Experiments Overview
Test configuration changes safely before rolling out to all users.
What are experiments?
Experiments let you A/B test changes on a percentage of your traffic:
Why use experiments?
| Direct deployment | Experiments |
|---|---|
| All-or-nothing | Test on % of traffic |
| Risk to 100% of users | Limited risk |
| Hard to measure impact | Clear before/after comparison |
| Can't undo easily | End anytime |
What can you test?
| Test type | Example |
|---|---|
| Floor prices | $0.75 vs $1.25 |
| Timeouts | 1200ms vs 1500ms |
| Ad slots | 4 vs 5 in-content slots |
| Layouts | Different slot arrangements |
| Partners | With/without a partner |
| Refresh | 45s vs 60s interval |
| Lazy loading | On vs off for specific slots |
Experiment workflow
1. Create experiment
Define what you're testing:
- Control (current configuration)
- Variation (new configuration)
- Traffic split (% to each)
2. Run experiment
Let it run for sufficient time:
- Minimum 7 days
- Ideally 14 days
- Enough traffic for significance
3. Analyze results
Compare metrics:
- Revenue per 1000 sessions
- CPM
- Fill rate
- Page speed (if relevant)
4. Act on results
- Winner found: Deploy winner to 100%
- No clear winner: Run longer or try different test
- Loser clear: End experiment, keep current
Common tasks
| Task | Link |
|---|---|
| Create experiment | Creating experiments → |
| Monitor results | Analyzing results → |
| End experiment | Ending experiments → |
Quick actions with AI Assistant
| You say | What happens |
|---|---|
| "Create an experiment for floor prices" | Guided experiment creation |
| "How is my floor price experiment doing?" | Shows current results |
| "End the timeout experiment" | Marks the experiment complete |
| "Roll out the winning variation" | Walks you through copying the winning config into a release to deploy |
Best practices
Do
- Test one variable at a time
- Run for at least 7 days
- Wait for statistical significance
- Document your hypothesis
Don't
- Test too many things at once
- End too early
- Ignore negative results
- Skip the analysis