Experiments Overview
Test configuration changes safely before rolling out to all users.
What are experiments?
Experiments let you A/B test changes on a percentage of your traffic:
Why use experiments?
| Direct deployment | Experiments |
|---|---|
| All-or-nothing | Test on % of traffic |
| Risk to 100% of users | Limited risk |
| Hard to measure impact | Clear before/after comparison |
| Can't undo easily | End anytime |
What can you test?
| Test type | Example |
|---|---|
| Floor prices | $0.75 vs $1.25 |
| Timeouts | 1200ms vs 1500ms |
| Ad slots | 4 vs 5 in-content slots |
| Layouts | Different slot arrangements |
| Partners | With/without a partner |
| Refresh | 45s vs 60s interval |
| Lazy loading | On vs off for specific slots |
Experiment workflow
1. Create experiment
Define what you're testing:
- Control (current configuration)
- Variation (new configuration)
- Traffic split (% to each)
2. Run experiment
Let it run for sufficient time:
- Minimum 7 days
- Ideally 14 days
- Enough traffic for significance
3. Analyze results
Compare metrics:
- Revenue per 1000 sessions
- CPM
- Fill rate
- Page speed (if relevant)
4. Act on results
- Winner found: Deploy winner to 100%
- No clear winner: Run longer or try different test
- Loser clear: End experiment, keep current
Ending an experiment archives it (it moves to the Archived state) — it doesn't delete it, and you won't see a separate "completed" state. The archived experiment stays in your history.
Common tasks
| Task | Link |
|---|---|
| Create experiment | Creating experiments → |
| Monitor results | Analyzing results → |
| End (archive) an experiment | Experiment lifecycle → |
Quick actions with the AI Assistant
The AI Assistant can scaffold and manage experiments from plain language — creating an A/B test, reporting how one is doing, ending it, or rolling the winning variation into a release. Review the drafted configuration before deploying. See the AI Assistant guide.
Best practices
Do
- Test one variable at a time
- Run for at least 7 days
- Wait for statistical significance
- Document your hypothesis
Don't
- Test too many things at once
- End too early
- Ignore negative results
- Skip the analysis