Analyzing Experiment Results
Wrapper → Experiments tracks hypothesis, variations, and traffic-allocation history — but it doesn't currently render a per-variation performance report with a significance indicator. Analyze experiments in Dashboard → Monetization, breaking down by experiment variation, and apply a bit of human judgment.
Where to look
Dashboard → Monetization
The primary surface for experiment analysis. Once an experiment is running, the wrapper tags each impression with which variation served it; that label flows into reporting.
- Go to Dashboard → Monetization
- Break down by experiment variation (or the equivalent dimension for your setup)
- Compare control vs. variation across your metrics
Wrapper → Experiments
The experiment detail page won't show a results table, but it will show:
- Current traffic allocation (and the Allocation History of how it's changed)
- Hypothesis and schedule
- Running status
Use this to cross-check that the experiment is actually live and getting the traffic share you intended.
Which metrics matter
| Metric | What it tells you |
|---|---|
| Revenue / 1000 sessions (RPS) | The best single success metric — accounts for both price and volume |
| CPM | Price per filled impression |
| Fill rate | % of requests that filled |
| Impressions | Total ads served |
| Bid rate | % of requests that got at least one bid |
| Core Web Vitals | Whether the change moved page speed (from Dashboard → Core Web Vitals) |
For most experiments, lead with revenue per 1000 sessions and cross-check with CWV to make sure a revenue lift isn't coming at the cost of a page-speed regression.
Reading the comparison
Pull control vs. variation numbers into a simple table. This is roughly what you're looking for:
| Metric | Control | Variation | Delta |
|---|---|---|---|
| Revenue / 1000 sessions | $12.50 | $14.20 | +13.6% |
| CPM | $2.10 | $2.85 | +35.7% |
| Fill rate | 85% | 72% | −15.3% |
| LCP (p75) | 2.1s | 2.2s | +0.1s |
Read the trade-off: higher CPM, lower fill, higher net revenue, no CWV regression → variation looks like a win.
How long to wait
The portal doesn't compute statistical significance for you today. Rough heuristics below.
Typical target: 7–14 days of running time, covering at least one full weekday/weekend cycle. Longer for low-traffic sites.
Rough rules of thumb for "is this signal real":
- Pre-14 days, or <1000 impressions/variation: Don't trust the numbers much. Early results can flip.
- Delta < ~3%: Probably inside the noise floor for most setups. Needs significantly more data or a larger change to be confident.
- Delta > 10% and stable for 7+ days: Likely a real effect.
- Delta oscillates (positive one day, negative the next): Variance, not signal. Wait.
If you need a proper significance calculation, pull the raw numbers (impressions, revenue per variation) from Dashboard → Monetization and plug them into any two-sample significance calculator.
Patterns in the read
Clear winner
Revenue / 1000 sessions: Control $12.50 → Variation $14.20 (+13.6%)
CWV: no regression
Stable for 10+ days
Action: stop the experiment, promote the variation. Typical path — copy the variation's source config into a standalone config and ship a release, then archive the experiment.
Clear loser
Revenue / 1000 sessions: Control $12.50 → Variation $10.80 (−13.6%)
Stable for 7+ days
Action: stop the experiment (archive), keep the control.
No clear difference
Revenue / 1000 sessions: Control $12.50 → Variation $12.65 (+1.2%)
Noisy across the run
Options:
- Run longer for more data
- The change genuinely has no effect — don't ship it
- Consider whether the hypothesis was worth testing; this is a signal to pick bigger bets
Mixed signals
Revenue / 1000 sessions: +8%
CPM: +25%
Fill rate: −15%
LCP (p75): +0.2s
Higher CPM, lower fill, net revenue up, but CWV slightly worse. Judgment call — depends on whether your CWV headroom tolerates the regression.
Segment the read
A variation can win on average and still hurt a specific segment. Break down Dashboard → Monetization by:
| Segment | Why it matters |
|---|---|
| Device | Mobile often behaves differently from desktop (different CPM floors, different viewability) |
| Layout | Article-only changes shouldn't be judged on homepage data |
| Geography | If your traffic mix shifted during the experiment window, geo can explain outliers |
| Browser | Safari (limited ITP) vs. Chrome can show identity-related variance |
If a winner is clean in aggregate but awful on one segment, consider a layout-scoped rollout rather than a global change.
Common mistakes
| Mistake | Why it's wrong | Fix |
|---|---|---|
| Ending too early | Early numbers flip frequently | Stick to 7–14 day minimum |
| Looking only at one metric | Misses trade-offs | Always cross-check CPM, fill, revenue, CWV together |
| Ignoring segments | Winners can hide losers | Segment by device and layout |
| Confirmation bias | Seeing the result you hoped for | Write the hypothesis before you look at data, not after |
Using the AI Assistant
| You ask | What happens |
|---|---|
| "How is my sidebar floor test doing?" | Summary of metrics for that experiment from Monetization data |
| "Compare variation vs control" | Delta table across key metrics |
| "Should I end the experiment?" | Heuristic recommendation based on the data |
The Assistant reads the same Monetization data you do — it's useful for speed, not for a deeper view than the dashboard provides.
After the decision
- Winner → copy the variation's source config into a standalone config, ship it as a release, archive the experiment. Watch Monetization for a few days to confirm the effect holds at 100% traffic.
- Loser → archive the experiment. Traffic returns to the control on the next release.
- Inconclusive → either archive and move on, or redesign the test (bigger change, more traffic allocation, longer run).