The real ROI of A/B testing: a calculator.
Testing infrastructure pays for itself on the second winner. Here's the math, the benchmarks, and a calculator to run your own numbers — with realistic assumptions, not vendor-inflated ones.
- ▸The median winning test in the dataset we analyzed produces $18K in annualized revenue.
- ▸A 25% hit rate on tests shipped means four winners per quarter at typical velocity — $72K in annualized revenue per quarter.
- ▸Testing infra pays back in under 60 days at any reasonable CR and traffic level.
- ▸The real cost of NOT testing is harder to calculate — but usually 3–5x the cost of testing.
Per-winner revenue impact.
Start from revenue per visitor (RPV), your hit rate on tests, and the average lift on winners. The math is deceptively simple: a 10% lift on a page seeing 100K visits and $0.50 RPV is $5K/month in incremental revenue. Annualized: $60K. One winner pays for most testing platforms for two years.
- ▸Incremental monthly revenue = traffic × RPV × lift.
- ▸Example: 100K visits × $0.50 RPV × 10% lift = $5K/mo.
- ▸Annualized per winner: ~$60K.
- ▸Most testing platforms cost <$24K/year. One winner = 2+ years paid off.
Realistic benchmarks.
Don't trust the vendor benchmarks ('60% of our customers see positive tests!'). Real hit rates for disciplined testing programs run 20–35%. Below that, your hypothesis quality is off. Above that, you're not being ambitious enough — the upside in A/B testing is fat tails, which requires some losers.
- ▸20–35% hit rate is the realistic band for disciplined programs.
- ▸Below 20%: hypothesis quality is the problem. Fix the inputs.
- ▸Above 40%: you're being too safe. Test more aggressively.
- ▸Losers are not failures — they are the cost of the winners.
Why shipping more matters more.
The single biggest driver of total ROI isn't hit rate — it's velocity. A team shipping 8 tests per quarter at a 25% hit rate produces 2 winners per quarter. A team shipping 2 tests per quarter at a 40% hit rate produces 0.8 winners per quarter. The fast team produces 2.5x the value.
- ▸Winners per quarter = tests shipped × hit rate.
- ▸Velocity compounds: 8 tests × 25% = 2 winners ($120K annualized).
- ▸2 tests × 40% = 0.8 winners ($48K annualized).
- ▸Hit rate maxes out around 35%. Velocity has no ceiling.
The invisible line item.
Every growth team has an invisible line item: the revenue that was left on the table because you didn't test. It's not zero, but it's hard to calculate. A rough estimate: the revenue from the tests you ran times the multiplier (5–10x) for the tests you didn't run. For most teams, that's 2–3 quarters of their current testing output.
- ▸Invisible cost: lost revenue from tests never shipped.
- ▸Rough estimate: 5–10x your current testing output.
- ▸This is why velocity matters more than anything else.
Use the interactive ROI calculator.
Google Doc, Notion, and PDF versions. Shared with you in one email.
Download →Related for your role
ALL RESOURCES →Stack Consolidation ROI Calculator
Enter what you pay Optimizely, Crayon, Hotjar, and Ahrefs today. See what Optimize Pilot would cost instead — and how many headcount the delta covers.
The Testing Velocity Playbook
How top CRO teams ship 6–8 experiments per quarter without sacrificing statistical rigor. Includes the idea-to-ship workflow we see in the top 10%.
Experiment Brief Template
The one-page brief every test should start from: hypothesis, variants, sample size, success metric, guardrails. Notion + Google Doc versions inside.
The math works. Start testing.
Navigator AI's ranked queue + Flight Deck's Bayesian stats = 3–4 tests shipped inside the 14-day trial. That's most of a quarter's output for the median team.