ARTICLE8 MINFOR CRO MANAGERSFOR GROWTH LEADERS

The real ROI of A/B testing: a calculator.

Testing infrastructure pays for itself on the second winner. Here's the math, the benchmarks, and a calculator to run your own numbers — with realistic assumptions, not vendor-inflated ones.

◆ TL;DR

▸A single winning test can produce tens of thousands of dollars in annualized revenue — enough to pay for testing infrastructure many times over.
▸At a realistic hit rate and steady velocity, a handful of winners per quarter adds up quickly. Run your own numbers below.
▸Testing infra pays back fast at any reasonable conversion rate and traffic level.
▸The real cost of NOT testing is harder to calculate — but it is rarely zero, and usually larger than the cost of testing.

01The math

Per-winner revenue impact.

Start from revenue per visitor (RPV), your hit rate on tests, and the average lift on winners. The math is deceptively simple: a 10% lift on a page seeing 100K visits and $0.50 RPV is $5K/month in incremental revenue. Annualized: $60K. One winner pays for most testing platforms for two years.

▸Incremental monthly revenue = traffic × RPV × lift.
▸Example: 100K visits × $0.50 RPV × 10% lift = $5K/mo.
▸Annualized per winner: ~$60K.
▸Most testing platforms cost <$24K/year. One winner = 2+ years paid off.

02Hit rate

Realistic benchmarks.

Don't trust the vendor benchmarks ('60% of our customers see positive tests!'). Real hit rates for disciplined testing programs run 20–35%. Below that, your hypothesis quality is off. Above that, you're not being ambitious enough — the upside in A/B testing is fat tails, which requires some losers.

▸20–35% hit rate is the realistic band for disciplined programs.
▸Below 20%: hypothesis quality is the problem. Fix the inputs.
▸Above 40%: you're being too safe. Test more aggressively.
▸Losers are not failures — they are the cost of the winners.

03Testing velocity

Why shipping more matters more.

The single biggest driver of total ROI isn't hit rate — it's velocity. A team shipping 8 tests per quarter at a 25% hit rate produces 2 winners per quarter. A team shipping 2 tests per quarter at a 40% hit rate produces 0.8 winners per quarter. The fast team produces 2.5x the value.

▸Winners per quarter = tests shipped × hit rate.
▸Velocity compounds: 8 tests × 25% = 2 winners ($120K annualized).
▸2 tests × 40% = 0.8 winners ($48K annualized).
▸Hit rate maxes out around 35%. Velocity has no ceiling.

04The cost of not testing

The invisible line item.

Every growth team has an invisible line item: the revenue that was left on the table because you didn't test. It's not zero, but it's hard to calculate. A rough estimate: the revenue from the tests you ran times the multiplier (5–10x) for the tests you didn't run. For most teams, that's 2–3 quarters of their current testing output.

▸Invisible cost: lost revenue from tests never shipped.
▸Rough estimate: 5–10x your current testing output.
▸This is why velocity matters more than anything else.

◆ TAKE IT WITH YOU

Use the interactive ROI calculator.

Google Doc, Notion, and PDF versions. Shared with you in one email.

Download →

Related for your role

ALL RESOURCES →

CALCULATOR

Stack Consolidation ROI Calculator

Enter what you pay Optimizely, Crayon, Hotjar, and Ahrefs today. See what Optimize Pilot would cost instead — and how many headcount the delta covers.

Run the numbers →

CALCULATOR

A/B Test Sample Size Calculator

Enter your baseline conversion rate, minimum detectable effect, and weekly traffic. Get the required sample size per variant and an estimated test duration.

Size your test →

PLAYBOOK18 MIN

The Testing Velocity Playbook

How high-performing CRO teams ship more experiments without sacrificing statistical rigor. Includes the idea-to-ship workflow we see work in practice.

OPEN →

◉ RUN THE NUMBERS

The math works. Start testing.

Navigator AI's ranked queue removes the prioritization meeting; Flight Deck's Bayesian stats tell you when a test has decided. 90-day money-back guarantee, no questions asked.

Book a 15-min stack audit →

◆ Loading…

Per-winner revenue impact.

▸Incremental monthly revenue = traffic × RPV × lift.

▸Example: 100K visits × $0.50 RPV × 10% lift = $5K/mo.

▸Annualized per winner: ~$60K.

▸Most testing platforms cost <$24K/year. One winner = 2+ years paid off.

Realistic benchmarks.

▸20–35% hit rate is the realistic band for disciplined programs.

▸Below 20%: hypothesis quality is the problem. Fix the inputs.

▸Above 40%: you're being too safe. Test more aggressively.

▸Losers are not failures — they are the cost of the winners.

Why shipping more matters more.

▸Winners per quarter = tests shipped × hit rate.

▸Velocity compounds: 8 tests × 25% = 2 winners ($120K annualized).

▸2 tests × 40% = 0.8 winners ($48K annualized).

▸Hit rate maxes out around 35%. Velocity has no ceiling.

The invisible line item.

▸Invisible cost: lost revenue from tests never shipped.

▸Rough estimate: 5–10x your current testing output.

▸This is why velocity matters more than anything else.