A/B testing was built for sites with hundreds of thousands of visitors. If yours has fewer, the method does not fail loudly. It just never finishes. This page shows where the line is, and how Optimize Pilot measures growth work on the wrong side of it.
Two things decide it: how often the page converts now, and how big a difference you are trying to see. The table shows the monthly visitors one page needs for a test to finish in 8 weeks.
| PAGE CONVERTS AT | 10% LIFT | 20% LIFT | 30% LIFT | 50% LIFT |
|---|---|---|---|---|
| 1%Typical B2B lead form | 175,000 | 46,000 | 21,000 | 8,300 |
| 2%A good B2B lead form | 86,000 | 23,000 | 10,500 | 4,100 |
| 5%Strong landing page | 33,000 | 8,700 | 4,000 | 1,600 |
| 10%A proxy goal, such as a click on the main button | 16,000 | 4,100 | 1,900 | 700 |
Monthly visitors to the page being tested, for a two-version test to finish in 8 weeks at 95% confidence and 80% power. Halve the time and the traffic needed doubles.
Read your own row. A B2B page that turns 2% of visitors into leads needs about 86,000 visitors a month to detect a 10% improvement. Most single changes, a new headline or a different button, improve a page by less than that. A site with 20,000 visitors a month across all its pages has no page that qualifies.
So the test runs for months, someone calls it early on a hunch, and the team concludes that testing does not work for them. The testing was fine. The site was too small for the question it was asked.
Look along any row of the table. Detecting a 50% lift takes about a twentieth of the visitors a 10% lift does. A small site should be rebuilding a page’s argument, not adjusting its button. This is why the Pilot builds whole sections and pages.
Look down the table. Moving from a goal 2% of visitors reach to one 10% reach cuts the traffic needed about fivefold. The risk is measuring something easy to move that predicts nothing, so every such signal is validated against the real goal first.
Hold back a share of visitors from every change and compare everyone else with them. All your traffic counts toward one answer: did the program work? Ten results too small to call on their own become one result you can.
For every change, Flight Path works out what the page could detect in 2, 4 and 8 weeks and sends the change down one of four routes.
If a page has the visitors, the change runs as a proper test against the current version. Optimize Pilot also uses what it already knows about each visitor to cut the number needed, and states the result as the risk of shipping, in plain words.
Typically pages above about 20,000 visitors a month, or bold changes on smaller ones.
Few visitors fill in a form, but many more reach the pricing section or start the form. Measuring a step that happens five times as often needs about a fifth of the traffic. A signal is only used after it has been checked against your real goal, on your own history and on similar sites.
Pages with a few thousand visitors a month.
Instead of splitting visitors evenly until the end, more of them are sent to the version that is doing better. You give up a clean verdict on small differences and lose less while finding out.
When being wrong for eight weeks would cost real leads.
Low-risk changes on pages too small to test are simply made. They are counted in the program result: a share of all your visitors never sees any change, and everyone else is compared with them. That comparison uses the traffic of the whole site, so it can find an answer where no single page could.
Most pages on most B2B sites.
A holdout is slow too. A small holdout protects more of your visitors and takes longer to answer. Detecting a 20% program lift at a 2% conversion rate takes about 42,000 visitors with an even split and about 120,000 with a 10% holdout. Small sites start with a larger holdout, and we show the expected wait before you begin.
A program result is not a verdict on each change. It says the work as a whole paid off. Which changes carried it becomes clearer over months, as results of the same type pool together.
Forecasts start wide. Until a site has a history, each forecast is a broad range. We publish how often real results landed inside the ranges, so you can judge them.
See what your site can measure, and let the Pilot do the work.
14-day free trial. Card required, US$0 today. Cancel anytime.