Most A/B Tests Don't Need More Traffic
The growth team paused four tests after three weeks with no significance. Traffic wasn't the problem — their read on what counts as a real difference was.
In October, the growth team paused four tests because none had reached statistical significance after three weeks. They blamed the traffic. The traffic wasn't the problem. Their calibration on what counts as a meaningful difference had been wrong since the program started.
Say a page gets 4,000 weekly visitors and you want a 3% lift detected at 80% statistical power: you need on the order of 90,000 visitors per variant, which is eleven weeks of traffic, not three. That was the first test. The assumption that three weeks should produce a verdict had nothing to do with the underlying math.
The other three tests had the same shape. Each hunted a small effect against modest traffic, with no power calculation run first. Each got paused the moment the team lost patience. Each would have needed two to three months to mean anything. None of them got it.
Most growth-stage A/B testing programs run too many tests against too little traffic, chasing too small an effect. The result is statistical theater: tests that almost reach significance, get interpreted anyway, and produce decisions the data doesn't support. The team gets the activity of running experiments without the learning of producing reliable answers.
Compute statistical power before launching any test. The team agrees on a minimum detectable effect they care about, usually ten percent or larger at small-company traffic, and checks whether the test can find it in a reasonable runtime. Tests that can't clear that bar don't launch. Cadence drops by roughly a third. The tests that do launch produce answers you can trust.
The uncomfortable part is that small-traffic companies can't detect small lifts. A two percent conversion bump is invisible to the underlying statistics. The team can accept that and run fewer, larger tests, or pretend the math doesn't apply and ship decisions calibrated to noise. Most pick the second, and the dashboards fill with wins that never replicate. It's the same trap as a homepage metric that measures comfort, and it ends the way a test that moved the wrong number does.