A/B Test Significance & Lift Calculator

Your test ran for two weeks. The variant shows a 12% conversion rate versus the control’s 10% — a 20% relative lift. The PM is ready to ship. But is that 2-percentage-point gap a genuine improvement, or could random variation explain it? An A/B test significance calculator takes the visitor counts and conversion counts from both arms, computes a z-statistic, and returns a p-value that answers exactly that question. If p is below your pre-set α (usually 0.05), the difference is statistically significant — meaning it is unlikely to have appeared by chance alone.

The mistake that ships bad changes: looking only at the lift percentage and ignoring significance. A 20% lift with 50 visitors per arm is meaningless noise. The same 20% lift with 5,000 visitors per arm is almost certainly real. The calculator forces you to consider both the magnitude and the reliability of the result before making a decision.

A p-value tells you whether the difference is likely real. A confidence interval tells you how big it might plausibly be. If control converts at 10% and variant at 12%, the 95% CI for the absolute difference might be [0.3%, 3.7%]. That means you are 95% confident the true lift is between 0.3 and 3.7 percentage points. The point estimate is 2 points, but it could be as small as 0.3 — barely worth the engineering effort — or as large as 3.7.

CIs are more useful than p-values for decision-making because they communicate uncertainty about the size of the effect, not just its existence. A significant result with CI [0.01%, 0.5%] says: yes, there is a difference, but it is probably tiny. A significant result with CI [1.5%, 4.2%] says: the difference is real and big enough to matter. Always look at both.

For proportions, the CI uses unpooled standard errors (each arm’s variance estimated separately). The p-value uses the pooled standard error (assuming no difference under the null). This is why the CI and the p-value can occasionally seem to disagree near the boundary — a p-value just above 0.05 with a CI that just barely excludes zero. The discrepancy is a rounding artefact of using two slightly different SE estimates.

Lift is the relative improvement: (variant − control) / control × 100. Going from 10% to 12% is a 20% relative lift but only a 2-percentage-point absolute difference. Which number you report changes how the result feels. “We improved conversion by 20%” sounds impressive. “We improved conversion by 2 points” sounds incremental. Both are the same result.

In practice, absolute difference determines how many additional conversions you get. If you have 100,000 visitors per month, 2 extra percentage points means 2,000 more conversions. Whether that justifies the change depends on the revenue per conversion, the cost of the variant, and the opportunity cost of not testing something else. Relative lift is useful for comparing across different baseline rates (a 20% lift on a 2% baseline is harder to achieve than on a 50% baseline), but absolute difference ties directly to business impact.

One trap: computing lift with the wrong denominator. Lift is always relative to the control, not to the variant. (12 − 10) / 10 = 20%. If you accidentally divide by the variant: (12 − 10) / 12 = 16.7%. Small difference, but it matters when you are reporting to stakeholders who will hold you to the number.

The right time to check significance is after you have collected the sample size you committed to before launching. Checking earlier and stopping when p < 0.05 is “peeking,” and it inflates your false-positive rate well above 5%. If you peek 10 times during a test, your actual α is closer to 20–30%, meaning one in four “significant” results is noise.

Duration matters beyond sample size. A test that reaches the required n in two days captures only weekend traffic (or only weekday traffic, depending on when it started). User behaviour shifts between weekdays and weekends, mornings and evenings, paydays and mid-month. Run for at least one full week — ideally two — to average over these cycles, even if you hit the target n sooner.

If your test reaches full sample and the result is not significant, resist the urge to “extend and see.” Extending without pre-specifying the new sample size is just peeking with extra steps. Either accept the inconclusive result, increase MDE for the next iteration, or redesign the variant with a bigger expected effect. Inconclusive tests are not failures — they are information that the difference, if it exists, is smaller than you planned for.

We checked significance daily and stopped on day 4 when p hit 0.03. Is the result valid?
Probably not at the α you intended. Each look is an independent chance for a false positive. With daily checks over 14 days, your effective α is roughly 25%, not 5%. If you need to monitor results mid-flight, use a sequential testing framework (like always-valid p-values or group sequential boundaries) that adjusts for repeated looks.

We tested 8 metrics and one was significant at p = 0.04. Can we claim a win?
Testing 8 metrics at α = 0.05 gives a roughly 34% chance that at least one is significant by chance alone (1 − 0.95⁸). Apply a Bonferroni correction: divide α by the number of metrics. With 8 metrics, your per-metric threshold is 0.05 / 8 = 0.00625. At p = 0.04, the result would not survive the correction. Designate one primary metric before launch and treat the rest as directional.

The variant won on conversion but lost on revenue. Which metric wins?
Neither “wins” automatically. This is a sign that the variant attracted more low-value conversions. Decide on the primary metric before the test, not after. If conversion was primary, the variant wins — but investigate the revenue drop before shipping. If revenue was primary, the variant lost. Post-hoc cherry-picking whichever metric looks best is not analysis; it is confirmation bias.

My p-value is 0.051. So close — can I round to significant?
No. The threshold is arbitrary, but once you set it, respect it. A p-value of 0.051 means you do not have enough evidence at α = 0.05. Report the result honestly: “The difference was not statistically significant at the 5% level (p = 0.051), though the observed lift was X%.” If the lift looks promising, increase sample size and re-test.

Three equations handle the full analysis:

Units note: p_A and p_B are proportions (0–1). c is conversion count, n is visitor count. The pooled SE is used for the hypothesis test; the unpooled SE is used for the confidence interval around the difference.

Scenario: You tested a simplified signup form. Control: 10,000 visitors, 820 signups (8.20%). Variant: 10,000 visitors, 910 signups (9.10%). α = 0.05, two-sided.

Step 1 — Pooled proportion and SE.
p̂ = (820+910)/(10,000+10,000) = 1,730/20,000 = 0.0865.
SE = √[0.0865 × 0.9135 × (1/10,000 + 1/10,000)] = √[0.0790 × 0.0002] = √0.0000158 = 0.00397.

Step 2 — z-statistic.
z = (0.0910 − 0.0820) / 0.00397 = 0.0090 / 0.00397 = 2.27.

Step 3 — p-value and significance.
Two-sided p = 2×(1−Φ(2.27)) = 2×0.0116 = 0.023. Since 0.023 < 0.05, the result is statistically significant.

Step 4 — Lift and CI.
Relative lift: (9.10−8.20)/8.20 × 100 = 11.0%. The 95% CI for the absolute difference (using unpooled SE) is approximately [0.12%, 1.68%]. The lift is real but could be as small as 0.12 points — worth monitoring post-launch to confirm the effect holds.