A/B test calculator: is the result significant?

A/B test significance calculator

What is an A/B test?

In an A/B test, visitors are split randomly into two groups. Group A sees the current version, group B the changed one. Then you compare the conversion rate of the two groups.

You decide what counts as a conversion, such as a sign-up or a purchase. In a checkout it is usually a completed order.

Two rates are almost never the same. The real question is whether the difference is real or chance. The calculator answers it with the two-proportion z-test.

How it's calculated

The calculator uses the formulas from the Wikipedia article on the two-proportion z-test. Say group A has n_A visitors and x_A conversions, and group B has n_B and x_B.

1. Conversion rates: p_A = x_A/n_A and p_B = x_B/n_B.
2. Pooled rate: p̄ = (x_A+x_B)/(n_A+n_B). The test assumes there is no difference between the versions, so it combines both groups into one rate.
3. Standard error: SE = √(p̄(1−p̄)(1/n_A+1/n_B)).
4. Test statistic: z = (p_B−p_A)/SE.
5. Two-sided p-value: 2 × (1 − Φ(|z|)). Φ is the cumulative distribution function of the standard normal distribution.

To calculate Φ in the browser I use one of Abramowitz and Stegun's approximations. It computes the error function (erf) with an error of at most 1.5 × 10⁻⁷. You get to the normal distribution through Φ(z) = ½(1 + erf(z/√2)).

The confidence interval doesn't use the pooled rate, but the variance of each group separately.

(p_B−p_A) ± 1.96 × √(p_A(1−p_A)/n_A + p_B(1−p_B)/n_B).

For the relative difference, (p_B−p_A)/p_A is calculated and shown as a percentage.

Because the two calculations use different standard errors, their results can differ slightly. The Wikipedia example is a borderline case: 1,000 visitors per group, 120 conversions in one and 150 in the other. The p-value is about 0.0497, just under 0.05, and the confidence interval only just excludes zero.

I don't read a result like that as “definitely won”. Enter the same numbers in the calculator and see for yourself.

How many visitors do you need?

Work out before you start how many visitors you need per group. Two numbers are enough: your current conversion rate and the smallest difference you want to detect.

The formula comes from the same Wikipedia article. Let p₁ be the baseline rate and p₂ the target rate. With two groups of the same size, the number of visitors per group is:

n = (z_α/2+z_β)² × (p₁(1−p₁)+p₂(1−p₂)) / (p₁−p₂)².

For α = 0.05, z_α/2 is about 1.96. For 80% power, z_β is taken as about 0.84. Example: with a baseline of 12% and a target of 15%, you need 2,033 visitors per group.

For a quick check, Evan Miller gives the rule n = 16σ²/δ². For a conversion rate, σ² = p(1−p), and δ is the smallest difference you're looking for.

Wikipedia calls the same rule Lehr's rule. It gives the number needed per group for a two-sided test with α = 0.05 and 80% power. For 12% and 3 points that is about 1,878, close to the formula's 2,033.

Small differences need a lot of traffic. To detect a rise of 1 point at a rate of 5%, you need 8,155 visitors per group.

When to read the result: peeking too early

The most common mistake is looking at the result every day and stopping the test on the first day it is significant. That's called peeking. Evan Miller shows in his 2010 piece that it inflates the false positive rate.

According to Miller's table, if you check the test 10 times and really want to keep the false positive rate at 5%, you have to lower the threshold to 1%. With 5 checks the threshold drops to 1.4%, with 2 checks to 2.2%.

In the piece's extreme example, a new logo with no effect at all is tested. After every observation someone looks, and the test stops at 5% significance or after 150 observations. The chance of finding a false significant result rises to 26.1%.

Miller's solution is simple: fix the sample size in advance and wait until the test is done. The sample size calculator above gives you that number. Don't decide by the badge on this page before you reach it.

My note

With teams that come to me I often see the same thing: the colour of a button gets tested, and the form that is the real problem stays as it is. A test is only as good as the idea behind it.

That's why I first look for where users get stuck. A usability test tells you what to change. The A/B test measures whether the change works.

With little traffic it can take weeks to reach the number of visitors you need. Then don't chase small differences; fix the big problems. If you don't know which flow to review first, I review up to 10 flows in UX Audit Pro.