How is this different from the general two-proportion test?

No mathematical difference: the z-test is the same. This calculator adds the confidence interval for the difference, Yates' correction and A/B-specific metrics (relative lift).

Is it valid with very low conversion rates?

Yes, but it requires larger sample sizes for the normal approximation to be reliable. As a rule of thumb, you need n·p ≥ 5 and n·(1−p) ≥ 5 in each group.

Should I use a two-tailed or one-tailed test?

If you don't know beforehand which direction the effect could go, use two-tailed. The one-tailed test has more power, but is only legitimate when the direction was fixed a priori.

Does a p-value of 0.04 mean B is better with 96% probability?

No. The p-value measures the probability of observing a statistic as extreme or more extreme if H₀ were true, not the probability that B is better.

Conversion z-test calculator for A/B testing

Calculator

Enter the successes and sizes of each group to get the p-value, the z statistic and the confidence interval for the difference.

Conversions group A (x_A)

Visitors group A (n_A)

Conversions group B (x_B)

Visitors group B (n_B)

Test type

Significance level (α)

Apply Yates' continuity correction

Result pending…

Hypotheses and test statistic

The test contrasts \(H_0: p_A = p_B\) against the selected alternative. Under \(H_0\), both groups share the same population proportion, so the pooled proportion \(\hat{p}\) is used to estimate the standard error.

\( z = \dfrac{\hat{p}_B - \hat{p}_A}{\sqrt{\hat{p}(1-\hat{p})\!\left(\tfrac{1}{n_A}+\tfrac{1}{n_B}\right)}}, \quad \hat{p}=\dfrac{x_A+x_B}{n_A+n_B} \)

Yates' continuity correction

With small samples, the discrete distribution of the statistic is better approximated by the normal by subtracting the correction:

\( z_{\text{corr}} = \dfrac{\operatorname{sgn}(\hat{p}_B - \hat{p}_A)\cdot\max\!\left(0,\;\left|\hat{p}_B - \hat{p}_A\right| - \tfrac{1}{2}\!\left(\tfrac{1}{n_A}+\tfrac{1}{n_B}\right)\right)}{\sqrt{\hat{p}(1-\hat{p})\!\left(\tfrac{1}{n_A}+\tfrac{1}{n_B}\right)}} \)

Confidence interval for the difference

The CI for \(\Delta = p_B - p_A\) is built with the Wald standard error (unpooled), more conservative than the one used in the test:

\( \Delta \pm z_{\alpha/2}\sqrt{\dfrac{\hat{p}_A(1-\hat{p}_A)}{n_A} + \dfrac{\hat{p}_B(1-\hat{p}_B)}{n_B}} \)

Worked example: z-test in A/B testing

With A = 10,000 visits and 420 conversions, and B = 10,000 visits and 480 conversions, the tool compares two independent proportions. Use a two-tailed test if you only want to detect a difference, or a one-tailed test only if the improvement hypothesis was fixed before seeing the data.

When to use Yates' correction

The continuity correction improves the approximation to the normal distribution when sample sizes are small (generally n < 50 per group) or when the proportion is close to 0 or 1. With large samples it produces results nearly identical to the uncorrected test and is slightly more conservative.

How to interpret the result

The z statistic measures how many standard deviations the observed difference \(\hat{p}_B - \hat{p}_A\) is away from zero under the null hypothesis. The larger \(|z|\) is in absolute value, the further the data are from what we would expect if there were no real difference. The p-value translates that statistic into a probability: it is the probability of obtaining a z as extreme or more extreme in repeated experiments where H₀ were true. A p-value < α (typically 0.05) does not mean that B is better with probability 1 − p; it means that the evidence against the hypothesis of equality is strong enough according to the chosen threshold.

The relative lift \(((\hat{p}_B - \hat{p}_A) / \hat{p}_A) \times 100\,\%\) expresses B's improvement as a percentage over A's baseline rate. It is the most direct metric for judging the practical relevance of the result: an experiment can be statistically significant with a lift of 0.5% that has no real business impact, and another may not reach significance but show a 15% lift that deserves further investigation with more samples.

The confidence interval for \(\Delta = p_B - p_A\) complements the p-value with information about the magnitude and uncertainty of the difference. If the CI does not include zero, the difference is significant at the chosen level. More importantly, the CI bounds indicate the plausible range of the true difference: if even the lower bound of the CI represents an improvement large enough for your business, you can confidently roll out B. If the CI includes both negative and positive values, the direction of the effect is uncertain and the experiment should be extended.

Frequently asked questions

How is this different from the general two-proportion test? No mathematical difference: the z-test is the same. This calculator adds the confidence interval for the difference, Yates' correction and A/B-specific metrics (relative lift).
Is it valid with very low conversion rates? Yes, but it requires larger sample sizes for the normal approximation to be reliable. As a rule of thumb, you need n·p ≥ 5 and n·(1−p) ≥ 5 in each group.
Should I use a two-tailed or one-tailed test? If you don't know beforehand which direction the effect could go, use two-tailed. The one-tailed test has more power, but is only legitimate when the direction was fixed a priori.
Does a p-value of 0.04 mean B is better with 96% probability? No. The p-value measures the probability of observing a statistic as extreme or more extreme if H₀ were true, not the probability that B is better.