Calculator
Select the mode based on what you want to compute.
How to read the chart
The horizontal axis represents the standardized Z statistic \(Z = (\hat{p}_B - \hat{p}_A)/SE_0\), where \(SE_0 = \sqrt{2\bar{p}(1-\bar{p})/n}\) and \(\bar{p}=(p_A+p_B)/2\).
- Blue curve (H₀): distribution of Z if there were no real difference between groups. It follows an N(0,1). The red zones are the rejection regions; their total area is α.
- Dashed orange curve (H₁): distribution of Z if the real difference were p_B − p_A. It is shifted relative to H₀. The green zone — the area under H₁ within the rejection region — is the power (1−β). The unshaded area under the orange curve within the acceptance region is β.
The larger the difference |p_B − p_A| or the sample size n, the more the curves separate and the greater the power.
Statistical foundation
For the two-proportion test \(H_0: p_A = p_B\) with equal sample sizes n, the test statistic is:
\( Z = \frac{\hat{p}_B - \hat{p}_A}{\sqrt{2\bar{p}(1-\bar{p})/n}} \;\sim\; N(0,\,1) \text{ under } H_0 \)
where \(\bar{p} = (p_A + p_B)/2\) is the pooled proportion. Under the alternative hypothesis \(H_1: p_B - p_A = \Delta\), the statistic is distributed approximately as:
\( Z \;\sim\; N\!\left(\delta,\; \sigma_1^2\right) \quad \text{where} \quad \delta = \frac{\Delta\sqrt{n}}{\sqrt{2\bar{p}(1-\bar{p})}},\quad \sigma_1 = \sqrt{\frac{p_A(1-p_A) + p_B(1-p_B)}{2\bar{p}(1-\bar{p})}} \)
Power formulas
- Two-tailed:
\( 1-\beta \;=\; 1 - \Phi\!\left(\frac{z_{\alpha/2}\cdot\sqrt{2\bar{p}(1-\bar{p})} - \Delta\sqrt{n}}{\sqrt{p_A(1-p_A)+p_B(1-p_B)}}\right) + \Phi\!\left(\frac{-z_{\alpha/2}\cdot\sqrt{2\bar{p}(1-\bar{p})} - \Delta\sqrt{n}}{\sqrt{p_A(1-p_A)+p_B(1-p_B)}}\right) \)
- Upper one-tailed (\(H_a: p_B > p_A\)):
\( 1-\beta \;=\; 1 - \Phi\!\left(\frac{z_\alpha\cdot\sqrt{2\bar{p}(1-\bar{p})} - \Delta\sqrt{n}}{\sqrt{p_A(1-p_A)+p_B(1-p_B)}}\right) \)
Sample size per group
\( n = \left\lceil \dfrac{\left(z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{1-\beta}\sqrt{p_A(1-p_A)+p_B(1-p_B)}\right)^2}{\Delta^2} \right\rceil \)
Minimum detectable effect (MDE)
The MDE is the smallest \(|\Delta|\) that achieves the target power with the available n. It is computed via binary search over \(\Delta\).
Worked example
An e-commerce team wants to test a new checkout flow (variant B) against the current one (group A). The conversion rate of A is p_A = 0.12, and B is expected to improve it to p_B = 0.138 (a +15% relative lift). The experiment is designed with α = 0.05, a two-tailed test, and n = 1,500 users per group.
Auxiliary parameters:
\( \bar{p} = (0.12 + 0.138)/2 = 0.129 \)
\( \text{NullScale} = \sqrt{2 \times 0.129 \times 0.871} = \sqrt{0.2247} \approx 0.4740 \)
\( \text{AltScale} = \sqrt{0.12 \times 0.88 + 0.138 \times 0.862} = \sqrt{0.1056 + 0.1190} = \sqrt{0.2246} \approx 0.4739 \)
Two-tailed power (\(z_{0.025} = 1.960\), \(n = 1500\)):
\( 1-\beta \approx 1 - \Phi\!\left(\frac{1.960 \times 0.4740 - 0.018\sqrt{1500}}{0.4739}\right) = 1 - \Phi\!\left(\frac{0.9290 - 0.6971}{0.4739}\right) = 1 - \Phi(0.489) \approx 0.313 \)
With n = 1,500 per group and a 15% lift, the power is only ≈ 31%: the experiment is severely underpowered for this effect. To reach 80% power, approximately 5,443 users per group would be needed.
This illustrates a common mistake in A/B testing: launching experiments without correctly sizing the sample to detect the effect of interest.
How to interpret the results
Power
A power of 80% means that if the real difference is p_B − p_A, the test will reject H₀ 80% of the time. Below 70%, the experiment has a high probability of missing real effects. Above 90%, the design is very conservative: you may be able to shorten the experiment's duration.
Type II error (β)
It is the probability of concluding there is "no difference" when there really is one. With β = 0.20, there is a 20% probability of missing a real improvement in variant B. This is the cost of running experiments with too small a sample.
Minimum detectable effect (MDE)
With the current sample, any difference smaller than the MDE will not be reliably detected. Compare the MDE with the minimum business-relevant impact: if the MDE is larger than that threshold, the experiment cannot make the right call even if it runs flawlessly.
Recommendations for A/B testing
- Always design the experiment before launching it. Stopping early as soon as you see significance inflates the type I error rate.
- Use a two-tailed test unless you can guarantee a priori that the variant can only improve the metric.
- With limited traffic, increase the minimum effect of interest or extend the duration instead of lowering α.
- For full sizing (sample size, power, and MDE all at once), use the full A/B power calculator.
Frequently asked questions
- What is statistical power in an A/B test? The probability of detecting a real difference between groups A and B when that difference exists. With power = 80%, the experiment will detect the effect in 8 out of every 10 repetitions.
- What is the MDE? The minimum detectable effect: the smallest difference between p_A and p_B that the experiment can detect at the target power. If the real effect is smaller, the test will probably miss it.
- How does the chart relate to power? The blue curve is H₀ (no difference). The red zones are the rejection regions (area = α). The orange curve is H₁ (the effect exists). The green area is the power: the fraction of H₁ that falls within the rejection region.
- When should I use a two-tailed test? In A/B testing, a two-tailed test is recommended because the variant could be better or worse than the control. A one-tailed test is only justified if you can guarantee a priori that the variant will never make the metric worse.