Calculator
Select the mode based on what you want to calculate and enter the remaining parameters.
How to read the chart
The chart shows the power curve: how the probability of detecting the effect varies as the sample size per group increases.
- X axis (horizontal) — sample size per group (n). The further right, the more users per variant.
- Y axis (vertical) — power (1−β), between 0 and 1. The usual target is at least 0.80.
- Each colored line represents a different effect size expressed as a relative lift over p_A (+5%, +10%, +20%). Larger effects (yellow line) are detected with smaller samples; small effects (blue line) require many more users.
- Dashed vertical line — appears when you calculate with a fixed n (Power and MDE modes) and marks exactly where your current sample sits on the curves.
How to use it in practice: if your n falls on the steep part of the curve, adding more users noticeably improves power. If you're already in the flat zone (top part), adding more sample barely helps. If the curves for the effect you care about are all below 0.80 at your current n, the experiment is too small to be reliable.
Key concepts before you start
These three parameters control the balance between risk, resources, and sensitivity of the experiment. Understanding them is essential to interpreting any result.
Significance level α — risk of a false positive
This is the probability of concluding that B is better than A when in reality there is no difference. With α = 0.05, one in every 20 experiments where no real effect exists will produce a "statistically significant" result by pure chance. Lowering α reduces false positives, but requires a larger sample to maintain power.
Statistical power 1−β — ability to detect a real effect
This is the probability of actually detecting the difference when it truly exists. With power = 80% and a real effect equal to the one you specified, the experiment will correctly detect it 8 out of 10 times. The remaining 20% are false negatives (β): the effect exists but the test doesn't see it. For critical decisions, it's worth raising power to 90% or more.
MDE — minimum detectable effect
This is the smallest absolute difference between p_A and p_B that the experiment can detect at the configured power and α. If the true effect is smaller than the MDE, the test will probably not detect it, even if it exists. The MDE depends directly on n: the larger the sample, the smaller the MDE, and therefore the greater the sensitivity.
Baseline rate p_A and expected rate p_B
p_A is the current conversion rate of the control group (for example, 12% clicks → 0.12). p_B is the rate you expect, or the minimum improvement you would care to detect in the variant. The difference \(\Delta = p_B - p_A\) is the effect of the experiment.
Formulas used
For a comparison of two proportions with 1:1 allocation, the required sample size per group is:
\( n = \left\lceil \dfrac{\left(z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_\beta\sqrt{p_A(1-p_A)+p_B(1-p_B)}\right)^2}{(p_A - p_B)^2} \right\rceil \)
where \(\bar{p} = (p_A + p_B)/2\), \(z_{\alpha/2}\) is the normal quantile for the two-tailed significance level (or \(z_{\alpha}\) for one-tailed) and \(z_{1-\beta}\) is the quantile for the desired power.
The power given n is obtained with the equivalent normal approximation; for a two-tailed test both rejection tails are added:
\( 1-\beta \approx P\!\left(\hat{p}_B-\hat{p}_A > z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})/n}\right) + P\!\left(\hat{p}_B-\hat{p}_A < -z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})/n}\right) \)
The MDE is obtained by binary search over the effect that achieves the target power with the available n.
Worked example
An e-commerce team wants to test a new, simplified checkout flow (variant B) against the current flow (variant A). The current conversion rate is 12%, and the team feels the change is only worth implementing if it improves conversion by at least 2 percentage points, i.e., the target rate for B is 14%. The experiment is designed with two-tailed α = 0.05 and 80% power (β = 0.20).
The auxiliary parameters are calculated. The weighted average of both rates is \( \bar{p} = (0.12 + 0.14)/2 = 0.13 \). The required quantiles are \( z_{\alpha/2} = z_{0.025} = 1.960 \) and \( z_{1-\beta} = z_{0.80} = 0.842 \). The sample size formula per group is:
\( n = \left\lceil \dfrac{\left(z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_\beta\sqrt{p_A(1-p_A)+p_B(1-p_B)}\right)^2}{(p_A - p_B)^2} \right\rceil \)
Substituting the values:
\( n = \left\lceil \dfrac{\left(1.960\,\sqrt{2 \times 0.13 \times 0.87} + 0.842\,\sqrt{0.12 \times 0.88 + 0.14 \times 0.86}\right)^2}{(0.12 - 0.14)^2} \right\rceil \)
\( = \left\lceil \dfrac{\left(1.960 \times 0.4757 + 0.842 \times 0.4754\right)^2}{0.0004} \right\rceil = \left\lceil \dfrac{(0.9324 + 0.4003)^2}{0.0004} \right\rceil = \left\lceil \dfrac{(1.3327)^2}{0.0004} \right\rceil \approx \mathbf{4{,}438} \text{ per group} \)
The experiment requires approximately 4,438 users per group (8,876 in total). If the site receives 3,000 visits per day and 50% is allocated to the experiment, it would take about 6 days to accumulate the necessary traffic. With a real conversion rate for B of 14%, the probability of detecting the improvement is 80%; if the real improvement is larger (e.g., 15%), the effective power would exceed 95%.
How to interpret the result
Depending on the selected mode, the calculator returns one of the three key metrics of experimental design. Each answers a different question and must be read differently.
Mode: Sample size
The calculator returns the minimum number of users per group needed for the experiment to have the desired power. For example, if you get n = 2,000, you need 2,000 users in group A and another 2,000 in group B, i.e., 4,000 in total. If you launch the experiment with fewer users, the probability of detecting the real effect drops below the threshold you set, and false negatives increase.
The result also shows the absolute effect Δ (difference in percentage points between p_A and p_B) and the relative lift (how much the rate increases in percentage terms relative to the baseline). These two numbers help you judge whether the effect you're designing to detect is realistic for your business.
Mode: Power
With a fixed sample (because you already have the data or know how much traffic you'll receive), power tells you the probability you have of detecting the specified difference. A power of 55%, for example, means that even if the variant is truly better by the stated amount, half the time the test won't detect it. If power is low, you should consider: (a) waiting longer to accumulate more sample, or (b) accepting that you'll only detect larger effects.
The result also includes how many users you would need to reach 80% power, giving you a reference for how far off you are.
Mode: MDE
Given the available sample, the MDE tells you the minimum difference you can reliably detect. If your MDE is ±3pp and the real improvement of your variant is 1pp, the experiment won't see it at the target power. Before launching, compare the MDE with the minimum effect that would have real business impact: if the MDE is larger, the test isn't useful for that decision.
The result also shows the specific p_B values that correspond to the MDE in the upward and downward directions, so you can reason in absolute terms of conversion rate.
Practical recommendations
- Power ≥ 0.80 is the recommended minimum; use 0.90 for critical decisions or when the cost of a false negative is high.
- α = 0.05 is the standard; consider α = 0.01 if the cost of a false positive is high (launching a variant that doesn't actually improve anything).
- Fix the sample size before launching the experiment. Stopping it early or continuing until you see significance inflates the type I error rate.
- The relative MDE is usually more relevant than the absolute one: a 10% MDE over a 5% baseline rate corresponds to p_B = 0.055, a difference of only 0.5pp.
- With limited traffic, accept detecting only larger effects (higher MDE) or extend the duration of the experiment instead of lowering α to compensate.
Frequently asked questions
- Does the sample size calculation assume 1:1 allocation? Yes. The formula implemented maximizes power with balanced allocation. For unequal allocation, use the general two-proportion sample size calculator.
- What is the difference between a two-tailed and a one-tailed test? The two-tailed test detects effects in both directions (p_B greater or smaller than p_A) and is the most commonly used. The one-tailed test only detects effects in one direction and needs a smaller sample, but is valid only if you know in advance that the variant can only improve, never worsen.
- Is the MDE the effect I expect, or the minimum effect that matters? The MDE is the minimum effect detectable with the given sample. You should size the experiment so that its MDE is equal to or smaller than the minimum effect that would have practical relevance for your business.
- Why does the chart show relative lifts instead of the exact effect I entered? The chart illustrates how power varies with n for three reference scenarios (+5%, +10%, +20% relative to p_A). Your specific calculation is shown in the results box; the chart provides visual context to understand the general sensitivity of the experiment.