Sample size calculator for two proportions

Explanation

This calculation determines the minimum number of observations per group needed to detect a difference between two independent proportions with a specified power and significance level. Unlike estimating a single proportion, the goal here is a two-tailed hypothesis test: \(H_0\!: p_1 = p_2\) versus \(H_1\!: p_1 \neq p_2\).

The formula implemented is that of Fleiss, Levin and Paik (2003), which is more precise than Cohen's arcsine approximation because it accounts for the separate variability of each group under \(H_1\).

The effect size here is the absolute difference \(|p_1 - p_2|\): the smaller it is, the harder it is to detect and the larger the required sample. For very small effects (differences < 0.05), consider whether the effect is clinically or practically relevant before planning the study.

Sample size formula

\( n_1 = \frac{\left(Z_{\alpha/2}\sqrt{(1+1/k)\bar{p}(1-\bar{p})} + Z_\beta\sqrt{p_1(1-p_1)+\frac{p_2(1-p_2)}{k}}\right)^2}{(p_1-p_2)^2} \)

\( n_2 = k\,n_1,\quad \bar{p}=\frac{p_1 + kp_2}{1+k} \)

• n1, n2: sample sizes per group.
• k: allocation ratio n2/n1 (k = 1 for a 1:1 design).
• \(\bar{p}\): pooled proportion under H0.
• \(Z_{\alpha/2}\) and \(Z_\beta\): normal quantiles for two-tailed alpha and power.

Quick settings

• p1 and p2: use estimates from a pilot study, historical data or the literature. If p2 is the control group, p1 is what you expect to observe in the intervention group.
• Alpha (α): 0.05 two-tailed is the most common value. For confirmatory studies use 0.01.
• Power: 0.80 as a minimum; 0.90 if the consequences of a false negative are serious.
• 1:1 allocation: the most efficient design when the cost of both groups is similar. Unequal allocations increase the total.
• Attrition: divide n by (1 − expected dropout rate).

Simple example

You are comparing conversion between two campaigns: p1 = 0.50, p2 = 0.40, α = 0.05, power 0.80, 1:1 allocation. Result: approximately 388 participants per group (total ≈ 776).

Worked example

A digital commerce company wants to compare the conversion rate of two versions of its product page. Design A (current) has a conversion rate of \(p_1 = 0.12\) (12%), and design B (new) is expected to reach \(p_2 = 0.16\) (16%). The goal is to detect this 4-percentage-point difference with 80% power (\(z_\beta = 0.842\)) and a two-tailed significance level \(\alpha = 0.05\) (\(z_{\alpha/2} = 1.960\)), with 1:1 allocation (\(k = 1\)).

First we compute the pooled proportion under \(H_0\) with \(k=1\): \(\bar{p} = (p_1 + p_2)/2 = (0.12 + 0.16)/2 = 0.14\). Then we apply the Fleiss formula:

\( n_1 = \frac{\left(z_{\alpha/2}\sqrt{2\,\bar{p}(1-\bar{p})} + z_\beta\sqrt{p_1(1-p_1)+p_2(1-p_2)}\right)^2}{(p_1-p_2)^2} \)

Substituting: the term under the square root of the first summand is \(2 \times 0.14 \times 0.86 = 0.2408\), so \(\sqrt{0.2408} = 0.4907\). The term under the square root of the second summand is \(0.12 \times 0.88 + 0.16 \times 0.84 = 0.1056 + 0.1344 = 0.2400\), so \(\sqrt{0.2400} = 0.4899\). Therefore:

\( n_1 = \frac{(1.960 \times 0.4907 + 0.842 \times 0.4899)^2}{(0.04)^2} = \frac{(0.9618 + 0.4125)^2}{0.0016} = \frac{(1.3743)^2}{0.0016} = \frac{1.889}{0.0016} \approx 1{,}180.5 \rightarrow n_1 = 1{,}181 \)

You need 1,181 sessions per version (2,362 in total). With this sample size, the test will have an 80% probability of detecting the 4-pp improvement if it truly exists, while keeping the false positive rate at 5%.

More demanding scenario (minimum difference of 3 pp): if the team considers it relevant to detect even a difference of \(p_2 = 0.15\) versus \(p_1 = 0.12\), the minimum difference drops to 3 pp. Since this is harder to detect, the required sample size grows considerably, to approximately 1,392 sessions per group (2,784 in total), which can mean additional weeks of A/B testing depending on the site's traffic volume.

Model assumptions

• The two groups are independent of each other.
• Random sampling within each group.
• The binomial distribution is approximated by the normal (valid for n·p ≥ 5 and n·(1−p) ≥ 5 in each group).
• Two-tailed test (if you have a directional hypothesis, the sample can be reduced somewhat using a one-tailed test, although this is less conservative).

Common uses

• A/B tests in marketing and digital product (conversion or click-through rates).
• Comparing response rates to two treatments in clinical trials.
• Evaluating changes in processes with a binary outcome (yes/no, success/failure).
• Epidemiological cohort studies with a dichotomous outcome.

How to interpret the result

The calculated \(n\) is the minimum size per group needed to detect the difference \(|p_1 - p_2|\) with the specified power and significance level. In a balanced design (\(k = 1\)) the total is \(2n\); in an unbalanced one (\(k \neq 1\)) the reference group needs \(n\) subjects and the intervention group needs \(k \cdot n\). Always round up, and divide by \((1 - \text{dropout rate})\) to get the actual recruitment target. If the expected non-response rate is 20%, plan for \(\lceil n / 0.80 \rceil\) per group.

The sensitivity to effect size is very pronounced: when \(p_1\) and \(p_2\) are both close to 0.5, or when the difference between them is small (< 0.05), \(n\) can balloon to impractical values. In this case, run a sensitivity analysis varying \(p_1\) and \(p_2\) by ±0.05 to assess the robustness of the plan. If the required difference is very small, consider whether it is practically relevant: a statistically detectable difference without practical relevance does not justify the cost of recruiting thousands of participants. Conversely, if \(p_1\) or \(p_2\) are very extreme (\(<0.05\) or \(>0.95\)), consider using the arcsine transformation or a logistic regression model in the analysis.

When the resulting \(n\) is small and the expected frequencies in some cell are below 5, the chi-square test may not be appropriate; in that case, use Fisher's exact test as an alternative. If the design is feasible, once the data has been collected, test the results with the hypothesis test calculator for two proportions or build a confidence interval for the difference if the goal is to estimate the magnitude of the effect.

References and further reading

• Wikipedia: Sample size determination — section on comparing proportions.
• Wikipedia: Statistical power — relationship between power, alpha and sample size.
• Fleiss, J. L., Levin, B. and Paik, M. C. (2003). Statistical Methods for Rates and Proportions (3rd ed.). Wiley. — source of the implemented formula.

Frequently asked questions

• How does this differ from Cohen's formula? Fleiss's formula is more precise because it models the variance of each group under H1; Cohen's formula uses the pooled variance in both terms.
• Can I use unequal allocation (k ≠ 1)? Yes, but the total sample size will always be larger than with k = 1. It's only worth it when recruiting one group is much cheaper.
• What is statistical power? The probability of rejecting H0 when H1 is true, i.e., of detecting the real difference.
• Is the result exact? It is a normal approximation; for very small samples or extreme proportions, consider exact methods.