A/B test simulation

What this tool simulates

In each replicate, the calculator generates random samples of size n from two Bernoulli distributions with the true rates p_A and p_B, computes the z statistic of the two-proportion test, and records whether H₀ is rejected. After N replicates, it obtains the empirical distribution of z and p-values, and compares the rejection rate with the theoretical power.

\( z^{(i)} = \dfrac{\hat{p}_B^{(i)} - \hat{p}_A^{(i)}}{\sqrt{\hat{p}^{(i)}(1-\hat{p}^{(i)})\left(\tfrac{1}{n_A}+\tfrac{1}{n_B}\right)}} \)

If p_A = p_B, the empirical rejection rate estimates the real type I error (it should approach α). If p_A ≠ p_B, it estimates the empirical power.

Worked example

A digital marketing team wants to evaluate whether a new email subject line (variant B) improves the open rate over the current subject line (variant A). Historical data shows the current open rate is p_A = 0.22 (22%). Variant B is expected to reach a rate of p_B = 0.26 (26%), an absolute improvement of 4 percentage points.

The team sets a two-tailed α = 0.05 and 80% power, which with the standard analytical formula yields approximately n = 476 users per group (952 in total). Before launching the real experiment, the team decides to validate these results via Monte Carlo simulation with N = 5,000 replicates.

In each replicate i, the tool generates \( n_A = n_B = 476 \) Bernoulli observations with the true rates, computes the z statistic of the two-proportion test, and records whether H₀ is rejected:

\( z^{(i)} = \dfrac{\hat{p}_B^{(i)} - \hat{p}_A^{(i)}}{\sqrt{\hat{p}^{(i)}(1-\hat{p}^{(i)})\!\left(\tfrac{1}{476}+\tfrac{1}{476}\right)}} \)

After the 5,000 replicates, the simulation yields the following results: the empirical rejection rate is 80.3%, in excellent agreement with the theoretical power of 80%. The histogram of the z statistic shifts to the right relative to the null N(0,1), with the empirical mean around \( \bar{z} \approx 1.45 \), reflecting the real effect of \( \Delta = 0.04 \). The p-value distribution shows a strong concentration near 0: in 80% of the replicates, the p-value falls below 0.05.

To check the control of the type I error, the team also runs the simulation under H₀ (setting p_B = p_A = 0.22). With the same 5,000 replicates, the empirical rejection rate turns out to be 4.9%, very close to the nominal level α = 0.05, and the p-value distribution is approximately uniform on [0, 1], exactly as theory predicts.

The practical conclusion is twofold: the analytically calculated sample size is correct (empirical power matches the theoretical power), and the test maintains good control of the type I error. The team can launch the experiment with confidence, knowing that if the real improvement is 4pp, they will have an 80% probability of detecting it with statistically significant results.

How to interpret the result

Monte Carlo simulation lets you empirically verify the test's properties: whether the type I error is really α when p_A = p_B, whether the power matches the theoretical value, and how the distribution of the z statistic behaves in practice. It is especially useful when you want to understand the test's behavior under specific conditions, or when the model's assumptions are not exactly met. The empirical rejection rate returned by the simulation is a direct estimate of power: how many times out of 100 identical experiments you would correctly conclude that B differs from A.

How to read the z histogram

Under H₀ (p_A = p_B), the z statistic follows approximately an N(0,1): the histogram should be symmetric, centered on 0, and the proportion of red bars (rejection zone) should approach α. Under H₁ (p_A ≠ p_B), the histogram shifts in the direction of the effect: the larger the real difference between rates or the sample size, the more pronounced that shift, and the greater the proportion of replicates that fall in the rejection zone.

How to read the p-value distribution

Under a perfect H₀, the p-values follow a Uniform(0,1) distribution: the bar from 0 to 0.05 should contain approximately 5% of the replicates, and all bars should have similar heights. Under H₁, the p-values concentrate near 0: the proportion of replicates with p < α is the empirical power. A p-value distribution with a marked concentration in the first bar and a rapid decline indicates that the test has good ability to detect the configured effect.

Frequently asked questions

• Does the simulation use binomial samples? Each replicate approximates x_A ~ Bin(n, p_A) and x_B ~ Bin(n, p_B) using the De Moivre-Laplace normal approximation. It is adequate when n·p and n·(1−p) are not small; for small samples or extreme proportions the simulation should be interpreted as approximate.
• Why doesn't the empirical power exactly match the theoretical one? There is always sampling variance. With 5,000 replicates, the standard error of the simulated proportion is approximately ±0.006 for powers around 80%.
• Can I simulate with p_A = p_B to check the α level? Yes: enter the same value for p_A and p_B and check that the empirical rejection rate approaches α.
• Are the replicates independent? Yes: each replicate generates new samples with no dependency on the previous ones.