Hypothesis tests

Online hypothesis test calculator for two proportions

Test whether two population proportions are equal in independent groups.

Parameters

Online calculator: enter the successes and sample size of each group to get the p-value and decision.

Result pending…

Rejection regions and observed statistic

What is a hypothesis test for two proportions?

It is the procedure for deciding whether the proportion of "successes" is the same in two independent groups. It lies behind a great many practical questions: does version B of the website convert more than version A? Do more patients recover with the new treatment than with the standard one? Do men and women vote the same way? Are there more defects on the night shift? In each group you count how many individuals meet the condition and compare the percentages.

You start from the null hypothesis that the two population proportions are equal, \(H_0: p_1 = p_2\), and measure how far apart the sample proportions \(\hat{p}_1\) and \(\hat{p}_2\) are. Even if the groups were identical, their sample proportions would never match exactly; the question is whether the observed difference is larger than chance can reasonably explain.

Hypotheses and test statistic

\(H_0: p_1 = p_2\)

\(H_1: p_1 \neq p_2\), \(p_1 > p_2\) or \(p_1 < p_2\)

\( z = \dfrac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}(1-\hat{p})\left(\dfrac{1}{n_1} + \dfrac{1}{n_2}\right)}}, \qquad \hat{p} = \dfrac{x_1 + x_2}{n_1 + n_2} \)

  • \(x_1, x_2\): number of successes in each group.
  • \(n_1, n_2\): group sizes.
  • \(\hat{p}_1 = x_1/n_1\), \(\hat{p}_2 = x_2/n_2\): sample proportions.
  • \(\hat{p}\): pooled proportion of both groups.

The denominator is the standard error of the difference if \(H_0\) were true. Under \(H_0\) both groups share the same proportion, and the best estimate of that common proportion is the pooled \(\hat{p}\), which combines all the successes and all the individuals. With large enough samples, the statistic approximately follows a standard normal distribution.

This z-test is equivalent to the chi-square test of independence on a 2×2 table: in the two-tailed case, \(z^2 = \chi^2\) and the p-values match. The advantage of z is that it allows one-tailed tests and directly gives the direction of the difference.

How to carry out the test, step by step

  1. Decide which group is 1 and which is 2, and what counts as a success. The difference is always computed as group 1 − group 2.
  2. State the hypotheses and set \(\alpha\), usually 0.05.
  3. Compute the proportions of each group and the pooled proportion.
  4. Compute the z statistic and the p-value, or compare it with the critical value (1.96 for a two-tailed test with \(\alpha = 0.05\)).
  5. Decide and interpret, reporting the decision together with the difference in percentage points and its confidence interval.

Decision rule

  • If the p-value is less than \(\alpha\), \(H_0\) is rejected: there is evidence that the proportions differ.
  • If the p-value is greater than or equal to \(\alpha\), there is not enough evidence to reject \(H_0\).
  • Not rejecting \(H_0\) does not prove that the proportions are equal.

With a one-tailed test, the direction is fixed before seeing the data: the right tail tests whether the proportion of group 1 is greater than that of group 2. In A/B tests the two-tailed test is usually recommended, because a change can also make results worse.

Difference, relative risk and odds ratio

There are three common ways of expressing how much two proportions differ, and they should not be confused:

MeasureFormulaWith 51.7% vs 36.9%
Difference (absolute)\(\hat{p}_1 - \hat{p}_2\)14.8 percentage points
Relative risk\(\hat{p}_1 / \hat{p}_2\)1.40 (40% more)
Odds ratio\(\dfrac{\hat{p}_1/(1-\hat{p}_1)}{\hat{p}_2/(1-\hat{p}_2)}\)1.83

The absolute difference is the most useful for decisions: a 50% increase sounds spectacular, but it may mean going from 2% to 3%. Relative risk is intuitive for communicating results, and the odds ratio is the natural measure in case-control studies and logistic regression. All three indicate no difference at the same time (0 or 1), so the test of equality is the same; what changes is how the size of the effect is expressed.

Assumptions of the test

  • Two independent groups: each individual is in only one group. If they are the same individuals at two times or under two conditions, use McNemar's test.
  • Independent observations within each group. In an A/B test, for example, count users rather than page views, because visits by the same user are not independent.
  • A large enough sample: as a rule of thumb, at least 5 expected successes and 5 expected failures in each group (using the pooled proportion). If not, use Fisher's exact test; the calculator warns you and links to it already filled in.

Worked example 1: one treatment against another

With the new treatment, 62 of 120 patients improve (\(\hat{p}_1 = 0.517\)); with the standard one, 48 of 130 (\(\hat{p}_2 = 0.369\)). We test \(H_0\colon p_1 = p_2\) against \(H_1\colon p_1 \neq p_2\) with \(\alpha = 0.05\).

The pooled proportion is \(\hat{p} = 110/250 = 0.44\), and the standard error under \(H_0\) is:

\( SE = \sqrt{0.44 \times 0.56 \left(\dfrac{1}{120} + \dfrac{1}{130}\right)} \approx 0.0628 \)

\( z = \dfrac{0.517 - 0.369}{0.0628} \approx 2.35 \)

Since \(|z| = 2.35 > 1.96\), the two-tailed p-value is \(p \approx 0.019\): \(H_0\) is rejected. The new treatment helps 14.8 percentage points more patients, with a 95% confidence interval from 2.5 to 26.9 points. The effect is real, but its size is poorly determined: it could be modest or very large.

In the calculator: x₁ = 62, n₁ = 120, x₂ = 48, n₂ = 130 and "Two-tailed (≠)". These are the values the page opens with.

Worked example 2: an A/B test with lots of traffic

A website tests two versions of its checkout page. Version A gets 420 purchases from 10,000 visitors (4.2%) and version B, 480 from 10,000 (4.8%). We test with \(\alpha = 0.05\), two-tailed, taking A as group 1.

\( \hat{p} = \dfrac{900}{20{,}000} = 0.045, \qquad SE = \sqrt{0.045 \times 0.955 \times \dfrac{2}{10{,}000}} \approx 0.00293, \qquad z = \dfrac{0.042 - 0.048}{0.00293} \approx -2.05 \)

The p-value is \(p \approx 0.041\): \(H_0\) is rejected, with version B ahead. Look at the size: 0.6 percentage points. With 10,000 visitors per version, the test detects with 80% probability differences of under 0.9 points, so small differences come out significant. Being significant does not mean it is worth it: you need to judge whether 0.6 more points of conversion justify the change. For this kind of analysis there are also the A/B testing calculators.

In the calculator: x₁ = 420, n₁ = 10000, x₂ = 480, n₂ = 10000 and "Two-tailed (≠)".

How to interpret the result

Rejecting \(H_0\) (p-value < α) indicates that the difference between the proportions is too large to attribute to chance if both groups were equal. To judge whether it matters, look at the difference in percentage points and its confidence interval.

Not rejecting \(H_0\) (p-value ≥ α) does not prove that the proportions are equal. If what you want to show is precisely that one treatment is not worse than another, the correct approach is a non-inferiority test.

In the calculator, the result shows the decision with the p-value, the z statistic with both proportions and a sentence stating the conclusion. On the chart, the green area is the non-rejection region, the red areas are the critical regions and the amber line marks the observed statistic. Under "Calculation details" you will find the difference in points, the pooled proportion, the standard error, the confidence interval and the relative risk.

Power and minimum detectable difference

Below the conclusion, the calculator shows the minimum detectable difference: the smallest real difference from the proportion of group 2 that, with your group sizes, the test would detect with 80% probability. It depends heavily on the starting proportion and on the sample size: with 120 patients per group only differences of about 18 points are reliably detected, and with 10,000 per group, under one point. The link opens the sample size calculator for two proportions with your proportions, your α, the ratio between the groups and the type of test.

Relationship with the confidence interval

The other link opens the confidence interval for the difference in proportions with your data. Test and interval almost always agree: if the interval does not contain 0, the test is significant. They can disagree in borderline cases, because the test uses the standard error with the pooled proportion and the Wald interval uses each group's own.

Common mistakes

  • Using it with paired data, such as the answers of the same people before and after.
  • Counting units that are not independent, such as page views instead of users.
  • Checking the A/B test every day and stopping as soon as it is significant: that practice inflates false positives. Fix the sample size in advance or use sequential methods.
  • Confusing percentage points with percentages: going from 4.2% to 4.8% is 0.6 points, or 14% more.
  • Concluding that two groups are equal because the test is not significant.
  • Applying the z-test with very few successes; in that case, use Fisher's exact test.

Frequently asked questions

  • What does it test? Whether the proportion of successes is the same in two independent groups (H₀: p₁ = p₂), against the alternative that they differ or that one is greater than the other.
  • Why is the pooled proportion used? Because, if H₀ is true, both groups share the same proportion, and the best estimate of that common proportion combines the data from both.
  • Is it the same as the chi-square test on a 2×2 table? Yes, for the two-tailed test: z² equals Pearson's χ² and the p-value is the same. The z-test also allows one-tailed tests.
  • Can it be used with paired data? No. If they are the same people at two times or under two conditions, use McNemar's test.
  • When should I use Fisher's exact test? When some group has fewer than about 5 expected successes or failures; in that case the normal approximation is not reliable.
  • What should I report besides the p-value? Both proportions, the difference in percentage points and its confidence interval; depending on the context, also the relative risk or the odds ratio.
  • What is the minimum detectable difference? The smallest real difference between the proportions that the test would detect with 80% probability given your group sizes.
  • What sample size do I need? It depends on the expected proportions, the difference you want to detect and the desired power. Work it out with the sample size calculator for two proportions.

Reference: Two-proportion Z-test — Wikipedia