Hypothesis tests

Online hypothesis test calculator for two means

Compare two independent groups using Welch's t-test.

Parameters

Online calculator: enter the means, standard deviations and sample sizes of both groups.

Result pending…

Rejection regions and observed statistic

What is a hypothesis test for two means?

It is the procedure for deciding, from two samples, whether the means of two independent populations are equal or whether the data show that they differ. It is probably the most widely used test in practice: does drug A lower blood pressure more than drug B? Do customers spend more on the new version of the website? Do students at two schools perform equally well? It is also known as the independent two-sample t-test.

The logic is that of any test. You start from the null hypothesis that the two population means are equal, \(H_0: \mu_1 = \mu_2\), and measure how far apart the sample means \(\bar{x}_1\) and \(\bar{x}_2\) are. Even if the populations had the same mean, two samples would never give exactly the same mean: the question is whether the observed difference is larger than sampling variation can reasonably explain. If it is, \(H_0\) is rejected in favour of the alternative hypothesis.

This calculator uses Welch's version of the t-test, which does not require the two populations to have the same variance. It is the recommended default, as explained below.

Hypotheses and test statistic

\(H_0: \mu_1 = \mu_2\) (that is, \(\mu_1 - \mu_2 = 0\))

\(H_1: \mu_1 \neq \mu_2\), \(\mu_1 > \mu_2\) or \(\mu_1 < \mu_2\)

\( t = \dfrac{\bar{x}_1 - \bar{x}_2}{\sqrt{\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}}} \)

  • \(\bar{x}_1, \bar{x}_2\): sample means of groups 1 and 2.
  • \(s_1, s_2\): sample standard deviations of each group.
  • \(n_1, n_2\): group sizes.

Development and additional details

The denominator is the standard error of the difference: it measures how much \(\bar{x}_1 - \bar{x}_2\) would vary if you repeated the study with other samples. Since each mean has its own uncertainty, the two variances add up. The t statistic therefore tells you how many standard errors the observed difference lies from zero, which is what \(H_0\) predicts.

Under \(H_0\), the statistic approximately follows a Student's t distribution with the Welch-Satterthwaite degrees of freedom, which are usually not whole numbers:

\( \nu = \dfrac{\left(s_1^2/n_1 + s_2^2/n_2\right)^2}{\dfrac{(s_1^2/n_1)^2}{n_1-1} + \dfrac{(s_2^2/n_2)^2}{n_2-1}} \)

\(\nu\) always lies between the smaller of \(n_1-1\) and \(n_2-1\) and the sum \(n_1+n_2-2\). When the variances and sizes are similar, it gets close to \(n_1+n_2-2\); when one group is much more variable or much smaller than the other, it drops, and the test becomes more demanding.

How to carry out the test, step by step

  1. State the hypotheses before looking at the data and decide which group is 1 and which is 2: the difference is always computed as group 1 − group 2.
  2. Set the significance level \(\alpha\), usually 0.05.
  3. Compute the standard error of the difference and the t statistic.
  4. Compute Welch's degrees of freedom and, with them, the p-value or the critical value from the Student's t table.
  5. Decide and interpret: report the decision together with the observed difference, its confidence interval and the effect size.

The calculator performs steps 3 and 4, plots the statistic on the t distribution and, under "Calculation details", shows all the intermediate values.

Decision rule

  • If the p-value is less than \(\alpha\), \(H_0\) is rejected: there is evidence that the means differ (in the direction of the chosen alternative).
  • If the p-value is greater than or equal to \(\alpha\), there is not enough evidence to reject \(H_0\).
  • Not rejecting \(H_0\) does not prove that the means are equal: the sample may be too small to detect the real difference.

Welch or Student's t with equal variances?

There are two classic versions of the independent two-sample t-test. Student's version assumes that both populations have the same variance and estimates it by combining both samples (pooled variance). Welch's version makes no such assumption: it uses each variance separately and corrects the degrees of freedom.

Student's t (equal variances)Welch's t
Assumption about the variances\(\sigma_1^2 = \sigma_2^2\)None
Standard error\(s_p\sqrt{1/n_1 + 1/n_2}\)\(\sqrt{s_1^2/n_1 + s_2^2/n_2}\)
Degrees of freedom\(n_1 + n_2 - 2\)Welch-Satterthwaite \(\nu\)
If the variances are equalCorrectAlmost identical, with a negligible loss of power
If both the variances and the sizes differCan give too many false positives (or too few)Keeps the type I error close to \(\alpha\)

That is why Welch is now recommended by default. When the variances are equal, hardly anything is lost; when they are not, especially if the smaller group is the more variable one, Student's version can go badly wrong. Nor is it a good idea to choose between them by first running a test of equal variances (such as the F test or Levene's test): that two-stage procedure does not control the type I error well.

If you really do know the population standard deviations \(\sigma_1\) and \(\sigma_2\), which is rare, the appropriate test is the two-sample z-test. With large samples, z and Welch give practically the same result.

Two-tailed or one-tailed test?

TypeAlternative hypothesisQuestion
Two-tailed\(H_1: \mu_1 \neq \mu_2\)Do the means differ, in either direction?
Right-tailed\(H_1: \mu_1 > \mu_2\)Is the mean of group 1 greater?
Left-tailed\(H_1: \mu_1 < \mu_2\)Is the mean of group 1 smaller?

A one-tailed test is only justified if, before seeing the data, you are interested in just one direction and a difference in the other direction would not change any decision. Choosing the tail after seeing which way the means point doubles the real risk of a false positive. When in doubt, use the two-tailed test: it is what most journals and reviewers expect.

Assumptions of the test

  • Two independent groups: each individual belongs to only one group and there is no pairing between them. If you measure the same people twice (before and after), or form pairs, use the paired means test.
  • Independent observations within each group: this fails, for example, with students from the same class or patients from the same hospital analysed as if they were independent.
  • A quantitative variable. To compare percentages, use the two-proportion test.
  • Approximate normality in each group, or samples large enough for the means to be approximately normal by the central limit theorem. With 30 or more observations per group, this is usually enough unless the distributions are very skewed.

The variances do not need to be equal, nor the groups the same size: Welch takes care of both. With small samples, it is worth checking normality (histogram or Shapiro-Wilk test) and outliers, which can move a mean a lot. If the data are clearly skewed and there are few of them, the non-parametric alternative is the Mann-Whitney U test. To compare three or more groups, use one-way ANOVA instead of several pairwise tests.

Worked example 1: two-tailed test

Two treatments for lowering blood pressure are compared. With treatment A (group 1), \(n_1 = 30\) patients achieve a mean reduction of \(\bar{x}_1 = 82\) mmHg with \(s_1 = 12\); with B (group 2), \(n_2 = 28\) patients achieve \(\bar{x}_2 = 76\) mmHg with \(s_2 = 15\). We test \(H_0\colon \mu_1 = \mu_2\) against \(H_1\colon \mu_1 \neq \mu_2\) with \(\alpha = 0.05\).

The standard error of the difference is:

\( SE = \sqrt{\dfrac{12^2}{30} + \dfrac{15^2}{28}} = \sqrt{4.800 + 8.036} \approx 3.583 \)

and the statistic:

\( t = \dfrac{82 - 76}{3.583} \approx 1.675 \)

Welch's degrees of freedom are:

\( \nu = \dfrac{12.836^2}{\dfrac{4.800^2}{29} + \dfrac{8.036^2}{27}} = \dfrac{164.76}{0.794 + 2.392} \approx 51.7 \)

The two-tailed critical value is \(t_{0.025;\,51.7} \approx 2.007\). Since \(|t| = 1.675 < 2.007\), the statistic does not fall in the rejection region; the two-tailed p-value is \(p \approx 0.100\). \(H_0\) is not rejected.

Conclusion: these data do not provide enough evidence that the two treatments produce different mean reductions. But that does not mean they are equally effective. The 95% confidence interval for the difference, \(6 \pm 2.007 \times 3.583\), runs from −1.19 to 13.19 mmHg: it includes 0, but also differences of more than 10 mmHg in favour of A, which would be clinically important. The study is simply small: with these group sizes it detects with 80% probability only differences of about 10.2 mmHg or more, and the probability of detecting a real difference of 5 mmHg is only about 28%.

In the calculator: x̄₁ = 82, x̄₂ = 76, s₁ = 12, s₂ = 15, n₁ = 30, n₂ = 28 and "Two-tailed (≠)". The result also shows the minimum detectable difference and a link that opens the sample size calculator with these values.

Worked example 2: one-tailed test

A school tries out a new teaching method and, before collecting the data, states that it only wants to know whether it improves grades. The \(n_1 = 40\) students taught with the new method (group 1) score \(\bar{x}_1 = 78\) points with \(s_1 = 10\); the \(n_2 = 45\) taught with the traditional method (group 2) score \(\bar{x}_2 = 72\) with \(s_2 = 12\). We test \(H_0\colon \mu_1 = \mu_2\) against \(H_1\colon \mu_1 > \mu_2\) with \(\alpha = 0.05\).

\( SE = \sqrt{\dfrac{100}{40} + \dfrac{144}{45}} = \sqrt{2.5 + 3.2} \approx 2.387, \qquad t = \dfrac{78 - 72}{2.387} \approx 2.513 \)

With \(\nu \approx 82.7\) degrees of freedom, the right-tailed critical value is \(t_{0.05;\,82.7} \approx 1.663\). Since \(t = 2.513 > 1.663\), \(H_0\) is rejected; the p-value is \(p \approx 0.0070\).

Conclusion: there is statistical evidence that the new method achieves a higher mean grade. The difference is 6 points, just over half a standard deviation (\(d \approx 0.54\)), a moderate effect. Bear in mind that, because this is an observational study of groups that already existed, the difference could also be due to other differences between the students; only random assignment to the method would allow it to be attributed to the method with confidence.

In the calculator: x̄₁ = 78, x̄₂ = 72, s₁ = 10, s₂ = 12, n₁ = 40, n₂ = 45 and "Right-tailed (>)".

How to interpret the result

Rejecting \(H_0\) (p-value < α) means that, if both populations had the same mean, it would be very unlikely to observe a difference between the sample means as large as the one obtained. But significant is not the same as important: with large samples, tiny differences become significant. Always look at the observed difference, its confidence interval and the effect size.

Not rejecting \(H_0\) (p-value ≥ α) does not prove that the means are equal. It only shows that the data are compatible with equality, and perhaps also with relevant differences, as in example 1. If what you want to show is precisely that two treatments are equivalent, the correct approach is an equivalence or non-inferiority test.

Development and additional details

In the calculator, the result shows the decision with the p-value, the t statistic with its Welch degrees of freedom and a sentence stating the conclusion. On the chart, the green area is the non-rejection region, the red areas are the critical regions and the amber line marks the observed statistic. Under "Calculation details" you will find the observed difference, the standard error, the critical value, the confidence interval (for the two-tailed test) and Cohen's d.

Power and minimum detectable difference

Power is the probability that the test detects a difference that really exists. It depends on the size of that difference, the variability, the sample sizes and \(\alpha\). Since the real difference is unknown, the calculator summarises it as the minimum detectable difference: the smallest difference between the population means that, with your group sizes and standard deviations, the test would detect with 80% probability.

If that difference is larger than the one that would matter in your problem, the study has low power: a non-significant result says little. The link in the result opens the sample size calculator for two means with your standard deviations, your α and the ratio between the groups; change the difference there to the one you care about and you will see how much data you would need.

It is not advisable to compute the so-called "observed power", that is, the power against the difference you observed: it depends only on the p-value and adds no information. The useful question is which differences the study is able to detect.

Relationship with the confidence interval

The two-tailed test and the confidence interval for the difference in means are equivalent: \(H_0\) is rejected at level \(\alpha\) exactly when the \(100(1-\alpha)\%\) interval for \(\mu_1 - \mu_2\) does not contain 0. The interval also gives the plausible size of the difference, which is what really matters for decisions.

Effect size: Cohen's d

To express the difference on a scale that can be compared across studies, Cohen's d is used: \(d = (\bar{x}_1 - \bar{x}_2)/s_p\), where \(s_p\) is the pooled standard deviation of the two groups. As a rough guide, 0.2 is a small effect, 0.5 medium and 0.8 large. In example 1, \(d \approx 0.44\); in example 2, \(d \approx 0.54\).

Common mistakes

  • Using this test with paired data (before and after in the same people). You lose the advantage of pairing and, usually, a lot of power.
  • Concluding that two groups are equal because the test is not significant.
  • Choosing between Welch and Student based on a preliminary variance test. Use Welch directly.
  • Running many pairwise tests when there are three or more groups: the probability of some false positive grows quickly. Use ANOVA followed by post hoc comparisons.
  • Ignoring the direction of the subtraction: the right-tailed test checks whether group 1 has a higher mean than group 2, not the other way round.
  • Attributing the difference to the cause under study without random assignment: in observational studies, the groups may differ in other ways.

Frequently asked questions

  • What hypothesis does it test? That the means of two independent populations are equal (H₀: μ₁ = μ₂), against the alternative that they differ or that one is greater than the other.
  • Why does the calculator use Welch's t? Because it does not require the two populations to have the same variance. When the variances are equal it gives practically the same result as Student's t, and when they are not it is much more reliable.
  • Why are the degrees of freedom not whole numbers? Because Welch approximates them with the Welch-Satterthwaite formula, which combines the variances and sizes of the two groups. Values such as 51.7 are normal.
  • Can it be used for paired samples? No. If each value in one group is paired with one in the other (for example, the same person before and after), use the paired means test.
  • Can the groups have different sizes? Yes. Welch's test works well with different sizes, although, for the same total amount of data, balanced groups give more power.
  • What if the data are not normal? With 30 or more observations per group the test is usually reliable thanks to the central limit theorem. With small samples and very skewed data, use the Mann-Whitney U test.
  • Two-tailed or one-tailed? Two-tailed, unless before seeing the data you are interested in only one direction of the difference. Remember that the difference is computed as group 1 minus group 2.
  • What is the minimum detectable difference? The smallest real difference between the means that the test would detect with 80% probability given your group sizes and variability. If it is larger than the difference you care about, the study has low power.
  • What sample size do I need? It depends on the difference you want to detect, the variability and the desired power. Work it out with the sample size calculator for two means.
  • What if I have three or more groups? Use one-way ANOVA. Running every pairwise test increases the probability of finding some significant difference by chance.

References: Student's t-test and Welch's t-test — Wikipedia