Hypothesis test calculators

All available hypothesis tests

Pillar page of the hypothesis testing cluster: it concentrates the broad statistical-testing intent and links to each specific test through differentiated anchors.

What is a hypothesis test?

A hypothesis test (also called a statistical test) is a formal procedure that lets you decide, from sample data, whether the empirical evidence is strong enough to reject a starting claim about the population. The core idea is simple: we state two competing hypotheses and use the data to see which one is more compatible with reality.

This kind of reasoning shows up across many fields: medicine (is this drug more effective than a placebo?), engineering (has the new production process improved?), marketing (which of two page versions converts better?), or social science (do scores differ between two groups?). In every case, the logical scheme is the same.

The logic of a test: statistical proof by contradiction

The mechanism works much like a proof by contradiction. You start from a reference hypothesis — the null hypothesis (H₀) — which represents the scenario you want to challenge: "there is no effect," "there is no difference," "the parameter equals exactly μ₀." You then work out what data you would expect to see if H₀ were true. If the observed data are very unlikely under that scenario, you conclude that H₀ is implausible and reject it in favor of the alternative hypothesis (H₁).

It's crucial to understand that a hypothesis test never "proves" anything with absolute certainty. What it does is quantify the compatibility between the data and H₀, and then, using a probability threshold (the significance level α), make an operational decision.

The hypotheses: null and alternative

Null hypothesis (H₀)

The null hypothesis is the conservative starting point. It usually states that there is no effect, that the means are equal, or that a parameter takes a specific value. It is always written with an equality (=, ≤ or ≥, depending on the type of test). Examples:

• H₀: μ = 50 (the population mean is 50)
• H₀: μ₁ = μ₂ (the two means are equal)
• H₀: p = 0.30 (the proportion is 30%)
• H₀: σ₁² = σ₂² (the variances are equal)

Alternative hypothesis (H₁ or Hₐ)

The alternative hypothesis expresses what you suspect or want to demonstrate: that there is an effect, a difference, or a change in a particular direction. It comes in three variants, depending on the type of test:

• Two-sided (two-tailed): H₁: μ ≠ μ₀ — rejected whether the statistic is very high or very low.
• Right-tailed (upper tail): H₁: μ > μ₀ — rejected only if the statistic falls in the upper tail.
• Left-tailed (lower tail): H₁: μ < μ₀ — rejected only if the statistic falls in the lower tail.

The choice between two-sided and one-sided must be made before looking at the data, based on the research question. Choosing the type of test after seeing the data artificially inflates the false-positive rate.

The test statistic and the p-value

Test statistic

The test statistic is a number computed from the sample that summarizes the evidence against H₀. Its exact distribution under H₀ is known mathematically, which lets you compute probabilities. The most common ones are:

• z: used when the population variance is known or the sample size is large (roughly n ≥ 30). It follows a standard normal distribution N(0, 1) under H₀.
• Student's t: used when the population variance is unknown and estimated from the sample. It follows a t distribution with n − 1 degrees of freedom.
• Snedecor's F: used when comparing variances or in ANOVA, where variances are compared across groups.
• χ² (chi-square): used in contingency tables and goodness-of-fit tests.

The p-value

The p-value is the probability of obtaining a statistic as extreme as, or more extreme than, the one observed, assuming H₀ is true. Formally:

• Two-sided test: p = P(|T| ≥ |t_obs| | H₀)
• Right tail: p = P(T ≥ t_obs | H₀)
• Left tail: p = P(T ≤ t_obs | H₀)

A small p-value indicates that the observed data would be very unlikely if H₀ were true, so there is evidence against H₀. The decision is made by comparing the p-value with the significance level α, set in advance:

• If p-value < α → reject H₀.
• If p-value ≥ α → do not reject H₀ (it is not "accepted" — there simply isn't enough evidence to reject it).

What the p-value is not

The p-value is one of the most misunderstood concepts in statistics. It's important to be clear about what it does not mean:

• It is not the probability that H₀ is true.
• It is not the probability that you made an error.
• It does not indicate the size or the practical importance of the effect.
• A p-value of 0.049 is not "almost significant"; 0.051 is not "almost not significant." The α threshold is an operational convention, not a boundary of knowledge.

Type I and Type II errors

Whenever you make a statistical decision there is always a chance of being wrong. There are two distinct kinds of error, and there's a fundamental tension between them:

The relationship between α and β is inverse, all else being equal: lowering α (being stricter about rejecting H₀) increases β (makes it harder to detect real effects). The only way to reduce both at once is to increase the sample size.

Statistical power: what it is and what it depends on

Power answers the question: "if the effect I suspect really exists, how likely is my study to detect it?" It's one of the most important criteria in study design, even though in practice it often gets overlooked.

Statistical power (1 − β) depends on four interacting factors:

• Sample size (n): the factor with the greatest practical impact. As n grows, the standard error shrinks, confidence intervals narrow, and the distribution of the statistic under H₁ moves further from the distribution under H₀, making detection easier.
• Effect size (d, f, w…): the larger the real difference between the parameter under H₁ and the value under H₀, the easier it is to detect. If the real effect is very small, a much larger sample is needed to achieve adequate power.
• Variability of the data (σ): greater spread "masks" the signal. With more homogeneous data, the same difference is easier to detect.
• Significance level (α): raising α (for example from 0.01 to 0.05) widens the rejection region and increases power, but at the cost of admitting more false positives.

An a priori power analysis — carried out before collecting data — lets you determine the sample size needed to reach a target power (typically 0.80 or 0.90) given a minimum effect size of interest. Doing this is essential practice in rigorous scientific studies.

Effect size: beyond the p-value

Statistically significant is not the same as practically important. This confusion is one of the most serious and common mistakes in applied statistics.

With a very large sample, a difference of 0.1 points between two groups can produce p < 0.001, even though it has no practical relevance whatsoever. Conversely, with small samples, a clinically relevant difference may fail to reach statistical significance.

Effect size indices quantify the magnitude of a difference in standardized units, independent of sample size:

• Cohen's d (difference of two means): d = (μ₁ − μ₂) / σ_pooled. Rough guidelines: small ≈ 0.2; medium ≈ 0.5; large ≈ 0.8.
• Pearson's r (correlation or ANOVA): r = 0.1 (small), 0.3 (medium), 0.5 (large).
• η² and partial η² (ANOVA): proportion of variance explained by the factor.
• Cohen's w (chi-square): w = 0.1 (small), 0.3 (medium), 0.5 (large).
• Relative risk, odds ratio (proportions): indicate how many times more likely the event is in one group relative to the other.

Current best practice recommends always reporting, alongside the p-value:

• The point estimate of the effect (difference of means, difference of proportions, odds ratio…).
• The confidence interval of the effect (typically at 95%).
• A standardized effect size index.
• An assessment of practical relevance in the context of the problem.

Steps to carry out a hypothesis test

Regardless of the type of test, the procedure always follows the same scheme:

1. State the hypotheses: formulate H₀ and H₁ based on the research question, and decide whether the test is two-sided or one-sided.
2. Set the significance level (α): typically 0.05, though stricter contexts use 0.01. It must be set before collecting data.
3. Check the test's assumptions: normality, independence of observations, homogeneity of variances, minimum expected frequencies, etc.
4. Choose the appropriate statistic: z, t, F, χ²… depending on the type of variable, the number of groups, and the assumptions verified.
5. Compute the statistic and the p-value from the sample data.
6. Make the decision: if p-value < α, reject H₀; if p-value ≥ α, do not reject H₀.
7. Compute the effect size and interpret the results in context.
8. Report the results: include the statistic, degrees of freedom, p-value, estimated effect, and confidence interval.

Common assumptions and how to check them

Every test rests on a set of assumptions that must hold (at least approximately) for the inference to be valid. Ignoring the assumptions is one of the most common sources of misleading results.

Normality

Parametric tests (t, z, F) assume that the data come from a normal distribution or, thanks to the Central Limit Theorem, that the sample size is large enough for the statistic to converge to the theoretical distribution. For n ≥ 30 the CLT is usually sufficient; with small samples it's worth checking normality with the Shapiro-Wilk test or the Kolmogorov-Smirnov test.

Independence of observations

Observations must be independent of one another. The most frequent violation occurs when there are repeated measures on the same subject (before/after) or when the data are clustered. In these cases you should use a paired-sample test or mixed models.

Homogeneity of variances (homoscedasticity)

The two-independent-samples t-test and ANOVA assume the group variances are equal. This can be checked with Levene's test. If it doesn't hold, robust alternatives exist, such as Welch's test for two means.

Minimum expected frequencies (chi-square)

In contingency tables, each cell should have an expected frequency of at least 5 (some sources say 1, provided no more than 20% of cells fall below 5). If this doesn't hold, use Fisher's exact test.

Common mistakes and myths about hypothesis testing

1. "p < 0.05 means there's a real effect"

A small p-value only indicates that the data are unlikely under H₀. It doesn't guarantee the effect is real or reproducible. With many simultaneous comparisons (the multiple-comparisons problem), obtaining p < 0.05 by pure chance is to be expected.

2. "p = 0.06 isn't significant, so there's no effect"

Absence of statistical significance is not evidence of absence of an effect. It can simply reflect an insufficient sample. Reporting only whether the 0.05 threshold was crossed is poor practice that discards valuable information.

3. "The p-value is the probability that H₀ is true"

The p-value is P(data as extreme or more | H₀ true), not P(H₀ true | data). Getting probabilities about hypotheses requires a Bayesian approach with a prior distribution.

4. "With a large n you always get significance"

True, and that's exactly the problem. With n = 1,000,000, an irrelevant difference of 0.001 units can have p < 0.001. That's why effect size and confidence intervals are indispensable.

5. "If the test doesn't meet its assumptions, the results are useless"

It depends on how severe the violation is. Many tests are robust to moderate departures from normality, especially with large n. In severe cases, non-parametric alternatives or data transformations exist.

6. "A hypothesis test is enough to make a decision"

A test is a tool, not an oracle. Decisions should integrate context, costs, the effect size, and the uncertainty represented by the confidence interval.

Best-practice recommendations

• Pre-register: Define the hypotheses, sample size, and α level before collecting data. Avoid p-hacking (tweaking the analysis until p < 0.05).
• Compute the required sample size a priori via a power analysis. Use the available sample size calculators.
• Report the estimated effect and its confidence interval, not just the p-value and the decision.
• Distinguish statistical significance from practical relevance. Interpret results in the context of the real-world problem.
• Check the test's assumptions before interpreting the results.
• Apply a multiple-comparisons correction (Bonferroni, Benjamini-Hochberg…) when running more than one test simultaneously.
• Consider a Bayesian approach as an alternative or complement, especially when the prior distribution carries relevant information.

Key reference

In 2016 the American Statistical Association published an official statement on the use and interpretation of the p-value, recommended reading for anyone working with hypothesis tests: ASA Statement on Statistical Significance and P-Values (Wasserstein & Lazar, 2016).