Distributions under H₀ and H₁
Standardized Z statistic. The shaded regions represent the probabilities of each type of error and the power of the test.
What are type I and type II errors?
When we run a hypothesis test we make a binary decision: reject the null hypothesis H₀ or fail to reject it. Like any decision based on incomplete information (a sample, not the entire population), this decision can be wrong in two distinct ways, and we call these two ways type I error and type II error.
A type I error occurs when we reject H₀ while it is actually true in the population. The probability of making this error is precisely the significance level α that we fix before running the test. If we set α = 0.05, we are accepting that there is a 5% probability of declaring a significant effect when in fact none exists. This error is also called a false positive or false alarm.
A type II error occurs when we fail to reject H₀ when it is false — that is, when a real effect exists in the population but our test fails to detect it. The probability of making this error is denoted by the letter β and is also called a false negative. Its complement, 1 − β, is the statistical power: the probability that the test correctly detects an effect when one exists.
A useful analogy is a courtroom verdict. Suppose H₀ is "the defendant is innocent." A type I error is equivalent to convicting an innocent person (false positive), while a type II error is equivalent to acquitting a guilty one (false negative). In criminal law, the tradition is to specifically protect against type I errors ("better that a hundred guilty go free than one innocent be convicted"), setting a very high evidentiary bar. In science and medicine, the optimal balance between the two errors depends on the practical consequences of each kind of mistake.
The relationship between α and β is inseparable: holding everything else constant, lowering α (being stricter) increases β (raises the probability of missing real effects), and vice versa. This trade-off is the central dilemma in the design of statistical studies.
| Decision \ Reality | H₀ true | H₀ false (H₁ true) |
|---|---|---|
| Fail to reject H₀ | Correct decision (1 − α) | Type II error (β) |
| Reject H₀ | Type I error (α) | Correct decision: Power (1 − β) |
Statistical power
Statistical power (1 − β) is the probability that a test correctly detects a real effect. It is, in a sense, the "sensitivity" of the test: a high-power test rarely misses effects that truly exist in the population.
In the simulator, power corresponds to the green region under the H₁ curve within the rejection zone. The greater the overlap between the two distributions (H₀ and H₁), the lower the power; the more separated they are, the higher the power. Four factors determine power:
- Effect size (Cohen's d): the real difference between means expressed in units of standard deviation. The larger the effect, the easier it is to detect. An effect of d = 0.2 is considered small, d = 0.5 medium and d = 0.8 large.
- Sample size (n): the factor most easily controlled in the design. As n increases, the standard error of the mean decreases, the distributions become narrower and their overlap shrinks. The non-centrality parameter δ = d × √n jointly captures the effect of the effect size and the sample size.
- Significance level (α): increasing α shifts the rejection region to the left (in a right-tailed test), capturing more area under the H₁ curve and increasing power. In exchange, it raises the false-positive rate.
- Measurement variability (σ): a measure with less noise produces more precise statistics. In the simulator, the standard deviation is standardized to σ = 1, but in practice reducing experimental variability (a better protocol, more homogeneous analysis units) directly increases power.
In scientific research and clinical trials it is common to set a minimum target power of 80% (β ≤ 0.20). This means we accept up to a 20% probability of missing a real effect when it exists. Some more demanding fields, such as pivotal pharmacology trials, require power of 90% or even higher.
A frequent and problematic practice is the underpowered study: a study with insufficient power that, if it obtains a significant result, probably did so due to chance (the Ioannidis effect, 2005). Underpowered studies that do detect effects also tend to overestimate the effect size, the so-called winner's curse.
The α vs β trade-off
The tension between the type I error and the type II error is one of the most fundamental problems in statistical inference. With a fixed sample size, it is not possible to minimize both simultaneously: any change in α inevitably affects β in the opposite direction.
To visualize this intuitively: the critical value zα is the point that separates the non-rejection region from the rejection region. Lowering α, for example moving from z0.05 = 1.645 to z0.01 = 2.326, shifts the decision threshold to the right. This shift means the rejection region captures less area of the distribution under H₁, which increases β and reduces power.
The only way to reduce both errors simultaneously is to increase the sample size. As n grows, the two distributions (under H₀ and under H₁) become narrower and more separated, reducing their overlap. This makes it possible to keep α low while also achieving high power.
From a theoretical standpoint, the Neyman-Pearson lemma (1933) establishes that the likelihood-ratio test is the most powerful test of level α for testing simple hypotheses. In other words, given fixed α and n, the likelihood-based test maximizes power — no other test of the same level can outperform it.
In practice, the choice of α depends on context: basic research conventionally uses α = 0.05; exploratory studies may relax it to α = 0.10 to avoid missing potentially interesting findings; applications with high consequences for type I error (medical diagnosis, safety quality control) use much stricter values (α = 0.001 or lower). In any case, the choice should be explicit, justified and set before collecting the data.
When the economic or human consequences of both types of error are quantifiable, statistical decision theory proposes balancing the two errors by weighting their respective costs. If the cost of a false positive is CI and the cost of a false negative is CII, the optimal decision threshold depends on the ratio CI/CII and on the prior prevalence of H₁, which connects to the Bayesian approach to inference.
Frequently asked questions
Why not always use α = 0.001 to be safer?
Using a very small α protects against false positives, but it also drastically increases the type II error (β) for a given sample size. If a real effect exists, a very strict α makes the test so demanding that it rarely reaches the rejection region, producing many false negatives. Moreover, in exploratory research, being too strict can lead to discarding true hypotheses before they can be investigated further. The choice of significance level should balance the cost of the two types of error according to the context of the study.
What is the p-value and how does it relate to α?
The p-value is the probability of obtaining a statistic as extreme as, or more extreme than, the one observed, assuming H₀ is true. It is not the probability that H₀ is true (a very common conceptual mistake). The decision rule is: if p ≤ α, we reject H₀; if p > α, we fail to reject H₀. The significance level α is the pre-set threshold against which we compare the p-value. Therefore, α directly controls the long-run type I error rate: if we apply the rule p ≤ α across many experiments in which H₀ is true, approximately a fraction α of them will produce significant results by chance.
How can I increase power without changing α?
The most direct way is to increase the sample size. The non-centrality parameter δ = d × √n grows with n, separating the distributions under H₀ and H₁ further apart and reducing their overlap. Other strategies include: (1) increasing the expected effect size, designing the study to maximize the difference you want to detect (for example, using higher doses in a clinical trial, if that is ethical and feasible); (2) reducing the variability of measurements through a more rigorous protocol, more precise measurements or a more homogeneous experimental design; (3) using more efficient designs, such as paired or block designs, which remove variability due to known covariates; and (4) performing an a priori power analysis to determine the required n before starting the study.
What is Cohen's effect size?
Cohen's d is the difference between two means expressed in units of standard deviation. Formally, d = (μ₁ − μ₀) / σ, where σ is the common standard deviation of the populations. This standardization allows effects to be compared across studies with different measurement scales. Jacob Cohen (1988) proposed the thresholds d = 0.2 (small effect), d = 0.5 (medium effect) and d = 0.8 (large effect). However, these thresholds are only a guide and should not be applied mechanically: an effect of d = 0.2 can be highly relevant in preventive medicine (if it affects millions of people) and irrelevant in experimental psychology. Effect size should always be interpreted in the substantive context of the research.