Interactive simulator
Select the mode, adjust the parameters, and click Simulate.
What does a 95% confidence interval actually mean?
The correct interpretation of a 95% confidence interval is frequently misunderstood, even by researchers with statistical training. The correct statement is:
"If we repeated the experiment infinitely many times and built the confidence interval on each repetition, 95% of those intervals would contain the true parameter."
This is the statement the simulator makes visible: each row in the chart is one repetition of the experiment. The blue bars are the intervals that captured the parameter; the red ones did not. If the procedure is correct, roughly 5% of the intervals will be red.
What it does NOT mean: "there is a 95% probability that the parameter is in this particular interval." Once you've computed the interval from your data, the parameter either is inside it or it isn't. There's no probability in that classical sense: the parameter is a fixed (though unknown) value, and the computed interval is also fixed. The 95% probability refers to the construction procedure, not to the individual interval.
This distinction matters because it has practical consequences: you can't say "I'm 95% sure μ is between 47 and 53." What you can say is "the method I used to compute this interval produces intervals that cover the true μ 95% of the time across repetitions of the experiment."
The Bayesian interpretation — "the probability that the parameter lies in this range is 95%" — is valid for Bayesian credible intervals, which require specifying a prior distribution and are conceptually different.
Nominal coverage vs actual coverage
The nominal level is the confidence value we specify when constructing the CI (90%, 95%, 99%). The actual (or empirical) coverage is the fraction of intervals that actually cover the true parameter across a long series of repetitions.
For the CI of the mean with known σ, the actual coverage matches the nominal level exactly for any sample size, because the distribution of the sample mean is exactly normal (if the population is too) or asymptotically normal by the CLT.
For proportions, the situation is more complicated. The parameter p is continuous, but the binomial variable X ~ Bin(n, p) is discrete, and this discretization causes the actual coverage to fluctuate around the nominal level:
- Wald interval: uses the normal approximation directly, \(\hat{p} \pm z_{\alpha/2}\sqrt{\hat{p}(1-\hat{p})/n}\). It works well for large n and p away from 0 and 1, but its coverage can be noticeably below the nominal level when n is small or p is near the extremes.
- Wilson interval: obtained by inverting Pearson's score test. It tends to have actual coverage closer to the nominal level, especially at the extremes, though it can be slightly conservative (coverage above nominal) in some cases.
You can observe this phenomenon directly with the simulator: select Proportion mode, set p = 0.05 and n = 20, and compare the empirical coverage of Wald vs Wilson.
How does sample size affect it?
The sample size n has a direct and clear effect on the width of the confidence interval. For the CI of the mean with known σ:
\[ \text{Width} = 2 \times z_{\alpha/2} \times \dfrac{\sigma}{\sqrt{n}} \]
The width is proportional to \(1/\sqrt{n}\): if we quadruple n, the width is cut in half. To reduce the width to a third, we need to multiply n by 9.
However, increasing n does not change the coverage: a 95% interval is still 95% regardless of whether n = 10 or n = 1000. What changes is how narrow that interval is, i.e. the precision of the estimate. Narrower intervals are more useful in practice, even though both have the same nominal coverage guarantee.
You can verify this in the simulator: keep the confidence level fixed at 95% and the number of intervals at 100, and vary n between 5 and 100. You'll observe that:
- The fraction of blue intervals stays at roughly 95% for all values of n.
- The intervals become progressively narrower as n increases.
- With small n, there's more variability in the position of the CI's endpoints.
Frequently asked questions
- If the CI doesn't contain the true value, was it calculated wrong? No. A 95% CI is designed to fail to cover the parameter roughly 5% of the time. Seeing red intervals in the simulation is exactly what's expected; it doesn't indicate any calculation error. If 95% of the intervals are blue, the procedure is working correctly.
- Why does the Wilson method give better coverage than Wald? The Wald interval applies the normal approximation directly to \(\hat{p}\), which deteriorates when \(\hat{p}\) is close to 0 or 1 or when n is small (the distribution of \(\hat{p}\) isn't symmetric enough). The Wilson interval gets its bounds by inverting Pearson's score test, which accounts for the curvature of the binomial distribution. This produces empirical coverage much closer to the nominal level across the whole range of p.
- Can I interpret the CI in Bayesian terms? Only if you use a Bayesian credible interval, which has a completely different construction. The frequentist CI doesn't assign probabilities to the parameter (which is treated as fixed but unknown). The Bayesian credible interval does express P(θ ∈ CI | data) = 1 − α, but it requires specifying a prior distribution over θ. Both tools are valid in their own context, but they shouldn't be confused.
- How many intervals do I need to simulate to see the coverage converge? With 20 intervals there's a lot of variability: the empirical coverage can swing between 80% and 100% even if the procedure is correct. With 100 intervals, the standard error of the proportion is ≈ √(0.95 × 0.05 / 100) ≈ 2.2%, so the empirical coverage will typically fall between 91% and 99%. With 200 intervals, the variability drops to ≈ 1.5%.
- Does the simulator use an exact normal distribution for the mean? Yes: in Mean mode (known σ) each observation is generated from an N(μ, σ) using the Box-Muller method. The sample mean has an exactly N(μ, σ/√n) distribution for any n, so the empirical coverage should match the nominal level even for small n.