Statistical simulation

Bootstrap method simulator

Generate a sample from the chosen distribution, apply bootstrap resampling with replacement, and build confidence intervals for any statistic without parametric assumptions.

Simulation parameters

Original sample

Histogram of the sample with theoretical density curve

Click "New sample" to start

Bootstrap distribution of the statistic

Histogram of the B replicates with the CI shaded

Click "New sample" to start

What is the bootstrap?

The bootstrap is a resampling technique proposed by Bradley Efron in 1979. Its name comes from the English expression «pull yourself up by your own bootstraps», which refers to the idea of learning about the variability of an estimator using only the data from the sample itself, without needing to know the population distribution.

The core intuition is simple: if we don't know the population distribution \(F\), we can use the empirical distribution \(\hat{F}_n\) — which assigns probability \(1/n\) to each observation in the sample — as a substitute. If we draw many bootstrap samples from \(\hat{F}_n\) (that is, with replacement from the original sample) and compute the statistic of interest on each one, the variability of those replicates approximates the variability of the actual estimator.

Formally, let \(\hat{\theta} = T(X_1,\ldots,X_n)\) be an estimator of \(\theta\). Each bootstrap replicate \(\hat{\theta}^* = T(X_1^*,\ldots,X_n^*)\) is computed on a sample \((X_1^*,\ldots,X_n^*)\) drawn with replacement from the original sample. The distribution of \(\hat{\theta}^* - \hat{\theta}\) approximates the distribution of \(\hat{\theta} - \theta\).

\[\widehat{SE}_{\text{boot}} = \sqrt{\frac{1}{B-1}\sum_{b=1}^{B}\left(\hat{\theta}^{*(b)} - \bar{\hat{\theta}}^*\right)^2}\]

where \(\bar{\hat{\theta}}^* = \frac{1}{B}\sum_{b=1}^{B}\hat{\theta}^{*(b)}\) is the mean of the bootstrap replicates.

How the resampling works

The non-parametric bootstrap algorithm follows these steps:

  1. Take the original sample \(\mathbf{x} = (x_1, \ldots, x_n)\) of size \(n\).
  2. Generate \(B\) bootstrap samples \(\mathbf{x}^{*(1)}, \ldots, \mathbf{x}^{*(B)}\), each of size \(n\), by drawing with replacement from \(\mathbf{x}\). In each replicate some original values will appear several times and others won't appear at all (on average, about \(1 - e^{-1} \approx 63.2\,\%\) of the original values are represented in each replicate).
  3. Compute the statistic of interest \(\hat{\theta}^{*(b)} = T(\mathbf{x}^{*(b)})\) for each replicate \(b = 1, \ldots, B\).
  4. Estimate the standard error as the standard deviation of the \(B\) replicates (formula above).
  5. Build the percentile CI at \((1-\alpha) \times 100\,\%\) as the interval between the \(\alpha/2\) and \(1-\alpha/2\) percentiles of the bootstrap distribution:
\[\text{IC}_{\text{percentil}} = \left[\hat{\theta}^*_{(\alpha/2)},\; \hat{\theta}^*_{(1-\alpha/2)}\right]\]

The percentile method is the simplest one, and the one implemented in this simulator. For small samples or biased estimators, more sophisticated methods such as BCa (bias-corrected and accelerated) offer better coverage. In practice, \(B = 1000\) replicates is enough to estimate the standard error and build confidence intervals with good precision; \(B = 5000\) provides greater stability in the extreme percentiles.

When to use the bootstrap

The bootstrap is especially useful when classical parametric methods aren't directly applicable or their assumptions don't hold:

  • Small samples with unknown distribution: classical methods require normality or large sample sizes for the CLT to guarantee the approximation. The bootstrap works reasonably well from \(n \approx 15\text{–}20\) onward, although the inherent uncertainty of a small sample still persists.
  • Complex statistics: the median, Pearson correlation, the coefficient of variation, regression coefficients in non-linear models, or the Kolmogorov-Smirnov statistic don't have simple analytical expressions for their standard error. The bootstrap sidesteps that problem.
  • Non-normal data: if the data distribution has heavy tails or marked skewness, the classical confidence interval \(\bar{x} \pm z \cdot s/\sqrt{n}\) may have coverage below the nominal level. The percentile bootstrap captures the skewness of the estimator's distribution.
  • Comparison with the parametric CI: for the mean with normal data and large \(n\), the percentile bootstrap CI and the classical \(t\) CI are practically identical. Differences appear with skewed distributions or for statistics other than the mean.

Limitations: the bootstrap doesn't work well with statistics that depend on extreme values (maximum, minimum) in distributions with bounded support, nor with data that have strong temporal dependence (that's what the block bootstrap is for). It also doesn't «create» information: if the original sample is small and unrepresentative, the bootstrap can only redistribute the existing uncertainty.

Frequently asked questions

Does the bootstrap always work?

Not always. The non-parametric bootstrap fails when the statistic is sensitive to the extreme values of the empirical distribution. A classic example is the maximum of a uniform sample: the bootstrap distribution of the maximum systematically underestimates the parameter. It also fails with very small sample sizes (\(n \leq 5\)), where the empirical distribution is too discrete to approximate the real distribution well. In those cases it's preferable to use Bayesian methods or exact intervals.

What is the BCa method?

The BCa method (Bias-Corrected and Accelerated), also proposed by Efron, corrects two shortcomings of the simple percentile method: the bias of the estimator (the difference between \(\bar{\hat{\theta}}^*\) and \(\hat{\theta}\)) and the acceleration (how the standard error changes with the parameter value). The CI percentiles are shifted according to these corrections, which improves the actual coverage for biased estimators or when the scale of the estimator's distribution isn't constant. The acceleration is usually computed via the jackknife method. For most practical applications with \(n \geq 30\) BCa and the percentile method give similar results.

How many replicates B do I need?

To estimate the bootstrap standard error, \(B = 200\text{–}500\) replicates is usually enough. To build 95% percentile confidence intervals, \(B \geq 1000\) is recommended. For extreme percentiles (99% or 99.9%) or for the BCa method, \(B = 5000\text{–}10\,000\) is advisable. Beyond \(B = 10\,000\), the precision gain is marginal and the computational cost can be high with large datasets. The variability of the bootstrap CI decreases roughly as \(1/\sqrt{B}\), similar to Monte Carlo error.

What is the difference between the bootstrap and the jackknife?

The jackknife (Quenouille, 1949; popularized by Tukey) is another resampling technique that estimates the bias and standard error of an estimator by systematically leaving out one observation at a time: it generates exactly \(n\) «leave-one-out» replicates. It's faster for large datasets but less general than the bootstrap. It doesn't capture the full distribution of the estimator well, nor can it be used to build percentile CIs. The bootstrap, because of its randomness, needs \(B \gg n\) replicates to be stable, but it can estimate the entire shape of the estimator's distribution and handle more complex statistics. The first-order jackknife is a linearization of the bootstrap and converges to it as \(B \to \infty\).