Bayesian A/B test — Beta-Binomial

Beta-Binomial model

In the Bayesian conjugate model for proportions, the prior is a Beta distribution and the likelihood is binomial. The resulting posterior is also a Beta, so the posterior distributions and their credible intervals can be obtained analytically. The probability P(B > A), on the other hand, is estimated by Monte Carlo and carries a small simulation error.

\( p_A \sim \text{Beta}(\alpha_0, \beta_0) \xrightarrow{\text{data}} p_A \mid \text{data} \sim \text{Beta}(\alpha_0 + x_A,\; \beta_0 + n_A - x_A) \)

\( p_B \sim \text{Beta}(\alpha_0, \beta_0) \xrightarrow{\text{data}} p_B \mid \text{data} \sim \text{Beta}(\alpha_0 + x_B,\; \beta_0 + n_B - x_B) \)

Choosing the prior

The Beta(1, 1) prior is uniform: it expresses total ignorance about the conversion rate. A Beta(α₀, β₀) prior with a large α₀ + β₀ is equivalent to having prior historical data. The calculator lets you customize it.

P(B > A) via Monte Carlo

The probability that B's conversion rate exceeds A's is estimated by sampling from both posteriors:

\( P(p_B > p_A) \approx \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\!\left[p_B^{(i)} > p_A^{(i)}\right], \quad p_A^{(i)}, p_B^{(i)} \sim \text{posteriors} \)

Worked example

A product team wants to improve the sign-up rate on a SaaS application's homepage. The current version (A) has a known conversion rate of roughly 8%. Variant B introduces a new headline and a more prominent call-to-action button. A non-informative Beta(1, 1) prior is adopted for both variants, expressing total ignorance about the true rate before observing any data.

After 14 days of the experiment, the following data are collected: group A receives 1,200 visitors and records 94 conversions; group B receives 1,200 visitors and records 113 conversions.

The Bayesian update is immediate thanks to Beta-Binomial conjugacy. The posterior distribution of each rate is:

\( p_A \mid \text{data} \sim \text{Beta}(1 + 94,\; 1 + 1{,}200 - 94) = \text{Beta}(95,\; 1{,}107) \)

\( p_B \mid \text{data} \sim \text{Beta}(1 + 113,\; 1 + 1{,}200 - 113) = \text{Beta}(114,\; 1{,}088) \)

The posterior mean of A is \( \hat{p}_A = 95/(95+1{,}107) \approx 7.9\% \) and that of B is \( \hat{p}_B = 114/(114+1{,}088) \approx 9.5\% \). Using Monte Carlo with 30,000 samples drawn from both posterior distributions, the probability that B beats A is estimated as:

\( P(p_B > p_A) \approx 0.916 \)

With 30,000 samples, the Monte Carlo standard error of this estimate is \( \sqrt{0.916 \cdot 0.084/30{,}000} \approx 0.002 \), so the result can change in the third decimal place from one run to the next.

The expected lift of B over A is approximately \( (9.5 - 7.9)/7.9 \approx +20\% \) in relative terms. With a 91.6% posterior probability that variant B is better, the evidence favours B, but the team does not reach the usual 95% decision threshold. It can extend the experiment to gather more data or decide by weighing the cost of each possible error; if it does roll out B, the central estimate is around 16 additional sign-ups per 1,000 visits, with considerable uncertainty.

Advantages of the Bayesian approach

• P(B > A) is directly interpretable by stakeholders without statistical training.
• It doesn't require fixing the sample size in advance; it can be updated with new data (although peeking must be controlled).
• The prior allows incorporating historical knowledge about the baseline conversion rate.
• It avoids the p-value problem: there is no binary significance threshold.

How to interpret the result

The central result of the Bayesian analysis is P(B > A): the posterior probability that B's conversion rate is genuinely higher than A's, given the observed data. Unlike the frequentist p-value, this probability is directly interpretable: if P(B > A) = 0.97, it means that, with the current data, there is 97% credibility that B beats A. In practice, values above 0.95 are usually considered sufficient evidence to decide to roll out variant B, although the exact threshold depends on the context and the relative cost of the possible errors.

The posterior distributions represent the updated knowledge about each variant's conversion rate after combining the prior with the experiment data. Following the Beta-Binomial model, A's posterior is \(\text{Beta}(\alpha_0 + x_A,\; \beta_0 + n_A - x_A)\) and B's is \(\text{Beta}(\alpha_0 + x_B,\; \beta_0 + n_B - x_B)\). The mean of each posterior is the point estimate of the true rate; when the two distributions overlap little in the chart, the difference between variants is clear; when the overlap is large, uncertainty is high and it's worth continuing to collect data.

The credible interval at the selected level (90%, 95% or 99%) indicates the range of conversion rate values that contains the selected credibility level of the posterior mass. Unlike the frequentist confidence interval, it can be interpreted directly: at a 95% level, "there is a 95% probability that B's true rate lies between these two values" (and analogously for 90% or 99%). If A's and B's intervals do not overlap, the evidence of a difference is very strong. The expected lift complements this analysis by showing the average relative improvement of B over A according to the posteriors, a more direct indicator of practical relevance than the p-value.

Frequently asked questions

• What prior should I use if I have no historical data? Beta(1, 1), the uniform prior. If you have historical conversion rates, you can parameterize the prior as if you had α₀ prior successes and β₀ prior failures.
• How many Monte Carlo samples does the calculator use? 30,000. The standard error of the P(B > A) estimate is approximately √(p(1−p)/30,000): ±0.003 for values around 0.5 and ±0.001 for values around 0.97.
• Can I use this with small sample sizes? Yes. The Bayesian calculation does not rely on asymptotic approximations, so it remains valid with small samples, where the z-test can be imprecise. In exchange, with little data the result depends more on the chosen prior, so check how it changes under reasonable alternative priors.
• What's the difference between P(B > A) = 0.97 and a p-value of 0.03? The p-value measures the probability of observing data as extreme under H₀; P(B > A) directly measures how much you believe B beats A given what you've observed. The latter is more intuitive.