Calculator
Enter the experiment data and the prior parameters to get the full Bayesian analysis.
Beta-Binomial model
In the Bayesian conjugate model for proportions, the prior is a Beta distribution and the likelihood is binomial. The resulting posterior is also a Beta, which allows exact analytical calculations.
\( p_A \sim \text{Beta}(\alpha_0, \beta_0) \xrightarrow{\text{data}} p_A \mid \text{data} \sim \text{Beta}(\alpha_0 + x_A,\; \beta_0 + n_A - x_A) \)
\( p_B \sim \text{Beta}(\alpha_0, \beta_0) \xrightarrow{\text{data}} p_B \mid \text{data} \sim \text{Beta}(\alpha_0 + x_B,\; \beta_0 + n_B - x_B) \)
Choosing the prior
The Beta(1, 1) prior is uniform: it expresses total ignorance about the conversion rate. A Beta(α₀, β₀) prior with a large α₀ + β₀ is equivalent to having prior historical data. The calculator lets you customize it.
P(B > A) via Monte Carlo
The probability that B's conversion rate exceeds A's is estimated by sampling from both posteriors:
\( P(p_B > p_A) \approx \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\!\left[p_B^{(i)} > p_A^{(i)}\right], \quad p_A^{(i)}, p_B^{(i)} \sim \text{posteriors} \)
Worked example
A product team wants to improve the sign-up rate on a SaaS application's homepage. The current version (A) has a known conversion rate of roughly 8%. Variant B introduces a new headline and a more prominent call-to-action button. A non-informative Beta(1, 1) prior is adopted for both variants, expressing total ignorance about the true rate before observing any data.
After 14 days of the experiment, the following data are collected: group A receives 1,200 visitors and records 94 conversions; group B receives 1,200 visitors and records 113 conversions.
The Bayesian update is immediate thanks to Beta-Binomial conjugacy. The posterior distribution of each rate is:
\( p_A \mid \text{data} \sim \text{Beta}(1 + 94,\; 1 + 1{,}200 - 94) = \text{Beta}(95,\; 1{,}107) \)
\( p_B \mid \text{data} \sim \text{Beta}(1 + 113,\; 1 + 1{,}200 - 113) = \text{Beta}(114,\; 1{,}088) \)
The posterior mean of A is \( \hat{p}_A = 95/(95+1{,}107) \approx 7.9\% \) and that of B is \( \hat{p}_B = 114/(114+1{,}088) \approx 9.5\% \). Using Monte Carlo with 30,000 samples drawn from both posterior distributions, the probability that B beats A is estimated as:
\( P(p_B > p_A) \approx 0.962 \)
The expected lift of B over A is approximately \( (9.5 - 7.9)/7.9 \approx +20\% \) in relative terms. With a 96.2% probability that variant B is genuinely better, the team exceeds the usual decision threshold (95%) and can roll out B as the new default version, estimating that the change will generate around 16 additional sign-ups per 1,000 visits.
Advantages of the Bayesian approach
- P(B > A) is directly interpretable by stakeholders without statistical training.
- It doesn't require fixing the sample size in advance; it can be updated with new data (although peeking must be controlled).
- The prior allows incorporating historical knowledge about the baseline conversion rate.
- It avoids the p-value problem: there is no binary significance threshold.
How to interpret the result
The central result of the Bayesian analysis is P(B > A): the posterior probability that B's conversion rate is genuinely higher than A's, given the observed data. Unlike the frequentist p-value, this probability is directly interpretable: if P(B > A) = 0.97, it means that, with the current data, there is 97% credibility that B beats A. In practice, values above 0.95 are usually considered sufficient evidence to decide to roll out variant B, although the exact threshold depends on the context and the relative cost of the possible errors.
The posterior distributions represent the updated knowledge about each variant's conversion rate after combining the prior with the experiment data. Following the Beta-Binomial model, A's posterior is \(\text{Beta}(\alpha_0 + x_A,\; \beta_0 + n_A - x_A)\) and B's is \(\text{Beta}(\alpha_0 + x_B,\; \beta_0 + n_B - x_B)\). The mean of each posterior is the point estimate of the true rate; when the two distributions overlap little in the chart, the difference between variants is clear; when the overlap is large, uncertainty is high and it's worth continuing to collect data.
The credible interval at the selected level (90%, 95% or 99%) indicates the range of conversion rate values that contains the selected credibility level of the posterior mass. Unlike the frequentist confidence interval, it can be interpreted directly: at a 95% level, "there is a 95% probability that B's true rate lies between these two values" (and analogously for 90% or 99%). If A's and B's intervals do not overlap, the evidence of a difference is very strong. The expected lift complements this analysis by showing the average relative improvement of B over A according to the posteriors, a more direct indicator of practical relevance than the p-value.
Frequently asked questions
- What prior should I use if I have no historical data? Beta(1, 1), the uniform prior. If you have historical conversion rates, you can parameterize the prior as if you had α₀ prior successes and β₀ prior failures.
- How many Monte Carlo samples does the calculator use? 30,000. This gives a standard error for the P(B > A) estimate of about ±0.003 for values around 0.5.
- Can I use this with small sample sizes? Yes. The biggest advantage of the Bayesian approach is that it works correctly with small samples, where the z-test can be imprecise.
- What's the difference between P(B > A) = 0.97 and a p-value of 0.03? The p-value measures the probability of observing data as extreme under H₀; P(B > A) directly measures how much you believe B beats A given what you've observed. The latter is more intuitive.