What is an A/B test and how does it work

What is an A/B test?

An A/B test —also called a split test or A/B experiment— is a controlled experimental method in which two versions of something are compared to determine which one produces better results. The original version is called the control (A), and the modified version is called the treatment or variant (B).

The principle is simple: users are randomly split into two groups, each group experiences one of the two versions, and at the end the metric of interest —conversion rate, click-through rate, time on page, revenue per visit, and so on— is compared using inferential statistics to determine whether the observed difference is real or simply the result of chance.

The A/B test is the central tool of growth and experimentation marketing, CRO (Conversion Rate Optimization) and data-driven product design. Companies like Google, Amazon and Netflix run thousands of A/B tests simultaneously to make evidence-backed decisions.

What is an A/B test used for?

An A/B test is used to make objective decisions in any context where a binary or continuous outcome can be measured. The most common use cases are:

• E-commerce and CRO: comparing two versions of a product page, a buy button, a headline or a sign-up form.
• Email marketing: testing subject lines, calls to action or formats to maximize open or click rates.
• Mobile and web apps: evaluating interface changes, onboarding flows or new features before rolling them out to 100% of users.
• Online advertising: comparing creatives, audiences or landing pages to optimize cost per acquisition.
• Clinical research and behavioral science: where randomized experimental design is the strongest standard of evidence.

How an A/B test works, step by step

A well-designed A/B experiment always follows the same sequence. Skipping any of these steps is the most frequent cause of invalid results.

1. Define the hypothesis and the primary metric. Before touching anything, document what you're changing, why you expect it to improve things, and which single metric you'll use to decide. A poorly defined hypothesis opens the door to confirmation bias.
2. Compute the required sample size. Based on the current baseline rate, the minimum detectable effect (MDE) you consider relevant to the business, the significance level α and the desired power (1−β). Use the MDE and power calculator for this step.
3. Randomize the assignment. Each unit of analysis (user, session, device) is independently assigned to group A or group B with equal probability. Randomization is what guarantees the groups are comparable.
4. Run the experiment for the planned duration. Don't stop it early just because the p-value crosses the threshold. Peeking —checking interim results and deciding based on them— seriously inflates the Type I error rate.
5. Analyze the results. With the complete data, compute the test statistic, the p-value and the confidence interval for the difference. Use the conversion Z-test if you prefer the frequentist approach, or the Bayesian analysis to obtain P(B > A) directly.
6. Make a decision and implement. Interpret the result in business terms: is the effect both statistically significant and practically meaningful? Does the cost of implementing B justify the estimated improvement?

Key statistical concepts in A/B testing

Null and alternative hypotheses

Every A/B test is framed as a hypothesis test. The null hypothesis (H₀) states that there is no difference between the two variants: the conversion rate of A and of B is the same. The alternative hypothesis (H₁) states that a difference does exist.

The goal of the statistical analysis is to assess whether the data are inconsistent enough with H₀ to reject it. Rejecting H₀ doesn't prove H₁ is true; it only indicates that the data are unlikely under the assumption that there's no effect.

Tests can be two-sided (is B different from A, in either direction?) or one-sided (is B better than A?). In practice, a two-sided test is recommended unless there's a very strong reason to rule out one of the directions.

Significance level (α) and p-value

The significance level α is the maximum Type I error probability you're willing to tolerate: the probability of rejecting H₀ when it's actually true (a false positive). The standard value in digital experimentation is α = 0.05, though lower values (0.01 or 0.001) are used in higher-risk contexts.

The p-value is the probability of obtaining, under H₀, a result as extreme as or more extreme than the one observed (in absolute value for a two-sided test). If p < α, H₀ is rejected and the difference is concluded to be statistically significant. It's important not to confuse the p-value with the probability that H₀ is true: they are different things.

A small p-value also doesn't imply the effect is important to the business: with very large samples, tiny differences turn out statistically significant. That's why it should always be accompanied by the confidence interval for the difference and the effect size.

Statistical power and Type II error

Statistical power (1−β) is the probability of detecting a real difference when one exists. In other words, it's the test's ability to avoid a false negative: concluding there's no effect when there actually is one. The standard power in experimentation is 0.80, though many teams use 0.90 to reduce the risk of missing real improvements.

Power depends on the sample size, the true effect size and α. Increasing n always increases power. Lowering α (being stricter about false positives) decreases power if everything else stays the same.

Sample size and minimum detectable effect (MDE)

Sample size is the number of observations needed in each variant for the test to reach the desired power. There's no universal rule: it depends on four parameters that must be fixed before the experiment:

• Baseline rate (p₀): the control's current conversion rate.
• Minimum detectable effect (MDE): the smallest improvement —absolute or relative— that you consider relevant to the business. Smaller effects require larger samples.
• Significance level α: typically 0.05.
• Power (1−β): typically 0.80.

The MDE is perhaps the most underused parameter. Setting it forces you to answer: "what's the minimum improvement that would justify the cost of implementing the change?" An overly ambitious MDE (detecting a 1% relative improvement) drives up the sample size dramatically. A realistic MDE aligns statistics and business from the outset.

Work out the sample size for your test with the MDE, power and sample size calculator.

Frequentist vs. Bayesian A/B testing

There are two major statistical frameworks for analyzing an A/B test. Each answers different questions and has advantages and drawbacks depending on the context.

Frequentist approach

The frequentist approach is the classic standard. It defines H₀, fixes α before the experiment and, at the end, computes the p-value and decides whether to reject H₀. The conclusion is binary: significant or not significant.

Its advantages are controlled interpretation of the Type I error rate and broad methodological consensus. Its main limitation is that the p-value doesn't say how likely it is that B is better than A: it only measures how compatible the data are with H₀.

Use the conversion Z-test for the frequentist analysis.

Bayesian approach

The Bayesian approach starts from a prior distribution over the conversion rates (typically a Beta distribution) and updates it with the observed data to obtain a posterior distribution. The main output is P(B > A): the direct probability that variant B is better than variant A.

This interpretation is more intuitive for product and business teams, since it answers the relevant question directly. The Bayesian analysis also doesn't rely on the frequentist notion of Type I error, though that doesn't mean peeking is harmless: repeatedly making decisions based on a P(B > A) threshold can still produce elevated false-positive rates over the long run. It also requires specifying a prior distribution.

Use the Beta-Binomial Bayesian analyzer for this approach.

Which one should you choose?

There's no single right answer. The frequentist approach is suitable when the decision must meet strict significance criteria (regulated environments, scientific publications). The Bayesian approach is preferable when you want to monitor the test in real time, when the team values the direct interpretability of probabilities, or when solid prior information about the conversion rate is available.

Type I and Type II errors in A/B tests

When making a statistical decision, there are four possible outcomes, two of which are errors:

The peeking problem

Peeking —stopping the test as soon as the p-value crosses the threshold for the first time— is one of the most frequent mistakes in A/B testing. The problem is that the p-value fluctuates throughout the experiment: if you check it repeatedly and stop at the first p < 0.05, the real Type I error rate can exceed 25% even if α is set to 0.05. The solution is to respect the sample size computed in advance, or to use sequential monitoring methods when flexibility is needed (always-valid p-values, SPRT).

Multiple comparisons

If multiple secondary metrics or multiple variants are tested without correction, the probability of getting at least one false positive grows quickly. The Bonferroni correction (dividing α by the number of comparisons) is the most conservative. In A/B tests with many variants, the Benjamini-Hochberg method is preferable, since it controls the false discovery rate (FDR) instead of the family-wise error rate.

When it makes sense to run an A/B test

An A/B test is the right tool when these conditions are met:

• Sufficient volume: the available traffic or sample lets you reach the required sample size within a reasonable timeframe. With too little traffic, tests take months and results get contaminated by seasonal effects.
• Measurable, stable metric: the outcome variable can be measured reliably and its baseline rate doesn't fluctuate abnormally during the experiment.
• Randomization is feasible: each unit can be randomly assigned to a variant without contamination between groups (network effects, spillover).
• The hypothesis is specific: you change a single thing, or a coherent set of changes, not multiple unrelated elements at once, which would make it impossible to attribute the effect to a specific cause.

When volume is insufficient or randomization isn't possible, alternatives exist: before-after tests with a synthetic control, difference-in-differences, regression discontinuity, or Bayesian analysis with informative priors.

Best practices and common mistakes in A/B testing

• Plan before you start. Define the hypothesis, primary metric, MDE, α, power and sample size before launching the experiment. A test without prior planning can almost never be interpreted correctly.
• Don't peek. Don't make decisions until you reach the planned sample size. Use the experiment simulation to see empirically how peeking inflates the Type I error rate.
• Use a single primary metric. Defining multiple primary metrics without correction multiplies the risk of false positives. Secondary metrics are exploratory and shouldn't drive the decision.
• Verify the randomization. Run an A/A test or check that pre-experiment covariates (age, device, acquisition channel) are balanced between groups before drawing conclusions.
• Report the confidence interval. The p-value only says "significant or not." The confidence interval for the difference in rates conveys the magnitude of the effect and its uncertainty, which is what matters for the business decision.
• Consider practical significance. A statistically significant difference may not be relevant to the business if the effect is too small to justify the implementation cost.
• Respect a minimum duration. Even if the sample size is reached earlier, it's advisable to keep the test running for at least one full week to capture day-of-week variability.
• Don't reuse past data. Comparing variant B against historical control data violates the randomization assumption and produces biased comparisons.

Frequently asked questions about A/B testing

How long should an A/B test run?

The minimum duration is calculated by dividing the required sample size by the available daily traffic. Regardless of the outcome, it's recommended to keep the test running for at least one full week to capture the day-of-week effect. In environments with strong weekly seasonality, two full weeks are preferable.

What if the p-value is exactly 0.05?

A p-value of exactly 0.05 doesn't change anything qualitatively compared to 0.049 or 0.051. The threshold is a convention, not an absolute truth. At that boundary, the confidence interval for the difference will touch zero at one of its ends, and the uncertainty about the effect is at its highest. Consider increasing the sample size or turning to Bayesian analysis for a complementary perspective.

Can I run an A/B test with very little traffic?

With little traffic, the test will take a long time to reach the required sample size, and seasonal effects will contaminate the results. In that case, the Bayesian approach with informative priors lets you incorporate prior knowledge about the conversion rate and reach conclusions faster, assuming the extra trade-offs that come with choosing the prior.

Can an A/B test have more than two variants?

Yes: it's called an A/B/n test or a multivariate test. The statistical complexity increases because multiple comparisons need to be controlled. The most common approach is a Bonferroni correction for pairwise comparisons, or an ANOVA followed by post-hoc comparisons. The required sample size also grows with the number of variants.

What is an A/A test?

An A/A test exposes both groups to the same experience. It's used to verify that the randomization system works correctly and that no pre-existing differences exist between groups. If the A/A test frequently concludes there are significant differences, there's a problem with the experiment's implementation.