Skip to content
TESTING HAVEN

IV · LEARN

Bayesian vs Frequentist A/B Testing: Which Should You Use?

By Testing Haven · Published · Updated

Bayesian versus frequentist is the closest thing A/B testing has to a holy war, and most of the shots fired miss the practical point: they are different questions, not different answers. This guide lays out what each framework actually computes and when each one earns its keep.

Two questions about the same data

The frequentist asks: if nothing were different, how surprising is this data? The answer is a p-value, plus a confidence interval built from a procedure guaranteed to cover the truth 95% of the time. The Bayesian asks: given this data (and my prior), what do I now believe about the true rates? The answers are direct probabilities — "there is a 96% chance B beats A" — plus credible intervals and, crucially, expected loss: how much conversion you sacrifice on average if you ship the variant and turn out wrong.

The intervals differ in the same way. A 95% credible interval says there is a 95% probability the true lift lies inside it, given the model and the prior — the reading most people wrongly give to confidence intervals. A 95% confidence interval describes the procedure instead: 95% of intervals built this way, across many experiments, contain the truth.

One dataset, two readings

Control: 5,000 visitors, 500 conversions (10.0%). Variant: 5,000 visitors, 560 conversions (11.2%).

The frequentist reading. z = 1.95 and the two-tailed p-value is 0.051. At 95% confidence that is not significant, narrowly. The 95% confidence interval on the relative lift runs from −0.8% to +24.8%.

The Bayesian reading, with the uniform Beta(1, 1) prior. The probability that the variant beats control is 97.4%. The median lift is +12.0%, with a 95% credible interval of −0.1% to +25.5%. The expected loss from shipping the variant is 0.006 percentage points of conversion rate, which is 0.06% of the baseline.

These are the same evidence in two vocabularies. With a flat prior, the probability that the variant beats control is almost exactly one minus the one-sided p-value — here 1 − 0.0256 = 97.4%. A 95% probability bar is therefore a one-tailed test at α = 0.05, a lower bar than the two-tailed test most teams run. Our combined calculator matches the bars: a two-tailed test at 95% confidence is compared against a 97.5% probability of beating control, a one-tailed test against 95%. On this data both readings then say the same thing: close, not there yet.

What the Bayesian reading adds is the expected loss. Even if you shipped now and were wrong, you would give up very little — well under the calculator's default risk threshold of 0.1% of the baseline rate. That is information a p-value cannot give you.

Where frequentist wins

P-values are the lingua franca of experimentation programs: auditable, convention-backed, and free of arguments about priors. The guarantee is specific: among tests where the variant truly changes nothing, at most 5% are declared significant. That holds across hundreds of tests regardless of what anyone believed beforehand. If your organization runs a large testing program with centralized review, frequentist defaults plus pre-committed sample sizes (see sizing) are hard to beat.

It is not a guarantee that 95% of your shipped winners are real. That share — the false discovery rate — depends on how often your ideas work and on power. Run 1,000 tests in which 10% of ideas truly help, at 80% power and α = 0.05. You detect 80 real winners, and 45 of the 900 duds come up significant by chance: 36% of all significant results are false. About half of those false alarms point in the losing direction and would never ship, which still leaves roughly 22% of shipped "winners" with nothing behind them. Drop power to 30% and 60% of significant results are false. Well-powered tests protect your win list as much as α does — the power calculator shows where a test stands.

Where Bayesian wins

Decisions, especially small-sample ones. "p = 0.11, fail to reject" leaves a PM nowhere; "94% chance it's better, and shipping it risks about 0.02 points of conversion on average" is a decision brief — and those are the same data (3,000 visitors per arm, 300 against 338 conversions). Expected loss lets you ship promising variants early when the downside is small, a decision-theoretic move frequentist tests simply do not offer.

Informative priors also let repeat experimenters stabilize noisy early data. Say 200 visitors and 30 conversions arrive on day one, a raw 15%. With the flat prior the posterior mean is 15.3%. With a Beta(100, 900) prior — the equivalent of 1,000 earlier visitors converting at 10% — it is 10.8%, which is a far better guess about a funnel that has converted at 10% for a year. The prior fades as data arrives; a badly chosen one biases everything until it does.

Try it on the Bayesian calculator, which reports probability-to-be-best, probability of beating control, expected loss, and a credible interval on lift. All four are computed by numerical integration of the Beta posteriors rather than by simulation, so the same inputs always return the same numbers. With three or more groups, watch probability-to-be-best rather than probability of beating control: two strong variants can each be near-certain to beat control while neither is clearly the best of the pair.

The myths

"Bayesian tests let you peek freely." Mostly overstated. In an A/A test checked ten times, stopping as soon as either arm reaches a 95% probability of being better declares a winner about 34% of the time. With a 97.5% bar it is about 19% — the same inflation as peeking at a two-tailed p-value. The posterior stays coherent; your win rate does not. If you want statistically safe peeking, that is precisely what sequential testing is for (and the peeking guide explains why).

"Priors are cheating." The uniform Beta(1, 1) prior adds almost nothing to the data — it is worth two visitors. Informative priors are only as biased as the data behind them.

"They disagree all the time." With flat priors and matched bars, the two readings agree almost by construction. Real disagreement comes from an informative prior, or from an expected loss that tells a different story from the probability.

A practical default

Run both — they are free. Use the significance calculator for the record and the program-level error guarantees, and the Bayesian view for the shipping decision and the stakeholder conversation. When they agree (usually), decide with confidence. When they disagree, find out why before acting: an informative prior pulling the posterior, or an expected loss still above your risk threshold. And when neither reading is decisive, the test is probably underpowered — which the power calculator will confirm.

Common mistakes

  • Comparing a 95% probability bar with a two-tailed p < 0.05 and calling the gap a philosophical disagreement. It is a one-tailed bar against a two-tailed one.
  • Reading 1 − p as the probability the variant is better. With a flat prior the right approximation uses the one-sided p-value.
  • Treating α as the share of shipped winners that are false.
  • Stopping the moment the probability crosses the bar, then quoting that probability as if the test had been run once.
  • Choosing an informative prior from optimism rather than from past data.
  • Skipping data-quality checks. No framework rescues a broken split — run the SRM checker first.

New to p-values and confidence intervals? Start with statistical significance, explained. For the Beta distribution behind the Bayesian model, see probability distributions.

  1. Casella, G., & Berger, R. L. (1987). Reconciling Bayesian and Frequentist Evidence in the One-Sided Testing Problem. Journal of the American Statistical Association, 82(397), 106–111.Why a one-sided p-value and a Bayesian probability line up.
  2. Stucchio, C. (2015). Bayesian A/B Testing at VWO. Visual Website Optimizer technical whitepaper.Expected loss as a Bayesian decision rule.
  3. Kohavi, R., Deng, A., & Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22).Why the share of false winners depends on how many ideas work.
  4. Robinson, D. (2015). Is Bayesian A/B Testing Immune to Peeking? Not Exactly. Variance Explained.Why Bayesian tests are not immune to peeking.
  5. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133.What a p-value does and does not tell you.