What this calculator does
This Bayesian A/B test calculator treats each variant's true conversion rate as an uncertain quantity and updates that uncertainty with your data. Instead of a p-value, you get answers to the questions stakeholders actually ask: What is the probability the variant is better? How big is the lift likely to be? How much do we risk if we ship it and we are wrong? It reports probability to be best, probability of beating control, expected loss against a configurable risk threshold, a credible interval on relative lift, and a chart of the full posterior distributions.
The same tool also runs the frequentist analysis of your data side by side — p-values, confidence intervals, and lift — plus an automatic sample ratio mismatch check. Comparing the two frameworks on identical data is the fastest way to understand what each one is (and is not) telling you.
The model
Conversions are modeled as draws from a Binomial distribution, with a Beta prior on the conversion rate. The Beta is the conjugate prior for the Binomial, so the posterior has a closed form: starting from a prior and observing conversions in visitors, the posterior is
The probability that variant B beats control A, and the expected loss of shipping B, are integrals over the two posteriors:
where and are each posterior's CDF and density. The calculator evaluates them by numerical integration, and does the same for probability-to-be-best across several variants and for the credible interval on lift. Nothing is simulated, so results are exact to many decimal places, identical every time you load the page, and unaffected by non-integer priors or very large samples.
A worked example
Control: 5,000 visitors, 500 conversions (10.0%). Variant B: 5,000 visitors, 560 conversions (11.2%). With the uniform prior, the posterior for control is Beta(501, 4501) and for B is Beta(561, 4441). The calculator reports a 97.4% probability that B beats control, a median lift of +12.0%, and an expected loss of 0.006% of conversion rate — 0.06% of the baseline. If B is secretly worse, you stand to lose almost nothing.
The frequentist reading of the same data is p = 0.051, just short of significance at 95%, and the Bayesian probability is likewise just short of the 97.5% bar that matches a two-tailed test. The two frameworks agree that the evidence is promising but not yet decisive; what the Bayesian reading adds is the expected loss, which tells you how little is at stake if you ship anyway.
When to use Bayesian over frequentist
- When you need a decision, not a publication: expected loss gives you a quantified risk to weigh against the cost of keeping the test running.
- When sample sizes are small and you have solid priors from previous tests on the same funnel.
- When communicating with non-statisticians: "93% chance B is better" lands better than "p = 0.14, fail to reject the null".
- Read the deeper comparison in our Bayesian vs frequentist guide.
Common mistakes
- Treating 95% probability as a magic gate. Combine it with expected loss — a coin-flip probability with negligible downside can still justify shipping.
- Overconfident priors. An informative prior based on wishful thinking rather than data will drag your posterior toward fiction. Beta(1, 1) is the honest default.
- Skipping data-quality checks. Bayesian math does not rescue broken randomization — this calculator runs a sample ratio mismatch check on your visitor counts automatically, and the standalone SRM checker remains available for checking splits on their own.
- Aggressive optional stopping. Checking constantly and stopping at the first 95% crossing still overstates your win rate; prefer expected-loss decision rules or planned durations.