Testing Haven

SRM Checker (Sample Ratio Mismatch)

Is your traffic split broken?

Traffic split

Control

Variant

Enter the expected share and observed visitor count for each group.

What this checker does

The SRM checker is a data-quality gate for your experiments. You tell it the traffic split you configured (50/50, 90/10, three-way — any allocation) and the visitor counts you actually observed, and it runs a chi-square goodness-of-fit test to determine whether the deviation is within the range random assignment produces, or evidence that something is systematically broken. The verdict is plain English: pass, suspicious, or SRM detected.

It is the least glamorous and highest-value 30 seconds in experimentation: an SRM check catches broken tests before you spend credibility shipping decisions based on them.

The formula

With observed counts OiO_i and expected counts EiE_i (your configured ratios scaled to the observed total), the test statistic is:

χ2=∑i=1k(Oi−Ei)2Ei\chi^2 = \sum_{i=1}^{k} \frac{(O_i - E_i)^2}{E_i}

which follows a chi-square distribution with k−1k - 1 degrees of freedom under the null hypothesis that assignment is working. The p-value is the probability of a deviation at least this large occurring by chance. Following standard practice, the checker alerts at p<0.001p < 0.001 and flags p<0.01p < 0.01 as worth a look.

A worked example

You configured a 50/50 test and observed 50,600 control users vs 49,400 variant users — a difference of just 1.2% between arms, easy to shrug off by eye. The expected count is 50,000 per arm, so χ2=600250000+600250000=14.4\chi^2 = \tfrac{600^2}{50000} + \tfrac{600^2}{50000} = 14.4, giving p ≈ 0.00015. That is below 0.001: this is a real SRM, not noise. Something is removing (or double-counting) roughly one in every 160 variant users — and whatever it is almost certainly correlates with behavior, biasing every downstream metric.

When to use it

  • Before reading the results of any completed test — make it a checklist item.
  • Daily during the first days of a new test, when integration bugs surface.
  • After any infrastructure change to assignment, redirects, bot filtering, or analytics.
  • When auditing a surprising win — a too-good-to-be-true lift with SRM is usually the SRM talking. Pair with the significance calculator only after the split passes.

Common mistakes

  • Eyeballing it. A 50.3/49.7 split "looks fine" but is a blatant SRM at 10 million users, while 52/48 is perfectly normal at 1,000 users. Only the test knows; intuition does not scale with n.
  • Checking percentages instead of counts. The test needs raw user counts. Rounded percentages destroy the signal.
  • Running the check on sessions when assignment is per user (or vice versa). Check the unit you actually randomize on.
  • "Fixing" SRM by resampling or reweighting. You cannot reweight away an unknown selection bias. The fix is finding the mechanism, not adjusting the data.
  • Skipping the check on wins. Teams scrutinize losses and wave through gains — precisely backwards, since SRM inflates apparent wins as often as losses. Read the full story in our SRM guide.

Frequently Asked Questions

What is a sample ratio mismatch (SRM)?

SRM is when the actual traffic split of your experiment differs from the configured split by more than randomness can explain — for example, a 50/50 test that logs 50,600 vs 49,400 users. It signals that assignment, redirects, or logging are biased, which means the two groups are no longer comparable and every metric in the test is suspect.

Why is the alert threshold p < 0.001 instead of 0.05?

Because SRM checks run constantly across many experiments, and a 5% false-alarm rate would bury teams in phantom incidents. The industry convention (from Microsoft's experimentation platform and the SRM literature) is p < 0.001: rare enough that alerts are almost always real problems, while genuine SRM bugs typically produce p-values far smaller — often 10⁻⁸ or beyond.

My test has SRM. Can I still use the results?

No — and this is the uncomfortable part. SRM means users were systematically, not randomly, sorted into groups, so the difference you measured mixes the treatment effect with the selection bias, in unknown proportions. Even a 0.5% traffic imbalance can produce metric deltas larger than typical treatment effects. Find the cause, fix it, and rerun.

What usually causes SRM?

The most common culprits: variant-specific redirects that lose slow connections; bot filtering that removes traffic unevenly (a variant that changes page structure can change bot classification); the treatment crashing or slowing for a subset of users so their telemetry never arrives; targeting or triggering conditions evaluated after assignment; and data pipeline bugs like deduplication rules interacting with variant behavior.

How often should I check for SRM?

At minimum once before you read any results, on the final data. Better: check daily while the test runs, since SRM caught on day 2 saves weeks of wasted runtime. Best: automate it in your experimentation pipeline so every test gets checked continuously. This page's URL encodes your inputs, so you can bookmark a pre-filled check for a running test.