IV · LEARN
Real or Noise?
Ten winning tests. Some of them are lying.
The rules of this world
Only 1 idea in 10 really works. The ones that work lift conversion by 3% to 10%; the rest do nothing. You will see ten tests, and every one of them came out statistically significant.
Dealing ten tests…
What the game is about
Every test in the game passed the usual bar: statistically significant at 95% confidence. It is tempting to read that as "95% likely to be real". It is not. In a world where only 1 idea in 10 works, there are so many ideas that do nothing that their occasional lucky results end up sitting beside the real wins. Typically about four in ten of the wins you are shown are noise.
The share of wins that are noise has a name, the false positive risk, and the false positive risk calculator works it out for your own testing program.
How to tell them apart
- Look at how far below 0.05 the p-value is. Noise that reaches significance usually only just makes it. A p-value of 0.04 is weak evidence; 0.001 is strong.
- Be suspicious of large lifts from small tests. Real ideas here lift conversion by a few percent. A measured lift of 30% from a couple of thousand visitors is far more likely to be luck than a breakthrough.
- Trust big samples more. A large test can detect a small real lift, so its wins are more often real.
After each guess the game shows the odds that the win was real, worked out from exactly the numbers you could see and the rules of the world. Nobody can be right every time: the best possible strategy is to go with those odds.
Even the real wins mislead
Notice how often a real win measures far more than its true lift. A test is only shown to you if it reached significance, and a small real lift only reaches significance when luck pushes it upward. That is the winner's curse, and the winner's curse calculator estimates how much of a measured lift to expect to keep.
How the tests are made
Each set is simulated. An idea is drawn (1 in 10 works), then a baseline conversion rate and a sample size, and a test is run on made-up visitors. Tests that do not reach significance in the variant's favour are thrown away, exactly as they would be forgotten in a real program, until ten wins are left. A set number always gives the same ten tests, so you can share a set with someone else. Today's set is the same for everyone.
Sources
- Kohavi, R., Deng, A., & Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22).How often significant results are false when few ideas work, and reported success rates.
- Ioannidis, J. P. A. (2005). Why Most Published Research Findings Are False. PLoS Medicine, 2(8), e124.The original argument that the share of true findings depends on how many tested ideas are true, and on power.
- Gelman, A., & Carlin, J. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9(6), 641–651.Why results that reach significance overstate the true effect, especially from small tests.