IV · LEARN · NOT SERIOUSLY
The Oracle of Significance
Ask the stars what your p-value means. They do not know either.
The Oracle is listening.
The Oracle is a joke. This part is not.
People read p-values the way they read horoscopes: looking for a sign that the thing they hoped for is true. A p-value cannot give that sign. It answers one narrow question: if the change made no difference at all, how often would a test show a difference at least this large?
What it does not tell you:
- The chance your result is a fluke. That depends on how often your ideas work in the first place. The false positive risk calculator works it out.
- The chance the variant is better. That is a Bayesian question, and the A/B test calculator answers it beside the p-value.
- How big the effect is. A tiny p-value can come with a tiny effect. Read the confidence interval, and expect a winning test to overstate its lift.
The line at 0.05 is a convention about a century old. Nothing changes at that number. A result at 0.049 and one at 0.051 are close to the same evidence, which is why the Oracle is hard on anyone who treats the line as a cliff, and harder on anyone who runs a test until it crosses. For the full story, read statistical significance, explained, or look a term up in the glossary.
Sources
- Wasserstein, R. L., & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133.What a p-value is, and the common ways it is misread.
- Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., et al. (2018). Redefine Statistical Significance. Nature Human Behaviour, 2(1), 6–10.Why a p-value just under 0.05 is weak evidence.