Skip to content
TESTING HAVEN

IV · LEARN · NOT SERIOUSLY

The Oracle of Significance

Ask the stars what your p-value means. They do not know either.

The Oracle is listening.

The Oracle is a joke. This part is not.

People read p-values the way they read horoscopes: looking for a sign that the thing they hoped for is true. A p-value cannot give that sign. It answers one narrow question: if the change made no difference at all, how often would a test show a difference at least this large?

What it does not tell you:

  • The chance your result is a fluke. That depends on how often your ideas work in the first place. The false positive risk calculator works it out.
  • The chance the variant is better. That is a Bayesian question, and the A/B test calculator answers it beside the p-value.
  • How big the effect is. A tiny p-value can come with a tiny effect. Read the confidence interval, and expect a winning test to overstate its lift.

The line at 0.05 is a convention about a century old. Nothing changes at that number. A result at 0.049 and one at 0.051 are close to the same evidence, which is why the Oracle is hard on anyone who treats the line as a cliff, and harder on anyone who runs a test until it crosses. For the full story, read statistical significance, explained, or look a term up in the glossary.

  1. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133.What a p-value is, and the common ways it is misread.
  2. Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., et al. (2018). Redefine Statistical Significance. Nature Human Behaviour, 2(1), 6–10.Why a p-value just under 0.05 is weak evidence.