What a p-value actually means
Few numbers carry as much weight, or as much confusion, as the p-value. It decides which findings get published and which get shelved — and it is routinely asked to mean things it simply cannot. Here is what it really says.
The one-sentence definition
A p-value is the probability of seeing a result at least as extreme as the one you got, if the null hypothesis were true. That last clause is everything. The p-value lives in a hypothetical world where there is no real effect, and it asks: in that world, how surprising is my data? A small p-value means “this data would be unusual if nothing were going on.”
What it does not mean
Almost every misuse of the p-value comes from forgetting the italicised clause above. In particular:
- It is not the probability that the null hypothesis is true. p = 0.03 does not mean “there’s a 3% chance there’s no effect.” The p-value assumes the null is true; it can’t also measure how likely that assumption is.
- It is not the probability your result was due to chance. That’s the same error in a different coat.
- It does not measure the size or importance of an effect. A tiny, meaningless difference can have a tiny p-value if the sample is large enough.
- A non-significant result is not proof of no effect. Absence of evidence isn’t evidence of absence — it often just means the study was too small. See statistical power.
A p-value tells you how surprising your data would be if nothing were happening. It does not tell you whether something is happening.
The 0.05 line is a convention, not a law
The threshold of 0.05 is a historical convention, not a boundary between truth and falsehood. A result at p = 0.049 and one at p = 0.051 are, for all practical purposes, the same result. Treating 0.05 as a cliff — significant on one side, worthless on the other — is how false discoveries and file-drawer bias creep into the literature.
Always pair it with an effect size
Because the p-value says nothing about magnitude, it should never travel alone. Report it beside an effect size and a confidence interval, which tell your reader how big the effect is and how precisely you’ve pinned it down. “Significant” is the beginning of the sentence, not the end.
Report it honestly
Good reporting states the test, the statistic, the exact p-value, the effect size, and the interval — in plain language and in APA form. thericerca does this for every result automatically, and its APA formatter is free to use on its own. Every number it prints is computed, then traced back to the analysis before the report ships — so a p-value is never a figure of speech.
The honest takeaway. A p-value is a useful, narrow instrument: a check on whether your data is surprising under the null. Ask it that one question, pair it with an effect size, and it will serve you well. Ask it to certify truth, and it will mislead you.