• Understand the basic concept and meaning of the p-value
Deeper study
• Compute and interpret p-values in a variety of tests
Real-world practice
• Recognize the limitations of the p-value and use complementary methods
At a glance
A p-value is the probability, assuming the null hypothesis is true, of obtaining a test statistic at least as extreme as the one observed; it ranges from 0 to 1, and smaller values are interpreted as stronger evidence against the null hypothesis.
Statistical significance is usually judged against a 0.05 threshold; in a two-sided test the p-value is p = 2P(|T| ≥ |t_obs|)
A p-value is P(data|H₀), not P(H₀|data), so it must not be read as the probability that a hypothesis is true
A small p-value does not mean a large effect (sample size drives it too); interpret it together with effect sizes and confidence intervals, and apply corrections such as Bonferroni when testing multiple hypotheses
The p-value is a core component of hypothesis testing
Basic: What Is a p-Value?
Difficulty 3/5
About 20 min
A p-value is the probability, assuming the null hypothesis is true, of seeing a result at least as extreme as the one you observed. It is used as a yardstick for statistical significance.
Think of a coin toss. Someone claims the coin is fair, but you toss it 10 times and get 9 heads. The probability that a result this extreme happens by pure chance is the p-value. If that probability is very small, you have reason to suspect the coin is not fair after all.
Key points
•
A conditional probability computed under the assumption that the null hypothesis is true
•
A value between 0 and 1
•
The smaller it is, the stronger the evidence against the null hypothesis
•
Usually judged against a 0.05 threshold
A simple example
A p-value of 0.03 means that, if the null hypothesis were true, a result this extreme would show up only 3% of the time.
Check your understanding
Answer:
If the null hypothesis were true, the observed result would occur only 0.1% of the time, so this is very strong evidence against the null hypothesis.
A p-value is computed as the area of the test statistic's distribution that lies beyond the observed value, that is, the region more extreme than what you observed. How you compute it depends on whether the test is two-sided or one-sided.
Mathematical formula
Two-sided test: Loading...
One-sided test: Loading... or Loading...
T is the distribution of the test statistic, and t_obs is the observed value of the test statistic.
Learn through examples
Example 1: p-Value in a One-Sided Test
Testing whether a new treatment is more effective than the existing one
H₀: μ ≤ 50, H₁: μ > 50
t = 2.1 (24 degrees of freedom)
P(T > 2.1) = 0.023
Answer:
Since p-value = 0.023 < 0.05, we conclude the new treatment is more effective.
p = 0.023 does not mean "there is a 97.7% chance the treatment works." It means that, under the assumption μ ≤ 50 is true, a result of t = 2.1 or larger occurs 2.3% of the time. A two-sided test on the same data would give p = 0.046, twice as large. The direction of the test must be fixed before you look at the data; switching to a one-sided test after seeing the result inflates the true Type I error rate.
Example 2: p-Value in a Two-Sided Test
Testing for a difference in means between two groups
H₀: μ₁ = μ₂, H₁: μ₁ ≠ μ₂
t = -2.5 (30 degrees of freedom)
P(|T| > 2.5) = 2 × P(T > 2.5) = 2 × 0.009 = 0.018
Answer:
Since p-value = 0.018 < 0.05, there is a significant difference between the two groups.
Because the test is two-sided, the one-tail probability 0.009 is doubled to give p = 0.018. The sign of t = −2.5 suggests the first group's mean is smaller, but the p-value itself carries no information about direction. And p = 0.018 does not mean "there is a 1.8% chance the groups are the same": a p-value is P(data|H₀), not P(H₀|data).
Common mistakes
Mistake:
Reading the p-value as the probability that a hypothesis is true
Why is it wrong?
A p-value is the probability of a result at least as extreme as the one observed, given that the null hypothesis is true
Correct approach:
A p-value is P(data|H₀), not P(H₀|data)
Mistake:
Mistaking the size of the p-value for the size of the effect
Why is it wrong?
A small p-value does not mean a large effect; it is also driven by sample size
Correct approach:
Compute the effect size separately and interpret it alongside the p-value
Mistake:
Mixing up two-sided and one-sided tests
Why is it wrong?
The testing procedure, and therefore the p-value calculation, depends on whether the research hypothesis is directional
Correct approach:
State the direction of your hypothesis clearly and choose the matching test
Case study: Misuse of p-Values in Medical Research
Background
An analysis of the problems caused by relying on p-values during drug development
The problem
We need to analyze the errors that arise when decisions rest on p-values alone in a multiple-testing setting.
The data
Among 20 tests for side effects, one comes out significant at p = 0.04
Method
Multiple-testing correction with the Bonferroni method and analysis of the error rate
Solution and results
After the multiple-testing correction, p = 0.04 × 20 = 0.8 > 0.05, so the result is not significant
It is most likely a chance finding
The result was significant on the raw p-value but not significant after correcting for multiple testing
Conclusion:
Rather than relying on a simple p-value threshold, weigh clinical relevance, effect sizes, and confidence intervals together.
References
Fisher, R. A. (1925), "Statistical Methods for Research Workers", Oliver and Boyd, Edinburgh — the origin of significance testing with p-values and of the P = 0.05 convention
Wasserstein, R. L. & Lazar, N. A. (2016), "The ASA's Statement on p-Values: Context, Process, and Purpose", The American Statistician 70(2), 129–133 (DOI 10.1080/00031305.2016.1154108) — the American Statistical Association's official statement on the definition of the p-value and its common misinterpretations (it is not the probability a hypothesis is true, nor an effect size)
Ioannidis, J. P. A. (2005), "Why Most Published Research Findings Are False", PLoS Medicine 2(8), e124 (DOI 10.1371/journal.pmed.0020124) — the foundational metascience paper on the replication crisis brought on by over-reliance on p-values
Dunn, O. J. (1961), "Multiple Comparisons Among Means", Journal of the American Statistical Association 56(293), 52–64 (DOI 10.1080/01621459.1961.10482090) — the basis for the Bonferroni multiple-testing correction used in the case study on this page
