Manuscript red-team reference · companion to Paper Review

p-hacking

Q: What is p-hacking?

p-hacking is the practice of running many different analyses on the same data (different subgroups, different outcomes, different cut points, different covariates) and reporting only the ones that came back statistically significant. The result looks like a clean finding; the underlying truth is that with enough freedom, randomly noisy data will produce 'significant' results.

Q: Why does a reviewer care about p-hacking?

Reviewers look for tells: p-values that cluster suspiciously close to 0.05; long lists of covariates with one or two starred; cell-by-cell comparisons in heat-maps; oddly specific subgroup analyses presented as primary findings. A pre-registration that names the primary analysis blocks p-hacking. Its absence is itself a flag.

Q: How do I fix p-hacking?

Pre-register your primary analysis and report it as primary, then label additional analyses as exploratory. Apply a multiple-comparisons correction. Show the full set of analyses you ran, not just the winners. A 'multiverse analysis' (Steegen et al., 2016) systematically reports the effect under many reasonable analytical choices and lets the reviewer see how robust your conclusion is.

Selectively reporting whichever analysis returned p < 0.05.

What it is

p-hacking is the practice of running many different analyses on the same data (different subgroups, different outcomes, different cut points, different covariates) and reporting only the ones that came back statistically significant. The result looks like a clean finding; the underlying truth is that with enough freedom, randomly noisy data will produce 'significant' results.

Why a reviewer cares

Reviewers look for tells: p-values that cluster suspiciously close to 0.05; long lists of covariates with one or two starred; cell-by-cell comparisons in heat-maps; oddly specific subgroup analyses presented as primary findings. A pre-registration that names the primary analysis blocks p-hacking. Its absence is itself a flag.

How to fix it

Pre-register your primary analysis and report it as primary, then label additional analyses as exploratory. Apply a multiple-comparisons correction. Show the full set of analyses you ran, not just the winners. A 'multiverse analysis' (Steegen et al., 2016) systematically reports the effect under many reasonable analytical choices and lets the reviewer see how robust your conclusion is.

This is one of ~15 canonical methodology explainers Paper Review's red-team report links to. To get a full review of your manuscript, start a Paper Review ($9).