The Search for Significance: A
Practical Guide to P-Hacking
@Neuro_Skeptic
neuroskeptic@googlemail.com
http://blogs.discovermagazine.com/neuroskeptic
The Nine Circles of Dante’s Inferno
First Circle: Limbo
Second Circle: Lust
Third Circle: Gluttony
Fourth Circle: Greed
Fifth Circle: Anger
Sixth Circle: Heresy
Seventh Circle: Violence
Eighth Circle: Fraud
Ninth Circle: Treachery
The Nine Circles of Scientific Hell
First Circle: Limbo
Second Circle: Overselling
Third Circle: Post-Hoc Storytelling
Fourth Circle: P-Value Fishing
Fifth Circle: Creative Use of Outliers
Sixth Circle: Plagiarism
Seventh Circle: Non-Publication of Data
Eighth Circle: Partial Publication of Data
Ninth Circle: Inventing Data
P-Fishing
Fourth Circle: P-Value Fishing
“Those who tried every statistical test in the book
until they got a p value less than 0.05 find
themselves here, an enormous lake of murky water.
Sinners sit on boats and must fish for their food.
Fortunately, they have a huge selection of different
fishing rods and nets. Unfortunately, only one in 20
fish are edible, so they are constantly hungry.”
P-Fishing
• Also known as…
▫
▫
▫
▫
▫
▫
▫
P-Hacking
Questionable Research Practices (QRPs))
Torturing the data
Outcome reporting bias
Undisclosed flexibility
Researcher Degrees of Freedom
…and more.
Related to…
Publication bias
P-hacking is about using multiple methods or
attempts to find a significant result.
Publication bias is the tendency to only publish
significant results.
Both go hand in hand in practice. But each could,
in theory, occur without the other.
P-Hacking Works!
• Collect some data
P-Hacking Works!
• Collect some data
• Try many statistical tests on the same data
P-Hacking Works!
• Collect some data
• Try many statistical tests on the same data
• Or try many variants of the same data (e.g.
removing ‘outliers’.)
P-Hacking Works!
• Collect some data
• Try many statistical tests on the same data
• Or try many variants of the same data (e.g.
removing ‘outliers’.)
• Or try looking at different variables within the
dataset
P-Hacking Works!
• Collect some data
• Try many statistical tests on the same data
• Or try many variants of the same data (e.g.
removing ‘outliers’.)
• Or try looking at different variables within the
dataset
• Report the analyses that give the most favourable
results (usually the lowest p-values).
“P-Hack the numbers, HARK the text”
• Hypothesizing After the Results Are Known
“P-Hack the numbers, HARK the text”
• Hypothesizing After the Results Are Known
• Allows any significant result to become an
interesting, hypothesis-confirming finding
“P-Hack the numbers, HARK the text”
• Hypothesizing After the Results Are Known
• Allows any significant result to become an
interesting, hypothesis-confirming finding
• HARKing is not to be confused with revising or
rejecting hypotheses in the light of new data –
which is essential (!)
“P-Hack the numbers, HARK the text”
• Hypothesizing After the Results Are Known
• Allows any significant result to become an
interesting, hypothesis-confirming finding
• HARKing is not to be confused with revising or
rejecting hypotheses in the light of new data –
which is essential (!)
• Rather, HARKing means that hypotheses are
never tested. The hypotheses are always “one
step ahead” of the data.
And now a demonstration…
fMRI Simulator
Why P-Hacking Is So Effective
• There are many choices (‘researcher degrees of
freedom’) in data analysis.
• For example, in a simple task-based fMRI data
analysis, Joshua Carp found 7000 combinations of
parameters (very conservative).
Carp, J. (2012).On the plurality of
(methodological) worlds: estimating the
analytic flexibility of fMRI experiments
Frontiers in Neuroscience
How To Spot It
• The p-curve…
Try it now!
http://www.p-curve.com/app2/
Simonsohn, U. Nelson, L. D.
Simmons, J. P. (2013). P-curve: a
key to the file-drawer.
Journal of Exp. Psychol General
Although it’s complicated
“Publication bias and underpowered studies
might be a bigger problem for science than
inflated Type 1 error rates…”
The Root of the Problem
The Root of the Problem (and Fixes)
Smulders YM (2013). A two-step
manuscript submission process
can reduce publication bias.
Journal of Clinical Epidemiology
The Root of the Problem (and Fixes)
Smulders YM (2013). A two-step
manuscript submission process
can reduce publication bias.
Journal of Clinical Epidemiology
Chambers CD (2013). Registered
Reports: a new publishing
initiative at Cortex
Cortex
Happy hacking!
@Neuro_Skeptic
neuroskeptic@googlemail.com
http://blogs.discovermagazine.com/neuroskeptic