Methods for Creating Reliable
and Reproducible Analyses
Jennifer Blackford, PhD
Assistant Professor of Psychiatry
VKC Stats & Methodology Core Training Coordinator
Goals
• Introduce a new idea
• Persuade you that reproducible analyses
are necessary
• Convince you that achieving reproducible
analyses is possible
• Provide the necessary tools for you to
begin creating reliable and reproducible
analyses
Outline
1.
2.
3.
4.
5.
6.
Reliability in research methods
Reliability in analyses
Dangers of point and click software
What are “reproducible results”?
Barriers to reproducible results
Methods for creating reproducible
analyses
7. Practical implementation
Reliability in
Research Methods
What is reliability?
re·li·a·ble [ri-lahy-uh-buh l]
adjective
that may be relied on; dependable in
achievement, accuracy, honesty, etc.: reliable
information.
From Random House Dictionary
How do we ensure reliability of our
research methods?
Examples?
How reliable is “reliable enough”?
a)
b)
c)
d)
70%
80%
90%
100%
What if the method is not reliable?
a)
b)
c)
d)
use it anyway
find a different method
revise method to increase reliability
develop a new method
Reliability in
(Statistical) Analyses
Scientific Method Steps
1.
2.
3.
4.
5.
Generate hypothesis
Perform experiment and collect data
Analyze data
Interpret data and draw conclusions
Publish results
What does “analysis” include?
Statistical Analysis
• Data entry
• Data cleaning
– Data entry errors
– Missing data
– Recoding
• Data preparation
– Checking data
distributions
– Handling outliers
– Transforming variables
– Creating new variables
• Performing statistical
analyses
• Recording results
• Creating graphs
Neuroimaging Analysis
• Image file pre-processing
–
–
–
–
Convert image files to standard format
Correct head motion within and across trials
Align structural and functional image
Transform images into a “standard” space
• Perform statistical analyses
– Input experimental design
– Assign event-timing information to image files
– Run statistical analyses
• Record Results
• Create Graphs
Should we expect the same
reliability (or reproducibility) from
our analytic methods as we do from
our other research methods?
a) No, analyses aren’t really methods
b) No, analytic methods are too complex
c) Maybe, but everything is fine, no need to
change methods
d) Yes, all parts of the scientific process
should be reliable
What are the “common” practices?
• Enter data in Excel
• Use Excel for data cleaning, etc
• Generate descriptive statistics (counts,
means, etc) in Excel
• Import data into SPSS for analysis
• Use point-and-click options to run
statistical analyses
• Type data back into Excel to create graphs
Now let’s evaluate reliability….
• Enter data in Excel
• Use Excel for data cleaning, etc
• Generate descriptive statistics (counts,
means, etc) in Excel
• Import data into SPSS for analysis
• Use point-and-click options to run
statistical analyses
• Type data back into Excel to create graphs
The Dangers of
Point-and-Click Software
Excel
•
•
•
•
Data entry inconsistencies and errors
No record of what steps were performed
Sorting dangers
Data export issues
– There is a bug in how Windows processing
Excel files which results in missing data being
exported.
Cryer (2001). Problems using Microsoft Excel for Statistics.
“Spreadsheet from Hell”
Dan Byrne
Excel—Statistical Issues
• Missing data are handled inconsistently
and sometimes incorrectly
• Excel uses poor algorithms to find the
standard deviation
• Excel does not treat tied observations
correctly when ranking
• Regression is especially problematic-computations are often erroneous due to
multiple incorrect procedures
Cryer (2001). Problems using Microsoft Excel for Statistics.
Other Point-and-Click
• There are also dangers with other
programs that offer user-friendly pointand-click capabilities, such as SPSS and
SAS.
• Neuroimaging analysis programs, like
Brain Voyager and SPM have the same
issues too.
Has this ever happened
to you…..
Vignette #1
You sit down to finish writing your
manuscript. You realize that you need to
clarify one result by running an additional
analysis. First, you re-run the primary
analysis……but the primary results don’t
match what you have in your paper.
Vignette #2
You get an email request from the graduate
student of a colleague at another university.
She has conducted an experiment similar to
one you published five years ago. She has
a question and would like for you to run an
analysis on a subset of subjects. When you
go to the data folder, you find thirty program
files and can’t remember which one has the
analysis. Where is that final dataset????
Would you like to increase the
reliability of your analytic methods?
a) No need, my current methods are
sufficient. I’m finding significant results
already.
b) Maybe, but I think it will be hard and I
don’t have the energy for something
new.
c) Yes! Show me how, I’ll do anything.
What are
“reproducible results”?
Reproducible Results
• Some consider the use of statistical
methods that can not be replicated as a
breach of research ethics.
• There is a new movement in statistics
called “reproducible results” or “literate
statistical programming”.
Claerbout’s Principle
“An article about computational science in a
scientific publication is not the scholarship itself, it
is merely advertising of the scholarship. The actual
scholarship is the complete software development
environment and the complete set of instructions
which generated the figures.”
Note: The same “Principle” obviously applies to
tables, standard errors, and so on.
deLeeuw, J (2001). Reproducible Results: The Botton Line.
Literate Statistical Practice
Literate Statistical Practice (LSP, Rossini,
2001) describes an approach for creating
self-documenting statistical results. It applies
literate programming (Knuth, 1992) and
related techniques in a natural fashion to the
practice of statistics. In particular,
documentation, specification, and
descriptions of results are written
concurrently with writing and evaluation of
statistical programs.
Rossini & Leisch (2003). Literate Statistical Practice.
Barriers to
Reproducible Results
Barriers
What are some barriers to “reproducible results”?
1. Awareness
We didn’t even know there was a problem.
2. Knowledge
What are the right methods?
3. Change
You mean I wasn’t doing the right thing?
4. Effort
It’s too hard. I don’t have time. I don’t have the
energy to learn something new.
Benefits
1. Reliable Results
The results you produce today will be the
same results you will produce tomorrow.
2. Time Savings/Efficiency
In the (not so) long run, you will save much
time.
3. Flexibility
When you don’t “point-and-click” you gain
many new analytic options.
4. Institutional (Lab) Memory
Baby Steps
We’ve already addressed the first barrier
Awareness
Now, we’ll address the second barrier
Knowledge
Methods for Creating
Reproducible Analyses
Caveats
1. This will take discipline.
– On the front end, it does take more time to
create a system or structure for your
analysis. It is much faster to use Excel to do
one quick analysis.
2. You might have to learn something new.
– This can be a major barrier. Learning takes
a commitment of time and effort.
Overarching Principle: Organization
• Organization for everything is key.
• This principle is as true for your research
study as for your data analysis.
– electronic folder structure
– file naming conventions
– analytic programs
• Things can get out of control.*
Data Entry
• Enter your data into a stable structure.
• Good examples of stable structures
– a relational database (like Microsoft Access)
• VKC provides database development
• VUMC REDCap (Research Electronic Data
Capture) https://www.mc.vanderbilt.edu/starbrite/
– SPSS or SAS tables
Data Cleaning and Preparation
The second step is to create methods or
procedures for documenting every step
How-To Guides
http://www.icpsr.umich.edu/access/dataprep.pdf
How-To Guides
By Fred Davidson, 1996
What About Data Analysis?
Now for the scary word….
programming
Which statement best
describes you?
a) I have never programmed before.
b) I have programmed a little….10 years
ago.
c) I have some basic programming skills
(and feel comfortable learning more)
d) I am a code junkie!
A code example
Final Product
• The final product is a single file that includes all
of the code necessary to reproduce all of the
analyses.
• The code file should be heavily commented so
that another reader can follow the workflow.
Note: Ultimately, the file will also print out text and
graphics.
Specifically…
• Data import
• Data cleaning/formatting/manipulation
– You must have an audit trail of every change made to
the data.
• Descriptive statistics
• Inferential/predictive statistics
– Include all of the workflow steps, including analyses
that didn’t work or that you don’t keep so that you’ll
remember what you did later
Practical Implementation
An illustration using SPSS
SPSS
• Starting simply, SPSS provides a
relatively easy way to create a document
that keeps track of all of your statistical
steps.
• You can begin with point-and-click, saving
the underlying code, while progressing to
programming.
• SPSS journaling (edit->options->session
journal, record syntax in journal) can track
your syntax selections
SPSS Example
Protect Yourself
• Track your changes
– version control software (like Subversion) tracks of
each iteration of document or program
• RCS software: http://www.componentsoftware.com/csrcs/
– Screen capture software can capture your mouse
movements: SnagIt (www.techsmith.com)
• Keep prior operating systems
– With Microsoft Virtual PC you can have other
operating systems on your computer
• http://www.microsoft.com/windows/products/winfamily/virtual
pc/default.mspx
Jim Steiger
Protect Yourself
• Back up, back up, back up
– Disk space is cheap, your time is not
– Amazon S3 “storage for the internet”
– Jungle Disk http://www.jungledisk.com/
• Acronis True Image Home
– www.acronis.com
– Note: first, go to Computer Management and make
certain that the Volume Shadow Copy service starts
automatically before you install the software
– make several copies of the Emergency Boot Disk,
because you can boot into a Windows system that will
not load windows and restore the whole hard drive (or
just the system settings) from OUTSIDE WINDOWS.
Jim Steiger
Future Steps
• SAS Enterprise Guide
• With programs like R and Sweave/LaTex,
you can incorporate the entire text of your
report with results and graphics
• Upcoming Seminar
– Terri Scott – May 16, 2008.