Rigor, Reproducibility and
Transparency
Oh my!
Sandra Taylor, Ph.D.
Clinical and Translational Science Center
University of California, Davis
11 January 2016
Scientific Rigor Criterion
Application of the scientific method to
ensure robust and unbiased results
– Experimental design
– Methodology
– Analysis
– Interpretation and reporting of results
Includes full transparency to allow
reproducibility and extensions
“Robust and Unbiased”
Robust results are obtained using
methods that
– Avoid bias
– Can be reproduced under well-controlled and
reported experimental conditions
“Robust” and “Unbiased” results are
goals, not absolute standards and may
vary across scientific disciplines.
“Rigor” standard operates over
entire project life
 Experimental Design/Study Protocol
 Data collection
 Data analysis
 Manuscript preparation
Studies need to be developed and
proposals written with rigor standards
in mind.
Rigor and reproducibility start
with the experimental design
 Specify inclusion/exclusion criteria
 Define and justify controls
 Identify confounders and biases
– Integrate measures to avoid/reduce
confounders/biases
 Include biological and technical replicates
 Include multiple reviewers
 Define key terms, conditions, requirements
up front
Key components for
demonstrating “rigor”
 Sample size/power analysis
 Randomization
 Blinding
 Data handling plan
 Statistical analysis plan
Sample size/power analysis
Ensure study has adequate power to
detect meaningful and realistic differences
– Conduct while designing study
– Be study specific and use correct procedure
– Specify procedure (e.g., two-sample t-test)
and assumptions (e.g., 80% power, 5%
significance level, standard deviation of 1,
difference of 2)
– Adjust for interim analyses and/or multiple
primary endpoints
Randomization
 Important for
– Avoiding/reducing bias
– Reduce likelihood of chance events impacting
study results
 Randomize wherever possible
– Treatment allocation
– Order of data collection (e.g., machine run
order, order or evaluator review)
Blinding
 Important for reducing bias and yielding
“robust” results
 Blind wherever possible
– Investigators, research personnel, animal
caretakers
– Treatment allocation
– Conduct of the experiment
– Assessment of outcome/data collection
 Acknowledge where you can’t blind and
incorporate practices to minimize
potential resultant biases
Data Handling
 Define stopping criteria in advance
 Prospectively define the primary
endpoint
 Define outliers and data exclusion
criteria prospectively
– Statisticians frown on dropping “outliers”
unless there is a good reason to
 Report missing data, why missing and
how handled
Statistical Analysis Plan
 Include an appropriately detailed
statistical analysis plan
 Address ALL endpoints - primary and
secondary
 Include adjustment for multiple
testing if necessary
Additional Information
 Landis el. 2012 A call for transparent reporting to
optimize the predictive value of preclinical research.
Nature 490(7419): 187-191.
 Collins, F.S. and L.A. Tabak. 2014. NIH plans to
enhance reproducibility. Nature 505:612-613.
 NIH Principals and Guidelines for Reporting Preclinical
Research http://www.nih.gov/research-training/rigorreproducibility/principles-guidelines-reportingpreclinical-research
 Updated Application Instructions to Enhance Rigor and
Reproducibility http://www.nih.gov/researchtraining/rigor-reproducibility/updated-applicationinstructions-enhance-rigor-reproducibility
How to Get Biostatistics Help
 Office Hours: 12-1:30 on Tuesdays
– Input regarding study design, data analysis,
interpretation or presentation of research
results in informal setting
– Reserve 15 min. spots on-line at
www.ucdmc.ucdavis.edu/ctsc/area/biostatistics/index.html
 Application for Resource Use
– More in-depth assistance with study design,
grant writing, data analysis and interpretation
– Submit application on line at
www.ucdmc.ucdavis.edu/ctsc/