The Basics of Reading A
Research Article
Amy Gutman MD ~ EMS Medical Director
prehospitalmd@gmail.com / www.TEAEMS.com
OBJECTIVES
Information Presented Meets or Exceeds NREMT Educational Standards
 EMS Research History
 Definitions
 Breaking down research articles
 Evidence-based medicine
 Research resources
 Summary
OVERVIEW
 Much “sensationalized” but bad
research
 Cellphones cause brain cancer
 Vaccines cause autism
 How do you tell “good” from “bad”?
 Is “good” research “important”?
 Review basics of reading a research
paper so you can keep up with current
trends in prehospital care
WHAT IS RESEARCH?
 Science is systematized knowledge derived
from observation & experimentation
 Research is a systematic investigation to
discover or establish facts by utilizing the
“scientific method”
 The scientific method is how researchers
construct an accurate, reliable &
repeatable answer to a question
SCIENTIFIC METHOD
 Anyone can perform &/or read research
 Observe something
 Ask a question, or “hypothesis”, about
observation
 Use hypothesis to make prediction
 Test prediction by experimentation
 Modify hypothesis after reviewing
experimentation results
 Repeat until satisfied hypothesis
answers question
 Ask more questions!
HYPOTHESIS & PREDICTIONS
 A hypothesis is a question or theory based
upon an observation
 Often common sense, but common sense
is neither common, nor true
 Examples:
 “Why do I get long-distance transports at
the end of my shift?”
 “Does anyone really drink ‘two beers’”?
SCIENTIFIC METHOD: EXAMPLE

Observation, Hypothesis & Prediction


Testing the Hypothesis / Methods:


“According to providers I asked, there are at least two awesome Cape Cod Medical Directors
who are not red-headed, short, chunky or female”
Results:


“My Medical Director is a an awesome, short, chunky,
red-headed female. Therefore I believe that ALL awesome
Medical Directors are short, chunky, red-headed females”
“Because there are awesome Medical Directors who didn’t meet the hypothesis, I modified
the hypothesis to “One awesome Medical Director is short, chunky & red-headed, but not all
awesome Medical Directors must be physically similar in appearance”
Reproducibility & Clinical Applicability:

Easily be reproduced in any prehospital environment

Medical Director physical characteristics may not indicate awesomeness
EMS RESEARCH
 The goal of EMS research is integrating results
into clinical practice after an evidence-based
initiative to improve patient outcomes in a
competitive & cost-conscious healthcare market
• Domains:
 Clinical: Study of direct patient care
 Systems: Effects of EMS system design & operations on resource utilization
 Educational: Training methodology for prehospital providers
EMS RESEARCH HISTORY

EMS practices initially modeled after battlefield/hospital care
 1st paper published in 1966 (BMJ) showed patients “suffocated” due to poor EMS training

1973 EMS Systems Act (Public Law 93-154) funded development of regional EMS
systems, protocols & research
 Pre-1980 only 3 “Efficacy Studies” published that evaluated if a practice or drug works in
ideal conditions (but not necessarily in the field!)
 A 1989 editorial stated “Impressively Deficient” data for “efficacy of scope of EMS practice”

1993 NHTSA FEMSC “EMS for Children Report” identified need for pediatric research

2001 NHTSA National EMS Database founded to answer research questions
 1st paper evaluated prehospital ETI in pediatric cardiac arrest & found that most providers
lacked training & that patients worse outcomes

Today: better funding, peer-reviewed journals & some clinically relevant research
EMS RESEARCH BARRIERS
 Few trained researchers with minimal funding
 Failure to translate research into clinical practice
 Lack of integrated IT systems linking prehospital data with patient outcomes
 Multiple locations / personnel / researchers
 HIPPA, IRBs
 Unstable patients often w/o diagnosis at time research conducted
 Uncontrolled research environment
 Critical nature limits practical & ethical experimental interventions
 Difficulty in obtaining informed consent
CURRENT NATIONAL EMS RESEARCH
(www.EMS.GOV/Research)
 National EMS Assessment (20092011) analyzed 20,000 agencies,
82,000 vehicles, 826,000 personnel
 National EMS Assessment
 EMS Workforce for the 21st Century:
Feasibility for Safety & Surveillance
 EMS Evidence-Based Guidelines
 EMS Systems Configuration Study
 National EMS Information System
(NEMSIS)
 EMS Performance Measures &
Outcomes Evaluation
 Longitudinal EMS Demographic Study
 National EMS Research Agenda
 Motor Vehicle Occupant Safety
Survey: Crash Injury & EMS
EMS RESEARCH ORGANIZATIONS / WEBSITES
PREHOSPITAL RESEARCH JOURNALS
 JEMS is most widely-read print journal
 Not peer-reviewed
 Opinions, summaries & commentaries that are
informative but not necessarily evidence-based
 Peer-Reviewed Journals
 Prehospital Emergency Care
 Prehospital & Disaster Medicine
 Annals of Emergency Medicine
 Pediatric Emergency Care
 Air Medical Journal
EVIDENCE-BASED MEDICINE (EBM)
 Scientific method to answer a question &
make patient care decisions
 Establish “standards of care” after multiple
studies replicate similar results, or a multicenter study shows undeniable results
 Example: Maine Selective Spinal Clearance
Study
 In cost-cutting age, insurance often will
not reimburse for non-EBM practices
 Litigation possible if non-EBM procedures
PEER-REVIEWED RESEARCH
 Rigorous review by persons of similar
credentials to researchers
 Maintains standards of quality & provides
credibility by critically evaluating research &
exposing flaws prior to publication
 A review will find the article:
 Acceptable (rare)
 Acceptable but requires revisions (common)
 Not acceptable (common)
HOW AN ARTICLE GETS TO PRINT (TIMELINE)
• Peer-reviewed EBM articles appear in many journals (some prestigious, some not)
• Most research happens years before publication

Days 1-7 Hypothesis Formulated


Days 1-60 Methodology formulated; Funding
application begins
Months 9-? For RCTs 12-24 months, Chart
Reviews usually 3-6 months

Year 2+ Abstract prepared & submitted for
conference presentation; article
prepared for submission

Year 2+ 8 Mo If accepted for revisions, editing
begins. If refused, hunt for another
journal begins

Year 2+ 10 Mo Article accepted; published 3-6
mos later unless special circumstances

Days 30-120 IRB proposal written, submitted,
resubmitted

Months 4-5 Identify & recruit investigators &
subjects

Month 6-9 Investigator training; Research,
data collection & article prep begins
(background, methodology)
JUDGING A RESEARCH PAPER

Credible source?

Applicable to EMS?

Design appropriate to answer question
being asked?

Variables controlled to ensure
reliability of results?

Results conclusive / logical?

Alternative explanations for results?

Do findings impact current evidence or
require changes in clinical practice?

How biased is the research?
BASIC VS APPLIED RESEARCH
BASIC


Performed in lab or clinical setting as
preliminary data collection to refine a
hypothesis
APPLIED / CLINICAL

Real subjects in real-world situations

Effects of a drug, device, or procedure
studied on humans
Independent variables manipulated to
observe & describe effect(s) on a dependent
variable


Often examines properties of drugs or used
in development of diagnostic tests

Every drug starts as a chemical whose
properties are examined through
experimentation
Test & prove (or disprove) therapies,
procedures & protocols
LEVELS OF SCIENTIFIC EVIDENCE
 Ia:
Randomized Controlled Trial
(RCT) meta-analysis
Strongest
 Ib:
RCT (single)
 IIa:
Prospective Non-RCT
 IIb:
Experimental study (case series,
animal studies)
 III:
Retrospective or Descriptive studies
(case-control, comparative,
correlation studies)
 IV:
Expert opinions, extrapolations
 V:
Weakest
Rational conjecture
RANDOMIZED CONTROLLED TRIAL (RCT)
 Subjects randomized into control &
experimental groups
 Groups evaluating for variables &
outcomes of interest
 Example: Outcomes of STEMI patients
given aspirin vs those not given aspirin
 Uses the scientific method & most
“valid” research method
RCT EXAMPLE
 Mattox KL, Bickell B, Pepe PE*, Burch J, Feliciano D.
Prospective MAST study in 911 patients. J. Trauma. 1989;29:1104-12
 911 adult trauma pts with SBP <90mmHg randomized by alternate days &
transported to Ben Taub
 Experimental Variable: MAST or no-MAST
 Dependent Variable: survival from trauma
 Independent Variables: etiology, age, race, sex, location of injury, trauma scores,
injury severity scores
 Results: Mortality 31% MAST group, 25% in non-MAST group
 Conclusions: MAST trousers in trauma pts increases mortality
RCT: COHORT STUDIES
 Subjects with a condition or
receiving a treatment compared
with another group not affected
by condition / treatment under
investigation
 Different than a double blind
study as this is an observational
study only
 No intervention by the researchers
EXAMPLE: COHORT STUDY

Gottesman BE, Gutman A et al. Radiation exposure in emergency
physicians working in an urban ED: a prospective cohort study*.

HYPOTHESIS: EM MDs are exposed to radiation levels above NCRP limits

METHODS: Prospective cohort study conducted at a level I Trauma ED. Dosimeter
radiation badges placed on all ED MDs & their phones carried 24/7/365. At the end
of the study exposure dose for each subject calculated

RESULTS: 75 MDs enrolled. Compliance 99%. Annual extrapolated exposure for
subject with highest radiation level was 50 mrem, below 5000 mrem exposure limit
for health care workers

CONCLUSION: EM MDs working in an urban, academic, level I trauma center ED are
not at risk of exceeding NCRP dose limits for ionizing radiation exposure
*Am J Emerg Med. 2010 Nov;28(9):1037-40
RCT: DOUBLE BLIND
STUDY
 Neither subject(s) nor researchers knows whether subject receiving treatment, or
control (placebo) treatment
 Example:
 Aufderheide TP, et al. Resuscitation Outcomes Consortium (ROC) PRIMED trial:
rationale & methodology for the impedance threshold device (ITD)*
 AIM: Compare OOHCA survival to discharge between CPR+ITD vs CPR vs fake ITD
 METHODS: Prospective, double-blind, RCT in OOHCA pts by EMS systems
participating in the ROC. 1.4% absolute survival difference found in CPR + ITD
 CONCLUSION: If ITD demonstrates hypothesized survival improvement, an
estimated 2700 deaths from OOHCA could be prevented annually
*Resuscitation. 2008 Aug;78(2):179-85
SYSTEMATIC REVIEWS (RCT META-ANALYSIS)
 Summary of literature & statistical analysis
after detailed search to determine
effectiveness of interventions / procedures
 Results of multiple studies evaluating a similar
question are pooled & statistically analyzed
 Highest level of scientific evidence as reduces
possibility of false negative results
 Poor quality studies are often excluded from
the statistical analysis
 Because patient numbers are so high (hundreds
to thousands), minimal chance of inaccurate
conclusion by “chance”
ANIMAL RESEARCH
 Precursor to evaluating a human intervention
 Computer models often initially used vs
animal models for ethical reasons
 Results often cannot be extrapolated to
draw conclusions on what will happen in humans
 Isoproterenol dosing based on rat studies resulted in 3500 British asthmatics dying
from overdoses Pharma, 1971; vol 18:272
 Domperidone withdrawn from market after 25 pts died from lethal arrhythmias
not seen in dogs with 70 X toxic human dose Drugs, 1982, vol 24:360-4
 Digitalis testing delayed in animals, while used safely for decades in humans
because of toxic rodent effects Tox in vitro, 1992, vol 6:47-52
EPIDEMIOLOGICAL / CORRELATION STUDY

Evaluate relationship between variables to
determine if there is any correlation


“Positive”, “Negative”, “Null”
Often miscategorize “association” with “cause”
 Positive correlation does not mean “A” causes “B”
 Example: Obese persons drink soda, therefore
soda causes obesity

Rely on memory recall or data documentation,
which is often incomplete or unreliable

Used in surveys or retrospective chart reviews
which may lead for a more controlled study
EPIDEMIOLOGICAL / CROSS-SECTIONAL SURVEYS
 Collects data at a single point in time to evaluate
a policy or public health issue
 Example: A case of hepatitis reported at
“Restaurant Q”, so all persons who ate at that
restaurant are questioned to discover if they have
symptoms consistent with hepatitis
 Limited by event memory, difficulty in
contacting subjects
EPIDEMIOLOGICAL / LONGITUDINAL OR
CASE CONTROL STUDIES
 Follows subjects over a period of
time, serially asking research
questions for study duration
 Example: Framingham Heart Study has
followed residents of Framingham, MA
since 1948 looking for heart disease
patterns
 In case control studies, subjects
with a condition are compared with
those who do not
 Observational, not interventional
therefore classified as
“epidemiological”
CASE REPORTS
 Case Study / Case Report
 Detailed info about a single subject or a
small group of subjects
 Examples: 1st time for new surgical
technique, “weird” presentations of diseases
 Case Series
 Report on a series of subjects with one
specific unifying factor but no control group
 Examples: Outcomes of pelvic trauma
patients with MAST trouser application
LITERATURE REVIEW
 Exhaustive search of all relevant
evidence-based literature related to a
topic using multiple databases
 Cochrane reviews (meta-analysis)
 PubMed (NIH study repository)
 CINAHL
 Asking a question of Google or Wikipedia
is NOT an exhaustive (nor reliable)
literature review
Accredited
Resources
Wikipedia
RETROSPECTIVE CHART REVIEWS /
HISTORICAL STUDIES
 Systematized searches for fact(s), then using
information to describe, analyze & interpret
the past
 Example:
 Bledsoe BE, Smith MG. Medical Helicopter Accidents
in the United States: A 10-Year Review.
 Methods: Review all US air medical helicopter
accidents from 1997-2002 from NTSB database
 Results: Majority of air medical helicopter accidents
resulted from pilot error or mechanical failure
*J Trauma. 2004;56:1325-1329
EXPERT OPINIONS
 Examples:
 Systematic reviews
 Narrative reviews
 Pure opinion pieces
 Position papers
 Based on a literature review
(including RCT meta-analysis) but
considered low on the “Scientific
Evidence” schematic due to not
being an actual “study”
RATIONALE CONJECTURE
 Lowest level of scientific validity
but often accurate & important
 Anecdotal reports often fit into
this category, as do “interesting
observations”
 Often amusing due to “duh” factor
“DUH” EXAMPLE: Back MD. Knowing Your Mate Value: Sex-Specific
Personality Effects on Accuracy of Expected Mate Choice*.

Males rated photos of women displaying
“exploitability cues” indicating appropriateness for
one-night stands vs “stability cues” appropriate for
marriage or long-term dating

22 exploitability cues correlated with “sexual
receptivity”, including:
 Lip licking/biting, appearing intoxicated, tight clothing,
obesity, dumb conversations, rubbing breasts / genitals

Conclusions: Males prefer drunk, slutty**, dumb
women for 1 night stands
*Psychol Sci. 2011 Aug;22(8):984-9.
**Authors’ words, not mine!
PSEUDOSCIENCE: THINGS WE BELIEVE ARE TRUE
THAT AREN’T
•
Epinephrine saves lives in OOHCA
• No RCT has shown better outcomes with epinephrine vs placebo in OOHCA
•
Amiodarone improves outcomes in pediatric OOHCA
• Only 1 RCT suggests risk > benefit of amiodarone (not statistically significant)
•
ATLS & the “Golden Hour”
• Evidence suggests ATLS has less impact on outcomes than believed
• Zero studies have shown anything magical about the 1st 60 minutes managing
trauma patients
ARTICLE COMPONENTS

Author

Title

Key Words

Abstract

Introduction / Background

Methodology

Results

Discussion

Conclusion

References / Citations

Acknowledgements
AUTHOR INFORMATION

Primary, secondary & associate authors
 Primary & secondary do majority of work
 Associates edit, collect data, produce diagrams
 If a multi-center, 1 author from each site listed
 If >10 sites, authors listed in acknowledgements

Lists primary site / contact information

Dirty secret…many “authors” have never seen the
research – they are department chairs, “famous” or
“friends of famous”, all of which lend credence to
research (& help get it published)
TITLE

“Headline”* clearly & briefly describing what paper examines

May be misleading to attract attention

Example:
 “Cellphones linked to brain cancer” is sexier than …
 “Environmental risk factors for cancers of the brain and nervous system: the use
of ecological data to generate hypotheses.”
*Keith Wesley
KEY WORDS
 Important “search terms”
 Different query results from:
 “Prehospital”
 “Paramedics”
 “EMTs”
 “EMS”
ABSTRACT
 Short summary of study
 Distills key study elements into a few easily understood sentences
 After reviewing, the reader determines if article is “worth my time to read”
INTRODUCTION / BACKGROUND
 Familiarize readers with subject &
research field
 Examines key question & reviews
current relevant literature
 Provides rationale for why research is
important
 Presents hypothesis attempting to
answer the key question
METHODOLOGY
 Evaluate research quality by showing
how study conducted
 Study design & population description
 Data collection & interpretation
methods
 Provides details so other researchers
can replicate & validate findings
 Reader “assumes” articles published
in a peer-reviewed journal are
statistically valid
 “Valid” data still may be misinterpreted
by the authors or media
RESULTS
 Objective description of data indicating
statistical significance / insignificance
using raw & calculated data
 A “p value” <0.05 indicates result not likely to
have happened by chance, and is “probably true”
 Data interpretation reserved for discussion section
 Results must be interpreted by reader keeping in mind validity of the
hypothesis, study methodology & author bias
DISCUSSION

Study results interpreted & evaluated against existing body of literature

Attempts to connect results to bigger picture & show how results applied clinically
 Statistically significant findings may not have much clinical application

Author’s opinion about results is valuable but taken with a grain of salt
 Any bias or unexpected results reviewed & explained
 No one starts a study expecting to “fail”
 Example: If a author states “trend towards significance”, it means that the study failed to
show a positive outcome, so the author “stretches the truth” to imply that data are really
significant, but the study itself was flawed

Even well-designed / conducted studies limited by imperfect nature of EMS research
CONCLUSIONS
 Summarizes info with recommendations
based upon data analysis
 Recommendations range from changing
clinical practice to more research
 Reader must weigh all parts of the study &
determine if methodologically sound ,
clinically reproducible & valid enough to
change or continue current practice
 Race, gender, geography, economic status &
“sensitive” factors should not be ignored due
to political correctness
 Example: Research links asthma incidence &
severity with race/ socio-economic status;
therefore providers should expect patients with
more severe respiratory distress in urban, black &
Hispanic areas vs rural Caucasian areas
REFERENCES / CITATIONS
 Each cited author & paper plus quotes,
ideas & content in a standard format
 Source of further info on a topic
 Decisions rarely made after one study
 References provide further info on that
subject in which you are interested
WHAT DOES THAT CITATION MEAN?
Authors / Researchers
“And Others”; usually part of the research team
Title of study, describes the hypothesis & results
Gottesman BE, Gutman A, et al. Radiation exposure in emergency physicians
working in an urban ED: a prospective cohort study. Am J Emerg Med. 2010
Nov;28(9):1037-40. Epub 2010 Mar 9. PMID: 20825934
PubMed ID#
Electronic publication date: year, month, date
Journal Name; does not indicate if peer-reviewed
Print publication date: year, month; journal year(issue):pages
ACKNOWLEDGEMENTS &
DISCLOSURES
 Research expensive & often sponsored by a university or company
 Authors must disclose financial or other support (i.e. airfare to a
conference) to explain potential conflicts of interest
 “Disclosures” indicate bias that may affect study’s outcome
 If design adheres to scientific method, little chance for bias to affect outcomes
 Example: If a study determines a drug to be ineffective / harmful, & if that study were
sponsored by a competitor of the drug then readers should know that there is not
only bias present, but potentially a data misinterpretation
NEMSRA RECOMMENDATIONS TO IMPROVE
FUTURE EMS RESEARCH

Structured training programs


Collaborative Centers of Excellence combining investigators, resources & public policy



i.e. Utstein criteria
The FDA & Office for Human Research Protections should re-evaluate exception from
informed consent


i.e. More RCTs & evidence-based practice
Standardized data collection using uniform prehospital data elements


i.e. The Resuscitation Outcomes Consortium (The “ROC”)
i.e. Recognizing that prehospital care is an actual “academic” branch of medicine
Better evidence before implementing procedures, devices, or drugs


i.e. Fellowships
i.e. HIPPA and human subject testing exemptions
Investment in infrastructure to overcome obstacles impeding EMS research

i.e. IT, funding, data-sharing
REFERENCES

Pierce, L. L. Twelve steps for success in the nursing research journey. The Journal of Continuing Education in Nursing, 2009. 40, 154-162

My Athletic Life (Scientific Research 101 series). 2012. http://www.eecs.harvard.edu/~michaelm/postscripts

Bledsoe B. An Introduction to EMS Research (presentatiaon). 2009

www.researchagenda.org (National EMS Research Agenda)

www.naemsp.org (National Association of EMS Physicians)

www.acep.org (American College of Emergency Physicians)

www.rescuehouse.com (Rescue House Fire Resources)

www.nhtsa.gov (National Highway Transportation Safety Agency)

www.signal51group.com (Shreveport Fire Department Signal 51)

www.ci.shreveport.la.us

Edenburn D. Medical research made easy (presentation). 2013

Fox J, Day S, Reynolds L, et al. Post-clinical trial survey of EMS providers: Research experience and attitudes. Air Med J. 2010;29(1):34–36

Schmidt TA, Nelson M, Daya M, et al. EMS providers’ attitudes and experiences regarding enrolling patients in clinical research trials. PEC.
2009;13(2):160–165

US Department of Health & Human Services. Code of Federal Regulations 45 CFR46. 2010

US FDA Regulations Relating to Good Clinical Practice and Clinical Trials. 2011

Ryan KJ, Brady JV, Cooke MD, et al. Belmont Report: Ethical principles and guidelines for the protection of human subjects of research. In
Office of Human Subjects Research. 1979

Literature / image searches: Wikipedia, Google, Bing, www.pubmed.com
SUMMARY
www.prehospitalmd@gmail.com / www.TEAEMS.com
 EMS must be driven by science rather than conjecture & pseudoscience
 Anyone can undertake a research project – just ask a question & try to
scientifically find out the answer
 EMS providers should analyze research & drawing appropriate conclusions
 By understanding evidence-based medicine, providers play an active role
in the advancement of patient care EMS must be driven by science