The Good, The Bad & The Ugly
of EMS Research
Definitions
Research: careful,
systematic, patient
study and investigation
in some field of
knowledge,
undertaken to discover
or establish facts or
principles.
Definitions
Science:
The state or fact of knowing; knowledge.
Systematized knowledge derived from
observation, study, and experimentation
carried on in order to determine the nature or
principles of what is being studied.
Is EMS Art or Science?
The knowledge of
EMS is science.
The way it is applied
is art.
Excellent EMTs know
the science of EMS
and use the art of
EMS to apply the
science.
Art or Science?
Some health practices
are more about art than
science.
Chiropractic, for
example, has little
science and a lot of art.
Art or Science?
Neurology today has
a great deal of
science and little art.
Art or Science?
When you have a life
threatening illness or
injury, would you seek
out a health care
provider whose
practice was more art
or more science?
Art or Science?
EMS is the safety net
of society.
We are often the first
to provide care to the
injured and the infirm.
Our practices must be
based on science.
But we must apply them with art!
Research
Most medical
research is based on
the application of the
scientific method.
The Scientific Method
The scientific method is the process by
which scientists, collectively and over time,
endeavor to construct an accurate (that is,
reliable, consistent and non-arbitrary)
representation of the world.
The Scientific Method
Steps:
1. Observe some aspect of the universe
2. Invent a tentative description, called a hypothesis,
that is consistent with what you observed.
3. Use the hypothesis to make predictions.
4. Test those predictions by experiments or further
observation and modify the hypothesis in the light of
your results.
5. Repeat steps 3 and 4 until there are no
discrepancies between theory and experiment and/or
observation.
The Scientific Method
Hypothesis:
groundwork,
foundation
supposition; an
unproven theory.
The Scientific Method
The Scientific Method
The great advantage of the scientific method is
that it is unprejudiced:
One does not have to believe a given researcher, one
can redo the experiment and determine whether his or
her results are true or false.
The conclusions will hold irrespective of the state of
mind, or the religious persuasion, or the state of
consciousness of the investigator and/or the subject of
the investigation.
Faith, defined as a belief that is not based on logical
proof or material evidence, does not determine
whether a scientific theory is adopted or discarded.
The Scientific Method
A theory is accepted not based on the
prestige or convincing powers of the
proponent, but on the results obtained
through observations and/or experiments
which anyone can reproduce.
The results obtained using the scientific
method are repeatable.
The Scientific Method
There are many types of “pseudoscientific”
theories which seem based on a mantle of
apparent experimental evidence but that,
when examined closely, are nothing but
statements of faith.
The Scientific Method
“Faith is believing
what you know ain’t
so.”
Mark Twain, 1894
“Following the Equator”
Science versus Pseudoscience
How to tell the difference:
Anecdotes do not make a science.
Scientific language does not make a science.
Bold statements do not make claims true.
Heresy does not equal correctness.
Reversal of the burden of proof.
Rumors do not equal reality.
Science versus Pseudoscience
How to tell the difference:
Failures are rationalized.
Use of emotive words and false analogies.
Ad ignorantiam reasoning (if you can’t
disprove a claim, it must be true).
Ad hominem or tu quoque statements.
Overreliance on authorities.
Circular reasoning.
Reductio ad absurdum reasoning.
The Scientific Method
Enzyte has to
work….after all,
Bob is smiling!
The Scientific Method
The Scientific Method
Pseudoscience is still
a major part of our
culture. Many
practitioners resort to
pseudoscience and
other anecdotal
practices.
The Scientific Method
Typically uses terms
and images to appear
“scientific”.
The Scientific Method
Reliance on “names” and
anecdotes.
Dr. Greg Cynaumon?
Physician?
Psychologist?
Nutritionist?
Chiropractor?
Master’s and Doctorate in
psychology from Sierra
University (known
diploma mill).
The Scientific Method
No clinical evidence CortiSlim works.
No information about contents (natural or
herbal).
Dr. Talcott fringe nutritionist with loose
affiliations with University of Utah.
One of the largest scams in recent history.
FDA issued a cease and desist order and
fined Talbot and Cynamoun.
Use of the Scientific Method in EMS
Observation: During a discussion you and
your coworkers bring up the idea that there are
more psychiatric emergencies when the moon
is full.
Hypothesis: Psychiatric emergencies are
more common when the lunar cycle is in the
full moon phase.
Use of the Scientific Method in EMS
Prediction: You predict that there will be
more EMS calls for psychiatric
emergencies when the moon is full as
compared to other times of the month.
Use of the Scientific Method in EMS
Testing: You set up an experimental
protocol to test your hypothesis.
For this you:
Determine you will conduct the study for a
year.
Determine when a full moon is scheduled to
appear.
Determine that you will declare two days
before and two days after the full moon a fiveday “full moon period.”
Use of the Scientific Method in EMS
For this you:
Define what a psychiatric emergency will be.
Gather information over the course of the year
that includes the date of all psychiatric
emergency patients.
When the data has been gathered for the
period previously determined, you determine:
1. How many psychiatric patients were there?
2. How many were treated during the full moon
periods?
Use of the Scientific Method in EMS
For this you:
Analyze the data based on the number of
emergencies that arose during full moon
periods and compare to the number of
emergencies during the other periods.
Perform any statistical tests necessary to
understand the data.
Use of the Scientific Method in EMS
Results:
Full Moon Period
Days=65
Other Period
Days = 300
Total
Days = 365
Ψ patients = 10
Ψ patients = 74
Total Ψ patients = 84
Total Patients =
2,105
Total Patients = 9,300 Total Patients = 11,315
Use of the Scientific Method in EMS
Results: Your study finds:
During full moon period 0.5% of patients were
Ψ patients.
During other period 0.8% of patients were Ψ
patients.
During the entire study period 0.7% of patients
were Ψ patients.
Use of the Scientific Method in EMS
Results: Your study finds:
The incidence of Ψ patients during the full
moon period was 0.15 patients per day.
The incidence of Ψ patients during the other
moon phases was 0.25 patients per day.
The incidence of Ψ patients overall was 0.23
patients per day.
Use of the Scientific Method in EMS
Psych patients are less common during the
full moon.
HYPOTHESIS DISSPROVED!
Use of the Scientific Method in EMS
Revise hypothesis:
New hypothesis: Psychiatric emergencies are
no more frequent during full moon lunar
phases than during other lunar phases.
HYPOTHESIS ACCURATE.
Report findings. Paper should be detailed enough that
anybody can follow it and repeat your experiment.
Repeat study to determine whether findings can be
repeated.
EMS Research
The more valid a
study, the closer it is
to the truth!
EMS Research
Bias:
A mental leaning or inclination; partiality;
prejudice; bent.
The more a study design adheres to the
scientific method, the less chance for bias to
affect the outcomes.
EMS Research
Types of biases:
Selection bias
Volunteer bias (tends to favor the treatment group)
Nonrespondent bias
Group not homogeneous
Measurement bias
Instrument bias
Insensitive measure bias
Expectation bias
Recall bias (people remember positive events rather
than negative events)
Attention bias (aware of participation)
EMS Research
Types of biases:
Intervention bias
Contamination bias
Compliance bias
Withdrawal bias
Proficiency bias
Publication bias
Cherry picking (mainly retrospective studies)
Trashcan bias
EMS Research
Avoiding bias:
Randomize subjects
Prospective design
Blind subjects and researchers
Contract to publish irregardless of findings
Admit conflicts of interest
Use null hypothesis
EMS Research
Bias (sometimes flat
deception) has been
a common practice in
medicine and EMS.
Quackery and bias
still permeates many
aspects of modern
healthcare.
EMS Research
Validity:
Whether the study measures what it was supposed to
measure.
Validity refers to the appropriateness of the
interpretation of the results of a study.
External Validity: The extent and appropriateness of
the generalizability of results.
Internal Validity: The basic minimum control,
measure, analysis and procedures necessary to make
results interpretable.
EMS Research
Constant:
A characteristic or condition that is the same for all
individuals in a study.
Variable:
A characteristic that takes on different values or
conditions for different individuals.
Dependent Variable: the variable being affected or assumed
to be affected my the independent variable.
Independent Variables: the variables that affects (or is
assumed to affect) the dependent variable under study.
Experimental Variable: at least one independent variable
being manipulated by the researcher.
Levels of Evidence
Levels of Evidence
Center for Evidence-Based Medicine
(Oxford)
Ia. Meta-analysis of RCTs
Ib. One RCT.
IIa. Controlled trial without randomisation.
IIb. One other type of quasi-experimental study.
III. Descriptive studies, such as comparative studies,
correlation studies, and case-control studies.
IV. Expert committee reports or opinions, or clinical
experience of respected authorities or both.
Levels of Evidence
American Heart Association
1. Positive randomized controlled trials.
2. Neutral randomized controlled trials.
3. Prospective, non-randomized controlled trials.
4. Retrospective, non-randomized controlled trials
5. Case series (no control group)
6. Animal studies
7. Extrapolations
8. Rational conjecture (common sense)
Study Designs
Randomized Controlled Trial (RCT):
A group of patients are randomized into an
experimental group and a control group. These
groups are followed up for the
variables/outcomes of interest.
RCTs most closely approximate the scientific
method and are the most valid of studies.
Randomized Controlled Trial
EMS Research
Houston MAST Study:
Constant: All patients with abdominal trauma
with hypotension transported by the HFD to
Ben Taub Hospital.
Dependent Variable: survival from trauma.
Independent Variables: age, sex, location of
injury, transport time, etc.
Experimental Variable: Application or nonapplication of the MAST.
Example
Mattox KL, Bickell B, Pepe PE, Burch J, Feliciano
D. Prospective MAST study in 911 patients. J.
Trauma. 1989;29:1104-12
911 trauma patients > 15 years of age with
systolic BP < 90 mmHg were randomized by
alternate day assignment. All transported by HFD
to Ben Taub hospital trauma center.
Experimental variable: MAST or no-MAST
Dependent variable: survival from trauma
Example
Independent variables: etiology, age, race,
sex, location of injury, trauma scores, injury
severity scores. Scores revealed the two
groups to be statistically identically
matched.
Findings: Mortality rate 31% in the MAST
group and 25% in the non-MAST group.
Difference statistically significant.
Double Blind Study
Double Blind Study
A double blind study is one in which neither
the patient nor the physician knows
whether the patient is receiving the
treatment of interest or the control
treatment.
It is a type of RCT.
Double Blind Study
Double Blind Study
Bracken MB, Shephard MJ, Collins WF, et al. A
randomized, controlled trial of
methylprednisolone or naloxone in the treatment
of acute spinal-cord injury. NEJM
1990;322(20):1405-11
Patients with spinal cord injury randomized by
computer to receive methyprednisolone,
naloxone, or placebo.
Researchers and patients did not know which
drug was which (prepared in pharmacy)
Quasi-Experimental Studies
Quasi-experimental studies use intact
groups of subjects rather than assigning
subjects to groups at random.
Quasi-Experimental Studies
Non-randomized controlled trial:
1 group receives intervention
1 group receives no-intervention (control)
Subjects assigned to groups by methods other
than randomization.
Cohort Studies
A Cohort Study is a study in which patients
who presently have a certain condition
and/or receive a particular treatment are
followed over time and compared with
another group who are not affected by the
condition under investigation.
Cohort Study
Cohort Studies
Framingham Heart Study:
People in Framingham, MA have been
followed for over 50 years.
Conducted by National Heart, Lung, and Blood
Institute (NHLBI) and Boston University.
5,209 residents between 30 and 60 years of
age initially enrolled.
In 1971, 5,124 children (and their spouses) of
the original cohort added.
500 minority members have been added.
Cohort Studies
Framingham Heart Study:
Every two to four years, study participants are
given extensive medical examinations
including a medical history, blood tests and
other sophisticated tests such as bone scans,
eye exams and echocardiograms assessing
multiple aspects of their current health status.
Cohort Studies
Framingham Heart Study (results):
Risk factors for heart disease identified.
Better understanding of the effects of lifestyle.
More than 1,000 papers published significantly
adding to the knowledge base of heart
disease.
Study continues….
Case-Control Studies
Case Control Studies: Case control studies
are studies in which patients who already
have a certain condition are compared with
people who do not.
Case-Control Studies
Case Series
A case series is a report on a series of
patients with an outcome of interest. No
control group is involved.
Case Report
A case report is a narrative report of an
interesting case.
Historical Studies
Historical studies are systematized
searches for the facts and then using the
information to describe, analyze, and
interpret the past.
Historical Studies
Bledsoe BE, Smith
MG. Medical
Helicopter Accidents
in the United States:
A 10-Year Review. J
Trauma.
2004;56:1325-1329
Historical Studies
Methods:Review of
all air medical
helicopter accidents
in the US from 19972002 from the NTSB
database.
Systematic Reviews
A summary of the medical literature that
uses explicit methods to systematically
search, critically appraise, and synthesize
the world literature on a specific issue.
Systematic Reviews
Bledsoe BE. Critical
Incident Stress
Management (CISM):
Benefit or Risk for
Emergency Services?
Prehosp Emerg Care.
2003;7:272-279
Meta-Analysis
A meta-analysis is a systematic review that
uses quantitative methods to summarize
the results.
Meta-Analysis
Meta-Analysis
Meta-analysis of RCTs represents the
highest level of scientific evidence.
Allows for more objective appraisal of the
evidence.
Reduces the possibility of false negative
results.
Meta-Analysis
van Emmerik AAP,
Kamphuis JH,
Hulsbosch AM,
Emmelkamp PMG.
Single-session
debriefing after
psychological trauma:
a meta-analysis.
Lancet.
2002;360:766-771
Meta-Analysis
Meta-Analysis (Observational Studies)
Less valid than RCTs.
Still valid in that it gives better information
of the population as a whole.
Many EMS interventions cannot be tested
with an RCT as it might be unethical to
withhold care from the control group.
Meta-Analysis (Observational Studies)
Bledsoe BE, Wesley AK, Eckstein M, Dunn
TM, O’Keefe MF. Helicopter transport of
trauma patients: a meta-analysis. J
Trauma. 2006;60:1257-1266
Observational meta-analysis examining
validated trauma scoring systems in trauma
patients transported from the scene to a
trauma center.
Meta-Analysis (Observational Studies)
Conclusions:
2 out of 3 trauma patients transported from the
scene to a trauma center have minor injuries
based on validated trauma scoring criteria.
1 out of 4 patients transported are discharged
from the emergency department.
Animal Studies
Animal studies provide insight into biology.
For ethical reasons, some studies cannot
be carried out on humans.
Computer modeling are replacing many
animal studies.
Expert Opinion
Expert opinions can take various forms:
Systematic reviews
Narrative reviews
Pure opinion pieces
Rational Conjecture
Lowest level of
scientific validity.
But, overall very
important.
“You don’t have to run
a Chi-Square test on
common sense”
Journals
Not all journals are
the same.
Impact rating:
Cites
Articles
Readership
Journals
Journal
Impact Rating (2005)
NEJM
44.016
Lancet
23.407
JAMA
23.332
Circulation
11.632
BMJ
9.052
Crit Care Med
5.077
Ann Emerg Med
2.782
Acad Emerg Med
1.789
J Trauma
1.722
Statistics
Statistics are essential to any scientific
analysis to eliminate bias and chance.
“Torture numbers and they’ll confess to anything.”
(Gregg Easterbrook)
“Satan delights equally in statistics and in quoting
scripture.... ” (H.G. Wells)
“The average human has one breast and one
testicle.” (Des McHale)
Summary
EMS must be driven by science.
Science is based upon quality research.
EMS providers of the future must be able to
understand, and in certain cases, conduct
valid research.
Summary
Break Time
Good
Gaucshe M, Lewis RJ,
StrattonSJ, et al.
Effect of Out-ofHospital Pediatric
Intubation on Survival
and Neurologic
Outcome: A Clinical
Trial. JAMA.
2000;283:783-790
Good
Methodology: RCT
Randomization: By
calendar day.
Journal: JAMA
Participants: 830
consecutive patients  12
years and weight < 40 kg
Interventions:
BVM only (odd days)
N= 410
BVM then ETI (even days)
N= 420
Outcome Measures:
Survival to discharge
Neurologic status at
discharge
Results:
No improvement in
survival or neurological
outcome in either
group.
Bad
Timmerman S, Cardoso
LF, Ramires JAF,
Halperin H. Improved
Hemodynamic
Performance with a
Novel Chest
Compression Device
During Treatment of inhospital Cardiac Arrest.
Resusc. 2004;61:273280
Bad
Methodology: Case
series
Randomization: None.
Journal: Resuscitation
Participants: 31
terminally-ill, fresh dead
Brazilians who failed 10
minutes of ACLS. Age 68
 10 years
Interventions:
Swan-Ganz catheters
placed
Interventions:
90 seconds A-CPR (60
compressions / minute)
followed by manual CPR
(100 compressions /
minute).
No ventilations
Subjects then intubated.
Ventilated (12 / minute)
Epinephrine given
Bad
Outcome Measures:
CPP
MAP
Results:
Usable blood
pressures obtained in
16/31 patients (52%)
Rosary or wake
interrupted (25%)
CPP:
A-CPR: 20  12
M-CPR: 15  11
PAP:
A-CPR: 115  42
M-CPR: 153  28
Results:
Increased CPP is
related to increased
chances of
resuscitation. A-CPR
improved CPP.
Ugly
Everly GS, Flannery
RB, Eyler V A. Critical
Incident Stress
Management (CISM):
A Statistical Review of
the Literature. Psych
Quart. 2002;73:171182
Ugly
Methodology: Metaanalysis?
Randomization: None.
Journal:
Participants: Metaanalysis of 8 CISM
investigations.
Evaluation: Results
pooled and Cohen’s d
calculated
Results: Significant effect
size (Cohen’s d) of 3.11
showing a beneficial
outcome from CISM.
You can’t obtain a
Cohen’s d without a
control group!
1 - mean2
mean
d=
1
2
v 2 + v 2 /2
Ugly
STUDY
CISM
MEASURES
PROBLEM
Busuttil (1995)
Used CISM to treat PTSD in
British Air Force personnel.
PTSD Scale
CISM is not a
treatment for PTSD
Flannery (1995)
ASAP
Assaults
Questionnaire
No mention of CISM in
paper
Flannery (1998)
ASAP
Assaults
Questionnaire
No mention of CISM in
paper
Flannery (1999)
ASAP
Assaults
Questionnaire
No mention of CISM in
paper
Richards (1999)
Debriefing following bank
robbery in NSW
IES
CISM provided at 10
0r 24 hours
Mitchell (2000)
CISM provided for volunteer
firefighters with PTSD 3 years
after a tornado
Diagnostic interviews
CISM is not a
treatment for PTSD
Flannery (2000)
ASAP
Assaults
No mention of CISM in
paper
Flannery (2000)
ASAP
Assaults
CISM mentioned
Good
Wang HE, Kupas DF,
Hostler D, Cooney R,
Yealey DM, Lave JR.
Procedural
Experience with outof-hospital
endotrachela
intubation. Crit Care
Med. 2005;33:
Good
Methodology: Analysis of
administrative database
Randomization: None
Journal: Critical Care
Medicine
Participants:
Records: 1,544,791
11,484 ETIs by 5,245
rescuers.
Outcome Measures:
Of 5,245 rescuers:
67% performed  2 ETIs.
39% performed 0 ETIs/
Mean ETIs = 1
Results:
Out-of-hospital intubation
is an uncommon event.
Bad
Amatangelo M,
Thomas SH, Harrison
T, Wedel SK. Analysis
of Patients Discharged
from Receiving
Hospitals within 24
Hours of Air Medical
Transport. Air Med J.
1997;16:44-14
Bad
Methodology:
Retrospective review of
flight records
Randomization: None
Journal: Air Medical
Journal
Participants:
945 flights
42 (4.4%) discharged less
than 24 hours of
transport.
Results: Patients met air
transport criteria.
A priori
Ugly
Ohshige K, Kawakami
C, Kubota K,
Tochikubo O. A
contingent valuation
study of the
appropriate user price
for ambulance service.
Acad Emerg Med.
2005;12:932-940
Ugly
Methodology:
Questionnaire survey
Randomization: None
Journal: Academic
Emergency Medicine
Participants:
3,363 ambulance
service customers sent
survey.
2,029 (60.3%) returned
survey.
Results:
Public inelastic (not
concerned) about cost
of ambulance service
when case is an
emergency.
Public fairly elastic
about price in
nonserious situations.
Thus, a ¥ 20,000 (US$
190.00) user fee will
reduce the number of
nonemergency calls.