EBM for Novice
Maria Kwok, MD, MPH
Assistant Clinical Professor
Section of Pediatric Emergency Medicine
Morgan Stanley Children’s Hospital of New York
Presbyterian
Outline
• Why is EBM important?
• What is EBM
• Understand foreground vs. background
questions
• Formulate PICO questions
• Find resources, do searches
• Validity of an article
• Appraise “Therapy” articles
• Appraise “Diagnostic” articles
Why is EBM Important?
• MEDLINE
– 400,000 new entries added each year
• To keep up-to-date
– Need to read 6,000 articles each day
Why is EBM Important?
Lag time from time of “knowing” to
time of “implementation”
– 13 years for thrombolytic therapy
– 10 years for corticosteroids for
acceleration of fetal lung maturity
(Antman EM, JAMA, 1992)
Traditional Approaches
Self-reported reading time per week.
(University setting)
– Medical students
– Interns
– Senior residents
– Fellows
– Attendings graduating
• Post 1975
• Pre 1975
60 min.
none
10 min.
45 min.
60 min.
30 min.
There is simply no way we
can keep up to date in
medicine using traditional
approaches!
Outline
• Why is EBM important?
• What is EBM?
• Understand foreground vs. background
questions
• Formulate PICO questions
• Find resources, do searches
• Validity of an article
• Appraise “Therapy” articles
• Appraise “Diagnostic” articles
Quiz: Define EBM
A) A means of camouflaging biostatistics
in medical training
B) An annoying and overused catch phrase
C) The conscientious, explicit, and
judicious use of current best evidence in
making decisions about the care of
individual patients
Research Evidence
Decision Making
Clinical Expertise
Patient Preference
Evidence Based Medicine: A Process
Patients
Appraise
Ask Questions
Searching
Validity
Results
Applicability
Outline
• Why is EBM important?
• What is EBM
• Understand foreground vs. background
questions
• Formulate PICO questions
• Find resources, do searches
• Validity of an article
• Appraise “Therapy” articles
• Appraise “Diagnostic” articles
What’s the difference
between foreground and
background questions?
Asking the Precise Question
• Background questions
• Basic aspect of a
disease
• Pathophysiology
• Etiology
• Basic treatment
• Who, what, when, how
• Foreground questions
• Specific knowledge
• Have 4 parts:
•Patient/problem
•Intervention
•Comparison
intervention
•Clinical outcomes
Background or Foreground?
• What is asthma?
• Is prednisone helpful in asthma?
• What are the newest medication for
asthma?
• Does atrovent used acutely make you
feel better?
Background or Foreground?
• What is asthma? (B)
• What are the newest medication for asthma?
(B)
• Does atrovent used acutely make you feel
better? (F)
• Is prednisone helpful in asthma? (F or B)
• Foreground if compare to other drugs
• Background if interested in how it works
Outline
• Why is EBM important?
• What is EBM
• Understand foreground vs. background
questions
• Formulate PICO questions
• Find resources, do searches
• Validity of an article
• Appraise “Therapy” articles
• Appraise “Diagnostic” articles
Structure of a Well-built Question
• Patient or population
• Be specific to capture the group you want
• Ex: Children w/ asthma
• Intervention
• Be specific
• Comparison group (if any)
• Compare to standard therapy or test
• Outcome
• Be precise
• What are the outcome of interest
Formulating the Clinical Question
PICO
Patient
In patients with
asthma discharged
home from the
emergency
department
Intervention
does the use of
both inhaled and
oral corticosteroids
Comparison
(if relevant)
compared with
oral steroids
alone
Outcome
lead to a
reduction in
asthma relapse?
Can You Identify PICO?
• In children under 6 months, how does
sleeping on back compared to sleeping on
the stomach in terms of risk of SIDS?
• In children under 6 months (P), how does
sleeping on back (I) compared to sleeping on
the stomach (C) in terms of risk of SIDS (O)?
Can You Form a PICO Question?
Clinical scenario:
5 yo with moderate persistent
asthma now in severe acute asthma
exacerbation. Intern gave 2 albuterol
and orapred with minimal
improvement. Intern asks why how
good is atrovent?
Searchable PICO Question
P: Population
I: Intervention/diagnostic test/risk
factor
C: Comparison
O: Outcome
In children with acute asthma exacerbation
(P), will the addition of atrovent (I) to
albuterol (C) decrease the rate of
hospitalization (O)?
Outline
• Why is EBM important?
• What is EBM
• Understand foreground vs. background
questions
• Formulate PICO questions
• Find resources, do searches
• Validity of an article
• Appraise “Therapy” articles
• Appraise “Diagnostic” articles
Searching Superhero
Best Bets
The Evidence Pyramid
If your question is about…
Look for a …
Intervention/Therapy
• Randomized controlled trial
Diagnosis/Screening
To assess the accuracy of the test
• Cohort study
To assess effect of test on health
outcome
• Randomized controlled trial
Prognosis
•Longitudinal cohort
Etiology/Risk factors
• Randomized controlled trial
• Cohort
• Case-control
Pre-appraised Resources
• Cochrane
– http://www.cochrane.org/cochrane/revabstr/mainindex.htm
– Very high quality reviews
– Mostly questions of therapy
• National Guideline Clearinghouse
– http://www.guideline.gov/
– Guidelines of varying levels of quality
– Do broad searches
• PEM database
– http://researchinpem.homestead.com/homepage.html
– Not pre-appraised
Pre-appraised Resources
• Best Evidence Topics
– http://www.bestbets.org
– Developed in the ED of Manchester Royal
Infirmary in UK
– Usually EM topics
– Free
• Clinical Evidence (CE Concise)
– http://www.clinicalevidence.com
– From UK
– Focus mostly on therapy
– Free
Primary Search Engines
• Pubmed
• Ovid
Primary Search Engines
• Both PubMed and Ovid
• Contains MEDLINE and other
bibliographic databases (Cochrane,
EMBASE, etc)
• Sells access to the databases to libraries
PubMed MEDLINE vs. Ovid MEDLINE
• Same DATABASE
• Different SEARCH ENGINES
• Searching differently may get you
slightly different results
• If you search both databases EXACTLY
the same, your results will be the same
What is MEDLINE?
• MEDLINE is a DATABASE
• MEDLINE is produced by the National
Library of Medicine
• MEDLINE is available free via Pubmed
• MEDLINE is sold to many vendors, like
Ovid, who search the DATA in MEDLINE
through different search engines
Database Producer
Vendors
Medline
Ovid
EMBASE
Pubmed
CINAHL
Personal PC
Dialog
Library PC
LEXIS/NEXIS
PubMed: Clinical Queries
• Pre-filtered searching
• Search on questions of
– Therapy
– Diagnosis
– Etiology
– Prognosis
Limit any of the categories to
sensitivity (broad) or specificity
(narrow)
• Or, search for Systematic Reviews
Sensitivity vs. Specificity
SPECIFICITY
• Narrower search
• More relevant items
• Possibility of losing
some useful
citations
• LESS information to
weed through
SENSITIVITY
• Broader search
• More irrelevant
items
• Less chance of
losing useful
citations
• MORE information
to weed through
OVID
• Most librarians favor this search engine
• Allow you to tailor your search
Design & Execute Search Strategy
• MeSH terms
• How articles are indexed
• Predefined categories from the National
Library of Medicine
• Explode
• MeSH + all items underneath
• Focus
• MeSH only
Design & Execute Search Strategy
• Limiters
• Limit by languages, human, age, etc
• Hedges
• “Clinical trials” for therapy
• “Sensitivity” & “Specificity” for diagnostic tests
• “Cohort studies” for prognosis
More on Searching Tips
• “Term$”
• Search engines will look for occurrences of the word
with any combination of letters following. (Ex:
“Hospital$” will include hospitalization, hospitals,
hospitalized)
• “Exp” for “explode”
• “Term.tw.”
• Allow search engine to look in the titles and abstract
More on Searching Tips
• “Term.af”
• Search all fields
• “..pg term”
• Perge/delete
For More Searching Tips
Ask your librarians!
Outline
• Why is EBM important?
• What is EBM
• Understand foreground vs. background
questions
• Formulate PICO questions
• Find resources, do searches
• Validity of an article
• Appraise “Therapy” articles
• Appraise “Diagnostic” articles
Appraise (Validity)
• http://www.cebm.utoronto.ca/teach/mate
rials/dx.htm
• http://www.cebm.utoronto.ca/teach/mate
rials/therapy.htm
Validity of Therapy Articles
• Randomized?
• Blinded?
• Were the subjects similar at the start of
the trial?
• Were all subjects treated the same
except for the treatment of interest?
• Were all patients accounted for at the
end of the trial?
Validity:
1. Were the Subjects Randomized?
“Methods” section
Validity:
2. Were They Blinded?
“Methods” section
Validity:
3. Were the Subjects Similar at
the Start of the Trial?
“Results” section
Validity:
4. Were All Subjects Treated the Same
except for the Treatment of Interest?
“Methods” section
Validity:
5. Were All Patients Accounted
for at the End of the Trial?
“Results” section
Validity of the Article
• Was the study randomized?
• Were the subjects blinded?
• Were the subjects similar at the start of
the trial?
• Were all subjects treated the same except
for the treatment of interest?
• Were all patients accounted for at the end
of the trial?
Outline
• Why is EBM important?
• What is EBM
• Understand foreground vs. background
questions
• Formulate PICO questions
• Find resources, do searches
• Validity of an article
• Appraise “Therapy” articles
• Appraise “Diagnostic” articles
Step 4: Appraise “Therapy” Study
• Therapy study
– Number Needed to Treat (NNT) to prevent an
adverse outcome
• Mathematically
– NNT = 1/ARR (absolute risk reduction)
• ARR = Risk of adverse outcome in the control
group (CER) – those in the experimental group
(EER)
– ARR* = CER** – EER***
*ARR: Absolute risk reduction
**CER: control event rate
***EER: experimental event rate
More on NNT
• Variables must be dichotomous
• Cannot calculate NNT from continuous
variables
• For NNT to be meaningful, the
difference between the 2 groups
must be significant
Let’s Calculate NNT
CER
EER
ARR
NNT
50%
20%
30%
3
3%
2.5%
0.5%
200
30%
25%
5%
20
CER: Control Event Rate
EER: Experimental Event Rate
ARR: Absolute Risk Reduction
NNT: Number Needed to Treat
NNT=1/ARR
Step 4: Appraise “Therapy” Articles
• ARR = CER – EER
• NNT = 1/ARR
• ARR = 52.6-37.5 =
15.1
• NNT = 1/15.1 x 100
=7
Step 5: Decision Making
Atrovent
– Low cost medication
– Minimal to no side effects
– Every 7 children with severe asthma
exacerbation, 1 hospitalization will be
prevented
The benefits outweigh the risks. Therefore
it would be worth treating the child with
Atrovent in the emergency department.
Outline
• Why is EBM important?
• What is EBM
• Understand foreground vs. background
questions
• Formulate PICO questions
• Find resources, do searches
• Validity of an article
• Appraise “Therapy” articles
• Appraise “Diagnostic” articles
Appraise (Validity)
• http://www.cebm.utoronto.ca/teach/mate
rials/dx.htm
• http://www.cebm.utoronto.ca/teach/mate
rials/therapy.htm
Validity of Diagnostic Articles
• Was there an independent, blind
comparison with a reference (“gold”)
standard of diagnosis?
• Was the diagnostic test evaluated in an
appropriate spectrum of patients (like
those in whom it would be used in
practice)?
Validity of Diagnostic Articles
• Was the reference standard applied
regardless of the diagnostic test
result?
• Was the test (or cluster of tests)
validated in a second, independent
group of patients?
Disease +
Disease -
Test +
A
True +
B
False +
A+B
Test -
C
False -
D
True -
C+D
A+C
B+D
Sensitivity & Specificity
• Sensitivity
The proportion of people with the target
disorder (A+C) who have a positive test (A)
A/(A+C)
• SnNout
If a test has a high sensitivity, a negative
result rules out the disease
Disease +
Disease -
Test +
A
True +
B
False +
A+B
Test -
C
False -
D
True -
C+D
A+C
B+D
Sensitivity & Specificity
• Specificity
The proportion of people without the
disease (B+D) who have a negative test (D)
D/(B+D)
• SpPin
When a test has a high specificity, a positive
result rules IN the diagnosis
Disease +
Disease -
Test +
A
True +
B
False +
A+B
Test -
C
False -
D
True -
C+D
A+C
B+D
Predictive Values
• Positive Predictive Value
Proportion of people with a positive test
(A+B) who have the target disorder
A/(A+B)
• Negative Predictive Value
Proportion of people with a negative test
(C+D) who are free of the disease (D)
D/(C+D)
Disease +
Disease -
Test +
A
True +
B
False +
A+B
Test -
C
False -
D
True -
C+D
A+C
B+D
What is Likelihood Ratio?
A likelihood that a given test result
would be expected in a patient w/
the target disorder compared w/
the likelihood that the same result
would be expected in a patient w/o
the target disorder
Likelihood Ratio
• Likelihood ratio (LR) of 1 means
 The test result identifies patients w/ the
disorder at the same rate as identifying
patients w/o the disorders
• LR >1
 The test result identifies patients w/ the
disorder at a faster rate; likelihood goes 
• LR <1
 The test result identifies patients w/ the
disorder at a slower rate; likelihood goes 
LR of 3 means:
The result is 3 times more likely to
occur in a patient w/ the target disease
than in a patient without.
In general, the LR for a test result is
the percentage of “disease +” patients
identified by that result divided by the
percentage of “disease –” patients so
identified.
Dis +
Dis -
Result A
a
b
Result B
c
d
Total
X
Y
Rate
Dis +
a/X
c/X
Rate
Dis b/Y
(a/X)
(b/Y)
d/Y
(c/X)
(d/Y)
What rate are Dis + patients identified by Result A?
What rate are Dis - patients identified by Result A?
What is the LR for Result A?
LR
Disease +
Disease -
Test +
A
True +
B
False +
Test -
C
False -
D
True -
X=A+C
Y=B+D
At what rate are Dis + patients identified by a “+ test” ? A/X
At what rate are Dis – patients identified by a “+ test”? B/Y
What is the LR for a “positive” test result? (A/X) ÷ (B/Y)
Positive Likelihood Ratio (LR +)
LR + = sensitivity/(1 – specificity)
= True pos rate/False pos rate
Disease +
Disease -
Test +
A
True +
B
False +
Test -
C
False -
D
True -
X=A+C
Y=B+D
At what rate are Dis + patients identified by a “- test” ? C/X
At what rate are Dis – patients identified by a “- test”? D/Y
What is the LR for a “negative” test result? (C/X) ÷ (D/Y)
Negative Likelihood Ratio (LR-)
LR - = (1 – sensitivity)/specificity
= False neg rate/True neg rate
Guide to Likelihood ratios
LR+
LR-
1–2
V Poor
1 – 0.5
2–5
Poor
0.5 – 0.2
5 – 10
Fair
0.2 – 0.1
10 – 20
Good
0.1 – 0.05
> 20
V. Good
< 0.05
Why Do We Care About LR?
Allow us to think about pre-test and posttest probabilities so that we can
personalize the test to our patients
Test & Treatment Thresholds in
the Diagnostic Process
Test
threshold
Treatment
threshold
0%
No test
needed
100%
Further testing
needed
Testing completed;
treatment starts
Step 1: Pre-test Probability
• Prevalence
The number of events (instances of a given
disease or other condition) in a given
population at a designated time.
It is not a rate.
It is used as pre-test probability
It is the number of “disease +” patients to the
“total”
Pre-test Probability
5 sources to determine this:





Clinical experience
Regional or national prevalence statistics
Practice databases
Original report on accuracy of the test
Studies that calculate pre-test probability
Step 3: Calculate LR
• Pretest probability is 30%
• LR+ = ?
Culture + Culture -
65
18
Test -
10
72
Total
75
90
Amir, 1994
Test +
LR + = TP rate/FP rate = (65/75)  (18/90) = 4.35
LR - = FN rate/TN rate = (10/75)  (72/90) = 0.16
Fagan Nomogram
Pre-test probability
on the left hand
column
LR in the middle
Post-test probability
on the right hand
column
Translation
• You see a 4 yo child who comes in c/o
sorethroat, no other complaints
• Your pre-test probability is that she has
a 30% chance of having strep throat
• You wonder how good is rapid strep
• You look up an article (Amir, 1994) and
find its sensitivity (86.5%) and
specificity (80%)
Translation
• Your patient rapid strep is positive
• You calculated the LR + to be 4.35
• You determined the post-test
probability to be 66%
• Is that enough of a threshold for you to
treat or would you rather do more test?
Test & Treatment Thresholds in
the Diagnostic Process
Test
threshold
Treatment
threshold
0%
No test
needed
100%
Further testing
needed
Testing completed;
treatment starts
A Different Article Showed…
• What if the LR+ is 12?
• What is the post-test probability?
“Diagnostic” Summary…
• Use the 2 x 2 table to determine
likelihood ratio (LR)
• LR + = True pos rate/False pos rate
• LR - = False neg rate/True neg rate
• Use Fagan Nomogram
Outline
• Why is EBM important?
• What is EBM
• Understand foreground vs. background
questions
• Formulate PICO questions
• Find resources, do searches
• Validity of an article
• Appraise “Therapy” articles
• Appraise “Diagnostic” articles
Knowledge is of two kinds; we know
a subject ourselves; or we know
where we can find information upon it.
- Samuel Johnson