EBM-Diagnostic Testing
K. Mae Hla, MD, MHS
Primary Care Faculty
Development Fellowship
November 13, 2010
Objectives
•
•
•
•
•
Develop pre-test probabilities
Derive treatment thresholds
Appraise evidence about a diagnostic test
-validity, accuracy and applicability
Calculate the results of diagnostic tests
-sensitivity, specificity and likelihood ratios
Apply evidence to patient care decisions
The Diagnostic Process
•
•
•
•
Working diagnosis- pretest probability
With each new finding/test we move
from the pre-test probability to a new
post-test probability
Clinicians estimate probability of
disease using probabilistic, prognostic
and pragmatic approaches
Compare disease probabilities to two
thresholds
Applying Diagnostic Tests
Example #1
8-year-old with fever, sore throat,
swollen cervical glands and tonsillar
exudates. No h/o cough.
You order a rapid strep test.
What’s your pretest probability of the
patient having group A strep
pharyngitis?
How much of a change would help you
decide to treat, not treat or test further?
Treatment Thresholds
No Tx
ZONE OF UNCERTAINTY
5%
Tx
90%
100%
0%
Probability of Strep
Pharyngitis
Rapid Strep Test Results
Rapid Strep test result comes back
negative
How does the rapid strep test result
change the probability of the patient
having or not having the disease?
A positive rapid strep test raises post
test probability of strep pharyngitis to
85% in one study
A negative strep test decreases
probability to 12 %
Pre-test Prob = 40%
LR+ = 7.2
LR- =
0.24
Treatment Thresholds
No Tx
ZONE OF UNCERTAINTY
Tx
X
0%
5%
12%
Probability of strep
when rapid strep test
is negative
90%
100%
Example #2
18-year-old female with ankle pain after a
roller-blading accident. States unable to walk
on her injured ankle. Exam demonstrates a
slightly swollen ankle but no tenderness
noted. Able to bear weight and take 4 full
steps upon encouragement.
What is the probability of ankle fracture?
Do you need to order an ankle x-ray?
Ottowa Ankle rule
An ankle x-ray is only necessary if there
is pain near the malleoli and any of the
following findings are present:
inability to bear wt. both
immediately and in the ED
bone tenderness at the posterior
edge or tip of the malleolus
How accurate is the Ottowa
ankle rule in ruling out ankle
fracture?
A prospective validation study in > 1000
pts presenting to the ED with ankle pain
Likelihood ratio + = 1.96
Likelihood ratio - = 0
Pre-test Prob = 10%
LR+ = 1.96
LR- =
0
Case 1
•
•
•
75-year-old woman with a hemoglobin
of 10, MCV was 80 on routine
checkup, a negative history and
physical except osteoarthritis, and on
no meds likely to suppress her marrow
or cause a bleed
Her probability of iron deficiency was
50%
You want to avoid doing a bone
marrow and order serum ferritin to
diagnose iron deficiency anemia
Case 1
•
•
•
•
•
•
P: In an elderly symptomless woman with
mild anemia
I: how useful is serum ferritin
C:
O: in diagnosing iron deficiency anemia
T(ype of question): Diagnosis
T(ype of study): Prospective Cohort
*Diagnosis of Iron Deficiency Anemia in the
Elderly (Guyatt, et al. Am J Med,
1990;(88):205-209
Three Main Questions
•
Validity-Is this evidence about the
accuracy of a diagnostic test valid?
•
Results-Does this evidence show that
this test can adequately distinguish
patients who do and do not have the
disorder?
•
Applicability-How can I apply this
valid, accurate diagnostic test to a
specific patient?
Validity
•
Measurement: was the gold standard
measured independently?
•
Representative: was the test
evaluated in appropriate spectrum of
patients?
•
Ascertainment: was the reference test
ascertained regardless of the
diagnostic test?
Validity- Measurement
•
All patients in study should have both the
diagnostic test in question (blood test, history,
physical exam) and the gold/reference standard
test (autopsy, bone marrow, biopsy, angiogram)
•
Independent- test not part of gold standard,
decision to perform gold standard should not
depend on result of diagnostic test under study
•
Blinding- reference test readers should be
unaware of results to avoid bias if tests/gold
standard have subjective component- x-rays,
biopsy, slides
Validity: Measurement
•
Was there an independent blind
comparison with a reference gold
standard?
• The gold standard was the bone marrow
aspirate results
• All patients got the serum ferritin and
bone marrow done independently
• Marrow aspirates and iron deficiency
status was determined by 2 hematologists
unaware of the lab result
Validity: Representative
•
Was the diagnostic test evaluated in
an appropriate spectrum of patients?
• Examples: risk markers such as CEA
were initially done in high risk patients
Validity: Representative
•
•
•
•
Diagnostic uncertainty
Patients with mild as well as severe
symptoms
Patients with early as well as late
disease
Patients with other commonly
confused diagnoses
Study spectrum representative?
•
•
•
•
•
Consecutive patients age 65 or older with
anemia were recruited
36% of patients had iron deficiency anemia
44% had anemia of chronic disease
8% megaloblastic anemia
Patients with other commonly confused
disorders- different types of anemia and
chronic medical conditions were included
Validity: Ascertainment
•
Was the reference standard
ascertained regardless of the
diagnostic test result?
•
Did all patients in the study both with
and without iron deficiency anemia get
the bone marrow done?
•
Yes
Ascertainment-Continued
•
Patients with negative diagnostic test
may not get the gold standard done if
the latter is invasive
•
How do we prove for sure that the
ones with negative tests truly do not
have the disease or vice versa?
•
Other ways to establish reference test
•
In the Pioped study looking at the
utility of V/Q scan in patients with
suspected pulmonary embolism-all
patients with negative V/Q scan did not
get pulmonary angiogram
•
Clinical followup in a year was the
additional reference standard to not
miss patients with false negative VQ
results
Results
•
Does the test accurately distinguish
between patients with and without the
disorder?
– Sensitivity and specificity
– Likelihood ratios
T
e
s
t
Positive
Negative
Disease
Present Absent
a
b
a+b
c
d
a+c
b+d
c+d
T
e
s
t
Disease
Present Absent
a
b
Positive
TP
FP
c
d
Negative
FN
TN
a+c
b+d
a+b
c+d
T
e
s
t
Positive
Negative
Disease
Present Absent
a
b
TP
FP
c
d
FN
a+c
TN
b+d
Sen=a/a+c Sp=d/d+b
a+b
c+d
T
e
s
t
Positive
Negative
Disease
Present
Absent
a
b
TP
FP
c
FN
a+c
d
a+b
c+d
TN
b+d
Sen=a/a+c “PID”
Sp=d/d+b “NIH”
Sensitive test-rules out the
disease (SnNout)
•
•
•
•
Test with high sensitivity (high TPR and
very low false negative rate), negative test
rules out the disease
Examples:
loss of retinal vein pulsation in increased
intracranial pressure-the presence of
pulsation (negative test) rules out IIP
HIV antibody- negative test rules out HIV
Specific test – rules in the
disease (SpPIN)
•
•
•
•
Test with high specificity (high TNR, very
low FPR)-positive test rules in the diagnosis
Features of child with Down’s syndromevery specific
Presence of features (positive test) rules in
the diagnosis
Western blot confirmatory testing for HIVhigh specificity: positive test rules in HIV
disease
Likelihood Ratio
LR =
likelihood of the test result in patients with the disease
likelihood of the same result in patients without disease
T
e
s
t
Positive
Negative
Disease
Present Absent
a
b
TP
FP
c
d
FN
a+c
TN
b+d
a+b
c+d
Sen=a/a+c Sp=d/d+b
(+)LR= + test result in pts with dz
+ test result in pts without dz
(-)LR= - test result in pts with dz
- test result in pts without dz
T
e
s
t
Positive
Negative
Disease
Present Absent
a
b
TP
FP
c
d
FN
a+c
TN
b+d
a+b
c+d
Sen=a/a+c Sp=d/d+b
(+)LR= + test result in pts with dz
+ test result in pts without dz
= Sn/1-Sp
(-)LR= - test result in pts with dz
- test result in pts without dz
= 1-Sn/Sp
Likelihood Ratio
LR =
probability of the test result in patients with the disease
probability of the same result in patients without disease
Pre-Test
Probability
Pre-Test
Odds
Odds = Probability/1-probability
Post-Test
Odds
Post-Test
Probability
Probability = Odds/1 + Odds
What do all these numbers
mean?!?
•
•
•
•
L.R.s indicate by how much a given diagnostic
test result will raise or lower the pre-test
probability of the target disorder
L.R. of 1 = post-test probability is same as pretest probability
L.R. > 1 increases the probability that the
target disorder is present; the higher the L.R.,
the greater the increase
L.R. < 1 decreases the probability of the target
disorder; the smaller the L.R., the greater the
decrease
Effects of different
likelihood ratios
•
•
•
>10 or <0.1 generate large and often
conclusive changes from pre- to posttest probability
5-10 and 0.1-0.2 generate moderate
shifts in pre- to post-test probability
Depending on pre-test probability,
change may or may not be large
enough to influence Rx decision
Back to our patient


Our patient’s serum ferritin comes
back at 40 mmol/L
How should we put all this together?
T
e
s
t
Positive
<45
Negative
>45
Iron Deficiency Anemia
Present Absent
70
15
a b
c d
15
135
a+c b+d
Totals
85
150
a+b+c+d
235
Low ferritin (<45) in
diagnosing Fe def anemia
Prevalence (study pre-test probability)
= 85/235= 36%
Sensitivity = True positive / all with disease
= a/a+c
= 70/85
= 82%
Specificity = True neg / all without disease
= d/b+d
= 135/150
= 90%
Low ferritin (<45) in
diagnosing iron deficiency
anemia
L.R.+
= sensitivity/(1-specificity)
= 82%/10% = 8.2
L.R. -
= (1-sens)/spec
= 18%/90% = 0.2
Simplifying Likelihood
Ratio Calculations
Bone Marrow:
iron deficient
Bone Marrow:
normal iron
< 45
> 46
70
15
15
135
Totals
85
150
Test Results:
Calculating Likelihood Ratios
at 45 cut point
Bone Marrow: Bone Marrow:
iron deficient normal iron
Likelihood
Ratios
< 45
70
70/85=0.824
15
15/150=0.1
0.824/0.1=
8.24
> 46
15
15/85=0.176
135
135/150=0.9
0.176/0.9=
0.196
Totals
85
150
Test Results:
Pre-test Prob = 36%
LR+ = 8.2
LR- =
0.2
Applying the Test
to the Patient
•
•
Is the diagnostic test available,
affordable, accurate and precise in our
setting?
Yes
Applicability (cont’d)
•
•
•
Test needs to be available, affordable
Interpreted in competent, reproducible
fashion in clinical setting
Potential consequences should justify
the cost
Applicability (Cont’d)
•
•
•
Are the study patients similar to our
own?
Are the results applicable to the patient
in my practice?
Will the patient be better off as a result
of the test?
Applicability-study
patients’ characteristics
•
•
•
•
•
•
•
•
235/259 patients had interpretable aspirates
Mean age 79.7, 46% men, 72% had no
medical diagnosis other than anemia
Early dementia 25
CHF 25
COPD 25
Rheumatoid arthritis 17
Osteoarthritis 14
Pneumonia 13
Can we generate a clinically
sensible estimate of our
patient’s pre-test probability?
How can we estimate pre-test probability?
•
Clinical experience
Regional or national prevalence statistics
Practice databases
Pretest probability observed in the study itself
•
Studies of pre-test probabilities
•
•
•
Risk of PE
Low = 3.6%
(2-6)
Inter = 20%
(17-24)
High = 67%
(54-77)

Wells PS, Anderson DR, Rodger M, et al. Derivation of a simple clinical model to
categorize patients' probability of pulmonary embolism: increasing the model's
utility with the SimpliRED D-dimer. Thromb Haemost 2000;83:416-420.
Will the post-test probabilities
affect our management and
help our patient?
•
•
•
Could the test result move us across a
test-treatment threshold?
Would the patient be willing to undergo
the test?
Would it help the patient?
Pre-test Prob = 50%
LR+ = 8.2
LR- =
0.2
Treatment Thresholds
No Tx
ZONE OF UNCERTAINTY
Tx
x
10%
90%
100%
0%
90% Probability of
Fe def Anemia when
Ferritin is <45
Likelihood Ratios for 4
levels of Serum Ferritin
Ferritin
Fe def #
Not Fe def L.R.
<18
47
2
41.47
>18<45
23
13
3.12
>45<100
7
27
0.46
>100
8
108
0.13
Total
85
150
Calculating Likelihood Ratios
Bone Marrow: Bone Marrow: Likelihood
normal iron
Ratios
iron deficient
Test Results:
< 18
19-45
46-100
>100
Totals
47
47/85=0.553
23
23/85=0.271
7
7/85=0.082
8
8/85=0.094
85
2
2/150=0.013
13
13/150=0.087
27
27/150=0.18
108
108/150=0.72
150
0.553/0.013=
42.5
0.271/0.087=
3.11
0.082/0.18=
0.456
0.094/0.72=
0.131
Clinical Scenario





52 y.o. male admitted to Orthopedics 3 days
ago for a R femur fracture after falling from a
ladder
Underwent ORIF 2 days ago
Last night developed SOB
No CP or cough
PE: Afeb 115/70 HR 110 RR 18 95% on 4L
– Otherwise unremarkable (Lungs clear, No
elevated JVP or RV heave, no lower ext swelling)


Labs: EKG sinus tach, CXR clear
D-dimer 0.5 mcg/mL (nl <0.6)
PICO
•
•
•
•
•
•
P: In a patient with acute onset of SOB,
hypoxia and sinus tachycardia after a major
orthopedic surgery for femur fracture
I: Does a negative D-dimer result
C:
O: Rule out pulmonary embolism
T(ype of question): Diagnosis
T(ype of study): Prospective Cohort
*De Monye W et al: The performance of two rapid d-dimer
assays in 287 patients with clinically suspected pulmonary
embolism. Thrombosis Res 2002;(107):283-286.
Risk of PE
Low = 3.6%
(2-6)
Inter = 20%
(17-24)
High = 67%
(54-77)

Wells PS, Anderson DR, Rodger M, et al. Derivation of a simple clinical model to
categorize patients' probability of pulmonary embolism: increasing the model's
utility with the SimpliRED D-dimer. Thromb Haemost 2000;83:416-420.
Treatment Thresholds
No Tx
ZONE OF UNCERTAINTY
Tx
X
5%
0%
20%
Probability of Pulm
Embolism
90%
100%
Sensitivity and Specificity
disease (+ ) disease (-)
T est (+ )
74
77
T est (-)
16
120
Sensitivity (“Positive in disease”) = true pos / all disease =
74 / 90 = 82.2%
Specificity (“Negative in health”) = true neg/all disease free
= 120 / 197 = 60.3%
Likelihood Ratios of D-Dimer Test
disease (+ ) disease (-)
T est (+ )
74
77
T est (-)
16
120
Likelihood ratio (+) = (74 / 90) / (77/197) = 2.1
Likelihood ratio (-) = (16/90) / (120/197) = 0.29
Pre-test Prob = 20%
LR+ = 2.1
LR- = 0.29
Treatment Thresholds
No Tx
ZONE OF UNCERTAINTY
Tx
X
5%
0%
5%
Probability of Pulm
Embolism if
D-Dimer is negative
90%
100%
LR (using lower cut off 95%
sensitive D-Dimer value)
disease (+ ) disease (-)
LR =
T est (+ )
86
149
T est (-)
4
48
probability of the test result in patients with the disease
probability of the same result in patients without disease
Likelihood ratio + test =
(86/90) / (149/197) = 1.3
Likelihood ratio - test =
(4/90) / (48/197) = 0.18
Pre-test Prob = 20%
LR+ = 1.3
LR- = 0.18
Treatment Thresholds
No Tx
ZONE OF UNCERTAINTY
Tx
x
3%
0%
90%
100%
3% Probability of
PE if D-Dimer is
negative
Summary
•
•
•
•
Develop pre-test probabilities
Steps in appraising a diagnostic test
Calculate and interpret sensitivity,
specificity and likelihood ratios
Evaluate how pre-test probability and
likelihood ratio results affect post-test
probability of disease to influence
further testing or treatment decisions