2) Multilevel Tests (Michael)
Likelihood ratios for results other
than “+” or “-”
Four Main Points
1) Dichotomizing a multi-level test by choosing
a fixed cutpoint reduces the value of the test.
2) The ROC curve summarizes the ability of the
test to differentiate between D+ and Dindividuals.
3) LR(result) = P(result|D+)/P(result|D-) =
slope of ROC curve.
(NOTE: Do not calculate an LR(+) or LR(-) for a
multilevel test.)
4) Pre-Test Odds x LR(result) = Post-Test Odds
Septic Arthritis
Bacterial infection in a joint.
Clinical Scenario
Does this Adult Patient Have Septic Arthritis?
Clinical Scenario
Does this Adult Patient Have Septic Arthritis?
A 48-year-old woman … presents to the emergency
department with a 2-day history of a red, swollen
right knee that is painful to touch. She reports no
prior knee swelling and no recent trauma or knee
surgery …. [S]he is afebrile and has a right knee
effusion. Her peripheral white blood cell (WBC) count
is 11 000/µL and her erythrocyte sedimentation rate
(ESR) is 55 mm/h. An arthrocentesis is performed,
and the initial Gram stain is negative.
You have the synovial white blood cell (WBC) count.
Margaretten, M. E., J. Kohlwes, et al. (2007). Jama 297(13): 1478-88.
Clinical Scenario
Does this Adult Patient Have Septic Arthritis?
Assume pre-test probability of septic arthritis is 0.38.
How do you use the synovial WBC result to determine
the likelihood of septic arthritis?
Margaretten, M. E., J. Kohlwes, et al. (2007). Jama 297(13): 1478-88.
Why Not Make It a Dichotomous
Test?
Synovial
WBC Count
Septic Arthritis
Yes
No
>25,000
77%
27%
≤ 25,000
23%
73%
TOTAL*
100%
100%
*Note that these could have come from a study where the patients with septic arthritis
(D+ patients) were sampled separately from those without (D- patients).
Margaretten, M. E., J. Kohlwes, et al. (2007). Jama 297(13): 1478-88.
Why Not Make It a Dichotomous
Test?
Sensitivity = 77%
Specificity = 73%
LR(+) = 0.77/(1 - 0.73) = 2.9
LR(-) = (1 - 0.77)/0.73 = 0.32
“+” = > 25,000/uL
“-” = ≤ 25,000/uL
Clinical Scenario
Synovial WBC = 48,000/mL
Pre-test prob: 0.38
Pre-test odds: 0.38/0.62 = 0.61
LR(+) = 2.9
Post-Test Odds = Pre-Test Odds x LR(+)
= 0.61 x 2.9 = 1.75
Post-Test prob = 1.75/(1.75+1) = 0.64
Clinical Scenario
Synovial WBC = 128,000/mL
Pre-test prob: 0.38
Pre-test odds: 0.38/0.62 = 0.61
LR = 2.9 (same as for WBC=48,000!)
Post-Test Odds = Pre-Test Odds x LR(+)
= 0.61 x 2.9 = 1.75
Post-Test prob = 1.75/(1.75+1) = .64
Why Not Make It a Dichotomous
Test?
Because you lose information. The risk
associated with a synovial WBC=48,000 is
equated with the risk associated with
WBC=128,000.
Choosing a fixed cutpoint to dichotomize a
multi-level or continuous test throws away
information and reduces the value of the test.
Main Point 1: Avoid Making
Multilevel Tests Dichotomous
Dichotomizing a multi-level or continuous test
by choosing a fixed cutpoint reduces the
value of the test
WBC (/uL)
Interval
% of
% of No
Septic
Septic
Arthritis Arthritis
>100,000
29%
1%
50,001-100,000
33%
7%
25,001-50,000
15%
19%
0 - 25,000
23%
73%
TOTAL
100%
100%
80%
70%
No Septic Arthritis
60%
Septic Arthritis
50%
40%
30%
20%
10%
0%
0 - 25,000
>25,00050,000
>50,000100,000
Synovial Fluid WBC Count
>100,000
Histogram



Does not reflect prevalence of D+ (Dark D+
columns add to 100%, Open D- columns add to
100%)
Sensitivity and specificity depend on the cutpoint
chosen to separate “positives” from “negatives”
The ROC curve is drawn by serially lowering the
cutpoint from highest (most abnormal) to lowest
(least abnormal).*
* Just said that choosing a fixed cutpoint reduces the value of the test. The key issues are
1) the ROC curve is for evaluating the test, not the patient, and 2) drawing the ROC curve
requires varying the cutpoint, not choosing a fixed cutpoint.
80%
Cutoff = ∞
Sensitivity = 0%
1 - Specificity = 0%
70%
60%
50%
Negative
40%
30%
20%
10%
0%
0 - 25,000
>25,00050,000
>50,000100,000
>100,000
Positive
80%
70%
60%
Cutoff = 100,000
Sensitivity = 29%
1 - Specificity = 1%
50%
Negative
Positive
40%
30%
20%
10%
0%
0 - 25,000
>25,00050,000
>50,000100,000
>100,000
80%
Cutoff = 50,000
Sensitivity = 62%
1 - Specificity = 8%
70%
60%
50%
Negative
Positive
40%
30%
20%
10%
0%
0 - 25,000
>25,00050,000
>50,000100,000
>100,000
80%
Cutoff = 25,000
Sensitivity = 77%
1 - Specificity = 27%
70%
60%
50%
Negative
Positive
40%
30%
20%
10%
0%
0 - 25,000
>25,00050,000
>50,000100,000
>100,000
80%
Cutoff = 0
Sensitivity = 100%
1 - Specificity = 100%
70%
60%
50%
Negative
Positive
40%
30%
20%
10%
0%
0 - 25,000
>25,00050,000
>50,000100,000
>100,000
WBC Count
(x1000/uL)
>∞
> 100
> 50
> 25
≥ 0
Sensitivity 1 - Specificity
0%
29%
62%
77%
100%
0%
1%
8%
27%
100%
Margaretten, M. E., J. Kohlwes, et al. (2007). Jama 297(13): 1478-88.
100%
90%
Cutoff ≥ 0
80%
Cutoff > 25k
Sensitivity
70%
60%
Cutoff > 50k
50%
40%
30%
Cutoff > 100k
20%
10%
Cutoff > ∞
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
1 - Specificity
Test Discriminates Fairly Well
Between D+ and DD+
D-
-40
-20
Test
0 Result20
40
60
Test Discriminates Well Between
D+ and D1
Sensitivity
0.8
0.6
0.4
0.2
0
0
0.2
0.4
0.6
1 - Specificity
0.8
1
Test Discriminates Poorly
Between D+ and D-
D+
D-
-40
-20
0 Result 20
Test
40
60
Test Discriminates Poorly
Between D+ and D1
Sensitivity
0.8
0.6
0.4
0.2
0
0
0.2
0.4
0.6
1 - Specificity
0.8
1
Area Under ROC Curve
100%
90%
Cutoff ≥ 0
80%
Cutoff > 25k
Sensitivity
70%
60%
Cutoff > 50k
50%
40%
30%
Cutoff > 100k
Area Under
Curve = 0.8114
20%
10%
Cutoff > ∞
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
1 - Specificity
Area Under ROC Curve
Summary measure of test’s
discriminatory ability
Probability that a randomly chosen D+
individual will have a more positive test
result than a randomly chosen Dindividual
Main Point 2
ROC Curve Describes the Test, Not
the Patient


Describes the test’s ability to discriminate
between D+ and D- individuals
Not particularly useful in interpreting a
test result for a given patient
ROC Curve Describes the Test,
Not the Patient
Clinical Scenario
Synovial WBC count = 48,000
Synovial WBC count = 128,000
Synovial WBC count = 48,000
100%
90%
Cutoff ≥ 0
80%
Cutoff > 25k
Sensitivity
70%
60%
Cutoff > 50k
50%
40%
30%
Cutoff > 100k
20%
10%
Cutoff > ∞
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
1 - Specificity
Sensitivity, Specificity, LR(+), and LR(-) of the
Synovial Fluid WBC Count for Septic Arthritis at 3
Different Cutoffs
WBC
(/uL)
>100,000
Sensitivity Specificity
29%
LR+ LR-
99%
29.0
0.7
>50,000
62%
92%
7.8
0.4
>25,000
77%
73%
2.9
0.3
Synovial WBC Count = 48,000/uL
Which LR should we use?
Sensitivity, Specificity, LR(+), and LR(-) of the
Synovial Fluid WBC Count for Septic Arthritis at 3
Different Cutoffs
WBC
(/uL)
>100,000
Sensitivity Specificity
29%
LR+ LR-
99%
29.0
0.7
>50,000
62%
92%
7.8
0.4
>25,000
77%
73%
2.9
0.3
Synovial WBC Count = 48,000/uL
Which LR should we use? NONE of THESE!
Likelihood Ratios
P(Result) in patient WITH disease
---------------------------------------------------P(Result) in patients WITHOUT disease
LR(result) = P(result|D+)/P(result|D-)
WOWO
With Over WithOut
Interval Likelihood Ratios
The ratio of the height of the D+ distribution to the height of the D- distribution
80%
70%
No Septic Arthritis
60%
Septic Arthritis
50%
40%
LR = 15%/19% = 0.8
19%
30%
15%
20%
10%
0%
0 - 25,000
>25,00050,000
>50,000100,000
>100,000
100%
90%
> 25k
80%
Sensitivity
70%
60%
50%
> 50k
15%
Slope = 15%/19% =0.8
19%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
1 - Specificity
Interval Likelihood Ratio
WBC (/uL)
Interval
% of
D+
% of D-
Interval
LR
>100,000
29%
1%
29.0
50,001-100,000
33%
7%
4.7
25,001-50,000
15%
19%
0.8
0 - 25,000
23%
73%
0.3
Common Mistake
When given an “ROC Table,” it is tempting to
calculate an LR(+) or LR(-) as if the test were
“dichotomized” at a particular cutoff.
Example: LR(+,25,000) = 77%/27% = 2.9
This is NOT the LR of a particular result (e.g.
WBC >25,000 and ≤ 50,000); it is the LR(+)
if you divide “+” from “-” at 25,000.
Common Mistake
WBC (/uL) Sensitivity Specificity
LR+ LR-
>100,000
29%
99%
29.0
0.7
>50,000
62%
92%
7.8
0.4
>25,000
77%
73%
2.9
0.3
Common Mistake
100%
90%
Slope = 0.8
80%
> 25,000
Sensitivity
70%
60%
50%
40%
77%
30%
Slope = 77%/27% =
2.9
20%
27%
10%
0%
0%
10% 20%
30% 40%
50% 60%
1 - Specificity
70% 80%
90% 100%
Common Mistake
From JAMA paper:
“Her synovial WBC count of 48,000/µL
increases the probability from 38% to
64%.” (Used LR = 2.9)
Correct calculation:
Her synovial WBC count of 48,000/µL
decreases the probability from 38% to
33%.” (Used LR = 0.8)
Main Point 3
Likelihood Ratio
P(Result) in patients WITH disease
-----------------------------------------------------P(Result) in patients WITHOUT disease
Slope of ROC Curve
Do not calculate an LR(+) or LR(-) for a multilevel
test.
Clinical Scenario
Synovial WBC = 48,000/uL*
Pre-test prob: 0.38
Pre-test odds: 0.38/0.62 = 0.61
LR(WBC btw 25,000 and 50,000) = 0.8
Post-Test Odds = Pre-Test Odds x LR(48)
= 0.61 x 0.8 = 0.49
Post-Test prob = 0.49/(0.49+1) = 0.33
*Can use slide rule, Excel, or web page
Clinical Scenario
Synovial WBC = 128,000/uL*
Pre-test prob: 0.38
Pre-test odds: 0.38/0.62 = 0.61
LR(128,000/uL) = 29
Post-Test Odds = Pre-Test Odds x LR(128)
= 0.61 x 29 = 17.8
Post-Test prob = 17.8/(17.8+1) = 0.95
*Can use slide rule, Excel, or web page
Clinical Scenario
WBC = 48,000/uL Post-Test Prob = 0.33
WBC = 128,000/uL Post-Test Prob = 0.95
(Recall that dichotomizing the WBC with a
fixed cutpoint of 25,000/uL meant that
WBC = 48,000/uL would be treated the
same as WBC = 128,000/uL and post-test
prob = 0.64)
Main Point 4
Bayes’s Rule
Pre-Test Odds x LR(result) = Post-Test Odds
What you knew before + What you learned = What you know now
Summary
1)
2)
3)
4)
Dichotomizing a multi-level test by choosing a fixed
cutpoint reduces the value of the test.
The ROC curve summarizes the discriminatory
ability of the test.
LR(result) = P(result|D+)/P(result|D-) = Slope of
ROC Curve (NOTE: Do not calculate an LR(+) or
LR(-) for a multilevel test.)
Pre-Test Odds x LR(result) = Post-Test Odds
Funny Times: 1-888-Funnytimes x 2111
3) Getting the most out of ROC
Curves (Tom)
THE WALKING MAN OR …
… WHAT CAN YOU LEARN FROM ROC
CURVES LIKE THESE?
Bonsu, B. K. and M. B. Harper (2003). "Utility of the peripheral blood white blood cell count for
identifying sick young infants who need lumbar puncture." Ann Emerg Med 41(2): 206-14.
“Walking Man” Approach to ROC
Curves




Divide vertical axis into d steps, where d is
the number of D+ individuals
Divide horizontal axis into n steps, where n is
the number of D- individuals
Sort individuals from most to least abnormal
test result
Moving from the first individual (with the
most abnormal test result) to the last (with
the least abnormal test result)…
“Walking Man” (continued)




…call out “D” if the individual is D+ and “N” if
the individual is DLet the walking man know when you reach a
new value of the test
The walking man takes a step up every time
he hears “D” and a step to the right every
time he hears “N”
When you reach a new value of the test, he
drops a stone.
… WHAT CAN YOU LEARN FROM ROC
CURVES LIKE THESE (Diagnosing meningitis
in infants from WBC counts)?*
*Bonsu BK, Harper MB. Ann Emerg Med 2003;41:206-14
Calculating the c Statistic


Actual values of the test are not important for the
shape of the ROC curve or the area under it--only
the ranking of the values
Area under an ROC (“C statistic”) curve uses the
same information as the Wilcoxon Rank Sum (or
Mann-Whitney U) and gives identical P values.
C= (Smax – S)/(Smax – Smin) where
Smin = d(d+1)/2 and Smax = Smin+dn
Illustration: shorter stature as a test
for female gender identification



Class arranges itself in order of height
(shortest person at the head of the line)
Make a grid on flip chart, Y axis has F
steps, X axis has M steps
Class draws ROC curve:



F draw a vertical line segment
M draw horizontal line segment
Tied height: diagonal line segment
10:35-10:45 Break
10:45-12:30 Small Groups
12:30 Lunch