Studying a Study and
Testing a Test:
Sensitivity Training,
“Don’t Make a Good Test Bad”,
and “Analyze This”
Borrowed Liberally from
Riegelman and Hirsch, 2nd Edition
What you need to know:
• When will ordering a given test be helpful
in making a decision on therapy?
• When is a test more likely to be
misleading?
• Which test should I order when?
• How do I evaluate the literature regarding
the value of different tests?
Sensitivity and Specificity:
NEW
TEST
GOLD
STANDARD
Diseased
GOLD
STANDARD
Disease-free
Positive
True
Positive
False
positive
Negative
False
negative
True
negative
• Sensitivity =
TP/(TP+FN)
• Specificity =
TN/(TN+FP)
Scenario #1
• You read an article describing a new rapid
test for diagnosis of Herpes Simplex virus
infection. It was used on samples from
1000 neonatal patients with CSF
pleocytosis (>30 wbc/mm3) elevated
protein (>50), with negative gram stain,
and the results were compared with a
gold-standard test. The following results
were obtained.
Sensitivity and Specificity:
• Question 1:
What are the
Sensitivity and
Specificity of this
test?
• Question 2:
Is this a Good
Test?
NEW
TEST
GOLD
STANDARD
Diseased
GOLD
STANDARD
Disease-free
Positive
400
50
Negative
100
450
500
500
Sensitivity and Specificity:
Answer to Questions 1 & 2
• Sensitivity = True Positives Divided by
Gold Standard Diseased (True positives
plus false negatives) = 400/500 = 80%
• Specificity = True Negatives Divided by
Gold Standard Disease-free (True
negatives plus false positives = 450/500 =
90%
• Good Test?
Take Home Message #1
• Sensitivity and Specificity are Properties of
the Test!!!
Scenario #2
• The same diagnostic test becomes
commercially available and your hospital
decides that for “medicolegal reasons”, it
should be done on all CSF samples,
regardless of cell count and protein
results. The results of the next 1,000 tests
are shown on the following table.
Sensitivity and Specificity
• Question 3:
What are the
sensitivity and
specificity now?
• Question 4:
Is this a good
test?
NEW
TEST
GOLD
STANDARD
Diseased
GOLD
STANDARD
Disease-free
Positive
80
90
Negative
20
810
100
900
Answers to 3 and 4
• Sensitivity = 80/100=80%
• Specificity = 810/900=90%
• Specificity and Sensitivity do not change
when you overuse the test, but the value
of the positive result is less
• Good test?
– …. 90 of the 170 patients with positive tests
are actually disease-free…
Predictive Values
• Predictive value of
a positive test =
TP/(TP+FP)
• Predictive value of
a negative test =
TN/(TN+FN)
NEW
TEST
GOLD
STANDARD
Diseased
GOLD
STANDARD
Disease-free
Positive
80
90
Negative
20
810
100
900
Positive and Negative
Predictive Value
• Predictive value of a positive test =
Proportion of those with a positive test
who have the disease = 80/170 = 47.1%
• Predictive value of a negative test =
Proportion of those with a negative test
who are disease-free = 810/830 = 97.6%
Take Home Message #2
• The Predictive Value of a test depends
upon the prevalence of the disease in the
population in which it is applied!!
• Corollary: You can make a good test into a
bad test by using it in a population with a
very low prevalence of the disease
Scenario #3
• A new extended screening test is being
piloted for medium chain acyl-CoA
dehydrogenase deficiency (MCAD) on
newborn blood spots. The following
results are obtained:
Predictive Value
• Question 3:
What are the
sensitivity,
specificity,
positive and
negative
predictive values
now?
• Question 4:
Is this a good
screening test?
NEW
TEST
GOLD
STANDARD
Diseased
GOLD
STANDARD
Disease-free
Positive
99
1800
Negative
1
8100
100
9900
Answers to Questions 5 and 6
• Sensitivity = 99/100 = 99%
• Specificity = 8100/9900 = 82%
• Predictive value of positive = 99/1899 =
5.2%
• Predictive value of negative= 8100/8101=
99.99%
Characteristics of a Good Screen
• Very high NEGATIVE PREDICTIVE value
(implies high sensitivity)
• Availability of follow-up Gold Standard test
to confirm (may be repetition of original
test)
• Availability of counseling and education
• Intervention that affects outcome
Risks and Odds
• Sometimes you don’t know the prevalence
of the disease within a given situation, so
you need to evaluate that from the
literature
• Two questions:
– What is the risk of a given pathology in the
context of a given risk factor?
– What are the odds before and after you
assess for the presence of a risk factor?
Scenario #4: You see a febrile infant in the midst of an
aseptic meningitis outbreak, but this one has never had
Prevnar. You have access to some raw prospective data
on the use of Prevnar in your area:
Invasive
Pneumococcal
disease
No Invasive
disease
No Prevnar
30
970
Prevnar
3
997
What is the relative risk of Invasive disease in an unimmmunized patient?
Relative Risk
• Relative Risk = probability of disease with
the risk factor ÷ probability without the risk
factor
• 0.030/0.003 = 10
• BUT, this only works if the data is
generated PROSPECTIVELY
What if this data were generated RETROSPECTIVELY,
i.e., taking kids who had invasive disease and selecting
matched controls.
Invasive
Pneumococcal
disease
No Invasive
disease
No Prevnar
90
45
Prevnar
10
55
What is the odds ratio of Invasive disease in an unimmmunized patient?
Odds ratio
• Odds ratio: odds of having a risk factor in
the diseased population ÷ odds of having
the risk factor in a matched control
population
• 90/10 ÷ 45/55 = 9 ÷ 0.82 = 11
Scenario #5
• An article outlines guidelines for referral to a
pediatric cardiologist for infants and children
with cardiac murmurs. The article describes
a decision analysis methodology that allows
you to compare the effectiveness of a two
different paradigms, one in which EKG is
combined with CXR at the same time, the
other in which only abnormal quality
murmurs are sent on for Echo.
Decision Analysis:
Serial Application of Tests
Decision Analysis:
Parallel Application of Tests
Which of the following is true?
A. In a serial testing approach, the initial test must
meet most criteria of a good screening test.
B. In a parallel approach, more diagnostic value is
gained if the outcomes of the 2 tests are
independent [e.g., CXR is less useful in
asthmatics, since wheeze and atelectasis occur
together in a non-random fashion]
C. A and B are both true.
Interventional Studies
•
•
•
•
Assignment to Groups
Assessment of outcomes
Analysis
Interpretation
Assignment (Avoid Bias)
•
•
•
•
Prospective vs. Retrospective
Randomized vs. Selected
Blinded vs. Open-label
Placebo
Assessment
• Define variables prior to initiation of study
• Choose a good test!!!
Analysis
• State Hypothesis as null: “There is no
difference between treatment and control
groups”
• Type I error: Falsely reject the null
hypothesis, P value (0.05) is likelihood of
type I
• Type II error: Falsely accept the null
hypothesis, Power = 1.0 - type II error
• Power of 0.8 is standard
3 determinants of Power
• Variability of the test
• Incremental change
• Sample size
Interpretation
• What was the study population?
• Do the results apply to your population?