Critical appraisal of a diagnostic paper
Shakila Thangaratinam
Professor of Maternal and Perinatal Health
Women’s Health Research Unit
R & D Director of Women’s Health
Barts Health NHS Trust
Relation of diagnosis and therapy in clinical
practice
Healthy and diseased population
Diagnosis
Diseased population
Therapy
Clinical outcome
A hierarchical model for evaluation of
tests
• Assessment of the reliability and other
technical aspects of a test
• Assessment of the diagnostic accuracy
• Assessment of the diagnostic effectiveness and
cost effectiveness
Fryback et al. Med Decis Making 1991;11:88-94
How does a diagnostic test help?
A test is useful if it changes our ability
to predict whether the patient has the
disease or not
• The likelihood of disease before the test
should be altered by the test result
• The likelihood of disease after the test
should be different compared to the
prevalence of disease in the population
Value of a test
Likelihood of
disease before test
TEST
RESULT
Prevalence of disease
+
(e.g. 7-11% of pregnant
women have preterm
delivery )
Likelihood of
disease after test
Should raise the
likelihood to ~100%
—
Should lower the
likelihood to ~ 0%
How to do systematic reviews
Formulate clear clinical questions from our
knowledge needs identified in patient encounters
Search the literature to identify relevant articles
Critically appraise the evidence for its validity and
usefulness
Synthesis the evidence
Implement useful findings in clinical practice
Critical Appraisal of the Medical
Literature
• Are the results of the study valid?
• What are the results?
• Will the results help in caring for patients?
Oxman et al. JAMA 1993;270:2093-5
A test accuracy study
Study Sample
Test
Test
positive
Test
Negative
Gold standard
Disease
present
Gold standard
Disease
absent
Disease
present
Disease
absent
Are the results valid?
• Was there an independent “blind” comparison
with a reference standard?
• Did the patient sample include an appropriate
spectrum of patients to whom the test will be
applied in clinical practice?
• Did the results of the test influence the decision
to perform the reference standard?
Are the results valid?
Study Sample
Appropriate spectrum of disease
Consecuitive or random sample
Prospectively recruited
Test
Test
Negative
Test
positive
Gold standard
Gold standard
Independent
“Blind”
Verification of all test positive cases
Independent
“Blind”
Verification of all test negative cases
Disease
present
Disease
absent
Disease
present
Disease
absent
Hierarchy of evidence for test accuracy
studies
1 An independent, blind comparison with reference standard among an
appropriate population of consecutive patients.
2 An independent, blind comparison with reference standard among an
appropriate population of non-consecutive patients or confined to a
narrow population of study patients.
3 An independent, blind comparison among an appropriate population of
patients, but reference standard not applied to all study patients.
4 Non-blind comparison or reference standard not applied independently
5 Expert opinion with no explicit critical appraisal, based on physiology,
bench research or first principles
Critical Appraisal of the Medical
Literature
• Are the results of the study valid?
• What are the results?
• Will the results help in caring for patients?
Oxman et al. JAMA 1993;270:2093-5
What are the results?
What is the “benefit” or “added value” of
test as a diagnostic aid? Can we measure
this added benefit?
• Are clinically meaningful measures of diagnostic
accuracy provided?
• Are the data necessary to calculate the diagnostic
accuracy measures provided?
Consequences of testing
Study Sample
Test
Test positive
Gold standard
Disease present
(True positive)
Test Negative
Gold standard
Disease absent
(False postive)
Disease present
(False negative)
Disease absent
(True negative)
Measures of test accuracy
T
e
s
t
Positive
Negative
Disease
Present Absent
TP
FP
FN
TN
• Sensitivity and Specificity
• Predictive values
• Likelihood ratios
• Diagnostic odds ratio
Sensitivity
Sensitivity
Sensitivity is the proportion of those people who really have
the disease (TP+FN) who are correctly identified as such (TP)
•
Sensitivity = TP/(TP+FN)
Disease
Positive
Present
Absent
TP
FP
FN
TN
Test
Negative
Sensitivity
Sensitivity is the proportion of those people who really have
the disease (TP+FN) who are correctly identified as such (TP)
•
Sensitivity = TP/(TP+FN)
Disease
Positive
Present
Absent
TP
FP
FN
TN
Test
Negative
Specificity
Specificity
Specificity is the proportion of those people who really do not
have the disease (TN+FP) who are correctly identified as such (TN)
•
Specificity = TN/(TN+FP)
Disease
Positive
Present
Absent
TP
FP
FN
TN
Test
Negative
Specificity
Specificity is the proportion of those people who really do not
have the disease (TN+FP) who are correctly identified as such (TN)
•
Specificity = TN/(TN+FP)
Disease
Positive
Present
Absent
TP
FP
FN
TN
Test
Negative
Predictive Values
PPV
Positive Predictive Value is the proportion of the people who test
positive (TP+FP) who truly have the disease (TP)
•
Positive predictive value = TP/(TP+FP)
Disease
Positive
Present
Absent
TP
FP
FN
TN
Test
Negative
PPV
Positive Predictive Value is the proportion of the people who test
positive (TP+FP) who truly have the disease (TP)
•
Positive predictive value = TP/(TP+FP)
Disease
Positive
Present
Absent
TP
FP
FN
TN
Test
Negative
NPV
Negative Predictive Value is the proportion of the people who test
negative (TN+FN) who truly do not have the disease (TN)
Negative predictive value = TN/(TN+FN)
Disease
Positive
Present
Absent
TP
FP
FN
TN
Test
Negative
NPV
Negative Predictive Value is the proportion of the people who test
negative (TN+FN) who truly do not have the disease (TN)
Negative predictive value = TN/(TN+FN)
Disease
Positive
Present
Absent
TP
FP
FN
TN
Test
Negative
Problems
• Sensitivity and specificity are
characteristics of the test
• Predictive values are dependent on
the prevalence of the disease
• Our population is often quite
different from the study population
The systematic review process
Formulate
research
question
Further selection of
primary studies
using inclusion
criteria
Extract data
Design
search
strategy
Search
bibliographic
databases
Retrieve papers
Identify possible
papers from
titles/abstracts
Quality
appraisal
Synthesis
Formulate research /
policy conclusions