Performance of a
diagnostic test
Dagmar Rimek
EPIET-EUPHEM Introductory Course 2012
Lazareto, Menorca, Spain
Based on the Lecture of 2011
by Steen Ethelberg
1
Outline
1. Performance characteristics of a test
– Sensitivity
– Specificity
– Choice of a threshold
2. Performance of a test in a population
– Positive predictive value of a test (PPV)
– Negative predictive value of a test (NPV)
– Impact of disease prevalence, sensitivity and
specificity on predictive values
2
1. Performance characteristics
of a test in a laboratory
setting
3
Population with affected and non-affected
individuals
Affected
Non-affected
4
A perfect diagnostic test identifies the
affected individuals only
Affected
Non-affected
5
In reality, tests are not perfect
Affected
Non-affected
6
Sensitivity of a test
The sensitivity of a test is the ability of the test
to identify correctly the affected individuals
Proportion of persons testing positive among affected individuals
Affected persons
Test result
+
-
True positive (TP)
False negative (FN)
Sensitivity (Se) = TP / (TP + FN)
7
Estimating the sensitivity of a test
• Identify affected individuals with a gold standard
• Obtain a wide panel of samples that are representative
of the population of affected individuals
– Recent and old cases
– Severe and mild cases
– Various ages and sexes
• Test the affected individuals
• Estimate the proportion of affected individuals that are
positive with the test
8
Example: Estimating the sensitivity of
a new ELISA IgM test for acute Q-fever
• Identify persons with acute Q-fever with a gold
standard (IgM Immunofluorescence Assay)
• Obtain a wide panel of samples that are representative
of the population of individuals with acute Q-fever
– Recent and old cases
– Severe and asymptomatic cases
– Various ages and sexes
• Test the persons with acute Q-fever
• Estimate the proportion of persons with acute Q-fever
that are positive with the ELISA IgM test
9
Example: Sensitivity a new ELISA IgM test
for acute Q-fever
Patients with acute Q-fever
ELISA IgM test result
+
True positive (TP)
-
False negative (FN)
148
2
150
Sensitivity =
TP / (TP + FN)
148 / 150 = 98.7%
10
What factors influence the sensitivity
of a test?
• Characteristics of the affected persons?
 YES: Antigenic characteristics of the pathogen in the area
(e.g., if the test was not prepared with antigens reflecting the
population of pathogens in the area, it will not pick up
infected persons in the area)
• Characteristics of the non-affected persons?
 NO: The sensitivity is estimated on a population of affected
persons
• Prevalence of the disease?
 NO: The sensitivity is estimated on a population of affected
persons
Sensitivity is an INTRINSIC characteristic of the test
11
Specificity of a test
The specificity of a test is the ability of the test
to identify correctly non-affected individuals
Proportion of persons testing negative among non-affected
individuals
Non-affected persons
Test result
+
-
False positive (FP)
True negative (TN)
Specificity (Sp) = TN / (TN + FP)
12
Estimating the specificity of a test
• Identify non-affected individuals
– Negative with a gold standard
– Unlikely to be infected
• Obtain a wide panel of samples that are representative
of the population of non-affected individuals
• Test the non-affected individuals
• Estimate the proportion of non-affected individuals
that are negative with the test
13
Example: Estimating the specificity of
a new ELISA IgM test for acute Q-fever
• Identify persons without Q-fever
– Persons without sign and symptoms of the infection
– Persons at low risk of infection, negative with gold standard
(IgM Immunofluorescence Assay)
• Obtain a wide panel of samples that are
representative of the population of individuals without
Q-fever
• Test the persons without Q-fever
• Estimate the proportion of persons without Q-fever
that are negative with the new ELISA IgM test
14
Specificity of a new ELISA IgM test
for acute Q-fever
Persons without acute Q-fever
ELISA IgM test result
+
False positive (FP)
10
-
True negative (TN)
190
200
Specificity =
TN / (TN + FP)
190 / 200 = 95%
15
What factors influence
the specificity of a test?
• Characteristics of the affected persons?
 NO: The specificity is estimated on a population of nonaffected persons
• Characteristics of the non-affected persons?
 YES: The diversity of antibodies to various other antigens in
the population may affect cross reactivity or polyclonal
hypergammaglobulinemia may increase the proportion of false
positives
• Prevalence of the disease?
 NO: The specificity is estimated on a population of nonaffected persons
Specificity is an INTRINSIC characteristic of the test
16
Performance of a test
Disease
Yes
No
+
TP
FP
-
FN
TN
Test
Se =
TP
TP + FN
Sp =
TN
TN + FP
17
To whom sensitivity and specificity
matters most?
INTRINSIC characteristics of the test
► To laboratory specialists!
18
Distribution of quantitative test results
among affected and non-affected people
Number of people tested
Ideal situation
Threshold for
positive result
TN
0
Non-affected:
Affected:
TP
5
10
15
20
Quantitative result of the test
19
Distribution of quantitative results
among affected and non-affected people
Number of people tested
Realistic situation
TN
FN
0
5
Non-affected:
Threshold for
positive result
Affected:
TP
FP
10
Quantitative result of the test
15
20
20
Effect of Decreasing the Threshold
Non-affected:
Number of people tested
Threshold for
positive result
Affected:
FP
TP
TN
FN
0
5
10
Quantitative result of the test
15
20
21
Effect of Decreasing the Threshold
Yes
Disease
No
+
TP
FP
-
FN
TN
Test
Se =
TP
TP + FN
Sp =
TN
TN + FP
22
Effect of Increasing the Threshold
Number of people tested
Threshold for
positive result
TN
Non-affected:
Affected:
TP
FN
FP
0
5
10
Quantitative result of the test
15
20
23
Effect of Increasing the Threshold
Yes
Disease
No
+
TP
FP
-
FN
TN
Test
Se =
TP
TP + FN
Sp =
TN
TN + FP
24
Performance of a test and threshold
• Sensitivity and specificity vary in opposite
directions when changing the threshold
(e.g. the cut-off in an ELISA)
• The choice of a threshold is a compromise
to best reach the objectives of the test
– consequences of having false negatives?
– consequences of having false positives?
25
Using several tests
• One way out of the dilemma is to use
several tests that complement each other
• First use test with a high sensitivity
(e.g. screening for HIV by ELISA, or for
syphilis by TPHA)
• Second use test with a high specificity
(e.g. confirmation of HIV or syphilis by
western blot)
26
ROC curves
• Receiver Operating Characteristics curve
• Representation of relationship
between sensitivity and specificity for a test
• Simple tool to:
– Help define best cut-off value of a test
– Compare performance of two tests
27
Prevention of blood transfusion malaria:
Choice of an indirect IFA threshold
Sensitivity (%)
100
80
1/80
1/40
1/160
60
IFA Dilutions
1/320
40
1/640
20
0
1/20 1/10
0
20
40
60
80
100
100 - Specificity (%): Proportion of false positives
28
Comparison of performance of IFA and ELISA
IgM tests for detection of acute Q-fever
Sensitivity (%)
100
80
IFA
ELISA
60
40
Area under the ROC curve (AUC)
20
0
0
25
50
75
100
100 - Specificity (%)
29
2. Performance of a test in a
population
30
How well does the test perform in a real
population?
• The test is now used in a real population
• This population is made of
– Affected individuals
– Non-affected individuals
• The proportion of affected individuals is the prevalence
Status of persons
Test
Affected
Non-affected
Positive
True +
False +
A+B
Negative
False -
True -
C+D
A+C
B+D
A+C+B+D
31
Predictive value of a positive test
The predictive value of a positive test is the
probability that an individual testing positive is
truly affected
Proportion of affected persons among those testing
positive
32
Positive predictive value (PPV) of a test
Test
Positive
Negative
Status of persons
Affected
Nonaffected
A
B
C
A+C
D
B+D
A+B
C+D
A+C+B+D
PPV = A / (A+B)
This is only valid for the sample of specimens tested
33
What factors influence
the positive predictive value of a test?
Status of persons
• Sensitivity?
 YES: To some extend.
Test
Affected
Nonaffected
Positive
A
B
A+B
Negative
C
D
C+D
• Specificity?
A+C
B+D
A+C+B+D
 YES: The more the test is specific, the more it will be negative
for non-affected persons (less false-positive results).
• Prevalence of the disease?
 YES: Low prevalence: Low pre-test probability for positives.
The test will pick up more false positives.
 YES: High prevalence: High pre-test probability for positives.
The test will pick up more true positives.
34
Positive predictive value of a test
according to prevalence and specificity
Specificity
0
10
90
80
70
60
50
40
30
70%
80%
90%
95%
20
0
10
100
90
80
VP % 70
P
PPV (%) 60
50
40
30
20
10
0
Prevalence (% )
35
Predictive value of a negative test
The predictive value of a negative test is the
probability that an individual testing negative
is truly non-affected
Proportion of non-affected persons among those testing
negative
36
Negative predictive value (NPV) of a test
Test
Positive
Negative
Status of persons
Affected
Nonaffected
A
B
C
A+C
D
B+D
A+B
C+D
A+C+B+D
NPV = D / (C+D)
This is only valid for the sample of specimens tested
37
What factors influence
the negative predictive value of a test?
Status of persons
Affected
Nonaffected
• Sensitivity?
Positive
A
B
A+B
Test
 YES:
Negative
C
D
C+D
A+C
B+D
A+C+B+D
The more the test is
sensitive, the more it captures affected persons (less false
negatives).
• Specificity?
 YES: But to a lesser extend.
• Prevalence of the disease?
 YES: Low prevalence: High pre-test probability for negatives.
The test will pick up more true negatives.
 YES: High prevalence: Low pre-test probability for negatives.
The test will pick up more false negatives.
38
Negative predictive value of a test
according to prevalence and sensitivity
Sensitivity
10
0
90
80
70
60
50
40
30
20
70%
80%
90%
95%
10
0
100
90
80
70
60
PVN
NPV
(%) % 50
40
30
20
10
0
Prevale nce (%)
39
Relation between predictive values and
sensitivity (Se), specificity (Sp), prevalence (Pr)
Disease
Yes
No
+
Se Pr
(1-Sp)(1-Pr)
Se Pr + (1-Sp)(1-Pr)
-
(1-Se)Pr
Sp(1-Pr)
(1-Se)Pr + Sp(1-Pr)
Test
Pr
1-Pr
40
Calculate PPV and NPV
Se Pr
PPV =
Se Pr + (1 - Sp)(1 - Pr)
Sp(1 - Pr)
NPV =
Sp(1 - Pr) + (1 - Se) Pr
41
Relation between predictive values
and sensitivity / specificity
Se Pr
PPV =
Se Pr + (1 - Sp)(1 - Pr)
Increasing specificity  increasing PPV
Sp(1 - Pr)
NPV =
Sp(1 - Pr) + (1 - Se) Pr
Increasing sensitivity  increasing NPV
42
Relation between predictive values
and prevalence
Se Pr
PPV =
Se Pr + (1 - Sp)(1 - Pr)
Increasing prevalence  increasing PPV
Sp(1 - Pr)
NPV =
Sp(1 - Pr) + (1 - Se) Pr
Decreasing prevalence  increasing NPV
43
Example: Screening for acute Q-fever in
two settings
• ELISA IgM test
– Sensitivity = 98%
– Specificity = 95%
• Population in low endemic area
– Prevalence = 0.5%
• Patients with atypical pneumonia
– Prevalence = 20%
• 10,000 tests performed in each group
44
Example: Screening for acute Q-fever in a
population in a low endemic area
IgM ELISA test sensitivity = 98%
IgM ELISA test specificity = 95%
Prevalence = 0.5%
Q-fever
IgM ELISA
Yes
No
Total
+
49
497
546
-
1
9,453
9,454
50
9,950
10,000
PPV = 8.97%
NPV = 99.98%
45
Example: Screening for acute Q-fever in
patients with atypical pneumonia
Prevalence = 20%
IgM ELISA test sensitivity = 98%
IgM ELISA test specificity = 95%
Q-fever
IgM ELISA
Yes
No
Total
+
1,960
400
2,360
-
40
7,600
7,640
2,000
8,000
10,000
PPV = 83.05%
NPV = 99.48%
46
To whom predictive values matters most?
• Look at denominators!
– Persons testing positive
– Persons testing negative
► To clinicians
– probability that a individual with a positive test is really sick?
– probability that a individual with a negative test is really healthy?
► To epidemiologists!
– proportion of positive tests corresponding to true patients?
– proportion of negative tests corresponding to healthy subjects?
47
Summary
• Sensitivity and specificity matter to laboratory specialists
– Studied on panels of positives and negatives
– Intrinsic characteristics of a test
• Capacity to identify the affected
• Capacity to identify the non-affected
• Predictive values matter to clinicians and epidemiologists
– Studied on homogeneous populations
– Dependent on the disease prevalence
– Performance of a test in real life
• How to interpret a positive test
• How to interpret a negative test
48
Where will you do your rain dance?
There?
Here?