Performance of a
diagnostic test
Steen Ethelberg
17th EPIET Introductory Course
Lazareto, Menorca, Spain
•Thierry Ancelle
September 2011
•Marta Valenciano
1
Outline
1. Performance characteristics of a test
– Sensitivity
– Specificity
– Choice of a threshold.
2. Performance of a test in a population
– Positive predictive value of a test (PPV)
– Negative predictive value of a test (NPV)
– Impact of disease prevalence, sensitivity and
specificity on predictive values.
2
1. Performance of a test
in an experimental setting
3
Population with ill and non-ill
individuals
4
Test should identify the ill only
5
In reality, tests are not perfect
6
Sensitivity & specificity
Se =
Identified patients
All patients
Se =
Sp =
18
= 90%
20
Identified non-patients
All non-patients
Sp =
78
= 97.5%
80
7
Sensitivity of a test
• Ability of a test to correctly identify affected
individuals
• Proportion of people testing positive
among affected individuals
Patients
Test
+
－
True positive (TP)
False negative (FN)
Sensitivity (Se) = TP / ( TP + FN )
8
Sensitivity of a PCR
for congenital toxoplasmosis
Patients with toxoplasmosis
Rapid test
True positive
False negative
54
4
58
Sensitivity = 54 / 58 = 0.931= 93.1 %
9
Specificity of a test
• Ability of test to identify correctly non-affected
individuals
• Proportion of people testing negative among nonaffected individuals
Non-affected people
Test
+
－
False positive (FP)
True negative (TN)
Specificity (Sp) = TN / ( TN + FP )
10
Specificity of a PCR
for congenital toxoplasmosis
Individuals without toxoplasmosis
Rapid test
False positive
True negative
11
114
125
Specificity = 114 / 125 =0.912 = 91.2 %
11
Performance of a test
Yes
+
Disease
No
TP
FP
FN
TN
Test
Se =
TP
TP + FN
Sp =
TN
TN + FP
12
Distribution of quantitative test results
among affected and non-affected people
Number of people tested
Ideal situation
Threshold for
positive result
TN
0
Non affected:
Affected:
TP
5
10
15
Quantitative result of the test
20
13
Distribution of quantitative results
among affected and non-affected people
More realistic situation
Non-affected:
Number of people tested
Threshold for
positive result
TN
FN
0
5
Affected:
TP
FP
10
Quantitative result of the test
15
20
14
Effect of Decreasing the
Threshold
Non affected:
Number of people tested
Threshold for
positive result
Affected:
FP
TP
TN
FN
0
5
10
Quantitative result of the test
15
20
15
Effect of Decreasing the
Threshold
Yes
+
Test
Se =
Disease
No
TP
FP
FN
TN
TP
TP + FN
Sp =
TN
TN + FP
16
Effect of Increasing the Threshold
Number of people tested
Threshold for
positive result
TN
Non-affected:
Affected:
TP
FN
FP
0
5
10
Quantitative result of the test
15
20
17
Effect of Increasing the Threshold
Yes
+
Disease
No
TP
FP
FN
TN
Test
Se =
TP
TP + FN
Sp =
TN
TN + FP
18
Performance of a test and
threshold
• Sensitivity and specificity vary in
opposite directions when changing the
threshold (e.g. the cut-off in an ELISA)
• The choice of a threshold is a
compromise to best reach the
objectives of the test
– consequences of having false positives?
– consequences of having false negatives?
19
When false diagnosis is worse
than missed diagnosis
• Example: Screening for congenital
toxoplasmosis
– One should minimise false positives
– Prioritise SPECIFICITY
20
When missed diagnosis
is worse than false diagnosis
• Example: Testing for Helicobacter pylori
infection
– One should minimise the false negatives
– Prioritise SENSITIVITY
21
Using several tests
• One way out of the dilemma is to use
several tests that complement each
other
• First use test with a high sensitivity
• Second use test with a high specificity
22
ROC curves
• Representation of relationship
between sensitivity and specificity for a
test
• Receiver Operating Characteristics curve
• Simple tool to:
– Help define best cut-off value of a test
– Compare performance of two tests.
23
Prevention of Blood Transfusion Malaria:
Choice of an Indirect IF Threshold
Sensitivity (%)
100
80
1/80
1/40
1/160
60
IIF Dilutions
1/320
40
1/640
20
0
1/20 1/10
0
20
40
60
80
100
100% - Specificity (%)
24
Comparison of Performance of ELISA and CATT
Test for Screening of Human Trypanosomiasis
Sensitivity (%)
100
80
ELISA
CATT
60
40
20
0
0
25
50
75
100
100 - Specificity (%)
25
Comparison of Performance of ELISA and CATT
Test for Screening of Human Trypanosomiasis
Sensitivity (%)
100
80
ELISA
CATT
60
Area under the ROC curve (AUC)
40
20
0
0
25
50
75
100
100 - Specificity (%)
26
Performance of a test
• Validity
– Sensitivity
– Specificity
• Reproducibility
• Concepts may also used more broadly
– Exposure status
– Case definitions
27
2. Performance of a test in a
population
28
Would also like to know…
• As a clinician
– probability that a individual with a positive test
is really sick?
– probability that a individual with a negative test
is really healthy?
• As an epidemiologist
– proportion of positive tests corresponding
to true patients?
– proportion of negative tests corresponding
to healthy subjects?
29
Predictive values
PPV=
Real patients
Positive patients
PPV=
NPV=
15
= 86%
22
Real non-patients
Negative patients
NPV=
75
= 94%
80
30
Positive Predictive Value
• Probability that an individual testing
positive is truly affected
– proportion of affected people among
those testing positive
Disease
Yes
No
Test
+
TP
FP
PPV = TP/(TP+FP)
31
Negative Predictive Value
• Probability that an individual testing
negative is truly non-affected
– proportion of non affected among
those testing negative
Disease
Yes
No
Test
FN
TN
NPV = TN/(TN+FN)
32
Predictive value of a positive
and a negative test
Disease
Yes
No
+
TP
FP
PPV = TP/(TP+FP)
FN
TN
NPV = TN/(TN+FN)
Test
PPV = VPP = PV+
NPV = VPN = PV33
Predicted values are not
constants
• The predicted values depend on the
sensitivity and on the specificity of the test
as well as on the prevalence of the
disease
• Will be different in different populations.
34
Relation between predictive values
and sensitivity / specificity
Disease
Yes
No
+
TP
FP
PPV = TP/(TP+FP)
FN
TN
NPV = TN/(TN+FN)
Test
35
Step 1: Specify the prevalence (Pr) of disease
Disease
Yes
No
+
Test
Pr
1-Pr
36
Step 2: Use sensitivity (Se) to distribute test
results among the diseased
Disease
Yes
No
+
Se Pr
Test
(1-Se)Pr
Pr
1-Pr
37
Step 3: Use specificity (Sp) to distribute test
results among the non-diseased
Disease
Yes
No
+
Se Pr
(1-Sp)(1-Pr)
(1-Se)Pr
Sp(1-Pr)
Pr
1-Pr
Test
38
Step 4: Determine the proportion testing
positive and the proportion testing negative
Disease
Yes
No
+
Se Pr
(1-Sp)(1-Pr)
(1-Se)Pr
Sp(1-Pr)
Pr
1-Pr
Se Pr + (1-Sp)(1-Pr)
Test
(1-Se)Pr+ Sp(1-Pr)
39
Step 5: Calculate PPV and NPV with
appropriate expressions from Step 4
Se Pr
PPV =
Se Pr + (1 - Sp)(1 - Pr)
Sp(1 - Pr)
NPV =
Sp(1 - Pr) + (1 - Se) Pr
40
Relation between predictive values
and sensitivity / specificity
Se Pr
PPV =
Se Pr + (1 - Sp)(1 - Pr)
Increasing specificity  increasing PPV
Sp(1 - Pr)
NPV =
Sp(1 - Pr) + (1 - Se) Pr
Increasing sensitivity  increasing NPV
41
Relation between predictive values
and prevalence
Se Pr
PPV =
Se Pr + (1 - Sp)(1 - Pr)
Increasing prevalence  increasing PPV
Sp(1 - Pr)
NPV =
Sp(1 - Pr) + (1 - Se) Pr
Decreasing prevalence  increasing NPV
42
PPV and NPV of a test according to the
prevalence (80% sensitivity and specificity)
Predictive value (%)
100
80
NPV
60
40
20
PPV
0
0
25
50
75
100
Prevalence (%)
43
Example: Two different populations,
Se=Sp=90%
Prevalence: 50%
 PPV = 90%
Ill
Test
+
Not ill
TP
FP
FN
TN
Prevalence: 10%
 PPV = 50%
44
Example: Screening for human
trypanosomiasis in two settings
• CATT test
– Sensitivity = 95%
– Specificity = 75%
• Endemic area
– Prevalence = 20%
• Low endemic area
– Prevalence = 0.5%
• 100,000 tests performed in each area
45
Example: Screening for human
trypanosomiasis in two settings
Prevalence = 20%
CATT test sensitivity = 95%
CATT test specificity = 75%
Trypanosomiasis
+
CATT
Yes
No
Total
19,000
20,000
39,000
1,000
60,000
61,000
20,000
80,000
100,000
PPV = 48.7%
NPV = 98.4%
46
Example: Screening for human
trypanosomiasis in two settings
CATT test sensitivity = 95%
CATT test specificity = 75%
Prevalence = 0.5%
Trypanosomiasis
+
CATT
Yes
No
Total
475
24,875
25,350
25
74,625
74,650
500
99,500
100,000
PPV = 1.90%
NPV = 98.97%
47
To sum up…
• Sensitivity and specificity
– intrinsic characteristics of a test
• capacity to identify the affected
• capacity to identify the non-affected
– independent from the disease prevalence
• Predictive values
– performance of a test in real life
• how to interpret a positive test
• how to interpret a negative test
– dependent on the disease prevalence
48
Thank you!
QUESTIONS?
49