Cochrane Diagnostic test accuracy reviews
Introduction to meta-analysis
Jon Deeks and Yemisi Takwoingi
Public Health, Epidemiology and Biostatistics
University of Birmingham, UK

 Analysis of a single study
 Approach to data synthesis
 Investigating heterogeneity
 Test comparisons
 RevMan 5

 What proportion of those with the disease does the test detect? (sensitivity)
 What proportion of those without the disease get negative test results?
(specificity)
 Requires 2 × 2 table of index test vs reference standard

Index test
Disease
(Reference test)
Present Absent
+
TP
-
FN
FP
TN
TP+FN FP+TN
TP+FP
FN+TN
TP+FP+
FN+TN sensitivity
TP / (TP+FN) specificity
TN / (TN+FP)

Heterogeneity in threshold within a study diagnostic threshold specificity=100% non-diseased sensitivity=100% diseased
0 40 80 test measurement
120 160

Heterogeneity in threshold within a study diagnostic threshold specificity=99% non-diseased sensitivity=69% diseased
0
TN FN FP TP
40 80 test measurement
120 160

Heterogeneity in threshold within a study diagnostic threshold specificity=98% non-diseased sensitivity=84% diseased
0
TN FN FP TP
40 80 test measurement
120 160

Heterogeneity in threshold within a study diagnostic threshold specificity=93% non-diseased sensitivity=93% diseased
0
TN FNFP TP
40 80 test measurement
120 160

Heterogeneity in threshold within a study diagnostic threshold specificity=84% non-diseased sensitivity=98% diseased
0
TN FN FP TP
40 80 test measurement
120 160

Heterogeneity in threshold within a study diagnostic threshold specificity=69% non-diseased sensitivity=99% diseased
0
TN FN FP TP
40 80 test measurement
120 160

Increasing threshold decreases sensitivity but increases specificity
Threshold Sensitivity Specificity
65 0.99
0.69
70 0.98
0.84
75
80
85
0.93
0.84
0.69
0.93
0.98
0.99
Decreasing threshold decreases specificity but increases sensitivity
1.0
0.8
0.6
0.4
specificity
0.2
0.0

Ex.1 Distributions of measurements and ROC plot no difference, same spread non-diseased diseased
0 40 80 test measurement
120
Uninformative test
1.0
0.8
0.6
0.4
specificity
0.2
0.0

Ex.2 Distributions of measurements and ROC plot small difference, same spread non-diseased diseased
0 40 80 test measurement
120 line of symmetry
1.0
0.8
0.6
0.4
specificity
0.2
0.0

Ratio of the odds of positivity in the diseased to the odds of positivity in the non-diseased
Diagnostic OR

TP

TN
FP

FN
DOR


 sensitivit
1
 y sensitivit y


 

1 specificit specificit y y



LR
 ve
LR
 ve

Sensitivity
Specificity 50% 60% 70% 80% 90% 95% 99%
50%
60%
70%
80%
90%
95%
99%
2
4
9
19
99
1
2
2
2
2
4
4
6
9
14
19
29
99
149
4
6
14
5
9
21
9
16
36
21
36
81
44
76
171
231
396
891
29 44 76 171 361 1881
149 231 396 891 1881 9801

(361)
(81)
(16)
(5)
(2)
(1) uninformative test
As DOR increases, the ROC curve moves closer to its ideal position near the upper-left corner.
line of symmetry
1.0
0.8
0.6 0.4 specificity
0.2
0.0

LOW DOR non-diseased diseased
HIGH
DOR
0 40 80 test measurement
120 1.0
0.8
0.6
0.4
specificity
0.2
0.0
ROC curve is asymmetric when test accuracy varies with threshold

 There are two summary statistics for each study – sensitivity and specificity – each have different implications
 Heterogeneity is the norm – substantial variation in sensitivity and specificity are noted in most reviews
 Threshold effects induce correlations between sensitivity and specificity and often seem to be present
 Thresholds can vary between studies
 The same threshold can imply different sensitivities and specificities in different groups

1.0
0.8
0.6
0.4
specificity
0.2
 Current statistical methods use a single estimate of sensitivity and specificity for each study
 Estimate the underlying ROC curve based on studies analysing different thresholds
 Analyses at specified threshold
 Estimate summary sensitivity and summary specificity
0.0
 Compare ROC curves between tests
 Allows comparison unrestricted to a particular threshold

ROC curve transformation to linear plot
 Calculate the logits of TPR and FPR
 Plot their difference against their sum

Moses-Littenberg SROC method
 Regression models used to fit straight lines to model relationship between test accuracy and test threshold
D = a + bS



Outcome variable D is the difference in the logits
Explanatory variable S is the sum of the logits
Ordinary or weighted regression – weighted by sample size or by inverse variance of the log of the DOR
 What do the axes mean?


Difference in logits is the log of the DOR
Sum of the logits is a marker of diagnostic threshold

Producing summary ROC curves
 Transform back to the ROC dimensions
 where ‘a’ is the intercept, ‘b’ is the slope
 when the ROC curve is symmetrical, b=0 and the equation is simpler

Example: MRI for suspected deep vein thrombosis
Study
Fraser 2003
Fraser 2002
Sica 2001
Jensen 2001
Catalano 1997
Larcom 1996
Laissy 1996
Evans 1996
Spritzer 1993
Evans 1993
Carpenter 1993
Vukov 1991
Pope 1991
Erdman 1990
TP
15
16
26
9
27
4
9
27
20
49
4
0
34
4
FP FN
2
3
0
5
3
0
0
0
4
3
1
2
1
4
0
0
0
1
0
1
0
3
3
6
0
6
0
3
TN Sensitivity Specificity
34
45
1.00 [0.83, 1.00]
0.94 [0.84, 0.99]
3
18
8
0.57 [0.18, 0.90]
0.00 [0.00, 0.46]
1.00 [0.90, 1.00]
191 0.40 [0.12, 0.74]
6
43
1.00 [0.78, 1.00]
0.94 [0.71, 1.00]
26
52
1.00 [0.87, 1.00]
1.00 [0.66, 1.00]
71
5
1.00 [0.87, 1.00]
0.80 [0.28, 0.99]
8
6
1.00 [0.66, 1.00]
0.90 [0.73, 0.98]
0.97 [0.85, 1.00]
0.92 [0.80, 0.98]
0.43 [0.10, 0.82]
0.86 [0.64, 0.97]
0.89 [0.52, 1.00]
0.99 [0.96, 1.00]
1.00 [0.54, 1.00]
0.90 [0.77, 0.97]
0.93 [0.76, 0.99]
0.95 [0.85, 0.99]
0.96 [0.89, 0.99]
1.00 [0.48, 1.00]
1.00 [0.63, 1.00]
1.00 [0.54, 1.00]
Sensitivity Specificity
0 0.2
0.4
0.6
0.8
1 0 0.2
0.4
0.6
0.8
1
Sampson et al. Eur Radiol (2007) 17: 175–181

SROC regression: MRI for suspected deep vein thrombosis weighted unweighted
1.0
0.8
0.6
0.4
specificity
0.2
0.0
-5 -4 -3 -2 -1
S
Linear transformation
0 1 2
Transformation linearizes relationship between accuracy and threshold so that linear regression can be used
3

SROC regression: MRI for suspected deep vein thrombosis weighted unweighted
-5 -4 -3 -2 -1
S
0 1 2 3
Inverse transformation
The SROC curve is produced by using the estimates of a and b to compute the expected sensitivity (tpr) across a range of values for 1-specificity (fpr)

SROC regression: MRI for suspected deep vein thrombosis a

4 .
721 , b

0 .
697
TPR

1

1 weighted e
4 .
721 
1

0 .
697

1




1

FPR
FPR 


1

0 .
697
1

0 .
697
-5 -4 -3 -2 -1
S
0 1 2 3
1.0
Inverse transformation
0.8
0.6
0.4
specificity
0.2
0.0
The SROC curve is produced by using the estimates of a and b to compute the expected sensitivity (tpr) across a range of values for 1-specificity (fpr)

SROC regression: MRI for suspected deep vein thrombosis weighted unweighted
-5 -4 -3 -2 -1
S
0 1 2 3
1.0
Inverse transformation
0.8
0.6
0.4
specificity
0.2
0.0
The SROC curve is produced by using the estimates of a and b to compute the expected sensitivity (tpr) across a range of values for 1-specificity (fpr)

Problems with the Moses-Littenberg
SROC method
 Poor estimation
 Tends to underestimate test accuracy due to zero-cell corrections and bias in weights

Problems with the Moses-Littenberg SROC method: effect of zero-cell correction
1.0
0.8
0.6
0.4
specificity
0.2
0.0

Problems with the Moses-Littenberg SROC method: effect of zero-cell correction
1.0
0.8
0.6
0.4
specificity
0.2
0.0

Problems with the Moses-Littenberg
SROC method



Poor estimation
 Tends to underestimate test accuracy due to zero-cell corrections and bias in weights
Validity of significance tests


Sampling variability in individual studies not properly taken into account
P-values and confidence intervals erroneous
Operating points
 knowing average sensitivity/specificity is important but cannot be obtained
 Sensitivity for a given specificity can be estimated




Hierarchical / multi-level
 allows for both within (sampling error) and
 between study variability (through inclusion of random effects)
Logistic
 correctly models sampling uncertainty in the true positive proportion and the false positive proportion
 no zero cell adjustments needed
Regression models
 used to investigate sources of heterogeneity

33

(12 studies)
1.0
0.8
Terasawa et al 2004
0.6
0.4
specificity
0.2
0.0

 Why do results differ between studies?

I.
II.
III.
IV.
V.
Chance variation
Differences in (implicit) threshold
Bias
Clinical subgroups
Unexplained variation

0.4
0.2
0.0
1.0
0.8
0.6
Chance variability: total sample size=40
Chance variability: total sample size=100
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
Specificity
0.2
0.0
1.0
0.8
0.6
0.4
Specificity
0.2
0.0

Investigating heterogeneity in test accuracy
 May be investigated by:
– sensitivity analyses
– subgroup analyses or
– including covariates in the modelling

Example: Anti-CCP for rheumatoid arthritis by
CCP generation (37 studies)
(Nishimura et al. 2007)

Anti-CCP for rheumatoid arthritis by CCP generation: SROC plot
1.0
0.8
Generation 1
0.6
0.4
specificity
Generation 2
0.2
0.0

(24 studies, 89,047 women)
1
0.6
0.5
0.4
0.3
0.9
0.8
0.7
0.2
0.1
0
1 0.9
0.8
0.7
0.6
0.5
Specificity
0.4
0.3
0.2
0.1
0

0.6
0.5
0.4
0.3
0.9
0.8
0.7
0.2
0.1
0
1
1
( = all ages; =aged 35 and over)
0.9
0.8
0.7
0.6
0.5
Specificity
0.4
0.3
0.2
0.1
0

Test +ve
(high risk)
Test -ve
(low risk)
Participants recruited
Down's Normal
Test +ve
(high risk)
Test -ve
(low risk)
50
50
250
4750
100 5000
Participants recruited
Down's Normal
50
50
250
4750
100 5000
AMNIO
AMNIO
AMNIO
BIRTH
16 lost
(33%)
Participants analysed
Down's Normal
50
50
100 5000
Participants analysed
Down's Normal
50
34
84
250
4750
250
4513
4763
Sensitivity = 50%
Specificity = 95%
Follow-up = 100%
Sensitivity = 60%
Specificity = 95%
Follow-up = 95%
237 lost
(5%)

0.6
0.5
0.4
0.3
0.9
0.8
0.7
0.2
0.1
0
1
1
( = all ages; =aged 35 and over)
0.9
0.8
0.7
0.6
0.5
Specificity
0.4
0.3
0.2
0.1
0

0.6
0.5
0.4
0.3
0.9
0.8
0.7
0.2
0.1
0
1
1
( = all ages; =aged 35 and over)
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Specificity
= all verified by amniocentesis
0.1
0

 Validity of covariate information
 poor reporting on design features
 Population characteristics
 information missing or crudely available
 Lack of power
 small number of contrasting studies

The same approach used to investigate heterogeneity can be used to compare the accuracy of alternative tests

75 HRP-2 studies and 19 pLDH studies

10 comparative studies

 Some systematic reviews pool all available studies that have assessed the performance of one or more of the tests.
 Can lead to bias due to confounding arising from heterogeneity among studies in terms of design, study quality, setting, etc
 Adjusting for potential confounders is often not feasible
 Restricting analysis to studies that evaluated both tests in the same patients, or randomized patients to receive each test, removes the need to adjust for confounders.
 Covariates can be examined to assess whether the relative performance of the tests varies systematically (effect modification)
 For truly paired studies, the cross classification of tests results within disease groups is generally not reported

 Different approach due to bivariate correlated data
 Moses & Littenberg method is a simple technique
 useful for exploratory analysis
 included directly in RevMan
 should not be used for inference
 Mixed models are recommended
 Bivariate random effects model
 Hierarchical summary ROC (HSROC) model

RevMan DTA tutorial included in version 5.1
Handbook chapters and other resources available at: http://srdta.cochrane.org