Meta-analysis with missing data:
metamiss
Ian White and Julian Higgins
MRC Biostatistics Unit, Cambridge, UK
Stata users’ group, London
10 September 2007
1
Motivation
• Missing outcome data compromise trials
• So they also compromise meta-analyses
• We may want to
– correct for bias due to missing data
– down-weight trials with more missing data
• NB missing data within trials, not missing
trials
2
Plan
Meta-analysis of binary data
• Haloperidol example
• Standard approaches to missing data
• Imputation methods
• IMORs
• Methods that allow for uncertainty
• Demonstration
3
Haloperidol meta-analysis
Haloperidol
r=successes
f=failures
m=missing
n=total
Placebo
r1
f1
m1
n1
r2
f2
m2
n2
%
missing
Arvanitis
25
25
2
52
18
33
0
51
2%
Beasley
29
18
22
69
20
14
34
68
41%
Bechelli
12
17
1
30
2
28
1
31
3%
Borison
3
9
0
12
0
12
0
12
0%
Chouinard
10
11
0
21
3
19
0
22
0%
Durost
11
8
0
19
1
14
0
15
0%
Garry
7
18
1
26
4
21
1
26
4%
Howard
8
9
0
17
3
10
0
13
0%
Marder
19
45
2
66
14
50
2
66
3%
Nishikawa 82
1
9
0
10
0
10
0
10
0%
Nishikawa 84
11
23
3
37
0
13
0
13
6%
Reschke
20
9
0
29
2
9
0
11
0%
Selman
17
1
11
29
7
4
18
29
50%
Serafetinides
4
10
0
14
0
13
1
14
4%
Simpson
2
14
0
16
0
7
1
8
4%
Spencer
11
1
0
12
1
11
0
12
0%
Vichaiya
9
20
1
30
0
29
1
30
3%
4
Standard approaches to missing
data
• Available cases (complete cases): ignore the
missing data
– assumes MAR: missingness is independent of
outcome given arm
• Assume missing=failure
– implausible, but not too bad for health-related
behaviours
• Neither assumption is likely to be correct
5
Other ideas
• Sensitivity analyses, e.g. do both
missing=failure and available cases
– but these could agree by chance
• Explore best / worst cases
• Use reasons for missingness
• Explicit assumptions about informative
missingness (IM)
– IM: missingness is dependent on outcome
6
metamiss.ado
• Processes data on successes, failures and missing by arm &
feeds results to metan
• Available cases analysis (ACA)
• Imputed case analyses (ICA):
–
–
–
–
–
–
–
impute as failure: ICA-0
impute as success: ICA-1
best-case: ICA-b (missing=success in E, failure in C)
worst-case: ICA-w
impute with same probability as in control arm: ICA-pC
impute with same probability as in experimental arm: ICA-pE
impute with same probability as in own arm: ICA-p (agrees with
ACA)
– impute using IMORs: ICA-IMOR (see next slide)
7
More general imputation: IMORs
• Measure Informative Missingness using the
Informative Missing Odds Ratio (IMOR):
– Odds ratio between outcome and missingness
• Can’t estimate IMOR from the data, but given any
value of IMOR, we can analyse the data
• Generalises other ideas: e.g.
–
–
–
–
–
ICA-0 uses IMORs 0, 0
ICA-1 uses IMORs , 
ICA-b uses IMORs , 0
ICA-p uses IMORs 1, 1
ICA-pC uses IMORs OR, 1 where OR is odds ratio
between arm and outcome in available cases
8
Getting standard errors (weighting)
right
• Weight 1: treat imputed data as real
• Weight 2: use standard errors from ACA
• Weight 3: scale imputed data to same sample size
as available cases
• Weight 4: algebraic standard errors
–
–
–
–
same as weight 1 for ICA-0, ICA-1, ICA-b, ICA-w
same as weight 2 for ICA-p
uses Taylor expansion for ICA-IMOR
for ICA-pC & ICA-pE, we condition on the IMOR (I
can explain…)
9
10
11
ACA
%
Study ID
ES (95% CI)
Weight
Arvantis
1.42 (0.89, 2.25)
18.86
Beasley
1.05 (0.73, 1.50)
31.22
Bechelli
6.21 (1.52, 25.35)
2.05
Borison_92
7.00 (0.40, 122.44)
0.49
Chouinard
3.49 (1.11, 10.95)
3.10
Durost
8.68 (1.26, 59.95)
1.09
Garry
1.75 (0.58, 5.24)
3.37
Howard
2.04 (0.67, 6.21)
3.27
Mander
1.36 (0.75, 2.47)
11.37
Nishikawa_82
3.00 (0.14, 65.90)
0.42
Nishikawa_84
9.20 (0.58, 145.76)
0.53
Reschke
3.79 (1.06, 13.60)
2.48
Selman
1.48 (0.94, 2.35)
19.11
Serafetinides
8.40 (0.50, 142.27)
0.51
Simpson
2.35 (0.13, 43.53)
0.48
Spencer
11.00 (1.67, 72.40)
1.14
Vichaiya
19.00 (1.16, 311.96)
0.52
Overall (I-squared = 41.4%, p = 0.038)
1.57 (1.28, 1.92)
100.00
.1
1
10
100
12
ICA-0
%
Study ID
ES (95% CI)
Weight
Arvantis
1.36 (0.85, 2.17)
24.38
Beasley
1.43 (0.90, 2.27)
25.01
Bechelli
6.20 (1.51, 25.40)
2.67
Borison_92
7.00 (0.40, 122.44)
0.65
Chouinard
3.49 (1.11, 10.95)
4.06
Durost
8.68 (1.26, 59.95)
1.42
Garry
1.75 (0.58, 5.27)
4.38
Howard
2.04 (0.67, 6.21)
4.29
Mander
1.36 (0.74, 2.47)
14.75
Nishikawa_82
3.00 (0.14, 65.90)
0.56
Nishikawa_84
8.47 (0.53, 134.46)
0.70
Reschke
3.79 (1.06, 13.60)
3.26
Selman
2.43 (1.19, 4.96)
10.42
Serafetinides
9.00 (0.53, 152.93)
0.66
Simpson
2.65 (0.14, 49.42)
0.62
Spencer
11.00 (1.67, 72.40)
1.50
Vichaiya
19.00 (1.16, 312.42)
0.68
Overall (I-squared = 25.8%, p = 0.158)
1.90 (1.51, 2.39)
100.00
.1
1
10
100
13
ICA-1
%
Study ID
ES (95% CI)
Weight
Arvantis
1.47 (0.93, 2.32)
5.95
Beasley
0.93 (0.77, 1.12)
35.81
Bechelli
4.48 (1.42, 14.15)
0.93
Borison_92
7.00 (0.40, 122.44)
0.15
Chouinard
3.49 (1.11, 10.95)
0.94
Durost
8.68 (1.26, 59.95)
0.33
Garry
1.60 (0.60, 4.25)
1.29
Howard
2.04 (0.67, 6.21)
0.99
Mander
1.31 (0.75, 2.28)
4.01
Nishikawa_82
3.00 (0.14, 65.90)
0.13
Nishikawa_84
10.68 (0.68, 167.43)
0.16
Reschke
3.79 (1.06, 13.60)
0.75
Selman
1.12 (0.95, 1.32)
47.38
Serafetinides
4.00 (0.51, 31.46)
0.29
Simpson
1.00 (0.11, 9.44)
0.24
Spencer
11.00 (1.67, 72.40)
0.35
Vichaiya
10.00 (1.36, 73.33)
0.31
Overall (I-squared = 60.3%, p = 0.001)
1.16 (1.04, 1.29)
100.00
.1
1
10
100
14
ICA-B
%
Study ID
ES (95% CI)
Weight
Arvantis
1.47 (0.93, 2.32)
22.59
Beasley
2.51 (1.69, 3.73)
30.05
Bechelli
6.72 (1.65, 27.28)
2.37
Borison_92
7.00 (0.40, 122.44)
0.57
Chouinard
3.49 (1.11, 10.95)
3.57
Durost
8.68 (1.26, 59.95)
1.25
Garry
2.00 (0.69, 5.83)
4.07
Howard
2.04 (0.67, 6.21)
3.76
Mander
1.50 (0.84, 2.69)
13.68
Nishikawa_82
3.00 (0.14, 65.90)
0.49
Nishikawa_84
10.68 (0.68, 167.43)
0.62
Reschke
3.79 (1.06, 13.60)
2.86
Selman
4.00 (2.09, 7.65)
11.08
Serafetinides
9.00 (0.53, 152.93)
0.58
Simpson
2.65 (0.14, 49.42)
0.54
Spencer
11.00 (1.67, 72.40)
1.31
Vichaiya
21.00 (1.29, 342.93)
0.60
Overall (I-squared = 25.9%, p = 0.156)
2.42 (1.95, 3.00)
100.00
.1
1
10
100
15
ICA-W
%
Study ID
ES (95% CI)
Weight
Arvantis
1.36 (0.85, 2.17)
13.99
Beasley
0.53 (0.39, 0.72)
33.33
Bechelli
4.13 (1.29, 13.20)
2.26
Borison_92
7.00 (0.40, 122.44)
0.37
Chouinard
3.49 (1.11, 10.95)
2.33
Durost
8.68 (1.26, 59.95)
0.82
Garry
1.40 (0.51, 3.85)
2.98
Howard
2.04 (0.67, 6.21)
2.46
Mander
1.19 (0.67, 2.10)
9.35
Nishikawa_82
3.00 (0.14, 65.90)
0.32
Nishikawa_84
8.47 (0.53, 134.46)
0.40
Reschke
3.79 (1.06, 13.60)
1.87
Selman
0.68 (0.48, 0.95)
26.57
Serafetinides
4.00 (0.51, 31.46)
0.72
Simpson
1.00 (0.11, 9.44)
0.60
Spencer
11.00 (1.67, 72.40)
0.86
Vichaiya
9.00 (1.21, 66.70)
0.76
Overall (I-squared = 74.2%, p = 0.000)
0.94 (0.79, 1.12)
100.00
.1
1
10
100
16
ICA-pc
%
Study ID
ES (95% CI)
Weight
Arvantis
1.40 (0.88, 2.23)
19.89
Beasley
1.03 (0.72, 1.49)
32.56
Bechelli
6.03 (1.47, 24.68)
2.17
Borison_92
7.00 (0.40, 122.44)
0.53
Chouinard
3.49 (1.11, 10.95)
3.29
Durost
8.68 (1.26, 59.95)
1.15
Garry
1.72 (0.57, 5.16)
3.57
Howard
2.04 (0.67, 6.21)
3.47
Mander
1.35 (0.74, 2.45)
12.05
Nishikawa_82
3.00 (0.14, 65.90)
0.45
Nishikawa_84
8.55 (0.54, 135.71)
0.56
Reschke
3.79 (1.06, 13.60)
2.64
Selman
1.30 (0.76, 2.23)
14.85
Serafetinides
8.40 (0.50, 142.27)
0.54
Simpson
2.35 (0.13, 43.53)
0.51
Spencer
11.00 (1.67, 72.40)
1.21
Vichaiya
18.42 (1.12, 302.65)
0.55
Overall (I-squared = 42.0%, p = 0.035)
1.53 (1.24, 1.88)
100.00
.1
1
10
100
17
ICA-p
%
Study ID
ES (95% CI)
Weight
Arvantis
1.42 (0.89, 2.25)
18.86
Beasley
1.05 (0.73, 1.50)
31.22
Bechelli
6.21 (1.52, 25.35)
2.05
Borison_92
7.00 (0.40, 122.44)
0.49
Chouinard
3.49 (1.11, 10.95)
3.10
Durost
8.68 (1.26, 59.95)
1.09
Garry
1.75 (0.58, 5.24)
3.37
Howard
2.04 (0.67, 6.21)
3.27
Mander
1.36 (0.75, 2.47)
11.37
Nishikawa_82
3.00 (0.14, 65.90)
0.42
Nishikawa_84
9.20 (0.58, 145.76)
0.53
Reschke
3.79 (1.06, 13.60)
2.48
Selman
1.48 (0.94, 2.35)
19.11
Serafetinides
8.40 (0.50, 142.27)
0.51
Simpson
2.35 (0.13, 43.53)
0.48
Spencer
11.00 (1.67, 72.40)
1.14
Vichaiya
19.00 (1.16, 311.96)
0.52
Overall (I-squared = 41.4%, p = 0.038)
1.57 (1.28, 1.92)
100.00
.1
1
10
100
18
ICA-IMOR 2 2
%
Study ID
ES (95% CI)
Weight
Arvantis
1.43 (0.90, 2.28)
14.67
Beasley
1.00 (0.74, 1.35)
35.19
Bechelli
6.12 (1.51, 24.87)
1.59
Borison_92
7.00 (0.40, 122.44)
0.38
Chouinard
3.49 (1.11, 10.95)
2.39
Durost
8.68 (1.26, 59.95)
0.84
Garry
1.74 (0.59, 5.16)
2.64
Howard
2.04 (0.67, 6.21)
2.52
Mander
1.35 (0.75, 2.45)
8.89
Nishikawa_82
3.00 (0.14, 65.90)
0.33
Nishikawa_84
9.57 (0.61, 151.28)
0.41
Reschke
3.79 (1.06, 13.60)
1.91
Selman
1.32 (0.93, 1.86)
26.22
Serafetinides
7.91 (0.47, 132.79)
0.39
Simpson
2.14 (0.12, 38.74)
0.37
Spencer
11.00 (1.67, 72.40)
0.88
Vichaiya
18.73 (1.14, 306.64)
0.40
Overall (I-squared = 48.2%, p = 0.014)
1.42 (1.19, 1.69)
100.00
.1
1
10
100
19
ICA-IMOR 1/2 1/2
%
Study ID
ES (95% CI)
Weight
Arvantis
1.40 (0.88, 2.23)
22.12
Beasley
1.12 (0.74, 1.70)
27.47
Bechelli
6.23 (1.52, 25.49)
2.41
Borison_92
7.00 (0.40, 122.44)
0.58
Chouinard
3.49 (1.11, 10.95)
3.66
Durost
8.68 (1.26, 59.95)
1.28
Garry
1.75 (0.58, 5.26)
3.96
Howard
2.04 (0.67, 6.21)
3.87
Mander
1.36 (0.75, 2.47)
13.34
Nishikawa_82
3.00 (0.14, 65.90)
0.50
Nishikawa_84
8.91 (0.56, 141.28)
0.63
Reschke
3.79 (1.06, 13.60)
2.94
Selman
1.74 (0.97, 3.12)
14.11
Serafetinides
8.68 (0.51, 147.44)
0.60
Simpson
2.49 (0.13, 46.31)
0.56
Spencer
11.00 (1.67, 72.40)
1.35
Vichaiya
19.06 (1.16, 313.20)
0.61
Overall (I-squared = 35.0%, p = 0.077)
1.70 (1.37, 2.11)
100.00
.1
1
10
100
20
Allowing for reasons (ICA-R)
• Specify number of missing individuals in
each arm to be imputed by each scheme
ICA-0, ICA-1, ICA-pC, ICA-pE, ICA-p,
ICA-IMOR.
• Can take these data from a different
outcome: metamiss scales to #missing
• If missing in a particular study, metamiss
imputes using combined studies
21
22
Allowing for uncertainty
• So far we have pretended we really know
the IMORs
• This is never really correct
• Now we allow them to be unknown but
from a user-specified distribution
23
Bayesian approach allowing for
uncertain IMORs (Rubin, 1977)
Prior for  E ,  C = log(IMOR) in experimental, control arm:
  E    E2
 E C  
 E 
 
  N    , 
2
C 
 C 
  C    E C
E , C measure your best guess about IM;
 E ,  C measure your uncertainty about IM;
 measures how similar you think  E ,  C are:
0 is most conservative, 1 often allows little impact of IM on results.
24
Bayesian analysis
Elicit prior for E, C or use N(0,12) or N(0,22)
Get posterior distribution by integrating over the 2dimensional distribution of E, C.
• metamiss does this fast & accurately by:
•
•
1. Standard normal approximation to posterior given E, C
2. Integrate using Gauss-Hermite quadrature.
•
Alternatives:
– Taylor expansion (inaccurate for large SD of log IMOR)
– Full Bayesian Monte Carlo (slow, little gain in accuracy)
25
Density
Understanding priors for log IMOR:
implied prior for P(success | missing)
when P(success | observed) = 1/2
0
.25
N(0,0.5^2)
.5
P(success | missing)
N(0,2^2)
.75
N(-1,0.5^2)
1
N(-1,2^2)
26
27
logimor ~ N(0,1)
%
Study ID
ES (95% CI)
Weight
Arvantis
1.42 (0.89, 2.25)
25.78
Beasley
1.06 (0.61, 1.85)
18.07
Bechelli
6.14 (1.50, 25.13)
2.80
Borison_92
7.00 (0.40, 122.44)
0.68
Chouinard
3.49 (1.11, 10.95)
4.26
Durost
8.68 (1.26, 59.95)
1.49
Garry
1.74 (0.58, 5.23)
4.60
Howard
2.04 (0.67, 6.21)
4.49
Mander
1.35 (0.74, 2.47)
15.46
Nishikawa_82
3.00 (0.14, 65.90)
0.58
Nishikawa_84
9.25 (0.58, 146.78)
0.73
Reschke
3.79 (1.06, 13.60)
3.41
Selman
1.54 (0.82, 2.88)
14.02
Serafetinides
8.17 (0.48, 139.06)
0.69
Simpson
2.27 (0.12, 42.39)
0.65
Spencer
11.00 (1.67, 72.40)
1.57
Vichaiya
18.74 (1.14, 308.29)
0.71
Overall (I-squared = 31.4%, p = 0.106)
1.75 (1.38, 2.22)
100.00
.1
1
10
100
28
logimor ~ N(0,2^2)
%
Study ID
ES (95% CI)
Weight
Arvantis
1.42 (0.89, 2.26)
30.37
Beasley
1.08 (0.51, 2.32)
11.36
Bechelli
5.98 (1.45, 24.71)
3.28
Borison_92
7.00 (0.40, 122.44)
0.81
Chouinard
3.49 (1.11, 10.95)
5.05
Durost
8.68 (1.26, 59.95)
1.77
Garry
1.73 (0.57, 5.22)
5.40
Howard
2.04 (0.67, 6.21)
5.32
Mander
1.35 (0.74, 2.47)
18.04
Nishikawa_82
3.00 (0.14, 65.90)
0.69
Nishikawa_84
9.32 (0.59, 148.16)
0.86
Reschke
3.79 (1.06, 13.60)
4.04
Selman
1.60 (0.67, 3.80)
8.77
Serafetinides
7.61 (0.43, 133.33)
0.80
Simpson
2.10 (0.11, 40.70)
0.75
Spencer
11.00 (1.67, 72.40)
1.86
Vichaiya
17.81 (1.06, 298.38)
0.83
Overall (I-squared = 23.6%, p = 0.181)
1.87 (1.44, 2.41)
100.00
.1
1
10
100
29
Proposal: 4 sensitivity analyses
IMORs
Options
(e.g.)
Sensitive to:
Works via:
fixed
equal
fixed
opposite
imor(2 2)
Imbalance in
missingness
Amount of
missing data
Point
estimates
random
equal
random
uncorrelated
sdlogimor(2)
corr(1)
Imbalance in
missingness
Amount of
missing data
Weightings
imor(2 1/2)
sdlogimor(2)
corr(0)
30
Summary
• Tool for sensitivity analysis
• Requires thought about plausible missing data
mechanisms
• Would be nice to overlay sensitivity analysis
with ACA
• Further work includes combining uncertainty
with reasons
• I also have a program mvmeta for
multivariate meta-analysis
31
References
• 1st part: Higgins JPT, White IR, Wood A. Imputation methods
for missing outcome data in meta-analysis of clinical trials.
Clinical Trials, submitted.
• 2nd part: White IR, Higgins JPT, Wood AM. Allowing for
uncertainty due to missing data in meta-analysis. 1. Two-stage
methods. Statistics in Medicine, in press.
• Related: White IR, Welton NJ, Wood AM, Ades AE, Higgins
JPT. Allowing for uncertainty due to missing data in metaanalysis. 2. Hierarchical models. Statistics in Medicine, in press.
• metamiss.ado available from
http://www.mrc-bsu.cam.ac.uk/BSUsite/Software/Stata.shtml
32
Extra slides
33
Gamble-Hollis
%
Study ID
ES (95% CI)
Weight
Arvantis
1.42 (0.86, 2.33)
33.58
Beasley
1.05 (0.34, 3.24)
6.58
Bechelli
6.21 (1.35, 28.50)
3.60
Borison_92
7.00 (0.40, 122.44)
1.02
Chouinard
3.49 (1.11, 10.95)
6.40
Durost
8.68 (1.26, 59.95)
2.24
Garry
1.75 (0.52, 5.92)
5.63
Howard
2.04 (0.67, 6.21)
6.75
Mander
1.36 (0.68, 2.72)
17.36
Nishikawa_82
3.00 (0.14, 65.90)
0.88
Nishikawa_84
9.20 (0.52, 162.90)
1.01
Reschke
3.79 (1.06, 13.60)
5.13
Selman
1.48 (0.37, 5.90)
4.39
Serafetinides
8.40 (0.50, 140.57)
1.05
Simpson
2.35 (0.12, 44.54)
0.97
Spencer
11.00 (1.67, 72.40)
2.36
Vichaiya
19.00 (1.13, 318.79)
1.05
Overall (I-squared = 16.7%, p = 0.257)
2.02 (1.51, 2.70)
100.00
.1
1
10
100
34