Strategies for handling missing data in
randomised trials
Peninsula Medical School, Exeter, 14th March 2011
Ian White, MRC Biostatistics Unit, Cambridge, UK
Plan
1. Why do missing data matter?
2. Intention-to-treat analysis strategy for randomised
trials with missing outcomes
3. Analysis methods
– regression and mixed models
4. Sensitivity analysis
– methods & examples
5. Missing baseline covariates
– simple methods are best
• Please ask questions!
• Work with: James Carpenter & Stuart Pocock (LSHTM),
Nick Horton (USA), Simon Thompson (BSU)
2
Why do missing data matter?
1. Loss of power (cf. power with no missing data)
– can’t regain lost power
2. Any analysis must make an untestable assumption
about the missing data
– wrong assumption  biased estimates
3. Some popular analyses with missing data get biased
standard errors
– resulting in wrong p-values and confidence intervals
4. Some popular analyses with missing data are inefficient
– confidence intervals wider than they need be
3
What to do: loss of power
Can’t solve by analysis (but can exacerbate it!).
Approach by design:
• Minimise amount of missing data
– Good communications with participants
– Aim to follow everyone up
– Make repeated attempts using different methods
• Also good to reduce the impact of missing data
– Collect reasons for missing data
– Collect information predictive of missing values
• Now assume that despite doing our best we still have
some missing data…
4
What to do: analysis
A
•
•
•
suitable method of analysis would:
Make the correct assumption about the missing data
Give an unbiased estimate (under that assumption)
Give an unbiased standard error (so that P-values and
confidence intervals are correct)
• Be efficient (make best use of the available data)
we can never be sure what is the correct assumption
 sensitivity analyses are essential
5
Missing data problems: observational studies
Loss of power
Bias
Outcome
Exposure
Confounders
Fail to control confounding
Potentially inefficient analysis
6
Missing data problems: randomised trials
Loss of power
Outcome
Bias
Baseline
covariates
Exposure
Randomised
group
Confounders
Fail to control confounding
Potentially inefficient analysis
7
Plan
1. Why do missing data matter?
2. Intention-to-treat analysis strategy for
randomised trials with missing outcomes
3. Analysis methods
– regression and mixed models
4. Sensitivity analysis
– methods & examples
5. Missing baseline covariates
– simple methods are best
8
Intention-to-treat (ITT) principle
• The ITT principle says that we should:
– include everyone randomised
– in the group to which they were assigned (whether
or not they completed the intervention given to the
group)
• With missing data, this is easier said than done
• I’m going to show it can also conflict with the statistical
principle that the analysis should be valid under
plausible assumptions
9
Note
• I’m addressing the ITT hypothesis
– “There is no difference in clinical outcome between
these two randomised groups”
• & estimating the ITT difference
– difference in clinical outcome between randomised
groups
10
What does intention-to-treat mean when
some outcome data are missing? (1)
• A strict view is that ITT requires complete outcome data
– but researchers with incomplete outcome data also
need a principle to guide them
• Should all randomised individuals still be included?
– “Although those participants [who drop out] cannot
be included in the analysis, it is customary still to
refer to analysis of all available participants as an
intention-to-treat analysis” (Altman et al, 2001)
– “Complete case analysis, which was the approach
used in most trials, violates the principle of intention
to treat” (Hollis & Campbell, 1999)
11
What does intention-to-treat mean when
some outcome data are missing? (2)
• In new advice, the European Medicines Agency (2010)
takes a more relaxed view:
– “Full set analysis generally requires the imputation of
values or modelling for the unrecorded data”
• The CONSORT (2010) statement says:
– “We replaced mention of ‘intention to treat’ analysis,
a widely misused term, by a more explicit request for
information about retaining participants in their
original assigned groups”
12
Difficulties
• Including all randomised individuals in the analysis isn’t
enough to make an analysis valid
• The desire to include all randomised individuals in the
analysis
– reduces emphasis on the appropriate assumptions
– leads to uncritical use of simple imputation methods,
esp. Last Observation Carried Forward (LOCF)
– leads to unnecessary use of complex methods,
esp. multiple imputation
• I’m going to say a bit about LOCF, then explain how we
have re-defined ITT as an analysis strategy
13
40
20
0
Y
60
80
Last observation carried forward (LOCF)
0
3
6
Months
9
12
14
LOCF: why it is used
• LOCF is widely advocated
• e.g. the European drugs regulator wrote
(CPMP 2001, but since revised):
– “The statistical analysis of a clinical trial generally
requires the imputation of values to those data that
have not been recorded …”
– “Last observation carried forward … is likely to be
acceptable if measurements are expected to be
relatively constant over time.”
15
LOCF: what it assumes
• Assumes last observation is representative of the
missing value
• Can’t verify this assumption from the data
– e.g. even if there was no observed trend over time,
drop-outs could have deteriorated
• Analysts rarely give a good justification, and instead
justify LOCF (wrongly) on the grounds that
– it is conservative: not true in general
– it respects ITT by analysing all individuals
• LOCF widely critiqued e.g. Mallinckrodt et al (2004)
• Should only be used if its assumption is plausible!
• So how can we unlink ITT from methods like LOCF?
16
Strategy for intention to treat analysis
with incomplete observations
(White et al, BMJ, 2011)
1. Attempt to follow up all randomised participants, even if
they withdraw from allocated treatment
2. Perform a main analysis of all observed data that is
valid under a plausible assumption about the missing
data
3. Perform sensitivity analyses to explore the effect of
departures from the assumption made in the main
analysis
4. Account for all randomised participants, at least in the
sensitivity analyses
17
Example 1: QUATRO trial
• European multicentre RCT to evaluate the effectiveness of
adherence therapy in improving quality of life for people
with schizophrenia (Gray et al, 2006)
• Primary outcome: quality of life measured by the SF-36
MCS scale, measured at baseline and 52-week follow up
Intervention
Control
Total n
204
205
Missing outcome
14%
6%
Mean of observed outcomes
40.2
41.3
SD of observed outcomes
12.0
11.5
18
QUATRO trial: analyses
• We did attempt to follow up all randomised individuals
• Main assumption: no difference between missing and
observed values, once adjusted for baseline variables
(MAR)
• Main analysis: analysis of covariance on complete cases
– intervention effect = -0.33 (s.e. 1.11)
• Sensitivity analysis: consider possible differences
between missing and observed values, allowed to be
different in each arm
– see later
19
Missing at random (MAR)
• The probability that data are
missing depends on the values of
the observed data, but does not
depend on the values of the
missing data
• p(M | Yobs, Ymis) = p(M | Yobs)
observed
data
M
missing
data
20
Plan
1. Why do missing data matter?
2. Intention-to-treat analysis strategy for randomised
trials with missing outcomes
3. Analysis methods
– regression and mixed models
4. Sensitivity analysis
– methods & examples
5. Missing baseline covariates
– simple methods are best
21
How to approach the analysis
• Principled approach to missing data:
– start by identifying a plausible assumption
– then choose an analysis method that’s valid under
that assumption
• Some analysis methods are simple to describe but have
complex assumptions
22
The analysis toolkit
Simple methods
• Last observation carried forward (LOCF)
• Complete-case analysis
• Mean imputation
• Missing indicator method
• Regression imputation
More complex methods
• Multiple imputation
• Likelihood-based methods
• Inverse probability weighting (IPW)
23
Properties of methods
Method
For missing covariate
For missing outcome
Not applicable
Valid under LOCF assn
Complete cases
Inefficient
Single Y: valid under MAR
Repeated Y: inefficient
Mean imputation
Missing indicator
Fails to control
confounding
Valid in RCT (see later)
Regression
imputation
Valid under MAR (no Y
in imp. model)
SE 
Valid under MAR
Valid under MAR
Inefficient or complex
Valid under MAR
Simple patterns only
LOCF
Multiple imputation
Maximum likelihood
IPW
SE 
Not applicable
Valid means valid & efficient
24
A comment on MAR
• A lot of statistical literature seems to regard MAR as the
correct starting point for analyses with missing data
• One argument in favour of MAR is that it tends to
become more plausible as more variables are included
in the model
– unlike other assumptions
• I think the correct assumption depends on the clinical
context
• However in this section I’m going to assume MAR
25
What method is best for missing
outcomes in a RCT, assuming MAR?
With a single outcome (not repeated measures):
• Regress outcome (Y) on randomised group (Z),
adjusting for baseline covariates (X)
– analysis of covariance, ANCOVA
– this is the likelihood-based method
• Which X?
– to make MAR valid, adjust for X that predict both
outcome and missingness
– to gain power, adjust for X that predict outcome
• Cases with missing Y contribute no information
– complete-cases analysis is correct!
26
What method is best for missing
outcomes in a RCT, assuming MAR?
With a repeated outcome:
• Use a mixed model (likelihood-based)
• Include all observed outcome data
• Exclude any individuals with no post-baseline
observations
• Include X’s as before
Software:
• Stata xtmixed, SAS proc mixed, R lme()
Some points to note:
• Don’t allow a treatment effect at baseline
• Allow a different treatment effect at each follow-up time
• If possible, allow for correlations between outcomes via
an unstructured variance-covariance matrix
27
What about multiple imputation?
Idea of multiple imputation:
• Impute missing (single or longitudinal) outcomes from
observed outcomes and baselines  repeat m times
• Analyse the m completed data sets
• Combine estimates by Rubin’s rules
Tutorial: White et al (2011)
28
Multiple imputation and ITT
• MI includes all randomised individuals in the analysis
– appealing
• Example with single outcome (Riper et al, 2008):
– “We then performed intention-to-treat analysis, using
multiple imputation to deal with loss to follow-up.”
• MI is the same as fitting a [mixed] model to the observed
data if imputation model and analysis model are the same
– but MI has additional random error (“Monte Carlo
error”) due to using a finite number of imputed data
sets
MI may be of value in a RCT
– so why do MI?
•if additional information (“auxiliary
variables”, e.g. compliance) can be
included in the imputation model
•as a way to do sensitivity analyses 29
Example: UK700 trial
• Intensive vs. Standard case management for severely
mentally ill people living in the community
• 708 patients randomised in 4 UK centres
• Outcome here: CPRS (psychopathology) score from 2-year
interview (missing in 11% of Intensive, 20% of Standard)
Method
Patients in
analysis
Treatment
effect
Standard
error
ANCOVA (adjusted
for baseline)
595
-0.3898
1.0348
Joint model for
baseline & outcome
705
-0.3898
1.0304
Multiple imputation
708
-0.3834
1.0369
Monte Carlo error
0.01
0.004
30
Plan
1. Why do missing data matter?
2. Intention-to-treat analysis strategy for randomised
trials with missing outcomes
3. Analysis methods
– regression and mixed models
4. Sensitivity analysis
– methods & examples
5. Missing baseline covariates
– simple methods are best
31
How to do sensitivity analyses?
• Not LOCF for main analysis, CC for sensitivity analysis
CC
MAR-based
9
8
7
6
Mean outcome
10
LOCF
0
1
2 0
1
2 0
1
2
time
Complete
Drop out at time 1
32
Principled sensitivity analysis
• Not LOCF for main analysis, CC for sensitivity analysis
– the alternative assumptions should contradict the
main assumption
• Instead, specify the numerical value of “sensitivity
parameter(s)” governing the degree of departure from
the main assumption (Kenward et al, 2001)
– e.g. the degree of departure from MAR
• Want methods to exactly match the main analysis when
sensitivity parameter is zero
• I’ll first describe methods based around an MAR-based
main analysis, then go further
33
Mean score approach (1)
• Analysis model: E[Y|X] = h(bAX)
– Y is outcome
– h(.) is inverse link function (e.g. identity or invlogit)
– X is a covariate vector including 1's, randomised
group Z and baseline covariates – we're interested in
the component of bA corresponding to Z
• Missing data occur in Y only
• MY is missingness of Y
34
Mean score approach (2)
• Analysis model: E[Y|X] = h(bAX)
• Imputation model: E[Y|X,MY] = h(bIX + MYd)
– d is a user-specified departure from MAR: could be
d1 if randomised to arm 1, d0 if randomised to arm 0.
• Estimate bI by regressing Y on X in complete cases
(MY=0)
• If we had complete data we would estimate the analysis
model using estimating equations S X{Y - h(bAX)} = 0
• Replace missing Y with Y* = E[Y|X,MY=1] = h(bIX + d)
• Solve!
• Standard errors by sandwich method (based on EEs for
both bI and bA) or by conditional variance formula
• Simplifies in the special case of linear regression
35
-2
0
2
Example: QUATRO data
-4
MAR
-10
-8
-6
-4
Missing - Observed
Intervention only
Both arms
Outcome SD ≈ 12
-2
0
Control only
36
Why do I allow different missing data
mechanisms in different arms?
• Because data may be further from MAR after more
intensive interventions
• Because results are most sensitive to this
37
Simple approach
• Assumed departures from MAR:
– let d = mean unobserved – mean observed outcome
– taking values d0 , d1 in arm 0, 1
• Values estimated from observed data:
– p0, p1 = fraction of missing data in arm 0, 1
– DCC = intervention effect in complete cases
– D = true intervention effect
• Then we can show (White et al, 2007) that
– D = DCC + d1p1 – d0p0
– se(D) ≈ se(DCC)
• Back-of-an-envelope calculation!
38
Standard error of
estimated intervention effect
1.1
1.09
1.11 1.12 1.13
Are std errors affected by d? (QUATRO)
-10
-8
-6
-4
d = Missing – Observed
Intervention only
Both arms
-2
0
Control only
39
Alternatives to mean score approach
• Can use MI
– Define imputation and analysis models as before
– Fit imputation model to complete cases as before
– Impute with offset d
» implemented in ice
– For sensitivity analysis, best to use same seed across
different values of d
• Can use IPW
– involves specifying d in missingness model
logit P(MY=1|Y,X) = aX + d Y
40
Demonstration using rctmiss in Stata
Missing data are 5 units (0.5 SD) worse than observed:
. rctmiss, pmmdelta(-5): reg sf_mcs alloc sf_mcsba
…in both arms
MNAR regression results
-----------------------------------------------------------------------------sf_mcs |
Coef.
Std. Err.
z
P>|z|
[95% Conf. Interval]
-------------+---------------------------------------------------------------alloc | -.7338669
1.096779
-0.67
0.503
-2.883514
1.415781
sf_mcsba |
.4697194
.0497254
9.45
0.000
.3722594
.5671794
_cons |
22.37387
2.058745
10.87
0.000
18.33881
26.40894
-----------------------------------------------------------------------------. rctmiss, pmmdelta(cond(alloc,-5,0)): reg sf_mcs alloc sf_mcsba
… in intervention arm only
2
0
41
-2
rctmiss also produces the graph
-4
Estimated intervention effect
MNAR regression results
-----------------------------------------------------------------------------sf_mcs |
Coef.
Std. Err.
z
P>|z|
[95% Conf. Interval]
-------------+---------------------------------------------------------------alloc | -1.016182
1.093445
-0.93
0.353
-3.159295
1.126931
sf_mcsba |
.4695661
.0496591
9.46
0.000
.372236
.5668962
_cons |
22.66207
2.052232
11.04
0.000
18.63977
26.68437
------------------------------------------------------------------------------
-10
-8
-6
-4
Missing - Observed
Intervention only
Both arms
-2
Control only
0
Corespondence with MAR when D=0
. reg sf_mcs alloc sf_mcsba
Source |
SS
df
MS
-------------+-----------------------------Model | 10497.5347
2 5248.76733
Residual | 36748.2816
346 106.208906
-------------+-----------------------------Total | 47245.8162
348
135.76384
Standard MAR analysis
Number of obs
F( 2,
346)
Prob > F
R-squared
Adj R-squared
Root MSE
=
=
=
=
=
=
349
49.42
0.0000
0.2222
0.2177
10.306
-----------------------------------------------------------------------------sf_mcs |
Coef.
Std. Err.
t
P>|t|
[95% Conf. Interval]
-------------+---------------------------------------------------------------alloc | -.3341637
1.108198
-0.30
0.763
-2.513817
1.84549
sf_mcsba |
.4703734
.0475847
9.88
0.000
.3767818
.5639651
_cons |
22.62969
2.05461
11.01
0.000
18.5886
26.67079
-----------------------------------------------------------------------------. rctmiss, pmmdelta(0): reg sf_mcs alloc sf_mcsba
Results allowing for MNAR
Using 349 complete cases and 37 incomplete cases
-----------------------------------------------------------------------------sf_mcs |
Coef.
Std. Err.
z
P>|z|
[95% Conf. Interval]
-------------+---------------------------------------------------------------alloc | -.3341637
1.108198
-0.30
0.763
-2.506193
1.837865
sf_mcsba |
.4703734
.0475847
9.88
0.000
.3771091
.5636377
_cons |
22.62969
2.05461
11.01
0.000
18.60273
26.65665
-----------------------------------------------------------------------------42
Main difficulty
• How to specify d?
• Need to discuss with investigators
– aim to pre-specify
• Alternatively report value of d that changes [significance
of] conclusions
43
Example 2: smoking cessation trial
• Smokers calling a telephone helpline were randomised
to tailored advice letters + standard counselling or
standard counselling
– Sutton & Gilbert (2007)
• Outcome: not smoking in months 4-6 (“quit”)
Intervention arm
(n=599)
Control arm
(n=565)
73 (12%)
51 (9%)
Didn’t quit
390 (65%)
364 (64%)
Missing
136 (23%)
150 (27%)
Quit
44
Smoking cessation trial: analyses
• Main assumption: assume missing = still smoking
– Includes everyone, but not convincing
• Main analysis: impute missing values as smokers
– OR 1.40 (95 CI 0.96 - 2.05)
• Sensitivity analyses: define the informatively missing
odds ratio (IMOR) as the odds ratio between quitting
and missing
– main analysis assumes IMOR=0
– sensitivity analysis varies IMOR (in one or both
arms)
45
Intervention odds ratio (95% CI)
1
2
Smoking trial: sensitivity analysis
Main analysis
0
0.1
0.2
0.3
IMOR in specified arm(s)
intervention arm
(23% missing)
both arms
0.4
0.5
control arm
(27% missing)
46
Plan
1. Why do missing data matter?
2. Intention-to-treat analysis strategy for randomised
trials with missing outcomes
3. Analysis methods
– regression and mixed models
4. Sensitivity analysis
– methods & examples
5. Missing baseline covariates
– simple methods are best
47
Missing baselines
Missing baselines in RCTs are a completely different
problem from missing outcomes
• Still assume we want to estimate the effect of
randomised intervention on outcome
• Baseline adjustment is used to gain precision, not avoid
bias
– missing baselines are not a source of bias
• Complete cases analysis is a very bad idea
• White & Thompson (2005) showed that almost anything
else is OK
– in particular, mean imputation or missing indicator
method
– provided randomisation is respected
48
Missing indicator method
• Regress Y on covariates
– MX = 1 if X observed, else 0
– Xfill = X if X observed, else 0 (or some fixed X0)
• Can use weights (good for large X-Y correlation)
• But missing indicator method is biased in observational
studies (Greenland and Finkle 1995)
– get imperfect adjustment for confounders (residual
confounding)
49
15
Confounder
5
10
0
1
0
Outcome
20
25
What’s wrong with the missing indicator method?
0
5
10
15
Exposure
20
25
50
15
Confounder
5
10
0
1
?
0
Outcome
20
25
Make some confounders missing
0
5
10
15
Exposure
20
25
51
15
Confounder
5
10
?
0
Outcome
20
25
Association is different
0
5
10
15
Exposure
20
25
52
Residual confounding
15
Confounder
5
10
0
1
?
0
Outcome
20
25
Missing indicator
method averages
over confounder
= 0, 1 and ?
0
5
10
15
Exposure
20
Residual confounding
doesn’t occur in
randomised trials:
25
Covariate
is independent
of “exposure” so not a
53
“confounder”
Missing indicator: how to do it (QUATRO)
. gen xmiss = missing(sf_mcsba)
. gen xfill = cond(xmiss, 39, sf_mcsba)
. reg sf_mcs alloc xmiss xfill
Source |
SS
df
MS
---------+-----------------------------Model | 10630.1786
3 3543.39287
Residual | 39646.5297
363
109.21909
---------+-----------------------------Total | 50276.7084
366 137.368056
Number of obs
F( 3,
363)
Prob > F
R-squared
Adj R-squared
Root MSE
=
=
=
=
=
=
367
32.44
0.0000
0.2114
0.2049
10.451
----------------------------------------------------------------------sf_mcs |
Coef.
Std. Err.
t
P>|t|
[95% Conf. Interval]
-------+--------------------------------------------------------------alloc | -.2364209
1.095984
-0.22
0.829
-2.391695
1.918854
xmiss | -2.682338
2.527628
-1.06
0.289
-7.65297
2.288295
xfill |
.4706946
.0482474
9.76
0.000
.3758151
.5655741
_cons |
22.57087
2.07815
10.86
0.000
18.48414
26.65759
-----------------------------------------------------------------------
(or use rctmiss)
n increased from 349 to 367
54
US report: “The Prevention and Treatment of
Missing Data in Clinical Trials”
• Commissioned by Food & Drug Administration
• Panel of top statisticians
•
National Research Council (Dec 2010)
1. Introduction and Background
2. Trial Designs to Reduce the Frequency of Missing Data
– focus on estimands (pre-trial)
3. Trial Strategies to Reduce the Frequency of Missing Data
4. Drawing Inferences from Incomplete Data
– covers it all
5. Principles and Methods of Sensitivity Analyses
– lots of suggestions
6. Conclusions and Recommendations
55
Conclusions & discussion
• Missing baselines: use simple methods that respect
randomisation
• Missing outcomes: focus on assumptions, not methods
• An intention-to-treat analysis strategy should include all
individuals in sensitivity analyses
– but not necessarily in main analyses
• ANCOVA and mixed models are usually the best
strategy for missing outcomes in RCTs
– use MI with auxiliary data (e.g. compliance) or
possibly as a way to do sensitivity analyses
• Sensitivity analyses can be done in various ways
– install my software rctmiss in Stata using net from
http://www.mrc-bsu.cam.ac.uk/IW_Stata/misc
56
References (1)
• Altman D et al (2001). The revised CONSORT statement for reporting randomized
trials: Explanation and elaboration. Annals of Internal Medicine 134: 663–694.
• Committee for proprietary medicinal products (2001). Points to consider on missing
data. http://www.emea.europa.eu/pdfs/human/ewp/177699EN.pdf
• European Medicines Agency (2010). Guideline on missing data in confirmatory
clinical trials. http://www.ema.europa.eu/ema/pages/includes/document/
open_document.jsp?webContentId=WC500096793
• Greenland S, Finkle W (1995). A critical look at methods for handling missing
covariates in epidemiologic regression analyses. American Journal of Epidemiology
142: 1255–1264.
• Gray R et al (2006). Adherence therapy for people with schizophrenia: European
multicentre randomised controlled trial. British Journal of Psychiatry 189: 508–514.
• Hollis S, Campbell F (1999). What is meant by intention to treat analysis? Survey
of published randomised controlled trials. British Medical Journal 319: 670–4.
• Kenward MG, Goetghebeur EJT, Molenberghs G (2001). Sensitivity analysis for
incomplete categorical tables. Statistical Modelling 1: 31–48.
• Mallinckrodt CH et al (2004). The effect of correlation structure on treatment
contrasts estimated from incomplete clinical trial data with likelihood-based
repeated measures compared with last observation carried forward ANOVA. Clinical
Trials 1:477–489.
57
References (2)
• Moher D, Schulz KF, Altman DA, for the CONSORT group (2001). The CONSORT
statement: Revised recommendations for improving the quality of reports of
parallel-group randomised trials. Lancet 357:1191–1194.
• National Research Council (2010). The Prevention and Treatment of Missing Data in
Clinical Trials. Panel on Handling Missing Data in Clinical Trials. Committee on
National Statistics, Division of Behavioral and Social Sciences and Education.
http://www.nap.edu/catalog.php?record_id=12955.
• Sutton S, Gilbert H (2007). Effectiveness of individually tailored smoking cessation
advice letters as an adjunct to telephone counselling and generic self-help
materials: randomized controlled trial. Addiction 102:994–1000.
• Riper H et al (2008). Web-based self-help for problem drinkers: a pragmatic
randomized trial. Addiction 103:218–227.
• White IR, Horton N, Carpenter J, Pocock SJ (2011). An intention-to-treat analysis
strategy for randomised trials with missing outcome data. British Medical Journal
http://www.bmj.com/content/342/bmj.d40.full.html?
ijkey=08IQJ8etzBYYGLw&keytype=ref.
• White IR, Thompson SG (2005). Adjusting for partially missing baseline
measurements in randomised trials. Statistics in Medicine 24:993–1007.
• White IR, Wood A, Royston P (2011). Multiple imputation using chained equations:
issues and guidance for practice. Statistics in Medicine; 30: 377–399.
58
The end
59