Handling (and Preventing) Missing Data in RCTs
ASENT
March 7, 2009
Janet Wittes
Statistics Collaborative
Topics
•Missing values: What, me worry?
•Methods of treatment
•Methods of prevention
•Moral: prevention is better than cure
• Or – “The moral of this tale is ‘care’.”
2
Topics turned into learning objectives
•Why are missing data important?
•What is the distinction between
 Ignorable and non-ignorable missing?
 MAR and NMAR?
•What are general approaches to analysis?
•How can we prevent (or minimize) missing?
3
Extent of missing primary outcome data
•Cardiovascular outcome trial:
1-2%
•Short-term blood pressure trial: 5-10%
•12 week pain trial:
20-40%
•12 week antipsychotic drug:
30-50%
•1 year Alzheimer’s Disease:
20-50%
Source: informal experience
4
Why people don’t care about missing data
•Many people care about missing data in outcome trials
•Many do not care about missing data in symptom trials
 Who cares about those who don’t take drug?
•“We know the drug won’t work if you don’t take it”
•“I am not interested in what happens after people stop.”
•No evidence that the two groups differ in Pr{missing}
•We are interested in what we observe – “complete cases”
•Too hard/expensive to bring back those who stop med
5
So let me convince you to care
•What many assume (explicitly or implicitly)
6
What many assume
7
What many assume
8
What statisticians fear (and assume)
9
What statisticians fear (and assume)
•Those still in study differ from those randomized
•We can’t characterize who is missing
•Who is missing differs by group
10
How papers typically report
(Hard to ferret out extent and timing of missing)
ADAS-Cog Over Time
50
40
30
Score
Active
Placebo
20
10
0
0 1 2 3 4 5 6 7 8 9 10 11 12
Month
“Missing values imputed using LOCF”
(hard to tell that 48% have missing 12 month data)
11
The fundamental dilemma
•Want to know: Effect of intervention if everyone took it
•Can learn: Effect of intervention among randomized
12
So what question should we ask
 What would have been the effect if people were forced to continue?
 What is the effect among people who can tolerate it?
 What would the effect have been if it were measured?
13
Language – Little/Rubin
•Ignorable
 Missing completely at random (MCAR): random number selected the missing
 Missing at random (MAR)
•Given observed data, missingness mechanism does not depend on unobserved.
•Rarely definitively determined from the data at hand
•Nonignorable: Not missing at random (NMAR)
14
Examples of MAR
(Probability of missing depends only on values of observed)
•2 measurements of same variable made at the same time.
 If they differ by more than a given amount a third is taken.
 3rd measurement missing for those who do not differ by the given amount.
•Subject removed from trial if condition is not sufficiently
controlled (criteria pre-defined)
15
Examples of NMAR
(Probability of missing depends on value of the missing data)
 AD patient too demented to come for measurement
 Chronic pain: VAS every two weeks
•Measured at Week 4 – pain still bad
•Patient feels better at week 5 so doesn’t come at Week 6
16
Rule of thumb: no benefit for missingness
Method of imputation shouldn’t give us stronger results than
what we would have seen from the complete cases
Simple example: More uncertainty but results strengthen.
80 patients: 40/group
20 success in treated and 12 in control
P-value (Fisher’s exact) = 0.11
--------------------------------------120 patients but 20 in each group missing
Assume missing data share results in observed
P-value now 0.040
17
Handling missing binary outcomes
•Just ignore the missing observations
•Impute missing on basis of
 Proportion in own group
 Best case – all pbo fail; all rx succeed
 Worst case – all pbo success; all rx fail
 Proportion in placebo group (“not unreasonable guess”)
 Proportion in opposite group (“reasonable worst case”)
•Multiple imputation
18
Problems with usual approach
•Too many degrees of freedom
•Some methods overstate effect
•Some methods understate effect
19
Loss of 4 points in ADAS-Cog
•Two groups – treated and control
•120 per group
•40% in placebo; 20% in treated
•Look at relative risk (<1 is “good”)
•Missing % equal in both groups
20
Example: loss of 4 points on ADAS-COG
Effect of Missing Data on Relative Risk
1.25
Worst
Opposite
Placebo
Own
Best
1.00
0.75
0.50
0.25
0.00
0
5
10
15
20
Percent Missing
21
Continuous, longitudinal, time-to-event outcomes
•Just ignore the missing observations
•Impute missing on basis of data in:
 Own group
 Combined group
 Placebo group
 Opposite group (“worst reasonable case”)*
•Last Observation Carried Forward
•Baseline Observation Carried Forward
•Last rank carried forward#
•Carry forward trajectories
•Multiple imputation
*Proschan et al (2001)., J Stat Planning 96: 155
# O’Brien, Zhang, Bailey (2005). Stat Med 24:34
22
Effect size and p-values for pain study
Effect size
P-value
Completers (n=500/650)
-0.45
0.06
LOCF
-0.51
0.03
BOCF
-0.28
0.25
Placebo trajectory
-0.49
0.03
23
Message
•Analyses produce very different results
•Can affect
 Direction of effect
 Effect size
24
Informed consent documents unclear
•Participation in this study is entirely
voluntary. Your treatment and your doctor’s
attitude toward you will not be affected
should you decide not to participate in this
study…
•You will be asked to return for follow-up
visits and to provide follow-up information.
•If you agree to participate, you may
withdraw from the study at any time
without affecting any benefits to which you
25
Permissive protocols encourage missing data
 “Drop-outs will not be replaced”
•Suggests that it would be ok to replace them
•Suggests that analysis will ignore them
 “Expect 10% drop out, therefore increase sample size by 10%”
 “The primary analysis will use the intent-to-treat pop”
•“The ITT pop is defined as all those randomized who…”
•The ITT pop is defined as the evaluable group
26
Language about withdrawal: an outcome trial
The reason that a subject discontinues from the study will be
recorded in the Case Report Form.
A discontinuation occurs when an enrolled subject ceases
participation in the study, regardless of the circumstances,
prior to completion of the protocol.
…
The final evaluation required by the protocol will be
performed at the time of study discontinuation.
27
Outcome: continuous measure at week 48
•Subjects must be withdrawn from the study (i.e., from any
further study medication or study procedure) for the following
reasons:
 At their own or their legally authorized representative’s request
 If, in the investigator’s opinion, continuation in the study would be
detrimental to the subject's well-being
 Occurrence of an intolerable treatment-emergent adverse event as
determined by the investigator and/or the subject
 Failure of the subject to return to the study site for scheduled visits
 Persistent noncompliance
 Pregnancy
28
Prevention of missing values
•Revise informed consent forms
•Make protocols less permissive
•Define outcome measures that don’t allow “success” for missing
 E.g., Define measures as success or failure and missing = failure
29
Improved informed consent document
•Participation in this study is entirely voluntary.
Your treatment and your doctor’s attitude toward
you will not be affected should you decided not
to participate in this study…
•If you agree to participate, you may withdraw
from the study at any time without affecting any
benefits to which you would otherwise be
entitled.
•You will be asked to return for follow-up visits
and to provide follow-up information even if you
are not taking study medication.
30
Protocols
•Be vigilant about permissive language
•Distinguish between
 Stopping meds
 Stopping active visits
 Withdrawing consent to be followed passively
•Understand the importance of full follow-up
 (even for those who stop study medication)`
31
Typical language about withdrawal in protocols
The reason that a subject discontinues from the study
medication will be recorded in the Case Report Form.
A discontinuation from the study occurs when an enrolled
subject ceases participation a participant in the study dies, is
permanently lost to follow-up, or withdraws consent,
regardless of the circumstances, prior to completion of the
protocol.
…
An final evaluation required by the protocol will be
performed at the time of study discontinuation of study
medication.
32
But, if there will be missing data
•Choose analytic methods that
 Do not add false precision
 Are reasonably conservative
 Are interpretable
•Recognize need for big increase in sample size
33