Developing a framework to allow for
informative missingness
in Cost Effectiveness Analysis
Alexina Mason
joint work with James Carpenter, Richard Grieve
and Manuel Gomes
UCL Statistical Methods in Health Economics Seminar
15 December 2015
Introduction
Modelling Approaches
Elicitation
Outline
Introduction
Missing data in CEA
Motivating example - IMPROVE
Modelling Approaches
Bayesian joint models
Pattern mixture models for IMPROVE
Elicitation
Designing the IMPROVE elicitation
Elicitation Tool
Results and Discussion
Elicitation results
Sensitivity analysis results
Results and Discussion
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Background
• Usual current methods for addressing missing data in CEA:
• complete-case analysis (assumes MCAR)
• multiple imputation (under MAR)
• but, MNAR received little attention
• Wider interest in MNAR in other areas (e.g. biostatistics)
• methods guidance generally advocate sensitivity analysis
• but there is debate about the form this should take
• good practical examples remain scarce, e.g. Bayesian models
using expert elicitation to inform sensitivity parameters
• Unclear how to adapt these methods to the CEA context
• Fully Bayesian approaches provide an appealing framework to
illustrate how uncertainty related to missingness translates into
decision uncertainty
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Missing quality-of-life in CEA
• NICE methods guide now encourages studies to collect
quality-of-life (QoL) alongside other effectiveness outcomes
• However, QoL is often missing in CEA studies
• e.g. patients fail to return QoL questionnaires
• True missing data mechanism cannot be determined from data
at hand
• Non-response in self-reported outcomes such as QoL is likely to
be informative
• Sensitivity analysis allowing for MNAR required to fully explore
uncertainty due to missing data
• relies on assumptions about the sensitivity parameters
• these can be informed by expert opinion
Introduction
Modelling Approaches
Elicitation
Results and Discussion
The IMPROVE trial
• Aim of the IMPROVE trial
• to assess the effectiveness and cost-effectiveness of an
emergency endovascular strategy (EVAR) compared with open
repair (OPEN) to treat patients with a ruptured abdominal aortic
aneurysm
• Multi-centre RCT (32 sites, N=613)
• Data available:
• 30 days (primary outcome and cost analysis published)
• 3 and 12 months (cost-effectiveness analysis published)
• EVAR is associated with gains in QoL
• unclear whether those gains depend on the assumptions about
the missing data
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Main findings from primary analysis (EVAR vs OPEN)
N
INC COST
INC QALY
INB?
?
CC
MAR
401
-£1152 (-£3755,£1452)
0.066 (-0.001,0.132)
£3120 (-£90,£6329)
613
-£2329 (-£5489,£922)
0.052 (-0.005,0.108)
£3877 (£253,£7408)
95% CI in brackets
incremental net benefits valuing QALY gains at £30,000 per QALY
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Why IMPROVE?
• Substantial missingness in QoL
• missing 3 mth QoL: 24% OPEN (36 out of 150 eligible); 18%
EVAR (30 out of 168 eligible);
• missing 12 mth QoL: 27% OPEN (38 out of 140 eligible); 21%
EVAR (34 out of 161 eligible);
• Typical setting
• unclear whether MAR is plausible and that may lead to misleading
inferences on the cost-effectiveness of EVAR vs OPEN
• IMPROVE will feed into a broader decision model on the
long-term effectiveness and cost-effectiveness of EVAR vs OPEN
• Practical advantages
• IMPROVE trial is ongoing (3-year follow-up)
• availability of ‘experts’: trial coordinators
Introduction
Modelling Approaches
Elicitation
Outline
Introduction
Missing data in CEA
Motivating example - IMPROVE
Modelling Approaches
Bayesian joint models
Pattern mixture models for IMPROVE
Elicitation
Designing the IMPROVE elicitation
Elicitation Tool
Results and Discussion
Elicitation results
Sensitivity analysis results
Results and Discussion
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Joint models
• Notation:
• Let z = (zij ) denote a rectangular data set of interest
i = 1, ..., n individuals and j = 1, ..., k variables
• Let m = (mij ) be a binary indicator variable such that
0: zij observed
mij =
1: zij missing
• Let β and θ denote vectors of unknown parameters
• Then the joint model (likelihood) of the full data is f (z, m|β, θ)
• Informative missingness requires modelling
1. data of interest
2. missing data mechanism
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Pattern mixture models
• Two factorisations of the joint model are commonly used
1. selection models
f (z, m|β, θ) = f (m|z, β, θ)f (z|β, θ)
2. pattern mixture models
f (z, m|β, θ) = f (z|m, β, θ)f (m|β, θ)
• We have chosen to use pattern mixture models
• allow a different model for z for each pattern of missing values
• assumptions about the missing data are more explicit
• corresponds more directly to what is actually observed (i.e. the
distribution of the data within subgroups having different missing
data patterns)
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Advantages of a Bayesian framework
• Fully Bayesian models
• are theoretically sound
• enable coherent model estimation
• allow uncertainty about imputed missing values to be fully
propagated through the model
• Bayesian models are formulated in a modular way
• ideal for iteratively building complex models
• Provides scope for including extra data or other information
through informative priors or sub-models
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Model overview
Survive to
12 months
Death between
3 & 12 months
Death before
3 months
Ineligible to
follow-up
OPEN: 140
EVAR: 161
OPEN: 10
EVAR: 7
OPEN: 93
EVAR: 91
OPEN: 54
EVAR: 57
Calculate mean QALY and cost, weighting means across 4 groups
Calculate Incremental Net Benefits
• Quality Adjusted Life Years (QALYs) calculated from QoL and
survival data
• QoL collected at 3 and 12 months using EQ-5D health
questionnaires
• Costs taken from administrative records
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Model for 12 month survivors
Survive to 12 months
Informative priors
for sensitivity
parameters based
on elicitation
*Code as marginal and
conditional distribution
Model QoL at 3 and 12 mths
assuming bivariate normality*
Pattern Mixture model to
impute missing scores (MNAR)
Calculate QALY from QoL scores
using properties of Normal
distribution
Model QALY and Costs assuming
bivariate normality*
Missing costs assumed MAR
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Marginal distribution for QoL at 3 months
Pattern 1: patients who completed their QoL questionnaire:
QoL3i ∼ N(µi , σ 2 )
µi = ηt(i) + βa agei + βs sexi + βh,hardman(i)
η1 , η2 , βa , βs , βh,1 , . . . , βh,5 , σ 2 ∼ prior distribution
t(i) trial arm indicator of individual i
age and sex are fully observed
Hardman index (hardman) is partially observed
⇒ covariate imputation model required
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Marginal distribution for QoL at 3 months
Pattern 1: patients who completed their QoL questionnaire:
QoL3i ∼ N(µi , σ 2 )
µi = ηt(i) + βa agei + βs sexi + βh,hardman(i)
η1 , η2 , βa , βs , βh,1 , . . . , βh,5 , σ 2 ∼ prior distribution
t(i) trial arm indicator of individual i
age and sex are fully observed
Hardman index (hardman) is partially observed
⇒ covariate imputation model required
Pattern 2: patients who did not complete their QoL questionnaire:
µi = ηt(i) + δt(i) + βa agei + βs sexi + βh,hardman(i)
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Marginal distribution for QoL at 3 months
Pattern 1: patients who completed their QoL questionnaire:
QoL3i ∼ N(µi , σ 2 )
µi = ηt(i) + βa agei + βs sexi + βh,hardman(i)
η1 , η2 , βa , βs , βh,1 , . . . , βh,5 , σ 2 ∼ prior distribution
t(i) trial arm indicator of individual i
age and sex are fully observed
Hardman index (hardman) is partially observed
⇒ covariate imputation model required
Pattern 2: patients who did not complete their QoL questionnaire:
µi = ηt(i) + δt(i) + βa agei + βs sexi + βh,hardman(i)
• Conditional distribution defined for QoL at 12 months using
similar principles
Introduction
Modelling Approaches
Elicitation
Outline
Introduction
Missing data in CEA
Motivating example - IMPROVE
Modelling Approaches
Bayesian joint models
Pattern mixture models for IMPROVE
Elicitation
Designing the IMPROVE elicitation
Elicitation Tool
Results and Discussion
Elicitation results
Sensitivity analysis results
Results and Discussion
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Who are our experts?
• People whose training and regular work mean they are likely to
have significant additional insights into patients with missing data
• Staff with direct contact with IMPROVE patients
• IMPROVE trial principal investigators and co-ordinators
• consultant vascular surgeons
• research nurses, vascular nurse specialist, consultant vascular
nurses, ...
• Ongoing involvement in trial
• 50 potential experts identified
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Chosen approach
• AIM: maximise number of elicitations
• Reliant on goodwill of ‘experts’
• minimise administrative burden
• easy to use elicitation tool
• limit completion time to 30 minutes
• Web based elicitation tool developed using Shiny (a web
application framework for R)
• graphical approach
• examples provided
• feedback given
• Received favourable ethical opinion from the LSHTM Research
Ethics Committee (Ref: 9597)
Introduction
Modelling Approaches
Elicitation
Results and Discussion
What information is required from experts?
• δ are unidentifiable sensitivity parameters
• relate to difference in mean QoL between patients who did and
did not return their QoL questionnaire
• Require a joint prior allowing for correlation
⇒ 3 pieces of information needed to build informative priors
• Hence we will elicit change in mean QoL score for patients who
did NOT return their questionnaire compared to those whose
questionnaire was returned for:
1. patients in OPEN REPAIR arm (marginal distribution)
2. patients in EVAR arm (marginal distribution)
3. patients in EVAR arm given the difference in the OPEN REPAIR
arm (conditional distribution)
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Who are Alfred, Bill and Chris?
• Main questions feature Alfred, Bill and Chris
• fictitious, but representative of typical IMPROVE patients
• all are 75 years old, survived their operation to repair a ruptured
aneurysm and had no major complications
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Main elicitation questions
• Focus on quality of life scores at 3 months for survivors
• Alfred’s score is marked on scale of possible QoL scores
• followed OPEN arm, returned QoL questionnaire at 3 months
• Expert asked about likely scores for
1. Bill (OPEN arm, did not complete questionnaire)
2. Chris (EVAR arm, did not complete questionnaire)
3. Chris, if Bill’s score is known (2 variants: low and high)
• The experts belief’s are represented by a Normal distribution,
controlled by 2 sliders
1. position of most likely value
2. degree of uncertainty
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Additional information collected
• Background of experts, including
• job title
• years in current role
• Familiarity with the results of IMPROVE trial
• Free text questions about basis of views
• Change in difference in QoL scores between arms for typical
non-responding patients at 12 months compared to 3 months
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Developing the elicitation tool
• Pre-piloting and piloting undertaken face-to-face
• short introduction
• elicitation carried out
• experience discussed
• Pre-piloting
• 3 LSHTM clinical trial staff (no involvement with IMPROVE)
• provided feedback on usability of tool and clarity of questions
• Piloting
• 4 experts with previous involvement in IMPROVE trial (2 trial
managers, 1 vascular surgeon and 1 clinical trials nurse)
• graphical approach with sliders well received
• helpful suggestions for instructions, explanatory text & questions
• tool refined between each pilot elicitation
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Administration of main elicitation
• Article on elicitation in IMPROVE newsletter
• Invitation to participate sent from IMPROVE CI, including
• participant information sheet
• web link
• Informed consent collected electronically
• questions disabled until consent box ticked
• Reminders sent at weekly intervals
• Unanticipated hitch - antiquated web browsers
• Provided option to complete at the The Vascular Societies
Annual Scientific Meeting 2015
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Elicitation Tool DEMO
Introduction
Modelling Approaches
Elicitation
Outline
Introduction
Missing data in CEA
Motivating example - IMPROVE
Modelling Approaches
Bayesian joint models
Pattern mixture models for IMPROVE
Elicitation
Designing the IMPROVE elicitation
Elicitation Tool
Results and Discussion
Elicitation results
Sensitivity analysis results
Results and Discussion
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Overview of respondents
• To date, 25 submitted responses from 50 invites
• a few invited experts nominated substitutes
• 15 doctors, 9 nurses and 1 radiologist
• 15 completed at the Vascular Societies ASM
• responses received from 18 different sites (out of 32 sites)
• 12 for high variant of Q5 and 13 for low variant (conditional
question)
• One expert expressed (almost) complete uncertainty about
scores
• but useful qualitative comments
• expert’s site had 100% returned questionnaires
• excluded from pooled priors
Modelling Approaches
Elicitation
Results and Discussion
Familiarity with trial results
some familiarity
read the paper
9
1
0
1
11
0
2
1
EVAR QoL > OPEN QoL
OPEN QoL > EVAR QoL
No difference
Not sure
Level of difference between treatment arms
Number of experts
Introduction
4
3
2
1
0
0.05
0.10
0.15
0.20
Level of Difference
Not sure
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Individual expert priors
12
doctor/radiologist
nurse
10
10
8
8
density
density
12
6
6
4
4
2
2
0
0
−0.2
0.0
0.2
0.4
0.6
EQ−5D for Bill
0.8
1.0
doctor/radiologist
nurse
−0.2
0.0
0.2
0.4
0.6
EQ−5D for Chris
0.8
1.0
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Certainty
EVAR: marginal distribution
Joint prior
0.8
0.6
0.2
10
0
5
−0.2 0.0
5
0
−0.2 0.0
0.2
0.4
0.6
0.8
1.0
0.4
EVAR
20
15
density
15
10
density
25
20
30
1.0
OPEN: marginal distribution
−0.2 0.0
0.2
0.4
0.6
0.8
1.0
−0.2 0.0
0.2
0.6
OPEN
OPEN: marginal distribution
EVAR: marginal distribution
Joint prior
0.
2
0.6
0.4
EVAR
1
0.8
0.2
0.2
0.4
EQ−5D
0.6
0.8
1.0
1.0
1.2
−0.2 0.0
0.2
0.0
0.2
−0.2 0.0
0.8
1.4
0.8
0.8
0.6
0.4
0.4
density
0.6
0.8
1.0
EQ−5D
0.0
density
0.4
EQ−5D
−0.2 0.0
0.2
0.4
EQ−5D
0.6
0.8
1.0
0.6
0.4
0.2
−0.2 0.0
0.2
0.4
OPEN
0.6
0.8
1.0
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Plausible values
Joint prior
1.0
EVAR: marginal distribution
10
0.8
8
8
OPEN: marginal distribution
−0.2 0.0
0.2
0.4
0.6
0.8
1.0
0.4
EVAR
−0.2 0.0
0.2
0.4
0.6
0.8
1.0
−0.2 0.0
0.2
0.4
0.6
EQ−5D
EQ−5D
OPEN
OPEN: marginal distribution
EVAR: marginal distribution
Joint prior
0.8
1.0
0.8
1.0
0.4
EVAR
0
−0.2 0.0
0.2
0.4
EQ−5D
0.6
0.8
1.0
15
5
10 20
−0.2 0.0
2
0.2
4
density
6
0.6
0.8
8
1.0
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5
density
−0.2 0.0
0
0
2
2
0.2
4
density
4
density
6
0.6
6
20
−0.2 0.0
0.2
0.4
EQ−5D
0.6
0.8
1.0
−0.2 0.0
0.2
0.4
OPEN
0.6
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Differences in mean scores elicited from experts
Chris − Alfred
Chris − Bill
7
6
6
6
5
5
5
4
3
2
4
3
2
1
1
0
0
−60
−40
−20
0
20
Bill − Alfred Scores
Frequency
7
Frequency
Frequency
Bill − Alfred
7
4
3
2
1
0
−60
−40
−20
0
20
Chris − Alfred Scores
−60
−40
−20
0
20
Chris − Bill Scores
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Correlation between Bill’s score and Chris’s score
Number of experts
10
8
6
4
2
0
[−1,−0.75]
(−0.75,−0.5]
(−0.5,−0.25]
(−0.25,−0.01]
(−0.01,0.01]
(0.01,0.25]
(0.25,0.5]
(0.5,0.75]
(0.75,1]
correlation range
Red: expert answered low variant of conditional Q (Bill’s score = 0.4)
Blue: expert answered high variant of conditional Q (Bill’s score = 0.9)
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Use of expert information
• ‘Community of Priors’ to explore sensitivity of results
• Individual priors: subset chosen to span range of views
• Expert 8: QoL score 0.2 higher for OPEN
• Expert 16: QoL score 0.29 higher for EVAR
• Pooled priors: created using algebraic pooling
• all experts
• all doctors
• all nurses
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Preliminary results of sensitivity analysis
CC
MAR
MNAR11
MNAR22
MNAR33
MNAR44
MNAR55
?
†
1
2
3
4
5
INC COST (£)
INC QALY
INB? (£)
INB>0†
-1147 (-3519,1216)
-871 (-3372,1662)
-849 (-3377,1695)
-854 (-3359,1668)
-847 (-3366,1650)
-848 (-3363,1705)
-854 (-3384,1676)
0.057 (0.035,0.080)
0.043 (0.019,0.067)
0.049 (0.000,0.099)
0.052 (0.006,0.100)
0.041 (-0.016,0.097)
0.020 (-0.048,0.087)
0.066 (0.044,0.088)
2859 (248,5463)
2173 (-590,4935)
2329 (-734,5368)
2424 (-566,5422)
2075 (-1066,5194)
1451 (-1869,4765)
2840 (119,5570)
0.984
0.937
0.932
0.944
0.904
0.804
0.980
posterior mean (95% credible interval)
incremental net benefits valuing QALY gains at £30,000 per QALY
posterior probability incremental net benefits are positive
MNAR1: pooled prior (23 experts)
MNAR2: pooled prior - doctors (15 experts)
MNAR3: pooled prior - nurses (8 experts)
MNAR4: prior from expert 8
MNAR5: prior from expert 16
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Graphical presentation of preliminary results
MNAR5
MNAR4
MNAR3
MNAR2
MNAR1
MAR
CC
−5000
0
5000
incremental net benefits (£)
10000
MNAR1: pooled prior; MNAR2: doctors pooled prior; MNAR3: nurses pooled prior;
MNAR4: expert 8 prior; MNAR5: expert 16 prior
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Comment
• Consensus of opinion in line with MAR
• Some difference between insights from doctors and nurses
• for the doctors, probability INB is positive = 94%
• for the nurses, probability INB is positive = 90%
• But extremes give markedly different results
• for an ‘optimistic’ expert, probability INB is positive = 98%
• for a ‘pessimistic’ expert, probability INB is positive = 80%
• However, even most pessimistic result would not change
conclusions
• so results robust
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Comment continued
• Successfully demonstrated a computational approach allowing
for informative missingness in CEA
• incorporating insights from experts provides more confidence in
conclusions
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Comment continued
• Successfully demonstrated a computational approach allowing
for informative missingness in CEA
• incorporating insights from experts provides more confidence in
conclusions
• Experts engaged with elicitation exercise
• raised awareness of difficulties posed by missing data
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Comment continued
• Successfully demonstrated a computational approach allowing
for informative missingness in CEA
• incorporating insights from experts provides more confidence in
conclusions
• Experts engaged with elicitation exercise
• raised awareness of difficulties posed by missing data
• Issues about Normal distribution assumptions in both analysis
and elicitation
• future enhancement to allow non-normal distributions
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Comment continued
• Successfully demonstrated a computational approach allowing
for informative missingness in CEA
• incorporating insights from experts provides more confidence in
conclusions
• Experts engaged with elicitation exercise
• raised awareness of difficulties posed by missing data
• Issues about Normal distribution assumptions in both analysis
and elicitation
• future enhancement to allow non-normal distributions
• Possible anchoring from starting positions of sliders
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Comment continued
• Successfully demonstrated a computational approach allowing
for informative missingness in CEA
• incorporating insights from experts provides more confidence in
conclusions
• Experts engaged with elicitation exercise
• raised awareness of difficulties posed by missing data
• Issues about Normal distribution assumptions in both analysis
and elicitation
• future enhancement to allow non-normal distributions
• Possible anchoring from starting positions of sliders
• Assumptions about missing QoL at 12 months also required
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Comment continued
• Successfully demonstrated a computational approach allowing
for informative missingness in CEA
• incorporating insights from experts provides more confidence in
conclusions
• Experts engaged with elicitation exercise
• raised awareness of difficulties posed by missing data
• Issues about Normal distribution assumptions in both analysis
and elicitation
• future enhancement to allow non-normal distributions
• Possible anchoring from starting positions of sliders
• Assumptions about missing QoL at 12 months also required
• Complements tipping point analysis
Introduction
Modelling Approaches
Elicitation
Results and Discussion
Acknowledgements
• Pinar Ulug (IMPROVE Trial Manager)
• Janet Powell (IMPROVE Chief Investigator)
• Ros Archer (assisted with piloting and brought Alfred, Bill and
Chris to life!)
• All the IMPROVE ‘Experts’ who made this research possible
• LSHTM clinical trial staff who assisted with pre-piloting
Experts 8 and 16
Joint prior
0.8
1.4
0.2
0.4
0.6
0.8
1.0
0.6
0.4
2.2
1.6
1
0.6
−0.2 0.0
0.2
−0.2 0.0
0.2
0.4
0.6
0.8
1.0
−0.2 0.0
0.2
0.4
0.6
EQ−5D
OPEN
OPEN: marginal distribution
EVAR: marginal distribution
Joint prior
0.8
1.0
0.8
1.0
0.4
EQ−5D
0.6
0.8
1.0
0.4
EVAR
0.2
6
density
−0.2 0.0
4
2
0
0.2
20
0.6
8
8
6
4
2
−0.2 0.0
10
0.8
10
1.0
10
EQ−5D
0
density
1.8
1.2
0.4
0.0
−0.2 0.0
2
2.4
0.2
EVAR
1.0
0.5
density
1.0
0.0
0.5
density
0.8
1.5
1.0
EVAR: marginal distribution
1.5
OPEN: marginal distribution
−0.2 0.0
0.2
0.4
EQ−5D
0.6
0.8
1.0
−0.2 0.0
0.2
0.4
OPEN
0.6