Addressing structural uncertainty in
health economic models
Simon Thompson
University of Cambridge
Chris Jackson, Linda Sharples
MRC Biostatistics Unit, Cambridge
Health economic evaluation
Cost-effectiveness analysis
• Necessary for health policy decisions and appropriate
use of resources
• Relevant for interventions which are more effective but
also cost more
• NICE threshold c. £20,000 per QALY
Health economic models
• Typically Markov models
• Involve evidence synthesis
• Usually involve extrapolation
UCL, 15 Sept 2011
Cost-effectiveness plane:
comparing two treatments
Markov model for AAA surgical repair
Incr. cost ∆C
+
Incr. effect ∆E
Incremental cost-effectiveness ratio:
ICER = ∆C / ∆E
Incremental net benefit:
INB = ∆E – ∆C
Probability of cost-effectiveness:
PCE() = Pr (INB > 0)
n o s c re e n in v ite
s c re e n in v ite
AAA
screening
long-term
model
Health economic models
No AAA
U n d e te c te d
D e te c te d
S m a ll
AAA
S m a ll
AAA
D ro p p e d o u t
f ro m fo llo w -u p
M e d iu m
AAA
• Make many assumptions
M e d iu m
AAA
In c id e n ta l
d e te c tio n
L a rg e
AAA
L a rg e
AAA
C o n tra in d ic a te d
c o n s u lta tio n
E le c o p
p e n d in g
ru p tu re
E m e rg e n c y
o p e ra tio n
E le c tiv e
o p e ra tio n
AAA
D e a th
S u rv iv e d
s u rg e ry
• Are simplifications of clinical reality
• Are subject to both:
parameter uncertainty
and structural uncertainty
N o n -A A A
D e a th
1
Uncertainty in health economic models
Parameter uncertainty
• Affects point estimates in (non-linear) costeffectiveness models
i.e. Uncertainty about values of parameters in a given
model
• Important for confidence in decision making
Usually handled by Monte Carlo simulation, drawing
parameter values from distributions representing their
uncertainty – probabilistic sensitivity analysis (PSA)
• Relevant to whether more research should be
undertaken
“Health economic evaluation requires a full
quantification of decision uncertainty.”
e.g. independently for each parameter
e.g. jointly from posterior distribution
Generally well handled in practice
Sculpher et al, Health Econ 2006
Structural uncertainty
Abdominal aortic aneurysm (AAA)
i.e. choice of structure to represent a complex process
e.g. what clinical events / states should be included?
e.g. what risk factors for each event should be included?
e.g. how do parameters change over time?
Handled by deterministic sensitivity analyses (if at all)
Can lead to confusion in decision making
Can we do better?
Example of EVAR
Markov model for AAA surgical repair
Comparison: Endovascular aneurysm repair (EVAR) vs. open
repair for elective AAA surgery
EVAR has
lower post-operative mortality
higher rates of complications & re-interventions
higher rates of late AAA mortality
higher costs
Health economic model based principally on 4-year follow-up
data from UK EVAR1 randomised trial of 1082 patients
Jackson et al, JRSS(A) 2009
2
Some structural uncertainties for EVAR model
(a) CVD mortality in AAA patients vs. general population
Hazard ratio 2.00 (95%CI 0.83 to 4.83) or 1 ?
(b) Effect of EVAR vs. open repair on CVD mortality in 2nd year
Hazard ratio 3.06 (95%CI 1.12 to 8.36) or 1 ?
(c) Effect of EVAR vs. open repair on long-term AAA mortality
Hazard ratio 5.84 (95%CI 0.70 to 48.5) or 1 ?
Model averaging to address structural uncertainty
More complex models with more parameters may
– fit the observed data better
– predict less well
Measure of predictive ability
Akaike Information Criterion (AIC) =
–2 log likelihood + 2  number of parameters
Give more weight to models with lower AIC
Report a model-averaged result which incorporates
– BOTH uncertainty over choice of model
– AND uncertainty for each given model
Model averaging techniques
Some structural uncertainties for EVAR model
AIC = –2 log likelihood + 2p
Model weight = exp(–0.5 AIC)
BIC = –2 log likelihood + 2p log(n)
Model weight = exp(–0.5 BIC)
(a) CVD mortality in AAA patients vs. general population
Hazard ratio 2.00 (95%CI 0.83 to 4.83) or 1 ?
(b) Effect of EVAR vs. open repair on CVD mortality in 2nd year
BIC
• larger penalty for complexity
• converges to a single model choice as n increases
• appropriate if one of the models is true
AIC
• aims to select model with best predictive ability
• better predictions often come from larger models as n
increases
• appropriate if models are approximate representations
of complex process
Lifetime cost-effectiveness of EVAR vs. open repair
ICER
Hazard ratio 3.06 (95%CI 1.12 to 8.36) or 1 ?
(c) Effect of EVAR vs. open repair on long-term AAA mortality
Hazard ratio 5.84 (95%CI 0.70 to 48.5) or 1 ?
Lifetime cost-effectiveness of EVAR vs. open repair
ICER
Probability cost-effective at:
(£/QALY)
£20K/QALY
£40K/QALY
Negative
353,000
0.01
0.02
0.08
0.15
Base case
(a)
(b)
44,800
0.08
0.46
(c)
51,400
0.07
0.38
Base case
(a)
…
(a), (b) & (c)
Probability cost-effective at:
(£/QALY)
£20K/QALY
£40K/QALY
Negative
353,000
0.01
0.02
0.08
0.15
(b)
44,800
0.08
0.46
(c)
51,400
0.07
0.38
(a), (b) & (c)
19,800
0.52
0.97
Model
averaged
92,800
0.07
0.30
…
19,800
0.52
0.97
3
Incremental net benefit of EVAR vs. open
repair at threshold of £20,000 per QALY
Implantable cardioverter defibrillators (ICDs)
Aim: Estimate cost-effectiveness of ICDs vs. anti-arrhythmic
drug (AAD) amiodorone for patients at high risk of sudden
cardiac death
Data: (i) Prospective study of 535 patients with ICDs in UK
(ii) RCT of 430 patients of ICD vs. AAD in Canada
Previous evidence: Two published trials of ICD vs. AAD
Bayesian framework: MCMC (WinBUGS / WBDev)
Solid – with covariates
Dashed – without (some) covariates
Bold – model average
Markov model for patients at risk of arrhythmia
Transition parameters
allowed to depend on:
Treatment (ICD vs. AAD)
Sex
Cardiac ejection fraction
Country (UK vs. Canada)
Age (<60, 60-69, 70+)
59 parameters in base
case model
Log odds (95% CI) of death by age
Jackson et al, JRSS(C) 2010
Structural uncertainties
M1: base case (59 parameters)
M2: age effects on length of stay (+12)
M3: fewer effects on AAD toxicity (–4)
M4: fewer sex effects (–5)
M5: quadratic age (+0)
M6: cubic age (+7)
M7: quartic age (+14)
M8: quadratic age x treatment (+14)
M9: cubic age x treatment (+28)
M10: quartic age x treatment (+42)
Model averaging to address structural uncertainty
Measure of predictive ability for Bayesian models
Deviance Information Criterion (DIC)
= Deviance at posterior mean
+ 2 x effective number of parameters
= –2 log likelihood at posterior mean + 2pD
For models with weak prior information and no random
effects DIC AIC
Model weight = exp(–0.5 DIC)
Technical note: Weight for model k = probability that model k
is selected by the predictive criterion; estimated by a bootstrap
procedure, approximated by DIC-based weight
4
Lifetime cost-effectiveness of ICD vs. AAD
M1
M2
…
M7
…
M10
Prob(Mk)
ICER
(£/QALY)
PCE at
£20K/QALY
0
0.24
17,000
17,000
0.67
0.68
0.47
38,000
0.06
0.24
71,000
0.17
30,000
0.25
Model
averaged
Incremental net benefit of ICD vs. AAD at
threshold of £20,000 per QALY
10%, median, and 90% quantiles shown
Blackness  posterior density
Extrapolation uncertainty
Example: Oral cancer screening
Alternative survival models following diagnosis
2-parameter Weibull model
Other 2-, 3- and 4-parameter models
Bayesian semi-parametric Cox model
Comparison of model predictive ability
Deviance Information Criterion (DIC) for Bayesian
models
Model averaging
Represent uncertainty over model choice in addition
to parameter uncertainty given the model
Jackson et al, Int J Biostats 2010
Conclusions
Structural uncertainty may be more important than
parameter uncertainty
Model averaging is an appealing approach for
addressing some aspects of structural uncertainty
Structural uncertainty may be especially important for
age effects and extrapolation
Model averaging without data
To address structural uncertainty, can use formal
averaging across different models with weights which
• reflect model predictive ability (e.g. AIC or DIC) if data
are available
• can include prior weights as well
• reflect elicited judgements (and associated uncertainty)
if no data are available
For extrapolation, model predictive ability based on shortterm data may be of limited usefulness, and elicited
judgements are necessary
Bojke et al, Value in Health 2010
Jackson et al, Med Decision Making 2011
References
Jackson CH, Thompson SG, Sharples LD. Accounting for uncertainty
in health economic decision models using model averaging. JRSS(A)
2009; 172: 383-404.
Jackson CH, Sharples LD, Thompson SG. Structural and parameter
uncertainty in Bayesian cost-effectiveness models. JRSS(C) 2010;
59: 233-253.
There are many unresolved issues: e.g.
Jackson CH, Sharples LD, Thompson SG. Survival models in health
economic evaluations: balancing fit and parsimony to improve
prediction. International Journal of Biostatistics 2010; 6: article 34.
- Choice of models to average over
- How to elicit judgements in real-life policy making
- Handling other structural uncertainties (e.g. choice of states)
- Better addressing uncertainty due to extrapolation
Jackson CH, Bojke L, Thompson SG, Claxton K, Sharples LD. A
framework for addressing structural uncertainty in decision models.
Medical Decision Making 2011; 31: 662–674.
5