Value of Information for
Complex Economic Models
Jeremy Oakley
Department of Probability and Statistics,
University of Sheffield.
Paper available from
www.sheffield.ac.uk/chebs/papers.html
Outline
1. Motivation
2. Expected value of perfect information (EVPI)
3. Emulators and Gaussian processes
4. Illustration: GERD model
1) Introduction
• An economic model is to be used to predict the
cost-effectiveness of a particular treatment(s).
• The economic model will require the
specification of various input parameters. Values
of some or all of these are uncertain.
• This implies the output of the model, the costeffectiveness of the treatment is also uncertain.
Introduction
• We wish to identify which input parameters are
the most influential in driving this output
uncertainty.
• Should we learn more about these parameters
before making a decision?
Introduction
• A measure of importance for an input variable
have been proposed, based on the expected
value of perfect information (EVPI) (Felli and
Hazen, 1998, Claxton 1999).
• Computing the values of these measures is
conventionally done using Monte Carlo
techniques. These invariably require a very large
numbers of runs of the economic model.
Introduction
• For computationally expensive models, this can
be completely impractical.
• We present an efficient alternative to Monte
Carlo, in terms of the number of model runs
required.
2) EVPI
• We work with net benefit: the monetary value
or utility of a treatment is
K x efficacy – cost
with K the monetary value of a unit increase in
efficacy.
• The net benefit of any treatment option will be
a function of the parameters in the economic
model.
EVPI
•
•
•
Denote the net benefit of treatment option t
given model parameters X to be
NB (t , X )
Given X, the economic model returns
NB (t , X ) for each t .
The ‘true’ values of the model parameters X
are uncertain.
EVPI
• The baseline decision is to choose t with the
largest expected net benefit:
NB* = maxt EX {NB (t , X )}
• The decision maker will have utility NB* if they
choose the best treatment now with no
additional information.
EVPI
• Now suppose the decision-maker chooses to
learn the value of all the uncertain input
variables X before choosing a treatment.
• They would then choose the treatment with the
highest net benefit conditional on X, i.e., they
would consider
maxt {NB (t , X )}
EVPI
• Before actually observing X, they will expect to
achieve a net benefit of
EX [maxt {NB (t , X )}]
• The expected value of this course of action is
the expected gain in net benefit over the
baseline decision:
EX [maxt {NB (t , X )}] – NB*.
• This is the (global) EVPI.
Partial EVPI
• Now suppose the decision-maker chooses to
learn the value of a single uncertain input
variable Y , an element of X before making a
decision.
• They would then choose the treatment with the
highest net benefit conditional on Y , i.e., they
would consider
maxt EX | Y {NB (t , X )}
Partial EVPI
• The expected value of learning Y before Y is
actually observed is then:
EY [maxt EX |Y {NB (t , X )}] – NB *
• This is the partial expected value of perfect
information (partial EVPI) for Y .
• The partial EVPI is zero if the decision-maker
would choose the same treatment for any
(plausible) value of Y .
Computing partial EVPIs
• We need to evaluate
EY [maxt EX |Y {NB (t , X )}]
for each element Y in X.
• The outer expectation EY is a one-dimensional
integral, and can be evaluated using numerical
integration.
• The term maxt EX |Y is the maximum of (several)
higher-dimensional integrals. This requires a
large Monte Carlo sample to be evaluated.
Patient Simulation Models
• Computing partial EVPIs for computationally
cheap models, while not trivial, is relatively
straightforward.
• However, for one class of models, patient
simulation models, a sensitivity analysis using
Monte Carlo methods will be out of reach for the
model user.
Patient Simulation Models
• An example is given in Kanis et al (2002) for
modelling osteoporosis:
• For an osteoporosis patient, a bone fracture
significantly increases the risk of a subsequent
fracture.
• Residential status of a patient needs to be
tracked, in order that costs are not doublecounted.
Patient Simulation Models
• Progress is to be modelled over a 10 year
period. Including the approptiate features in the
model necessitates a patient simulation
approach.
• The net benefit for a given set of input
parameters is obtained by sampling events for a
large number of patients.
• The model takes over an hour for a single run at
one set of input parameters.
Patient Simulation Models
• For a model with 20 uncertain input variables,
computing the partial EVPI reliably using Monte
Carlo for each input variable would require a
possible minimum of 500,000 model runs.
• At one hour for each run, this would take 57
years!
• Something more efficient is needed…
3) Emulators
• For each treatment option t, and given values
for the input parameters X = x, the economic
model returns NB (t , x )
• We think of the model as a collection of
functions
NB (t , x ) = ft (x)
• Partial EVPIs can be computed more efficiently
by exploiting the `smoothness’ of each ft (x)
Emulators
• We can compute partial EVPIs more efficiently
through the use of an emulator.
• An emulator is a statistical model of the original
economic model which can then be used as a
fast approximation to the model itself.
• An approach used by Sacks et al (1989) for
dealing with computationally expensive
computer models.
Gaussian processes
• Any regression technique can be used. We
employ a nonparametric regression technique
based on Gaussian processes (O’Hagan, 1978).
• The gaussian process model for the function
ft (x) is non-parametric; the only assumption
made about ft (x) is that it is a continuous
function.
Gaussian processes
• In the Gaussian process model, ft (x) is thought
of as an unknown function, and uncertainty
about ft (x) is described by a normal
distribution.
• Correlation between ft (x1) and ft (x2) is
modelled parametrically as a function of
||x1-x2||
Gaussian processes
•
•
•
The partial EVPI for input variable Y is given by
EY [maxt EX |Y {NB (t , X )}] – NB *
We need to evaluate EX |Y {NB (t , X )} for each
t at various values of Y.
Denote G (X |Y) to be the distribution of X
given Y. Then
EX |Y {NB (t , X )} =  ft (x) dG (x |y )
Gaussian processes
• We can use Bayesian quadrature (O’Hagan,
1993) to rapidly speed up the computation:
• Under the Gaussian process model for ft (x),
 ft (x) dG (x |y )
has a normal distribution, and can be evaluated
(almost) instantaneously.
• This reduces the number of model runs required
from 100,000s to 100s.
4) Example: GERD model
• The GERD model, presented in O’Brien et al
(1999) predicts the cost-effectiveness of a range
of treatment strategies for gastroesophageal
reflux disease.
• Various uncertain inputs in the model related to
treatment efficacies, resource uses by patients.
• Model outputs mean number of weeks free of
GERD symptoms, and mean cost of treatment
for a particular strategy.
Example: GERD model
•
We consider a choice between three treatment
strategies:
›
›
›
Acute treatment with proton pump inhibitors (PPIs)
for 8 weeks, then continuous maintenance
treatment with PPIs at the same dose.
Acute treatment with PPIs for 8 weeks, then
continuous maintenance treatment with hydrogen
receptor antagonists (H2RAs).
Acute treatment with PPIs for 8 weeks, then
continuous maintenance treatment with PPIs at the
a lower dose.
Example: GERD model
• There are 23 uncertain input variables.
• Distributions for uncertain inputs detailed in
Briggs et al (2002).
• We estimate the partial EVPI for each input
variable, based on 600 runs of the GERD model.
• We assume a value of $250 for each week free
of GERD symptoms. (It is straightforward to
repeat our analysis for alternative values).
Example: GERD model
Conclusions.
• The use of the Gaussian process emulator
allows partial EVPIs to be computed
considerably more efficiently.
• Sensitivity analysis feasible for computationally
expensive models.
• Can also be extended to value of sample
information calculations.