A TWO LEVEL MONTE CARLO APPROACH TO CALCULATING EXPECTED VALUE
OF PERFECT INFORMATION:- RESOLUTION OF THE UNCERTAINTY IN METHODS.
Alan Brennan, , Jim Chilcott, Samer Kharroubi, Tony O’Hagan, University of Sheffield, England. a.brennan@sheffield.ac.uk
Purpose: Calculating EVPI in health economic decision model enables
understanding of variables’ uncertainty and quantifies the value of
obtaining further information on the uncertain variables. Recent authors
suggest Monte Carlo simulation, but several different algorithms appear to
be used (in particular for subsets of model parameters) and there is some
confusion as to the most appropriate algorithm to be used. This paper resolves the issue,
describes a generalised two level Monte Carlo simulation approach, and identifies when the
approach is necessary. We set out a mathematical notation for overall EVPI, and EVPI for
parameter(s), and use an illustrative model to show feasibility of the method and
differences in results.
.
.
The Two level Algorithm
6) Simulate a data collection exercise for your parameter subset of interest by:
sampling each parameter of interest once from its prior uncertain range (1st level)
Cost of drug
% admissions
Days in Hospital
Cost per Day
% Responding
Utility change if respond
Duration of response (years)
9) calculate the best strategy given this new knowledge on the fixed parameter of
interest i.e. strategy which gives the most net benefit over all the simulations in (8)
% Side effects
Change in utility if side effect
Duration of side effect (years)
70%
%
Responding 80%
0.3000
Utility change0.3000
if respond
3.0
3.0 (years)
Duration
of response
£
T0
1
2%
1.00
200
Sampled Values
T1
1 £
2%
1.00
200 £
Increment
T1 (T1 over T0)
1,499 £
499
11%
0%
7.05
2.05
589 £
-
T0
1,000 £
10%
5.01
589 £
10%
-
10%
0.1000
0.5
10%
0.0500
1.0
88%
0.3119
3.1
70%
0.2120 4.1
-18%
0.0998
1.0
25%
-5%
%
Side effects 20%
-0.10
0.00
Change in utility -0.10
if side effect
0.50
Duration of side 0.50
effect (years) -
10%
0.02
0.20
5%
0.02
0.20
31%
-0.06
0.70
24%
-0.14
0.65 -
-7%
-0.08
0.05
1,208
£
Total Cost
1,695
0.6175
Total QALY
0.7100
£
487
£
1,308
£
1,937
£
629
£2,000
A “UK school” version of this algorithm is identical except that for each simulated parameter
value at step 11 we would measure the reduction in opportunity loss achieved, and at step 12
simply average the reductions in opportunity loss .
Opportunity loss = net benefit of optimal decision - net benefit of baseline decision
1 level EVPI for Parameters Approximations
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
-0.2446
3,256
-£2,570
£4,011
-£3,075
1.6
£
-£500
-£1,000
1,558
£
-£1,500
-£2,000
Inc QALY
£7,086
Overall EVPI = £1,351
Expression can be seen to be equivalent to a 1 level simulation procedure for overall EVPI
Mathematical Formulation: EVPI for Parameters
i = the parameters of interest for partial EVPI
 -i = the other parameters (those not of interest, i.e. remaining uncertainty)


E i max E i NB(d, ) |  i
d
8) EVPI (parameters) =
8) EVPI (parameters) =
the overall opportunity loss (5)
opportunity loss if parameters were
average net benefit given perfect
information on parameters of interest (7)
Equation 5: Partial EVPI for parameters i
(4) – (1)
E i max E i NB(d,  ) |  i  max E NB(d,  )
Partial EVPI =
known (7)
– net benefit of the baseline decision (3)
This is a 2 level simulation due to 2 expectations
Rationale:
•Takes account of remaining uncertainty
having obtained the information on the
parameter of interest
Rationale:
•Allows the parameter of interest to vary
across its range of uncertainty.
Implications of Further Mathematics
Why it’s wrong:
•The parameter of interest may not be
discovered at its prior mean – it could be
found at any point across the range of
uncertainty.
•Does not take account of remaining
uncertainty having obtained information
on the parameter of interest. All other
parameters should be allowed to vary.

d



(4)
(5)
% Side effects T1
Duration of side effect (years) T1
Duration of response (years) T1
Change in utility if side effect T1
Cost per Day T1
% Responding T1
Utility change if respond T1
Cost of drug T1
% admissions T1
Days in Hospital T1
% Side effects T0
Duration of side effect (years) T0
Duration of response (years) T0
Change in utility if side effect T0
Cost per Day T0
% Responding T0
Utility change if respond T0
(3)
d
£% Side effects T1
-£100
Duration of side effect (years) T1

d
£100
£0
Duration of response (years) T1

E max NB(d, )  max E NB(d, )
£200
£100
Change in utility if side effect T1
Global EVPI =
£300
£200
admissions T0
Equation 3: Overall or “Global” EVPI
(2) - (1) Net benefit of full information minus the net benefit of current information.
£400
£300
Cost of drug T0
(2)
£500
£400
Cost per Day T1

d
1 level UK
£500
2 level US
2 level UK
Days in Hospital T0
(1)
% Responding T1

£800
£700
1 level US
£600
£600
d
Equation 2: full information
With full perfect information on parameters  we always choose the optimal strategy.
Expected net benefit of full information = E max NB(d,  )
£900
£700
Utility change if respond T1
Equation 1: no further information (the value of the baseline decision)
Given current information chose decision giving maximum expected net benefit.
Expected net benefit (no further info) = max E NB(d,  )
Equation 4: Perfect Information on i
Expected Net benefit, perfect info only on i =
Why it’s wrong:
£5,267
£800
Cost of drug T1
7) For each sample of parameters, work
out the optimal strategy given that
particular “possible future world”, record
the net benefit
-0.4
• 1 level UK and US results are different, 2 level approaches are equivalent
• a linear model is accurately analysed using 1 level ‘US’.
EVPI for Parameters - 2 level methods
% admissions T1
7) For each sample of parameters, work
out the optimal strategy given that
particular “possible future world”, record
the net benefit, and the opportunity loss.
Opportunity loss =
net benefit of optimal decision
- net benefit of baseline decision
-0.6
T1 is optimal on 54.5% of simulations
Days in Hospital T1
6) Simulate a “perfectly accurate” data
collection exercise for your parameter
subset of interest by :
•allow parameter set of interest to vary
according to prior uncertainty
•keep remaining unknown parameters
constant at their prior mean value
-0.8
Illustrative Model Conclusions on EVPI for Parameter Methods:
Mathematical Formulation: Overall EVPI

= the parameters for the model (uncertain currently).
d
= set of possible decisions or strategies.
NB(d, ) = the net benefit for decision d, and parameters 
% Side effects T0
6) Simulate a “perfectly accurate” data
collection exercise for your parameter
subset of interest by :
•keeping parameter set of interest
constant at their prior mean values
•allow remaining unknown parameters
to vary according to prior uncertainty
-1
EVPI for parameters shows duration of T1 response is most important uncertainty
Change in utility if side effect T0
“US school” 1 level algorithm
EVPI for parameters
(e.g. Felli and Hazen) 3
Do steps 1 to 5 above and then ….
Baseline decision = T1
Duration of side effect (years) T0
“UK school” 1 level algorithm
EVPI for parameters
(incl. Claxton and ourselves) 1, 2
Do steps 1 to 5 above and then ….
-1.2
Illustrative Model Results:
% Responding T0
i
0.5948
£0
-1.4
Illustrative Model :2 treatments T1 versus T0, 10 parameters each on benefits, side effects and costs,
all uncertainties characterised as normal distributions, decision rule = £10,000 per QALY
Utility change if respond T0
 NetBenefitOfOptimalStrategy  NetBenefitOfBaseline Strategy
Net Benefit of T1 versus
T0 £ 5,405 £437.80
£ 4,967
Duration of response (years) T0
5) With full or “perfect” information (i.e no uncertainty in the values of each parameter) we
would always choose the optimal strategy. The overall expected value of perfect
information is therefore
10, 000
1,956
£ QALY 2,388
Cost per
0.8394
£500
Inc Cost
12) The average value of gaining perfect information on the parameter of interest is then the
average net benefit given information on the parameter (11) minus the average net benefit
given no further information i.e. of the baseline decision (3)
£
Net Benefit of T1 versus
T0
admissions T0
4) For each sample set of parameters, work out the optimal strategy given that particular
“possible future world”, record the net benefit, and the opportunity loss.
£1,000
0.0925
Cost per QALY
Cost per Day T0
3) Work out the baseline decision – i.e. strategy giving (on average over the 1,000
simulations) the highest net benefit.
11) Repeat steps 6 to 10, say 1,000 times, and calculate the average of the net benefits
obtained in (10)
Cost of drug T0
2) Simulate (say 1,000) sample sets of uncertain parameter values (monte carlo).
£1,500
Total QALY
Days in Hospital T0
1) Characterise uncertain parameters with probability distributions e.g. normal(μ,σ2),
beta(a,b), gamma (a,b), triangular(a,b,c) … etc
–
e
10) calculate the mean net benefit of this new best strategy
0) Set up a decision model comparing different strategies and set up a decision rule e.g.
“select strategy if marginal cost per QALY is < $50,000”. This is equivalent to “chose
strategy with highest net benefit” where, net benefit = $50,000 * QALY – Cost
i 1
£
8) simulate the other remaining unknown parameters 1,000 times allowing them to
vary according to prior uncertainty (2nd level)
d
Uncertainty in Parameter Means
Standard Deviations
Increment
T0
T1 (T1 over T0)
Mean Model
1,000
500
Cost of£drug 1,500 £
10%
8%
-2%
%
admissions
5.20
0.90
Days in Hospital6.10
400 per
£ Day
400 £
Cost
£
Total Cost
Agreed First Steps in the Algorithm
EVPI  PopulationToBenefit 
a
1,000
Number of Patients in the UK
Threshold cost per QALY
£10,000
Mean Model
7) set the parameter of interest fixed constant at its sampled value
Illustrative
Model
b
c
Illustrative Model
Number of Patients in the UK
Threshold cost per QALY
Conclusions
1) A 2 level monte carlo method provides a feasible and accurate solution to EVPI for parameter(s)
2)The 2 level mathematics solves the uncertainty on UK/US approximations to EVPI for parameters.
3) Both are correct using the 2 level approach!
4) Jim Chilcott has applied 2 level monte carlo in a forthcoming NICE appraisal 4
5) Further work is needed on computational issues- how many samples? Non-linear approximations?
- see Matt Stevenson, Jeremy Oakley posters
d
a) The 2 level ‘UK’ and ‘US’ expressions, are mathematically equivalent.
b) Expressions for 1 level ‘US’ and ‘UK’ approximations can be derived –
The 1-level ‘US’ approximation is accurate if net benefit is a linear function of the model
parameter subsets i and -I, .irrespective of the shape of the uncertain probability
distributions for parameters.
1 Claxton et al. Bayesian Value of Information Analysis: An application to a Policy Model of Alzheimer's Disease,
International Journal of Technology Assessment in Health Care, Volume 17 Issue 1, 2001
2 Chilcott et al. Liquid-based cytology in cervical screening: a rapid and systematic review; Health Technology
Assessment 2000; Vol. 4: No. 18
3 Felli, Hazen. Sensitivity Analysis and Expected Value of Perfect Information. Medical Dec Making 1998; 18:95-109
4 Chilcott et al. Kidney Preservation Systems: a rapid review. Health Technology Assessment 2002 (In Press)
Acknowledgements:
Co-authors: Jim, Samer and Tony (for help in thinking, working through maths and
encouragement to get it done). I am a member of CHEBS: The Centre for Bayesian
Statistics in Health Economics. Particular thanks to Karl Claxton and Tony Ades who
helped finalise our thinking at a CHEBS “focus fortnight” event in April 2002.