Transforming the efficiency
of Partial EVSI computation
Alan Brennan
Health Economics and Decision Science (HEDS)
Samer Kharroubi
Centre for Bayesian Statistics in Health Economics
(CHEBS)
University of Sheffield, England
IHEA July 2005
a.brennan@sheffield.ac.uk
s.a.kharroubi@sheffield.ac.uk
1
Expected Value of Sample
Information (EVSI)
•
•
EVSI works out the expected impact on
decision making if we collect more data
We
1. Simulate a collected sample dataset
2. Update uncertainty in parameters given data
3. ? Choose a different decision option given data
4. Quantify increase in benefit over baseline decision
5. Repeat for many sample datasets
6. Calculate the expected increase in benefit
2
EVSI
The Computational Problem
•
•
EVSI works out the expected impact on
decision making if we collect more data
Conventional Computations required
1. “Outer” Monte Carlo sample
2.
Bayesian Update – analytic or MCMC
3.
“Inner” Monte Carlo sample e.g. 10,000 times
4.
Evaluate each net benefit function each time
5. Repeat for many sample datasets e.g. 10,000 times
6. Total e.g. 100,000,000 evaluations of net benefit
3
Mathematical Notation
EVSI =


E Xi max E NBt ,  | X i   max E NB(t,  )
t
Expectation
over sampled
datasets
Expected Payoff for each
Decision given particular
new data Xi
t
Expected Payoff given
only current information

= uncertain model parameters
t
= set of possible treatments (decision options)
NB(d, ) = net benefit (λ*QALY – Cost) for decision d, 
i
= parameters of interest – possible data collection
Xi
= data collected on the parameters of interest i
4
Laplace approximation
• Sweeting and Kharroubi (2003) developed a 2nd
order approximation to evaluate the posterior
expectation of any real valued smooth function
v() with a vector of d uncertain parameters 
given new available data X.
 
 
^
^ d  - 




E v  | X   v     i v  i   i v  i  v  
  i1 
 
------
1st order
term
---------------------------------2nd order
term
5
Eureka
• For EVSI the first term in the formula is

E X max E NB(t , ) | X 
t
• We can adapt Laplace approximation to evaluate
the EVSI inner expectation !




 ^  d   ^ 



E NBt ,  | X   NB t ,     i NB t , i   i NB t , i  NB t ,  
  i 1 
 
-------- ----------------------------------------1st order
2nd order
• Only requires 1+3d evaluations of net benefit
(Kharroubi and Brennan 2005)
6
Univariate Explanation: +
• + and - are 1 standard deviation away from
the posterior mode ^

Posterior Probability Density Function
0.07
0.06
0.05
θ-
0.04
0.03
^

θ+
0.02
0.01
0
-4
-3
-2
-1
0
1
2
3
4
7
Univariate Explanation: α+
• α+ and α- are weights, functions of the ratio of
the slopes of the log density function at θ+, θLog (Posterior Density)
0
-4
-3
-2
-1
θl'(θ-)
0
-2
-4
-6
-8
1
2
θ+
l'(θ+)
3
4
1
 
 '    
1 

 '    
If distribution
is symmetric
then
α+ = α - =½
8
Multivariate Requires Matrix
Algebra for each dataset Xi
• θi+, θi- are vectors.
• each is the i th row of a matrix θ+, θ-
^i-1
› The first i -1 components are posterior modes^
θ1 ...θ
› i th is ^
θi ± (ki)-1/2 , where ki is 1/first entry of {J(i)}-1
› Remaining i +1 to d components are chosen to maximise the
posterior density given the first i components
• αi+ and αi- are vectors of weights, which are calculated
based on partial derivatives of the log posterior density
function at θi+, θi• Requires numerical optimisation
9
Case Studies
• Case Study 1
›
›
›
›
2 treatments – T1 versus T0
Uncertainty in …… 19 independent parameters
Univariate Normal prior and data
Net benefit function is sum-product form
› NB1=  (θ5θ6θ7+θ8θ9θ10) – (θ1+ θ2θ3θ4 )
• Case Study 2
› Uncertainty in …… 19 correlated parameters
› Multivariate Normal prior and data
10
Illustrative Model
Illustrative Model
Parameters
Cost of drug
% admissions
Days in Hospital
Cost per Day
% Responding
Utility change if respond
Duration of response (years)
a
b
c
d
Parameter Mean Values given
Existing Evidence
e
f
g
Uncertainty in Means Patient Level Variability
Standard Deviations Standard Deviations
T0
T1
(T1 - T0)
T0
T1
T0
T1
£ 1,000 £ 1,500 £
500 £
1 £
1 £
500 £
500
10%
8%
-2%
2%
2%
25%
25%
5.20
6.10
0.90
1.00
1.00
4.00
4.00
£
400 £
400 £
£
200 £
200 £
200 £
200
70%
0.3000
3.0
80%
0.3000
3.0
10%
-
10%
0.1000
0.5
10%
0.0500
1.0
20%
0.2000
1.0
20%
0.2000
2.0
% Side effects
25%
20%
-5%
Change in utility if side effect
-0.10
-0.10
0.00
Duration of side effect (years)
0.50
0.50
Total Cost
£ 1,208 £ 1,695 £
487
Total QALY
0.6175
0.7100
0.0925
Cost per QALY
£ 1,956 £ 2,388 £ 5,267
Net Benefit
(threshold = £10,000 per QALY)
£ 4,967 £ 5,405 £
438
10%
0.02
0.20
5%
0.02
0.20
20%
0.10
0.80
10%
0.10
0.80
11
Case Study 1 Results (5 sets)
1st order Laplace is accurate
EVSI :- 1st Order Laplace Approximations (150,000
simulations) versus Monte Carlo (1,000 * 1,000)
£1,400
£1,200
EVSI (£)
£1,000
£800
£600
£400
£200
£0
0
50
100
150
200
250
Sample Size (n)
12
Case Study 2: 1st order wrong
2nd order is accurate
Partial EVSI: Param eters 6 & 15
EVSI (£)
500
400
2level Monte Carlo
10,000 * 1,000
300
1st order Laplace
200
2nd order Laplace
100
0
0
50
100
150
200
250
Sam ple Size
13
Accuracy of inner integral
approximation
• Parameters 6,15
• Sample size n=50
Laplace
Monte Carlo
• Out of 1000 datasets
the resulting decision
between 2 treatments
was different in 7
i.e. 0.7% error
14
Trade-off in Computation
Time
Number of outer samples
Time to produce 1 outer parameter
sample, 1 sample dataset and do
Bayesian update parameters | data
(around 120 per second)
Time to produce 1 outer parameter
sample, 1 sample dataset and
Evaluate theta and alpha's | data
Time for 1 net benefit function
evaluation (around 700 per second)
No. of net benefit function evaluations
to quantify inner expectation
Total Computation Time (Hours)
Efficiency (Ratio of CPU Times)
2 level
10,000
Laplace
10,000
0.00858
-
0.00144
10,000
39.9
4.6
3.03000
0.00144
58
8.6
Pentium 4 1.8GHz, 512Mb RAM
15
Computation Time
What-If Analyses
• Efficiency gain due to Laplace approximation
increases rapidly as model run time for one
evaluation of net benefit increases
Time to
evaluate
net benefit
function
(seconds)
Time for outer sample and Bayes update
(seconds)
4.6
0.00858
0.1
1
60
0.00144
4.6
4.6
4.9
23.9
0.01
27.7
27.7
28.0
44.3
0.1
113
113
113
120
1
164
164
164
165
16
Limitations
• Any Type of Net benefit function
› analytic function of model parameters
› result of probabilistic model e.g. individual level
simulation
• Characterisation of Uncertainty
›
›
›
›
Need functional form for probability density function
Smooth and differentiable,
i.e. not just a histogram to sample from
write down the equations for posterior density
function and its derivative mathematically
17
Conclusions
•
•
•
EVSI calculations using the Laplace
approximation are in line with those using 2
level Monte-Carlo method in case studies so far
Method is very generalisable once you
understand the mathematics and algorithm
Computation time reductions depend on times
to compute net benefit functions
18
Thankyou
• 'Wisest are they
who know they
do not know‘
• ‘Especially if they
can calculate
whether it’s
worth finding out’
19
References
•
•
•
•
Brennan, A. B., Chilcott, J. B., Kharroubi, S. A, O'Hagan, A. A Two Level
Monte Carlo Approach to Calculation Expected Value of Sample
Information: How To Value a Research Design. Presented at the 24th
Annual Meeting of SMDM, October 23rd, 2002, Washington. 2002.
http://www.shef.ac.uk/content/1/c6/03/85/60/EVSI.ppt
Ades AE, Lu G, Claxton K. Expected value of sample information
calculations in medical decision modelling. Medical Decision Making. 2004
Mar-Apr;24(2):207-27.
Sweeting, T. J. and Kharroubi, S. A. (2003). Some new formulae for
posterior expectations and Bartlett corrections. Test, 12(2): 497-521.
Kharroubi, S. A. and Brennan, A. (2005). A Novel Formulation for
Approximate Bayesian Computation Based on Signed Roots of Log-Density
Ratios. Research Report No. 553/05, Department of Probability and
Statistics, University of Sheffield. Submitted to Applied Statistics.
http://www.shef.ac.uk/content/1/c6/02/56/37/Laplace.pdf
20