Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
1
2
Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models.
3
4
Alan Brennan, MSc(a)
5
Samer Kharroubi, PhD(b)
6
Anthony O’Hagan, PhD(b)
7
Jim Chilcott, MSc(a)
8
9
(a) School of Health and Related Research, The University of Sheffield, Regent Court, Sheffield S1
10
4DA, England.
11
(b) Department of Probability and Statistics, The University of Sheffield, Hounsfield Road, Sheffield S3
12
7RH, England.
13
14
Reprint requests to:
15
Alan Brennan, MSc
16
School of Health and Related Research,
17
The University of Sheffield,
18
Regent Court,
19
Sheffield S1 4DA,
20
England.
21
a.brennan@sheffield.ac.uk
22
1
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
23
ABSTRACT
24
25
Partial EVPI calculations can quantify the value of learning about particular subsets of uncertain
26
parameters in decision models. Published case studies have used different computational approaches.
27
This paper aims to clarify computation of partial EVPI and encourage its use. Our mathematical
28
description defining partial EVPI shows two nested expectations, which must be evaluated separately
29
because of the need to compute a maximum between them. We set out a generalised Monte Carlo
30
sampling algorithm using two nested simulation loops, firstly an outer loop to sample parameters of
31
interest and only then an inner loop to sample the remaining uncertain parameters, given the sampled
32
parameters of interest. Alternative computation methods and ‘shortcut’ algorithms are assessed and
33
mathematical conditions for their use are considered. Maxima of Monte Carlo estimates of expectations
34
are always biased upwards, and we demonstrate the effect of small samples on bias in computing partial
35
EVPI. A series of case studies demonstrates the accuracy or otherwise of using ‘short-cut’ algorithm
36
approximations in simple decision trees, models with increasingly correlated parameters, and many
37
period Markov models with increasing non-linearity. The results show that even if relatively small
38
correlation or non-linearity is present, then the ‘shortcut’ algorithm can be substantially inaccurate. The
39
case studies also suggest that fewer samples on the outer level and larger numbers of samples on the
40
inner level could be the most efficient approach to gaining accurate partial EVPI estimates. Remaining
41
areas for methodological development are set out.
42
43
44
45
2
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
46
Acknowledgements:
47
48
The authors are members of CHEBS: The Centre for Bayesian Statistics in Health Economics,
49
University of Sheffield. Thanks go to Karl Claxton and Tony Ades who helped our thinking at a
50
CHEBS “focus fortnight” event, to Gordon Hazen, Doug Coyle, Maria Hunink and others for feedback
51
on the poster at SMDM. Finally, thanks to the UK National Coordinating Centre for Health Technology
52
Assessment which originally commissioned two of the authors to review the role of modelling methods
53
in the prioritisation of clinical trials (Grant: 96/50/02).
54
55
3
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
56
INTRODUCTION
57
58
Quantifying expected value of perfect information (EVPI) is important for developers and users of
59
decision models.
60
sensitivity analysis (PSA)1,2 and EVPI is seen as a natural and coherent methodological extension3,4.
61
Partial EVPI calculations are used to quantify uncertainty, identify key uncertain parameters, and inform
62
the planning and prioritising of future research5. Some of the few published EVPI case studies have
63
used slightly different computational approaches6 and many analysts, who confidently undertake PSA to
64
calculate cost-effectiveness acceptability curves, still do not use EVPI. The aim of this paper is to
65
clarify the computation of partial EVPI and encourage its use in health economic decision models.
Many guidelines for cost-effectiveness analysis now recommend probabilistic
66
67
The basic concepts of EVPI concern decisions on policy options under uncertainty. Decision theory
68
shows that a decision maker’s ‘adoption decision’ should be the policy with the greatest expected pay-
69
off given current information7. In healthcare, we use monetary valuation of health (λ) to calculate a
70
single payoff synthesising health and cost consequences e.g. expected net benefit E(NB) = λ *
71
E(QALYs) – E(Costs). In general, expected value of information (EVI) is a Bayesian 8 approach that
72
works by taking current knowledge (a prior probability distribution), adding in proposed information to
73
be collected (data) and producing a posterior (synthesised probability distribution) based on all available
74
information. The value of the additional information is the difference between the expected payoff that
75
would be achieved under posterior knowledge and the expected payoff under current (prior) knowledge.
76
On the basis of current information, this difference is uncertain (because the data are uncertain), so EVI
77
is defined to be the expectation of the value of the information with respect to the uncertainty in the
78
proposed data. In defining EVPI, ‘Perfect’ information means perfectly accurate knowledge, or absolute
79
certainty, about the values of some or all of the unknown parameters. This can be thought of as
80
obtaining an infinite sample size, producing a posterior probability distribution that is a single point, or
81
alternatively, as ‘clairvoyance’ – suddenly learning the true values of the parameters.
82
information on all parameters implies no uncertainty about the optimal adoption decision. For some
Perfect
4
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
83
values of the parameters the adoption decision would be revised, for others we would stick with our
84
baseline adoption decision policy. By investigating the pay-offs associated with different possible
85
parameter values, and averaging these results, the ‘expected’ value of perfect information is quantified.
86
87
The expected value of obtaining perfect information on all the uncertain parameters gives ‘overall
88
EVPI’, whereas ‘Partial EVPI’ is the expected value of learning the true value(s) of an individual
89
parameter or of a subset of the parameters. For example, we might compute the expected value of
90
perfect information on efficacy parameters whilst other parameters, such as those concerned with costs,
91
remain uncertain. Calculations are often done per patient, and then multiplied by the number of patients
92
affected over the lifetime of the decision to quantify ‘population (overall or partial) EVPI’.
93
94
The limited health-based literature reveals several methods, which have been used to compute EVPI5.
95
Early literature9,10 used stylised decision problems and simplifying assumptions, such as normally
96
distributed net benefit, to calculate overall EVPI analytically via standard ‘unit normal loss integral’
97
statistical tables11, but gave no analytic calculation method for partial EVPI. In 1998, Felli and Hazen4
98
gave a fuller exposition of EVPI method, setting out some mathematics using expected value notation,
99
with a suggested general Monte Carlo random sampling procedure (‘MC1’) for partial EVPI calculation.
100
This procedure appeared to suggest that only the parameters of interest (ξI) need to be sampled but,
101
following discussions with the authors of this paper, this was recently corrected 12 (both ξI and ξIC
102
sampled), to show mathematical notation with nested expectations. Felli and Hazen also provided a
103
‘shortcut’ simulation procedure (‘MC2’), for use when all parameters are assumed probabilistically
104
independent and the payoff function is ‘multi-linear’. In the late 1990s, some UK case studies employed
105
a different 1 level algorithm to compute partial EVPI13,14,15, analysing the “expected opportunity loss
106
remaining” if perfect information were obtained on a subset of parameters.
107
5
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
108
Other recent papers discuss the general value of partial EVPI, comparing either with alternative
109
‘importance’ measures for sensitivity analysis16,17,18,19, or with ‘payback’ methods for prioritising
110
research5, concluding that partial EVPI is the most logical, coherent approach without discussing the
111
EVPI calculation methods required. Very few studies examine the number of simulations required, and
112
Coyle uses quadrature (taking samples at particular percentiles of the distribution) rather than random
113
Monte Carlo sampling to speed up the calculation of partial EVPI for a single parameter 17. Separate
114
literature examines the case when the information gathering itself is the intervention of interest e.g. a
115
diagnostic test or screening strategy that gathers information to inform decision making concerning an
116
individual patient20,21. Here, the value of perfect information is typically the net benefit given
117
‘clairvoyance’ as to the true disease state of an individual patient. EVPI methods in risk analysis
118
literature were also recently reviewed22. In 1999, building upon previous work by Gould23, Hilton24,
119
Howard25 and Hammitt26, a case study by Hammitt and Shlyakhter27 set out similar mathematics to Felli
120
and Hazen and examined the use of elicitation methods to quantify prior probability distributions if data
121
were sparse.
122
6,28
123
Since first presenting our mathematics and algorithm
a small number of case studies have been
124
developed. For the UK National Institute for Clinical Excellence and NCCHTA, Claxton et al. present
125
six such case studies29. In Canada, Coyle at al. have used a similar approach for the treatment of severe
126
sepsis30. Development of the approach to calculate expected value of sample information (EVSI) is also
127
ongoing31,32,33. Recent case studies include analysis of pharmaco-genetic tests in rheumatoid arthritis34.
128
Partial EVPI of course represents an upper bound on the expected value of sample information for data
129
collection on a parameter subset.
130
131
The objective of this paper is to examine the computation of partial EVPI. We mathematically define
132
partial EVPI using expected value notation, assess the alternative computation methods and algorithms,
133
investigate the mathematical conditions when the alternative computation approaches may be used, and
6
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
134
use case studies to demonstrate the accuracy or otherwise of ‘short-cut’ algorithm approximations.
135
Because a general 2 level Monte-Carlo algorithm is relatively computationally intensive, we also assess
136
whether relatively small numbers of iterations are inherently biased and investigate the number of
137
iterations required to ensure accuracy.
138
calculation in health economic decision models.
Our overall aim is to encourage the use of partial EVPI
139
140
MATHEMATICAL FORMULATION
141
142
Overall EVPI Mathematics
143
144
Let,
145

they have a joint probability distribution.
146
147
be the vector of parameters in the model. Since the components of  are uncertain,
d
denote an option out of the set of possible decisions; typically, d is the decision to adopt
or reimburse one treatment in preference to the others.
148
be the net benefit function for decision d for parameters values .
149
NB(d,)
150
Overall EVPI is the value of finding out the true value of . If we are not able to learn the value of , and
151
must instead make a decision now, then we would evaluate each strategy in turn and choose the baseline
152
adoption decision with the maximum expected net benefit. Denoting this by ENB0, we have
153
Expected net benefit | no additional information,
154
ENB0 = max E NB(d, )
d
(1)
Notice that E denotes an expectation over the full joint distribution of .
155
156
Now consider the situation where we might conduct some experiment or gain clairvoyance to learn the
157
true values of the full vector of model parameters . Then, since we now know everything, we can
158
choose with certainty the decision that maximises net benefit i.e. max NB(d,  true ) . This naturally
d
7
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
159
depends on true, which is unknown before the experiment, but we can consider the expectation of this
160
net benefit by integrating over the uncertain .
161
Expected net benefit | perfect information

162
The overall EVPI is the difference between these two (2)-(1),
163
EVPI
164

= E max NB(d,  )


d
= E max NB(d, )  max E NB(d, )
d
d
(2)
(3)
It can be shown that this is always positive.
165
166
Partial EVPI Mathematics
167
168
Now suppose that  is divided into two subsets, i and its complement c, and we wish to know the
169
expected value of perfect information about i. If we have to make decision now, then the expected net
170
benefit is ENB0 again, but now consider the situation where we have conducted some experiment to
171
learn the true values of the components of i. Now c is still uncertain, and that uncertainty is described
172
by its conditional distribution, conditional on the value of i. So we would now make the decision that
173
maximises the expectation of net benefit over that distribution. This is therefore NB(i) =
174
max E **  * NB(d, ) , whose value is again unknown prior to the experiment because it depends on i.
d


175
Taking the expectation with respect to the distribution of i therefore provides the relevant expected net
176
benefit,
177
Expected Net benefit | perfect info only on i
 
d
178
and the partial EVPI for i is the difference between (4) and ENB0, i.e.
179
PEVPI(i)
180
 

= EI max EC I NB(d, )  max E NB(d, ) :
d

= EI max EC I NB(d, )
d
(4)
(5)
This is necessarily positive and is also necessarily less than the overall EVPI.
181
8
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
182
The conditioning on i in the inner expectation is significant. In general, we expect that learning the
183
true value of i would also provide some information about c. Hence the correct distribution to use for
184
the inner expectation is the conditional distribution that represents the remaining uncertainty in c after
185
learning i. The exception is when i and c are independent, allowing the unconditional (marginal)
186
distribution of c to be used in the inner expectation. Although such independence is often assumed in
187
economic model parameters (as we do in Case Study 1), the assumption is rarely fully justified.
188
Equation (5) clearly shows two expectations. The inner expectation evaluates the net benefit over the
189
remaining uncertain parameters c conditional on i. The outer evaluates the net benefit over the
190
parameters of interest i.
191
192
Residual EVPI
193
194
Finally, we define the residual EVPI for i by REVPI(i) = EVPI – PEVPI(c)
195
REVPI(i)
196
This is a measure of the expected additional value of learning about i, if we are already intending to
197
learn about all the other parameters c. It is a measure of the residual uncertainty attributable to i, if
198
everything else were known. From a decision maker’s perspective it might be interpreted as answering
199
the question, ‘Can we afford not to know i’?


 

= E max NB(d,  )  EC max EI C NB(d,  )
d
d
(6)
200
201
COMPUTATION
202
203
Having explicitly set out the algebraic formulae for the different forms of EVPI, it is now possible to
204
identify valid ways to compute them. The key to the various approaches is how we evaluate
205
expectations. Notice that in (5) there are terms with two nested expectations, one with respect to the
206
distribution of i and the other with respect to the distribution of c given i. Although this may seem to
9
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
207
involve simply taking an expectation over all the components of , it is important that the two
208
expectations are evaluated separately because of the need to compute a maximum between the two
209
expectations. It is this that makes the computation of partial EVPI complex.
210
211
Three techniques are commonly used in statistics to evaluate expectations.
212
(a) Analytic solution.
213
It may be possible to evaluate an expectation exactly using mathematics. For instance, if X has a normal
214
distribution with mean μ and variance σ2 then we can analytically evaluate various expectations such as
215
E(X2) = μ2 + σ2 or E(exp(X)) = exp(μ + σ2/2). This is the ideal but is all too often not possible in
216
practice. For instance, if X has the normal distribution as above, there is no analytical closed-form
217
expression for E((1 + X2)-1).
218
(b) Quadrature.
219
Also known as numerical integration, quadrature is a computational technique to evaluate an integral.
220
Since expectations are formally integrals, quadrature is widely used to compute expectations. It is
221
particularly effective for low-dimensional integrals, and therefore for computing expectations with
222
respect to the distribution of a small number of uncertain variables.
223
(c) Monte Carlo Sampling.
224
This is a very popular method, because it is very simple to implement in many situations. To evaluate
225
the expectation of some function f(X) of an uncertain quantity X, we randomly sample a large number,
226
say N, of values from the probability distribution of X. Denoting these by X1;X2,: : : ;XN, we then
227
1
estimate E{f(X)} by the sample mean Eˆ { f ( X )} 
N
N
 f (X
n)
. This estimate is unbiased and its
n 1
228
accuracy improves with increasing N. Hence, given a large enough sample we can suppose that
229
Eˆ { f ( X )} is an essentially exact computation of E{f(X)}.
230
Each of these methods might be used to evaluate any of the expectations in EVPI calculations.
231
10
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
232
Two-level Monte Carlo computation
233
234
A straightforward general approach is to use Monte Carlo sampling for all the expectations. Box 1
235
displays the calculations for overall EVPI and partial EVPI in a simple, step-by-step algorithm.
236
237
Summation notation can also be used to describe these Monte Carlo sample mean calculations. If we
238
define  ki to be the vector of kth random Monte Carlo samples of the parameters of interest i, and  n to
239
be the vector of the nth random Monte Carlo samples of the full set of parameters  then
240
Overall EVPI =
241
Partial EVPI =
1
N
1

 max NB(d, ) max  L  NB(d, )
N
n 1
L
d 1toD
n
 
d 1toD
1 J
1 K 
  NB d , cj  ki
max

K k 1  d 1toD J j 1
(3s)
l
l 1
   max  L1  NB(d, )
L

d 1toD

l 1
l

(5s)
242
Where for partial EVPI, we denote by D, the number of decision policies; K, the total number of
243
different sampled values of parameters of interest i; J, the total number of sets of values for the other
244
parameters c for each given  ki ; and L, the number of sampled sets of all the parameters together when
245
calculating the expected net benefit of the baseline adoption decision.
246
247
Several important points about the algorithm in Box 1 should be noted.
248
249
Firstly, the process involves two nested simulation loops. This is because partial EVPI requires, through
250
the first term in (5), two nested expectations. Box 1 is actually a more complete and detailed description
251
of Felli and Hazen’s MC1 approach. The nested nature of the sampling is implicit in the mathematics of
252
MC1 step 2, rather than explicitly set out in the MC1 algorithm. The revised MC1 procedure12 seems to
253
suggest concurrently generating random samples of the parameters of interest (ξI) and the remaining
254
uncertain parameters (ξIC). Our algorithm shows the nested loops transparently – first sample parameters
255
of interest and only then sample the remaining uncertain parameters, given the sampled parameters of
11
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
256
interest. The MC1 procedure also assumes there is an algebraic expression for the expected net benefit
257
of the revised adoption decision given new data (step 2i). For simple decision models, algebraic
258
integration of net benefit functions can be tractable, but the inner loop in Box 1 provides a generalised
259
method for any model. The MC1 step 2ii suggests calculating the improvement (i.e. net benefit of the
260
revised minus the baseline decision) within an inner loop, which is correct, but not necessary. In fact, the
261
computation of the 2nd term can be done once for the whole decision problem, rather than within the
262
loop or for each partial EVPI. Finally, note that overall EVPI is just partial EVPI for the whole
263
parameter set, so the inner loop is redundant because there are no remaining uncertain parameters.
264
265
Secondly, in the inner loop it is important that values of c are sampled from their conditional
266
distribution, conditional on the values for i that have been sampled in the outer loop. For each sampled
267
i (in the outer loop), we need to sample (in the inner loop) many instances of c from this conditional
268
distribution. In practice, most economic models assume independence between all their parameters, so
269
that i and c are independent. In such cases, we can sample in the inner loop from the unconditional
270
distribution of c. However, not only is the assumption of independence very strong, but it is also rarely
271
justified.
272
273
Thirdly, the EVPI calculation depends on , but does not need repeating for different thresholds. If
274
mean cost and mean effectiveness are recorded separately for each strategy at the end of each inner loop
275
(the end of step 5), then partial EVPI is quick to calculate for any .
276
277
Fourthly, as always with Monte Carlo, increasing the sample sizes will increase the accuracy of
278
computation i.e. reduce the likely error (variance). In considering the number of samples required (J, K
279
and L), we need to consider the precision required of the resulting partial EVPI estimate. An equal
280
number for inner and outer sampling is not necessary and may not be efficient. We examine this in
281
detail in the later case studies.
12
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
282
283
Finally, it is important to note also that the resulting Monte Carlo estimates of overall and partial EVPI
284
are biased.
285
286
Bias of Monte Carlo Maxima
287
288
In the second term of (3) and in both terms of (5) we need to evaluate the maximum of two or more
289
expectations. If these expectations are computed by Monte Carlo, then although the estimates of the
290
expectations for each decision d are themselves unbiased, the resulting estimate of the maximum over all
291
the decisions will always be biased upwards. This is because, if X and Y are random quantities, then in
292
general max(E[X],E[Y]) is upward biased if E[X], and E[Y] are estimated with error. The bias will
293
decrease with increasing sample sizes, but for a chosen acceptable accuracy we typically need much
294
larger sample sizes when computing EVPI than when computing a single expectation.
295
296
To illustrate this, consider 2 treatments with a very simple pair of net benefit functions. NB1 =
297
20,000*1, NB2 = 19,500*2, where 1 and 2 are statistically independent uncertain parameters each
298
with a normal distribution N(1,1). Then analytically, we can evaluate max{E(NB1), E(NB2)} as
299
max{20000,19500} = 20,000. If instead we use Monte Carlo sampling to evaluate the expectations of
300
NB1 and NB2 with very small samples, then the sampling error can be seen to introduce a bias to the
301
calculation to the maximum of the two expectations. Table 1 shows the first fifteen estimates of E(NB1)
302
and E(NB2), each of which has been estimated using just L=10 Monte Carlo samples to evaluate the
303
expectations.
304
305
Insert Table 1
306
13
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
307
The variation of the values in each of the first two columns is due to Monte Carlo sampling error on the
308
10 samples. On the second bottom line of the table is the average of the full 10,000 estimates in each
309
column. For the first two columns, these are essentially the true expected net benefit values for
310
treatments 1 and 2, from which we see that treatment 1 truly has higher expected net benefit and hence
311
the true value of the maximum should be 20,000 (estimated at 20,040 by Monte Carlo). However, the
312
figure at the bottom of the third column is larger. This is the effect of the bias. To see why the bias
313
happens, notice that on lines 2, 5, 6, 12 and 14 the sampled expected net benefit for treatment 2 is higher
314
than that for treatment 1. It is these occasions, when because of Monte Carlo sampling error the
315
maximum of the estimated expectations leads to the wrong optimal decision, that create the bias. In this
316
illustration, the estimated bias with just 10 Monte Carlo samples is 3,260 i.e. an overestimate of around
317
16% in the maximum of expectations of NB1 and NB2.
318
319
It is clear, therefore, that the magnitude of this bias is directly linked to the chance of the wrong decision
320
being taken due to Monte Carlo sampling error. Increasing the sample size reduces the sampling error
321
and so reduces the bias. In the situation illustrated in Table 1, doubling the sample size to L=20 would
322
reduce the bias to about 2,200 and doubling it again to L=40 takes the bias down to about 1,560.
323
However, the sample size that is needed to reduce the bias to a negligible value depends on the
324
separation between the true expected net benefits. If the expected net benefits for competing treatments
325
are close, then this is when the bias is most marked. Unfortunately of course, this is also when value of
326
information analysis is important. For instance, if parameter subset i is to have an appreciable partial
327
EVPI, then the maximum within the first term of (5) should select different decisions as i varies. There
328
will therefore be a range of i values where the expected net benefits of different decisions are very
329
close and the bias will be appreciable.
330
331
This does not necessarily mean that Monte Carlo based partial EVPI estimates are always upward
332
biased. Taking maximums of Monte Carlo expectations, and hence the upward bias effect, occurs in
14
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
333
both terms of the computation of partial EVPI. Whether the overall bias in equation (5s) is upwards or
334
downwards will depend on the net benefit functions, the characterised uncertainty and the relative sizes
335
of J and L. Increasing the sample size J reduces that bias of the first term. Increasing the sample size L
336
reduces that bias of the second term. The size K of the outer sample in the 2-level calculation is less
337
crucial because it does not induce bias. If we compute the baseline adoption decision’s net benefit with
338
very large L, but compute the first term with very small number of inner loops J, then such partial EVPI
339
computations will be upward biased.
340
341
For overall EVPI, only the second term of the computation is biased. The first term in (3s) is unbiased.
342
Hence, the Monte Carlo estimate of overall EVPI is biased downwards.
343
344
Because of the nested loops and the potential need for large numbers of samples, the two-level Monte
345
Carlo computation of partial EVPI, although simple and general, can be computationally intensive. It is
346
feasible as long as the economic model is simple enough for the computation of net benefit NB(d,) to
347
be more or less instantaneous. The many models runs might make it impractical if the computation of
348
NB(d,) takes more than a few seconds. Because of this last point, it is important to look at alternative,
349
more efficient, computation methods.
350
351
Use of analytical expectations
352
353
In many simple economic models, it is possible to evaluate expectations of net benefit exactly,
354
particularly if parameters are independent. Suppose for instance that net benefit is simply calculated as
355
NB(d;) = λ*Qd - Rd*C where λ is the willingness to pay (assumed known), Qd is the effectiveness of
356
treatment d, Rd is the resource usage of treatment d, and C is the cost per unit of resource. The
357
parameters  of this extremely simple model comprise Qd and Rd for each d, and C. Now if Rd and C are
358
independent, simple properties of expectations imply E NB(d, ) = λ* Q d - R d * C , where Q d, R d
15
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
359
and C are the expectations of Qd, Rd and C. So in this case we can evaluate the expectation of net
360
benefit analytically, simply by running the model with the parameters set equal to their mean values.
361
For partial EVPI, analytical calculation could be undertaken both for the expectations in the 2nd term of
362
(5), and for the inner expectations in the 1st term of (5).
363
364
The same property will hold whenever the model effectively calculates net benefit as a sum (or
365
difference) of products, provided that the parameters in each product are independent. Although simple,
366
there are economic models in practice, particularly decision tree models, which are of this form.
367
However, we reiterate that the assumption of independence that is widely made in economic modelling
368
is rarely justifiable.
369
370
The ‘Short-Cut’ 1 Level Algorithm
371
372
For such models, when all parameters are independent, we can evaluate the expectation of net benefit in
373
the second term of (3) and in both terms of (5) by simply running the model with the relevant parameters
374
set to their expected values. This calculation is set out as a simple, step-by-step algorithm in Box 2,
375
which performs a one level Monte Carlo sampling process, allowing parameters of interest to vary,
376
keeping remaining uncertain parameters constant at their prior means. Note that the expectations of
377
maxima cannot be evaluated in this way. Thus, the expectation in the first term of (3) and the outer
378
expectation in the first term of (5) are still evaluated by Monte Carlo in Box 2.
379
380
Felli and Hazen4 give a similar procedure, which they term a ‘shortcut’ (MC2). It is much more
381
efficient than the two- level Monte Carlo method, since we replace the many model runs by a single run
382
in each of the expectations that can be evaluated without Monte Carlo. Felli and Hazen suggest the
383
procedure can apply successfully “when all parameters are assumed probabilistically independent and
384
the pay-off function is multi-linear i.e. linear in each individual parameter”. Mathematically, we can
16
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
385
describe the algorithm as follows. The outer level expectation over the parameter set of interest i is as
386
per equation (5), but the inner expectation is replaced with net benefit calculated given the remaining
387
uncertain parameters c set at their prior mean.
388
1 level partial EVPI for i =
 

EI max NB(d, | C  C )  max E NB(d, )
d
(7)
d
389
390
The 1-level approach is equivalent to the correct 2 level algorithm if (5)  (7), i.e.
391
if
 

EI max EC I NB(d, )
d

 
EI max NB(d, | C  C )
d

(8)
392
This is true if the left hand side inner bracket (expectation of net benefit, integrating over c) is equal to
393
the net benefit obtained when c are fixed at their prior means (i.e.
394
Felli and Hazen’s condition is sufficient to ensure that this requirement holds. Specifically, for a
395
particular parameter set of interest i it is sufficient that:
396
1. For each d, the function NB(d, ) = NB(d, i, c) is a linear function of the components of c, whose
397
coefficients may depend on d and i. If c has m components, this linear structure takes the form NB(d,
398
i, c) = A1(d, i)×c(1) + A2(d, i)×c(2) + … + Am(d, i) ×c(m) + b(d, i).
399
2. The parameters c are probabilistically independent of the parameters i.
400
In order to get conditions 1 and 2 to hold for all partitions i, c of the parameter vector , one needs:
401
1a. For each d the function NB(d, ) to be expressed as a sum of products of components of .
402
2a. The components of  are mutually probabilistically independent.
403
This specification of sufficient conditions is actually slightly stronger than the necessary condition
404
expressed mathematically in (8) but it is unlikely in practice that the one-level algorithm would correctly
405
compute partial EVPI in any economic model for which Felli and Hazen’s condition did not hold.
C  C ) in the right hand side.
406
407
Alternative 1 level Algorithm
408
17
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
409
Misunderstanding of the Felli and Hazen ‘short cut’ method has sometimes led to the use of a quite
410
inappropriate algorithm by some authors
411
overall opportunity loss inherent in a decision problem is given by the overall EVPI from equation (3).
412
The idea is that if we had perfect information on the parameters of interest this overall measure of
413
uncertainty would reduce. In order to calculate the opportunity loss remaining after perfect information
414
is gained on a subset of parameters, this alternative 1 level algorithm works with a similar 1 level
415
sampling process to that in Box 2, but keeps the parameters of interest constant at their prior mean
416
values  * , and allows the remaining unknown parameters to vary according to prior uncertainty. Finally
417
it calculates the difference between the current overall opportunity loss (3) and the opportunity loss
418
remaining after perfect information is gained that a subset of parameters have true values exactly
419
equivalent to their prior means.
13,14
. This focuses on the reduction in opportunity loss. The
420
421
The result is actually a measure of how much remaining uncertainty there would be in the decision
422
problem if we had perfect knowledge that the parameters of interest are at their prior mean values. It is
423
not actually measuring partial EVPI. Under Felli and Hazen’s conditions, in fact it computes the
424
residual EVPI (6) for i. This method should therefore not be used.
425
426
Other methods
427
428
The use of quadrature has already been mentioned as an alternative for calculating expectations. Because
429
of the large number of parameters that are typically uncertain in economic models, quadrature has
430
limited use for EVPI calculations. However, if the number of parameters in either i or c is small, then
431
quadrature can be used for the relevant computations in partial EVPI. The former situation is common,
432
where we are interested in the partial EVPI of either a single parameter or a small number of parameters.
433
Then quadrature can be an efficient method for computing the outer expectation in the second term of
434
(5). For the case where i is a single parameter, this can cut the number of values of i that need to be
18
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
435
used for this calculation from around 1000 (which is what would typically be needed for Monte Carlo)
436
to 10 or fewer.
437
438
A quite different approach is set out in by Oakley et al.35, who use a Bayesian approach based on the
439
idea of emulating the model with a Gaussian process. Although this method is technically much more
440
complex than Monte Carlo, it can dramatically reduce the number of model runs required. It should
441
therefore be the method of choice when individual model runs take more than a few seconds.
442
443
CASE STUDIES
444
445
To investigate the computation of partial EVPI further, we developed three case studies: firstly, where
446
the cost-effectiveness model is made up of simple sum-products of its statistically independent
447
parameters (a simple decision tree); secondly, where the cost-effectiveness model has increasing levels
448
correlation between model input parameters, and finally, where the cost-effectiveness model has
449
increasing levels of non-linearity (such as many period Markov models).
450
451
The first, and simplest case study is a simple decision tree model. The model compares two drug
452
treatments T0 and T1 (Figure 1). Costs for each strategy are for drugs and hospitalisations (e.g. central
453
estimate costT0 = “cost of drug”, plus “percentage admitted to hospital” x “days in hospital” x “cost per
454
day” = £1,000 + 10% x 5.20 x £400 = £1,208). QALYs gained come from response and side-effect
455
decrement (e.g. central estimate QALYT0 = “% responding” x “utility improvement” x “duration of
456
improvement”, and “% side-effects” x “a utility decrement” x “duration of decrement” = 70%
457
responders x 0.3 x 3 years + 25% side effects x –0.1 x 0.5 years = 0.6175). The decision threshold cost
458
per QALY, which we have we arbitrarily set =£10,000, enables net benefits calculations (central
459
estimate NetBenT0 = 10,000* 0.6175 – £1,208 = £4,967. The net benefit functions are therefore:
460
NBT0 = λ*(5*6*7 + 8*9*10) - (1 + 2*3*4)
19
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
461
NBT1 = λ*(14*15*16 + 17*18*19) - (11 + 12*13*4)
462
The 19 uncertain model parameters in Case Study 1 are characterised with independent normal
463
distributions with a mean (columns a, b) and a standard deviation (columns d, e). Monte Carlo sampling
464
is undertaken using EXCEL formulae (=NORMINV( rand(), mean, standard deviation) and loop macros
465
to record the input and output values for each sample – similar to producing a cost-effectiveness plane36
466
or a cost-effectiveness acceptability curve37. A single sample result is shown in columns f and g (cost
467
difference =£324, QALY = -0.1286, and so net benefit in favour of T0 in this sample). Averaging 1,000
468
sample results (Figure 1b) shows that the baseline decision given current information is to adopt strategy
469
T1. The cost effectiveness plane shows uncertainty to be larger in QALY differences than in costs,
470
whilst the CEAC at threshold = £10,000 shows T1 to be cost-effective with a probability of 54.5%.
471
472
Case study 2 uses the same model and net benefit functions, but investigates correlation. We examined
473
5 different levels of correlation (0, 0.1, 0.2, 0.3, 0.6) between 6 different parameters. Firstly, positive
474
correlations are anticipated between four of the parameters concerning the two drugs’ mean response
475
rates and the mean durations of response. Thus each of the parameters 5, 7, 14 and 16 is correlated
476
with the other three. Secondly, positive correlations are anticipated between the two drugs’ utility
477
improvements, 6 and 15. For Monte-Carlo sampling, we convert this correlation structure to a
478
variance-covariance matrix for the multi-variate normal distribution for the full parameter set 1 to 19.
479
We randomly sample the correlated values from this distribution using R software, and also
480
implemented an extension of Cholesky decomposition in EXCEL Visual Basic to create a new EXCEL
481
function =MultiVariateNormalInv (see CHEBS website)38.
482
483
Case study 3 uses an extension of the Case study 1 model incorporating a Markov model for the natural
484
history of duration of response to investigate the effects of non-linear net benefit functions. The
485
parameters for mean duration of response (7 and 16) are replaced with 2 Markov models of natural
486
history of response to each drug with health states “still responding”, “no longer responding” and “died”.
20
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
487
The new parameters are transition probabilities as set out for treatment T0 in the transition matrix M0
488
below. This matrix has as elements 6 uncertain parameters 20 to 25, and a similar matrix M1 with
489
parameters 26 to 31 exists for treatment T1. Note, 22 is defined by the other 2 parameters in its row
490
i.e. 22 = 1-20-21. Thus, the mean duration of response to each drug is a function of multiples of
491
Markov transition matrices.
492
493
Matrix Figure
494
495
We have characterised the level of uncertainty in these probabilities by assuming that each is based on
496
evidence from a small sample of just 10 transitions e.g. of 10 patients still responding at the start of
497
period 1, we have of 6 responders, 3 non responders and 1 death by the period end. We have assumed
498
statistical independence between the transition probabilities for those still responding and those no
499
longer responding. We also assumed statistical independence between the transition probabilities for T1
500
and T0. We sampled from the Dirichlet distribution in R, and also extended the method of Briggs 39 to
501
create a new EXCEL Visual Basic function =DirichletInv38.
502
503
The resulting net benefit functions are:
504
Ptotal


T
p
NBT0 =  *   ( S 0) * ( M 0) * U 0   θ 8  θ 9  10   1   2 *  3 *  4
 p 1


505
Ptotal


T
p
NBT1 =  *   ( S1) * ( M 1) * U 1   θ17  θ18  19   11   12 * 13 *  4

 p 1

506
where for treatment T0, (S0)T is the transpose of the initial response state vector i.e. (5,1-5,0) , U0 is
507
the utility improvement vector (6,0,0)T, and for treatment T1, (S1)T is the transpose of the initial
508
response state vector i.e. (14,1-14,0) , and U1 is the utility improvement vector (15,0,0)T. To
509
investigate the effects of increasingly non-linear models, we have analysed time horizons of Ptotal = 3,
510
5, 10, 15 and 20 periods. Thus, in case study 3a the mean duration of response is a function of up to




21
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
511
three multiples of the transition matrix M0 above, whilst in case study 3e it is a function of up to 20
512
multiples resulting in greater model non-linearity.
513
514
For each case study, we analysed partial EVPI for individual parameters and for groups. The groups
515
represent different types of proposed data collection exercises – a trial to obtain data on response rate
516
parameters alone (5 and 14), a utility study to obtain data on mean utility gain of responders (6 and
517
15), and a long term follow-up observational study could obtain data on duration of response to both
518
drugs (7 and 16, or in case study 3 the parameters making up M0 and M1).
519
520
CASE STUDY RESULTS
521
522
Computing with 2 level or 1 level algorithm
523
524
The partial EVPI results for each parameter subset in Case study 1 are shown in Table 2. Using 1,000
525
simulations the overall EVPI per patient is estimated at £1,352. With our assumed one thousand patients
526
affected over the lifetime of the decision, the population overall EVPI is £1,352,000, usually interpreted
527
as the expected maximum benefit to society of a research study. The 2 level partial EVPI estimates,
528
calculated with 1,000 outer and 1,000 inner samples, are very similar to the 1 level partial EVPI
529
estimates based on 1,000 samples. When expressed as a percentage of the overall EVPI, the largest
530
absolute difference between 2 level and 1 level results is 3%. The 2 level algorithm results are slightly
531
higher than the 1 level results in every case. This reflects the mathematical results shown earlier.
532
Firstly, the 1 level and 2 level EVPI calculations are mathematically equivalent because the cost-
533
effectiveness model has net benefit functions that are sum-products of statistically independent
534
parameters. And secondly, the 2 level estimates in this case are upwardly biased due to the maximisation
535
of Monte-Carlo estimate in the inner loop. In this case study, 6 of the 19 individual parameters are
536
important i.e. response rates (5 and 15), utility parameters (6 and 15) and durations of response (7
22
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
537
and 16). The remaining thirteen parameters, not shown, had zero partial EVPI. Note that in this case
538
study, the partial EVPI for groups of parameters is always lower than the sum of the EVPIs of the
539
individual parameters (e.g. utility parameters combined (6 and 15) = 57%, compared with individual
540
utility parameters = 46%+24% = 70%).
541
542
We also compared the one level opportunity loss reduction method. The results were substantially lower
543
for every individual parameter and group, sometimes less than half that of the 2 level algorithm (e.g.
544
utility parameters combined estimate = 27%), and confirming that this incorrect method could
545
substantially under-estimate partial EVPI.
546
547
TABLE 2
548
549
The case study 2 results show that if correlations are present between the parameters, then the 1 level
550
EVPI results can substantially underestimate the true EVPI. The 1 level and 2 level EVPI estimates are
551
broadly the same when small correlations are introduced between the important parameters.
552
example, with correlations of 0.1, the 2 level result for the utility parameters combined is 58%, 6 points
553
higher than the 1 level estimate. However, if larger correlations exist, then the 1 level EVPI ‘short-cut’
554
estimates can be very wrong. With correlations of 0.6, the 2 level result for the utility parameters
555
combined (6 and 15) is 18 percentage points higher than the 1 level estimate, whilst for the response
556
rate parameters combined (5 and 14) shows the maximum disparity seen, at 36 percentage points. As
557
correlation is increased, Figure 2 shows that the average disparity between 2 level and 1 level estimates
558
increases substantially.
For
559
560
FIGURE 2
561
23
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
562
For Case study 2, the 1 level EVPI results are the same no matter what level of correlation is involved.
563
This is because the 1 level algorithm sets the remaining parameters (those not of interest) at their prior
564
mean values no matter what values are sampled for the parameters of interest. The 2 level algorithm
565
correctly accounts for correlation, by sampling the remaining parameters from their conditional
566
probability distributions within the inner loop i.e. given the values for the parameters of interest sampled
567
in the outer loop. Note that these results further demonstrate that having linear or sum-product net
568
benefit functions is not a sufficient condition for the 1 level EVPI estimates to be accurate. The second
569
mathematical condition, i.e. that parameters are statistically independent, is just as important as the first.
570
It could be sensible to put the conditional mean for c given i into the 1 level algorithm rather than the
571
prior mean, but only in the very restricted circumstance when the elements of c are conditionally
572
independent given i and the net benefit function is multi-linear.
573
574
The Case study 3 results show the effects of increasingly non-linear Markov models (Table 2). The 1
575
level estimate is substantially lower than the 2 level for the trial and utility parameters. Indeed, it is
576
actually negative for the trial parameters for the 3 most non-linear case studies. This is because the net
577
benefit
578
E * max NB(d,  | C  C ) is actually lower than the second term max E NB(d, ) . Thus, when we
function is
 
d
so non-linear that the first

term
in
the
1 level
EVPI equation
d
579
set the parameters we are not interested in (c) to their prior means in term 1, the net benefits obtained
580
are lower than in term 2 when we allow all parameters to vary. We investigated the extent of non-
581
linearity for each of the Markov models 3a to 3e by expressing the net benefits as functions of the
582
individual parameters using simple linear regression and noting the resulting adjusted R 2 for each.
583
Figure 3 shows that more non-linear net-benefit functions often produce less accurate 1 level EVPI
584
estimates. However, this is not always the case. Partial EVPIs for the Markov transition probabilities
585
for duration of disease show a high degree of alignment between the 1 level and 2 level methods. It is
586
very important to note that even quite high adjusted R2 (e.g. 0.973 in 3a as compared to 0.829 in 3e)
587
does not imply that 1 level and 2 level estimates will be equal or even of the same order of magnitude.
24
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
588
For example in 3a, the 2 level EVPI for trial parameters is 30 compared with a 1 level result of 19. The
589
2 level EVPI algorithm is necessary, even in non-linear Markov models very well approximated by
590
linear regression.
591
592
FIGURE 3
593
594
Numbers of Inner and Outer Samples Required
595
The number of Monte-Carlo samples required for accurate unbiased estimates of partial EVPI has been
596
examined in case studies 1 and 3. Case study 1, (satisfying the multi-linear and independence criteria
597
for equivalence) showed 2 level results with 1000 inner samples that were always slightly higher than
598
the 1 level due to the upward bias. We examined the response parameters combined (5 and 14) with
599
larger samples. Figure 4 shows how the 2 level result changes depending on the number of inner level
600
samples.
601
602
FIGURE 4
603
604
The correct indexed result is 25. With very small numbers of inner samples the 2 level results can be
605
wrong by an order of magnitude (e.g. 10 inner samples and 1000 outer samples gives an indexed result
606
of 36). With 1,000 inner level samples the indexed result is 27 (£359) and with 10,000 inner samples the
607
2 level result is 25 (£334). Once over 500 outer level samples (K), the partial EVPI estimate converges
608
to within 1 percentage point. This contrasts with the inner level (J), where the partial EVPI shows a 4
609
point difference between 750 and 1000 samples, and where it required samples of between 5,000 and
610
10,000 for the partial EVPI estimates to converge to within 1 percentage point. The number of samples
611
needed for convergence is not symmetrical for J and K, indeed the results suggest that fewer samples on
612
the outer level and larger numbers of samples on the inner level could be the most efficient approach to
25
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
613
gaining accurate EVPI estimates. However, case study 1 does suggest that the order of magnitude of the
614
partial EVPI estimates might be stable over 100 outer and 500 inner samples.
615
616
The stability of partial EVPI estimates using 100 outer and 500 inner samples has been tested on four
617
different parameter groups using the 5 models in case study 3. Table 3 shows the variability in 2 level
618
partial EVPI estimates for different numbers of inner and outer samples.
619
620
TABLE 3
621
622
The results show that increasing the number of outer samples from 100 to 300, the standard deviations
623
fall substantially. In fact, averaging over the 20 case studies (3a to 3 e times 4 parameter subsets, using
624
100 inner samples) the standard deviations fall by a factor of 0.5329. This is in line with a reduction in
625
proportion to the square root of the number of samples i.e. reduction in standard deviation 
626
(100)/(300)=0.5774.
627
628
In contrast, the reductions in standard deviation due to increases in the number of inner samples are not
629
so marked. Again averaging over 20 case studies, the standard deviations fall by a factor of just 0.8853,
630
as we move from 100*100 to 100*500 inner samples. If the reductions were to be in proportion to the
631
square root of the number of samples, then this factor would be (100)/(500)= 0.4472.
632
demonstrates that improving the accuracy of partial EVPI estimates requires proportionately greater
633
effort on the inner level than the outer.
This
634
635
It is clear that the higher the partial EVPI, the greater the level of noise that might be expected (Figure
636
5). By plotting the standard deviations in each case study against the adjusted R2, we found that the
637
“noise” was independent of the level of non-linearity in the net benefit functions.
638
26
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
639
FIGURE 5
640
641
Of course, the acceptable level of error when calculating partial EVPI depends upon their use. If
642
analysts want to clarify broad rankings of sensitivity or information value for model parameters then
643
knowing whether the indexed partial EVPI is 62, 70 or 78 is probably irrelevant and a standard deviation
644
of 4 may well be acceptable. If the exact value needs to be established within 1 indexed percentage
645
point then higher levels of samples will be necessary.
646
647
DISCUSSION
648
649
This paper mathematically describes the correct way to calculate partial EVPI, with the evaluation of
650
two expectations, an outer expectation over the parameter set of interest and an inner expectation over
651
the remaining parameters. In specific circumstances, a ‘short-cut’ 1 level algorithm is equivalent to the
652
2 level algorithm and can be recommended for use in simple enough models. If net benefits are non-
653
linear functions of parameters or where model parameters are correlated, the 1 level algorithm can be
654
substantially inaccurate. The scale of inaccuracy often increases with non-linearity and correlation, but
655
not always predictably so in scale or direction. The formerly used alternative 1 level algorithm does not
656
measure partial EVPI (but rather residual EVPI) and so is not recommended. The 2 level Monte Carlo
657
algorithm is recommended for general use.
658
659
The number of inner and outer level simulations required depends upon the number of parameters, their
660
importance to the decision, and the model’s net benefit functions. Our empirical approach, in a series of
661
alternative models, suggests they do not need to be equal. 500 inner loops for each of the 100 outer loop
662
iterations (i.e. 50,000 iterations in total) proved capable of estimating the order of magnitude of partial
663
EVPI reasonably well in our examples, although it is likely that higher numbers may be needed in some
664
situations. For very accurate calculation or in computationally intensive models, one might use an
27
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
665
adaptive process to test for convergence in the partial EVPI results, within a pre-defined threshold.
666
Work is ongoing on a theoretical description of the Monte Carlo bias in partial EVPI calculation, and on
667
using this theory to develop algorithms to quantify the inner level sample size required for a particular
668
threshold of accuracy40. This will be the subject of forthcoming papers.
669
670
Our mathematics has also shown the existence of an over-estimating bias in evaluating maximum
671
expected net benefit across strategies using small numbers of Monte Carlo samples. This can result in
672
over or under-estimating the partial EVPI depending on the number of iterations used to evaluate the
673
first and second terms. Previous authors have investigated mathematical description of Monte Carlo
674
bias outside the EVPI context41. Further theoretical investigation of Monte Carlo bias in the context of
675
partial EVPI would be useful.
676
677
In practice, the 1 level ‘short-cut’ algorithm could be useful to screen for parameters which do not
678
require further analysis. If parameters do not affect the decision, then their partial EVPI will be very
679
close to zero using both the 2 level and the 1 level algorithm. Thus, the 1 level algorithm might be used
680
with a relatively small number of iterations (e.g. 100) to screen for groups of parameters in very large
681
models. The use of such an approach in non-linear or correlated models is a trade-off.
682
683
There is a relationship between the common representations of uncertainty, the cost-effectiveness plane
684
and the cost-effectiveness acceptability curve42, and the overall EVPI results. The cost-effectiveness
685
plane shows the relative importance of uncertainty in costs and effectiveness. Partial EVPI shows the
686
breakdown by parameter, so decision makers see clearly the source and scale of uncertainty.
687
688
Methodological research would be useful for population EVPI, which demands estimated patient
689
numbers involved in the policy decision. Incidence and prevalence are important, as are the likely
690
lifetime of the technology and potential changes in competitor strategies. There are arguments over the
28
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
691
validity of analysing phased adoption of the intervention over time explicitly versus full adoption
692
implied by the decision rule. Timing of data collection is important too. Some parameters may be
693
collectable quickly (e.g. utility for particular health states), others take longer (e.g. long term side-
694
effects), and still others may be inherently unknowable (e.g. the efficacy of an influenza vaccine prior to
695
the arrival of next years strain of influenza). There are trade-offs on the investment in different data
696
collection, quicker and cheaper versus fuller reduction in uncertainty. Again, EVSI will provide an
697
important refinement of the approach to explore these trade-offs.
698
699
EVPI is important, both in decision-making, and in planning and prioritising future data collection.
700
Policy makers assessing interventions are keen to understand the level of uncertainty, and many
701
guidelines recommend probabilistic sensitivity analysis20. This paper seeks to encourage analysts to
702
extend the approach to calculation of overall and partial EVPI. The theory and algorithms required are
703
now in place. The case study models have shown the feasibility and performance of the method,
704
indicating the numbers of samples needed for stable results. Wider application will bring greater
705
understanding of decision uncertainty and research priority.
706
29
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
707
Figure 1: Case Study 1 Model Framework and Basic Results
708
709
Part a: Model Framework
Case Study 1
a
b
c
Treatment T0 Treatment T1 Increment
Param
Param
No. Prior Mean
No. Prior Mean
710
711
712
713
Cost of drug
% admissions
Days in Hospital
Cost per Day
% Responding
Utility change if respond
Duration of response (years)
% Side effects
Change in utility if side effect
Duration of side effect (years)
Total Cost
Total QALY
Cost per QALY
Net Benefit (lamda= £10000)









 1
£1,000
2
10%
3
5.20
4
£400
5
70%
6 0.3000
7
3.0
8
25%
9 -0.1000
10
0.50
£1,208
0.6175
£4,967




 11
 12
 13
 4
 14
15
16
17
18
 19
£1,500
8%
6.10
£400
80%
0.3000
3.0
20%
-0.1000
0.50
£1,695
0.7100
£5,405
(T1 minus T0)
£500
-2%
0.90
£0
10%
0.00
0.00
-5%
0.00
0.00
£487
0.0925
£5,267
£438
d
e
Standard Deviation in
Param
No.
T0
T1
1
2%
1.00
£200
10%
0.1000
0.5
10%
0.0200
0.20
1
2%
1.00
£200
10%
0.0500
1.0
5%
0.0200
0.20
Part b: Summary Results
f










1
2
3
4
5
6
7
8
9
10
g
Sampled Values
h
Param
No.
T0
£999  11
11%  12
5.92  13
£685  4
72%  14
0.5222  15
2.7  16
30%  17
-0.1015  18
0.36  19
£1,441
1.0060
£1,432
£8,620
Increment
T1 minus T0)
T1
£1,500
£500
9%
-2%
4.34
-1.58
£685
£0
89%
17%
0.4022
-0.1200
2.5
-0.2
25%
-5%
-0.0949
0.0066
0.94
0.58
£1,765
£324
0.8774
-0.1286
£2,011
-£2,517
£7,010
-£1,610
Part c: Cost Effectiveness Plane / CEAC
£2,000
Inc Cost
Results Statistics
Mean of 1,000
Samples
£1,500
Central Estimate
Using Prior
Means
£1,000
£500
£1,210
£1,692
0.6216
0.7044
£5,006
£5,352
£483
0.08280
£5,833
£ 345
£1,208
£1,695
0.6175
0.7100
Percentage of Monte Carlo Samples When T1 is
The Correct Decision
54.5%
-
Expected Net Benefit Achievable if Perfect
Information Were Available
Value of Perfect Information
£6,704
-
£1,352
-
714
715
716
717
-1.6
-1.2
-0.8
£0
-0.4
0
-£500
0.4
0.8
1.2
1.6
Inc QALY
-£1,000
-£1,500
-£2,000
£487
0.0925
£5,267
£438
Cost Effectiveness Acceptability of T1 versus T0
Probability Cost Effective
Mean Cost T0
Mean Cost T1
Mean QALY T0
Mean QALY T1
Mean Net Benefit T0
Mean Net Benefit T1
Mean Incremental Cost (T1 v T0)
Mean Incremental QALY (T1 v T0)
Mean Incremental Cost Per QALY (T1 v T0)
Mean Incremental Net Benefit (T1 v T0)
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
£0
£10,000
£20,000
£30,000
£40,000
£50,000
£60,000
Threshold
30
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
718
719
Box 1: General 2 level Monte Carlo Algorithm for Calculation of Partial EVPI on a Parameter Subset of Interest
720
Preliminary Steps
0) Set up a decision model comparing different strategies and set up a decision rule e.g. Cost per QALY is < λ
1) Characterise uncertain parameters with probability distributions
e.g. normal(μ,σ2), beta(a,b), gamma (a,b), triangular(a,b,c) … etc
2) Simulate (say 10,000) sample sets of uncertain parameter values (Monte Carlo).
3) Work out the baseline adoption decision given current information
i.e. the strategy giving (on average over the 10,000 simulations) the highest expected net benefit.
Partial EVPI for a parameter subset of interest
The algorithm has 2 nested loops
4) Simulate a perfect data collection exercise for your parameter subset of interest by:
sampling each parameter of interest once from its prior uncertain range (outer level simulation)
5) calculate the best strategy given this new knowledge on the parameter of interest by
- fixing the parameters of interest at their sampled values
- simulating the other remaining uncertain parameters (say 10,000 times) allowing them to
vary according to their conditional probability distribution (conditional upon the parameter of
interest at its sampled value)
(inner level simulation)
- calculating the mean net benefit of each strategy
- choosing the revised adoption decision to be the strategy which has the highest expected
net benefit given the new data on the parameters of interest
6) Loop back to step 4 and repeat steps 4 and 5 (say 10,000 times) and then calculate the average net
benefit of the revised adoption decisions given perfect information on parameters of interest
7) The EVPI for the parameter of interest =
average net benefit of revised adoption decisions given perfect information on parameters (6)
minus
average net benefit given current information i.e. of the baseline adoption decision
(3)
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
Overall EVPI
The algorithm for overall EVPI requires only 1 loop
8) For each of the 10,000 sampled sets of parameters from step (3) in turn,
- work out the optimal strategy given that particular sampled sets of parameters,
- record the net benefit of the optimal strategy
9) With “perfect” information (i.e. no uncertainty in the values of each parameter) we would always
choose the optimal strategy.
Overall EVPI =
average net benefit of optimal adoption decisions given perfect information on all parameters (8)
minus
average net benefit given current information i.e. of the baseline adoption decision
(3)
737
31
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
738
739
Box 2: One level Monte Carlo Algorithm for Calculation of Partial EVPI on a Parameter Subset of Interest
Preliminary Steps ….
As in Box 1
One level Partial EVPI for a parameter subset of interest
The algorithm has 1 loop
4) Simulate a perfect data collection exercise for your parameter subset of interest by:
sampling each parameter of interest once from its prior uncertain range (one level simulation)
5) calculate the best strategy given this new knowledge on the parameter of interest by
- fixing the parameters of interest at their sampled values
- fixing the remaining uncertain parameters of interest at their prior mean value
- calculating the mean net benefit of each strategy given these parameter values
- choosing the revised adoption decision to be the strategy which has the highest
net benefit given the new data on the parameters of interest
6) Loop back to step 4 and repeat steps 4 and 5 (say 10,000 times) and then calculate the average net
benefit of the revised adoption decisions given perfect information on parameters of interest
7) The EVPI for the parameter of interest =
average net benefit of revised adoption decisions given perfect information on parameters (6)
minus
average net benefit given current information i.e. of the baseline adoption decision
(3)
740
741
32
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
742
743
Table 1: Illustration of Bias in Monte Carlo Estimates of Maxima of Several Expectations
Estimate
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
……….
Mean of 10,000
estimates
Analytic Mean
Monte Carlo Estimates of Expected Net
Benefit based on L=10 samples (‘000s)
E(NB1)
E(NB2) max{E(NB1),E(NB2)}
25.57
20.85
25.57
14.78
16.92
16.92
30.58
14.04
30.58
25.61
18.76
25.61
15.19
17.16
17.16
9.19
17.70
17.70
22.17
16.20
22.17
17.58
13.96
17.58
21.29
20.47
21.29
14.54
13.27
14.54
26.82
18.00
26.82
23.42
23.68
23.68
29.13
18.06
29.13
21.66
26.81
26.81
27.02
18.47
27.02
20.04
20
19.45
19.5
23.26
Bias
5.57
-3.08
10.58
5.61
-2.84
-2.30
2.17
-2.42
1.29
-5.46
6.82
3.68
9.13
6.81
7.02
3.26
744
745
746
33
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
747
Table 2: Case Study Results Comparing the Algorithms
---------------------------------- Partial EVPI
Parameters
Parameter No.
(Indexed - Overall EVPI = 100) ----------------------------------
Utility Duration of
Utility Duration of
Change if Response
Change if Response
% Resp T0 Respond T0
T0
% Resp T1 Respond T1
T1
5
6
7
14
15
16
Independent Linear Cross-product model
2 level
17
46
18
1 level
14
45
17
Opportunity Loss 1 Level
10
24
10
Trial
5,14
Trial +
Utility Only Utility
6,15
5,6,14,15
Overall
EVPI
Durations
7,16
All
Case Study 1
16
14
12
Case Study 2
(a) 0.1
(b) 0.2
(c) 0.3
(d) 0.6
(a) 0.1
(b) 0.2
(c) 0.3
(d) 0.6
Correlated Parameters Linear Cross-product model
2 level
14
46
21
14
2 level
15
47
28
23
2 level
24
48
28
27
2 level
26
50
46
44
1 level
12
47
16
12
1 level
12
47
16
12
1 level
12
47
16
12
1 level
12
48
16
12
Case Study 3
(a) N=3
(b) N=5
(c) N=10
(d) N=15
(e) N=20
(a) N=3
(b) N=5
(c) N=10
(d) N=15
(e) N=20
Non Linear Markov Models
2 level
2 level
2 level
2 level
2 level
1 level
1 level
1 level
1 level
1 level
-
-
-
24
23
16
59
57
32
27
24
6
57
56
27
68
67
35
66
65
33
£ 1,352
£ 1,352
£ 1,352
19
21
22
23
20
20
20
21
56
55
70
85
56
56
57
58
26
41
47
55
20
19
17
19
58
59
68
71
52
50
47
53
60
74
81
91
62
58
55
62
65
72
76
93
62
62
64
64
£
£
£
£
£
£
£
£
1,306
1,295
1,275
1,255
1,306
1,295
1,275
1,255
30
21
20
14
11
19
5
-13
-20
-26
51
69
69
44
65
64
47
25
15
6
79
84
58
46
55
69
52
28
17
9
68
76
59
82
82
54
67
75
80
81
£
£
£
£
£
£
£
£
£
£
903
1,154
1,616
1,898
2,119
903
1,154
1,616
1,898
2,119
-
-
748
749
34
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
750
751
752
Matrix Figure
End State
No Longer
Responding
Still
Responding
Start state
Still
Responding
No Longer
Responding
Died


20
0.60
23
0.00
0.00


21
0.30
24
0.90
0.00
Died


22
0.10
25
0.10
1.00
35
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
753
Figure 2: Impact of Increasing Correlation on Inaccuracy of 1 level method to calculate partial EVPI
Indexed Difference
Case Study 2: 2 level versus 1 level Partial EVPI
40
35
30
25
20
15
10
5
-
Average Disparity
Max. Disparity
0
0.2
0.4
0.6
0.8
Correlation
754
755
36
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
756
757
Figure 3: Impact of Increasing Non-Linearity on Inaccuracy of 1 level method to calculate partial EVPI
Difference between
2 level partial EVPI and 1 level estimate
Index -overall EVPI =
100
80
3e
Trial parameters
60
3d
3c
Utility Parameters
3b
40
3a
20
Trial + Utility
0
0.8
0.85
0.9
0.95
1
Markov transition
probability parameters
-20
Adjusted R sq
<----- increasing non-linearity of
net benefit function
758
759
760
37
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
Figure 4: Number of Monte Carlo Iterations and the Effect on Accuracy of EVPI estimate for Response Parameters
5,14. in an Independent Linear Cross-product model
K (Outer level)
14
15
15
500
33
27
24
24
24
750
29
26
23
24
24
1000
31
30
26
27
27
2000
32
30
27
27
27
5000
30
27
24
24
24
10000
30
27
24
25
25
600
550
500
450
400
350
300
250
200
150
100
50
0
550-600
500-550
450-500
400-450
350-400
250-300
200-250
K - outer level
766
300-350
£334
100
10
16
500
24
750
36
100
1000
1000
37
2000
750
36
5000
500
40
1000
10000
100
44
500
10
10
750
763
764
765
EVPI estimates - Impact of More Samples
10
J (Inner
Level
100
761
762
J - inner level
150-200
100-150
50-100
0-50
767
38
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
768
769
Table3 Standard Deviation in 2 level partial EVPI estimates in Case Study 3 (indexed to overall EVPI)
100
300
outer
outer
Parameter Subset
Parameter Subset

5,

6,
5, 6,
Case
5, 14 6, 15 7, 16 14, 15
Study 5, 14 6, 15 7, 16 14, 15
15.46
10.86
9.73
9.08
7.65
7.82
11.98
16.01
15.69
19.06
11.85
12.50
11.35
11.31
9.59
2.60
3.06
2.93
1.24
1.12
9.77
6.06
5.49
4.50
4.91
3.30
6.87
7.38
8.05
13.55
6.27
6.55
5.67
4.76
3.78
3a
4.36
11.04
3b
3.50
12.35
500
3c
2.31
9.73
Inner
3d
2.71
8.58
3e
2.39
6.34
Standard deviations based on 30 runs.
9.21
12.50
11.07
18.71
16.44
11.64
11.26
9.70
8.97
6.32
3.29
1.77
1.32
1.98
1.81
6.77
10.96
4.68
3.92
3.54
5.19
6.02
7.05
11.55
8.13
8.23
6.92
6.38
5.08
4.77
100
Inner
770
3a
3b
3c
3d
3e
5.66
4.79
3.75
3.09
2.56
771
772
773
39
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
774
Figure 5: Stability of partial EVPI estimates using 100 outer and 500 inner samples
Standard deviation
in Estimate
775
Standard Deviation of 100 by 500
runs
20
15
10
5
0
0
50
100
Estimated partial EVPI
776
777
778
40
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
779
REFERENCES
1 National Institute for Clinical Excellence. Guidance for manufacturers and sponsors: Technology
Appraisals Process Series No. 5. 2001. London, National Institute for Clinical Excellence.
2 Manning WG, Fryback DG, Weinstein MC. Reflecting uncertainty in cost-effectiveness analysis. In
Gold MR, Siegel JE, Russell LB, Weinstein MC, eds. Cost-Effectiveness in Health and Medicine. New
York: Oxford University Press, 1996. p.247-75.
3 Coyle D, Buxton MJ, O’Brien BJ. Measures of importance for economic analysis based on decision
modeling. Journal of Clinical Epidemiology 2003:56;989–997.
4 Felli C, Hazen GB. Sensitivity analysis and the expected value of perfect information. Medical
Decision Making 1998;18:95-109.
5 Chilcott J, Brennan A, Booth A, Karnon J, Tappenden P. The role of modelling in prioritising and
planning clinical trials. Health Technol Assess 2003;7(23).
6. Brennan, A., Chilcott, J., Kharroubi, S. A. and O'Hagan, A.. A two level Monte Carlo approach to
calculating expected value of perfect information:- Resolution of the uncertainty in methods. Poster at
the 24th Annual Meeting of the Society for Medical Decision Making, October 23rd, 2002, Baltimore.
2002. Available at http://www.shef.ac.uk/content/1/c6/03/85/60/EVPI.ppt
7 Raiffa H. Decision analysis: Introductory lectures on choice under uncertainty. 1968. AddisonWesley, Reading, Mass.
8 Spiegelhalter DJ, Myles JP, Jones DR, Abrams KR. Bayesian Methods in Health Technology
Assessment: A Review. Health Technology Assessment 2000;4.
9 Claxton K,.Posnett J. An economic approach to clinical trial design and research priority-setting.
Health Economics 1996;5:513-24.
10 Thompson KM, Evans JS. The value of improved national exposure information for
perchloroethylene (Perc): a case study for dry cleaners, Risk Analysis 1997;17:253-71.
11 Lapin, L. L. Quantitative methods for business decisions. 1994. Orlando, Florida, Dryden Press.
41
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
12 Felli JC, Hazen GB. Correction: Sensitivity Analysis and the Expected Value of Perfect Information,
Medical decision Making, 2003;23:97
13 Fenwick E, Claxton K, Sculpher M, Briggs A. Improving the efficiency and relevance of health
technology assessment: The role of iterative decision analytic modelling. CHE Discussion Paper 179,
Centre for Health Economics 2000.
14 Claxton K, Neuman PJ, Araki SS, Weinstein M. The value of information: an application to a policy
model of Alzheimer's disease. International Journal of Health Technology Assessment 2001;17:38-55.
15 Payne N, Chilcott JB, McGoogan E. Liquid-based cytology in cervical screening. Health Technology
Assessment 2000.
16 Felli JC, Hazen GB. A Bayesian approach to sensitivity analysis. Health Economics 1999;8:263-68.
17 Coyle D, Grunfeld E, Wells G. The assessment of the economic return from controlled clinical trials.
A framework applied to clinical trials of colorectal cancer follow-up. Eur J Health Econom 2003;4:6-11.
18 Karnon J, Brennan A, Chilcott J. Commentary on Coyle et al. The Assessment of the economic return
from controlled trials. Eur J Health Econom 2003;4:239-40.
19 Meltzer D. Addressing uncertainty in medical cost-effectiveness analysis—implications of expected
utility maximization for methods to perform sensitivity analysis and the use of cost-effectiveness
analysis to set priorities for medical research. J Health Econ. 2001;20:109-129.
20 Phelps CE, Mushlin AI. Focusing technology-assessment using medical decision-theory. Med Decis
Making. 1988;8:279-289.
21 Bartell SM, Ponce RA, Takaro TK, Zerbe RO, Omenn GS, Faustman EM. Risk estimation and valueof-information analysis for three proposed genetic screening programs for chronic beryllium disease
prevention. Risk Anal. 2000;20:87-99.
22 Yokota F, Thompson K. Value of Information Literature Analysis: A Review of Applications in
Health Risk Management, Medical Decision Making, 2004;24:287-98.
23 Gould J. Risk,stochastic preference and the value of information, Journal of Economic Theory,
1974;8: 64–84.
42
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
24 Hilton R. Determinants of information value, Management Science, 1981;27: 57–64.
25 Howard RA. Decision Analysis: Practice and Promise, Management Science, 1988;34:679–695.
26 Hammitt JK. Can More Information Increase Uncertainty? Chance, 1995;8:15–17.
27 Hammitt JK, Shlyakhter AI. The Expected Value of Information and the Probability of Surprise, Risk
Analysis, 1999;19:135-152.
28 Brennan, A., Chilcott, J., Kharroubi, S. A. and O'Hagan, A. (2002). A two level Monte Carlo
approach to calculating expected value of perfect information:- Resolution of the uncertainty in methods.
Available at http://www.shef.ac.uk/content/1/c6/02/96/05/Brennan.doc
29 Claxton K, Eggington S, Ginnelly L, Griffin S, McCabe C, et al. A pilot study of value of
information analysis to support research recommendations for the National Institute for Clinical
Excellence, Society for Medical Decision Making Annual Conference, (abstract), 2004.
30 Coyle, D., McIntyre, L., Martin, C., and Herbert, P. Economic analysis of the use of drotrecogin alfa
(activated) in the treatment of severe sepsis in Canada. Poster #9 presented at the 24th Annual Meeting
of the Society for Medical Decision Making, October 20th, 2002, Washington. 2002.
31 Brennan, A. B., Chilcott, J. B., Kharroubi, S., and O'Hagan, A. A two level monte carlo approach to
calculation expected value of sample information: how to value a research design. Presented at the 24th
Annual Meeting of the Society for Medical Decision Making, October 23rd, 2002, Washington. 2002.
Available at http://www.shef.ac.uk/content/1/c6/03/85/60/EVSI.ppt
32 Brennan, A., Chilcott, J., Kharroubi, S. A. and O'Hagan, A. A two level Monte Carlo approach to
calculating expected value of sample information:- How to value a research design. Discussion paper,
2002. . Available at http://www.shef.ac.uk/content/1/c6/02/96/29/EVSI.doc
33 Ades AE, Lu G, Claxton K. Expected value of sample information calculations in medical decision
modelling. Medical Decision Making. 2004 Mar-Apr;24(2):207-27.
34 Brennan, A. B., Bansback, N., Martha, K., Peacock, M., and Huttner, K. Methods to analyse the
economic benefits of a pharmacogenetic test to predict response to biological therapy in rheumatoid
43
Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models
arthritis and to prioritise further research. Presented at the 24th Annual Meeting of the Society for
Medical Decision Making, October 22nd, 2002, Washington. 2002.
35 Oakley J. Value of information for complex cost-effectiveness models. Research Report No. 533/02
Department of Probability and Statistics, University of Sheffield, 2002.
36 Briggs AH, Gray A. Handling uncertainty when performing economic evaluation of healthcare
interventions. Health Technology Assessment 1999;3.
37 Van Hout BA, Al MJ, Gordon GS, Rutten FF . Costs, effects and C/E-ratios alongside a clinical trial.
Health Economics 1994;3:309-19.
38 http://www.shef.ac.uk/chebs/news/news1.html
39 Briggs AH, Ades AE, Price MJ. Probabilistic Sensitivity Analysis for Decision Trees with Multiple
Branches: Use of the Dirichlet Distribution in a Bayesian Framework, Medical Decision Making 2003;
23:4:341-350
40 Tappenden P, Chilcott J., Eggington, S., Oakley, J., McCabe C. Methods for expected value of
information analysis in complex health economic models: developments on the health economics of
interferon-β and glatiramer acetate for multiple sclerosis. Health Technol Assess 2004;8(27).
41 Linville CD, Hobbs BF, Venkatesh BN. Estimation of error and bias in Bayesian Monte Carlo
decision analysis using the bootstrap, Risk Analysis, 2001;21:63-74.
42 Van Hout BA, Al MJ, Gordon GS, Rutten FF . Costs, effects and C/E-ratios alongside a clinical trial.
Health Economics 1994;3:309-19.
44