Cost-effectiveness models
to inform trial design:
Calculating the
expected value of sample information
Alan Brennan
and
J Chilcott, S Kharroubi, A O’Hagan
Overview
• Principles of economic viability
• 2 level Monte-Carlo algorithm & Mathematics
• Calculating EVSI (Bayesian Updating) case studies
– Normal, Beta, Gamma distributions
– Others – WinBUGS, and approximations.
• Illustrative and real example
• Implications
• Future Research
Example:- Economic viability
of a proposed oil reservoir
•
•
•
•
Some information suggesting there is oil
Could do further sample drilling to “size” the oil reservoir
Decision = “Go / No go”
Criterion = expected profit (net present value or NPV)
Is the sampling worthwhile? … that depends on …
• Costs of collecting the data
• Current uncertainty in reservoir size
Expected gain from sampling =
(P big reservoir*Big profits)+(P small reservoir*Big loss)–(Sample cost)
Analogies
• Drug Development Project
–
–
–
–
Go / No go decisions
Trial supports consideration of next decision (Phases to launch)
Criterion = Expected profit (NPV)
Correct decision
 profit if good drug, avoided financial loss if not a good drug
• NICE / NCCHTA decision
–
–
–
–
Approval or not
Is additional research required before decision can be made
Criterion = Cost per QALY…. i.e. net health benefits
Correct decision  better health (efficiently) if good drug,
avoided poor health investment if not a good drug
Principles
• Strategy options with uncertainty about their performance
• Decision to make
• Sampling is worthwhile if
Expected gain from sampling - expected cost of sampling > 0
• Expected gain from sampling =
Function (Probability of changing the decision|sample,
.
amount of gain made / loss avoided)
• Applies to all decisions
Algortihm
2 Level EVSI - Research Design4, 5
0)Decision model, threshold, priors for uncertain parameters
1) Simulate data collection:
• sample parameter(s) of interest once ~ prior
• decide on sample size (ni)
(1st level)
• sample a mean value for the simulated data | parameter of interest
2) combine prior + simulated data --> simulated posterior
3) now simulate 1000 times
parameters of interest ~ simulated posterior
unknown parameters ~ prior uncertainty (2nd level)
4) calculate best strategy = highest mean net benefit
5) Loop 1 to 4 say 1,000 times Calculate average net benefits
6) EVSI parameter set = (5) - (mean net benefit | current information)
Mathematics
2 Level EVSI - Mathematics 4, 5
Mathematical Formulation: EVSI for Parameters

= the parameters for the model (uncertain currently).
d
= set of possible decisions or strategies.
NB(d, ) = the net benefit for decision d, and parameters 
Step 1: no further information (the value of the baseline decision)
Given current information chose decision giving maximum expected net
benefit.
max E NB(d,
Expected net benefit (no further info) =
(1)  )
d


i = the parameters of interest for partial EVPI
 -i = the other parameters (those not of interest, i.e. remaining uncertainty)
2 Level EVSI - Mathematics 4, 5
Step 6: Sample Information on i
Expected Net benefit, sample on i =


E X max E i NB(d, ) |  i
d
(6)

Step 7: Expected Value of Sample Information on i
(6)
E X– (1)
max E i NB(d,  ) |  i  max E NB(d,  )
d
d
Partial EVSI
=





(7)
This is a 2 level simulation due to 2 expectations
4 Brennan et al Poster
SMDM 2002
5 Brennan et al Poster
SMDM 2002
Bayesian Updating
Normal
Beta
Gamma
Normal Distribution
Normal Distribution
0= prior mean for the parameter
0= prior uncertainty in the mean (standard deviation)
I  1 /  2 = precision of the prior mean
0
0
2pop = patient level uncertainty from a sample ( needed for Bayesian
update formula)
X = sample mean (further data collection from more patients /
clinical trial study entrants).
I s  1 /  2 = precision of the sample mean .
 2   2pop / n = sample variance
4 Brennan et al Poster
SMDM 2002
5 Brennan et al Poster
SMDM 2002
Normal Distribution
 0 0   s 
1 
0   s
= implied posterior mean
(the Bayesian update of the mean
following the sample information)
2


/n  2
pop
2
 0
1   2
  2 / n 
pop
 0

= implied posterior standard deviation
(the Bayesian update of the std dev
following the sample information)
Normal Distribution - Implications
• Implied posterior variance will always be smaller than the
prior variance because the denominator of the adjustment
term is always larger than the numerator.
• If the sample size is very small then the adjustment term will
almost be equal to 1 and posterior variance is almost
identical to the prior variance.
• If the sample size is very large, the numerator of the
adjustment term tends to zero, the denominator tends to the
prior variance and so, posterior variance tends towards zero.
Normal Distribution
Normal Bayesian Update (n=50)
Prior
Frequency (1000 samples)
300
250
Posterior after
sample 1
200
150
Posterior after
sample 2
100
Posterior after
sample 7
50
0
0%
20%
40%
60%
Response Rate (T0)
80%
100%
Beta Distribution
Beta / Binomial Distribution
• e.g. % responders
• Suppose prior for % of responders is ~ Beta (a,b)
• If we obtain a further n cases, of which y are successful
responders then
• Posterior ~ Beta (a+y,b+n-y)
Gamma Distribution
Gamma / Poisson Distribution
e.g. no. of side effects a patient experiences in a year
• Suppose prior for mean number of side effects per person
is ~ Gamma (a,b)
• If we obtain a further n samples, (y1, y2, … yn) from a
Poisson distribution then
• Posterior for mean number of side effects per person ~
Gamma (a+ yi , b+n)
Bayesian Updating
Other Distributions
Other Distributions
Beta
Inverted Beta
Cauchy 1
Cauchy 2
Chi
Chi² 1
Chi² 2
Erlang
Expon.
Fisher
Gamma
Inverted Gamma
Gumbel
Laplace
Logistic
Lognormal
Normal
Pareto
Power
Rayleigh
r-Distr.
Uniform
Student
Triangular
Weibull
Bayesian Updating without a Formula
• WinBUGS
• Put in prior distribution
• Put in data (e.g. sample of patients or parameter)
• Use MCMC to generate posterior (‘000s of iterations)
• Use posterior in model to generate new decision
• Loop round and put in a next data sample
• Other approximation methods (talk to Samer!)
Illustrative Model
First (Illustrative) Model
• 2 treatments – T1 versus T0
• Criterion = Cost per QALY < £10,000
• Uncertainty in ……
• % responders to T1 and T0
• Utility gain of a responder
• Long term duration of response
• Other cost parameters
Illustrative Model
a
Number of Patients in the UK
Threshold cost per QALY
1,000
Number of Patients in the UK
£10,000
Threshold cost per QALY
Mean Model
Cost of drug
% admissions
Days in Hospital
Cost per Day
£
£
% Responding
Utility change if respond
Duration of response (years)
£
d
e
Uncertainty in Parameter Means
Standard Deviations
Increment
T0
T1 (T1 over T0)
Mean Model
1,000
500
Cost of£drug 1,500 £
10%
8%
-2%
%
admissions
5.20
0.90
Days in Hospital6.10
400 per
£ Day
400 £
Cost
70%
%
Responding 80%
0.3000
Utility change0.3000
if respond
3.0
3.0 (years)
Duration
of response
% Side effects
Change in utility if side effect
Duration of side effect (years)
Total Cost
b
c
Illustrative
Model
T0
1
2%
1.00
200
T1
1 £
2%
1.00
200 £
10%
-
10%
0.1000
0.5
10%
0.0500
1.0
25%
-5%
%
Side effects 20%
-0.10
0.00
Change in utility -0.10
if side effect
0.50
Duration of side 0.50
effect (years) -
10%
0.02
0.20
5%
0.02
0.20
1,208
£
Total Cost
1,695
£
487
£
£2,000
Total QALY
0.6175
Total QALY
0.7100
£1,500
0.0925
£1,000
Cost per QALY
£
1,956
£ QALY 2,388
Cost per
£5,267
Inc Cost
£500
£0
-1.4
-1.2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
-£500
Net Benefit of T1 versus
T0
Net Benefit of T1 versus
T0 £ 5,405 £437.80
£ 4,967
-£1,000
-£1,500
-£2,000
Inc QALY
0.4
0.6
0.8
1
1.2
1.4
£
1.6
s
a
b
c
Illustrative
Model
1,000
Number of Patients in the UK
£10,000
Threshold cost per QALY
£
£
e
Uncertainty in Parameter Means
Standard Deviations
Increment
T0
T1 (T1 over T0)
Mean Model
1,000
500
Cost of£drug 1,500 £
10%
8%
-2%
%
admissions
5.20
0.90
Days in Hospital6.10
400 per
£ Day
400 £
Cost
70%
%
Responding 80%
0.3000
Utility change0.3000
if respond
3.0
3.0 (years)
Duration
of response
£
d
T0
1
2%
1.00
200
Sampled Values
T1
1 £
2%
1.00
200 £
T0
1,000 £
10%
5.01
589 £
Increment
T1 (T1 over T0)
1,499 £
499
11%
0%
7.05
2.05
589 £
-
10%
-
10%
0.1000
0.5
10%
0.0500
1.0
88%
0.3119
3.1
70%
0.2120 4.1
-18%
0.0998
1.0
25%
-5%
%
Side effects 20%
-0.10
0.00
Change in utility -0.10
if side effect
0.50
Duration of side 0.50
effect (years) -
10%
0.02
0.20
5%
0.02
0.20
31%
-0.06
0.70
24%
-0.14
0.65 -
-7%
-0.08
0.05
1,208
£
Total Cost
1,695
£
487
£
1,308
£
1,937
£
629
£2,000
0.6175
Total QALY
0.7100
£1,500
0.0925
0.8394
£1,000
0.5948
-0.2446
3,256
-£2,570
£4,011
-£3,075
£
1,956
£ QALY 2,388
Cost per
£5,267
Inc Cost
£500
£0
-1.4
-1.2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
£
1.6
1,558
£
-£500
Net Benefit of T1 versus
T0 £ 5,405 £437.80
£ 4,967
-£1,000
-£1,500
-£2,000
Inc QALY
£7,086
Illustrative Model Results
• Baseline strategy = T1
• Cost per QALY = £5,267
• Overall EVPI = £1,351 per person
EVSI for Parameter Subsets
EVSI (n=50)
£1,400
£1,319
Value of Information
£1,200
£1,000
£867
£878
£716
£800
£600
£330
£400
£200
£-
All Six
Durations
Trial +
Utility
Utility
Study
Trial on %
Response
Value of Information
EVI for % Responders to T0
250
200
150
EVSI (n)
EVPI
100
50
0
0
50
100
Sample Size (n)
150
200
Expected Net Benefit of Sampling
Illustrative data collection cost = £100k fixed plus £500 marginal
Value of Information
Expected Net Benefit of Sampling
300
EVSI
200
100
Cost of
Sampling
0
-100 0
50
100
150
-200
Sample Size (n)
200
ENBS
Real Example
Second Example
• Pharmaco-genetic Test to predict response
• Rheumatoid Arthritis
• Up to 20 strategies of sequenced treatments
• U.S. - 2 year costs and benefits perspective
• Criterion = Cost per additional year in response
• Range of thresholds ($10,000 to $30,000)
• Real uncertainty (modelled by Beta’s)
“Biologics”
Anakinra ($12,697), Etanercept ($18,850), Infliximab ($24,112)*
Is Response Genetic?
*Costs include monitoring
Anakinra 100mg
Etanercept 25mg eow
Infliximab 3mg/kg 8 weekly
% achieving "Swollen50"
91 patients, 150mg Anakinra, 24 week RCT1,2, gene = IL-1A +4845
Positive response = reduction of at least 50% in swollen joints
100%
80%
60%
40%
20%
1 Camp et al. American Human
0%
Placebo Anakinra
100%
Gene
+ve
50%
Gene -ve
50%
Genetics Conf abstract 1088, 1999
2 Bresnihan
Arthritis & Rheumatism, 1998
A Pharmaco-Genetic Strategy
Before
0 - 6 months
Strategy 1
Gene +ve?
Yes
Respond ?
Yes
…..
No
…..
Respond?
Yes
…..
No
…..
Anakinra
PGt
No
Before
Strategy 2
Etanercept
0 - 6 months
Respond?
Yes
Anakinra
No
Partial EVSI: PGt Research only
Sample Size
Caveat: Small No.of Simulations on 1st Level
rfe
ct
Pe
50
00
20
00
10
00
50
0
20
0
10
0
50
20
$30.0
$25.0
$20.0
$15.0
$10.0
$5.0
$0.0
10
EVSI $m
EVSI for PGt Research only
(for threshold = $20,000 per responder year gained)
Doing Fewer Calculations?
Properties of the EVSI curve
• Fixed at zero if no sample is collected
• Bounded above by EVPI
• Monotonic
• Diminishing return
• Suggests perhaps exponential form?
• Tried with 2 examples –
fitted curve is exponential function of the square root of n
Fitting an Exponential Curve to EVSI:
Illustrative Model - % response to T0
EVI for % Responders to T0
Value of Information
EVSI = EVPI * [1-
EXP( -0.3813 * sqrt(n) ]
250
200
EVSI (n)
150
EVPI
100
50
Exponential fit
0
0
50
100
150
Sample Size (n)
200
Fitting an Exponential Curve to EVSI:
Pharmaco-genetic Test response
Pharmaco-genetic Test Response
EVSI = EVPI * [1- EXP( -0.3118 * sqrt(n) ]
Value of Information
30
25
20
EVSI
15
Exponential Fit
10
EVPI
5
0
0
100
200
300
400
Proposed Sample Size (n)
500
Unresolved Question
• Does the following formula always provide a good fit?
• EVSI (n) = EVPI * [1 – exp -a*sqrt(n) ]
• The 2 examples are Normal and Beta
• Is it provable by theory?
Discussion Issues
Phase III trials
Future Research Agenda
Discussion Issues – Phase III trials
• Based on proving a clinical DELTA
• Implication is that if clinical DELTA is shown then
adoption will follow i.e. it is a proxy for economic viability
• Often FDA requires placebo control (lower sample size),
which implies DELTA versus competitors is unproven
• Could consider economic DELTA …….
Discussion Issues – Phase III trials
Early “societal” economic models provide a tool for assessing:
1. What would be an economic DELTA?
2. Implied sample needed in efficacy trial for cost-effectiveness
3. What other information is needed to prove cost-effectiveness?
4. Will proposed clinical DELTA be enough for decision makers
Similar commercial economic models could link
• proposed data collection with
• probability of re-imbursement and hence with
• expected profit (NPV)
Discussion Issues – Problems
& Development Agenda
1. Technical - Bayesian Updating for other distributions
2. Partnership and case studies - to develop Bayesian tools
for researchers who currently use frequentist only sample
size calculation
3. Methods for complexity in Bayesian updating e.g. the new trial will have slightly different patient group
to the previous trial (meta-analysis and adjustment)
Conclusions
• Can now do EVSI calculations from a societal
perspective using the 2 level Monte-Carlo algorithm
• Bayesian Updating works for case studies
– Normal, Beta, Gamma distributions
– Others need – WinBUGS, and/or approximations.
• Future Research Issues
– Bayesian Technical
– Collaborative Issues with Frequentist Sample Size
Thankyou