Value of Information Calculations to
Inform and Prioritize
Clinical Research Investments
David Meltzer MD, PhD
The University of Chicago
Background
• Cost-effectiveness analysis has long been used to
assess the value of medical treatments and the
information that comes from diagnostic tests
– Cost-effectiveness measured in Cost/QALY
– Net health benefits
= gain in QALYs – opportunity cost of spending in QALYs
– Net monetary benefit
= $ value of improved health - costs
• Newer value of information techniques have extended
these tools to assess the value of medical research
Value of Information Approach to Value of Research
• Without information
– Make best compromise choice not knowing true state of the
world (e.g. don’t know if intervention is good, bad)
• With probability p:
get V(Compromise|G)
• With probability 1-p: get V(Compromise|B)
• With information
– Make best decision knowing true state
• With probability p:
get V(Best choice|G)
• With probability 1-p: get V(Best choice|B)
• Value of information
= E(outcome) with information - E(outcome) w/o information
= {p*V(Best choice|G) + (1-p)*V(Best choice|B)} {p*V(Compromise|G) + (1-p)*V(Compromise|B)}
= Value of Research
Practical Applications of Value of Information
• VOI requires modeling population value of information
VOI 
where

 t  D (t )  I (t )  N t  IV O I
t
 t is time preference discount factor
D ( t ) is depeciation of knowledge over time
I (t ) is extent of implementation
N t is number of eligible individuals in each cohort
IVOI is individual VOI
• VOI based on decision models
– IVOI modeled with decision model
– UK (NICE): Alzheimer’s Disease Tx, wisdom teeth removal
• Minimal modeling approaches to VOI
– IVOI comes (nearly) directly from clinical trial
– US (NIH): CATIE Trial of atypical antipsychotics
• Bound with more limited data (burden of illness)
Information Requirements for Value of Information Calculations
(Meltzer. J Health Econ 2003)
Information Required
Conceptual
Basis
Expected Gain
in Welfare
from Research
Expected Gain
Expected Value from Perfectly
of Perfect
Informative
Information
Specific
Experiment
Maximum
Maximum Value Possible Gain
of Information
from Specific
Experiment
Maximum Value
Maximum
of (DiseasePossible Gain
Specific)
for Target
Research
Disease
Expected Value
of Information
Priors for Posteriors
Burden of
Subject of for Subject
Illness
Research of Research
Yes
Yes
Yes
Yes
Yes
Yes
Minimal
Bounds
Yes
Missing
Elements
Serendipity
Serendipity,
Likelihood
Potential
Gains
Serendipity,
Likelihood
Potential
Gains
Serendipity,
Likelihood
Potential
Gains
AHRQ EPC Methods Paper: Minimal Modeling Approaches to VOI
(Forthcoming, Medical Decision Making)
Approaches
Full Modeling
No Modeling
Limited
Modeling
Definitions*
Full characterization of the
disease/ treatment using a
decision model or other
simulation model of
relevant health state
Direct replication or direct
calculation of
(incremental) effects on
comprehensive health
outcomes (e.g. QALYs,
and/or net benefits)
Any modeling necessary
(e.g., modeling of patient
survival, mapping of
treatment effect to utilities
or aggregate
approximation of costs)
without using a decision
model or other simulation
model of relevant health
states
VOI Calculations
Simulation/
bootstrapping,
parametric and/or
nonparametric
Equation-based
computation,
parametric
Simulation/
bootstrapping,
parametric and/or
nonparametric
Equation-based
computation,
parametric
Simulation/
bootstrapping,
parametric and/or
nonparametric
Equation-based
computation,
parametric
Data
Requirements
Data on all model
parameters
Clinical
Application(s)
Chronic conditions,
complex diseases
Distributions of
comprehensive
health outcomes
or, QALYs and/or
net benefits
Acute conditions,
end of life
treatments
Direct
measurement of
final health
outcomes
Intermediate
measures for
health outcomes
or QALYs, costs
and/or NBs;
Survival data
Acute conditions,
end of life
treatments
Advantages (+) and
Disadvantages (-)
- Complex and time-consuming modeling
exercises
+ Detailed uncertainty analysis and VOI
estimates, including calculation of
EVPPI
+ No need for complex and timeconsuming modeling
+ Complementary to adaptive clinical trial
design
- Requires clinical trial that can provide
comprehensive measure of net benefit
- No comprehensive uncertainty analysis
and VOI estimates (EVPPI)
+ Reduced need for complex and timeconsuming modeling
+ Complementary to adaptive clinical trial
design
- Requires clinical trial that can requires
only modeling of survival or other limited
modeling to generate comprehensive
measure of net benefit
- No comprehensive uncertainty analysis
and VOI estimates (EVPPI)
* All approaches seek to address specific treatment or coverage decisions, to characterize decision uncertainty and to establish VOI estimates
EVPPI: expected value of partial perfect information
Full Modeling:
“Bayesian Value of information analysis: An
application to a policy model of Alzheimer's disease.”
Uncertainty in Incremental Net Benefits
Value of Research by Value of Health
Contributors to Value of Research
Practical Applications of Value of Information
• VOI requires modeling population value of information
VOI 
where

 t  D (t )  I (t )  N t  IV O I
t
 t is time preference discount factor
D ( t ) is depeciation of knowledge over time
I (t ) is extent of implementation
N t is number of eligible individuals in each cohort
IVOI is individual VOI
• VOI based on decision models
– IVOI modeled with decision model
– UK (NICE): Alzheimer’s Disease Tx, wisdom teeth removal
• Minimal modeling approaches to VOI
– IVOI comes (nearly) directly from clinical trial
– US (NIH): CATIE Trial of atypical antipsychotics
• Bound with more limited data (burden of illness)
Limited Modeling: The Clinical Antipsychotic Trials in
Intervention Effectiveness (CATIE)
• $42.6 million, NIMH-funded randomized trial of Atypical
Antipsychotic Drugs (A-APDs) and a Neuroleptic (Perhphenazine) in
patients with established schizophrenia
• Major findings
o Discontinuation rates similar with A-APDs and Perphenazine
o Perphenazine cost-effective first-line treatment
• Impact
o Frequently discussed in coverage decisions
o Some have argued results should be considered definitive
CATIE Cost-Effectiveness Results
Monthly Costs
Mean (sd) ($)
QALY Mean
(sd)
ICER
($/QALY)
Perphenazine
817 ( 728)
0.722 (0.0064)
Olanzapine
1619 (1442)
0.723 (0.0063)
9,624,000
Risperidone
1635 (1457)
0.706 (0.0066)
Dominated
Quetiapine
1680 (1497)
0.721 (0.0065)
Dominated
-
(Ref: Rosenheck et al , 2006; Private Communications with Dr. Rosenheck)
Only statistically significant difference:
QALYPerphenazine > QALYRisperidone (p-val < 0.001)
Simulated Distribution of Mean QALYS
(Based on uncertainty around CATIE results)
Density
Olanzapine: 0.723 (0.0063)
Quetiapine: 0.721 (0.0065)
Risperidone: 0.706 (0.0066)
Perphenazine: 0.722 (0.0064)
.65
.7
.75
E(QALY)/per patient per year
.8
.85
Simulated Distribution of Mean Costs
(Based on uncertainty around CATIE results)
Density
Olanzapine: $1606 (1421)
Quetiapine: $1685 (1485)
Risperidone: $1621 (1439)
Perphenazine: $ 810 ( 723)
0
5000
10000
E(QALY)/per patient per year
15000
Realizations of Value of Research Over Time
12
Value of Future Research to Prevalent and Incident Cohorts
at $50k/QALY
0
2
Value (in Billion $)
4
6
8
10
Incident in 2012-2036
Incident in 2011
Incident in 2010
Incident in 2009
Incident in 2008
Incident in 2007
Prevalent Cohort
2007
2017
2027
2037
2047
YEAR
2057
2067
2077
2087
Total Value to Prevalent Cohort: $207 billion
Total Value to Each Incident Cohort: $6.6 billion
Total Value to Prevalent & Next 20 Incident Cohorts: $342 billion
EVSI (in Billions)
325
350
Net Expected Value of Sample Information
(at $50K, $100K and $150K/QALY)
|
|
|
|
|
300
|
|
v
at $50K/QALY
at $100K/QALY
at $150K/QALY
5000
10000 15000 20000 25000 30000 35000 40000 45000 50000
Sample size for each arm
Cost of Research: $3 mill + (sample size*4)*($5000/month)*18 months
Optimal sample size for each arm = 22,500
No Modeling Approach:
Azithromycin vs. Augmentin in Acute Sinusitis
• Azithromycin more costly but easier and more effective
• RCT (Marple et al 2010)
– Primary outcome resolution of symptoms within 5 days
• 70/236 patients (29.7%) in the azithromycin extended-release arm
• 45/238 patients (18.9%) in the amoxicillin/clavulanate arm
• Difference: 10.8%; 95% confidence interval [CI]: 3.1–18.4%
– By day 28:
• 26/236 patients (11.0%) in the azithromycin extended release arm
• 27/238 patients (11.3%) in the amoxicillin/clavulanate arm
• Difference: -0.4%; 95% CI: -6.1% to 5.3%
– Completion of trial to equal resolution is key to avoid building
model
Estimation of Symptom Duration
P r o p .o2 r t i o n w i t h . 4c o n t i n u e d . 6A B S s y m . p8 t o m s
1
Appendix B. Curves for symptom resolution in azithromycin extended release versus amoxicillin/clavulanate in acute
sinusitis
Difference:
Azithromycin
7.55 (SE 0.260) days
vs.
Amoxicilin/Clavulanate
8.12 (SE 0.210) days
Estimated Hazard ratio = 1.13 (95% CI: 0.91-1.42)
0
Amoxicillin/Clavulanate (N=238)
Azithromycin Extended Release (N=236)
0
ORIGINAL
2
4
6
8
10
Days after enrollment
TRANSCRIBED AND ESTIMATED
12
14
Gemcitabine + Erlotnib vs
Gemcatabine in Pancreatic CA
100
Appendix C. Survival curves for gemcitabine plus erlotinib versus gemcitabine alone in metastatic pancreatic cancer
Survival Probability (%)
40
60
80
Estimated HR = 0.82
95% CI: (0.69, 0.97)
p-value = 0.023
Erlotinib (n=285)
0
20
Placebo (n=284)
0
ORIGINAL
Moore et al. J Clin Oncol 2007; 25(15): 1960-1966.
6
12
Time (months)
TRANSCRIBED AND ESTIMATED
18
24
VOI Estimation
•
Cost-Effectiveness Results and Decisions Based on Current Information
– Azithromycin mean time to resolution of 7.55 days (SE=0.260) vs. 8.12 days
(SE=0.210) for amoxicillin/clavulanate (p value for difference 0.077).
– WTP $73.2/day (SE=6.66) in 2010 U.S. dollars (Tolley et al.)
– Cost (AWP Drug Topics Red Book, 2010)
• 10 days oral amoxicillin/clavulanate 875/125 mg every 12 hours: $31.99
• Single oral dose of azithromycin extended release (2 g): $55.68
– Net benefit = WTP – Costs
– Current treatment decision is azithromycin (Net Benefit= (8.12 - 7.55)*$73.2 ($55.68-$31.99) = $18.2).
•
•
Estimate IVOI by bootstrap sampling out of distribution of average
benefits and costs and averaging the net benefit of over
amoxicillin/clavulanate over azithromycin/clavulanate whenever
amoxicillin had a positive net benefit.
Incidence, Time Horizon and Discounting
– 10 million cases/year.
– 10-year horizon based on patent expiration for azithromycin
– Discount rate of 3%.
VOI Results
•
EVPI Benefits alone: $40 million (range $13 million - $109 million over WTP
$25 to $200/day of avoided sinus congestion).
– Chance information change current decision of using azithromycin based
on effectiveness results only about 4%.
EVPI with costs: $250 million at baseline threshold of $73/day
Max VOI : $400 million at threshold value of $50/day avoided symptoms
where the probability of current decision azithromycin cost-effective 50%.
•
•
.8
1
Figure 2. Acceptability curve and population expected value of perfect information (EVPI) curve for comparing azithromycin extended
release versus amoxicillin/clavulanate in the treatment of acute sinusitis
.2
.4
EVPI, in Billion $
.6
.4
0
0
.2
Pr(Cost-Effective)
.6
.8
EVPI(Cost & effectiveness)
Partial EVPI(Effectiveness)
Partial EVPI(Costs)
.02
(a)
.04
.06
.08
.1
.12
.14
Threshold, 1000 $/Day
.16
.18
.2
.02
(b)
.04
.06
.08
.1
.12
.14
Threshold, 1000 $/Day
.16
.18
.2
Conceptual Value of Information
• VOI requires modeling population value of information
VOI 
where

 t  D (t )  I (t )  N t  IV O I
t
 t is time preference discount factor
D ( t ) is depeciation of knowledge over time
I (t ) is extent of implementation
N t is number of eligible individuals in each cohort
IVOI is individual VOI
• I VOI
–
–
–
–
p(change decision) * Expected value of change given change desirable
IVOI low if either of these gets small enough unless other is very large
In principle, all items above
Exploring use to prioritize topics for systematic reviews for AHRQ EPCs
Conclusions
• Value of information (VOI) analysis can be used to develop
prospective estimates of the value of research
• VOI can be used to inform priorities for clinical research or choose
among study designs
• Standardization of methods across studies critical
• Decision models or minimal modeling approaches
• Additional minimal model applications in progress
• Erlotinib (Tarceva) (+gemcitabine) in advanced pancreatic CA
• Azithromycin vs. Augmentin in acute sinusitis
• Incorporating VOI into research prioritization is a work in progress
• Use by funding agencies?
• Use by investigators?
• Developing algorithms
Acknowledgments
• Collaborators: Anirban Basu Ph.D., Jeanette Chung
Ph.D., Ties Hoomans Ph.D., Herbert Meltzer M.D.
• Funding: AHRQ BCBS Evidence-Based Practice
Center, AHRQ Hospital Medicine and Economics
Center for Education and Research in Therapeutics,
Best Practice Inc, National Institute of Aging,
National Institute of Mental Health
Cost-Effectiveness Acceptability Curve
Value of Research by Value of Health
Value of Research by Time Horizon
Cost-Effectiveness Acceptability Curve
Research as Value of Information:
Analogy to Diagnostic Testing
U(T|S)
S
pU(T|S)+(1-p)U(N|H)
Test
U(N|H)
H
S
Don’t Test
H
Max{pU(T|S)+(1-p)U(T|H),
pU(N|S)+(1-p)U(N|H)}