Health Economics Tools for Research
Prioritization : Value of Information (VOI)
Methods and Applications for
Comparative Effectiveness Research
ITHS Institute
Sept. 28, 2010
David Veenstra, PharmD, PhD
Josh Carlson, MPH, PhD
Pharmaceutical Outcomes Research and Policy Program, University of Washington
1
Agenda
• Value of information theory
• VOI Examples:
– EGFR testing in advanced refractory NSCLC
– Lung volume reduction surgery (NETT trial)
• Incorporating VOI into research prioritization
2
Scenario
• You have been appointed director of the National
Heart Lung and Blood Institute. Your budget is $2.9
billion, most of which is devoted to investigatorinitiated projects (R01s). Still there is $14 million in
“roadmap funds”, and many researchers have
approached you with ideas for large multicenter
collaborative projects to address “critical” questions.
• Question: How will you allocate this money?
A.
B.
C.
D.
Your own personal preferences
Expert opinion
What the political winds dictate is necessary
Some other way
Expected Value of Information (VOI)
• Is there a way to make better decisions about funding future
research so that uncertainty is reduced and the greatest
number of people benefit from the knowledge that is gained?
• VOI is proposed as a structured set of methods to prioritize
areas where further research is needed to reduce uncertainty
in decision making.
Rationale: Expected value of information
• Premise: Objective of the health care system is to maximize
gains in health subject to a budget constraint
• Current decisions are made under uncertainty  possible
that adopted medical interventions do not maximize net
benefits
• i.e. We could be making ‘wrong’ decisions about which
treatments to use.
• Health care system should be willing to pay for additional
information if the value of additional information is greater
than the cost to generate additional information
How do we assess the value of additional
information?
• Value of additional evidence is evaluated by estimating the
expected cost of uncertainty.
• Probability decision based on existing information will be
wrong (probability of error)
• Consequences of a wrong decision (opportunity costs of
error).
• Applied over the impacted population
• Applied over the time horizon for the technology
Methods
• Utilizes decision analysis
– Compares a treatment and an alternative
• Calculate the incremental net benefit of the treatment vs.
alternative
– INB = λ(B1-B0) – (C1-C0)
– λ = societal willingness to pay for a unit of health benefit
(e.g., a QALY)
– OR,
– INB = Willingness to pay X Incremental health gain –
Incremental Costs
Methods
• Issue:
– The parameters used to inform the model are uncertain
(e.g., benefit of screening)
– If the wrong decision is made, INB is negative
Purpose
• Estimate the impact of collecting additional primary data (e.g.,
clinical trial) on the decision
– The additional information reduces the uncertainty
regarding the decision
– What is the value of this information relative to it’s cost?
Expected Value of Perfect Information
(EVPI)
• If we had perfect information, we would not make a “wrong”
decision
• EVPI can be thought of as the value of new information
obtained from a trial in which we have an infinite number of
subjects
• An upper bound on the returns from further research
Expected Value of Perfect Parameter
Information (EVPPI)
• If we had perfect information about an individual parameter
(e.g. mean survival for intervention) or set of parameters
(mean survival for multiple interventions), we would not
make a “wrong” decision due to these specific parameter(s).
• EVPPI can be used to estimate the value of information about
a specific set of parameters that might be collected with a
particular type of study.
– Clinical parameters (clinical trial)
– Cost parameters (costing study)
• An upper bound on the returns from further research for the
parameter set
11
Expected Value of Sample Information
(EVSI)
• The value of new research before conducting a trial of a given
sample size
• Note that EVSI < EVPI
• If the cost of research is less than the EVSI, the net benefit
from the information is positive
• Expected net benefit of sampling (ENBS)
– The EVSI less the cost of sampling (i.e., a trial)
– ENBS = EVSI – cost of sampling
Past EVI studies:
• Alzheimer’s disease drugs
• Pentoxifylline to treat chronic venous leg ulcers
• Preoperative patient management before major
elective surgery
EXAMPLES
14
Example: Cost-utility of EGFR-guided
treatment of refractory NSCLC
• Developed a decision analytic model to evaluate the cost-utility of EGFR
protein expression or gene copy number testing compared to standard
care with erlotinib in refractory advanced NSCLC patients.
• Motivation for EVPI:
• No clear ‘best’ treatment strategy
• High uncertainty in the parameter estimates (especially survival
estimates)
Carlson JJ, Garrison LP, Ramsey SD, Veenstra DL.Value Health. 2009
Jan;12(1):20-7.
Example: Cost-utility of EGFR-guided treatment of refractory NSCLC
EVPI steps
1. Assign distributions to parameters (95% CI)
2. Run simulations to generate a series of iterations of model
outputs (e.g. costs and QALYs)
–
Possible realization of the true outcomes given uncertainty
3. Calculate the net benefit
4. Generate per person EVPI
5. Calculate effective population
• # of patients impacted over lifetime of technology
6. Generate population EVPI
• Per person EVPI times effective population
Population EVPI
EVPI @ $100,000/QALY for US pop.
%
Incidence of lung cancer (US 2007)
Count
Source
213,380
SEER (2007)
Advanced/Distant stage
41%
87,486
SEER website (1988-2003)
NSCLC
80%
69,989
Ramsey et al. (2006)
Likely to be treated with chemotherapy
79%
55,151
Ramsey et al. (2006)
Likely to receive 2nd line treatment
32%
17,496
Kutakova et al. (2005)
EVPI per person @ $100,000/QALY
$381
Discounted EVPI in US over 5 years
1
$6,663,196
2
$6,469,122
3
$6,280,701
4
$6,097,768
5
$5,920,163
Total
Abbreviations: EVPI=expected value of perfect information, NSCLC=non-small cell lung cancer
$31,430,951
Cost-effectiveness frontier
CE frontier indicates the probability that the alternative with the
highest net-benefit will be cost-effective
Next Steps
• Expected value of perfect parameter information (EVPPI)
• Difference between expected net benefit (ENB) of perfect information
about a set of related parameters (e.g. survival estimates) and ENB
with current information
• Better represents how future research would be conducted
• Expected value of sample information (EVSI)
• Difference between ENB with increased sample size and ENB with
current information
• Future research will not provide perfect information, but rather better
estimates (smaller confidence intervals)
Expected value of perfect parameter
information (EVPPI)
Effective population*
17,496
Ceiling ratio
Cost
Safety and survival
Utilities
$50,000
$0
$76,245,000
$0
$100,000
$0
$9,838,000
$13,193,000
$150,000
$10,894,000
$119,839,000
$101,905,000
21
Assessing the value of a study - EVSI
Example:
A Multicenter Trial of Lung Volume
Reduction Surgery For Patients with
Severe Emphysema
Lung Volume Reduction Surgery
(LVRS)
• Palliative surgical treatment for severe emphysema
• Multiple wedges resected from damaged lung
• Preliminary data show promising results:
– Pulmonary function ~ 30% improvement
– Reduced need for supplemental oxygen
– Exercise capacity (6-minute walk) ~ 25% increase
– Improved quality of life (SF-36, lung disease-specific QOL
measures)
LVRS: Concerns
• Morbidity:
– 30-50% post-surgery air leaks
• Mortality (30 day): 5-10%
• Limitations of published clinical trial data:
– Unclear & nonstandardized patient selection criteria
– Lack of control populations
– Small sample size (<50 patients/study)
– Short follow-up periods
– Limited information on quality of life
Economics of LVRS
• Mean reimbursement (HCFA):
– $25,268 (max $251,222)
• Estimated U.S. expenditure, 7/94-12/95
– $100,000,000 ($30m HCFA)
• Potential expenditure?
– 2 million individuals with COPD
– ~150,000 new cases annually
National Emphysema Treatment Trial (NETT)
• 18 centers throughout U.S.
• 2,600 patients to be randomized
• Three study arms
– Usual care + intensive pulmonary rehab
– Pulmonary rehab + LVRS by median sternotomy
– Pulmonary rehab + LVRS by thoracoscopy
• Follow-up 3-5 years
Question
• What is the value of an investment in a trial to learn
more about the effectiveness and cost-effectiveness
of LVRS?
Value of Information
National Emphysema Treatment Trial
• Consider
– Impact of LVRS and MT on quality of life, survival
– Cost of LVRS and MT
– Expected life of the technology
– Eligible patients over life of technology
– Value of a Quality Adjusted Life Year
• $50,000/QALY, $100,000/QALY
Inputs for the Expected Value of Information
Calculations (1996 dollars)
Parameter†
Cost
Distribution
Mean, Lung Volume
Reduction Group
Normal
$53498± $3527
n = 23
$26032 ± $3219
n = 228
$8935±$1105
n = 228
$8935±$1105
n = 228
0.34
0.34
3
3
1.78 ± 0.93
n = 22
1.69 ± 0.82
n = 22
Cost of medical therapy, per year*
2-year perioperative mortality
rate**
Long term survival (years)
Quality of life (utilities)
Number of persons in the United
States expected to utilize
LVRS if found effective, per
year
Duration of use of LVRS in
clinical practice, years
Normal
20,000
10
Mean, Medical
Therapy Group
Costs Associated with Conducting the
National Emphysema Treatment Trial
Expense
Expenditure
(2003 dollars)
Study costs (investigators, support staff,
clinical study coordinators, etc)
$37,225,000
Participant travel costs (paid from study funds) $91,000
Patient evaluation and preparation costs,
including the pre-trial rehabilitation program*
$2,683,000
Post-therapy rehabilitation
$1,052,000
Trial-related treatment costs*†
$18,112,000
Total Estimated Expenditure
$59,163,000
*Based on CMS claims for NETT participants.
VOI for LVRS at WTP Thresholds of $50,000 and
$100,000 per QALY assuming 1250 subjects per
arm
Threshold
WTP/QALY
Incremental net
benefit
Expected net benefit
of sampling
Probability of
change in
decision
WTP =
$50,000/QALY
WTP =
$100,000/QALY
$305,000/QALY
$305,000/QALY
$3.41 billion
$7.23 billion
0.04
0.24
Ramsey SD, Blough DK, Sullivan SD.Med Care. 2008 May;46(5):542-8.
NETT Outcomes: Mortality and Exercise Capacity†
All Patients
Mortality RR = 1.01
Exercise OR = 6.27
Non High Risk Patients
Mortality RR = 0.89
Exercise OR = 6.78*
Upper Lobe
Low Exercise
Death RR = 0.47*
Exercise OR = ∞*
Upper Lobe
High Exercise
Death RR = 0.98
Exercise OR = 5.81*
High Risk Patients
Mortality RR = 1.82*
Exercise OR = 3.48
Non Upper Lobe
Low Exercise
Death RR = 0.81
Exercise OR = 1.77
Non Upper Lobe
High Exercise
Death RR = 2.06*
Exercise OR = 0.90
†Δ ≥ 10 watts * = p≤0.001
Cost-Effectiveness for Subgroups
at 3 Years
Cost per QALY Gained
at 3 years
Subgroup
High Risk (n=138)
Dominated*
Upper Lobe + Low Exercise Capacity
(n=137)
Upper Lobe + High Exercise Capacity
(n=204)
Non Upper Lobe and Low Exercise
Capacity (n=82)
Non Upper Lobe and High Exercise
Capacity (n=108)
$98,000
$240,000
$330,000
Dominated*
*Dominated: higher costs and less favorable outcomes for the LVRS group
Ramsey SD, Berry K, Etzioni R, Kaplan RM, Sullivan SD, Wood DE; National Emphysema
Treatment Trial Research Group.N Engl J Med. 2003 May 22;348(21):2092-102.
VOI and Research Prioritization:
CANCERGEN
• CER project evaluating genomic tests in cancer
• 6 top candidates for RCT selected using landscape
analysis
• 3 top candidates selected by stakeholder group
• Using VOI approaches to help inform final selection
for development of CER RCT
Candidates
• Tumor markers for detection of BrCA recurrence
– validity not established
– commonly used
• EGFR mutation testing for tyrosine kinase inhibitor
therapy in lung cancer
– mutations relatively rare
– large effect on treatment outcomes
• ERCC1 testing in early-stage lung cancer
– significant side effects of chemo
– recurrence clinically impactful
Process
• Develop disease models
– assumptions with expert input
• Calculate EVPI and EVPPI
– rank tests
– identify study types
• Calculate EVSI
– provide support for study funding
– refine study design
Conclusions
Expected Value of Information
• A promising method for allocating scarce research
funds to maximize the benefit of each funded study
• But
– Conceptually difficult
– Computationally complex
– Requires time consuming simulation modeling
• Ongoing research may yield a tractable approach
Extra Slides
Population EVPI over range of willingness to pay thresholds
Population EVPI (Millions)
$300
$250
$200
$150
$100
$50
$0
$0
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
Societal Willingness to Pay (per QALY)
• Increasing willingness to pay  opportunity cost of wrong decisions is higher  higher EVPI
• But will also increase the expected net benefit (QALY x WTP) and probability of error will fall
which may decrease the EVPI.
• EVPI will ultimately increase  decision uncertainty falls at a declining rate
while the value of opportunity losses increases at a constant rate
EVPI per person
Value of information
• Expected value of perfect information (EVPI)
• Difference between the expected net benefit (ENB) of perfect
information (no wrong decisions) and the expected net benefit with
current information
• Provides the upper bound value of additional information
• Expected Net Benefits = QALYs x willingness to pay for health
gains – costs
• Current decision = treatment with highest expected net
benefits (i.e. maximum of the average net benefit over all
iterations)
• Decision with perfect information = average of the maximum
net benefits for each iteration
Calculating EVPI
Treatment
Optimal
choice
Maximum
net benefit
Opportunity
loss
A
B
Iteration 1
9
12
B
12
0
Iteration 2
12
10
A
12
2
Iteration 3
14
20
B
20
0
Iteration 4
11
10
A
11
1
Iteration 5
14
13
A
14
1
Expectation
12
13
13.8
0.8
• Each iteration represents a possible future realization of the model
under existing uncertainty