Expected Value of Perfect Information:
Active Learning Through User-Friendly
Computations, Displays, and
Application Discussions
R Brett McQueen, PhD1
D. Eldon Spackman, PhD2
Jonathan D. Campbell, PhD1
1. University of Colorado, Aurora, CO
2. University of York, Heslington, York, UK
Workshop Objectives
• Review theory and concepts of value of
information
• Expected Value of Perfect Information (EVPI)
active learning
– Gain understanding through a workshop exercise
– Display EVPI case
• Discuss EVPI applications
Welfare Economics &
Bayesian Decision Theory
Decision Analytic Model
Uncertainty in Net
Benefit
Mean Net benefit
Expected Value of
Perfect Information
Q1: Adopt
Intervention with
Present Information?
Claxton K. The irrelevance of inference: a decision-making approach to the stochastic
evaluation of health care technologies. Journal of Health Economics 18 (1999): 341-364.
Q2: Generate Future
Evidence?
Decision Analytic Model
(e.g., Population, intervention, comparator(s), outcome(s),
time that best represent disease state)
Maximize Social Welfare Function
(normative weights included)
Mean Net benefit
Mean Net Benefit = QALYs*Willingness to pay - Costs
Positive analysis
Q1: Adopt
Intervention with
Present Information?
Answer = maximum mean net benefit
Welfare Economics &
Bayesian Decision Theory
Decision Analytic Model
Uncertainty in Net
Benefit
Mean Net benefit
Expected Value of
Perfect Information
Q1: Adopt
Intervention with
Present Information?
Q2: Generate Future
Evidence?
Decision Analytic Model
Uncertainty analysis (i.e., probabilistic sensitivity analysis)
Uncertainty in Net
Benefit
Decision Uncertainty: probability of error
Perfect information (infinite sample) – Present information =
E(max net benefits for each iteration) – max of E(net benefits
of all iterations)
Expected Value of
Perfect Information
Upper bound on value of generating future evidence
Q2: Generate Future
Evidence?
EVPI Active Learning
• Definition:
E(max net benefits for each iteration) – max of E(net
benefits of all iterations)
• Inputs to calculate EVPI:
– Willingness to pay (how much society is willing to
pay for an extra unit of health i.e., QALY)
– QALYs for each intervention and Monte Carlo
iteration
– Costs for each intervention and Monte Carlo
iteration
– Net benefit = (QALYs)*willingness to pay - Costs
Solution
Net Benefit: A
$
25,000.00
$
75,000.00
$
(50,000.00)
$
50,000.00
$
25,000.00
Expected Net Benefit: A
$
25,000.00
EVPI
$
10,000.00
Net Benefit: B
$
$
50,000.00
$
25,000.00
$
50,000.00
$
125,000.00
Expected Net Benefit: B
$
50,000.00
Choose
Max Net Benefit
A or B?
$
25,000.00
A
$
75,000.00
A
$
25,000.00
B
$
50,000.00
B
$
125,000.00
B
Expected Max Net Benefit
$
60,000.00
Display EVPI Case
$40,000
$35,000
$30,000
$25,000
$20,000
$15,000
$10,000
$5,000
$$25,000
$50,000
$75,000
$100,000
$125,000
$150,000
$175,000
$200,000
$225,000
$250,000
$275,000
$300,000
$325,000
$350,000
$375,000
$400,000
$425,000
$450,000
$475,000
$500,000
EVPI
EVPI across varying WTP values
Willingness to pay (US dollars)
Reason for a Data Table
• The EVPI example was based on one
willingness-to-pay value
• What if we wanted to know what EVPI was at
$50,000/QALY or $75,000/QALY?
• To answer this we could manually calculate
EVPI for different values of willingness-to-pay
• Excel can automatically calculate this and
record the values in one table using a data
table
Create the Data Table
• Requirements for Data Table
– input cell (willingness-to-pay) that is used to calculate the
output (net benefit used to calculate EVPI)
– A column of varying input values willingness-to-pay of your
choosing (e.g., $5,000 - $100,000/QALY)
• Highlight cells and click the "Data" tab > "What-If Analysis" >
"Data Table"
• This will prompt a box with row and column input cells.
– If your WTP values are set up as columns (inputs moving
down) enter the in the column input cell
• To update the calculation please press F9 on a PC or CMD += on
a mac
Excel Screen Shots
Adoption and research decisions
A number of conceptually distinct but simultaneous decisions must be
made:
•
Which technology should be adopted into clinical practice given the
existing evidence base and the uncertainty surrounding outcomes
and resource use?
•
Is additional evidence required to support the use of the
technology?
- How uncertain are the expected benefits?
- Does this uncertainty matter (will it change the adoption decision)?
- How much does it matter (consequences of getting it wrong)?
•
What type of evidence would be most valuable?
•
Which research designs would be worthwhile?
•
When to approve the technology?
- Early approval? Can the evidence be provided with approval?
Using Value of Information Analysis to Prioritise Health
Research: Some Lessons from UK Experience
Pilot studies for NICE and NCCHTA (Sculpher and Claxton, 2006)
• Is further research required
–
–
•
What type of research
–
–
•
All subgroups should be included in research
Only worth while for certain groups
Which comparators
–
–
•
•
RCTs of treatment effect
Quality of life and costs
Which subgroups
–
–
•
Research is not needed
Research is a priority
Head to head comparisons are needed
Some comparators could be ruled out
Which endpoints
Length of follow-up
Is further research required?
Physiotherapy for adults
with asthma or COPD did
not appear to be CE and
additional information was
unlikely to change this
assessment.
EVPI is very high and
additional evidence should
be required to support
reimbursement.
Is further research required?
What type of research?
Further evidence about
quality of life with influenza
which is most important
rather than additional
evidence about the effect
on symptoms which would
require an RCT
Is further research required?
What type of research?
Which endpoints and what
length of trial is needed?
If a RCT is undertaken, they
need longer term follow-up
than is currently collected.
Using Value of Information Analysis to Prioritise Health
Research: Some Lessons from UK Experience
Conclusions
• The results were considered by the NCCHTA and NICE
– In general regarded the analysis to be interesting and potentially
useful
• In both cases the formal analysis failed to have a significant
impact on the decisions taken
– The NICE Research and Development committee decided to adopt
a subjective scoring system
• The decisions made by the panels and commissioning
board did not seem to be informed by the results of the
analysis
Formalization of OIR and AWR for NICE
•
Expected cost-effectiveness
•
Irrecoverable costs
– Costs committed by approval that cannot be recovered
– Capital costs of long lived equipment (training and learning)
– Initial losses (negative NB) offset by later gains
– Significance depends on whether initiation of treatment can be delayed
•
Value of additional evidence
•
The need for evidence, type of evidence, design of research
•
Uncertainty that cannot be resolved by research but only over time
•
Are the benefits of early approval greater than the opportunity costs?
Combined assessments will establish the most appropriate policy
option: ‘Approve’, ‘Reject’, ‘Only in Research’, or ‘Approve with
Research’
Current projects with EVPI
• NIHR HTA: Graduated compression stockings
for prevention of deep vein thrombosis in
postoperative surgical patients
• NIHR HTA: PROMIS trial MRI use for the
diagnosis of prostate cancer
• Early phase modelling
Expected Value of Perfect Information:
Active Learning Through User-Friendly
Computations, Displays, and
Application Discussions
R Brett McQueen, PhD1
D. Eldon Spackman, PhD2
Jonathan D. Campbell, PhD1
1. University of Colorado, Aurora, CO
2. University of York, Heslington, York, UK