The uptake of value of
information methods
Solutions found and
challenges to come
Alan Brennan
Director of Operational Research
ScHARR
Aim of this session
•
•
•
•
•
Discuss the roles of VoI analysis
Demonstrate recent technical progress
Growth in interest in and use of VoI methods
Discuss challenges for further uptake
Some recommendations
• Personal Views !
What is Value of Information?
• Given ….
a choice between strategies,
a decision rule for adoption of strategies,
some uncertainty
VoI analysis tells us…
› How valuable more information would be
to reduce uncertainty, to help us choose
› Which uncertain parameters are most crucial
› How valuable a sample of size n=10 would be
compared to n=100, or n=1000 ….
Roles of VoI
Probabilistic Sensitivity
Analysis Without VoI
Cost Effectiveness Acceptability of T1 versus T0
£2,000
100%
90%
£1,500
80%
£1,000
Inc Cost
£0
-1.4
-1.2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
-£500
0.4
0.6
0.8
1
1.2
1.4
1.6
Probability Cost Effective
70%
£500
60%
50%
T1
40%
30%
-£1,000
20%
-£1,500
10%
-£2,000
Inc QALY
0%
£-
£20,000
£40,000
£60,000
£80,000
Threshold (MAICER)
• C-E plane
and
CEAC
• i.e. describes how uncertain we are
£100,000
£120,000
£140,000
Probabilistic Sensitivity
Analysis With VoI adds (1)
EVI for Parameters: research options
EVSI n=50
Trial
Trial + Utility
All (trial, util +
durations)
£1,400
£1,200
£1,000
£800
£600
£400
£200
£-
EVSI n=50
2 level EVPI
• Which parameters or groups of parameters are important
• i.e. the causes - why we are uncertain
Probabilistic Sensitivity
Analysis With VoI adds (2)
EVSI :- 2 Level Algorithm Results
(1,000 outer x 1,000 inner simulations)
£1,400
% response/ utility trial +
duration obsn'l study
£1,200
EVSI (£)
£1,000
% response and utility trial
£800
£600
duration of response
observational study
£400
utility observational study
£200
% response trial
£0
0
50
100
150
200
250
Sample Size (n)
• Value of different samples and research design combinations
• i.e. how best to resolve the uncertainty
Probabilistic Sensitivity
Analysis - VoI difficulty myth
• “It is complex”
• “It takes a long time”
• “Methods are not developed”
• “People can’t understand it”
Probabilistic Sensitivity
Analysis - VoI truth
• If you have done a probabilistic sensitivity
100%
£1,500
90%
£1,000
80%
£0
-1.4
-1.2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
-£500
0.4
0.6
0.8
1
1.2
1.4
1.6
Probability Cost Effective
70%
£500
Inc Cost
analysis i.e.
Cost Effectiveness Acceptability of T1 versus T0
£2,000
60%
50%
T1
40%
30%
-£1,000
20%
-£1,500
10%
-£2,000
Inc QALY
0%
£-
£20,000
£40,000
£60,000
£80,000
£100,000
£120,000
£140,000
Threshold (MAICER)
• Then you are 90% to 95% there
Prioritising and Planning HTA
• Recommendations from systematic review of the
use of modelling in planning and prioritising trials
(HTA 2003; Vol 7: number 23)
• Results of Pilot Study show ..
• Beneficial in refining research design and
quantifying uncertainty
• choice of timing and topic are vital
• who should do the analysis is important
Industry Roles –
Societal Perspective
• Undertaking probabilistic sensitivity analysis and
VoI to be ahead of re-imbursement authorities
• VoI to identify likelihood of cost-effectiveness
and priorities for further data collection
› Very early Discovery phase work
› Phase II results available – now design phase III
› Phase III results available – what else do we need to
make the economic case?
Industry Roles –
Commercial Perspective
• Approaches still developing
(and commercial in confidence)
• Define Net benefit not as λ * QALY – cost
but rather as commercial sales / profit
Or
• model linking sales to likelihood of reimbursement or extent of cost-effectiveness
Growth in Uptake
A personal view
• 1996 – Claxton and Posnett – “what are all
these squiggles, it will never catch on”
• 1999 – doing our review, realising the
conceptual validity and practicability of VoI
• 2000 – some applications but people are unsure
on methods
• 2001 – IHEA York / MDM San Diego
2 or 3 speeches (all UK)
Growth in Uptake
A personal view
• 2002 – CHEBS Focus fortnight with ourselves
Karl Claxton and Tony Ades – methods sorted
• Nice Appraisals beginning to use VoI
• 2002 – MDM Baltimore
-7 people UK and Canada – lots of interest
• 2003 – MDM Chicago
-15 to 20 people (US, Canada, Netherlands,
UK) even more interest
Technical Problems
Recently Solved
Technical Problems
Recently Solved:EVPI
• Correct method for EVPI = 2 level simulation
EVPI =



E i max E i NB(d,  ) |  i  max E NB(d,  )
d
d
• 1 level simulation EVPI works if ….
(a) the net benefit functions are linear functions of
the -i for all of the decisions d and all of the possible
values of the parameter set of interest i, and
(b) if i and -i are independent. “
Technical Problems
Recently Solved:EVSI
• Correct Method for EVSI = 2 level simulation
(Bayesian update given simulated collected data)


EVSI  E Xi max E NBd ,  | X i   max E NB(d, )
d
d
• EVSI easy with conjugate distributions
(i.e. retain same functional form when additional
data is collected and synthesised)
› Normal, Beta, Gamma, Lognormal
• EVSI more complex without conjugacy
Technical Problems
Recently Solved: Shortcuts
• Complex models can be ‘emulated’ e.g. Gaussian
Processes, which also offer quicker EVPI and EVSI for 1
variable calculation functionality
• Laplace approximation can help shortcut to a 1 level
simulation for EVSI calculations (poster)


EVSI  E Xi max E NBd ,  | X i   max E NB(d, )
d
d
• The EVSI curve has a common but not universal shape –
exponential of square root of n
• Valuing additonal data in survival trials (poster)
Technical Problems
Recently Solved: Software
• EXCEL for models
• R / Splus advantages
›
›
›
›
More sampling functionality
More optimisation functionality
Faster running times
Easier code
• WinBUGS for complex posterior distributions –
interlinkage with R and Splus
Key Challenges: Uptake for
Sensitivity Analysis - AIDA
• Awareness  Interest  Desire  Action
• Publications / Conferences / Seminars
• Training
 York / Oxford Advanced Modelling Course
 CHEBS characterising uncertainty and
analysing outputs courses (2004)
 Bristol WinBUGS course (2004)
• Network of experts
• Web resources
Key Challenges:
Uptake in trial design
• Engaging with traditional trial designers
• Clinically significant difference is a proxy for
the decision rule
• Need to work together to compare results
and approaches on pilot trials
• particularly ones with Economic and Quality
of life data to be collected
Key Challenges:
Meeting Criticisms
1. How do you know uncertainty is properly
characterised i.e. you might be uncertain about
the uncertainty
2. You need to multiply by the number of people
affected by the decision - How long does the
technology / decision last – 2 years, 5 years, 10
years? An issue even without doing EVI
3. Information has value beyond the jurisdiction
Key Challenges:
Technical
• Correlation, Correlation, Correlation
• Integrated VoI and evidence synthesis quickly
• Reversible decisions
• Deciding to wait – options pricing
Conclusions
• Significant recent progress, interest and growth
• Key challenges
› Speeding up and teaching standard routines to do
the calculations
› Working through when exactly VoI is likely to be
most valuable
› Engaging sceptics to collaborate
• Now is the time ….
Conclusions