When is there Sufficient Evidence?
Karl Claxton,
Department of Economics and Related Studies and
Centre for Health Economics,
University of York.
Sufficient evidence for decision making
• Informing decisions
• Prioritising research
– Examples from the NHS HTA Pilot
•
•
•
•
•
Designing research
Evaluating regulatory policies
Implementation
UK policy impact
Implications and challenges
The decisions
• Should a technology be adopted given existing information?
– Which clinical strategies are cost-effective?
– For which patient groups?
• Is additional evidence required?
– What type of evidence
– What type of studies
– For which patient groups
– How much evidence
Informing decisions
• Requirements
–
–
–
–
Structuring clinical decision problems
Characterisation of decision uncertainty
Value of additional research
Decisions consistent with objectives and constraints
Structuring clinical decision problems
• Requirements
–
–
–
–
–
Compare all alternative interventions/strategies
Explore the full range of clinical policies
For range of patient groups
Over an appropriate time horizon
Combine evidence from variety sources
Characterising uncertainty
• Evidence for all model parameters
– Systematic searching for all parameters
– Methods of synthesis
• Direct evidence
– Different types of study (potential bias/exchangeability)
• Networks of evidence
– Indirect and mixed comparisons
– Evidence on functions of parameters
• Probabilistic analysis of the decision model
Mixed and indirect comparisons
Alternative interventions
• Compare all
interventions
• No RCT of all
• Pair wise comparisons?
• Use all the information
– Mixed comparisons
– Same assumptions as
random effect meta
analysis
– Estimate posterior LOR
with correlations
A
B
1
x
x
2
x
3
x
RCTs
D
x
x
4
x
5
x
6
C
x
x
x
x
Networks of evidence
A
A+B
A+C
A+B+C
B
C
B+C
Value of additional research
Is further evidence required?
• Expected value of perfect information (EVPI)
– Maximum return to research (decision problem)
– Necessary condition (EVPI>costs)
What type of evidence?
• EVPI for parameters
– Maximum return to different types of research
– Focusing research design
An example
• A pilot study of value of information analysis to
inform the NHS health technology assessment
programme
– Screening for age-related macular degeneration
– Manual chest physiotherapy techniques for asthma and
chronic obstructive pulmonary disease
– *long-term antibiotic treatment for preventing recurrent
urinary tract infections (UTI) in children*
Structuring the decision problem
Frequency of
recurrent UTIs
Number of
pyelonephritic attacks
Progressive
renal scaring
End-stage renal disease
No UTI
1 UTI
Pyelonephritic
attack
2 UTIs
Pyelonephritic
attack
3 UTIs
Pyelonephritic
attack
4 UTIs
Pyelonephritic
attack
Transplant
Number of
attacks
Progressive
renal
scaring
Development
of ESRD
Age at
ESRD onset
Dialysis
The evidence
• Effectiveness
– Existing reviews (variable quality)
– Meta analysis, Multiple parameter synthesis
• Natural history
– Epidemiological studies
– Pooled trial baselines
– Registry studies
• Quality of life
– Published studies
– Survey
• Costs
– Published studies
– Published unit costs and dosage (BNF, PSSRU, CIPFA)
Characterising decision uncertainty
1
Intermittent
0.9
Cotrimoxazole
Nitrofurantoin
0.8
Trimethoprim
Probability cost-effective
0.7
Frontier
0.6
0.5
0.4
0.3
0.2
0.1
0
£0
£10,000
£20,000
£30,000
£40,000
Threshold for cost-effectiveness
£50,000
£60,000
The irrelevance of inference?
• The choice between alternative technologies should
be based on expectation. Inference is irrelevant to
treatment choice
• The only valid reason to characterise the uncertainty
surrounding outcomes of interest is to establish the
value of additional information
• Distinguish the separate steps of deciding which
technology should be chosen, given existing
information, from the question of whether more
evidence is required to support this decision
Is further evidence required?
• Expected cost of uncertainty
– Adoption based on existing information is uncertain
– There is a probability of making the wrong decision
– There are costs of making the wrong decision
• Expected value of perfect information (EVPI)
– Maximum return to research (decision problem)
– Necessary condition (EVPI>costs)
EVPI for the decision problem
Treatment A
Treatment B
Treatment C
Optimal treatment
Iteration 1
11
12
13
C
13
Iteration 2
12
10
9
A
12
Iteration 3
13
18
15
B
18
Iteration 4
14
16
17
C
Iteration 5
15
14
11
A
Expected NB
13
14
13
Current information = 14
Perfect information = 15
EVPI
= 15- 14 = 1
EVPI = Eθ maxj NB(j, θ) - maxj Eθ NB(j, θ)
Max NB
17
15
15
Is further
evidence
required?
Population
EVPI
£4,000,000
£3,500,000
Populaion EVPI
£3,000,000
£2,500,000
£2,000,000
£1,500,000
£1,000,000
£500,000
£0
£0
£10,000
£20,000
£30,000
£40,000
Cost-effectiveness threshold
£50,000
£60,000
EVPI for parameters
EVPPIφ = Eφ maxj Eψ |φ NB(j, φ,ψ) – maxj Eθ NB(j,θ)
Where: φ = parameter of interest
ψ = other uncertainties
Some implications:
• Information about an input is only valuable if it
changes our decision
• Information is only valuable if the parameter does
not resolve at its expected value
General solution (linear and non linear models)
m
od
el
D
ev
ES
el
op
R
D
m
e
nt
C
on
of
se
ES
q
Ef
ue
R
fe
D
nc
ct
e
s
of
of
Lo
ES
ng
R
-te
D
rm
Ef
an
fe
ti b
ct
io
of
ti c
s
co
tri
Ef
m
fe
ox
ct
az
of
ol
ni
e
tro
fu
Ef
ra
fe
na
ct
to
of
in
tri
m
et
ho
Ef
pr
fe
im
ct
<
6
m
on
Ef
th
fe
s
ct
>
6
m
on
th
s
of
U
of
TI
py
el
on
ep
hr
i ti
s
U
ti l
R
i ty
is
k
of
of
U
PR
TI
S
/p
ye
lo
ne
ph
ri t
is
eq
ue
nc
y
Fr
eq
ue
nc
y
Fr
Fu
ll
Expected Value of Perfect Information
What type of evidence?
£2,500,000
£2,000,000
£1,500,000
£1,000,000
£500,000
£0
Prioritising research
• Other methods
– Unrelated to the returns from research
– Research as a means changing clinical practice
• Statistical decision theory
– Reduction in the costs of decision uncertainty
– Value consistent with objective and constraints of
service provision
– Allocation within and between clinical areas
– Allocation between research and service provision
Design of future research
• EVPI exceeds cost of research?
– Necessary but not sufficient
• Benefit of sample information
– Predict possible future samples and posteriors
– Expected net benefits with and with out sample information
– Expected value of sample information (EVSI)
• Societal payoff from proposed research
– EVSI - cost of sampling
– Expected net benefit of sampling (ENBS)
EVSI: predicting the results of proposed research
N=1
NBj
NBj
NBk
N=20
NBj
pdf
pdf
Prior Distribution
Predicted Posterior Distributions
Prior Distribution
Predicted Posterior Distr ibutions
0.0
0.2
0.4
NBj
Probability
0.6
0.8
1.0
0.0
0.2
0.4
NBj
Probability
0.6
0.8
1.0
Expected net benefit of sample information (all parameters)
£3,500,000
EVSI
Cost(n)
ENBS
£3,000,000
EVSI, cost(n), ENBS
£2,500,000
£2,000,000
£1,500,000
£1,000,000
£500,000
£0
0
100
200
300
400
500
600
700
800
Sample size (n)
900
1000
1100
1200
1300
1400
1500
ENBS and n* for a portfolio of research designs
£1,800,000
PIP
RSD,PHZ
UPA
Expected net benefit of sample information
£1,600,000
£1,400,000
£1,200,000
£1,000,000
£800,000
£600,000
£400,000
£200,000
£0
0
100
200
300
400
500
600
700
800
Sample size (n)
900
1000
1100
1200
1300
1400
1500
Efficient research designs
•
•
•
•
•
•
•
•
•
Sufficient condition (ENBS>0)
Optimal sample size
Allocation of patients to the arms of a trial
Optimal stopping of sequential trial
Which endpoints should be included?
Optimal follow-up
What are relevant alternatives
Optimal portfolios of research
Optimal development programmes
Regulation of health technologies
“...A reasonable basis for a claim [of cost-effectiveness]
depends on a number of factors relevant to the benefits and
costs of substantiating a particular claim. These factors
include: the type of product, the consequences of a false
claim, the benefits of a truthful claim, the costs of
developing substantiation for the claim ...”
(FDA, section 114, modernization act of 1997)
Regulation of health technologies
• Is it efficient to conduct further research?
– Define a claim/submission as “substantiated” and evidence
as “competent and reliable” when it is not efficient to
conduct further research
• Should demand:
– More evidence for some technologies as compared to others
– Different types of evidence for different technologies
– Different amounts of evidence for the same technology in
different circumstances
• Role of the regulator?
– Policing the priors
Evaluating regulatory policies
• Societal value of information
– Socially optimal development decisions
– Socially optimal amount/type of evidence
• Commercial value of information
–
–
–
–
Commercial payoff function
Payoff conditional on licensing and reimbursement
Commercially optimal development decisions
Commercially optimal amount/type of evidence
• Optimal policies
– Were commercial development and information decisions
match societal needs
Implementation
• Value of information and the value of implementation
Information
Implementation
Current
“Perfect”
Current
A
B
“Perfect”
C
D
• Realising the value of information
- EVPI = D-C
- “Realisable” EVPI = B-A
• Value of changing clinical practice
- Current information = C-A
- With additional information = D-B
• Evaluate policies to change clinical practice
- Can invest in information, implementation or both = D-A
UK Policy impact
• National Institute for Clinical Excellence
– Pilot study to inform research recommendations
– Guidance on methods of technology appraisal
– Decision analytic modelling required
– Evidence synthesis and probabilistic analysis required
– Value of information analysis recommended
• NHS Health Technology Assessment Programme
– Pilot study of setting research priorities
• Medical Research Council
– Programme on value of information
– Programme on evidence synthesis for decision making
– Cancer Trials Unit Programme
Implications and challenges
• Rational decision making
- Adoption decisions
- Research decisions
- Additional objectives, constraints and irreversibility
• Characterising uncertainty
- Indirect comparisons and networks of evidence
- Bias and exchangeability
- Model structure
• Computation
- Correlation and complex data structures
- Non multi-linear and complex models
- Dimensions of design space