Finding your way through formal
benefit-risk approaches for decisionmaking in drug regulation
PSI Annual Conference
12th May 2014
London
Professor Deborah Ashby
School of Public Health
Imperial College London
The IMI-PROTECT
• PROTECT1 (Pharmacoepidemiological Research on
Outcomes of Therapeutics by a European ConsorTium)
• “Improving and strengthening the monitoring of the
benefit/risk of medicines marketed in the EU” including
graphical representation of risk-benefit led by EMA with
31 public and private partners, 2009-2014 (www.imiprotect.eu)
1
PROTECT is receiving funding from the European Community’s Seventh Framework
Programme (F7/2007-2013) for the Innovative Medicine Initiative (www.imi.europa.eu)
4
Partners in PROTECT
Public
Private
Regulators:
EMA (Co-ordinator)
DKMA (DK)
AEMPS (ES)
EFPIA companies:
GSK (Deputy Coordinator)
MHRA (UK)
Sanofi
Roche
Academic Institutions:
University of Munich
FICF (Barcelona)
INSERM (Paris)
Mario Negri Institute
(Milan)
Poznan University of
Medical Sciences
University of Groningen
University of Utrecht
Imperial College London
University of Newcastle
Novartis
Pfizer
Amgen
Genzyme
Merck Serono
Others:
WHO UMC
GPRD
SMEs:
Outcome Europe
PGRx (LA-SER)
IAPO
CEIFE
Bayer
Astra Zeneca
Lundbeck
NovoNordisk
Takeda
Eli Lilly
5
IMI- PROTECT Work Package 5
Benefit-risk integration and representation
6
Decision makers – who are they?
Patients
• Make decisions for themselves
Healthcare providers
• Make decisions based on prescribing
lists
NICE
• Makes decisions on cost-effectiveness
EMA/MHRA etc.
• Makes decisions on quality, safety,
efficacy and benefit-risk balance to
individuals and public health
Pharmaceutical companies
• Makes decisions on what to develop
for which licenses to apply
7
Challenges in medical decision-making
• Should we formalise decision-making at all?
• Which quantitative approach(es) to use?
• Whose value preferences take priority – regulators,
pharma, physicians or patients?
• How do we find these preferences – simple elicitation,
decision conferencing, discrete choice experiments….?
• Do we need stakeholders’ preference a priori, or should
we provide tools to allow individual decision-makers to
explore their own preferences and the consequent
decisions?
• How do we communicate benefits and risks?
8
Methodologies available (and tested)
PrOACT-URL
NNT
QALY
INHB
NNH
BRAT
PSM
BRR
Q-TWiST
MCDA
Impact numbers
ITC
DCE
MTC
SMAA
SBRAM
http://www.imi-protect.eu/benefitsRep.shtml
9
Disclaimers
“The processes described and conclusions drawn from
the work presented herein relate solely to the testing
of methodologies and representations for the
evaluation of benefit and risk of medicines.
This report neither replaces nor is intended to replace
or comment on any regulatory decisions made by
national regulatory agencies, nor the European
Medicines Agency.”
10
Recommendation Roadmap
Exploration
Evidence
gathering and
data preparation
Conclusion and
dissemination
Analysis
Planning
11
Stage 1: Planning
• encourages stakeholders to focus on critical issues
related to BR assessment
• encourages sufficient thinking and thorough
discussions between stakeholders to clearly define
the purpose and context of the BR assessment
• ensures clear detailed summary documentation of
discussions and results
• allows future analyses and updates to utilise the
same foundations
12
Planning Toolbox
PrOACT-URL
BRAT
Problem
Define decision
context
Objective
Identify benefit
and risk outcomes
Alternative
Define the decision
context
Consequence
Extract source
data
Customise
framework
Trade-off
Assess outcome
importance
Uncertainty
Display & interpret
key BR metrics
Risk tolerance
Useful methodologies
include:
• non-quantitative /
descriptive frameworks
to organize data
• tree diagrams and
structured tables
providing useful means
of visualisation
Linked decisions
13
An example of (value) tree diagram from
natalizumab case study
14
Recommendation Roadmap
Exploration
Evidence
gathering and
data preparation
Conclusion and
dissemination
Analysis
Planning
15
Stage 2: Evidence gathering and data preparation
• Identifies and extracts evidence relevant to the BR assessment
in relation to the set criteria
• Determines what data to be collected from anticipated type of
BR analysis
• Aggregating multiple sources of evidence, may require the use
of estimation techniques
• Encourages systematic handling of missing data
• Requires engagement of clinical, statistical, epidemiological
and database expertise
16
Evidence Gathering and Data Preparation
Toolbox
• Useful methodologies include:
– Indirect/Mixed Treatment Comparison (ITC/MTC)
– Probabilistic Simulation Method (PSM)
– visualisation techniques such as structured and colourcoded tables, and network graphs to enhance the
communication of data.
17
An example of MTC network in the
natalizumab case study
Natalizumab
Direct
(Polman 2006, EPAR)
Indirect
Indirect
Placebo
Direct
Direct
(Johnson 1998)
(Jacobs 1996)
Glatiramer
acetate
Interferon
beta-1a
Indirect
18
An example of colour-coded tables of data
summary
Outcome
Natalizumab
/ pts
Tysabri
Risk Risk
/ 1000
Benefits
1000 pts
Convenience Benefits
Risk Difference (95% CI)/
1000 pts
-
-
-
(-, -)
Relapse (weight 3.9%)
280
450
-170
(-, -)
Disability Progression (weight 5.6%)
110
140
-30
(-, -)
Reactivation of serious herpes viral infections (weight 6.7%)
PML (weight 55.9%)
80
2
70
0
10
2
(-26, 45)
(-, -)
Liver Toxicity
Transaminases elevation (weight 11.2%)
50
40
10
(-16, 38)
Reproductive Toxicity
Congenital abnormalities (weight 5.6%)
-
-
-
(-, -)
Neurological Disorders
Seizures (weight 5.6%)
0
11
-11
(-23, 0)
236
312
-76
(-, -)
Medical Benefits
Infection
Risks
Convenience (weight 0.6%)
Comparator Risk /
1000 pts
Infusion/Injection reactions (weight 2.8%)
Other
Hypersensitivity reactions (weight 1.1%)
90
40
50
(20, 82)
Flu-like reactions (weight 1.1%)
399
608
-209
(-320, -98)
Higher for Drug A
Higher for Tysabri
Higher for Comparator
19
Recommendation Roadmap
Exploration
Evidence
gathering and
data preparation
Conclusion and
dissemination
Analysis
Planning
20
Stage 3: Analysis
• Evaluates data collected at previous stage in a BR
assessment
• Quantifies the magnitudes of benefits and risks
• Weighs or integrates quantitative measures of the
BR balance depending on the type of analysis
21
Analysis toolbox - methodologies
• Useful methodologies include
– metric indices which provide numerical representations
of benefits and risks e.g. Number Needed to Treat /
Number Needed to Harm (NNT/NNH), Impact numbers
– quantitative frameworks which model benefit-risk
trade-off and balance benefits and risks e.g. MultiCriteria Decision Analysis (MCDA), Stochastic Multicriteria Acceptability Analysis (SMAA)
– utility survey techniques which elicit stakeholders’
preference information e.g. Discrete Choice Experiment
(DCE)
22
Analysis toolbox – example visualisations
Different visuals for different tasks…
…and how to produce them
23
Recommendation Roadmap
Exploration
Evidence
gathering and
data preparation
Conclusion and
dissemination
Analysis
Planning
24
Stage 4: Exploration
• Assesses the robustness and sensitivity of the main
results to various assumptions and sources of
uncertainties
• Assesses further consequences of a decision
• Considers any impact or added value to the RMPs
• Requires both statistical and clinical input
25
Exploration toolbox
• Useful methodologies include:
– ITC/MTC, PSM, SMAA
– Utility survey techniques e.g. DCE, AHP, Swingweighting, MACBETH
• Preferred visualisation techniques include:
– the box, distribution, scatter, and forest/interval plots;
tornado diagram; and techniques that are interactive
with the user.
26
Natalizumab: Probabilistic MCDA
Distribution of benefit-risk score
P(natalizumab
ranked 1st) ≈ 1
27
Recommendation Roadmap
Exploration
Evidence
gathering and
data preparation
Conclusion and
dissemination
Analysis
Planning
28
Stage 5: Conclusion and dissemination
• The point at which a conclusion is reached
• The results and consensus from the BR assessment
are communicated to a wider audience
• Explicitly states findings and conclusions that could
influence future actions
• Emphasises a transparent audit trail of the whole
assessment process i.e. brings everything together
and sets the course of action
• Ensures the "big picture" overview is not lost
29
Summary
• Choice of approach should match the complexity of
the problem.
• In most simple problems, simple descriptive
framework is likely to be sufficient.
• For more complex problems, a framework
supplemented by quantitative models can facilitate
consideration of trade-offs amongst the benefits and
risks, address uncertainty, and potentially lead to a
more comprehensive overall assessment.
• To understand the perspective of a particular
stakeholder, elicitation of preference values for
weighing benefits and risks may be required.
30
Final remarks
• Benefit-risk assessment methodologies
support decision-making and are not
intended to replace medical expertise.
• It is not a linear or sequential but an
iterative process.
• Stakeholders such as patients and public
involvement may add value and would lead
to more clinically relevant decisions.
31
Acknowledgements
• The research leading to these results was conducted
as part of the PROTECT consortium
(Pharmacoepidemiological Research on Outcomes of
Therapeutics by a European ConsorTium, www.imiprotect.eu) which is a public-private partnership
coordinated by the European Medicines Agency.
• The PROTECT project has received support from the
Innovative Medicine Initiative Joint Undertaking
(www.imi.europa.eu) under Grant Agreement n°
115004, resources of which are composed of
financial contribution from the European Union's
Seventh Framework Programme (FP7/2007-2013)
and EFPIA companies’ in kind contribution.
32
Click or
scan me!
http://www.imi-protect.eu/results.shtml#
33
Introducing the IMI “Get Real” project
PSI Conference May 12-14 2014
Chrissie Fletcher, Amgen
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
Disclaimer
• The views expressed herein represent those of the
presenter and do not necessarily represent the views
or practices of Amgen or the views of the general
Pharmaceutical Industry.
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
Two problems needing a joint solution
R&D
decision
• What combination of possible studies will provide the most valuable
information to HTA/payers?
• What is the feasibility of the study options pre-launch and what would be
commitments post launch?
• How do options reconcile with the regulatory process?
HTA
decision
• Would we predict an improvement in patient outcome or care pathway
efficiency over and above current practice in my healthcare?
• Would we accept the uncertainty for a period of time while waiting for
studies to complete?
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
Background and motivation
1. HTA / healthcare decision making require additional information to
Regulatory to address the “efficacy – effectiveness” gap
2. Current initiatives on relative/comparative effectiveness research show how
real world data can be harnessed post launch
3. There is an opportunity to adapt such techniques pre-launch... but there are
practical issues
– HTA <-> regulatory objectives
– Ethical and legal obligations
– Managing cost and operational feasibility
4. Integrating heterogeneous trial and observational data to inform predictive
modelling of effectiveness requires new analytical techniques
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
38
GetReal: Project Vision
For pharmaceutical R&D and healthcare decision makers to better
understand how real-world data and analytical techniques can be
used to improve the relevance of knowledge generated during
development, e.g., through innovation in clinical trial design.
Lasting impact of the project
To provide a methodological and analytical framework that
informs policy and process evolution beyond the project and at
an international level; and to provide tools, techniques and
training that ensure that the potential of real world data can be
exploited in drug development.
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
39
39
Innovative
Medicines
Initiative:
Joining Forces
in the
Healthcare
Sector
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
Nat Med 2014;20:5.
41
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
Nat Med 2014;20:5.
42
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
Public/SME partners
Universities, research organisations, public
bodies, non-profit groups
Universitair Medisch Centrum Utrecht, the Netherlands
Academisch Ziekenhuis Groningen, the Netherlands
College voor Zorgverzekeringen, the Netherlands
European Medicines Agency, UK
European Organisation for Research and Treatment of
Cancer, Belgium
Haute Autorité de Santé, France
London School of Hygiene and Tropical Medicine, UK
National Institute for Health and Care Excellence, UK
Panepistimio Ioanninon, Greece
Universität Bern, Switzerland
University of Leicester, UK
Small and medium-sized enterprises (SMEs)
LA Santé Epidemiologie Evaluation et Recherche,
France
Patients’ organisations
International Alliance of Patients' Organizations, UK
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
EFPIA partners
GlaxoSmithKline Research and Development
Amgen
AstraZeneca
Bayer Pharma
Boehringer Ingelheim
Bristol Myers Squibb
Eli Lilly
F. Hoffmann-La Roche
Janssen Pharmaceutica
Merck KGaA
Merck Sharp & Dohme Corp.
Novartis Pharma AG
Novo Nordisk
Sanofi-Aventis
Takeda Development Centre
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
Structure and topics of
GetReal
WP3
WP1
WP2
WP3
WP2
WP3
Choice of
comparator
Innovative
Development
options / study
designs
WP2
Drivers of
relative
effectiveness
III a options /
study designs
WP4
Operational
cost/ feasibility
/ solutions
Ethics &
Regulations
Full Evidence
Integration
Predictive
Modelling
WP4
Predictive
power / residual
uncertainty
WP1
Frameworks of
relative
effectiveness
assessment
International
Acceptable
Reg & HTA
uncertainty?
policy
implications
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
Reg + HTA
Process
Simulations
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
WP1: The WP1 case study approach
Development strategy redesign workshops
Summaries
Information
Sources
Workshop 1
Outputs
Company Governance
• Publicly available
documents (reg, HTA)
• Stakeholder interviews
• Company
commentaries &
presentations
• Original company
source documents
Simulations
• Discussion summary /
minutes
• Key scientific
questions (sources of
bias and uncertainty
in REff)
• Alternative
development design
options
Workshop 2
Outputs
• Discussion summary
/ minutes
• Stakeholder insight
& reactions to
potential options
• Scenario Summary
• Contribute to
decision framework
• Publications
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
Slide 46
WP 2: better understand the gap between
efficacy & effectiveness
Clinical trials
(efficacy)
Real world
(effectiveness)
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
WP3: overcoming practical barriers in
undertaking pre-launch RE research.
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
WP4: Promote best practice in evidence
synthesis and predictive modelling
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
Developing a predictive
model for relative effectiveness
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
50
Project deliverables and
benefits
• Frameworks developed jointly by Regulatory, HTA and Industry experts for
use in:
• Practical Solutions to enable implementation of studies with greater value of
information for relative effectiveness assessment
• Advances in methodology to reliably predict effectiveness from available
data
• Aligning innovation in evidence generation with evolution of regulatory and
HTA processes
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu
51
For more information:
• Visit the GetReal website:
imi-getreal.eu
The research leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant
agreement no [115303], resources of which are composed of financial contribution from the European Union’s Seventh Framework
Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
www.imi.europa.eu