Jasvinder Singh, MBBS, MPH
Associate Professor of Medicine, UAB School of Medicine & VA
Medical Center, Birmingham, Alabama
Mayo Clinic School of Medicine, Rochester, MN
IMMPACT-XIV, Arlington, VA June 16-17, 2011

Entire OMERACT Executive: Developed
OMERACT 3 X 3

Slides: Dr. Maarten Boers

Comments: Drs. Maarten Boers & Lee Simon




Definitions
Summary and Discussions at 2008 OMERACT
Drug Safety Summit
OMERACT approach
Benefit Risk Action Team (BRAT) approach
◦ Pharmaceutical Research and Manufacturers of
America


Questions and potential solutions
The Future

Risk:
◦ Oxford: “a situation involving exposure to danger”
◦ Merriam-Webster: “possibility of loss or injury”
◦ Medical context: an effect that is harmful to the
patient’s or public’s health and which can relate to
safety, efficacy or quality of a product.

Benefit:
◦ Oxford: “an advantage or profit gained from
something”
◦ Merriam-Webster: “something that promotes wellbeing “




Definitions
Summary and Discussions at 2008 OMERACT
Drug Safety Summit
OMERACT approach
Benefit Risk Action Team (BRAT) approach
◦ Pharmaceutical Research and Manufacturers of
America


Questions and potential solutions
The Future

Assigning causality:
◦ Bradford-Hill criteria + biological plausibility +
Bayesian methods
◦ No agreement on magnitude of frequency, i.e.,
when does something become a “signal”

Utility of large trials to define risk
◦ Large, simple trials such as prospective randomized
open blinded endpoint (PROBE)
◦ Randomization to two effective treatments
◦ Alternative: observational studies with confounding
by indication and channeling bias

Utility of metanalyses of RCTs
◦ Multiple small RCTs ?statistically equivalent
to a large single trials with same
denominator
◦ 200-300 events needed for credible
estimates
◦ Role for indirect comparisons- unclear
◦ Observational studies: may be included;
 Selection and confounding bias issues
 Outcome definition and measurement and
follow-up

Postmarketing Surveillance
◦ Essential for drug safety evaluation
◦ All drugs in a class should be analyzed identically and
concurrently
◦ Desired components:
More than one defined population
Full protocol with outcomes assessed at regular intervals
Concurrent control subjects
Outcomes- patient-recoded + documented in electronic
records
 Pre-defined hypothesis
 Oversight by FDA and data management council





Utilities of Drug registries
◦ Advantages: real world experience, long-term FU
◦ Disadvantages: lack of comparator group, data
quality, patients not obligated to follow protocol,
loss to follow-up

Pharmacoepidemiologic studies
◦ Cohort studies generally better than case-control
studies in providing risk estimates
◦ Issues need to addressed
 Misclassification bias
 Validation of outcomes
 Guidelines for confounding bias and methods for
adjustment
Simple versus complex metric
◦ Simple: OMERACT 3 x 3
 Benefit and risk categorized into 3 levels
each- none, substantial, (near) remission or
death
◦ Complex: multicriteria decision analysis
• Complex Frameworks: Quantitative methods
◦ Decision analysis method
◦ Conjoint analysis
◦ Incremental net-benefit
◦ BRAT approach



GRADE approach
◦ Classifies evidence into 4 levels: high, moderate, low
and very low based on
 Study design, weaknesses, special strengths (large
effect, dose response)
◦ 2 recommendations: strong and weak
◦ RCTs always starting at high and non-RCTs starting at
low?
Other models of Risk: Nontreatment
◦ Risk and value of available treatment versus
nontreatment options
◦ Type, intensity and severity of adverse event
◦ Acute, subacute or chronic; manageable, treatable or
not




Definitions
Summary and Discussions at 2008 OMERACT
Drug Safety Summit
OMERACT approach
Benefit Risk Action Team (BRAT) approach
◦ Pharmaceutical Research and Manufacturers of
America


Questions and potential solutions
The Future
Create three categories of harm and
benefit
 Benefit: none/minimal, major, (near)
remission
 Harm: none/minimal, major, (near)
death
 Key components of this approach:
◦ Harms versus benefits

 measurement
of benefit is specific
and evolved, but measurement of
risk (or harm) is generic and more
primitive
 Benefit and harm not placed on a
single scale
single scale benefit-harm
14

placing value judgments on scientific facts
◦ values will vary depending on
the perspective of the assessor

comparing benefit and risk involves:
◦ comparing apples and pears
◦ trading off (and discounting)
long-term against short-term effects
◦ assessing multiple benefits and risks

assessment mostly driven
by the need to make decisions,
whereas most research is ‘truth-driven’
single scale benefit-harm
15
...placing value judgments on scientific facts
Requires qualitative research
...multiple comparisons and trade-offs
Difficult science
Requires decision analysis
...mostly driven by decision-makers
Low speed of developments
Literature difficult to access
single scale benefit-harm
16
Benefit
none/
minimal
major
(near)
remission
none/
minimal
H a r m major
(near)
death
single scale benefit-harm
17


COBRA trial
VIGOR trial
single scale benefit-harm
18


RCT comparing combination therapy (COBRA)
including step-down high-dose prednisolone
with single drug therapy (sulfasalazine, SSZ)
in early rheumatoid arthritis
Main result showed COBRA to be more
effective and result in less side effects than
SSZ
single scale benefit-harm
19
SSZ (n=79)
Harm
(%)
none/minimal
major
Benefit (%)
none/
major
(near)
minimal
remission
23
29
25
14
5
4
total
77
23
(near) death
0
0
0
0
total
37
34
29
100
none/minimal
major
(near) death
12
5
0
38
3
0
42
0
0
92
8
0
total
17
41
42
100
COBRA (n=76)
Harm
(%)
Major benefit = Disease activity score <=3.7
(near) Remission = Disease activity score <=2.4
Severe Harm = treatment discontinuation due to serious harm, loss of efficacy or
both
(near) death = death or severe inacapacitation
20
Major Benefit (%)
SSZ (n=79)
Major
Harm (%)
no
no
23
yes
54
total
77
yes
14
9
23
total
37
63
100
no
12
80
92
yes
5
3
8
total
17
83
100
COBRA (n=76)
Major
Harm (%)
ARR: Unqualified success: 80-54% = 26%; NNT = 4
ARR: Unmitigated failure: 5-14% = –9%; NNH = ?
single scale benefit-harm
21
80
60
Unqualified success
40
20
0
-20
Unmitigated failure
SSZ
COBRA
single scale benefit-harm
22
large industry-sponsored trial to compare
high-dose rofecoxib with naproxen in RA
object was GI safety, not efficacy
rofecoxib proved safer for GI,
but less safe for cardiovascular events
Merck withdrew the drug voluntarily amidst
much controversy
no access to individual patient data
safety and efficacy assumed independent
single scale benefit-harm
23
rofecoxib (3900 py)
Major Harm
(per 100 pt-yrs)
Major Benefit (per 100 pt-yrs)
no
no
62.9
yes
32.4
total
95.3
yes
3.1
1.6
4.7
total
66.0
34.0
100
no
yes
62.1
3.9
32.0
1.9
94.1
5.8
total
66.0
33.9
100
naproxen (3100 py)
Major Harm
Data from original report and from Strand V. Lancet 2007;370:2138-51.
Major harm is defined as cardiovascular event; gastrointestinal event; death.
single scale benefit-harm
24
40
Unqualified success
20
0
Unmitigated failure
-20
rofecoxib
naproxen
single scale benefit-harm
25




Definitions
Summary and Discussions at 2008 OMERACT
Drug Safety Summit
OMERACT approach
Benefit Risk Action Team (BRAT) approach
◦ Pharmaceutical Research and Manufacturers of
America


Questions and potential solutions
The Future

Specifying the therapeutic context
◦ Explicit definition of product, indication, target
population, formulation, dosage and
contraindications

Specifying the comparator
◦ Standard of care, best in class, watchful waiting,
placebo and nonpharmacological intervention

Defining the time horizon
◦ Duration of exposure and time period over which
benefit-risk events are measured

Specifying the stakeholder perspective
◦ Sponsor, regulators, patients, physicians
1Coplan
PM Clin Pharma Therap 2011; 89:2: 312-15




Identifying pool of candidate outcomes for
the assessment
Deciding which outcomes to include or
exclude from the framework
Documenting all critical assumptions for
these inclusion and exclusion decisions
Value tree is a visual hierarchical presentation
of key ideas, values or concepts
1Coplan
PM Clin Pharma Therap 2011; 89:2: 312-15


Information may come from multiple data
sources
A repository of all data, called data source
table is kept
1Coplan
PM Clin Pharma Therap 2011; 89:2: 312-15


Customization done based on data quality
and limitations
Study end-points are organized in value tree
at 2 levels
◦ Body system category of the benefit or risk
◦ The end-point measured in studies

Approaches to capturing level of severity of
outcomes identified and the value tree is
enhanced
1Coplan
PM Clin Pharma Therap 2011; 89:2: 312-15

Assess relative weights
◦
◦
◦
◦

Stated choice methods
QALYs
Utilities
Value functions
Application of weighted approach in analysis
◦ Net clinical benefit
◦ Number needed to harm (NNH)
◦ Multicriteria decision analysis
1Coplan
PM Clin Pharma Therap 2011; 89:2: 312-15

Key risk benefit table
◦ Flexible table summarizing the key information
needed to quantify outcomes in the value trees

Additional items to enhance key table
◦ Heat map color coding
◦ Embedded graphs
1Coplan
PM Clin Pharma Therap 2011; 89:2: 312-15
1Levitan
Clin Pharma Therap 2011; 89:2: 217-234
1Levitan
Clin Pharma Therap 2011; 89:2: 217-234
1Levitan
Clin Pharma Therap 2011; 89:2: 217-234




Definitions
Summary and Discussions at 2008 OMERACT
Drug Safety Summit
OMERACT approach
Benefit Risk Action Team (BRAT) approach
◦ Pharmaceutical Research and Manufacturers of
America


Questions and potential solutions
The Future

How to come up with method (s) for risk-benefit
assessment that is/are
◦ Easily understood by regulators, patients and physicians
◦ Take into account non-treatment or alternative
treatments
◦ Allow sensitivity analyses?

How to mandate and conduct efficient
postmarketing surveillance studies?
◦
◦
◦
◦
Requirement by FDA and EMEA etc.
Funding
Preventing bias
Design and follow-up methodology


Can efficacy trials be better designed to allow
better short-term risk-benefit analysis?
What steps must be taken to use large
databases to conduct postmarketing studies?
◦ Standardization of methods for reduction of bias
◦ Data integrity and privacy issues

How to achieve an agreement on methods by
major regulators (FDA, EMEA)?




Definitions
Summary and Discussions at 2008 OMERACT
Drug Safety Summit
OMERACT approach
Benefit Risk Action Team (BRAT) approach
◦ Pharmaceutical Research and Manufacturers of
America


Questions and potential solutions
The Future






Collaborative networks of academia, regulatory
agencies, patients and pharma
Use of preferences, values, cost and delay in
drug approval (risks of alternatives) in riskbenefit analysis
Universally applicable methods
Ability to vary values, weights and perform
sensitivity analyses
Utilize non-randomized studies
Design and conduct useful large simple
comparative effectiveness trials