STATISTICAL CONSIDERATIONS IN
POSTMARKETING SAFETY EVALUATION
A. Lawrence Gould
Merck Research Laboratories
West Point, PA [goulda@merck.com]
FDA/Industry Workshop
29 September 2006
Washington, DC
OVERVIEW
September 29, 2006
1
Spontaneous AE Reports
• Clinical trial safety information is limited & relatively
short duration
• Safety data collection continues after drug approval
o
Detect rare adverse events
o
Obtain tolerability information in a broader population
• Large amount of low-quality data collected
o
Not usable for trt comparisons or risk assessment
o
Unknown sensitivity & specificity
• Evaluation by skilled clinicians & epidemiologists
• Long history of research on issue
September 29, 2006
2
Information Available Postmarketing
• Previously undetected adverse and beneficial effects
that may be uncommon or delayed, i.e., emerging
only after extended treatment
• Patterns of drug utilization
• Effect of drug overdoses
• Clinical experience with study drugs in their “natural”
environment
September 29, 2006
3
The Pharmacovigilance Process
Traditional
Methods
Insight from
Outliers
Type A
(Mechanism-based)
Type B
(Idiosyncratic)
September 29, 2006
Data
Mining
Detect Signals
Generate Hypotheses
Refute/Verify
Public Health
Impact, Benefit/Risk
Estimate
Incidence
Act
Inform
Change Label
Restrict use/
withdraw
4
Considerations & Issues (An Incomplete List!)
• Incomplete reports of events, not reactions
• Bias & noise in system
• Difficult to estimate incidence because no. of pats at
risk, pat-yrs of exposure seldom reliable
• Significant under reporting (esp. OTC)
• Synonyms for drugs & events → sensitivity loss
• Duplicate reporting
• No certainty that a drug caused the reaction reported
• Cannot use accumulated reports to calculate
incidence, estimate drug risk, or compare drugs
September 29, 2006
5
DATA MINING
September 29, 2006
6
Data Mining is a Part of Pharmacovigilance
• Identify subtle associations (e.g., drug+drug+event) and
complex relationships not apparent by simple summary
• Identify potential toxicity early
• Finding ‘real’ D-E associations similar to finding potential
active compounds or expressed genes – not exactly the
same (no H0) – more like model selection
• Still need initial case review
respond to reports involving severe, potential lifethreatening events eg., Stevens-Johnson syndrome,
agranulocytosis, anaphylactic shock
• Clinical/biological/epidemiological verification of apparent
associations is essential
September 29, 2006
7
Typical Data Display
No. Reports Target AE Other AE Total
Target Drug
a
b
nTD
Other Drug
c
d
nOD
nTA
nOA
n
Total
Basic idea:
Flag when
R = a/E(a) is
“large”
Some possibilities
Reporting Ratio:
E(a) = nTD  nTA/n
Proportional Reporting Ratio: E(a) = nTD  c/nOD
Odds Ratio:
E(a) = b  c/d
• Need to accommodate uncertainty, especially if a is small
• Bayesian approaches provide a way to do this
September 29, 2006
8
Currently Used Bayesian Approaches
• Empirical Bayes (DuMouchel, 1998) & WHO (Bate, 1998)
• Both use ratio nij / Eij where
nij = no. of reports mentioning both drug i & event j
Eij = expected no. of reports of drug i & event j
• Both report features of posterior dist’n of ‘information
criterion’
ICij = log2 nij / Eij = PRRij
• Eij usually computed assuming drug i & event j are
mentioned independently
• Ratio > 1 (IC > 0)  combination mentioned more often
than expected if independent
September 29, 2006
9
Comparative Example (DuMouchel, 1998)
• No. Reports = 4,864,480, Mentioning drug = 85,304
Reports
Mentioning
Reporting Ratio
Expected RR
5% Quantile
Excess n
September 29, 2006
Headache
AE
Both
71,209 1,614
Polyneuritis
AE
Both
262
3
1.23
2.83
WHO
FDA
WHO
FDA
1.29
-300
1.23
1.18
225
0.76
-0
1.42
0.58
0
10
DATA MINING EXAMPLES
INCORPORATING STATISTICAL
REFINEMENTS
September 29, 2006
11
Result From 6 Years of Reports on Lisinopril
Events w/Lower 5% RR Bnd > 2 (Bold  N  100)
N
6
8
9
51
53
50
124
225
696
904
99
214
102
216
E
0.55
0.82
1.15
8.39
9.37
11.5
30.9
60.5
195.9
290.6
31.0
81.6
38.6
91.9
AE (preferred term)
toxic erythema
obstipation
labile hypertension
erythrocytes decreased
peripheral vascular disorder
angina pectoris
hyperkalemia
palpitation
cough
dizziness
serum creatinine increased
angioedema
renal failure
edema
September 29, 2006
RR
8.19
7.97
6.15
5.85
5.41
4.08
3.91
3.66
3.54
3.10
3.09
2.59
2.57
2.32
5% Lwr Bnd Excess N
2.73
0.9
3.30
1.9
2.79
2.1
4.53
29.6
4.21
30.1
3.18
25.0
3.36
72.7
3.28
137.7
3.32
454.5
2.93
562.0
2.61
49.9
2.31
107.0
2.18
45.5
2.08
98.8
12
Persistence (& Reliability) of Early Signals
As of Dec 1996
Mean
Lower
Adverse Event
N EBGM 5% Bnd
renal artery stenosis
6
6.96
2.41
exanthema
23
4.74
3.23
peripheral vascular disorder 23 4.74
3.23
angina pectoris
15 4.36
2.68
serum creatinine increased
36 3.94
2.95
dizziness
349 3.86
3.53
myocardial infarction
26
3.67
2.62
palpitation
73 3.59
2.95
hyperkalemia
32 3.46
2.55
renal failure
53 3.39
2.69
pulmonary edema
10
3.16
1.82
cough
209 3.11
2.77
migraine
19
2.87
1.95
vertigo
22
2.51
1.75
angioedema
62 2.35
1.91
edema
72 2.32
1.91
September 29, 2006
headache
255 2.21
2.00
As of Oct 2000
N
7
48
53
50
99
904
-225
124
102
-696
-84
214
216
--
Mean Lower
EBGM 5% Bnd
4.78
2.03
2.73
2.14
5.41
4.23
4.08
3.18
3.09
2.60
3.1
2.93
--3.66
3.27
3.91
3.36
2.57
2.17
--3.54
3.32
--2.36
1.97
2.59
2.31
2.32
2.07
13 ---
Accumulating Information over Time
• Lower 5% quantiles of RR stabilized fairly soon
4
Lower 5% RR Bnd
3.5
dizziness
3
cough
palpitation
2.5
edema
2
angioedema
1.5
hyperkalemia
renal failure
1
incr. serum creatinine
0.5
0
96
00
nJu 9
-9
ec
D
99
nJu 8
-9
ec
D
98
nJu 7
-9
ec
D
97
n-
ec
Ju
D
95
96
n-
ec
Ju
D
September 29, 2006
14
Time-Sliced Evolution of Risk Ratios
September 29, 2006
a
20
00
a
19
99
a
19
98
a
19
97
a
2
1.5
1
0.5
0
19
96
kalemia =
hyperkalemia
edema =
angioedema
Cough
edema
kalemia
tension
Failure
a
tension =
hypotension
failure =
heart failure
4
3.5
3
2.5
19
95
Change in ICij
for reports of
selected events
on A2A from
1995 to 2000
Exp. RR
• See how values of criteria change over time within time
intervals of fixed length
Half-year interval
15
Masking of AE-Drug Relationships (1)
• Company databases smaller than regulatory
databases, more loaded with ‘similar’ drugs
eg, Drug A is 2nd generation version of Drug B,
similar mechanism of action, many reports with B
• Elevated reporting frequency on Drug B could mask
effect of Drug A
• May be useful to provide results when reports
mentioning Drug B are omitted
September 29, 2006
16
Table 6.
Effect of omitting Drug B. Quantities tabulated are lower 5% quantil
of EBGM
and corresponding
excess
cases over independence
Masking
of AE-Drug
Relationships
(2)
Drug B
Preferred Term
atopic dermatitis
hypotension
left cardiac failure
lichen planus
pharyngeal edema
psoriasis vulgaris
pulmonary congestion
pulmonary edema
renal insufficiency
sudden death
tachycardia
tongue edema
vertigo
September 29, 2006
Included
Omitted
EBGM05 Excess
1.96
9.8
1.87
29.5
1.99
3.0
1.79
4.1
1.47
5.8
1.92
8.4
1.65
3.8
EBGM05 Excess
2.11
10.9
2.44
38.2
2.20
3.7
2.04
5.1
2.32
10.8
2.37
10.4
2.23
5.4
2.12
6.1
2.10
12.2
2.58
4.0
2.21
49.0
2.73
10.7
2.51
41.7
1.96
1.86
3.0
40.9
1.97
33.4
17
Example 2: Vaccine-Vaccine Interaction
• From FDA VAERS database, reports from 1990-2002
• Intussusception is a serious intestinal malady observed to
affect infants vaccinated against rotavirus
• Look at reports of intussusception that mention rotavirus
vaccine (RV) and DTAP vaccine
• DTAP is a benign combination vaccine commonly
administered to infants
• Demonstration question: Intussusception very commonly
reported with RV – but does the reporting rate depend on
whether DTAP was co-administered?
• Not easy to address using standard pharmacovigilance
procedures
September 29, 2006
18
Outline of Analysis
• Standard tools provide intussusception reporting rate
for pairs of vaccines, and for vaccines singly
• Result is a 3-way count table (corresponding to RV +
or -, DTAP + or -, and intussusception + or -)
• Use log-linear model to see if intussusception is
mentioned with the two vaccines together more often
than the separate vaccine-intussusception reporting
associations would predict
• Turns out that there is an association – Likelihood
ratio chi-square is 17.41, 1 df, highly significant
September 29, 2006
19
Observed and Expected Report Rates
Observed
Expected
8
Symbol area  no. of reports
Report Rate, %
6
4
2
RV+
DTAP+
RV+
DTAPRV DTAP+
RVDTAP-
0
-2
September 29, 2006
20
Comments
• Intussusception seems to be reported more often
than expected when RV and DTAP are given together
than when RV is given without DTAP, after adjusting
for individual vaccine-intussusception associations
• Reports of intussception without RV are very rare,
about 4.5/10,000 reports if RV is not mentioned
• The joint effect of RV and DTAP on intussusception
reporting is small, but does reach statistical
significance
• Not clear that apparent association means anything - actual synergy between RV and DTAP seems
unlikely, but explanation requires clinical knowledge
September 29, 2006
21
A NEW BAYESIAN APPROACH
(Gould, Biometrical Journal 2006, to appear)
September 29, 2006
22
Model for Process Generating Observations
• ni = no. of reports mentioning i-th drug-event pair ~
Poisson (true for EB approach as well)
f(ni | Ei, i) = fPois(ni ; iEi)
Expected count
under independence
Association
measure
• i drawn from a gamma(a0, b0) distribution or from a
gamma(a1, b1) distribution
o
A model selection problem
o
Dist’ns reflect physician/epidemiologist’s judgment
as to what range of  values corresponds to
‘signals’, and what does not
September 29, 2006
23
Prior/Model Density of 
• Bayes approach starts with a random mixture of
gamma densities,
Analyst specifies
f0( ; , a0, b0, a1, b1)
parameter values
= (1 - )fgam(; a0, b0) + fgam(; a1, b1)
Use value of Ppost( = 1) for inference
• EB approach starts with expectation wrt  given p 
nonrandom mixture of gamma densities,
Data determine
f0( ; p, a0, b0, a1, b1)
parameter values
= pfgam(; a0, b0) + (1-p)fgam(; a1, b1)
Use quantiles of posterior dist’n of  for inference
September 29, 2006
24
Comments
• Bayes and EB approaches both model strength of
drug-event reporting assn as a gamma mixture
• Diagnostic properties of Bayes method can be
determined analytically or by simulation
• Unknown separation of the true alternative dist’ns for
 more important than prior dist’n used for analysis
• Methods described here can be applied to other
models – Scott & Berger (2005) used normal
distributions – could also use binomial instead of
Poisson, beta instead of gamma distributions to
develop screening methods for AEs in clinical trials
September 29, 2006
25
DISCUSSION
September 29, 2006
26
Discussion
• Bayesian approaches may be useful for detecting
possible emerging signals, especially with few events
• MCA (UK) currently uses PRR for monitoring emergence
of drug-event associations
• Signal detection combines numerical data screening,
statistical interpretation, and clinical judgement
• Most apparent associations represent known problems
• ~ 25% may represent signals about previously unknown
associations
• The actual false positive rate is unknown
September 29, 2006
27
What Next?
• PhRMA/FDA working group has published a white paper
addressing many of these issues
Drug Safety (2005) 28: 981-1007
• Further refine methods, look for associations among
combinations of drugs and events, timing of reports
• Data mining is like screening, need to evaluate
diagnostic properties of various approaches
• Need good dictionaries: many synonyms  difficult
signal detection
º Event names: MedDRA may help
º Drug names: Need a common dictionary of drug
names to minimize dilution effect of synonyms
September 29, 2006
28
Data Used to Construct Plot
Intussception +
Intussception -
Observed Expected Observed Expected
RV + DTAP +
85
74
1111
1122
DTAP 29
40
608
597
RV - DTAP +
4
15
33520
33509
DTAP 293
282
610714
610725
September 29, 2006
29