Using Disproportionality Analysis to
Systematically and Simultaneously
Detect Safety Signals in AERS
Jonathan G. Levine, PhD
FDA/CDER
Office Of Pharmacoepidemiology &
Statistical Science
1
Disclaimer
• The views expressed in this presentation
are mine, and do not necessarily represent
the views of FDA.
2
Overview
•
•
•
•
•
•
•
Goals of adverse event analysis.
What is AERS?
Limitations of AERS
AERS versus Clinical Trials.
What can we learn from AERS?
Disproportionality Analysis.
Interpreting Disproportionality Scores
3
Goals of Adverse Event Analysis
• Adverse event analysis seeks to answer these
three questions:
– Which drugs or combinations of drugs cause which
adverse events?
– Which patients are most likely to experience the
adverse event?
– Is an alternate therapy more or less likely to cause
the event?
• AERS data cannot by itself answer these
questions
• AERS data can help to answer these questions
4
AERS Database
• Computerized adverse case reporting system
– Voluntary reporting by health care workers and the general public.
– Mandatory reporting by manufacturers for serious, unexpected events
•
Adverse event reports
– Coded according to the standardized terminology of the Medical
Dictionary for Regulatory Activities(MedDRA)
– Over 2.5 million voluntary reports from 1968 to the present.
– Small number of data elements (drugs, events, age, sex, etc.)
– Lots of missing data
• Safety issues leading to drug withdrawal from the market have been
discovered using the AERS database
5
Limitations of AERS
•
No denominator
– Population rates cannot be estimated.
•
Differential under-reporting of events can bias results.
– Consistent under-reporting for a drug or event does not bias results
– Under-reporting that is a function of drug and event does bias results
•
Drug name errors
– Data entry is not done following strict drug naming standards.
•
•
Missing data
MedDRA is often too granular
– Numerous terms for seizures, strokes, etc.
•
MedDRA codes are often difficult to aggregate
– Higher level groupings may contain an event and its opposite, e.g. hypertension
and hypotension.
•
MedDRA is not comprehensive
– A very unusual adverse event may not have a MedDRA term.
•
Duplicate reports
– Multiple manufacturers may submit the same report.
6
AERS versus Clinical Trials
• “Representative” population?
– Clinical trials: non-random samples, certain types of patients are
intentionally excluded.
– AERS: non-random sample, no one intentionally excluded.
• Generalizability
– Clinical trials: results may not apply to other patient populations.
– AERS: results apply to the entire patient population (in theory);
reporting may be driven by events incompletely addressed in the
labeling.
• Number of patients studied
– Clinical trials: small (hundreds, maybe thousands of patients).
– AERS: everyone (in theory…but under-reporting biases and
labeling of an event may in practice reduce the number of
patients effectively studied).
7
AERS versus Clinical Trials
• Ascertainment?
– Clinical trials: good ascertainment for serious events with fairly
rapid onset and more subtle events that are the focus of
monitoring
– AERS: under-reporting
• Causality
– Clinical trials: Randomization makes causality assessment
straightforward.
– AERS: causality assessment difficult without using external
information. ( e.g. good patient descriptions, linked to an
objective marker such as digoxin levels, potassium levels, and a
correlated EKG and positive dechallenges and rechallenges)
• Complexity of research question
– Clinical trials: limited by cost and ethics.
– AERS: limited by actual drug use and reporting.
8
What can a statistician learn from
AERS?
• Has adverse event Y been reported in
patients taking drug X?
• Are reports in AERS containing drug X
more likely to also contain adverse event
Y?
Question: How can we analyze AERS data
in order to answer these questions?
9
Disproportionality Analysis Using
DuMouchel’s MGPS Method
•
Calculate observed and expected number of reports for a particular drugevent combination.
Observed rate = Number of reports for event X with drug Y
Number of reports for drug Y
Expected rate = Number of reports for event X in AERS
Number of reports in AERS
Reporting Ratio (RR)= Observed rate
Expected rate
•
•
“Shrink” the RR towards 1. The shrunk RR is referred to as the EBGM
score.
The amount of shrinkage is a function of the amount of information in AERS
about the drug-event combination.
10
Disproportionality Analysis
• MGPS done on a “cleaned” version of AERS
– Duplicate removal, drug name standardization
• All scores are calculated simultaneously.
• Expected rates are usually calculated stratifying
by age, sex, and report year.
• The strata-level expected rates are combined
using strata weights proportional to strata size.
• A common measure of the importance of a drugevent combination is the lower limit of the 90%
credible region (“confidence interval”) for the
EBGM, referred to as EB05.
11
What Does an EB05>1 Mean?
• An EB05>1 suggests that in AERS there is an
association between the drug and the event.
• An EB05>1 does not prove that a drug causes
an event.
• What is a large EB05 value?
– A signal having an EB05 > 2 indicates that the D-E is
at least twice the expected
– In many cases, an EB05 >1, or EB05 >1.5 may be a
more useful definition.
12
When Are Findings from Disproportionality
Analysis Most Compelling?
• Very low background rate in general
population, e.g. “growing feathers”
• Very low background rate in similar
patients
• Objective outcome
• Any finding must be viewed in context.
13
A. B. Hill not R. A. Fisher
(http://www.edwardtufte.com/tufte/hill)
• Hill asked: “What aspects of that association should we
especially consider before deciding that the most likely
interpretation of it is causation?”
–
–
–
–
–
–
–
–
–
Strength.
Consistency
Specificity
Temporality
Biological gradient
Plausibility
Coherence
Experiment
Analogy
• Determination of causality requires more than statistics.
14
Does Disproportionality Analysis
Work?
• MGPS has identified emerging adverse events.
• EB05s > 1 are usually seen for known adverse
events.
• Implausible signals are rarely seen, (but
sometimes the event causes the drug; need
medical input to determine direction of
causality).
• Innocent bystanders problem and signal leakage
in cases of polypharmacy with independence
model are still a problem. As a means to
address these issues, Bayesian logistic
regression methods are being studied.
15
Conclusions
• AERS provides useful information about adverse events.
• Clinical trials cannot replace the information provided by
AERS.
• Biases exist in AERS, and the exact nature of the biases
is impossible to determine.
• Disproportionality analysis can provide an understanding
of the associations between drug-event pairs in AERS.
• Disproportionality analysis of AERS cannot by itself
determine if there is a causal link between a drug-event
pair.
16
Selected References
•
•
•
•
•
•
•
•
•
Almenoff, J. S., W. DuMouchel, A. Kindman, X. Yang and D. M. Fram (2003).
"Disproportionality analysis using empirical Bayes data mining: a tool for the evaluation of
drug interactions in the post-marketing setting." Pharmacoepidemiology and Drug Safety
12(6): 517-521.
DuMouchel, W., Bayesian data mining in large frequency tables, with an application to the
FDA Spontaneous Reporting System. The American Statistician, 1999. 53(3):177-190.
DuMouchel, W. and D. Pregibon, Empirical Bayes screening for multi-item associations. In
7th ACM SigKDD Intl Conference on Knowledge Discovery and Data Mining. 2001. San
Francisco: ACM Press.
Fram DM, Almenoff JS, DuMouchel W (2003) Empirical Bayesian Data Mining for
Discovering Patterns in Post-Marketing Drug Safety Data Proc. ACMSIGKDD 2003 Intl.
Conf. on Knowledge Discovery from Data.
Niu, M.T., D.E. Erwin, and M.M. Braun, Data mining in the US Vaccine Adverse Event Reporting
System (VAERS): early detection of intussusception and other events after rotavirus vaccination.
Vaccine, 2001. 19: 4627-37.
O'Neill, R.T. and A. Szarfman, Discussion: Bayesian data mining in large frequency tables, with an
application to the FDA Spontaneous Reporting System by William DuMouchel. The American
Statistician, 1999. 53(3):190-6.
O'Neill, R.T. and A. Szarfman, Some FDA perspectives on data mining for pediatric safety
assessment. Curr Ther Res Clin Exp, 2001. 62:650-663.
Szarfman, A., S.G. Machado, and R.T. O’Neill. Use of Screening Algorithms and Computer
Systems to Efficiently Signal Higher-Than-Expected Combinations of Drugs and Events in the US
FDA’s Spontaneous Reports Database. Drug Safety, 2002. 25(6): p. 381-392.
van Puijenbroek EP, Diemont WL, van Grootheest K Application of Quantitative Signal Detection
in the Dutch Spontaneous Reporting System for Adverse Drug Reactions Drug Safety 2003; 26
17
(5): 293-301