From signal detection to
verification
Marie Lindquist
Safety assessment steps
Knowledge
Benefit > harm
Signal verification
Signal quantification
Signal strengthening
Signal detection
Effectiveness - risk
assessment
Evidence of causality
and probability
Incidence assessment
Characterisation of problem
Hypothesis generation
Time
Marie Lindquist, Uppsala Monitoring Centre
Spontaneous reporting
- advantages
• A unique source of safety signals
• Gives information about effects in ’real
world’ use
– as opposed to the often idealised conditions in
e.g. clinical trials
• Has a potential to identify rare reactions
• Gives indications about which problems are
considered most troublesome in practice
Marie Lindquist, Uppsala Monitoring Centre
Spontaneous reporting
- advantages
• Continuous data collection
• Low cost
• Covers whole populations (at least in
theory)
• Exists in 90+ countries
• All countries have access to each other’s
data through the WHO Programme network
Marie Lindquist, Uppsala Monitoring Centre
Spontaneous reporting
- possibilities
• Adverse reaction reports can tell us a lot about
the concerns of
– treating and prescribing doctors/dentists/midwifes
– other health professionals involved in patients’
medicines treatment
– patients themselves!
Marie Lindquist, Uppsala Monitoring Centre
Spontaneous reportingpossibilities
• New players
– can give more/different information and better
data quality
• More resources for data analysis
– instead of spending all money on perfect
reporting systems and bureaucratic processes
• Widened horizons
– focus on patient safety instead of drug problems
Marie Lindquist, Uppsala Monitoring Centre
Spontaneous reporting
- problems
• Low levels of participation
• Bureaucratic systems
• Lack of data quality
• Causality assessment difficult
• Generalisability?
• Lack of dialogue and collaboration between
stakeholders
Marie Lindquist, Uppsala Monitoring Centre
Spontaneous reporting
- problems
• Not suited for detection of
– Delayed reactions
– ADRs with a high disease background incidence
• Confounding
– where disease under treatment could be linked to the
outcome
• Diagnostic bias
• Channeling
– Susceptible patients switched to a new drug thought to be
better
• Underreporting and missing data
Marie Lindquist, Uppsala Monitoring Centre
Spontaneous reporting
- problems
We have a numerator
Frequency of reporting
Frequency of exposure
But not a denominator!
Marie Lindquist, Uppsala Monitoring Centre
Epidemiology basics
Cases
Controls
Exposed
population
A
B
Non-exposed
population
C
D
Frequency
Risk
Incidence = # cases developing over a defined time period
Prevalence = # cases at a given point in time
Absolute risk (exp)
A/(A+B)
Reference risk (non-exp)
C/(C+D)
Marie Lindquist, Uppsala Monitoring Centre
Relative risk
Risk (exp)/Risk (non-exp)
With case reports alone, we cannot
• Calculate incidence/prevalence
– Do not know the frequency of the ADR*)
– Do not know the size and characteristics of the exposed
population
• Calculate relative risk
– Do not know the frequency of the reaction among the
non-exposed
– Do not know the size and characteristics of the nonexposed population
*) due to (variable) under-reporting
Marie Lindquist, Uppsala Monitoring Centre
Spontaneous reporting or
epidemiological studies?
• Case report
• ?
• Case series
• ?
• Signal => Hypothesis
• Hypothesis
• Analysis
• Study design
– Qualitative
• Study execution
– Reporting frequency
• Analysis
– Causality
– Incidence
– Probability
Marie Lindquist, Uppsala Monitoring Centre
– Frequency
– Incidence
– Probability
– Causality
Spontaneous reporting or
epidemiological studies?
Signal
detection
Signal
Signal
strengthening quantification
Case reports
Signal
verification
Epidemiology
Statistical evidence
Clinical & pharmacological
information
Marie Lindquist, Uppsala Monitoring Centre
We need both
Spontaneous reporting
• Good information on
individual harm
Epidemiology
• Good information on
studied population
but
• Generalisation to whole
populations?
Marie Lindquist, Uppsala Monitoring Centre
• Incidence not translatable
to individual risk
Signal strenghtening
Using spontaneous reports
Qualitative signal assessments
• Documentation level
– Are all relevant facts present?
• Credibility of cases
– Result and accuracy of individual causality
assessment
• Credibility of signal
– Enough cases?
– Pointing in the same direction?
– Absence of reverse findings?
Marie Lindquist, Uppsala Monitoring Centre
Building a signal
4
feasible
cases
3
index
cases
2
substantial
cases
1
index
case
7
3
case reports case reports
Marie Lindquist, Uppsala Monitoring Centre
Qualitative signal assessments
• Confounding
– Other risk factors than drug present?
• Bias
– Selection, publication, measurement bias
present?
• Individual and public health impact of the
signal
– Degree and duration of harm caused
Marie Lindquist, Uppsala Monitoring Centre
Quantitative signal detection
using surrogate denominators
• Use other reports in database
– as surrogates for exposure and controls
– to calculate statistical disproportionality
Marie Lindquist, Uppsala Monitoring Centre
Quantitative signal detection
methods
Cases
Exposed
population
Non-exposed
population
Controls
Drug X+
ADR Y
Drug X+
Other ADRs
Other
drugs +
ADR Y
Other drugs+
Other ADRs
Marie Lindquist, Uppsala Monitoring Centre
Signal strengthening
Using drug usage denominator
data
A realistic first approach
Make an estimate of the incidence in the
exposed population
• Use existing ADR data (numerator) AND
• Drug use data (denominator)
Marie Lindquist, Uppsala Monitoring Centre
Possible measures of drug use
• How much was actually taken by patients?
• How much was dispensed?
• How much was prescribed?
• How much was sold?
Level of
certainty of
measure
Marie Lindquist, Uppsala Monitoring Centre
What measures are available?
• Actual consumption data
• Dispensing data
• Prescription data
• Sales data
Availability
of data
Marie Lindquist, Uppsala Monitoring Centre
Sources of drug usage data
• Sales data
– IMS (worldwide)
– National wholesales information
– Hospital/pharmacy information
• Prescription data
– IMS medical indices (worldwide)
– National prescription databases
– Healthcare databases
• GPRD, Thin/EPIC, Saskatchewan, Medicaid etc.
Marie Lindquist, Uppsala Monitoring Centre
Sales data as denominator
Marie Lindquist, Uppsala Monitoring Centre
Combining ADR and sales data
Reporting rates are calculated as
Number of ADR reports
Amount drug sold (exposure surrogate)
Marie Lindquist, Uppsala Monitoring Centre
Prescription data as denominator
Marie Lindquist, Uppsala Monitoring Centre
Combining ADR and prescription data
Comparable reporting rates are calculated as
Number of ADR reports
# prescriptions (exposure surrogate)
Marie Lindquist, Uppsala Monitoring Centre
Reporting rates using sales or
prescription data
Can give an indication of
– The size of the problem (incidence estimate)
– Differences between the target drug and its
comparator drugs
– Differences between countries
– Differences over time (if longitudinal data is
used)
Marie Lindquist, Uppsala Monitoring Centre
Reporting rates using prescription
data
In addition to when sales data is used, analyses can
be made involving
– Patient age
– Gender
– Indication
– Dosage
Marie Lindquist, Uppsala Monitoring Centre
Using this method were able to
• Identify new problems on old drugs
• Elucidate reasons for country differences, e.g.
– Differences in drug usage
– Publication bias influences reporting
• Identify differences in reporting patterns between
drugs
• Evaluate if an identified signal for one drug may be
a group/class effect
• Show that the public health impact of a signal could
have been recognised earlier
Marie Lindquist, Uppsala Monitoring Centre
Reports of carbamazepine and Stevens
Johnson syndrome in WHO database
Country
# reports
Thailand
318
United States
155
Germany
144
Malaysia
129
United Kingdom
104
Spain
37
Sweden
31
Canada
28
Switzerland
16
Italy
10
Netherlands
8
Denmark
2
Marie Lindquist, Uppsala Monitoring Centre
Adding sales data
Country
Malaysia
Thailand
Sweden
Switzerland
United Kingdom
Netherlands
Germany
Spain
Canada
United States
Denmark
Italy
Marie Lindquist, Uppsala Monitoring Centre
# reports sum KG sales sum
129
4,705.9
318
28,797.7
31
95,036.4
16
53,641.7
104
604,275.6
8
53,917.6
144
990,518.6
37
307,818.3
28
258,180.9
155
2,086,026.5
2
44,577.6
10
448,681.8
# reports/
mill DDD
27.4
11.0
0.3
0.3
0.2
0.1
0.1
0.1
0.1
0.1
0.0
0.0
Lessons learned
• Direct comparisons drugs/countries should not be made
without investigating possible reasons for differences
• The choice of comparator drug/s is important
– problems if not used for the same indication, or if the
severity of the disease treated is different
– analyses should be made at similar times in their
marketed lives
• Biases in denominator and numerator can be different
– E.g. under-reporting in numerator and not in
denominator
Marie Lindquist, Uppsala Monitoring Centre
Lessons learned
• Method only useful when there is a relatively large
amount of reports
– particularly when involving more variables than drug/
country/ year
• 'Push button' merging of the different data sets is
not possible
– more or less extensive manual mapping/data washing
needed
Marie Lindquist, Uppsala Monitoring Centre
Adding drug usage denominators conclusion
• Combining drug use data and spontaneous ADR data is
helpful when analysing signals
– But reporting rate
≠
incidence
– Gives a more or less good estimate of incidence
– Need data on background rate of disease to
compare risk in exposed/non-exposed populations
• Be careful when making comparisons between
drugs/countries/different time periods
– be aware of all factors that can influence
denominator and numerator
Marie Lindquist, Uppsala Monitoring Centre
Spontaneous reporting or
epidemiological studies?
Signal
detection
Signal
Signal
strengthening quantification
Case reports
Signal
verification
Epidemiology
Statistical evidence
Clinical & pharmacological
information
Marie Lindquist, Uppsala Monitoring Centre
Towards signal verification
• Cohort event monitoring
• Signal testing using
longitudinal health care
data
Cases + controls (exposed)
• Case-control studies
As above + non-exposed
cases and controls
• Cohort studies
• Other study designs
• Randomised clinical trials
Marie Lindquist, Uppsala Monitoring Centre
As above + study
population randomly
divided into exposed/nonexposed
Type
Question answered
Weaknesses
Case reports
(spontaneous
reporting)
Reporting frequency of No controls, no information
a particular outcome on exposure
Case reports +
usage data
Reporting rate
(number of cases
compared with data on
how much a drug was
used)
Marie Lindquist, Uppsala Monitoring Centre
Strengths
Cheap, covers whole country
populations (in theory). Can
detect rare reactions, postmarketing
Cases and users (exposure
Quick, cheap
surrogate) not from the same
population. No comparison
with non-exposed. Gives only
a crude estimate of incidence
in the exposed
Type
Case-control study
Question answered
Strength of relationship
outcome - exposure (how many
were exposed and how many
were not among the cases; and
the same for the controls).
Starting with identified cases.
Can be done retrospectively
Cohort study
Weaknesses
Matching cases and controls Relatively quick and cheap. Can
controls not necessarily
study multiple exposures.
representative of the whole study Works for rare diseases/ADRs
population. Risk of bias and
confounding. Less certain
causality
Incidence - how many will
Takes time, and high cost.
develop the outcome (cases) and Exposure decided on clinical
how many will not in the
grounds (by study leader -risk of
exposed population and in the bias). Sometimes limited in size
non-exposed population.
and power (to detect rare
outcomes)
Starting by allocating exposure.
Prospective
Randomised controlled As with cohort studies, but
trial
population randomly divided
into exposed/non-exposed
Marie Lindquist, Uppsala Monitoring Centre
Strengths
Takes time, and high cost.
Practically limited in size and
power. Sometimes ethical
problems withholding treatment
from controls
Can look at multiple outcomes.
Less risk of bias and
confounding than in casecontrol studies. Gives incidence
Risk more likely to be correct
since possible selection bias can
be ruled out (random allocation
of exposure), and confounding
can be controlled for