Using Social Media to Support
Safety Surveillance Efforts
Nabarun Dasgupta, MPH, PhD
Epidemico, Inc.
September 11, 2015 – M-CERSI Workshop
Slides available now
@epidemico
REAL-WORLD EXPERIENCE
00
01
06
09
10
Disclosure
Data
Taming
Supply Chain
Clinical Trials
Post-marketing
11
13
14
16
20
Space + Time
Patient Benefits
Limitations
DEMO
Minutes
Disclosures Funding Sources
EPIDEMICO, Inc. + EPIDEMICO, Ltd.
Authors are employees or consultants. Epidemico is commercializing aspects of this software platform. Epidemico is a wholly
own subsidiary of Booz Allen Hamilton.
01. USA
02. EUROPE
FDA has been funding the
research into the technology for
3 years.
A public-private partnership entering Year 2.
The WEB-RADR project is supported by the
Innovative Medicines Initiative Joint
Undertaking (IMI JU) under grant agreement n°
115632, resources of which are composed of
financial contribution from the European
Union's Seventh Framework Programme
(FP7/2007-2013) and EFPIA companies’ in kind
contribution.
Office of the Chief Scientist
(OCS)
Center for Tobacco Products
(CTP)
Office of International Programs
(OIP)
03. INDUSTRY
GSK has been a development
partner to extend the technology
platform to meet the standards of
regulated software for the last year
and half. Epidemico provides data to
other pharmaceutical companies
and CROs.
How to Tame Social Media
The Drug I’m Responsible
Taking
for Monitoring
DATA PROCESS OVERVIEW
Collect public
social media data
Remove false
positives
(optional)
Synthesis of
information from
other sources
Acquire
Curate
Causation
Filter
Statistics
Automated processes
to identify events and
clean data
Hypothesis
generation
SCRANTON B, DE VRIES J, BERINATO S, BORTZ C. HARVARD BUSINESS REVIEW, JUNE 2013
AUTOMATED PROCESSES
4
1
3
2
Geotag
Consolidate
Translate
Identify Events
Automated NLP Bayesian
classifier
Machine learning
Internet vernacular to
MedDRA v18 preferred
terms
Multiplicate copies &
posts from same user
consolidated
(PII removed)
Geography identified
if available using
Google API
visual snow dont want to eat cross vision lost my appetite
apetite surpressed googley eyed no appetitey doublevision
blind appetite is nonexistent blurry vision miss feeling hung
anorexic double vision #notevenhungry blindness cant eat
uldn’t see making me eat like a mouse vision change blurry
st appetite lost their eyesight didn’t get hungry seeing doub
can’t eat killed my apetite googly eyed lost teh appetite
lost apetite crosseyed killed my appetite seeing weird cou
cross eyed seeing weird colour stomach small changes in vis
lack of apetite never want to eat seeing weird color never h
Internet
Combined
Spelling
Implied
Vernacular
or
Typos
invented
Variations
phrases
Translation
words
visual snow dont want to eat cross vision lost my appetite
apetite surpressed googley eyed no appetitey doublevision
blind appetite is nonexistent blurry vision miss feeling hung
anorexic double vision #notevenhungry blindness cant eat
uldn’t see making me eat like a mouse vision change blurry
st appetite lost their eyesight didn’t get hungry seeing doub
can’t eat killed my apetite googly eyed lost teh appetite
lost apetite crosseyed killed my appetite seeing weird cou
cross eyed seeing weird colour stomach small changes in vis
lack of apetite never want to eat seeing weird color never h
Visual impairment
Decreased appetite
MedDRA 10047571
MedDRA 10061428
Visual impairment
Loss of appetite
SNOMED 397540003
SNOMED 79890006
Proto-AE
Posts with Resemblance to Adverse Events
SYSTEM PERFORMANCE
~100% with manual curation
60M
380k
68%
88%
Posts mentioning
medical products
ML classifier
trained by manual
review of posts
Precision
Positive predictive
value
Recall
Sensitivity
APPLICATION ACCESS (API)
Supply Chain Safety for Drugs & Food
OIP investigated further on Tingyi Holding Corp:
- The company actively ships to the U.S. with
increasing shipments from 2011-2014, two
refused entry lines in 2012 and four detained or
reconditioned lines from 2011-2013
- Products include instant noodles, tea, fruit
juices, and prepared baked goods
- Multiple distributors of its products with the
U.S.
- OIP presented its findings to the Centers with a
mapping of potential networks of suppliers
identified using FDA internal systems.
Counterfeit Products: Weibo
Consumer purchased
extremely cheap ointment sold
under brand name Bactroban
but was not manufactured by
GSK.
GSK’s Bactroban
Clinical Trial Unblinding and Drop-out
CLINICAL TRIAL DISCUSSION
I'm in a [Company] double blind ######### trial and could receive
the placebo up until week 16 at which point I'm guaranteed to
receive the drug.
What I don't understand is that I can't be told if I was on the
placebo for the first 16 weeks. I'm not seeing any positive benefit
from whatever I've been taking for 8 weeks so far, and I would
really like to know at week 16 if it's the drug because then I can
bail from the study at that time and move on to something else…
19
CLINICAL TRIAL DISCUSSION
It sounds like [your father] very well may be on the real thing.
Those are definite improvements.
…I rather doubt my bro is on the real thing, even with the physical
things that have happened to him (constipation, UTI and prostate
infections.) Either that, or it's not a medication that's going to work
for him. I'm so afraid the latter is the case. If so, what next??
…If he's been on it, I can't see any reason to continue…If he
hasn't been getting it, then certainly it's worth continuing with the
real thing. Problem is we won't find out until this study is
completed for everyone currently in it…
20
CLINICAL TRIAL DISCUSSION
He started on Thursday. He told me yesterday that he tried to chew a
capsule (I'm not sure why, I think he wanted to see if it had a flavor) & it
was hard as a rock. I'm assuming maybe it was eneric coated or
something. He took the no flavor to mean he's on the placebo, but I
actually wonder if they would bother enteric coating a placebo.
Probably doesn't matter & I know we're not supposed to know anyway,
just wondered if anyone else's was enteric coated.
21
Rectifying A False Negative in Spontaneous
Report Data
SOCIAL DATA
HELPING
SPONTANEOUS DATA
Routine Portfolio Surveillance
Review Company’s
Safety Database
Twitter Post Identified
“Did this product cause x, y, z? I’ve
had 3 administrations of this
Submit to Regulator
product and ever since then have
felt a, b, c.”
172
Real-time Geo-temporal Surveillance for Adverse
Events
Asking New Kinds of Questions in Pharmacovigilance
DATA PROCESSING
18 month study period
Exclude Chantix/Champix
DTC advertising allowed in non-US countries
58 Rx Drugs w/DTC ads
4,499
5,535
Identify Posts with Proto-AEs
Automated classifier
11,265
218,512
Limit to English Language
Only 6 posts with unknown language
338,263
408,867
Import Twitter Posts
All DTC products
1,245,530
Keep posts with in-scope geography
Keep posts from US, UK, CA, AU
Identifiable geography for 55% of Proto-AEs
Consolidate multiplicate copies
35% Excluding literal duplicates and retweets
Remove Spam, news, coupons
67% of posts were spam
18 month study period
58 Rx Drugs w/DTC ads
Daytime Television
Proto-AEs
Prime Time
350
300
200
16:00
12:00
20:00
0:00
150
8:00
100
4:00
50
0:00
24:00
Hour of the Day (local time)
0
Tweets per 15 Minute Increment
250
W h a t H a p p e n s a t Results
10 O’Clock?
“The majority of postings [about
migraine] occurred in the United
States, peaking on weekdays at
10:00h and then gradually again at
22:00h…”
"Most AEs occurred between 08:00
and 12:00a.m. with a peak around
10:00 a.m."
Can we quantify patient benefit?
12 month study period
15 GSK Rx & OTC Products
Source: Powell GE, Duke SP, Bell HG, Anderson LS, Metcalf MA. In Their Own Words: Social Listening for “Real-World
Benefits” from Prescription and OTC Products. Poster, ISPOR 2015.
DEMO
LIMITATIONS OF SOCIAL MEDIA
01
02
03
CAUSALITY & VERACITY
Patients may not correctly assess
causality. Can system be gamed?
Develop tools to assess
manipulation.
VOLUME
Volume of reports likely to be large.
Reduce false positives and create
automated tools to triage
information.
SIGNAL
DETECTION
Very limited statistical methods to
detect problems. Collaborate with
academia, industry and regulators
to refine methods.
04
PRIVACY & GENERALIZABILITY
Patient privacy expectations and
fear of government oversight. Use
publicly available data only.
05
REGULATION UNCLEAR
When is there an obligation to monitor
or report? Work with regulators and
industry to clarify guidance.
Carrie
Mike
Aranka
Matt
Melissa
Ciara
Nabarun
Nico
Wenjie
Carly
Harold
Derek
Essam
John
Robin
Kyra
Lou
Chi
Lucas
Sue
Kelly
Kate
Clark
Chris
Robin
WE WANT 2 HEAR FROM U
@epidemico
@WEBRADR
demo.medwatcher.org
nabarun@epidemico.com