Informatics an
nd Observational
Res
search
h
Mark Weiner
Weiner, MD
mweiner@mail.med.upenn.edu
Division of Gen
neral Internal Medicine
Institute for Translational Medicine and Therapeutics (ITMAT)
Office of Hum
man Research (OHR)
University of Pennsyylvania School of Medicine
RCTs vs Observatio
onal Research


Randomized Controlled Trials ((RCTs)) are
considered to be the hig
ghest tier of evidence
because the randomizattion balances clinical issues
th t may confound
that
f
d the
th association
a
i ti b
between
t
treatment and outcome..
H
However,
need
d tto consiider
d bi
biases and
d ethical
thi l
issues in the conduct off RCTs
–
–
–
RCTs tend
RCT
t d to
t define
d fi coh
horts
t that
th t are very restrictive.
t i ti
Even among patients witth similar clinical characteristics
as the study subjects
subjects, pe
eople who choose to enroll are
different from the genera
al population
Monitoring and use of me
edication is different in clinical
studies from
f
the real worrld
Issues in conductin
ng RCTs

Logistics
g
–
–
–
–

Small RCTs have focused po
opulations, not necessarily
generalizable.
Large RCTs are needed to e
enable subgroup analyses where the
relative effectiveness of two interventions may be altered
RCTs need to be long enoug
gh to ascertain the temporal
differences in effectiveness b
between two interventions
RCTs require larger sample sizes
s
to reliably distinguish the
effectiveness of therapies tha
at are somewhat similar.
Ethics
–
Is it ethical to randomize patients into different therapies when
current standards of care exist, even if the purpose of the research
ness of the standard?
is to assess the appropriaten
The alternative to RCTs

The Goal – Observation
nal analyses
–

Analyses of routinely colllected clinical data looking for
differences in outcome re
elated to differences in treatmen
The problem
–
–
–
The data are not always accurate
Treatments are not rando
omized/Imbalance of measured
and especially unmeasurred confounders
Data collection occurs att scheduled visits which are
arranged based on cliniccal need, not study protocol
Informatics to supp
port Observational
Research

Can informatics meth
hods be applied
–
–
–
To better organize and analyze existing clinical
data for more reprodu
ucible results?
To more formally colle
ect essential data that is not
currently
y being
g collectted consistently,
y or not at
all?
To identify scenarios w
where analyses of
observational data ma
ay or may not provide valid
findings?
Informatics and data accuracy
Technology
gy Solutions
–
–
–
Enable discrete information from non-structured sources (NLP of
clinical notes)
Incorporate information from labss and other ancillary studies that help
support the presence of diagnose
e
Create a truly longitudinal medica
al record that crosses institutional
boundaries and sites of care
Training Solutions
–
Help analysts and investigators u
understand real world clinical issues in
working with clinical data
Policy Solutions
–
–
–
Doctors
D
t
are lousy
l
coders
d
– but
b t its nott always
l
their
th i ffault!
lt!
If we are serious about performin
ng good comparative effectiveness
research, coding
g rules need to be
etter reflect research uses of data, no
just billing purposes
Embrace and work with the ambiiguity inherent in the practice of
medicine
Understanding diffe
erences in clinical
and research data collection
c

Absence of evidence is not evidence of absence
–

Vigilance in exploring for a condition,
c
even if it turns out
NOT to be present, has clinical relevance
–

Just because you don’t see evidence
e
of a disease doesn’t mean
the patient doesn’t have the disease
d
Providers may look harder forr the presence of a condition or a
physical finding in some patie
ents more than others
Clusters of visits are significcant,
cant but not necessarily directly
related to the condition bein
ng studied
–
Someone with a lot of visits re
elated to asthma mayy get
g a
cholesterol drawn sooner than a patient without asthma whose
cholesterol management is be
eing done more routinely
Understanding diffe
erences in clinical
and research data collection
c

To find p
patients with a certain dissease,, you
y need to consider all the
ways the disease may be repressented in diagnosis codes and
ancillary test results
–
–



Spectrum Bias – Some diseasess and conditions are more vigorously
sought in the setting of other disseases and conditions.
Left Censoring -- the first instancce of a disease in the database is
not necessarily the time the dise
ease first appeared
Right Censoring –
–
–

URI/bronchitis/tracheitis/pharyngitiss/sinusitis are harder to reliably distinguish
that most people would like to thinkk.
Green sputum is not always bacterrial and Yellow sputum is not always viral
data must cover a time interval long enough for the exposure to result in an
outcome
t
that
th t is
i captured
t d in
i the
th d
data.
t
Does the exposure influence the occ
currence of the outcome in 1 year, or 5 years
Rigorous outcomes data collection at scheduled clinic visits may
miss
i changes
h
iin status b
between visits
i i
Understanding diffe
erences in clinical
and research data collection
c




Data from uncaptured clinica
al settings
– Received Angioplasty at big
b city institution A. Hospitalized
at local community hospita
al for GI bleed.
Logical inconsistencies acrosss information systems
– Allergy to Drug Z listed fro
om data source A
– Prescriptions for Drug Z from data source B
Temporal issues
– Presence of murmur AFTE
ER echo report (or cardiology
visit)
– Heart failure reported AFT
TER starting ACE-I
Uncoded information
– Patients with certain chara
acteristics are cared for in certain
locations
Understanding diffe
erences in clinical
and research data collection
c

Treatment bias
–
–

Testing
g bias
–

Insurer rules limiting the use off some meds to patients with certain
underlying conditions or who have
h
already failed other meds?
Practice variation
–

Patients who test positive for certain
c
tests are more likely to receive
additional testing (related or un
nrelated)
Formulary issues
–

Medications are prescribed because the provider believes the
patient needs them
Sicker patients may systematiccally receive certain types of medicine
more than others
Some meds are favored becau
use of non-clinical issues
OTC Meds
–
Aspirin, other NSAIDs, PPIs
Addressing the Id
diosyncrasies


Informatics solutions to obse
ervational data q
quirks may
y
increase the accuracy of the
e data, but make the analytical
data set less generalizable
–
Requiring an echocardiogram
m to definitively rule in or rule out
diagnosis of CHF limits your cohort to people who were sick
enough
h tto require
i an echo
h – even among patients
ti t who
h turn
t
outt
NOT to have CHF by echo
–
Finding incident cases by lim
miting a cohort to people who have
existed in the system for a while without evidence for a disease of
pp
interest,, and then suddenlyy a code for the disease appears.
–
Finding cases through NLP…
….
Analyses are not necessarily inaccurate, but be mindful of the
impact
p
of the Spectrum
p
Bias
Difficulties with info
ormatics solutions

Bigger is Better -- Integrrating more databases
offers the p
promise of filliing
g in g
gaps
p in the
continuum of care,
– But it also increases the likelihood of finding
clinical conflicts in the
e data for an individual

Semantic interoperabilityy will enable different
information technology ssystems to understand the
true meaning of data be
eing sent
– But
B t have
h
you ever see
en two
t
DOCTORS agree
upon the meaning of what
w
they hear?
Difficulties with info
ormatics solutions

Standards will enable comp
puter systems to share a
common language
g g to descrribe clinical concepts
p
– But the precision inheren
nt in these vocabularies often
exceeds the p
precision of medicine
– More precise terms imply
y a “truthiness” that may not be
valid

User friendly GUIs will enab
ble a broader set of people to
perform better quality resea
arch using the data
– Beware of masking the c
complexity of the underlying
data!
Despite these issu
ues…

High quality research CAN still be performed
with observational da
ata with results that have
convergent validity with RCT
Women s Health Initiative
I
(
(WHI)
)
Clinical Trial Simulation
 Prior
observational stu
udies (mostly
retrospective surveys)) had suggested that
HRT was protective frrom adverse
cardiovascular outcom
mes in addition to their
mes,
beneficial effect on syymptoms and bones
 These findings were disproven
d
by WHI, a
randomized trial of HR
RT use
 Could better data and more rigorous analysis
from clinical systems h
have led to the correct
conclusions?
Observational Me
ethods





Started with General Pracctice Research Database, a
10+ year longitudinal data
abase of the care of 8 millio
people in Great Britain
Selected study “subjects” from database in
contemporaneous time pe
eriod as actual study
Applied
pp
the same inclusio
on/exclusion criteria as stud
Intervention cohort derive
ed from patients who receive
HRT during recruitment p
period.
period
Age matched controls sellected from among people
who did not receive HRT during recruitment period.
Observational Me
ethods
Followed subjects forward
d in time looking for
cardiovascular outcomes and
a other relevant endpoint
Conducted intention to treat and “as-treated” analyses
Conducted analyses that a
adjusted for comorbidities a
concurrent therapies.
P f
Performed
d additional
dditi
l senssitivity
iti it analyses
l
off outcomes
t
on patient subgroups
ubject Selection fo
or HRT simulation
ka why you need to start wiith a large database)
779
905,234
18 472
18,472
R
Results
lt
A comparison of Hazard Ratios
s of HRT use for various outcom
Other RCT Replica
ations
Conclusions
ssues in analyzing clinica
al data to conduct researc
do not invalidate the results.
–
However y
you must be mindful of the impact
p
of the
caveats!
Complexities of data maskked by intuitive (looking)
GUIs do not mean these tools are not valuable
–
However it d
H
does mean use
ers mustt be
b wellll educated
d
t d
about limitations of the too
ol and perhaps motivate them
become more educated ab
bout informatics
The need to address these
e issues requires trained
nformaticians as well as statisticians
s
in the process