Pragmatic cohort studies and
comparative effectiveness
Eric S Johnson, PhD
Kaiser Permanente’s
Center for Effectiveness and Safety Research
AHRQ’s Effective Healthcare Program:
Scientific Resource Center
June 18, 2010
UC Davis School of Medicine
Copyright H Photography 2006 iStockphoto.com
http://www.itv.com/Drama/family/JeevesandWooster/picturegallery.html
The risk of death associated with the use of
conventional versus atypical antipsychotics
among elderly patients
Sebastian Schneeweiss, Soko Setoguchi, Alan Brookhart, Colin Dormuth,
Philip S Wang
Canadian Medical Association Journal 2007;176:627-32
Copyright Matt Knannlein 2005 iStockPhoto.com
BMJ’s PICO format: Study question
Do conventional antipsychotic drugs pose risks equal to or
greater than those associated with newer, atypical drugs
in an elderly population?
Copyright Matt Knannlein 2005 iStockPhoto.com
BMJ’s PICO format: Design, intervention and control
We conducted a retrospective cohort study of elderly patients
with a diagnosis of dementia who started a conventional or an
atypical antipsychotic drug.
Copyright Matt Knannlein 2005 iStockPhoto.com
BMJ’s PICO format: Participants and setting
We followed 37,241 patients 65 years or older who started
an antipsychotic drug between 1996 and 2004 and were
residents of British Columbia.
Copyright Matt Knannlein 2005 iStockPhoto.com
BMJ’s PICO format: Outcomes and their timing
We compared the 180-day all-cause mortality rate for patients
who started conventional versus atypical antipsychotic drugs.
Copyright Matt Knannlein 2005 iStockPhoto.com
BMJ’s PICO format: Results and role of chance
Of the 24,359 (control cohort) patients who started an atypical
antipsychotic drug, 9.6% died by 180 days. In adjusted
analyses, patients who started a conventional drug were 32%
more likely to die (hazard ratio=1.32; 95% CI, 1.23 to 1.42)—
an excess of 3.5 deaths per 100 patients.
Copyright Matt Knannlein 2005 iStockPhoto.com
A pragmatic-explanatory continuum indicator summary
(PRECIS): a tool to help trial designers
Kevin Thorpe, Merrick Zwarenstein, Andrew Oxman, and others
J Clin Epidemiology 2009;62:464-75.
“Pragmatic trials seek to answer the question:
‘Does the intervention work under usual conditions.’ ”
Copyright Matt Knannlein 2005 iStockPhoto.com
The pragmatic to explanatory trials continuum*
Flexibility of
comparison
intervention
Practitioner
expertise
(comparison)
Follow-up
intensity
Practitioner
Expertise
(experimental)
Flexibility of the
experimental
intervention
Eligibility
criteria
E
Primary
analysis
Outcomes
Participant
compliance
Copyright 2007 Robert Churchill
iStockPhoto.com
Practitioner
adherence
* J Clin Epidemiology 2009;62:464-75
A pragmatic cohort study: new users
New atypical
antipsychotic drug
New conventional
antipsychotic drug
No antipsychotic drugs
filled: year-long
baseline
Copyright 2007 Robert Churchill
iStockPhoto.com
Time
Zero
Intention-totreat analyses
to 180 days
Evaluating medication effects outside of clinical trials:
New-user designs
Wayne Ray
Am J Epidemiol 2003;158:915-20
“[P]revalent users can introduce two types of biases:
(1) Underascertainment of events that occur early in therapy
(2) Inability to control for predictors that may have been altered
by the study drugs.”
Prevalent users also preclude meaningful propensity scores.
Copyright Matt Knannlein 2005 iStockPhoto.com
Valid comparisons with usual care?
One class of intervention
Usual care:
New or current users?
Consistency of alternatives?
Any criteria for
medications filled in
year-long baseline?
Copyright 2007 Robert Churchill
iStockPhoto.com
Time
Zero
How might we design a
score to predict the usefulness of
pragmatic cohort studies?
How well does the cohort answer
decision-makers’ questions?
Inferiority or non-inferiority?
How closely does the cohort resemble
a pragmatic trial? Design? Analysis?
.
How easy is the cohort for systematic
reviewers to GRADE and synthesize?
An unintended outcome, a harm?
Copyright 2006 Ethan Myerson iStockPhoto.com
“[A] national CER program
should promote the
professional ethos that
places the interests of
patients and the larger
community above all other
considerations.”
Comparative effectiveness
research: a report from the
Institute of Medicine
Harold Sox and Sheldon Greenfield
Annals of Intern Med 2009;151:203-5
Copyright 2006 Ethan Myerson iStockPhoto.com
“He wants to look at his plate,
see a steak, and say,
‘I like steak!’ ”
“Give people what they want,
then later you can give them
what you want.”
Pascal in Big Night (1996)
Copyright Juan Monino 2006 iStockPhoto.com
Variation in the risk of suicide attempts and
completed suicides by antidepressant agent
in adults
Sebastian Schneeweiss, Amanda Patrick, Daniel Solomon, Jyotsna Mehta,
Colin Dormuth, Matthew Miller, Jennifer Lee, Philip S Wang
Archives of General Psychiatry 2010;67:497-506
Copyright Matt Knannlein 2005 iStockPhoto.com
Design, intervention and control:
Advanced strategies for propensity scores
“After plotting and comparing the distribution of propensity
scores…we truncated our study population to the area of
overlap. [S]ubjects with non-overlapping propensity scores may
have absolute clinical indications or contraindications for a
particular treatment.”
Copyright Matt Knannlein 2005 iStockPhoto.com
Design, intervention and control:
Advanced strategies for propensity scores
“We constructed a high-dimensional propensity score based on
the top 500 identified variables in addition to the variables
identified by us…We consider results from this
analysis to be the most completely adjusted… ”
Copyright Matt Knannlein 2005 iStockPhoto.com
Design, intervention and control:
Advanced strategies for propensity scores
“We created adjusted Kaplan-Meier plots by weighting each
subject by the inverse of his or her probability for treatment
as estimated in the propensity score analysis.”
Copyright Matt Knannlein 2005 iStockPhoto.com
Probability of remaining free of suicide or suicide attempt
since treatment with sertraline (36,135) or fluoxetine
(22,207), weighted by the inverse probability of treatment
from the propensity score.
From Schneeweiss et al. Arch Gen Psychiatry 2010;67:497-506
Sertraline vs. fluoxetine
Composite event-free probability
1.000
0.998
0.996
0.994
Sertraline
Fluoxetine
0
3
6
9
Time since treatment initiation (months)
12