Principles of Comparative
Effectiveness Research
Mark Helfand, MD
Director, Oregon EPC
http://www.ohsu.edu/epc/
OREGON EPC
Outline
•
•
•
•
•
Antecedents
Comparative effectiveness reviews
Principles for CE research
Applying the principles
Methods research agenda
OREGON EPC
Table published in the third edition of Florence
Nightingale's Notes on Hospitals [9]
Iezzoni, L. I. Ann Intern Med 1996;124:1079-1085
OREGON EPC
Percent Mortality for Inpatients
1863
N
24
12
25
30
13
1
Setting
London hospitals
Hospitals in large towns
County hospitals
Other hospitals
Naval & military hospitals
Margate infirmary
%
91%
83%
39%
40%
16%
13%
Florence Nightingale: Measuring Hospital Care Outcomes by D. Neuhauser. Joint Commission on
Accreditation of Health Care Organization, 1999, 260 pages, ISBN 0 866 88559 5.
OREGON EPC
OREGON EPC
90
5
0
1
0
Tonsillectomy
Hys
Prost
Chol
Appy Hernia
OREGON
EPC
How sure are we?
Expert estimates of breast implant
rupture rates
0% 0.2% 0.5% 1% 1% 1% 1.5% 2% 3% 3%
4%
5% 5% 5% 5% 5% 5% 5% 5% 6% 6% 6%
8%
10% 10% 10% 10% 13% 13% 15% 15% 18%
20% 20% 20% 25% 25% 25% 30% 30% 40%
50% 50% 50% 62% 70% 73% 75% 75% 75%
Source:
Dr. David
75%
80%
Eddy
80% 80% 80% 80% 80% 100%
OREGON EPC
Experts’ estimates of probability of
acute retention in men with BPH
35
30
25
20
Number of
respondents
15
10
5
0
0%
20%
40%
Source: Dr. David Eddy
60%
80%
100%
OREGON EPC
Experts estimates of the effect of
colon cancer screening on chance of
dying
0%
Source: Dr. David
Eddy
25%
50%
75%
100%
OREGON EPC
Summary
•Practice and outcomes Vary (WENNBERG)
•Clinicians are not very good at probabilities (Eddy, Tversky)
•They may cite literature selectively or inaccurately
•They even make logical errors in medical thinking, including
“argument from authority” and “post hoc” reasoning
• In making recommendations to a patient, they may ignore
important information, such as what the patient values
(McNeil)
•Clinical research was often poorly conceived (Feinstein) and
lacked relevance to everyday practice (Fry)
OREGON EPC
1990’s
“Evidence-based Medicine”
Outcomes Research
1994 OTA report
S. 580, the "Healthcare Research
and Quality Act of 1999”
OREGON EPC
Outline
•
•
•
•
•
Antecedents
Comparative effectiveness reviews
Principles for CE research
Applying the principles
Methods research agenda
OREGON EPC
2000’s
2001-
Oregon’s Practitioner-Managed
Prescription Drug Plan (PMPDP)
2004-
AHRQ’s Effective Health Care
Program
OREGON EPC
The Question:
What is the kind and
strength of the evidence
you are relying on to make
a recommendation?
OREGON EPC
What does evidence-based
mean?
• A comprehensive, systematic, open-minded
review of all the evidence
• The evidence determines the conclusion, not
vice versa
• Not, the citation of papers supporting a
preformed conclusion (and trashing of those
that don’t)
• Not, the use of evidence when it is ‘positive’
but judgement when it isn’t
OREGON EPC
Systematic literature reviews
• Are systematic to remove bias in
finding and reviewing the literature.
OREGON EPC
Systematic literature reviews
• Are systematic to remove bias in
finding and reviewing the
literature.
– Experts may interpret the data
(and their own experience)
differently.
OREGON EPC
Systematic literature reviews
• Are systematic to remove bias in
finding and reviewing the
literature.
– Studies with disappointing results
may get less attention
OREGON EPC
Trial Number Groups
RESULTS
114
302
40mg bid
80mg bid
Total improvement at all doses
compared with PLACEBO
115
419
20 mg bid
60 mg bid
100mg bid
Total improvement at all doses
compared with PLACEBO
106
139
20 mg bid,
60mg bid
Borderline improvement at 60
mg dose compared with
PLACEBO
104
153*
20 mg bid, 40
mg bid
20 mg bid, 60
mg bid 80mg
bid
No improvement compared
with placebo at either dose.
Lower relapse rate (31% to
36%) vs. PLACEBO (57%)
303 294
(32
wks)
*Excludes 5 mg bid group
OREGON EPC
Systematic literature reviews
• Are systematic to remove bias in
finding and reviewing the
literature.
– Critical details may be unavailable.
OREGON EPC
Trial 114
OREGON EPC
OREGON EPC
Included Drugs
Clozapine
not posted
risperidone (1993) not posted
olanzapine (1996)
not posted
quetiapine (1997)
not posted
ziprasidone (2001) posted
aripiprazole (2002) posted
OREGON EPC
Systematic literature reviews
• Are systematic to remove bias in
finding and reviewing the
literature.
– Experts may underplay controversy
or select only supportive evidence
OREGON EPC
Simpson et al, 2004
OREGON EPC
Simpson et al, 2004
OREGON EPC
OREGON EPC
In a double-blind study vs risperidone…
GEODON sustained control of
positive symptoms at 1 year
OREGON EPC
Systematic literature reviews
• Are systematic to remove bias in finding and
reviewing the literature.
– Experts may underplay controversy or
select only supportive evidence
• Emphasize the best evidence
OREGON EPC
The best evidence
• addresses health outcomes rather than
intermediate outcomes.
A health outcomes is something a person
can feel or experience
(such evidence is called “direct”)
OREGON EPC
A
Lipid-lowering drugs
Lipid lowering
Angiographic results
B
Heart
Congestive
Strokes
attacks heart failure
C
Mortality
Function
Quality of life
OREGON EPC
The best evidence
• addresses health outcomes and not just
intermediate outcomes
• includes the spectrum of patients to
whom a drug will be prescribed or test
will be needed in, not just highly
selected patients in research studies.
OREGON EPC
The best evidence
• addresses health outcomes and not just
intermediate outcomes
• includes the spectrum of patients to
whom a drug will be prescribed
• considers the potential harms as well as
the benefits of the intervention being
considered.
OREGON EPC
The best evidence
• addresses health outcomes and not just
intermediate outcomes
• is from “real” patients like ours, not just
highly selected patients in studies.
• considers the potential harms as well as the
benefits of the intervention being considered.
• is from well-designed, well-conducted studies.
OREGON EPC
Systematic literature reviews
• Define the strengths and limits of the
evidence.
• Clarify what is based on evidence and what is
based on other grounds.
• Do not necessarily tell you what to do when
the evidence is limited. Other factors, such
as equity, judgment, values, and preferences
play a role in using the evidence.
• In fact, the evidence base is usually
inadequate to inform good decisions.
OREGON EPC
An evidence-based decision
process
Makes use of an independent,
systematic review of the evidence
Employs rules for linking evidence to
recommendations
Produce explicit, defensible
recommendations
OREGON EPC
OREGON EPC
Oregon Approach
What were we after?
• Systematic drug-class reviews should address
questions that reflect clinicians’ and patients’
concerns.
• Decision-makers should begin to wrestle with
the idea of what is good evidence.
• Manufacturers should gain market share if
they produce good evidence of superiority
over other drugs in a class.
• Patients, caregivers, payers should demand
better evidence about outcomes that matter.
OREGON EPC
Oregon Approach
• An evidence-based process, not just
systematic reviews
• a process for selecting and refining
questions that puts providers’ and
patients’ concerns center stage
OREGON EPC
Selecting questions
• Researchers often use their own
curiosity or research interest as the
basis for selecting questions.
• This can introduce bias into a study or a
review.
OREGON EPC
Selecting questions
• Our premise is that important questions
arise from practice, and from life.
“Experts in practice”--and patients--
select the populations, interventions,
and outcome measures of interest.
OREGON EPC
Selecting Questions
• by using citizen panels, our process for
selecting and refining questions puts
providers’ and patients’ concerns center
stage
• the process illustrates how the evidence
people need to make decisions and the
evidence researchers provide is often a
mismatch
OREGON EPC
This process
• Defines the populations, interventions,
outcomes for the systematic review
• Distinguishes health outcomes from
intermediate outcomes
• Identifies what types of studies will be
considered suitable to answer the
questions.
OREGON EPC
OREGON EPC
2000’s
2001-
Oregon’s Practitioner-Managed
Prescription Drug Plan (PMPDP)
2004-
AHRQ’s Effective Health Care
Program
2008-
Knowing What Works
ARRA IOM panel, FCC
OREGON EPC
CER Key Characteristics
1. The objective is to inform decisions
2. Compares at least 2 alternatives, each
with potential to be best practice
3. Analysis at the individual and group levels
4. Measure outcomes important to patients
(both benefits AND harms)
5. Conducted in real world settings
OREGON EPC
Principles
1. Emphasize getting the questions right
2. Start with open-minded inquiry, not
beating others on the head
3. Patient-centered: Anyone can
nominate but formulating the
questions must be broad-based
4. High standards regarding conflicts
5. Collaborate with policy makers but
maintain separate identities
OREGON EPC
Principles
6. Decision-makers should wrestle with
what is good evidence
7. Clinicians should have high standards
for evidence, while taking into account
risk attitude and preferences
8. Market share should be determined by
genuine promise and demonstrated
value
OREGON EPC
Funding CER
• ARRA
• CTSAs
infrastructure
training
• PCORI
infrastructure
training
OREGON EPC
Research Priorities:
Improve methods for
1. Involving patients and the public
2. Evaluating the role of observational
studies
3. Increasing the efficiency of trials
4. Addressing heterogeneity of treatment
effects within studies
5. Incorporating preferences, values, and
individual biological differences into the
design of clinical research studies
OREGON EPC
Identify 2-3 characteristics
• Important to decision makers
– Gaps in current evidence
– Involve the public in selecting or refining
research questions
•
•
•
•
•
Start with viable alternatives
Direct comparison
Benefits and harms
Heterogeneity of treatment effects
OREGON EPC
Recognize and address methods needs
Review
• Be open-minded
• Be clear about the phase or stage of
research
– Efficacyeffectivenessimplementation
– Exploratory?
OREGON EPC
Resources
• JCE articles (Vol 63 Issue 5 (May 2010)
• Methods library
http://www.citeulike.org/user/SRCMethodsLibrary
• Methods mailing list: query to
relevo@ohsu.edu
OREGON EPC
OREGON EPC