Infusion of Statistical Science Into
Comparative Effectiveness Research:
What, Why, How, Who?
Sally C. Morton
Statistics and Epidemiology, RTI International
www.rti.org
RTI International is a trade name of Research Triangle Institute
What is
Comparative Effectiveness Research (CER)?
“The generation and synthesis of evidence that
compares the benefits and harms of alternative
methods to prevent, diagnose, treat, and monitor
a clinical condition or to improve the delivery of
care. The purpose of CER is to assist consumers,
clinicians, purchasers, and policy makers to make
informed decisions that will improve health care at
both the individual and population levels.”
Institute of Medicine Committee on Comparative Effectiveness Research
Prioritization (2009)
www.rti.org
4/27/10
2
Other Names for CER: “Clinical Effectiveness Research”
or “Patient-Centered Research”
•
•
•
•
•
“Which treatment works best, for whom, and under what
circumstances?” (Slutsky and Clancy. Am J Med Qual
2009;24:67-70)
Informs a clinical decision (screening, diagnosis, treatment)
Focuses on effectiveness not efficacy
Compares at least 2 alternatives, one may be “usual care”
Measures outcomes that matter to patients, including harms
and benefits, at both the population and subgroup levels
Includes generation and analysis of new evidence;
secondary data analysis; and synthesis
Institute of Medicine Committee on Comparative Effectiveness Research Prioritization (2009)
www.rti.org
4/27/10
3
Why CER Now?
•
•
•
•
Focus on healthcare reform
Clinical practice often not based on evidence
Coverage decisions often unrelated to effectiveness
Large and escalating volume of information
– Synthesis required
– Relevance to clinical practice unclear
• CER expected to increase quality, effectiveness, and
efficiency by
– Summarizing existing knowledge
– Identifying knowledge gaps
4
www.rti.org
4/27/10
It’s Where the Money Is
Dramatic increases in funding at AHRQ for CER
• $ 30 million in FY07
• $ 50 million in FY08, now $88 million
ARRA (Stimulus Bill) targeted for CER
• $ 1.1 billion in FY09 and FY10
– $300 million to AHRQ
– $400 million to NIH
– $400 million to HHS Office of the Secretary
• Institute of Medicine (IOM) Committee to
recommend priorities for CER
5
www.rti.org
4/27/10
How Do Statistical Methods Contribute To
Comparative Effectiveness Research?
Prioritization of gaps
in evidence generation,
synthesis,
communication, or
translation
Evidence
generation:
prospective and
restrospective
analysis of
secondary
databases such
as medical
registries and
claims data
Evidence
generation:
primary data
collection via
clinical trials
and
observational
studies
Evidence
synthesis:
costeffectiveness
analysis and
decision
modeling
Evidence
communication:
dissemination of
findings
Evidence
synthesis:
systematic
reviews and
meta-analyses
Evidence
translation:
generation of
information for
different decisionmakers, e.g.,
clinical practice
guidelines
Challenge: Statistical methods that are scientifically rigorous,
relevant (patient-centered), and timely
www.rti.org
4/27/10
6
Factors That Enter Into Clinical Decisions
Mulrow et. al. Ann Intern Med 1997;126:389-391
www.rti.org
4/27/10
7
Continuum From Research Studies to
Clinical Practice Guidelines
8
www.rti.org
4/27/10
Focus on Evidence Generation
Via Primary Data Collection Studies
• RCTS were recommended for 49 of the top 100 CER
research priorities identified by the IOM
• “The paradox of the clinical trial is that it is the best
way to assess whether an intervention works, but
arguably the worst way to assess who will benefit
from it.” (Mant. Lancet 1999;353:743-46)
• Knowing a treatment is efficacious is necessary but
not sufficient for knowing it is effective
9
www.rti.org
4/27/10
Distinguishing Efficacy and Effectiveness
Study
Elements
Efficacy Studies
Effectiveness Studies
Study design
Protocol driven; visits
conducted at regular intervals;
medication often masked
Clinician visits driven by patient needs and
physician practice; treatment usually
unmasked, allows modification of treatment
Patient
population
Homogeneous; highly selected
for patient characteristics,
adherence, no comorbidities
Heterogeneous; more representative of
patients in the real world
Study sites
Research specialists, academic General practice physicians; “usual”
medical centers
caregivers, facilities, settings
Sample size
Sufficient only to detect trial
outcome; often not adequate
for subgroup analyses
Adequate to assessing minimally important
difference from patient perspective,
sometimes for subgroups as well
Study
endpoints
Often intermediate; primary
endpoint is often a physiologic
endpoint (surrogate)
May include patient- reported outcomes;
clinically relevant outcomes
www.rti.org
4/27/10
10
Distinguishing Efficacy and Effectiveness
(Cont)
Study
Elements
Efficacy Studies
Effectiveness Studies
Duration
Often short —several weeks Often longer to parallel use in practice
to months
and to allow for nonadherence
Comparator
Usually placebo (for pivotal
efficacy trials for FDA
approval)
Relevant clinical comparator or “usual
care”
Data
Collection
Stops with event or study
discontinuation; data
collected as part of
treatment visits
Captures data even after event or
study discontinuation; data collected as
part of routine care
Analysis
Typically intention to treat
Various, depending on study design
External
validity
Limited applicability
Broader applicability
11
www.rti.org
4/27/10
How Do We Expand CER Beyond
Explanatory (Ideal Setting) RCTs
How can we improve clinical trial organizational
efficiency, analytic efficiency, and usability for decisionmakers? (Luce et al. Ann Intern Med 2009;151:206-209)
• Pragmatic trials (real-world settings)
• Cluster randomized trials
• Bayesian adaptive designs
How can we improve methods to control for confounding
in observational studies?
• Propensity scores
• Instrumental variables
12
www.rti.org
4/27/10
Focus on Evidence Synthesis:
Unadjusted Indirect Comparison
How does one
compare
A vs. B
when no direct
(head to head)
evidence is
available?
A
P
B
P
A
P
B
P
A
P
B
P
A
P
B
C
A
D
B
C
A
D
B
D
A
E
B
E
A
F
B
F
13
www.rti.org
4/27/10
Adjusted Indirect Comparison
In this approach,
the comparison is
adjusted by a
common
comparison
group
A
P
B
P
A
P
B
P
A
P
B
P
A
P
B
C
A
D
B
C
A
D
B
D
A
E
B
E
A
F
B
F
Alternative approach: network analysis
14
www.rti.org
4/27/10
Overarching Principle: “Fit For Purpose”
“Decisions about the use of therapeutic interventions,
whether for individuals or entire healthcare systems,
should be based on the totality of available evidence.
The notion that evidence can be reliably or usefully
placed in ‘hierachies’ is illusory. Rather, decision
makers need to exercise judgement about whether
(and when) evidence gathered from experimental or
observational sources is fit for purpose.”
Rawlins. Clin Med 2008;8:579-88.
15
www.rti.org
4/27/10
Questions to Ponder
• How can RCTs be designed to most benefit CER?
• When and how can observational studies complement
experimental studies to answer CER questions?
RCTs ≡ efficacy
Observational Studies ≡ effectiveness
16
www.rti.org
4/27/10
Questions to Ponder
• What methods for observational studies adequately
address confounding?
• What meta-analytic methods for indirect evidence will
inform CER questions?
• Whose data are they?
• What are the statistical methods research priorities?
Priority list: A product of this conference
17
www.rti.org
4/27/10
Comparative Effectiveness Research
• (Sort of) new wine
– Emphasis is driven by technology availability, payer
interest, rising chronic disease burden
• New bottle
– Federal and payer interest high in the next few years
– Critical to be successful early
• I encourage the statistical community to have a glass!
Adapted from lecture by Tim Carey on CER
www.rti.org
4/27/10
18