Comparative Effectiveness Research
and scientific governance
OR
Wh t can go wrong with
What
ith CER that
th t hasn’t
h
’t gone
wrong already?
Mark Helfand
Dept of Hospital Medicine
Portland VAMC
Di
Director,
O
Oregon EPC
AcademyHealth
June 2010
Existing research…
research
‐‐Doesn’t address the right questions
‐‐emphasizes
h
triviall comparisons and
d
unrepresentative patient groups
‐‐Skimps on outcomes
‐‐Is
I nott always
l
accessible
ibl
Often,
f
then, existing
g research is not a g
good guide
g
for
f
practice because it doesn’t answer what patients,
clinicians and other researchers need to know .
clinicians,
CBO 2007
2
Critique of existing agencies
• NIH – “extensive
extensive experience overseeing
clinical trials, but may not see research on
comparative effectiveness as central to its
mission”
• AHRQ– “substantial expertise in many areas
of comparative
p
effectiveness but has limited
experience managing trials”
• Institute of Medicine “widely respected but
does not have an extensive organizational
capacity to conduct or oversee primary
CBO 2007
research”
3
• might have more influence on doctors and
other health professionals
• might have greater autonomy
• might have a better chance of survival
CBO 2007
4
• might have more influence on doctors and
other health professionals
• might have greater autonomy
• might have a better chance of survival
1978 ‐1981 Nat’l Center for Health Care
Technology.
 1972
1972‐1995
1995 Office of Technology Assessment
 1989‐ ?
AHCPR/AHRQ
CBO 2007
5
independence
1. a dependable source of funding
2. separation from guideline development and
coverage decision making
3 freedom
3.
f d
from
f
political
li i l interference
i
f
4. freedom from conflicts of interest in research
6
Functions of the Governing Board
• Provide guidance to the entity
• establish priorities for its research projects
• create an independent process for reviewing
and approving findings
• Serve as a channel for interested parties to
pa c pa e
participate.
CBO 2007
7
Pitfalls of a top
top‐down
down approach
*
*
*
*
undue influence by one or more sectors
conflicts of interest instead of a public trust
watered down evidence
less latitude and less room for competing
perspectives
p
p
on evidence
• political trading and compromise instead of
adherence
dh
tto principles
i i l off CER
• craziness
CBO 2007
8
Functions of the Methodology
Committee
“Develop methodological standards for
research. Such methodological
g
standards shall
provide specific criteria for internal validity,
generalizability feasibility,
generalizability,
feasibility and timeliness of
research and for health outcomes measures,
risk adjustment,
adjustment and other relevant aspects of
research and assessment with respect to the
design of research.”
9
Rationale for a Methodology
committee
• TTo create
t a “l
“levell playing
l i fi
field”
ld”
• To help
p the p
public identifyy high‐quality
g q
y research
• To promote characteristics of CER
– Patient‐centered
P i
d
– inclusive to avoid overgeneralized method
– Direct comparisons of viable alternatives
– Balance of benefits and harms
– Addresses gaps that underlie uncertainty for decision
makers
10
– Broad view of evidence
Worries about a Methodology
Committee
Requirements
q
too strict
Dismissive of anything but RCTs
Requirements too lax
Di i i off RCT
Dismissive
RCTs
Unresponsive
p
to innovation
11
Status quo
• The literature misses important questions,
patients, comparisons, outcomes
• Researchers have a stake in certain methods
• We can’t trust that what is published is true
– You can get any answer you want
– You can hide or distort the answers you don’t
don t like
• Editors may be fooled or may not be able to
provide
id the
h needed
d d llevell off scrutiny
i
12
Types of Reporting bias
• Deliberately
lib
l preventing
i or delaying
d l i publication
bli i or
public disclosure
• Selective outcome reporting
• Selective analysis
y
• Selective pooling bias
• Ghost and guest authorship
• Reframing and spin
• “Publication
“ bli i strategy””
• Subverting the peer review process
Simpson et al, 2004
Simpson et al, 2004
Pitfalls of Current Research
Eff t
Efforts
Industry
NIH
Economic
Marketing
Career
AHRQ
Foundations
VA
Infrastructure
Norms
Good Research Governance
• Clear rules
– Responsiveness
p
with separate
p
identities and
Independence
– (Whoever asks the question
question, invite broad
participation in refining it)
– Freedom from conflicts of interest
– Responsive to criticism
• Innovation and methodological development,
not just following rules
17
Good Research Governance
• Deliberation about what is good evidence,
i l i scientists,
involving
i i
policymakers,
li
k
and
d the
h
public
• considering preferences and uncertainties
underlying decision
decision‐making
making
• Commitment to research information as a
public good
• Change the marketplace so that industry can
benefit from conducting CER
18
Steps for improving (all) studies
• Publish the protocol (all of them)
• Authority over what data are collected
• Because they are not randomized,
randomized
observational studies need better, not weaker,
prote tions aagainst
protections
ainst bias
and
T k a skeptical
Take
k ti l scientific
i tifi perspective
ti
19
Types of CER
•
•
•
•
•
Systematic Reviews
Analysis of claims records and EHRs
Registries
RCTs
Modeling
CBO 2007
20