Comparative
Effectiveness Research
Shalini Kulasingam, PhD
University of Minnesota
Overview







Learning objectives
Background: why do we need a “special type” of research
agenda?
Definition: comparative effectiveness research
What areas/conditions have been prioritized for study?
Role of nursing?
Methods for conducting comparative effectiveness research
Examples




RCTs
Observational studies
Simulation modeling
Funding
Learning Objectives




Why there is a need for CER?
Priority CER topics
Study designs for conducting CER
Examples of CER studies
Background


Health care expenditures were $2.4 trillion in 2008 and
are projected to grow by an average of 6.2 percent per
year for the next 10 years, more than triple the
projected rate of overall gross domestic product (GDP)
growth (Sisko et al., 2009)
The Congressional Budget Office (CBO) projects that
under current law, health care will consume more than
30 percent of GDP by 2035 (CBO, 2008).
IOM report, 2009
Background

Regional variations in treatment patterns and
cost growth provide deeper insight into the need
for more informed medical decision making.

Patients in the highest-spending regions of the
country receive 60 percent more health services than
those in the lowest-spending regions, yet this
additional care is not associated with improved
outcomes (Fisher et al., 2003).
Background
Research suggests that physicians in
higher-spending areas are more likely than physicians in
other regions to recommend costly interventions that
have not been definitively shown to be effective (Fisher et
al., 2009).

Nationwide, the Institute of Medicine (IOM) has
estimated that less than half of all treatments delivered
today are supported by evidence (IOM, 2007).

Background

A recent review of practice guidelines developed
by the American College of Cardiology and the
American Heart Association found that
relatively few recommendations were based on
high-quality evidence—randomized controlled
trials, for instance—and many were based solely
on expert opinion, individual case studies, or
standard of care (Tricoci et al., 2009).
What is comparative effectiveness
research?

Comparative effectiveness research (CER) is the
generation and synthesis of evidence that compares the
benefits and harms of alternative methods to prevent,
diagnose, treat, and monitor a clinical condition or to
improve the delivery of care. The purpose of CER is to
assist consumers, clinicians, purchasers, and policy
makers to make informed decisions that will improve
health care at both the individual and population levels.
CER Summary and Investment in
Research




How good is the intervention/treatment/test?
In what patients?
Under what circumstances?
American Recovery and Reinvestment Act of
2009
$1.1 billion “down payment” to support CER
 $400 million given to the NIH
 $300 million given to the AHRQ
 $400 million to Health and Human Services

What are the priority areas for
research?

The American Recovery and Reinvestment Act of
2009 called on the Institute of Medicine to
recommend a list of priority topics to be the
initial focus of a new national investment in
comparative effectiveness research. The IOM’s
recommendations are contained in the report,
Initial National Priorities for Comparative Effectiveness
Research.
What are the priority areas for
research?






Treatment strategies for atrial fibrillation
Treatment for hearing loss
Primary prevention versus clinical treatments in
preventing falls in older adults
Biologics for inflammatory diseases
Upper endoscopy for patients with
gastroesophageal reflux disease
Dissemination and translation of techniques for
use of CER by clinicians
IOM, 2009
What are the priority areas for
research?






Comprehensive care programs for people with chronic
disease
Interventions for MRSA
Strategies to reduce health care associated infections
Management of prostate cancer
Registry for lower back pain
Detection and management of dementia in a
community setting
IOM, 2009
What are the priority areas for
research?






Management of behavioral disorders associated with
dementia
School-based interventions for treating obesity in
children
Interventions to reduce hypertension, obesity etc. in
urban poor and Native American populations
Management strategies for ductal carcinoma in-situ
Use of imaging technologies for cancer
Genetic and biomarkers for cancer
IOM, 2009
What are the priority areas for
research?







Prevention of dental caries in children
Treatment strategies for children with ADHD
Management of serious emotional conditions in
children and adults
Interventions to reduce health disparities
Literacy sensitive disease management
Interventions to reduce adverse birth outcomes in
women especially African American women
Prevention of unintended pregnancies
IOM, 2009
Role of Nursing

Statement by MaryJean Schumann, Chief Program
Officer, ANA, 2009
Perspective is based on two types of nurses
 the registered nurse providing direct care
 the advanced practice registered nurse




Certified Registered Nurse-Anesthetists (CRNAs) who provide
critical anesthesia services;
Clinical Nurse Specialists (CNSs) who provide acute care expertise
for complex patients;
Certified Nurse-Midwives (CNMs) who provide health care to
women across the lifespan;
Nurse Practitioners (NPs) who deliver a wide range of primary care
services.
Role of Nursing

“Nursing’s holistic view – attention to the whole
person – makes nurses particularly effective in
advancing these priorities. Nurses, with their
expertise in health promotion, disease
prevention, and health literacy, can contribute to
changing the current sickness care system into a
true health care system.”
MaryJean Schumann,, ANA,. 2009
Setting priorities based on ANArelated work and data


National Quality Forum National Priorities and Goals
Six Priority areas:






What are the most effective tools and systems to engage
patients in their care?
What are the most effective models for care coordination?
How do we reduce 30-day readmission rates?
How is palliative care best provided?
“How do we eliminate unnecessary or risky care?
“Improve health by ensuring that patients receive the most
effective preventive services recommended by the U.S.
Preventive Services Task Force. “
MaryJean Schumann,, ANA,. 2009
Setting priorities based on ANA
related work and data
CER priorities based on quality indicators developed by
ANA
 1998, ANA established the National Database of
Nursing Quality Indicators® (NDNQI®), the only
national database that provides nursing data and patient
outcomes at the unit level where care occurs.
 Data are collected on structure, process and outcome
measures in approximately 1400 hospitals of all sizes, in
all 50 states and the District of Columbia.
 Data is collected on 17 measures, 11 of which have
been endorsed by the National Quality Forum.
MaryJean Schumann,, ANA,. 2009
Area NOT recommended for further
research

A Cochrane review concluded that “appropriately
trained nurses can produce as high quality care as
primary care doctors and achieve as good health
outcomes for patients.” It was noted that the research
available is limited and some may call for further
comparative studies. There are, however, no other
professionals who have been subjected to the depth of
study that NPs and CNMs have, and we question the
need to expend limited resources on additional studies
comparing professional groups, though we stand ready
to play a role in the design and conduct of such studies
should they be deemed necessary.
MaryJean Schumann, ANA. 2009
Study Designs for CER



Synthesis of existing data
Analysis of observational data
Randomized controlled trials
Study Design

Synthesis of existing data
Systematic review
 Meta-analysis
 Decision modeling

Study Design

Observational data
Administrative claims
 Electronic medical records
 Registries
 Case control or cohort studies

Study Design

Randomized controlled trial
Luce et al. Annals Internal Medicine, 2009
 How to change RCTs for comparative effectiveness
research
 Analytic and operational efficiency

Reduce costs of running a trial, and be able to up date
trials on an ongoing basis, dropping
tests/drugs/interventions that are not promising
 Accomplish this using Bayesian approaches


Pragmatic clinical trial
Study Design

Pragmatic RCT







CER objective is to provide information to help patients,
consumers, clinicians, and payers make informed decisions.
Trials tend to exclude relevant patient populations, commonly
used comparators, long term outcomes, and non-expert
providers
Clinically effective comparators
Study patients with common co-morbid conditions
Diverse study patients
Providers from community settings
Provider and patient chosen outcomes
Potential Sequence for Identifying
and Proposing CER



IOM list of priority topics lists those that are
most likely to get funded
IOM report notes that systematic reviews and
meta-analyses provide information on areas for
further study.
Question: can you use take a topic from the
IOM priority list and identify a study for grant
purposes?
Examples of CER Studies

Detection of dementia in a community setting
Systematic review
 Decision modeling
 Pragmatic trial


Patient falls
Systematic review
 Patient record review
 Randomized controlled trial – with pragmatic
aspects

Detection of dementia in
a community setting
The original CMS-sponsored TA



Aim: Assess the benefits of FDG-PET scanning
in patients with dementia, with mild cognitive
impairment (MCI) and in asymptomatic patients
with a family history of AD, subsequent to the
standard evaluation as described in the American
Academy of Neurology (AAN) guidelines.
CMS requested that the AHRQ identify an
Evidence Practice Center to perform a
Technology Assessment (TA)
Duke EPC assigned the TA in 2001
Methods of the original TA


Literature review
Decision model to provide an understanding of
the decisional context
http://www.cms.hhs.gov/coverage/download/id64.pdf
Direct inference
Delayed
progression
Test
Decreased
mortality
Indirect inference: causal pathway
True -
Test
False -
True +
Treat
False +
Adverse
Event
Delayed
progression
Decreased
mortality
Model: Part 1
Truepos
AD
sens
prevAD
PET
Falseneg
AD, treat
#
Falsepos
NoAD
#
AD, no treat
#
Trueneg
No AD, treat
spec
AD
Mild dementia
No PET/Rx
Truepos
No AD, no treat
Falsepos
AD, treat
Falseneg
No AD, treat
Trueneg
AD, no treat
prevAD
NoAD
#
AD
No PET/NoRx
prevAD
NoAD
#
No AD, no treat
Test performance
No AD by
clinical
evaluation
Test +
AD by
clinical
evaluation
True +
Test -
False -
True -
False +
Sensitivity = True +/AD
Specificity = True - /No AD
ROC curve of PET test accuracy based on the literature review
Tree results: mild dementia
True +
(%)
False +
(%)
False –
(%)
True –
(%)
Correct
(%)
No
PET/
Rx
56
44
0
0
56
PET/
Rx+
49
6
7
38
87
No
PET/ no
Rx
0
0
56
44
44
Asymptomatic
MCI
Model: Part 2
Dead
Mild
Dementia
Moderate
Dementia
Severe
Dementia
Illustrative patient history
Year
1
Asy
MCI
MiD
MoD
SeD
D
2
Asy
MCI
MiD
MoD
SeD
D
3
Asy
MCI
MiD
MoD
SeD
D
4
Asy
MCI
MiD
MoD
SeD
D
Markov results: mild dementia
QALY
LE
SDFLE
No PET/ Rx
4.10
7.89
4.02
PET/
Rx+
No PET/ no
Rx
4.09
7.88
4.00
4.02
7.82
3.86
Primary conclusion


PET could improve the overall accuracy
compared to accuracy of an exam based on
AAN guidelines.
Treatment based on an AAN-recommended
examination leads to better health outcomes
than treatment based on PET results
How can this make sense?


While net accuracy with PET may be better, this is
because there are many fewer false positives but a few
more false negatives
Incorrectly not treating (due to a false negative result a
patient misses an opportunity for a Rx benefit) is worse
than incorrectly treating (the patient unnecessarily
receives medication, however the Rx is relatively benign,
may be beneficial even if they don’t have AD, and the
personal downside is that their cognitive
impairment/disability is not correctly labeled)
When testing is preferred
1. If a new treatment becomes available that is not
only more effective than AChEIs but is also
associated with a risk of severe adverse effects.
When testing is preferred
2.If testing could be demonstrated to be a better
reference standard than an examination based
on AAN guidelines. (i.e., testing would need to
better distinguish patients who respond to
therapy than is possible with a standard
examination.)
When testing is preferred
3.If the results have demonstrable benefits beyond
informing AChEI use.

This “value of knowing” could have both positive
and negative components.
A research agenda in service to
decision-making
Designs:

Minimize bias

Maximize generalizability

Why not a trial?

In particular, why not a pragmatic clinical trial?
Kulasingam et al. Am J Alzheimers Dis Other Demen. 2006
Design of a pragmatic clinical trial
Matched Communities
R
PET
reimbursed
Patients
identified,
1 page evaluation
completed*
PET not
reimbursed
Medicare claims (primary outcome = resource use from index date
to 3 months)
* Medicare claims at the community-level for
individuals with relevant ICD codes will be examined (see
Methods)
Design of a pragmatic clinical trial






Design: A demonstration project in which matched
communities are randomly assigned to have FDG-PET
reimbursed by Medicare or not.
Allocation: Communities are allocated to intervention or control
by concealed randomization.
Blinding: Blinded outcome assessors/data collectors,
biostatisticians.
Follow-up period: 3 years
Setting: Communities in which state-of-the-science FDG-PET
is reasonably available for various Medicare-covered clinical
applications.
Patients: Patients will be enrolled based on (a) age ≥ 65, (b)
free-living, (c) presenting without prior specific workup for a
complaint of memory deficit, and (d) the physician specifies that
some degree of workup is planned.
Design of a pragmatic clinical trial

Intervention: All participating communities will have a general
education program regarding the diagnosis and evaluation of
cognitive impairment, and will be informed how to enroll
patients into the study. To ensure comparable patient
identification in all communities, providers will be compensated
on completion of a basic evaluation form for an eligible patient.
Communities randomized to have FDG-PET reimbursed will
have payment coordinated by the regional Medicare carrier.
Communities randomized to not have FDG-PET reimbursed by
Medicare will not have restrictions on FDG-PET if covered
under other payment arrangements.
Design of a pragmatic clinical trial

Measures: Measures will consist of (a) a simple (i.e., 1 page)
form completed on the date of presentation (index date) by the
patient’s provider regarding basic demographic and clinical
features, diagnosis/further diagnostic plan, treatment plan, and
prognosis; and (b) resource utilization related to initial evaluation
and management of individuals with CI, as assessed via linked
Medicare claims files. Cumulative resource costs from the index
date to three-months (short-term) will serve as the primary
outcome measure for purposes of sample size calculation.
Additional measures will include resource counts (e.g., imaging
studies, specialty referrals, laboratory testing, and so on), FDGPET diffusion (in terms of proportion of candidates who have
had a FDG-PET), as well as trajectory of resource use over time.
Review of Steps



Priority topic
Literature review/decision modeling to identify
areas for further research
Proposed pragmatic clinical trial
Patient Falls
Potential Areas of Research

“Patient falls are one example of how our quality work
informs a CER priority. As an outcome of interest, falls
are of critical importance, highlighted by CMS’ decision
to include falls on the list of Hospital Acquired
Conditions for which they no longer pay. There are
many validated fall assessment tools, but there has not,
to date, been any comparative research on the tools to
determine which is more effective in determining fall
risk assessment and which interventions are most
effective for preventing falls.”
MaryJane Schumann, ANA, 2009
Falls





Coussement J et al. American Geriatrics Society, 2008
Interventions for Preventing Falls in Acute- and Chronic-Care
Hospitals: A Systematic Review and Meta-Analysis
Goal: To determine the characteristics and the effectiveness of
hospital fall prevention programs
Results: Review showed that most studies were conducted on
long-stay (mean length of stay (LOS) >1.5 years) and
rehabilitation units (mean LOS 36.9 days).
Results: For analysis of the number of falls, one unifactorial and
two multifactorial studies showed a significant reduction of 30%
to 49% in the intervention group, with the greatest effect
obtained in the unifactorial study that assessed a pharmacological
intervention.
Falls





Lakatos BE et al. Psychosomatics, 2009
Objective: to determine the prevalence of diagnosed and
undiagnosed delirium in patients who fell during their hospital
stay.
Study design: Retrospective chart review
Methods: Falls were categorized by their severity (i.e., minor,
moderate, and major). Demographic information, patient
outcomes, and diagnostic criteria for delirium (per DSM–IV)
were collected on the day of admission, the day of the fall, and
the 2 days preceding the patient’s fall
Results: Falls in the general hospital were associated with
delirium (both diagnosed and undiagnosed), advanced age, and
specific surgical procedures
Falls




Vass et al. Reducing Falls in In-patient Elderly, Trials, 2009
Summary of an RCT that aims to reduce falls in an elderly inpatient population in an acute care setting.
Background: More than half of all in-patient falls in elderly
people in acute care settings occur at the bedside, during
transfers or whilst getting up to go to the toilet. In the majority
of cases these falls are un-witnessed.
Background: New patient monitoring technologies have the
potential to offer advances in fall prevention. Bedside sensor
equipment can alert staff, not in the immediate vicinity, to a
potential problem and avert a fall. However no studies utilizing
this assistive technology have demonstrated a significant
reduction in falls rates in a randomized controlled trial setting.
RCT for fall prevention


The research design is an individual patient randomized
controlled trial of bedside chair and bed pressure
sensors, incorporating a radio-paging alerting mode to
alert staff to patients rising from their bed or chair,
across five acute elderly care wards in Nottingham
University Hospitals NHS Trust.
Participants will be randomized to bedside chair and
bed sensors or to usual care (without the use of
sensors). The primary outcome is the number of
bedside inpatient falls.
RCT – proposed data collection
Data collection
Baseline
Discharge
Demographics
X
Previous falls/med. Hx.
X
Cognitive Ability
X
X
Quality of Life
X
X
Activities of Daily Living X
X
Discharge Destination
X
X
Length of stay
X
X
Fear of falling
questionnaire
X
X
Total # of in-patient falls X
X
Vass et al. Trials, 2009
Review of Steps



Priority topic
Meta-analysis and chart review study to identify
gaps
Proposed clinical trial
Pragmatic aspects are community setting
 Range of outcomes

Conclusions





CER is new focus of funding at the NIH and AHRQ
List of priority topics (IOM)
Evidence reports and meta-analyses can provide
information on gaps in knowledge base that require
further study (AHRQ)
Search grants.gov for RFAs or other announcements re:
new funding opportunities
Lots of potential colleagues/collaborators at the U MN



School of Public Health
School of Medicine
School of Dentistry
Funding and Acknowledgements



Shalini Kulasingam is supported by NCI grant K07-CA113773
Previously funded by:
 Grants: Merck, CSL-Australia, SP-MSD, CDC, NIH, mtm
 Consultant: SP-MSD, CSL – New Zealand, Medtronic
Collaborators
 Evan Myers, Duke University
 George Sawaya, University of California, San Francisco
 Joy Melnikow, University of California, Davis
 Mark Schiffman, Philip Castle, NCI
 Eduardo Franco, Raghu Rajan, McGill University
 Laura Koutsky and Akhila Balasubramanian, University of
Washington
 Patti Gravitt, Johns Hopkins University
 Levi Downs, Rahel Ghebre, Ruby Nguyen, Karen Kuntz,
University of Minnesota