Patient-Centered Outcomes
Research Institute
What is it and what does it mean for health
services researchers?
Patrick S. Romano, MD MPH
UC Davis Center for Healthcare Policy and Research
March 28, 2012
Speaker’s Verbal Disclosure Statement:
Have you (or your spouse/partner) had a personal financial relationship
in the last 12 months with the manufacturer of the products or services
that will be discussed in this CME activity?
_x_ Yes
___ No
(If yes, please state disclosures and resolutions)
I received a $200 honorarium and reimbursement of travel
expenses for participating in a PCORI grant review meeting.
Educational objectives for this seminar:
Please type between two and four educational objectives here.
To describe the origin and mission of PCORI
To differentiate Patient Centered Outcomes Research from
Comparative Effectiveness Research (if possible?)
To explain the PCORI grant review process and discuss future
opportunities for UC Davis/CHPR research support
What is CER?
• Comparative effectiveness research is the conduct and
synthesis of systematic research comparing different
interventions and strategies to prevent, diagnose, treat
and monitor health conditions. The purpose of this
research is to inform patients, providers, and decisionmakers, responding to their expressed needs, about
which interventions are most effective for which
patients under specific circumstances. To provide this
information, comparative effectiveness research must
assess a comprehensive array of health-related
outcomes for diverse patient populations…
– Federal Coordinating Council for Comparative Effectiveness
Research, definition; [cited 28 July 2010].
What is PCORI?
• Prior to several years ago, no coordinated effort for
comparative effectiveness research (CER)
• 2009 stimulus package provided $1.1 billion for
CER (most went to AHRQ and NIH)
• 2010 Affordable Care Act created Patient Centered
Outcomes Research Institute
• Funded by tax on government and private insurers
for each covered life (~$500 million per year)
What is PCORI?
• Independent, nonprofit, quasi-governmental organization
– AHRQ almost eliminated in 1990s after published guideline
that surgery for back pain was unnecessary
– Provide political protection
– PCORI funding does not depend on annual Congressional
appropriation
• No infrastructure for determining funding priorities,
issuing opportunities, reviewing and administering grants
– Outsourcing (NIH, AHRQ, CMS)
• Peer review, contract management
• May receive $ from PCORI to originate grants
– PCORI will be its own entity
1. Identifying national priorities for research.
2. Creating a research agenda based on identified priorities.
3. Funding research consistent with these priorities and agenda.
4. Providing patients and their caregivers with useful research
information.
PCORI Board of Directors
• Director of AHRQ (Carolyn Clancy)
• Director of NIH (Francis Collins)
• 17 members appointed by the Comptroller General of the
United States as follows:
– 3 members representing patients and health care consumers
– 5 members representing physicians and providers, including at least 1
surgeon, nurse, State-licensed integrative practitioner, and hospital
– 3 members representing private payers, including at least 1 health
insurance issuer and at least 1 self-insured employer
– 3 members representing pharmaceutical, device, and diagnostic
manufacturers or developers
– 1 member representing quality improvement or independent health
service researchers (Harlan Krumholz or Arnold Epstein)
– 2 members representing the Federal Government or the States,
including at least 1 representing a Federal health program or agency
PCORI Leadership
• Eugene Washington, MD, MSc, (chair), Vice
Chancellor of UCLA Health Sciences and Dean
of the David Geffen School of Medicine
• Steven Lipstein, MHA (vice chair), President
and Chief Executive Officer of BJC HealthCare
• Joe Selby, MD, MPH
Executive Director
• Anne C. Beal, MD, MPH
Chief Operating Officer, abeal@pcori.org
PCORI Mission
PCORI helps people make informed health care
decisions and improves health care delivery and
outcomes by producing and promoting high
integrity, evidence-based information that comes
from research guided by patients, caregivers, and
the broader health care community.
PCOR vs. CER
• Why PCORI, not CERI?
• Political concerns that determining which treatments
better → “rationing”
– Restrict patients’ ability to choose inferior treatment
• Compromise focus on patient-centered outcomes
research
• Legislation barred PCORI from
– Developing or using QALYs
– Considering cost in making coverage recommendations
– In practice, PCORI will not fund cost-effectiveness analyses
Patient-Centered Outcomes Research
• Not just a semantic difference
• Focus on PCOR essential role in shaping funding
priorities and “rules of the game” for getting funding
• Can’t just replace placebo control with active
comparison group, do everything else the same
• Distinct elements
–
–
–
–
Outcomes should be patient-centered
Effectiveness, not efficacy
Heterogeneity of treatment effects
Involvement of stakeholders
Patient-Centered Outcomes Research
• Outcomes should be patient-centered
– “…outcomes that people notice and care about
such as survival, function, symptoms, and healthrelated quality of life”
– Not biomarkers (BP, serum cholesterol, etc.)
Patient-Centered Outcomes Research
• Effectiveness
– Research done in clinically relevant populations
– Representative of patients
– Representative of settings
• Heterogeneity of treatment effects
– “Given my personal characteristics, conditions and
preferences, what should I expect will happen to me?”
– “What are my options and what are the benefits and
harms of those options?”
Role of stakeholders
• Government payors and regulators; health plans;
hospitals and other care providers; industry
• Consider how results will shape policy and practice
through dissemination
• Stakeholders should be intimately involved in the study
design process to identify important outcomes
– Trials of anti-seizure drugs in children
– Traditional measure – seizure frequency
– Patient-centered measure – parent want to know how
drug will affect school performance
– Drug ↓ seizures but ↑ sedation may not be preferable
Implications for research design
• RCTs
– Pragmatic - real-world settings; large N’s to assess subgroups
– Focus on effect size, not just superiority
– Bayesian and adaptive approaches to design, accounting for
prior probabilities
• Observational studies
– Causal inference modeling
– Tension for studies using pure administrative data –
powerful and low-cost; lack high-quality patient-centric data
• Cost and cost-effectiveness
– PCORI not allowed to evaluate cost in making coverage
recommendations, won’t fund “pure” CEA
– Cost can be studied and used by decision-makers
Proposed National Priorities
• Assessment of Prevention, Diagnosis, and Treatment Options. The
research goal is to determine which option(s) work best for distinct
populations with specific health problems.
• Improving Healthcare Systems. Focuses on ways to improve healthcare
services, such as the coordination of care for patients with multiple
chronic conditions.
• Communication and Dissemination. Looks at ways to provide information
to patients so that they, in turn, can make informed healthcare decisions
with clinicians.
• Addressing Disparities. Assures that research addresses the healthcare
needs of all patient populations. This is needed as treatments may not
work equally well for everyone.
• Accelerating Patient-Centered and Methodological Research. Includes
patients and caregivers in the design of research that is quick, safe, and
efficient.
PCORI Funding Opportunities
• Developmental
• RFPs (Sept 2011)
• ~$30-$100K each
• Methods for setting research priorities
• Pilot Projects Grant Program (Dec 2011)
•
•
•
•
$13 million in year 1 for ~40 projects
Help develop national research priorities for PCOR
Develop ways to engage patients and bring stakeholders together
Develop measurement tools and data collection methods for novel
patient-centered outcomes
PCORI Funding Opportunities
• $150 million in current FY; ultimately perhaps $500
million/year
• Mechanisms unknown, priorities under discussion
– Likely NOT “traditional” research
– Patient-centered outcomes
• Including developing methods
– Real-world populations
– Heterogeneity of treatment effects
– Involvement of stakeholders
Other Funding Opportunities
• Existence of PCORI does not preclude government agencies or
foundations from funding CER
• AHRQ has Effective Healthcare Program
– Large evidence-synthesis and dissemination projects
– EBPCs, DEcIDE network, CERTS
– Does not support investigator-initiated projects
• Areas of CER explicitly outside PCORI’s mandate
– CER with “traditional” outcomes
– Cost-effectiveness analyses
• Remains to be seen
PCORI Pilot Research Projects
• $13 million in 2012
• Approximately 40 awards
• 2 years maximum support
– 2nd year of funding must be well justified and will be
evaluated for successful completion of the first year goals
• $250,000 per year (plus indirect costs)
– Travel to annual PCORI meeting is an allowable expense
• Requires stakeholder involvement as co-investigators
– unless not feasible as for an application on analytic
methods
Principles of grant review
•
•
•
•
•
Scientific and technical merit
Fair, competent and thorough
Conflict of interest
Confidentiality
Difference between overall impact and significance
– Overall impact: Probability of whether the research will
exert a sustained, powerful influence on the research field.
– Significance: Does the project address an important
problem or a critical barrier to progress in the field? If the
aims are achieved, how will scientific knowledge, technical
capability, and/or clinical practice be improved?
Structure of application
• Signed Face Page, Table of Contents
• Project Summary: single paragraph using plain language to summarize the
– Broad objectives
– Specific aims
– Relevance to PCORI Objectives
– Research Design and methods for achieving the goals
• Project/Performance Site(s)
• Key Personnel and Other Significant Contributors
• Biographical Sketches–Principal Investigator, Other Biographical Sketches
– Personal statement allows each person to explain their role and goals for the project
• Budget and Justification
• Resources
– Provide information on the facilities to be used in the proposed research project
– Describe how the research environment contributes to the probability of success
– Describe how project will benefit unique features of the research environment or
community involvement or allow for useful collaborative arrangements
– Describe institutional and community investment in the success of the research
• Checklist
• Addendum Form: Checklist to identify PCORI Areas of Interest
Research Plan
•
•
•
•
•
•
Specific Aims (1 page)
Research Strategy (10 pages)
References Cited
Protection of Human Subjects
Consortium/Contractual
Letters of Support
• No appendices are allowed and will be removed by
PCORI before sending to reviewers
PCORI Review Criteria
•
•
•
•
•
•
Significance (Research Plan)
Patient/Stakeholder Engagement (Research Plan)
Investigator (Biosketch)
Innovation (Research Plan)
Approach (Research Plan)
Environment (Project/ Performance Sites)
PCORI Review Criteria Significance
• Does the project address an important problem or
critical barrier to PCOR?
• Will the result produce new knowledge that can
advance PCOR methods or infrastructure?
• Does the investigator demonstrate thorough
knowledge of previous and ongoing work related to
their proposed topic?
PCORI Review Criteria Patient/Stakeholder Engagement
• Will the project make a unique contribution to
learning about engagement of patients and
stakeholders?
• Does the research team demonstrate authentic,
feasible partnerships with the patients/stakeholders?
• Is there evidence that the stakeholders were involved
in the preparation of the research proposal?
PCORI Review Criteria Investigator
• Is the research team well suited to the project?
• Is there appropriate scientific expertise?
• Does at least one member of research team have
experience in patient/stakeholder engagement?
• Will the research team be able to achieve the study
aims?
• Does the research team have complementary and
integrated expertise? Is the approach and structure
of the research team appropriate for the project?
PCORI Review Criteria Innovation
• Does the project address a new method or approach
for the field of PCOR?
• Does the project apply a proven method or approach
in a novel way to the field of PCOR?
• Is there a new application or refinement of
theoretical concepts, approaches, methodologies,
instrumentation or interventions proposed?
PCORI Review Criteria Approach
• Are the overall strategy, methodology, and analyses
appropriate to accomplish the specific aims?
• Are potential problems, alternative strategies, and
benchmarks for success presented?
• Will the strategy establish feasibility and will particular
risky aspects be managed
• Is the budget and timeframe appropriate for the research
plan?
• Are there plans for dissemination among key PCOR
stakeholders in education, practice and policy?
PCORI Review Criteria Environment
• Will the environment contribute to the probability of
success?
• Are the institutional support, equipment and other
physical resources adequate for the project?
• Will the project benefit from unique features of the
research environment, community involvement,
patient populations, or non-traditional stakeholder
collaborative arrangements?
PCORI Review
Template Based
OVERALL IMPACT SECTION
Overall Impact
SCORED REVIEW CRITERIA SECTIONS: The six review criteria, Significance,
Patient/Stakeholder Engagement, Investigators, Innovation, Approach,
Environment, with a separate score for the first five criteria.
1. Significance
Strengths
Weaknesses
33
When Writing Critiques
Overall Impact paragraph –
– requires a summary narrative (paragraph) of the strengths
and weaknesses. Do not cut and paste.
– Focus on major strengths and weaknesses that impacted
your overall rating of the application
– Overall impact score is not an average of the criterion
scores; weights can differ across reviewers
• Use bulleted points to make succinct, focused comments
(Be careful of being too brief – the reader needs to
understand your comment)
• Any criterion score between 3 to 9 needs to have weaknesses
listed
34
Scoring
9-point score scale is used to provide:
• Criterion Scores for each of the core review criteria
• Overall Impact/Priority Score based on but not an average of the
core criterion scores plus additional criteria
All applications receive scores:
• Not discussed applications (bottom half by pre-meeting scores) will
receive only initial criterion scores from the 3 assigned reviewers.
• Discussed applications also receive an averaged overall impact score
from eligible panel members (i.e., those without conflicts of
interest).
• These scores will be averaged to one decimal place, and multiplied
by 10. The 81 possible priority scores will thus range from 10-90.
9-Point Score Scale Descriptors
Impact
High
Medium
Low
Score
Descriptor
Additional Guidance
1
Exceptional
Exceptionally strong with essentially no weaknesses
2
Outstanding
Extremely strong with negligible weaknesses
3
Excellent
Very strong with only some minor weaknesses
4
Very Good
Strong but with numerous minor weaknesses
5
Good
6
Satisfactory
Some strengths but also some moderate weaknesses
7
Fair
Some strengths but with at least one major weakness
8
Marginal
9
Poor
Strong but with at least one moderate weakness
A few strengths and a few major weaknesses
Very few strengths and numerous major weaknesses
Process During the Meeting
• Chair calls for preliminary overall impact scores
• Primary reviewer
– Brief description
– Discuss major/ score-driving strengths and weaknesses
• Second and third reviewers add only major/score-driving
strengths and weaknesses
• Open discussion among all reviewers
• Human subjects issues
• Final scores from assigned reviewers
• Unassigned reviewers may vote outside the score range
• Record scores (in IAR and on hard copy)
• Budget (does not influence score!)
• 10-15 minutes per application
Reviewer Voter Sheet
(available only during the meeting)
Acknowledgments
• Michael Steinman MD, UCSF CTSI Director of
Comparative Effectiveness Research
• Jonathan Arias, PhD, NIH Center for Scientific
Review
• All of you