Using the PCORI Methodology
Standards to Generate Robust,
Relevant, and Timely Evidence for
Patient-Centered Outcomes Research
Joe V. Selby, David Hickam, Brian Mittman, and
Sean Tunis
June 4, 2013
The audio and slide presentation will
be delivered directly to your
computer

Speakers or headphones are required to hear the
audio portion of the webinar.

If you do not hear any audio now, check your
computer’s speaker settings and volume.

If you need an alternate method of accessing audio,
please submit a question through the Q&A pod.
Technical Assistance

Live technical assistance:
– Call Adobe Connect at (800) 422-3623

Refer to the ‘Technical Assistance’ box
in the bottom left corner for tips to
resolve common technical difficulties.
Questions may be submitted at
any time during the presentation
To submit a question:
1.
2.
Click in the Q&A
box on the left side
of your screen
Type your question
into the dialog box
and click the Send
button
Welcome

Erin Holve, Ph.D., M.P.H.,
M.P.P.
– Senior Director of Research
& Education, AcademyHealth
A Collaborative Effort to
Improve Training in PCOR
PCORI Mini-Course

Two morning panels
– Developing a Program of Comparative Effectiveness
Research at the Patient Centered Outcomes Research
Institute” (9:45-11:15am)
• Chair: Joe Selby, PCORI
• Faculty: Chad Boult, Rachael Fleurence, Romana HasnainWynia, and David Hickam, PCORI
– “PCORI Methods Pilot Projects: Learning from Engagement”
(11:30am-1:00pm)
• Chair: Sue Sheridan, PCORI
• Faculty: Katherne Bevans, Children’s Hospital of Philadelphia;
Renee Robinson, Southcentral Foundation; and Tiffany
Haynes, University of Arkansas for Medical Sciences
PCORI Mini-Course

Afternoon workshop (June 25, 2:00-6:00pm ET)
– Heterogeneity of Treatment Effects
• Chair: Brian Mittman, PhD, Department of Veterans Affairs
• Faculty: Ravi Varadhan, PhD, Jodi Segal, MD, MPH, Johns Hopkins
University School of Medicine
– Adaptive and Bayesian Trial Designs
• Chair: Sally Morton, PhD, University of Pittsburgh
• Faculty: Jason Connor, PhD, Berry Consultants
– Data Registries
• Chair: Robin Newhouse, PhD, RN, NEA-BC, FAAN, University of
Maryland
• Faculty: Priscilla Velentgas, Ph.D., Quintiles Outcome
Learning Objectives

Provide an overview of the PCORI Methodology
Standards and the process used to develop the
standards;

Provide general guidance on using the standards
when responding to PCORI funding announcements;
and

Review plans for implementing the standards and
updating them in the future.
Webinar Faculty
Joe V. Selby, MD,
MPH
David Hickam, MD, Brian Mittman,
MPH
PhD
Sean Tunis, MD,
MSc
Using the PCORI Methodology
Standards to Generate Robust,
Relevant, and Timely Evidence
for Patient-Centered Outcomes
Research
Joe V. Selby
Executive Director, PCORI
Academy Health Webinar
June 4, 2013
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
About PCORI
• An independent, non-profit organization authorized by
Congress.
• Mission is to fund research that will provide patients, their
caregivers and clinicians with the evidence-based
information needed to make better-informed health care
decisions.
• Committed to continuously seeking input from patients
and a broad range of stakeholders to guide its work.
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
What is Patient Centered Outcomes
Research?
Patient-Centered Outcomes Research (PCOR) helps people and their
caregivers communicate and make informed health care decisions, allowing
their voices to be heard in assessing the value of health care options.
PCOR has the following characteristics:
•
Actively engages patients and key stakeholders throughout the research
process.
•
Compares important clinical management options.
•
Evaluates the outcomes that are most important to patients.
•
Addresses implementation of the research finings in clinical care
environments.
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Approximate PCORI Funding
* 20% of this amount goes to AHRQ/HHS each year for dissemination and capacity-building
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Current Funding Opportunities and Events
Announcement
Letter of Intent Due
Assessment of Prevention, Diagnosis, and Treatment Options
06/15/13
Improving Healthcare Systems
06/15/13
Communication and Dissemination Research
06/15/13
Addressing Disparities
06/15/13
Improving Methods for Conducting Patient-Centered Outcomes
Research
06/15/13
Improving Infrastructure for Conducting Patient Centered Outcomes
Research: Phase One —Clinical Data Research Networks
06/19/13
Improving Infrastructure for Conducting Patient Centered Outcomes
Research: Phase One —Patient-Powered Research Networks
06/19/13
http://www.pcori.org/funding-opportunities/funding-announcements/funding-center/
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
David Hickam, MD, MPH, PCORI
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Developing the PCORI Methodology
Standards
• Congressional Requirements- Patient Protection and Affordable
Care Act, Subtitle D, Paragraph (6)(C)(i)
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
PCORI Methodology Committee
•
Sherine Gabriel, MD, MSc (chair),
Mayo Clinic
•
John Ioannidis, MD, DSc, Stanford
University
•
Robin Newhouse, PhD, RN (vice
chair), University of Maryland
•
Michael S Lauer, MD, National Heart,
Lung, and Blood Institute
•
Naomi Aronson, PhD, BlueCross
BlueShield
•
David O. Meltzer MD, PhD,
University of Chicago
•
Ethan Basch, MD, MSc, Memorial
Sloan-Kettering Cancer Center
•
Brian S. Mittman, PhD, Dept. of
Veterans Affairs
•
Alfred O. Berg, MD, MPH, University •
of Washington
Sebastian Schneeweiss, MD, ScD,
Harvard Medical School
•
David Flum, MD, MPH, University of •
Washington
Jean R. Slutsky, Agency for
Healthcare Research and Quality
•
Steven Goodman, MD, MHS, PhD,
Johns Hopkins School of Public
Health
•
Mary Tinetti, MD, Yale University
School of Medicine
•
Clyde Yancy, MD, MSc, Northwestern
University Feinberg School of
Medicine
•
Mark Helfand, MD, MS, MPH, Dept.
of Veterans Affairs
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
PCORI Methodology Standards
PCORI
Methodology
Committee
Formed
(2011)
Draft
Methodology
Report
Posted (July
2012)
Public
Comment
Period (JulySeptember
2012)
http://www.pcori.org/research-we-support/methodology/
Proposed
Revisions to
Methodology
Standards
(November
2012)
Revised
PCORI
Methodology
Standards
(December
2012)
Final
Methodology
Report
(Spring 2013)
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
11 Categories of Methodology Standards
47 Individual Standards
• Formulating Research
Questions
• Preventing/Handling Missing
Data
• Patient-Centeredness
• Data Networks
• Systematic Reviews
• Adaptive and Bayesian
Trial Designs
• General and Cross-cutting
Methods for all PCOR
• Causal Inference
• Data Registries
• Studies of Diagnostic Tests
• Heterogeneity of
Treatment Effects
*Bold indicates topics to be discussed in detail at the June 25 in-person workshop at the AcademyHealth Annual Research
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Characteristics of the Methodology
Standards
• Are minimal standards for performing comparative effectiveness
research.
• Are intended to provide helpful guidance to researchers and those
who use research results.
• Reflect generally accepted best practices.
• Provide guidance for both project protocols and reporting of
results.
• Are used to assess the scientific rigor of funding applications.
• Context of research should drive use of the standards.
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Standards that Apply to Most Projects
• Formulating research questions (6 standards)
• Patient-centeredness (4 standards)
• General and cross-cutting (6 standards)
• Missing data (5 standards)
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Formulating Research Questions

Identify affected populations and health decision(s), and how
study results will inform the health decision (RQ-3)

Identify and assess participant subgroups (RQ-4)

Select appropriate interventions and comparators and make
explicit how each were selected (RQ-5)
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Patient-Centeredness

Engage the population of interest and other relevant stakeholders
as appropriate and necessary in a given research context (PC-1)

Identify, select, recruit, and retain study participants
representative of the population of interest (PC-2)

Use patient-reported outcomes when patients or people at risk of a
condition are the best sources of information. (PC-3)

Support dissemination and implementation of study results (PC-4)
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
General and Cross-cutting Methods
• Assess the suitability of the data source (GM-1)
• Document data analysis plans for major aims (GM-3)
• Document validated scales and tests
• Include characteristics of the scales and psychometric
properties (GM-4)
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Missing Data
• Use Validated Methods to Deal with Missing Data that
Properly Account for Statistical Uncertainty Due to
Missingness (MD-3)
• Record and Report All Reasons for Dropout and Missing
Data, and Account for All Patients in Reports (MD-4)
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Standards that Apply to Specific Data
Sources and Study Designs
• Systematic reviews (1 standard)
• Causal inference (6 standards)
• Heterogeneity of treatment effects (4 standards)
• Data networks (2 standards)
• Adaptive trials (5 standards)
• Registries (3 standards)
• Diagnostic tests (5 standards)
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Causal Inference
• Precisely Define the Timing of the Outcome Assessment
Relative to the Initiation and Duration of Exposure (CI-3)
• Measure Confounders before Start of Exposure. Report data
on confounders with study results (CI-4)
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Diagnostic Tests
• Assess the Effect of Factors Known to Affect Diagnostic
Performance and Outcomes (DT-3)
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Brian Mittman, PhD, Department of Veterans Affairs
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
How to Use the Standards
• The standards represent “best practice” recommendations for
improving the quality and value of PCOR and CER.
• The standards aim to achieve greater transparency, rigor and other
improvements to strengthen the science and methods of PCOR and
CER.
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Implementing the standards: improving PCOR
research practices
•
“Research practices” include decisions and actions by numerous
stakeholders (funders, reviewers, researchers, etc.)
•
Changing these practices is challenging
– Researchers value independence and autonomy
– Researcher actions are guided by knowledge, experience,
judgment
•
Analogue: the challenge of improving research practices through
standards is comparable to the challenge of improving clinical
practices through evidence-based clinical practice guidelines.
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Implementing research standards
•
Professional practices are influenced by multiple factors operating at
multiple levels
•
Implementation gaps must be thoroughly diagnosed to identify “root
causes”
•
Implementation campaigns must include multiple, coordinated
components targeting the full spectrum of determinants of gaps
•
Influences on research practices – and points of leverage to change
practices – should be depicted in a causal, conceptual model
There are “no magic bullets”: examples of dramatic change in research
practices (e.g., trial registration) are rare and exceptional.
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Conceptual model of the research process
•
Research question identification and prioritization (researchers,
funding agencies)
•
Study design and methods selection (researcher judgment guided
by training, professional norms, published standards)
•
Peer review (reviewer judgment guided by training, professional
norms, published standards)
•
Study governance and conduct (researcher judgment, possibly
guided by reporting requirements, IRB review, DSMB review)
•
Documentation and reporting (researcher judgment guided by
reporting guidelines, journal requirements, peer review)
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Dissemination and Future Directions
• Initial set of standards released in December 2012.
• PCORI Methodology Report will be released in Summer 2013.
• No change in the set of standards.
• Provides further guidance on interpretation and use.
• Responding to future PCORI Funding Announcements (PFAs).
• All future submissions must demonstrate adherence
to/adoption of/consideration of the standards.
• PCORI will continue to develop tools and resources for
researchers, reviewers and other stakeholders.
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Sean Tunis MD, MSc - CMTP
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Purpose of Standards: PMC draft report 7/12
“Departures from basic good research practices are partially
responsible for the mismatch between the quality and relevance of
the information research provides and the information patients need
to make informed clinical decisions.”
“The initial range of topics was chosen to reflect areas in which the
Committee believed that there were either substantial deficiencies or
inconsistencies in how available methods were applied in practice, or
for which there was specialized knowledge in how best to conduct
research that had not been effectively disseminated.”
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Required by Law!
(ACA)
“Methodological standards shall provide specific criteria for”:
• Internal validity (robust)
• Generalizability (relevant)
• Timeliness (timely)
• Feasibility
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Where to Focus
• Craft an important, relevant research question
• Informed by gaps in systematic reviews and meaningful
input from patients and other decision makers
• Patient-centered care requires attention to subgroups and PROs
• Apply best practices in application of selected methods
• Provide full transparency
Clear, detailed protocol – explain deviations from standards
Follow existing reporting standards
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Asking the Right Question
“To produce information that is meaningful and useful to people when
making specific health decisions, research proposals and protocols
should describe: 1) the specific health decision the research is
intended to inform; 2) the specific population for whom the health
decision is pertinent; and 3) how study results will inform the health
decision.”
• Requires dialogue with decision makers (more is better)
• Methods should serve research question; not vice versa
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Use of Patient Reported Outcomes
“Select outcomes based on input directly elicited from patient
informants, people representative of the population of interest, either
in previous studies or in the proposed research.”
• Use existing core outcomes measures sets when available
• COMET (Core Outcomes Measures in Effectiveness Trials)
• Effectiveness Guidance Documents (Green Park Collaborative)
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Feasibility: An Unresolved Issue
Standard for Studies of Dx Tests:
“Studies of clinical outcomes after diagnostic testing should use a
prospective randomized study design when possible. If a nonrandomized design is proposed, the reason for using an observational
study (or modeling and simulation) should be addressed and efforts
to minimize confounding documented.”
P A T IENT -CENTERED O U T COM ES R ES EA RCH IN S T ITUTE
Looking Forward
• Specifics of implementation of standards not yet clear
• Initial emphasis of methods standards on patients and
caregivers as decision makers. Intent to expand scope to
clinicians, payers/purchasers, policy makers, others.
• PCORI thinking continues to evolve based on experience and
public feedback from researchers and other stakeholders
• How best to achieve optimal balance of internal validity,
generalizability, timeliness and feasibility remains an
important challenge
Submitting Questions
To submit a question:
1.
2.
Click in the Q&A
box on the left side
of your screen
Type your question
into the dialog box
and click the Send
button
Expert Q&A
Erin Holve (facilitator)
 Dave Hickam
 Brian Mittman
 Sean Tunis

Still time to sign up for the PCORI Mini-Course
In-person events at AcademyHealth’s Annual Research Meeting in Baltimore, MD (June 25)
•
Two morning panels
• “Developing a Program of Comparative Effectiveness Research at the Patient Centered
Outcomes Research Institute” (9:45-11:15am)
• “PCORI Methods Pilot Projects: Learning from Engagement” (11:30am-1:00pm)
Afternoon workshop (June 25, 2:00-6:00pm ET)
• Heterogeneity of Treatment Effects
• Chair: Brian Mittman, PhD, Department of Veterans Affairs
• Faculty: Ravi Varadhan, PhD, Jodi Segal, MD, MPH, Johns Hopkins University School of
Medicine
• Adaptive and Bayesian Trial Designs
• Chair: Sally Morton, PhD, University of Pittsburgh
• Faculty: Jason Connor, PhD, Berry Consultants
• Data Registries
• Chair: Robin Newhouse, PhD, RN, NEA-BC, FAAN, University of Maryland
• Faculty: Priscilla Velentgas, Ph.D., Quintiles Outcome
•
•
Click here to register: $50 for AcademyHealth members, $100 for nonmembers
Thank You
Please take a moment to fill out the
brief evaluation which will appear in your browser.