Improving Healthcare Systems Research Cycle 2, 2015 PFA
Offline Critique Template
Improving Healthcare Systems (IHS)
The IHS program is seeking studies that will affect healthcare delivery by determining which system
features lead to improved patient-centered outcomes (PCOs) and which provide valuable knowledge
to patients, their caregivers, and clinicians, as well as other key stakeholders, including payers and
employers.
Important Reminders:
 Before beginning your first critique, read the IHS Program PCORI Funding Announcement. Key
aspects of the funding announcement can also be found in the reference table at the end of this
document.
 Use this offline template to draft a written critique for each criterion you have been assigned to
review. The offline reviewer critique template begins on page 2.
 Please use the scoring rubric on page 8 to ensure that your comments and numeric scores align.
This helps other reviewers and Program staff to better understand your scores and to use that
information in preparing for the in-person meeting.
 Once you have completed your written critique based on the review criteria, double-check to make
sure that you have addressed the key points listed on page 7.
 Once you have finished drafting your entire critique, please copy/paste your comments into PCORI
Online.
1
Improving Healthcare Systems Research Cycle 2, 2015 PFA
Offline Critique Template
Please use this template only as a placeholder for your critiques. Once you have finished your critique, please
copy/paste your comments into the PCORI Online Critique Form.
Keep comments for each criterion to no more than 5000 characters, including spaces. Comments for each criterion
should list out strengths and weaknesses using a bulleted format. Comments for the Protection of Human Subjects
and Overall Narrative sections should be written in paragraph form.
Request ID:
Criterion 1. Impact of the condition on the health of individuals and populations
The proposal addresses the following questions:
 Is the condition or disease associated with a significant burden in the U.S. population, in terms
of prevalence, mortality, morbidity, individual suffering, or loss of productivity?
 Alternatively, does the condition or disease impose a significant burden on a smaller number of
people who have a rare disease?
 Does the proposal include a particular emphasis on patients with one or more chronic
condition(s)?
Strengths:



Weaknesses:



Criterion 1 Score:
Criterion 2. Potential for the study to improve health care and outcomes
The proposal has the potential to lead to meaningful improvement in the quality and efficiency of care
and to improvements in outcomes important to patients. It addresses the following questions:
 Does the research question address a critical gap in current knowledge as noted in systematic
reviews, guideline development efforts, or previous research prioritizations?
 Has the research been identified as important by patient, caregiver, or clinician groups?
 Do wide variations in practice patterns suggest current clinical uncertainty?
2
Improving Healthcare Systems Research Cycle 2, 2015 PFA
Offline Critique Template


Is the research novel or innovative in its methods or approach, in the population being studied,
or in the intervention being evaluated, in ways that make it likely to improve care?
Do preliminary studies indicate the potential for a sizable benefit of the intervention relative to
current practice? How likely is it that positive findings could be disseminated and implemented
quickly, resulting in improvements in practice and patient outcomes?
Strengths:



Weaknesses:



Criterion 2 Score:
Criterion 3: Technical merit
The proposal has sufficient technical merit in the research design to ensure that the study goals will be
met. It addresses the following questions:
 Does the proposal delineate a clear conceptual framework/theory/model that anchors the
background literature and informs the design, key variables, and relationships being tested?
 Does the research plan describe rigorous methods that demonstrate adherence to PCORI’s
Methodology Standards?
 Are the comparison interventions realistic options that exist in current practice?
 Are the sample sizes and power estimates presented based on realistic and careful evaluations
of the anticipated effect size?
 Is the project timeline realistic, including specific scientific and engagement milestones?
 Does the research team have the necessary expertise to conduct the project?
 Are the organizational structure and the described resources appropriate to carry out the
project?
 Is there a diverse study population with respect to age, gender, race, ethnicity, and clinical
status, appropriate for the proposed research?
3
Improving Healthcare Systems Research Cycle 2, 2015 PFA
Offline Critique Template
Strengths:



Weaknesses:



Criterion 3 Score:
Criterion 4: Patient-centeredness
The proposal demonstrates patient-centeredness at every stage of the research. It addresses the
following questions:
 Is the research focused on questions that affect outcomes of interest to patients and their
caregivers?
 Does the research address one or more of the key questions mentioned in PCORI’s definition of
PCOR?
o “Given my personal characteristics, conditions, and preferences, what should I expect
will happen to me?”
o “What are my options, and what are the potential benefits and harms of those
options?”
o “What can I do to improve the outcomes that are most important to me?”
o “How can clinicians and the care delivery systems they work in help me make the best
decisions about my health and health care?”
Strengths:



Weaknesses:



4
Improving Healthcare Systems Research Cycle 2, 2015 PFA
Offline Critique Template
Criterion 4 Score:
Criterion 5: Patient and stakeholder engagement
The proposal demonstrates patient-centeredness at every stage of the research. It addresses the
questions:
 Are patients and other stakeholders engaged in:
o Formulating research questions?
o Defining essential characteristics of study participants, comparators, and outcomes?
o Identifying and selecting outcomes that the population of interest notices and cares
about (e.g., survival, function, symptoms, health-related quality of life) and that inform
decision making relevant to the research topic?
o Monitoring study conduct and progress?
o Designing/suggesting plans for dissemination and implementation activities?
 Are the roles and the decision-making authority of all research partners clearly stated?
 Does the proposal demonstrate the principles of reciprocal relationships, co-learning,
partnership, trust, transparency, and honesty?
Strengths:



Weaknesses:



Criterion 5 Score:
5
Improving Healthcare Systems Research Cycle 2, 2015 PFA
Offline Critique Template
Protection of Human Subjects:
Does the application have acceptable risks and/or adequate protections for human subjects? (Yes/No)
Please provide comments related to human subjects protections, if any.
Overall Score:
Overall Narrative:
Please provide your overall narrative here. The narrative should be written in paragraph form and
provide a summary of strengths and weaknesses that drove the overall score.
6
Improving Healthcare Systems Research Cycle 2, 2015 PFA
Offline Critique Template
Reference: Key Points
Criterion 2
 Does the proposed research study the benefits and harms of interventions delivered in realworld settings, and will the study provide practical information to help patients make informed
healthcare decisions?
 How likely is it that positive findings could be disseminated and implemented quickly, resulting in
improvements in practice and patient outcomes?
Criterion 3 (note that Patient and Stakeholder Reviewers are not required to evaluate Criterion 3)
 Does the proposed research compare at least two viable alternatives, each supported by efficacy
data?
 Does the proposed research have sufficient statistical power to detect clinically meaningful
effects? Are proposed sample sizes justified? Are assumptions used in power calculations
explained?
 Does the proposal delineate a clear conceptual framework/theory/model that anchors the
background literature and informs the design, key variables, and relationships being tested?
Criterion 4
 Does the proposed research include health outcomes that are meaningful to the patient
population, their caregivers, and family members under study, and that are likely to guide their
decisions?
All Criteria (as appropriate for each type of reviewer):
 Does the proposed research adhere to all relevant PCORI Methodology Standards? (Note
relevant Review Criteria for addressing each standard.)
o Identify a true gap in evidence (RQ-1; Crit. 1)
o Identify the comparators and explain why they were selected (RQ-5; Crit. 3)
o Provide a rationale and define “usual care,” if it is proposed (RQ-5; Crit. 2 & 3)
o Show that the outcomes chosen are relevant to patients and end users (RQ-6; Crit. 4)
o Show that study participants represent target populations and are properly recruited,
enrolled, and retained (PC-2; Crit. 3)
o Describe rigorous data analysis plans (IR-1, IR-3; Crit. 3)
o Describe their plan and statistical methods for handling missing data in their project
(MD-1, MD-2; Crit. 3)
o Address heterogeneity of treatment effects and state the aims of HTE analysis (HT-1; Crit.
3)
Notify your MRO if the application proposes to:
 Measure cost-effectiveness or conduct a formal cost-effectiveness analysis as a primary aim.
 Develop, test, and/or validate individual decision aids/tools.
7
Improving Healthcare Systems Research Cycle 2, 2015 PFA
Offline Critique Template
Reference: PCORI Scoring Chart
Definitions of strength and weakness modifiers:






Minor Strength: An attribute that could lead to improvements in healthcare and/or
outcomes
Moderate Strength: An attribute that would probably lead to improvements in
healthcare and/or outcomes
Major Strength: An attribute that is likely to lead to improvements in healthcare and/or
outcomes
Minor Weakness: An easily addressable weakness that does not substantially lessen the
impact of the study’s results on healthcare and/or outcomes
Moderate Weakness: A weakness that would lessen impact of the study’s results on
healthcare and/or outcomes
Major Weakness: A weakness that would seriously limit impact of the study’s results on
healthcare and/or outcomes
8
Improving Healthcare Systems Research Cycle 2, 2015 PFA
Offline Critique Template
Reference: IHS PFA Summary
What is IHS interested in?
The IHS program is seeking studies that will affect healthcare delivery by determining
which system features lead to improved patient-centered outcomes (PCOs) and which
provide valuable knowledge to patients, their caregivers, and clinicians, as well as other
key stakeholders, including payers and employers.
IHS is interested in comparative effectiveness research
focused on:




Patients’ access to care, high quality of care, support for
self-care, and coordination of care across healthcare
settings
Professional decision making on the basis of patients’
personal values
Experiences that are important to patients and their
caregivers, such as overall health, functional ability,
health-related quality of life, stress, severity of
symptoms, survival, and unanticipated healthcare
utilization, such as unexpected hospital stays or visits to
the emergency department
The efficiency of healthcare delivery, as measured by the
amount of ineffective, duplicative, or wasteful care
provided to patients
IHS applications must describe research that:
 Compares two or more strategies, each of which has
established efficacy.
 Studies the benefits and harms of interventions and
strategies delivered in real-world settings.
 Is based on health outcomes that are meaningful to the
patient population, their caregivers, and family members
under study, and that are likely to guide their decisions.
9
Although all letters of intent
have been screened for
programmatic fit, please
notify your MRO if you
believe any of your assigned
applications do not align
with the IHS program’s
research interests.
Programmatic alignment
concerns should NOT affect
your scores or your written
critiques.
Please notify your MRO as
soon as possible if you
believe any of your assigned
applications do not include
a comparator. Your MRO
will discuss any identified
issues with program staff
and provide you with
further instructions.
Improving Healthcare Systems Research Cycle 2, 2015 PFA
Offline Critique Template
What is IHS not interested in?
PCORI cannot fund applications that measure cost effectiveness.


Research cannot conduct a formal cost-effectiveness
analysis in the form of dollar-cost per quality-adjusted life
year.
Research cannot measure the relative costs of care of
two or more alternative approaches as the primary
criterion for choosing the preferred alternative.
PCORI cannot fund applications that develop practice guidelines,
coverage recommendations, payment or policy
recommendations.
PCORI cannot fund research whose findings will include:
• Practice guidelines
• Coverage recommendations
• Payment or policy recommendations
• Creation of clinical practice guidelines or care pathways
• Establishing efficacy for a new clinical strategy
• Pharmacodynamics
• Study of the natural history of disease
• Fundamental science or study of biological mechanisms
AD applications should not focus on the following categories:





Instrument development
Developing, testing and validating new decision
aids/tools or clinical prognostication tools
Pilot studies intended to inform larger efforts
Comparisons of patient characteristics rather than clinical
strategy options
Studies comparing interventions for which the primary
focus in the role of community health workers or patient
navigators
10
If an application meets any of
these conditions, please
notify your MRO as soon as
possible. Your MRO will
discuss all identified issues
with program staff and
provide you with further
instructions.