Assessment of Prevention, Diagnosis, and Treatment Options Cycle 2, 2015 PFA
Offline Reviewer Critique Template
Assessment of Prevention, Diagnosis, and Treatment Options (APDTO)
The APDTO program is seeking applications for comparative effectiveness research designed
to provide evidence-based information to inform critical decisions that face patients and
caregivers, clinicians, policy makers, and healthcare system leaders.
Important Reminders:





Before beginning your first critique, read the APDTO Program PCORI Funding Announcement. Key
aspects of the funding announcement can also be found in the reference table on pages 9 and 10 in
this document.
Use this offline template to draft a written critique for each criterion you have been assigned to
review. The offline reviewer critique template begins on page 2.
Please use the scoring rubric accompanying each criterion, as well as the definitions of strength and
weakness modifiers provided on page 8, to ensure that your comments and numeric scores align.
This helps other reviewers and Program staff to better understand your scores and to use that
information in preparing for the in-person meeting.
Once you have completed your written critique based on the review criteria, double-check to make
sure that you have addressed the key points listed on page 6.
Once you have finished drafting your entire critique, please copy/paste your comments into PCORI
Online.
1
Assessment of Prevention, Diagnosis, and Treatment Options Cycle 2, 2015 PFA
Offline Reviewer Critique Template
Please use this template only as a placeholder for your critiques. Once you have finished your critique, please
copy/paste your comments into the PCORI Online Critique Form.
Keep comments for each criterion to no more than 5000 characters, including spaces. Comments for each criterion
should list out strengths and weaknesses using a bulleted format. Comments for the Protection of Human Subjects
and Overall Narrative sections should be written in paragraph form.
Request ID:
Criterion 1. Impact of the condition on the health of individuals and populations
The proposal addresses the following questions:
 Is the condition or disease associated with a significant burden in the U.S. population, in terms
of prevalence, mortality, morbidity, individual suffering, or loss of productivity?
 Alternatively, does the condition or disease impose a significant burden on a smaller number of
people who have a rare disease?
 Does the proposal include a particular emphasis on patients with one or more chronic
condition(s)?
Strengths:



Weaknesses:



Criterion 1 Score:
Criterion 2. Potential for the study to improve health care and outcomes
The proposal has the potential to lead to meaningful improvement in the quality and efficiency of care
and to improvements in outcomes important to patients. It addresses the following questions:
 Does the research question address a critical gap in current knowledge as noted in systematic
reviews, guideline development efforts, or previous research prioritizations?
 Has the research been identified as important by patient, caregiver, or clinician groups?
 Do wide variations in practice patterns suggest current clinical uncertainty?
2
Assessment of Prevention, Diagnosis, and Treatment Options Cycle 2, 2015 PFA
Offline Reviewer Critique Template


Is the research novel or innovative in its methods or approach, in the population being studied,
or in the intervention being evaluated, in ways that make it likely to improve care?
Do preliminary studies indicate the potential for a sizable benefit of the intervention relative to
current practice? How likely is it that positive findings could be disseminated and implemented
quickly, resulting in improvements in practice and patient outcomes?
Strengths:



Weaknesses:



Criterion 2 Score:
Criterion 3: Technical merit
The proposal has sufficient technical merit in the research design to ensure that the study goals will be
met. It addresses the following questions:
 Does the proposal delineate a clear conceptual framework/theory/model that anchors the
background literature and informs the design, key variables, and relationships being tested?
 Does the research plan describe rigorous methods that demonstrate adherence to PCORI’s
Methodology Standards?
 Are the comparison interventions realistic options that exist in current practice?
 Are the sample sizes and power estimates presented based on realistic and careful evaluations
of the anticipated effect size?
 Is the project timeline realistic, including specific scientific and engagement milestones?
 Does the research team have the necessary expertise to conduct the project?
 Are the organizational structure and the described resources appropriate to carry out the
project?
 Is there a diverse study population with respect to age, gender, race, ethnicity, and clinical
status appropriate for the proposed research?
3
Assessment of Prevention, Diagnosis, and Treatment Options Cycle 2, 2015 PFA
Offline Reviewer Critique Template
Strengths:



Weaknesses:



Criterion 3 Score:
Criterion 4: Patient-centeredness
The proposal demonstrates patient-centeredness at every stage of the research. It addresses the
questions:
 Is the research focused on questions that affect outcomes of interest to patients and their
caregivers?
 Does the research address one or more of the key questions mentioned in PCORI’s definition of
PCOR?
o “Given my personal characteristics, conditions, and preferences, what should I expect
will happen to me?”
o “What are my options, and what are the potential benefits and harms of those
options?”
o “What can I do to improve the outcomes that are most important to me?”
o “How can clinicians and the care delivery systems they work in help me make the best
decisions about my health and health care?”
Strengths:



Weaknesses:



4
Assessment of Prevention, Diagnosis, and Treatment Options Cycle 2, 2015 PFA
Offline Reviewer Critique Template
Criterion 4 Score:
Criterion 5: Patient and stakeholder engagement
The proposal demonstrates that people representing the population of interest and other relevant
stakeholders are engaged in ways that are appropriate and necessary in a given research context. It
addresses the questions:
 Are patients and other stakeholders engaged in:
o Formulating research questions?
o Defining essential characteristics of study participants, comparators, and outcomes?
o Identifying and selecting outcomes that the population of interest notices and cares
about (e.g., survival, function, symptoms, health-related quality of life) and that inform
decision making relevant to the research topic?
o Monitoring study conduct and progress?
o Designing/suggesting plans for dissemination and implementation activities?
 Are the roles and the decision-making authority of all research partners clearly stated?
 Does the proposal demonstrate the principles of reciprocal relationships, co-learning,
partnership, trust, transparency, and honesty?
Strengths:



Weaknesses:



Criterion 5 Score:
5
Assessment of Prevention, Diagnosis, and Treatment Options Cycle 2, 2015 PFA
Offline Reviewer Critique Template
Protection of Human Subjects:
Does the application have acceptable risks and/or adequate protections for human subjects? (Yes/No)
Please provide comments related to human subjects protections, if any.
Overall Score:
Overall Narrative:
Please provide your overall narrative here. The narrative should be written in paragraph form and
provide a summary of strengths and weaknesses that drove the overall score.
6
Assessment of Prevention, Diagnosis, and Treatment Options Cycle 2, 2015 PFA
Offline Reviewer Critique Template
Reference: Key Points
Once you have completed your written critique based on the review criteria, double-check to make
sure that you have addressed these key points:
Criterion 2
 Does the proposed research study the benefits and harms of interventions delivered in realworld settings, and will the study provide practical information to help patients make informed
healthcare decisions?
Criterion 3
 Does the proposed research compare at least two viable alternatives, each supported by efficacy
data?
 Does the proposed research have sufficient statistical power to detect clinically meaningful
effects? Are proposed sample sizes justified? Are assumptions used in power calculations
explained?
Criterion 4
 Does the proposed research include health outcomes that are meaningful to the patient
population, their caregivers, and family members under study, and that are likely to guide their
decisions?
All Criteria (as appropriate for each type of reviewer):
 Does the proposed research adhere to all relevant PCORI Methodology Standards? (Note
relevant Review Criteria for addressing each standard.)
o Identify a true gap in evidence (RQ-1; Crit. 1)
o Identify the comparators and explain why they were selected (RQ-5; Crit. 3)
o Provide a rationale and define “usual care,” if it is proposed (RQ-5; Crit. 2 & 3)
o Show that the outcomes chosen are relevant to patients and end users (RQ-6; Crit. 4)
o Show that study participants represent target populations and are properly recruited,
enrolled, and retained (PC-2; Crit. 3)
o Describe rigorous data analysis plans (IR-1, IR-3; Crit. 3)
o Describe their plan and statistical methods for handling missing data in their project
(MD-1, MD-2; Crit. 3)
o Address heterogeneity of treatment effects and state the aims of HTE analysis (HT-1; Crit.
3)
Notify your MRO if the application proposes to:
 Measure cost-effectiveness or conduct a formal cost-effectiveness analysis as a primary aim.
 Develop, test, and/or validate individual decision aids.
7
Assessment of Prevention, Diagnosis, and Treatment Options Cycle 2, 2015 PFA
Offline Reviewer Critique Template
Reference: PCORI Scoring Chart
Definitions of strength and weakness modifiers:






Minor Strength: An attribute that could lead to improvements in healthcare and/or
outcomes
Moderate Strength: An attribute that would probably lead to improvements in
healthcare and/or outcomes
Major Strength: An attribute that is likely to lead to improvements in healthcare and/or
outcomes
Minor Weakness: An easily addressable weakness that does not substantially lessen the
impact of the study’s results on healthcare and/or outcomes
Moderate Weakness: A weakness that would lessen impact of the study’s results on
healthcare and/or outcomes
Major Weakness: A weakness that would seriously limit impact of the study’s results on
healthcare and/or outcomes
8
Assessment of Prevention, Diagnosis, and Treatment Options Cycle 2, 2015 PFA
Offline Reviewer Critique Template
Reference: APDTO PFA Summary
What is APDTO interested in?
The APDTO program is seeking applications for comparative effectiveness research designed
to provide evidence-based information to inform critical decisions that face patients and
caregivers, clinicians, policy makers, and healthcare system leaders.
APDTO must fund applications that compare the effectiveness of
at least 2 viable alternatives that have established efficacy.
PCORI seeks to fund investigator-initiated research that:
 Compares the effectiveness of two or more strategies for
prevention, treatment, screening, diagnosis, or
management that are known to be efficacious but have
not been adequately compared in previous studies; it
may be appropriate to include as a comparator a
generally accepted practice that occurs with insufficient
evidence of efficacy or effectiveness; PROI is particularly
interested in studies that are conducted in typical clinical
populations and that address the full range of relevant
patient-centered outcomes (PCOs)
 Addresses a high-priority evidence gap, as identified by
prior systematic reviews, clearly defined gaps in clinical
guidelines, or other credible evidence reviews
 Among compared groups, investigates various factors
that account for variation in treatment outcomes, with
attention paid to demographic, biological, clinical, social,
economic, or geographic factors, comorbidities, and other
factors that may influence those outcomes; strategies
may focus on patient populations with a single condition
or involve patients with a range of conditions.
9
Please notify your MRO as
soon as possible if you
believe any of your
assigned applications do
not include a comparator.
Your MRO will discuss any
identified issues with
program staff and provide
you with further
instructions.
Assessment of Prevention, Diagnosis, and Treatment Options Cycle 2, 2015 PFA
Offline Reviewer Critique Template
What is APDTO not interested in?
PCORI cannot fund applications that measure cost effectiveness.
 Research cannot conduct a formal cost-effectiveness analysis in
the form of dollar-cost per quality-adjusted life year to
compare two or more alternatives.
 Research cannot measure the relative costs of care of two or
more alternative approaches as the primary criterion for
choosing the preferred alternative.
PCORI cannot fund applications that develop practice guidelines,
coverage recommendations, payment or policy recommendations.
PCORI cannot fund research whose findings will include:
• Practice guidelines
• Coverage recommendations
• Payment or policy recommendations
• Creation of clinical practice guidelines or care pathways
• Establishing efficacy for a new clinical strategy
• Pharmacodynamics
• Study of the natural history of disease
• Fundamental science or study of biological mechanisms
APDTO applications should not focus on the following categories:





Instrument development
Developing, testing and validating new decision aids/tools or
clinical prognostication tools
Pilot studies intended to inform larger efforts
Comparisons of patient characteristics rather than clinical
strategy options
Studies comparing interventions for which the primary focus in
the role of community health workers or patient navigators
10
If an application meets
any of these
conditions, please
notify your MRO as
soon as possible. Your
MRO will discuss all
identified issues with
program staff and
provide you with
further instructions.