Enhancing Peer Review at NIH
Cheryl A. Kitt, Ph.D.
Deputy Director, CSR
National Institutes of Health
U.S. Department of Health and Human Services
National
N ti
l Institutes
I tit t off Health
H lth
Office of the Director
National Institute
on Aging
National Institute
on Alcohol Abuse
and Alcoholism
National Institute
of Allergy and
Infectious Diseases
National Institute
of Arthritis and
Musculoskeletal
and Skin Diseases
National Cancer
Institute
National Institute
of Child Health
and Human
Development
National Institute on
Deafness and Other
Communication
Disorders
National Institute
of Dental and
Craniofacial
Research
National Institute
of Diabetes and
Digestive and
Kidney Diseases
National Institute
on Drug Abuse
National Institute
of Environmental
Health Sciences
National Eye
Institute
National Institute
of General
Medical Sciences
National Heart,,
Lung, and Blood
Institute
National Human
Genome Research
Institute
National Institute
of Mental Health
National Institute
of Neurological
Disorders and
Stroke
National Institute
of Nursing Research
National Institute of
Biomedical Imaging
and Bioengineering
National Center
for Complementary
and Alternative
Medicine
Fogarty
International
Center
National Center
for Research
Resources
National Library
of Medicine
National Center on
Minority Health and
Health Disparities
Clinical Center
Center for
Information
Technology
Center for
Scientific Review
Scientific Review Process
Dual Review System for Grant Applications
First Level of Review
CSR or Institute Review
S i tifi Review
R i
G
Scientific
Group
(Study Section)
Second Level of Review
NIH Institute/Center
Council
CSR Review Divisions
Neuroscience, Development
and Aging
Brain Disorders &
Clinical Neuroscience
Molecular, Cellular &
Developmental
Neuroscience
p
Integrative, Functional &
Cognitive Neuroscience
Emerging Technologies &
Training in Neuroscience
Biology of Development
& Aging
AIDS, Behavioral
and Population Sciences
Biobehavioral &
Behavioral Processes
Risk, Prevention&
Health Behaviors
Epidemiology &
Population Sciences
Healthcare Delivery
& Methodologies
AIDS &
Related Research
Basic and Integrative
Biological Sciences
Biological Chemistry &
Macromolecular
Biophysics
Bioengineering Sciences
& Technologies
Cell Biology
Genes, Genomes
& Genetics
Oncology 1 – Basic
Translational
Interdisciplinary
Molecular
& Training
Physiological and
Pathological Sciences
Translational and
Clinical Sciences
Endocrinology,
Metabolism,
Nutrition &
Reproductive Sciences
Cardiovascular and
Respiratory Sciences
Immunology
Surgical Sciences,
Biomedical
I
Imaging
i and
d
Bioengineering
Infectious Diseases
& Microbiology
Musculoskeletal, Oral
And Skin Sciences
Digestive, Kidney &
Urological Systems
Oncology 2 –
Translational Clinical
Vascular
V
l andd
Hematology
Get Your Application to the Right Study
Section !
• Review CSR Integrated Review Group
(IRG) and Scientific Review Group (Study
Study
Section)) Guidelines to Identify
Section
y a Home for
Your Application.
• Submit a Cover Letter !
2008: The Year of Peer Review
Enhancing Peer Review
“Fund the best science,, by
y the best scientists,,
with the least administrative burden…”
Elias
as Zerhouni,
e ou , MD,, Former
o e Director,
ecto , NIH
Amended Applications:
To speed the funding of meritorious science and
minimize reviewer burden:
•
As of January 25, 2009, all original new applications (i.e.,
never submitted)) and competing
p
g renewal applications
pp
are
permitted only a single amendment (A1).
(A1)
Balanced
Across
Career
B l
d and
d Fair
F i Reviews
R i
A
C
Stages and Scientific Fields
• New Investigator (NI):
PD/PI who has not yet competed successfully for a
substantial NIH research grant
o For multiple PD/PIs-all PD/PIs must meet requirements for NI status
• Early Stage Investigator (ESI):
PD/PI who qualifies as a New Investigator AND is within 10
years of completing the terminal research degree or is within
10 years of completing medical residency (or equivalent)
•
•
•
Applies only to R01 applications
New Investigators/Early Stage Investigators will be clustered
together for review
New Investigators/Early Stage Investigators Apply Now!!!
Enhanced Review Criteria
• Overall Impact:
Assessment of the likelihood for the project to
exert a sustained, powerful influence on the
research field(s)
( ) involved
• New Core Criteria Order:
Significance
Investigator(s)
Innovation
Approach
Environment
o Review criteria enhanced and expanded
and each scored from 11-9
Scoring
To improve
p
the transparency
p
y of the scoring
g
process:
•
Score applications
S
li ti
on five
fi review
i
criteria
it i using
i
a scale
l off
1-9.
•
score using
1-9 scale.
scale
Preliminary
y overall impact
p
g 1-
•
Discussed applications will receive an overall score from
each eligible (i.e., without conflicts of interest) panel
member and these scores will be averaged to one decimal
place, and multiplied by 10. The 81 possible priority scores
will thus range from 10-90.
P
Percentiles
til will
ill be
b reported
t d in
i whole
h l numbers.
b
All applications will receive scores:
scores:
Not Discussed applications
pp
will receive initial criterion
scores from the three assigned reviewers
•
•
Scoring Descriptions
Impact
Hi h IImpactt
High
Moderate Impact
Low Impact
Score Descriptor
1
Exceptional
2
O t t di
Outstanding
3
Excellent
4
V
Very
G
Good
d
5
Good
6
Satisfactory
7
Fair
8
Marginal
9
Poor
Cii
Critiques
To improve
p
the quality
q
y of the critiques
q
and to focus
reviewer attention on the review criteria:
•
Electronic template for critiques that will prompt for
strengths and weaknesses for each criterion.
•
Summary statement are shorter and more focused
focused.
•
Discussed applications only have a summary of the
panel’s discussion at the meeting.
•
ALL applications will be scored.
scored and receive critiques
Not discussed applications will receive criterion scores
only
l
Template--Based Critiques
Template
• The objective is to write evaluative statements and to
discourage summarizing the application
• Comments will be in the form of bullet points or if
necessary short narratives
1. Significance
Please limit text to ¼ page
Strengths
•
•
•
Weaknesses
•
•
•
14
What Happens at the Study Section Meeting?
Discuss applications in order of average
preliminary score:
Why:
•Concern - variation of scores during different times
of the meeting.
One
scores
O
e recommendation
eco
e dat o was
as to recalibrate
eca b ate sco
es at
the end of the meeting
Solution:
•Recalibrate dynamically throughout meeting.
Requirement:
R i
Reviewers
must participate
i i
in
i entire
i meeting.
i
Not Discussed
Discuss ~ 50% of applications
SRO then asks if there are any other applications
panel wishes to discuss
The remaining applications will not be discussed
(applications receive criterion scores only)
Final Scores
• Discussed applications receive an overall score
from each eligible (i.e., without conflicts of interest)
panel member and these scores will be averaged to
one decimal place
place, and multiplied by 10.
10 The
priority scores will thus range from 10
10--90
90.
• Percentiles
e ce t es a
are
e reported
epo ted in whole
o e numbers.
u be s
SUMMARY
Enhancing Peer Review Implementation Timeline
January 2009 Due Dates (for potential FY2010 1.
1.Early
Early Stage Investigator (ESI) and New
funding)
Investigator Policy (NOT(NOT-OD
OD--09
09--013;
http://grants.nih.gov/grants/guide/notice-http://grants.nih.gov/grants/guide/notice
files/NOT--OD
files/NOT
OD--0909-013.html)
013.html)
2.New
2.
New NIH Policy on Resubmissions (NOT(NOT-OD
OD-09--003:
09
http://grants.nih.gov/grants/guide/notice-http://grants.nih.gov/grants/guide/notice
files/NOT--OD
files/NOT
OD--0909-003.html)
003.html)
May 2009 Review Meetings (for potential 1.9
1.9-Point Scoring System
FY2010 funding)
2.Enhanced
2.
Enhanced Review Criteria
3.Formatted
3.
Formatted Reviewer Critiques
4 Scoring
4.Scoring
4.
S
i off Individual
I di id l Review
R i
Criteria
Ci i
5.Clustering
5.
Clustering of New Investigator
Applications During Review
J
January
2010 D
Due D
Dates
t (for
(f potential
t ti l FY2011 1.Shorter
1.Shorter
1
Sh t A
Applications
li ti
ffor R01
R01s and
d Oth
Other
funding)
Mechanisms
2.Restructured
2.
Restructured Applications to Align with Review
Criteria
kitt @
ih
kittc@csr.nih.gov
301--435301
435-1112