NIH Peer Review
Challenges and Opportunities in Peer Review
The Strategic National Importance of Peer Review
A Vision for Ensuring Its Strategic National Value
toni scarpa
NIH and Peer Review
a.scarpa@csuohio.edu
440 991 6022
NIH Center for Scientific Review
The Drivers for Change
Enhancing Peer Review
Cleveland State University
Cleveland, OH
April 12, 2013
The Strategic National Importance of Peer Review
NIH Support to Scientists Who Later Received Nobel
Prizes in Physiology/Medicine or Chemistry
30 Years of Medical Innovation
•
•
•
•
•
•
•
•
•
MRI and CT Imaging
ACE inhibitors
Angioplasty
Statins
Mammography
Coronary Interventions
H inhibitors and H2 Blockers
Antidepressants
Cataract and Lens
Replacement
• Ultrasound Imaging
• Asthma Treatment
•
•
•
•
•
•
•
•
•
•
Cardiac Enzymes
Fluoroquinolones
Hypoglycemic Agents
HIV Testing and Intervention
Tamoxifen
PSA
H. Pylori Test and Treatment
Bone Densitometry
Cephalosporins
Calcium Blockers
Fuchs and Sox, Health Affairs, 20, 30-42
Why Has The U.S. Biomedical-Behavioral
Research Been So Successful?
Major Differences Between Europe and U.S.
In Funding Biomedical Research
It is not the money but the way it is spent
Nobel Prizes
and Major
Discoveries
Others
Global Federal
Expenditure for
Biomedical Research in
Developed Countries
USA
9%
USA 87%
U.S.
Organization Ministry of university,
science/research/education
Department Health and
Human Services
Goal
Promote biomedical research
Provide employment for
scientists
Promote cures
Reengineer medicine
How funds
Large entitlement to
are allocated universities, research centers
Small fraction through peer
review to researchers
100% through peer
review to researchers
The
Rest
91%
1
The Rules and the Results of the Process
Funding Longevity of NIH Investigators
• Researchers are “Contractors” who bid in an
open competition
• Peer Review is the judge of the competition
• Universities and Research Institutions receive
funds only to the extent they have competitive
Faculty
How a Mosquito Helped Creating the NIH
(and the USA)
NIH Peer Review
Yellow Fever and the Decline of French America
Memphis: Death and Exodus
United
States
Louisiana Purchase 1803
2
Carlos Finley and Walter Read
The First Peer Review
Responses to
Yellow Fever
• A $30,000 bid (RFA) from the US Army for
Universities
• 1st peer-reviewed applications for research.
1879
Penicillin and
World War II
The Fundamental Tenets for NIH (1946)
1. The only possible source for adequate support of our medical
research is the taxing power of the federal government.
2. The federal government and politicians must assure complete
freedom for individual scientists in developing and conducting
their research work.
1940
•
President Roosevelt set up the National Defense Research
Committee.
Awarded contracts for rapid production projects
Identified 700 universities for future contracts
1942
• Medical Research funding grew from $2.3 million to $7.5
million, rating applications with an “A”, “B”, or “C”.
• 21 penicillin production plants (led to a 97% survival rate
for wounded soldiers.)
3. Reviews should be conducted by outside experts essentially
without compensation.
4. Program management and review functions should be
separated.
Surgeon General Thomas Parran, Jr.
Department of Health and Human Services
Total = $592 Billion
Total = $52.6
HRSA
11%
CDC
8%
The National Institutes of Health
Medicare
58%
Medicaid
33%
FDA 3%
Discretionary
Programs
9%
NIH
54%
`
Other 24 %
3
FY 2012 NIH Budget
Distribution of Funding
24 NIH Institutes and Centers Fund Grants
NINDSNIGMSNCRR NIA
NIAID
NIAAA
NIDCR
NEI
NIH Center For Scientific Review
NINR
NCI
NIEHS
NIDCD
NLM
NCCAM
Types of NIH Grants
• Research Project Grant
NIAMS
Center for
Scientific
Review
FIC
NIBIB NIDA
NIMH
NHLBI
NHGRI
NICHD
NIDDK
NCMHD
The Operating Principles of NIH Peer Review
NIH has ownership of the process
• The Scientific Review Officer , a full time federal employee,
nominates the review panel, assigns applications and is responsible
for the meeting
• Small Business Grants
• Training and Career Development Grants
• Exploratory/Development Grants
The study section (review panel) has ownership of the
science.
• Shared Instrumentation Grants
•
•
• Resource Grants
Is composed by experienced scientists in the field.
Hundreds of study sections reviewing different biomedical behavioral
science
4
This is CSR
The Drivers for Change
September 2009
1st Driver: The NIH Budget
2nd Driver: Number of Applications
120
40
100
35
Thousands
$, Billion
30
25
20
15
80
60
40
10
20
5
0
0
1998
2000
2002
2004
2006
2008
2010
2012
1996
70
12.00
60
6.00
Applications
Per Reviewer
4.00
2.00
$ Millions
14.00
8.00
2000
2002
2004
2006
2008
2010
4th Driver: CSR Budget
3rd Driver: Reviewer’s Load
10.00
1998
50
40
30
20
10
0
2004
2005
2006
2007
2008
2009
2010
2011
19
97
19
98
19
99
20
00
20
01
20
02
20
03
20
04
20
05
20
06
20
07
20
08
20
09
20
10
20
11
0.00
CSR
Reviewer Cost
Constant $
5
Annual Savings in Reviewers’ Expense Budget
• Sending application electronically
$3.8 million
• Non-refundable tickets with one possible
change
$15 million
• 3,000 fewer reviewers
$3 million
• 15% reviews using electronic platforms
$5 million
• One meeting a year on the West Coast
$1.8 million
5th Driver: The World Is Changing
• The way research is done has changed
• The diseases of Americans have changed
The Land of the Free, The Home of the Fries
Major Complaints About NIH Peer Review
• The process was too slow
Enhancing Peer Review
• There were not enough senior/experienced
reviewers
• The process favored predictable research instead of
significant, innovative, or transformative research
• The time and effort required to write and review
were a heavy burden on applicants and reviewers
2005
Enhancing Peer Review: The Overall Goal
1. Modernizing the Process
2. Realigning Study Sections, Divisions, IRG
3. Investing in the Future: Young Investigators
4. Funding the Most Promising Research Earlier
5. Focusing More on Impact and Significance
6. Attracting and Maintaining the Best Reviewers
7. Others
6
NIH Review Changes
The Process
Diagnostic
June 2007 – Feb. 2008
The Process
Design
Implementation
Plan
March 2008 – June 2008
Begin Phased
Implementation
of Selected
Actions
September 2008
•
Input from the community
•
Open Houses
•
PRAC (Peer Review Advisory Committee)
•
NIH Intramural and NIH IC Directors
•
Department Health and Human Services
•
OMB and White House
2
1: Processing Applications
Enhancing Peer Review
1. Modernizing the Process
2. Realigning Study Sections, CSR Divisions and
IRGs
3. Investing in the Future: Young Investigators
4. Reviewing Expeditiously
5. Focusing More on Impact and Significance
6. Attracting the Best Reviewers
7. Others
July 1, 2005
CSR: February
1, 2007
1. One Review Platform for 63 years
The First NIH Study Section
1946
60 years later
2005
1. Advancing Additional Review Platforms
• Electronic Reviews
Video Assisted Meeting
Internet Assisted Meeting
• Editorial Board Review
Complex Science
Small Business
Transformative R01
ARRA, IRS, FDA applications
7
1. The First Telepresence Study Section
Enhancing Peer Review
1. Modernizing the Process
2. Realigning Study Sections, CSR Divisions and
IRGs
3. Investing in the Future: Young Investigators
4. Reviewing Expeditiously
5. Focusing More on Impact and Significance
6. Attracting the Best Reviewers
7. Others
2. CSR Organization
Divisions
Integrated Review Groups
Scientific Review
Officers
2. Redesign Study Sections
Positional Map of Membrane Biology and Protein
Processing Study Section
2. Redesign Study Sections
Positional Map of Musculoskeletal Tissue
Engineering Study Section
Enhancing Peer Review
1. Modernizing the Process
2. Realigning Study Sections, CSR Divisions and
IRGs
3. Investing in the Future: Young Investigators
4. Reviewing Expeditiously
5. Focusing More on Impact and Significance
6. Attracting the Best Reviewers
7. Others
8
3. Projection of Age Distribution of NIH RPG
Investigators: 2020
3. Early Stage Career Investigators
7%
PIs in 1980
Definition of New Investigator:
• Not previously competed successfully as PD/PI for a significant
NIH independent research award.
6%
Percent of PIs
5%
Definition of Early Stage Investigator:
• Within 10 years of completing terminal research degree or within
10 years of completing medical residency (or the equivalent).
4%
3%
• The NIH corporate policy is to fund R01s of New
Investigators and ESIs at different paylines
•
• 3 Paylines for R01s Applies only to R01 applications
2%
1%
0%
25
30
35
40
45
50
55
60
65
70
75
80
85
90
Age
19
3. Funding New Investigators
Enhancing Peer Review
1. Modernizing the Process
2. Realigning Study Sections, CSR Divisions and
IRGs
3. Investing in the Future: Young Investigators
4. Funding the Most Promising Research Earlier
5. Focusing More on Impact and Significance
6. Attracting the Best Reviewers
7. Others
4. Funding the Best Research Earlier
4. Shortening the Review Time
• Abolish A2 applications
2005
2007
2009
0
2
4
6
Months: Submission to Posting Critiques
To enable resubmission, when doable and desirable, 4
months earlier than in the past.
9
4. Percent of R01 Awards by Amendment
4. Investigators’ time-to-award. New R01
applications, by year of original A0 application.
70%
Percent of Total Awards
60%
50%
40%
30%
20%
10%
0%
1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Enhancing Peer Review
1. Modernizing the Process
2. Realigning Study Sections, CSR Divisions and
IRGs
3. Investing in the Future: Young Investigators
5. Focusing More on Impact and Significance
• Shorter Applications
• Template Based Review
• Score on Impact and Significance
• Training of Reviewers
4. Reviewing Expeditiously
5. Focusing More on Impact and Significance
6. Attracting the Best Reviewers
7. Others
Enhancing Peer Review
6.Attracting the Best Reviewers
16000
1. Modernizing the Process
2. Realigning Study Sections, CSR Divisions and
IRGs
3. Investing in the Future: Young Investigators
14000
12000
10000
8000
6000
4. Reviewing Expeditiously
4000
5. Focusing More on Impact and Significance
2000
6. Attracting the Best Reviewers
7. Others
0
200020012002200320042005200620072008200920102011
Chartered
Temporary
10
6. Attracting the Best Reviewers
Academic Rank of ALL CSR Reviewers
6. Attracting the Best Reviewers
Some Successful Strategies
80%
•Move a meeting a year to the West Coast
70%
•Additional review platforms
60%
•Develop a national registry of volunteer reviewers
•Searchable database with 5,000 reviewers
50%
•Provide tangible rewards for reviewers
•No submission deadlines for chartered members
of study sections and frequent reviewers
40%
30%
20%
•Provide flexible time for reviewers
•Choice of 3 times/year for 4 years or
•2 times/year for 6 years
10%
0%
1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
PROFESSOR
ASSOCIATE PROFESSOR
ASSISTANT PROFESSOR
Enhancing Peer Review
7. Others
1. Modernizing the Process
1. Reviewing Transformative Research
2. Realigning Study Sections, CSR Divisions and
IRGs
2. Reviewing for ARRA (The Stimulus), FDA,
Treasury Department and Various Countries
3. Investing in the Future: Young Investigators
3. Changing Scoring
4. Reviewing Expeditiously
4. Changing the Order of Reviews
5. Focusing More on Impact and Significance
5. Training of Reviewers and SS Chairs
6. Attracting the Best Reviewers
7. Others
7. Training Study Section Chairs
Clouds at the Horizon
Clouds on the Horizon
• The Business Model of Many Universities and Medical
Schools is not sustainable
•
The NIH is Becoming a Faculty Employment Agency
• Intra and Extramural mandates and Bureaucracy are
Mushrooming
•
Too Many Institutes, not Aligned with Today’s Science
•
Leadership Unwilling to Make Strategic Decisions
11
Coronary Heart Disease
Age-Adjusted Death Rates in U.S.:
Actual (blue) vs. Expected (yellow)
National International Hope
Deaths per 100,000
500
400
300
200
100
0
1950
Average annual
investment
per American
~ 1,329,000 Projected Deaths
in 2000
~$3.70
815,000 Deaths
Prevented in 2000
World Video.mpg
~ 514,000 Actual Deaths
in 2000
1955 1960 1965 1970 1975 1980 1985 1990 1995 2000
Year
12