Revising and Resubmitting:
Practical Considerations Based
on the Psychology of ReReviews
Marc I. Rosen, M.D.
Most Successful Grants are
Revise and Resubmits
Initial or
Revised?
Number
Applications
Success Rate
Initial
Submission
19,259
8.6%
Revise and
Resubmit
5,373
37.2%
http://www.report.nih.gov/success_rates/index.aspx
2012 data for new R01s
Revising and Resubmitting:
Lecture Structure
 Deciding Whether to Revise and Resubmit
 Suggestions for Revising and Resubmitting
 Example
 Moral
Review Group Actions
 Discussed
and Scored with Impact Rating
(score and percentile)
 Triaged/Not
Discussed with no Impact
Rating but criteria scores (lower half)
 Deferral; Not
Recommended; Abstention
Deciding Whether to
Resubmit: Consider the Score
 Triaged/Not



Initial decision to triage at meeting rarely
overturned at meeting
Hard to completely convert a critic to a gungho booster
Committee is busy, focuses on close calls
 2010-2012




Discussed: How decided?
R01s
2.3% of unscored new R01s funded on resubmit
8.7% of unscored continuation R01s funded
Includes those not resubmitted
http://report.nih.gov/FileLink.aspx?rid=880
Reading the Critiques
•
Read critiques carefully and calmly
• Even if you are angry
• Assume you got a good-faith, intelligent review
Let colleagues and mentors read the
reviews for reality testing, support, and
input
• Give more weight to comments that
•
• Are in the “Summary”
• Are made by more than one reviewer
Deciding Whether to
Revise and Resubmit: Get
More Information
Contact Program Officer
-Ask about study section discussion
-Ask about NIH Institute
interest in area?
advice?
Talk through reviews with co-investigators
and peers
Deciding Whether to
Resubmit: Keep Perspective
• Reviewers assess your submitted material
• Reviewers are never totally wrong or right
• Extremely competitive process:
• Resubmission is common
• Avoid WYSIATI (what you see is all there
is)---other talented people out there
Deciding Whether to Revise and
Resubmit: Prospect Theory
Sunken Cost Fallacy
• Staying to the end of a boring movie hoping to
recoup loss of spent money
• Using a fitness plan even when it’s painful
• It’s a fallacy
Loss aversion: It’s not a rejection if you don’t give
up
Thinking Fast and Slow (Daniel Kahneman)
Deciding Whether to Revise
and Resubmit: Psychology
Overconfidence: Excessive Optimism
•
•
•
•
Only 5% of U. Chicago MBA students predict they
are in bottom 50%; most predict second decile
90% of drivers think they are above average
Entrepreneurs say success rate for new business is
50% but predict personal success rate of 100%
Few newlyweds expect to be among 50% who
eventually divorce
Nudge, Thaler and Sunstein
Deciding Whether to Revise and
Resubmit: Overconfidence?
Test of Overconfidence: 90% Confidence
Interval for
• Weight of earth in tons
• 6.0 X 1024
• Percentage of world’s population who
are Native English speakers (per CIA
World Factbook 2009)
• 4.83%
Reviews that Should NOT
Make You Overconfident





“This grant addresses an important topic”
“Yale has superb facilities for this research”
“The investigator is qualified”
Only the first reviewer was critical of the
application and the grant was un-scored/poor
score
Mild praise and the grant was un-scored/poor
score
Meta-Critiques that May
Not be Answerable
 ”There
are already a lot of grants in this
area”
 “Not innovative”
 “Not significant”
 “Not exportable”
Consider Alternatives
A
smaller grant (R21 instead of R01)
 Another funding agency
 A substantial change that you can submit
as a new grant
◦ If you can answer critiques  revise and resubmit
◦ If you cannot answer critiques new grant
Deciding Whether to
Revise and Resubmit
• Do
you have something better to work on
for two-plus months?
Revising and Resubmitting:
Lecture Structure
 Deciding Whether
to Revise and
Resubmit
 Suggestions for Revising and
Resubmitting
 Examples
 Moral
Consider the Person Behind
the Review
◦ Look into research interests of people on
the committee
◦ NIH REPORTER search of what
committee has funded in the past
◦ Talk to people who have been on the
committee
◦ Talk to your project officer
Consider the Person Behind
the Review
◦ Reviewers want to avoid cognitive dissonance
 Cognitive dissonance
 Inner drive to hold our attitudes and beliefs in harmony
 Drive to avoid dissonance between them
 Examples
 The Fox and the Sour Grapes
 The review group that found fault with your grant
◦ So, don’t say the reviewer was wrong
Consider the Person Behind
the Review
◦ Even Reviewers Who Change Their Minds Impacted by
Anchoring to Prior Score
 Roulette Wheel Study of Anchoring 1:
Volunteers shown rigged roulette wheel that stops
at 10 or 65
What % of countries in the U.N. are in Africa?
Wheel stops at 10---------25% average
Wheel stops at 65---------45% average
 http://en.wikipedia.org/wiki/Anchoring; Tversky and Kahneman, 1974
Consider the Reviewer’s
Perspective
The reviewer who likes your application
has to justify your response to a
committee of 12+ very smart people
Make your response easy-to-follow
Use tables for complicated concepts
(the reviewer can say to the
committee—“He’s got a table laying
that out.”)
Consider the Reviewer’s
Perspective
• If the reviewer likes the application, he/she
is more likely to agree with your
justifications
• How juries decide:
• They do not weigh the evidence
• They do arrive at a narrative that appears to
fit the data
• Answer meta-critiques
Revising and Resubmitting:
Content Issues
Address any grant-killing meta-questions
early in your response emphatically and
clearly
Examples of Meta-Critiques
Critique:
“They’ll never be able to pull this off – the
project is not feasible.”
Answers:
-Pilot data
-Bring in collaborators who can pull it off
Examples of Meta-Critiques
Critique:
“This was written by a slob who just does not
know this topic well.”
Answers:
-Emphasize how much the application has
been cleaned up
-Consider adding expert who would have
caught all your mistakes the first time.
Examples of Meta-Critiques
Critique:
“This was written by Kathy Carroll’s
(Stephanie O’Malley’s, Rajita Sinha’s, Marc
Potenza’s…) go-fer and is not really an
independent application.”
Answer:
-Spell out what is yours and what is not
Examples of Meta-Critiques
Critique:
“[zzzz’s] always make a hash out of [yyyy]
research– it requires someone in my field of
specialization. “
“The application would be strengthened by
the involvement of a biostatistician.”
Answer:
-Include someone with the recommended
expertise
Examples of Meta-Critiques
Critique:
”The study design is from hunger.”
Answer:
-Table and/or figure justifying and explaining
the study design
Revising and Resubmitting:
Process Issues

Respond constructively and positively
◦ The reviewer is always right (even if not).
No more than 1-2 areas of disagreement, but
justify decision thoughtfully and respectfully
 If not ready to submit at next deadline, DON’T

◦ Reviewers generally need a reason to improve your
score
Revising and Resubmitting:
General Content Issues
 “Thank
you for the careful review of our proposal
to […] We appreciate the careful, critiques”
 Indicate method of highlighting changes (e.g., bold,
italics in text)
 Main criticism and response
 List more minor criticisms (in italics) and
responses
 “Thank you for reconsidering our application”
Revising and Resubmitting:
The Best Responses
 Pilot
Data
 Re-analysis of your own data
 Literature
Revising and Resubmitting:
Weaker Responses
◦ Logic
◦ Your opinion
◦ “In my clinical experience…”
Revising and Resubmitting:
Don’t, Don’t, Don’t





(Usually) don’t answer questions that were not raised
Don’t malign the review process or the reviewer
Don’t spend much effort pointing out that one
reviewer liked what another reviewer critiqued
If the reviewer asks for something that was already in
the application, be humble, e.g. “The information is
presented more clearly this time in the methods as
follows…”
Don’t get personal (no jokes, personal opinions, etc.)
Revising and Resubmitting:
Don’t, Don’t, Don’t
•
Don’t repeat every critical word from a
review
• Summarize criticisms (it was bad enough the first
time)
•
Don’t over-answer minor criticisms by
writing a long essay that makes the criticism
seem more major than it is
Revising and Resubmitting:
Lecture Structure
 Deciding Whether
to Revise and
Resubmit
 Suggestions for Revising and Resubmitting
 Example
 Moral
Example of Grant Review
4-year clinical trial to test computerdelivered counseling to improve
engagement in work
Grant Review
•
•
•
Lousy score of 270, 67th percentile
Program officer tells me they liked it, wanted
to see it back
Reviewer response:
• Reviewer one liked
• Reviewer two mixed
• Reviewer three (statistician) gave it terrible
score
Reviewer’s Potentially GrantKilling Responses
“However, no data exists whether veterans
would actually use the intervention.”
Summary Statement Recommends
“Further conceptually develop and pilot test
the internet-based intervention. Provide that
data as a part of the proposal.”
Planned Response
•
Agree with everything reviewers say and
propose three-year, pilot-type, therapy
development study to address it
Revising and Resubmitting:
Lecture Structure
 Deciding Whether to Revise and Resubmit
 Suggestions for Revising and Resubmitting
 Example
 Moral
Morals
 It
helps to enjoy the process
◦ Doing your best
◦ Advocating for something you
believe in
◦ Promoting yourself
 Your CV lists grants
◦ No lasting harm from unfunded
application
Moral

“At the length, truth will out”
◦ Shakespeare (Merchant of Venice)

“In the long run, we are all dead.”
◦ John Maynard Keynes
Thank you