Ming Tai‐Seale, PhD, MPH
Senior Investigator
Palo Alto Medical Foundation Research Institute
Professor
Texas A&M Health Science Center
g
Agenda
 JIT
 Priority Score and Percentile
P i it S
d P
til
 Summary Statement
 Working with Program Staff
 Revise and Resubmit
JIT = Just‐In‐Time
 NIH
 A call to IMMEDIATE action
A ll t IMMEDIATE ti




Immediately contact the project officer
Get IRB approval
Budget Respo d to ev e o p og a o ce to use advocacy
Respond to review for program officer to use in advocacy
 AHRQ
 Do nothing 280
130
157,, 18th %
190
257
217
118
198
5
The New Scoring System
Reviewers will see new drop‐down menus in IAR for entering scores for each criterion
drop down
drop‐down
7
Scoring Process
Score Terminology
IAR
g -Submit
Before Meeting
Critique Screen
Rev 1
Rev 2
Rev 3
Peer
Review
(REV )
IAR
g After Meeting
Final Score Screen
Rev 1
Rev 2
Rev 3
Rev
Rev11
Rev 2
Rev n
3
Rev
2 … Rev
Significance
Approach
EDITABLE
AFTER THE
MEETING
Innovation
Investigator (s)
Environment
Preliminary
Impact / overall
Score
Average Preliminary
Score
((for streamlining
g )
Final
Score
B OTH ASSIGNED AND UNASSIGNED
REVIEWERS
CAN PROVIDE CRITERION SCORES ;
B OTH ASSIGNED AND UNASSIGNED REVIEWERS CAN
PROVIDE PRELIMINARY SCORES ;
EVERY PARTICIPANT ( EXCEPT M AIL REVIEWERS ) CAN
PROVIDE FINAL SCORES
Priority Score (avg x 10 )
and Percentile if
applicable
7–9
4–6
1–3
Respond to Review before Funding
Respond to Review before Funding Decision
g
g
Program Staff Meeting
The Council
Grateful to Dr. Will Manning for Some of the Slides
13
p
g
y
Responding to Study Section's Concerns
 Expect a second submission
 Very few proposals funded first time unless oneone
shot announcement
 Prepare for a critical review
14
Study Section Members
• Disciplines
–
–
–
–
–
–
–
–
–
–
 Methodological Anthropology Biostatistics
Economics
Epidemiology Health services research
Medicine Nursing
Organizational Theory
S i l
Sociology
Social work
O i t ti
Orientations
 Quantitative
 Qualitative
 Mixed
15
Facts about Review and Reviewers
 Reviewers may not be from same discipline or
specialty
p
y
 Check the section roster, then PubMed/Google
 Many reviewers will have trouble with jargon
 Your technical terminology is unintelligable jargon to
me unless explained
 "Collective
"C ll i efficacy"
ffi
" means what?
h ?
 "Diff-n-diff" means what?
 HSR is multidisciplinary
 Unless the study section has many from your
discipline you must talk to a wider audience
discipline,
16
Facts about Review & Reviewers (cont'd)
(cont d)
 Basic Reviewing Principle
 Burden of proof of idea and approach is on
the proposer
 If they cannot find what they need easily,
they often will treat it as missing or
inadequately described
 Reviewer's language is often very frank!
 But remember reviewers are:
 Looking for problems
 Trying to help, esp. if they see some merit in
proposal
17
Facts about Review & Reviewers (cont'd)
(cont d)
 Reviewers
 doing the review "cold“
cold
 working under pressure of short deadline while
still teaching, seeing patients, writing their own
proposals, …
 do not have benefit of lengthy discussions with
research team
 do not have time to read the proposal over and
over again
I iti l R
Initial
Response tto S
Summary
Statements
 Sulking is normal
 Do
D not take
k iit personally
ll
 Get advice from agency staff ASAP
 GET SENIOR HELP TO INTERPRET COMMENTS
 Plan to resubmit unless "fatally
fatally flawed"
flawed
19
Responding to Reviews
 Leave plenty of time to
 Overhaul
O h l in
i response to
t study
t d section
ti
and
d agency
staff comments
 Solicit reaction to revised submission from cold
reviewer, preferably with study section or area
experience
 Respond accordingly
 Do
D not rush
h to resubmit
b i (J
(July
l 1
1st after
f
receiving
i i
June 6th)
20
g the Proposal
p
Revising
 Outrage is OK for a personal reaction but never in a
response!
p
 Take the feedback seriously as indicative of
 Gaps in exposition or logic
 Overly terse in key areas
 Organizational issues
 Id
Identify
tif common themes
th
across reviewers
i
 Respond to all concerns in "Response" as well as in
text
 Thank the reviewers for their valuable comments
 Apologize
po og e for
o inadequacy
adequacy o
of …
21
g the Proposal
p
(
)
Revising
(cont'd)
 Even if you are "right," clean up the exposition to
make the logic
g more transparent
p
 Revise the whole proposal if needed
 Ask research team
team, senior colleague
colleague, cold reviewer
for reaction
 Revise again
 For substance
 For ease of re
reviewers
ie ers to e
evaluate
al ate
22
Common problems
• Specific Aims
– Not specific
– Largely rhetorical
– Process aims rather than hypothesis-driven
• Significance
g
– Lack clear focus
• Innovations
– Not truly innovative
 Low Impact
• Career development plan (for K-series)
– Lack specificity
– Lack linkage with research career goal
23
Common p
problems ((cont'd))
 Research design and methods lack
 Conceptual framework -- A flow chart is useful but
inadequate
 Alternative approaches
pp
not considered
 Weak or no link to hypotheses
 Research design flaws or weak design
 Failure to address reliability and validity of data
 Questions of internal or external validity
 Wrong sample for research question, esp. if using a
convenient sample you happen to have
24
C
bl
((cont'd)
t'd)
Common
problems
• Not feasible as constituted
• Budget too large or too small
• Budget
B d t llacks
k adequate
d
t jjustification
tifi ti
ffor
– Length of grant
– Level of effort
• Scope too broad
• Research team lacks appropriate qualifications
– Substantial data analysis but no
statistician/econometrician
– Economic analysis but no economist
– Large clinical component but no clinician
– Qualitative component without card-carrying
qualitative researcher
q
25
Common p
problems ((cont'd))
 PI is too junior, not enough senior involvement
 Project full of 5 percenters
 Lacks
k sense off own llimitations
 Errors or omissions in human subjects,
subjects minority
minority,
gender areas
 Lacking AHRQ priority populations
26
g
g
Working with Program Staff
What Can You Expect From
 Project officers
P j ffi

Read your concept paper and draft 


Send it in EARLY!
Interpret the fundability of your priority score Guide you in responding to review / resubmission
 Scientific Review Officer (SRO) 
Assign reviewers who may have expertise to review your proposal