THE REVIEW PROCESS
(AND RESPONDING TO
REVIEWER FEEDBACK)
June 20, 2014
Rick
Gibbons
CENTER FOR SCIENTIFIC
REVIEW (CSR)
VS. NIH
REVIEWING @ CSR
Initial Review Group (aka “Study
Section;” SRG)
1. People
SRO
Chair (voting member)
Program Official* (with Institute)
Assigned Reviewers (primary, secondary, tertiary)
Voting Members (24 - ~42)
 *maybe
REVIEW PROCESS @ CSR
2. Process
 New / early investigators grouped
Initial scores presented by assigned
reviewers
Reviews presented in order
Discussion
Final vote by ARs
Others vote as well – stay within
parameters (hi/lo score); if not, why
FEEDBACK FROM CSR
3. Feedback
Overall summary from each reviewer
Criterion scores
Discussed?
Impact score
Percentile
Impact score vs. percentile
SCORES
CRITERION SCORES
WEIGHTING FOR EACH
CRITERION
PAYLINES
NIDA
Success
Payline (%ile)
1992
2013
40%
28
21%
8
NCI
Success
33%
Payline
22
“Zone of Consideration”:
20%
9
10 – 20%ile
SUCCESS: EXPERIENCED
SUCCESS: NEW
SHOULD I RESPOND?
 Yes.
 70% likelihood of better score
 In 2010: 3 times as likely to get funded
 11.5% success rate for initial vs. 35% for resubmission
 How?
 Appeal?
 … unlikely to succeed (and if it does, then back to
study section)
 Take a look at a different study section;
 25% of funding decisions would have been different
with a different panel (Johnson, 2008).
RESUBMISSIONS
OLD
NEW
A0
A0
A1 (w/intro)
A1
A2
A0 (no intro)
A1
(etc.)
RATINGS
Significance:
Investigator(s):
Innovation:
Approach:
Environment:
Reviewer 1
Reviewer 3
•
•
•
•
•
•
•
•
•
•
5
1
6
6
1
1
1
1
2
1
RESUME AND SUMMARY OF
DISCUSSION
In this very strong application, the investigators propose
collecting a seventh wave of data from participants in the
Family and Community Health Study (FACHS). This study will
expand the original study in several important ways to examine
how early stress experiences and cumulative stress predict
continued substance use and risky health behaviors for African
American young adults through their late 20s. The investigators
have been mostly responsive to previous critiques. There was a
divergence of opinion among reviewers about the potential
overall impact of this study.
RESUME AND SUMMARY OF
DISCUSSION (CONT.)
While most reviewers indicate the application is likely to
have an extraordinarily high impact on the field and larger
social world, a minority of reviewers were unsure about the
impact due to limitations of the methodology. The strengths of
the application include the broad significance of the work,
outstanding and productive research team, use of high quality
longitudinal data with excellent retention, addition of biomarker
data and biological indicators of substance use, and the
addition of community level data and a White comparison
sample. The application is weakened by some limitations in the
approach and an ambitious and underdeveloped analysis plan.
Overall, the strengths of the application outweigh the
weaknesses.
CRITIQUE 1: OVERALL IMPACT
This is important work, and the team is well positioned to
conduct it. In the resubmission, the investigators manifest a
serious commitment to addressing prior weaknesses. They clearly
thought through their options, and arrived at a reasonable
solution to the absence of data on White people and the addition
of team members with quantitative expertise. These are laudable
changes. The resubmission, however, fails to provide a persuasive
argument on why Wave 7 is necessary. Questions remain over the
comparability and viability of the White sample. Some issues
raised in the previous review have gone unaddressed. Too much
material is devoted to early sections of the application at the
expense of later, more critical ones. The absence of clearly
delineated changes to the resubmission makes finding the
improvements difficult.
CRITIQUE 3: OVERALL IMPACT
In this application , the investigator s propose to conduct a seventh
wave of data in the Family and Community Health Study (FACHS). The ongoing
study has followed samples of African American children in Iowa and Georgia
star ting at age 10 onward. The initial sample represented nearly 900
families. To date, the project has examined an array of factor s predicting
health risk behavior s (e.g., substance use) and health outcomes as well as
social cognitive predictor s of these (par ticularly perceive d racial
discrimination), not only in the target children but their parents and siblings.
The study has documente d the cross -over ef fect, indication that African
Americans begin to use substances later than do Whites, but they also are
less likely to mature out of substance use. This next wave of data collection
would allow the investigator s to test this cross -over ef fect in the best way
possible: with data that track the par ticipants into early adulthood. Moreover,
the propose d work expands the original study in 6 impor tant ways, not only
simply moving for ward with the initial study goals (which themselves remain
impor tant) but also in expanding to include 1) a White comparis on sample, 2)
biomarker data for genetic analyses, 3) biological indicator s of substance
use, and 4)additional community level data relevant to the outcomes of
interest. The propose d data will also account for age -related changes in the
lives of the target par ticipants, incorporating 5) romantic par tner repor ts and
6) data about the target children in parental roles as many of the original
targets are now parents.
CRITIQUE 3: OVERALL IMPACT (CONT.)
The project is likely to have an extraordinarily high impact on the
field as well as the larger social world. Under standing the ef fects of
perceived racial discrimination on health and wellness in minority groups
are an impor tant policy problem and the investigator s have taken pains to
tie the science here to the real world context of their par ticipants. The
proposed work is immensely expansive in its scope and promises to
provide the kind of evidence necessar y to under stand the problems
addressed as well as to guide inter ventions. The major strengths of the
application include the ambitious research plan, the exper tise of the
collection of scholar s who will carr y it out, he proven track record of the
research team, and the ver y high broad significance of the work . The key
weaknesses are best viewed as imper fections or limitations. Cer tainly,
although similar to the original FACHS samples in some ways, the Oregon
sample is likely to dif fer in ways beyond race/ethnicity. It is clear that the
investigator s are aware of these issues and will make the best use of
these data. In addition, some aspects of the application remain a bit
underdeveloped, such as the integration of statistical procedures that
account for interdependence in the dataset. Impor tantly, the team that is
in place suggests that these limitations will be addressed by, perhaps, the
ver y best scholar s in the field.
1. SIGNIFICANCE
 Strengths:
 Excellent progress report; very productive group.
 Questions about health disparities are sorely in need of better data.
This application promises to continue to generate those data.
 Weaknesses:
 Questions about what an additional wave of data collection will
bring remaining unpersuasively answered.
2. Investigator(s)
• Strengths:
▫ Outstanding team.
▫ Very productive in executing this study thus far.
▫ Investigators are uniquely suited to continue the research they have
been doing over the past several years.
▫ Adding Dr. Kenny to the team gives added strength.
• Weaknesses:
▫ None noted.
3. APPROACH
 S t r e n g th s :
 The resubmission represents an earnest attempt to address weaknesses in the
prior submission.
 More detail on measures and data analysis is helpful.
 We a k n e s s e s :
 The absence of delineations for text revisions is regrettable. Locating the many
improvements to the application was a challenge.
 Heavy use of italicized words and phrases further inhibited the task of discerning
changes. Though some resubmitted applications use italics to indicate changes,
apparently this application employs italics to underscore particularly salient
points. Excessive italicization, however, may have served to obscure, rather than
to clarify what is special about the planned work.
 The potential lack of comparability between the FACHS and OYSUP samples is
worrisome. That both studies are up for renewed funding is another concern. If
OYSUP is not awarded, how does this affect FACHS?
 Timeline is too sketchy. This is particularly troublesome owing to the absence of a
timeline being cited in the last review.
 Too much space is taken up with background material and the progress report, at
the expense of procedure. The absence of detail on procedure was cited in the
prior review.
 In the prior review, issues about heterogeneity and interdependence were noted.
4. INNOVATION
 Strengths:
 The addition of more detail on innovation
demonstrated the team’s intent to respond to the
prior review.
 Weaknesses:
 Notwithstanding arguments put forth in the
applications, innovations seem minor.
5. Environment
• Strengths:
▫ Excellent settings.
• Weaknesses:
▫ None noted.
1. SIGNIFICANCE
 Strengths:
 Understanding racial differences in substance use and abuse is an
extremely important scientific and social goal.
 Understanding how perceived racial discrimination influence health
risk behavior is an important goal for social science and policy
makers.
 The breadth of the data collected, including self and other ratings as
well as biological measures of substance use and genetic markers,
suggests that this work will influence a broad range of fields.
 The inclusion of this next wave of data collection will allow for a true
test of the processes identified thus far in the earlier waves. Such
data are needed to fully understand the cross-over effect.
 The capacity to follow the original participants into their young adult
lives, as romantic partners and parents surely enhances the
significance of the work.
 Weaknesses:
 The inclusion of the White (Oregon) sample without very specific
plans to deal with the differences that likely exist between this
2. INVESTIGATOR(S)
 Strengths:
 The team itself is a clear strength of the application. Principal
Investigators Gibbons and Gerrard are productive scholars who have
made important contributions to the scholarly literature of direct
relevance to the proposed project.
 A large number of Co-Investigators and consultants with direct
expertise specific to the FACHS data and/or the specific complex
issues included in this wave of data collection, including HIV
prevention (Fisher), genetic epidemiological research and GXE effects
(Todorov), tobacco use and prevention (Wills), stress, discrimination
and genes and biological processes in African American samples
(Brody, Cutrona) analyzing biomaterials (Philibert), and the complex
quantitative task that is implied in these rich data ( Todorov, Kenny).
 The inclusion of Co-Investigators Andrews and Hampson from ORI to
collaborate on the integration of the Oregon dataset with the FACHS
samples.
 Weaknesses:
 None noted.
3. INNOVATION
 Strengths:
 The FACHS is innovative in that it focused initially on Black
participants who did not reside solely in urban environments.
 The inclusion of biological as well as psychological measures.
 Adding community level variables.
 Applying more sophisticated quantitative analytical tools to these
complex data.
 Including genetic markers to fully contextualize the psychosocial
variables and processes.
 Adding bio-measures of substance use.
 Examining, now, multigenerational processes.
 Incorporating a White comparison sample.
 Weaknesses:
 None noted.
4. APPROACH
 Strengths:
 Relatively large samples, studied longitudinally.
 The very rich data already collected about these individuals.
 The fact that, on the variables of interest, these is indeed enough
variability to fully explore the central aims of the project.
 The addition of a White/comparison sample.
 The ambitious range of data to be collected about these individuals,
including not only self-reports and other (i.e. parent, romantic
partner, sibling) reports but also, now, genetic measures, bio measures for substance use, and additional measures pertaining to
the community level aspects of the participants’ lives.
 Weaknesses:
 Although the inclusion of the Oregon sample is an innovative step, it
is possible that the differences between the samples (beyond,
race/ethnicity) will be difficult to surmount.
 This is clearly a massive undertaking and it is possible that the plan
is over ambitious.
5. ENVIRONMENT
 Strengths:
 Adequate facilities and resources are available for this research.
 Weaknesses:
 None noted.