NIH IS CHANGING/ENHANCING THE
REVIEW CRITERIA AND FOCUSING
ON MISSING OPPORTUNITIES TO
EXAMINE SEX DIFFERENCES
JOURNALS ARE CHANGING/ENHANCING THE REVIEW CRITERIA
Jeffrey W. Elias, PhD
UCD SOM Office of Research
Grants Facilitation
http://www.ucdmc.ucdavis.edu/medresearch/grant_facilitation.html
Sex Differences are part of Reproducibility
Issues in Scientific Studies
Changes in the NIH 2016 submission/review cycle
1) NIH enhances reproducibility standards for
application science and review – primary issues
of bias in design (to include exclusion of sex
differences) and bias in reporting data.
2) Focus on enhancing animal research – to include
sex differences
3) Journal standards for enhancing reproducibility
– less obvious emphasis on sex differences.
Sex Differences are part of Reproducibility
Issues in Scientific Studies
• Sex refers to a set of biological attributes in
humans and animals. It is primarily associated
with physical and physiological features
including sex chromosomes, gene expression,
hormone levels and function, and
reproductive/sexual anatomy. Sex is usually
categorized as female or male but there is
variation in the biological attributes that
comprise sex and how those attributes are
expressed.
Sex Differences are part of Reproducibility
Issues in Scientific Studies
• Gender refers to the socially constructed roles, behaviors,
expressions and identities of girls, women, boys, men,
and gender diverse people. It influences how people
perceive themselves and each other, how they act and
interact, and the distribution of power and resources in
society. Gender is usually conceptualized as a binary
(girl/woman and boy/man) yet there is considerable
diversity in how individuals and groups understand,
experience and express it.
Addressing Reproducibility Issues to
Include Sex Differences
BIAS: From a personal perspective as a researcher, journal
reviewer, study section reviewer, and journal editor the
most frequent source of bias is a hypothesis:
1) hypotheses often ignore sources of variance that
should be known or explored (i.e., “lazy” hypothesizing).
We have been lazy about sex differences.
2) investigators try to fit the data to the hypotheses as
proposed and do not follow the results, which is a form of
biased reporting.
Addressing Reproducibility Issues to
Include Sex Differences
• Funding Opportunity
• PA-16-066, Administrative Supplements for
Research on Sex/Gender Differences
• The Office of Research on Women’s Health
(ORWH) Most institutes have signed on to PA
• Due March 4 – non-competing –
administrative supplements 100K
• Must have 18 months remaining on parent
grant
Addressing Reproducibility Issues to
Include Sex Differences
• PA-16-066, Administrative Supplements for Research
on Sex/Gender Differences
• “In 1993, the NIH Revitalization Act required the
inclusion of women in NIH-funded clinical research.”
(from supplement on sex differences)
• “There has not been a corresponding revolution in
experimental design and analyses in cell and animal
research — despite multiple calls to action.
Publications often continue to neglect sex-based
considerations and analyses in preclinical studies.
Addressing Reproducibility Issues to
Include Sex Differences
• PA-16-066, Administrative Supplements for
Research on Sex/Gender Differences
• “The NIH plans to address the issue of sex and
gender inclusion across biomedical research
multi-dimensionally — through program
oversight, review and policy, as well as through
collaboration with stakeholders including
publishers. This move is essential, potentially very
powerful and need not be difficult or costly.”
Addressing Reproducibility Issues to
Include Sex Differences
M. Parikh, (2015), Subject selection bias in animal studies,
Current Science, Vol. 109, No. 4. 25 August.
• Stroke events are more prevalent in females
compared to males, but female animals are used in
only 38% of studies.
• Hypertension is found to be equally present in both
sexes, but 65% of scientific reports have males as
the study subjects.
• Male biases were found in 8 out of 10 biological
fields.
Addressing Reproducibility Issues to
Include Sex Differences
M. Parikh, (2015), Current Science ……….
• 80% of the studies in surgical journals used male
animals
• Women are 1.5 times more susceptible to pain
than men and women show higher adverse
drug reactions, but still studies use males exclusively,
which resulted in withdrawal of 8 out of 10
prescription drugs from the US market as side effects
manifested differently in both sexes
..
Addressing Reproducibility Issues to
Include Sex Differences
Can this effort be successful?
• Canadian Institute of Health Research (CIHR)
requires its applicants to answer mandatory
questions on sex and gender; unlike the NIH, it
does not mandate the inclusion of specific
populations in research designs, nor does it
have corresponding peer review criteria on
sex and gender
Addressing Reproducibility Issues to
Include Sex Differences
Can this be successful?
• In recognition of the important influences of sex
and gender on health, in December 2010 CIHR
made a change to its grant application forms,
requiring that all applicants respond to two
questions: Are sex (biological) considerations
taken into account in this study? Are gender
(socio-cultural) considerations taken into account
in this study?
Addressing Reproducibility Issues to
Include Sex Differences
Canadian Institute of Health & Research
motivations for change.
• With ratios of male-only against female-only
studies the ratios were: 5.5 in neuroscience,
5.0 in pharmacology and 3.7 in physiology.
• 75% articles in reputed immunology journals
did not indicate the sex of animals used in the
study.
Figure 2. Applicant by research area, competition and responses to sex and gender items.
Johnson J, Sharman Z, Vissandjée B, Stewart DE (2014) Does a Change in Health Research Funding Policy Related to the Integration of Sex and
Gender Have an Impact?. PLoS ONE 9(6): e99900. doi:10.1371/journal.pone.0099900
http://journals.plos.org/plosone/article?id=info:doi/10.1371/journal.pone.0099900
Suggestions for approaching the enhancement
of rigor and sex differences mandated by NIH
READ:
1) NOT-OD-15-102 for further consideration of NIH
expectations about sex as a biological variable.
2) General FAQs from NIH found at
http://grants.nih.gov/reproducibility/faqs.htm
3)Most recent article by Clayton & Collins for tips
on developing sex differences approach. Clayton, J. &
Collins F. (2014), NIH to balance sex in cell and animal
studies .Nature, Vol. 509, | 15 MAY 2014, 282-283.
Suggestions for approaching the enhancement
of rigor and sex differences mandated by NIH
Read:
• Landis et al (2012) A call for transparent
reporting to optimize the predictive value of
preclinical research. Nature, Oct
11;490(7419):187-91. doi:
10.1038/nature11556.
Suggestions for approaching the enhancement
of rigor and sex differences mandated by NIH
Pay attention to:
By November 25, 2015 application guide instructions will be
updated to include the following additional guidance for.
Significance
Describe the scientific premise for the proposed project,
including consideration of the strengths and weaknesses of
published research or preliminary data crucial to the support
of your application.
Approach
Describe the experimental design and methods proposed and
how they will achieve robust and unbiased results.
Suggestions for approaching the enhancement
of rigor and sex differences mandated by NIH
Pay attention to:
Using too subtle an approach to enhanced review
issues – “But I did mention it on page 102!” Make
reviewers pay attention.
Be clear if you are developing hypotheses (e.g., sex
differences) or testing hypotheses.
Don’t be lazy in literature review.
Suggestions for approaching the enhancement
of rigor and sex differences mandated by NIH
Pay attention to:
adding sex as a factor in statistical design.
• There are seven effects involved in a threeway factorial
• Three main effects (one for each independent
variable)
• Three 2-way interactions
• One 3-way interaction
Suggestions for approaching the enhancement
of rigor and sex differences mandated by NIH
Pay attention to:
adding sex as a factor in a statistical design.
• There are 15 effects involved in a four-way design
• Four main effects (one for each independent
variable)
• Six 2-way interactions
• Four 3-way interactions
• One 4-way interaction
Suggestions for approaching the enhancement
of rigor and sex differences mandated by NIH
Pay Attention To:
Major concern for NIH grants and clinical trials - the focus
is on detail of methods.
No additional application pages are allotted.
Reviewers are refocused before the first round of reviews
after January 16, 2016.
Work in progress for reviewers and applicants.
Landis et al (2012) NINDS
• Nature. Author manuscript; available in PMC
2013 Apr 11.
• Published in final edited form as:
• Nature. 2012 Oct 11; 490(7419): 187–191.
• doi: 10.1038/nature11556
• PMCID: PMC3511845
• NIHMSID: NIHMS422888
• A call for transparent reporting to optimize the
predictive value of preclinical research
Landis – Enhancing Animal Research
• The main workshop recommendation is that
at a minimum studies should report on
sample-size estimation, whether and how
animals were randomized, whether
investigators were blind to the treatment, and
the handling of data.
Landis – Enhancing Animal Research
• Standards
Encourage the use of community-based standards
(such as nomenclature standards and reporting
standards like ARRIVE), where applicable.
• Replicates
Require that investigators report how often each
experiment was performed and whether the results
were substantiated by repetition under a range of
conditions. Sufficient information about sample
collection must be provided to distinguish between
independent biological data points and technical
replicates.
Landis – Enhancing Animal Research
• Core set of standards for rigorous reporting
of study design (Adapted from Landis et al.)
• Include these reporting standards in
Information for Authors or other public place.
Require authors to fill out a checklist, ideally
upon submission, to state where the required
information is located in the manuscript.
Landis – Enhancing Animal Research
• Blinding
Require authors to state whether experimenters
were blind to group assignment and outcome
assessment, at a minimum for all animal
experiments.
• Sample-size estimation
Require authors to state whether an appropriate
sample size was computed when the study was
being designed and include the statistical method
of computation. If no power analysis was used,
include how the sample size was determined.
Landis – Enhancing Animal Research
• Inclusion and exclusion criteria
Require authors to clearly state the criteria
that were used for exclusion of any data or
subjects. Include any similar experimental
results that were omitted from the reporting
for any reason, especially if the results do not
support the main findings of the study.
Describe any outcomes or conditions that
were measured or used and are not reported
in the results section.
Landis – Enhancing Animal Research
• Statistics
Require that statistics be fully reported in the
paper, including the statistical test used, exact
value of N, definition of center, dispersion and
precision measures (e.g., mean, median, SD, SEM,
confidence intervals)
• Randomization
Require authors to state whether the samples
were randomized and specify method of
randomization, at a minimum for all animal
experiments.
Reproducibility Journal Standards
• Journals should have no limit or generous limits
on the length of methods sections (including
online options), while at the same time
encouraging efficient and clear presentation to
ensure a thorough examination by reviewers.
• Journals should use a checklist during editorial
processing to ensure the reporting of key
methodological and analytical information to
reviewers and readers. (A proposed set of key
information is listed below).
Reproducibility: Proposed Journal
Standards
• Rigorous statistical analysis
• A section outlining the journal’s policies for
statistical analysis should be included in the
Information for Authors, and the journal
should have a mechanism to check the
statistical accuracy of submissions.
Reproducibility Journal Standards
• Data and material sharing
• Stipulate, at the minimum, that all datasets on which
the conclusions of the paper rely must be made
available upon request (where ethically appropriate)
during consideration of the manuscript (by editors and
reviewers) and upon reasonable request immediately
upon publication.
• Recommend deposition of datasets in public
repositories, where available. Datasets in repositories
should be bidirectionally linked to the published article
in a way that ensures proper attribution of data
production.
Reproducibility Journal Standards
• Encourage presentation of all other data
values in machine readable format in the
paper or its supplementary information.
Require materials sharing after publication.
• Encourage sharing of software and require at
the minimum a statement in the manuscript
describing if software is available and how it
can be obtained.
Reproducibility Journal Standards
• Consideration of refutations
• Have a policy stating that if the journal
publishes a paper, it assumes responsibility to
consider publication of refutations of that
paper, according to its usual standards of
quality.
Reproducibility Journal Standards
• Consider establishing best practice guidelines for:
• Image based data (image screening for manipulation,
Western blots, for example)
• Description of biological material with enough information
to uniquely identify the reagents (for example unique
accession number in repository), in particular for:
• antibodies: also report source, characteristics, dilutions and
how they were validated
• cell lines: also report source, authentication and
mycoplasma contamination status
• animals: also report source, species, strain, sex, age,
husbandry, inbred and strain characteristics of transgenic
animals
Reproducibility Journal Standards
• antibodies: also report source, characteristics, dilutions and
how they were validated
• cell lines: also report source, authentication and
mycoplasma contamination status
• animals: also report source, species, strain, sex, age,
husbandry, inbred and strain characteristics of transgenic
animals