Appendix B. Publication biases.
We explored a suite of variables that could potentially bias our results presented in the main
paper: average study duration, sample size, publication year, and journal impact factor.
Table B1. Summary of the bias variables examined.
Variables
Average Duration
Description
Continuous variable, log-
Levels of Variable
-Length of study in days
transformed
Sample Size
Continuous variable, log-
-Average sample size for each treatment
transformed
combination
Publication Year
Continuous variable
-Integer year of study
Journal Impact Factor
Continuous variable
-Impact factor during the year study was
published
Methods
We collected data on potential bias variables: average study duration, sample size per treatment
combination, the year the study was published, and journal impact factor during the year the
study was published (Table B1). Average duration of the study was recorded as the number of
days from the start of the experiment to the day the response variable was measured. We
expected that studies conducted over a longer period would be more likely to include extreme
interaction outcomes, leading to greater variation. Sample size was calculated as the mean
sample size across the control and experimental treatments. Likewise, we expected that studies
with greater sample sizes would be more likely to capture strong variation in interaction
outcome. We recorded the publication year. We expected no biological differences in effect
sizes through time, so variation in effect sizes with publication year would indicate some kind of
publication bias, perhaps reflective of general interest in the theme of context-dependency.
Journal impact factor was the ISI journal impact factor during the year the study was published,
derived from Journal Citation Reports ® (prepared by Thomson Reuters ®, Inc., Philadelphia,
PA, USA). We expected lower levels of context-dependency for studies published in higher
impact journals, which may select for studies with larger and less variable effect sizes. There was
one publication with an exceptionally high impact factor, and three with very low impact factors,
which skewed the distribution of this covariate; these outliers were excluded prior to analysis
because transformations did not correct the distribution skew.
Statistical analyses
Analyses followed those in described in the main text.
Results
The average duration of the study affected the CV*RII and sign change, and in both cases this
effect depended on both the context and species interaction class (duration × context gradient ×
species interaction: CV*RII: F6,133 = 14.59, P < 0.0001; sign change: F6,133 = 86.12, P < 0.001). In
three cases (mutualism–third party [β = 1.70, P = 0.017], predation–abiotic [β = 0.13, P = 0.040],
and predation–temporal [β = 0.33, P < 0.0001]) longer studies resulted in higher CV*RII than
shorter studies as we predicted. However, for competition studies examining spatial variation,
longer studies were associated with lower CV*RII values (β = –0.61, P = 0.011). In two out of
eight cases, longer study durations were associated with higher probability of detecting a change
in sign as we had anticipated (predation–temporal, β = –0.39, P < 0.001; mutualism–temporal, β
= 0.42, P < 0.001); interestingly, these were both combinations examining temporal variation.
However, in two other cases (predation–spatial [β = –0.99, P < 0.001], predation–abiotic [β = –
0.56, P < 0.001]) longer study durations were associated with a lower probability of detecting a
change in sign, contrary to our prediction. In four combinations of context × species interaction
type, analyses were not conducted due to small sample sizes (not all studies reported duration
with sufficient accuracy).
The magnitude of CV*RII generally increased over time with year of publication (β =
0.07; F1,223 = 10.39, P = 0.0015), but the direction of this relationship varied with both the
context type and interaction class (year of publication × context gradient × species interaction:
F6,223 = 4.42, P < 0.001). The only significantly positive relationship between year of publication
and CV*RII occurred for predation–third party studies (β = 0.07, P = 0.012), and competition–
temporal, competition–third party, and mutualism–spatial studies all showed significantly
negative relationships. In contrast, studies generally had a lower propensity to show a change in
time in more recent years of publication (β = –0.09; F1,223 = 381.09, P < 0.001). The magnitude
of this negative relationship depended on both the context type and interaction class (year of
publication × context gradient × species interaction: F6,223 = 49.66, P < 0.001). Specifically, the
propensity for an interaction to change in sign was reported significantly less frequently in more
recent publication years for four of 12 cases: most strongly for predation–temporal studies (β = –
0.39, P = 0.008), followed by predation–abiotic (β = –0.15, P < 0.0001), competition–temporal
(β = –0.10, P < 0.0001), and lastly predation–temporal (β = –0.09, P = 0.003). Results highlight
a particular decline over time in reports of a sign change in studies of predator-prey interactions.
We expected that studies published in journals with higher impact factors would have
less variable effect sizes. We were particularly interested in testing whether studies showing high
context-dependency in mutualisms would be published in journals with higher impact factors
than studies on competition and predation, due to the greater emphasis on context-dependency as
an overarching theme in the study of mutualism. Direct tests confirmed our expectation of a
greater emphasis on context dependency in high impact mutualism studies. The influence of
journal impact factor on both CV*RII and sign change significantly varied among species
interaction classes (journal impact factor × species interaction, CV*RII: F6,219 = 23.73, P < 0.001;
sign change: F6,219 = 140.82, P < 0.001). Mutualism studies with higher levels of contextdependency were published in relatively higher impact journals (CV*RII : β = 0.09, sign change: β
= 0.00), whereas both competition and predation studies showed less positive relationships
between metrics of variation and journal impact factor (competition: CV*RII : β = 0.03, sign
change: β = -0.44; predation: CV*RII : β = 0.00, sign change: β = -0.32).