BRIEF COMMUNICATIONS
Evolution, 56(5), 2002, pp. 1059–1062
THE COST OF FLUCTUATING INBREEDING DEPRESSION
PIERRE-OLIVIER CHEPTOU1 AND DANIEL J. SCHOEN2
Department of Biology, McGill University, 1205 Dr. Penfield Avenue, Montreal, Quebec H3A 1B1, Canada
1 E-mail: pchept@po-box.mcgill.ca
2 E-mail: danpschoen@maclan.mcgill.ca
Abstract. We present a phenotypic model for the evolution of self-fertilization in an infinite population of annual
hermaphrodites for the case in which fitness and inbreeding depression vary among generations (e.g., due to fluctuations
in the environment from year to year). Conditions for the evolution of selfing, mixed mating, and outcrossing are
derived and are compared with results from numerical calculations that assume a normal distribution of inbreeding
depression. In contrast to the situation in which inbreeding depression does not vary, when inbreeding depression
fluctuates in a stochastic manner among generations with a mean less than 0.5, selfing is not necessarily selected.
Thus, fluctuating inbreeding depression can be viewed as an additional cost of selfing that may stabilize mixed mating
systems. These results emphasize the need to take into account fluctuating inbreeding depression in empirical studies
aimed at understanding mating system evolution in annuals.
Key words.
Negative feedback, selfing, stochastic environment.
Received October 15, 2001.
Both genetic and ecological factors have been invoked as
selective factors in models for the evolution of self-fertilization (Barrett and Harder 1996). Two main genetic factors
influence the selection of mating system modifiers. First,
genes that increase the rate of selfing are automatically selected because they bias their own transmission (Fisher
1941). Second, such genes are counter selected because of
reduced survivorship and fecundity in the progeny of selfed
parents (Darwin 1876). In both empirical and theoretical studies, the reduction in performance of progeny derived from
selfing is embodied as a single parameter, inbreeding depression, that is typically expressed as 1 2 (the relative fitness
of selfed progeny) (Charlesworth and Charlesworth 1987).
When automatic selection and inbreeding depression are the
only forces in play, there are two evolutionarily stable outcomes (Lloyd 1979): either complete selfing evolves (when
inbreeding depression is below 0.5) or complete outcrossing
evolves (when inbreeding depression is greater than 0.5); see,
however, Uyenoyama et al. (1993) and Holsinger (1996) for
a discussion of some limitations of this model.
An underlying assumption of the vast majority of mating
system evolution models is that ecological factors that contribute to fitness, and consequently to inbreeding depression,
do not vary in time (but see Lloyd 1980). This contrasts with
other demographic (Caswell 1989) and life-history models
(Kisdi and Meszena 1995), in which the consequences of
environmental fluctuations are explicitly taken into account.
Moreover, there is emerging evidence for variation in both
absolute and relative fitnesses of selfed progeny that is associated with varying ecological conditions (Dole and Ritland
1993; Cheptou et al. 2000).
Cheptou and Mathias (2001) produced a model for the
evolution of selfing under specific conditions of varying inbreeding depression. This paper generalizes the problem in
such a way that varying inbreeding depression emerges as a
cost of selfing. This more general model also makes it clear
under what conditions this cost is large enough to influence
the stable outcome of mating system evolution.
Accepted January 23, 2002.
THE EVOLUTION
SELFING WHEN INBREEDING DEPRESSION
VARIES
Consider an infinite monomorphic population composed
of annual hermaphrodites with selfing rate R. Because both
absolute and relative fitness may vary among generations,
we assume that fitness of outbred progeny and inbreeding
depression are random variables denoted as G and d, respectively. The selection of selfing is analyzed by considering
the fitness of a rare mutant phenotype with selfing rate r (close
to R). The fitness of an individual is measured as the number
the gametes transmitted to the next generation (Uyenoyama
et al. 1993) and is treated as the sum of three components:
the selfed seeds, outcrossed seeds, and the seeds of other
individuals produced by fertilization with exported pollen
(Lloyd 1992). We assume that the selfing rate does not influence pollen export (i.e., there is no pollen discounting).
Accordingly, the fitness of mutant phenotype with selfing rate
r at generation t is:
[
wt (r, R) 5 Gt r(1 2 d t ) 1
]
12r
12R
1
.
2
2
(1)
But in an environment with stochastically varying fitness and
inbreeding depression, the fitness of the mutant is characterized by its long-term exponential growth rate or invasion
function s(r, R) (Geritz et al. 1998). This quantity is given
by the expectation of the logarithm of the annual growth rate
(Kisdi and Meszena 1995):
s(r, R) 5 E[Ln w(r, R)].
(2)
We assume that the population is demographically stable in
the long term, which means that the average over time of the
log annual growth rate is zero for the resident morph with
selfing rate R (see Kisdi and Meszena 1995). Thus, we may
write:
s(R, R) 5 E[Ln w(R, R)] 5 0.
Consequently,
1059
q 2002 The Society for the Study of Evolution. All rights reserved.
OF
(3a)
1060
BRIEF COMMUNICATIONS
E[Ln G] 5 2E[Ln R(1 2 d) 1 (1 2 R)].
(3b)
Under stable demographic conditions, the fate of the mutant
can then be analyzed by considering the algebraic quantity:

R 


12r
12
 r(1 2 d) 1
1
2
2
 
s(r, R) 5 E Ln 
R(1 2 d) 1 (1 2 R)
 



(4)
obtained by substituting (3b) into (2). Equation (4) defines
the criteria for the evolution of selfing in an environment
with stochastically varying fitness and inbreeding depression
when the population is stable on the long term. Analysis of
(4) is dependent on the choice of a distribution for d, which
is, to our knowledge, not known in the wild. However, several
different approaches to analyzing (4) can be taken. These are
discussed in the following two sections.
ANALYTICAL APPROXIMATION FOR VARYING INBREEDING
DEPRESSION
We first approach the problem of the evolution of selfing
in a varying environment using an analytical approximation
of (4) based on the mean and the variance of the distribution
of d. In particular, approximations to equation (4) can be
obtained using the second-order approximation:
E[Ln(l)] ø Ln[E(l)] 2
1
V(l)
2E(l) 2
(5)
(Bulmer 1994), where l is a random variable and V(l) is its
variance. Letting the mean and the variance of the inbreeding
depression (d) over time be denoted as m and s2 and using
the classical properties of the mean and the variance, the
second-order approximation for s(r, R) is:
[
s(r, R) 5 Ln r(1 2 m) 1
]
12r
12R
1
2
2
NUMERICAL ANALYSIS UNDER NORMALLY DISTRIBUTED
INBREEDING DEPRESSION
2 Ln[R(1 2 m) 1 (1 2 R)]
2
1
r2s2
[
]
12r
12R
2 r(1 2 m) 1
1
2
2
R2s2
.
2[R(1 2 m) 1 (1 2 R)] 2
2
(6)
Equation (6) is always defined when m , 1. The direction
of selection on the mating system in a population with arbitrary selfing rate R can be studied by analyzing the local
fitness gradient, D(R), which is the partial derivative of the
invasion function with respect to the mutant strategy r, evaluated at R (see Geritz et al. 1998). The fitness gradient is:
11 2 22 R
R
D(R) 5
0.5 2 m
2
s2.
1 2 mR
(1 2 mR) 3
selected. On the contrary, if D(R) is negative, then only phenotypes with r , R can invade, and higher outcrossing will
be selected. Thus, the sign of D(R) is a sufficient to predict
the direction of mating system evolution.
The first term in (7) gives the fitness gradient in absence
of temporal fluctuations in inbreeding depression (s2 5 0),
whereas the second term captures the effects of temporal
fluctuations in inbreeding depression. In absence of temporal
fluctuations, the selfing rate in the population is determined
only by m. It is positive when m , 0.5, leading to the evolution of complete selfing, or negative if m . 0.5 leading to
complete outcrossing, a result that agrees with classical predictions (Lloyd 1979; Lande and Schemske 1985).
Because the second part of the equation is always negative
for 0 , R , 1, it follows that, for a given mean inbreeding
depression, the fitness gradient will always be lower in the
presence of temporal variation. The biological interpretation
is that, under varying inbreeding depression, the evolution
of selfing is more difficult compared with constant inbreeding
depression of the same mean. Moreover, because the second
part of (7) is a monotonically decreasing function of R, for
a given distribution of inbreeding depression, the evolution
of selfing becomes increasingly more difficult as the selfing
rate increases. Thus, in a population evolving toward selfing,
there is negative feedback effect caused by fluctuating inbreeding depression.
Of particular note, the fitness gradient can go to zero for
some intermediate values of R, which means that selection
ceases and a mixed selfing rate evolves as an evolutionary
stable strategy (Maynard Smith 1982). More generally, Figure 1 illustrates the direction of mating system evolution in
the parameter space of m and s2. There are three regions
corresponding, respectively, to complete selfing (R 5 1), stable mixed selfing (0 , R , 1), and complete outcrossing (R
5 0).
(7)
Inferences based on (7) assume that the mating system
evolves by mutations that cause small changes in the selfing
rate. If D(R) is positive, then only phenotypes with r . R
can invade, and consequently, higher selfing rates will be
If one assumes that temporal variation in inbreeding depression is normally distributed, it is possible to numerically
evaluate the criteria for invasion of a phenotype with mutant
selfing rate r using equation (4). Because the relative survival
of inbred progeny is positive, this implies that d , 1. If we
further assume that the fitness of outbred progeny is greater
than that of inbred progeny, then d . 0. Because of these
limits on d, we can use a truncated normal distribution (0 ,
d , 1), and normalize the probability density (see Lynch and
Walsh 1998).
Because an analytical derivative cannot be obtained in this
case, the direction of selection can be determined by numerical evaluation of the fitness gradient. This is obtained
by taking a mutant with selfing rate r close to R (mutation
step: 1025) and calculating its long-term growth rate (eq. 4)
using Mathematica 4.1. A positive growth rate when r 2 R
. 0 indicates that more selfing is selected. A positive growth
rate when r 2 R , 0 indicates that more outcrossing is selected. The evolutionary stable selfing rate R* is the one for
which mutants with selfing rates that vary in either direction
have negative growth rates (Geritz et al. 1998).
1061
BRIEF COMMUNICATIONS
FIG. 1. The evolution of selfing when there is temporal variation
in inbreeding depression. The graph illustrates where complete selfing, mixed selfing, and complete outcrossing are evolutionarily stable in the parameter space of m and s2, where m and s2 define the
mean and the variance of the inbreeding depression distribution,
respectively. The solid curve defines the boundary between complete selfing and mixed selfing for normally distributed inbreeding
depression. The dotted curve defines the boundary between complete selfing and mixed selfing, based on the second-order approximation.
As in the previous section, the selection of selfing can be
illustrated in the parameter space of m and s2 in Figure 1.
The variance of the truncated normal distribution, s2, was
calculated numerically. The results show that for small variance, the boundary between complete selfing and mixed selfing is nearly equivalent for both analytical and numerical
solutions. As the variance increases, the analytical approximation gives increasingly unreliable results, which is expected (Bulmer 1994). For s2 , 0.015, the approximation
appears to be good. But for higher variance, results based on
the approximation do not conform to those based on numerical analysis with normally distributed inbreeding depression (Fig. 1). In particular, the domain of mixed selfing
rate is larger than that based on approximation, suggesting
that fluctuating inbreeding depression can be a substantial
force maintaining intermediate selfing rate.
DISCUSSION
This model shows that the presence of temporal variation
in inbreeding depression can qualitatively modify the selection of selfing. Contrary to the classical conclusions where
the direction of selection is independent of the population
selfing rate, we have shown that the evolution of selfing depends on the interaction between inbreeding depression over
time and population selfing rate. The reason for this is quite
simple. It is known that for a given mean, the time average
of the log annual growth rate is reduced by temporal fluctuations (cf. Jensen’s inequality E[Ln(l)] , Ln[E(l)]; Caswell 1989). As a consequence, fluctuations in inbreeding
depression decrease the long-term fitness of phenotypes practicing selfing and a predominant selfer will pay a high cost
of fluctuating inbreeding depression. However, because inbred progeny contribute little to the total fitness of a predominantly outcrossing phenotype, it will be little affected
by inbreeding depression fluctuations. This phenomenon can
produce negative feedback on the selection of selfing, which
can lead to the maintenance of stable mixed selfing rate.
The existence of inbreeding depression fluctuations in the
wild is probably common, but has been examined in only a
few empirical studies. Importantly, the biological relevance
of varying inbreeding depression depends crucially on the
magnitude of the variation. Some studies have shown that
temporal variance can be important in the field. For example,
Dole and Ritland (1993) showed by direct field measurements
in Mimulus species that the relative fitness of selfed progeny
varied three- to fourfold between years (i.e., 0.1 to 0.3 for
M. guttatus and 0.17 to 0.47 for M. platycalyx). Eckert and
Barrett (1994) measured inbreeding depression in 10 different
natural populations of Decodon verticillatus, and they showed
variations of a similar order of magnitude. Unfortunately,
there have been no long-term studies of variation, and so
empirical measures of the mean and variance of inbreeding
depression are not available. But the model results together
with these few findings suggest that temporal variation in
inbreeding depression could be more important than previously assumed.
The phenotypic selection model described here has been
adopted for mathematical convenience. The results have been
checked by simulation using a diploid genetic model based
on the recurrence equations described by Holsinger et al.
(1984) and show that selected selfing rates are similar to those
predicted by the phenotypic model when equivalent inbreeding depression distribution are assumed (data not shown).
However, the phenotypic model described above does not
consider the possible effects of genetic associations generated
by selfing, and these are known to modify conditions for the
evolution of selfing (Holsinger 1988).
In the context of the temporally varying inbreeding depression and its effects on mating system evolution, the consideration of perenniality could be an interesting direction
for future research direction. Perennials, by virtue of their
longevity, may produce progeny cohorts that experience different levels of inbreeding depression. This could impact the
evolution of the mating system. Indeed, Morgan et al. (1997)
demonstrated that evolution of selfing is more difficult in
perennials because of the larger number of selective episodes
in the their life.
ACKNOWLEDGMENTS
We thank K. Holsinger and D. Waller for advice and criticism. This work was supported by scholarship support from
the government of Quebec to POC and by a research grant
from the National Science and Engineering Research Council
of Canada to DJS.
LITERATURE CITED
Barrett, S. C. H., and L. D. Harder. 1996. Ecology and evolution
of plant mating system. Trends Ecol. Evol. 11:73–79.
1062
BRIEF COMMUNICATIONS
Bulmer, M. 1994. Theoretical evolutionary ecology. Sinauer, Sunderland, MA.
Caswell, H. 1989. Matrix population models: construction, analysis,
and interpretation. Sinauer, Sunderland, MA.
Charlesworth, D., and B. Charlesworth. 1987. Inbreeding depression and its evolutionary consequences. Annu. Rev. Ecol. Syst.
18:237–268.
Cheptou, P. O., and A. Mathias. 2001. Can varying inbreeding
depression select for intermediairy selfing rates? Am. Nat. 157:
361–373.
Cheptou, P. O., E. Imbert, J. Lepart, and J. Escarre. 2000. Effects
of competition on lifetime estimates of inbreeding depression in
the outcrossing plant Crepis sancta (Asteraceae). J. Evol. Biol.
13:522–531.
Darwin, C. R. 1876. The effects of cross and self fertilization in
the vegetable kingdom. Murray, London.
Dole, J., and K. Ritland. 1993. Inbreeding depression in two Mimulus taxa measured by multigenerationnal changes in inbreeding coefficient. Evolution 47:361–373.
Eckert, C. G., and S. C. H. Barrett. 1994. Inbreeding depression in
partially self-fertilizing Decodon verticillatus: population genetic and experimental analysis. Evolution 48:952–964.
Fisher, R. A. 1941. Average excess and average effect of a gene
substitution. Ann. Eugen. 11:53–63.
Geritz, S. A. H., E. Kisdi, G. Meszena, and J. A. J. Metz. 1998.
Evolutionnary singular strategies and the adaptive growth and
branching of the evolutionnary tree. Evol. Ecol. 12:35–57.
Holsinger, K. E. 1988. Inbreeding doesn’t matter: the genetic basis
of mating system evolution. Evolution 42:1235–1244.
———. 1996. Pollination biology and the evolution of mating systems in flowering plant. Evol. Biol. 29:107–149.
Holsinger, K. E., M. W. Feldman, and F. B. Christiansen. 1984.
The evolution of self-fertilization in plants: a population genetic
model. Am. Nat. 124:446–453.
Kisdi, E., and G. Meszena. 1995. Life history with lottery competition in stochastic environment: ESSs which do not prevail.
Theor. Popul. Biol. 47:191–211.
Lande, R., and D. W. Schemske. 1985. The evolution of self fertilization and inbreeding depression in plants. I. Genetic models.
Evolution 39:24–40.
Lloyd, D. G. 1979. Some reproductive factors affecting the selection
of self-fertilization in plants. Am. Nat. 113:67–79.
———. 1980. Demographic factors and mating patterns in Angiosperms. Pp. 67–88 in O. T. Solbrig, ed. Demography and evolution in plant populations. Univ. of California Press, Berkeley.
———. 1992. Self and cross fertilization in plants. 2. The selection
of self fertilization. Int. J. Plant Sci. 153:370–380.
Lynch, M., and B. Walsh. 1998. Genetic analysis of quantitative
traits. Sinauer, Sunderland, MA.
Maynard Smith, J. 1982. Evolution and the theory of games. Cambrige Univ. Press. Cambridge, U.K.
Morgan, M. T., D. J. Schoen, and T. M. Bataillon. 1997. The evolution of self fertilization in perennials. Am. Nat. 150:618–638.
Uyenoyama, M. K., K. E. Holsinger, and D. M. Waller. 1993. Ecological and genetic factors directing the evolution of self fertilization. Oxf. Surv. Evol. Biol. 9:327–381.
Corresponding Editor: M. Morgan