Supplementary material
Case 2: epistatic trade-off with direct, divergent selection (full model)
In the full model, we allow the B locus to experience divergent ecological selection (t1, t2
> 0).
In Case 2, the b allele can invade Population 2 if
m2 <
s2 (r + s2 + eA + ea )(t 2 + ea )(1+ t 2 )
r(s2 + eA + ea )
Following invasion, the equilibrium frequencies of the A allele and B allele in the two
populations are
p˜ A,1 = 1p˜ A,2 =
m2 (r + s2 + eA + ea )
+ O(x 2 )
(s2 + eA + ea )(r + s2 + t 2 + ea )
p˜ B,1 = 1and
p˜ B,2 =
m1 (r + s1 )
+ O(x 2 )
s1 (r + s1 + t1 - ea )
m1 (r + t1 + eA )
+ O(x 2 )
(t1 + eA )(r + s1 + t1 - ea )
m2 (r + t 2 + ea )
+ O(x 2 )
(t2 + ea )(r + s2 + t 2 + ea )
When linkage is loose (r >> s), stability requires
é s (e + t ) s (e + t - e t )ù
m1 < minê 1 A 1 , 1 A 1 A 1 ú
ë2eA + ea + t1 eA + ea + s1 û
and
é
e (1- ea - s2 ) + (1- ea )(ea + s2 ) (1- ea )(ea + t 2 )(eA + ea + s2 )ù
m2 < minê(ea + t 2 ) A
,
ú
eA + 2ea + t 2
2(eA + ea ) + s2
ë
û
The intuitive but important point is that divergent ecological selection on the B locus
makes it much easier to maintain the two-locus polymorphism at higher levels of gene
flow.
Intrinsic isolation can be calculated for Case 2 as in Case 1 (see online
Mathematica file). There are no qualitative differences with respect to the major
conclusions because the additional ecological selection in Case 2 is not expected to affect
intrinsic isolation. When there is ecological selection on B, it is possible to have
divergence at this locus even if the b allele has beneficial effect on the a background but
no counterbalancing negative interaction on the A background. Despite this, it is still
possible to observe intrinsic isolation in the formal sense because "parental types" (e.g.,
AB and ab) will have higher intrinsic fitness, on average, than "hybrid types" (e.g., Ab
and aB). Of course, it is not possible to observe strong intrinsic isolation such as hybrid
inviability or sterility in this type of model because there is no intrinsic "incompatibility"
in the intuitive sense of that term.
The barrier to gene flow of alleles from Population 2 into Population 1 is
31
B1 = 32
(s1 + t1 - ea ) + 321 r(57 - 42r + 22r 2 - 7r 3 + r 4 )(eA + ea ) + O(x 2 )
whereas the barrier into Population 2 is
31
B2 = 32
(s2 + t 2 + ea ) + 321 r(57 - 42r + 22r 2 - 7r 3 + r 4 )(eA + ea ) + O(x 2 )
Ecological selection on the B locus (t1, t2 > 0) increases the barrier (i.e., lowers gene
flow) into both populations. This is unsurprising because a neutral allele that migrates
into a population will be disadvantaged by the ecological selection acting against both the
foreign A and B locus alleles with which it is associated. In Case 1 (t1 = t2 = 0), the
evolution of the epistatic modifier often reduced the barrier into Population 1. This
outcome is less likely in case 2 where t1 > 0 because of direct ecological selection against
b alleles entering Population 1.
Diploid Simulations
We performed simulations using a diploid model that closely paralleled the haploid
model reported in the main text. To avoid a proliferation of parameters in dealing with
diploidy, we assumed most effects combined multiplicatively across homologous
chromosomes (i.e., intermediate dominance). However, we explored three different
levels of dominance for the modifier's deleterious epistatic effect as the dominance of
such effects are believed to be important in explaining the ubiquitous pattern of sex-based
asymmetries in hybrid breakdown known as Haldane's rule.
Let Xi and Yi be the number of a and b alleles, respectively, carried by diploid
genotype i (Xi,Yi  {0, 1, 2}). The fitness of genotype i in Population 1 is
wi,1 = (1- s1)X i (1- t1)Yi (1+ ea )X i Yi / 2 (1- eA ) f [X i ,Yi ]
The fitness of genotype i in Population 2 is
wi,2 = (1- s2 )2-X i (1- t 2 )2-Yi (1+ ea )X i Yi / 2 (1- eA ) f [X i ,Yi ]
To model cases where modifier's deleterious epistatic effects are recessive, multiplicative,
or dominant, we use the following functions, respectively:
f recessive [X i ,Yi ] = (2 - X i )Yi (Yi -1)/2
f additive [X i,Yi ] = (2 - X i )Yi /2
and
f dominant [X i ,Yi ] = (2 - X i )(2 - (2 -Yi )(1-Yi )) /2
Figures S3 and S4 show results for the recessive model for Case 1 (t1 = t2 = 0) and Case 2
(t1 = t2 = 0.5s), respectively. Figures S5 and S6, show Cases 1 and 2, respectively, for
the multiplicative model. Figures S7 and S8, show Cases 1 and 2, respectively, for the
dominant model. Other details are the same as in Figures 2 and 3 in the main text.