Copyright 0 1986 by the Genetics Society of America THE SELECTION LIMIT DUE TO THE CONFLICT BETWEEN TRUNCATION AND STABILIZING SELECTION WITH MUTATION ZHAO-BANG ZENG AND WILLIAM G. HILL Institute of Animal Genetics, University of Edinburgh, Edinburgh EH9 3JN, Scotland Manuscript received February 1 1, 1986 Revised copy accepted August 2 5 , 1986 ABSTRACT Long-term selection response could slow down from a decline in genetic variance or in selection differential or both. A model of conflict between truncation and stabilizing selection in infinite population size is analysed in terms of the reduction in selection differential. Under the assumption of a normal phenotypic distribution, the limit to selection is found to be a function of K , the intensity of truncation selection, w2, a measure of the intensity of stabilizing selection, and U*, the phenotypic variance of the character. The maintenance of genetic variation at this limit is also analyzed in terms of mutation-selection balance by the use of the "House-of-cards" approximation. It is found that truncation selection can substantially reduce the equilibrium genetic variance below that when only stabilizing selection is acting, and the proportional reduction in variance is greatest when the selection is very weak. When truncation selection is strong, any further increase in the strength of selection has little further influence on the variance. It appears that this mutation-selection balance is insufficient to account for the high levels of genetic variation observed in many long-term selection experiments. I N artificial selection experiments and breeding programs, there may be a conflict between artificial and natural selection. Usually, the aim of artificial selection is to improve some particular traits; thus, it is generally directed toward extreme phenotypic values. It is observed that natural selection often favors intermediate expression of metric traits unless these traits are very closely associated with fitness (e.g., LINNEY,BARNESand KEARSEY 1971). So plateaux obtained in artificial selection experiments could result from the opposing forces of directional and natural selection, rather than from a loss of additive genetic variance, as indicated by some long-term selection experiments (e.g., LERNERand DEMPSTER1951; CLAYTONand ROBERTSON1957; LATTER 1966; ROBERTS1966; WILSONet al. 1971; Yoo, NICHOLAS and RATHIE1980). Two extreme models have been proposed for the action of natural selection on quantitative characters. One is stabilizing (or optimum) selection, another is overdominant selection. Stabilizing selection has received considerable attention over the last several decades (e.g., FISHER 1930; WRIGHT1935; HALDANE 1954; ROBERTSON 1956; KOJIMA 1959; LATTER1960, 1970; LEWONTIN1964; LANDE 1975). In this selection model it is assumed that the fitness of an Genetics 114: 1313-1328 December. 1986. 1314 Z.-B. ZENG AND W. G. HILL individual is solely a function of its phenotypic deviation from an intermediate optimum. In overdominant selection, on the other hand, it is assumed that heterozygotes at the loci affecting the quantitative character have higher fitness, and this model has been thought by many people to be important as a possible explanation for the maintenance of genetic variability in both artificial and natural populations (e.g., LERNER1950, 1954; ROBERTSON 1956; BULMER 1973; GILL~ESPIE 1984). JAMES (1962) considered the case where the conflict is between truncation and stabilizing selection. With the assumption that heritability is not greatly altered during the course of selection, he was able to develop an approximate expression for the selection limit that is expressed in terms of K , the intensity of truncation selection, w2, a measure of the intensity of stabilizing selection, and a ' , the phenotypic variance of the character. It is not clear, however, whether heritability is zero or not at this selection limit. In contrast, ROBERTSON ( 1 956), VERGHESE (1974) and NICHOLAS and ROBERTSON (1980) investigated another model, called the homeostatic model, for which the conflict is between directional and overdominant selection. The impact of such a model on selection limits has been explored in depth. However, in all these studies, mutation as a source of introducing fresh variability has been ignored. The dynamics and maintenance of genetic variability of quantitative characters under stabilizing selection and mutation have been studied intensively by many authors (LATTER 1960; KIMURA 1965; BULMER1972, 1980; LANDE 1975; FLEMING1979; TURELLI 1984). They examined the balance between mutation and stabilizing selection to see whether this balance could account for the high levels of heritable variation usually observed for many quantitative characters in natural populations. LANDE(1 975), in particular, has forcefully argued that high heritabilities could be maintained by this mutation-selection balance even with strong stabilizing selection. Recently, TURELLI (1984) has critically reviewed this argument and numerically given the domains of applicability of the various approximations produced by him and other authors. Based on a Lerch's zeta function analysis and numerical work, TURELLI argued that the approximation he used would give a better estimate of the equilibrium genetic variance when the mutation rate per locus is of the order of or less. Elsewhere we have investigated the consequences of recurrent mutation on selection response in finite populations, but in the absence of any opposing effects of natural selection (HILL 1982). In this paper, we examine the state of the population at the selection limit due to the conflict between truncation and stabilizing selection, i.e., we investigate the selection differential, heritability and the population mean at this limit. We assume that selection is on a single trait, gene effects are additive and the population is infinitely large. First, in the section below, we present an analysis on the phenotypic change due to selection; and then, in the following section, we investigate the genetic structure of the population at the limit. ANALYSIS OF THE PHENOTYPIC CHANGE Consider a metric character having phenotypic value x with the following probability density function among juveniles (before the operation of selection) in generation t f ( x ) = TU^)-"^ expi-(x - u,)'/(2a2j), (1) 1315 LIMIT T O SELECTION where U' is the phenotypic variance which is assumed to be independent of the mean ut. The fitness of individuals under stabilizing selection with phenotypic value x is assumed to decrease with deviation from the optimum value according to the relation wl(x) = exp(-x'/(2~')), (2) where the optimum value of x is taken to be zero, and U' is a measure of intensity of stabilizing selection, being less intense the larger U' is in relation to u2. Stabilizing selection may act on individuals through the whole life cycle, by differential viability or reproductivity, or both. But here, for convenience, it will be assumed that the stabilizing selection occurs before truncation selection. Then it is readily shown that, among the survivors after stabilizing selection, the phenotypic distribution becomes f '(4= f(x)wl(x)/Sf(.)~l(x)dx = ( 2 ~ c a ~ ) -exp(-(x '/~ (3) - cu,)'/(2cu2)], + where c = w'/(u2 U * ) is called the coefficient of centripetal selection by LATTER(1970). T h e distribution (3) is still normal with mean U ; = cut and variance U'' = cu2. Truncation selection induces the fitness function (4) where T is the truncation point in absolute value. Using (3) and (4), we can then obtain the proportion of individuals surviving the two kinds of selection (the mean fitness of the population): = P[u'/(u' + u ' ) ] ~ /exp(' u:/[2(u2 + U')]). where P = J: f ' ( x ) d x is the proportion of individuals surviving stabilizing selection that are then selected by truncation. T h e change in the population mean by truncation after stabilizing selection is = KWU/(U' + W2)1/2, where K is the standardized selection differential of truncation selection, corresponding to P. Then the total selection differential is given by st = up - U* = ( u p - U:) = KU(T/(U2 + + (U: - ut) - UtU2/(U2 (7) + U'). 1316 Z.-B. ZENG AND W. G. HILL This relation would hold for every generation if the phenotypic distribution remained normal before selection and U' were constant. When the second assumption is violated, U' has to be replaced by U? in (7). The change in the average phenotypic value in response to selection must equal the product of heritability and selection differential, i.e., where hf is the heritability of the character at generation t. Thus, at a selection limit, where Au = 0, either s or h' should be zero. If a limit is attained due to the attenuation of selection differential, the total predicted selection advance can be obtained by setting s = 0 in (7) and is = Um + KW(U' (94 W2)1'2/U or Um = KW(U' + W2)lD/U - Uo if uo # 0. This relation was first obtained by derivation. Furthermore, from (7) and (8), we can get JAMES h'u' - = (1 %+I =) (9b) (1962) by a different h'xwu ut + + w')"' (U' T h e first term on the right side of (10) shows the effect of stabilizing selection on the phenotypic change which is the same as shown in Equation 14 of LANDE (1975). The second is the effect of truncation selection that could be constant in standard units, if the indicated assumptions hold; that is, the phenotypic distribution is always normal before selection, h' and U' do not change very much during the course of experiment, and the same proportion of individuals is selected by truncation in every generation. If uo = 0, the selection response in the first generation of selection may be expressed as u1 = h2Kuw/(u2 + w')"', (1 1) and the ratio of the total response over the response in the first generation becomes um/ul = Finally, solving (10) gives ut = uo exp {- h'u't ___ U' (U' ]+ [ + w2 + w')/(u'h'). 1 -exp (12) {-~ )] h'u't U' + KU(U' U' + w2)"* U . (13) Then we can get the "half-life" of the selection process, the number of generations taken to go halfway to the limit (ROBERTSON 1960). This is to.5 = ln2(u2 + w2)/(u'h2) generations. (14) There are several notable features to this analysis. First, as indicated in (13), LIMIT TO SELECTION 1317 the response curve is exponential, which is similar to that predicted by ROBERTSON'S (1960) theory for finite populations with directional selection alone. In this analysis the response rate is h2a2/(a2 U'), a function of h2 and 02/a2. Therefore, the half-life is expected to be longer if selection is on a character with a lower heritability and/or less intense stabilizing selection (i.e., higher value of w2/a2) (14). Second, although the response rate is a function of heritability, the selection limit predicted is independent of heritability. Third, in a large population, the total response is maximized by having the smallest possible P (9), but in a small population, P should be 0.5 to obtain the largest total response (ROBERTSON 1960). + ANALYSIS OF THE EQUILIBRIUM: DISTRIBUTION OF ALLELIC EFFECTS In this section, we shift our analysis from the phenotypic level to the genotypic level, and we focus attention on the distribution of allelic effects at the selection limit. The method of analysis is similar to that of TURELLI (1984), but with an extension to include truncation selection. The model: Consider a randomly mating diploid population of infinite size and a quantitative character that is affected by n additive loci and an independent environmental effect. At each locus it is assumed that there is potentially an infinite number of allelic states, and the phenotypic effects of these alleles are continuously distributed. Let x denote the phenotype of this quantitative character with x=y+e (15) and where z: (z!) is the allelic effect of the maternally (paternally) inherited gene at the ith locus in an individual, and e is the environmental effect, assumed to be normally distributed with mean zero and variance U:. The object is to find an approximation for the equilibrium distribution of zi at the limit to selection. First some other assumptions need to be made. Usually, it is assumed that the phenotypic effects of mutant alleles from a given allele state are normally distributed around the phenotypic effect of the original allele (e.g., KIMURA1965). Then, ifft(zi) denotes the distribution of allelic effects among gametes in generation t and if mutation is assumed to occur will be during gametogenesis which follows selection, 1318 Z.-B. ZENG AND W. G . HILL is the density function of allelic effects after selection, where w(zJ is the selection function on zi, and g(zJ = (ZT")-"~ exp(-z?/(Zm?)] (19) is the density function of mutant effects, where m? is the variance of mutant effects for the ith haploid locus. Interpreting (17) in words: at the beginning of generation t 1, an allele zi has probability 1 - pi of coming from allele zi in the previous generation, having survived selection and without mutation, and probability pi of coming from another allele in the previous generation, having survived selection and mutated to zi. Generally speaking, it is difficult to analysis (17) directly without some simplification. Here we simplify (17) in the following way. By expanding g(zi - vi) in Taylor's series about (zi - U : ) , where U{ = J vJ:(vi)dv;, we have + + ( Y 2 ) (U, - U:)2g'2'(z1 - U:) + . . .Id Ut - Pz[g(Z, - U : ) + (%) v:g'2' where V: = J (U, (z,- U : ) + . .I, * - u tr )2f t (r v Z ) d v , If . g(zJ is expressed by (19), g(*)(zz - U:) - u:)'/m; = (l/m,2)g(z, - u:)[(z, - 11. Then ,- An important observation of TURELLI (1984) is that, under the reasonable assumption that pz 5 m,2 >> v:, (21) in which m,' denotes the variance of effects associated with mutation and V: denotes the equilibrium allelic variance after selection. Then the second term and the terms in higher order in the bracket in (20) can be ignored without serious error, and (17) can be approximated by fi+l(ZZ) = (1 - Pz)fi(z*) + P*g(z*- U:) (22) under the condition of (21). This is the "House-of-cards" approximation used by TURELLI (1984) and originally introduced by KINGMAN (1978). In this approximation the effects of new mutants are assumed to be distributed around the population mean, and to be essentially independent of their premutation state. The density function of the equilibrium distribution of z,in this approximation is given by fm(zz) = Pzg(zz - ~ t > N 1 4w(~,)], where 4 is a constant such that (23) 1319 LIMIT TO SELECTION S fm(%i)dZi = 1, (24) and Thus, if w(zJ and g(zi - ui)are known, f&) can be approximated by (23) providing pi C: Allelic effect distribution at the limit to selection: In this section, (23) is used to find the equilibrium distribution of allelic effects zi under stabilizing and truncation selection. First, consider w(zi), the selection function on zi. Let ai = zi - ui be the excess of the allele zi over the mean of alleles at the ith locus. The fitness of this allele, relative to the mean ui,is usually given by wheref(x) is the density function of phenotype and w(x) is the selection function on phenotype. By expandingf(x - a,) in a Taylor series about x, we then have that, to order a', W ( Z ~ )= 1 + Cui + (?'z)Du?, (27) where D = sf(2)(x)w(x)dJsf(x)w(~)dx. Iff(x) is the normal density function with mean U and variance u2, then C = Au/u2 D = (Aa2 + Au2)/u4, where Au and Aa2 are the changes in the mean and variance as a result of selection (BULMER1980). Since there are two kinds of selection, Au and Ao2 are determined by two components. We have already found that in (7) Au = K + w ' ) ~ ' ~- uu2/(u2+ W U / ( ~ ~ (28) U'). From (3), we have that Aa2 (due to stabilizing) = a2(1 - w2/(u2 + w2)) = -U'/(.' and it is well known that Au2 (due to truncation) = -K(K - Z)U~W~+ /(U U'), ~ + w2), 1320 Z.-B. ZENG AND W. G . HILL where K is the intensity of truncation selection and 2 is the standard deviate of truncation point 7. So the total change in the variance due to selection is + Aa' = -(a4 + - Z)a2w2)/(o' K(K (29) U'). With ( 2 8 ) and (29), ( 2 7 ) becomes w(z,) = 1 + + u ~ )-~uu2/ ~ + - Z)w2 ai a,. 22(2+ + U') KUU(U' U' K(K 02) 62(U' (30) In (30) Au2 is not included in the term of a:. As shown later, this does not influence the results. When a, = z,- U , is small in magnitude, w(zJ can also be approxi mated by w(z,) = exp KUU(C2 E U2)1'2 a'(u' = exp(- (z, where + + - UO' U2) a, - + a ' K(K 2a2(a2 - B)*/(2A)Je, - Z)W' + 02) (31) is a constant, -t (r'(a2 A = + U' K(K U') - 2)U' and KUCJ(U' B=Ui+ ' l c + + U2)1'2 K(K - U6' - z)U2 ' Now inserting (31) into (23) and letting g(zi - U,) be defined by (19), we then have the following approximation for the density function of the distribution of allelic effects at the limit to selection It can be shown that in (32) U , = B, i.e., U =K U ( 2 + U2)1'2/U (33) (see APPENDIX for proof). The result (33) has two implications: First, since uiin (32) could, in theory, take any value, but U = KW(C' u ' ) ' / ~ / u ,this shows that the mean genotype, as well as the mean phenotype (since U , = ur = U by assumption), is a fixed value at the limit to selection, but the mean effect of the alleles at a particular locus is not fixed. Their values at the selection limit would then largely depend on the initial conditions, historical influences and chance events at this particular locus. As a consequence, different lines or replicates in an experiment could be quite different in genetic constitution, even though they might show similar phenotypic expressions (see also LANDE 1975). Second, in contrast to the traditional argument that the maximum 'response to artificial selection is a function of the number of loci, i.e., as the number of loci increases, the maximum response increases (e.g., ROBERTSON1960), this model predicts that the maximum response on the phenotype is independent of the number of genes + 1321 LIMIT TO SELECTION responsible for the character. T h e increase in the number of loci is accompanied by a decrease in the effects of individual genes. Equation (33) for the maximum response at the limit to selection is identical to (9). Now let ai = zi - ui.With U = KO(CT‘ w2)1’2/u, (32) reduces to + pi fm(ai) = (27rmf)’” exp(-a?/(Zm?)} [ l - [ exp{-a?/(2A)}] ’ (34) equilibrium distribution of allelic effects under which is the same as TURELLI’S K(K - Z)w‘]. By using stabilizing selection alone. Here A = u2(u2 w’)/[u‘ Lerch’s zeta function, TURELLI (1984) has been able to show that + + 5 = exp{-p?7rA/m?] Vi = E(a?) = 2pZA r2 (35) = E(a4)/[3{E(~?)}’] m?/(6pJ) as pi + 0 for (34), where Vi is the equilibrium genetic variance due to locus i (haploid) and r2 is the coefficient of kurtosis for this distribution. As he pointed out, the approximations rest on the condition that pi << m?/A << 1, (36) which will be justified numerically in the next section. In addition, it can easily be proved that this distribution is symmetric. Considering all relevant loci, the total genetic variance can be approximated by n 0,’ 4piA, = (374 i= I if a global linkage equilibrium is assumed. In particular, if the mutation rate is equal for all loci, U,’ = 4npA = 4npu2(2 U* + K(K + w2) - z)w2. A check on the approximations: The results of (35) were obtained by TUR(1984) from (34) as approximations, as pi + 0, i.e., p j << m?/A << 1. This condition is internally consistent with m? >> Vl (21) which leads (17) to (34) (TURELLI 1984). By a simulation of (17), TURELLI provided a numerical test of the results of (35), which clarified the conditions for the house-of-cards approximation. In this section, we provide another numerical test of (35) directly from the moment calculation of (32) with truncation selection, which relies on TURELLI’S numerical calculation to support (32). The numerical analysis was carried out to minimize five functions, each with five real variables ([, U , V, rl, 7-2) by the Newton-Raphson method (see GILL, MURRAYand WRIGHT1981), where rl is the coefficient of skewness. These functions were the cumulative distribution function and the first four moment functions of (32). T h e process of minimization was as follows: First, an initial guess for the real variables [, U , V , rl and r2 was made, and then the increments ELLI 1322 Z.-B. ZENG AND W. G. HILL TABLE 1 Effects of varying the mutation rate ( p ) on the analytically predicted (Pre.) and numerically determined (Obs.) equilibrium genetic distribution for P = 0.5, V, = 10 and tn2 = 0.05, given pi = 0.1 € P IO-’ U V rvi n 5.61 6.70 Pre. Obs. 0.9999067 0.9999164 7.57 7.57 2.97 x 10-3 2.75 X lo-’ 2.00 x 10-2 1.28 X lo-* Pre. Obs. 0.999999067 0.999999077 7.57 7.57 2.97 2.97 2.00 X lo-’ 1.89 X lo-’ 5.61 5.76 he. Obs. 0.99999999067 0.99999999068 7.57 7.57 2.97 x 10-5 2.99 x 10-5 2.00 x 1 0 - ~ 1.99 x 5.61 x 1 0 2 5.67 x 102 Pre. Obs. 0.99999999991 0.9999999999 1 7.57 7.57 2.97 x 10-6 2.99 x 10-6 2.00 x 1.99 x 10-5 5.61 x 103 5.67 x 109 X X X X 10 10 + [ is a parameter of (32); U = K U ( ~ U‘)”/. is the mean genotype; V is the variance of the distribution; [VI is the variance without truncation selection, i.e., when P = 1; and r2 = E(r, - u,)~/ { 3 [ E ( z ,- u , ) ~ is ] ~the ) coefficient of kurtosis of the distribution. of these variables toward the solutions were calculated. This calculation was iterated until all increments were under the tolerance error, and the final values of the variables were regarded as the solutions. In this study, the initial guess was supplied by (33), (35) and r l = 0 with U * = 0.1, and the tolerance error was IF4. In the computations, for convenience, all measurements except p were scaled so that U = 1. T h e results of the computations are shown in Tables 1-4, which illustrate the effects on the analytically predicted and numerically observed equilibrium genetic distribution of varying p, m2,V, and P separately around p = m2 = 0.05, V, = 10 and P = 0.5, with ui = 0.1 where V, = u2 U‘. These values of parameters are chosen to be consistent with TURELLI’S analysis, so VJu2 = 10 is equivalent to V5/u: = 20 in TURELLI (1984). The equilibrium genetic variance [VI without truncation selection (P = 1) is also presented in the tables for comparison. Since the distribution is symmetric, rl was found always to be zero and thus was excluded from the tables. T h e results of the variance [Ufor P = 1 are very consistent with those of TURELLI.As p and V, (V, = A in this case) decrease and m 2 increases, the approximate values of the variance became close to the observed values, and reasonable agreement between predicted and observed variances is achieved whenever 50p 5 m2/VSapproximately (Tables 1-3). When P = 0.5, the value of A (A = u2V5/[u2 K ( K - Z)w2]) is severely reduced. Although V, ranges from 2 to 50 in Table 3, the value of A ranges only from 1.22 to 1.55. So the predicted variances displayed in Tables 1-3 are a little closer to the observed variances when P = 0.5 than when P = 1. Table 4 shows the effect of truncation selection on reducing the equilibrium genetic variance (see also Figure 1 below). It is notable how severely the genetic variance is reduced by truncation selection. In all four tables, the predicted means (KW(U’ ~ ~ ) ‘ / ~ are /a) matched by the observed means. + + + 1323 LIMIT TO SELECTION TABLE 2 Effects of varying the variance of the effect for the mutants (m’)for 1.1 V, = 10, given 1.1~ = 0.1 = lo4, P = 0.5 and ~ E m2 V U [VI r2 0.001 Pre. Obs. 0.99995333 0.99997010 7.57 7.57 2.97 X 2.08 x 2.00 X IO-’ 5.83 x 1.12 2.14 0.005 Pre. Obs. 0.99999067 0.99999164 7.57 7.57 2.97 x 10-4 2.73 X 2.00 x 10-3 1.28 X lo-’ 5.61 6.65 0.01 Pre. Obs. 0.99999533 0.99999559 7.57 7.57 2.97 x 1 0 - ~ 2.85 X 2.00 x 10-3 1.55 X lo-’ 1.12 x 10 1.23 X 10 0.05 Pre. Obs. 0.99999907 0.99999908 7.57 7.57 2.97 x 2.97 X 2.00 x 10-3 1.89 X IO-’ 5.61 x I O 5.76 X 10 0.1 Pre. Obs. 0.999999533 0.999999536 7.57 7.57 2.97 x 10-4 3.01 x 1 0 - ~ 2.00 x 1 0 - ~ 1.94 x io4 1.12 x 1 0 2 1.15 x 102 0.5 Pre. Obs. 0.9999999067 0.9999999068 7.57 7.57 2.97 3.24 2.00 x IO-’ 2.01 X 5.61 x IO2 6.07 X lo2 X X IO-* TABLE 3 Effects of varying the intensity of stabilizing selection (V,) for 1.1 and m 2 = 0.05, given 1.18 = 0.1 2 = P = 0.5 Pre. Obs. 0.999999232 0.999999240 1.13 1.13 2.44 2.45 5 Pre. Obs. 0.9999991 14 0,999999 124 3.57 3.57 2.82 x 10-4 2.82 X 1.00 x 10-3 9.72 X 5.91 x I O 6.07 x 10 10 Pre. Obs. 0.99999907 0.99999908 7.57 7.57 2.97 x 10-4 2.97 X 2.00 x I O + 1.89 X IO-’ 5.61 x I O 5.76 X 10 15 Pre. Obs. 0.99999905 0.99999906 11.56 11.56 3.03 x 3.02 X 3.00 x 1 0 - ~ 2.75 X lo-’ 5.51 x I O 5.66 X 10 20 Pre. Obs. 0.99999904 0.99999905 15.56 15.56 3.05 3.04 4.00 3.57 5.46 5.61 Pre. Obs. 0.99999903 0.99999904 39.50 39.50 3.11 x 1 0 - 4 3.10 x IO-‘ 50 4.00 3.98 X X X X loT4 6.82 6.99 X X X IO-’ X LOO x 10-2 7.73 x IO-’ X X 10 10 X 10 X 10 5.37 x 10 5.52 x I O DISCUSSION Apart from the decline in heritability that causes plateaux in some experiments (e.g., BROWNand BELL1961; ROBERTS 1966), a reduction in selection differential seems to be responsible for some other selection limits, particularly in long-term selection experiments (see FALCONER 1981, chapter 12, for review). It has been observed that a decline in mean was rapid after relaxation of selection, and that fertility and reproductivity were much poorer in many lines-even to the extent of causing extinction of selection lines-in many long-term selection experiments (e.g., LERNERand DEMPSTER 1951; CLAYTON and ROBERTSON1957; LATTER1966; ROBERTS1966; WILSONet al. 1971; YOO, NICHOLAS and RATHIE 1980). This clearly indicated that additive genetic var- 1324 Z.-B. ZENG AND W. C . HILL TABLE 4 Effects of varying the proportion of truncation selection (P) for and m 2 = 0.05, given pi = 0.1 t P U p = V, = 10 V 72 8.33 9.39 1 .o Pre. Obs. 0.9999937 0.9999942 0.00 0.00 2.00 x 1.89 x l o + 0.9 Pre. Obs. 0.99999825 0.99999829 1.85 1.85 5.57 x 10-4 5.50 x 10-4 2.99 3.11 0.7 Pre. Obs. 0.99999887 0,99999889 4.71 4.71 3.59 x 1 0 - ~ 3.58 x 10-4 4.64 X 10 4.78 X 10 0.5 Pre. Obs. 0.99999907 0,99999908 7.57 7.57 2.97 x 1 0 - ~ 2.97 x 10-4 5.61 5.76 X 0.3 Pre. Obs. 0.999999 176 0.999999184 1 1 .oo 11.00 2.62 x 1 0 - ~ 2.63 x 10-4 6.35 6.52 X 10 X 10 0.1 Pre. Obs. 0.999999258 0.999999265 16.65 16.65 2.36 x 1 0 - 4 2.37 x 1 0 - 4 7.06 7.24 X X X X X 10 10 10 10 10 10 iance was not exhausted in these populations at apparent limits and also suggested that natural selection opposing artificial selection might cause the reduction in the effective selection differential. In this paper, a conflict between natural and artificial selection is examined. It is shown that stabilizing and truncation selection can take a large population to a selection limit where the population mean is determined by the intensities of truncation and stabilizing selection and the phenotypic variance of the character (9). There is still genetic variance at this limit, and its amount can be approximated by (37) in terms of the mutation-selection balance. However, it is necessary to examine whether this balance can account for high levels of genetic variation observed in some long-term selection experiments. T h e balance between mutation and stabilizing selection has been examined in detail by many authors. Both LANDE'S(1975) and TURELLI'S (1984) mathematical calculations and parameter estimations from relevant data suggested that high levels of variation could be maintained by mutation in the face of stabilizing selection. However, when additional truncation selection is taken into account in the model, this balance may or may not be sufficient to explain the maintenance of genetic variability. Figure 1 shows the ratio of genetic variances maintained by the balance with and without truncation selection for different values of u2/a2 and P. It can be seen that truncation selection is a crucial factor in quantifying the equilibrium genetic variance. Even a small amount of selection can severely reduce the variance. But when selection is strong, any further increase in the strength of selection has little further influence on the variance. Relevant data for estimating n p and w'/u' have been reviewed by TURELLI (1984), who considered that n p =: 0.01 and w2/o' =: 510 might be typical estimates for many quantitative characters. With those values, the house-of-cards approximation predicts heritabilities (h2 = 4np(a2 + w')/[u' + K ( K - Z)w']) ranging from 0.240-0.440 without truncation selection, but 0.098-0.1 13 for P = 0.9 and 0.057-0.060 for P = 0.5. This seems to LIMIT TO SELECTION 1325 1. v) al ;0. ld .r( c g 0. CI 0 0 +-' .r( 0. (d p: FIGURE1.-The ratio of genetic variances maintained by the balance with and without truncation selection is plotted against the proportion of truncation selection (P%)for different values of w'/u'. The ratio is equal to [ l K ( K - Z)w2/aZ]-'. + suggest that, in the presence of opposing truncation selection, the heritability can only be maintained at a low level with np = 0.01. In this case the change in the intensity of stabilizing selection does not make a significant difference to the variance. However, if n p is 0.02, rather than 0.01, the heritability at the selection limit would be about 0.303-0.088 for P = 0.95-0.01 and u2/a2 = 10. Clearly, the argument relies on the estimation of the relevant parameters, particularly the total mutation rate of the loci controlling the character concerned. In view of some experimental evidence that indicates that mutations in the broad sense occurring during the experiments from whatever sources may have contributed to the long-term responses (FRANKHAM1980) and that plateaux have not been achieved in some long term selection experiments, for example the Illinois corn experiments (DUDLEY1977), the present analysis is not a complete description of the process. In due course it will be necessary to synthesize theories in predicting responses from mutations in the absence of natural selection (HILL 1982), the present theory incorporating stabilizing selection at the phenotypic level, and theories of natural selection acting at the genetic level through heterozygote superiority. Nevertheless, it is hoped that the calculation presented here will be of value in interpreting some long-term selection experiments which reached a selection limit. We thank M. TURELLI,R. LANDEand M. LYNCHfor their helpful comments on an earlier for his reading of the APPENDIX. draft, and R. THOMPSON LITERATURE CITED BROWN, W. P. and A. E. BELL,1961 Genetic analysis of a "plateaued" population of Drosophila melanogaster. Genetics 46: 407-425. BULMER,M. G., 1972 The genetic variability of polygenic characters under optimizing selection, mutation and drift. Genet. Res. 19: 17-25. 1326 Z.-B. ZENC AND W. G. HILL BULMER,M. G., 1973 T h e maintenance of the genetic variability of polygenic characters by heterozygous advantage. Genet. Res. 22: 9-1 2. BULMER, M. G., 1980 The Mathematical Theory of Quantitative Genetics. Clarendon Press, Oxford. CLAYTON,G. A. and A. ROBERTSON,1957 An experimental check on quantitative genetical theory. 11. T h e long-term effects of selection. J. Genet. 55: 152-170. J. W., 1977 76 generations of selection for oil and protein percentage in maize. pp. DUDLEY, 459-473. In: Proceedings of the International Conference on Quantitative Genetics, Edited by E. POLLAK, 0. KEMPTHORNE and T. B. BAILEY,JR. Iowa State University Press, Ames. FALCONER, D. S., 198 1 Introduction to Quantitative Genetics, Ed. 2. Longman, London. FISHER,R. A., 1930 The Genetical Theory of Natural Selection. Clarendon Press, Oxford. FLEMING,W. H., 1979 Equilibrium distributions of continuous polygenic traits. SIAM J. Appl. Math. 36: 148-168. R., 1980 Origin of genetic variation in selection lines. pp. 56-68. In: Selection ExFRANKHAM, periments in Laboratory and Domestic Animals, Edited by A. ROBERTSON. Commonwealth Agricultural Bureaux, Slough. GILL, P. E., W. MURRAY and M. H. WRIGHT,1981 Practical Optimization. Academic Press, London. GILLESPIE,J. H., 1984 Pleiotropic overdominance and the maintenance of genetic variation in polygenic characters. Genetics 107: 321-330. J. B. S., 1954 T h e measurement of natural selection. Proc. 9th Int. Congr. Genet. (Part HAIDANE, 1, Cargologia) 6 (Suppl.): 480-487. HILL,W. G., 1982 Predictions of response to artificial selection from new mutations. Genet. Res. 40: 255-278. JAMES,J. W., 1962 Conflict between directional and centripetal selection. Heredity 17: 487-499. KIMURA,M., 1965 A stochastic model concerning the maintenance of genetic variability in quantitative characters. Proc. Natl. Acad. Sci. USA 54: 731-736. J. F. C., 1978 A simple model for the balance between selection and mutation. J. KINGMAN, Appl. Probab. 15: 1-12. KOJIMA,K., 1959 Stable equilibria for the optimum model. Proc. Natl. Acad. Sci. USA 45: 989993. LANDE,R., 1975 T h e maintenance of genetic variability by mutation in a polygenic character with linked loci. Genet. Res. 2 6 221-235. LATTER, B. D. H., 1960 Natural selection for an intermediate optimum. Aust. J. Biol. Sci. 13: 30-35. LATTER,B. D. H., 1966 Selection for a threshold character in Drosophila. 11. Homeostatic behavior on relaxation of selection. Genet. Res. 8: 205-218. LATTER, B. D. H., 1970 Selection in finite populations with multiple alleles. 11. Centripetal selection, mutation, and isoallelic variation. Genetics 66: 165-1 86. LERNER,I. M., 1950 Population Genetics and Animal Improvement. University Press, Cambridge. LERNER,1. M., 1954 Genetic Homeostasis. Oliver & Boyd, Edinburgh. LERNER,I. M. and E. R. DEMPSTER,1951 Attenuation of genetic progress under continued selection in poultry. Heredity 5: 75-94. R. C., 1964 T h e interaction of selection and linkage. 11. Optimum models. Genetics LEWONTIN, 50: 757-782. LINNEY,R., B. W. BARNESand M. J. KEARSEY,1971 Variation for metrical characters in Drosophila populations. Heredity 27: 163-1 74. 1327 LIMIT TO SELECTION NICHOLAS, F. W. and A. ROBERTSON, 1980 The conflict between natural and artificial selection in finite population. Theor. Appl. Genet. 5 6 57-64. ROBERTS,R. C., 1966 The limits to artificial selection for body weight in the mouse. 2. The genetic nature of the limits. Genet. Res. 8: 361-375. ROBERTSON, A., 1956 The effect of selection against extreme deviants based on deviation or on homozygosis. J. Genet. 5 4 236-248. ROBERTSON, A., 1960 A theory of limits in artificial selection. Proc. Roy. Soc. Lond. (Biol.) 153: 234-249. TURELLI, M., 1984 Heritable genetic variation via mutation-selection balance: Lerch's zeta meets the abdominal bristles. Theor. Pop. Biol. 25: 138-193. VERCHESE, M. W., 1974 Interaction between natural selection for heterozygotes and directional selection. Genetics 7 6 163-168. WILSON,S. P., H. D. GOODALE, W. H. KYLE and E. F. GODFREY, 1971 body weight in mice. J. Hered. 62: 228-234. Long-term selection for WRIGHT,S., 1935 Evolution in populations in approximate equilibrium. J. Genet. 3 0 257-267. YOO, B. H., F. W. NICHOLAS and K. A. RATHIE,1980 Long-term selection for a quantitative character in large replicate populations of Drosophila melanoguster. 4. Relaxed and reverse selection. Theor. Appl. Genet. 57: 113-1 17. Communicating editor: W. J. EWENS APPENDIX From (32), (24) and ( 2 5 ) , we have where [ is a constant, such that and In this APPENDIX, we prove U ; = B in (Al). First, we need to clarify the range of the constant p; is a small value, this implies that 0<1 always, which suggests that 0 [ 1 -t - C; exp(- C;. Since ( A l ) is a density function and (z; - B)'/(2A)} < [ < 1. T h e n it follows that exp(- VI]' exp(- = 1. n(zi2A - B)* ?l=O Therefore, (AI) can be written as A + nm2 A + nm? 1328 Z.-B. ZENG AND W . G . HI1.L after some calculation. Note that the last two terms in the right-hand side of (A4) define a normal density function. By using this property and putting (A4) into (A2) and (A3), we then obtain and and Let and b, = a, A + nm?B/u, A + nmg ' (A5) and (A6) are then equivalent to m m a, = 1 b, = -. A n=o "=O cr=o Sufficiency: It is easy to show that a , is a monotonically decreasing series with positive terms. Then, if U , > R , it follows that a, > b, for n = 1 , 2, . . . , except n = 0, where a,, = bo. Consequently, c:=,) a, > cr=p=o b, which contradicts the condition of (A7). Similarly, if U , < B , c,"=~ a, < c,"=p=o b,, which also contradicts the condition of (A7). While, when U , = B, a , = b, holds for every n = 0, 1 , 2, . . .; therefore, (A7) is satisfied. Necessity: (A5) and (AS) can be rewritten as and A n ( u , - B)' A + nm:B/u, = rd,, e x p \ - 2(A 4- n m f ) j A + nm: "=n J io'[ X I I/' which are two power series. According to the identity theorem for power series, if the two power series m 2 n=o m yc. and rd, n=o have the sanie sun1 in an interval 0 < [ < 1 (in this case) in which both of them converge, then the t w o series are entirely identical. That is to say, for every n = 0, I , 2, . . ., c, = d,, since (AS) and (A9) do converge with 0 < f < 1 . I t then gives U , = B always. Thus, the proof is completed.