1
1
Incorporating economic weights into Radiata pine breeding selection decisions
2
David C. Evison* and Luis A. Apiolaza
3
4
School of Forestry, University of Canterbury, Private Bag 4800, Christchurch, New Zealand.
5
*Corresponding author: David.Evison@canterbury.ac.nz, telephone: +64-3-364 2987
6
2
7
Abstract
8
This article introduces the concept of “robust selection”, which proposes selection based on the
9
stochastic simulation of economic values to account for the inherent uncertainty of economic
10
weights used in tree selection for breeding programmes. The proposed method uses both median
11
ranking and variability of ranking as criteria for breeding selection. Using consensus genetic and
12
economic parameters from the New Zealand Radiata Pine Breeding Company programme we
13
compare three selection strategies: deterministic application of economic weights from a vertically-
14
integrated bio-economic model, an equal-weight index often used in operations and robust
15
selection. All strategies aim to increase value for a breeding objective that includes volume, stem
16
sweep, branch size and wood stiffness based on a selection index that considers stem diameter,
17
straightness, branching score, wood density and modulus of elasticity.
18
Two-thirds of the selected trees were unique for each of the strategies. Robust selection achieved
19
the best realised gain for three out of the four selection criteria, and is the middle performer in the
20
other one. Considering the large intrinsic uncertainty of economic weights, we suggest that the
21
relevant criterion for selection of individuals is the maximum median ranking, subject to an
22
acceptable level of variation in that ranking, rather than their narrow performance under a single
23
economic scenario. This will lead to selections that perform well under a wide range of economic
24
circumstances.
25
26
27
Key words: breeding objectives, economic weights, simulation, economic evaluation, risk analysis
3
28
Introduction
29
The selection of superior trees is central to breeding programs, aiming to identify individuals with
30
maximum economic-genetic value for a combination of characteristics of economic importance.
31
These trees are mated to produce the next generation and deployed in commercial forests. Optimal
32
selection decisions depend not only on the economic relevance of each trait, but also on many
33
estimates of their genetic parameters, including heritabilities and genetic correlations (Hazel 1943,
34
Schneeberger et al. 1992, Van Vleck 1993).
35
The New Zealand forest industry has used selection indices since the mid-1970s (Burdon 1979), but
36
initially focussed on collecting data to estimate genetic parameters, while relying on crude estimates
37
of economic weights; for example, 2 for growth and 1 for basic density (see Cotterill and Jackson
38
1985 for alternative methods). Although the incorporation of economic information into the
39
selection of trees for breeding is supported both on theoretical and logical grounds, it has not been
40
implemented fully by tree breeders (see Apiolaza and Greaves 2001 for common reasons). Formal
41
estimation of economic weights was first attempted for eucalypts in the early 1990s (Borralho et al.
42
1993), while modelling the economic importance of Radiata pine traits started in the late 1990s (e.g.
43
Chambers and Borralho, 1999, Apiolaza and Garrick, 2001, Ivkovic et al., 2006a). In spite of a
44
significant volume of research on this topic, researchers and industry still feel more comfortable (or
45
less uncomfortable) dealing with genetic parameters than with economic weights. Both genetic and
46
economic parameters are subject to substantial uncertainty, for the following reasons.
47
1. The genetic architecture of the traits; their variability, genetic control and association
48
between traits is also subject to uncertainty, because they are estimates that are obtained
49
from different samples of genotypes, growing in different sites and over a long period of
50
time.
4
51
2. There are gaps in the available genetic information, particularly if a trait has only recently
52
been identified as important, and filling those gaps may take decades; particularly for age-
53
age correlations.
54
3. Economic values are subject to a high degree of uncertainty, as they will be realised several
55
decades in the future. Even for relatively fast-growing tree species such as Radiata pine, the
56
lag between breeding decision-making, and realisation of value from harvested trees can be
57
fifty years or more.
58
59
60
61
4. The forest growers, who are the ultimate customers of tree breeders, may have different
views of the future, different silvicultural strategies and different target products or markets.
5. A particular selection may be deployed for a number of years, and over the period that this
population is harvested, the technology and market requirements may change.
62
While the uncertainty associated with genetic parameters might be reduced through additional data
63
and analysis, the uncertainty in economic parameters is unavoidable, because it is generated by the
64
necessarily long time frames from selection to harvest of the improved crop, and the different views
65
of the future among forest growers who will use the improved seed. The long time frames make this
66
a much larger issue for tree breeding than for the animal species where selection index theory was
67
initially developed. Selection methodologies that account for this inherent uncertainty in tree
68
breeding are required.
69
Bio-economic models (which show the relationship between changes in the management of the
70
resource, and the subsequent quality and quantity of output from the forest) have been the most
71
popular approach to estimate economic weights in tree breeding. Ivkovic et al. (2006a), constructed
72
separate bio-economic models for a forest grower, a processor and an integrated firm, obtaining
73
different values for economic weights, mirroring the experience of animal breeders, as discussed by
74
Goddard (1998). For a discussion of other methods of calculating economic weights in tree breeding,
75
see Alzamora (2010).
5
76
This paper outlines an alternative approach to incorporating economic information into selection
77
decisions. We use genetic and economic data for Radiata pine in New Zealand to first calculate
78
deterministic selection index coefficients. This is followed by the introduction of a simulation
79
approach to the use of economic weights and proposal for the selection of “robust” genotypes,
80
which are stable (in the sense of being highly-ranked performers) under the likely variation in values
81
for the economic weights.
82
Materials and methods
83
Selection indices combine predicted breeding values (si, often BLUP) of multiple traits for each
84
individual to calculate an index I on which to base selection:
85
I = b1s1 + b2s2 + …. + bnsn = b’s
86
where s is a vector of breeding values for selection criteria (often assessed at 1/4 to 1/3 of rotation)
87
and b is the vector of index coefficients. While selecting on index values, breeders aim at maximising
88
the breeding objective function H, a linear combination of the genetic values for each trait (ai)
89
weighted by their economic values vi (value per unit of increase in the trait):
90
H = v1a1 + v2a2 + …. + vnan = v’a
91
where v is the vector of economic values and a is the vector of genetic values for the traits we wish
92
to improve (often at rotation age). Therefore, H represents the genetic-economic worth of an
93
individual. The index coefficients (b) that maximize the correlation between I and H are calculated as
94
(Schneeberger et al. 1992):
95
b = Gss-1 Gso v
96
for Gss and Gso the addititive covariance matrices for selection criteria, and between selection criteria
97
and objective traits respectively, and the b coefficients combine genetic and economic information
(1)
6
98
(heritabilities, genetic correlations and economic weights). Equation 1 is an extension of Hazel’s
99
(1943) and van Vleck’s (1993) work.
100
Selection indices link characteristics that breeders would like to improve (objective traits in H) with
101
the ones they assess (selection criteria in I). This distinction is very relevant for tree breeders, who
102
often use criteria expressed early in the life of the trees and that are easier to assess than objective
103
traits. For example, radiata pine progeny trials are routinely assessed for dbh at age 8 rather than for
104
volume at rotation age.
105
These indices assume that Gss, Gso and v are known with certainty; however, we have already
106
pointed out that genetic and, particularly, economic parameters are subject to much uncertainty.
107
Because the sources of uncertainty of economic parameters are largely related to the long time
108
frames, and the different views of the future that may be held by those who plant improved seed,
109
they are not able to be removed through further analysis. Therefore, selection decisions should
110
recognise the inherent uncertainty in the estimation and application of economic weights. What is
111
required is the best estimate of the economic value of the trait, and an estimate of the uncertainty
112
in these estimates.
113
We propose a method, which we call “robust selection”, which uses i) economic weights drawn
114
from distributions centred around published values to calculate selection index coefficients and ii)
115
selecting individuals that have both high rankings and low variability of rankings. Therefore, the
116
selection process will involve n times of 1) drawing a set of economic weights, 2) calculating
117
coefficients using Equation 1, 3) applying the index to all trees, 4) ranking the trees. Trees are then
118
evaluated on the basis of their median ranking across all draws of economic weights, subject to a
119
variability of less than a predetermined threshold.
120
Robust selection will be compared to the deterministic application of economic weights calculated
121
by Ivkovic et al. (2006), using the estimates for the integrated direct sawlog option presented in
7
122
Table 1, and an equal weight index often used by the New Zealand Radiata Pine Breeding Company
123
(where all coefficients of b are equal to 1) for selecting trees for breeding from an elite population.
124
125
We assume that all genetic parameters in Gss and Gso are known without error to focus the
126
comparison only on varying economic weights. Robust selection was implemented using the R
127
statistical system (R Core Team 2014) and both the code and the breeding values used in the test are
128
available as complementary material for this article.
129
To illustrate the proposed method we have assumed, for each trait with a positive mean value, a
130
minimum value of zero, and a maximum value of twice the mean value. For a trait with a negative
131
mean value, the maximum value was assumed to be zero, and the minimum value was assumed to
132
be twice the mean value. A triangular distribution was used for each trait (see Figure 1).
133
While this assumed level of variation is somewhat arbitrary, it is supported by the variation in
134
economic weights from bio-economic models (see for example Table 1 above). It seems intuitively
135
reasonable to assume for wood quality traits that, at the minimum point on the distribution, the
136
trait has no economic value. To ensure a symmetrical distribution, the maximum point on the
137
distribution is a value double the mean estimate.
138
139
140
141
142
143
8
144
145
146
The economic weights shown in Table 1 above were used in the analysis, except that for volume
147
they were changed from value for m3/tree to m3/ha by using a scaling factor of 350 trees/ha to
148
expand volume from tree level to hectare level, matching Ivkovic et al. (2010).
149
Genetic parameters
150
“Consensus” or “accepted” genetic parameters for selection criteria and objective traits for the New
151
Zealand Radiata pine breeding program are presented in Tables 2 to 4. This information is exactly as
152
presented by Ivkovic et al. (2010); however, Table 4 contains updated phenotypic variances for
153
objective traits, as they were incorrectly reported in 2010 (Kumar S, personal communication).
154
155
Breeding values
156
We will test the performance of the different selection indices by applying them to a preliminary set
157
of 481 candidates of a 2007 elite population. This population was a selection of (more or less) 10
158
individuals from 50 families, from the 268 Progeny trial, Compartment 1350, Kaingaroa Forest (Dr
159
Paul Jefferson, pers. comm.) For each tree there are predicted breeding values (univariate BLUP,
160
expressed as deviation from the overall population mean) for stem diameter at breast height, stem
161
straightness, branching habit, wood basic density and modulus of elasticity (estimated using time-of-
162
flight acoustic tools). The data set is presented as supplementary material, although the genetic
163
identities have been changed.
164
Results
165
We will show the results using three alternatives for incorporating economic weights into selection
166
decisions. Selection Index coefficients were calculated using deterministic economic weights for a
9
167
direct sawlog vertically integrated regime and equal index weights using data from Tables 1 to 4. The
168
indices were applied to the breeding values for all genotypes, which were then ranked in decreasing
169
order of index value. This is the standard implementation of economic weights in a breeding
170
program. Figure 2 shows a simple ranking based on index value, with some obvious candidates for
171
selection, some obvious candidates for rejection and a majority of genotypes with near-average
172
performance, where the economic penalty for choosing the next-lowest ranked candidate is
173
relatively small.
174
In the case of robust selection the assumed triangular distribution of economic weights was used to
175
calculate the median ranking and the range of rankings for these selections (Figure 3). It should be
176
noted that the ranking distributions are highly skewed – we are selecting from those individuals at
177
the superior end of the curve, so most of their index values are high. Figure 3 shows that some of
178
the selections made by maximising selection index value are ranked highly throughout the range of
179
economic circumstances represented by the probability distributions described above (for example,
180
trees 140, 214 and 126), others are ranked much more poorly in some circumstances (for example
181
trees 151, 18 and 62).
182
183
Robust selection targets genotypes that avoid the wide range of rankings displayed by some of the
184
selections chosen by simply maximising selection index value. It does this by selecting the individuals
185
with the largest median ranking, but making that selection subject to a threshold acceptable
186
variability in ranking.
187
The second selection reduces the variability considerably. If we look at the top 42 trees in terms of
188
their average index value (Figure 4), we can see again that there are a few top-performers with high
189
rankings and low variability, but below that point there is a clear trade-off between median ranking
190
and variability of ranking
10
191
Comparing conventional selection using economic weights with robust selection and equal weight
192
selection (another method often employed by tree breeders) shows that assumptions of future
193
economic circumstances certainly makes a difference in terms of the trees selected under each
194
strategy (Figure 5). Only 4 out of the top 15 trees are common to all strategies, while 4 to 6 trees are
195
common to two strategies. In other words, 2/3 of the selections are unique for each of the
196
strategies.
197
198
Table 5 shows the genetic gain achieved under the three strategies described above – robust
199
selection provides the best gain for three out of the four traits, and is the middle performer in one.
200
When compared with the deterministic economic weights for a direct sawlog regime, it performs
201
better for all traits except volume. Equal weights is not recommended, as the selection of values for
202
weights is entirely arbitrary, ignoring all economic information and genetic information generated by
203
progeny trials.
204
Discussion
205
Robust selection uses two decision criteria that are absent from the standard implementation of
206
economic weights. The implementation of the method focuses on ranking, rather than selection
207
index value. It is fundamental to the method to select individuals with a low variability of ranking.
208
These are the individuals that perform most reliably under the full range of economic circumstances.
209
Tree breeders always wish to select the individuals that rank the highest. In robust selection we
210
propose selecting those that have the maximum median ranking, subject to a pre-determined
211
variability of ranking. Figure 6 shows that both very high and very low-ranked individuals tend to
212
have a low variability of ranking.
213
11
214
In order to implement robust selection, a probability distribution of economic weights is required.
215
While these distributions have been assumed in the example shown above, they could be provided
216
from the results of sensitivity analysis conducted using a bio-economic model. A sensitivity analysis
217
of this type would include historic variation in log prices and other data inputs. In addition, we have
218
assumed no correlations between the economic weights. If these correlations did exist, however and
219
were known they could be accommodated using the standard tools of risk analysis.
220
A useful selection tool should 1) provide appropriate weightings to reflect the relative importance of
221
a number of different traits, 2) recognize the inherent uncertainty in economic information,
222
generated by the long time frames in tree breeding, in both the breeding and deployment cycles,
223
and the different objectives of stakeholders (very relevant for breeding programs involving multiple
224
firms), and 3) lead to agreement among members on economic weights for breeding selection. The
225
process of implementing economic weights should include further analysis to investigate the impact
226
of different choices of probability distribution and mean value of economic weights.
227
As has been noted by Hoogstra and Schanz (2008), there is a tendency among decision makers in
228
forestry and other business areas to ignore uncertainty. This indicates a potential for much wider
229
application of the methods outlined in this paper. For example, the uncertainties in the economic
230
values also exist for the genetic parameters. We have assumed perfect knowledge of genetic
231
parameters to emphasize the effect of uncertainty present in economic information, but the same
232
approach could be expanded to incorporate knowledge of the uncertainty of these parameters as
233
well. The methods outlined in this paper could also be used to evaluate silvicultural and other
234
investment options in forestry where there is uncertainty about future values of input data.
235
236
Conclusion
237
There has been an extensive research investment in New Zealand to enhance the selection of
238
improved trees to increase the competitive advantage of radiata pine, and enhance the species
12
239
suitability and market acceptance in a range of end uses. Breeding theory requires that the optimal
240
selection of multiple traits uses economic weights to specify the value of each trait. The focus of
241
implementation of economic weights to date has been to choose the individuals that maximise the
242
index value. Where the economic weights and other inputs into the calculation of the index are
243
known with certainty this method will provide the highest ranked candidates.
244
Nevertheless, where input values are not certain, we contend that the relevant criterion for
245
selection of individuals is the maximum median ranking, subject to an acceptable level of variation in
246
that ranking. This will provide the selections that are most likely to be high-ranked, under the full
247
range of expected future economic values. We have shown that there are a number of reasons why
248
there is inherent uncertainty in any estimate of economic weights. We have also provided a method
249
to implement selection for breeding that will minimise the impact of this uncertainty on the value of
250
the final crop. This approach is consistent with investment theory which proposes that we should
251
not only consider the expected value of the return from an investment, but also the variability of
252
that return, as a measure of risk.
253
The investment in tree improvement is significant, and the methods that we are proposing will
254
provide a more defensible selection for use by the industry in the future, and will lead to a more
255
complete adoption of a rigorous method of choosing among investment alternatives, taking account
256
of uncertain information.
257
258
Acknowledgements
259
260
This research was funded by the Radiata Pine Breeding Company Ltd. The support of Dr John Butcher
261
and Dr Paul Jefferson, and the Technical Committee of the Radiata Pine Breeding Company is
262
gratefully acknowledged
263
13
264
References
265
Alzamora RM (2010) Valuing breeding traits for appearance and structural timber in radiata pine.
266
PhD thesis, School of Forestry, University of Canterbury. http://hdl.handle.net/10092/5077
267
Apiolaza LA and Garrick DJ (2001) Breeding objectives for three silvicultural regimes of radiata pine.
268
Canadian Journal of Forest Research 31: 654-662.
269
Apiolaza LA and Greaves BL (2001) Why are most breeders not using economic breeding objectives?
270
In IUFRO Conference Developing the Eucalypt of the Future, Valdivia, Chile.
271
Borralho NMG, Cotterill PP and Kanowski PJ (1993) Breeding objectives for pulp production of
272
Eucalyptus globulus under different industrial cost structures. Canadian Journal of Forest Research
273
23: 648-656.
274
Burdon RD (1979) Generalisation of multi-trait selection indices using information from several sites.
275
New Zealand Journal of Forestry Science 9: 145-152.
276
Chambers PGS and Borralho NMG (1999) A simple model to examine the impact of changes in wood
277
traits on the costs of thermomechanical pulping and high-brightness newsprint production with
278
radiata pine. Canadian Journal of Forest Research 29: 1615-1626.
279
Cotterill PP and Jackson N (1985). On index selection I. Methods of determining economic
280
weight. Silvae Genetica 34: 56-63.
281
Goddard ME (1998) Consensus and debate in the definition of breeding objectives. Journal of Dairy
282
Science, 81(Suppl 2): 6–18.
283
Hazel LN (1943) The genetic basis for constructing selection indexes. Genetics 28: 476–490.
284
Hoogstra, M.A.; Schanz, H. (2008): How (Un)Certain is the Future in Forestry? A comparative
285
Assessment of Uncertainty in the Forest and Agricultural Sector. Forest Science, 54 (3), 316-327.
14
286
287
Ivkovic M, Kumar S and Wu H (2006) Development of Breeding Objectives for Structural and
288
Appearance products: Bio-economic model and economic weights for structure timber production in
289
New Zealand. RPBC Client Report IO003 (unpublished)
290
Ivkovic M, Wu HX, McRae TA and Powell MB (2006a) Developing breeding objectives for radiata pine
291
structural wood production. I. Bioeconomic model and economic weights. Canadian Journal of
292
Forest Research 36: 2920-2931.
293
Ivkovic M, Wu HX, McRae TA and Powell MB (2006b). Developing breeding objectives for radiata
294
pine structural wood production. II. Sensitivity analyses. Canadian Journal of Forest Research 36:
295
2920-2931.
296
Ivkovic M, Wu HX and Kumar S (2010) Bio-economic modelling as a method for determining
297
economic weights for optimal multiple-trait tree selection. Silvae Genetica 59: 77-90.
298
R Core Team (2014) R: A Language and Environment for Statistical Computing. R Foundation for
299
Statistical Computing, Vienna, Austria. http://www.R-project.org
300
Schneeberger M, Barwick SA, Crow GH and Hammond K (1992) Economic indices using breeding
301
values predicted by BLUP. Journal of Animal Breeding and Genetics 107: 180–187.
302
Van Vleck LD (1993) Selection index and introduction to mixed model methods. CRC Press, Boca
303
Raton.
304
15
305
Table 1: Economic weights for New Zealand radiata pine.
Viewpoint
Regime
Vol
SWE
BRS
MoE
Grower
Direct sawlog
12.42
-40.88
-9.2
387.6
Min. tending
9.031
-53.85
-63.06
497.4
Direct sawlog
-0.0065
-2.958
-0.416
8.25
Min. tending
0.0346
-6.5
-2
14.25
Direct sawlog
12.14
-499.3
-54.94
700.1
Min. tending
17.97
-1087
-186.5
1242
Sawmiller
Integrated
306
307
308
Table 2: Additive genetic correlation among selection criteria.
DBH08
DBH08
STR08
BR08
DEN08
MOE08
1
0.05
0.29
-0.25
-0.35
1
0.15
0.05
0.05
1
-0.05
-0.10
1
0.48
STR08
BR08
DEN08
MOE08
1
309
310
311
312
Table 3: Additive genetic correlations between selection criteria and objective traits.+
VOL25
SWE25
BIX25
MOE25
DBH08
0.70
0.10
0.45
-0.30
STR08
0.14
-0.70
0.05
-0.10
BR908
0.15
0.05
-0.70
-0.11
DEN08
-0.19
0.09
-0.11
0.45
MOE08
-0.20
-0.25
-0.14
0.70
16
313
Table 4: Heritability and variance components for selection criteria and objective traits.
Trait
h2
s 2p
s a2 = s 2p h2
DBH08
0.18
1100
198
STR08
0.25
3.8
0.95
BR08
0.35
4.7
1.65
DEN08
0.65
552
358.8
MOE08
0.55
2.08
1.14
VOL25
0.25
0.3
0.08
SWE25
0.14
0.49
0.07
BIX25
0.15
95
14.25
MOE25
0.40
1.7
0.68
314
315
Table 5: Breeding value gain under the 3 candidate strategies
Deterministic
Equal weights
Robust selection
316
317
Diameter
21.03
9.69
15.47
Straightness
-0.08
-0.34
0.20
Branch size
1.23
0.80
1.25
Stiffness
0.81
0.71
0.88
17
318
Figure 1: Assumed distributions for economic weights
319
Figure 2: Selection index values for all 482 individuals, ranked in decreasing order
320
Figure 3: Violin plots of the distribution of ranking positions for the top 10 selections, based on
321
maximising selection index value under 1000 scenarios of economic weights.
322
Figure 4: Top 42 individuals based on selection index value
323
Figure 5: Trees selected under different strategies
324
Figure 6: RPBC elite population rankings, triangles represent trees chosen by robust selection
325
326
327
328
329
330
331
332
333
334
335
336
337
338
Min
Volume
0
Sweep
-998.6
Branch size -109.88
Stiffness
0
Mean
12.14*350
-499.3
-54.94
700.1
Max
24.28*350
0
0
1400.2
Vertically integrated direct sawlog regime index ($/ha)
18
●
●
●
●
●
2000
●
●
●
●●●
●●
●●●
●●●●
●●●
●●●
●●●
●●●●●●●
●●●●●
●●●●
●●●●●●●
●●
●●●●●●●
●●●●
●●●●●●●●
●●●●●
●●●●●●●●●●●
●●●●●●●●●●●●●
●●●●●●
●●●●●●●●●
●●●●●●●●●●●
●●●●●●●
●●●●●●●●●●
●●●●●●●●●●
●●●●●●●●●
●●●●●●●●
●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●
●●●●●●●●●●●●
●●●●●●●●●●●●●●●
●●●●●●●●●
●●●●●●●●
●●●●●●●●●●●●●
●●●●●●●●●●●●●●●
●●●●●●●●●
●●●●●●●●●●●●●●
●●●●●●
●●●●●●●●●●
●●●●●●
●●●●●
●●●●●●●●●●
●●●●●●●●●●●●●
●●●●●●●●●
●●●●●●●●
●●●●●●●
●●●
●●●
●●●●●●
●●●●●
●●●●●●●
●●●●●
●●●●●●●
●●●●●●
●●●●●
●●●●●●●
●
●●●●●●●
●●
●●●●
●●●●
●
●
1000
0
−1000
●
●●
●
●
0
100
200
300
400
500
Ranking
339
340
250
200
Ranking
150
100
50
●
●
●
●
150
137
368
●
●
341
●
●
●
●
140
125
214
●
●
246
179
●
●
●
240
126
Genotype ID
334
146
123
232
113
19
●
●
●
●
400
●
●
●
●
●
●
●
Ranking range
●
●
300
●
●
●
●
●
●
●
200
●
●
●
●
●
●
●
●
●
●
●
●
100
●
●
●
●
●
●
●
●
●
●
0
342
343
344
10
20
Median ranking
30
40
Standard deviation of ranking over 1000 iterations
20
●
●
346
347
●
●
●
●
●
●
●
●
●
●
●
50
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
● ●● ●●
●
●
●● ●● ●
● ●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●●
● ●
●
●●
●
●
● ●
●
●
●
●
● ●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
● ● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
●
● ●
● ●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
100
●
●
●
●●
● ●●●
● ●
●
●
●
●
●
●
● ●
● ●●
●
●
●
● ●● ●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
● ●●
●
●
●
●
●
● ● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
● ●●
●
● ●● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ● ●● ● ● ● ●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
● ●
●● ● ● ● ● ●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●● ●
●
●
●
●
●
●
●
0
0
345
●
●
100
200
300
Median ranking over 1000 iterations
400
500