DRAFT
ECOLOGICAL EVALUATION
METHODOLOGY
For
Combined Structural and Operational Plan
CSOP Ecological Sub-team
Bruce Boler, ENP
Joffre Castro, ENP
Christine Chan, ENP
Barbara Cintron, USACE
Ernest Clarke, USACE
Inger Hansen, FDEP
Paul Linton, SFWMD
Brenda Mills, SFWMD
Dan Nehler, USFWS
Kevin Palmer, USFWS
Deborah Peterson, USACE
Tim Towles, FFWCC
April 2005
-1CSOP Ecological Evaluation Methodology_041105.d0c
DRAFT
ECOLOGICAL EVALUATION METHODOLOGY
For
Combined Structural and Operational Plan (CSOP)
INTRODUCTION
A fundamental and essential element of any planning process is the selection of a
preferred alternative plan through evaluation and comparison of impacts. Choosing an
appropriate, practical, and easy to use method is the first step in ensuring that this task is
fair and unbiased. It also makes the selection process transparent and, therefore, more
acceptable to all interested parties, including the general public. Although there are
many types of impacts that need to be estimated, this methodology addresses only those
that are ecologically significant.
BACKGROUND
The Everglades landscape is very complex, and its natural processes are not fully
understood. Anthropogenic changes that started as early as 1881 with reclamation efforts
by Hamilton Disston and others have continued until today, including construction of
canals, have significantly modified and altered this unique ecosystem. As much as one
half of the original Everglades wetlands have been lost through drainage. The remaining
habitat has been drastically changed and even compartmentalized. About two million
acre-feet of water is annually lost to tide; poor-quality water from agricultural and urban
runoff, which is now being discharged into the Everglades, has contaminated fish and
wildlife and has prompted human health advisories; and the spread of exotic species,
facilitated by the system of canals, has impacted the native habitat resulting in nearly 70
species being included in the endangered or threatened federal lists. In recent years, the
Federal Government in partnership with the State of Florida has launched a
comprehensive restoration effort to save and preserve the ecological integrity of the
Everglades while providing flood-protection and water-supply benefits to urban and
agricultural areas.
-2CSOP Ecological Evaluation Methodology_041105.d0c
DRAFT
In the Everglades, the relationship between hydrology and ecology is strong but has not
always been defined. Although ongoing research has provided a great deal of new
information, there is a great deal more that is still unknown. Most of the ecological
performance measures in the CSOP project are hydrological surrogates. The reasons for
this are limited ecological information and the absence of appropriate ecological models.
The Ecological Evaluation Methodology (EEM)
The proposed EEM is commensurable and compatible with the type, quantity, and quality
of data available on these restoration projects. Although the EEM concept is simple, the
methodology is flexible and accounts for the limitations of the data and intricacies of the
natural system. The method consists of three steps: (a) and (b) must be completed first,
in any order, before proceeding with (c):
a.
Weighting performance measures1,
b.
Scoring alternatives, and
c.
Computing the Ecological Benefit Index (EBI)
The basic information for evaluation of ecological scores comes from PM (performance
measures) and modeled alternatives. PMs are indexes that predict wetland-attribute
responses to changes in the ecosystem. A PM is defined by a metric and a target. The
former defines the wetland attribute to be predicted, and the latter is a desired value for
the metric. Alternatives are a set of options that are being proposed to achieve a PM’s
target. Using hydrodynamic models, alternatives are simulated. For each alternative, a
numeric output is produced and represents the PM’s metric. By comparing the numeric
output (metric) with the target, the ecological scores of alternative water control plans for
CSOP are compared (scored).
1
Ecological evaluations will be completed both with and without weighting of PMs to ensure transparency
of methodology.
-3CSOP Ecological Evaluation Methodology_041105.d0c
DRAFT
The EEM takes advantage of scientific knowledge and understanding of the natural
system by establishing relationships between wetland attributes through a system of
weights. The weights reflect the individual contribution PMs may have toward the
restoration effort. These performance-measure weights are combined with an alternative
plan’s score to estimate the EBI. The alternative plan’s score is computed by scaling and
comparing the alternative plan’s numeric output. An ecological benefit index (EBI) is
estimated by multiplying weights and scores and then summarized by CSOP’s objectives
and also by similar landscape type or other similar endpoint (e.g. tree islands, wading
bird habitat, etc.). Finally, the ecological benefit indexes (EBIs) are presented as a
percentage of an Ecological Restoration Target, which is a yardstick by which alternative
plans can be compared, and will also be compared to ecological score of existing
conditions (planning conditions). The multi-agency Ecological Sub-team is in the
process of determining the appropriate ERT for use in comparing alternative plans.
Uncertainty in Estimating Benefits
Ecological scores are evaluated using PM weights and modeled output. Both types of
measurements have errors associated with their computation that will propagate into the
estimation of ecological scores. As to the weights, the EEM computes the mean and
standard deviations, providing a measure of its variability and, therefore, of its
uncertainty. It is simple and, for relative comparisons as those carried out herein, it is
adequate. As to the numeric output from the models, the methodology does not directly
account for its error. It does, however, provide a mechanism to deal with it indirectly.
During the process of assigning weights, the reliability of the models used in simulating
the PMs is assessed and is one of the factors considered in estimating the weight of a PM.
PMs that are not easily modeled or whose output contains a high level of uncertainty
would receive a lower weight.
There is another type of unquantifiable error that may significantly affect the results of
this and any other evaluation. It has to do with whether or not all critical wetland
attributes are being monitored by the PMs. If the PMs failed to address any of the most
important features of restoration—because it is not known or because there are not
-4CSOP Ecological Evaluation Methodology_041105.d0c
DRAFT
enough data—the EEM won’t provide a realistic evaluation of ecological scores. It is
important, therefore, that the PMs be proposed using stress-effect relationships developed
from Everglades’ conceptual models.
METHODOLOGY
Projects are defined by their scope, goals, and objectives. A project’s success is
intimately related to, and often measured by, the degree to which its goals and objectives
are completed or met. Consequently, objectives are practical and realistic milestones to
be used in the evaluation of ecological scores. In the EEM, PMs are grouped by
objectives and results are presented and summarized by (1) objectives and (2) landscape
type and other similar endpoints. Each of the two presentation options provide different
types of information for the decision-maker. Presenting results by objective serves the
following purposes:
a. Illustrates how well and to what extent objectives are completed,
b. Allows objectives to be assigned levels of significance, including
the same level, and
c. Presents results in a form that is clear and concise, facilitating
comparison of alternatives.
Presenting results by landscape type and other similar endpoints (e.g. tree islands, wading
bird habitat, etc.) ensures:
a. Transparent comparison among PMs, which represent distinct resources,
within an objective, and
b. Poor performance of an individual PM within an objective is readily apparent.
Because projects are multipurpose and address a wide range of objectives, an initial
selection of pertinent ecological objectives must be made at the beginning of the process.
Only those that provide ecological benefits are included in this analysis.
CSOP has seven authorized objectives, but only five are considered in the evaluation of
ecological scores. The other two objectives, which provide flood protection benefits, will
-5CSOP Ecological Evaluation Methodology_041105.d0c
DRAFT
be considered elsewhere. Because each of the seven authorized objectives are equally
important for the project, they are assigned an equal weight, which in this case is 1.
Additionally, CSOP also has four additional objectives, but only two of which (Future
Restoration Actions and Natural Values of WCA-3A and WCA-3B) are considered in the
evaluation of ecological scores. Each of these objectives are assigned the same weight of
1.
STEP ONE—WEIGHTS AND PERFORMANCE MEASURES
In this step of the methodology, ecological performance measures are chosen and
organized according to project objectives. Because PMs may address a variety of project
features, only those that evaluate ecological scores are considered in this analysis. As
with the objectives, the Project Delivery Team, or a similar group, must decide on the
PMs that will be included in the evaluation of ecological scores. The selection should
take place early on in the process and before the evaluation of ecological scores begins.
10-to-1 SCALE. PMs are indicators that measure key effects on the ecosystem as the
result of modifications to wetland attributes by an alternative plan. The PM and also the
specific wetland attribute may have different levels of importance within the context of
restoration. Some PMs may address issues that affect larger areas than others and some
attributes may be more important than others. To establish their level of significance,
PMs are relatively compared by using a 10-to-1 scale. Using this scale, each PM is
weighted according to its merits and contributions to the ecological restoration effort.
Ten represents the best outcome and one the worst. This is the only step of the processes
where a subjective input is used to derive a quantitative output. To minimize the
subjectivity of this step and to maintain consistency, a set of guidelines is provided.
Ecological evaluations will be completed both with and without weighting of PMs to
ensure transparency of this methodology.
To measure the individual contribution of a PM to ecological restoration, a weight is
selected using the 10-to-1 scale. The scale has been divided into four groups and each
-6CSOP Ecological Evaluation Methodology_041105.d0c
DRAFT
group has three or two possible values. At the top of the scale—HIGH and MEDIUM—
are the groups associated with greater restoration potential. At the bottom of the scale—
LOW and VERY LOW—are the groups associated with the least restoration potential,
perhaps, because there is a greater uncertainty associated with how the PMs are estimated
and modeled.
10-TO-1 SCORING SCALE
VERY LOW
LOW
MEDIUM
HIGH
LEVEL
VALUE
10
9
8
7
6
5
4
3
2
1
DESRIPTION
10 to 8 HIGH. Fully supports one or more system-wide
restoration objectives, including increasing total spatial
extent of natural areas, improving habitat and functional
quality, and improving relative plant and animal species
abundance and diversity.
7 to 5
MEDIUM. Provides improvements in ecological
functions; however, improvements are less than optimal
and/or too localized to significantly support system-wide
restoration.
4 to 3
LOW. Maintains existing ecological functions, but does
not enhance system-wide ecological functions.
2 to 1
VERY LOW. Does not support ecological functions,
locally or system wide.
Several factors must be taken into account in this process. For example:
a. Role of the PM in achieving the objective under which it is listed,
b. Role of the PM in the overall project and its contributions to other
objectives, and
c. Ability of numeric models to simulate and predict the PMs response to
an alternative plan.
-7CSOP Ecological Evaluation Methodology_041105.d0c
DRAFT
Normalization of Weights
Since CSOP’s objectives are considered equally important (same level of significance),
the PMs’ weights need to be normalized within the objective. Otherwise, objectives with
more PMs will have more weight than those with fewer PMs. Within an objective, the
weight of a PM is normalized by dividing it by the sum of the objective’s total PM
weights. For convenience, the normalized weight is multiplied by 100 and rounded to the
nearest whole number (see Example 2).
Uncertainty in Estimating Weights
Achieving group consensus on PM weights may be difficult and may represent an
insurmountable challenge for a PDT. A simple solution to this difficulty is to request
input on weights for PMs from as many interested and qualified parties as possible. Once
an adequate sample of weights is collected, about 20 values for each PM, the means and
standard deviations are computed. While the former is a reliable measure of the center of
a distribution, the latter is a measure of the spread of a distribution around the mean. In
the present context, the larger the standard deviation, the less reliable the PM’s weight is.
It shows that there was not real agreement on the PM’s significance to ecological
restoration.
A special form and a set of guidelines have been developed for this purpose. The form
has the guideline questions and presents the PMs by objectives. Using a spreadsheet, the
form provides the following information for each PM: objective title, PM number, and
short descriptions of the metric and target. The last two columns, which are blank, are to
be filled with weights and justifications for the weights. The form has been completed by
agencies that are involved in the project and that have the necessary technical expertise
for answering the questions.
-8CSOP Ecological Evaluation Methodology_041105.d0c
DRAFT
EXAMPLE 1. PM’s Individual and Normalized Weights
OBJECTIVE
WEIGHT
DESCRIPTION
a
1. ECOSYSTEM
RESTORATION OF
TAYLOR SLOUGH
AND ENP EASTERN
PANHANDLE
b
c
Marl Prairie Habitat
i
Spatial distribution of marl-forming wetlands
ii F&I (Fish and invertebrates):..increase biomass...
iii F&I (Fish and invertebrates): refugia…water depths
iv Marl prairie vegetation
Slough Habitat
i
F&I (Fish and invertebrates): min. water depth…
ii F&I (Fish and invertebrates): flow distribution…
iii F&I (Fish and invertebrates): reversals…inundation…
Coastal Wetlands and Estuarine Habitat
i
Max. dry season flows in Shark Slough…
ii Max. dry season flows in Taylor Slough…
iii Max. dry season flows in ENP and Eastern panhandle…
iv Min. high-salinity events …in coastal basins…
v Max. low-salinity days…in coastal basins…
vi Min. early dry season salinity…
INDIVIDUAL
NORMALIZED
7.7
7.3
7.0
5.7
8
8
7
6
8.7
6.7
8.3
9
7
9
7.3
7.3
7.3
7.7
7.7
7.7
96.4
8
8
8
8
8
8
100
In the example above, weights are assigned to PMs for the Ecosystem Restoration of Taylor Slough
and ENP Eastern Panhandle objective of the CSOP project. Next to the description of each PM, the
individual and normalized weights are shown.
For the analysis, the mean and the 1 and 2 standard deviations are needed. The mean,
mean +1 standard deviation, and mean -1 standard deviation are used to display the range
of uncertainty associated with the EBI. The mean +/- 2 standard deviations are used to
check the acceptability of the PM weight. If the PM weight is outside the mean +/2STD, it must be carefully reviewed before it is included in the analysis.
Once the mean and standard deviations are computed for each PM, the three weights are
normalized. Because of the normalization, the weights are no longer symmetric around
the mean; the new weights, however, define a lower, middle, and upper value, which will
allow the estimationMEAN
of a benefit range associated with the EBI,
uncertainty
In thisreflecting
hypotheticalthe
case,
weights
MEAN + STDEV
- STDEV
in the process MEAN
(see Example
3).
PM1
PM2
EXAMPLE 2. Measure of PM weight’s variability
for four PMs are shown; they are
mean, mean + 1 STD, and mean 1 STD. While PM3 has the
widest range of weights—
between 2 and 6.8—PM2 has the
narrowest range—between 8 and
9.7.
PM3
Based on the spread of the
weights, PM2 is more reliable
than PM4, which is more reliable
than PM1; PM3 is the least
reliable of the PMs.
PM4
-9CSOP Ecological Evaluation Methodology_041105.d0c
1
2
3
4
5
6
7
8
9
10
DRAFT
STEP TWO—ALTERNATIVES AND SCORES
The next step in the EEM is to score the alternative plans. All alternative plans are
compared to each other, ranked, and assigned a score between 1 and 10. Using output
obtained from hydrodynamic models, the best performing alternative plan is given a 10,
the worst performing alternative plan is given a 1, and the other alternative plans are
interpolated between these two values. The output selected for scoring of alternatives is
the PM’s metric.
Score of 10. The highest score of 10 is assigned to either the PM’s target, if specified, or
to the best performing alternative. If the alternatives fail to reach the target, the 10 is
assigned to the target. For PMs whose target is to maximize or minimize a metric, the
score of 10 will be assigned to the alternative with either the highest or lowest metric,
respectively.
Score of 1. The lowest score, which is 1, is assigned to the worst performing alternative.
If additional information is available for the PM and a minimum threshold can be
specified, this value could be assigned the score of 1.
In-between values. For the rest of the alternatives that are not the best or worst of the
group, a simple interpolation method is used to assign a score between 1 and 10. Two
interpolation methods will be utilized: linear and exponential function of the form
b*Exp(x/a). A simple, popular method is linear interpolation and it will be used here for
presentation purposes (see Example 4). Although both methods are arbitrary, the
exponential interpolation provides the added benefit of favoring better performing
alternatives over lesser performing alternatives. At the end of the analysis, the leading
alternatives will consistently score better and move ahead of the others, thus, simplifying
the selection process. The Ecological Sub-team will determine the appropriate algorithm
to be utilized prior to presentation of results.
- 10 CSOP Ecological Evaluation Methodology_041105.d0c
DRAFT
EXAMPLE 3. Scoring Alternatives
SCHEMES FOR SCORING ALTERNATIVES
12
Target
Linear
10
EXP
SCORE
8
6
4
2
0
0
10
20
30
40
50
60
70
80
90
100
110
120
METRIC
A. Assign a 10 to the target or
to the best performing
alternative.
B. Assign a 1 to the worst
performing alternative.
C. Interpolate values, for the
other alternatives, using
b*EXP(x/a), where a and b
are constants.
Example (left): a metric of 52
gives a score of 3 in the
exponential scheme and of 5.2
in the linear. The best and the
worst metrics receive the same
score in both schemes.
THIRD STEP—ECOLOGICAL BENEFIT INDEX
The last step of the EEM is to combine the PM’s weight with the alternative’s scores to
derive an index that is a quantitative measure of the ecological benefits. The product of
the PM’s weight times the alternative’s score is called the Ecological Benefit Index (EBI)
(see Example 5).
It is a relative measure of potential benefits that each of the alternatives could realize. It
combines into a single value both the contribution of the PM to the restoration effort and
the effectiveness of the alternative in producing a desired ecological change.
Because each PM has three weights (mean and mean +/- 1STD), defining a range, the
EBI also has three values. The maximum and minimum values represent both ends of the
EBI range, and the middle value represents the most probable value.
- 11 CSOP Ecological Evaluation Methodology_041105.d0c
DRAFT
EXAMPLE 4. Ecological Benefit Index
WEIGHTS
Middle
To compute the Ecological Benefit Index, the
PM’s weight is multiplied by the alternative
score.
In the example (left), the weight for PM1, which
is 14, is multiplied by the score of each of the
alternatives:
Weight* Score =
EBI
14 * 5.27 =
73.87
14 * 6.99 =
97.86
14 * 2.93 =
41.02
SCORES
PM1
14
PM2
11
PM3
14
PM1
5.27
6.99
2.93
1.43
1.00
PM4
16
PM2
2.27
1.36
5.41
10
1.00
PM5
12
PM3
6.25
7.54
6.84
10
9.45
Middle
PM6
9
PM7
7
PM8
17
Alt1
Alt2
Alt3
Alt4
Alt5
PM4
The EBI can be easily grouped and summarized
by objectives.
Summary of Ecological Benefit Indexes--(continued)
ALT1 ALT2
Ecosystem Restoration in TS and EPh
ENP Natural Values
Damaging Freshwaters Flows
Flood Protection for C-111 Basin
East Everglades Mitigation
Quality of Waters Diverted to ENP
Water Deliveries into ENP
TOTAL
Additional Objectives
ALT3 ALT4 ALT5
120
150
140
130
120
210
200
200
180
190
190
180
170
170
150
210
120
1120
230
110
1130
190
190
130
100
1060 1030
180
120
950
220
170
190
250
350
A hypothetical table with Ecological
Benefit Indexes is presented in the
example (left). The indexes have been
summarized by objectives to facilitate
the comparison of alternatives. In this
case, Alt1 and Alt2 are the best
performing alternatives, with less than
1 percent difference.
COMPARING ALTERNATIVES
To facilitate comparison of CSOP’s alternative Water Control Plans, EBIs are computed
and displayed in both tabular and graphical forms. Whereas it would be simpler, and
tempting, to select the best performing alternative based solely on cumulative EBI values,
the analysis should include and consider individual PM’s indexes, to ensure that
alternatives perform well at all levels. In addition, other project objectives and
constraints will be considered when comparing and selecting the recommended Water
Control Plan for CSOP, e.g. flood damage reduction, water supply, water quality
constraints, etc.
- 12 CSOP Ecological Evaluation Methodology_041105.d0c
DRAFT
The EBI, which is a combination of the PM’s weight and alternative’s score, has been
estimated by making a relative comparison among hypothetical scenarios—from modeled
output. To bridge the “hypothetical” world with the real word—model land vs. field
conditions—the quantification results are expressed as functions of two “achievable
marks”: a minimum and a maximum desirable goal. The minimum goal is established by
an alternative representing base conditions—defined by the project’s authorizing
legislation—as the minimum level of ecological performance from which ecological
restoration benefits are compared. The maximum goal is established by ERT (Ecological
Restoration Target). ERT is an alternative that has been designed to achieve the most
gain of ecological benefits, over other alternatives, within the scope of the project. The
ecological benefit index is modified in two ways:
1. The EBI is printed in red, if it is less than that of the Base Condition, and
2. The EBI is expressed as a percent of the ERT (ecological restoration target).
In tables, results in red are undesirable, because ecological restoration benefits have not
reached levels provided by existing conditions. In graphs, EBIs that are less than the
base conditions are indicated by bars extending to the left of dashed lines. In tables and
graphs, EBI is expressed as a percent of the ERT. It is important to note that this
percentage shows the percentage point of ERT that an alternative has achieved, without
implying or suggesting that the same amount of true ecological restoration has been
completed.
EXAMPLE 5. Summary of Results
ALT1 -- 82%
OB1
OB2
OB6
OB7
10 20 30 40 50 60 70 80 90 100
ALT4 -- 63%
OB1
OB2
OB6
OB7
10 20 30 40 50 60 70 80 90 100
- 13 CSOP Ecological Evaluation Methodology_041105.d0c
The results for two hypothetical
alternatives are presented in this
example (left). Ecological benefits (xaxis) are expressed as a percentage of an
ERT (Ecological Restoration Target),
and the dashed lines are Base Conditions
(existing conditions) for each objective.
The horizontal bars show the range of
expected benefits for each objective and
the red mark the most probable level of
benefits. The larger the range of
benefits, the less reliable the outcome is,
meaning that there is a greater
uncertainty associated with the estimate
of those benefits.
By comparison, ALT1, which achieved
an overall 82% of ERT, performed much
better than ALT4, which achieved only
63% of ERT.
DRAFT
SUMMARY AND CONCLUSIONS
The selection of a preferred alternative is a multi-step process that includes the evaluation
of ecological scores, as well as other authorized project objectives, e.g. flood damage
reduction, water supply, water quality constraints, etc. A methodology is herein proposed
for evaluation of ecological scores that accounts for shortcomings of existing data and
that takes advantage of scientists’ understanding of the natural system. This knowledge
and insight do not easily translate into mathematical relations quantitatively defined by
equations. The relations, however, are more easily expressed qualitatively by a system of
weights. The results of the analysis are organized by (1) objectives, providing a real
assessment of the effectiveness and completeness of a project’s objectives, and (2) by
landscape type or other similar endpoint, e.g. tree islands, wading bird habitat, etc.,
providing transparent comparison among PMs, which represent distinct resources, within
an objective, and ensuring poor performance of an individual PM within an objective is
readily apparent.
The EEM consists of three steps:
 STEP ONE: weighting PMs based on the contribution to
ecological restoration,
 STEP TWO: scoring alternatives by using an exponential
function, and
 STEP THREE: computing an ecological benefit index by
multiplying weights times scores.
STEP ONE. PMs are grouped by objectives and assigned a weight between 1 and 10.
PMs that have a significant influence in restoration—for example, improve wetland’s
functions and species distribution in a regional scale—merit a weight of 10. PMs that
have a lesser influence in restoration or whose benefits are limited in quality and extent
receive a lower weight. The lowest weight of 1 is for PMs that have negligible
- 14 CSOP Ecological Evaluation Methodology_041105.d0c
DRAFT
(measurable) contributions to ecological restoration. There are PMs that are difficult to
simulate with existing numeric models. Some of these PMs may deserve a low weight,
even a 1, due to the model’s inability to provide a reliable output. Ecological evaluations
will be completed both with and without weighting of PMs to ensure transparency of this
methodology.
STEP TWO. To score alternatives, the numeric output for the PM’s metric is rescaled
between 1 and 10. The target or the best performing alternative is assigned a 10 and the
worst performing alternative a 1. The other alternatives are interpolated between these
two numbers using an exponential expression of the form: b*EXP(x/a), where a and b are
constants and using a linear expression, which is very commonly used. The advantage of
the exponential over the linear expression is that the former consistently promotes better
performing alternatives ahead of mediocre and worst performing alternatives, which may
facilitate and simplify the selection process. The Ecological Sub-team will determine the
appropriate algorithm to be utilized prior to presentation of results.
STEP THREE. The Ecological Benefit Index, a relative measure of potential benefits
that each of the alternative Water Control Plans could realize, is computed. The EBI is
the product of a PM’s weight times an alternative’s score. In terms of ecological
restoration, this index represents the significance of a PM and the effectiveness of
alternatives to achieve desired levels of change in wetland attributes conducive to
restoration. The results are expressed as percent of ERT (ecological restoration target)
and are bounded by benefit levels provided by planning conditions.
The EEM is simple in concept and yet flexible for addressing the complexities of the
natural ecosystem of the Everglades. By using an index (EBI), a relative measure of
potential ecological benefits are qualified and quantified. The results are easily and
clearly presented in tabular and graphical form facilitating the analysis and selection of
the best performing alternative.
CSOP Ecological Evaluation Methodology_04-11-05.doc
- 15 CSOP Ecological Evaluation Methodology_041105.d0c