Scenario-Based Planning and Decision-Making: Guidelines for Use in the U.S. Army
Corps of Engineers Planning Studies and Literature Review
By Charles Yoe
1. Introduction
The overarching question behind this research project seems quite clear. It is, quite simply, how
will the U.S. Army Corps of Engineers cope with the world of uncertainty that envelops their
Civil Works Program? In other words, what will the Corps’ culture of uncertainty look like?
Planning is future-oriented decision making. The future is not already written. It always
remains to be created. This leaves the future open to a wide variety of uncertain possibilities.
The Corps planning process, summarized in Figure 1, and described at length in the Planning
Manual (1996), uses scenarios to describe the future. Traditionally, the Corps has used a single
forecast of the most likely future condition if no action is taken by the planners (without
condition). Key variables and values in this future scenario are compared to a separate single
forecast of the most likely future condition if a specific action is taken (with condition).
In a sense, this reliance on a single estimate of the future
treats the future, for plan formulation and analytical
purposes, as if the future is already written. At times, this
approach fails to deal with significant uncertainties about
the course of future events. Uncertainty is the key to
scenario planning. Scenario planning has developed as a
systematic approach to coping with uncertainty in strategic
planning and policy-making contexts.
Figure 1: Planning Process (P&G)
The purpose of this paper is to describe scenario planning and its potential utility to the U.S.
Army Corps of Engineers Civil Works Program. In brief, three options for modifying current
planning practices to better address the significant uncertainties of contemporary planning
studies are offered for consideration. The first of these is closest to adopting classical scenario
planning methods. In the Corps’ own jargon, this would entail the construction of multiple
without project conditions (scenarios) and then formulating and assessing 1 plans that would
perform “best” in the eventuality of any of these uncertain future scenarios. The second
approach would produce multiple without conditions but one would be designated the most
likely future condition and planning would proceed as usual. Once a plan was selected however,
1
Assessment is used here to include the planning steps of evaluation, comparison, and selection.
1
its performance in the event of the realization of one of the alternative futures would be
evaluated. Significant differences in outcomes could lead to reformulation of the plan. The third
approach relies on the use of a structured sensitivity analysis of a single without condition
scenario.
A key argument advanced in this paper is that the Corps’ planning process needs to develop a
culture of uncertainty. This entails a recognition of the breadth and extent of the uncertainty
encountered in the planning process and an organizational commitment to honestly
acknowledging and addressing that uncertainty.
The next section discusses uncertainty and the need for a culture of uncertainty. This is followed
by a section that describes the technique of scenario planning, along with its history through a
review of the literature. The fourth section presents three options for incorporating some of the
principles and methods of scenario-driven planning into the P&G 2 planning process used by the
Corps. The report is accompanied by two appendices and a bibliography. The first appendix
provides annotated review of scenario planning articles related to water resource contexts.
Expert opinion is the focus of the second appendix.
2. Uncertainty and a Culture of Uncertainty
2.1 Uncertainty in the World
One of the emerging constants in the modern world is uncertainty. There is good reason to
believe that complexity and rapid as well as unpredictable change should be considered normal
parts of the 21st century landscape. Faced with this reality, planners can bemoan the difficulty of
decision-making or they can devise simple, effective strategies to enable themselves to cope with
and even thrive in an uncertain world.
In a quiet but quite dramatic way the world has moved from its Dark Ages to its Age of
Enlightenment and is now poised for a retreat forward into an Age of Uncertainty. Uncertainty
is not going away. We will see more of it, not less of it. And that is true for many good reasons.
The world grows more complex. Complexity as used here, is generally understood to refer to
such things as the size of a society, the number of its parts, the distinctiveness of those parts, the
variety of specialized social roles that it incorporates, the number of distinct social personalities
present, and the variety of mechanisms for organizing these into a coherent, functioning whole.
Augmenting any of these dimensions increases the complexity of a society (Tainter 1996). For
over 99% of human history we lived as low-density foragers or farmers in egalitarian
communities of no more than a few dozen persons and fewer distinct social roles. In the 21st
century we live in societies with millions of different roles and personalities. Our social systems
grow so complex as to defy understanding. Consequently, our systems of problem solving
2
Economic and Environmental Principles and Guidelines for Water and Related Land Resources Implementation
Studies, March 10, 1983.
2
develop greater complexity. Unconvinced? Consider the need for a planning process that more
fully addresses the complexity of the world in which we live as exhibit A!
The world faces an increasingly rapid pace of change in almost every arena. Scientific
breakthroughs make things, once impossible to conceive, commonplace. Much of this change is
driven by rapid advances in technology. The level of complexity in our social, economic, and
technological systems is increasing to a point that is too turbulent and rapidly changing to be
predicted by human beings. We see rapid increases in social, economic, and technological
connectivity taking place around the world. Social movements, e.g., environmentalists, women’s
rights, WTO opposition and the like are global in their pervasiveness. We are increasingly a
global economy. Fashions are designed in New York, approved in London, patterns are cut in
Honk Kong, clothes are made in Taiwan, and sold in Europe and North America. A computer
virus spreads around the world in hours. A human virus spreads in weeks or months.
Relentless pressure on costs is now a fixture in all public decision making. Patterns of
competition are becoming unpredictable. It is getting harder and harder to understand and
anticipate who the competition is for a job, for US grain, for land use and so on. For businesses
and government agencies alike customer/client profiles are changing rapidly and unpredictably.
We see fast-increasing and diversified customer demands. There is a growing role for one-of-akind production, and rapid sequences of new tasks in business and government. A media
explosion is just one of the consequences of an increase in the number and speed of
communication channels.
As a result of these and other changes we have entered a world where irreversible consequences,
unlimited in time and space are now possible. This is or should be extremely important to
planners. Fifteen years after the accident at Chernobyl some of the victims haven’t even been
born yet. Some of the wicked problems planners face can have a long latency period. For
example, the Corps’ ongoing efforts with landscape scale ecosystem restoration problems in the
Florida Everglades, Coastal Louisiana, and the Columbia River provide clear examples of
problems that took decades to emerge and be recognized. The implications of the solutions
being formulated may similarly take decades to be understood.
A new phenomenon of “known unawareness” has entered our lexicon. As a society we are
beginning to realize that despite all we know the unknown far outweighs what is known.
Knowledge is as much to create more questions as it is to provide definitive answers. Clearly
scientists now know much more about BSE (mad cow disease) than when the crisis started. But
even now, more than 12 years after the disease’s discovery, its origins, its host range, its means
of transmission, the nature of the infectious agent and its relation to its human counterpart new
variant Creutzfeldt-Jakob disease remains mostly unknown. We have begun to suspect that there
are some risks for which there may be no narrative closure, no ending by which the truth is
recovered and the boundaries of the risk established.
Although we are organized and live in nations our risks and our interactions are global in nature.
It becomes increasingly difficult to affix responsibility for problems and their solutions. Who is
destroying the ozone, causing global warming, spreading BSE, AIDS and SARS?
3
And despite the world’s rapid advances in all kinds of sciences we are increasingly dominated by
public perception. When it comes to uncertainties and risks, acceptability depends on whether
those who bear the losses also receive the benefits. When this is not the case, the situation is
often considered unacceptable. Risks and uncertain situations have a social context. It is folly
to regard social and cultural judgments as things that can only distort the perception of risk.
Without social and cultural judgments, there are no risks.
As a result, possibility is often accorded the same significance as existence in the public’s view.
And this view can find its way into policy. This is in part because many things that were once
considered certain and safe, and often vouched for by authorities, turned out to be deadly. The
recent BSE experience in Europe and SARS experience in China provide vivid examples of this.
Applying knowledge of this experience to the present and the future devalues the certainties of
today. This is what makes conceivable threats seem so possible and what fuels our fears of
uncertainty. It is also what makes criticism of a plan that forecasts the future as a determined
one embarrassingly easy.
Responsibility in this more connected world has become less clear. Who has to prove what?
What constitutes proof under conditions of uncertainty? What norms of accountability are being
used? Who is responsible morally? And who is responsible for paying the costs? These
questions plague planners nationally and transnationally.
The Corps’ planning process operates in this uncertain reality. Yet it clings stubbornly to a
deterministic approach to planning and decision-making that belies the experience of business
and government the world over. The planning process needs a ‘culture of uncertainty’. Recent
experience with planning on the Upper Mississippi River Tributaries has proven this point
convincingly. The future is fundamentally unknowable and there must be recognition of the
central importance of demonstrating the collective will to act responsibly and accountably with
regard to our efforts to grapple with this fundamental uncertainty and the inevitable losses that
will occur despite every best planning effort to account for this uncertainty. In an uncertain
world we cannot know everything and we will make mistakes despite our best efforts to the
contrary.
2.2 Three Decision Contexts for the Corps
The language of uncertainty is messy. It is also inexact and not infrequently, contradictory. We
have been using uncertainty in a rather overarching global sense of not being sure for any reason.
As we begin to examine the nature of our uncertainty the language and its usage becomes more
complex. Three types of decision contexts for Corps planners and decision makers can be
distinguished. In increasing order of challenge they are situations of risk, uncertainty, and
ignorance. Despite the fact that most planning decisions will be characterized by risk,
uncertainty or ignorance, Corps leaders must still make critical decisions to address water
resource problems and opportunities.
Risk involves known probabilities of events with potentially undesirable (or less desirable)
outcomes, or at least probabilities that can be reasonably well-estimated from data. There are
numerous situations of risk confronting Corps planners. Flood damage reduction represents one
4
of the best-developed areas of risk analysis in government. Few ecosystem restoration project
decisions are of this type at present. A principal goal of risk analysis is to progress from
probabilistic estimates that are mere guesses, to those that are informed estimates. Over time, the
continued practice of risk analysis should allow more Corps decisions to approach situations of
risk.
Situations of uncertainty involve known states of the world, but with unknown probabilities. In
these instances, subjective probabilities developed by trained experts, rather than the objective
probabilities of risk analysis are used to inform decisions. Analytic problems, such as how to
combine differing and perhaps conflicting expert judgments, how to exchange information and
points of view so as to make reasoned decisions, remain to be solved. From a practical as well as
a theoretical standpoint, the extent to which the subjective information overlaps or may come
from sources with different levels of credibility are key matters of concern. One practical
problem is that people, even experts in a field, often perform very badly when asked to make
probability judgments. This problem is compounded when they are confronted with multiple
information sources. For example, there is a tendency to overreact to the more alarmist or
extreme information when confronted with divergent estimates from different sources.
Ecosystem restoration planning provides many examples of uncertain situations.
The final class of decisions consists of those involving situations of ignorance. These occur
when both the states of the world and their probabilities are unknown. Catastrophic events such
as the 9/11 attacks and the Exxon Valdez oil spill, are often not even on the list of potential
outcomes that analysts consider before the events occur. Ignorance is often the most difficult
form of uncertainty for planners to confront. Scenario planning challenges planners to think
outside the box and to explore the boundaries of ignorance.
Methods for dealing with uncertainty in these various decision contexts are needed. Uncertainty
usually scares planners and decision makers alike. They prefer to solve clearly stated problems
using clearly stated scientific, engineering, policy or other principles. This has always been an
illusion.
We have never known all the relevant facts. Rarely do we fully understand how a problem is
nested in a far more complex problem context. Worse, we can never precisely predict the effect
of our plans on natural or social relations. The certainty we cling to in our uncertainty ignoring
methods is and has always been a self-induced delusion.
Other disciplines are ahead of planning in their perceptions of uncertainty. Economics,
engineering, sociology, political science, and finance have long investigated the significance of
decision making under uncertainty. If the Corps of Engineers is to improve its decision making
in general and its planning process in particular it needs to create a culture of uncertainty within
the organization that openly embraces the fact of uncertainty. That would represent a significant
change in the manner in which the Corps conducts its business, especially its planning process.
Scenario planning can contribute to consideration of uncertainty in the Corps’ planning process.
5
2.3 A Culture of Uncertainty
In a culture of uncertainty stakeholders and principals are ready, prepared and able to openly talk
about an approach to risk, uncertainty and ignorance. There is a willingness to acknowledge
what is known and what is not known. In matters of uncertainty there can be multiple
rationalities and in a culture of uncertainty there is a willingness to negotiate between these
different rationalities, rather than to engage in the mutual denunciation of conflicting views of
the future that so often has characterized the past. This culture begins with a recognition of the
breadth and extent of the uncertainty that confronts all partners to the planning process. It is
followed by an organizational commitment to honestly acknowledge and address that
uncertainty. All of this is underwritten by an understanding that it is impossible to plan
flawlessly but it is essential to plan honestly.
It's no longer enough to characterize our world as fast-moving and global. Whether in water
resources planning or business nobody can be sure what will happen next. Consequently, we
need to be ready for anything. Successful planners like successful companies will be those best
able to handle uncertainty through adaptive attitudes, structures, and processes. Dealing
explicitly with uncertainty is an absolutely essential task set before Corps planners. Current
approaches for doing so are non-existent to haphazard. Until a more rational culture of
uncertainty can be constructed for the Corps, scenario planning represents one option for
augmenting the Corps’ responsiveness to an uncertain world.
The P&G encourage the consideration of risk and uncertainty. Progress has been made in
creating a culture in risk-based analysis within the Corps’ planning function as evidenced by
recent changes in guidance, best practice and the development of risk-based analytical tools for
flood damage reduction, major renovations, and waterway improvements, and other analytical
tasks. Comparable progress has not been made in creating a culture of uncertainty. Planning,
because it is explicitly future-oriented, faces more uncertainty than many of the Corps other
functions. And it faces this uncertainty across the business purposes of navigation, flood damage
reduction including coastal or storm damage reduction, ecosystem restoration, water supply,
hydropower, and recreation. There is a pressing need to acknowledge, recognize, identify,
describe, and address the uncertainty that planners, their partners and stakeholders grapple with
on a regular basis.
To successfully establish a culture of uncertainty in planning and the use of uncertainty analysis 3
as part of an effective and modern water resource planning process there are four things that are
essential. These are: knowledge of uncertainty, an environment that supports uncertainty
analysis; an effective planning infrastructure; and, stakeholder involvement.
2.3.1 Knowledge of Uncertainty
3
In the current context uncertainty analysis is used in an inclusive context to include risk analysis. Establishing a
culture of uncertainty is a logical next step in expanding the significant progress made in incorporating risk-based
analysis into benefit calculations, rather than a completely new initiative.
6
Uncertainty analysis is not new in the sense that it lacks definition and techniques. It is,
however, new in the sense that it has not come to be broadly applied in the Corps’ water
resources planning initiatives. Uncertainty analysis represents a shift in phase rather than in
paradigm for water resources planners. Uncertainty analysis does necessitate some new ways of
thinking about the planning process. Agency personnel with the authority to implement or
require the use of uncertainty analysis must be aware of the meaning and methods of uncertainty
analysis and the value it adds to the resource and economic concerns of the Nation. Likewise,
planners who will become responsible for planning under and for uncertainty need to be aware of
the concept, its language and approaches to uncertainty analysis.
Assuring that planners and their
supervisors learn what uncertainty
analysis is, why it is done, and how to
do it represents the first concrete and
crucial step in effectively addressing
uncertainty in water resources planning.
Spreading knowledge of and proficiency
in the use of uncertainty analysis beyond
the Corps to its planning partners and
other stakeholders is almost an
immediate need, once the level of
awareness of uncertainty and the need to
address it is raised within the Corps.
Uncertainty Is Not Variability
Rainfall is variable. So are the performance of an
economy, commodity movements and the price of
gasoline. Variability can be handled in a variety of
ways within the Corps existing planning framework.
Uncertainty is different. Uncertainty may result from
the cumulative and complex variability in the system
for which the Corps is planning. Uncertainty may
result for a great many reasons. See, for example,
Chapter Three of An Introduction To Risk And
Uncertainty In The Evaluation Of Environmental
Investments
http://www.iwr.usace.army.mil/iwr/pdf/96r08.pdf
Scenario planning is best suited to situations where the
uncertainties are many and the consequence of being
wrong about the future are great.
Planners basically have two options:
reducing the amount of uncertainty, or
coming to a substantive decision, the remaining uncertainty notwithstanding. Although
uncertainty can be an objective fact, it is socially perceived and construed. It is part of the
political process to highlight selected uncertainties or risks and to divert attention away from
others. Good public decision-making and a culture of uncertainty requires transparency in
addressing significant uncertainties in planning.
2.3.2 Environment That Supports Uncertainty Analysis
It is not enough to be aware of the need for and the value of uncertainty analysis. To
successfully use uncertainty analysis to improve water resources planning, the Corps must
provide an environment that supports the use of uncertainty analysis in planning. At the most
basic level, that means the agency and other government reviewers like OMB and GAO must
value uncertainty analysis. At a minimum this includes some common definitions and
taxonomy, a commitment to openly and honestly identify key sources of uncertainty encountered
in the planning process and to address them in an open and reasonable manner. If uncertainty
analysis is recognized and valued, resources will be directed to its use. People will become
proficient in its use.
Uncertainty analysis must have the support of decision makers at the highest levels of the Corps.
Non-federal planning partners must find value in the results of uncertainty analysis. Academia
must help produce information that meets the needs of uncertainty analysis. Stakeholders and
7
citizens must be the ultimate beneficiaries of uncertainty analysis. Therefore, effective
partnerships among these key players are vital to the production of an environment that values
uncertainty analysis. Training, cooperation, collaboration, and research are just a few arenas in
which partnerships can further the use of uncertainty analysis.
2.3.3 Infrastructure
The infrastructure needed to support uncertainty analysis would be relatively modest. It
includes such elements as:
 Official Corps guidance
 Adjustments to the Corps’ planning
process
 Development of models to appropriately
support uncertainty analysis
 Information, education, communication
and training
Interdisciplinary Approach Needed
2.3.4 Stakeholder Involvement
Uncertainty analysis cannot succeed anywhere
unless key stakeholders in the planning process
have legitimate means to participate in the
process. This does not mean they would do the
analysis but that they would have an appropriate
role in the identification of key uncertainties at a
meaningful point in the planning process.
Some potential goals of stakeholder
involvement might include the following:
An interdisciplinary approach to uncertainty
in planning seems more promising than
isolated disciplinary approaches. At the time
of this writing there is a tendency for the
Corps’ risk-based analysis to be discipline
based. Economic analysis makes the most
use of risk-based techniques with a
smattering of engineering applications
sprinkled throughout the Corps official
guidance. Risk-based analysis has yet to
permeate the “culture” of the Corps. A
prime example of this is the EC 1105-2-101
requirement of risk-based analysis of flood
damage reduction benefits. There is no
similar requirement that the uncertainties and
risks inherent in the cost estimate be
investigated. This results in an asymmetry of
information for decision makers considering
net benefit measures. An interdisciplinary
approach to uncertainty analysis throughout
the organization is needed for a culture of
uncertainty.
1. Promote awareness and understanding
of the specific uncertainties under consideration during the planning process, by all
participants;
2. Promote consistency and transparency in arriving at and implementing decisions made
under uncertain conditions;
3. Improve the overall effectiveness and efficiency of planning under uncertainty;
4. Contribute to the development and delivery of effective information and education
programs to the Corps and its stakeholders;
5. Foster public trust and confidence in the planning process;
6. Strengthen the working relationships and mutual respect among all planning process
participants;
7. Promote the appropriate involvement of all interested parties in identifying and
addressing significant uncertainty; and
8
8. Exchange information on the knowledge, attitudes, values, practices and perceptions of
interested parties concerning uncertainties associated with a specific planning
investigation.
2.4 Uncertainty Not Always Significant
It is important to recognize that uncertainty, although always present, is not always going to be
significant to the planning process. To supply water to a house the options are pretty much a
water supply line, a well, a spring, a cistern, or a water tank. Although yields and such may be
uncertain there is no need for a formal process of addressing uncertainty. The scale of the
problem is small, the uncertainties are rather familiar, and there is neither complexity nor
controversy. Let it, therefore, be clear that nothing in this paper should be construed to mean
that uncertainty is always a significant issue. And when it is not there is no need to deviate from
the Corps’ current planning procedures. Forecasting a single most likely alternative future
condition will be more than adequate for a great many circumstances and no change is needed.
However, there will be landscape scale investigations, watershed studies, projects with global
connections and implications, controversial projects and so on for which uncertainty will be a
significant issue. The methodologies discussed in section 4 are targeted at the class of projects
for which uncertainty is significant.
3. Scenario Planning
This section provides an introduction to the scenario planning literature. As the longest section
of this report, casual readers may choose to skim for topics of interest.
Scenario planning developed in the latter part of the 20th century in response to the failure of
more traditional planning methods in dealing effectively with uncertainty. Traditional Corps’
planning methods, at that time as now, relied on forecasts of the future with and without a plan in
place. These forecasts were treated moreorless as a deterministic view of the future. During the
postwar years the future could indeed be adequately described as an extension of the immediate
past. As the pace of change in the world accelerated and as complexity grew by leaps and
bounds, however, the future could no longer be characterized accurately as an extension of the
past.
New methods were needed to characterize the future, because the forecasts were usually wrong.
Scenario planning offered one such method. Scenario planning is a purposeful examination of a
complete range of futures that could be realized. It is done to address the uncertainty inherent in
planning. Unlike forecasts, scenarios do not indicate what the future will look like so much as
what the future could look like. Scenario construction stimulates creative ways of thinking that
help planners, decision makers and stakeholders break out of established patterns of assessing
situations and plans so that they can better adapt to a rapidly changing and complex future.
9
Consequently, scenarios are most appropriate under conditions where complexity and
uncertainty are high.
Planning can be differentiated from ordinary problem solving by its future focus, i.e., its
planning horizon. The future is fundamentally uncertain and scenario planning is one technique
for addressing this uncertain future. The two major threads in the scenario planning/scenario
analysis literature are summarized below in the context of the Corps’ planning process and
jargon. Subsequent sections consider the topic in its own context and jargon.
The first and major thread of scenario planning can be described as a process that develops
several without project conditions rather than a single most likely alternative future without a
project, as the Corps normally does. This method, developed for strategic planning by industry,
recognizes large uncertainties in the future. Different realizations of the future could lead to
quite different views about the best actions to take in the present. The uncertainties are
addressed by describing different scenarios for each relevant future state of the world. Then,
rather than to choose a plan based on its differences between a without and with project
conditions comparison as the Corps currently does, a plan would be evaluated against each of the
future scenarios (i.e., the multiple without project conditions). The plan that performs best
across all future without project conditions is deemed the best plan.
A stylized example is offered to illustrate the concept. Consider a hypothetical deep draft
navigation project where the future is quite uncertain. Commodity tonnage is considered here as
an indicative example of the process, in fact there would be multiple variables and values
considered. Suppose there are four candidate scenarios with a reasonable chance of being
realized if no improvements to the harbor are made.
Scenario A is a constant amount of tonnage based on current capacity levels. Scenario B is
tonnage growing at 3% annually, assuming capacity would be expanded to accommodate the
growth. Scenario C is based on normalization of trade with Cuba which could triple or
quadruple tonnage within three years. Scenario D is a worsening of geopolitical tensions that
result in a bunker mentality that cuts trade in half. These are four different futures. The first two
are variations on a status quo theme. The last two are radical but quite plausible departures.
The Corps’ process would be to identify one of these futures as the most likely and then to
evaluate all planned improvements against it. Alternatively, one application of scenario planning
would evaluate each plan against Scenarios A, B, C, and D. The plan that had the best overall
performance would be the recommended plan. This oversimplified description of this thread
overlooks a great many details but it does no great harm in return for the general understanding
of the process.
A second thread of the scenario planning/scenario analysis literature focuses on one or a few
uncertain variables in either or both of the without and with project conditions. So if the first
thread sees the future as black and white options, this second thread sees more shades of gray.
The many shades of gray depend on the specific values the uncertain variables take. For
example, the actual rate of growth of commerce in Scenario B above may be uncertain. The
10
basic scenario is set as one of growth but the actual future, hence project benefits for example,
will depend on the actual rate of growth that is realized.
The approach here is to use one or more of the many methods devised to address this more
limited uncertainty in a more or less agreed upon scenario. These methods could include
classical forecasting techniques, subjective probability elicitations, probabilistic risk analysis,
and so on. This relies on a without and with condition comparison similar to the current Corps
planning process. The critical difference is that some degree of uncertainty analysis is explicitly
introduced into this planning process. This is the type of approach used in the Restructured
Upper Mississippi River-Illinois waterway System Navigation Study.
In summary, the greatest difference between the Corps’ current planning process and scenario
planning/scenario analysis is that the latter explicitly addresses uncertainty and it does so in the
development of scenarios rather than in sensitivity analysis or probabilistic analyses. Many
Corps planning studies make explicit investigations of uncertainties and risks encountered, but
these are often done as part of the evaluation of differences between two most likely alternative
future scenarios. In practice, when compared to the second thread described above, this may be
a distinction without a difference. The possibility that addressing uncertainties in the definition
of scenario rather than in the evaluation of plans may mark significant difference and an
improvement in some instances is not precluded at this point, however.
3.1 Scenarios in the Corps Context
The Corps’ planning process is presented in the Planning Guidance Notebook (ER 1105-2-100,
2000) and it is described in detail in The Planning Manual (Yoe and Orth 1996). Scenarios have
always played a significant role in the Corps’ planning process. Planning can, figuratively, be
thought of as standing at a juncture in the present and trying to choose the most desirable
alternative future from among the many alternative futures that could occur. Planning relies on
descriptions of existing and historic conditions as well as scenarios that describe the future as
best we can.
In best planning practice these future conditions describe objective realities and their subjective
social contexts as honestly as possible. Scenarios are developed for a specific study area based
on different sets of assumptions about actions society will take (with condition) or will not take
(without condition) as a result of the planning process.
The future is uncertain. It can unfold in many different ways. And the future will change, based
upon the actions society takes in the present. If a planning partnership between the U.S. Army
Corps of Engineers and its nonfederal partners decides to take no action at all following a
planning study, the future will unfold in one way. This scenario is called the “without condition”
by Corps of Engineers planners; it refers to a future without any specific action taken by the
partnership to alter the path of the future.
If the planning partnership finds that without condition future scenario undesirable it may decide
to take action in the present in order to alter that future. This scenario is called the “with
11
condition” because it identifies the future that is expected to occur if a specific action is taken by
the partnership. The scenario that accompanies a specific action plan will be different from the
“without condition” scenario. Identifying differences in important situations, events, values,
effects, conditions and resources by comparing these two future scenarios provides the evidence
that forms the basis for decision making in the planning process.
Because the future is uncertain, it is impossible to forecast the future without any action or with a
specific action with complete accuracy; but these conditions can always be forecast with more or
less accuracy. What is essential in forecasting future scenarios is that the fundamental
uncertainty of the scenario construction task be recognized, acknowledged and dealt with in an
open, honest and transparent manner.
Increasing complexity and an increasingly rapid pace of change are shaping the 21st century
world. The spread of an Internet worm affects computers and commerce around the world in a
matter of hours. Middle East peace negotiations seem to defy resolution. The frenetic efficiency
of Wall Street trading and its instantaneous response to world events gives witness to our global
economy. It has become increasingly difficult to anticipate future conditions with any degree of
certainty. The pace of change continues to accelerate with globalization of the world’s economy
and technological advances that affect everything from crop growth and tow boat horsepower to
information flows. In addition, the Corps’ increasing involvement with larger and larger scale
projects has complicated the forecast of future conditions with and without a project. Landscape
scale projects such as the Upper Mississippi River Navigation Study, the Comprehensive
Everglades Restoration Plan, and Coastal Louisiana Wetlands 2050 embrace far more complex
natural and social systems to consider in forecasts than do smaller localized projects.
These are but a few examples of systems that are hard to control and even harder to predict. The
very complexity of these systems makes it difficult for any actor to know what actions to take;
much depends on what others do and how their strategies change over time. The many actors in
this world interact in intricate ways that continually reshape their collective future. In such
complex and rapidly changing systems, the actors keep revising their strategies, trying to adapt to
shifting circumstances. As they do, they constantly change the circumstances to which other
participants are trying to adapt and the pace of change accelerates even more as
telecommunications and other technological advances reduce the lag time between action and
reaction.
Change has forced us to clear new paths in business, in education, in research, in science and
engineering. And it may be forcing us to consider new approaches to decision making and
planning. With the complexity and rapid change comes uncertainty. It is ubiquitous. With
uncertainty comes the necessity for decision-making that is flexible and agile in adapting to
change.
Planners Explore the Future
This research has been undertaken to assist the Corps of
Engineers in its search for flexible and agile decision support
tools. The focus of this section is the scenario planning
literature. Although scenario planning as a discipline is nearly
half a century old, its relevance is newly affirmed by the
12
"The capacity to tolerate
complexity and welcome
contradiction, not the need for
simplicity and certainty, is the
attribute of an explorer." Heinz
Pagels, Perfect Symmetry .
growing prevalence of the need to make significant decisions in the face of large uncertainties.
3.2 Early History of Scenario Planning
3.2.1 What is it?
There is a difference between an uncertain variable and an uncertain scenario. There are a great
many techniques for addressing the uncertainty in a single critical variable. These include
classical forecasting techniques, risk assessment, sensitivity analysis and a variety of other
techniques that are not addressed further in this literature review. Scenario planning is a method
that is more suitable for addressing the uncertain, rapidly changing and challenging future in a
much broader context.
“Scenario” literally means an outline or synopsis of a play. The word is derived from the Italian,
from scena, scene, that comes from the Latin scaena and it dates from about 1878. Herman
Kahn introduced the word to its planning context, roughly a description of possible actions or
events in the future, at the RAND Corporation in the 1950s. The first applications of scenarios
in a planning context are thought to have been in the military strategy studies done by RAND for
the U.S. government.
By the 1960’s the Wharton School’s H. Ozbekahn had used scenarios in an urban planning
project for Paris, France. The theoretical foundations of scenario forecasting, an important
component of scenario planning, were principally developed in the 1970s. Royal Dutch Shell is
regularly credited with popularizing and modernizing the use of scenario planning for strategic
planning in the early 1970s (Wack 1985a, 1985b). In fact, Wack asserts it was Royal Dutch
Shell that came up with the idea of scenario planning. French (Godet 1987) and German
(Brauers & Weber 1988) planners have also made early use of these methods. The use of
scenario-driven planning spread in the 1970s and by the 1980s it seems to have emerged as a
distinct field of study with an extensive literature.
Known variously as scenario-driven planning, scenario
forecasting, scenario analysis, and scenario planning,
by the 1980’s these approaches had developed a range
of sophisticated techniques for addressing uncertainty
inherent in a rapidly changing world. Part of the spread
of scenario analysis in the United States is attributed to
their early use by military strategists who subsequently
took jobs in other government agencies and industry,
taking their techniques with them (Becker, 1983).
There have always been significant differences of
opinion about what scenarios are, how they can and
should be prepared and how they can and should be
used. Several formal methods of scenario construction
13
A scenario
“. . . is intended to describe a possible
but by no means certain set of future
conditions. A scenario can present future
conditions in two different ways. . . .a
snapshot in time, that is, conditions at
some particular instant in the future.
Alternatively, . . .the evolution of events
from now to some point of time in the
future. . .a “future history.”. . .The latter
approach is generally preferred . .
.because it provides cause and effect
information.” (Becker, 1983 p. 96)
were used at the RAND Corporation and in the literature. Examples include, Delphi techniques
and cross-impact matrices. A synthesis of scenario methods began in the 1970s that drew
together in a single framework a variety of perspectives, including those of professional
planners, analysts and line managers (Georgantzas and Acar, 1995).
Huss & Honton (1987) described scenario-driven planning as a hybrid of many disciplines
which, unlike techniques that simply extrapolated from the past, encouraged planners and
managers to think more broadly about the future. They describe a variety of approaches to
scenario-driven planning that fall under three major categories: intuitive logics, trend-impact
analysis, and cross-impact analysis.
3.2.3 Why do it?
Scenario is a frequently used word that probably has a different meaning to everyone who uses
it. It is important to be clear how the term is used in the current context. Consider a without
project condition for a flood problem that had different possible rates at which land in the
watershed would be developed and converted to impervious surface. Consider a deep draft
navigation project with alternative growth rates for commodities. Consider an ecosystem
restoration that could produce differing marsh salinities. Each of these examples includes a
critical uncertain value that could result in potential wide variations in the future outcomes of
these projects. These differences are not scenarios as the term is used here.
In the somewhat dated language of the literature these might be called ‘reference projections,’
(forecasts in the Corps’ planning jargon) that is, piecemeal extrapolations of past trends (Ackoff,
1981). Ackoff distinguishes these reference projections from the overall reference scenario
resulting from putting them together. In other words, the without (or with) condition scenario
consist of more than a forecast of a key variable. A scenario is the cumulative result of
numerous factors.
Scenario-driven planning is a systematic approach to the increasingly important responsibility of
general management of advantageously positioning today's organization in a rapidly changing
and complex global environment. In this sense it might be described as using scenarios to
achieve a well-structured process of managing uncertain strategic situations. The future neither
aligns nor reveals itself for our convenience and as such it is ill-structured for organizational
decision-making paradigms.
Unlike forecasts, scenarios do not indicate what the future will look like so much as what the
future could look like. Scenario construction stimulates creative ways of thinking that help
decision makers and stakeholders break out of established patterns of assessing situations and
plans so that they can better adapt to the future. Scenarios are most appropriate under conditions
where complexity and uncertainty are high (Schoemaker, 1993).
The demand for scenario-driven planning in business has been increasing. Acar notes two
reasons for this. First, there is abundant evidence that the strength of the US economy is
declining or at least stagnating. Second, using scenarios as a strategic tool has provided a
handsome return on the investment it requires. The benefits of scenario planning far outweigh
14
the costs of doing so. He concludes that firms need more people capable of generating strategic
change scenarios. Because of its multidisciplinary nature, Acar notes it has been successfully
applied in a variety of applications, namely, capital budgeting, career planning, competitive
analysis, crisis management, decision support systems (DSS), macroeconomic analysis,
marketing, portfolio management, and product development. And although scenarios have been
used in strategic planning principally to explore future corporate environments, scenario-driven
planning is increasingly of interest to functional managers in diverse business areas.
Becker (1983) identifies at least three purposes for scenarios: 1) to estimate if various policies
and actions can assist or prevent the conditions of a scenario from coming about; 2) to assess
how well alternate policies and strategies would perform under the conditions depicted; and 3) to
provide a common background for various groups or individuals involved in planning within an
organization. This latter point refers to the so-called “corporate scenario” that provides a
common point of departure for all of an organization’s strategic planning. The Corps has
struggled with the notion of a corporate scenario for navigation commerce off and on over the
years.
3.2.4 Who’s doing it?
A survey by Linneman & Klein (1983) showed how scenario-driven planning was emerging as a
common and useful tool among the Fortune 1,000 Industrials. They found only a handful of
firms used scenarios in 1974. That number doubled in the next two years. By 1977, 47 firms
used multiple scenarios. In 1981, 50 percent (108) of their respondents reported using scenarios.
Not all industries adapted this technique with equal vigor; the greatest concentration of users was
found in the aerospace and process industries. The following table, adapted from Acar shows
some of the industries benefiting from scenario planning.
Industry
Aerospace & telecommunications
Agriculture
Banking
Chemicals
Data processing
Petroleum
Public utilities
Sample Literature
Millett & Randles (1986)
Helgason T. & Wallace (1991)
Imundo (1986) Prebble & Reichel (1988)
Zentner (1987)
Schultz (1986)
Gross (1984) Jones (1985); Wack (1985a
& b); Wright & Hill (1986)
Ports (1985)
Table 1: Early Users of Scenario-Driven Planning
A primary lesson to be taken from this early experience is that a variety of business areas have
benefited by using strategic scenarios to help them frame problems and situations. This suggests
that scenario-driven planning may need to be modified some to suit the nature of the “business.”
Thus, the Corps would be well advised to take the best of what scenario planning has to offer and
field modify it for their needs. This is not a tool that requires rigid adherence to a method or set
of rules of use. The other lesson taken from these early applications is that managers should
learn to construct their own scenarios. Experience shows that only then will they use them in
15
strategy design. The literature repeatedly makes the point that consultants and scenario experts
should not impose their own models and scenarios on managers. This suggests that although
Corps planners may be ultimately responsible for the development of scenarios, decision makers
may be productively involved in their construction.
Top management support is needed to build an organization-wide capability of scenario
planning. Organization-wide capability and top management support are pervasive requirements
in the literature. Schwartz (1991) describes these two elements as a pair of switches that can turn
a firm's managers and executives into partners in taking the long view. If both switches are on,
then a firm will benefit from scenario-driven planning. The benefit is not more accurate forecasts
but "better decisions about the future."
There are fundamental differences between the corporate culture where scenario planning first
flourished and the culture of a government agency with a military heritage. This raises some
concern about such an agency’s ability to incorporate some of the more valuable lessons learned
in the early years of scenario planning. Scenarios are valuable as long as they cause a new form
of interaction among those who must decide and act. In the Corps’ context this would suggest
new forms of interaction among decision-makers, planners, and stakeholders. Decision makers
who at times are removed from the details of the planning process would seem to need to take a
more proactive role in an effective adaptation of scenario planning. Acar reports that Wack, who
led the scenario planning effort for Shell, had less interest in predicting the future than in
liberating management insight and inspiring the long-term view. In scenario planning, managers
can reperceive a strategic situation, and discern their assumptions about the situation so that they
can improve their decision quality. Successful scenario planning will require the Corps to
establish an effective culture of uncertainty throughout the planning process, including top
management in some critical planning studies.
Schwartz (1991) says scenario thinking is an art, not a science, as he describes the
macroenvironmental scenarios that allow a whole array of possible futures to be addressed
before a firm's managers evaluate their responses to them. There are still many methodological
problems and difficulties with scenario planning, but experience and research are reducing these
issues (Chandler & Cockle, 1982). Chandler provides some discussion of the differences among
methods for generating scenarios, but he does not focus on scenario construction processes, a
topic considered briefly below.
Klein & Linneman (1984) found trend extrapolation to be the most widely used forecasting
technique among Fortune 1,000 corporations followed by scenario writing. But they also noted
that most companies use a "very informal scenario writing approach, with little reliance on
rigorous methodologies" (p. 72). Bearing in mind that scenario-driven planning is more than a
forecasting technique, attention turns to scenario generating techniques in Section 3.4.
3.3 Purpose of Scenario Planning
3.3.1 Scenario Planning is Strategic
16
Scenario planning is for strategic decision making under uncertainty. A review of the extensive
scenario planning literature shows increasingly more attention was paid to the methodology of
scenario planning through the 1980s. Since that time the literature has continued to grow. The
emphasis in the more recent literature has been on the use of scenarios as a tool for addressing
uncertainty. Much of this literature is more oriented toward a discussion of the tools, models and
methodologies used to generate scenarios rather than on the methodology of scenario planning
itself. This section focuses principally on the methodology of scenario planning.
Scenario analysis has been developed as a strategic planning tool to deal explicitly with
overconfidence, reliance on one certain estimate for an uncertain future and anchoring in the
present (Clemons 1995). It is a conscious move away from forecasts that rely on trend
extrapolation. Scenario planning in its early applications to private industry was used as a
strategic planning tool. Firms would not only anticipate their future through environmental
scanning and scenario constructions, but also actively engage in creating it.
Modern strategic planning, as separate from scenario planning, has generated its own extensive
literature that is not a focus of this review. Clemons describes applications of scenario planning
designed to anticipate a company’s future environmental and operational uncertainties and to
achieve consensus on the changes that need to be made in the present. Specific examples include
financial risks—delays and cost overruns, lack of feasibility; technical risks—staying within the
present while making the most of what technology can do; project risks—choosing the right
project and completing the chosen project; functionality risks—completed systems may not have
the right capabilities because planners misunderstood problems and opportunities or because the
problems and opportunities have changed; and political risks—organizational resistance keeps
the system from being completed or adopted.
Scenario planning explores several alternative futures. Scenarios are: “Developed by blending
data and analysis with intuition and creativity, scenario plots must ‘hang together’ like a wellcrafted novel, stretch the imagination without going outside the bounds of believability, and
consistently address issues that are critical to decision makers.” (Schriefer and Mercer, 1996)
Although there is not a single monolithic definition of scenario analysis, there are some
reasonably consistent characteristics of scenario analysis in most of its forms. Among them are
the following. Scenario analysis does:
1. Acknowledge uncertainty and highlight the key, critical sources of uncertainty and
ambiguity.
2. Develop a range of possible future scenarios for exploration, acknowledging that not all
are equally likely and that the future may indeed have aspects from more than one scenario.
3. Develop a range of strategies and future indicators of which strategies may be most
critical.
4. Acknowledge that future uncertainties may create discontinuities.
Scenario analysis does not:
1. Hide uncertainty or ambiguity.
17
2. Develop a single most likely answer.
3. Develop a single strategy to which the firm can commit and which the firm can pursue. 4.
4. Obtain unavailable data or make decisions on available data that may not be relevant to
the future planning process.
3.3.2 The Shell Experience
Wack’s experience with Shell is worth a brief review in itself. His corporate planning group
focused on the development of a first tier of six different scenarios about the global environment.
They used a fifteen-year planning horizon. Some of these scenarios suggested that an energy
crisis might not be far away. Although many managers were skeptical of these warnings (they
were not consistent with extrapolated trends), the energy crisis of 1973 gave Wack's planners
new credibility with Shell.
The long-run global scenarios developed by Wack's corporate planning team were disseminated
to the planning departments of Shell's operating companies. These helped inform the corporate
environmental scenarios. Smaller business units were encouraged to use the corporation’s
environmental scenarios in their own strategic planning. A second tier of short-term scenarios
was developed for near-term planning at Shell. These scenarios differed in their time horizon
and their focus on short-term economic and business cycle developments. These scenarios were
developed in an effort to directly aid the planning of Shell’s smaller business units. The secondtier scenarios proved to be another important step in the company's learning process.
The third level of scenarios developed at Shell consisted of environmental change scenarios
developed by individual business units.
The Corps in Context of Shell’s Experience
When the global scenarios did not focus on
the factors critical to business unit
To aid thinking about the Shell experience in the context
of the Corps’ planning process, consider the following
managers, they were encouraged to develop
hypothetical analogies. A first tier of scenarios might be
their own scenarios. Some of Shell's
prepared by the Corps to describe the global navigation
business unit managers developed their own
environment or perhaps the budget environment of the
global scenarios. The culture at Shell
Federal government. In fact, a corporate scenario might
welcomed these differences in scenarios
be prepared for each of the Corps’ business purposes.
because they stimulated thought and were
Second tier scenarios might be prepared for individual
considered creative rather than destructive.
studies in a District or watershed. Third tier scenarios
Nonetheless, the corporation could ill afford
might be prepared by analysts working on a study. For
to have related business units like chemicals
example, hydrologists or economists may make use of
and refining developing conflicting
scenarios.
scenarios and plans.
Nothing about scenario planning obviates the need for
forecasts. Commodity forecasts will still be critical for
navigation studies. But they will be of decidedly
secondary importance to the overall without and with
condition scenarios that would result from a scenario
planning approach.
18
As Shell gained experience, scenario-driven
planning came to be accepted as a helpful
alternative to traditional forecasting.
Quantitative forecasts still had a role in
planning; variables such as GNP, inflation,
and oil prices were still quantified. Their
role was secondary to the overall scenario. Shell’s long-run scenarios remained qualitative in
nature.
Although they started with six scenarios, Shell managers planned without seriously considering
the outlying scenarios. Consequently, Shell corporate planners eventually reduced the number of
scenarios to two. In practice these two scenarios were often strongly opposing pairs chosen
perhaps to provide a devil's advocate position (Cosier, 1981a, 1981b; Schwenk, 1984). The Shell
approach, which might be important for water resource planners to note, was that it is impossible
to predict the future exactly and dangerous to try. The effective operating principle was that
neither scenario is right. But if you're prepared for both, you'll be ready to cope with the real
world.
The essence of the planning process then, was to develop management alternatives that perform
best when compared to any of the possible scenarios. This is an important fundamental
difference between scenario-driven planning as practiced by Shell and others and the Corps’
planning process. The Corps identifies a most likely scenario and formulates and evaluates plans
against that scenario. Scenario-driven planning says the scenarios are wrong. Any of the
scenarios is possible. Plans are then formulated and evaluated against all the possible future
scenarios.
Shell found it valuable to develop flexible strategies, capable of modification in case of rapid
response from rival firms. Managerial plans and strategies were valued for their resilience in all
possible combinations of global developments. This suggests that resilience or flexibility might
be an evaluation criterion for scenario-driven planning. Scenarios, as used in classical scenario
planning, are not about predicting the future so much as perceiving and then reperceiving
possible alternative futures. Consequently, adapting the Corps’ planning process to scenario
planning would require a shift in practice from focusing on a most likely alternative future to
considering a range of possible futures. Good scenarios challenge decision makers to examine
carefully their biases and mindsets. The intent is to have decision makers say, “I can see how
that might happen and what we should do about it if it does.” Developing good scenarios helps
organizations to learn, anticipate and plan to cope with uncertainty.
Acar offers these summary observations on the process. “Scenario-driven planning has an
overriding goal and an underlying mind-set to help companies confront looming challenges and
render themselves efficiently adaptive. Its’ overriding goal is to enrich the way managers and
their consultants think, learn, and feel about strategic situations in the turbulent global
environment. “ It is in times of rapid and unexpected change that scenario-driven planning has
leverage and can make the difference between good and poor decisions.
3.3.3 The Role of Stakeholders
Adaptive comanagement (ACM) relies on iterative social learning among stakeholders and the
on-going adjustment of management decisions to be acceptable to relevant actors. In this respect
it is compatible with the planning process, although the learning in ACM was built on the
monitoring of past actions. Wollenberg (2000) shows how scenarios can be used as a tool for
19
adaptive comanagement to enable groups of forest users to not only respond to changes, but also
anticipate them.
Wollenberg argues that community-level decision-making will be more effective to the extent
that it takes account of social and ecological processes at the scale of landscapes or larger. This
context mirrors some Corps studies where multiple stakeholders are involved. The author
asserts the need for new methods to facilitate community-level decision-making that can account
for risks and opportunities with origins at larger scales.
Learning is an important part of this process and that learning is facilitated by the use of
scenarios. Scenarios are described as stories or `snapshots' of what might be. Decision makers
use them to evaluate what to do now, based on different possible futures. As already mentioned,
the term scenario is associated with several distinct approaches for gaining information about the
future (Millett, 1988; Fischhoff, 1988; Sapio, 1995). Scenario methods can be said to refer to a
general category of techniques associated with creative visioning.
Participatory Rapid Appraisal (PRA) techniques have been used to elicit people's vision about
the future. Some PRA techniques include the use of possible futures (Slocum and Klaver, 1995)
and guided imagery (Borrini-Feyerabend,1997) exercises. Scenarios differ from these in part by
their focus on the analysis of uncertainties, i.e., what other authors call the drivers of change and
causal relationships associated with a potential decision. Scenarios encourage critical thinking
about risks and systems relationships and this supports learning about problems and solutions. In
corporate cultures scenarios have been used to adapt current mental models to rapidly changing
circumstances because the “existing mental models include assumptions that are no longer valid
or habits of observation that prevent seeing new relationships” (Wack, 1985b). The ability to
break out of their highly structured and policy-based mode of thinking will present a substantial
challenge to the Corps ability to adapt scenario-based planning methods or ACM techniques.
Scenarios enable people to overcome cognitive biases to (1) undervalue that which is hard to
remember or imagine, (2) better remember and give more weight to recent events, (3)
underestimate uncertainties, (4) deny evidence that does not support one's views, (5)
overestimate their ability to influence events beyond their control, (6) be overconfident about
their own judgements and (7) overestimate the probability of desirable events (Becker, 1983;
Barnes, 1984; Bunn & Salo, 1993; Schoemaker, 1993).
Van de Klundert (1995) suggests that the application of scenarios has evolved to reflect the
historical context in planning. In the 1960s scenarios emphasized prediction based on existing
stable trends. In the 1970s and 1980s scenarios began to focus on uncertainty. In the 1990’s
scenarios have included `stakeholders' in the discussions around public and shared decisionmaking. What is the role of stakeholders in scenario analysis? In the Corps case they can help
with the scenario structuring.
Scenarios need to be able to integrate different interest groups in the planning process if they are
to be useful for public agencies. This need not mean that all groups participate equally in every
stage of scenario construction and analysis. Stewart and Scott (1995) found that differences in
20
sophistication among stakeholders in community forests required designing understandable,
transparent methods for each participating stakeholder group.
Scenario methods are themselves adaptable, and have used various forms of stakeholder input to
inform the scenario process and help make it relevant to users. Techniques for stimulating
creativity and overcoming biases include: (1) using extreme outcomes, not just predictable ones,
(2) creating disruptions to historic trends, (3) selecting scenarios that are distinct, not ones that
reflect a gradient such as high, medium and low values, or a positive and negative scenario, (4)
including undesirable scenarios, (5) starting the construction of the scenario from an imagined
future, rather than from extrapolation of current trends (Schoemaker, 1991, 1993; Bunn and Salo,
1993; Wack, 1985a). Remember the purpose of scenarios is not to predict the future but to
improve abilities to adapt to it. Thus, such techniques are not as extreme and discontinuous as
they may first appear. The cultural attitudes of organizations may make overcoming the natural
barriers to creativity difficult to overcome.
3.3.4 Forecasts Are Usually Wrong
It is much easier to predict the effects of changes in the environment in which we operate than it
is to predict the primary causes of change. It would be far easier to predict the changes in airline
security than it would be to predict the World Trade Center attack. And it is easier to predict the
World Trade Center attack than it is to explain its root causes. Forecasting is an indispensable
tool for Corps planners. But it is limited in its utility and range. Forecasts of future trends, such
as fleet compositions and commodities, are seldom value free. They have often been offered as
self-serving prophecies for the purpose of shepherding resources and efforts toward specific
planning objectives.
Scenario planning is a purposeful examination of a complete range of futures that could be
realized. It is done to address the uncertainty inherent in planning. It is a process engaged in by
decision makers and staff alike. In the Corps context, it would appropriately include
stakeholders as well.
When planners believe they can forecast with high certainty, or if they believe they are dealing
with a single scenario (the case with traditional Corps planning) their approach to the future is
called deterministic. Renn (2000) suggests that such deterministic views are a form of negation
or suppression of ambivalence and uncertainty that were often produced by technocracies. In the
past, citizens relied on experts to remove uncertainty and ambivalence from their lives. When
alternative futures are possible or are believed to exist, planners are said to hold a probabilistic
view of the future (Becker, 1983). When a range of future possibilities is used, multiple
scenarios can generate important insights that would escape the use of a single scenario. Plans
that lead to the more desirable futures or that fare better against all the possibilities are obviously
the more attractive plans.
Deterministic scenarios are rooted in the desire for a single right answer, anchored in the present,
with overconfidence in knowledge and current models. These often provide dangerously
conservative strategies. Scenarios enable managers to address strategic uncertainties. Rather
21
than embracing a single view of the future, scenario planning embraces uncertainty (Clemons,
1995).
Scenarios mark an improvement over forecasts because they are data based and managers feel
better when decisions are data-based. It’s not possible to collect “facts” about the future.
Scenario planning tests pseudo-facts for their likelihood, plausibility, fit, logical connections and
links between the present reality and future conditions. Because decisions are rooted in mental
models, they are prone to become out-of-date as the environment changes. Yet they remain
understandably hard to give up. Scenarios challenge individual and organizational mental
models. Scenarios enable decision makers to practice solutions under a wide variety of
conditions.
A technocracy or expertocracy is no longer an acceptable approach to handling uncertainty. The
future is not deterministic and forecasts are usually wrong.
3.4 Constructing Scenarios
The literature does not yet include a good, thorough methodological review of how to construct
scenarios. In the absence of such a review, this section reviews several techniques found in the
literature. Any one of these may be used or adapted by Corps planners to develop scenarios.
The reader is, of course, directed to the source articles for more detailed explanations of the
methods summarized here.
General Electric used a scenario construction approach (Jauch and Glueck, 1988) based on a
Delphi expert panel and both trend-impact and cross-impact analyses. The outputs of these
techniques were used to develop a range of probable future scenarios. Delphi forecasting has
been described as constrained guesswork, but its results are trusted because the panelists selected
are experts in their fields. GE's approach started with an initial determination of the key trends
by their planning analysts. This was followed by ‘constrained expert guesswork’ by one or
several panels of outside experts.
Trend-impact analysis begins with an outside expert's assessment of the Delphi panel's forecast
of an environmental trend. Cross-impact analysis is a more complex technique. Its output is
summarized in a matrix that shows the favorable or unfavorable interaction of likely
developments generated earlier by the Delphi panel. The output from the Delphi panel, crossimpact analysis and trend-impact analysis was then used to develop a series of probable future
scenarios. The details of that process were incomplete.
There are many different techniques presented in the literature. Georgantzas and Acar (1995)
offer a summary of the GE scenario process used to perform environmental analysis from 1960
to the 1980s in the table below.
22
Table 2. General Electric Scenario Process
Prepare
background
Assess overall environmental factors of the sector under investigation
Demographic and lifestyle
General business and economic
Legislative and regulatory
Scientific and technological
Select Critical
Identify the industry’s key indicators (trends)
Indicators
Undertake literature search to identify potential future events impacting
the key trends
Nominate Delphi panel participants whose expert opinion is credible in
evaluating the industry’s future
Indicator
Potential future events
Experts on indicator
Establish past
Establish historical performance for each indicator
behavior for
Enter these data into data base
each indicator
Analyze reasons for past behavior of each trend
Demographic and social
Economic
Political
Technological
Construct Delphi panel interview artifact
Interrogate Delphi panel
Verify
potential future
Evaluate past trends
events
Assess potential impact of future events
Assess probability of future events
Forecast future values
Specify and document assumptions for forecasts
Specific and document rationale for projected values
Forecast each
Operate trend impact analysis and cross impact analysis on the
indicator
literature search and Delphi output to establish ranges of future values
Analyze forecast results
Write scenarios Document scenarios
Schreifer and Mercer (1996) argued it is becoming increasingly difficult to anticipate future
conditions for any industry. Favoring scenario planning for its penchant to explore several
alternative futures, she describes a scenario construction process that includes the following
steps. The step identification is Schriefer’s, the summary description this reviewer’s.
1. Know the now—understand current situation and dynamics. Give maximum
exposure to present and seek agreement on any assumptions used.
2. Keep it simple—complex scenarios are useless, people cannot understand them.
3. Work up the group carefully. Five to seven members.
23
4. Try to stick to an 8- to 10-year setting. Less time than that and people extend
what is going on now. More time than that and people are guessing.
5. Be iterative—go back and summarize, remove contradictions, stay on track.
6. Blend drivers (key factors causing change)—make them work together do not let
one drive the entire scenario—challenge conventional wisdom.
7. Have a “hang-together” check at the end—challenge it, look for fatal flaws, try to
break the scenario
8. Plan to use a given scenario several times—multiple uses lowers cost, fits
business better than Corps
9. Use group again and again—there is skill acquisition here, people get better at it
the more they do it.
Clemons (1995) identified five simple steps that provide a nice conceptual overview of what a
scenario might look like. First, identify key uncertainties facing the partnership (firm). This
includes environmental and operational uncertainties. Second, rank the environmental
uncertainties. Third, select two or three critical uncertainties as the driving uncertainties. Fourth,
combine them for future scenarios. Finally, explore each scenario and develop strategies for
each scenario.
Mercer (1995) argues that scenarios can be simple and the simpler the better. He identifies three
basic groups of activities: environmental analysis, scenario planning, and
corporate strategy.
Environmental analysis is stressed by Mercer because the scenarios are only as good as the
information they are based upon. High quality analysis of the environment is essential. This is a
strong point of the Corps’ planning process. The planning team needs to be totally immersed in
the facts that define the environment they are studying. Scenario development depends less on
the facts available that what is in the team’s heads.
Environmental analysis is followed by scenario planning. The sequence Mercer describes is very
compatible with the Corps’ way of doing things. That is a plus for possibly integrating the best
of the two models. Mercer’s scenario planning has six steps.
Step one--decide on the drivers for change. The results of environmental analysis are examined
to determine the most important factors that will decide the nature of the future environment
within which the organization, or in the case of Civil Works, the plan operates. Planners 4 must
carefully decide the broad assumptions upon which the scenario will be based. Only then should
the key drivers be specifically defined. The purpose of these two tasks is to free people from the
preconceptions they bring to the scenario structuring process. Too short a time frame leads to
reliance on extrapolation from the present. This is to be avoided.
4
‘Planners’ is intentionally used without careful definition here. There is no effort to prescribe how or if this should
be done by the Corps. This reviewer has attempted to generalize the original author’s work to the Corps context to
make the ideas more accessible to people who have not read the original work. This, unfortunately, results in some
rather arbitrary use of language at times.
24
Brainstorm lists of drivers. Apply the 80:20 rule so managers and others can focus on what is
truly important. We’re looking for drivers that are subject to significantly different alternative
futures, i.e. things that are important and uncertain. Factors that are important but predictable
(e.g. hydrology) should be identified in the introduction to the scenarios.
Step two--bring drivers together into a viable framework. Scenarios are more than simple
forecasts of unknown variables. Uncertainty is more than variability in a system. It is necessary
to link the critical drivers together in a viable framework. Build “event strings” that link drivers
so as to see the linkages and their progression over time. Group drivers into combinations that
are meaningful to participants. For example, many individual drivers may be aggregated into
groups like commerce, navigation, ecosystem. This framework is often the most difficult step
conceptually.
Step three--produce mini-scenarios. Step two often produces seven to nine groups of drivers.
Now it is time to work out the connections among these groups. What does each group
represent? Develop a mini-scenario around each group of drivers. The scenario architects need
to stretch their imaginations while remaining believable.
Step four--reduce scenarios. Reduce the mini scenarios to a few (two or three) larger scenarios.
Experience suggests managers cannot cope effectively with too many more than 3 scenarios.
Shell used two complementary scenarios. ‘Complementary scenarios’ means there is no
preferred scenario. There is no most likely condition as the Corps uses. So this would mark a
change. The idea is not to produce a good/bad scenario or a high/medium/low scenario so much
as to produce balanced and reasonable scenarios. Any scenario produced here should be tested.
Does it hang together? Does it make sense? Make sure the assumptions are not unrealistic.
Step five--write the scenarios. Produce the scenarios in the form most suitable for use by
planners and managers who are going to use the scenarios to develop plans/strategies. They are
usually in word form and qualitative. In the Corps process they would become more quantitative
as evaluation proceeds. There is a potential issue here in that planners are not personally
involved in the way a business’s employees would be. Planners are going to walk away at some
point. When the plan is implemented they go on to the next plan. A corporation must continue
to live with the impacts of its strategic decisions. So the personnel doing the planning and
making the decisions will differ from the corporate model. This may or may not make a
difference in terms of the agency’s actors’ willingness to engage in the scenario construction
process.
Step six--identify issues arising. Examine the scenarios to identify the most critical outcomes.
These are the branching points related to the issues that will have the greatest impact on the
future. The scenarios are then used in planning. They become logical devices, ways to present
the most important topics to planners and managers so they can address them. They provide a
way to test our consistency. Are they internally consistent? The insight they offer as to the
general shape of the future is important. At this point it is no longer a theoretical exercise, it is a
genuine framework for dealing with the future.
25
Scenarios have been used for corporate strategy, they are a means to an end. They identify
forces and long term consequences that must be addressed by plans. The idea is to search not
just for an optimal outcome but the best overall outcome. This is defined as one that protects as
far as possible against all future threats and exploits major opportunities. In the Corps’ planning
context, the scenarios would be used for formulation, evaluation, comparison and, as a result, for
plan selection. The search for a best overall outcome could present a conceptual problem for the
Corps planning process, given its current emphasis on the NED plan, an optimal outcome.
Becker’s (1983) general approach allows a place for the “genius scenario.” This is a scenario
created by one person based on his or her knowledge and experience. Becker offers several
common sense thoughts that should not be taken for granted, as common sense is not all that
common. Scenarios must avoid containing mutually exclusive events. They need to be
internally consistent. Ideally people with different viewpoints participate in their formulation
(this of course rules out the genius scenario). The future’s history is evolved in a sequential
fashion. Events leading to a branch point are described; it is cause and effect oriented.
Becker’s basic scenario characteristics include the following:
1. Select basic characteristics, i.e., the few conditions most important to shaping the system
or marketplace being studied. These are the drivers.
2. Set the possible range of values that will be studied for the basic characteristics,
quantified if possible. Decide the extremes that will be used. Bound the analysis. This
needs to be more than variations on a theme.
3. Select the number of scenarios that will be studied. This may be based on combinations
of basic characteristics that are internally consistent and sufficiently plausible. Use a
middle ground or most likely and two other scenarios that represent demanding situations
in the marketplace. Recognize the most likely may have a low probability of being
realized.
4. Designate the indicator and trends that will be treated in each scenario. Drivers were
identified early on, now identify other details of scenarios that are important and useful.
5. List important events. These are developments necessary for the conditions of each
scenario to come about and those important to shaping the indicators and trends. This is
getting to the more detailed effects.
6. Estimate probabilities of each event in each scenario and the impacts of each on the
indicators. This includes estimates of the likelihood of occurrence and influence on each
indicator.
7. Project the indicators; quantified values vs. time.
8. Prepare narratives. Describe the evolution of conditions in each scenario spotlighting key
events/developments, important trends, implications for the system or market place
studied and, where possible, implications for strategies, policies and actions.
The resulting scenarios are then used to nominate strategies for implementation.
Schoemaker in his book, “When and How to Use Scenario Planning: A Heuristic Approach with
Illustration” (1991) offers a list of ten steps in scenario construction.
26
1. Define the issues you wish to understand better in terms of time frame, scope, and
decision variables. Review the past to get a feel for degrees of uncertainty and volatility.
2. Identify the major stakeholders or actors who would have an interest in these issues, both
those who may be affected by them and those who could influence matters appreciably.
Identify their current roles, interests, and power positions.
3. Make a list of current trends, or predetermined elements, that will affect the variable(s) of
interest. Constructing a diagram may be helpful to show interlinkages and causal
relationships.
4. Identify key uncertainties, whose resolution will significantly affect the variables of
interest to you. Briefly explain why and how these uncertain events matter and examine
how they interrelate.
5. Construct two forced scenarios by placing all positive outcomes of key uncertainties in
one scenario and all negative outcomes in the other. Add selected trends and
predetermined elements to these extreme scenarios.
6. Assess the internal consistency and plausibility of these artificial scenarios. Identify
where and why these forced scenarios may be internally inconsistent (in terms of trends
and outcome combinations).
7. Eliminate combinations that are not credible or impossible, and create new scenarios (two
or more) until you have achieved internal consistency. Make sure these new scenarios
cover a reasonable range of uncertainty.
8. Assess the revised scenarios in terms of how the key stakeholders would behave in them.
Where appropriate, identify topics for further study that might provide stronger support
for your scenarios, or might lead to revisions of these learning scenarios.
9. After completing additional research, reexamine the internal consistencies of the learning
scenarios and assess whether certain interactions should be formalized via a quantitative
model. If so, use this model to run some Monte Carlo simulations after obtaining
subjective uncertainty ranges (or entire distributions) for key independent variables.
10. Finally, reassess the ranges of uncertainty of the dependent (i.e., target) variables of
interest, and retrace Steps 1 through 9 to arrive at decision scenarios that might be given
to others to enhance their decision making.
Wollenberg (2000) offers a four step process for creating scenarios. The four elements common
to scenario analysis are:
1.
2.
3.
4.
Definition of the purpose of the scenarios.
Information about a system's structure and major drivers of change.
Generation of the scenarios.
Implications of the scenarios and use by decision makers.
Ringland’s.Checklist (Ringland 2003b) for developing scenarios comprises the following 12
steps:
1. Identify focal issue or discussion—the key question
2. Key forces in the local environment—factors that will influence success or failure of
decisions
27
3. Driving forces—forces in macroenvironment that drive key forces in local environment
4. Rank by importance and uncertainty—identify two or three factors most important and
most uncertain
5. Selecting the scenario logics---the two or three factors used to create a visual map of
scenarios
6. Fleshing out the scenarios—How would we get from here to there? What events have to
happen for that to come true?
7. Implications for strategy—how does question to be decided look against the scenarios?
8. Selection of leading indicators and signposts—how will the history of scenario be
tracked?
9. Feed the scenarios back to those consulted—get feedback on created scenarios
10. Discuss the strategic options—generate complete set of options against the scenarios
11. Agree on the implementation plan—owner of process
12. Publicize the scenarios—this is akin to an evaluation step.
Some of the literature has suggested that companies sometimes do not know what to do with the
scenarios once constructed and so they do not achieve desired results. The key is to ask what the
scenarios really mean for the company. In the case of Corps planning, one might put each plan
up against each scenario. For example, if we implement Plan A and Scenario1 is realized we
will have this problem or that risk or opportunity. What will we do about it?
4. Scenarios in Corps of Engineers Planning
How might scenario-planning techniques be used to improve The Corps’ decision-making for
inland navigation, deep draft navigation, flood damage reduction and ecosystem restoration
studies? This section takes the opportunity to summarize some of the concepts reviewed to this
point and to place them in a more familiar Corps planning context. First, it is useful to consider
when scenario planning might be appropriate for Corps planners.
Consider the figure below. It presents a four quadrant grid based on the intersection of
uncertainty and consequence axes. When uncertainty is great and the consequences of a wrong
decision are grave, scenario planning is desirable. If the uncertainty is great but the
consequences of an error are minor, traditional Corps planning is appropriate. Likewise,
traditional planning is indicated when there is little uncertainty. Ordinary decision-making
processes, i.e. requiring no planning process, is best when the consequences of a mistake are
minor and most of the necessary facts are known. The figure below suggests that there is a
continuum of gravity of consequences and extent of uncertainty as well. A study in the far upper
left corner of the Scenario Planning quadrant and the far lower right corner of the same quadrant
could be very different. For example, one might be a landscape scale multiple purpose project
and the other a local continuing authorities project. It is the relative extent and consequence of
uncertainty that will dictate the appropriateness of scenario planning moreso than any simple
categorization of reconnaissance, feasibility, watershed or other types of studies. Nonetheless
28
Consequence
Grave
Scenario Planning
P&G Planning
Uncertainty
Much
Little
P&G Planning
Standard Decision Making
Minor
Figure 2: When to Use Scenario Planning
one might speculate that the kinds of planning studies most likely to land in the upper left
quadrant are inclined toward large landscape scale studies with complex problems (ecosystem
problems tend toward more complexity) that involve conflicting stakeholder interests and new or
technologically sophisticated solutions.
There are two useful distinctions to make in how scenarios are used in scenario-driven planning.
One involves distinctly different scenarios. This use of scenarios is when there is genuine
uncertainty. The other use of scenarios involves variations of a single scenario. This occurs
when there is not so much overall uncertainty about the future as there is variability in one or
more important variables in a without or with condition future forecast. Translating these two
notions into the language of the Corps’ planning process is helpful.
Scenario planning as described in the literature involves the identification of distinctly different
alternative future without project conditions. This is appropriate to do when the future is
uncertain. Consider a hypothetical navigation study for a south Atlantic port. One without
condition scenario might be an extension of the present into the future. A distinctly different
future will result if the United States normalizes relations with Cuba and begins trade with them.
Another future might be described by a world situation in which the US and other nations
withdraw from the global economy as a result of terrorist activity. Yet another scenario would
result if the world moves to global government. The key here is that each scenario is distinctly
different. If we go back to the original meaning of the word, these scenarios offer very different
29
plotlines or scripts for the port’s future. Notice these futures involve a great deal more than a
commodity forecast that varies by a few percent each year or by a high/medium/low view of
basically the same future.
The various scenarios in the hypothetical example can differ in a variety of ways including
commodity forecasts, fleet forecasts, origin-destination pairs for imports and exports, commodity
mixes, trade agreements and so on. Contrast these kinds of scenarios with the more common
without condition that forecasts that a port will continue to do what it is now doing but will do
more of it. With this kind of scenario there are usually a couple of variables that are subject to
considerable variability. Let us suppose commodity forecasts is one of these. All the scenarios
follow essentially the same plotline, they only vary in their specific details for a single variable.
For example, the port may realize a decrease in tonnage, no change, a low increase, a moderate
increase or a high increase in tonnage. This makes a great deal of difference to project benefits
and its ultimate feasibility but it is not a different scenario.
The two will be distinguished in this report by considering the first example one of different
scenarios and the second example one of key uncertainties (or variability) within a given
scenario. That leaves open the question of how different the key uncertainties must be before
they constitute different scenarios, but that is a minor concern. If the plotlines differ then an
uncertain future, i.e., different scenarios, is the principle challenge for planners to address. If the
plotline is essentially set while the details differ then variable uncertainty (or variability) is the
principle challenge in considering the scenario. It is this latter situation which rises to the
surface most often within the Corps’ Civil Works Program. It could, however, be validly argued
that the former is the real challenge to Corps planners.
In its traditional Shell sense, scenario planning involves the identification of alternative future
scenarios and then devising a plan or plans that would perform well against any and all of these
scenarios. Translating this notion into the Corps’ planning process is not difficult. This amounts
to an admission that the future without a project is uncertain and unknowable. Any one scenario
will be wrong. Instead of a single most likely without condition scenario, the Corps would
identify multiple without condition scenarios. Then instead of identifying the NED plan for the
comparison of an effectively deterministic without and with condition, decision makers would
seek the plan that performs best against all of the reasonable future scenarios identified. The
conceptual issue of reconciling the new notion of a best plan with the notion of an NED plan
remains. That is not insurmountable in concept, however, because if the scenarios are assigned
probabilities an expected value can be calculated. Pragmatically, it is not yet clear that pursuing
the maximum expected value of net NED benefits is desirable. At one level, scenario planning
can be characterized as essentially the identification and use of alternative without project
conditions. This approach is further explored in Section 4.1.
Adapting Becker (1983), multiple without condition scenarios can be used for at least three
purposes. First, they can be used to estimate if various policies and actions can assist or prevent
the undesirable conditions of a scenario from coming about. Second, they can be used to assess
how well alternate policies and strategies would perform under the conditions depicted, i.e., to
estimate risks in choosing certain courses of action. Third, they can be used to provide a
30
common background for various groups or individuals involved in planning within an
organization.
Scenario analysis helps determine actions organizations should take now; actions they should
stop in the future if they are headed toward a scenario that makes them unnecessary or less
attractive; and, actions they should take if they are headed toward a scenario that makes them
necessary or more attractive. A plan with a series of triggers or thresholds might be in order and
as scenario uncertainties are resolved, the strategy or measures for responding are there and
ready for use. In this sense, we can see scenario planning melding into adaptive management
trains of thought, especially if the planning horizon is shortened from 50 years to something
closer to a decade or two. Planning then expands to include identifying contingent possibilities
(triggers and the like) so managers can anticipate and respond quickly. This is fundamentally
more difficult when considering large scale public works projects than it is for a business making
decisions about a product line, however.
Scenarios, however constructed, are used for forecasting alternative futures. When these
scenarios are subsequently analyzed and used as the basis for developing corporate strategies
they give rise to scenario planning, a planning model used by business that represents an
alternative to the six step planning model presented in the P&G.
Scenarios can be used for strategic planning as the literature amply illustrates or for project
planning. In fact, the Corps has tried to do some scenario planning in the past by coming up with
a set of commodity forecasts that would be used by all Districts. The Inland Waterway Users
Board has been used somewhat like this. Experts on the Board help provide scenarios for inland
waterborne commerce that can be used to establish infrastructure investment strategies. These
examples, though limited and informal applications at best, are suggestive of the notion of the
Corps’ corporate scenario. Such scenarios provide a common point of departure for planning
efforts. In addition, Corps guidance has provided some ad hoc scenario guidance, e.g., no net
gains in tonnage from other ports.
Using guidance to settle scenario issues is, however, an anathema to scenario planning. This is
precisely the kind of rigid thinking and behavior scenario planning is trying to overcome.
Scenario planning is open to all the possibilities in an uncertain future. Much of the Corps’
guidance functions to constrain the realities Corps planners can consider. In this regard there is a
fundamental clash between scenario-driven planning and the Corps regulation-heavy culture.
For scenario-driven planning to succeed in the Corps’ Civil Works planning program a culture of
uncertainty must be established. Planning and senior management must embrace uncertainty. It
must provide the framework to support solutions that may be radically different. It needs to
move from precision and inaccuracy to accuracy and imprecision.
The Corps planning process can itself be described as a variation on the scenario planning theme.
Corps planners use scenarios already. Consequently, scenario-driven planning does not
represent a significant change. But it does represent a change.
4.1 Current Corps Planning Methodology
31
Corps planners identify one future scenario if no action is taken and a separate future scenario for
each plan they are investigating. They choose a plan based upon the differences in key social
values and variables that are identified in a comparison of these scenarios.
Every Corps planning process identifies a most likely future condition without any specific
action taken by the Corps and its planning partners. This is the “without condition” scenario
described earlier. The purpose of Corps planning is to avert the undesirable features of this future
condition. The without condition often has the qualities of a forecast, i.e. planners are saying the
future will most likely look like this and the “this” is very often an extension of the present into
the future. The without condition serves as a benchmark against which other alternative future
scenarios are compared. For each plan formulated, Corps planners develop a most likely future
condition with that plan implemented. Plans are evaluated by comparing such values as
expected annual flood damages, habitat units, water quality, costs and the like in their “without
project” and “with project” future condition scenarios.
Scenario planning, by contrast, says we do not know what the future will look like. It could look
like this, that or some other thing. Scenarios are not forecasts, they are possible futures. And no
one of them is singled out as more likely than any other.
The second step of the planning process, the inventory and forecasting step, is where the without
condition is described. It is considered a forecast. Planners make forecasts to identify
differences among future scenarios that make a difference to decision makers. Consider a
hypothetical planning investigation that focuses on acres of wetland available and tons of cargo
that move through a port. If the planning partners do nothing at all the values for these two
variables in 20 years are expected to be as shown in Table 3. Corps planners would call such a
description of the future a scenario, using the language somewhat differently than it is used in the
scenario planning literature.
Item
Without
Condition
Forecast
Acres of Wetlands
1,200
Tons of Commodities 3,000,000
Table 3: Future Without Action
Now suppose there are actions that can be taken in the present to alter the course of the future.
For example, suppose plan A is expected to result in the values shown below. The with
condition presents another scenario, as the Corps now uses the language. The differences the
plan makes are easy to see when the two scenarios are compared.
32
Item
Without
Condition
Forecast
1,200
3,000,000
Acres of Wetlands
Tons of
Commodities
Table 4: Plan A Differences
With
Condition
Forecast
2,500
1,500,000
Differences
+1,300
-1,500,000
Each plan that is considered has its own unique with condition scenario. The table below shows
the differences Plan B makes to the future.
Item
Without
Condition
Forecast
1,200
3,000,000
Acres of Wetlands
Tons of
Commodities
Table 5: Plan B Differences
With B
Condition
Forecast
1,000
3,500,000
Differences
-200
+500,000
Now decision makers can chose from two very distinct futures. One has more wetlands and less
commerce, the other has more commerce and fewer wetlands. Plans are chosen based on an
evaluation and comparison of the differences the plans cause in the alternative future forecasts
identified. It is a selective comparison of key variables across scenarios. Table 6 summarizes
the significant differences between the choice of two proactive futures, a future with Plan A or a
future with Plan B.
Item
Plan A
Differences
+1,300
-1,500,000
Plan B
Differences
-200
+500,000
Acres of Wetlands
Tons of
Commodities
Table 6: Comparing Differences That Make a Difference
Planning can be differentiated from ordinary problem solving by its future focus, its planning
horizon. The future is fundamentally uncertain and scenario planning is one technique for
addressing the uncertain future. Traditional scenario planning can be described as a process that
develops several without project conditions rather than a single most likely alternative future
without a project, as the Corps normally does. It relies on multiple scenarios rather than a single
forecast.
In scenario planning the scenarios would include the evidence and the narrative explanations of
how and why wetlands and commodity projections might come to be as planners consider them.
Wetland acreage and commodity tonnage are, in the language of scenario planning, indicators of
what is important about the different scenarios. Three basic approaches to incorporating some
degree of scenario planning into the Corps’ planning process are summarized below. Each
approach provides a focus for the three sections that follow.
33
Although several techniques
have been summarized above,
one common approach is to
identify two key uncertainties
and prepare a scenario cross as
shown below. The two driving
uncertainties, A and B, when
considered together define four
different scenarios. Scenario 1
might have a low level of
uncertainty B and a high level of
uncertainty A, and so on.
Uncertainty A
Scenario 1
Scenario 2
Scenario 3
Scenario 4
Uncertainty B
The concept may best be
demonstrated with a simple
Figure 32: Generic Scenario Cross
example. Suppose the tables
above relate to a deep draft navigation project. Suppose a scenario planning process has
identified two key uncertainties that will drive the shape of the future. One of these is the
terrorist threat level, the other is the extent to which the global economy continues to flourish.
These two uncertainties might produce a scenario cross like the one below.
High Threat
Each scenario is given a title that
will be memorable and descriptive.
If the terror threat is high and the
Deal
Bunker
global economy is very open, this
‘em
Mentality
presents the potential for great gain
Open
or great loss, hence a poker-related
Closed
Economy title for this scenario. If the threat
Economy
is high and the economy closes
down on the world we have a
Secure
Peace
At Last
On Earth
bunker mentality. Each of these
scenarios would be described in a
rich narrative description, which in
Low Threat
turn, would be supported by
Figure 4
3: Deep Draft Navigation Scenarios
quantitative estimates of the key
indicator variables, such as commodity forecasts and wetlands, necessary for plan formulation
and assessment.
Instead of a single forecast of the most likely without project condition, planners would now
have four future scenarios. No one of them is a forecast or prediction of what will be. But
together they present a reasonable picture of what could be in an uncertain future. In most
scenario planning models no effort is made to attach probabilities to the different scenarios.
And, although, there are many other methods used to construct scenarios, most of the literature
suggests three to five scenarios are most reasonable.
34
The challenge for planners would be to formulate plans that would be effective in any future
scenario. This presents a difficulty to Corps planners because scenario planning has been
favored by industry and others who are considering product lines or management strategies for
the next 5 to possibly 20 years rather than irreversible infrastructure investments intended to last
five or more decades. In addition, adapting a scenario approach to planning would entail more
complex evaluation and comparison processes. It would seem reasonable, for example, to expect
commodity forecasts to be quite different for some navigation study scenarios. That would mean
four different benefit estimates that would ultimately have to be considered in some way.
This method, developed for strategic planning by industry, recognizes large uncertainties in the
future. Different realizations of the future could lead to quite different views about the best
actions to take in the present. The uncertainties are addressed by describing different scenarios
for each relevant future state of the world. Then, rather than choosing a plan based on its
differences between a without and with project conditions comparison as the Corps currently
does, a plan would be evaluated against each of the future scenarios. The plan that performs best
across all future without project conditions is deemed the best plan.
35
Scenarios and P&S
Scenarios are not new to the Corps. P&G relies on deterministic scenarios with and without the
project in place. In the past it seems the Corps may have been influenced by the scenario analysis
trend as seen in the Principles and Standards that preceded the P&G. The future is uncertain.
During the early 1970s when scenario planning was gaining credence in the business world the
Corps’ planning guidance seems to have reflected some of the values of scenario planning.
“A first requirement is to determine the general types of alternatives to be developed under alternative
assumptions concerning the level and magnitude of component needs in the future. Given alternative
assumptions concerning the future economic and demographic trends for the planning setting and the
total range of component needs related thereto, a set of alternative plans should be prepared for each
major assumption concerning the future. In those planning situations where there does not exist a
strong linkage between water and land development and major shifts in economic and demographic
trends, the Council’s baseline projections will generally be used as a single set of assumptions about
the future level of component needs required. Where the linkage is sufficiently strong so that water
and land development may materially alter future economic or demographic trends, this relation
should be reflected in alternative assumptions. Where the planning area may be unusually
susceptible to other factors that could easily change in the future, it will be appropriate to establish a
basis for a different set of alternative plans based on alternative assumptions concerning future
change. In this instance, a sensitivity check should be made to ascertain the extent to which
component needs will vary significantly given different assumptions concerning the future. If no
significant variation is found, only one set of alternative plans will have to be developed.” Water and
Related Land Resources Establishment of Principles and Standards for Planning, FR Vol. 38 No. 174,
Monday September 10, 1973 Page 101
Section 711.17 Forecasting, says the following: “(a) Formulation and evaluation of alternative plans
are to be based on the most likely conditions expected to exist in the future with and without the plan.
. . (b) The forecasts of with- and without-plan conditions shall use the inventory of existing conditions
as the baseline . . .” Principles and Standards for Water and Related Land Resources Planning-Level
C: Final Rule, 18 CFR Part 711, Monday September 29, 1980
A second adaptation of scenario planning for the Corps would be to identify the scenarios of
Figure 4 but to follow a more traditional planning process. That means one future would be
singled out as the most likely alternative future. It might be one of these scenarios or a synthesis
of them. The planning process would proceed to the point of at least the final comparison step,
possibly all the way to the selection of a plan. At that point, the selected plan (or the final array
of plans) would be evaluated against each of the possible scenarios for the purpose of testing the
sensitivity of the selected plan’s performance to the eventual future state of the world. If a plan
fails miserably in one or more scenarios, it may be important to work on reformulation or
reducing the uncertainty the planners face. This has the appeal of minimizing the analytical
difficulties that would have to be resolved (e.g., weighting different benefit estimates) to make
the multiple scenario approach consistent with the Corps’ current planning practice.
A third approach, arguably being used already to some extent in a few cases, would focus on one
or more uncertain variables in the without and with condition forecasts. This is more closely
attuned with what the Corps calls its risk-based analysis of flood damage reduction benefits, for
36
example. The actual rate of growth of commerce in the scenarios above may be uncertain. The
approach here is to use one or more of the many methods devised to address this more limited
uncertainty in a more or less agreed upon scenario. These methods could include classical
forecasting techniques, subjective probability elicitations, probabilistic risk analysis, and so on.
This relies on a without and with condition comparison similar to the current Corps planning
process. The critical difference is that some degree of uncertainty analysis is explicitly
introduced into this planning process.
In summary, the greatest difference between the Corps’ current planning process and scenario
planning/scenario analysis is that the latter explicitly addresses uncertainty and it does so
principally in the development of multiple scenarios. Scenarios are not forecasts of the future,
they are potential future states of the world in which planners are interested.
Many Corps planning studies make explicit investigations of uncertainties and risks encountered,
but these are often done as part of the evaluation of differences between two most likely
alternative future forecasts. Scenario analysis could mean a change in the basic planning model.
The Corps could switch to a model closer to the corporate sector’s strategic planning.
Alternatively, the Corps could adapt scenario planning’s strengths in addressing an uncertain
future to its own planning process.
4.2 Planning with Multiple Without Conditions
This section considers how the six-step planning process of the Corps might be adapted to
incorporate scenario planning strengths. In brief, the variant of scenario planning considered
here is based on the identification of multiple future scenarios. Plans are formulated to perform
“best” no matter which future is realized.
4.2.1 Problem Identification
Step one, problem identification would remain essentially the same. In those instances where
uncertainty is known to be significant and scenario planning is desired, the problems,
opportunities, objectives and constraints should be expressed as richly as possible. Scenario
planning tends to place significantly more emphasis on the external environment of the
organization. Translated to the Corps’ planning process this implies the need to take a much
broader view of the context of the problems and to examine factors in the macroenvironment that
could affect the problems or their social contexts.
4.2.2 Inventory and Forecast
Step two, inventory and forecast would be substantially different in parts. The principle new
output of the second step of the planning process would be the identification of multiple without
project conditions.
Scenario planning is predicated on the belief that the future is uncertain, it is not yet written and
it is unknowable. Thus, there is no attempt to forecast what that future will look like. There is
no most likely alternative future. There are many possible futures and scenarios are constructed
37
to characterize the range of them. In a statistical analogy, traditional Corps planning was based
on an expected future value, “the most likely alternative future.” By contrast, scenario planning
is based on the variation in the future. It is like the difference between planning for one value or
for all possible realizations of an unknown variable.
The key to the changes in this step center around the method used to construct alternative
scenarios. The textbox on scenarios and the Principles and Standards above shows that the idea
of alternative futures is not new to the Corps. In the 1970s when scenario planning was growing
in popularity, the Corps recognized the reasonableness of identifying alternative futures.
Common practice in the Corps quickly focused on the designation of one of these futures as the
most likely future. At that point any alternative futures were left behind. In actual practice there
is little evidence to suggest that Corps planners ever considered multiple future scenarios. A
definitive statement on actual practice would require a review of planning reports from that time
period.
Although the idea of using more than one scenario to characterize what the future of a study area
might be like if no planning action is taken may appear new to many planners, it actually
represents the revival of an old idea (P&S). Because there is little evidence to suggest multiple
scenarios were ever used by the Corps there are no methods found in the Corps’ guidance to
suggest how these scenarios should be constructed. The literature is quite rich on this point and
the literature review above summarizes numerous methods of identifying future scenarios
(Georgantzas and Acar (1995), Schriefer (1995), Clemons (1995), Mercer (1995), Becker (1983),
Schoemaker (1991), Wollenberg (2000), Ringland’s (2002), Lindgren and Bandhold (2003)).
It is not the purpose of this paper to define the method to be used. Nonetheless, an example is
needed to move the discussion forward in realistic terms. The method described below is
constructed from the methods presented in the literature.
1. Issue. Define the issues you wish to understand better in terms of time frame, scope, and
decision variables. Review the past to get a feel for degrees of uncertainty and volatility.
Know the now, i.e., understand the current situation and dynamics. Identify the focal
issue or decision, this is the key question your planning investigation is to answer.
Planners will need information on many variables to derive the answers they need for
decision making. These variables are the typical variables and criteria encountered in
Corps planning studies.
2. Identify Stakeholders. Identify the major stakeholders or actors who would have an
interest in these issues and variables, both those who may be affected by it and those who
could influence matters appreciably. Identify their current roles, interests, and power
positions.
3. Core Scenario Group. Work up the core scenario group carefully. This will be the
interdisciplinary group of planners responsible for constructing the scenarios. The group
should include five to seven members, based on common practice. This group may,
however, use any number of techniques to elicit information relevant to future scenarios
from a larger group of stakeholders and experts.
4. Identify Macroenvironmental Forces. Identify key forces in the local/project
environment, i.e. the factors that will influence the success or failure of planning
38
decisions. These factors might be hydrologic, economic, political, and the like. Make a
list of current trends, predetermined elements, or other factors that will affect the
variable(s) of interest. Constructing a diagram may be helpful to show interlinkages and
causal relationships.
5. Identify Driving Forces. Considering these key forces select the few basic conditions
most important to shaping the system being studied. These conditions are the drivers.
These drivers, found in the macroenvironment, ‘drive’ the other key forces in the local
environment. Identify the possible range of values that will be used for these drivers,
quantified if possible. Think carefully about the extremes that will be used to bound the
planning analysis. This needs to be more than variations on a theme, it is intended to
exploit the creativity and imagination of planners in a realistic sense.
6. Consolidate Drivers. Blend drivers, i.e., the key factors causing change. Make them
work together. Do not let one drive the entire scenario. Challenge conventional wisdom.
7. Identify Key Drivers. Identify key uncertainties, whose resolution will significantly
affect the variables of interest to you. Briefly explain why and how these uncertain
events matter and examine as well how they interrelate. Rank the drivers by importance
and uncertainty. Identify the two or three factors that are most important and most
uncertain
8. Define Scenario Logic. Select the scenario logics by identifying the factors that will be
used to create a visual map of scenarios. This may be based on combinations of drivers
that are internally consistent and sufficiently plausible. More often it will be based on the
two or three factors that are most uncertain and most important. The generic scenario
cross presented earlier illustrates a useful technique for identifying four different
scenarios. This process will identify the number of scenarios that will be constructed.
Keep it simple. Too many scenarios as well as complex scenarios are useless, people
cannot understand them.
9. Define Scenario. Fleshing out the scenarios. Develop them with a narrative. Describe
the evolution of conditions in each scenario. Spotlight key events and developments,
important trends, and implications for the natural and human systems of interest to your
planning study. Where possible point out the implications of each strategy for alternative
plans, policies and actions. This step is significantly different from traditional Corps
planning. The without condition is often not so much a narrative description of a possible
future with quantitative measurements of key variables to support the process so much as
it is a collection of forecasts of key variables that hang more or less loosely together.
These scenarios are different. They are more complete.
10. Iterate. Be iterative. It is often easiest to develop the scenarios qualitatively at first.
How would we get from here in the present to there in the future? What events have to
happen for that to come true? Assess the internal consistency and plausibility of these
scenarios. Identify where and why they may be internally inconsistent in terms of trends
and outcome combinations. Go back and summarize, remove contradictions, stay on
track, get to the finish point. Eliminate combinations that are not credible or are
impossible. Have a “hang-together” check at the end. Challenge your scenarios. Look
for fatal flaws, try to break the scenario. Where appropriate, identify topics for further
study that might provide stronger support for your scenarios. Create new scenarios if
necessary to achieve internal consistency and to cover a reasonable range of uncertainty.
Assess the revised scenarios in terms of how the key stakeholders would behave in them.
39
11. Get Feedback. Feed the scenarios back to those who were consulted in their
construction. Get feedback from stakeholders and others on your created scenarios.
Publicize the scenarios. When satisfied with the details of the scenarios, these will be
used in the remainder of the planning process.
Section 4 above provides discussion of the generic scenario cross as one example of a common
technique for identifying four future scenarios. The output of the work in step two includes a set
of alternative without project conditions.
4.2.3 Plan Formulation
Plan formulation is substantively unchanged by these scenario planning techniques. The basic
method of building plans from measures that meet the planning objectives is unchanged.
Scenario planning may require a new strategy, however.
In formulation, planners are trying to meet the planning objectives and thereby solve the
problems and realize the opportunities of the study area. In scenario planning they are trying to
do this in the face of considerable uncertainty. The P&G prescribe four planning evaluation
criteria for planners. These are: completeness, efficiency, effectiveness, and acceptability. To
these we now might add robustness. Formulated plans must be flexible enough, adaptable
enough and robust enough to perform at an acceptable level no matter which future is actually
realized. The methods of formulating plans are not expected to be significantly different, but the
planners’ awareness of alternative future conditions brings a new dimension to the formulation
task.
4.2.4 Evaluation
In traditional Corps planning the most likely future with a project in place is forecasted and
described after the plans are formulated in the evaluation step. It is here that the scenario
planning process diverges again from traditional planning. Evaluation still requires criteria as
always. But there will no longer be a single with project condition. Navigation benefits may
depend on how the war on terrorism progresses. It may depend on normalization of trade with
Cuba, or on global recession scenarios. This is quite different from benefits estimates that vary
with an uncertainty forecast of a commodity growth rate.
The multiple without project conditions will result in multiple performances of the project,
effectively multiple with project conditions, at least one for each of the without project condition
scenarios. How these are to be presented and how they might affect the notion of an NED plan
would be substantive matters to be resolved by policy.
The scenario planning literature is not singular on the point of how to evaluate a plan’s
performance against multiple futures. In fact, evaluation is one of the most overlooked details of
scenario planning in the literature because evaluation is a more closed process in business than it
is in water resources planning. One option is not to attach probabilities to any of the alternative
futures. This would lead to an array of plan effects that would have to be evaluated as some sort
of range of potential impacts. Alternatively, probabilities can be attached to the scenarios and all
40
quantitative plan effects then effectively become expected values. Qualitative plan effects would
remain somewhat problematic in their handling.
Although this discussion does little to resolve the issue of how best to evaluate the performance
of a plan across several future scenarios, this is a problem with a solution. There are many ways
in use in the scenario planning literature and the statistical and decision theory literature provide
numerous other options.
4.2.5 Comparison
The comparison of plan effects would seem to follow directly on the resolution of the evaluation
issues. As long as the plan effects can be quantified or summarized, it would not seem too
difficult to make comparisons.
A potential complication emerges from the fact that scenario planning offers a new dimension
for tradeoffs. Comparison of plan effects is almost formulaic, except for the necessity to make
tradeoffs. Tradeoffs require someone to say this effect is more important than that one. With
alternative futures it is quite likely that there will be plans that perform better against one future
than another. That will require someone to say which of these futures it is more important to do
well against. The answer to that cannot be prescribed in advance. It may not be the scenario that
is most likely. Instead it could be the scenario with the potential to cause the most disruption of
the system under study. Tradeoffs represent a significant issue to be addressed under a scenario
planning approach. The comparison step is where planners need to ultimately decide how the
question to be decided looks against the scenarios for each of the individual plans.
4.2.6 Selection
Selecting a plan for implementation remains a value judgment to be made by decision makers. If
the value added by scenario planning is to be realized, it is absolutely essential that decision
makers understand the culture of uncertainty under which planners operate. Decision makers
have to be engaged in the alternative futures process from early in the process. They need to
understand and value the process. They need to have had meaningful opportunities to participate
in the shaping of the scenarios.
4.3 Plan Sensitivity to Alternative Futures and Regret
Change is not easy. The incorporation of scenario analysis principles, as described above, into
the Corps’ planning process would necessitate substantial changes in the planning process. First,
there would be the need to develop alternative future scenarios. Then the plan assessment
(evaluation, comparison, and selection steps) process would be challenged to accommodate these
changes. This section presents a second option for incorporating scenario planning strengths into
the Corps’ planning process.
It would seem a point of easy agreement to assert that a necessary condition for uncertainty to be
important to decision makers is that it could affect their decisions. That means the decision
41
makers’ beliefs about the uncertainty can affect the choice of the most desirable alternative
course of action. Uncertainty matters. The sufficient condition is that there must be a significant
economic or other loss associated with that change in the decision. This is an important and
useful finding for Corps planning that can be used to modify the traditional Corps’ planning
approach.
In essence this approach relies on using alternative future without condition scenarios to conduct
a rigorous sensitivity analysis of a more traditional planning process. The first step would be to
develop alternative without project conditions as described in the previous section. After the
alternative future scenarios are identified via the above procedures, one of them is designated as
the most likely alternative future. With this as a starting point two options are described below.
4.3.1 A Selected Plan’s Sensitivity to the Future
In this approach, a recommended plan is selected according to the Corps’ traditional planning
process. The difference, to this point, is that multiple without project conditions are identified.
Suppose for illustration purposes we have four scenarios: A, B, C, and D. Let scenario A have
been designated as the most likely alternative future and Plan 1 is the selected plan. Further
suppose the plan effects of most interest to decision makers are net national economic
development (NED) benefits and habitat units (HUs). Based on an assumption of scenario A as
most likely, Plan 1 is the best plan. Plan 1 would then be evaluated against Scenario B. If either
HUs or net NED benefits for Plan 1 decrease, these losses would represent regrets for another
future materializing. A similar sensitivity would be conducted for each scenario, so Plan 1 is
evaluated against scenarios A, B, C, and D. If any of the regrets are significant, for example,
suppose net NED benefits were negative under Scenario D, then a decision must be made.
One option is to reformulate Plan 1, looking for ways to make it perform better under Scenario
D. Another option would be to try to reduce the uncertainty that keeps us from knowing whether
Scenario A or D is most likely. This could mean do some research or just waiting for uncertain
matters to begin to reveal and resolve themselves. For example, the course of the war on
terrorism will be better known in five years than it is now.
The matter of deciding what sort of regret is too great to bear will ultimately be a subjective one.
But systematically revealing the regrets is a step in the right direction.
4.3.2 Plan Selection’s Sensitivity to the Future
An alternative approach would be to assume Scenario A is the most likely future and choose the
selected plan based on traditional Corps planning methods. Then assume Scenario B is the most
likely future and choose the selected plan using traditional methods. There would be a separate
selection process for each of the future scenarios.
Finally the results of these four analyses are compared. They could produce the same plan for
each scenario or a different plan for each scenario. Thus, planners would be comparing for
example, the net benefits of the two, three or four best plans as well as the loss of net benefits for
42
each plan if an alternative scenario is realized instead. If this process is repeated many times for
alternative without and with conditions it can result in probability distributions of the relevant
evaluation metrics. Thus, if Plan 1 under scenario A yields $100, and under scenario B it yields
$85, and under scenario C it yields $46 and so on, a distribution of both net benefits (and the
expected losses) can be estimated if the scenarios are assigned probabilities of occurrence.
4.3.3 Two Scenarios
As a special case, there may be instances where there is a single uncertain driver that leads to
two distinct future scenarios. For example, suppose deep draft container traffic is diverted to
specific designated ports of entry into the US in the future.
The steps suggested in such a case (Hobbs, Chao, and Venkatesh (1997)), in brief, follow:
1. Determine whether the decision has characteristics that suggest the uncertain variable
could be relevant to the decision.
2. Evaluate the alternative plans under a scenario without the uncertain factor being
realized. That is suppose container traffic is not diverted in the future.
3. If net benefits are significantly affected by the alternative assumption (that container
traffic is diverted), assess the “regret” that would occur if a decision was made assuming
no diversion but diversion occurs anyway.
4. If the amount of regret is important, construct a decision tree with two or more diversion
scenarios for evaluating the plans. Then evaluate the expected performance of the plans
under a range of subjective probabilities for the scenarios.
5. For larger projects the benefits of waiting a decade or longer for better information on the
uncertainty could be assessed.
4.4 Sensitivity Analysis
Sensitivity analysis is already done in some Corps studies. The approach suggested here would
institutionalize sensitivity analysis as an approach to decision making under uncertainty. This
would require the least change to the Corps’ current planning process. A plan would be
formulated and assessed in the usual manner. Sensitivity analysis could either be applied to all
plans in the final array just prior to plan selection or it could be applied to the selected plan only.
As the analytical approaches are basically the same in either event no further distinction is made
between the two.
Future condition scenarios may have dozens of input and output variables that are linked by
calculations, systems of equations, assumptions, and so on. Planners and decision makers must
understand the relative importance of the various components of a scenario and they must
appreciate to some extent the influence of these many variables on the results of the planning
process.
Some outcomes and decisions are sensitive to minor changes in assumptions and input values. It
is not always immediately obvious which assumptions and uncertainties may most affect outputs,
conclusions and decisions. Consequently, thorough and rational decision-making requires an
43
explicit examination of such sensitivities. The purpose of sensitivity analysis is to systematically
investigate and find out what elements of the future condition scenarios have the greatest
influence on the planners’ recommendations and the decision makers’ decision.
Sensitivity analysis has been described as “what if” analysis. It is basically a systematic
investigation of the model parameters, inputs and assumptions as well as other factors and inputs
used in the assessment of alternative plans, including the future condition scenarios, the focus of
this discussion. There are a number of sensitivity analysis techniques to use. Parametric
variation of an input variable’s values to examine its effects on one or more output variables is a
popular approach. This variation is done in a variety of ways that include:
 Deterministic one-at-a-time analysis of each input under consideration;
 Deterministic joint analysis;
 Scenario analysis;
 Subjective estimates;
 Parametric analysis of a range of values.
 Importance analysis (regression or correlation of inputs and outputs)
Instead of approaching the entire future scenario as a holistically uncertain condition, sensitivity
analysis as described here focuses on individual components of a future scenario. Different input
values can lead to different output values. For example, if the commodity forecast for a
navigation study varies so will the estimate of project benefits. Typically a few key inputs
account for most of the variation in the output. One purpose of sensitivity analysis is to identify
these key inputs.
A good sensitivity analysis will aid the planning investigation by revealing the most important
variables in the plan assessment process. It provides insight into the conditions that contribute
the most to good and bad outcomes. Once the key inputs are identified assessors can then focus
their attention on addressing the uncertainty in these variables or carefully describing their
variability. In other words, sensitivity analysis can focus a planner’s attention on the most
important inputs. The following sections describe some techniques and methods that may prove
useful in the conduct of a sensitivity analysis.
4.4.1 Assumptions Sensitivity
Describing future condition scenarios requires analysts to make many assumptions. One useful
and often overlooked way to conduct a sensitivity analysis is to simply systematically scrutinize
the assumptions made by analysts in defining the future scenarios. The basic steps in such an
analysis follow:




List the key assumptions of your future scenario
Explore what happens as you drop each one individually
Do your decisions change?
Explore what happens as you drop them in combinations
Assumption dropping is a very effective and seldom used kind of sensitivity analysis. When a
change in an assumption results in a different decision the planning team must respond
44
accordingly. This includes reformulation of the plan or any effort to reduce the critical
uncertainty.
4.4.2 One-At-A-Time Analysis
This may be the most common analytical approach to
sensitivity analysis. It is done by holding each parameter,
input, assumption, variable and factor in your analysis constant
except one that is suspected of being a critically uncertain
value. That value is then allowed to change in a way that
captures the range of uncertainty about its true value. The
effects of the different values are then examined to ascertain
what effect, if any, it has on the planners’ decisions.
An example might be to hold everything constant in a
navigation study but the commodity forecast. Then net
benefits are recalculated several times using different
commodity forecasts.
One-at-a-time analysis (OAATA) can be useful but it can also
be dangerous, depending on the context of the analysis. For
example, imagine a complex mathematical model that is used
to define a future scenario. It is important for the analyst
conducting the sensitivity analysis to be intimately familiar
with the structure of the model. See the textbox for an
example.
Suppose a model includes
the following function.
If A<50 then
C=B+1
Else
C = B100
Imagine an analyst trying
various values of B to see
how it affects C. When the
analyst holds all other things
constant what value of A of
A is assumed? In a case
like this, OAATA can provide
misleading results.
4.4.3 Joint Analysis
This is another ceteris paribus analysis, like one-at-a time analysis, but it changes more than one
thing at a time. This is in part a solution to the kinds of problems illustrated in the text box. It
enables analysts to let several variables change at once in a rational way, thus taking known
dependencies explicitly into account. It can have the same limitations as one-at-a-time analysis,
however, if all dependencies are not accounted for.
4.4.4 Scenario Analysis
This approach would have analysts prepare an entire alternative future condition scenario simply
to test the sensitivity of the planning decision to an entirely different future outcome. It is a
specific and limited application of the multiple without condition scenarios in the two preceding
sections.
45
Used in this way, scenario analysis can help identify important variables, i.e., variables to which
planners’ decisions are sensitive. This differs from the multiple scenario approaches previously
in that planners may choose to use scenarios not chosen so much for their likelihood as for their
ability to enable the analysts to explore the robustness of their decision. See the textbox for
examples of scenarios that might be used in this kind of sensitivity analysis.
4.4.5 Subjective Estimates
Subjective estimates of the value of an uncertain variable can be
used to identify threshold values of importance to the analysis.
For example, one could use trial and error to find a discount rate
at which net benefits become negative. One fairly common
approach to this method is to identify the 10th, 50th, and 90th
percentile values of each potentially important variable in your
analysis. Holding all other variables constant the model outputs
of interest, for example, net benefits, are calculated using all
possible combinations of these percentiles for the variables for
which such subjective estimates were prepared.
4.4.6 Importance Analysis
Sample Scenarios

Optimistic/Best Case

Pessimistic/Worst case

Maximum/minimum

Most Likely

No Action

Locally preferred

Regulatory options
As mentioned previously, typically a few key uncertain inputs
account for most output uncertainty. These are the important inputs. Investigations that make
use of simulation analyses produce numerous input and output values that can be used in
importance analysis. In essence, importance analysis uses the simulation data for an output of
interest identified by analysts and regresses or correlates each input that contributes to the
estimation of that output against it. The statistical results enable the analyst to rank inputs by
their statistical significance. This is computationally demanding and requires a separate
regression or correlation for each input and output pair.
4.4.7 What Does It All Mean?
When a planning decision is sensitive to changes or uncertainties in a future scenario that are
within the realm of possibility then more precision and additional information may be required.
It may be necessary to do additional analysis, conduct some research or to inform decision
makers so their choices can appropriately weight the evidence uncovered by the sensitivity
analysis.
With each sensitivity analysis result analysts test fly their alternative plans. Sensitivity analysis
enables planners to experiment with the performance of their plan and to revise it according to
what they learn about its performance. Each sensitivity analysis provides new information about
the plan and helps planners decide whether to modify it in some way to perform better. Planners
can modify the features, scale, timing, space, material and so on of a plan in response to
46
sensitivity analysis. The objective is to improve the performance (as defined by planning
objectives) of the plan in the face of often-controversial uncertainty.
5.0 Conclusions
The Corps’ planning process relies on forecasts of the future that use a single most likely
alternative future. The use of such deterministic scenarios is rooted in the desire for a single
right answer. Far too often deterministic scenarios are anchored in the present and overconfident
in its projections of the future and its current models. As the world grows more complex and the
pace of change becomes increasingly more rapid this approach often results in dangerously
conservative strategies in the face of significant uncertainties. It also can result in adversarial
planning processes, based on legitimate differences in stakeholder views of an uncertain future.
Scenario planning is an alternative to the Corps traditional planning process that offers a method
more suitable for addressing the uncertain, rapidly changing and challenging future in a much
broader context. Scenario analysis has been developed as a strategic planning tool to
counterbalance the deficiencies of a planning process rooted in trend extrapolation.
The recent controversy over the Upper Mississippi Locks stands as a singular example of how a
different future would make all the difference for the economics of a project to improve the
waterway on the Upper Mississippi. Scenario planning offers the opportunity in cases such as
these to address the uncertainty about the future in a more straightforward and perhaps less
controversial way.
Scenario planning would encourage Corps planners, managers and stakeholders to think more
broadly about the future. The benefit is not more accurate forecasts but "better decisions about
the future." This is because the purpose of scenarios is not to predict the future but to improve
the Crops’ abilities to adapt to it.
The culture of the Corps and its planning process, however, could make it difficult to overcome
the barriers to creativity that must fall if scenario planning is to succeed. Scenarios are valuable
as long as they cause a new form of interaction among those who must decide and act. In the
Corps’ context this would suggest new forms of interaction among decision-makers, planners,
and stakeholders. Scenarios need to be able to integrate different interest groups in the planning
process if they are to be useful for public agencies. The Corps’ planning process is heavily
regulated and lacking in flexibility. The Corps does not currently have a culture that is oriented
toward addressing and dealing with the uncertainty that results form our complex and rapidly
changing world.
It is quite clear that the Corps’ military culture with its deference to higher authority, its reliance
on established (synonymously past) policy, and practices codified in copious detail in
Engineering Circulars, Regulations, and Pamphlets and the like operates in a reality that is
47
heavily burdened by the past. That heritage, along with the military command structure, makes
the Corps less flexible and able to respond effectively and efficiently to the rapidly changing
environment of the present. This reality will go a long way in determining if and how the Corps’
planning process is modified to take advantage of any potential benefits offered by scenario
analysis.
As the scenario literature has developed from the 1970s there seems to be a maturation process in
the scenario concept and its use. Initially, scenarios represented an alternative to deterministic
forecasts of the future that were increasingly inaccurate and incapable of identify turning points
and significant changes in the environment. Over time, however, there has been a decided shift
toward the notion that scenarios are not forecasts of the future. Much as the story of the ghost of
Christmas future in Dicken’s A Christmas Carol, scenarios are more like descriptions of what
could be than they are what will be. And as was true for Scrooge, changes in present behavior
can help one not only adapt to the future but to shape it as well.
Scenario analysis focuses on the definitions and techniques of developing scenarios. It also
includes using scenarios in a structured sensitivity analysis of a management measure of some
sort. Scenario planning or scenario-driven planning overlaps the definition and development of
scenarios but it takes a different focus on the use of the scenarios. The scenarios are used to help
formulate, evaluate and choose future courses of action. And although there are examples of
such use in water resources (See Appendix) and public investment, the greatest use of these
planning techniques has been in the arena of strategic planning.
This usage is significantly different from the Corps’ current Civil Works water resources
planning practices. One of the major ways in which this is true, is in the private sector’s
emphasis on flexible plans and the Corps emphasis on public works infrastructure investments
that are to a great extent irreversible, the antithesis of flexibility. One of the keenest innovations
of scenario planning is its ability to get managers to focus on the future by taking a 15 to 20 year
view into the distance. In the 1960’s it was commonplace for the Corps to be using a 100-year
planning horizon, which has been shortened to 50 years.
For the Corps, its heritage and these differences in planning applications present a choice with
respect to making use of scenario-driven planning techniques. Should the Corps adopt or adapt
the common practice of scenario planning? Or perhaps it has nothing of value to offer the Corps.
As noted above, the Corps relies on the use of most likely forecasts of the future without a
project and with a project. These scenarios, once defined, are treated as rather deterministic.
However, individual elements of the scenarios are often treated as risky or uncertain. Expanding
and formalizing the manner in which uncertainties within a scenario are treated is one possible
response by the Corps. Another would be to take what is useful and adaptable that provides
value added to the Corps planning process in at least some situations and adapt it for use. A
third, and far less likely option is to adopt the scenario-driven planning process as its own.
To make any use of scenario planning will require that the Corps see the future differently. This
is not a subtle change that can be made for this study here or that study there. The organization
has to recognize the pressing need to acknowledge and address the uncertainty that pervades
their Civil Works function. The first step is an organizational one. The Corps must first
48
acknowledge its inability to control the uncertainty in the world around them, and then resolve to
change its behavior by creating a culture of uncertainty.
49
Appendix A
Scenarios in Water and Other Resource Contexts
The literature pertaining to water and natural resources is voluminous. An Ingenta search for key
words “scenario water resources” for example, produced 54 articles from 1997 to 2003. All the
abstracts were read and the most promising articles were read in their entirety. The majority of
this literature took a much narrower view of scenario planning and analysis than is generally
helpful to the Corps planning process. A sample abstract, chosen because it is representative of
the general lack of germaneness of most of the literature, is provided in the textbox.
Upper Yellowstone River Flow and Teleconnections with Pacific
Basin Climate Variability during the Past Three Centuries
Climatic Change,
July 2003, vol. 59, no. 1-2,
pp. 245-262(18)
Graumlich L.J.[1]; Pisaric M.F.J.[2]; Waggoner L.A.[2]; Littell J.S.[2]; King
J.C.[2]
[1] The Big Sky Institute, Montana State University, Bozeman, MO 59717-3490,
U.S.A.; E-mail: lisa@montana.edu [2] The Big Sky Institute, Montana State
University, Bozeman, MO 59717-3490, U.S.A.
Abstract:
Climate variability, coupled with increasing demand is raising
concerns about the sustainability of water resources in the western
United States. Tree-ring reconstructions of stream flow that extend the
observational record by several centuries provide critical information
on the short-term variability and multi-decadal trends in water
resources. In this study, precipitation sensitive Douglas-fir
(Pseudotsuga menzeisii) tree ring records are used to reconstruct
annual flow of the Yellowstone River back to A.D. 1706. Linkages
between precipitation in the Greater Yellowstone Region and climate
variability in the Pacific basin were incorporated into our model by
including indices Pacific Ocean interannual and decadal-scale climatic
variability, namely the Pacific Decadal Oscillation and the Southern
Oscillation. The reconstruction indicates that 20th century streamflow
is not representative of flow during the previous two centuries. With
the exception of the 1930s, streamflow during the 20th century
exceeded average flows during the previous 200 years. The drought of
the 1930s resulted in the lowest flows during the last three centuries,
however, this probably does not represent a worst-case scenario for the
Yellowstone as other climate reconstructions indicate more extreme
droughts prior to the 18th century.
Relatively few articles provide examples of
the primary thread (multiple future
conditions against all of which a plan is
evaluated) described in Section Three
above. Most articles that address scenario
analysis tend to be of the second thread
(techniques for dealing with specific sources
of significant uncertainty). Consequently,
the format of this section will differ from
that of the rest of this review. This section
will provide a brief review of the most
relevant articles found in this research.
One of the more promising areas of the
resource literature for scenario planning is
climate change. There have been a great
number of articles that make some use of or
reference to scenario analysis. It is quite
common to see articles that make use of
different scenarios for climate change.
Doubling the CO 2 levels is an example of
such a scenario. The foci of articles are
often on the model used to generate these
scenarios or the techniques needed to scale
these scenarios from a large spatial scale to
a smaller scale. Climatic Change is a journal
dedicated to the topic that contains
numerous articles that make some use of scenarios. The September 1997, Volume 37, Issue 1
edition of the journal contained several useful articles, three of which are summarized below.
Climate Change and Water Resources, Kenneth D. Frederick and David C. Major Climatic
Change 37 (1): 7-23, September 1997
50
The authors introduce some current perspectives on global climate change based on then-recent
reports of the Intergovernmental Panel on Climate Change (IPCC). Changes in precipitation
and runoff patterns, sea level rise, land use and population shifts following from these effects,
and changes in water demands are presented as potential impacts of a greenhouse effect that
would affect water planning and evaluation. Irrigation water demands were identified as
particularly sensitive to some of these climate changes. A key emphasis of the article was the
substantial uncertainty that remains as to how and when climate will change and how these
changes will affect the supply and demand for water at the river basin and watershed levels. In
an effort to bound the uncertainty of climate factors the authors explored the influence of nonclimate factors such as population, technology, economic conditions, social and political factors,
and the values society places on alternative water uses are considered on the supply and demand
for water. The authors conclude that our track record anticipating these uncertainties has not
been especially good and neither should we expect our ability to anticipate climate change
uncertainties to be.
Assessing Urban Water Use and the Role of Water Conservation Measures under Climate
Uncertainty, John J. Boland Climatic Change 37 (1): 157-176, September 1997
This article was unique for its focus on the effects of climate change on urban water use. The
paper takes a look at the suitability of various water use forecasting models for predicting
climate impacts or of the best procedures for assessing this issue. The paper argues that a
scenario approach to describing possible changes in climate is useful. It uses six climate change
scenarios. None of them is intended as a prediction of climate change but taken together they
capture the likely range of uncertainty about climate. This range permits an assessment of the
sensitivity of different water management alternatives to climate change. The author also
evaluates the IWR-MAIN model as a source of plausible water use forecasts given uncertain
future climate. This article used scenarios to describe a portion of the range of uncertainty, and
as such it represents a slightly different application of scenarios to a problem evaluation.
Using Decision Analysis to Include Climate Change in Water Resources Decision Making,
Benjamin F. Hobbs, Philip T. Chao, and Boddu N. Venkatesh, Climatic Change, 37 (1):
177-202, September 1997
The authors’ take off point for this article is that a necessary condition for uncertainty to be
important to managers is that it could affect their decisions. That means the manager’s beliefs
about the uncertainty can affect the choice of the most desirable alternative. Uncertainty matters.
The sufficient condition is that there must be a significant economic or other loss associated with
that change in the decision. This is an important and useful finding for Corps planning.
Using the expected cost of ignoring uncertainty (ECIU) the authors offer a framework for
approaching uncertainty using a climate change example. Briefly, the ECIU is found by
evaluating an expected value like net NED benefits when the probability of the alternative
scenario is zero and subtracting from it the expected value of net benefits for the plan that is
optimal under the alternative scenario. So, for example (using this reviewer’s numbers) if the
best plan with a no climate change scenario has benefits of $100 and the best plan with a climate
change scenario has benefits of $60 the ECIU is $40.
51
The article provides a good example of a well-articulated climate change scenario (p. 182) that
was applied to two test cases. It may be worth emulating if an example is needed at some future
point. A no climate change scenario and one climate change scenario are compared. Of greatest
value to the Corps’ planning process is a five step process using a scenario tree that holds
considerable promise for addressing alternative scenarios. The steps, in brief, follow:
1. Determine whether the decision has characteristics that suggest that climate change
could be relevant.
2. Evaluate the options under a climate change scenario.
3. If net benefits are significantly affected, assess the “regret” that would occur if a decision
was made assuming no climate change but global warming occurs anyway.
4. If the amount of regret is important, construct a decision tree with two or more climate
scenarios for evaluating the options. Then evaluate the expected performance of the
options under a range of subjective probabilities for the scenarios.
5. For larger projects the benefits of waiting a decade or longer for better information on
climate change could be assessed.
Assessing Climate Change Implications for Water Resources Planning, Andrew W. Wood,
Dennis P. Lettenmaeir, and Richard N. Palmer, Climatic Change, 37 (1): 203-228,
September 1997
Noting that many studies show water supply systems are sensitive to climate change, the authors
ask if planning methods should be modified accordingly. This paper identifies three principal
sources of climate change uncertainty in water resources planning. They are: a) climate
modeling; b) scaling model results from global to regional levels; and, c) water demands. In the
process of going from GCM-based studies to water resource management strategies there can be
a cascade of uncertainty. The resulting uncertainty makes it difficult to prescribe a set of steps
for addressing the unknown pragmatically.
Of interest to this review is a simple approach proposed and used in a case study to demonstrate
how the planning process might be modified to address these uncertainties. The methodology is
generalized from its climate change application to a more traditional planning approach. For
ease of explication assume two alternative scenarios A and B. They could be alternative without
conditions or alternative with conditions for the same plan. One could be with climate change
the other without; or one with Cuban trade the other without; or one with 5% commodity growth
the other with 1%. The possibilities are endless and there is no need to restrict the approach to
two scenarios.
First, a plan is selected using traditional planning techniques, i.e., most likely alternative futures
without and with the project are identified. Assume scenario A is the most likely with condition.
The best plan using scenario A is identified. Then this plan is evaluated under scenario B.
Second, the best plan is picked assuming scenario B. This plan is then evaluated under
conditions of scenario A.
52
Finally the results of these two analyses are compared. For example, the net benefits of the two
best plans could be compared as could the loss of net benefits for each plan if the alternative
scenario is realized instead. The authors suggest that if this process is repeated many times for
alternative without and with conditions (if appropriate to the situation) it can result in probability
distributions of the relevant evaluation metrics. Thus, if Plan 1 under scenario A yields $100,
and under scenario B it yields $85, and under scenario C it yields $46 and so on a distribution of
both net benefits and the expected losses can be estimated if the scenarios are assigned
probabilities of occurrence. This latter point is this reviewer’s observation.
In the case study done by the authors, it was found that the prospect of climate change did not
play an important role in the best reallocation plan.
Water Resources Planning Principles and Evaluation Criteria for Climate Change:
Summary and Conclusions, Kenneth D. Frederick, David C. Major and Eugene Z. Stakhiv
Climatic Change, 37 (1): 291-313, September 1997
This paper summarizes the findings of the many articles in this volume of Climatic Change. The
prospect of anthropogenically-induced climate change presents water planners with a variety of
challenges. With respect to the six-step planning process detailed in the Economic and
Environmental Principles and Guidelines for Water and Related Land Resources Implementation
Studies (P&G) the methods of sensitivity analysis, scenario planning, and decision analysis that
are encouraged by the P&G are found to be generally appropriate for planning and project
evaluation under the prospect of climate change.
The review found here does not summarize Frederick et al.’s summary of the articles. The most
useful of the articles are reviewed elsewhere. There were several findings among the conceptual
issues raised in this article that are interesting because they can be generalized for consideration
of other planning problems with substantial uncertainty. Much of what follows is based on the
arguments of Frederick et al. and the opinions of this author.
Bearing in mind the climate change orientation of these water resource issues, the authors note it
is problematic because the timing and nature of the change are both uncertain. And the scale of
the uncertainty is, perhaps, different from what planners have approached in the past. These
considerations seem to be easily generalized beyond climate change uncertainty for some
planning investigations.
Any significant uncertainty can raise issues about intergenerational equity. Generalizing from
the authors’ findings it can be argued that when faced with large and significant uncertainties,
water resource plans should maintain options and build in dynamic flexibility. The climate
change article suggests that it may well be worth waiting and building a project later rather than
now if doing so reduces the probability of a bad outcome.
Particularly interesting were the authors’ comments on adaptation through infrastructure
investments, beginning at p. 300. Generalizing, uncertainties can sometimes cause significant
shifts in hydrologic regimes (in the case of climate change), or in economic factors like supply
53
and demand or supply chains, social values, and ecosystem performance. When the magnitude,
timing and direction of the shifts are uncertain it can be very difficult to plan infrastructure
investments.
Matalas, N.C. (1997) Stochastic Hydrology in the Context of Climate Change
Climatic Change, September 1997, vol. 37, no. 1, pp. 89-101(13)
Kluwer Academic Publishers
Matalas suggests using what-if analysis to assess the robustness of alternative designs. Frederick
et al. suggest that perhaps a robustness criterion can be introduced to the planning process. This
would presumably join the completeness, effectiveness, efficiency and acceptability criteria of
the P&G. The authors suggest a move away from finding the “right” design as has traditionally
been done in plan evaluation to a new emphasis on finding the robust design. A robust design
may not be best under any given scenario but it is fairly good under a wide range of outcomes.
Such an idea is intriguing but it represents a sea change in attitude for some planners. This is
also an idea that fits well into the scenario planning mentality.
Developing resource management systems that are more flexible and responsive to changes in
the underlying assumptions about future conditions is a valid approach to addressing uncertainty
from any source. The authors assert that institutional flexibility that complements or substitutes
for costly infrastructure projects is important for water resources planning. This is a very timely
idea. But it marks a change in the way of doing planning that will cause some challenges. For
example, the goal of maximizing net NED benefits has, arguably, always been one of hitting a
moving target. There have always been uncertainties in project costs and benefits for a variety of
reasons, project performance and the future being only two of them. Consequently,
identification of an NED plan was an informed guess at best. Creating a culture of uncertainty
within the Corps will require flexibility. And in the current instance that flexibility will require
making explicit many things that have not been made explicit in the past. For example, the NED
plan must be more formally and publicly recognized as a moving target. And the best economic
plan may not always be the one with the highest expected value for net NED benefits.
The authors here and in several articles in this volume of Climatic Change argue persuasively
that the advantages of postponing costly, one-of-a-kind and perhaps irreversible responses to
potential future situations with significant uncertainties for as long as possible or at least until the
uncertainty can be reduced may often be a viable strategy.
The next two articles demonstrated the continuing relevance of both threads of scenario planning
to resource management. They demonstrate that multiple and contrasting scenarios can be
incorporated into resource management decisions. The first article indicates the importance of
sensitivity analysis to explore the residual uncertainty in different scenarios. The scenarios in
these instances resulted more from a model-based process than an overtly expert-judgment
process such as the Corps is likely to use. They differ from the preceding articles in that they do
not focus on water resources.
54
Climate change and winter wheat management: A modeling scenario for Southe-Eastern
England, Ghaffari, A., Cook, H.F., Lee, H.C., Climatic Change vol. 55, no. 4, December
2002, pp. 509-533
Weather and climate represent the two major uncertainties in agricultural production. Obtaining
a consensus on the direction of climate change is no simple matter as more and more scenarios
are created. The authors use the dynamic crop-growth model, CERES-Wheat, to examine crop
management responses, including six climate change scenarios for the years 2025 and 2050. It is
noted that these scenarios are “probable” but the probability is not developed. Large
uncertainties remain in these scenarios due to a general lack of data. Differences in temperature,
CO 2 and rainfall were used to create the basic scenarios. The focus of this article is more on the
CERES-wheat crop simulation methodology than it is on generally applicable principles that can
be adapted by Corps planners. A sensitivity analysis was conducted using a one-at-a-time
investigation of increases in temperature alone. Wheat yields were then constrained to assess
crop performance under water-limited production scenarios with different soils. What was
interesting was that different management practices like planting dates and nitrogen application
were applied to find the best adaptation strategies across all scenarios. This was one of the few
examples of the first thread of scenario planning found in the resource literature. There was
nothing of specific value to the Corps planning process in this article.
Assessing Winter Wheat Responses to Climate Change Scenarios: A Simulation Study in
the U.S. Great Plains, Weiss A., Hays C.J., Won J., Climatic Change vol. 58, no. 1-2, May,
2003, pp. 119-147
The uncertainty associated with climate change has spawned a large literature. As is common in
much of the professional literature a string of related articles often appears on a topic and this
article is closely related to the preceding one. The authors consider the effect of climate on hard
red winter wheat in the Great Plains region of the U.S. It investigated the effects of two
contrasting global climate change projections (one from the UK and one from Canada), on the
yield and percent kernel nitrogen content of winter wheat at three locations in Nebraska. These
locations represented different moisture conditions. In this article the emphasis was more on the
tools used and the implications for agricultural management. The climate scenarios were based
on projections using the LARS-WG weather generator along with data from automated weather
stations. CERES-Wheat was also used in this study to simulate the responses for two kinds of
wheat and two sowing dates. From the resulting analysis proactive steps to meet the challenges
of global climate change as represented by these climate scenarios were recommended.
A scenario-based stochastic programming model for water supplies from highland lakes,
Watkins, D.W., McKinney, D.C., Lasdon, L.S., Nielsen, S.S., Martin, Q.W.
International Transaction in Operational Research 7 (2000), pp. 211-230
The authors use a scenario-based, multistage stochastic programming model to explore
management of the Highland Lakes by the Lower Colorado River Authority (LCRA) in Central
Texas. Thirty scenarios were generated by the model and were solved using both a primal
simplex method and Benders decomposition. The results show the amount of water to contract
55
for the coming years is highly dependent on the initial reservoir storage levels. The article tends
toward a more technical focus on the modeling technique.
Unlike deterministic models that select values of decision variables with ‘perfect
knowledge of the future,’ scenario-based stochastic programming models consider a number of
possible futures. This particular model is proposed for use in ‘here and now’
decision making as well as providing a number of `wait and see' strategies dependent on which
scenario unfolds. The method allows large-scale problems to be decomposed by scenario and
solved in a nested manner with inputs represented as a scenario tree. The objective function for
the model was a weighted combination of two goals, roughly maximizing water sales revenues
and maximizing recreational benefits.
A scenario in this model was defined to be a sequence of monthly available flows. Available
flows are defined as those in excess of environmental needs and water rights senior to the
LCRA’s. Monthly inflow data were used to generate scenarios. The model is solved using
GAMS and SP/OSL. The authors argue that by explicitly considering a number of inflow
scenarios, the stochastic model can determine a contract level that balances interruptible water
supply and recreational goals while appropriately hedging against the effects of drought. This
article offered little pragmatic information for the Corps’ planning process.
Toward a scenario analysis framework for energy footprints, Jiun-Jiun Ferng
Ecological Economics 40 (2002) pp. 53-69
The ecological footprint is an index developed for quantifying humanity’s dependence on
ecosystems. It measures land and water areas required to support the resource provision and
environmental assimilation necessary to satisfy the consumption needs of a human population.
The author proposes a framework that enables scenario analyses of policy instruments that could
reduce energy footprints. In other ways he uses scenario analyses to look for ways to reduce
energy needs. The purpose of this paper was to demonstrate the feasibility of the framework
rather to actually apply it. The author compared the results of one hypothetical scenario to a
baseline.
The paper uses a six-stage calculation. Final domestic demand is estimated for a new policy
instrument using a computable general equilibrium (CGE) model. The use of the CGG model
(an equation-based simulation model) is a centerpiece of this research. Next, the sectoral outputs
necessary to satisfy the final demand are estimated using an input-output analysis. Stage three
estimates the various final energy consumption required to produce the sectoral outputs. Then
the primary energies of various kinds needed are calculated using the I-O analysis. Energy
footprints are calculated for the various primary energy requirements estimated and the final
stage is to estimate the deficits of energy lands. Energy footprints and deficits can be compared
for different scenarios in this manner.
An adaptive approach to planning and decision-making, Gene Lessard, Landscape and
Urban Planning 40 (1998) 81-87
The conclusion of this article is a good place to begin. It says:
56
“So why an adaptive approach? Clearly, there are critical uncertainties in our knowledge
base which will provide a continuous supply of surprise events. Since we will never have
perfect information, we will continually learn from the response of ecosystems to
implementation of our decisions. Planning for and adapting to surprise will provide an
actionary rather than a reactionary basis for more informed decisions.”
This article proved quite helpful in stimulating potentially useful thoughts on the Corp’s
planning process. This review departs from the standard approach found here by presenting the
abstract verbatim. The article, although geared toward adaptive management, lends itself well to
the more traditional planning processes like the Corps through simple analogical thinking, which
follows the abstract in the next paragraph.
Article Abstract
A formal process of adaptive management will be required to maximize the benefits of any
option for land and natural resource management and to achieve long-term objectives through
implementation of ecosystem management. The process itself is straightforward and simple: new
information is identified, evaluated, and a determination is made whether to adjust strategy or
goals. While relatively straightforward, applying the concept of adaptive management to
complex management strategies requires answers to several critical questions. What new
information should compel an adjustment to the management strategy? What threshold should
trigger this adjustment? Who decides when and how to make adjustments? What are the
definitions and thresholds of acceptable results? Adaptive ecosystem management depends on a
continually evolving understanding of cause-and-effect relationships in both biological and
social systems. Planning for and adapting to surprise will provide an actionary rather than a
reactionary basis for more informed decisions.
This article stimulated some ideas that are captured here. To adapt the author’s ideas, the initial
hypothesis of interest in the context of the Corps’ planning process is the performance of a plan
using the traditional evaluation process that compares the without and with project scenarios.
New information becomes alternative scenarios based on a sensitivity analysis of key
uncertainties in the process. This new information (an alternative without or with project
condition) is evaluated to provide feedback on the formulated plan for the purpose of
reaffirming, refining, or reformulating the plan.
With each scenario created by the planning team, planners can experiment with the performance
of their project. In the process they can learn effectively about the potential performance of their
project under scenarios other than the one most anticipated. Each scenario provides new
information about the plan. Planners can use that information to decide whether to modify it in
some way to improve its performance.
Reformulation could effectively become a new planning step. It is not warranted in all or
perhaps even most cases. But reformulation may be an effective strategy when: 1) uncertainties
are large or numerous; 2) plans are irreversible; or, 3) there is controversy and disagreement.
This step would include examination of uncertainties through the use of alternative without or
57
with condition scenarios, as the case may warrant. Unacceptable project performance 5 under
reasonable alternative scenarios would result in a “back to the drawing board” effort to
reformulate the plan specifically to improve its performance under the alternative scenarios as
well as the most likely scenario.
Environmental Management scenarios: Ecological implications, Ogden. J.C., Browder,
J.A., Gentile, J.H., Gunderson, L.H., Fennema, R., Wang, J., Urban Ecosystems 3 (1999)
pp. 279-303
The authors assert that prevailing scientific consensus is that the current spatial extent and
patterns of ecology and hydrology do not support a sustainable Everglades or South Florida
ecosystem. As a part of the U.S. Man and the Biosphere (US MAB) Human-Dominated Systems
Directorate (HDS) project, five plausible, regional-scale environmental management scenarios
were developed to illustrate the potential for recovery of the physical defining characteristics of
the South Florida system. In essence the authors describe the management of a range (five
scenarios) of possible land additions to core and buffer areas intended to meet spatial-scale
requirements and to achieve different degrees of hydrological improvement to examine their
potential for ecological sustainability. Their evaluation was based on specific hydrologic
characteristics for each scenario. The measure of ecological sustainability is the degree to which
a scenario recovers the defining ecological characteristics of the regional landscape mosaic.
A set of eight “risk hypotheses” was used to show the relationship between the human-caused
alterations in the defining physical characteristics and the resulting losses of ecological
sustainability in these wetlands. The hypotheses identify the physical parameters of interest and
enable each scenario to be evaluated on the basis of its capacity to recover the predrainage
conditions for each parameter. The authors’ assessment of the five scenarios suggests they would
all improve the problems addressed by the eight hypotheses.
A transferable point for the Corps’ planning process would be to explicitly identify the
hypotheses that underlie a formulated plan. These hypotheses would serve as a criteria for plan
evaluation and comparison, albeit more complex criteria than are normally encountered. Stating
hypotheses would be a trivial exercise in busy work for some studies. For example, to develop a
hypothesis for how a floodwall would reduce flood damages is silly, as would be a hypothesis to
suggest that a larger lock can pass more cargo. But the Everglades project is also a Corps
activity and it is evident from this article and much other literature that hypotheses can be
valuable management and decision aids for complex or large projects with significant complexity
and uncertainty. The art then is in ascertaining when the planning process might best be aided by
the definition of specific hypotheses that capture and embody the essence of the critical
uncertainties in a planning investigation.
Component ecological footprint: developing sustainable scenarios, Barrett J.
Impact Assessment and Project Appraisal, 1 June 2001, vol. 19, no. 2, pp. 107-118(12)
The focus of this article is more on the concept of a “component ecological footprint” than it is
5
Examples of unacceptable performance might be negative net NED benefits, unacceptable ecosystem losses, or
significant stakeholder opposition.
58
on scenario planning, although the footprint analysis is proposed for use as a regional planning
tool. The author calculated the ecological footprint of waste, transport, energy, water, and bioresources for Guernsey (Channel Islands) using an integrated resource accounting framework, an
apparent innovation of this approach. By altering the inputs to the model one is able to identify
a footprint that is considered sustainable. The inputs leading to this desired result then provide
guidance for regional planners. The ecological footprint was used to offer new insights into
regional sustainability and what a sustainable society might look like. This research is more
valuable for its ecological footprint, but it does demonstrate that scenarios of idealized
benchmark futures can be imagined or created and plans can be developed to achieve these
scenarios.
WARSYP: a robust modeling approach for water resources system planning under
uncertainty, Escudero L.F., Annals of Operations Research, 2000, vol. 95, no. 1/4, pp. 313339(27)
This article was of interest because it explicitly addressed decision making under uncertainty.
The main elements of this problem were water resource sources (surface and groundwater),
water demands (hydropower generation, irrigation, industrial, domestic, recreation, and ecology),
and infrastructure (reservoirs and distribution systems). A multiperiod optimization model
(WARSYP) was used to address the uncertainty in these main elements.
Instead of using a mathematical programming model to capture the uncertainty the authors used
scenarios to capture the uncertainty caused by random parameters in the model. The basic
problem was to balance water resource supplies and demands over a multiperiod planning
horizon. A multistage scenario tree was used along with “full recourse” techniques to solve the
model.
The scenario model is described in considerable detail and it has value in its thoroughness. The
paper is a good demonstration of the feasibility of addressing uncertainty in a scenario tree
model. The paper is not without application to the Corps but it’s true value would be limited to
problems of a water balance. It is not a generally applicable approach.
Development of an environmental flows decision support system, Young W.J.; Lam D.C.L.;
Ressel V.; Wong I.W., Environmental Modelling and Software with Environment Data
News, March 2000, vol. 15, no. 3, pp. 257-265(9)
This article provides a good example of a reasonably common use of the scenario terminology.
The authors examine the desirability of different flow management scenarios. This is much
more closely aligned to the Corps’ planning process than other uses of the terminology.
Scenarios as used in this article are like alternative plans in the Corps’ jargon. The Murray–
Darling Basin in Australia is severely environmentally degraded as a result of a range of
anthropogenic changes. Withdrawal of irrigation water is the principle stressor. The resulting
environmental problems include eutrophication of rivers and storages, elevated salinity levels,
widespread blooms of toxic blue–green algae, decline of native fish and bird populations, and
reduction of area of riverine wetlands. To facilitate the on-going trade-off process between
competing water uses, the authors are developing a decision support system (DSS) that will
59
enable explicit prediction of the likely response of key features of the riverine environment to
proposed flow management scenarios. The DSS, which is the true focus of this article, will
integrate a range of simple qualitative and quantitative models of riverine ecology that are being
developed.
Water Resources Implications of Global Warming: A U.S. Regional Perspective
Lettenmaier D.P.; Wood A.W.; Palmer R.N.; Wood E.F.; Stakhiv E.Z., Climatic Change,
November 1999, vol. 43, no. 3, pp. 537-579(43)
The original formulation of scenario planning created alternative scenarios and analyzed the
performance of strategies or plans across all the scenarios. This study reversed that sequence and
examined the implications of global warming for the performance of six existing U.S. water
resource systems. The six case study sites represent a range of geographic and hydrologic, as
well as institutional and social settings. The studies essentially examined the sensitivity of six
water resources systems to changes in precipitation, temperature and solar radiation. Thus
alternative scenarios were “run” against a variety of existing projects, not for the purpose of
choosing the best project but to examine the robustness with which the projects would perform.
A standard experimental design, consisting of specific base case and hypothetical altered climate
simulations and similar evaluation measures for all sites, was used to maintain consistency between the
six parts of the study. A sequence of models was used to infer the water resources effects of each of the
climate change scenarios.
The climate change scenarios used in this study are based on results from transient climate change
experiments performed with coupled ocean-atmosphere GCMs for the 1995 Intergovernmental Panel
on Climate Change (IPCC) assessment. The effects of climate change on system performance
varied from system to system, from climate change model to climate change model, and for each
system operating objective, such as hydropower production, municipal and industrial supply,
flood control, recreation, navigation and instream flow protection.
Where possible, the effects of climate change were compared with the effects of non-climate related
changes that could plausibly take place over the same period as the climate changes measured in these
assessments (1990–2050). The studies showed, among other things, that these non-climate
changes, such as demand growth and operational changes, had effects as large or larger than
climate change. This study represents a useful data point in the investigation of climate change
impacts on water resource infrastructure and systems. It did not contribute appreciably to
scenario planning. Although it did use a variation of the original scenario-planning concept in its
research design, this was more incidental to the study purpose.
Water Resources Planning Principles and Evaluation Criteria for Climate Change:
Summary and Conclusions, Frederick K.D.; Major D.C.; Stakhiv E.Z., Climatic Change,
September 1997, vol. 37, no. 1, pp. 291-313(23)
The prospect of anthropogenically-induced climate change presents water planners with a variety
of challenges. Drawing on work presented in this volume, these challenges are summarized and
conceptual issues surrounding strategies for adapting water planning and project evaluation
practices to this prospect are examined. The six-step planning process detailed in the Economic
and Environmental Principles and Guidelines for Water and Related Land Resources
60
Implementation Studies (P&G) is described; its’ ability to incorporate consideration of and
responses to possible climate impacts is assessed. The methods of sensitivity analysis, scenario
planning, and decision analysis that are encouraged by the P&G are found to be generally
appropriate for planning and project evaluation under the prospect of climate change. However,
some important planning and evaluation criteria require review and possible adaptation. The
Intergovernmental Panel on Climate Change (IPCC) impact assessment procedures are found to
be particularly useful as a framework for climate change impact and sensitivity analyses, and
would fulfill the requirements for future environmental impact statements. The ideas and
principles are compatible with those found in the P&G. The water resources guidelines in the
P&G deal explicitly with the specific comparison, appraisal, and selection of project alternatives
based on normative decision rules associated with benefit cost analysis and maximizing national
welfare. These basic rules and normative decision criteria for evaluating alternative adaptation
measures were validated to a large degree by the IPCC Working Group III report (Bruce,
et.al.1996) on economic and social dimensions of climate change. Neither IPCC guidelines nor
general environmental impact procedures possess comparable prescriptive decision criteria. The
paper concludes with guidance to planners as to: (1) climate-related factors that are of concern
and should be monitored; (2) conditions under which climate change should receive particular
attention; and (3) adaptation opportunities.
61
Appendix B
Expert Opinion, Subjective Probability and Sensitivity Analysis in Related
Literatures
This section provides a brief discussion of a limited amount of literature on two topics of interest
to the Corps for the current research topic. These are expert opinion and subjective probability.
Both of these topics appear in the scenario planning literature. Each has spawned its own
literature. It is beyond the scope of this task order to consider these bodies of work at length but
it is important to consider them, however briefly.
This section considers some early lessons in the Aerospace and Intelligence fields. These were
chosen because as government agencies involved in high stakes outcomes in the presence of
great uncertainties they seemed to have something in common with some of the Corps’ own
decision making situations. Indeed there seem to be some valuable lessons to be learned from
these two fields.
The section next turns to the use of subjective probability in probabilistic risk analysis in
government applied policy or planning contexts. The section concludes with the briefest
consideration of the use of subjective probabilities in policy making. The main points there
being that probabilistic forecasting is better than deterministic forecasting when there are
substantial uncertainties; and, decision makers need to be aware of the significance of the
uncertainties behind the deterministic estimates that are often treated as certain.
Aerospace Sector
The aerospace sector has an extensive and now very public history of using expert opinion to
assess safety. The history of this agency contains salient lessons for the Corps or any public
agency that is required to make significant decisions under uncertainty. The basic lesson is that
good techniques in estimating the probabilities of unobserved events is important and deserves
careful attention from the agency’s highest management down throughout the agency.
NASA managers needed to assess the risks associated with rare or unobserved catastrophic
events to protect its astronauts and to respond to politicians’ concerns. Estimating the
likelihoods of events that could occur but that have never been observed via traditional statistical
methods was clearly not possible. Problems associated with estimating such likelihoods were
dramatically brought out on January 28, 1986, with the tragic loss of the Challenger space shuttle
and its crew.
A review of a National Aeronautics and Space Administration (NASA) sponsored estimate of
shuttle failure modes and failure probabilities (Colglazier and Weatherwax, 1986) indicated an
estimate of the solid rocket booster failure probability per launch, based on subjective
probabilities and operating experience, was roughly 1 in 35. This estimate was rejected by
NASA management, which relied on its own “engineering judgment,” and used a figure of 1 in
62
100,000. This, in retrospect, clearly indicates a significant danger in ignoring good technique in
making decisions under uncertainty. An excerpt from the Colglazier and Weatherwax article
explained:
‘We estimated in 1983 that the probability of a solid rocket booster (SRB) failure
destroying the shuttle was roughly 1 in 35 based on prior experience with this
technology. . . Our estimates of SRB failure were based on a Bayesian analysis
utilizing the prior experience of 32 confirmed failures from 1902 launches of
various solid rocket motors. We also found that failure probabilities for other
accident modes were likely to have been underestimated by as much as a factor of
1000. . . NASA had decided to rely upon its engineering judgment and to use 1 in
100,000 as the SRB failure probability estimate for nuclear risk assessments. We
have recently reviewed the critiques and stand by our original conclusions. . . We
believe that in formulating space policy, as well as in assessing the risk of
carrying RTGs on the shuttle, the prudent approach is to rely upon conservative
failure estimates based upon prior experience and probabilistic analysis.’
Formal risk assessment is a useful decision support tool for dealing with uncertain situations.
NASA developed an interest in the risk assessment methodology after the fire on Apollo flight
AS-204 on January 27, 1967, killed three astronauts. Before the Apollo fire NASA relied on its
contractors to apply “good engineering practices" to provide quality assurance and quality
control. This was an agency that needed something that would both serve their decision making
needs and stand up to the withering scrutiny of politics and the public better than good
engineering practice did.
The Space Shuttle Task Group, formed April 5, 1969, developed "suggested criteria" for
evaluating the safety policy of the shuttle program. The probability of mission completion was
to be at least 95% and the probability of death or injury per mission was not to exceed 1%. These
numerical safety goals were not adopted in the subsequent shuttle program (Wiggins, 1985). This
perhaps marked another unfortunate instance of ignoring good scientific technique to quantify
key uncertainties.
Following a lead from the military, NASA adopted what they called risk assessment matrix
tables to "quantify and prioritize" risks. An example follows. Risks are presumed here to have a
consequence and a likelihood. The first table shows the consequences considered by NASA.
The second table adds a frequency of occurrence dimension to the consequence to produce a
matrix in which the frequency-consequence pairs receive a numerical index. Higher numbers are
lesser risks. The third table “judges” the risks.
Hazard Severity Categories
Description
Catastrophic
Critical
Marginal
Category Mishap Definition
I
Death or system loss
II
Severe injury, severe occupational illness, or major system
damage
III
Minor injury, minor occupational illness, or minor system
63
Negligible
IV
damage
Less than minor injury, occupational illness, or system
damage
Hazard Risk Management Matrix
Hazard Categories
Frequency of
Occurrence
(A) Frequent
(B) Probable
(C) Occasional
(D) Remote
(E) Improbable
I
Catastrophic
1
2
4
8
12
II
Critical
3
5
6
10
15
III
Marginal
7
9
11
14
17
IV
Negligible
13
16
18
19
20
Suggested Criteria
Hazard Risk Index
1-5
Unacceptable
6-9
Undesirable (project management decision required)
10-17
Acceptable with review by project management
18-20
Acceptable without review
Table 6.1.1 NASA Risk Assessment Matrix Tables a, b, c
The published reason for using the index was that the low numerical assessments of accident
probability produced by quantitative risk estimates do not guarantee safety. A report describing
the NASA safety program described the problem like this: ". . . the problem with quantifying risk
assessment is that when managers are given numbers, the numbers are treated as absolute
judgments, regardless of warnings against doing so. These numbers are then taken as fact,
instead of what they really are: subjective
Lessons from Aerospace Experience
evaluations of hazard level and probability"
(Wiggins, 1985).

Use good science and good technique
to estimate the probability of nonobserved events.
Quantitative estimates of likelihoods
can be misused, especially if the
limitations on their use are not
understood and appreciated.
Qualitative estimates of risks and their
likelihoods are more useful than no
estimates.
Self-serving estimates of likelihoods
are the worst of all options.
This published explanation seems at odds with the
practice according to Colglazier and Weatherwax

who suggest NASA managers did not treat
quantitative risk assessments as absolute numbers,
instead, they ignored them and relied on their own

judgment. In hindsight this self-serving decision
met with tragedy. There are persistent rumors in
the aerospace world that the primary motive for

abandoning quantitative risk assessment was not
distrust in overoptimistic reliability assessments so
much as the fact that the estimates of catastrophic failure probabilities were so high they would
have threatened the political viability of the entire space program. For example, a risk assessment
64
Comment [CY1]: This is an argument
against quantifying the probabilities of
scenarios occurring!
Comment [CY2]: But using
quantitative estimates of experts is better
than using one’s own self-serving
judgments!
on the likelihood of a successful manned moon landing done by General Electric estimated the
chance of success was "less than 5%." When the NASA administrator was presented with the
results, he "felt that the numbers could do irreparable harm, and disbanded the effort" (Bell and
Esch, 1989).
Military Intelligence
There is insufficient evidence in the available literature to make any truly informed commentary
on the way the intelligence community deals with uncertainty. However, the Defense
Intelligence Agency/Directorate of Estimates (DIA/DE), did make the results of a report
available in 1980 (Morris and D'Amore, 1980). Morris and D’Amore suggest an intelligence
service actively concerned with problems of dealing with uncertainty at a high level of
mathematical sophistication.
A major focus of this work was predicting future force levels for the USSR. The failure to have
predicted the fall of the Shah of Iran also appears as a motive for this report. The strategic
planning of the intelligence community depends on the intentions and capabilities of an enemy.
This differs some from the kinds of strategic planning done by the Corps. However, the highly
uncertain motives and intentions of an enemy have easy corollaries in highly uncertain motives
and corollaries of the markets and global politics that can affect navigation projects as well as the
highly uncertain structure and function of ecosystems.
The data upon which intelligence assessments were based were often, if not usually, of dubious
reliability. The so-called expert sources of these data included the testimony of defectors, reports
from informants, and interpretation of high-altitude photographs. A significant issue for the
analysts in these situations was how to convey estimator's uncertainty to the "consumers" of
intelligence reports so they could take this into account in making decisions. This problem
remains critically important in the Corps’ planning process. How can planners convey their
degree of confidence in the uncertain forecasts and judgments they must rely upon in the
planning process? How can the uncertainty in a without or with project condition best be
portrayed? How can the uncertainty in plan effects derived from the comparison of uncertain
without and with conditions be conveyed? There may be lesson to be learned from the
intelligence community.
Intelligence estimates in the 1960s contained phrases like "it is likely that," "it is unlikely that,"
and “it is probable that.” The Directorate of Estimates introduced more precise forms for
expressing uncertainty in 1976. For example, terms like "there is a 60% probability that,"
replaced the text phrases of the sixties. These estimates were sometimes accompanied by a
colored sheet of paper on which was printed:
Numeric forms are used to convey to the reader this degree of probability more
precisely than is possible in the traditional verbal form. Our confidence in the
supporting evidence is taken into account in making these quantifications. . . . All
efforts at quantifying estimates are highly subjective, however, and should be
treated with reserve." ( Morris and D'Amore, 1980, pp. 2-3)
65
Qualitative expressions were still given alongside these numeric estimates. Some years later, the
practice of using a "high," "low," and "best" estimate was introduced. The high and low values
were chosen such that there would be a 75% probability the true value falls between the high and
low values. This practice was proposed for use in the interval estimation cost estimating
approach (Yoe, 2000). Intelligence analysts were not satisfied with this system for two principal
reasons. First, the numbers were indeed "highly subjective," and there was no clearly defined
method for obtaining them. Second, the numbers were generally ignored by the intelligence
consumers. Consumers tend to take the best estimates (or sometimes the highest values) as if it
were certain, and disregard the uncertainty attached to them.
The Directorate of Estimates’ response to these weaknesses in their then existing methods was to
sponsor research for improving the quality and usefulness of numerical assessments of
uncertainty. The resulting program was, at the time, considered better than any other method
described in the public literature in both scale and sophistication. The communication and
evaluation of uncertainty assessments, as envisioned in this program, are summarized below for
their potential interest to the Corps’ planning process.
Communicating uncertainty to “consumers” so they will make proper use of it, was and remains
a formidable challenge. Several systems were implemented and subsequently discarded. An
early system, combining reliability and accuracy ratings, is shown in the table below.
“Information” would be assigned a rating such as C2 or A5. The problem with this technique
was that intelligence officers placed too much emphasis on the assessed accuracy of reports and
not enough on the reliability of the source. In addition, there was a wide disparity in interpreting
the meanings of the qualitative ratings. (Samet, 1975)
Source Reliability
A: Completely reliable
B: Usually reliable
C: Fairly reliable
D: Not usually reliable
E: Unreliable
F: Reliability cannot be judged
Table 6.2.1 Reliability and Accuracy Ratings
Information Accuracy
1: Confirmed
2: Probably true
3: Possibly true
4: Doubtfully true
5: Improbable
6: Accuracy cannot be judged
The Directorate of Estimates also made use of Kent charts. These charts (not all of them agree
on the assignments) provide a quantitative interpretation of natural language expressions of
uncertainty. Kent charts define the terms most frequently used to describe the range of
likelihood in the key judgment of intelligence (Morris and D'Amore, 1980, p. 5-21)
66
Order of
Likelihood
Near certainty
Probable
Even chance
Improbable
Near impossibility
Synonyms
Virtually (almost) certain, we are convinced,
highly probable, highly likely
Likely
We believe
We estimate
Chances are good
It is probable that
Chances are slightly better than even
Chances are about even
Chances are slightly less than even
Probably not
Unlikely
We believe . . . not
Almost impossible
Only a slight chance
Highly doubtful
Chances Percent
in 10
9
99
90
8
60
7
6
5
4
40
3
2
10
1
1
Tab
le
6.2.
2A
Ken
t
Cha
rt
for
Esti
mat
ing
Ter
ms
and
Deg
rees
of
Pro
bab
ility
The Kent charts were abandoned in favor of direct numeric estimations of probability. Most of
these numeric estimates were provided by experts using subjective probability estimates. There
remains a substantial issue with the validity of numerical or calibration of these subjective
probabilities. A subjective probability expert is said to be "well calibrated" if statements with an
assessed probability of X% turn out to be true X%. In other words, if an expert says it rains 60%
of the time and it does rain 60% of the time, the expert is well calibrated. If it rains 30% of the
time he overestimates and if it rains 80% of the time he underestimates. Calibration of expert
opinion has produced its own voluminous literature.
The Directorate of Estimates proposed a three-pronged effort to improve calibration. It included:
debiasing; use of proper scoring rules; and, use of feedback and systematic evaluation. The
psychometric literature identifies a number of biases
Lessons from Intelligence
that tend to impair experts’ opinions. And procedures
for eliciting and calibrating subjective probability

The intelligence community was/is
estimates have been developed (Morgan and Henrion,
confronted with uncertainty on a large
scale.
1988).




They have been forced to confront
virtually all the problems known from
the literature.
The intelligence community
literature on this topic is understandably
not very open.
The response of the intelligence
community has been to try to improve
the numerical estimations of uncertainty
and to support the consumers in making
the best possible use of these numerical
assessments.
It would be very useful to have
access to more recent reports on the
success of such initiatives.
These procedures are all quite standard and are well
described in the literature. Many universities and
some consultants offer decision support centers that
rely on computer-supported interactive elicitation
processes that incorporate one or more of techniques
described in the literature. Debiasing efforts are
67
directed toward the process of eliciting subjective probabilities.
The intelligence community has made use of scoring rules for their probability assessors.
A scoring rule scores or rates a probability assessor on the basis of whether a “predicted”
outcome is later observed. The most intuitively appealing scoring rules are not truly useful. The
"hit or miss" rule simply counts the number of projections that have proven true and divides this
by the total number of projections. This rule leads to evaluations like "70% of the projections
from estimator A have come out." A rule like this encourages estimators to "hedge," that is, to
give projections that are more cautious than they really think appropriate. A second flawed but
appealing rule is the "direct rule." It rewards a probability assessment like "A will happen with
probability 60%" by giving the score 60 if A happens and the score 40 otherwise. This rule
encourages overconfident assessments. Scoring rules are difficult to apply in intelligence
estimates because many estimates concern events 10 or 20 years into the future, much as some
Corps forecasts might.
The last initiative described by Morris and D'Amore for improving the quality of probability
assessments concerned an elaborate computerized data bank of intelligence estimates called the
"institutional memory" (IM). The purposes of IM were: to support the elicitation process; to
provide feedback on past performance; and, to inform the consumer of the quality of the
Directorate of Estimates’ estimates.
In 1980 the IM may have been the most advanced system of its kind for handling uncertainty. It
is not possible to know without a better understanding of the techniques in general usage at that
time. It is not likely that such sophisticated analysis would well suit the Corps’ planning process,
however.
Probabilistic Risk Analysis
Probabilistic risk analysis is a science-based decision support framework that first introduced
subjective probabilities on a large scale. Many of the advances in expert elicitation, especially in
the area of probability estimation have owed some debt to the practice of risk analysis. The
notion of probabilistic risk assessment is not new. Its practice as a formal decision making
paradigm is a rather recent development. More or less explicit risk assessment can be found in
the writings of Arnobius the Elder in the fourth century (Covello and Mumpower, 1985).
In the United States radiation biology was the birthplace of modern probabilistic risk analysis.
The Atomic Energy Commission, now the Nuclear Regulatory Commission (NRC), applied risk
analysis concepts to the assessment of the "maximum credible accident." The best known
example of such a study is WASH-740 released in 1957. It focused on three scenarios of
radioactive releases from a 200 Megawatt-electric power nuclear power plant operating 30 miles
from a large population center (Cooke, 1992).
68
Comment [CY3]: Note the practice of
scoring/rewarding the estimator.
The report essentially concluded that the probability of such an event cannot be known. As
larger reactors were being proposed, the desire to quantify and evaluate the associated risks led
to the introduction of probabilistic risk analysis (PRA).
The first full-scale probabilistic risk assessment was undertaken in the Reactor Safety Study
called the Reactor Safety Study: An Assessment of Accident Risks in U.S. Commercial Nuclear
Power Plants, Nuclear Regulatory Commission, NUREG 75/014, Washington, D.C., October
1975.
With respect to the use of subjective probabilities in risk assessment a review of WASH-1400
said in part:
The Reactor Safety Study (RSS) had to use subjective probabilities in many
places. Without these, RSS could draw no quantitative conclusions regarding
failure probabilities at all. The question is raised whether, since subjective
probabilities are just someone's opinion, this has a substantial impact on the
validity of the RSS conclusions.
It is our view that the use of subjective probabilities is necessary and appropriate, and provides a
reasonable input to the RSS probability calculations. But their use must be clearly identified, and
their limits of validity must be defined. (Lewis et al., 1979, p. 8)
The National Research Council distanced itself from the results of the Reactor Safety Study in
January 1979 when it said:
In particular, in light of the Review Group conclusions on accident probabilities, the
Commission does not regard as reliable the Reactor Safety Study's numerical estimate of the
overall risk of reactor accident." (U.S. NRC, 1979)
The future of probabilistic risk analysis improved significantly after this initial foray into the
field as the Food and Drug Administration, the U.S. Department of Agriculture,the Occupational
Safety and Hazard Administration, and the U.S. Environmental Protection Agency began to
make regular and continuous use of quantitative and qualitative risk assessment techniques to
support their decision-making processes. During the Carter Administration the so-called
“Benzene case” made it to the Supreme Court. Justice Stevens, arguing for the majority, said
risk assessment is feasible and OSHA must do one before taking rule-making action to reduce or
eliminate the risk associated with benzene. Shortly after, the “Cotton-dust” case gave the
Supreme Court the opportunity to reaffirm the Stevens decision.
As a common practice among U.S. federal government agencies that has been legitimized by the
Supreme Court of the United States, probabilistic risk assessment is a well-established tool for
public decision making.
69
Returning to the matter of subjective probabilities, the methodology of the Zion Probabilistic
Safety Study has been well documented in the Journal of Risk Analysis (Kaplan and Garrick,
1981) The first issue of this journal defines "probability" as follows:
". . . 'probability' as we shall use it is a numerical measure of a state of knowledge,
a degree of belief, a state of confidence." (Kaplan and Garrick, 1981, p. 17 )
Expert elicitation of subjective probabilities and less structured approaches have since been used
in several large studies. These include, for example, the risk study of the fast breeder reactor at
Kalkar, Germany (Hofer, Javeri, and Loffler, 1985); studies of seismic risk by Okrent (1975) and
Bernreuter, et al, (1984); a study of fire hazards in nuclear power plants (Sui and Apostolakis,
1985); and numerous food safety risk assessments including but not limited to the more than 100
examples that are found at the Food Safety Risk Analysis Clearinghouse
(http://www.foodriskclearinghouse.umd.edu/risk_assessments.cfm).
Two sources that are most useful to anyone who wants to access the early literature on this topic
include the Handbook of Human Reliability (Swain and Guttmann, 1983) and (Humphreys
(1988). Together they give a good review of the literature and methods. Volume 93 (1986) of
Nuclear Engineering and Design is entirely devoted to the role of data and judgment in risk
analysis.
Four practical issues emerged early in the subjective probability literature and they remain
germane to this day. They include the spread of expert opinion, the dependency between
experts, the reproducibility of the results, and finally the calibration of the results. Cooke’s
(1992) excellent review of the topic, which provides the structure for much of this section, is
used to introduce these issues.
Subjective Data: Spread
A nuclear reactor core meltdown is a good example of a quantity that can’t be computed without
substantial use of subjective probabilities. The Reactor Safety Study estimated the value to be
4.7 x 10-7. The Okrent study (1975) estimated the same value to be probability as 8 x 10-5.
Another study (Lee, Okrent, and Apostolakis, 1979) estimated a probability of 1.77 x 10-4 per
reactor year for reactors with no design errors and 2.32 x 10-3 per reactor year for systems having
a "maximal number of design errors and reduced original safety factors." These subjective
probability estimates span four orders of magnitude, illustrating a common concern about
subjective probability estimates. Expert probability assessments in risk analysis typically show
extremely wide spreads (Cooke, 1992).
Subjective Data: Dependence
A study of the Reactor Safety Study (Cooke, 1992) shows that the estimates experts offered for
probabilities of various events were not independent. Experts pessimistic with respect to one
component of a reactor tended to be pessimistic about other components as well. For example,
an expert whose estimate for a given component lies above the median estimate often tends to be
above the median for other components as well. An expert is "rank independent" if his responses
70
Comment [CY4]: This could be
useful.
show no tendency to cluster either toward optimism nor pessimism. Other studies (Shooman
and Sinkar, 1977) have confirmed the high degree of clustering in the probability estimates of
experts.
Subjective Data: Reproducibility
There have been some studies that attempt to gage the extent to which risk assessment results are
reproducible. And although they do not focus solely on subjective probability estimates, they are
primary quantities of interest in explaining the different results experts obtained in these studies.
These studies are called bench mark studies. The Joint Research Centre at Ispra, Italy
(Amendola, 1986) undertook such a study on the new Paluel Auxiliary feedwater system. Ten
teams from different European countries were formed to independently estimate the probability
that this system would not fulfill its design requirements. The study design was quite unique for
this field and is worth some description. In stage one, the teams carried out their first
probabilistic analysis. This was the "blind evaluation." Without discussing the results
beforehand, the teams were then brought together to compare their analyses qualitatively. Stage
two followed this this comparison. In stage two the teams performed an intermediate "fault tree
analysis." The teams were unable to agree on a common fault tree model and the spread in
results after this stage was rather large. In stage three the researchers separated the effects of
different fault tree models from the effects of different failure data by providing the teams with a
common fault tree. The goal of stage four was to determine whether different methods of
calculation played a significant role in the results observed. The results show a broad dispersion
of outcomes.
Brune, et al (1983) conducted another bench mark study concerning human reliability. Human
error is implicated in anywhere from 30% to 80% of serious accidents with sophisticated
technical systems (Levine and Rasmussen, 1984). A study was designed to determine to what
extent techniques for quantifying human error probabilities in risk analysis developed at the
Sandia National Laboratories would lead to similar results when applied by different human
reliability experts. Qualified persons using the same techniques failed to obtain an answer within
the prescribed confidence intervals in about 38% of the experts' responses. The differences in
the experts’ answers were not caused by different values for individual failure probabilities.
These were given in the exercise. The differences were caused by the fact that the experts
analyzed the human reliability problems differently. For example, they identified different tasks
that might be performed incorrectly. The evidence suggests that reproducibility of risk
assessment results is not easy to achieve. The relevance of this for the Corps’ planning process
would seem to be that subjective processes are fundamentally prone to human error. And
planning is always going to rely on some subjective processes.
Subjective Data: Calibration
Having experts agree is important but it is less important than having experts who are accurate
and whose assessments are good. This is the calibration issues, which is concerned with the
extent to which the assessed probabilities agree with observed relative frequencies.
71
It is always difficult and often not even possible to calibrate subjective probability estimates
when the events whose probabilities are assessed are rare. Ordinary events are not often the
subject of risk analysis. It is the rare events that most often and most controversially capture the
risk assessors’ attentions. Because rare events can often be deconstructed to a series of
component or ingredient events it is sometimes possible to calibrate assessments for ingredient
events.
The literature on the calibration of these rare event probabilities is sparse. There has been some
effort to calibrate some of the subjective probability assessments in the Reactor Safety Study
(Minarick & Kukielka, 1982; Cottrell & Minarick, 1984). These studies suggested two sorts of
biases in the Reactor Safety Study estimates. First, there is a "location" or "first moment" bias.
This caused causing the estimate to be too low in the RSS case, but it could as easily cause an
overestimate. Second, there is a "scale" or "overconfidence" bias, causing the confidence bounds
to be too narrow. To compound this entire issue there is little documented evidence to suggest
how confident the experts were of their flawed estimates.
Mosleh, Bier, and Apostolakis (1988) propose the use of a degree of confidence indicated by
"range factors." To obtain the range factor for a quantity you first find the ratio of the 95th and
5th percentiles of the distribution for this quantity and then take the square root of that quantity.
Thus, if a range factor of 3.2 is reported the 95th percentile value is a factor of 10 larger than the
5th percentile value. Cooke (1992) applied this concept in a study and concluded the experts’
assessments reflect significant overconfidence.
There is a great deal of “gray literature” on the matter of subjective probability assessment. This
field is dominated by a great deal of applied work and only a fraction of it ever finds its way into
print. This reviewer has participated in the conduct of over a dozen such assessments all of
which resulted in proprietary reports. A review of more than 3500 abstracts on judgment and
reasoning by Christensen-Szalanski & Beach (1984) says that poor performances with expert
opinion were reported more than good performances. It is unclear whether that is because poor
performances outnumber good performances or simply whether “bad news” makes for better
articles than good news.
The conclusions from this limited review of the literature related to the use of subjective
probabilities in risk analysis shows that expert opinions in probabilistic risk analysis have
exhibited extreme spreads, have shown clustering, and have led to results with low
reproducibility and poor calibration. There have been significant gains and advances in
methodological approaches to the use of expert opinion. A review of that literature is beyond the
scope of this task. But, if the Corps relies on methodologies dependent on subjective probability
estimates in particular, or expert opinion in general, it is best to make use of one or more of the
more recent protocols and methodologies.
The Internet is one of the best sources of information about the most recent advances in expert
knowledge elicitation, which is a much broader and more modern field of inquiry. Many
university courses make notes and unpublished papers available to their students and any other
members of the public with an interest in this area.
72
Policy Analysis
Continuing with Cooke’s useful structure, "policy analysis" is a catchall for those things that do
not fit into the areas discussed above. It would seem that the Corps’ planning process as
currently practiced would fall more comfortably into this category than into aerospace,
intelligence, or probabilistic risk analysis. Morgan et al (1984),. Morgan and Henrion (1988) and
Merkhofer and Keeney ( 1987) provide good examples of policy analysis that makes explicit use
of subjective probability assessments. In general policy tends to rely far more on the use of
deterministic methods of expert opinion. This brief review highlights some of the differences in
probabilistic and deterministic methods of using expert opinion. The Corps has tended more
toward the deterministic methods than the probabilistic methods when using experts in its
planning process, hence some review of the differences may be useful.
Many planning uncertainties revolve around the planners’ ability to forecast the future
(scenarios) or the future value of some variable. Planners must predict values, e.g. fleet
composition, commodities, flood damages, flows, that will eventually become known. Once
known they may lend themselves very well to objective probabilistic analysis. Before they are
known, however, subjective probabilistic assessment is often the best that can be done. And
although the Corps is quite on the leading edge in the use of probabilistic methods for some of
these quantities, in others there has been very little use of probabilistic assessment.
The Corps’ early efforts to make commodity forecasts for specific harbors employed various
mathematical models in economic forecasting, sometimes in conjunction with expert forecasts.
At times the experts have had impressive credential and at other times the subjective judgments
were being offered by Corps’ analysts with more planning experience than specific expertise in
the field.
Experts' forecasts combined with model outputs have often been used to produce estimates of
critical variables based on past performance and other factors. These methods are deterministic
when they make no attempt to assess or communicate the uncertainty attending the estimate of
this critical variable. Decision makers are given no information to help them gage the extent to
which their decisions should or should not hedge for the uncertainty inherent in the planning
studies upon which they base their decisions. These naively deterministic estimates of future
values of critical variables are even more primitive than the qualitative attempts, like the Kent
chart above, to represent uncertainty. In practice this often means that the decision maker treats
the forecast values as certain. This can be problematic when others, especially opponents of a
specific course of action, recognize the uncertainty in these forecasts.
Granger (1980) said there was no practical way to evaluate a forecasting technique because some
values are more easily forecast than others, so the technique is not independent of the nature of
the task. For example, it is easy to predict how old someone will be in ten years. It is not as easy
to predict the number of heads in ten tosses of a coin. How would one compare and evaluate the
techniques used for such different tasks?
73
Comment [CY5]: This may be more
in tune with the COE
Cooke (1992) suggests this is perhaps the single most important difference between probabilistic
and deterministic methods, i.e., probabilistic forecasters can be evaluated independently of the
“forecastability” of the things they are forecasting, while deterministic forecasters cannot.
Granger (1980) and Beaver (1981) were two early studies to suggest that the consensus of
several experts is generally a better forecaster than any of the experts individually. Beaver
illustrated this with an example of forecasting winners of football games. He reviewed the
published predictions of the sports staff of the Chicago Daily News from 1966 to 1968. The
study found the consensus prediction outperformed all the other predictors, for the entire period.
Building on Beaver's idea, the advice of several experts is to be preferred to the advice of any
one expert.
A potential problem with Beaver’s argument is that in the real world, forecasters are not always
unbiased. Experience shows they are not "as likely to overestimate as to underestimate" a value.
Worse, it is possible to have all experts with identical biases if the selection pool itself is biased.
Thus, a team of Corps experts, or a team of industry experts, or a team of environmental interest
group experts may be biased. And in this case a team of biased experts may not perform as well
as the single best expert. This suggests that a diverse group of bona fide experts may be
desirable. When such biases are absent there are techniques that may work very well.
A much more recent example of this principle in action was advanced by the Defense Advanced
Research Projects Agency (DARPA) in a program, called the Futures Markets Applied to
Prediction (FutureMAP). This program would have involved many investors betting small
amounts of money that a particular event -- a terrorist attack or assassination -- would happen. A
large number of independent and unbiased “experts” would produce a better composite forecast
than any one or two experts could have produced on their own. A story posted by CNN on
Wednesday, July 30, 2003 (http://www.cnn.com/2003/ALLPOLITICS/07/29/terror.market/)
reported that FutureMAP had part of the Total Information Awareness program. DARPA
acknowledged, under considerable political pressure, that the program faced "a number of major
technical challenges and uncertainties. Chief among these are: Can the market survive and will
people continue to participate when U.S. authorities use it to prevent terrorist attacks? Can
futures markets be manipulated by adversaries?" So the debate over who is an expert and how
many experts does it take to make a good consensus forecast continues to the present moment.
Sensitivity Analysis
Scenario analysis is not sensitivity analysis. Sensitivity analysis is changing variables, often one
at a time, all other things equal. There is no one at a time variation in scenario analysis and there
is no attempt to identify an average case. Sensitivity analysis often runs off the average case to
see how it is affected by a change in a variable here or there. Nonetheless, sensitivity analysis is
a common way of addressing uncertainty. This section briefly reviews a few articles dealing
with sensitivity analysis.
‘‘In a sensitivity analysis, one systematically and comprehensively tests to see how changes in
the parameters of the model affect the model’s output’’ (Starfield & Bleloch, 1991. The goal is
to learn which of the model parameters exert significant influence on the output variables and
74
Comment [CY6]: So our
recomemdnations should be to engage a
team to develop scenarios and their
likelihoods.
Comment [CY7]: This is a strong
argument in favor of a diverse group of
forecasters and a group that is likely to be
unbiased, i.e. experts.
which are inconsequential. In order to understand the decision support models in use, analysts
need to know, for example, if small increments in any parameters produce unexpectedly large
alterations in results. Most importantly, rcommendations based upon a model without explicit
sensitivity analysis lack foundation (Beres & Hawkins, 2001).
Despite the growing interest and acknowledged importance of sensitivity analysis, there is ‘‘a
dearth of information’’ (Henderson-Sellers & Henderson-Sellers, 1996) on how to do sensitivity
analysis. ‘‘There is no single, universally accepted procedure for the sensitivity analysis of
stochastic models’’ (McCarthy et al., 1995). Perhaps the most common approach to sensitivity
analysis is to explore the effects of changing parameters, one at a time, on a target output
variable (Henderson-Sellers & Henderson-Sellers, 1996; Swartzman & Kaluzny, 1987). This
ceteris paribus procedure holds all variables constant at a mean or representative value while
each parameter is allowed to change to determine its’ effect in isolation from the possible effects
of other variables. In an interdependent system this approach frequently can result in serious
errors.
Daniel (1973) suggested that even when the sensitivity of all individual parameters is
investigated, the one-at-a-time (OAAT) method cannot uncover potentially important
interactions between two parameters.
Experimental designs for sensitivity analysis of more than one parameter at a time are needed.
Analysts are often interested in the effects of individual parameters, the 2-way and multi-way
‘‘interactions’’ of pairs and more of parameters (Swartzman & Kaluzny, 1987). Some such
methods have been devised, but a review of that literature is beyond the scope of this task order.
One such design includes the complete factorial design, which consists of all possible
combinations of selected high and low values for the parameters.
Beres and Hawkins describe the Plackett–Burman Sensitivity Analysis (PBSA) design that
provides an alternative that is both convenient and informative. They offer a rationale for using
this approach based on the following points:
 PBSA is not a OAAT method
 PBSA finds 2-way interactions
 PBSA is not restricted to any particular type of model
 PBSA is prescriptive, using pre-determined designs
 PBSA is efficient in terms of number of scenarios needed
 PBSA designs for up to 100 parameters are readily available
 PBSA rankings are easy to compute
 PBSA works with categorical as well as numerical parameters
 PBSA does not require parameters to be considered over identical intervals
 PBSA is statistically sound
Having noted these limitations of sensitivity analysis, there is some potential for using a
sensitivity analysis where the effects of different scenarios rather than different parameters are
examined for their impact on planning outcomes of interest.
75
Bibliography
Ackoff R. L. Creating the Corporate Future. New York: Wiley, 1981.
Adya, M., Et Al. “An Application of Rule-Based Forecasting To A Situation Lacking Domain
Knowledge” International Journal of Forecasting, October 2000, Vol. 16, No. 4, Pp. 477-484(8)
Al-Jayyousi, O.R. “Scenarios for Public-Private Partnerships in Water Management: a Case
Study from Jordan,” International Journal of Water Resources Development, June 2003, Vol. 19,
No. 2, Pp. 185-201(17)
Amendola, A., "Systems Reliability Benchmark Exercise Parts I and II," EUR-10696, EN/I,
1986.
Anderson, J., Clement, J., and Crowder, L.V.1999, “Pluralism in Sustainable Forestry and Rural
Development-An Overview of Concepts, Approaches and Future Steps,” in Food and
Agriculture Organization, Pluralism and Sustainable Forestry and Rural Development. In:
Proceedings of an International Workshop, December 1997, Pp. 9± 12
Ataie-Ashtiani, B., Et Al. “Numerical Modelling of Two-Phase Flow in a Geocentrifuge,”
Environmental Modelling and Software, April 2003, Vol. 18, No. 3, Pp. 231-241(11)
Athanassopoulos, A.D., Lambroukos, N., and Seiford, L. “Data Envelopment Scenario
Analysis for Setting Targets to Electricity Generating Plants,” European Journal of Operational
Research, 6 June 1999, Vol. 115, No. 3, Pp. 413-428(16)
Bailenson, J.N., Et Al. A Bird's Eye View: Biological Categorization and Reasoning Within and
Across Cultures. Cognition, May 2002, Vol. 84, No. 1, Pp. 1-53(53)
Baker, S., Ponniah, D., and Smith, S. “Techniques for the Analysis of Risks in Major Projects,”
Journal of the Operational Research Society, June 1998, Vol. 49, No. 6, Pp. 567-572(6)
Bardram, J., “Scenario-Based Design of Cooperative Systems,” Group Decision and Negotiation,
May 2000, Vol. 9, No. 3, Pp. 237-250(14)
Barnes, Jr., J.H. “Cognitive Biases and Their Impact On Strategic Planning,” Strategic
Managment. Journal, 1984, Vol. 5, No.2, Pp.129±137
Barrett, J. “Component Ecological Footprint: Developing Sustainable Scenarios,” Impact
Assessment and Project Appraisal, 1 June 2001, Vol. 19, No. 2, Pp. 107-118(12)
Baumert, L., Golomb, S.W., and Hall, M. “Discovery of an Hadamard Matrix of Order 92,”
American Mathematical Society Bulletin 1962, No. 68, Pp. 237–238
76
Baumont, G., Et Al. “Quantifying Human and Organizational Factors in Accident Management
Using Decision Trees: the Horaam Method”. Reliability Engineering and System Safety,
November 2000, Vol. 70, No. 2, Pp. 113-124(12)
Bazerman, M.H., and Neale, M.A., “Negotiating Rationally,” the Free Press, 1992
Beaver, W. M. Financial Reporting: An Accounting Revolution. Prentice Hall, Englewood Cliffs,
N. J., 1981.
Becker, H.S., “Scenarios: a Tool of Growing Importance to Policy Analysts in Government and
Industry,” Technology Forecasting and Social Change, 1983, Vol. 23, No.2, Pp. 95±120
Beliakov G., and Warren J. “Fuzzy Logic For Decision Support in Chronic Care”
Artificial Intelligence in Medicine, January 2001, Vol. 21, No. 1, Pp. 209-213(5)
Bell, T. E., and Esch, K., "The Space Shuttle: A Case of Subjective Engineering," IEEE
Spectrum, pp. 42-46, June 1989.
Belzer, R.B. “Getting Beyond 'Grin and Bear It' in the Practice of Risk Management”
Reliability Engineering and System Safety, May 2001, Vol. 72, No. 2, Pp. 137-148(12)
Bennett, R.L., Et Al. “Fabry Disease in Genetic Counseling Practice: Recommendations of the
National Society of Genetic Counselors”. Journal of Genetic Counseling, April 2002, Vol. 11,
No. 2, Pp. 121-146(26)
Beres, D.L. “A Methodological Study of Modeling for California Condors,” PhD Dissertation,
University of Minnesota
Beres, D. L. and Hawkins, D. M. (2001), 'Plackett-Burman technique for sensitivity analysis of
many-parametered models', Ecological Modelling, 141, 171-183.
Beres, D.L., Clark, C., Swartzman, G. and Starfield, A.M. “Truth in Modeling,” Natural
Resource Modeling 2000, Vol. 14, No. 3, in Press
Bernreuter, D. L., Savy, J. B., Mensing, R. W., and Chung, D. H., "Seismic Hazard
Characterization of the Eastern United States: Methodology and Interim Results for Ten Sites,"
NUREG/CR-3756, 1984.
Bialasiewicz, L. “Another Europe: remembering Habsburg Galicja,”. Cultural Geographies, 1
January 2003, vol. 10, no. 1, pp. 21-44(24)
Bilkle O. and Knolker, U. “The Installation of a Psychiatric Out-Patient Drug and Addiction
Service for Adolescents -- First Results on Clinical and Institutional Problems”
European Psychiatry, 1998, vol. 13, no. 1004, pp. 229s-229s(1)
77
Blair, A.R., Et Al. “Forecasting the Resurgence of the US Economy in 2001: An Expert
Judgment Approach Socio-Economic Planning Sciences,” June 2002, Vol. 36, No. 2, Pp. 7791(15)
Blythe, M.J., and Young, R. “Scenario Analysis: a Tool for Making Better Decisions for the
Future” Evaluat. J. Australasia, 1994, Vol.6, No. 1, Pp. 1±17
Bocco, G., and Toledo, V.M. “Integrating Peasant Knowledge and Geographic Information
Systems: a Spatial Approach to Sustainable Agriculture,” Indig. Know. Develop. Monitor 1997,
Vol. 5, No. 2, Pp. 10-13
Bommer, J. J., Scott, S. G., and Sarma, S. K. “Hazard-Consistent Earthquake Scenarios Soil
Dynamics and Earthquake Engineering,” June 2000, Vol. 19, No. 4, Pp. 219-231(13)
Bood R., and Postma T. “Strategic Learning with Scenarios,”. European Management Journal,
December 1997, Vol. 15, No. 6, Pp. 633-647(15)
Borrini-Feyerabend, G. “Beyond Fences: Seeking Social Sustainability in Conservation, a
Resource Book,” 1997, Vol. 2
Bower, B.T., and Kerry, Turner R. “Characterizing and Analyzing Benefits from Integrated
Coastal Management (ICM) - Market and Intervention Failures,” Ocean and Coastal
Management, 1 January 1998, Vol. 38, No. 1, Pp. 41-66(26)
Box, G.E.P., and Draper, N.R. “Empirical Model-Building and Response Surfaces,” 1987, P. 669
Brauers, J., Weber, M. 1988. “A New Method of Scenario Analysis for Strategic Planning,”
Journal of Forecasting, 1988, Vol.7, No.1, Pp. 31-47
Briand, L.C., and Wust, J. “Integrating Scenario-Based and Measurement-Based Software
Product Assessment,” Journal of Systems and Software, 15 October 2001, Vol. 59, No. 1, Pp. 322(20)
Brown, R.A., Et Al. “Integrated Assessment of Hadley Centre (Hadcm2) Climate Change
Projections On Agricultural Productivity and Irrigation Water Supply in the Conterminous
United States - I. Climate Change Scenarios and Impacts On Irrigation Water Supply Simulated
With the Humus Model,” Agricultural and Forest Meteorology, June 30 2003, Vol. 117, No. 1,
Pp. 73-96(24)
Brown, S., Middleton D., Lightfoot G. “Performing the Past in Electronic Archives:
Interdependencies in the Discursive and Non-Discursive Ordering of Institutional
Rememberings”. Culture & Psychology, June 2001, vol. 7, no. 2, pp. 123-144(22)
Bruce, J.P., H. Lee and E.F. Haites (Eds). 1996. Climate Change 1995: Economic and Social
Dimensions of Climate Change. Contribution of Working Group III to the Second Assessment
of the Intergovernmental Panel on Climate Change. Cambridge University Press, United
Kingdom.
78
Brune, R., Weinstein, M. and Fitzwater, M., "Peer Review Study of the Draft Handbook for
Human Reliability Analysis with Emphasis on Nuclear Power Plant Applications," NUREG/CR1278, Human Performance Technologies, Inc., Thousand Oaks, Calif., 1983.
Bunn, D.W., and Salo, A.A. “Forecasting With Scenarios,” European J. Oper. Res. 1993, Vol.
68, No. 3, Pp. 291-303
Burgman M.A., Et Al. “A Method For Setting the Size of Plant Conservation Target Areas” A
Method For Setting the Size of Plant Conservation Target Areas. Conservation Biology, June
2001, Vol. 15, No. 3, Pp. 603-616(14)
Burgman, M.A., Ferson, S., and Akcakaya, H.R. “Risk Assessment in Conservation Biology,”
1993, P. 314
Carroll J. M. “Becoming Social: Expanding Scenario-Based Approaches in Hci,” Behaviour and
Information Technology, 1 July 1996, Vol. 15, No. 4, Pp. 266-275(10)
Carroll, J.M. “Five Reasons for Scenario-Based Design,” Interacting With Computers,
September 2000, Vol. 13, No. 1, Pp. 43-60(18)
Chandler J. & Cockle P. ( 1982). Techniques of Scenario Planning. London: McGraw-Hill.
Chandler, A.M. and Lam, N.T.K. “Scenario Predictions for Potential Near-Field and Far-Field
Earthquakes Affecting Hong Kong” Soil Dynamics and Earthquake Engineering, January 2002,
Vol. 22, No. 1, Pp. 29-46(18)
Chao, P.T., Hobbs, B.F., and Venkatesh, B.N. “Using Decision Analysis to Include Climate
Change in Water Resources Decision Making,” Climatic Change, September 1997, Vol. 37, No.
1, Pp. 177-202(26)
Chau, K.W., Albermani, F. “Expert System Application on Preliminary Design of Water
Retaining Structures”. Expert Systems With Applications, February 2002, Vol. 22, No. 2, Pp.
169-178(10)
Chen, F., Chu, H.-S, and Yeh, C. “Effects of Porosity Change of Gas Diffuser On Performance
of Proton Exchange Membrane Fuel Cell,” Journal of Power Sources, 15 September 2003, Vol.
123, No. 1, Pp. 1-9(9)
Choi, Y.S., Lee, S.H., and Lee, B.W. “Expert Judgment For Nuclear Energy”
Annals of Nuclear Energy, May 2000, Vol. 27, No. 7, Pp. 575-588(14)
Christensen-Szalanski, J. J. J., and Beach, L. R., "The Citation Bias: Fad and Fashion in the
Judgment and Decision Literature," American Psychologist, vol. 39, pp. 75-78, 1984.
79
Cionco, R.M., and Ellefsen, R. “High Resolution Urban Morphology Data for Urban Wind Flow
Modeling,” Atmospheric Environment, January 1998, Vol. 32, No. 1, Pp. 7-17(11)
Clemen, R.T., and Winkler R.L. “Combining Probability Distributions from Experts in Risk
Analysis” Risk Analysis, April 1999, Vol. 19, No. 2, Pp. 187-203(17)
Clemons E.K. “Using Scenario Analysis to Manage the Strategic Risks of Reengineering,” Long
Range Planning, December 1995, Vol. 28, No. 6, Pp. 123-123(1)
Coen F., Et Al. “Nitrogen Removal Upgrade of a Wastewater Treatment Plant Within Existing
Reactor Volumes: a Simulation Supported Scenario Analysis,” Water Science and Technology,
1996, Vol. 34, No. 3, Pp. 339-346(8)
Coffey J.W., and Hoffman R.R. “Knowledge modeling for the preservation of institutional
Memory” Journal of Knowledge Management, 9 July 2003, vol. 7, no. 3, pp. 38-52(15)
Cojazzi, G.Et Al. “Benchmark Exercise on Expert Judgment Techniques in Psa Level 2”
Nuclear Engineering and Design, November 2001, Vol. 209, No. 1, Pp. 211-221(11)
Coles S.; Rowley J. “Spreadsheet Modelling for Management Decision Making,”
Industrial Management & Data Systems, 1 July 1996, Vol. 96, No. 7, Pp. 17-23(7)
Colfer, C.J.P. “Who Counts Most in Sustainable Forest Management?” Cifor Working Paper,
1995, No.7
Colglazier, E. W., and Weatherwax, R. K., "Failure Estimates for the Space Shuttle," Abstracts
for Society for Risk Analysis, Annual Meeting 1986, Boston, Mass., p. 80, Nov. 9-12, 1986.
Cooke, R. 1992. Experts in Uncertainty: Opinion and Subjective Probability in Science. New
York: Oxford University Press.
Cosier R. A. ( 1981a). "Dialectical inquiry in strategic planning: A case of premature
acceptance?" Academy of Management Review, 6: 643-648.
Cosier R. A. ( 1981b). "Further thoughts on dialectical inquiry: A rejoiner to Mitroff and
Mason". Academy of Management Review, 6: 653-654.
Cottrell W., and Minarick, C., "Precursors to Potential Severe Core Damage Accidents: 19801982, a Status Report," NUREG/CR-3591, 1984.
Courtney, H. “Decision-Driven Scenarios for Assessing Four Levels of Uncertainty,” Strategy
and Leadership, 2003 Vol. 31, No. 1, Pp. 14 – 22
Covello, V. T., and Mumpower, J. "Risk Analysis and Risk Management: An Historical
Perspective," Risk Analysis, vol. 5, no. 2, pp. 103-120, 1985.
80
Crane, S.A. “Memory, Distortion, and History in the Museum”. History and Theory, December
1997, vol. 36, no. 4, pp. 44-63(20)
Croise, J., and Kaleris, V. “Estimation of Cleanup Time in Layered Soils by Vapor Extraction,”
Journal of Contaminant Hydrology, 15 February 1999, Vol. 36, No. 1, Pp. 105-129(25)
Croizet, J.-C., and Fiske, S.T. “Moderation of Priming By Goals: Feeling Entitled To Judge
Increases Judged Usability of Evaluative Primes”. Journal of Experimental Social Psychology,
March 2000, Vol. 36, No. 2, Pp. 155-181(27)
Daniel, C. “Applications of Statistics to Industrial Experimentation,” 1976, P. 294.
Daniel, C. “One-At-A-Time Plans,” Journal of the American Statistical Association 1973, Vol.
68, Pp.353–360
Das, A., and Kandpal, T.C." Energy-Environment Implications of Cement Manufacturing in
India: a Scenario Analysis,” Fuel and Energy Abstracts, November 1997, Vol. 38, No. 6, Pp.
430-430(1)
Davis, J.E. “Accounts of False Memory Syndrome: Parents, "Retractors," and the Role of
Institutions in Account Making”. Qualitative Sociology, 2000, vol. 23, no. 1, pp. 29-56(28)
De Wit, S., Augenbro,E G. “Analysis of Uncertainty in Building Design Evaluations and Its
Implications”. Energy and Buildings, October 2002, Vol. 34, No. 9, Pp. 951-958(8)
Delic, J.I., Et Al. “The Utility of Pbpk in the Safety Assessment of Chloroform and Carbon
Tetrachloride”. Regulatory Toxicology and Pharmacology, October 2000, Vol. 32, No. 2, Pp.
144-155(12)
Deshler, D. “Techniques for Generating Futures Perspectives,” In: Brockett, Ralph G. (Ed.),
Continuing Education in the Year 2000. New Directions for Continuing Education, 1987, No. 36,
Pp. 79-82
Diaper, D. “Task Scenarios and Thought,” Interacting With Computers, October 2002, Vol. 14,
No. 5, Pp. 629-638(10)
Diaz, Iii J., and Hansz J.A. “The Use of Reference Points in Valuation Judgment”. Journal of
Property Research, 1 June 2001, Vol. 18, No. 2, Pp. 141-148(8)
Doran C. M., Et Al. “General Practitioners' Role in Preventive Medicine: Scenario Analysis
Using Smoking As a Case Study,” Addiction, 1 July 1998, Vol. 93, No. 7, Pp. 1013-1022.
Dorp, J.R.V, and Kotz, S. “A Novel Extension of the Triangular Distribution and Its Parameter
Estimation”. Journal of the Royal Statistical Society Series D (The Statistician), 2002, Vol. 51,
No. 1, Pp. 63-79(17)
81
Dreybrodt, W., Gabrovsek, F., and Romanov, D. “Dam Sites in Soluble Rocks: a Model of
Increasing Leakage By Dissolutional Widening of Fractures Beneath a Dam,” Engineering
Geology, October 2003, Vol. 70, No. 1, Pp. 17-35(19)
Driscoll, D.-M., and M. Hoffman W. “Gaining the ethical edge: procedures for delivering
values-driven management”. Long Range Planning, April 1999, vol. 32, no. 2, pp. 179-189(11)
Ducey M.J. “Representing Uncertainty in Silvicultural Decisions: An Application of the
Dempster-Shafer Theory of Evidence,”. Forest Ecology and Management, 15 September 2001,
Vol. 150, No. 3, Pp. 199-211(13)
Ducot, C., and Lubben, G.J. “A Typology for Scenarios,” Futures, 1980 Vol. 12, No.1, 51±57
Duval, A., Fontela, E., and Gabus, A. “Cross-Impact Analysis: a Handbook on Concepts and
Applications,” In: Baldwin, M.M. (Ed.), Portraits of Complexity: Applications of Systems
Methodologies to Societal Problems, 1975, Pp. 202±222
Edmond G. “Legal Engineering: Contested Representations of Law, Science (And Non-Science)
and Society”. Social Studies of Science, June 2002, Vol. 32, No. 3, Pp. 371-412(42). Sage
Publications Inc.
Elander J., Hardman D. “An Application of Judgment Analysis To Examination Marking in
Psychology”. British Journal of Psychology, 1 August 2002, Vol. 93, No. 3, Pp. 303-328(26)
ER 1105-2-100. Planning Guidance Notebook. Department of the Army, U.S. Army Corps of
Engineers, Washington, DC, April 2000.
Escudero, L.F. “Warsyp: a Robust Modeling Approach for Water Resources System Planning
Under Uncertainty,” Annals of Operations Research, 2000, Vol. 95, No. 1/4, Pp. 313-339(27)
Evans J.S.T.B.T., Et Al. “Explicit and Implicit Processes in Multicue Judgment”
Memory & Cognition, 1 June 2003, Vol. 31, No. 4, Pp. 608-618(11)
Evans, R.J. “History, Memory, and the Law: the Historian As Expert Witness”
History and Theory, October 2002, Vol. 41, No. 3, Pp. 326-345(20)
Fahey L. Strategy and Leadership, 15 January 2003, Vol. 31, No. 1, Pp. 32-44(13)
Fahey, L., and Randall, R.M. “Learning From the Future: Competitive Foresight Scenarios,”
1998
Farrington, J. “Socioeconomic Methods in Natural Resources Research,” Natural Resource
Perspectives, 1996, No. 9
Fayerweather W.E., Et Al. “Quantifying Uncertainty in a Risk Assessment Using Human Data,”
Risk Analysis, December 1999, Vol. 19, No. 6, Pp. 1077-1090(14)
82
Ferraro, D.O., Ghersa, C.M., and Sznaider, G.A. “Evaluation of Environmental Impact
Indicators Using Fuzzy Logic To Assess the Mixed Cropping Systems of the Inland Pampa,
Argentina”. Agriculture, Ecosystems and Environment, June 2003, Vol. 96, No. 1, Pp. 1-18(18)
Fisher, E. “Drowning By Numbers: Standard Setting in Risk Regulation and the Pursuit of
Accountable Public Administration” Oxford Journal of Legal Studies, 2000, Vol. 20, No. 1, Pp.
109-130(22)
Fischhoff, B. “Judgmental Aspects of Forecasting: Needs and Possible Trends,” Int. J.
Forecasting, 1988, Vol. 7, Pp. 421±433
Floris, F.J.T., and Peersmann, M.R.H.E. “Integrated Scenario and Probabilistic Analysis for
Asset Decision Support,” Petroleum Geoscience, 2002, Vol. 8, No. 1, Pp. 1-6(6)
Foran, B., and Wardle, K. “Transitions in Land Use and the Problems of Planning: a Case Study
From the Mountain Lands of New Zealand,” J. Environ. Manage. 1995, Vol. 43, Pp. 97±127
Ford, Peters D. “Examining Child Sexual Abuse Evaluations: the Types of Information
Affecting Expert Judgment*1”. Child Abuse and Neglect, January 2001, Vol. 25, No. 1, Pp.
149-178(30)
Foxen, P. “Cacophony of Voices: A K'iche' Mayan Narrative of Remembrance and Forgetting”
Transcultural Psychiatry, September 2000, vol. 37, no. 3, pp. 355-381(27)
Frank, M. V. “Treatment of Uncertainties in Space Nuclear Risk Assessment with Examples
From Cassini Mission Applications,” Safety Factor Associates, Inc., Received 25 April 1998;
Accepted 11 December 1998. Available Online 13 October 1999.
Frederick, K.D., Major, D.C., and Stakhiv, E.Z. “Water Resources Planning Principles and
Evaluation Criteria for Climate Change: Summary and Conclusions,”Climatic Change,
September 1997, Vol. 37, No. 1, Pp. 291-313(23)
Fuller, D., Akinwande, A. and Sodini, C. “Leading, Following or Cooked Goose? Innovation
Successes and Failures in Taiwan’s Electronics Industry”. Industry and Innovation, June 2003,
vol. 10, no. 2, pp. 179-196(18)
Ganzach, Y. “Nonlinear Models of Clinical Judgment: Communal Nonlinearity and Nonlinear
Accuracy”. Psychological Science, September 2001, Vol. 12, No. 5, Pp. 403-407(5)
Garthwaite, P.H., and O’hagan, A. “Quantifying Expert Opinion in the Uk Water Industry: An
Experimental Study”. The Statistician, Journal of the Royal Statistical Society - Series D,
December 2000, Vol. 49, No. 4, Pp. 455-477(23)
Georgantzas, Nicholas C., and William Acar. Scenario-Driven Planning: Learning to Manage
Strategic Uncertainty. Westport, CT: Quorum Books, 1995.
83
Ghaffari, A., Cook, H.F., and Lee, H.C. “Climate Change and Winter Wheat Management: a
Modelling Scenario for South-Eastern England,” Climatic Change, December 2002, Vol. 55, No.
4, Pp. 509-533(25)
Ghyym, S.H. “A Semi-Linguistic Fuzzy Approach To Multi-Actor Decision-Making:
Application To Aggregation of Experts’ Judgments”. Annals of Nuclear Energy, August 1999,
Vol. 26, No. 12, Pp. 1097-1112(16)
Giraud, G., Impara, J.C., and Buckendahl C. “Making the Cut in School Districts: Alternative
Methods For Setting Cutscores”. Educational Assessment, 1 August 2000, Vol. 6, No. 4, Pp.
291-304(14)
Godet, M. “Scenarios and Strategic Management: Prospective ET Plantification Stratégique,”
1987, (D. Green and A. Rodney Translators).
Gomez-Limon, J.A., Arriaza, M. and Berbel, J. “Conflicting Implementation of Agricultural and
Water Policies in Irrigated Areas in the Eu,” Journal of Agricultural Economics, 1 July 2002,
Vol. 53, No. 2, Pp. 259-281(23)
Granger, C. W. J., Forecasting in Business and Economics, Academic Press, New York, 1980.
Gray V. “Developing the corporate memory: the potential of business archives”. Business
Information Review, March 2002, vol. 19, no. 1, pp. 32-37(6)
Grimble, R., and Chan, Man-Kwun, “Stakeholder Analysis for Natural Resource Management in
Developing Countries,” Natural Resources Forum 1995, Vol. 19, No.2, Pp.113-124
Gross A. D. ( 1984). "How Golf Oil deals with uncertainty". Planning Review, 12 ( 2): 8-13.
Gudjonsson, G. H. “The Ira Funeral Murders: the Confession of Pk and the Expert
Psychological Testimony”. Legal and Criminological Psychology, 1 February 1999, Vol. 4, No.
1, Pp. 45-50(6)
Gyldenkaerne, S., Et Al. “Pesticide Leaching Assessment Method for Ranking Both Single
Substances and Scenarios of Multiple Substance Use,” Chemosphere, April 1998, Vol. 36, No.
10, Pp. 2251-2276(26)
Gumbo, B., Et Al. “Coupling of Digital Elevation Model and Rainfall-Runoff Model in Storm
Drainage Network Design” Physics and Chemistry of the Earth, Parts A/B/C, 2002, Vol. 27, No.
11, Pp. 755-764(10)
Gumerman E.; Koomey J.G.; Brown M.A. “Strategies for Cost-Effective Carbon Reductions: a
Sensitivity Analysis of Alternative Scenarios,” Energy Policy, November 2001, Vol. 29, No. 14,
Pp. 1313-1323(11)
84
Hampton, M., and Sperling, J. “Positive Negative Identity in the Euro-Atlantic Communities:
Germanys Past, Europe’s Future?”. Journal of European Integration, 1 January 2002, vol. 24, no.
4, pp. 281-301(21)
Hanso, M. “Institutional Theory and Educational Change”. Educational Administration
Quarterly, December 2001, vol. 37, no. 5, pp. 637-661(25)
Harrell, A.T. “New Methods in Social Science Research: Policy Sciences and Future Research,”
Praeger, 1978
Harris K.M. “Can High Quality Overcome Consumer Resistance To Restricted Provider Access?
Evidence From A Health Plan Choice Experiment, ” Health Services Research, June 2002, Vol.
37, No. 3, Pp. 551-571(21)
Harvey D.C. Jones R. “Custom and habit (us): The meaning of traditions and legends in early
medieval western Britain”. Geografiska Annaler: Series B, Human Geography, 1999, vol. 81B,
no. 4, pp. 223-233(11)
Harwell, C.C., Et Al. “Use of a Conceptual Model of Societal Drivers of Ecological Change in
South Florida: Implications of an Ecosystem Management Scenario,” Urban Ecosystems,
11 October 1999, Vol. 3, No. 3/4, Pp. 345-368(24)
Helgason T. & Wallace S. W. ( 1991). "Approximate scenario solutions in the progressive
hedging algorithm: A numerical study with an application to fisheries management". Annals of
Operations Research, 31 ( 2): 425-444.
Henderson-Sellers, B., and Henderson-Sellers, A. “Sensitivity Evaluation of Environmental
Models Using Fractional Factorial Experimentation,” Ecological Modelling, 1996, Vol. 86, Pp.
291–295
Hofer, E., Javeri, V., and Loffler, H., "A Survey of Expert Opinion and Its Probabilistic
Evaluation for Specific Aspects of the SNR-300 Risk Study," Nuclear Technology, vol. 68, pp.
180-225, 1985.
Holling, C.S. “Adaptive Environmental Assessment and Management,” Wiley International
Series On Applied Systems Analysis, 1978, Vol. 3
Hsia P., Kung D., and Sell C. “Software Requirements and Acceptance Testing,” Annals of
Software Engineering, 1997, Vol. 3, No. 1, Pp. 291-317(27)
Humphreys, P., Human Reliability Assessors Guide, Safety and Reliability Directorate, United
Kingdom Atomic Energy Authority, 1988.
85
Hurley, W.J., and Lior, D.U. “Combining Expert Judgment: on the Performance of Trimmed
Mean Vote Aggregation Procedures in the Presence of Strategic Voting”. European Journal of
Operational Research, 1 July 2002, Vol. 140, No. 1, Pp. 142-147(6)
Huss, W.R., and Honton, E.J. “Scenario Planning: What Style Should You Use?” Long Range
Planning, 1987, Vol. 20, No.4, Pp.21-29
Hunt, E.R., Piper, S.C., and Winslow, J.C. “The Influence of Seasonal Water Availability On
Global C3 Versus C4 Grassland Biomass and Its Implications for Climate Change Research,”
Ecological Modelling, 1 May 2003, Vol. 163, No. 1, Pp. 153-173(21)
Ichii, K., Et Al. “A Simple Global Carbon and Energy Coupled Cycle Model for Global
Warming Simulation: Sensitivity to the Light Saturation Effect,” Tellus B, April 2003, Vol. 55,
No. 2, Pp. 676-691(16)
Imundo A. ( 1986). Are you optimizing returns on your check processing dollar? Bank Systems
& Equipment, 23 ( 9): 76-78.
Jackson, D.A., Et Al. “Developing A Generalized Flaw Distribution For Reactor Pressure
Vessels” Nuclear Engineering and Design, 1 September 2001, Vol. 208, No. 2, Pp. 123-131(9)
Jauch and Glueck, Business Policy and Strategic Management, 1st Edition, Jauch and Glueck,
McGraw-Hill, 1988.
Jinno, K. “Performance Risk Analysis for Fukuoka Water Supply System,” Water Resources
Management, February 1998, Vol. 12, No. 1, Pp. 13-30(18)
Jones S. T. ( 1985). Multiple scenario planning: Atlantic Richfield's experience. l Journal of
Business Forecasting, 4 ( 3): 19-23.
Jorgensen, S.E. Fundamentals of Ecological Modelling, 1994, 2nd Edition, P. 628
Kahane, A. “Scenarios for Energy: Sustainable World vs. Global Mercantilism,” Long Range
Planning, 1992, Vol. 25, No. 4, Pp. 38-46
Kahn, H., 1965. On Escalation: Metaphors and Scenarios. Praeger, New York.
Kaluarachchi, J.J., and Khadam, I. “Applicability of Risk-Based Management and the Need for
Risk-Based Economic Decision Analysis At Hazardous Waste Contaminated Sites,”
Environment International, July 2003, Vol. 29, No. 4, Pp. 503-519(17)
Kaplan, S., and Garrick, B., "On the Quantitative Definition of Risk," Risk Analysis, vol. 1, pp.
11-27, 1981.
86
Kaplan, S., Haimes, Y.Y., and Garrick, B.J. “Fitting Hierarchical Holographic Modeling into the
Theory of Scenario Structuring and a Resulting Refinement to the Quantitative Definition of
Risk,” Risk Analysis, October 2001, Vol. 21, No. 5, Pp. 807-807(1)
Kayse,R K., Et Al. “Recent Developments and Present Status of Telepathology”. Analytical
Cellular Pathology, 2000, Vol. 21, No. 3-4, Pp. 101-106(6)
Kazman, R., Carrière, S.J., and Woods, S.G. “Toward a Discipline of Scenario-Based
Architectural Engineering,” Annals of Software Engineering, 2000, Vol. 9, No. 1/4, Pp. 5-33(29)
Kemp, W. “Who is Taking Notes? Institutional Memory and the OSCE”. Helsinki Monitor,
2001, vol. 12, no. 4, pp. 243-244(2)
Kennedy, P., Perrottet, C., and Thomas, C., “Scenario Planning After 9/11: Managing the Impact
of a Catastrophic Event,” Strategy and Leadership, 2003, Vol. 31 No. 1 Pps. 4 – 13
Kessler, J.H., and Mcguire, R.K. “Total System Performance Assessment for Waste Disposal
Using A Logic Tree Approach”. Risk Analysis, October 1999, Vol. 19, No. 5, Pp. 915-931(17)
Khan, Faisal, I. Husain, Tahir, and Abbasi, S. A. “Design and Evaluation of Safety Measures
Using a Newly Proposed Methodology "SCAP"” Journal of Loss Prevention in the Process
Industries, March 2002, vol. 15, no. 2, pp. 129-146(18)
Kim M-S. “Cloning and Deliberation: Korean Consensus Conference”. Developing World
Bioethics, November 2002, Vol. 2, No. 2, Pp. 159-172(14)
Klein. H. E. & Linneman R. E. ( 1984). "Environmental assessment: An international study of
corporate practices". Journal of Business Strategy, 5: 66-84.
Kraan B.C.P., and Cooke R.M. “Uncertainty in Compartmental Models For Hazardous Materials
- A Case Study”. Journal of Hazardous Materials, 7 January 2000, Vol. 71, No. 1, Pp. 253268(16)
Kumar, M.J. “The White House As City Hall: A Tough Place to Organize”. Presidential Studies
Quarterly, March 2001, vol. 31, no. 1, pp. 44-55(12)
Lad, F., and Di Bacco, M. “Assessing the Value of A Second Opinion: the Role and Structure
of Exchangeability”. Annals of Mathematics and Artificial Intelligence, May 2002, Vol. 35,
No. 1-4, Pp. 227-252(26)
Leden, L., Garder, P., and Pulkkinen, U. “An Expert Judgment Model Applied To Estimating
the Safety Effect of A Bicycle Facility”. Accident Analysis and Prevention, July 2000, Vol. 32,
No. 4, Pp. 589-599(11)
87
Lee, Y. T., Okrent, D. and Apostolakis, G., "A Comparison of Background Seismic Risks and
the Incremental Seismic Risk Due to Nuclear Power Plants," Nuclear Engineering & Design,
vol. 53, pp. 141-154, 1979.
Lee, H., Lee, C., and Yoo, C., “A Scenario-Based Object-Oriented Hypermedia Design
Methodology,” Information and Management, September 1999, Vol. 36, No. 3, Pp. 121-138(18)
Legesse, D., Vallet-Coulomb, C., and Gasse, F. “Hydrological Response of a Catchment to
Climate and Land Use Changes in Tropical Africa: Case Study South Central Ethiopia,”. Journal
of Hydrology, 25 April 2003, Vol. 275, No. 1, Pp. 67-85(19)
Lelong, S., and Santos, C. “The child that shames dead as indication of a psychotic potentiality”.
Annales medico-psychologiques, May 2001, vol. 159, no. 4, pp. 336-340(5)
Lessard, G. “An Adaptive Approach to Planning and Decision Making,” Landscape and Urban
Planning 1998, Vol.40, No. 1-3, Pp.81-87
Lettenmaier, D.P., Et Al. “Water Resources Implications of Global Warming: a U.S. Regional
Perspective,” Climatic Change, November 1999, Vol. 43, No. 3, Pp. 537-579(43)
Leufkens H., Hekster Y., and Hudson S. “Scenario Analysis of the Future of Clinical Pharmacy,”
Pharmacy World and Science, August 1997, Vol. 19, No. 4, Pp. 182-185(4)
Levine, S., and Rasmussen, N., "Nuclear Plant PRA: How Far Has It Come," Risk Analysis, vol.
4, 247-255, 1984.
Lewis, H. W., Budnitz, R. J., Kouts, H. J. C., Lowenstein, W. B., Rowe, W. D., Von F. Hippel,
and Zachariasen, F., Risk Assessment Review Group Report to the U.S. Nuclear Regulatory
Commission, NUREG/CR-0400, 1979.
Liken, M.A. “Managing Transitions and Placement of Caring for a Relative With Alzheimer's
Disease”. Home Health Care Management & Practice, December 2001, vol. 14, no. 1, pp. 3139(9)
Linneman R. E. & Klein H. E. ( 1983). "The use of multiple scenarios by US industrial
companies: A comparison study, 1977-1981". Long Range Planning, 16 ( 6): 94-101.
Lind, M.R., and Sulek, J.M. “A Methodology For Forecasting Knowledge Work Projects”.
Computers and Operations Research, September 2000, Vol. 27, No. 11, Pp. 1153-1169(17)
Lloyd, D. “Colonial Trauma/Postcolonial Recovery?” Interventions: International Journal of
Postcolonial Studies, 1 July 2000, vol. 2, no. 2, pp. 212-228(17)
88
Locke, A., and Edwards, D. “Bill and Monica: Memory, emotion and normativity in Clintons
Grand Jury testimony”. British Journal of Social Psychology, 1 June 2003, vol. 42, no. 2, pp.
239-256(18)
Lucius-Hoene, G., and Deppermann, A. “Narrative Identity Empiricized: A Dialogical and
Positioning Approach to Autobiographical Research Interviews”. Narrative Inquiry, December
2000, vol. 10, no. 1, pp. 199-222(24)
Lundberg, C.G., and Nagle, B.M. “Post-Decision Inference Editing of Supportive and
Counterindicative Signals Among External Auditors in A Going Concern Judgment”. European
Journal of Operational Research, 16 January 2002, Vol. 136, No. 2, Pp. 264-281(18)
Lupton D., and Tulloch J. “Theorizing Fear of Crime: Beyond the Rational/Irrational
Opposition”. British Journal of Sociology, 1 September 1999, Vol. 50, No. 3, Pp. 507-523(17)
Maarleveld, M., and Dangbegnon, C. 1998. “Managing Natural Resources in Face of Evolving
Conditions: a Social Learning Perspective,” 76 E. Wollenberg Et Al. / Landscape and Urban
Planning 47 (2000) 65±77 Paper Presented At the Conference Crossing Boundaries, 7th
Conference of the International Association for the Study of Common Property, Vancouver,
Canada, 10±14, 1998, June.
Macdonell S.G., Shepperd M.J. Combining Techniques To Optimize Effort Predictions in
Software Project Management”. Journal of Systems and Software, 15 May 2003, Vol. 66, No.
2, Pp. 91-98(8)
Malafant, K.W.J., and Fordham, D.P. “Gis, Dss and Integrated Scenario Modelling Frameworks
for Exploring Alternative Futures,” In: Uso, J.L., Brebbia, C.A., and Power, H. (Eds.), Advance
in Ecological Sciences: Ecosystems and Sustainable Development, 1997, Vol. 1, Proceedings of
a Conference, Peniscola, Spain, 14-16 October 1997, Pp. 669-678
Mantovani, G., “Social Context in Hcl: a New Framework for Mental Models, Cooperation, and
Communication,” Cognitive Science, 6 April 1996, Vol. 20, No. 2, Pp. 237-269(33)
Marcot, B.G., Et Al. “Using Bayesian Belief Networks To Evaluate Fish and Wildlife
Population Viability Under Land Management Alternatives From An Environmental Impact
Statement,” Forest Ecology and Management, 1 October 2001, Vol. 153, No. 1, Pp. 29-42(14)
Marti P., Pucci F., and Rizzo A. “External Aids for Social Memory,” Information,
Communication and Society, 1 June 2001, Vol. 4, No. 2, Pp. 261-273(13)
Mathevet, R., Et Al. “Agent-Based Simulations of Interactions Between Duck Population,
Farming Decisions and Leasing of Hunting Rights in the Camargue (Southern France),”
Ecological Modelling, 15 July 2003, Vol. 165, No. 2, Pp. 107-126(20)
Matsuoka U. “Fuel and Energy Abstracts,” November 1995, Vol. 36, No. 6, Pp. 467-467(1)
89
McCarthy, M.A., Burgman, M.A., and Ferson, S. “Sensitivity Analysis for Models of Population
Viability,” Biological Conservation 1995, Pp. 73, 93–100
McGovern, T.J .and Samuels, H.W. “Our institutional memory at risk: collaborators to the
rescue”. Campus-Wide Information Systems, 14 August 1998, vol. 15, no. 3, pp. 103-107(5)
Mcintosh, N., Et Al. “Genetic Counseling For Fragile X Syndrome: Recommendations of the
National Society of Genetic Counselors”. Journal of Genetic Counseling, August 2000, Vol. 9,
No. 4, Pp. 303-325(23)
Mclain, R.J., and Lee, R.G. “Adaptive Management: Promises and Pitfalls,” Environ. Manage.
1996, Vol. 20, No.4, Pp. 437-448
Mercer, D. “Scenarios Made Easy”. Long Range Planning, August 1995, vol. 28, no. 4, pp. 78+81(2), Elsevier Science
Merkhofer, M., and Keeney, R., "A Multiattribute Utility Analysis of Alternative Sites for the
Disposal of Nuclear Waste," Risk Analysis, vol. 7, no. 2, pp. 173-194, 1987.
Millett, S.M. “How Scenarios Trigger Strategic Thinking,” Long Range Planning, 1988, Vol. 21,
No. 5, Pp. 61-68.
Millett S. M. & Randles F. ( 1986). "Scenarios for strategic business planning: A case history for
aerospace and defence companies". Interfaces, 16 ( 6): 64-72.
Minarick, J., and Kukielka, C., "Precursors to Potential Severe Core Damage Accidents: 19691979," NUREG/CR-2497, 1982.
Mitr, S.K. “Language and federalism: the multi-ethnic challenge”. International Social Science
Journal, March 2001, vol. 53, no. 167, pp. 51-60(10)
Mitter, R. “Old Ghosts, New Memories: Chinas Changing War History in the Era of Post-Mao
Politics,” Journal of Contemporary History, 1 January 2003, vol. 38, no. 1, pp. 117-131(15)
Mizina, S.V., et al. “An Evaluation of Adaptation Options For Climate Change Impacts on
Agriculture in Kazakhstan”. Mitigation and Adaptation Strategies For Global Change, 1999,
Vol. 4, No. 1, Pp. 25-41(17)
Montgomery, D.C. “Design and Analysis of Experiments,” 1997, Wiley, New York, P. 704
“Movements”. Qualitative Sociology, 2002, vol. 25, no. 3, pp. 443-458(16)
Morgan, Granger M ,Amaral, D., Henrion, M., and Morris, S., "Technological Uncertainty in
Quantitative Policy Analysis--A Wulfur Pollution Example," Risk Analysis, vol. 3, pp. 201-220,
1984.
90
Morgan, G.M., and M. Henrion, Uncertainty; a Guide to Dealing with Uncertainty in
Quantitative Risk and Policy Analysis, Department of Engineering and Public Policy, Carnegie
Mellon University, 1988.
Morris, J. M., and D'Amore, R. J., "Aggregating and Communicating Uncertainty," Pattern
Analysis and Recognition Corp., 228 Liberty Plaza, Rome, New York, 1980.
Mosleh, A., Bier, V., and Apostolakis, G., "Critique of Current Practice for the Use of Expert
Opinions in Probabilistic Risk Assessment," Reliability Engineering and System Safety, vol. 20,
pp. 63-85, 1988.
Mozina M., “Can we remember differently? A case study of the new culture of memory in
voluntary organizations,”. International Journal of Social Welfare, October 2002, vol. 11, no. 4,
pp. 310-320(11)
Mulvey J.M., Rosenbaum D.P., and Shetty B. “Parameter Estimation in Stochastic Scenario
Generation Systems,” European Journal of Operational Research, 1 November 1999, Vol. 118,
No. 3, Pp. 563-577(15)
Nobel, C.E., and Allen, D.T. “Using Geographic Information Systems (Gis) in Industrial Water
Reuse Modelling,” Process Safety and Environmental Protection, 1 July 2000, Vol. 78, No. 4,
Pp. 295-303(9)
Nordmann, L.H., and Luxhoj, J.T. “Application of A Performance Measure Reduction
Technique To Categorical Safety Data,” Reliability Engineering and System Safety, January
2002, Vol. 75, No. 1, Pp. 59-71(13)
Nuclear Engineering and Design Volume 93 (1986)
Nyhart, L.K. “The Importance of the "Gegenbaur School" for German Morphology”
Theory in Biosciences, 1 May 2003, vol. 122, no. 2-3, pp. 162-173(12)
O'Connor A. “WHOS EMMA AND THE LIMITS OF CULTURAL STUDIES”. Cultural
Studies, 1 October 1999, vol. 13, no. 4, pp. 691-702(12)
Ohe, T. “Simple Assignment of Partitioning & Transmutation Objectives to Reduce the Overall
Repository Impacts Due to Thermal Load and Nuclide Migration,” Progress in Nuclear Energy,
April 2002, Vol. 40, No. 3, Pp. 423-430(8)
Okrent, D., "A Survey of Expert Opinion on Row Probability Earthquakes," in Annals of Nuclear
Energy, Pergamon Press, pp. 601-614, 1975.
Ozbekahn, H. “The Future of Paris: a Systems Study in Strategic Urban Planning,” Philosophical
Transactions of the Royal Society of London, 1977, Pp. 387: 523-544
91
Phelps, R., Chan, C., and Kapsalis, S.C. “Does Scenario Planning Affect Performance? Two
Exploratory Studies,” Journal of Business Research, March 2001, Vol. 51, No. 3, Pp. 223232(10)
Plackett, R.L., and Burman, J.P. “The Design of Optimum Multifactorial Experiments,”
Biometrika, 1946, Vol. 33, Pp305–325
”Political Studies Books”. Political Studies, August 2002, vol. 50, no. 3, pp. 587-658(3)
Pollitt, C. “Institutional Amnesia: A Paradox of the 'Information Age'?” Prometheus, 1 March
2000, vol. 18, no. 1, pp. 5-16(12)
Ports R. E.E. ( 1985). "A strategic planning system in a Canadian gas utility". Public Utilities
Fortnightly, 116 ( 8): 22-30.
Prebble J. F. & Reichel A. ( 1988). "Scanning the future environment for banking". Mid.
American Journal of Business, 3 ( 2): 23-31.
Ramachandran, G. “Retrospective Exposure Assessment Using Bayesian Methods” The Annals
of Occupational Hygiene, November 2001, Vol. 45, No. 8, Pp. 651-667(17)
Ramachandran G., Banerjee S., and Vincent J.H. “Expert Judgment and Occupational Hygiene:
Application To Aerosol Speciation in the Nickel Primary Production Industry”. Annals of
Occupational Hygiene, August 2003, Vol. 47, No. 6, Pp. 461-475(15)
Ramachandran, G., and Vincent, J. H. “A Bayesian Approach To Retrospective Exposure
Assessment”. Applied Occupational and Environmental Hygiene, 1 August 1999, Vol. 14, No.
8, Pp. 547-557(11)
Regnell, B., Runeson, P., and Thelin, T. “Are the Perspectives Really Different? – Further
Experimentation On Scenario-Based Reading of Requirements,” Empirical Software
Engineering, December 2000, Vol. 5, No. 4, Pp. 331-356(26)
Renn, O. Precautionary Principle: Risk Uncertainty and Rational Action
Prof. Ortwin Renn, University of Stuttgart, Germany
http://www.allchemeseminars.org/downloads/03-10-01/20031001report.pdf
Ribera, I., Et Al. “Discovery of Aspidytidae, a New Family of Aquatic Coleoptera”.
Proceedings: Biological Sciences, 22 November 2002, Vol. 269, No. 1507, Pp. 2351-2356(6)
Rienstra S.A., Nijkamp P. “The Role of Electric Cars in Amsterdam's Transport System in the
Year 2015, a Scenario Approach,” Transportation Research Part D: Transport and Environment,
January 1998, Vol. 3, No. 1, Pp. 29-40(12)
Ringland, Gill “Using Scenarios to Focus R&D,” Strategy and Leadership, 2003a, Vol. 31, No.1,
Pp.45-55
92
Ringland, G. (2003b) Scenario planning: persuading operating managers to take ownership
Strategy and Leadership 1 June 2003, vol. 31, no. 6, pp. 22-28(7)
Risbey, J.S. “Sensitivities of Water Supply Planning Decisions to Stream Flow and Climate
Scenario Uncertainties,” Water Policy, June 1998, Vol. 1, No. 3, Pp. 321-340(20)
Rothman, H. K. “Institutional Memory and Management Needs: History in the Park Service's
Northwest” The Public Historian, 1 April 2001, vol. 23, no. 2, pp. 87-91(5)
Rothstein, B. Trust, Social Dilemmas and Collective Memories. Journal of Theoretical Politics,
October 2000, vol. 12, no. 4, pp. 477-501(25)
Rowe, G., and Wright, G. “Differences in Expert and Lay Judgments of Risk: Myth Or
Reality?” Risk Analysis, April 2001, Vol. 21, No. 2, Pp. 341-356(16)
Rushton, M. “Political Oversight of Arts Councils: A Comparison of Canada and the United
States”. International Journal of Cultural Policy, 1 January 2002, Vol. 8, No. 2, Pp. 153-165(13)
Saccomanno, F., and Haastrup, P. “Influence of Safety Measures on the Risks of Transporting
Dangerous Goods Through Road Tunnels”. Risk Analysis, December 2002, Vol. 22, No. 6, Pp.
1059-1069(11)
Sakellaropoulos, G.C., and Nikiforidis, G.C. “Prognostic Performance of Two Expert Systems
Based on Bayesian Belief Networks”. Decision Support Systems, January 2000, Vol. 27, No. 4,
Pp. 431-442(12)
Samet, M. G., "Quantitative Interpretation of Two Qualitative Scales Used to Rate Military
Intelligence," Human Factors, vol. 17, no. 2, pp. 192-202, 1975.
Sapio, B. “Search (Scenario Evaluation and Analysis Through Repeated Cross Impact
Handling): a New Method for Scenario Analysis With an Application to the Videotel Service in
Italy,” Int. J. Forecast, 1995, Vol.11, No. 1, Pps.113-131.
Sarangi, S., Clarke A. “Zones of Expertise and the Management of Uncertainty in Genetics
Risk Communication”. Research on Language and Social Interaction, 1 April 2002, Vol. 35, No.
2, Pp. 139-171(33)
Satterfield, T., Slovic, P., and Gregory, R. “Narrative Valuation in A Policy Judgment Context”.
Ecological Economics, September 2000, Vol. 34, No. 3, Pp. 315-331(17)
Schoemaker, P.J.H. “Multiple Scenario Development: It’s Conceptual and Behavioral
Foundation,” Strategic Manage. J. 1993, Vol. 14, No. 3, Pp. 193-213.
Schoemaker, P.J.H. “When and How to Use Scenario Planning: a Heuristic Approach With
Illustration,” J. Forecast, 1991, Vol. 10, Pp. 549-564.
93
Schriefer, A. and D. Mercer. “Scenarios Made Easy - Getting the Most Out of Scenarios” The
Journal of Product Innovation Management, March 1996, vol. 13, no. 2, pp. 175-176(2) Elsevier
Science
Schuenemeyer, J.H. “A Framework For Expert Judgment To Assess Oil and Gas Resources”.
Natural Resources Research, June 2002, Vol. 11, No. 2, Pp. 97-107(11)
Schultz D. I. ( 1986). "Strategic information systems planning sharpens competitive edge". Data
Management, 24 ( 6): 20-24, 38.
Schwartz P. ( 1991). The Art of the Long View. New York: Doubleday Currency.
Schwenk C. R. ( 1984). "Cognitive simplification processes in strategic decisionmaking".
Strategic Management Journal, 5 ( 2): 111-128.
Seraphine, A.E. “The Performance of Dimtest When Latent Trait and Item Difficulty
Distributions Differ” Applied Psychological Measurement, March 2000, Vol. 24, No. 1, Pp. 8294(13)
Shetle,R K., Karlsdottir, A., and Froelicher, V. “Assessing Patients With Possible Heart Disease
Using Scores” Sports Medicine, 1 June 2001, Vol. 31, No. 6, Pp. 387-408(22)
Shooman, M., and Sinkar, S., "Generation of Reliability and Safety Data by Analysis of Expert
Opinion," Proc. 1977 Annual Reliability and Maintainability Symposium, pp. 186-193, 1977.
Shu, Y., Furuta, K., and Kondo, S. “Team Performance Modeling for Hra in Dynamic
Situations,” Reliability Engineering and System Safety, November 2002, Vol. 78, No. 2, Pp.
111-121(11)
Silberfeld, M., and Checkland, D. “Faulty Judgment, Expert Opinion, and Decision-Making
Capacity” Theoretical Medicine and Bioethics, August 1999, Vol. 20, No. 4, Pp. 377-393(17)
Siu T.K., and Yang H. “Subjective Risk Measures: Bayesian Predictive Scenarios Analysis,”
Insurance: Mathematics and Economics, 16 November 1999, Vol. 25, No. 2, Pp. 157-169(13)
Slocum, R., and Klaver, D. “Time Line Variations,” In: Slocum, R., Wichart, L., Rocheleau, D.,
Thomas-Slayter, B. (Eds.), Power, Process and Participation -Tools for Change. 1995, Pp. 194197.
Squarcini, F. “In Search of Identity within the Hare Krishna Movement: Memory, Oblivion and
Thought Style”. Social Compass, June 2000, vol. 47, no. 2, pp. 253-271(19)
Stamelos, I., Et Al. “On the Use of Bayesian Belief Networks For the Prediction of Software
Productivity”. Information and Software Technology, 1 January 2003, Vol. 45, No. 1, Pp. 5160(10)
94
Starfield, A.M., and Bleloch, A.L. “Building Models for Conservation and Wildlife
Management,” Burgess International Group Inc., 1991, P. 253.
Steelman, T.A., and Ascher, W. “Public Involvement Methods in Natural Resource Policy
Making: Advantages, Disadvantages and Trade-Offs,” Policy Sciences, 1997. Vol. 30, Pp. 71-90.
Stern, P.C., and Druckman, D. “Evaluating Interventions in History: the Case of International
Conflict Resolution”. The International Studies Review, Spring 2000, Vol. 2, No. 1, Pp. 3363(31)
Stewart, J.T., and Scott, L., “A Scenario-Based Framework for Multicriteria Decision Analysis
in Water Resources Planning,” Water Resources, 1995, Vol. 31, No. 11, Pp. 2835-2843
Sui, N., and Apostolakis, G., "Combining Data and Judgment in Fire Risk Analysis," 8th
International Conference on Structural Mechanics in Reactor Technology, Brussels, Belgium,
August 26-27, 1985.
Svartdal, F. “Persistence During Extinction: Are Judgments of Persistence Affected By
Contingency Information?”. Scandinavian Journal of Psychology, December 2000, Vol. 41, No.
4, Pp. 315-328(14)
Swain, A., and Guttman, H., Handbook of Human Reliability, Analysis with Emphasis on
Nuclear Power Plant Applications, NUREG/CR-1278, 1983.
Swaminathan, S., and Smidts, C. “The Mathematical Formulation for the Event Sequence
Diagram Frame Work,” Reliability Engineering Program, Received 22 June 1998; Accepted 10
October 1998; Available Online 31 March 1999
Swartzman, G.L., and Kaluzny, S.P. “Ecological Simulation Primer,” 1987, Macmillan, New
York, P. 370
Tainter, Joseph A. Getting Down To Earth: Practical Applications of Ecological Economics,
Island Press, 1996; ISBN 1-55963-503-7
Tan-Kim-Yong, U. “Participatory Land-Use Planning for Natural Resource Management in
Northern Thailand,” Network Paper 1992, No. 14b, Rural Development Forestry Network.
Tao F., Et Al. “Future Climate Change, the Agricultural Water Cycle, and Agricultural
Production in China,” Agriculture, Ecosystems and Environment, April 2003, Vol. 95, No. 1, Pp.
203-215(13)
Taylor, B., Kremsater, L., and Ellis, R. “Adaptive Management of Forests in British Columbia,”
British Columbia Ministry of Forests, Canada, Report, 1997
95
Taylor, G., and Horan, A. “From cats, dogs, parks and playgrounds to IPC licensing: policy
learning and the evolution of environmental policy in Ireland”. British Journal of Politics and
International Relations, October 2001, vol. 3, no. 3, pp. 369-392(24)
Temkin, M., ”Avec un certain malaise: The Paxtonian Trauma in France, 1973-74”. Journal of
Contemporary History, 1 April 2003, vol. 38, no. 2, pp. 291-306(16)
Todorovska, M.I., and Trifunac, M.D. “Northridge, California, Earthquake of 1994: Density of
Pipe Breaks and Surface Strains,” Soil Dynamics and Earthquake Engineering, 1997, Vol. 16,
No. 3, Pp. 193-207(15)
Tol, R.S.J., Et Al. “Adapting to Climate: a Case Study On Riverine Flood Risks in the
Netherlands,” Risk Analysis, June 2003, Vol. 23, No. 3, Pp. 575-583(9)
Trenberth, K. “Science Considers Global Climate Change? No Longer Any Reasonable
Disagreement Says Us Expert”. Refocus, November 2002, Vol. 2002, No. 11, Pp. 50+52-53(4)
Trepel, M., and Kluge, W. “Ecohydrological Characterization of a Degenerated Valley Peatland
in Northern Germany for Use in Restoration,” Journal for Nature Conservation, November 2002,
Vol. 10, No. 3, Pp. 155-169, (15) & #169
Tuljapurkar, S., and Boe,C. “Validation, Probability-Weighted Priors, and Information in
Stochastic Forecasts”. International Journal of Forecasting, July 1999, Vol. 15, No. 3, Pp. 259271(13)
U.S. NRC, "Nuclear Regulatory Commission Issues Policy Statement on Reactor Safety Study
and Review by the Lewis Panel," NRC Press Release, no. 79-19, January 19, 1979.
U.S. Water Resources Council. 1983. Economic and Environmental Principles and Guidelines
for Water and Related Land Resources Implementation Studies. Washington, DC: U.S.
Government Printing Office (March 10, 1983).
Van De Klundert, A.F., 1995 “The Future's Future: Inherent Tensions between Research, Policy
and the Citizen in the Use of Future Oriented Studies,” In: Scenario Studies for the Rural
Environment. Proceedings of the Symposium Scenario Studies for the Rural Environment,
Schoute, J.F.T., Finke, P.A., Veeneklaas, F.R., and Wolfert, H.P. (Eds.), 12-15 September 1994,
Pp. 25-32
Van Der Fels-Klerx, I.H.J., Et Al. “Elicitation of Quantitative Data From A Heterogeneous
Expert Panel: Formal Process and Application in Animal Health”. Risk Analysis, February 2002,
Vol. 22, No. 1, Pp. 67-81(15)
Van Der Linden, S., Et Al. “Sensitivity Analysis of Discharge in the Arctic Usa Basin, EastEuropean Russia,” Climatic Change, March 2003, Vol. 57, No. 1, Pp. 139-161(23)
96
Van Dorp, J.R., Et Al. “A Risk Management Procedure For the Washington State Ferries”. Risk
Analysis, February 2001, Vol. 21, No. 1, Pp. 127-142(16)
Van Huylenbroeck, G., and Coppens, A. “Multicriteria Analysis of the Conflicts Between Rural
Development Scenarios in the Gordon District Scotland,” J. Environ. Plann Manage. 1995 Vol.
38 No. 3, Pp.393-407
Van Imhoff E., and Henkens K. “The Budgetary Dilemmas of an Ageing Workforce: a Scenario
Study of the Public Sector in the Netherlands,” European Journal of Population/ Revue
Europenne De Dmographie, March 1998, Vol. 14, No. 1, Pp. 39-59(21)
Van Noortwijk, J.M. “Optimal Maintenance Decisions on the Basis of Uncertain Failure
Probabilities”. Journal of Quality in Maintenance Engineering, 1 February 2000, Vol. 6, No. 2,
Pp. 113-123(11)
Veldkamp, A., and Fresco, L.O. “Exploring Land Use Scenarios: an Alternative Approach Based
On Actual Land Use,” Agric. Syst. 1997 Vol. 55 No. 1, Pp.1-17
Violato, C., Marini, A., and Lee, C. “A Validity Study of Expert Judgment Procedures For
Setting Cutoff Scores on High-Stakes Credentialing Examinations Using Cluster Analysis”.
Evaluation & the Health Professions, 1 March 2003, Vol. 26, No. 1, Pp. 59-72(14)
Wack, P. “Scenarios: Uncharted Waters Ahead,” Harvard Business Review 1985a Vol.63 No.
5, Pp. 72-89
Wack, P. “Scenarios: Shooting the Rapids,” Harvard Business Review 1985b Vol. 63 No.6, Pp.
139-150
Wagner, D. “Assessment of the Probability of Extreme Weather Events and Their Potential
Effects in Large Conurbations,” Available Online 23 July 1999
Walters, C. “Adaptive Management of Renewable Resources,” 1986, Macmillan
Werritty, A. “Living With Uncertainty: Climate Change, River Flows and Water Resource
Management in Scotland,” the Science of the Total Environment, 22 July 2002, Vol. 294, No. 1,
Pp. 29-40(12)
Wiggins, J., "ESA Safety Optimization Study," Hernandez Engineering, HEI-685/1026,
Houston, Texas, 1985.
Wohlin C., Andrews A.A. “Prioritizing and Assessing Software Project Success Factors and
Project Characteristics Using Subjective Data”. Empirical Software Engineering, September
2003, Vol. 8, No. 3, Pp. 285-308(24)
Wolf, J., and Diepen, C.A. “Effects of Climate Change On Grain Maize Yield Potential in the
European Community,” Climate Change 1995, Vol. 29, No. 3, Pp. 299-331
97
Wollenberg, E. et al. / Landscape and Urban Planning 47 (2000) 65±77 67
Woods B. “Dementia care: progress and prospects” Journal of Mental Health, 1 April 1995, vol.
4, no. 2, pp. 115-124(10)
Worsley, H. “The Impact of the Inner-child on Adult Believing”. Journal of Beliefs and Values,
1 October 2002, vol. 23, no. 2, pp. 191-202(12)
Wright P. A. & Hill T. V. ( 1986). "Cost estimating: Dealing with uncertainty". AACE
Transactions,: E.5. 1-E.5.8.
Wright ,G., Pearman, A., and Yardle, K. “Risk Perception in the U.K. Oil and Gas Production
Industry: Are Expert Loss-Prevention Managers Perceptions Different From Those of Members
of the Public?” Risk Analysis, October 2000, Vol. 20, No. 5, Pp. 681-690(10)
Yin, Y., Pierce, J.T., and Love, E., “Designing a Multisector Model for Land Conversion Study,”
J. Environ. Manage. 1995, Vol. 44, Pp. 249-266.
Yoe, C.E. Risk Analysis Framework for Cost Estimation. December 2000, IWR Report 00-R-9,
U.S. Army Corps of Engineers , Alexandria, Virginia.
Yoe, C.E., and K. D. Orth. Planning Manual. U.S. Army Corps of Engineers, Institute for Water
Resources, IWR Report 96-R-21, November 1996.
Young, W.J., Et Al. “Development of an Environmental Flows Decision Support System,”
Environmental Modelling and Software With Environment Data News, March 2000, Vol. 15,
No. 3, Pp. 257-265(9)
Yu D., Park W.S. “Combination and Evaluation of Expert Opinions Characterized in Terms of
Fuzzy Probabilities” Annals of Nuclear Energy, May 2000, Vol. 27, No. 8, Pp. 713-726(14)
Zavirsek, D., “Pictures and silences: memories of sexual abuse of disabled people,”
International Journal of Social Welfare, October 2002, vol. 11, no. 4, pp. 270-28
98