Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
Exploratory Modelling and Analysis, an approach for model-based
foresight under deep uncertainty
1 Introduction
Future oriented technology analysis (FTA) is understood as the field interested in analysing
future technology and its consequences. FTA relies on a wide array of methods for exploring
futures. FTA is an umbrella term for the work done in various future-oriented research fields,
including technology forecasting, future studies, foresight, and innovation studies. These various
fields have their own methods and techniques. Across these fields, various methods and
techniques rely at least in part on mathematical and computer models.
We call decision support through models model-based decision support. The reason for using
models might be understood in light of the rise of Newtonian mechanics and its success in
predicting a wide array of phenomena. This success gave rise to a mechanistic worldview,
according to which the world is like a clock. If the mechanisms of the clock are known, any future
state of the clock can be predicted. Similarly, if the mechanisms underlying a phenomenon are
known, we can predict how this phenomenon will develop in the future. With the rise of
computers, more and more mechanisms can be codified into a model, and more and more
phenomena can be predicted.
However, the use of models to make predictions can be seriously misleading if there are
profound uncertainties. The system of planets is a relatively small system in terms of
components and can be very well observed, and thus its behaviour can be predicted with great
accuracy. However, for many other phenomena, such as the world’s climate, or systems in
which humans are involved, the situation is different. In these cases, there are many
components and mechanisms that interact in a variety of ways, and the system can only partly
be observed. The use of predictive models for such systems is problematic. There have been
scientists who have realized this. Some claim “the forecast is always wrong” (Ascher, 1978),
others say “all models are wrong” (Sterman, 2002), and yet again others speak of “useless
arithmetic” (Pilkey and Pilkey-Jarvis, 2007). Such comments raise the question whether models
can be used at all in decisionmaking under uncertainty.
In their agenda setting paper on FTA, Porter et al. (2004) note that “there are many irreducible
uncertainties inherent in the forces driving toward an unknown future beyond the short term and
predictions need not be assumed to constitute necessary precursors to effective action”. There
is a need for model-based support for the design of robust strategies across this spectrum of
irreducible uncertainties. The RAND Corporation developed a technique called Exploratory
Modelling and Analysis (EMA) tailored to this. This paper explores the potential of EMA for FTA.
It thus explicitly addresses one of the FTA challenges of Porter et al. (2004). Particular attention
is given to the potential of EMA in offering decision support for shaping systemic and structural
transformation.
This paper first discusses the wide ranging literature on uncertainty and uncertainty
classification, resulting in a typology of levels of uncertainty. This typology is used to specify
more clearly the types of uncertainty for which EMA is well suited. It is argued that EMA is
particularly well suited for model-based decision support given heterogeneous or even
conflicting information. Next, EMA is introduced and the steps of an EMA study are specified. To
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
-1-
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
further elucidate what EMA is and in order to assess the potential of EMA for FTA, three case
studies are reported on. Each of these cases is related to a grand societal challenge, is
grounded in a systems perspective, and aims to offer decision-support for crafting policies that
can shape and drive change. The paper will close with a discussion of the potential of EMA for
FTA in light of these cases. Section 2 present an uncertainty typology. Section 3 provides more
background on EMA. Section 4 contains three short cases. Section 5 contains the concluding
remarks and a discussion on the potential of EMA for FTA.
2 Classifying Uncertainty
To assess the extent to which models can be used for model-based decision support, it is
necessary to first specify in more detail what is meant with uncertainty. A variety of conceptual
schemes, definitions, and typologies of uncertainty have been put forward in different scientific
fields. For example, in risk analysis the distinction between aleatory and epistemic uncertainty
emerged (Hoffman and Hammonds, 1994, Helton, 1994). Epistemic uncertainty denotes the lack
of knowledge or information in any phase or activity of the modelling process. Aleatory
uncertainty denotes the inherent variation associated with the physical system or the
environment under consideration. Others have tried to clarify where uncertainty manifests itself
in the form of a source or location of uncertainty (e.g. Walker et al., 2003, Morgan and Henrion,
1990), and still others have tried to classify the severity of the uncertainty in the form of a level of
uncertainty (e.g. Courtney, 2001, van Asselt, 2000, Walker et al., 2003). That is, where does the
uncertainty manifest itself along the continuum ranging from deterministic knowledge to total
ignorance?
The question of where uncertainty manifests itself along the continuum ranging from
deterministic knowledge to total ignorance might very well have the longest history of the
different dimensions of uncertainty, dating back to philosophical questions debated among the
ancient Greeks about the certainty of knowledge and perhaps even further. Its modern history
begins around 1921 when Knight made a distinction between risk and uncertainty (Samson et
al., 2009, Rechard, 1999, Knight, 1921). More recent, various alternative conceptualizations
have been put forward (e.g. Makridakis et al., 2009, Kwakkel et al., 2010b, van Asselt, 2000,
Walker et al., 2003, Funtowicz and Ravetz, 1990, van der Sluijs, 1997, Petersen, 2006,
Courtney, 2001). A common theme in these various conceptualizations of the level of
uncertainty is the identification of more than two levels. For this paper, we define four levels of
uncertainty. The least uncertain is Level 1 uncertainty, or shallow uncertainty. In case of Level 1
uncertainty, probabilities can be used to specify the likelihood or plausibility of the uncertain
alternatives. In case of Level 2 uncertainty, or medium uncertainty, alternatives can be
enumerated and rank ordered in terms of their likelihood, but how much more likely or less likely
cannot be specified. This level of uncertainty is encountered when one is able to enumerate
alternatives and is able to say whether they are more likely, equally likely, or less likely, without
being able or willing to quantify this further. In case of Level 3 uncertainty, or deep uncertainty,
alternatives can be enumerated, but for various reasons, such as that decisionmakers or experts
cannot agree or don’t know, even a rank ordering is ruled out. The strongest form of uncertainty
is Level 4 uncertainty, or recognized ignorance. Here, alternatives cannot even be enumerated.
However, even when alternatives cannot be enumerated, merely keeping open the possibility of
being wrong or of being surprised is still possible. Table 1 shows the resulting levels of
uncertainty, their description, and some examples.
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
-2-
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
Table 1: The Four Levels of Uncertainty
Level of
Description
Examples
Being able to enumerate multiple alternatives
and being able to provide probabilities
(subjective or objective)
Being able to enumerate multiple alternatives
and being able to rank order the alternatives in
terms of perceived likelihood. However, how
much more likely or unlikely one alternative is
compared to another cannot be specified
Being able to enumerate multiple alternatives
without being able to rank order the
alternatives in terms of how likely or plausible
they are judged to be
Being unable to enumerate multiple
alternatives, while admitting the possibility of
being surprised
Being able to enumerate multiple possible
futures or alternative model structures, and
specify their probability of occurring
Being able to enumerate multiple possible
futures or alternative model structures, and
being able to judge them in terms of perceived
likelihood
Uncertainty
Level 1
(shallow uncertainty)
Level 2
(medium uncertainty)
Level 3
(deep uncertainty)
Level 4
(recognized
ignorance)
Being able to enumerate multiple possible
futures or specify multiple alternative model
structures, without being able to specify their
likelihood
Keeping open the possibility of being wrong or
being surprised
Extensive model-based techniques are available for handling Level 1 uncertainty. Exploring the
sensitivity of model outcomes to variations in the values of input parameters, is for example well
supported by Monte Carlo sampling and related techniques. Handling Level 2 and Level 3
uncertainty in model-based decision support is more limited. Here, analysts need to take
recourse to techniques such as scenario’s, Delphi, etc. Level 4 uncertainty is even more
problematic. How can we handle unknown unknowns? Adaptivity and flexibility in planning
appear to be the main techniques for preparing for this (Collingridge, 1980, Holling, 1978,
Albrechts, 2004, Erikson and Weber, 2008, Kwakkel et al., 2010a).
To what extent can models be used across the different levels of uncertainty? For level 1
uncertainty, there is not really a problem. Monte Carlo sampling and related techniques,
uncertainty propagation methods etc, offer the analyst ample opportunity to assess the
implications of uncertainties about the exact values of model parameters on outcomes. For level
2-4, the situation is different. Uncertainty about the future world can be addressed through for
example scenarios. These scenarios can be quantified and serve as input for model-runs. But
even these techniques are not free of problems. Goodwin and Wright (2010) (p. 355) argue that
“all the extant forecasting methods – including the use of expert judgment, statistical forecasting,
Delphi and prediction markets – contain fundamental weaknesses.” And Popper, et al. (2009)
state that the traditional methods “all founder on the same shoals: an inability to grapple with the
long-term’s multiplicity of plausible futures.” Moreover, if the uncertainty is not only about the
future, but is also about the models or aspects of the models, the situation becomes more
problematic. In addition, how are models to be used to support the development of flexible
adaptive plans that are robust across the uncertainties? This suggests that there is a need for a
methodology for using models for deeper levels of uncertainty (Porter et al., 2004).
3 Exploratory Modelling and Analysis
Various scientific fields are involved in providing model-based decision support. In these various
fields, people are grappling with the treatment of deeper levels of uncertainty while using
models. A common theme across these different sciences appears to be a shift away from
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
-3-
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
predictive use of models towards more explorative use of models (Pilkey and Pilkey-Jarvis,
2007, Sarewitz et al., 2000).
Exploratory Modelling and Analysis (EMA) is a research methodology that uses computational
experiments to analyse complex and uncertain systems (Bankes, 1993, Agusdinata, 2008).
Porter et al. (2004), in their agenda setting paper on FTA, explicitly mention EMA as being of
potential interest to FTA. To our knowledge, the potential of EMA for FTA has however not been
investigated yet. This paper can be seen as a (belated) attempt to rectify this.
EMA can be contrasted with consolidative modelling techniques in which known facts are
consolidated into a single model. Most models are intended to be predictive and use
consolidative modelling techniques, in which known facts are consolidated into a single ‘best
estimate’ model. The consolidated model is subsequently used to predict system behaviour
(Hodges, 1991, Hodges and Dewar, 1992). In such uses, the model is assumed to be an
accurate representation of that portion of the real world being analysed. However, the
consolidative approach is valid only when there is sufficient knowledge at the appropriate level
and of adequate quality available. Under deeper uncertainties, these conditions are not met.
Even if the consolidative modelling approach cannot be used, there is often a wealth of
information, knowledge, and data available that can be used to inform decisionmaking. For
example, through databases, scientific research, and in the form of mental models. EMA is a
research methodology that aims at utilizing the available information, knowledge, and data. EMA
specifies multiple models that are consistent with the available information. Instead of building a
single model and treating it as a reliable representation of the information, an ensemble of
models is created and the implications of these models are explored. A single model run drawn
from this set of models is not a prediction. Rather, it provides a computational experiment that
reveals how the world would behave if the assumptions any particular model makes about the
various uncertainties were correct. By conducting many such computational experiments, one
can explore the implications of the various assumptions. EMA aims at offering support for
exploring this set of models across the range of plausible parameter values and drawing
inferences from this exploration (Agusdinata, 2008, Bankes, 1993) that can be used for
decisionmaking, without falling into the pitfall of trying to predict that which is unpredictable.
EMA is not focused narrowly on optimizing a (complex) system to accomplish a particular goal or
answer a specific question, but can be used to address ‘beyond what if’ questions, such as
“Under what circumstances would this policy do well? Under what circumstances would it likely
fail?” It is exceptionally valuable in stimulating ’out of the box’ thinking and supporting the
development of adaptive plans. The typical steps in EMA are: (i) conceptualize the problem; (ii)
develop an ensemble of fast and simple model of the system of interest; (iii) specify the
uncertainties that are to be explored; (iv) specify a variety of policy options; (v) calculate and
compare the performance of the various options across the ensemble of future worlds; (vi)
iterate through steps (iii) to (v) until a satisfying policy option emerges. EMA is first and foremost
an alternative way of using models, knowledge, data, and information. Many well established
techniques, such as Monte Carlo sampling, factorial methods, and optimization techniques, can
be usefully and successfully employed in the context of EMA.
In this paper, we argue that by using models differently, the challenges associated with
decisionmaking under deep uncertainty can be overcome. Instead of trying to predict, the
models are used to explore what could happen across various uncertainties. In this way,
decisionmaking can proceed despite the presence of deep uncertainty, for decisions can be
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
-4-
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
designed to be robust across the explored range of possible futures. This exploratory way of
using models fits with the trend in foresight to move away from prediction and more towards
exploring what can happen. Exploratory modelling provides a rigorous methodology for exploring
the uncertainty space. Thus addressing one of the often mentioned shortcomings of foresight,
namely its impressionistic character (Erikson and Weber, 2008).
4 Illustrations of EMA
In this paper, EMA is illustrated using three cases. These cases differ in application domain, the
type of models used, and the purpose of the study. In this way, together these cases offer a
good overview of what EMA is about, what can be done with it, and what its potential is. Each of
the cases is related to important societal challenges. The first case explores uncertainties
related to the availability of minerals/metals that are crucial for the sustainable development of
all developed and developing societies. The second case shows how EMA can be used to
develop adaptive plans for guiding airport development. Airports are a major driver for regional
and national economic development. Future uncertainty is increasing because contextual
conditions are less stable, new technical solutions are emerging, and evaluation criteria are
contested (e.g. noise and emissions versus economic benefits) (Störmer et al., 2009). EMA
offers a suitable technique to explore the potential implications of these uncertainties and assists
in developing a plan that can adapt over time to how uncertainties unfold. The third case
presents an EMA study into transition pathways for the Dutch electricity system. Recent
contextual developments constitute a backdrop of change for the Dutch electricity system.
Institutional change driven by liberalization, changing economic competitiveness of the dominant
fuels, new technologies, and changing end-user preferences regarding electricity supply are
some examples of these developments. EMA is used to explore plausible transition trajectories
in the face of these developments given technological uncertainty about investment and
operating costs, and fuel efficiency of various alternative technologies; political uncertainty about
future CO2 abatement policies such as emission trading; and socio-economic uncertainty about
fuel prices, investment decisions of suppliers, and load curves.
4.1 Mineral scarcity
The first case explores uncertainties related to the availability of minerals/metals that are crucial
for the sustainable development of all developed and developing societies. Potential
mineral/metal scarcity poses a serious challenge for civil protection in at least three ways:
1. many crucial high-volume minerals are expected to become exhausted –or at least much
more expensive to mine– in the coming decades posing a threat to the sustainability of
current welfare levels;
2. the disparity between the expected exponential growth of metal demand and the
expected limited growth of metal supply (especially of crucial low-volume metals such as
rare earth metals that are required in ever bigger quantities for many innovative
technologies and electronics) may result in temporary and/or chronic scarcity; and
3. strategic and/or speculative behavior of countries that have a quasi-monopoly on the
extraction of (rare earth) metals may seriously hinder the transition of modern societies
towards more sustainable ones.
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
-5-
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
The asynchronous dynamics of supply and demand, aggravated by reinforcing behaviors and
knock-on effects, is a breeding ground for acute and/or chronic crises (Pruyt, 2010): how and
when these dynamics and crises may materialize remains highly uncertain. This case aims at
creating insight into the kinds of dynamics that can occur.
4.1.1 Model
Figure 1 displays the main feedback loops of a generic simulation model developed to create
insight into the types of dynamics that can occur. The two recycling loops displayed in red are
reinforcing loops. The linked extraction sector feedback loops are controlling loops. This
suggests that if recycling takes off, it is intrinsically limited by the extraction sector loops. These
extraction sector loops in turn are limited by the steeply increasing costs of extraction. More
details on the model can be found in Pruyt (2010). The model has been implemented as a
system dynamics model using Vensim software.
Figure 1: Causal loop diagram of the scarcity model (Pruyt, 2010)
4.1.2 Uncertainties
The future evolution of the extraction and recycling is intrinsically uncertain. The presented
model allows us to explore alternative evolutions across the various uncertainties. Table 2 gives
a high level overview of the key uncertainties that are taken into consideration. Note that we
explore across both parametric variations and what typically would be considered more
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
-6-
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
structural variations. The exploration is handled using the python programming software, utilizing
the Vensim DLL to parameterize the model, run the model and extract the results.
Table 2: High level uncertainties
Name
Description
Ranges
Parametric uncertainties
A wide variety of parametric uncertainties are
explored, including the lifetime of mines and
recycling facilities, the initial values, and
behavioral parameters such as price elasticity
and desired profit margins
There are various time delays, such as the
building of new recycling capacity and mines
There are various non-linear relations, modeled
with lookups. Examples include learning effects,
the impact of shortage on price, and
substitutions in case of shortages. These nonlinear relations are varied by changing the start,
end and slope
Typically plus and minus
50% of the default value
Orders of time delays
Non-linear lookups
First order, third order and
tenth order, 1000th
Start, end, slope
4.1.3 Analysis of Results
Figure 2 shows the dynamics for 7 different outcomes of interest. It illustrates clearly the wide
variety of dynamics that can occur. It also shows that under specific conditions cyclical
behaviour can emerge, with overshoots in supply, followed by undershoot, in turn causing a
highly dynamic market price.
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
-7-
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
Figure 2: Results of 100 runs
To investigate the emergence of crises and cycles in prices, we investigated the behaviour of
this one outcome in some more detail. Figure 3 shows the behaviour of the relative price for a
thousand runs. This figure shows even more examples of cyclical behaviour and it even appears
that the cycles can become worse over time. This display however also shows the need for
further analysis: the individual runs are difficult to trace in this plot. Therefore, there is a need for
data reduction techniques
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
-8-
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
Figure 3: Evolution of market price for 1000 runs
One way of analysing the results is to identify runs that share the same dynamic behaviour over
time. The behaviour over time can be understood as being a concatenation of atomic behaviour
patterns (Ford, 1999). By combining atomic behaviour patterns with the sign of the first order
derivate, six different behaviours are possible (i.e. positive logarithmic, negative logarithmic,
positive exponential, negative exponential, positive linear, negative linear). Each time series can
then be converted into a concatenation of these six patterns. Clustering then takes place on the
basis of the concatenations. If a hard clustering is used, which is to say that the entire
concatenation needs to be identical, Table 3 is the result. This table shows the wide variety of
behaviours that the model can generate by sampling across the various uncertainties.
More thorough analysis of the time series is still needed, for even roughly 6000 behaviours in
case of 50.000 runs is still unwieldy for supporting decision making. There is an emerging field
that studies the clustering of time series data. A wide variety of methods and techniques are
being explored (Liao, 2005). Application domains include chemical process monitoring and
control, and gene expression research. This area of research might contain useful techniques
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
-9-
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
for further reducing the results and supporting the interpretation. Along another dimension, more
research is currently on going for particular metals, such as copper, indium, and tantalum. The
purpose of these studies is to create insight into the possible dynamics and to explore possible
strategies to cope with or prevent certain dynamics. In this research, models are designed for
EMA.
Table 3: Identification of behaviors
Number of
runs
1000
Number of behavior
patters
371
5000
1214
10000
2042
15000
2742
20000
3386
25000
3894
30000
4547
35000
4976
40000
5511
45000
5972
50000
6404
4.2 Adaptive planning for Airport Development
The air transport industry operates in a rapidly changing context. Changes in ownership
structure, initiatives like Single European Sky and the Open Skies treaty between the United
States and Europe, the introduction of new aircraft such as the Airbus A380 and the Boeing 787,
and advances in Air Traffic Management (ATM) technology radically alter the functioning of the
sector. Airports are a crucial element in this system and are major drivers of regional and
national economies. Their long-term planning is therefore of crucial importance. Amsterdam
Airport Schiphol has been working over the last couple of years on a plan for guiding its longterm development. In this section, a stylized version of this decision making problem is explored.
4.2.1 Model
The uncertainty airport planners face is mainly located in the external environment affecting the
airport. The uncertainty about the internal workings of an airport are comparatively small.
Therefore, a single fast and simple model, utilizing existing tools for aspects of airport
performance calculations has been developed (Kwakkel, 2010). Table 4 gives an overview of
the components that make up this fast and simple model. This model is a general purpose
model, by parameterizing it for the specifics of a particular airport (runway locations, etc), the
performance of that airport can be calculated.
Table 4: Tools integrated in the Fast And Simple Model for Airport Performance Analysis
Airport Performance Aspect
Capacity
Tool
FAA Airfield Capacity Model (FCM) − an extension of the classic
Blumstein model (de Neufville and Odoni, 2003, FAA, 1981).
Area Equivalent Method (AEM ) a model that approximates
Integrated Noise Model results (FAA, 2008).
Noise
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
- 10 -
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
Emission Dispersion Modeling System (EDMS) − the FAA required
tool for emission analysis (FAA, 2009),.
Methodology developed by the National Air Traffic Services (NATS)
for third-party risk (Cowell et al., 1997, Cowell et al., 2000) − the
NATS methodology has been extended to apply to multiple runways
(Heblij, 2004, Heblij and Wijnen, 2008).
Emissions
Third Party Risk
4.2.2 Uncertainties
A wide variety of uncertainties are important in long-term airport planning. Table 5 gives an
overview of the major uncertainties that are explored in this case. For more details on the
parametric ranges of the various uncertainties see Kwakkel (2010). By picking a functional form
for each uncertainty, a scenario generator is specified. In total, 48 different generators are
possible. Each generator in turn has its own parametric ranges over which one can sample.
Table 5: Major uncertainties (adapted from Kwakkel, 2010)
Name
Demand
Wide body vs. narrow body aircraft mix
Population
ATM technology
Engine technology (noise/emissions)
Weather
Description
Change in demand, the curves can
be parameterized in various ways
Change in aircraft mix, the curves can
be parameterized in various ways
Change in population density, the
curves can be parameterized in
various ways
Change in air traffic management
technology, the curves can be
parameterized in various ways
Change in air traffic management
technology, the curves can be
parameterized in various ways
Percentage of change in days with
severe wind conditions per year. This
effects the availability of runway
configuration’s
Range
Exponential growth, logistic growth, or
logistic growth followed by logistic decline
Linear or logistic change
Logistic growth or logistic growth to a
maximum followed by logistic decline
Exponential or logistic performance
increase
Exponential or logistic performance
increase
-1% - +4%
4.2.3 Analysis of Results
One key challenge for airport planners is to design a plan for guiding the future developments of
the airport that is robust with respect to the future (de Neufville and Odoni, 2003). For
Amsterdam Airport Schiphol the design of such a robust plan is particularly challenging because
its serves as a secondary hub of KLM-Air France, implying that this airline can move operations
to its primary hub, Charles de Gaul, easily. Moreover, Schiphol is located in a wind prone area,
necessitating a runway layout that covers the various wind directions. At the moment, Schiphol
is considering expanding the airport by adding a new runway that is to become operational in
2020. Moreover, a participatory process has resulted in the agreement that no more than
510.000 operations can be scheduled at Schiphol in 2020. Up to 70.000 short haul operation are
to be relocated from 2015 onwards to the existing airport Eindhoven, and Lelystad Airport which
is to be developed in the coming years.
Using a conjugant gradient optimization algorithm, the lower bound and the upper bound for
each of the performance indicators can be calculated. These bounds are calculated across the
48 scenario generators and their associated parameter ranges. The column ‘static plan’ in Table
6 shows the results of this analysis. Looking at the various outcome indicators, and in particular
the ratio capacity to demand, it is clear that the outlined plan does not succeed in robustly
guiding the future development of the airport. There is the potential for significant under use of
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
- 11 -
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
the provided infrastructure, or a significant overshoot in the negative external effects (noise,
external safety and emissions).
Table 6: Performance bounds of the static and the adaptive plan
Outcome indicators
Size of noise contour after thirty years (km2)
Cumulative Average Casualty Expectancy (ACE)
Ratio practical capacity to demand after 30 years
Maximum ratio practical capacity to demand
Accumulated latent demand (flights)
Cumulative CO emission (kg)
Static Plan
13.2-63.8
0.9-2.7
0.25-2.48
0.9-2.48
0-5,058,504
21,520.9-195,729.1
Adaptive Plan
10.2-47.4
1.1-2.3
0.89-1.1
0.52-1.1
0-8,290,622
19,773.9-103,899.5
A sensible redesign of the plan would be to make it dynamically adaptive (de Neufville and
Odoni, 2003, Kwakkel et al., 2010a). The construction of a new runway and the moving of
operations are in this approach not planned for a particular moment in time, but are triggered by
the evolution of external conditions. Thus, a new runway is only built if there is sufficient
demand, or problems with wind conditions. However, preparatory actions, such as land use
reservations, designs for the runway, etc. are taken. Minimizing the time required to realize the
change. To address the potential overshoot of negative external affects, this modified dynamic
adaptive plan is complemented with a stricter slot allocation mechanism. The column ‘adaptive
plan’ in Table 6 shows the performance bounds for this redesigned plan. It is clear that this plan
has a much narrower bandwidth in expected outcomes across all the uncertainties. It is thus
better able to guide the future developments of the airport in light of the uncertainties.
To investigate in more detail the difference in performance between both plans, and to create
insight into the specific range of conditions under which one plan would perform better than the
other, Figure 5 has been generated. The performance difference is calculated using the
Euclidian norm and the normalized performance vectors for each plan (Kwakkel, 2010). This
figure shows that under the conditions most favourable for the static plan, if demand is high
and/or the increase of aircraft size is high, there is a big advance in using the adaptive plan.
Conversely, if the growth of demand is minor, the static plan performs slightly better. This figure
could serve as a starting point for slightly modifying the outlined dynamic adaptive plan, for
example by modifying the exact values that are used to trigger actions, or by modifying the
stricter slot allocation regime.
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
- 12 -
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
Figure 4: Performance difference of the Adaptive Plan compared to the Static Plan for that
combination of uncertain parameters that most favor the Master Plan. Below 0 the static
plan is better, above 0 the Adaptive Plan performs better
4.3 Identification of plausible transition pathways for the future Dutch electricity
generation system
Recent contextual developments constitute a backdrop of change for the Dutch electricity
system. Institutional change driven by liberalization, changing economic competitiveness of the
dominant fuels, new technologies, and changing end-user preferences regarding electricity
supply are some examples of these developments. In this case, we explore plausible transition
trajectories in the face of these developments given technological uncertainty about investment
and operating costs, and fuel efficiency of various alternative technologies; political uncertainty
about future CO2 abetment policies such as emission trading; and socio-economic uncertainty
about fuel prices, investment decisions of suppliers, and load curves. Various alternative
developments for these uncertainties are specified. The consequences of each of these
alternative developments is assessed using an agent-based model (Yücel, 2010) of the Dutch
electricity system. The outputs are analysed using various data-mining and data visualization
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
- 13 -
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
techniques in order to reveal arch-typical transition trajectories and their conditions for occurring.
Policy recommendations are derived from this.
4.3.1 Model
ElectTrans is an agent-based simulation model, which explicitly focuses on multiple actor groups
within the electricity system, most importantly the end users and the generation companies. Four
groups of end-users are represented in the model, which are industrial users, commercial users,
horti-/agricultural users, and households. It is possible to name two major supply options for all
actor groups, i.e. using electricity supplied via central generation, and adoption of distributed
generation options for self-generation. There are two grid-based options in the model: gray
electricity and green electricity. Various distributed generation options are also available, such
as wind turbines and gas engine CHPs. Generation companies are mainly responsible for shortterm operation, as well as long-term management of their generator park. The short-term
operation involves unit commitment decisions, and price bidding in the electricity market, which
are directly related to load dispatching to take place in the market. The long-term decisions are
related to capacity investment and decommissioning. Decommissioning decisions are mainly
based on expected lifetime of the technology used in a generation unit. A unit at the end of it is
lifetime, and/or an old unit making loss can be decommissioned. Generation companies’
expansion decisions are mainly driven by profit expectations, and are dependent on forecasts
about fuel prices, demand, active generation capacity connected to the grid, and feasible
investment options. A more detailed description of the ElectTrans can be found in (Yücel, 2010).
4.3.2 Uncertainties
Table 7 presents an overview of the uncertainties that are explored in the EMA study. In total, 13
uncertainties are explored across the specified range. Most of uncertainties are multiplier factors
that will be used to alter the base value of the corresponding parameters. For example, assume
the investment cost of wind turbine is 100, and the Investment Cost Factor for wind is 0.8; then
the model will be initialized with an investment cost of 100x0.8=80. To be more precise, it is not
the initial investment cost we are altering, but the expected future investment cost towards which
the option evolves during the time horizon of the simulation.
Table 7: The major uncertainties and their ranges
Name
Investment Cost Factor
operational Cost Factor
Coal and Gas Price
Increase Percentage
Demand Growth Fraction
Load Slope Change
Fraction
Planning Horizon of the
generation companies
Mean return on
investment of generation
companies
Description
Multiplier factor to alter the future investment cost of new generation
options
Multiplier factor to alter the future variable operating costs of a
technology
Yearly fractional increase in coal prices
Range
Yearly fractional change in the demand of end users
Yearly fractional change in the slope of the load-duration curve
0-0.03
Upper bound for the planning horizon of the generation companies.
(Planning horizon for each generation company is randomly initialized
using a uniform distribution with a lower (i.e. 5 years) and an upper
bound)
Average expected return-on-investment for the generation companies
0.6-1.25
0.6-1.25
0.002-0.03
-0.01-0.01
6-12
0.1-0.25
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
- 14 -
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
4.3.3 Analysis of Results
Figure 5 shows a performance envelope for five outcome indicators.

Total generation: Total amount of electric energy generated

Total fossil: Total amount of electric energy generated using fossil fuels as the energy
source

Total non-fossil: Total amount of electric energy generated using non-fossil fuels as the
energy source (renewables and nuclear)

Capacity: Total installed power generation capacity

Price: Average price of electricity on the central grid
The figure shows the upper and lower bounds that are encountered across the 15.000 runs. The
figure also shows the distribution of outcomes at the end of the runtime. It appears from these
results, that there is a limited development of non-fossil generation, suggesting that under most
uncertainties a transition towards more sustainable generation does not take place.
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
- 15 -
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
Figure 5: Performance envelopes and distribution of end states for five outcome
indicators.
To provide insight into how the various uncertainties jointly determine outcomes, a classification
tree was made based on the results. Classification trees are a frequently employed data mining
technique. They are used to predict class membership based on a set of attributes. In the
context of this paper, we used the 13 uncertainties as attributes. As class we used the terminal
value for the fraction of fossil fuel-based generation. This terminal value was split. If it was lower
or equal to 0.6, it is coded as 0, else it is codes as 1. Figure 6 shows a classification tree that
results from this analysis. The tree was generated using the open source data mining package
Orange. This is a C++ library with python bindings to many useful data mining and machine
learning algorithms.
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
- 16 -
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
Figure 6: Classification tree of fraction fossil-based generation
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
- 17 -
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
5 Concluding remarks
In this paper, we have introduced EMA. EMA is an approach for using models under conditions
of deep uncertainty. Instead of using a model predictively, EMA aims at developing an ensemble
of models that is consistent with the available information, data, and knowledge and at exploring
the implications of this ensemble. EMA allows for a systematic rigors exploration of the
uncertainties that are present.
We illustrated EMA using three cases. These cases differed in the modelling paradigm that was
being used, in the application domain, and in the type of problem being investigated. The first
case showed how EMA can be combined with system dynamics to investigate the types of
behaviour that can occur with respect to rare earth metals and minerals shortages. In this case
both parametric and some structural uncertainties were being taken into account. The case also
showed that further research on time-series clustering is necessary to facilitate the interpretation
of results. For FTA, the case illustrates how EMA can be used to explore a wide variety of
uncertainties and assess their joint implications. If a high tech company is dependent on rare
earth minerals, the results of the case could be used to identify early indicators of for example
cyclical pricing behaviour.
The second case used an ensemble of hybrid models to facilitate the design of a good plan for
shaping and guiding the future development of an airport. The case illustrated how through the
use of non-linear optimization techniques a performance bandwidth could be established across
all the uncertainties. This bandwidth showed unacceptable behaviour of the basic plan,
necessitating the redesign of the plan. A basic redesign, drawing on adaptivity and flexibility
already showed a significant shrinkage of the performance bandwidth. To investigate the
conditions under which one plan would outperform the other, a further analysis was presented.
These results could be used for further improve the plan. For FTA, this case illustrates how EMA
can help in developing plans that are robust across the various uncertainties that are affecting
the decision making problem.
The third case illustrated how EMA can be combined with agent based models. In the case, we
investigated the transition patterns that could occur and which combinations of uncertainties
resulted in which transition pattern. In particular the use of the classification tree in order to
create insight into how uncertainties are mapped to classes of outcomes is useful here. For FTA,
this case illustrates how the results of a significant number of runs can be converted into a tree
that can usefully be communicated to a decision maker.
The techniques used in each of the tree cases do not exclude each other. Classification trees
can for example be used to define more precisely trigger values for an dynamically adaptive plan
or the tree can be used to understand which type of behaviour pattern emerges under which
combination of uncertainties. The three cases also illustrate the need for combining EMA with
machine learning or data mining techniques, for otherwise, the problem of incompletely taking
into account uncertainty is being replaced by an information overload problem.
References
AGUSDINATA, D. B. 2008. Exploratory Modeling and Analysis: A promising method to deal with
deep uncertainty. Ph.D. thesis Ph.D. thesis, Delft University of Technology.
ALBRECHTS, L. 2004. Strategic (spatial) planning reexamined. Environment and Planning B:
Planning and Design, 31, 743-758.
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
- 18 -
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
ASCHER, W. 1978. Forecasting: An Appraisal for Policy Makers and Planners, Baltimore, Johns
Hopkins University Press.
BANKES, S. 1993. Exploratory Modeling for Policy Analysis. Operations Research, 4, 435-449.
COLLINGRIDGE, D. 1980. The Social Control of Technology, London, UK, Frances Pinter
Publisher.
COURTNEY, H. 2001. 20|20 Foresight: Crafting Strategy in an Uncertain World, Boston,
Harvard Business School Press.
COWELL, P. G., FOOT, P. B., GERRARD, R. J., KENT, D., MANSON, S. M. & RIVOIRE, A.
2000. A methodology for calculation individual risk due to aircraft accidents near airports.
London, Uk.: National Air Traffic Services Ltd.
COWELL, P. G., GERRARD, R. & PATERSON, D. S. 1997. A Crash Location Model for Use in
the Vicinity of Airports. London, UK: National Air Traffic Services Ltd.
DE NEUFVILLE, R. & ODONI, A. 2003. Airport Systems: Planning, Design, and Management,
New York, McGraw-Hill.
ERIKSON, E. A. & WEBER, K. M. 2008. Adaptive Foresight: navigating the complex landscape
of policy strategies. Technological Forecasting and Social Change, 75, 462-482.
FAA 1981. Upgraded FAA Airfield Capacity Model, Washington D.C., Federal Aviation
Administration.
FAA 2008. FAA Aerospace Forecast Fiscal Years 2008-2025, U.S. Department of
Transportation, Federal Aviation Administration.
FAA 2009. Emissions and Dispersion Modeling System User's Manual. Washington, DC:
Federal Aviation Administration, Office of Envronment and Energy.
FORD, D. N. 1999. A behavioral approach to feedback loop dominance analysis. System
Dynamics Review, 15, 3-36.
FUNTOWICZ, S. O. & RAVETZ, J. R. 1990. Uncertainty and Quality in Science for Policy,
Dordrecht, Kluwer Academic Publishers.
GOODWIN, P. & WRIGHT, G. 2010. The limits of forecasting methods in anticipating rare
events. Technological Forecasting and Social Change, 77, 355-368.
HELTON, J. C. 1994. Treatment of Uncertainty in Performance Assessment for ComplexSystems. Risk Analysis, 14, 483-511.
HODGES, J. S. 1991. Six (or so) Things You Can Do With a Bad Model. Operations Research,
39, 355-365.
HODGES, J. S. & DEWAR, J. A. 1992. Is it You or Your Model Talking? A Framework for Model
Validation. Santa Monica: RAND.
HOFFMAN, F. O. & HAMMONDS, J. S. 1994. Propagation of Uncertainty in Risk Assessments:
the Need to Distinguish between Uncertainty Due to Lack of Knowledge and Uncertainty
Due to Variability Risk Analysis, 14, 707-712.
HOLLING, C. S. 1978. Adaptive Environmental Assessment and Management, New York, John
Wiley & Sons.
KNIGHT, F. H. 1921. Risk, Uncertainty and Profit, Mineola, New York, Dover Publication.
KWAKKEL, J. H. 2010. The Treatment of Uncertainty in Airport Strategic Planning. PhD, Delft
University of Technology.
KWAKKEL, J. H., WALKER, W. E. & MARCHAU, V. A. W. J. 2010a. Adaptive Airport Strategic
Planning. European Journal of Transportation and Infrastructure Research, 10, 227-250.
KWAKKEL, J. H., WALKER, W. E. & MARCHAU, V. A. W. J. 2010b. Classifying and
communicating uncertainties in model-based policy analysis. International Journal of
Technology, Policy and Management, 10, 299-315.
LIAO, T. W. 2005. Clustering of time series data - a survey. Pattern Recognition, 38, 1857-1874.
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
- 19 -
Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
SEVILLE, 12-13 MAY 2011
MAKRIDAKIS, S., HOGARTH, R. M. & GABA, A. 2009. Forecasting and uncertainty in the
economic and business world. International Journal of Forecasting, 24, 794-812.
MORGAN, M. G. & HENRION, M. 1990. Uncertainty: A Guide to Dealing with Uncertainty in
Quantitative Risk and Policy Analysis, Cambridge, Cambridge University Press.
PETERSEN, A. C. 2006. Simulating nature: A philosophical study of computer-simulation
uncertainties and their role in climate science and policy advice. Ph.D. Ph.D. thesis,
Universiteit Amsterdam.
PILKEY, O. H. & PILKEY-JARVIS, L. 2007. Useless Arithmetic: Why Environmental Scientists
Can't Predict the Future, New York, USA, Columbia University Press.
POPPER, S., GRIFFIN, J., BERREBI, C., LIGHT, T. & MIN, E. Y. 2009. Natural Gas and Israel’s
Energy Future: A Strategic Analysis Under Conditions of Deep Uncertainty. Santa
Monica, California: RAND.
PORTER, A. L., W.B., A., CLAR, G., COATES, J. F., CUHLS, K., CUNNINGHAM, S. W.,
DUCATEL, K., VAN DER DUIN, P., GEORGEHIOU, L., GORDON, T., LINDSTONE, H.,
MARCHAU, V. A. W. J., MASSARI, G., MILES, I., MOGEE, M., SALO, A., SCAPOLO, F.,
SMITS, R. E. H. M. & THISSEN, W. 2004. Technology Futures Analysis: Twards
integration of the field and new methods. Technological Forecasting and Social Change,
71, 287-303.
PRUYT, E. 2010. Scarcity of Minerals and Metals: A Generic Exploratory System Danamics
Model. In: MOON, T. H. (ed.) The 28th International Conference of the System Dynamics
Society. Seoul, Korea.
RECHARD, R. P. 1999. Historical Relationship between Performance Assessment for
Radioactive waste Disposal and Other Types of Risk Assessment. Risk Analysis, 19,
763-807.
SAMSON, S., RENEKE, J. A. & WIECEK, M. W. 2009. A Review of Different Perspectives on
Uncertainty and Risk and an Alternative Modeling Paradigm. Reliability Engineering and
System Safety, 94, 558-567.
SAREWITZ, D., PIELKE, R. A. & BYERLY, R. (eds.) 2000. Prediction, Washington, D.C.: Island
Press.
STERMAN, J. D. 2002. All models are wrong: reflections on becoming a systems scientist.
System Dynamics Review, 18, 501-531.
STÖRMER, E., TRUFFER, B., DOMINGUEZ, D., GUJER, W., HERLYN, A., HIESSL, H.,
KASTENHOLZ, H., KLINKE, A., MARKARD, J., MAUER, M. & RUEF, A. 2009. The
exploratory analysis of trade-offs in strategic planning: Lessons from Regional
Infrastructure Foresight. Technological Forecasting and Social Change, 76, 1150-1162.
VAN ASSELT, M. B. A. 2000. Perspectives on Uncertainty and Risk, Dordrecht, Kluwer
Academic Publishers.
VAN DER SLUIJS, J. P. 1997. Anchoring Amid Uncertainty: On the Management Of
Uncertainties in Risk Assessment of Anthropogenic Climate Change. Ph.D. Ph.D. thesis,
University of Utrecht.
WALKER, W. E., HARREMOËS, J., ROTMANS, J. P., VAN DER SLUIJS, J. P., VAN ASSELT,
M. B. A., JANSSEN, P. H. M. & KRAYER VON KRAUSS, M. P. 2003. Defining
Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision
Support. Integrated Assessment, 4, 5-17.
YÜCEL, G. 2010. Analyzing Transition Dynamics. PhD, Delft University of Technology.
THEME: BUILDING FTA CAPACITIES FOR SYSTEMIC AND STRUCTURAL TRANSFORMATIONS
- 20 -