Santos: IIM with Multiple Experts
Submitted to IIOA
Inoperability Input-Output Model (IIM) with Multiple Probabilistic
Sector Inputs
Joost R. Santos
Department of Systems and Information Engineering, University of Virginia, Charlottesville, Virginia
22904, jrs8e@virginia.edu
The increasing degree of interdependencies among sectors of the economy can likely make the impacts of natural and
human-caused disruptive events more pronounced and far-reaching than before. An extended input-output model is implemented in this paper to analyze risk scenarios to a particular sector and to estimate the resulting ripple effects to other sectors. The proposed extension is capable of combining likelihood and consequence estimates from multiple experts, which incorporates traditional expected value and extreme-event measures of risk. The probability densities of
ripple effects are generated via Monte Carlo simulation; hence, providing estimates of the mean and extreme values of
economic losses and corresponding levels of sector disruptions. In investing for additional airline security, for example,
the “breakeven” level of investment cost should account for the potential consequences associated with both “average”
and “worst-case” scenarios. Ultimately, the ranking of the sectors that are most critically affected by a given disruptive
event can provide guidance in identifying resource allocation and other risk management strategies to minimize the
overall impact on the economy. The methodology is demonstrated through an air transportation sector case study.
Key words: {input output analysis}, {decision and risk analysis}, {cost-benefit analysis}, {probability and distribution
comparisons}, {simulation and statistical analysis}
____________________________________________________________________________________
1. Introduction
Decision-making processes at any level of a hierarchical system involve management of multiple objectives (costs, benefits, and risks). The US is a large-scale system that is comprised of interdependent sectors such as manufacturing, transportation, service, commerce, and workforce. The nation is a hierarchical
system as it involves several layers of management authorities ranging from national policymakers to operators of specific critical infrastructure subsystems. The economic sectors in the US (and the entire global economy) are becoming more and more interdependent—for example; different organizational functions (e.g., supply chain, accounting, inventory management, customer support, etc.) now heavily depend
on the cyberspace to enhance business efficiency. Furthermore, these functions are oftentimes subcontracted to external firms, which can add a business-to-business layer of interdependence in addition to
other existing interdependencies (cyber and physical). Given this premise, the increasing degree of inter-
1
Santos: IIM with Multiple Experts
Submitted to IIOA
dependencies among the economic sectors can likely make the impacts of natural and human-caused disruptive events more pronounced and far-reaching than before. Such disruptive events upset the “businessas-usual” production levels of the affected systems and lead to a variety of economic losses, such as demand/supply reductions.
Using the North American Industry Classification System (NAICS) for input-output accounting,
this paper focuses on the interdependencies that exist at the macroeconomic level of the US economy.
The premise is that an “attack” to a particular sector can render adverse ripple effects to other sectors depending on the degree of interdependencies. The inoperability input-output model (IIM) is implemented
in this paper to analyze risk scenarios to a particular sector (i.e., air transportation) and to estimate the
resulting ripple effects to other sectors. In previous applications of the IIM, the disruption inputs (e.g.,
demand reductions) to initially-affected sectors were typically expressed as single-point estimates (in contrast to a range of values). Hence, the paper develops an extension to the IIM that allows an expert to
specify initial disruptions to a particular sector in the form of a probability distribution. In addition, the
proposed extension is capable of combining probability distribution inputs from multiple experts to enhance the credibility of the initial estimates. A mixture distribution (or “effective” distribution) is then
generated from the individual distributions elicited from multiple experts. This mixture allows the computation of statistics such as the expected value (or mean) and the conditional expected value of extreme
risk. The resulting expressions for the expected and conditional expected values of risk are rigorously derived in the paper. A partition of the resulting mixture distribution, usually the upper-tail region, reflects
those events with low-likelihoods, but with extreme outcomes. Both the classical definition of expected
value (which represents “normal” events) and the conditional expected value (extreme events) can provide a holistic insight into the uncertain outcomes of a disruptive event. In investing for additional airline
security, for example, the breakeven point for investment cost should account for the potential costs associated with both “average” and “worst-case” scenarios.
Given the distribution of direct effects to a particular sector, the IIM can be implemented to determine the ripple effects to other sectors using the “inoperability” metric. Inoperability is a measure of
2
Santos: IIM with Multiple Experts
Submitted to IIOA
risk (expressed in %) that indicates the extent to which a given sector deviates from its nominal production level due to a disruptive event. The IIM has the capability to pinpoint the ordinal ranking of adverse
effects to the interdependent sectors of the economy. In addition to the ordinal ranking, the proposed
Monte Carlo simulation algorithm enables the generation of the actual distribution of inoperability estimates for the most-affected sectors.
The remainder of the paper is organized as follows. Section 2 describes the overall research
framework that includes overview of risk analysis, IIM background, extreme-event analysis, and expert
elicitation. Section 3 discusses a method for combining multiple expert distributions and provides the detailed derivations for expected (f5) and conditional expected (f4) values. Section 4 demonstrates how the f5
and f4 values can be calculated using triangular and beta distributions. Section 5 shows an IIM-based case
study that illustrates demand reduction scenarios to the air transportation sector and develops a Monte
Carlo simulation algorithm to forecast the resulting effects to other sectors of the US economy. Finally,
Section 6 summarizes the paper and enumerates areas for future research.
2. Methodological Framework
Figure 1 depicts the methodological framework employed in this paper. The left side of the figure highlights a procedure for pooling probability distributions from multiple experts, where measures of risk such
as expected and conditional expected values can be subsequently calculated. The principles of risk analysis (Section 2.1) should serve as a guide in the generation of a probability distribution, which encompasses the enumeration of the possible consequences and their associated likelihoods. The right side of
the figure shows a schematic of sector interdependency modeling using the IIM (Section 2.2). A method
for combining multiple expert assessments is developed using probabilistic and extreme-event analysis
tools (Section 2.3). This method incorporates a computer module that allows multiple experts to input
their likelihood and consequence ratings (Section 2.4), which can then be fed to the IIM module.
3
Santos: IIM with Multiple Experts
Submitted to IIOA
Mixture of Probability Distributions from
Multiple Experts & Extreme-Event Analysis
Inoperability Input-Output Modeling with
Monte Carlo Simulation
Ripple Effects to
Other Sectors
Expert 1
Effective PDF
Expert 1
Expert 2
Direct Effect
to Sector k
Random
Sample
Expert 2
I
N
O
P
E
R
A
B
I
L
I
T
Y
N
Sample
Size
Achieved
?
Y
Figure 1. Methodological framework for the IIM with multiple probabilistic inputs
2.1 Risk Analysis
Risk connotes an outcome of any deviations relative to a predefined normative state of a system; hence
the process of risk analysis is an important component of any decision-making processes. In project management, for example, risk takes the form of schedule delays and cost overruns. Broadly defined,
Lowrance (1976) defines risk as a function of the likelihood of an unwanted event (e.g., disaster) and the
severity of potential consequences. From the perspective of policymakers dealing with homeland security,
risk can be described as the outcome of a threat being applied to a vulnerable system (or subsystem) resulting in adverse consequences. In preparing for any disasters, decision-makers need to consider the two
phases of risk analysis. The first phase is risk assessment, which aims to answer the three questions: (i)
What can go wrong? (ii) What is the likelihood? and (iii) What are the consequences? (Kaplan and Garrick 1981). The second phase deals with risk management, which aims to address the next three questions: (i) What can be done and what options are available? (ii) What are the tradeoffs in terms of costs,
benefits, and risks? and (iii) What are the impacts of current decisions on future options? (Haimes 2004).
In the current environment of heightened threat to homeland security, the six questions of risk
analysis can provide insights into policy formulation geared towards disaster preparedness. The Homeland Security Council (2004) identifies the 15 planning scenarios to create a strong risk analysis focus on
catastrophic events. The expected value is a commonly-used metric for estimating risk; however it tends
4
Santos: IIM with Multiple Experts
Submitted to IIOA
to discount catastrophic risks because of their relatively low likelihoods. In this paper, we supplement the
expected value metric with an extreme-event metric that permits the inclusion of a risk parameter for partitioning high consequence/low probability events. Extreme events typically have low historical precedence and sparse data availability; hence risk analysts would typically resort to expert elicitation.
Another important aspect of risk analysis relates to the decomposition of consequences. In the
context of this paper, the consequences of a disruptive event to a particular segment of the economy will
cascade to the entire nation due to interdependencies. The interdependency modeling feature of the IIM
can provide insights into the distribution of the likely ripple effects of a targeted attack to a particular sector. Key sector analysis can be based on the ranking of critical sector effects, which can ultimately provide guidance in identifying resource allocation and other risk management strategies to minimize the
overall risk to the economy.
2.2 Inoperability Input-Output Model (IIM)
Presently, the economic sectors in the US and across the entire global economy are becoming more and
more interdependent; hence the consequences due to natural and human-caused disruptive events can be
more pronounced and far-reaching than before. The IIM was developed as an interdependency analysis
tool for the assessment of the ripple effects triggered by various sources of disruption, including terrorism, natural calamities, and accidents. Previous IIM-based works on infrastructure interdependencies and
risks of terrorism include Santos and Haimes (2004), Haimes et al. (2005a; 2005b), Crowther and Haimes
(2005), Lian and Haimes (2006), and Santos (2006).
The IIM was developed as an extension of Wassily Leontief’s input-output (I-O) model (Leontief
1936). Publications on the I-O model and its applications are highly transparent in the literature (see Miller and Blair (1985)). The I-O model has been used in a highly diverse set of applications including (but
not limited to) assessment of economic interdependencies (Midmore et al. 2006), environmental modeling
(Hoekstra and Janssen 2006), and disaster impact analysis (Cho et al. 2001, Rose 2004).
5
Santos: IIM with Multiple Experts
Submitted to IIOA
The formulation of the IIM is as follows:
q = A*q + c* = (I–A*)-1c*
(1)
The details of model derivation and discussion of the model components are found in Santos and
Haimes (2004). In summary, the terms in the IIM formulation in Eq. (1) are defined as follows:

q is the inoperability vector expressed in terms of normalized economic loss. The elements of q
represent the ratio of unrealized production (i.e., “business-as-usual” production minus degraded
production) with respect to the “business-as-usual” production level of the industry sectors;

A* is the interdependency matrix which indicates the degree of coupling of the industry sectors.
The elements in a particular row of this matrix can tell how much additional inoperability is contributed by a column industry to the row industry; and

c* is a demand-side perturbation vector expressed in terms of normalized degraded final demand
(i.e., “business-as-usual” final demand minus actual final demand, divided by the “business-asusual” production level).
In previous applications of the IIM, we assumed that the vector c* is composed of elements that
are constants. For example, a 20% demand reduction scenario for the air transportation (Sector 29) will
correspond to a perturbation value of c*29 = 0.2, and all the rest of the elements are zeroes if we assume
that the air transportation sector is the only directly-affected sector. For the current paper, we explore the
uncertainty in the specification of the perturbation input by considering probability distributions, instead
of constants. Furthermore, the paper also examines the possibility of eliciting probability distributions
from more than one expert. The resulting “effective” distribution is aggregated based on the inputs from
multiple experts, and then fed to the model in (1). This will result in inoperability values (q) for the interdependent sectors that also follow the form of probability distributions. Subsequently, the rankings in
terms of the inoperability metric can then be represented as a range of possible values, from where we can
calculate statistics such as averages and extreme-event values.
6
Santos: IIM with Multiple Experts
Submitted to IIOA
2.3 Extreme-Event Analysis
The Partitioned Multi-objective Risk Method (PMRM) is a specific extreme-event analysis tool that has
the capability to analyze various scenarios (e.g., average vs. extreme scenarios) using conditional expectations from a given distribution. A conditional expectation refers to the expected value of the possible realizations of a random variable within a prespecified interval (Asbeck and Haimes 1984, Haimes 2004). In
survival analysis literature, the conditional expected value of a random variable within an upper-tail partition is typically referred to as the mean expected life (Klein and Moeschberger 1997). In finance, the term
“conditional value-at-risk” (CVaR) is used to refer to the lower-tail conditional expectation of potential
portfolio losses (Rockafellar and Uryasev 2000). In this paper, we use the notation f4 to denote the conditional expectation of a random variable within a prespecified upper-tail partition—encompassing events
that have catastrophic effects, albeit the low likelihoods. The conditional expectations used in the PMRM
can effectively distinguish low-consequence/high-probability events from high-consequence/lowprobability events (i.e., extreme events). The upper-tail conditional expectation can complement and supplement the commonly used measure of central tendency—the expected value or mean. For a given probability distribution f(x), the expected value and conditional expected value (denoted by f5 and f4, respectively) are defined as follows.

f 5   xf ( x)dx
(2)


 xf ( x)dx
f4 

(3)

 f ( x)dx

The  in equation (3) is a specified upper-tail partitioning along the x-axis (i.e., x<+ ), which
corresponds to an exceedance probability of  (i.e., Pr(x>β)=). The function f(x) appearing in (2) and (3)
denotes a probability density for demand-based perturbations (i.e. f( ci* ), where ci* is an element of the
vector c* in (1)).
7
Santos: IIM with Multiple Experts
Submitted to IIOA
2.4 Expert Elicitation
When eliciting information from experts, using probability distributions (as opposed to single-point estimates) allows an analyst to factor uncertain components of a problem into a mathematical model. The
importance of expert elicitation is well-documented in the literature. O’Hagan and Oakley (2004) described various sources of uncertainty (parameter, model, variability, and code) and related these sources
to the two main categories of uncertainty (aleatory and epistemic). They asserted that instead of finding
alternative ways to measure uncertainty (aside from the probability concept), attention can be put more
productively on enhancing elicitation methods. Hammitt and Shlyakhtel (1999) described the importance
for expert elicitation in the context of data-sparse applications and using the expected value of information, they presented a process for determining the need to update prior distributions to capture new
information. In the context of unsafe human actions, Forester et al. (2004) proposed an approach for eliciting probabilities that accounts not only the actual elicited probability value(s) but also the experts’ experience and the relevance of the information that they provide to the failure scenario that is being studied.
Cagnoa et al. (1999) explored the use of the analytic hierarchy process (AHP) for eliciting gas pipeline
failure distributions from experts and presented a Bayesian approach to integrate historical data.
3. Combination of Probability Distribution Functions from Multiple Experts
Garthwaithe et al. (2005) discussed issues surrounding the process of expert elicitation and how a risk
analyst can fit the resulting expert estimates into probability distributions. Several types of probability
distributions and regression methods were discussed in their paper as well as cases that require pooling of
multiple expert probability distributions. Lipscomb et al. (1998) implement a Bayesian hierarchical approach for pooling expert-specified distributions of service times across several medical centers, which
can be used for estimating staffing requirements. Clemen and Winkler (1999) discussed the importance of
obtaining probability distributions from multiple experts as this leads to a more robust risk analysis; hence
integrating the expert-elicited information using aggregation and behavioral approaches becomes neces-
8
Santos: IIM with Multiple Experts
Submitted to IIOA
sary. They argue that the linear approach for aggregating expert-elicited probabilities gives comparable
results than the more complex mathematical approaches (e.g. Bayesian models).
3.1 Effective Probability Distribution Function from Multiple Experts
From this point forward, we use the additive pooling technique in deriving the effective distribution that
would result from combining multiple expert distributions. We use the advantages and properties pointed
out by Clemen and Winkler (1999) in choosing the additive technique including its simplicity, relatively
smaller number of parameter requirements, and comparability of results with the more sophisticated techniques (e.g., multiplicative and Bayesian pooling).
Let pi (x) be the probability distribution function (pdf) elicited from expert i. Also, let  i be the
N
relative credibility (or weight) of the evidence from expert i such that

i 1
i
 1 . The effective pdf, denot-
ed by p(x), can be established using the additive pooling technique as shown in (4) below.
p( x)  1 p1 ( x)   2 p2 ( x)   N p N ( x)
(4)
3.2 Expected Value of the Weighted Sum of Probability Distribution Functions
Theorem 1. The expected value of an effective pdf (denoted by f5) is equal to the weighted sum of the expected values for each of the N pdf’s.
N
f 5  E[ x] 
 i Ei [ x]
(5)
i 1
… where  i and Ei [x] are the weight and expected value, respectively, of the ith pdf.
Proof:
Multiply x to both sides of Equation (4):
xp( x)  1 xp1 ( x)   2 xp2 ( x)   N xpN ( x)
Take the integral of (6) from lower limit  L to upper limit  U :
9
(6)
Santos: IIM with Multiple Experts
Submitted to IIOA
U
U
U
U
L
L
L
L
 xp( x)dx  1  xp1 ( x)dx   2  xp2 ( x)dx     N
U
N
L
i 1
 xp
N
( x)dx
(7)
U
 xp( x)dx    xp ( x)dx
i
(8)
i
L
From equation (8), the expected value f5, is obtained by setting  L   and  U   :

N

N

i 1

i 1
 xp( x)dx   i  xpi ( x)dx  f5  i Ei x
//q.e.d.
3.3 Conditional Expected Value of the Weighted Sum of Probability Distribution Functions
Theorem 2. The conditional expected value of an upper-tail region of an effective pdf (denoted by f4) is
related to each of the N component pdf’s according to the following formula:
N

i 1
L
N

i 1
L
 i  xpi ( x)dx
f4 
(9)
 i  pi ( x)dx
… where  i is the weight of the ith pdf pi (x) .
Proof:
Take the integral of Equation (4) from lower limit  L to upper limit  U :
U
U
U
 p( x)dx    p ( x)dx    p
1
L
1
2
L
U
N
L
i 1
2
( x)dx     N
U
p
N
( x)dx
(10)
L
L
U
 p( x)dx    p ( x)dx
i
(11)
i
L
Divide equation (8) with (11):
10
Santos: IIM with Multiple Experts
Submitted to IIOA
U
U
N
 xp( x)dx   xp ( x)dx
L
U

i 1
i
i
L
(12)
U
N
  p ( x)dx

 p( x)dx
i 1
L
i
i
L
Any conditional expected value fk bounded by the region x   L and x   U is defined exactly
as the left-hand side (12). Hence:
U
N
 i  xpi ( x)dx
fk 
i 1
N
L
U
i 1
L
(13)
 i  pi ( x)dx
The conditional expected value f4 of an upper-tail region of the weighted sum of N pdf’s corresponding to a high-consequence/low-probability region is obtained from (13) by setting  U   .
N

i 1
L
N

i 1
L
 i  xpi ( x)dx
f4 
 i  pi ( x)dx
//q.e.d.
Definition. An alternative form for the expression for f4 given in equation (9) is obtained by letting

k in

 xpi ( x)dx and k
L

d
i

 p ( x)dx , where the superscripts n and d represent the contribution of the
i
L
ith pdf in the numerator and denominator, respectively, of f4.
N
f4 
  i k in
i 1
N

i 1
(14)
 i k id
11
Santos: IIM with Multiple Experts
Submitted to IIOA
The upper-tail probability , corresponding to a specific  L happens to be the denominator of f4
as shown in equation (14).
N
   i k id
(15)
i 1
4. Examples of Combining Probability Distributions from Multiple Experts
The following discussions highlight nonparametric distribution—a mathematical function whose required
inputs can describe intuitively the shape and range of the resulting probability distribution (Vose 1997).
For example, the distribution of the magnitude of terrorist-induced demand reductions could be described
by minimum, maximum, most likely, or any desired percentile estimates. Examples of distributions that
require such estimates include the triangular, beta, and quantile distributions. For brevity, the paper focuses only on the triangular and beta distributions. The use of probability distributions for expert elicitation has been demonstrated in various risk analysis literature (see, for examples, Haimes and Chittister
(1995); Schoof (1996); and Johnson-Payton et al. (1999)).
4.1 Triangular Probability Distribution Function
A triangular distribution function is defined as follows:
 ( c 2 (ax)(ba)a )

p ( x)   ( c 2 (ac)(cx)b )
 0

a xb
bxc
otherwise
(16)
The parameters of a triangular distribution are: a is the minimum value of x; b is the most likely
value of x (i.e., mode); and c is maximum value of x. The aim in this section is to present a form of f4 for
an effective pdf resulting from the weighted sum of multiple triangular pdfs. Based on (16), the pdf of a
triangular distribution specified by expert i can be written as follows:
12
Santos: IIM with Multiple Experts
Submitted to IIOA
 ( c 2a( x)(bai ) a )
 i 2(ici  xi ) i
pi ( x)   ( ci  ai )( ci bi )

0

ai  x  bi
bi  x  ci
otherwise
(17)
…where the parameters a i , bi and c i are defined in the same manner as a, b, and c (i.e., minimum, most likely, and maximum values). The expected value (or mean µi) of a triangular distribution,
with parameters a i , bi and c i is given by:
i 
ai  bi  ci
3
(18)
To derive the extreme value f4, we refer to equation (14) to derive k in and k id . The possible values of k in are summarized as follows:
0

3

ci  3ci  L2  2  L3

3(ci  ai )(ci  bi )

n
2
2
k i    ai bi  ai ci  bi ci2  bi2 ci  ai bi ci  2  L3  3ai  L2

3(ci  ai )(bi  ai )

ai  bi  ci


3
 L  ci
bi   L  ci
ai   L  bi
(19)
 L  ai
It is worth noting that if the lower limit  L is equal or lower than the minimum value ai, the expression for k in simply becomes the mean of the triangular distribution.
Furthermore, the possible values of k id are summarized as follows:
 L  ci
0


(c i   L ) 2

(ci  ai )(ci  bi )

d
ki  
2
  ai bi  ai ci  bi ci   L  2ai  L

(ci  ai )(ci  bi )

1

bi   L  ci
(20)
ai   L  bi
 L  ai
13
Santos: IIM with Multiple Experts
Submitted to IIOA
The expressions k in and k id for a triangular distribution (see (19) and (20)) can be derived using
straightforward integration and have been omitted here in the interest of space.
Example 1: Two experts provide parameter estimates to triangular distributions representing the direct
effects (expressed in %) of a terrorist attack on the demand reduction for the air transportation sector.
Calculate the expected value and conditional expected value (at upper-tail partition of L=18%).
Table 1. Estimates of the direct effects on reduced demand for air transportation (in %)
Minimum
Most Likely
Maximum
Expert 1
6
12
24
Expert 2
12
16
20
This example aims to calculate both the expected value and conditional expected value of the direct demand reduction to the air transportation sector given two expert-elicited triangular distributions. It
is assumed that the two experts are equally credible. Figure 2 depicts the triangular distributions corresponding to experts 1 and 2, along with the resulting effective pdf.
Probability Density
0.3
0.25
0.2
0.15
0.1
0.05
0
0
5
10
15
20
25
Demand Reduction for Air Transportation Sector (%)
Expert 1
Expert 2
Effective PDF
Figure 2. Effective distribution resulting from triangular distributions of two experts
14
Santos: IIM with Multiple Experts
Submitted to IIOA
Using equation (18), the expected values of the direct effects are 14% and 16% for experts 1 and
2, respectively. The effective mean (f5) of the combined distribution is just the weighted sum of the means
of each expert’s triangular distribution. Earlier, we noted that the two experts are equally credible and this
suggests a weight of 0.5 for each of the two experts, hence from equation (5):
f 5  0.5(14%)  0.5(16%)  15%
For the conditional expectation (f4) of the combination of the two distributions, we first use equations (19) and (20) at a partition of  L  18 % to get: k1n  3.3333, k 2n  2.3333, k1d  0.1667, and
k 2d  0.1250. From equation (14) we can calculate:
f4 
0.5(3.3333)  0.5(2.3333)
= 19.4%
0.5(0.1667)  0.5(0.1250)
Note that the probability corresponding to f 4  19.4% is just the value of the denominator in
the previous equation, which happens to be   0.5(0.1667)  0.5(0.1250)  0.15 . The location of f4 in
the effective pdf and the area of the shaded upper-tail region  are depicted in Figure 3.
Probability Density
0.2
0.15
0.1
  0.15
0.05
0
0
5
10
Demand Reduction for
Air Transportation Sector (%)
15
 L  18 %
20
25
f 4  19 .4%
Effective PDF
Figure 3. Conditional expectation for combined triangular distributions
15
Santos: IIM with Multiple Experts
Submitted to IIOA
4.2 Beta Probability Distribution
The beta distribution is typically used in the analysis of uncertainty in project management applications. The BetaPERT distribution is an extension of the original beta distribution that makes use of
three parameters, a, b, and c, just like in a triangular distribution, to estimate a project’s minimum, most
likely, and maximum duration. The BetaPERT parameters (referred to as “beta” hereafter, for simplicity)
for expert i (ai, bi, and ci) are used to compute the additional parameters i and i required by the beta pdf
as follows:
pi ( x)   ( x; i ,  i ai , ci )
 1  (   )  x  a
i
i
i



  c  a  ( )(  )  c  a
i 
i
i  i
i
 i

otherwise




 i 1
 ci  x

 ci  a i



 i 1
a i  x  ci
(21)
0
The beta distribution  ( x; i ,  i ai , ci ) makes use of the Gamma function. The Gamma of a
constant k  0 is:

(k )   x k 1e  x dx
(22)
0
In (21), the parameters  i and  i are related to ai, bi, and ci (i.e., minimum, most likely, and
maximum estimates) using the following definitions (Vose 1997).
i 
( i  ai )( 2bi  ai  ci ) 4bi  5ai  ci

(bi   i )(ci  ai )
ci  ai
(23)
i 
 i (c i   i )
(  i  ai )
(24)

ai  4bi  ci
6
(25)
Theorem 3. The following identity simplifies the calculation of conditional expected values associated
with the beta distribution.
16
Santos: IIM with Multiple Experts
Submitted to IIOA
xpi ( x)  x ( x; i ,  i , ai , ci )
 i
 (ci  ai )
i  i

 ( x; i  1,  i , ai , ci )  ai ( x; i ,  i , ai , ci )

(26)
Proof:
xpi ( x)  x ( x; i ,  i ai , ci )
Equation (21) can be written as:
 x  ai  ai
x ( x; i ,  i ai , ci )  
 ci  a i
 ( i   i )  x  ai


 ( i )(  i )  ci  ai
  ci  a i
  
  ci  a i
 x  ai
 
 ci  a i
  ci  ai  ( i   i )  x  ai
  


c

a

(

)

(

)
i 
i
i  ci  a i
  i



 ai
 
 ci  ai
  ci  ai  ( i   i )  x  ai
  


c

a

(

)

(

)
i 
i
i  ci  a i
  i



 i 1
 i 1
 ci  x

 ci  a i



 i 1
 ci  x

 ci  a i



 i 1



 i 1
 ci  x

 ci  a i



i 1
Term 1
Term 2
For Term 1:
 ci  a i

 ci  a i
 ( i   i ) ( i  1)(  i ) ( i  1   i )  x  ai




 ( i )(  i ) ( i  1   i ) ( i  1)(  i )  ci  ai



( i 1) 1
 ci  x

 ci  a i
Using the Gamma identity (k  1)  k(k ) :
 ci  a i   i
( i  1   i )  x  ai




 ci  ai   i   i ( i  1)(  i )  ci  ai



( i 1) 1
 ci  x

 ci  a i
 
 1 ( i  1   i )  x  ai
  

ci  ai ( i  1)(  i )  ci  ai
 

 i 
 ( x; i  1,  i , ai , ci )
 (ci  ai )



i 
 i
 i
 (ci  ai )
i  i
For Term 2:
 1  (   )  x  a
i
i
i


ai 
 ci  ai  ( i )(  i )  ci  ai



 i 1
 ci  x

 ci  a i
17



 i 1 









i 1
( i 1) 1
 ci  x

 ci  ai



 i 1








i 1
Santos: IIM with Multiple Experts
Submitted to IIOA
 ai ( x; i ,  i , ai , ci )
Adding Terms 1 and 2:
xpi ( x)  x ( x; i ,  i , ai , ci )
 i
 (ci  ai )
i  i

 ( x; i  1,  i , ai , ci )  ai ( x; i ,  i , ai , ci )

//q.e.d.
The terms that are needed for the conditional expected value f4, namely k id and k in from equation (14) can be computed using the following approach.
Let (  L ; i ,  i , ai , ci ) denote the cumulative probability function of a beta distribution for the
interval    x   L :
Knowing that k id 
k id  1 


L
L
n
 pi ( x)dx and k i 
L
 p ( x)dx  1   ( 
i
L
 xp ( x)dx as suggested in equation (14) then:
i
; i ,  i , a i , ci )
(27)

Let i denote the expected value of the ith pdf. For the numerator term k in , we have:
k in


L
L

 xpi ( x)dx  i   xpi ( x)dx
(28)
Substituting equation (26) to (28):
L

 i
k in   i   (ci  ai )
i  i
 
 i
k in   i  (ci  a i )
i  i


 ( x; i  1,  i , ai , ci )  ai ( x; i ,  i , ai , ci )dx



(  L ;  i  1,  i , a i , ci )  a i (  L ;  i ,  i , a i , ci )

18
(29)
(30)
Santos: IIM with Multiple Experts
Submitted to IIOA
…where  L is the x-axis partition for the upper-tail of a beta distribution (i.e., xL). The notation  L should not be confused with  i , which is one of the parameters of a beta distribution.
Example 2. Consider the data in Table 1 and construct beta distributions based on the expert-elicited parameters. Calculate the expected value and conditional expected value using the same upper-tail partition
of L=18%. Assume that both distributions are equally credible.
The expected value for the effective pdf can be computed using equations (5) and (25):
1 
6  4(12)  24
12  4(16)  20
= 13%;  2 
= 16%; and f 5  0.5(13)  0.5(16) = 14.5%.
6
6
Recall that only three parameters were elicited from expert i, namely ai, bi, and ci. However, the
values of i and i can be determined using equations and (24).
For expert 1:
4b1  5a1  c1 4(12)  5(6)  24

 2.3333
c1  a1
24  6
 (c  1 ) 2.3333(24  13)
1  1 1

 3.6667
( 1  a1 )
13  6
1 
For expert 2:
4b2  5a2  c2 4(16)  5(12)  20

3
c2  a2
20  12
 (c   2 ) 3(20  16)
2  2 2

3
( 2  a2 )
16  12
2 
Figure 4 superimposes the effective pdf with the two underlying expert pdfs. The terms that are
needed for the conditional expected value f4, namely k in and k id can be determined from equations (30)
and (27), respectively. It is convenient to do this using predefined software functions; for example there is
19
Santos: IIM with Multiple Experts
Submitted to IIOA
a ready-made function for (  L ; i ,  i , ai , ci ) in MS Excel, which is “= BETADIST (  L ; i ,  i , ai , ci ) ”. The calculated k in and k id values are as follows: k1n  1.5802,
k 2n  1.9199, k1d  0.0815, and k 2d  0.1051. Hence from equation (14):
N
f4 
 i kin
i 1
N
 i kid

1 (k1n )   2 (k 2n ) 0.5(1.5802)  0.5(1.9199)

 18.76 %
1 (k1d )   2 (k 2d ) 0.5(0.0815)  0.5(0.1051)
i 1
The denominator of the f4 tells the area (i.e., probability) of the upper tail, which happens to be 
= 0.0933. Numerical procedures can be done if the upper-tail probability  is the one that is specified instead of  L . For example, if  = 0.05 one can calculate  L  18.6% . Following the steps above, this
new specification of  = 0.05 will yield f 4  19.5% , as shown in Figure 5.
Probability Density
0.25
0.2
0.15
0.1
0.05
0
0
5
10
15
20
25
30
Demand Reduction for Air Transportation Sector (%)
Expert 1
Expert 2
Effective PDF
Figure 4. Effective distribution resulting from beta distributions of two experts
20
Santos: IIM with Multiple Experts
Submitted to IIOA
0.18
0.16
Prob Density
0.14
0.12
0.1
0.08
0.06
  0.05
0.04
0.02
0
0
5
10
15
Demand Reduction for  L  18.6%
Air Transportation Sector (%)
20
f 4  19 .5%
25
30
Effective PDF
Figure 5. Upper-tail conditional expectation for combined beta distributions
5. IIM Case Study
As described in Section 2.2., the IIM is based on Leontief’s I-O model, which has the capability to estimate the ripple effects of economic disruptions across interdependent sectors. Given a demand reduction
in one sector (or a set of sectors), the IIM can generate the resulting inoperabilities for all sectors given
their interdependencies. In this section, we use the 59-sector I-O data (household sector excluded) from
the Bureau of Economic Analysis (BEA 2006a) to model the ripple effects due to a direct demand reduction of the air transportation sector (S29). The sector definitions are provided in Table 2 along with the
case study results that will be described later in this section. Previous IIM applications only considered
single-point estimates for the input perturbation (i.e., c* in equation (1)). Here, a distribution is used to
reflect the possible realizations of demand reduction scenarios that a particular sector could experience.
Another extension to the IIM that is embedded within this uncertainty modeling is constructing an effective pdf from multiple expert-elicited distributions.
For the case study, direct demand reduction scenarios are assumed for S29. While the air transportation sector has proven its importance in promoting commerce and tourism over the years, the demand for this sector declined when it was used as a weapon of terrorism during 9-11. For example, a
21
Santos: IIM with Multiple Experts
Submitted to IIOA
comparison of 2002 versus 2001 economic data for New York reveals at least a 15% decline in the air
transportation sector component of its gross domestic product (BEA 2006b). Furthermore, the Federal
Aviation Administration (FAA) enplanement data indicate a significant reduction in the demand for air
transportation pursuant to the 9-11 terrorist attack (FAA 2002). Suppose that instead of applying a singlepoint estimate to represent the direct demand reduction to S29, a distribution is used based on the two
underlying beta distributions shown in Figure 4. In addition, suppose that only S29 suffers a direct effect
for simplicity (Note that Santos (2006) identified other directly affected sector such as accommodation
and tourism and pointed out supply effects in addition to demand reduction).
To generate the inoperability rankings given the distribution of possible S29 demand reduction
scenarios (represented by the effective pdf in Figure 5), a Monte Carlo simulation algorithm is implemented as follows:
(i) Draw a random sample from the distribution of demand reduction to S29
(ii) Assign a perturbation value to S29 (c*29) using the sample obtained from (i)
(iii) Solve for the inoperability vector using q = (I–A*)-1c*
(iv) Repeat (i) to (iii) until the desired number of iterations is achieved
Since the IIM is a linear model, the sector rankings in terms of inoperability remain the same regardless of the perturbation value that is assigned to S29 (i.e., the random sample obtained from (i)
above). Nevertheless, this information does not discount the importance of performing the above Monte
Carlo simulation algorithm. Aside from the ordinal rankings, the actual distribution of inoperability values for each sector can be obtained using the algorithm, which ultimately enables the calculation of the
expected and conditional expected values (f5 and f4). Figure 6 shows the inoperability distributions for the
top-10 most affected sectors due to S29 demand reduction scenarios. These sectors are: (1) S29-Air transportation ; (2) S35-Other transportation and support activities; (3) S3-Oil and gas extraction; (4) S24Petroleum and coal products manufacturing; (5) S49-Administrative and support services; (6) S16-Other
transportation equipment manufacturing; (7) S34-Pipeline transportation; (8) S23-Printing and related
22
Santos: IIM with Multiple Experts
Submitted to IIOA
support activities; (9) S11-Fabricated metal product manufacturing; and (10) S58-Food services and
drinking places.
A summary of the expected inoperability value and conditional inoperability expected value at 
= 0.05 (denoted by f5 and f4, respectively) for each of the 59 sectors is shown in Table 2. To interpret the
entries in this table, we use S29 as a reference for the subsequent discussions. The expected value (f5) of
inoperability for S29 given the distribution of possible demand reduction scenarios depicted in Figure 5 is
14.55%. For an extreme-case scenario corresponding to the worst 5% realizations (i.e.,  = 0.05), the
conditional expected value (f4) of inoperability for S29 increases to 19.56%.
One might question the need for carrying out the Monte Carlo simulation when the ultimate goal
is to calculate the f5 and f4 for the resulting inoperability distributions in Figure 6. Note that the values f5
and f4 of are directly computable from the underlying pdf in Figure 5. For example, to get the f4 corresponding to the extreme-case inoperability values for all the 59 sectors, we use the “initial” f4 value
shown in Figure 5 and assign this value (19.5%) to the c*29 element within the c* vector. Then, we solve q
= (I–A*)-1c*, which would give the f4 values for all the 59 sectors—without having to resort to the Monte
Carlo simulation. However, this approach will work only when one sector is perturbed. For cases where
the number of directly perturbed sectors exceeds 1, the solution to the IIM will involve summation of
random variables (corresponding to the distribution of demand reductions for multiple sectors), which can
be solved using the method of convolution. To better understand this proposition, consider a 2-sector instantiation of q = (I–A*)-1 c* = Dc* (i.e., the solution to the IIM in (1), where D = (I–A*)-1):
 q1   d11 d12  c1*   d11c1*  d12c2* 
q   d
 *   
*
*
 2   21 d 22  c2  d 21c1  d 22c2 
(31)
Equation (30) implies that if both the demand reduction inputs ( c1* and c2* ) are random variables,
the inoperability for either sector (q1 and q2) can be related to these random variables through convolution. The analytical solution to the convolution of random variables is difficult to derive; and can be impossible to establish for complex functions. Hence, the above Monte Carlo simulation algorithm can be
23
Santos: IIM with Multiple Experts
Submitted to IIOA
used as an efficient solution approach to numerically solve the IIM equation given demand reductions to
multiple sectors.
Rank 1 = Sector S29
Rank 2 = Sector S35
2000
2000
1000
1000
0
0.05
0.1
0.15
0.2
0
0.005
0.25
Rank 3 = Sector S3
0.015
0.02
Rank 4 = Sector S24
2000
2000
1000
1000
0
0.005
0.01
0.01
0
0.015
4
6
8
10
12
-3
x 10
Rank 5 = Sector S49
Rank 6 = Sector S16
2000
2000
1000
1000
0
2
3
4
5
6
0
7
1
1.5
2
2.5
3
-3
-3
x 10
x 10
Rank 7 = Sector S34
Rank 8 = Sector S23
2000
2000
1000
1000
0
1
1.5
2
2.5
3
0
0.6
3.5
0.8
1
1.2
1.4
-3
x 10
Rank 9 = Sector S11
Rank 10 = Sector S58
2000
2000
1000
1000
0.8
1
1.2
1.4
1.6
-3
x 10
0
0.6
3.5
0
0.6
1.6
-3
0.8
1
1.2
1.4
1.6
-3
x 10
x 10
Figure 6. Frequency distribution of inoperability for the top-10 most-affected sectors due to disruption in air
transportation sector (S29) based on 10,000 simulated observations (see Table 2 for sector definitions)
24
Santos: IIM with Multiple Experts
Submitted to IIOA
Table 2. Expected (f5) and Conditional Expected (f4, at  = 0.05) Inoperability Values Due to Disruption in Air Transportation Sector (S29)
Codes
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
S16
S17
S18
S19
S20
S21
S22
S23
S24
S25
S26
S27
S28
S29
S30
Sector Description
Crop & animal production
Forestry, fishing, & related activities
Oil & gas extraction
Mining, except oil & gas
Support activities for mining
Utilities
Construction
Wood products
Nonmetallic mineral products
Primary metal products
Fabricated metal products
Machinery manufacturing
Computer & electronic products
Electrical equipment & appliances
Motor vehicle, body, trailer, & parts
Other transportation equipment
Furniture & related products
Miscellaneous manufacturing
Food, beverage, & tobacco products
Textile & textile product mills
Apparel, leather, & allied products
Paper manufacturing
Printing & related support activities
Petroleum & coal products
Chemical manufacturing
Plastics & rubber products
Wholesale trade
Retail trade
Air transportation
Rail transportation
f5
0.03%
0.04%
1.00%
0.05%
0.05%
0.04%
0.00%
0.02%
0.04%
0.08%
0.10%
0.03%
0.03%
0.03%
0.01%
0.24%
0.01%
0.02%
0.02%
0.02%
0.01%
0.06%
0.11%
0.79%
0.05%
0.04%
0.06%
0.01%
14.55%
0.08%
f4
0.04%
0.05%
1.34%
0.07%
0.07%
0.05%
0.01%
0.03%
0.05%
0.11%
0.13%
0.04%
0.05%
0.04%
0.02%
0.32%
0.01%
0.02%
0.03%
0.03%
0.01%
0.08%
0.14%
1.06%
0.07%
0.06%
0.08%
0.01%
19.56%
0.11%
Codes
S31
S32
S33
S34
S35
S36
S37
S38
S39
S40
S41
S42
S43
S44
S45
S46
S47
S48
S49
S50
S51
S52
S53
S54
S55
S56
S57
S58
S59
25
Sector Description
Water transportation
Truck transportation
Transit & ground passenger
Pipeline transportation
Other transportation
Warehousing & storage
Publishing including software
Motion picture & sound recording
Broadcasting & telecommunications
Information & data processing
Federal banks, credit intermediation
Securities, commodity contracts
Insurance carriers & related activities
Funds, trusts, & other financial vehicles
Real estate
Rental & leasing services
Professional, scientific, & technical services
Management of companies & enterprises
Administrative & support services
Waste management & remediation services
Educational services
Ambulatory health care services
Hospitals, nursing & residential care
Social assistance
Performing arts, museums, & related activities
Amusements, gambling, & recreation
Accommodation
Food services & drinking places
Other services
f5
0.07%
0.05%
0.04%
0.23%
1.27%
0.08%
0.03%
0.03%
0.07%
0.09%
0.06%
0.03%
0.06%
0.01%
0.02%
0.09%
0.08%
0.08%
0.42%
0.04%
0.01%
0.00%
0.00%
0.00%
0.04%
0.02%
0.03%
0.10%
0.01%
f4
0.10%
0.07%
0.05%
0.31%
1.71%
0.10%
0.05%
0.04%
0.10%
0.12%
0.09%
0.05%
0.08%
0.01%
0.03%
0.13%
0.11%
0.10%
0.56%
0.05%
0.02%
0.00%
0.00%
0.00%
0.05%
0.02%
0.05%
0.13%
0.02%
Santos: IIM with Multiple Experts
Submitted to IIOA
6. Summary and Conclusions
This paper examines an extension to the IIM where input distributions to a particular sector can be elicited from multiple experts. Many articles on risk modeling concur that the representation of uncertainty is
better captured as the number of inputs from credible experts increases. In previous IIM applications, single-point estimates were used to represent the model inputs (i.e., the % demand reduction to each directlyaffected sector). The paper explores the case where multiple probabilistic inputs from experts are available, from where metrics such as expected values and extreme-event conditional expected values can be
subsequently computed. For the expected value (f5) of a combination of probability distributions, the required calculation is straightforward to do because f5 is simply the weighted sum of the expected values of
the underlying distributions. For the conditional expected value (f4), the paper derives a formula (see (9))
that can be used regardless of the form of the underlying distributions. The computations of f5 and f4 for
multiple probabilistic expert inputs are demonstrated using the triangular and beta distributions.
A case study that considers demand reduction scenarios of the air transportation sector (S29) is
presented using a mixture of two beta distributions. Due to their interdependencies with S29, the ideal (or
target) productions of the other sectors are also affected; hence, resulting into some levels of production
opportunity losses measured via the inoperability metric. For example, when the demand for S29 reduces
(which was the case in the aftermath of 9-11), other sectors such as: Other transportation and support activities; Oil and gas extraction; Petroleum and coal products manufacturing; and Administrative and support services (among others) are also rendered inoperable to some extent. It should be noted that the case
study employed a demand-side analysis (as opposed to a supply-based model) and the ranking of the
most-affected sectors were generated using “backward” linkages (see Miller and Blair (1985)). This
means that those sectors that provide the most production inputs to S29 are expected to suffer the greatest
when the demand for S29 decreases (e.g., less “oil and gas” products will be consumed by S29 due to a
reduced air transportation sector activity). For disasters that predominantly affect public consumption behavior (e.g., effect of SARS on air transportation and tourism), demand-side analysis is reasonable to use.
Nevertheless, most disasters can affect both demand and supply aspects of an interdependent economy
26
Santos: IIM with Multiple Experts
Submitted to IIOA
and a combined analysis is necessary. For the case of 9-11, public apprehension on airline safety led to a
direct reduction of demand for S29; at the same the destruction of buildings (e.g., World Trade) and critical infrastructures led to a direct supply reduction for sectors such as finance and government services.
While this paper shows a pure demand-side perturbation analysis to S29, the analysis can be extended in
the future to permit exogenous specification of supply perturbations to other directly-affected sectors. The
primary aim of this paper is to demonstrate how demand reduction scenarios from multiple experts (in the
form of multiple probabilistic inputs to a particular sector) can be pooled together to generate an effective
distribution that can serve as input to a Monte Carlo simulation-based implementation of the IIM. An area
for future research is to extend the probabilistic concept to encompass disruptions to more than one sector
(this will involve convolutions of several random variables); as well as to model mixtures of demand reduction scenarios with exogenously specified supply constraints.
Policy formulation and research focus on disaster impact analysis have gained momentum in light
of the recent calamities and global war on terrorism. While historical data for natural disasters are fairly
obtainable (e.g., hurricane events), other disasters such as terrorism are relatively rare in frequency and
hence data availability can be sparse. For cases where specific data are inadequate (or nonexistent), expert
elicitation becomes an acceptable process for conducting risk analysis. Predicting the impacts of highlyuncertain events become more accurate as the number of experts increases, granted that the elicited distributions can be pooled based on the confidence and credibility of the experts. It should be noted that the
term “expert” can also be broadened to include usage of data from events that have some degree of similarity with the event that is being examined (e.g., assessing the resilience of economic and infrastructure
sectors in the event of a fictitious but plausible high-altitude electromagnetic pulse attack using lessons
learned from Hurricane Katrina).
Acknowledgments
This study was supported by the National Science Foundation, under a grant to the University of Virginia
Center for Risk Management of Engineering Systems (NSF 0301553: Input-Output Risk Model of Critical Infrastructure Systems, May 2003June 2007). This work was also supported under Award number
27
Santos: IIM with Multiple Experts
Submitted to IIOA
2003-TK-TX-0003 from the U.S. Department of Homeland Security, Science and Technology Directorate. Points of view in this document are those of the authors and do not necessarily represent the official position of the U.S. Department of Homeland Security or the Science and Technology Directorate.
The I3P is managed by Dartmouth College.
References
Asbeck, E., Y. Haimes. 1984. The partitioned multiobjective risk method. Large Scale Systems 6 13-38.
Bureau of Economic Analysis (BEA). 2006a. Regional input-output multiplier system (RIMS) II industry
aggregations. Retrieved October 31, 2006, http://www.bea.gov/bea/regional/rims/appc.cfm.
BEA. 2006b. Gross domestic product by state. Retrieved October 31, 2006,
http://www.bea.gov/bea/regional/gsp/.
Cagnoa, A., F. Carona, M. Mancinia, F. Ruggerib. 1999. Using AHP in determining the prior distributions on gas pipeline failures in a robust Bayesian approach. Reliability Engineering & System
Safety 67 275-284.
Cho, S., P. Gordon, J.E. Moore II, H.W. Richardson, M. Shinozuka, S. Chang. 2001. Integrating transportation network and regional economic models to estimate the costs of a large urban earthquake.
Journal of Regional Science 41 39-65.
Clemen R., L. Winkler. 1999. Combining probability distributions from experts in risk analysis. Risk
Analysis 19 187-203.
Crowther, K., Y. Haimes. 2005. Applications of the inoperability input-output model (IIM) for systemic
risk assessment and management of interdependent infrastructures. Systems Engineering 8 323341.
Federal Aviation Administration (FAA). 2002. Aviation industry overview fiscal year 2001. FAA Office
of Aviation Policy and Plans, Washington, DC.
Forester, J., D. Bley, S. Cooper, E. Lois, N. Siu, A. Kolaczkowski, J. Wreathall. 2004. Expert elicitation
approach for performing ATHEANA quantification. Reliability Engineering & System Safety 83
207–220.
28
Santos: IIM with Multiple Experts
Submitted to IIOA
Garthwaite, P., J. Kadane, A. O’Hagan. 2005. Statistical methods for eliciting probability distributions.
Journal of the American Statistical Association 100 680-700.
Haimes, Y. 2004. Risk Modeling, Assessment, and Management, Second Edition. John Wiley and Sons,
Inc., NY.
Haimes, Y., C. Chittister. 1995. An acquisition process for the management of nontechnical risks associated with software development. Acquisition Review Quarterly Spring 121-154.
Haimes, Y., B. Horowitz, J. Lambert, J. Santos, C. Lian, K. Crowther. 2005a. Inoperability input-output
model (IIM) for interdependent infrastructure sectors: Theory and methodology. Journal of Infrastructure Systems 11 67-79.
Haimes, Y., B. Horowitz, J. Lambert, J. Santos, K. Crowther, C. Lian. 2005b. Inoperability input-output
model (IIM) for interdependent infrastructure sectors: Case study, Journal of Infrastructure Systems 11 80-92.
Hammitt, J., A. Shlyakhtel. 1999. The expected value of information and the probability of surprise. Risk
Analysis 19 135-152.
Hoekstra, R., M. Janssen. 2006. Environmental responsibility and policy in a two-country dynamic inputoutput model. Economic Systems Research 18 61-84.
Homeland Security Council. 2004. Planning scenarios: Executive Summaries, Created for use in national,
federal, state, and local homeland security preparedness activities. Retrieved October 12, 2006,
http://www.globalsecurity.org/security/library/report/2004/hsc-planning-scenarios-jul04.htm.
Johnson-Payton, L., Y. Haimes, J. Lambert. 1999. A methodology for risk management of compound
failure modes, in Y. Haimes and R.E. Steuer (editors), Proceedings of the XIVth International
Conference on Multiple Criteria Decision Making (MCDM) Lecture Notes in Economics and
Mathematical Systems. Springer-Verlag, New York, NY.
Kaplan, S., B. J. Garrick. 1981. On the Quantitative Definition of Risk. Risk Analysis 1 11-27.
Klein, J., M. Moeschberger. 1997. Survival analysis: Techniques for censored and truncated data.
Springer-Verlag, New York, NY.
29
Santos: IIM with Multiple Experts
Submitted to IIOA
Leontief, W. 1936. Quantitative input and output relations in the economic system of the United States.
Review of Economic and Statistics 18 105-125.
Lian, C., Y. Haimes. 2006. Managing the risk of terrorism to interdependent infrastructure systems
through the dynamic inoperability input-output model. Systems Engineering 9 241–258.
Lipscomb, J., G. Parmigiani, V. Hasselblad. 1998. Combining expert judgment by hierarchical modeling:
an application to physician staffing. Management Science 44 149-161.
Lowrance, W. 1976. Of Acceptable Risk. William Kaufmann, Inc., Los Altos, CA.
Midmore, P., M. Munday, A. Roberts. 2006. Assessing industry linkages using regional input-output tables. Regional Studies 40 329–343.
Miller, R., P. Blair. 1985. Input-output analysis: Foundations and extensions. Prentice-Hall, Inc., NJ.
O’Hagan, A., J. Oakley. 2004. Probability is perfect, but we can’t elicit it perfectly. Reliability Engineering and System Safety 85 239–248.
Rockafellar, R., S. Uryasev. 2000. Optimization of Conditional Value-at-Risk. Journal of Risk 2 21-41.
Rose, A. 2004. Economic principles, issues, and research priorities in hazard loss estimation, in Y.
Okuyama and S. Chang (editors), Modeling Spatial and Economic Impacts of Disasters. Springer-Verlag, New York, NY.
Santos, J. 2006. Inoperability input-output modeling of disruptions to interdependent economic systems,
Systems Engineering 9 20-34.
Santos, J., Y. Haimes. 2004. Modeling the demand reduction input-output (I-O) inoperability due to terrorism of interconnected infrastructures. Risk Analysis 24 1437-1451.
Schooff, R. 1996. Hierarchical Holographic Modeling for Software Acquisition Risk. Ph.D. Dissertation,
Department of Systems Engineering, University of Virginia, Charlottesville, VA.
Vose, D. 1997. Monte Carlo risk analysis modeling, in V. Molak (editor), Fundamentals of Risk Analysis
and Management. CRC Press, New York, NY.
30