Uncertainty and Sensitivity Analysis Methods for
Improving Design Robustness and Reliability
by
Qinxian He
Bachelor of Science in Engineering, Duke University, 2008
Master of Science, Massachusetts Institute of Technology, 2010
Submitted to the Department of Aeronautics and Astronautics
in partial fulfillment of the requirements for the degree of
Doctor of Philosophy
at the
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
June 2014
c Massachusetts Institute of Technology 2014. All rights reserved.
Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Department of Aeronautics and Astronautics
May 7, 2014
Certified by . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Karen E. Willcox
Professor of Aeronautics and Astronautics
Thesis Supervisor
Certified by . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Ian A. Waitz
Jerome C. Hunsaker Professor of Aeronautics and Astronautics
Committee Member
Certified by . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Douglas L. Allaire
Assistant Professor of Mechanical Engineering, Texas A&M University
Committee Member
Accepted by . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Paulo C. Lozano
Associate Professor of Aeronautics and Astronautics
Chair, Graduate Program Committee
2
Uncertainty and Sensitivity Analysis Methods for
Improving Design Robustness and Reliability
by
Qinxian He
Submitted to the Department of Aeronautics and Astronautics
on May 7, 2014, in partial fulfillment of the
requirements for the degree of
Doctor of Philosophy
Abstract
Engineering systems of the modern day are increasingly complex, often involving numerous components, countless mathematical models, and large, globally-distributed
design teams. These features all contribute uncertainty to the system design process
that, if not properly managed, can escalate into risks that seriously jeopardize the
design program. In fact, recent history is replete with examples of major design setbacks due to failure to recognize and reduce risks associated with performance, cost,
and schedule as they emerge during the design process.
The objective of this thesis is to develop methods that help quantify, understand,
and mitigate the effects of uncertainty in the design of engineering systems. The
design process is viewed as a stochastic estimation problem in which the level of
uncertainty in the design parameters and quantities of interest is characterized probabilistically, and updated through successive iterations as new information becomes
available. Proposed quantitative measures of complexity and risk can be used in the
design context to rigorously estimate uncertainty, and have direct implications for
system robustness and reliability. New local sensitivity analysis techniques facilitate
the approximation of complexity and risk in the quantities of interest resulting from
modifications in the mean or variance of the design parameters. A novel complexitybased sensitivity analysis method enables the apportionment of output uncertainty
into contributions not only due to the variance of input factors and their interactions,
but also due to properties of the underlying probability distributions such as intrinsic
extent and non-Gaussianity. Furthermore, uncertainty and sensitivity information
are combined to identify specific strategies for uncertainty mitigation and visualize
tradeoffs between available options. These approaches are integrated with design
budgets to guide decisions regarding the allocation of resources toward improving
system robustness and reliability.
3
The methods developed in this work are applicable to a wide variety of engineering
systems. In this thesis, they are demonstrated on a real-world aviation case study to
assess the net cost-benefit of a set of aircraft noise stringency options. This study
reveals that uncertainties in the scientific inputs of the noise monetization model are
overshadowed by those in the scenario inputs, and identifies policy implementation
cost as the largest driver of uncertainty in the system.
Thesis Supervisor: Karen E. Willcox
Title: Professor of Aeronautics and Astronautics
Committee Member: Ian A. Waitz
Title: Jerome C. Hunsaker Professor of Aeronautics and Astronautics
Committee Member: Douglas L. Allaire
Title: Assistant Professor of Mechanical Engineering, Texas A&M University
4
Acknowledgments
The journey toward a PhD is one that cannot be undertaken alone. It is only with help
from countless people along the way that I have made it thus far, and I am eternally
grateful for their support. First and foremost, I would like to thank my advisor,
Professor Karen Willcox, for her guidance, mentorship, and encouragement over the
past four years. It has truly been a privilege to work with Karen, and I am greatly
indebted to her for everything she has taught me about research, life, and making
it all balance together. I also extend thanks to my committee members, Dean Ian
Waitz and Professor Doug Allaire, for lending their knowledge and support to every
aspect of this research, and for providing invaluable feedback and guidance along the
way. I would also like to acknowledge my thesis readers, Professor John Deyst and
Professor Olivier de Weck for contributing insightful comments and suggestions that
have helped to improve this thesis.
To my labmates in the ACDL, past and present, thank you so much for sharing
this journey with me. It has been an honor to work with and learn from all of you.
I would like to especially recognize Leo Ng, Andrew March, Sergio Amaral, Chad
Lieberman, Giulia Pantalone, Harriet Li, Laura Mainini, Rémi Lam, Marc Lecerf,
Tiangang Cui, Patrick Blonigan, Alessio Spantini, Eric Dow, Hemant Chaurasia, and
Xun Huan for making my stay in Building 37 such a memorable experience. I am also
grateful to the larger AeroAstro community for providing a welcoming and dynamic
environment in which to learn and grow. I have made so many friends in AeroAstro
and will always fondly remember the good times we’ve shared. A special thank you
to the ladies of the Women’s Graduate Association of Aeronautics and Astronautics
(WGA3 ) — Sunny Wicks, Sameera Ponda, Farah Alibay, Whitney Lohmeyer, Abhi
Butchibabu, Sathya Silva, and Carla Perez Martinez — it has been such a joy getting
to know you and I can’t wait to see the amazing things you will continue to accomplish
in the future. I also owe a great deal of thanks to Philip Wolfe. From a technical
standpoint, thank you for helping me set up and run the CAEP/9 noise stringency
case study on such short notice; from a personal perspective, thank you for your
5
friendship over the past 10 years — I have really appreciated our project meetings,
commiseration lunches, and random adventures both at Duke and at MIT. Finally, I
would like to acknowledge the AeroAstro staff members who work tirelessly to make
life easier and more enjoyable for graduate students. Jean Sofronas, Joyce Light,
Meghan Pepin, Marie Stuppard, Beth Marois, Bill Litant, Robin Palazzolo, and Sue
Whitehead — thank you for being so generous with your time, help, smiles, and words
of encouragement.
Outside of research, I have been extremely fortunate to have the support of a
diverse group of friends who help me keep things in perspective. Matt and Amanda
Edwards, Jim Abshire, David Rosen, Emily Kim, Audrey Fan, Lori Ling, Yvonne
Yamanaka, Tiffany Chen, Tatyana Shatova, and Robin Chisnell are fellow graduate
students in the Boston area, and dear friends who have accompanied me on this
journey. Thank you for the potluck dinners, ski trips, hiking outings, gym workouts,
and runs along the Charles River — you have done so much to lift my spirits and
bring joy to my life. Another huge source of strength comes from the Sidney-Pacific
Graduate Community, without a doubt the best graduate dormitory in the world.
Living at SP was one of the highlights of my time at MIT, and has taught me so
much about friendship, leadership, and the spirit of community. In particular, I wish
to thank the former Housemaster team — Professor Roger Mark and Mrs. Dottie
Mark, and Professor Annette Kim and Dr. Roland Tang — thank you for welcoming
me into your home and making sure that I am always well-fed. To my friends from SP
— Amy Bilton, Mirna Slim, Ahmed Helal, Jit Hin Tan, Brian Spatocco, George Lan,
Po-Ru Loh, Kendall Nowocin, George Chen, Boris Braverman, Fabián Kozynski, and
many, many others — thank you for sharing your kindness and talents, for inspiring
me to set ambitious goals, and for showing me that there are few problems in life that
can’t be solved through teamwork and more bacon.
Finally, I would like to thank my family for their steadfast love and support. My
parents, Jie Chen and Helin He, left behind friends, family, and successful careers in
China to immigrate to the United States in pursuit of better opportunities. Without
their sacrifice, I would not be where I am today. I credit them for instilling in me the
6
importance of a good education, the determination to overcome whatever challenges
may arise, and the confidence to forge my own path. Last but not least, I want to
thank Tim Curran for being my co-pilot on this incredible journey. Thank you for
always believing in me, for teaching me to be patient with myself and others, and for
giving me the strength to push through the tough times. I look forward to starting
the next chapter of our lives together.
The work presented in this thesis was funded in part by the International Design Center at the Singapore University of Technology and Design, by the DARPA
META program through AFRL Contract Number FA8650-10-C-7083 and Vanderbilt
University Contract Number VU-DSR #21807-S7, by the National Science Foundation Graduate Research Fellowship Program, and by the Zonta International Amelia
Earhart Fellowship Program.
7
Contents
1 Introduction
19
1.1
Motivation for Uncertainty Quantification . . . . . . . . . . . . . . .
20
1.2
Terminology and Scope . . . . . . . . . . . . . . . . . . . . . . . . . .
21
1.2.1
System Design Process . . . . . . . . . . . . . . . . . . . . . .
21
1.2.2
Characterizing Uncertainty . . . . . . . . . . . . . . . . . . . .
24
1.2.3
Some Common Probability Distributions . . . . . . . . . . . .
26
Designing for Robustness and Reliability . . . . . . . . . . . . . . . .
28
1.3.1
Defining Robustness and Reliability . . . . . . . . . . . . . . .
29
1.3.2
Background and Current Practices . . . . . . . . . . . . . . .
30
1.3.3
Opportunities for Intellectual Contribution . . . . . . . . . . .
33
1.4
Thesis Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
33
1.5
Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
34
1.3
2 Quantifying Complexity and Risk in System Design
2.1
2.2
2.3
37
Background and Literature Review . . . . . . . . . . . . . . . . . . .
37
2.1.1
Quantitative Complexity Metrics . . . . . . . . . . . . . . . .
38
2.1.2
Differential Entropy . . . . . . . . . . . . . . . . . . . . . . . .
40
A Proposed Definition of Complexity . . . . . . . . . . . . . . . . . .
41
2.2.1
Exponential Entropy as a Complexity Metric . . . . . . . . . .
42
2.2.2
Interpreting Complexity . . . . . . . . . . . . . . . . . . . . .
44
2.2.3
Numerical Complexity Estimation . . . . . . . . . . . . . . . .
46
Related Quantities . . . . . . . . . . . . . . . . . . . . . . . . . . . .
47
9
2.3.1
Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
49
2.3.2
Kullback-Leibler Divergence and Cross Entropy . . . . . . . .
51
2.3.3
Entropy Power . . . . . . . . . . . . . . . . . . . . . . . . . .
52
2.4
Estimating Risk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
53
2.5
Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . .
54
3 Sensitivity Analysis
3.1
3.2
3.3
3.4
3.5
55
Background and Literature Review . . . . . . . . . . . . . . . . . . .
56
3.1.1
Variance-Based Global Sensitivity Analysis . . . . . . . . . . .
56
3.1.2
Vary-All-But-One Analysis . . . . . . . . . . . . . . . . . . . .
59
3.1.3
Distributional Sensitivity Analysis . . . . . . . . . . . . . . . .
60
3.1.4
Relative Entropy-Based Sensitivity Analysis . . . . . . . . . .
61
Extending Distributional Sensitivity Analysis to Incorporate Changes
in Distribution Family . . . . . . . . . . . . . . . . . . . . . . . . . .
62
3.2.1
Cases with No Change in Distribution Family . . . . . . . . .
65
3.2.2
Cases Involving Truncated Extent for X o . . . . . . . . . . . .
67
3.2.3
Cases Involving Resampling . . . . . . . . . . . . . . . . . . .
71
Local Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . .
75
3.3.1
Local Sensitivity to QOI Mean and Standard Deviation . . . .
76
3.3.2
Relationship to Variance-Based Sensitivity Indices . . . . . . .
77
3.3.3
Local Sensitivity to Design Parameter Mean and Standard Deviation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
79
Entropy Power-Based Sensitivity Analysis . . . . . . . . . . . . . . .
81
3.4.1
Entropy Power Decomposition . . . . . . . . . . . . . . . . . .
81
3.4.2
The Effect of Distribution Shape . . . . . . . . . . . . . . . .
85
Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . .
88
4 Application to Engineering System Design
91
4.1
R-C Circuit Example . . . . . . . . . . . . . . . . . . . . . . . . . . .
91
4.2
Sensitivity Analysis of the R-C Circuit . . . . . . . . . . . . . . . . .
94
4.2.1
94
Identifying the Drivers of Uncertainty . . . . . . . . . . . . . .
10
4.2.2
4.3
4.4
Understanding the Impacts of Modeling Assumptions . . . . .
Design Budgets and Resource Allocation . . . . . . . . . . . . . . . . 101
4.3.1
Visualizing Tradeoffs . . . . . . . . . . . . . . . . . . . . . . . 102
4.3.2
Cost and Uncertainty Budgets . . . . . . . . . . . . . . . . . . 104
4.3.3
Identifying and Evaluating Design Alternatives . . . . . . . . 107
Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
5 Application to a Real-World Aviation Policy Analysis
5.1
5.2
5.3
5.4
94
115
Background and Problem Overview . . . . . . . . . . . . . . . . . . . 115
5.1.1
Aviation Noise Impacts . . . . . . . . . . . . . . . . . . . . . . 115
5.1.2
CAEP/9 Noise Stringency Options . . . . . . . . . . . . . . . 117
APMT-Impacts Noise Module . . . . . . . . . . . . . . . . . . . . . . 119
5.2.1
Model Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.2.2
Estimating Monetary Impacts . . . . . . . . . . . . . . . . . . 125
Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 127
5.3.1
Comparison of Stringency Options and Discount Rates . . . . 127
5.3.2
Global and Distributional Sensitivity Analysis . . . . . . . . . 128
5.3.3
Visualizing Tradeoffs . . . . . . . . . . . . . . . . . . . . . . . 132
Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
6 Conclusions and Future Work
137
6.1
Thesis Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.2
Thesis Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
6.3
Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
A Bimodal Distribution with Uniform Peaks
141
A.1 Specifying Distribution Parameters . . . . . . . . . . . . . . . . . . . 142
A.2 Options for Variance Reduction . . . . . . . . . . . . . . . . . . . . . 143
A.3 Switching to a Bimodal Uniform Distribution . . . . . . . . . . . . . 145
B Derivation of Local Sensitivity Analysis Results
147
B.1 Perturbations in Mean . . . . . . . . . . . . . . . . . . . . . . . . . . 147
11
B.1.1 Effect on Complexity . . . . . . . . . . . . . . . . . . . . . . . 148
B.1.2 Effect on Risk . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
B.2 Perturbations in Standard Deviation . . . . . . . . . . . . . . . . . . 149
B.2.1 Effect on Complexity . . . . . . . . . . . . . . . . . . . . . . . 150
B.2.2 Effect on Risk . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
C Derivation of the Entropy Power Decomposition
153
D CAEP/9 Analysis Airports
157
Bibliography
161
12
List of Figures
1-1 The effect of complexity on system development time in the aerospace,
automobile, and integrated circuit industries [1, Figure 1] . . . . . . .
20
1-2 Schematic of the proposed system design process . . . . . . . . . . . .
22
1-3 Examples of uniform, triangular, Gaussian, and bimodal uniform probability densities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
26
1-4 Schematics of design for robustness and reliability. Red and blue probability densities represent initial and updated estimates of fY (y), respectively. Dashed line indicates the location of the requirement r.
Shaded portions denote regions of failure in which the requirement is
not satisfied. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
29
2-1 Probability density of a standard Gaussian random variable (left) and
a uniform random variable of equal complexity (right) . . . . . . . . .
45
2-2 Probability density of a multivariate Gaussian random variable with
µ = (0, 0) and Σ = I2 (top left), and uniform random variables of equal
complexity in R2 and R (top right and bottom, respectively) . . . . .
45
2-3 Numerical approximation of h(Y ) using k bins of equal size . . . . . .
47
2-4 Comparison of variance, differential entropy, and complexity between
a bimodal uniform distribution and its equivalent Gaussian distribution 50
2-5 Probability of failure associated with the requirement r . . . . . . . .
53
3-1 Apportionment of output variance in GSA [8, Figure 3-1] . . . . . . .
56
13
3-2 Examples of reasonable uniform, triangular, and Gaussian distributions. In each figure, the solid red line denotes fX o (x), the dashed blue
lines represent various possibilities for fX 0 (x), and the dotted black
lines show κfX o (x), which is greater than or equal to the corresponding density fX 0 (x) for all x.
. . . . . . . . . . . . . . . . . . . . . . .
65
3-3 Examples of reasonable uniform, triangular, and Gaussian distributions for cases involving a truncated extent for X o
. . . . . . . . . .
69
3-4 Examples of reasonable uniform, triangular, and Gaussian distributions for cases involving resampling. The left column shows instances
where δ is too large, and resampling is required. The right column
shows instances where δ is sufficiently small, and AR sampling can be
used to generate samples of X 0 from samples of X o .
. . . . . . . . .
73
3-5 The relative locations of r and µY greatly impact the change in risk
associated with a decrease in σY . Moving from the red probability
density to the blue, P (Y < r) decreases if µY − r > 0, and increases if
µY − r < 0. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
77
3-6 Examples of Y = X1 + X2 with increase, decrease, and no change in
Gaussianity between the design parameters and QOI . . . . . . . . .
87
3-7 Sensitivity indices Si , ηi , and ζi for three examples of Y = X1 + X2 .
For each factor, Si equals the product of ηi and ζi . . . . . . . . . . . .
88
4-1 R-C high-pass filter circuit . . . . . . . . . . . . . . . . . . . . . . . .
92
4-2 Bode diagrams for magnitude (top) and phase (bottom). The dashed
black line indicates the required cutoff frequency of 300 Hz. . . . . . .
92
4-3 Histogram of cutoff frequency for Case 0 generated using 10,000 MC
samples. The dashed black line indicates the required cutoff frequency
of 300 Hz. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
93
4-4 Global and distributional sensitivity analysis results for Case 0 . . . .
94
4-5 Distributional sensitivity analysis results with changes to distribution
family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14
95
4-6 Examples of uniform (U), triangular (T), Gaussian (G), and bimodal
uniform (BU) distributions for X2 , each with a variance of 0.30 µF2 .
96
4-7 Examples of uniform (U), triangular (T), Gaussian (G), and bimodal
uniform (BU) distributions for X2 supported on the interval [3.76, 5.64] µF 98
4-8 Histogram of Y resulting from various distributions for X2 considered
in Case 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
99
4-9 Main effect and entropy power-based sensitivity indices for Case 2 . . 100
4-10 Contours for variance, complexity, and risk corresponding to reductions
in factor variance, generated from 10,000 MC simulations (solid colored
lines) or approximated using distributional or local sensitivity analysis
results (dashed black lines) . . . . . . . . . . . . . . . . . . . . . . . . 103
4-11 Notional cost associated with a 100(1 − δ)% reduction in the variance
of X1 or X2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
4-12 Contours for variance, complexity, and risk corresponding to reductions
in factor variance (solid colored lines) overlaid with contours for cost
of implementation (dashed green lines) . . . . . . . . . . . . . . . . . 106
4-13 Contours for risk (solid blue lines) overlaid with contours for cost
(dashed green lines). Shaded region denotes area in which all budget constraints are satisfied. . . . . . . . . . . . . . . . . . . . . . . . 107
4-14 Uncertainty contours for variations in resistance and capacitance, generated from 10,000 MC simulations (solid colored lines) or approximated using entropy power-based sensitivity analysis results (dashed
black lines) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
4-15 Uncertainty contours for variations in resistance and resistor tolerance 108
4-16 Uncertainty contours for variations in capacitance and capacitor tolerance109
4-17 Uncertainty contours for variations in resistor and capacitor tolerance
109
4-18 Tradeoffs in complexity and risk for various design options. Dashed
green lines bound region where both complexity and risk constraints
are satisfied. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
15
5-1 Joint and marginal distributions for the regression parameters obtained
using 10,000 bootstrapped samples . . . . . . . . . . . . . . . . . . . 122
5-2 Schematic of APMT-Impacts Noise Module [142, adapted from Figure 4]126
5-3 Comparison of mean estimates for Y in Stringencies 1–3 and discount
rates between 3%–9%. Error bars denote 10th , 90th percentile values.
127
5-4 Main effect sensitivity indices for Stringency 1, DR = 3%, with IC
modeled probabilistically in (a) and as a deterministic value in (b) . . 129
5-5 Adjusted main effect sensitivity indices for Stringency 1, DR = 3%,
with IC modeled probabilistically in (a) and as a deterministic value
in (b) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
5-6 Histogram of Y under various IC assumptions for Stringency 1, DR =
3% . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
5-7 Uncertainty contours associated with changes in the mean and variance
of BNL and IC, generated from 10,000 MC simulations (solid colored
lines) or approximated using sensitivity analysis results (dashed black
lines) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
5-8 Uncertainty contours associated with changes in the mean and variance
of CU and IC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
5-9 Uncertainty contours associated with changes in the mean and variance
of BNL and CU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
A-1 Bimodal uniform distribution with peak bounds aL , bL , aR , and bR ,
and peak heights fL and fR . . . . . . . . . . . . . . . . . . . . . . . 141
A-2 Examples of distributions for X 0 obtained using Algorithms 1–3 (solid
o
lines), based on an initial distribution X BU (dashed green line) and
δ = 0.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
A-3 Examples of switching from a uniform, triangular, or Gaussian distribution to a bimodal uniform distribution. In each figure, the solid
red line denotes fX o (x), the dashed blue line represents fX 0 (x), and
the dotted black line shows κfX o (x), which is greater than or equal to
fX 0 (x) for all x. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
16
List of Tables
2.1
Range, mean, variance, differential entropy, exponential entropy, and
entropy power for distributions that are uniform, triangular, Gaussian,
or bimodal with uniform peaks
2.2
. . . . . . . . . . . . . . . . . . . . .
The properties of variance, differential entropy, exponential entropy,
and entropy power under constant scaling or shift . . . . . . . . . . .
3.1
48
Ratio of δ =
var (X 0 )
var (X o )
49
for changes in factor distribution between the
uniform, triangular, and Gaussian families. Rows represent the initial
distribution for X o and columns correspond to the updated distribution
for X 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
64
3.2
Local sensitivity analysis results for complexity and risk . . . . . . . .
76
4.1
Change in uncertainty estimates for Cases 1-T, 1-G, and 1-BU, as
compared to Case 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2
Change in uncertainty estimates for Cases 2-T, 2-G, and 2-BU, as
compared to Case 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3
96
98
Best achievable uncertainty mitigation results given individual budgets
for cost and uncertainty. Entries in red denote budget violations. . . . 107
4.4
Local sensitivity predictions for the change in mean and standard deviation (SD) of each factor required to reduce risk to 10% . . . . . . . 112
4.5
Uncertainty and cost estimates associated with various design options.
Entries in red denote budget violations. . . . . . . . . . . . . . . . . . 113
17
5.1
Inputs to CAEP/9 noise stringency case study. The definitions for
BNL, CU, RP, SL, and IGR correspond to the midrange lens of the
APMT-Impacts Noise Module, whereas IC and DR are used in the
post-processing of impacts estimates to compute net cost-benefit. . . 124
5.2
Comparison of mean and uncertainty estimates of Y for Stringencies 1–
3 and discount rates between 3%–9%. All monetary values (mean, 10th
and 90th percentiles, standard deviation, and complexity) are listed in
2006 USD (billions). . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
5.3
Main effect sensitivity indices for Stringency 1 at various discount rates 129
D.1 CAEP/9 US airports and income levels . . . . . . . . . . . . . . . . . 160
18
Chapter 1
Introduction
Over the years, engineering systems have become increasingly complex, with astronomical growth in the number of components and their interactions. With this rise
in complexity comes a host of new challenges, such as the adequacy of mathematical
models to predict system behavior, the expense and time to conduct experimentation
and testing, and the management of large, globally-distributed design teams. These
issues all contribute uncertainty to the design process, which can have serious implications for system robustness and reliability and potentially disastrous repercussions
for program outcome.
To address some of these issues, this thesis presents a stochastic process model
to describe the design of complex systems, in which uncertainty in various parameters and quantities of interest is characterized probabilistically, and updated through
successive design iterations as new estimates become available. Incorporated in the
model are methods to quantify system complexity and risk, and reduce them through
the allocation of resources to enhance robustness and reliability. The goal of this
approach is to enable the rigorous quantification and management of uncertainty,
thereby serving to help mitigate technical and programmatic risk.
Section 1.1 describes the motivation for uncertainty quantification, followed by
Section 1.2, which defines the scope of the research and the terminology used to
describe it. Section 1.3 gives an overview of current methodologies used in design
for robustness and reliability, and identifies several potential areas for intellectual
19
contribution. Sections 1.4 and 1.5 outline the main research objectives and provide a
roadmap for the remainder of the thesis.
1.1
Motivation for Uncertainty Quantification
In the aerospace industry, the challenges associated with complexity are particularly
daunting. A recent study by the Defense Advanced Research Projects Agency shows
that as aerospace systems have become more complex, their associated costs and
development times have also reached unsustainable levels (Figure 1-1) [1].1 Until this
unmitigated growth can be contained, it poses a looming threat to the continued
viability and competitiveness of the industry.
Figure 1-1: The effect of complexity on system development time in the aerospace,
automobile, and integrated circuit industries [1, Figure 1]
To understand the potentially devastating effects of complexity in its various manifestations, one needs only to look to recent history, which is replete with examples
of major design setbacks due to failure to recognize and reduce performance, cost,
1
The complexity metric used in the study is the number of parts plus the source lines of code.
For a discussion of various measures of complexity and their applicability, see Section 2.1.
20
and schedule risks as they emerge during the design process. A notable example is
the Hubble Space Telescope which, when first launched, failed its optical resolution
requirement by an order of magnitude. A Shuttle repair mission, costing billions of
additional dollars, was required to remedy the problem [2]. An investigation later
uncovered that data revealing optical system errors were available during the fabrication process, but were not recognized and fully examined; in fact, crucial error
indicators were disregarded and key verification steps omitted. The V-22 Osprey tiltrotor aircraft is another example: over the course of its 25-year development cycle,
the program was fraught with safety, reliability, and affordability challenges, resulting in numerous flight test crashes with concomitant loss of crew and passenger lives
[3]. More recently, the Boeing 787 Dreamliner transport aircraft program has experienced a number of major prototype subsystem test failures, causing budget overruns
of billions of dollars and service introduction delays of about three years. Many of
the program’s problems were attributed to Boeing’s aggressive strategy to outsource
component design and assembly, which created heavy program management burdens
and led to unforeseen challenges during vehicle integration [4].
In these cases and numerous others, the design program was unaware of the mounting risks in the system, and was surprised by one or more unfortunate outcomes. Although these examples are extreme, they are suggestive that improved system design
practices are required in order to identify and address performance, cost, and schedule
risks as they emerge. Tackling the problem of complexity — and more broadly — of
uncertainty in the design of engineering systems is an enormous yet urgent task that
requires innovation and collaboration on both sides of industry and academia.
1.2
1.2.1
Terminology and Scope
System Design Process
Figure 1-2 shows a schematic of the system design process proposed in this thesis. It
outlines the key steps in the design of engineering systems, by which is meant col-
21
Set
Targets
Define
Parameters
𝑋1
Requirements,
Constraints
𝑋2
𝑋3
Perform
Quantify
Uncertainty Sensitivity
Analysis
Evaluate
Quantities
of Interest
Generate
Designs
Models,
Governing
Equations,
Analysis
Tools, etc.
𝑌1
0.35
0.3
0.25
0.2
𝑌2
0.15
0.1
0.05
0
-0.05
-1
-0.5
0
0.5
1
1.5
2
2.5
3
QOI
Design
Feedback
Variance, Complexity,
and Risk
Uncertainty
Feedback
Robustness and Reliability
Resource
Allocation
Figure 1-2: Schematic of the proposed system design process
lections of interrelated elements that interact with one another to achieve a common
purpose [5]. A system can be a physical entity, such as a vehicle or consumer product,
or an abstract concept, such as a software program or procedure. Whatever its form,
a system must satisfy a set of targets in order to be deemed successful. These targets
typically refer to functional requirements that dictate the system’s performance, but
may also include additional constraints that stipulate conditions such as budget or
time.
From the system targets, a set of design parameters can be enumerated which
are used to characterize the system. Design parameters are variables that relate
the requirements in the functional domain to aspects of the design in the physical
domain that can be manipulated to satisfy those requirements [6]. They can include
quantities that define the system itself, such as individual component dimensions, as
well as data describing the procedures and other elements by which the system can
be manufactured, operated, and maintained [7]. In this thesis, we assume that all
design parameters within a system are independent, and denote them using the m × 1
22
vector x = [x1 , x2 , . . . , xm ]T , where xi is the ith design parameter among a total of m
design parameters.
In addition to the design parameters, there are also quantities that are used to
characterize aspects of the system’s performance that are of interest to the designer.
We refer to them generically as quantities of interest (QOI), and represent them using
the n × 1 vector y = [y1 , y2 , . . . , yn ]T , where yj is the j th QOI out of a total of n.
Quantities of interest are typically evaluated indirectly as functions of the design
parameters using available models and analysis tools. For example, in aerospace
engineering, relevant QOI can include the gross weight of an aircraft, which can be
computed from design parameters regarding vehicle geometry and material selection,
or the lift-to-drag ratio, which can be estimated from the external geometry using
results from computational fluid dynamics and wind tunnel testing. For simplicity,
we will assume that there already exist models and tools (may be black-box) with
which to evaluate the QOI of a system from the design parameters. We express this
relationship as:
y = g(x),
(1.1)
where g : Rm → Rn denotes the mapping from design parameters to QOI. For the j th
QOI yj , this mapping is given by:
yj = gj (x),
(1.2)
where gj : Rm → R1 , and j = 1, 2, . . . , n.
With regard to analysis models and tools used to aid system design, we will also
use the terms factor and output. Consistent with the definitions set forth in Ref. [8],
a factor refers to an external input to a model, whereas output is a model result of
interest. While these terms are similar to design parameter and quantity of interest
in the context of a system, they are not equivalent. A factor can refer to any model
input, and need not correspond to a parameter that can be manipulated by the
designer. Similarly, the outputs of a model are not necessarily limited to the QOI of
a system.
23
1.2.2
Characterizing Uncertainty
For most realistic engineering systems, the design process is made more difficult by
the presence of non-deterministic features that contribute to stochasticity in the outcome, which comprise the uncertainty associated with the QOI [9, 10]. In this thesis, uncertainty quantification refers to “the science of identifying, quantifying, and
reducing uncertainties associated with models, numerical algorithms, experiments,
and predicted outcomes or quantities of interest” [11]. We focus specifically on the
development of methods to quantify uncertainty during the intermediate stages of
system design, where requirements, design parameters, and QOI have already been
defined, but no feasible design has yet been realized. Uncertainty pertaining to the
formulation of functional requirements, development of analysis tools, manufacturing,
deployment, maintenance, and organizational structure will not be discussed.
Uncertainty can come in many flavors; for example, it can be broadly categorized
as epistemic, owing to insufficient or imperfect knowledge, or aleatory, which arises
from natural randomness and is therefore irreducible [10–13]. Furthermore, there
are several types of uncertainty associated with the use of computer-based models in
system design [14]:
• Parameter uncertainty, resulting from not knowing the true values of the inputs
to a model;
• Parametric variability, relating to unspecified conditions in the model inputs or
parameters;
• Residual variability, due to the inherently unpredictable or stochastic nature of
the system;
• Observation error, referring to uncertainty associated with actual observations
or measurements;
• Model inadequacy, owing to the insufficiency of any model to exactly predict
reality;
• Code uncertainty, arising from not knowing the output of a model given a
particular set of inputs until the code is run.
24
The framework presented in this thesis is intended to be general enough to describe
any of aforementioned types of uncertainty, provided that they can be described
probabilistically. However, we note that although both epistemic and aleatory sources
of uncertainty can be characterized within this framework, only epistemic uncertainty
can be reduced through additional research and improved knowledge. Thus, while
it may be possible to study the effects of aleatory sources of uncertainty, it is not
appropriate to allocate resources toward their reduction.
To characterize the propagation of uncertainty, we employ continuous random
variables to represent design parameters2 and quantities of interest [10, 18, 19], and
model the time evolution of design as a stochastic process whose outcome is governed
by some probability distribution. We simulate the behavior of the system using
Monte Carlo (MC) sampling, which generates numerous QOI estimates from which
a probability density can be estimated.3 We use X = [X1 , X2 , . . . , Xm ]T and Y =
[Y1 , Y2 , . . . , Yn ]T to denote vectors of random variables that correspond to each entry in
x and y, respectively. The probability densities associated with Xi and Yj are given by
fXi (xi ) and fYj (yj ), respectively. In general, these densities can be of arbitrary form,
and closed-form expressions may not be available. The stochastic representations of
(1.1) and (1.2) can be written as:
Y = g(X),
(1.3)
Yj = gj (X).
(1.4)
Once the QOI have been estimated, we use selected metrics to quantitatively
describe their levels of uncertainty. This information is compared to the system’s
2
The process of assigning design parameters to specific probability distributions will not be discussed in this thesis. Instead, the reader is referred to Refs. [15–17], which detail several methods
addressing this topic. They include density estimation techniques to leverage historical data, elicitation of expert opinion to supplement empirical evidence, and calibration and aggregation procedures
to combine information from multiple sources.
3
Monte Carlo simulation is computationally expensive, and may be intractable for some systems.
In practice, computational expense can be reduced by using quasi-random sampling techniques
[20, 21], reducing the dimensionality of the input space [22–24], or employing less-expensive surrogate models to approximate system response [25–29]. In this thesis, we will assume that forward
simulation using MC sampling is computationally feasible.
25
uncertainty targets through the uncertainty feedback loop. We also perform sensitivity
analysis to identify the key drivers of uncertainty, enumerate uncertainty reduction
strategies, and evaluate tradeoffs in the context of resource allocation decisions for
uncertainty mitigation, thus enhancing system robustness and/or reliability.
1.2.3
Some Common Probability Distributions
Throughout this thesis, we will focus on four particular types of probability distributions for characterizing uncertainty in the design parameters: uniform, triangular,
𝑓𝑋 𝑥
𝑓𝑋 𝑥
Gaussian, and bimodal with uniform peaks, which are shown in Figure 1-3.
𝑎
𝑏
𝑥
𝑎
𝑥
𝑐
𝑏
𝑓𝑋 𝑥
𝑓𝑋 𝑥
(a) Uniform distribution with lower bound (b) Triangular distribution with lower
a and upper bound b
bound a, upper bound b, and mode c
𝑓𝐿
𝜎
𝜇
𝑥
𝑎𝐿
𝑓𝑅
𝑏𝐿
𝑥
𝑎𝑅
𝑏𝑅
(c) Gaussian distribution with mean µ and (d) Bimodal distribution composed of
standard deviation σ
uniform peaks with bounds aL , bL , aR ,
and bR , and peak heights fL and fR
Figure 1-3: Examples of uniform, triangular, Gaussian, and bimodal uniform probability densities
26
A uniform distribution is specified by two parameters, a and b, which correspond
to the lower and upper bound of permissible values, respectively. All values on the
interval [a, b] are equally likely, and therefore a uniform distribution is often assumed
when no knowledge exists about a parameter except for its limits. The probability
density of a uniform random variable X is given by:

1



fX (x) = b − a


0
if a ≤ x ≤ b,
(1.5)
otherwise.
Similar to the uniform distribution, a triangular distribution is also parameterized
by a lower bound a and an upper bound b. There is also an additional parameter, c,
which represents the mode. A triangular distribution is appropriate when information
exists about the bounds of a particular quantity, as well as the most likely value. The
corresponding probability density is:

2(x − a)




(b − a)(c − a)



2(b − x)
fX (x) =


(b − a)(b − c)




0
if a ≤ x ≤ c,
if c < x ≤ b,
(1.6)
otherwise.
Unlike uniform and triangular distributions, a Gaussian distribution does not have
finite bounds. It is often used when there is strong confidence in the most likely value
[17]. A Gaussian distribution is parameterized by its mean µ and standard deviation
σ. Its probability density is nonzero everywhere along the real line:
(x − µ)2
fX (x) = √
exp −
.
2σ 2
2πσ 2
1
(1.7)
In practice, it is often unrealistic to work with random variables with infinite support.
Instead, it may be necessary to approximate a Gaussian distribution with a truncated
form [30].
Bimodal distributions are used to characterize quantities consisting of two distinct
27
sets of features or behaviors. They can arise in a variety of science and engineering
contexts: for example, in materials science, to characterize the size and properties
of polymer particles [31–33]; in medicine, to explain disease epidemiology within and
between populations [34, 35]; in human factors, to analyze individual and group communication patterns [36, 37]; in earth science, to describe the occurrence of natural
phenomena such as rainfall and earthquakes [38–41]; and in electrical engineering, to
relate disparities in circuit component lifetime to distinct modes of failure [42, 43].
For simplicity, we will restrict our consideration to bimodal distributions composed
of two uniform peaks, henceforth termed “bimodal uniform,” as shown in Figure 13(d). We use [aL , bL ] and [aR , bR ] to denote the lower and upper bounds of the left
and right peaks, respectively, and fL and fR to designate the peak heights. It takes
five parameters to uniquely specify a distribution of this type; therefore, given any
five of {aL , bL , fL , aR , bR , fR }, the sixth can be directly computed from the condition
that:
fL (bL − aL ) + fR (bR − aR ) = 1.
(1.8)
The probability density of a bimodal uniform distribution is given by:



fL



fX (x) = fR




0
1.3
if aL ≤ x ≤ bL ,
if aR ≤ x ≤ bR ,
(1.9)
otherwise.
Designing for Robustness and Reliability
Uncertainty in engineering design is closely related to the robustness and reliability of
the resulting systems. In this section, we provide definitions for these terms, discuss
current practices in design for robustness and reliability, as well as identify several
opportunities for intellectual contribution.
28
1.3.1
Defining Robustness and Reliability
According to Knoll and Vogel, robustness is “the property of systems that enables
them to survive unforeseen or unusual circumstances” [44]. That is to say, to improve
a system’s robustness is to make it less susceptible to exhibit unexpected behavior in
the presence of a wide range of stochastic elements [45, 46].
Reliability, on the other hand, describes a system’s “probability of success in
satisfying some performance criterion” [9], and is closely tied to system safety [45].4
It is also related to the definition of risk that we will introduce in Section 2.4, which
poses risk and reliability as complementary quantities.
Refinement: Change Std Dev
Redesign: Shift Mean
0.8
0.6
0.6
𝑓𝑌 𝑦
Constant
Scaling
0.4
0.2
𝑃 𝑌<𝑟
0
-0.2
-4
f X(x)
f X(x)
𝑓𝑌 𝑦
0.8
0.4
𝑃 𝑌<𝑟
0.2
𝑦
𝑟
-3
Constant Shift
0
𝑦
𝑟
-0.2
(a)-2 Design
for
robustness
(b)
-1
0
1
2
3
4
5
-4
-3
-2 Design
-1
0 for 1reliability
2
3
Redesign
and
Refinement
x
4
5
x
𝑓𝑌 𝑦
0.8
Constant
Shift &
Scaling
f X(x)
0.6
0.4
0.2
0
𝑃 𝑌<𝑟
𝑦
𝑟
-0.2
-4
(c) Design
and
-3
-2 for
-1 robustness
0
1
2
x
reliability
3
4
5
Figure 1-4: Schematics of design for robustness and reliability. Red and blue probability densities represent initial and updated estimates of fY (y), respectively. Dashed
line indicates the location of the requirement r. Shaded portions denote regions of
failure in which the requirement is not satisfied.
4
Alternate definitions of reliability also ascribe an element of time. For example, reliability is
sometimes described as the probability of failure within a given interval of time [47], or the expected
life of a product or process, typically measured by the mean time to failure or the mean time between
failures [46]. In this thesis, we will not address the temporal aspect of reliability.
29
Graphically, we illustrate the distinction between design for robustness and design
for reliability in Figure 1-4, in which our estimate of a QOI Y is updated from the
red probability density to the blue. The dashed line y = r represents a performance
criterion below which the design is deemed unacceptable; the area of the shaded region
corresponds to P (Y < r), the probability of failure.5 Figure 1-4(a) shows a constant
scaling of fY (y) with no change in the mean value, whereas Figure 1-4(b) depicts
a constant shift (horizontal translation) of fY (y) with no change in the standard
deviation. Both activities have direct implications for the uncertainty associated with
Y . In practice, improvements in robustness and reliability can occur concurrently,
as depicted in Figure 1-4(c). In the following chapters, however, we will treat them
as distinct activities in order to investigate how robustness and reliability are related
to two proposed measures of uncertainty — complexity and risk — as well as how
sensitivity analysis can be used to predict changes in these quantities.
1.3.2
Background and Current Practices
The origins of robust design, also known as parameter design, can be traced to postWorld War II Japan, where it was pioneered by Genichi Taguchi as a way to improve
design efficiency and stimulate industries crippled by the war [48]. The aim of the
so-called “Taguchi Methods” is to reduce the sensitivity of products and processes
to various noise factors, such as manufacturing variability, environmental conditions,
and degradation over time [49, 50]. To achieve this goal, Taguchi applied design of
experiment techniques [51, 52], in particular the orthogonal array, to systematically
vary the levels of the control factors (e.g., the design parameters) and noise factors,
quantify the effect in the output, and compute a signal-to-noise ratio. In order to
improve system robustness, the designer seeks to determine the parameter values
that maximize the signal-to-noise ratio, thus minimizing the system’s sensitivity to
variability. Finally, a loss function is used to estimate the expected decrement in
quality due to deviation of the system response from a target value.
5
The probability of failure can also be defined as P (Y > r) in the case that the QOI must not
exceed r, noting that P (Y > r) = 1 − P (Y < r).
30
More recently, the robust design literature has expanded to include techniques
such as the combined array format [53, 54], response modeling [53–55], and adaptive
one-factor-at-a-time experimentation [56, 57]. Whereas traditional Taguchi methods
evaluate the influence of control factors and noise factors separately, the combined
array format merges the two sets in a single experimental design, thereby reducing the
number of required experiments. In response modeling, the output of the system is
modeled and used as the basis for design optimization, rather than the loss function.
One advantage of this approach is that it allows the designer to model the mean
and variance of the output separately, and thus gain an improved understanding
of underlying system behaviors [55, 58]. Finally, in adaptive one-factor-at-a-time
experimentation, the designer begins with a baseline set of factor levels, then varies
each factor in turn to a level that has not yet been tested while holding all other
factors constant. Along the way, the designer also seeks to optimize the system
response by keeping a factor at its new level if the response improves, or reverting to
its previous level if the response worsens. Although this approach does not guarantee
optimal factor settings, it has been shown in certain situations to be more effective
than traditional fractional factorial experimentation for improving design robustness
[56, 57, 59, 60].
Another design philosophy used in many industrial sectors today is what is known
as “Six Sigma.” First introduced by Motorola, Inc. in the mid-1980s, Six Sigma is
a set of strategies and techniques aimed to minimize risk by reducing the sources of
variability in a system, not just the system’s sensitivity to said variability [46, 61].
Qualitatively, to design for Six Sigma is to ensure that the mean system performance
far exceeds the design requirement such that failures are extremely rare. In this
approach, uncertainty in the QOI is represented using a Gaussian distribution with
mean µ and standard deviation σ (or “sigma”), which is subject to a failure criterion
r. Based on experimental testing, the number of sigmas that a particular design
achieves is estimated as the standard score (also known as the z value or z score) of
31
r with respect to the observed µ and σ, expressed by:
z=
r−µ
.
σ
(1.10)
From (1.10), the standard Gaussian cumulative distribution function Φ(z) can be
used to compute the reliability of the system with respect to the QOI. Quantitatively,
Six Sigma is defined by a 99.99966% probability of success, or 3.4 defects per million.6
Note that because Six Sigma seeks to maximize the probability of success through
variance reduction, it has implications for both system robustness and reliability (i.e.,
see Figure 1-4(a)).
In the design for reliability context, a common approach to minimize risk is to
ensure that a system’s nominal performance deviates from a critical requirement by a
specified amount. This is typically done by applying a multiplicative factor of safety
or an additive margin of safety (usually expressed as a percentage) to the requirement,
thus injecting conservatism into the design [5].
Another class of well-known methods are the First- and Second-Order Reliability
Methods (FORM and SORM, respectively). Originally used in civil engineering to
assess the reliability of structures under combined loads, FORM linearize a system
about the current design point in order to estimate the mean and standard deviation
of the resulting QOI [9, 63, 64]. As in Six Sigma, all random variables are assumed to
be Gaussian and uncorrelated, a z score (termed the “reliability index” in FORM) is
computed from the estimates of µ and σ, and the system is assigned a probability of
failure according to 1 − Φ(z). The goal of FORM is to employ various optimization
techniques to modify the design in such a way as to maximize the reliability index.
A similar approach is adopted in SORM, except that a second-order approximation
of the system is used [65, 66].
6
Confusingly, the 3.4 defects per million benchmark does not actually correspond to six standard
deviations of a Gaussian distribution. In fact, a failure probability of 3.4 × 10−6 corresponds to
1 − Φ(4.5), or “4.5 sigma.” By convention, the extra 1.5 sigma shift is introduced to account for
degradation in reliability over time [62]. The actual six sigma failure probability is 1 − Φ(6) =
9.9 × 10−10 , or approximately 1 defect per billion.
32
1.3.3
Opportunities for Intellectual Contribution
In view of current practices in designing for robustness and reliability, this thesis
proposes to make intellectual contributions in three main areas.
First, many existing methods use variance as the sole measure of uncertainty.
Although variance has many advantages as an uncertainty metric (see Section 2.3.1),
we will show in the following chapters that it also has several limitations, and should
be supplemented with additional measures such as complexity and risk in order to
provide a richer description of system robustness and reliability.
Second, a common theme in existing methods is the use of Gaussian distributions to represent uncertainty in a system’s design parameters and QOI. As we will
demonstrate, this simplifying assumption can lead to under- or overestimates of various uncertainty measures, as well as erroneous prioritization of efforts for uncertainty
mitigation. In this thesis, we will investigate the influence of distribution shape in
system design as it relates to uncertainty and sensitivity estimates.
Third, current methods for designing for robustness and reliability typically emphasize variance reduction, but do not provide specific details on how it is to be
achieved. Thus, a potential opportunity lies in using uncertainty and sensitivity information to identify specific strategies to mitigate uncertainty. We propose to do
this by evaluating the tradeoffs between available design options with respect to robustness and reliability, and by incorporating cost and uncertainty budgets to aid
decisions regarding resource allocation.
1.4
Thesis Objectives
The primary goal of this thesis is to quantify and understand the effects of uncertainty
in the design of engineering systems in order to guide decisions aimed to improve
system robustness and reliability. Specifically, the thesis objectives are:
1. To define and apply an information entropy-based complexity metric to estimate
uncertainty in the design of engineering systems.
33
2. To develop a sensitivity analysis methodology that identifies the key contributors to uncertainty, evaluates tradeoffs between various design alternatives, and
informs decisions regarding the allocation of resources for uncertainty mitigation.
3. To apply uncertainty quantification and sensitivity analysis methods to an engineering case study to demonstrate their utility for enhancing system robustness
and reliability.
1.5
Thesis Outline
This thesis is composed of six chapters. The structure and content of the remaining
chapters are outlined below.
• Chapter 2 gives an overview of various metrics used to describe uncertainty,
and proposes complexity and risk as two possible uncertainty measures in the
context of design for robustness and reliability. Definitions are presented for
each of these terms, as well as a discussion of how they can be applied to
characterize uncertainty in various quantities of interest.
• Chapter 3 provides a discussion of existing sensitivity analysis methods and
how they can be used to identify the key drivers of uncertainty. New local
sensitivity analysis techniques are introduced, which can be used to predict
changes in complexity and risk associated with a reduction in variance. In
addition, a novel approach is presented that decomposes output uncertainty
into contributions due to the intrinsic extent and non-Gaussianity of the input
factors and their interactions.
• Chapter 4 describes how uncertainty and sensitivity information can be used
in conjunction with design budgets to identify specific uncertainty mitigation
strategies, visualize tradeoffs between available options, and guide the allocation
of resources aimed at improving system robustness and reliability.
34
• Chapter 5 presents a case study in which the methods developed in this thesis
are applied to assess the net cost-benefit of a set of real-world aviation noise
stringency options and support the design of cost-effective policy implementation responses.
• Chapter 6 summarizes the key contributions of this thesis and highlights some
areas of future work.
35
Chapter 2
Quantifying Complexity and Risk
in System Design
There are many possible ways to quantify uncertainty in a system. In this chapter,
we propose complexity and risk as two specific measures of uncertainty, which can
be used to supplement variance and convey system robustness and reliability. Section 2.1 presents a brief overview of the literature on complexity in system design, with
a particular focus on Shannon’s differential entropy. Section 2.2 proposes a new metric for complexity based on Campbell’s exponential entropy, presents the associated
qualitative and quantitative definitions, and provides some possible interpretations
of complexity. Section 2.3 discusses how this complexity metric relates to several
other quantities that can be used to characterize uncertainty. An analogous set of
definitions is also proposed for quantifying risk, which is presented in Section 2.4.
2.1
Background and Literature Review
Complexity in system design is an elusive concept for which many definitions have
been proposed. Early work in the field of complexity science by Warren Weaver
posited complexity as the nebulous middle ground between order and chaos, a region
in which problems require “dealing simultaneously with a sizeable number of factors
37
which are interrelated into an organic whole” [67]. Another interpretation of this
idea considers a set of “phase transitions” during which the fundamental features of
a system undergo drastic changes [68]. As an illustrative example, consider the phase
transitions of water [69]. On one end of the spectrum, water is frozen into a simple
lattice of molecules whose structure and behavior are straightforward to understand.
At the other extreme, water in gaseous form consists of millions of molecules vibrating
at random, and the study of such a system requires methods of statistical mechanics
or probability theory [70]. In between the two lies the complex liquid state, wherein
water molecules behave in a manner neither orderly nor chaotic, but at once enigmatic
and mesmerizing, which has captured the imagination of fluid dynamicists past and
present.
Although the above example makes the idea of complexity relatable to a large
audience, the debate over its definition still persists. When asked for the meaning of
complexity, Seth Lloyd once replied, “I can’t define it for you, but I know it when I
see it” [69]. While complexity may be in the eye of the beholder, many researchers
agree that there are several properties that complex systems tend to share [71–73]:
• They consist of many parts.
• There are many interactions among the parts.
• The parts in combination produce synergistic effects that are not easily predicted and may often be novel, unexpected, even surprising.
• They are difficult to model and to understand.
2.1.1
Quantitative Complexity Metrics
In addition to qualitative descriptions of complexity, there have also been many attempts to explain complexity using quantitative measures. These metrics can be classified into three general categories: structure-based, process-based, and informationbased.
Structure-based metrics quantify the complexity associated with the physical representation of a system [74]. They typically involve counting strategies: in software
38
engineering, the source lines of code can be used to describe a computer program [75];
in mechanical design, analogous measures include the number of parts [76], functions
[77], or core technologies [78] embodied in a product. Although appealing in their
simplicity, these counting metrics may be susceptible to different interpretations of
what constitutes a distinct component — depending on the level of abstraction, a
component may be as high-level as an entire subsystem, or as basic as the nuts and
bolts holding it together. More sophisticated structure-based metrics also attempt to
address the issue of component interactions. For example, McCabe proposed the idea
of cyclomatic complexity in software engineering, which uses graph theory to determine the number of control paths through a module [79]. Numerous others have also
recommended approaches to estimate system complexity by characterizing the number, extent, and nature of component interactions, which govern the interconnectivity
and solvability of a system [73, 80–82]. Overall, structure-based complexity metrics
are usually easy to understand and to implement, but they may not be meaningful
except in the later stages of design, after most design decision have been made, and
the system is well-characterized [83].
A second class of complexity metrics quantifies system uncertainty in terms of
processes required to realize the system. Metrics in this category can measure, for
example, the number of basic operations required to solve a problem (computational
complexity) [84, 85], the compactness of an algorithm needed to specify a message
(algorithmic complexity or Kolmogorov complexity) [86–88], or the amount of effort
necessary to design, modify, manufacture, or assemble a product [6, 74, 83, 89].
A third possible interpretation of complexity is related to a system’s information
content, often described using the notion of entropy. Like complexity, entropy is
also an abstruse concept.1 Early work by thermodynamicists Carnot and Clausius
sought to use entropy to describe a system’s tendency towards energy dissipation;
later, Gibbs and Boltzmann formalized this notion by giving it a statistical basis,
conceptualizing entropy as a measure of the uncertainty in the macroscopic state of
1
The Hungarian-American mathematician John von Neumann once famously said “nobody knows
what entropy really is, so in a debate you will always have the advantage” [90].
39
a system associated with the microstates that are accessible during the course of
the system’s thermal fluctuations. For a system with probability p(l) of accessing
microstate l, the Gibbs entropy S is defined as:
S = −kB
X
p(l) log p(l) ,
(2.1)
l
where kB = 1.38065 × 10−23 J/K is the Boltzmann constant.
From its origins in thermodynamics and statistical mechanics, entropy has since
been adapted to describe randomness and disorder in many other contexts, including
information theory [91], economics [92], and sociology [93–95]. Most relevant to this
thesis is the concept of information entropy, which was originally proposed by Claude
Shannon to study lossless compression schemes for communication systems [91]. It is
discussed in more detail in the following section; in Section 2.2.1, we will introduce a
complexity metric that is based on information entropy.
2.1.2
Differential Entropy
Shannon’s information entropy is a well-known measure of uncertainty, and represents
the amount of information required on average to describe a random variable. For
a discrete random variable Y that can assume any of k values y (1) , y (2) , . . . , y (k) , the
Shannon entropy is defined in terms of the probability mass function pY (y):
H(Y ) = −
k
X
pY (y (l) ) log pY (y (l) ).
(2.2)
l
The quantity H(Y ) is always nonnegative. In the limiting case, if Y is deterministic with a value of y ∗ , then pY (y) = 1 for y = y ∗ , and zero elsewhere, which
results in H(Y ) = 0. The units of H(Y ) depend on the base of the logarithm used in
(2.2); base 2 and base e are the most common, resulting in units of “bits” or “nats,”
respectively. In this thesis, we will deal exclusively with the natural logarithm. Comparing (2.1) and (2.2), it is easy to see that information entropy has an intuitive and
appealing analogy to thermodynamic entropy as a measure of a system’s tendency
40
toward disorder [96].
In the design of complex systems, design parameters and quantities of interest
typically can vary over a range of permissible values. In order to avoid restricting
these quantities to take only discrete values, we model them as continuous random
variables with associated probability densities. For a continuous random variable Y ,
the analogy to Shannon entropy is differential entropy, which is defined in terms of
fY (y) over the support set Y:
Z
h(Y ) = −
fY (y) log fY (y) dy.
(2.3)
Y
Unlike Shannon entropy, differential entropy can take on values anywhere along
the real line. For example, let Y be a uniform random variable on the interval [a, b].
Using (2.3) with fY (y) = (b − a)−1 for y ∈ [a, b], the differential entropy is computed
to be h(Y ) = log(b − a). Clearly, h(Y ) > 0 if b − a > 1, h(Y ) = 0 if b − a = 1,
and h(Y ) → −∞ as b − a → 0. This result comes about because whereas Shannon
entropy is an absolute measure of randomness in a thermodynamic sense, differential
entropy measures randomness relative to a reference coordinate system [88, 91].
2.2
A Proposed Definition of Complexity
In this thesis, we use complexity as a measure of uncertainty in a system. Qualitatively, we define complexity as the potential of a system to exhibit unexpected
behavior in the quantities of interest, whether detrimental or not [97]. This definition
is consistent with many of the theoretical formulations of complexity discussed in Section 2.1, such as the notions of emergent behavior, nonlinear component interactions,
and information content. It is also in line with the view that reducing complexity
translates to minimizing a system’s information content and maximizing its likelihood
of satisfying all requirements [74, 98].
41
2.2.1
Exponential Entropy as a Complexity Metric
Quantitatively, we use exponential entropy as our complexity metric. Exponential entropy was first proposed by L. Lorne Campbell to measure the extent of a probability
distribution [99]. For a continuous random variable Y , the simplest measure of extent is the range, ν(Y ), defined as the length of the interval on which the probability
density is nonzero:
Z
dy.
ν(Y ) =
(2.4)
Y
In the case where Y is a uniform random variable on the interval [a, b], ν(Y ) = b−a
as expected. One disadvantage of range, however, is that it gives equal weight to
all possible values of y ∈ Y, and consequently can be infinite, as is the case for
probability distributions such as Gaussian or exponential. In these cases, range is
no longer a useful measure of uncertainty; instead, it is desirable to formulate a
measure of “intrinsic extent.” Campbell put forward such a quantity in exponential
entropy, which is closely related to Shannon’s differential entropy [99]. The derivation
of exponential entropy is given below.2
First, consider Mt (Y ), the generalized mean of order t of fY (y) over Y [100]:
 Z !1/t
t

1



fY (y) dy

Y fY (y)
Mt (Y ) =
Z



1

exp
log
fY (y) dy
fY (y)
Y
if t 6= 0,
(2.5)
if t = 0.
From (2.5), it is easy to see that ν(Y ) = M1 (Y ); that is, range is the generalized
mean of order one. Furthermore, when t = 0, M0 (Y ) is the geometric mean of fY (y)
over Y [101].
For arbitrary orders s and t, where s < t, the relationship between Ms (Y ) and
2
Instead of working with fY (y) and Y, Campbell’s derivation uses the Radon-Nikodym derivative
of a probability space (Ω, A, P ), where Ω is the sample space, A is a σ-algebra, and P is a probability
measure. Thus, it is more general than our presentation, which applies to probability densities
defined over real numbers, in that it also extends to probability measures defined over arbitrary
sets.
42
Mt (Y ) can be described by the generalized mean inequality [100]:
Ms (Y ) ≤ Mt (Y ),
(2.6)
where equality holds if and only if Y is a uniform random variable.3 Setting s = 0
and t = 1, we can rewrite (2.6) to obtain a general relationship between M0 (Y ) and
ν(Y ):
M0 (Y ) ≤ M1 (Y ),
(2.7)
M0 (Y ) ≤ ν(Y ).
(2.8)
Campbell terms the quantity M0 (Y ) the intrinsic extent of Y , which is always
less than or equal to the range ν(Y ) (i.e., the “true” extent). Another interpretation
of M0 (Y ) is as “the equivalent side length of the smallest set that contains most of
the probability” [88]. That is to say, whereas ν(Y ) gives the length of the interval
(potentially infinite) that contains all possible values y for which Y = y, M0 (Y )
represents the (finite) extent of Y once the various values of y have been weighted by
their probability of occurrence. In the case where Y is a uniform random variable,
M0 (Y ) and ν(Y ) are equivalent.
In addition, M0 (Y ) is also closely related to Shannon’s differential entropy h(Y ).
To see this, consider the natural logarithm of M0 (Y ):
Z
1
fY (y) dy
log
log M0 (Y ) = log exp
fY (y)
Y
Z
1
=
fY (y) log
dy
fY (y)
Y
Z
= − fY (y) log fY (y) dy
(2.9)
(2.10)
(2.11)
Y
= h(Y ).
(2.12)
From (2.12), it follows that M0 (Y ) is the exponential of the differential entropy of Y ,
3
If Y is a uniform random variable, Mt (Y ) = ν(Y ) = b − a for all t.
43
hence the term “exponential entropy”:
M0 (Y ) = exp[h(Y )].
(2.13)
Using exponential entropy as our metric for complexity, a QOI represented by Y thus
has a complexity given by:
C(Y ) = exp[h(Y )].
2.2.2
(2.14)
Interpreting Complexity
As exponential entropy is a measure of intrinsic extent, it has the same units as the
original random variable. We note that it is also consistent with our proposed qualitative definition of complexity, as well as fits with the notion of robustness introduced
in Section 1.3. In the limiting case, a deterministic system devoid of any potential to
exhibit unexpected behavior in the QOI has no complexity, and thus its exponential
entropy is zero.
From (2.8), (2.13), and (2.14), we have that C(Y ) ≤ ν(Y ), with equality in the case
where Y is a uniform random variable. This relationship implies that of all probability
distributions of a given range, the uniform distribution is the one with the maximum
complexity. This result is consistent with the principle of maximum entropy, which
states that if only partial information is available about a system, the probability
distribution which best represents the current state of knowledge is the one with
largest information entropy [102, 103]. More specifically, if the only knowledge about
a system is the range [a, b] of permissible values, the largest differential entropy (and
hence the greatest exponential entropy) is achieved by assuming a uniform random
variable on [a, b] [104].
Another interpretation of exponential entropy is as the volume of the support
set Y, which is always nonnegative [88]. For an arbitrary continuous random variable Y , the complexity of the associated quantity of interest is equivalent to that
of a uniform distribution for which C(Y ) is the range. Figure 2-1 illustrates this
statement for a standard Gaussian random variable Y ∼ N (0, 1). In this case,
44
Y ~ N( = 0,  = 1)
Y ~ U[0, 4.13]
0.5
0.5
𝐶 𝑌 =
0.4
2𝜋𝑒𝜎 2
0.4
𝐶 𝑌 = 4.13
0.3
fY(y)
fY(y)
0.3
0.2
0.2

Range = 4.13
0.1
0.1
0
-3
-2
-1
0
1
2
0
-1
3
0
1
2
3
4
5
y
y
Figure 2-1: Probability density of a standard Gaussian random variable (left) and a
uniform random variable of equal complexity (right)
Y ~ N( = [0, 0],  = I )
Y ~ U([0, 4.13], [0, 4.13])
2
0.2
0.15
fY(y1, y2)
fY(y1, y2)
0.2
0.1
0.05
𝐶 𝑌 =
0
2
2
0
y
-2
2
0
-2
y
2𝜋𝑒
𝑛
Σ
0.15
Area = 17.08
0.1
0.05
0
4
𝐶 𝑌 = 17.08
𝐶 𝑌 = 4.13 2
4
2
0
y
1
2
2
0
y
1
Y ~ U[0, 17.08]
0.2
fY(y)
0.15
0.1
0.05
Range = 17.08
0
0
5
10
15
y
Figure 2-2: Probability density of a multivariate Gaussian random variable with
µ = (0, 0) and Σ = I2 (top left), and uniform random variables of equal complexity
in R2 and R (top right and bottom, respectively)
C(Y ) =
√
2πeσ 2 = 4.13, which suggests that Y is equivalent in complexity to a
uniform random variable with a range of 4.13. This interpretation also holds in
higher dimensions. For example, if Y is instead a multivariate Gaussian random vari-
45
able with mean µ ∈ Rd and covariance Σ ∈ Rd×d , then C(Y ) represents the volume
of a d-dimensional hypercube that captures the intrinsic extent of Y , and there is
a corresponding uniform random variable in Rd with the same complexity. Alternatively, C(Y ) can be also be represented as the range of a uniform random variable in
R. Figure 2-2 illustrates this example for a standard multivariate Gaussian random
variable with d = 2.
2.2.3
Numerical Complexity Estimation
In realistic complex systems, estimates of the QOI are often generated from the
outputs of numerous model evaluations, such as through Monte Carlo simulation.
The resulting distribution for Y can typically be of any arbitrary form; that is, no
closed-form expression is available for fY (y). This precludes computing C(Y ) through
the exact evaluation of h(Y ) using (2.3), requiring instead numerical integration [105].
Assuming that fY (y) is Riemann-integrable, h(Y ) can be estimated by discretizing
fY (y) into k bins of size ∆(l) , where l = 1, ..., k. Letting fY (y (l) ) denote the probability
density of Y evaluated in the lth bin, and h(Y ∆ ) represent the estimate of h(Y )
computed using numerical approximation, we have [88]:
∆
h(Y ) = −
=−
=−
k
X
l=1
k
X
l=1
k
X
fY (y (l) )∆(l) log[fY (y (l) )∆(l) ]
fY (y (l) )∆(l)
fY (y )∆ log
∆(l)
(l)
(l)
(l)
(l)
(l)
(2.15)
(l)
(2.16)
fY (y )∆ log[fY (y )∆ ] +
l=1
k
X
fY (y (l) )∆(l) log ∆(l) .
(2.17)
l=1
If we select bins of equal size (∆
(l)
= ∆ ∀ l), then
k
X
fY (y (l) )∆ ≈ 1, which gives:
l=1
∆
h(Y ) = −
k
X
fY (y (l) )∆ log[fY (y (l) )∆] + log ∆.
l=1
46
(2.18)
Note that h(Y ∆ ) → h(Y ) as ∆ → 0. Figure 2-3 provides an illustration for evaluating
h(Y ∆ ) using this discretization procedure.
𝑓𝑌 𝑦
𝑓𝑌 𝑦 𝑙
∆
𝑦
bin 𝑙
Figure 2-3: Numerical approximation of h(Y ) using k bins of equal size
2.3
Related Quantities
In this section, we examine the relationship between exponential entropy and several other quantities that can be used to describe uncertainty. As its connection to
differential entropy and range have already been addressed in previous sections, we
will focus on how exponential entropy is related to variance and entropy power. We
also introduce the Kullback-Leibler divergence, which will play a key role in entropy
power-based sensitivity analysis, to be discussed in Chapter 3.
Table 2.1 lists the expressions for several uncertainty metrics for the four types
of distributions introduced in Section 1.2.3. Table 2.2 shows how the various metrics
are affected by the transformations shown in Figures 1-4(a) and 1-4(b), where a QOI
Y is updated to a new estimate Y 0 through scaling or shifting by a constant amount.4
4
In Figure 1-4(a), fY (y) is scaled by a constant α with no change in µY . To represent this update
using Y 0 = αY requires that µY = 0. Instead, the action depicted in Figure 1-4(a) is more accurately
described by Y 0 = α(Y − µY ). However, the results presented in Table 2.2 for multiplicative scaling
by α hold in general, and do not restrict Y to be zero mean.
47
b−a
Uniform
Y ∼ U[a, b]
b−a
Triangular
Y ∼ T (a, b, c)
Infinite
Gaussian
Y ∼ N (µY , σY )
(bL − aL ) + (bR − aR )
Bimodal Uniform
Y ∼ BU(aL , bL , aR , bR , fL , fR )
Range
ν(Y )
fL
fR 2
2
(b2 − aL2 ) + (bR
− aR
)
2 L
2
a+b
2
fL
L
fR
2πe
−2fL (bL −a ) −2fR (bR −aR )
fL
−fL (bL −aL ) −fR (bR −aR )
fR
−fL (bL − aL ) log fL − fR (bR − aR ) log fR
µY
Mean
E [Y ]
(b − a)2
12
σY2
log(2πeσY2 )
2
q
2πeσY2
a+b+c
3
Variance
var (Y )
log(b − a)
a2 + b2 + c2 − ab − ac − bc
18
1
b−a
+ log
2
2
√
(b − a) e
2
fL
fR 3
3
(b3 − aL3 ) + (bR
− aR
) − (E [Y ])2
3 L
3
Diff. Entropy
h(Y )
b−a
σY2
Exp. Entropy
C(Y )
(b − a)2
2πe
(b − a)2
8π
Entropy Power
N (Y )
Table 2.1: Range, mean, variance, differential entropy, exponential entropy, and entropy power for distributions that are
uniform, triangular, Gaussian, or bimodal with uniform peaks
48
Constant Scaling
Y 0 = αY, α > 0
Constant Shift
Y 0 = β + Y, β ∈ R
Variance
var (Y 0 )
α2 var (Y )
var (Y )
Diff. Entropy
h(Y 0 )
h(Y ) + log α
h(Y )
Exp. Entropy
C(Y 0 )
αC(Y )
C(Y )
Entropy Power
N (Y 0 )
α2 N (Y )
N (Y )
Table 2.2: The properties of variance, differential entropy, exponential entropy, and
entropy power under constant scaling or shift
2.3.1
Variance
Variance is one of the most widely used metrics for characterizing uncertainty in
system design. It provides a measure of the dispersion of a random variable about
its mean value [106]. For a continuous random variable Y , variance is denoted by
var (Y ) or σY2 , and can be expressed in integral form as:
var (Y ) =
σY2
Z
=
ZY
=
(y − µY )2 fY (y) dy
(2.19)
y 2 fY (y) dy − µ2Y ,
(2.20)
Y
where µY denotes the mean (also known as the expected value or first moment) of Y ,
given by:
Z
E [Y ] = µY =
yfY (y) dy.
(2.21)
Y
There are several advantages to using variance for quantifying uncertainty. For
most distributions, variance can easily be estimated from sample realizations of the
random variable, without explicit knowledge of the probability density.5 Furthermore,
5
It is important to note, however, that variance is not always well-defined. For example, the
variance of a Cauchy distribution, which has heavy tails, is undefined. However, its differential
49
there is a rich body of literature in variance-based sensitivity analysis, which provides
rigorous methods to apportion a system’s output variability into contributions from
the inputs (see Section 3.1.1).
Although variance and differential entropy can both be used to describe a system’s
uncertainty or complexity, in general there is no explicit relationship between the two
quantities [107, 108]. Consider the example shown in Figure 2-4: let Y be a bimodal
random variable consisting of two symmetric uniform peaks, such that fY (y) = 0.5
for −2 ≤ y ≤ −1 and 1 ≤ y ≤ 2, and zero elsewhere. We define Y G to be its corresponding “equivalent Gaussian distribution” — that is, a Gaussian random variable
with the same mean and variance, parameterized by Y G ∼ N (µY , σY ).6 Intuitively,
we might expect that the complexity of Y is less than that of Y G , as the former can
take on values in one of only two groups, whereas the latter has infinite support. For
this case, var (Y ) = var (Y G ) whereas C(Y ) < C(Y G ), and thus exponential entropy
more accurately captures the limited range of possible outcomes in Y as compared to
Y G.
0.6
Probability Density
0.5
fY(y)
Y
YG
var (·)
2.33
2.33
h(·)
0.69
1.84
C(·)
2.00
6.31
0.4
0.3
0.2
fYG(y)
0.1
0
−4
−2
0
2
4
y
Figure 2-4: Comparison of variance, differential entropy, and complexity between a
bimodal uniform distribution and its equivalent Gaussian distribution
Based on exponential entropy, the bimodal uniform distribution has the same
complexity as a uniform random variable whose range is equal to two. This illustrates
another key feature of our proposed complexity metric — unlike variance, which
entropy is given by log(γ) + log(4π), where γ > 0 is the shape parameter, and thus its exponential
entropy is well-defined. This represents one potential advantage of our proposed complexity metric,
in that it can be estimated for any probability density using the procedure outlined in Section 2.2.3.
6
The notion of an equivalent Gaussian distribution will be explored in more detail in Section 3.4,
where it is used to characterize the non-Gaussianity of the factors and outputs of a model.
50
measures dispersion about the mean, exponential entropy reflects the intrinsic extent
of a random variable irrespective of the expected value.
2.3.2
Kullback-Leibler Divergence and Cross Entropy
Another quantity relevant to our discussion of complexity is the Kullback-Leibler
(K-L) divergence, which is a measure of the difference between two probability distributions [109]. For continuous random variables Y1 and Y2 , the K-L divergence from
Y1 to Y2 is defined as:
Z
∞
DKL (Y1 ||Y2 ) =
fY1 (y) log
Z−∞
∞
fY1 (y)
dy
fY2 (y)
(2.22)
Z
∞
fY1 (y) log fY1 (y) dy −
=
−∞
fY1 (y) log fY2 (y) dy
(2.23)
−∞
= −h(Y1 ) + h(Y1 , Y2 ).
(2.24)
As shown in (2.24), the K-L divergence is equal to the sum of two terms: the
negative of the differential entropy h(Y1 ) and the cross entropy between Y1 and Y2 ,
defined as:
Z
∞
h(Y1 , Y2 ) = −
fY1 (y) log fY2 (y) dy.
(2.25)
−∞
Note that neither K-L divergence nor cross entropy is a true metric as they are
not symmetric quantities — that is, in general DKL (Y1 ||Y2 ) 6= DKL (Y2 ||Y1 ) and
h(Y1 , Y2 ) 6= h(Y2 , Y1 ). Because of this, K-L divergence is also commonly referred to as
the relative entropy of one random variable with respect to another, which is always
non-negative.7 In the limiting case where Y1 = Y2 , we have that h(Y1 , Y2 ) = h(Y1 )
and h(Y2 , Y1 ) = h(Y2 ), and thus DKL (Y1 ||Y2 ) = DKL (Y2 ||Y1 ) = 0.
7
In (2.22), Y2 is designated as the reference random variable. If fY2 (y) = 1 over the support set
Y2 (e.g., a uniform random variable with an extent of one), then DKL (Y1 ||Y2 ) equals −h(Y1 ). This
lends some intuition about the statement that differential entropy is a measure of randomness with
respect to a fixed reference [88, 91].
51
2.3.3
Entropy Power
In his seminal work on information theory, Shannon proposed not only the use of
differential entropy h(Y ) to quantify uncertainty in a random variable, but also introduced a derived quantity N (Y ), which he termed entropy power [91]:
N (Y ) =
exp[2h(Y )]
.
2πe
(2.26)
Entropy power is so named because in the context of communication systems, it
represents the power in a white noise signal that is limited to the same band as the
original signal and having the same differential entropy. As Shannon writes, “since
white noise has the maximum entropy for a given power, the entropy power of any
noise is less than or equal to its actual power” [91]. Due to the 2πe normalizing
constant in (2.26), entropy power also has the interpretation of the “variance of a
Gaussian random variable with the same differential entropy, [which is] maximum
and equal to the variance when the random variable is Gaussian” [110].8 That is to
say, for an arbitrary random variable Y , N (Y ) ≤ var (Y ), with equality in the case
where Y is a Gaussian random variable (see Table 2.1).
Additionally, entropy power is always non-negative, and is proportional to the
square of exponential entropy. Rewriting (2.26), we obtain:
exp[h(Y )] exp[h(Y )]
2πe
C(Y )2
=
.
2πe
N (Y ) =
(2.27)
Whereas C(Y ) is a measure of intrinsic extent and has the same units as Y , N (Y )
exhibits properties of variance and has the same units as Y 2 (see Table 2.2). This
provides a very appealing analogy for the relationship between standard deviation
and variance to that between exponential entropy and entropy power.
8
This is consistent with the principle of maximum entropy, which states that if we known nothing
about a system’s behavior except for the first and second moments, the distribution that maximizes
differential entropy (and therefore entropy power) given our limited knowledge is the Gaussian [104].
52
2.4
Estimating Risk
Similar to complexity, the concept of risk in system design is also difficult to formalize. In engineering systems, the risk associated with a particular event is often
characterized by two features: the probability of the event’s occurrence (likelihood)
and the severity of the outcome should the event occur (consequence) [5, 111]. In
one common risk management approach, these attributes are multiplied together to
generate a quantitative measure of risk associated with the event, which is then used
to set design priorities accordingly [112].
In this thesis, we focus only on the first component of risk, namely, the probability
of a system incurring an undesirable outcome, such as a technical failure or the
exceedance of a cost constraint. Quantitatively, we compute risk as the probability of
failing to satisfy a particular criterion [9, 113]. Note that in this definition, risk is the
complement of reliability, which instead describes the probability of success. Suppose
that a QOI is subject to the requirement that it must be greater than or equal to a
specified value r. In this case, risk corresponds to the probability that the random
variable Y takes on a value less than r, given by:
Z
r
fY (y) dy.
P (Y < r) =
(2.28)
−∞
This notion of risk is illustrated in Figure 2-5, where P (Y < r) corresponds to
the area of the shaded region in which the requirement is not met. Conversely,
P (Y > r) = 1 − P (Y < r).
𝑓𝑌 𝑦
𝑃 𝑌<𝑟
𝑦
𝑟
Figure 2-5: Probability of failure associated with the requirement r
53
2.5
Chapter Summary
In this chapter, we introduced complexity and risk as possible measures of uncertainty in a system with respect to a quantity of interest, and provided definitions
for these terms. In particular, the use of exponential entropy as a complexity metric
was described in detail, and placed in the context of information-based measures of
uncertainty.
In the following chapter, we will explore the topic of sensitivity analysis, which
is used to explain how the various factors of a model contribute to uncertainty in
the output, and how this information can be leveraged to reduce uncertainty in the
context of improving robustness and reliability.
54
Chapter 3
Sensitivity Analysis
The focus of this chapter is sensitivity analysis in the context of engineering system
design. Within the framework of Figure 1-2, sensitivity analysis takes place after
we have quantified uncertainty using metrics such as variance, complexity, and risk.
Sensitivity analysis allows us to better understand the effects of uncertainty in a
system in order to make well-informed decisions aimed at uncertainty reduction. For
example, it can be used to study how “variation in the output of a model ... can be
apportioned, qualitatively or quantitatively, to different sources of variation, and how
the given model depends upon the information fed into it” [114]. This information
can help to answer questions such as “What are the key factors that contribute
to variability in model outputs?” and “On which factors should research aimed at
reducing output variability focus?” [8].
Section 3.1 provides a brief overview of the sensitivity analysis literature. New
contributions of this thesis are presented in Sections 3.2 through 3.4. Specifically,
Section 3.2 extends existing variance-based sensitivity analysis methods to permit
changes to distribution family and examine the effect of input distribution shape
on output variance. Section 3.3 details how to predict local changes in complexity
and risk due to perturbations in mean and standard deviation, which have direct
implications for system robustness and reliability. Section 3.4 describes a sensitivity
analysis approach based on entropy power, in which uncertainty is decomposed into
contributions due to the intrinsic extent and non-Gaussianity of each factor. These
55
sections set the stage for Chapter 4, which explores how sensitivity information can be
combined with design budgets to guide decisions regarding the allocation of resources
for uncertainty mitigation.
3.1
Background and Literature Review
There are several key types of sensitivity analysis: factor screening, which identifies
the influential factors in a particular system; global sensitivity analysis (GSA), which
apportions the uncertainty in a model output over the entire response range into
contributions from the input factors; distributional sensitivity analysis (DSA), which
extends GSA to consider scenarios in which only a portion of a factor’s variance is
reduced; and regional sensitivity analysis, which is similar to GSA and DSA but
focuses instead on a partial response region — for example, the tail region of a
distribution in which some design requirement is not satisfied. The sections below
discuss several methods that are most relevant to the current thesis: variance-based
GSA and DSA, and relative entropy-based sensitivity analysis.
3.1.1
Variance-Based Global Sensitivity Analysis
Global sensitivity analysis uses variance as a measure of uncertainty, and seeks to
apportion a model’s output variance into contributions from each of the factors, as
well as their interactions. This notion is illustrated in Figure 3-1.
Figure 3-1: Apportionment of output variance in GSA [8, Figure 3-1]
There are two main methods to perform GSA: the Fourier amplitude sensitivity
test (FAST), which uses Fourier series expansions of the output function to repre-
56
sent conditional variances [115–117], and the Sobol’ method, which utilizes Monte
Carlo simulation [118–121]. The Sobol’ method is based on high-dimensional model
representation (HDMR), which decomposes a function g(X) into the following sum:
X
g(X) = g0 +
gi (Xi ) +
i
X
gij (Xi , Xj ) + . . . + g12...m (X1 , X2 , . . . , Xm ),
(3.1)
i<j
where g0 is a constant, gi (Xi ) is a function of Xi only, gij (Xi , Xj ) is a function of Xi
and Xj only, etc. Although (3.1) is not a unique representation of g(X), it can be
made unique by enforcing the following constraints:
Z
1
gi1 ,...,is (Xi1 , . . . , Xis ) dxk = 0,
∀
0
k = i1 , . . . , is ,
(3.2)
s = 1, . . . , m.
In the above equation, the indices i1 , . . . , is represent all sets of s integers that satisfy
1 ≤ i1 < . . . < is ≤ m. That is, for s = 1, the constraint given by (3.2) applies to
all sub-functions gi (Xi ) in (3.1); for s = 2, the constraint applies to all sub-functions
gij (Xi , Xj ) with i < j in (3.1), etc. Furthermore, in (3.2) we have defined all factors
Xi1 , . . . , Xis on the interval [0, 1]; this is merely for simplicity of presentation, and
not a requirement of the Sobol’ method. With the sub-functions gi1 ,...,is (Xi1 , . . . , Xis )
now uniquely specified, we refer to (3.1) as the analysis of variances high-dimensional
model representation (ANOVA-HDMR) of g(X).
In the ANOVA-HDMR, all sub-functions gi1 ,...,is (Xi1 , . . . , Xis ) are orthogonal and
zero-mean, and g0 equals the mean value of g(X). Assuming that g(X) and therefore
all comprising sub-functions are square-integrable, the variance of g(X) is given by:
Z
D=
g(X)2 dx − g02 ,
(3.3)
and the partial variances of gi1 ,...,is (Xi1 , . . . , Xis ) are defined as:
Z
Di1 ,...,is =
gi1 ,...,is (Xi1 , . . . , Xis )2 dxi1 . . . dxis .
57
(3.4)
Squaring then integrating both sides of (3.1), we obtain:
D=
X
Di +
i
X
Dij + . . . + D12...m ,
(3.5)
i<j
which states that the variance of the original function equals the sum of the variances
of the sub-functions, as shown in Figure 3-1. We will return to this notion of uncertainty apportionment in Section 3.4.1, where the ANOVA-HDMR is used in deriving
the entropy power decomposition.
In the Sobol’ method, global sensitivity indices are defined as:
Si1 ,...,is =
Di1 ,...,is
,
D
s = 1, . . . , m.
(3.6)
By definition, the sum of all sensitivity indices for a given function is one. Two
particularly important types of sensitivity indices are the main effect sensitivity index
(MSI) and the total effect sensitivity index (TSI). For a system modeled by (1.3), the
MSI of the ith factor, denoted by Si , represents the expected relative reduction in the
variance of Y if the true value of Xi is learned (i.e., the variance of Xi is reduced to
zero):
Si =
var (Y ) − E [var (Y |Xi )]
.
var (Y )
(3.7)
Alternatively, Si can be expressed as the ratio of the variance of gi (Xi ) to the variance
of g(X):
Si =
Di
.
D
(3.8)
Whereas the MSI signifies the contribution to output variance due to Xi alone,
the total effect sensitivity index ST i denotes the contribution to output variance due
to Xi and its interactions with all other factors. Using the notation X∼i to denote
all factors except Xi , ST i represents the percent of output variance that remains
unexplained after the true values of all factors except Xi have been learned:
ST i =
var (Y ) − E [var (Y |X∼i )]
.
var (Y )
58
(3.9)
Equivalently, ST i equals the sum of all global sensitivity indices of the form specified
by (3.6) that contain the subscript i.
For a particular model,
Pm
i=1
Si ≤ 1, whereas
Pm
i=1
ST i ≥ 1; deviation from unity
reflects the magnitude of the interaction effects. The various factors of a model
can be ranked according to their MSI or TSI in a factor prioritization or factor
fixing setting [122]. The goal of factor prioritization is to determine the factors that,
once their true values are known, would result in the greatest expected reduction in
output variability; these correspond to the factors with the largest values of Si . In
factor fixing, the objective is to identify non-influential factors that can be fixed at a
given value without substantially affecting the output variance; in this case, the least
significant factors are those with the smallest values of ST i .
3.1.2
Vary-All-But-One Analysis
A related concept to Sobol’-based GSA is vary-all-but-one analysis (VABO), in which
two Monte Carlo simulations are conducted to determine a factor’s contribution to
output variability [123, 124]. In the first MC simulation, all factors are sampled
from their nominal distributions, whereas in the second, one factor is fixed at a
particular point on its domain while the other factors are allowed to vary. The
observed difference in output variance between the two simulations is then attributed
to the contribution of the fixed factor to output variability.
While the VABO method is simple to implement, its main limitation is that there is
no clear guidance as to where each factor ought to be fixed on its domain; depending
on the chosen location, the resulting apportionment of variance among the factors
can change [122]. By contrast, GSA circumvents this problem by accounting for all
possible values of each factor within its domain, such that the resulting sensitivity
index represents the expected reduction in variance, where the expectation is taken
over the distribution of values at which the factor can be fixed.
59
3.1.3
Distributional Sensitivity Analysis
Building on global sensitivity analysis methods, distributional sensitivity analysis
(DSA) can also be applied to parameters whose uncertainty is epistemic in nature.
The DSA procedure was developed by Allaire and Willcox in order to address one of
the inherent limitations of GSA [8, 125]: the assumption that all epistemic uncertainty
associated with a particular factor may be reduced to zero through further research
and improved knowledge. This generalization is optimistic, and can in fact lead
to inappropriate allocations of resources. Distributional sensitivity analysis, on the
other hand, treats the portion of a factor’s variance that can be reduced as a random
variable, and therefore may be more appropriate than GSA for the prioritization of
efforts aimed at uncertainty reduction, as it could convey for which input(s) directed
research will yield the greatest return.
Letting Xio be the random variable corresponding to the original distribution for
factor i, and Xi0 be the random variable whose distribution represents the uncertainty
in that factor after some design modification, the ratio of remaining (irreducible)
variance to initial variance for factor i is defined by the quantity δ:
δ=
var (Xi0 )
.
var (Xio )
(3.10)
When we do not know the true value of δ, we treat it as a uniform random
variable on the interval [0, 1]. Whereas GSA computes for each factor Xi the main
effect sensitivity index Si , the analogous quantity in DSA is adjSi (δ), the adjusted
main effect sensitivity index of Xi given that it is known that only 100(1 − δ)% of its
variance can be reduced.1 Additionally, the average adjusted main effect sensitivity
index, or AASi , is the expected value of adjSi (δ) over all δ ∈ [0, 1].
Another advantage of DSA is that it does not require additional MC simulations;
rather, it can be implemented by reusing samples from a previous Sobol’-based GSA.
Finally, the current DSA methodology assumes that a design activity that reduces
the variance of Xi by 100(1−δ)% does not change the distribution family to which Xi
1
The main effect sensitivity indices computed from GSA correspond to the case where δ = 0.
60
2
belongs. That is to say, if Xi is initially a Gaussian random variable with variance σX
,
i
its distribution remains Gaussian after the specified activity, with an updated variance
2
of δσX
. In Section 3.2, we extend the existing DSA methodology to permit changes
i
to a factor’s distribution type. This contribution will complement the discussion in
Section 3.4, which demonstrates that a distribution’s shape can play a significant role
in uncertainty estimation.
3.1.4
Relative Entropy-Based Sensitivity Analysis
In addition to variance-based methods, there has also been work done to develop
methods for global and regional sensitivity analysis using information entropy as a
measure of uncertainty. One method uses the Kullback-Leibler divergence, or relative
entropy, to quantify the distance between two probability distributions [126].
Suppose that Y o represents the original estimate of a system’s quantity of interest,
and Y 0 the modified result once some factor Xi has been fixed at a particular value
(e.g., its mean), as in a VABO approach. In relative entropy-based sensitivity analysis,
(2.22) is used to compute the K-L divergence between the two QOI estimates, which
serves to quantify the impact of the factor that has been fixed. That is to say, the
larger the value of DKL (Y o ||Y 0 ), the more substantial the contribution of factor i to
uncertainty in the QOI.
This approach can be used to perform both global and regional sensitivity analysis
by simply adjusting the limits of integration, and is suitable for distributions where
variance is not a good measure of uncertainty (e.g., bimodal or skewed distributions).
However, as in the VABO approach, the results of relative entropy-based sensitivity
analysis vary depending on where a particular factor is fixed on its domain. Furthermore, the K-L divergence is not normalized and does not have physical meaning —
it can be used to rank factors, but does not have a proportion interpretation such as
shown in Figure 3-1 for variance-based sensitivity indices.
61
3.2
Extending Distributional Sensitivity Analysis
to Incorporate Changes in Distribution Family
In the existing distributional sensitivity analysis methodology, it is assumed that when
a design activity is performed that reduces the variance of factor X by 100(1 − δ)%,
only the parameters of X’s distribution are altered, not the type of distribution itself.
If the activity were to also change the distribution family to which X belongs (e.g.,
from Gaussian to uniform), any additional fluctuations expected in the variance of
the QOI due to the switch would not be reflected in the computed adjusted main
effect sensitivity indices. Of this, Allaire writes:
“[If] the distribution family of a given factor was expected to change
through further research (e.g. from an original uniform distribution to a
distribution in the triangular family), then reasonable distributions from
the new family, given that the original distribution was from another
family, could be defined” [8, pp. 60].
Here, we extend DSA to permit changes in distribution family by defining reasonable new factor distributions, which represent the result of further research or additional knowledge about a factor [8]. These new distributions reflect a 100(1 − δ)%
reduction in the factor’s variance; however, for each value of δ ∈ [0, 1], there is usually
no unique distribution, and thus we average over k possible reasonable distributions
to estimate the adjusted main effect sensitivity index.2
As a note on terminology, we will use X o to denote the original random variable for
some factor, and X 0 to represent the updated random variable such that var (X 0 ) =
δvar (X o ), as in (3.10). Allaire’s DSA methodology uses acceptance/rejection (AR)
sampling to generate samples from a desired distribution by sampling from a different
distribution. In the current work, we also employ AR sampling to obtain samples of
X 0 (the desired distribution) from samples of X o ; in order to do this, there must
2
To illustrate this point, Allaire provides the example of a factor X that is uniformly distributed
√
on the√interval [0,√1]. For δ√= 0.5, there are infinitely many new distributions (e.g., U[0, 2/2],
U[1 − 2/2, 1], U[ 2/4, 1 − 2/4], etc.) for which the variance is 50% of the original variance.
62
exist a constant κ with the property that κfX o (x) ≥ fX 0 (x) for all x [105]. The AR
sampling procedure is outlined below; the steps may be repeated until sufficiently
many samples of X 0 have been obtained.
Acceptance/rejection method
1. Draw a sample, xo , from X o .
2. Draw a sample, u, from a uniform random variable on [0, 1].
3. If
fX 0 (xo )
fX o (xo )
≥ κu, let x0 = xo .
4. Otherwise, discard xo and return to Step 1.
In Sections 3.2.1 through 3.2.3, we propose procedures to select reasonable new
distributions and perform AR sampling for cases where X o and X 0 are uniform,
triangular, or Gaussian. For these cases, the analytical expressions for δ are listed
in Table 3.1. Note that the proposed procedures represent but one possible set of
methods to generate reasonable distributions for X 0 , and are not intended to be
definitive. Accordingly, the distributions obtained using the procedures are but a
subset of possible manifestations of X 0 , and thus should not be taken to be exhaustive.
In Appendix A, we will examine the corresponding cases in which X 0 is a bimodal
uniform random variable.
63
Xo
X0
Uniform
U[ao , bo ]
Uniform
U[a0 , b0 ]
2
b0 − a0
b o − ao
(b0 − a0 )2
12σ o2
2 02
(a
3
Triangular
T (a0 , b0 , c0 )
+ b02 + c02 − a0 b0 − a0 c0 − b0 c0 )
(bo − ao )2
a02 + b02 + c02 − a0 b0 − a0 c0 − b0 c0
18σ o2
Gaussian
N (µ0 , σ 0 )
12σ 02
(bo − ao )2
σ 02
σ o2
3 0
(b − a0 )2
Triangular
a02 + b02 + c02 − a0 b0 − a0 c0 − b0 c0
18σ 02
2
T [ao , bo , co ] ao2 + bo 2 + co2 − ao bo − ao co − bo co ao2 + bo 2 + co2 − ao bo − ao co − bo co ao2 + bo 2 + co2 − ao bo − ao co − bo co
Gaussian
N [µo , σ o ]
0
var (X )
for changes in factor distribution between the uniform, triangular, and Gaussian families.
Table 3.1: Ratio of δ = var
(X o )
Rows represent the initial distribution for X o and columns correspond to the updated distribution for X 0 .
64
3.2.1
Cases with No Change in Distribution Family
First, we focus on cases where X o and X 0 belong to the same family of distributions
(i.e., the diagonal entries of Table 3.1). In these cases, the extent of X 0 is always
less than or equal to that of X o , and thus AR sampling works well for generating
samples of X 0 from samples of X o . The procedures provided below are adapted from
Allaire [8]. Examples of reasonable distributions generated using the procedures are
Probability density
provided in Figure 3-2 for select values of δ and k.
3.5
7
3
6
2.5
5
2
4
1.5
3
1
2
0.5
1
0
−0.2
0
0.2
0.4
0.6
0.8
1
0
0
1.2
0.2
0.4
0.6
0.8
1
x
(a) Uniform to uniform: X o ∼ U[0, 1], δ = 0.1,
k=3
(b) Triangular to triangular: X o ∼ T (0, 1, 0.7),
δ = 0.5, k = 3
0.8
Probability density
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
−3
−2
−1
0
x
1
2
3
(c) Gaussian to Gaussian: X o ∼ N (0, 1),
δ = 0.3, k = 1
Figure 3-2: Examples of reasonable uniform, triangular, and Gaussian distributions.
In each figure, the solid red line denotes fX o (x), the dashed blue lines represent various
possibilities for fX 0 (x), and the dotted black lines show κfX o (x), which is greater than
or equal to the corresponding density fX 0 (x) for all x.
65
In the uniform to uniform and triangular to triangular cases, a key distinction
between the current work and the procedures proposed by Allaire is that the k distributions for X 0 are made to be evenly spaced on the interval [ao , bo ]; this is to
ensure the inclusion of a wide spectrum of possible distributions when estimating the
adjusted main effect sensitivity index.
Uniform to uniform: X o ∼ U[ao , bo ] → X 0 ∼ U[a0 , b0 ]
Selecting a0 and b0 :
1. From δ =
b0 −a0 2
,
bo −ao
solve for extent L = b0 − a0 .
2. Evenly space k samples of b0 on the interval [ao + L, bo ].
3. For each sample of b0 , find a0 = b0 − L.
Specifying κ:
1. Set κ =
bo −ao
.
b0 −a0
For triangular distributions, the variance is not solely a function of extent, but
√
also of the peak location. For a factor X o , the extent can vary between 18δσ o2 and
√
24δσ o2 . The former case occurs when the peak coincides with either the lower or
upper bound (i.e., co = ao or co = bo ), resulting in a skewed distribution. The latter
case corresponds to a symmetric triangular distribution in which the peak coincides
with the mean value (i.e., co = (ao + bo )/2).
Triangular to triangular: X o ∼ T (ao , bo , co ) → X 0 ∼ T (a0 , b0 , c0 )
Selecting a0 , b0 , and c0 :
1. Compute the minimum and maximum possible extent for X 0 : Lmin =
√
√
18δσ o2 and Lmax = 24δσ o2 .
2. Evenly space k samples of L on the interval [Lmin , Lmax ].
3. For each sample of L, randomly sample midpoint M from the interval [ao , bo ].
• Find a0 = M −
L
2
and b0 = M + L2 .
66
• If a0 ≥ ao and b0 ≤ bo , accept a0 and b0 . Otherwise, repeat Step 3.
4. From δ =
a02 +b02 +c02 −a0 b0 −a0 c0 −b0 c0
,
ao2 +bo 2 +co2 −ao bo −ao co −bo co
solve for c0 . Select the root that is closer
to co (this is to circumvent the problem of sample impoverishment towards
the tails of the distribution).
Specifying κ:
1. Using (1.6), evaluate fX 0 (c0 ) and fX o (c0 ).
2. Set κ =
fX 0 (c0 )
.
fX o (c0 )
Gaussian to Gaussian: X o ∼ N (µo , σ o ) → X 0 ∼ N (µ0 , σ 0 )
Selecting µ0 and σ 0 :
1. Set µ0 = µo .
√
2. Set σ 0 = δσ o2 .
Specifying κ:
1. Using (1.7), evaluate fX 0 (c0 ) and fX o (c0 ).
2. Set κ =
3.2.2
fX 0 (µ0 )
.
fX o (µ0 )
Cases Involving Truncated Extent for X o
In this section, we consider three cases in which it is not always feasible to generate
samples of X 0 from the entire distribution for X o . These cases are triangular to
uniform, Gaussian to uniform, and Gaussian to triangular, which form the lower left
entries of Table 3.1. In the first case, the probability density of a triangular random
variable X o equals zero at ao and bo ; thus, there are very few samples of X o in the
tails of the distribution, making it nearly impossible to generate samples of X 0 from
those regions using the AR method. Similarly, in the second and third cases, X o is
Gaussian and has infinite extent; however, sample impoverishment in the tails again
limits the feasibility of the AR method to generate sufficiently many samples of X 0 .
67
To circumvent this problem, we instead use the AR method to generate samples
of X 0 from a truncated approximation for X o . We specify lower and upper bounds
(BL and BU , respectively) for the interval on which X 0 can be supported, such that
the vast majority of the probability mass of X o is contained in [BL , BU ].
The lower and upper bounds can be computed for a desired level using the cumulative distribution function FX o (x), which is equal to the area under the probability
density fX o (x) from −∞ to x:
FX o (x) = P (X o ≤ x)
Z x
=
fX o (ξ) dξ.
(3.11)
(3.12)
−∞
As an example, to specify the bounds [BL , BU ] that delimit the middle 95% of the
probability mass of X o , we can simply solve (3.12) for the values of x corresponding
to FX o (x) = 0.025 and FX o (x) = 0.975, respectively. The analytical expressions for
the cumulative distribution function of triangular and Gaussian random variables are
given in (3.13) and (3.14), respectively:
X o ∼ T (ao , bo , co ) :
o
o
o
X ∼ N (µ , σ ) :

(x − a)2




(b − a)(c − a)




(b − x)2
FX o (x) = 1 −

(b − a)(b − c)






0
if a ≤ x ≤ c,
if c < x ≤ b,
(3.13)
otherwise.
x − µo
1
FX o (x) =
1 + erf √
.
2
2σ o2
(3.14)
Empirical evidence suggests that reasonable bounds for the three cases in this
section (and the percentage of the area under fX o (x) that is captured), are:
• Triangular to uniform: BL = FX−1o (0.025), BU = FX−1o (0.975); middle 95%.
• Gaussian to uniform: [BL , BU ] = µo ∓ 2σ o ; middle 95.4%.
• Gaussian to triangular: [BL , BU ] = µo ∓ 3σ o ; middle 99.7%.
68
The procedures to estimate the parameters of X 0 and specify κ for AR sampling are
listed below. Examples of reasonable distributions for these three cases are shown in
12
3
10
2.5
Probability density
Probability density
Figure 3-3 for select values of δ and k.
8
6
4
2
0
0
2
1.5
1
0.5
0.2
0.4
0.6
0.8
0
−2
1
−1.5
−1
−0.5
x
(a) Triangular to uniform: X o ∼ T (0, 1, 0.3),
δ = 0.5, k = 2
0
x
0.5
1
1.5
2
(b) Gaussian to uniform: X o ∼ U[0, 1], δ = 0.7,
k=2
1.4
Probability density
1.2
1
0.8
0.6
0.4
0.2
0
−3
−2
−1
0
x
1
2
3
(c) Gaussian to triangular: X o ∼ N (0, 1),
δ = 0.6, k = 2
Figure 3-3: Examples of reasonable uniform, triangular, and Gaussian distributions
for cases involving a truncated extent for X o
69
Triangular to uniform: X o ∼ T (ao , bo , co ) → X 0 ∼ U[a0 , b0 ]
Selecting a0 and b0 :
1. Using (3.13), find bounds BL = FX−1o (0.025) and BU = FX−1o (0.975).
2. From δ =
3(b0 −a0 )2
,
2(ao2 +bo 2 +co2 −ao bo −ao co −bo co )
solve for extent L = b0 − a0 .
3. Evenly space k samples of b0 on the interval [BL + L, BU ].
4. For each sample of b0 , find a0 = b0 − L.
Specifying κ:
1. Using (1.6), evaluate fX o (a0 ) and fX o (b0 ).
2. If fX o (a0 ) ≤ fX o (b0 ), set κ =
1
.
fX o (a0 )(b0 −a0 )
Otherwise, set κ =
1
.
fX o (b0 )(b0 −a0 )
Gaussian to uniform: X o ∼ N (µo , σ o ) → X 0 ∼ U[a0 , b0 ]
Selecting a0 and b0 :
1. Set lower bound BL = µo − 2σ o and upper bound BU = µo + 2σ o .
2. From δ =
(b0 −a0 )2
,
12σ o2
solve for extent L = b0 − a0 .
3. Evenly space k samples of b0 on the interval [BL + L, BU ].
4. For each sample of b0 , find a0 = b0 − L.
Specifying κ:
1. Using (1.7), evaluate fX o (a0 ) and fX o (b0 ).
2. If fX o (a0 ) ≤ fX o (b0 ), set κ =
1
.
fX o (a0 )(b0 −a0 )
Otherwise, set κ =
1
.
fX o (b0 )(b0 −a0 )
Gaussian to triangular: X o ∼ N (µo , σ o ) → X 0 ∼ T (a0 , b0 , c0 )
Selecting a0 , b0 , and c0 :
1. Compute the minimum and maximum possible extent for X 0 : Lmin =
√
√
18δσ o2 and Lmax = 24δσ o2 .
70
2. Evenly space k samples of L on the interval [Lmin , Lmax ].
3. Set lower bound BL = µo − 3σ o and upper bound BU = µo + 3σ o .
4. For each sample of L, randomly sample midpoint M from the interval
[BL , BU ].
• Find a0 = M −
L
2
and b0 = M + L2 .
• If a0 ≥ BL and b0 ≤ BU , accept a0 and b0 . Otherwise, repeat Step 4.
5. From δ =
a02 +b02 +c02 −a0 b0 −a0 c0 −b0 c0
,
18σ o2
solve for c0 . Select the root that is closer to
µo .
Specifying κ:
1. If a0 6= c0 , find αL and a0 < xL < c0 such that αL fX o (x) and fX 0 (x) are
tangent at xL . Otherwise, set xL = a0 .
2. If b0 6= c0 , find αR and c0 < xR < b0 such that αR fX o (x) and fX 0 (x) are
tangent at xR . Otherwise, set xR = b0 .
3. Using (1.6) and (1.7), find κL =
fX 0 (xL )
,
fX o (xL )
κR =
fX 0 (xR )
,
fX o (xR )
and κC =
fX 0 (c0 )
.
fX o (c0 )
4. Set κ = max{κL , κR , κC }.
3.2.3
Cases Involving Resampling
Finally, we consider the uniform to triangular, uniform to Gaussian, and triangular
to Gaussian cases, which comprise the upper right corner of Table 3.1. In these cases,
AR sampling may not be a valid method to generate samples of X 0 from the existing
samples of X o . This is because for larger values of δ, although var (X 0 ) < var (X o ),
the extent of X 0 can actually exceed that of X o . For example, in Figure 3-4(a), X o is a
uniform random variable with variance σ o2 ; two possibilities for X 0 are shown, where
X 0 is a triangular random variable and δ = 0.9. In both cases the new extent b0 − a0
exceeds original extent bo − ao , making it impossible to generate samples in the tail
regions of X 0 through AR sampling. Therefore, in order to compute the adjusted main
71
effect sensitivity index for δ = 0.9, we must perform a new Monte Carlo simulation to
generate samples of X 0 and propagate them through the appropriate analysis models
to obtain estimates of Y .
Recall that the minimum extent for a triangular distribution is Lmin =
√
18δσ o2 ;
for Lmin to be wholly encompassed within the bounds of a uniform distribution on
[ao , bo ] requires that δ ≤ 2/3. Thus, in the uniform to triangular case, AR sampling
is feasible for δ ≤ 2/3; otherwise, resampling is necessary.
For the uniform to Gaussian and triangular to Gaussian cases, the extent necessarily increases between X o and X 0 due the Gaussian distribution having infinite
support. Here, we employ the same strategy as in Section 3.2.2 by using a truncated
distribution to approximate X 0 and ensuring that it lies entirely within the interval
[ao , bo ]. By requiring that ao ≤ µ0 − 3σ 0 and bo ≥ µ0 + 3σ 0 , this approximation is
sufficient to capture more than 99.7% of the probability mass of X 0 . This condition
is satisifed for δ ≤ 1/3 in the uniform to Gaussian case; in the triangular to Gaussian
case, the corresponding threshold varies between 1/2 and 2/3, depending on the skewness of the triangular distribution for X o . If δ exceeds the specified value, resampling
must be used in lieu of the AR method to generate samples of X 0 .
For the three cases in this section, the procedures for specifying the parameters
of X 0 and generating samples from the distribution are provided below. Examples of
reasonable distributions for these cases are shown in Figure 3-4 for select values of δ
and k.
72
3
2.5
2.5
Probability density
Probability density
3
2
1.5
1
1.5
1
0.5
0.5
0
−0.2
2
0
0.2
0.4
0.6
0.8
1
0
−0.2
1.2
0
0.2
0.4
x
0.6
0.8
1
1.2
x
3
3
2.5
2.5
Probability density
Probability density
(a) Uniform to triangular: X o ∼ U[0, 1], δ = 0.9, (b) Uniform to triangular: X o ∼ U[0, 1], δ = 0.4,
k=2
k=3
2
1.5
1
1.5
1
0.5
0.5
0
−0.2
2
0
0.2
0.4
0.6
0.8
1
0
−0.2
1.2
0
0.2
0.4
x
0.6
0.8
1
1.2
x
6
6
5
5
Probability density
Probability density
(c) Uniform to Gaussian: X o ∼ U[0, 1], δ = 0.7, (d) Uniform to Gaussian: X o ∼ U[0, 1], δ = 0.3,
k=1
k=1
4
3
2
3
2
1
1
0
0
4
0.2
0.4
0.6
0.8
0
0
1
x
(e) Triangular to Gaussian: X o ∼ T (0, 1, 0.8),
δ = 0.5, k = 1
0.2
0.4
0.6
0.8
1
x
(f) Triangular to Gaussian: X o ∼ T (0, 1, 0.8),
δ = 0.2, k = 1
Figure 3-4: Examples of reasonable uniform, triangular, and Gaussian distributions
for cases involving resampling. The left column shows instances where δ is too large,
and resampling is required. The right column shows instances where δ is sufficiently
small, and AR sampling can be used to generate samples of X 0 from samples of X o .
73
Uniform to triangular: X o ∼ U[ao , bo ] → X 0 ∼ T (a0 , b0 , c0 )
Selecting a0 , b0 , and c0 :
1. Compute Lmin =
√
√
18δσ o2 and Lmax = 24δσ o2 .
2. Sample L and set lower and upper bounds for sampling M .
• If δ ≤ 2/3:
– Evenly space k samples of L on the interval [Lmin , min{Lmax , bo −
ao }].
– Set lower bound BL = ao and upper bound BU = bo .
• If δ > 2/3:
– Evenly space k samples of L on the interval [Lmin , Lmax ].
– Set lower bound BL = bo − Lmax and upper bound BU = ao + Lmax .
3. For each sample of L, randomly sample midpoint M from the interval
[BL , BU ].
• If L = BU − BL , set M =
• Find a0 = M −
L
2
BL +BU
.
2
and b0 = M + L2 .
• If a0 ≥ BL and b0 ≤ BU , accept a0 and b0 . Otherwise, repeat Step 3.
4. From δ =
2(a02 +b02 +c02 −a0 b0 −a0 c0 −b0 c0 )
,
3(bo −ao )2
solve for c0 . Select either root at random.
Specifying κ:
• If δ ≤ 2/3, set κ = (bo −ao )fX 0 (c0 ) and use AR sampling to generate samples
of X 0 ∼ T (a0 , b0 , c0 ) from samples of X o ∼ U[ao , bo ].
• If δ > 2/3, use MC sampling to generate samples of X 0 ∼ T (a0 , b0 , c0 ).
74
Uniform to Gaussian: X o ∼ U[ao , bo ] → X 0 ∼ N (µ0 , σ 0 )
Selecting µ0 and σ 0 :
1. Set µ0 =
0
2. Set σ =
ao +bo
.
2
q
δ(bo −ao )2
.
12
Specifying κ:
• If δ ≤ 1/3, set κ = (bo −ao )fX 0 (µ0 ) and use AR sampling to generate samples
of X 0 ∼ N (µ0 , σ 0 ) from samples of X o ∼ U[ao , bo ].
• If δ > 1/3, use MC sampling to generate samples of X 0 ∼ N (µ0 , σ 0 ).
Triangular to Gaussian: X o ∼ T [ao , bo , co ] → X 0 ∼ N (µ0 , σ 0 )
Selecting µ0 and σ 0 :
1. Set µ0 =
2. Set σ 0 =
ao +bo +co
.
3
q
δ(ao2 +bo 2 +co2 −ao bo −ao co −bo co )
.
18
Specifying κ:
• If ao ≤ µ0 − 3σ 0 and bo ≥ µ0 + 3σ 0 :
1. Using (1.6) and (1.7), find x∗ and α such that αfX o (x) and fX 0 (x) are
tangent at x∗ .
2. Set κ =
fX 0 (x∗ )
fX o (x∗ )
and use AR sampling to generate samples of X 0 ∼
N (µ0 , σ 0 ) from samples of X o ∼ T [ao , bo , co ].
• Otherwise, use MC sampling to generate samples of X 0 ∼ N (µ0 , σ 0 ).
3.3
Local Sensitivity Analysis
In this section, we explore how complexity and risk estimates are affected by small
changes in the mean or standard deviation of a system’s design parameters or QOI.
Revisiting Figures 1-4(a) and 1-4(b), we consider the separate cases of a constant
75
scaling of fY (y) that shrinks the standard deviation versus a constant shift in fY (y)
that alters the mean value. These design activities have direct implications for system
robustness and reliability. The results derived herein represent local sensitivities
obtained by linearizing the system about the current estimate.
3.3.1
Local Sensitivity to QOI Mean and Standard Deviation
To evaluate the local sensitivity of complexity and risk with respect to design activities
that perturb the mean (µY ) or standard deviation (σY ) of the QOI Y , we compute
the partial derivative of C(Y ) and P (Y < r) with respect to µY and σY . The partial
derivatives can then be linearized about the current design to predict the change
in complexity (∆C) or risk (∆P ) associated with a perturbation in mean (∆µY ) or
standard deviation (∆σY ). These expressions are derived in Appendix B; the key
results are summarized in Table 3.2.
Sensitivity of C(Y ) with respect to µY and σY
Partial derivatives
∂C(Y )
=0
∂µY
∂C(Y )
C(Y )
=
∂σY
σY
(3.15)
(3.16)
Local approximations
∆C(Y ) = 0
(3.17)
∆C ≈
∆σY
C(Y )
σY
(3.18)
Sensitivity of P (Y < r) with respect to µY and σY
Partial derivatives
∂P (Y < r)
= −fY (r)
∂µY
∂P (Y < r)
(µY − r)
=
fY (r)
∂σY
σY
(3.19)
(3.20)
Local approximations
∆P ≈ −∆µY fY (r)
(3.21)
∆P ≈
∆σY
(µY − r)fY (r)
σY
(3.22)
Table 3.2: Local sensitivity analysis results for complexity and risk
From (3.15), we see that perturbations in µY do not affect complexity. This is
76
unsurprising, as we know that C(Y ) contains no information about the expected value
of Y . From (3.16), we see that the sensitivity of complexity to σY is a constant, and
simply equals the ratio of C(Y ) to σY . Since both C(Y ) and σY are non-negative,
this implies that as a local approximation (see (3.18)), reducing standard deviation
also reduces complexity.
More interestingly, (3.19) and (3.20) show that the sensitivity of risk to both µY
and σY is proportional to the probability density of Y evaluated at the requirement
r. Since fY (y) ≥ 0 for all values of y, this implies that the sign of ∆µY or µY − r
determines whether risk is increased or decreased (see (3.21) and (3.22)). Figure 3-5
helps to illustrate this point for (3.22), where a reduction in standard deviation can
alter risk in either direction, depending on the relative
locations of r and µY .
Refinement: Change Std Dev
Refinement: Change Std Dev
0.8
𝑓𝑌 𝑦
0.6
f X(x)
f X(x)
0.6
0.4
𝑃 𝑌<𝑟
0.2
0
-0.2
-4
𝑓𝑌 𝑦
0.8
-2
𝑃 𝑌<𝑟
0.2
𝑦0
𝑟 𝜇𝑌
-3
0.4
(a)
µY −
r > 10
-1
0
2
3
𝑦
𝜇𝑌 𝑟
-0.2
-4
4
x
-3
-2
(b) µY − r < 0
-1
0
1
2
3
4
x
Figure 3-5: The relative locations of r and µY greatly impact the change in risk
associated with a decrease in σY . Moving from the red probability density to the
blue, P (Y < r) decreases if µY − r > 0, and increases if µY − r < 0.
3.3.2
Relationship to Variance-Based Sensitivity Indices
In this section, we seek to relate the local approximations in (3.18) and (3.22) to
sensitivity indices computed from variance-based GSA and DSA. We let Y o and Y 0
represent initial and new estimates of the QOI (the red and blue densities in Figure
3-5, respectively) corresponding to a design activity that reduces variance (and thus
standard deviation). Similarly, we let var (Y o ) and var (Y 0 ) represent the variance of
77
the QOI before and after the update, respectively. If the activity is one that results
in learning the true value of factor i, then we know from (3.7) that:
var (Y 0 ) = var (Y o ) − Si var (Y o ).
(3.23)
The ratio of var (Y 0 ) to var (Y o ) is given by:
var (Y 0 )
= 1 − Si .
var (Y o )
(3.24)
We can relate the above ratio to the quantity ∆σY /σY from (3.18) and (3.22):
p
p
var (Y 0 ) − var (Y o )
p
var (Y o )
s
var (Y 0 )
−1
=
var (Y o )
p
= 1 − Si − 1.
∆σY
=
σY
(3.25)
This allows us to rewrite (3.18) and (3.22) as:
p
∆C ≈ ( 1 − Si − 1)C(Y ),
p
∆P ≈ ( 1 − Si − 1)(µY − r)fY (r),
(3.26)
(3.27)
noting that C(Y ), µY , and fY (r) in the above expressions refer to the complexity,
mean, and probability density (evaluated at r) of the QOI for the initial design.
Finally, if the design activity is instead one that reduces the variance of factor i by
100(1 − δ)%, then (3.23) can be modified to:
var (Y 0 ) = var (Y o ) − adjSi (δ)var (Y o ),
(3.28)
and we can similarly substitute adjSi (δ) for Si in (3.26) and (3.27).
Since both Si and adjSi (δ) can only assume values in the interval [0, 1], the above
local approximations imply that a decrease in variance is concurrent with a reduction
78
in complexity, as well as in risk if µY − r > 0. While an attractive result, these
relations are simply local approximations and should not be generalized. As we will
demonstrate in Chapter 4, variance, complexity, and risk each describes a different
aspect of system uncertainty, and can therefore trend in opposite directions.
3.3.3
Local Sensitivity to Design Parameter Mean and Standard Deviation
In order to use sensitivity information about the QOI to update the design, it is
necessary to translate this information into tangible actions for modifying the design
parameters. For this, we extend (3.19) and (3.20) to compute the sensitivity of risk
to perturbations in the mean or standard deviation of the design parameters.
We define the vectors µX = [µX1 , µX2 , . . . , µXm ]T and σX = [σX1 , σX2 , . . . , σXm ]T ,
which consist of the mean and standard deviation estimates of the entries of X. Using
the chain rule, the partial derivative of P (Y < r) with respect to µXi is given by:
∂P (Y < r) ∂µY
∂P (Y < r)
=
.
∂µXi
∂µY
∂µXi
(3.29)
To estimate ∂µY /∂µXi , we use (1.4) and make the following approximation for µY :
µY = E [g(X)] ≈ g(E [X]) = g(µX ).
(3.30)
This approximation is exact if g(X) is linear. Therefore, we have:
∂µY
∂g(µX )
≈
.
∂µXi
∂Xi
(3.31)
Combining (3.19), (3.29), and (3.31), we obtain:
∂P (Y < r)
∂g(µX )
≈ −fY (r)
,
∂µXi
∂Xi
∂g(µX )
∆P ≈ −∆µXi fY (r)
.
∂Xi
79
(3.32)
(3.33)
Similarly, the partial derivative of P (Y < r) with respect to σXi is given by:
∂P (Y < r)
∂P (Y < r) ∂σY
=
.
∂σXi
∂σY
∂σXi
(3.34)
To approximate ∂σY /∂σXi , we can use results from DSA to estimate ∆σY /∆σXi . The
adjusted main effect sensitivity index adjSi (δ) relates the expected variance remaining
in Y (given by (3.28)) to the variance remaining in factor i (given by (3.10)) as δ
ranges between 0 and 1, which allows us to compute ∆σY /∆σXi as follows:
p
p
var (Y o ) − adjSi (δ)var (Y o ) − var (Y o )
p
p
δvar (Xio ) − var (Xio )
p
σY ( 1 − adjSi (δ) − 1)
√
=
.
σXi ( δ − 1)
∆σY
=
∆σXi
(3.35)
Thus, we can combine (3.20), (3.34) and (3.35) to obtain:3
p
fY (r)(µY − r)( 1 − adjSi (δ) − 1)
∂P (Y < r)
√
≈
,
∂σXi
σXi ( δ − 1)
p
∆σXi fY (r)(µY − r)( 1 − adjSi (δ) − 1)
√
∆P ≈
.
σXi ( δ − 1)
(3.36)
(3.37)
The key result of this section is that we can relate system risk to the specific
changes in the design parameters. As an example, let ∆P denote the desired change
in risk. Rearranging (3.33) and (3.37), we can estimate the requisite change in the
mean and standard deviation of each design parameter in order to achieve that goal.
For small values of ∆P , these changes are approximated by:
∆µXi
∆σXi
−1
∆P ∂g(µX )
≈−
,
fY (r)
∂Xi
√
∆P σXi ( δ − 1)
p
≈
.
fY (r)(µY − r)( 1 − adjSi (δ) − 1)
3
(3.38)
(3.39)
While (3.35) is a valid statement of ∆σY /∆σXi for any value of δ ∈ [0, 1), the choice of δ can
greatly affect the accuracy of the local approximation ∂σY /∂σXi ≈ ∆σY /∆σXi , especially for small
values of δ (corresponding to large reductions in factor variance), or if the relationship between
var (Y 0 ) and var (Xi0 ) is highly nonlinear.
80
The above expressions allow the designer to obtain a first-order estimate of the parameter adjustments needed to achieve a desired decrease in risk. They can also highlight different trends, tradeoffs, and design tensions present in the system. However,
we note again that these relations are merely local approximations whose predictive
accuracy cannot be guaranteed. Furthermore, they do not account for interactions
among the design parameters, nor do they imply that the suggested changes in mean
or standard deviation are necessarily feasible, as limitations due to physical or budgetary constraints are not accounted for. It is the responsibility of the designer to
use these tools as a guideline for cost-benefit analysis of various design activities, and
ultimately select the most appropriate action for risk mitigation.
3.4
Entropy Power-Based Sensitivity Analysis
In this section, we extend the ideas of variance-based GSA to an analogous methodology for complexity-based GSA. For this, we use entropy power as the basis for our
sensitivity analysis approach, recalling from (2.27) that entropy power is proportional
to the square of exponential entropy.
3.4.1
Entropy Power Decomposition
We rewrite the ANOVA-HDMR presented in (3.1) using the random variables Y and
Zi , Zij , . . . , Z12...m to denote the outputs of the original function and various subfunctions, respectively. That is, we let Y = g(X) and Zi1 ,...,is = gi1 ,...,is (Xi1 , . . . , Xis ),
where the indices i1 , . . . , is represent all sets of s integers that satisfy 1 ≤ i1 < . . . <
is ≤ m, as described in Section 3.1.1. The ANOVA-HDMR can thus be expressed as:
Y = g0 +
X
i
Zi +
X
Zij + . . . + Z12...m .
(3.40)
i<j
Since all sub-functions in (3.1) are orthogonal and zero-mean, Zi , Zij , . . . , Z12...m
(henceforth termed “auxiliary random variables”) are all uncorrelated. According
to (3.5), the variance of Y is the sum of the partial variances of Z1 through Zm and
81
their higher-order interactions:
var (Y ) =
X
var (Zi ) +
X
i
var (Zij ) + . . . + var (Z12...m ).
(3.41)
i<j
Normalizing by var (Y ), the proportional contribution to var (Y ) from each of the
auxiliary random variables is given by:
1=
X var (Zi )
i
var (Y )
+
X var (Zij )
i<j
var (Y )
+ ... +
var (Z12...m )
.
var (Y )
(3.42)
We desire to derive a similar relationship for N (Y ), the entropy power of the
QOI. As discussed in Section 2.3.3, although there is no general relationship between
entropy power and variance, for a given random variable, entropy power is always
less than or equal to variance, with equality in the case where the random variable is
Gaussian (see Table 2.1). Therefore, we seek to relate the disparity between variance
and entropy power to the random variable’s degree of non-Gaussianity, quantified
using the K-L divergence. For this, we define an “equivalent Gaussian distribution,”
which is a Gaussian random variable with the same mean and variance as the original
random variable. For an arbitrary random variable W , we denote its equivalent
Gaussian distribution using the superscript G, and define it as W G ∼ N (µW , σW ).
Using (2.22), we compute the K-L divergence between each random variable in
(3.40) and its equivalent Gaussian distribution to obtain DKL (Y ||Y G ), DKL (Z1 ||Z1G ),
DKL (Z2 ||Z2G ), etc. In Appendix C, we derive the following sum, which relates the
entropy power and non-Gaussianity of the QOI to the corresponding quantities for
the auxiliary random variables:
N (Y )exp[2DKL (Y ||Y G )] =
X
N (Zi ) exp[2DKL (Zi ||ZiG )]
i
+
X
N (Zij ) exp[2DKL (Zij ||ZijG )] + . . .
i<j
G
+ N (Z12...m ) exp[2DKL (Z12...m ||Z12...m
)].
82
(3.43)
The basis for (3.43) is that for any arbitrary random variable W , var (W ) =
N (W ) exp[2DKL (W ||W G )].4 This implies that the variance of each of Y and Zi , Zij ,
. . . , Z12...m is equal to the product of the variable’s entropy power and the exponential
term, where the argument of the exponential is two times the K-L divergence of the
variable with respect to its equivalent Gaussian distribution. Thus, (3.41) and (3.43)
are equivalent.
Note that in the derivation we have not made any assumptions about the underlying distributions of Y or the auxiliary random variables. The only requirement is
that all the auxiliary random variables must be uncorrelated, which is automatically
satisfied by the ANOVA-HDMR. We emphasize the distinction between uncorrelated
random variables Zi and Zj (i 6= j), for which the variance of the sum is equal to
the sum of the variances (i.e., var (Zi + Zj ) = var (Zi ) + var (Zj )), and independent
random variables Xi and Xj , for which the joint probability density fXi ,Xj (xi , xj ) is
equal to the product of the marginal densities fXi (xi ) and fXj (xj ). While independent
implies uncorrelated, the converse is not true. Recall that in Section 1.2 we made the
assumption that all design parameters X1 , . . . , Xm are independent. On the other
hand, the auxiliary random variables Z1 , Z2 , . . . , Z12...m are uncorrelated. While the
auxiliary random variables representing main effects (e.g., Z1 , Z2 , . . . , Zm ) are also
mutually independent, those that correspond to interaction effects are functions of
two or more design parameters, and thus are necessarily dependent. For example,
while Zi and Zj , i 6= j, are independent, Zij is dependent on both Zi and Zj , but all
three variables are uncorrelated.
Having obtained analogous expressions for the decomposition of output variance
and entropy power into comprising terms, we can normalize (3.43) to establish a
similar interpretation in terms of proportional contributions from the input factors
4
This result is consistent with the principle of maximum entropy, which states that for a given
mean and variance, differential entropy (and therefore entropy power) is maximized with a Gaussian
distribution. Since the K-L divergence is a non-negative quantity, exp[2DKL (W ||W G )] ≥ 1, which
implies that var (W ) ≥ N (W ). In the limiting case that W is itself a Gaussian random variable,
DKL (W ||W G ) = 0, exp[2DKL (W ||W G )] = 1, and var (W ) = N (W ), as required.
83
and their interactions:
1=
X N (Zi ) exp[2DKL (Zi ||Z G )]
i
N (Y ) exp[2DKL (Y ||Y
+ ... +
i
G )]
+
X N (Zij ) exp[2DKL (Zij ||ZijG )]
i<j
N (Y ) exp[2DKL (Y ||Y G )]
G
)]
N (Z12...m ) exp[2DKL (Z12...m ||Z12...m
.
G
N (Y )
exp[2DKL (Y ||Y )]
(3.44)
In (3.44), the proportion of output uncertainty directly due to Xi consists of two
parts: one that is the ratio of the entropy power of Zi to that of Y , and one that is the
ratio of the exponential of twice the K-L divergence from Zi to ZiG to the analogous
quantity for Y . Recalling that exponential entropy — proportional to the square root
of entropy power — measures the intrinsic extent of a random variable, we conclude
that the first ratio is directly influenced by the intrinsic extent of Xi (and thus Zi ).
On the other hand, the second ratio is directly related to the non-Gaussianity of Xi
(and thus Zi ). To illustrate this, we introduce the entropy power sensitivity indices
ηi and ζi , defined as:
N (Zi )
,
N (Y )
exp[2DKL (Zi ||ZiG )]
ζi =
.
exp[2DKL (Y ||Y G )]
ηi =
(3.45)
(3.46)
The above expressions correspond to the main effect indices for factor i; analogous
expressions can also be derived for the higher-order interactions. Due to the equivalence of (3.42) and (3.44), ηi and ζi are related to the variance-based main effect
sensitivity index Si as follows:
Si = ηi ζi .
(3.47)
Substituting ηi and ζi into (3.44) gives:
1 =η1 ζ1 + η2 ζ2 + . . . + η12...m ζ12...m .
(3.48)
The proportion of output uncertainty due to each factor directly can be broken down
84
into an intrinsic extent effect characterized by the sensitivity index ηi , which is related
to complexity, and a non-Gaussianity effect characterized by the sensitivity index ζi ,
which is related to distribution shape. The product of the two effects equals the MSI
of the factor. This allows us to associate (3.44) with the uncertainty apportionment
notion depicted in Figure 3-1.
Note that in the previous paragraph we used the phrase “due to each factor
directly” instead of “due to each factor alone.” This choice of wording reflects the
fact that a change in any auxiliary random variable Zj can affect both the intrinsic
extent and the non-Gaussianity of Y . Thus, the quantities N (Y ) and DKL (Y ||Y G ) are
impacted, which indirectly affects the indices ηi and ζi of all other auxiliary variables
Zi where i 6= j. Because it is usually difficult to determine a priori how changing
Zj would modify N (Y ) and DKL (Y ||Y G ), it is typically impractical to decouple the
direct and indirect effects that alterations to Zj would impose on the entropy power
decomposition. Despite this limitation, the entropy power sensitivity indices still
reveal useful information about how the spread and distribution shape of each input
factor contribute to uncertainty in the output quantity of interest.
3.4.2
The Effect of Distribution Shape
The K-L divergence between Zi and its equivalent Gaussian distribution is both shiftand scaling-invariant. The former is easy to comprehend, since by definition Zi and
ZiG have the same mean. To see the latter, we examine the case where Zi0 = αZi . It
is easy to show that h(Zi0 ) = h(Zi ) + log α (see Table 2.2) [88]. Making use of the
relations fZi0 (z) = α1 fZi ( αz ) and ξ = αz , we can compute the cross entropy h(Zi0 , Zi0G )
85
as follows:
h(Zi0 , Zi0G )
∞
1
z
1
z
fZi ( ) log
f G ( ) dz
=−
α
α Zi α
−∞ α
Z ∞
1
1
=−
fZi (ξ) log
f G (ξ) α dξ
α Zi
−∞ α
Z ∞
Z
=−
fZi (ξ) log fZiG (ξ) dξ + log α
Z
−∞
∞
fZi (ξ) dξ
−∞
= h(Zi , ZiG ) + log α.
(3.49)
Combining (2.24) and (3.49), we obtain:
DKL (Zi0 ||Zi0G ) = DKL (Zi ||ZiG ).
(3.50)
The above result implies that the non-Gaussianity of a random variable remains
constant if its underlying distribution maintains the same shape to within a multiplicative constant. For example, if Zi and Zi0 are both uniform random variables, but
Zi0 has a narrower distribution, the fundamental shape of probability density is not
affected, and therefore the two variables have the same degree of non-Gaussianity.
However, if Zi and Zi0 are both triangular random variables, but one has a symmetric
distribution and the other a skewed distribution, then Zi and Zi0 have differing levels
of non-Gaussianity. This is because the underlying distribution shape has changed,
even though both variables are of the triangular family. Finally, as a note of caution,
we reiterate that even if a design activity has no effect on DKL (Zi ||ZiG ), ζi can still
be impacted indirectly through changes to DKL (Y ||Y G ) imparted by other factors.
Since it is a measure of distance, the K-L divergence is always non-negative. As a
measure of non-Gaussianity, DKL (Zi ||ZiG ) = 0 if and only if Zi itself is Gaussian, and
is positive otherwise. This implies that exp[2DKL (Zi ||ZiG )] ≥ 1. However, ζi can be
greater than, less than, or equal to one, depending on the relative magnitude of the
numerator and denominator in (3.46). That is to say, ζi indicates whether an auxiliary
random variable Zi is less Gaussian (ζi > 1), more Gaussian (ζi < 1), or equally as
Gaussian (ζi = 1) as the QOI Y . Below, we consider the three cases separately for a
86
system modeled by Y = X1 + X2 , where X1 and X2 are either uniform or Gaussian
random variables. Note that this simple system does not contain interaction effects,
and thus Z1 = X1 , Z2 = X2 , and Z12 = 0.
𝑿𝟏 ~𝑼 −𝟎. 𝟓, 𝟎. 𝟓
-0.5
0
𝑿𝟐 ~𝑼 −𝟎. 𝟓, 𝟎. 𝟓
0.5
-0.5
0
𝒀~𝑻 −𝟏, 𝟏, 𝟎
0.5
-1 -0.5
0
0.5
1
0.5
1
(a) Case 1: Example with ζ1 > 1 and ζ2 > 1
2.5
𝑿𝟏 ~𝑼 −𝟎. 𝟓, 𝟎. 𝟓
2
2.5
𝑿𝟐 ~𝑵 𝟎, 𝟎. 𝟐𝟓
1.5
1
1
0.5
0.5
0
0
-0.5
0
0.5
-0.8
𝒀
2
1.5
-0.5
-0.6
-0.4
-0.2
0
0
0.2
0.5
0.4
0.6
0.8
-1.5
-1 -0.5
-1
-0.5
0
0
0.5
1
1.5
(b) Case 2: Example with ζ1 > 1 and ζ2 < 1
2.5
2
2.5
𝑿𝟏 ~𝑵 𝟎, 𝟎. 𝟐𝟓
2
1.5
𝑿𝟐 ~𝑵 𝟎, 𝟎. 𝟐𝟓
2
1.5
1.5
1
1
1
0.5
0.5
0.5
0
-0.8
0
-0.5
-0.6
-0.4
-0.2
0
0
0.2
0.5
0.4
0.6
0.8
-0.8
𝒀~𝑵 𝟎, 𝟎. 𝟑𝟓
2.5
0
-0.5
-0.6
-0.4
-0.2
0
0
0.2
0.5
0.4
0.6
0.8
-1.5
-1 -0.5
-1
-0.5
0
0
0.5
0.5
1
1
1.5
(c) Case 3: Example with ζ1 = ζ2 = 1
Figure 3-6: Examples of Y = X1 + X2 with increase, decrease, and no change in
Gaussianity between the design parameters and QOI
For the first case, we let X1 and X2 be i.i.d. uniform random variables on the
interval [−0.5, 0.5]. The sum Y = X1 + X2 has a symmetric triangular distribution
on the interval [−1, 1]. In this case, both ζ1 and ζ2 exceed one (Figure 3-7(a)). Figure
3-6(a) illustrates that in moving from the design parameters to the QOI, the system
increases in Gaussianity. Figure 3-6(b) shows an example where X1 is uniform and
X2 is Gaussian, resulting in a distribution for Y that is more Gaussian than X1 but
less Gaussian than X2 . In this case, the corresponding indices for non-Gaussianity
are ζ1 > 1 and ζ2 < 1 (Figure 3-7(b)). Finally, if both X1 and X2 are Gaussian, then
X1 + X2 is also Gaussian (Figure 3-6(c)), and ζ1 = ζ2 = 1 (Figure 3-7(c)).
87
1.6
1.4
1.2
1.6
Si
1.37
1.37
ηi
ζi
1
0.8
0.8
0.4
0.6
0.50
0.50
0.99
0.57
0.43 0.43
0.4
0.2
0
ζi
0.40
0.37
0.37
1.42
ηi
1.2
1
0.6
Si
1.4
0.2
Factor 1
0
Factor 2
(a) Case 1: X1 , X2 ∼ U[−0.5, 0.5]
Factor 1
Factor 2
(b) Case 2: X1 ∼ U[−0.5, 0.5],
X2 ∼ N (0, 0.25)
1.6
1.4
1.2
Si
ηi
ζi
1.00
1.00
1
0.8
0.6
0.50 0.50
0.50 0.50
0.4
0.2
0
Factor 1
Factor 2
(c) Case 3: X1 , X2 ∼ N (0, 0.25)
Figure 3-7: Sensitivity indices Si , ηi , and ζi for three examples of Y = X1 + X2 . For
each factor, Si equals the product of ηi and ζi .
3.5
Chapter Summary
In the design of complex systems, sensitivity analysis is crucially important for understanding how various sources of uncertainty influence estimates of the quantities of
interest. In this chapter, we built upon the existing methodology for variance-based
distributional sensitivity analysis to consider changes in distribution family and thus
explicitly study the effect of distribution shape. We also computed the local sensitivity of complexity and risk to perturbations in mean and standard deviation, and
interpreted the results in the context of design for robustness and reliability. Furthermore, we connected these sensitivities to indices from GSA and DSA to generate
local approximations for the change in complexity or risk associated with a reduction
in variance.
88
Another contribution of this chapter is the derivation of the entropy power decomposition, which categorizes uncertainty contributions into intrinsic extent and
non-Gaussianity effects. The entropy power decomposition is analogous to the notion of variance apportionment in GSA; however, it provides additional information
about how different aspects of a factor’s uncertainty are compounded in the system,
which can be valuable to the designer when selecting specific strategies to reduce
uncertainty.
In Chapter 4, we will connect the sensitivity analysis methods from this chapter
with design budgets to trade off various options for uncertainty mitigation and guide
decisions for the allocation of resources.
89
Chapter 4
Application to Engineering System
Design
In this chapter, we apply the framework shown in Figure 1-2 to a simple engineering
system in order to showcase its utility for designing for robustness and reliability.
Throughout this discussion, we will use as an example a R-C high-pass filter circuit,
which is introduced in the Section 4.1. Section 4.2 demonstrates how the sensitivity
analysis techiques developed in this thesis can be used to better our understanding
of the effects of uncertainty in the system, as well as analyze the influence of various
modeling assumptions. In Section 4.3, we introduce design budgets for cost and uncertainty, and connect them with sensitivity analysis results in order to identify specific
options for uncertainty mitigation, visualize tradeoffs between various alternatives,
and inform resource allocation decisions.
4.1
R-C Circuit Example
As an example system, consider the circuit shown in Figure 4-1, which consists of a
resistor and a capacitor in series. The design parameters of the system are resistance
(R) and capacitance (C), which have nominal values 100 Ω and 4.7 µF, respectively.
The corresponding component tolerances are Rtol = ±10% and Ctol = ±20%.
91
𝐶
𝑉in
𝑅
𝑉out
Figure 4-1: R-C high-pass filter circuit
As configured, the circuit acts as a passive high-pass filter, whose magnitude and
phase Bode diagrams are shown in Figure 4-2. We select the quantity of interest to be
the cutoff frequency fc , which has a nominal value of 339 Hz. To achieve the desired
circuit performance, fc must exceed 300 Hz.
(dB)
Magnitude
Magnitude (dB)
0
-10
-20
-30
(deg)
Phase
Phase (deg)
-40
90
45
0
1
10
10
2
10
3
10
4
Frequency (Hz)
(Hz)
Frequency
Figure 4-2: Bode diagrams for magnitude (top) and phase (bottom). The dashed
black line indicates the required cutoff frequency of 300 Hz.
We use the random variables X1 , X2 , and Y to represent uncertainty in R, C, and
fc , respectively. Due to stochasticity in the system, the functional requirement on
the QOI will be treated as a probabilistic design target (i.e., the probability that the
cutoff frequency falls below 300 Hz must not exceed a specified limit). The relationship
between the design parameters and the QOI is given by:
Y =
1
.
2πX1 X2
92
(4.1)
We model X1 and X2 as uniform random variables on the intervals specified
by their component tolerances, such that X1 ∼ U[a1 , b1 ] = U[90, 110] Ω and X2 ∼
U[a2 , b2 ] = U[3.76, 5.64] µF. For simplicity, we shall refer to this initial scenario as
Case 0. For this system, we can evaluate the ANOVA-HDMR of (4.1) in closed form
to obtain the following expressions for the expected value of Y , as well as the auxiliary
random variables Z1 , Z2 , and Z12 :
log(b1 /a1 ) log(b2 /a2 )
,
2π(b1 − a1 )(b2 − a2 )
(4.2)
Z1 =
log(b2 /a2 )
− E [Y ],
2π(b2 − a2 )X1
(4.3)
Z2 =
log(b1 /a1 )
− E [Y ],
2π(b1 − a1 )X2
(4.4)
E [Y ] =
Z12 =
1
log(b2 /a2 )
−
2πX1 X2 2π(b2 − a2 )X1
log(b1 /a1 )
log(b1 /a1 ) log(b2 /a2 )
−
+
.
2π(b1 − a1 )X2
2π(b1 − a1 )(b2 − a2 )
(4.5)
1400
var(Y) = 2015 Hz2
Number of samples
1200
C(Y) = 181 Hz
1000
P(Y<300Hz) = 18.6%
800
600
400
200
0
250
300
350
400
450
500
Cutoff frequency (Hz)
Figure 4-3: Histogram of cutoff frequency for Case 0 generated using 10,000 MC
samples. The dashed black line indicates the required cutoff frequency of 300 Hz.
In this example, we use Monte Carlo simulation with 10,000 samples to characterize the propagation of uncertainty from the design parameters to the quantity of interest. Figure 4-3 shows the histogram of Y for Case 0. The corresponding uncertainty
93
estimates are: var (Y ) = 2015 Hz2 , C(Y ) = 181 Hz, and P (Y < 300 Hz) = 18.6%.
4.2
4.2.1
Sensitivity Analysis of the R-C Circuit
Identifying the Drivers of Uncertainty
Having obtained uncertainty estimates for Case 0, we next apply existing variancebased sensitivity analysis methods to identify the key drivers of output variance.
Global sensitivity analysis of Case 0 reveals that approximately 80% of the variability
in Y can be attributed to X2 (capacitance), and the remaining 20% to X1 (resistance),
with interactions playing a negligible role (Figure 4-4(a)). Distributional sensitivity
analysis (assuming fixed distribution shape) indicates that the relationship between a
decrease in factor variance and the expected decrease in output variance is linear for
both X1 and X2 (Figure 4-4(b)). These results point to prioritizing X2 for uncertainty
reduction, as it dominates X1 in terms of contribution to output variance.
< 1%
1
0.8
20%
Factor 1
Factor 2
Interactions
Factor 1
Factor 2
adjSi(δ)
0.6
80%
0.4
0.2
0
−0.2
0
0.2
0.4
0.6
0.8
1
1−δ
(a) Main effect sensitivity indices
(b) Adjusted main effect sensitivity indices
Figure 4-4: Global and distributional sensitivity analysis results for Case 0
4.2.2
Understanding the Impacts of Modeling Assumptions
In this section, we extend our sensitivity analysis of the R-C circuit to study the effects
of modeling assumptions, in particular assumptions regarding the input distributions.
94
Focusing on X2 , we apply several methods developed in this thesis, including entropy
power-based sensitivity analysis and DSA that permits changes to distribution family.
Two specific cases are considered: in Case 1, we allow the distribution of X2 to vary
between several types while holding its variance constant; in Case 2, the interval on
which X2 is supported is fixed while the distribution type is varied across several
families. In both cases, we are interested in studying how assumptions regarding the
input factor distribution impact uncertainty in the QOI.
Case 1: Fixing the Variance of X2
As described in Case 0, we initially assumed that X1 and X2 are uniformly distributed
on the intervals [90, 110] Ω and [3.76, 5.64] µF, respectively, which correspond to to
variances of var (X1 ) = 33.3 Ω2 and var (X2 ) = 0.30 µF2 . Here, we use the procedures
developed in Section 3.2 to extend DSA from Figure 4-4(b) to permit changes between
distribution families.
1
0.8
1
Uni −> Uni
Uni −> Tri
Uni −> Gaussian
Uni −> Bmd Uni
0.8
0.6
adjSi(δ)
adjSi(δ)
0.6
0.4
0.4
0.2
0.2
0
0
−0.2
0
Uni −> Uni
Uni −> Tri
Uni −> Gaussian
Uni −> Bmd Uni
0.2
0.4
0.6
0.8
−0.2
0
1
1−δ
0.2
0.4
0.6
0.8
1
1−δ
(a) Factor 1: Resistance
(b) Factor 2: Capacitance
Figure 4-5: Distributional sensitivity analysis results with changes to distribution
family
Figure 4-5(a) shows that for X1 , there is virtually no difference in the adjusted
main effect sensitivity indices between the four scenarios considered: the initial case
where var (X1 ) is reduced while maintaining X1 as a uniform random variable, and
the alternative options where a decrease in var (X1 ) is concomitant with a switch to
a triangular, Gaussian, or bimodal uniform distribution. This result suggests that
95
the shape of the underlying distribution for X1 does not contribute significantly to
uncertainty in the QOI; that is, the relationship between a decrease in var (X1 ) and
the expected reduction in var (Y ) is not affected by the distribution family (of the
four considered) to which X1 belongs.
Figure 4-5(b) shows a similar trend for X2 ; however, the four curves diverge slightly
for small values of 1−δ. To study this region, we concentrate on the leftmost points in
Figure 4-5(b) where 1−δ = 0, which correspond to no reduction in var (X2 ) from Case
0. Figure 4-6 shows four possible distributions for X2 , each of which has a variance
of 0.30 µF2 . Note that the triangular and Gaussian distributions have supports that
extend beyond the original interval.1
Probability density
1.5
U
T
G
BU
1
0.5
0
3
3.5
4
4.5
5
5.5
6
6.5
Capacitance (µF)
Figure 4-6: Examples of uniform (U), triangular (T), Gaussian (G), and bimodal
uniform (BU) distributions for X2 , each with a variance of 0.30 µF2
Case
var (X2 )
C(X2 )
var (Y )
C(Y )
P (Y < 300 Hz)
1-T
0%
+16.6%
+2.1%
+1.2%
−1.5%
1-G
0%
+19.3%
+1.3%
−1.9%
−2.3%
1-BU
0%
−50.0%
−2.9%
−4.5%
+2.6%
Table 4.1: Change in uncertainty estimates for Cases 1-T, 1-G, and 1-BU, as compared
to Case 0
1
Although a Gaussian random variable is infinitely supported, in this example we have truncated the Gaussian distribution at ±3 standard deviations (which captures more than 99.7% of the
probability), as it is not realistic for capacitance to approach ±∞.
96
Table 4.1 shows how uncertainty estimates for the system are altered as a result
of changing the distribution of X2 from uniform (Case 0) to triangular (Case 1-T),
truncated Gaussian (Case 1-G), or bimodal uniform (Case 1-BU). It is easy to see
that estimates of variance, complexity, and risk can trend in opposite directions. For
example, in both Cases 1-T and 1-G there is a significant increase in C(X2 ), which results in a slight increase in output variance but a slight decrease in risk. Furthermore,
Case 1-G, which represents the maximum entropy distribution for X2 given a variance
of 0.30 µF2 , does not necessarily correspond to the most conservative estimates for
output uncertainty. In Case 1-BU, the complexity of X2 is reduced by 50%, which in
turn decreases both output variance and complexity but increases risk. These results
demonstrate the presence of competing objectives for uncertainty mitigation, such
that efforts to reduce one measure of uncertainty can result in an increase in the others. In general, the extent to which the various measures of uncertainty trend jointly
or divergently is problem-dependent.
Case 2: Fixing the Extent of X2
In Case 2, we investigate the effect of our initial assumption of a uniform distribution for X2 to capture the uncertainty associated with a ±20% capacitor tolerance.
Figure 4-7 shows several alternative distributions on the interval [3.76, 5.64] µF: triangular and truncated Gaussian distributions centered at 4.7 µF (Cases 2-T and 2G, respectively),2 and a symmetric bimodal uniform distribution parameterized by
aL = 3.76 µF, bL = 4.23 µF, aR = 5.17 µF, and bR = 5.64 µF (Case 2-BU). Since these
distributions lie wholly within the original interval, we can sample from them using
the acceptance/rejection procedures provided in Section 3.2; thus, no additional MC
simulations are required for the following analysis.
The distributions shown in Figure 4-7 differ greatly in variance as well as complexity (see Table 4.2). Cases 2-T and 2-G can represent reasonable choices for X2 if there
is strong confidence in the nominal capacitance. A possible scenario for Case 2-BU is
2
As before, we approximate the Gaussian distribution with a truncated version in which ±3
standard deviations of the distribution fall within the specified interval.
97
Probability density
1.5
U
T
G
BU
1
0.5
0
3
3.5
4
4.5
5
5.5
6
6.5
Capacitance (µF)
Figure 4-7: Examples of uniform (U), triangular (T), Gaussian (G), and bimodal
uniform (BU) distributions for X2 supported on the interval [3.76, 5.64] µF
Case
var (X2 )
C(X2 )
2-T
−50.0%
2-G
2-BU
var (Y )
C(Y )
P (Y < 300 Hz)
−17.6% −41.2%
−21.6%
−7.4%
−66.7%
−31.1% −54.6%
−30.8%
−10.5%
+75.0%
−50.0%
+11.9%
+11.2%
+60.7%
Table 4.2: Change in uncertainty estimates for Cases 2-T, 2-G, and 2-BU, as compared
to Case 0
one in which the capacitors with ±10% tolerance are removed during the manufacturing process and sold for a higher price, leaving the worst-performing components
in the ±20% tolerance category.
Figure 4-8 shows the histograms of Y generated using 10,000 MC samples for the
various cases.3 It is easy to see that the choice of X2 greatly affects the shape of the
resulting distribution for Y , as well as the corresponding uncertainty estimates. Table
4.2 shows that in Cases 2-T and 2-G, there is a significant decrease in both var (X2 )
and C(X2 ) over Case 0, which also results in large reductions in the variance, complexity, and risk in the QOI. These outcomes are consistent with the adjusted main effect
sensitivity indices shown in Figure 4-5(b), which predict the expected reduction in
var (Y ) to be approximately 40% and 55% for Cases 2-T and 2-G, respectively, corresponding to reductions in var (X2 ) of 50% and 66.7%. Case 2-BU, on the other hand,
3
Note that Figure 4-3 is reproduced as Figure 4-8(a) as a point of comparison in Case 2.
98
1400
1400
2
var(Y) = 2015 Hz
C(Y) = 181 Hz
1000
P(Y<300Hz) = 18.6%
800
600
400
200
0
250
var(Y) = 1182 Hz2
1200
Number of samples
Number of samples
1200
C(Y) = 142 Hz
1000
P(Y<300Hz) = 10.9%
800
600
400
200
300
350
400
450
0
250
500
300
Cutoff frequency (Hz)
350
(a) Case 0
500
1400
2
var(Y) = 898 Hz
1200
C(Y) = 124 Hz
1000
var(Y) = 3270 Hz2
1200
Number of samples
Number of samples
450
(b) Case 2-T
1400
P(Y<300Hz) = 7.7%
800
600
400
200
0
250
400
Cutoff frequency (Hz)
C(Y) = 203 Hz
1000
P(Y<300Hz) = 29.7%
800
600
400
200
300
350
400
450
0
250
500
Cutoff frequency (Hz)
300
350
400
450
500
Cutoff frequency (Hz)
(c) Case 2-G
(d) Case 2-BU
Figure 4-8: Histogram of Y resulting from various distributions for X2 considered in
Case 2
represents a 50% reduction in C(X2 ) and a 75% increase in var (X2 ), and produces
sizable increases the uncertainty estimates for the QOI.4 This illustrates that, as in
Case 1, the maximum entropy distribution for X2 given the available information (in
this case, the uniform distribution since we are focusing on the interval [3.76, 5.64] µF)
does not necessarily lead to the most conservative estimates for output uncertainty.
Figure 4-9 shows the entropy power-based sensitivity indices ηi and ζi alongside
the main effect indices Si computed from GSA. We see that for Cases 0, 2-T, and
2-G, Si is similar to ηi for both factors, and that ζi is equal to or slightly greater than
4
Case 2-BU does not fall along the DSA curves shown in Figure 4-5(b). Since it entails a
75% increase in var (X2 ), one would need to extrapolate the uniform → bimodal uniform curve to
1 − δ = −0.75; doing so confirms that a 60.7% increase in var (Y ) can reasonably be expected.
99
2.5
2.5
S
S
i
i
ηi
2
ζi
1.5
ζi
1.5
1.17
1.14
1.11
0.98
1
0.80
0.72
0.67
0.5
0.20 0.18
0.00 0.00
Factor 1
Factor 2
0
Interactions
1.13
1.02
1
0.5
0
ηi
2
0.33
0.59
0.27
0.00 0.00
Factor 1
Factor 2
(a) Case 0
Interactions
(b) Case 2-T
2.5
2.5
Si
Si
ηi
2
2
ζi
1.5
ηi
2.06
ζi
1.5
1.19
1.18
1.00
1
1
0.85
0.87
0.75
0.57 0.57
0.5
0.43
0.5
0.35
0.38
0.13 0.13
0
0.00 0.00
Factor 1
Factor 2
0
Interactions
(c) Case 2-G
0.00 0.00
Factor 1
Factor 2
Interactions
(d) Case 2-BU
Figure 4-9: Main effect and entropy power-based sensitivity indices for Case 2
one. This indicates that uncertainty apportionment using variance and entropy power
produce similar results. Furthermore, it reveals that Gaussianity increases in moving
from the input factors to the QOI. For Case 2-BU, however, ζ1 is less than one, ζ2
is exceedingly large, and η2 is significantly smaller than S2 . This implies that the
proportion of output variance due to Factor 2 alone (87%) is dominated by the nonGaussianity of the bimodal uniform distribution, with only a small contribution due
to the intrinsic extent of X2 . This is consistent with the result that the complexity of
Y increases by only 11.9% over Case 0, whereas variance rises by a substantial 60.7%
(see Table 4.2).
For Case 2-BU, our analysis has highlighted that the propagation of uncertainty
from the inputs to the QOI is dominated by the bimodality of X2 , which introduces
100
significant non-Gaussianity into the system. Thus, in using entropy power decomposition, we learned more about the underlying features of the system that contribute to
output uncertainty than can be revealed by variance-based sensitivity analysis alone.
We conclude from Case 2 that initial assumptions regarding distribution shape can
greatly impact uncertainty in the QOI, especially for systems involving multimodality
or other highly non-Gaussian features. Therefore, one must be careful when selecting
the appropriate distribution to characterize uncertainty in the system inputs, as misattributions can lead to gross over- or underestimates of uncertainty in the quantities
of interest.
4.3
Design Budgets and Resource Allocation
Once the sources of uncertainty in a system are identified through sensitivity analysis, the task of uncertainty mitigation centers on making decisions regarding design
modifications in the subsequent iteration [5, 6, 127]. Such decisions often relate to
the allocation of resources in order to improve one or more aspects of the design.
Resources can be allocated to a variety of different activities — for example, to direct
future research, to conduct experiments, to improve physical and simulation-based
modeling capabilities, or to invest in superior hardware or additional personnel. These
steps complete the resource allocation feedback loop shown in Figure 1-2.
Typically, the appropriate decisions for minimizing system uncertainty at each step
of the design process are not known a priori. Instead, sensitivity analysis results are
often used to infer the potential of various options for mitigating uncertainty. As discussed in Chapter 3, variance-based GSA and DSA, as well as relative entropy-based
sensitivity analysis all enable the prioritization of factors according to contribution to
output variability. However, these methods do not address how to identify or select
between specific strategies.
In the following sections, we use the R-C circuit example to demonstrate how the
sensitivity analysis methods developed in this thesis can be combined with budgets
for cost or uncertainty and used to uncover design alternatives for reducing uncer-
101
tainty, visualize tradeoffs among the available options, and ultimately guide decisions
regarding the allocation of resources to improve system robustness and reliability.
4.3.1
Visualizing Tradeoffs
First, we are interested in studying how uncertainty estimates for the QOI are affected
by reductions in the variance of one or both design parameters. For simplicity, we will
assume that uncertainty in both resistance and capacitance can be well-characterized
using uniform random variables. This assumption is justified by the DSA results in
Figure 4-5, which showed that the variance of Y is not significantly affected by the
shape of the underlying distributions for X1 and X2 . Therefore, reductions in var (X1 )
and var (X2 ) serve to shrink the extent of the corresponding uniform distributions.5
We note, however, that the techniques employed in the following analysis can be
adjusted to also accommodate changes in the distribution family of X1 or X2 .
Figure 4-10 shows how different levels of reduction in var (X1 ) and var (X2 ) trade
against one another with respect to variance, complexity, and risk remaining in the
QOI. The bottom left corner of each figure represents the initial design (Case 0).
The colored contours were generated using 10,000 MC simulations to evaluate the
QOI at each design parameter combination. Hence, they represent the most accurate
uncertainty estimates available for the system.
For many realistic engineering systems, however, evaluating the QOI is computationally expensive, and it may not be feasible to perform numerous MC simulations
in order to visualize tradeoffs in the design space. In those situations, it is still
possible to approximate variance, complexity, and risk using sensitivity information.
For variance, we can use the adjusted main effect sensitivity indices from DSA (i.e.,
Figure 4-5) to relate a 100(1 − δ)% decrease in var (X1 ) or var (X2 ) to the expected
reduction in var (Y ). Similarly, we can combine the DSA results with local sensitivity
estimates from (3.26) and (3.27) to predict the expected change in complexity and
risk. The contours generated using these approximations are indicated by the dashed
5
We assume that reductions in variance reflect improvements in component tolerance, and thus
decrease distribution extent symmetrically about R = 100 Ω or C = 4.7 µF.
102
1
200
500
200
Percent reduction in var(X2)
Percent reduction in var(X2)
1
500
0.8
800
500
800
0.6
1100
800
1100
0.4
1400
1100
0.2 170
0
02000
0
0.2
1400
1400
0.6
0.8
0.6
90
105
90
135
120
105
135
120
150
135
0.4 150
0.2 165
150
165
0180
0
1
75
120
Percent reduction in var(X1)
0.2
0.4
0.6
0.8
1
Percent reduction in var(X1)
(a) Lines of constant variance
(b) Lines of constant complexity
1 0
Percent reduction in var(X2)
75
1
0.8 05
1700
0.4
75
90
0
0.025
0.8 0.05
0.075
0.025
0.6 0.1
0.07
0.0
25
0.0
5
0.075
0.1
5
0.1
0.1
0.4 25
0.125
0.15
0.2
0.125
0.15
0.175
0
0
0
0.05
0.2
0.175
0.4
0.6
0.15
0.8
0.175
1
Percent reduction in var(X1)
(c) Lines of constant risk
Figure 4-10: Contours for variance, complexity, and risk corresponding to reductions
in factor variance, generated from 10,000 MC simulations (solid colored lines) or
approximated using distributional or local sensitivity analysis results (dashed black
lines)
black lines in Figure 4-10. In Figure 4-10(a), there is a close match between the variance contours predicted from DSA results and those generated using MC simulations.
For complexity and risk (Figures 4-10(b) and 4-10(c)), however, the correspondence
deteriorates for large deviations from Case 0, especially as var (X1 ) is decreased.
Nevertheless, the trends in variance, complexity, and risk resulting from reductions in
var (X1 ) and var (X2 ) are correctly captured, which render the approximated contours
useful for visualizing tradeoffs between uncertainty mitigation options in the input
factors, while circumventing the need to perform additional MC simulations.
Figure 4-10 confirms that decreasing var (X2 ) yields the most dramatic reductions
103
in output uncertainty; in fact, similar trends are observed for variance, complexity,
and risk. This is consistent with the prioritization of X2 over X1 in Section 4.2.1.
Next, we will connect the potential for uncertainty reduction with the cost of implementing the requisite design change, which leads to a discussion of budgets for cost
and uncertainty.
4.3.2
Cost and Uncertainty Budgets
In typical system design processes, uncertainty reduction efforts are constrained by
the availability of resources, such as funds to acquire superior tools or components,
manpower to conduct additional research or experiments, or time to allot to computational models. In this thesis, we assume that there is a cost associated with each
activity that mitigiates uncertainty in the QOI. As an illustrative example, let us consider the notional curves shown in Figure 4-11(a) for the R-C circuit, which depicts
the cost associated with a 100(1 − δ)% reduction in the variance of X1 or X2 .6 In
Figure 4-11(b), the same information is visualized as contour lines of equal cost with
respect to different proportions of variance reduction in the two factors. We observe
that although reducing var (X2 ) is more effective in abating output uncertainty than
decreasing var (X1 ), it is also more expensive to implement. Furthermore, the reduction in var (X2 ) is capped at 75% (that is, 25% of the variability in X2 is irreducible),
whereas var (X1 ) can be reduced by as much as 99%.
In order to make design decisions that adequately mitigate uncertainty yet are
feasible to implement, we must take into account the available budgets, both in terms
of cost and uncertainty. A cost budget specifies the amount of resources that can be
expended to improve the design. An uncertainty budget, on the other hand, refers
to the total level of uncertainty that is deemed tolerable for the system. In both
cases, we seek to determine how much of the prescribed amount ought to be allocated
to each design parameter. For the R-C circuit, we impose the following budgetary
6
In general, it may be difficult to determine the relationship between cost and reductions in
input uncertainty. We assume in this thesis, however, that such relationships exist and have already
been established using, for example, historical data, cost estimation methods, or expert opinion
elicitation.
104
1
25
20
Cost
15
10
5
0
0
0.2
0.4
0.6
0.8
0.8
22.5
20
17.5
0.6
15
12.5
10
7.5
0.2
5
0.4
0
0
1
0.2
12
.5
10
7.5
5
0.4
0.6
3027.525 2.5 20 7.5 15
2
1
Percent reduction in var(X2)
Factor 1
Factor 2
0.8
1
Percent reduction in var(X1)
1−δ
(a) Cost of reducing variance of X1 or X2
(b) Lines of constant cost
Figure 4-11: Notional cost associated with a 100(1 − δ)% reduction in the variance
of X1 or X2
constraints:
1. Complexity: The complexity with respect to the QOI shall not exceed 150 Hz.
2. Risk: The probability of violating the cutoff frequency requirement shall not
exceed 10%.
3. Cost: The cost of uncertainty reduction shall not exceed 20 units of cost.
Next, we examine each constraint individually in turn to study the resulting optimal allocation. Figure 4-12 overlays contours of variance, complexity, and risk in the
QOI (solid colored lines) with those corresponding to cost of implementation. These
contours allow us to consider the cost and uncertainty budgets in conjunction. We
see that for a given cost, the maximum possible uncertainty reduction is achieved
by decreasing the variance of both X1 and X2 simultaneously, rather than focusing on either factor alone. Table 4.3 lists the lowest possible uncertainty and cost
estimates for the system when each of the budgetary constraints (C(Y ) ≤ 150 Hz,
P (Y < 300 Hz) ≤ 10%, Cost ≤ 20) is in turn made active. Of the three budgets, the
complexity constraint is the cheapest to satisfy; however, the resulting allocation of
factor variance reduction does not satisfy the risk constraint. To ensure that risk does
not exceed 10%, a minimum cost of 14.8 is required; this corresponds to a complexity
of 132 Hz. Finally, a cost budget of 20 is more than adequate to guarantee that the
105
1
200
500
500
0.8
800
20
10
1400
0.4
0.6
10 1400
0.8
120
0.6
135
15
0.4 150
0180
0
1
75
105
90
120 20
105
120
135 15
10
150
0.2 165
5
Percent reduction in var(X1)
135
10 1
50
5
165
0.2
0.4
0.6
0.8
1
Percent reduction in var(X1)
(a) Lines of constant variance and cost
1 0
0
0.025
0.8 0.05
0.075
20
15
0.0
0.07 20
5
0.1 1
5
25
0.0
5
0.075
0.1
0.2
0.175
0.4
0.6
0.125
10 0.15
15
0.15
5
5
20
0.125
10
0.175
0
0
0.05
25
0.15
0.2
0
0.025
0.6 0.1
0.1
0.4 25
(b) Lines of constant complexity and cost
30
Percent reduction in var(X2)
20
15
5
1700
90
20
02000
0
0.2
1100
15
0.2 170
50
110015
20
0.4
1400
800
75
1
0.8 05
25
15
800 20
25
0.6
1100
500
75
90
30
Percent reduction in var(X2)
200
30
Percent reduction in var(X2)
1
0.8
0.175
1
Percent reduction in var(X1)
(c) Lines of constant risk and cost
Figure 4-12: Contours for variance, complexity, and risk corresponding to reductions
in factor variance (solid colored lines) overlaid with contours for cost of implementation (dashed green lines)
uncertainty budgets are also satisfied. As we observe in Figures 4-12(b) and 4-12(c),
the expenditure of 20 units of cost is sufficient to decrease var (X2 ) by up to 67%;
even without a simultaneous reduction in var (X1 ), it is enough to reduce complexity
and risk to acceptable levels of 125 Hz and 8%, respectively.
The results in Table 4.3 suggest that the cost and uncertainty budgets for the
R-C circuit design are mutually compatible and achievable. There are a host of
solutions that satisfy all three budgetary constraints, which lie within the shaded
region in Figure 4-13. The main tradeoff in these solutions is between cost and risk:
when both of those constraints are satisfied, the complexity budget is automatically
satisfied as well. In the following section, however, we will consider the cost and
106
Active constraint
var (Y ) C(Y )
P (Y < 300 Hz)
Cost
Complexity
1378
150
13.4%
8.8
Risk
1055
132
10%
14.8
Cost
780
114
6.4%
20
Table 4.3: Best achievable uncertainty mitigation results given individual budgets for
cost and uncertainty. Entries in red denote budget violations.
Figure 4-13: Contours for risk (solid blue lines) overlaid with contours for cost (dashed
green lines). Shaded region denotes area in which all budget constraints are satisfied.
uncertainty budgets alongside available design options, which can lead to different
conclusions regarding their feasbility.
4.3.3
Identifying and Evaluating Design Alternatives
In this section, we study how the nominal value and tolerance of the components
in the R-C circuit trade against one another in terms of contribution to uncertainty
in the QOI. The present analysis builds upon the design budget discussion from
Section 4.3.2, and enables designers to visualize strategies for uncertainty mitigation
in terms of actionable items in the design space.
For each of X1 and X2 , there are two parameters that can be modified: the nominal
value of the circuit component (R or C) and the tolerance (Rtol or Ctol ). We vary
R and C from their nominal values by up to ±20%, and decrease Rtol and Ctol from
the initial tolerances of ±10% and ±20%, respectively, down to ±1%. Figures 4-14
107
6
18
0.5
0.7
0.9
0
5.5 .1
0
5 210
Capacitance (µF)
Capacitance (µF)
3
0
5.5
150
180
4.5
240
18
21
0
0
4
270
240
3.5
3
80
0.
6
15
300
330
210 0
Case
Option A
Option B
240
110
120
270
90
100
5
0.0
1
0.7
0.5
0.3
0.1
4.5
0.0
1
0.3
0.1
4
3.5
3
80
0.01
Case 0
Option A
Option B
90
100
Resistance (Ω)
110
120
Resistance (Ω)
(a) Lines of constant complexity
(b) Lines of constant risk
0.5
0.3
0
20
22220
0
21201
0
140
140
130
0.02
Case 0
Option A
Option C
110
0
80
120
Resistance (Ω)
0.1
0.04
0.2
150
160
15
0
170
16
0
100
Resistor tolerance
16
0
18
17
0
0
0
19
180
0
17
190
0
18
200
200
0.6
190
0.6
Resistor tolerance
0.5
0.4
90
0.06
0.01
0
80
0.4
0
14
0.02
0.2
0.08
0
15
0.06
0.04
0.1
0.08
0.1
0.01
0.1
0.3
Figure 4-14: Uncertainty contours for variations in resistance and capacitance, generated from 10,000 MC simulations (solid colored lines) or approximated using entropy
power-based sensitivity analysis results (dashed black lines)
Case 0
Option A
Option C
90
100
110
120
Resistance (Ω)
(a) Lines of constant complexity
(b) Lines of constant risk
Figure 4-15: Uncertainty contours for variations in resistance and resistor tolerance
through 4-17 show contour lines for complexity (left) and risk (right) for different
combinations of the four quantities; a black X in each figure denotes the location of
Case 0 in the design space.
These contours reveal several interesting trends. First, we see that raising either
R or C lowers complexity in the QOI but also increases risk, and vice-versa (Figure
4-14); this indicates the presence of competing objectives. For a fixed value of R,
decreasing Rtol appears to have little effect on complexity, and almost no effect on
risk (Figure 4-15). Figure 4-16(a), on the other hand, shows that both C and Ctol
108
1
0
100
80
0
0.05
14
80
12
0
10
3.5
4
4.5
5
5.5
4.5
5
Case 0
Option B
Option D
0
3.5
6
0.5
0.6
0.05
Case 0
Option B
Option D
0
0
3
0.1
0.4
0.3
0.2
0.1
12
0.7
0
16
0.1
0.15
0.8
120
40
0.5
0.6
0
18
4
Capacitance (µF)
0.7
0.8
0.9
0
20
Capacitor tolerance
0.15
0.01
Capacitor tolerance
140
0.4
0
16
0.3
0.2
0
18
0.1
0
20
0
22
0.2
0
26 240
1
0.0
0.2
5.5
6
Capacitance (µF)
(a) Lines of constant complexity
(b) Lines of constant risk
Figure 4-16: Uncertainty contours for variations in capacitance and capacitor tolerance
0.2
0.2
160
160
140
120
0.15
120
100
0.1
Capacitor tolerance
Capacitor tolerance
140
100
80
80
60
0.05
40
60
20
0
0
0.02
0.04
0.06
0.15
0.1
0.16
0.13
0.1
0.07
0.04
0.04
0.0
1
0.
04
0.01
0.0
0.05
Case 08
0
Option C
Option D
0.08
0.16
0.16
0.13
0.13
0.1
0.1
0.07
0.04
0.010.07
1
Case 0
Option C
Option D
0
0
0.1
0.02
Resistor tolerance
0.04
0.06
0.08
0.1
Resistor tolerance
(a) Lines of constant complexity
(b) Lines of constant risk
Figure 4-17: Uncertainty contours for variations in resistor and capacitor tolerance
are influential with respect to complexity; however, Figure 4-16(b) indicates that
depending on the value of C, decreasing Ctol can either increase or decrease risk.7
Finally, Figure 4-17 shows that when trading Rtol against Ctol , decreasing the latter
is more effective for reducing both complexity and risk.
As was the case in Section 4.3.1, the colored contours in Figures 4-14 through 4-17
were generated using 10,000 MC simulations to evaluate Y at each design parameter
combination. The dashed black lines correspond to local approximations obtained
7
This is because changing C also shifts the distribution of Y relative to the requirement r = 300 Hz
(see Figure 3-5).
109
using sensitivity information, and can be produced without performing additional MC
simulations. The approximated risk contours in Figures 4-14(b), 4-15(b) 4-16(b), and
4-17(b) were computed using local sensitivity results from (3.33) and (3.37). Although
these approximations vary in accuracy, the overall trends in risk due to changing R,
C, Rtol , and Ctol are captured, especially for design parameter combinations close
to Case 0. The approximated complexity contours (Figures 4-14(a), 4-15(a) 4-16(a),
and 4-17(a)), on the other hand, rely on the results of entropy power-based sensitivity
analysis, and were estimated using the procedure below.
First, we introduce the notation X1o and X2o , corresponding to the random variables
for resistance and capacitance in Case 0, respectively, as well as X10 and X20 , denoting
the updated estimates for those factors after some design modificaton. Similarly,
letting i = 1, 2, we write the following relations:
Zio = gi (Xio )
Zi0 = gi (Xi0 )
Y o = g(X1o , X2o )
Y 0 = g(X10 , X20 )
ζio =
exp[2DKL (Zio ||ZioG )]
exp[2DKL (Y o ||Y oG )]
ζi0 =
exp[2DKL (Zi0 ||Zi0G )]
exp[2DKL (Y 0 ||Y 0G )]
Based on the entropy power decomposition given in (3.44), we can express the
entropy power of Y o as the sum of the following terms:
o
o
N (Y o ) = N (Z1o )ζ1o + N (Z2o )ζ2o + N (Z12
)ζ12
.
(4.6)
Since the interaction effects are negligible for this system, we can drop the last term
in (4.6). Then, we employ the following approximation to predict N (Y 0 ), which uses
the non-Gaussianity indices ζ1o and ζ2o evaluated in Case 0:
N (Y 0 ) ≈ N (Z10 )ζ1o + N (Z20 )ζ2o .
(4.7)
The relationship between each input factor Xi and the corresponding auxiliary random variable Zi is available from the ANOVA-HDMR (computed either analytically
110
or numerically); for the R-C circuit, these relationships are given in (4.2)–(4.5). Thus,
we are able to calculate N (Z10 ) and N (Z20 ) based on X10 and X20 , which then allows
us to estimate N (Y 0 ) without having to explicitly evaluate Y 0 using MC simulations.
Finally, recognizing that entropy power is proportional to the square of complexity,
we can rewrite (4.7) in terms of C(Y 0 ):
C(Y 0 ) ≈
p
2πe[N (Z10 )ζ1o + N (Z20 )ζ2o ].
(4.8)
We observe that the approximations obtained using (4.8) closely match the colored
contours generated from brute force MC simulation. The largest discrepancies occur
in Figure 4-15(a) for small resistor tolerances, which is consistent with the trend
observed in Figures 4-10(b) and 4-10(c) for small values of var (X1 ). Even for this case,
however, the overall trends in complexity due to varying Rtol and R are preserved.
Thus, the approximation given in (4.8) provides an easy way to visualize trends in
complexity and identify regions in the design space that warrant additional analysis
— toward which additional MC simulations can be directed to generate more rigorous
estimates of uncertainty.
Figures 4-14 through 4-17 depict tradeoffs among R, C, Rtol , and Ctol as continuous within the design space, implying infinitely many design combinations. For many
engineering systems, however, the design space instead contains a discrete number of
feasible options dictated by component availability. Thus, we next investigate how
local sensitivity analysis results can be used to identify distinct design alternatives.
Using (3.38) and (3.39), we can compute the requisite change in the mean and
standard deviation of X1 and X2 needed to achieve a desired decrease in risk. In this
case, we set ∆P = −0.086 so as to meet the constraint that P (Y < 300 Hz) ≤ 10%.
Table 4.4 lists the change in mean and standard deviation of each factor required
to achieve this risk reduction as predicted using local sensitivities, as well as the
corresponding component nominal value and tolerance. Consistent with the risk
contours in Figure 4-14(b), we see that reducing risk requires decreasing R or C.
This suggests that we use a resistor with a nominal value of no more than 96.6 Ω, or a
111
Component
Change mean only
Change SD only
∆µXi
Nominal value
∆σXi
Tolerance
Resistor
−3.4 Ω
R = 96.6 Ω
−8.4 Ω
N/A
Capacitor
−0.16 µF
C = 4.54 µF
−0.17 µF
Ctol = ±14.4%
Table 4.4: Local sensitivity predictions for the change in mean and standard deviation
(SD) of each factor required to reduce risk to 10%
capacitor with a nominal value of no more than 4.54 µF. Alternatively, we can reduce
risk by using components with tighter tolerances. For the resistor, the change in the
standard deviation of X1 needed to satisfy the risk constraint is −8.4 Ω; however, in
Case 0, σX1 = 5.8 Ω, which implies that it is not possible to achieve the requisite risk
reduction by decreasing Rtol alone. This is consistent with the risk contours in Figures
4-15(b) and 4-17(b). For the capacitor, we observe that the risk constraint can be
met by decreasing the standard deviation of X2 by −0.17 µF, which corresponds to a
component tolerance of ±14.4%.
The local sensitivity analysis predictions point the designer to values of R, C,
Rtol , and Ctol that would satisfy the uncertainty budget for risk. Consulting standard
off-the-shelf resistor and capacitor values [128], we identify four design alteratives
— Options A–D — which correspond most closely to the desired component specifications shown in Table 4.4. Options A–D are listed in Table 4.5 along with their
associated estimates of variance, complexity, risk, and cost of implementation. Each
option perturbs one of R, C, Rtol , and Ctol from its initial value in Case 0. We assume in this analysis that it is comparatively cheaper to purchase components with
different nominal values (Options A and B) than it is to acquire components with
significantly tighter tolerances (Options C and D).8 The locations of the four options
within the design space are indicated in Figures 4-14 through 4-17.
Figure 4-18 illustrates the tradeoffs between complexity and risk for the various
8
Assuming that X1 and X2 are both uniform random variables, a decrease in Rtol from ±10%
in Case 0 to ±1% in Option C corresponds to a 99% reduction in var (X1 ), and a decrease in
Ctol from ±20% in Case 0 to ±10% in Option D corresponds to a 75% reduction in var (X2 ). The
associated costs of implementation correspond to the rightmost red and blue points in Figure 4-11(a),
respectively.
112
Option
R (Ω)
Rtol
C (µF)
Ctol
var (Y )
C(Y ) P (Y < 300 Hz)
Cost
Case 0
100
±10%
4.7
±20%
2015
181
18.6%
N/A
A
82
±10%
4.7
±20%
2997
221
0%
5
B
100
±10%
3.9
±20%
2927
218
0%
5
C
100
±1%
4.7
±20%
1627
154
18.1%
13.2
D
100
±10%
4.7
±10%
788
115
7.1%
22.5
Table 4.5: Uncertainty and cost estimates associated with various design options.
Entries in red denote budget violations.
options. Immediately, we see that none of the options satisfies both the cost and
uncertainty budgets. Options A and B fall well within the cost budget and reduce
risk to zero; however, they both increase complexity to unacceptable levels, and thus
can be eliminated from consideration.9 In Option C, the resistor tolerance is reduced
to just ±1% at a cost of 13.2; yet, there is virtually no change in risk compared to
Case 0. Option D appears to be the most promising choice, as it is the only one to
satisfy both the complexity and risk constraints. However, it exceeds the cost budget
of 20.
0.25
0.2
Case 0
Option A
Option B
Option C
Option D
Risk
0.15
0.1
0.05
0
0
50
100
150
200
250
Complexity (Hz)
Figure 4-18: Tradeoffs in complexity and risk for various design options. Dashed
green lines bound region where both complexity and risk constraints are satisfied.
9
Had we instead identified options that raised the nominal values of R and C, the result would
have decreased complexity and increased risk, which is similarly undesirable.
113
These results suggest that in order to meet both the cost and uncertainty budgets, the designer must either seek alternative options, or relax one or more of the
constraints. In Figure 4-18, we see that Option D falls well within the region of acceptable uncertainty. Focusing on the top right corner of that region, we find that when
Ctol = ±12.5%, the system achieves C(Y ) = 130 Hz and P (Y < 300 Hz) = 10.0%
at a cost of 18.3, which meets all design budgets. However, while this design is
mathematically possible, it is not realistically feasible given limitations in component
availability, as standard capacitor tolerances include ±20%, ±10%, ±5%, ±2%, and
±1%. From this analysis, we conclude that the most promising course of action is
to allocate resources toward improving capacitor tolerance; furthermore, we recommend increasing the cost budget so that all uncertainty targets can be satisfied using
standard components.
4.4
Chapter Summary
In this chapter, we demonstrated the uncertainty quantification and sensitivity analysis methods introduced in Chapters 2–3 on a simple R-C high-pass filter circuit.
This example highlighted the applicability of our methods to assess the implications
of initial assumptions about the system for uncertainty in the quantities of interest.
We also showed how various sensitivity results can be used to formulate local approximations of output uncertainty, which enable the visualization of tradeoffs in the
design space without expending additional computational resources to perform model
evaluations. Finally, we connected our sensitivity analysis techniques to budgets for
cost and uncertainty, which facilitates the identification and evaluation of various
design alternatives and helps inform decisions for allocating resources to mitigate
uncertainty.
In Chapter 5, we will apply the methods developed in this thesis to perform costbenefit analysis of a real-world aviation environmental policy, further demonstrating
their utility for improving system robustness and reliability.
114
Chapter 5
Application to a Real-World
Aviation Policy Analysis
This chapter presents a case study in which the methods developed in this thesis are
applied to analyze a set of real-world aviation policies pertaining to increases in aircraft noise certification stringency. Section 5.1 provides some background on aviation
noise impacts and frames the objectives of the case study. Section 5.2 introduces
the APMT-Impacts Noise Module and explains how it is used in the present context
to perform a cost-benefit analysis of the proposed policy options, identify sources of
uncertainty, and analyze assumptions and tradeoffs. Section 5.3 discusses the case
study results and recommends specific courses of action to support the design of
cost-beneficial policy options.
5.1
5.1.1
Background and Problem Overview
Aviation Noise Impacts
The demand for commercial aviation has risen steadily over the last several decades,
and is expected to continue growing at a rate of 5% per year over at least the next two
decades [129–131]. This wave of growth brings with it increasing concerns regarding
the environmental impacts of aviation, which include aircraft noise, air quality degra-
115
dation, and climate change. Of these issues, aircraft noise has the most immediate
and perceivable community impact [132–134], and was the first to be regulated by
the International Civil Aviation Organization (ICAO) in 1971 with the publication of
Annex 16: Environmental Protection, Volume I: International Noise Standards [135].
There are many effects associated with exposure to aircraft noise, which can be
broadly classified as either physical or monetary. The physical effects of aviation noise
span a range of severities, and include annoyance, sleep disturbance, interference with
school learning and work performance, and health risks such as hypertension and heart
disease [136, 137]. The monetary effects, on the other hand, include housing value
depreciation, rental loss, and the cost associated with lost work or school performance.
The assessment of these monetary impacts is of particular interest to policymakers,
researchers, and aircraft manufacturers, as they provide quantitative measures with
which to perform cost-benefit analysis of various policy options.1
Motivated by a 2004 report to the US Congress on aviation and the environment
[138], the Federal Aviation Administration (FAA) is developing a comprehensive suite
of software tools that can characterize and quantify a wide spectrum of environmental
effects and tradeoffs, including interdependencies among aviation-related noise and
emissions, impacts on health and welfare, and industry and consumer costs under
various scenarios [139]. This effort, known as the Aviation Environmental Tools Suite,
takes as inputs proposed aviation policies or scenarios of interest, which can pertain to
regulations (e.g., noise and emissions stringencies, changes to aircraft operations and
procedures), finances (e.g., fees or taxes), or anticipated technological improvements.
These inputs are processed through various modules of the Tools Suite — in particular
the Aviation environmental Portfolio Management Tool for Impacts (APMT-Impacts)
— to produce cost-benefit analyses that explicitly detail the monetized noise, climate,
and air quality impacts of the proposed policy or scenario with respect to a welldefined baseline [140].
1
While the physical and monetary effects of aviation noise are usually presented separately, the
two categories are not necessarily independent – that is, monetary impacts (typically assessed from
real estate-related damages) often serve as surrogate measures for the wider range of interrelated
effects that are difficult to quantify individually. However, it is recognized that such estimates may
undervalue the full impacts of noise.
116
The Noise Module within APMT-Impacts was developed by He et al. and relates
city-level per capita income to an individual’s willingness to pay for one decibel (dB) of
noise reduction [141, 142]. It is based on a meta-analysis of 63 hedonic pricing aviation
noise studies, which used observed differences in housing markets between noisy and
quiet areas to determine the implicit value of quietness (or conversely, the cost of
noise).2 Because income data are typically readily available for most airport regions
around the world, the income-based model facilitates the assessment of aviation noise
impacts on a global scale. For example, a previous study used the APMT-Impacts
Noise Module to estimate that capitalized property value depreciation due to aviation
noise in 2005 totaled $23.8 billion around 181 airports worldwide, including $9.8
billion around 95 airports in the US [142]. In the following sections, we will discuss
the APMT-Impacts Noise Module in more detail, as well as introduce the aviation
policy case study to which it is applied.
5.1.2
CAEP/9 Noise Stringency Options
Since ICAO’s initial publication of Annex 16 in 1971, international aircraft noise
certification standards have been updated on three different occasions to reflect advances in technology. The most recent stringency increase was adopted in 2001 as
part of the ICAO Committee for Aviation Environmental Protection’s (ICAO-CAEP)
CAEP/5 cycle. In 2007, a formal review of existing noise certification standards was
initiated as part of CAEP/7. Subsequently, five options for noise stringency increase
were proposed during CAEP/9 in 2013, which correspond to changes in noise certification stringency of −3 (Stringency 1), −5 (Stringency 2), −7 (Stringency 3), −9
(Stringency 4), and −11 (Stringency 5) cumulative EPNdB relative to current limits
2
Hedonic pricing (HP) studies typically derive a Noise Depreciation Index (NDI) for one airport,
which represents the percentage decrease in property value corresponding to a one decibel increase
in the noise level in the region. Previous meta-studies have shown that NDI values tend to be
similar across countries and stable over time [143–145]. An alternative to HP is contingent valuation
(CV), which uses survey methods to explicitly determine individuals’ willingness to pay for noise
abatement, or alternatively, willingness to accept compensation for noise increases. While HP and
CV methods both have respective advantages, some key drawbacks which limit their applicability
to cost-benefit analysis of aviation policies include insufficient availability of real-estate data (HP)
[146], inconsistency of results between different study sites (CV) [147, 148], and prohibitive cost of
performing new valuation studies (both).
117
[149].3
The CAEP/9 noise stringency goals are achieved through a mixture of changes
to fleet composition and aircraft operation projected by the ICAO Forecasting and
Economic Support Group (FESG). For the purpose of policy cost-benefit analysis,
the noise impacts of the proposed stringency options are modeled for 99 airports in
the US using the FAA’s Aviation Environmental Design Tool (AEDT) [152]. These
99 airports are listed in Appendix D and belong to the FAA’s database of 185 Shell-1
airports worldwide that account for an estimated 91% of total global noise exposure
[153]. The noise contours for each airport are expressed in day-night average sound
level (DNL) in increments of 5 dB: 55–60 dB, 60–65 dB, 65–70 dB, 70–75 dB, and 75 dB
and above.4 The noise contours corresponding to each stringency option, along with
a baseline scenario with no stringency change, are forecasted from 2006 (the reference
year of aviation activity) to 2036, with a policy implementation year of 2020.
Along with forecasts of future noise, each CAEP/9 noise stringency option also
has an associated cost of implementation, which is the sum of the recurring and nonrecurring costs estimated by the FESG.5 Recurring costs are incurred annually and
consist of fuel costs, capital costs, and other direct operating costs, and were calculated using the Aviation environmental Portfolio Management Tool for Economics
(APMT-Economics) under the assumptions documented in Ref. [149]. Non-recurring
costs include loss in fleet value and cost to manufacturers, and are provided as low
and high estimates for each stringency option.
A policy is said to be cost-beneficial (resulting in a “net benefit”) if its associated
environmental benefits (typically negative) exceed the implementation costs (typically
3
The effective perceived noise in decibels (EPNdB) is the standard metric used in aircraft noise
certification limits, and represents the weighted sum of the sound pressure level from nine contiguous
frequency ranges, with added penalties for discrete pure tones and duration of noise [150, 151].
4
The day-night average sound level (DNL) is the 24-hour A-weighted equivalent noise level with a
10 dB penalty applied for nighttime hours. A similar measure, the day-evening-night average sound
level (DENL), is commonly used in Europe; DENL is very similar to DNL, except that it also applies
a 5 dB penalty to noise events during evening hours.
5
Recurring and non-recurring costs were estimated on a global scale. To obtain US-only policy
implementation costs, it was assumed that US operations comprise approximately 27% of the global
sum, based on previous analyses conducted using APMT-Economics [154].
118
positive), producing a negative value for net monetary impacts.6 Conversely, a policy
is not cost-beneficial (resulting in a “net dis-benefit”) if the net monetary impact is
a positive value.
The objectives of this case study are to explicitly model uncertainties in the
CAEP/9 noise stringency options and the APMT-Impacts Noise Module, understand
their implications for policy net cost-benefit, and identify resource allocation strategies for improving robustness and reliability. The outcomes of this study can help
inform decisions to support the design of cost-beneficial policy implementation responses. Prior work by Wolfe and Waitz used various modules within APMT-Impacts
to estimate the net present value of each stringency option with respect to noise, climate, and air quality impacts [155]. The cost of policy implementation was treated
as a deterministic value equal to the low estimate for each stringency and discount
rate. It was found that Stringencies 4 and 5 resulted in a net dis-benefit across all
discount rates considered, even though the lowest (least conservative) cost estimates
were used. In this case study we will use random variables to characterize uncertainty
in the implementation cost, and focus only on CAEP/9 Stringencies 1–3. Furthermore, we will limit the scope of environmental effects to consider only noise impacts.
Under these assumptions, it is expected that the present analysis will provide a more
conservative estimate of policy net cost-benefit for Stringencies 1–3.
5.2
APMT-Impacts Noise Module
The APMT-Impacts Noise Module is a suite of scripts and functions implemented in
R
the MATLAB
(R2013a, The MathWorks, Natick, MA) numerical computing envi-
ronment. The next sections describe the inputs, assumptions, and impacts estimation
procedure used in the module.
6
We note, however, that depending on the policy, environmental impacts can be either positive
or negative; the same is true of implementation costs.
119
5.2.1
Model Inputs
There are a number of inputs to the APMT-Impacts Noise Module, which can be
classified into three broad categories: scenario, scientific, and valuation. Below, we
describe each category and its respective inputs in greater detail.
Scenario Inputs
The scenario inputs correspond to the policy under consideration, and include demographic data for population and personal income, alternative forecasts of future
aviation activities or situations, and the cost of implementing the proposed policy.
As discussed in Section 5.1.2, noise contours from AEDT correspond to the DNL
of aircraft noise at a particular location, and are computed as yearly averages around
each airport. To evaluate the monetary impacts of a particular aviation policy, two
sets of noise contours are needed: baseline and policy. The baseline contours for the
reference year are constructed using actual aircraft movement data from a representative day of operations. The baseline forecast for future years represents an estimate
of the most likely future noise scenario while maintaining the status quo for technology, fleet mix, and aviation demand. The policy forecast reflects the expected future
noise levels after the implementation of the proposed policy. The monetary impacts
due to policy implementation correspond to the difference between the policy and the
baseline scenarios (henceforth termed the “policy-minus-baseline” scenario).
As will be further explained in Section 5.2.2, the APMT-Impacts Noise Module
assesses monetary noise impacts based on a per person willingness to pay (WTP)
for one decibel of noise reduction, which is computed as a function of the average
city-level personal income. Thus, locality-specific population and income data are
required. Income data may be obtained from a variety of sources, such as national
or local statistical agencies, or the US Bureau of Economic Analysis, which publishes
the annual personal income for each Metropolitan Statistical Area (MSA) [156]. The
income level for each of the 99 airports in the CAEP/9 analysis is listed in Appendix
D. Similarly, population data can be obtained from several sources, including the
decadal US Census, the European Environmental Agency’s population maps, the
120
Gridded Rural-Urban Mapping Project, and national or local statistical agencies.
For the CAEP/9 noise stringency analysis, noise and population data are provided
jointly in the form of the number of persons residing within each 5 dB contour band
surrounding an airport in the years 2006, 2026, and 2036. The population data were
derived from the 2000 US Census, and thus population changes since that time are
not accounted for. Finally, under the current capabilities of the APMT-Impacts Noise
Module, uncertainty associated with income and population data are not modeled.
The cost of policy implementation (IC) is another example of a scenario input. In
the CAEP/9 analysis, the estimated total cost for a particular stringency option can
be obtained by summing the recurring and non-recurring costs for a selected discount
rate. There are two cost totals for each stringency — low and high — corresponding
to the best- and worst-case estimates, respectively. To characterize uncertainty in the
implementation cost, we model this input using a uniform random variable on the
interval bounded by the low and high estimates.
Scientific Inputs
The scientific inputs enable the estimation of monetary impacts for a scenario of
interest, and consist of the background noise level (BNL), noise contour uncertainty
(CU), and the regression parameters (RP) that relate personal income to the WTP for
noise abatement. The uncertainties associated with these inputs arise from limitations
in scientific knowledge or modeling capabilities and are represented probabilistically
using random variables.
Background noise level refers to the level of ambient noise in an airport region,
which is important since monetary impacts are assessed only when the level of aviation
noise exceeds this threshold. Although BNL can vary from region to region, typical
values range between 50–60 dB DNL [143, 148]. To capture uncertainty in this input,
we model BNL as a symmetric triangular distribution between 50 dB and 55 dB, with
a mode of 52.5 dB.
Under the current capabilities of AEDT, the computed noise contours are fixed
values. In order to account for uncertainty in the noise contour level, we assume
121
a triangular distribution for CU with minimum, maximum, and mode at −2 dB,
2 dB, and 0 dB, respectively. This distribution represents an engineering estimate and
captures only uncertainty in the noise contour level, not in the area of the contour,
which can also affect the estimated monetary noise impacts [157].
500
0
−500
Intercept ($/dB/person)
1000
Mean = 39.11
SD = 159.65
−1000
2000
1500
1000
500
0
0
Income coefficient (1/dB/person)
−0.02
0
0.02
0.04
0.06
Number of samples
Number of samples
500
1000
Mean = 0.0114
SD = 0.0061
1500
2000
Figure 5-1: Joint and marginal distributions for the regression parameters obtained
using 10,000 bootstrapped samples
Finally, the regression parameters consist of two quantities, income coefficient and
intercept, which specify a linear relationship between income and WTP for 1 dB of
reduction in aviation noise. For US airports, this relationship is given by:
WTP = Income coefficient × Income + Intercept.
122
(5.1)
Since income coefficient and intercept are not independent, they are modeled jointly
as a random variable in R2 . The joint distribution and corresponding marginal distributions are shown in Figure 5-1.7 These distributions are approximately Gaussian
in shape, and were obtained by bootstrapping the 63 meta-analysis observations of
WTP versus per capita income and performing weighted least-squares regression using
a robust bisquare estimator [142].
Valuation Inputs
Finally, the valuation inputs pertain to subjective judgments regarding the value of
money over time or the significance of environmental impacts, rather than quantities
rooted in scientific knowledge. These factors are modeled as deterministic values instead of random variables. The valuation inputs of the APMT-Impacts Noise Module
consist of the discount rate, income growth rate, and significance level.
The discount rate (DR) captures the depreciation in the value of money over
time, and is expressed as an annual percentage. It is closely related to the time
span of the analyzed policy, which is based on the typical economic life of a building
and the duration of future noise impacts that is considered by the house buyers. In
this analysis, the policy time span is 30 years (2006–2036). Consistent with previous
analyses of the CAEP/9 noise stringency options, the nominal discount rate is selected
to be 3% [155]. However, other discount rates (5%, 7%, 9%) will also be considered
to investigate the effect of this parameter on net cost-benefit. We assume that the
same discount rate is used to compute the net present value of both monetary noise
impacts and recurring implementation costs.
The income growth rate (IGR) represents the annual rate of change in the citylevel average personal income. It is universally applied to the income levels of all
7
In the case of non-US airports, a third regression parameter, the interaction term, is added to
(5.1), such that WTP is given by:
WTP = (Income coefficient + Interaction term) × Income + Intercept.
(5.2)
The interaction term has an approximately Gaussian marginal distribution with mean and standard
deviation equal to 39.11 and 159.65, respectively [142]. For this US-based case study, however, we
will not further discuss the interaction term.
123
airports in the analysis when calculating the WTP for noise abatement. While this
parameter may be user-selected to be any reasonable value (even negative growth
rates), in this analysis it is set to zero so as to consider noise impacts solely due to
the growth of aviation, rather than due to changes in economic activity.
The significance level (SL) is the threshold DNL above which aircraft noise is
considered to have significant impact on the surrounding community. It does not affect
the value of the computed monetary noise impacts, but rather designates impacts as
significant or insignificant, and thereby includes or excludes them from the reported
results. In this analysis, the significance level is set to 52.5 dB, as consistent with
the midrange lens definition in the APMT-Impacts Noise Module.8 Noting that the
lowest noise contour level in the CAEP/9 analysis is 55 dB, this choice implies that
any aviation noise above the background noise level is perceived as having a significant
impact on the community.
Symbol
Factor (Units)
Definition
BNL
Background noise level (dB)
T (50, 55, 52.5)
CU
Contour uncertainty (dB)
T (−2, 2, 0)
Income coefficient (1/dB/person)
Mean = 0.0114, SD = 0.0061
Intercept ($/dB/person)
Mean = 39.11, SD = 159.65
SL
Significance level (dB)
52.5 dB
IGR
Income growth rate (%)
0%
IC
Implementation cost ($)
U [low estimate, high estimate]
DR
Discount rate (%)
3%, 5%, 7%, 9%
RP
Table 5.1: Inputs to CAEP/9 noise stringency case study. The definitions for BNL,
CU, RP, SL, and IGR correspond to the midrange lens of the APMT-Impacts Noise
Module, whereas IC and DR are used in the post-processing of impacts estimates to
compute net cost-benefit.
Table 5.1 summarizes the main model inputs presented in this section. We note
that with the exception of implementation cost, which is added directly to the net
present value of monetary noise impacts (to be discussed in Section 5.2.2), all inputs
8
A lens is a ready-made set of input values that can be used to evaluate policy options given a
particular perspective or outlook. The midrange lens represents a most likely scenario, where all
model inputs are set to their nominal value or distribution.
124
listed in the table are used in impacts estimation for both the baseline and policy
scenarios. Thus, while uncertainties in these inputs affect the baseline or policy
scenario results to the first-order, when considering a policy-minus-baseline scenario,
their effects become second-order. Finally, for each of the probabilistic inputs (BNL,
CU, RP, and IC), 10,000 MC samples are used to characterize the distribution.9
5.2.2
Estimating Monetary Impacts
In this case study, we use the APMT-Impacts Noise Module to assess the net costbenefit of each CAEP/9 noise stringency option, which we estimate as the net present
value of the sum of the monetary noise impacts and implementation cost. A schematic
of the module is shown in Figure 5-2.
For the k th airport in the analysis, the capitalized monetary impacts of aviation
in year t in the dth 5 dB noise contour band is given by V (k, d, t):
V (k, d, t) = WTP(k, t) × ∆dB(k, d, t) × Number of persons(k, d, t),
(5.3)
where WTP(k, t) is obtained using (5.1), and ∆dB(k, d, t), the aviation noise level
above the BNL, is computed as:
∆dB(k, d, t) = Noise contour level(k, d, t) + CU − BNL.
(5.4)
To estimate the total capitalized monetary impacts in year t, we sum V (k, d, t) over
all noise contour bands and across all airports:
V (t) =
XX
k
V (k, d, t).
(5.5)
d
It is important to note that V (t) is a capitalized value that encompasses not only
the property value depreciation due to aviation noise in year t, but also the future
9
In order to check for convergence of the MC samples, the running mean and variance of each
input were plotted versus the sample number. After 10,000 samples, fluctuations in the running
mean and variance were on the order of 0.1% for all inputs.
125
Model Inputs
Scenario
Scientific &
Valuation
Quantities of
Interest
Contour
Uncertainty
Exposure
Area
(dB)
(m2)
(dB)
AEDT
Background
Noise Level
Exposed
Population
Exposed
Population
(dB)
(Persons)
(Persons)
(Persons/m2)
US Census,
EEA, GRUMP
Significance
Level
ΔdB
Personal
Income
Income
Growth Rate
Willingness
to Pay
Capitalized
Impacts
($)
USBEA,
Statistical
Agencies
(%)
($/Person/dB)
($)
Regression
Parameters
Capital
Recovery
Factor
Annual
Impacts
Implementation Cost
($)
Net Present
Value
Physical Impacts
Noise
Contours
Intermediate
Results
Population
Data
(dB)
(dB)
($)
APMTEconomics
Discount
Rate
(%)
($)
Monetary Impacts
Implementation Cost
($)
Figure 5-2: Schematic of APMT-Impacts Noise Module [142, adapted from Figure 4]
noise damages anticipated by the house buyers. This quantity can be transformed
into a net present value (NPV) by applying a selected discount rate:
NPV = V (0) +
T
X
V (t) − V (t − 1)
t=1
(1 + DR)t
,
(5.6)
where V (0) denotes the capitalized impacts in the reference year (2006), and T = 30
is the policy time span. Because baseline and policy scenarios are provided for only a
subset of the years in the policy timespan (2006, 2026, 2036), the year-by-year results
are obtained by linearly interpolating the monetary impacts between fixed contour
years.
We select the quantity of interest for this case study to be the net cost-benefit
of each noise stringency option, given as the net present value of the policy-minus-
126
baseline scenario plus the corresponding cost of implementation. This quantity can
be written as:
Y = NPVpolicy − NPVbaseline + IC.
(5.7)
All monetary values are expressed year 2006 USD. The complexity of Y represents
the intrinsic extent of the distribution for the net monetary impacts of the proposed
stringency option. The risk associated with Y is defined as P (Y > 0), and can be
interpreted as the probability that the stringency option is not cost-beneficial.
5.3
5.3.1
Results and Discussion
Comparison of Stringency Options and Discount Rates
NPV (Billion 2006 USD)
4
3
2
1
0
-1
-2
Str 1
DR = 3%
Str 2
DR = 5%
DR = 7%
Str 3
DR = 9%
Figure 5-3: Comparison of mean estimates for Y in Stringencies 1–3 and discount
rates between 3%–9%. Error bars denote 10th , 90th percentile values.
Figure 5-3 shows a comparison of the mean net monetary impacts of Stringencies
1–3 at discount rates of 3%, 5%, 7%, and 9%. Table 5.2 provides the corresponding
uncertainty estimates. We see that across all discount rates considered, the mean of
Y is negative in Stringency 2, positive in Stringency 3, and mixed in Stringency 1.
The complexity of Y remains relatively constant within each stringency, but increases
significantly between the stringencies. This suggests that an increase in noise stringency is concurrent with a growth in complexity (and thus, a decrease in robustness).
127
Str.
Stringency 1
DR
3%
Mean
-0.12
0
10th
-0.31
-0.19
th
0.07
0.19
SD
0.14
Comp.
Risk
90
5%
7%
Stringency 2
Stringency 3
9%
3%
5%
7%
9%
3%
5%
7%
9%
0.08
0.13
-1.19
-0.72
-0.41
-0.19
2.91
2.29
1.86
1.56
-0.11
-0.06
-1.42
-0.94
-0.63
-0.42
2.42
1.80
1.37
1.07
0.26
0.32
-0.96
-0.50
-0.19
0.03
3.41
2.78
2.35
2.05
0.14
0.14
0.14
0.17
0.16
0.16
0.16
0.37
0.36
0.36
0.35
0.56
0.54
0.53
0.53
0.70
0.66
0.64
0.63
1.49
1.43
1.40
1.38
24%
49%
66%
77%
0%
0%
0%
15%
100% 100% 100% 100%
Table 5.2: Comparison of mean and uncertainty estimates of Y for Stringencies 1–
3 and discount rates between 3%–9%. All monetary values (mean, 10th and 90th
percentiles, standard deviation, and complexity) are listed in 2006 USD (billions).
The risk associated with Y increases with discount rate within Stringency 1, equals
zero in Stringency 2 (except when DR= 9%), and remains at 100% in Stringency 3.
From these results, we conclude that under the current assumptions, Stringency 2
is virtually always cost-beneficial and Stringency 3 is never cost-beneficial. The most
interesting option is Stringency 1, which straddles the border, with risk estimates
ranging between 24–77%. Because of this, we will focus our attention on Stringency
1, and in the following section investigate the factors that drive its uncertainty.
5.3.2
Global and Distributional Sensitivity Analysis
Figure 5-4(a) shows global sensitivity analysis results for Stringency 1 when a discount
rate of 3% is assumed. With a MSI of 0.93, implementation cost is by far the largest
contributor to output variability. On the other hand, the scientific inputs (BNL,
CU, and RP), along with factor interactions (Int.), each contributes only a minuscule
proportion. This apportionment result does not change when the discount rate is
increased (see Table 5.3, left side). Thus, in the remainder of this section, we will
focus on the case where DR = 3%.
The distributional sensitivity analysis results for this case are shown in Figure
5-5(a). We see that the adjusted main effect sensitivity index for each factor varies
linearly with δ. If we permit BNL, CU, RP, and IC to deviate from their nominal
distribution families specified in Table 5.1, the resulting effect on adjSi (δ) is negli-
128
2%
3%1% 1%
4%
32%
43%
93%
BNL
CU
RP
IC
Int.
BNL
CU
RP
Int.
20%
(a) IC ∼ U[low est., high est.]
(b) IC = mean estimate
Figure 5-4: Main effect sensitivity indices for Stringency 1, DR = 3%, with IC modeled probabilistically in (a) and as a deterministic value in (b)
IC ∼ U[low est., high est.]
IC = mean estimate
DR
3%
5%
7%
9%
3%
5%
7%
9%
BNL
0.02
0.01
0
0
0.32
0.32
0.32
0.32
CU
0.01
0
0
0
0.20
0.20
0.20
0.20
RP
0.03
0.01
0
0
0.43
0.43
0.43
0.43
IC
0.93
0.97
0.99
0.99
N/A
N/A
N/A
N/A
Int.
0.01
0.01
0.01
0.01
0.04
0.04
0.04
0.04
Table 5.3: Main effect sensitivity indices for Stringency 1 at various discount rates
gible.10 Thus, we can conclude that irrespective of distribution shape, the relative
factor contribution to output variance does not change with δ.
Next, we investigate the influence of our initial assumption that IC is a uniform
random variable on the interval delimited by the low and high cost estimates. Figures
5-6(a) through 5-6(c) show the distribution of Y for 10,000 MC samples when IC is
assumed to be a uniform, symmetric triangular, or Gaussian random variable on this
interval.11 The variance of IC is decreased steadily (by 50% between the uniform and
10
In DSA, the regression parameters are modeled as a multivariate Gaussian random variable
(MVN) in R2 parameterized by the vector of means M and the covariance matrix Σ. This is an
approximation since the RP distribution is not truly Gaussian (as can be seen from Figure 5-1).
Furthermore, we have not permitted changes to the RP distribution family, as its shape is specified
according to empirical evidence rather than modeling assumptions.
11
Consistent with the approach in Chapter 4, we approximate the Gaussian distribution with
129
0.8
0.8
0.6
0.6
adjSi()
1
i
adjS ()
1
0.4
0.2
0
0
BNL: Tri->Uni
BNL: Tri->Tri
BNL: Tri->Gaussian
BNL: Tri->Bmd Uni
CU: Tri->Uni
CU: Tri->Tri
CU: Tri->Gaussian
CU: Tri->Bmd Uni
IC: Uni->Uni
IC: Uni->Tri
IC: Uni->Gaussian
IC: Uni->Bmd Uni
RP: MVN->MVN
0.4
0.2
0.2
0.4
0.6
0.8
0
0
1
0.2
1-
0.4
1-
0.6
0.8
1
(a) IC ∼ U[low est., high est.]
1
adjSi(δ)
0.8
0.6
0.4
0.2
0
0
0.2
0.4
0.6
0.8
1
1−δ
(b) IC = mean estimate
Figure 5-5: Adjusted main effect sensitivity indices for Stringency 1, DR = 3%, with
IC modeled probabilistically in (a) and as a deterministic value in (b)
triangular cases, and by 66.7% between the uniform and Gaussian cases), which in
turn reduces the standard deviation and complexity of Y . Since µY < 0, a reduction in var (Y ) also decreases P (Y > 0), hence increasing the probability that the
policy will result in a net benefit (i.e., improving reliability).12 Despite these significant reductions in uncertainty, however, IC still remains the largest driver of output
variability, as its main effect sensitivity index (SIC ) equals 0.86 for the symmetric
triangular case and 0.82 for the Gaussian case.
In the limiting case, Figure 5-6(d) shows the distribution of Y when IC is fixed
at a deterministic value equal to the mean cost estimate for Stringency 1. Compared
to the probabilistic cases, there are significant reductions in the standard deviation
a truncated version in which ±3 standard deviations of the distribution fall within the specified
interval.
12
We note that this outcome is not necessarily true of other stringencies and discount rates in the
CAEP/9 analysis. For example, when DR = 7% in Stringency 1, µY > 0 (see Table 5.2), and thus
a decrease in var (Y ) actually increases risk.
130
2500
Mean = $−0.12B
SD = $0.14B
Comp = $0.56B
Risk = 23.9%
2000
Number of samples
Number of samples
2500
1500
SIC = 0.93
1000
500
2000
1500
SIC = 0.86
1000
500
0
−0.6
−0.4
−0.2
0
0.2
0
−0.6
0.4
Net present value (Billion 2006 USD)
−0.4
−0.2
0
0.2
0.4
Net present value (Billion 2006 USD)
(a) IC ∼ U[low est., high est.]
(b) IC ∼ T [low, high, mean]
2500
2500
Mean = $−0.12B
SD = $0.09B
Comp = $0.36B
Risk = 7.3%
2000
Number of samples
Number of samples
Mean = $−0.12B
SD = $0.10B
Comp = $0.43B
Risk = 12.7%
1500
SIC = 0.82
1000
500
2000
Mean = $−0.11B
SD = $0.04B
Comp = $0.15B
Risk = 0%
1500
1000
500
0
−0.6
−0.4
−0.2
0
0.2
0
−0.6
0.4
Net present value (Billion 2006 USD)
−0.4
−0.2
0
0.2
0.4
Net present value (Billion 2006 USD)
(c) IC ∼ N [mean, (high − low)/6]
(d) IC = mean estimate
Figure 5-6: Histogram of Y under various IC assumptions for Stringency 1, DR =
3%
and complexity of Y . Furthermore, the risk of a net policy dis-benefit is reduced
to zero. Figure 5-4(b) shows the GSA results for the deterministic IC case. Of the
scientific inputs, the regression parameters contribute the largest proportion to output
variance at 43%, followed by BNL at 32% and CU at 20%, with interactions again
playing a minor role. As in the probabilistic case, the variance apportionment for
the deterministic IC case remains constant across all discount rates considered (see
Table 5.3, right side). Likewise, the DSA results in Figure 5-5(b) reveal that the
adjusted main effect sensitivity indices tend to vary linearly with δ, and incur only
small deviations when changes between distribution families are considered.
131
5.3.3
Visualizing Tradeoffs
Building off of the variance-based sensitivity analyses from the previous section, we
now investigate how changes in the mean or variance of the model inputs affect complexity and risk in the QOI. For this analysis, we focus on BNL, CU, and IC, whose
distributions are specified according to engineering assumptions. The regression parameters are excluded, as additional research is needed to extend local sensitivity
analysis and visualization techniques to multivariate random variables.
0.2
0.7
0.2
0.8
0.6
IC mean ($B)
0.1
0
−0.1
50
0.56
0.57
−0.2
0.55
IC mean ($B)
0.1
51
52
53
0
0.7
0.5
0.6
0.4
0.5
0.3
0.4
0.2
0.3
0.1
0.2
−0.1
0.1
−0.2
54
55
50
51
52
BNL mean (dB)
53
54
55
BNL mean (dB)
(a) Lines of constant complexity due to changes (b) Lines of constant risk due to changes in
in mean
mean
1
0.25
0.8
Percent reduction in var(IC)
Percent reduction in var(IC)
1
0.25
0.3
0.3
0.35
0.35
0.4
0.4
0.6
0.4
0.2
0
0
0.45
0.45
0.5
0.5
0.55
0.2
0.4
0.55
0.6
0.8
0.6
0.04
0.08
0.08
0.12
0.16
0.12
0.16
0.16
0.4
0.2
0.08
0.04
0.12
0.2
0.2
0
0
1
Percent reduction in var(BNL)
(c) Lines of constant complexity due to
reductions in variance
0.08
0.8
0.12
0.16
0.2
0.2
0.2
0.4
0.6
0.8
1
Percent reduction in var(BNL)
(d) Lines of constant risk due to reductions in
variance
Figure 5-7: Uncertainty contours associated with changes in the mean and variance
of BNL and IC, generated from 10,000 MC simulations (solid colored lines) or approximated using sensitivity analysis results (dashed black lines)
In Figures 5-7 through 5-9, we let the mean of BNL, CU, and IC vary between the
132
0.2 0.8
0.7
0.2
IC mean ($B)
0.1
0
−0.1
−2
−1
0.57
0.55
−0.2
0.56
IC mean ($B)
0.1
0.7
0
0
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
−0.1
−0.2
1
2
−2
−1
CU mean (dB)
0
1
2
CU mean (dB)
(a) Lines of constant complexity due to changes (b) Lines of constant risk due to changes in
in mean
mean
1
0.25
0.8
0.6
0.4
0.2
0.25
0.3
0.3
0.35
0.35
0.4
0.4
0.45
0.45
0.5
0.5
0.55
0
0
0.2
0.4
0.55
0.6
0.8
Percent reduction in var(IC)
Percent reduction in var(IC)
1
0.8
0.6
0.08
0.12
0.16
0.12
0.16
0.16
0.4
0.2
0.08
0.04
0.12
0.08
0.04
0.2
0.2
0
0
1
Percent reduction in var(CU)
(c) Lines of constant complexity due to
reductions in variance
0.08
0.12
0.16
0.2
0.2
0.2
0.4
0.6
0.8
1
Percent reduction in var(CU)
(d) Lines of constant risk due to reductions in
variance
Figure 5-8: Uncertainty contours associated with changes in the mean and variance
of CU and IC
lower and upper bounds of their respective distributions, as well as allow their variance
to be reduced by 0–100% (assuming that such reductions decrease the extent of the
underlying distribution symmetrically about the mean). No changes in distribution
family are considered. The solid colored lines denote contours generated using 10,000
MC simulations at each design parameter combination, whereas the dashed black
lines indicate approximations based on distributional, local, or entropy power-based
sensitivity analysis results. A black X in each figure designates the location of the
initial case as defined in Table 5.1.
Figures 5-7 and 5-8 show how changes in IC trade off against BNL and CU,
respectively. Figures 5-7(a) and 5-8(a) reveal that varying the mean implementation
133
2
2
9
56
.5
1.5 0
0.
1
56
0.
54
0
25
15
0.
.
−0.5 0
−1
0.
0.57
55
−1
0.5
−2
50
51
52
−1.5
53
54
0.
0.2
−2
50
55
35
0
−0.5
3
0.
2
0.
0.
0.5
5
5
0.
CU mean (dB)
0.
.
1 0
7
5
0.
58
0.
CU mean (dB)
1
−1.5
25
0.
15
0.
1.5
51
52
BNL mean (dB)
3
53
54
55
BNL mean (dB)
0.4
0.6
0.8
0.238
0.2
37
0.2
0.237
7
23
0.2
36
38
0.2
0.237
0.4
0
0
1
Percent reduction in var(BNL)
(c) Lines of constant complexity due to
reductions in variance
0.6
0.
55
0.2
38
37
0.
0
0
56
0.2
0.2
0.2
0.4
0.8
0.2
Percent reduction in var(CU)
0.6
0.
Percent reduction in var(CU)
5
5
0.5
0.5
0.8
0.23
1
37
1
7
(a) Lines of constant complexity due to changes (b) Lines of constant risk due to changes in
in mean
mean
0.2
0.4
0.6
0.8
1
Percent reduction in var(BNL)
(d) Lines of constant risk due to reductions in
variance
Figure 5-9: Uncertainty contours associated with changes in the mean and variance
of BNL and CU
cost estimate has no effect on output complexity. This is unsurprising since IC is
simply added to the NPV of the policy-minus-baseline noise impacts according to
(5.7), and thus contributes only a constant shift to the estimates of Y . However,
when the mean value of IC is decreased, the risk of incurring a net policy dis-benefit
is greatly reduced (Figures 5-7(b) and 5-8(b)). A reduction in the mean BNL value
slightly increases complexity and decreases risk, whereas a reduction in the mean
CU value slightly decreases complexity and increases risk. This incongruity can be
explained by (5.4), which states that a decrease in BNL results in a larger value of
∆dB and thus a higher level of estimated noise impacts, whereas a decrease in CU
has the opposite effect. Figures 5-7(c), 5-7(d), 5-8(c), and 5-8(d) show that reducing
134
the variance of IC greatly decreases both complexity and risk in the QOI. This effect
is so dominant that reductions in the variance of BNL or CU have negligible impact
by comparison. These results corroborate the findings of variance-based sensitivity
analysis, and confirm that the most effectual strategy for reducing uncertainty in
policy net cost-benefit is to invest additional resources toward improving estimates
of implementation cost.
Finally, Figure 5-9 illustrates the tradeoffs in output complexity and risk associated with changes in the mean or variance of BNL and CU. Consistent with the trends
in Figures 5-7(a), 5-7(b), 5-8(a), and 5-8(b), we see that complexity decreases when
the mean of BNL is increased, or when the mean of CU is decreased, whereas risk
decreases when the mean of BNL is decreased, or when the mean of CU is increased.
Figures 5-9(c) and 5-9(d) show that reductions in the variance of BNL and/or CU
have little effect on complexity, and virtually no effect on risk. These results validate
our conclusion that from an uncertainty mitigation point of view, it is not worthwhile
to expend resources to learn more about background noise level or noise contour uncertainty; instead, efforts should be directed toward reducing uncertainty in policy
implementation cost.
5.4
Chapter Summary
In this chapter, we applied the uncertainty and sensitivity analysis methods developed
in this thesis to study the net cost-benefit of the CAEP/9 noise stringency options
as estimated using the APMT-Impacts Noise Module. Based on our analysis, we
conclude that among all sources of uncertainty considered, scenario uncertainty in
the form of forecasted policy implementation cost outpaces scientific uncertainty in
the noise monetization model by an order of magnitude. This result holds true across
all stringencies and discount rates considered, as well as for several reasonable specifications of input distribution shape. Additional sensitivity analyses revealed that
as defined, the scientific inputs of the APMT-Impacts Noise Module comprise only a
small contribution to the complexity and risk associated with the net cost-benefit of
135
the proposed policy options. For Stringency 1, we recommend that the best course of
action for ensuring net policy benefit is to improve economic forecasting capabilities
and obtain more precise estimates of implementation cost.
In this case study, we have also highlighted the disparity in uncertainty estimates
for policy cost-benefit when implementation cost is modeled probabilistically versus as
a deterministic value, which represents an extension to previous cost-benefit analyses
of the CAEP/9 noise stringency options. We note, however, that a key difference
between this work and the prior analysis by Wolfe and Waitz lies in having modeled
only the monetized noise impacts of the proposed stringency options, such that the
net present value estimates do not capture the full suite of environmental benefits
associated with policy implementation. A natural extension of the current work is to
incorporate the climate and air quality co-benefits of the CAEP/9 noise stringency
options, as well as characterize the effects of uncertainty in those estimates.
136
Chapter 6
Conclusions and Future Work
The overarching goal of this thesis was to develop methods that help quantify, understand, and mitigate the effects of uncertainty in the design of engineering systems. In
this chapter, we summarize the progress made toward the thesis objectives, highlight
key intellectual contributions, and suggest directions for future research.
6.1
Thesis Summary
For many engineering systems, current design methodologies do not adequately quantify and manage uncertainty as it arises during the design process, which can lead
to unacceptable risks, increases in programmatic cost, and schedule overruns. In this
thesis, we developed new methods for uncertainty quantification and sensitivity analysis, which can be used to better understand and mitigate the effects of uncertainty.
In Chapter 2, we defined complexity as the potential of a system to exhibit unexpected behavior in the quantities of interest, and proposed the use of exponential
entropy to quantify complexity in the design context. This complexity metric can be
used alongside other measures of uncertainty, such as variance and risk, to afford a
richer description of system robustness and reliability.
In Chapter 3, we extended existing variance-based global sensitivity analysis to
derive an analogous interpretation of uncertainty apportionment based on entropy
power, which is proportional to complexity squared. This derivation also revealed
137
that a factor’s contribution to output entropy power consists of two effects, which
are related to its spread (as characterized by intrinsic extent) versus distribution
shape (as characterized by non-Gaussianity). To further examine the influence of
input distribution shape on estimates of output variance, we also broadened the
existing distributional sensitivity analysis methodology to accommodate changes to
distribution family.
Another key aspect the work in this thesis is the development of local sensitivity
analysis techniques to predict changes in the complexity and risk of the quantities of
interest resulting from various design modifications. These local approximations are
particularly useful for systems whose simulation models are computationally expensive, as they can be used to identify tradeoffs and infer trends in the design space
without performing additional model evaluations. Furthermore, they can be connected with budgets for uncertainty and cost in order to identify options for improving robustness and reliability and inform decisions regarding resource allocation. In
Chapters 4 and 5, we demonstrated the uncertainty and sensitivity analysis methods
developed in this thesis on two engineering examples.
6.2
Thesis Contributions
The main contributions of this thesis are:
• The proposal of an information entropy-based complexity metric, which can be
used in the design context to supplement existing measures of uncertainty with
respect to specified quantities of interest;
• The development of local sensitivity analysis techniques that can approximate
changes in complexity and risk resulting from modifications to design parameter
mean or variance, as well as enable the visualization of design tradeoffs;
• The identification of intrinsic extent and non-Gaussianity as distinct features
of a probability distribution that contribute to complexity, as well as the development of techniques to analyze the effect of distribution shape on uncertainty
138
estimates;
• A demonstration of how uncertainty estimates and sensitivity information can
be combined with design budgets to guide decisions regarding the allocation of
resources to improve system robustness and reliability.
6.3
Future Work
There are a number of possible extensions to the research presented in this thesis.
The first is to broaden the application of the uncertainty and sensitivity analysis
methods to problems of higher dimensionality. Although the theory underlying many
of the techniques (e.g., estimation of complexity and entropy power, characterizing
non-Gaussianity using K-L divergence) can easily be extended to higher dimensions,
practical issues often hinder their applicability. For example, density estimation —
upon which entropy estimation is prefaced — is typically straightforward in 1-D; for
higher dimensions, however, it becomes much more challenging, or even downright
intractable. High-dimensional density estimation is an active area of research; it is
expected that advancements in this field will allow the methods developed in this
thesis to be applied to larger classes of engineering problems.
A related issue in computational tractability is the expense of pseudo-random
Monte Carlo simulation. One avenue of future work could lead to the incorporation
of efficient sampling methods, such as Latin hypercube or quasi-random Monte Carlo,
to reduce computational cost. In particular, it would be interesting to study how
such techniques can be combined with acceptance/rejection sampling or importance
sampling to enable sample reuse in distributional sensitivity analysis.
In this thesis, we focused primarily on characterizing the downstream effects of parameter uncertainty, which is due to not knowing the true values of the model inputs.
Another important area for future research is to extend the proposed system design
framework to consider other sources of uncertainty, such as model inadequacy and
observation error. This issue is particularly relevant for systems such as the CAEP/9
noise stringency case study presented in Chapter 5, in which we discovered that by
139
modeling implementation cost probabilistically, we identified a major contributor to
uncertainty in policy net cost-benefit. Within the context of the APMT-Impacts
Noise Module, other potential sources of uncertainty include the spatial and temporal resolution of the AEDT noise contours (which are currently provided in increments
of 5 dB for only three of the years in the 30-year policy duration), as well as changes
in population and income level during the analysis timespan. By constructing a more
accurate picture of the various sources of uncertainty present in a system, we can
better understand their effects on the quantities of interest, as well as take steps to
mitigate potentially detrimental impacts in order to ensure system robustness and
reliability.
140
Appendix A
Bimodal Distribution with
Uniform Peaks
In Section 1.2, we introduced a bimodal distribution consisting of two uniform peaks
(i.e., a “bimodal uniform” distribution), parameterized by six quantities: aL , bL , fL ,
aR , bR , and fR . Below, we reproduce Figure 1-3(d) as well as the expressions for the
𝑓𝑋 𝑥
mean and variance of such a distribution from Table 2.1.
𝑓𝐿
𝑎𝐿
𝑓𝑅
𝑏𝐿
𝑥
𝑎𝑅
𝑏𝑅
Figure A-1: Bimodal uniform distribution with peak bounds aL , bL , aR , and bR , and
peak heights fL and fR
fL 2
(b − a2L ) +
2 L
fL
σ 2 = (b3L − a3L ) +
3
µ=
fR 2
(b − a2R )
2 R
fR 3
(b − a3R ) − (E [X])2
3 R
141
(A.1)
(A.2)
A.1
Specifying Distribution Parameters
Because there are five degrees of freedom in the parameter specification, mean and
variance alone are not sufficient to uniquely define a bimodal uniform random variable.
Thus, we must make additional assumptions about the shape of the distribution. In
this thesis, we assume that the two peaks are symmetric about the mean and that
each peak has width equal to 1/4 of the extent of a uniform random variable with
the same variance. This allows us to compute aL , bL , fL , aR , bR , and fR using the
following procedure.
Algorithm 1 Specify a bimodal uniform random variable X given µ and σ
1. Find the lower and upper bound of a uniform distribution with mean µ and
standard deviation σ:
√
⇒
a = µ − (b − a)/2, b = µ + (b − a)/2
b − a = σ 12
2. Assume that the peaks of X are symmetric, each with a width equal to (b−a)/4.
Thus, the peak heights are fL = fR = 2/(b − a).
3. Let b∗R = b and a∗R = 43 (b − a) + a.
4. Find a∗L and b∗L such that the bimodal uniform random variable X ∗ ∼
BU{a∗L , b∗L , fL , a∗R , b∗R , fR } has a variance of σ 2 . This requires solving the following system of equations:
(b − a)/4 = b∗L − a∗L
2
fL ∗3
fR ∗3
fL ∗2
fR ∗2
∗3
∗3
∗2
∗2
σ = (bL − aL ) + (bR − aR ) −
(b − aL ) + (bR − aR )
3
3
2 L
2
2
5. The mean of X ∗ is µ∗ = (b∗L + a∗R )/2. Compute the difference ∆µ = µ∗ − µ.
Then, shift each of a∗L , b∗L , a∗R , and b∗R by ∆µ to obtain aL , bL , aR , and bR :
aR = a∗R − ∆µ
bR = b∗R − ∆µ
aL = a∗L − ∆µ
bL = b∗L − ∆µ
6. Define X ∼ BU(aL , bL , fL , aR , bR , fR ).
142
A.2
Options for Variance Reduction
Algorithm 1 can be used to define a symmetric bimodal uniform random variable
o
X BU ∼ BU(aoL , boL , fLo , aoR , boR , fRo ) with mean µo and variance σ o2 . For a given value
of δ, we desire to specify parameters for a new bimodal uniform random variable X 0 ∼
BU(a0L , b0L , fL0 , a0R , b0R , fR0 ) such that σ 02 = δσ o2 . Some possible options for variance
reduction include:
1. Decreasing the width of one or both peaks.
2. Decreasing the separation between the peaks.
3. Increasing the height of one peak versus the other.
In this section, we will focus on Options 2 and 3, as Option 1 is typically not
sufficient to reduce variance to the requisite level for most values of δ. Algorithms 2
and 3 provide the procedures to define X 0 using Options 2 and 3, respectively. Note
that neither option is valid for the full range of possible values δ ∈ [0, 1]; instead, each
option is applicable for δ ∈ (δ ∗ , 1].
Using Option 2, the minimum achievable variance occurs when there is no separation between the peaks (i.e., the symmetric bimodal uniform distribution becomes
a uniform distribution), such that b0L = a0R . In this case, σ 02 = (boL − aoL )2 /3 =
(boR − aoR )2 /3. Similarly, using Option 3, variance is minimized when the height of
one peak is reduced to zero while that of the other is increased, such that σ 02 =
(boL − aoL )2 /12 = (boR − aoR )2 /12. The values of δ ∗ for these options are listed below. Finally, we note that Algorithm 1 can also be used to specify a new symmetric bimodal
o
uniform random variable X 0 based on X BU for all values of δ ∈ [0, 1].
Algorithm 1:
Algorithm 2:
Algorithm 3:
δ∗ = 0
(boL − aoL )2
3σ o2
o
(bL − aoL )2
∗
δ =
12σ o2
δ∗ =
143
Algorithm 2 Decrease peak separation to achieve σ 02 = δσ o2
1. Set peak heights fL0 = fLo and fR0 = fRo .
2. Select at random which peak to move. If the left peak is chosen, go to Step 3.
Else, go to Step 4.
3. Set a0R = aoR and b0R = boR . Find a0L and b0L by solving the following system of
equations:
b0L − a0L =
δσ
o2
fRo o
(b − aoR )
fLo R
o
2
fLo 03
fRo o3
fL 02
fRo o2
03
o3
02
o2
= (bL − aL ) + (bR − aR ) −
(b − aL ) + (bR − aR )
3
3
2 L
2
4. Set a0L = aoL and b0L = boL . Find a0R and b0R by solving the following system of
equations:
b0R − a0R =
δσ
o2
fLo o
(bL − aoL )
o
fR
o
2
fL o2
fRo 03
fRo 02
fLo o3
o3
03
o2
02
(b − aL ) + (bR − aR )
= (bL − aL ) + (bR − aR ) −
3
3
2 L
2
5. Define X 0 ∼ BU(a0L , b0L , fL0 , a0R , b0R , fR0 ).
Algorithm 3 Vary peak heights achieve σ 02 = δσ o2
1. Set peak bounds a0L = aoL , b0L = boL , a0R = aoR , and b0R = boR ,
2. Find fL0 and fR0 by solving the following system of equations:
1 = fL0 (boL − aoL ) + fR0 (boR − aoR )
δσ
o2
0
2
fL0 o3
fR0 o3
fL o2
fR0 o2
o3
o3
o2
o2
= (bL − aL ) + (bR − aR ) −
(b − aL ) + (bR − aR )
3
3
2 L
2
3. Select at random which peak height to decrease. If the left peak is chosen, set
fL0 equal to the smaller of {fL0 , fR0 } obtained in Step 2, and fR0 equal to the
larger. Otherwise, set fR0 to equal the smaller of {fL0 , fR0 }, and fL0 equal to the
larger.
4. Define X 0 ∼ BU(a0L , b0L , fL0 , a0R , b0R , fR0 ).
144
o
Figure A-2 shows a bimodal uniform distribution X BU obtained using Algorithm
1 for µo = 0.5 and σ o = 0.29 as well as several examples of bimodal uniform distributions X 0 obtained using Algorithms 1–3 for δ = 0.5. Each realization of X 0 has
σ 0 = 0.20.
5
o
Alg. 1 (XBU )
Alg. 1 (X’)
Alg. 2 (X’)
Alg. 3 (X’)
Probability density
4
3
2
1
0
0
0.2
0.4
0.6
0.8
1
x
Figure A-2: Examples of distributions for X 0 obtained using Algorithms 1–3 (solid
o
lines), based on an initial distribution X BU (dashed green line) and δ = 0.5
A.3
Switching to a Bimodal Uniform Distribution
In this section, we outline the procedure for switching from a uniform, triangular,
or Gaussian random variable X o to a bimodal uniform random variable X 0 . As in
Appendix 3.2, the key steps are to estimate the parameters of X 0 and to specify κ
so that the acceptance/rejection method can be used to generate samples of X 0 from
samples of X o . Examples of reasonable bimodal uniform distributions generated using
the procedure below are shown in Figure A-3.
Selecting a0L , b0L , fL0 , a0R , b0R , and fR0 :
o
1. Use Algorithm 1 to define X BU ∼ BU(aoL , boL , fLo , aoR , boR , fRo ) based on µo ,
σo.
2. Determine which of Algorithms 1–3 can be used to achieve σ 02 = δσ o2 .
145
3. Of the possible algorithms, select one at random and use it to specify X 0 ∼
BU(a0L , b0L , fL0 , a0R , b0R , fR0 ).
Specifying κ:
• Evaluate fX o (x) at each of a0L , b0L , a0R , and b0R .
o
n 0
f0
f0
f0
f
• Set κ = max fX o L(a0 ) , fX o L(b0 ) , fX o R(a0 ) , fX o R(b0 ) .
L
R
R
3.5
7
3
6
2.5
5
Probability density
Probability density
L
2
1.5
1
0.5
0
−0.2
4
3
2
1
0
0.2
0.4
0.6
0.8
1
0
0
1.2
0.2
0.4
x
0.6
0.8
1
x
(a) Uniform to bimodal uniform: X o ∼ U[0, 1],
δ = 0.7, k = 1
(b) Gaussian to uniform: X o ∼ T (0, 1, 0.6),
δ = 0.3, k = 1
2.5
Probability density
2
1.5
1
0.5
0
−3
−2
−1
0
x
1
2
3
(c) Gaussian to triangular: X o ∼ N (0, 1),
δ = 0.9, k = 1
Figure A-3: Examples of switching from a uniform, triangular, or Gaussian distribution to a bimodal uniform distribution. In each figure, the solid red line denotes
fX o (x), the dashed blue line represents fX 0 (x), and the dotted black line shows
κfX o (x), which is greater than or equal to fX 0 (x) for all x.
146
Appendix B
Derivation of Local Sensitivity
Analysis Results
In Section 3.3, we introduced expressions for the sensitivity of complexity and risk to
local changes in the mean or standard deviation of the quantity of interest. Here, we
provide the derivations for (3.15), (3.16), (3.19), and (3.20). Letting Y and Y 0 represent initial and new estimates of the QOI, respectively, the corresponding changes in
mean, standard deviation, complexity, and risk, are given below:
∆µ = µY 0 − µY ,
∆σ = σY 0 − σY ,
∆C = C(Y 0 ) − C(Y ),
∆P = P (Y 0 < r) − P (Y < r).
B.1
Perturbations in Mean
First, we consider the case where Y and Y 0 are related by a constant shift ∆µ, while
∆σ = 0:
Y 0 = Y + ∆µ.
147
(B.1)
B.1.1
Effect on Complexity
A constant shift has no effect on a random variable’s differential entropy, as h(Y +
∆µ) = h(Y ) [88]. Therefore, the exponential entropy of Y and Y 0 are also equivalent:
C(Y 0 ) = exp[h(Y + ∆µ)] = C(Y ).
(B.2)
Since ∆C = 0 for all values of ∆µ, we have that the partial derivative of C(Y ) to µY
is zero:
∂C(Y )
= 0.
∂µY
B.1.2
(B.3)
Effect on Risk
Here, we define a new variable z = y − ∆µ, and note that the probability densities of
Y and Y 0 are related by:
fY 0 (y) = fY (y − ∆µ) = fY (z).
(B.4)
Therefore, the failure probability associated with Y 0 is:
Z
0
r
Z
P (Y < r) =
r
fY 0 (y) dy =
−∞
fY (z) dy.
(B.5)
−∞
Taking the partial derivative of P (Y 0 < r) with respect to µY 0 , and noting that
∂z/∂µY 0 = ∂[y − (µY 0 − µY )]/∂µY 0 = −1, we obtain:
∂P (Y 0 < r)
=
∂µY 0
Z
r
−∞
∂fY (z) ∂z
dy = −
∂z ∂µY 0
Z
r
−∞
∂fY (z)
dy.
∂z
(B.6)
In the limit as ∆µ → 0, z → y and µY 0 → µY . Then, the above expression becomes:
∂P (Y < r)
=−
∂µY
Z
r
−∞
r
∂fY (y)
dy = −fY (y) .
∂y
−∞
148
(B.7)
Commonly, fY (y) → 0 as y → −∞, so we have:
∂P (Y < r)
= −fY (r).
∂µY
B.2
(B.8)
Perturbations in Standard Deviation
Next, we shrink the standard deviation of Y while leaving the mean unchanged. For
this derivation, we scale Y by a positive constant γ, and define new random variable
Y 0 such that:1
Y0 =
Y − µY
,
γ
(B.9)
fY 0 (y) = γfY (γy + µY ).
(B.10)
Note that we have included in Y 0 a constant shift of ∆µ = −µY , such that µY 0 = 0.
This shift serves to recenter Y 0 at the origin so as to simplify the derivation; in
general we do not restrict Y 0 to be zero mean. To account for the shift, we adjust
the requirement on Y 0 accordingly by defining a new constant r0 = r − µY . We also
define a new variable z = γy + µY . The variance of Y 0 is then given by:
σY2 0
Z
∞
1
=γ
y fY (γy + µY ) dy = 2
γ
−∞
2
Z
∞
(z − µY )2 fY (z) dz =
−∞
σY2
,
γ2
(B.11)
which corresponds to the following change in standard deviation:
σY
∆σ =
− σY = σY
γ
1
−1 .
γ
(B.12)
Thus, for γ > 1, standard deviation decreases between Y and Y 0 . The differential
entropy estimates h(Y ) and h(Y 0 ) are related by:
0
Z
∞
h(Y ) = −
fY (z) log[kfY (z)] dz = h(Y ) − log γ.
(B.13)
−∞
Note that in Sections 2.3 and 3.4.2, we described a multiplicative scaling of Y as Y 0 = αY or
Y = α(Y − µY ); in this section we use γ for convenience, where γ = 1/α.
1
0
149
B.2.1
Effect on Complexity
The change in complexity is given by:
exp[h(Y )]
− exp{h(Y )} = C(Y )
∆C =
exp[log γ]
1
−1 .
γ
(B.14)
Combining (B.12) and (B.14), we obtain:
∆C
C(Y )
=
.
∆σ
σY
(B.15)
∂C(Y )
C(Y )
=
.
∂σY
σY
(B.16)
In the limit as ∆σ → 0, we have:
B.2.2
Effect on Risk
The probability of failure associated with Y 0 is:
0
0
Z
r0
P (Y < r ) = γ
fY (γy + µY ) dy = P (Y < γr0 + µY ).
(B.17)
−∞
Taking the partial derivative with respect to γ, we obtain:
∂P (Y < γr0 + µY )
∂
=
∂γ
∂γ
Z
γr0 +µY
fY (z) dz = r0 fY (γr0 + µY ).
(B.18)
−∞
Then, if we let γ → 1 and make the substitution r0 = r − µY , we have:
∂P (Y < r)
= (r − µY )fY (r).
∂γ
(B.19)
We seek to compute the partial derivative of P (Y < r) with respect to σY . Using
the chain rule, we obtain:
∂P (Y < r)
∂P (Y < r) ∂γ
=
∂σY
∂γ
∂σY
150
(B.20)
Recall that from (B.11), γσY 0 = σY . Differentiating both sides with respect to σY 0
allows us to estimate ∂γ/∂σY :
γ+
∂γ
σY 0 = 0,
∂σY 0
∂γ
−γ
=
.
∂σY 0
σY 0
(B.21)
(B.22)
As γ → 1, σY 0 → σY , and the above expression becomes:
∂γ
−1
=
.
∂σY
σY
(B.23)
Combining (B.19), (B.20), and (B.23), we have:
(µY − r)
∂P (Y < r)
=
fY (r).
∂σY
σY
(B.24)
Finally, we note that in the case where risk is expressed as P (Y > r) = 1 − P (Y <
r), the corresponding sensitivities ∂P (Y > r)/∂µY and ∂P (Y > r)/∂σY are related
to (B.8) and (B.24), respectively, through only a change in sign.
151
Appendix C
Derivation of the Entropy Power
Decomposition
In Section 3.4.1, we introduced the following entropy power decomposition, which
apportions N (Y ), the entropy power of the QOI, into contributions due to the entropy
power and non-Gaussianity of the auxiliary random variables Z1 , . . . , Zm and their
interactions:
N (Y )exp[2DKL (Y ||Y G )] =
X
N (Zi ) exp[2DKL (Zi ||ZiG )]
i
+
X
N (Zij ) exp[2DKL (Zij ||ZijG )] + . . .
i<j
G
+ N (Z12...m ) exp[2DKL (Z12...m ||Z12...m
)].
(C.1)
The superscript G in the above expression denotes equivalent Gaussian random variables that have the same mean and variance as the original distribution.
In this section, we derive the entropy power decomposition starting from the
ANOVA-HDMR given in (3.1) and (3.40). We limit our consideration to a system
consisting of only two input factors (m = 2), although the results can be generalized
153
to higher dimensions. For such a system, (3.40) and (3.41) reduce to:
Y = g0 + Z1 + Z2 + Z12 ,
(C.2)
var (Y ) = var (Z1 ) + var (Z2 ) + var (Z12 ).
(C.3)
The auxiliary random variables Z1 , Z2 , and Z12 are uncorrelated, though not necessarily independent.1 For this two-factor system, the distributions of the equivalent
Gaussian random variables and the corresponding probability densities are:
Z1G
∼ N (µZ1 , σZ1 )
fZ1G (z) =
Z2G ∼ N (µZ2 , σZ2 )
fZ2G (z) =
G
Z12
∼ N (µZ12 , σZ12 )
G (z) =
fZ12
Y G ∼ N (µY , σY )
fY G (z) =
−(z − µZ1 )2
q
exp
2σZ2 1
2πσZ2 1
−(z − µZ2 )2
1
q
exp
2σZ2 2
2πσZ2 2
−(z − µZ12 )2
1
q
exp
2σZ2 12
2πσZ2 12
−(z − µY )2
1
p
exp
2σY2
2πσY2
1
To verify the entropy power decomposition, we need to show that:
G
exp[2h(Y, Y G )] = exp[2h(Z1 , Z1G )] + exp[2h(Z2 , Z2G )] + exp[2h(Z12 , Z12
)].
(C.4)
where h(Z1 , Z1G ) represents the cross entropy between Z1 and its equivalent Gaussian
random variable Z1G (likewise Y , Z2 , and Z12 ). By definition, h(Z1 , Z1G ) is computed
as:
1
Since Z12 is a function of both X1 and X2 , it is necessarily dependent on both Z1 and Z2 , which
are respectively functions of X1 and X2 alone. However, Z1 and Z2 are independent, due to our
assumption in Section 1.2 that all design parameters are independent.
154
h(Z1 , Z1G )
Z
∞
=−
fZ1 (z) log fZ1G (z) dz


Z ∞

2 
1
−(z − µZ1 )
fZ1 (z) log q
=−
dz
exp
 2πσ 2

2σZ2 1
−∞
Z1


Z ∞
1
 dz
fZ1 (z) log  q
=−
2
−∞
2πσZ1
Z ∞
−(z − µZ1 )2
fZ1 (z) log exp
−
dz
2σZ2 1
−∞
Z ∞
Z ∞
q
1
2
fZ1 (z) dz + 2
(z − µZ1 )2 fZ1 (z) dz.
= log 2πσZ1
2σ
−∞
Z1 −∞
(C.5)
−∞
Z
∞
Noting that
Z
−∞
h(Z1 , Z1G )
(C.7)
(C.8)
∞
fZ1 (z) dz = 1 and
−∞
(C.6)
(z − µZ1 )2 fZ1 (z) dz = σZ2 1 , (C.8) simplifies to:
q
1
= log 2πσZ2 1 + .
2
(C.9)
Multiplying both sides by two and taking the exponential, we get:
exp[2h(Z1 , Z1G )]
q
1
2
= exp 2 log 2πσZ1 + 2
2
= exp log(2πσZ2 1 ) + 1
= 2πeσZ2 1 .
(C.10)
(C.11)
(C.12)
Similarly, the other terms of (C.4) are given by:
exp[2h(Z2 , Z2G )] = 2πeσZ2 2 ,
(C.13)
G
exp[2h(Z12 , Z12
)] = 2πeσZ2 12 ,
(C.14)
exp[2h(Y, Y G )] = 2πeσY2 .
(C.15)
155
Substituting (C.12)–(C.15) into (C.4), we obtain:
2πeσY2 = 2πeσZ2 1 + 2πeσZ2 2 + 2πeσZ2 12 ,
σY2 = σZ2 1 + σZ2 2 + σZ2 12 .
(C.16)
(C.17)
We have already established that (C.17) is true, as it is equivalent to (C.3). We
conclude therefore that (C.4) also holds true.
Using (2.24) to relate cross entropy to differential entropy and K-L divergence, we
can rewrite (C.4) as follows:
exp[2h(Y ) + 2DKL (Y ||Y G )] = exp[2h(Z1 ) + 2DKL (Z1 ||Z1G )]
+ exp[2h(Z2 ) + 2DKL (Z2 ||Z2G )]
G
+ exp[2h(Z12 ) + 2DKL (Z12 ||Z12
)].
(C.18)
Using (2.26), we can rewrite the above expression in terms of N (Y ):
(2πe)N (Y ) exp[2DKL (Y ||Y G )] = (2πe)N (Z1 ) exp[2DKL (Z1 ||Z1G )]
+ (2πe)N (Z2 ) exp[2DKL (Z2 ||Z2G )]
G
+ (2πe)N (Z12 ) exp[2DKL (Z12 ||Z12
)].
(C.19)
Dividing both sides by (2πe), the result becomes:
N (Y ) exp[2DKL (Y ||Y G )] = N (Z1 ) exp[2DKL (Z1 ||Z1G )]
+ N (Z2 ) exp[2DKL (Z2 ||Z2G )]
+ N (Z12 )
G
exp[2DKL (Z12 ||Z12
)]
,
G
exp[2DKL (Y ||Y )]
(C.20)
which is exactly (C.1) for m = 2. Thus, we have verified the entropy power decomposition from Section 3.4.1.
156
Appendix D
CAEP/9 Analysis Airports
Table D.1 lists the 99 US airports used in the CAEP/9 noise stringency case study in
Chapter 5, along with the average personal income in the corresponding Metropolitan
Statistical Area obtained from the US Bureau of Economic Analysis [156]. Each
airport is identified using its four-letter ICAO designation. All income figures are in
2006 USD.
No.
Airport
City
State
Income
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
KABE
KABQ
KACK
KALB
KATL
KAUS
KBDL
KBFI
KBHM
KBNA
KBOI
KBOS
KBUF
KBWI
KCAE
KCLE
KCLT
KCMH
Allentown
Albuquerque
Nantucket
Albany
Atlanta
Austin
Hartford
Seattle (Boeing Field)
Birmingham
Nashville
Boise
Boston
Buffalo
Baltimore
Columbia
Cleveland
Charlotte
Columbus
PA
NM
MA
NY
GA
TX
CT
WA
AL
TN
ID
MA
NY
MD
SC
OH
NC
OH
35,717
32,935
71,837
37,828
39,160
37,022
46,456
47,006
38,087
38,471
35,591
52,018
33,402
44,704
33,330
38,156
39,542
36,695
157
No.
Airport
City
State
Income
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
KCOS
KCVG
KDAL
KDAY
KDCA
KDFW
KDSM
KDTW
KELP
KEWR
KFAT
KFLL
KGEG
KGON
KGRR
KGSO
KHOU
KHUF
KIAD
KIAH
KILN
KIND
KJAX
KJFK
KLAS
KLAX
KLGA
KLIT
KMCI
KMCO
KMDW
KMEM
KMHT
KMIA
KMKE
KMMH
KMSP
KMSY
Colorado Springs
Cincinnati
Dallas (Love Field)
Dayton
Washington (Reagan)
Dallas/Fort Worth
Des Moines
Detroit
El Paso
Newark
Fresno
Fort Lauderdale
Spokane
Groton/New London
Grand Rapids
Greensboro
Houston (Hobby)
Terre Haute
Washington (Dulles)
Houston (Bush)
Wilmington
Indianapolis
Jacksonville
New York (Kennedy)
Las Vegas
Los Angeles
New York (LaGuardia)
Little Rock
Kansas City
Orlando (International)
Chicago (Midway)
Memphis
Manchester
Miami
Milwaukee
Mammoth Lakes
Minneapolis/St. Paul
New Orleans
CO
OH
TX
OH
DC
TX
IA
MI
TX
NJ
CA
FL
WA
CT
MI
NC
TX
IN
DC
TX
OH
IN
FL
NY
NV
CA
NY
AR
MO
FL
IL
TN
NH
FL
WI
CA
MN
LA
35,649
38,096
40,614
33,649
53,384
40,614
40,432
37,928
24,718
50,843
29,111
42,148
31,626
41,792
32,734
33,978
42,984
26,936
53,384
42,984
31,423
38,338
39,197
50,843
38,183
42,329
50,843
35,840
38,914
34,625
43,294
36,417
43,160
42,148
41,162
39,404
45,002
42,416
158
No.
Airport
City
State
Income
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
KOAK
KOKC
KOMA
KONT
KORD
KORF
KORL
KPDX
KPHL
KPHX
KPIT
KPVD
KRDU
KRIC
KRNO
KROC
KRSW
KSAN
KSAT
KSDF
KSEA
KSFO
KSHV
KSJC
KSLC
KSMF
KSRQ
KSTL
KSWF
KSYR
KTEB
KTOL
KTPA
KTUL
KTUS
KTYS
KVNY
PANC
Oakland
Oklahoma City
Omaha
Ontario
Chicago (O’Hare)
Norfolk
Orlando (Executive)
Portland
Philadelphia
Phoenix
Pittsburgh
Providence
Raleigh/Durham
Richmond
Reno
Rochester
Fort Myers
San Diego
San Antonio
Louisville
Seattle/Tacoma
San Francisco
Shreveport
San Jose
Salt Lake City
Sacramento
Sarasota
St. Louis
Newburgh
Syracuse
Teterboro
Toledo
Tampa
Tulsa
Tucscon
Knoxville
Van Nuys
Anchorage
CA
OK
NE
CA
IL
VA
FL
OR
PA
AZ
PA
RI
NC
VA
NV
NY
FL
CA
TX
KY
WA
CA
LA
CA
UT
CA
FL
MO
NY
NY
NJ
OH
FL
OK
AZ
TN
CA
AK
59,640
36,357
40,476
29,330
43,294
36,567
34,625
38,416
43,548
37,173
38,808
37,777
39,215
39,730
43,803
35,385
41,089
43,967
32,553
35,954
47,006
59,640
33,418
56,124
36,906
38,852
47,497
39,126
36,563
32,574
50,843
32,812
36,470
38,470
33,263
33,150
42,329
42,256
159
No.
Airport
City
State
Income
95
96
97
98
99
PHKO
PHLI
PHNL
PHOG
PHTO
Kailua-Kona
Lihue
Honolulu
Kahului
Hilo
HI
HI
HI
HI
HI
29,185
32,442
39,938
34,160
29,185
Table D.1: CAEP/9 US airports and income levels
160
Bibliography
[1] P. Eremenko. Formal Model-Based Design & Manufacture: A Template for
Managing Complexity in Large-Scale Cyber-Physical Systems. In Conference
on Systems Engineering Research, Atlanta, GA, 2013.
[2] L. Allen, R. Angel, J. D. Mangus, G. A. Rodney, R. R. Shannon, and C. P.
Spoelhof. The Hubble Space Telescope Optical Systems Failure Report. Technical Report NASA-TM-103443, National Aeronautics and Space Administration,
1990.
[3] C. Bolkcom. V-22 Osprey Tilt-Rotor Aircraft: Congressional Research Service
Report for Congress. Technical report, Congressional Research Service, Jan. 7,
2005.
[4] M. Hiltzik. 787 Dreamliner Teaches Boeing Costly Lesson on Outsourcing. Los
Angeles Times, Feb. 15, 2011.
[5] National Aeronautics and Space Administration, Washington, DC. NASA Systems Engineering Handbook, NASA/SP-2007-6105 Rev1 edition, 2007.
[6] N. P. Suh. The Principles of Design. Oxford University Press, New York, NY,
1990.
[7] J. J. Deyst. The Application of Estimation Theory to Managing Risk in Product
Developments. In Proceedings of the 21st Digital Avionics Systems Conference,
volume 1, pages 4A3–1–12, Irvine, CA, 2002.
[8] D. L. Allaire. Uncertainty Assessment of Complex Models with Application
to Aviation Environmental Systems. PhD thesis, Massachusetts Institute of
Technology, Cambridge, MA, 2009.
[9] A. Haldar and S. Mahadevan. Probability, Reliability and Statistical Methods
in Engineering Design. John Wiley & Sons, Inc., New York, NY, 2000.
[10] C. J. Roy and W. L. Oberkampf. A Comprehensive Framework for Verification,
Validation, and Uncertainty Quantification in Scientific Computing. Computer
Methods in Applied Mechanics and Engineering, 200:2131–2144, 2011.
161
[11] R. C. Smith. Uncertainty Quantification: Theory, Implementation, and Applications. SIAM Computational Science & Engineering, Philadelphia, PA, 2014.
[12] W. L. Oberkampf, S. M. DeLand, B. M. Rutherford, K. V. Diegert, and K. F.
Alvin. Error and Uncertainty in Modeling and Simulation. Reliability Engineering and System Safety, 75:333–357, 2002.
[13] A. H-S. Ang and W. H. Tang . Probability Concepts in Engineering: Emphasis
on Applications to Civil and Environmental Engineering. John Wiley & Sons,
Inc., Hoboken, NJ, 2nd edition, 2007.
[14] M. C. Kennedy and A. O’Hagan. Bayesian Calibration of Computer Models.
Journal of the Royal Statistical Society, Series B, 63(3):425–464, 2001.
[15] B. M. Ayyub. Elicitation of Expert Opinions for Uncertainty and Risks. CRC
Press, Boca Raton, FL, 2001.
[16] A. Babuscia. Statistical Risk Estimation for Communication System Design.
PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, 2012.
[17] A. Babuscia an K. Cheung. A Quantitative Approach to Perform Expert Elicitation for Engineering Risk Analysis: Methodology and Experimental Results.
Royal Statistical Society (Series A), 2013 (to appear).
[18] H. Agarwal, J. E. Renaud, E. L. Preston, and D. Padmanabhan. Uncertainty
Quantification Using Evidence Theory in Multidisciplinary Design Optimization. Reliability Engineering and System Safety, 85:281–294, 2004.
[19] J. C. Helton, J. D. Johnson, C. J. Sallaberry, and C. B. Storlie. Survey of
Sampling-Based Methods for Uncertainty and Sensitivity Analysis. Reliability
Engineering and System Safety, 91:1175–1209, 2006.
[20] I. M. Sobol’. On the Distribution of Points in a Cube and the Approximate
Evaluation of Integrals. Zh. Vychisl. Mat. Mat. Fiz., 7(4):784–802, 1967.
[21] H. Niederreiter. Random Number Generation and Quasi-Monte Carlo Methods.
Society for Industrial and Applied Mathematics, Philadelphia, PA, 1992.
[22] T. Bui-Thanh, K. Willcox, and O. Ghattas. Model Reduction for Large-Scale
Systems with High-Dimensional Parametric Input Space. SIAM Journal on
Scientific Computing, 30(6):3270–3288, 2008.
[23] C. Lieberman, K. Willcox, and O. Ghattas. Parameter and State Model Reduction for Large-Scale Statistical Inverse Problems. SIAM Journal on Scientific
Computing, 32(5):2523–2542, 2010.
162
[24] O. Bashir, K. Willcox, O. Ghattas, B. van Bloemen Waanders, and J. Hill.
Hessian-Based Model Reduction for Large-Scale Systems with Initial Condition
Inputs. International Journal for Numerical Methods in Engineering, 73(6):844–
868, 2008.
[25] D. Allaire and K. Willcox. Surrogate Modeling for Uncertainty Assessment
with Application to Aviation Environmental System Models. AIAA Journal,
48(8):1791–1803, 2010.
[26] M. S. Eldred, A. A. Giunta, and S. S. Collis. Second-Order Corrections for
Surrogate-Based Optimization with Model Hierarchies. In 10th AIAA/ISSMO
Multidisciplinary Analysis and Optimization Conference, AIAA Paper 20044457, 2004.
[27] T. W. Simpson, J. D. Peplinski, P. N. Koch, and J. K. Allen. Metamodels
for Computer-Based Engineering Design: Survey and Recommendations. Engineering with Computers, 17(2):129–150, 2001.
[28] N. M. Alexandrov, R. M. Lewis, C. R. Gumbert, L. L. Green, and P. A. Newman. Approximation and Model Management in Aerodynamic Optimization
with Variable-Fidelity Models. Journal of Aircraft, 38(6):1093–1101, 2001.
[29] T. D. Robinson, M. S. Eldred, K. E. Willcox, and R. Haimes. Surrogate-Based
Optimization Using Multidelity Models with Variable Parameterization and
Corrected Space Mapping. AIAA Journal, 46(11):2814–2822, 2008.
[30] R. Spence and R. S. Soin. Tolerance Design of Electronic Circuits. Imperial
College Press, London, UK, 1997.
[31] K. Premphet and W. Paecharoenchai. Polypropylene/Metallocene EthyleneOctene Copolymer Blends with a Bimodal Particle Size Distribution: Mechanical Properties and Their Controlling Factors. Journal of Applied Polymer
Science, 85(11):2412–2418, 2002.
[32] M. E. Fowler, H. Keskkula, and D. R. Paul. Synergistic Toughening in Rubber
Modified Blends. Polymer, 28(10):1703–1711, 1987.
[33] T. Usami, Y. Gotoh, and S. Takayama. Generation Mechanism of Short-Chain
Branching Distribution in Linear Low-Density Polyethylenes. Macromolecules,
19(11):2722–2726, 1986.
[34] E. A. Murphy. One Cause? Many Causes?: The Argument from the Bimodal
Distribution. Journal of Chronic Diseases, 17(4):301–324, 1964.
[35] P. Seeman, C. Ulpian, C. Bergeron, P. Riederer, K. Jellinger, E. Gabriel, G. P.
Reynolds, and W. W. Tourtellotte. Bimodal Distribution of Dopamine Receptor
Densities in Brains of Schizophrenics. Science, 225(4663):728–731, 1984.
163
[36] Y. Wu, C. Zhou, J. Xiao, J. Kurths, and H. J. Schellnhuber. Evidence for a
Bimodal Distribution in Human Communication. Proceedings of the National
Academy of Sciences, 107(44):18803–18808, 2010.
[37] D. G. Xenikos. Modeling Human Dialogue – The Case of Group Communications in Trunked Mobile Telephony. Physica A, 388(23):4910–4918, 2009.
[38] D. Conway, E. Allison, R. Felstead, and M. Goulden. Rainfall Variability in
East Africa: Implications for Natural Resources Management and Livelihoods.
Philosophical Transactions of the Royal Society A, 363(1826):49–54, 2005.
[39] M. E. Angeles, J. E. González, N. D. Ramı́rez-Beltrán, C. A. Tepley, and D. E.
Comarazamy. Origins of the Caribbean Rainfall Bimodal Behavior. Journal of
Geophysical Research: Atmospheres, 115(11):D11106, 2010.
[40] S. Touati, M. Naylor, and I. G. Main. Origin and Nonuniversality of the Earthquake Interevent Time Distribution. Physical Review Letters, 102(16):168501,
2009.
[41] T. Parsons. Global Omori Law Decay of Triggered Earthquakes: Large Aftershocks Outside the Classical Aftershock Zone. Journal of Geophysical Research:
Solid Earth, 107(B9):ESE 9–1–20, 2002.
[42] K. v. Sichart and R.-P. Vollertsen. Bimodal Lifetime Distributions of Dielectrics for Integrated Circuits. Quality and Reliability Engineering International, 7(4):299–305, 1991.
[43] K. L. Wong. The Physical Basis for the Roller-Coaster Hazard Rate Curve for
Electronics. Quality and Reliability Engineering International, 7(6):489–495,
1991.
[44] F. Knoll and T. Vogel. Design for Robustness. IABSE (International Association
for Bridge and Structural Engineering), Zurich, Switzerland, 2009.
[45] A. Karl. Robust Design and Stochastics: Key Methods, Case Studies and
Challenges. In MDO Consortium Workshop, Stanford, CA, 2013.
[46] C. M. Creveling, J. L. Slutsky, and D. Antis, Jr. Design for Six Sigma in
Technology and Product Development. Prentice Hall, Upper Saddle River, NJ,
2003.
[47] K. C. Kapur and L. R. Lamberson. Reliability in Engineering Design. John
Wiley & Sons, Inc., New York, NY, 1977.
[48] L. A. Ealey. Quality By Design: Taguchi Methods and U.S. Industry. ASI Press,
Burr Ridge, IL, 2nd edition, 2009.
164
[49] G. Taguchi. The System of Experimental Design: Engineering Methods to Optimize Quality and Minimize Costs (2 vols.). UNIPUB/Kraus International
Publications and American Supplier Institute, White Plains, NY and Dearborn, MI, 1987.
[50] M. S. Phadke. Quality Engineering Using Robust Design. Prentice Hall, Englewood Cliffs, NJ, 1989.
[51] R. A. Fisher. The Design of Experiments. Oliver & Boyd, Edinburgh, UK,
1935.
[52] G. E. P. Box, W. G. Hunter, and J. S. Hunter. Statistics for Experimenters:
An Introduction to Design, Data Analysis, and Model Building. John Wiley &
Sons, Inc., New York, NY, 1978.
[53] W. J. Welch, T.-K. Yu, S. M. Kang, and J. Sacks. Computer Experiments for
Quality Control by Parameter Design. Journal of Quality Technology, 22(1):15–
22, 1990.
[54] A. C. Shoemaker, K.-L. Tsui, and C. F. J. Wu. Economical Experimentation
Methods for Robust Design. Technometrics, 33(4):415–427, 1991.
[55] R. H. Myers, A. I. Khuri, and G. Vining. Response Surface Alternatives to
the Taguchi Robust Parameter Design Approach. The American Statistician,
46(2):131–139, 1992.
[56] D. D. Frey, F. Engenhardt, and E. M. Greitzer. A Role for One Factor a a
Time Experimentation in Parameter Design. Research in Engineering Design,
14(2):65–74, 2003.
[57] D. D. Frey and H. Wang. Adaptive One-Factor-at-a-Time Experimentation and
Expected Value of Improvement. Technometrics, 48(3):418–431, 2006.
[58] K.-L. Tsui. An Overview of Taguchi Method and Newly Developed Statistical
Methods for Robust Design. IIE Transactions, 24(5):44–57, 1992.
[59] D. D. Frey and R. Jugulum. The Mechanisms by Which Adaptive One-Factorat-a-Time Experimentation Leads to Improvement. Journal of Mechanical Design, 128(5):1050–1060, 2006.
[60] T. B. Savoie and D. D. Frey. Detecting Mistakes in Engineering Models: The
Effects of Experimental Eesign. Research in Engineering Design, 23(2):155–175,
2012.
[61] G. Eckes. The Six Sigma Revolution. John Wiley & Sons, Inc., New York, NY,
2001.
165
[62] G. Tennant. Six Sigma: SPC and TQM in Manufacturing and Services. Gower
Publishing Company, Burlington, VT, 2001.
[63] A. M. Hasofer and N. C. Lind. Exact and Invariant Second-Moment Code
Format. Journal of the Engineering Mechanics Division, 100(1):111–121, 1974.
[64] R. Rackwitz and B. Fiessler. Structural Reliability Under Combined Random
Load Sequences. Computers and Structures, 9(5):489–494, 1978.
[65] B. Fiessler, H. J. Neumann, and R. Rackwitz. Quadratic Limit States in Structural Reliability. Journal of Engineering Mechanics, 1095(4):661–676, 1979.
[66] H. O. Madsen, S. Krenk, and N. C. Lind. Methods of Structural Safety. Prentice
Hall, Englewood Cliffs, NJ, 1986.
[67] W. Weaver. Science and Complexity. American Scientist, 36:536–544, 1948.
[68] B. A. Huberman and T. Hogg. Phase Transitions in Artificial Intelligence Systems. Artificial Intelligence, 33:155–171, 1987.
[69] G. Johnson. Researchers on Complexity Ponder What It’s All About. The New
York Times: Technology, May 6, 1997.
[70] S. Balestrini-Robinson. A Modeling Process to Understand Complex System
Architectures. PhD thesis, Georgia Institute of Technology, Atlanta, GA, 2009.
[71] P. A. Corning. Complexity is Just a Word! Technological Forecasting and Social
Change, 59:197–200, 1998.
[72] S. C. Bankes. Tools and Techniques for Developing Policies for Complex and
Uncertain Systems. In Proceedings of the National Academy of Sciences of the
United States of America, volume 99, pages 7263–7266, 2002.
[73] J. D. Summers and J. J. Shah. Mechanical Engineering Design Complexity Metrics: Size, Coupling, and Solvability. Journal of Mechanical Design,
132(2):021004–1–11, 2010.
[74] D. Braha and O. Maimon. The Measurement of a Design Structural and Functional Complexity. IEEE Transactions on Systems, Man and Cybernetics, Part
A: Systems and Humans, 28(4):527–535, 1998.
[75] A. J. Albrecht and Jr. J. E. Gaffney. Software Function, Source Lines of Code,
and Development Effort Prediction: A Software Science Validation. IEEE
Transactions on Software Engineering, SE-9(6):639–648, 1983.
[76] P. A. Murmann. Expected Development Time Reductions in the German Mechanical Engineering Industry. Journal of Product Innovation Management,
11(3):236–252, 1994.
166
[77] A. Griffin. The Effect of Project and Process Characteristics on Product Development Cycle Time. Journal of Marketing Research, 34(1):24–35, 1997.
[78] M. H. Meyer and J. M. Utterback. Product Development Cycle Time and Commercial Success. IEEE Transactions on Engineering Management, 42(4):297–
304, 1995.
[79] T. J. McCabe. A Complexity Measure. IEEE Transactions on Software Engineering, 2(4):308–320, 1976.
[80] M. V. Tatikonda and S. R. Rosenthal. Technology Novelty, Project Complexity, and Product Development Project Execution Success: A Deeper Look at
Task Uncertainty in Product Innovation. IEEE Transactions on Engineering
Management, 47(1):74–87, 2000.
[81] S. Novak and S. D. Eppinger. Sourcing by Design: Product Complexity and
the Supply Chain. Management Science, 47(1):189–204, 2004.
[82] J. Kim and D. Wilemon. An Empirical Investigation of Complexity and Its
Management in New Product Development. Technology Analysis & Strategic
Management, 21(4):547–564, 2009.
[83] K. M. M. Hölttä and K. N. Otto. Incorporating Design Effort Complexity
Measures in Product Architectural Design and Assessment. Design Studies,
26(5):463–485, 2005.
[84] J. Hartmanis and R. E. Stearns. On the Computational Complexity of Algorithms. Transactions of the American Mathematical Society, 117:285–306,
1965.
[85] S. Arora and B. Barak. Computational Complexity. Cambridge University
Press, New York, NY, 2009.
[86] A. N. Kolmogorov. Three Approaches to the Quantitative Definition of Information. Problems of Information Transmission, 1(1):1–7, 1965.
[87] G. J. Chaitin. On the Simplicity and Speed of Programs for Computing Infinite
Sets of Natural Numbers. Journal of the ACM, 16(3):407–422, 1969.
[88] T. M. Cover and J. A. Thomas. Elements of Information Theory. John Wiley
& Sons, Inc., New York, NY, 1991.
[89] C. A. Rodriguez-Toro, S. J. Tate, G. E. M. Jared, and K. G. Swift. Complexity
Metrics for Design (Simplicity + Simplicity = Complexity). In Proceedings of
the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture, volume 217, pages 721–725, 2003.
167
[90] M. Tribus and E. C. McIrvine. Energy and Information (Thermodynamics and
Information Theory). Scientific American, 225(3):179–188, 1971.
[91] C. E. Shannon. A Mathematical Theory of Communication. The Bell System
Technical Journal, 27:379–423 and 623–656, 1948.
[92] J. Rifkin and T. Howard. Entropy: A New World View. The Viking Press, New
York, NY, 1980.
[93] J. S. Coleman. An Introduction to Mathematical Sociology. The Free Press,
New York, NY, 1964.
[94] K. D. Bailey. Sociological Entropy Theory: Toward a Statistical and Verbal
Congruence. Quality and Quantity, 18(1):113–133, 1983.
[95] K. D. Bailey. Social Entropy Theory. State University of New York Press,
Albany, NY, 1990.
[96] S. Lloyd and H. Pagels. Complexity as Thermodynamic Depth. Annals of
Physics, 188:186–213, 1988.
[97] D. Allaire, Q. He, J. Deyst, and K. Willcox. An Information-Theoretic Metric of
System Complexity With Application to Engineering System Design. Journal
of Mechanical Design, 134(10):100906–1–10, 2012.
[98] N. P. Suh. Complexity: Theory and Applications. Oxford University Press, New
York, NY, 2005.
[99] L. L. Campbell. Exponential Entropy as a Measure of Extent of a Distribution.
Probability Theory and Related Fields, 5(3):217–225, 1966.
[100] G. H. Hardy, J. E. Littlewood, and G. Pólya. Inequalities. Cambridge University
Press, Cambridge, UK, 1934.
[101] H. V. Dannon. Power Means Calculus, Product Calculus, Harmonic Mean
Calculus, and Quadratic Mean Calculus. Gauge Institute Journal, 4(4):1–88,
2008.
[102] E. T. Jaynes. Information Theory and Statistical Mechanics. The Physical
Review, 106(4):620–630, 1957.
[103] E. T. Jaynes. Information Theory and Statistical Mechanics II. The Physical
Review, 108(2):171–190, 1957.
[104] B. M. Ayyub and G. J. Klir. Uncertainty Modeling and Analysis in Engineering
and the Sciences. CRC Press, Boca Raton, FL, 2010.
[105] M. H. DeGroot and M. J. Schervish. Probability and Statistics. Addison-Wesley,
Boston, MA, 3rd edition, 2002.
168
[106] D. P. Bertsekas and J. N. Tsitsiklis. Introduction to Probability. Athena Scientific, Nashua, NH, 2nd edition, 2008.
[107] D. Mukherjee and M. V. Ratnaparkhi. On the Functional Relationship Between
Entropy and Variance with Related Applications. Communications in Statistics
– Theory and Methods, 15(1):291–311, 1986.
[108] N. Ebrahimi, E. Maasoumi, and E. S. Soofi. Ordering Univariate Distributions
by Entropy and Variance. Journal of Econometrics, 90(2):317–336, 1999.
[109] S. Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of
Mathematical Statistics, 22(1):79–86, 1951.
[110] S. Verdú and D. Guo. A Simple Proof of the Entropy-Power Inequality. IEEE
Transactions on Information Theory, 52(5):2165–2166, 2006.
[111] S. Kaplan and B. J. Garrick. On the Quantitative Definition of Risk. Risk
Analysis, 1(1):11–27, 1981.
[112] L. A. Cox, Jr. What’s Wrong with Risk Matrices? Risk Analysis, 28(2):497–512,
2008.
[113] J. Li and D. Xiu. Evaluation of Failure Probability Via Surrogate Models.
Journal of Computational Physics, 229(23):8966–8980, 2010.
[114] A. Saltelli, K. Chan, and E. M. Scott. Sensitivity Analysis. John Wiley & Sons,
Inc., New York, NY, 2000.
[115] R. I. Cukier, C. M. Fortuin, K. E. Shuler, A. G. Petschek, and J. H. Schaibly.
Study of the Sensitivity of Coupled Reaction Systems to Uncertainties in Rate
Coefficients I: Theory. Journal of Chemical Physics, 59:3873–3878, 1973.
[116] J. H. Schaibly and K. E. Shuler. Study of the Sensitivity of Coupled Reaction Systems to Uncertainties in Rate Coefficients II: Applications. Journal of
Chemical Physics, 59:3879–3888, 1973.
[117] A. Saltelli and R. Bolado. An Alternative Way to Compute Fourier Amplitude
Sensitivity Test. Computation Statistics & Data Analysis, 26:445–460, 1998.
[118] T. Homma and A. Saltelli. Importance Measures in Global Sensitivity Analysis
of Nonlinear Models. Reliability Engineering & System Safety, 52(1):1–17, 1996.
[119] I. M. Sobol’. Sensitivity Estimates for Nonlinear Mathematical Models. Mathematical Modeling and Computational Experiment, 1(4):407–414, 1993.
[120] I. M. Sobol’. Global Sensitivity Indices for Nonlinear Mathematical Models
and Their Monte Carlo Estimates. Mathematics and Computers in Simulation,
55:271–280, 2001.
169
[121] I. M. Sobol’. Theorems and Examples on High Dimensional Model Representation. Reliability Engineering & System Safety, 79(2):187–193, 2003.
[122] A. Saltelli, S. Tarantola, F. Campolongo, and M. Ratto. Sensitivity Analysis
in Practice: A Guide to Assessing Scientific Models. John Wiley & Sons, Inc.,
Hoboken, NJ, 2004.
[123] J. J. Lee, I. A. Waitz, B. Y. Kim, G. G. Fleming, L. Maurice, and C. A.
Holsclaw. System for Assessing Aviation’s Global Emissions (SAGE), Part
2: Uncertainty Assessment. Transportation Research Part D: Transport and
Environment, 12(6):381–395, 2007.
[124] K. Marais, S. P. Lukachko, M. Jun, A. Mahashabde, and I. A. Waitz. Assessing
the Impact of Aviation on Climate. Meteorologische Zeitschrift, 17(2):157–172,
2008.
[125] D. Allaire and K. Willcox. A Variance-Based Sensitivity Index Function for
Factor Prioritization. Reliability Engineering and System Safety, 107:107–114,
2012.
[126] H. Liu, W. Chen, and A. Sudjianto. Relative Entropy Based Method for Probabilistic Sensitivity Analysis in Engineering Design. Journal of Mechanical Design, 128(2):326–336, 2006.
[127] H. Takeda, P. Veerkamp, T. Tomiyama, and H. Yoshikawa. Modeling Design
Processes. AI Magazine, 11(4):37–48, 1990.
[128] J. D. Irwin and R. M. Nelms. Basic Engineering Circuit Analysis. John Wiley
& Sons, Inc., Hoboken, NJ, 8th edition, 2005.
[129] FAA Aerospace Forecast, Fiscal Years 2009-2025. Technical report, United
States Federal Aviation Administration, 2009.
[130] B. Metz, O. R. Davidson, P. R. Bosch, R. Dave, and L. A. Meyer, editors.
Contribution of Working Group III to the Fourth Assessment Report of the
Intergovernmental Panel on Climate Change. Cambridge University Press, New
York, NY, 2007.
[131] A. Schäfer and I. Waitz. Air Transportation and the Environment. Transport
Policy, 2014. http://dx.doi.org/10.1016/j.tranpol.2014.02.012.
[132] Results From a Survey of the Nation’s 50 Busiest Commercial Service Airports.
Technical Report GAO/RCED-00-222, United States General Accounting Office, 2000.
[133] Y. Schipper. Environmental Costs in European Aviation. Transport Policy,
11(2):141–154, 2004.
170
[134] P. J. Wolfe, S. H. L. Yim, G. Lee, A. Ashok, S. R. H. Barrett, and I. A. Waitz.
Near-airport Distribution of the Environmental Costs of Aviation. Transport
Policy, 2014. http://dx.doi.org/10.1016/j.tranpol.2014.02.023.
[135] Annex 16: Environmental Protection Volume I: Aircraft Noise. Technical report, International Civil Aviation Organization, 2005.
[136] S. McGuire. Sleep Disturbance Due to Aircraft Noise. Technical report, Partnership for AiR Transportation Noise and Emissions Reduction (PARTNER),
Project 25, 2009.
[137] H. Swift. A Review of the Literature Related to Potential Health Effects of
Aircraft Noise. Technical report, Partnership for AiR Transportation Noise
and Emissions Reduction (PARTNER), Project 19, 2009.
[138] I. Waitz, J. Townsend, J. Cutcher-Gershenfeld, E. Greitzer, and J. Kerrebrock.
Aviation and the Environment: A National Vision Statement, Framework for
Goals and Recommended Actions. Technical report, Report to the United States
Congress, 2004.
[139] A. Mahashabde, P. Wolfe, A. Ashok, C. Dorbian, Q. He, A. Fan, S. Lukachko,
A. Mozdzanowska, C. Wollersheim, S. R. H. Barrett, M. Locke, and I. A. Waitz.
Assessing the Environmental Impacts of Aircraft Noise and Emissions. Progress
in Aerospace Sciences, 47(1):15–52, 2011.
[140] I. Waitz, S. Lukachko, Y. Go, P. Hollingsworth, K. Harback, and F. Morser.
Requirements Document for the Aviation Environmental Portfolio Management
Tool. Technical Report PARTNER-COE-2006-001, Partnership for AiR Transportation Noise and Emissions Reduction (PARTNER), Project 19, 2006.
[141] Q. He. Development of an Income-Based Hedonic Monetization Model for the
Assessment of Aviation-Related Noise Impacts. Master’s thesis, Massachusetts
Institute of Technology, Cambridge, MA, 2010.
[142] Q. He, C. Wollersheim, M. Locke, and I. Waitz. Estimation of the Global
Impacts of Aviation-Related Noise Using an Income-Based Approach. Transport
Policy, 2014. http://dx.doi.org/10.1016/j.tranpol.2014.02.020.
[143] J. P. Nelson. Meta-Analysis of Airport Noise and Hedonic Property Values:
Problems and Prospects. Journal of Transport Economics and Policy, 38(1):1–
28, 2004.
[144] J. P. Nelson and R. B. Palmquist. Review of APMT Benefit Valuation Methods and Suggestions for Modifications. Technical report, Prepared for the Massachusetts Institute of Technology, 2008.
171
[145] Z. Wadud. A Systematic Review of Literature on the Valuation of Local Environmental Externalities of Aviation. Technical report, Omega: Aviation in a
Sustainable World, Cambridge, UK.
[146] C. Kish. An Estimate of the Global Impact of Commercial Aviation Noise.
Master’s thesis, Massachusetts Institute of Technology, Cambridge, MA, 2008.
[147] P. A. Diamond and J. A. Hausman. Contingent Valuation: Is Some Number
Better Than No Number? Journal of Economic Perspectives, 8(4):45–64, 1994.
[148] S. Navrud. The State-of-the-Art on Economic Valuation of Noise. Technical
report, Final Report to the European Commission DG Environment, 2002.
[149] Economic Assessment of the Noise Stringency Scenarios. Technical Report
CAEP/9-IP/12, ICAO Committee for Aviation Environmental Protection,
2013.
[150] K. D. Kryter. The Meaning and Measurement of Perceived Noise Level. Noise
Control, 6(5):12–17, 1960.
[151] Federal Aviation Regulations, Noise Standards: Aircraft Type and Airworthiness Certification. Technical Report 14 CFR Part 36, United States Federal
Aviation Administration, 2009.
[152] Report on the Environmental Portion of the Noise Stringency Assessment. Technical Report CAEP/9-IP/20, ICAO Committee for Aviation Environmental
Protection, 2013.
[153] Environmental Tool Suite Frequently Asked Questions.
http:
//www.faa.gov/about/office_org/headquarters_offices/apl/research/
models/toolsfaq/. United States Federal Aviation Administration [Online
Resource]; Accessed March 11, 2014.
[154] Cost-Benefit Analysis of CAEP/9 Noise Stringency Options. Technical Report CAEP/9-IP/25, ICAO Committee for Aviation Environmental Protection,
2013.
[155] P. J. Wolfe and I. A. Waitz. Aircraft Noise Certification Stringency: Benefits,
Costs, and Environmental Co-benefits. Transportation Research Part D, 2014
(in review).
[156] Local Area Personal Income. http://www.bea.gov/regional/. United States
Department of Commerce Bureau of Economic Analysis [Online Resource]; Accessed June 15, 2010.
[157] R. Tam, P. Belobaba, K. R. Polenske, and I. Waitz. Assessment of Silent
Aircraft-Enabled Regional Development and Airline Economics in the UK. In
Proceedings of the 45th AIAA Aerospace Sciences Meeting and Exhibit, Reno,
NV, 2007.
172