Douglas Van Bossuyt
PhD Qualifier
June 11, 2009
 OSU Triple
Play
 Research Interests
• Collaborative design
• Complex system design
• Design for cultures
• Psychology
• Business management
•
Overview of Design Trade Studies
• Methods, Tools, Groups Who Perform, Examples
•
Overview of Risk and Uncertainty
• Risk quantification and mitigation tools
• Uncertainty assessment methods
• Uncertainty mitigation tools
•
•
•
•
Possible ways to incorporate risk and
uncertainty into Trade Studies
Future areas of research
Expected contributions of overall research
Research plan
 Complex
systems are here to stay
 Every complex system design tries to
maximize system utility
• System utility metrics: ROI, system integrity,
public perception of project, etc…
 Subsystems
optimized to achieve high
overall system utility
 Design
parameters (mass, power, volume,
cost, heat dissipation, etc.) used to define
subsystem parameters that determine
system-level utility
 Parameters are traded between
subsystems to optimize design in Trade
Studies
 Risk and uncertainty of systems is
another important factor in complex
system design
 Definitions
• Risk: probability of event X impact of event
 Sometimes more narrowly means probability of
catastrophic event X impact of event
• Uncertainty: caused by variability and doubt in
the status of an output that is either predictable
or unpredictable, or caused by an unknown
process or device
• Reliability: Ability of a device to perform as
intended over a given period of time
 Definitions
(Con’t)
• Robustness: Ability of a device to continue to
properly function under changes in input
variables.
• Design Margins: Quantify the influence of
uncertainties in the design process. Often a high
and low bounding around a central parameter
quantification.
 Methods
developed from this research
will improve system utility and integrity
 Improved utility and integrity uses
resources more efficiently and produces
more desirable results
•
Overview of Design Trade Studies
• Methods, Tools, Groups Who Perform, Examples
•
Overview of Risk and Uncertainty
• Risk quantification and mitigation tools
• Uncertainty assessment methods
• Uncertainty mitigation tools
•
•
•
•
Possible ways to incorporate risk and
uncertainty into Trade Studies
Future areas of research
Expected contributions of overall research
Research plan
 Trade
Studies in complex systems design
 Both design and decision tool
 Trade Studies attempt to find maximum
system utility
 System utility defined by many metrics:
• Cost
• Return on Investment
• System Reliability
 Multi-step process to perform a Trade Study
• Understand system goals, objectives and constraints
•
•
•
•
•
•
(Eg: Functional requirements)
Develop alternative conceptual design solutions
Evaluate alternatives based on system utility
Use mathematical models where appropriate to
determine system utility
Rank alternatives according to their system utility
Remove less desirable alternatives
Either refine and continue to eliminate alternatives
or select most desirable alternative
Image Source: NASA Systems Engineering Handbook
 Trade
Studies search for max system utility
 Many
mathematical ways to find max utility
 Modern software packages available to find
optimum design points
• ICEMaker: Used by many Collaborative Design Centers
to find optimum designs
• Advanced Trade Space Visualization (ATSV): Used to
graphically view and explore optimum design points
ATSV Screenshots
Image Source: https://webhosting.its.psu.edu/atsv/webfiles/glyphscatter
/WebStart_files/image003.jpg
 Software
(Con’t)
• ModelCenter: Integrates capabilities of ATSV
with ability to link together many different types
of programs
 Many
other types of software available
that help perform Trade Studies
Image Source: Jensen, et. al.: ME 519 Class Project
 Many
CDCs exist in government
organizations, academia, and industry
 Original is Team-X housed at NASA JPL
• Helped NASA reduce time to finish Trade Studies
from 3-9 months to 2-3 days
• Reduced cost by a factor of five
 Other
NASA facilities with CDCs: Langley
Research Center, Goddard, Johnson
Space Center
Image Source: http://jplteamx.jpl.nasa.gov/images/teamx/team.jpg
 European
Space Agency uses CDCs and
Trade Studies
 Boeing, Aerospace Corporation, TRW,
and other aerospace companies use
Trade Studies
 Several academic institutions also use
trade studies
 Many
examples in literature of Trade
Studies
 Most come from Team-X but some also
from academic institutions
 Very few from private industry
• Due to proprietary information, etc
 For
those interested, long list of Trade
Studies is available
•
Overview of Design Trade Studies
• Methods, Tools, Groups Who Perform, Examples
•
Overview of Risk and Uncertainty
• Risk quantification and mitigation tools
• Uncertainty assessment methods
• Uncertainty mitigation tools
•
•
•
•
Possible ways to incorporate risk and
uncertainty into Trade Studies
Future areas of research
Expected contributions of overall research
Research plan
 Overview
of methods to account for risk
 Overview of uncertainty and how to
account for it in design process
 Note: many methods not reviewed here
due to space and time constraints
 Many
methods and tools available
 Some used in practice, some only in
academia
 Practice
• RBD, Databases, FMEA/FMECA , ETA, FTA, PRA,
QRA
 Theory
• FFDM, FFIP, RED, HiPHOPS, RUBIC, FFA
 Used
for understanding fault tolerance
 Energy, information, or material flow
through block diagram
Image source: http://www.itemsoft.com/rbd.shtml
 Contain
failure and reliability data on
systems, subsystems, components, and
processes
 Proprietary and industry-specific
 High amount of front-end work to have
worthwhile database
 Often used in oil, automotive, and
aerospace industries
 FMEA used to examine:
• Potential failures modes
• Effects of failures
• Severity of the effects
• Potential causes of the failures
• Probability or potential probability of failure
• Current detection methods of failure
• Detectability of failure
• Recommendations to mitigate cause or effects of
failure
 FMEA
also can be used to assign a Risk
Priority Number
 RPN = Severity x Occurrence x Detection
• Severity of each failure is rated
• Likelihood of each occurrence is rated
• Likelihood of prior detection is rated
 FMECA
is an extension of FMEA. Adds
criticality analysis to FMEA.
 Mode Criticality = Expected Failures X Mode Ratio of
Unreliability X Probability of Loss
 Item Criticality = SUM of Mode CriticalitieS
Image Source: http://www.weibull.com/basics/fmea_fig1.htm
 ETA
is visual representation of failure
events and mitigating events in a system
 Used in safety system analysis
 Starting point is failure event
 Subsequent levels show additional
failures and mitigations
Image Source: http://www.event-tree.com/images/et_example.JPG
 FTA
starts with failure at top-level and
proceeds down to analyze all possible
causes of failure
 Boolean operators and logic gates used
Image Source: http://www.isograph-software.com/ftpoverdgc.htm
 PRA
is used to quantify the risk of failure
in a system
 Employs FTA, ETA, and other techniques
as desired
 PRA quantifies risk by magnitude and
likelihood of each possible failure
 PRA is essentially an umbrella for several
other risk methods
 Used
when quantitative assessment is not
possible
• Not enough time, money, expertise
 Relies
on expert opinions
 Usually performed by interviewing key
designers to determine their belief in the
level of risk of a design
 FFDM
used to investigate potential failure
modes during conceptual design
 Uses failure databases to find failure
rates of generic components
 Improves on FMEA and related
techniques
Image Sources: Stone, Tumer, Van Wie: The Function-Failure Design Method
 FFIP
estimates potential failures and their
propagation paths through systems
 Three components to FFIP:
• Graphical system model
• Behavioral simulation
• Reasoning scheme called Function Failure Logic
Image Sources: Kurtoglu and Tumer: A Graph-Based Fault Identification and
Propagation Framework for Functional Design of Complex Systems
 An
extension of FFDM
 Quantifies risks identified in FFDM
 Automated process for combining
historical risk data with new system
architectures
 Uses fever charts to show risks
 Displays riskiest failure states
Image Source: Lough, Stone, Tumer: Implementation Procedures for the Risk in
Early Design (RED) Method
 HiPHOPS
uses elements of FMEA, FTA,
and others to assess risk in systems
 Model of system is annotated with
formalized logical component failure
descriptions and expected effects
 This method is too complex to ever gain
widespread adoption
A
continuous risk management tool
 Used to identify risk elements during
conceptual design
 RUBIC continuously optimizes budgetary
resources to mitigate risks
 Graphical tool helps find Pareto optimal
sets of resource allocations
Image Source: Mehr, Tumer: Risk-Based Decision-Making for Managing Resources
During the Design of Complex Space Exploration Systems
 FFA
captures physical system
architecture including connections of
energy, material, and data flows in a
functional model
 Model contains sensor information,
failure modes of each component,
propagation effects of failure modes, and
propagation timing
 Approach requires high level of detail in
system before it is useful
 State
Event Fault Tree Analysis
 Component Fault Tree Analysis
 Simulation-Based Probabilistic Risk
Analysis
 Component Stress and Conceptual
Strength Interference Theory
 Various Bayesian Network Analysis tools
 Many others
 All
try to identify and quantify risk
 All good for identifying riskiest points in
designs
 In practice, lists of failures versus failure
paths methods
 Most theoretical tools trying to find
subsystem and component interaction
risks
•
Overview of Design Trade Studies
• Methods, Tools, Groups Who Perform, Examples
•
Overview of Risk and Uncertainty
• Risk quantification and mitigation tools
• Uncertainty assessment methods
• Uncertainty mitigation tools
•
•
•
•
Possible ways to incorporate risk and
uncertainty into Trade Studies
Future areas of research
Expected contributions of overall research
Research plan
 Definitions
of uncertainty
 Assessing System Uncertainty
 Mitigating Uncertainty
Image Source: http://www.martin-koser.de/images/enjoy%20uncertainty.jpg
 Many
different ways to define uncertainty
and many different places for it to be
found
 Easiest to think of uncertainty as being
made of many different types and falling
into two categories
 Categories:
• Intrinsic: Caused by randomness in nature
• Epistemic: Caused by lack of knowledge or data
 Several
ways to assess uncertainties:
• Probabilistic Methods
• Bayesian Techniques
 1st, 2nd, 3rd level Bayesian Analysis
 Bayesian Team Support
• Stimulation Methods
 Monte Carlo Methods
 These
methods quantify the behavior of
model uncertainties as a result of random
model design input variables
 This allows engineers to find variables
that are the most sensitive to change
• Engineers then concentrate on these variables
 Based
on Bayesian statistics and
probability
 Probability interpreted as a state of
knowledge
 Bayesian probability assumes that
posterior probability is proportional to
prior probability
 All Bayesian analysis based on Baye’s
Theorem
 First Level Bayesian Analysis:
• Used for creating system success rate probabilities
based on past success and failure data
 Second Level Bayesian Analysis:
• Used for systems with no prior data but that are
similar to existing systems. Existing system data
used
 Third Level Bayesian Analysis:
• Same as Second Level but with normalizing
available comparable systems data. Makes best
estimate of future system success rates
 Bayesian Team
Support helps to solve
Arrow’s Paradox
• Arrow’s Paradox is not being able to rank order a
group’s voting choices (eg: A-B-C-A)
 BTS
assumes all information is “Uncertain,
incomplete, inconsistent, and evolving”
 BTS implemented in Accord software
package
 BTS helps groups make decisions robustly
by showing where information needs to be
improved, etc…
Image Source: Ullman. Making Robust Decisions
 Many
techniques to simulate systems
 Most work by simulating system design
or input variables
• Use random or semi-random numbers
 Simulation
Methods useful for when
direct analytic solution is not available
 Monte Carlo Methods
 Class
of computational algorithms
 Models with a high degree of uncertainty in
input variables use MCM
 Model outputs calculated using random or
well-chosen semi-random input variables
with many repetitions
• Large dataset is created to adequately explore
design space
 Many
different algorithms available
•
Overview of Design Trade Studies
• Methods, Tools, Groups Who Perform, Examples
•
Overview of Risk and Uncertainty
• Risk quantification and mitigation tools
• Uncertainty assessment methods
• Uncertainty mitigation tools
•
•
•
•
Possible ways to incorporate risk and
uncertainty into Trade Studies
Future areas of research
Expected contributions of overall research
Research plan
 Natural
Uncertainties:
• Hard to address without changing natural
environment
• Eg: dikes to hold back rivers, oceans; bridges
over bodies of water; snow sheds for rail lines…
 Model
Structure and Parameter
Uncertainties:
• Usually addressed by refining model and
parameters
 Data
Uncertainties
• Hard to discover and diagnose. If found then:
 Inaccurate measurements: Better measurement tools
 Data gauging network problems: Improve networks
 Data handling and transcription errors: Fix process
 Alternative approach: Relax requirements so
uncertainties are acceptable
 Computational Uncertainties
• Faulty hardware: Very hard to find and diagnose. If
found, replace bad hardware.
• Faulty software: Also hard to find. Fix software as
needed. (Truncation and rounding errors can be
fixed with longer integer registers, etc…)
 Operational Uncertainties
• Use Total Quality Management, Human Factors
Engineering, etc to reduce this uncertainty
 Behavioral and Ambiguity Uncertainties
• Various corrective behavioral techniques available
•
Overview of Design Trade Studies
• Methods, Tools, Groups Who Perform, Examples
•
Overview of Risk and Uncertainty
• Risk quantification and mitigation tools
• Uncertainty assessment methods
• Uncertainty mitigation tools
•
•
•
•
Possible ways to incorporate risk and
uncertainty into Trade Studies
Future areas of research
Expected contributions of overall research
Research plan
 No
examples beyond Thunnissen of risk
and uncertainty being traded in Trade
Studies
 Thunnissen proposed a method to do this
 This method is rudimentary and needs to
be expanded
 Thunnissen also proposed a method of
design margins for Trade Studies. Also
needs to be implemented.
 Thunnissen
was influenced by Antonsson
and others
• Grayscale System Reliability: Quantifies
influence of partial failure states on system
integrity and reveals tradeoffs between system
reliability and cost
• Method of Imprecision: Represents uncertainty
and preference on 0-1 scale. System to trade
attribute levels without allowing any attribute to
go to zero performance.
 Some
interest at Team-X to implement
• Thunnisson’s methods
• Bayesian Team Support
• Risk methods
 Work
conducted at JPL this summer to
integrate some methods into Trade
Studies using ModelCenter (replacing
ICEMaker)
•
Overview of Design Trade Studies
• Methods, Tools, Groups Who Perform, Examples
•
Overview of Risk and Uncertainty
• Risk quantification and mitigation tools
• Uncertainty assessment methods
• Uncertainty mitigation tools
•
•
•
•
Possible ways to incorporate risk and
uncertainty into Trade Studies
Future areas of research
Expected contributions of overall research
Research plan
 The
valuation and perception of risk and
uncertainty is not well accounted for in the
complex system design process
• It is researched in insurance industry, stock market,
and elsewhere. Need to move this knowledge into a
workable form for engineers.
 Cultural
factors that affect risk and
uncertainty need to be better understood
by engineers
• Several methods in business to quantify culture.
Need to move and adapt into engineering.
 Method
to maximize system utility and
integrity via expanding Trade Studies
• Risk methods
• Uncertainty methods
• Design Margins
• Risk and uncertainty perception and valuation
• Cultural methods
 Integration
of method with ModelCenter
May 09
Complete PhD qualifier. Research design Trade Studies and
risk and uncertainty. Prepare for summer internship at JPL.
June-August
09
Work with Steve Wall and Team-X at JPL. Prepare first
example of design margins in Trade Studies using
ModelCenter for Team-X.
September 09 Submit paper to Journal of Engineering Design detailing
work at JPL.
Sept-Dec 09
Revise design margins in Trade Studies tool. Start work on
integrating risk and uncertainty into Trade Studies.
January 10
Submit conference papers to IDETC and others.
Jan-March 10
Take final required coursework. Continue work on
integrating risk and uncertainty into Trade Studies.
April-May 10
PhD Prelim. Publish journal paper on integrating risk and
uncertainty into Trade Studies.
June-Aug 10
Work either at JPL, overseas, or another company to
continue work. Test risk and uncertainty methods in an
industrial setting. Begin investigating perception and
valuation of risk, and cultural constructs of risk.
Sept-Dec 10
Finish work on integrating risk and uncertainty into Trade
Studies. Implement valuation and perception metrics, and
cultural standards.
January 11
Submit conference papers to IDETC, ICED and others.
Jan-March 11
Submit journal paper.
April-June 11
Prepare and defend dissertation. Take long vacation.
 Why
so focused on aerospace?
 Why worthy of a PhD?