NASA: Engineering Space Exploration
National Aeronautics and Space Administration
Presented to the Conference on Quality in the
Space and Defense Industries 2007
Cape Canaveral, Florida
March, 2008
www.nasa.gov
C. Herbert Shivers, PhD, PE, CSP
Deputy Director, Safety and Mission
Assurance Directorate
NASA/Marshall Space Flight Center
Launching to the Moon, Mars,
and Beyond
From
Determinism to
“Probabilism”
National Aeronautics and Space Administration
www.nasa.gov
Changing our
mindsets, or why
PTC isn’t an
easy sell - yet
My Muddled Mind
Robust Design
?
Deterministic Design
SDF
Safety
Block Diagrams,
Probabilistic Design
DOE
Predictive
Models &
Algorithms
PRA
Factors
Redundancy
Life Cycle Prediction
Systems of
Systems
Models
PDF
FMECA, FTA,
DFMR
SWP
PD
RAM Design
Diagnostics &
Prognostics
System Safety Tools Feed Risk
Informed Decision Making
Method
Technique
Qualitative
Qualitative
Risk
Risk
Assessment
Assessment
FMEA.
FMEA.
HA,
HA,
ESD,
ESD,
ETA,
ETA,
FTA,
FTA,
RBD
RBD
Probabilistic
Probabilistic
Risk
Risk
Assessment
Assessment
Management
Management
System
System
Decision
Decision
Analysis
Analysis
Bryan O’Connor, Chief, Safety and Mission Assurance, NASA
SEPT 8, 2007, Huntsville, AL
Actuarial/
Actuarial/
Statistical
Statistical
Analyses
Analyses
Legend:
FMEA HA ESD
ETA
FTA
RBD -
Application
Technical
Technical
Risk
Risk
and/or
and/or
Program
Program
Risk
Risk
Failure Modes & Effects Analysis
Hazard Analysis
Event Sequence Diagram
Event Tree Analysis
Fault Tree Analysis
Reliability Block Diagram
S&MA in Ares Design – Summary
• Using the Ares quantitative safety and reliability
requirement to enforce the “design for reliability and
safety” paradigm shift
• Improving system safety by using a functional analysis
system approach to model and understand integrated
system failures similar to the Shuttle foam problem.
• Improving design reliability by using probabilistic
engineering physics-based modeling
• Evaluating and understating design uncertainty and
design margins using probabilistic engineering
techniques
Excerpted from Chris Cianciola, S&MA, MSFC, CQSDI
March 2008
Uncertainty Quantification and Risk Assessment
risk-informed decision making
Uncertainties sources
manufacturing, storage, aging, use environments, and scenarios
Uncertainties types
randomness, lack of knowledge
Model to predict uncertainties in systems
Uncertainties outside acceptable limits?
Uncertainties produce inadequate margins?
Most important contributors to uncertainty (sensitivity analyses)?
Physics model form uncertainty
Sensitivity Analysis Only several contribute most of the uncertainty in system
response
Requirements Uncertainty
Probabilistic Models Limitations
One of many decision variables
Subjective information - scientific and engineering judgment is
necessary
Known unknowns and unknown unknowns exist
Resources are limited
Results uncertainty and credibility – challenges decision makers
Social, economic, and political factors exist
Credibility
• Is the methods framework credible?
• Address the credibility concern with
verification and validation
Special study data
• Start with robustness and margin in design
• A 30% design margin is not plausible for space
flight – it’s just too expensive
• Our margins are so thin that we really need to
investigate margin
• In our case we want to whittle the margin down
as much as possible but still maintain the
robustness we need
• Getting those design margin trades into the
system analysis is critical
NASA special study data 2006
Special study data
• We still use deterministic design and build margin into designs even
though we have the computational capabilities to us PT methods
• We do not have probabilistic design methods in place
• Probabilistic methods will have to be an investment by NASA into
the long term technology base and tools
• First you need a proper deterministic goal and then you can
successfully apply a probabilistic model
• Numerical Propulsion System Simulation is something that must be
implemented to get this country to Mars
• The entire thought process needs to change, this is not a
deterministic world – everything is probabilistic
• In the deterministic method there is no role for uncertainty or margin
• Right now technology is so fast we are dumping technology on
people who are not educated to use these technologies
appropriately
NASA special study data 2006
Special study data
• Redundancy is a part of our culture right now
because it is much easier than looking for a
solution using margin
• When you consider uncertainty in the design
phase you are able to examine the trade space
more efficiently and thoroughly.
• The nature of the probabilistic skills dealing in
design is very different from the traditional PRA.
NASA special study data 2006
Why PTC Now?
• Design engineers don’t understand how to
account for uncertainty in the design process
and how to explore the trades throughout
• In the past we’ve had so much money and time
for testing, but we don’t have this anymore
• Apollo did 14,000 tests on the LEM, do you think
we could do that now?
• We must trade ability to test versus desired
reliability demonstration
NASA special study data 2006
Other Notable Thoughts
• Probabilistic requirements ultimately relate to risk in
achieving a level of performance or safety.
• The most important reason for using a structured
approach is to ensure that the potential failure modes of
a design are identified early in the design process to
better understand the risks.
• An understanding of risks due to incomplete or
inaccurate modeling of a design and the operational
environment, or due to uncertainty related to the data
used in a model is necessary so that a requirement can
be verified with confidence.
Surendra N. Singhal, Engineering Directorate,
Marshall Space Flight Center, Sept. 18, 2007
A Way Forward
• Need a coherent institutionalized effort
• Need a regular training of the entire
Center staff (many unaware of PT and its
benefits)
• Need a core group where people and
projects can go for help
PT Selling Points
Utilizes Physics based behavioral model
Considers inherent uncertainties, modeling uncertainties, lack of
data, human error, measurement error
Compensates for unknowns using statistical methods
Utilizes past performance data to develop behavioral model
Quantifies safety measures
Qualifies prediction accuracy
Robert J. Kuper, Executive for Reliability and Quality,
Systems Engineering Conference, 23 October 2002