National Aeronautics and Space Administration
Engineer’s Perceptions of Risk
Lynne P. Cooper, Ph.D.
Jet Propulsion Laboratory
California Institute of Technology
- and Daniel J. Epstein Department of Industrial and
Systems Engineering
University of Southern California
March 6, 2012
Presented at the USC CSSE Annual Research Review 2012
lynne.p.cooper@jpl.nasa.gov or lynnecoo@usc.edu
©2012 L.P.Cooper
Project Risk Scenario
What is this judgment based on?
System Design
That doesn’t
make sense
Experts
x1
x2
x3
Y
x4
xn
Inputs
Team
Model
Output
2
“Risk” from a different perspective:
One that includes:
• Emotions
• Intuition
• Relationships
• Collective belief
“Pre-Quantitative Risk”
3
Research Approach
• Real world teams
• Working on high risk, high technology projects
• Over time
• Discussion Analysis:
• Language used
• Flow of conversation through topics
• Development of topics throughout project
• Actions taken
4
Observed Behavior
• Very little discussion “about” risk
• Circular conversations
• Déjà vu discussions
• Very few formal decisions
• Quantification for justification – after decision
was made
• General perception of decreasing risk
5
Language of Risk
• Rarely talked about risk explicitly
• But language of risk and uncertainty
permeated team discussions
Neg Outcome
Negative Outcome - Uncertainty
Uncertainty - Opportunity
Neg Outcome Trend
Opportunity
Uncertainty Trend
Opportunity Trend
Normalized Frequency
80
70
60
50
40
30
20
10
6
0
0
20
40
60
80
Day of Project
100
120
140
Components of Risk Discussions
• Goals
• Design
• Environment
• How things interact
• Bad things that could happen
• What they don’t know
• What they could influence/control
• Acceptable levels of risk
7
How Components Fit Together
Goal
Uncertainty
Design
Interactions
Ability to
Influence
Bad things
that could
happen
Judgment of
Acceptability
Risk
Action
Environment
8
= interactions
Visualizing Risk: Sliders
Degree of Risk
Too Little
Low
Acceptable Risk
Medium
Too Much
High
Acceptable Risk Thresholds
9
Aggregating Risk
Site Selection
Thermal Environment
Energy Demands
Power System
Batteries
O&M Costs
Solar Arrays
• Not linear
• Highly dynamic
• Strange couplings
10
Results & Practical Implications
(continued)
• Very little discussion “about” risk
• Circular conversations
• Déjà vu discussions
11
Circular Conversations
Verbal Simulation of System
Diagnostic value
• Scope problem areas
Builds Intuition
Intervention Value
• Accelerated cycles
• Model development
Identify Interactions
Test Changes
Insight into Nature of Relationships
12
Déjà vu Discussions
Developing a Story
Diagnostic value
Integrate ideas
Refine messages
Synchronization
• Assess progress based on
clarity
Intervention Value
• Accelerate story
development
• On-boarding
Improved external
communications
13
Discussion-based Evolution of Risk
Déjà vu Discussions
Story
Circular Conversations
Evidence
High Confidence Risk Assessments
14
In Summary
• Engineers base critical decisions on prequantitative conceptions of risk
• Pre-Quantitative Risk includes
• Interactions among goals, design elements,
environments, risks, assessments of
riskiness
• Broad view of uncertainty
• Ability to influence
• Judgments about the acceptability of risk
• Team discussion can be used as a
diagnostic, as well as for interventions
• Visualization techniques can help
develop shared understanding of risk
15
“The revolutionary idea that defines the
boundary between modern times and the
past is the mastery of risk: the notion that
the future is more than a whim of the gods
and that men and women are not passive
before nature”
Peter L. Bernstein, 1996, p.1
16
Thank You
17