Week 3 – Assessing Risk
Risk Analysis Process





Technical & systematic process
Examine events
Focus on causes, not symptoms
Determine interrelationships
Document impact in terms of probability
& consequence
Analysis Phase

May actually start during identification
process





Availability of experts
As a natural by product of interviewing, etc.
Define likelihood or probability ratings
Define ratings for severity of consequence
Establish assessment matrix
RM Execution Phases
Assessment Phase

Primary objective – Identify & analyze
program risks




To control the most critical risks
Id factors that contribute most to achieving
desired results
Id factors to use in setting cost, schedule,
and performance objectives
Problem definition stage
Assessment Process




Basis of most RM actions
Quality of the assessment determines
the effectiveness of the RM program
Tools are available but no one tool has
all the answers
Definitizes probability & consequence of
potential events
Identify Risk Drivers



Compile potential risk events
Describe in detail to understand
significance & causes
Are events that have significant impact
to program


Adverse Consequence
Significant Opportunity
Address Root Sources of Risk



Assess processes vs acceptable best practices
Consider cost, schedule & technical impacts
Use Willoughby templates in DoD 4245.7




Templates describe an area of risk
Specify technical methods for risk reduction
Correlate with acquisition phases & milestones
Primarily applicable during development
Willoughby Templates
Risks in Acquisition Process

Templates address risks by common
DoD program elements




Discussion of risk
Outline for reducing risk
Relates to program phase timeline
Industrial process for design, test &
production of low risk products
Process-Oriented Assessment


Program critical technical processes
Evaluate program baselines




Against current & historical data
Critical paths
Process constraints
Critical inputs from outside program
Process Metrics

Track process of developing, building &
introducing the system


Meeting established milestones
Variances from baselines




Earned value
Parametric comparisons
Details of critical path, constrained process
items
Dependencies beyond program scope
Product Focused Assessment

Address risk by program output




Utilize WBS breakout
Constraints of master schedule
Interfaces with other programs/products
Use independently or in conjunction
with process analysis techniques
Product Metrics


Track development of the product
Measures of effectiveness &
performance





Progress in meeting requirements
Test & analysis results
Ability to produce & deliver
Availability of resources
Comparison to past experience
Areas for Assessment
Cost Assessment

Use probability distributions




Define distribution & range by WBS
element
Use Monte Carlo simulation to assess &
aggregate
Utilize expert opinion
Address performance and schedule
causes of cost risk
Ranges & Distributions
Risk Element
Scrap/Rework
(Firm Fixed
Price negotiated)
Schedule Delay
(Level of effort
Labor)
Meet Performance
Reqmts (new
technology)
Total Impact:
Min
Most Likely
(2,000)
10,000
Max
12,500
-24000
0
24000
0
2000
20000
Simulated
Value
Rationale
10,774 Min: potential underrun in scrap/rework rates
Most Likely: Expect current rates to increase 1%
Max: Exceed scrap/rework rates by 25%
(13,364) Min: complete 1 week early
Most Likely: Meet negotiated schedule
Max: Exceed negotiated schedule by 1 week
3,263 Min: no add'l testing
Most Likely: 1 day add'l testing
Max: 10 days add'l testing
672
Distributions
Statistics
Forecast: Total Cost Impact
Summary:
Display Range is from -30,000 to 50,000
Entire Range is from -20,864 to 47,999
After 10,000 Trials, the Std. Error of the Mean is 112
Statistics:
Trials
Mean
Median
Mode
Standard Deviation
Variance
Skewness
Kurtosis
Coeff. of Variability
Range Minimum
Range Maximum
Range Width
Mean Std. Error
Value
10000
14,069
13,993
--11,164
124,630,124
0.00
2.68
0.79
-20,864
47,999
68,863
111.64
Percentile
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Value
-20,864
-628
4,351
8,035
11,203
13,993
16,966
20,181
23,762
28,667
47,999
Schedule Assessment

Extension of Critical Path Method




Define duration ranges & distributions for
scheduled activities
Use analytical techniques to identify
schedule drivers
Address external schedule impacts
Assess probability & magnitude of an
overrun
Schedule Assessment
Min
Most
Likely
Max
Simulated
Value
AA
90
100
120
115.5
Min: 10% challenge of program estimates
AB
72
80
85
72.9
Most Likely: program estimate
AC
45
50
60
56.6
Max: worst case scenario
AD
67.5
75
95
79.4
Task
Total A
305
324.3
Rationale
Cost-Schedule Containment Chart
Modeling & Simulation




Physical, mathematical, or logical
representation of system or process
Implementation of a model over time
Use data or expert opinion to select
PDF (probability density function)
Preferred for assessing cost or schedule
risk
Which Distribution to Use?
Which Distribution to Use?
More Modeling & Simulation

As Virtual Prototyping




Replica of a system flow
Duplication of a physical product
Representation of a process flow
May be the only way to verify or
validate design or process, or assess
risk
Before Using any Model

Verify


Validate


Functions as designed
Represents the system it models
Accredit

Is acceptable for the special purpose
Exercise
By study group, identify how you would verify,
validate, and accredit these simulation methods

Group 1 – Virtual simulation



Group 2 – Constructive simulation



Physical & electrical system representation
Ex: Built-in training
Represents the system & its usage
Ex: Mock-up
Group 3 – Live simulation


Uses real operators & equipment
Ex: Operational tests
You have __ minutes for this exercise. Be prepared to
discuss your results.
Establish Rating Criteria

From empirical data if possible


Significance based on expert opinion


Else, define rigorous qualitative measures
Polling program & industry experts
Variance from best practices

Accepted rating definitions
Best Practices example

Low



Medium



Little or no anticipated impact
Normal mgmt attention should control at
acceptable level
May cause some impact
Special action & attention may be required
High


Likely to cause significant impact
Significant additional action & attention would be
required
Concurrency Impact




Overlap between program phases
From combining phases / activities
Schedule adequacy
Assess with best practices or historical data
Overlap in DT&E, Production
DT&E % Complete
>67%
33% - 67%
<33%
0%
Concurrency Risk
Low
Moderate
High
Very High
Developing Measurement Scales

Qualitative analysis



Ordinal scales
Defines a relative relationship
Quantitative analysis



Numerical methods
Calibrated ordinal or cardinal scales
May be linear or nonlinear
Qualitative Scales

Levels defined by experts



Criteria used coordinated with PM
Early definition avoids bias
Reflect relative relationship between
risk /consequence levels


Absolute value on scale not known
Not valid for mathematical manipulation
Ordinal scales




Generally reflect ranked data
Difference between scale values is
unknown, and not necessarily constant
Misleading if scale is defined
numerically
Mathematical operations:


Are at best meaningless
At worst: misleading
Quantitative Scales

Reflect measurable relationship
between risk /consequence levels



Cardinal or validated ordinal scales
Valid for mathematical manipulation
Tendency to use for calculating a ‘risk
value’


Empirical data
Simulation & decision analysis results
Qualitative vs. Quantitative

Depends on several factors





Information available
Nature & phase of program
Available resources – personnel, schedule,
budget
Availability of experts
Generally qualitative at first, then
quantitative as needed or feasible
Probability Ratings



Use empirical data if available
Otherwise, use expert opinion, etc.
Important to know if scales are ordinal
Probability Level What is the Likelihood the Risk Event Will Happen
a
Remote
<10%
b
Unlikely
10% to 35%
c
Likely
35% to 65%
d
Highly Likely
65% to 90%
e
Near Certainty
>90%
Consequence Ratings

Define for technical, schedule and cost
Assessment Matrix

Define overall risk ratings
Prioritization
Multi-Voting Technique



Each team member receives votes
equal to ½ the number of risks
Team members vote for risk items they
think have the highest priority
Risks are ranked according to the vote
Benefits v. Biases ?
Borda Ranking Method

Ranks risk by criticality based on
identified criteria



Rank by impact of consequence
Rank on probability of occurrence
Borda count used to rank risk by
criticality
Impact Frequencies
Risk #
1
2
3
4
5
Conseq
Critical
Serious
Serious
Critical
MInor
Conseq
C.freq
Critical
2
Serious
2
Moderate
Minor
1
Negligible
Prob
60%
100%
90%
60%
40%
C.rank
C.rank
P.rank
Borda Cnt Borda Rnk Per Matrix
Probaility P.freq
91-100%
1
61-90%
1
41-60%
2
11-40%
1
0-10%
P.rank
Rank Consequence, Probability
Risk #
1
2
3
4
5
Conseq
Critical
Serious
Serious
Critical
MInor
Conseq
C.freq
Critical
2
Serious
2
Moderate
Minor
1
Negligible
Prob
60%
100%
90%
60%
40%
C.rank
1.5
3.5
5
C.rank
P.rank
Borda Cnt Borda Rnk Per Matrix
Probaility P.freq
91-100%
1
61-90%
1
41-60%
2
11-40%
1
0-10%
P.rank
1
2
3.5
5
C, P-rank = 1/2 [ 2 (# risks @ higher levels) +1 + # risks @ current level ]
Determine Borda Count
Risk #
1
2
3
4
5
Conseq
Critical
Serious
Serious
Critical
MInor
Prob
60%
100%
90%
60%
40%
C.rank
1.5
3.5
3.5
1.5
5
P.rank
3.5
1
2
3.5
5
Borda Cnt Borda Rnk Per Matrix
5
5.5
4.5
5
0
for N total risks, Borda Count = ( N - C.rank) + ( N - P.rank)
Highest count is most critical
Borda v. Matrix Rank
Risk #
1
2
3
4
5
Conseq
Critical
Serious
Serious
Critical
MInor
Prob
60%
100%
90%
60%
40%
C.rank
1.5
4
4
1.5
4
P.rank
3.5
1
2
3.5
5
Borda Cnt Borda Rnk Per Matrix
H
1
5
H
0
5.5
M
3
4.5
H
1
5
L
4
0
Borda Rank = # risks with a higher Borda count
Risk Matrix
Negligible
P\C
L
0-10%
L
11-40%
L
41-60%
M
61-90%
M
91-100%
Minor
L
L
M
M
H
Moderate
L
M
M
M
H
Serious
M
M
M
M
H
Critical
M
H
H
H
H
Assessment Documentation


Goal: communicate to customer,
program management, team
Define aggregation criteria


Voting method
Summary level



Process, e.g. WBS break out
Area of risk – cost, schedule, performance
By criticality
Impact & Rating Criteria
Risk Aggregation
Summarize Rank Frequency
P\C
Negligible
0-10%
11-40%
41-60%
61-90%
91-100%
L
M
H
Minor
Moderate
Serious
Critical
1
2
1
1
Minimum impact; minimum oversight required
Some disruption; may need additional attention
Unacceptable; major disruption likely; priority attention required
Order Risks by Borda Count
Risk #
2
1
4
3
5
Conseq
Serious
Critical
Critical
Serious
MInor
Prob
100%
60%
60%
90%
40%
C.rank
4
1.5
1.5
4
4
P.rank
1
3.5
3.5
2
5
Borda Cnt Borda Rnk Per Matrix
5.5
0
H
5
1
H
5
1
H
4.5
3
M
0
4
L
Aggregating Results

Define reporting format that
communicates the best



Group by phase, product, WBS, …
Order by color, Borda count, …
Information provided



Description, relationship to requirements
Action required
Risk owner, …
Aggregation Results
Common Failures in RM Process

Definition phase too focused on activities


Unclear relationships or motives



Need detail on motives, timelines, resources
Between organizations, analysis methods, models
In identifying sources of uncertainty, risk,
consequences
Addressing commonality between issues

Links, interdependencies
Document RM process flow to clarify
Next Time: Risk Handling,
Monitoring

Read: Risk Management Guide
section 5.7
Project – Part II
Submit a paper copy of results with your final exam.
1. Explain how to map risks on your class project into the integrated master
schedule you prepared for the mid-term exam and
Provide a cost-schedule containment chart with a description of the steps you took along
with intermediate calculations you made to construct the cost-schedule containment chart,
Or, use MS Project (or similar tool) to calculate optimistic, expected, and pessimistic
project cost and schedule estimates.
Modifications to your mid-term schedule and task loadings are allowed. (12 pts)
2. Provide a comprehensive trade study for your class project that makes use of
program performance measures (technical, cost, and schedule) for decision
criteria. Explain how risk considerations were included in your trade study.
(12 pts)
3. Provide a risk management process flowchart suitable for presentation that has
been tailored for the projected life cycle of your class project. (12 pts)
4. Provide briefing charts (in addition to #3) suitable for orientation and training of
project personnel on the procedures in which risk management would be
conducted on your project. (14 pts)
Mid-term




Closed book, closed notes
Turn Part I of your project in with your
exam paper
You have 90 minutes for exam.
Any questions?