Project Risk Analysis –
An Overview
INTRODUCTION
There is often scrutiny in industry regarding Monte Carlo simulation on projects.
The key objections are that the process is too subjective and the real world is much
more complicated than you can effectively model; concepts like accuracy and
complex correlations between cost areas are too difficult to quantify. The reality is
that this is correct. However, accuracy is not the intent here. It’s important to
understand that in estimating there are no guarantees. The only thing we can
strive for is reduction of uncertainty – to simply get a better understanding of what
we need to account for on our projects.
Values in 10^ -8
Values in 10^ -8
A common misconception in the business world is that risks are too complex to
quantify – whether it is the risks themselves, or their correlation to each other.
Because of this, decisions are often made to do no quantification at all because of a
perceived lack of value or confidence in the
Distribution for Total Project Cost/Q88
Distribution for Total Project Cost/Q88
results it will produce. The truth is that
4.000
4.000
Mean=1.570811E+08
even some level of uncertainty reduction is
3.500
Mean=1.570811E+08
3.500
valuable to your project.
3.000
3.000
2.500
2.500
2.000
2.000
1.500
1.500
1.000
1.000
0.500
0.500
0.000
110
0.000
110
In the scientific world measurement is
defined as “a set of observations that
reduce uncertainty where the result is
expressed as a quantity” (ref. p21, “How to
Measure Anything – Finding the Value of
135
160
185
210
135
160
185
210
Intangibles in Business”, Douglas W.
Values in Millions
Values in
Millions
15%
70%
15%
Hubbard).
By that definition exactness
15% 145.2439 70%
15%
169.2682
145.2439
169.2682
should
never
be expected.
Fully
understanding this concept yields a significant change in one’s perspective on
modeling risk. The result is openness to quantification at some level of that which
would be impossible to exactly or accurately model. Even a very simplistic
approach to quantifying your project uncertainty will bring you to a much better
understanding of your level of project definition than if you do none at all.
AN OVERVIEW OF MONTE CARLO SIMULATION ON PROJECTS
For all projects, no matter the size, location, industry, etc. there are inherently
some unknowns. Even though risk and variability are both a subset of uncertainty
we categorize the risk on projects as variability or uncertainty. Variability refers to
a random nature of a process, i.e. where the outcomes are random even though
the process and its parameters are well understood. Examples might be the
variability of labour rates, estimated quantities, estimate material and equipment
costs, productivity etc. Uncertainty, in this case, refers to the lack of knowledge
about the value of a parameter, or the factors that determine its behaviour.
overview of risk analysis-rev1.doc
- ENGCOMP Confidential -
Page 1 of 7
Phone (306) 978.7730 · Fax (306) 978.7729 · 3318 Millar Ave · Saskatoon · SK Canada S7K 7G9 · info@engcomp.ca
Project Risk Analysis –
An Overview
Examples might be the number of days where weather prevents construction from
occurring, the possibility of a strike, delays in obtaining environmental approval, ice
road availability, labour shortages, currency fluctuations, safety, business risk, Acts
of God, etc. Uncertainty usually applies to factors on projects that are not required
to execute the project, but if they occur could adversely affect the cost or schedule
or sometimes both.
A contingency analysis typically only takes in
to account variability in the cost estimate. A
Project Risk Analysis (PRA) takes in to
account variability and uncertainty.
The
contingency analysis uses a model based on
the capital cost estimate and the PRA uses a
model based on the project schedule. @Risk
is used to perform the Monte Carlo Simulation
in both cases and works within Microsoft
Excel and Microsoft Project. Listed below is a
typical outline of each analysis procedure.
1.000
1.000
Distribution for Total Project Cost/Q88
Distribution for Total Project Cost/Q88
0.800
0.800
Mean=1.570811E+08
Mean=1.570811E+08
0.600
0.600
0.400
0.400
0.200
0.200
0.000
110
0.000
110
135
160
135
160
Values in Millions
Values
in
Millions
15%
70%
15% 145.2439
145.2439
70%
169.2682
185
185
210
210
15%
15%
169.2682
COST ESTIMATE CONTINGENCY ANALYSIS
Contingency is an amount of money that
you add to a budget to account for all of
the things you haven’t thought of yet. The
Decontaminate D213, D264, .../Q58
.63
Decontaminate
D213,
D264, .../Q58
.63
Decontaminate
Site
D141/Q22
.431
less work you’ve done at that time to define
Decontaminate
Office
Renos/Q65 Site D141/Q22
.353 .431
Renos/Q65
.353
the project, the more uncertain you are
WBOffice
Enclosure
Dir Cost/Q36
.339
WB Enclosure
Dir Civ,
Cost/Q36
Electroplating
Arch,
.../Q48
.207 .339
about the total cost, and therefore the
Electroplating
Civ, .../Q48
Electroplating
IndArch,
Cost/Q52
.154.207
Cost/Q52
.154
WBElectroplating
Floor Slab DirInd
Cost/Q31
.134
more contingency you require. Uncertainty
WBEsc
Floor
Slab Dir Cost/Q31
.134
2008
Factor/Q80
.131
2008 Mov
Esc es/Q68
Factor/Q80
.131
Interim
.129
in a cost estimate is a concept that is
Mov es/Q68
.129
WBInterim
Enclosure
Ind Cost/Q37
.109
WB
Enclosure
Ind
Cost/Q37
.109
readily accepted in the industry no matter
Consulting Costs/Q74
.098
Consulting
.098
CANTASS
EnclCosts/Q74
Dir Cost/Q41
.087
what stage of project definition you’re at.
CANTASS
Encl
Dir
Cost/Q41
.087
Electroplating Mech/Q49
.077
Electroplating
Mech/Q49
.077
2009
Esc Factor/Q81
.076
The question, however, is how much
2009
Esc Factor/Q81
.076
DCC
Support/Q73
.069
DCC
Support/Q73
.069
2010
Esc
Factor/Q82
.065
uncertainty do you have? Rules of thumb
2010 Esc Factor/Q82
.065
-1
-0.75 -0.5 -0.25
0
0.25
0.5
0.75
1
and good practice give cost estimators a
-1
-0.75 -0.5 -0.25
0
0.25
0.5
0.75
1
Std b Coefficients
good idea what percentage of contingency
Std b Coefficients
should be applied to a cost estimate for a
given level of project definition, but the problem is that it’s just a rule of thumb and
it relies on a subjective estimate of the overall level of project definition. If projects
are small then it’s easier to make this subjective decision, but as they get larger it
is much more difficult to do so effectively. This is where Monte Carlo simulation
becomes the preferred solution for quantifying the uncertainty in order to make a
good assessment of how much contingency should be allocated to your project.
Regression Sensitivity for Total Project
Regression Sensitivity
Cost/Q88 for Total Project
Cost/Q88
overview of risk analysis-rev1.doc
- ENGCOMP Confidential -
Page 2 of 7
Phone (306) 978.7730 · Fax (306) 978.7729 · 3318 Millar Ave · Saskatoon · SK Canada S7K 7G9 · info@engcomp.ca
Project Risk Analysis –
An Overview
Although it is more crucial that this process be applied to larger projects, we feel
that it is beneficial at any level. It gives you a better understanding of the level of
project definition at a given stage and provides tools for more effectively managing
your project. Contingency is a very important aspect of any budget, and needs to
be accounted for properly in order to ensure the success of your projects.
Contingency Analysis Description
Traditionally, contingency is set as a factor of the total estimate based on
experience and the level of project definition. We are still using this concept to
establish a value, but we back it up with a statistical analysis of the estimate that
generates a confidence level for the total estimate. This has proven to be a very
useful tool for our Clients when it comes to decision making and approvals for
projects.
The Contingency Analysis we are discussing is a quantitative approach that seeks to
statistically determine the possible outcomes of a project total cost. In general, the
techniques in a Contingency Analysis encompass four steps:
Developing a Model — by defining your problem or situation in a spreadsheet
format
Identifying Uncertainty — in the variables in your spreadsheet; specifying their
possible values with probability distributions, and identifying the uncertain
worksheet results you want analyzed
Analyzing the Model with Simulation — to determine the range and probabilities
of all possible outcomes for the results of your worksheet
Making a Decision — based on the results provided and individual preferences
The goal of this method is to help the decision-maker choose a course of action,
given a better understanding of the possible outcomes that could occur.
Results
Contingency @ 55% Level of Confidence =
$29,449,000
22.9%
Contingency @ 65% Level of Confidence =
$32,521,000
25.3%
Contingency @ 75% Level of Confidence =
$36,251,000
28.2%
Contingency @ 85% Level of Confidence =
$40,746,000
31.7%
Contingency @ 95% Level of Confidence =
$48,162,000
37.5%
overview of risk analysis-rev1.doc
- ENGCOMP Confidential -
Page 3 of 7
Phone (306) 978.7730 · Fax (306) 978.7729 · 3318 Millar Ave · Saskatoon · SK Canada S7K 7G9 · info@engcomp.ca
Project Risk Analysis –
An Overview
Process Overview
The following is an outline of the tasks and procedures that ENGCOMP facilitates in
order to perform the contingency analysis.
Review Estimates: ENGCOMP will gather all the cost information, review it
with the client and organize it all into one spreadsheet.
Estimate Roll-Up: This is a collaborative process by which the cost estimate is
summarized into 20 to 30 lines in preparation for the analysis input model.
The result of this meeting is the Variability Brainstorming Worksheet.
Planning Team Calibration: This is a process in which the planning team
members are trained to be good estimators of risk. Variability Brainstorming:
This is a facilitated meeting to brainstorm the upper and lower bounds of the
cost summary lines on the variability brainstorming worksheet. The output
of this meeting is the input data required for the analysis model.
Contingency Analysis Modelling: Once all the data is collected it is compiled
into a spreadsheet model, validated and the simulation is executed.
Delivery of Results: The results are compiled into comprehensive output
reports. Discussion and dissemination of the results are included in a report
and the final recommendation for contingency is given. Reports can be
customized to suit the specific needs of the client.
Approximate Contingency Breakdown For Total Project Cost
PMO & Consultant Total
1.1%
Demo D141 Total
1.3%
Interim Moves Total
-3.5%
Demo D191B Total
0.2%
Office Renos Total
18.8%
Decont D141 Total
14.5%
West Bay Floor Total
5.9%
Cafeteria Total
2.1%
West Bay Enclosure Total
14.6%
Decont D213, D264, D265 Total
23.3%
Electroplating Total
16.5%
CANTASS Enclosure Total
4.0%
Demo D213, D264, D265 Total
1.3%
overview of risk analysis-rev1.doc
- ENGCOMP Confidential -
Page 4 of 7
Phone (306) 978.7730 · Fax (306) 978.7729 · 3318 Millar Ave · Saskatoon · SK Canada S7K 7G9 · info@engcomp.ca
Project Risk Analysis –
An Overview
A PROJECT RISK ANALYSIS
Project Risk Analysis (PRA) takes the contingency analysis to the next level,
quantifying the effects of all reasonable risks and uncertainty on your project.
This process takes into account the variability of costs, schedule durations, outside
project risks and their collective impact on your project. The results of this Monte
Carlo simulation help to define the schedule contingency required to complete
the project for a given confidence level as well as the risk reserve budget
required.
One might question – why do you need to quantify contingency separate from risk
reserve on projects? The answer lies in the control authority of these two pots of
money. The project manager should be given full control of the contingency to
manage and allocate those funds as necessary amongst the project cost centers or
work packages. Risk reserve, however, usually requires authorization at a more
senior level based on appropriate justification for its use. However one does not
have to go as high as a senior funding authority to use this money, which would
typically add significant delay to the project schedule.
The objective of ENGCOMP’s process is to provide a framework and methodology to
derive a reliable project cost estimate as well as to identify and manage risks
associated with the project.
100%
100%
90%
90%
85%
85%
80%
80%
Expected Project
Expected
Cost
BeforeProject
Cost Before
Contingency
Contingency
70%
70%
60%
60%
Total Project Cost
Total
Project Cost
with
Contingency
with Contingency
50%
50%
40%
40%
Contingency = $40,746,000
Contingency = $40,746,000
Risk Reserve = $22,615,000
Risk Reserve = $22,615,000
30%
30%
20%
20%
Total Cost with
Total Cost with
Contingency
&
Contingency
&
Project
Risk
Project Risk
Reserve
Reserve
10%
10%
0%
0%
Overlay of Cumulative Probability Curves
overview of risk analysis-rev1.doc
- ENGCOMP Confidential -
Page 5 of 7
Phone (306) 978.7730 · Fax (306) 978.7729 · 3318 Millar Ave · Saskatoon · SK Canada S7K 7G9 · info@engcomp.ca
Project Risk Analysis –
An Overview
Definition of Risk Reserve
Risk Reserve is an amount of money that the project carries to cover reasonable
project risks that may affect the project cost. They key here is that this portion of
the budget does not account for any items that are required to be built. In fact if
all things go well you won’t even need to spend this portion of the budget. But for
projects that need to have some level of comfort that they won’t get shut down due
to budget overruns related to reasonable risks, then this fund needs to be defined.
Process Overview
The following is a high level list of the tasks that ENGCOMP facilitates in order to
perform a project risk analysis.
Review Project Scope and Schedule:
This involves a detailed study of the
project schedule and scope of work document. A review of the project with
the project team is also required in order to get an effective understanding of
the project needs and issues.
Planning Team Calibration:
This is a process during which
the planning team members
are trained to be good
estimators of risk.
Schedule Roll-Up: This is a
collaborative process by which
the schedule is simplified by
grouping the tasks that have
a common variability in their
durations. These groupings
will also have commonality
with respect to overall project
areas,
disciplines
and/or
major accounts. The result is
a
Schedule
Variability
Brainstorming Worksheet.
Statistic
Summary Statistics
Value
%tile
Value
Minimum
9 Jan 2013
5%
25 Apr 2013
Maximum
30 Sep 2014
10%
27 May 2013
Mean
30 Sep 2013
15%
14 Jun 2013
Std Dev
101.6
20%
03 Jul 2013
Variance
10323
25%
19 Jul 2013
Skewness
0.284
30%
31 Jul 2013
Kurtosis
2.767
35%
14 Aug 2013
Median
24 Sep 2013
40%
28 Aug 2013
Mode
19 Jul 2013
45%
10 Sep 2013
Left X
25 Apr 2013
50%
24 Sep 2013
Left P
5%
55%
08 Oct 2013
Right X
26 Mar 2014
60%
22 Oct 2013
Right P
95%
65%
06 Nov 2013
Diff X
335
70%
21 Nov 2013
Diff P
90%
75%
06 Dec 2013
0
80%
25 Dec 2013
#Errors
Filter Min
85%
16 Jan 2014
Filter Max
90%
14 Feb 2014
95%
26 Mar 2014
#Filtered
0
Schedule Variability Brainstorming: The goal of the brainstorming session is to
scrutinize the duration variability for each line of the model as a group and
come up with values that everyone (or most everyone) buys into.
Risk Quantification: This is another brainstorming meeting that is facilitated
by ENGCOMP. Before quantitative analysis or modelling can occur, it is
necessary to develop a risk register. The risk register contains a description
overview of risk analysis-rev1.doc
- ENGCOMP Confidential -
Page 6 of 7
Phone (306) 978.7730 · Fax (306) 978.7729 · 3318 Millar Ave · Saskatoon · SK Canada S7K 7G9 · info@engcomp.ca
Project Risk Analysis –
An Overview
of the risks that may be encountered and quantification of the probability of
occurrence and impact on the project. The risk register is a direct output of
this meeting.
Project Risk Analysis: Once all the data is collected it is compiled into an MS
Project based model, validated and the Monte Carlo simulation is executed.
Delivery of Results: The results are compiled into comprehensive output
reports. Discussion and dissemination of the results are included in formal
report and the final recommendation for project risk reserve is given.
Example Risk Brainstorming Record
Line
No.
WP
No.
1
Data
Type
Task Description
Phase IV Total Project Schedule
Start
Date
Estimate
Jan 2008
Finish
Date
Duration
(Work Days)
Apr 2013
1360
1562
Analysis
Jan 2008
Jan 2014
2
Demolish 191B (Under Current EA) - Planned
9g
Completion Sep 2008
Estimate
Apr 2008
Aug 2008
80
Analysis
Oct 2008
Mar 2009
101
3
TB Approval EPA(R) Phase IV – Planned
Milestone Jan 2009
Estimate
Jan 2009
Feb 2009
Analysis
Apr 2009
May 2009
Estimate
Apr 2009
May 2009
Analysis
Sep 2009
Oct 2009
5
BOD West Bay – Planned Completion Jun
10d
2011
Estimate
Apr 2009
Jun 2011
558
Analysis
Jul 2009
Nov 2011
621
6
BOD Electro-Plate Shop – Planned
11a
Completion Jul 2011
Estimate
Feb 2009
Dec 2010
480
7
BOD Cafeteria & Administration Fit-Up –
13
Planned Completion Oct 2012
8
9 & Demolitions & Decontamination – Planned
11 Completion Apr 2013
4
a
Construction Contract Award (WP10/11/13) –
Planned Milestone Apr 2009
Analysis
Jul 2009
Jul 2011
528
Estimate
Mar 2012
Aug 2012
120
Analysis
Aug 2012
Jan 2013
121
Estimate
Jul 2009
Apr 2013
980
Analysis
Jul 2011
Mar 2014
700
Schedule
Contingency
Y
Q
Gantt Chart
2008
2009
2010
2011
2012
2013
2014
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
M
277
217
155
212
156
351
Example Schedule Milestones Variability Results
overview of risk analysis-rev1.doc
- ENGCOMP Confidential -
Page 7 of 7
Phone (306) 978.7730 · Fax (306) 978.7729 · 3318 Millar Ave · Saskatoon · SK Canada S7K 7G9 · info@engcomp.ca