Using Cost Models to Capture Project Risk:
A Knowledge-Based Approach
Dr. Ricardo Valerdi
Massachusetts Institute of Technology
October 22
22, 2009
[rvalerdi@mit.edu]
Share
A Goal: Excellence
We Share AWe
Goal:
Enterprise
Enterprise
Excellence
4
http://lean.mit.edu
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 4
Risk Assessment Lessons Learned
in the U.S. Department of Defense
1. Systems engineering can be the blessing or the
curse
•
Resource estimation methods are being developed
2 T
2.
Technology
h l
maturity
t it and
d requirements
i
t stability
t bilit
are controllable risks
• Cost models help understand this relationship
3. People risks are often underestimated
•
Experience and capability are not interchangeable
4. By the time the risk is identified, it’s too late!
• Need leading indicators (not lagging indicators)
http://lean.mit.edu
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 5
Cost Commitment on Projects
C om m itm ent to Technology,
C onfiguration, Perform ance, C ost, etc.
%
100
C ost Incurred
75
System -Specific K now ledge
50
25
Ease of C hange
N
E
E
D
C onceptualPrelim inary
D esign
D etail D esign
and
D evelopm ent
C onstruction
and/or
Production
System U se, Phaseout,
and D isposal
Blanchard, B., Fabrycky, W., Systems Engineering & Analysis, Prentice Hall, 1998.
6
http://lean.mit.edu
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 6
COSYSMO
Operational
Concept
Constructive
Systems Engineering
Cost Model
# Requirements
q
# Interfaces
# Scenarios
# Algorithms
+
3 Adj. Factors
Size
Drivers
Effort
Multipliers
- Application factors
-8 factors
- Team factors
-6 ffactors
t
http://lean.mit.edu
COSYSMO
Effort
Calibration
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 7
Systems Engineering Processes
EIA/ANSI 632, Processes for Engineering a System (1999).
•
•
•
Acquisition and Supply
• Supp
Supply
y Process
ocess
• Acquisition Process
Technical Management
• Planning Process
• Assessment Process
• Control Process
System Design
• Requirements
q
Definition Process
• Solution Definition Process
http://lean.mit.edu
•
•
Product Realization
• Implementation Process
• Transition to Use Process
Technical Evaluation
• Systems
y
Analysis
y Process
• Requirements Validation Process
• System Verification Process
• End Products Validation Process
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 8
COSYSMO
COSYSMOData
DataSources
Sources
Boeing
Integrated Defense Systems (Seal Beach, CA)
Raytheon
Intelligence & Information Systems (Garland, TX)
Northrop Grumman
Mission Systems (Redondo Beach, CA)
Lockheed Martin
Transportation & Security Solutions (Rockville, MD)
Integrated Systems & Solutions (Valley Forge, PA)
Systems Integration (Owego, NY)
Aeronautics (Marietta, GA)
Maritime Systems & Sensors (Manassas, VA;
Baltimore, MD; Syracuse, NY)
General Dynamics
Maritime Digital Systems/AIS (Pittsfield, MA)
Surveillance & Reconnaissance Systems/AIS
(Bloomington, MN)
BAE Systems
National Security Solutions/ISS (San Diego, CA)
Information & Electronic Warfare Systems (Nashua,
NH)
SAIC
Army Transformation (Orlando, FL)
Integrated Data Solutions & Analysis (McLean
(McLean, VA)
L-3 Communications
http://lean.mit.edu
Greenville, TX
9
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 9
Policy & Contracts
Commercial Implementations
COSYSMO Model
E

 14
PM NS  A    ( we ,k  e ,k  wn ,k  n ,k  wd ,k  d ,k )    EM j
 k
 j 1
10 Academic
Theses
Academic Curricula
Proprietary Implementations
•
•
•
•
http://lean.mit.edu
SEEMaP
COSYSMO-R
SECOST
Systems Eng
Eng.
Cost Tool
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 10
Traditional Cost and Schedule
Risk Estimation
• Risk
Taxonomy
• Organizatio
nal Lessons
Learned
Assess/
Estimate
Possible
Risk Impacts
• Ranges of
Estimate Cost
& Schedule
Risks
• Ranges of
Cost &
Schedule
Values &
Probabilities
Values of
Size,
Productivity,
& Other
Parameters
100%
95%
90%
85%
80%
75%
70%
65%
60%
55%
50%
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
0
25000
50000
Person Months
Risk Assessment
http://lean.mit.edu
Select
Cost and
Schedule
Person Months Confidence (Cumulative Probability)
Cumulative Probability of Person
Months
Determine
Sources of
Risk
75000
100000
Management
Decision
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 11
Expert COSYSMO Operation
Integrated Estimation and Risk Analysis
Size
Drivers
User Input
Cost
Drivers
Cost
Estimating
Relationship
Rule-Based
Risk Heuristics
Cost Estimate with
Uncertainty
y Ranges
g
Risk Assessment
- Identification
- Analysis
- Prioritization
P i iti ti
Risk Control
- Planning
- Monitoring
Madachy, R. & Valerdi, R., Knowledge-Based Risk Assessment for Systems
Engineering: Expert COSYSMO, working paper, 2009.
12
http://lean.mit.edu
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 12
ARCH
LSVC
MIGR
TRSK
DOCU
INST
RECU
TEAM
PCAP
PEXP
PROC
SITE
TOOL
SIZE (REQ + INTF + ALG + OPSC)
Requirements Understanding
Architecture Understanding
Level of Service Requirements (the ilities)
Migration Complexity (legacy system considerations)
Technology Risk (maturity of technology)
Documentation match to life cycle needs
Number and Diversity of Installations or Platforms
Number of Recursive Levels in the Design
Stakeholder Team Cohesion
Personnel/team capability
Personnel Experience and Continuity
Process Capability
Multisite Coordination
Tool Support
RQMT
SIZE
Initial Risk Conditions
21
21
17
9
9
9
12
7
10
5
5
8
12
7
8
4
3
3
4
1
2
7
5
7
5
10
8
2
10
9
11
3
1
6
3
4
8
5
6
6
4
4
4
3
4
9
10
11
4
7
9
4
5
8
7
11
8
11
4
7
5
2
6
7
9
12
7
5
5
2
3
3
6
4
7
3
9
10
6
4
6
3
5
3
2
8
2
8
8
8
5
7
1
4
2
4
5
3
5
5
3
5
3
8
8
high risk
small x = 0.5; big X = 1
medium risk
n = 19
l
low
risk
i k
Valerdi, R. & Gaffney, J., Reducing Risk and Uncertainty in COSYSMO Size and Cost Drivers:
Some Techniques for Enhancing Accuracy, 5th Conference on Systems Engineering Research,
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 13
http://lean.mit.edu
Hoboken, NJ, March 2007.
Risk Network
Risk Categories
Product
Risk Items
Mitigation Guidance Items
ARCH RECU
ARCH_RECU
Prototype
PRR =  RE
REProduct =  RE
ARCH_PCAP
People
REPeople=  RE
Hire
PRR =  RE
Rescope
PRR =  RE
ARCH_MIGR
Platform
REPlatform =  RE
RECU PCAP
RECU_PCAP
RE = Risk Exposure
PRR = Potential Risk Reduction
Madachy, R. & Valerdi, R., Knowledge-Based Risk Assessment for Systems
© 2009 Massachusetts Institute of Technology
Expert COSYSMO, working paper, 2009.
http://lean.mit.edu
Engineering:
Valerdi 2009 - 14
Expert COSYSMO Inputs
http://csse.usc.edu/tools/ExpertCOSYSMO.php
15
http://lean.mit.edu
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 15
Expert COSYSMO Outputs
16
http://lean.mit.edu
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 16
Outputs - Risk Mitigation Advice
17
http://lean.mit.edu
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 17
Risk Exposure Trends
as Leading Indicators
•
Risk burndown tracked as mitigation actions are executed
and other changes occur
18
http://lean.mit.edu
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 18
Publicly Available Resources
•
•
•
U.S. General Accountability Office (http://gao.gov/)
• Investigative
I
ti ti arm off the
th U.S.
US C
Congress
RAND Corporation (http://rand.org/)
• Public think tank
• “Managing Risk in USAF Force Planning”
Defense Acquisition University (https://acc.dau.mil)
• One
O off severall U.S.
U S Milit
Military Universities
U i
iti
• “DoD Risk Management Guidebook “
http://lean.mit.edu
© 2009 Massachusetts Institute of Technology Valerdi 2009 - 19