Affordability Analysis: The Role
Process, Cost and ROI Modeling In
Improved Program Performance
COCOMO @ SEI October 2012
Galorath Incorporated
Daniel D. Galorath: Founder & CEO
Copyright Galorath Incorporated 2012
Key Points
Viable
affordability
decisions yield
project
achievements
Positive ROI
when
decisions
use
affordability
modeling
Parametric
Models provide
affordability
decision support
2
© 2012 Copyright Galorath Incorporated
2
Cost Overruns Are Everywhere
•
“GAO: Staggering cost overruns dwarf modest
improvements in Defense acquisition”
• R&D costs of weapons programs increased 42% over
original estimates
• Average delay 22 months in delivering initial capabilities
• Evolving technical requirements
• Shortage of qualified government staff to manage
• "Every dollar of cost growth on a DoD weapon system
represents a lost opportunity to pay for another national
priority"
•
“The Collins Class submarine program: Murphy
was an optimist”
http://epress.anu.edu.au/apps/bookworm/view/Agenda,+Volume+19,+Number+1,+2012/9601/Ergas.htm
•
“BG Group shares hit by $5.4B cost overruns on
Australian Liquified Natural Gas project”
Software Is A Key Risk Item In
Weapons Systems
•
Navy Mobile User Objective Satellite Communication
System delays to the Joint Tactical Radio
System, a set of software-defined radios causes
advanced MUOS capabilities to be drastically
underused… GAO
•
GAO identified 42 programs at risk for cost & schedule
1. military requirements changes
2. software development challenges
3. workforce issues
•
National Institute of Standards and Technology (NIST)
• Software defects cost nearly $60 Billion Annually
• 80% of development costs involve identifying and
correcting defects
Software, not Hardware or technology readiness levels
were called out
Example: Project Cost Alone Is not
The Cost of IT Failure (Source: HBR)
•
Case Study: Levi Strauss
• $5M ERP deployment contracted
• Risks seemed small
• Difficulty interfacing with customer’s systems
• Had to shut down production
• Unable to fill orders for 3 weeks
• $192.5M charge against earnings on a $5M IT project
failure
“IT projects touch so many aspects of organization
they pose a new singular risk”
http://hbr.org/2011/09/why-your-it-project-may-be-riskier-than-you-think/ar/1
An ROI Analysis of A New System:
Should We Fund This?
•
• Can we do better?
Will stakeholders tolerate a
loss for 3 years?
• What is the risk?
Bad Estimates Are A Root Cause of
Project Failure
• An estimate is the most knowledgeable statement you
can make at a particular point in time regarding:
• Effort / Cost
• Schedule
• Staffing
• Risk
• Reliability
•
•
Estimates more precise with progress
A WELL FORMED ESTIMATE IS A
DISTRIBUTION
© 2012 Copyright Galorath Incorporated
7
Poor Project Performance Often Points
Back To Inadequate Estimating
•
When performance doesn’t meet the estimate, there
are two possible causes:
1. Poor performance
2. Poor estimates
•
“In the software world, we have ample evidence that
our estimates stink, but virtually no evidence that
people in general don’t work hard or intelligently
enough.”
- Tom DeMarco, Why Does Software Cost So Much?
Why should we care: Good people working with poor
estimates cause project challenges.
© 2012 Copyright Galorath Incorporated
8
Greatest Leverage Is In Tackling
Inherent and Structural Costs
© 2010 Copyright Galorath Incorporated
9
© 2010 Copyright Galorath Incorporated
10
US Better Buying Power Initiatives
•
•
•
June 28, 2010 Mandate
September 14, 2010 Guidance
November 3, 2010
Implementation
• Five Specific Areas of Concern:
• Target Affordability and Control
Cost Growth
• Reduce Non-Productive
Processes and Bureaucracy
• Incentivize Productivity and
Innovation in Industry
• Promote Real Competition
• Improve Tradecraft in Services
Acquisition
Affordability Initiatives With
“Should Cost” and “Will Cost”
Will Cost
Performance
-
Cost Initiatives
(Applied practices
& improvements)
=
Should Cost
Performance
Many View Bottoms up estimates as the requirement
for Should Cost / Will Cost Analysis
But parametrics can do analysis faster as well as
provide more tradeoffs
© 2012 Copyright Galorath Incorporated
12
System Description (Parametrics Can
Estimate More, Earlier)
Adapted from CEBOK
“If you can’t tell me what it is,
I can’t tell you what it costs.”
-Mike Jeffers
“If you can tell me the range of
what it might be, I can tell you the
range of cost, schedule &
probability.”
-Dan Galorath
Affordability Process
Step 1.
Procure
Key Performance
Parameters that are
inviolate
Step 2. Identify
Affordability Goals &
Figures of Merit
(development, life
cycle, payback, ROI,
NPV, kill ratio, Budget
constraints, etc.)
Step 3. Gather
Requirements,
Features, Performance
Step 4. Define
Baseline Alternatives
Step 5. Perform
Technical Design
Analysis for Each
Alternative
Step 6. Perform
Cost Schedule
Analysis of Each
Alternative
Step 7. Assess
Benefits Based
on Figures of
Merit
Step 8. Perform
Probabilistic Risk
Analysis
Step 9. Assess
Alternatives &
Select Optimal
Alternative
Step 10.
Document
Analysis and
Lessons Learned
Should Cost: Trade Study Flow
Requirements And Features
Analysis
Preliminary
Design
Process
Alternative 1
Alternative 2
Alternative 3
Alternative 4
Other Factors
•Human Factors
•Security
•Reliability
•Availability
•Survivability
•Supportability
•Testability
•Producibility
•Reuse
•Transportability
Selection
Process
4
3
2
SEER
Assessment
•Performance
•Schedule
•Risk Assessment
•Life Cycle Cost
Iterate
5
1
No
OK?
Optimized?
Yes
Cost
Performance
Schedule
Risk
Bottoms Up Estimation
as Required
© 2012 Copyright Galorath Incorporated
15
Top-Down versus Bottom-Up Estimating
(Adapted From: Project Management: the
Managerial Process, McGraw Hill 2011)
Condition
Macro Level /
Parametric
Strategic decision
making
X
Micro level /
Bottoms up
Internal, Small Project x
Fixed price contract
x
Customer wants detail
x
Unstable scope
•
x
Bottoms up can be more accurate IF
• All work elements are captured and
• Inter WBS work is not forgotten
•
Parametrics can be more accurate since it won’t forget
items
•
Use parametrics AT LEAST for analysis of alternatives
© 2010 Copyright Galorath Incorporated
16
Complex Parametric Modeling:
Repeatable Based on Program
Parametric Estimating
•
Mathematical relationships
between anticipated
characteristics and effort,
schedule, risk, etc. &
Traditional Estimating
•
•
Guessing
•
Simple Analogy
•
Spreadsheets
•
Bottoms up
• Development
•
Quantity x price
• Production
•
Etc.
Technology, programmatics,
cost, schedule, To estimate
• Effort / cost / Total Cost
of Ownership
• Schedule
• Manufacture
• Modification
• Deployment of end items
• Much more
•
Many techniques from:
Part of a repeatable process
•
Generally manual in
nature…
•
Most are not repeatable
Firm Fixed
Price?
Feel lucky?
What is likely
to happen
Understand the risk before you commit!
18
Dealing With the “Problem of
Assumptions”
•
•
Assumptions are essential but…
•
Use an assumption verification process
Incorrect assumptions can drive an estimate to
uselessness
2. Rank
order
assumptions
based on
estimate
1. Identify
assumptions impact
© 2012 Copyright Galorath Incorporated
3. Identify
high ranking
assumptions
that are risky
4. Clarify high
ranking, high
risk
assumptions
& quantify
what happens
if those
assumptions
change
5. Adjust
range of
SEER inputs
to describe
the
uncertainty
in
assumptions
19
Key Points
Viable
affordability
decisions yield
project
achievements
Positive ROI
when
decisions
use
affordability
modeling
Parametric
Models provide
affordability
decision support
20
© 2012 Copyright Galorath Incorporated
20