Developing An Early Project Cost Estimate:
An Applied Case Study From Project Concept
through Contract Award
Linda Esker
Kathleen Dangle
Fraunhofer Center Maryland
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Topics Covered
• Project Overview
• The Early Project Cost
Estimate
• Estimate Phases
– Challenges
– Strategy
– Approach: Methodology and
Models Used
– Decisions to go forward
• Summary / Lessons Learned
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
2
Project Overview
• A large Ground Space
Communication System
• COTS-intensive
– For both HW and SW
• 5-year project duration
• Many areas highly scientific
– Need appropriate technical expertise as well as those who
understand what is involved in the cost estimate, especially SW
– Would require a blended estimation team
• Development of the estimate began in the earliest initial
concept phase
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
3
Cost Estimate Began Before Typical 1st Milestone
Life-Cycle
Phases
RFP
Reviews
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
4
The Early Project Cost Estimate
•
Cost estimate developed by the project to estimate
government and contractor project costs for
budget/funding approval
•
Activity played a significant role in the development of the
project
–
•
Value of the activity included not just the resulting estimate, but also
drove understanding, evolution, and “selling” of the project concept
State of the practice for early project estimates:
–
Typically use general, high-level, “personal” heuristics/ rules of
thumb by subject matter experts
–
Not able to effectively use estimation tools because details required
as inputs are not known early in the project
–
Process followed not as disciplined or structured (or repeatable) as
expected in estimates during later phases of project lifecycle
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
5
The Early Project Cost Estimate (cont.)
• The good news: With government emphasis on
better project estimates, projects now need a basis
of estimate and confidence levels
• The bad news: Solid estimates face challenges that
make use of estimation tools an uphill battle
• Politics — Organizational budgets have a life of their
own, are highly competitive, and defy
comprehension for those not in the political know
• Personal opinion and aggressive (or naive) optimism,
“This should be easy…”
• Distrust of what they do not understand
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
6
Early Project Estimate Phases
“In the Beginning…”
“Let there be light…”
“And it was good…”
“And it was better…”
• Forming the concept
• Maturing the project’s
architecture and
requirements
• Defining a project
lifecycle & WBS
• Defining the estimation
process and models
• Iterating on the
architecture,
schedule, models, and
estimate
• Evaluating options
and “what ifs”
• Formalizing Estimate
Confidence
Lessons
Learned
Lessons
Learned
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Lessons
Learned
7
Early Project Estimate Phase 1
• Started with:
–
–
–
–
Top-down approach
Fully understanding of the scope
Understanding SW and HW requirements of the project
Examined use of a commercial estimation tool
• Found it was difficult to use at this very early stage
– Needed more details than what was available
– Input was constantly changing as project investigated technologies
and needs and tool data cumbersome to update
• Management did not trust estimation tool results
– Wanted greater insight into and control on how estimate was
determined
– Wanted costs broken down into their areas of interest
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
8
Early Project Estimate Phase 1
Approach
Phase Start: Blank page; immature
requirements; forming team; gathering
historical data; top-down approach
Strategy:
Formulate
concept
•
•
•
•
•
•
Challenge:
Create initial
estimate with
minimum
information
Concept studies performed to develop
notional architecture
Estimate used:
•
•
•
General parametric models
Expert judgment
COCOMO
Spreadsheets
HW: Developed MEL (Master Equip. List)
SW: Used analogies/LOC
Percentages used to estimate many areas:
•
•
•
•
Management
Contingency, reserve & inflation
Spread of labor, HW, SW by fiscal year
Other technical unknowns
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Decision:
Next use
bottoms-up
approach for
greater accuracy
Lessons Learned
 Initial informal “off-thecuff” estimates can
derail a project from the
start
 Do not count on Mgmt.
acceptance of tool
estimate results
9
Early Project Estimate Phase 2
• Enhanced the Estimate with some
structure: a notional architecture,
schedule, and bottoms up-analysis
• Stabilized requirements to refine the
HW and SW estimate
• COCOMO schedule and effort
distributions allowed us to distribute
costs over the timeframe to allocate
budget to fiscal years
• COSYSMO could now be used to
validate engineering SME estimate
o
Needed to use requirement expansion factors
• Project still going through definition
and needed to adjust for changes
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
10
Early Project Estimate Phase 2
Decisions:
Approach
Strategy:
Use bottomsup approach;
define models;
HW/SW by arch
Phase Start: Immature requirements; notional
architecture beginning to evolve
• Migrated approach from parametric overlay to
bottoms up
• Established physical notional architecture to
organize costs
Challenge:
Pursue 2
independent paths
(dev., deploy.); focus
on things missing
–
Approach helped drive engineer thought process
–
Estimate used:
– Expert judgment
– Analogous estimation - some historical
data
– COCOMO for SW effort & duration
–
% still used to estimate
– Management (based historical data)
– Contingency, reserve, inflation, &
unknowns
– Spread of labor, HW, SW by fiscal year
Establish
sound basis for
good estimate
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Lessons Learned
 Hybrid of bottoms-up
& parametric
modeling works best
 Everyone understood
what was known
(justifiable) and
unknown
11
Iterative Development of the Estimate
Notional System Architecture
$70,000,000
Labor, Material, Cost
Phasing
WBS
(By Notional Architecture)
$60,000,000
$50,000,000
$40,000,000
$30,000,000
$20,000,000
$10,000,000
$0
Estimation
Model
Project Lifecycle Schedule
RESOURCES
Hardware
– Antenna
o Antenna part 1
o Antenna part 2
– Computers
o Server
o Workstation
•
•
•
Software
– Software item 1
– Software item 2
•
•
•
Master Resource
Sheet of
•
•
•
•
•
Engineering tasks
Integration Tasks
Transition tasks
Items,
Costs
SLOC COCOMO
Model
Effort,
Duration
Effort
Hardware
Software
Engineering Effort
Integration Effort
Transition Effort
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
12
Early Project Estimate Phase 3
• Areas of Focus
– Identifying a good system model on which to base confidence in
the estimate—having a solid Basis of Estimate (BOE)
– Ensuring Implementation schedule is realistic
– Understanding changing expectations and being flexible to deal
with change
– SW: ► Filling voids in analogous systems by using SW SMEs to provide sizing information
– HW:► Using a notional system to obtain sufficient information for an early project estimate
and trying to avoid over-engineering the perfect system
•
Key Accomplishments
– Met goals of project: Early Project Estimate for RFP;
presentation to HQ; understandable/supportable basis for
budget
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
13
Early Project Estimate Phase 3
Approach
Strategy:
Use 2 separate
Teams (dev. &
deploy.);
merge results
Phase Start: Matured requirements & team; near
complete notional architecture
Ready to go;
freeze estimate
for RFP
• Incorporated inputs from Trade
Studies and another independent cost
estimate
• Aligned cost structure, schedule, WBS,
and notional architecture
Challenges:
Decisions:
–
PM worked with all technical teams to understand basis
of estimate (BOE)
–
Loaded Resources by skill level & labor rates
–
Spread labor, HW, SW costs by fiscal year based on
schedule
–
Only PM remained %
–
Enhanced estimate model to allow:
Honing the estimate;
• Many different views (architecture elements, high cost
drivers)
• Investigation of options ("what if" a ground station were
eliminated?)
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Lessons Learned
 BOE review sessions
need a strong driver
 The better the
estimate fidelity the
more useful for “holes”
& “what ifs”
 Working as a team
creates buy-in and
project team is much
smarter
14
Early Project Estimate Phase 4
Approach
Phase Start: RFP Released; Project estimate
frozen; fewer distractions
Strategy:
Examine Estimate
Cost Risks
Challenge:
Understand
Confidence in
Estimate
•
•
•
•
•
Decisions:
Project has
confidence with
expected
range
Performed Risk Cost Analysis
Focused on high cost drivers
–
Optimistic
–
Most likely
–
Pessimistic estimate
Used triangular distribution and Monte
Carlo to simulate confidence ranges
Compared dispersions with expected
ranges
Detailed resource items enabled
analysis of procurement long lead
items & phasing
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Lessons Learned
 A solid early Project
cost estimate enables
developing confidence
levels and
understanding where
the estimate falls
within the levels
before contract start
15
COCOMO Lessons Learned
• At this phase, the
summary COCOMO
tables are very helpful.
However,
– They do not give
consistent information
– Percentages do not add
up to 100%
– Total effort has two
different values
• This makes them
appear to be
unreliable
– Had to make
adjustments
Module Name
Total Size
Total Effort
Overall
Plans And Requirements
Product Design
Programming
Integration and Test
TOTALS
Sample
37975
230.308313
Schedule (%)
22.53%
27.27%
42.93%
29.80%
122.53%
Schedule
(Months) Effort (%)
Effort
Staff
14.763379
7.00% 16.121582
1.091998
17.864799
17.00% 39.152413
2.191595
28.130345
54.20% 124.82886
4.437516
19.524292
28.80% 66.327036
3.397155
80.282815
107.00%
246.4299
11.118264
Plans and
Product
Integration
EFFORT
Requirements
Design
Programming and Test
Requirements Analysis
7.211762 4.894052
4.993155
1.658176
Product Design
2.842752 16.052489
9.986309
3.316352
Programming
0.929637 5.337729
70.528308 26.220951
Test Planning
0.666338 2.401298
7.031867
2.078163
Verification and Validation
1.230594 2.988584
10.776733
18.63815
Project Office
1.972248 3.810935
7.323452
4.554541
CM/QA
0.462173 0.926657
7.947597
5.217811
Manuals
0.806079 2.740669
6.241443
4.642893
TOTALS
16.121583 39.152413
124.828864 66.327037
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
TOTALS
18.757145
32.197902
103.016625
12.177666
33.634061
17.661176
14.554238
14.431084
246.429897
16
COCOMO Lessons Learned
• Modular approach of COCOMO
products is ideal
• Project had definite need for a
COCOTS-like tool, but
– Seemed too immature
– Decisions on COTS were not controlled by
government
• Overlap of COSYSMO and COCOMO II makes
management wary of using them together
– Need some better quantification of the overlap in the
estimate
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
17
Summary / Lessons Learned
Start early: Plan for iterations of the estimate
– The project estimate will iterate through
phases and different estimation techniques as
more information is learned
– The estimation process can aid in the
maturation of the project concept,
requirements, and schedule
Be Adaptable/Flexible: Standard tools cannot handle all of
management’s needs
– Early on there is not a clear definition/agreement on how to
proceed and implement the project; need to change and adapt
– Be sure your cost estimation tools are flexible
• Need to handle what is known about the project at early and also later
phases of the evolution
• COTS may not match all your needs—a hybrid approach works well
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
18
Summary/Lessons Learned
Be Mindful of Management: Need to work
with high-level management and keep them in
the loop
– Early “off-the-cuff” promises can derail a
thorough estimation process
• Don’t promise too much too soon—early
promises cannot be rescinded
– It is imperative that the higher management teams have faith in
the estimation work
• You need to thoroughly understand (and justify) how a tool’s model
handles all aspects of the estimate
– Even if not accepted for an end result, use tools to also give a
sanity check for any ad hoc estimation practices
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
19
Summary / Lessons Learned
WBS is Critical: Projects need a WBS that
can help them collect data and learn
– Lobby for better WBS to facilitate data
and information collection when
necessary
– Current WBS templates/guidelines make it difficult to
extract software from other parts of the system
development work
• Project was afraid to specify that WBS differentiate between
software and other activities
• Would love to see this addressed as a SW best practices
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
20
Questions
For more information contact us at:
Linda Esker
lesker@fc-md.umd.edu
Kathleen Dangle
kdangle@fc-md.umd.edu
© 2012 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
21