Should-Cost: A Use for
Parametric Estimates
Additional uses for estimation tools
Presenters:
Bob Ferguson (SEMA)
Date:
November 3, 2011
Location:
COCOMO Forum 2011
© 2011 Carnegie Mellon University
“Should-Cost” Background
Ashton Carter Memo, September 14, 2010:
Driving for affordability by enrolling contractors in mid-stream process
improvement.
© 2011 Carnegie Mellon University
2
Methods Used to Analyze Contractor Processes
Activity based costing (Consulting firm)
• Audit or use contractor’s data to allocate costs to activity
• Roll-up activity by Role (programmer, analyst, …), Process (design, …) and
Technology (language, database, platform …)
• Compare to benchmarks
Parametric Estimation (what I did)
• Use data from contractor’s basis of estimate (size, productivity, lifecycle)
• Construct parametric estimate that (nearly) duplicates contractor’s estimate.
• Test sensitivity of parameters and estimate benefits of improvement
Process Assessment (not used)
• Based on detailed interviews and artifact analylsis
© 2011 Carnegie Mellon University
3
Methods and Data for Parametric Review
Methods
• Park, R., A Manager's Checklist for Validating Software Cost and Schedule
Estimates, SEI-95-SR-004
• SEER-SEM, COCOMO parametric estimation tools used
Park’s checklist
• Checklist used to see whether I had sufficient information to reproduce the
estimate. Answers the question: Can I trust the estimate?
• Process, estimating experience, documented assumptions, …
Data used (COCOMO and SEER are similar):
•
•
•
•
Software size (LOC) by subsystem with new, modified, reused and base
Product complexity and unprecedentness factors (embedded vs. simulation…)
People and team capabilities and experience
Benchmarked performance (CMMI, or other)
© 2011 Carnegie Mellon University
4
Results of Analyzing 11 Estimates
1 – Estimated schedule and cost corresponded to very high productivity
and quality so there was little room for cost improvement (~5%).
2 – Very good in productivity (<10%)
4 – Estimate was padded by assuming people would not be as
productive compared to previous products. (~15%)
4 – Very difficult to determine a basis of estimate because insufficient
detail about size was presented.
Comparison to industry for productivity and quality
© 2011 Carnegie Mellon University
5
Why Both SEER & COCOMO
SEER
• Component level breakdown could be performed
• Sensitivity analysis reports
• One of the contractors was using SEER so BOE was visible
COCOMO
• One contractor did not provide component level detail. It was difficult to
understand their usage of re-use vs. modified vs. base code. Simplified
assumptions were easier to apply to COCOMO.
• SEER takes longer to edit for iteration of multiple estimates.
• My own inexperience with SEER.
© 2011 Carnegie Mellon University
6
Should-Cost Results
Improved value
• Up to 47% improvement on projected component cost – overall 22%!
• At least modest cost improvement (~5%) identified on every component
evaluated (all 11 estimates).
Some improvement suggestions (first 2 related to parameters)
Most contractors claimed low cohesion on teams. People pulled to fight fires.
Some contractors wanted to claim “inexperience” in domain and claim “best
in class” salaries in the same breath.
• Systems integration needs better synchronization. Added overhead
appeared directly in the BOE documentation. Delays waste valuable
resources and result in higher costs..
• Project management overhead was too high for every contractor.
• Program office needs both better processes and more technical
personnel. Product quality problems existed in prior efforts.
© 2011 Carnegie Mellon University
7
Conclusions
Parametric estimates are a useful method for should-cost evaluations.
• It was important to validate the BOE first as otherwise contractor can
challenge the conclusions.
Parametric method should be supplemented with bottom-up evaluation.
• Non-software services and products must be included.
• Product and Process Quality must be checked (e.g. estimation of defects)
• Interfaces between contractor and program office are a critical source of data.
Benchmark data sources are required.
• Estimation tools
• Capers Jones’ books
•…
© 2011 Carnegie Mellon University
8
Contact Information
Presenters/ Points of Contact
Dave Zubrow
SEMA
Telephone: +1 412-268-5243
Email: dz@sei.cmu.edu
Bob Ferguson
SEMA
Telephone: +1 412-268-9750
Email: rwf@sei.cmu.edu
Web:
www.sei.cmu.edu
www.sei.cmu.edu/measurement/
U.S. mail:
Software Engineering Institute
Customer Relations
4500 Fifth Avenue
Pittsburgh, PA 15213-2612
USA
Customer Relations
Email: info@sei.cmu.edu
Telephone:
+1 412-268-5800
SEI Phone:
+1 412-268-5800
SEI Fax:
+1 412-268-6257
© 2011 Carnegie Mellon University
9
NO WARRANTY
THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE
MATERIAL IS FURNISHED ON AN “AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY
MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO
ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR
PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM
USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY
WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT,
TRADEMARK, OR COPYRIGHT INFRINGEMENT.
Use of any trademarks in this presentation is not intended in any way to infringe on the
rights of the trademark holder.
This Presentation may be reproduced in its entirety, without modification, and freely
distributed in written or electronic form without requesting formal permission. Permission
is required for any other use. Requests for permission should be directed to the Software
Engineering Institute at permission@sei.cmu.edu.
This work was created in the performance of Federal Government Contract Number
FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software
Engineering Institute, a federally funded research and development center. The
Government of the United States has a royalty-free government-purpose license to use,
duplicate, or disclose the work, in whole or in part and in any manner, and to have or
permit others to do so, for government purposes pursuant to the copyright license under
the clause at 252.227-7013.
© 2011 Carnegie Mellon University
10