Headquarters U.S. Air Force
Integrity - Service - Excellence
Next-Generation
Systems and Software
Cost Estimation
Wilson Rosa
Technical Advisor
Air Force Cost Analysis Agency
(AFCAA)
October 28, 2008
Problem Statement
Emerging technologies such as
•
•
•
•
•
•
Systems of Systems (SoS)
Model Driven Architecture (MDA)
Enterprise Resource Planning (ERP)
Service-Oriented Architecture (SOA)
Commercial Off the Shelf (COTS) Services
Reuse-Driven (RUSE)
are complicating AFCAA's job of producing
accurate software cost estimates
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
2
Next-Generation System
Challenges
1. Lines of Code not appropriate for


Model Driven Architecture
COTS-Based Systems (SOA, ERP, etc.)
2. No guidelines for estimating beyond software design





Infrastructure (servers, LAN, routers, etc.)
Concurrent Users
Enterprise Services (collaboration, discovery, portal, etc.)
Data Migration, External Interfaces
Interoperability and Interdependency
3. Unfamiliar with total system cost drivers
4. Lack of Empirical Research – SOA, ERP, SoS, MDA
3
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Data Challenges
AFCAA has multiple software datasets
Unable to combine software datasets because
of inconsistencies and lack of standardization
 Schedule seems to be reported at program and not CSCI







level -- all CSCI’s have same schedule
No reporting of % re-design, % re-coding, % re-test
No common counting method – logical, physical, etc.
No standard application type definitions
No common code counting tool
Product size only reported in lines of code
No reporting of COCOMO, SEER, PRICE parameters
No reporting of quality measures – defects, MTBF, etc.
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
4
Parametric Model Challenges
Most AF program offices rely on cost models
Have these models been calibrated with the
most recent DoD data? (2002-2008)
Most models only cover a fraction, not total
system costs – infrastructure, users, etc.
Model calibration with recent data will help
reduce the program office estimating error rate
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
5
Consequence:
Significant Cost Growth (%)
Statistics
Total System*
Software Only**
Minimum
-64%
-80%
Mean
45%
37%
Median
27%
8%
High
471%
623%
Standard Deviation
71%
107%
Milestone Phase
Development
Development
Sample Size
137
111
Year of Data
1993-2003
2002-2008
Source : *John McCrillis, 36th DOD Cost Analysis Symposium (2003)
**Defense Automated Cost Information System
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
6
Headquarters U.S. Air Force
Integrity - Service - Excellence
Software Cost
Metrics Manual
OVERVIEW
7
Scope
Cost Agencies in conjunction with University
of Southern California will publish a manual to
help analysts develop quick software estimates
using reliable metrics from recent programs
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
8
Software Cost Manual Content
Chapter 1: Basic Software Estimation
Chapter 2: Product Size Estimation
Chapter 3: Schedule and Cost Growth
Chapter 4: Default ESLOC Parameters
Chapter 5: Historical Productivity
Chapter 6: Default COCOMO/SEER/True-S Parameters
Chapter 7: SLIM-ESTIMATE Calibration
Chapter 8: Risk and Uncertainty Parameters
Chapter 9: Data Normalization
Chapter 10: Space Software Cost Guide
Chapter 11: Software Maintenance
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
9
Data Sources
Commodity
Source
Format
Year
Projects
CSCIs
Space, Ground, Air
Software Resource Data Reports
DD-Form 2630
2002-2008
32
103
Space
AEHF
SEER
2008
6
53
Air
F-22 EMD and Increment II
Boeing
2004
13
112
Space
MILSTAR
SEER
1990s
4
28
Space
FAB-T
DD-Form 2630
2008
7
21
Space
NPOESS
SEER
2008
3
67
Space
TSAT
DD-Form 2630
2007
3
3
Air, Ground
Northrop Grumman
COCOMO, SEER
1997-2008
15
32
Space, Ground
Raytheon
COCOMO
1997-2008
33
49
Air, Ship, Ground
Naval Center for Cost Analysis
TECHNOMICS
1992-2001
21
68
Air
Lockheed Martin
COCOMO
1996-2004
2
2
Air
Army Cost and Economics Analysis Center
TECHNOMICS
2001-2004
16
16
Ground
Future Combat System
DD-Form 2630
2003-2008
13
42
Space, Intel
Intelligence Community (NRO, NGA, DNI)
SEER
TBD
TBD
TBD
Space, Ground
Aerospace, Space & Missile System Center
Unknown
Unknown
TBD
TBD
Space
NASA JPL
Unknown
Unknown
TBD
TBD
Space, Air, Ground
USC Affiliates
Unknown
Unknown
TBD
TBD
>168
>598
Note: Expecting over 1600 CSCIs by 2010
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
10
Data Normalization
 USC will interview program offices and
developers to obtain additional information…
1.
2.
3.
4.
5.
Cost Model Parameters – True S, SEER, COCOMO
Reuse Type – auto generated, re-hosted, translated, modified
Reuse Source – in-house, third party
Degree-of-Modification – %DM, %CM, %IM
Method – Model Driven Architecture, Object-Oriented, Traditional
 Available Data
1. DoDAF – System Views, Operational Views, etc.
2. Software Resource Data Report – Software Size, Effort, Schedule
3. Cost Analysis Requirements Description (CARD)

System Description, Users, Infrastructure, locations, interfaces, etc.
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
11
Chapter 4: ESLOC Parameters
Default values from recent programs
Based on Reuse Type and Reuse Source
Design
Modified
Code
Modified
Integration
Modified
ESLOC
Reuse Type
Reuse Source
Auto Generated
In-House
0%
0%
50%
15%
Third Party
0%
0%
100%
30%
In-House
0%
0%
100%
30%
Third Party
0%
24%
100%
37%
In-House
0%
100%
100%
60%
Third Party
15%
100%
100%
66%
In-House
0%
100%
100%
60%
100%
100%
100%
100%
In-House
0%
0%
32%
10%
Third Party
0%
0%
100%
30%
Re-Host
Translated
Modified
Third Party
Unmodified
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
12
Chapter 5: Historical Productivity
Overview and Guidelines
 Historical Productivity Dataset by Application
 Default Productivity Ranges by Application
IOC
Productivity
(ESLOC/MM)
Raw
(KSLOC)
ESLOC
Effort
(MM)
Peak
Effort
(FTE)
Schedule
(Months)
CSCI
Application
1999
Signal
Processing
Avionics
60
90000
90000
1000
69
71
2008
Spot Antenna
Control
Payload
39
78000
5000
2000
95
69
2008
Bootstrap
Bus
44
35000
31000
800
70
60
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
13
Significance of
Proposed Manual
Collected data can be used for
 Systems of Systems cost research
 Cost Model improvement initiatives
 SEER, True-S, COCOMO, etc.
Understanding relationships between NextGeneration Processes and COCOMO cost drivers
can encourage researchers to explore new
strategies to improve available cost models…
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
14
Way Ahead
Short Term
 NCCA/AFCAA Software Cost Handbook (Nov 2008)
 Software Cost Metrics Manual (2009)
Long Term





Cost Risk and Uncertainty Handbook Volume II (2011)
ERP Cost Guide (2010)
Impact of MDA on Cost Modeling (2010)
SOA Cost Study (2012)
Data Center Cost Study (2010)
Note: Any data you provide will not be attributed to your company or program,
but will be combined with like data from other sources and generic zed"
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
15
Backup Slides
Integrity - Service - Excellence
16
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e