Headquarters U.S. Air Force
Integrity - Service - Excellence
Next-Generation
Systems and Software
Cost Estimation
Wilson Rosa
Technical Advisor
Air Force Cost Analysis Agency
(AFCAA)
October 28, 2008
Problem Statement
Emerging technologies such as
•
•
•
•
•
•
Systems of Systems (SoS)
Model Driven Architecture (MDA)
Enterprise Resource Planning (ERP)
Service-Oriented Architecture (SOA)
Commercial Off the Shelf (COTS)
Design for Reuse (RUSE)
are complicating AFCAA's job of producing
accurate software cost estimates
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
2
Next-Generation Systems
Challenges
1. Lines of Code not appropriate for


Model Driven Architecture
COTS-Based Systems (SOA, ERP, etc.)
2. No guidelines for estimating beyond software design:





Infrastructure (servers, LAN, routers, etc.)
Concurrent Users
Enterprise Services (collaboration, discovery, etc.)
Data Migration, External Interfaces
Interoperability and Interdependency
3. Unfamiliar with total system size and cost drivers
4. Lack of Empirical Research – SOA, ERP, SoS, MDA
3
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Data Challenges
AFCAA has multiple software datasets
Unable to combine software datasets because
of inconsistencies and lack of standardization
 Schedule seems to be reported at program and not CSCI







level -- all CSCI’s have same schedule
No reporting of % re-design, % re-coding, % re-test
No common counting method – logical, physical, etc.
No standard application type definitions
No common code counting tool
Product size only reported in lines of code
No reporting of COCOMO, SEER, PRICE parameters
No reporting of quality measures – defects, MTBF, etc.
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
4
Parametric Model Challenges
Most DoD Program Offices rely on software
parametric models which have not been
calibrated with recent DoD data
Electronic Systems Center (Hanscom AFB)
SEER-SEM
Aeronautical Systems Center (WPAFB)
True-S
Software Technology Center (Hill AFB)
Sage
Space and Missile Systems Center
SEER-SEM
Parametric Models only cover software design
not total system – infrastructure, users, etc.
Calibration will help reduce the program office
estimating error rate
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
5
Consequence:
Significant Cost Growth (%)
Statistics
*Total System
**Software Only
Minimum
-64%
-80%
Mean
45%
37%
Median
27%
8%
High
471%
623%
Standard Deviation
71%
107%
Milestone Phase
Development
Development
Sample Size
137
111
Year of Data
1993-2003
2002-2008
Source : *John McCrillis, 36th DOD Cost Analysis Symposium (2003)
**Defense Automated Cost Information System
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
6
Headquarters U.S. Air Force
Integrity - Service - Excellence
Software Cost
Metrics Manual
OVERVIEW
7
Scope
Cost Agencies in conjunction with University
of Southern California will publish a manual to
help analysts develop quick software estimates
using reliable metrics from recent programs
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
8
Data Sources
Commodity
Source
Format
Year
Projects
CSCIs
Space, Ground, Air
Software Resource Data Reports
DD-Form 2630
2002-2008
32
103
Space
AEHF
SEER
2008
6
53
Air
F-22 EMD and Increment II
Boeing
2004
13
112
Space
MILSTAR
SEER
1990s
4
28
Space
FAB-T
DD-Form 2630
2008
7
21
Space
NPOESS
SEER
2008
3
67
Space
TSAT
DD-Form 2630
2007
3
3
Air, Ground
Northrop Grumman
COCOMO, SEER
1997-2008
15
32
Space, Ground
Raytheon
COCOMO
1997-2008
33
49
Air, Ship, Ground
Naval Center for Cost Analysis
TECHNOMICS
1992-2001
21
68
Air
Lockheed Martin
COCOMO
1996-2004
2
2
Air
Army Cost and Economics Analysis Center
TECHNOMICS
2001-2004
16
16
Ground
Future Combat System
DD-Form 2630
2003-2008
13
42
Space
NRO
SEER
TBD
TBD
TBD
Space, Ground
Aerospace
Unknown
Unknown
TBD
TBD
Space, Ground
SMC
Unknown
Unknown
TBD
TBD
Space
NASA JPL
Unknown
Unknown
TBD
TBD
>168
>598
Note: Expecting over 1600 CSCIs by 2010
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
9
Data Normalization
 USC will interview program offices and
developers to obtain additional information…
1.
2.
3.
4.
5.
COCOMO II Parameters
Reuse Type – auto generated, re-hosted, translated, modified
Reuse Source – in-house, third party
Degree-of-Modification – %DM, %CM, %IM
Method – Model Driven Architecture, Object-Oriented, Traditional
 Available Data
1. DoDAF – System Views, Operational Views, etc.
2. Software Resource Data Report – Software Size, Effort, Schedule
3. Cost Analysis Requirements Description (CARD)

System Description, Users, Infrastructure, locations, interfaces, etc.
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
10
Software Cost Manual Content
Chapter 1: Basic Software Cost Estimation
Chapter 2: Product Size Metrics
Chapter 3: Historical Growth
Chapter 4: Default Effective Size (ESLOC) Parameters
Chapter 5: Historical Productivity Dataset
Chapter 6: Default COCOMO Parameters
Chapter 7: SLIM-ESTIMATE Calibration
Chapter 8: Risk and Uncertainty Parameters
Chapter 9: Data Cleansing
Chapter 10: Space Software Cost Estimation
Chapter 11: Software Maintenance
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
11
Chapter 4: ESLOC Parameters
Default values from recent programs
Based on Reuse Type and Reuse Source
Design
Modified
Code
Modified
Integration
Modified
ESLOC
Reuse Type
Reuse Source
Auto Generated
In-House
0%
0%
50%
15%
Third Party
0%
0%
100%
30%
In-House
0%
0%
100%
30%
Third Party
0%
24%
100%
37%
In-House
0%
100%
100%
60%
Third Party
15%
100%
100%
66%
In-House
0%
100%
100%
60%
100%
100%
100%
100%
In-House
0%
0%
32%
10%
Third Party
0%
0%
100%
30%
Re-Host
Translated
Modified
Third Party
Unmodified
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
12
Chapter 5: Historical Productivity
Overview and Guidelines
 Historical Productivity Dataset by Application
 Default Productivity Ranges by Application
IOC
Productivity
(ESLOC/MM)
Raw
(KSLOC)
ESLOC
Effort
(MM)
Peak
Effort
(FTE)
Schedule
(Months)
CSCI
Application
1999
Signal
Processing
Avionics
60
90000
90000
1000
69
71
2008
Spot Antenna
Control
Payload
39
78000
5000
2000
95
69
2008
Bootstrap
Bus
44
35000
31000
800
70
60
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
13
Significance of
Software Cost Metrics Manual
Collected data can be used for
 Systems of Systems cost research
 COCOMO improvement initiatives
 Understanding relationships between NextGeneration Processes and COCOMO cost drivers
can encourage researchers to explore new strategies
to improve available cost models…
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
14
Way Ahead
Short Term (2009-2010)
 Send Software Data Call to program offices,
developers, and USC Affiliates
 Write Chapters 4 & 5 (2009)
 Publish Software Cost Metrics Manual (2010)
Long Term (2010-2012)
 ERP Cost Guide (2010)
 Impact of MDA on Software Productivity (2010)
 SOA Cost Study (2012)
Note: Any data you provide will not be attributed to your company or program,
but will be combined with like data from other sources and generic zed"
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e
15
Backup Slides
Integrity - Service - Excellence
16
UNCLASSIFIED I n t e g r i t y - S e r v i c e - E x c e l l e n c e