Software Maintenance, Sustaining Engineering, and Operational Support
Software Maintenance, Sustaining Engineering,
and Operational Support
Estimating Software Maintenance Costs for U
U.S.
S Army Weapons
Systems
Office of the Deputy Assistant Secretary of the Army for Cost and Economics (ODASA-CE)
PSM User’ Group Meetings and Workshops
29 July 2012
1 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
Obj ti
Objective
Provide the Department of the Army with the
ability to accurately estimate, budget, allocate,
and justify the software maintenance, sustaining
engineering, and operational support resources
required to meet evolving mission and service
affordability requirements across the program
life-cycle
2 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
P
Presentation
t ti Outline
O tli
• Current software maintenance estimation technology
does not support objective resource decisions in the
emerging Army systems sustainment environment
• Defining a viable software maintenance cost estimation
methodology for Army programs - project
requirements, approach, and initial results
• Integrated software maintenance life cycle cost
estimation model - linking software maintenance
resource requirements to program and functional
domain sustainment profiles
3 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
Estimating Software Maintenance Costs
in the Current Army Sustainment
Environment
4 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
A
Army
S
Software
ft
M
Maintenance
i t
Environment
E i
t
• Software is the “default component” for increasing system
capability and performance
• Operational requirements are dynamic and complex
• Maintenance budgets are becoming more constrained and
vulnerable
l
bl
• Difficult to defend program software life cycle cost estimates
and annual maintenance budget requests
• Limited understanding of the relationships between software
investments, applied resources, product outputs, and mission
capability
p
y
• Multiple perspectives: enterprise - program - maintenance
organization - user base
• Existing software maintenance estimation methodologies are
inadequate
5 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
S ft
Software
System
S t
Growth
G
th
107 - AH-64As
1620 - AH-64Ds
Apache Software Growth
300 KSLOC to Over 1.4 Million SLOC
6 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
S ft
Software
Configuration
C fi
ti Complexity
C
l it
•
Multiple system variants
drive multiple parallel
software release baselines
•
Different “types” of
concurrent software releases
-
•
4,300 - M1A1 & variants
580 - M1A2 & variants
580 - M1A2 SEP & variants
7 - 27 June 2012
Correct defects
Fault prevention - mandates
Enhance functionality
Adapt to new environments
Multiple change drivers
-
End user requirements
Mission evolution
System interoperability
Change mandates
Technology
Technical debt
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
Significant
Si
ifi
t Growth
G
th in
i Army
A
Software
S ft
Maintenance Resource Requirements ($)
• Reliance on software changes to
meet evolving mission
requirements
• Proliferation of system software
variants
• Increasingly complex system to
system functional interfaces
• Proliferation of change drivers
• Functional change
g backlogs
g
• Budget realignments
“We’ve lived in a rich man’s world where there has
been less emphasis on cost over the past 10 years.”
years ”
Dr. Jacques Gansler
8 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
Currentt S
C
Software
ft
M
Maintenance
i t
Estimation Methods
• Parametric models
- Only include corrective, perfective, adaptive changes &
enhancements
- Invalid assumptions about sustaining engineering tasks
- Historical data not visible in all models
- Not calibrated - validated
• Past software maintenance funding - effort
-
Limited access to historical FTE - Budget data
Represent “rolled-up” costs
Requirements or LOE funding?
Availability of correlated program development sustainment cost data
9 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
C
Current
tE
Estimation
ti ti Methods
M th d (continued)
(
ti
d)
• Number of lines of code per software engineer
- Each engineer can maintain 20K-25K LOC/ESLOC
- Does not reflect the impact of software reuse or COTS
• Software maintenance estimated as a p
percentage
g of
development costs
- Rule(s) of thumb - development based:
• S/W maintenance costs - 2/3 of total S/W life cycle costs
• S/W maintenance costs - 60% to 75% of total S/W life cycle costs
• Annual S/W maintenance costs - 5% to 10% of total S/W life cycle costs
- Ignores total system life cycle software growth and maintenance
requirements/strategy/tasks
q
gy
All of these methods have significant limitations
in the current and future Army environment
10 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
K E
Key
Estimation
ti ti Issues
I
• Current software maintenance estimation methods:
- Do not align with an individual program’s unique life cycle
sustainment profile
- Assume software maintenance resource requirements are
consistent after system deployment
- Do not take into account specific software maintenance
products and related activities for a given program (what’s
in and what
what’s
s out)
- Do not encompass multiple types of cost relationships
- Do not address the differences across functional software
domains
- Are high level constructs based on significant assumptions
of commonality
11 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
Notional
N
ti
l Software
S ft
Maintenance
M i t
Life-Cycle Cost Model
PDSS
PPSS
Development
Production
Maintenance
Cost
EMD
Design
Obsolescence
Technical
Debt
DEV
A
B
12 - 27 June 2012
Minimal Maintenance
C
Software Maintenance Estimation Period
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
Summary
• Army system mission capability is dependent upon the
adequate resourcing of program software maintenance
requirements
• To ensure this the Army needs a software maintenance
estimation methodology that better reflects the
software change profile and related employment
characteristics of the system
13 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
Software Maintenance Estimation Project
Requirements, Approach, and
Initial Results
14 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
E ti ti Methodology
Estimation
M th d l
Requirements
R
i
t
• Define a software maintenance cost estimation
methodology that integrates the product release and
software support costs over the life-cycle of a program
- Configurable to the specific program software maintenance
improvement and release strategy
- Encompasses PDSS and PPSS efforts
- Applicable at Milestones A, B, and C
- Integrates
I
component cost estimates
i
from
f
multiple
l i l CER based
b
d
product and activity estimates (WBS defined)
- Directly related to development parameters and performance
- Can incorporate cost risk uncertainty at component and higher
levels
- Practical - defensible - data derived - adaptable
15 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
Work Breakdown
Structure
Software Policy
OPS-29
Funding Model
Data
Architecture
Terminology
Software Source
Data
• information requirements
• program context data
• planned/actual cost &
technical data
• risk
risk-uncertainty
uncertainty data
Structured
Cost Data
• data set characterization
- availability/content
- integrity
- usability
• data demographics
• data resource access
16 - 27 June 2012
program
environment
Analysis
Data & Reporting
Requirements
• data requirements
• measures - data sources
Cost Risk
Uncertainty Model
• cost uncertainty
factors
• risk relationships
Analysis
• best practices
• requirements
Cost Estimating
Relationships
• parametric
relationships
• cost ratios
• product-activity
specific
• domain specific
Approved for public release - Distribution is unlimited
I t
Integrated
t d Cost
C t
Estimation Model
• estimation process model
• product-activity-context
driven estimation
methodologies
• risk based cost uncertainties
• predictive cost performance
• life cycle phase relationships
Software Maintenance, Sustaining Engineering, and Operational Support
P j t Approach
Project
A
h
• Establish software maintenance technical standards and
b
baselines
li
-
Software maintenance products and activities
Configurable and adaptable WBS
Army software maintenance and sustaining engineering process model
Common terminology
• Collect, evaluate, and structure relevant Army software
maintenance data
-
Data collection



-
Data evaluation - availability, integrity, usability
Data schema and accessible data store


-
Army program cost data - all relevant parameters
Army program, organizational, and enterprise context data
Navy,
y Air Force, and contractor data
Historical cost data
Derived CERs
E ti ti
Estimation
data
d t requirements
i
t
17 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
P j t Approach
Project
A
h (continued)
(
ti
d)
• Independent analysis and model development
-
Data and information model
Product sizing models
Cost estimation relationships
Cost risk - uncertainty model
Life cycle integrated cost estimation model
• Define Army software maintenance data collection
and policy change requirements
-
Information driven program and organization data requirements
Post development policy requirements
Required information infrastructure changes
• Implement and improve
-
Data stores - estimation assets
Estimation and risk-uncertainty model
Estimation processes
Emerging stakeholder information requirements
18 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
W k Breakdown
Work
B kd
St
Structure
t
•
•
•
•
•
•
•
•
•
Common structure that includes all potential products and
activities - “what’s
what s in - what
what’s
s out”
out
Common definitions - terminology
Emphasis on DOD weapons systems
B i for
Basis
f identifying
id tif i specific
ifi costt elements
l
t attributable
tt ib t bl to
t a given
i
program or system maintenance/sustaining engineering effort
Product based - driven by changes to the software baseline(s)
Release focused - primary software maintenance product
Intended to be tailored and adapted for each program or
organization:
- Addition/deletion of lower level cost elements
- Re-binning of lower level cost elements
Foundation for cost estimation process/models
Basis for defining cost estimating relationships
19 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
Work Breakdown Structure
Software Maintenance, Sustaining Engineering,
and Operational Support
1.0 Software
Maintenance
Release N
Planning - Management
Software Requirements
Architecture & Design
Change Implementation
Integration & Test
Acceptance Test
Rework
Emergency Repairs
Hardware Updates
Release N+1
Release N+2
Release N+3
2.0 Software
Licenses
Deployed Systems
Facility Systems
3.0 Information
Assurance
5.0 Sustaining
Engineering
Analysis and Studies
Test Support
Software Delivery
User Training
User Support
Field Support
Development Facilities
Integration and Test
Facilities
Tactical Equipment
Test Equipment and
T l
Tools
IAVA
4.0 Certification
& Accreditations
Mission, Safety, Performance
DIACAP
Version 2.82
20 - 27 June 2012
6.0 Facilities &
Infrastructure
Approved for public release - Distribution is unlimited
7.0 Management
Program Management
Contract Management
Change Management
Data Management
Quality Assurance
Process Management
Personnel Management
Software Maintenance, Sustaining Engineering, and Operational Support
W k Breakdown
Work
B
kd
St
Structure
t
1.0 Software Maintenance - products and activities associated with modifying an
operational software product or system
2.0 Software Licenses - products and activities associated with the procurement and
renewal of software licenses for operational software
3.0 Information Assurance - products and activities associated with ensuring that the
software is compliant with externally defined information assurance requirements
4.0 Certifications and Accreditations - products and activities associated with
verifying a software system against externally defined domain performance
criteria
5.0 Sustaining Engineering - products and activities associated with supporting a
deployed software product or system in its operational environment
6.0 Facilities & Infrastructure - products and activities associated with establishing
and operating the facilities and processes required to modify, integrate, and test
operational software products or systems
7.0 Management - products and activities associated with planning, organizing,
funding and controlling the resources required to support operational software
funding,
products or systems
21 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
P
Program
WBS Element
El
t Coverage
C
WBS
1.0
1.1
1.1.1
1.1.2
1.1.3
1.1.4
1.1.4.1
1.1.4.2
1.1.5
1.1.5.1
1.1.5.2
1.1.5.3
1.1.5.4
1.1.6
1161
1.1.6.1
1.1.6.2
1.1.6.3
1.1.6.4
1.1.7
1.1.8
1.1.9
1.2
1.3
1.4
2.0
3.0
31
3.1
3.2
4.0
Title
# 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Software Maintenance
Software Release (N)
Planning and Management
Software Requirements
Architecture and Design
Change Implementation
Change Development
Unit Testing
Integration and Test
Test Planning
Test Scenario and Test Case
Development
Test Tool Development
Test Conduct
Acceptance Testing
Test Planning
Test Scenario and Test Case
Development
Test Tool Development
Test Conduct
Rework
Emergency Repairs
Hardware Updates
Software Release (N+1)
Software Release (N+2)
Software Release (N+3)
Licenses
Information Assurance
IAVA
DIACAP
Certifications & Accreditations
22 - 27 June 2012
17
7
19
24
24
23
7
7
24
8
8
8
8
29
7
7
7
7
7
25
15
0
0
0
11
27
12
11
12
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
P
Program
WBS Element
El
t Coverage
C
(cont.)
(
t)
5.0
5.1
5.2
53
5.3
5.4
5.5
5.6
6.0
6.1
6.1.1
6.1.2
6.1.3
6.2
6.2.1
6.2.2
623
6.2.3
6.4
6.5
7.0
7.1
7.1.1
7.1.2
7.1.3
7.2
7.3
7.4
7.5
7.5.1
7.6
7.7
Software Sustaining Engineering
Analysis and Studies
Test Support
Delivery
User Training
User Support
Field Support
Software Facilities and
Infrastructure
Development
p
Facilities
Equipment
Licenses
Operations
Integration and Test Facilities
Equipment
Licenses
O
Operations
ti
Tactical Equipment
Test Equipment and Tools
Management
Program Management
Project Release Management
Risk Management
Measurement/ Analysis
Contract Management
Change Management
Data Management
Quality Assurance
IV&V
Process Management
Personnel Management
23 - 27 June 2012
0
18
16
15
15
16
26
16
11
7
11
7
27
7
14
11
11
27
11
29
12
11
26
17
27
24
29
26
12
0
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
A
Army
S
Software
ft
M
Maintenance
i t
Data
D t
Org 1 Program 1
Program 2
Program 3
Program 4
Program 5
Program 6
Program 7
Program 8
Program 9
Program 10
g
Program 11
Program 12
Program 13
Org 2 Program 1
Program 2
Program 3
Program 4
Program 5
Program 5
Program 6
Program 7
Program 8
Program 9
Org 3 Program 1
Org 4 Program 1
Program 2
Program 3
Program 4
Org 5 Program 1
Program 2
Program 3
Org 6 Program 1
Program 2
Program 3
Program 4
Program 5
Number of WBS Task Interfaces
Releases Performed
3
Effort
Cost ($)
Schedule Software Change Requirements
Requests (SCRs)
Defects
1
1
4
1
3
1
1
1
4
4
1
1
2
2
5
3
1
4
2
2
1
1
6
2
3
7
Multiple
Multiple
Multiple
3
6
3
8
1
88
24 - 27 June 2012
Approved for public release - Distribution is unlimited
Backlog
Source Lines OPS-29 Data
of Code
Software Maintenance, Sustaining Engineering, and Operational Support
D t A
Data
Analysis
l i Summary
S
• Availability of data varied significantly by organization and
program
• Execution data (actuals) was very limited - most data was
planning data aligned with the OPS-29 funding model
• Multiple
M lti l funding
f di
streams
t
exist
i t for
f similar
i il work
k
• Data was cost based - not product/activity based
• Software engineering
g
g data derived from different source
artifacts for a given release were sometimes inconsistent
• The aggregation levels of provided data were inconsistent
(details vs
vs. summary)
• The data that was available provided significant insight
into potential cost estimating relationships and
maintenance products and activity allocations
25 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
O
Operating
ti Environments
E i
t - Domains
D
i
Count of Operating Environment
Helicopter
Mission Processing
Real Time Embedded ‐ Other
3
System
Training
Vehicle Control
Vehicle Payload
Grand Total
Ground
5
3
Missile
5
4
6
2
Count of Super‐Domain
Helicopter
Mission Critical ‐
E b dd d
Embedded
3
Mission Critical ‐
Non‐Embedded
Mission Support
Grand Total
3
26 - 27 June 2012
UAV
Grand Total
5
8
4
6
2
2
22
2
2
4
2
29
UAV
Ground
Missile
Grand Total
2
7
2
14
2
9
6
22
2
9
6
29
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
A li ti Domain
Application
D
i Types
T
Embedded
Non Embedded
• Sensor Control and
Signal Processing
• Vehicle
V hi l Control
C t l
• Vehicle Payload
• Real Time Embedded
• Mission Processing
• Systems Software
• Automation and Process
C t l
Control
• Simulation & Modeling
Mission Support
• Training
• Test
• Data Processing
27 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
C tE
Cost
Estimating
ti ti Relationships
R l ti
hi
•
Types of cost estimating relationships:
- Systemic ratios
- Parametric models
- Simple relationships
- Historical cost trends
•
Approach:
- Ratios useful to identify performance relationships
- Product costs structured into software releases - estimated
parametrically based on functional change content
- Activity costs estimated using simple CER relationships unique
u
que variables
a ab es and
a d drivers
d es
- Fixed costs based on historical resource expenditures
- Related factors influence and/or modify the outcomes in all
cases
28 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
C
Correlation
l ti Coefficients
C ffi i t Matrix
M ti
Correlation Coefficients Matrix
8 Critical value (10%)
1.94
Sample size
Interfaces
A
Age
Est_Total_KSLOC
Req_Fund_FY11
Exec_Fund_FY11
29 - 27 June 2012
Pearson Correlation
R Standard Error
t
p-value
H0 (10%)
P
Pearson
Correlation
C
l ti
R Standard Error
t
p-value
H0 (10%)
Pearson Correlation
R Standard Error
t
p-value
H0 (10%)
Pearson Correlation
R Standard Error
t
p-value
H0 (10%)
Pearson Correlation
R Standard Error
t
p value
p-value
H0 (10%)
Coefficie
Coefficie
C ffi i
Coefficie
Coefficie
Coefficie
IInterfaces
t f
1.00
A
Age
0 47
0.47
0.13
1.31
0.24
accepted
0.70
0 08
0.08
2.43
0.05
rejected
0.39
0.14
1 04
1.04
0.34
accepted
0.50
0.13
1.41
0 21
0.21
accepted
1 00
1.00
0.23
0 16
0.16
0.58
0.58
accepted
0.30
0.15
0 77
0.77
0.47
accepted
0.59
0.11
1.78
0 13
0.13
accepted
E t T t l KSLOC Req_Fund_FY11
Est_Total_KSLOC
R
F d FY11 Exec_Fund_FY11
E
F d FY11
1.00
0.87
0.04
4 41
4.41
0.00
rejected
0.81
0.06
3.40
0 01
0.01
rejected
Approved for public release - Distribution is unlimited
1.00
0.94
0.02
6.87
0 00
0.00
rejected
1.00
Software Maintenance, Sustaining Engineering, and Operational Support
F di vs. Software
Funding
S ft
Size
Si
30 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
P li i
Preliminary
Cost
C t Relationships
R l ti
hi
• Results
- Estimated total system KSLOC and number of interfaces
had the highest correlation to both funded and executed
funding
- Requested and executed funding varied by a factor of 3
- There is an implied prioritization of applied funding by
subsystem (domain - weapons system vs. training system)
- System age had no appreciable cost impact
- There is a non-linear relationship between software system
size and cost for software maintenance (resource
constraints)
- Measurable software development characteristics available
early in the life cycle may be early predictors of software
maintenance life cycle cost
31 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
WBS B
Based
dC
Costt E
Estimating
ti ti R
Relationships
l ti
hi
WBS
Title
1.1
Software Release
1.1.1 Planning, Req'ts, Design,
to
Implementation, Development,
1.1.5 Unit Testing, Integration & Test
(typical SW Development)
Army OPS-29 Mapping
Variability
Capability Sets FY(XX/XX)
System Mission Capability
Variable
Cost
116
1.1.6
A
Acceptance
T
Test
Capability
C
bili Sets
S
FY(XX/XX)
System Mission Capability
Variable
V
i bl
Cost
2.0
3.1
Licenses
IAVA
Licenses
IAVA
Semi-Var
Semi-Var
Cost
4.0
5.0
5.1 to
55
5.5
5.6
6.0
Certifications & Accreditations
SW Sustaining Engineering
Analysis, Test Support, Delivery,
Training
Field Support
Software Facilities &
Infrastructure
C&A
Fixed
Roll-up
Semi-Var
Cost
Semi-Var
Fixed
7.0
Management
Organic
7.1 to
7.7
Program / Release / Risk /
Contract Management
32 - 27 June 2012
Organic Labor
FSEs
System Infrastructure or
System Open Door
Roll-up
Fixed
Cost
Cost Drivers Examples
Lower level cost roll-up
Parametric modeling - Number of
Requirements (Enhancements),
Defects, Test Cases - added, reused,
modified; App. Domain, Complexity,
Reliability, Personnel Factors
N b off requirements
Number
i
/ capabilities
bili i
/ … / test cases / etc.)
Type and number of COTS products.
Parametric model (percentage
distribution); Application domain, size,
effort staffing, duration, productivity
Cost per C&A
Lower level cost roll-up
LOE
LOE, No. of Field Sites
# People at facility, Simulation / Test
Equip Maintenance
Lower level cost roll-up
LOE
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
T
Types
off Cost
C t Estimating
E ti ti Relationships
R l ti
hi
Variable
Cost
Semi Variable
Fixed
Time
33 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
S ft
Software
Maintenance
M i t
Cost
C t Estimation
E ti ti Model
M d l
• Objectives
- Use available data to construct a multi
multi-CER
CER cost estimate
for a sample program - compare to program actuals
- Identify program cost allocations
- Validate integrity - usability of provided data
• Scope
-
5 Year PPSS time p
period - 4 capability
p
y sets (2006-2011)
(
)
All included software maintenance WBS elements
Applied CERs - Proof of Concept
Parametric model - capability sets / releases (SLIM)
Semi-variable - IAVAs, licenses, certification, etc.
Fixed costs - infrastructure and management
34 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
S ft
Software
Maintenance
M i t
Cost
C t Estimation
E ti ti Model
M d l
• Results for one sample program
- S
Software
ft
capability
bilit sets
t are mostt significant
i ifi
t costt item
it
based on program allocations and expenditures (55%)
- Licenses, IAVAs, and Certifications & Accreditations were
relatively stable year-to-year (small adjustments for
inflation) - due in part to limited COTS on this program
- Facilities and Management
g
were also relatively
y stable yeary
to-year
- Requested funding is relatively consistent across the
timeline
- Data was noisy - inconsistent parameters from different
program data sources
- Model cost prediction was consistent with executed costs
35 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
I iti l Model
Initial
M d l
Core Measures
Fiscal Yearly Avg Staff (L2)
Monthly Gantt Chart (L4)
< Program X Example >
< Program X Example >
50
40
20
peoplle
30
10
TOTAL SUPPORT
1.0 SOFTWARE MAINTENANCE
CAPABILITY SETS
SB 21
CS 9/10
CS 11-12
CS 13 -14
2.0 SOFTWARE LICENSES
0
1
FY07
2
FY08
3
FY09
4
FY10
5
FY11
6
FY12
7
FY13
4 10 16 22 28 34 40 46 52 58 64 70 76 82
Oct Apr Oct Apr Oct Apr Oct Apr Oct Apr Oct Apr Oct Apr Oct
'06 '07
'08
'09
'10
'11
'12
'13
FY14
Fiscal Yearly Cost Rate (L2)
Fiscal Yearly Cum Cost (L2)
< Program X Example >
< Program X Example >
20
5
80
60
40
20
0
1
FY07
36 - 27 June 2012
2
FY08
3
FY09
4
FY10
5
FY11
6
FY12
0
7
FY13
1
FY14
FY07
2
FY08
3
FY09
Approved for public release - Distribution is unlimited
4
FY10
5
FY11
6
FY12
7
FY13
FY14
$ (millions)
10
100
$ (millions)
15
120
Software Maintenance, Sustaining Engineering, and Operational Support
Integrated Software Maintenance Life Cycle
Cost Estimation Model
37 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
I t
Integrated
t d Cost
C t Estimation
E ti ti Life
Lif Cycle
C l Model
M d l
• Two key
y components
p
• Software maintenance cost risk-uncertainty model
- Risk information model
- Risk interdependency analysis model
- Cost estimation application process model
• Integrated life cycle cost allocation model
- Program software life cycle product-activity profile



Post-development software releases
Fielded software sustaining engineering support
Software maintenance site infrastructure
- Maintenance resource-funding allocation model(s)
- Integrated cost estimation results
38 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
C t Ri
Cost
Risk-Uncertainty
kU
t i t Model
M d l
• Create a mechanism for identifying
y g and evaluating
g
software maintenance program risks at all
pertinent decision levels
• Identify those risks that directly impact program
software maintenance cost uncertainty
• Develop a methodology for integrating riskuncertainty information into program software
maintenance
i
life
lif cycle
l cost estimates
i
39 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
Program Software Maintenance Release Profile
Cycles
y
are different - p
platform dependent
p
User needs drive release content
40 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
N ti
Notional
l Software
S ft
Maintenance
M i t
Life-Cycle
Lif C l
Cost Allocation Model
TD / EMD
PDSS
PPSS
Cost
R l
Releases
Change Driven Cost
Change Related Cost
DEV
A
B
41 - 27 June 2012
C
Software Maintenance Estimation Period
Approved for public release - Distribution is unlimited
Releases
Software Maintenance, Sustaining Engineering, and Operational Support
P
Program
IInformation
f
ti Requirements
R
i
t
• System and software maintenance strategy
• Included products and activities
• Software release profile - PDSS/PPSS
-
Release type and schedule (high level)
Release profile stability
User base - deployment strategy
Configuration information
Performing organizations
Projected operational environment
Program and software risk analysis
Historical profile information (later)
42 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
Program
SW Maintenance
Profile
• specific time period
• products and activities
• SW maintenance
strategy
SW Maintenance
WBS Elements
• software release
• sustainment activities
• infrastructure
Cost Estimating
Relationships
CER Cost
Uncertainty
• cost per WBS element
Integrated Cost
Profile
Risk
Analysis
Program Cost
Uncertainty
• cost p
per time p
period
43 - 27 June 2012
Approved for public release - Distribution is unlimited
Integrated SW
Maintenance Cost
Estimate
Software Maintenance, Sustaining Engineering, and Operational Support
Long Term Objective
44 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
AH-64D Longbow “Night Fury”
10,000 Flying Hours Reached on 28 June 2011
3 0,000 hours
340,000
ou s of
o maintenance
a e a ce by hundreds
u d eds o
of weapons
eapo s technicians
ec c a s in co
conjunction
ju c o
with countless hours of repairs and inspections performed by avionics, electrical
and environmental, engine, fuels and structural personnel
If there was a 20% cut on avionics software maintenance
maintenance,
what would be the mission impact?
45 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
S f
Software
Maintenance C
Cost-Capability
C
Framework
Software
Products
Personnel
Capability
Cost
Software
Services
Facilities
• applied resources
• maintenance process
performance
• risk
Software
S
ft
Changes
• software baseline
architecture - design
• requirements change drivers
46 - 27 June 2012
Approved for public release - Distribution is unlimited
• functional domain
• system interfaces
• operational
environment
Software Maintenance, Sustaining Engineering, and Operational Support
Ma
aintenance
Requirem
ments - Cos
st ($M)
H
How
much
h should
h ld software
ft
maintenance
i t
cost?
t?
Increased Risk - Increased Capability
"It's All About the Money“, Dr. Chien Huo, CAPE, November 2011
47 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
Summary
• The projected Army operational and economic
environment places an increasing emphasis on the
performance of software maintenance and sustaining
engineering efforts
• Accurate software maintenance life cycle cost
estimates are critical to ensuring that objective
resource information
i f
ti is
i available
il bl to
t program decision
d i i
makers
• The estimation methodology must be configured to
address the unique characteristics and projected
sustainment profile of each program
48 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
C t t IInformation
Contact
f
ti
Jim Judy
U.S. Army Office of the Deputy Assistant
Secretary of the Army for Cost &
Economics (ODASA-CE)
5701 21st Street
Building 216
Fort Belvoir, VA 22060-5546
(703) 697-1612
james.m.judy.civ@mail.mil
Cheryl Jones
U.S. Army RDECOM - ARDEC
Quality Engineering & System
Assurance
RDAR-QES-A
Building 62
Picatinny Arsenal, NJ 07806-5000
(973) 724-2644
cheryl.jones5@us.army.mil
Jeramia Poland
U.S. Army ODASA-CE
5701 21st Street
Building 216
Fort Belvoir, VA 22060-5546
(703) 697-1640
Jeramia.poland.civ@mail.mil
Dr. Brad Clark
Software Metrics, Inc.
4345 High Ridge Road
Haymarket, Va. 20169
(703) 754-0115
brad@software-metrics.com
49 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
Additional Information
50 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
A
Army
S
Software
ft
M
Maintenance
i t
Study
St d
• Sponsored by U.S. Army Office of the Deputy Assistant
Secretary of the Army for Cost & Economics (ODASA
(ODASACE)
• Collaborative environment - diverse perspectives
• Army, Air Force, Navy, OSD, Industry participation
• Initial focus on ACAT-1
ACAT 1 weapons system software
maintenance costs
• Technical approach based on measurement and
estimation best practices
51 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
S ft
Software
M
Maintenance
i t
Product
P d t Sizing
Si i
•
•
•
•
Overall concept of “E size” for maintenance products
E Size for types of changes (different drivers)
Composite E Size for different types of releases
Concept - profile the program maintenance strategy based on
numbers and types of releases (variable costs) - add semi-fixed
costs related to supporting the fielded software (costs not
based on amount of work to construct a release)
• Variable costs driven by type and number of product changes
• Semi-fixed costs driven by number of deployed platforms,
domain, operating tempo, etc.
• Similar to software development sizing “E Sloc” but focused
on change drivers/release content and support scope
52 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
S ft
Software
M
Maintenance
i t
Change
Ch
Drivers
Di
•
•
•
•
•
•
The factors that cause the changes that are made to an operational
software baseline
Software changes characterized by:
- source of change
- type
t
off change
h
- impact of change (scope, complexity, etc.)
- priority of implementation
Change drivers can impact the operational software configuration,
associated sustaining engineering activities, and the implemented
maintenance infrastructure
Most change requirements are allocated to planned releases per
time period
Deferred change
g requirements
q
are defined as “backlog”
g
Drivers with different characteristics are costed differently
53 - 27 June 2012
Approved for public release - Distribution is unlimited
Software Maintenance, Sustaining Engineering, and Operational Support
S ft
Software
Change
Ch
Drivers
Di
External Drivers
Internal Drivers
• Operational User
- Functional modifications
- Functional additions
- Functional deletions
• Stakeholder
-
- Technical debt
- Deferred functionality
• Maintenance Operations
- Maintainer
M i t i
skill
kill sett
Threat
Mission doctrine
System interoperability
E t
External
l testing/IV&V
t ti /IV&V
External audits
• Mandate
- Legal/Regulatory/Policy
• Technology
- Technology obsolescence
- Infrastructure changes
54 - 27 June 2012
• Legacy
Types of Changes
• Corrective (identified defects)
• Preventive (latent faults)
• Perfective (functional
enhancements)
• Adaptive
Ad ti ((new HW/SW
environments)
Approved for public release - Distribution is unlimited