An Application of COCOMO and COSYSMO in
Acquisition Research
Dr. Peter Hantos and Nancy Kern
The Aerospace Corporation
COSYSMO Workshop, 25th International Forum on COCOMO,
Los Angeles, CA, November 4, 2010
1
© The Aerospace Corporation 2010
Acknowledgements
•
This work would not have been possible without the following:
– Reviewers
• Suellen Eslinger
• Dr. Sergio Alvarado
• B. Zane Faught
– Sponsors
• Rosalind Lewis
• Dr. Malina Hills
– Funding Source
• The Aerospace Corporation’s Independent Research & Development
Program
2
Outline
•
•
•
•
•
•
•
•
•
•
3
Objectives
Changes to DOD Acquisition Policy
The Impact on Systems Engineering Effort
Detour: Software Life Cycle Modeling Basics
Life Cycle Phases and Software Effort Distribution in the Previous
DOD 5000.2 Framework
Life Cycle Phases and Software Effort Distribution in the Current
DOD 5000.02 Framework
The Impact of Incremental Software Development
Conclusions
Acronyms
References
Objectives
•
•
•
•
4
The presentation’s objective is to highlight some consequences of
selected changes in the DOD 5000 Instructions
The focus of inquiry is on the alignment of the Preliminary Design
Review (PDR) with the Milestone B decision
Another aspect of the analysis is to determine the impact of the
mandate for increased competition in the Technology Development
phase
Using life cycle modeling and cost estimation research results, we
also explore the impact of different, basic software life cycle models
Changes to DOD Acquisition Policy
DOD 5000.2 (May 12, 2003)
Pre-Systems Acquisition
Technology
Development
Approval
A
JROC
Milestones:
ICD
Concept
Refinement
Systems Acquisition
System
Development &
Demonstration
Approval
Design
Readiness
Review
Low-Rate
Initial Prod
Approval
C
B
System Development
and
Demonstration
Technology
Development
SRR
Concept
Decision
SFR
PDR
CDR
IOC
Production
and
Deployment
TRR
SVR
DOD 5000.02 (December 8, 2008)
Pre-Systems Acquisition
Technology
Development
Approval
JROC
Milestones:
ICD
Materiel
Solution
Analysis
Materiel
Development
Decision
A
Systems Acquisition
Engineering and
Manufacturing
Development
Approval
Post-CDR
Assessment
B
SFR
PDR
CDR
TRR
PDR now conducted prior Milestone B
5
C
Engineering and
Manufacturing
Development
Technology
Development
SRR
Low-Rate
Initial Prod
Approval
IOC
Production
and
Deployment
SVR
The Rationale Behind the Changes
•
Selected aspects of the discussed changes were proposed earlier
in the 2006 Defense Acquisition Performance Assessment (DAPA)
report*:
– For Acquisition Category (ACAT) I and II programs, create contract terms
and conditions that require formal subcontractor level competition
instead of internal make-or-buy assessments by the prime
• According to the report, this higher level of visibility would allow the
government to better understand the technical and management risks
of the prime contractor’s plans
– Reposition the Milestone B decision to occur at PDR
• According to the report, the maturity of the designs at this phase
would allow more realistic program cost determination
• Industry and Government would be in a better position to agree on a
high confidence cost estimate for the desired capability
• Source Selection Authorities would have a competitive range
available to consider the proposals’ affordability
* Source: [DAPA 2006]
6
Constructive Systems Engineering Cost Model
(COSYSMO) Systems Engineering Effort Distribution*
% of
Total
Effort
36%
30%
27%
23%
20%
14%
10%
Time
(Notional)
ISO/IEC Conceptualization
15288 Phases
DOD 5000.02
Technical
Reviews
SRR
Development
SFR PDR CDR TRR
*Sources: [Valerdi 2008], [Hantos-Kern 2009]
7
Transition to
Operation
OT&E
SVR
The Impact on the Systems Engineering Effort
DOD 5000.2 (May 12, 2003)
Pre-Systems Acquisition
Concept
Refinement
Systems Acquisition
System
Development &
Demonstration
Approval
Technology
Development
Approval
JROC
Milestones:
ICD
Source Selection
A
Pre-Systems Acquisition
C
B
Concept
Decision
SRR
System Development
and
Demonstration
SFR
PDR
CDR
Technology
Development
Approval
Low-Rate
Initial Prod
Approval
Design
Readiness
Review
Technology
Development
IOC
Production
and
Deployment
TRR
DOD 5000.02 (December 8, 2008)
JROC
Milestones:
ICD
Contractor 1
11%
Post-CDR
Assessment
SRR
SFR
Low-Rate
Initial Prod
Approval
C
Engineering and
Manufacturing
Development
Technology
Development
PDR
CDR
TRR
IOC
Production
and
Deployment
SVR
54%
54%
21%
Systems Acquisition
B
Materiel
Solution
Analysis
Materiel
Development
Decision
SVR
A
Source Selection
Engineering and
Manufacturing
Development
Approval
32%
14%
14%
Contractor 1
32%
Contractor 2
11%
Contractor 2
Since program baseline is formalized after MS B, the cost and duration of acquisitions appear to decrease
- However, the overall cost of acquisitions, particularly the costs associated with the initial systems
engineering effort involving multiple contractor teams, will significantly increase
- Program Office effort, leading up to and carrying out source selection, needs to significantly increase
Program risk after MS B may be reduced but at increased cost for the overall acquisition
Conclusions from [Hantos-Kern 2009]
8
Moving Milestone B Increases the Overall Systems
Engineering Effort
“New”
Systems 200
Engineering
190
Effort
(as Percentage of 180
Total Systems
Engineering Effort
for One System)
164-122 .34
=
122
132-111 .19
=
111
170
160
150
140
130
120
110
100
164
+ 34%
132
+ 19%
“Old”
111
1
2
Minimum Number of Competitors
Source: [Hantos-Kern 2009]
9
122
3
Number of
4 Competing
Contractors
Detour: Software Life Cycle Modeling Basics
10
Positioning the Classic Software Waterfall Life Cycle
Model in Software-Intensive System Development
System Requirements
Definition &
System Requirements
Allocation
Software Plans
&
Requirements
Software Development
High-Level Design
Detail Design
Software Design
Program m ing
(Includes Unit Test)
Softw are Integration &
Test
•
Waterfall characteristics
– All system requirements are defined/allocated to software prior
System Requirem ents
System Integration &
Definition &
to development
Test
System Requirem ents
Allocation
– Software intended to be developed all at once (One single build)
– No significant overlap allowed between development phases
– Typically marred by late problem discovery and resolution
– Can be mitigated via incremental development (Next slide)
11
System Integration &
Test
“Big Bang”
Position of the Incremental Life Cycle Model in
Software-Intensive System Development
Waterfall Development of Software Increments
Software
Requirements
Detailed
Design
Re-Assessment
Software Design
System Requirements
Definition &
System Requirements
Allocation
Programming
Software Integration &
Regression Test
Software Requirements
&
Increment Planning
Software Increment 1
Software Increment 2
Software Increment 3
•
•
•
Software Increment n
System Integration &
Test
Overlap of increments (concurrent builds) are allowed
12
Software Engineering Effort Distribution in the
Waterfall Model (COCOMO)
% of Average
Manpower
Level
150%
44%
100%
32%
50%
18%
6%
•
•
13
56%
29%
0%
Software
Life Cycle Phases
Schedule
Plans &
Requirements
Software
Design
79%
Programming
100%
Software
Integration &
Test
Chart shows the effort-distribution of a typical software system [Boehm 1981]
The shaded area is an approximation of a Rayleigh curve that is the basis for
computing effort-distribution in parametric models*
* Note that the effort required for the Plans & Requirements phase is not
covered by the Rayleigh curve
Life Cycle Phases in the Previous DOD 5000.2
Framework (May 12, 2003)
DOD 5000.2 (May 12, 2003)
Pre-Systems Acquisition
Milestones:
Acquisition
Concept
Refinement
Systems Acquisition
System
Development &
Demonstration
Approval
Technology
Development
Approval
A
Low-Rate
Initial Prod
Approval
Design
Readiness
Review
C
B
System Development
and
Demonstration
Technology
Development
Concept
Decision
SRR
SFR
PDR
CDR
IOC
Production
and
Deployment
TRR
SVR
ISO/IEC 15288 - COSYSMO
Systems
Engineering
Software
Engineering
Early
Technology
Development
Conceptualization
Development
OT&E
System
Integration
& Test
Software Support
to
Conceptualization
SW Plans
&
Reqs
Software
Development
Pre-A Systems Engineering and Pre-B Software Engineering
efforts are not comprehended in any estimation models
14
Transition
To
Operation
SW Effort Distribution* in the Previous DOD 5000.2
Framework (May 12, 2003)
DOD 5000.2 (May 12, 2003)
Source Selection
Pre-Systems Acquisition
Technology
Development
Approval
Milestones:
Concept
Refinement
Concept
Decision
A
Systems Acquisition
System
Development &
Demonstration
Approval
Design
Readiness
Review
Low-Rate
Initial Prod
Approval
C
B
System Development
and
Demonstration
Technology
Development
SRR
SFR
PDR
CDR
IOC
Production
and
Deployment
TRR
SVR
44%
32%
Contractor1
Contractor2
Software Support
to
Conceptualization
Software Support
to
Conceptualization
18%
6%
Contractor2 does not do
any substantial software
development
Structure and Pre-MS B phase naming implies that software is not a “technology”
Simplified assumption: Only one software increment and it is Waterfall
15
Life Cycle Phases in the Current DOD 5000.02
Framework (December 8, 2008)
DOD 5000.02 (December 8, 2008)
Pre-Systems Acquisition
Technology
Development
Approval
Milestones:
Acquisition
Systems Acquisition
Engineering and
Manufacturing
Development
Approval
A
B
Materiel
Solution
Analysis
SRR
SFR
C
Engineering and
Manufacturing
Development
Technology
Development
Materiel
Development
Decision
Low-Rate
Initial Prod
Approval
Post-CDR
Assessment
PDR
CDR
IOC
Production
and
Deployment
TRR
SVR
ISO/IEC 15288 - COSYSMO
Systems
Engineering
Software
Engineering
Early
Technology
Development
Conceptualization
Development
OT&E
Transition
To
Operation
System
Integration
& Test
Software Support
to
Conceptualization
SW Plans
&
Reqs
Software
Development
Note that the Systems Engineering
Software Engineering life cycle
alignment is the same; only the acquisition life cycle part has changed
16
SW Effort Distribution in the Current DOD 5000.02
Framework (December 2, 2008)
Pre-Systems Acquisition
Source Selection DOD 5000.02 (December 2, 2008)
Systems Acquisition
Engineering and
Manufacturing
Development
Approval
Technology
Development
Approval
Milestones:
Materiel
Solution
Analysis
Materiel
Development
Decision
A
B
CDR
PDR
SFR
44%
32%
Contractor1
Software Support
to
Conceptualization
Contractor2
TRR
IOC
Production
and
Deployment
SVR
Cannot exactly determine/plan
Pre-MS B software effort*
18%
6%
?%
Software Support
to
Conceptualization
C
Engineering and
Manufacturing
Development
Technology
Development
SRR
Low-Rate
Initial Prod
Approval
Post-CDR
Assessment
Contractor2 not awarded the
contract, so no effort after MS B
18%
6%
PDR is an arbitrary, system-level review from the software development perspective
* An additional uncertainty, which is not formally indicated, is that effort curves do
not account for any possible software technology development
17
Moving Milestone B, Combined with Mandated Pre-MS B
Competition, Increases the Software Engineering Effort
Software 200
Engineering
190
Effort
(as Percentage of 180
Total Software
Engineering Effort
for One System)
148-100 .48
=
100
124-100 .24
=
100
“New”
170
160
150
140
130
120
110
100
148
+ 48%
124
+ 24%
100
1
2
100
3
Minimum Number of Competitors
18
Due to the mentioned uncertainties, the chart only shows the minimum
increase; actual values are always higher
“Old”
Number of
4 Competing
Contractors
Incremental Software Development in the Previous DOD
5000.2 Framework
DOD 5000.2 (May 12, 2003)
Pre-Systems Acquisition
Technology
Development
Approval
Milestones:
Acquisition
Concept
Refinement
Concept
Decision
Systems Acquisition
System
Development &
Demonstration
Approval
A
Low-Rate
Initial Prod
Approval
Design
Readiness
Review
C
B
System Development
and
Demonstration
Technology
Development
SRR
SFR
PDR
CDR
IOC
Production
and
Deployment
TRR
SVR
ISO/IEC 15288 -COSYSMO
Systems
Engineering
Software
Engineering
Early
Technology
Development
Conceptualization
Development
OT&E
Transition
To
Operation
System
Integration
& Test
Software Support
to
Conceptualization
SW Reqs
& Incr
Planning
Increment 1
Increment 2
Increment 3
Incremental Development decision took place only after MS B and did not have an impact
19
Incremental Software Development in the Current DOD
5000.02 Framework
DOD 5000.02 (December 8, 2008)
Pre-Systems Acquisition
Technology
Development
Approval
Milestones:
Acquisition
A
Systems Acquisition
Engineering and
Manufacturing
Development
Approval
Post-CDR
Assessment
B
Materiel
Solution
Analysis
SRR
SFR
C
Engineering and
Manufacturing
Development
Technology
Development
Materiel
Development
Decision
Low-Rate
Initial Prod
Approval
PDR
CDR
TRR
IOC
Production
and
Deployment
SVR
ISO/IEC 15288-COSYSMO
Systems
Engineering
Software
Engineering
Early
Technology
Development
Conceptualization
Development
OT&E
Transition
To
Operation
System
Integration
& Test
Software Support
to
Conceptualization
SW Reqs
& Incr
Planning
Increment 1
Increment 2
Increment 3
Incremental Development does not change the main conclusions, only increases uncertainty
20
Conclusions
•
•
•
The repositioning of MS B and new rules for competition will
explicitly increase both the systems and software engineering effort
for the program (minimum 19% and 24%, respectively)
Various scenarios have been analyzed, but actual cost/schedule
impacts still remain to be seen
– The vision for systems during the Pre-MS A phase is usually quite vague;
consequently, estimates based on that vision have always high level of
uncertainty
– The expected effort increase in both cases is in the front-end, i.e., in the
Technology Development phase
• However, in addition to technical considerations, the reality is that the
actual determination of Technology Development funding will be based
on various component and other, higher level negotiations, adding
further uncertainty and instability to the estimates
Both under- or over-estimation of resources for Technology
Development can put the program in jeopardy
– If the estimates seem to be too high at MS A then the program might not
be even initiated or Technology Development could be underfunded
– Underestimation of resources will definitely cause major tensions and
schedule/cost problems later
21
Acronyms
ACAT
CDR
COCOMO
COSYSMO
DAPA
DOD
IEC
ISO
IOC
PDR
SFR
SRR
SVR
SW
TRR
22
Acquisition Category
Critical Design Review
Constructive Cost Model
Constructive Systems Engineering Cost Model
Defense Acquisition Performance Assessment
Department of Defense
International Electrotechnical Commission
International Standards Organization
Initial Operational Capability
Preliminary Design Review
System Functional Review
System Requirements Review
System Verification Review
Software
Test Readiness Review
References
Boehm 1981
DAPA 2006
Hantos-Kern 2009
Valerdi 2008
23
Software Engineering Economics, Prentice-Hall, 1981
Defense Acquisition Performance Assessment (DAPA)
Report, March 2006
Hantos, P., Kern, N., Cost and Risk Impacts of the New
DOD 5000 Defense Acquisition Framework, NDIA 12th Annual
Systems Engineering Conference, San Diego, CA, 27
October 2009
Valerdi, R., The Constructive Systems Engineering Cost
Model (COSYSMO), VDM Verlag Dr. Mueller, 2008
Use of Trademarks, Service Marks and Trade Names
Use of any trademarks in this material is not intended in
any way to infringe on the rights of the trademark holder.
All trademarks, service marks, and trade names are the
property of their respective owners.
24