Harvesting Reference Points for
Cost Estimation: A Step in the
SEI’s Cost Estimation Method
Software Engineering Institute
Carnegie Mellon University
Pittsburgh, PA 15213
Robert W. Stoddard
Dennis Goldenson
Rhonda Brown
24 October 2013
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
Introduction
QUELCE (Quantifying Uncertainty for Early Lifecycle Cost Estimation) is
a multi-year research project led by the Software Engineering
Measurement and Analysis (SEMA) team within the SEI Software
Solutions Division.
Research team membership comprises SEI technical staff with cost
estimation background in collaboration with several external faculty (Dr.
Ricardo Valerdi, Univ of Arizona, & Dr. Eduardo Miranda, CMU).
This research is motivated by (1) the WSARA Act requiring cost
estimates pre-Milestone A and (2) DoD’s need for more accurate cost
estimation methods that provide continuous monitoring of changing
assumptions and constraints.
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
3
Information Flow for Early Lifecycle Estimation
Proposed Material Solution & Analysis of Alternatives
Expert Judgments
Information from Analogous Programs/Systems
Program Execution Change Drivers
System Characteristics
Trade-offs
•KPP selection
•Systems design
•Sustainment issues
...
Operational Capability
Trade-offs
•Mission / CONOPS
•Capability-based analysis
...
Technology Development
Strategy
•Production quantity
•Acquisition mgt
•Scope definition/responsibility
•Contract award
Driver States & Probabilities
Plans, Specifications, Assessments
Probabilistic
Modeling (BBN) &
Monte Carlo
Simulation
Cost Estimates
•Analogy
•Parametric
Models
•Engineering
•CERs
Program Execution
Scenarios with Conditional
Probabilities of
Drivers/States
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
4
Repository: Analyze Existing Data to Model
Program Execution Uncertainties – 1
Materiel Solution Analysis Phase – Pre-Milestone Estimate
2. Reduce Cause and
Effect Relationships
via Dependency
Structure Matrix
Techniques
1. Identify
Change
Drivers &
States
Program Change
Repository
Prog
DDG51
JTRS
F22
State
cond 1
Driver
CONOPS
cond 2
cond 3
cond 1
cond 2
cond 1
cond 2
cond 3
System
CapDef
InterOp
Produc
Contract
Function
CONOPS
3. Assign
Conditional
Probabilities to
BBN Model
Records show that Strategic
Vision changed in 45% of the
MDAPS
Change Driver Nominal State
Alternative States
Additional
Production
Scope Reduction
deliverable (e.g.
downsized
(funding reduction)
training & manuals)
Program
New echelon
becomes Joint
Trade-offs
Subtraction
Variance
[performance vs
affordaility, etc.]
Obligated vs.
FFRDC ceiling Funding change for Funding spread
allocated funds
issue
end of year
out
shifted
Advocate
Service owner
Senator did not Change in senior
requires change
different than
get re-elected
pentagon staff
in mission
CONOPS users
scope
Scope
Definition
Mission /
CONOPS
Stable
Users added
Additional
(foreign)
customer
defined
New condition
New mission
Capability
Definition
Stable
Addition
Funding
Schedule
Established
Funding delays tie up
resources {e.g.
operational test}
Stable
Joint service program
loses particpant
Advocacy
Change
Closing
Technical
Gaps (CBA)
●
●
●
Selected Trade
studies are
sufficient
Technology does not
achieve satisfactory
performance
Technology is
too expensive
Selected solution
cannot achieve
desired outcome
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
Technology not
New technology not
performing as
testing well
expected
~~~~
~~~~
5. Monte Carlo
Simulation to
Compute Cost
Distribution
The Materiel Solution of a
global network command and
control system anticipates a
possible change in Strategic
Vision that will include allied
participation.
For C2 systems,
how often does
Strategic Vision
change?
Driver State Matrix
4. Calculate Cost
Factor
Distributions for
Program Execution
Scenarios
A
Sharing information with allies
creates new encryption
requirements (a change in
Mission/CONOPs).
These changes lead to
changes in Capability
Definition.
Repository identifies probability of
change in MDAP cost drivers.
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
5
Repository: Analyze Existing Data to Model
Program Execution Uncertainties – 2
Materiel Solution Analysis Phase – Pre-Milestone Estimate
2. Reduce Cause and
Effect Relationships
via Dependency
Structure Matrix
Techniques
Sharing information with allies
creates new encryption
requirements (a change in
Mission/CONOPs).
70% of the time the
Mission/CONOPS changes
2
2
2
1
1
1
2
2
1
1
2
2
1
2
2
1
2
1
2
2
1
1
1
1
2
2
1
1
2
1
2
2
2
0
1
0
1
1
1
1
1
1
1
1
1
1
3
1
1
2
2
2
1
2
2
2
1
2
1
1
3
1
1
1
1
1
1
1
1
1
2
1
2
0
3
2
0
2
2
2
1
2
2
2
1
1
2
1
1
1
1
3
2
1
0
1
2
1
1
2
2
1
1
1
1
1
1
1
2
2
1
2
2
2
1
1
1
1
1
1
2
1
1
1
1
2
3
2
2
2
2
1
1
2
1
1
1
3
3
1
2
2
1
1
2
1
1
2
2
2
0
2
1
1
1
1
1
1
1
0
1
1
1
2
1
1
1
1
1
1
1
2
1
1
1
1
1
1
1
1
2
1
1
2
2
2
2
2
3
1
2
1
6
29
16
6
34
27
29
21
33
16
5
10
5
19
12
14
6
5
6
10
7
7
4
0
0
2
2
0
0
0
0
0
0
1
3
5
0
0
1
0
2
2
2
1
2
5
2
4
3
0
0
0
0
0
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
1
1
2
1
1
Number right of diagonal
1
1
1
3
Total
2
0
2
1
Product Challenge
1
2
2
1
0
2
3
2
Project Challenge
1
3
3
Size
3
Contractor Performance
3
3
Data Ownership
Mission / CONOPS
Change in Strategic Vision
Capability Definition
Advocacy Change
Closing Technical Gaps (CBA)
Building Technical Capability & Capacity (CBA)
Interoperability
Systems Design
Interdependency
Functional Measures
Scope Definition
Functional Solution Criteria (measure)
Funding Schedule
Acquisition Management
Program Mgt - Contractor Relations
Project Social / Dev Env
Prog Mgt Structure
Manning at program office
Scope Responsibility
Standards/Certifications
Supply Chain Vulnerabilities
Information sharing
PO Process Performance
Sustainment Issues
Contract Award
Production Quantity
Data Ownership
Production Quantity
~~~~
Contract Award
~~~~
Sustainment Issues
~~~~
PO Process Performance
~~~~
~~~~
Information sharing
~~~~
~~~~
Supply Chain Vulnerabilities
~~~~
~~~~
Causes
Standards/Certifications
~~~~
Technology not
New technology not
performing as
testing well
expected
Scope Responsibility
Selected solution
cannot achieve
desired outcome
~~~~
Manning at program office
Technology is
too expensive
~~~~
Prog Mgt Structure
Technology does not
achieve satisfactory
performance
~~~~
Project Social / Dev Env
●
●
●
Selected Trade
studies are
sufficient
Program Mgt - Contractor Relations
Joint service program
loses particpant
Closing
Technical
Gaps (CBA)
Acquisition Management
Funding delays tie up
resources {e.g.
operational test}
Stable
Advocacy
Change
Funding Schedule
Addition
Established
Functional Solution Criteria (measure)
Stable
Funding
Schedule
Scope Definition
Capability
Definition
Change Drivers - Cause & Effects Matrix
Effects
Functional Measures
New mission
Interdependency
Additional
(foreign)
customer
New condition
Systems Design
Users added
defined
Interoperability
Stable
Building Technical Capability & Capacity (CBA)
Alternative States
Additional
Production
Scope Reduction
deliverable (e.g.
downsized
(funding reduction)
training & manuals)
Program
New echelon
becomes Joint
Trade-offs
Subtraction
Variance
[performance vs
affordaility, etc.]
Obligated vs.
FFRDC ceiling Funding change for Funding spread
allocated funds
issue
end of year
out
shifted
Advocate
Service owner
Senator did not Change in senior
requires change
different than
get re-elected
pentagon staff
in mission
CONOPS users
scope
Scope
Definition
Mission /
CONOPS
Closing Technical Gaps (CBA)
Change Driver Nominal State
These changes lead to
changes in Capability
Definition.
DSM Cause-Effect Matrix
Advocacy Change
Driver State Matrix
Test & Evaluation
System De
CapDef
InterOpera
Production
Contract
Functional
CONOPS
Capability Definition
F22
Driver
CONOPS
Change in Strategic Vision
JTRS
cond 2
cond 3
cond 1
cond 2
cond 1
cond 2
cond 3
If Strategic Vision
changes, what
else changes?
Mission / CONOPS
Prog
State
DDG51 cond 1
5. Monte Carlo
Simulation to
Compute Cost
Distribution
The Materiel Solution of a
global network command and
control system anticipates a
possible change in Strategic
Vision that will include allied
participation.
Cost Estimate
Program Change
Repository
4. Calculate Cost
Factor
Distributions for
Program Execution
Scenarios
3. Assign
Conditional
Probabilities to
BBN Model
Industry Company Assessment
1. Identify
Change
Drivers &
States
A
1
3
1
1
1
2
1
1
2
2
2
6
Repository: Analyze Existing Data to Model
Program Execution Uncertainties – 3
Materiel Solution Analysis Phase – Pre-Milestone Estimate
4. Calculate Cost
Factor
Distributions for
Program Execution
Scenarios
3. Assign
Conditional
Probabilities to
BBN Model
When
both Strategic Vision & Mission/CONOPs
experience change, the BBN calculates that
Capability Definition will also change
Sharing information with allies
creates new encryption
requirements (a change in
Mission/CONOPs).
95% of the time.
2
1
1
2
2
1
2
2
1
2
1
2
2
1
1
1
1
1
2
0
1
1
1
2
1
1
1
1
1
1
1
1
1
3
2
1
1
1
2
2
2
2
2
1
2
1
1
3
1
1
1
1
1
1
1
2
1
2
0
3
2
0
2
2
2
1
2
2
2
1
1
1
3
2
1
0
1
2
1
1
2
2
1
1
1
1
1
1
1
2
2
1
2
2
2
1
1
1
1
1
1
2
1
1
1
1
2
3
2
2
2
2
1
1
2
1
1
1
3
3
1
2
2
1
1
2
1
1
2
2
2
0
2
1
1
1
1
1
1
1
0
1
1
1
2
1
1
2
1
1
1
1
1
1
1
2
Product Challenge
1
2
0
1
Contractor Performance
2
2
1
2
1
2
1
1
1
1
1
1
1
1
1
1
2
1
1
1
1
1
2
2
2
2
2
3
1
1
1
2
1
Number right of diagonal
2
2
2
1
1
1
2
Total
1
1
1
3
Scope Responsibility
Acquisition Management
Funding Schedule
Functional Solution Criteria (measure)
2
0
2
1
Test & Evaluation
~~~~
1
2
2
1
0
2
3
2
Cost Estimate
~~~~
1
3
3
Industry Company Assessment
~~~~
3
Data Ownership
~~~~
3
3
Production Quantity
~~~~
Mission / CONOPS
Change in Strategic Vision
Capability Definition
Advocacy Change
Closing Technical Gaps (CBA)
Building Technical Capability & Capacity (CBA)
Interoperability
Systems Design
Interdependency
Functional Measures
Scope Definition
Functional Solution Criteria (measure)
Funding Schedule
Acquisition Management
Program Mgt - Contractor Relations
Project Social / Dev Env
Prog Mgt Structure
Manning at program office
Scope Responsibility
Standards/Certifications
Supply Chain Vulnerabilities
Information sharing
PO Process Performance
Sustainment Issues
Contract Award
Production Quantity
Data Ownership
Industry Company Assessment
Cost Estimate
Contract Award
~~~~
Technology not
New technology not
performing as
testing well
expected
Sustainment Issues
~~~~
PO Process Performance
~~~~
Information sharing
Selected solution
cannot achieve
desired outcome
~~~~
~~~~
Supply Chain Vulnerabilities
Technology is
too expensive
~~~~
~~~~
Standards/Certifications
Technology does not
achieve satisfactory
performance
~~~~
Manning at program office
Selected Trade
studies are
sufficient
Prog Mgt Structure
●
●
●
Project Social / Dev Env
Closing
Technical
Gaps (CBA)
Scope Definition
Advocacy
Change
Causes
Functional Measures
Joint service program
loses particpant
Interdependency
Funding delays tie up
resources {e.g.
operational test}
Stable
Systems Design
Addition
Established
Interoperability
Stable
Funding
Schedule
Building Technical Capability & Capacity (CBA)
Capability
Definition
Closing Technical Gaps (CBA)
New mission
Advocacy Change
Additional
(foreign)
customer
New condition
Capability Definition
Users added
defined
Change in Strategic Vision
Stable
Mission / CONOPS
Effects
Alternative States
Additional
Production
Scope Reduction
deliverable (e.g.
downsized
(funding reduction)
training & manuals)
Program
New echelon
becomes Joint
Trade-offs
Subtraction
Variance
[performance vs
affordaility, etc.]
Obligated vs.
FFRDC ceiling Funding change for Funding spread
allocated funds
issue
end of year
out
shifted
Advocate
Service owner
Senator did not Change in senior
requires change
different than
get re-elected
pentagon staff
in mission
CONOPS users
scope
Scope
Definition
Mission /
CONOPS
Program Mgt - Contractor Relations
Driver State Matrix
Change Driver Nominal State
These changes lead to
changes in Capability
Definition.
BBN Model
Change Drivers - Cause & Effects Matrix
Project Challenge
DSM Cause-Effect Matrix
5. Monte Carlo
Simulation to
Compute Cost
Distribution
The Materiel Solution of a
global network command and
control system anticipates a
possible change in Strategic
Vision that will include allied
participation.
Size
2. Reduce Cause and
Effect Relationships
via Dependency
Structure Matrix
Techniques
1. Identify
Change
Drivers &
States
A
6
29
16
6
34
27
29
21
33
16
5
10
5
19
12
14
6
5
6
10
7
7
4
0
0
2
2
0
0
0
0
0
0
0
0
1
3
5
0
0
1
0
2
2
2
1
2
5
2
4
3
0
0
0
0
0
0
0
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
3
1
1
1
2
2
2
2
7
Develop Efficient Techniques to Calibrate Expert
Judgment of Program Uncertainties
Solution
Step 2: Iterate
through a series
of domainspecific tests
Step 3: Feedback on
test performance
Step 1: Virtual
training using
reference
points
Outcome: Expert
renders calibrated
estimate of size
Program Change
Repository
1)
Size of ground combat
vehicle targeting feature xyz
in 2002 consisted of 25
KSLOC Ada
2)
Size of Army artillery firing
capability feature abc in 2007
consisted of 18 KSLOC C++
3)
…
Un-Calibrated
Calibrated
Calibrated = more
realistic size and
wider range to
reflect true expert
uncertainty
Estimate of SW Size
Used with permission from Douglas Hubbard Copyright HDR 2008 dwhubbard@hubbardresearch.com
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
8
Research Challenge
QUELCE team found the “mining” of domain reference points challenging
in terms of requisite knowledge, effort, and schedule.
The use of an NVivo-like platform was hypothesized to address the
intermediate step of identifying potential change driver experiences
documented in historical MDAP artifacts
• without losing the context of a given artifact,
• enabling queries by attributes across MDAPs, and
• achieving review by others prior to committing to a refined domain reference
point statement
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
9
How CAQDAS May Help
Computer-Assisted/Aided Qualitative Data AnalysiS (CAQDAS) is
the “use of computer software to aid qualitative research such as
transcription analysis, coding and text interpretation, recursive
abstraction, content analysis, discourse analysis, grounded theory
methodology, etc.”1
CAQDAS software initially developed for social science researchers
needed a way to assemble, annotate, and make sense out of a myriad
of textual artifacts.
The QUELCE team conceived of the use of CAQDAS in the sense that
“research” has always been inherent in the use of analogy for cost
estimation. CAQDAS enables a more formal treatment of that activity.
1
Wikipedia, http://en.wikipedia.org/wiki/CAQDAS
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
10
NVivo Experiment
Phase 1: Domain experts coded an assigned set of DoD program
artifacts likely to exhibit potential change drivers
•
Checkpoint synchronization in the middle of this phase
Phase 2: Domain experts and the research team conducted queries
against the artifacts using NVivo text analytic and search capabilities
Phase 3: Feedback gathered from all regarding NVivo usage
•
•
•
On the NVivo experience
On high-value documents
On the co-occurrence and/or cascading of change drivers
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
11
Experimental Procedure
Approximately 200 artifacts of an inventory of 1,500 artifacts were
identified for coding.
We operated in standalone mode, with each domain expert assigned a
unique NVivo project on the server.
Each NVivo project possessed a unique subset of the 200 artifacts (we
balanced the workload among 7 experts).
Experts were asked to open assigned artifacts, one at a time, within
NVivo and identify the apparent change drivers, if any.
We asked that experts to track their time in hours during the coding so
that we could gauge the productivity rate in forecasting future coding
projects.
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
12
Detailed Experimental Work Flow
STEP 1: Open
NVivo project
STEP 2: Open a
file to code
STEP 4: Sequentially
identify change events
STEP 6: For each change
instance, highlight the text
and tag with a Change
Driver code
STEP 3: Peruse the
entire artifact for
context
STEP 5: Within a change
event, identify one or more
change instances
STEP 7: For each change instance,
optionally highlight and tag
separate elements of Stimulus,
Response, and Outcome
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
13
An NVivo Demonstration Platform
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
14
NVivo Coding Artifacts with Change Drivers
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
15
NVivo Summary of Change Driver Coding
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
16
NVivo Simple Query on a Change Driver
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
17
NVivo Search on “Aerospace” and Schedule
Change Driver
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
18
NVivo Word Frequency
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
19
NVivo Word Search Cluster Analysis
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
20
NVivo Analysis of Reproducibility
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
21
NVivo Search on Co-occurrence of Change
Drivers Within Artifacts
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
22
Early Experimental Feedback
Although individual NVivo projects may be employed and then merged
together, efficiencies may be possible via the team server setup.
The NVivo performance across sensitive networks can be troublesome.
Installing NVivo with today’s security infrastructure remains quite
challenging.
Experts found the coding within NVivo to be painless, e.g., as easy as
highlight and drag or highlight and click on a pull-down list of change
drivers.
Queries provide a means to scan a large portfolio of artifacts for key
words, change driver codes, co-occurrence of change drivers, etc…
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
23
Future Work
Need to enlist additional experts to code the same artifact set and
evaluate reproducibility.
Need to design and implement expert judgment experiments for
improvement in judgment “calibration” using a coded NVivo repository
as a reference.
Need to refine the query results from an extensive NVivo code
repository with the aim of producing refined domain reference points.
Need to consider the at-scale implications and logistics using a tool such
as NVivo, e.g., as a living DoD asset.
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
24
Contact Information
Robert W. Stoddard
Principal Researcher
Software Solutions Division, SEAP
Telephone: +1 412-268-1121
Email: rws@sei.cmu.edu
U.S. Mail
Software Engineering Institute
Customer Relations
4500 Fifth Avenue
Pittsburgh, PA 15213-2612
USA
Web
www.sei.cmu.edu
www.sei.cmu.edu/contact.cfm
Customer Relations
Email: info@sei.cmu.edu
Telephone: +1 412-268-5800
SEI Phone: +1 412-268-5800
SEI Fax: +1 412-268-6257
2013 COCOMO Forum
Stoddard, 24 October 2013
© 2013 Carnegie Mellon University
25