USC-CSE 2004
6/27/2016
Contextual Models of Cost, Quality and Cycle Time
Authors
Cvetan Redzic, Michael Crowley, Nancy Eickelmann, Jongmoon Baik
Motorola Labs
Schaumburg, IL 60196
We present a value-based approach to controlling software cost, quality and cycle time
using multiple models of the software development process. These models are presented
in the context of two meta-models, the CMM and Knox model. The Software
Engineering Institute (SEI) Capability Maturity Model (CMM) provides a guideline for
the required processes that need to be in place for a software development team and their
organization to develop quality software, on time and within budget. The CMM
decomposes the essential control elements required to conduct software engineering in an
efficient and effective manner. These elements include documented requirements,
communications with the customer(s) and user(s), agreed-to commitments, planning,
documented process and work breakdown structure WBS. These elements are embedded
in a five tier ordinal hierarchy comprised of Key Process Areas (KPAs). It is assumed
that the higher the maturity level the more cost effective the software development
process, specifically the cost of poor quality is incrementally reduced by creating the
product “right” the first time, resulting in producing software of quality, on time and on
budget.
This paper examines evaluating process improvements based on results, or the
improvement in cost structure in four quality cost categories: prevention, appraisal,
internal failures and external failures. The Knox model of software quality provides a
theoretical account of changing quality cost structure as a function of a maturing software
development process. The Knox model has been found to serve as a reasonable predictor
of the cost of poor quality in a number of empirical studies. Knox divides the cost of poor
quality (CoPQ) into two categories (Internal and External Failure), and adds two
categories (Appraisal and Prevention) to complete the Costs of Quality (CoQ). The
category Create is not a CoQ category, but rather the total development cost less all CoQ.
See Figure 1.
Prevention Appraisal
Int Failure
Ext Failure TCoSQ
60
Cost as a Percent of Development
50
40
30
20
10
0
1
2
3
4
SEI CMM Level
5
USC-CSE 2004
6/27/2016
Figure 1 Knox’s Theoretical Model for Cost of Software Quality (Digital Technical Journal, vol.5,
No. 4., Fall 1993, Stephen T. Knox.)
Figure 1 provides a graphical representation of a hypothetical cost structure that would
correspond to the five levels of the CMM; the x-axis plots the five CMM levels, the yaxis plots the four cost of quality categories and a cumulative total cost of quality
(TC0SQ). The two basic assumptions are that total cost of quality is about 60% for a
level 1 organization and that this diminishes by about two thirds, to about 20%, for a
level 5 organization. A level 1 organization has no defined process and constructs
software in an ad hoc manner. The cost structure associated with a level 1 organization
has a total cost of quality of approximately 60%, prevention 1%, appraisal 3%, internal
failures 22%, and external failures 34%. These percentages are sensitive to the definitions
of quality and how failures and appraisal activities are recorded. For instance, the cost of
appraisal, if there were no failures would be the cost of executing unique tests once.
However, if failures are found this contributes to the cost of internal failure. Each
maturity level has a corresponding cost structure for the cost of quality.
The Knox model for cost of software quality was developed using the Cost of Quality
model developed in manufacturing environments and extending it across the SEI CMM
to produce a theoretical cost of software quality model. The definitions associated with
the model are provided in Table 1.
Table 1. Definition of Cost Categories
Cost
Category
Definition
Conformance
Appraisal
Discovering the condition
of the product
Prevention
Efforts to ensure product
quality
Nonconformance
Internal
failures
External
failures
Quality failures detected
prior to product shipment
Quality failures detected
after product shipment
Typical Costs for Software
Testing and associated
activities, product quality audits
SQA administration,
inspections, process
improvements, metrics
collection and analysis
Defect management, rework,
retesting
Technical support, complaint
investigation, defect notification
We now present three models, COQUALMO, COQ-DES, and MotoROI, which are used
in an integrated manner to provide and value-based approach to software development
and process improvement.
COQUALMO
COQUALMO (COnstructive QUALity MOdel) is one of COCOMO II suite of
models; it is be used to predict the number of introduced defects and removed defects by
USC-CSE 2004
6/27/2016
phase and residual defects in a software product development [Boehm et al. ,2000]. It can
be applied in the early stage of the software development life cycle such as analysis and
design, as well as in the later stages for refining the estimates when more information is
available. It enables 'what-if' analyses that demonstrate the impact of various defect
removal techniques and the effects of personnel, project, product and platform
characteristics of software development. It also provides insights into determining ship
time, assessment of payoffs for quality investments and understanding of interactions
amongst quality strategies.
COQUALMO is comprised of two separate models, Defect Introduction (DI) and
Defect Removal (DR) to estimate the rates of software defect introduction and defect
removal. The COQUALMO-DI model uses the same project attributes (size, personnel,
platform, project, and product characteristics) as used in COCOMO II to estimate the
member of requirements, design, and code defects introduced into the project. Its DR
model uses three defect removal profiles, analysis tools, peer reviews, and execution
testing, to determine what fraction of the introduced defects are removed.
Figure 2 COQUALMO model of defect introduction and removal. (Software Cost Estimation with
COCOMO II; Barry Boehm, et al.; Prentice Hal PPT; 2000)
CoQ-DES
CoQ-DES (Cost of Quality Discrete Event Simulator) is a Monte Carlo simulation
of a software development process that produces output distributions, by development
phase, of injected and removed faults, and per-phase costs of quality (and hence, costs of
poor quality). The inputs to the model are development phase dates, a priori fault
injection rate, quantified characterization of peer review and test activities, and hourly
loaded engineering costs. The per-phase a priori injection rates are well accounted for by
using historical per-phase COQUALMO DI rates applied to a current project estimations;
DR rates are currently unused. Calibration is in terms of historical CoQ categories:
Create, Prevent, Appraise, Internal and External Failure.
The model is developed as a number of Fault Injecting Phases, Fault Removal
Phases, and Customer Use Phases, as shown in figure 3. Because of the modular
USC-CSE 2004
6/27/2016
structure of CoQ-DES, it can be shown to be extensible as to number of phases and nonwaterfall methodologies. CoQ-DES is calibrated using the Knox Cost of Quality
categories outlined above.
M Sequential Create
Phases
C&P
N Sequential Test
Phases
Ac
Defects
*PCE (%)
At
Defects
Defects
Yes
Ic
New
Faults
Customer Use
Phase
Inject Faults
X
*PSE (%)
No
Defects
Characterize Defects
Defects
Repair Faults
Yes
It
Repair Defects
Re-inject Faults
New
Faults
System Requirements
Box Test
System/Early Integration
Code & Unit Test
* PCE = Phase Containment Effectiveness
Repair Defects
X
Deploy Repairs
Re-inject Faults
Box Requirements
Design
X
System Test
New
Faults
Re-inject Faults
Field Use
PSE = Phase Screening Effectiveness
Figure 3. Waterfall instantiation of CoQ-DES.
The “what-if” experiments using CoQ-DES are directed to per-phase fault and
cost distributions. The interplay of practitioner cost, phase duration and engineering
activity are seen as contributors to cost and savings. Many of the experiments can drive
MotoROI, such that an optimizing software development organization can use the tools
together to quantify process advances.
MotoROI
The return on investment (ROI) or yield of applying verification and validation (V&V)
to a given project consists of the following steps:
1) The Cost of Software Quality/ Cost of Poor Software Quality is calculated using the
Knox Model.
2) The costs associated with phase containment or leakage are derived from historical
data sets obtained from large industrial projects. Using this historical data, rework
costs for V&V applied at different lifecycle phases or at combinations of different
phases is calculated.
3) Based on the rework costs and industry guidelines, Potential Maximum Return from
investment in V&V is calculated.
4) The actual expected ROI is also calculated based on the investment made in V&V.
The expected return is bounded by the actual budgets and V&V scope selected.
USC-CSE 2004
6/27/2016
By considering the CMM level of the software organization, different phases of the
lifecycle as well as different combinations of V&V activities, our ROI model fulfills the
needs in a new way. A prototype tool has also been developed that takes the input factors
such as CMM Level, Software Budget, V&V Budget, V&V Scope, etc. and outputs the
expected ROI and Potential Maximum Return. An example of model output is given
below, the output restates the critical input parameters such as CMM level and project
budget information. Sample scenarios of ROI estimation and resulting COPQ savings are
shown in Table 2. Report 1 on the table presents the results of RM improvements for a
CMM level 5 organization with 50% technological effectiveness for the investment in
Report 1
CMM Level
Total SLOC
Scope of SLOC
Software Budget
Investment Budget
Investment Effectiveness
Allocated investment
Scope of Investment
Potential Maximum
Return
Potential Maximum
Return/Scope
Potential Maximum ROI
Pot
Expected Return
Expected ROI
% COPQ Savings
Report 2
Report 3
5
$9,780,700
$9,780,700
$37,209,669
$400,000
50%
$400,000
Requirements
$5,953,547
$2,232,580
Report 4
3
$9,780,700
$9,780,700
$37,209,669
$400,000
100%
$400,000
Requirements
$13,023,384
$4,837,257
3
$9,780,700
$9,780,700
$37,209,669
$400,000
75%
$400,000
Requirements
$13,023,384
$4,837,257
3
$9,780,700
$9,780,700
$37,209,669
$400,000
50%
$400,000
Requirements
$13,023,384
$4,837,257
1:1.6
$120,000
1:3.5
$520,000
1:3.5
$390,000
1:3.5
$260,000
1:0.3
1:1.3
1:.0.97
1:0.65
2%
4%
3%
2%
improvement. At high maturity levels it is difficult to achieve cost savings due to the high
efficiency level of the organization. Reports 2 through 4 present CMM level 3 results
with 100%, 75%, and 50% technology investment effectiveness ratings. Report 2 is
generated using 100% effectiveness, the ROI is positive but the likelihood of perfect
Table 2. MotoROI scenarios of technology investments to reduce COPQ and improve effectiveness.
results through technology insertion is low. Report 3 has a 1:1 ROI and reduces COPQ at
75% effectiveness; this is a highly likely outcome. Report 4 has a negative ROI at 50%
effectiveness, which could be used to bound a worst-case scenario when evaluating the
proposed technology change.
Results Summary**
This section will provide several examples from applying the models in an industrial setting
resulting in significant savings and cycle time improvements while reducing defect injection.