Lab _003 --- Process Improvement
SE 3730 / CS 5730 – Software Quality
Due: Oct 18 NOON
This exercise makes use of your team’s knowledge of metrics and process to optimize a quality
assurance process. The objective is to use this knowledge to improve a software processes. Part of
this lab is a list of defects, see the link to Lab 003: Defect Data to Analyze. The defect list contains
information about Injection Points (Col C).
Definitions:
Injection point: The lifecycle stage in which a defect was introduced into a system.
Detection point: The lifecycle stage in which a defect was discovered.
Standard Cost: The organization’s average cost to do something – in this case it includes the cost of
performing a quality assurance technique or the cost of fixing a defect at several lifecycle stages.
Total cost of finding and fixing a defect: The sum of detection and fixing costs.
Here is how this works:
1. Use appropriate metrics to compute and analyze the total cost of finding and fixing all defects.
a. See Table 1 below for the standard costs for fixing a defect based on life-cycle detection
point. We cannot fix a defect before it is injected and before it is detected.
b. See Table 2 below for:
i. The cost of running a particular quality assurance technique (like requirements
review, code review, unit testing, …)
ii. The probability of finding a defect using a particular quality assurance technique.
Note that performing a quality assurance technique will NOT catch all defects that
are in the system, only a certain percentage of them. Also, it can only detect defects
that have already been injected into a system.
2. Example: It cost 100 pieces of gold to perform a requirements review for this hypothetical project
and the review will catch 50% of defects (table 2). If there were 10 defects to start with, the
requirements review would find 5 of them that could be fixed (leaving 5 unfound). We could run
another requirements review using another team of independent reviewers and it would find 50%
of residual 5 defects or 2 (we round down to the nearest whole defect).
Table 1: Cost of Fixing Defect by Detection Point
Detection Point
Cost Fixing a Defect
(in pieces of gold)
Req. Review
50
Design Review
100
Unit Test
200
Code Review
210
Integration Test
300
System Test
400
Alpha Test
800
Beta Test
1000
Customer Report
4000
©2011 Mike Rowe
Page 1
Stage in which we can run these
(see Col C of Excel Spreadsheet)
Rqmts
Design
Code
Code
Integration
SystemTest
SystemTest
SystemTest
SystemTest
02/10/2016 1:20 AM
Lab _003 --- Process Improvement
SE 3730 / CS 5730 – Software Quality
Table 2: Cost of Quality Assurance Techniques and Probability of finding latent defect
Quality Assurance
Can find defects
Cost of Running Quality
Probability of finding
Technique
injected in
Assurance Technique (in
defect
pieces of gold)
Req Review
Rqmts
100
.50
Design Review
Rqmts, Design
150
.40
Unit Test
Rqmts, Design, Code
350
.30
Code Review (prior Rqmts, Design, Code
300
.60
to Integration test)
Integration Test
Rqmts, Design, Code,
250
.40
Integration
System Test
Rqmts, Design, Code,
400
.30
Integration, System Test,
Alpha Test
Rqmts, Design, Code,
800
.40
Integration, System Test,
Documentation
Beta Test
Rqmts, Design, Code,
1000
.50
Integration, System Test,
Documentation
Customer finds
Rqmts, Design, Code,
0
1.00 This means that
after release
Integration, System Test,
the customer will
Documentation
ALWAYS find all defects
left in the product.
DELIVERABLES:
0. Use a Microsoft Excel compatible spreadsheet to show your calculations and formulas. See the
REQUIRED format on the last page of this document.
1. Cost of the initial process, assuming each quality assurance technique is performed once and only
once. Note: this will be easy –
a. Find the number of defects injected at each point.
b. Run the each quality assurance technique once at that point in the lifecycle to determine
how many defects were found, and thus could be fixed
c. Determine the cost of running that quality assurance technique. Note: Techniques must be
run in the order given in Table 1
d. Determine the cost of fixing the discovered defects
e. Compute how many defects were not found at each detection point (residue defects)
f. Add the residual defects to the defects injected at the next life-cycle
g. And repeat a-f until all quality assurance techniques have been used at their appropriate
lifecycle stage. Assume that each technique was run only once.
©2011 Mike Rowe
Page 2
02/10/2016 1:20 AM
Lab _003 --- Process Improvement
SE 3730 / CS 5730 – Software Quality
2. Now for the fun part! Engineer a new quality assurance process consisting of up to three iterations
of each quality assurance technique that minimizes the total cost of quality.
 For example:
o Assume we have 9 requirements defects.
o The First Requirements Review cost 100 pieces of gold to run and will catch 50% of the
defects or 4 (remember we round down assuming we must find all of a defect to be
useful).
o After fixing the requirements’ defects identified in the initial review which cost 50 pieces
of gold per defect or 200; we can do another (Second) Requirements Review at a cost of
100 more pieces of gold.
o This Second review will find 50% of the remaining (5 * 50% =2.5 OR 2 defects – rounding
down ) resulting in 3 remaining defect.
o We can then do this a third time costing another 100 pieces of gold to perform which
removes 1 more defect. Leaving 2 remaining defects.
o Running the second or third requirements review may or may not be cost effective. It
may be cheaper to move onto the next quality assurance technique, Design Review.
o Note: a quality assurance technique cannot find defects that have not yet been
injected. For instance, a Requirements Review cannot find defects that will be injected
in the Code Phase. See the second column from the left in Table 2 for guidance.
 The goal of this exercise is to engineer our quality assurance process, so as to minimize the
Total Cost of Quality (as defined by: total cost of fixing defects plus performing the QA
techniques).
 Use the defect data in the spreadsheet to determine a strategy for minimizing our cost. Use
the Table on the last page of this document to display your calculations and results in an Excel
spreadsheet.
3. Find the Cost of Detecting and Fixing per Defect for each iteration. For example on third Code
Review iteration, if 3 defects were discovered and fixed, we would add the cost of running a Code
Review to the cost of fixing three code defects and then divide this quantity by 3 defects. HINT:
this will help you answer the question of whether or not to perform the next iteration of a quality
assurance technique.
4. Find the difference between the original process (Deliverable #1) and your new optimized
processes Total Cost (Deliverable #2).
5. Produce a lab report describing what your team did including a discussion of why you did what you
did. Time worked by each individuals and a summary of how long (person-minutes) each task
took. Copy the grading rubric to the top of your lab report. Email the lab report and your Excel
compatible spreadsheet to me by the due time.
©2011 Mike Rowe
Page 3
02/10/2016 1:20 AM
Lab _003 --- Process Improvement
SE 3730 / CS 5730 – Software Quality
GRADING CRITERIA 0-22 POINTS
Task
Points
Total Cost of Defects
Deliverable 1
Improved QA
 Devise a better QA
strategy
 For each iteration
calculate the cost of
detecting and fixing
per defect.
 Calculation of the
total cost fixing all
defects
Calculation of the
difference between the
original Total Cost and the
new Total Cost.
Lab Report and following all
instructions.
TOTAL
©2011 Mike Rowe
Continuum of Earned Points / Comments:
0% - no supported and very wrong answer
50% - weakly supported process and an
answer based on process – errors in using
data.
100% - strongly support and valid process,
reasonable answer.
2
10
2
2
2
4
22
Page 4
02/10/2016 1:20 AM
Lab _003 --- Process Improvement
SE 3730 / CS 5730 – Software Quality
Required Excel Spreadsheet table format.
A
# Defects
Injected
this
iteration
B
Current
Lifecycle
Stage
C
Defects
available
to fix
now
Nn
Rqmts
nnn
A:
B:
C:
D:
E:
F:
G:
H:
I:
J:
K:
L:
D
E
Per
Defect
cost2fix
QA
Tech
Req.
Review
F
G
H
I
J
K
L
(Tbl 1)
Type of Defect
Review Can
Find
QA
Cost
(Tbl 2)
Probability
of Detection
(Tbl 2)
Fixed
Defects
Residual
Defects
Cost:
Fix +
QA
Cost
per
defect
50
Rqmts
100
0.5
iiii
jjj
kkkkk
llllll
The first iteration of a lifecycle stage this is the total defects injected at this point. For subsequent iterations of a technique this will be 0.
The current lifecycle stage
Col A plus col K of previous row
The quality assurance technique being run
Cost to fix each defect discovered at this point from table 1
List of the Injection points that this technique can find defects in. See table 2.
Cost of running the current quality assurance technique. See table 2.
Probability of finding a defect using this quality assurance technique. See table 2.
Defects found by this quality assurance technique. This is the floor of col C multiplied by col H.
Defects not found. This is col C minus col I.
Col E * col I + col G
Col l / col K
©2011 Mike Rowe
Page 5
02/10/2016 1:20 AM
Lab _003 --- Process Improvement
SE 3730 / CS 5730 – Software Quality
©2011 Mike Rowe
Page 6
02/10/2016 1:20 AM