a stitch in time saves nine Economics of defect prevention
2
a stitch in time saves nine – Economics of defect prevention
Abstract
Do you think ‘high quality’ means more money and more time? Many
organizations think so.
In a way it is true for the simple fact that one half of most software
development project budgets and two thirds of the typical development
team’s time are spent fixing poor quality or software defects. A study
conducted by National Institute of Standards and Technology (NIST)
reports that software defects cost the U.S. economy $59.5 billion annually.
Caper Jones in his 2001 IEEE paper, “Measuring Software Process
Improvement,” gave a hope to software executives around the world by
revealing that software process improvement leads to 80% reduction in
post-release defects and a 65% increase in productivity, while reducing
project schedules and costs by 20%.
This paper highlights the need for an early detection of defects, its impact
on the business & also emphasizes on the cost of fixing a defect.
Introduction - Cost of Poor Quality is nothing
but “Defects”:
An important strategy of any IT organization is to deliver defect free or
quality products. But developing the quality product comes with a price
tag.
Dr. Joseph Juran, the “father of the quality” developed the “Cost of
Quality” concept (figure1) – a measure that takes into account all the costs
associated with good and poor quality. In essence the cost of quality
represents the costs to achieve good quality as well as the costs incurred
due to poor quality.
Figure 1
Cost of poor quality attributes to internal failure costs and external
failure costs.
?
Internal failure costs - the cost incurred due to the defects, before
shipping the products to production or live environment. This is preproduction cost.
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
3
a stitch in time saves nine – Economics of defect prevention
?
External failure costs - the cost incurred due to the defects, after
shipping the products to production or live environment. This is postproduction cost.
It’s important to correctly classify quality costs to determine how much
poor quality is costing your organization.
Not all defects in the software development are caused by coding errors.
Some examples of defects coming upstream in the software development
lifecycle (SDLC) are defects due to misunderstanding of requirements,
contradicting requirements, missing requirements, wrong analysis of
requirements, incomplete requirements and more importantly unable to
identify non-functional requirements such as testability, scalability,
maintainability, usability, performance etc.
A programmer makes a mistake, which results in a defect in the software
source code. If this defect is executed, in certain situations the system will
produce false results, causing a failure. However all defects will not
necessarily result in failures. For example, defects in dead code will never
result in failures. A defect can turn into a failure when the environment is
changed. For example, software being run on a new hardware platform,
changes in source data or interacting with different software. A single
defect may result in a wide range of failure symptoms.
Why defects need to be detected and
fixed early? :
What industry trends on defects say:
Industry trends indicate that though 56% of the defects are injected in
requirements phase, most of these defects are actually detected in
integration / system testing phase only. The longer the defect stays in the
program, the more likely to have additional code written around it, the
more re-testing to be done and incurring additional cost to fix. So it is
evident that it is always less expensive to detect and fix the defects early in
SDLC than fixing in the production.
Industry trends on defects injection
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
4
a stitch in time saves nine – Economics of defect prevention
Industry trends on defects detection
Figure 2: Source: Extracts from chapter 5 from the National Institute of
Standards and Technology (NIST) Report on “The Economic Impacts of
Inadequate Infrastructure for Software Testing”.
Relative cost of defects & its impact on the business:
The concept of cost of fixing a defect is not well understood.
Figure 3
This cost feeds back into SDLC and many managers do not actually know
what it costs their organizations to fix the defects.
An estimate by Boehm & Basili suggests that the cost of fixing the defect
grows exponentially for each downstream phase of the software
development lifecycle in which it remains undetected. The fixing of the
defect after product delivery can be 100 times more costly than doing so in
the requirements and design phases.
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
5
a stitch in time saves nine – Economics of defect prevention
Table 1 also shows the cost of fixing the defect depending on the SDLC phase & the time it is detected. If
the defect in the requirements is detected post-release, then it would cost 10–100 times more to fix than if the
defect had already been detected in the requirements phase.
Cost to Fix a Defect
Time Detected
Requirements Architecture Construction System testing Post-release
Time
Introduced
Requirements
1×
3×
5–10×
10×
10–100×
Architecture
-
1×
10×
15×
25–100×
Construction
-
-
1×
10×
10–25×
Table 1
Table 2 indicates some real time impacts of software defects on the
business:
Business
Cost per Hour Downtime
Automated teller machines (medium –sized bank)
$14,500
Package shipping service
$28,250
Telephone ticket sales
$69,000
Catalog sales center
$90,000
Airline reservation center (small airline)
$89,500
Table 2: Source :Information Technology Project Management, Fourth Edition
Calculation of Cost of Fixing a Defect:
Formula:
Average cost
(Number of resources X Duration to fix the defects)
to fix a defect = --------------------------------------------------------- X Cost per person-day
(Number of defects fixed)
Below you would find two hypothetical cases of a project that will help
you understand how to determine the cost of fixing a defect & its impact
in your project.
Project X is developing a financial software product and below are the
project details:
Duration
11 months
Team Size
10 FTE
Scope
Requirements, Design, Construction, Unit Testing and System Testing
Billing Rate
$500 per person-day
Table 3
Project has spent initial 10 months (as per estimated schedule) in
developing the product. Now the project
is in the system testing phase.
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
6
a stitch in time saves nine – Economics of defect prevention
Before: “All defects are detected in system testing”
The project has detected 300 defects in system testing phase and zero
defects in previous phases. Refer figure 4.
Figure 4
Before Scenario
Cost of fixing the defects in system testing:
# of resources deployed
10
# of valid defects detected
300
# of defects fixed
300
Duration to fix the defects
60 days
Effort to fix the defects
10*60=
Bef
ore
Sce
nar
io
600 Person days
Defect fixing rate
2 person days per fix
Billing rate
$500 per person-day
Average cost to fix a defect in
system testing
500*2=$1000 per fix
Table 4
System testing defects cost: 1000*300 = $300,000. This is preproduction cost.
During post production stage, Project X has spent on an average 5
person-days per fix. At $500 per person-day, the cost per fix is $2500.
Project X has released 50 fixes, Total post-release fix cost = $125,000. This
is post- production cost.
Total cost incurred for fixes or defects (pre production +post
production) = $425,000 + Indirect cost*.
(Indirect Cost* - system unavailability, service loss, environment setups,
technology complexity, project management effort, QA effort etc…).
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
7
a stitch in time saves nine – Economics of defect prevention
After: “Defects detected & fixed throughout SDLC”
The project has detected and fixed 180 defects in the previous phases and
remaining 120 defects are detected in system testing. Refer figure 5.
Figure 5
After Scenario
Cost of fixing the defects in system testing:
# of resources deployed
10
# of valid defects detected
120
# of defects fixed
120
Duration to fix the defects
20 days
Effort to fix the defects
10*20=
200 Person days
Defect fixing rate
1.6 person days per fix
Billing rate
$500 per person-day
Average cost to fix a defect in
system testing
500*1.6=$800 per fix
Table 5
System test defects cost: 800*120 = $96,000 +RDC Cost*.
(RDC Cost* - Requirements, Design, Construction & Unit Testing (RDC)
defects cost should also be calculated same as above table5).
For the calculation purpose, let’s assume the RDC cost incurred is 30%
system testing defects i.e $28,800.
System Test defects cost: $124,800. This is pre-production cost. Post
production cost is ‘NIL’.
Total cost incurred for fixes or defects (pre production +post
production) = $124,800.
In the ‘after’ scenario, project X has saved over $300K in addition
to saving associated with indirect costs. Other benefits include
cycle time reduction, achieving project schedule, better
productivity and customer confidence.
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
8
a stitch in time saves nine – Economics of defect prevention
Conclusion
It takes courage and commitment to make a ‘Zero defect’ policy. But with
focused effort one can certainly commit to achieving minimal defects and
reduce the cost associated with them. Product managers can absolve
themselves from the dilemma of whether we need to do one more round
of testing. Defect prevention simplifies the test cycles, reduces overall
cost and improves product quality.
About the Author:
SUDHEER CHIKKERUR is a senior process consultant at HCL
Technologies Limited.
HCL’s IV&VS - Quality Consulting Group works with client
organizations on software process improvements to improve product
quality while reducing operational cost. For more information contact us
at process.consulting@hcl.com
References:
Boehm B, Basili VR. Software defect reduction top 10 list. IEEE
Computer. January, 2001.
Principles of Quality Costs: Principles, Implementation, and Use,Third
Edition, ed. Jack Campanella, ASQ
What Does It Cost to Fix a Defect? By Johanna Rothman
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
9
a stitch in time saves nine – Economics of defect prevention
ABOUT HCL
HCL Technologies
HCL Technologies is a leading global IT services company, working with
clients in the areas that impact and redefine the core of their businesses.
Since its inception into the global landscape after its IPO in 1999, HCL
focuses on ‘transformational outsourcing’, underlined by innovation and
value creation, and offers an integrated portfolio of services including
software led IT solutions, remote infrastructure management,
engineering and R&D services and BPO. HCL leverages its extensive
global offshore infrastructure and network of offices in 31 countries to
provide holistic, multi-service delivery in key industry verticals including
Financial Services, Manufacturing, Consumer Services, Public Services
and Healthcare. HCL takes pride in its philosophy of ‘Employees First’
which empowers our 72,267 transformers to create real value for
customers. HCL Technologies, along with its subsidiaries, had
consolidated revenues of US$ 3.1 billion (Rs. 14,101 crores), as on 31st
December 2010 (on LTM basis). For more information, please visit
www.hcltech.com
About HCL Enterprise
HCL is a $5.7 billion leading global technology and IT enterprise
comprising two companies listed in India - HCL Technologies and HCL
Infosystems. Founded in 1976, HCL is one of India's original IT garage
start-ups. A pioneer of modern computing, HCL is a global
transformational enterprise today. Its range of offerings includes product
engineering, custom & package applications, BPO, IT infrastructure
services, IT hardware, systems integration, and distribution of
information and communications technology (ICT) products across a
wide range of focused industry verticals. The HCL team consists of over
79,000 professionals of diverse nationalities, who operate from 31
countries including over 500 points of presence in India. HCL has
partnerships with several leading Global 1000 firms, including leading IT
and Technology firms. For more information, please visit www.hcl.com
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.