PRESENTED BY CATHERINE OEHL
COQAA Monthly Meeting
December 5, 2012

• What is Quality and Why is it important
• What is Total Cost of Quality (TCoQ)
• What is the purpose and value of measuring TCoQ
• How to measure and reduce TCoQ
• Hypothetical Scenario applying TCoQ
• Q&A
12/5/2012 2

• Definition: A measure of excellence or a state of being free from defects, deficiencies and significant variations. It is brought about by strict and consistent commitment to certain standards that achieve uniformity of a product in order to satisfy specific customer or user requirements.
• ISO 8402-1986 standard defines quality as "the totality of features and characteristics of a product or service that bears its ability to satisfy stated or implied needs."
• If an automobile company finds a defect in one of their cars and makes a product recall, customer reliability and therefore production will decrease because trust will be lost in the car's quality.
Source: http://www.businessdictionary.com/definition/quality.html
12/5/2012 3

12/5/2012 4

o Current projects spend 40-50% of the effort on avoidable rework o About 80% of avoidable rework comes from 20% of the defects o About 80% of defects comes from 20% of the modules and about half of the modules are defects free o 90% of the downtime comes from at most 10% of the defects o Peer Reviews catch 60% of the defects o Perspective-based reviews catch 35% more defects than non directed reviews o Disciplined personal practices can reduce defect introduction rates by up to 75% o While several organizations measure and track application total cost of ownership (TCO), very few measure the total cost of quality
12/5/2012 5

• Total Cost of Quality (TCoQ) consists of the costs associated with preventing, assessing, finding and correcting defective work.
• Note that the TCoQ analysis looks only at our
(producers) costs and not the customer’s cost due to poor quality.
12/5/2012 6

Prevention cost + Inspection cost + Internal Failure cost + External Failure cost
 Prevention costs arise from efforts to keep errors or defects from occurring at all

Inspection costs arise from detecting errors/defects via inspections, tests, audits

Internal failure costs arise from defects caught internally and dealt with by discarding or repairing the defective items
 External failure costs arise from defects that actually reach customers. Often other costs are incurred in addition to correcting the failure.
12/5/2012 7

Addresses and measures the following questions:
• What is the state of quality of your applications?
• How does quality impact your business?
• How is quality measured?
• When are errors/defects introduced? Why?
• When are errors/defects corrected? What are the costs?
• What are the processes that impact quality?
• How do you quantify the effectiveness of each process?
• How much do failures cost?
• What is the root cause of the failure?
12/5/2012 8

Requirements
56%
Design
27%
Unknown
Coding
7%
10%
Source: Writing Testable Requirements, Dick Bender
12/5/2012
Source: NIST Software Quality Study
9

Source: NIST Software Quality Study
10 12/5/2012

Source: B. Boehm and V. Basili, “Software Defect Reduction Top 10 List”, IEEE Computer
12/5/2012 11

Prevention /
Reviews
Appraisal /
Inspection
Failures /
Reaction
Greatest
Leverage
Good
Leverage
Least
Leverage
Source: http://en.wikipedia.org/wiki/Quality_costs
12/5/2012
Cost Savings
12

GOAL: Prevent defects before they happen
(Work done right the first time)
Recommended Activities:
– Formal Requirements Gathering Processes
– Requirements Analysis and Management
– Early Prototyping
– Usability Analysis
– Static Testing
– Peer Reviews
– Release and Configuration Management Processes
– Standards for Test Environments
– Quality Training
– Accurate Internal Communication
12/5/2012 13

GOAL: Detect the Errors (Find them early)
Recommended Activities:
– Code inspections
– Design reviews
– Glass box and black box testing
– Test Automation
– Usability Testing
– Pre-release / User Acceptance Testing
– Structured Personal reviews focused on verification and validation
12/05/2012 14

GOAL: Report and fix defects before production release (Fix and Re-Test)
Incurred Activities:
• Diagnose Defects
• Defect Fixes
• Defect Management
• Re-Test and Regression Testing
• Rework code and recompile
• Wait states
12/5/2012 15

GOAL: Reduce and remediate impact to customers
(Apologize and recover)
Incurred Activities:
• Technical Support Calls
• Recalls
• Customer Service
• Scrap
• Overtime
• Regulatory / Legal Costs
12/5/2012 16

• Suppose we have a software product that has one production release per quarter.
• On average each release contains 1000 must-fix defects that will be identified and repaired.
• We will assume developers catch 25% of defects during unit and integration testing at a cost of $10/defect
12/5/2012 17

Non Formal
Testing Inspection Testing Costs
Staff
Infrastructure
Tools
Total Investment
$0
$0
$0
$0
Development
Must-Fix Defects Found
Fix Cost (Internal Failure - $10/defect)
Testing (Pre-Release)
Must-Fix Defects Found
Fix Cost (Internal Failure - $100/defect)
250
$2,500
0
$0
Customer Support (Production)
Must-Fix Defects Reported
Fix Cost (External Failure - $1000/defect)
Cost of Quality
Conformance (Inspection Costs)
NonConformance (Internal Failures + External
Failures)
TCoQ
ROI
12/5/2012
750
$750,000
$0
$752,500
$752,500
N/A
18

Inspection Testing Costs
Staff
Infrastructure
Tools
Total Investment
Non Formal
Testing
$0
Manual
Testing
$60,000
$0
$0
$0
$10,000
$0
$70,000
Development
Must-Fix Defects Found
Fix Cost (Internal Failure - $10/defect)
Testing (Pre-Release)
Must-Fix Defects Found
Fix Cost (Internal Failure - $100/defect)
250
$2,500
0
$0
250
$2,500
350
$35,000
Customer Support (Production)
Must-Fix Defects Reported
Fix Cost (External Failure - $1000/defect)
Cost of Quality
Conformance (Inspection Costs)
NonConformance (Internal Failures + External
Failures)
TCoQ
ROI
12/5/2012
750
$750,000
$0
$752,500
$752,500
N/A
400
$400,000
$70,000
$437,500
$507,500
350%
19

Inspection Testing Costs
Staff
Infrastructure
Tools
Total Investment
Development
Must-Fix Defects Found
Fix Cost (Internal Failure - $10/defect)
Testing (Pre-Release)
Must-Fix Defects Found
Fix Cost (Internal Failure - $100/defect)
Non Formal
Testing
$0
$0
Manual
Testing
$60,000
$10,000
Automated
Testing
$60,000
$10,000
$0
$0
$0
$70,000
$12,500
$82,500
250
$2,500
0
$0
250
$2,500
350
$35,000
250
$2,500
500
$50,000
What if we added some simple
Prevention activities like
Static Reviews,
Peer Reviews,
Change
Management ?
Customer Support (Production)
Must-Fix Defects Reported
Fix Cost (External Failure - $1000/defect)
Cost of Quality
Conformance (Inspection Costs)
NonConformance (Internal Failures +
External Failures)
TCoQ
ROI
12/5/2012
750 400
$750,000 $400,000
$0 $70,000
$752,500 $437,500
$752,500 $507,500
N/A 350%
250
$250,000
$82,500
$302,500
$385,000
445%
20

Prevention Costs
Static Reviews (10 people every 2 weeks reviewing 5 sets of documents x $75/hr)
Peer Reviews (2 developers every 2 weeks reviewing 10 modules x $75)
Change Management (10% Configuration Mgr & 10% Requirements Lead & 10% Test Lead)
Inspection Testing Costs
Staff
Infrastructure
Tools
Total Investment (Prevention & Inspection costs)
Development
Must-Fix Defects Found
Fix Cost (Internal Failure - $10/defect)
Testing (Pre-Release)
Must-Fix Defects Found
Fix Cost (Internal Failure - $100/defect)
Customer Support (Production)
Must-Fix Defects Reported
Fix Cost (External Failure - $1000/defect)
Cost of Quality
Conformance (Inspection Costs)
NonConformance (Internal Failures + External Failures)
TCoQ
ROI
Non Formal Testing Manual Testing Automated Testing
$0 $60,000 $60,000
$0
$0
$0
$10,000
$0
$70,000
$10,000
$12,500
$82,500
250
$2,500
0
$0
750
$750,000
$0
$752,500
$752,500
N/A
250
$2,500
350
$35,000
400
$400,000
$70,000
$437,500
$507,500
350%
250
$2,500
500
$50,000
250
$250,000
$82,500
$302,500
$385,000
445%
Resource Costs
$97,500
$39,000
$46,800
$265,800
500
$5,000
350
$35,000
50
$50,000
$265,800
$90,000
$355,800
57%
12/5/2012 21

• Quality creates value and reduces risk exposure
• The “cost of quality” isn’t the price of creating a quality product or service. It’s the cost of NOT creating a quality product or service.
• Errors / defects are often introduced early and typically found later
• Total Cost of Quality (TCoQ) consists of the costs associated with preventing, assessing, finding and correcting defective work.
• The central theme of quality improvement is that larger investments in prevention drive even larger savings in quality-related failures and
Inspection efforts.
• Any Quality improvement effort should use TCoQ to quantify the effectiveness of the process.
12/5/2012 22

Catherine Oehl oehlc@yahoo.com
614-440-6477
LinkedIn Profile: http://lnkd.in/u8kcyy
12/5/2012 23

 Software Defect Reduction Top 10 List by Barry Boehm and Victor Basili
 Using Cost Benefit Analysis to Develop Software Process Improvement
Strategies, Data Analysis Center of Software
 Traditional P-A-F method by Juran (1951) and Feigenbaum (1956) –
Prevention Appraisal and Failure costs
 Phil Crosby’s Model from the book Quality is Free (1979) – conformance to requirements – cost of conformance (done Right) and non-conformance
(done Wrong)
 Process Cost Model by Ross (1977)
 http://asq.org/learn-about-quality/cost-of-quality/overview/overview.html
 http://www.authorstream.com/Presentation/Hassanasif-493959-cost-ofquality
 http://www.bexcellence.org/Cost-Of-Quality.html
 http://www.isixsigma.com/implementation/financial-analysis/cost-quality-notonly-failure-costs/
 http://www.projectconnections.com/articles/051005-koch.html
12/5/2012 24

• Process (Productivity & Prevention)
– Effectiveness of Test Processes
• Release Mgt, Configuration Mgt
– Efficiency of Test Processes
• Defect Mgt, Test Environment Mgt, Test Data Mgt
– Consistency of Test Activities
• Test Planning, Test Estimation, Test Execution
• Schedule
– Testing Planning
• Duration of lifecycle phases
• Duration of Test Phase/Stage
– Test Estimating/Projections
• Planning Estimate
• Component Based Estimates
12/5/2012 25

• Failures
– Volume of Defects
• Open, Closed, New, Test Phase/Stage Contained,
Defect Aging (# of days open)
– Severity of Defects
• Criticals, Highs, Mediums, Lows, Problem Areas,
Low to all reported ratio
12/5/2012 26

• Build Test Mgt Costs (P+I+IF)
– Manual Test Hour Costs
– Automated Test Hour Costs
– Test Hour Costs per Test Phase/Stage
– % of Total Build Costs
• Breakfix/Maintenance Costs (EF)
12/5/2012 27

12/5/2012 28