Software Defect Prevention
Using Orthogonal Defect
Classification
Twin-SPIN
January 6, 2005
Presented by:
Megan Graham, ASQ CSQE
megan@metagraham.net
Overview
• Defect prevention is a process used to
improve software quality
• Orthogonal Defect Classification is a tool
that characterizes defect data used in defect
analysis
2
Agenda
•
•
•
•
About Defect Prevention
ODC Concepts
Applying ODC to the DP Process
Summary
3
Agenda
•
•
•
•
About Defect Prevention
ODC Concepts
Applying ODC to the DP Process
Summary
4
Defining Defect Prevention
• What does it mean to you?
• Defect prevention is a process whose
purpose is to:
– identify the common causes of defects, and
– change the relevant process(es) to prevent that
type of defect from recurring. (SEI)
• Take what we already know and apply it to
what we think we know to produce quality
software.
5
A Bug’s Life
Requirements
Design
Code
Test
6
A Bird’s Eye View of DP
Historical
Project
Defects
Increas
e
Defects
Detecte
d
Defect
Prevention
Current
Project
Defects
Decrease
Defects
Injected
7
Applying Defect Prevention
• A defect in the software is also a defect in
the process (injection and/or detection)
• For DP to work, we need to turn software
defects into actionable process defects
Software
defects
DP
Process
defects
8
And then there was ODC…
• Orthogonal Defect Classification
• Developed at IBM in the 1990s by Ram
Chillarege
• Methodology to characterize software
defects and translate into process defects
9
Agenda
•
•
•
•
About Defect Prevention
ODC Concepts
Applying ODC to the DP Process
Summary
10
Simple ODC Classification Scheme
Type
Defect
Trigger
11
Defect Types
•
•
•
•
•
•
•
FUNCTION: Affects significant capability, end-user interfaces, product
interfaces, interface with hardware architecture or global data structure(s).
LOGIC: Affects the functionality of a code module and can be fixed by reimplementing an algorithm or local data structure without a need for
requesting a high level design change.
INTERFACE: Affects the interaction of components via macros, call
statements and/or parameters.
CHECKING: Affects program logic that would properly validate data and
values before they are stored or used in computation.
ASSIGNMENT: Requires change in a few lines of code, such as initialization
of control blocks or data structures.
TIMING/SERIALIZATION: Affects the proper management of shared and
real-time resources.
BUILD/PACKAGE/MERGE: Affects product version or configuration;
requires a formal changes to reconfigure or rebuild the product.
12
Defect Triggers
Trigger 1
Trigger 2
Fault
Failure
Trigger n
13
Review/Inspection Triggers
•
•
•
•
•
•
•
•
•
DESIGN CONFORMANCE: Comparing the implemented design against a
reference –design document, pattern, or guideline.
LOGIC/FLOW: Checking for correctness or flaws using knowledge of the
practice.
BACKWARD COMPATIBILITY: Examining compatibility with prior version
of the product.
LATERAL COMPATIBILITY: Examining for compatibility with other
products and platforms that need to work with this release.
CONCURRENCY: Serialization, shared resources, multi-threaded tasks,
timing, etc.
INTERNAL DOCUMENT: Inconsistencies in prologs, and sections in the
same work product.
LANGUAGE DEPENDENCY: Programming standards, specific
implementation considerations, environment restrictions, execution modes,
etc.
SIDE EFFECTS: Usage behavior beyond design, but relevant in context. Do
A; B happens.
RARE SITUATION: Unusual issues related to idiosyncrasy of environment,
14
hardware, or software.
Function Test Triggers
•
•
•
•
•
•
SIMPLE PATH: White box – Straightforward usage of code path and
branches.
COMPLEX PATH: White box – Exercising conditionals, and circuitous
coverage.
COVERAGE: Black box – Straightforward usage of function or single
parameterized execution.
VARIATION: Black box – Straightforward like coverage, but with a variety of
parameters.
SEQUENCING: Black box – Multiple functions, with a specific sequence
(order is important).
INTERACTION: Black box – When two or more bodies of code are involved
(order is not important).
15
System Test Triggers
•
•
•
•
•
•
WORKLOAD STRESS: Pushing the limits of performance, resources, users,
queues, traffic, etc.
RECOVERY: Invoke exception handling, recovery, termination, error
percolation, etc.
STARTUP/RESTART: Major events of turning on, off, or changing the degree
of service availability.
HARDWARE CONFIGURATION: Issues surfaced as a consequence of
changes in hardware setup.
SOFTWARE CONFIGURATION: Issues surfaced as a consequence of
changes in software setup.
BLOCKED TEST/NORMAL MODE: Test could not run during System Test,
or customer found nonspecific trigger. Look for additional trigger.
16
ODC v5.11 Classification Scheme
Trigger
Activity
Impact
Opener
Defect
Closer
Age
Target
Type
Source
Qualifier
Source: ODC v5.11, IBM Center for Software Engineering, www.research.ibm.com/softeng 17
Agenda
•
•
•
•
About Defect Prevention
ODC Concepts
Applying ODC to the DP Process
Summary
18
When to Apply DP
• DP can be applied to one or more phases of
the software lifecycle
Analysis
Analysis
Analysis
Analysis
TEST
Action
CODE
Action
DSGN
Action
RQTS
Action
Action
Analysis
Defect Prevention
Field
• Depends on maturity, goals, etc.
19
Defect Analysis
Orthogonal Defect
Classification
What are the attributes of the
defects?
Defect Causal
Analysis
Defect Trigger
Analysis
When are defects
being injected?
How are defects
being detected?
ODC provides a structure for the defect data
20
Defect Causal Analysis
• Purpose: Characterize process issues that
lead to injection of defects.
Step 1: ID a set of defects.
Step 2: Identify Defect Types with team of experts.
Step 3: Plot the distribution of Defect Types.
Step 4: Map Defect Types back to development activity.
Step 5: Develop action plan to address process deficiencies.
Step 6: Monitor process to ensure changes were effective.
21
Defect Type Mapped to Phase
Function
Architecture/HLD
Interface
H/L Level Design
Logic
Checking
Low Level Design
LLD/Implementation
Assignment
Implementation
Timing/Serialization
Implementation
Build/Package/Merge
Configuration Mgmt
22
Sample Distribution
100%
80%
60%
40%
20%
0%
er
t
In
ce
fa
B/
M
P/
L
ic
g
o
in
k
c
he
C
g
ig
ss
A
t
en
nm
g/
n
i
im
a
ri
e
S
l
F
tio
c
un
n
T
23
Defect Trigger Analysis
• Purpose: Characterize process issues that
allowed defects to escape to later phases.
Step 1: ID a set of defects.
Step 2: Identify Defect Trigger with team of experts.
Step 3: Plot distributions: Defect Trigger by Family, Review Triggers,
Function Test Triggers, System Test Triggers.
Step 4: Map Defect Trigger family back to detection activity(ies).
Step 5: Develop action plan to increase missed Defect Triggers.
Step 6: Monitor process to ensure changes were effective.
24
Defect Trigger Family Mapped to
Phase
Review/Inspection
Function Test
System Test
Peer Reviews or
Inspections
Testing of specified
scenarios
Testing in real-world
scenarios or extreme
scenarios
25
Sample Distributions
Defect Triggers by Family
60
50
40
30
20
10
0
System Test
Function Test Triggers
Complex
Path
Simple
Path
Coverage
Interaction
18
16
14
12
10
8
6
4
2
0
Variation
Function Test
Sequencing
Review/Inspection
26
Additional Use for DTA
• Purpose: Determine quality status.
Step 1: Plot distribution of Defect Triggers after each detection activity.
Step 2: Compare to historical defect profile to determine if profile is as
expected.
Step 3: Develop action plan, if necessary, to get back on track.
• If your project thinks it’s one phase, but the distribution is
of an earlier phase, then you’re really in the earlier phase!
27
Agenda
•
•
•
•
About Defect Prevention
ODC Concepts
Applying ODC to the DP Process
Summary
28
The Big Picture: DP & ODC
• Mission: Improve software quality by using
readily available data to decrease defects injected
and increase defects detected.
Analysis
Analysis
Analysis
Analysis
TEST
Action
CODE
Action
DSGN
Action
RQTS
Action
Action
Analysis
Defect Prevention
Field
ODC Classification of Software Defects
29
Important Points about ODC
• A defect in the software is a defect in the process.
• Implementing ODC is very cost-effective
– Enhances data already collected (software defects)
– Adding fields that are completed real-time make data
collection virtually free!
– Tooled to quickly identify process defects (mapping)
• ODC can be implemented in stages
– Start with field defects, then move to in-process
analysis
– Utilize defect profiling in-process to predict quality and
project status
• Fields can be tailored to your own organization
30
References
• Handbook of Software Reliability Engineering
(Michael R. Lyu, Editor; IEEE Computer Society
Press/McGraw-Hill)
• Ram Chillarege (www.chillarege.com)
• IBM Research Center for Software Engineering
(www.research.ibm.com/softeng)
• Soheil Khajenoori, SIAGroup
(soheil@siagroup.us)
31