COQUALMO and Orthogonal
Defect Classification(ODC)
Keun Lee
( keunlee@sunset.usc.edu)
&
Sunita Chulani
(Sunita_Chulani@us.ibm.com)
Feb 6. 2001
© USC-CSE
1
COQUALMO and Orthogonal
Defect Classification(ODC)
• Current COQUALMO Model
- Results and Challenges
• COQUALMO – ODC Research Approach
• Example
• Results
• Issues and Research Plans
Feb 6. 2001
© USC-CSE
2
Current COQUALMO System
COCOMO II
COQUALMO
Software Size Estimate
Software platform,
Project, product and
personnel attributes
Software development
effort, cost and schedule
estimate
Defect
Introduction
Model
Number of residual defects
Defect density per unit of size
Defect removal profile
levels
Automation, Reviews,
Testing
Feb 6. 2001
Defect
Removal
Model
© USC-CSE
3
Partion of COQUALMO Rating
Scale
COCOMO II p.263
Very Low
Low
Nominal
High
Very High
Extra High
Automated
Analysis
Simple
compiler
syntax
checking
Basic
compiler
capabilities
Compiler
extension
Basic req. and
design
consistency
Intermediatelevel module
Simple
req./design
More
elaborate
req./design
Basic distprocessing
Formalized
specification,
verification.
Advanced
distprocessing
Peer Reviews
No peer
review
Ad-hoc
informal
walk-through
Well-defined
preparation,
review,
minimal
follow-up
Formal review
roles and
Well-trained
people and
basic checklist
Root cause
analysis,
formal follow
Using
historical data
Extensive
review
checklist
Statistical
control
Execution
Testing and
Tools
No testing
Ad-hoc test
and debug
Basic test
Test criteria
based on
checklist
Well-defined
test seq. and
basic test
coverage tool
system
More advance
test tools,
preparation.
Distmonitoring
Highly
advanced
tools, modelbased test
Feb 6. 2001
© USC-CSE
4
COQUALMO Defect Removal
Estimates
- Nominal Defect Introduction Rates
70
60
60
50
40
Delivered Defects
/ KSLOC
30
28.5
20
14.3
10
7.5
0
VL
Low
Nom
High
3.5
VH
1.6
XH
Composite Defect Removal Rating
Feb 6. 2001
© USC-CSE
5
Multiplicative Defect Removal
Model
- Example : Code Defects; High Ratings
• Analysis : 0.7 of defects remaining
• Reviews : 0.4 of defects remaining
• Testing : 0.31 of defects remaining
• Together : (0.7)(0.4)(0.31) = 0.09 of defects remaining
• How valid is this?
- All catch same defects : 0.31 of defects remaining
- Mostly catch different defects : ~0.01 of defects remaining
Feb 6. 2001
© USC-CSE
6
Example UMD-USC CeBASE Data
Comparisons
“Under specified conditions, …”
• Peer reviews are more effective than functional testing for faults of
omission and incorrect specification(UMD, USC)
• Functional testing is more effective than reviews for faults
concerning numerical approximations and control flow(UMD,USC)
• Both are about equally effective for results concerning typos,
algorithms, and incorrect logic(UMD,USC)
Feb 6. 2001
© USC-CSE
7
ODC Data Attractive for Extending
COQUALMO
- IBM Results (Chillarege, 1996)
Percent within activity
50
40
40
10
30
30
30
20
40
40
30
30
25
20
20
10
20
20
20
10
10
0
Design
Code review
Function
Feb 6. 2001
Assignment
© USC-CSE
Function test
Interface
System test
Timing
8
COQUALMO Extension Research
Approach
• Extend COQUALMO to cover major ODC categories
• Collaborate with industry ODC users
- IBM, Motorola underway
- Two more sources being explored
• Obtain first-land experience on USC digital library projects
- Completed IBM ODC training
- Initial front-end data collection and analysis
Feb 6. 2001
© USC-CSE
9
Digital Library Analysis to Date
- in ODC terms
•
•
•
Feb 6. 2001
Artifacts
- Operational Concept, Requirements, Software
Architecture documents
Activities
- Perspective-based Fagan inspections
Triggers
- Environment or condition that causes defect
© USC-CSE
10
Front End (Information
Development) Triggers
1.Clarity – confusing or difficulty to understand information.
2.Style – inappropriate or difficulty to understand the manner of
expression
3.Accuracy – incorrect information
4.Task Orientation - inappropriate presentation to perform task
5.Organization – relationship between parts is not conveyed
6.Completeness – missing information.
7.Consistency – the expression manner is not displayed in a consist
manner
Feb 6. 2001
© USC-CSE
11
Initial Digital Library Project ODC
Analysis
- Trigger percentage Distribution by Team
50
48.3
46.7
45
40
40
38.5
35
30
25.9 25.9
25.9
25
23.1
20.7
20
17.2 15.4
15 13.8
13.3
10
7.7 7.7
7.4
7.4 7.4
5
3.8 3.8
0 00
0
00 00
0
Team4
Team8
Team17
Team22
Feb 6. 2001
© USC-CSE
Clarity
Style
Accuracy
Task Orientation
Organization
Competeness
Consistency
12
Initial Digital Library Project ODC
Analysis
- Number of Triggers Defects by Team
35
33
30
30
25
20
15
10
5
0
15
14
8
4
22
4
0
Clarity
Feb 6. 2001
0
6
2
10
6 77
01 00 1
0
Style
Accuracy
Task Orientation
© USC-CSE
0
22
4
67 6
7
1 2
0
Oranization
5
Team 4
Team 8
Team 17
Team 22
Total
Completeness
Consistency
13
Issues and Research Plans
• Understand anomalies in Digital Library Data
- Number of Team 22 defects
- Team 4 completeness defects
- Due to differences in artifacts or procedures?
• Continue Digital Library ODC collection &
analysis
- Detailed Design, code, test
• Obtain, analyze industry ODC data
- Looking for more sources of ODC Data
Feb 6. 2001
© USC-CSE
14