Improved Software
Defect Prediction
SPIN
23 February 2006
Norman Fenton
Agena Ltd
and
Queen Mary University of London
Slide 1
Using fault data to predict reliability
Pre-release testing
Post-release operation
delivery
Slide 2
Pre-release vs post-release faults?
30
?
20
Post-release
faults
10
0
0
40
80
120
160
Pre-release faults
Slide 3
Pre-release vs post-release faults:
actual
30
20
Post-release
faults
10
0
0
40
80
120
160
Pre-release faults
Slide 4
Software metrics….?
Slide 5
Regression models….?
Slide 6
Solution: causal models (risk maps)
Slide 7
What’s special about this approach?
• Structured
• Visual
• Robust
Slide 8
The specific problem for Philips
Defects
found
actual
predicted
time
Slide 9
Background to work with Philips
Agena Risk
MODIST
Fixed Risk Map (AID)
Reliability models
2000
2001
2002
2003
2004
Slide 10
Projects in the trial
MM&C (9)
28%
DVD (7)
22%
TV (16)
50%
(actual number of projects shown in brackets)
Slide 11
Factors used at Philips
•
•
•
•
Existing code base…
Complexity and management of new requirements ...
Development, testing, rework, process …
Overall project management …
Slide 12
Actual versus predicted defects
2000
1800
predicted defects
1600
1400
1200
1000
Correlation
coefficient
95%
800
600
400
200
0
0
500
1000
1500
2000
2500
Actual defects
Slide 13
Validation Summary (Philips’ words)
“Bayesian Network approach is innovative and one of
the most promising techniques for predicting
defects in software projects”
“Our evaluation showed excellent results for project
sizes between 10 and 90 KLOC”
“Initially, projects outside this range were not within
the scope of the default model so predictions were
less accurate, but accuracy was significantly
improved after standard model tailoring”
“AgenaRisk is a valuable tool as is”
Slide 14
Specific benefits
•
•
•
•
•
•
•
•
Accurate predictions of defects at different phases
Know when to stop testing
Identify where to allocate testing
Minimise the cost of rework
Highlight areas for improvement
Use out of the box now if you have no data
Approach fully customisable
Models arbitrary project life-cycles
Slide 15
Summary
• Risk maps – the way forward
• Validation results excellent
Slide 16
…And
You can use the
technology NOW
www.agenarisk.com
Slide 17