R&D Metrics
Todd Little
What Makes a Good Metric?
 What is it measuring?
 Why do we care?
 How easy is it to gather & compute?
 Could you do it well and still suck?
 Could you do it poorly and still be great?
 How do you cheat?
 How do you balance the cheating?
Low Hanging Fruit
 Estimation Quality Factor (EQF)
 Defect Detection Efficiency (DDE)
 Defect Removal Efficiency (DRE)
 Cumulative Swamp Report
 Customer Satisfaction
Estimation Quality Factor (EQF)
Value to be Estimated
Actual Value
Blue Area
EQF =
Initial Estimate
Red Area
Actual
End Date
Elapsed Time
Link to article by Tim Lister
EQF from Lister/DeMarco
 An EQF of 5 is pretty good (i.e. averaging about 1/5
or 20 percent off.)
 The median for schedule estimating is about a 4, with
the highest sustained scores at 8 to 9.
 Lister and DeMarco have never known anybody to
sustain a 10 (just 10 percent off).
 Typical disaster project is 1.8
EQF Distribution Curve (LGC)
EQF for duration
has a theoretical
minimum of 2.0
We slip one day at a time, EQF=2
Value to be Estimated
Actual Value
Blue Area
EQF =
Initial Estimate
Elapsed Time
Red Area
Actual
End Date
(EQF-2) Distribution Curve (LGC data)
LGC Estimation Quality
 LGC’s EQF measurement is pretty good.
 Our p(50) is 4.8, versus an industry average
around 4 and a best sustained in the ~8-10.
 Our p(10) is 2.8, which is not bad.
 Calendar projects should have EQF = ~∞
 EQF can be used with scope on Y-axis
Defect Detection Efficiency (DDE) &
Defect Removal Efficiency (DRE)
Defects found before release
DDE 
Total Defects
Defects fixed
DRE 
Total Defects
Comments about DDE and DRE
 Can be for a release or a product
 Can be all defects or just P1/P0
 Can be all defects or just customer defects
Customer D efects fixed
DRE( customer ) 
Total Customer D efects
DDE and DRE SCOPUS Queries
 Ignore Closed, Duplicate, and
Enhancements
 Per product is easy
 Per release requires release date info and is
subject to version reporting
Defect Detection and Defect Removal
Efficiency
Overall
P0/P1 DDE
Rate
Overall
P0/P1 DRE
Rate
P0/P1
Customer
DRE
A
88%
77%
47%
B
83%
84%
62%
C
92%
91%
83%
D
34%
68%
60%
E
71%
94%
88%
F
94%
100%
100%
G
77%
95%
85%
H
75%
95%
96%
I
81%
91%
95%
Product
Defect Injection vs. Defect Removal
Cumulative Swamp over time
Swamp projections
Customer Product Satisfaction
Summary
 EQF calculated from time tracking
 DDE and DRE computed from defect
tracking
 Cumulative Swamp over time from time
tracking and from defect tracking
 Customer Product satisfaction from
Customer Support Survey
Summary
Metric
EQF
DDE
Description
Estimation Quality
Factor.
Defect Detection
Efficiency.
DRE
Defect Removal
Efficiency.
Customer
Satisfaction
Customer support
product satisfaction
How to Cheat
How to
balance
Project
Tracking
Sandbagging,
Cut scope, Ship
crap
Throughput,
DDE,
Customer
Satisfaction
Defect
Tracking
Find lots of
meaningless
defects. Mark
meaningless
defects as P1 or
P0.
Defects /
LOC?,
Customer
Satisfaction
Easy
Defect
Tracking
Re-class
customer defects
as P2 and only
consider P0
Defects /
LOC?,
Customer
Satisfaction
Easy
Customer
Support
Ask the right
questions
Revenue
How Easy
Easy
Easy
Data