Metrics: A Path for Success
Kim Mahoney, QA Manager, The Hartford
kim.mahoney@thehartford.com
QAAC
1
Session objectives: to leave this room
with knowledge of metrics and be able
to apply these learning’s to achieve
success in your organization.
QAAC
2
2
Success
My definition of success is:
– Test case pass rates > 70%
– Test environment availability/stability > 90%
– Requirements are stable with minimal changes
– No defects leaked into production
– Root causes of defects tell a story
What is yours?
QAAC
3
3
Agenda
What metrics to use in my organization?
Where to get the data to create the metric?
Key metrics for ensuring successes








Test case pass rate
Defect leakage into production
Requirements stability index
Test environment availability
Root cause of defects
Test effort variance
Error discovery rate
Automation script results
How can metrics pave a path to success?
QAAC
4
4
What metrics to use in my
organization?
QAAC
5
5
What metrics to use in my organization?
What do you want to gauge?
•Quality of code deploys
•Environment stability
What do you want to determine?
•Go/no-go decisions
•Quality of requirements
What makes sense?
•Not all metrics make sense for every project
Who to distribute to?
•Distribute to folks who can do something about it!
QAAC
6
6
Where to get the data?
QAAC
7
7
Where to get the data to create the metric?
Testing Tool




Test case execution results
Defects & root cause information
Production defects
Automation results
Requirements Tool
 Requirements changes
Time Tracking Tool
 Actual effort
Vendor Partner
 Environmental information
QAAC
8
8
Key metrics for ensuring
successes
QAAC
9
9
Test Case Pass Rate
QAAC
10
10
Test case pass rate
What is your target pass rate in your
organization?
Pass rate = # test cases passed / # test cases
executed
Sample: 238 test cases passed / 278 test cases
executed = 86% pass rate
What can be learned from this 86%?
QAAC
11
11
Test Case Pass Rate con’t
86% pretty good, but….
• What if the 14% that is failing is the most
critical part of the system?
• What if this is the last cycle of testing and
14% of those test cases cannot be fixed
before production?
• What was the project goal’s pass rate?
QAAC
12
12
Test Case Pass Rate con’t
How to use test pass rate?
• Comparing cycle to cycle
• Comparing similar test efforts
• Review test pass rate during and
after test execution
Who to tell? How to tell them?
• Management, Project Team
• Lessons learned meeting, post
mortem
QAAC
13
13
Defect Leakage into Production
QAAC
14
14
Defect Leakage into Production
Capture for each release
Capture root cause
Configuration, data, requirements, training, etc..
Capture severity – impact to biz
Critical, high, medium, low
Compare release to release
QAAC
15
15
Defect Leakage - Sample
What can we learn from this?
Who would want to know this?
QAAC
16
16
Requirement Stability Index
QAAC
17
17
Requirements Stability Index
RSI indicates the level of change to the original set of
customer approved requirements
Why is this a good metric to measure?
• Measuring and controlling RSI within the defined ranges
• Leads to a stabilized & controlled requirement thus
reducing rework effort & defect leakage.
• Increases test effectiveness and quality of application
implemented in production.
QAAC
18
18
RSI cont’d
Calculation:
= (Total # Original Requirements + Total # of
Requirements Changed + Total # of Requirements
Added + Total # of Requirements deleted) / Total #
Original Requirements
Green
1.00 to 1.1 (requirements are stable)
Yellow 1.12 to 1.15 (requirements stability is average)
Red
>1.15 (requirements are unstable due to frequent
requirement changes)
QAAC
19
19
RSI cont’d
When will we calculate RSI?
 RSI for a project/application will be calculated
every time when a change is requested.
 Note: RSI can be published at the end of a release
during project closure phase.
QAAC
20
20
RSI cont’d - Sample
• Total number of original
requirements:
28
• Requirement changes: 2
• Requirements added:
1
• Requirements deleted: 3
So……..
• RSI = (28+2+1+3)/28 = 1.18
• RSI > 1.15, Red (requirements are unstable)
QAAC
21
21
Test Environment Availability
QAAC
22
22
Test environment availability
Test environment availability and stability
– Total minutes due to issues / total minutes available
to track
• Example: 120 / 480 = .25
• 25% of the time the environment was not available for
testing
So… what is the impact when
the test environment is not
so stable?
QAAC
23
23
A picture is worth 1000 words…
What can be said about the above?
QAAC
24
24
Test environment availability cont’d
– Track daily and report out by release
– Track for trending
QAAC
25
25
Root Cause of Defects
QAAC
26
26
Root Cause of Defects
Defects can be caused for a number of reasons:
– Code issues
– Ambiguous requirements
– Data
– Test case
– Database
– Existing production defect
– Configuration
– Not an issue (all other)
Track root causes for trending to proactively avoid anticipated
defects in future
QAAC
27
27
Root Cause of Defect – cont’d
QAAC
28
28
Test Effort Variance
QAAC
29
29
Test effort variance
Planned vs. actuals for a test
effort
– Why track?
• Learn from
• Refine your estimating skills
– Who cares?
• QA management, Project Managers,
Finance
– How to mitigate variance
• Some reasons for variance: changes in requirements,
environmental issues, offshore network issues, late code
deployments, unusually high defects, etc…
• Easier to explain a week in variance
– Trending
• Are you always over or under estimating?
QAAC
30
30
Error Discovery Rate
QAAC
31
31
Error Discovery Rate
• EDR = total defects / total test cases executed
QAAC
32
32
Automation Script Results
QAAC
33
33
Automation script results
What is the value in automation? The reasons
are obvious.
What is not so obvious?
– The kinds of defects that are found over and over
– Script re-work that the automation team put in
due to code changes
– Applications that always have a low pass rate
when the automation bed is run
QAAC
34
34
Metrics can be a path to success…
QAAC
35
35
… because you can…
•
•
•
•
Learn from the metrics
Compare similar projects
Make things better
Continuously improve
One step at a time to
achieve success.
QAAC
36
36
In Summary…
• Reliable tools are needed which
house the data
• Metrics are objective
• Need to know which metrics
make sense for your
organization
• Distribute to folks who can
make a difference
• Pass rates can be deceiving
• Using metrics displays proactive
thought leadership
QAAC
37
37
QAAC
38
38