Software Quality - Getting Right
Metrics, Getting Metrics Right
How to set the right performance
metrics and then benchmark it for
continuous improvement?
While metrics are important means to quantify
performance of a process, we often define metrics to
comply with process standards, overlooking actual
alignment with business factors. We may put
substantial effort and discipline in getting metrics
deployed, but we might not be using the ones that
really matter. How can we define metrics that remain
anchored to business needs? In other words, how can
we translate business requirements to the right set of
metrics? Once the right metrics are defined, how can
we monitor and benchmark it to control performance
and make breakthrough improvements? This white
paper discusses best practices around metrics, from its
very definition to using it for continuous improvement.
Software Quality - Getting Right Metrics, Getting Metrics Right
About the Authors
Dr. Kanhaiya Jethani
Kanhaiya is a senior Process and Quality consultant at Tata Consultancy
Services. He has extensive experience in the fields of software project
management, software process and quality, CMM and CMMI. He has led
and executed process and quality consulting engagements for various IT
organizations in USA, Spain, Poland, Taiwan, Singapore, Japan and India.
He is a Certified Software Quality Analyst (CSQA), Project Management
Professional (PMP) and Candidate SCAMPI® Lead Appraiser. A Doctorate in
Chemical Engineering from Oklahoma State University, USA, he was also a
Post-Doctoral Research Fellow at Oklahoma State University, USA and
University Institute of Chemical Technology (UICT), Mumbai. He is a
member of PMI.
Vanishri Veloo
Vanishri is a senior Process and Quality consultant at Tata Consultancy
Services. She is a Candidate Lead Assessor for People CMM®, a qualified
auditor for ISO 9001:2000, a trained assessor for CMM® and SCAMPISM and
an External Assessor for Tata Business Excellence Model (TBEM). She has
led and executed process and quality consulting engagements for various
IT organizations in USA, UK, Germany, The Netherlands, Spain, Taiwan and
India.
Vanishri has worked in the financing of renewable energy, e-business and
financial services industry before moving to the Information Technology
Industry. Vanishri is a graduate in Biomedical Engineering from Mumbai
University, India and has done her Masters in Management Studies
(M.M.S.) in Finance from the Sydenham Institute of Management Studies,
Research & Entrepreneurship Education, Mumbai University, India.
Pragnya Misra
Pragnya Misra is a senior Process and Quality consultant at Tata
Consultancy Services. She has extensive experience in the fields of
application development and maintenance, software project
management, software process and quality, CMM and CMMI®. She has led
and executed process definition and deployment engagements for
various customers across the world. She is a Certified Software Quality
Analyst (CSQA).She is an SEI authorized assessor for CMM® and CMMI® and
also ISO-9000 Internal Auditor. She is a Masters in Business Administration
(Finance) and a Bachelor of Engineering (Instrumentation and Control)
from Gujarat University, India.
1
Software Quality - Getting Right Metrics, Getting Metrics Right
Table of Content
1. Introduction
3
2. Ask the Right Question
3
3. GQM Keeps Us Anchored to Changing
5
Business Imperatives
4. Using SPC to Control Process Performance
5
5. Continuous Improvement Needs
7
Breakthroughs where it Matters
6. Conclusion
9
2
Software Quality - Getting Right Metrics, Getting Metrics Right
Introduction
Pursuit of process frameworks and quality standards like CMMI® or ITIL has brought more discipline and maturity to
software development and IT services. The flip side is that we go overboard with a plethora of metrics, which often
are dictated by our preoccupation for compliance to such frameworks. With little alignment to the business-needs,
many of these metrics can turn ritualistic and time consuming, let alone the questionable sanity of the data. For
example, while we may remain complacent with low defect levels; we may ignore a high defect concentration in a
critical module or the severity of defects.
Many IT organizations have a metrics handbook that suggests the set of metrics to be followed. If such a handbook
prescribes a set of mandatory metrics, we would like such metrics to be rather small in number. A larger set of metrics
can remain optional, leaving it to the discretion of the project. Now the question is - how can one decide what would
be the right metric. It is the question of finding the right metric first before getting the metric right; the irony that
comes out of Joseph Juran’s adage do right things before doing things right.
Ask the Right Question
Consider this: a module in software was found to have abnormally high defects. The metric that was closely
monitored was the defect density - defects per unit functionality (for example defects per function point). This
module was escalated for re-engineering. On inspection of the design, it was found to be good in terms of
maintainability and defect prevention. Questioning the rationale for re-engineering, the defects were revisited. It
was found that surprisingly, the defects were more in number but low in severity, being concentrated mostly on
output formats. Unfortunately, the severity of the defects was not factored in the defect density metric.
Metrics will remain meaningless ceremonies unless they are aligned to business goals. In this example, the business
goal was to reduce injection of defects critical to the quality of the product.
While we inherit or improvise the metrics that would support the monitoring of the project, it is essential to be
anchored to the business needs. In fact, the business goal should translate to the right set of metrics required to
monitor and control quality. A three step process, Goal-Question-Metric (GQM), would be very handy here: It is a way
to identify the right metrics by asking questions around the business goal. GQM helps translate the business needs
from conceptual level to the operational level by asking appropriate questions, thereby identifying the measures
that can provide objective insight into the achievement of the business goals. For example, if our goal is to improve
estimation, what questions would we have in mind? We would most probably ask how good the current estimation
method is. Effort overrun would then be a good metric to indicate the accuracy of our estimation method.
3
Software Quality - Getting Right Metrics, Getting Metrics Right
Conceptual Level
Establish measurement
goals aligned with
business needs i.e. what
we want to achieve
Operational Level
Generate a set of
questions that can be
used to determine
achievement of goals
Quantitative Level
Establish the set of data,
objective or subjective,
that will answer each
question in a
quantitative way
Figure 1: Translating business needs at conceptual to operational and quantitative parameters
Goal
Approach
Question
Metric
Improve
Estimation
- Estimating effort
based on requirements
- Planning based
on estimation
How good
is effort
estimation?
Effort
Slippage
Minimize
Defects
- Detecting defects
as early as possible
- Minimize acceptance
testing defects
What are
current
defect
levels?
Defect
Density
Reduce
Costs
- Identifying non-value added activities
- Detecting defects early
- Reducing rework
What is
the rework
Effort?
Rework
Effort
4
Software Quality - Getting Right Metrics, Getting Metrics Right
GQM Keeps Us Anchored to Changing Business Imperatives
A large project calculated billable value in terms of the project effort expended until the time of costing. However, the
value in terms of project deliverables and milestones met were found to be lesser than the effort spent. This was
primarily due to schedule slippages at certain phases in the project. Due to the enormity of the project, the actual
value delivered during the project remained confusing. In this case, the business need was to deliver the milestones
as per schedule. The question that comes to mind is what milestones have been delivered and what was the
corresponding estimated effort? This suggested tracking the delivered value in terms of Earned Value. Earned value
is the value of the deliverables made in terms of estimated effort, irrespective of the slippages. This gave the project a
better understanding of how much of true value is being delivered by the project at a given point in time, vis- a- vis
the total effort spent.
Now, when we have selected the right metric for monitoring project or process performance, how do we ensure that
the performance suggested by the metric is within acceptable limits? Or, how do we know that the project will be
able to deliver the deliverable as per plan? Statistical Process Control (SPC) is a technique that is useful in this. It
defines the operational limits for acceptable variation in the process data.
Using SPC to Control Process Performance
Statistical Process Control primarily follows four steps. These are:
Measuring the process
!
Reducing variation to acceptable limits
!
Monitoring and controlling
!
Improving the process to achieve target value
!
While many tools are used in process improvement, with each being applicable to specific contexts, we will discuss
the use of Control Charts to control process variation. Control chart is plotting of the process parameter over time to
identify causes of any abnormal variation. Any parameter would have some variation that would be confined to an
acceptable or “normal” range. An instance of the high variation would suggest an underlying assignable or “special”
cause, which would then have to be eliminated. Normally, an instance of variation lying outside the range of u + 3s (u
is the mean and s the standard deviation) is considered to be abnormal or “outlier”. The limits u + 3s and u – 3s are
referred to as Upper Control Limit (UCL) and Lower Control Limit (LCL) respectively. Processes having variation within
the range LCL-UCL are considered stable.
5
Software Quality - Getting Right Metrics, Getting Metrics Right
Out of
control
Abnormal variation
due to assignable sources
1020
UCL
1010
1000
Mean
Normal variation
due to chance
990
LCL
Abnormal variation
due to assignable sources
980
970
0
1
2
3
4
5
6
8
7
9
10
11
12
13
14
15
Sample number
Figure 2
Process
Capability
Band
USL
LSL
LCL
UCL
3s
3s
The reason we chose to discuss control chart in this paper is that we find the technique useful in gauging the
capability of the process, than just the stability. Besides UCL and LCL, the process can set its own limits (target for
process performance). This is referred to as Lower Specification Limit (LSL) and Upper Specification Limit (USL). A
capable process would have a variation well within the LSL-USL range. The LCL-UCL of a capable process is called
Process Capability Baseline (PCB). As the process matures to be more capable, the USL-LSL range is shortened with a
more tapered peak (“high Kurtosis”) at the mean.
6
Software Quality - Getting Right Metrics, Getting Metrics Right
The bell-curve becomes:
“Taller”
“Thinner”
with more process improvements
!
!
Further Improvements
Improvements
Time X
Time X
Time X
Capability Over Time
Continuous Improvement Needs Breakthroughs where it Matters
Continuous improvement essentially involves improving the performance of the process so that the specification
limits are satisfied and the process becomes more capable. Once the specification limits are satisfied, tighter
specification limits or “stretch targets” are defined and another cycle of continuous improvement is initiated. We call it
a continuous improvement journey because the improvement actions are not warranted by exceptions or “special
causes”, but are more proactive in nature. It addresses the “common causes”, which are inherent in the nature of the
process. In the context of control charts, capability improvement in processes requires identifying the right
breakthroughs, which would shift the mean and reduce variation to make it centered on the mean. In simpler words,
this means reducing variation to the extent that it improves end quality and shifts the mean to the desired
specifications. Often, we reduce variation on a parameter even when the process is robust with the existing level. For
example, we may go overboard on improving the performance variation of a transaction when its highest variation is
well accommodated in the actual transaction sequence. Rather, the business objective would be to reduce the mean
time taken to perform the series of related transactions in the practical business scenario. Reduction of the mean
would require a “breakthrough” action or process change. This is because the old process may have already hit its
maximum performance state given the existing process definition. To improve from here, one would need to
introduce new interventions or activities within the process. For example, we may often find that, despite sufficient
testing, there are a high number of defects during production. A process alteration such as peer review of source
code can help in detection of potential defects.
7
Software Quality - Getting Right Metrics, Getting Metrics Right
As depicted in the following figure, breakthrough improvements will lead to change in the mean performance of the
process.
Original zone of
Quality control
Chronic waste
New zone of
Quality control
Less chronic waste
Breakthrough
Improvement
To summarize, quality improvement starts with identifying the right metrics. We apply techniques such as control
charts to improve the stability of the process. Thereafter, we embark on a continuous improvement journey by
increasing the capability of the process. All through, a fact-based approach is needed. The typical approach is
illustrated as follows:
Business
Objectives
Process
Change
Sub-process
Selection
Measure
Piloting
Project-level
Change
Remove
Special
Causes
Remove
Common
Causes
Org.
Change
Process/
Technology
Change
No
No
Statistical
Analysis
(OPP)
Process
Stable?
Yes
8
Process
Capable?
Yes
Software Quality - Getting Right Metrics, Getting Metrics Right
Conclusion
A quantitatively managed process would have metrics aligned to business needs. Such needs are better
understood when we define the business goals. One can translate the goals to a set of representative metrics by
asking the right questions, which is an important part of the exercise. The GQM approach helps in this translation.
Once the right metrics are defined, SPC helps in monitoring the performance of the metrics and bringing the
process under control. We have discussed control chart as one of the tools that is useful here. As we move into the
continuous improvement journey, we try to reduce variance where it matters and find breakthroughs/ process
changes to shift the mean. This makes the process more capable.
At the end of the day, all a good process needs is the right metrics used to monitor it and good benchmarks set to
control its performance and continuously improve its performance.
9
About Tata Consultancy Services (TCS)
About TCS IT Process and Service
Management Consulting
Tata Consultancy Services is an IT services, business solutions and
IT Process and Service Management Consulting partners with
outsourcing organization that delivers real results to global
organizations to enable IT process improvement aligned to
businesses, ensuring a level of certainty no other firm can match.
business needs. Leveraging TCS' experience in successfully
TCS offers a consulting-led, integrated portfolio of IT and IT-
adopting best-inclass frameworks like CMMI®, ISO-9001, ITIL®,
enabled services delivered through its unique Global Network
P-CMM®, BS15000, BS7799, and Six Sigma, the consulting group
Delivery ModelTM recognized as the benchmark of excellence in
provides services with a practitioner’s approach to assessing,
software development.
defining, improving and deploying IT processes.
l
CMMI® appraisal services and training
A part of the Tata Group, India's largest industrial conglomerate,
l
ITIL deployment and assessment
TCS has over 100,000 of the world's best trained IT consultants in 50
l
Agile development for distributed environment
countries. The company generated consolidated revenues of US
l
Six Sigma-based improvements
$5.7 billion for fiscal year ended 31 March 2008 and is listed on the
l
Quality improvement initiatives
National Stock Exchange and Bombay Stock Exchange in India. For
Digitized enablers which reduce 'Time to Deploy' and sustain
l
more information, visit us at www.tcs.com.
productivity for process
global.consulting@tcs.com
All content / information present here is the exclusive property of Tata Consultancy Services Limited
(TCS). The content / information contained here is correct at the time of publishing.
No material from here may be copied, modified, reproduced, republished, uploaded, transmitted,
posted or distributed in any form without prior written permission from TCS. Unauthorized use of the
content / information appearing here may violate copyright, trademark and other applicable laws,
and could result in criminal or civil penalties.
Copyright © 2008 Tata Consultancy Services Limited
www.tcs.com