SERVICE PRODUCTIVITY MEASURES /
EVALUATION TECHNIQUES/
MEASUREMENT CHALLENGES
:CASE OF SOFTWARE INDUSTRY
Abide COŞKUN
Murat Umut İZER
Sarp KOHEN
OUTLINE

Introduction

Service Productivity

Software Productivity

Conclusion
WHAT IS PRODUCTIVITY?

Productivity shows whether the activity of an
organization is efficient and effective.

But simply it is defined as Output over Input
WHAT IS PRODUCTIVITY?

Productivity measures express relationships between the
outcomes or outputs of service processes and the resources
or inputs required to operate them (McLaughlin and Coffey,
1990).

Input
Resources spent to generate the output
(ex. Effort)

Output
The Value delivered
WHAT IS PRODUCTIVITY?
There are many different productivity measures.
 The choice between them depends on the purpose of
productivity measurement and on the availability of
input and output

WHAT IS PRODUCTIVITY?
Productivity Measures
Definition
Interpretation
Shows the time profile of how
Labour productivity,
Quantity index of gross output /
productively labour is used to
based on gross output
Quantity index of labour input
generate gross output.
Purpose
It traces the labour requirements per unit of
(physical) output.
Analysis of micro-macro links, such as the
industry contribution to economy-wide labour
Labour productivity,
Quantity index of value added /
based on value added
Quantity index of labour input
Shows the time profile of how
productivity and economic growth.
productively labour is used to
generate value added.
At the aggregate level, a direct link to a widely
used measure of living standards, income per
capital.
Capital productivity,
Quantity index of value added /
based on value added
Quantity index of capital input
Capital-labour MFP based
on value added
KLEMS Multifactor
productivity
Quantity index of value added /
Quantity index of combined
labour and capital input*
Shows the time profile of how
productively capital is used to
generate value added.
Changes in capital productivity indicate the
extent to which output growth can be
achieved with lower welfare costs in the form
of foregone consumption.
Show the time profile of how
Analysis of micro-macro links, such as the
productively combined labour and
industry contribution to economy-wide MFP
capital inputs are used to generate
growth and living standards, analysis of
value added.
structural change.
Quantity index of gross output /
Shows the time profile of how
Quantity index of combined
productively combined inputs are
inputs*
used to generate gross output.
Analysis of industry-level and sectoral
technical change.
PRODUCTIVITY MEASUREMENT

Nordhaus explained the sectors that can have wellmeasured outputs as:

Agriculture, forestry, and fishing
Mining
Manufacturing
Transportation and Public Utilities
Wholesale trade
Retail trade





PRODUCTIVITY MEASUREMENT

Productivity measurement is difficult in some sectors
because outputs and inputs are typically quite diverse
and are often themselves difficult to measure
(Nordhaus, 2001).

Construction
Real estate
Finance
Government
Services




SERVICE AND SOFTWARE PRODUCTIVITY

Productivity in the service sector was not analyzed
before the end of the twentieth century, while
productivity in manufacturing has been analyzed for
more than two hundred years (Rutkauskas, Paulavicien,
2005).

Since input and output of service sector productivity
consist not only of quantitative elements but also
qualitative, measuring service productivity is difficult.
SERVICE AND SOFTWARE PRODUCTIVITY
Anselmo and Ledgard (2002) state that software is the
most important industry among others.
 While it is noticed that software productivity is
declining more rapidly than other industries (Groth,
2004).

PRODUCTIVITY IN SERVICE
SECTOR
THE U.S. BUREAU OF LABOR STATISTICS (BLS)

173 - industry titles

39 - the broad service, or nongoods,
sector
PRODUCTIVITY RATIO
SERVICE PRODUCTIVITY RATIO
QUANTITY ASPECTS OF SERVICE PRODUCTIVITY
Inputs
 Labor
 Capital
 Raw material
Output
 Service volume
QUALITY ASPECTS OF SERVICE PRODUCTIVITY
Inputs
 Tangible elements
 Intangible elements
Output
 Customer perceived
quality
MEASURE REFLECTION

changes in technology

scale of production

educational levels of workers

managerial techniques
CHALLENGES
What makes measuring service
productivity so hard?
the nature of service
IHIP-CRITERIA

Intangibility - incapable of being perceived

Heterogeneity - non-standardized

Inseparability - consumed at the point of production

Perishability – transitory nature
MEASUREMENT OUTPUTS FOR SERVICE INDUSTRIES
Trade
 gross sales
 labor cost
 employee hour
Transportation
 the movement of goods
 passengers over distance
 labor input – ex. trucking, air
transportation, and bus
carriers
Communication
 number of calls (or call minutes)
Retail banking
 number of access (main) lines
 numbers of transactions
 revenues
and deposit
 labor hours
 loan accounts revenues
 number of pieces of mail
MEASUREMENT OUTPUTS FOR SERVICE INDUSTRIES
Real Estate
 real estate loans.
the number of residential mortgage loans
the number of construction loans
the number of commercial mortgage loans
Business and personal services
 number of calls (or call minutes)
 number of access (main) lines
 revenues
 labor hours
 number of pieces of mail
BRINKERHOFF & DRESSLER’S 7 STEPS OF MEASUREMENT
1. Mission Statement
Write a mission statement for the unit that identifies the
major goals and customer of the unit.
BRINKERHOFF & DRESSLER’S 7 STEPS OF MEASUREMENT
2. Expectations
Expectation must be clearly identified and explaining quality
needs.
BRINKERHOFF & DRESSLER’S 7 STEPS OF MEASUREMENT
3. Key Outputs
Identify outputs that are important to the unit’s mission.
BRINKERHOFF & DRESSLER’S 7 STEPS OF MEASUREMENT
4. Major Functions
Identify and describe the major functions of the unit.
BRINKERHOFF & DRESSLER’S 7 STEPS OF MEASUREMENT
5. Output Measurement Selection
Construct measurement techniques for one or more key
outputs.
BRINKERHOFF & DRESSLER’S 7 STEPS OF MEASUREMENT
6. Input Measurement Selection
Construct measurement techniques for one or more key
inputs.
BRINKERHOFF & DRESSLER’S 7 STEPS OF MEASUREMENT
7. Index Construction
Construct one or more productivity measures to incorporate
the output and input measures.
PRODUCTIVITY IN SOFTWARE
INDUSTRY
THE SOFTWARE INDUSTRY

Introduction of the computer technology from the
1960s

Software industry consists of the activities:
Development,
 Maintenance,
 Production
 Software services (documentation, support and training)

THE SOFTWARE INDUSTRY
Personal computers became widespread in the mid
1970s
 The need for operating systems and software


MS-DOS
Introduction of workstations and servers
 The need for enterprise software
 Size and complexity of software projects increased
rapidly

(Groth, 2003)
PRODUCTIVITY OF THE SOFTWARE INDUSTRY
AMONG OTHERS
Annual productivity growth rates for industry from
1998 to 2003
(Groth, 2003)
WHY TO MEASURE SOFTWARE PRODUCTIVITY?
To control and improve the performance of software
development
(Petersen, 2011)
 Benchmarks
 Time to market and speed of delivery is more important
economically
 Reliable and consistent measurement
 Trade-offs with speed of delivery and quality

(Symons, 2010)
CHAIN LINKING MEASUREMENT
(Symons, 2010)
CHALLENGES OF SOFTWARE PRODUCTIVITY

Problems determining the software productivity:
Poor definition of measures
 Unclear root-cause effect for function points
 Lack of combining managerial and technical aspects

(Scacchi, 1991)
What represents quality for the user?
 What represents quality for the developer?

(Wong and Jeffery, 2002)
CHALLENGES OF SOFTWARE PRODUCTIVITY

Why software productivity is a complex process?
It is difficult to measure regarding to the traditional
definition
 It is affected by lots of factors, some of them are difficult to
measure
 Interaction between these factors


(It is easier to be productive while disregarding quality)
(Premraj et al., 2005)
CHALLENGES OF SOFTWARE PRODUCTIVITY
Two
different
programs
with exactly
the same
functionality
(Premraj et al., 2005)
CHALLENGES OF SOFTWARE PRODUCTIVITY
Metrics don’t work properly
complexity of the software
 No trend data for productivity

increasing
(Symons, 2010)

The reason of declining productivity:
No industrywide standard definition
 Increasing compexity of software applications
 Need for more formalized processes

(Groth, 2004)
LOW PRODUCTIVITY
(Jeet et al., 2011)
CHALLENGES OF SOFTWARE EVALUATION

No consensus on generic software evaluation criteria

A framework must be developed to help decision
makers
(Jadhav et al., 2008)

Selection and the evalutation of software packages
being more complex
Large number of software packages
 Incompabilities between hardware and software packages
 Lack of technical skills of decision makers
 Ongoing improvements in IT
(Jadhav et al., 2011)

MODELS FOR SOFTWARE EVALUATION
(Petersen, 2011)
MODELS FOR SOFTWARE EVALUATION
(Hernandez-Lopez et al., 2010)
DECISIONS OF SOFTWARE EVALUATION
Keep or change,
 Make or buy,
 Commercial product evaluation,
 Tender evaluation,
 Software certification,
 Software process evaluation,
 Software system design selection

(Stamelos et al., 2003)
SOFTWARE EVALUATION CRITERIA
(Jadhav et al., 2011)
DECISION HIERARCHY FOR SOFTWARE EVALUATION
(Jadhav et al., 2011)
SOFTWARE MEASUREMENT PROCESS
(Farooq et al., 2011)
IN CONCLUSION

The improvement in software productivity cannot be
predicted without measuring it likewise the service
sector

The measures of a software product such as
functionality, complexity or quality differ from service
sector in terms of intangibility, heterogeneity,
inseparability and perishability.
THANK YOU FOR LISTENING =)
MIS 518 SELECTED TOPICS
Abide COŞKUN
Murat Umut İZER
Sarp M. KOHEN
SOFTWARE PRODUCTIVITY
MEASUREMENTS
Weider D. Yu, D. Paul Smith, and Steel T. Huang
Abide COŞKUN
ABOUT THE PAPER
describes the specific effort that has been taken to establish
and improve the measurement of software productivity in the
US 5ESS
 summaries some of the results of the effort.

ABOUT THE SOFTWARE
The 5ESS switch developed for the United States market.
 The 5ESS Switch project is one of the most extensive
software projects at AT&T BellLaboratories.
 The total size of the delivered source code and internal
support code is several million lines of code.

MEASURING PRODUCTIVITY

To measure software productivity, two fundamental
measures must be established:

The measure of the output from a software
development process

The measure of the input to a software development
process.
OUTPUT-INPUT

software program size - output

development effort - input
SOFTWARE PROGRAM SIZE MEASUREMENT
-Software Size Viewed at Project Level

production code and support code.
SOFTWARE PROGRAM SIZE MEASUREMENT
Source Code
 base code
 modified code
 new code
 bug code
 ported code
DEVELOPMENT EFFORT MEASUREMENTS
Direct hardware development effort: circuit design, physical
design, diagnostic and resident software (firmware)
development.
 Direct software development effort
 Support efort: integration, load building, tools, system
verification, field testing, training, field documentation,
resource improvement, system labs, management and
project coordination.

SOFTWARE PRODUCTION RATE
software program size
direct sofiware development effort
SOFTWARE PRODUCTION RATE

Software program size is measured in KNCSL.

Direct software development effort is measured in ATHCTY
(Average Technical Head Count Year).
Software Production Rate (NCSL / ATHCT)
Release 1
Release 2
Release 3
Release 4
Release 5
Release 6
Feature Software Production Rate (NCSL /
ATHCT)
FP1
FP2
FP3
FP4
FP5
FP6
FP7
FP8
FP9
THE RESULT
The results of software productivity measurements
have been applied in the following areas:
1. The productivity measurements and the identified
software productivity factors have been used to
develop a SESS estimation model, 5ESS SIZER
THE RESULT
The results of software productivity measurements
have been applied in the following areas:
2. Software size measurement has been used extensively
as an important normalizing factor in baselining the
major characteristics of the SESS releases and
development process such as a variety of code sizes,
fault density profiles, software production rates and
test densities
THE RESULT
The results of software productivity measurements
have been applied in the following areas:
3. The identification of high impact productivity factors
has helped the project focus its process improvement
efforts.
THE RESULT
The results of software productivity measurements have
been applied in the following areas:
3.1 Feature requirement completeness and stability. A
requirement traceability methodology has been introduced
to the SESS development community to emphasize
requirement specification and requirement verification. It
assists SESS engineers to identify requirement faults and
omissions during the earlier development stage and to
facilitate further feature design and testing work.
3.2 Feature interaction complexity. For some large and
complex features, a feature interaction matrix is constructed
to show all the interactions with other features. This
improves the effectiveness of feature design and testing.
THE RESULT
The results of software productivity measurements
have been applied in the following areas:
3.3 Stuff experience. Critical expertise and shared resources
were reorganized into functional units to better utilize
subject experts. Management is currently considering a
proposal. which gives additional incentives to engineers who
are willing to stay on the same job function for a specific
period of time.
3.4 Feature development environment impact. A number of
development tools have been introduced to improve the
development environment, such as DOC, which allows
SOFTWARE MEASUREMENTS
AND METRICS: ROLE IN
EFFECTIVE SOFTWARE TESTING
(Farooq et al., 2011)
Murat Umut İZER
SOFTWARE MEASUREMENT
An important issue in the software engineering industry
 Software testing should be done properly and
effectively
 Increasing the effectiveness of software testing process
 Output: Quality software and/or improved processes

SOFTWARE METRICS
Still mainly used for cost estimation
 Used to evaluate the software development process
and the quality of the product
 Should be effectively implemented
 Contribute to the quality of the software
 Helps to deliver the software in time and within budget

SOFTWARE MEASUREMENT
SOFTWARE MEASUREMENT

Characteristics of a good measurement process:
Reliability
 Validity
 Sensitivity


Measurement plays an important role for
prediction,
 progress,
 process improvement

KEY COMPONENTS OF AN EFFECTIVE
MEASUREMENT
Clearly defined development issues
 Graphical or tabular reports
 Analysis of indicators
 Use of analysis results


Why is measurement process used?

To determine quality, progress, and performance
throughout all life cycle phases
WHAT IS A METRIC?
A measurement of a software product, process or
project that is directly observed.
 Metrics are derived from measures
 Using software metrics, it is able to:

View requirements,
 Predict development resources,
 Track development progress,
 Determine maintenance costs

AN EFFECTIVE MEASUREMENT PROCESS
METRIC ELEMENTS
USABLE METRICS
CATEGORIES OF METRICS
Commercial
Perspective
Significance
Perspective
Observation
Perspective
Measurement
Perspective
Technical
metrics
Core metric
Primitive
metrics
Direct
measurement
Process
metrics
Objective
metrics
Defect metrics
Non-core
metric
Computed
metrics
Indirect/
derived
measurement
Product
metrics
Subjective
metrics
End-user
satisfaction
metrics
Warranty
metrics
Reputation
metrics
Other Perspectives
TESTING MEASURES AND METRICS
SOFTWARE TEST METRICS
THE CHALLENGE OF PRODUCTIVITY
MEASUREMENT
David N. Card , 2006
Sarp M.KOHEN
WHAT IS THE CHALLENGE OF PRODUCTIVITY
MEASUREMENT?

No single productivity measure applies in all situations
for all purposes

Organizations must craft productivity measures
appropriate to their processes and information needs
COMMONLY USED INTERNATIONAL STANDARDS

IEEE Standard 1045, Software Productivity Measurement:

Describes the calculation of productivity in terms of effort
combined with counts of lines of code or function points.

It recommends variations to address software re-use and
maintenance scenarios. It provides a project characterization form

does not discuss how different characteristics might lead to different
productivity measures.
COMMONLY USED INTERNATIONAL STANDARDS

ISO/IEC Standard 15939, Software Measurement Process:

This standard is the basis for the Measurement and Analysis
Process Area of the Capability Maturity Model Integration.

The process model identifies the principal activities required for
planning and performing measurement. The ISO/IEC information
model defines three levels of measures: indicators, base measures,
and derived measures

does not discuss how different characteristics might lead to different
productivity measures.
COMMONLY USED INTERNATIONAL STANDARDS

SEI technical reports:

Discuss how to define effort [12] and size measures [13],

but give little guidance on how they can be combined to
compute things such as productivity
COMMONLY USED INTERNATIONAL STANDARDS

None of these standards systematically
addresses the factors that should be
considered in choosing appropriate base
measures and constructing indicators of
productivity for specific purposes
THE CONCEPT OF PRODUCTIVITY
THE CONCEPT OF PRODUCTIVITY

Numerator
Amount of product
 Volume of requirements
 Value of product


Denominator
Amount of resources expended
 Cost of resources expended

THE CONCEPT OF PRODUCTIVITY

• Scope of outputs (product)

• Scope of resources

• Requirements (or other input) churn

• Quality at delivery
SIZE MEASUREMENT

Function Points: A function point is a unit of measurement to
express the amount of business functionality an information
system provides to a user. The cost (in dollars or hours) of a single
unit is calculated from past projects. As of 2012, there are five
recognized ISO standards for functionally sizing software.

Lines of Codes: Lines of code (LOC) is a software metric used to
measure the size of a software program by counting the number of
lines in the text of the program's source code. SLOC is typically
used to predict the amount of effort that will be required to
develop a program, as well as to estimate programming
productivity or maintainability once the software is produced.
RESOURCE MEASUREMENT

1) the categories of cost and effort to include and

2) the period of the project life cycle over which they
are counted

Four categories of labor may be considered in
calculating productivity: engineering, testing,
management, and support (N.Card, 2006)
REQUIREMENTS CHURN AND QUALITY AT DELIVERY
TYPICAL PRODUCTIVITY CALCULATIONS

Physical Productivity

The ratio of the amount of product to the resources
consumed
TYPICAL PRODUCTIVITY CALCULATIONS

Functional Productivity

The ratio of the amount of the functionality delivered to the
resources consumed.
TYPICAL PRODUCTIVITY CALCULATIONS

Economic Productivity

The ratio of the value of the product produced to the cost of
the resources used to produce it.
COMPARING PRODUCTIVITY NUMBERS

Having chosen a productivity calculation along with
appropriate definitions of resource and size measures,
productivity numbers can be produced. Comparing
productivity numbers from a series of closely related
projects (e.g., members of a product line) is
straightforward. However, making comparisons across
different projects or organizations requires greater
care. (N.Card, 2006)
IN CONCLUSION

No single measure of productivity is likely to be able to
serve all the different needs of a complex software
organization


including project estimation, tracking process performance
improvement, benchmarking, and demonstrating value to
the customer.
Multiple measures of productivity may be needed.
Each of these must be carefully designed.
THANK YOU FOR LISTENING =)