ESPRIT Project 28015 TRUMP
Trial Usability Maturity Process
Cost benefit analysis
Nigel Bevan
Serco Usability Services
D3.0
Version 1.1
7-Mar-16
Abstract
Explains how to calculate the cost-benefits of user centred design, by comparing the costs
with an estimate of the potential benefits to the organisation during development, sales, use
and support. Vendors can benefit in development, sales and support. Purchasers can benefit
in use and support. Systems developed for in-house use can benefit in development, use and
support.
Keywords: cost benefit analysis, user centred design, usability.
Cost benefit analysis
TRUMP D3.0
1. DOCUMENT HISTORY
Issue
Date
Author
Description of change
0.1
4/12/99
N Bevan
Draft for comment
0.11
20/12/99
N Bevan
Feedback from Dorothy Kushner
1.0
30/3/00
N Bevan
Included example
1.1
8/9/00
N Bevan
Executive summary improved
Page 2
© 2000 Serco Ltd
Approved
Version 1.1 7-Mar-16
Cost benefit analysis
TRUMP D3.0
CONTACTS
This report was produced by.
Nigel Bevan
Serco Usability Services
Alderney House
4 Sandy Lane, Teddington
TW11 ODU, Middx, UK
Tel: + 44 20 8614 3811
Fax: + 44 20 8614 3765
nbevan@usability.serco.com
Page 3
© 2000 Serco Ltd
Version 1.1 7-Mar-16
Cost benefit analysis
TRUMP D3.0
Contents
2
EXECUTIVE SUMMARY
5
3
THE PROBLEMS (AND THEIR COSTS)
7
4
POTENTIAL BENEFITS OF USABILITY
8
4.1
Development
8
4.2
Sales
8
4.3
Use
8
4.4
Support and Maintenance
9
ESTIMATING BENEFITS
11
5
5.1
Development
11
5.2
Sales
11
5.3
Use
11
5.4
Support
12
6
MAKING THE COST-BENEFIT CASE
13
6.1
Calculating cost-benefits
13
6.2
Example of cost-benefit calculation
13
6.3
Other barriers to user-centred design
14
REFERENCES
Page 4
15
© 2000 Serco Ltd
Version 1.1 7-Mar-16
Cost benefit analysis
TRUMP D3.0
2 Executive summary
To calculate the cost-benefits of user centred design:

estimate the potential savings during development, sales, use and support

estimate the costs that would be incurred.
The extent of the financial benefits will depend on how completely user centred design can
be implemented. A balance needs to be obtained so that a convincing case can be made for
benefits that are substantially larger than the costs of additional user centred activities.

Vendors can benefit in: development, sales and support.

Purchasers can benefit in: use and support.

Systems developed for in-house use can benefit in: development, use and support.
1. Development
What savings will be made as a result of:
1.1 Reduced development time and cost to produce a product which has only relevant
functionality and needs less late changes to meet user needs?
1.2 Reduced cost of future redesign of the architecture to make future versions of the product
more usable?
2. Sales
What increase in revenue will result from:
2.1 Increased competitive edge: customers expect products to be easy to use. What
percentage increase in sales can be obtained by marketing your product as easier to use than
the competition?
2.2 More satisfied customers: difficult to use products create dissatisfied customers. What
percentage increase will there be in repeat customers more satisfied with the current product?
2.3 Higher ratings for usability in the trade press?
Customers will be more satisfies as a result of the potential savings listed below.
3. Use
What savings will be made as a result of:
3.1 Reduced task time and increased productivity?
3.2 Reduced user errors that have to be corrected later?
3.3 Reduced user errors leading to an improved quality of service?
3.4 Reduced training time for users?
3.5 Reduced staff turnover as a result of higher satisfaction and motivation?
4. Support
What savings will be made as a result of:
Page 5
© 2000 Serco Ltd
Version 1.1 7-Mar-16
Cost benefit analysis
TRUMP D3.0
4.1 Reduced costs of producing training materials?
4.2 Reduced time providing training?
4.3 Reduced time spent by other staff providing assistance when users encounter difficulties?
4.4 Reduced help line support?
5. Calculating cost-benefits
For organisations already committed to user-centred design a cost-benefit analysis is not
essential but it can provide a valuable input when formulating a usability plan. The technique
can be used repeatedly as a development project progresses to reassess the importance of
various activities. The process can also be used to compare different usability methods and so
aid selection of the most cost effective method.
Page 6
© 2000 Serco Ltd
Version 1.1 7-Mar-16
Cost benefit analysis
TRUMP D3.0
3 The problems (and their costs)
Landaur (1995) provides evidence that computers have not improved the efficiency and
quality of most information work, and that difficult-to-use computer systems are a major
factor in restricting economic growth. Productivity growth in the seven richest nations has
fallen in the last 30 years, from an average of 4.5% a year during the 1960s, to a rate of 1.5%
in recent years. The slowdown has hit the biggest IT spenders – service-sector industries –
hardest (Gibbs, 1997). 31% of the computer systems designed for in house use are either
cancelled or rejected as unacceptable, primarily because of inadequate user design input.
When they are installed "computers often create pathologies and increased costs" (Standish
Group, 1995). The result is a loss of approximately $80 billion annually to the economy
(Standish Group 1995).
Some of the specific costs created by badly designed and difficult to use systems have been
identified. Because computers are often hard to use, companies often have to provide about
$3150 worth of technical support for every user (Gartner Group, 1997). In addition, nontechnical employees take 4-10% of their time to help co-workers solve their computer
problems at a cost of $10,500 a year for each computer (Nolan Norton Institute, 1997).
Unproductive activities with computers (support and housekeeping, response time delays,
checks for accuracy, etc) cost another $5590 per computer per year (Gartner Group, 1997).
Page 7
© 2000 Serco Ltd
Version 1.1 7-Mar-16
Cost benefit analysis
TRUMP D3.0
4 Potential benefits of usability
The objective of introducing user centred methods is to ensure that real products can be used
by real people to achieve their tasks in the real world. This requires not only easy-to-use
interfaces, but also the appropriate functionality and support for real business activities and
work flows. According to IBM (1999) “It makes business effective. It makes business
efficient. It makes business sense”.
4.1 Development
Many existing development processes focus exclusively on adherence to technical and
process specifications. User centred methods can reduce development time and cost. The
user interface may account for up to 66% of the lines of code in an application and 40% or
more of the development effort (MacIntyre et al 1990). During software development,
usability engineering can reduce the time and cost of development efforts through early
definition of user goals and usability objectives, and by identification and resolution of
usability issues. A change may cost 1.5 units of project resource during conceptual design, 6
units during early development, 60 during systems testing and 100 during post-release
maintenance (Pressman, 1992).
The value of user centred methods in improving the quality of computer systems is confirmed
by a survey by Keil and Carmel (1995) which showed that the probability of project success
is improved by user-centred design.
4.2 Sales
In today's market, usable products are desirable products. Ease of use differentiates them in a
highly competitive market place. Ease of use brings an added value that culminates in a
higher degree of customer satisfaction, continued business and higher revenues. Customer
service and satisfaction provide market differentiation (Jones and Sasser, 1995, Prokesch,
1995). Companies committed to ease of use do more than meet customer expectations, they
can actually exceed anticipated earnings (Karat, 1997).
User-centred design provides methods and tools for identifying, and catering for, a wide
range of user needs, thus potentially making a product accessible to a wider range of users,
including those with special needs.
Products with a higher rate of employee satisfaction along with improved performance,
produce a chain reaction that leads directly, and ultimately, to more satisfied customers. And
satisfied customers provide the base for business growth and competitive success.
Wixon and Jones (1992) document a case study of a usability-engineered product that
achieved revenues that were 80% higher than for the first release developed without usability
engineering, and 60% above project expectations.
4.3 Use
Companies that purchase or produce usable systems for their employees also see impressive
returns on their investment.
Page 8
© 2000 Serco Ltd
Version 1.1 7-Mar-16
Cost benefit analysis
TRUMP D3.0
 Improved productivity. The average software program has 40 design flaws that impair
employees' ability to use it. The cost in lost productivity is up to 720% (Landauer 1995).
Design changes due to usability work at IBM resulted in an average reduction of 9.6
minutes per task, with projected internal savings at IBM of $6.8 Million in 1991 alone
(Karat 1990).
 Better task focus. A good interface to a well-designed product will allow the user to
concentrate on the task rather than the tool. If the interface is designed appropriately, it
will allow them to operate effectively and efficiently, rather than lose vital time struggling
with a poorly designed user interface and badly thought-out functionality.
 Reduced errors. A significant proportion of so-called “human error” can be attributed to a
product with a poorly designed interface that is not closely matched to the users' task
needs, or to their mental model of the task. Avoiding inconsistencies, ambiguities or other
interface design faults, while adhering to user expectations in terms of task structure and
sequencing, has the potential to significantly reduce user error and improve quality.
 Reduced training. A poorly designed user interface and dialogue can prove a barrier to an
otherwise technically sound system. A well-designed system designed with a focus on the
end-user can reinforce learning, thus reducing training time and effort. Karat (1993b)
documents a case study of a usability engineered product that required a one-hour training
session as compared to one week of training for similar systems built and used within the
organisation. The investment in usability save the organisation millions of dollars in
training costs and in the opportunity costs of the employees' time in the first year alone.
 Organisations that ensure their employees are furnished with easy-to-use products see
dramatic reductions in training time and, subsequently, great reductions in training cost. In
certain cases,.
 Improved acceptance. This is particularly important where usage is discretionary. Users
are more likely to use and trust a well-designed, accessible system, which has been
designed so that information is both easy to find and is provided in a form that is easy to
assimilate and use.
 Reduced staff turnover resulting in lowered personnel costs associated with the employees
who have lower turnover rates and higher satisfaction with their work environment. (Karat
1994a).
4.4 Support and Maintenance
When an organisation tries to recover from significant usability problems in a released
product, the costs of post-release changes are unaffordable for many organisations.
 80% of software life cycle costs occur during the maintenance phase (Pressman 1992).
 80% of maintenance is due to unmet or unforeseen user requirements; only 20% is due to
bugs or reliability problems (Martin and McClure 1993; Pressman 1992).
 Microsoft tracks its support call costs and has seen a significant cost savings resulting
from improving the usability of its products, such as Word (Reed 1992).
 Design changes due to usability work on one project at IDS/American Express resulted in
estimated savings of $45 million (Chalupnik and Rinehart 1992).
Page 9
© 2000 Serco Ltd
Version 1.1 7-Mar-16
Cost benefit analysis
TRUMP D3.0
 Design changes from one usability study at Ford Motor Company reduced the number of
calls to the help line from an average of 3 calls to none, saving the company an estimated
$100,000 (Kitsuse 1991).
 Reduced support. Significant savings of help-desk calls and service costs are another
added bonus when products are made to meet user needs.
Page 10
© 2000 Serco Ltd
Version 1.1 7-Mar-16
Cost benefit analysis
TRUMP D3.0
5 Estimating benefits
To calculate the cost-benefits of user centred design, first estimate the potential benefits for
your organisation during development, sales, use and support.

Vendors can benefit in: development, sales and support.

Purchasers can benefit in: use and support.

Systems developed for in-house use can benefit in: development, use and support.
5.1 Development
What savings will be made as a result of:

Reduced development time and cost to produce a product which has only relevant
functionality and needs less late changes to meet user needs?

Reduced cost of future redesign of the architecture to make future versions of the product
more usable?
5.2 Sales
What increase in revenue will result from:

Increased competitive edge: customers expect products to be easy to use. What
percentage increase in sales can be obtained by marketing your product as easier to use
than the competition?

More satisfied customers: difficult to use products create dissatisfied customers. What
percentage increase will there be in repeat customers more satisfied with the current
product?

Higher ratings for usability in the trade press?
Customers will be more satisfies as a result of the potential savings during use listed below.
5.3 Use
What savings will be made as a result of:

Reduced task time and increased productivity?
This can be estimated by measuring efficiency in a usability test.

Reduced user errors that have to be corrected later?
This can be estimated by measuring effectiveness in a usability test.

Reduced user errors leading to an improved quality of service?
Page 11
© 2000 Serco Ltd
Version 1.1 7-Mar-16
Cost benefit analysis
TRUMP D3.0
This can be estimated by measuring effectiveness in a usability test.

Reduced staff turnover as a result of higher satisfaction and motivation?
This can be estimated by measuring satisfaction in a usability test.

Reduced training time for users?
5.4 Support
What savings will be made as a result of:

Reduced costs of producing training materials?

Reduced training time for trainers and users?

Reduced time spent by other staff providing assistance when users encounter difficulties?

Reduced help line support?
Page 12
© 2000 Serco Ltd
Version 1.1 7-Mar-16
Cost benefit analysis
TRUMP D3.0
6 Making the cost-benefit case
6.1 Calculating cost-benefits
The basic principle is to estimate the total cost of producing, maintaining, and selling the
product, and compare current costs with estimated costs after use of user centred design
methods.
The extent of the financial benefits will depend on how completely user centred design can
be implemented. A balance needs to be obtained so that a convincing case can be made for
benefits that are substantially larger than the costs of additional user centred activities.
Most user centred design techniques are relatively simple to apply. The major costs are the
time of the people who apply the methods. The methods chosen will depend not only on the
budget available, but also on the available skills and experience and practical constraints such
as project deadlines and the availability of users.
A full implementation of user centred design would conform to ISO 13407 and use all of the
relevant capabilities in the Usability Maturity Model (Earthy 1998). A minimalist approach
could implement the methods recommended in Getting started in user centred design (Bevan
1999) and only involve 10-15 person-days effort.
For more information on calculating the costs of methods, see Daly-Jones et al (1998) and
Bias and Mayhew (1994).
6.2 Example of how to calculate cost-benefits
After using a range of user-centred design techniques during the development of a system:

For each technique, sum the person days required to perform it:
(preparation time x people) + (application time x people) + (documentation time x people)

Multiply the person-days by the appropriate day rate(s) to give a total labour cost, and add
any other costs.

Decide which of the benefits listed in section 2 the technique contributed to.

Describe the specific benefits obtained from each technique.

Group together techniques that have been used in combination (typically at the same
stage of development) and provide shared benefits.

Estimate the financial savings resulting from the use of each inter-related group of
techniques (avoiding any double counting).

Calculate the cost-benefit ratio for each group of techniques, and the overall cost-benefit
ratio.
Page 13
© 2000 Serco Ltd
Version 1.1 7-Mar-16
Cost benefit analysis

TRUMP D3.0
To estimate the cost-benefits of individual techniques (particularly those regarded as of
marginal value) re-calculate the estimated cost benefits for a group of techniques when
one or more individual techniques has been excluded.
Use the results to establish a policy for which techniques should be used as a normal part of
development, and to decide how to tailor the selection of techniques to particular
development situations.
6.3 Other barriers to user-centred design
A cost-benefit case alone is not sufficient to introduce user centred design. Organisations
frequently do not budget for the total cost of producing, supporting and maintaining a
product. The development teams’ main priority is to complete development on time and on
budget. Organisations typically only prioritise usability if they suffer from lost revenues, lost
bids, dissatisfied customers, and high support costs. Introducing successful user-centred
design into a large organisation requires cultural and technical change as well as strategic
commitment:
 cultural: all those involved in the development of a system or product must be aware of
the issues and activities involved in user-centred design to effect the best design decisions
at a micro- and macro- level. Others affected by the system or product, for example endusers, who previously have not been involved in its development, need to be made aware
of their new role in design. They must also put into place mechanisms for achieving the
technical and cultural change outlined below.
 technical: development processes and procedures must include the user centred methods
and activities which are appropriate for the organisation and whose purpose and benefit
can be clearly demonstrated to the developers. Techniques are required for the selection
of appropriate methods for each project, and for the dissemination of lessons learned to
other projects in the organisation.
 strategic: the organisation and its management must set quality in use as a principal
objective for systems development, but until recently there has been no way to accurately
specify quality in use requirements prior to design. There is often a contractual
requirement to deliver a product that matches the specification, but if the specification is
not precise about quality in use, there is little incentive to make this a priority in design.
Product developers regard it as inevitable that different users will have different
perceptions of the quality of the product. As user perceived quality is regarded as a
subjective judgement outside the control of the developer, meeting the technical
specification becomes the sole objective of design.
Page 14
© 2000 Serco Ltd
Version 1.1 7-Mar-16
Cost benefit analysis
TRUMP D3.0
References
Bevan N (1995) Measuring usability as quality of use. Journal of Software Quality, 4, 115130.
Bevan N (1997a) Quality and usability: A new framework. In: van Veenendaal, E, and
McMullan, J (eds) Achieving software product quality, p 25-34, Tutein Nolthenius,
Netherlands.
Bevan N (1997b) Usability Issues in Web Site Design. In: Proceedings of HCI
Internationalí97, San Francisco, 24-30 August 1997.
Bevan N (1999) Getting started with user centred design.
http://www.usability.serco.com/trump/ucdstart/index.html
Bevan N and Azuma M, Quality in use: Incorporating human factors into the software
engineering lifecycle, in Proceedings of the Third IEEE International Software
Engineering Standards Symposium and Forum (ISESS’97), p169-179, 1997.
Bevan N and Curson I (1998) Planning and Implementing User-Centred Design, CHI’98
Adjunct Proceedings.
Bevan N and Macleod M, Usability measurement in context. Behaviour and Information
Technology, 13, 132-145, 1994.
Bevan N, Usability is quality of use, in Anzai & Ogawa (eds) Proc. 6th International
Conference on Human Computer Interaction, July 1995. Elsevier, 1995b.
Bevan, N (1999) Industry standard usability tests. In: Human-Computer Interaction –
INTERACT ’99 (Volume II). S. Brewster, A. Cawsey & G. Cockton (editors). British
Computer Society, Swindon, UK.
Bias, G. and Mayhew, D. (Eds) (1994). Cost-Justifying Usability. Academic Press,
NewYork.
Chalupnik and Rinehart 1992 cited at
http://wwwwseast.usec.sun.com:80/usability/benefits.html
Daly-Jones, O, Thomas, C, Bevan, N. (1997) Handbook of user-centred design. Serco
usability Services.
Earthy, J (1998) Usability Maturity Model: Processes. INUSE deliverable D5.1.4p,
http://info.lut.ac.uk/research/husat/inuse/f3_assurance.html
Manning, H, McCarthy JC, Souza RK (1998) Why most web sites fail. Forrester Report,
3(7). Forrester Research Inc, Cambridge, MA.
Gartner Group (1997). Cited in Gibbs (1997).
Gibbs, W.W. (1997) Taking computers to task. Scientific American, 7/97.
http://www.sciam.com/0797/issue/0797trends.html
Hurley, M (1997) cited in Gibbs, W.W. (1997)
IBM (1999) Cost Justifying Ease of Use http://www3.ibm.com/ibm/easy/eou_ext.nsf/Publish/23
INUSE, see http://www.npl.co.uk/inuse, 1998.
ISO 13407 (1999) User-centred design process for interactive systems.
ISO 9241-11 (1998) Ergonomic requirements for office work with visual display terminals
(VDT)s - Part 11 Guidance on usability.
ISO/IEC FDIS 9126-1 (2000) Software product quality - Part 1: Quality model.
ISO/IEC TR 15504, Software process assessment, 1998.
Page 15
© 2000 Serco Ltd
Version 1.1 7-Mar-16
Cost benefit analysis
TRUMP D3.0
Jones, T.O., and Sasser, W.E. (1995) Why satisfied customers defect. Harvard Business
Review, 6(1), 88-99.
Karat, C. (1990). Cost-benefit analysis of usability engineering techniques. Proceedings of
the Human Factors Society. Orlando. Fl.
Karat, C.M. (1992). Cost-justifying human factors support on development projects. Human
Factors Society Bulletin, 35 (11), 1-8.
Karat (1993a) Usability Engineering in dollars and cents. IEEE Software 10(3), 88-89
Karat (1993b) Cost-benefit and business case analysis of usability engineering. SIGCHI’93,
Amsterdam.
Karat, C. (1997). Cost-justifying usability engineering in the software life cycle. In
Helander, M., Landauer, T., and Prabhu, P. (Eds), Handbook of Human-Computer
Interaction. Elsevier Science, Amsterdam.
Keil, M, & Carmel, E. (1995). Customer developer links in software-development.
Communications of the ACM. 38(5), pp33 - 44.
Kitsuse, A (1991) Why aren't computers... Across the Board (October) 28, 44-48
Landaur (1995) The Trouble with Computers. MIT Press
MacIntyre, F. Estep, K.W., and Sieburth, J.M. (1990). Cost of user-friendly programming.
Journal of Forth Application and Research 6(2), 103-115
Martin and McClure (1993) cited at:
http://wwwwseast.usec.sun.com:80/usability/benefits.html
Prokesch, 1995 Competing on customer service. Harvard Business Review, NovemberDecember 1995.
Pressman, R.S. (1992). Software Engineering: A Practitioner's Approach. McGraw Hill, NY.
Reed, S. (1992) Who defines usability? You do! PC Computing (Dec), 220-232
RESPECT, see http://www.npl.co.uk/respect, 1998.
Standish Group (1995) CHAOS Report. http://www.standishgroup.com/chaos.html
Thomas C and Bevan N, eds (1995) Usability Context Analysis: A practical guide, Version
4. Serco Usability Services, Teddington, UK
Wixon et al 1994
Wixon, D. and Jones, S. (1995). Usability for fun and profit: A case study of the re-design of
the VAX RALLY. In: Human-Computer Interface Design: Success Stories, Emerging
Methods, and Real-World Context. Marianne Rudisill, Clayton Lewis, Peter G. Polson,
Tim McKay (eds) Morgan Kaufmann Publishers.
Page 16
© 2000 Serco Ltd
Version 1.1 7-Mar-16