Cost-Benefit Analysis for HCI
Mark W. Newman
University of Michigan School of Information
mwnewman@umich.edu
University of Michigan
Human Factors Engineering Short Course
2011
Learning Objectives
• How to argue for the value of user-centered design
– General arguments for incorporating usability methods
– Cost/benefit projections for specific projects
User-Centered Design
• Involve users throughout process
– At beginning: Needs assessment
• e.g., contextual inquiry
– Throughout: Iterative design & evaluation
• e.g., user testing, heuristic evaluation
– After deployment: Feedback for next version
• e.g., surveys, log analysis
• Incorporates User Experience Research and Interaction
Design
You Don’t Need User-Centered Design
You Don’t Need User Centered Design
• … if you’re a genius
Alternatives to UCD1
• I’m a user, and I can use it
• This thing is so new and different there are no way
users can evaluate it
• This technology is so awesome that people will be
willing to learn it
• We’ll just release it and fix any problems in the next
release
• Users just want features, so we need to devote all
resources to implementation
1 These
are not recommended
The Bottom Line
General Arguments for UCD
• External ROI
– Increased sales
– Decreased support
– Increased perception of
value
– Early identification of
problems
– (Loyalty)
General Arguments for UCD
• Internal ROI
–
–
–
–
Increased productivity
Decreased support
Decreased errors
Decreased training
costs
– Early identification of
problems
– (Morale)
Specific Arguments:
Costs vs. Benefits
• Benefits
– How much will we save?
– How much will we make?
– What risks will be reduced?
• Costs
– What resources will be required?
• Human effort
• Recruiting, compensation, equipment, etc.
– How long will it take to complete?
• Effects on time-to-market
• Lost opportunities
Metrics for Cost/Benefit
• ROI*
V f - Vi V f
ROI =
=
-1
Vi
Vi
– ROI > 0: profit
– ROI < 0: loss
– ROI = 1: 100% profit (return is 2x investment)
• Payback
– Time to recoup initial investment
– Payback time = Vi / Vp (expressed in time period p)
*: we are ignoring the fact that Vf could generate interest if not invested here.
High-level example
• You believe a user-centered design project will
result in a sales increase of 5%
– Current sales are $100,000 per month
– The project will take 6 months and cost $50,000
• What is the payback period?
– Vi = $50,000; Vp = $5,000
– Payback = Vi/Vp = 10 months
• What is the ROI after 1 year? 3 years?
– Vf_1yr = 60,000; Vf_3yr = $180,000
– Vf_1yr/Vi – 1 = 0.2 (20%); Vf_3yr/Vi – 1 = 2.6 (260%)
Usability ROI: Evidence
• 53 redesign projects with
before-and-after metrics
– 66 measurements in all
Metric Class
Improvement
Sales/conversion rate (20 cases)
87%
Traffic/visitor count (13 cases)
96%
User performance (14 cases)
119%
Feature use (10 cases)
223%
Overall average
114%
(Note: 8 outliers with extreme improvements were removed)
Usability ROI: Decline over time
• Data represents two studies: 2000 & 2006
– 2000: 135%
(Aggregate: 114%)
– 2006: 83%
– Excludes huge gains (5 measures 900%+ & 3 measures ∞%)
• Possible reasons for lower numbers in 2006
– Low hanging fruit has been picked
– Usability budgets have not increased as web has gotten
“better”
Has the web gotten better?
 Avg sales conversion rate in 2000: 1%
 Avg sales conversion rate in 2006: 2%
Different Benefits
Packaged SW/HW
(e.g., Apple, Adobe)
✔
Content (e.g., Huff
Post, WebMD)
✔
✔
Intranet (e.g.,
Wolverine Access)
Web Presence (e.g.,
SI website, UMHS)
✔
Registration
✔
Return Visitors
Support Costs
✔
Training Costs
Visitors
✔
Ad Clickthrough
Purchase Amount
✔
Page Views
Conversion Ratio
✔
Process Efficiency
Sales
eCommerce (e.g.,
Amazon, Zappos)
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
Estimating the Costs
• What work will be done?
– Selecting methods
– Estimating amount of effort for each method
• How much does it cost?
– How much do people get paid?
– Are there other costs?
Example Project: Baconfreak.com
•
•
•
•
Daily Visitors: 10,960
Conversion rate: 1%
Average sale: $28.40
The plan: We’re going
to rework the site to
have a “modern look”
Development Plan
• Designer will make info architecture, page layouts,
and graphics
• Software developers (3) will design and implement
the functionality
• Database manager will redesign database
• QA people (2) will test for bugs
• Project manager will keep things on track
• It will take 6 months:
– 1 month for up front system design, site architecture, page layouts
– 5 months for implementation of website code and database changes
– 2 months for QA at end, overlapping with development
Your Proposal
• Add 2 months at the front to provide for:
– Interviews/Contextual inquiry
– Survey
– Comparative analysis
• Hire Usability Specialist for 8 months (total) to
oversee
–
–
–
–
–
Initial Needs-Finding
Low-fi prototype and test
Heuristic evaluation(s)
User test(s)
Web analytics setup
Original Development Plan
Design
Development
QA
Original Development Plan
Design
Development
QA
UX Research
Added Costs
• Existing costs
–
–
–
–
Project manager for 6 months
Designer for 3 months
3 Developers for 5 months
2 QA people for 2 weeks
• Added costs
– 8 months of UX Researcher salary
*http://www.simplyhired.com/a/salary/search/q-user+experience+researcher/l-ann+arbor,+mi
Revised Budget
Role
FTE
Salary
Overhead
Cost
Project
Manger
6/12
80,000
50%
60,000
Designer
3/12
80,000
50%
30,000
Developers
15/12
(3@5mo)
90,000
50%
179,550
Database Mgr 5/12
70,000
50%
43,750
QA Engineers 4/12
(2@2mo)
60,000
50%
29,700
Total
343,000
50%
80,400
Total
423,400
(23%)
UX
Researcher
8/12
80,000
Projected Benefits
• User-centered process yields
– Conversion increase of 87%
– Traffic increase of 96%
• Generously assume that the non-UX development
process would yield 50% increase in each
• What is the ROI after 1 year?
• What is the payback period?
Previous
W/O UX
W/UX
Conversion rate 1%
1.5%
1.87%
Visitor Count
10,960
16,440
20,500
Avg sale
$28.40
$28.40
28.40
Daily revenue
$3,113
$7,003
$10,887
Yearly revenue
$1,136,090
$2,556,256
$3,973,806
1 year value
(Vf)
0
$1,420,166
$2,837,716
Investment (Vi)
0
$343,000
$423,400
ROI (1 year)
0
3.1
5.7
.24 yr (3 mo)
.15 yr (7.5 wk)
Payback
Usability Budgets: Best Practices
• Conducted at “Usability Week 2006” conferences in
NY, SF, Sydney, London
• Based on 143 projects
–
–
–
–
–
Avg project size: 9 person-years
Avg usability investment: 6 person-months (~1040 hrs)
Median usability investment: 10%
Mean usability investment: 17%
Regression model:
usability _budget = 0.343project _ size 0.63
(units: person months) (note: budget grows roughly as square root.)
Other Considerations
• Value of incorporating usability earlier into process
– Gilb’s $1 - $10 - $100 rule
• Cost of fixing problems increases by an order of magnitude as you
progress from design – development – release
Serious Risks: Accuracy Matters
Although Democrats
are listed 2nd, they are
the 3rd hole
Punching the 2nd hole
votes for the Reform
party
• 5,330 voters doublepunched Gore &
Buchanan
– 2,908 Gore/McReynolds
– 1,631 Bush/Buchanan
• 3,407 (0.76%) voted for
Buchanan
– pre-election polls
predicted ~600
• Bush won Florida by 537
votes
Horizontal lines lead the
eye to the wrong holes,
causing additional
confusion
[Tognazzini 2001]
Serious Risks: Therac 25
• Radiation therapy machine
– Electron mode: low power, short burst
– X-ray mode: high power, longer duration,
concentrated on target
Serious Risks: Therac 25
• 6 accidents; 4 fatalities (85-87)
• Usability flaws + Software bugs resulted in huge
doses of radiation (100x expected amts)
– Minor malfunctions were common (~40 per day)
– Easy to restart after failure (Press “P”)
– Common path through system led to error state
(“Cursor Up” to change modes, followed quickly by “B”
to start treatment)
– Operators unable to interpret cryptic “Malfunction 54”
message
[Levenson 95]
UCD and Risks
• UCD is helpful for certain kinds of risks:
– Helps anticipate expectations, skills, existing practices of
users
– Avoid likely misconceptions and sources of error
– Reduce engineering risks: feature creep, fluid requirements
– Reduce likelihood of user rejection
• However, UCD does not protect against rare events
– Formal analysis methods are recommended for safety
critical features!
Thimbleby 2007, Boehm 1991
Parting Thoughts: The Long View
• Wherever possible
– Collect metrics before and after UCD-based redesign
– Institute an ongoing measurement process
– Insist on similar metrics for other projects
Parting Thoughts: The Method Muddle
• User-centered design is our best defense against
getting it wrong (and best chance of getting it right!)
• This lecture treats user-centered design as a black box
• Different methods are good for different things
• During the rest of this course, ask:
– What risks does this method help minimize?
• Wrong requirements, learnability, poor performance, likelihood of
errors, user resistance
– Where could this method fit into an iterative process?
• Early, middle, late, post-deployment
References
•
•
•
•
•
•
•
•
Nielsen, J., & Gilutz, S. (2008) Usability return on investment, 3 ed. available at
http://www.NNgroup.com/reports/roi
Bias, R. G. and Mayhew, D. J. (2005) Cost Justifying Usability: An update for the
Internet Age. San Francisco, CA: Morgan Kaufman.
Usability First: Usability ROI: Case Studies.
http://www.usabilityfirst.com/roi/studies.txl [retrieved 7/1/08].
Nielsen, J. Return on Investment for Usability (Jakob Nielsen’s Alertbox).
http://www.useit.com/alertbox/roi-first-study.html [retrieved 7/1/08]
Nielsen, J. Usability ROI Declining, but Still Strong (Jakob Nielsen’s Alertbox).
http://www.useit.com/alertbox/roi.html [retrieved 7/1/08].
Nielsen, J. Do Government Agencies and Non-Profits Get ROI from Usability?
http://www.useit.com/alertbox/government-nonprofit.html [retrieved 7/1/08].
Usability Professionals Association. Resources for Selling Usability
http://www.upassoc.org/usability_resources/usability_in_the_real_world/selling
_usability.html. [retrieved 7/1/08].
Donoghue, K. (2002) Built for Use: Driving Profitability Through the User
Experience. McGraw Hill.
References
•
•
•
•
Thimbleby, H. 2007. User-centered methods are insufficient for safety critical
systems. In Proceedings of the 3rd Human-Computer interaction and Usability
Engineering of the Austrian Computer Society Conference on HCI and Usability
For Medicine and Health Care (Graz, Austria). A. Holzinger, Ed. Lecture Notes In
Computer Science. Springer-Verlag, Berlin, Heidelberg, 1-20.
Boehm, B. W. 1991. Software Risk Management: Principles and Practices. IEEE
Software 8(1). January 1991
Levenson, N. Safeware: System Safety and Computers. Addison-Wesley. 1995.
(contains a chapter on the Therac-25 accidents)
Tognazzini, B. The Butterfly Ballot: Anatomy of a Disaster. AskTog, January, 2001.
http://www.asktog.com/columns/042ButterflyBallot.html
These slides
http://mwnewman.people.si.umich.edu/courses/hfsc2011
Rohrer, C. http://www.useit.com/alertbox/user-research-methods.html
What People Do
What People Say
Rohrer, C. http://www.useit.com/alertbox/user-research-methods.html
Rohrer, C. http://www.useit.com/alertbox/user-research-methods.html
Rohrer, C. http://www.useit.com/alertbox/user-research-methods.html
Other considerations:
• Should I employ users at all?
• Where should method go in this
process?
• What prerequisites (e.g.
prototypes) do I need for the
method?
Rohrer, C. http://www.useit.com/alertbox/user-research-methods.html