Measuring Website Usability:
Instrument Development, Validation,
and Application
Big XII IS Research Symposium
April 5, 2003
Younghwa “Gabe” Lee
University of Colorado at Boulder
leey@colorado.edu
Agenda
 Background
 Previous Studies
 Research Objective
 Research Design
 Expected Contribution
 Discussion
Background
 Online business failures are increasing as customers turn
away from unusable or unfriendly sites. ‘Build it and they
will come’ mentality has led to the demise of e-commerce
sites when sites are too late, too buggy, or too complex
- Becker and Mottay, 2001
 In a poorly designed EC environment, users might be
uncomfortable with the uncertainty and ambiguity caused
by lack of interaction with websites.
-
Jahng et al, 2000
 The number of shoppers and total sales are still marginal,
mainly because of poor interfaces
- Jarvenpaa and Todd, 1997
Background
 Building a usable website is important since website is the
only source for online customers to touch, feel, search,
communicate, and experience the products or services
available at the online store
 Usable websites
 Build positive attitude (Singh and Dalal, 1999)
 Increase stickiness (Rettie, 2001)
 Increase revisit rates (Klein, 1998)
 Increase online purchase (Palmer, 2002)
 Increase performance (Nielsen, 2000)
 Provide more satisfaction (Lund, 1999)
Website Usability Research
How to measure?
Instruments
Development
 HCI
 Company-specific
 Industry Gurus
 Few IS researchers
Website
Usability
How to verify?
Design and
Testing
 HCI
 MIS SIG HCI
Previous Studies
 Measurement problems of website usability
 No consensus on the definition and dimensions of
website usability
 A number of single-item constructs
 Intuition and experience-based: Few efforts to develop
measurement using scientific methods
 HCI-oriented objective variables (error rate and
download time)
 No Investigation of the relationship between
website usability constructs
 No process model
Agarwal and
Venkatesh (2002)
Palmer (2002)
Kim et al. (2002)
• Content
• Ease of Use
• Promotion
• Made-for-the-medium
• Emotion
• Download Delay
• Navigability
• Information Content
• Interactivity
• Responsiveness
• Firmness
• Convenience
• Delight
McKinney et al.
(2002)
Zhang and
von Dran (2002)
• Access
• Usability
• Entertainment
• Hyperlinks
• Navigation
• Interactivity
• Content
• Enjoyment
• Privacy
• User empowerment
• Visual appearance
• Technical Support
• Navigation
• Credibility
• Organization
Download
Delay
Download
Delay
Navigability
Navigability
Information
Content
Perceived
Success
Information
Content
Interactivity
Interactivity
Responsiveness
Responsiveness
Palmer (2002)
Perceived
Success
Motivation of the Study
 Current inconsistency and incompleteness among
website usability measurement is the crucial
problem of website usability studies.
 There is very little in the way of concrete
measurement that tells us how good a website
really is. Current guidelines, methods, and metrics
do help to design better websites, but there is room
for improvement
- Tarasewich (2000)
Research Objective
 Develop measurement of website usability
 18 constructs and 62 instruments have been identified
 Investigate the causal relationship between website
usability constructs
 Revealed Causal Mapping approach (Nelson et al., 2000)
 Examine the effects of website usability constructs to
multiple dependent variables
 Satisfaction, purchase intention, revisit intention, actual purchase,
affect, and loyalty
 Investigate generalizability of the new measurement
and identify different causal maps under different
boundary conditions
 Gender, Product, Industry and Culture
Research Design

Instruments Development






Literature review
Interviews with web usability experts
 SUN, IBM, 37 Signals.com website designers
A Focus Group Study (IS-majored master-level subjects)
A Major survey to 400 Business undergraduate students
Exploratory Factor Analysis
Causal Relationship Between Website Usability
Constructs



Interviewed with experts (n = 20)
Interviewed with experienced online customers (n = 40)
Data Analysis suggested by Nelson et al. (2000)
Research Design

Effects of Website Usability Factors to Diverse
Dependent Variables




The effects of Price, Time, Scarcity, Convenience, Fun, Usefulness
will be examined together
CFA and Path Analysis will perform
Data Analysis: LISREL
Boundary Conditions





Compare websites with different gender-focused
Compare websites with different types of products (Hedonic vs
Utilitarian)
Compare websites with different cultures (U.S. vs Japan)
Compare websites with different stakeholders (Customers,
Designers, and Managers)
Data Analysis: PLS
Website Usability Constructs
Simplicity
Timeliness
Scope
Readability
Scanability
Consistency
Relevancy
Learnability
Content
Navigability
Privacy
Credibility
TelePresence
Security
Reliability/
Accessibility
Community
Flexibility
Interactivity
Causal Relationship between
Website Usability Constructs
Simplicity
Scanability
Timeliness
Scope
Readability
+
Consistency
Content
Relevancy
Learnability
+
Navigability
Privacy
-
Security
TelePresence
-
Credibility
+
Reliability/
Accessibility
+
Flexibility
+
Community
Interactivity
Effects of Usability Constructs to
Diverse Dependent Variables
Simplicity
Scanability
Timeliness
Scope
Readability
+
Consistency
Actual
Purchase
Revisit
Intention
Purchase
Intention
Satisfaction
Loyalty
Affect
Content
Relevancy
Learnability
+
Navigability
-
+
-
Privacy
Reliability/
Accessibility
Credibility
+
TelePresence
+
Community
Security
Flexibility
Interactivity
Price
Time
Scarcity
Convenience
Fun
Usefulness
Expected Contribution
 Develop new measurement of website usability
 provide a better means to evaluate website design quality
 Identify causal relationship between website usability
constructs
 Provide a deeper understanding of how online customers build their
usability perception
 Examine effects of website design factors to diverse
dependent variables
 justify the investment on website usability
 Investigate measurement’s generalizability and causal
maps under different boundary conditions
 help to perform future study using the measurement
 help designers to allocate limited resources to the most important
usability factors
Discussion
Discussion
Revealed Causal Mapping
Repertory Grid
Causal Relationship Identification
Construct Identification
Guided Interviews
Open Interviews
No requirement for multiple elements
Multiple elements are required
Identify constructs and causal
Relationship based on repeated
Interviews for identifying
Identify constructs using
Triadic Methods to identify
Mid-theory exists
No requirement for mid-theory
Discussion
 Alternative ways of conducting the research
 Process Model (e.g TAM) vs Multiple Dependent Variables
 Boundary conditions valuable to be observed