Empirically Validated Web Page
Design Metrics
Melody Y. Ivory, Rashmi R. Sinha, Marti A. Hearst
UC Berkeley
CHI 2001
The Usability Gap
196M new Web
sites in the next 5
years [Nielsen99]
~20,000 user
interface
professionals
[Nielson99]
2
Empirically Validated Web Page Design Metrics
The Usability Gap
196M new Web
sites in the next 5
years [Nielsen99]
A shortage of
user interface
professionals
[Nielson99]
Most sites have inadequate
usability [Forrester, Spool, Hurst]
(users can’t find what they want 3966% of the time)
3
Empirically Validated Web Page Design Metrics
The Usability Problem


NON-professionals need to create websites
Guidelines are helpful, but
–
There are MANY usability guidelines


Survey of 21 web guidelines found little overlap [Ratner et al. 96]
Why?
–
–
–
4
One idea: because they are not empirically validated
Sometimes imprecise
Sometimes conflict
Empirically Validated Web Page Design Metrics
Possible Solutions:
Tools to Help Non-Professional Designers

Examples:
–
A “grammar checker” to assess guideline
conformance


–
–

5
Imperfect
Only suggestions – not dogma
Automatic comparison to highly usable pages/sites
Automatic template suggestions
How to create these?
Empirically Validated Web Page Design Metrics
Current Design Analysis Tools

Some tools report on easy-tomeasure attributes
–
Compare measures to thresholds


–
Guideline conformance

–
Faraday (Design Advisor)
Small subset of features [Brajnik00]
–
–
6
CAST (Bobby), Scholtz & Laskowski
(WebSAT), Lift Online
Perceptually based heuristics


Stein (Rating Game), Theng & Marsden,
Thimbley (Gentler)
Not empirically validated
Simplistic
Not empirically validated
Empirically Validated Web Page Design Metrics
The WebTANGO Approach

Models of good design by looking at existing
designs
–
–

7
Empirical foundation for easy-to-measure attributes
Focus on information-centric sites
First work to take a large set of sites and
analyze them
Empirically Validated Web Page Design Metrics
The Investigation

Using quantitative measures to predict Web site ratings
–

Given
–
–

1898 pages from 400+ sites
11 quantitative measures to assess various Web page aspects
Questions
–
–
–
8
Followup investigation [HFW00]
Which features distinguish well-designed web pages?
Can metrics predict ratings?
Are there differences for categories of pages?
Empirically Validated Web Page Design Metrics
Webby Awards 2000

2000 sites initially
–
27 topical categories

We studied sites from information-centric categories
–

100 judges
–
–
9
Finance, education, community, living, health, services
International Academy of Digital Arts & Sciences
3 rounds of judging
Empirically Validated Web Page Design Metrics
Webby Awards 2000

6 criteria
–
–
–
–
–
–



10
Content
Structure & navigation
Visual design
Functionality
Interactivity
Overall experience
Scale: 1-10 (highest)
Nearly normally distributed
What are judgments about?
Empirically Validated Web Page Design Metrics
Webby Awards 2000



Content criterion is best
predictor
Visual design criterion is
worst predictor
User study of 57 sites
–
11
Ratings reflect usability
Empirically Validated Web Page Design Metrics
Quantitative Measures:
Aspects Impacting Usability


Identified 42 attributes from the literature
Roughly characterized:
–
Page Composition

–
Page Formatting

–
fonts, lists, colors, …
Overall Page Characteristics

12
words, links, images, …
information & layout quality, download speed, …
Empirically Validated Web Page Design Metrics
Quantitative Measures:
Word Count
13
Empirically Validated Web Page Design Metrics
Quantitative Measures:
Body Text %
14
Empirically Validated Web Page Design Metrics
Quantitative Measures:
Emphasized Body Text %
15
Empirically Validated Web Page Design Metrics
Quantitative Measures:
Text Positioning Count
16
Empirically Validated Web Page Design Metrics
Quantitative Measures:
Text Cluster Count
17
Empirically Validated Web Page Design Metrics
Quantitative Measures:
Link Count
18
Empirically Validated Web Page Design Metrics
Quantitative Measures:
Page Size (Bytes)
19
Empirically Validated Web Page Design Metrics
Quantitative Measures:
Graphic %
20
Empirically Validated Web Page Design Metrics
Quantitative Measures:
Graphic Count
21
Empirically Validated Web Page Design Metrics
Quantitative Measures:
Color Count
22
Empirically Validated Web Page Design Metrics
Quantitative Measures:
Font Count
23
Empirically Validated Web Page Design Metrics
Characterizing Measures:
A View of Web Site Structure

Information design
–

interaction with information
structure
Graphic design
–
24
structure, categories of information
Navigation design
–

[Newman et al. DIS00]
visual presentation
Courtesy of Mark Newman
Empirically Validated Web Page Design Metrics
Characterizing Measures:
Web Site Structure Assessed
Metric
Word Count
Body Text %
Emp. Body Text %
Text Positioning Count
Text Cluster Count
Link Count
Page Size
Graphic %
Graphics Count
Color Count
Font Count
Information
Design
Navigation Graphic
Design
Design
Study Method

The Webby factor
–
Principle components analysis of the 6 criteria


Two comparisons
–
Model 1: Highly rated sites (top 33%) vs. the rest

–
Using the overall Webby score
Model 2: Highly rated sites vs. bottom 33%

26
Accounted for 91% of the variance
Using the Webby factor
Empirically Validated Web Page Design Metrics
Findings

We can accurately classify web pages
–
–
Linear discriminant analysis
Model 1: For highly-rated sites vs. rest


–
Model 2: For highly-rated sites vs. bottom


27
67% correct when not considering content categories
73% correct when taking content categories into account
65% correct when not considering content categories
80% correct when taking content categories into account
Empirically Validated Web Page Design Metrics
Findings

Top vs. bottom
–


28
Webby factor
Linear
discriminant
analysis
Better for
categories
Empirically Validated Web Page Design Metrics
Deeper Analysis

Which metrics matter?
–
Linear regression analysis

–

All metrics played a role
Compared small, medium, and large pages
–
Across the board (preliminary profiles)



29
Backward elimination until adjusted R² reduced
Good pages had significantly smaller graphics percentage
Good pages had less emphasized body text
Good pages had more colors (on text)
Empirically Validated Web Page Design Metrics
Small pages (66 words on
average)

Good small pages have
(according to beta
coefficients)
–
–
–
–

Examples
–
Slightly more content
Smaller page sizes
Fewer graphics
More font variations
Suggests that these pages
–
Have faster download times

–
30

corroborated by a download
time metric
Use different fonts for
headings vs. the rest of the
text
Empirically Validated Web Page Design Metrics
Services (Home Pages)


Top
Bottom
Metric
Top Bottom
Word Count
114
58
Page Size
47227 54843
Graphic Count
18
21
Font Count
5
3
Medium pages (230 words on
average)

Good medium pages
–
–
–

Suggests that these pages
–
31
Emphasize less of the body text
Appear to organize text into clusters (e.g., lists and
shaded table areas)
Use colors to distinguish headings from body text
Are easier to scan
Empirically Validated Web Page Design Metrics
Large pages (827 words on
average)

Good large pages have
–
–
–
More headings
More links
Are larger but have fewer graphics


Suggests that good large pages
–
32
Probably attributable to style sheets
Are easier to scan and facilitate information seeking
Empirically Validated Web Page Design Metrics
Why does this work?


Content is most important predictor
BUT there’s predictive power in other aspects
–
–


Possibly: Good design is good design all over
Note: we are NOT suggesting we can
characterize:
–
33
Visual and navigation design
Verifies preliminary investigation [HFW00]
Aesthetics or subjective preferences
Empirically Validated Web Page Design Metrics
How might we use this?
Web Site Design
Comparable Designs
Profiles
Analysis
Tool
Favorite Designs
34
•Prediction
•Similarities
•Differences
•Suggestions
Empirically Validated Web Page Design Metrics
Future work

Distinguish according to page role
–

Better metrics
–


More aspects of info, navigation, and graphic design
Site level as well as page level
Category-based profiles
–
–
35
Home page vs. content vs. index …
Use clustering to create profiles of good and poor sites
These can be used to suggest alternative designs
Empirically Validated Web Page Design Metrics
In Summary


Automated tools should help close the Web Usability
Gap
We have a foundation for a new methodology
–

We can empirically distinguish good pages
–
–

36
Empirical, bottom up
Empirical validation of design guidelines
Can build profiles of good vs. poor sites
Eventually build tools to help users assess designs
Empirically Validated Web Page Design Metrics
More Information

37
http://webtango.berkeley.edu
Empirically Validated Web Page Design Metrics