Information Design Journal 21(2), 129–145
© 2014 John Benjamins Publishing Company
D O I : 10.175/idj.21.2.05wes
Hans Westerbeek, Marije van Amelsvoort, Alfons Maes & Marc Swerts
Effects of cognitive design principles on user’s
performance and preference
A large scale evaluation of a soccer stats display
Keywords: efficiency, information design, natural
mapping, preference, visual variables
We present an analytic and a large scale experimental
comparison of two informationally equivalent information displays of soccer statistics. Both displays were
presented by the BBC during the 2010 FIFA World Cup.
The displays mainly differ in terms of the number and
types of cognitively natural mappings between visual
variables and meaning. Theoretically, such natural formmeaning mappings help users to interpret the information
quickly and easily. However, our analysis indicates that
the design which contains most of these mappings is
inevitably inconsistent in how forms and meanings are
mapped to each other. The experiment shows that this
inconsistency was detrimental for how fast people can
find information in the display and for which display
people prefer to use. Our findings shed new light on the
well-established cognitive design principle of natural
mapping: while in theory, information designs may
benefit from natural mapping, in practice its applicability
may be limited. Information designs that contain a
high number of form-meaning mappings, for example,
for aesthetic reasons, risk being inconsistent and too
complex for users, leading them to find information less
quickly and less easily.
1. Introduction
Soccer is one of the most popular sports worldwide.
Print and online media present reports to communicate
what happened during important games. Often, these
reports contain displays that visualize quantitative
information about the games, such as possession of the
ball, free kicks taken, and goals scored. Such displays
come in visually attractive formats, designed not only
to be clear and efficient, but also to be engaging and fun
to consult. In this paper, we analyze and experimentally
evaluate two examples of such displays, focusing on how
efficiency and use are affected by their design.
During and after the 2010 FIFA World Cup, the BBC
presented two different displays of soccer stats (British
Broadcasting Corporation 2010). There was an innovative field-based display (reproduced in Figure 1), and a
more conventional and simple list-based one (Figure 2).
Both displays show the same statistics (i.e., they are
informationally equivalent (Larkin & Simon 1987), but
do so in different ways. The field display is more “realistic” as it shows some analogies to the real world: it looks
like a football field, and there is a clock in the middle of
the field. A more general difference between the displays
is that the field display uses more design features to
convey meaning. Information elements (e.g., the number
129
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
Figure 1. Top: the field display, as used in our analysis and
experiment. Bottom: the “more stats” panel.
130
idj 21(2), 2014, 129–145
Figure 2. The list display, as used in our analysis
and experiment.
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
of corners) are defined by using different colors, different sizes, and different forms, and the positioning of
these elements also plays a role in their interpretation.
These differences raise the question: which of these
visualizations is better? Do the added design features and
analogies in the field display help users to find information more quickly, or do they lead to a “design overload”
that may be more pleasurable, but harder to interpret?
The goal of asking these questions, however, is
not to propose a more efficient, more pleasurable,
and easier-to-interpret redesign of the stats display.
Instead, the relevance of our question goes beyond
these two displays and soccer (stats). It touches upon a
more fundamental question in data visualization and
human cognition, namely, how complex real-world
data should be visualized, tailored to the workings of
human perception (Larkin & Simon 1987; Hegarty 2011;
Tversky 2011a; Zhang & Norman 1994). Our research
focuses on the theoretical assumption that natural
mapping in an information display is beneficial for its
users. According to this assumption, an information
display that is designed in such a way that elements
in the display express a meaning that is cognitively
natural is more efficient to use than a display that is
not designed in that way. Cognitively natural design is
design that exploits the workings of human perception
(Tversky, Kugelmass & Winter 1991; Hegarty 2011). For
example, by making a visual element larger, it naturally
expresses something that is “greater than” or “more”
than an element that is smaller. As shown in the lower
panel of Figure 1, for instance, the cross that displays
10 fouls of one team is smaller than the cross displaying
the 12 fouls of the other team. Another example of
cognitively natural design in Figure 1 is that elements
of one team are shown in another color than elements
of the other team. By giving things different colors, the
design naturally conveys that these things belong to
different categories.
idj 21(2), 2014, 129–145
This idea of natural mapping is proposed in the
literature in several ways, ranging from practical design
advice to the cognitive underpinnings of the visual
system. Designers are advised to map the visual appearance of elements in a display to meaning in a way that
is “compatible” with visual perception (Kosslyn 2006)
or that “capitalizes on the human facility for processing
visual information” (Agrawala, Li & Berthouzoz 2011;
Vande Moere & Purchase 2011). Cognitive scientists
argue that natural design-meaning mappings are
those that have their origins in the body and the world
(Tversky et al. 1991; Tversky 2011b). For example, a large
quantity of something in the everyday world takes up
more space, so a visualization of a large quantity should
be sized accordingly. This approach is closely related to
Conceptual Metaphor Theory (Lakoff & Johnson 1980):
natural design-meaning mappings are grounded in
everyday perception and action. The naturalness of some
mappings has also gained support in experiments in
which people produce visualizations, which show that
people converge on mappings of time, places, and people
(Kessell & Tversky 2008, 2011) and of quantities or
qualities (Tversky et al. 1991). This makes it conceivable
that information can be displayed in ways that are more
natural than others. As such, an information display
that makes extensive use of natural mapping is easier
to understand and preferred by users over displays that
make little use of such techniques. However, it remains
unclear whether adding more of such mappings to a
display makes it better.
The beneficial effects of using such natural mappings
in information designs have been investigated using
relatively simple designs in experiments, such as single
charts (Zacks & Tversky 1999; Shah & Friedman 2011).
How the design principle of natural mapping scales up
to more complex designs remains largely unexplored,
however (see Hegarty 2011). In order to explore natural
mapping in a more complex, real-life display, we analyze
131
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
and compare the BBC’s soccer stats displays. These
displays, shown in Figures 1 and 2, differ in terms of
the amount of natural mappings they employ to convey
meaning. In our analysis, we will show that the field
display contains a considerable number of different
design features (i.e., differences in color, size, form, and
space), which are mapped onto different meanings (i.e.,
different teams, different quantities). Each information
element obtains meaning from a number of these
features, for which we use the term visual variables
(Bertin 1981; Carpendale 2003). The alternative,
list-based display represents the same information in
a design that uses a smaller amount of such designmeaning mappings.
A comparison of the displays is a benchmark test
of the general applicability of natural mapping as a
design principle and an exploration into what happens
when the number of mappings in a display is relatively
high. On the one hand, increasing the number of visual
variables in an information display may be “the more,
the better”: when more design-meaning mappings are
used to convey meaning to information elements, the
design provides users with more handles to understand
the meaning of the elements. This leads to a more
efficient display in which information can be found more
easily (more quickly) and that is enjoyable to use. On
the other hand, increasing the number of visual variables can lead to “overload.” We use the term overload
because interpreting the meaning of one information
element (e.g., the number of corners of a team) can
depend on considering as many as four visual variables
(color, size, form, and space). This may impede finding
information and could also discount appreciation for the
visualization. Furthermore, potentially, when a design
encompasses more design choices, the risk of applying
these choices inconsistently may increase.
To explore the effects of using many visual variables
in a richly designed information display on both
132
idj 21(2), 2014, 129–145
efficiency and user preferences, we compare the two
displays qualitatively and quantitatively. The qualitative
analysis describes the field display in terms of how
design choices convey meaning and compares this to the
list display. This analysis is aimed at providing a detailed
description of the two displays under discussion and
exploring possible advantages or problems of the designs.
In the quantitative comparison, we address the question
whether our findings from the qualitative analysis have
repercussions for actual efficiency of the displays and for
user preferences. Therefore, we conduct a large-scale user
evaluation experiment in which we measure how quickly
people can find information and which display they
prefer given a number of usage scenarios.
2. Analysis of the displays
This qualitative analysis is structured in terms of Bertin’s
(1981) description of visual variables (see Carpendale
2003 for a comprehensive overview). Bertin describes
how visual features of information elements can vary in
a display and how this variation can convey meaning.
Elements can, for example, vary in terms of color, size,
location, and form—where the latter, form, is the appearance of an element and encompasses what an element
looks like (i.e., its shape and texture). For example, in the
field display, fouls are represented as crosses, and goals
are visualized as little balls. As such, form also includes
any visual analogies (i.e., iconicity) in the element.
If elements differ on a visual variable, this difference
expresses meaning by defining what the elements
visualize (Bertin 1981; Tversky 2001, 2011a). In the
case of the soccer display, for instance, elements with
different colors belong to different soccer teams. So
color expresses meaning by defining different groups in
the data. But visual variables can express meaning on
different levels of precision (nominal, ordinal, interval,
ratio; see Tversky et al. 1991; Tversky 2001). A difference
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
in appearance between elements creates groups, which
is information on the nominal level. Meaning can also
be expressed on an ordinal level: the location of elements in the space of the visualization can bring order
to them, such that elements can precede or follow each
other. And when the distance between these elements
is meaningful, interval level information becomes
available (much alike Bertin’s associative and order
characteristics of visual variables; Carpendale 2003).
Changes in visual variables can also express information
on the ratio level by displaying proportions, for example,
in a segmented bar chart, where the sizes of segments
represent proportions (MacDonald-Ross 1977). The four
levels of precision (nominal, ordinal, interval, ratio) are
ordered inclusively (Tversky 2001): information that is
defined on one level of precision (e.g., interval) implies
definition on the previous levels as well (e.g., nominal
and ordinal).
The analyses below describe how different visual
variables (color, size, form, and space) are used to convey
meanings on different levels of precision (nominal,
ordinal, and interval) in the two soccer stats displays
under discussion. Tables 1 and 2 summarize these
design-meaning mappings for both displays.
2.1 The field display
The display presented in Figure 1 is reminiscent of an
actual soccer field, with information elements distributed on this field. On the center line is a clock displaying
events that occurred during the ninety minutes of
the game (goals, bookings, substitutions). Additional
information elements are available when a user clicks
on the “more stats” button in the bottom of the display.
This action reveals a panel that shows attendance, fouls,
free kicks, and offsides (Figure 1, lower panel). Table 1
summarizes how each visual variable adds meaning to
the individual information elements in the field display.
idj 21(2), 2014, 129–145
Below, we discuss the role of each of the four visual
variables in the display.
All information elements in the field display have a
color that corresponds to a soccer team. So color adds
information on the nominal level: it assigns elements
to each of the teams. This usage of color is consistent
throughout the display because it applies to every
element. Independent of this grouping of elements by
team, the color of red and yellow cards on the clock
is analogous to real soccer. Note that these cards are
grouped by team through colored lines that connect
them to the clock.
The size of elements in the field display represents
quantity. The rectangles that show shots on/off goal
are larger as the number of shots increases and thus
show interval information. In doing that, they also
imply ordinal information since it allows users to see
which team shot the most. The same holds for corners,
free kicks, and fouls (Figure 2). The use of size in the
display is, however, somewhat inconsistent since it is
not available for all elements (it does not apply to those
placed on the clock). And size is used to display ratio
information too, namely in the segmented bar chart (see
MacDonald-Ross 1977) in the center of the display that
shows possession of the ball.
Each type of element in the display has its own
distinct form, including the graphs in the “more stats”
panel (Figure 1, lower panel). So form provides meaning
on the nominal level. What is inconsistent about the
use of form in the display, however, is the relationship
between an element’s form and its meaning. This
relationship ranges from analogies with real soccer
(balls represent goals, little cards represent yellow and
red cards awarded in the game) to something that is
more symbolic (e.g., crosses for fouls, quarter circles
for corners) and to shapes that bear no visual similarity
to what they represent at all (e.g., circles for free kicks,
rectangles for shots on goal).
133
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
idj 21(2), 2014, 129–145
Table 1. Analysis of the field display
Visual variables
Element
Color
Size
Form
Space/location
Shots on/off goal
Nominal - color of the
element groups it by team
Ordinal - size of the
element increases
by quantity
Nominal - shots on/off
goal have got a unique
square shape
Nominal - left-right grouping
by team (relational), and
Nominal - location is related
to real soccer event (iconic)
Corners
Nominal - color of the
element groups it by team
Ordinal - size of the
element increases
by quantity
Nominal - corners have
got a unique circular
angle shape
Nominal - left-right grouping
by team (relational), and
Nominal - location is related
to real soccer event (iconic)
Goals
Nominal - color of the line
groups it by team
Nominal - goals have
got a unique icon (ball)
Ordinal - location on the
time circle stands for point
in time (relational)
Bookings
(yellow and
red cards)
Nominal - color of the line
groups it by team, and
Nominal - color of the card
icon corresponds to type
of booking
Nominal - bookings
have got a unique icon
(card)
Ordinal - location on the
time circle stands for point
in time (relational)
Substitutions
Nominal - color of the line
groups it by team
Nominal - subs have
got a unique symbol
(double arrow)
Ordinal - location on the
time circle stands for point
in time (relational)
Possession of
the ball
Nominal - color of the
bar segment corresponds
to team
Ratio - size of the bar
segment corresponds
to percentage
Free kicks
Nominal - color of the
element groups it by team
Ordinal - size of
the chart element
increases by quantity
Nominal - free kicks
have got a unique
shape (circle)
Nominal - left-right
corresponds with team
(relational)
Fouls
Nominal - color of the
element groups it by team
Ordinal - size of
the chart element
increases by quantity
Nominal - fouls have
got a unique shape (X)
Nominal - left-right
corresponds with team
(relational)
Offside
Nominal - color of the
element groups it by team
Ordinal - height of
the bar increases
by quantity
Nominal - offsides is
the only bar chart
Nominal - left-right
corresponds with team
(relational)
134
Nominal - left-right
distribution of percentages
(relational)
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
Each element has a location in the field display, and
this location bears meaning. The field display is a hybrid
display (Hegarty 2011: 449) because it uses space in two
ways. The elements displaying shots on/off goal and
corners are located iconically: their location is analogous
to locations on an actual football field, and space in the
display is thus used to represent space in the real world.
Location is, however, also used relationally: it groups
and orders all elements in the display. This hybrid usage
of space means that the meaning of the visual variable
space is ambiguous within the display.
Looking at the relational use of location only, more
inconsistencies become apparent. Elements are grouped
by team: Elements on the left of the field belong to one
team, and elements on the right belong to the other. This
left-right organization relative to the field is applied to
shots and corners. However, for fouls, free kicks, and
offsides, the left-right organization of elements is not
exerted relative to the field, but per pair of elements. The
left element of each pair corresponds to one team, the
right to the other. The location of these element pairs
relative to the field does not bear meaning.
Another relational use of space is found in how the
clock in the center of the display visualizes time. It is important to note that mapping time on space is arguably
very natural. For example, people make this mapping
spontaneously when producing visualizations (Tversky et
al. 1991). And when talking about time, people often use
spatial metaphors (Clark 1973; Lakoff & Johnson 1980),
for example, when we say that “something lies behind us”
or when something is “far away into the future.” It has
even been argued that mental representations of time
are essentially partly built out of representations of space
(Casasanto & Boriditsky 2008).
While mapping time on space is very natural, it
introduces additional inconsistencies in the meaning
of location in the field display. Besides the left-right
nominal grouping of elements, the clock in the center
idj 21(2), 2014, 129–145
of the display uses space to define ordinal and interval
aspects of elements. The location of event elements on
the clock (goals, bookings, and substitutions) expresses
the order in which events took place. The distance
between events on the clock is also meaningful since it
expresses temporal intervals between these elements.
Taken together, space is used to define all elements
in the display, but it is used in an ambiguous way. It
can be iconic (space means space) or relational (space
groups and orders), and this is different for different
elements. Furthermore, the grouping sometimes works
on the display as a whole and sometimes per pair of
elements. Finally, space is used to express both nominal
and interval information. Analogies with the real world
also play an inconsistent role when it comes to space in
the display. Some elements derive meaning from their
location on the football field, and the way in which
events are placed on a circle is analogous to how clocks
work. And then there is inconsistency in these analogies
as well. The location of elements on the field ignores the
fact that, in reality, soccer teams shoot on the other side
of the field, and that teams switch sides halfway through
a game. And unlike real clocks, a full circle in the display
does not make up sixty but ninety minutes.
2.2 The list display
The list display appears to be more abstract than the field
display since there are no salient visual analogies like
football fields and clocks in it. Although this display is
more text-reliant than the field display, and therefore
significantly fewer visual variables are used to express
meaning, still some forms of graphic organization play a
role in the list display.
The table-like structure of the list display organizes
information elements along two axes: the vertical (y) axis
and the horizontal (x) axis. The segmented bar charts on
the bottom end of the display also fit in this x/y structure.
135
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
All organization along the x axis is nominal since
information of one team is on the left and information
of the other team is on the right. This left-right grouping
by space is consistent throughout the whole display.
Space is only used in this relational way and does not
represent any other type of information (like it displays
time intervals in the field display). Comparable to the
field display, elements are consistently grouped by team
by using different colors for each team. Finally, size is
consistently used to display ratio-level information since
all segmented bar charts in the list display work along
the same principles. Table 2 summarizes the designmeaning mappings in the list display.
There is only one analogy to actual soccer in the
display, and that is in the display of bookings. These are
visualized with little red and yellow cards next to player
names in the list display, similar to what is the case in the
field display.
2.3 Conclusions qualitative analysis
From the analyses of the two displays, a generalizable
observation emerges. When visual variables are “stacked”
(i.e., a high number of visual variables is used to define
information elements), problems with consistency of
design-meaning mappings may become hard to avoid.
Furthermore, analogies in a display can lead to problems
as well. They can mislead because they may suggest that
they are important in defining elements in a display
(while they are not). It may be difficult to apply analogies
to all elements in hybrid displays as well, so analogies are
easily inconsistent in a display (i.e., some elements derive
meaning from visually salient analogies while other
elements do not).
We have established that the field display contains
more visual variables that convey meaning than the list
display, providing more handles to find information.
But it may also confuse users due to inconsistencies in
136
idj 21(2), 2014, 129–145
mappings and the applicability of analogies. Does this
have repercussions for users when they are searching for
information? The analyses raise a number of expectations about how easily users can find information in the
displays and about preferences that users might have.
These expectations are based on the observation that
resulted from our qualitative analysis: when many visual
variables express design-meaning mappings within
a single display, inconsistencies and ambiguities may
arise, which may slow down users in finding information. It may also influence preferences that users have
for different types of displays. The BBC’s soccer stats
displays form an interesting design case that we use to
attest these expectations.
So for these displays, we expect that finding and
combining information is faster when the list display
is used, relative to the field display. This is not just
because some information in the field display is on the
“more stats” panel and clicking that button obviously
takes some time, but because of the differences in the
information design between the two displays. Previous
studies suggest that natural mappings of design variables
to meaning in a display helps users to find information
efficiently (Tversky 2001, 2011a) and that such natural
mappings aid in making inferences (Kessell & Tversky
2011) and in comparing information elements to each
other. One might interpret these findings as leading to
the expectation that finding and combining information
in the field display is faster than in the list display since
the former display employs more natural mappings than
the latter. However, our observations in the qualitative
analysis leads to a contrary expectation. We predict that
the list display is faster than the field display because the
stacking of visual variables in the field display can lead to
inconsistency and ambiguity.
In comparing the displays, we take special interest
in how easily users can deduce information about
temporal aspects of the games. For such information,
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
idj 21(2), 2014, 129–145
Table 2. Analysis of the list display
Visual variables
Element
Color
Size
Form
Space/location
Shots on/off goal
Nominal - color of the
element groups it by team
Ratio - size of the bar
segment corresponds
to percentage
Nominal - left-right
grouping by team
(relational)
Nominal - color of the
element groups it by team
Corners
Nominal - color of the
element groups it by team
Ratio - size of the bar
segment corresponds
to percentage
Nominal - left-right
grouping by team
(relational)
Nominal - color of the
element groups it by team
Goals
Bookings
(yellow and
red cards)
Nominal - left-right
grouping by team
(relational)
Nominal - color of the card
icon corresponds to type
of booking
Nominal - left-right
grouping by team
(relational)
Substitutions
Nominal - color of the card
icon corresponds to type
of booking
Nominal - left-right
grouping by team
(relational)
Possession of
the ball
Nominal - color of the
element groups it by team
Ratio - size of the bar
segment corresponds
to percentage
Nominal - left-right
grouping by team
(relational)
Nominal - color of the
element groups it by team
Free kicks
Nominal - color of the
element groups it by team
Ratio - size of the bar
segment corresponds
to percentage
Nominal - left-right
grouping by team
(relational)
Nominal - color of the
element groups it by team
Fouls
Nominal - color of the
element groups it by team
Ratio - size of the bar
segment corresponds
to percentage
Nominal - left-right
grouping by team
(relational)
Nominal - color of the
element groups it by team
Offside
Nominal - color of the
element groups it by team
Ratio - size of the bar
segment corresponds
to percentage
Nominal - left-right
grouping by team
(relational)
Nominal - color of the
element groups it by team
137
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
the field display is expected to have a clear advantage
over the list display because the field display visualizes
temporal aspects of events that take place during a game
in its clock.
Our expectations about appreciation of the displays
and preferences that the users have for the displays in
certain usage contexts are more speculative. Because
the field display contains more design features than
the list display (i.e., it has more variation in visual
variables and it has salient likenesses to a soccer field),
this could mean that people appreciate it more and
prefer to use it when they want to be entertained. The
mapping of temporal aspects of the game on a spatial
representation (i.e., the clock) can lead to people
preferring the field display when they want to see the
time course of the game and figure out how the game
developed. The list display, on the other hand, may
appear simpler and more conventional and could be
preferred for tasks like getting a quick idea of the game
and remembering data.
We test our expectations in a large-scale user evaluation of the two displays.
3. Quantitative comparison of the displays
In the quantitative comparison, we address the question
whether the differences between the displays described
in the qualitative analysis have repercussions for the
efficiency and appeal of the displays when they are
actually used. Therefore, we compare how quickly users
can find information in the field display, compared to
the list dis­play, and whether they prefer one display
over the other.
3.1 Method
3.1.1 Participants. 539 Participants (210 females and 329
males, median age 23 years, range 8–74) volunteered to
138
idj 21(2), 2014, 129–145
take part in the study. They were recruited by students in
an introductory methodology course.
3.1.2 Materials. The BBC displays for three games played
during the actual 2010 World Cup served as the basis
for the experimental materials. In the previous sections,
we analyzed how the two displays differ in terms of the
levels of precision (nominal, ordinal, interval, ratio)
on which visual variables (color, size, form, and space)
express meaning. In the field display, the high amount
of mappings between visual variables and meaning has
the consequence that mappings become inconsistent or
ambiguous throughout the display. This can be seen in
Table 1: the meaning expressed by one visual variable
(one column in the table) is not the same for all information elements in the field display. Additionally, analogies
with reality sometimes add meaning. This is also
inconsistent though: only a few elements get additional
meaning by analogies with real football or clocks.
While the display’s most salient spatial characteristic is
that it looks like a soccer field, this analogy has a very
limited scope.
The list display, on the other hand, is very consistent
in the few design-meaning mappings that it employs.
Table 2 shows that the meaning of the different visual
variables is consistent throughout the list display: mappings for one visual variable (i.e., one column in Table 2)
are nearly identical for all elements. The list display does
not use salient analogies to the real world, such as a
football field and a clock.
A few slight alterations were made to the BBC
displays prior to the quantitive experiment. To make sure
that participants would not recognize the actual games
and base their answers on that knowledge, team names
were replaced by generic animal names (e.g., Wolves
vs. Bears), and player names were replaced by common
surnames. The displays were placed on a dark gray
background. To make the views fully informationally
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
equivalent, timings of bookings and the number of free
kicks were added to the list displays (these were not
present on the original BBC displays). A footnote was
written below the field display to emphasize that clicking
on the display would reveal more stats. Figures 1 and 2
show both displays for one of the games because they
were used in the experiment.
Fourteen questions about the games were composed.
To answer each question, participants were required to
find information in the displays. Four questions related
to information that was on the “more stats” panel in the
field display and, thus, required participants to click a
button to find the information. The other ten questions
inquired information that was directly available in the
two displays. Within this set of ten non-click questions,
we defined two subsets. There was one subset of three
questions which inquired temporal information, for
example, by asking in which half of the game the first
goal was scored. The second subset (six questions)
required participants to combine multiple information
elements by requiring a comparison to be made (e.g.,
deciding which team shot on goal the most) or inferences to be drawn (e.g., finding which player scored the
winning goal). Some questions required information
to be combined and inquired temporal aspects of the
games at the same time, while other questions did
neither of the two.
3.1.3 Procedure. The study was carried out through a webbased interface. The participants completed the experiment individually. They first read a general introduction,
telling them that they were going to take part in a study
about soccer. Then, the fourteen questions had to be
answered for one game using one of the two displays,
and then for another game with the other display. The
questions were answered one at a time: participants
typed an answer, and by pressing enter or clicking “next,”
the next question appeared on the screen. The order of
idj 21(2), 2014, 129–145
the displays and the games used were counterbalanced
throughout the experiment, and the order of the
questions was randomized for each participant and
display. The questions and the displays were presented in
a split-screen view with the soccer stats on the left and
the questions on the right. After answering all questions
using both displays, appreciation for the displays was
measured. The participants chose one of the two displays
based on three statements concerning clarity, usability,
and, completeness. Then, preferences were inquired:
participants chose a display on the basis of seven short
usage scenarios (e.g., “which display is better if you want
to see how the match developed?”).
3.2 Research design and statistical analysis
Search times were calculated by logging the time span
between the appearance of a question on the screen
and the appearance of the next question. Therefore, the
measured search times included reading the questions,
searching the display, and typing the answer. This was
the same throughout both conditions in the experiment.
Search times for the four questions that required
participants to click the “more stats” button in the field
display were not analyzed because this would only affect
search times in the field display, leading to a delay that
cannot be (solely) ascribed to the difference in visual
properties of the displays. For the other ten questions,
10,780 search times were recorded. We discarded all
search times times for incorrect answers (n = 844) and
correct responses that took longer than sixty seconds
(n = 269). This outlier procedure resulted in discarding
10% of the data. For each participant, we calculated mean
search times for the ten non-click questions and the
subsets of questions.
The search times were analyzed using repeated
measures ANOVA’s, with display and type of question
as within-participants factors, and search time as the
139
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
dependent measure. The appreciation and preference
measures were analyzed with chi-square tests against
equal proportions.
3.3 Results
3.3.1 Search times. Analysis of search times, shown in
Figure 3, revealed a significant effect of display used,
F (1,476) = 95.87, p < .001, η² = .168. Search times for answering questions using the field view (mean 16.5 s) were
slower than when the list view was used (mean 14.4 s).
We also looked at the mean search time for questions
that inquire temporal information since we expected
that the field display would have an advantage over the
list display because it visualizes time. This analysis of a
subset of the questions, however, revealed an opposite
effect of display used on search times, F (1,476) = 62.14,
p < .001, η² = .115): search times were slower when the
field display was used (mean 18.3 s) than when the list
display was used (mean 15.0 s).
Field display
Mean response time (seconds)
List display
20
10
0
All
questions
Time
questions
Inference
questions
Combination
questions
Figure 3. Mean search time (and standard deviation) per
display and question type.
140
idj 21(2), 2014, 129–145
Analysis of search times for questions that required
inferences or comparisons between elements again
revealed a main effect of the display that was used,
F (1,506) = 31.19, p < .001, η² = .058). This main effect was
qualified by an interaction with the type of question (display × question type, F (1,506) = 35.99, p < .001, η² = .066).
When answering a question required an inference to be
made, search times were slower when the field display
was used (mean 18.4 s) than when the list display was
used (mean 15.3 s), F (1,506) = 57.96, p < .001, η² = .103).
This difference between the two displays was not present
when answering a question required information
elements to be compared: the difference between search
times when the field display was used (mean 15.6 s) and
when the list display was used (mean 15.4 s) was not
significant, F < 1.
3.3.2 Appreciation of the displays. Analysis of the twoalternative forced choices concerning appreciation,
shown in Figure 4, revealed that the list display was
found to be useful by more participants than the
list display (χ²(1) = 79.14, p < .001). The same holds
for clarity (χ²(1) = 79.14, p < .001) and completeness
(χ²(1) = 5.14, p < .025).
3.3.3 Preference for the displays. Analysis of the preferences, shown in Figure 4, revealed that more participants
preferred the list display over the field display for three of
the seven usage scenarios. The list display was preferred
over the field display for “having an overview of the
match” (χ²(1) = 73.64, p < .001). The list display was not
only faster with regard to the search times, but more
participants preferred the list display of the field display
for wanting to “view the game quickly” (χ²(1) = 66.28,
p < .001). The list display was also preferred by most
people for “remembering the information” (χ²(1) = 22.74,
p < .001).
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
Field display
List display
Appreciation: which display is the most useful?
31%
69%
Appreciation: which display is the most clear?
31%
69%
Appreciation: which display is the most complete?
45%
55%
Preference: which display do you prefer when you want to…
… have an overview of the game?
31%
69%
… view the game quickly?
32%
68%
… remember the information?
40%
60%
… see the development of the game?
66%
34%
… watch the information for fun?
70%
31%
… explain the game to someone else?
46%
54%
… understand the game?
52%
48%
Figure 4. Preferences for the displays, expressed as the
percentage of users preferring each display.
Concerning the other four scenarios, in line with
our expectations users showed a preference for the field
display when it came to “seeing the development of the
game” (χ²(1) = 55.28, p < .001). There was also a preference for the field display for “watching the information
idj 21(2), 2014, 129–145
for fun” (χ²(1) = 80.37, p < .001). There was no significant
difference between the proportion of participants that
preferred the field display and those that preferred the
list display when it came to “explaining the game to
someone else” (χ²(1) = 2.88, p = .09) and “understanding
the game” (χ²(1) = 1.19, p = .28).
3.4 Conclusions quantitative experiment
The search time analyses revealed that the field display
led to significantly slower responses than the list display.
We expected that the field display would be slower
than the list display overall, but we did not expect that
this was also the case when participants had to answer
questions that inquired temporal aspects of the games.
No advantage of the field display’s visualization of time
was found in the search times—on the contrary, the field
display is slower than the list display.
We also compared search times for the two displays
for questions that required an inference to be made
(e.g., “which player scored the winning goal?”) and that
required multiple information elements to be compared
(e.g., “which team had the most shots on goal?”). We
found that the field display is slower than the list
display for answering questions that require inferences
(Kessell & Tversky 2011; Tversky 2001; Tversky 2011a).
But concerning questions that required information
elements to be compared, no significant difference in
search time was found.
Looking at appreciation and preferences, our
(tentative) expectations were supported by the data.
The list display was found to be more useful, clear, and
complete than the field display. The field display was only
preferred over the list display when it came to watching
the information for fun and for seeing the development
of the match. This latter preference contrasts with the
findings from the search time analysis: after using the
displays, participants indicated that they appreciate the
141
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
visualization of temporal information in the field display,
yet they do not seem to take advantage of it in terms
of efficiency.
4. General discussion
In this study, we have compared two real-world informationally equivalent displays of soccer stats (Figures 1
and 2) to address the more general question of how
data should be visually displayed. We have compared
the displays in a qualitative analysis and in a large-scale
quantitative user evaluation to examine the assumption
that cognitively natural design-meaning mapping in an
information display is beneficial for users.
The results of the qualitative analysis, summarized in
Tables 1 and 2, show that the displays differ in terms of
the number of design-meaning mappings they employ
and that having more of such mappings increases the
risk of using the same visual variable for different meanings. Also, some mappings do not apply to all information elements. To test whether these differences between
the displays have repercussions for the efficiency and
appeal of the displays when they are actually used, we
conducted a large-scale quantitative user evaluation of
the displays to measure performance (search times) and
preferences of the users.
The quantitative evaluation showed that the more
richly designed (and, as a consequence, inconsistent)
field-based display (Figure 1) led to slower search times
than the list display (Figure 2) when users were asked
to find information and answer questions about soccer
games. This is largely in line with our expectations since
our analysis of the displays yielded the observation that
the field display’s performance may be compromised
as a result of having inconsistencies and ambiguity in
design-meaning mappings. This also led to the field
display being slower for answering questions that
required inferences to be made. However, for questions
142
idj 21(2), 2014, 129–145
concerning temporal aspects of the soccer games, we
expected the opposite. Because the field display visualizes time on a clock, which is an arguably very natural
design-meaning mapping, we expected that the field
display would be faster for such questions. However, this
turned out to be reversed, and even for these questions,
the field display was the slower of the two.
The experiment also pointed out that users mostly
preferred the list display over the field display, except
when they want to see the development of the game or
when they want to see the stats for fun. Because of the
large number of participants in this evaluation (539),
we not only have high statistical power, but we can also
generalize over different cognitive styles, age groups, and
levels of expertise.
Our findings challenge the view that natural mappings of design features to meaning in information
displays always improve the effectiveness of a display
by making it more efficient and more fun to use
(Kosslyn 2006; Hegarty 2011; Tversky 2011a). Natural
mappings are useful, but we have seen that too many of
these mappings may in fact be detrimental for performance and preference. For example, the field display is
“more designed” but is also less efficient to use and not
always preferred over the simpler list display. So stacking
simple design principles into one design may counterfeit
the efficiency entailed by the individual principles.
This also raises some interesting new research
questions, which concern both the efficiency of information designs and preferences of users for such designs.
Our findings on efficiency suggest that there may be
some kind of a threshold in design choices: when some
number of natural mappings used to define information
elements is exceeded, a design can become less efficient
(i.e., users find information less quickly). In the displays
that we compared, we found that one information
element is defined by multiple visual variables, that
there may be inconsistencies in how these multiple
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
visual variables define the element, and that analogies
in a design may be misleading. To disentangle these
potential causes for a rich design to become less efficient,
experiments can be carried out that take performance
measures while systematically varying the number of
visual variables, consistency, and analogies in a design.
Can the differences in search times between the two
displays also be ascribed to the possibility that the list
display may be more conventional as a display of soccer
stats than the field display is? Users of information
displays apply knowledge of display conventions (i.e.,
display schema’s; Hegarty 2011: 454) when interpreting
an information display. So if the list display adheres to
these display schema’s where the field display does not,
this may greatly affect search times to the benefit of the
list display. We have looked into this by studying whether
the search times that we report may fluctuate as a factor
of a user’s familiarity with soccer and soccer stats. We
did ask participants in the quantitative analysis whether
they liked soccer and whether they were familiar with
soccer stats. However, we could not find reliable interaction effects between liking soccer (stats) and the display
that was used on search times or preferences. Future
research can address the question as to what extent
knowledge of particular visualization styles plays a role
in assessing the effectiveness of information displays.
We have found that a more richly designed information display is often not preferred over a simpler display
by users, but our data do not allow us to see which
aspects of the displays drive these judgments. Again,
further experiments are warranted that disentangle the
effects of different aspects of information designs (e.g.,
inconsistency, analogy) on user preferences. Additionally,
participants in our study expressed preferences after
having used both displays to find information about
soccer games, such that they can be compared on the
same merit. However, it may also be interesting to
compare these judgments to preferences of people who
idj 21(2), 2014, 129–145
have not (yet) used the designs since intuitions about
visual representations may not always be in line with
actual efficiency (Smallman & Cook 2011).
4.1 Implications for design
Mapping decisions on the color, size, form, and location
of information to meaning in a natural or compatible
way is a well-known adagium in the design literature
(Agrawala et al. 2011; Vande Moere & Purchase 2011).
However, our case study we shows that a design that
contains many of such natural mappings is not a better
design by definition. Efficiency and users’ preferences
were compromised by the richness of the design in
terms of the number of such design-meaning mappings employed.
This can be regarded as an argument in favor of more
minimalistic design: using fewer visual variables in a
design can make the design more efficient and more
likable. But it may be even more important to keep an
eye on the consistency of design-meaning mappings.
When many different design features are used to define
information elements on different levels of precision
(e.g., grouping, ordering), the risk of being inconsistent
in these mappings increases. A design that contains
many mappings, but is inconsistent, may not necessarily
lead to users finding the design more efficient and
more preferred.
Proposing a redesign for the information displays
that we analyzed and evaluated lies outside the scope
of the current research. However, our work does allow
us to recommend some more general guidelines for
information design, related to our findings concerning
the number of design-meaning mappings employed and
potential inconsistencies in design-meaning mapping.
The first guideline we formulate is to avoid stacking
visual variables in an information display. Our analyses
and evaluation suggest that when multiple visual
143
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
variables are used to assign meaning to an information
element, efficiency of the information display may be
compromised. A rule-of-thumb may be not to use more
than two visual variables (e.g, color, size, location, form)
to define an information element.
This guideline is advised to be used in correspondence with a second recommendation: Avoid inconsistent
design-meaning mappings. When a visual variable (e.g,.
color, size, and position) is used to assign meaning to an
information element (e.g., on a nominal or ordinal level),
choose these design-meaning mappings in concordance
with other mappings in the same display.
Although applying these guidelines to the field-based
display discussed in this paper probably leads to a more
efficient display (and better user evaluations), future
experimental research is needed to address the scope of
these guidelines and to investigate their generalizability.
Acknowledgements
We thank Annemarie Quispel and Lisanne van Weelden for
fruitful discussions on this research and helpful comments on
earlier versions of this manuscript.
Submission date: 8/22/14
Accepted date: 1/12/15
References
References
Agrawala, M., Li, W., & Berthouzoz, F. (2011). Design principles for
visual communication. Communications of the ACM, 54(4),
607–69. DOI: 10.1145/1924421.1924439
Bertin, J. (1981). Graphics and graphic-information-processing.
New York: Walter de Gruyter. DOI: 10.1515/9783110854688
British Broadcasting Corporation. (2010). Netherlands – Brazil.
BBC Sport World Cup 2010. Retrieved March 6, 2011,
144
idj 21(2), 2014, 129–145
from http://news.bbc.co.uk/sport2/hi/football/
world_cup_2010/8729486.stm
Carpendale, M.S.T. (2003). Considering visual variables as a
basis for information visualisation. University of Calgary,
Department of computer science, 2001–693–16.
Casasanto, D. & Boroditsky, L. (2008). Time in the mind: Using
space to think about time. Cognition, 106, 579–593. DOI:
10.1016/j.cognition.2007.03.004
Clark, H.H. (1973). Space, time, semantics, and the child. In T.E.
Moore (Ed.), Cognitive development and the acquisition of
language (pp. 27–63). New York, NY: Academic Press.
Hegarty, M. (2011). The cognitive science of visual-spatial
displays: implications for design. Topics in Cognitive Science, 3,
446–474. DOI: 10.1111/j.1756-8765.2011.01150.x
Kessell, A.M., & Tversky, B. (2008). Cognitive methods for
visualizing space, time, and agents. In G. Stapleton, J. Howse,
& J. Lee (Eds.), Theory and application of diagrams. Dordrecht,
The Netherlands: Springer.
Kessell, A.M., & Tversky, B. (2011). Visualizing space, time, and
agents: production, performance, and preference. Cognitive
Processes, 12, 43–52. DOI: 10.1007/s10339-010-0379-3
Kosslyn, S.M. (2006). Graph design for the eye and the mind.
Oxford, England: Oxford University Press. DOI: 10.1093/acprof
:oso/9780195311846.001.0001
Lakoff, G., & Johnson, M. (1980). Metaphors we live by. Chicago, IL:
University of Chicago Press.
Larkin, J.H., & Simon, H.A. (1987). Why a diagram is (sometimes)
worth ten thousand words. Cognitive Science, 11, 65–99. DOI:
10.1111/j.1551-6708.1987.tb00863.x
MacDonald-Ross, M. (1977). How numbers are shown: A review of
research on the presentation of quantitive data in texts. AV
Communication Review, 25(4), 359–409.
Shah, P., & Freedman, E.G. (2011). Bar and line graph comprehension: An interaction of top-down and bottom-up
processes. Topics in Cognitive Science, 3, 560–578. DOI:
10.1111/j.1756-8765.2009.01066.x
Smallman, H.S., & Cook, M.B. (2011). Naïve realism: folk fallacies
in the design and use of visual displays. Topics in Cognitive
Science, 3, 579–608. DOI: 10.1111/j.1756-8765.2010.01114.x
Tversky, B. (2001). Spatial schemas in depictions. In M. Gattis (Ed.),
Spatial schemas and abstract thought (pp. 79–111). Cambridge,
MA: MIT Press.
H. Westerbeek, M. van Amelsvoort, A. Maes & M. Swerts • Effects of cognitive design principles
Tversky, B. (2011a). Visualizing thought. Topics in Cognitive Science,
3, 499–535. DOI: 10.1111/j.1756-8765.2010.01113.x
Tversky, B. (2011b). Spatial thought, social thought. In T.W.
Schubert & A. Maass (Eds.), Spatial dimensions of social
thought (pp. 17–38). Berlin, Germany: De Gruyter Mouton.
Tversky, B., Kugelmass, S., & Winter, A. (1991). Cross-cultural and
developmental trends in graphic productions. Cognitive
Psychology, 23, 515–557. DOI: 10.1016/0010-0285(91)90005-9
Vande Moere, A., & Purchase, H. (2011). On the role of design in
information visualization. Information Visualization, 10(4),
356–371. DOI: 10.1177/1473871611415996
Zacks, J., & Tversky, B. (1999). Bars and lines: A study of graphic
communication. Memory and Cognition, 27, 1073–1079. DOI:
10.3758/BF03201236
Zhang, J., & Norman, D.A. (1994). Representations in distributed
cognitive tasks. Cognitive Science, 18, 87–122. DOI: 10.1207/
s15516709cog1801_3
About the authors
Hans Westerbeek is a PhD Candidate at the
Tilburg center for Cognition and Communica­
tion (Tilburg University). His multidisciplinary
research focuses on perceptual aspects of
information design, language production,
navigation, memory, and learning.
idj 21(2), 2014, 129–145
Marije van Amelsvoort (PhD 2006) is an
assistant professor at the Department of
Communication and Information Sciences
(Tilburg University). Her research is aimed at
understanding learning with visualizations,
argumentation, and computer-supported
(collaborative) learning.
Alfons Maes (PhD 1991) is a full Professor and
Head of the Department of Communication
and Information Sciences (Tilburg University).
His work includes research on multimodal
aspects of human communication with
studies on visual metaphor, visual and
linguistic saliency, gesture, visual health
communication, and multimodal navigation communication.
Marc Swerts (PhD 1994) is a full professor
at the Department of Communication and
Information Sciences of Tilburg University.
His research is concerned with the use
and function of prosody and nonverbal
features in human-human and humanmachine interactions.
Email: h.g.w.westerbeek@tilburguniversity.edu
145