APRIL
13-~s,
1996
CHi96
,r
Structuring Information With Mental Models:
A Tour of Boston
Lokuge
Stephen A. G i l b e r t
Whitman R i c h a r d s
MIT Media Lab
Brain & Cognitive Sciences
20 Ames Street
MIT, E10-120
MIT Media Lab
20 Ames Street
Cambridge, MA 02139
Cambridge, MA 02139
Cambridge, MA 02139
ishi@media.mit.edu
stephen@psyche.mit.edu
whit @ media.mit.edu
Ishantha
ABSTRACT
We present a new systematic method of structuring information using mental models. This method can be used both
to evaluate the efficiency of an information structure and to
build user-centered information structures. In this paper we
present the method using Boston tourist attractions as an
example domain. We describe several interfaces that take
advantage of our mental models with an activation spreading network. Multidimensional Scaling and Trajectory
Mapping are used to build our mental models. Because of
the robustness of the technique, it is easy to compare individual difference in mental models and to customize interfaces for individual models.
Keywords
Cognitive models, multidimensional scaling, visualization,
interaction design, evaluation.
INTRODUCTION
We all know that a curious person can more efficiently
absorb information when it is well structured than when it
is arbitrarily scattered. The question that every information
architect then asks is, "How might I best organize the information for that person?" This question contains three
issues: what structures are useful for organizing information in general, what structures are useful for organizing
that information, and what structures are useful for organizing information for that person.
Before continuing on to details, we offer definitions of
"information structure" and "mental model." By "information structure" we mean an arrangement of pieces of information. The arrangement might be a 2-D array on a table
or a screen, as in a card game or a spreadsheet. It might
also be a 1-D ordering of items, like a shopping list. It
could also consist of a set of information nodes with connecting association links, such as a hypertext. All the
pieces of information should belong to the same conceptual type category, such as numbers to add, tasks to do, or
words to remember. By "mental model" we mean not the
explanatory model offered by Johnson-Laird [5], but rather
a more general definition: the cognitive layout that a person uses to organize information in his or her memory.
THE EXAMPLEDOMAIN
Consider the map of Boston shown in Figure la. We have
chosen fifteen different sites of activity taken from a tour
guide book (listed in Figure lb). This set of information
pieces (or "stimuli," as we will call them) is a very highdimensional in feature space, and it has a relatively large
variance across people. That is, there are many features
To answer these above questions, we propose (i) collecting
experimental data from a number of subjects, (ii) analyzing
the mental models of those subjects with Multidimensional
Scaling (MDS) [7, 15, 16] and Trajectory Mapping (TM)
[4, 9, 13], and then (iii) using those models to design information structures. Such models should lead more quickly to
suitable interfaces rather than beginning with trial and error
explorations.
Permission to make digital/hard copies of all or part of this material for
personal or classroom use is granted without fee provided that the copies
are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication and its date appear, and notice is
given that copyright is by permission of the ACM, Inc. To copy otherwise,
to republish, to post on servers or to redistribute to lists, t~qutr~s specific
permission and/or fee.
CHI 96 Vancouver, B C Canada
¢ 1996 A C M 0-89791-777-4/96/04..$3.50
Figure la: Geographic layout of activities
413
~,,~| ql~ APRIL 1 3 - 1 8 ,
to illustrate the capabilities of Multidimensional Scaling
(MDS), first used by Torgerson [t6], Shepard [15], and
Kruskal [7], we collected data from two subjects for each of
these mental models (Figure 3). Note that the judgments
based on geographic similarity (3a) are completely different
from judgments based on content (3b). The input data for
MDS takes the form of pairwise similarity judgments, and
the output is an arrangement of the stimuli in a metric space.
Stimulus Set: Boston Tourist Attractions
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
1996
Sports Museum
Children's Museum
Science Museum
Aquarium
Swan Boats (boat tour of garden)
Newbury Street (elegant shopping)
Quincy Market (outdoor mall)
Trinity Church (historic site)
Magic Show
Salem (nearby historic town)
Harvard University
Museum of Fine Arts
Zoo
Fenway Park (baseball stadium)
Arboretum (nature preserve)
To clarify the procedure for obtaining these MDS plots, we
offer a simple example; consider a collection of seven black
and white circles of different sizes. In Figure 2a, these circles are arranged randomly along the edges of a similarity
matrix. The numbers show a subject's similarity rating on a
scale of 1 to 7. From these numbers the MDS algorithm
arranges the circles as shown in Figure 4, with neighboring
circles being the most similar.
Note: Activities 9-15 are outside the scale of the map.
Figure lb: The stimuli for the MDS and TM experiments.
that can be used to describe the stimuli, and people do so in
significantly different ways. As a contrasting example, a
deck of playing cards contains very few features, namely the
number of a card, its suit, and perhaps its color, and different people describe playing cards similarly.
MDS AND TM METHODS
A person who lives in Boston and knows the sites will have
at least two mental models of them, one based on their geographic locations, and one based on their content. In order
The distances between the points in the output is usually a
non-linear transformation of the values in the similarity
matrix. For Figure 3a, we asked the subjects for every pair
of stimuli, "On a scale from 1 to 7, how similar are stimuli
X and Y in terms of their distance from each other?" For
Figure 3b, we asked, "How similar are stimuli X and Y in
terms of their c o n t e n t or theme?" We then used KYST2 [6]
to run the MDS and generate the graphs shown.
As one might expect, the distance-based MDS plot is similar to the actual map of Boston, though somewhat warped;
the warping could stem from differing familiarity with the
sites [11] or from thinking of distance as travel time instead
of geographic distance. In the content-based MDS plot,
,OoOoo.
II
2
4
4
6
5
0
6
5
4
3
1
2
3
5
1
2
5
4
1
2
0
qll
0
®
4
Figure 2a: Typical input data for the MDS algorithm; similarity data for black and white circles of different sizes.
7 = very similar; 1 = very dissimilar.
414
Figure 2b: Typical input data for the TM algorithm;
extrapolations and interpolations for black and white circles of different sizes. Here are 12 of 21 possible pairs.
APRIL
1996
t3-~8,
Salem
~'~
9~
Magic Show
................ ~ t ....................... t _ _ _
@
..................... ?!....L._.. .............
o
~
Sports e Science
Museum
Museum
o
Harvard
Aquarium
•
• Quincy
Market
ENTE ~TAINMENT
Magic Show
Children's
Museum
:.
o
®
Museum
~
Swan
Science
i Fenway
Newbury St,~ ®Trinity
Church
Children's e
Museum
Park
"Fenway
• Fine Arts
Park
Museum
..................................
. ........................ 7~ ..............................................
....
Trinity
Church
Zoo
o
Aquarium
Swar~Boats
@
g
•
Museum Fine Arts
Museum
O
Arboretum
HISTORCAL
Harvard
o
Sports
ZOO~
o
Salem
NATURE
Arboretum
Neewbury
St.
®
, HOPPING
Quincy Market
Figure 3a: MDS plot based on geographic similarity
Figure 3b: MDS plot based on content similarity
similar activity sites like the Aquarium and the Zoo appear
near each other, as do shopping areas Newbury Street and
Quincy Market. One can use the groupings of points in an
MDS plot to assign features to clusters of points, as we have
done in Figure 3b with the features, "historical," etc.
Likewise, many researchers who use MDS would attempt to
assign meaning to the axes of the plot, suggesting perhaps
that the X-axis runs from "playful to serious" and Y-axis
runs from "outdoors to indoors". Such labels or category
assignments must be done carefully and should be verified
with separate experiments, however, because they are often
biased by the experimenter's a priori knowledge of the data.
have known what types of similarity to separate if we knew
nothing about the data to begin with. To answer this question, we introduce the relatively new Trajectory Mapping
(TM) procedure [4, 13, 14].
The reader might wonder what would be the outcome if we
had asked subjects to give general similarity ratings without
specifying what type of similarity. Would the resulting
MDS plot have been an unfortunate mixture of the two plots
in Figure 3? This issue raises the question of how we could
For high-dimensional feature spaces, often TM can better
delineate the features of the data than MDS. Instead of giving similarity judgements as in MDS, the subject's task in
TM is to imagine a conceptual feature or property that links
a given pair of stimuli. The subject then extrapolates that
feature in both directions to pick two stimuli from the
remaining set that would be appropriate. The subject also
picks an interpolant, i.e. a stimulus that would fit well within the pair.
Returning to our black and white circle example, a TM set
of input data might appear as in Figure 2b. The members of
the original pair are in columns A and B; the extrapolants are
-t
-4
i;
e
4~
w
....
t)
,0
Figure 4: MDS and TM output for black and white circles.
Figure 5: Trajectory map for the tourist site data
!~ii!
jil i
415
i~ i~
C~"~
in the two "ex" columns, and the interpolant in the "int" column. As well as using a stimulus for each slot, the subject
may also enter an "X" or a "...". An "X" indicates that the
subject did not reel comfortable choosing a stimulus for that
spot, and the "..." indicates that the subject could imagine a
stimulus that would fit there, but that such a stimulus was
not present in the given data set to choose from.
From this set of quintuples, we can now extract a connected
graph of the stimuli in which the maximum number of quintuples fit. For example, in the top row of Figure 2b, we have
an ordered set of white circles ending in a small dot in one
column, and a set of black circles ending in the same small
dot in the other column. If these quintuples are equally considered as constraints on the graph, the ending trajectory
map will contain a path from the large white circle to the
large black circle, going through the small dot, as shown in
Figure 4.
By pruning the trajectory map to only its strongest links, one
can see rough feature clusters emerge (see the heavier links
APRIL
Mag c Show .8~;4;~ ~4
~;~,
.................
; 4:
t3-18,
~&
1996
Salem
lildren's
mourn
Science
,
(
M i i~..11 i m
i
Museum
"i
~iiiiii~
O0
Newbury St.
Arboretum
,:.
BOSTON DATA
Figure 5 shows a trajectory map for tourist site data gathered
from the authors. Note here that the positions of the nodes
are not important; the mental model lies in the connections
between the nodes, i.e. the topology of the graph. The
weights on the links are based not on similarity, but rather
on the robustness of that link across a gamut of parameters
within the TM algorithm. The weight on a link can thus be
thought of as the strength of the connection in the mental
model. (A TM algorithm is being fine-tuned for release by
Gilbert.)
@~
~,C?
-0!
Quincy Market
( )
~,
,
Figure 6: The mental models combined: a rough TM
path superimposed over the MDS content plot.
in Figure 5). In our example, the clusters are roughly similar to the estimated clusters in the MDS plot (Figure 2b). In
a trajectory map, however, the s~imuli are ordered within
each cluster, e.g. Arboretum, Swan Boats, Aquarium/Zoo.
Figure 6 shows the combined mental models, a rough TM
path drawn over the MDS content-based plot.
Because the tourist sites have several possible features that
could be used by subjects to report their TM extrapolations
and interpolations, one can discover the most salient features of the data set by running the algorithm across many
i~ ::~L¸
II
tmi
m
m
Fine Arts Museum Trinity
Church
F ¢nw,~tf Pw~
Fenway Park
Z~
O
Aquarium
Z
q
Figure 7a: Children's activities energized by the regional activation spreading network.
416
m
Figure 7b: Adults' activities energized by the regional
activation spreading network.
APRIL 1 3 - 1 8 ,
1
1996 ~ |
~
l
l
M
Figure 8: The network activates the path segments when the user examines the sites Sports Museum, Fenway Park and
Aquarium. The activation levels are mapped to image and typographic size, enabling the system to suggest to the user
an appropriate next step based on the MDS and TM results.
subjects. It is likely with this tourist site information, for
example, that some subjects would do the TM based on geographic distance, while others would do it based on the content of the sites. This feature difference would eventually
manifest itself from obviously disparate trajectory maps,
and lead to the phrasing of the MDS questions described
earlier: similarity in terms of distance and similarity in
terms of content. Examples of different trajectory maps for
the same domain can be seen in [12].
EXPERIMENTATION
The mental models from TM and MDS provide an excellent
basis for measuring the efficiency of an information structure that has been built from the models. One might arrange
the stimuli serially according to the TM paths or distribute
them according to the MDS plot. After collecting data from
a new pool of subjects for measures of readability, ease of
remembering, etc. (see [2, 3] for a typical array of memory
measures), the information structure can be systematically
varied by changing the parameters that produce the MDS
and TM models.
EXAMPLE INTERFACES
If one considers the set of tourist attractions as an information space to be explored, we can also use the mental models to give the user well-founded suggestions as to his or her
next step of exploration. As a demonstration of feasibility,
we have designed a visualization system that allows the
exploration of the Boston tourist attractions. Since we are
no longer restrained to the geography of Boston, as an actual tourist would be, we must now answer the question,
"How should we order the various sites?" To explore the
different possible answers, we have built three different
interfaces for the system, each of which defines an "attentional window" as the current region of interest in the information space. An activation spreading network based on the
mental models defines the size of the attentional window as
the user explores. Thus, if the aquarium node of the network
is currently activated, and the models suggest that the swan
boats node is closely related, then the activation will spread
to swan boats, leading the attentional window to include
swan boats as a potential next focus.
In the first two interfaces, the 15 sites are arranged twodimensionally according to the MDS plot in Figure 3b.
Whichever sites fall within the attentional window are rendered larger than the others. In one interface, the attentional window envelops MDS regions (Figure 7), and in the
other, the window spreads across TM paths (Figure 8).
Figure 7a shows the display of regional activation with
emphasis on children's activities. Figure 7b shows the activities in an adult context. In contrast, Figure 8 shows several successive frames of path activation: when the user
investigates the Sports Museum, the network suggests that
Fenway Park or the Children's Museum might be a good
next choice (those two sites are slightly enlarged in Figure
8a). As the user shifts attention to the next event, past activities fade away and activities further along the path become
more conspicuous.
The difference between these two styles of exploration can
be characterized by the width of the attentional window:
narrow in the case of paths, and wide in the case of regions.
By smoothly varying the width, one might change smoothly between modes of exploration.
The third interface attempts to combine the path and region
following ideas of the first two. This interface depicts the
sites as semitransparent multimedia cubes in a 3-D space
(Figure 9). The cubes are arranged along the TM paths, but
at each cube, other cubes within the same region can be seen
orbiting the current cube nearby. Thus, the user can "fly"
smoothly through the space along the TM routes, or branch
off to a similar site within the current region. Also, the floor
of each cube displays a geographic map with the site's location high-lighted, thus incorporating all of the features that
we have discussed thus far.
417
~"H ~
APRIL 13-18, 1996
Figure 9: An interface that combines path and region following; the sites are represented by cubes in a 3-D space. The cubes
lie along the TM paths, and the other sites from the MDS region encircle the cube within the attentional window.
The activation spreading network was designed by Lokuge
and Ishizaki [10], and the visualization system is implemented on a Silicon Graphics Onyx workstation. This network links nodes in a space (Figure 10) and uses the ordering in the space to control the activation level of a node [ 1,
12]. When attention "jumps" to a new node, the network
partially reduces the activation levels in the original set of
nodes, leaving a faint trace as a history of the attentional
sequence.
DISCUSSION
We have designed this visualization system with its various
interfaces both as an existence proof that such a system
could be devised and as an illustration of our proposed
methodology for structuring information. We mentioned
above the fact that a variety of mental maps exist for each
data set, both within and across individuals. A further development would be an intelligent activation spreading network, i.e. one which contained the full repertoire of different mental models for a given data set and chose the best
model based on the user's behavior. Such a repertoire could
be described as a "hyper-mental-map" and could likely
formed by gathering MDS and TM data on the mental mod-
Fenway Park
|
MagiclShow
Science Museum
/
Salem~
Figure 10: Schematic diagram of a regional activation
spreading network. Activation energy spreads in the direction of the arrows initiated by the Goal: Kids' Tour of
Boston.
418
We have proposed and implemented a method of organizing
information space around cognitive maps. We plan next to
explore the degree to which this method can be used with
more generalized domains of knowledge, but examples
from previous MDS and TM papers lends us hope that the
method could become generally applicable. By accommodating an individual's search context (e.g. through paths or
regions) and his or her particular model of a domain (e.g. a
particular MDS or TM model), one can offer both a more
personalized tour of the information space and a more easily absorbable mass of information.
ACKNOWLEDGMENTS
This work was in part sponsored by ARPA, JNIDS, NYNEX
and Alenia. However, the views and conclusions expressed
here are those of the authors and do not necessarily represent that of the sponsors.
REFERENCES
1. Anderson, J.A. Spreading activation theory of memory.
Journal of Learning and Verbal Behavior 22, 261-295,
1983.
. Atkinson, R.C. & Shiffrin, R.M. The control of shortterm memory. Scientific American, 225, 82-90, 1971.
Goal: Kids' Tour
of Boston
Children's Museum
els as stimuli themselves.
. Craik, EI.M. & Lockhart, R.S. Levels of processing: A
framework for memory research. Journal of Verbal
Learning and Verbal Behavior, 11,671-684, 1972.
. Gilbert, S. A. & Richards, W. Using trajectory mapping
to analyze musical intervals. Proceedings of the Sixteenth
Annual Conference of the Cognitive Science Society.
1994.
. Johnson-Laird, EN. Mental models. Cambridge, MA:
Harvard University Press, 1983.
. Kruskal, J.B.
kyst2a. AT&T Bell Laboratories,
Computing Science Research Center, Netlib repository of
software, http://plan9.att.com/netlib/mds/index.html.
i
APRIL 1 3 - 1 8 ,
1996 C l " l | 9 6
7. Kruskal, J.B. Nonmetric multidimensional scaling: A
numerical approach. Psychometrika, 29, 28-42, 1964.
8. Lokuge, I. Dynamic magical environments: Engaging
interaction based on the art of illusion. Masters Thesis,
MIT Media Lab, 1995. To receive a copy email
docsOmit.edu or call MITDocument Services at
6171253-5650.
12. Maes, P. Situated agents can have goals in Designing
Autonomous Agents: Theory and Practice from Biology
to Engineering and Back, ed. P. Maes, MIT
Press/Bradford Books, 1992.
13. Richards, W. & Koenderink, J.J. Trajectory mapping
("TM"): A new non-metric scaling technique.
Proceedings of European Conference on Visual
Perception, Edinburgh (Abstract), 1993. In press
Perception.
9. Lokuge, I., Gilbert, S.A. & Richards, W. Visualizing
information spaces: A tour through Boston. Presented at
European Conference on Visual Perception '95,
Ttibingen. Abstract in Perception, 24 Supplement, p. 76.
14. Saint-Arnaud, N. Class~cafion of Sound Textures.
Masters Thesis, MIT Media Lab, 1995. To receive a
copy email d o c s @ m i t . e d u or call MITDocument
Services at 617/253-5650.
10. Lokuge, I. & Ishizaki, S. GeoSpace: An interactive visualization system for exploring complex information
spaces. Proceedings of ACM SIGCHI'95, Denver,
Colorado, 1994.
15. Shepard, R.N. The analysis of proximities: Multidimensional scaling with an unknown distance function. I. & II. Psychometrika, 27, 125-140, 219-246,
1962.
11. Lynch, K. Image of the City. Cambridge, MA: MIT
Press, 1960.
16. Torgerson, W.S. Multidimensional scaling: I. Theory
and method. Psychometrika, 4, 401-419, 1952.
419