UNIVERSITY OF CALIFORNIA
IRVINE
Space for Interpretation:
Designing and Evaluating Spatially and Socially Embedded Tangible Interaction
THESIS
submitted in partial satisfaction of the requirements
for the degree of
Master of Science in Information and Computer Science,
with a concentration in
Interactive and Collaborative Technologies
by
Amanda Marisa Williams
Thesis Committee:
Professor Paul Dourish, Chair
Assistant Professor Beatriz da Costa
Assistant Professor Donald J. Patterson
2007
Chapter 6  2005 Springer-Verlag
Chapter 7  2007 Association for Computing Machinery
All other materials  2007 Amanda Marisa Williams
ii
The thesis of Amanda Marisa Williams is approved:
__________________________________________
__________________________________________
__________________________________________
Committee Chair
University of California, Irvine
2007
iii
Table of Contents
Table of Contents ............................................................................................................... iv
List of Figures ................................................................................................................... vii
Acknowledgements .......................................................................................................... viii
Abstract of the Thesis ......................................................................................................... x
Introduction ......................................................................................................................... 1
1.1 A Short Story about an Abacus ................................................................................. 1
1.2 Research Questions ................................................................................................... 3
1.3 Theoretical Framework and Methodology................................................................ 3
1.4 A Roadmap ............................................................................................................... 5
Related work in collaborative tangible interfaces ............................................................... 6
2.1 Atoms and Bits: Tangible Artifacts as Representations of Digital Data .................. 7
2.2 Tangible Interfaces for Physical Learning .............................................................. 10
2.2.1 Learning Programming and Abstract Concepts ............................................... 11
2.2.2 Supporting Bricolage ....................................................................................... 12
2.3 Tangible Systems in Support of Emotional Intimacy ............................................. 14
2.4 Representational and Interpretive ........................................................................... 17
2.4.1 Representational Interfaces .............................................................................. 17
2.4.2 Interpretive Interaction..................................................................................... 18
2.4.3 Summary .......................................................................................................... 20
Damage: An Interpretive Prototype .................................................................................. 23
3.1 Introduction ............................................................................................................. 23
3.2 Mobile Group Communication ............................................................................... 24
3.2.1 Prior work ........................................................................................................ 24
3.2.2 Slam ................................................................................................................. 25
3.3 Damage ................................................................................................................... 26
3.3.1 Design .............................................................................................................. 26
3.3.2 A Usage Scenario ............................................................................................. 28
3.3.3 Implementation ................................................................................................ 29
3.3.4 Early Feedback................................................................................................. 30
3.4 Design Considerations for Wearable Ambient Communication ............................ 31
3.4.1 Quiet Technology............................................................................................. 32
3.4.2 Negotiated Interpretations ................................................................................ 32
3.4.3 Semi-public display ......................................................................................... 33
3.4.4 Tangibility and Intimacy .................................................................................. 33
3.5 Discussion ............................................................................................................... 34
Meaning in Tangible Interaction ....................................................................................... 35
4.1 Intentionality and Indexicality ................................................................................ 36
4.1.1 Tokens, Icons and Indices ................................................................................ 37
4.1.2 Indices in Natural Language ............................................................................ 38
4.2 Ethnomethodological Approaches to Meaning-Making ......................................... 40
4.2.1 Common Ground ............................................................................................. 43
4.3 Intersubjectivity and Phenomenology..................................................................... 46
4.3.1 Representational and Interpretive Revisited .................................................... 47
4.3.2 The Role of the Physical .................................................................................. 47
iv
Space and the Body in Tangible Interaction ..................................................................... 49
5.1 Collaborative Tangible Interfaces… In Space ........................................................ 50
5.1.1 A Foray Into Theory ........................................................................................ 51
5.2 Space as a Tool for Understanding ......................................................................... 54
5.3 Phenomenology of Space, Body, and Action ......................................................... 56
5.3.1 Action, Being and Heidegger ........................................................................... 57
5.3.2 Action, Space, and Heidegger .......................................................................... 59
5.3.3 The Being-Action-Space Story So Far ............................................................ 62
5.3.4 Body, Space and Merleau-Ponty...................................................................... 62
5.4 Social and Spatial .................................................................................................... 64
5.4.1 What Spatially Oriented Collaboration Studies Inherit From Phenomenology64
5.4.2 Sociality is Spatial and Embodied ................................................................... 68
5.4.3 Space is Social ................................................................................................. 73
5.5 Summary ................................................................................................................. 76
SignalPlay: Configuring Space through Embodied Interaction ........................................ 78
6.1 Motivation ............................................................................................................... 79
6.2 Related Work .......................................................................................................... 81
6.2.1 Tangible Interaction with Sound...................................................................... 81
6.2.2 Gallery Studies ................................................................................................. 83
6.2.3 Understanding Space ....................................................................................... 84
6.3 System Design and Implementation ....................................................................... 85
6.3.1 Infrastructure .................................................................................................... 86
6.3.2 Interface Objects .............................................................................................. 87
6.3.3 Sound Controls................................................................................................. 90
6.4 Exploring and Interpreting Space ........................................................................... 92
6.4.1 Modes of Object Interaction ............................................................................ 94
6.4.2 Collective Encounters and Interpretation....................................................... 101
6.4.3 Reading the Space .......................................................................................... 104
6.5 Conclusions and Implications ............................................................................... 107
Nimio: Combining Tangible Interfaces and Ambient Displays for Collaborative Groups
......................................................................................................................................... 111
7.1 Introduction ........................................................................................................... 111
7.2 Background ........................................................................................................... 112
7.2.1 Passive Awareness ......................................................................................... 113
7.2.2 Ambient Displays........................................................................................... 114
7.2.3 Tangible Interfaces......................................................................................... 115
7.3 Context-Specific Ambient Displays ..................................................................... 115
7.4 Site Study .............................................................................................................. 117
7.5 A Context-Specific Design ................................................................................... 120
7.5.1 How Nimio Works ......................................................................................... 121
7.5.2 Example Interaction ....................................................................................... 123
7.6 Implementation ..................................................................................................... 124
7.6.1 Design Rationale ............................................................................................ 126
7.7 What Did People Make of It? ............................................................................... 129
7.8 Discussion & Conclusions .................................................................................... 134
7.8.1 Context for Interpretability ............................................................................ 135
v
7.8.2 Accumulation of Meaning Over Space and Time.......................................... 136
7.8.3 Legibility and Ambiguity ............................................................................... 137
Discussion ....................................................................................................................... 140
8.1 Design Considerations .......................................................................................... 140
8.1.1 Objects That Invite Interaction ...................................................................... 140
8.1.2 Publicly Available Feedback ......................................................................... 141
8.1.3 Building For Accountability .......................................................................... 142
8.2 Tangible Systems as Social Probes ....................................................................... 144
8.3 Technologically Mediated Space .......................................................................... 146
Future Work .................................................................................................................... 151
9.1. Distributed Displays and Wireless Hot Spots ...................................................... 151
9.1.1 Distributed Displays In The Wild .................................................................. 152
9.1.2 Spot Clocks .................................................................................................... 154
9.2 Technology and Urban Space ............................................................................... 156
Bibliography ................................................................................................................... 159
vi
List of Figures
Figure 2.1: Ullmer and Ishii's MCRpd interaction model
Figure 2.2: A mapping application on the metaDESK
Figure 2.3: curlybot
Figure 2.4: Montessori-inspired systemblocks
Figure 2.5: Topobo
Figure 2.6: InTouch
Figure 2.7: Buddy Bead charm bracelets
Figure 2.8: Iconic and Interpretive Tangible Systems
Figure 3.1: Slam main screen
Figure 3.2: Damage design sketch
Figure 3.3: An early prototype of Damage.
Figure 6.1: System Diagram of SignalPlay
Figure 6.2: SignalPlay Interactive Objects
Figure 6.3: Stackable Chess Pieces
Figure 6.4: “Playing the compass” by tilting, closing and opening.
Figure 7.1: Ambient Display Design Considerations
Figure 7.2: Jasmine Blossom Distributed Display
Figure 7.3: Nimio!
Figure 7.4: One possible interaction with Nimio
Figure 9.1: Urban Distributed Displays
Figure 9.2: Infrastructural Access Points
Figure 9.3: SpotClocks
vii
Acknowledgements
This work was supported in part by the National Science Foundation under awards
0133749, 0205724 and 0326105, by a grant from Intel Corporation, and by a National
Science Foundation Graduate Research Fellowship Award.
The problem with writing acknowledgements is that it forces me to translate the complex
topology of my wonderful support network into a simple linear order, and to say in so
many words what my family, friends, and associates have done for me. I can’t help but
feel that such an ordering and articulation does violence to the reality of how these
relationships have helped sustain me. But sometimes you have to commit to representing
these things.
Thanks first and foremost to my advisor, Paul Dourish, for his support and insight. His
good opinion gives me the confidence to take risks. I would also like to extend my
gratitude to the other members of my thesis committee, Beatriz DaCosta and Don
Patterson.
Despite my name being the only one on the title page, this thesis had been a collaborative
project. Co-authors on previous publications have been personally supportive and
contributed significantly to the realization of this work. Thank you, Johanna Brewer, Eric
Kabisch, Shelly Farnham, and Scott Counts. Thanks also to Erica Robles, Joseph
Kugelmass, Jennifer Rode and Paul DiGioia for reading my ugly rough drafts and helping
viii
me articulate myself. Simon Penny, Tom Jennings, Ted Ediss, Mike Sinclair, Li Zhong,
Danyel Fisher, AJ Brush, Marc Smith and the Community Technologies group at
Microsoft Research, Jason Hill, Joe Polastre, and Gregory Gallardo have all helped me
with various aspects of the individual projects presented here. I also owe a big thank you
to focus group members, SignalPlay’s participants, Shelly, Albert and everyone in the
Calit2 group, who gave me something to write about. I’m especially grateful to JeanBaptiste of the lent soldering iron, who saved our demo. Ken Anderson, Dawn Nafus,
Silvia Lindtner, Charlotte Lee, David Nguyen, Marisa Cohn, Sharon Ding, Wendy Ju, Jan
Chong, and Karen Martin have all stimulated my brain and done work that inspires.
Hiroshi Ishii and the good people at PARC (Ruth Rosenholtz and Jeff Breidenbach in
particular) provided impetus. Scott Klemmer and Jeff Heer made it look easy. Eric
Baumer, Arianna Bassoli, Judy Chen, danah boyd, Irina Shklovski, Ken Conley, Sarah
Wu, Christine Rhee, Alex Shearer, Chris Beckmann, Bryan Pendleton, Blair Thornton,
and Stephen Allison have provided sanity, or insanity, as needed.
Pai and Steve Williams are constantly supportive and have inspired my work in ways that
they may not realize. Scott Lederer keeps me grounded, and for that he has my love and
gratitude.
My thesis also owes its existence to several non-person entities: my powerbook, Roxy’s
soldering iron, Soul Coughing (they reassure me that I am a “supra genius”), Franz
Ferdinand (the band not the Habsburg), and coffee. A lot of coffee.
ix
Abstract of the Thesis
Space for Interpretation:
Designing and Evaluating Spatially and Socially Embedded Tangible Interaction
by
Amanda Marisa Williams
Master of Science in Information and Computer Science
with a concentration in Interactive and Collaborative Technologies
University of California, Irvine
Professor Paul Dourish, Chair
As computation increasingly becomes a part of everyday life, ubiquitous computing
researchers are becoming increasingly aware of the importance of interactions that are
both physically and socially embodied. This thesis presents the design, implementation
and evaluation of three collaborative tangible interactive systems. These designs and
studies explore collective appropriation of meaningful technological systems and the
important role of action, bodies, and space in people’s negotiations of contingent
intersubjective understandings.
Damage is a wearable tangible-ambient messaging device, providing a minimalist display
meant to allow close friends to negotiate the meanings of messages around the device.
SignalPlay is an interactive gallery installation, allowing participants to collectively
control a soundscape by manipulating tangible artifacts. Nimio is a set of networked
x
desktop toys that convey information about sound and movement to members of a
workgroup, allowing for peripheral and interpretable awareness of others’ presence.
These deployments serve as technological probes as well as systems in themselves,
providing opportunities for social analysis. I hope that these designs can foreground
important issues to consider more generally in the deployment of ubiquitous computing
systems into preexisting spatial and social milieus.
xi
Chapter 1
Introduction
As computation increasingly becomes a part of everyday life, ubiquitous computing
researchers are becoming increasingly aware of the importance of interactions that are
both physically and socially embodied (Dourish 2001). Research in tangible interfaces
and ambient displays has a great deal of potential to put embodied interaction principles
into practice, but while such research addresses physical embodiment quite well, social
embodiment has remained largely neglected.
1.1 A Short Story about an Abacus
This excerpt from Hiroshi Ishii’s seminal Tangible Bits paper hints at the social, as well
as physical, richness of tangible interaction:
Ishii met a highly successful PDA (Personal Digital Assistant) called the
“abacus” when he was 2 years old. This simple abacus-PDA was not merely a
computational device, but also a musical instrument, imaginary toy train, and a
back scratcher. He was captivated by the sound and tactile interaction with this
simple artifact. When his mother kept household accounts, he was aware of her
activities by the sound of her abacus, knowing he could not ask for her to play
with him while her abacus made its music. We strongly believe this abacus is
suggesting to us a direction for the next generation of HCI.
(Ishii and Ullmer 1997)
Many different things propelled me toward graduate school, and many kind people have
supported me, so I can’t give Ishii all the credit. But it was this paragraph that provided
the initial impetus. It is so full of subtle suggestion that it bears unpacking.
1
Besides its instrumental use as a mathematical assistant, certain intrinsic physical
characteristics of the abacus inspired and allowed the young Hiroshi to appropriate it for
a variety of purposes, as a “musical instrument, imaginary toy train, and a back
scratcher”. Sounds and tactile pleasure are not necessary properties of mathematical
assistants, but these seemingly extraneous characteristics are what make the object so
rich. It is captivating in ways that a dialogue box could never be.
These intrinsic characteristics also allow the abacus to serve as an ambient display of
sorts, indicating when his mother was too busy to play. In this secondary use (awareness
of activity), it is apparently just as pleasant as it is in its primary use (as a calculator). I
imagine how his childish understanding of the sound of the abacus would have
accumulated over time, perhaps with the help a few scoldings, as his mother repetitively
paid the monthly bills. I imagine this interaction as particular to that household and yet
recurring in infinite variety in many households. I speculate that to this day, the sound of
the abacus still reminds him of his childhood. Ishii’s account recalls the small intimacies
of our everyday lives with our loved ones, intimacies that form around simple, yet
somehow intricate, everyday objects.
This paragraph hints at three aspects of the abacus (and by extension, tangible interfaces
in general) that contribute to the richness of interaction with it. The first, and the primary
focus of the tangible bits research agenda, is the abacus as a way to represent and
manipulate the data, using beads to signify numbers. But the intrinsic physicality of the
abacus also plays an important role in Ishii’s relationship with the object, and it is
2
engaging as a thing that rolls, makes sounds, and has texture, not only as a data
manipulation device. And lastly, as an everyday object in a social setting, the abacus
accumulates meanings particular to its situation, constructed by people’s actions and
interactions over time.
1.2 Research Questions
My research explores collaborative tangible interfaces as spatially and socially situated.
While spatial and social situatedness are analytically distinct, I will argue that in practice
they are tightly intertwined. The work and theoretical analysis presented in this thesis
explore the question of how considerations of spatial sociality can help us think of
collaborative tangible interfaces in new and interesting ways, as systems that are made
meaningful by people’s collective actions. I would suggest that in addition, this leads us
to a broader consideration. If space is experienced as an arena for action with and around
objects, then how can ubiquitous computing provide new opportunities for action in the
spaces that we inhabit?
1.3 Theoretical Framework and Methodology
My system design work is conducted within a theoretical framework of embodied
interaction, which implies a focus on “the creation, manipulation, and sharing of meaning
through engaged interaction with artifacts” (Dourish, 2001). This interaction with
artifacts may be physical, but more relevantly to this project the creation and sharing of
meaning is a social process. The fact that technological systems are deployed and
appropriated into pre-existing, rich social milieus and communities of practice is a
3
primary focus of my design work. The theoretical framework for my evaluation work, in
keeping with the focus on embodied interaction, is influenced by Ethnomethodology,
with a focus on the ways in which people produce orderliness and meaning through
situated interaction.
A theoretical focus on physically and socially embodied interaction has methodological
implications. A close linkage of ethnography, design and evaluation is called for. All
three appear in this thesis, and the evaluation of one system may inform the design of the
next in an iterative process. Additionally, the designs presented here are site-specific in
order to best understand the situations into which they are deployed. Certain assumptions
about tangible systems inform my research:

People appropriate technological systems into complex social practices and
situations. They make those systems meaningful through their practices, over
time.

Tangible systems have intrinsic physical features in addition to the strictly
functional. They are interpretable, and thus foster rich, multi-layered, multimeaningful interactions.

The body and space are often a crucial part of the building of shared experience
and shared understanding. Tangible systems involve bodies and physical space in
ways that graphical systems do not.
4
My interest in tangible systems, then, is twofold. First, they provide rich interaction,
making them an excellent place to consider the interactional potential of ubiquitous
computing. Second, tangible systems make good technological probes. Much of the work
presented there involves introducing a novel tangible system and then watching the
process of appropriation and meaning-making. Tangible systems can be both prototype
and probe.
1.4 A Roadmap
In this thesis I will first discuss trends in prior tangible interaction work, discussing those
systems in terms of their approaches to meaning and interpretability (Chapter 2). In
Chapter 3 I present Damage, a highly interpretable wearable prototype incorporating
tangible interaction and ambient display in order to allow for social connectivity. As an
early prototype it raises some useful questions about designs of tangible systems and how
they come to be meaningful. In Chapters 4 and 5 I explore some theoretical background
for considering meaning in computational and tangible systems, as well as the importance
of bodies and space in coordination, collaboration, and shared understanding. In Chapter
6 I present SignalPlay, a collaborative, tangibly controlled soundscape, and discuss how
it was used by participants to construct the space that they inhabited. In Chapter 7 I
present Nimio, a tangible interface and ambient display meant to facilitate shared
awareness amongst a group of office workers. Chapter 8 is a discussion of some of the
implications of this body of work, and in Chapter 9 I present some future directions.
5
Chapter 2
Related work in collaborative tangible interfaces
Research in tangible interfaces has, unsurprisingly, exploded since Ishii wrote the
Tangible Bits paper. I will focus on three areas of research within tangible interaction.
The first wave of tangible systems, as discussed in Ishii and Ullmer (1997), used physical
objects to represent and manipulate digital data; physical portions of the system serve as
icons. A second body of research produces tangible systems for learning, based on the
theory that their intrinsic physical features encourage exploration. Third, tangible
interfaces have been designed primarily for emotional and intimate connection, especially
when physical presence is not possible; the evolution of shared understanding is crucial
to the use of these systems.
This review of prior work in tangible interaction is not exhaustive; rather I attempt to
trace outlines of a few research agendas that are directly relevant to this discussion. Each
of these areas can be characterized by the activities the tangible system is meant to
support; the tangible interactions in each of these research areas are designed for distinct
and different purposes. Another possible reading of these areas, however, focuses on their
tendency to highlight one of the three aspects of the abacus discussed in the previous
chapter: the object as data representation, the object in itself, and the object as a site for
constructing meaning through social interaction.
6
In the final section, I attempt a cross-cutting characterization of these tangible systems in
terms of their modes of meaningful interaction. I am interested here in how these three
aspects interrelate, in the tension between stably representational objects and objects that
invite negotiation of meaning, and how the object’s intrinsic physicality plays into its
meaningfulness.
2.1 Atoms and Bits: Tangible Artifacts as Representations of Digital Data
Following the suggestion that the “coupling of bits and atoms” (ibid) is one of the key
concepts in tangible interfaces, Brig Ullmer and Hiroshi Ishii assert that the value of
tangible interfaces lies in the “seamless integration of representation and control”
(Ullmer and Ishii 2000), as opposed to distinguishing sharply between input devices and
the data being manipulated, as graphical user interfaces do. To return to the example of
the abacus, they claim that its beads serve as representations of abstract numerical values,
and that to move the beads is to manipulate their underlying associations. Drawing from
the Model-View-Controller architectural pattern in graphical user interfaces, they propose
the MCRpd interaction model. In this interaction model, the “view” component is
replaced with physical and digital representations (Rpd). The physical representation is
tightly coupled with a tangible control component (C), while the digital representation
may be a projected image or sound that represents the state of digital data along with the
tangible component of the interface. While the model may be digital, control and
representation are both manifest in the physical world.
7
Figure 2.4: Ullmer and Ishii's MCRpd interaction model
The MetaDESK platform (Ishii and Ullmer 1997; Ullmer and Ishii 1997) puts the
MCRpd theoretical framework into practice, going so far as to actually create physical
versions of pre-existing GUI metaphors. Icons became phicons, menus became trays, and
handles became “phandles” (a seemingly redundant name meaning “physical handles”).
The metaDESK consists of a desk, a back-projected graphical display and horizontal
surface on which the phicons sit, an active lens, which is a movable arm-mounted flatpanel display showing a digitally augmented view of items on the desk, and a passive
lens, which can trigger the desk to change the display in its vicinity. Components are
designed to seem similar to other tools that might be familiar to users; the active lens, for
example, looks and acts like a jeweler’s magnifying glass. By eliminating the GUI’s
locus of control (at the cursor), the metaDESK allows users to make fuller use of their
bodies and the desk space. However, many of the GUI’s desktop metaphors persist,
ironically translated from desk to screen to augmented desk. The phicons and displays
8
serve primarily as representations of digital data, while the lenses are means to access
different views of that data.
One of several applications built atop the metaDESK platform, the Urban Planning
Workbench (Ben-Joseph, Ishii et al. 2001; Ishii, Underkoffler et al. 2002) was meant to
support collaborative exploration of urban planning problems. Phicons represented
buildings, and environmental characteristics like shadow and wind could be projected on
the table.
Another system conforming to the MCRpd model, MediaBlocks (Ullmer, Ishii et al.
1998; Ullmer and Ishii 2000) were small physical blocks that were dynamically coupled
with digital media. Media could be played by inserting blocks into a player, or sequenced
by arranging blocks in order on a “sequence rack”.
Figure 5.2: A mapping application on the metaDESK. Two phicons represent buildings on the MIT
campus. The passive lens shows a satellite view of the mapped region, while the active lens displays a
stripped-down 3D view.
(Source: http://www.usenix.org/publications/library/proceedings/tcl97/
full_papers/poster_ullmer/ullmer_html/desk.jpg)
9
Using a whiteboard-based system similar in nature to the metaDESK, the Designer’s
Outpost (Klemmer, Newman et al. 2001; Klemmer, Thomsen et al. 2002) was meant to
support collaborative web design using a combination of manipulable tangible objects
and projected digital imagery. The design teams involved in the study used post-its
arranged on whiteboards during their design process, which the Designer’s Outpost
digitally augmented. Web designers could edit the digital model of a website’s
architecture by moving post-its, or placing event-triggering icons in the workspace within
view of the system’s camera.
2.2 Tangible Interfaces for Physical Learning
Reflecting a long history of research around cognitive development and constructionist
learning, inspired by the work of Piaget (Inhelder and Piaget 1958), tangible interfaces
have been developed, largely at the MIT Media Lab, to support learning. Piaget
distinguishes between a concrete operational stage, in which children are capable of
logical thought but operate primarily in concrete rather than abstract situations, and a
formal operational stage, in which one can think abstractly, plan, and apply procedures to
solve unfamiliar problems. One of the strengths of tangibles appears to be support of
concrete operational learning by allowing physical manipulation of real materials.
Tangible learning systems have found their way into commercial products in the form of
the Logo programmable turtle and more recently Lego® Mindstorms (Papert 1980).
10
2.2.1 Learning Programming and Abstract Concepts
The distinction between concrete and formal is again evident in the design of Froebelinspired manipulatives for modeling of real-world phenomena, and Montessori-inspired
manipulatives for modeling of abstract structures (Zuckerman, Arida et al. 2005). It is
argued that tangibles are powerful learning tools because they provide multi-sensory
engagement, are more accessible, and facilitate group learning, however they do not go in
depth into how, exactly, these characteristics come about. That is, it is stated that tangible
learning interfaces are more accessible to novices, and people with learning disabilities,
but no explanation is forthcoming of what makes them accessible. Similarly, though it is
noted that they provide a multi-hand interface, no other reasons are given as to just how
group learning is facilitated, and most of their evaluation is done pertaining to
individuals. Their tangible interfaces to model complex concepts such as flow,
accumulation, and feedback loops did, however, facilitate learning of those concepts both
among 10-year-olds and some 4-year-olds (ibid). One pitfall in the functioning of their
system lay in the tension between representative and intrinsic physical characteristics of
the Montessori-inspired manipulatives; rather than use the tangible interfaces to create
analogies to abstract processes, some children focused on the physical form of the
components, using them to build letters or shapes. The paper recommends designing
tangible learning systems to encourage the building of analogies rather than concrete
structures.
Tangible Computation Bricks (McNerney 1999) embed both sensors and programmable
microprocessors inside of stackable Lego® bricks, with the goal of reducing the
11
conceptual difference between sensing data and programmatically manipulating it. Each
brick may perform a certain operation, such as counting, multiplying, or triggering. Order
of operations is determined by the order in which bricks are stacked. Because the bricks
can only stack one on top of another (i.e. forking or building two-dimensional structures
is not possible), physical limitations introduce obvious constraints on what can be
represented, which serves to simplify the problem for novices. While intrinsic physical
characteristics
in
Montessori-inspired
manipulatives
were
thought
to
hinder
representation, and therefore learning, the Tangible Computation Bricks leverage their
intrinsic characteristics to structure the interaction and ease comprehension.
2.2.2 Supporting Bricolage
Curlybot (Frei, Su et al. 2000), a simple, cute, round children’s toy, makes algorithms
spatial and tangible. A user presses a button telling curlybot to record a motion, then
when she pushes the button again it repeats the initial motion ad infinitum. A fairly openended toy, curlybot could be used to illustrate narratives, convey emotions, or play with
geometry. Evaluation consisted of observing the play activities of children using curlybot
in a loosely social setting, but little was said about collaboration among the children.
Zuckerman et al (2005) characterize Topobo (Raffle, Parkes et al. 2004) as a Froebelinspired manipulative for modeling concrete phenomena. Topobo is a 3D modeling
system, programmable by direct manipulation. Children can connect physical
components to build a structure or creature, then record motions, which the system then
plays back. It is thus capable of teaching children about balance, motion, and center of
12
gravity, as well as construction. Where many tangible systems have emphasized
representation, either of digital data or abstract concepts, Topobo’s design guidelines
focus on action, manipulation, and interpretation, with items such as “be meaningful even
if the power is turned off”, “be expressive”, and “engage multiple senses”. Using the
system “correctly”, then, is not an all-or-nothing affair; it is meaningful at several levels.
In their evaluation, both 2nd graders and 8th graders believed it to be designed for their
own age group.
Figures 2.6, 2.4, 2.5: Curlybot, Montessori-inspired Systemblocks, and Topobo.
(Sources: http://tangible.media.mit.edu/projects/curlybot/,
http://www.iseesystems.com/community/connector/Zine/may-june_2003/systemblocks.html,
http://www.rafelandia.com/topobo/topobo-photos/pages/2_griffinWalking.html)
These systems have been built for use by children, and particularly for elementaryschool-aged children in what Piaget would characterize as the concrete operational stage,
which dominates from age 7 to 11. Turkle and Papert, in contrast, claim that concrete
operation is not a stage but a style, and one that works (Turkle and Papert 1992). They
refer to work done in such a style as “bricolage”. By their definition, “bricoleurs
construct theories by arranging and rearranging, by negotiating and renegotiating with a
set of well-known materials”. They describe bricoleurs’ experience even of seemingly
insubstantial computational objects as “tactile and concrete,” and their relationship to
13
their materials as close, even intimate. In other words, material matters; it is manipulated
and negotiated by adults as well as children.
2.3 Tangible Systems in Support of Emotional Intimacy
A number of tangible interfaces have also been designed purely for emotional intimate
communication. Lumitouch (Chang, Resner et al. 2001) involved LEDs situated on a
picture frame that could be activated by partners across distance to indicate presence and
emotional connectivity. InTouch (Brave and Dahley 1997) provides a tangible channel
for long-distance communication using devices with wooden rollers, sensors and
actuators, connected via Ethernet. Playing with one device causes the other to move.
Interactions were interpretable by the users, who might engage in games of resistance,
roll the rollers smoothly, or in other ways flexibly interact with the system.
Figure 2.6: InTouch. (Source: http://tangible.media.mit.edu/projects/intouch/intouch.gif)
Expanding on the idea of virtual intimate objects (Kaye, Levitt et al. 2005), physical
intimate objects (Kaye 2005) present a minimal interface for communicating with another
person: a button and two LEDs attached to an Altoids tin. When one user pushes the
14
button on their object, a red LED lights up on their partner’s object, and gradually fades
over time. A green LED on one’s own object indicates the state of the partner’s LED.
Despite the extreme simplicity of the interaction, users attributed their own meanings to
it, and those meanings would change depending on the participants’ situation. Clicks
might take the form of gifts, contests, or simple good mornings. Kaye’s minimal intimate
objects pose the question: “what is the minimum amount of digital data needed for
computer-mediated communication to be interpretable?” The answer: very little if both
participants share common understandings. These highly interpretable objects were
intended for communication between intimates and were evaluated with couples in longdistance relationships.
Sensing Beds (Goodman and Misilim 2003) also explore the possibility of intimate
tangible communication between distant partners. Each of the paired beds senses body
position and triggers a grid of heating elements to warm the same points on it counterpart.
Each person, then, may feel the ghost of their partner’s body heat, even if they are
sleeping apart. The interaction here is highly context-dependent:
The Sensing Beds deliberately limit the data they sample. They do not recognize
who is in the bed, or whether the bed's owner is in the room. Their heat may be a
comforting reminder of a lover's presence — or perhaps create insecurity. …
They can only be comforting when they are supported through emotional trust
built with other, more active, communications methods: the phone, the email, the
Instant Messenger (IM).
The Sensing Beds derive their meaning from people, not the other way around.
They echo and amplify a relationship's dynamic.
Buddy Beads expands intimate interpretable communication beyond couples to tight-knit
groups of friends, specifically teenage girls (Kikin-Gil 2005). Mobile phones act as a
15
transparent means of communication between Buddy Beads, a wearable tangible device
much like a charm bracelet. By pressing a “message bead” a user sends a combination of
lights and vibrations to one of her friends, specified by pressing one of the “friend
beads”. Each bead is designed to be distinguishable by touch as well as by sight, in case
messages need to be sent surreptitiously. Buddy Bead bracelets are meaningful not only
because of the messages communicated through them, but also by the social interactions
around them; as a piece of jewelry worn on the body it is semi-public and visibly
signifies group membership. Like the minimal intimate objects, Buddy Beads depend on
interpretability due to shared context for their utility.
Figure 2.7: Buddy Bead charm bracelets. (Source: http://www.ruthkikin.com/Images/bracelets.jpg)
These projects feature touch, one of the more intimate human sensations, as a way of
conveying action in absence. The common theme here is emotion and intimacy, and their
support for the development of private meanings and messages that are not meant to be
widely understandable.
16
2.4 Representational and Interpretive
The tangible interfaces discussed in this chapter can be broadly characterized by two
ways of being meaningful in interaction (Figure 2.8). In the first, which I refer to as
representational, tangible components are meant to be clearly representative of some
stable, well-defined idea or piece of data. The second group I refer to as interpretive.
Interpretive interfaces may convey gestures or indicate that the user is doing something.
They are conduits and catalysts for visible action, but the meaning of those actions are
heavily dependent on context. I do not wish to imply that these two modes of interactivity
are mutually exclusive. Any experienced icon designer knows all too well that users will
interpret their symbols in unpredictable ways; and meaningful actions need not be free of
representation. However, among the aforementioned tangible systems one mode or the
other tends to dominate interaction with the system.
2.4.1 Representational Interfaces
Representational interfaces are explicitly concerned with the coupling of physical objects
and digital information in a representational relationship. In the case of the Urban
Planning workbench (Ben-Joseph, Ishii et al. 2001) this coupling is static and defined by
the designer, each physical building tied to its digital model. With MediaBlocks (Ullmer,
Ishii et al. 1998), it is dynamic and user-defined; the user decides which block represents
which media clip. In both cases, though, each physical object represents a well-defined
digital object or function; their spatial relationships or relational ordering both determine
and reflect the digital state of the system. Tangible Computation Bricks (McNerney
17
1999) are a particularly clear example of this principle: each brick represents an operation
in a program, and stacking the bricks represents the order of execution.
The representative nature of these interfaces is not different in principle from the icons
present in any graphical user interface – indeed they invite the comparison by using
“phicons”. The benefit of tangible over graphical interaction in these systems is that
control and representation are more tightly bound, making the task at hand more
immediate and less mediated. The digital state of the system is perpetually visible and
immediately understandable by the physical state of the system.
The System Blocks (Zuckerman, Arida et al. 2005) are a slight departure from the theme.
Their primary purpose, as an educational toy, is to accurately represent an abstract
concept, such as feedback loops or accumulation, rather than a particular piece of system
data. Nonetheless, all of these systems share a central concern, the “design and selection
of appropriate physical representations” of something more ethereal (Ullmer and Ishii
2000).
2.4.2 Interpretive Interaction
Interpretive tangible systems are characterized by their potential multiplicity of
meanings, their context dependence, and their emphasis on action over representation.
Brave and Dahley (1997) describe a crucial aspect of these interfaces in their discussion
of inTouch:
18
With inTouch, the idea is not to create a device to represent the physical form of
the user at the other end, but rather to create a physical link for expressing the
movements or gestures of that person.
This expressive mode of interaction allowed for quite a bit of ambiguity and a broad
variety of engagement with the system. curlybot (Frei, Su et al. 2000), a similarly openended system, was used by children to create narrative motions, expressive gestures, and
spatially constrained geometrical shapes. In contrast to interfaces that strive for
“appropriate” representation (implying that there are inappropriate representations that
might lead to misuse of the system) there appears to be no “right” or “wrong” way to use
these systems. Their intrinsic physical forms may afford certain actions, but they are not
stand-ins for well-defined bits of data.
It is user action that renders these systems meaningful, an aspect that is highlighted
particularly by Kaye’s Minimal Intimate Objects (Kaye 2005). In context, and with
shared interpersonal understandings, a simple light on an altoids box or dot on the
computer screen were imbued with highly personal meanings. Buddy Beads (Kikin-Gil
2005) and Sensing Beds (Goodman and Misilim 2003) worked on a similar principle; all
projects emphasized the legibility of the system as part of a larger ecology of
communication between intimates that provided the shared context necessary for users to
agree upon and attribute meaning to the system.
The issue here is not just the interpretability of the object, but the interpretability of
people’s action through and around the object, and how they might come to agree upon
the meanings that they attribute to the interaction.
19
Atoms and Bits
Educational
Representational
 metaDESK
 Urban Planning
Workbench
 Digital Outpost
 Montessori-Inspired
Manipulatives
 Tangible Computation
Bricks
Intimate
Interpretive


Topobo
Curlybot



Lumitouch
InTouch
Minimal Intimate
Objects
Sensing Beds
Buddy Beads


Figure 2.8: Iconic and Interpretive Tangible Systems
2.4.3 Summary
It is noticeable from Figure 2.8 that my crosscutting categorizations of tangible systems
as atoms-and-bits/educational/intimate and representational/interpretive are not exactly
orthogonal. Intimate and emotional interactive systems were all interpretive, while
systems based on the atoms-and-bits model were all representational. Educational
systems were split down the middle.
The first set of categories can be seen as characterizing the intent of the system, the
activities that it is meant to support. The atoms-and-bits systems, in part, attempted to
ease the performance of tasks normally done in a graphical user interface (e.g. zoomable
maps, organizing media, running simulations) by creating a tighter coupling between
controller and data. In so doing, the designers retained many of the metaphors of the
graphical user interface, using icons, containers, and controllers. They utilized stable,
20
widely known representations in order to make a system that was easy to walk up and
use.
Systems in support of intimacy, on the other hand, are not meant to be universally
accessible; that would defeat the purpose, users of those systems do not wish to feel
emotionally connected to the whole world. Each intimate emotional relationship is
particular, constituted of experienced shared within that relationship and snippets of
private language. These shared experiences provide a background for co-creating
particular relationship-appropriate meanings from interactions that are fairly blank-slate,
rendering them interpretable only to their intended audience.
The interpretive mode of tangible interaction, however, is not always directed at creating
meanings that are opaque to all but a few people. Among educational tangible interfaces
interpretable systems like curlybot or Topobo could encourage actions that were visible
to others and foster collaborative, open-ended exploration. Representative systems for
education, though also designed for collaboration and exploration, seemed geared
towards teaching specific concepts.
So one observation here, which may sound terribly obvious, is that these different modes
of interaction will support different activities. Yet another way of characterizing these
three types of system is by which aspect of tangible interaction they highlight, and how
these aspects relate to meaningfulness in interaction. The atoms-and-bits systems provide
a physically embodied experience of data manipulation, allowing users to understand and
21
act on data through the object. Tangible artifacts in these systems provide a pathway to a
meaning. In this way, the beads on an abacus can represent numbers. The Piaget-inspired
educational systems, in comparison, begin to foreground intrinsic physical features,
which can either encourage exploration or provide misleading cues. Here the object is
interactionally interesting in and of itself. It is the physicality of the abacus that invites it
to be used as a back scratcher or toy train. Tangible systems in support of intimacy focus
on the object as a site to create and negotiate meaning in collaboration with other people.
It is just such socially negotiated meaningfulness that allowed the abacus to alert
members of Ishii’s household to his mother’s bill-paying activities. I am interested in
drawing attention to the ways in which these aspects interact: the tension between
representation and interpretability, and the roles of physicality, sociality and space in
rendering these systems meaningful.
The original tangible systems presented in this master’s thesis are for the most easily
characterized as interpretive. Though representational aspects are sometimes present,
they are generally not the dominant mode of these systems. My project here is not simply
to present the systems that we have designed, but to evaluate them, and to use them as
opportunities to build theories on how people form meanings around interactions with
tangible systems.
22
Chapter 3
Damage: An Interpretive Prototype
Damage is a prototype device for mobile ambient awareness of a social group. It is in
some ways inspired by prior work in interpretive intimate interfaces, and as such is
conceived as a device that would appeal primarily to tight-knit groups of friends. Mobile
phones, an omnipresent computing platform, drive a wearable tangible interface and
ambient display for minimalist, interpretable communication.
This chapter is based on material that has been previously published in the proceedings of
the 2006 ACM Conference on Human Factors in Computing (CHI), Montréal, Québec,
Canada (Williams, Farnham et al. 2006). The project was done at Microsoft Research
during the summer of 2005 as part of a larger project developing a group messaging tool
for Microsoft Smartphones.
3.1 Introduction
Mobile devices are an accessible and increasingly ubiquitous computational tool.
Potential as a computing device notwithstanding, users still view the mobile phone as a
phone; its primary appeal is not computing power but the ability to engage in both remote
and collocated communication. It is the computational power of the mobile phone,
however, that enables multiple modes of communication, allowing it to go beyond the
one-to-one voice communication of the traditional telephone. The use of text messaging
(Grinter and Eldridge 2001) and mobile push-to-talk (Woodruff and Aoki 2003) have
23
been explored by HCI researchers, and it has been noted that constant availability and
constant awareness are becoming commonplace. How, then, can we leverage the
computational power of the mobile phone to allow people to communicate and connect in
innovative, peaceful, emotionally satisfying ways?
In this chapter we first examine related research in mobile group communication and
present Slam, a mobile group messaging system for the Smartphone. Driven by issues
raised by this investigation as well as user feedback on Slam, we designed Damage, a
wearable ambient display of group activity. Early user feedback on Damage highlighted a
diverse and highly situated set of mobile communication needs. We then discuss design
considerations for mobile ambient awareness devices.
3.2 Mobile Group Communication
3.2.1 Prior work
Devices for mobile communication are becoming increasingly ubiquitous, with the
number of mobile subscriptions per capita estimated at 84% in the UK, 92% in Italy, and
106% in Taiwan in 2002, pinpointing that year as the pivotal moment at which mobile
phone use exceeded that of land line phones (Srivastava 2004). Research in use of SMS
among young people reveal patterns of lightweight continuous contact (Grinter and
Eldridge 2001; Ito 2001) as well as sharing of messages amongst collocated friends
(Palen
2002).
Mobile
push-to-talk,
having
noticeably
different
interactional
characteristics from phone service, has also been observed to maintain long-term remote
presence (Woodruff and Aoki 2003). The cell phone hardware itself can be a site for
24
enacting one’s identity, signifying stylistic preferences or membership in a subculture (Ito
2001).
One of the most compelling reasons stated for buying and using cell phones was to
coordinate a family’s complex schedule (Crabtree, Nathan et al. 2003). This coordination
tends to be cobbled together using currently available pairwise communication
capabilities of the mobile phone, but recent research indicates that group communication
might add to these practices. By “group communication” we refer not to a collection of
one-to-one communications amongst a group of friends, but rather a form of one-to-many
or many-to-many communication by and for all members of a group. Farnham et al
(2004) note that close-knit local social groups depend heavily on group communication in
the form of email lists in order to coordinate activities. While one-to-one communication
is, of course, an important part of their social lives, group-wide communication makes
coordination for face-to-face interaction much more tractable.
3.2.2 Slam
Slam is a group messaging application for the SmartPhone running Windows Mobile. It
provides a user interface for lightweight formation of and navigation between groups. For
each group, messages are archived and displayed on the phone in reverse chronological
order, providing a clear view of a group’s conversation. Text messages are sent to the
entire group. Photos can easily be included in a Slam message and are integrated into the
display of messages in recipients’ phones.
25
Figure 3.1: Slam main screen
Feedback from participants in a 10-day-long study of Slam use indicated that they liked
the feeling of being “always on”, and felt significantly more connected to their social
group. An average of 3.98 messages were sent per day, including simple and repeated
“good night” messages, which replicates previous findings in studies of SMS use (Grinter
and Eldridge 2001; Ito 2001). However, some users regarded the constant vibration of
their phone throughout the day, often at work, as a nuisance. This feedback led us to
design Damage, in order to allow unobtrusive, yet intimate, communication based on the
Slam application.
3.3 Damage
3.3.1 Design
We approached the design of Damage (so named because it provides tangible evidence of
Slamming) with the goal of creating a wearable ambient connection to a social group.
Because the display was to be a semi-public piece of jewelry, we aimed for flexible
26
interpretability, allowing groups of users to socially assign meaning to the display in
ways that would not necessarily be evident to strangers.
After several iterations, the design we arrived at was similar to a studded bracelet. Each
bracelet contains six studs, five representing individuals, and one representing wholegroup activity. This number seemed appropriate due to prior research into SMS use
indicating that people typically communicated regularly and frequently with 4-7 friends
(Grinter and Eldridge 2001). Each individual stud contains four LEDs in red, blue, green,
and white. A pulsing white LED on one of these studs indicates that a Slam message is
waiting from that person. The colored LEDs glow steadily and their meanings are
determinable by members of the group.
Previous projects in social communication (Nelson, Bly et al. 2002; Kaye, Levitt et al.
2005; Kikin-Gil 2005) indicate that people readily develop meaningful codes, give
minimal but flexible and interpretable means of communication. Red, blue and green
messages can be sent from the bracelet by opening and closing three metal snaps. The
sender of the message can send another signal to turn the LEDs off, otherwise they will
Figure 3.2: Damage design sketch
27
fade out after half an hour. The group stud contains one white LED that glows
incrementally brighter as messages are sent amongst the group, with the glow
diminishing over time if no messages are sent. If the group is larger than five people, then
messages sent by individuals not mapped onto individual studs will still affect the group
stud. The group stud is visually and tactilely distinguishable from the individual studs.
3.3.2 A Usage Scenario
Alice is a college student who routinely engages in both group and individual messaging
on her phone in Slam with a group of seven of her friends. She most frequently
corresponds with her three roommates (Carol, Louis and Hattie) and a couple of close
friends (Kat and Bunny); these five friends are usually displayed on her Damage bracelet
so she can feel aware of them even in situations where she cannot use her phone, though
when she was at a conference recently she switched out her roommates for her three
classmates with whom she was attending. Alice, Kat and Bunny have long ago agreed on
a few basic codes; when Alice gets out of class at 3:30, she closes a snap on Damage to
light her friends’ bracelets blue. At 3:50 she notices Kat’s stud on her own bracelet is also
lit up in blue, and moments later it starts pulsing white. She checks Slam, and there is a
message from Kat to the whole group asking if they want to meet at the coffee shop.
Alice sends another message to the group: “I’m down. I’ll light up red when I get there.”
On her way, Bunny’s stud lights up red and Alice finds her waiting when she arrives.
Alice sets her bracelet again to let Kat know that both friends are meeting her. After a
few minutes, Bunny, impatient, opens and closes a snap on her bracelet, causing her stud
on her friends’ bracelets to flash red on and off. Kat responds by briefly flashing red,
reassuring the two that she is aware of them; she arrives at the coffeeshop a few minutes
28
later, apologetic for her lateness. During coffee, Alice notices Louis’s stud light up green,
which they have agreed means he has arrived at home.
3.3.3 Implementation
The SmartPhone on which Slam runs is Bluetooth enabled. A Promi-ESD-02 Bluetooth
chip was used to enable Damage to communicate with the phone. A PIC16F877A
microcontroller communicated with the Bluetooth chip over UART, sending and
receiving short messages to and from the phone. Messages specify the color of the LED,
which stud to activate, and whether to turn the light on or off. The PIC parses the
message it receives and controls the pulsing, on/off status, and fade-out time of the LEDs
on the bracelet accordingly. On button presses, a message is sent back to the phone
specifying color and on/off. The message is sent to other group members’ phones via
Slam, and the mapping of individual contacts to bracelet studs is managed on the
recipient’s Smartphone, also in the Slam application. The prototype bracelet consisted of
two layers of leather sewn together with circuitry sandwiched between. The studs were
5/8” in diameter and molded of polyurethane with four LEDs embedded in each. The
“buttons” were metal snaps on leather tabs that could be snapped in place or unsnapped
to signal friends’ bracelets to blink on or off.
29
Figure 3.3: An early prototype of Damage. A BasicSTAMP is used to control the pulsing of the LEDs.
3.3.4 Early Feedback
Two focus groups were conducted, one comprised of three men and two women between
the ages of 20 and 30, the other of 4 men and 3 women between the ages of 16 and 19, all
frequent users of cell phones and SMS to coordinate social activity. Slam was described
and demonstrated to them, as well as the idea of small accessories that can communicate
with the phone. Participants were given pencils, paper, crayons and modeling clay, and
asked to design devices for group communication. Amongst focus group participants,
designs involving bracelets and watches were popular, though a few suggested necklace
displays that would be more visible to others than to the user. There was some desire to
display photos of family members. The ability to receive alerts were important to
participants with children, as well as younger participants who lived with their parents.
Participants were next shown a physical prototype of Damage and asked for feedback.
The younger set was generally more positive, with some requests for more variety in
physical styles. Almost all participants wanted greater flexibility, some requesting
30
multiple group studs, some stating that they only wanted to keep track of a significant
other, and many indicating that they wanted more flexibility in tracking individuals from
multiple social groups. Members of the younger group expressed concerns that a
wearable social display might get them in trouble in class (something we’d been
designing to avoid) and suggested a design that included a watch, giving them a
legitimate reason to look down at their wrist, with subtler indicator lights surrounding the
watch face.
Based on our experiences with our early prototypes and feedback from these sessions, we
present several design considerations for wearable tangible/ambient communicative
devices.
3.4 Design Considerations for Wearable Ambient Communication
Brewer (2004), in proposing design considerations for site-specific ambient displays,
describes “sites” to include “an abstract situation (e.g. a group of friends)”. This resonates
with our own findings; the site for which Damage was designed is not a place but a group
of people. We contend that such a display of social activity must be well-situated in the
social practices of the person and group that uses it. In the course of designing and
prototyping Damage, several considerations emerged as particularly important for our
design.
31
3.4.1 Quiet Technology
Our goal in creating Damage is to provide a non-interrupting visual indicator of Slam
activity, fostering a sense of connection without constant distraction, and enabling a
smooth transition to active messaging when desired. Conveying information in a
peripheral, non-distracting way is a design consideration for any ambient display, but also
a particular concern for mobile and wearable devices. Worn by the user in many different
physical and social settings, a wearable communication device has the potential to
distract not only the user, but also anyone nearby (a phenomenon that we know only too
well from our experience with cell phones). Quiet peripheral display, then, can be
particularly valuable for wearable devices, as noted by (Hansson and Ljungstrand 2000).
3.4.2 Negotiated Interpretations
In his work with intimate objects (Kaye, Levitt et al. 2005) Kaye explores the amount of
interpretation that can be applied to the simplest possible communication – a single bit –
depending on the context of the communicating couples and their current situations.
Buddy Beads’ (Kikin-Gil 2005) semi-secretive means of sending coded communications
(by tapping beads) informed our design. Damage was designed to engage users in a very
open-ended interaction, in which the meanings of the minimal light display could be
negotiated amongst users. Weilenman’s ethnographic work studying ski instructors’ use
of a minimal mobile awareness device (Weilenmann 2001) indicates that, with
interactionally flexible communication devices, norms of use develop through face to
face interaction about the systems as well as mobile interaction through the system.
Mobile users may switch back and forth between these modes of interaction, sometimes
32
encountering situations in which rich communication is not possible. The context that
renders these displays interpretable may tend to be almost entirely social and negotiated,
since physical settings are apt to be unpredictable.
3.4.3 Semi-public display
Feedback from focus groups indicated that our users were quite savvy to the semi-public
nature of a wrist-worn display, even mentioning the inclusion of a watch face to provide
social camouflage for their use of Damage. The general visibility of a wearable display
has implications for how the information is displayed, and dovetails with our
considerations around “negotiated interpretations”. Social negotiation may render an
ambient wearable device meaningful to a group of friends, but conversely strangers may
find it unintelligible or regard it as purely aesthetic.
3.4.4 Tangibility and Intimacy
A critical component of Damage and other wearable communicative interfaces is their
physicality; they are tangible tokens of social connection. They could quite easily have
been conceived as mere displays of sensor data or location, but instead physical action
must be taken to communicate. Certainly there are reasons for this that are related to
personal privacy and the wish to avoid surveillance, but we contend that there is another
purpose to the physical input. Designed for groups of close friends, these devices seek to
reproduce certain aspects of the intimacy these groups manifest when co-present, an
intimacy that is often physically expressed through touch and proximity.
33
3.5 Discussion
Damage as an exploratory project was successful, inasmuch as such a project is meant to
raise interesting questions. In designing this system in support of intimacy, we echoed
practices of intimate groups of friends, such as visible indicators of solidarity (e.g.
friendship bracelets, matching cell phone charms), or secret languages and inside jokes.
In considering the negotiated interpretations aspect of our design, however, I began to
think about how those practices might inform design in deeper ways. Rather than simply
echoing them, why not question what they accomplish, how they come about, how they
work? Designing for practices that are interpretable only to a select few people who share
a great deal of common experience raised questions about how we collectively render
actions understandable, and how interpretive systems might fit into this process.
34
Chapter 4
Meaning in Tangible Interaction
Issues raised by my work on Damage focused in large part on how one’s interactions
with the system might come to mean something, and how the system might be
meaningful in different ways to different users and the people around them. At core, the
representational and interpretive tangible systems presented in Chapter 2 typify different
approaches to creating meaningful interaction. In order to fully understand what makes
these systems understandable, and how their approaches differ, it will be necessary to
unpack how things mean.
In a prior, and more thorough, discussion of meaning in embodied interaction, Dourish
(2001) breaks meaning down into three aspects: ontology, intersubjectivity, and
intentionality. Ontology is concerned with how the world breaks down into individuated
objects and entities that can be referred to in meaningful ways. Intersubjectivity is
concerned with how multiple individuals can come to shared understandings of the
world, with how meaning is shared. Intentionality deals with the directedness of meaning,
of how one entity can refer to another.
These three aspects, while analytically distinguishable, are inextricably intertwined. A
thorough treatment of all three, however, is beyond the scope of this thesis. I am
concerned first, with the design of tangible artifacts as representational objects, and
35
unpacking just how those objects can be taken as representational. In addressing how
representational tangible interfaces manage to be meaningful, I will first discuss
intentionality and indexicality, and their implications for how an entity can be taken to
represent something. Secondly, I will discuss ethnomethodology and its ways of dealing
with how people collectively negotiate contingent meanings. Ethnomethodology focuses
in large part on how people use language, but it also draws heavily from Alfred Schutz’s
Phenomenology of the Social World, which is concerned primarily with the
phenomenology of intersubjectivity, which I will discuss last in this chapter. Schutz and
other phenomenologists, to be discussed in Chapter 5, exhibit a concern with action,
bodies and space, as well as language, providing valuable insights into interpretable
tangible systems and technologically mediated creation of shared meanings.
4.1 Intentionality and Indexicality
Intentionality is, to put it simply, the property of being representational, of being about
something. Less simply, “Intentionality is the term philosophers use to refer to the
‘directedness’ of meaning. Intentionality proposes meaning as a relationship between
some entity (perhaps a thought or utterance) and some other entity (its meaning)” (ibid).
As such, it has historically been a major concern both of philosophers, linguists and more
recently cognitive scientists. Philosophers such as Brentano (1995), Husserl (Husserl and
Cairns 1977), and Dennett (1993) have all exhibited an interest in the intentionality of
thought, the ways in which thoughts can be thoughts about something. Dourish (2001)
refers to this as “original” intentionality, requiring a consciousness that is capable of
meaning something. The intentionality of language – or even, I would argue, gesture,
36
icons, and action – is “derived” intentionality, meaningful because of the original
intentionality of its users and interpreters. Even among philosophers who know a great
deal more than I do about mental phenomena, original intentionality is a hornet’s nest of
unanswered questions and contradicting speculations; my concern here is with the
derived intentionality of language and action, which we are capable of actually seeing at
work in our everyday lives. How does a map refer to territory? How does an icon (or
phicon) refer to a file? How does a lit LED refer to your partner’s emotional state?
4.1.1 Tokens, Icons and Indices
Pragmatist philosopher Charles Sanders Peirce considered three ways in which one entity
might stand in a relationship of meaning with another. His primary focus was on the
symbols of formal logic, but his ideas have been usefully applied to natural language and
sociology by Suchman (1987) and Garfinkel (1984). He broke the signifying relationship
down into three types: tokens, indexes and icons. The meaning of tokens is conventional
or arbitrary, yet necessary for generality. The way in which the spoken sound (or written
image) of “tree” is understood to stand for a tall, sturdy, brown and green, leafy sort of
vegetation is an example of this. Icons resemble the things they represent. Indexes point:
The index asserts nothing; it only says “There!” It takes hold of our eyes, as it
were, and forcibly directs them to a particular object, and there it stops.
Demonstrative and relative pronouns are nearly pure indices, because they
denote things without describing them; so are the letters on a geometrical
diagram, and the subscript numbers which in algebra distinguish one value from
another without saying what those values are.
(Peirce 1885)
37
Any adequate system of notation, claims Peirce, would need all three sorts of signs.
Indexes in particular are necessary to specify the subject of the discourse, which “can
only be indicated. The actual world cannot be distinguished from a world of imagination
by any description. Hence the need of pronouns and indices, and the more complicated
the subject the greater the need of them” (ibid).
Indexicality, then, could be seen as a kind of intentional relationship. It is unavoidable,
and arguably, it is a kind of intentional relationship that provides us with insight into
meaningful embodied interaction.
4.1.2 Indices in Natural Language
While Peirce was concerned with logical and algebraic notation, ethnomethodologists
Sacks and Garfinkel (1970) expanded and applied his ideas of indexicality to natural
language interactions. Drawing heavily from Husserl, they enumerate a number of
important traits about indexical expressions:

Their sense cannot be decided without knowing something about the speaker and
the circumstances of the utterance.

At a given use, an indexical expression refers to one thing, but on a different
occasion may refer to something else.

Thus, indexical statements can change in truth value.

Replicating the expression does not necessarily name the same thing as the
original expression in its original situation named.
38

The denotation of the expression is relative to the user.

“Their use depends upon the relation of the use to the object with which the word
is concerned.”

In the case of temporal indexicals, the time of the utterance matters.

In the case of spatial indexicals, the location of the utterance matters.

Because of these particularities, indexical expressions are not freely repeatable or
reproducible.
Garfinkel argues against the idea, prevalent at the time in the social sciences, that every
indexical expression can, in theory, be replaced with an objective, reproducible
expression (Garfinkel 1984). To illustrate this point, he had his students “translate”
normal conversations from an actual transcript into accurate, clear, and distinct
explanations that could be read literally and understood by someone with absolutely no
prior knowledge. The assignment proved impossible:
We now see what the task was… that they found increasingly difficult and finally
impossible…. I had required them to take on the impossible task of “repairing”
the essential incompleteness of any set of instructions no matter how carefully or
elaborately written they might be. I had required them to formulate the method
that the parties had used in speaking as rules of procedure to follow the order to
say what the parties said, rules that would withstand every exigency of situation,
imagination, and development.
(ibid)
This experiment is interesting not simply because the students could not manage to do it,
but because of the reasons that they could not do it. The impossibility of the assignment
was, ultimately, not an issue of insufficient time, motivation, or paper, but the structure of
the task itself. The explanation, it seemed, could always somehow be made clearer, more
39
accurate, or more distinct, developing the conversation into a “branching texture of
relevant matters”. In describing this experiment Suchman (1987) summed up the problem
as the fact that “every utterance’s situation comprises an indefinite range of possibly
relevant features.” The entire conversation was infinitely indexical and could not be
extracted from its context.
Thus far we know that indexicality is a kind of intentional relationship, an unavoidable,
crucial component of expressive, meaningful systems. We know quite a bit about how
not to derive meaning from indexical expressions: we can not indiscriminately repeat
them and retain their meaning. We cannot derive their meaning without understanding
their context. We can not translate them into objective expressions; it’s indexicals all the
ways down. And yet, despite the ubiquity of indexical statements in natural language, we
manage to create representations that work, we manage to understand one another, and
we manage to agree upon and share an objective social reality. So how do we derive
meaning from our interactions?
4.2 Ethnomethodological Approaches to Meaning-Making
One of Garfinkel and Sacks’ central claims was that the objective production and display
of knowledge and meaning is routinely achieved despite the indexicality of natural
language, that practical sociological reasoning is actually directed towards this
achievement. They specifically note:
(1) that the properties of indexical expressions are ordered properties, and (2)
that they are ordered properties is an ongoing, practical accomplishment of every
actual occasion of commonplace speech and conduct
(Sacks and Garfinkel 1970)
40
To illustrate this assertion they point out the practice of “glossing”, a set of practices in
which speakers “mean something different from what they can say in so many words.”
The meaning, therefore, remains undefined unless or until the context is understood, and
the course of the dialogue composes the context, providing the glossed phrase with its
functional character. Natural language is thus reflexive: “whatever he says provides the
very materials to be used in making out what he says.” Members of a conversation may
use a part of the conversation as an opportunity to describe what that conversation is
doing, as a part of that conversation. Such a formulating statement is both part of and
about the conversation. It is available, observable, and reportable to everyone involved in
the conversation. These practices are simply “phases of interactional enterprises”.
Garfinkel uses the term “ethnomethodology” (the study of member’s methods) “to refer
to the investigation of the rational properties of indexical expressions and other practical
actions as contingent ongoing accomplishments of organized artful practices of everyday
life.” (Garfinkel 1984) As a theoretical framework, it rejects a top-down social-sciences
approach that focuses on how social rules and conventions affect behavior; rather they
are concerned with how members – as defined by their competence at understanding
what is being done, and making others understand what they are doing – produce and
manage everyday affairs in ways that are accountable and understandable to each other.
The objective reality of the social patterns that professional sociologists study is an
ongoing achievement of socially aware practitioners, not an overarching force that directs
the actions of “cultural dopes”.
41
Ethnomethodology contributes several important points an investigation of meaningful
interaction. The first has already been discussed at length: our everyday use of natural
language is indexical. More broadly, many of our actions are directed towards the
situation in which we act, and their meanings depend on context. Second, language in
particular, and our practices more generally, are reflexive; as we speak and act, we
provide the tools for understanding what we are saying or doing. What we do generates
the “context” that renders our doings understandable. Third, these actions are
accountable, in the sense that we are able to give an understandable account of what we
are doing as we do it. Our actions are observable and reportable, resources not only for
ourselves but for others. Fourth, to understand what someone is saying (or doing) is to
know what they are trying to accomplish. To understand is not simply to see that
someone does something, but to attribute those actions to an aware and rational actor
with goals. And finally, these actions and understandings are contingent. We make sure
that they are good enough for the situation they occur in.
According to the ethnomethodological accounts, then, people’s speech and action
generate their own context and render themselves interpretable. Suchman (1987) states
that “language takes its significance from the embedding world, in other words, even
while it transforms the world into something that can be thought of and talked about.”
This hints at the possibility that representation doesn’t just “point”, it touches. Ontology
is not just a collection of objects, and intentionality the choice of which object to point to.
Rather, representations help construct the entities that they represent.
42
4.2.1 Common Ground
Another phenomenon highlighted by Garfinkel’s experiment is that our processes of
understanding take place against a background of common expectations, just as, visually,
figure and ground constitute one another. This background is not something that is
ennumerable, or listable, or articulable, but rather is constituted over time in the doing of
everyday actions. We are constantly filling in the sketchy bits. In order to share
understanding, we need shared ground.
What are some of these unarticulated commonsense expectations? Garfinkel draws
heavily on Afred Schutz to compile a list of things that “the person assumes, assumes that
the other person assumes as well, and assumes that as he assumes it of the other person
the other person assumes the same for him1” (Garfinkel 1984). To be clear, neither
Garfinkel or I would claim that these assumptions always hold true, simply that those
situations in which they do not all hold true would require some coping. My paraphrase is
considerably simplified:

A witness’s determination of an event will be objectively accurate. (Though
objectivity here is collectively achieved.)
1

Things are what they appear to be.

One can affect and be affected by events.

Meanings are the products of a socially standardized process, such as language.
Yes, he really did write that.
43

The meanings of an event were intended on previous occasions and will be again.

One remembers this instance of the event.

An event’s context of interpretation is commonly understood, the sort of thing
anyone knows.

Other people would take an event to mean the same as the witness would if they
were in the witness’s place.

Interpretations are dependent on each witness's personal biography, but internal
individual determinations are irrelevant to the task at hand. Both witnesses select
interpretations that are sufficient for their shared practical purposes.

There is a disparity between public and personal interpretations, and some private
knowledge is held in reserve.

People can alter this disparity in a controlled manner.
This body of shared background assumptions seems, at first blush, to conflict with the
earlier focus on order and meaning as produced by people’s collective actions, locating
the organization of meaning outside of the action itself. The key here is that first,
people’s judgments regarding these expectancies remain situated, contingent and
collaboratively negotiated. And second, accumulated actions, interactions and
experiences contribute to these assumptions, and cyclically these assumptions contribute
to the active organization of meaningful action. (In this sense, “background” can be
thought of not only in the figure/ground sense, but also in the sense of biographical
background.) A discussion of these background expectations is not meant to detract from
ethnomethodology’s fundamental focus on actors and action. However, the accumulation
44
through action of background expectancies is worth considering in the case of
technological deployments or augmented spaces that come to be collectively understood
and appropriated over time.
These expectations serve as the starting point for making sense of and accounting for
speech and action. If these assumptions are violated, then people’s cooperative processes
of sense-making are undermined. This was, in fact, the structure of many of Garfinkel’s
social experiments. "Procedurally it is my preference to start with familiar scenes and ask
what can be done to make trouble," he claims (ibid. 37). Thus he directed many of his
students experimentally to violate some of these expectations. An assignment might
require a student to behave as if things were not as they appeared to be on surface, by
treating all conversational partners as if they had hidden and suspicious motives. Or one
might behave as if meanings were not socially standardized, and request clarification for
commonplace conversational idioms. The results were often disruptive and alienating.
While I am wary of the deception involved in these social experiments, and I am certainly
the last person to condone using ones’ students as guinea pigs, Garfinkel’s probes did
serve to defamiliarize our everyday social interactions in informative ways. The
introduction of novel technological systems often proves disruptive (or more
optimistically, provides occasions for social restructuring). Ethnomethodology’s tradition
of social probes indicates that these instances may be valuable opportunities to examine
how people construct meaning around and through ubiquitous technologies.
45
Furthermore, ethnomethodological analysis frames indexicality – and thus a significant
aspect of intentionality and representation – as, at its core, a problem of shared
understanding. The rest of this chapter will therefore focus on a precursor to
ethnomethodology, Alfred Schutz’s phenomenological approach to intersubjectivity.
4.3 Intersubjectivity and Phenomenology
Schutz, like Garfinkel, was primarily interested in how meaningful social acts actually
become meaningful to people and how they may be recognized as such. How, if we only
experience our own mind, can we arrive at shared understandings?
First of all, visible action is critical: "I apprehend the lived experiences of another only
through signitive-symbolic representation, regarding either his body or some cultural
artifact he has produced as a 'field of expression' for those experiences." (Schutz 1967)
We know that we are visible to others as they are visible to us. We know that they know
their actions are visible.
Secondly, we evaluate others’ actions according to the assumption that they are similar to
ourselves. Schutz refers to the “Thou”, “that consciousness whose intentional Acts I can
see occurring as other than, yet simultaneous with, my own.”
While Schutz denies that multiple people can actually share a stream of consciousness, he
claims that as we mutually watch each others’ lived experiences “our respective streams
of consciousness intersect”. While we do not directly share experiences, we do share
46
artifacts, and we do see each others actions. He refers to this simultaneity as “growing old
together”.
4.3.1 Representational and Interpretive Revisited
Ethenomethodology does not admit to degrees of indexicality or context-dependence –
everything, apparently, is infinitely indexical. So then where is the difference between
representational and interpretive interactions? Does a seemingly obvious interactional
disctinction break down upon closer examination? We’ve seen, however, that indexical
expressions and context dependent meanings are mutually understandable in part because
of shared ground. The representational interfaces, I would argue, take advantage of
ground that is more widely shared. Many people have used maps. Many people
understand how to stack things such as legos. They use assemblages whose ways of being
meaningful are already familiar to users. As for interpretive systems, there is less there
that has been ready-made. People have to work and construct meaning from the ground
up.
4.3.2 The Role of the Physical
While Garfinkel and Sacks focus largely on speech and conversation – using language to
achieve meaning – Schutz was concerned just as much with physical embodiment and the
ways in which it enables us continually to achieve intersubjectivity. Animate bodies and
produced artifacts feature as important indicators for understanding other selves. Schutz
47
suggests that bodily movements are comprehended much like objects, until we are able to
attribute them to another consciousness structured like our own.
Garfinkel points out the usefulness of probes that disrupt the unarticulated background, as
a way to figure out just what that background is. Suchman (1987) formulates people’s use
of a technological system as an opportunity to analyze their situated processes of
understanding. The importance of the physical to Schutz’s account of the social world
indicates that specifically tangible computational systems, situated in physical settings,
could be valuable socio-technical probes. Interpretable systems, as I have stated, provide
the opportunity to construct personal, intimate, situational meanings “from scratch”.
Tangible systems situate the work of constructing meaning visibly in the physical world,
where it is observable (and reportable) by both users and researchers.
The next chapter addresses the critical role of the body and space in achieving shared
understandings.
48
Chapter 5
Space and the Body in Tangible Interaction
In the previous chapter I discussed the importance of shared understanding in making
systems meaningful. In unpacking how these computational artifacts can be interpreted as
meaningful, I addressed their context-dependence, introduced the ethnomethodological
framework for understanding how people produce socially situated meanings, and then
traced ethnomethodology back to some of its roots in Schutz’s phenomenology. In going
back to Schutz, it becomes evident that physical bodies and space are important resources
for intersubjective understandings.
Collaborative tangible systems are well-positioned to take advantage of – and elucidate
the role of – bodies and space as resources for collective meaning-making. Among many
tangible systems discussed in Chapter 2, however, exploration of social context and
speculation as to how the systems’ very tangibility might support collaboration have been
cursory – along the lines of noting that because the table was large, many people could
gather around it. In this chapter I will discuss a number of tangible systems that attempt
to leverage intrinsic properties of interaction in physical space specifically in order to
foster collaboration, as well as observational studies of people’s use of space to
externalize and organize tasks and thought processes. Next, I will examine
phenomenological approaches to space and the body, and introduce Merleau Ponty’s
work on perception. Finally I will attempt to clarify some of the ways in which the spatial
49
and the social intertwine, how our actions and the spaces in which we perform them coconstruct one another.
5.1 Collaborative Tangible Interfaces… In Space
In Ishii’s initial discussion of tangible bits (Ishii and Ullmer 1997), the other major
project discussed, beside metaDESK, was the ambientROOM. A counterpart to the
metaDESK’s tangible components that demanded focused attention from the user,
ambientROOM was meant to render the environment informative in nonintrusive ways,
to take advantage of the user’s background awareness. This does not mean that the user’s
foreground attention was never used, simply that the room did not demand it at all times,
as one of the example interactions suggests:
A steady pattering of rain might remain at the periphery of the user’s attention,
allowing the user to concentrate on foreground activities such as reading e-mail.
However, if the sound of rain suddenly stops or grows louder, it will attract his
attention, causing him to grasp the object and bring it onto the metaDESK, for
example, displaying more detailed interactive graphical information about recent
web page activity.
(ibid)
While the work on ambientROOM is useful insofar as it highlights the ways in which
environments can be informative, it does not foreground the ways in which people
actively construct and interpret the places that we inhabit.
An examination of socially constructed and technologically augmented spaces does take
place in the “Re-Tracing the Past” project. Based on an ethnographic analysis of museum
space, Ciolfi (2004) creates a museum exhibit designed as a room with interactive objects
meant to provoke curiosity amongst visitors, who slowly uncover, bit by bit, an
50
explanatory narrative. Moving about the exhibit and examining the artifacts within,
visitors gradually built up a “sense of place”, and could even leave a bit of themselves by
recording vocal statements to be heard later by other visitors.
In support of, perhaps, a more open-ended activity than visiting a museum exhibit, Ju et
al (Ju, Lee et al. 2007) constructed a computationally augmented whiteboard that
switches to appropriate modes of interaction based on users’ proximity to the board. This
system is based on the observation that collaborative uses of whiteboards switch between
different modes – writing, reviewing, considering, debating, showing – and that these can
be characterized partly by where people locate themselves relative to the whiteboard;
their spatial situation is meaningful. Though this design was based on how people act
around unaugmented whiteboards, the augmented whiteboard still influenced their
behavior; over time users learned where to stand in order to switch the augmented
whiteboard into the mode they wanted, their spatial situations taking on new layers of
meaning.
While successful designs of spatial-tangible systems are in themselves informative to
other researchers, a theoretical approach to the area is increasingly becoming recognized
as necessary.
5.1.1 A Foray Into Theory
In a series of deployments of tangible collaborative systems, Eva Hornecker devises
design themes for tangibles. She asserts that one of the salient features of tangible
51
interaction is its embeddedness in physical space. In her discussion of past work, she
notes some – mostly cursory – discussion of “visibility of actions and distributed loci of
control” (Hornecker and Buur 2006) and notes also that we “lack concepts for analyzing
and understanding the collaborative aspects of tangible interaction and design knowledge
on how to design for collaboration” (ibid). She characterizes past work as data-centered,
expressive-movement-centered, and space-centered. Hornecker’s data-centered and
expressive-movement-centered categories correlate by and large with the representational
and interpretive impulses in tangible systems that I introduced in Chapter 2. Her spacecentered category of tangible interaction is quite similar to the work I have just discussed
in the previous section.
Hornecker’s contribution to theory is to point out four thematic aspects of tangible
interaction. First, tangible manipulation is characterized by observable and legible bodily
interaction with physical objects. It can allow for implicit communication, peripheral
awareness. Second, spatial interaction emphasizes the body as a reference point for
perception. Space has social meanings. Space is situated, inhabited, a site for
performance. Third, embodied facilitation focuses on structures in tangible interaction
that encourage, direct, or limit certain behaviors. Fourth, expressive representation is
concerned with how tangible interaction can be “about physical representation of data.”
She notes that “in interaction we ‘read’ and interpret representations, act on, modify and
create them.” These themes each reify and analyze one of several interlinked aspects of
tangible interaction: objects, space, action, and meaning.
52
In (Hornecker 2005), focusing primarily on the embodied facilitation theme, Hornecker
notes that certain constraints are inherent in physical interaction and suggests exploiting
constraints that might induce helping and coordination, while also providing a shared
transaction space. Multiple access points to the interface and the ability for simultaneous
action are necessary for collaboration to take place. She focuses mainly on how the
design of physical objects and the spatial orientation of the system can affect the actions
of users. Her design recommendations are useful, but I am concerned not only with the
effect of the system on the user. Rather I am interested in the user’s interpretation of the
system, and of each other’s actions, and how these might influence each other. My
central claim here is that representation is not an a priori relationship, but rather is
constructed by people in particular situations -- situations that involve an intertwining of
spaces, bodies, artifacts, and actions. Tangible systems leverage this process; they can
also foreground this process for observation in the visible and physical world, and are
therefore valuable as an analytic tool.
While Hornecker’s reification of these four themes in tangible interaction is a useful
analytical standpoint, I believe that a more holistic view of these aspects as intertwined is
also valuable. Proceeding forward then, I will examine views of space as more than just a
container for action, but rather as integral to meaning-making, as socially defined, as
constructed by people along with artifacts. The next two sections will examine first,
space as tool for understanding, and second, space as fundamental aspect of being, along
with bodies and actions.
53
5.2 Space as a Tool for Understanding
Much of what we know is spatial. This is widely understood, if in a simplistic sense; a
classic mnemonic technique attributed to Cicero pairs items on a list with locations along
a familiar route – mentally re-tracing the route helps one recall items on their list
(Anderson 1995). In both cognitive psychology and human-computer interaction,
research has been conducted into “spatial cognition” (Curiel and Radvansky 1998;
Roskos-Ewoldsen, McNamara et al. 1998) and leveraging “our pre-attentive ability to
recognize spatial relationships” (Robertson, Czerwinski et al. 1998) in order to organize
and browse through digital documents. Designers of onscreen 3-dimensional virtual
spaces exhibit a concern with “natural” representations of objects and spatial
relationships (Ark, Dryer et al. 1998).
Moving off the screen, tangible interaction that is manifest in physical space should
certainly be able to take advantage of our everyday tendencies towards spatial cognition.
Patten and Ishii (2000) present a study in which they asked users to read, organize, and
interpret ten news articles using a tangible and a graphical interface. In the graphical
condition, each article was represented by an icon on screen. In the tangible condition,
each article was represented by a physical block. This study produced two principle
findings. First, those who used the tangible system used spatial organization more than
those who used the graphical system. Moreover, they spatially organized the articles in
different ways: "TUI subjects employed spatial encoding techniques which relied on the
position of the blocks within an external reference frame [the physical environment],
while GUI subjects did not." (Patten and Ishii 2000) One specific difference in spatial
54
organization between the two systems is that users of the tangible system organized the
blocks in relation to their own bodies, placing things from left to right centered around
their body, placing a block representing a story about “arms” (as in weapons) near their
own arm, or placing a story about heart disease near their own heart. Second, users who
spatially organized the articles demonstrated better recall both of the content of the
articles and of the relationships between the articles than those who did not, whether they
used the graphical or the tangible interface. Those who used the tangible system and
spatially organized the articles demonstrated the best recall.
A more thorough analysis of the phenomenon uncovered in Patten and Ishii’s study can
be found in some of the work of cognitive psychologist David Kirsh (1995). After
extensive study and video analysis of cooking, packing, assembly, and even playing
Tetris, Kirsh notes that, in performing skilled activities, people spatially organize their
environment and components of the task in order make the task easier. Experts are
distinguished by their ability to “jig” the task environment. He documents three
categories of spatial organization: spatial arrangements that simplify choice (e.g. laying
out ingredients in the order that they are to be cut), arrangements that simplify perception
(e.g. grouping similar objects), and spatial dynamics that simplify internal computation
(e.g. rotating tetris shapes on screen rather than mentally, which would take longer). His
research emphasizes the utility of space as a tool for cognition; he speaks of it as “an
invaluable resource to exploit in order to facilitate everyday problem solving and
planning.” (Kirsh 1995)
55
The main point of the research discussed here is that we think spatially and we use space
to think. The phenomenological view of space would take that assertion a step further and
assert that our experience of space, like that of our bodies, like our actions in the world, is
pre-ontological. Space, bodies, and actions are intertwined, prior to and generative of
meaning.
5.3 Phenomenology of Space, Body, and Action
Within the field of Human-Computer Interaction, embodied interaction turns to the
phenomenological tradition in philosophy to clarify just what “embodiment” is and why
it matters for technology design. Dourish (2001) discusses the ramifications of
phenomenology for tangible and social interaction, arguing that a sound basis in the
phenomenological tradition reframes several problematic issues within HCI. More
broadly, phenomenology’s concern with everyday lived experience and action correlates
with many of the concerns of interaction designers. J.J. Gibson’s idea of affordances has
taken hold in the field of HCI (Baerentsen and Trettvik 2002; Norman 2002), though
some argue that it has been broadly misinterpreted (Ingstrup 2006) as being a property of
an object, rather than a subject’s interaction with object and environment, influenced by
memory and culture, as Gibson originally claimed. This distinction is crucial, because it
puts the emphasis on interaction, rather than static designed features of an object or
system. Phenomenology as well emphasizes action and use, and while that aspect of the
phenomenological approach has been explored by HCI researchers, that branch of
philosophy may also provide some insights about how we experience space in terms of
action within it.
56
5.3.1 Action, Being and Heidegger
Heidegger’s particular brand of phenomenology, emphasizing a being’s engagement with
the world (Heidegger 1996) as being a factor of it acting in the world, has justifiably
become influential in the field of Human-Computer Interaction. Heidegger is
fundamentally concerned with being, specifically, what is means for a self-aware being2
(as opposed to, say, a brick) to be. He refers to this sort of entity as Dasein, and claims
that its fundamental mode of being is care. It is, intrinsically, engaged, with the world,
with self-understanding, etc.
Winograd and Flores (1987) summarize the features of Heidegger’s phenomenology that
seem most relevant to HCI. Interpretation, they claim, pervades our everyday lives, and
to divide the world into subjects that understand and objects that are understood is to
misunderstand “the primacy of experience and understanding that operates without
reflection” (ibid). That is, we are not distant analyzers of the world, but directly bound up
as actors within it. HCI designers must understand that our implicit beliefs and
assumptions cannot all be made explicit, that our practical understandings are more
fundamental than detached theoretical understanding, that we do not relate to things
primarily through having representations of them, and that meaning is fundamentally
social and cannot be reduced to the meaning-giving activity of individual subjects. (These
statements bear more than a passing resemblance to the assertions of Garfinkel and the
This being is often charmingly and confusingly referred to as a “being for whom its
being is a concern”. Hopefully, “self-conscious being” or “self-aware being” conveys the
idea well enough for my purposes.
2
57
ethnomethodologists, due to their shared phenomenological heritage.) Heidegger asserts
as well that we are thrown into an already existing world. As such, we cannot avoid
acting, and we cannot necessarily step back and reflect on our actions. We do not always
have stable representations of the situations we are in, and every representation is also an
interpretation. While we sometimes engage in "conscious reflection and systematic
thought", usually we deal with pre-reflective experience. Rarely can we really disengage
from the situations we find ourselves in. (And disengaging isn't really escaping
thrownness, just shifting our concern.)
All this is often summed up in the phrase “being-in-the-world”. Being-in-the-world
manifests in one particularly illustrative concept that is directly relevant for interaction
designers: that of tools being ready-to-hand or present-at-hand. Those two states do not
belong to the object itself, but rather to a being’s relationship with the object. For a tool
to be present-at-hand, we see it as an object outside ourselves, we are aware of its
properties as a thing. This is one, but not the only or even necessarily the best, way of
understanding an object. Heidegger contrasts the two: “When we just look at things
"theoretically," we lack an understanding of handiness. But association which makes use
of things is not blind, it has its own way of seeing which guides our operations and gives
them their specific thingly quality” (Heidegger 1996). He continues, in explaining what it
means to be ready-to-hand, “what is peculiar to what is initially at hand is that it
withdraws, so to speak, in its character of handiness in order to be really handy. What
everyday association is initially busy with is not tools themselves, but the work” (ibid).
The tool, in effect, becomes an extension of the doer.
58
In everyday life we may find numerous examples of ready-to-handedness and present-athandedness that clarify the concepts. To borrow from Daniel Dennett (1993), try this
experiment: find a pen, stick, or chopstick (don’t forget to cap the pen, or this can go
badly). As you pick it up, stare at it, and wonder what I am about to tell you to do with it,
it is present-at-hand. Hold one end of it, close your eyes, and run the other end of the
stick across things. You can probably discern something about the texture of the thing
that you are touching through the stick, and yet you are probably not registering this as a
conscious interpretation of the vibrations of the stick against your fingers. In this
capacity, the stick is an extension of your perception and action; it is ready-to-hand.
Dourish (2001) points out that when you click on menus in a GUI the mouse is ready-tohand, but when you reach the edge of the mousepad and move it back to the center it is
present-at-hand.
Heidegger’s treatment of action and meaning is enormously useful in understanding how
people approach and appropriate technology. His treatment of space is discussed less in
the HCI literature, yet seems an important element to examine when entering a discussion
of spatially situated, tangible, interactional systems.
5.3.2 Action, Space, and Heidegger
According to Heidegger, space is grounded in our experience of the world, not the other
way around. Dasein’s spatiality is attributable to it on the basis of it being in the world. In
order to use things, we must bring them close. He calls this ent-fernung, which Heidegger
59
scholar Hubert Dreyfus explains as “the establishing and overcoming of distance”, and
which is sometimes translated as “de-distancing”. De-distancing puts an object in the
realm of possibility of use; it may then be more or less available, nearer or farther from
Dasein. De-distancing is prior to any particular Dasein’s nearness to or farness from
objects. (Dreyfus 1991) This is the basis of Dasein’s spatiality, and how we are able to
encounter things in the world as spatial. Experientially, nearness means being “at hand”.
Nearness of something is measured not in terms of Cartesian distance from the location
of a being, but by its availability for use or action. What is near is that with which we are
“absorbedly coping” (ibid). More broadly, space is experienced in the context of the
availability of useful things.
Thus, “space is neither in the subject nor is the world in space.” (Heidegger 1996)
Instead, Dasein is necessarily constituted as being-in-the-world, and that discloses space.
Space as we experience it is interactional. It is revealed by our interactions with our
world.
Heidegger primarily talks about space in terms of nearness and remoteness to Dasein. He
grants, however, that a Cartesian understanding of measurable space may underlie the
physical world, but that is not how beings in the world experience and understand it.
“Bare space is still veiled.” So, the objective way that the world is may involve
measurable space, but it is our subjective experience that “perhaps discovers what is most
60
real about the ‘reality’ of the world.” The relationship between these two modes of
spatiality is never quite made clear3.
In this particular way, Heidegger’s approach to space is “fundamentally confused”
(Dreyfus 1991). Dreyfus points out that Heidegger has an unresolved conflict between
Dasein’s personal, subjective space – in which the world is more or less available, farther
and nearer, from the point of view of a particular Dasein – and public space that is
available to anyone and in which Dasein is situated. He discusses subjective experience
of space a great deal, but leaves it at that, aside from including a brief concession to
Cartesian space.
I would argue that Heidegger’s fundamental mistake here is his exclusion of the body as
a crucial component of a being’s experience of space. He explicitly states that he does not
find the body to be important: “Bringing near is not oriented toward the I-thing
encumbered with a body, but rather toward heedful being-in-the-world, that is, what that
being-in-the-world initially encounters.” (Heidegger 1996) Heidegger conceptualizes
Dasein as “pure concern, not as a physical body located at a certain point in objective
space.” (Dreyfus 1991) Dasein’s experience of space is purely about its concern or
attention, not about its physical embodiment.
The body, however, bridges private and public experience. It is both a fundamental part
of subjective experience, and an object that is available to others. In a following section, I
3
Inasmuch as anything Heidegger writes can be said to be made “clear”.
61
will discuss Merleau-Ponty’s idea of the body-subject, and attempt to show how this can
lead us to a more tenable approach to action and spatiality.
5.3.3 The Being-Action-Space Story So Far

We are in the world. We can’t not interact with it. In fact, that is our primary
mode of being.

Space is revealed by our interactions with our world.

We experience space, subjectively, in terms of the availability of useful things.

This is useful so far, but…

Heidegger does not adequately address collective experience of space.

Bodies might be important, after all.
5.3.4 Body, Space and Merleau-Ponty
Merleau-Ponty places primary importance on the body and the things that we can do with
it. It is both a vital part of the self-aware being, and part of the world; it links us. “Our
own body is in the world as the heart is in the organism: it keeps the visible spectacle
constantly alive, it breathes life into it and sustains it inwardly, and with it forms a
system” (Merleau-Ponty, Smith et al. 2002). No sensory field, by itself, possesses an a
priori spatial orientation. Rather, we need an anchoring point, the body. Our sense of
place, then, is based on our “area of possible actions”, which so far is in agreement with
Heidegger. Where Merleau-Ponty departs from Heidegger is in asserting that our spatial
meanings develop in no small part from the spatial nature of our bodies.
62
He asserts that “every sensation is spatial”, and this shapes our encounters and our
knowledge of the physical objects in our world:
From the point of view of my body I never see as equal the six sides of the cube,
even if it is made of glass, and yet the word ‘cube’ has a meaning; the cube itself,
the cube in reality, beyond its sensible appearances, has its six equal sides. As I
move round it, I see the front face, hitherto a square, change its shape, then
disappear, while the other sides come into view and one by one become squares.
But the successive stages of this experience are for me merely the opportunity of
conceiving the whole cube with its six equal and simultaneous faces, the
intelligible structure which provides the explanation of it… it is… by conceiving
my body itself as a mobile object that I am able to interpret perceptual
appearance and construct the cube as it truly is.
(ibid)
In this example, he makes clear that our experience of this cube comes from a particular
physical and spatial point of view. It is true that we never see all six sides of the cube as
equal, and yet we gather that the cube has six equal sides because over space and time
one’s point of view changes, experiencing cumulatively more of the object. Our bodies
are mobile, allowing us to take in the characteristics of the cube in this way. Thus, even
in our experience of a single object, space, time, and movement are crucial.
While both Heidegger and Merleau-Ponty claim that our perceptions of space hinge on
our capacity for action within it, the latter’s theory is based soundly on the idea that we
act with our bodies, which are already inherently spatial and located. "Space is not the
setting (real or logical) in which things are arranged, but the means whereby the position
of things becomes possible" (ibid). Primarily, according to Merleau-Ponty, we live and
act in space, and this is prior to our perception of space, perception being “the knowledge
that a disinterested subject might acquire of the spatial relationships between objects and
63
their geometrical characteristics” (ibid). It is through action that we learn that places have
structures and “geometrical characteristics” that can be schematized or mapped, and it is
places that make the experience of space possible. We attach meanings and significance
to space according to our body’s capabilities within it. Thus, argues Merleau-Ponty in
one example, do small spaces come off as confining or depressing.
While Merleau-Ponty, like Heidegger, concentrates on the individual experience of
space, by including the physical, visible body in the discussion, Merleau-Ponty gives us a
foundation on which to build theories of collective spatial experience.
5.4 Social and Spatial
By bringing the body into the discussion, we now have the components we need to talk
about both the spatial and the intersubjective. As mentioned in my discussion of Schutz
in the previous chapter, bodies are sites of visible action and important resources for
collective construction of shared understandings. In this section I wish to discuss first,
how phenomenological approaches to embodiment and space can provide a framework
for understanding real-world collaboration; second, the demonstrated importance of space
and bodies to shared understanding and collaboration; and third, the ways in which social
processes render space meaningful.
5.4.1 What Spatially Oriented Collaboration Studies Inherit From Phenomenology
In describing passive awareness in the field of Computer Supported Cooperative Work,
Toni Robertson draws heavily on Merleau-Ponty’s theories of embodied perception, in
64
the hopes that such an analysis can provide some insights for design decisions (Robertson
2002). Perception, cognition, and action are lived together, she claims, and their mutual
constitution is the basis for the rich interpersonal coordination that makes collaborative
work possible:
I want to explore important concepts from Merleau-Ponty’s phenomenology of
perception that do provide this richness. The first is his analysis of perception as
active, embodied and always generative of meaning; perception is not some
private, inner representation of some public, outer world but instead a practical
and material involvement in that world (Williams and Bendelow, 1998, pp. 52–
53). The second is the interconnection of skilful action and perception; our
bodies determine what is available to us in our world in different ways
according to their specific history. The third is his analysis of the two, always
intertwined, aspects of our embodiment; our bodies are physical structures
available to our own and others’ perceptions at the same time as they are lived
by us as our thinking and acting selves. Finally, his notion of reversibility
provides a name for the complex intertwining between the perceiver, the
perceived and the physical environment that is the essential condition for our
interaction with the world and with others.
(ibid. emphasis mine)
Robertson identifies four components, emphasized in the above quote, of MerleauPonty’s phenomenology that are of particular importance to the study of collaborative
work.
First is the idea of perception as the means of connection between a body-subject and the
world it is situated in. “This”, she claims, “is how we have the capacity for peripheral
awareness in the first place.” Our recognition of events in our environment, and the
actions of people around us, will depend on our personal histories and prior experiences.
Shared understandings therefore, must be provided by some amount of common
experiences.
65
Second, perception is an embodied skill; that is, bodies structure it, it is learned over
time, and it requires cultural skill. Perception is learned. Therefore, “awareness” cannot
be a property of a technology or workplace in itself, rather it is achieved by the skilled
practices of participants if they are sharing a space and have publicly available resources
for building understanding.
Third, as we perceive the world, we also perceive ourselves. We live in our bodies, but
they are also objects in the world that are available to our own and others' perceptions. It
is the dual nature of bodies as both sensible and sentient that enables us to “recognise and
understand others' actions by the same process that we shape our own actions for their
interpretation by others.”
Fourth, perception is reversible. This is a stronger statement than simply “if you can see
me, then I can see you.” Reversibility is about the fundamental and continual intertwining
of the sentient and sensible body. It is based on the fact that physical bodies and the
physical world are made of the same stuff, with the same material properties.
Reversibility requires “that as an inseparable part of any single perceptual act the person
being perceived can perceive not just that they are being perceived but how they are
being perceived by others. An individual has to simultaneously be both perceiver and
perceived of their own actions as well as the perceiver of others' actions if those actions
are to fulfill a communicative role that is controlled by the participants in the
interaction.” This is an understanding regarding ones mode of material existence; it is
ontological, not just interpersonal.
66
This framework privileges the social agency of human actors, rather than regarding them
as bound by the situations that they find themselves in. Awareness is learned. In practice,
perception and action, the sensory and the motor, are intertwined. Perception is not a
private, internal act, but instead “is active, embodied and always generative of meaning.
As such it belongs to neither body or mind but is constitutive of both” (Robertson 1997).
Public availability of actions for others’ consumption is an active process. The role of
technology in this ecology then, is to support and provide a basis for the creation of
shared meaning. Robertson (1997) then applies this framework to an in-depth study of
workplace collaboration among a group of software designers in which she devises a
detailed taxonomy of workers’ embodied actions.
In this study, she takes care to emphasize that all the actions she catalogues are indexical,
they are all performed in relation to objects, other bodies, or the physical space in which
they happen. She catalogues individual actions that are performed in relation to physical
objects (moving an object, creating a private physical representation), to other bodies
(emitting signs that are meant to be observed, monitoring other people’s signs), and to the
physical space (moving around, pointing, shifting the direction of gaze). Critically, these
individual actions do not occur in isolation, but also constitute group activities like
conversation, the creation of shared representations, or observing the same thing at the
same time. Mutual perception is crucial to the constitution of all these actions, as is selfperception.
67
Not every workplace study draws so explicitly from Merleau-Ponty’s embodied
phenomenology, but there are common threads that run through many of them: the
importance of visible and accountable action, of mutual monitoring, of bodily
comportment, of publicly available objects and resources, of bodies performing as both
observing subjects and observable objects, in constructing a space in which collaborative
work can happen. Such studies tend also to draw from the ethnomethodological tradition
in analyzing the ways in which people use the resources available to them to actively
generate order and meaning.
5.4.2 Sociality is Spatial and Embodied
Embedded in a discussion of designing systems for social translucence, Thomas Erickson
and Wendy Kellogg provide a vivid example of space and bodies as crucial components
of the social organization of a collaborative task (Erickson and Kellogg 2000). In this
example, thirty authors have gathered to organize the chapters of a book to which they
are all contributing. In an open process, authors could suggest section names, placing
chapters near appropriate section names, or picking up and moving a chapter if they
found a better fit. Participants could change section names and chapter orders at any
point. Though the process might seem unregulated and perhaps chaotic, within a half
hour, the book was organized and everyone had participated in the discussion. What
made it work so well?
68

Visible action played a role. For example, if people are gathered around a section
and an author moves her chapter there, she then ends up accounting for the move
and a discussion ensues.

Spatial constraints were also important. Authors could only be in one place at
once so it was difficult for any one person to dominate the decision process.
Authors had to pick their battles, and that manifested in space by someone staying
close to a section or chapter they cared about. By staying near a certain section
they might gain an in-depth understanding of the rationale for that section over
repeated/varying discussions, but their influence over other sections were also
limited.

Underlying all this was the fact that the ability to see and hear what is going on
decreases with distance -- physical space is translucent, not transparent.
The action was shaped not only the constraints of bodies and space, but also by the fact
that people are familiar with these characteristics and could anticipate how they would
structure action. The background awareness of physical and spatial characteristics is
shared; we all know that we all know the constraints and we hold each other accountable
accordingly.
A comparative study of two software development teams (Chong 2006) highlights the
importance of the spaces they occupy and the physical cues available to collaborating
programmers. Chong spent eight months on site, studying a conventional software
development team and an eXtreme Programming (XP) team. The XP methodology
69
involves several unconventional practices, such as pair programming, minimal explicit
documentation, and a lack of individual ownership of code modules. One of the most
important differences between the XP team and the conventional team, and the difference
that made the XP methodology work smoothly in this case, was the XP team’s shared
space, shared objects, and the visibility and accountability of the developers’ actions. As
a shared resource, a basic diagram of the system’s architecture was tacked onto the wall
of the shared space. Note cards and task descriptions were freely shared. Most
importantly, the conversations and audible knowledge sharing required as part of pair
programming were publicly available as peripheral cues. Programmers routinely made
public announcements to the entire group regarding the state of the project. These
announcements might pertain to global issues like data migration, wide-ranging changes,
or potential interdependencies. Chong notes that these announcements were often
triggered by overheard conversations between other pair members in the room, and thus
could be offered when they were most immediately relevant. The physical characteristics
of the shared space in this case were constantly used by the XP developers to coordinate
their work.
In the previous chapter I discussed Garfinkel and Sacks’ analysis of everyday use of
language in constructing shared understanding. They inherit much of their theoretical
framework from Schutz, but in the work I discussed they tended to emphasize linguistic
action over physical action. Other social scientists using ethnomethodological
frameworks, however, have focused a great deal on bodies and space. A group of
researchers at King’s College London have produced a series of studies of collaboration
70
with detailed analysis of interaction around objects, bodily stances, spatial arrangements
and visible action, generally gathered from long-term engagements and painstakingly
close video analysis. By their own claim they focus on “the full gamut of embodied
conduct evident in face-to-face (and object-mediated) interaction. Thus, there are not
only discrete turns at talk to be considered but also the accompanying layers of bodily
conduct that are produced along with, and alongside, that talk” (Hindmarsh and Heath
2000).
In studies of performance-oriented and intensely collaborative settings such as a Line
Control Room in the London Underground (Heath and Luff 1992) or the Restoration
Control Office of a telecom company (Hindmarsh and Heath 2000) they uncover the
ways that objects, space and bodies are used to coordinate ongoing actions. The London
Underground control room, for example, features a fixed line diagram showing traffic
movement along their line. It is a crucial shared resource, large, spatially prominent, and
publicly available, around which coworkers could coordinate action. Its utility as a shared
object is not only grounded in its own visibility, but also the fact that openly examining it
– indicated by stance and position of the body in the room – can be a publicly available
indicator that certain actions need to be taken.
The visible and audible actions of the people in the room were also informative. Control
room workers constantly monitored each other’s activities even as they performed their
own tasks. Someone’s bodily stance towards the fixed line diagram might be an
important visual clue; at the same time audible clues such as the content of the Line
71
Controller’s phone conversation (sometimes peppered with exclamations that are quite
deliberately meant to draw attention) could suggest appropriate action, such as
announcing a delay to riders on the platform without being explicitly told to do so. Yet
even as coworkers monitored each other, they used their bodies to indicate that they did
not intend to interfere with the performance of other people’s duties: “Through his bodily
comportment and the ways in which he warily accomplishes his actions, the DIA
preserves a careful balance of involvement… whilst avoiding overt attention to the
Controller’s conduct” (Heath and Luff 1992)4.
In contrast to the densely interactive and intensely work-oriented environments of the XP
programming pit, line control room, and the collaborative authoring melee, even the
comparatively relaxed mall food court provides opportunities for embodied coordination
(Manzo 2005). Food court patrons employed numerous strategies to negotiate, claim, and
relinquish space; they used space visibly, spreading newspapers and coffee cups of over
their area, or signaling with clothing, food and drinks whether they would be leaving
soon or staying a while.
These studies all provide detailed examples of the ways that people use material
properties of space, bodies and objects to actively construct meaning. These studies
demonstrate that people make meanings using the resources that are available to them, a
4
To clarify the roles discussed there, the DIA is the Divisional Information Assistant,
who makes announcements to passengers on a public announcement system, and
communicates with station managers. The Controller coordinates the daily running of the
trains – he would generally be the first to hear about problems like train delays, which
would then need to be announced.
72
practice that is crucial for us as technology designers to understand. Yet another
important point, addressed to some extent by Garfinkel’s discussion of background
expectancies, is that these meanings accumulate, evolve, solidify and dissipate over time.
They become the shared background necessary for meaningful interaction. The physical
world is a resource for generative sociality, but at the same time it takes on social
meanings that render it intelligible.
5.4.3 Space is Social
Action socially constructs space, but also social space constructs action. Practices
accumulate, rendering places meaningful, and indicating what actions that place might
afford, both physically and socially, to its inhabitants. This background understanding of
meaningful spaces affects how people collectively act within them. Social and historical
studies of urban spaces provide accounts of this phenomenon.
Geographer Keith Lilley (2004) suggests that residents viewed medieval cities through
the concepts of centrality and peripherality, high and low, a hierarchical Christian
cosmology, and a Neoplatonic interpretation of the body. “Through medieval Christian
interpretations of classical cosmology and biblical cosmogony, the city was understood as
a scaled-down version of the wider world, a microcosm, as well as a macrocosm, a
‘body’ writ large, both sharing in a divine order.” Both the body and the cosmos were
based on threes: God ruling over the angels ruling over man, and reason ruling over
passions ruling over appetites. In the city, the elevated and centrally placed castle ruled
over soldiers and below them tradesmen and ordinary citizens. Norman conquerors,
73
understanding this, were able to re-center Saxon cities architecturally, rendering their
original inhabitants socially peripheral as they rendered them spatially peripheral. Their
social marginalization was simultaneously literally achieved and symbolically
represented.
Historian David Garrioch (Garrioch 2003) reports on urban ambient sounds as a semiotic
system in early modern European cities. Merchants, for example, plied their wares
vocally, using distinguishable rhythms and cadences. Bells were particularly significant,
as they were versatile and one of the loudest noises that could be heard in the city. Bells
from different churches might be tuned differently, and besides the time, they marked the
liturgical season or day of the week. Their ringing pattern might also signify a birth,
death, wedding, mass, or closing of the city gates.
With a complex grammar relying on local knowledge, time of day, and other senses,
inhabitants could read the noises of the early modern city. Additionally, sounds might be
understood differently, depending on people’s affiliation, neighborhood, or social
position. Protestants might respond differently to church bells than Catholics, while
servants could distinguish merchants’ cries that their masters could not understand. In
addition, people could be distinguished not just by the sounds they made but by their
reactions to sounds around them.
The residents of ethnographer William Kelleher’s Northern Irish town of “Ballybogoin”
demarcate the city by “whether a space is populated by people wearing uniforms or not,
74
whether a house is two stories or one story, whether or not you and your neighbor know
each others’ first names, whether one’s movements through space are ‘Protestant’ or
‘Catholic,’ whether you park your car east or west of the town square, whether you talk
or do not” (Kelleher 2004). Inhabitants further distinguish space by painted curbs; the
colors of political posters; people’s clothes, haircut, and accent; and whether soldiers
defaced or ignored graffiti. Though they could see and read the same signs, Protestants
and Catholics had profoundly different interpretations of which spaces were safe and
welcoming.
As the examples in this and previous sections illustrate, bodies, action, space and
meaning intertwine. People read spaces according to who they find inhabiting them.
Meanings are attached to people’s bodily stance and motion, whether it indicates
attention to a shared display in a collaborative workspace, or distinguishes Protestants
and Catholics in Ballybogoin or an early modern European town. Conversely, who you
are, your history, and how you move will affect how a space might be meaningful to you;
note the masters’ and servants’ different understandings of merchant’s cries, or the
divergent interpretations of safe places in Ballybogoin.
Computational technologies and their infrastructures continue the legacy of
environmental features like church bells in constructing socially meaningful spaces.
Enhanced 911 (or E911) allows emergency response services to determine the address of
the calling telephone. In 1996, the US FCC decided to require that wireless service
providers be able to determine the location of cell phones placing calls, such that they
75
could also be incorporated into the E911 system. (Curry, Phillips et al. 2004) argue that
the E911 system renders the landscape – and the people within it – “legible” in new
ways. This process is not new, but a continuation of the rationalization of space practiced
by the making of maps, Jefferson’s systematic survey system (still visible in patterns of
roads and outlines of farms in rural America), the standardization of postal addressing,
and ZIP codes. Personal data (even anonymized) correlated to location renders space
meaningful not only to emergency responders but also to direct marketers, who may find
quite different interpretations of those representations, and be inspired to quite different
sorts of actions.
The socially constructed meaningfulness of spaces is a particularly rich area of inquiry
for researchers of ubiquitous computing. Our constructions and representations of space
are enabled by the resources available to us within our environments, some of which are
now computational. This raises a set of questions concerning technology and space. To
whom is the space meaningful and in how many different ways? Who creates the the
dominant representations? Who controls the technological infrastructure? What
opportunities might we find to reinterpret the spaces we live in, and to what end? What
resources do we have available to do so?
5.5 Summary
Meaning is messy, but for the most part we manage to make it work well enough. And I
use the word “manage” deliberately here – meaning is an active process. In Chapter 4 I
discussed the ways in which meaning is contingent and situated, dependent on the
76
achievement of intersubjectivity for any broad applicability. In our everyday use of
language we provide the very resources needed to interpret that use of language, and
demonstrate our own competence and contingent understandings. Chapter 5 generalizes
this discussion of meaning beyond verbal discourse and brings in the importance of
actions, bodies, objects, and space. We understand the world through our actions and we
are part of it as embodied beings, both perceiving and perceived. If we consider the
tangible interfaces in Chapter 2 as being concerned with different ways of being
meaningful, we now have the tools to understand how tangible computational systems
can be sites of meaning, how they may be constructed by their users, and how they may
be used to construct and make a space meaningful.
In the next two chapters, I will present two tangible systems: SignalPlay and Nimio. Both
were designed to be collaborative, open-ended enough to be freely interpretable, and
spatially situated as part of a particular site. In both projects, the focus was as much on
observing and analyzing people’s use of the systems, as on the initial design and
development phases.
77
Chapter 6
SignalPlay: Configuring Space through Embodied Interaction
When computation moves off the desktop, how will it transform the new spaces that it
comes to occupy? How will people encounter and understand these spaces, and how will
they interact with each other through the augmented capabilities of such spaces? We have
been exploring these questions through a prototype system in which augmented objects
are used to control a complex audio 'soundscape.' The system involves a range of objects
distributed through a space, supporting simultaneous use by many participants. We have
deployed this system at a number of settings in which groups of people have explored it
collaboratively. Our initial explorations of the use of this system reveal a number of
important considerations for how we design for the interrelationships between people,
objects, and spaces.
SignalPlay is a sensor-based interactive sound environment in which familiar objects
encourage exploration and discovery of sound interfaces through the process of play.
Embedded wireless sensors form a network that detects gestural motion as well as
environmental factors such as light and magnetic field. Human interactions with the
sensors and with each other cause both immediate and systemic changes in a spatialized
soundscape. Here we present observations on embodied network interaction and suggest
opportunities for further investigation in this field.
78
This chapter is based on material previously published with Eric Kabisch and Paul
Dourish in the proceedings of Ubicomp 2005, Tokyo, Japan (Williams, Kabisch et al.
2005) and in the proceedings of the 2005 ACM Conference on Human Factors in
Computing (CHI), Portland, OR (Kabisch, Williams et al. 2005).
6.1 Motivation
In ubiquitous computing settings, computation moves “off the desktop,” and, by the same
token, interaction moves into physical space. Interaction between humans and computers
becomes subsumed within, and mediated by, interaction between humans and their
surroundings. This transformation is significant, given that conventional user interface
design has traditionally exploited metaphors of real-world objects that are, nonetheless,
critically distinct from that real world (Ferris and Bannon 2002).
We use the term “embodied interaction” to refer to forms of interaction with computer
systems that are embedded in the physical and social worlds (Burrell, Gay et al. 2002).
Artifacts and spaces become meaningful for individuals largely through the way they are
used by other social actors; collective social action, both real-time and accreted,
configures the ways in which we interact with the physical. So, two critical issues arise
for interaction with ubiquitous computing – first, how can people understand and make
sense of complex, embodied computational systems, and second, how do these
understandings arise out of the collective actions of many participants?
79
We have been investigating these questions through an interactive sound installation
called SignalPlay, in which participants use physical interface objects to explore a
complex auditory environment. SignalPlay takes sensor data collected in real time and
uses it to generate and manipulate sounds that are fed back to the participants. The
configuration and movement of different objects control aspects of the soundscape; as
participants explore the space, they begin to associate their own actions with the response
of the system and advance their process of discovery.
Critically, this is a collaborative endeavor; multiple participants share the same space and
interact with the system simultaneously. A user’s interaction not only affects sounds they
can readily associate with their own actions, but it also induces global systemic changes,
thereby affecting the sounds generated in response to other participants’ actions. This
creates a collaborative mesh of interaction where the user is not only engaging in physical
action coupled to auditory response, but also where visual cues of co-participants and
their behaviors are woven into the fabric of cause-effect associations. SignalPlay is an
initial foray into the phenomenology of augmented space.
In this system, a series of physical objects with embedded computational properties
collectively control a dynamic “sound-scape” which responds to the orientation,
configuration, and movement of the component objects (Kabisch, Williams et al. 2005).
The system and its component objects – chess pieces, building blocks, bongo drums, an
antique compass, and a toy light saber – are large enough that they cannot all be used by
a single person at once; spread through a space, they create a sonic environment which is
80
experienced and transformed collectively by multiple people. SignalPlay has been
exhibited a number of times, generally in gallery spaces, and we have observed people’s
interactions with and through the system. In this paper, we will explore some of our early
experiences with SignalPlay, and set out an initial framework for describing and
understanding people’s encounters with augmented objects and augmented spaces.
We will begin by discussing some current work that explores similar technological and
design concerns and which examines the collective configuration of space, particularly in
gallery settings. After a brief presentation of the design of SignalPlay, we will discuss our
observations of its use and the framework for interaction that is emerging from our
analysis. Finally, we will discuss some of our further investigations and the potential
implications of this work.
6.2 Related Work
Our investigations were informed by several areas of previous work, including uses of
complex audience spaces as a focus for embodied interaction, use of representational
objects in tangible interaction, studies of the collective experience of exhibits and gallery
spaces, and considerations of how people come to understand a space that they inhabit.
6.2.1 Tangible Interaction with Sound
Art practice has long explored ideas of computational sensory feedback based on physical
interaction. These ideas appear in 1950’s and 1960’s explorations such as a photoelectric
and microphone controlled sound system designed by Billy Klüver for a series of
81
performances held in October 1966 under the title Nine Evenings: Theatre and
Engineering (Dinkla 1996). Installation and performance artists such as Myron Krueger
and David Rokeby have continued to explore the use of sensor technologies with realtime sound generation. Our use of chess pieces as an interface device evokes a 1968
game of chess played by John Cage and Marcel Duchamp in which the movement of
pieces controlled a composition of light and sound.
Work on the use of gestural user interfaces for electronic instruments includes that of the
Hyperinstruments group at MIT Media Lab. The Beatbug system (Weinberg, Aimi et al.
2002) in particular focuses on users’ ability to manipulate musical system behavior at
different levels of collaboration and complexity using simple toy-like objects. In contrast,
SignalPlay uses music as a means of exploring a novel interface; we do not think of it
purely as a musical instrument, but as an experience. It draws on the idea of tangible bits
(Ishii and Ullmer 1997) and phicons (Ullmer and Ishii 2000) for the physical design of
the objects. Unlike the metaDESK phicons, however, SignalPlay’s objects can be thought
of not only as nonrepresentational icons that stand in for a digital interaction possibility,
but also, and more noticeably, they are more literal icons (and in some cases what Ullmer
and Ishii refer to as “actualities”) that represent real-world objects with known
interactional rules.
In this sense, SignalPlay bears some resemblance to ensemble (Anderson 2004), in which
common wardrobe items are augmented to turn the childhood game of dress-up into a
music manipulation activity. As well, the Cardboard Box Garden (Ferris and Bannon
82
2002) uses physically embodied audio spaces to investigate the augmentation of familiar
objects with computational capabilities.
6.2.2 Gallery Studies
Partly because SignalPlay was deployed in a gallery space, it is in many ways related to
the Ghost Ship installation described by Hindmarsh et al (2002). In the Ghost Ship
exhibit, interactive components were distributed throughout a gallery space such that
visitors could interact knowingly with a component in their immediate proximity; but
sometimes unbeknownst to them, they might also influence other components in the
exhibit. Our system is heavily audio-based, while theirs used video images, yet both
installations elicited strikingly similar expressions of confusion, surprise and playfulness.
These detailed studies of interaction and collaboration in a public place, using close video
analysis, informed our methods of observation.
The Ghost Ship study is one of a number of detailed studies of interaction in gallery and
exhibit spaces conducted by researchers at Kings College London (Vom Lehn, Heath et
al. 2001; Heath, Luff et al. 2002). A central feature of these studies is that they turn their
attention away from HCI’s traditional focus on how a single individual might interact
with an exhibit, and focus instead on how a group of gallery-goers might interact around
a particular object or exhibit. The issue here is not simply that most people visit gallery
and exhibit spaces in groups, although this is true (Grinter, Aoki et al. 2002). Rather,
drawing on a range of studies into the role of objects in the collective production of
orderly action, they focus on the ways in which people’s actions essentially “configure”
83
the space for each other. People encounter spaces as ones that are populated with others,
and exhibits as visible sites of other’s activity. Detailed studies of video records show the
ways in which people attend to each other’s interactions with exhibits, which in turn
shape aspects of their own encounters with them. Encounters with exhibits are collective
experiences, and individual actions around them are organized with regard to the
presence, orientation, activities, and gaze of others. The Kings College group has used
these observations in support of design activities (Heath, Luff et al. 2002).
6.2.3 Understanding Space
In an evaluation of the Sotto Voce system (Aoki, Grinter et al. 2002) it is noted that
mutual eavesdropping through the system, and consequent lack of sound attenuation with
distance, could affect couples’ spatial interaction with each other. However, the role of
sound in shaping understandings of space is not extensively addressed in the ubiquitous
computing literature. Anthropology and urban studies have addressed the topic as it
relates to spaces on the scale of cities. Dourish and Bell (Dourish and Bell In press)
discuss space as infrastructure, shaping and shaped by peoples actions in it and beliefs
about it. They present an example of auditory organization of space: children in the
British Commonwealth memorize the sounds of London’s churchbells through a nursery
rhyme, and aural map of the city. Indeed, most European cities of the early modern era
generated informative ambient soundscapes, conveying not only neighborhood, but time,
significant events and power structures, and encouraging or forbidding certain actions
(Garrioch 2003).
84
The aural “landscape” is one of the ways in which the city takes on a shape; similarly,
patterns of movement, religious activity, historical patterns of migration and habitation,
etc, all serve to shape landscapes and make them collectively intelligible (Smail 1999).
Dourish and Bell argue that ubiquitous computing technologies and the infrastructures
upon which they depend similarly offer an infrastructure through which space can be
encountered and understood.
In his discussion of context-aware technologies, Svanaes (2001) notes that space “comes
into being through interaction” and discusses simple technological probes aimed at
highlighting how people come to understand augmented space. It may be informative to
think of SignalPlay as just such a probe.
6.3 System Design and Implementation
We take two lessons from these studies. First, at a broad level, they demonstrate the
complexity of the relationship between technologies and spatial encounters. Interactive
technologies are encountered not simply in their own right, but also as elements in spaces
populated by other technologies and people, and which is a site of social action and social
meaning. Second, that, although gallery spaces are outside the primary traditional
domains of ubiquitous computing application, the exploration and creative engagement
that they encourage can provide us with a site for exploring these questions.
Our goal, then, was to create a system that we could use as an experimental testbed for
understanding how people explore and understand ubicomp technologies as spatially
85
situated phenomena. The primary criteria were that, first, that the system should be
distributed in space; second, that it should allow for simultaneous use by multiple
individuals acting independently or in concert; and third, that it slowly disclose its
operation. Our prototype system, SignalPlay, addresses these goals by using augmented
objects as collective controls for a complex audio space. Deployed in gallery settings, it
allows us to explore the ways in which people individually and collectively explore the
intersection between spatiality and activity.
Max/MSP
Figure 6.1: System Diagram of SignalPlay
6.3.1 Infrastructure
The infrastructure for this project was developed using Crossbow Mica2 motes running
TinyOS, developed at the University of California, Berkeley. The motes are capable of
forming ad-hoc networks through radio communication and at a size of 1-3/4” x 2-1/2”,
could be embedded on or in our objects with reasonable ease.
The motes were fitted
with sensor boards that included accelerometers, magnetometers, thermistors,
microphones, and light sensors. We used primarily data from the accelerometers and
86
light sensors, as the magnetometer data was prone to being affected by the immediate
environment (large speakers, for example) and the microphones had a low sampling rate.
The sensor data from each mote was transmitted at regular intervals (50, 100, or 200
milliseconds depending on what was required to allow fluid interaction with the object)
via radio to a receiver mote, which was attached via serial connection to a PC laptop
running a slightly modified version of XListen, a sensor listener supplied by Crossbow.
The data was read and reformatted and sent as TCP packets to a Macintosh laptop, which
received and parsed the formatted sensor data and generate audio content based upon the
user actions indicated by the sensor data. The behaviors and musical content were
programmed in Max/MSP and Reason. These two programs communicated with each
other using the MIDI protocol and the spatialized audio was output through a multichannel sound interface.
While using multiple laptops connected by TCP was probably not the most efficient way
to design the system, given our time constraints it was most convenient; the Macintosh
did not have a serial port, nor is there a TinyOS install for it, while we did not have
access to the sound manipulation software for the PC. While there was some latency in
the system, we do not believe that the laptop setup was the source of it.
6.3.2 Interface Objects
We designed or selected specific objects based on their capacity to elicit certain
behaviors and on their relation to the theme of “play.” On the one hand, the objects must,
through their physical affordances, suggest how they should be handled; on the other
87
hand, their effect upon a complex audio environment is difficult to convey through form
alone.
The objects were three giant chess pieces (a rook and two pawns), five oversized building
blocks, two bongo drums, a navigational compass in a wooden box, and a Star Wars
lightsaber.
The three chess pieces sat on the ground amid a “chess board” of six disjoint squares,
designed to cue the participant to move the pieces around the space, but with gaps and
shifts in the grid arrangement to indicate that rule-based chess was not required. Each
piece was about two feet high. A mote was placed inside each chess piece such that
moving and setting down the chess piece triggered its behavior.
Five 12” cube building blocks were arranged on and around several small pedestals. Each
had a hole in the top under which a light sensor is placed. The expected behavior of
stacking blocks on top of one another dropped the light reading below a set threshold and
the system responded to that stimulus.
The bongos had holes in the top, at the center of the drumming surface, with light sensors
inside each drum. In striking the center of the drums the user could affect the light
readings and thereby controls a bass line in the system. The system behavior was
sensitive to which drum was struck and how long the light source was obscured. Our
augmentations did not greatly affect the sound of simply drumming on the bongos.
88
The box-mounted compass was hinged in two directions, allowing it to swivel when
tilted. A mote was attached to the outside of the box and readings from the attached
accelerometer and magnetometer were used to control sound. When the compass was at
rest or the compass lid closed it was silent. By opening the lid, the user activated its
sound and controlled various parameters of a waveform synthesizer by moving, tilting
and rotating the compass.
The lightsaber was an off-the-shelf plastic Star Wars lightsaber fitted with a mote
mounted to the handle. It sounded upon sensing motion and was silenced after several
seconds at rest. When swung by a participant, the speed at which it moved dictated the
enacting of sampled sounds.
SignalPlay was deployed first at the opening event for a new research building at the UCI
campus, and subsequently in a gallery space for several days. The installation, both at the
building opening and in the gallery space, was arranged such that the chess set occupied
territory – indicated by the squares placed on the floor – that was roughly central to the
piece. Blocks were placed as a group to one side. The bongos, compass, and lightsaber
were placed on two small pedestals to the other side. During the gallery showings the
room, approximately 15’x18’, was shared with another installation. The two installations
were spatially distinguishable, but not separated; a set of three interactive sculptures was
mounted on the wall while SignalPlay was placed on the floor and other horizontal
surfaces.
89
There are several salient features of the design of the objects and the space that we wish
to highlight here. First, the actions initially elicited by SignalPlay’s objects do produce
discernable effects on the system, but other effects can be gradually revealed through use
over time. Second, the size and design of each object makes it difficult to operate more
than one object at a time, making collaboration necessary to reveal all behaviors of the
system. And third, the spatial distribution of the objects throughout a single room
provided enough space for participants to play individually, but also allowed enough
visual and auditory awareness to coordinate with others.
Figure 6.2: (a) Antique compass with attached mote. (b) A participant poses
with the lightsaber. To the left is one of the chess pieces. Behind and to the right
are the bongos. (c) Another participant stacks blocks.
6.3.3 Sound Controls
Each interface object affects the system in a readily apparent way through discrete sound
events (direct controls) that occur in immediate response to participant interaction. In
addition, most of the objects have effects on a system-wide level (systemic controls),
90
thereby changing the ways in which the sounds of other objects are processed. Through
this second mode of feedback, participants begin to engage in a process of interaction not
just between themselves and the system, but also indirectly (and directly through social
behavior) with other participants. Participants may thus play with the system
individually, affect the response of other people’s instruments, or play in concert.
The systemic control of sound feedback is currently based on control of tonal harmony
(keys, scales and intervals), tempo, and timbre. For all of the objects except the
lightsaber, we base the direct sounds on a globally specified pitch we call the tonal
center; if the tonal center is changed, their sounds are transposed in pitch by the same
interval. These objects, except for the compass, are also governed by a scale of specified
intervals relative to that tonal center. The object sounds base their tonal harmony on a set
of pitches defined by the tonal center and scale intervals. However, object sounds are not
confined only to pitches within that set, but can also deviate by a chosen interval from
specific pitches within the set.
Most of the directly controlled sounds have an associated tempo, be it the rate at which
samples and notes are triggered, the delay and decay times of signal processing modules,
or the enacting of dependent processes. Interaction with certain objects causes systemwide tempo changes that affect these parameters. For instance, a transient “hit” on the
bongos will trigger that instrument’s sound while instead holding your hand continuously
over the light sensor will cause the tempo to speed up or slow down (depending on which
drum you trigger).
91
The object behaviors form a continuum from simple and direct control to complex and
systemic control in the following order: lightsaber, compass, bongos, chess pieces, and
blocks. The lightsaber uses only direct controls with no affect on a system- wide level.
This allows its behavior to be very easily understood. The compass is affected by the
tonal center but not the pitch sets. The rest of the objects have direct controls with an
increasing level of system controls. In addition, there are sounds that are not related to the
physical objects; these are based entirely on systemic changes and have no direct control.
6.4 Exploring and Interpreting Space
We deployed SignalPlay in four showings. The first was the building opening noted
above; the other three were showings at the Arts, Culture and Technology building at
UCI. During the building opening and two of the gallery showings, we video-recorded
people’s interactions with the exhibit and received informal feedback from them during
and after their interactions with SignalPlay. The video was a mix of handheld, manual
recording, allowing close-ups of participant interaction with the system, and stationary
video taken at a vantage point from which the entire installation could be viewed. The
observations presented here are the results of an initial analysis of these video materials.
As is clear from the earlier description, SignalPlay is both inherently collaborative (since
it is physically too large for a single person to explore) and responsive to transformations
in its physical configuration; our goal, then, was to use it as a basis for understanding
aspects of the interactions between people, actions, and artifacts in augmented spaces.
92
One starting point for this analysis is Ullmer and Ishii’s (2000) MCRpd interaction model
for representational tangible interfaces. Based on the model-view-controller approach to
graphical user interface development, MCRpd presents a framework for tangible
interaction in which the “view” component is distributed between the digital and the
physical. A physical controller cum physical representation affects a digital model, which
may output a digital representation. They point to audio from a speaker as an example of
digital representation, and chess pieces and chess boards as examples of physical
representations.
We distinguish between two aspects of people’s experience in forming an understanding
of SignalPlay. The first is learning to control the system through the objects; the second
is learning to “read” or interpret the sound output of the whole system as being a result of
purposive human action. These two attributes are analytically distinguishable, as
suggested by the MCRpd model, but not separable in practice. We interpret participants’
perceptions of SignalPlay to be inextricably bound with their actions within it (Robertson
2002). Control and interpretation are tied to participants’ interactions with each other and
with the space they inhabit. As we have illustrated above, people encounter ubiquitous
computing technologies in socially-organized settings. Even when they are alone, they
act nonetheless in spaces that have social and cultural meanings and interpretations.
These factors – not just how people encountered the system along with others, but also
how they encountered it in terms of sedimented understandings and metaphors – were
significant aspects of our observations.
93
In what follows, we will discuss some of the experiences of SignalPlay drawn from the
video materials. We organize these into three related topics. First, we consider individual
interactions with the devices, and how both the material and metaphorical aspects of the
artifacts shapes interaction. Second, we move from an individual to a collective level,
discussing how people used aspects of the system to play not simply with the technology
but also with each other. Finally, we approach the question of “reading the space” and
discuss the ways in which learned how to interpret the actions of the system as the
outcome of the embodied practices of actors.
6.4.1 Modes of Object Interaction
Our first consideration is the ways in which individuals encountered the system, and how
the properties of the artifacts out of which it was constructed – both material properties
and metaphorical properties – shaped and constrained their interactions. Objects were
designed to evoke certain behaviors by resembling everyday artifacts; however we also
wanted to invite exploration by making it evident that these objects were augmented.
Physical cues indicated that the objects were not exactly what they represented: the chess
set was incomplete, the chess board strewn across the floor in a not-quite-grid, and the
motes’ antennae poked out of the blocks. As participants learned to exploit the digital
augmentation of SignalPlay’s toys, their engagements with the objects varied, reflecting
different forms of engagement both with the objects themselves and with the effects that
they controlled.
94
Certain objects exhibit little tension between symbolic physical cues and augmented
behavior; the lightsaber, for example, is enhanced merely to make lightsaber noises when
swung like a sword. This object had enormous initial appeal, and was quickly understood.
Other objects support richer modes of interaction. Over the course of five minutes, we
observed a man learning to control aspects of the system’s sound generation through the
compass. Initially, he held the compass stationary in front of his torso and walked around,
changing direction periodically. As it became evident that direction influenced the
compass’s associated sound, he stood still and rotated his torso, holding the compass
rigid. This behavior turned to rotating the compass with his hands while holding his torso
stationary. He then tilted it, resulting in a dramatic pitch change. With more
experimentation, he combined direction with tilt, as well as opening and closing the lid to
abruptly start and stop the sound. His interaction with the object departed significantly
from ordinary compass use as he learned to understand the augmented compass as a new
object in its own right.
We observed three major categories of use: iconic, intrinsic, and instrumental. Iconic
interaction entails interacting with a physical icon in the ways afforded by the object it
represents. Examples of iconic interaction with objects in SignalPlay include moving
chess pieces from one square to another, stacking the blocks, beating on the bongos, or
holding the compass in front of oneself while walking around the room. For example, a
few participants, while playing with the chess pieces, limited themselves to legal moves,
never moving the rook diagonally or the pawns more than one square over. Iconic use,
then, is shaped primarily by the metaphors suggested by the physical objects themselves;
95
they are appropriated as augmented versions of their traditional analogs.
Intrinsic interaction takes advantage of the intrinsic physical characteristics of an object.
For example, because our chess pieces were hollow, a pair of participants (playing
together) proceeded to stack them on top of one another. This mode of play had nothing
to do with the object’s status as a physical icon of a chess piece, but rather responded to
the physical configurations of the objects themselves. Turkle and Papert (Turkle and
Papert 1992) report a wonderful illustration of intrinsic interaction in their dicussion of
bricolage among elementary school students learning engineering concepts. Given an
assignment to propel a small robot forward using a motor, many of the children used the
motors to drive wheels; one boy, however, used a motor to drive a robot around directly
by the force of its vibration. He did not think of the tool as an instance of the category
motor, but rather as a thing that vibrates in such a way that might move a small robot
around. Similarly, the idea to tilt our compass does not come from its “compassness”, but
rather from the fact that it happens to swivel in an interesting way when tilted.
Figure 6.3: Stackable Chess Pieces
In comparison to the two earlier modes of interaction, instrumental interaction is not
96
focused on the physical objects themselves, but on the effects that they engender; people
engaged in instrumental interaction reach “through” the objects, focused on using them as
controllers of a digital system. In the case of SignalPlay, users took advantage of the
ways in which the musical sounds were influenced by manipulation of the object, treating
it similarly to a musical instrument. For example, we observed a participant “playing the
compass” by a combination of tilting, swiveling his wrist, and closing and opening the
lid. A pair of women played with the blocks by a combination of stacking and covering
light holes with their hands or other objects. Instrumental interaction may exploit the
intrinsic physical features of the augmented object, (as in covering light holes) or it may
be externally the same as the iconic interaction (as in stacking the blocks), or it may
constitute a combination of the two; the critical aspect of instrumental interaction is the
user’s understanding of the object and system.
Our observations of participants’ play revealed in each object a different interrelation
between these three modes of interaction. For example, the lightsaber had been
augmented simply to make the sounds that might be associated with it through the Star
Wars films; it did not affect any other sounds in the system. In this case, instrumental
interaction did not differ significantly from iconic interaction; it acted just as a lightsaber
is “supposed” (or might be expected) to act. In contrast, iconic interaction with the
compass, triggered only a subset of the possible sounds. The intrinsic interaction of tilting
the box allowed participants greater control over the pitch of the compass’s sound.
Participants generally understood the lightsaber right away, and we observed numerous
instances where a participant might pick it up, play for just a few seconds, and quickly
97
put it down or try to hand it off to another person. In the case of the augmented compass,
we found many instances of extended interaction over several minutes, frustration,
exploration, discovery and failure.
Figure 6.4: “Playing the compass” by tilting, closing and opening.
Initial Conditions and Sequential Experience. Participants’ interaction with SignalPlay
proceeded in an approximate sequence. A tentative poke may lead to engaged iconic
interaction. Further exploration may involve intrinsic interaction, then confident use of
the object as instrument. Instrumental interaction may then lead a participant to exploit
more of the object’s intrinsic characteristics. This sequence describes only a general
trend. Participants’ behavior could be influenced by their initial experience, which helped
determine which exploratory actions they tried. A man who tried raising and lowering the
compass had some success affecting pitch change in that manner. When he subsequently
played with the bongos, failing to make them trigger a sound by drumming them, he then
tried to raise and lower them as he had with the compass. This action is not particularly
afforded by the bongos, either physically or instrumentally.
98
We logged numerous instances of participants playing while a friend watched, sometimes
right at their shoulder, pointing and suggesting actions. As the crowd grew, we logged an
increasing number of participants watching and being watched by strangers who simply
stood back and did not interact with the person at play. Mutual watching informed
participants’ understanding of how to control the system through objects; as watching
increased, participants tended to become less tentative and more engaged. Some
participants, after watching for some time, skipped iconic interaction altogether, imitating
a more experienced participant’s instrumental interaction.
Space and Modes of Interaction. In the case of the compass, when people thought of it
iconically, they tended to cover more space, walking about the room holding the
compass. When they started thinking of it more instrumentally, they were more likely to
play it standing stationary and changing only direction and tilt. We saw this
transformation take place in the case of one man who was bent on understanding the
compass; though he roamed the room at first, after five minutes playing with it he was
controlling the sound confidently and with his feet planted in one spot. Playing with the
blocks or the chess pieces as a collocated set reinforced the iconic nature of the objects.
This became evident during one of the gallery showings when two participants moved the
blocks and bongos onto the chess board into the middle of the room, disrupting the
objects’ clearly demarcated territories. Their treatment of the objects changed drastically
as a result of this move. One covered the light hole on the bongo with one hand, swinging
the lightsaber with the other and using it to cover a light hole on one of the blocks.
Meanwhile her friend, as she bent to set it down a chess piece with one arm, covered the
99
second light hole on the bongos with her other hand. Other participants followed their
lead and adhered far less to iconic interaction than previously. That this disruption of the
exhibit’s spatial setup had such a noticeable effect on participants’ object interactions
indicates that their understanding of the system is affected not a little by how they think
of it within the space of the gallery.
We set out to examine how people can understand physical interfaces to complex
systems, in light of Smith’s tension between “literal” and “magical” for graphical user
interfaces. The critical difference between our physical objects and Smith’s graphical
objects is that physical objects cannot be separated from their literal component. If
graphical interface objects could be charted as points on a continuum between literal and
magical, augmented physical objects would have to be represented as lines anchored in
their inherent literal form and extending into the magical, leading the eye from one end of
the spectrum to the other. Our observations focus on the transition from symbolic to
instrumental use of physical objects; a transition that is not inherent in the objects but an
emergent property of embodied encounters with them.
An artifact, through its physical form and socially generated meaning, elicits specific
actions from people. In a system that employs symbolic interface objects, these implied
actions must generate a perceptible response in order to engage people. Our bongo
drums, due to issues with system latency, were often unresponsive to participants’ initial
taps, and failed to engage them. When sufficiently engaged however, participants were
inclined to explore deeper interactional possibilities.
100
In instances where gradual or
serendipitous departures from the initial interaction generated noticeable system
response, our participants readily reformulated their understanding of objects and the
modes in which they engage with them.
6.4.2 Collective Encounters and Interpretation
People tended to encounter SignalPlay in groups. One interesting set of issues, then,
concern the ways in which it mediated collective experiences. People respond both the
technology and to the setting within which it is encountered – in our cases, a
technological demonstration or a gallery space. These settings lend meaning to the
technology, as something to be explored and understood, but not necessarily to be used as
a tool. These contexts shape and limit forms of engagement; the socially understood
settings both “script” people’s encounters with the technology (time-limited, to be shared
with others, not to be taken away, etc) as well as making the space and the technology
“legible” (in terms of, for example, how the various elements of our system could be seen
as part of a single “piece” but distinguishable from others nearby.)
Playing With Others. Like Hindmarsh et al’s Ghost Ship, the interactional capabilities of
SignalPlay manifested themselves fully when the gallery space was crowded. Crowds
lent themselves to group play and observation of participants by other participants, both
of which encouraged instrumental interaction. The chess set is a case in point. Due to the
size and dispersal of the chess pieces, one person could not move them rapidly enough to
make the tonal change obvious.
101
At one showing, once the workings of the system were explained to the participants, two
pairs of women gravitated towards the chess set, which had previously generated interest
only in a couple individuals. These two groups remained engaged for longer than the
previous solo players and, in attending to the objects’ capacity as sound controllers,
departed more from the iconic cues of the chess pieces; illegal moves were made more
readily and conventions of turn-taking were discarded. One pair was quite aware of their
departure from iconic interaction, commenting that “no one can win this game!” and
cracking jokes about how they should have a chess timer.
In later gallery showings that lasted longer and drew larger crowds, participants would
roll a chess piece around the edge of its base, or hold it up and swing it, triggering chord
changes in quicker succession than they would have if making chess moves. Indeed, it
was during games of “speed chess”, and other interactions that triggered rapid change,
that the effect of the rook on the tonal center of the system became evident. Those of us
who do not have perfect pitch depend on our imperfect memory in order to hear intervals.
A single person engaging in iconic interaction with the objects in SignalPlay, then,
typically does not reveal the systemic sound effects of some of the objects because of this
temporal aspect of the system. On Hindmarsh’s Ghost Ship, space was the key element in
understanding the exhibit, since video images taken in one part of the room were
displayed to other people in another part of the room. This was true for SignalPlay, since
moving the rook in one part of the room would affect the tonal center for other objects
scattered about the space, however time was also a critical factor. In SignalPlay, systemic
sound controls were most evident when several users interacted at the same time,
102
triggering objects in quick succession.
Peripheral awareness and mutual monitoring. Unsurprisingly, participants’ attention
might be drawn to one another due to loud talking or sudden motions. Copresence and
peripheral awareness of companions’ locations proved to be a crucial component in
visitors’ understanding of Ghost Ship (Hindmarsh, Heath et al. 2002). In SignalPlay as
well, awareness of people in space was a necessary step towards an understanding of
system sound in space. However, participants’ awareness of each other in the Ghost Ship
installation was based on vision more exclusively than in SignalPlay, where awareness of
others’ actions did not necessarily depend on the direction of ones’ gaze. Participants
frequently monitored each other through the system. For instance, a girl playing with the
blocks demonstrated awareness of her friend playing with the compass, turning towards
the camera, widening her eyes and smiling when the compass sound suddenly changes in
quality. Additionally, participants are aware of each others’ awareness, and explorations
took on a certain aspect of performance. Two girls playing with the blocks dance to the
music, and people playing with the lightsaber adopt dramatic poses.
This mutual monitoring through audio was not deliberately designed into the system, but
rather the result of simple, but public interaction. Grinter et al (Grinter, Aoki et al. 2002)
noted a similar phenomenon in the Sotto Voce system: the system was meant to allow
pairs of museum visitors to share audio content regarding the exhibits, but it was used in
addition to monitor the location of companions. In this case the information shared is not
so explicit, but it is shared more widely, to strangers and friends alike.
103
6.4.3 Reading the Space
Finally, here, the experiences with SignalPlay also highlight our concern with the ways in
which actions in space become readable and interpretable to others. We encounter spaces
as particular kinds of places (Harrison and Dourish 1996); as public or private, as spaces
of work or leisure, as rowdy or dignified, etc. In our deployments, we were particularly
interested in the “legibility” of space and technology – that is, in how people could learn
to read it or interpret it, and in particular how they could read the system’s activity as
being a consequence of their own and others’ actions.
A direct physical mapping between the gallery space and SignalPlay’s audio output
would identify the sounds as coming from the speakers, located in certain corners of the
room. On only one occasion, however, did a participant actually indicate the speakers as
the source of the sound, an 8-year-old boy who wanted to know how we got the sounds
from “there” (the bongos) to “there” (pointing at speakers). Though he knew
intellectually where the source of the sounds were physically located, interactionally he
mapped the sounds to the space quite differently. Seconds after he pointed out the
speakers, the rook was moved, triggering the associated sound. Looking up from the
bongos, he pointed towards the chess set. In this section, we examine how our
participants might come to understand SignalPlay’s audio output in space as something
more than a simple physical correspondence. Participants’ interpretation of the
SignalPlay space was built upon their awareness of people in space, as previously
discussed, as well as a strong association between sounds and objects, objects and
104
territory, and awareness of each other’s sound-producing actions.
Transferring Focus to Objects. We saw numerous instances of participants examining
motes that were attached externally to the lightsaber and compass. However, we also saw
a man peer inside the compass box, despite the visible mote. We also noted a woman who
put the compass up to her ear, as if expecting the sound to emanate directly from it. These
were the most noticeable illustrations of the general tendency to focus on the physical
objects as the source of the sounds and regard the digital system as transparent.
Universally, when a participant’s attention was attracted by a sound associated with a
certain object, they turned not towards the physical source of the sound – the speakers –
but to the causal source of the sound, the object.
Physical Objects Demarcate Space. At one point during one of the gallery showings, a
participant separated one of the blocks from the set, placed it on the floor next to one of
the sculptures from the other installation sharing the room, and ran an Ethernet cable
from that sculpture into the hole on top of the block that allowed light to reach the light
sensor inside. This breached the grouping of the blocks, expressed by keeping them all in
the same territory, not to mention the spatial distinction between the two installations.
The displacement of that block proved to be an exception that proved the rule; it drew
looks, comments, and jokes from other participants.
Different objects elicited different spatial behaviors. The lightsaber, compass and bongos
tended to “wander” but return home. A participant might roam around the room with the
105
lightsaber, poking their friends and swinging it around. Participants commonly walked
around with the compass, and in fact that movement can be considered an example of
iconic interaction encouraged by the compass. However, participants almost always put
them back exactly where they found them. The chess pieces on the other hand, were
placed on a chess board, a clearly demarcated piece of territory. Though they were moved
around, they were rarely moved off of the chess board. The blocks, for the most part,
stayed on the pedestals on which they were originally placed. Territory was not marked
for the blocks, any more than it was for the compass, which traveled more. The key
difference was that while the iconic interaction with the compass required movement
through space, the blocks encouraged stacking in place. Thus the interactional properties
of the objects affected how participants fit them into the space of the exhibit.
Sounds and Sound-Producing Action. Sounds in SignalPlay are caused by visible action,
allowing watchers to associate a sound with an object and the person controlling it, and
thereby making the system’s audio output interpretable. During a gallery session, three
women off camera are discussing “the bong” and in order to clarify its source to them
another participant simply picks up the rook and moves it. This demonstration makes
explicit a usually implicit process of monitoring other participants’ actions and
associating them with system sounds.
These three aspects of interaction – with the artifacts themselves, with others, and as a
means of reading space – are not separate behaviors; they arise in concert with each
other. Here they provide us with a starting point for understanding the relationships
106
between people and activities in augmented spaces, and how ubicomp technologies
transform the legibility of actions in space. Although gallery settings differ from office,
domestic, or mobile settings in which ubicomp technologies may be deployed, those
settings are also populated by people and by technologies, and ones that must be
interpreted and transformed through practical engagement. Our data illustrate that the
collective, spatial, and sequential aspects of encounters with ubicomp environments are
critical factors in how those technologies will be put to collective use.
6.5 Conclusions and Implications
SignalPlay demonstrates that our world is both physical and social. While we might
distinguish between these as analytic concerns, they are fundamentally intertwined as
practical matters. Just as it is impossible for us to encounter space independently of its
physical characteristics, it is equally impossible for us to encounter it independently of its
social character and organization. This social character means that spaces are not
“given”; they are the products of active processes of interpretation. The meaningfulness
of space is a consequence of our encounters with it.
For ubiquitous computing, this is an important consideration. We are engaged in the
development of technologies that are rapidly moving out of traditional computational
settings – laboratories and workplaces – and into everyday environments. Ubiquitous
computing research is actively concerned with domestic environments, with technology
in leisure settings, with mobile technologies, and with a range of computational
embeddings in space. The research challenge, then, is to understand how it is that
107
computationally augmented spaces will be legible; with how people will be able to
understand them and act within them. Taking this perspective highlights some aspects
that are traditionally hidden in the ways in which we think about ubiquitous computing
and interaction. Our traditional focus, drawn from decades of research on HCI, is on how
people might interact with technologies. However, as we can see from observations with
SignalPlay, this is a narrow perspective. Instead, we have been looking at how people
engage with space and with each other through the technologies that we provided to
them. We focus on how people collectively act in space, and through that participation,
achieve concerted social action.
Our design models must address space not as a passive container of objects and actions,
but as something that is explicitly constructed, managed, and negotiated in the course of
interaction; and at the same time, we need to be conscious of ways in which new
technologies provide new ways of encountering space.
For tangible interaction, SignalPlay demonstrates some of the ways that a relatively
simple system can be used as a collective resource for rich interaction. It foregrounds the
fact that tangible interfaces are not just physical objects coupled with data, they are also
social and spatial. The very tangibility of SignalPlay’s various components rendered
interaction with the system publicly available, and participants could use multiple,
mutually reinforcing sensory channels to monitor each other’s actions. The physical
forms and spatial arrangements of objects, while they did not constitute an exact
representation of the state of the digital system, did intuitively indicate what actions were
108
available for participants to perform within the system. If technology renders space
legible in new and interesting ways, then situating computational systems in a physical
setting makes them observable and interpretable as well. We are interested in how
computational technology can reconstitute space: tangible systems put those processes
out in the open where we can examine them.
SignalPlay is an initial examination of people’s interactions with and through
computationally augmented objects and spaces. Our focus is less on technical innovation
and more on uncovering behaviors and understandings that will inform future work. This
includes augmenting a new interdisciplinary research building with a sensor network
infrastructure that will support ambient displays of presence and activity, and
enhancements to SignalPlay itself, incorporating network topology and radio signal
strength in order to tie the system more closely to physical space. More broadly, our
research in this area further develops the ‘embodied interaction’ paradigm, which
concerns itself with how technologies and artifacts take on meaning for their users
through their embedding into systems of practice (Dourish 2001). This relationship
between people, objects, and activities, cast in terms of the ways in which practice
evolves, is a central consideration for future developments in ubiquitous computing.
Our SignalPlay deployments scarcely scratch the surface of this topic. They were limited
in both scope and duration, and so provide only a brief snapshot of the ways in which
people engage with augmented spaces. In the next chapter I will present Nimio, another
site-specific collaborative tangible system, this time deployed in an office setting for
109
several months.
110
Chapter 7
Nimio: Combining Tangible Interfaces and Ambient Displays for
Collaborative Groups
While tangible interfaces open up new possibilities for input and interaction, they are also
interesting because of the ways in which they occupy the physical world just as we do.
Nimio sits at the intersection of three research areas – tangible interfaces, ambient
displays, and collaboration awareness. Our system, Nimio, uses engaging physical
objects as both input devices (capturing aspects of individual activity) and output devices
(expressing aspects of group activity). It is situated in a rich ecology of multiple
communication media and opportunities for mutual monitoring in everyday lived space.
We present our design and experiences, focusing in particular on the tension between
legibility and ambiguity and its relevance in collaborative settings.
This chapter is based on material previously published with Johanna Brewer and Paul
Dourish in the proceedings of Tangible and Embedded Interaction 2007 (Brewer,
Williams et al. 2007).
7.1 Introduction
Ubiquitous computing applications reimagine the everyday world as a site for interaction.
Where traditional interaction has been bound to personal computers and desktop
environments, the move of computation into the world means that the physical
environment itself becomes an interface to a diffuse coalition of computational devices
and services. The everyday world is, however, populated by other people, objects, and
111
activities. This suggests that one important area for ubiquitous computing system
development lies at the intersection of ubiquitous and collaborative systems. Significant
research questions in this area remain to be answered. Some of these include, how can
ubiquitous technology be used to support group cohesion and interaction? How can
people understand the operation of augmented spaces? How can collective behavior
emerge in interaction mediated by ubiquitous computing technology?
Motivated by these questions, we have been experimenting with simple devices that can
be used to maintain informal contact and interaction for distributed groups. Nimio is a
system comprising a series of physical objects designed as individual playthings, but
wirelessly networked to act as both input and output devices for a collective visualization
of distributed activity (see Fig. 7.1). These hand-held, translucent silicone toys have
embedded sensors (for input) and LEDs (for output) that allow them to be reactive to
both sound and touch. Action around one Nimio will cause the others to glow in different
patterns and colors. The interaction design is deliberately open-ended, in order to allow
the emergence of distinctive patterns of collaborative engagement in real groups. This
tension between legibility and ambiguity is a central aspect of our design.
7.2 Background
The research presented here, and the devices that have resulted from it, draw upon and
combine previous research into three primary areas – passive awareness, ambient
displays, and tangible interfaces.
112
7.2.1 Passive Awareness
Although early CSCW research focused on the formal, task-oriented aspects of
collaboration (Streitz, Geißler et al. 1994; Glance, Pagani et al. 1996), it rapidly became
clear that computer systems could also usefully support the informal aspects of collective
interaction. For instance, IM is not a collaborative technology in itself, but supports
collaboration as part of a broader ecology of tools. IM’s messaging component supports
informal interaction, quick questions, and social chit-chat; at the same time, the presence
indicators visualize individual and collective presence (Nardi, Whittaker et al. 2000). The
range of mechanisms by which this is achieved have generally been glossed as
“awareness.”
Embodied action in space is critical to studying how groups maintain an informal
connection and understanding of mutual activity. Heath and Luff’s (1992) classic study
of London Underground controllers shows how they coordinate their actions so as both to
display and to monitor the activities in which they are engaged, through the ways in
which they share a physical space within which these actions are performed. Audio-video
environments have attempt to reproduce aspects of this experience for distributed groups,
but everyday interaction takes place in a three-dimensional space, not on a twodimensional plane, and so communicative gestures, for example, lose their interactional
effectiveness on a screen (Heath and Luff 1991).
113
7.2.2 Ambient Displays
One approach that attempts to deal with some of these problems is to move the source of
interaction back out into the world. A number of researchers have noted the ways in
which people interpret the rich cues that they find in the everyday world as a means to
understand activities around them, and have begun to explore the ways in which the
everyday world can be used as a medium for display. There are two key elements to this
work – a focus on passive understandings, and a focus on ambient information. These are
related but different concepts. By stressing passive understandings, some research draws
attention to the ways in which information may be provided to users without explicit
effort on their part, perhaps serendipitously encountered in the course of interaction. This
may encompass passive awareness displays based on “push” models (Dourish and Bly
1992; Fitzpatrick, Parsowith et al. 1998), or it may suggest approaches that annotate
information objects with indicators of activities that others have performed, in much the
same way as physical objects carry markers of previous activity (Hill, Hollan et al. 1992;
Höök, Gersie et al. 2003). Relatedly, a focus on ambient information considers the ways
in which information can be conveyed in the environment, through the use of peripheral
cues such as background sounds, light levels, etc. The primary consideration here is the
way in which information display features as an aspect of the environment. Ambient
displays, however, have largely operated as just that – displays, concerned primarily with
output. While some have augmented these displays with devices such as video cameras or
RFID (Sawhney, Wheeler et al. 2001), these have largely been efforts in local
information customization. In contrast, our focus on collaboration requires that we be
concerned also with the displays as input sites.
114
7.2.3 Tangible Interfaces
One source of inspiration is Brave et al’s (1998) “InTouch,” which uses wooden rollers
connected to both sensors and actuators to create a shared physical experience across
distance, and hence to provide a tangible channel for communication between two
parties. Its design is simple and compelling, but, while it supports potentially
sophisticated interaction between two people, it has no support for broader engagements
amongst distributed groups.
Research on tangible interfaces, particularly in collaborative settings of this sort, suggests
that they might provide an effective mechanism for combining ambient displays with
social connection through activity awareness. The system we have developed, Nimio, is
designed for this purpose. Our previous research has suggested that the local
environmental and organizational context is a critical issue for ambient display design.
We will begin by outlining aspects of this approach, and then discuss our field studies of
the group and the space into which Nimio is to be introduced.
7.3 Context-Specific Ambient Displays
Brewer (2004) outlined several considerations for the design of ambient displays,
primarily stressing the importance of context. Naturally-occurring sources of ambient
information are in a sense ideally suited for their situations. Both the sound of rain and
shadows from the sun are inherently wed to their location; hearing rainfall means that it is
raining right here. They are integrated into their surroundings, or rather, they constitute
115
the surroundings. These displays are part and parcel of the information that they convey.
However, that information also can be interpreted to have more complex connotations.
From seeing many people on the street in the business district of a city, for example, one
may infer that it is lunch time or quitting time. This is a crucial feature of these displays:
their ability to support inference of more complicated or nuanced situations. Our
framework highlights the situated nature of information and inference.
 Is the Information Dependent Upon the Context?
It also emphasizes that the display
 Is the Information Specific to a Certain Group?
must address a need, however subtle
 Is the Display Meant to Be Interactive?
or currently unacknowledged, while
 How Does the Ambient Information Relate to Other
Information and Information Practices?
 Is the Primary Purpose of the Display to be
Aesthetic or Informative?
remaining flexible enough to be
adopted into the user’s lifestyle as
 How Rapidly Does the Information Change?
they see fit. This is achieved not only
 Is the Information Already Displayed in Some Way
or is it Intangible?
 Does Past Information Persist in the Present?
by choosing carefully the information
to display, but also by understanding
 Is the Physical Location Cohesive or Fragmented,
Mobile or Static?
Figure 7.1: Ambient Display Design Considerations
how that information figures into the
users’ daily practices. To this end we
have developed several design considerations (Figure 7.1), in the form of elucidatory
questions, which provide a rationale for choosing certain methods of display based upon
the various features of the information.
116
7.4 Site Study
Our principles suggest that any design effort must begin with a detailed examination of
current practice. We worked with a group of ten people who manage a multi-disciplinary
information technology research institute. They reside in two spacious suites across the
hall from each other. Another member of the group has an office on another floor; as
facilities manager, his job requires him to roam about the building. This group was the
first to occupy the new building, having recently moved from a single, cramped hallway
in another building. In their new area, the larger of their two suites contains six offices, a
conference room, a reception area, and a multi-purpose copier/coffee/mail room; the
smaller annex has four offices and a reception desk. Impromptu social gatherings tend to
happen in the multi-purpose room.
Although the group is influenced by the space they occupy, the social topology of the
group is continually changing, as they often work in smaller subgroups depending on the
current project schedule. Their office layout is not necessarily optimal for collaboration,
so they are highly mobile, traveling both within and outside of the office suites. They
estimated their trips across the hall to see colleagues in the other suite in dozens. Thus,
the context for the display is primarily influenced by both the group’s social and spatial
configurations.
We observed patterns of the users interacting with and through the environment, and
conducted semi-structured ethnographic interviews with most of the group members.
117
Interview questions focused on daily work routines, collaboration with colleagues, and
use of physical artifacts for both work and decorative purposes.
Collaborating around Artifacts. Several people described day-to-day work that was
outside of their job descriptions, things they did to “help out” until more staff were hired.
One effect of this flexibility of duties was that people did not work with fixed groups.
Certainly, some sets of people worked closely and consistently together, but none of
these sub-groups worked in isolation. In describing their tasks, some interviewees pulled
out calendars, paper documentation, spreadsheets, and floor plans. In one notable
instance, a spreadsheet of tasks served not only as a to-do list, but a reminder that she was
planning a great deal of collaboration with a particular colleague over the coming weeks.
This document that had his name all over it served as a low-tech long-term indicator of
his presence.
Collaboration and Interruption. Perhaps more striking than the content of the interviews
was the fact that every interview we conducted was interrupted at least once by a
colleague checking the availability of the interviewee. Two interviewees reported calling
first to see if the other is available, though when this happened during the interview, she
let the phone ring and the colleague checked on her in person anyway. Incidents such as
Figure 7.2: Jasmine Blossom Distributed Display
118
these, as well as comments made during interviews, led us to believe that one advantage
of their previous cramped quarters was that they allowed peripheral awareness of each
other’s presence. From one group member:
When we were in [our old offices] we were in close proximity, so it was very easy
to know where people were or if they were on the phone or if they were talking
with somebody else or if they were out of the office... But in here, now that we’re
separated into two suites, it’s difficult because there’ll be times where I don’t
want to call someone on the phone I want to talk to them in person, so I’ll walk
over there, and I’ll have another excuse to go over there for coffee or whatever,
and I’ll find that they’re either on the phone with somebody or they’re not there…
So that’s… you end up making five trips for the one trip.
Furthermore, we were intrigued by the way in which the group chooses to portray itself.
They consciously present themselves as a tight-knit and heavily collaborative group.
They go out to lunch together, don matching shirts emblazoned with the institute logo for
building functions, and tell us in interviews how close they are.
Artifacts and Ornaments. This closeness is also reflected, on occasion, on the physical
objects in the environment, which were a particular focus of our attention. At our first
visit on site, most group members had similar jasmine blossoms in each of their offices
(see Fig. 7.2). We were told later that one member had brought in several clippings from
the same bush to share with her colleagues.
Though theirs are not technical positions, the members of the group present their
organization’s innovative technology to visitors on a daily basis. This being the case, it is
in their interest to project a certain technophilic image to visitors, which is currently done
in part by the office’s décor. In the waiting area stands a large plasma disc that responds
119
to voice and touch with moving bolts of colored light. In interviews, several group
members expressed an interest in objects or displays that could serve as “talking points”
for newcomers, though at the same time they did not wish for distractions from their dayto-day work.
7.5 A Context-Specific Design
The goal of our observational work was to understand the opportunities and parameters
for informal collaboration support in this setting. Clearly, the move from a common
space to a larger distributed space has introduced problems for the group. Some of these
are purely coordinative, such as knowing when people are around and knowing when
they might be available. However, we were struck too by the sense of closeness within
the group; an important goal then is to support not just coordination but cohesion. So this
less formal element was a critical design issue. It was notable that this cohesion was
expressed not least through physical objects. Finally, we noted the importance of
providing them with the means to demonstrate to visitors that their organization fosters
innovative and aesthetic technology. This trio of coordination, cohesion and comportment
became the focus for our design process.
Our studies suggested that the information which would be most beneficial and suitable
for display would be the activities of the other group members. Activity, however, is not
a trivial thing to capture and express. Some other displays attempt to monitor the amount
of work the members of the group are performing and then display the activity as an
availability level based upon the work being done. From our observations though, it
120
seems that the users are more attuned to the inverse: they were used to having an
awareness not of each others work but of all of each other’s attendant activities. They
inferred what one another were doing from observing the peripheral cues of each other’s
actions within the workplace. Understanding how busy or free another person is not a
straightforward operation; one must learn their co-workers patterns of behavior and the
peripheral signs which result from those behaviors. Thus, the peripheral information
which they previously relied on was not a clear representation of work level, in fact it
could vary from person to person, but rather it was a channel through which the group
became accustomed to expressing themselves and learning about each other. This
channel, then, does not reflect activity level as a scale of interruptibility, and so work
done on recovering interruptibility (Fogarty, Hudson et al. 2004) is not sufficient for this
application as the group is accustomed to communicating and understanding richer
information about the nature of each other’s activities.
We then set out to examine that information with our framework and to try and develop a
method to gather and display it. In this section we will give a detailed account of the way
in which Nimio works and highlight, when applicable, the questions of the framework
that influenced our design decisions.
7.5.1 How Nimio Works
The system is a set of 12 touchable translucent white silicone toys that can detect when
there is sound around them and when they are being moved. They transmit this
information to each other wirelessly, and display it via red, green and blue lights.
121
A single Nimio is defined by its shape and its color. There are four shapes (pyramid,
cube, dome and cylinder) and three colors of bases and lights (red, green and blue). Each
Nimio can display all three colors, but can only trigger distributed display of the color on
its base. Each toy is a unique combination of a shape and color (e.g., there is only one
“red cube” or “blue dome”). These two properties create two types of family groups:
shape groups (e.g. dome Nimios) and color groups (e.g, blue Nimios). The family groups
and the type of interaction detected govern they way in which information is displayed on
the Nimios.
Nimio can detect and display sound and two different levels of movement. If any Nimio
detects sound around it, all other Nimios pulse their matching light in rhythm with the
sound. So, for example, if a cube-shaped Nimio with a red base senses sound all other
Nimios will display red.
Figure 7.3: Nimio!
Next, a Nimio detects when it is being moved gently. This activity will only be displayed
by Nimios of the same color or shape as the sensing Nimio, and it will be displayed as a
122
steady pulsing at a frequency of one pulse per second. So, if the red cube is moved
gently, all cubes and red-based Nimios will slowly pulse red, but, for instance, the greenbased pyramid will not pulse.
Finally, a Nimio detects when it is shaken vigorously. This activity will also only be
displayed by Nimios that share its shape or color, as a solid light for 5 seconds. For
example, if the red cube is shaken, all cubes and red-based Nimios will light up solid red,
but, for instance, the blue-based dome will not be affected.
7.5.2 Example Interaction
Figure 7.3 depicts an example scenario of how Nimio might be used, showing all of the
Nimios in the system and their reactions. Prairie-Dawn has two Nimios in her hands, the
green and red cubes. In a moment of whimsy, she gently wiggles both Nimios like
maracas. The red and green cubes detect this fidgeting and transmit this signal to the
other Nimios. As a result all three of the cube-shaped Nimios begin pulsing red and
green because fidgeting is detected within the same “shape group.” The red-based
pyramid, dome and cylinder pulse red because fidgeting is detected within the same
“color group.” Likewise, the green-based pyramid, dome and cylinder pulse green. The
blue-based pyramid, dome and cylinder do not light up because they are not in the same
color or shape group with either of the two Nimios that have been fidgeted with. Oscar,
who currently has the blue cube atop his desk above his monitor, knows from experience
that the steady, rhythmic pulsing he sees means that someone is fidgeting with the red
and green cubes. Oscar frequently stops by Prairie-Dawn’s desk so he knows that she has
123
the red and green cubes. He chuckles to himself figuring that she is fooling around a bit,
and decides he will go and see what she’s up to, since he is a bit bored himself.
Figure 7.4: One possible interaction with Nimio
7.6 Implementation
The physical artifacts are constructed using Smooth-On Smooth-Sil 920, a flexible,
translucent silicone. Each individual shape and its detachable base were cast and molded
by hand. The shapes are hollow and constructed to contain a Berkeley mote. Each mote is
powered by two AA batteries, which must be replaced every month or so depending on
usage. The bottoms are designed to be sturdy enough to withstand daily use, but easy
enough to open for semi-regular maintenance.
124
Nimio was developed using Crossbow MICA2 motes communicating on the 900MHz
band, with attached sensor boards. From the sensor board we use the accelerometer and
the microphone as inputs. The microphone samples at 14KHz, not a fast enough sampling
rate to reproduce sound well, but certainly high-resolution enough for a visual display of
the energy of local sounds. The accelerometer reads every 50 ms, which has proven
adequate to detect fidgeting or shaking. For output, we have replaced each of the three
surface-mount LEDs on the mote with an array of three ultra-bright LEDs connected in
parallel. Each toy, then, contains a total of nine LEDs in three different colors: red, green
and blue. The original positive solder points for the surface-mount LEDs, which would
have provided 5mA of current, serve as triggers; the ultra-bright LEDs are connected
both to the original solder points and directly to the batteries using transistors, and so are
able to draw a full 20mA for maximum brightness.
Due to the layout of the suites, multi-hop messaging must be used to communicate within
the wireless network that supports the toys. The toys themselves require a total of 12
motes. We use additional unaugmented motes mounted on the ceiling to relay messages
between the two suites. An additional base station would be necessary to better support
the group member with an office on another floor, but because his job requires him to
wander the building outside the range of the suites’ wireless network, even this would not
fully support him. Though not a part of the initial deployment, development has been
undertaken to integrate Nimio with the Equip Component Tookit (Greenhalgh, Izadi et al.
125
2004), which would provide us with infrastructure to connect Nimio with the wireless
LAN at Calit(2), enabling its use more widely throughout the building.
Nimio’s behavior was initially prototyped in Max/MSP so we could gain a more
immediate understanding of what our proposed interaction felt like. Development then
proceeded in NesC, a variant of C refactored for use in wireless sensor networks.
Technical concerns included the need to route messages appropriately. Most routing
algorithms for wireless sensor networks are concerned with routing data from sources in
the environment to a sink where the data can be analyzed. This is not the case with
Nimio, in which each node is both an input and an output. Rather than routing messages
to a single node, each node must broadcast widely, and each receiving node must respond
to messages meant for it, and also forward the message along. This sort of routing is more
prone to clogging the network with many messages, and for that reason (as well as
prolonging battery life) we attempt to optimize between fluid interaction and minimizing
the number of messages sent. Microphone, accelerometer, and mote-of-origin data is
conveyed in 23 bytes. A limited table of message ID numbers is kept on each mote, and if
the mote has seen the message before it does not forward it. The algorithmic problem of
routing well for a system like Nimio is still unsolved, but our results are satisfactory for a
system composed of a couple dozen nodes.
7.6.1 Design Rationale
Because we were trying to create a socially and spatially situated channel of information,
we were acutely aware of the dangers of violating existing social practices. We decided
126
to support inference rather than try to represent something more specific. Instead of
supposing that we could know a priori what cues and actions were most significant for
the other members of the group, we choose to create a medium that would support a more
ambiguous set of behaviors. We felt that a limited and explicit modality of input and
output would constrain the users in ways that would force them to be more focused on the
system itself instead of developing a new way of becoming attuned to one another.
To this end we chose to base our system around a set of toys. Even if they have some
predefined mode of interaction, toys are typically re-appropriated by the user for
whatever purpose they see fit. It is not taboo or unusual to do so, and by designing a toy
we are trying to draw the users away from thinking of our system as a tool which they
can only use in a certain way; we are encouraging them to explore the capabilities of the
system. Additionally, because the toy is both the means of input and output, the
disjunction between monitoring and display is erased. We hoped the users would not feel
as if they are being watched, but rather feel similarly to how one feels as they are moving
physically through a space, aware of the effects their presence has on others, and in
control of those effects.
We chose to allow the users to identify themselves with the toy, rather than identifying
the toys with a specific office or user. By allowing the users themselves to constitute the
context and negotiate the configuration of that context by means of exchange of the toys,
we are giving them the power to represent the workplace as a social space rather than a
physical one, and we are transforming the problem of representing individual activity into
127
one of group flow. Also, the fact that the users must actively maintain a mapping between
themselves and the space they inhabit supports the behaviors they are accustomed to
engaging in.
Our three levels of interaction – sound, fidgeting, and shaking – and the shape and color
resonances are meant to support the different types of working relationships to which our
participants were accustomed. We treat fidgeting and shaking as a more intimate gesture
than the widely broadcast sounds, limiting the display to members of the same color or
shape groups. If, for example, the blue-based pyramid is shaken, the owners of the red
and green pyramids will be able uniquely to identify the shaker (because only the blue
pyramid can make the other pyramids light solid blue), while the owners of other blue
shapes will have a more nebulous awareness of the action (because any other blue shape
can create that effect in that group). With this property, we tried to reflect the ways in
which members of certain close subgroups tend to be more aware of each other’s
presence, and better able to interpret each others’ actions, and give them a means to
support that behavior.
We also attempted to design for an experience similar to the way the user’s functioned in
their old office space. When the area is quiet it may be easier to distinguish between a
few sources of activity, but during busy times sounds from the offices of many coworkers are heard by all simultaneously, and become harder to distinguish. Similarly, if
sound is present near multiple red-based Nimios, those sounds and rhythms will be
additively combined and displayed on all of the Nimios. In this way the display draws on
128
the properties of sound in a small office. However, even if the office is busy, the display
does not become overwhelming, as there is an upper bound on the complexity of the
display. Because the display is somewhat ambiguous, it is adaptable to different rates of
information flow.
Finally, like the jasmine blossoms found on the desks of the group, Nimio presents an
outward display of group cohesiveness, and occupies the workplace alongside the people.
The physical presence of an object on the desk serves an additional purpose. When trying
to determine how the ambient information would fit in with other information practices,
we noted the desire of many users to present a high-tech image. A desktop toy fulfills this
need since it is usable at the very least as a physical object by anyone, including visitors,
who happen to be in the office. Additionally, the group has a preference for tangible
representations of information throughout their workday. A “high-tech” toy can strike a
balance between the aesthetics, informativeness, and usability that the group desired.
However, we did not want to create a toy that was so “high-tech” looking that it would
seem fragile or austere. Nimio’s physical design attempts to reflect a tech-friendly
atmosphere while still remaining intriguing and inviting.
7.7 What Did People Make of It?
Nimio was deployed in a complex social setting, and with very open-ended goals.
Accordingly, our assessment is not so much focused on evaluating its fitness for specific
instrumental purposes, but rather on examining whether and how it was incorporated into
the social life of the group as they settled into their routines in the new building.
129
Individual Meanings. We were surprised, after witnessing the ease with which group
members socialized, that no one ever traded Nimios with each other (as we had hoped).
We had not counted on the Nimios being regarded as highly personal objects. Yet in
follow-up interviews it became clear that certain shapes had particular meanings to our
participants. One women thought the domes were “mysterious” while another man
thought they were “boring”, but pyramids reminded him of “Egypt and Mexico”. People
poked, prodded, shook, shouted at, and even disassembled their Nimios. As one woman
said of her Nimio, “we became friends”.
Performing Group Identity. As noted earlier, the group portrayed itself as tight-knit when
we spoke to them shortly after their move to the new building. In the year following the
move-in, the group’s identity remained cohesive (though not completely undifferentiated)
and was articulated in several ways. First, individuals maintained that they worked in a
“good group” where “everyone gets along” even as they added several new members to
the original core staff. Second, the group’s informal style was manifest both in interviews
and during participant observation. Meetings tended to be opportunistic and impromptu,
rather than formally scheduled; this style was regarded as a more productive use of time,
and frequently contrasted in interviews to people’s experiences in old jobs and in other
groups. Opportunistic interaction meant that the public spaces in the office suite were
frequently used, and collaboration was often quite visible. One interviewee asserted that
the “tone” of the office “comes from the top”, establishing it as a characteristic of
everyone working for the group manager. Finally, along with this intensive intragroup
130
collaboration, many group members also interfaced with “outsiders”, whether in public
relations roles, or collaborating with funded faculty, or meeting with promising
undergraduate researchers. Nimio became a part of the image they presented to these
outside collaborators, a tangible conversation piece, a visually appropriate and eyecatching office decoration shared by most group members, and an example of the kind of
research done at their institute:
I think it’s visually interesting and unique… I have a lot of people who come and
go from my office and so I mean it’s an opportunity for me to casually just say…
‘oh yeah, we’re just part of a research project, just one of many fascinating things
we do here’… it seems appropriate in this context.
Generating group identity: shared understandings and interpretable actions. As we had
seen in our initial study, people had developed a strong sense of each others’ work habits
in the course of their interactions. People attended to each others’ habits and tailored their
communication strategies to relevant personal habits and characteristics of the person
they wanted to talk to.
Group members acted visibly or audibly in physical space (leaving office doors open or
closed), on the phone (by having audible conversations), on email and on IM (status
messages, responsiveness or unresponsiveness). Over time and with trial and error these
actions became accurately interpretable to others as meaning specific things like presence
at one’s desk, or participation in a conference call. We intended Nimio to become another
channel in this diverse communication ecology, but encountered several difficulties. The
simplest set of difficulties was purely technical. Short battery life left the Nimios
sometimes unresponsive, and routing problems cut off a few users who were spatially
131
peripheral. This meant that Nimio’s responses to its users and immediate environment
were often not consistent enough for people to develop a mental model of what the
system was doing. Another sort of difficulty stemmed from the fact that Nimio was a
distributed display. A Nimio might be reacting to a distant, otherwise imperceptible
event, but precisely because that event was distant there was no way to correlate the
distant action with Nimio’s reaction. Similarly there was no convenient way to know if
your actions were creating an effect on someone else’s Nimio. After the initial
deployment, when we set all the Nimios out in the kitchenette/copy room, no one played
with their Nimios in the same place so that they could see immediate reactions. Nimio
was designed to be unobtrusive; placed out of one’s line of sight, the blinking lights did
not demand attention the way an audio alert would. But once placed out of sight, because
it was difficult to understand, it was also out of mind. For all these reasons, Nimio did not
become a medium through which group members could act visibly and be understood by
the rest of the group.
Subgroup identity: reinforcing strong ties. While it is true that “everyone gets along” and
collaborations are fluid, some ties within the group were stronger than others. One
participant noted, with regard to potential collaborations “each of us is the center of a bell
curve of possibilities”: one might collaborate with anyone, but would be more likely to
work with some than others. Another noted that these ties are mutually reinforcing;
opportunistic face-to-face meetings with collaborators were also moments where one
might get “sucked into” yet another project. Strong ties might mean nearly constant faceto-face interaction:
132
we literally just get up, and you can see, we are coming and going out of each
other’s offices all the time, so it’s very personal contact.
Strong ties tended to coincide with physical proximity, since people had offices in the
main suite or the annex according to their job duties and organizational positions. In these
cases, some people were able to find enough correlations to render Nimio interpretable.
There were two main ways in which this happened: awareness of Nimio as a physical
object in a coworker’s office, and awareness of a coworker’s activity through Nimio. In
the first case, people were seldom aware of who possessed which shapes and colors
among their weak ties, but could identify those possessed by a few of their close
collaborators. Three women who worked closely together (identified by one as “the three
talkers”) chose their Nimios together, grabbing the three pyramids as soon as they were
available, making a conscious decision to use a set of Nimios that could communicate
most closely, making their affiliation visible. In their case, Nimio could call their
attention to what had been a merely background awareness of their closest coworkers’
activities:
I’d see the red lighting up and then if I listen – like I can tune everybody out, but
then if I stopped and listened yeah it would be her.
The tiered interaction model we had designed using a gradient of communication
between similar shapes, similar colors, and the whole group did indeed reflect the varying
levels of collaboration within the group. While Nimio did not strengthen ties uniformly
throughout the group, it made stronger and weaker ties more visible to us.
133
Three themes emerge from these observations. First, Nimio was highly interpretable to
individuals. Some seemed to have a better grasp than others of “how it works”, and many
interpreted the system in ways we wouldn’t have expected, though explanations were
available either from the researchers or from other coworkers. To some degree,
misunderstandings were due to some of the aforementioned technical and interactional
difficulties with the system, yet some interpretations were incredibly specific and
surprisingly accurate. Second, awareness of Nimio and other people was gained largely
through face-to-face interaction. For the most part, people were aware of what their most
frequent face-to-face collaborators possessed. At the same time most participants were
aware of the two receptionists, whose desks and Nimios were publicly accessible. Lastly,
Nimio highlighted and clarified some of the social complexities of the group, providing a
lens that helped us understand how these coworkers collaborated. While participants
expressed a desire to be more aware of all their colleagues, in practice that awareness
emerges from their communications over primary work activities, and each reinforces the
other. Nimio made this process more visible to us but did not break the feedback loop.
7.8 Discussion & Conclusions
During the design process of Nimio, we tried to situate the system within its users’
particular social practices and the physical environment in which they act. The space they
inhabit is made meaningful through their relationships and actions. The office space is
one in which they sit and work, but also one in which people gather, a space one must
walk through to find one’s colleague, etc. This group of people – like any group of people
– have assigned collective meaning to their relationships and interactions with each other,
134
to simple interactions like the sharing of some jasmine blossoms. Nimio was designed
from the bottom up to be similarly openly interpretable, to be an artifact to which users
can attach their own collective meanings through a process of use and negotiation. The
particular resources available to users for the generation of meaning around Nimio’s
ambiguous output affected whether they found the system interpretable or simply
confusing.
7.8.1 Context for Interpretability
Nimio was deployed into an already richly interactive setting. In its least technologically
augmented form, “awareness” is maintained simply by our sense of hearing, glances in
the right direction, or even senses of touch or smell. In their original workplace, the group
members we talked to were able to maintain awareness quite naturally because their
offices did not muffle sound. Studies out of the Kings College group (Heath and Luff
1992; Hindmarsh and Heath 2000) examine these methods of shared awareness quite
thoroughly, pointing also at the importance of shared objects. Dourish and Bellotti
(Dourish and Bellotti 1992) discuss awareness in shared workspaces, hinting at
awareness through documents that is similar to what we saw in our study.
While Nimio allows coarse activity awareness, it also depended on the development of
shared understandings, which in this case come about due to routine co-present
interactions that occur throughout the day. This was an active, interactional process.
Actions are observable and presumed to be rational. Social contexts arise from this
ongoing accounting-in-action, and it was the near-constant peripheral availability of
135
visible and accountable action that made Nimio’s minimal display intelligible to some.
Those to whom that action was less frequently available had a much harder time
interpreting Nimio. Thus, to adapt an easily understood example from one of our
interview, if one hears their coworker’s door close, and then sees their Nimio pulsate a
certain color, they may infer that their coworker is talking to someone on the phone,
because in their past experience she also sometimes closes her door for phone meetings.
To another coworker who does not hear her close her door on a regular basis, and who is
less familiar with the frequency of her phone meanings, the colored lights she triggers on
their Nimio will not mean much.
7.8.2 Accumulation of Meaning Over Space and Time
In the course of designing Nimio, we noticed a corollary to the situated display approach
that we initially took. In many environments – the office being one – “situations” may
tend to repeat themselves. Thus we see many offices in an office building, each with
similar layouts and serving similar purposes. We see similar desks in each office, and
meeting held in the same room, at the same time every week, and about similar subject
matter. In such environments, situated displays may also become distributed displays.
As mentioned before, at our first visit on site, most group members had jasmine blossoms
in their offices, all from the same bush. While one of these jasmine branches may
constitute a mere pretty office decoration, the set of them, distributed throughout many
coworkers’ offices, conveys information about the working relationships of this group.
Additionally, socially conveyed knowledge (i.e. that they all came from the same bush,
136
that group member X brought in the branches from her neighbor’s bush, that each group
member picked a branch) adds to the legibility of this display, though even without that
knowledge it is not wholly uninformative.
While each toy may allow awareness of the presence of several coworkers, the system as
a whole can convey on one level who is active, and on another level, by visible shapes
and colors, who is aware of whom. This sort of knowledge would be accumulated over
time, and require the knower to move throughout the space in order to take it in. In
practice, though the group of administrators we worked with were collaborative and
friendly, they did not all spend time in everyone else’s offices; thus no one we talked to
had a global awareness of Nimio as a complete system. Most people could tell us which
Nimio a few of their closest coworkers had, but no more; they had a partial knowledge of
Nimio as a distributed system based on whose offices they visited the most. The objects
were part of a set of local awarenesses based on each participant’s closest working
relationships, closest friendships, and most frequented paths about the office.
7.8.3 Legibility and Ambiguity
Technologies and technology use take their meaning not simply from individual
encounters; rather, they evolve over time through collective use, through the ways in
which people adopt and adapt them, just as our participants’ visible actions became
legible to each other over time and through collective participation. These collective
experiences are inherently varied, open-ended, and situated in practice. Wenger (1999)
137
describes practice as a process by which we can find the world and our encounters with it
as meaningful; in this view, practice and meaning reside within communities.
Elsewhere, we have considered these issues as they relate to the broader program of
embodied interaction (Dourish, 2001). Here, we want to elaborate on the twin central
concerns we encountered with Nimio: legibility and ambiguity, and identity and
anonymity. The legibility/ambiguity tension concerns the extent to which the device is
broadly understandable but retains enough mystery both to be engaging and to allow
users to project their own meanings onto it. When translated from interaction to
collaboration, a similar issue arises in the tension between identity and anonymity. This
arises because the goal of Nimio is to foster group cohesion rather than interpersonal
communication, and yet, in order to be meaningful and legible, people must be able to
associate visible actions with people, and to distinguish between the actions of different
actors or groups.
Our tiered interaction design – based on color groups and shape groups – is a response to
this tension. They do not allow for a precise identification of objects or activities, but
allow participation “affinity groups” to develop. While the Nimio devices do not reflect
the activity of specific individuals, neither are they simply reflections of the group as a
whole; activities can be associated with different subsets of individuals, and so they allow
for people to make distinctions between activities. Distinctions – between activities, and
between groups – are critical here, since they are the foundation of distinguishable
meaning and therefore patterns of collective signaling and interpretation.
138
Critically, the focus here is on collectives and their interactions. Traditionally,
collaborative technologies, and most particularly awareness technologies, have focused
on individual interactions or on group presence, and have articulated these in terms of
predefined forms of expression. Here, we have left the forms of expression open, and we
have used different visual and interactional forms to delineate different social groups.
What is central to the design is the overlap between different groups (color groups and
shape groups), precisely because of the ambiguity that they introduce in the interpretation
of the actions of any particular object.
The central concern that our design demonstrates is a sensitivity towards the ways in
which these devices can be meaningful to users only in the contexts of their own
practices. This is not simply an argument for customization; instead, it reflects the
observation that these devices must be designed for appropriation. A jasmine blossom is
not a device for social cohesion except when used as one; similarly, the ways in which
patterns of connection and engagement arise around ubiquitous computing technologies
suggest that the meanings of advanced technologies are also going to arise only through
real use. We have attempted to illustrate here how the tension between legibility and
ambiguity has been a central element of our design. For embodied devices, this is a key
technology challenge.
139
Chapter 8
Discussion
8.1 Design Considerations
As previously discussed in Chapter 5, Hornecker (2006) reifies four different aspects of
tangible interactive systems, corresponding essentially to objects, space, action, and
meaning. She provides design guidelines for each of those aspects, recommending, for
example, multiple access point to faciliate action, or lightweight incremental interaction
with objects. These are all solid design guidelines and I have no wish to quibble with
them. Rather, by approaching tangible systems from a framework of embodied
interaction, one tends to view these categories of objects, space, action and meaning as
richly and inextricably intertwined, and this will ultimately affect how design
considerations are framed.
8.1.1 Objects That Invite Interaction
To start with the most straightforward consideration, tangible systems should be
comprised of objects that invite interaction and exploration. This in itself is no revelation,
but in our experience we found interesting ways in which objects might be inviting to
people.
In the design of Nimio, we focused on creating objects whose intrinsic physical features –
shape, texture, size, heft, squishiness – invited touching and manipulation. We
140
deliberately avoided giving them the typical grey, metallic, or hard appearance of
(stereo)typical computational devices. Ultimately participants felt comfortable not only
touching and shaking the devices, but also taking them apart and examining the guts. (I
consider this to be a positive.)
However, it may also be pre-existing meanings or associations with an object that invite
interaction, not merely an expectation of tactile pleasantness. Representational objects
can provide an indication of what sort of data or function they are coupled with. In this
case they give the user some idea of what to expect if they are manipulated.
Signalplay, however, did not work like this, and yet representation still played a role in
inviting action. In this case, by resembling familiar objects that users know how to use,
they suggested actions that might be done with them. (The metaDESK’s passive lens may
work this way – suggesting what it does by physically resembling a normal magnifying
glass, but actually behaving a bit differently.) Engaging in a familiar action would trigger
a system response that might in turn suggest action. A practiced user, then, would
eventually act with the augmented object quite different than they would act with the
object it resembles, but the original suggested action would have started them down the
path towards familiarity with the system.
8.1.2 Publicly Available Feedback
It would be trite to suggest that users’ actions should have an immediately observable
effect on the system. I wish to suggest, rather, that in a collaborative system, in order to
141
generate the shared context needed for intelligible use, users’ actions should have an
immediate publicly available effect on the system. Numerous workplace studies have
shown people’s everyday actions to be visible and accountable; if the system’s responses
are not, then witnesses are only hearing one half of the conversation. Conversely, if a
user, personally, sees a system action, but does not have a visible triggering action with
which to correlate it, the actions of the system will be difficult to understand.
The visibility of the actions of both the system and other people will be very much
influence by the physical and social configuration of the space in which that system is to
be deployed. Different senses will have different interactional aspects; for example
SignalPlay combined visible human action with spatialized sounds made by the system.
While vision requires a clear line of sight, sound pervades the room and can serve to
publicize actions or signal participants to turn around and look. This reinforces the utility
of site-specific system designs.
8.1.3 Building For Accountability
What, then, if you are designing for a situation where there is not a great deal of visible
action available? Perhaps users are working in separate offices, perhaps they are not
collocated at all. Thus far I have focused on how collocated individuals communicate
peripherally in physical space, but this is just one particular sort of situation into which
tangible and ubiquitous computing applications may be deployed.
142
First, rather than regarding a system as standing alone, it may be important to consider
how it fits in to an ecology of communicative technologies. Damage, for example was a
tangible component of a larger system in which text, photos, sound and video could be
shared with a group of friends. Kaye’s intimate interfaces as well were intended for
couples who also communicated by phone, IM, and email. Remote actions can be known
and accounted for – though the experiential resources for doing so may differ – via
communicative media as well as physical space, and it is those accounts that may allow
people to collectively generate meaning for minimal, highly interpretable interactions.
Second, and as discussed in the previous chapter, the tension between legibility and
ambiguity is crucial. Minimal interfaces are interesting case studies, but how can one
design for a highly interpretable system that leaves ample room for individual and
particular meanings, while also providing a rich enough interaction to account for actions
through the system itself (rather than other channels)? Intimate interactions at the
intersection of the visual, the bodily and the tactile (Bitton 2006; Motamedi 2007)
explore this question, marrying expressive screen images with tangible interaction. In an
ethnographic analysis of a mixed reality system (Kirk, Crabtree et al. 2005), a simple
shared visual of hand gestures projected onto a collaborative construction task allowed
for rich, indexical gestures such as pointing, negating, imitating, and indicating spatial
orientation, with little overhead for users. Gestures were not predefined, but were legible
given the rich interactive context shared and generated by the users.
143
Third, a system may utilize mappings and representations that are relatively stable and
agreed-upon, based upon previous share experience. This possibility brings us back
around to the representational tangible systems discussed in Chapter 2. The experience of
using maps is widely (though perhaps not universally) shared. Phicons of buildings (BenJoseph, Ishii et al. 2001) are easily recognized because the experience of seeing buildings
(from many angles) is widely shared. Within a certain domain of practice, phicons that
resemble lenses and mirrors are also easily recognized (Underkoffler and Ishii 1998).
Within a certain workgroup, icons resembling certain group members could leverage preexisting shared understandings specific to that group. Those users who were confused by
Nimio often requested the use of well-established mapping either to people (using
personal icons) or to space (using maps).
Many of these design recommendations are applicable to non-tangible collaborative
interactions as well, but our attention was drawn to these issues by our evaluation of
specifically tangible systems. This brings me to my next point of discussion, regarding
the utility of tangible interactions as technological probes.
8.2 Tangible Systems as Social Probes
Tangible interfaces, especially those meant for education, are already well recognized as
objects for users to think with, but they are also objects for designers and researchers to
think with. Our use of tangible systems like SignalPlay and Nimio as social probes was
effective in uncovering people’s mundane collaborative practices. The idea of probes in
HCI is not new; cultural probes (Gaver, Dunne et al. 1999; Gaver, Boucher et al. 2004)
144
and more recently urban probes (Paulos and Jenkins 2005) are sometimes used to inspire
design. These terms are familiar enough that they may color the reader’s interpretation of
our use of probes. While we share an interest in probes’ abilities to defamiliarize
situations that we would otherwise take for granted, I would argue that our tangible
probes work in a significantly different way from cultural probes as intended by Gaver
and Dunne.
Gaver and Dunne (1999) describe cultural probes as akin to astronomical probes. They
send their probes out into the environment they wish to discover – for example, by
providing elders with packets of postcards and maps containing provocative questions to
be answered – and over time, snippets of information come back to them. Designers use
the probes as triggers for inspiration, but not for in depth analysis. Gaver et al. (2004)
actually argue against rationalizing the process by analyzing, summarizing results, or
asking unambiguous questions of participants.
By contrast, we did engage in extensive ethnomethodological analysis of video footage
and ethnographic grounded-theory analysis of field notes and interviews, in order to
understand people’s interactions with our systems. Far from dulling the inspirational
quality of the data gathered from probes, we uncovered rich and sometimes surprising
practices. In this respect, ambiguity may have been advantageous. Our tangible probes,
being also functioning systems, do not just gather information about people’s practices;
145
they perturb those practices5, and the settings into which they are deployed, in
informative ways. Nimio, for example, served as a sort of litmus test after its deployment,
uncovering social configurations that we would not have noticed from simply observing
and interviewing. Tangible probes can be objects that trigger discussions, objects around
which to talk about people’s practice.
Tangible probes situate collaboration and interaction in the physical, visible world,
making these practices more easily available for evaluation. Collaboration in the physical
world employs all available resources: space, bodies, actions, language, meanings. This is
not to say that shared understanding cannot be achieved without copresence, simply that
the wide variety of resources and observability of action in space are useful tools for
researchers of collaboration.
8.3 Technologically Mediated Space
As I have previously discussed, space is socially mediated and technology has a role in
that process. James Scott (Scott 1999) documents the ways in which modern
transportation, communication, and government have rationalized and standardized urban
and national space in ways that make them easier to regulate. These practices of
rationalizing space involve making grids (as one sees both in city blocks and in the
selling of plots of rural land) making maps (ZIP codes being a well-known example), and
making categories (e.g. direct marketing by ZIP code). Maps are a commonly used
technology today, so it is easy to overlook the important role that technology plays in
In this way our probes are perhaps not dissimilar from Garfinkel’s social and
conversational probes, though I hope ours are less anger-inducing.
5
146
mapmaking. However, even in the earliest days of surveying coast lines in the New
World, map-making required enormous resources (Turnbull 2000): money, ships and
astrolabes (Law 1986), writing and drawing techniques, standardized systems of
measurement, the organizational infrastructure to coordinate the findings of many
explorers, standardization of latitude and longitude, and still the task of compiling many
disparate bits of local knowledge into a coherent map of the New World was massively
problematic.
Infrastructures that map and locate are a crucially important part of ubiquitous
computing. “Location aware” urban computing applications frequently employ wifi
access points and cell towers (LaMarca, Chawathe et al. 2005), or satellite GPS (Benford,
Crabtree et al. 2006) in order to locate users and act accordingly. In everyday life, we can
probably all think of times when online mapping services – google maps, yahoo maps, or
mapquest – have influenced our ways of navigating a city.
When technologies affect our representations of space, inevitably, they affect our
experience of space. Mapping neighborhoods by categories of mortgage risk as assessed
by the Federal Housing Administration during the late 1930s had a profound effect on
what options might be available to homeowners and in turn how those neighborhoods
developed. A homeowner in a neighborhood rated as high risk would have far more
difficulty securing a home loan or insurance, ensuring that neighborhoods determined to
be unstable (and the presence of a single non-white family on a block could result in such
147
a rating) stayed that way (Jackson 1987). The representations that we choose (or worse,
representations that we do not choose) become reality.
Yet to focus on how technology intertwines with the rationalization and mapping of
space is to leave out much of how we experience space on an everyday basis. Curry
describes the mapping instinct as geographic, but asserts that other ways of knowing
space – the chorographic and the topographic – predate the dominance of geographic
knowledge (Curry 2005). Chorographic knowledge began as a branch of astrology and,
according to Curry, “attempts to find some order in the world by seeing a relationship
between events and the places and times at which they have occurred.” In a description
that echoes predictions for ubiquitous computing, he adds that “the world itself—
terrestrial and celestial—acted as what one today might think of as a kind of information
storage device, one that operated via what amounted to a set of signs or symbols.” In
short, the world tells us appropriate times and places to act.
Topography, on the other hand, is wrapped up in narrative. Originally, the word referred
to a verbal description of a place. “In the topographic tradition one creates a new place by
acting, routinizing, narrating, and in the process, creating an account of what constitutes a
place, of what in a place is possible and what is not possible. Places are performed.”
(Curry 2005). Indeed, even after the geographic rationalization of the ZIP code system,
he points out, postal routes on the ground still exploit topographic local knowledge.
148
Exploring the possibilities of spatially situated ubicomp applications, Can You See Me
Now? (Benford, Crabtree et al. 2006) exploits multiple ways of knowing a space. A
mobile mixed reality game of tag played in urban settings, it involves online players who
can move their avatars within a map of the city, in order to avoid being caught by GPSlocated runners on the streets of the city itself (which occurs when the runner’s physical
location overlaps the online player’s map location). On face, this might seem to give the
advantage to the online players; they, after all, have a wide birds-eye view of the space of
gameplay, while runners merely eavesdrop on the online players’ dialogue. The runners,
however, developed tactics in which they would hide in the GPS shadows of hills, where
they couldn’t be tracked, ambushing the online players. They developed a nuanced
topographic knowledge of physically invisible, but computationally detectable, features
of the territory of the game.
I do not deny the usefulness of maps, I am simply pointing out the usefulness of on-theground knowing. Maps are powerful representations, and they do affect action – state
action being one significant example. But for individuals, there is a disconnect between
the picture of the space and actually acting in the space: the map is not the territory.
Chorographic and topographic ways of knowing are concerned with our experience of
space as a setting for action and for meaning. Computational technology marries
representation and action, and ubiquitous computing locates this in real-world spaces; so
how might computational technologies be incorporated into chorographic and geographic
ways of knowing space? What does it mean to build for the rich experience of the
149
territory? The spatialized tangible systems discussed in this thesis begin to address this
question, but it is only a beginning.
150
Chapter 9
Future Work
The tangible systems presented in this thesis raise as many questions as they answer and
present some additional avenues for exploration. These systems and studies have focused
closely on the collective construction of meanings from minimal or ambiguous indicators,
and as such provide valuable insights into how people make things meaningful and how
we can study these processes. Technological probes such as SignalPlay and Nimio
provide new opportunities for these processes to happen. However, another aspect of
socially contructed meanings, discussed in Chapter 5, is that they accumulate, solidify,
shape our experiences as well as being constructed by our actions. Nimio in particular
scratches the surface of my concerns with distributed displays and the sort of habituated
environmental knowing that is accumulated as one moves through a space over time.
Informed by some of the issues foregrounded by our Nimio deployment, I am designing a
distributed display explicitly directed towards making accumulated action over time more
visible, as well as highlighting the presence of infrastructural access points.
Another line of inquiry is also suggested by this work: a scaling up of my concern with
embodied collective practice from office to urban spaces.
9.1. Distributed Displays and Wireless Hot Spots
During the Nimio deployment, we became aware of it as a distributed display: a situated
display that recurs as the situations appropriate to it themselves recur. For example we
151
see many offices in an office building, each with similar layouts and serving similar
purposes. And similarly, in a city we will find many coffee shops, many intersections,
many parking garages, bus stops, highway on-ramps, trash bins, drains and manholes. In
such environments, situated displays may also become distributed displays.
9.1.1 Distributed Displays In The Wild
Distributed displays can be a powerful way for inhabitants to read the spaces. When
examining distributed displays “in the wild”, we should consider that much of their value
as informative objects is imparted by those who read them as such. Some questions:
What is the intent behind the object? Behind the set of objects? How do inhabitants
render these objects legible? How do inhabitants’ readings confirm or confound the intent
behind these signs?
Figure 9.1: Urban Distributed Displays
Graffiti tags, for example, are ubiquitous in many urban areas. (Note the octopus stickers
on the signs in each of the above photos.) While each individual sticker conveys personal
identity, their distribution may mark off territory – indeed, this is the supposed purpose of
gang-related graffiti. In the case of sticker graffiti in the Capitol Hill area of Seattle,
taggers do not seem exclusively possessive of territory; often several stickers and markerdrawn tags shared space on a single sign without encroaching on each other. A walk
152
through the neighborhood and an attentive eye reveal not just the territorial claims of a
single octopus artist, but an overlapping network of turfs claimed by several local stickerartists.
Figure 9.2: Infrastructural Access Points
Infrastructural access points can easily be read as distributed displays, a trait hinted at by
the fact that Capitol Hill taggers chose to tag parking signs, access points to a legal
infrastructure regulating parking in the city. Manhole covers typically fade into the
background, a generally unnoticeable part of the urban landscape (unless the covers are
removed and the hole revealed). However, a closer look reveals care in labeling: covers
providing access to the water system are distinguishable from those leading to drainage,
which are different from those leading to electric. Still others are mysterious. The
distribution of manhole covers can be used to discern the routing of these infrastructures,
an activity occasionally undertaken by students motivated to explore the “steam tunnels”
beneath Stanford University.
153
9.1.2 Spot Clocks
Information and computational technologies add layers of invisible infrastructure on top
of familiar electrical and communicative networks. WiFi coverage, for example, is not
directly visible (though the savvy geek will look for gatherings of like-minded people
with their laptops out). The Spot Clock attempts to make visible not only the presence of
a nearby wireless access point, but the current amount of activity on it, and the long-term
temporal patterns of use of that access point. In an office building containing several
access points, it can provide spatially distributed information concerning activity in
different portions of the building, providing clues to human activity based on wireless
internet activity. I anticipate that spatially linked historical displays of wireless internet
activity can reveal the differing patterns of presence and work habits among groups of
people that inhabit different portions of the Calit2 building.
Figure 9.3: SpotClocks
The Spot Clock is, first of all, a working clock. A row of resistors is arrayed along the
hour hand, and that row constitutes a linear gauge of how many unique MAC addresses
154
are accessing the local wireless access point. Current will be run through more resistors,
heating them, as more computers sign onto the access point. The clock face is cloth,
saturated with thermochromic pigment. The hour hand actually rotates just beneath this
layer. Thermochromic pigment lightens with application of heat, so a red dye would turn
clear if a hot resistor were held to it. This trait persists even when mixed with regular
pigments – so orange could be made to turn yellow, or violet to turn light blue. Over
time, thermochormic pigment inevitably degrades. Instead of changing back to red when
the heat is removed, over time it will return to a lighter and lighter shade of pink. While
this is typically regarded as a bug in the material, the Spot Clock uses it as a feature.
Temporal patterns of internet use are slowly worn into the clock face and given physical
shape. Scattered at different points through the building, they display not only how many
people are online at this moment, but also different groups of people’s different temporal
rhythms of inhabiting the building. In order to prevent 24-hour temporal patterns from
overwriting each other on a clock face that rotates every 12 hours, the inner portion of the
clock face may be used to display nighttime activity and an outer ring for daytime
activity.
The emphasis here is dual. First the Spot Clocks would be distributed through space, each
one local to the access point it displays. They would mark infrastructural points of access
and make the wireless internet infrastructure of the building a bit more visible. Second,
they are distributed over time. It takes time for the history of access point use to be worn
into the pigment as the material gradually wears. As well, the viewer also experiences
155
these in time; one would not see all of the Spot Clocks at once but rather encounters them
as a part ones movements around the building, in the situation appropriate to them.
Spot clocks are currently in the process of being built. Physical fabrication is underway.
9.2 Technology and Urban Space
My work thus far on tangible interfaces has focused very much on the construction of
space and meaning as an active practice of embodied agents. Urban computing exhibits
an interest in the space into which computation is deployed, but tends to treat The City as
a generic location. But if the character of a space comes about from the actions of its
inhabitants, it stands to reason that each city is culturally and historically specific in ways
that are relevant to technology design.
I intend to examine mobility and computing technology in Bangkok, culminating in the
design of a site-specific technology or system. With the support of Intel’s People and
Practices Research Group, I have already conducted an ethnography of Thai transnational
retirees, focusing on conceptions of “home” across continents; important links between
Bangkok, New York, St. Louis, and smaller Thai towns; how ordinary people use
technology to achieve personal and cultural values.
I have identified three initial areas of interest, all aimed at understanding subjective
experiences of mobility in and around Bangkok, articulating the ways in which it may
differ from the mobility experiences of previously studied sites, and finding mobility
156
practices that can provide inspiration for technology design that is truly native to
Bangkok.
Styles of mobility. The Chao Praya River is one of Bangkok’s major thoroughfares,
indeed sometimes the only one not completely immobilized by traffic. The river affords a
certain kind of mobility, one in which movement is relatively slow, but hardly ever stops
fully. This style of mobility is echoed in the vibrant street-market life that occupies a
large portion of Bangkok’s sidewalks. How can such conceptions of place and mobility
affect mobile technology design? Between river markets, sidewalk markets, and vendors
who peddle goods primarily to drivers in rush-hour traffic, how does this form of
mobility lend itself so readily to commerce (or how does commerce foster this form of
mobility)?
Being mobile together. The car is not a private place to be alone, but a semi-private social
space for groups of people to cooperate in accomplishing a goal, a space of warmth,
closeness, cooperation and conflict with people you trust and depend on. Driving is often
an activity that everyone in the car may participate in, because everyone has strong
opinions on what is the best way to get there, which way will avoid traffic this time of
day, and passengers in the back are frequently recruited to check blind spots and parking
jobs. The cell phone is often used a way to extend that cooperation beyond those
physically in the car.
157
Distributed households: distance, rather than weakening the role of the family, may allow
members to better fulfill responsibilities. Examples include contributing to family income
through remittances from high-paying but distant jobs, or helping siblings send their
children to school in Bangkok or the US by providing them with lodging and support.
How do people use information and communication technology to fulfill their familial
responsibilities?
A theoretical focus on embodied interaction informed much of my work in tangibles, and
I am continuing to apply that framework to my work in urban spaces. A central concern
here is still the creation and sharing of meaning as a social process, and the fact that
technological systems are deployed and appropriated into pre-existing, rich social milieus
and communities of practice.
158
Bibliography
Anderson, J. R. (1995). Cognitive psychology and its implications, WH Freeman.
Anderson, K. (2004). "'ensemble': Playing with Sensors and Sound." Extended Abstracts
ACM Conf Human Factors in Computing Systems (CHI): 1239-1242.
Aoki, P. M., R. E. Grinter, et al. (2002). "Sotto voce: exploring the interplay of
conversation and mobile audio spaces." Proceedings of the SIGCHI conference on
Human factors in computing systems: Changing our world, changing ourselves:
431-438.
Ark, W. S., C. Dryer, et al. (1998). "Representation Matters: The Effect of 3D Objects
and a Spatial Metaphor in a Graphical User Interface." Proceedings of HCI on
People and Computers XIII: 209-219.
Baerentsen, K. B. and J. Trettvik (2002). "An Activity Theory Approach to Affordance."
Proceedings of NordiCHI 2002: 51-60.
Ben-Joseph, E., H. Ishii, et al. (2001). "Urban Simulation and the Luminous Planning
Table: Bridging the Gap between the Digital and the Tangible." Journal of
Planning Education and Research 21(2): 196-203.
Benford, S., A. Crabtree, et al. (2006). "Can you see me now?" ACM Transactions on
Computer-Human Interaction (TOCHI) 13(1): 100-133.
Bitton, J. (2006). Distance and Sexuality: where HCI meets convenience and affinity.
Workshop on Sexual interactions: Why We Should Talk About Sex in HCI, at
CHI 2006. Montréal, Québec, Canada.
Brave, S. and A. Dahley (1997). "inTouch: a medium for haptic interpersonal
communication." Conference on Human Factors in Computing Systems: 363-364.
Brewer, J., A. Williams, et al. (2007). A Handle on What's Going On: Combining
Tangible Interfaces and Ambient Displays for Collaborative Groups. First
International Conference on Tangible and Embedded Interaction (TEI). Baton
Rouge, LA.
Burrell, J., G. Gay, et al. (2002). "Context-Aware Computing: A Case Study." Proc. Intl.
Conf. Ubiquitous Computing Ubicomp (2002): 1-15.
159
Chang, A., B. Resner, et al. (2001). "LumiTouch: an emotional communication device."
Conference on Human Factors in Computing Systems: 313-314.
Chong, J. (2006). Transparency and Organizational Knowledge: Knowledge Sharing
Through Work Practice. Academy of Management Meeting. Atlanta, GA.
Crabtree, J., M. Nathan, et al. (2003). "MobileUK: Mobile phones and everyday life."
Work Foundation/iSociety (2003).
Curiel, J. M. and G. A. Radvansky (1998). "Mental organization of maps." Journal of
experimental psychology. Learning, memory, and cognition 24(1): 202-214.
Curry, M., D. Phillips, et al. (2004). "Emergency Response Systems and the Creeping
Legibility of People and Places." The Information Society 20(5): 357-369.
Curry, M. R. (2005). "Toward a Geography of a World Without Maps: Lessons from
Ptolemy and Postal Codes." Annals of the Association of American Geographers
95(3): 680.
Dinkla, S. (1996). "From Participation to Interaction: Toward the Origins of Interactive
Art." Clicking In: Hot Links to a Digital Culture, Bay Press.(1996): 279-290.
Dourish, P. (2001). Where the Action Is: The Foundations of Embodied Interaction,
Bradford Book.
Dourish, P. and G. Bell (In press). "The Experience of Infrastructure and the
Infrastructure of Experience: Meaning and Structure in Everyday Encounters with
Space." Environment and Planning B: Planning and Design.
Dourish, P. and V. Bellotti (1992). Awareness and Coordination in Shared Workspaces.
ACM Conference on Computer Supported Cooperative Work.
Dourish, P. and S. Bly (1992). "Portholes: Supporting Awareness in a Distributed Work
Group."
Dreyfus, H. L. (1991). Being-in-the-world : a commentary on Heidegger's Being and
time, division I. Cambridge, Mass., MIT Press.
Erickson, T. and W. A. Kellogg (2000). "Social Translucence: An Approach to Designing
Systems that Support Social Processes." ACM Transaction of Human-Computer
Interaction 7(1): 59-83.
160
Ferris, K. and L. Bannon (2002). ""... a load of ould boxology!"" Proceedings of the
conference on Designing interactive systems: processes, practices, methods, and
techniques: 41-49.
Fitzpatrick, G., S. Parsowith, et al. (1998). "Tickertape: awareness in a single line."
Conference on Human Factors in Computing Systems: 281-282.
Fogarty, J., S. E. Hudson, et al. (2004). "Examining the robustness of sensor-based
statistical models of human interruptibility." Proceedings of the 2004 conference
on Human factors in computing systems: 207-214.
Frei, P., V. Su, et al. (2000). "Curlybot: designing a new class of computational toys."
CHI 2000- Conference on Human Factors in Computing Systems'The Future is
Here': 129-136.
Garfinkel, H. (1984). Studies in ethnomethodology. Cambridge, UK, Polity Press.
Garrioch, D. (2003). "Sounds of the city: the soundscape of early modern European
towns." Urban History 30(01): 5-25.
Gaver, B., T. Dunne, et al. (1999). "Cultural Probes." Interactions 6(1): 21-29.
Gaver, W. W., A. Boucher, et al. (2004). "Cultural probes and the value of uncertainty."
interactions 11(5): 53-56.
Glance, N. S., D. S. Pagani, et al. (1996). "Generalized process structure grammars
GPSG for flexible representations of work." Proceedings of the 1996 ACM
conference on Computer supported cooperative work: 180-189.
Goodman, E. and M. Misilim (2003). "The Sensing Beds." UbiComp 2003 Workshop.
Greenhalgh, C., S. Izadi, et al. (2004). "ECT: A Toolkit to Support Rapid Construction of
Ubicomp Environments." Proceedings of Ubicomp 2004.
Grinter, R. E., P. M. Aoki, et al. (2002). "Revisiting the visit:: understanding how
technology can shape the museum visit." Proceedings of the 2002 ACM
conference on Computer supported cooperative work: 146-155.
Grinter, R. E. and M. Eldridge (2001). "y do tngrs luv 2 txt msg." Proceedings of the
Seventh European Conference on Computer-Supported Cooperative Work
ECSCW 1: 219-238.
161
Hansson, R. and P. Ljungstrand (2000). "The reminder bracelet: subtle notification cues
for mobile devices." Conference on Human Factors in Computing Systems: 323324.
Harrison, S. and P. Dourish (1996). "Re-place-ing space: the roles of place and space in
collaborative systems." Proceedings of the 1996 ACM conference on Computer
supported cooperative work: 67-76.
Heath, C. and P. Luff (1991). "Disembodied conduct: communication through video in a
multi-media office environment." Proceedings of the SIGCHI conference on
Human factors in computing systems: Reaching through technology: 99-103.
Heath, C. and P. Luff (1992). "Collaboration and control: Crisis management and
multimedia technology in London Underground Line Control Rooms." Computer
Supported Cooperative Work (CSCW) 1(1): 69-94.
Heath, C., P. Luff, et al. (2002). "Crafting participation: designing ecologies, configuring
experience." Visual Communication 1(1): 9-33.
Heidegger, M. (1996). Being and Time: A Translation of Sein und Zeit,(J. Stambaugh,
Trans.).
Hill, W. C., J. D. Hollan, et al. (1992). "Edit wear and read wear." Proceedings of the
SIGCHI conference on Human factors in computing systems: 3-9.
Hindmarsh, J. and C. Heath (2000). "Sharing the Tools of the Trade: The Interactional
Constitution of Workplace Objects." Journal of Contemporary Ethnography
29(5): 523-562.
Hindmarsh, J., C. Heath, et al. (2002). "Creating assemblies:: aboard the Ghost Ship."
Proceedings of the 2002 ACM conference on Computer supported cooperative
work: 156-165.
Höök, K., A. J. Gersie, et al. (2003). Designing Information Spaces: The Social
Navigation Approach, Springer.
Hornecker, E. (2005). "A Design Theme for Tangible Interaction: Embodied
Facilitation." Conference on Computer-Supported Cooperative Work 18: 22.
Hornecker, E. and J. Buur (2006). "Getting a grip on tangible interaction: a framework on
physical space and social interaction." Proceedings of the SIGCHI conference on
Human Factors in computing systems: 1-59593.
162
Husserl, E. and D. Cairns (1977). Cartesian Meditations: An Introduction to
Phenomenology, Springer.
Ingstrup, M. (2006). "A Perceptual Approach to Understanding the Physical-Digital
Design Space." forthcoming.
Inhelder, B. and J. Piaget (1958). The Growth of Logical Thinking from Childhood to
Adolescence. New York, Basic Books.
Ishii, H. and B. Ullmer (1997). "Tangible bits: towards seamless interfaces between
people, bits and atoms." Proceedings of the SIGCHI conference on Human factors
in computing systems: 234-241.
Ishii, H., J. Underkoffler, et al. (2002). "Augmented Urban Planning Workbench:
Overlaying Drawings, Physical Models and Digital Simulation." Proceedings of
IEEE and ACM International Symposium on Mixed and Augmented Reality,
20002.: 203-211.
Ito, M. (2001). "Mobile Phones, Japanese Youth, and the Replacement of Social
Contact." Proc. Ann. Mtg. Society for Social Studies of Science.
Jackson, K. T. (1987). Crabgrass Frontier: The Suburbanization of the United States,
Oxford University Press US.
Ju, W. G., B. Lee, et al. (2007). Range: Exploring Proxemics in Collaborative
Whiteboard Interaction. Research Boaster, Human-Computer Interaction
Consortium (HCIC). Fraser, CO.
Kabisch, E., A. Williams, et al. (2005). "Symbolic objects in a networked gestural sound
interface." Conference on Human Factors in Computing Systems: 1513-1516.
Kaye, J. J. (2005). Intimate Objects: A Site for Affective Evaluation. Workshop Paper:
Workshop on Innovative Approaches to Evaluating Affective Interfaces, CHI
2005. Portland, OR.
Kaye, J. J., M. K. Levitt, et al. (2005). "Communicating intimacy one bit at a time."
Proceedings of CHI 5.
Kelleher, W. (2004). The Troubles in Ballybogoin: Memory and Identity in Northern
Ireland, University of Michigan Press.
Kikin-Gil, R. B. B. (2005). Mediating social relationship through mobile communication,
Masters Thesis, Interaction Design Ivrea (2005).
163
Kirk, D., A. Crabtree, et al. (2005). "Ways of the Hands." European Conference on
Computer-Supported Cooperative Work 18: 22.
Klemmer, S. R., M. W. Newman, et al. (2001). "The Designers' Outpost: A Tangible
Interface for Collaborative Web Site Design." The 14th Annual ACM Symposium
on User Interface Software and Technology: UIST2001, CHI Letters 3(2): 1-10.
Klemmer, S. R., M. Thomsen, et al. (2002). "Where do web sites come from?: capturing
and interacting with design history."
LaMarca, A., Y. Chawathe, et al. (2005). "Place Lab: Device Positioning Using Radio
Beacons in the Wild." Proceedings of Pervasive.
Law, J. (1986). "On the methods of long-distance control: vessels, navigation and the
Portuguese route to India." Power, Action and Belief: A New Sociology of
Knowledge: 234-263.
Manzo, J. (2005). "Social Control and the Management of" Personal" Space in Shopping
Malls." Space and Culture 8(1): 83-97.
McNerney, T. S. (1999). Tangible Programming Bricks: An approach to making
programming accessible to everyone, Massachusetts Institute of Technology.
Merleau-Ponty, M., C. Smith, et al. (2002). Phenomenology of perception. London ; New
York, Routledge.
Motamedi, N. (2007). Keep in Touch: A Tactile-Vision Intimate Interface. Proc. Tangible
and Embedded Interaction. Baton Rouge, LA.
Nardi, B. A., S. Whittaker, et al. (2000). "Interaction and outeraction: instant messaging
in action." Proceedings of the 2000 ACM conference on Computer supported
cooperative work: 79-88.
Nelson, L., S. Bly, et al. (2002). "Quiet Calls: Talking Silently on Mobile Phones."
Proceedings of CHI 2: 174-181.
Norman, D. A. (2002). The design of everyday things, Basic Books.
Palen, L. (2002). "Mobile telephony in a connected life." Communications of the ACM
45(3): 78-82.
Papert, S. (1980). Mindstorms: children, computers, and powerful ideas, Basic Books.
164
Patten, J. and H. Ishii (2000). A comparison of spatial organization strategies in graphical
and tangible user interfaces. Proceedings of DARE 2000 on Designing augmented
reality environments. Elsinore, Denmark, ACM Press.
Paulos, E. and T. Jenkins (2005). "Urban probes: encountering our emerging urban
atmospheres." Conference on Human Factors in Computing Systems: 341-350.
Peirce, C. S. (1885). "On the Algebra of Logic: A Contribution to the Philosophy of
Notation." American Journal of Mathematics 7: 180-202.
Raffle, H., A. Parkes, et al. (2004). "Topobo: A Constructive Assembly System with
Kinetic Memory." Proceedings of CHI 2004: 869-877.
Robertson, G., M. Czerwinski, et al. (1998). "Data mountain: using spatial memory for
document management." Proceedings of the 11th annual ACM symposium on
User interface software and technology: 153-162.
Robertson, T. (1997). "Cooperative Work and Lived Cognition: A Taxonomy of
Embodied Actions." Proceedings of the European Conference on ComputerSupported Cooperative Work ECSCW 97.
Robertson, T. (2002). "The Public Availability of Actions and Artefacts." Computer
Supported Cooperative Work (CSCW) 11(3): 299-316.
Roskos-Ewoldsen, B., T. P. McNamara, et al. (1998). "Mental representations of large
and small spatial layouts are orientation dependent." J Exp Psychol Learn Mem
Cogn 24(1): 215-26.
Sacks, H. and H. Garfinkel (1970). "Formal Structures of Practical Action." Theoretical
Sociology.
Sawhney, N., S. Wheeler, et al. (2001). "Aware Community Portals: Shared Information
Appliances for Transitional Spaces." Personal and Ubiquitous Computing 5(1):
66-70.
Schutz, A. (1967). The phenomenology of the social world. Evanston, Ill., Northwestern
University Press.
Scott, J. C. (1999). Seeing Like a State: how certain schemes to improve the human
condition have failed, Yale University Press.
Smail, D. L. (1999). Imaginary Cartographies: Possession and Identity in Late Medieval
Marseille, Cornell University Press.
165
Srivastava, L. (2004). "Social and Human Considerations for a More Mobile World."
Published in ITU/MIC conference workshop on shaping the future of mobile
information society.
Streitz, N. A., J. Geißler, et al. (1994). "DOLPHIN: integrated meeting support across
local and remote desktop environments and LiveBoards." Proceedings of the 1994
ACM conference on Computer supported cooperative work: 345-358.
Turkle, S. and S. Papert (1992). "Epistemological Pluralism and the Revaluation of the
Concrete." Journal of Mathematical Behavior 11(1): 3-33.
Turnbull, D. (2000). Tricksters and Cartographers: Maps, Science and the State in the
Making of a Modern Scientific Knowledge Space. . Excerpt from "Masons,
Tricksters, and Cartographers: Comparative Studies in the Sociology of Scientific
and Indigenous Knowledge, Harwood Academic: 89-129.
Ullmer, B. and H. Ishii (1997). "The metaDESK: Models and Prototypes for Tangible
User Interfaces." Proceedings of the 10th annual ACM symposium on User
interface software and technology: 223-232.
Ullmer, B. and H. Ishii (2000). "Emerging frameworks for tangible user interfaces." IBM
Systems Journal 39(3): 915-931.
Ullmer, B., H. Ishii, et al. (1998). "mediaBlocks: physical containers, transports, and
controls for online media." Proceedings of the 25th annual conference on
Computer graphics and interactive techniques: 379-386.
Underkoffler, J. and H. Ishii (1998). "Illuminating light: an optical design tool with a
luminous-tangible interface." Proceedings of the SIGCHI conference on Human
factors in computing systems: 542-549.
Vom Lehn, D., C. Heath, et al. (2001). "Exhibiting Interaction: Conduct and
Collaboration in Museums and Galleries." Symbolic Interaction 24(2): 189-216.
Weilenmann, A. (2001). "Negotiating Use: Making Sense of Mobile Technology."
Personal and Ubiquitous Computing 5(2): 137-145.
Weinberg, G., R. Aimi, et al. (2002). "The Beatbug network: a rhythmic system for
interdependent group collaboration." Proceedings of the 2002 conference on New
interfaces for musical expression: 1-6.
Williams, A., S. Farnham, et al. (2006). "Exploring wearable ambient displays for social
awareness." Conference on Human Factors in Computing Systems: 1529-1534.
166
Williams, A., E. Kabisch, et al. (2005). "From Interaction to Participation: Configuring
Space Through Embodied Interaction." Proceedings of Ubicomp. Tokyo, Japan:
287-304.
Woodruff, A. and P. M. Aoki (2003). "Media Affordances of a Mobile Push-To-Talk
Communication Service." Arxiv preprint cs.HC/0309001.
Zuckerman, O., S. Arida, et al. (2005). "Extending tangible interfaces for education:
digital montessori-inspired manipulatives." Conference on Human Factors in
Computing Systems: 859-868.
167