Graceful Interaction In Intelligent Environments
Mikael Wiberg, PhD
Department of Informatics
Umeå University, 901 87 Umeå,
Sweden
mwiberg@informatik.umu.se
+46 (0)90 786 61 15
ABSTRACT
The current trend of blending digital technology into our physical
surrounding, and the current attempts made to design for
disappearing UIs open up for important questions related to how
people will be able to make sense of, and interact with, these new
intelligent environments. In this paper we present an empirical
interaction study of how people try to make sense of an everyday
sensing system that might be a natural element of a more complex
intelligent environment in a nearby future. In our study we
collected video-recorded data on how 8 subjects explored an
interactive lamp (a Mathmos Airswitch interactive lamp). In our
study we were able to observe how the subjects quickly developed
different personal, technical and dramatic/ affective interaction
schemas as ways of exploring the functionality of the interactive
lamp and how they then fine-tuned their interaction schemas to
effectively interact with the lamp (i.e. switch it on and off and
brighten it/dim it). We could also observe how the subjects, in
their fine-tuning of their interaction schemas, also adjusted their
overall performance in the intelligent environment and how they
developed a meta-level interaction schema that we choose to label
“graceful interaction”. We discuss our findings and its theoretical
and practical implications in relation to design of everyday
intelligent environments.
user interfaces (UIs) so that we, as users in and of these
environments, can interact with and control these smart objects.
However, along with the current trend of blending digital
technology into our physical surrounding, and along with the
current attempts made to design for disappearing UIs, these user
interfaces will to a large extent be intentionally visually hidden to
the user as this digital technology gets highly interwoven and
blended with the physical design. This is the current trend within
several emerging areas including e.g. interactive architecture
[3,7,8], ambient intelligence, and ubiquitous computing. The
“ambient alarm clock” (see figure 1) serves as a good illustration
of this current trend. When not switched on the ambient alarm
clock appears to be only a piece of wood. However, when touched
it switches on and glows just like an ordinary alarm clock.
However, the UI is visually hidden to its user and even when
switched on the device provide no guidance or interface to its user
for e.g. setting the alarm, setting the time, or switching it off
again.
Keywords
Ambient
technology,
Graceful
interaction,
Intelligent
environments, Interaction schemas, Smart objects, Ubiquitous
computing.
1. INTRODUCTION
In a nearby future our homes, and probably also a lot of public
places, are likely to be filled with more or less smart objects that
taken together can be viewed as intelligent environments in terms
of computationally augmented physical objects designed to do
various things and services for us as users of these embedded
systems. These smart objects that taken together will constitute
larger intelligent environments will also have various kinds of
Figure 1. The ambient alarm clock. (Source: [8]).
Another illustrative example of a more large scale and more
environmentally embedded implementation is the Hello.Wall
[14,15,16] (see figure 2), a 1.8-meter-wide by 2-meter-high
ambient display with integrated light cells and sensing technology
designed to show continuously aesthetic patterns conveying
presence and mood parameters [14].
An important aspect of the Hello.Wall installation is that it
illustrates how the technology is designed to mainly serve as a
background technology, i.e. an ambient technology.
Further on, this installation also illustrates an important social
aspect of intelligent environments, i.e. that these intelligent
environments might be populated by several persons who are
simultaneously interacting with each other while interacting with
various smart objects and ambient displays in an intelligent
environment.
While all these five questions as formulated by [2] are highly
relevant to address when designing ubiquitous information
environments, the list of questions can be further extended if not
only considering the system view, but also the user’s view of the
system. These questions might then also include e.g. how the user
should be able to e.g. find out, understand, and be able to explore
what kind of services and functionality an intelligent environment
can offer to them, and by which means and through which
interaction modalities the user is supposed to interact with the
system given that the user interface to a large extent will be
visually hidden to the user.
In an attempt to address these questions from a user-centered
perspective we have in our project explored this sense making
issue empirically. Our overall objective in this project has thus
been to address the following research questions:
- How do people make sense of sensing systems?,
- How do people go about discovering what a sensing system can
do for them?,
- How do people develop ways of controlling a sensing system?
Figure 2. The Hello.Wall [14,15,16].
The Hello.Wall shares the same UI design principles as the
ambient alarm clock in that the UI is visually hidden to the user
and there are no help buttons, help menus, or wizards
implemented to help or guide the user in interpreting the
representations on the Hello.Wall or in his/her understanding of
how to execute system commands in front of this ambient display.
On a more general level, and as stated by Mark Weiser [19,20],
who originally envisioned this movement towards embedded
computational power in everyday objects under the label
“ubiquitous computing”, this is in fact one of the ultimate goals
with this current movement, i.e. to actually hide away the user
interfaces so good that it will be hard for the user to even know
that a smart object has a user interface:
“For ubiquitous computing one of the ultimate goals is to design
technology so pervasive that it disappears into the surrounding
[19,20].”
While this goal has the advantage of not cluttering up our
surrounding with all various kinds of graphical user interfaces and
buttons on every single smart object that will be part of e.g. a
future smart home environment it does at the same time present us
with a delicate problem of how we as users of these systems
should be able to make sense of various smart objects or “sensing
systems” [2] that will surround us. As formulated by Bellotti, et al
[2] in their paper “Making sense of sensing systems” these
sensing systems/ambient technologies present us with the
following five questions:
- When I address a system, how does it know I am addressing it?
- When I ask a system to do something how do I know it is
attending?
- When I issue a command, how does the system know what is
relates to?
- How do I know the system understands my command and is
correctly executing my intended action?
- How do I recover from mistakes?
- And, do people develop any more general interaction schemas
that they can reuse whenever they need to interact with a sensing
system?
The basic motivation for undertaking this kind of empirically
driven, open-ended, and user-centered research is at least three
folded: First, we need a better understanding of how people go
about exploring intelligent environments and smart objects. This
is crucial knowledge for every interaction design project aimed at
designing usable ambient devices or more complex intelligent
environments. Second, we believe that this kind of empirical
interaction studies, with an explicit focus on the user’s behavior in
intelligent environments, can be valuable for extracting more
general design guidelines, and requirements for design of
intelligent environments. Finally, in a long-term perspective, we
believe that this kind of empirical interaction studies is a valuable
and important element in the development of interaction theories
capable of describing, analyzing and predicting human interaction
in and with intelligent environments.
The rest of the paper is structured as follows: In the next section
we present some related work that has inspired us in our own
research throughout this project. We then present our empirical
study in which we report on our collected data from an
experiment in which 8 subjects were set out to explore and
interact with a Mathmos Airswitch interactive lamp as a single
example of a smart object that might be a natural part of future,
and more complex, intelligent environments. We then present our
results from this study followed by a discussion of its theoretical
and practical implications in relation to design of everyday
intelligent environments before concluding the paper.
2. RELATED WORK
The trend towards intelligent environments, pervasive and
ubiquitous computing, and smart objects has been well covered in
the literature (e.g. [4]). Further on, e.g. Malcolm McCullough
[11] describes in his book “Digital ground” how architecture and
mobile, ubiquitous and pervasive technologies will be completely
blended in a nearby future and how new intelligent environments
will be built upon this complete blend of our physical and virtual
world.
Another indicator of this trend towards intelligent environments
can be found in current research in the area of ubiquitous
computing. Here, a strong trend is towards design of
computational things for our homes and for individuals in their
everyday life. However, while this trend is very promising most of
the work we have found is directed towards design of various
smart objects and computationally augmented physical appliances
rather than reporting on empirical interaction studies of such
smart objects in use. One exception is the paper “unremarkable
computing” [17] in which the authors reports from an empirical
study of people’s everyday routines and its implications for design
of ubiquitous information environments. In their paper they report
from an empirical study in which they had an explicit focus on
understanding what people do in relation to how the technologies
and everyday objects that they use becomes highly invisible or
unremarkable to the users. The basic motivation behind their
research project was to do an empirical study along the idea of
designing this technology in a way that makes it truly invisible,
i.e. to the extent that it disappears into the periphery of the users
attention. While this empirical interaction study present very
interesting empirical results in terms of everyday routines in
relation to transparency of everyday objects they did at the same
time focus upon the disappearance rather than the appearance of
this technology to the user. Thus, their did not focus on how
people go about discovering and understanding what an
intelligent environment can do for them in their everyday life, and
how they effortlessly, and with good precision, can interact with
different smart objects in such an environment.
Along this line of thought we found very few empirical studies
conducted with an explicit focus on how people, in a very openended way, and without any further instructions, go about when
asked to explore intelligent environments and smart objects. For
one exception see e.g. the research conducted by [1] in which they
looked into users interaction with sensing systems from the
perspectives of expected, sensed, and desired movements. Most of
the studies we found focused on experiments with intelligent
environments and smart objects in which clear descriptions or a
quite explicit set of instructions have been given to the subjects as
a introduction to the environment they should interact with. The
typical focus reported has been on task-solving and navigation in
these environments (e.g. computer augmented museums) and not
on how people develop their own ways of interacting with these
environments per se. Thus, the purpose with our study is to
contribute to the area of ubiquitous computing and intelligent
environments with an empirical study with an explicit purpose on
understanding how people actually go about when making sense
of highly embedded/ambient sensing systems.
3. EMPIRICAL EXPERIMENT WITH
MATHMOS AIRSWITCH
Below we describe the overall design of our empirical study.
3.1 Research questions
In our empirical study we wanted to address the following
research questions: How do people make sense of sensing
systems?, How do they go about discovering what a sensing
system can do for them?, How do they develop ways of
controlling a sensing system? And, do they develop any more
general interaction schemas that they can reuse whenever they
need to interact with a sensing system?
3.2 Material
In order to explore these research questions empirically we
decided to set up a simple experiment environment in which
persons could be recorded on video while interacting with a smart
object.
One idea that we had for this experiment was that we wanted to
keep the technology on a fairly simple level, and limit our study to
a quite simple smart object that the subjects could fully explore
within a rather short period of time. As a result of this we decided
to do our experiment with the Mathmos Airswitch interactive
lamp [10] (see figure 3).
Simply described, the Mathmos Airswitch interactive lamp has no
on/off-switch on the cord like any traditional lamp. Instead, the
user is suppose to control the lamp with various hand movements
or gestures, e.g. if the user slowly passes his/her hand above the
Mathmos Airswitch it will switch on and if the user moves his/her
hand higher or lower above the Airswitch it will brighten or dim
the light.
Figure 3 illustrates this idea with the different positions of the
hand above the lamp and the corresponding brightness provided.
Figure 3. The Mathmos Airswitch interactive lamp used in our
experimental study. (Source: [10]).
On a technical level, the interactive lamp consists of a traditional
light bulb connected to two photo sensors via a microcontroller.
The photo sensors are mounted vertically inside the lamp directed
towards the opening on the top of the lamp.
3.3 Subjects
In our experiment we collected video-recorded data on how 8
subjects explored the interactive lamp and its functionality. All
subjects chosen for this experiment are academics in the area of
IT. Thus, we accounted them to be very used to traditional
computers. We found this aspect to be important since we did not
wanted any unfamiliarity with ordinary computers to affect this
study. Table 1 below shows some demographic data of the
subjects that took part in this study.
1
12
Subject
Male/female
Age
2
8
1
Female
29
3
7
9
2
Female
31
4
3
Female
38
5
15
4
Female
30
6
11
5
Male
51
7
10
6
Male
30
8
8
7
Male
50
8
Male
32
Table 1. User demographics.
3.4 Procedure
The experiment lasted for a maximum of 30 minutes/subject and
the only instructions given to each subject was as follows:
- “As you can see, this is not a traditional lamp since it lacks the
typical on/off switch on the cord. Therefore, your mission is to
explore how you can interact with the lamp and how you can
control it (i.e. switch it on and off, dim it and brighten it). You are
supposed to freely explore it and just find out how it works.
Please let me know when you feel that you have full control over
your interaction with the lamp”.
As seen in this transcription no further instructions were given.
There was no information given about the photo sensors or that it
reacted to gestures or hand movements. Instead, this lack of
instructions was part of the experiment design.
After each experiment a follow-up discussion was initiated to
collect comments from the subjects concerning their experiences
of this and their own reflections on how they had approached this
problem of making sense of this rather simple sensing system.
4. PRELIMINARY RESULTS
On a general level, every subject that took part in our study found
out how to interact with the lamp within 15 minutes. Table 2
below present a more precise summary of the time spent by each
individual from the time the instructions were given to the
moment when the subject said that they now had a good
understanding of how to interact with the lamp.
On a more detailed level of analysis we could observe how the
subjects quickly developed different interaction schemas as ways
of exploring the functionality of the lamp and how they then finetuned their interaction schemas to effectively interact with the
lamp (i.e. switch on and off the lamp and dim it).
We could also observe how the subjects, in their fine-tuning of
their interaction schemas, also adjusted their overall performance
in the intelligent environment and how they developed a metalevel interaction schema that we choose to label “graceful
interaction”.
Subject
Minutes spent to get in full
control over the interactive lamp
Table 2. Summary of time spent in order to find out how to
interact with the lamp.
While it was an interesting observation taken in isolation that the
subjects rather quickly understood how to interact with the lamp,
an even more interesting thing was how the subjects reached this
understanding. Below we present three different techniques, or
interaction schemas that the subjects applied to build up a good
understanding of how the lamp worked and how they could
interact with it and control its functionality. The term “interaction
schemas” was chosen here as a way of talking about an almost
ritualistic behavior that every subject developed in their
interaction with the lamp, i.e. they tried out one interaction
modality or one interaction technique. They then tested it over
and over again to validate if it actually could be a part of the
interaction with the lamp and, if it worked, they tried to extend
their interaction schema a little bit further until they could both
switch the lamp on and off and dim/brighten the lamp using their
own developed interaction schema.
Below we present three different kinds of interaction schemas that
were possible to identify during this rather limited experiment.
These three interaction schemas include 1) personal interaction
schemas, 2) technically-driven interaction schemas, and 3)
dramatic/affective interaction schemas.
4.1 Personal interaction schemas
Every subject developed their own personal interaction style or
interaction schema, but these schemas where not just personal in
the sense of being made up by a single individual. Instead, it was
very clear that the interaction schema constructed heavily
reflected the individual behind the schema. This, through the
incorporation of very personal body language into their
interaction schema, ways of moving their hands, gestures applied,
and the speed chosen for executing a gesture. The gesture speed
issue was actually a quite big problem to three of the subjects who
executed exactly the same kind of gesture for brightening the lamp
as described in the manufactures manual to the lamp but the
gesture was executed to fast so the photo sensors inside the lamp
did not manage to register it as a valid system command.
The interaction styles applied for interacting with the lamp was
further on very personal since it was highly influenced by the
subjects prior experiences which influenced what kind of mental
models and interaction metaphors that were chosen and tried out
when approaching and exploring the interactive lamp, e.g. as
described under the next section on “technically driven interaction
schemas” one subject compared the lamp with some toys that
react to loud noises. E.g. a noise-controlled toy car can go in
another direction when its “driver” makes loud noises by clapping
his/her hands.
Another subject in our study thought about the lamp as an active
volcano or a fire possible to put out if placing ones hand in direct
directly above and in direct contact with the lamp in an attempt to
“smother the fire” (see figure 4).
tried several different sensor-driven interaction techniques to
make the lamp react to their actions. They clapped their hands to
see if it reacted to sound or noises (sound sensors), they touched
the lamp both on the outside but also putted their fingers down the
lamp from the top (touch sensors), they tried to tilt the lamp (tilt
sensors/accelerometers) and they tried to warm the lamp with their
hands (heat sensors).
One interesting observation made was that while one of the
subjects were tilting the lamp the top of the lamp was facing the
table resulting in a dimming of the light. This was taken by the
subject experimenting with the lamp as a preliminary confirmation
that the lamp did in fact react upon tilting. However, after having
tried to tilt the lamp a couple of times without managing to dim it
again this idea was abandoned.
4.3 Dramatic/affective interaction schemas
Figure 4. The “smother the fire” interaction technique
Another subject viewed the interactive lamp as a candle and
leaned over the lamp in an attempt to “blow out the candle” (dim
the light) or blow on it to give it some fresh air (brighten it).
These two subjects interaction techniques were quite different (i.e.
putting ones hand over the lamp vs. trying to blow out the
“electric candle”) and it also builds upon two different mental
models. At the same time these two examples illustrates
something of rather great importance here. The thing is that both
of these two techniques work. The first technique works since the
distance between the photo-sensor and the hand is rather short,
thus resulting in a dimming of the lamp. Also, the second
technique worked quite well. Not because of the actual attempts to
blow out the candle, but, since the subject placed his/her head
above the light and then lean towards it in order to blow it out the
photo-sensor interpreted this head movement in the same way as a
hand movement which also result in a dimming of the lamp.
On the other hand, these personally made up interaction schemas
also presented their creators with some breakdowns in their
mental models when they discovered that the interactive lamp did
not corresponded 100% to their formulated ideas of how it
worked. One example of this kind of breakdown was that it was
rather confusing for one of the subjects who had sought of the
interactive lamp as a candle when it did not started to glow
brighter again (like a normal candle) when the hand was removed
after an attempt to smother it out by placing his/her directly above
and in direct contact with the lamp.
4.2 Technically driven interaction schemas
Three of the subjects started out their exploration of how to
interact with the lamp from a highly technical point of departure.
They lifted up the lamp, looked underneath it, looked inside it,
and tried to see where different sensors or buttons could be placed
on the lamp. When they looked from above straight down into the
inside of the lamp they could see the two photo sensors placed
beside the light bulb. However, no one of these subjects
recognized the sensors as photo sensors and could thus not find
out what the sensors would react to. However, after having found
out that there were in fact sensors placed inside the lamp they
We could also observe some quite dramatic, affective or
emotionally driven approaches to the exploration of the
interactive lamp. E.g. one of the subjects in our experiment started
his/her exploration of the lamp very explicitly by smashing his/her
hand as hard as he/she could against the table top on which the
lamp was placed. This caused the lamp to literary jump up a
couple of centimeters above the table, but it did not, as a big
disappointment to the person trying out this affective interaction
technique, switch the lamp on. When the lamp did not reacted at
all to this quite bold gesture the subject started to laugh and then
he/she leaned over the lamp to see if it was damaged after this
rather bold “attack”. However, the leaning over the lamp switched
it on and this in turn resulted in another affective response from
the subject. This time, however, the response was calmer and it
was almost with a relief in the voice that subject said to the lamp:
–“So, so… that was not so bad… you glow now so everything
seems to be ok…”.
This behavior was both personally and technically related. The
somewhat bold expressive way of approaching the lamp is natural
for a person that can be said to have an extrovert personality.
Further on, the choice of smashing his/her hand on the table top
was not randomly chosen but was also motivated by the
individuals prior experiences of technologies that react to
shakings or loud noises. The follow-up discussion with this
particular subject also revealed that this person where used to
children’s toys (e.g. monster truck toys) that react upon loud
noises and the bumping into things, which makes the toy vehicle
go in an alternative direction.
This kind of dramatic and very emotionally expressive way of
interacting with a sensing system has also been reported by e.g.
[13] in their study of how people interacted with a tangible input
device called SenToy, a toy-like affective interface to a computer
game.
4.4 Graceful interaction in intelligent
environments
A completely different approach to the dramatic one presented
above was also possible to identify when observing several of the
subjects. After a while, when the subjects felt that that their
interaction with the lamp was stable, effortless and
straightforward 6 of the 8 subjects in our study started to fine-tune
their interaction schemas and their gestures as to make them more
graceful, sophisticated, and elegant. This kind of graceful
interaction was not primarily done for the purpose of interacting
with the smart object in front of them, but the gracefulness was
more closely related to their own appearance in the wider
intelligent environment they now considered themselves to be part
of. As such, the subjects added an aesthetic level to their
interaction schemas in that they did not solely wanted to find out a
straightforward functional way of controlling the smart object in
front of them, but they also wanted themselves to appear rational
and elegant in their body movements in relation to the observer of
this experiment, the video camera and other persons that might
observe them interacting with the smart object. Figure 5 illustrates
the hand of a user that even from the beginning of the experiment,
i.e. before even knowing how the lamp actually worked, made
various very gentle and smooth hand movements up and down
along the side of the lamp. This, almost with dignity or at least
with some respectfulness as expressed by slow and graceful hand
movements.
Figure 5. The smooth/gentle interaction technique
While this particular attempt made to address the lamp did not
work this subject continued with this gentle exploration of the
lamp until he/she finally understood how to fully control the
lamp.
It should be said though that these three categorizes of interaction
schemas as identified in our study are not mutually exclusive. Nor
are the categorizes general on a statistical level to anyone that
would approach an intelligent environment or an ambient device
due to the limited number of subjects included in our study and
due to the limited complexity of our intelligent environment
consisting of only one smart object. On the other hand, the three
categorizes presented above cover quite well the different
interactional aspects of our empirical data and this blend between
these categorizes also illustrate an important aspect of our study
on its own, i.e. that the subjects were faced with a very complex
task when set to freely explore this interactive lamp in which they
had to try out a quite large repertoire of different interaction
techniques, while simultaneously trying to interpret and make
sense of their own interaction with the sensing system as to extract
functional and valid system commands from their own body
movements and gestures. In facing this complex task it was
natural to the subjects to approach this task from several different
angles, e.g. relying on prior personal experiences of similar
objects, trying out various interaction metaphors, approach the
task from a technical horizon, and express his/her frustration over
this complex task as well as happiness and pride when the system
responded accurately to given commands.
5. DISCUSSION
So, what implications can be drawn from this rather limited
study? From our viewpoint we believe that the study has
highlighted both theoretical and practical implications, and in this
section we will try to address these different dimensions.
From a theoretical viewpoint the current movement towards
information technology explicitly designed to be highly embedded
in everyday physical objects present us with several issues that so
forth has been taken for granted in traditional usability guidelines
for interactive system design. One example is the traditional
usability concept of user interface visability. Here, any kind of
visual user interface is, by purpose, hidden to the user. Further on,
and highly related, these smart objects clearly have some
affordances [12] that the physical form can communicate quite
well (e.g. that the smart object used in our experiment was a lamp
that quite certainly would be possible to switch on and off. At the
same time, this physical form provided no further guidelines or
help functions that could guide the user in figuring out exactly
how to gain access to this functionality. Finally, as for today, most
of the ambient objects available today are not designed according
to any known or traditional interactional conventions [9]. Instead,
each user is left rather alone and has to experientially go about
and explore how they can get in control.
From a more practical, or design-oriented viewpoint our study
suggests that it might be a good idea if not only the physical form
of the smart object could reveal its functional affordances, but
also if the digital part of the object could reveal its interaction
modality affordances and provide the user with some feedback
and clues for how to continue a more well-informed exploration
of an, although well hidden, user interface.
Another design implication from our study is that it might be a
good idea if the intended interaction schema for a smart object
was related to a couple of the most obvious interaction metaphors
of their intended users. E.g. as our experiment shown the subjects
tried to understand the lamp as a fire, a candle, or a volcano.
These different associations were probably not randomly chosen
by the subjects but instead probably highly related to the fact that
they interacted with a lamp with a light bulb that can shine very
bright and also become quite warm if the light is on for a couple
of minutes. If a design would support a number of such obvious
mental models it might be more likely that the user would
understand how to interact with it in an even shorter time.
In our experiment we could also notice how every subject that
took part in this experimental study approached the lamp by
relying on several different senses. They touched the lamp, lifted
it up, tilted it, tried to warm it with their hands, screamed at it,
clapped their hands, etc. In a sense they all had a very tangible
and sense-based approach in their exploration of the lamp. This
rimes well with the following statement made by Gershenfeld [4]
in his book “when things start to think”:
“we need to be able to use all of our senses to make sense of the
world” [4, p. 93]
What was clear though was that while the subjects were highly
involved in their exploration of the interactive lamp there was at
the same time something quite paradoxical about the subject’s
own immediate interpretations of what actually triggered any
system event. In one sense all subjects were highly aware of the
fact that some short sequences of body movements triggered a
specific system event, e.g. one user discovered that the lamp
switched on when she clapped her hands quite hard just above the
lamp. Thus, whenever the lamp by mistake was switched off while
she was struggling with learning how to control it she returned to
this particular sequence for switching the lamp on again. This, we
label repetition of successful interaction schemas. On the other
hand, while all subjects consciously relied on this repeated
interaction technique for going back to a somewhat safe starting
point for further exploration of the interactive lamp they were at
the same time totally unable to cut out a specific sequence of their
own body movements which had triggered a response from the
system and reuse such a sequence as a valid system command. We
label this the interaction continuum problem, i.e. the users
problem of doing in situ extraction of a particular short sequence
of his/her own body movements, which has resulted in an
unintended system command, from a longer session of bodily
interaction, and then immediately adopt this particular sequence
as one possible way of issuing a more specific system command.
Although the subjects were quite fast in getting a good
understanding of how to interact with the lamp we think that this
reliance on repeated interaction in combination with the
interaction continuum problem prevented them from an even
faster learning cycle and in their development of functional
interaction schemas.
Another reflection we have had after this study relates to the
observed gracefulness of the subjects interaction with the
interactive lamp. This gracefulness might be possible to analyze
even further e.g. if interpreting the subjects graceful interaction
with the lamp from the viewpoint of Goffman’s [5] theories of
human public behavior as everyday social performances on a
stage with an audience populating the intelligent environment
who are watching and judging every action taken by the single
individual. Given this perspective, a single individual interacting
with a smart object is not isolated, but instead part of a larger
social interplay in which it becomes important to the individual to
appear rational and aesthetical in front of the system and in front
of any other person that might be present in the same intelligent
environment.
The term “graceful interaction” has been used before as a general
guiding idea for design of dialogue systems and [6] describes that
“graceful interaction” (from a system perspective):
“involves dealing appropriately with anything a user happens to
say” [6].
Related to the area of intelligent environments, and given a user
perspective rather than a system perspective, a slight modification
to this paragraph in the form of a paraphrase might be possible to
construct as follows:
Graceful interaction in intelligent environments is about:
“dealing appropriately with anything a system happens to do”
i.e. to always, whatever the response (or lack of response) from
the sensing system is, act in a way that could be interpreted as
rational by others while at the same time serve as a system
command. According to this idea we define “graceful interaction”
as:
“any way in which a person interact with an intelligent
environment that is both effective and effortless to him/her while
at the same time appearing to be rational and elegant to anyone
else observing him or her while interacting with the system”.
Another aspect of the gracefulness in the subject’s interaction
schemas has to do with sorting out actual attempts made to
execute system commands or address the sensing system from just
any other kind of non-reflected hand movements. The acting out
of really explicit and graceful hand movements can be a clear
signal to both the system and to other persons nearby in that it
communicates that this specific gesture is purposeful and directed
towards the system.
6. CONCLUSION
In this paper we have presented an empirical study of how people
make sense of an everyday sensing system that might be a natural
part of a more complex intelligent environment in a nearby future.
In our empirical study we collected video-recorded data on how 8
subjects explored an interactive lamp. In our study we could
observe how the subjects quickly developed different personal,
technical and dramatic/affective interaction schemas as different
ways of exploring the functionality of the lamp and how they then
fine-tuned their interaction schemas to effectively interact with the
lamp (i.e. switch on and off the lamp and dim it/brighten it).
In this paper we have further described how we, during our study,
also could observe how the subjects, in their fine-tuning of their
interaction schemas, also adjusted their overall performance in the
intelligent environment and how they developed a meta-level
interaction schema that we have chosen to label “graceful
interaction”.
Given these empirical observations we have in this paper also
pinpointed some theoretical and practical implications in relation
to design of everyday intelligent environments.
It should be said though, that the interaction study conducted have
some limitations. The study conducted was quite small in that the
experiment only included 8 persons. It was also limited in terms
of complexity of the intelligent environment in that our study only
covered one individual’s interaction with one smart object, i.e. the
interactive lamp. While these two factors can be interpreted as a
methodological problem we have in our project instead viewed
this as a strength in that we have been able to focus on very
detailed and particular aspects of these persons different, and
highly personal ways of making sense of a sensing system, and we
have been able to look closely into individual interaction schemas
and how such schemas develops around one single smart object.
As such we believe that our study has been valuable since it has
provided us with the insights reported in this paper concerning
e.g. various interaction schemas, the value of openness in design
of intelligent environment, and knowledge on how the physical
form of a smart object can communicate some interaction related
affordances to its user.
The next step in our research project will be to take these valuable
experiences from this initial interaction study as a point of
departure for conducting a bigger comparative study in which we
will compare whether there are any cultural differences between
interaction schemas in Sweden vs. in the US. A similar
experiment as the one reported in this paper will be conducted in
which we plan look into questions like e.g., what kind of mental
models were used to make sense of the sensing system? Were
there any significant differences in how the users approached the
system in Sweden vs. in the US?
To address the two limitations to this reported study as discussed
above we also plan to extend our study in terms of technical and
functional complexity when it comes to the set up of the
intelligent environment. Thus, a second direction for our future
work includes the realization of a more complex intelligent
environment consisting of both several smart objects as well as
tight integration between these objects. Here, we are also planning
to set up this environment in a public place as to allow for several
persons to simultaneously interact with this environment and
allow for these persons to observe and affect each other while
interacting with the intelligent environment. In more concrete
terms this public interactive environment will be done in tight
collaboration with two artists and we plan to design some kind of
public, multi-user, and gesture-controlled light installation as part
of this years Umeå Autumn Light Festival in Umeå, Sweden [18].
In this forthcoming study we will specifically look into questions
related to the social development and use of cross-device
interaction schemas, and how such schemas are applied, finetuned, and exchanged for effective and graceful interaction in and
with intelligent environments.
7. ACKNOWLEDGEMENTS
Thanks are due to the 8 subjects that took part in this experimental
study. Thanks are also due to prof. Victor Kaptelinin and prof.
Erik Stolterman for their valuable comments on this project.
8. REFERENCES
[1] Benford, S., Schnädelbach, H., Koleva, B., Anastasi, R.,
Greenhalgh, C., Rodden, T., Green, J., Ghali, A., Pridmore,
T., Gaver, B., Boucher, A., Walker, B., Pennington, S.,
Schmidt, A., Gellersen, H., Steed, A. (2005) Expected,
sensed, and desired: A framework for designing sensingbased interaction, ACM Transactions on Computer-Human
Interaction (TOCHI), Volume 12 Issue 1, March 2005
[2] Bellotti, V., et al. (2002) Making sense of sensing systems:
Five questions for designers and researchers, In proceedings
of CHI 2002, p. 415-422, Minneapolis, 20-25 April, 2002.
[3] Bullivant, L. (2005) 4dspace: Interactive Architecture,
Wiley-Academy, UK.
[4] Gershenfeld, N. (1999). When Things Start to Think, Owl
books, New York.
[7] Inhabitat – Future-forward design for the world you inhabit,
http://www.inhabitat.com/
[8] Interactive Architecture dot Org
dhttp://www.interactivearchitecture.org/
[9] Lund, A. & Wiberg, M. (2004) Ambient Displays Beyond
Conventions, Presented at the "Designing for attention"
workshop at HCI 2004, The 18th British HCI Group Annual
Conference, Leeds Metropolitan University, UK 6-10.
[10] Mathmos Airswitch, http://www.mathmos.se/
[11] McCullough, M. (2004) Digital ground: Architecture,
Pervasive Computing, and Environmental Knowing, The
MIT Press, Cambridge.
[12] Norman, D. (1988) The Design of Everyday Things,
Currency doubleday, New York.
[13] Paiva, A., Andersson, G., Höök, K., Mourão, D., Costa, M.,
Martinho, C. (2002) SenToy in FantasyA: Designing an
Affective Sympathetic Interface to a Computer Game,
Personal and Ubiquitous Computing, Volume 6 Issue 5-6.
[14] Streitz, N., Magerkurth, C., Prante, T., Röcker, C. (2005)
From Information Design to Experience Design: Smart
Artefacts and the Disappearing Computer, Interactions, JulyAugust, 2005, p. 21-25.
[15] Streitz, N., Prante, T., Röcker, C., Alphen, D., .Magerkurth,
C., Stenzel, R., Plewe, D. (2003) Ambient Displays and
Mobile Devices for the Creation of Social Architechtural
Spaces: Supporting informal communication and social
awareness in organizations. In: O´Hara, K. et. al. (Eds.),
Public and Situated Displays: Social and Interactional
Aspects of Shared Display Technologies, Kluwer. Pp. 387409.
[16] Streitz, N., Röcker, C., Prante, T., Alphen, D., . Stenzel,
R.,Magerkurth, C. (2005) Designing Smart Artifacts for
Smart Environments. IEEE Computer, March 2005, p. 4149.
[17] Tolmie, P., et al. (2002) Unremarkable Computing, In
proceedings of CHI 2002, p. 415-422, Minneapolis, 20-25
April, 2002.
[18] Umeå Autumn Light Festival, http://www.umea.se/
kulturfritid/parknatur/umeahostljusumeaautumnlight20
05.4.3aa33d98107237db7c67fff1204.html
[5] Goffman, E., (1990) Presentation of Self in Everyday Life,
Paperback, Penguin Books Ltd, UK, 1990.
[19] Weiser, M. (1991). The computer of the 21st century,
Scientific American, 265(3):66-75, September 1991.
[6] Hayes, Philip J., Reddy, D. Raj (1983): Steps Toward
Graceful Interaction in Spoken and Written Man-Machine
Communication. In International Journal of Man-Machine
Studies, 19 (3) p. 231-284.
[20] Weiser, M., Gold, R., Brown, J. S. (1999). The origins of
ubiquitous computing research at PARC in the late 1980s.
IBM Systems Journal, 38(4):693-696.