INVESTIGATION OF HUMAN AFFECT DURING ELECTRONIC COMMERCE ACTIVITIES
P Goulev, E Mamdani
Imperial College London, UK
ABSTRACT
With the development of e-commerce an ever
increasing number of customers shop online,
interacting with their personal computers more
often and for longer than with human clerks.
However a major limitation of today’s technology
is that machines do not have the means of
observing and interpreting the user’s response to
the service provided. In particular, we look at
users’ emotional responses. This paper presents
an evaluation of the use of skin conductance
measurements to this end, using a wearable
sensing device called AffectiveWare. The
investigation was carried out as part of a joint
project between the London Science Museum’s
Live Science program, and Imperial College
London. The aim is to discover more about the
role of affect in our interaction with computers.
computing (6), the main goal of which is to study
and simulate social interaction between people
and their computers.
However a major limitation of today’s computer
technology is that machines do not have the
ability to observe, interpret and deduce human
reactions. Here we focus on the measurement of
affect or emotional states (5). The concept of
sending affective information to a computer
system is explained in the next section. In the
remaining sections chapters the architecture for
the AffectiveWare system is described followed by
an overview of an experimental setup for
measurements of affective feedback of subjects
shopping for wine bottles via the Internet. We then
discuss some diagrams resulting from the
experiments and look at some experimental
difficulties encountered. Finally we present some
possibilities for future work.
THE AFFECTIVE LOOP
INTRODUCTION
With the development of e-commerce an ever
increasing number of customers shop online,
interacting with their personal computers more
often and for longer than with human clerks.
Evidence for this trend can be seen in the sharp
increase in Internet turnover (1) in recent years.
The main reason is the availability of detailed
information on many web sites. Often in stores the
personnel’s knowledge does not cover some
aspects of particular products (2). Especially in
large superstores when a lot of products are on
sale, the customer can only avail themselves of
the information printed on the packaging to make
their choice.
In contrast web based software can easily provide
comparisons between different parameters of
similar products (3). An advantage technically
based on the searchability of online databases.
Most likely the important field in the item record is
the price. Often the value of the item is the
customer’s major reason for purchase.
An advantage human sales assistants have is that
they are able to recognize reactions in their
customers relatively easily by means of body
language, voice intonation, facial expression,
overall behaviour and so on. This problem has
been of increasing interest in the area of affective
Traditionally the study of Human-Computer
Interaction (HCI) has focused on the interaction
between a human and a personal computer (Fig.
1), but in the last few years many people have
started using Personal Digital Assistants (PDAs),
such as an electronic diary or mobile phone.
Normally interactions with a computer are limited
to using a mouse and keyboard as input and
speakers and a video monitor as output.
Figure 1: AffectiveWare dataflow diagram
Interactions with the PDA are even more
restrictive. Typically the user can use just a few
keys or touch the screen in a small number of
predefined areas, making working with PDAs
even more limited. Given that they don’t have
good quality sound and usually have a screen
with limited colours, the interaction is often less
satisfying for the user.
We want to improve this situation by introducing
an additional information flow generated by the
human body (Figure. 1, see the light arrows).
Sensors can be placed near or on the body which
are able to detect physiological changes during
the human-computer interaction (7).
Sensors should the constructed in such a way that
they will not obstruct the usage of the computer
itself. An additional constraint is the safety of the
subject using the proposed system. The devices
attached to the body cannot be connected
electrically with the computer because typically it
is connected with the main power supply (8).
Unfortunately such sensors often generate an
analogue signal. This means that physiological
data should be coded in a digital format that can
then be easily transferred to the computer or the
PDA. The affective coder shown in Fig. 1 has to
use a universal (device independent) coding
protocol which can be interpreted by the large
variety of computer devices currently available on
the market.
AFFECTIVEWARE ARCHITECTURE
The functionality of the AffectiveSensors and
AffectiveCoder were explained already. In the
next few subsections more detailed information is
given about the rest of the components.
AffectiveData
In the last few years a new standard for
representing data called eXtensible Mark-up
Language (XML) has been broadly implemented
(9). It has been chosen as the base of
AffectiveData which is a data format that
represents the physiological readings that are
passed from the AffectiveCoder to computer. An
example of AffectiveData packet is shown in Fig.
3.
<AffectiveWare>
<channel
name=“SkinConductance"
reading=“124"
value=“254"
dimension=“Siemens“
connector=“Plug" />
<channel … />
</AffectiveWare>
Figure3: Example of AffectiveData packet
The proposed dataflow is achieved with the
architecture shown in Fig. 2. The information
coming via the sensors is coded and transferred
to the computer using the AffectiveLink. Software
running on the computer called AffectiveMonitor
receives the data and stores it in AffectiveSQL
server. The AffectiveAnalyser checks the pattern
of the signal every second and deduces the user
mood. There is a Java based application interface
called AffectiveAPI that can pass the mood to the
application running on the PC/PDA.
Figure 2. AffectiveWare architecture.
The top three components are hardware devices
built especially for the experiment in this article.
The bottom four are software components (Fig. 2).
The first attribute, ‘Name’, shows which
parameters of the human body have been
monitored. The next attribute shows the actual
reading. The ‘Value’ is the reading according to
the International System of Units and ‘Dimension’
is the name of the Unit measured, in this instance
‘Siemens’. The term ‘Connector’ shows in which
channel the signals are operating.
<complexType name="UserMoodType">
<sequence>
<element
name="MoodNow"
type="MoodType"
minOccurs="1"
maxOccurs="unbounded
/>
<element
name="MoodHistory"
type="MoodType"
minOccurs="0"
maxOccurs="unbounded"
/>
</sequence>
</complexType>
Figure 4: XML definition of UserMood
AffectiveLink
The AffectiveLink is based on RS 232, a
widespread communication protocol supported by
most computers. The two media used in this
research are:
Fibreoptics: where a transfer is made with
RS323 coding via electrically isolated fibre optics.
The advantage of this media is the low noise and
virtually no loss of packets
Bluetooth: which is a popular radio transmission
based protocol. The advantage here is that PDAs
and sophisticated mobile phones, which used
Bluetooth themselves, can also use the
AffectiveData.
AffectiveMonitor
AffectiveMonitor is Java based program which
monitors the input through a serial interface of the
PC/PDA. It is able to recognise the AffectiveData
packet and to extract the reading. If a valid
reading is found the program inserts the reading
in the database for further analysis.
reading for SkinConductance and Battery, one
table for personal information called “UserInfo”
and a table for the predicted user mood called
“UserMood”.
AffectiveAnalyser
AffectiveAnalyser is a Java based program which
checks the AffectiveSQL every second. When it
discovers that a new reading is stored a pattern
recognition algorithm is executed. It tries to
deduce the user mood based on the last readings.
The result is stored back in the database in the
“User Mood” table.
This program also checks the level of the battery.
This is important because if the batteries are
discharged the reading will be wrong.
This program has been under constant
development from the beginning of this research
and will be improved when more data from
different experiments are collected.
AffectiveAPI
AffectiveAnalyser is a Java based program which
allows software modules written by other
developers to read the results of the database.
The scheme of XML data packet which
AffectiveAnalyser produces is shown on Fig. 4.
The architecture could be viable even without this
module. This is due to the fact that the data stored
into “UserMood” table of AffectiveSQL can be
accessed easily via an ODBC connection.
Figure 5: Screenshot of the AffectiveMonitor
This program also displays the readings on the
screen numerically and graphically as shown in
Fig. 5. On the X axis is the time and on the Y axis
the last few readings are represented. The graph
is shifted towards the left to allow the last reading
to be displayed. The interface is designed in a
way that allows a video recording of the panel with
the current results to be taken during the
experiments.
AffectiveSQL
In our research we decided to use a Linux
Mandrake 9.0 based server because of the high
price of commercial ones. To overcome license
limitations for SQL server we choose MySQL
3.23. We have one database called “affective”
with four tables. Two further tables store the
SKIN CONDUCTANCE
Given that the experiments and the use of this
equipment will take place in a public space the
sensors could not be put on places that are not
accessible without removing their clothes.
Furthermore it will be better that the sensors are
placed (and removed) on the human body for a
couple of seconds.
Skin conductance is a well-established method of
measuring physiological changes that have been
linked to emotional responses (7). In this instance,
sensors were devised that could easily be
attached to the hand of the user leaving the
subject free to use the computer in a normal
fashion. Two different devices were developed
and evaluated in this project.
Affective rings are placed on the fingers and are
the mouse. This solution gives us independence
of the two modules inside the mouse.
Figure 6: AffectiveRings
able to read physiological changes continuously.
The disadvantage of the rings is that they have to
be attached before any human-computer
interaction and therefore not necessarily practical
for everyday use (Fig 6).
Affective mouse which is able to measure skin
conductance without disturbing the user was also
created (Fig. 7).
AffectiveRings
The sensors shown in Fig. 6, consists of elastic
tape containing electro-conductive material. Two
of them should be are placed on the index and
fourth fingers. The sensor’s electronics are
attached to the wrist of the wearer, and are similar
in weight to an ordinary watch. A subject’s hand
wearing the Affective Rings is shown in Fig. 6.
There is no internal power supply for this device.
Due to its low consumption it is powered directly
from the AffectiveCoder.
Figure 7: AffectiveMouse
EVALUATION SETUP
As mentioned in the abstract the evaluation took
place in The Science Museum London. The
location was chosen firstly because the Science
Museum has a good age distribution throughout
its visitors. Furthermore, only volunteers were to
participate in this experiment, which would create
an additional motivation for them to browse the
web site for longer. Additionally the Live Science
area in the new Welcome Wing of the museum
has an exceptional ambiance allowing people to
relax before and during the experiment.
Finally the Science Museum is the perfect place
for measuring skin conductance as the humidity
and temperature are controlled and stable.
AffectiveMouse
The AffectiveMouse is shown in Fig. 6 and it looks
like an ordinary mouse. The difference is that it is
covered with electro-conductive layer. Skin
conductance is measured between the left or right
button and the body of the mouse.
The electronic module inside the mouse is the
same as that in the wrist box in the AffectiveRings
scenario. The cable which in on the front of the
mouse on the picture is the connection with the
AffectiveCoder. The mouse itself uses a wireless
connection to transfer position, clicks and so on,
to the computer.
The power supply for the skin conductive sensor
comes from the coder as in the previously
described sensor. The radio track of the mouse is
supplied by set of researchable batteries inside
Figure 8: Evaluation setup diagram
Fig. 8 shows a subject working with an ordinary
PC. The camera on the top of the monitor records
the subjects face. A camera on the table is
recording the mouse. The signals from the screen
and the AffectiveMonitor are combined with the
signal from the two cameras by a QuadVideo
mixer. This ensures that all the images are
synchronised perfectly. This is recorded on a DVD
enabled computer.
The real the computer system is shown in Fig 9.
On the table is the PC which the subject uses and
in front is the videoQuad which is positioned over
the DVD enabled PC.
time an observation of the computer screen helps
to discover the response of the subject to the
software behaviour.
Figure 11: Age distribution
Figure 9: Real evaluation setup
Diagrams of readings of two different subject
wearing different sensors are presented below.
Record of subject using the AffectiveRings is
shown on figure 12. At one point on the computer
screen appears a new window on. At the same
time, the affective monitor shows a sharp change
of the skin conductance. After that follows a
relaxation period.
RESULTS
In the whole study 86 people took part. Due to the
fact that not all of them permitted us to record
their session on tape 74 videos are valid. A
photograph of people using the system during the
evaluation is shown on Fig.10.
Figure 12: Use of the AffectiveRings by S102
Record of subject using the AffectiveMouse is
shown on figure 13. The skin conductance
readings drop significantly when an error
message that pops up on the screen.
Figure 10: People using the system
The total length of the video is 16162 seconds, a
little bit less than 5 hours. The male subjects (43)
were more than the female (31) but this difference
is not so big. The majority were aged between 20
and 60, though with some exceptions. The age
distribution is wide (Fig. 11), which adds to the
validity of the experimental results.
To analyse the reading is necessary to compare
the video record of the subject’s face with the
changes of the skin conductance. On the same
Figure 13: Use of the AffectiveMouse by S114
The readings shown on Fig. 13 are not continuous
like these shown on Fig 12. This is due to the fact
that the subjects do not keep their hands on the
mouse at all times. Despite this, the results are
still interpretable using interpolation techniques
known from the mathematics.
person using the mobile) and even
AffectiveRoom (a whole set of sensors).
DIFFICULTIES
It would also be desirable to repeat this
experiment using more and different sensors in
search for better defined correlations.
We were very keen to include measures of
personality in the questionnaire that we devised
for subjects. However there were some
constraints which are explained below.
Qualifications, which we didn’t have, are
required by law in order to conduct personality
tests even on volunteers.
Performing even very standard tests such as the
Eysenck Personality Test cost money, so no
personality measures were unfortunately taken.
Collecting personal data and especially Video
recording can cause problems with the Data
Protection Act, so consent forms detailing all of
the parts of the experiment are essential.
Automatical recognition of face emotions by
computers is not possible yet. Thus the video
analysis and matching is very labour-intensive for
the researcher and takes a long time.
Despite the mentioned problems there are still a
lot of investigations to be done.
an
This research will also pave the way for
theoretical improvements in emotional models.
Indeed, it is hoped that improvements might be
made using the data obtained from this
experiments conducted at the London Science
Museum.
ACKNOWLEDGMENTS
The evaluation had been organised by Sabiha
Foster and Mark Witkowski. The hardware has
been constructed in the laboratories at the
Department of EEE under supervision of Neil
Todd. The editing of this article was made by
Kaveh Kamyab.
The research of the hardware and software has
been kindly supported from two IST projects –
SAFIRA(IST 1999-11683) and ISIS(IST 200237253) .
CONCLUSIONS
REFERENCES
Evaluation of the AffectiveWare system has taken
place since the time at the Science museum.
Preliminary results, such as the screenshot
shown in Fig. 5, reveal that the system is
effective.
The AffectiveRings and AffectiveMouse worked
without problems for two weeks with plenty of
subjects, although some of the subjects scored
out of the range of our skin conductance
equipment.
The research confirms and explores further the
connection between skin conductance and
human-computer interaction already shown at the
University of Huddersfield (7).
FUTURE WORK
In future it would be highly desirable to study a
relatively big numbers of subjects and to analyse
their data carefully. It is very likely that
correlations between emotional expressions of the
subject and changes in skin conductance will be
discovered as a result of this work. There is also
potential for the development of new sensors such
as an: AffectiveChair (which measures the heart
rate of the person sitting in it), an AffectiveMobile
(which could sense the skin conductance of the
(1) M O’Hara, Virtual Internet triples turnover,
Guardian 15.09.2000
(2) Riseberg J., Klain J., Fernandez R., Picard R.
W.(1998), Frustrating the User on Puspose, ASM
CHI 98, page 227-228, ISBN 1-58113-028-7
(3) Kelkoo Web Site, http://www.kelkoo.co.uk
(4) J Tanenbaum, “Marketing relationships, like
others, require trust”, 20.08.2001, Washington
Business Journal
(5) Picard, R. W. (2000) "Toward computers that
recognize and respond to user emotion." IBM
System Journal 39(3 & 4).
(6) Picard, R. W. (1997). Affective computing.
London, The MIT press.
(7) Ward R D, Marsden P H, Cahill B and Johnson
C (2001). Using skin conductivity to detect
emotionally significant events in human-computer
interaction. Procs IHM-HCI2001, II, 25-28.
(8) W Hofheinz, Electrical Safety – Ungrounded
Power Supply Systems in Medical Sites, Hospital
Engineering and Facilities Management, Business
Briefings, London, 2003
(9)
Extensible
Markup
Language,
http://www.w3.org/XML
(10) Cacioppo, J. T. and L. G. Tassinary. (1990).
"Inferring
psychological
significance
from
physiological signals." American Psychologist
1(45): 16-28.