A Study of Non-verbal Expressions in a Computer-Mediated
Communication context (CMC)
- Focusing on the mouse behaviors in the web contents -
Shin-Young, PARK*, Akira HARADA**
*University of Tsukuba Major of Kansei, Cognitive and Neuroscience
1-1-1 Tennodai Tsukuba-shi Ibaraki 305-8574 JAPAN, emosign@geijutsu.tsukuba.ac.jp
** University of Tsukuba Institute of Art & Design
1-1-1 Tennodai Tsukuba-shi Ibaraki 305-8574 JAPAN, aharada@kansei.geijutsu.tsukuba.ac.jp
Abstract: This paper presents an effective and direct method to study Non-verbal Cue in computer-mediated
communication context (CMC).
The real digital communications such as the Internet are used to the logical information converted easily into a
digital signal for the communication media. However, a human communication is composed of not only logical
information, but also sensitivity-like non-verbal one. Non-verbal information like the human voice, gesture, and
emotion rather than verbal one is more useful to understand the user's inclination exactly. However it is quite difficult
to catch the meaning of non-verbal information without using a specially purposed device such as the motion capture.
Although the devices can use for the experimental purposes of understanding the meaning of non-verbal
information, the confidence of results is not enough because the devices give an invisible pressure to a testee wearing
the equipments.
In this paper, we develop a mouse-tracking system, namely Recoding tool Of Mouse trace In The Web (ROMIT),
for the purpose of studying the non-verbal information on CMC. ROMIT supports on-line communication between
communicators through the Internet. It can provide more exact remote users' inclination for the system manager from
avoiding the invisible pressure on the users.
We investigate the user’s mouse behavior on the web by using ROMIT system, and our demonstration show that the
mouse behavior of each user is the effective and direct method to understand the user's inclination. Then, we confirm
that the mouse behavior can be used as the medium for transferring the KANSEI or non-verbal information on CMC.
Key words: non-verbal Information, mouse behavior, ROMIT, Computer-Mediated Communication
1. Introduction
1-1 Motivation and Objective
One of the important communication technologies is Computer-Mediated Communication (CMC) such as the
Internet communication, ICQ, peer to peer communication, etc, which will become a common communication
media and affects more and more human communication as CMC is popular to the communication method among
people. Moreover, The human communication field is expanded into not only the physical space but also the
virtual space by using CMC since it does not need to the physical space and the specific companion to talk.
However, most of previous researches show that CMC is an uncertain method in human communication because it
is not able to represent non-verbal information, i.e., personal background, personality, and feeling, which are the
important factors in the human communication field.
In spite of the uncertain communication method, CMC will become important communication media and will
substitute human communication gradually by the progress of CMC technologies. Even if CMC can express the
non-verbal information somewhat, it is still not enough to express the niceties of feeling or emotion exactly.
We set up several questions to study non-verbal communication on CMC as follows.
•
Is not possible to express non-verbal information in CMC?
•
Does understanding non-verbal information help us communicate well?
In this paper, we provide the efficient method for developing CMC tools as near as human communication from
converting non-verbal information to visible one.
1-2 The effects of our study
How to convey non-verbal information among communicators on CMC is greatly important to develop CMC
tools and realize human communication on CMC. We present the efficient and direct way of analyzing the user’s
inclinations and behaviors on the web from capturing mouse behaviors, and also show that the captured mouse
behaviors provide the useful way of analyzing the non-verbal information on CMC.
2. Non-verbal Information
2.1 Definition of non-verbal information
Non-verbal communication is language-independent communication, for which conveying messages with using
unconsciousness expressions to communicators instead of using verbal expression. The unconsciousness
expressions, i.e., gesture, voice, body language, etc, are referred to as non-verbal information
Among many factors contributing to non-verbal communication are sending and receiving ability and accuracy,
perception of appropriate social roles, and cognitive desire for interpersonal involvement or assessment.
Difficulties may arise if communicators are unaware of the types of messages they are sending and how the
receiver is interpreting those messages. Discrepancy may also arise if the sender’s message does not fit the
receiver’s perception of social norms for the particular situation. Although these difficulties and discrepancy are
occurred in human communication, the non-verbal information gives cues for solving the difficulties and
discrepancy of communication since it has many common components and meaning in a group with the common
social, situational and cultural background
Burgoon (1985) shows that the factors of non-verbal information providing clarity or contradiction for a messages
being sent are
•
Motion of body: defined as a movement in body area such as head, face, shoulder, hands, or feet.
•
Voice factors: defined as a voice without a language expression such as voice tone, speed, silence, etc.
•
Cosmetic factors: defined as the appearance of a person such as costume, cosmetic, etc.
•
Relation factors: defined as the contact of communicators such as handshake, embrace, kiss, etc.
•
Geographical space factors: defined as the distance and position among communicators.
•
Time space factors: defined as the communication time.
•
Environment factors: defined as surroundings to affect communication including the geographical space and
time space factors
2.2 The characteristics of non-verbal information on CMC
Communication is a dynamic process with the interacting components of sending, receiving messages.
Non-verbal communication regulates relationships between messages and meaning, and may support or replace
verbal communication. From the previous study (Birdwhistell, 1970), non-verbal communication occupies about
60-65% of human communication, which means that it is impossible to complete human communication without
non-verbal one. The comparisons of verbal and non-verbal communication are as follows.
•
An adult interprets messages more depending on nonverbal information compared with verbal one.
•
Children interpret messages more depending on verbal information.
•
Verbal communication is used to transfer the actual, concise, and reliable messages, while non-verbal
communication is used to provide accuracy, clarity, and contradiction for ambiguous messages.
Non-verbal information can be used to the following situations more efficiently; (Kurokawa 1994)
•
There is not appropriate verbal information to express the message.
•
There is difficult to express the message by only verbal information.
•
There is to emphasize non-verbal information.
•
There is to avoid the straightforward expression.
•
There is to emphasize the meaning of messages by supporting verbal information.
2.3 Aspects of non-verbal information in the neurophysiology
The properties of non-verbal communication can be also found in the neurophysiology.
Left brain
Right brain
Verbal
Nonverbal
Perceptual
Global
Simultaneous
Synthetic
Intuitive
Concrete
Nontemporal
Spatial
Syntactical
Linear
Sequential
Analytic
Logical
Symbolic
Fig 1 The functions of a right brain and a left brain
In the view of non-verbal information as the neurophysiology, a left brain operates the analytic, logical, linguistic
process of thinking, while a right brain operates the emotional, intuitive process of thinking as shown in Fig 1.
In general, humans are able to communicate complex emotion through behavioral cues and the ambiguous
nature of emotion creates a wide variety of possible interpretations for both sender and receiver. Non-verbal
information is the most important factor to concrete the meaning of ambiguous emotion or thought.
3. Non-verbal information in Computer-Mediated Communication (CMC)
3.1 The characteristics of CMC
Computer-Mediated Communication is a process of human communication via computers, involving people,
situated in particular contexts, engaging in processes to shape media for a variety of purposes. In general, the term
computer-mediated communication refers to both task-related and interpersonal communication conducted by
computer. This includes communication both to and through a personal or a mainframe computer, and is generally
understood to include asynchronous communication via email or through use of an electronic bulletin board;
synchronous communication such as "chatting" or through the use of group software; and information
manipulation, retrieval and storage through computers and electronic databases.1
CMC becomes more popular communication form, and replaces the traditional face-to-face communication
fields gradually. In CMC, there consists of four major parts as follows,
Table 1 Examples of Computer-Mediated Communication (CMC)
Synchronism
Purpose
Style
Transmission of Message
Electronic –mail (e-mail)
Asynchronism
Information exchange
Text
One - to - one
One - to - many
Electronic bulletin board
Asynchronism
Information exchange
Text, discussion
One - to - many
Chat
Synchronism
Social interaction
Text
One- to- server
MUD
Synchronism
Social interaction
Text, graphics
One - to - server
The properties of CMC can be defined as follows,
•
There does not restrict geographical and time space.
•
There is the limit of expressing non-verbal information.
•
There is based on basically nameless.
•
There is based on face-to-monitor communication instead of face-to-face communication.
•
There is based on the literal communication instead of the linguistic communication.
•
There is to support to reutilize the information.
1
Pixy Ferris, “What is CMC?” (http://www.december.com/cmc/mag/1997/jan/ferris.html)
3.2 Expression of non-verbal information in CMC
One of the problems in CMC is a lot of limit of expressing non-verbal information. To solve the problem,
several approaches are tried in CMC, but they still depend on most of verbal communication formatted text to
express non-verbal information. The text communication is significantly difficult to express the internal feeling or
emotion since it does not include the non-verbal information such as gesture, tone of voice, motion of body, etc, in
the conveying messages. One of challenges to express the non-verbal information in CMC is to use the electronic
paralanguage such as emoticon or articon.
3.2.1 Electronic paralanguage (emoticon, articon)
At present, an electronic paralanguage, i.e., face mark, emoticon, articon, is widely used to express non-verbal
information in CMC. The electronic paralanguage is started to directly convey the ambiguous messages such as
feeling and emotion to other person on the computer network communication, which varies from the face mark
based on text format to the avatar supporting the virtual motion and human appearance as shown in Fig 2.
Strange to say, the usage of the electronic paralanguage is more widely used to the oriental people compared
with the western people. It may be caused by the thought of orient that the non-verbal communication is more
important method to transfer the message to others.
Face mark Of the e-mail
^^;
:-)
(^.^)
(>.<)
:-C
(_ _)
^o^
T_T
(^-^;)
Smile icon of the Instant messenger
Avatar of the e-mail application
Fig 2 Examples of electronic paralanguage
3.2.2 Non-verbal communication using sensor technology
The captured motion by using electronic equipments like a motion capture is more helpful to understand the
meaning of non-verbal information compared with using the electronic paralanguage.
The motion capture is the process of capturing the movement or gesture of a person and mapping it onto a
computer generated object. The motion capture can provide more real non-verbal information for others than the
electronic paralanguage, However it is difficult to adopt in the real computer communication with easy, and the
confidence of results is not enough because the devices give a strain to a testee wearing the equipments. Thus, it is
need to avoid the invisible pressure from users in order to get the confidence results.
4. Case study
4.1 The purpose of our study
From the previous studies, non-verbal information has an important function as much as verbal information on
communication. In current, several methods are used in computer-mediated communication (CMC). However,
most of expression of non-verbal information is based on literal, and then it has the limit of representing the
non-verbal information and difficult to express accurately and easily.
This paper presents an effective and direct method to support to understand non-verbal information (emotion,
sense. etc.) in CMC. We capture the mouse movement and obtain its position on the web contents. From the
analysis of the captured mouse movement, we show that the mouse behavior is one of methods to express
non-verbal information and can describe well the characteristics of user’s behaviors and inclinations.
To sum up, the purposes of capturing the mouse movement are
•
It can be used to express the non-verbal information on CMC.
•
It avoids the invisible pressure of a testee, and is easy to experiment since it does not need to the specially
purposed equipments like a motion capture.
•
The captured log data provides the easy analysis and comparison of verbal communication and non-verbal
communication.
In this paper, we focus on the communication between a web manager and remote users in the web, which is
one of examples of CMC. For the task of verification experiments of capturing the mouse movements, we analyze
the degree of interest about the web contents.
4.2. Development of Recording tool Of Mouse trace In The Web (ROMIT) with JAVA
We develop a mouse trace tool for studying the user’s behaviors on the web, namely Recording tool of Mouse
trace In The Web (ROMIT) with JAVA. Users communicate with a web manager on the Internet in the real time
by using ROMIT. The web manager records the log information periodically, which is the mouse position on the
web contents. The obtained log information is used to
analyze the user’s behaviors and inclinations on the web.
The ROMIT system is shown in Fig 3
Save of the Log data
IE
N4
4.2.1. Basic operations
Mozilla
ROMIT
is
based
on
a
client-server
network
communication. After an user is connected to a server
(web master or web manager) on the Internet, he sends
messages including the mouse position on the web
A different script according to a browser is prepared
Access
httpd(Web server)
contents and other extra information to server periodically.
The server receives messages consisted of the mouse
position, the current page of web contents, view time, etc,
from users, and he stores the messages in the log
management systems.
From the obtained information, the web manager can
Explorer user
Netscape user
Mozilla user
analyze the user’s behavior on the web by visualizing the
user’s mouse movements and positions on the web
contents.
Fig 3 Structure of a ROMIT
Collection of the log data
For the purpose of collecting the mouse position on the web browser, ROMIT is programmed by JAVA script
supporting on-line communication in the Internet, which ROMIT gets the mouse positions and other information
of each user and send the messages to the web manager periodically. The obtained information are
1. Time which road page by each user
2. X Coordinates of the mouse point
3. Y Coordinates of the mouse point
4. Mouse behaviors (move, click, release)
Transmission of the Log data
The obtained log data including several information is sent to a web manager through ROMIT.
Preservation of the log data
The web manager stores the log data in the log management databases programmed by PHP and Mysql by the
time order.
4.2.2. Display the obtained log data
ROMIT supports to display the obtained log data on the server side (web manager). ROMIT is designed to
operate the current popular browsers not only internet explorer but also Netscape. A kind of web browsers is
distinguished by the Java applet. In addition, for the purpose of reducing the overhead of the server, the period of
sending data can be set dynamically. In our experiment, we set to 0.1sec for the period of sending data from the
client to the server.
Log data : The obtained
IP Address
ID
130.158.244.210
130.158.081.192
130.158.098.192
130.158.244.192
3e18c32711856 2002-12-06
2002-12-0623:43:35
23:43:35/lunar/trace/main-ie.php
/lunar/trace/main-ie.php
3e18c4990dffb 2002-12-08
2002-12-0816:49:45
16:49:45/lunar/trace/main-mozi.php
/lunar/trace/main-mozi.php
3e18c4d15ed7e 2002-12-10
2002-12-1008:50:51
08:50:51/lunar/trace/main-n4.php
/lunar/trace/main-n4.php
3e1981b86d4dd 2002-12-10
2002-12-1022:16:40
22:16:40/lunar/trace/main-ie.php
/lunar/trace/main-ie.php
.
.
Time
①
②
③ ④ ⑤
3e1981b86d4dd move 114 231 212
3e1981b86d4dd move 270 221 256
3e1981b86d4dd move 350 373 114
3e1981b86d4dd move 450 394 127
3e1981b86d4dd move 551 453 165
3e1981b86d4dd move 651 334 127
3e1981b86d4dd move 751 586 239
3e1981b86d4dd move 851 345 234
3e1981b86d4dd move 951 235 211
.
.
Fig 4 Form of the Log data of two kinds
Page Name
log data is shown in Fig 4.
The log data consists of the
IP addresses of users, a
identifying
number,
the
① ID
mouse points on the web
② Position of the mouse
page, sending time, and the
③ Time which the mouse moved
④ X coordinates
⑤ Y coordinates
current
web
page.
The
mouse points of each user
includes the current position
of the mouse, time which
the
mouse
coordinates,
moved,
X
and
Y
coordinates.
4.3 Verification experiments
For the task of verification experiments of capturing the mouse movements, we analyze the degree of interest
about the web contents.
The testees for experimental studies
We investigate the mouse behaviors on 12 testees with an internet career from a year to over 5 years.
The testees are organized by a group of 8 males and 4 females, which the group is divided into 3 persons with an
internet career less than one year, 3 persons about one and two years, and 5 persons over 5 years.
Explanation of
an experiment
Maudsley
Personality
Inventory
Evaluation value
Presentation
of an
image stimulus
Logdata generation
of mouse coordinates
Server
Impression
evaluation to
an image
stimulus(SD)
Evaluation value
End
Fig 5 Experiment procedures
Experiment procedures (see Fig 5)
1.For
investigation
purpose,
test
web
pages
are
constructed by Maudsley Personality Inventory:
M.P.I (Question item 80).
2. Testees will surf the web pages consisted of dynamic
images (Flash pages) and static images.
3. The impression of the images is evaluated by measuring
the staying time of the mouse point on the image
(Semantic Differential enforcement using a smile
icon.)
Fig 6 The executive screen of the ROMIT (JAVA Applet)
4. Analysis and comparison of each testee’s mouse
behavior.
4.5 Experiment results
Users stay on the high impressive images for a long time.
We score the different point of each image by the degree of impressiveness. By measuring the staying time on
the images, we know that the staying time on the high scored images is longer than that of the low scored images.
As a result, we easily catch the user’s interest in which web contents from capturing the mouse points since they
stay on the high impressive images for a long time shown in Fig 6.
Common mouse behaviors on users are appeared.
Users with a major of art and design have common mouse behaviors, which the trace of mouse point lines
through a left down and a right upper on the web page, compared with others without the experience of art and
design shown in Fig 7. It is shown that an user’s life style and habit affects the mouse behaviors unconsciously.
Then, we easily catch the user’s behaviors in which web contents from capturing the mouse points showing the
user’s life patterns and habit which are the important factors consisting of non-verbal information.
Visualization of a group of users’
interest.
It is difficult to measuring of a group of
users’ interest and behavior because the
group interest is different from the
experimental environment and condition,
and also it is impossible to maintain the
same experimental condition for testees.
ROMIT supports a web manager to
communicate with several remote users in
the same experimental condition. Moreover,
visualization of mouse behavior of not only
a user but also a group of users is useful to
analyze the group of users’ interest with
Fig 7 The result of impression evaluation
easy.
Personality of users is appeared in the impression of image
The impression of images is various as much as the difference of user’s personalities. In spite of the difference
of user’s personality, we know that the common characteristics are appeared in the impression of images. An
introvert scores the high point to the images, while an extrovert varies the impression of images.
5. Conclusions and future works
This paper presents an effective and direct method to study KANSEI (non-verbal information) in
computer-mediated communication (CMC). The real digital communications such as the Internet uses the logical
information converted easily into a digital signal. However, human communication is composed of not only such
logical information but also sensitivity-like non-verbal one. Non-verbal information such as the human voice,
gesture, and emotion rather than verbal one is more useful to understand the user's inclination exactly. However it
is quite difficult to catch the meaning of non-verbal information. We develop a mouse-tracking system, namely
Recoding tool Of Mouse trace In The Web (ROMIT), for the purpose of studying the non-verbal information on
CMC. ROMIT provides more exact remote users' inclination and behavior in CMC for the web master by
avoiding the invisible pressure from users. Moreover, it is enable us to easily catch the user’s interests and
behaviors in web contents.
We experiment the user's inclination and behavior on the web by using the ROMIT system, which captures the
mouse movement in the web contents and visualizes the user’s behavior. From the analysis of the captured mouse
points, we confirm that the mouse behavior shows the user’s life style and habit which are the important factors in
non-verbal communication, and then it can be used as the medium for transferring the KANSEI or non-verbal
information on CMC.
References
•
Barker D., Carey M.S., Taylor R.G., “Factors underlying mouse pointing performance”, contemporary
Ergonomics, pp359-364 (1990)
•
Jacob Nielsen, “Designing Web”, New Riders Publishing (2000)
•
Kiesler,S., Seigel, J,. & McGuire, T, “Social psychological aspects of computer-mediated communication”,
American Psychologist, 39, pp1123-1134 (1984)
•
Kurokawa, “Non-verbal Interface”, Ohm, Inc (1994)
•
Shinyoung,Park, ChangWoo Pyo, Harada Akira, “The extraction of KANSEI information using the
mouse-tracking tool in World Wide Web”, Proceedings of the 4th Annual conference of JSKE, Tokyo,
146(2002)
•
Shinyoung,Park, Masakatsu Sakata, Harada Akira, “Development of the Communication Tool using the mouse
Log data”, Proceedings of the 50th Annual conference of JSSD, Tokyo, 124-125 (2003)
•
HeungRyong,Woo, “The Cognitive Graphic Thinking Processes of Idea Sketches by Two Brain Modes”,
Bulletin of 5th Asian Design Conference CD Rom (2001)