The Design and Implementation of a Multi-User
Interactive Public Display
by
Sonia Garg
Submitted to the Department of Electrical Engineering and Computer Science
in partial fulfillment of the requirements for the Degree of
Master of Engineering in Electrical Engineering and Computer Science
at the
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
1ay 2003
©Sonia Garg, MMIII. All rights reserved.
The author hereby grants MIT permission to reproduce and
distribute publicly paper and electronic copies of this
thesis and to grant others the right to do so.
Author ..........................................................
Department of Electrical Engineering and Computer Science
May 21, 2003
C ertified by ...
.........................................................
....................................................
Larry Rudolph
Tjesis Supervisor
A ccepted by .................
. .
......
........
. .............
........................................
Arthur C. Smith
Chairman, Department Committee on Graduate Theses
MASSACHUSETTS INSTITUTE
OF TECHNOLOGY
JUL3 02003
1
LIBRARIES
The Design and Implementation of a Multi-User Interactive Public Display
by
Sonia Garg
Submitted to the
Department of Electrical Engineering and Computer Science
May 21, 2003
in partial fulfillment of the requirements for the Degree of
Master of Engineering in Electrical Engineering and Computer Science
Abstract
This thesis explores a new approach to exchanging information using an interactive
public display. The system was designed to support multiple users in dynamic
environments, where people connect and disconnect to the displays at will. The key
goals were to ensure a seamless end user experience and provide the anonymous
exchange of information. Also, users have the option of customizing and
personalizing their interaction with the public displays. A prototype of the exchange
system was built to demonstrate a connection scheme for Bluetooth enabled devices.
The interactive display system can be applied and extended by various pervasive
computing endeavors.
Thesis Supervisor: Larry Rudolph
Title: Principle Research Scientist
2
Acknowledgments
I would like to thank:
Larry Rudolph for his inspiration and guidance throughout this project. He has
been a motivational advisor and I am grateful to have had the opportunity of
working with him.
John Ankcorn and Jorge Ortiz for their help in designing and developing the
Content Exchange system.
Jonathan Brunsman, Glenn Eguchi, Hana Kim, Kan Liu, Arjun Narayanswamy for
being fun and energetic officemates.
Members of the MEng '02 group for helping to initiate my involvement with the
Oxygen Research Group.
My parents, Gobind and Kamlesh Garg as well as my brother and sister, Rohit and
Nitasha Garg for their endless support and encouragement which have carried me
this far.
3
Contents
1 Introduction .....................................................................................................
6
2 Motivation and Background ............................................................................
9
2.1
Group Background ......................................................................................
9
2.2
Context-Aware Pervasive Computing Efforts.........................................
10
2.3
Efforts with Public Displays......................................................................
11
3 The Content Exchange .........................................
13
3.1
User Scenarios...........................................................................................
13
3.2
Design Goals .............................................................................................
14
3.3
System Architecture .....................................................................................
16
3.4
Functional Design ....................................................................................
20
4 Functional Im plem entation ............................................................................
26
4.1
Bluetooth Stack & Interfaces ...................................................................
27
4.2
CEA Discovery...........................................................................................28
4.3
Establishing a Channel with a CEA via Bluetooth ..................................
4.4
Security......................................................................................................34
4.5
Partial State of the System .....................................................................
36
5 Analysis & Extensions ........................................................................................
39
5.1
Tradeoffs with Bluetooth..........................................................................
39
5.2
User-Experience ......................................................................................
41
5.3
Pros and Cons of Custom izations.............................................................
43
4
30
5.4
Extensions to the CEA N etw ork ..................................................................
45
5.5
Location Aw are Technology .....................................................................
47
5.6
Multi-M odal Input...................................................................................
48
6 Conclusion ............................................................................................................
5
50
Chapter 1
Introduction
The focus of this thesis was to change the process of sharing and exchanging data by
creating a system in which users could interact with information shown on public
displays. A system called the Content Exchange was designed to demonstrate a
multi-user system that can perform assorted functionality based on a dynamic set of
attributes provided by the users. A prototype was implemented to define a
connection and resource negotiation protocol for Bluetooth enabled devices.
In the past 30 years, technological revolutions have redefined the way people think
and operate. With the birth of the Internet, information has been made widely
available. Databases, servers, and personal computers are filled with an abundance
of information. Knowledge portals and data repositories are one way to organize
and store such large amounts of information. In order to make the information
repositories useful, effective mechanisms for saving and retrieving data need to be
established. The key challenges in managing data and information effectively are
one, to identify the proper way to display the information and two, to devise the
correct scheme to transfer the information.
First, displaying information effectively depends on the content of the information
and the display devices available. Many research institutes and industry
6
corporations have moved towards using public displays to advertise products and
services, show press releases, and publicize events. These displays are primarily
used for the purposes of viewing static information.
Second, methods of exchanging information have drastically changed with the
emergence of the Internet. Network improvements and increased bandwidth have
led to the adoption of peer-to-peer file transfer systems and chat programs.
Exchanging information is an important element of creative interactive and
collaborative environments. A system needs to be established that will allow
multiple users to have similar interaction but where the information is on a public
or private display.
This thesis presents a system designed to change the paradigm for transferring data
into and out of an information repository. The idea is to combine a public display
with peer-to-peer file transfer. The result is a large-scale display that people can
use to download and upload content. The system is designed to account for dynamic
environments where users a free to move around. Additionally the system has
multi-user support so that viewers can interact with the system simultaneously.
Finally, the system allows users to personalize their experience and customize its
behavior to fits their individual needs.
A system called the Content Exchange was designed to achieve a new degree of
personalization and interaction when exchanging information on a public display.
In order to provide a significant level of customization, avenues within the system
7
were constructed to use real-time input from users and their environment. A
prototype of the Content Exchange was implemented to illustrate how devices
connect and disconnect to the system using Bluetooth. Implementing a prototype for
Bluetooth communication exposed limitations of the communication protocol causing
the prototype to deviate from original design decisions. In theory it makes sense for
mobile clients to be considered Bluetooth masters because this gives users more
control over their devices. Further, users may have their mobile devices already
configured as Bluetooth masters, such as cell phones that would have Bluetooth
slave accessories such as hands-free headsets. Also, in order to make use of many of
the Bluetooth features, communicating directly over the Bluetooth stack was
required. Since Bluetooth stacks are proprietary it was uncertain that
communication between different stacks could be achieved. Therefore a TCP
interface to Bluetooth was used to manage and control communication. Although
this added complexity to the communication procedures, as discussed in this thesis,
it helped make communication between Bluetooth devices possible.
This thesis discusses the design and implementation of a multi-user interactive
public display. Section 2 outlines the motivation behind the project and describes
various related projects. Section 3 discusses the design of the Content Exchange,
goals, system architecture and functionality. Section 4 provides an overview of the
implementation of the pilot system. Finally, an analysis of the pros and cons of the
system and possible future extension is discussed in Section 5.
8
Chapter 2
Motivation and Background
There have been many paradigm shifts when devising architectures that store and
circulate information. Given the new requirements about mobility and dynamic
systems, present day information transfer models can no longer address important
user scenarios. Current advertisements and other large-scale information displays
are static and stand-alone. There is minimal interaction between the general public
and the devices that are displaying information. Other systems such as public
kiosks that do have user interaction are limited in functionality and do not provide
multi-user support.
Various systems and mechanisms have tried to address context-aware and information exchange
issues in order to provide users with personalized and more relevant services. The
research group that supported this thesis has developed a suite of systems that help
incorporate technology into people's daily routines. Other proposed solutions include
public kiosks and information exchange protocols. The aim of this section is to
provide an overview of the Oxygen Research Group and other work related to public
displays and content exchange.
2.1
Group Background
The Oxygen Research Group is focused on developing a pervasive computing environment that is
responsive to people completing routine tasks and going about their everyday lives. Project
9
Oxygen is a wide-ranged pervasive computing project that combines the use of different
technologies.
The key theme of Project Oxygen is to "help people do more by doing less." [1] The project
strives to make technology fit into the everyday lives of people as opposed to making people
adapt around the needs of technology. The vision of the project is to create a single portable
device that will perform all the desired communication tasks [2]. The human-centered approach
addresses various issues, which include providing location-aware support and enabling the
customization of anonymous devices so that people can have constant access to information [3].
2.2
Context-Aware Pervasive Computing Efforts
Efforts to develop a pervasive environment are widespread. Many different approaches are being
investigated to achieve a human-centric environment that is responsive and customizable. The
basic idea across the board is making computers fit the needs of humans.
Georgia Institute of Technology runs a program called the Aware Home which aims at creating a
system which can understand the needs and desires of the its residents. Considerable effort is
also placed into understanding the emotions of its residents. Using sensors including video
cameras, wearable monitors, and location sensors, the system analyzes it's residents gestures,
facial expressions, and movement. Based on this information, the aware home attempts to better
predict the desires of its residents. [4, 5]
The University of California Berkeley has developed a software agent called the Communication
Information Agent (CIA). They have used a service infrastructure approach to context-aware
computing where middle-ware technologies can be accessed through a network. The CIA uses
10
context and human-to-human interaction to gather and deliver relevant and useful information.
The information retrieval methods are based on the relationship of the information with the user
that relies on location and speech technologies. [6, 7]
2.3
Efforts with Public Displays
Various other approaches to the dynamic public displays have been designed and
implemented. Some solutions, although create an interactive display, are only valid
for scenarios where the users are part of a small, related group. These solutions
would not be applicable in a public setting where users are constantly connecting
and disconnecting.
The GVU Center at Georgia Institute of Technology has designed Semi-Public
Displays, which are shared displays targeted for small groups [8]. These SemiPublic Displays are used in environments were there is a group of people working
closely together. The purposes of the displays are twofold, one to help people
collaborate and brainstorm on ideas and two, to inform people of the group's
activities and interests. Although Semi-Public Displays promote collaborative
environments and advertise events, the content on the displays is only partially
dynamic. The information shown on the display is parsed from a weekly status
report. Also, Semi-Public Displays are geared towards small, private groups and
cannot be extended to public use.
At the University of Calgary, researchers have developed an interaction system
called the Notification Collage (NC) [9]. The NC is designed to serve as a public
11
bulletin board for small, co-located groups. The NC has a separate interface for the
general public display and personalized views for each user. The content on the NC
is dynamic so that it can show updated video captures, sticky notes that show the
message being typed, and various other multi-media elements. The NC is successful
in creating an interactive environment, but similar to Semi-Public Displays it
targets a small private group. Extending the NC to a public domain would be
nontrivial.
Tbe Software Infrastructures Group at Stanford has developed an IRoom where
"people can work together in a technology rich space" [10]. The IRoom is an
interactive workspace that allows users to collaborative on school work using their
mobile devices. The IRoom software infrastructure lets users enter and leave the
workspace without affecting their experience. IRoom similar to other solutions
focuses on building a collaborative environment and does not tackle the issue of a 2way exchange of information.
Other display solutions are similar to Appliance Studio's Web Signs [11]. The Web
Signs project is a flat panel display appliance that can run external applications.
Web Signs provide interactive support, host dynamic content, and communicate with
each other. Although Web Signs achieve a great deal of user interaction, it is mostly
one-on-one interaction. Web Signs have a touch screen for users to "dig deeper" and
get more information. However, Web Signs do not provide a framework for multiple
users or for users to interact with the display using mobile devices.
12
Chapter 3
The Content Exchange
The Content Exchange is a system that is designed to serve as a mechanism for
users to interact with information on public displays, called Content Exchange
Appliances (CEA). These appliances are designed to be an information portal that
people can interact and exchange information with. This section serves to outline
the design goals, architecture and functionality of the system.
3.1
User Scenarios
The basic scenario for the Content Exchange system is to display general
information to the public. For example, viewers of the information can interact with
the display appliance while waiting for an appointment or a meeting. The displayed
content could include general health tips if deployed in a doctor's office or product
advertisements in the first floor lobby of a software company's building.
A second situation for the CEA is in a meeting room where information is shared
between multiple parties. The system could be used to assist in teleconferencing
because not only would an image be transferred but content could be exchanged as
well. This video relaying could also be applied in a family setting where different
members of the family can communicate from the home, office or while they are
walking around.
13
Another scenario for the display appliance is to support a collaborative environment
where users can take full advantage of the display by uploading and downloading
content. Co-workers or group mates can use the display to work together on a
project by uploading and displaying content.
3.2
Design Goals
The CEA can be considered a next generation public kiosk that serves multiple users
at one time. The key aspect for the project was to keep the CEA human-centric
because that would help instill faith in the end-user. In order for the CEA to be a
success users have to feel comfortable enough to interact with and trust the
intentions of the system. Various goals were kept in mind when designing the CEA.
Seamless End-User Experience
To maximize the impact and usability of a CEA, an important aim is to provide a
seamless end-user experience. The idea is that from arrival to departure, a user can
just interact with the display without experiencing delays, configuring settings or
establishing connection interfaces. Device discovery and communication is initiated
when a user is within local proximity of the public display. Similarly, a session ends
when the user walks away from the public display. In addition, users do not have to
be concerned with the actions of other users. Users should not have to wait for
another user to finish their session or interaction. The system ensures that the
navigation and actions of each user are independent.
14
Customized Interaction and Exchange
Users have the option to configure their settings to achieve a customized and more
personalized experience with a display. Each user can define the type of information
they download and upload. Since users of the CEA have personal mobile devices,
they can define how they want to save the information they receive from the display.
An important goal of the system is to build flexibility into the application that would
allow users to personalize their experiences. Providing users the ability to control
their personal settings regarding information retrieval and privacy makes the CEA
applicable in more situations and more a larger variety of people.
Privacy and Security
Ensuring the privacy of users is important since the system is open to the public.
The public displays are used as information portals and require the user to establish
a connection via one of the various communication standards that exist. A
connection using Bluetooth or Wireless LAN may indirectly expose specific
information about a user to the display application. The goal of the Content
Exchange is to make it possible for users to anonymously interact with displays.
Despite knowing the location of the different displays, a user's location can remain
unknown because the Bluetooth information associated with a device can be
changed with each session.
Therefore, users can interact with public displays with
the assurance that their sessions or actions are not tracked or recorded.
The goal is to provide avenues for users to specify their preferred security level in
order to achieve the desired level of privacy. Although security was not the focus of
15
this project, putting structures in place that would support various secure protocols
was important. The goal was to achieve a minimal level of security that did not
require a lot of computation by the mobile devices. For this reason, the use of the
Secure Socket Layer protocol was minimized by creating a negotiation scheme (see
Section 4.4). Since the information in the CEA repository is not necessarily
sensitive, the default aim for security was minimal, but can be extended and
improved upon.
3.3
System Architecture
The overall architecture of the Content Exchange involves a public display running
content exchange services, called a Content Exchange Appliance (CEA) and
handheld devices (see Figure 1). The CEA can communicate and transfer data via
different protocols, including Bluetooth, Internet Protocol, and IR.
Content
Echange
INTERNET-A
802.11b
0I
E 0
Content Exchange
Appliance
ith
IPAQ
Bluetooth
Phone
B
ce
Discovery
IR / Bluetooth
IPAQ
Figure 1: CEA connects to mobile devices over Bluetooth, IR, IP, etc.
Each handheld device is running an application that recognizes and understands the
messages sent by a CEA. In a typical scenario, a user will approach a CEA and
establish a connection. Once the CEA has secured a connection the user can proceed
16
to view, download and upload information to and from the CEA accordingly. Several
components make up the multi-user content exchange system.
Content Exchange Appliance
CEA's are public displays hosting information, which can be viewed, transferred,
and edited externally by users. A CEA is considered a "super-computer" because its
computational ability, power supply, and memory dramatically outweigh the
properties handheld devices.
They support the communication mediums of
Bluetooth, 802.11b, Ethernet, and IR. CEA's accept connections from devices that
would like to interact with the information on display.
CEA's can be controlled by typical network administrators to ensure that the
information displayed and transferred is accurate and secure. A CEA is similar to
the server side of an application, where people can use mobile devices to issue
requests and make transactions.
Mobile Device
A mobile device serves as the gateway between the user and the CEA. Users can
interact with a CEA with any type of mobile device that supports one of the listed
communication protocols (see Content Exchange Appliance). The mobile device runs
a client application that lets the device recognize a CEA. This application is
responsible for initiating a connection to the CEA and managing the information
exchange. The mobile device provides the user with a personal viewing platform
17
regarding the contents of the display as well as an input panel when uploading and
downloading information.
Personal Area Network
A personal area network is an ad-hoc network that exists for each CEA and the
devices to which the CEA is connected. This project focuses on creating a
communication scheme using Bluetooth. A Bluetooth network is called a piconet,
where one device, called a Bluetooth master, is connected to a maximum of 7 other
devices, called Bluetooth slaves. The master of a piconet is responsible for
maintaining the frequency hopping for the connected devices. Bluetooth devices
operate within the 2.45MHz frequency range. Bluetooth devices can communicate
without interfering with each other by performing what is called "spread spectrum
frequency hopping." Bluetooth devices communicate by hopping 1600 times per
second between 79 different frequencies each 1 MHz apart. Bluetooth piconets do
not interfere with neighboring Bluetooth networks because the masters of each
piconet ensure that the connected devices hop to different frequencies in unison. [12]
In the original design, the mobile devices were considered the Bluetooth masters of
the system. This was because as users would most likely have their devices preconfigured as masters, regardless of their interaction with a CEA. In order to
achieve this design and time scheduling routine must be put in place to balance the
requests of the mobile clients, since the CEA would be the slave. To avoid that
complexity, in the prototype of the CEA, the CEA was made the master which runs a
module devoted to listening for incoming connections from other Bluetooth devices.
18
The piconet isolates the communication associated for each display and allows users
to interact with display anonymously. Once a user disconnects or leaves the vicinity
of a piconet they no longer have to be associated with the temporary IP they had
when connected.
Connection Channel
A connection channel is a communication link established between a CEA and a
mobile device. A connection channel is created when a user walks in the vicinity of a
public display and chooses to start interacting with the information. The channel is
an understood agreement between the CEA and the device regarding the properties
and settings of the user for a given device, user, session, or request.
Attributes
An attribute is a name value pair associated with a user, handheld or public display.
They are a means of personalizing and customizing one's interaction with a CEA.
Depending on the type of attribute, they are exchanged between the display and
handheld at the time of connection or when transferring data. There are two types
of attributes, session-based and action-based.
Session-based attributes are properties of a user's handheld and are therefore device
specific. They are transferred to the CEA when a connection channel is established.
Session-based attributes also describe privacy and security settings of a user (see
Section 3.2).
19
An action-based attribute is a name-value pair transferred over a connection
channel when a user motions to exchange information with the CEA. Action-based
attributes describe the type of information that is to be downloaded or uploaded.
These attributes act as parameters to the different function calls on a CEA.
3.4
Functional Design
Interaction with the multi-user content exchange system is a result of various
methods and functions built into the system. The functionality of the CEA depends
on its purpose, the environment in which it is deployed, the users that interact with
it, and the computational power available. The CEA is designed to be versatile
enough to mold to various user scenarios and requirements, defined either by the
administrator or the users. This section defines how the system is designed to
operate and adjust to various users.
Configuring Settings & Attributes
An important goal the CEA is to provide a framework where users can customize
their mobile client application in order to personalize their experience. Users can
configure a variety of settings or use the pre-configured default values. Minimal
setup is required if users choose the default implementation, where as customizing
the settings requires a nominal amount of configuration time.
There are various common attributes that a user is more likely to configure to fit
individual needs. For example a user can change the default connection behavior
(see Section 4). When a user wants to create a connection channel with a CEA, the
20
user can choose to automatically connect to the closest CEA or see a list of possible
devices before connecting. The default value, automatically connecting to a CEA,
can be changed by users if they wish to have more control over their exchange
sessions. Other attributes may be specific to the mobile device, such as the amount
of free memory or the format and size of the device's screen.
Similarly, a user can configure their privacy settings, depending on how private they
want to keep their interaction with a CEA. When connecting using Bluetooth a user
can change the Bluetooth device descriptor and address that is affiliated with the
mobile device. If a user wishes to remain anonymous then they have to option of
changing the settings of their Bluetooth configuration so that each time they use the
CEA they are connecting under a different address.
Navigation
In the most common scenario, a CEA will have a top level page for the display
application. This page remains visible on the screen for the public, even when users
are interacting with the information. Users can "navigate" through the information
because mouse movements from the users' mobile devices are captured and sent to
the CEA (see Section 5). This positioning information is used to determine the
actions intended by a user.
Pages below the top level or home page of the CEA are viewed on a user's mobile
device. Depending on the type of the device and the session-based attributes defined
by the user, the CEA sends the appropriate viewing format. For example, if a user
21
is connected to a CEA via a cell phone, then the CEA will respond to a request by
sending pages with a specific format, in this case, WAP. This processing can be done
by the CEA, since its computational power outweighs that of a mobile device.
Other possibilities for navigating and viewing information depend on the type of
CEA deployed. Some CEA's may have space set aside on their display for users to
view information. This space can be used when session-based attributes are set to a
lower level of privacy. This would mean that a user does not mind having their
actions or navigation paths followed, perhaps because the information they are
viewing is public or accessible by others. The designated space would have to be
shared by all the users connected to a CEA and may require the implementation of a
scheduling system. This functionality is mentioned as a possibility but was not
explored in detail for this project.
Downloading Content
In addition to viewing information on a CEA, a user can download information
pertaining to the displayed information. Some possible examples of data
downloaded are information about events, advertisements, news, etc. Depending on
the use of a CEA, a variety of information can be made available.
22
HOME PAGE
EET
abstract
calendar
launch
text
sync
email
Phone
PC
2
3
1
Figure 2: Multiple functions exist within a single a location on a CEA.
The unique behavior of a CEA is that the function behind a button, link, or pixel of
its display can have different results (see Figure 2). In past models, a position (x, y)
of a display has a particular value associated with it, such a denoting a color. The
current approach has a particular (x, y) coordinate associated with a value and a
function, {(x, y), val, function}. This function is what makes displays and web pages
dynamic. Touching a button or clicking a position on a page calls a function
associated with that position. Similar to current representation, on a CEA a
position (x, y) is associated with a value and a function. The difference is that
functions on a CEA also take in parameters. These parameters a passed in by the
user as attributes, defined in Section 4.2. This means that on a CEA a point can
appear to have multiple functions, {(x, y), val, ({attr}, function)). Since attributes
define the type of information downloaded, one person may synchronize their
calendar with the CEA events calendar, while another person might download email
information regarding a particular event.
23
Uploading Content
In addition to downloading information and user can upload content depending on
the settings of a CEA. When a user uploads content they can specify action-based
attributes in order to create or add functionality to a particular pixel or position (x,
y) associated with that content. The attribute associated with the information a
person uploaded must be unique and not currently in use by the CEA. Users can
also assign a value to a particular point on the CEA, but that requires additional
access privileges on the side of the user. Content that is meant to be displayed and
edited by particular people would need to have to proper attributes and permissions
associated with it.
Modes of Operation
There are various modes in which a CEA can operate. Each mode pertains to
specific functionality that depends on the configuration and purpose of the CEA.
The basic level of activity is when the CEA is in the display mode. In this mode the
CEA displays static information for the public. The information can be updated or
changed by system administers, but appears static to the user during a session.
When in the display mode, the user may have limited interaction with the
information, since the primary purpose is to view what is being presented.
In the receive mode, the CEA performs a partial set of the capabilities. The purpose
of a CEA in this mode would be to distribute information to users. Users would only
be able to download available information. Since users cannot upload information in
24
this mode, the content that they download is virtually static unless changed by the
CEA administrator.
The exchange mode is when the CEA is at full activity. In this mode, users can
freely download and upload information. Downloading information is similar to the
process when the CEA is in the receive mode. When uploading information, users
can specify the format and type of information by sending the appropriate actionbased attributes with the request. The CEA interprets the attributes and store the
information accordingly. Users also have the option of uploading collaborative
information to leave for other users or the CE administrators.
25
Chapter 4
Functional Implementation
The pilot system that was implemented is focused on building a connection
framework for the CEA. On top of this framework, applications and extensions can
be built (see Section 5). The backbone developed defined the protocol of establishing
a connection between a CEA and a mobile client and extending that to building a
personal area Bluetooth network. A connection establishment protocol is defined in
detail because of its important role in public display systems such as the Content
Exchange. Designing a way to create and manage connections between mobile
clients and a CEA is difficult because of the system's unique requirements.
The default communication medium used in the pilot is the Bluetooth protocol, along
with various software modules and interfaces to help establish connectivity among
the devices. Bluetooth was chosen because it has a built in a device discovery
protocol and supports efficient close range communication. Devices that are
Bluetooth enabled, such as Compaq Ipaq, Nokia Cell Phone, and the Acer Tablet PC,
fall into the appropriate category of devices that are well suited for the purposes of a
data exchange system.
26
4.1
Bluetooth Stack & Interfaces
There are two main Bluetooth stacks available for Linux, which is the default
operating system used for this research. The stacks, OpenBT, contributed by Axis,
and Bluez, originally contributed by Qualcomm, are both open source and have
strengths and weaknesses of their own. Bluez was the stack chosen for the
prototype of the Content Exchange. Bluez, although new compared to OpenBT, has
several benefits that are relevant to the requirements of a CEA. Bluez is included in
the standard Linux kernel which allows for better integration. Second, the stacks
have different interfaced for drivers and applications. OpenBT uses a serial
abstraction, and BlueZ uses a network abstraction [website 1]. The network
abstraction in Bluez is a key benefit because establishing connectivity between
various devices is at the center of making the Content Exchange system work.
Bluez achieves a solid network abstraction over Bluetooth by using various profile
and interface tools. PAN is the Bluetooth profile that defines how to do networking
on the Bluez stack. PAN achieves IP support over Bluetooth, by using an interface
called BNEP. BNEP is an interface, similar to ETH, which encapsulates Ethernet
frames in an L2CAP session. L2CAP, Logic Link Control and Adaptation Protocol,
runs on the control data link layer, and is a control protocol used for Bluetooth and
sometimes Ethernet [12]. Finally a driver, either hci_uart or hci_usb was needed,
depending on how Bluetooth is connected.
The BNEP interface for the mobile devices and the CEA are pre-configured with the
following commands:
27
%> modprobe bnep
%> ifconfig bnepO <PAN IP ADDR>
The first command starts a new Ethernet emulation and the second command
configures TCP/IP on the newly created bnep interface. The PAN IP ADDR is an
internal IP address unique to the personal area network of a specific CEA. In the
pilot system a default PAN IP of 10.0.0.1 was given to the CEA. Once the interfaces
are configured, the devices and CEA can run their respective exchange applications.
4.2
CEA Discovery
A user with a handheld device detects a CEA when it is in local proximity to the
display, via Bluetooth. The user can manually connect to a CEA or rely on the
device discovery mechanism in Bluetooth. In either case, the discovery is done by
the user's device and not the CEA. By giving users the option to connect to a CEA
they can be assured that their privacy is protected.
Device discovery of Bluetooth works because the CEA is configured as a Bluetooth
master. As a master, the display is responsible for maintaining the frequency
hopping of its piconet. The PAN profile is used to run a server or daemon process on
the CEA so that it is constantly listening for incoming Bluetooth connection
requests. This process remains running despite the activity of the clients connected
to the CEA because it must stay ready for incoming connection from other clients.
A sequence of commands is issued after the BNEP interface is setup, to allow mobile
devices to discover a CEA. On the CEA side the commands are:
28
%> pand -master
%> pand -listen
These commands start a process that remains running on the CEA as long as it is
powered and functioning. It is important to have these processing running in order
to accept new connection from mobile Bluetooth enabled devices.
On the mobile client side, the users' handhelds are most often configured as
Bluetooth slaves. As slaves the mobile devices can also use the PAN profile in order
to connect to the CEA. The mobile client has a loop that is constantly scanning the
environment in order to keep a look out for Bluetooth devices, using the command,
%>hcitool scan
Scanning ...
00:80:37:B5:A8:3A
CEA1-LCS
As shown, this command lists the descriptors and addresses of the Bluetooth devices
that are in a local proximity to the mobile client. This means that the result of a
Bluetooth scan will be a list of CEA's as well as the devices of the viewers using
these displays. When other Bluetooth devices are detected, the client application
prompts the user with a list of available Bluetooth enabled devices, listed the CEA
marked entries first.
If there is only one CEA in the area, then depending on the user's settings, the client
application will connect to that display. The default setting on the client is to
automatically connect to the display if there is only one option. Users can change
their settings ahead of time.
29
If multiple CEA devices are found during the Bluetooth scanning loop, then the
client can use local proximity as criteria to choose a CEA. The signal strength of the
Bluetooth signal can be used to the degree of closeness, which would suggest that a
user is closer to one CEA over another. Similar to other settings, the user can
change the default setting from automatically connecting to the closest display, and
instead choose to see a list of the options before connecting.
Finally in the case where the user has a hard time distinguishing between CEA's
and other mobile clients, the client will have to parse out the CEA devices from the
scan results. Before presenting a list to the user, the client application can use the
device descriptor field to determine whether the corresponding Bluetooth address is
a CEA or another mobile client.
If the user is presented with a list of possible CEA and Bluetooth devices, the user
has the option of connecting to a CEA or not. The user must decide which Bluetooth
device to connect to if there are multiple options. If the user decides not to connect
to the display at that time, they have the option to continue scanning for Bluetooth
devices or to quit the client application altogether.
4.3
Establishing a Channel with a CEA via Bluetooth
Once a CEA has been discovered by the client application, it is up to the client
application on the handheld to initiate a Bluetooth link before establishing a
communication channel. There are a series of steps in the communication protocol
that must be followed in order to ensure a successful connection to a CEA. The
30
connection protocol moves the client to and from various transition states, see
Figure 3.
set name
to
detect
CEnA
r Scansofor
nBeeti'
T
S23S
Si
BT
for
Nait
Wonct
SET
ecan
TCP Socket
ooenec
ST
t~tno
c
BT
Send
ACX to
r
disconnect
Figure 3: This figure represents the transition states of the CE client application.
When establishing a connection, the client starts in SO when scanning for devices.
The client ends in S4 when it has received an ACK from the CEA, confirming that a
connection channel has been established.
While the client application is scanning for Bluetooth devices the client remains in
the initial state, state 0. The first step in connecting to a CEA occurs after the client
application detects the presence of a CEA. After the discovery takes place the client
moves to state 1. In this state, if the user has decided to connect to a CEA, the client
application chooses a random IP address from 10.0.0.2 to 10.0.0.255 to use for
communicating in the CEA's personal area network. If the user decides not to
connect to a CEA then the client returns to state 0. The random number is the IP
used to configure bnep interface of the client as described in Section 4.1. Once
interface is configured, the client changes the name of the device to the PAN IP
address using the command,
%> hciconfi -a hciO name 10.0.0.*
The name of the device is changed in order to provide the CEA with the IP
information necessary to establish a connection channel. Once the name has been
31
changed the client transitions to state 2, where the client issues a Bluetooth
connection request by calling,
%> pand -c
<bdaddr>
The command is part of the PAN profile and the argument, bdaddr, is the Bluetooth
device address of the CEA. The client application gets the Bluetooth device address
of the CEA from the scan request it loops through when searching for a CEA. After
issuing the Bluetooth connection request, the client moves to state 3, where it waits
for a response from the CEA. When a Bluetooth connection is received by the CEA,
the device address and name of the handheld are made available, where the name is
the PAN IP address of the mobile client requesting a connection.
Before a communication channel can be established, the CEA must check the
validity of the device that is requesting the new connection. The CEA needs to
ensure that the IP address submitted by the client is not currently in use within its
network of devices. Second, the CEA must also make sure that there isn't another
active session affiliated with the Bluetooth device address of the new client
If there is an IP address conflict that signals the CEA that the new client has chosen
an IP address that is already in use. The CEA simply kills the Bluetooth connection
using the command,
%>pand -k
<bdaddr of new client>
32
At this time the client is in state 3, waiting for a response from the CEA. If the
client application has waited for longer then 10 seconds then it checks the Bluetooth
connection it made with the CEA. Since the CEA just ended the connection, the
client is transitions back to state 1 where it will generate a new IP address and then
try to progress again. On the other hand, if the Bluetooth connection has not been
killed, then client may experience delays because of an error with the CEA or the
TCP socket request. In this case the client kills the Bluetooth connection and
transitions to state 2, where it requests the connection again. The client does not
need to go back to state 1 in this case because the delay was not necessarily due to
an IP conflict.
If the display application finds a Bluetooth device address conflict then the CEA
does not know whether the new client has changed it's Bluetooth address or if it is
an old client that has recently crashed and is trying to reconnect. The CEA then
pings the IP on record for that particular Bluetooth address. If the ping is
successful, then the CEA kills the Bluetooth connection request made by the new
client. Disconnecting the mobile device sends the client application back to state 1.
On the other hand, if there is no response to the ping request, the CEA assumes that
the old client is no longer connected to the network and therefore the CEA can
reassign the IP to the new client. The CEA now continues with the connection
procedures as would occur if there was no conflict.
If there are no conflicts with the IP addresses and the Bluetooth addresses then the
CEA pings the client's IP. Issuing a ping to the mobile device creates a route to and
33
from the CEA. Without a ping from the display, the client would not be able to
reach the CEA via its IP address. If the ping request is successful the server opens a
TCP socket connection to the mobile device using the PAN IP address, in this case,
10.0.0.*. The CEA informs the client that the connection request has been accepted
by sending a "CONNETACCEPTED" message. The CEA temporarily stores
information about the client's Bluetooth and IP addresses (see Partial State of the
System).
Now the client device can communicate with the CEA. When the client receives the
"CONNECTACCEPTED" message, it transitions to state 4. In this state, the client
sends an ACK, or acknowledgement, back to the CEA. Once this is complete a
connection channel has been established between the CEA and the client device.
4.4
Security
In order to provide viewers a minimal level of security and protocol was designed to
help guard against adversaries. After a connection channel is established between a
client and a CEA, various factors can effect the status of the channel. For example
the user could walk away from the display, the client application could fail, the
mobile device could power down, or various other random events. In order to
preserve the security of a connection channel, the user and the CEA negotiate a
secret key and nonce (see Figure 4). The negotiation tells the CEA that the mobile
device connected on the other end is in fact the expected device and not someone
pretending to be that device.
34
Mobile
CEA
Step 1
Device
(Shared Key)
Cient
-,
Step 2
generate
e-
Step 3
(nonce), -
notice
Step 4
decrypt
nonce
ACK
4-
Data
(nonce -text)S K
Figure 4: This figure illustrates the negotiation between the CEA and a mobile
client that takes place to ensure a secure connection channel in the CE.
The negotiation takes place after a connection channel has been established, which
is when the client has reached state 4, as described in Section 4.3. When the
connection channel is first established the Secure Socket Layer protocol is used to
keep the connection secure. In step 1 of the negotiation is the CEA sends a 32-bit
shared key to the client device over the created TCP socket. The shared key is
encrypted with the client's SSL public key. Once the shared key is received, the
client decrypts uses its private key to decrypt the message. From this point forward
SSL, a computationally expensive protocol, is no longer necessary because the client
and the mobile device can use the shared key to encrypt their messages. In step 2,
the client generates a random 32-bit number as a nonce. In the next step, the client
sends the nonce encrypted with the shared key to the display application. Here,
encryption is simply the XOR of the shared key and the nonce. In step 4 the CEA
receives and decrypts the nonce message using the previously sent shared key. The
CEA now has the nonce that will the client append to the beginning of its messaging
35
as authentication. In the final step, the CEA sends and ACK message to the client
which acknowledges the receipt of the nonce.
Now that a secure channel has been established, all data sent by a client for a CEA
will have the nonce at the beginning of the transmission. The nonce will signal to
the CEA that the data it is about to receive is valid because it is coming from the
correct client.
4.5
Partial State of the System
The partial state of the Content Exchange system is necessary because information
regarding the mobile clients and the public displays is temporarily stored by the
client and display applications. Information is not permanently stored because of
the dynamic environment in which a CEA is deployed.
The client application keeps track of the history of displays that a user has visited.
The table, PASTCONNTABLE, stores the name of the CEA, the Bluetooth device
address of the CEA, and the BNEF IP addresses the client used to connect with the
CEA, see Figure 5. The client also saves the nonce and shared key that was
exchanged in case it can still be used.
CEA
Name
CEA
btaddr
BNEPO
IP
Nonce
(32-bit)
Secret Key
(32-bit)
Figure 5: This figure shows the table, PASTCONNTABLE, which is stored on the
client which allows for speedy re-connection.
36
The table is preserved until the user quits the client application or the mobile device
dies.
Storing the last used IP for a previously visited CEA will save the client time when
it tries to reconnect to that display, further discussed in Section 5.
On the CEA, the application stores information about the client, the connection, and
the user's device. The nonce and IP information affiliated with a particular
Bluetooth device address is saved in the CEA's CONNECTIONTABLE, see Figure
6. Each record is saved with a timestamp in order to keep the records up to date.
The expiration of the timestamp is determined by the settings of the CEA which
depends on the environment in which the CEA is deployed. The nonce is stored in a
table and mapped to the PAN IP address of the client. The nonce is used to
determine the authenticity of messages coming from a particular PAN IP.
Timestamp
Secret Key
Nonce
Client
Client IP
(32-bit)
(32-bit)
btaddr
address
Figure 6: This figure shows the format of the CONNECTIONTABLE stored on the
CEA. This table helps determine the validity of incoming requests and the
expiration of previously used nonces and secret keys.
The CONNECTIONTABLE also maps clients' Bluetooth device addresses to their
PAN IP addresses. This mapping allows the CEA to recognize mobile clients that
have visited in the recent past.
Session-based attributes, defined in Section 3.3, are also stored temporarily by the
CEA. They are marked for deletion when a client device disconnects from a CEA.
Session-based attributes are stored in order to prevent clients from having to send
37
repeated information with the action requests when navigating through the
information on a CEA. A set of session-based attributes is only active during a
single session or interaction between a user and a display.
38
Chapter 5
Analysis & Extensions
The Content Exchange was designed to propose a new way for users to transfer data
into and out of a public kiosk type device. The public display component added
different user scenarios and usability requirements to the standard information
repository problem. Multiple users accessing and interacting with the information
simultaneously adds complexity to the system. The concept of a public display
required devising proper protocols to protect the viewers and the CEA.
Efforts for the project are focused on furthering the development of pervasive computing
technologies. The Content Exchange system can incorporate the different pervasive
computing technologies of the Oxygen Research Group, such as location-aware
support, speech recognition, and face and gesture detection. This section discusses
the tradeoffs of the design and prototype of the CEA, and then proceeds to address
possible extensions to the system.
5.1
Tradeoffs with Bluetooth
The Bluetooth prototype described in Section 4 provided the means necessary to
achieve many of the original design goals for the Content Exchange. Using
Bluetooth made device discovery in a local proximity possible. Although the device
scanning described in Section 4.2 successfully identifies CEA's in the area, all other
39
Bluetooth enabled devices are also found. The option presented in Section 4.2 solves
the problem by using the Bluetooth device descriptor to extract the CEA entries.
Despite its feasibility, relying on a descriptor suggests that a standard naming
system is defined for every CEA, where the name is set to the value of the Bluetooth
device descriptor. Another possible solution is to scan for Bluetooth devices that
have been declared as Bluetooth masters [footnote?]. Since each CEA is supposed to
be a master of its own piconet, then scanning for masters will return a list of display
devices. Using this mechanism, mobile clients would not be detected because they
are configured as Bluetooth slaves.
Although Bluetooth was the preferred communication protocol used in this project,
using Bluetooth did pose certain networking limitations. According to the Bluetooth
specifications a piconet can consist of one master and a maximum of 7 slaves
[Bluetooth Spec]. The limit on the number of Bluetooth connections for a given CEA
limits the number of viewers that connect to the display using Bluetooth. This
means that once the 7 slots are taken, users with Bluetooth enabled devices will
have to wait until someone leaves, try connecting to a different CEA, or connect to
the same CEA using a different communication protocol, such as 802.11 or IR
Another limitation of Bluetooth was that there are different proprietary stacks that
can communicate over Bluetooth. Further research into Bluetooth needed to be done
in order to devise a way to communicate directly over the Bluez stack. Instead the
PAN profile on the BlueZ was used to assist with the prototyping effort, since it
supported the widely accepted and practiced use of TCP socket connections. Since
40
communication directly over the Bluetooth stack was not achieved, using the BlueZ
stack was both necessary and beneficial since it provided the foundation for
establishing a personal area network for each CEA. The PAN and bnep modules
gave the CE the appropriate level of user control. Using the PAN module made
development work simpler because communication from software could be done
through standard TCP sockets. The BlueZ stack was also easy to integrate into a
common mobile device used in the testing environment, Compaq's Ipaq.
On the other hand, using BlueZ also requires setup time when the CEA's are
powered up. This is understandable since a CEA is designed to stay powered on
most of the time. For mobile clients that are using BlueZ the initial setup is autoconfigured with the exchange client application. Another drawback to developing a
solution for BlueZ is that this stack is not used on every Bluetooth enabled device.
The portability of the exchange client applications to other Bluetooth stacks is not
certain. Separate client applications may have to be written for various Bluetooth
stacks. Although the programming logic would be similar, the connection and
scanning commands may differ among the different stacks. This is evidence that
further supports the argument for developing a Bluetooth industry standard.
5.2
User-Experience
An important goal for the CEA was to establish and maintain a seamless end-user
experience. Each user is supposed to remain oblivious to the variability of each
CEA's personal area network. The connections of each CEA are dynamic since
41
users come and go at random times. In the current design, viewers are not affected
by the arrival or departure of other users.
The system is designed to help make the users visit to each CEA enjoyable. The
partial state of the system helps users achieve a good degree of mobility without
many repercussions. The table maintained by the client application that stores a
history of the displays visited saves the user time when revisiting a display. The
client saves time because instead of attempting to connect with random IP address
that might conflict with other devices, the client can use the IP address in it has on
record for the corresponding display. The history table on the client only remains
refreshed while the client application stays running. This helps ensure that the
PAN IP addresses stored in the table are up to date and can be used with confidence.
In addition to saving time upon connection, the stored information also saves the
client from having to negotiate a shared key and nonce. The partial state of the
system has helped make the user experience efficient and friendly.
A user may experience delays when trying to connect to a CEA if the randomly
generated IP is already in use. Although the probability of two devices picking the
same PAN IP is low, the IP must also be sure not to overlap with LAN and WLAN
IP's that are in use. The issue of IP overlap or collisions is important and can effect
the performance of the CE. In this situation, the client applications can be
configured to chose random IP's from unregulated space. The current
implementation limits devices to the range 10.0.0.2 - 10.0.0.255. Increasing the
42
range of possible PAN IP addresses would further reduce the chances of IP
collisions.
Another point in the system where users may be required to input information or
experience delays is when they are trying to establish a connection channel with a
CEA. Connection to a CEA may not happen automatically because that depends on
the environment of the CEA. The client application is responsible for scanning the
area for Bluetooth devices and parsing the list for display devices, which may
require input from the user. In the other situation if the viewer is using the default
settings, then the display chosen by the client application may not be accepting
connections or may not be the CEA the user intended to connect to. These scenarios
occur when displays are in high traffic areas or when there are multiple displays in
close proximity to each other.
5.3
Pros and Cons of Customizations
Providing users with a high degree of customization and personalization was an
important goal for the Content Exchange system. The system was designed to give
users the opportunity to set their own desired level of privacy and connection
behavior. In addition, users could define any number of personal settings that
described their mobile devices, storage destination when downloading, data source
when uploading, and various other session-based attributes. With this level of
customizations come various tradeoffs and outcomes.
43
Processing the different user settings and customizations levels takes place on the
CEA. Several operations performed by the CEA's require a lot of computation,
processing and storage. For example, for a predetermined length of time CEA's
must store information regarding every mobile client that has connected to it. This
information is maintained whether the client is still connected or not. Second,
CEA's must monitor their current connections to keep track of when a mobile client
is active or not. In addition, CEA's also have to manage the session-based attributes
for each connected device. These attributes must be accessed each attempts
download or upload data, because the client does not send all the attribute
information with every request. Finally, when a user navigates to a new page the
CEA must determine the format of the data sent to the mobile client so that it can be
correctly displayed on that particular device. By design the CEA has to do a lot of
computation and processing, but this was done intentionally since CEA's are
supercomputers compared to mobile devices. This design benefits the users since
mobile devices have less bandwidth, slower processors, and minimal power resources
compared to a CEA.
In the design of the CE users can achieve a strict level of privacy, but that requires a
higher degree of customizations on the user's part. The default is not insecure
communication, but if a user wants complete anonymous communication they need
to take appropriate steps to make that happen. The decisions to make a high level
of privacy added work for the user was made because the common scenario for the
CEA is to exchange public data in either a confined environment such as a company
lobby or other workplace. In scenarios where displays are completely out in the
44
open, users that care about their privacy would not mind making extra effort to
achieve that. Setting session-based attributes requires work done by the user as
well. This is an initialization step that can be completed just once if the user decides
to keep the settings the same. If the user wishes to achieve varying behavior at
different times, then they must spend the extra time to make that happen.
Although this requires effort by the user, it is argued that users desiring this level of
personalization would not mind spending time setting up their attributes. The goal
of the system was to provide a framework that allowed users to customize their
exchanges with a CEA.
5.4
Extensions to the CEA Network
The implementation of the CEA does allow for various networking extensions. The
current solution was implemented to handle separate piconets for each CEA. One
possible extension is to form ad-hoc networks composed of piconets that correspond
to different CEA's. In Bluetooth these ad-hoc networks are called scatternets. The
added benefit from forming scatternets is that information can be transferred
between two or more CEA's. A example scenario involves a user that has uploaded
data to one CEA but wants it to appear on a collection of other CEA's. Instead of
manually uploading information to each, if the CEA's formed a scatternet the data
could be forwarded to the respective CEA's. Synchronizing displays could also be
achieved using other communication protocols, such as 802.11 or LAN. Another
scenario that would benefit from connected CEA's is that data uploaded by the user
could follow the user around as they moved from CEA to CEA. Similarly a user can
set their mobile device to act as a slave-master bridge and start to communicate
45
with two CEA's at once. This would give users the ability to exchange information
with two different display devices without having to go through the connection setup
sequence each time. Further research is being conducted to identify efficient
methods of forming Bluetooth scatternets [13, 14].
A second possibility to increase the number of Bluetooth clients that can interact
with a CEA is to make the CEA a Bluetooth slave. As originally imagined, having
the mobile devices be Bluetooth slaves would mean that clients would create their
own point-to-point piconet with a CEA. In this scheme, the CEA would be a slave to
multiple masters, and therefore would not be able to communicate with other CEA's.
Further exploration into Bluetooth will determine whether this design when
implemented can be applied to a larger number of clients.
In the design of the Content Exchange, each CEA is intended to communicate with
devices over Bluetooth, WLAN, IR and other wireless communication technologies.
The implementation can be extended to include other communication mediums,
namely 802.11. The benefit of using the BlueZ stack is that communication in
software is done through TCP sockets. This makes the integration with 802.11
trivial because the code does not change. Instead of opening socket connections to
the PAN IP address of a Bluetooth enabled device, a socket is opened to the IP
address of a WLAN device. When communicating over WLAN, devices would be set
to the ad-hoc infrastructure mode as opposed to the peer-to-peer mode. This means
that they would expect to receive an IP from a base station which could be the CEA
itself. Another interesting issue that arises when moving to the WLAN case is
46
device discovery. Similar to the Bluetooth scanning tool used discover Bluetooth
devices, an 802.11 device can scan for other 802.11 devices. In this situation, the
scan will return a list of IP addresses of devices that are using the same access
point. This list comes close to depicting the devices in close proximity because
WLAN devices use access points based on the distance between them. The problem
is a user may not know which IP address corresponds to the CEA to which they want
to connect. A simple solution to this problem is to have the CEA display its Internet
IP address on the display so the viewers can see it when they want to connect.
5.5
Location Aware Technology
The current design of the Content Exchange used Bluetooth device discovery
mechanisms to determine the proximity of a mobile client to a CEA. A different
approach to locate a CEA that is close to a user would be to use independent location
detection technology. Location aware systems can be applied to the exchange
system because they are built to track the positions of mobile clients.
AT MIT, the Laboratory for Computer Science has developed a location aware
system that does not use a GPS tracking system [15]. The Cricket Location-Support
System, the location aware system at LCS, uses a system of signal emitters and
receivers, called beacons and listeners respectively. A cricket beacon is mounted on
a ceiling or wall and emits an RF signal and an ultrasonic pulse. Listeners are
attached to mobile devices, such as Ipaqs. The listeners pick up the RF signals and
the corresponding ultrasonic pulses from the emitting beacon. The distance between
a listener and a beacon is determined by the distance between each ultrasonic pulse.
47
The Cricket Technology at LCS can be used to identify the CEA that a mobile client
should connect to. The beacons would be mounted on the ceilings close to the CEA's
or on the CEA's themselves. The listeners would attach to the mobile devices and
used to find the distances between the devices and the CEA's. The distance
information can be inputted into the exchange client application to determine which
CEA is closer to the user at any given time. This system is targeting towards group
communities in indoor environments.
5.6
Multi-Modal Input
The current version of the Content Exchange is designed to capture mouse
movements to control navigation, downloading, and uploading. There are various
input and control methods that people can use to interact with CEA's. Users do not
have to be limited to using the standard mouse or navigation unit on their mobile
device. Instead, they can interact and use the information from a display using
speech technology, location aware systems, and other movement tracking devices.
This section outlines the different ways to control the exchange of information.
Speech Input
A person can use speech to govern the activity between the mobile computer and the
public display. Another team of the Oxygen Research Group has developed speech
recognition technology. The speech technology has been developed since 1994, and
has revolved around enabling human to computer conversations. The project could
interlace to spoken language systems called Galaxy and SpeechBuilder.
48
SpeechBuilder is a suite of tools that can be used by external applications in order to
process spoken language [16]. This architecture allows for speech recognition,
language understanding, information retrieval, and language generation. Current
efforts have been successful in transporting the speech technology from a server to
light weight mobile devices [17]. The Content Exchange could be extended to
incorporate the speech technology as a way for users to communicate with a CEA.
Viewers could say navigation, download, or upload commands into their mobile
devices. Then the recognition system on the mobile device could convert the spoken
phrase into text and send the command over to the CEA. This mode of input would
complement or even replace mouse movements, especially if the viewer is using a
cellular phone.
Vision & Gesture Recognition
Improvements in vision technologies have made it possible to control computer
systems with facial expressions and body gestures. The Vision team of the Oxygen
Research Group has put together visual processing systems that can be used to
recognize and classify gestures. The Vision Interface Group has built systems that
can accurately detect the presence, posture, and gaze of a person [18, 19]. Using this
technology, viewers of a CEA can point or look in certain directions in order to
navigation through the information on display. The vision interfaces would make
the users' experience more natural because they wouldn't be confined to the input
methods on their mobile devices.
49
Chapter 6
Conclusion
The overall goal in pervasive computing endeavors is to understand and support everyday
practices of people. The human-centered approach in pervasive computing is
paralleled by efforts to build a context-aware system that still maintains a seamless
end-user experience. This thesis project illustrated a context-aware system that
created a new way for people to exchange information with public displays.
The prototype of the interactive display implemented a Bluetooth based solution.
Using Bluetooth sparked a series of questions regarding the organization of the
piconets and the mode of communication. In the prototype personal area networks
were assigned to each CEA, however it is important to investigate the performance
of the display if it was made a slave node in piconets controlled by mobile devices. A
TCP/IP interface was used in the prototype to assist in software development.
Further research should be done to explore the possibility of communicating directly
over the different proprietary Bluetooth stacks.
Related work completed in this area has successfully created interactive
environments that support group collaboration. The Content Exchange Appliance
distinguishes itself because it can be deployed in both small groups and the general
public. Also, the interactive display provided a framework for people to customize
how they interact with public displays. People can control the format and context of
50
the information they download and upload to the display appliance. The CEA
balances the responsibilities of the display appliance and the user. Users are
responsible for configuring their personal settings if they choose to change the
default behavior. The display appliance handles many of the computationally
expensive tasks such as managing the connections of the CEA, storing mobile client
information, and formatting data according to the specifications of each mobile
device.
The system designed and implemented in this project can also be extended. There
are various input methods that can be applied to the CEA, which would give users
options for how they communicate with the display appliance. Although this project
focused on designing an building a framework from public display interaction, many
applications can designed for such a system. Interactive public displays have the
potential to revolutionize the way people think and operate.
51
Bibliography
[1] Dertouzos, Michael L., "The future of computing," Scientific American, July 1999.
[2] Guttag , John V., "Communicating Chameleons," Scientific American, July 1999
[3] Brown , Eric S., Project Oxygen's New Wind, Technology Review, December 2001.
[4] Essa, Irfan, "Computers Seeing People," Al Magazine, Volume 20, Summer 1999.
[5] Kidd, Cory D., Robert J. Orr, Gregory D. Abowd, Christopher G. Atkeson, Irfan
A. Essa, Blair MacIntyre, Elizabeth Mynatt, Thad E. Starner and Wendy
Newstetter. Proceedings of the Second International Workshop on Cooperative
Buildings. Position paper, October 1999.
[6] Hong, J.I. and Landay, J.A., "A Context/Communication Information Agent."
Personal Technologies (Special Issue on Situated Interaction and Context-Aware
Computing), 2001.
[7] Hong, J.I. and Landay, J.A., "An Infrastructure Approach to Context-Aware
Computing," Human-Computer Interaction (HCI) Journal, 2001.
52
[81 Huang, E., Mynatt, E., Tailoring Public Display for Small, Co-located Groups; In
the Companion Proceedingof the ACM Conference on Computer Supported
Cooperative Work (CSCW), 2002.
[91 Greenberg, S., Rounding, M., The Notification Collage: Posting Information to
Public and Personal Displays, In Proceedingsof ACM Conference of Human
Factorsin Computing Systems, pages 514-521, 2001.
[10] Johanson, B., et al., "Multibrowsing: Moving Web Content Across Multiple
Displays," In Proceedings of Workshop on Handheld, in Conj. with ACM
Conference on Computer Supported Cooperative Work, 1998.
[11] Lewis, S., "Web Signs, Display Appliance Platform," Appliance Studio Technical
Report.
[12] Bluetooth, Specifications of Bluetooth Core 1.OB, 2003.
[13] Tan, G., Miu, A., Guttag, J., Balakrishnan, H., "Forming Scatternets from
Bluetooth Personal Area Networks," MIT LCS Technical Report, 2001.
[14] G. Zussman and A. Segall, CapacityAssignment in Bluetooth Scatternets -Analysis and Algorithms, Proc. 2nd IFIP TC-6 Networking Conference, Pisa,
Italy, 2002.
53
[15] Nissanka B. Priyantha, Anit Chakraborty, Hari Balakrishnan, The Cricket
Location-Support system, In Proceedingsof the
6 th
Annual ACM International
Conference on Mobile Computing and Networking (MobiCom '00), pages 32-43.
ACM, August 2000.
[16] E. Weinstein, SpeechBuilder:FacilitatingSpoken Dialogue System
Development, M.Eng. thesis, MIT Department of Electrical Engineering and
Computer Science, May 2001.
[17] Zue, Victor, "Talking with your computer," Scientific American, July 1999.
[18] Trevor Darrell, Konrad Tollmar, Frank Bentley, Neal Checka, Louis-Philippe
Morency, Ali Rahimi, Alice Oh, Face-responsive interfaces: from direct
manipulation to perceptive presence, International Conference of Ubiquitous
Computing, 2002.
[19] Alice Oh, Harold Fox, Max Van Kleek, Aaron Adler, Krzysztof Gajos, LouisPhilippe Morency, Trevor Darrell, Evaluating Look-to-Talk: A Gaze-Aware
Interface in a Collaborative Environment, Proceedingsof the Conference on
Human Factors in Computing System (CHI), 2002.
54