Ubiquitous Computing
Toolkits
Tara Matthews
Advance User Interface Software Fall 2004
Goals of Ubicomp

Scalable interfaces




Natural interaction




multiple devices
inter-device communication
multiple users
getting away from the desktop
augmenting surrounding environment
provide context-appropriate services
Calmness


more info w/o overwhelming users
aware of user’s interruptiblity
Toolkits for Goals

Scalable interfaces




Natural interaction




multiple devices
inter-device communication
multiple users
getting away from the desktop
augmenting surrounding environment
provide context-appropriate services
Calmness


more info w/o overwhelming users
aware of user’s interruptiblity
Ubicomp Infrastructure
Toolkits for Goals

Scalable interfaces




Natural interaction




multiple devices
inter-device communication
multiple users
getting away from the desktop
augmenting surrounding environment
provide context-appropriate services
Calmness


more info w/o overwhelming users
aware of user’s interruptiblity
CSCW (not covered)
Toolkits for Goals

Scalable interfaces




Natural interaction




multiple devices
inter-device communication
multiple users
getting away from the desktop
augmenting surrounding environment
provide context-appropriate services
Calmness


more info w/o overwhelming users
aware of user’s interruptiblity
Physical UIs / Mobile
(other lectures)
Toolkits for Goals

Scalable interfaces




Natural interaction




multiple devices
inter-device communication
multiple users
getting away from the desktop
augmenting surrounding environment
provide context-appropriate services
Calmness


more info w/o overwhelming users
aware of user’s interruptiblity
Context-Aware
Toolkits for Goals

Scalable interfaces




Natural interaction




multiple devices
inter-device communication
multiple users
getting away from the desktop
augmenting surrounding environment
provide context-appropriate services
Calmness


more info w/o overwhelming users
aware of user’s interruptiblity
Peripheral Displays
Toolkits for Goals

Scalable interfaces




Natural interaction




multiple devices
inter-device communication
multiple users
getting away from the desktop
augmenting surrounding environment
provide context-appropriate services
Calmness


more info w/o overwhelming users
aware of user’s interruptiblity
Interruptibility
(not covered)
Toolkits for Goals

Scalable interfaces




Natural interaction




getting away from the desktop
augmenting surrounding environment
provide context-appropriate services
Calmness



multiple devices
inter-device communication
multiple users
more info w/o overwhelming users
aware of user’s interruptiblity
Goals accomplished?
Evaluation
Toolkit Categories

Ubicomp infrastructure
Context-aware
Peripheral displays
Evaluation

Not covering:







CSCW (separate field)
Interruptibility (no toolkits yet)
Physical UIs (covered in Jack’s lecture)
Mobile Devices (covered in an unclaimed lecture)
Ubicomp Infrastructure



iROS
 Stanford Interactive Room Operating System
iStuff
 Physical application toolkit build on iROS
Aura



CMU “Distraction-Free Ubiquitous Computing” project
Distributed services infrastructure
Gaia


University Of Illinois at Urbana-Champaign
Infrastructure for Active Spaces
iROS



Interactive Room (iRoom) Operating System
Definition: A ubiquitous computing
environment where groups come together to
collaborate on solving problems (not a toolkit)
Contains:


Embedded devices: large display screens, tabletop display, & other output devices
Mobile devices: mobile devices brought into
space can be used with embedded facilities
iROS Goals


Facilitate common Interactive Workspace usages observed
in iRoom prototype workspace:
 Move data between devices
 Control devices & apps from other devices
 Coordinate apps, including ones not written to work together
Additional goals:
 Provide for heterogeneity of devices in workspace,
& new devices being added over time
 Allow easy integration of legacy devices
 Robust to transient failures
 Portable across installations
 Actions appear instantaneous to users
iROS

3 sub-systems:



Event Heap: stores and forwards events
Data Heap: facilitates data movement in an
interactive workspace
iCrafter: provides a system for service
advertisement and invocation, along with a user
interface generator for services.
Event Heap


Tuple space model
 Associated memory: content is addressable using a pattern
 Distributed systems based on blackboard model are easily
created in tuple space
 Decouples apps in space & time
Added semantics:
 Apps not designed to work together, so need


Apps can snoop or interpose


Tuple sequencing
Data shared & can build up


Self-describing tuples, Flexible typing, Typed tuples
Tuple expiration
Benefits
 Anonymous coordination
 Interposability
 Snooping
Event Heap
Figure: Example operation showing placed events, and using template events for matching
Data Heap


Provides type & location
independent storage
Machine independent:


Don’t need explicit location
of data
Type independent:

If a set of type converters
are available, system
automatically transforms the
data
Figure: Shows automatic retrieval of
a Word document in PostScript form
iCrafter




Universal control system: control anything
from anywhere
Services advertise how they can be controlled
System returns appropriate interface per
device
Interfaces can range from fully custom to
dynamically generated
iCrafter
Appliance
Description
Environment
Database
Service
Description
Remote Invocation
1
User
Active
Interface
UI
Renderer
2
4
Appliance
3
Generator
Selector
Generator
Processor
Interface
Manager
Discovery
Generator
Database
1. USER REQUEST
The user requests an interface for a
service. The appliance sends this request
to the Interface Manager, along with
appliance description information.
Service
iCrafter
Appliance
Description
Environment
Database
Service
Description
Remote Invocation
1
User
Active
Interface
UI
Renderer
2
4
Appliance
3
Generator
Selector
Generator
Processor
Interface
Manager
Discovery
Generator
Database
2. GENERATOR SELECTION
Upon receiving the request, the Generator
Selector selects an appropriate generator
based on the service(s) selected and the
appliance description.
Service
iCrafter
Appliance
Description
Environment
Database
Service
Description
Remote Invocation
1
User
Active
Interface
UI
Renderer
2
4
Appliance
3
Generator
Selector
Generator
Processor
Interface
Manager
Discovery
Generator
Database
3. GENERATOR PROCESSING
The Generator Processor runs the selected
generator with access to all context
(appliance, service, environment), which
outputs a UI description.
Service
iCrafter
Appliance
Description
Environment
Database
Service
Description
Remote Invocation
1
User
Active
Interface
UI
Renderer
2
4
Appliance
3
Generator
Selector
Generator
Processor
Interface
Manager
Discovery
Generator
Database
4. USER INTERFACE RENDERING
The UI description is sent to the UI
Renderer on the appliance, which renders
the interface and handles user interactions,
including service invocations over the EH.
Service
iStuff

Toolkit to prototype physical UIs






Flexible, lightweight components
Protocol independence
Platform independence with cross-platform capabilities
Ease of integration with existing applications
Support for multiple simultaneous users
Built on iROS



iStuff wireless devices (e.g., iButton, iSlider, iLight)
send/receive events to/from Event Heap via a proxy
PatchPanel translates events (e.g., iButton: light on | light off)
Apps get events from devices via PatchPanel + Event Heap
iStuff Device
Wireless connection
Transceiver
Application
PatchPanel
Proxy
TCP/IP connection
Event Heap
iStuff component
iStuff Architecture
iStuff Devices
iButtons
iSlider
iStylus
iMike
X10
RF
Input
iDog
Anoto Pen
iMouse
iBuzzer
Output
iSpeaker
iLight
Aura

CMU “Distraction-Free Ubiquitous Computing”
theme:





Mobility: allow users to move computational tasks
from one place to another
Maximize use of available resources
Minimize user distraction & drains on attention
Personal information aura: spans wearable,
handheld, desktop, & infrastructure devices
Requires new system design: hardware,
network, OS, middleware, & UI support
Aura Research Framework
Research Examples


Users

UI / Agents / Speech

Task-driven computing
Rapid service configuration
Tasks


Services



OS / Network



Physical Devices


Energy-aware adaptation
Multi-fidelity computation
Nomadic data access
Intelligent networking
Power-aware devices
Wearable computers
Aura Example Scenario





Fred prepares presentation & is late
Fred gets PDA & Aura transfers slides to it
Finishes slides on PDA walking to meeting
Aura finds location of meeting from calendar
& uploads slides to projection machine
Aura recognizes unfamiliar meeting
attendees faces & skips budget slides
Aura Architecture


User tasks are represented as set of distributed
services
 Database query interface to access distributed services
 Easily search mixed environments for needed services
 A key service is People Location Service
Environments self-monitor & re-negotiate task support
given runtime variation of capabilities & resources
 Can detect when task requirements (e.g., min response
time) aren’t met, & search & deploy alternative
configurations to support task
 Uses software architecture models (graph of
components & connectors) for runtime adaptability
Aura Architecture




Task Manager: tracks user context, environment, & task changes
Context Observer: provides info on physical context & reports
relevant events Task & Environment Managers
Environment Manager: handles access to distributed services
Suppliers: provide services for tasks (text editing, video playing,...)
Privacy in Aura

Aura services depend on user location info


Location policies: define who gets location
info, where, when, and at what granularity



Introduces privacy concerns
e.g., “Bob is allowed to know Alice’s location when
she is in NSH”
Similar to Bell Labs’ Houdini system’s “rules”
Not based on how user’s actually disclose
location
GAIA


Infrastructure for Active Spaces, UIUC
Like Aura:


Distributed services architecture accessed via
structured queries
Main difference:

Focus on user customization of apps based on
resources in current space
Toolkit Categories




Ubicomp infrastructure
Context-aware
Peripheral displays
Evaluation
Context



Context is key to ubicomp environment that
reacts correctly to user’s current situation
Definition: Info used to characterize the
situation of a person, place or object relevant
to the interaction between a human & a
computational service
Primary context types:


Identity (who), Location (where), Time (when), Activity (what)
Secondary context types:

e.g., for identity: email address, phone number, birthdate, etc
Context-Aware Applications


Definition: Uses context to provide relevant
info and/or services to the user, where
relevancy depends on the user’s task
3 categories:

Present information & services to users


Automatically execute services


e.g., nearby wireless networks, directions
e.g., phone vibrates when in meeting
Tag info with context for later retrieval

e.g., class lectures and notes
Toolkits for Context

3 main approaches:
1.
2.
3.
Widget model
 Context Toolkit
 Dey, Abowd, & Salber, 2001
 Based on architecture of GUIs
Distributed services model
 Context Fabric
 Hong & Landay, 2001
 Based on client-server dialog
Blackboard model
 iRoom
 Winograd, 2001
 Based on artificial intelligence apps (Engelmore, 1988)
Widget Model: Context Toolkit

Operating system view



Abstraction to hardware (sensors);
data is secondary
Similar to GUI input handling: uses widget
abstractions for handling context input
Uses 3 abstractions:

widgets, interpreters & aggregators
Benefits



Separates semantics from low-level input
sensor handling
Apps can focus on context, not sensors or
analysis of sensor input
Allows re-use of context widgets, interpreters
& aggregators
Drawbacks


Tight coupling – less robust to component
failure
Single-manager (Discoverer) control – less
flexible
Context Toolkit Architecture
Widgets




Encapsulate info about a single piece of
context (e.g., location or activity)
Hide details of underlying sensors from apps
Provide customizable & reusable building
blocks of context sensing
Provide historical context info
Widgets

Include optional services, or actuation of
output in environment



Like Interactors: responsible for input & output
e.g., light widget could sense light level & adjust
lights accordingly
Get context by subscription or polling

Callback methods used to notify subscribers
Aggregators




Widgets + ability to aggregate context info
Collect sensory info about an entity (person,
place, thing, etc.) from multiple sources into
one widget
Hide more complexity about context-sensing
mechanisms by combining multiple sensors
Enable maintainability and efficiency
Interpreters


Abstract sensor info
Use lower level sensory information to
deduce higher level info

e.g.,
identity + location + sound sensors 
“is there a meeting?”
Putting an App Together


Discoverer enables apps to locate and
subscribe to context components
Distributed infrastructure uses p2p comm.
Distributed Services Model

Database view (vs. OS view of CTK)



Middleware technologies that can be accessed
through a network


Data is primary, hardware is secondary
Focus on how data is modeled, distributed, protected, & used
Example: infrastructure=Internet; service=DNS
Any kind of device or application can use context
gathering & processing services by adhering to
predefined data formats & network protocols
Benefits



Interoperable: independent of hardware
platform, OS, & programming language
Decoupling: Sensors, services, & apps can
be changed w/o affecting each other
Simpler devices: sensors, processing power,
data, & services w/in infrastructure shared
Drawbacks


Applications must poll for data (proactively)
Servers are specialized per sensor
Context Fabric
Includes 3 pieces

1.
Distributed data model of people, places, things


2.
Context Specification Language (like SQL)


3.
each device has a space: private data goes to local space
multiple views of context (mine, yours, theirs)
apps specify what context info they need in uniform way
e.g., "What are the nearby movie theaters?”
Context service as API (per device)

interpret CSL queries and provide context info
Privacy in Context Fabric

Decentralized architecture allows it to
capture, store, process, & share in privacysensitive manner



Capture, store, & process data on user’s device
Provides mechanisms for end-user control of
sharing
Plausible deniability built-in

e.g., if request for context info denied, system can
reply “unknown”
Aura



Also follows distributed services model
Uses SQL-like interface to access a set of
distributed contextual services
Handles privacy through user-set policies
Blackboard Model





Communication goes through a centralized server
Components
 post messages shared blackboard
 subscribe to get posted messages matching specified pattern
Route by matching message content to subscriber's pattern
 Pattern matching uses AI (often apply sophisticated inference
procedures to logical representations)
Direct communication between components simulated by including
an identifier for path endpoints as a field in message
Announce-listen style
 Components that provide services periodically post events
 Component that uses the data listens to these events, & if one does
not appear within the expected time can initiate restart operations
Benefits


Ease of configuration
Robustness to component failure



When component fails, only communication link
broken is from it to blackboard
Dependent components can detect failure and
call for restart
Simplicity provided by a uniform
communications path (to the blackboard)

No complex protocol for finding ports or
resources, establishing connections
Drawbacks

Communication less efficient


Each communication requires 2 hops
General message structure not optimized for
particular data or interaction protocol
iRoom & iStuff

Event Heap: the blackboard & central
communication server
Review of Context Approaches

Widget Model – CTK


Distributed Infrastructure Model – ConFab, Aura, Gaia


Widget architecture where Discoverer acts as a manager of
context widgets
Client-server architecture where client apps communicate
directly w/ services using structured language
Blackboard Model – iRoom, iStuff

Central server architecture where apps communicate only w/
server & events are routed using subscriber pattern matching
Toolkit Categories




Ubicomp infrastructure
Context-aware
Peripheral displays
Evaluation
Peripheral Displays



Provide awareness with min attention
Separate from primary task
Important to ubicomp vision of many devices to 1 user


Non-focal display not used unless providing peripheral info
Enable you to monitor many info sources while
maintaining a calm environment

But, only calm if designed to manage attention they attract
Why is creating PDs hard?


Need to abstract info to be glance-able
Need mechanisms for dynamically
managing attention PDs attract:



Deciding attention levels to attract
(notification levels)
Displaying info appropriately (transitions)
Peripheral Display Toolkit (PTK)

Supports these 3 key issues in PD creation
Example PTK Applications



Remote Activity
 Social Guitar
 Audio Monitor
 Motion Monitor
 Remote Awareness Display
Bus Displays
 Bus Mobile
 Bus LED
Instant Messenger Status
Social Guitar
IM Picture Frame
Orb showing remote activity
Bus LED
BusMobile
PTK Architecture
1.
Support for managing impact on human
attention using abstraction, notification
levels, and transitions
2.
Simplified code design and
code re-use
3.
Library of common PD components
Input
Abstractor
Output
Notification
Map
Transition
Architecture Diagram
Input-Side
Input 1
Abs. 1
Discovery
Server
Output-Side
Peripheral Display 1
(Abstractors)
(Notification
Maps)
Abs. 1
N.M. 1
.
.
.
Abs. 2
N.M. 2
Out 2
…
Input N
Abs. N
N.M. N
Out N
Abs. N
Input 2
PTK
Discovery
Server
(Output Widgets w/
optional Transition)
Trans
Out 1
Peripheral Display 2
Abs. 1
N.M. 1
Abs. 2
N.M. 2
Abs. N
N.M. N
Out 1
Trans
Out 2
…
Out N
Library Components

Input

audio, camera, Phidgets, Context Toolkit, online calendars,
news, stocks, weather, Web page parser, serial port
Library Components

Input


audio, camera, Phidgets, Context Toolkit, online calendars,
news, stocks, weather, Web page parser, serial port
Output

ticker text, Ambient Orb, Phidgets
Library Components

Input


Output


audio, camera, Phidgets, Context Toolkit, online calendars,
news, stocks, weather, Web page parser, serial port
ticker text, Ambient Orb, Phidgets
Abstractors

motion, people counting, voices, phone ringing
Library Components

Input


Output


ticker text, Ambient Orb, Phidgets
Abstractors


audio, camera, Phidgets, Context Toolkit, online calendars,
news, stocks, weather, Web page parser, serial port
motion, people counting, voices, phone ringing
Notification

exact match, threshold, contains, degree of change
info urgency
user attention
Toolkit Categories




Ubicomp infrastructure
Context-aware
Peripheral displays
Evaluation
Evaluation

3 tools for evaluating people in natural settings
from Intille et al.:
1. Context-aware experience sampling based on
GPS or other sensors

2.
Ubiquitous sensing system that detects
environmental changes

3.
e.g., ask questions when user goes to store
e.g., tape-on touch, proximity, & light sensors
Image-based experience sampling system

i.e., take a picture & ask user about it later
Summary: Ubicomp Goals

Scalable interfaces




Natural interaction




multiple devices
inter-device communication
multiple users
getting away from the desktop
augmenting surrounding environment
provide context-appropriate services
Calmness


more info w/o overwhelming users
aware of user’s interruptiblity
Summary: Ubicomp Toolkits

Ubicomp infrastructure


Context-aware


Context Toolkit, Context Fabric, iRoom
Peripheral displays


iROS, iStuff, Aura, Gaia
Peripheral Display Toolkit
Evaluation

Intille’s toolkit
Thanks!
CSCW

Groupware Architectures



Phillips, W.G., 1999. Architectures for Synchronous
Groupware, Tech. Rep.. http://phillips.rmc.ca/greg/pub/
Greenberg, S. and Roseman, M., 1999. Groupware
Toolkits for Synchronous Work. In: Beaudouin-Lafon, M.
(Ed.), Trends In CSCW'99, No. 7 in Trends in Software,
John Wiley & Sons, New York, NY, USA, ch. 6, pp. 135–
168.
Roseman, M. and Greenberg, S., 1992. GROUPKIT: a
groupware toolkit for building real-time conferencing
applications. In: Proceedings of the conference on
Computer-supported cooperative work, ACM Press,
pp. 43–50. http://doi.acm.org/10.1145/143457.143460