Toolkits for Physical Objects
(TUIs)
Jack Li
Advanced User Interface Software
November 30
Fall 2004
1
Outline

Motivation for TUIs

Input Technologies

TUI Toolkits
2
MOTIVATION FOR TUIS
Current Limitations of UIs
Desktops and laptops suck in our entire
attention
 As a result we live in two worlds: the real
physical world and the virtual cyber world
 Constantly “wired” in order to be in both
worlds
 Loss of rich interactions found in everyday
things (particularly haptic)

3
MOTIVATION FOR TUIS
Motivation
Driven by ubiquitous computing
 Both people and technology moving away
from the desktop into the environment
 Input: different ways of interacting
 Output: different ways of displaying

4
MOTIVATION FOR TUIS
Tangible User Interfaces





TUIs augment the physical world by integrating
digital information with everyday physical
objects
Goal: to bridge the gap between bits and atoms
1. Interactive surfaces
2. Coupling information with physical objects
3. Ambient media for background awareness
5
Outline

Motivation for TUIs

Input Technologies

TUI Toolkits
6
INPUT TECHNOLOGIES
Physical Objects as Input
Pretty much anything … walls, papers,
utensils, clothes hangers!
 Inspiration

sci-fi novels, own imagination
 everyday life, current practices

7
INPUT TECHNOLOGIES
Devices



Cameras
Electronic tags (RFID technology)
Sensors






Light
Motion
Heat
Force
Motors
Orbs
8
INPUT TECHNOLOGIES
Framework for the GUI
(a)model-view-controller
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
9
INPUT TECHNOLOGIES
Framework for the TUI
(b)MCRpd: model-controlrepresentation(physical and digital)
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
10
Outline

Motivation for TUIs

Input Technologies

TUI Toolkits
Phidgets (2001)
 Calder (2004)
 Papier-Mâché (2004)

11
Toolkit Goals
Same as when GUIs were built
 Make easy for non-specialized experts to
create (should not need a vision or
hardware guru on the team)
 Hide dirty implementation details
 Abstract proper functionality with welldefined API

12
PHIDGETS
Overview
First toolkit for physical objects
 Reusable physical devices with proper
abstraction for creating a wide variety of
applications
 An API that had the right amount of
abstraction

13
PHIDGETS
GUI Differences

Connection manager


Identification


Way of detecting when devices go on/off
Every physical object should have a virtual
counterpart
Simulation

A way to test code even if phidget not
present
14
PHIDGETS
Architecture
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
15
PHIDGETS
The Physical Device

Primitive device components


QuickTi me™ a nd a
TIFF (LZW) de com press or
are need ed to se e th is p icture.


Circuit board


Sensors
Motors
Switches
CY7C63000 USB micro-controller
Communications layer

USB communication standard
16
PHIDGETS
Wire Protocol


Low-level software in Windows and
microcontroller for communication
that’s hidden from end programmers
Commands from phidgets




QuickTi me™ and a
TIFF ( LZW) decompressor
are needed to see thi s pi ctur e.
Unique Id numbers
Device type (servo, powerbar)
Specific events (tag for RFID reader)
Commands from Windows

Set device state (output like servo)
17
PHIDGETS
Computer Software

PhidgetManager

Connection manager
•
•
•
•

Event onAttach()
Event onDetach()
Int count
Enum item
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Phidget-specific devices

All have:
• String deviceType
• Boolean isAttached
• Long serialNumber

RFID
• onTag()

Servo
• onPositionChanged(), motorPosition(index)
18
PHIDGETS
Simple Application

Phidget Manager
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.

Phidget Servo
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
19
Outline

Motivation for TUIs

Input Technologies

TUI Toolkits
Phidgets (2001)
 Calder (2004)
 Papier-Mâché (2004)

20
CALDER
Main goals
Primarily geared towards
interaction/product designers
 Support for more refined products
 Support on the hardware side

21
CALDER
Supported Devices
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Small size is big win!
22
CALDER
Fluid prototyping
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Objects easily embedded!
23
CALDER
Communication

Wired
PIC16C745 C with USB interface engine
 Same as Phidgets (and iStuff)


Wireless
Tradeoff between size and power (perhaps
Moore’s law will remedy)
 Small batteries, antenna, and pulse-width
modulation

24
CALDER
Software

Surrogates (GUI widgets) to manipulate
physical object output
25
Outline

Motivation for TUIs

Input Technologies

TUI Toolkits
Phidgets (2001)
 Calder (2004)
 Papier-Mâché (2004)

26
PAPIER-MACHE
Overview
Emphasis on software side side
 Input model that provides common
programming abstractions
 Focus on objects being lightweight

27
PAPIER-MACHE
Software Abstractions

Phob—encapsulates information related to this
physical object




PhobListener—listens for Phobs produced



All—source producer, time
RFIDPhob—tag
VisionPhob—mean color, bounding box, rotations
phobAdded(), phobUpdated(), phobRemoved()
PhobProducer—actual devices that generates
events
PhobManager—maintains PhobProducers
28
PAPIER-MACHE
A Common Event Model


PhobListener is the interface among all devices
Methods are the same


Vision


phobAdded() when new object detected, phobUpdated() when
old object changed (size, color), phobRemoved() when old
object disappears
RFID (different from what the paper says)


phobAdded(), phobUpdated(), phobRemoved()
phobAdded() when tag detected, no phobUpdated(),
phobRemoved() when tag disappears
Facilitates easier retargetting and substitutions (allows
for more exploration and faster, prototyping)
29
PAPIER-MACHE
Marble Answering Example

1. Get the manager that keeps track of all
input devices (PhobProducers)
RFIDManager manager = RFIDManager.getManager();

2. Set up a map associating all objects with
an audio clip
AssociationFactory factory = new AudioClipFactory();
AssociationTreeTable assocMap = new
AssociationTreeTable(factory);

3. Attach this map so new objects are
assigned new clips and old ones get played
manager.addPhobListenerForAllProducers(assocMap);
30
PAPIER-MACHE
A Behavior: Associations
A common behavior we noticed
 New objects get stored/assigned, old
objects get retrieved a database of sorts
 An extension of PhobListener

31
PAPIER-MACHE
A Behavior: Associations

AssociationTreeTable


ObjectClassifier


addClassifierFactoryPair(Classifier, Factory)
isMemberOfClass(Phob phobToCompare)
AssociationFactory

createAssociationEltForPhob(Phob phob)
32
PAPIER-MACHE
Monitoring Window
33
PAPIER-MACHE
Device state—connection mgr
34
PAPIER-MACHE
Camera Feedback
35
PAPIER-MACHE
Association Map
36
PAPIER-MACHE
Extras
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
37
Summary
38
Major Toolkit differences

Phidgets


Calder


Complete, balanced, varied assortment of
physical objects easy to program with
Small size allows for more fluid prototyping at
the PHYSICAL level
Papier-Mâché

Unified event model allows easier
development and retargetting at the
SOFTWARE level
39
Conclusion
TUIs seek to augment the physical world
with digital information
 Toolkit support allows non-experts to
develop TUI application, like GUI toolkits
 Successful toolkits must manage physical
connections, gather/alter relevant physical
state information, and provide good
feedback for these added complexities

40
Future work

Input adaptation


Input models


M to N mapping of input devices
What other kinds of behaviors can be
abstracted
Will be more clear with the development
of more applications
41
References





Hiroshi Ishii , Brygg Ullmer, Tangible bits: towards seamless interfaces between people, bits and atoms,
Proceedings of the SIGCHI conference on Human factors in computing systems, p.234-241, March 2227, 1997, Atlanta, Georgia, United States
Ullmer, B. and H. Ishii, Emerging Frameworks for Tangible User Interfaces, in Human-Computer
Interaction in the New Millennium, Addison-Wesley. pp. 579 -- 601, 2001.
Saul Greenberg, Chester Fitchett. "Phidgets: easy development of physical interfaces through physical
widgets," Proceedings of the 14th annual ACM symposium on User interface software and technology,
November 11-14, 2001, Orlando, Florida
Scott R. Klemmer, Jack Li, James Lin, James A. Landay. "Papier-Mache: toolkit support for tangible
input," Proceedings of the 2004 conference on Human factors in computing systems, p.399-406, April
24-29, 2004, Vienna, Austria
Lee, J.C.; Avrahami, D.; Hudson, S.E.; Forlizzi, J.; Dietz, P.H.; Leigh, D.L., "The Calder Toolkit: Wired
and Wireless Components for Rapidly Prototyping Interactive Devices", Designing Interactive Systems
(DIS), ISBN: 1-58113-787-7, pp. 167-175, August 2004
42