Lecture 19:
Physical Gadgets and their
Interaction Techniques
Brad Myers
05-440/05-640 :
Interaction Techniques
Spring, 2016
© 2016 - Brad Myers
1
Definitions and Synonyms






Ubiquitous computing (UbiComp) - computing everywhere
and anywhere
Pervasive computing – (no separate definition)
Ambient intelligence (mostly used in Europe) – environment
is instrumented so it is sensitive and responsive to people
Information appliances – Smartphone or PDA
Context-aware computing – mobile device that knows its
surroundings, such as location, light, sound, etc.
Tangible user interfaces (TUIs) -- person interacts
with digital information through the physical
environment


Formerly “graspable UIs”
Has its own conference series:


TEI’16: 10th International Conference on
Tangible, Embedded and Embodied Interaction,
The Netherlands, February 14-17, 2016
Internet of Things (IoT)
2
© 2016 - Brad Myers
Definitions, cont.

Physical Gadgets

Are to physical (tangible) user interfaces what
interaction techniques are to graphical user interfaces


An interaction technique embodied in a physical entity


Adapted from [Greenberg’01]
Must be reusable
Many other TUIs are
tabletop interactions
with physical objects
sensed on a table with
a projector
3
© 2016 - Brad Myers
Scope


There are lots of interesting, cute,
even useful applications of tangible
and ubiquitous user interfaces
Most are not interaction techniques

E.g., Ambient displays – no interaction


Mankoff’s BusMobile
E.g., Tangible applications – not a
reusable widget

Bottles that play sounds when opened
4
© 2016 - Brad Myers
Logo “Turtle”



From 1967 by Daniel G. Bobrow, Wally
Feurzeig, Seymour Papert and Cynthia Solomon
at MIT
Simplified programming for children
Originally drove a physical turtle on the floor with
a physical pen


Pen up / down
Walk forward / turn
5
© 2016 - Brad Myers
Lego Mindstorms

Introduced in 1998



Original kit contained light sensors,
buttons, touch sensors, motors, etc.
1st version programmed using
“RCX code”



Named after Seymour Papert’s book
Blocks language implemented in
Macromedia Director
Could also be downloaded
from other languages
Slow processor, low-quality
sensors and actuators
6
© 2016 - Brad Myers
Phidgets



Saul Greenberg and Chester Fitchett. 2001. Phidgets: easy development of physical interfaces through
physical widgets. In Proceedings of the 14th annual ACM symposium on User interface software and
technology (UIST '01). ACM, pp. 209-218. http://doi.acm.org/10.1145/502348.502388
“Physical widgets”
Previously was very difficult to build TUIs




Encapsulated complexities of using physical objects



Had to build custom hardware and microprocessors
Soldering, circuit design (EE), assembly-language programming, etc.
Lots of new sensors
Lights, motors, sensors, cameras, switches, etc.
Mostly USB
Interactive since sensors for motion, light, sound, etc.

Sensor+control counts as interaction technique, not just a moving flower

Programmed (originally) in Visual Basic
Simulation mode to help create the software
Formed a company to market his phidgets

Video, 6:10 (2001)


7
© 2016 - Brad Myers
Controllers







Started about 2005 with Arduino
http://arduino.cc/
Single-board microcontroller
Open source electronics prototyping platform
Now about $9 to $30 each
Easy to program and attach
devices to
Newer alternatives (from Chris Harrison)
 https://www.raspberrypi.org/
 http://beagleboard.org/
 http://www.pjrc.com/teensy/ - USB
 http://leaflabs.com/devices/maple/
 If you want to start from a phone, there is the IOIO-OTG
board for android:

https://www.sparkfun.com/products/12633
© 2016 - Brad Myers
8
Toys and Robots

Allison Druin. 1988. NOOBIE: the animal design
playstation. SIGCHI Bull. 20, 1 (July 1988), 45-53.
http://dx.doi.org/10.1145/49103.49106


Many toys


Giant stuffed animal with sensors and a
screen
E.g., Furby from 1998 by Tiger Electronics
Many robots




Some programmed
by example
Rethink Robotics
Nao
Boston Dynamics
9
© 2016 - Brad Myers
Research: Hiroshi Ishii


Tangible Media Group: http://tangible.media.mit.edu/
Dozens of projects dating back to 1990


But most are not “interaction techniques”
One that is:

John Underkoffler and Hiroshi Ishii. 1999. Urp: a luminous-tangible
workbench for urban planning and design. In Proceedings of the SIGCHI
conference on Human Factors in Computing Systems (CHI '99). ACM, pp.
386-393. http://doi.acm.org/10.1145/302979.303114

Physical tools for measuring, changing building material,
turning on wind, changing light paths, etc.
10
© 2016 - Brad Myers
Ishii, cont.

Hiroshi Ishii, Dávid Lakatos, Leonardo Bonanni, and Jean-Baptiste Labrune. 2012. Radical atoms: beyond
tangible bits, toward transformable materials. interactions 19, 1 (January 2012), 38-51.
http://doi.acm.org/10.1145/2065327.2065337

Includes a survey of tangible Uis
Lists lots of toolkits to create TUIs
“Tangible design seeks an amalgam of thoughtfully



designed interfaces embodied in different materials and
forms in the physical world—soft and hard, robust and
fragile, wearable and architectural, transient and enduring.”
Future: physical-digital “atoms” that can transform, conform
and inform

E.g., “clay” that changes its own shape based on rules, user
commands, & constraints
11
© 2016 - Brad Myers
Recent Research: Skweezee

Karen Vanderloock, Vero Vanden Abeele, Johan A.K. Suykens, and Luc
Geurts. 2013. The skweezee system: enabling the design and the
programming of squeeze interactions. InProceedings of the 26th annual
ACM symposium on User interface software and technology(UIST '13).
ACM, pp. 521-530. http://doi.acm.org/10.1145/2501988.2502033

Soft tangible objects, filled with conductive
padding and embedded sensors
(eight electrodes)
Toolkit for defining
squeeze gestures by example


Learns from a single example
12
© 2016 - Brad Myers
“Wearable” technology


Fitbits, etc.
Heddoko “smart” clothing, etc.

Sensors in shirts/pants (video, 1:21)
13
© 2016 - Brad Myers
Hand-Held Projectors

Xiang Cao, Clifton Forlines, and Ravin Balakrishnan. 2007. Multi-user
interaction using handheld projectors. UIST '07. ACM, 43-52.
http://dx.doi.org/10.1145/1294211.1294220

Each person holds a projector
with integrated camera
Interact by moving projector
or items in the scene
Extra buttons on projector
or externally
Video (6:08)
Current projectors are the size
of a smartphone





Sony Portable HD Mobile Projector (MPCL1)
14
© 2016 - Brad Myers
Hybrid mobile device and
physical


Sensors on smartphones – physical
manipulate the phone itself
Also mobile + environment

Early example: Jun Rekimoto and Masanori
Saitoh. 1999. Augmented surfaces: a spatially continuous
work space for hybrid computing environments. CHI '99, 378385. http://dx.doi.org/10.1145/302979.303113 video 3:14

Andy Wilson at Microsoft
research has lots, e.g.: Andrew D. Wilson. 2005.
PlayAnywhere: a compact interactive tabletop
projection-vision system. In UIST '05. ACM, 83-92.
http://dx.doi.org/10.1145/1095034.1095047 videos
15
© 2016 - Brad Myers
Scott Hudson’s class




“Gadgets, Sensors and Activity Recognition
in HCI”
http://www.hcii.cmu.edu/courses/appliedgadgets-sensors-and-activity-recognition-hci
Every spring
This semester, in SCR 172 – big room
16
© 2016 - Brad Myers