CMPUT 301: Lecture 32
Out of the Glass Box
Martin Jagersand
Department of Computing Science
University of Alberta
Ubiquitous Computing
• What:
– any computing technology that permits human
interaction away from a single workstation
– e.g.,
– handheld or portable devices, pen-based devices
– Intelligent interfaces to common appliances
– Information capture and vizualization
– Automated buildings
– Service robotics
2
Automated Capture,
Integration, and Access
• Problem:
– remove the burden of human record or note
taking
– provide automatic capture of individual and
group experiences
– support access to richly integrated record of
events
– synchronize multiple information streams
3
Automated Capture,
Integration, and Access
• Electronic whiteboard:
– e.g.,
SMART Board
4
Automated Capture,
Integration, and Access
• Education:
– e.g., eClass
– promote “heads up”
learning
– electronic whiteboard,
audio, video,
projectors, network,
etc.
– web-based timeline of
events during the class
– effective?
5
Automated Capture,
Integration, and Access
• Research issues:
– granularity of integration
– should every action be linked to audio or video at
that instant?
– supporting revision during access
– provide the capability to modify a record afterwards
during access?
– supporting networked interaction
– collaborative note taking
– real-time, anonymous feedback
6
Applications for vision in User Interfaces
• Interaction with machines and robots
– Service robotics
– Surgical robots
– Emergency response
• Interaction with software
– A store or museum information kiosk
7
Service robots
• Mobile manipulators, semi-autonomous
DIST
TU Berlin
KAIST
8
TORSO with 2 WAMs
9
Service tasks
10
This is completely hardwired! Found no real task on WWW
But
• Maybe first applications in tasks humans
can’t do?
11
Why is humanlike robotics so hard to
achieve?
• See human task:
– Tracking motion, seeing gestures
• Understand:
– Motion understanding: Translate to correct
reference frame
– High level task understanding?
• Do:
– Vision based control
12
Types of intelligent machine systems
Auton
omy
Preprogrammed
systems
Programming by
demonstration
Tele-assistance
Supervisory control
Generality
13
Interaction styles
If A then
end
Conventional
:
• Low bandwidth interaction
• Partial or indirect system state
displayed
• User works from internal
mental model
14
Interaction styles
Direct Manipulation:
•High bandwidth interaction
•Interact directly and intuitively with objects (affordance)
•See system state (visibility)
•(Reversible actions)
15
Examples of Direct Manipulation
•
•
•
•
•
•
Drawing programs e.g. Mac Paint
Video games, flight simulator
Robot/machine teaching by showing
Tele-assistance
Spreadsheet programs
Some window system desktops
But can you always see effects (visibility)?
16
Why direct manipulation?
• Recognition quicker than recall.
• Human uses “the world” as memory/model
• Human skilled at interacting spatially
How quick is direct?

Subsecond! Experiments show human
performance decreased at 0.4s delay.
17
Vision and Touch based UI
• Typical UI today: Symbolic, 1D (slider), 2D
• But human skilled at 3D, 6D, n-D spatial
interaction with the world
18
Seeing a task
• Tracking movement
– See directions, movements in tasks
• Recognizing gestures
– Static hand and body postures
• Combination: Spatio-temporal gestures
19
Tracking movement
• Tracking the human is hard:
– Appearance varies
– Large search space, 60 parameters
– Unobservable: Joint angles have to be inffered
from limb positions, clothing etc.
– Motion is non-linear.
– Difficult to track 3D from 2D image plane info
– Self occlusion of limbs
20
Trick 1:
Physical model
• Reduce number of DOF’s by coupled model
of articulated motion (Hedvig, Mike)
21
Human body tracking
22
Trick 2:
Use uniqueness of skin color
• Can be tracked at real time
23
Gestures:
• Identifying gestures is hard
– Hard to segment hand parts
– Self occlusion
– Variability in viewpoints
24
Trick 3:
Scale space
• Define hand gesture in course to fine terms
25
Trick 4:
Variability filters
26
Programming by Demonstration
• From assembly relations
• From temporal assembly sequence
– Segmenting manipulation sequence into parts
(subtasks) is hard
• Using a gesture language
27
Tele-assistance:
• Gestures + context
• Here based on exoskeleton (intrusive)
28
Robust manipulations
29
System for Vision-based User Interfaces:
• Vision based
“Tele
Assistance”
• Describe task
and objects by
gestures and
pointing
• Visual language
maps to physical
actions
30
Vision and Sensory based UI
31
Composite Task: Solving a Puzzle
32
Vision based UI: summary
• Most aspects of robot see – robot do are
hard
• Conventional methods are
– Incapable of seeing task
– Incapable of understanding what’s going on
– Incapable of performing human manipulation
tasks
• Uncalibrated methods are more promising
33
Challenge: How to replicate biological
sensory-motor capabilities?
• Sensory and motor areas in cortex
34
Context-Aware Computing
• Problem:
– free the user from the constraints of stationary
desktop computing
– customize computational services based on
knowledge of the user’s physical context
35
Context-Aware Computing
• Location-aware
computing:
– e.g.,
Xerox PARC
PARCTAB
– wireless network of
pen-based, palm-sized
computers
– constant connectivity
– location reporting
– nearest resources
36
Context-Aware Computing
• Location-aware
computing:
– e.g.,
Olivetti Active Badge
– locate individuals
within a building
– badge transmits unique
infra-red signal
– networked sensors in
the building
37
Context-Aware Computing
• Location-aware
computing:
– e.g.,
Georgia Tech
Cyberguide …,
car navigation systems
– tour guide
– locate position of user
on a map
– Global Positioning
System
38
Context-Aware Computing
• Research issues:
– there is more to context than position
– facial expressions
– emotions
– context-awareness behind the scenes
– automatic context tracking
– loss of privacy?
39
Ubiquitous Software Services
• Problems:
– scalable interface
– availability on any device handy to the user
– e.g., Java virtual machine …
– service integration
– adaptability to a changing set of services the user
wants
– i.e., loose, end-user integration
– e.g., Apple OpenDoc
40
Ubiquitous Software Services
• Research issues:
– flexible input/output modality
– avoid input mode used by the sender (e.g., phone,
email, etc.) dictating the output mode used by the
receiver
– unobtrusive ubiquity
– provide computing power in a less intrusive user
interface than that of a desktop computer
41
Ubiquitous Software Services
• Personal Digital
Assistant:
– focused function
– optimized for mobility
– tapping, writing, and
drawing with a pen
– intelligent assistant
– new and custom
applications
– simple
• Portable
Computer:
– general purpose
– optimized for stationary
operation
– typing, pointing, and
clicking with trackpoint and
keyboard
– scripting and
macros
– existing desktop applications
– complex
42
Virtual Reality
• What:
– computer-generated simulation of a world in
which the user can interact
(e.g., move about)
43
Virtual Reality
• Technologies:
–
–
–
–
–
–
–
head-mounted displays
shutter glasses
glove-based input
3D trackers
surround sound
vests
etc.
44
Virtual Reality
• Examples:
– Sensorama
(stereo sound,
vibrations, aromas,
wind)
45
Virtual Reality
• Examples:
– 3D games,
flight simulators, war
simulators
46
Virtual Reality
• Examples:
– architectural tours,
phobia therapy, product
information, protein
chemistry
QuickTime™ and a
Sorenson Video decompressor
are needed to see this picture.
47
Virtual Reality
• Immersive VR:
– e.g., VizRoom
48
Virtual Reality
• Augmented reality:
– project electronic
images over the real
world
– e.g., heads-up displays,
road navigation, virtual
objects
49
Telepresence
• What:
– work is controlled and
experienced in a
different physical
location than where the
work itself resides
– e.g., remote surgery,
video conferencing,
telecubicles, etc.
50
Information Visualization
• Cone trees:
51
Information Visualization
• Tree maps:
52
Information Visualization
• Hyperbolic trees:
QuickTime™ and a
GIF decompressor
are needed to see this picture.
53
Information Visualization
• Perspective wall:
54
Information Visualization
• SHriMP views:
55
Information Visualization
56
End
• What did I learn today?
• What questions do I still have?
57