The Right Information
John S. Zelek
Dept. of Systems Design Engineering
Intelligent Human Machine Systems Lab
University of Waterloo
Research Problems
• making sense of
where I am & what
is around me
(computer vision)
• how to convey
information in
tactile form &
provide
telepresence
(wearable haptics)
Where am I ?
WhatÕs out
there?
Applications
• Assistive Technology
• way-finding for people who are
blind
• Surveying & Mapping
• robotics, vehicles, wearable
computing
• MIS Surgery haptics
• laparoscopic, robotic
• rehabilitation
Talk Structure
1.Way-finding for People who are
Blind (5-19)
2.Computer Vision (20-35)
depth, object detection/recognition, context, SLAM, IR,
etc.
3.Wearable Haptics (36-52)
4.Design (53-56)
5.In closing (57-58)
1. Wayfinding...
Statistics
• 11.4 million visually impaired in U.S.
• blindness prevalence increases with age &
average age gradually increasing
• older population also subjected to loss of
hearing, physical & cognitive impairments
• variety: when (children, young, old) & type
(retinal, cortical) of sight loss
• quality of life, dignity, independence
1. Wayfinding...
more statistics
• 180 million people worldwide have visual
impairments, 40-45% are blind (WHO)
• people with visual impairments lose their lives to
pedestrian accidents more frequently then their
non-disabled peers
• US (2001) almost 5,000 pedestrians died, 78,000
were injured while crossing the street, walking to
school, waiting for a bus
• cars at intersections hit 8% of visually impaired
respondents (ACB), 28% reported white canes
were run over by careless motorists
1. Wayfinding...
definitions
• A person with a visual impairment:
•
•
•
•
•
would be unable to recognize a friend from across a
room, even when wearing glasses
not be able to read a regular newspaper print, even
when wearing glasses
report their own vision as poor or very poor
report some other trouble seeing, even with glasses
be blind in one or both eyes
• A person with a severe visual impairment:
• A person with a moderate impairment:
1. Wayfinding...
Assistive Technologies
• Technologies for VI have been
developed in the areas of:
1.activities of daily living
2.computer access
3.access to graphics
4.way-finding.
1. Wayfinding...
Existing AT (Assistive Technology)
• most prevalent:
• long cane & guide dog
• Electronic travel aids (ETAs) using sonar or laser
shortcomings:
• user must actively scan environment (time-consuming,
requires conscious effort
• user must perform additional measurements to determine
dimensions & shape of obstacles
• interference between environmental sound cues & acoustic
feedback
• ETAs using GPS (Braille Note, Trekker) - economics
1. Wayfinding...
Spaces
• personal
• what can we touch
• long cane
• intermediate
• what can we see
• cameras (vision)
• global frame of reference
• GPS, compass
1. Wayfinding...
Spatial Representation
Auditory
Information
Linguistic
Spatial
Representation
Representations
Visual
Information
Haptic
Motor
Information
Information
1. Wayfinding...
Our Work
QuickTime™ and a
H.264 decompressor
are needed to see this picture.
1. Wayfinding...
Lab Experiments
1. Wayfinding...
Details
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
left
? obstacles upcoming = motor on
? free space = motor off
? intensity strength inversely
proportional to depth
In front of
right
1. Wayfinding...
Experiment Results
Existing aid glove
# of
subjects
Obstacle
course 1
65%
76%
5
Obstacle
course 3
65%
57%
4
? reference is minimal hitting probability path
? quantitatively, inconclusive as a result of small sample size
? qualitatively, comparable
1. Wayfinding...
Anecdotal comments
• “I’m taking this home. It’s unbelievable… its
like an extension of my cane, like having many
canes…ever neat, I don’t want to take it off”
JIM
• “felt really comfortable, not cumbersome at all,
fit really well, I got some good information”
MARTY
• “thought it was good”
• “I want one”
• “neat”
1. Wayfinding...
Subject
(2002-3)
Age
Date
Classified
Visual
Acuity
Eye Condition
1
49
2
25
1978
Total
Retrolental fibroplasia
AC, B
3
30
2000
Total
Brain tumour
AC
4
33
1996
20/400
Diabetic retinopathy
AC, SM
5
45
2001
20/200
Diabetic retinopathy
AC, M
6
50
1989
20/400
Macular degeneration
AC, M
7
57
1991
20/300
Lebers optic atrophy
CCTV, M,AC
8
72
1997
Light
Brain tumour
SM,M,CCTV
9
60
1991
20/200
Cystoid macular degeration AC
Total
Adaptive
Equipment
AC, B
1. Wayfinding...
Experimental Factors
•
•
•
•
•
•
•
•
•
when was vision lost? (birth, childhood, later in life, recently)
where is the visual impairment? (retinal, optic nerve, cortical)
subject’s age
subject’s sex
subject’s mental capability (mental imagery, memory)
emotional state
general health
left/right handed
current independence level
•
•
•
•
indoor-outdoor
obstacle configuration complexity
static/dynamic obstacles
terrain complexity (flat, staircases, ramps, uneven)
1. Wayfinding...
Haptic Compass
Affordable, tactile interpolation,
orientation & distance tactile
communication
2. Computer Vision
• Probabilistic Inference - Tracking
• Face Detection
• Object Detection/Recognition
• Depth - stereo, optical flow
• Context
• SLAM (Simultaneous Localization &
Mapping)
• Other Modalities, Contexts (IR)
2. Computer vision
Visual Tracking (Probabilistic
Inference)
QuickTime™ and a
H.264 decompressor
are needed to see this picture.
QuickTime™ and a
H.264 decompressor
are needed to see this picture.
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
2. Computer Vision
Face Detection (Wu)
2. Computer Vision
Face Detection (Fazl)
QuickTime™ and a
H.264 decompressor
are needed to see this picture.
Face Recognition:
Statistical
Local Feature Analysis (LFA) for feature
point
extraction.
FERET: 97.1%
2. Computer Vision
Object Category
Detection/Recognition
QuickTime™ and a
H.264 decompressor
are needed to see this picture.
novel probabilistic constellation model of
triangular graphs, robust against partial
occlusion, scale, rotation, and affine
transformations
2. Computer Vision
Depth
2. Computer Vision
Depth (stereo)
QuickTime™ and a
Microsoft Video 1 decompressor
are needed to see this picture.
2. Computer Vision
Commodity Based Stereo (Bromley)
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
QuickTime™ and a
H.264 decompressor
are needed to see this picture.
2. Computer Vision
Context
• Top-down - Use the frequency content of the image at
various scales and orientations (holistically)
Indoor office
Indoor lounge
Outdoor urban
Outdoor Suburban
2. Computer Vision
Context - Sense of Place
2. Computer Vision
SLAM
QuickTime™ and a
GIF decompressor
are needed to see this picture.
2. Computer Vision
SLAM
QuickTime™ and a
H.264 decompressor
are needed to see this picture.
2. Computer Vision
Optical Flow
Towards Bayesian real-time Optical Flow
(Zelek:02,04)
2. Computer Vision
Optical Flow
•
•
•
•
time to collision (depth)
foe
motion detection
inertial information when fused
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
2. Computer Vision
Optical Flow - Particle Filter (spatial,
temporal)
QuickTime™ and a
MPEG-4 Video decompressor
are needed to see this picture.
2. Computer Vision
IR
Sparse Disparity Map from Uncalibrated
Infrared Stereo Images
Approach:
? Feature extraction (using phase congruency)
? Stereo matching (Gabor coefficients)
? Matching refinement (outlier elimination using RANSAC)
Left
Right
Ground Truth
Disparity map
3. Wearable Haptics
• From Greek
“haptikos” - ‘able to
touch’
• Tactile
(cutaneous) &
kinesthetic
(proprioceptive)
3. Wearable Haptics
Physiology
2 point thresholds (cm)
Chin
0.55
Forehead
1.9
Nose
0.9
Ear
2.4
Top shoulder
4.4
Inner forearm
4.6
Back of hand
1.7
2nd finger front
0.25
2nd finger back
0.25
chest
3.8
back
4.2
Lower leg front
4.0
3. Wearable Haptics
Physiology
Channel
Pacinian
Meissner Õ
s
Corpuscle Corpuscle
NPI
Ruffini
Ending
NPII
Merkel
Cells
Hz
40-800
U-shaped
3-100
flat
15-400
U-shaped
< 100
flat
attribute
Vibration,
tickle
Flutter,
edges
stretch,
shear
pressure
Receptive
field mm 2
(median)
10-1000
(101)
1-100 (12.6)
10-500
(59)
2-100
(11.0)
Receptors/c 21 (9)
m2 fingertip
(palm)
140 (25)
9 (15)
70 (8)
Temp. dep.
Yes
Yes
Yes
Yes
Temporal
sum.
Yes
No
Yes
No
Spatial
sum.
Yes
No
?
no
4 mechano-receptors for glabrous (non-hairy) skin
properties. (left out hairy receptors, tactile disks)
3. Wearable Haptics
Receptive Fields
3. Wearable Haptics
Rabbit/Saltation Effect
3. Wearable Haptics
Tactile Glove
• Vibrotactile pager
motors (tactors)
• Receptive field
uniqueness
• Non-fingertip regions
• Code intensity
• Minimize lateral,
maximize vertical
conductivity
3. Wearable Haptics
Tactor Stimulus Variables
?
?
?
?
?
Intensity
Pattern
Location
Duration
Inter-stimulus
interval
? Inter-activity
? waveform
3. Wearable Haptics
Pager Motors
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
Mild steel casing
Coil embedded
nylon rotor
Permanent
magnetic ring
Front view
Back view
Soldering pad
(a) Outline
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
(b) Interior
Winding (coil)
Commutator
3. Wearable Haptics
Telepresence
from Stacked Pager Motors?
3. Wearable Haptics
• Electrical to
mechanical
transducer?
• Mechanical to
perceptual?
• Maximize
communication
bandwidth
FPGA, midi
3. Wearable Haptics
Bandwidth
Intensity &
Frequency
inherently
connected
max. intensity
levels is 3
perhaps,
stacked?
3. Wearable Haptics
investigating other technology..
SMA actuators
Electroactiv
e polymer
motors power dense
3. Wearable Haptics
Tactile & Inertial Patterns in
Long Cane
3. Wearable Haptics
Robotic Surgery
QuickTime™ and a
Sorenson Video 3 decompressor
are needed to see this picture.
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
3. Wearable Haptics
Understanding Conventional Surgical Haptics
(Xin)
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
3. Wearable Haptics
Surgical SLAM
QuickTime™ and a
H.264 decompressor
are needed to see this picture.
3. Wearable Haptics
Telepresence - Military driving force (TATRC)
QuickTime™ and a
Sorenson Video 3 decompressor
are needed to see this picture.
4. Design (of assistive devices)
process
team work
communication
user relevancy
innovation (methods)
prediction: mathematical
model
iterative (many
prototypes)
analysis & synthesize
again
entrepreneur (business,
economics)
Course, 3rd year,
focus on product
design, theme is
disabled elderly =
design assistive
devices
4. Design
(2005)
QuickTime™ and a
MPEG-4 Video decompressor
are needed to see this picture.
4. Design
(2006)
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
4. Design
(2006 - another group)
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
5. In closing
• real world testing (theory, research,
engineering)
• surgical haptics require understanding
need
• computer vision - hard to make sense of
our world
• wearable haptics, various embodiments communications/telepresence
• design (understanding user problems and
needs) can identify research problems
• other applications include rehabilitation
- providing the right information for
people to do the right thing
5. really in closing
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
All of this
research
could also be
called
robotics
research but
I prefer not
to even
though we
dabble (play)
with robots
to amuse
students