Human Interface
where the bits meet the flesh
bits meets eyes and ears
Think beyond the box
Tom Zimmerman
IBM Almaden Research Center
San Jose, California
tzim@almaden.ibm.com
October 2007
Research Theme
Transducers; Converting one thing into another
User Interface, User Experience, First Person
Hooking people to gadgets
– Bits Meet the Flesh
– Bits Meet Eyes & Ears
Transducer Examples
– Dance into Music (dance synthesizer, suction cup)
– Hand into Finger Spelling (DataGlove)
– Sound into Light (Singing Coach)
Fantasy & Necessity
Are the Mothers of Invention
VR Data Glove
Personal Area Network
Violin
Home Health Monitoring
Honda Smart Seat
Contact Microscope
Discrete Events
Converting continuous input into
discrete events.
Continuous input examples
brightness, sound level, pressure
Discrete output; 0/1, off/on, mouse click
Comparator- Ideal Signal
Sensor
+
_
Digital Output
Threshold
Threshold
Sensor
Continuous Input
Quantized Digitized
Output
Comparator- Real World Signal
Noise in input causes output jitter
threshold
cross high for on
cross low for off
Triggering Discrete Events
with Hysteresis
Hysteresis, “play”
High threshold
Continuous Input
(e.g. brightness,
sound level, pressure)
Low threshold
Quantized/Digitized
Output
cross high for on
cross low for off
TV Synth Interface
Hacking Remotes, Mice, Kbd
Sensor
T
V
Signal Processing
Switch
Mouse
Keyboard
IR Remote
Synthesizer
Circuit Board
Photo
Transistor
60 Hz
Filter
Comparator
w/ hysteresis
4066
Quad
CMOS
Switch
• Left button
• Right button
• X direction
• X increment
• Y direction
• Y increment
1 to 100 Hz
Mouse Sensors
• Microphone; Speaker Radiation Pattern
• Light; Water Ripple Tank
• Ultrasonics; Wavelength, Horn Radiation Pattern
• Temperature; Weather Station
• Wind speed; See Jet Stream, Car Aerodynamics
• Pressure
• Tilt
• Rpm
• Vibration
• Pendulum; Simple Harmonic Motion
• Acceleration; Seismograph, Earthquakes, Cars/Trucks
• Voltage
• Current
• Resistance
• Capacitance
• Magnetic field
• Electric field
Wireless Doorbell
Wireless Motion Detector
Working with Continuous Output
PowerGlove Protoype (VPL 1983)
DataGlove
(Fiber Optic)
Young Harvill
sensor
source
Fiber Optic
Flex Sensors
Polhemus Magnetic
Position & Orientation
Head Mounted Display
(NASA 1986)
PowerGlove (Mattel 1987)
Computer Data Entry And
Manipulation Apparatus And Method
1.3 Million Sold
$89 Retail
The Spirit Chair (MIT 1994)
Mediums use spirit cabinets to guide fields
to contact spirits who make sounds ...
Near-Field Communication
(MIT/IBM 1995)
Data In
Data Out
Contagious Information
(MIT 1995)
Fred
Jan
7/11
Bill
Car Seat Sensor (NEC 1998)
Musical Instruments
Human
Actions
Musical
Pitch
Amplitude
Timbre
Duration
Instrument
Vocalizer (Breakaway 1981)
Singing Coach (Exploratorium 1990)
RePerformer (Zim/Wantman 1992)
Pen Input
Pen That Remembers Everything You Write
(IBM 2000-2002)
Premiere UI
unobtrusive, low cognitive load, small, portable, quick, cheap
Modeless
Universal
text, graphics, doodle
comment on document
direct manipulation
any language
popular input method
classic form factor
BUT
hard to index, search,
archive, distribute
Other Pen Systems
PDA; writing on plastic
LCD; writing on glass
Tablet; invisible writing
CrossPad; writing on 2 lbs.
Toy Pens
Pen Research (IBM 2002)
IRDA
Ink
CPU and
Flash Memory
Battery (AAA)
CMOS Imager
Lens
Normal or dot code paper
Ultrasonic
Optical
Electric Field Pen (IBM 1997)
Dynamic Signature Verification
“Sign and Go” (2003)
•
•
•
•
Biometric verification using handwriting cadence (timing)
A means to electronically verify customer signatures
Customer signs on a digital signature capture station
Software checks signature against reference signatures and
checks for major changes in signature
Biometrics
Classification of Biometrics
Physical Feature
–Iris, face, finger, hand, DNA
–Can alter presentation
Behavior Trait
–Signature, voice, gait, keystroke
–Can change behavior
Identification & Verification
Identification
– “Who am I?”
– 1:N matching (difficult, big search)
– Example: 100k customers @ 0.1% error = 100
errors
– Use: Discover identity, Watch lists
Verification
– “Am I who I claim to be?”
– 1:1 matching (easier)
– Requires identifier (not secure but unique)
token (loyalty card)
memory (phone number)
– Use: Verify identity
Performance Metrics
False Accepts (FA)
– likelihood that someone may be falsely accepted bad guy gets in
False Rejects (FR)
– likelihood that a genuine user may be rejected good guy gets rejected
Lower the FA and FR, the better the system performance
Acceptance threshold trade-off FA to FR
Equal Error Rate when FA=FR
Failure to enroll, Failure to acquire
– Light eyes (iris), thin skin (finger), simple/changing signature,
noisy environment (voice)
– Need backup/alternative
Example FA=2% FR=1%
– 2 out of 100 good customers get rejected
– 1 out of 100 bad guy attempts get in
– If 1 in 1000 customers are bad guys, 1 forgery per 100,000
transactions
Biometric Comparison
Biometri
c
EER*
Advantage
Disadvantage
Iris
< 0.1%
Most accurate, no contact
Critical alignment,
expensive reader
Finger
0.1-10%
Wide use, standard
databases, competitive
market, legal precedence
Special reader, criminal
association, gummy
attack, modest cost reader
Signature
1.7-3%
Uses existing POS signature
capture hardware, social
acceptance, signature dual
purpose (agreement &
biometric)
Behavior (requires
cooperation), name
changes, ergonomics
Voice
2-10%
Use with cell phones
Sensitive to noisy
environment, requires
* Equal Error Rate estimates for relative comparison only. Actual
results are highly
talking
dependent on system design, equipment, user population and experience.
Tangible Programming
Logo Cards
Bar code
LEFT
RIGHT
FORWARD
LEFT angle
RIGHT angle
FORWARD distance
Rotates the turtle
counterclockwise by a
specific angle
Turns the turtle clockwise
by an angle
Moves turtle forward by a
specified distance
LEFT 90 turns the turtle
one quarter to its left
RIGHT 120 turns the turtle
one third to its right
FORWARD 100 moves
turtle 100 turtle steps in the
direction it is facing
BACK
HOME
FORWARD
BACK distance
HOME
FORWARD distance
Moves turtle backwards by
a specified direction {turtle
still faces forward}
Turtle returns to the center
of the screen
Moves turtle forward by a
specified distance
BACK 60 causes turtle to
move 60 steps backwards
Instruction
Usage
Definition
Example code
FORWARD 100 moves
turtle 100 turtle steps in the
direction it is facing
35mm slide holder
Students Programming With LOGO Playing Cards
Queuing and Scanning Programs into Computer
Programming Robots with
Playing Cards
The Robosapian™ robot
comes with a remote
control to program robot
movements.
The remote control is
replaced with a deck of
instruction cards and bar
code reader.
Students choreograph
movements by arranging
the cards, then program
the robot by scanning the
cards with the bar code
reader.
Telepresence
When you need to be there
… but you don’t want to travel there
Dummy Head Electronics (IBM 2003)
Digital Head Components
Frame and Handle
VGA Camera
MegaPixel Camera
Tilt Gimble
Yaw Motor
Speaker Phone
Battery
Ethernet
Tiny PC w/WLAN
Ethernet
AC Power
Analog Phone Line
DIGITAL HEAD
(x-ray view)
BASE STATION
(100 m range to head)
Techno Head
Binaural Mics
QVGA Screen
Digital Camera
20 W Speaker
Seat Clamp
Head Appearances
Standards Committee
Breakfast Keynote
Symposium
Board Meeting Talk
Trade Show Booth
Conference Panel
Shopping Buddy
Infrared Beacon
Infrared Receiver
Wireless (Bluetooth)
Barcode Scanner
WLAN
Location-aware shopping cart provide real-time “you are
here” navigation map and location-based content-driven
offers such as coupons and recipes.
Museum Interfaces
Sizzling Jello
Bubble Machine
Bubble Strobe Machine
Questions???
Badge Computer (IBM 1998)
NZ Airport Travel Card (IBM 1998)
PDA running XML Browser, linked by 802.11b WLAN
Multi-Function PDA (IBM 2001)
Camera with 270 deg swivel
Microphones/Structured Light
802.11 WLAN antennas running
along each side
Color Screen (no touch)
Keypad overlay
Touch Pad
FM antenna along perimeter
Speakers
BioPDA Wireless Biometric Terminal
InfraRed Lights
Iris Camera
Fingerprint Reader
Face & Document Camera
Field Replaceable
Biometric Module
Symbol 8000
Handheld Terminal
802.11b WLAN
Intel XScale Processor
Numeric Keypad
Using the Biometric
Wireless Terminal
Capturing
Face and Iris
Lining up
Iris Camera
lines up and frames
Face Camera
Capturing
Fingerprint