5.2.2. Interaction Techniques
• Interaction task
• Interaction device
• Interaction technique
– A way of using a device to perform a generic HCI task
– Widget in a toolkit
• Two information processors communicating via
constrained interface [Tufte]
– Goal: Increase bandwidth across channel
The Teletype
• 10
characters
per second
in both
directions
• Photo
shows
Thompson
and Ritchie,
creators of
Unix
Chord Keyboard
• Douglas Engelbart,
“Mother of all Demos”
– http://sloan.stanford.e
du/MouseSite/1968De
mo.html
Image courtesy of Louisa Billeter on Flickr.
Image courtesy of Amarand Agasi on Flickr.
Image courtesy of Frau Bob on Flickr.
Image courtesy of Mike Marttila on Flickr.
Light Pen
Image by MIT OpenCourseWare.
Image by MIT OpenCourseWare.
Image by MIT OpenCourseWare.
Not Multi-touch
Fast accurate finger or stylus
Image by MIT OpenCourseWare.
Bare finger or
conducting
stylus
Not multi-touch
Image by MIT OpenCourseWare.
Image by MIT OpenCourseWare.
Image by MIT OpenCourseWare.
Image by MIT OpenCourseWare.
Other Manual Input
• Bier, Taxonomy of See-Through Tools
– https://open-video.org/details.php?videoid=4563
• Han, Perceptive Pixel
– https://www.youtube.com/watch?v=ysEVYwa-vHM
• Harrison, Skinput
– http://research.microsoft.com/enus/um/redmond/groups/cue/skinput/
Google Glass Predecessors, Steve Mann
Skype/Facetime Predecessor 1964 (!)
Interaction Tasks
• Select
• Position
• Orient
• Quantify
• Text
Selection Factors
• Continuity
• Parallelism
• Experimental results
• Costs
• Reliability
Design issues
• Control/display ratio
• Control/display compatibility
• Direction relations
• Fitts' law
• Position vs. velocity control
• Relative vs. absolute
• Direct vs. indirect
© ACM, Inc. All rights
reserved.
© ACM, Inc. All rights
reserved.
Eye Movement-Based Interaction
• Highly-interactive, Non-WIMP, Non-command,
Lightweight
– Continuous, but recognition algorithm quantizes
– Parallel, but implemented on coroutine UIMS
– Non-command, lightweight, not issue intentional commands
• Benefits
– Extremely rapid
– Natural, little conscious effort
– Implicitly indicate focus of attention
– “What You Look At is What You Get”
Issues
• Midas touch
– Eyes continually dart from point to point, not like relatively
slow and deliberate operation of manual input devices
– People not accustomed to operating devices simply by
moving their eyes; if poorly done, could be very annoying
• Need to extract useful dialogue information from
noisy eye data
• Need to design and study new interaction
techniques
Approach to Using Eye Movements
• Philosophy
– Use natural eye movements as additional user input
– vs. trained movements as explicit commands
• Technical approach
– Process noisy, jittery eye tracker data stream to filter,
recognize fixations, and turn into discrete dialogue tokens
that represent user's higher-level intentions
– Then, develop generic interaction techniques based on the
tokens
Other Work
• A taxonomy of approaches to eye movement-based
Natural
interaction
Unnatural
response
(real-world)
response
Unnatural
(learned)
eye movement
Most work,
esp. disabled
N/A
Natural
eye movement
Jacob
Starker & Bolt,
Vertegaal
Methods for Measuring Eye Movements
• Electronic
– Skin electrodes around eye
• Mechanical
– Non-slipping contact lens
• Optical/Video - Single Point
– Track some visible feature on eyeball; head stationary
• Optical/Video - Two Point
– Can distinguish between head and eye movements
Optical/Video Method
•
Views of pupil, with corneal reflection
•
Hardware components
Use CR-plus-pupil Method
•
Track corneal reflection and outline
of pupil, compute visual line of gaze
from relationship of two tracked
points
•
Infrared illumination
•
Image from pupil camera
The Eye
• Retina not uniform
– Sharp vision in fovea, approx. 1 degree
– Blurred vision elsewhere
• Must move eye to see object sharply
• Eye position thus indicates focus of attention
Types of Eye Movements Expected
• Saccade
– Rapid, ballistic, vision suppressed
– Interspersed with fixations
• Fixation
– Steady, but some jitter
• Other movements
• Eyes always moving; stabilized image disappears
Eye Tracker in Use
• Integrated with head-mounted display
Fixation Recognition
•
Need to filter jitter, small saccades,
eye tracker artifacts
•
Moving average slows response
speed; use a priori definition of
fixation, then search incoming data
for it
•
Plot = one coordinate of eye
position vs. time (3 secs.)
•
Horizontal lines with o's represent
fixations recognized by algorithm,
when and where they would be
reported
User Interface Management System
•
Turn output of recognition algorithm
into stream of tokens
–
EYEFIXSTART, EYEFIXCONT,
EYEFIXEND, EYETRACK,
EYELOST, EYEGOT
•
Multiplex eye tokens into same
stream as mouse, keyboard and
send to coroutine-based UIMS
•
Specify desired interface to UIMS
as collection of concurrently
executing objects; each has own
syntax, which can accept eye,
mouse, keyboard tokens
Interaction Techniques
•
Eye tracker inappropriate as a
straightforward substitute for a
mouse
•
Devise interaction techniques that
are fast and use eye input in a
natural and unobtrusive way
•
Where possible, use natural eye
movements as an implicit input
•
Address “Midas Touch” problem
Eye as a Computer Input Device
• Faster than manual devices
• No training or coordination
• Implicitly indicates focus of attention, not just a pointing device
• Less conscious/precise control
• Eye moves constantly, even when user thinks he/she is
staring at a single object
• Eye motion is necessary for perception of stationary objects
• Eye tracker is always "on"
• No analogue of mouse buttons
• Less accurate/reliable than mouse
Object Selection
•
Select object from among several
on screen
•
After user is looking at the desired
object, press button to indicate
choice
•
Alternative = dwell time: if look at
object for sufficiently long time, it is
selected without further commands
–
•
Poor alternative = blink.
Dwell time method is convenient,
but could mitigate some of speed
advantage
Object Selection (continued)
•
Found: Prefer dwell time method
with very short time for operations
where wrong choice immediately
followed by correct choice is
tolerable
•
Long dwell time not useful in any
cases, because unnatural
•
Built on top of all preprocessing
stages-calibration, filtering, fixation
recognition
•
Found: 150-250 ms. dwell time feels
instantaneous, but provides enough
time to accumulate data for
accurate fixation recognition
Continuous Attribute Display
• Continuous display of attributes of selected object,
instead of user requesting them explicitly
• Whenever user looks at attribute window, will see
attributes for the last object looked at in main window
• If user does not look at attribute window, need not be
aware that eye movements in the main window
constitute commands
• Double-buffered refresh of attribute window, hardly
visible unless user were looking at that window
– But of course user isn't
Moving an Object
•
Two methods, both use eye position
to select which object to be moved
–
Hold button down, “drag” object by
moving eyes, release button to stop
dragging
–
Eyes select object, but moving is
done by holding button, dragging
with mouse, then releasing button
•
Found: Surprisingly, first works
better
•
Use filtered “fixation” tokens, not
raw eye position, for dragging
Menu Commands
•
Eye pull-down type menu
•
Use dwell time to pop menu, then to
highlight choices
–
If look still longer at a choice, it is
executed; else if look away, menu is
removed
–
Alternative: button to execute
highlighted menu choice without
waiting for second, longer dwell time
•
Found: Better with button than long
dwell time
•
Longer than people normally fixate
on one spot, hence requires
“unnatural” eye movement
Eye-Controlled Scrolling Text
•
Indicator appears above or below
text, indicating that there is
additional material not shown
•
If user looks at indicator, text itself
starts to scroll
•
But never scrolls while user is
looking at text
•
User can read down to end of
window, then look slightly lower, at
arrow, in order to retrieve next lines
•
Arrow visible above and/or below
text display indicates additional
scrollable material
Listener Window
• Window systems use explicit mouse command to
designate active window (the one that receives
keyboard inputs)
• Instead, use eye position: The active window is the
one the user is looking at
• Add delays, so can look briefly at another window
without changing active window designation
• Implemented on regular Sun window system (not
ship display testbed)
Object Selection Experiment
•
Compare dwell time object selection
interaction technique to
conventional selection by mouse
pick
•
Use simple abstract display of array
of circle targets, instead of ships
•
Subject must find and select one
target with eye (dwell time method)
or mouse
•
Circle task: Highlighted item
•
Letter task: Spoken name
Results
• Eye gaze selection significantly and substantially
faster than mouse selection in both tasks
• Fitts slope almost flat (1.7 eye vs 117.7 mouse)
Task (time in msec.)
Device
Circle
Letter
Eye gaze
503.7 (50.56)
1103.0 (115.93)
Mouse
931.9 (97.64)
1441.0 (114.57)
Time-integrated Selection
•
Alternative to “stare harder”
–
Subsumed into same implementation
•
Retrieve data on 2 or 3 most looked-at objects over
last few minutes
•
Integrate over time which objects user has looked at
•
Select objects by weighted, integrated time function
(vs. instantaneous look)
•
Matches lighweight nature of eye input
X-ray Vision
•
1. Entire virtual room
•
2. Portion of virtual room
–
•
No object currently selected
3. Stare at purple object near top
–
Internal details become visible
Experimental Results
• Task: Find object with given letter hidden inside
• Result: Eye faster than Polhemus, More so for
distant objects
• Extra task: Spatial memory better with Polhemus
time (sec)
150
close
100
distant
50
overall
0
Eye
Polh
Laptop prototype
• Lenovo + Tobii
– http://www.youtube.c
om/watch?v=6bYOB
XIb3cQ
– http://www.youtube.c
om/watch?v=GFwhx
0Wy8PI