Dialog design
How do we communicate with computers?
Agenda
 Syllabus modification
 Part 1 due next week!
 Interaction modalities and styles
Dialog Design
 How does a user interact with the interface?
Interaction Styles
 1. Command languages
 2. WIMP - Window, Icon, Menu, Pointer
 3. Direct manipulation
 4. Speech/Natural language
 5. Pen & Mobile
General Issues in Choosing
Dialogue Style








Who is in control - user or computer
Initial training required
Learning time to become proficient
Speed of use
Generality/flexibility/power
Special skills - typing
Screen space required
Computational resources required
Command Languages
 Earliest UI interaction paradigm
 Examples: MS-DOS shell, UNIX, Linux
 Little or nothing is visible so…
 Work primarily by recall, not recognition
 Heavy memory load
 Poor choice for novices but...
CL Advantages
 Advantages for experts
 Speed, conciseness
 % ls (hard to beat)
 Can express actions beyond a limited set
 Flags, piping one command to another
 Repetition, extensibility
 Scripting, macros
 Easier implementation, less overhead
 Power
 Abstraction, wild cards
CL Dangers
 With added power, comes added
responsibility and danger

UNIX


% rm -r *
Deletes every file that you have, and you can’t
get them back
CL Design Goals
 Consistency
 Have options and arguments expressed
the same way everywhere
 Good naming and abbreviations
 UNIX fails here because commands
were developed by lots of different
people at different organizations

No guidelines provided
General Issues - CL








Who is in control - user or computer
Initial training required
Learning time to become proficient
Speed of use
Generality/flexibility/power
Special skills - typing
Screen space required
Computational resources required
Direct Manipulation
Definition:
 1) Continuous visibility of the objects and
actions of interest
 2) Rapid, reversible, incremental actions
whose effect is immediately noticeable
 3) Replacement of command language
syntax by direct manipulation of object of
interest (physical actions, buttons, etc.)
Shneiderman ‘82
DM Essence
 Representation of reality that can be
manipulated
 The user is able to apply intellect directly to
the task
 The tool itself seems to disappear
 Related to


Directness
Engagement
Direct Manipulation
 See pgs 171-174 in DFAB
 Examples





WYSIWYG editors and word processors
VISICALC - 1st electronic spreadsheet
CAD
Desktop metaphor
Video games
Example: Homefinder
DM Issues
Advantages
 Flexible, easily reversible
actions helps reduce
anxiety in users
 Easy to learn and
remember
 WYSIWYG provides
context & instant visual
feedback
 Exploits human use of
visual spatial cues
 Limits types of errors that
can be made
Disadvantages
 visual representation not selfexplanatory, no prompts
 Mouse ops may be slower than
typing
 Screen space intensive
 Not good at:
 Repetition
 History keeping
 Certain tasks (Change all
italics to bold)
 Abstract elements (variables)
General Issues - DM








Who is in control - user or computer
Initial training required
Learning time to become proficient
Speed of use
Generality/flexibility/power
Special skills - typing
Screen space required
Computational resources required
WIMP
 Windows, Icons, Menus, Pointers
 Focus: Menus, Buttons, Forms
 Predominant interface paradigm now (with
some direct manipulation added)
 Advantages:
 ?
Menus
 Advantages:
 1 keystroke or mouse operation vs. many
 No memorization of commands
 Limited input set
 Disadvantages:
 Less direct user control - have to find correct menu /
menu item
 Not so readily extensible
 Slower than keyboarding for experienced users, at
least without accelerators
Menu Items
 Various types:



Flat
Cascading
Pop-up (contextual)
 Organization strategies




Create groups of logically similar items
Cover all possibilities
Ensure that items are non-overlapping
Keep wording concise, understandable
A Good Example
 Logical grouping
 Visual separation of
groups
 Disabled items “grayed
out”
 Shortcuts shown
 … indicates leads to
dialogue
Presentation Sequence
 Forms, dialogue boxes, menus
 Use natural if available
 Time
 e.g. Breakfast, Lunch, Dinner
 Numeric ordering
 e.g. Point sizes for font
 Other possibilities:
 Alphabetical
 Group related items
 Frequently used first
 Most important first
Pointers
 Allows users to specify location parameter of a
command


Provides for some of that direct manipulation
Put this file HERE
 Also displays system state info:
 Tracking
 Busy
 Hints
 Modes (text cursor vs. selection pointer)
Other WIMP issues
 Windows management



How to locate, move, find
Transfer information between
Switch attention between
 Icons

Need graphic design attention
 Toolbars
Newer icons
Simple icons
Simple icons
General Issues - WIMP








Who is in control - user or computer
Initial training required
Learning time to become proficient
Speed of use
Generality/flexibility/power
Special skills - typing
Screen space required
Computational resources required
Natural input
 Universal design
 Take advantage of familiarity, existing
knowledge
 Alternative input & output
 Multi-modal interfaces
 Getting “off the desktop”
Speech dialogue
 Why use it?






Hands busy
Mobility required
Eyes occupied
Conditions preclude use of keyboard
Visual impairment
Physical limitation
Speech Input
 Speaker recognition


Tell which person it is (voice print)
Monitoring, recording
 Speech recognition


Identify words
IBM ViaVoice, Dragon Dictate, ...
 Natural language understanding

(does not necessarily involve audio)
Recognition Dimensions
 Speaker dependent/independent
 Parametric patterns are sensitive to speaker
 With training (dependent) can get better
 Vocabulary
Did you
 Some have 50,000+ words
vs.
 Isolated word vs. continuous speech
Didja
 Continuous: where words stop & begin
 Typically a pattern match, no context used
Recognition Example:
My flight experience uh,
I started off as a private
pilot and then I spent 20
years flying as a navigator
on C130s in the air force
and I currently fly as a
first officer on the SAAB
340 turboprop.
my flight experience a and
I started off as a private
pilot and I spent 20
years flying is a navigator
on C-130s in the Air Force
and I currently fly as a
first officer on the sound
340 turboprop.
And another example
Here here's one of the
thoughts I'd throw in on
that. I'd agree with
Howard that the
combination of checklists
and flows is the way way
to manage things best.
hair and here's one of the
outside troodon and
an eye to agree with
power that the
domination checklists
and flows is the way
that managed things best
Errors
 Systems make four types of errors:




Substitution - one for another
Rejection - detected, but not recognized
Insertion - added
Deletion - not detected
 So how do you recover from an error?
Natural Language Understanding
 Putting meaning to the words
 Input might be spoken or typed
 Holy grail of Artificial Intelligence
problems
Eliza - a “Faked” Understanding

Person : Men are all alike.
ELIZA : In what way?
Person : They're always bugging us about something or other.
ELIZA : Can you think of a specific example?
Person : Well, my boyfriend made me come here.
ELIZA : Your boyfriend made you come here?
Person : He says I'm depressed much of the time.
ELIZA : I am sorry to hear that you are depressed.
Person : It's true. I am unhappy.
ELIZA : Do you think coming here will help you not to be unhappy?
Person : I need some help, that much seems certain.
ELIZA : What would it mean to you if you got some help?
Person : Perhaps I could learn to get along with my mother.
ELIZA : Tell me more about your family
 http://www-ai.ijs.si/eliza/eliza.html

Weizenbaum, J., "ELIZA -- A computer program for the study of natural
language communication between man and machine", Communications of the
ACM 9(1):36-45, 1966
NL Factors/Terms
 Syntactic
 Grammar or structure
 Prosodic
 Inflection, stress, pitch, timing
 Pragmatic
 Situated context of utterance, location, time
 Semantic
 Meaning of words
SR/NLU Issues
Advantages
 Easy to learn and
remember
 Powerful
 Fast, efficient (not
always)
 Little screen real
estate
Disadvantages
 Assumes domain knowledge
 Doesn’t work well enough yet


Requires confirmation
And recognition will
always be error-prone
 Expensive to implement
 Unrealistic expectations can
generate mistrust
Speech Output
 Tradeoffs in speed, naturalness and
understandability
 Male or female voice?


Technical issues (freq. response of phone)
User preference (depends on the application)
 Rate of speech
 Technically up to 550 wpm!
 Depends on listener
 Synthesized or Pre-recorded?
 Synthesized: Better coverage, flexibility
 Recorded: Better quality, acceptance
Speech Output
 Synthesis
 Quality depends on software ($$)
 Influence of vocabulary and phrase choices
 http://www.research.att.com/~ttsweb/tts/demo.php#top
 Recorded segments
 Store tones, then put them together
 The transitions are difficult (e.g., numbers)
Designing Speech Interaction
 Constrain vocabulary
 Limit valid commands
 Structure questions wisely (Yes/No)
 Manage the interaction
 Examples?
 Slow speech rate, but concise phrases
 Design for failsafe error recovery
 Visual record of input/output
 Design for the user – Wizard of Oz
Speech Tools/Toolkits
 Java Speech SDK

FreeTTS 1.1.1 http://freetts.sourceforge.net/docs/index.php
 IBM JavaBeans for speech
 Microsoft speech SDK (Visual Basic, etc.)
 OS capabilities (speech recognition and synthesis
built in to OS) (TextEdit)
 VoiceXML
General Issues – Speech/NL








Who is in control - user or computer
Initial training required
Learning time to become proficient
Speed of use
Generality/flexibility/power
Special skills - typing
Screen space required
Computational resources required
Non-speech audio
 Good for indicating changes, since we ignore
continuous sounds

Traditionally used for warnings, alarms or status
information
 Provides secondary representation
 Supports visual interface
 Provides information that helps reduce error
 Tradeoff in using natural (real) sounds vs.
synthesized noises.
Non-speech audio examples







Error ding
Info beep
Email arriving ding
Recycle
Battery critical
Logoff
Logon
Others?
Pen, Touch, & Mobile interaction
Pen, Touch, & Mobile dialog
 Stylus or finger
 Tradeoffs of each?
 Pen as a standard mouse (doubleclick?)
 Variety of platforms
 Desktop touch screens or input pads (Wacom)
 Tablet PCs
 Handheld and Mobile devices
 Electronic whiteboards
 Platforms often involve variety of size and other
constraints
Mobile devices
 More common as more platforms available
 PDA
 Cell phone
 Ultra mobile tablets
 GPS
 Smaller display (160x160), (320x240)
 Few buttons, different interactions
 Free-form ink
 Soft keyboard
 Numeric keyboard => text
 Stroke recognition
 Hand printing / writing recognition
http://www.blackberry.com/
http://www.oqo.com/
Soft Keyboard
 Presents a small diagram of
keyboard
 You click on buttons/keys with pen
 QWERTY vs. alphabetical


Tradeoffs?
Alternatives?
Numeric Keypad
 You press out letters of your word, it matches the most
likely word, then gives optional choices
 Faster than multiple presses per key
 Used in mobile phones
 http://www.t9.com/
Other pen text input
 Graffiti – Palm mobile devices

Unistroke recognition
 Experimental

Cirrin

World level unistroke
Quickwriting
Harder to learn than
graffiti
Hand Printing / Writing
Recognition
 Recognizing letters and numbers and special
symbols
 Lots of systems (commercial too)
 English, kanji, etc.
 Not perfect, but people aren’t either!


People - 96% handprinted single characters
Computer - >97% is really good
Recognition Issues
 Boxed vs. Free-Form input
 Sometimes encounter boxes on forms
 Printed vs. Cursive
 Cursive is much more difficult
 Letters vs. Words
 Cursive is easier to do in words vs individual letters, as
words create more context
 Usually requires existence of a dictionary
 Real-time vs. off-line
Pen Gesture Commands
-Might mean delete
-Insert
-Paragraph
Define a series of (hopefully) simple drawing gestures
that mean different commands in a system
Pen Use Modes
 Often, want a mix of free-form drawing and
special commands
 How does user switch modes?



Mode icon on screen
Button on pen
Button on device
Error Correction
 Having to correct errors can slow input
tremendously
 Strategies
 Erase and try again (repetition)
 When uncertain, system shows list of best
guesses (n-best list)
 Others??
Free-form Ink
 Ink is the data, take as is
 Human is responsible for
understanding and
interpretation
 Often time-stamped
 Applications




Signature verification
Notetaking
Electronic whiteboards
Sketching
Electronic whiteboards
 Smartboard and Mimio
 Can integrate with projection
 Large surface to interact with

Issues?
http://www.mimio.com/
http://www.smarttech.com/
Touch tables
 Which techniques
might be similar to
smaller
touchscreens?
 Which would differ?
 How similar and
different from
interactive white
boards?
Microsoft Surface
Real paper
 Anoto digital paper and pen technology
(http://www.anoto.com/)
 Other pens available:
http://www.logitech.com/
http://www.epos-ps.com/
 Issues?
General Issues – Pen input








Who is in control - user or computer
Initial training required
Learning time to become proficient
Speed of use
Generality/flexibility/power
Special skills - typing
Screen space required
Computational resources required
Other interesting interactions
 Gesture input
 Wii
 Lots of other specialized hardware for tracking
 3D interaction
 Stereoscopic displays
 Virtual reality
 Immersive displays such as glasses, caves
 Augmented reality
 Head trackers and vision based tracking
 Tangible interaction
 Use physical objects to express input