HCI
The Computer
Interacting with computers
To understand Human-Computer Interaction
… need to understand computers!
what goes in and out of computer
devices, paper, sensors, etc
what can the computer do?
memory, processing, etc
Von Neumann Architecture
Timeline
Vacuum Tube
Triode (1908)
Transistor
1947
(silicon,
germanium)
Intel Itanium (Tukwila)
2008: 2 billion transistors
Very Large Scale
Integrated (VLSI)
Circuits; 1970s-(> 1,000 transistors
per chip)
The Computer
A computer system is made up of various elements
Each of these elements affects the interaction
input devices – text entry and pointing
 output devices – screen (small&large), digital paper
 virtual reality – special interaction and display devices
 physical interaction – e.g. sound, haptic, bio-sensing
 paper – as output (print) and input (scan)
 memory and processing– memory, access, speed of processing,
networks

How many computers …
in your house?
PC
 DVD, HiFi, TV
 microwave, cooker,
washing machine, fridge
 Air Condition
 security system
can you think of more?

in your pockets?
PDA
 smartphone
 digital camera
 smart card
 electronic car key
 USB memory
try your pockets and
bags

Richer interaction
sensors
and devices
everywhere
TEXT ENTRY DEVICES
keyboards (QWERTY et al.)
chord keyboards, phone pads
handwriting, speech
Input (1)

Batch
 Large
amounts
 Pre-prepared
information
 Standardised
data loaded
quickly using automated routines
 Used
a lot in the past (punch
cards)
Keyboards
Most common text input device
 Allows rapid entry of text by experienced
users
 Keypress closes connection, causing a
character code to be sent
 Usually connected by cable, but can be
wireless

Keyboards

QWERTY Keyboard (1878)

the official standard of computer
keyboards [ISO9995]

was designed for two finger typing

prevents stuck keys (on a typewriter)

the position of the keys the hands are
held close together helping to cause
Carpal Tunnel Syndrome

can be used without training

is cheap and easy to find

Any other keyboard?
Keyboards (ctd) - layout
Alphabetic layout
keys arranged in alphabetic order
 not faster for trained typists
 not faster for beginners either!

Dvorak layout
common letters under dominant fingers
 biased towards right hand
 common combinations of letters alternate between hands
 10-15% improvement in speed and reduction in fatigue
 But no significant adoption, when compared to QWERTY

Keyboards

Over the years, attempts were made to physically
improve keyboards …
 Tilting
 Curving
 Splitting
Keyboards (ctd) - special keyboards
Maltron left-handed keyboard
Virtual keyboard
Projection keyboard
Chord keyboards
only a few keys - four or 5
 letters typed as combination of keypresses
 compact size – ideal for portable applications
 short learning time – keypresses reflect letter shape
 fast – once you have trained
BUT - social resistance, plus fatigue after extended use
NEW – niche market for some wearables

Keyboards

Chord Keyboards
Don’t use an alphabet
 Use 4/5 keys
 Rely on combination of keys
 Ideal for one hand or mobile devices
 Can suffer fatigue after use
 New application in mobile devices?

Phone keypad and T9 entry

use numeric keys with
multiple presses
2–abc
3-def
4-ghi
5-jkl
6-mno
7-pqrs
8-tuv
9-wxyz
hello = 4433555[pause]555666
surprisingly fast!

T9 predictive entry




type as if single key for each letter
use dictionary to guess the right word
hello = 43556 … ‘hello’ not ‘gdjjm’
note 26 could mean ‘am’ or ‘an’; phone gives those options
Handwriting recognition

Text can be input into the computer, using a pen and a
digesting tablet


natural interaction
Technical problems:
capturing all useful information (path, pressure, etc) in a natural
manner
 segmenting joined up writing into individual letters
 coping with different styles of handwriting
 etc


Used in PDAs, tablet computers, smart white boards …
… leave the keyboard on the desk!
Speech recognition

Most successful when:
 single
user – initial training and learns peculiarities
 limited vocabulary systems

Improving, but challenges such as:
 external
noise interfering
 imprecision of pronunciation/articulation e.g.
vagueness, pauses, clearing throat, repetition, etc
 large vocabularies
 different speakers
POSITIONING, POINTING AND
DRAWING
mouse, touchpad
trackballs, joysticks etc.
touch screens, tablets
eyegaze, cursors
Mouse

Movement of mouse moves screen cursor



screen cursor oriented in (x, y) plane
mouse movement in (x, z) plane …
Since the mouse sits on a desk
moving it about is easy
 users suffer little arm fatigue
 cursor is small and can thus be manipulated without obscuring
the display


Disadvantage

hand-eye coordination problems for novice users
Mouse (ctd)
Two methods for detecting motion
 Mechanical
ball on underside of mouse turns/rotates as mouse is moved
 can be used on almost any flat surface


Optical
light emitting diode on underside of mouse
 may use special grid-like pad or just on an appropriate surface
 detects fluctuating alterations in reflected light intensity to
calculate relative motion
 less susceptible to dust and dirt

Mouse (ctd)
Footmouse

Motivation


Relevant



foot pedals are used heavily in musical instruments (pianos, organs,
drums) and in mechanical equipment (cars, cranes, sewing machines)
hands-occupied situations
users with disabilities or with high-back or neck problems
Challenges


feet tend to be less precise than hands
practicality of having foot controls in the work environment (e.g. pedals
under a desk may be operated accidentally, etc)
Touchpad




a pointing device featuring a specialized surface that
can translate the motion and position of a user's fingers
to a relative position on screen
operated by stroking a finger over the surface
used mainly in laptop computers
good ‘acceleration’ settings important

fast stroke



lots of pixels per inch moved
initial movement to the target
slow stroke


less pixels per inch
for accurate positioning
Trackball and thumbwheels
Trackball

ball is rotated inside static housing

like an upside down mouse!
separate buttons for picking/selection
 very fast for gaming
 used in some portable and notebook computers

Thumbwheels
two dials for X-Y cursor position
 for fast scrolling – single dial on mouse

Joystick and keyboard nipple
Joystick



pressure of stick translates to velocity of movement
buttons for selection e.g. on top or on front like a trigger
often used for computer games, aircraft controls and 3D navigation
Keyboard nipple or pointing stick





a pointing device typically mounted in a keyboard on laptops
pointing stick can be considered to be a miniature joystick
pointing stick operates by sensing applied force
the velocity of the pointer depends on the applied force
force applied on the pointing stick is echoed on the screen by movements of
the pointer
Touch-sensitive screen


Detect the presence of finger or stylus on the screen.
Increasingly being accepted in the mass market especially in
mobile devices e.g. smartphones and tablets



Advantages:




direct pointing device
more natural gestures through multitouch (i.e. ability to recognize the presence of two
or more points of contact with the screen surface, and therefore supporting more
natural gestures such as pinching and reverse pinching)
fast, and requires no specialised pointer
good for menu selection
suitable for use in hostile environment: clean and safe from damage.
Disadvantages:



finger can mark screen
imprecise (finger is a fairly blunt instrument!)
if screen is vertical, lifting arm can be tiring unless supported
Stylus




small pen-like pointer to draw directly on screen
may use touch sensitive surface or magnetic detection
used in PDA, tablets PCs and drawing tables
pros
very direct
 obvious to use


cons

can obscure screen
Skinput: Body as an Input Surface
30



CHI’2010
Use a tiny projector on body to show menus
Microphones to listen to taps on hand/arm

Signal processing and machine learning to
differentiate positions
Pointing Devices

Integrated gaze and face tracking system. A camera and IR
light source tracks gaze by computing the angle between the
corneal reflection and the centroid of the pupil.
Eye gaze or eye tracking


a laser light reflects off the retina
provides a control interface by tracking eye/visual
movement on the screen





e.g. look at a menu item to select it
high accuracy requires headset
cheaper and lower accuracy devices available
mainly used for evaluation of interactive systems e.g. in
order to provide information on clicking and scrolling
patterns of web users
it is also relevant for disabled users
Discrete positioning controls

in phones, TV controls etc
 mini-joysticks
(mobile phones)
 central cursor pads & discrete left-right, updown buttons (TV remote controls)
 e.g.

+/- for controlling volume & navigating channels
mainly for menu selection
Pointing Devices (6)
Smart Systems
Minority Report … reality or fiction?
DISPLAY DEVICES
bitmap screens (CRT & LCD)
large & situated displays
digital paper
Output Devices

Permanent output
 Paper
 Microfiche
 Film

Impermanent output
 Screen
 Sound
 Tactile
Bitmap displays



bitmap display or raster scan
the whole screen is scanned sequentially and horizontally
screen is vast number of coloured dots
Visual Output

Cathode Ray Tube (CRT)
Text based to high resolution
 Produce minor health hazards




Since they are radiation emitting
devices!
Liquid Crystal Display (LCD)
Alternative systems might not
use a screen …
Lights
 Gauges
 Dials
 Can you give examples?

Large and situated displays
Large displays
 Used for meetings, lectures, etc
 Some use gas plasma technology to create large flat bitmap
displays
 Where very large screen areas are required, several smaller
screens, either LCD or CRT, can be placed together in a video
wall where each screen displays a portion of the whole (or
separate sub-image) and the result is an enormous image
Situated displays
 As well as for lectures and meetings, display screens can be
used in various public places to display information; they could
even be interactive
 These are often called situated displays as they take their
meaning from the location in which they are situated i.e. the
meaning of information or interaction is related to the location
PAPER: PRINTING AND SCANNING
print technology
fonts, page description, WYSIWYG
scanning, OCR
Permanent Output

On paper using printers
 Impact
Printers (Noisy)
 Dot
Matrix (Slow + Low Quality)
 Daisy Wheel (Very Slow + 1 Font)
 Chain and Band printers (Fast + Expensive)
 Non-Impact
Printers (Silent, High Quality, Fast)
 Inkjet
 Thermal
(Expensive)
 Laser (Cost effective + Colour expensive)
Permanent Output

Printers
Speed measured in Pages Per Minutes
 Resolution measured in Dots Per Inch (dpi)




Store information in firmware such as




80 dpi (dot matrix) = Near Letter Quality
300 dpi – 1200 dpi (non-impact) = Near Book Quality
Typeface (Fonts)
Character set
Size
More complex printers use Page Description Language (PDL)


Communicates complex commands directly to printer
Used in pdf and postscript
Fonts

Font – the particular style of text
Courier font
Helvetica font
Palatino font
Times Roman font
§´  (special symbol)

Size of a font
This is ten point Helvetica
This is twelve point
This is fourteen point
This is eighteen point
and this is twenty-four point
Fonts (ctd)
Pitch
 fixed-pitch
– every character has the same width e.g.
Courier
 variable-pitched – some characters wider e.g.
compare the “i” and the “m” in Times New Roman
Serif or Sans-serif
 sans-serif
– square-ended strokes
e.g. Helvetica
 serif
– with splayed ends (such as)
e.g. Times Roman
Readability of text

lowercase
All
lowercase text considered to be
easier to read than all uppercase

UPPERCASE
better
for individual letters and non-
words
e.g. flight numbers: BA793 vs. ba793
Scanners
Take paper and convert it into a
digital document
 Useful in

 desktop
publishing for incorporating
photographs and other images
 document storage and retrieval systems,
instead of physical paper storage
Optical character recognition


OCR converts bitmap back into text
different fonts
 create
problems for simple “template matching”
algorithms
 more complex systems segment text, decompose it into
lines and arcs, and decipher characters that way

page format
 columns,
pictures, headers and footers
VIRTUAL REALITY AND 3D
INTERACTION
positioning in 3D space
moving and grasping
seeing 3D (helmets and caves)
Positioning and viewing in 3D

Positioning in 3D space





3D mouse: six-degrees of movement: x, y, z + roll, pitch, yaw
data glove: to detect finger position
VR helmets: to detect head motion and possibly eye gaze
whole body tracking: accelerometers, image/video processing, etc
Viewing in 3D

often use stereoscopic vision ( two 2D images are presented separately to
the left and right eye of the viewer, which are then combined in the brain
to give the perception of depth), but also possible to use real 3D display
(displaying an image in three full dimensions)
Simulators and VR caves




scenes projected on walls
realistic environment
real controls
other people
PHYSICAL CONTROLS, SENSORS
ETC.
special displays and gauges
sound, touch, feel, smell
physical controls
environmental and bio-sensing
Sounds

can take many forms
beeps
 bongs (deep ringing sounds)
 clonks (loud thudding sounds)
 whistles
 whirrs (like the sound of rapidly vibrating wings)


used for
error indication (e.g. incorrect command/input) or risky
situations (e.g. when deleting files)
 confirmation of actions e.g. keyclick
 notification of events/status e.g. a new email has arrived,
download has been completed

Haptic devices - example
BMW iDrive
 for selecting and controlling menus
 one feels little bumps along the way as one maneuvers through
options and menus
 makes it easier to select options by feel
 uses haptic technology from Immersion Corp
Physical controls

specialist controls needed …
 industrial
controls, consumer products, etc.
easy-clean
smooth buttons
large buttons
multi-function
control
clear dials
tiny buttons
Environment and bio-sensing

sensors all around us
 car
courtesy light
 small
switch on door
 ultrasound
 security,
 RFID
detectors
washbasins
security tags in shops
 temperature, weight, location
MediaCup

cup has sensors



heat, movement, pressure
broadcasts state (IR)
used for awareness

user is moving, drinking, …
Han's purpose to drink coffee
incidentally colleagues are aware
OTHER ASPECTS
Compression
reduce amount of storage required
 lossless

 recover
exact text or image – e.g. GIF, ZIP
 look for commonalities:
 text:

AAAAAAAAAABBBBBCCCCCCCC
10A5B8C
lossy
 recover
something like original – e.g. JPEG, MP3
 exploit perception
 JPEG:
lose rapid changes and some colour
 MP3: reduce accuracy of drowned out notes
Storage formats - media

Images:
many storage formats :
(PostScript, GIFF, JPEG, TIFF, PICT, etc.)
 plus different compression techniques
(to reduce their storage requirements)


Audio/Video
again lots of formats :
(QuickTime, MPEG, WAV, etc.)
 compression even more important
 also ‘streaming’ formats for network delivery

Finite processing speed
Designers tend to assume fast processors, and
make interfaces more and more complicated
 But problems occur, because processing cannot
keep up with all the tasks it needs to do

 cursor
overshooting because system has buffered
keypresses
 icon wars - user clicks on icon, nothing happens, clicks
on another, then system responds and windows fly
everywhere

Also problems if system is too fast - e.g. help
screens may scroll through text much too rapidly
Computer Memory

Short Term
 RAM

Long Term
 Secondary
 Hard
Backing Storage
disk
 Disk drive
 CD’s
 Memory Sticks
Permanent Storage




Cheaper
Slower
Permanent (Non-volatile)
Several types



Different capacities







Magnetic
Optical
Floppies 1.4 Mb
Memory Stick 64 Mb +
CD 640 Mb
DVD 4 Gb – 18 Gb
Blue Ray 15 Gb – 30 Gb
Harddisk 120 Gb + … Tera Bytes
Two writing types


Rewritable (Hard disk)
Write Once Read Many WORM (CD, punch cards)
Recap: Boolean Logic
Boolean Expression
Boolean Circuit
Truth table:
Value of E for every
possible D, S.
TRUE=1; FALSE= 0.
E = S AND D
S
E
D
D
0
S
0
E
0
0
1
1
1
0
0
1
1
0

Truth table has
2 k rows if
the number of
variables is k
Combinational circuit for binary addition?

25
11001
+ 29
11101
54
110110
Want to design a circuit to add any two N-bit
integers.
Is the truth table method useful for N=64?
A Full Adder (from handout)
R-S Flip-Flop
S
R
M
A more convenient form of memory
No
“undefined”
outputs ever!
Fact: “Data Flip-Flop” or “D flip flop”;
can be implemented using R-S flip flops.
“Register” with 4 bits of memory
RAM: Implementing “Write”
Data
Decoder
(Demux)
Clock
RAM
The decoder selects
which cell in the RAM
gets its “Write” input
toggled
(simple combinational
circuit; see logic handout)
K-bit address
(in binary)
Greatly simplified view
of modern CPUs.
Program (in binary)
stored in memory
Memory Registers
Arithmetic and
Logic Unit
(ALU)
Control FSM
Instruction PointerLots of Custom Hardware
RAM
The “symphony” inside a computer
Clock
Combinational
circuit
Memory
Clocked
Sequential
Circuit
(aka
Synchronous
Circuits)
Limitations on interactive performance
Computation bound

Computation takes too long, causing frustration for the user
Storage channel bound

Bottleneck in transfer of data from disk to memory
Graphics bound

Common bottleneck: updating displays requires a lot of effort sometimes helped by adding a graphics co-processor optimised
to take on the burden
Network capacity

Many computers networked - shared resources and files,
multiple access to printers, etc - but interactive performance can
be reduced by slow network speed
Viruses and Worms
Automated ways of breaking in;
Use self-replicating programs
(Recall self-replicating programs:
Print the following line twice, the second time
in quotes. “Print the following line twice, the
second time in quotes.” )
Email Viruses
Infected program, screen saver, or Word
document launches virus when opened
 Use social engineering to entice you to open the
virus attachment
 Self-spreading: after you open it, automatically
emails copies to everyone
in your address book

The Melissa Virus (1999)





Social engineering: Email says
attachment contains porn site
passwords
Self-spreading: Random 50
people from address book
Traffic forced shutdown of
many email servers
$80 million damage
20 months and $5000 fine
David L. Smith
Aberdeen, NJ
Combating Viruses
Constant battle between attackers and defenders
Example:
Anti-virus software finds “signature” of known virus
 Attacker response: Polymorphic virus –
to thwart detection, change code when reproduced
 Anti-virus software adapts to find some kinds of
polymorphism
 But an infinite number of ways to permute viruses available
to attackers

The Morris Worm (1988)






First Internet worm
Created by student at Cornell
Exploited holes in email servers,
other programs
Infected ~10% of the net
Spawned multiple copies,
crippling infected servers
Sentenced to 3 years probation,
$10,000 fine, 400 hours
community service
Robert Tappan Morris
The Slammer Worm (2003)







Fastest spreading worm to date
Only 376 bytes—Exploited buffer overflow in Microsoft
database server products
Spread by sending infection packets to random servers as fast
as possible, hundreds per second
Infected 90% of vulnerable systems within 10 minutes!
200,000 servers
No destructive payload, but packet volume shut down large
portions of the Internet for hours
911 systems, airlines, ATMs — $1 billion damage!
Patch already available months previously,
but not widely installed
Zombies
Attacker’s
Program
Bot
Bot program runs silently in the background, awaiting
instructions from the attacker
DDOS Attacks
“Distributed Denial of Service”
“Attack www.store.com”
Objective: Overwhelm target site with traffic
Sending Spam
“Forward this message:
Subject: Viagra!
…”
Messages are hard to filter because there are
thousands of senders
Spyware/Adware



Hidden but not
self-replicating
Tracks web
activity for
marketing, shows
popup ads, etc.
Usually written by
businesses: Legal
gray area
Conclusion
HCI depends on computer operating as an
intermediary, achieving appropriate interaction
through a variety of I/O devices
WHAT COMPUTERS
JUST CANNOT DO.
M.C. Escher
Print Gallery
High-level description of program that selfreproduces
Print 0
A
Print 1
.
.
.
Print 0
......
B
......
......
......
}
}
Prints binary code of B
Takes binary string on tape,
and in its place prints (in
English) the sequence of
statements that produce it,
followed by the translation
of the binary string into
English.
How do you represent music?

Score:

Audio samples

Spectrogram
Creating music: Synthesis
Review Questions





What are the most common interaction devices?
What are their limitations?
How can some of them be overcome by novel devices?
Suggest ideas for an interface which uses speech and
sound effectively.
How can sound be used to convey information?
Review Questions



What are some of the factors you might consider when
designing systems like a public access interactive kiosk?
Identify the strengths and weaknesses of multimedia
and hypermedia systems.
Discuss how the speed of access to memory can affect
the speed of an interactive computer system. Which
other factors can affect such speed?
Revision