A Toolkit for Managing User
Attention in Peripheral Displays
Tara Matthews, Anind K. Dey*‫٭‬, Jennifer Mankoff‫٭‬, Scott Carter, Tye Rattenbury
EECS Department, UC Berkeley
*Intel-Berkeley Research
‫٭‬currently of HCII, Carnegie Mellon
Problematic Scenario
• Bus schedule
– Requires a task switch to use
– Doesn’t afford continuous monitoring
RTE
NUM
65
65
65
65
65
65
65
65
65
65
65
65
65
65
65
65
65
65
65
65
10/27/2004
LAWR
HALL
605a
635a
735a
835a
935a
1035a
1135a
1235p
135p
235p
335p
GRIZ
SR.
703a
803a
903a
1003a
1103a
1203p
103p
203p
303p
EUCL
GRIZ
616a
646a
716a
746a
816a
846a
916a
946a
1016a
1046a
1116a
1146a
1216p
1246p
116p
146p
216p
246p
316p
346p
CNTR
SHAT
630a
700a
730a
800a
830a
900a
930a
1000a
1030a
1100a
1130a
1200p
1230p
100p
130p
200p
230p
300p
330p
400p
Peripheral Display Toolkit
2
Solution: Peripheral Displays
• Provide awareness with min attention
• Separate from primary task
• Bus LED Display
+ Get bus info at a glance
+ Dynamically manages attention it attracts
– Hard to accomplish
10/27/2004
Peripheral Display Toolkit
3
Why is creating PDs hard?
• Need to abstract info to be glance-able
• Need mechanisms for dynamically
managing attention PDs attract:
– Deciding attention levels to attract
(notification levels)
– Displaying info appropriately (transitions)
• Our contribution:
– Peripheral Display Toolkit (PTK) supports
these key issues in PD creation
10/27/2004
Peripheral Display Toolkit
4
Example PTK Applications
• Remote Activity
–
–
–
–
Social Guitar
Audio Monitor
Motion Monitor
Remote Awareness Display
• Bus Displays
– Bus Mobile
– Bus LED
Orb showing remote activity
• Instant Messenger Status
Social Guitar
10/27/2004
IM Picture Frame
Bus LED
Peripheral Display Toolkit
BusMobile
5
Example PTK Applications
• Remote Activity
–
–
–
–
Social Guitar
Audio Monitor
Motion Monitor
Remote Awareness Display
• Bus Displays
– Bus Mobile
– Bus LED
Orb showing remote activity
• Instant Messenger Status
Social Guitar
10/27/2004
IM Picture Frame
Bus LED
Peripheral Display Toolkit
BusMobile
6
Example PTK Applications
• Remote Activity
–
–
–
–
Social Guitar
Audio Monitor
Motion Monitor
Remote Awareness Display
• Bus Displays
– Bus Mobile
– Bus LED
Orb showing remote activity
• Instant Messenger Status
Social Guitar
10/27/2004
IM Picture Frame
Bus LED
Peripheral Display Toolkit
BusMobile
7
Example PTK Applications
• Remote Activity
–
–
–
–
Social Guitar
Audio Monitor
Motion Monitor
Remote Awareness Display
• Bus Displays
– Bus Mobile
– Bus LED
Orb showing remote activity
• Instant Messenger Status
Social Guitar
10/27/2004
IM Picture Frame
Bus LED
Peripheral Display Toolkit
BusMobile
8
Example PTK Applications
+ Manage attention with
abstraction, notification
levels, & transitions
+ Glance-able
+ Few lines of code
+ Simplified code design
& code re-use
Social Guitar
10/27/2004
IM Picture Frame
Orb showing remote activity
Bus LED
Peripheral Display Toolkit
BusMobile
9
Simple Example: Motion Monitor
no/low motion
medium motion
high motion
10/27/2004
Peripheral Display Toolkit
10
Outline
• PTK: Motivation & Examples
• Managing attention
– Abstraction
– Notification levels
– Transitions
• Toolkit Architecture & Library
• Conclusion
10/27/2004
Peripheral Display Toolkit
11
Managing User Attention
• Attention is a crucial issue in design
– Significant impact on success
– More info can be monitored w/o overburdening
• Attention management is difficult
– Used to designing apps that attract full attention
– Attention attracted should match info urgency
info urgency
user attention
none
Attention
focal
• Literature surveys led to key characteristics
of attention management
10/27/2004
Peripheral Display Toolkit
12
Attention Management Characteristics
From survey of peripheral displays:
1. Abstraction
•
enables lower-attention info consumption
2. Notification levels
•
help determine how much attention to
attract based on urgency of info
3. Transitions
•
enable display of info to match urgency
info urgency
10/27/2004
user attention
Peripheral Display Toolkit
13
Abstraction
• Enables lower-attention consumption of info
• Extracting features or reducing fidelity of info
• Motion Monitor: replaces video images with
general motion amount (high, medium, none)
video
images

abstraction

amount of
motion
• high motion
• medium motion
• no/low motion
10/27/2004
Peripheral Display Toolkit
14
Notification Levels
• Indicate the urgency of info
– Low levels = non-urgent info
– High levels = more urgent info
• Motion Monitor: doesn’t show urgent
info, so uses low notification levels
10/27/2004
Peripheral Display Toolkit
15
More on Notification Levels
awareness
unconscious
0%
attention
100%
• Attention is (roughly) scalar – PD designers must
decide where on the scale displayed info should fall
10/27/2004
Peripheral Display Toolkit
16
More on Notification Levels
awareness
unconscious
0%
attention
100%
Attention Type
1. None
10/27/2004
Peripheral Display Toolkit
17
More on Notification Levels
awareness
unconscious
0%
Attention Type
1. None
10/27/2004
100%
attention
Notification Level
Ignore
Peripheral Display Toolkit
18
More on Notification Levels
awareness
unconscious
0%
Attention Type
1. None
2. Inattention
10/27/2004
100%
attention
Notification Level
Ignore
Peripheral Display Toolkit
19
More on Notification Levels
awareness
unconscious
0%
Attention Type
1. None
2. Inattention
10/27/2004
100%
attention
Notification Level
Ignore
Change blind
Peripheral Display Toolkit
20
More on Notification Levels
awareness
unconscious
0%
Attention Type
1. None
2. Inattention
3. Divided
10/27/2004
100%
attention
Notification Level
Ignore
Change blind
Peripheral Display Toolkit
21
More on Notification Levels
awareness
unconscious
0%
Attention Type
1. None
2. Inattention
3. Divided
10/27/2004
100%
attention
Notification Level
Ignore
Change blind
Make aware
Peripheral Display Toolkit
22
More on Notification Levels
awareness
unconscious
0%
Attention Type
1. None
2. Inattention
3. Divided
4. Focused
10/27/2004
100%
attention
Notification Level
Ignore
Change blind
Make aware
Peripheral Display Toolkit
23
More on Notification Levels
awareness
unconscious
0%
Attention Type
1. None
2. Inattention
3. Divided
4. Focused
10/27/2004
100%
attention
Notification Level
Ignore
Change blind
Make aware
Interrupt
Peripheral Display Toolkit
24
More on Notification Levels
awareness
unconscious
0%
10/27/2004
100%
attention
Attention Type
1. None
2. Inattention
3. Divided
4. Focused
5. Focused + interaction
Notification Level
Ignore
Change blind
Make aware
Interrupt
Demand action
Peripheral Display Toolkit
25
Transitions
• Enable display of info to match
urgency
• Depend on notification level
–
–
–
–
–
Ignore: no changes
Change blind: fading, slow motion, tiny updates
Make aware: discrete, abrupt updates
Interrupt: flashing, beeping, vibration
Demand action: interrupt until user does action
• Motion Monitor: varies pulse speeds
for different notification levels
10/27/2004
Peripheral Display Toolkit
26
Outline
• PTK: Motivation & Examples
• Managing attention
– Abstraction
– Notification levels
– Transitions
• Toolkit Architecture & Library
• Conclusion
10/27/2004
Peripheral Display Toolkit
27
PTK Architecture
1. Support for managing impact on
human attention using abstraction,
notification levels, and transitions
2. Simplified code design and
code re-use
3. Library of common PD components
Input
Abstractor
Output
Notification
Map
Transition
10/27/2004
Peripheral Display Toolkit
28
Motion Monitor Architecture
Input-Side
Discovery
Server
Output-Side
Motion Monitor
Camera
Input
10/27/2004
PTK
Discovery
Server
Motion
Abstractor
Peripheral Display Toolkit
Notification
Map
Trans
Orb
Widget
29
Motion Monitor Architecture
Input-Side
Discovery
Server
Output-Side
Motion Monitor
Camera
Input
10/27/2004
PTK
Discovery
Server
Motion
Abstractor
Peripheral Display Toolkit
Notification
Map
Trans
Orb
Widget
30
Motion Monitor Architecture
Input-Side
Discovery
Server
Output-Side
Motion Monitor
Camera
Input
PTK
Discovery
Server
Motion
Abstractor
Notification
Map
Trans
Orb
Widget
motion = medium
10/27/2004
Peripheral Display Toolkit
31
Motion Monitor Architecture
Input-Side
Discovery
Server
Output-Side
Motion Monitor
Camera
Input
PTK
Discovery
Server
Motion
Abstractor
Notification
Map
Trans
Orb
Widget
motion = medium
notification level
= change blind
10/27/2004
Peripheral Display Toolkit
32
Motion Monitor Architecture
Input-Side
Discovery
Server
Output-Side
Motion Monitor
Camera
Input
PTK
Discovery
Server
Motion
Abstractor
Notification
Map
Trans
Orb
Widget
motion = medium
notification level
= change blind
10/27/2004
Peripheral Display Toolkit
33
Motion Monitor Architecture
Input-Side
Discovery
Server
Output-Side
Motion Monitor
Camera
Input
PTK
Discovery
Server
Motion
Abstractor
Notification
Map
Trans
Orb
Widget
5 lines of logic code
10/27/2004
Peripheral Display Toolkit
34
Remote Awareness Display
Add notification
Count # people,
How much motion? level
Phone ring?
Voices?
10/27/2004
Peripheral Display Toolkit
35
Motion Monitor 
Remote Awareness Display
Input-Side
Camera
Input
Discovery
Server
Output-Side
Motion Monitor
PTK
Discovery
Server
Exact Match
(Motion)
Trans
Orb
Widget
Motion
Abstractor
10/27/2004
Peripheral Display Toolkit
36
Motion Monitor 
Remote Awareness Display
Input-Side
Camera
Input
10/27/2004
Motion
Abstractor
Discovery
Server
Output-Side
Motion Monitor
PTK
Discovery
Server
Peripheral Display Toolkit
Exact Match
(Motion)
Trans
Orb
Widget
37
Motion Monitor 
Remote Awareness Display
Input-Side
Camera
Input
Motion
Abstractor
# People
Abstractor
10/27/2004
Discovery
Server
Output-Side
Motion Monitor
PTK
Discovery
Server
Peripheral Display Toolkit
Exact Match
(Motion)
Trans
Orb
Widget
38
Motion Monitor 
Remote Awareness Display
Input-Side
Camera
Input
Motion
Abstractor
# People
Abstractor
Audio
Input
Discovery
Server
Output-Side
Motion Monitor
PTK
Discovery
Server
Exact Match
(Motion)
Trans
Orb
Widget
Phone
Abstractor
Voice
Abstractor
10/27/2004
Peripheral Display Toolkit
39
Motion Monitor 
Remote Awareness Display
Input-Side
Camera
Input
Motion
Abstractor
# People
Abstractor
Audio
Input
Discovery
Server
Output-Side
Motion Monitor
PTK
Discovery
Server
Trans
Orb
Widget
Exact Match
(Audio)
Phone
Abstractor
Threshold
(# People)
Voice
Abstractor
10/27/2004
Exact Match
(Motion)
Peripheral Display Toolkit
40
Motion Monitor 
Remote Awareness Display
Input-Side
Camera
Input
Motion
Abstractor
# People
Abstractor
Audio
Input
Discovery
Server
Output-Side
Motion Monitor
PTK
Discovery
Server
Orb Wrapper
Trans
Orb
Widget
Exact Match
(Audio)
Phone
Abstractor
Threshold
(# People)
Voice
Abstractor
10/27/2004
Exact Match
(Motion)
Peripheral Display Toolkit
41
Remote Awareness Display
Input-Side
Camera
Input
Motion
Abstractor
# People
Abstractor
Audio
Input
Discovery
Server
Output-Side
Remote Awareness Display
PTK
Discovery
Server
Exact Match
(Motion)
Orb Wrapper
Trans
Exact Match
(Audio)
Phone
Abstractor
Threshold
(# People)
Voice
Abstractor
Orb
Widget
Tikr. Wrapper
Trans
Ticker
Widget
54 – total lines of code
10/27/2004
Peripheral Display Toolkit
42
General Architecture Diagram
Input-Side
Input 1
Abs. 1
Discovery
Server
Output-Side
Peripheral Display 1
(Abstractors)
(Notification
Maps)
Abs. 1
N.M. 1
.
.
.
Abs. 2
N.M. 2
Out 2
…
Input N
Abs. N
N.M. N
Out N
Abs. N
Input 2
PTK
Discovery
Server
(Output Widgets w/
optional Transition)
Trans
Out 1
Peripheral Display 2
10/27/2004
Abs. 1
N.M. 1
Abs. 2
N.M. 2
Abs. N
N.M. N
Peripheral Display Toolkit
Out 1
Trans
Out 2
…
Out N
43
Library Components
• Input
– audio, camera, Phidgets, Context Toolkit,
online calendars, news, stocks, weather,
Web page parser, serial port communication
10/27/2004
Peripheral Display Toolkit
44
Library Components
• Input
– audio, camera, Phidgets, Context Toolkit,
online calendars, news, stocks, weather,
Web page parser, serial port communication
• Output
– ticker text, Ambient Orb, Phidgets
10/27/2004
Peripheral Display Toolkit
45
Library Components
• Input
– audio, camera, Phidgets, Context Toolkit,
online calendars, news, stocks, weather,
Web page parser, serial port communication
• Output
– ticker text, Ambient Orb, Phidgets
• Abstractors
– motion, people counting, voices, phone ringing
10/27/2004
Peripheral Display Toolkit
46
Library Components
• Input
– audio, camera, Phidgets, Context Toolkit,
online calendars, news, stocks, weather,
Web page parser, serial port communication
• Output
– ticker text, Ambient Orb, Phidgets
• Abstractors
– motion, people counting, voices, phone ringing
• Notification
– exact match, threshold, contains,
degree of change
10/27/2004
Peripheral Display Toolkit
47
Transitions
• PTK provides architectural support
for implementing as animations
• Timing
• Number of steps
• Flashing (single, repeated)
• Tied to output
– Depend on output modality
– Example transitions: color changing and
pulsing (orb); fading, motion, & flashing
(ticker text)
10/27/2004
Peripheral Display Toolkit
48
Future Work & Conclusion
• PTK supports the creation of PDs
• Focuses on managing user attention
• To do this, uses abstraction,
notification levels, transitions
• Validated with 7 example applications
– Supports attention management
– Facilitates code re-use
• Future work
– Incorporate interruptibility and
local context sensing
– Cross-modality animations
– PD evaluation support
10/27/2004
Peripheral Display Toolkit
49
Questions?
• For more information & downloads:
www.cs.berkeley.edu/projects/io/ptk/
tmatthew@cs.berkeley.edu
Thanks to…
NSF, Ed De Guzman, Jeff Heer,
Scott Klemmer, Berkeley & CMU
10/27/2004
Peripheral Display Toolkit
50
What about user context?
• PDs must 2 sets of attention issues:
– User context
• interruptibility, primary activity, focus of attention
– Attention management
• balancing info urgency and user attention
• PTK focuses on attention management
– User context invasive to sense, not practical to
assume PDs have access
– Management issues are independent of user
context
• Context input can be considered on a
library level (in notification maps)
10/27/2004
Peripheral Display Toolkit
51
Related Work
• AROMA: architectural support for abstraction
• Context Toolkit: infrastructure for context
input and some abstraction
• Phidgets: library of hardware widgets for input
and output
• Real World Interfaces Toolkit: library of
X10 output widgets
• iStuff: infrastructure for distributed inputs and
outputs
10/27/2004
Peripheral Display Toolkit
52
Focus Attention
awareness
unconscious
0%
attention
100%
• Peripheral display + attracting focused attention
= contradiction? Nope!
– Notifications enable displays to be peripheral
when not displaying urgent info
– Attention is scalar: PDs should cover entire range
10/27/2004
Peripheral Display Toolkit
53
PTK Validation
Created 7 example applications that illustrate:
• Supports attention management
• 3 used abstraction
• 5 used notification levels
and transitions
• Easier to create
• 5-55 lines of logic code
• Facilitates code re-use
• used same input for 4 displays
Social Guitar
10/27/2004
IM Picture Frame
Bus LED
Peripheral Display Toolkit
Orb showing remote
activity
BusMobile
54
Notification Maps
Exact Match Notification Map
If event value equal to:
Set notification level to:
no motion
ignore
med motion
change blind
high motion
make aware
(null)
interrupt
(null)
demand action
10/27/2004
Peripheral Display Toolkit
55
Notification Maps
• Exact match
• checks event for equality w/specified values
• Threshold
• determines if event within specified ranges
• Contains
• checks if event contains specified strings
• Degree of change
• determines if event has changed by a
specified amount since last event
10/27/2004
Peripheral Display Toolkit
56
Introduction
• Peripheral displays (PDs) enable you
to monitor many info sources while
maintaining a calm environment
– But, only calm if designed to manage
the attention they attract
• PDs are hard to create
– No existing tools or guidelines for
managing human attention
• Peripheral Display Toolkit (PTK):
– Enables easier creation of PDs that
manage human attention
10/27/2004
Peripheral Display Toolkit
57
Why Peripheral Displays?
• Provide awareness w/ minimal attention
• Separate from user’s primary activity
• Non-focal display not used unless
providing peripheral info
• Important to ubicomp vision of many
devices to one user
10/27/2004
Peripheral Display Toolkit
58
Peripheral Display Toolkit (PTK)
• Supports the creation of PDs
1. Focuses on managing user attention
2. Simplifies code
3. Provides library of common components
• Managers user attention with
– Abstraction
– Notification levels
– Transitions
10/27/2004
Peripheral Display Toolkit
59
Characteristics of Attention
Management
• Abstraction of info to be glance-able
• Dynamically managed attention
– Notification levels for deciding how
much attention to attract
– Transitions for displaying info
appropriately
10/27/2004
Peripheral Display Toolkit
60
Peripheral Display Toolkit (PTK)
• Supports the creation of PDs
• Focuses on managing user attention
– Balance info urgency and user attention
• To do this, uses
– Abstraction
– Notification levels
– Transitions
10/27/2004
Peripheral Display Toolkit
61
More on Notification Levels
y = awareness
indirect cognitive
influence
direct cognitive
influence
alerting
focused
attention
ambient
divided
attention
change blind
x = attention
0%
preattention
10/27/2004
setting (layout, color, objects, names)
inattention
Peripheral Display Toolkit
100%
conscious perception (x > 0)
62