A case study in UI design and
evaluation for computer security
Rob Reeder
Dependable Systems Lab
Carnegie Mellon University
February 16, 2006
Memogate: A user interface scandal !!
2
Overview

Task domain: Windows XP file permissions

Design of two user interfaces: native XP
interface, Salmon

Evaluation: Which interface was better?

Analysis: Why was one better?
3
Part 1: File permissions in Windows XP

File permissions task: Allow authorized
users access to objects, deny unauthorized
users access to objects

Objects: Files and folders

Users: People with accounts on the system

Access: 13 types, such as Read Data, Write
Data, Execute, Delete
4
Challenges for file permissions UI design

Maybe thousands of users – impossible to
set permissions individually for each

Thirteen access types – hard for a person to
remember them all
5
Grouping to handle users

Administrators

Power Users

Everyone

Admin-defined
6
A problematic user grouping
Xu
Ari
Bill
Cindy
Group A
Miguel
Yasir
Zack
Group B
7
Precedence rules

No setting = Deny by default

Allow > No setting

Deny > Allow
(> means “takes precedence over”)
8
Grouping to handle access types
Moral

Setting file permissions is quite complicated

But a good interface design can help!
10
The XP file permissions interface
11
The Salmon interface
ProjectF
Example task: Wesley

Initial state


Final state


Wesley allowed READ & WRITE from a group
Wesley allowed READ , denied WRITE
What needs to be done

Deny Wesley WRITE
13
What’s so hard?

Conceptually: Nothing!

Pragmatically:



User doesn’t know initial group membership
Not clear what changes need to be made
Checking work is hard
14
Learning Wesley’s initial permissions
1
2
Click “Effective
Permissions”
Click “Advanced”
3
4
View Wesley’s
Effective Permissions
Select Wesley
15
Learning Wesley’s group membership
Bring up Computer
5
Management interface
6
Click on
“Users”
Read
Wesley’s
group
membership
9
7
Doubleclick
Wesley
Click
“Member
Of”
8
Changing Wesley’s permissions
10
11
Deny
Write
Click
“Add…”
Click
“Apply”
12
17
Checking work
13
14
Click “Effective
Permissions”
Click “Advanced”
15
16
View Wesley’s
Effective Permissions
Select Wesley
18
XP file permissions interface: Poor
19
Part 2: Common security UI design problems

Poor feedback

Ambiguous labels

Violation of conventions

Hidden options

Omission errors
20
Problem #1: Poor feedback
1
2
Click “Effective
Permissions”
Click “Advanced”
3
4
View Wesley’s
Effective Permissions
Select Wesley
21
Salmon: immediate feedback
ProjectF
Problem #2: Labels (1/3)
Full control
Modify
Read & Execute
Read
Write
Special Permissions
Problem #2: Labels (2/3)
Full control
Traverse Folder/Execute File
List Folder/Read Data
Read Attributes
Read Extended Attributes
Create Files/Write Data
Create Folders/Append Data
Write Attributes
Write Extended Attributes
Delete
Read Permissions
Change Permissions
Take Ownership
Salmon: clearer labels
ProjectF
Problem #3: Violating interface conventions
Normally, a
clicking on a
checkbox only
changes that
checkbox –but
click a checkbox
in one of these
sets, and some
other
checkboxes may
also be changed
– confusing!
Salmon: better checkboxes
ProjectF
Problem #4: Hidden options
28
Problem #4: Hidden options
1
2
Click “Advanced”
Double-click entry
3
Click “Delete”
checkbox
29
Salmon: All options visible
ProjectF
Problem #5: Omission errors (1/2)


Omission error: Failure to complete a
necessary step in a procedure
Classic examples:



Forgetting to take your card out of the ATM after
receiving cash
Forgetting to take your original after making
photocopies
Omission errors are quite common in
security-based tasks


Forgetting to change a default password
Forgetting to restart a service after making
changes
31
Problem #5: Omission errors (2/2)

XP interface showed much potential for
omission errors:


Users failed to make necessary changes to
permissions that were hidden from view (e.g.
Change Permissions and Take Ownership)
User failed to check group membership, because
the group membership information was so hard
to find
32
Salmon: Feedback helps prevent omission errors
ProjectF
FLOCK: Summary of design problems

Feedback poor

Labels ambiguous

Omission error potential

Convention violation

Keeping options visible
34
Part 3: Evaluation of XP and Salmon

Conducted laboratory-based user studies

Formative and summative studies for
Salmon

I’ll focus on summative evaluation
35
Advice for user studies

Know what you’re measuring!

Maintain internal validity

Maintain external validity
36
Common usable security metrics




Accuracy – with what probability do users
correctly complete tasks?
Speed – how quickly can users complete
tasks?
Security – how difficult is it for an attacker
to break into the system?
Etc. – satisfaction, learnability, memorability
37
Measure the right thing!
Keystroke dynamics analysis poses a real threat to any
computer user. Hackers can easily record the sounds of users'
keystrokes and obtain sensitive passwords from them. We
address this issue by introducing a new typing method we call
"Babel Type", in which users hit random keys when asked to
type in their passwords. We have built a prototype and tested
it on 100 monkeys with typewriters. We discovered that our
method reduces the keystroke attack by 100%. This approach
could potentially eliminate all risks associated with keystroke
dynamics and increase user confidence. It remains an open
question, however, how to let these random passwords
authenticate the users.
38
Measurement instruments

Speed – Easy; use a stopwatch, time users

Accuracy – Harder; need unambiguous
definitions of “success” and “failure”

Security – Very hard; may require serious
math, or lots of hackers
39
Internal validity



Internal validity: Making sure your results
are due to the effect you are testing
Manipulate one variable (in our case, the
interface, XP or Salmon)
Control or randomize other variables





Use same experimenter
Experimenter reads directions from a script
Tasks presented in same text to all users
Assign tasks in different order for each user
Assign users randomly to one condition or other
40
External validity


External validity: Making sure your
experiment can be generalized to the real
world
Choose real tasks

Sources of real tasks:
• Web forums
• Surveys
• Your own experience

Choose real participants

We were testing novice or occasional filepermissions users with technical backgrounds
(so CMU students & staff fit the bill)
41
User study compared Salmon to XP

Seven permissions-setting tasks, I’ll discuss
two:



Wesley
Jack
Metrics for comparison:



Accuracy (measured as deviations in users’ final
permission bits from correct permission bits)
Speed (time to task completion)
Not security – left that to Microsoft
42
Study design

Between-participants comparison of
interfaces

12 participants per interface, 24 total

Participants were technical staff and
students at Carnegie Mellon University

Participants were novice or occasional file
permissions users
43
Wesley and Jack tasks
Wesley task

Initial state



Deny Wesley
WRITE
Initial state


Wesley allowed
READ, denied
WRITE
What needs to be
done


Wesley allowed
READ & WRITE
Final state

Jack task
Final state


Jack allowed READ,
WRITE, & ADMINISTRATE
Jack allowed READ,
denied WRITE &
ADMINISTRATE
What needs to be
done

Deny Jack WRITE &
ADMINISTRATE
Salmon outperformed XP in accuracy
Percent successful completions by
task
300%
100
100
50
25
0
Wesley task
25
XP
58
Salmon
75
improvement
Salmon
83
XP
Percent of Users Who
Correctly Completed
Tasks
43% improvement
Jack task
Task Name
45
Salmon outperformed XP in accuracy
p = 0.09
83
100
p < 0.0001
50
25
0
Wesley task
25
XP
58
Salmon
75
Salmon
100
XP
Percent of Users Who
Correctly Completed
Tasks
Percent successful completions by
task
Jack task
Task Name
46
Salmon did not sacrifice speed
Speed (Time-to-Task-Completion)
Results
250
173
150
Salmon
100
50
XP
183
Salmon
208
200
XP
Time (seconds)
208
Successful XP users
Successful Salmon users
0
Wesley task
Jack task
Task Name
47
Salmon did not sacrifice speed
250
Speed (Time-to-Task-Completion)
Results
p = 0.35
p = 0.20
173
150
Salmon
100
50
XP
183
Salmon
208
200
XP
Time (seconds)
208
Successful XP users
Successful Salmon users
0
Wesley task
Jack task
Task Name
48
Part 4: Analysis

What led Salmon users to better
performance?
49
How users spent their time - Wesley
45
40
35
All XPFP users
30
25
20
15
All Salmon users
Successful XPFP users
Successful Salmon users
wo
Ch
rk
ec
k
gr
ou
Le
ps
ar
n
in
te
rfa
Co
ce
ns
ul
tH
Se
el
tp
p
er
m
iss
M
io
an
ns
ag
e
wi
Re
nd
m
ow
ov
s
e
fro
m
AC
L
to
Ad
d
Ch
ec
k
AC
L
an
Pl
ta
sk
10
5
0
Re
ad
Time (seconds)
Average behavior time per participant for Wesley task
Behavior
50
Where Salmon did better - Wesley
45
40
35
All XPFP users
30
25
20
15
All Salmon users
Successful XPFP users
Successful Salmon users
wo
Ch
rk
ec
k
gr
ou
Le
ps
ar
n
in
te
rfa
Co
ce
ns
ul
tH
Se
el
tp
p
er
m
iss
M
io
an
ns
ag
e
wi
Re
nd
m
ow
ov
s
e
fro
m
AC
L
to
Ad
d
Ch
ec
k
AC
L
an
Pl
ta
sk
10
5
0
Re
ad
Time (seconds)
Average behavior time per participant for Wesley task
Behavior
51
Where XP did better - Wesley
45
40
35
All XPFP users
30
25
20
15
All Salmon users
Successful XPFP users
Successful Salmon users
wo
Ch
rk
ec
k
gr
ou
Le
ps
ar
n
in
te
rfa
Co
ce
ns
ul
tH
Se
el
tp
p
er
m
iss
M
io
an
ns
ag
e
wi
Re
nd
m
ow
ov
s
e
fro
m
AC
L
to
Ad
d
Ch
ec
k
AC
L
an
Pl
ta
sk
10
5
0
Re
ad
Time (seconds)
Average behavior time per participant for Wesley task
Behavior
52
How users spent their time - Jack
Average behavior time per participant for Jack task
60
All XPFP users
40
All Salmon users
30
Successful XPFP users
20
Successful Salmon users
10
wo
Ch
rk
ec
k
gr
ou
Le
ps
ar
n
in
te
rfa
Co
ce
ns
ul
tH
Se
el
tp
p
er
m
iss
M
io
an
ns
ag
e
wi
Re
nd
m
ow
ov
s
e
fro
m
AC
L
to
Ad
d
Ch
ec
k
AC
L
an
Pl
ta
sk
0
Re
ad
Time (seconds)
50
Behavior
53
Where Salmon did better - Jack
Average behavior time per participant for Jack task
60
All XPFP users
40
All Salmon users
30
Successful XPFP users
20
Successful Salmon users
10
wo
Ch
rk
ec
k
gr
ou
Le
ps
ar
n
in
te
rfa
Co
ce
ns
ul
tH
Se
el
tp
p
er
m
iss
M
io
an
ns
ag
e
wi
Re
nd
m
ow
ov
s
e
fro
m
AC
L
to
Ad
d
Ch
ec
k
AC
L
an
Pl
ta
sk
0
Re
ad
Time (seconds)
50
Behavior
54
Where XP did better - Jack
Average behavior time per participant for Jack task
60
All XPFP users
40
All Salmon users
30
Successful XPFP users
20
Successful Salmon users
10
wo
Ch
rk
ec
k
gr
ou
Le
ps
ar
n
in
te
rfa
Co
ce
ns
ul
tH
Se
el
tp
p
er
m
iss
M
io
an
ns
ag
e
wi
Re
nd
m
ow
ov
s
e
fro
m
AC
L
to
Ad
d
Ch
ec
k
AC
L
an
Pl
ta
sk
0
Re
ad
Time (seconds)
50
Behavior
55
Common UI problems summary

Feedback poor

Labels ambiguous

Omission error potential

Convention violation

Keeping options visible
56
User interface evaluation summary

Know what you’re measuring

Internal validity: Control your experiment

External validity: Make your experiment
realistic
57
Good UI design  Peace on Capitol Hill?
58
Thanks

Rob Reeder

Email: reeder@cs.cmu.edu
59
x-x-x-x-x-x-x END x-x-x-x-x-x-x-x
60