An Empirical User Study
of a Smartphone-Based
Access-Control System
Kami Vaniea
Joint work with Lujo Bauer, Lorrie Cranor, Mike Reiter and Rob Reeder
CMU Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
Physical access control
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
2
Limitations

Must delegate all access tokens in advance

Necessary to hide an access token for
emergency situations

Problems getting access tokens back

Once given out key can be copied

Requires users to carry additional objects
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
3
Smartphones

What about using smartphones for access
control?

Smartphone capabilities
• User interface
• Computing ability
• Communication

Smartphones are increasing in popularity
• Computational power of mobile phones also
increasing
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
4
Research questions

What are the usability challenges in building
a smartphone-based access-control
system?

How well does a deployed smartphonebased access-control system match users’
needs?
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
5
Outline
Introduction
Grey Overview & Deployment
Study 1: System Acceptance
Study 2: Policy Creation
Related Work
Conclusion
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
6
Grey

Smartphone based accesscontrol system

Used to open doors in the CIC
building

Allows users to grant access to
their doors from anywhere at any
time
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
7
Grey example
Lorrie
Kami
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Lorrie’s
Office
8
Grey advantages

Can easily delegate authority

In advance of the access

At the time of the access

Guarantee access is no longer allowed
after specified time
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
9
Field trial: environment
 30 doors
 Perimeter doors to a
large research area
 Offices
 Storage closets
 Conference room
 A lab
 A machine room
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
10
Users

Chose participants who work together

Wanted groups of users who share
resources

29 users
• 9 faculty
• 11 graduate students
• 7 technical staff
• 2 administrative assistants
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
11
Interview procedure
 Interviewed participants
• Security practices
• Types of resources managed and needed
 Gave participants a smartphone with Grey preinstalled and brief instruction on use
 Interviewed one month later
• Changes in security practices
• General reactions to Grey

Periodically conducted follow-up interviews at
approximately one month intervals
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
12
Data

Recorded approximately 30 hours of
interviews

System was actively used
• Logged 19,500 Grey accesses for 29 users
• Active users averaged 12 accesses a week
• Five users accessed their office almost
exclusively with Grey
• Users interacted with an average of 7.4
different doors during the study

Study lasted a year
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
13
Outline
Introduction
Grey Overview & Deployment
Study 1: System Acceptance
Study 2: Policy Creation
Related Work
Conclusion
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
14
Research question

What are the usability challenges in
building a smartphone-based accesscontrol system?
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
15
Design issues
 Analyzed interview data and identified five
different design issues
 Speed
 Failures
 Complex features
 Non-Grey users
 New uses
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
16
Issue 1: Perceived speed
Users quickly began to complain about
speed and convenience of unlocking doors
We knew Grey and keys required similar
amounts of time to open a door
Videotaped a highly trafficked door to better
understand how doors are opened
differently with Grey and keys
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
17
Issue 1: Videotaping
Videotaped participants
accessing kitchenette door
Videotaped two hours daily
after 6pm for two weeks
18 users taped
• 5 Grey participants
• 13 additional participants were
solicited as they passed
through the door
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
18
Issue 1: Average access times
Keys
Getting
keys
3.6 sec
5.4 sec
σ = 3.1
σ = 3.1
5.7 sec
σ = 3.6
Door
Closed
Door
opened
Stop in
front of
door
Total
14.7
sec
σ = 5.6
Grey
Getting
phone
8.4 sec
2.9 sec
σ = 2.8
σ = 1.5
Stop in
front of
door
Total
15.1
sec
3.8 sec
σ = 1.1
Door
opened
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Door
Closed
σ = 3.9
19
Issue 2: Failure
Cost of failure is potentially high
Rebooting a phone or door was considered
very inconvenient
Several users stopped using Grey actively
after a single inopportune failure
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
20
Issue 2: Delays interpreted as
failures
Delays can be interpreted as failures even
when the system is functioning perfectly
• Humans can be slow or unresponsive
Providing feedback on the
status of the request is very
important
• Did it arrive?
• Is a human currently responding?
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
21
Issue 3: Confusing features
Users would rather choose
a suboptimal solution that
they understand than one
with an uncertain outcome
Initially tried for concise
interface (top)
Adopted wizard solution
(bottom)
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
22
Issue 4: Non-Grey users
Grey is a service that becomes more
valuable as more people use it
Our participants were selected so that their
work network included others with Grey
Still had many people who would have
benefited if Grey participant could have
given access
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
23
Issue 4: Alice’s colleagues
Bob
No Grey
Marie
Frank
Lillian
Alice
Sue
Jake
Mark
Joe
Have Grey
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
24
Issue 5: Unanticipated uses
Unlocking door from inside the office
without having to stand
Unlocking nearby door for someone else
without leaving office
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
25
Study 1: summary
1.
Perceived speed and convenience are critical to
user acceptance
2.
A single failure can strongly discourage
adoption
3.
Users won’t use features they don’t understand
4.
Important to consider occasional users of the
system
5.
Unanticipated uses can improve acceptance
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
26
Outline
Introduction
Grey Overview & Deployment
Study 1: System Acceptance
Study 2: Policy Creation
Related Work
Conclusion
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
27
Research question

How well does a deployed smartphonebased access-control system match users’
needs?

Do users make more or less secure
access-control decisions when using Grey
than when using physical keys?
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
28
Policies

A policy is a collection of rules

A rule is a tuple containing a user, resource
and condition (Bob, Alice’s office, true)
Bob
True
Alice’s
Office
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
29
Methodology overview

Examined access-control policies created
by 8 resource owners
• 8 offices
• 1 machine room

Using interviews we created ideal, key and
Grey policies for each of 9 resources

Compared ideal and implemented rules
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
30
Ideal policies

Ideal Policy – Policy the user would enact
if not restricted by technology

Based on interview data

Looked at not only what was enacted but
endeavored to determine why
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
31
Policy synthesis
. . . Garry Frank
True
Rick
Larry
Logged
Joan
Mary . . .
Lab owner
is notified
Logged
Logged
False
Charlie’s
Lab
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
32
Ideal conditions

True (can access anytime)

Logged

Owner notified

Owner gives real-time approval

Owner gives real-time approval and witness
present

Trusted person gives real time approval and is
present

False (no access)
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
33
Policy analysis

We compared each of the 244 ideal access rules,
with the key and Grey rules and marked them as:
• False Accept – User not required to fulfill all conditions
required by the ideal policy
• False Reject – User must fulfill conditions not required
by the ideal policy
• Faithfully Implemented – Matched the ideal policy
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
34
Policy analysis example
Charlie’s
Lab
Faithfully
False Accept False Reject
implemented
Alice
Ideal
Conditions
Key
Conditions
Access
anytime
Has a key
Bob
Owner
Notified
Has a key
Sue
Logged
Doesn’t have
a key
35
Keys vs.
ideal
User
28
User
29
Alice
Bob
Sue
User
4
User
27
User
5
User
26
20 Faithful Implementations (Green)
User
25
User
6
4 False Accepts (Red)
5 False Rejects (Yellow)
User
7
User
24
Charlie’s
Lab
User
23
User
8
User
22
User
9
User
21
User
10
User
20
User
11
User
19
User
18
User
17
User
16
User
15
User
14
User
13
User
12
Conditions
Ideal
Keys

True (can access anytime)

True (has a key)

Logged


Owner notified
Ask trusted person with
key access

Know location of hidden
key

Ask owner who contacts
witness

False (no access)

?
Owner gives real-time
approval

Owner gives real-time
approval and witness
present

Trusted person gives real
time approval and is
present

False (no access)
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
37
Rules
Key implementation accuracy
Ideal Conditions
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
38
Conditions
Ideal
Grey

True (can access anytime)

True (has a delegation)

Logged


Owner notified
Ask trusted person with
Grey access

Ask owner via Grey

Ask owner who contacts
witness

False (no access)


Owner gives real-time
approval
Owner gives real-time
approval and witness
present

Trusted person gives real
time approval and is
present

False (no access)
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
39
Rules
Implementation accuracy
Ideal Conditions
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
40
Study 2: Contributions

Documented the collection of ideal policy
data

Developed a metric and methodology for
quantitatively comparing accuracy of
implemented policies

Showed that a smarphone access-control
system outperformed keys in overall
security and effectiveness
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
41
Outline
Introduction
Grey Overview & Deployment
Study 1: System Acceptance
Study 2: Policy Creation
Related Work
Conclusion
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
42
Related work

Several Grey-like systems have been
proposed but not implemented
• Digital Key system [Beaufour and Bonnet]
• The Master Key [Zhu, Mutka and Ni]

Access-control tokens are not very easy to
use and those that are tend to be less
secure [Braz and Robert; Piazzalunga et. al.]
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
43
Related work

Usability of access control for file systems
• Manipulating access-control lists is difficult for
users to do accurately [Cao and Iverson]
• Users have difficulty understanding how rules
interact to form the effective policy [Maxion and
Reeder]

Studies of users’ access-control needs
• Identified several different approaches to
access control management [Ferraiolo et al.]
• Users have dynamic access-control needs that
very by task [Whalen et al.]
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
44
Summary

Study 1
• Users have low tolerance for failure and treat Grey like
an appliance

Study 2
• Policies made using Grey were less permissive than
key policies and better matched the ideal policies

Related work
• Unlike previous work we study an actual working
system and examine gathered empirical data
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
45
Future work

Explore the tasks policy authors engage in

Explore the use of a Grey like system in
large organizations

Develop technologies that assist in the
authoring of policies
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
46
CMU Usable Privacy and Security
Laboratory
http://cups.cs.cmu.edu/
Bibliography

X. Cao and L. Iverson. Intentional access management:
Making access control usable for end-users. In
Symposium On Usable Privacy and Security, 2006.

A. Beaufour and P. Bonnet. Personal servers as digital
keys. In 2nd IEEE International Conference of Pervasive
Computing and Communications, 2004.

C. Braz and J. Robert. Security and usability: The case of
the user authentication methods. In IHM ’06, p 199-203,
2006.

D. F. Ferraiolo, D. M. Gilbert and N. Lynch. An
examination of federal and commercial access control
policy needs. In 16th National computer Security
Conference, p 107-116, 1993.
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
48
Bibliography

R. A. Maxion and R. W. Reeder. Improving user-interface
dependability through mitigation of human error. International Journal
of Human-Computer Studies, 63(1-2), 2005.

U. Piazzalunga, P. Salveneschi, and P. Confetti. The usability of
security devices. In L. F. Cranor and S. Garfinkel, editors, Security
and Usability: Designing Secure Systems that People Can Use, p
221-241. O’Reilly, 2005.

T. Whalen, D. Smetters, and E. F. Churchill. User experiences with
sharing and access control. In CHI ’06 extended abstracts on Human
factors in computing systems, p 1517-1522, 2006.

F. Zhu, M. W. Mutka, and L. M. Ni. The master key: A private
authentication approach for pervasive computing environments. In 4th
IEEE Interantional Conference on Pervasive Computering and
Communications, p 212-221, 2006.
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
49
Number of Accesses
Grey accesses per week
Week
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
50