Understanding the
Human in the Loop
January 16, 2008
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
1
Humans
“Humans are incapable of securely storing highquality cryptographic keys, and they have
unacceptable speed and accuracy when
performing cryptographic operations. (They are
also large, expensive to maintain, difficult to
manage, and they pollute the environment. It is
astonishing that these devices continue to be
manufactured and deployed. But they are
sufficiently pervasive that we must design our
protocols around their limitations.)”
-- C. Kaufman, R. Perlman, and M. Speciner.
Network Security: PRIVATE Communication in a PUBLIC World.
2nd edition. Prentice Hall, page 237, 2002.
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
2
Humans are weakest link
Most security breaches attributed to
“human error”
Social engineering attacks proliferate
Frequent security policy compliance
failures
Automated systems are generally more
predictable and accurate than humans
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
3
Why are humans in the loop at all?
Don’t know how or too expensive to
automate
Human judgments or policy decisions
needed
Need to authenticate humans
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
4
The human threat
Malicious humans who will attack system
Humans who don’t know when or how to
perform security-critical tasks
Humans who are unmotivated to perform
security-critical tasks properly or comply
with policies
Humans who are incapable of making
sound security decisions
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
5
Need to better understand humans
Do they know they are supposed to be
doing something?
Do they understand what they are
supposed to do?
Do they know how to do it?
Are they motivated to do it?
Are they capable of doing it?
Will they actually do it?
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
6
Proposed framework
 Cranor interactions article: What do they
"indicate?": evaluating security and privacy
indicators
 The Handbook of Warnings, edited by Michael S.
Wogalter
• Wogalter’s Communication-Human Information
Processing (C-HIP) Model
QuickTime™ and a
 Applied C-HIP to security indicators evaluation from
TIFF (Uncompressed) decompressor
are needed to see this picture.
interactions article
 Expanded it to model other types of human
interaction with secure systems
 Developed “Human in the loop security framework”
and “Human threat identification and mitigation
process” - paper under review
 Need validation and more work on mitigation and
how to operationalize process
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
7
C-HIP Model
 CommunicationHuman Information
Processing (C-HIP)
Model
• Wogalter, M. 2006.
CommunicationHuman Information
Processing (C-HIP)
Model. In Wogalter, M.,
ed., Handbook of
Warnings. Lawrence
Erlbaum Associates,
Mahwah, NJ, 51-61.
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
8
Human in the loop security framework
Communication
Impediments
Environmental
Stimuli
Knowledge
and
Experience
Communication
Interference
Intentions
Attitudes
and Beliefs
Motivation
Capabilities
Communication
Processing
Demographics
and Personal
Characteristics
Application
Personal
Variables
Communication
Delivery
Human Receiver
Attention Switch
Attention
Maintenance
Comprehension
Behavior
Knowledge
Acquisition
Knowledge
Retention
Knowledge
Transfer
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
9
Communication processing model
Framework is based on communication
processing model
• Many models in the literature
• Used to model all sorts of different types of
communications: individual, group, media, etc.
Most end-user security actions are
triggered by some form of communication
• Pop-up alert, email, manual, etc.
Expert self-discovery of a security process
can be modeled as communication to
oneself
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
10
Communication
Communication
Impediments
Environmental
Stimuli
Knowledge
and
Experience
Communication
Interference
Intentions
Attitudes
and Beliefs
Motivation
Capabilities
Communication
Processing
Demographics
and Personal
Characteristics
Application
Personal
Variables
Communication
Delivery
Human Receiver
Attention Switch
Attention
Maintenance
Comprehension
Behavior
Knowledge
Acquisition
Knowledge
Retention
Knowledge
Transfer
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
11
Types of security communications
 Warnings
• Alert users to take immediate action to avoid hazard
 Notices
• Inform users about characteristics of entity or object
 Status indicators
• Inform users about system status information
 Training
• Teach users about threat and how to respond
 Policy
• Inform users about policies
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
12
Active versus passive communications
Active
Firefox
Anti-Phishing
Warning
Passive
Indicators
with audio
alerts
Bluetooth
indicator in
Mac menu bar
Indicators
with
animation
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
13
Communication impediments
Communication
Impediments
Environmental
Stimuli
Knowledge
and
Experience
Communication
Interference
Intentions
Attitudes
and Beliefs
Motivation
Capabilities
Communication
Processing
Demographics
and Personal
Characteristics
Application
Personal
Variables
Communication
Delivery
Human Receiver
Attention Switch
Attention
Maintenance
Comprehension
Behavior
Knowledge
Acquisition
Knowledge
Retention
Knowledge
Transfer
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
14
Environmental stimuli
Divert user’s attention
Greatest impact on passive communication
Examples
• Other communications
• Ambient light and noise
• User’s primary task
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
15
Interference
 Anything that may prevent a communication from
being received as the sender intended
 Caused by
• Malicious attackers
• Technology failures
• Environmental stimuli that obscure the communication
 Focus of traditional secure systems analysis
• How can attacker interfere with communications?
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
16
Human receiver
“The human in the loop”
Communication
Impediments
Environmental
Stimuli
Knowledge
and
Experience
Communication
Interference
Intentions
Attitudes
and Beliefs
Motivation
Capabilities
Communication
Processing
Demographics
and Personal
Characteristics
Application
Personal
Variables
Communication
Delivery
Human Receiver
Attention Switch
Attention
Maintenance
Comprehension
Behavior
Knowledge
Acquisition
Knowledge
Retention
Knowledge
Transfer
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
17
Communication delivery
 Attention switch
• Noticing communication
 Attention maintenance
• Paying attention long enough to process
communication
 Breakdowns
• Environmental stimuli, interference
• Characteristics of communication
• Habituation
 Tendency for the impact of stimuli to decrease over time
 Just because the communication appeared on
the user’s screen, doesn’t mean the user actually
saw it
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
18
Communication processing
Comprehension
• Ability to understand communication
Knowledge acquisition
• User’s ability to learn what to do in response
Breakdowns
• Unfamiliar symbols, vocabulary, complex
sentences, conceptual complexity
Even if a user understands the
communication, they still may not know
what they are supposed to do
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
19
Application
Knowledge retention
• Ability to remember communication
Knowledge transfer
• Ability to recognize situations where the
communication is applicable and figure out
how to apply it
Some security communications are always
applied immediately (for example, pop-up
warnings) so retention and transfer may
not be necessary
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
20
Personal variables
Demographics and personal characteristics
• Age, gender, culture, education, occupation,
disabilities
Knowledge and experience
• Education, occupation, prior experience
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
21
Intentions
 Attitudes and beliefs
• Beliefs about communication accuracy
• Beliefs about whether they should pay attention
• Self-efficacy - whether they believe they can complete
actions effectively
• Response-efficacy - whether they believe the actions
they take will be effective
• How long it will take
• General attitudes - trust, annoyance, etc.
 Motivation
• Incentives, disincentives
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
22
Capabilities
User’s level of ability
• Cognitive or physical skills
• Availability of necessary software or devices
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
23
Behavior
Communication
Impediments
Environmental
Stimuli
Knowledge
and
Experience
Communication
Interference
Intentions
Attitudes
and Beliefs
Motivation
Capabilities
Communication
Processing
Demographics
and Personal
Characteristics
Application
Personal
Variables
Communication
Delivery
Human Receiver
Attention Switch
Attention
Maintenance
Comprehension
Behavior
Knowledge
Acquisition
Knowledge
Retention
Knowledge
Transfer
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
24
Behavior
Users may complete recommended action,
but do so in a way that follows a predictable
pattern that can be exploited by attackers
• Example: password choice
Users may intend to comply, but may fail to
complete necessary action
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
25
Gulfs
Don Norman. The Design of Every Day
Things.1988.
 Gulf of Execution
• Gap between a person’s intentions to carry out an
action and the mechanisms provided by a system to
facilitate that action
 “I can’t figure out how to make it do what I want it to do”
 Gulf of Evaluation
• When a user completes an action but is unable to
interpret the results to determine whether it was
successful
 “I can’t figure out whether it worked”
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
26
Generic Error-Modeling System
James Reason. Human Error. 1990.
Mistakes
• When people formulate action plans that will
not achieve the desired goal
Lapses
• When people formulate suitable action plans,
but forget to perform a planned action (for
example, skipping a step)
Slips
• When people perform actions incorrectly (for
example, press the wrong button)
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
27
Human threat identification and
mitigation process
Task
Identification
Task
Automation
Failure
Identification
Failure
Mitigation
Human-in-the-loop
Framework
User Studies
User Studies
 Task identification
• Identify all points where the system relies on humans to perform securitycritical functions
 Task automation
• Find ways to partially or fully automate some of these tasks
 Failure identification
• Identify potential failure modes for remaining tasks
 Failure mitigation
• Find ways to prevent these failures
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
28
Why don’t users follow password
policies?
Typical password policy
Pick a hard to guess password
Don’t use it anywhere else
Change it often
Don’t write it down
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
30
Typical password practice
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
31
Why don’t users follow password policies?
Task
Identification
Task
Automation
Failure
Identification
Failure
Mitigation
Human-in-the-loop
Framework
User Studies
User Studies
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
32
Why don’t users follow password policies?
Communication
Impediments
Environmental
Stimuli
Knowledge
and
Experience
Communication
Interference
Intentions
Attitudes
and Beliefs
Motivation
Capabilities
Communication
Processing
Demographics
and Personal
Characteristics
Application
Personal
Variables
Communication
Delivery
Human Receiver
Attention Switch
Attention
Maintenance
Comprehension
Behavior
Knowledge
Acquisition
Knowledge
Retention
Knowledge
Transfer
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
33
Why don’t user’s heed browser
security warnings?
Do users notice them?
“What lock icon?”
• Few users notice lock icon in browser chrome,
https, etc.
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
35
Do users know what they mean?
Web browser lock icon:
• “I think that it means secured, it symbolizes
some kind of security, somehow.”
Web browser security pop-up:
• “Yeah, like the certificate has expired. I don’t
actually know what that means.”
J. Downs, M. Holbrook, and L. Cranor. Decision Strategies and Susceptibility to
Phishing. In Proceedings of the 2006 Symposium On Usable Privacy and
Security, 12-14 July 2006, Pittsburgh, PA.
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
36
Do they do what they advise?
“I would probably experience some brief,
vague sense of unease and close the box
and go about my business.”
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
37
Why don’t users heed browser security warnings?
Task
Identification
Task
Automation
Failure
Identification
Failure
Mitigation
Human-in-the-loop
Framework
User Studies
User Studies
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
38
Why don’t users heed browser security warnings?
Communication
Impediments
Environmental
Stimuli
Knowledge
and
Experience
Communication
Interference
Intentions
Attitudes
and Beliefs
Motivation
Capabilities
Communication
Processing
Demographics
and Personal
Characteristics
Application
Personal
Variables
Communication
Delivery
Human Receiver
Attention Switch
Attention
Maintenance
Comprehension
Behavior
Knowledge
Acquisition
Knowledge
Retention
Knowledge
Transfer
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/
39