User Studies II
With your instructor, Jeremy Hyland
Plan for Today
• Discuss the reading:
– Why Johnny Can’t Encrypt
– Johnny 2: Judgment Day
• Do a little testing of our own…
Why Johnny Can’t Encrypt
• Who’s Johnny and why can’t he
encrypt?
Posner says
What’s Johnny
trying to hide?
Why Johnny Can’t Encrypt
• Whitten and Tygar, 1999
–
http://www.usenix.org/publications/library/proceedings/sec99/full_papers/whitten/whitten_html/index.ht
ml
• A Usability Evaluation of PGP 5.0
Why Johnny Can’t Encrypt
• “Security mechanisms are only effective when used correctly”
So:
If Usable then
else
Why Johnny Can’t Encrypt
•
Defining Usable Security Software
–
Whitten and Tygar:
Security software is usable if the people who are expected to use it:
1. are reliably made aware of the security tasks they need to
perform.
2. are able to figure out how to successfully perform those
tasks
3. don't make dangerous errors
4. are sufficiently comfortable with the interface to continue
using it.
Why Johnny Can’t Encrypt
•
Why is usable security hard?
You have no usable
security, get over it.
McNealy says
Why Johnny Can’t Encrypt
•
Why is usable security hard?
– Five reasons:
1. The unmotivated users
•
“Security is usually a secondary goal”
2. Policy Abstraction
•
Programmers understand the representation but
normal users have no background knowledge.
Why Johnny Can’t Encrypt
•
Why is usable security hard?
– Five reasons:
3. The lack of feedback
•
We can’t predict every situation.
4. The proverbial “barn door”
•
Need to focus on error prevention.
5. The weakest link
•
Attacker only needs to find one vulnerability
Why Johnny Can’t Encrypt
• Usability Evaluation
– PGP 5.0
•
•
•
•
Pretty Good Privacy
Software for encrypting and signing data
Plug-in provides “easy” use with email clients
Modern GUI, well designed by most standards
Why Johnny Can’t Encrypt
• Usability Evaluation
– Whitten and Tygar focus their evaluation on
a question based off their definition of
usable secure software:
If an average user of email feels the need for privacy and
authentication, and acquires PGP with that purpose in mind, will PGP's
current design allow that person to realize what needs to be done,
figure out how to do it, and avoid dangerous errors, without becoming
so frustrated that he or she decides to give up on using PGP after all?
Loaded question?
Why Johnny Can’t Encrypt
• Usability Evaluation
– Cognitive walk through
• Mentally step through the software as if we
were a new user. Attempt to identify the
usability pitfalls.
• Focus on interface learnablity.
Why Johnny Can’t Encrypt
• Usability Evaluation
– Cognitive walk through results:
Visual metaphors
– Public vs. Private keys
– Signatures and verification
Why Johnny Can’t Encrypt
• Usability Evaluation
– Cognitive walk through results:
Different key types
– Compatibility increases complexity
– Keys listed as users
Why Johnny Can’t Encrypt
Keys listed as users
Why Johnny Can’t Encrypt
• Usability Evaluation
– Cognitive walk through results:
Key server
– Hidden?
– What is it doing?
– Revocation not automatic
Would that help?
Why Johnny Can’t Encrypt
• Usability Evaluation
– Cognitive walk through results:
Key management policy
– Unneeded confusion
– What’s the difference between trust and validity?
Why Johnny Can’t Encrypt
• Usability Evaluation
– Cognitive walk through results:
Irreversible actions
– Need to prevent costly errors
Consistency
– “Encoding”?!?
Too much information
– More unneeded confusion
– Show the basic information, make more advanced information
available only when needed.
Why Johnny Can’t Encrypt
• Usability Evaluation
– User Test
• PGP 5.0 with Eudora
• 12 participants all with at least some college and none
with advanced knowledge of encryption
• Participants were given a scenario with tasks to complete
within 90 min
• Tasks built on each other
• Participants could ask some questions through email
Why Johnny Can’t Encrypt
• Usability Evaluation
– User Test Results:
• 3 users accidentally sent the message in clear text
• 7 users used their public key to encrypt and only 2 of the 7
figured out how to correct the problem
• Only 2 users were able to decrypt without problems
• Only 1 user figured out how to deal with RSA keys correctly.
• A total of 3 users were able to successfully complete the
basic process of sending and receiving encrypted emails.
• One user was not able to encrypt at all
Why Johnny Can’t Encrypt
• Conclusion
If an average user of email feels the need for privacy and
authentication, and acquires PGP with that purpose in mind, will PGP's
current design allow that person to realize what needs to be done,
figure out how to do it, and avoid dangerous errors, without becoming
so frustrated that he or she decides to give up on using PGP after all?
→Nope
– Is this a failure in the design of the PGP 5.0 interface or is it
a function of the problem of traditional usable design vs.
design for usable secure systems?
• Security as the primary function vs. a secondary function
Johnny 2
• Garfinkel and Miller, 2005
–
http://www.simson.net/clips/academic/2005.SOUPS.johnny2.pdf
• Follow-up to Why “Johnny Can’t encrypt”
• Test of new encryption technology
– Key Continuity Management
– S/MIME certificates
• Better interface
– Simple buttons
Johnny 2
• Garfinkel and Miller:
– Johnny couldn’t encrypt because of the key
architecture behind PGP.
“….the fundamental usability barriers that
Whitten identified could be overcome by
replacing the underlying third-party certification
model with Key Continuity Management.”
Johnny 2
• User Test
– Tried to stay as close to the Johnny
experiment as practical
• Same methods of user solicitation/selection
• Same basic scenario
• Similar user tasks
– Added attackers
Johnny 2
• User Test
– Attacks:
• new key attack
• new identity attack
• unsigned message attack
– How well does the interface enable users
to respond to these attacks?
Johnny 2
• User Test
– Test application: CoPilot
• “Wizard of Oz” prototype
• S/MIME certificate handling:
–
–
–
–
–
First time = Yellow
Trusted certificate = Green
Changed certificate = Red
Unsigned message = White
Unsigned message from a sender that normal
sends signed messages = Gray
• Better tools allow for a more automated
and scientific test
Johnny 2
• User Test
– 43 test subjects
– Three groups:
No KCM
Color
Color+Briefing
Johnny 2
• User Test
– Results:
• Users generally understood the basics
• Little understanding of signature integrity guarantees
• Verifying attack message authenticity was difficult for most users
• No group resisted attacks 100% of the time
• Color and Color+Briefing resisted new key attack and the unsigned
message attack better then No KCM
• The interface did not help against new identity attacks
Johnny 2
• User Test: Conclusions
– A few surface interface issues
– Do not trust button
– Misconceptions about the security of sealed messages
– Generally, the new interface simplifies email encryption
– Still problems with determining certificate trust, however
some of these problems may be unavoidable.
So Now What?
• Now its time to do your own test!
User Test
• 3 groups:
– Cell Phone
– CD player
– Calculator
• Take a few minutes to create a simple user
test
• One member of each group switches to be a
tester
User Test
• Guidance:
– Decide whose going to do what!
– Create a Use Case Scenario
– Define user tasks for completion of the
scenario
– Set up metrics for results evaluation
– What qualifies as success vs. failure?
User Test
• Results!?