Privacy Notice and Choice
in Practice
Thesis Defense
Pedro Giovanni Leon
Department of Engineering and Public Policy
Carnegie Mellon University
August 28, 2014
Committee:
Lorrie Faith Cranor, EPP & SCS, CMU
Alessandro Acquisti, Heinz College, CMU
Jon M. Peha, EPP & ECE, CMU
Joel Reidenberg, School of Law, Fordham University
Engineering &
Public Policy
Today’s Privacy Challenges
• Extensive data collection
–
–
–
–
–
Online tracking
Smartphone / Smart home / Smart store / Smart …
Social media
Web transactions
Credit card transactions
• Data aggregation and linkage from different data
streams
• Extensive sharing
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
2
Source: BlueKai
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
3
Source: BlueKai
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
4
Privacy Enables Fundamental Values
• Freedom of speech
• Anonymity
• Innovation
• Reputation
• Autonomy
• Equality
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
5
Transparency-based Regulations
Notice
Company’s
response
Company’s
assessment
Users’
perceptions
and
assessment
Users’
choices
Source: Fung A., Graham M, and Weil D. “Full Disclosure: The Perils and Promise of Transparency.” 2007
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
6
Vehicle’s 5-Star Safety Ratings
Source: http://www.safercar.gov/
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
7
Restaurant’s Hygiene
Source: http://www.nyc.gov/html/doh/html/services/restaurant-inspection.shtml
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
8
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
9
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
10
Regulators Ask for Better Privacy Protections
White House – Feb. 2012
FTC – March 2012
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
11
Current Protections Strongly Rely on Notice
and Choice
Notice
• Consumers should be given notice of an entity’s
information practices
• Before any personal information is collected from
them
• Notice required for informed decisions as to
whether and to what extent to disclose personal
information
Choice
• Giving consumers options as to how any personal
information collected from them may be used
Source: http://www.ftc.gov/reports/privacy3/fairinfo.shtm
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
12
Thesis Goals
[Need to work on content]
• To investigate the effectiveness of today’s most
widely deployed privacy protection mechanisms
based on notice and choice
• To evaluate new notice and choice alternatives
• To propose both interface design improvements
and policy recommendations
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
13
Thesis Scope and Approach
Scope
• [Need to work on content]
Approach
• Laboratory and field studies, usability testing, semistructured interviews, and online surveys
• Evaluation of companies’ privacy practices and
uses of privacy notice and choice mechanisms
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
14
Thesis Outline
Chapter 1: Introduction
Chapter 2: The Misrepresentation of Website Privacy Policies Through the
Misuse
of P3P Compact Policy Tokens
Chapter 3: A Large-Scale Evaluation of U.S. Financial Institutions Privacy
Notices
Chapter 4: An In-depth Analysis of Online Advertising Companies’ Privacy
Chapter
Policies 5: Perceptions of Online Behavioral Advertising
Chapter 6: A Usability Evaluation of Tools to Limit Online Behavioral Advertising
Chapter 7: What Do Online Behavioral Advertising Privacy Disclosures
Communicate to Users?
Chapter 8: Factors That Affect Users’ Willingness to Share Information with
Online
Advertisers
Chapter 9: A Field Trial of Privacy Nudges for Facebook
Chapter 10: Public Policy Implications
Chapter 11: Conclusions
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
15
Thesis Outline
Chapter 1: Introduction
Chapter 2: The Misrepresentation of Website Privacy Policies Trough the
Misuse
of P3P Compact Policy Tokens
Chapter 3: A Large-Scale Evaluation of U.S. Financial Institutions Privacy
Notices
Chapter 4: An In-depth Analysis of Online Advertising Companies’ Privacy
Chapter
Policies 5: Perceptions of Online Behavioral Advertising
Chapter 6: A Usability Evaluation of Tools to Limit Online Behavioral Advertising
Chapter 7: What Do Online Behavioral Advertising Privacy Disclosures
Communicate to Users?
Chapter 8: Factors That Affect Users’ Willingness to Share Information
with Online
Advertisers
Chapter 9: A Field Trial of Privacy Nudges for Facebook
Chapter 10: Public Policy Implications
Chapter 11: Conclusions
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
16
Scope of this presentation
A Large-Scale Evaluation of U.S. Financial Institutions
Privacy Notices
[To be submitted to the Journal of Legal Studies 2014, WEIS 2013]
What Do Online Behavioral Advertising Privacy
Disclosures Communicate to Users?
[WPES 2012]
Factors that Affect Users’ Willingness to Share
Information with Online Advertisers
[SOUPS 2013, TPRC 2014]
Public Policy Implications
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
17
A Large-Scale Evaluation of U.S. Financial
Institutions Privacy Notices
[To be submitted to the Journal of Legal Studies 2014, WEIS 2013]
What Do Online Behavioral Advertising Privacy
Disclosures Communicate to Users?
[WPES 2012]
Factors that Affect Users’ Willingness to Share
Information with Online Advertisers
[SOUPS 2013, TPRC 2014]
Public Policy Implications
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
18
Research Questions
• What is the impact of government regulation
on US financial institutions’ privacy practices?
• What is the spectrum of choices that customers
have with respect to sharing practices of their
personal information?
• What are the internal and external factors that
impact US financial institutions information
sharing practices?
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
19
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
20
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
21
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
22
Methodology
• Public list of 19K+ FDIC-Insured financial
institutions
• Crawled the Internet and retrieved 6K+ notices
• Parsed notices to extract information
• Merged extracted info with companies’
additional info (size, geographical location,
regulator, etc.)
• Built binary logistic regression model to predict
sharing practices
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
23
Important Differences in Practices
• Need to add a graph here
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
24
Company’s Size And Location Affect
Sharing Practices
• Need to add a content showing the regression
coefficients
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
25
Results
• Substantial differences between sharing
practices
• Minimum compliance
• A few instances of non-compliance
• Larger institutions are more likely to share
consumers’ personal information
• Geographical location impacts sharing practices
• Weaknesses in the regulation lead to reduced
transparency of collection practices
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
26
Conclusions
• TBD
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
27
A Large-Scale Evaluation of U.S. Financial Institutions
Privacy Notices
[To be submitted to the Journal of Legal Studies 2014, WEIS 2013]
What Do Online Behavioral Advertising Privacy
Disclosures Communicate to Users?
[WPES 2012]
Factors that Affect Users’ Willingness to Share
Information with Online Advertisers
[SOUPS 2013, TPRC 2014]
Public Policy Implications
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
28
Ads Based on Online Behavior
www.webmd.com
User
www.webmd.com
Search: “Scalp conditions”
Ad Network
www.expedia.com
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
29
Study Goal
Do OBA disclosures empower users to make
privacy choices?
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
30
Methodology
• Between-subjects online study
• Recruitment: Amazon Mechanical Turk
• Random assignment to:
•
•
•
•
1 of 2 priming conditions
1 of 2 icons
1 of 7 taglines
1 of 5 landing pages
• Browsing scenario with online survey
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
31
Two Priming Conditions
Imagine that you are…
1) Planning your next vacation to Paris…
2) Planning to buy a new car…
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
32
Two Icons
Advertising option icon
Asterisk man icon
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
33
Seven Taglines
 “AdChoices”
 “Interest based ads”
 “Why did I get this ad?”
 “Learn about your ad choices”
 “Configure ad preferences”
 “Sponsor ads”
 Blank (no tagline)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
34
Five Landing Pages
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
35
Demographics
• 1,505 participants
• Mean age = 32, SD = 11.5
• 59% female
• > 200 participants per tagline treatment
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
36
First Exposure to OBA Disclosures
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
37
Recalling Ads and OBA Disclosures
• More likely to remember specific ads (49.3%)
than icons (27.6%). p<0.001
• Tagline recall rate was only 11.9%
• “Why did I get this ad?” recalled more often
(22.3%) than others. p<0.05
• “AdChoices” (7.9%) and “Sponsor ads” (7%)
recalled at about the same rate
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
38
OBA Disclosures in Isolation
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
39
To what extent, if any, does this combination of the
symbol and phrase [icon+tagline placed here], placed
on the top right corner of the above ad suggest the
following?
“This ad has been tailored based on websites
you have visited in the past”





Definitely not
Probably not
Not sure
Probably
Definitely
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
40
This ad has been tailored based on websites you have visited
in the past. TRUE
•
“Why did I get this ad?” correct at a higher rate (p < 0.05)
•
No statistically significant difference between icons
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
41
What do you think will happen if you click on that symbol or
that phrase?
• It will take you to a page where you can tell the
advertising company that you do not want to
receive tailored ads. TRUE (27% agreement)
• It will take you to a page where you can buy
advertisements on this website. FALSE (30%)
• You will let the advertising company know that you
are interested in those products. FALSE (51%)
• More ads will pop up. FALSE (53%)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
42
It will take you to a page where you can tell the
advertising company that you do not want to receive
tailored ads. TRUE
“Configure ad preferences” correct at a higher rate (p < 0.01)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
43
It will take you to a page where you can buy
advertisements on this website. FALSE
“Configure ad preferences” and “Why did I get this ad?” correct at a higher
rate (p < 0.0005)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
44
More ads will pop up. FALSE

Advertising option icon more likely to be incorrect than asterisk man icon (p = 0.003
)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
45
Notice Provided by Landing Pages
To what extent, if at all, does the information on the
‘landing page’ suggest to you that…
• The ads you see in the news website are based on
your visits to this news website and other
websites. TRUE (77% agreement)
• This news website protects your privacy by not
sharing your information. FALSE (24% agreement)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
46
Indicate your agreement with the following
statements defining what ‘opt out’ means in the
context of Internet advertising:
• Stop advertising companies from collecting
information about your browsing activities. FALSE
(63% agreement)
• Stop seeing ads based on your browsing activities.
TRUE (80% agreement)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
47
Conclusions
• OBA disclosures are not noticed
• “AdChoices” one of the worst taglines
• Users are unlikely to click on disclosures
• Landing pages provide proper notice
• Users misunderstood the meaning of opting out
• Current OBA disclosures are falling short
• Plenty of opportunities for improvement, but
• User education is needed
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
48
A Large-Scale Evaluation of U.S. Financial Institutions
Privacy Notices
[To be submitted to the Journal of Legal Studies 2014, WEIS 2013]
What Do Online Behavioral Advertising Privacy
Disclosures Communicate to Users?
[WPES 2012]
Factors that Affect Users’ Willingness to Share
Information with Online Advertisers
[SOUPS 2013, TPRC 2014]
Public Policy Implications
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
49
Benefits
Concerns
• Relevant ads
• What info is collected?
• Find things of interest
• How collected
information is used?
• Personalized Internet
experience
• Inaccurate
assumptions
• Discounts
• Lack of transparency
and control
User
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
50
Users often make decisions without
having relevant information or
under the wrong assumptions
What actually matters to users?
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
51
What Matters to Users?
• Type of data collected?
• Extent of data sharing?
• Retention period?
• Familiarity with the visited website?
• Control mechanisms?
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
52
Methodology
• Between-subjects Mturk study [N = 2,912]
– 15 scenarios representing different advertising-related
data practices
• Simulated web-browsing scenario
• Explained value proposition of OBA
• Collected willingness to share 30 types of
information
• Collected sharing preferences under different
hypothetical controls
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
53
Treatments
• Website familiarity
–
or
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
54
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
55
Treatments
• Website familiarity
–
or
• Scope of Sharing
– Only WebMD / WebDR
– WebMD / WebDR and other visited sites
– WebMD / WebDR and Facebook
• Retention Period
– One day or Indefinitely
• Access to view and edit collected information
– Access or No access
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
56
Analysis
• Dependent variables: 30 types of personal
information
• Independent variables: Different data usage
scenarios
• Performed factor analysis to group types of
personal information into categories
• Built MANOVA regression model to investigate
factors affecting disclosure preferences
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
57
Willingness to disclose information
60%
50%
40%
30%
20%
10%
0%
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
58
Willingness to disclose information
60%
50%
40%
30%
•
•
•
•
•
Country
Web browser
Gender
Operating system
State
20%
10%
0%
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
59
Willingness to disclose information
60%
50%
40%
30%
20%
10%
•
•
•
•
•
Credit score
Exact current location
Phone number
Address
Credit card #
0%
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
60
Categories of Data Types
Category
Data Types
Browsing
Pages visited, search terms, time spent on
pages, + 2 more
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
61
Categories of Data Types
Category
Data Types
Browsing
Pages visited, search terms, time spent on
pages, + 2 more
Computer
Web browser and OS
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
62
Categories of Data Types
Category
Data Types
Browsing
Pages visited, search terms, time spent on
pages, + 2 more
Computer
Web browser and OS
Location
Country, State, Town/City, and ZIP code
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
63
Categories of Data Types
Category
Data Types
Browsing
Pages visited, search terms, time spent on
pages, + 2 more
Computer
Web browser and OS
Location
Country, State, Town/City, and ZIP code
Demographic
Gender, political views, income bracket, religion,
sexual orientation, + 4 more
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
64
Categories of Data Types
Category
Data Types
Browsing
Pages visited, search terms, time spent on
pages, + 2 more
Computer
Web browser and OS
Location
Country, State, Town/City, and ZIP code
Demographic
Gender, political views, income bracket, religion,
sexual orientation, + 4 more
PII
Email and name
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
65
What Factors Matter?
Data
Category
Scope of
Sharing
Retention
Period
Browsing
✔
✔
✔
✔
✔
Access
Site
familiarity
Computer
Location
Demographic
PII
✔
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
66
Example: Browsing (Pages visited)
80%
70%
60%
50%
40%
Indefinitely
One day
30%
20%
10%
0%
WebMD + FB
WebMD + Other
Visited
Scope of Sharing
Only WebMD
* Scope and retention statistically significant treatments (p-value < 0.001)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
67
Example: Browsing (Pages visited)
80%
70%
60%
50%
+15%
+11%
40%
Indefinitely
One day
30%
20%
10%
0%
WebMD + FB
WebMD + Other
Visited
Scope of Sharing
Only WebMD
* Scope and retention statistically significant treatments (p-value < 0.001)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
68
Example: Browsing (Pages visited)
80%
70%
60%
50%
40%
+25%
Indefinitely
One day
30%
20%
10%
0%
WebMD + FB
WebMD + Other
Visited
Scope of Sharing
Only WebMD
* Scope and retention statistically significant treatments (p-value < 0.001)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
69
Example: Browsing (Pages visited)
80%
70%
+15%
60%
50%
+25%
40%
Indefinitely
One day
30%
20%
10%
0%
WebMD + FB
WebMD + Other
Visited
Scope of Sharing
Only WebMD
* Scope and retention statistically significant treatments (p-value < 0.001)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
70
Example: PII (name)
30%
25%
20%
15%
+12%
Indefinitely
One day
10%
5%
0%
WebMD + FB WebMD + Other Only WebMD
Visited
Scope of Sharing
* Scope statistically significant treatment (p-value < 0.001)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
71
Limitations
• Survey data can differ from behavioral data
• Mturk recruitment biases
• Limited scope (a health related site)
• The purpose of collection was limited to targeted
ads
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
72
Conclusions
• Notice and choice tools should allow users to
allow or block trackers on the basis of factors
users find important (e.g., retention and scope)
• Ad companies should be transparent about their
data practices
• Standardized (and machine-readable) notices
would allow for automation
• Research on user preferences can assist the
design of usable interfaces and policy
frameworks
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
73
A Large-Scale Evaluation of U.S. Financial Institutions
Privacy Notices
[To be submitted to the Journal of Legal Studies 2014, WEIS 2013]
What Do Online Behavioral Advertising Privacy
Disclosures Communicate to Users?
[WPES 2012]
Factors that Affect Users’ Willingness to Share
Information with Online Advertisers
[SOUPS 2013, TPRC 2014]
Public Policy Implications
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
74
Better Notices
• Understand users’ expectations
• Understand what information is relevant for
users
• Standard models similar to “Financial Notices”
can help to facilitate comprehension and
comparison
• Usability testing is needed
• Are worthless without meaningful choices
• Are not sufficient, additional protections are
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
75
Standardization
• Facilitate automatic checking of compliance with
notice requirements
• Facilitate automatic implementation of policies
• Facilitate use of privacy agents
• Facilitate comparison of notices (creating
incentives to do better)
• Standardization is also needed to enforce and
automate information flow policies (e.g., sticky
policy)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
76
Incentives
• Government regulation is necessary to achieve
accountability and enforcement
• Reduce information asymmetries
– Enhance transparency mechanisms
• Guarantee real alternatives
• Data minimization and limited retention
• Limit and control secondary uses
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
77
Regulation requirements may include
–
–
–
–
–
–
Establishment of privacy program
Establishment of Internal Review Boards
Ongoing risk assessment (PIA, Audits, etc.)
Employee education and training
Limits and constraints for data retention and sharing
Usable transparency and control mechanisms
(awareness, standard notices, redress procedures)
– Real sanctions for violations
– Establish first parties responsibility for the tracking
that happens on their websites
– Alternative options requirements (offer free of-tracking
paid services)
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
78
Acknowledgements
Thank you to:
My advisor: Lorrie Faith Cranor
My committee members: Alessandro Acquisti, Jon M. Peha, Joel
Reidenberg
Co-authors and CUPS Lab Members: Idris Adjerid, Lujo Bauer,
Rebecca Balebako, Laura Brandimarte, Cristian Bravo Lillo, Justin
Cranshaw, Mihai Christodorescu Jim Graves, Alain Forget, Manoj
Hastak, Kelly Idouchi, Patrick Kelley, Saranga Komanduri, Abigail Marsh,
Robert McGuire, Aleecia McDonald, Eyal Peer, Norman Sadeh, Rich
Shay, Sonam Smat, Florian Schaub, Many Sleeper, Blase Ur, Yang
Wang, Guzi Xu
My family: Alejandra Penilla, Josué León, Mario León. Maria Antonia
Nájera, Bella León, and Yonathan León
Institutions: Carnegie Mellon University, Mexican Council of Science and
CyLab Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
Technology, the Central
Bank of Mexico, (other funding agencies)
79