Do security toolbars actually
prevent phishing attacks?
Min Wu, Robert Miller, Simson Garfinkel
MIT Computer Science and Artificial Intelligence Lab
?
Address bar
Status bar
eBay Account Guard
SpoofStick
Netcraft Toolbar
SpoofGuard
TrustBar
Are security toolbars effective?
• Many implemented (or proposed) antiphishing schemes use a security toolbar
• But no systematic study of the
effectiveness of different toolbars
• User study goals:
– Test security toolbars
– Find out why users are fooled by phishing
3 types of toolbars
SpoofStick
Neutral-information Toolbar
Netcraft Toolbar
eBay Account Guard
System-decision Toolbar
SpoofGuard
TrustBar
SSL-verification Toolbar
Main Frame
Address bar frame
http://tigermail.co.kr/cgi-bin/webscrcmd_login.php
Toolbar frame
Toolbar frame
Status bar frame
Study protocol
• Users should be given real tasks, which
contain security-related subtasks.
• In order to be realistic:
– User should care about security in the
study.
– But we cannot use their own personal data.
– User should not put security as the main
goal.
Study protocol
• We set up dummy accounts as John
Smith at various websites
• “You are the personal assistant of
John Smith. John is on vacation now.
During his vacation, he sometimes
sends you emails asking you to do
some tasks for him online.”
• “Here is John Smith’s profile.”
Study protocol
• Users deal with 20 emails forwarded by
John Smith.
• Most of the emails are about managing
John’s wish lists at various sites
Phishing attacks
• 5 out of the 20 emails are phishing
emails
1. Similar name attack
Bestbuy.com  www.bestbuy.com.ww2.us
2. IP attack
Bestbuy.com  212.85.153.6
3. Hijacked-server attack
Bestbuy.com  www.btinternet.com
Popup-window attack on
hollywoodvideo.com
Greedy attack on paypal.com
Tutorial email
• When and how to give users tutorial about the
tested toolbar?
• Pilot test with 11 users (with another scenario)
– First 5 users were presented with a printout of the
tutorial before the study
• Success attack rate: 1 out of 15
• The printout explicitly told users where to check for phishing
– Second 6 users did not see the printout but the
toolbar has “What’s this?” link to the tutorial
• Success attack rate: 17 out of 18
• Everyone claimed to notice the toolbar but none clicked
“What’s this?”. So users did not know that the toolbar is used
for detecting phishing
Attack pattern
1-9
10
Paypal attack
11
Tutorial email
12-20
Results
• 30 users
– Recruited at MIT, paid $15 for one hour
– 10 for each toolbar
Neutral-info Toolbar
System-decision Toolbar
SSL-verification Toolbar
– Average age 27 [18-50]
– 14 females, 16 males
– 20 MIT students, 10 not
Users cared about security
• 18 out of 30 unchecked “remember me”.
www.deepdiscountdvd.com
www.travelocity.com
• 13 out of 30 logged out (or tried to) after at
least one task
Users cared about security
1
6
"Remember Me" or logged out
Reported fraud
Showed suspicion
23
Phishing spoof rates
1
0.9
0.8
0.7
0.6
0.54
system-decision
neutral-information
ssl-verification
0.5
0.4
0.39
0.4
0.33
0.31
0.33
0.32
0.3
0.3
0.19
0.2
0.1
0
before tutorial
after tutorial
total
Spoof rates between familiar sites
vs. unfamiliar sites
1
0.9
0.8
0.7
0.6
0.5
0.4
0.32
0.31
unfamiliar sites
familiar sites
0.3
0.2
0.1
0
Why did users get fooled?
• Users explained away spoofing URLs:
– www.ssl-yahoo.com: “ssl-yahoo.com is a subdirectory
of Yahoo, like mail.yahoo.com”
– www.walmart-onlineservice.global-update2.com:
“legitimate sites may direct you to another site, for
example, amazon.com to amazon.co.uk.”
– sign.travelocity.com.zaga-zaga.us: “it may be an
outsourcing site.”
– www.mytargets.com: “sometimes the company has to
register a different name from its brand, for example,
what if target.com has already been taken by another
company.”
– www.btinternet.com attacks buy.com: “sometimes I go
to a website and the site directs me to another
address which is different from the one I typed in.”
– 200.114.156.78 attacks kmart.com: “I have been to
sites with IP addresses.”
Why did users get fooled?
• Users explained away toolbar
messages:
– Potential fraudulent site: “it is triggered
because the web content is ‘informal’, just
like my spam filter says ‘this email is
probably a spam.’”
– New Site
[BR]: “Yahoo may have a
branch in Brazil and thus registered there.”
Why did users get fooled?
• Users explained away spoofing
behavior from the fraudulent site:
– “The popup window in
www.hollywoodvideo.com was mistakenly
triggered by me. I clicked ‘Register for new
account’ by mistake, instead of ‘Sign in for
existing account’.”
Why did users get fooled?
• Users had wrong expectations about
fraudulent sites:
– “I think that it is good for John to clear his wish list
from this fraudulent site.” (But she had to log in
first)
– “If the site looks suspicious, I will call their
customer service using the number listed in the
page.”
– “If a site works well with all its links, then the site
is authentic. I cannot imagine that an attacker will
make the attack so elegantly to mirror a whole
site.”
– “The site is authentic because it has a privacy
policy, VeriSign seal, contact information, and the
submit button says ‘sign in using our secure
server’.”
Why did users get fooled?
• Users did not take the toolbar message
seriously
– “This site is maybe fake but since I am
only dealing with the wish list, I have to
take the risk to get the task done.”
Fooled by Paypal attacks
• 4 out of 25 (16%) users who had been to
Paypal before were fooled by the Paypal
attack
– “I used Paypal before and the site looks exactly
the same. If I trust the site from my experience I
am not suspicious.”
– “Repetitive tasks made me lose focus and lower
my judgment.”
– “They [Paypal] need this information [credit card
and bank account] in order to charge me”
False positive errors
• Users reported fraud at good sites 3.3% of the time
(13/390)
– 4 because of web content
• The contact information of www.deepdiscountdvd.com only has PO
Box instead of street number
• The Verisign seal at www.hsn.com is not clickable
• The webpage (www.deepdiscountdvd.com) is less cluttered and has
less popups compared to other video sites.
– 1 because of suspicious url: https://www-ssl.bestbuy.com/...
– 3 because of browser’s warning
– 5 because of the warning of the SSL-verification toolbar
Conclusion
• Browser toolbars are not effective
enough to prevent phishing attacks.
– If the website’s appearance, which is totally
controlled by the attacker, looks good or
not risky, many users (≥19%) do not take
seriously or believe the message displayed
by the toolbars about phishing attacks.
– Badly executed e-commerce sites lead
users to explain away weird behavior from
phishing sites
Thank you!
• Q/A
Popup window
• Spoofguard, eBay account guard, netcraft
toolbar use popup window as well
– When the toolbar THINK that the site/page is
dangerous
• Heuristic rules
• Blacklist
– Display negative information
– Pros:
• Always noticeable
• Larger area for detailed explanation
– Cons:
• Let security issue interrupt user’s main task
• What if the toolbar thinks wrong
– False positive: users won’t believe it and the uninstall it
– False negative: false sensitive of security
Hypotheses
• Properties of the tested security
indicators
– Using peripheral area
– Talking about security
– Limited space for display
• Ineffectiveness because users
– Do not notice them
– Do not care them
– Do not understand or believe them
Address bar frame
Toolbar frame
Title frame
Toolbar config:
co.kr: New Site, KR
Main frame to
https://www.paypal.com/cgi-bin/
webscrcmd_login.php
Status bar frame
Phishing config:
paypal.com  tigermail.co.kr
https  http
Fake browser
Real sites