18th ACM Conference on Computer and Communications Security
(CCS 2011)
Cloak and Dagger: Dynamics of Web
Search Cloaking
David Y. Wang, Stefan Savage, and Geoffrey
M. Voelker
University of California, San Diego
左昌國
Seminar @ ADLab, NCU-CSIE
2
Outline
• Introduction
• Methodology
• Results
• Related Work
• Conclusion
3
Introduction
• Search Engine Optimization (SEO)
• “Search engine optimization (SEO) is the process of improving the
visibility of a website or a web page in search engines via the
"natural" or un-paid ("organic" or "algorithmic") search results.” --Wikipedia
• SEO could be used as benign techniques
• Cloaking
• Up to 1999
• One of the notorious blackhat SEO skills
• Delivering different content to different user segments
• ie. Search engine crawlers and normal users
4
Introduction
Normal User
Search Engine Crawler
5
Introduction
• Types of cloaking
• Repeat Cloaking
• Cookies or IP tracking
• User Agent Cloaking
• User-Agent field in the HTTP request header
• Referrer Cloaking
• Referer field in the HTTP header
• IP Cloaking
6
Introduction
• This paper…
• Designs a system, Dagger, to identify cloaking in near real-time
• Uses this system to
• Provide a picture of cloaking activity as seen through three search
engines(Google, Bing and Yahoo)
• Characterize the differences in cloaking behavior between
undifferentiated “trending” keywords and targeted keywords.
• Characterize the dynamic behavior of cloaking activity
7
Methodology
• Dagger consists of five functional components
• Collecting search terms
• Fetching search results from search engines
• Crawling the pages linked from the search results
• Analyzing the pages crawled
• Repeating measurements over time
8
Methodology
• Collecting Search Terms
• Collecting popular search terms from
• Google Hot Searches
• Alexa
• Twitter
• Constructing another source of search terms using keyword
suggestions from “Google Suggest.”
• ex: User enter -> viagra 50mg
Suggestion -> viagra 50mg cost
viagra 50mg canada
…
9
Methodology
• Querying Search Results
• Submitting the search terms to search engines(Google, Yahoo, and
Bing)
• Google Hot Searches and Alexa each supply 80 terms per 4-hour
• Twitter supplies 40
• Together with 240 additional suggestions based on Google Hot
Searches (80 * 3)
Total 440 terms
• Extracting the top 100 search results for each search term(44,000)
• Removing whitelist URLs
• Grouping similar entries (same URL, source, and search term)
average roughly 15,000 unique URLs in each measurement period
10
Methodology
• Crawling Search Results
• Web crawler
• A Java web crawler using the HttpClient 3.x package from Apache
• Crawling 3 times for each URL
• Disguised as a normal user using Internet Explorer, clicking through the
search result
• Disguised as the Googlebot Web crawler
• Disguised as a normal user again, NOT clicking through the search
result
• Dealing with IP cloaking?
• Fourth crawling using Google Translate
• More than half of cloaked results do IP cloaking
11
Methodology
• Detecting Cloaking
• Removing HTTP error response (average 4% of URLs)
• Using Text Shingling to filter out nearly identical pages
• 90% of URLs are near duplicates ( “near duplicates” means 10% or less
differences between 2 sets of signatures)
• Measuring the similarity between the snippet of the search result
and the user view of the page
• Removing noise from both the snippet and the body of the user view
• Search substrings from the snippet
• Number of words from unmatched substrings divided by the total
number of words from all substrings
• 1.0 means no match
• 0.0 means fully match
• Threshold: 0.33  filter out 56% of the remaining URLs
12
Methodology
• Detecting Cloaking(cont.)
• False positives may still exist
• Examining the DOMs as the final test
• Computing the sum of an overall comparison and a hierarchical
comparison
• Overall comparison: unmatched tags from the entire page divided by the
total number of tags
• Hierarchical comparison: the sum of the unmatched tags from each
level of the DOM hierarchy divided by the total number of tags
• 2.0 means no match
• 0.0 means fully match
• Threshold: 0.66
13
Methodology
• Detecting Cloaking(cont.)
• Manual inspection
• False positive: 9.1% (29 of 317) in Google search
12% (9 of 75) in Yahoo
(benign websites but delivering different content to search engines)
• Advanced browser detection
• Temporal Remeasurement
• Dagger remeasures every 4 hours for up to 7 days
14
Results
• Cloaking Over Time
15
Results
16
Results
• Sources of Search Terms
17
Results
18
Results
19
Results
20
Results
• Search Engine Response
21
Results
22
Results
23
Results
24
Results
25
Results
• Cloaking Duration
26
Results
• Cloaked Content
27
Results
28
Results
• Domain Infrastructure
29
Results
• SEO
30
Conclusion
• Cloaking is an standard skill of constructing scam pages.
• This paper examined the current state of search engine
cloaking as used to support Web spam.
• New techniques for identifying cloaking(via the search
engine snippets that identify keyword-related content
found at the time of crawling)
• Exploring the dynamics of cloaked search results and
sites over time.