Searching the World Wide Web:
Meta Crawlers vs. Single Search
Engines
By: Voris Tejada
The Beginning
• Since its inception the internet has grown
at a staggering rate with an extremely
large number of pages being added every
day
• Search engines such as Alta Vista, Excite,
HotBot, Infoseek, Lycos, and Northern
Light attempted to turn the internet into a
“15-billion-word encyclopedia”
Performance Measures For a Search
Engine
•
•
•
•
Coverage: also called “recall” in IR
Relevance: also called “precision” in IR
Freshness of pages in the index
Speed
What is the main problem with this?
• Coverage
– Despite their claims, no single search engine
could index the entire web
– Traditional IR systems were really designed
for static collections
• They could not keep up with the growth of the
internet
Experiment by Selberg and Etzioni
• They did an experiment using the results from
logs from their MetaCrawler web sites.
– “unique documents”
• What were the problems with this experiment?
– They only took the first X pages returned from
each engine
– Ranking system of each search engine was
different
Experiment by Lawrence and Giles
• Produced statistics on the coverage of the
major web search engines and the
estimated size of the web
• Compared the number of documents
returned by each and analyzed the results
• Problems
– They did not know if they were indexing
unique URLs or subsets of the same URLs
– Returned first X amount of documents
They found…
• Using the estimate that the web contains 320 million
pages they calculated the following:
– HotBot: 34% coverage
– Alta Vista: 28% coverage
– Northern Light: 20% coverage
– Excite: 14% coverage
– Infoseek: 10% coverage
– Lycos: 3% coverage
*Note: both experiments were concerned with coverage
What method did they find that could
increase coverage?
• Combining results from multiple engines
– By combining all six search engines they were
able to yield 3.5X the amount of results
– Selberg and Etzioni had created a
MetaCrawler which gathered a “market share”
of the results of each engine
*A solution better than MetaCrawler?
The First MetaCrawler
• Softbot
– Invented by Selberg and Etzioni at the
University of Washington
– What important qualities did it provide?
• A single interface to query through multiple search
engines such as Lycos and Alta Vista
• Obtained higher quality results as opposed to just
combining results
Modular Design
• User Interface
– Translates user queries and options into appropriate
parameters
• Aggregation Engine
– Obtains references, eliminates duplicates, collates &
outputs results
• Parallel Web Interface
– Downloads HTML pages from the Web, sends
queries and obtains results
• Harness
– Where service specific information is kept
Motivation
• Growth of the Web
• Difficulty in finding information
• Search engines index different documents and
use different ranking algorithms
– By using a single search engine you could miss over
77% of the most relevant references
• Interfaces of many search engines were difficult
to use
Softbot Addresses These Problems
• Aggregates web search services under a unified
interface
– Interface was much easier to use
– Forwards queries to single search engines and ranks
results into one composite list
• Obtains higher quality results
– Allows users to be more specific
– Eliminates duplicates using comparison algorithm
– Adapts to a rapidly changing environment
Formatting and Ranking
• MetaCrawler translates each query into the
appropriate format for use in each search engine
• Uses a “confidence score” to rank
– Allows each service to vote on relevancy for a
particular document
– Higher total score = higher ranking on final list
Speed
• Has user modifiable timeouts
• References only downloaded when needed or
only when user chooses to
• Shows partial results
– Doesn’t wait for full results list to be generated before
showing you something
Adaptability, Portability, Scalability
• Modular design allows for services to be added,
modified, and removed quickly
• Does not require large databases/large amounts
of memory, can run on most machines
• Has ability to scale without adding more
machines
MetaCrawlers Today
• “The Big Four””
– Dogpile
– Metacrawler
– Excite
– Webcrawler