Yahoo’s Spider Slurp
How to write for Slurp the spider
As the world's second most popular search tool, Yahoo moves a tremendous amount of traffic
and is a very credible alternative to Google. Yahoo receives over 2.76 billion page views per
day from hundreds of millions of unique users. It boasts over 157 million registered users
enjoying mail, shopping and discussion groups and an increasingly personalized search and
news services. For the past two years, Yahoo, Google and MSN have been embroiled in a
hard-fought battle for the loyalty of search engine users forcing all three firms into the hyperevolution we are witnessing today. Over the next three Wednesdays we are going to examine
how the Big-3 spiders work, what they look for and how to best prepare your sites for multiple
visits from the bots that rank them. Today, we are starting with Yahoo's bot, SLURP.
Getting found by Slurp
The first thing to know about Slurp is that like its better known cousin, Google-bot, Slurp
"discovers" sites by following links from one site to another, reading and recording nearly
everything it finds in its path. The majority of websites referenced by Yahoo were originally
included in its database because they were accessed by Slurp following links from another
site.
Yahoo suggests adding an inbound link to all pages in your site to guarantee those pages will
be discovered by Slurp. They also recommend an internal sitemap linked to from the Index (or
home) page of the site. To encourage Slurp to spend more time deep-crawling your content,
Yahoo recommends the addition of "good authoritative links pointing into your site", from
highly reputable sources such as news sites, established business partners and other sites
relevant to your business or service.
Manual submission of the site is only recommended if for some reason or another Slurp does
not find the site on its own. This is increasingly rare however as server-logs show Slurp is one
of the most active spiders out there. In other words, if a site Slurp has already indexed links
to your site, Slurp will almost certainly be visiting very soon. Webmasters should never have
to pay submission fees to get into Yahoo's index as according to Yahoo's Tim Mayer, 99% of
Yahoo's index is crawled by Slurp for free.
It is still important to make sure your site is ready to receive a visit from Slurp. To ensure
Slurp is able to travel across your entire site, provide standard HREF text links as opposed to
forms, Flash or java script navigation tools. Webmasters are encouraged to avoid tracking and
communication methods that rely on using cookies across every page of the site. If you have a
database driven site or a site that creates unique sessions for each user, avoid embedding
session IDs in URL's. Lastly, use 404 pages to redirect users (and spiders) to the root (index)
page if a page or site URL becomes invalid. Yahoo also asks webmasters of sites with shopping
carts to use robot.txt exclusions in the source of the shopping carts.
Where your site has been included. Results May Vary...
Yahoo has seen enormous change over the past seven years. What started as a paid-inclusion,
human edited search directory, has grown into the second largest database of indexed
content. Yahoo is on the cutting edge of integrating several forms of media into their search
offerings and will likely soon produce its own entertainment content like an online HBO. Yahoo
is flirting with the concept of becoming an infotainment portal again but the core of its
offerings remains firmly rooted in search.
Yahoo search results come in multiple formats including: Yahoo-Local, Yahoo-products, MYYahoo (personalized results), specific nation-based Yahoo's, and the standard Yahoo.com One
of Yahoo's goals appears to be presenting individual search-users with results that best match
their personal needs. For instance, Yahoo would like to present constantly updated
geographic-specific references when a user searches for daily-use items such as groceries,
repair-workers, real estate and other services one would normally use a telephone directory to
find. Similarly, Yahoo wants to present the entire global database of references when a user
searches for international news, trans-national products or vacation plans. Being certain your
website gets served up for all levels of search, local, regional and global, will be important if
you wish to serve a market larger than your general region or community.
Getting Rankings
Yahoo's search engine ranks sites based on a formula that is very similar to the algorithms
used by rivals Google and MSN. Yahoo values many of the same elements other search
engines do including keyword enriched domain names, titles, meta tags, and content. Yahoo
also values keywords found in the anchor text of internal links, though the effect at Yahoo is
not as powerful as it is on Google.
According to Yahoo, well optimized pages and sites will continue to get good results across all
versions of their search engine. By opening your site to Yahoo Slurp and performing wellplanned optimization services across every page, a good SEO can nearly always achieve Top
placements on Yahoo. The trick is in offering Slurp the information it needs to read, record and
rank your site. If that information is included on each page, a set of text-based links is woven
through the site to provide easy passage for Slurp, and Yahoo is told what your business is,
where your business is located, and who your business serves, your site should achieve strong
rankings.
Due to the advent of personal, local, regional and global search results, it is highly
recommended to add full contact and address information on every page of a site. This
information should be as precise as possible and should include street address, unit or suite
number, zip or postal code, state or province, county and country, full telephone information
(including area codes), and if possible, the approximate longitude and latitude of your
business location. (look up your longitude and latitude here: http://www.astro.com/atlas)
When writing for Slurp here are a few basic fundamentals:












Have a descriptive URL
Use keyword enriched titles on each page of the site
Place keyword enriched description and keywords meta tags on each page of the site
Use robot.txt files to keep Slurp out of your shopping cart or log in pages
Place keyword enriched text in the first paragraphs of your site-copy
Use HREF links to direct Slurp through each page of the site
Add a sitemap page and be certain there is a link from the INDEX page to the sitemap
Be certain that geographic specific information is mentioned on each page of the site.
Always have a contact page that also lists geographic specific information.
Write a press release and send it to as many blogs, news-wires and press release sites
as possible.
Acquire strong, relevant incoming links from sites with topics similar to yours.
Update your site frequently.
Enjoy and value your placements.