Getting Your Web Site Listed
Danny Sullivan
Editor, Search Engine Watch
http://searchenginewatch.com/
Session’s Goal
 Cover the best way to increase the
percentage of your web site listed with
crawler-based search engines
 Discuss importance of submitting to
human-powered directories
 Not covering issues of improving ranking!
Types of Search Services
 “Search engines” crawl the web and read
your web pages
 “Directories” like Yahoo depend on
human-provided information to categorize
sites
 Most search services are a hybrid of both,
but more dominant in one area
How Search Engines Work
 Read pages they find on the web
 Store text in an “index”
 When you search, they look for pages
with matching text
 Other factors involved in “ranking” those
pages, such as “link popularity”
The Major Search Engines








Northern Light (big, but low traffic)
FAST Search (big, but even less traffic)
AltaVista (big and popular)
Inktomi (HotBot, UK Max)
Google (Netscape Search)
Excite (WebCrawler)
Go (Infoseek)
Lycos
(links at end)
Submitting To Search Engines
 Search engines should find you naturally,
but submitting helps speed the process
and can increase your representation
 Look for Add URL link at bottom of home
page
 Submit your home page and a few key
“section” pages
 Turnaround from a few days to 2 months
Deep Crawlers
 AltaVista, Inktomi, Northern Light will add
the most, usually within a month
 Excite, Go (Infoseek) will gather a fair
amount; Lycos gathers little
 Index sizes are going up, but the web is
outpacing them…nor is size everything
 Here are more actions to help even the
odds…
“Deep” Submit
 A “deep” submit is directly submitting pages
from “inside” the web site – can help improve
the odds these will get listed.
 At Go, you can email hundreds of URLs.
Consider doing this.
 At HotBot/Inktomi, you can submit up to 50
pages per day. Possibly worth doing.
 At AltaVista, you can submit up to 5 pages per
day. Probably not worth the effort.
 Elsewhere, not worth doing a “deep” submit.
Big Site? Split It Up
 Expect search engines to max out at
around 500 pages from any particular site
 Increase representation by subdividing
large sites logically into subdomains
 Search engines will crawl each subsite to
more depth
 Here’s an example...
Subdomains vs. Subdirectories
Instead of
Do this
gold.ac.uk/science/
gold.ac.uk/english/
gold.ac.uk/admin/
science.gold.ac.uk
english.gold.ac.uk
admin.gold.ac.uk
I Was Framed
 Don't use them. Period.
 If you do use them, search engines will
have difficulty crawling your site.
Dynamic Roadblocks
 Dynamic delivery systems that use ? symbols in
the URL string prevent search engines from
getting to your pages
 http://www.nike.com/ObjectBuilder/ObjectBuilder.iwx?ProcessN
ame=IndexPage&Section_Id=17200&NewApplication=t
 Eliminate the ? symbol, and your life will be rosy
 Look for workarounds, such as Apache rewrite
or Cold Fusion alternatives
 Before you move to a dynamic delivery system,
check out any potential problems.
How Directories Work
 Editors find sites, describe them, put them
in a category
 Site owners can also submit to be listed
 A short description represents the entire
web site
 Usually has secondary results from a
crawler-based search engine
The Major Directories
 Yahoo
 The Open Directory
 (Netscape, Lycos, AOL Search, others)
 LookSmart
 UK Plus
 Snap
Submitting To Directories
 Directories probably won't find you or may list
you badly unless you submit
 Find the right category (more in a moment),
then use Add URL link at top or bottom of page
 Write down who submitted (and email address),
when submitted, which category submitted to
and other details
 You’ll need this info for the inevitable resubmission
attempt – it will save you time.
Submitting To Directories
 Take your time and submit to these right
 Write 3 descriptions: 15, 20 and 25 words long,
which incorporate your key terms
 Search for the most important term you want to
be found for and submit to first category that's
listed which seems appropriate for your site
 Be sure to note the contact name and email
address you provided on the submit form
 If you don't get in, keep trying
Subdomain Advantage
 Directories tend not to list subsections of
a web site.
 In contrast, they do tend to see
subdomains as independent web sites
deserving their own listings
 So, another reason to go with subdomains
over subdirectories
Other Tips / Misc
 Under represented? Get in touch!
 Need Search Engine Links? See:
http://searchenginewatch.com/links/