how does a web search
engine work?
 google
(started 1998 … now worth $365 billion)
 bing
 amazon
search
 web, images, news, maps, books, shopping, apps, videos, music – and
much more!
 sometimes people tell you what they want
 type your query into google
 sometimes you have to guess and offer stuff to them
 amazon gives you shopping ideas
search engines help people find things
our objective
saves them time and effort
as a search engine, your job is to make sure they can
find things quickly and easily!
1. collect the things that you want to search
... take a snapshot of all the internet
2. figure out what those things are “about”
overview
... words in text documents, speech in videos,
notes in sound
3. allow people to find what they want from your
collection
… decide which things you have are relevant
… like finding a needle in a haystack!
crawling
3 core parts
the web ‘spider’
crawls across all the
pages on the internet
indexing
retrieving
like a librarian
categorising books
letting people find the
stuff they want
 the internet is huge (trillions of webpages)
 useless information (old, poorly written, advertising,
duplicates)
but it gets
complicated…
 inappropriate stuff
 different languages
 spam
 lots of computers to do you work for you!
 but you need to tell them what work to do – programming.
 they all have to work together
 what if some break?
search engines
are expensive
 24/7/365
 mobile phones, ipads, computers etc
 in every corner of the world
 lots of fast internet connections
servers
cooling
let’s take a little look at how each part works…
 start with a bunch of website you know about
 and just follow the links…
 Imagine if you kept clicking all the links forever
1. crawling
 How long would it take to get back to the page you started on, if you were
clicking on a different link each time?
 Could you cover all the pages on the internet?
 Is it equally likely you will cover all pages? What about more popular pages,
for example: bbc.com, facebook.com etc?
2. indexing
activity...
3. retrieval
activity...
 words like ‘the’, ‘a’, ‘and’, ‘what’ – useless!
 tiger/tigers, bengali, tyger, big cat – plurals, spelling mistakes, synonyms
challenges
 what people search for ‘mismatches’ exactly what people write, even though
it means the same!
 how easy to read, or “helpful” is a web page?
 how about the search ‘topic’ is the page, really?