Spring
Ferret: 360o Search
Prafulla Mahindrakar
Aniket Patil
Ketan Umare
Advisor: Dr. Ling Liu
CS8803:Advanced Internet Application Development, Group Project.
09
2
FERRET: 360O SEARCH
Table of Contents
1. MOTIVATION AND OBJECTIVES
4
2. RELATED WORK
5
2.1 GOOGLING
2.2 SOCIALLY RELEVANT SEARCH
2.2 CATEGORIZATION OF SEARCH RESULTS
5
6
6
3. ARCHITECTURE
7
3.1 SYSTEM ARCHITECTURE DIAGRAM
3.2 PATTERN ORIENTED ARCHITECTURE
3.2.1 DESIGN PATTERNS
3.2.2 DESIGN PATTERNS USED IN FERRET
3.3 HIGH PERFORMANCE
3.3.1 THREADING
3.3.2 CACHING
3.4 DATABASE SCHEMA
3.4.1 ER DIAGRAM
3.4.2 USER TABLE
3.4.3 PAGE KEYWORD TABLE
3.4.4 USER SESSION TABLE
7
8
8
8
9
9
9
9
9
10
10
10
4. COMPONENTS
11
4.1 AUTHENTICATION
4.2 STANFORD TAGGER
4.2 WEB SEARCH
4.3 MEDIA SEARCH
4.4 PRODUCT SEARCH
4.4.1 CLUSTERING OF RESULTS
4.5 SOCIAL SEARCH
4.5.1 SESSIONS
4.5.2 LISTEN TO USER CLICKS
4.5.3 HEARTBEAT MESSAGES
4.6 CATEGORIZATION ENGINE
4.6.1 WHY DOCUMENT CLUSTERING
4.6.2 APPROACHES
4.6.3 BUILDING BLOCKS
4.6.4 LINGO ALGORITHM
4.7 PRESENTATION ENGINE
4.7.1 SEARCH RESULTS TAB CREATOR
4.8 VIEW
11
11
11
11
11
12
12
13
13
13
14
14
14
15
16
18
18
18
5. EVALUATION FRAMEWORK
19
2 FERRET: 360O SEARCH
FERRET: 360O SEARCH 3
5.1 SOCIAL NETWORK SIMULATION
5.1.1 ER DIAGRAM
5.2 JMETER
5.2.1 TEST CASES
5.3 COMPARISON TO OTHER SEARCH ENGINES
19
20
20
20
22
6. TESTING AND RESULTS
22
6.1 PROTOTYPE SYSTEM
6.1.1 SOFTWARE
6.1.2 HARDWARE
6.1.3 OPERATING SYSTEM
6.2 RESULTS
6.2.1 WITHOUT MEMCACHED
6.2.2 WITH MEMCACHED
6.2.3 LOAD RESPONSE WITH MEMCACHED
22
22
23
23
23
23
23
25
7. FUTURE WORK
25
8. CONCLUSIONS
25
9. BIBLIOGRAPHY
26
9. APPENDIX
27
FERRET: 360O SEARCH 3
4
FERRET: 360O SEARCH
1. Motivation and Objectives
fer·ret (v) (\ˈfer-ət\)
to find and bring to light by searching
Imagine trying to find a pair of the latest Ray-Ban glasses in the Lenox Square Mall.
It is not an easy task! Now think about doing the same across the World Wide Web.
Feeling tizzy? The World Wide Web with its astronomical amount of information
presents an enormous challenge for resource discovery. Precise navigation is
impossible with the increasingly large collection of hyperlinks that users must
traverse. Commercial search engines like Google and Yahoo have solved the
problem at a fundamental level by making available a hypertext-based index for
pages across the web. Web Users can query the index for documents about a specific
topic to find the desired document.
While search engines have become quite popular and are helping to redefine how
people access information scattered across the wide-area network, they are not well
suited to the case when users do not know what exactly they are looking for. In such
a situation, using one of the popular search engines can be a messy, frustrating
experience. What do you do when you don’t know where to start? Give Ferret a try!
For any topic in the universe, Ferret provides a neatly organized view of the web.
Our category guides bring meaningful and relevant information that makes
browsing for a topic fast. Rather than the messy back-and-forth clicking of search
results, we do the processing so that you can learn, explore and discover the things
that matter to you. Ferret offers you a new way to discover the Web – it’s the place
you should be when you want to browse and discover everything the Web has to
offer. Come to Ferret when you want to learn about a topic or explore what’s
happening now on the Web. We’ll show you content that you may have never
discovered otherwise and we’ll give you an at-an-glance look at everything related
to the query. Think of Ferret as your guide for exploring the Web.
For instance, consider the search term ‘Transformers’. A Google search result
returns a list arranged serially that speaks about the movie ‘Transformers’, and
electrical transformers on the first page. However, a user who is interested in
knowing about the class ‘Transformer’ in Java or about the comics on Transformers
needs to browse several pages before such results are discovered. Our system
graphically arranges and classifies results into categories such as text, multimedia,
entertainment, discussions, blogs and more. A user simply needs a single click to
have a 360 degree view of content associated with the query term.
4 FERRET: 360O SEARCH
FERRET: 360O SEARCH 5
Ferret. Your guide to the world!
2. Related Work
Search has been a constantly evolving and a continuously researched topic. There
have been great success stories and even greater debacles in this industry. Web
search has become such an important part of our life that it has contributed to our
vocabulary in some cases. Following are some of the most different systems
currently available online, from which we derive and drive our inspiration.
Figure 1: Taxonomy of Existing Search Technologies
2.1 Googling
In their seminal work [1], the authors described a new way of ranking web
documents, based on the idea of citation. The Search engine instantly became a hit
and overtook all of its competitors. The webpage [2] is the most highly visited page
online and everyone knows “The Google Story”.
Google uses a simple keyword based search, but the most important point is the
ranking of content. Thus Google successfully demonstrates the idea that just the
content is not important, but the way we present it is highly important. Google has
continued to innovate and come up with great innovative new features, but still it
has a long way to go.
FERRET: 360O SEARCH 5
6
FERRET: 360O SEARCH
2.2 Socially relevant search
Social search or a social search engine is a type of web search method that determines
the relevance of search results by considering the interactions or contributions of
users.[3]
Based on this simple idea is Delver[4], which uses the social network of a user to
come up with better recommendations. It enables you to find, experience and
benefit from the wealth of information created and referenced by your social world.
Socially relevant search can really benefit a user, as what matters to him is usually
what matters to his peers. Paper [5] talks about the benefits of integrating the web
search and social search and quantifies it with great results. It also delineates the
challenges in doing so.
2.2 Categorization of search results
Search results categorization is another important way to present the search results.
Take an example of the word Transformers. For the same word we could have
different implications – an electrical device, a movie, the cartoon series, a toy, there
could be a review about the movie, or some news about the invention of some new
efficient transformer, etc. So how do you show these results? Which is more
important?
These questions are almost impossible to answer. Papers[6-9] show a variety of
ways in which we can classify the web search results and quantify them with
interesting results. But Kosmix[10], is one of the most promising sites that has
leveraged from this idea. It uses the search provided by Google, and creates a
wrapper for its own classification system. It has been voted as one of the best new
startups[11] and that just makes a statement about the importance of classification
of results.
6 FERRET: 360O SEARCH
FERRET: 360O SEARCH 7
3. Architecture
The following sections give an outline of the System architecture and a small
description of the important components.
3.1 System Architecture Diagram
Figure 2 System Architecture
Ferret can operate in two modes, Logged in or Private mode. Each of these modes
are described in detail in the later sections. In the logged in mode alongwith the
typical web results, ferret also provides socially relevant search results, using the
FERRET: 360O SEARCH 7
8
FERRET: 360O SEARCH
users profile form one of the major social network databases, for example facebook
OR twitter. The typical web results are categorized into 3 broad categories, namely,
Web Search, Media Search and Product Search. Each Category is further categorized
using our clustering algorithm.
3.2 Pattern Oriented Architecture
The aim while developing ferret was to keep it flexible enough so that we can add
new features with relative ease. Also performance was a major concern, so each of
the components built was built for a large-scale system. This could be easily
achieved using Pattern oriented architecture. The following section describe the
various patterns used in ferret.
3.2.1 Design Patterns
A Design Pattern can be defined as a particular recurring design problem that arises in
specific design contexts, and presents a well-proven generic scheme for its solution.
Describing its constituent components, their responsibilities and relationships, and the
ways in which they collaborate specifies the solution scheme[12, 13].
3.2.2 Design Patterns used in Ferret
Ferret uses these design patterns.
3.2.2.1 Front Controller
A Front controller pattern enables centralized request processing. This enables
changes to the levels below to be transparent. Even communication, threading can
be abstracted easily from the presentation layer.
3.2.2.2 Abstract Factories
Factories is a creational pattern that abstracts creation of objects from the place
where it is used. This provides ease of adding modules.
3.2.2.3 Strategy
A strategy pattern allows ferret to change clustering algorithms easily and thus
allowing new algorithms to be plugged in with relative ease. This especially was
vital during testing out various algorithms.
3.2.2.4 Adapter
Adapter pattern is used to abstract the search/fetch/cluster logic from the
presentation generator. This generator can also be modified easily irrespective of
changes to the prior system.
3.2.2.5 Singleton
Many things needs single connections and to avoid the overhead we used thread
controllers in singletons so that we could reduce the thread creation overhead. Also
tagger library is loaded just once so that we avoid the cost associated with re
reading it.
8 FERRET: 360O SEARCH
FERRET: 360O SEARCH 9
3.2.2.6 Spring (OS) Doors
Just as the Spring system developed at Sun labs we have Controller which abstracts
the access of data from the presentation layer. This allows us to deploy individual
systems remotely, which could be employed in the future for large scale distributed
computing.
3.3 High Performance
For a search engine Performance is critical. Ferret achieves performance using large
scale threading, distributed caching and easily allowing separation of modules onto
separate physical hosts.
3.3.1 Threading
Ferret uses pre-spawned thread pools to offset the overhead of thread spawning. It
also uses threads to perform searches across various domains parallely.
3.3.2 Caching
Ferret uses memcached[14, 15] to cache recent results. To maintain freshness of
the results, each cached entry is associated with Expiry value. Currently the expiry
time is arbitrarily fixed, but future efforts would aim at arriving at this number
using a learning algorithm. For example, it is known that google doesnot refresh its
index for atleast n hours. In that case we could cache till the results are updated.
3.4 Database Schema
The database used by ferret is minimal, and this is essential to enhance the
performance. The following section describes the schema in detail.
3.4.1 ER Diagram
usr_user
PK
uid
puk_pagekeyword
PK
username
password
name
profession
imageUrl
pageid
page
keyword
title
uss_usersession
PK
uid
pageid
historycount
timestamp
sessionid
Figure 3 Database Model for Social Search
Table usr_user:
Column Name
Uid
Description
Auto-generated primary key for usr_user table
FERRET: 360O SEARCH 9
10 FERRET: 360O SEARCH
Username
Password
Name
Profession
ImageUrl
Login name for the user
User’s password
User’s name
User’s profession
Pathname for the user image
Table puk_pagekeyword:
Column Name
pageid
page
keyword
title
Description
Auto-generated
primary
key
for
Puk_pagekeyword
table
URL of the page
Processed query term for which page was
retrieved
Title for page
Table uss_usersession:
Column Name
uid
pageid
historycount
timestamp
sessionid
Description
Auto-generated
primary
key
for
uss_usersession
table
Refers to puk_pagekeyword.pageid
Frequency of usage of search results
Time at which user selected a page for reading
Server-generated session id for user
3.4.2 User Table
The user table is needed to maintain login information of the user in case the Google
Authentication system isn’t used. Also it stores the uid’s which again could be
directly from facebook, bt would be needed in case of multiple networks.
3.4.3 Page Keyword Table
This table is used to maintain a list of popular keyword and page combinations
accessed by the users. Based on freshness criteria, this table should be cleaned
every x number of days.
3.4.4 User Session table
This table is used to track the user and his favorite links. This table is essential to
implement the Good page Bad page algorithm.
10 FERRET: 360O SEARCH
FERRET: 360O SEARCH 11
4. Components
This section explains the various modules that constitute the ferret search engine.
4.1 Authentication
Ferret uses its own database to authenticate the user. It is easy to instead use the
Google OpenId system for authentication. The system treats the user as a guest and
does not track your activities. This enables private browsing.
4.2 Stanford Tagger
The Stanford Tagger used by Ferret is a Part-Of-Speech Tagger. It is a piece of
software that reads text in some language and assigns parts of speech to each word
(and other tokens), such as noun, verb, adjective, etc.
The tagger is used to identify relevant keywords in a query and store them in the
database. The tagger is used in the following components:

Dictionary Search: The tagger identifies nouns (personal, common, both
singular and plural) as keywords to be sent to WordNet for query.

Social Search: The tagger identifies nouns (personal, common, both
singular and plural), verbs and adverbs from the user’s query.
4.2 Web Search
This engine is multithreaded and accepts the raw query and dispatches it to the
various worker threads, which aim at collecting the search results from variety of
search engines like Google[2], A9[16], IMDB[17] etc. The worker threads use WSDL
to communicate to the various search engines. The external interface is extensible
since collecting results from a new search engine simply requires the
implementation of a WSDL interface. This enables our system to be augmented by
additional search results through Yahoo, Windows Live or any other search engine.
4.3 Media Search
This engine is multithreaded and accepts the raw query and dispatches it to the
various worker threads, which aim at collecting the search results from variety of
search engines like Google[2], A9[16], IMDB[17] etc. The worker threads use WSDL
to communicate to the various search engines. The external interface is extensible
since collecting results from a new search engine simply requires the
implementation of a WSDL interface. This enables our system to be augmented by
additional search results through Yahoo, Windows Live or any other search engine.
4.4 Product Search
Ferret product search uses Amazon E-Commerce API to retrieve product
information. The API exposes Amazon's product data and e-commerce functionality.
This allows Ferret to leverage the data that Amazon uses to power its own business.
FERRET: 360O SEARCH 11
12 FERRET: 360O SEARCH
Ferret is able to retrieve product results over a huge range of categories. For every
product, Ferret retrieves the product name, product cost as on Amazon and a
product image.
All searches are performed for US locale. In the future, it may be possible to detect
the geographical region from where the query originates and adjust the locale
accordingly.
4.4.1 Clustering of Results
The search results are clustered dynamically on the basis of categories that are
retrieved for the query term. All products belonging to a single category are
arranged together using seed list based clustering. Any Amazon product can be
classified into one of the following categories:
Apparel, Automotive, Baby, Beauty, Blended, Books, Classical, Digital Music, DVD,
Electronics, Foreign Books, Gourmet Food, Health Personal Care, Hobbies, Home Garden,
Jewelry, Kitchen, Magazines, Merchants, Miscellaneous, Music, Musical Instruments, Music
Tracks, Office Products, Outdoor Living, PC Hardware, Pet Supplies, Photo, Restaurants,
Software, Software Video Games, Sporting Goods, Tools, Toys, VHS, Video, Video Games,
Wireless, Wireless Accessories
Figure 4 List of product categories in Amazon
We used the above categories as a seed list and use the retrieved product
information to detect the category and cluster appropriately.
Due to the extensible nature of the product search component, we can easily obtain
results from other e-commerce providers such as Buy.com and Ebay. We also plan
to integrate functionality to sort results by cost and social relevance.
4.5 Social Search
Ferret adds a new spin to search: social networking. One of the most innovative
features of Ferret is the ability to retrieve search results that are relevant to the
user’s social network.
The feature allows the user to leverage searches performed by the user’s friends.
Social search recommends the best pages found by people in the user’s network that
are relevant to the user’s query. Ferret’s social search tries to match the user’s query
term with a larger set of searchers in the user’s social network that are looking for
the same things. The results are clustered by the friend’s name and are listed
serially. Each result contains the page name and the page url which is clickable for
the purpose of viewing.
The feature is an opt-in: no one can see what the user is searching for unless the
user logs in. This ensures user privacy.
12 FERRET: 360O SEARCH
FERRET: 360O SEARCH 13
Social Search is implemented using the following primitives:



Sessions
Listen to user clicks
Heartbeat Messages
4.5.1 Sessions
In Ferret, a session stores the state of communication between a server and the user
enabling the server to identify that user across multiple page requests or visits to
that site.
A session is created when a user logs in with his username and password. The
session for a user stores the following attributes:




User ID: The primary key generated by the database for the user
Query: The query term currently being searched by the user
Page URL: The page currently being viewed by the user
Timestamp: The time at which the user clicked on the page url
4.5.2 Listen to user clicks
A request is sent to the server each time the user clicks on a page for browsing. This
is used to store associate the query term processed by Stanford Tagger (keywords)
with the user id previously stored in the session for future processing. The
algorithm for the user clicks is as follows:
1. If session is invalid
2.
Return
3. Else if no timestamp exists in session
4.
Insert URL into session
5.
Insert Keyword into session
6.
Insert Timestamp into session
7.
Return
Figure 5 User-Click Algorithm
4.5.3 Heartbeat Messages
A heartbeat message is an event-driven message which is sent to the server when
there is a search results page is reloaded. This message is used to detect if the user
likes the page he has just viewed. We use heuristics to differentiate such a page from
one the user does not like.
The heuristic Ferret uses is as follows: If the user spends more time on a certain
page, we can assume he does so because he likes the page. If the user returns back
from a page “quickly”, he does not like the page. Currently, we have set a timeout of
30 seconds to differentiate a good page from a bad page. If the user spends 30
seconds or greater on a specific page, the system records the page as a good page
FERRET: 360O SEARCH 13
14 FERRET: 360O SEARCH
and stores it in the database. If the user returns from the page in less than 30
seconds the page is not associated with the user.
The algorithm for the heartbeat process can be summarized as follows:
1. If session is invalid
2.
Return
3. Else if no timestamp exists in session
4.
Return
5. Else if page is liked by user
6.
Associate page-keyword combination with userid
7.
Return
8. Else if page is disliked by user
9.
Remove page-keyword association with user
10.
Return
Figure 6 Heartbeat Algorithm
In the future, social search can be improved by deducing the “meaning” of the query
being searched using natural language processing query techniques and using the
meaning to retrieve search results. For instance, if the user is searching for “what
drug treats a headache” Ferret can process the semantic relationships between
words and may deduce that someone searching for “what medicine relieves
migraines” is a match. In addition, it may be possible to rank a set of results
retrieved for a specific user’s friend by freshness or relevance to the query.
4.6 Categorization Engine
The results collected through the various websites are then categorized using Lingo
clustering, and then grouped into different categories.
4.6.1 Why Document Clustering
With an enormous growth of the Internet it has become very difficult for the users
to find relevant documents. In response to the user’s query, currently available
search engines return a ranked list of documents along with their partial content
(snippets). If the query is general, it is extremely difficult to identify the specific
document which the user is interested in. The users are forced to sift through a long
list of off-topic documents. Moreover, internal relationships among the documents in
the search result are rarely presented and are left for the user.
One approach is to automatically group search results into thematic groups
(clusters) which would help the user to see various perspective of the same query
grouped into categories.
4.6.2 Approaches
Clustering of web search results was first introduced in the Scatter-Gather system.
Several algorithms followed; Suffix Tree Clustering, (STC), implemented in the
Grouper system pioneered in using recurring phrases as the basis for deriving
14 FERRET: 360O SEARCH
FERRET: 360O SEARCH 15
conclusions about similarity of documents. MSEEC and SHOC also made explicit use
of words proximity in the input documents. Apart from phrases, graph-partitioning
methods have been used in clustering search results
All the above approaches follow a scheme where cluster content discovery is
performed first, and then, based on the content, the labels are determined. But very
often intricate measures of similarity among documents do not correspond well
with plain human understanding of what a cluster’s “glue” element has been. To
avoid such problems Lingo algorithm reverses this process and attempt to ensure
that it can create a human-perceivable cluster label and only then assign documents
to it. This the approach we have followed in our implementation of clustering web
results.
4.6.3 Building Blocks
The following section describes the building blocks for the implementation of the
clustering algorithm used in ferret.
4.6.3.1 Vector Space model
Vector Space Model (VSM)[18] is a technique of information retrieval that
transforms the problem of comparing textual data into a problem of comparing
algebraic vectors in a multidimensional space. Once the transformation is done,
linear algebra operations are used to calculate similarities among the original
documents. Every unique term (word) from the collection of analyzed documents
forms a separate dimension in the VSM and each document is represented by a
vector spanning all these dimensions.
For example, if vector v represents document j in a k-dimensional space ,then
component t of vector v, where t 1 . . . k, represents the degree of the relationship
between document j and a term corresponding to dimension t. This relationship is
best expressed as a t X d matrix A, usually named a term-document matrix , where t
is the number of unique terms and d is the number of documents. Element aij of
matrix A is therefore a numerical representation of relationship between term i and
document j. There are many methods for calculating aij , commonly referred to as
term weighting methods.
4.6.3.2 Calculating Relevance
We use the tf-idf method for calculating the term weights. The tf–idf weight (term
frequency–inverse document frequency) is a weight often used in information
retrieval and text mining. This weight is a statistical measure used to evaluate how
important a word is to a document in a collection or corpus. The importance
increases proportionally to the number of times a word appears in the document
but is offset by the frequency of the word in the corpus.
4.6.3.3 Suffix Arrays
Let A = a1a2a3 . . . an be a sequence of objects. Let us denote by Ai a suffix of A
starting at position i ! 1 . . . n, such as Ai =aiai+1ai+2 . . . an. An empty suffix is also
FERRET: 360O SEARCH 15
16 FERRET: 360O SEARCH
defined for every A as An+1 = #. A suffix array[19] is an ordered array of all suffixes
of A. Suffix arrays are used as an efficient data structure for verifying whether a
sequence
of objects B is a substring of A.The complexity of this operation is O(P + logN), a
suffix array can be builtin O(NlogN).
4.6.3.5 Singular value Decomposition
An algebraic method of matrix decomposition called Singular Value
Decomposition[20] is used for discovering the orthogonal basis of the original termdocument matrix. This basis consists of orthogonal vectors that, at least
hypothetically, correspond to topics present in the original term-document matrix.
SVD breaks a t X d matrix A into three matrices U, ∑ and V , such that A = U∑ VT . U is
a t X t orthogonal matrix whose column vectors are called the left singular vectors of
A, V is a d X d orthogonal matrix whose column vectors are called the right singular
vectors of A, and ∑ is a t X d diagonal matrix having the singular values of A ordered
decreasingly along its diagonal. The rank rA of matrix A is equal to the number of its
non-zero singular values. The first rA columns of U form an orthogonal basis for the
column space of A—an essential fact used by Lingo.
4.6.4 Lingo Algorithm
At the very high level lingo[21] first finds frequent phrases from the input
documents, hoping they are the most informative source of human-readable topic
descriptions. Next, by performing reduction of the original term-document matrix
using SVD, it tries to discover any existing latent structure of diverse topics in the
search result. Finally, it match group descriptions with the extracted topics and
assign relevant documents to them.
4.6.4.1 Preprocessing
The aim of the preprocessing phase is to prune from the input all characters and
terms that can possibly affect the quality of group descriptions. Two steps are
performed: text filtering removes HTML tags, entities and non-letter characters
except for sentence boundaries. Next, appropriate stemming and stop words
removal end the preprocessing phase.
4.6.4.Phrase Extraction
We define frequent phrases as recurring ordered sequences of terms appearing in
the input documents. Intuitively, when writing about something, we usually repeat
the subject-related keywords to keep a reader’s attention. Obviously, in a good
writing style it is common to use synonymy and pronouns and thus avoid annoying
repetition. To be a candidate for a cluster label, a frequent phrase or a single term
must:
 appear in the input documents at least certain number of times (term
frequency threshold),
 not cross sentence boundaries,
16 FERRET: 360O SEARCH
FERRET: 360O SEARCH 17
 be a complete phrase ,
 not begin nor end with a stop word.
We use suffix arrays to find such complete phrases.
4.6.4.2 Cluster Label Induction
Once frequent phrases (and single frequent terms) that exceed term frequency
thresholds are known, they are used for cluster label induction. There are three
steps to this: term-document matrix building, abstract concept discovery,
phrase matching and label pruning.
The term-document matrix is constructed out of single terms that exceed a
predefined term frequency threshold. Weight of each term is calculated using the
standard term frequency, inverse document frequency (tfidf) formula, terms
appearing in document titles are additionally scaled by a constant factor.
In abstract concept discovery, Singular Value Decomposition method is applied to
the term-document matrix to find its orthogonal basis.Vectors of this basis (SVD’s U
matrix) represent the abstract concepts appearing in the input documents.
Phrase matching and label pruning step, where group descriptions are discovered,
relies on an important observation that both abstract concepts and frequent phrases
are expressed in the same vector space—the column space of the original termdocument matrix A.The classic cosine distance is used to calculate how “close” a
phrase or a single term is to an abstract concept. Let us denote by P a matrix of size
t X (p+t) where t is the number of frequent terms and p is the number of frequent
phrases. Having the P matrix and the i-th column vector of the SVD’s U matrix, a
vector mi of cosines of the angles between the i-th abstract concept vector and the
phrase vectors can be calculated.
mi = UI T P.
The phrase that corresponds to the maximum component of the mi vector should be
selected as the human-readable description of i-th abstract concept.
4.6.4.2 Cluster Content Discovery
In the cluster content discovery phase, the classic Vector Space Model is used to
assign the input documents to the cluster labels induced in the previous phase. In a
way, we re-query the input document set with all induced cluster labels. The
assignment process resembles document retrieval based on the VSM model. Let us
define matrix Q, in which each cluster label is represented as a column vector. Let C
= QTA, where A is the original term-document matrix for input documents. This way,
element cij of the C matrix indicates the strength of membership of the j-th
document to the i-th cluster. A document is added to a cluster if cij exceeds the some
threshold yet another control parameter of the algorithm. Documents not assigned
to any cluster end up in an artificial cluster called Others.
FERRET: 360O SEARCH 17
18 FERRET: 360O SEARCH
4.7 Presentation Engine
This module is responsible for displaying and painting the results for the user
browser. It uses the Adapter pattern to abstract the search part from the display
part.
4.7.1 Search Results Tab creator
This interface creates a tab and each type of tab can be separated into a different
class. The most important functions are written in the base class and whenever a tab
is needed to be different a simple class can be easily written.
4.8 View
The clustered results and the socially relevant search results are then showed to the
end user in tabbed format, which allows the user to easily find his appropriate
content. The view uses Mootools[22], which is an opensource Javascriptig
framework, which enables it to be browser agnostic.
The following chart shows the performance comparison of mootools with other
java-scripting frameworks. The performance alongwith the ease of use makes it one
of the preferred choices.
18 FERRET: 360O SEARCH
FERRET: 360O SEARCH 19
Figure 7 Performance comparison of various Java-Scripting Frameworks (source:Blog)
5. Evaluation Framework
Ferret tries to use many old and some new ideas to combine them into a new
exciting product. Hence evaluation of such a system is critical. The Evaluation falls
under Three broad categories
Social Network based relevance
Performance
Comparison to other contemporary search engines
5.1 Social Network Simulation
Ferret needs the social network to provide information about a user and his friends
so that it can perform and maintain social relevance search results. Though it has a
facebook engine ready, Facebook authentication system requires a static IP or a URL
to work with. Due to this limitation it became essential to simulate the social
network. The following section describes a simple social network simulation
FERRET: 360O SEARCH 19
20 FERRET: 360O SEARCH
5.1.1 ER Diagram
Ferret presently simulates a social network to implement Social Search. The
database model used is as follows:
ufl_userfriends
uhb_userhobbies
usk_usersearchpage
PK
PK
PK
uid
fid
uid
hobbies
uid
pageid
Figure 8 Database Model for Social Network
Table ufl_userfriends:
Column Name
Uid
fid
Description
Refers to usr_user.uid
Refers to usr_user.uid
Table uhb_userhobbies:
Column Name
Uid
hobbies
Description
Refers to usr_user.uid
User hobby name
Table Usk_usersearchpage:
Column Name
Uid
pageid
Description
Refers to usr_user.uid
Refers to puk_pagekeyword.pageid
Ferret does not use uhb_userhobbies table currently in simulation. It is possible to
consider the user’s friends’ hobbies when recommending social search results to the
user.
5.2 JMeter
Performance of a search engine is critical and JMeter is an open source tool that can
simulate multiple clients sending post request[23, 24]. It can also load test the
application. Ferret was tested using JMeter and various performance stats were
collected. This section provides details on the test cases.
5.2.1 Test Cases
20 FERRET: 360O SEARCH
FERRET: 360O SEARCH 21
First screenshot shows the Jmeter Test plan setup screen. The Testplan is called
Ferret Testplan. Screenshot 2 shows the type of parameter, namely the search
query, to be passed and type of HTTP request to be sent, for example POST or GET.
Screenshot 3 shows the expected amount of load (number of users), number of time
each query is executed and the gap between consecutive queries.
FERRET: 360O SEARCH 21
22 FERRET: 360O SEARCH
5.3 Comparison to Other search engines
Search engines performance has an important component, which deals with the
quality of results for a particular query. Such an evaluation is very subjective. To
compare the result of ferret to contemporary search engine method of surveying
was used.
6. Testing And Results
6.1 Prototype system
We have build a prototype systems for the demo using the hardware and software
listed in the following sections.
6.1.1 Software
 Java 1.6
 Eclipse IDE
 J2EE 1.4
 Apache Tomcat 5.5
 MySQL 5.0
 Clustering Algorithms (Developed by us)
 Mootools
 Multibox
 MySQL JDBC Connector
 JUnit 4.4
 Open Source Web / REST API’s for Google, IMDB, Facebook etc.
22 FERRET: 360O SEARCH
FERRET: 360O SEARCH 23
6.1.2 Hardware
We need simple commodity hardware, as it will not be a live system, but a proof of
concept. Currently a Desktop PC with a browser and internet connectivity would
suffice. We would primarily develop on our laptops.
6.1.3 Operating system
The primary development and test platforms would be
 Windows 98/XP/Vista
 MacOSX 10.5.5 (Leopard)
Though most of the technologies we are using are completely portable and we
should be able to run on most systems that support JAVA.
6.2 Results
We conducted results using memcached and Tomcat. For every search engine
response times are very important. Since we use Google as our search provider our
times can never be better than Google. Each Tab is separated on different threads
and page is created parallely.
6.2.1 Without Memcached
Without memcached, the same query takes approximately constant response times.
This is because the entire result set is constructed for the same query al over again
for every request.
6.2.2 With Memcached
FERRET: 360O SEARCH 23
24 FERRET: 360O SEARCH
Memcached improves the performance but a small amount of time is spent as the
social results are never cached. But since they are stored locally on Ferrets own
database, the bottleneck is because of the remote servers and the clustering system.
When the user is not logged in the complete page is constructed completely using
the cached results. The Thread pools are not interrupted and thus the performance
is very high.
24 FERRET: 360O SEARCH
FERRET: 360O SEARCH 25
6.2.3 Load Response with Memcached
The above graph shows the response time of Ferret system with multiple
concurrent users search for the same query. It is evident that we need a server or a
host of servers to handle multiple concurrent users.
7. Future Work
There are a lot of changes that we dream of, and we have a long way to go. This
serves as a good demo tool, but not a final product. Following are some things we
have planned for Ferret.
o
Using up our summer vacation to build on it
o
Notion of Social Rank
o
Adding blogs, forums, reservations, email search to search results
o
Using Digg interface to re-rank sites
o
Learning better categories
o And the list goes on...
8. Conclusions
This was a very good learning experience. One of the most important things we
learnt was how to develop an idea and get a working prototype. From our
perspective, there are two navigation paradigms on the Web – Search and Browse.
Search lets you find specific bits of information quickly or navigate to sites you
already know. Browse gives you a more immersive way to explore a topic so that
FERRET: 360O SEARCH 25
26 FERRET: 360O SEARCH
you can learn more about something or discover something new. Ferret is about
reinventing Browse just as Google reinvented Search.
9. Bibliography
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
Brin, S. and L. Page, The anatomy of a large-scale hypertextual Web search
engine. Computer Networks and ISDN Systems, 1998. 30(1-7): p. 107-117.
larry page, S.B. Google. Available from: http://www.google.com.
Wikipedia-The free Encyclopedia. Available from: http://www.wikipedia.com.
Liad Agmon, A.y., Sagie Davidovitch(co-founders), Delver.
Mislove, A., K. Gummadi, and P. Druschel. Exploiting Social Networks for
Internet Search. 2006.
Chen, H. and S. Dumais. Bringing order to the Web: automatically categorizing
search results. 2000: ACM Press New York, NY, USA.
Thet, T., J. Na, and C. Khoo, Automatic Classification of Web Search Results:
Product Review vs. Non-review Documents. LECTURE NOTES IN COMPUTER
SCIENCE, 2007. 4822: p. 65.
Vogel, D., et al., Classifying search engine queries using the web as background
knowledge. SIGKDD Explor. Newsl., 2005. 7(2): p. 117-122.
Yeung, A., N. Gibbins, and N. Shadbolt, A k-Nearest-Neighbour Method for
Classifying Web Search Results with Data in Folksonomies. 2008.
Venky
Harinarayan,
A.R.C.-f.
Kosmix.
Available
from:
http://www.kosmix.com.
Read Write Web- Top 10 Alternative Search Engines of 2008. Available from:
http://www.readwriteweb.com/archives/top_10_alternative_search_engi.ph
ps.
Buschmann, F., Pattern-oriented software architecture: a system of patterns.
2002: Wiley.
Buschmann, F., K. Henney, and D. Schmidt, Pattern-oriented software
architecture. 1996: Wiley New York.
Fitzpatrick, B., Distributed caching with memcached. Linux Journal, 2004.
2004(124).
Interactive, D., Memcached. 2006.
Bezos, J. A9-Amazons Seach Engine. Available from: http://www.a9.com.
Needham, C. Internet Movie Database. Available from: http://www.imdb.com.
Wong, S., W. Ziarko, and P. Wong. Generalized vector spaces model in
information retrieval. 1985: ACM New York, NY, USA.
Manber, U. and G. Myers. Suffix arrays: A new method for on-line string
searches. 1990: Society for Industrial and Applied Mathematics Philadelphia,
PA, USA.
Golub, G. and C. Reinsch, Singular value decomposition and least squares
solutions. Numerische Mathematik, 1970. 14(5): p. 403-420.
Osinski, S., J. Stefanowski, and D. Weiss. Lingo: Search results clustering
algorithm based on singular value decomposition. 2004: Springer.
26 FERRET: 360O SEARCH
FERRET: 360O SEARCH 27
22.
23.
24.
Proietti, V., MooToolsÐthe compact javascript framework.
Foundation, A., Apache JMeter.
Hansen, K., Load Testing your Applications with Apache JMeter. Java Boutique
Internet, http://javaboutique. internet. com/tutorials/JMeter/, as viewed
November, 2004.
9. Appendix
Figure 9: Home Page
FERRET: 360O SEARCH 27
28 FERRET: 360O SEARCH
Searching for ‘metallica’ in Ferret…
Figure 10: Search Query
Search (Web) results for ‘metallica’ in Ferret.
Figure 11: Search Results Page > Web Search Tab
28 FERRET: 360O SEARCH
FERRET: 360O SEARCH 29
Media results for ‘metallica’ in Ferret. Results are clustered by STC and Lingo
Algorithm
Figure 12: Search Results Page: Media Results Tab
Playing ‘Nothing Else Matters’ video.
Figure 13: Search Results Page > Media Results Tab > Media Player
FERRET: 360O SEARCH 29
30 FERRET: 360O SEARCH
Product results for ‘metallica’ in Ferret. Results are clustered by seed-list based
clustering.
Figure 14: Search Results Page > Product Search
Social results for ‘metallica’ in Ferret. No results shown since user is not logged in.
Figure 15: Search Results Page > Social Search (Not Logged in)
User logs into Ferret to see social search results.
30 FERRET: 360O SEARCH
FERRET: 360O SEARCH 31
Figure 16: Ajax Login Option
User’s friend has already searched for ‘metallica’ and his favorite ‘metallica’ pages are
displayed.
Figure 17: Search Results Page > Social Results Tab (Logged in)
FERRET: 360O SEARCH 31
32 FERRET: 360O SEARCH
Search (Web) results for ‘ipl’ in Ferret. User ‘ketan’ is logged in and clicks on a URL.
Figure 18: Search Results Page > Web Search > On Clicking a Query
User ‘ketan’ logs out and ‘praful’ logs in and searches for ipl again.
Figure 19: When a Friend Logs in!
32 FERRET: 360O SEARCH
FERRET: 360O SEARCH 33
Social results for ‘ipl’ display the URL ketan had liked when he searched for ipl.
Figure 20: The Socially Relevant Query turns up on Friends Page
Search (Web) results for ‘yoyo’ in Ferret. Dictionary Search is able to get a definition
for ‘yoyo’
Figure 21: Wikipedia, Wordnet Dictionary search, Images from yahoo and google
FERRET: 360O SEARCH 33