Investigating Automated Assistance and Implicit Feedback for Searching
Systems
Bernard J. Jansen
School of Information Sciences and Technology
The Pennsylvania State University
329F IST Building, University Park PA 16802
Email: jjansen@ist.psu.edu
Michael D. McNeese
School of Information Sciences and Technology
The Pennsylvania State University
307IST Building, University Park PA 16802
Email: mmcneese@ist.psu.edu
ABSTRACT
Information retrieval systems offering personalized
automated assistance have the potential to improve
the searching process. There has been much work in
this area for several years on a variety of systems.
However, there has been little empirical evaluation of
automated assistance to determine if it is of real
benefit for searchers. We report the results of
empirical evaluation investigate how searchers use
implicit feedback and automated assistance during the
searching process. Results from the empirical
evaluation indicate that searchers typically use
multiple implicit feedback actions, usually bookmark –
copy.
The most commonly utilized automated
assistance was for query refinement, notable the use
of the thesaurus. We discuss the implications for Web
systems and future research.
1. Introduction
There has been considerable research into automated
assistance in order to address some of the issues users have
when interacting with information retrieval (IR) systems
(Jansen, Spink, Bateman, and Saracevic, 1998; Yee, 1991).
The need for automated assistance is especially acute with
Web searching systems, as research shows that users of
Web search engines have difficulty successfully
implementing query syntax (Jansen, Spink, Bateman, and
Saracevic, 1998), and the performance of major Web
search engines (i.e., number of relevance documents
retrieved within the first ten) is approximately 50%
(Eastman and Jansen, 2003; Jansen and Spink, 2003).
Automated assistance systems usually attempt to assist the
user during the search process by either executing search
tactics for or offering assistance to the user in order to help
locate relevant information.
We define automated
assistance as any expressions, actions or responses by an IR
system with the aim of improving the information searching
experience for the user as measured by some external
metric. These external metrics are usually relevance related
ones, such as precision (Korfhage, 1997).
However, there has been little empirical evaluation of
automated assistance. Therefore, it is not clear whether or if
automated assistance is beneficial to users during the search
process. Is automated assistance helpful? If so, what type
of assistance? Is it helpful for certain types of searches?
When in the search process do searchers desire assistance?
The research results presented in this article address a
portion of these issues. We examine whether automated
assistance is beneficial to users during the search process.
We begin with a review of literature concerning Web and
IR systems offering automated assistance. We then provide
a short description of the automated assistance application
we developed and utilized in the user study. Next, we
discuss the empirical test we conducted to evaluate the
effect of automated assistance on system performance. We
present the results of our experiment and the implications
for Web IR system design and then discuss directions for
future research.
2. Literature Review
Many searchers have difficulty effectively utilizing IR
systems. These issues occur across the spectrum of IR
systems, including online public access catalogs (Peters,
1993) and Web systems (Jansen, Spink, and Saracevic,
2000). For Web systems, issues include lack of query
syntax (e.g., AND, MUST APPEAR, etc.), improper use of
query syntax, retrieving too many results, retrieving zero
results (Silverstein, Henzinger, Marais, and Moricz, 1999;
Yee, 1991), among many others.
Although there has been considerable research and
development on advanced searching features in Web search
engines, users generally do not use these features (Spink,
Jansen, Wolfram, and Saracevic, 2002) and have problems
with them when they do (Jansen, Spink, and Saracevic,
1998). In order to assist the user with the searching issues
and to better utilize advanced searching methods, there
have been efforts to develop IR systems that automate
various aspects of the searching process. We refer
collectively to these as automated assistance systems.
Meadow and fellow researchers (1982a; 1982b) present one
of the first design and analysis of a system offering
contextual help. Chen and Dhar (1991) developed a system
for key word selection and thesaurus browsing. Using the
agent paradigm, researchers have explored intelligent IR
systems for Web, including Alexa (1999) and Letizia
(1995) to aid in the browsing process. ResearchIndex
(Lawrence, Giles, and Bollacker, 1999) utilizes an agent
paradigm to recommend articles based on a user profile.
From this brief literature review, it is apparent that there has
been considerable work into developing automated
assistance IR systems. There has been much less research
into evaluating: how do searchers utilize these systems
during the searching process. We conducted a user study
utilizing an automated assistance application that we
developed in order to investigate these questions.
3. Research Evaluation
We are interested in examining how users interact with
automated assistance with the aim of improving system
performance during a session. A session is one episode of a
searcher using an IR system during which a series of
interactions occur between the searcher and system. In this
research, we specifically focus on implicit feedback actions
and use of automated assistance.
We next provide a short description of the automated
assistance techniques we employed, and the component we
developed.
4. System Development
We designed and developed a client-side software
component to integrate a suite of automated assistance
features with a range of existing Web-based IR systems.
Our system development goal was that the system would
rely on implicit feedback, gleaning information solely from
normal user – system interactions during the search process.
The automated assistance component uses these interactions
to determine what assistance to provide.
4.1 System Design
The system builds a model of user – system interactions
using action - object pairs (a, o) (Jansen, 2003). A single
action - object (a, o) pair captures an instance of a user –
system interaction. An a is some user initiated interaction
with the system. An o is the receiver of that action.
A series of (a, o) pairs models a searcher’s chain of
interactions during the session. The system can use these
(a, o) pairs to determine the user’s information need and
provide appropriate assistance by associating certain
actions with specific types of assistance.
Using (a, o) pairs has several advantages compared to other
methods of gathering information from a user during a
session.
The query is usually the only source of
information from the user during traditional IR system
interaction. Other techniques (e.g., answering questions,
completing profiles, judging relevance judgments) require
the user to take additional actions beyond those typical of
user interactions during an online search.
Using (a, o) pairs, the user’s query is not the sole
representation of the information need. The system gathers
additional information from other user actions, such as
bookmarking, printing, emailing, without requiring
additional user actions. As such, the (a, o) pair
methodology is ideally suited for the implicit feedback that
one expects on the Web and other client – server
architectures.
4.2 System Overview
The system currently monitors the searcher’s interactions
with the system, tracking actions of bookmark, copy, print,
save, submit, and view results. Previous research has
identified these actions as implicit indications of possible
document relevance (Oard and Kim, 2001). There are
currently three objects that the system recognizes, which are
documents, passages from documents, and queries.
The system monitors the user for one of the six actions, via
a browser wrapper. When the system detects a valid action
(i.e., (a, o) pair), it records the action and the specific
object receiving the action. For example, if a searcher was
viewing this_Web_Page and saved it, the system would
record this as (save this_Web_Page). The system then
offers appropriate search assistance to the user based on the
particular action and the system’s analysis of the object.
The more (a, o) pairs the system records, the more complex
the model of the information need.
4.3 Automated Assistance Offered
We currently focus on five user – system interaction issues
and corresponding system assistance, which are:
 Managing Results: Searchers have trouble managing the
number of results. Using the (submit query) pair and the
number of results, the automated assistance application
provides suggestions to improve the query in order to either
increase or decrease the number of results. If the number of
results is more than thirty, the application provides
suggestions to restrict the query. If the number of results is
less than ten, the system provides advice on ways to
broaden the query. We chose thirty and ten results as the
boundary conditions based on research studies showing that
approximately 80% of Web searchers never view more than
twenty results (Jansen, Spink, and Saracevic, 2000).
However, one can adjust the result thresholds to any
targeted user population.
 Query Refinement: In general, most Web searchers do
not refine their query, even though there may be other terms
that relate directly to their information need. With a
(submit query) pair and a thesaurus, the system analyzes
each query term and suggests synonyms of the query terms.
The system uses the Microsoft Office thesaurus, but the
application can utilize any online thesaurus via an
application program interface (API).
 Query Reformulation: Some search engines, such as
AltaVista (Anick, 2003) offer query reformulation based on
similar queries from previous users. We incorporated this
feature into our automated assistance system. With a
(submit query) pair, the system accesses a database of all
previous queries and locates queries within the database
containing similar terms. The system displays the top three
similar queries based on number of previous submissions.
 Relevance Feedback: Relevance feedback has been
shown to be an effective search tool (Harman, 1992);
however, Web searchers seldom utilize it when offered. In
the studies on the use of relevance feedback on the Web
(Jansen, Spink, and Saracevic, 1999), Web searchers
utilized relevance feedback less than 10% of the time. In
this study, we automate the process using term relevance
feedback. When the (a, o) pairs of (bookmark document),
(print document), (save document), or (copy passage)
occur, the system implements a version of relevance
feedback using terms from the document or passage object.
The system provides suggested terms from the document
that the user may want to implement in a follow-on query.
 Spelling: Searchers routinely misspell terms in queries,
which will usually drastically reduce the number of results
retrieved. A (submit query) pair alerts the automated
assistance application to check for spelling errors. The
system separates the query into terms, checking each term
using an online dictionary. The system’s online dictionary
is Microsoft Office Dictionary, although it can access any
online dictionary via the appropriate application program
interface.
4.4 System Overview
The automated assistance system has five major modules,
which are:
The Query Terms module uses the (submit query) pairs
during a session. For each (submit query) pair, the module
parses each query into separate terms, removing query
operators such as the MUST APPEAR, MUST NOT
APPEAR and PHRASE operators. The module then
accesses the Microsoft Office dictionary and thesaurus,
sending each term to the process. If there are possible
misspellings, the module records the suggested corrections.
The Relevance Feedback module uses (bookmark
document), (print document), (save document), or (copy
passage) pairs. When one of these pairs occurs, the module
removes all stop words from the object using a standard
stop word list (Fox, 1990) and all terms from previous
queries within this session. The system then randomly
selects terms remaining from the results listing abstract that
was displayed or from the passage of copied text,
depending on the (a, o) pair.
The Reformulation module uses the (submit query) pair to
provide suggested queries based on submitted queries of
previous users. When a (submit query) pair occurs, the
modules queries a database of all previous queries that
contain all the query terms, attempting to find at least three
queries to present to the user. If the database contains three
queries that contain all the terms, the module selects these
queries, unless one is identical to the current query. If the
database contains more than three, the module selects the
top three based on frequency of occurrence. If the database
contains less than three, the module queries the database for
queries that contain at least one of the terms, beginning with
the first term in the current query. The module repeats the
process until it has at least three queries to present to the
searcher. One can alter the number of queries the module
returns.
The Refinement module uses a (submit query) pair and the
number of results retrieved to suggest alternate queries to
either tighten or loose the retrieval function. If the system
retrieves more than thirty results, the module first checks
the query for the AND, MUST APPEAR, or PHRASE
operators.
If the module detects no operators, it
reformulates the queries using the existing terms and the
appropriate AND, MUST APPEAR, or PHRASE operators.
If the module detects AND or MUST APPEAR operators in
the query, the module refines the query with the PHRASE
operator. If the module detects PHRASE operators in the
query, the module does no refinement to tighten the query.
If the system retrieves less than twenty results, the module
performs a similar process to broaden the query by
removing of AND, MUST APPEAR and PHRASE and
replacement them with the OR operator.
The Tracking module monitors user interactions with the
browser, including all interactions with the browser tool
bars, along with the object of the interaction. The Tracking
module then formulates the (a o) pair, passing the pair to
the appropriate module.
The Assistance module receives the automated assistance
from the Query Terms, Relevance Feedback, Reformulation
and Refinement modules, presenting the automated
assistance to the searcher via an ASP script, which the
browser loads with the Web document. For the spelling
assistance, each term is presented, followed by a list of
possible correct spellings. The same format is followed for
synonyms. Queries with spelling corrections, similar
queries, relevance feedback terms, and re-structured queries
are presented as clicked text (i.e., the searcher can click on
these to generate a new search). Figure 1 presents a
complete system diagram.
Browser Interface
Assistance Modules
Assistance
Assistance
Query Terms
Table 1. Demographic of Subjects
Age
Mean
St Dev Mode
21.4
1.96
21
Gender Male
Female
35
5
Experience with Search Engines
<1
1-3
3 -5
>5
Year
Years
Years
Years
0
2
12
26
Self-Reported Skill Rating
1
2
3
4
(Novice)
0
2
Google
Yahoo!
38
7
Search Engine
Use (Daily)
Search Engine
Use (Weekly)
7
23
Alta
Others
Vista
2
4
Mean Range =
4.6 – 5.5
Mean Range =
30.5 – 33.1
Total
40
40
5
(Expert)
8
40
51
St Dev Range =
3.6 – 4.1
St Dev Range =
27.2 – 28.6
Relevance Feedback
Reformulation
Refinement
Actions and Objects
other information.
Table 1 presents the pertinent
demographic information.
Tracking
Figure 1: Automated Assistance Modules and Information
Flow with Interface
5. User Study
In the following sections, we outline our empirical
evaluation.
5.1 Study Design
The backend IR system utilized for the empirical study was
Microsoft Internet Information Service (IIS).
The IIS
system is running on an IBM-compatible platform using the
Windows XP operating system and Microsoft Internet
Explorer as the system interface. For the automated
assistance system, we integrated the automated assistance
application via a wrapper to the Internet Explorer browser.
The subjects for the evaluation were 40 college students (35
males and 5 females) attending a major U.S. university. All
were familiar with the use of Web search engines. We gave
them no additional training. We did administer a preevaluation survey to collect a variety of demographic and
The average age of the subjects was 21 years. Of the
subjects, twenty-six reported more than five years
experience using Web search engines. The subjects selfrated their searching skills. Of the forty subjects, thirty-one
rated themselves as expert or near expert. None rated
themselves as novice. We also asked which search engines
they used frequently. The subjects could list more than one.
The most frequently reported search engine was Google.
There were four search engines listed once (AOL, MSN,
Ask.com, Meta-crawler). Reported frequency of search
engine usage per day (subjects could report a range)
averaged 4.6 to 5.5 occurrences. Weekly search engine
usage averaged 30.5 to 33.1 occurrences.
We utilized the Text REtrieval Conference (TREC)
volumes number 4 and 5 as the document collection for the
evaluation. The document collection is more than 2GB in
size, containing approximately 550,000 documents.
Each TREC collection comes with a set of topics for which
there are relevant documents in the collection. We loaded a
list of topics in a spreadsheet and coded a script to select
six topics at random. The topics selected, and the ones that
we utilized are:

Number 304: Endangered Species (Mammals)

Number 311: Industrial Espionage

Number 323: Literary/Journalistic Plagiarism

Number 335: Adoptive Biological Parents

Number343: Police Deaths

Number350: Health and Computer Terminals
There were 904 TREC-identified relevant documents for
the six topics, which is 0.2% of the collection.
analysis, we instructed the subjects to think out loud during
the searching process.
All forty subjects utilized the full fifteen minutes on the
system for a total of 600 minutes of video for analysis.
During the sessions, we used the thinking-aloud protocol
(Dumas and Redish, 1993) where the subject verbalization
occurs in conjunction with a task. We used these utterances
to help clarify user interactions with the system recorded in
the TL.
During the search sessions, we also recorded extensive lab
notes on subject actions, notable utterances, and other
searching behaviors.
After the search session, each
searcher completed a subjective evaluation of the
automated assistance (Chin, Diehl, and Norman, 1988).
The combination of the pre-surveys, protocol analysis, TL,
and user evaluations provided a robust data source to
conduct our analysis.
6. Results
Figure 2: Automated Assistance after Submitting a Query
At the start of the evaluation, we provided each of the
subjects a short statement instructing them to search on a
given topic in order to prepare a report, which is in line
with the definition of relevance judgments for the TREC
documents. The subjects had fifteen minutes to find as
many relevant documents as possible. We determined the
length of the search session based on reported measures of
the “typical” Web search session (Jansen and Spink, 2003).
When the subjects utilized the automated assistance system,
we notified them that the system contained an automatic
feature to assist them while they were searching. When the
system had searching advice to offer, an assistance button
would appear on the browser. We showed them a screen
capture of the assistance button. We instructed them that
they could access the assistance by clicking the button, or
they could ignore the offer of assistance with no detrimental
effect on the system. The system offered assistance
whenever assistance was available. Figure 2 shows the
Internet Explorer browser with the automated assistance
displayed.
For the searching sessions, we then gave each of the
subjects one of the six search topics, read the one paragraph
explanation provided with the TREC collection, and then
afforded the written explanation to them. We asked the
subjects to search as when they normally conducting online
research, taking whatever actions they usually take when
locating documents of interest online. We rotated the 6
topics among the searchers.
The users were video taped during the searching process
and a transaction log (TL) recorded user – system
interactions. In order to add further robustness to the
In the following section, we present the results of the
empirical evaluation, presenting results on use of implicit
feedback, and use of automated assistance.
During the evaluation, ten subjects did not view or
implement the automated assistance. We eliminated the
results of these ten users from the evaluation in order to not
to skew the evaluation. We instead report data on the thirty
subjects who interacted with the automated assistance
feature.
6.1 Use of Implicit Feedback
Given the client – server architecture of the Web, there is a
natural reliance on implicit feedback (Oard and Kim, 2001)
in the identification of relevant documents for evaluation
purposes. In Table 3, we present the occurrences of the
implicit actions that our automated assistance application
uses as surrogates for the identification of relevant
documents.
Table 3. Use of Implicit Relevance Actions
System
System
Relevance
Without
With
Total
Action
Assistance Assistance
186
84
102
Bookmark
87
38
49
Copy
%
64.4%
30.1%
Print
6
10
16
5.5%
Save
4
0
4
1.4%
130
159
289
100.0
%
Total
For each of these actions, we verified from either lab notes
or video analysis whether or not these actions implied the
location of a relevant document. In all cases, these actions
denoted the location of relevant information.
Bookmark (64%) and Copy (30%) were the most common
implicit actions (94%). What may be surprising though is
that not all users relied on only one implicit action. When
we analyzed what specific implicit actions individual
searchers utilized, ten subjects used only Bookmark, six
used only Copy, one only used Print, and one only used
Save. However, ten users utilized a combined of Bookmark
and Copy and three used a combination of Copy and Print.
What this implies is that there may be inherent
characteristics of the information objects that lend
themselves to specific relevance actions by users
6.2 Use of Automated Assistance
In Table 4 we present the results on the use of the
automated assistance.
Table 4. Use of Automated Assistance Types
Assistance Type
Occurrences
%
Query Refinement
Synonyms
22
29%
Query
Reformulation
Similar Queries
13
17%
Spelling
Spelling
13
17%
Managing Results
AND
PHRASE
REMOVE
PHRASE
Total
28
16
11
37%
21%
14%
The thirty users accepted the offer of automated assistance
(i.e., viewed the assistance) 108 times. They implemented
some form of the assistance 76 times for a 70.4%
acceptance rate. We took this high acceptance rate to be an
indication that the users viewed the assistance as beneficial.
Of the thirty users, three viewed the assistance at least once
during the session but did not implement any assistance.
Twenty-seven users viewed and implemented the assistance
at least once. The mean number of viewings per user was
3.6 (sd=2.4), and the mean number of implementations was
2.5 (sd=1.8).
We designed the system to offer help whenever there was
assistance, meaning after every query and every implicit
relevance action, the assistance button would appear on the
browser. There were 233 total queries submitted by the
thirty users on the automated assistance system
(mean=7.77, sd=3.68) and 159 implicit relevance actions.
Therefore, the assistance button appeared 392 times, of
which users viewed the assistance 108 times (28%). We
thought this percentage low, assuming that the novelty
factor alone would entice users at least to view the
assistance.
From analysis of comments during the session and the postsearch survey, there appears to be least two major reasons
for not viewing the assistance. There is a sizeable portion
of searchers who just like to attempt things on their own
first and seek assistance only when needed. If they believe
they are doing okay, they see no need for system assistance.
The second major reason concerns interface design issues.
We had attempted to make the assistance button very nonintrusive. However, it appears that there must be some
intrusiveness into order to attract the attention of searchers.
7. Discussion
1
76
1%
100%
From Table 4, we see that the most commonly implemented
assistance was Managing Results (37%). Of these, all but
one was assistance to decrease the number of results (i.e.
the use of the AND and PHRASE operators). The subjects
also utilized the other forms of assistance quite frequently,
with Query Refinement also being commonly used
assistance. The subjects did not use the Spelling feature as
much as we expected, since research reports that Web
searchers frequently misspell terms (Jansen, Spink, and
Saracevic, 1998). However, we did furnish the searchers
with the written TREC topic description for their search.
Most searchers referred to this sheet for formulating some
of their initial queries.
Returning to our research question (How do searchers
utilize these systems during the searching process?),
searchers interact with automated assistance system in some
interesting and occasionally unexpected ways.
There are certain implicit relevance actions that searchers
prefer, primarily Bookmark, which was the action of choice
by twenty of the thirty users, either alone (10) or in
combination with the action Copy (also 10). Copy was also
a top choice of the searchers, again either alone (6) or in
combination with Bookmark (10). Why searchers make
these choices could lead to improvements in system support
during the searching process. Are these choices solely user
specific or are there some characteristics of the information
objects that lead themselves to certain searcher relevance
actions?
Concerning the use of automated assistance, users most
commonly implemented assistance to Manage Results, and
overwhelmingly these selections were actions to tighten the
query, thereby attempting to reduce the number of results.
This would indicate that users consider result management,
particularly precision, a key issue for Web searching.
Searchers also used Query Refinement, specifically the use
of synonyms, extensively (29% of all implementations).
Given the low percentage of relevant documents in the
collection (0.2%), many subjects found the searching
difficult, expressing their frustration with the topic
difficulty during the search. In these searching situations,
term selection is extremely important, perhaps even more
important than selection of query operators (Eastman and
Jansen, 2003).
In general though, searchers were inclined not to view
assistance (72%) when it was offered. Searchers have a
tendency to want to search on their own. Several searchers
did not even notice the search button until well into the
search session, even though the button was placed just
below the search text box. However, we did attempt to
make the assistance button non-intrusive. Given the
cognitive load that searching requires, it appears that some
intrusiveness is necessary to gain the attention of the
searcher.
Of the thirty searchers who viewed the assistance, twentyseven (90%) implemented the assistance. Of the 108 times
that searchers viewed the assistance, they implemented it 76
times (70%). This indicates that once they did view system
assistance, they deemed the assistance beneficial. Again,
this may be a justification for more direct and target
intrusion into the searching process. This is the technique
that Google and AltaVista employ with their Did you mean
spelling assistance.
8. Conclusion and Future Research
The results of the research conducted so far are very
promising. Results indicate that searchers interact with
these systems in predictable ways, which might be utilized
to improve the design of future systems.
We are continuing our performance evaluation of the
automated assistance system at the query and session level
of analysis using the TREC identified relevant documents.
We also plan to conduct a temporal analysis to see if there
are patterns of interaction that would predict when a
searcher desires assistance from the system. By detecting
the patterns of user – system interaction, designers can
tailor IR systems to provide targeted assistance at the
proper temporal states when the user is willing to view or
implement the assistance. With this information, IR
systems may more effectively assist searchers in locating
the information they desire.
9. References
Anick, P. (2003). Using Terminological Feedback for Web Search
Refinement - a Log-Based Study. In Proceedings of the
Twenty-Sixth Annual International ACM SIGIR Conference
on RE search and Everyone in Information Retrieval, pp. 8895. Toronto, Canada. 28 July - 1 August.
Chen, H. C., & Dhar, V. (1991). Cognitive Process as a Basis for
Intelligent Retrieval-Systems Design. Information
Processing and Management, 27(5), 405-432.
Chin, J. P., Diehl, V. A., & Norman, K. L. (1988). Development
of an Instrument Measuring User Satisfaction of the HumanComputer Interface. In Proceedings of the ACM
International Conference on Computer Human Interaction,
pp. 213-218. Washington, DC, USA. 15-19 May.
Dumas, J. S., & Redish, J. (1993). A Practical Guide to Usability
Testing. Norwood, NJ: Ablex.
Eastman, C. M., & Jansen, B. J. (2003). Coverage, Ranking, and
Relevance: A Study of the Impact of Query Operators on
Search Engine Results. ACM Transactions on Information
Systems, 21(4), 383 - 411.
Fox, C. (1990). A Stop List for General Text. ACM SIGIR Forum,
24(2), 19 - 21.
Harman, D. (1992). Relevance Feedback Revisited. In
Proceedings of the 15th Annual International ACM SIGIR
Conference on Research and Development in Information
Retrieval, pp. 1-10. Copenhagen, Denmark. 21-24 June.
Jansen, B. J. (2003). Designing Automated Help Using Searcher
System Dialogues. In Proceedings of the 2003 IEEE
International Conference on Systems, Man & Cybernetics,
pp. 10 - 16. Washington, D.C., USA. 5-8 October.
Jansen, B. J., & Spink, A. (2003). An Analysis of Web
Information Seeking and Use: Documents Retrieved Versus
Documents Viewed. In Proceedings of the 4th International
Conference on Internet Computing, pp. 65 - 69. Las Vegas,
Nevada. 23 - 26 June.
Jansen, B. J., Spink, A., Bateman, J., & Saracevic, T. (1998). Real
Life Information Retrieval: A Study of User Queries on the
Web. SIGIR Forum, 32(1), 5-17.
Jansen, B. J., Spink, A., & Saracevic, T. (1998). Failure Analysis
in Query Construction: Data and Analysis from a Large
Sample of Web Queries. In Proceedings of the 3rd ACM
Conference on Digital Libraries, pp. 289-290. Pittsburgh,
PA.
Jansen, B. J., Spink, A., & Saracevic, T. (1999). The Use of
Relevance Feedback on the Web: Implications for Web Ir
System Design. In Proceedings of the 1999 World
Conference of the World Wide Web and Internet, pp.
Honolulu, HI.
Jansen, B. J., Spink, A., & Saracevic, T. (2000). Real Life, Real
Users, and Real Needs: A Study and Analysis of User
Queries on the Web. Information Processing and
Management, 36(2), 207-227.
Kahle, B. (1999). Alexa Internet Press Releases [Web site]. Alexa
Internet. Retrieved 12 February, 2000, from the World Wide
Web: http://www.alexa.com/company/recent_articles.html.
Korfhage, R. (1997). Information Storage and Retrieval. New
York, NY: Wiley.
Lawrence, S., Giles, C. L., & Bollacker, K. (1999). Digital
Libraries and Autonomous Citation Indexing. IEEE
Computer, 32(6), 67-71.
Lieberman, H. (1995). Letizia: An Agent That Assists Web
Browsing. In Proceedings of the 14th International Joint
Conference on Artificial Intelligence, pp. 924 -929.
Montreal, Quebec, Canada. August.
Meadow, C. T. (1982a). A Computer Intermediary for Interactive
Database Searching I: Design. Journal of the American
Society for Information Science, 33, 325-332.
Meadow, C. T. (1982b). A Computer Intermediary for Interactive
Database Searching Ii: Evaluation. Journal of the American
Society for Information Science, 33, 357-364.
Oard, D., & Kim, J. (2001). Modeling Information Content Using
Observable Behavior. In Proceedings of the 64th Annual
Meeting of the American Society for Information Science and
Technology, pp. 38-45. Washington, D.C., USA. November.
Peters, T. (1993). The History and Development of Transaction
Log Analysis. Library Hi Tech, 42(11), 41-66.
Silverstein, C., Henzinger, M., Marais, H., & Moricz, M. (1999).
Analysis of a Very Large Web Search Engine Query Log.
SIGIR Forum, 33(1), 6-12.
Spink, A., Jansen, B. J., Wolfram, D., & Saracevic, T. (2002).
From E-Sex to E-Commerce: Web Search Changes. IEEE
Computer, 35(3), 107-111.
Yee, M. (1991). System Design and Cataloging Meet the User:
User Interfaces to Online Public Access Catalogs. Journal of
the American Society for Information Science, 42(2), 78-98.