Modern Information Retrieval
Chapter 7: Text Operations
Ricardo Baeza-Yates
Berthier Ribeiro-Neto
Document Preprocessing





Lexical analysis of the text
Elimination of stopwords
Stemming
Selection of index terms
Construction of term categorization structures
Lexical Analysis of the Text

Word separators





space
digits
hyphens
punctuation marks
the case of the letters
Elimination of Stopwords

A list of stopwords


words that are too frequent among the documents
articles, prepositions, conjunctions, etc.

Can reduce the size of the indexing structure
considerably

Problem

Search for “to be or not to be”?
Stemming

Example





connect, connected, connecting, connection,
connections
effectiveness --> effective --> effect
picnicking --> picnic
king -\-> k
Removing strategies




affix removal: intuitive, simple
table lookup
successor variety
n-gram
Index Terms Selection

Motivation



A sentence is usually composed of nouns, pronouns,
articles, verbs, adjectives, adverbs, and connectives.
Most of the semantics is carried by the noun words.
Identification of noun groups

A noun group is a set of nouns whose syntactic distance
in the text does not exceed a predefined threshold
Thesauri


Peter Roget, 1988
Example
cowardly adj.
Ignobly lacking in courage: cowardly turncoats
Syns: chicken (slang), chicken-hearted, craven, dastardly,
faint-hearted, gutless, lily-livered, pusillanimous,
unmanly, yellow (slang), yellow-bellied (slang).

A controlled vocabulary for the indexing and
searching
The Purpose of a Thesaurus



To provide a standard vocabulary for indexing
and searching
To assist users with locating terms for proper
query formulation
To provide classified hierarchies that allow the
broadening and narrowing of the current query
request
Thesaurus Term Relationships



BT: broader
NT: narrower
RT: non-hierarchical, but related
Term Selection
Automatic Text Processing
by G. Salton, Chap 9,
Addison-Wesley, 1989.
Automatic Indexing

Indexing:


assign identifiers (index terms) to text documents.
Identifiers:


single-term vs. term phrase
controlled vs. uncontrolled vocabularies
instruction manuals, terminological schedules, …

objective vs. nonobjective text identifiers
cataloging rules define, e.g., author names, publisher names,
dates of publications, …
Two Issues

Issue 1: indexing exhaustivity



exhaustive: assign a large number of terms
nonexhaustive
Issue 2: term specificity

broad terms (generic)
cannot distinguish relevant from nonrelevant documents

narrow terms (specific)
retrieve relatively fewer documents, but most of them are
relevant
Parameters of
retrieval effectiveness

Recall
Number of relevant items retrieved
Total number of relevant items in collection
Precision
Number of relevant items retrieved
P
Total number of items retrieved
Goal
high recall and high precision
R


b
Nonrelevant
Items
c
a
Recall 
a +d
a
Retrieved
Part
Relevant
Items
d
a
Precision 
a+b
A Joint Measure

F-score




(  2  1)  P  R
F
2  P  R
 is a parameter that encode the importance of recall
and procedure.
=1: equal weight
<1: precision is more important
>1: recall is more important
Choices of Recall and Precision



Both recall and precision vary from 0 to 1.
Particular choices of indexing and search policies
have produced variations in performance ranging
from 0.8 precision and 0.2 recall to 0.1 precision
and 0.8 recall.
In many circumstance, both the recall and the
precision varying between 0.5 and 0.6 are more
satisfactory for the average users.
Term-Frequency Consideration


Function words
 for example, "and", "or", "of", "but", …
 the frequencies of these words are high in all texts
Content words
 words that actually relate to document content
 varying frequencies in the different texts of a collect
 indicate term importance for content
A Frequency-Based Indexing Method



Eliminate common function words from the document
texts by consulting a special dictionary, or stop list,
containing a list of high frequency function words.
Compute the term frequency tfij for all remaining terms Tj
in each document Di, specifying the number of occurrences
of Tj in Di.
Choose a threshold frequency T, and assign to each
document Di all term Tj for which tfij > T.
Inverse Document Frequency

Inverse Document Frequency (IDF) for term Tj
N
idf j  log
df j
where dfj (document frequency of term Tj) is the
number of documents in which Tj occurs.
 fulfil both the recall and the precision
 occur frequently in individual documents but rarely in
the remainder of the collection
TFxIDF

Weight wij of a term Tj in a document di
N
wij  tf ij  log
df j



Eliminating common function words
Computing the value of wij for each term Tj in each
document Di
Assigning to the documents of a collection all terms with
sufficiently high (tf x idf) factors
Term-discrimination Value

Useful index terms


Distinguish the documents of a collection from
each other
Document Space


Two documents are assigned very similar term
sets, when the corresponding points in
document configuration appear close together
When a high-frequency term without
discrimination is assigned, it will increase the
document space density
A Virtual Document Space
Original State
After Assignment of
good discriminator
After Assignment of
poor discriminator
Good Term Assignment

When a term is assigned to the documents of a
collection, the few objects to which the term is
assigned will be distinguished from the rest of
the collection.

This should increase the average distance
between the objects in the collection and hence
produce a document space less dense than
before.
Poor Term Assignment

A high frequency term is assigned that does not
discriminate between the objects of a collection.
Its assignment will render the document more
similar.

This is reflected in an increase in document
space density.
Term Discrimination Value

Definition
where
dvj = Q - Qj
Q and Qj are space densities before and
after the assignments of term Tj.
N
N
1
Q
sim( Di , Dk )


N ( N  1) i 1 k 1
ik

dvj>0, Tj is a good term;
dvj<0, Tj is a poor term.
Variations of Term-Discrimination Value
with Document Frequency
Document
Frequency
N
Low frequency
Medium frequency
High frequency
dvj=0
dvj>0
dvj<0
TFij x dvj

wij = tfij x dvj

compared with wij  tf ij  log


N
df j
N
df j
: decrease steadily with increasing document
frequency
dvj: increase from zero to positive as the document
frequency of the term increase,
decrease shapely as the document frequency
becomes still larger.
Document Centroid


Issue: efficiency problem
N(N-1) pairwise similarities
Document centroid C = (c1, c2, c3, ..., ct)
cj 

N
w
i 1
ij
where wij is the j-th term in document i.
Space density
1
Q
N
N
 sim (C , D )
i 1
i
Probabilistic Term Weighting


Goal
Explicit distinctions between occurrences of
terms in relevant and nonrelevant documents of
a collection
Definition
Given a user query q, and the ideal answer set of the
relevant documents

From decision theory, the best ranking algorithm
for a document D
Pr( D | rel )
Pr( rel )
g ( D )  log
 log
Pr( D | nonrel )
Pr( nonrel )
Probabilistic Term Weighting

Pr(rel), Pr(nonrel):
document’s a priori probabilities of relevance
and nonrelevance

Pr(D|rel), Pr(D|nonrel):
occurrence probabilities of document D in the
relevant and nonrelevant document sets
Assumptions

Terms occur independently in documents
t
Pr( D | rel )   Pr( xi | rel )
i 1
t
Pr( D | nonrel )   Pr( xi | nonrel )
i 1
Derivation Process
g ( D )  log
Pr( D | rel )
Pr( rel )
 log
Pr( D | nonrel )
Pr( nonrel )
t
 log
 Pr( x |rel )
t
 Pr( x |nonrel )
i 1

t
 log
i 1
i
i 1
 constants
i
Pr( xi |rel )
 constants
Pr( xi |nonrel )
For a specific document D

Given a document D=(d1, d2, …, dt)
Pr( xi  di |rel )
g ( D)   log
 constants
Pr( xi  di |nonrel )
i 1
t

Assume di is either 0 (absent) or 1 (present).
Pr(xi=1|rel) = pi
Pr(xi=0|rel) = 1-pi
Pr(xi=1|nonrel) = qi
Pr(xi=0|nonrel) = 1-qi
Pr( xi  di |rel )  pi (1  pi )
di
1  di
Pr( xi  di |nonrel )  qi (1  qi )
di
1  di
g ( D) 
t
 log
i 1

t
 log
i 1
Pr( xi  di | rel )
 constants
Pr( xi  di |nonrel )
1di
p d (1 p )
1d
d
q (1 q )
i
i
i
i
i
i
 constants
i
(1  p )
p
(
1

q
)
  log
 constants
d
d
q (1 p ) (1  q )
t
i 1
di
di
i
i
i
i
i
i
i
i
(1 p )
(
p
(
1

q
))
  log
 constants
d
(q (1 p )) (1 q )
t
i 1
di
i
i
i
i
i
i
i
Term Relevance Weight
t
1  pi
p (1  qi )
g ( D)   log
  di log i
 constants
i 1
1  qi i 1
qi (1  pi )
t
pj (1  qj )
trj  log
qj (1  pj )
Issue

How to compute pj and qj ?
pj = rj / R
qj = (dfj-rj)/(N-R)


R: the total number of relevant documents
N: the total number of documents
Estimation of Term-Relevance

The occurrence probability of a term in the nonrelevant
documents qj is approximated by the occurrence
probability of the term in the entire document collection
qj = dfj / N

The occurrence probabilities of the terms in the small
number of relevant documents is equal by using a
constant value pj = 0.5 for all j.
Comparison
pj (1  qj )
trj  log
 log
qj (1  pj )
0.5 * (1 
df j
N
 log
df j
N
)
* 0.5
( N  df j )
df j
When N is sufficiently large, N-dfj  N,
trj  log
( N  df j )
df j

log
N
df j
= idfj
Estimation of Term-Relevance

Estimate the number of relevant documents rj in the
collection that contain term Tj as a function of the known
document frequency tfj of the term Tj.
pj = rj / R
qj = (dfj-rj)/(N-R)
R: an estimate of the total number of relevant documents
in the collection.
Summary

Inverse document frequency, idfj


Term discrimination value, dvj


tfij*dvj
Probabilistic term weighting trj


tfij*idfj (TFxIDF)
tfij*trj
Global properties of terms in a document
collection