Major goal:
Find the best possible representation
Minor goals:
Improve storage and speed
First: Need to transform sequence of bytes into sequence of characters
Problems involved:
Encoding, language, …
Different formats (some might be proprietary)
Content vs. structure information etc.

Goal : Transform stream of characters into stream of words / terms / tokens
Terms should be consistent and ‘reasonable’
Lots of potential problems here, e.g.
Apostrophes
England’s: England s – Englands?
haven’t: haven t – havent – have not?
user’s vs. users’ vs. users’s?
Hyphenation state-of-the-art vs. state of the art co-authors vs. coauthors vs. co authors

Special characters
Punctuation (Normally removed. But what about www.uni-freiburg.de?)
Names: C++, A* Algorithm, M*A*S*H, …
Use just white spaces for segmentation?
San Francisco? Los Angeles?
Nuclear power plant?
Language specific issues, e.g.
German: ‘Komposita’, e.g. Atomkraftwerk
Case sensitivity?
Conclusion : Implementation is easy, but what and how is critical (often heuristics, etc.)

Goal : Remove words with no relevance for any search in order to reduce disk space, speed up search time, and improve result
Problems:
What about ‘ To be or not to be ’?
What about phrases, e.g. ‘ The name of the rose ’ (book/movie vs. flower names)?
Usability?
Implementation :
Often via manually generated stop words lists
(depending on the task and data, e.g. small vs. big, general vs. topic specific, etc.)

Thesaurus : A data structure composed of (1) a pre-compiled list of important words in a given domain of knowledge and (2) for each word in the list, a list of related (synonym) words.
(Glossary, page 453, Baeza-Yates & Ribeiro-Neto [1])
Implementation :
Either manually generated (hard & expensive) or automatic (e.g. with statistical methods)
Usefulness for retrieval is controversial
Topic-specific thesauri seem to work well, but limited usage with large, dynamic data base(?)

Usefulness for indexing unclear, but it can be useful in another way:

Usefulness for indexing unclear, but it can be useful in another way:

Usefulness for indexing unclear, but it can be useful in another way:

Stemming : A technique for reducing words to their grammatical roots.
(Glossary, page 452, Baeza-Yates & Ribeiro-Neto [1])
Motivation: reduce different cases, singular and plural, etc. to the root (stem) of the word
Example: connected, connecting, connection, connections -> connect
But how about: operation research, operating system, operative dentistry -> opera???

Table lookup
Generation is complex
Final tables are often incomplete
Affix removal
Suffix vs. prefix (e.g. mega-volt)
Doesn’t always work, esp. not in German
Successor variety stemming
More complex than suffix removal
Uses (e.g.) linguistic approaches and techniques from morphology
N-grams
General clustering approach which can also be used for stemming

Special algorithm for the English language based on suffix removal
Basic idea: Successive application of several rules for suffix removal (5 phases of condition and action rules)
Example: Remove plural ‘s’ and ‘sses’
Rules: sses -> ss, s -> NIL (obey order!)
Advantage: Easy algorithm with good results
Disadvantage: Not always correct, e.g.
Same root for police – policy, execute –executive, …
Different root for european – europe, search – searcher, …

Conventions for the pseudo language used to describe the
- a consonant variable is represented by the symbol C which is used to refer to any letter other than a,e,i,o,u and other than the letter y preceded by a consonant;
- a vowel variable is represented by the symbol V which is used to refer to any letter which is not a consonant;
- a generic letter (consonant or vowel) is represented by the symbol L;
- the symbol NIL is used to refer to an empty string (i.e., one with no letters);
- combinations of C, V, and L are used to define patterns;
- the symbol * is used to refer to zero or more repetitions of a given pattern;
- the symbol + is used to refer to one or more repetitions of a given pattern;
- matched parenthesis are used to subordinate a sequence of variables to the operators
* and +;
- a generic pattern is a combination of symbols, matched parenthesis, and the operators * and +;
- the substitution rules are treated as commands which are separated by a semicolon punctuation mark;
- the substitution rules are applied to the suffixes in the current word;
- a conditional if statement is expressed as ``if (pattern) rule'' and the rule is executed only if the pattern in the condition matches the current word;
- a line which starts with a % is treated as a comment;
- curled brackets are used to form compound commands;
a “select rule with longest suffix” statement selects a single rule for execution among all the rules in a compound command. The rule selected is the one with the largest matching suffix.

Thus, the expression (C)* refers to a sequence of zero or more consonants while the expression ((V)*(C)*)* refers to a sequence of zero or more vowels followed by zero or more consonants which can appear zero or more times. It is important to distinguish the above from the sequence (V*C) which states that a sequence must be present and that this sequence necessarily starts with a vowel, followed by a subsequence of zero or more letters, and finished by a consonant.
Finally, the command if (*V*L) then ed -> NIL states that the substitution of the suffix ed by nil (i.e., the removal of the suffix ed) only occurs if the current word contains a vowel and at least one additional letter.

% Phase 1: Plurals and past participles. select rule with longest suffix { sses -> ss; ies -> i; ss -> ss; s -> NIL;
} select rule with longest suffix {
if ( (C)*((V)+(C)+)+(V)*eed) then eed -> ee; if (*V*ed or *V*ing) then { select rule with longest suffix { ed -> NIL; ing -> NIL; } select rule with longest suffix { at -> ate; bl -> ble; iz -> ize; if ((* C1C2) and ( C1 = C2) and (C1 not in {l,s,z})) then C1C2 -> C1; if (( (C)*((V)+(C)+)C1V1C2) and (C2 not in {w,x,y})) then C1V1C2 -> C1V1C2e; }
}
} if (*V*y) then y -> i;

% Phase 2 if ( (C)*((V)+(C)+)+(V)*) then select rule with longest suffix { ational -> ate; tional -> tion; enci -> ence; anci -> ance; izer -> ize; abli -> able; alli -> al; entli -> ent; eli -> e; ousli -> ous; ization -> ize; ation -> ate; ator -> ate; alism -> al; iveness -> ive; fulness -> ful; ousness -> ous; aliti -> al; iviti -> ive; biliti -> ble; }

% Phase 3 if ( (C)*((V)+(C)+)+(V)*) then select rule with longest suffix { icate -> ic; ative -> NIL; alize -> al; iciti -> ic; ical -> ic; ful -> NIL; ness -> NIL; }

% Phase 4 if (
(C)*((V)+(C)+)((V)+(C)+)+(V)*) then select rule with longest suffix { al -> NIL; ance -> NIL; ence -> NIL; er -> NIL; ic -> NIL; able -> NIL; ible -> NIL; ant -> NIL; ement -> NIL;
ment -> NIL; ent -> NIL; ou -> NIL; ism -> NIL; ate -> NIL; iti -> NIL; ous -> NIL; ive -> NIL; ize -> NIL;
if (*s or *t) then ion -> NIL; }

% Phase 5 select rule with longest suffix { if ( (C)*((V)+(C)+)((V)+(C)+)+(V)*) then e -> NIL; if (( (C)*((V)+(C)+)(V)*) and not (( *C1V1C2) and (C2 not in {w,x,y}))) then e -> nil; } if ( (C)*((V)+(C)+)((V)+(C)+)+V*ll) then ll -> l;
Source: Baeza-Yates [1], Appendix (available online at www.sims.berkeley.edu/~hearst/irbook/porter.html)

Usefulness for retrieval is controversial
Seems useful for languages such as German
(lots of cases, compound nouns, etc.), i.e. languages where it is hard to do successfully!
Hard for large, multilingual data collections
(such as the web)
Another problem: Usability (wrong results often hard to understand by the users)

Term processing, e.g.
Normalization
Lexical analysis & tokenization
Removal of stop words
Thesaurus
Stemming (Porter’s Algorithm) and much more
How is it usually done?
Often depends on the application and data
Is it useful? How does it influence the result?
We need a measure for that!

Evaluation of IR systems is often done based on some test data (benchmark tests), i.e. a set of (document-query-relevance)-labels
The most important measures to quantify IR performance are Precision and Recall
PRECISION =
# FOUND & RELEVANT
# FOUND
RECALL =
# FOUND & RELEVANT
# RELEVANT

RESULT: 1. DOC. B
2. DOC. E
3. DOC. F
4. DOC. G
5. DOC. D
6. DOC. H
DOCUMENTS:
A
E
D
F
B
G
H
I
C
J
PRECISION =
# FOUND & RELEVANT
# FOUND
RECALL =
# FOUND & RELEVANT
# RELEVANT

How does term proc. influence IR performance?
Generally:
More specific terms: Precision , Recall
Broader terms: Precision , Recall
More specific:
Thesaurus: Precision , Recall
Phrases: Precision , Recall
Stemming: Precision , Recall
Compound nouns: Precision , Recall
Note: Often increase in recall rather than precision.
Maybe a reason why it’s hardly used for web search?

INFORMATION NEED
QUERY
QUERY PROCESSING
(PARSING & TERM
PROCESSING)
LOGICAL VIEW OF THE
INFORMATION NEED
DOCS.
User Interface
RESULTS
RESULT
REPRESENTATION
RANKING
SEARCHING
DOCUMENTS
SELECT DATA FOR
INDEXING
PARSING & TERM
PROCESSING
LOGICAL VIEW OF THE
DOCUMENTS (INDEX)
PERFORMANCE EVALUATION

Which documents should be indexed?
Big issue for web search (because we can’t do all of it: too big, changes too quickly, …)
General problems: Confidential information, formats, etc.
What is a reasonable unit for indexing ?
In most cases: 1 document = 1 file
What about email archives w. more than one email in one file (file = folder)?
What about attachments?
What about one document split over several files (e.g. HTML version of PPT file)

INFORMATION NEED
QUERY
QUERY PROCESSING
(PARSING & TERM
PROCESSING)
LOGICAL VIEW OF THE
INFORMATION NEED
DOCS.
User Interface
RESULTS
RESULT
REPRESENTATION
RANKING
SEARCHING
DOCUMENTS
SELECT DATA FOR
INDEXING
PARSING & TERM
PROCESSING
LOGICAL VIEW OF THE
DOCUMENTS (INDEX)
PERFORMANCE EVALUATION

[1] R. BAEZA-YATES, B. RIBEIRO-NETO:
‘MODERN INFORMATIN RETRIEVAL’,
ADDISON WESLEY, 1999
CHAPTER 7 (TERMPROCESSING)
CHAPTER 8.1, 8.2 (INDEX, INV. FILES)
[2] C. MANNING, P. RAGHAVAN, H. SCHÜTZ:
‘INTRODUCTION TO IR’ (TO APPEAR 2007)
CHAPTER 1 (INDEX, INV. FILES)
CHAPTER 2 (TERMPROCESSING)
AVAILABLE ONLINE AT http://www-csli.stanford.edu/~schuetze/ information-retrieval-book.html