Introduction to (still more)
Computational Linguistics
Pawel Sirotkin
28.11-01.12.2008, Riga
Rule-based CL
Rule-based CL




Rules have to be generated by hand
Easily tailored to fit (or test) a particular theory
First results with just a handful of rules
But:




4
Very hard to get “all” the rules
Rules may conflict
Rules are language- and domain-specific
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
Statistical CL
Needed: an algorithm that can create rules
Algorithm needs training data to learn




More and more data around
Digitalized literature, official documents, corpora
These rules can be applied to new texts
Good points:




5
Largely independent from language, domain etc.
Computational power available in abundance
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
A brief aside: Corpora
First major corpus: Brown Corpus (mid-60ies)
•
–
–
–
500 samples of 2000 words each
From newspapers, fiction and non-fiction books
Around 80 part-of-speech-tags
•
Tagging took over 15 year to be completed
Modern corpora: BNC, COCA, ...
•
–
–
–
6
Sometimes hundreds of millions of words
Written and spoken texts
More or less syntactic and semantic annotation
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
Part-of-Speech Tagging
Linguistic background



What are parts of speech?
How do we recognize them?
Practical usage



What are POS taggers good for?
What should they do?
Implementation



7
What are the possible problems?
What are the possible solutions?
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
Parts of speech
Nouns, verbs, adjectives…

I have a dream that my four little children will one day live
in a nation where they will not be judged by the color of
their skin but by the content of their character.
(Martin Luther King)
How many nouns are there in this text?

8
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
Parts of speech
Nouns, verbs, adjectives…

I have a dream that my four little children will one day live
in a nation where they will not be judged by the color of
their skin but by the content of their character.
(Martin Luther King)
What defines a noun?

9
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
What defines a part of speech?

Noun

a word (other than a pronoun) used to identify any of a class of
people, places, or things (common noun), or to name a particular
one of these (proper noun) [OED]


any member of a class of words that typically can be combined with
determiners to serve as the subject of a verb, can be interpreted as
singular or plural, can be replaced with a pronoun, and refer to an
entity, quality, state, action, or concept [Merriam-Webster]

10
Semantic definition
Syntactic and semantic definition
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
What parts of speech are there?
Open word classes
Closed word classes
Nouns (table, time, Wiebke)
Determiners (the, some, what)
Verbs (go, use, sleep)
Auxiliary verbs (be, have, must)
Adjectives (nice, white, absent)
Pronouns (I, ourselves, his)
Adverbs (quickly, clockwise, yesterday) Prepositions (on, by, after)
Interjections (wow, ouch, er)
Conjunctions (and, while, either ... or ...)
• More (closed) word classes in English
• More (or less, or different) word classes in other
languages
• Different word classes in different linguistic
models
11
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
How to recognize word classes?


Substitution test
The small boy sits in a car.





12
The, a, this: determiner
Small, big, angry, clever: adjectives
Boy, girl, cat, doll: nouns
Sits, cries, sleeps: verbs
In, on, outside: prepositions
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
Why do we need POS tags?
•
•
Main aim: disambiguation
Useful for most advanced CLP applications
–
–
–
–
13
Machine translation
Named Entity Recognition/Extraction
Anaphora resolution
etc.
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
Part-of-Speech Tagger

Not surprisingly, an application for determining parts of
speech in a text

NotADV surprisinglyADV, anDET applicationN forPREP
determiningV partsN ofPREP speechN inPREP aDET textN
14
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
Part-of-Speech Tagging – rules?

Rule-based POS Tagging?

Possible rules (simplified):

If ends in „est“, then it‘s an adjective (superlative form)


If ends in „ed“, it‘s a verb (past or participle form)



15
Pest? Rest?
Bed? Sled?
Rules of this kind are few and unreliable
Largest problem: they don’t help with the ambiguous words!
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
Part-of-Speech Tagging – rules!
•
The wind is blowing.
–
–
How do we know wind is a noun and not a verb?
Because it appears after an article and before a verb
•
•
•
ART ___ VERB  ART NOUN VERB
We need rules about inter-word relations
Hard to say what the rules are:
–
–
16
The cromulent wind
The cromulent wind up
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
Part-of-Speech Tagging: Stats
•
•
•
Wind: 76% noun usage, 24% verb usage
ART ___ VERB: 72% noun, 1% adverb
The wind blows:
–
–
–
Verb probability: 24% x 0% = 0%
Adverb probability: 0% x 1% = 0%
Noun probability: 76% x 72% = 55%
Careful!
The numbers are invented, and the calculation is more complex than that.
17
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
What do we need?
This is a simple sentence.
This text, excogitated by Dr. Samākslots of New York, is a
bit more complicated. It consists of a few longer-thanusual sentences; also, it has punctuation etc. It will help us
to learn the complexities of part-of-speech tagging, or
POST.
18
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
We need…


A tokenizer to split the text into tokens
Tag probabilities for the tokens


Tag sequence probabilities



E.g. left: 46% adjective, 31% noun, 23% verb
E.g. ADJ ___ NOUN: 57% noun, 43% adjective
How long should the sequences be?
Methods for estimating unknown words


19
E.g. 80% proper noun probability if capitalized
No closed word classes
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
Tag probabilities
The wind blows.
•
•
•
•
•
•
•
The: 98% article, 2% adverb
Wind: 76% noun, 24% verb
Blows: 53% verb, 47% noun
Article  Noun: 72%, Article  Verb 1%
Adverb  Noun 0%, Adverb  Verb 6%
Noun  Verb 61%, Noun  Noun 4%
Verb  Verb 3%, Verb  Noun 59%.
20
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
Tag probability calculation
The wind blows.
•
•
•
•
•
•
Article – noun – verb: 98% x 72% x 76% x 61% x 53% = 17%
Article – noun – noun: 98% x 72% x 76% x 4% x 47% = 10%
Article – verb – noun: 98% x 1% x 24% x 39% x 47% = 0.04%
Article – verb – verb: 98% x 1% x 24% x 3% x 53% = 0.0004%
…
The complexity of calculations explodes when the length of
the sentences and the number of tags increase.
21
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
Hidden Markov Models
The
98%
?
?
22
wind
72%
2%
76%
blows
61%
24%
6%
52%
47%
59%
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
Viterbi Algorithm
The
98%
wind
72%
2%
adverb: 2%
23
61%
0.22%
24%
6%
article: 98%
54%
76%
blows
52%
47%
59%
article – noun: 54% article – noun – verb: 18%
17%
article – verb: 0.22%
0.2% article – noun – noun: 1%
adverb – noun: 0% article – verb – verb: 0.02%
adverb – verb: 0.02%article – noun – noun: 0.05%
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
HMMs – the theory

A five-tuple (S, K, Π, A, B)

Set of states S


Output alphabet K


here: tag sequence probabilities
Symbol emission probabilities B

24
here: probabilities for first item in a sentence/text
State transition probabilities A


here: the possible tokens
Initial probabilities Π


here: the possible tags at any point
Here: token-tag-probabilities
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin
POST: Current state


Baseline approach (tagging each token with most
frequently used tag) delivers up to 90% accuracy
State-of-the-art taggers reach 96-97% accuracy



But: Given an average sentence length of 20 words in a
newspaper text, we get errors in most sentences!
POS taggers are used as a first step in most complex CL
applications
Some free online taggers: CLAWS, CST, CCG…
25
Computational Linguistics, NLL Riga
2008, by Pawel Sirotkin