Natural Language Processing
Readings:
Russell and Norvig’s AI: a Modern approach (Chapter 22)
James Allen’s NLU (Chapter 3)
What is Natural Language
Processing?
NLP: applications
• Speech recognition and synthesis
• Machine translation
• Document processing
– information extraction
– summarization
• Text generation
• Dialog systems (typed and spoken)
Levels of language analysis
• Phonology: What words (or sub words) are we dealing
with?
• Morphology: How words are constructed from more basic
meaning units?
• Syntax: What phrases are we dealing with?
• Semantics: What’s the context-free meaning?
• Pragmatics: What is the more exact (context-dependent)
meaning?
• Discourse Knowledge: how the immediately preceding
sentences affect the interpretation of the next sentence?
• World knowledge: Using general knowledge about the
world
Levels of language analysis
• Phonetics: sounds -> words
– /b/ + /o/ + /t/ = boat
• Morphology: morphemes -> words
– friend + ly = friendly
• Syntax: word sequence -> sentence structure
• Semantics: sentence structure + word meaning -> sentence
meaning
• Pragmatics: sentence meaning + context ->
more precise meaning
• Discourse and world knowledge
Levels of language analysis (cont.)
1. Language is one of fundamental aspects of
human behavior and is crucial component
of our lives.
2. Green frogs have large noses.
3. Green ideas have large noses.
4. Large have green ideas nose.
5. I go store.
Why is NLP Hard?
“At last, a computer that understands you like
your mother”
Ambiguity
•
“At last, a computer that understands you
like your mother”
1. (*) It understands you as well as your
mother understands you
2. It understands (that) you like your mother
3. It understands you as well as it
understands your mother
• 1 and 3: Does this mean well, or poorly?
Ambiguity at Many Levels
At the acoustic level (speech recognition):
1. “ ... a computer that understands you like
your mother”
2. “ ... a computer that understands you lie
cured mother”
Ambiguity at Many Levels
At the syntactic level:
Different structures lead to different
interpretations.
More Syntactic Ambiguity
Ambiguity at Many Levels
At the semantic (meaning) level:
Two definitions of “mother”
• a woman who has given birth to a child
• a substance consisting of bacteria, used to
produce vinegar (i.e., mother of vinegar)
This is an instance of word sense ambiguity
Ambiguity at Many Levels
At the discourse level:
• Alice says they’ve built a computer that
understands you like your mother
• But she ...
• ... doesn’t know any details
• ... doesn’t understand me at all
Syntactic analysis
• Syntax can make explicit when there are
several possible interpretations
– (Rice flies) like sand.
– Rice (flies like sand).
• Knowledge of ‘correct’ grammar can help
finding the right interpretation
– Flying planes are dangerous.
– Flying planes is dangerous.
Syntax shows how words are related in a
sentence.
Visiting aunts ARE boring.
vs
Visiting aunts IS boring.
Subject verb agreement allows us to
disambiguate here.
How do we represent syntax?
Parse Tree
An example:
– Parsing sentence:
– "They are cooking apples."
Parse 1
Parse 2
How do we represent syntax?
List
Sue hit John
[ s, [np, [proper_noun, Sue] ] ,
[vp, [v, hit],
[np, [proper_noun, John] ]
What strategies exist for trying to find the structure
in natural language?
Top Down vs. Bottom Up
Bottom - Up
John, hit, the, cat
prpn, hit, the, cat
prpn, v, the, cat
prpn, v, det, cat
prpn, v, det, n
np, v, det, n
np, v, np
np, vp
s
Top - Down
s
s -> np, vp
s -> prpn, vp
s -> John, v, np
s -> John, hit, np
s -> John, hit, det,n
s -> John, hit, the,n
s -> John, hit, the,cat
What strategies exist for trying to find the structure
in natural language?
Top Down vs. Bottom Up
Bottom - Up
John, hit, the, cat
prpn, hit, the, cat
prpn, v, the, cat
prpn, v, det, cat
prpn, v, det, n
np, v, det, n
np, v, np
np, vp
s
Top - Down
s
s -> np, vp
s -> prpn, vp
s -> John, v, np
s -> John, hit, np
s -> John, hit, det,n
s -> John, hit, the,n
s -> John, hit, the,cat
Better if many alternative rules for a phrase
Worse if many alternative terminal symbols for
each word
Better if many alternative terminal symbols
for each word
Worse if many alternative rules for a phrase
What does an example grammar for
English look like?
• Re-write rules
1.sentence -> noun phrase , verb phrase
2.noun phrase -> art , noun
3.noun phrase -> art , adj , noun
4.verb phrase -> verb
5.verb phrase -> verb , noun phrase
Top down parsing (as a search procedure)
1
The
2
dog 3 cried 4
Step Current state
Backup States
1
2
3
((S) 1)
((NP VP) 1)
((ART N VP) 1)
4
((N VP) 2)
initial position
Rule 1
Rules 2 & 3
((ART ADJ N VP) 1)
Match Art with the
((ART ADJ N VP) 1)
Match N with dog
((ART ADJ N VP) 1)
Rules 4 & 5
((V NP) 3)
((ART ADJ N VP) 1)
Success
5 ((VP) 3)
6
7
((V) 3)
comment
Chart Parsing
General Principles:
A Bottom-Up parsing method
– Construct a parse starting from the input symbols
– Build constituents from sub-constituents
– When all constituents on the RHS of a rule are matched, create a
constituent for the LHS of the rule
The Chart allows storing partial analyses, so that they can be shared.
Data structures used by the algorithm:
– The Key: the current constituent we are attempting to “match”
– An Active Arc: a grammar rule that has a partially matched RHS
– The Agenda: Keeps track of newly found unprocessed constituents
– The Chart: Records processed constituents (non-terminals) that span
substrings of the input
25
Chart Parsing
Steps in the Process:
Input is processed left-to-right, one word at a time
1. Find all POS of the current word (terminal-level)
2. Initialize Agenda with all POS of the word
3. Pick a Key from the Agenda
4. Add all grammar rules that start with the Key as active
arcs
5. Extend any existing active arcs with the Key
6. Add LHS constituents of newly completed rules to the
Agenda
7. Add the Key to the Chart
8. If Agenda not empty – go to (3), else go to (1)
26
The Chart Parsing Algorithm
Extending Active Arcs with a Key:
Each Active Arc has the form:
<pi> [A  X1…C…Xm] <pj>
- A Key constituent has the form: <pj >C<pk>
- When processing the Key <p1>C<p2>, we search the active
arc list for an arc <p0>[A  X1…C…Xm]<p1>, and
then create a new active arc
<p0>[A  X1…C… Xm]<p2>
- If the new active arc is a completed rule:
<p0>[A  X1…C]<p2>, then we add <p0>A<p2> to the
Agenda
- After “using” the key to extend all relevant arcs, it is entered
into the Chart
27
The Chart Parsing Algorithm
The Main Algorithm: parsing input x = x1…xn
1. i = 0
2. If Agenda is empty and i < n then set i = i + 1, find all
POS of xi and add them as constituents <pi>C<pi+1> to
the Agenda
3. Pick a Key constituent <pj>C< pk > from the Agenda
4. For each grammar rule of form A  CX1…Xm, add
<pj>[A   CX1…Xm]<pj> to the list of active arcs
5. Use Key to extend all relevant active arcs
6. Add LHS of any completed active arcs into the Agenda
7. Insert the Key into the Chart
8. If Key is <1>S<n> then Accept the input Else goto (2)
28
Chart Parsing - Example
The Grammar:
(1) S  NP VP
(2) NP  ART ADJ N
(3) NP  ART N
(4) NP  ADJ N
(5) VP  AUX VP
(6) VP  V NP
29
Chart Parsing - Example
The input:
“x = The large can can hold the water”
POS of Input Words:
–
–
–
–
–
the: ART
large: ADJ
can: N, AUX, V
hold: N, V
water: N, V
30
The large can can hold the water
31
The large can can hold the water
32
The large can can hold the water
33
The large can can hold the water
34
The final Chart
35