Competitive Grammar Writing
Jason Eisner
Johns Hopkins
Noah A. Smith
Carnegie Mellon
TeachCL Workshop @ ACL – June 20, 2008
1
1. Welcome to the lab exercise!


Please form teams of ~3 people …
Programmers, get a linguist on your team


And vice-versa
Undergrads, get a grad student on your team

And vice-versa
We always run this exercise on the 1st day
of the Johns Hopkins Summer School
in Human Language Technology
every summer
since 2002
thank you JHU,
NSF, and NAACL …
We’ve also run variants in our JHU & CMU classes
2
2. Okay, team, please log in



The 3 of you should use adjacent workstations
Log in as individuals
Your secret team directory:
cd …/03-turbulent-kiwi



You can all edit files there
Publicly readable & writeable
No one else knows the secret directory name
Minimizes
permissions fuss
3
3. Now write a grammar of English

You have 2 hours. 
Actually, as the deadline
approaches, the teams usually
vote to stay an extra hour
4
3. Now write a grammar of English

You have 2 hours. 
What’s a grammar?
Here’s one to start with.

1
S1  NP VP .

1
VP  VerbT NP


20 NP  Det N’
1 NP  Proper

20 N’  Noun
1 N’  N’ PP

1

PP  Prep NP
5
3. Now write a grammar of English
Here’s one to start with.
Plus initial terminal rules.











1
1
1
1
1
1
1
1
1
1
1
Noun  castle
Noun  king
…
Proper  Arthur
Proper  Guinevere
…
Det  a
Det  every
…
VerbT covers
VerbT rides
…
Misc  that
Misc  bloodier
Misc  does
…

1
S1  NP VP .

1
VP  VerbT NP


20 NP  Det N’
1 NP  Proper

20 N’  Noun
1 N’  N’ PP

1

PP  Prep NP
Any PCFG is okay
6
3. Now write a grammar of English
Here’s one to start with.
S1
NP
1
VP
.

1
S1  NP VP .

1
VP  VerbT NP



20 N’  Noun
1 N’  N’ PP

1

Sample a sentence
on the blackboard
20 NP  Det N’
1 NP  Proper
PP  Prep NP
Any PCFG is okay
7
3. Now write a grammar of English
Here’s one to start with.
S1
NP
VP
.

1
S1  NP VP .

1
VP  VerbT NP

Det
N’


20 N’  Noun
1 N’  N’ PP

1

Sample a sentence
on the blackboard
20 NP  Det N’
1 NP  Proper
PP  Prep NP
Any PCFG is okay
8
3. Now write a grammar of English
Here’s one to start with.
S1
NP
VP
.

1
S1  NP VP .

1
VP  VerbT NP

Det
every
N’ drinks [[Arthur [across

the [coconut in the castle]]]
Noun [above another chalice]] 
castle
Sample a sentence
on the blackboard
20 NP  Det N’
1 NP  Proper

20 N’  Noun
1 N’  N’ PP

1
PP  Prep NP
Arbitrary PCFG is okay
9
4. Okay – go!
How will
we be tested
on this?
10
5.
4. Evaluation
Okay – go! procedure


We’ll sample 20 random sentences
from your PCFG.
Human judges will vote on whether
each sentence is grammatical.

How will
we be tested
on this?
By the way, y’all will be the judges
(double-blind). this is educational

You probably want to use the
sampling script to keep testing
your grammar along the way.
11
5. Evaluation procedure




You’re right:
This only tests precision.
How about recall?

1 S1 
NP VP .

1 VP 
VerbT NP

20 NP 
 1 NP 
Det N’
Proper
20 N’ 
 1 N’ 
Noun
N’ PP


We’ll sample 20 random sentences
from your PCFG.
Human judges will vote on whether
each sentence is grammatical.
1 PP 
Prep NP
Ok, we’re done!
All our sentences
are already
grammatical.
12
Development set
You might want your grammar to generate …














Arthur is the king .
covered by initial grammar
Arthur rides the horse near the castle .
riding to Camelot is hard .
We provide a file
do coconuts speak ?
of 27 sample sentences
what does Arthur ride ?
illustrating a range of
who does Arthur suggest she carry ?
grammatical phenomena
why does England have a king ?
are they suggesting Arthur ride to Camelot ?
five strangers are at the Round Table . questions, movement,
(free) relatives, clefts,
Guinevere might have known .
Guinevere should be riding with Patsy . agreement, subcat frames,
conjunctions, auxiliaries,
it is Sir Lancelot who knows Zoot !
gerunds, sentential
either Arthur knows or Patsy does .
subjects, appositives …
neither Sir Lancelot nor Guinevere will speak of it .
13
Development set
You might want your grammar to generate …













the Holy Grail was covered by a yellow fruit .
Zoot might have been carried by a swallow .
Arthur rode to Camelot and drank from his chalice .
they migrate precisely because they know they will grow .
do not speak !
Arthur will have been riding for eight nights .
Arthur , sixty inches , is a tiny king .
Arthur knows Patsy , the trusty servant .
movement,
Arthur and Guinevere migrate frequently questions,
.
(free) relatives, clefts,
he knows what they are covering with that story .
agreement, subcat frames,
Arthur suggested that the castle be carried .
conjunctions, auxiliaries,
the king drank to the castle that was his home
.
gerunds,
sentential
when the king drinks , Patsy drinks .
subjects, appositives …
14
5’. Evaluation of recall (= productivity!!)
What we could have done:
Cross-entropy on a similar, held-out test set

every coconut of his that the swallow dropped sounded like a horse .
No OOVs allowed
in the test set.
Fixed vocabulary.
How should we
parse sentences
with OOV words?
15
5’. Evaluation of recall (= productivity!!)
What we could have done
done:(good for your class?):
Cross-entropy on a similar, held-out test set
What we actually did, to heighten competition & creativity:
Test set comes from the participants!
You should try to
generate
sentences that
your opponents
can’t parse.
In Boggle, you get
points for finding
words that your
opponents don’t
find.
Use the fixed vocabulary creatively.
16
Initial terminal rules
The initial grammar sticks
to 3rd-person singular
transitive present-tense
forms. All grammatical.




But we provide 183 Misc
words (not accessible from
initial grammar) that you’re
free to work into your
grammar …





Use the fixed
vocabulary creatively.


1
1
1
1
1
1
1
1
1
1
1
Noun
Noun
…
Proper
Proper
…
Det
Det
…
VerbT
VerbT
…
Misc
Misc
Misc
…
castle
king
Arthur
Guinevere
a
every
covers
rides
that
bloodier
does
17
Initial terminal rules
The initial grammar sticks
to 3rd-person singular
transitive present-tense
forms. All grammatical.
But we provide 183 Misc
words (not accessible from
initial grammar) that you’re
free to work into your
grammar …
pronouns (various cases),
plurals,
various verb forms,
non-transitive verbs,
adjectives (various forms),
adverbs & negation,
conjunctions & punctuation,
wh-words,
…

Use the fixed
vocabulary creatively.


1
1
1
Misc
Misc
Misc
that
bloodier
does
…
18
5’. Evaluation of recall (= productivity!!)
What we could have done (good for your class?):
Cross-entropy on a similar, held-out test set
What we actually did, to heighten competition & creativity:
Test set comes from the participants!
You should try to
generate
sentences that
your opponents
can’t parse.
In Boggle, you get
points for finding
words that your
opponents don’t
find.
19
5’. Evaluation of recall (= productivity!!)
What we could have done (good for your class?):
Cross-entropy on a similar, held-out test set
What we actually did, to heighten competition & creativity:
Test set comes from the participants!
We’ll score your cross-entropy
You should try to
when you try to parse the sentences
generate
that the other teams generate.
sentences that
your opponents
can’t parse.
(Only the ones judged grammatical.)

You probably want to use the
parsing script to keep testing
your grammar along the way. 20
5’. Evaluation of recall (= productivity!!)
What we could have done (you could too):
Cross-entropy on a similar, held-out test set
What we actually did, to heighten competition & creativity:
Test set comes from the participants!
0 probability??
You get
the
We’ll score your
cross-entropy
penalty.
when you try toinfinite
parse
the sentences
thatifthe
What
my other teams generate.
grammar can’t parse
one(Only
of thethe
testones judged grammatical.)
sentences?
So don’t do that.
21
Use a backoff grammar: Bigram POS HMM
S2
Initial backoff grammar
i.e., something that
_Verb starts with a Verb


Verb
rides
i.e., something that
_Misc starts with a Misc 

Misc _Punc
‘s
Punc
!
_Noun




Noun
swallow

S2  
S2  _Noun
(etc.)
S2  _Misc
_Noun  Noun
_Noun  Noun _Noun
_Noun  Noun _Misc
_Misc  Misc
_Misc  Misc _Noun
_Misc  Misc _Misc
22
Use a backoff grammar: Bigram POS HMM
Init. linguistic grammar

S1  NP VP .

VP  VerbT NP





NP  Det N’
NP  Proper
Initial backoff grammar





N’  Noun
N’  N’ PP

PP  Prep NP



S2  
S2  _Noun
(etc.)
S2  _Misc
_Noun  Noun
_Noun  Noun _Noun
_Noun  Noun _Misc
_Misc  Misc
_Misc  Misc _Noun
_Misc  Misc _Misc
23
Use a backoff grammar: Bigram POS HMM
Initial master grammar
Mixture
model


START  S1
START  S2
Init. linguistic grammar

S1  NP VP .

VP  VerbT NP





NP  Det N’
NP  Proper
Choose these
weights wisely!
Initial backoff grammar





N’  Noun
N’  N’ PP

PP  Prep NP


S2  
S2  _Noun
(etc.)
S2  _Misc
_Noun  Noun
_Noun  Noun _Noun
_Noun  Noun _Misc
_Misc  Misc
_Misc  Misc _Noun
24
6. Discussion


What did you do? How?
Was CFG expressive enough?



features, gapping
How should one pick the weights?



How would you improve the formalism?
Would it work for other languages?
And how could you build a better backoff grammar?
Is grammaticality well-defined? How is it related to probability?
What if you had 36 person-months to do it right?



What other tools or data do you need?
What would the resulting grammar be good for?
What evaluation metrics are most important?
25
7. Winners announced
26
7. Winners announced

Of course, no one finishes their ambitious plans.
Helps to
favor
backoff
grammar
Anyway, a lot of work!
yay
unreachable

Alternative: Allow 2 weeks (see paper) …
27
What did they do? (see paper)












More fine-grained parts of speech
do-support for questions & negation
Movement using gapped categories
X-bar categories (following the initial grammar)
Singular/plural features
Pronoun case
Verb forms
Verb subcategorization; selectional restrictions (“location”)
Comparative vs. superlative adjectives
Appositives (must avoid double comma)
A bit of experimentation with weights
One successful attempt to game scoring system (ok with us!)
28
Why do we recommend this lesson?

Good opening activity



No programming
Only very simple probability
No background beyond linguistic intuitions



Though w/ time constraints, helps to have a linguist on the team
Works great with diverse teams
Social, intense, good mixer, sets the pace
http://www.clsp.jhu.edu/grammar-writing
29
Why do we recommend this lesson?


Good opening activity
Introduces many topics – touchstone for later teaching

Grammaticality




Generative probability models: PCFGs and HMMs



Backoff, inside probability, random sampling, …
Recovering latent variables: Parse trees and POS taggings
Evaluation (sort of)



Grammaticality judgments, formal grammars, parsers
Specific linguistic phenomena
Desperate need for features, morphology, gap-passing
Annotation, precision, recall, cross-entropy, …
Manual parameter tuning
Why learning would be valuable, alongside expert knowledge
http://www.clsp.jhu.edu/grammar-writing
30
A final thought

The CS curriculum starts with programming



Accessible and hands-on
Necessary to motivate or understand much of CS
In CL, the equivalent is grammar writing

It was the traditional (pre-statistical) introduction


Our contributions: competitive game, statistics,
finite-state backoff, reusable instructional materials
Much of CL work still centers around grammar formalisms






We design expressive formalisms for linguistic data
Solve linguistic problems within these formalisms
Enrich them with probabilities
Process them with algorithms
Learn them from data
Connect them to other modules in the pipeline
Akin to
programming
languages
31