Textual Entailment Recognition for Web Based
Question-Answering
Fabio Massimo Zanzotto
University of Rome “Tor Vergata”
Roma, Italy
University of Rome “Tor Vergata”
Operational Scenarios
What’s the
weather in
Macao?
© F.M.Zanzotto
When is my
paper
scheduled in
the World
Intelligence
Congress?
University of Rome “Tor Vergata”
Operational Scenarios
Answering a Question using existing texts
Q: Who did Roma play with?
Web
Snippet: Roma won against Milan (2-1)
© F.M.Zanzotto
University of Rome “Tor Vergata”
Recognizing Textual Entailment
Reframing the two problems …
S: Roma won against Milan (2-1) Q: Who did Roma play against?
Roma played against X
Text (T)
Roma won against Milan
(2-1)
© F.M.Zanzotto
entails
Hypothesis (H)
Roma played against Milan
University of Rome “Tor Vergata”
Outline
• Recognizing Textual Entailment (RTE): Problem
definition
• Systems and Approaches for RTE
• Supervised Machine Learning Methods for RTE
• Semi-supervised Knowledge Induction for RTE
© F.M.Zanzotto
University of Rome “Tor Vergata”
Classical Entailment Definition
Chierchia & McConnell-Ginet (2001):
A text t entails a hypothesis h if h is true in every
circumstance (possible world) in which t is true
Strict entailment - doesn't account for some
uncertainty allowed in applications
(Dagan, Roth, Zanzotto, ACL Tutorial 2007)
© F.M.Zanzotto
University of Rome “Tor Vergata”
Language Variability
The Dow Jones Industrial Average closed up 255
Dow ends up
Dow gains 255 points
Dow climbs 255
Stock market hits a
record high
(Dagan, Roth, Zanzotto, ACL Tutorial 2007)
© F.M.Zanzotto
University of Rome “Tor Vergata”
Natural Language and Meaning
Meaning
Variability
Ambiguity
Language
(Dagan, Roth, Zanzotto, ACL Tutorial 2007)
© F.M.Zanzotto
University of Rome “Tor Vergata”
Applied Textual Entailment
A directional relation between two text fragments:
Text (T) and Hypothesis (H):
T entails H (TH)
if humans reading t will infer that h is most
likely true
For textual entailment to hold we require:
T + previous knowledge K H
and not
KH
© F.M.Zanzotto
University of Rome “Tor Vergata”
Applied Textual Entailment
• Operational (applied) definition:
– Human gold standard - as in NLP applications
– Assuming common background knowledge
For textual entailment to hold we require:
T + previous knowledge K H
and not
KH
© F.M.Zanzotto
University of Rome “Tor Vergata”
Applied Textual Entailment
Model variability as relations between text
expressions:
• Equivalence: text1  text2 (paraphrasing)
Text (T)
Roma won against Milan
entails
• Entailment: text1  text2
Text (T)
Roma won against Milan
(2-1)
© F.M.Zanzotto
entails
Hypothesis (H)
Roma defeated Milan
the general case
Hypothesis (H)
Roma played against Milan
University of Rome “Tor Vergata”
Operational Definition
The task has been operationally defined in the
challenges of
Recognizing Textual Entailment (RTE)
(Dagan et al. 2005)
under
the PASCAL EU Network (RTE 1-2-3)
the NIST (RTE 4-5-6-7)
the SEMEVAL conference (RTE-8)
© F.M.Zanzotto
Current
Challenge
University of Rome “Tor Vergata”
Operational Definition
The task has been defined on the basis of other NLP
tasks:
–
–
–
–
–
–
–
Question Answering
Information Extraction
“Semantic” Information Retrieval
Comparable documents / multi-doc summarization
Machine Translation evaluation
Reading comprehension
Paraphrase acquisition
• Most data created from actual applications output
(Dagan, Roth, Zanzotto, ACL Tutorial 2007)
© F.M.Zanzotto
University of Rome “Tor Vergata”
Some RTE Challenge Examples
TEXT
HYPOTHESIS
TASK
ENTAILMENT
Regan attended a ceremony in
1 Washington to commemorate the
landings in Normandy.
Washington is located in
Normandy.
IE
False
2 Google files for its long awaited IPO.
Google goes public.
IR
True
…: a shootout at the Guadalajara
airport in May, 1993, that killed
3
Cardinal Juan Jesus Posadas Ocampo
and six others.
Cardinal Juan Jesus
Posadas Ocampo died in
1993.
QA
True
IE
True
The SPD got just 21.5% of the vote
in the European Parliament elections,
4 while the conservative opposition
parties
polled 44.5%.
© F.M.Zanzotto
The SPD is defeated by
the opposition parties.
(Dagan, Roth, Zanzotto, ACL Tutorial 2007)
University of Rome “Tor Vergata”
Outline
• Recognizing Textual Entailment (RTE): Problem
definition
• Systems and Approaches for RTE
• Supervised Machine Learning Methods for RTE
• Semi-supervised Knowledge Induction for RTE
© F.M.Zanzotto
University of Rome “Tor Vergata”
Systems for RTE
Problem
We want to build a system that recognize whether:
a text T entails an Hypothesis H
Questions:
• How many possibilities do we have?
• What kind of knowledge do we need?
• Is there a baseline system?
© F.M.Zanzotto
University of Rome “Tor Vergata”
Baseline RTE system
Lexical Overlap System
Count how many words/tokens are in common ore
“related” between T and H, if this number is large
(above a threshold) then
say
ENTAILMENT
otherwise
say
NOT-ENTAILMENT
© F.M.Zanzotto
University of Rome “Tor Vergata”
Baseline RTE system
Text: The Cassini spacecraft arrived at Titan in July, 2006.
Hyp:
The Cassini spacecraft has reached Titan.
(Dagan, Roth, Zanzotto, ACL Tutorial 2007)
© F.M.Zanzotto
University of Rome “Tor Vergata”
Baseline RTE system
Some examples:
T1  H1
(Zanzotto, Moschitti, 2006)
T1
“At the end of the year, all solid companies pay dividends.”
H1
“At the end of the year, all solid insurance companies pay dividends.”
T1  H2
T1
“At the end of the year, all solid companies pay dividends.”
H2
“At the end of the year, all solid companies pay cash dividends.”
The problem is not so simple, but this is a good baseline!
© F.M.Zanzotto
University of Rome “Tor Vergata”
Systems for RTE
Problem
We want to build a system that recognize whether:
a text T entails an Hypothesis H
Questions:
• How many possibilities do we have?
• What kind of knowledge do we need?
• Is there a baseline system?
© F.M.Zanzotto
University of Rome “Tor Vergata”
What kind of knowledge do we need?
Roma won against Milan
Roma defeated Milan
Roma won against Milan
Roma played against Milan
we need (Lexical Knowlegde):
• the equivalence
win against  defeat
• the implication
win  play
© F.M.Zanzotto
University of Rome “Tor Vergata”
What kind of knowledge do we need?
T2  H2
T2
H2
“Kesslers team conducted 60,643 face-to-face interviews with adults
in 14 countries”
“Kesslers team interviewed more than 60,000 adults in 14 countries”
we need (first-order rules/rules with variables):
• the equivalence
“X conducted Y interviews with Z” = “X interviewed Y Z”
• the implication
“X”  “more than Y” if X>Y
© F.M.Zanzotto
University of Rome “Tor Vergata”
Residual problems
• How do we encode this knowlegde
– It depends on the level of language interpretation
• How do we use this knowledge
– Rule based systems + threshold
– Machine learnt systems
• How do we learn this knowledge
– Supervised learning
– Unsupervised/Semisupervised Learning
© F.M.Zanzotto
University of Rome “Tor Vergata”
Textual Entailment and Language
Interpretations
Meaning
Representation
Inference
Logical Forms
Semantic
Representation
Representation
Syntactic Parse
Local Lexical
Raw Text
Textual Entailment
© F.M.Zanzotto
University of Rome “Tor Vergata”
Symbolic Langague Interpretation Models
• Constituency-based Syntactic Interpretation
S
VP
NP
© F.M.Zanzotto
NNS
VB
Farmers
feed
NP
NP
NNS
NN
cows
animal
NNS
extracts
University of Rome “Tor Vergata”
Symbolic Langague Interpretation Models
• Dependency-based Syntactic Interpretation
(Dagan, Roth, Zanzotto, ACL Tutorial 2007)
© F.M.Zanzotto
University of Rome “Tor Vergata”
Symbolic Langague Interpretation Models
• Semantic Interpretation
Semantic Role Labelling (or Semantic Parse)
T: The government purchase of the Roanoke building, a former prison,
took place in 1902.
take
The govt. purchase… prison
PRED
ARG_0
place
in 1902
ARG_1
ARG_2
purchase
PRED
The Roanoke building
ARG_1
© F.M.Zanzotto
(Dagan, Roth, Zanzotto, ACL Tutorial 2007)
University of Rome “Tor Vergata”
Symbolic Langague Interpretation Models
• Logical Forms
[Bos & Markert]
The semantic representation
language is a first-order
fragment a language used in
Discourse Representation
Theory (DRS), conveying
argument structure with a
neo-Davidsonian analysis and
Including the recursive DRS
structure to cover negation,
disjunction, and implication.
(Dagan, Roth, Zanzotto, ACL Tutorial 2007)
© F.M.Zanzotto
University of Rome “Tor Vergata”
Textual Entailment and Language
Interpretations
• Rules at different levels
© F.M.Zanzotto
Logical Forms
One rule R
Semantic
Representation
Possibly still one
rule R
Syntactic Parse
Many rules
corresponding to R
Local Lexical
Many rules
corresponding to R
University of Rome “Tor Vergata”
Rules (with Variables) at different levels
Logical Forms
Semantic
Representation
xy.win(x,y)play(x,y)
win(Arg0:x,Arg1:y)play(Arg0:x,Arg1:y)
Syntactic Parse
Local Lexical
© F.M.Zanzotto
X wins against Y X plays against Y
X won against Y X played against Y
X defeatedY X played against Y
X defeatedY X played againstY
Y has been defeated by X X played againstY
University of Rome “Tor Vergata”
Strategies for building a RTE system
• Rewriting systems (RS)
• Distance/Similarity Systems (DSS)
• Hybrid Systems = RS+DSS
© F.M.Zanzotto
University of Rome “Tor Vergata”
Strategies for building a RTE system
Rewriting Systems
t1
Meaning
Representation
t2
r1
tn-1
r2
…
rn-1
h=tn
rn
Raw Text
T
© F.M.Zanzotto
H
University of Rome “Tor Vergata”
Strategies for building a RTE system
Distance/Similarity Systems
t1
Meaning
Representation
h
sim(t1,h)
<t
>t
NO
YES
Raw Text
T
© F.M.Zanzotto
H
University of Rome “Tor Vergata”
Strategies for building a RTE system
Hybrid Systems
t1
Meaning
Representation
tj
h=tn
tk
…
…
sim(tj, tk)
Raw Text
T
© F.M.Zanzotto
<t
>t
NO
YES
H
University of Rome “Tor Vergata”
Strategies for building a RTE system
Residual Problems
• How to estimate the threshold t?
Supervised Machine Learning Approaches
• How to accumulate a large knowledge base of
rules?
Semi-supervised Machine Learning Approaches or
Knowledge Induction Methods
© F.M.Zanzotto
University of Rome “Tor Vergata”
Outline
• Recognizing Textual Entailment (RTE): Problem
definition
• Systems and Approaches for RTE
• Supervised Machine Learning Methods for RTE
• Semi-supervised Knowledge Induction for RTE
© F.M.Zanzotto
University of Rome “Tor Vergata”
Quick background
on Supervised Machine Learning
Instance
xi
Classifier
yi
Instance in a
feature space
Learnt Model
{(x1,y1)
Training Set (x2,y2)
…
(xn,yn)}
© F.M.Zanzotto
Learner
University of Rome “Tor Vergata”
Quick background
on Supervised Machine Learning
• Some Machine Learning Methods exploit the distance
between instances in the feature space
• For these machines, we can use the Kernel Trick:
– define the distance K(x1 , x2)
– instead of defining the feautures
K(x1,x2)
x1
x2
© F.M.Zanzotto
University of Rome “Tor Vergata”
RTE and Classification
If Recognizing Textual Entailment (RTE) is a
classification task:
T2  H2
T2
H2
“Kesslers team conducted 60,643 face-to-face interviews with adults
in 14 countries”
“Kesslers team interviewed more than 60,000 adults in 14 countries”
We can learn a classifier from annotated examples
Problem: Defining the feature space
© F.M.Zanzotto
University of Rome “Tor Vergata”
RTE and Classification
Hybrid Systems
Meaning
Representation
t1
tj
h=tn
tk
…
…
Classifier
Raw Text
NO
YES
T
H
We can learn a classifier from annotated examples
Problem: Defining the feature space
© F.M.Zanzotto
University of Rome “Tor Vergata”
RTE and Classification
Defining the feature space for RTE Classifiers
• Classes of models and feature spaces for sentence pairs
• A particular model: First-order rewrite rule feature spaces
for sentence pairs
© F.M.Zanzotto
University of Rome “Tor Vergata”
RTE and Classification
Defining the feature space for RTE Classifiers
• Classes of models and feature spaces for sentence pairs
• A particular model: First-order rewrite rule feature spaces
for sentence pairs
© F.M.Zanzotto
University of Rome “Tor Vergata”
Defining the feature space
How do we define the feature space?
T1  H1
T1
“At the end of the year, all solid companies pay dividends.”
H1
“At the end of the year, all solid insurance companies pay dividends.”
• Possible features
– “Distance Features” - Features of “some” distance between T and H
– “Entailment trigger Features”
– “Pair Feature” – The content of the T-H pair is represented
• Possible representations of the sentences
– Bag-of-words (possibly with n-grams)
– Syntactic representation
– Semantic representation
© F.M.Zanzotto
Page 45
University of Rome “Tor Vergata”
Similarity Features
T H
T
“At the end of the year, all solid companies pay dividends.”
H
“At the end of the year, all solid insurance companies pay dividends.”
Possible features
–
–
–
–
© F.M.Zanzotto
Number of words in common
Longest common subsequence
Longest common syntactic subtree
…
Page 46
University of Rome “Tor Vergata”
Similarity Features
Limits
T1  H1
T1
“At the end of the year, all solid companies pay dividends.”
H1
“At the end of the year, all solid insurance companies pay dividends.”
T1  H2
T1
“At the end of the year, all solid companies pay dividends.”
H2
“At the end of the year, all solid companies pay cash dividends.”
% of H covered words = 6/7
% of H covered syntactic relations = 6/7
© F.M.Zanzotto
University of Rome “Tor Vergata”
Entailment Triggers
Possible features
from (de Marneffe et al., 2006)
– Polarity features
• presence/absence of neative polarity contexts (not,no or few, without)
– “Oil price surged”“Oil prices didn’t grow”
– Antonymy features
• presence/absence of antonymous words in T and H
– “Oil price is surging”“Oil prices is falling down”
– Adjunct features
• dropping/adding of syntactic adjunct when moving from T to H
– “all solid companies pay dividends” “all solid companies pay cash
dividends”
– …
© F.M.Zanzotto
Page 48
University of Rome “Tor Vergata”
Pair Features
T H
T
“At the end of the year, all solid companies pay dividends.”
H
“At the end of the year, all solid insurance companies pay dividends.”
• Possible features
– Bag-of-word spaces of T and H
insurance_H
dividends_H
pay_H
companies_H
solid_H
…
year_H
…
end_H
dividends_T
pay_T
H
companies_T
solid_T
year_T
…
end_T
T
…
– Syntactic spaces of T and H
© F.M.Zanzotto
Page 49
University of Rome “Tor Vergata”
Pair Features: what can we learn?
• Bag-of-word spaces of T and H
– We can learn:
• T implies H as when T contains “end”…
• T does not imply H when H contains “end”…
It seems to be totally irrelevant!!!
© F.M.Zanzotto
insurance_H
dividends_H
pay_H
companies_H
solid_H
…
year_H
…
end_H
dividends_T
pay_T
H
companies_T
solid_T
year_T
…
end_T
T
…
University of Rome “Tor Vergata”
RTE and Classification
Defining the feature space for RTE Classifiers
• Classes of models and feature spaces for sentence pairs
• A particular model: First-order rewrite rule feature spaces
for sentence pairs
© F.M.Zanzotto
University of Rome “Tor Vergata”
Motivation
For example, in textual entailment…
Training examples
P1: T1  H1
P2: T2  H2
T1 “Farmers feed cows animal extracts”
T2 “They feed dolphins fishs”
H1 “Cows eat animal extracts”
H2 “Fishs eat dolphins”
feed
X
Y

X
eat
Y
Relevant Features
First-order rules
P3: T3 H3
T3 “Mothers feed babies milk”
© F.M.Zanzotto
H3 “Babies eat milk”
Classification
University of Rome “Tor Vergata”
In this part of the talk…
• First-order rule (FOR) feature spaces: a challenge
• Tripartite Directed Acyclic Graphs (tDAG) as a solution:
– for modelling FOR feature spaces
– for defining efficient algorithms for computing kernel functions
with tDAGs in FOR feature spaces
• An efficient algorithm for computing kernels in FOR
spaces
• Experimental and comparative assessment of the
computational efficiency of the proposed algorithm
© F.M.Zanzotto
University of Rome “Tor Vergata”
In this part of the tutorial…
• First-order rule (FOR) feature spaces: a challenge
• Tripartite Directed Acyclic Graphs (tDAG) as a solution:
– for modelling FOR feature spaces
– for defining efficient algorithms for computing kernel functions
with tDAGs in FOR feature spaces
• An efficient algorithm for computing kernels in FOR
spaces
• Experimental and comparative assessment of the
computational efficiency of the proposed algorithm
© F.M.Zanzotto
University of Rome “Tor Vergata”
First-order rule (FOR) feature spaces:
challenges
We want to exploit first-order rule (FOR) feature
spaces writing the implicit kernel function
K(P1,P2)=|S(P1)S(P2)|
that computes how many common first-order rules
are activated from P1 and P2
Without loss of generality, we present the problem
in syntactic-first-order rule feature spaces
© F.M.Zanzotto
University of Rome “Tor Vergata”
First-order rule (FOR) feature spaces:
challenges
T1  H1
Adding placeholders
Propagating placeholders
T1 “Farmers feed cows animal extracts”
H1 “Cows eat animal extracts”
S
S
1
NP
VP
NP
Pa= 
VP
VB
NNS
NP
1
NP
,
3
NNS
1
Cows
feed
Farmers
NNS
NN
1
2
cows
animal
1
2
NNS
3
VB
NP
3

eat
NN
1
NNS
2
animal
extracts
extracts
2
3
3
3
S
S(Pa)=
{
S
NP
,
S
VP NP
1
VP
,
VP
 ,
VB
NP
1
NP
NP
1
3
VB
© F.M.Zanzotto
 ,...
VP
feed
eat
NP
3
}
University of Rome “Tor Vergata”
First-order rule (FOR) feature spaces:
challenges
T3 H3
T3 “Mothers feed babies milk”
H3
“Babies eat milk”
S
S
NP
Pb=
VP
VP
NP

1
VB
NNS
NP
1
NP
,
2
NNS
1
Babies
Mothers
feed
NNS
1
NN
2
NP
VB
eat
2
NN

2
1
milk
babies
milk
1
2
2
S
S(Pb)= {
S
NP
,
S
VP NP
1
VP
,
VP
 ,
VB
NP
1
NP
NP
1
2
VB
© F.M.Zanzotto
 ,...
VP
feed
eat
NP
2
}
University of Rome “Tor Vergata”
First-order rule (FOR) feature spaces:
challenges
K(Pa,Pb)=|S(Pa)S(Pb)|
S
S(Pa)= {
,
S
VP NP
1
S
NP
 ,
VP
,
VP
VB
NP
1
NP
NP
1
VP

VB
NP
NP
Y
, NP
 ,...
}
3
eat
VP
X
VB
feed
NP
feed
S
X
}
3
VB
VP
 ,...
NP

=
=
Y
=
eat
S
S(Pb)= {
S
NP
,
S
VP NP
1
VP
,
VP
 ,
VB
NP
1
NP
NP
1
VP
2
VB
© F.M.Zanzotto
feed
eat
NP
2
University of Rome “Tor Vergata”
In this part of the tutorial…
• First-order rule (FOR) feature spaces: a challenge
• Tripartite Directed Acyclic Graphs (tDAG) as a solution:
– for modelling FOR feature spaces
– for defining efficient algorithms for computing kernel functions
with tDAGs in FOR feature spaces
• An efficient algorithm for computing kernels in FOR
spaces
• Experimental and comparative assessment of the
computational efficiency of the proposed algorithm
© F.M.Zanzotto
University of Rome “Tor Vergata”
A step back…
• FOR feature spaces can be modelled with
particular graphs
• We call these graphs tripartite direct acyclic
graphs (tDAGs)
• Observations:
– tDAGs are not trees
– tDAGs can be used to model both rules and sentence
pairs
– unifying rules in sentences is a graph matching problem
– graph macthing algorithms are, in general, exponential
© F.M.Zanzotto
University of Rome “Tor Vergata”
Tripartite Directed Acyclic Graphs (tDAG)
As for Feature Structures…
VP
VB
S
NP
NP
X
NP
Y
X
VP
VB
feed
NP
Y
eat
S
S
NP
NNS
VB
NP
1
NP
© F.M.Zanzotto
feed
NNS
1
NN
2
cows
animal
1
2
1
VB
NP
3
3
Cows
Farmers
VP
VP
NP
NNS
1
NNS
extracts
3
3
1
eat
NN
animal
2
2
NNS
extracts
3
3
University of Rome “Tor Vergata”
Tripartite Directed Acyclic Graphs (tDAG)
As for Feature Structures…
VP
VB
S
NP
NP
X
NP
Y
X
VP
VB
feed
NP
Y
eat
S
S
NP
NNS
VB
NP
1
NP
© F.M.Zanzotto
feed
NNS
1
NN
2
cows
animal
1
2
1
VB
NP
3
3
Cows
Farmers
VP
VP
NP
NNS
1
NNS
extracts
3
3
1
eat
NN
animal
2
2
NNS
extracts
3
3
University of Rome “Tor Vergata”
Outline
• Recognizing Textual Entailment (RTE): Problem
definition
• Systems and Approaches for RTE
• Supervised Machine Learning Methods for RTE
• Semi-supervised Knowledge Induction for RTE
© F.M.Zanzotto
University of Rome “Tor Vergata”
Semi-supervised Knowledge Induction
Acquisition of Explicit Knowledge
• Learning Lexical Knowledge or Rules
Acquisition of Implicit Knowledge
• Acquiring Corpora for Supervised Machine
Learning Models
© F.M.Zanzotto
University of Rome “Tor Vergata”
Semi-supervised Knowledge Induction
Acquisition of Explicit Knowledge
• Learning Lexical Knowledge or Rules
Acquisition of Implicit Knowledge
• Acquiring Corpora for Supervised Machine
Learning Models
© F.M.Zanzotto
University of Rome “Tor Vergata”
Acquistion of Explicit Knowledge
The questions we need to answer
• What?
– What we want to learn? Which resources do we need?
• Using what?
– Which are the principles we have?
• How?
– How do we organize the “knowledge acquisition”
algorithm
© F.M.Zanzotto
University of Rome “Tor Vergata”
Acquisition of Explicit Knowledge: what?
Types of knowledge
• Equivalence
– Co-hyponymy
Between words: cat  dog
– Synonymy
Between words: buy  acquire
Sentence prototypes (paraphrasing) : X bought Y  X acquired Z% of the Y’s shares
• Oriented semantic relations
Words: cat  animal , buy  own , wheel partof car
Sentence prototypes : X acquired Z% of the Y’s shares 
X owns Y
© F.M.Zanzotto
University of Rome “Tor Vergata”
Acquisition of Explicit Knowledge : Using what?
Underlying hypothesis
• Harris’ Distributional Hypothesis (DH) (Harris,
1964)
“Words that tend to occur in the same contexts tend to
have similar meanings.”
sim(w1,w2)sim(C(w1), C(w2))
• Robison’s Point-wise Assertion Patterns (PAP)
(Robison, 1970)
“It is possible to extract relevant semantic relations with
some pattern.”
w1 is in a relation r with w2 if the context pattern(w1, w2 )
© F.M.Zanzotto
University of Rome “Tor Vergata”
Distributional Hypothesis (DH)
simw(W1,W2)simctx(C(W1), C(W2))
Words or Forms
w1= constitute
Context (Feature) Space
C(w1)
Corpus: source of contexts
… sun is constituted of hydrogen …
…The Sun is composed of hydrogen …
w2= compose
C(w2)
© F.M.Zanzotto
Page 88
Page 88
University of Rome “Tor Vergata”
Point-wise Assertion Patterns (PAP)
w1 is in a relation r with w2 if the contexts patternsr(w1, w2 )
relation
w1 part_of w2
patterns
“w1 is constituted of w2”
“w1 is composed of w2”
Statistical Indicator
Corpus: source of contexts
… sun is constituted of hydrogen …
…The Sun is composed of hydrogen …
Scorpus(w1,w2)
selects correct vs incorrect relations
among words
© F.M.Zanzotto
part_of(sun,hydrogen)
University of Rome “Tor Vergata”
DH and PAP cooperate
Distributional Hypothesis
Words or Forms
w1= constitute
Point-wise assertion Patterns
Context (Feature) Space
C(w1)
Corpus: source of contexts
… sun is constituted of hydrogen …
…The Sun is composed of hydrogen …
w2= compose
C(w2)
© F.M.Zanzotto
University of Rome “Tor Vergata”
Knowledge Aquisition: Where methods differ?
Words or Forms
w1= cat
Context (Feature) Space
C(w1)
w2= dog
C(w2)
On the “word” side
• Target equivalence classes: Concepts or Relations
• Target forms: words or expressions
On the “context” side
• Feature Space
• Similarity function
© F.M.Zanzotto
University of Rome “Tor Vergata”
Oriented relations
Equivalence
Types of knowledge
KA4TE: a first classification of some methods
Verb Entailment
(Zanzotto et al., 2006)
Noun Entailment
(Geffet&Dagan, 2005)
Relation Pattern Learning (ESPRESSO)
(Pantel&Pennacchiotti, 2006)
ISA patterns
(Hearst, 1992)
ESPRESSO
(Pantel&Pennacchiotti, 2006)
Hearst
Concept Learning
(Lin&Pantel, 2001a)
Inference Rules (DIRT)
(Lin&Pantel, 2001b)
Distributional
Hypothesis
© F.M.Zanzotto
Point-wise assertion
Patterns
Underlying hypothesis
University of Rome “Tor Vergata”
Noun Entailment Relation
(Geffet&Dagan, 2006)
• Type of knowledge: oriented relations
• Underlying hypothesis: distributional hypothesis
w1  w2
• Main Idea: distributional inclusion hypothesis
Words or Forms
w1
Context (Feature) Space
if
All the prominent features
of w1 occur with w2 in a
sufficiently large corpus
C(w1)
I(C(w1))
+
+
+
+
w1  w2
w2
I(C(w2))
C(w2)
© F.M.Zanzotto
+
+
+
+
+
University of Rome “Tor Vergata”
Verb Entailment Relations
(Zanzotto, Pennacchiotti, Pazienza, 2006)
• Type of knowledge: oriented relations
• Underlying hypothesis: point-wise assertion
patterns ? win  play ! player wins
• Main Idea:
relation
v1  v2
patterns
“agentive_nominalizatio
n(v2) v1”
© F.M.Zanzotto
Statistical Indicator
S(v1,v2)
Point-wise Mutual information
Zanzotto, F. M.; Pennacchiotti, M. & Pazienza, M. T. Discovering asymmetric
entailment relations between verbs using selectional preferences, Coling-ACL, 2006
University of Rome “Tor Vergata”
Verb Entailment Relations
(Zanzotto, Pennacchiotti, Pazienza, 2006)
Understanding the idea
• Selectional restriction
fly(x)  has_wings(x)
in general
v(x)  c(x)
(if x is the subject of v then x has the property c)
• Agentive nominalization
“agentive noun is the doer or the performer of an action v’”
“X is player” may be read as play(x)
c(x) is clearly v’(x) if the property c is derived by v’ with
an agentive nominalization
© F.M.Zanzotto
Zanzotto, F. M.; Pennacchiotti, M. & Pazienza, M. T. Discovering asymmetric
entailment relations between verbs using selectional preferences, Coling-ACL, 2006
University of Rome “Tor Vergata”
Verb Entailment Relations
Understanding the idea
Given the expression
player wins
 Seen as a selctional restriction
win(x)  play(x)
 Seen as a selectional preference
P(play(x)|win(x)) > P(play(x))
© F.M.Zanzotto
Zanzotto, F. M.; Pennacchiotti, M. & Pazienza, M. T. Discovering asymmetric
entailment relations between verbs using selectional preferences, Coling-ACL, 2006
University of Rome “Tor Vergata”
Knowledge Acquisition for TE: How?
The algorithmic nature of a DH+PAP method
• Direct
– Starting point: target words
• Indirect
– Starting point: context feature space
• Iterative
– Interplay between the context feature space and the
target words
© F.M.Zanzotto
University of Rome “Tor Vergata”
Direct Algorithm
1.
sim(w1,w2)sim(C(w1), C(w2))
2.
sim(w1,w2)sim(I(C(w1)), I(C(w2)))
Words or Forms
w1= cat
Context (Feature) Space
C(w1)
sim(C(w1), C(w2))
I(C(w1))
sim(w1, w2)
w2= dog
sim(I(C(w1)), I(C(w2)))
I(C(w2))
C(w2)
© F.M.Zanzotto
3.
Select target words wi from the
corpus or from a dictionary
Retrieve contexts of each wi and
represent them in the feature
space C(wi )
For each pair (wi, wj)
1.
Compute the similarity
sim(C(wi), C(wj )) in the
context space
2.
If sim(wi, wj )=
sim(C(wi), C(wj ))>t,
wi and wj belong to the
same equivalence class W
University of Rome “Tor Vergata”
Indirect Algorithm
1.
sim(w1,w2)sim(C(w1), C(w2))
sim(w1,w2)sim(I(C(w1)), I(C(w2)))
2.
Words or Forms
Context (Feature) Space
3.
w1= cat
C(w1)
sim(C(w1), C(w2))
4.
5.
sim(w1, w2)
w2= dog
C(w2)
© F.M.Zanzotto
Page 99
Page 99
Given an equivalence class W,
select relevant contexts and
represent them in the feature
space
Retrieve target words (w1, …,
wn) that appear in these
contexts. These are likely to be
words in the equivalence class
W
Eventually, for each wi, retrieve
C(wiI) from the corpus
Compute the centroid I(C(W))
For each for each wi,
if sim(I(C(W), wi)<t, eliminate
wi from W.
University of Rome “Tor Vergata”
Iterative Algorithm
1.
sim(w1,w2)sim(C(w1), C(w2))
sim(w1,w2)sim(I(C(w1)), I(C(w2)))
2.
Words or Forms
Context (Feature) Space
3.
4.
w1= cat
C(w1)
sim(C(w1), C(w2))
sim(w1, w2)
w2= dog
C(w2)
© F.M.Zanzotto
Page 100
Page 100
For each word wi in the
equivalence class W, retrieve the
C(wi) contexts and represent
them in the feature space
Extract words wj that have
contexts similar to C(wi)
Extract contexts C(wj) of these
new words
For each for each new word wj,
if sim(C(W), wj)>t, put wj in W.
University of Rome “Tor Vergata”
Knowlege Acquisition using DH and PAH
• Direct Algorithms
– Concepts from text via clustering (Lin&Pantel, 2001)
– Inference rules – aka DIRT (Lin&Pantel, 2001)
– …
• Indirect Algorithms
– Hearst’s ISA patterns (Hearst, 1992)
– Question Answering patterns (Ravichandran&Hovy, 2002)
– …
• Iterative Algorithms
– Entailment rules from Web – aka TEASE (Szepktor et al., 2004)
– Espresso (Pantel&Pennacchiotti, 2006)
– …
© F.M.Zanzotto
University of Rome “Tor Vergata”
TEASE
(Szepktor et al., 2004)
Type: Iterative algorithm
On the “word” side
• Target equivalence classes: fine-grained relations
prevent(X,Y)
• Target forms: verb with arguments
subj
mod
X
call
obj
mod
indictable
finally
Y
On the “context” side
• Feature Space
X_{filler}:mi?,Y_{filler}:mi?
© F.M.Zanzotto
Idan Szpektor, Hristo Tanev, Ido Dagan and Bonaventura Coppola. 2004. Scaling
Web-based Acquisition of Entailment Relations. In Proceedings of EMNLP 2004.
University of Rome “Tor Vergata”
TEASE
(Szepktor et al., 2004)
Lexicon
Input template:
Xsubj-accuse-objY
WEB
TEASE
Sample corpus for input template:
Paula Jones accused Clinton…
BBC accused Blair…
Sanhedrin accused St.Paul…
…
Anchor Set Extraction
(ASE)
Anchor sets:
{Paula Jonessubj; Clintonobj}
{Sanhedrinsubj; St.Paulobj}
…
Sample corpus for anchor sets:
Template Extraction
Paula Jones called Clinton indictable…
St.Paul defended before the Sanhedrin
…
(TE)
Templates:
© F.M.Zanzotto
X call Y indictable
Y defend before X
and
… Bonaventura
Idan Szpektor, Hristo Tanev, Ido Dagan
Coppola. 2004. Scaling
iterate
Web-based Acquisition of Entailment Relations. In Proceedings
of EMNLP 2004.
University of Rome “Tor Vergata”
TEASE
(Szepktor et al., 2004)
Innovations with respect to reasearches < 2004
• First direct algorithm for extracting rules
• A feature selection is done to assess the most informative features
• Extracted forms are clustered to obtain the most general sentence
prototype of a given set of equivalent forms
S1:
S2:
call
subj {2}
mod {1}
mod {2}
Y
{1}
subj {1,2}
mod {1,2}
obj {1,2}
obj {2}
obj {1}
X
{1}
{1,2}
{2}
{1}
subj {1}
call
call
indictable
{1}
X
{2}
Y
{2}
for {1}
indictable
{2}
X
{1,2}
Y
{1,2}
indictable
{1,2}
for {1}
mod {2}
mod {2}
harassment
{1}
finally
{2}
finally
{2}
© F.M.Zanzotto
harassment
{1}
Idan Szpektor, Hristo Tanev, Ido Dagan and Bonaventura Coppola. 2004. Scaling
Web-based Acquisition of Entailment Relations. In Proceedings of EMNLP 2004.
University of Rome “Tor Vergata”
Espresso
(Pantel&Pennacchiotti, 2006)
Type: Iterative algorithm
On the “word” side
• Target equivalence classes: relations
compose(X,Y)
• Target forms: expressions, sequences of tokens
Y is composed by X, Y
© F.M.Zanzotto
is made of
X
Patrick Pantel, Marco Pennacchiotti. Espresso: A Bootstrapping Algorithm for Automatically
Harvesting Semantic Relations. In Proceedings of COLING/ACL-06, 2006
University of Rome “Tor Vergata”
Espresso
(Pantel&Pennacchiotti, 2006)
(leader , panel)
(city , region)
(oxygen , water)
1.0
0.9
0.7
0.6
0.6
0.2
SEEDS
(tree , land)
(atom, molecule)
(leader , panel)
(range of information, FBI report)
(artifact , exhibit)
(oxygen , hydrogen)
Instance Extraction
Pattern instantiation
Pattern Induction
Pattern Ranking / Selection
Instance Ranking / Selection
GENERIC PATTERN FILTERING
Sentence retrieval
Pattern Reliability
ranking
Generic Test
Google
yes
Web Instance
Filter
Instance Reliability
ranking
no
Sentence generalization
Instance selection
Pattern selection
Frequency count
Low Redundancy
Test
yes
Syntactic Expansion
Web Expansion
Y is composed by X
X,Y
Y is part of Y
© F.M.Zanzotto
1.0
0.8
0.2
Y is composed by X
Y is part of X
X,Y
(tree , land)
(oxygen , hydrogen)
(atom, molecule)
(leader , panel)
(range of information, FBI report)
(artifact , exhibit)
…
Patrick Pantel, Marco Pennacchiotti. Espresso: A Bootstrapping Algorithm for Automatically
Harvesting Semantic Relations. In Proceedings of COLING/ACL-06, 2006
University of Rome “Tor Vergata”
Espresso
(Pantel&Pennacchiotti, 2006)
Innovations with respect to reasearches < 2006
• A measure to determine specific vs. general
patterns (ranking in the equivalent forms)
1.0
0.8
0.2
Y is composed by X
Y is part of X
X,Y
• Both pattern and instance selections are performed
• Differnt Use of General and specific patterns in
the iterative algorithm
© F.M.Zanzotto
Patrick Pantel, Marco Pennacchiotti. Espresso: A Bootstrapping Algorithm for Automatically
Harvesting Semantic Relations. In Proceedings of COLING/ACL-06, 2006
University of Rome “Tor Vergata”
Structure & Lexico-Syntactic Patterns
Observation
• Distributional Models (DH)
Target Relations
Hyperonymy (IS_A)
Cotopy (Similarity)
Use of structural
properties
Transitivity is implicitly exploited
• Lexico-Syntactic Pattern Models (LSP)
© F.M.Zanzotto
Target Relations
All possible semantic
relations
Use of structural
properties
Transitivity is NOT
exploited
Fallucchi, F. & Zanzotto, F. M. Inductive Probabilistic Taxonomy Learning using
Singular Value Decomposition, NATURAL LANGUAGE ENGINEERING, 2011
University of Rome “Tor Vergata”
Structure & Lexico-Syntactic Patterns
Target Relations
All possible semantic
relations
Use of structural
properties
Transitivity is effectively
exploited
Exploiting Transitivity within Lexico-Syntactic Pattern
Models
•we exploit structural properties of target relations to
determine the probability
•we focus on the transitivity to reinforce or lower the
probability
© F.M.Zanzotto
Fallucchi, F. & Zanzotto, F. M. Inductive Probabilistic Taxonomy Learning using
Singular Value Decomposition, NATURAL LANGUAGE ENGINEERING, 2011
University of Rome “Tor Vergata”
Structure & Lexico-Syntactic Patterns
isa
relation
0.2
animal
0.7
mammal
0.648
0.8
cat
P ( R animal , cat | E )
P ( Rˆ animal , cat | E )
© F.M.Zanzotto
Direct Probabilities for Corpus Observation
(E) with Lexico-Syntactic Patterns
Induced Probabilities
Fallucchi, F. & Zanzotto, F. M. Inductive Probabilistic Taxonomy Learning using
Singular Value Decomposition, NATURAL LANGUAGE ENGINEERING, 2011
University of Rome “Tor Vergata”
Structure & Lexico-Syntactic Patterns
P ( R k1 , j | E )
vegetable
(k1)
food (j)
P ( Rˆ i , j | E )
P ( R i , k1 | E )
P ( Rk2 , j | E )
animal (k2)
P ( Ri ,k 2 | E )
lettuce (i)
P ( Rˆ i , j | E ) = P ( R i , j  ( R i , k1  R k1 , j )  ( R i , k 2  R k 2 , j ) | E )
© F.M.Zanzotto
Fallucchi, F. & Zanzotto, F. M. Inductive Probabilistic Taxonomy Learning using
Singular Value Decomposition, NATURAL LANGUAGE ENGINEERING, 2011
University of Rome “Tor Vergata”
Semi-supervised Knowledge Induction
Acquisition of Explicit Knowledge
• Learning Lexical Knowledge or Rules
Acquisition of Implicit Knowledge
• Acquiring Corpora for Supervised Machine
Learning Models
© F.M.Zanzotto
University of Rome “Tor Vergata”
Acquistion of Implicit Knowledge
The questions we need to answer
• What?
– What we want to learn? Which resources do we need?
• Using what?
– Which are the principles we have?
© F.M.Zanzotto
University of Rome “Tor Vergata”
Acquisition of Explicit Knowledge: what?
Types of knowledge
• Equivalence
– Nearly Synonymy between sentences
Acme Inc. bought Goofy ltd.  Acme Inc. acquired 11% of the Goofy ltd.’s shares
• Oriented semantic relations
– Entailment between sentences
Acme Inc. acquired 11% of the Goofy ltd.’s shares 
Acme Inc. owns Goofy ltd.
Note: ALSO TRICKY NOT-ENTAILMENT ARE
RELEVANT
© F.M.Zanzotto
University of Rome “Tor Vergata”
Acquisition of Explicit Knowledge : Using what?
Underlying hypothesis
• Structural and content similarity
“Sentences are similar if they share enough content”
sim(s1,s2) according to relations from s1 and s2
• A revised Point-wise Assertion Patterns
“Some patterns of sentences reveal relations among
sentences”
© F.M.Zanzotto
University of Rome “Tor Vergata”
Types of knowledge
Oriented relations
Equivalence
entails not entails
A first classification of some methods
Wikipedia Revisions
Relations among sentences
(Hickl et al., 2006) (Zanzotto&Pennacchiotti, 2010)
Relations among sentences
(Burger&Ferro, 2005)
Paraphrase Corpus
(Dolan&Quirk, 2004)
Revised Point-wise
Structural and content
assertion Patterns
similarity
Underlying hypothesis
© F.M.Zanzotto
University of Rome “Tor Vergata”
Entailment relations among sentences
(Burger&Ferro, 2005)
• Type of knowledge: oriented relations
(entailment)
• Underlying hypothesis: revised point-wise
assertion patterns
• Main Idea: in headline news items, the first
sentence/paragraph generally entails the title
relation
s2  s1
patterns
“News Item
Title(s1)
First_Sentence(s2)”
© F.M.Zanzotto
This pattern works on the structure
of the text
University of Rome “Tor Vergata”
Entailment relations among sentences
examples from the web
© F.M.Zanzotto
Title
New York Plan for DNA Data in Most Crimes
Body
Eliot Spitzer is proposing a major expansion of New
York’s database of DNA samples to include people
convicted of most crimes, while making it easier for
prisoners to use DNA to try to establish their innocence.
…
Title
Chrysler Group to Be Sold for $7.4 Billion
Body
DaimlerChrysler confirmed today that it would sell a
controlling interest in its struggling Chrysler Group to
Cerberus Capital Management of New York, a private
equity firm that specializes in restructuring troubled
companies. …
University of Rome “Tor Vergata”
Tricky Not-Entailment relations among sentences
(Hickl et al., 2006)
• Type of knowledge: oriented relations (tricky notentailment)
• Underlying hypothesis: revised point-wise assertion
patterns
• Main Idea:
– in a text, sentences with a same name entity generally do not
entails each other
– Sentences connected by “on the contrary”, “but”, … do not entail
each other
relation
patterns
s1  s2
s1 and s2 are in the same text and
share at least a named entity
“s1. On the contrary, s2”
© F.M.Zanzotto
University of Rome “Tor Vergata”
Tricky Not-Entailment relations among sentences
examples from (Hickl et al., 2006)
© F.M.Zanzotto
T
One player losing a close friend is Japanese pitcher
Hideki Irabu, who was befriended by Wells during spring
training last year.
H
Irabu said he would take Wells out to dinner
when the Yankees visit Toronto.
T
According to the professor, present methods of cleaning
up oil slicks are extremely costly and are never
completely efficient.
H
In contrast, he stressed, Clean Mag has a 100
percent pollution retrieval rate, is low cost and can be
recycled.
University of Rome “Tor Vergata”
Wikipedia for Extracting Examples
(Zanzotto&Pennacchiotti, 2010)
Extracting pairs of sentences from
Wikipedia revision system
Wikipedia : open encyclopedia, where every person can behave
as an author, inserting new entries or modifying existing ones.
HYPOTHESIS
Given an original entry S1 a piece of text in Wikipedia before it is
modified by an author, and the revision S2 the modified text:
(S1, S2) extracted from the Wikipedia revision
database, represent good candidate of both positive
and negative entailment pairs (T,H).
© F.M.Zanzotto
University of Rome “Tor Vergata”
Wikipedia for Extracting Examples
(Zanzotto&Pennacchiotti, 2010)
• Type of knowledge: oriented relations (tricky notentailment)
• Underlying hypothesis: revised point-wise assertion
patterns
• Main Idea:
© F.M.Zanzotto
University of Rome “Tor Vergata”
Wikipedia for Extracting Examples
(Zanzotto&Pennacchiotti, 2010)
• Here an example
© F.M.Zanzotto
University of Rome “Tor Vergata”
Nice properties of Wikipedia revisions
(Zanzotto&Pennacchiotti, 2010)
Wikipedia revisions are ideal for co-training:
given a pair entry–revision (S1, S2) , we can define
two independent views:
• content-pair view : features modeling the actual textual
content (S1, S2).
• comment view : features regarding the comment
inserted by the author of the revision S2 (usually, the
reason and the explanation of the changes he wrote).
© F.M.Zanzotto
University of Rome “Tor Vergata”
What we have seen
• Recognizing Textual Entailment (RTE): Problem
definition
• Systems and Approaches for RTE
• Supervised Machine Learning Methods for RTE
• Semi-supervised Knowledge Induction for RTE
© F.M.Zanzotto
University of Rome “Tor Vergata”
RTE Resources
Current RTE Challenge
http://www.cs.york.ac.uk/semeval-2013/task7/
Textual Entailment Resource Pool
http://aclweb.org/aclwiki/index.php?title=Textual_Entailment_Resource_Pool
Book on Recognizing Textual Entailment
I. Dagan, D. Roth, M. Sommons, F.M.Zanzotto, Recognizing Textual Entailment: Models
and Applications, Morgan&Claypool Publishers (forthcoming)
© F.M.Zanzotto
University of Rome “Tor Vergata”
Learning RTE Systems on Rule Spaces
Initial Idea
• Zanzotto, F. M. & Moschitti, A. Automatic learning of textual entailments with crosspair similarities, ACL-44: Proceedings of the 21st International Conference on
Computational Linguistics and the 44th annual meeting of the Association for
Computational Linguistics, 2006
First refinement of the algorithm
• Moschitti, A. & Zanzotto, F. M. Fast and Effective Kernels for Relational Learning from
Texts, Proceedings of 24th Annual International Conference on Machine Learning, 2007
Analysis of different feature spaces
• Pennacchiotti, M. & Zanzotto, F. M. Learning Shallow Semantic Rules for Textual
Entailment, Poceeding of International Conference RANLP - 2007, 2007
A comprehensive description
• Zanzotto, F. M.; Pennacchiotti, M. & Moschitti, A. A Machine Learning Approach to
Textual Entailment Recognition, NATURAL LANGUAGE ENGINEERING, 2009
© F.M.Zanzotto
University of Rome “Tor Vergata”
Learning RTE Systems on Rule Spaces
Adding Distributional Semantics
•
Mehdad, Y.; Moschitti, A. & Zanzotto, F. M. Syntactic/Semantic Structures for Textual Entailment
Recognition, Human Language Technologies: The 2010 Annual Conference of the North American
Chapter of the Association for Computational Linguistics, 2010
A valid kernel with an efficient algorithm
•
Zanzotto, F. M. & Dell'Arciprete, L. Efficient kernels for sentence pair classification, Conference on
Empirical Methods on Natural Language Processing, 2009
•
Zanzotto, F. M.; Dell'arciprete, L. & Moschitti, A. Efficient Graph Kernels for Textual Entailment
Recognition, FUNDAMENTA INFORMATICAE
Applications
•
Zanzotto, F. M.; Pennacchiotti, M. & Tsioutsiouliklis, K. Linguistic Redundancy in Twitter,
Proceedings of 2011 Conference on Empirical Methods on Natural Language Processing (EmNLP),
2011
Extracting RTE Corpora
•
Zanzotto, F. M. & Pennacchiotti, M. Expanding textual entailment corpora from Wikipedia using cotraining, Proceedings of the COLING-Workshop on The People's Web Meets NLP: Collaboratively
Constructed Semantic Resources, 2010
Learning Verb Relations
•
Zanzotto, F. M.; Pennacchiotti, M. & Pazienza, M. T. Discovering asymmetric entailment relations
between verbs using selectional preferences, ACL-44: Proceedings of the 21st International
Conference on Computational Linguistics and the 44th annual meeting of the Association for
Computational Linguistics
© F.M.Zanzotto
University of Rome “Tor Vergata”
References
[1] Rod Adams. Textual entailment through extended lexical overlap. In Proceedingsof the Second PASCAL Challenges Workshop on Recognizing Textual
Entailment,2006.
[2] E. Akhmatova. Textual entailment resolution via atomic propositions. In Proceed-ings of RTE 2005, 2005.
[3] R. Bar-Haim, J. Berant, I. Dagan, I. Greenthal, S. Mirkin, E. Shnarch, andI. Szpektor. Efficient semantic deduction and approximate matching over compactparse
forests. In Text Analysis Conference (TAC), 2009.
[4] Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, BernardoMagnini, and Idan Szpektor. The second pascal recognising textual
entailmentchallenge. In Proceedings of the Second PASCAL Challenges Workshop on Recog-nising Textual Entailment. Venice, Italy, 2006.
[5] Roy Bar-Haim, Ido Dagan, Iddo Greental, and Eyal Shnarch. Semantic inferenceat the lexical-syntactic level. In Proceedings of the 22nd National Conference
onArtificial Intelligence (AAAI), Vancouver, Canada, July 2007.
[6] Roy Bar-haim, Ido Dagan, Iddo Greental, Idan Szpektor, and Moshe Friedman.Semantic inference at the lexical-syntactic level. In Proceedings of AAAI,
pages131-136, 2007.
[7] Roy Bar-Haim, Ido Dagan, Iddo Greental, Idan Szpektor, and Moshe Friedman.Semantic inference at the lexical-syntactic level for textual entailment
recognition.In Proceedings of the ACL-PASCAL Workshop on Textual Entailment andParaphrasing, pages 131-136, Prague, June 2007. Association for
ComputationalLinguistics.
[8] Roy Bar-Haim, Idan Szpecktor, and Oren Glickman. Definition and analysis ofintermediate entailment levels. In Proceedings of the ACL Workshop on
EmpiricalModeling of Semantic Equivalence and Entailment, pages 55-60. Association forComputational Linguistics, Ann Arbor, Michigan, June 2005.
[9] Regina Barzilay and Kathleen McKeown. Extracting paraphrases from a parallelcorpus. In Proceedings of the 39th ACL Meeting. Toulouse, France, 2001.
[10] Samuel Bayer, John Burger, Lisa Ferro, John Henderson, and Alexander Yeh.Mitre's submissions to the eu pascal rte challenge. In Proceedings of RTE
2005,160 Mesh Refinement for Time-Domain Numerical Electromagnetics2005.
[11] Roni Ben Aharon, Idan Szpektor, and Ido Dagan. Generating entailment rulesfrom framenet. In Proceedings of the ACL 2010 Conference Short Papers,
pages241-246, Uppsala, Sweden, July 2010. Association for Computational Linguistics.
[12] Luisa Bentivogli, Elena Cabrio, Ido Dagan, Danilo Giampiccolo, Medea Lo Leggio,and Bernardo Magnini. Building textual entailment specialized data sets:
amethodology for isolating linguistic phenomena relevant to inference. In NicolettaCalzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk,
SteliosPiperidis, Mike Rosner, and Daniel Tapias, editors, LREC. European LanguageResources Association, 2010.
[13] Luisa Bentivogli, Elena Cabrio, Ido Dagan, Danilo Giampiccolo, Medea Lo Leggio,and Bernardo Magnini. Building textual entailment specialized data sets:
amethodology for isolating linguistic phenomena relevant to inference. In NicolettaCalzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph
Mariani,Jan Odijk, Stelios Piperidis, Mike Rosner, and Daniel Tapias, editors, Proceedingsof the Seventh conference on International Language Resources and
Evaluation(LREC'10), Valletta, Malta, may 2010. European Language Resources Association(ELRA).
[14] Luisa Bentivogli, Peter Clark, Ido Dagan, Hoa T. Dang, and Danilo Giampiccolo.The sixth PASCAL recognizing textual entailment challenge. In The Text
AnalysisConference (TAC 2010), 2010.
[15] Luisa Bentivogli, Peter Clark, Ido Dagan, Hoa T. Dang, and Danilo Giampiccolo.The seventh PASCAL recognizing textual entailment challenge. In The
TextAnalysis Conference (TAC 2011), to appear, 2011.
© F.M.Zanzotto
University of Rome “Tor Vergata”
References
[16] Luisa Bentivogli, Ido Dagan, Hoa T. Dang, Danilo Giampiccolo, and BernardoMagnini. The fifth PASCAL recognizing textual entailment
challenge. In The TextAnalysis Conference (TAC 2009), 2009.
[17] Jonathan Berant, Ido Dagan, and Jacob Goldberger. Global learning of focusedentailment graphs. In Proceedings of the 48th Annual Meeting
of the Associationfor Computational Linguistics, pages 1220-1229, Uppsala, Sweden, July 2010. Associationfor Computational Linguistics.
[18] Richard Bergmair. A proposal on evaluation measures for rte. In Proceedings ofthe 2009 Workshop on Applied Textual Inference, pages 10-17,
Suntec, Singapore,August 2009. Association for Computational Linguistics.
[19] C. M. Bishop. Neural networks for pattern recognition. Oxford University Press,Oxford, UK, 1996.
[20] Johan Bos and Katja Markert. When logical inference helps determining textual entailment (and when it doesn't). In Proceedings of the Second
PASCAL Chal-lenges Workshop on Recognizing Textual Entailment, 2006.
[21] R. Braz, R. Girju, V. Punyakanok, D. Roth, and M. Sammons. An inference modelfor semantic entailment in natural language. In Proceedings
of the National Con-ference on Artificial Intelligence (AAAI), pages 1678-1679, 2005.
[22] C. Brockett. Aligning the rte 2006 corpus. Technical Report MSR-TR-2007-77,Microsoft Research, 2007.
[23] John Burger and Lisa Ferro. Generating an entailment corpus from news headlines.In Proceedings of the ACL Workshop on Empirical
Modeling of Semantic Equiv-alence and Entailment, pages 49-54. Association for Computational Linguistics,Ann Arbor, Michigan, June 2005.
[24] Chris Callison-Burch, Philipp Koehn, and Miles Osborne. Improved statisticalmachine translation using paraphrases. In Proceedings of the
Human LanguageTechnology Conference of the NAACL, Main Conference, pages 17-24, New YorkCity, USA, June 2006. Association for
Computational Linguistics.
[25] Jean Carletta. Assessing agreement on classification tasks: The kappa statistic.Computational Linguistics, 22(2):249-254, 1996.
[26] Bob Carpenter. The Logic of Typed Feature Structures. Cambridge UniversityPress, Cambridge, England, 1992.
[27] Xavier Carreras and Lluìs Marquez. Introduction to the CoNLL-2005 Shared Task:Semantic Role Labeling. In Proceedings of the Ninth
Conference on ComputationalNatural Language Learning (CoNLL-2005), pages 152-164, Ann Arbor, Michigan,June 2005. Association for
Computational Linguistics.
[28] Asli Celikyilmaz, Marcus Thint, and Zhiheng Huang. A graph-based semisupervisedlearning for question-answering. In Proc. of the Annual
Meeting ofthe ACL, pages 719-727, Suntec, Singapore, August 2009. Association for ComputationalLinguistics.
[29] M. Chang, D. Goldwasser, D. Roth, and V. Srikumar. Discriminative learning overconstrained latent representations. In Proc. of the Annual
Meeting of the NorthAmerican Association of Computational Linguistics (NAACL), Jun 2010.
[30] M. Chang, L. Ratinov, and D. Roth. Constraints as prior knowledge. In ICMLWorkshop on Prior Knowledge for Text and Language
Processing, pages 32-39,July 2008.
© F.M.Zanzotto
University of Rome “Tor Vergata”
References
[31] M. Chang, V. Srikumar, D. Goldwasser, and D. Roth. Structured output learningwith indirect supervision. In Proc. of the
International Conference on MachineLearning (ICML), 2010.
[32] Ming-Wei Chang, Dan Goldwasser, Dan Roth, and Vivek Srikumar. Discriminative162 Mesh Refinement for Time-Domain
Numerical Electromagneticslearning over constrained latent representations. In Proceedings of HLT: NAACL,pages 429-437, 2010.
[33] E. Charniak. A maximum-entropy-inspired parser. In Proceedings of NAACL 2000,pages 132{139, Seattle, Washington, 2000.
[34] Gennaro Chierchia and Sally McConnell-Ginet. Meaning and Grammar: An in-troduction to Semantics. MIT press, Cambridge,
MA, 2001.
[35] Timothy Chklovski and Patrick Pantel. VerbOcean: Mining the Web for Fine-Grained Semantic Verb Relations. In Proceedings of
Conference on EmpiricalMethods in Natural Language Processing (EMNLP-04), pages 33{40, 2004.
[36] Timoty Chklovski and Patrick Pantel. VerbOCEAN: Mining the web for finegrainedsemantic verb relations. In Proceedings of the
2004 Conference on Empir-ical Methods in Natural Language Processing. Barcellona, Spain, 2004.
[37] Naom Chomsky. Aspect of Syntax Theory. MIT Press, Cambridge, Massachussetts,1957.
[38] Kenneth Ward Church and Patrick Hanks. Word association norms, mutual informationand lexicography. In Proceedings of the
27th Annual Meeting of theAssociation for Computational Linguistics (ACL). Vancouver, Canada, 1989.
[39] Philipp Cimiano, Andreas Hotho, and Steffen Staab. Learning concept hierarchiesfrom text corpora using formal concept analysis.
Journal of Artificial Intelligenceresearch, 24:305-339, 2005.
[40] P. Clark, W. R. Murray, J. Thompson, P. Harrison, J. Hobbs, and C. Fellbaum.On the role of lexical and world knowledge in RTE3. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 54-59, 2007.
[41] Peter Clark and Phil Harrison. An Inference-Based Approach to Recognizing Entailment.In Text Analysis Conference (TAC),
pages 63-72, 2009.
[42] Peter Clark and Phil Harrison. An inference-based approach to recognizing entailment.In Notebook papers and Results, Text
Analysis Conference (TAC), pages63-72, 2009.
[43] Peter Clark, Phil Harrison, John Thompson, William Murray, Jerry Hobbs, andChristiane Fellbaum. On the role of lexical and
world knowledge in rte3. In Pro-ceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing,pages 54-59,
Prague, June 2007. Association for Computational Linguistics.
[44] Michael Collins and Nigel Duffy. New ranking algorithms for parsing and tagging:Kernels over discrete structures, and the voted
perceptron. In Proceedings ofACL02. 2002.
[45] Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, JanJaspars, Hans Kamp, David Milward, Manfred
Pinkal, Massimo Poesio, and Steve Pulman. Using the framework. Technical report, 1996.
© F.M.Zanzotto
University of Rome “Tor Vergata”
References
[46] Courtney Corley and Rada Mihalcea. Measuring the semantic similarity of texts.In Proc. of the ACL Workshop on Empirical
Modeling of Semantic Equivalence andEntailment, pages 13-18. Association for Computational Linguistics, Ann Arbor,Michigan, June
2005.
[47] C. Cortes and V. Vapnik. Support vector networks. Machine Learning, 20:1-25,1995.
[48] Nello Cristianini and John Shawe-Taylor. An Introduction to Support Vector Ma-chines and Other Kernel-based Learning
Methods. Cambridge University Press,March 2000.
[49] C. Cumby and D. Roth. On kernel methods for relational learning. In Proc. of theInternational Conference on Machine Learning
(ICML), pages 107-114, 2003.
[50] I. Dagan and O. Glickman. Probabilistic textual entailment: Generic applied modelingof language variability. In Learning Methods
for Text Understanding andMining, Grenoble, France, 2004.
[52] Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. Recognizing textualentailment: Rational, evaluation and approaches.
Natural Language Engineering,15(Special Issue 04):i-xvii, 2009.
[53] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textualentailment challenge. In Quionero-Candela et
al., editor, LNAI 3944: MLCW 2005,pages 177-190. Springer-Verlag, Milan, Italy, 2006.
[54] Marie-Catherine de Marneffe, Trond Grenager, Bill MacCartney, Daniel Cer,Daniel Ramage, Chloe Kiddon, and Christopher D.
Manning. Aligning semanticgraphs for textual inference and machine reading. In AAAI Spring Symposiumat Stanford 2007, 2007.
[55] Marie-Catherine de Marneffe, Bill MacCartney, Trond Grenager, Daniel Cer, AnnaRafferty, and Christopher D. Manning.
Learning to distinguish valid textual entailments.In Bernardo Magnini and Ido Dagan, editors, Proceedings of the SecondPASCAL
Recognizing Textual Entailment Challenge, Venice, Italy, 2006. Springer-Verlag.
[56] Marie-Catherine de Marneffe and Christopher Manning. The stanford typed dependenciesrepresentation. In COLING Workshop
on Cross-framework and Cross-domain Parser Evaluation, 2008.
[57] Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning.Finding contradictions in text. In Proceedings of
ACL-08: HLT, pages 1039-1047,164 Mesh Refinement for Time-Domain Numerical ElectromagneticsColumbus, Ohio, June 2008.
Association for Computational Linguistics.
[58] Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. L, and RichardHarshman. Indexing by latent semantic
analysis. Journal of the American Societyfor Information Science, 41:391-407, 1990.
[59] Q. Do and D. Roth. Constraints based taxonomic relation classifier. In EMNLP,Massachussetts, USA, 10 2010.
[60] Quang Do, Dan Roth, Mark Sammons, Yuancheng Tu, and V.G.Vinod Vydiswaran.Robust, Light-weight Approaches to compute
Lexical Similarity. ComputerScience Research and Technical Reports, University of Illinois, 2010.
© F.M.Zanzotto
University of Rome “Tor Vergata”
References
[61] Bill Dolan, Chris Quirk, and Chris Brockett. Unsupervised construction of largeparaphrase corpora: Exploiting massively parallel
news sources. In Proceedings ofColing 2004, pages 350-356. COLING, Geneva, Switzerland, Aug 23-Aug 27 2004.
[62] FRANCESCA FALLUCCHI and FABIO MASSIMO ZANZOTTO. Inductiveprobabilistic taxonomy learning using singular
value decomposition. Natural Lan-guage Engineering, 17(01):71-94, 2011.
[63] C. Fellbaum. WordNet: An Electronic Lexical Database. MIT Press, 1998.
[64] Christiane Fellbaum, editor. WordNet: An Electronic Lexical Database. MIT Press,Cambridge, MA, 1998.
[65] Charles John Fillmore, Christopher R Johnson, and M R L Petruck. Backgroundto framenet. International Journal of
Lexicography, 16(3):235-250, 2003.
[66] Abraham Fowler, Bob Hauser, Daniel Hodges, Ian Niles, Adrian Novischi, andJans Stephan. Applying cogex to recognize textual
entailment. In Proceedings ofRTE 2005, 2005.
[67] Konstantina Garoufi. Towards a better understanding of applied textual entailment:Annotation and evaluation of the rte-2 dataset.
Master's thesis, SaarlandUniversity, 2008.
[68] Thomas G•artner. A survey of kernels for structured data. SIGKDD Explorations,2003.
[69] Maayan Geffet and Ido Dagan. The distributional inclusion hypotheses and lexicalentailment. In ACL '05: Proceedings of the 43rd
Annual Meeting on Associationfor Computational Linguistics, pages 107-114, Morristown, NJ, USA, 2005. Associationfor
Computational Linguistics.
[70] Danilo Giampiccolo, Hoa T. Dang, Bernardo Magnini, Ido Dagan, and Bill Dolan.The fourth PASCAL recognizing textual
entailment challenge. In The Text Anal-ysis Conference (TAC 2008), 2008.
[71] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal recognizing textual entailment challenge. In
Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9. Association for Computational
Linguistics, Prague, June 2007.
[72] Oren Glickman and Ido Dagan. Probabilistic textual entailment: Generic appliedmodeling of language variability. In Proceedings
of the Workshop on LearningMethods for Text Understanding and Mining. Grenoble, France, 2004.
[73] Oren Glickman, Ido Dagan, and Moshe Koppel. A lexical alignment model forprobabilistic textual entailment. In Joaquin
Qui~nonero Candela, Ido Dagan,Bernardo Magnini, and Florence d'Alche Buc, editors, MLCW, volume 3944 ofLecture Notes in
Computer Science, pages 287-298. Springer, 2005.
[74] Isabelle Guyon and Andre Elisseeff. An introduction to variable and feature selection.Journal of Machine Learning Research,
3:1157-1182, March 2003.
[75] Sanda Harabagiu and Andrew Hickl. Methods for using textual entailment in opendomainquestion answering. In Proceedings of
the 21st International Conferenceon Computational Linguistics and 44th Annual Meeting of the Association forComputational
Linguistics, pages 905-912, 2006.
© F.M.Zanzotto
University of Rome “Tor Vergata”
References
[76] Sanda Harabagiu and Andrew Hickl. Methods for using textual entailment in opendomainquestion answering. In Proceedings of the 21st International
Conference onComputational Linguistics and 44th Annual Meeting of the Association for Com-putational Linguistics, pages 905-912, Sydney, Australia, July 2006.
Associationfor Computational Linguistics.
[77] Sanda Harabagiu and Andrew Hickl. Methods for Using Textual Entailment inOpen-Domain Question Answering. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Associationfor Computational Linguistics, pages 905-912, Sydney, Australia, July 2006.
Associationfor Computational Linguistics.
[78] Sanda Harabagiu, Andrew Hickl, and Finley Lacatusu. Satisfying informationneeds with multi-document summaries. Information Processing &
Management,43(6):1619 - 1642, 2007. Text Summarization.
[79] Zellig Harris. Distributional structure. In Jerrold J. Katz and Jerry A. Fodor,editors, The Philosophy of Linguistics. Oxford University Press, New York, 1964.
[80] P. Harrison and M. Maxwell. A new implementation of gpsg. In Proceedings of the6th Canadian Conference on AI (CSCSI'86), pages 78-83, 1986.
[81] Marti A. Hearst. Automatic acquisition of hyponyms from large text corpora. InProceedings of the 15th International Conference on Computational
Linguistics(CoLing-92). Nantes, France, 1992.
[82] Andrew Hickl. Using discourse commitments to recognize textual entailment. In166 Mesh Refinement for Time-Domain Numerical
ElectromagneticsProceedings of the 22nd COLING Conference, 2008.
[83] Andrew Hickl and Jeremy Bensley. A Discourse Commitment-Based Frameworkfor Recognizing Textual Entailment. In Proceedings of the ACL-PASCAL
Work-shop on Textual Entailment and Paraphrasing, pages 171-176, 2007.
[84] Andrew Hickl, John Williams, Jeremy Bensley, Kirk Roberts, Bryan Rink, andYing Shi. Recognizing textual entailment with LCCs GROUNDHOG system.
InBernardo Magnini and Ido Dagan, editors, Proceedings of the Second PASCALRecognizing Textual Entailment Challenge, Venice, Italy, 2006. Springer-Verlag.
[85] Andrew Hickl, John Williams, Jeremy Bensley, Kirk Roberts, Bryan Rink, andYing Shi. Recognizing textual entailment with LCCs GROUNDHOG system.
InBernardo Magnini and Ido Dagan, editors, Proceedings of the Second PASCALRecognizing Textual Entailment Challenge. Springer-Verlag, Venice, Italy, 2006.
[86] J. R. Hobbs, M. Stickel, P. Martin, and D. Edwards. Interpretation as abduction.In Proceedings of the 26th Annual Meeting of the Association for
ComputationalLinguistics (ACL), pages 95-103, 1988.
[87] A. Iftene and M.-A. Moruz. Uaic participation at rte5. In Notebook papers andResults, Text Analysis Conference (TAC), pages 367-376, 2009.
[88] Christian Jacquemin. Spotting and Discovering Terms through Natural Lan-guage Processing. Massachusetts Institue of Technology, Cambrige,
Massachussetts,USA, 2001.
[89] Jay J. Jiang and David W. Conrath. Semantic similarity based on corpus statisticsand lexical taxonomy. In Proc. of the 10th ROCLING, pages 132-139.
Tapei,Taiwan, 1997.
[90] Valentin Jijkoun and Maarten de Rijke. Recognizing textual entailment using lexicalsimilarity. In Proceedings of the 1st Pascal Challenge Workshop,
Southampton,UK, 2005.
© F.M.Zanzotto
University of Rome “Tor Vergata”
References
[91] Johannes K•obler, Uwe Sch•oning, and Jacobo Toran. The graph isomorphism prob-lem: its structural complexity. Birkhauser Verlag, Basel, Switzerland,
Switzerland,1993.
[92] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico,Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard
Zens,Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: opensource toolkit for statistical machine translation. In ACL '07: Proceedings of
the45th Annual Meeting of the ACL on Interactive Poster and Demonstration Ses-sions, pages 177-180, Morristown, NJ, USA, 2007. Association for
ComputationalLinguistics.
[93] Milen Koulyekov and Bernardo Magnini. Recognizing textual entailment with treeBibliography 167edit distance algorithms. In Proceedings of RTE 2005, 2005.
[94] D. Lin. Automatic retrieval and clustering of similar words. In Proceedings ofCOLING/ACL-98, pages 768-774, 1998.
[95] D. Lin and P. Pantel. Induction of semantic classes from natural language text.In Proceedings of ACM SIGKDD Conference on Knowledge Discovery and
DataMining, pages 317-322, 2001.
[96] Dekang Lin. Dependency-based evaluation of minipar. In Proceedings of the Work-shop on Evaluation of Parsing Systems at LREC 1998, Granada, Spain,
1998.
[97] Dekang Lin and Patrick Pantel. DIRT-discovery of inference rules from text. InProceedings of the ACM Conference on Knowledge Discovery and Data
Mining(KDD-01), San Francisco, CA, 2001.
[98] Dekang Lin and Patrick Pantel. DIRT: discovery of inference rules from text. InKnowledge Discovery and Data Mining, pages 323-328, 2001.
[99] B. MacCartney, T. Grenager, and M. de Marneffe. Learning to recognize featuresof valid textual entailments. In Proceedings of RTE-NAACL 2006, 2006.
[100] Bill MacCartney, Michel Galley, and Christopher D. Manning. A phrase-basedalignment model for natural language inference. In Proceedings of the
Conferenceon Empirical Methods in Natural Language Processing (EMNLP-2008), 2008.
[101] BillMacCartney and Christopher D. Manning. An extended model of natural logic.In The Eighth International Conference on Computational Semantics
(IWCS-8),Tilburg, Netherlands, 2009.
[102] J. B. MacQueen. Some methods for classification and analysis of multivariateobservations. In L. M. Le Cam and J. Neyman, editors, Proc. of the fifth
BerkeleySymposium on Mathematical Statistics and Probability, volume 1, pages 281-297.University of California Press, 1967.
[103] M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz. Building a large annotatedcorpus of english: The penn treebank. Computational Linguistics, 19:313330, 1993.
[104] Y. Mehdad, M. Negri, and M. Federico. Towards cross-lingual textual entailment.In Human Language Technologies: The 2010 Annual Conference of the
NorthAmerican Chapter of the Association for Computational Linguistics, pages 321-324. Association for Computational Linguistics, 2010.
[105] Y. Mehdad, F. M. Zanzotto, and A. Moschitti. Semker: Syntactic/semantic kernelsfor recognizing textual entailment. In Notebook papers and Results, Text
AnalysisConference (TAC), pages 259-265, 2009.
© F.M.Zanzotto
University of Rome “Tor Vergata”
References
[106] Yashar Mehdad, Matteo Negri, Elena Cabrio, Milen Kouylekov, and BernardoMagnini. EDITS: An Open Source Framework for Recognizing Textual
Entail168Mesh Refinement for Time-Domain Numerical Electromagneticsment. In Text Analysis Conference (TAC), pages 169-178, 2009.
[107] Yashar Mehdad, Matteo Negri, Elena Cabrio, Milen Kouylekov, and BernardoMagnini. Edits: An open source framework for recognizing textual entailment.
InNotebook papers and Results, Text Analysis Conference (TAC), pages 169-178,2009.
[108] G. Miller, R. Beckwith, C. Fellbaum, D. Gross, and K.J. Miller. Wordnet: An onlinelexical database. International Journal of Lexicography, 3(4):235-312,
1990.
[109] George A. Miller. WordNet: A lexical database for English. Communications ofthe ACM, 38(11):39-41, November 1995.
[110] Shachar Mirkin, Ido Dagan, and Sebastian Pado. Assessing the role of discoursereferences in entailment inference. In Proceedings of the 48th Annual Meeting
ofthe Association for Computational Linguistics, pages 1209-1219, Uppsala, Sweden,July 2010. Association for Computational Linguistics.
[111] Shachar Mirkin, Lucia Specia, Nicola Cancedda, Ido Dagan, Marc Dymetman,and Idan Szpektor. Source-language entailment modeling for translating
unknownterms. In Proceedings of the Joint Conference of the 47th Annual Meeting of theACL and the 4th International Joint Conference on Natural Language
Processingof the AFNLP, pages 791-799, Suntec, Singapore, August 2009. Association forComputational Linguistics.
[112] Shachar Mirkin, Lucia Specia, Nicola Cancedda, Ido Dagan, Marc Dymetman,and Idan Szpektor. Source-language entailment modeling for translating
unknownterms. In Proceedings of ACL/AFNLP, pages 791-799, Suntec, Singapore, August2009. Association for Computational Linguistics.
[113] D. Moldovan, C. Clark, S. Harabagiu, and S. Maiorano. Cogex: A logic prover forquestion answering. In Proceedings of HLT-NAACL 2003, 2003.
[114] A. Moschitti and F. Zanzotto. Fast and effective kernels for relational learningfrom texts. In Zoubin Ghahramani, editor, Proc. of the International
Conferenceon Machine Learning (ICML), pages 649-656. Omnipress, 2007.
[115] Alessandro Moschitti. Making tree kernels practical for natural language learning.In Proceedings of EACL'06. Trento, Italy, 2006.
[116] Alessandro Moschitti and Fabio Massimo Zanzotto. Fast and effective kernels forrelational learning from texts. In Proceedings of the International Conference
ofMachine Learning (ICML). Corvallis, Oregon, 2007.
[117] Eamonn Newman, Nicola Stokes, John John Dunnion, and Joe Carthy. Ucd iirgapproach to the textual entailment challenge. In Proceedings of the 1st
PascalChallenge Workshop, Southampton, UK, 2005.
[118] Rodney d. Nielsen, Wayne Ward, and James h. Martin. Recognizing entailmentBibliography 169in intelligent tutoring systems*. Nat. Lang. Eng., 15:479-501,
October 2009.
[119] Sebastian Pado, Marie-Catherine de Marneffe, Bill MacCartney, Anna N. Rafferty,Eric Yeh, and Christopher D. Manning. Deciding entailment and
contradictionwith stochastic and edit distance-based alignment. In Text Analysis Conference(TAC), 2008.
[120] Sebastian Pado, Michel Galley, Dan Jurafsky, and Chris Manning. Robust machinetranslation evaluation with entailment features. In Proceedings of the
JointConference of the 47th Annual Meeting of the ACL and the 4th International JointConference on Natural Language Processing of the AFNLP: Volume 1 Volume1, ACL '09, pages 297-305, Stroudsburg, PA, USA, 2009. Association for ComputationalLinguistics.
© F.M.Zanzotto
University of Rome “Tor Vergata”
References
[121] Sebastian Pado, Michel Galley, Dan Jurafsky, and Christopher D. Manning. Robustmachine translation evaluation with entailment features. In Proceedings
ofACL/AFNLP, pages 297-305, Suntec, Singapore, August 2009. Association forComputational Linguistics.
[122] Patrick Pantel and Marco Pennacchiotti. Espresso: Leveraging generic patternsfor automatically harvesting semantic relations. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meetingof the Association for Computational Linguistics, pages 113-120. Association
forComputational Linguistics, Sydney, Australia, July 2006.
[123] M. Pazienza, M. Pennacchiotti, and F. Zanzotto. Terminology extraction: An analysisof linguistic and statistical approaches. In S. Sirmakessis, editor,
KnowledgeMining Series: Studies in Fuzziness and Soft Computing. Springer Verlag, 2005.
[124] Ted Pedersen, Siddharth Patwardhan, and Jason Michelizzi. Wordnet::similarity -measuring the relatedness of concepts. In Proc. of 5th NAACL. Boston, MA,
2004.
[125] Anselmo Pe~nas, Alvaro Rodrigo, Valentìn Sama, and Felisa Verdejo. Overviewof the answer validation exercise 2006. In Carol Peters, Paul Clough, Fredric
C.Gey, Jussi Karlgren, Bernardo Magnini, Douglas W. Oard, Maarten de Rijke,and Maximilian Stempfhuber, editors, CLEF, volume 4730 of Lecture Notes
inComputer Science, pages 257-264. Springer, 2006.
[126] Anselmo Pe~nas, Alvaro Rodrigo, and Felisa Verdejo. Overview of the answer validationexercise 2007. In Carol Peters, Valentin Jijkoun, Thomas Mandl,
HenningM•uller, Douglas W. Oard, Anselmo Pe~nas, Vivien Petras, and Diana Santos, editors,CLEF, volume 5152 of Lecture Notes in Computer Science, pages
237-248.Springer, 2007.
[127] V. Punyakanok, D. Roth, and W. Yih. Natural language inference via dependencytree mapping: An application to question answering. In submission, 2004.170
Mesh Refinement for Time-Domain Numerical Electromagnetics
[128] J. Quinlan. C4:5:programs for Machine Learning. Morgan Kaufmann, San Mateo,1993.
[129] R. Raina, A. Ng, and C. Manning. Robust textual inference via learning andabductive reasoning. In Proceedings of AAAI 2005, 2005.
[130] L. Ratinov, D. Roth, D. Downey, and M. Anderson. Local and global algorithmsfor disambiguation to wikipedia. In Proc. of the Annual Meeting of the
Associationof Computational Linguistics (ACL), 2011.
[131] Deepak Ravichandran and Eduard Hovy. Learning surface text patterns for aquestion answering system. In Proceedings of the 40th ACL Meeting.
Philadelphia,Pennsilvania, 2002.
[132] Philip Resnik. Selection and Information: A Class-Based Approach to Lexical Re-lationships. PhD thesis, Department of Computer and Information Science,
Universityof Pennsylvania, 1993.
[133] Harold R. Robison. Computer-detectable semantic structures. Information Storageand Retrieval, 6(3):273-288, 1970.
[134] Alvaro Rodrigo, Anselmo Pe~nas, and Felisa Verdejo. Evaluating question answeringvalidation as a classification problem. Language Resources and
Evaluation,pages 1-9, March 2011.
[135] Alvaro Rodrigo, Anselmo Pe~nas, and Felisa Verdejo. Overview of the answer validationexercise 2008. In Carol Peters, Thomas Deselaers, Nicola Ferro, Julio
Gonzalo,Gareth J. F. Jones, Mikko Kurimo, Thomas Mandl, Anselmo Pe~nas, andVivien Petras, editors, CLEF, volume 5706 of Lecture Notes in Computer
Science,pages 296-313. Springer, 2008.
© F.M.Zanzotto
University of Rome “Tor Vergata”
References
[136] Lorenza Romano, Milen Kouylekov, Idan Szpektor, Ido Dagan, and AlbertoLavelli. Investigating a generic paraphrase-based approach for relation
extraction.In EACL, 2006.
[137] F. Rosenblatt. The perceptron: A probabilistic model for information storage andorganization in the brain. Psych. Rev., 65:386-407, 1958. (Reprinted in
Neurocom-puting (MIT Press, 1988).).
[138] D. Roth and W. Yih. Global inference for entity and relation identification via alinear programming formulation. In Lise Getoor and Ben Taskar, editors, Introduction to Statistical Relational Learning. MIT Press, 2007.
[139] Dan Roth, Mark Sammons, and V.G.Vinod Vydiswaran. A Framework for EntailedRelation Recognition. In Proc. of the Annual Meeting of the Association
ofComputational Linguistics (ACL), Singapore, August 2009. Association for ComputationalLinguistics.
[140] Mark Sammons, V.G.Vinod Vydiswaran, and Dan Roth. \Ask not what TextualBibliography 171Entailment can do for you...". In ACL, Uppsala, Sweden, July
2010. Associationfor Computational Linguistics.
[141] Mark Sammons, V.G.Vinod Vydiswaran, T. Vieira, N. Johri, M.-W. Chang,D. Goldwasser, V. Srikumar, G. Kundu, Y. Tu, K. Small, J. Rule, Q. Do, andD.
Roth. Relation Alignment for Textual Entailment Recognition. In Text Analy-sis Conference (TAC), 2009.
[142] Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003shared task: Language-independent named entity recognition. In Proceedings
ofCoNLL-2003, pages 142-147, 2003.
[143] scar Ferrndez, Christian Spurk, Milen Kouylekov, Iustin Dornescu, Sergio Ferrndez,Matteo Negri, Rubn Izquierdo, David Toms, Constantin Orasan,
GuenterNeumann, Bernardo Magnini, and Jose Luis Vicedo. The qall-me framework: Aspecifiable-domain multilingual question answering architecture. Web
Semantics:Science, Services and Agents on the World Wide Web, 9(2):137 - 145, 2011.
[144] Rion Snow, Daniel Jurafsky, and A. Y. Ng. Semantic taxonomy induction fromheterogenous evidence. In In ACL, pages 801-808, 2006.
[145] Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. Cheap andfast|but is it good?: evaluating non-expert annotations for natural
languagetasks. In Proceedings of the Conference on Empirical Methods in Natural LanguageProcessing, EMNLP '08, pages 254-263, Stroudsburg, PA, USA, 2008.
Associationfor Computational Linguistics.
[146] Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. Yago: A Core ofSemantic Knowledge. In 16th international World Wide Web conference
(WWW2007), New York, NY, USA, 2007. ACM Press.
[147] Jana Z. Sukkarieh and Svetlana Stoyanchev. Automating model building in crater.In Proceedings of the 2009 Workshop on Applied Textual Inference,
TextInfer'09, pages 61-69, Stroudsburg, PA, USA, 2009. Association for ComputationalLinguistics.
[148] Idan Szpektor and Ido Dagan. Learning entailment rules for unary templates. InProceedings of the 22nd International Conference on Computational
Linguistics(Coling 2008), pages 849-856, Manchester, UK, August 2008. Coling 2008 OrganizingCommittee.
[149] Idan Szpektor, Ido Dagan, Roy Bar-Haim, and Jacob Goldberger. Contextualpreferences. In Proceedings of ACL-08: HLT, pages 683-691, Columbus,
Ohio,June 2008. Association for Computational Linguistics.
[150] Idan Szpektor, Hristo Tanev, Ido Dagan, and Bonaventura Coppola. Scaling webbasedacquisition of entailment relations. In Proceedings of the 2004
Conference172 Mesh Refinement for Time-Domain Numerical Electromagneticson Empirical Methods in Natural Language Processing. Barcellona, Spain, 2004.
© F.M.Zanzotto
University of Rome “Tor Vergata”
References
[151] L. Tesniere. Elements de syntaxe structural. Klincksiek, Paris, France, 1959.
[152] Lucy Vanderwende and William B. Dolan. What syntax can contribute in the entailmenttask. In Joaquin Qui~nonero Candela, Ido Dagan, Bernardo Magnini,
andFlorence d'Alche Buc, editors, Machine Learning Challenges Workshop, volume3944 of Lecture Notes in Computer Science, pages 205-216. Springer, 2006.
[153] V. N. Vapnik. The Nature of Statistical Learning Theory. Springer-Verlag, NewYork, 1995.
[154] Ellen M. Voorhees and Donna Harman. Overview of the seventh text retrieval conferencetrec-7. In Proceedings of the Seventh Text REtrieval Conference
(TREC-7,pages 1-24, 1998.
[155] Annie Zaenen, Lauri Karttunen, and Richard Crouch. Local textual inference: Canit be defined or circumscribed? In Proc. of the ACL Workshop on Empirical
Mod-eling of Semantic Equivalence and Entailment, pages 31-36, Ann Arbor, Michigan,June 2005. Association for Computational Linguistics.
[156] A. Zanzotto, F.M. adn Moschitti, M. Pennacchiotti, and M.T. Pazienza. Learningtextual entailment from examples. In Bernardo Magnini and Ido Dagan,
editors,Proceedings of the Second PASCAL Recognizing Textual Entailment Challenge.Springer-Verlag, Venice, Italy, 2006.
[157] Fabio Massimo Zanzotto and Lorenzo Dell'Arciprete. Efficient kernels for sentencepair classification. In Conference on Empirical Methods on Natural
Language Pro-cessing, pages 91-100, 6-7 August 2009.
[158] Fabio Massimo Zanzotto, Lorenzo Dell'arciprete, and Alessandro Moschitti. Effi-cient graph kernels for textual entailment recognition. FUNDAMENTA
INFOR-MATICAE, 107 (2-3):199-222, 2011.
[159] Fabio Massimo Zanzotto and Alessandro Moschitti. Automatic learning of textualentailments with cross-pair similarities. In Proceedings of the 21st Coling and
44thACL, pages 401-408. Sydney, Australia, July 2006.
[160] Fabio Massimo Zanzotto and Marco Pennacchiotti. Expanding textual entailmentcorpora fromwikipedia using co-training. In Proceedings of the 2nd Workshop
onThe People's Web Meets NLP: Collaboratively Constructed Semantic Resources,pages 28-36, Beijing, China, August 2010. Coling 2010 Organizing Committee.
[161] Fabio Massimo Zanzotto, Marco Pennacchiotti, and Alessandro Moschitti. Amachine learning approach to textual entailment recognition. NATURAL LANGUAGE ENGINEERING, 15-04:551-582, 2009.
[162] Fabio Massimo Zanzotto, Marco Pennacchiotti, and Maria Teresa Pazienza. Discoveringasymmetric entailment relations between verbs using selectional
preferBibliography173ences. In Proceedings of the 21st International Conference on Computational Lin-guistics and 44th Annual Meeting of the Association for
Computational Linguis-tics, pages 849-856. Association for Computational Linguistics, Sydney, Australia,July 2006.
[163] F.M. Zanzotto and A. Moschitti. Automatic learning of textual entailments withcross-pair similarities. In ACL-44: Proceedings of the 21st International
Conferenceon Computational Linguistics and the 44th annual meeting of the Association forComputational Linguistics, pages 401-408, 2006.
[164] Zhi Zhong and Hwee Tou Ng. It makes sense: a wide-coverage word sense disambiguationsystem for free text. In Proceedings of the ACL 2010 System
Demon-strations, ACLDemos '10, pages 78-83, Stroudsburg, PA, USA, 2010. Associationfor Computational
© F.M.Zanzotto