Question Answering Passage Retrieval
Using Dependency Parsing
Hang Cui
Renxu Sun
Keya Li
Min-Yen Kan
Tat-Seng Chua
Department of Computer Science
National University of Singapore
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
1/28
Passage Retrieval in Question
Answering
QA
System
Document
Retrieval
• To narrow down the
search scope
• Can answer questions
with more context
Passage
Retrieval
Answer
Extraction
August 17, 2005
• Lexical density based
• Distance between
question words
Question Answering Passage Retrieval Using
Dependency Parsing
2/28
Density Based Passage Retrieval
Method
• However, density based can err when …
<Question> What percent of the nation's cheese does Wisconsin
produce?
Incorrect: … the number of consumers who mention California when asked
Relationships
between
about cheese has risen by 14 percent, while
the number specifying
Wisconsin
has dropped 16 percent.
matched words differ …
Incorrect: The wry “It's the Cheese” ads, which attribute California's allure to
its cheese _ and indulge in an occasional dig at the Wisconsin stuff'' … sales
of cheese in California grew three times as fast as sales in the nation as a
whole 3.7 percent compared to 1.2 percent, …
Incorrect: Awareness of the Real California Cheese logo, which appears on
about 95 percent of California cheeses, has also made strides.
Correct: In Wisconsin, where farmers produce roughly 28 percent of the
nation's cheese, the outrage is palpable.
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
3/28
Our Solution
• Examine the relationship between words
– Dependency relations
• Exact match of relations for answer
extraction
• Has low recall because same relations are often
phrased differently
• Fuzzy match of dependency relationship
– Statistical similarity of relations
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
4/28
Measuring Sentence Similarity
Sim (Sent1, Sent2) = ?
Sentence 1
Sentence 2
Matched words
Lexical matching
+
Similarity of relations
between matched words
Similarity of individual
relations
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
5/28
Outline
•
•
•
•
•
Extracting and Paring Relation Paths
Measuring Path Match Scores
Learning Relation Mapping Scores
Evaluations
Conclusions
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
6/28
Outline
•
•
•
•
•
Extracting and Paring Relation Paths
Measuring Path Match Scores
Learning Relation Mapping Scores
Evaluations
Conclusions
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
7/28
What Dependency Parsing is Like
• Minipar (Lin, 1998) for dependency
parsing
• Dependency tree
Root
ad
wh
n
he
– Nodes: words/chunks in the sentence
– Edges (ignoring the direction): labeled by
relation types
produce
p re
de
t
percent
subj
p
what
of
Wisconsin
p co
mpn
What percent of the nation's cheese
does Wisconsin produce?
cheese
ge
n
nation
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
8/28
Extracting Relation Paths
• Relation path
– Vector of relations between two nodes in the tree
Root
ad
wh
n
he
produce
p re
de
t
percent
subj
p
what
of
produce < P1: subj > Wisconsin
percent < P2: prep pcomp-n > cheese
Wisconsin
p co
Two constraints for relation paths:
1. Path length (less than 7 relations)
2. Ignore those between two words that
are within a chunk, e.g. New York.
mpn
cheese
ge
n
nation
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
9/28
Paired Paths from Question and
Answer
In Wisconsin, where farmers produce roughly 28
percent of the nation's cheese, the outrage is
palpable.
What percent of the nation's cheese
does Wisconsin produce?
Root
mo
n
ad
wh
i
he
d
Root
produce
in
produce
percent
subj
p
mpn
< P1(Sent) : pcomp-n mod i >
farmers
Wisconsin
Wisconsin
mo d
of
28 percent
p co
t
de
p re
what
su b
j
obj
< P1(Q) : subj >
of
p co
n
p co m p
mp-
-n
Paired Relation Paths
cheese
cheese
n
nation
nation
August 17, 2005
gen
ge
SimRel (Q, Sent) = ∑i,j Sim (Pi (Q), Pj(Sent))
Question Answering Passage Retrieval Using
Dependency Parsing
10/28
Outline
•
•
•
•
•
Extracting and Paring Relation Paths
Measuring Path Match Scores
Learning Relation Mapping Scores
Evaluations
Conclusions
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
11/28
Measuring Path Match Degree
• Employ a variation of IBM Translation
Model 1
– Path match degree (similarity) as translation
probability
• MatchScore (PQ, PS) → Prob (PS | PQ )
• Relations as words
• Why IBM Model 1?
– No “word order” – bag of undirected relations
– No need to estimate “target sentence length”
• Relation paths are determined by the parsing tree
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
12/28
Calculating Translation
Probability (Similarity) of Paths
Given two relation paths from the question and a candidate sentence
Prob( PS | PQ ) 

mn
m
m
n
(S )
(Q )
    Pt ( Reli | Reli )
11
 n 1 i 1
Considering the most probable alignment (finding
the most probable mapped relations)
Prob( PS | PQ ) 

m
n
(S )
(Q )
P
(
Re
l
|
Re
l
)

t
i
A
n
i
i 1
Take logarithm and ignore the constants (for all
sentences, question path length is a constant)
MatchScore ( PS )  Prob( PS | PQ )

August 17, 2005
'
n
n
 log Pt ( Reli( S ) | Rel A(Qi ) )
i 1
MatchScores of paths are
combined to give the sentence’s
relevance to the question.
?
Question Answering Passage Retrieval Using
Dependency Parsing
13/28
Outline
•
•
•
•
•
Extracting and Paring Relation Paths
Measuring Path Match Scores
Learning Relation Mapping Scores
Evaluations
Conclusions
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
14/28
Training and Testing
Testing
Training
Sim ( Q, Sent ) = ?
Prob ( PSent | PQ ) = ?
1.
Mutual
Q - A pairs
Similarityinformation
between (MI)
relationbased
vectors
2. Expectation
Maximization
(EM)between
based Paired Relation Paths
Similarity
individual relations
P ( Rel (Sent) | Rel (Q) ) = ?
Relation Mapping Model
Relation
Mapping
Scores
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
15/28
Approach 1: MI Based
• Measures bipartite co-occurrences in
training path pairs
• Accounts for path length (penalize those
long paths)
• Uses frequencies to approximate mutual
information
Pt ( MI ) ( Reli( S ) | Rel (j Q ) )  log
August 17, 2005
(Q )
(S )
    ( Rel j , Reli )
| Rel (j Q ) |  | Reli( S ) |
Question Answering Passage Retrieval Using
Dependency Parsing
16/28
Approach – 2: EM Based
• Employ the training method from IBM
Model 1
– Relation mapping scores = word translation
probability
– Utilize GIZA to accomplish training
– Iteratively boosting the precision of relation
translation probability
• Initialization – assign 1 to identical
relations and a small constant otherwise
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
17/28
Outline
•
•
•
•
Extracting and Paring Relation Paths
Measuring Path Match Scores
Learning Relation Mapping Scores
Evaluations
–
–
–
–
Can relation matching help?
Can fuzzy match perform better than exact match?
Can long questions benefit more?
Can relation matching work on top of query expansion?
• Conclusions
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
18/28
Evaluation Setup
• Training data
– 3k corresponding path pairs from 10k QA
pairs (TREC-8, 9)
• Test data
– 324 factoid questions from TREC-12 QA task
• Passage retrieval on top 200 relevant
documents by TREC
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
19/28
Comparison Systems
• MITRE –baseline
– Stemmed word overlapping
– Baseline in previous work on passage retrieval evaluation
• SiteQ – top performing density based method
– using 3 sentence window
• NUS
– Similar to SiteQ, but using sentences as passages
• Strict Matching of Relations
– Simulate strict matching in previous work for answer selection
– Counting the number of exactly matched paths
• Relation matching are applied on top of MITRE and NUS
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
20/28
Evaluation Metrics
• Mean reciprocal rank (MRR)
– On the top 20 returned passages
– Measure the mean rank position of the correct
answer in the returned rank list
• Percentage of questions with incorrect
answers
• Precision at the top one passage
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
21/28
Performance Evaluation
• All improvements
are statistically significant (p<0.001)
MRR Comparison
• MI and EM do not make much difference given our training data
el
at
io
St
n(
ri c
M
IT
t_
R
R
el
E
)
at
i
on
R
el
(N
at
U
io
S)
n_
R
el
M
at
I
(M
io
n_
IT
SM
RE
T1
)
(M
R
el
IT
at
R
io
E)
n_
R
el
M
at
I(
io
NU
n_
S
SM
)
T1
(N
U
S)
S
Rel_EM
(NUS)
0.3625
0.4161
0.4218
0.4756
0.4761
N/A
N/A
N/A
+38.26
N/A
N/A
+33.88
N/A
N/A
+49.50
+8.14
+11.69
+81.25
+31.10
+35.41
+108.09
+50.50
+55.43
+110.94
+52.57
+57.56
+137.85
+72.03
+77.66
+138.08
+72.19
+77.83
% Incorrect
45.68%
37.65%
33.02%
41.96%
32.41%
29.63%
29.32%
24.69%
24.07%
Precision at top one
passage
0.1235
0.1975
0.1759
0.2253
0.2716
0.3364
0.3457
0.3889
0.3889
August 17, 2005
St
ri c
t_
R
N
U
0.2990
Si
te
Q
0.2677
% MRR improvement
over
MITRE
SiteQ
NUS
0.2000
Rel_MI
(NUS)
0.2765
M
IT
R
MRR
E
MRR
0.60
– EM needs more training data
Fuzzy matching
0.50
– MI is more susceptible to noise, so may not scale well
outperforms strict
0.40
0.30
matching significantly.
Passage 0.20
retrieval
Rel_Strict
Rel_Strict
Rel_MI
Rel_EM
MITRE
SiteQ
NUS
systems 0.10
(MITRE)
(NUS)
(MITRE) (MITRE)
0.00
Question Answering Passage Retrieval Using
Dependency Parsing
22/28
Performance Variation to
Question Length
• Long questions, with more paired paths,
tend to improve more
– Using the number of non-trivial question terms
to approximate question length
0.65
0.6
0.55
MRR
0.5
Rel_NUS_EM
0.45
0.4
Rel_MITRE_EM
0.35
0.3
0.25
0.2
0
2
4
6
8
# Question Term s
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
23/28
Error Analysis
• Mismatch of question terms
• e.g. In which city is the River Seine
• Introduce question analysis
• Paraphrasing between the question and
the answer sentence
• e.g. write the book → be the author of the book
• Most of current techniques fail to handle it
• Finding paraphrasing via dependency parsing (Lin
and Pantel)
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
24/28
Performance on Top of Query
Expansion
• On top of query expansion, fuzzy relation matching
brings a further 50% improvement
• However
– query expansion doesn’t help much on a fuzzy relation matching
system
– Expansion terms do not help in paring relation paths
Passage Retrieval
Systems
NUS
(baseline)
NUS+QE
Rel_MI
(NUS+QE)
Rel_EM
(NUS+QE)
MRR
(% improvement
over baseline)
0.2677
0.3293
(+23.00%)
0.4924
(+83.94%)
0.4935
(+84.35%)
% MRR
improvement over
NUS+QE
N/A
N/A
+49.54%
+49.86%
% Incorrect
33.02%
28.40%
22.22%
22.22%
Precision at top one
passage
0.1759
0.2315
0.4074
0.4074
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
Rel_EM
(NUS)
0.4761
25/28
Outline
•
•
•
•
•
Extracting and Paring Relation Paths
Measuring Path Match Scores
Learning Relation Mapping Scores
Evaluations
Conclusions
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
26/28
Conclusions
• Proposed a novel fuzzy relation matching
method for factoid QA passage retrieval
– Brings dramatic 70%+ improvement over the state-ofthe-art systems
– Brings further 50% improvement over query
expansion
– Future QA systems should bring in relations between
words for better performance
• Query expansion should be integrated to relation
matching seamlessly
August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
27/28
Q&A
Thanks!

August 17, 2005
Question Answering Passage Retrieval Using
Dependency Parsing
28/28