Question Answering (QA)
Lecture 2
© Johan Bos
April 2008
Lecture 1
•
•
•
•
•
•
•
•
What is QA?
Query Log Analysis
Challenges in QA
History of QA
System Architecture
Methods
System Evaluation
State-of-the-art
• Question Analysis
• Answer Typing
• Background Knowledge
Lecture 3
•
•
•
•
•
Query Generation
Document Analysis
Semantic Indexing
Answer Extraction
Selection and Ranking
© Johan Bos
April 2008
What is Question Answering?
What is Question Answering?
© Johan Bos
April 2008
• Questions in natural language,
not simple queries of keywords!
• Answers, not documents!
What is Question Answering?
• Questions, no queries!
• Answers, not documents!
© Johan Bos
April 2008
Q: Where did Olof Palme die?
A:
What is Question Answering?
• Questions, no queries!
• Answers, not documents!
© Johan Bos
April 2008
Q: Where did Olof Palme die?
A: In Stockholm.
QA versus IR
• Traditional method for information
access: IR (Information Retrieval)
© Johan Bos
April 2008
– Think of IR as finding the “right book in a
library”
– Think of QA as a “librarian giving you the
book and opening it on the page with the
information you’re looking for”
QA versus IE
• Traditional method for information
access: IE (Information Extraction)
– Think of IE as finding answers to a predefined question (i.e., a template)
© Johan Bos
April 2008
– Think of QA as asking any question you
like
Why do we need QA?
© Johan Bos
April 2008
• Information overload problem
• Accessing information using traditional
methods such as IR and IE are limited
• QA increasingly important because:
– Size of available information grows
– There is duplicate information
– There is false information
– More and more “computer illiterates”
accessing electronically stored information
Information Avalanche
• Available information is growing*:
– 1999: 250MB pp for each person on earth
– 2002: 800MB pp for each person on earth
© Johan Bos
April 2008
• People want specific information
• People ask questions (rather than queries)
* source: M.de Rijke 2005
Information Pinpointing
© Johan Bos
April 2008
For example --- information required:
Average number of car accidents per
year in Sweden.
Two ways of getting this information:
- Ask Google or a similar search engine
(good luck!)
- Ask a QA system the question:
What’s the rate of car accidents in
Sweden?
Excite Query Log
• Search Engine Log with 2,477,283 queries
• Collected at 20 December 1999
• Around 15% natural language questions
© Johan Bos
April 2008
queries
questions
© Johan Bos
April 2008
Excite Query Log
090039
090039
090039
090039
090039
090039
090039
090039
090039
090039
090039
090039
090039
090039
090039
090039
090039
090039
090039
82F30745D2BF5C2C
9252B344DB1B7047
E6D3DCFA0A39BA3E
AFCD0FD4A65D34C3
F54BC573FF3C94B1
3DF4E9B0AFF6B808
009CD46686D4A1FD
F26E5D31D804ED98
D87CE5C149126B4B
91255EEF958730CE
03D685B3F001A858
F54BC573FF3C94B1
DAF1CD4E35BB6D98
BDF5F884F42DB05F
1246BA311FD5AF65
9F0FF534C84D7873
E00F9C2509761702
70B9F6D94EB08EB7
5AE9D0A42F05FDE1
golf
www.Odcgov/cia/Publication/Fatbook/index
www.ufallfahrzeug.de
montogery ward
What is the website for the newspaper in Scranton, PA?
Universities in Wellington
pantyfreek
carman ministries
Where can I find information about the television show Felici
how can i convert liquid audio files to wav files?
"CD duplication software"
What does the FDA say is its mandate?
ebony ayres+homepage
hacking VP Instant Message readers messanger
thiazolidisediones
chicago newspapers
yahoo
New Century Professions, Inc.
What presidents were born in texas
Question Types
• Wh-Questions:
• Yes/No-Questions:
342,075
23,882
Wh-Questions
© Johan Bos
April 2008
Yes/NoQuestions
© Johan Bos
April 2008
Question Types
How is the U.N. funded?
Was Don lying about the shooting?
Who invened the rangefinder?
what exactly is a hydraulic system?
When was Fisher Price started?
Where can i find exams on the internet about database management systems course with sol
Where can I find yesturday's weather?
What is the probability of extraterrestrial life existing in the Universe?
What vacation rentals are available in the pacific Northwest?
Who was president Grant's private secretary?
Where can I get a recipe for sloppy joes?
What are the times of sunrises and sunsets in various cities in the US?
pros and cons of professional managers in government?
Where can I find information about FLSA?
where can I find details of property in Scotland?
Who sings the song"Ice, Ice Baby"?
what percentage of schools have internet access?
When were gingerbread houses invented?
how do I find out the purchase price of a house that sold in California?
Are cmputer chips made from sand?
what is lupus?
where can i find how to write a resume?
WHERE CAN I FIND ON-LINE GREETING CARDS?
Wh-Question Types
© Johan Bos
April 2008
where (185,351)
what
(71,374)
how
(53,963)
who
(19,477)
when
(5,541)
why
(4,428)
which
(1,781)
whom
(97)
whose
(63)
How-Question Types
© Johan Bos
April 2008
how AUX
(43,966)
how ADJ
(4,558)
how many
(3,024)
how much
(2,415)
© Johan Bos
April 2008
How-Question Types
How
How
How
How
How
How
How
How
How
How
How
How
How
How
How
How
How
How
How
How
How
How
How
How
How
do I register a domain name?
do genetics affect childrens diseases?
do I stop a Bios I/O COnflict?
does the Greenhouse effect work
many hours do you practice a week for basketball?
do I learn Unix?
were the Christians persecuted?
maney bicycle accidents in US by year?
long does a DUI stay on your record in California
do you make a curcitboard?
does airplane wing shape affect flight?
were the Christians persecuted?
do you play the saxophone?
do I start a travel writing career?
do I stop a COnflict I/O Ports?
many bicycle accidents in US by year?
much weight can you lose per week?
are you?
do I shop online?
do you take apart an alkaline battery
to manufacture?
does dry ice work?
tall is the average man
do I make a pecan pie
can I find someone's Yahoo e-mail address?
Natural search
© Johan Bos
April 2008
• Google gets about 200 million
searches per day
• That would mean about 30 million
natural language searches
Question Answering (QA)
© Johan Bos
April 2008
Lecture 1
•
•

•
•
•
•
•
What is QA?
Query Log Analysis
Challenges in QA
History of QA
System Architecture
Methods
System Evaluation
State-of-the-art
© Johan Bos
April 2008
QA – The Seven Challenges
1) Accounting for flexibility in natural
language questions and answers
2) Dealing with ill-formed input
3) Merging across documents
4) Acquiring relevant background
knowledge
5) Semantic analysis
6) User modelling
7) Answer precision
Challenge 1 (flexibility)
• Variation in questions:
–
–
–
–
What is the population of Rome?
How many people live in Rome?
What’s the size of Rome?
How many inhabitants does Rome have?
© Johan Bos
April 2008
• Variation in answers:
…is estimated at 2.5 million residents…
… current population of Rome is 2817000…
…Rome housed over 1 million inhabitants…
Challenge 2 (ill-formed input)
© Johan Bos
April 2008
• Just look at the Excite query log!
Who invened the rangefinder?
what exactly is a hydraulic system?
When was Fisher Price started?
Where can I find yesturday's weather?
What is the probability of extraterrestrial life existing in the Universe?
What vacation rentals are available in the pacific Northwest?
Who was president Grant's private secretary?
Where can I get a recipe for sloppy joes?
What are the times of sunrises and sunsets in various cities in the US?
pros and cons of professional managers in government?
Where can I find information about FLSA?
where can I find details of property in Scotland?
Who sings the song"Ice, Ice Baby"?
what percentage of schools have internet access?
When were gingerbread houses invented?
how do I find out the purchase price of a house that sold in California?
Are cmputer chips made from sand?
what is lupus?
where can i find how to write a resume?
WHERE CAN I FIND ON-LINE GREETING CARDS?
Challenge 2 (ill-formed input)
© Johan Bos
April 2008
• Just look at the Excite query log!
Who invened the rangefinder?
what exactly is a hydraulic system?
When was Fisher Price started?
Where can I find yesturday's weather?
What is the probability of extraterrestrial life existing in the Universe?
What vacation rentals are available in the pacific Northwest?
Who was president Grant's private secretary?
Where can I get a recipe for sloppy joes?
What are the times of sunrises and sunsets in various cities in the US?
pros and cons of professional managers in government?
Where can I find information about FLSA?
where can I find details of property in Scotland?
Who sings the song"Ice, Ice Baby"?
what percentage of schools have internet access?
When were gingerbread houses invented?
how do I find out the purchase price of a house that sold in California?
Are cmputer chips made from sand?
what is lupus?
where can i find how to write a resume?
WHERE CAN I FIND ON-LINE GREETING CARDS?
Challenge 3 (merging)
• Answers could be spread across different
documents
• Example 1:
– Which European countries produce wine?
[Document A contains information about Italy, and
document B about France]
© Johan Bos
April 2008
• Example 2:
– What does Bill Clinton’s wife do for a living?
[Document A explains that Bill Clinton’s wife is
Hillary Clinton, and Document B tells us that she’s
a politician]
Challenge 4 (knowledge)
Q: When was NATO established?
A: NATO launched its first attack against
Yugoslavia on March 24.
© Johan Bos
April 2008
WordNet: launch=establish
Challenge 4 (knowledge)
Q: Bing Crosby.
What was his profession?
A: Crosby sang her first song, Starlight. She
© Johan Bos
April 2008
scribbled notes for the song on the back of a
menu at a New York jazz bar in 1931.
.
WordNet: bar hyponym of profession
Challenge 5 (semantics)
Q:
Where did Ricky Williams, American football
player, grow up?
A: Texas running back Ricky Williams is from
© Johan Bos
April 2008
California. Those who follow the Longhorns
don't like Williams any less because he didn't
grow up in Texas.
Challenge 6 (user modelling)
Q: Where is the Taj Mahal?
© Johan Bos
April 2008
A:
The Taj Mahal is a mausoleum located in Agra,
India, that was built under Mughal Emperor Shah
Jahan in memory of his favorite wife, Mumtaz Mahal.
Challenge 6 (user modelling)
Q: Where is the Taj Mahal?
© Johan Bos
April 2008
A:
A:
The Taj Mahal is a mausoleum located in Agra,
India, that was built under Mughal Emperor Shah
Jahan in memory of his favorite wife, Mumtaz Mahal.
The Taj Mahal first opened its door to a receptive but
cautious public back in 1964. Being the first
restaurant of its kind in Stevenage, our main
problems were initially to get people to try foods that
they had never tried before.
Challenge 7 (precision)
• Where is 3M based?
© Johan Bos
April 2008
– In Maplewood
– In Maplewood, Minn.
– In Minnesota
– In the U.S.
– In Maplewood, Minn., USA
Challenge 7 (precision)
• Where did Franz Kafka die?
© Johan Bos
April 2008
– In his bed
– In a sanatorium
– In Kierling
– Near Vienna
– In Austria
Question Answering (QA)
© Johan Bos
April 2008
Lecture 1
•
•
•

•
•
•
•
What is QA?
Query Log Analysis
Challenges in QA
History of QA
System Architecture
Methods
System Evaluation
State-of-the-art
© Johan Bos
April 2008
History of QA (de Rijke & Webber 2003)
• QA is by no means a new area!
• Simmons (1965) reviews 15
implemented and working systems
• Many ingredients of today’s QA
systems are rooted in these early
approaches
• Database oriented systems, domain
independent, as opposed to today’s
systems that work on large sets of
unstructured texts
© Johan Bos
April 2008
Examples of early QA systems
• BASEBALL (Green et al. 1963)
Answers English questions about scores,
locations and dates of baseball games
• LUNAR (Woods 1977)
Accesses chemical data on lunar material
compiled during the Apollo missions
• PHLIQA1 (Scha et al. 1980)
Answers short questions against a database
of computer installations in Europe
Recent work in QA
• Since the 1990s research in QA has by
and large focused on open-domain
applications
© Johan Bos
April 2008
• Recently interest in restricted-domain
QA has increased, in particular in
commercial applications
– Banking
– Medical applications
– Entertainment
Open vs restricted domain
• QA in open domain
– General questions
– Large collection of documents (web)
– Answer redundancy
© Johan Bos
April 2008
• QA in restricted domain
– Very specific questions
– Relatively small set of documents
– Answers might appear only once in
document collection
Question Answering (QA)
© Johan Bos
April 2008
Lecture 1
•
•
•
•

•
•
•
What is QA?
Query Log Analysis
Challenges in QA
History of QA
System Architecture
Methods
System Evaluation
State-of-the-art
Architecture of a QA system
corpus
question
Question
Answering
System
© Johan Bos
April 2008
answers
Architecture of a QA system
corpus
question
Question
Analysis
query
documents/passages
expected
answer-type
question
representation
© Johan Bos
April 2008
answers
IR
Answer
Extraction
Document
Analysis
passage
representation
Question Analysis
• Input:
Natural Language Question
© Johan Bos
April 2008
• Output:
Expected Answer Type
(Formal) Representation of Question
• Techniques used:
Machine learning
syntactic & semantic parsing
Document Analysis
• Input:
Documents or Passages
© Johan Bos
April 2008
• Output:
(Formal) Representation of Passages
that might contain the answer
• Techniques used:
Tokenisation, Named Entity
Recognition, Parsing
Answer Extraction
• Input:
Expected Answer Type
Question (formal representation)
Passages (formal representation)
© Johan Bos
April 2008
• Output:
Ranked list of answers
• Techniques used:
Matching, Re-ranking, Validation
Example Run
corpus
question
Question
Analysis
query
documents/passages
answer-type
question
representation
© Johan Bos
April 2008
answers
IR
Answer
Extraction
Document
Analysis
passage
representation
Example Run
How long is the
river Thames?
corpus
question
Question
Analysis
query
documents/passages
answer-type
question
representation
© Johan Bos
April 2008
answers
IR
Answer
Extraction
Document
Analysis
passage
representation
Example Run
length river thames
corpus
question
Question
Analysis
query
documents/passages
answer-type
question
representation
© Johan Bos
April 2008
answers
IR
Answer
Extraction
Document
Analysis
passage
representation
Example Run
corpus
question
Question
Analysis
MEASURE
query
documents/passages
answer-type
question
representation
© Johan Bos
April 2008
answers
IR
Answer
Extraction
Document
Analysis
passage
representation
Example Run
corpus
question
Question
Analysis
query
answer(x) &documents/passages
length(y,x) &
river(y) & named(y,thames)
answer-type
question
representation
© Johan Bos
April 2008
answers
IR
Answer
Extraction
Document
Analysis
passage
representation
Example Run
A: NYT199802-31
B: APW199805-12
C: NYT200011-07corpus
question
Question
Analysis
query
documents/passages
answer-type
question
representation
© Johan Bos
April 2008
answers
IR
Answer
Extraction
Document
Analysis
passage
representation
Example Run
A: 30(u) & mile(u) &
length(v,u) & river(y)
query
B: 60(z)
& centimeter(z)
&
Question
question
height(v,z)
& dog(z)
Analysis
C: 230(u) & kilometer(u) &
length(x,u) answer-type
& river(x)
question
representation
© Johan Bos
April 2008
answers
Answer
Extraction
corpus
IR
documents/passages
Document
Analysis
passage
representation
Example Run
corpus
question
Question
Analysis
query
documents/passages
C: 230
kilometer
answer-type
A: 30 miles
B: 60question
centimeter
representation
© Johan Bos
April 2008
answers
IR
Answer
Extraction
Document
Analysis
passage
representation
Question Answering (QA)
© Johan Bos
April 2008
Lecture 1
•
•
•
•
•

•
•
What is QA?
Query Log Analysis
Challenges in QA
History of QA
System Architecture
Methods
System Evaluation
State-of-the-art
Method 1: Guessing
© Johan Bos
April 2008
• Limit the amount of (deep) natural
language processing tools
• Usually based on simple pattern
matching
• Use shallow features, such as
frequency, to guess an answer
Methods in QA
• Viewed from a distance, roughly three
different methods can be distinguished:
© Johan Bos
April 2008
– Guessing
– Googling
– GOFAI (good old-fashioned AI)
Guessing example
Q: What country is Berlin in?
© Johan Bos
April 2008
Ties between Iran and Germany strained
after a Berlin court verdict ruled on April 10
A: that Iran's top leaders were behind the
assassination of four exiled Kurdish
opposition leaders in a Berlin restaurant.
Guessing example
Q: What country is Berlin in?
© Johan Bos
April 2008
Ties between Iran and Germany strained
after a Berlin court verdict ruled on April 10
A: that Iran's top leaders were behind the
assassination of four exiled Kurdish
opposition leaders in a Berlin restaurant.
Another guessing example
Q: Port Arthur Massacre.
© Johan Bos
April 2008
What was the killer's nationality?
A nation asks why, the portrait of a lone
gunman, Martin Bryant, should have no
A: reason to be a killer. The man arrested after
the Port Arthur massacre should be a wealthy
man, only four years ago he inherited more
than 500,000 Australian dollars 375,000 US
dollars from one of the heirs of George
Adams's great Tattersalls fortune.
Method 2: Googling
© Johan Bos
April 2008
• Use other, larger, corpora to find the
answer (such as the internet).
• Use Google tricks.
Googling example
Q: What is the state bird of Alaska?
A:
Google search:
© Johan Bos
April 2008
"The state bird of Alaska is the * "
Googling example
Q: What is the state bird of Alaska?
A:
Google results:
© Johan Bos
April 2008
The state bird of Alaska is the ptarmigan.
The state bird of Alaska is the willow ptarmigan.
Googling example
Q: What is the state bird of Alaska?
A:
Google results:
© Johan Bos
April 2008
The state bird of Alaska is the ptarmigan.
The state bird of Alaska is the willow ptarmigan.
The state bird of Alaska is the mosquito.
The state bird of Alaska is the Mosquito.
Method 3: GOFAI
© Johan Bos
April 2008
• Using background knowledge to give
support for an answer
• Use (logical) inference when possible
GOFAI example
© Johan Bos
April 2008
Q: Where did Olof Palme die?
A: Stockholm.
GOFAI example
Q: Where did Olof Palme die?
© Johan Bos
April 2008
A:
……….
given clause #13: (wt=2) 69 [hyper,47,16] vehicle($c3).
given clause #14: (wt=2) 71 [hyper,50,16] vehicle($c4).
given clause #15: (wt=2) 74 [hyper,52,18] building($c5).
given clause #16: (wt=3) 51 [] have(vincent,$c4).
given clause #17: (wt=2) 76 [hyper,55,18] building($c6).
given clause #18: (wt=2) 78 [hyper,61,16] vehicle($c6).
given clause #19: (wt=2) 80 [hyper,65,10] organism($c1).
given clause #20: (wt=2) 84 [hyper,67,10] organism($c2).
given clause #21: (wt=3) 53 [] die(palme,$c5).
given clause #22: (wt=2) 86 [hyper,69,15] instrument($c3).
given clause #23: (wt=2) 88 [hyper,71,15] instrument($c4).
given clause #24: (wt=2) 90 [hyper,74,7] artifact($c5).
given clause #25: (wt=2) 94 [hyper,76,7] artifact($c6).
given clause #26: (wt=3) 56 [] $c7=$c6.
given clause #27: (wt=2) 96 [hyper,78,15] stockholm($c6).
-----> EMPTY CLAUSE at 0.01 sec ----> 113 [hyper,96,24,76] $F.
GOFAI example (future?)
Q: Where did Olof Palme die?
© Johan Bos
April 2008
A:
1. You are looking for location.
2. I know that Stockholm is a city.
3. Every city is a location.
4. If x is shot to death then x died.
5. I found the following evidence in document
APW20000227.0124:
"In 1986, Swedish Prime Minister Olof Palme
was shot to death in central Stockholm. “
6. Hence Stockholm is the answer.
Question Answering (QA)
© Johan Bos
April 2008
Lecture 1
•
•
•
•
•
•

•
What is QA?
Query Log Analysis
Challenges in QA
History of QA
System Architecture
Methods
System Evaluation
State-of-the-art
Evaluating QA systems
© Johan Bos
April 2008
• International evaluation campaigns for
QA systems (open domain QA):
– TREC (Text Retrieval Conference)
http://trec.nist.gov/
– TAC (Text Analysis Conference)
– CLEF (Cross Language Evaluation Forum)
http://clef-qa.itc.it/
– NTCIR (NII Test Collection for IR Systems)
http://www.slt.atr.jp/CLQA/
TREC-QA (organised by NIST)
• Annual event, started in 1999
• Difficulty of the QA task increased over
the years:
– 1999: Answers in snippets, ranked list of
answers;
– 2005: Exact answers, only one answer.
© Johan Bos
April 2008
• Three types of questions:
– Factoid questions
– List questions
– Definition questions
QA@CLEF
• CLEF is the “European edition” of TREC
• Monolingual (non-English) QA
– Bulgarian (BG), German (DE), Spanish (ES),
Finnish (FI), French (FR), Italian (IT), Dutch (NL),
Portuguese (PT)
• Cross-Lingual QA
© Johan Bos
April 2008
– Questions posed in source language, answer
searched in documents of target language
– All combinations possible
Open-Domain Question Answering
• TREC (1999 - 2007)
– Acquint corpus
– Factoid and Definition questions
• TAC (2008 - ….)
© Johan Bos
April 2008
– Blog06 corpus
– List and opinion questions
TREC-type questions
• Factoid questions
– Where is the Taj Mahal?
• List questions
– What actors have played Tevye in
“Fiddler on the Roof?”
© Johan Bos
April 2008
• Definition/biographical questions
– What is a golden parachute?
– Who is Vlad the Impaler?
What is a correct answer?
• Example Factoid Question
– When did Franz Kafka die?
© Johan Bos
April 2008
• Possible Answers:
– Kafka died in 1923.
– Kafka died in 1924.
– Kafka died on June 3, 1924 from
complications related to Tuberculosis.
– Ernest Watz was born June 3, 1924.
– Kafka died on June 3, 1924.
What is a correct answer?
• Example Factoid Question
– When did Franz Kafka die?
© Johan Bos
April 2008
• Possible Answers:
Incorrect
– Kafka died in 1923.
– Kafka died in 1924.
– Kafka died on June 3, 1924 from
complications related to Tuberculosis.
– Ernest Watz was born June 3, 1924.
– Kafka died on June 3, 1924.
What is a correct answer?
• Example Factoid Question
– When did Franz Kafka die?
© Johan Bos
April 2008
• Possible Answers:
Inexact
(under-informative)
– Kafka died in 1923.
– Kafka died in 1924.
– Kafka died on June 3, 1924 from
complications related to Tuberculosis.
– Ernest Watz was born June 3, 1924.
– Kafka died on June 3, 1924.
What is a correct answer?
• Example Question
– When did Franz Kafka die?
© Johan Bos
April 2008
• Possible Answers:
Inexact
(over-informative)
– Kafka died in 1923.
– Kafka died in 1924.
– Kafka died on June 3, 1924 from
complications related to Tuberculosis.
– Ernest Watz was born June 3, 1924.
– Kafka died on June 3, 1924.
What is a correct answer?
• Example Question
– When did Franz Kafka die?
© Johan Bos
April 2008
• Possible Answers:
– Kafka died in 1923.
Unsupported
– Kafka died in 1924.
– Kafka died on June 3, 1924 from
complications related to Tuberculosis.
– Ernest Watz was born June 3, 1924.
– Kafka died on June 3, 1924.
What is a correct answer?
• Example Question
– When did Franz Kafka die?
© Johan Bos
April 2008
• Possible Answers:
Correct
– Kafka died in 1923.
– Kafka died in 1924.
– Kafka died on June 3, 1924 from
complications related to Tuberculosis.
– Ernest Watz was born June 3, 1924.
– Kafka died on June 3, 1924.
Answer Accuracy
© Johan Bos
April 2008
# correct answers
Answer Accuracy = --------------------------# questions
Correct answers to list questions
Example List Question
Which European countries produce wine?
System A:
© Johan Bos
April 2008
France
Italy
System B:
Scotland
France
Germany
Italy
Spain
Iceland
Greece
the Netherlands
Japan
Turkey
Estonia
Evaluation metrics for list questions
• Precision (P):
# answers judged correct & distinct
P = ---------------------------------------------# answers returned
• Recall (R):
# answers judged correct & distinct
R = -----------------------------------------------# total answers
© Johan Bos
April 2008
• F-Score (F):
2*P*R
F = -----------P+R
Correct answers to list questions
Example List Question
Which European countries produce wine?
System A:
© Johan Bos
April 2008
France
Italy
P = 1.00
R = 0.25
F = 0.40
System B:
Scotland
France
Germany
Italy
Spain
Iceland
Greece
the Netherlands
P = 0.64
Japan
R = 0.88
Turkey
F = 0.74
Estonia
Other evaluation metrics
System A: Ranked answers (Accuracy = 0.2)
Q1
Q2
Q3
Q4
Q6
Q7
Q8
Q9
….
Qn
A1
W
W
C
W
C
W
W
W
….
W
A2
W
W
W
W
W
W
W
W
….
W
A3
W
W
W
W
W
W
W
W
….
W
A4
W
W
W
W
W
W
W
W
….
W
A5
W
C
W
W
W
C
W
W
….
W
© Johan Bos
April 2008
System B: Ranked answers (Accuracy = 0.1)
Q1
Q2
Q3
Q4
Q6
Q7
Q8
Q9
….
Qn
A1
W
W
W
W
C
W
W
W
….
W
A2
C
W
C
W
W
C
C
W
….
C
A3
W
C
W
W
W
W
W
W
….
W
A4
W
W
W
C
W
W
W
W
….
W
A5
W
W
W
W
W
W
W
W
….
W
Mean Reciprocal Rank (MRR)
• Score for an individual question:
– The reciprocal of the rank at which
the first correct answer is returned
– 0 if no correct response is returned
• The score for a run:
© Johan Bos
April 2008
– Mean over the set of questions in the test
MRR in action
System A: MRR = (.2+1+1+.2)/10 = 0.24
Q1
Q2
Q3
Q4
Q6
Q7
Q8
Q9
….
Qn
A1
W
W
C
W
C
W
W
W
….
W
A2
W
W
W
W
W
W
W
W
….
W
A3
W
W
W
W
W
W
W
W
….
W
A4
W
W
W
W
W
W
W
W
….
W
A5
W
C
W
W
W
C
W
W
….
W
© Johan Bos
April 2008
System B: MRR = (.5+.33+.5+.25+1+.5+.5+.5)/10=0.42
Q1
Q2
Q3
Q4
Q6
Q7
Q8
Q9
….
Qn
A1
W
W
W
W
C
W
W
W
….
W
A2
C
W
C
W
W
C
C
W
….
C
A3
W
C
W
W
W
W
W
W
….
W
A4
W
W
W
C
W
W
W
W
….
W
A5
W
W
W
W
W
W
W
W
….
W
Question Answering (QA)
© Johan Bos
April 2008
Lecture 1
•
•
•
•
•
•
•

What is QA?
Query Log Analysis
Challenges in QA
History of QA
System Architecture
Methods
System Evaluation
State-of-the-art
State of the art
• QA is hard
• Best systems use
extensive answer
typing and semantics
• Systems complex
Accuracy TREC 2004
(n=28)
0.6-0.7
0.4-0.5
0.2-0.3
0.0-0.1
© Johan Bos
April 2008
0
10
Unsolved problems
• Yes/No-questions
Can you give me the names of Mr Jones?
Are debit cards expensive?
© Johan Bos
April 2008
Are growing stock piles an indication of trouble in a
company?
Unsolved problems
• Conditional questions
Can I get a mortgage if the house needs repairs?
• Why-questions
Why did Obama change his name?
© Johan Bos
April 2008
• How-questions
How did Hillary Clinton win 58% in Rhode Island?
Unsolved problems
• Comparison-questions
Is X better than Y?
Of the two governments involved over Kinmen,
which has air superiority?
© Johan Bos
April 2008
What is the difference between X and Y?
Unsolved problems
• Questions with negation
Which European countries do not charge VAT tax
on gold?
Are there hearing aids that do not have to go inside
your ear?
© Johan Bos
April 2008
My computer is set up to reject cookies.
What will I not be able to do on your website?
Unsolved problems
• Multiple questions in disguise
© Johan Bos
April 2008
Where and when was Franz Kafka born?
Who ordered what?
Where is each department located?
Unsolved problems
• Questions asking for opinions
What do people like about IKEA?
What do young people like about IKEA?
© Johan Bos
April 2008
What do women like about IKEA?
Question Answering (QA)
Lecture 2
© Johan Bos
April 2008
Lecture 1
•
•
•
•
•
•
•
•
What is QA?
Query Log Analysis
Challenges in QA
History of QA
System Architecture
Methods
System Evaluation
State-of-the-art
• Question Analysis
• Answer Typing
• Background Knowledge
Lecture 3
•
•
•
•
•
Query Generation
Document Analysis
Semantic Indexing
Answer Extraction
Selection and Ranking
Architecture of PRONTO
parsing
question
answer
ccg
answer
reranking
boxing
drs
answer
selection
WordNet
NomLex
knowledge
© Johan Bos
April 2008
query
answer
typing
Indri
answer
extraction
Indexed Documents
Lecture 2
parsing
question
answer
ccg
answer
reranking
boxing
drs
answer
selection
WordNet
NomLex
knowledge
© Johan Bos
April 2008
query
answer
typing
Indri
answer
extraction
Indexed Documents
Lecture 3
parsing
question
answer
ccg
answer
reranking
boxing
drs
answer
selection
WordNet
NomLex
knowledge
© Johan Bos
April 2008
query
answer
typing
Indri
answer
extraction
Indexed Documents