(Spoken) Dialogue
and Information Retrieval
Antoine Raux
Dialogs on Dialogs Group
10/24/2003
Outline



Interactive Information Retrieval Systems
(Belkin et al)
EUREKA: Dialogue-based IR for Low
Bandwidth Devices
Voice Access to IR
Cases, Scripts, and InformationSeeking Strategies

Belkin, Cool (Rutgers)
Stein, Thiel (GMD-IPSI)

Long journal article (1995)

From the IR community (Expert Systems)
IR as Interaction



Traditional IR research focuses on
document/query representation and
comparison
Need to focus on the user
Represent IR as a dialogue between an
information seeker and an information
provider
Information-Seeking Strategies

Represent information-seeking behavior
along 4 dimensions:





Method of Interaction (scanning vs searching)
Goal of Interaction (learning vs selecting)
Mode of Retrieval (recognition vs specification)
Resource Considered (information vs meta-info)
Binary values  16 strategies (ISS)
Dialogue Structures for Information
Seeking

Mix of different formalisms:



Recursive state-based schemas (COR)
e.g. Request  Promise  Inform
 Be contented
Scripts: prototypical interaction for each ISS
Goal trees
Retrieve Specified Items
Specify Characteristic
Offer choice
Recognize Desired Items
Select and Specify
Deriving Scripts from Data




Case-based approach: problem solving
using previously stored solved instances
Match a sequence of action to a statebased schema
Extract goal tree
Identify goal (which ISS?)
The MERIT System




Theory vs Practice…
Graphical interface (not NL dialogue)
User does case selection (for eventual
case-based reasoning)
Example task is relational database (not
free text IR): uses form filling (!)
Discussion


Contribution to IR: user-centered view,
application of many non-IR theories
(discourse, CBR)
BUT: too complicated for the user?
Discussion


Contribution to Dialogue Systems: difficult
task (not often dealt with in DS), CBR (can
we learn dialogue structure from data?)
BUT: lacks a good, unified, practical
framework (too many different paradigms
applied…)
Dialogue-based IR: Why?


Google-like interface still predominant
(despite MERIT)
Why?




Users receives a lot of information (document
titles, summaries) and use it as they want
Very simple to learn
Very flexible
BUT: works on high bandwidth devices
Dialogue-based IR: Why?




For low bandwidth devices (PDA, phone),
information-rich interface don’t work
Only small pieces of information
exchanged at a time
System has to select
Less information, more interaction
EUREKA: Idea

Use dialogue to submit queries to a web
search engine, browse through the
hierarchically clustered results, perform
query reformulation/refinement, etc…
EUREKA: Overview




Backend: Vivisimo (through web scraper)
Dialogue Management: RavenClaw
(successor of CMU Communicator)
Language Understanding: Light Open
Vocabulary Parser
NLG/TTS: template-based & Festival
Backend: Vivisimo


Available clustering meta-search engine
www.vivisimo.com
Hand-written Perl web scraper
(hope Vivisimo doesn’t change their page
design by the end of the semester…)
LOV Parser


Problem: traditional NL parsers require a
dictionary  not applicable to open
domain IR
Solution (implemented in C++):


fix a small number of one-word commands
(new_query, open, list_clusters)
parse each line as “[command] [arguments]”
or “[command]” or “[arguments]”
Dialogue Management: RavenClaw

Hierarchical agent architecture:
EUREKA
Greet User
Submit
Query
Prompt
Query
Get
Cluster List
New Query
Get
Doc List
Open
Cluster
Inform
Results
Close
…
Cluster
NLG/TTS


Template-based Language Generation
(e.g. “I found <n_doc> documents.”)
General purpose Festival voice for TTS
NB: Browsing through lists is not efficient
with speech, even for lists of clusters
Already Implemented


Working prototype
Commands:




new_query
list_clusters, list_documents
open, close (cluster)
more, back (list of clusters/documents)
Demo
Future Work


Add more functionalities (query refinement,
summarization…)
Make clever use of the dialogue (not only
command and control + browsing)


System can provide advice to user on search
strategies (e.g. “you need to refine the query”)
User and system can negotiate to specify the
user’s information need
(cf Belkin: overview vs specific document)
Future Work/Discussion



Advantage of dialogue: more feedback
from the user
How can dialogue improve the efficiency
of low bandwidth IR?
Do we need to tailor IR techniques (e.g.
clustering) for dialogue, or even design
new techniques?
Vocal Access to IR


Problem: ASR introduces a lot of
erroneous words in a spoken query
(for an open domain, speaker independent
system)
However, in an IR system: access to many
text documents to help language
modeling…
Vocal Access to a Newspaper
Archive (Crestani 02)



Presents studies for a full voice-controlled IR
system
No dialogue:
user query  list of summaries
Focuses on issues of:


TTS: can user make relevance judgments when they
hear document descriptions synthesized over the
phone? (answer: yes)
ASR: how does IR perform with recognized queries?
Using IR Techniques to Deal with
Recognition Errors




WER does have an impact on precision, although
not much variation for WER in 27%-47%
Relevance feedback: use documents judged
relevant by the user as query
Use prosodic stress to estimate information
content of query terms
Include semantically/phonetically close terms in
the query
Improving ASR (Fujii et al 02)

Fujii et al propose LM adaptation based on
the IR corpus:




Offline “adaptation”: train on the whole corpus
Online adaptation: adapt on the top retrieved
documents (then reperform ASR and IR)
Good results with offline trained LM (WER
< 20%, AP loss of 20-30% from text IR)
No evaluation of online adaptation…
Vocal Access to IR: Discussion

Seems to work ok for some tasks
Clever use of IR techniques
BUT queries are not spontaneous nor
natural (maybe)
LM for Web queries??

What about dialogue?


