TARGETED HELP FOR SPOKEN
DIALOGUE SYSTEM
SREEDHAR ELLISETTY
TARGETED HELP FOR SPOKEN DIALOGUE SYSTEM :
INTELLIGENT FEEDBACK IMPROVES NAIVE USERS’ PERFORMANCE
MAIN ISSUES
» What is TARGETED HELP?
» How it provides Feedback?
» How Target help Module Works?
» How SLM Recognizer makes system Robust?
» Experimental Evidence
» How some ITSs providing Feedback?
INTRODUCTION - 1
» Targeted Help: Gives immediate Intelligent Feedback to Naïve
users’ about their Out-of-Coverage spoken Utterances.
» Goal: Producing Intelligent Feedback to the Naive users’ and
guiding users Towards In-Coverage Utterances.
» Key words
- Expert Users
- Naive Users
- In-Coverage Utterance
- Out-of-Coverage Utterance
- Grammar based Recognizer
- SLM Recognizer
INTRODUCTION - 2
» Expert users know the limitations and capabilities of the system
» Grammar-based recognizers tuned to a domain can handle well,
the in-coverage Utterances.
» Naive users don’t know the Grammar of the System and they
produces Out-of-Coverage Utterances.
» Grammar based Recognizers rejects and produces messages like
“Sorry, I didn't Understand”
» Statistical Language Model (SLM) Recognizer handles Out-of-Coverage
Utterance.
INTRODUCTION - 3
» SLM Recognizer produces a recognition hypothesis, which is
used by the Targeted Help agent to give the users feedback
» Feedback message consists of Diagnostics of the user Utterance
and In-Coverage examples.
» In-Coverage example uses In-coverage words in the users’
Utterances and the Same Dialogue move
» Encourages users’ to align the users Utterance with the Language
model of the System
» Makes users expert in quick time, improves performance,
Reduces time.
Grammar – Based Recognizer
» Tuned to a domain performs well for all In-Coverage utterances.
» Example: Grammar of FAIRY TALE READER
» Grammar can be written quickly
Drawbacks:
- Puts constraints to the user on the usage of the system
- Cant’ handle Out-of-Coverage utterance
- Takes lot of time for simple sentences.
» Defining a Grammar is to predict future user utterance as good as possible
by observing the real users in the desired application
» Problem in Defining appropriate Grammar
- It should cover as much as possible
- It shouldn’t contain unnecessary words, which leads to recognition errors
> It should satisfy these issues: Precise , Coverage.
SLM RECOGNIZER
» Also called Secondary, Fall-back, Category-based statistical Language model
Recognizer.
» Handles all the Out-of-Coverage Utterances well
» Produces a recognition hypothesis, which is used by the targeted help agent to
Give Feedback
» Problems with SLM :
- Needs to be trained to perform efficiently
- Collecting a large enough corpus to create an SLM is time consuming
and expensive
- A separate parser must be implemented to extract semantic content.
Advantages
- Don’t put constraints to the users’
System Description
» Targeted Help was developed and tested as part of the WITAS dialogue
system
» WITAS: A command and control and mixed-initiative dialogue system for
interacting with a simulated robotic helicopter or UAV
» Interaction in WITAS involves joint-activities between an autonomous
system and human operator .
» Interactions with such a system are not scriptable in advance ,
and rely on mixed task and dialogue initiatives in conversation.
» So, the system is a good test-bed for Targeted Help
THE TARGETED HELP MODULE
» Is a separate component that can be added to an existing dialogue system
with minimal changes .
» Module design makes it quite portable
» Goal is to handle Utterance that cant’ be processed by the Main Dialogue system
» Aligns the users’ input with the coverage of the system as much as possible
PARTS IN THE MODULE:
- The secondary Recognizer
- The Targeted Help Activator
- The Targeted Help Agent
ARCHITECTURE OF DIALOGUE SYSTEM WITH TARGETED HELP MODULE
Parser
Dialogue Manager
MAIN DIALOGUE SYSTEM
Primary speech
recognizer
Speech Synthesizer
Speech in
Speech
out
Secondary speech
recognizer
Targeted help
activator
Targeted help agent
TARGETED HELP MODULE
Normal response path
Targeted help response path
Targeted help response path (if secondary SR result parses)
THE TARGETED HELP ACTIVATOR
» The Activator’s behavior is as follows for the four possible combinations of
Recognizer outcomes -
1. Both Recognizers get a recognition hypothesis
2. Main recognizer gets a recognition hypothesis and secondary recognizer
rejects
3. Main recognizer rejects, secondary recognizer gets a recognition hypothesis
and it can be Parsed (rare)
4. Main recognizer rejects, secondary recognizer gets a recognition hypothesis
and it cant’ be Parsed
5. Both recognizers reject.
BOTH RECOGNISERS GET A RECOGNITION HYPOTHESIS
Parser
Dialogue Manager
MAIN DIALOGUE SYSTEM
Primary speech
recognizer
Speech Synthesizer
Speech in
Speech
out
Secondary speech
recognizer
Targeted help
activator
Targeted help agent
TARGETED HELP MODULE
Normal response path
Targeted help response path
Targeted help response path (if secondary SR result parses)
MAIN RECOGNISER ACCEPTS, SECONDARY RECOGNISER REJECTS
Parser
Dialogue Manager
MAIN DIALOGUE SYSTEM
Primary speech
recognizer
Speech Synthesizer
Speech in
Speech
out
Secondary speech
recognizer
Targeted help
activator
Targeted help agent
TARGETED HELP MODULE
Normal response path
Targeted help response path
Targeted help response path (if secondary SR result parses)
MAIN RECOGNISER REJECTS, SECONDARY RECOGNISER HYPOTHESIS
CAN BE PARSED
Parser
Dialogue Manager
MAIN DIALOGUE SYSTEM
Primary speech
recognizer
Speech Synthesizer
Speech in
Speech
out
Secondary speech
recognizer
Targeted help
activator
Targeted help agent
TARGETED HELP MODULE
Normal response path
Targeted help response path
Targeted help response path (if secondary SR result parses)
MAIN RECOGNISER REJECTS, SECONDARY RECOGNISER
HYPOTHESIS CANNOT BE PARSED
Parser
Dialogue Manager
MAIN DIALOGUE SYSTEM
Primary speech
recognizer
Speech Synthesizer
Speech in
Speech
out
Secondary speech
recognizer
Targeted help
activator
Targeted help agent
TARGETED HELP MODULE
Normal response path
Targeted help response path
Targeted help response path (if secondary SR result parses)
FEEDBACK MESSAGE
» Target Help Agent constructs a message based on SLM recognizer hypothesis
» Spots the In-Coverage words in the SLM recognizer hypothesis and Dialoguemove type.
» Feedback Message consists:
•What the System Heard:
- A Report of the Backup SLM recognition hypothesis
•What the Problem was
- Problem with the users’ Utterance
•What you might say instead
- A Similar In-Coverage example.
PROBLEM DIAGNOSTIC
» Diagnostics are of three major types which accounts for major failed utterances
1. End pointing Errors:
Eg.
User says
: Search for the red car
System heard
: For the red car
Category
: Push-to-talk button too late
2. Unknown Vocabulary :
Eg: Fly over to the Hospital.
“The system doesn't understand the word over” vocabulary which is not the
Main recognizer”.
3. Sub categorization Mistakes:
Saying “Zoom in on the Red car” but system knows only intransitive “Zoom in”.
Zooming in on a particular object is not part of the functionality of the system
In-Coverage Example
• Helps Naive users learn the coverage of the system.
• Takes in-coverage words produced by the user .
• Helps the user to imitate in-coverage pattern .
• Leads to linguistic alignment between the user and the
system.
• It always matches the utterance dialogue-move type.
• Eg: Wh-Question, Yes/No-Question, Command,etc.
• Users are likely to produce atleast few in-coverage words.
• Eg :
User
: Fly over to the Hospital.
In-Coverage Example : Fly to the Hospital.
Constructing In-Coverage Example
•
In what sense they should be similar?
•
It takes into account information from a wide variety of sources,
including discourse history, user model and pedagogical strategy.
•
Constraints on not only what words should be included in the
example, but also what syntactic structures, semantic
representations, and word order.
•
Definition of generation task is: Given a grammar G, and a set of
target words T , find a word string W=W1,…,Wn, W ε L(G) such that
T {W1,…….,Wn} .
•
Algorithms for generating In-Coverage Example:
1. Iterative-Deepening (ID) Algorithm.
2. A* Algorithm
3. A combined A*-ID Algorithm.
4. Anytime Algorithm.
Example session with targeted help
version of the Home Control System
User
System
User
System
: Hello?
: Hello, How can I help ?
: Um, Can I have some lights on ?
: Sorry, I didn’t quite catch that. To turn a device on or off, you
could try something like “turn on the kitchen light”.
User
: Turn on the kitchen light .
System : The kitchen light is now on.
User
: What other sort of things are there ?
System : I didn’t quite catch that . To find out what devices there are you
could try something like “ what devices are there” , or “What
devices are there in the lounge”.
User
: What devices are there in the lounge ?
System : There are three things in the lounge; the computer, the VCR and
the television.
DESIGN OF EXPERIMENTS
• To assess the effectiveness of the targeted help of this system.
• Compared the performance of 2 groups of 20 users , One that
received help and one that didn’t.
• Provided minimal written instruction on how to use the system.
• Task is to direct a helicopter within a city to complete various tasks.
• Given task is ended when one of the following is met.
- The task was accurately completed.
- The user thinks he completed the task, though it wasn’t.
- The user gave up.
• The system produced feedback to group in help group.
• They concluded from the experiment that , users achieved
-> HIHER TASK COMPLETION .
-> REDUCED TIME.
EXPERIMENTAL RESULTS
•
Found clear evidence that targeted help improves performance in
this environment, by showing statistical analysis of the experiment.
•
Users who received help were less likely to give up than those who didn’t
received help.
- During first task 11% Vs 45%
•
Users who received help took less time to complete tasks than those who
didn’t , 290.4s Vs 440.6s.
•
Users who didn’t receive help took 67.0% longer to complete than those
who received.
•
Finally, 89% users in the help group and 40% of the users in the No Help
group accurately completed the task.
•
Conclusion: It is possible to construct effective Targeted help messages even
from fairly low quality secondary recognizers.
•
Such an approach can improve the speed of the training for Naive users and
may result in lasting improvements in the quality of their understanding.
My Suggestions
• It is always better to provide instructions to users on how to use the
system and some in-coverage examples.
• If possible system should give mock session to the user and evaluate
the users level of expertise.
• This leads to better interaction between the user and the system.
• Giving the system a way to adjust its help strategy to the level of
expertise of the user is to be accomplished.
• We should use both recognizers simultaneously and select the one
which gives higher confidence value.
• Adding multimodal Input/Output and Animated Agents along with
this approach in ITSs is very useful.
feedback of geometry tutor
• The cognitive tutor component determines what feedback to give to
the student , based on the classification of the explanation.
• NLU component uses a knowledge-based approach to recognize
sentences as correct or partially correct explanations.
• It uses a Statistical Text Classifier (STC) when the knowledge-based
method fails which acts as a Backup. This is based on the Naive
Bayes classification method.
• System falls back on the STC to determine whether the student’s
explanation is focusing on the right geometry rule.
• The tutor say “You appear to be focusing on the right geometry rule.
However , the tutor doesn’t understand your explanation, Could you
please state it in a different way”.
• Thus, the STC enables the tutor to provide more informative
feedback in response to unexpected input.
feedback of geometry tutor: Example
• Explanation of the Triangle sum theorem
Student : they are 180.
Tutor
EQUAL-180
: Could you be more specific? Who are “they”? And what tells
you that they are 180.
Student : the angles are 180.
Tutor
ANGLES-180
: You are heading in the right direction, but you need to state
this rule more precisely. First where does the 180 come
from? Are all angles 180?
Student : the angles in a triangle are 180.
ANGLES-OF TRIANGLE-180
Tutor
: You are awfully close . But is each angle measure equal to
180? Precisely what is 180?
Student
: the sum of angles in a triangle are 180.
Tutor
: Correct.
TRIANGLE-SUM
SUMMARY
• Seen experimental evidence that providing Naive users an
immediate help message improves their performance and achieves,
- HIHER TASK COMPLETION .
- REDUCED TIME
• Adding SLM Recognizer to the Dialogue system makes system
robust.
• Targeted help makes Naïve users expertise in the usage of the
system in a quick time.
• In-Coverage example leads to greater linguistic alignment between
the user and the system.
• This approach is easy to add to an existing system with little
modifications.
• Seen how this approach is useful in some ITSs .
references
1.
2.
3.
4.
5.
6.
Generating Canonical examples using candidate words. Dowding,
G.Aist , B.A. Hockey, and E.O. Bratt. 2002.
G.Gorell, I.Lewin, and M.Rayner. 2002. Adding intelligent help to
mixed-initiative spoken dialogue systems.
B.A. Hockey, G.Aist, J.Dowding, and J.Hieronymus. 2002a.
Targeted help and dialogue about plans.
B.A. Hockey, G.Aist, J.Dowding, J.Hieronymus, and O. Lemon .
2002b. Targeted Help: Embedded training and methods for
evaluation.
B.A. Hockey, G.Aist, J.Dowding, J.Hieronymus, O. Lemon, E.
Campana, L. Hiatt, A. Gruenstein. Targeted Help for spoken
Dialogue Systems: Intelligent feedback improves Naïve users
performance.
V. Aleven, O. Popescu, and K.R. Koedinger. Towards Tutorial
Dialog to support Self Explanation : Adding Natural Language
Understanding to a Cognitive Tutor.
THANK YOU !
ANY QUESTIONS?