error handling –
Higgins / Galatea
Dialogs on Dialogs Group
July 2005
1
work by …

Gabriel Skantze
ph.d. student
KTH, Stockholm
“I am doing research on spoken dialogue
systems. More specifically, I am interested in
studying miscommunication and error
handling, but also in the representation and
modelling of utterances and dialogue, as
well as conducting experiments with users.“

and co-authors:
J. Edlund, D. House, R. Carlson
2
3 papers

Higgins
Higgins – a spoken dialogue system for investigating error handling techniques,
Edlund, Skantze, Carlson [2004]

Galatea
GALATEA: A Discourse Modeller Supporting Concept-Level Error Handling in
Spoken Dialog Systems, Skantze [2005]

Prosody & Clarifications
The Effects of Prosodic Features on the Interpretation of Clarification
Ellipses, Edlund, House, Skantze [2004]
3
1st paper

Higgins
Higgins – a spoken dialogue system for investigating error handling techniques,
Edlund, Skantze, Carlson [2004]

Galatea
GALATEA: A Discourse Modeller Supporting Concept-Level Error Handling in
Spoken Dialog Systems, Skantze [2005]

Prosody & Clarifications
The Effects of Prosodic Features on the Interpretation of Clarification
Ellipses, Edlund, House, Skantze [2004]
4
Higgins

practical goal of Higgins project


build a collaborative dialog system in which error handling
ideas can be tested empirically
error handling issues, plus




incremental dialogue processing
on-line prosodic feature extraction
robust interpretation
flexible generation and output
5
domain

pedestrian city navigation and guiding



complex




user gives system a destination
system guides user by giving verbal instructions
large variety of error types
semantic structures can be quite complex
reference resolution
domain can be extended even further
6
architecture


follow-up from Adapt
everything is XML






domain objects
utterance semantics
discourse model
database content
system output (before surface)
3D city model
7
research issues




early detection and correction
late detection
incrementality
error recovery
8
early detection and correction


KTH LVCSR – output likely to contain
errors 
robust interpretation Pickering:



some syntactic analysis is needed
 e.g. relations between objects
but handles insertions and non-agreement phrases
humans - good at early detection (woz)
9
late detection and correction

discourse modeller (GALATEA)



joins several results from Pickering into a discourse model
adds grounding information
can be manipulated later
 remove concepts which turn out not to be grounded
10
incrementality

end-pointers cause trouble

even more so in this domain
better:
11
incrementality [2]


all components support incremental
processing
several issues



when to barge in? (semantic content and prosody)
longer-than-utterance units: interpreter or dialog manager?
rapid and unobtrusive feedback: challenge for synthesis
12
error recovery

signaling non-understandings



decreased experience of task success
slower recovery
ask other task-related question
13
2nd paper

Higgins
Higgins – a spoken dialogue system for investigating error handling techniques,
Edlund, Skantze, Carlson [2004]

Galatea
GALATEA: A Discourse Modeller Supporting Concept-Level Error Handling in
Spoken Dialog Systems, Skantze [2005]

Prosody & Clarifications
The Effects of Prosodic Features on the Interpretation of Clarification
Ellipses, Edlund, House, Skantze [2004]
14
GALATEA

a discourse modeller for
conversational spoken dialog systems



builds a discourse model (what has been said during the
discourse)
resolution of ellipses & anaphora
tracks the grounding status


who said what when (plus confidence information)
can be used for concept-level error handling
15
should do grounding at
concept level

explicit and implicit verification on whole
utterance can be tedious and unnatural

45% of clarifications in BNC are fragmentary
/ elliptical
16
should do grounding at
concept level





Traum (1994) – utterance level computational model of
grounding
Larsson (2002) – issue-level computational model of
grounding in Issue-Based DM
Rieser (2004), Schlangen (2004): systems capable of
fragmentary clarification requests, but models do not handle
user reactions
systems should keep grounding information at
the concept level
like RavenClaw? 
17
semantic representation

rooted unordered trees of semantic
concepts

nodes: attr-value pairs, objects, relations, properties
18
semantic representation

enhanced with “meta”-information



confidence
communicative acts
info is new / given
19
ellipsis resolution


transforms
ellipsis into full
propositions
rule based


~10 rules
domain-specific
20
anaphora resolution

keeps a list of entities (talked about)





assigns ids
when given entities are added to the
discourse, look up the antecedent
if found, unification (and move to the top of
the entity list)
unification also allows entities to be referred
to in new ways
how does this fare and compare?
21
grounding status

who added the concept?
in which turn?
how confident?

may be used by the action manager



for instance remove all items with high grounding status
when referring to an entity
22
updating grounding status
23
late error detection


discover inconsistencies in discourse
model
look at grounding status to see where
error may be

concept can be removed
24
future






methods for automatic tuning of strategy
selection
extend to track confidence and grounding
status at different levels
evaluate
how people respond to incorrect
confirmations, and how can that information
be used to update grounding status
error recovery after non-understandings
other domains
25
3rd paper

Higgins
Higgins – a spoken dialogue system for investigating error handling techniques,
Edlund, Skantze, Carlson [2004]

Galatea
GALATEA: A Discourse Modeller Supporting Concept-Level Error Handling in
Spoken Dialog Systems, Skantze [2005]

Prosody & Clarifications
The Effects of Prosodic Features on the Interpretation of Clarification
Ellipses, Edlund, House, Skantze [2004]
26
prosody in clarifications

effects of prosodic features on
interpretation of elliptical clarifications




U: Further ahead on the right I see a red building…
S: Red (?)
vary prosodic features
study impact on user’s understanding
of the system’s intention
27
motivation

long (whole utterance) confirmations
are not good



short confirmations


tedious, unnatural
BNC corpus: 45% of clarifications are elliptical
make dialog more efficient by focusing on the actual
problematic fragments
however

interpretation depends on context and prosody
28
3 readings





U: Further ahead on the right I see a red building…
S: Red (?)
Ok, red [all positive]
Do you really mean red? What do you mean
by red? [positive perception, negative understanding]
Did you say red? [positive contact, negative perception]
29
stimuli



3 test words [red, blue, yellow]
di-phone voice (MBROLA)
manipulated



peak position [mid, early, late / 100ms]
peak height [130Hz / 160 Hz]
vowel duration [normal, long / +100ms]
30
subjects + design




8 speakers: 2f / 6m, 2nn / 6n
introduced to Higgins
listen to all 42 (only once); random
order
3 options



Okay, X
Did you really mean X?
Did you say X?
31
results

no effects for


color, subject, duration
significant effects for

peak position, peak height, & their interaction
32
results



Statement: early, low peak
Question: late, high peak
Clear division between “did you mean” and “did you say”
33
food for thought





how about English?
red
red?
red!?
how many ways can you say it?
34
conclusion




strong relationship between intonation
and meaning
statement: early, low peak
question: late, high peak
clear division between “did you mean”
and “did you say”
35
the end
36