ACL, June 2007, Prague
On the role of context and prosody
in the interpretation of ‘okay’
Agustín Gravano, Stefan Benus, Julia Hirschberg
Héctor Chávez, and Lauren Wilcox
Spoken Language Processing Group
Columbia University
Overview



What information do subjects use to interpret
the word ‘okay’ in dialogue?
Perception Study.
Findings:


Contextual cues stronger predictors than acoustic
/ prosodic / phonetic cues.
Final rising pitch: Strongest prosodic cue.
Agustín Gravano - ACL - June 2007
2
Overview

that’s pretty much okay

Speaker 1: between the yellow mermaid and
the whale
Speaker 2: okay
Speaker 1: and it is

okay we gonna be placing the blue moon
Agustín Gravano - ACL - June 2007
3
Cue Words

Linguistic expressions that can be used




to convey information about the discourse
structure, or
to make a semantic contribution.
Discourse markers, cue phrases, clue
words, …
Examples:

now, well, so, alright, and, okay, first, on the other
hand, by the way, for example, …
Agustín Gravano - ACL - June 2007
4
Research Questions

In spoken dialogue, how do hearers
disambiguate cue words?



How important is acoustic/prosodic information?
What is the role of phonetic variation?
What is the role of discourse context?
Agustín Gravano - ACL - June 2007
5
Why do we care?

Spoken dialogue systems


Need to convey potentially ambiguous terms with
a particular intended meaning.
Must interpret the user’s input correctly.
Agustín Gravano - ACL - June 2007
6
Previous Work

Cues to cue phrase disambiguation


Cues to Dialogue Act identification


Hirschberg & Litman ’87, ’93; Hockey ’93; Litman ’94
Jurafsky et al ’98; Rosset & Lamel ’04
Contextual cues to the production of
backchannels

Ward & Tsukahara ’00; Sanjanhar & Ward ’06
Agustín Gravano - ACL - June 2007
7
The Columbia Games Corpus


12 spontaneous task-oriented dyadic conversations
in Standard American English.
2 subjects playing a computer game, no eye contact.
Describer:
Follower:
Agustín Gravano - ACL - June 2007
8
The Columbia Games Corpus
Annotation of Affirmative Cue Words
Cue Words











alright
gotcha
huh
mm-hm
okay
right
uh-huh
yeah
yep
yes
yup
Functions




count
1. the
4565
2. of
1534
3. okay 1151
4. and
886
5. like
753
…






Acknowledgment / Agreement
Backchannel
Cue beginning discourse segment
Cue ending discourse segment
Check with the interlocutor
Stall / Filler
Back from a task
Literal modifier
Pivot beginning
Pivot ending
Agustín Gravano - ACL - June 2007
9
Perception Study
Experiment Design
Speaker 1: yeah um
there's like there's/ Agreement
some space there's
 Acknowledgment
Speaker 2: okay I think
I got it
Backchannel
 Cue beginning discourse segment

okay
Speaker 1: but it's gonna be below the onion
Speaker 2: okay
Speaker 1: okay alright I'll try it okay
Speaker 2: okay the owl is blinking
Agustín Gravano - ACL - June 2007
10
Perception Study
Experiment Design
54 instances of ‘okay’ (18 for each function).
2 tokens for each ‘okay’:
 Isolated condition: Only the word ‘okay’.
 Contextualized condition: 2 full speaker turns:
•
•
The turn containing the target ‘okay’; and
The previous turn by the other speaker.
speakers
okay
contextualized ‘okay’
Agustín Gravano - ACL - June 2007
11
Perception Study
Experiment Design

Two parts:
Part 1: 54 isolated tokens
Part 2: 54 contextualized tokens

Subjects asked to classify each token of ‘okay’
as:



Acknowledgment / Agreement, or
Backchannel, or
Cue beginning discourse segment.
Agustín Gravano - ACL - June 2007
12
Perception Study
Experiment Implementation

Subjects:





20 paid subjects (10 female, 10 male).
Ages between 20 and 60.
Native speakers of English.
No hearing problems.
GUI on a laboratory workstation with
headphones.
Agustín Gravano - ACL - June 2007
13
Results
Inter-Subject Agreement

Kappa measure of agreement with respect to
chance (Fleiss ’71)
Isolated Condition
Contextualized Condition
Overall
.120
.294
Ack / Agree vs. Other
.089
.227
Backchannel vs. Other
.118
.164
Cue beginning vs. Other
.157
.497
Agustín Gravano - ACL - June 2007
14
Results
Cues to Interpretation

Phonetic transcription of okay:

Isolated Condition
Strong correlation for realization of
:
 Backchannel
 Ack/Agree, Cue Beginning

Contextualized Condition
No strong correlations found for phonetic variants.
Agustín Gravano - ACL - June 2007
15
Results
Cues to Interpretation
Isolated Condition
Shorter /k/
Shorter latency between turns
Shorter pause before okay
Lower intensity
Higher final pitch slope
Longer 2nd syllable
Higher final pitch slope
More words by S2 before okay
Fewer words by S1 after okay
Ack / Agree
Backchannel
Contextualized Condition
Lower final pitch slope
Longer latency between turns
Cue beginning Lower overall pitch slope More words by S1 after okay
Lower final pitch slope
S1 = Utterer of the target ‘okay’. S2 = The other speaker.
Agustín Gravano - ACL - June 2007
16
Results
Cues to Interpretation
Final intonation using the ToBI conventions.
(Both isolated and contextualized conditions.)
H-H%  Backchannel
H-L%
L-H%  Ack/Agree, Backchannel
L-L%  Ack/Agree, Cue beginning
Agustín Gravano - ACL - June 2007
17
Conclusions

Agreement:



Availability of context improves inter-subject
agreement.
Cue beginnings easier to disambiguate than the
other two functions.
Cues to interpretation:


Contextual features trump features of word okay.
Exception: Final pitch slope of okay in both
conditions.
Agustín Gravano - ACL - June 2007
18
Further Work

Benus et al, 2007


“The prosody of backchannels in American
English”, ICPhS 2007, Saarbrücken, Germany,
August 2007.
Gravano et al, 2007

“Classification of discourse functions of affirmative
words in spoken dialogue”, Interspeech 2007,
Antwerp, Belgium, August 2007.
Agustín Gravano - ACL - June 2007
19
ACL, June 2007, Prague
On the role of context and prosody
in the interpretation of ‘okay’
Agustín Gravano, Stefan Benus, Julia Hirschberg
Héctor Chávez, and Lauren Wilcox
Spoken Language Processing Group
Columbia University
The Columbia Games Corpus
Annotation






Orthographic transcription and alignment.
Laughs, coughs, breaths, smacks, throat-clearings.
Self repairs.
Intonation, using the ToBI convention.
Function of affirmative cue words (alright, mm-hm,
okay, right, uh-huh, yeah, yes, …).
Question form and function.
Agustín Gravano - ACL - June 2007
21
The Columbia Games Corpus
Annotation of Intonation: ToBI


Tones:
L*, H*, L*+H, H+!H*, …

Pitch accents:

Phrase accents: L-, H-, !H-

Boundary tones: L%, H%
Break Indices:

Degrees of junction
0 = no word boundary ... 4 = full intonational phrase boundary

Miscellanea:

Disfluencies, non-speech sounds, etc.
Agustín Gravano - ACL - June 2007
22
The Columbia Games Corpus
Annotation of Intonation: ToBI
waveform
fundamental frequency (F0)
Agustín Gravano - ACL - June 2007
23
Perception Study
Definitions Given to the Subjects

Acknowledge/Agreement:


Backchannel:


The function of ‘okay’ that indicates “I believe what you
said” and/or “I agree with what you say”.
The function of ‘okay’ in response to another speaker's
utterance that indicates only “I’m still here” or “I hear you
and please continue”.
Cue beginning discourse segment

The function of ‘okay’ that marks a new segment of a
discourse or a new topic. This use of ‘okay’ could be
replaced by ‘now’.
Agustín Gravano - ACL - June 2007
24