Intonational and Its Meanings
Julia Hirschberg
CS 6998
7/15/2016
1
What do speech researchers do?
Study human production and perception
Try to embody it in machines
Production: TTS, CTS
Perception: ASR, ASRU, speaker ID, language
ID
7/15/2016
2
Pitch Accent/Prominence in ToBI
 Which items are made intonationally prominent and
how?
 Accent type:
H*
L*
L*+H
L+H*
simple high (declarative)
simple low (ynq)
scooped, late rise (uncertainty/ incredulity)
early rise to stress (contrastive focus)
H+!H* fall onto stress (implied familiarity)
7/15/2016
3
•Downstepped accents:
•!H*,
•L+!H*,
•L*+!H
•Degree of prominence:
within a phrase: HiF0
across phrases
7/15/2016
4
Functions of Pitch Accent
 Given/new information
S: Do you need a return ticket.
U: No, thanks, I don’t need a return.
 Contrast (narrow focus)
U: No, thanks, I don’t need a RETURN…. (I need
a time schedule, receipt,…)
 Disambiguation of discourse markers
S: Now let me get you the train information.
U: Okay (thanks) vs. Okay….(but I really want…)
7/15/2016
5
Prosodic Phrasing in ToBI
 ‘Levels’ of phrasing:
intermediate phrase: one or more pitch accents
plus a phrase accent (Hor L)
intonational phrase: 1 or more intermediate
phrases + boundary tone (H%
or L%
)
 ToBI break-index tier
0 no word boundary
1
word boundary
2
3
4
7/15/2016
strong juncture with no tonal markings
intermediate phrase boundary
intonational phrase boundary
6
Functions of Phrasing
 Disambiguates syntactic constructions, e.g. PP
attachment:
S: You should buy the ticket with the discount
coupon.
 Disambiguates scope ambiguities, e.g. Negation:
S: You aren’t booked through Rome because of
the fare.
 Or modifier scope:
S: This fare is restricted to retired politicians and
civil servants.
7/15/2016
7
L-L%
L-H%
H-L%
H-H%
H*
L*
L*+H
7/15/2016
8
L-L%
L-H%
H-L%
H-H%
L+H*
H+!H*
H* !H*
7/15/2016
9
Contour Examples
http://www.cs.columbia.edu/~julia/cs6998/card
s/examples.html
7/15/2016
10
Contours: Accent +
Phrasing
 What do intonational contours ‘mean’ (Ladd ‘80,
Bolinger ‘89)?
Speech acts (statements, questions, requests)
S: That’ll be credit card? (L* H- H%)
Propositional attitude (uncertainty, incredulity)
S: You’d like an evening flight. (L*+H L- H%)
Speaker affect (anger, happiness, love)
U: I said four SEVEN one! (L+H* L- L%)
“Personality”
S: Welcome to the Sunshine Travel System.
7/15/2016
11
Propositional attitude (uncertainty)
Did you feed the animals?
I fed the L*+H goldfish L-H%
Distinguish direct/indirect speech acts
Can you open the door?
7/15/2016
12
And Other Things Contribute:
Pitch Range and Timing (Rate, Pause)
Level of speaker engagement
Hello vs. HELLO
 Contour interpretation
Rise/fall/rise (L*+H L-H%): Elephantiasis isn’t
incurable
 Discourse/topic structure: paratones
7/15/2016
13
Prosodic Generation for TTS
Corpus-based approaches
Train prosodic variation on large labeled
corpora using machine learning
techniques
Accent and phrasing decisions
Associate prosodic labels with simple
features of transcripts
To do:
Contour variation
7/15/2016
14
Timing and backchanneling
Disfluencies?
Emotion and ‘personality’
Personalized voices
7/15/2016
15
Concept to Speech
Decisions in TTS depend on text analysis
Concept-to-Speech (CTS) systems should be
able to do better
System knows what it wants to say and can
specify how
But….
Still need labeled corpora to train on
CTS features may be hard to label (focus,
given/new,…)
How to decide how to realize these?
7/15/2016
16
Prosody in ASRU
Little success in improving ASR transcription
More promise in other areas:
Improving rejection
Shrinking search space
Automatic topic segmentation for
browsing/retrieval
Identifying ‘salient’ words in turns
Disambiguating speech/dialogue acts: okay
7/15/2016
17
Recognizing communicative ‘problems’
ASR errors
User corrections
‘Aware’ turns
‘Problematic’ dialogues
Disfluencies and self-repairs
Recognizing speaker emotion
7/15/2016
18
Some Research Topics
Meaning of intonational contours:
Rise/fall/rise (L*+H L-H%)
A: Did you take out the garbage?
B: Sort of.
A: Sort of!
High rise questions (H* H-H%)
This is the chicken Chermula?
I’m from Skokie?
7/15/2016
19
Compositional theory of intonational meaning
(w/Pierrehumbert)
Intonational disambiguation across languages:
Spanish, Italian and English (w/Avesani &
Prieto)
William isn’t drinking because he’s unhappy
• Disfluencies: self-repairs (w/Nakatani)
I want to go to Ba- Baltimore.
• Cue phrases (w/Litman)
• Now let’s go to work.
7/15/2016
20
Accent and strict/sloppy interpretations of
ellipsis (w/Ward)
People who live in Los Angeles adore it’s
beaches and so do people who live in New
York
7/15/2016
21
•Accent and given/new (w/Terken)
The ball touches the circle.
The ball touches the triangle.
The ball touches the cone.
The square touches the ball.
Intonation and discourse structure (w/Grosz &
Nakatani)
Boston Directions Corpus
Automatic assignment of accent and phrasing
for TTS (w/Wang, Sproat, Koehn, Abney, Collins,
Rambow)
7/15/2016
22
ToBI prosodic labeling conventions w/many)
Prosody in dialogue systems (w/Litman &
Swerts): generation and understanding (TOOT)
Audio browsing and retrieval: SCAN and
SCANMail (w/many)
7/15/2016
23
Potential Projects
Build a TTS system in a limited domain
Build a speech recognizer
Study a speech phenomenon (disfluencies,
accenting, contours, pitch range variation)
Do some experiments (production, perception).
Examples:
Speech summarization, eye tracking and
emotion, deceptive speech, given/new and
contour,….
7/15/2016
24