Module u1:
Speech in the Interface
6: Human Communication
Jacques Terken
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
1
Motivation


provide knowledge about basic concepts of human
communication relevant to the design of speech interfaces
explore use of knowledge about human communication as
model for man-machine interaction
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
2
Contents








basic scheme for communication
the structure of messages
the structure of communication
“how to do things with words”1: communication as process
implications for man-machine interaction
key points
human-human communication as model for human-computer
interaction
1
J.L. Austin: “How to do things with words”, 1962
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
3
basic scheme
context
sender
channel
context
receiver
feedback

Context
– Situational context  situated communication: time, place, objects
– Social context
– Cultural context
– Dialogue context
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
4
intentions and effects


Sender: intention to convey a message
 message needs to be encoded
Receiver: decoding  effect
Ideal communication: intention = effect
Distortions underway: intention  effect’
Furthermore: effect is more than intention (sender also
communicates unintentional information)
complex messages require sequence of steps
 communication as structured action / collaboration

SAI User-System Interaction U1, Speech in the interface: 6. Human communication
5
Contents







basic scheme for communication
the structure of messages
the structure of communication
communication as process
implications for man-machine interaction
key points
human-human communication as model for human-computer
interaction
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
6
formal aspects (1)

Simple case: messages containing a single dialogue act
(inform, request, confirmation etc)

Sentence/utterance
– put the red block before the yellow block next to the ball
– I herewith pronounce you man and wife
– What time does the next flight to Boston leave?
Sentences made up of word groups
Word groups made up of words
Words made up of syllables and phonemes (~40) or
graphemes (~25)
/ @, e, a, r, … / ; { e, e, a, r, … }



SAI User-System Interaction U1, Speech in the interface: 6. Human communication
7
formal aspects (2)

rules determine well-formedness of representations at different
levels:
– pragmatics: sentence meaning in context (speech act,
dialogue act)  contextual meanings
*the king of France is bald (but may be okay in a story)
– semantics: sentence meanings  possible meanings
*the 50 year old bachelor surprised his wife
– syntax: sentence structure  possible sentences
– morphology: word structure  possible words
– Dutch: loopje; *loopetje; kommetje; *kompje
– Phonology: sound structure  possible sound sequences
Dutch: herfst; Polish: przewalski
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
8
formal aspects (3)



mapping between levels not fully specified (not one - to - one):
 ambiguity
Mapping sounds onto words:
/Aiscri:m/  {I scream} or {Ice cream}
Mapping words onto sentence structure:
Syntactic ambiguities
i saw the girl with the binoculars
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
9
put the red block before the yellow block next to the ball
(1) put the red block
before the yellow block [ that is ] next to the ball
(2) put the red block before the yellow block,
next to the ball
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
10
Contents







basic scheme for communication
the structure of messages
the structure of communication
communication as process
implications for man-machine interaction
key points
human-human communication as model for human-computer
interaction
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
11
levels of analysis
topic
…
topic
…
topic
transaction
exchange exchange exchange
turn
turn
turn
move
move
move
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
12
transaction


(part of) conversation dealing with a single topic
consists of one or more exchanges
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
13
exchange




smallest interactive unit
consists minimally of two turns produced by two different
speakers
involves the negotiation of a single piece of information
examples:
request - inform - confirm
presentation - acceptance
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
14

adjacency pairs:
question
answer
request
accept / turn down
invitation
accept / decline
greeting
greeting
apology
smoother

Interactive pressure: opening move induces obligation to answer
with closing move (conversational obligation)
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
15

nesting of adjacency pairs:
s: question X
what time is the next train to amsterdam
a: question Y
amsterdam central?
s: answer Y
yes
a: answer X
in seven minutes at platform 5
mixed initiative vs. system initiative
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
16
turn



Turn: everything the speaker says before the next speaker takes
over
Rules for turn-taking behaviour
1. Speaker selects next speaker
2. Next speaker selects self
Window of opportunity for turn switching (minimizes overlap and
interruption; interruptions only permissible in particular
circumstances
Turn consists of one or more moves
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
17
temporal aspects of turn-taking

dialog not as a sequence of well-formed non-overlapping turns:
– time window for turn taking
– overlap, interruptions
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
18
move/dialog act




no fixed inventory, but dependent on application
Common types:
 Dialogue control acts
Primary dialogue acts
– acknowledge
– request for action
– check
– inform
– confirm
– question
– greeting
– thanks
Game theory conceives of communication as a sequence of
moves
Expresses the assumption that communication is governed by a
set of rules
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
19
Indirect communication


No one-to-one mapping between form and function: indirect
communication (politeness forms)
Questions:
– Can you tell me what time it is (yes/no question)
– I would like to know what time it is (statement)
– Do you have a watch (yes/no question with indirect
communication )
Direct communication:
– What time is it (wh-question)
– Please tell me the time (request)
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
20
spoken dialog systems

Identify dialog act performed by user in order to decide what
dialog act to produce as an an answer

Machine learning techniques in order to learn best continuation
in context x (and to learn best path)
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
21
Contents







basic scheme for communication
the structure of messages
the structure of communication
communication as process
implications for man-machine interaction
key points
human-human communication as model for human-computer
interaction
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
22
spoken communication as joint
action/cooperation



Speech is volatile
Mechanisms to cope with volatile nature of speech
co-operation
receiver must be in receiving mode, i.e., be open to message
sender must adjust to channel and receiver characteristics
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
23
spoken communication as joint
action/cooperation (2)



Goal achievement is usually accomplished through a
number of steps involving both participants: dialogue
Monitoring of transmission success by sender on the basis
of conscious and unconscious feedback from receiver
Immediate action in case of problems either by user or
sender
– Role of eye gaze in staying in touch, monitoring and
turn taking
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
24
speech generation

two simultaneous processes:
– conceptualisation (“what to say”: content)
– formulation (“how to say it”: converting content into
utterance)
simultaneous, but formulation lagging behind
(  spontaneous speech phenomena)
– restarts, self-corrections, hesitations (silence, ehm),
disfluencies
– grammar of spoken language
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
25
message formulation: issues

continuity, coherence: what we say usually links to what was
said before:
context provides search space for interpretation and allows
for efficient expression
– pronouns, anaphora
– prosody
– bridging: establishing a relation between expressions in
consecutive utterances
can you take the picnic supplies out of the van? the beer is
probably warm, so please put it in the fridge.
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
26
grice’s cooperativity principle




human communication (supposed to be) governed by grice’s
cooperativity principle, consisting of the following maxims
(guidelines for communication):
quantity: make your contribution as informative as is required
I want a flight; I want a flight from A to B because ….
quality: do not say what you believe is false or for which you
lack adequate evidence
relation: be relevant
manner: be perspicuous, brief and orderly; avoid obscurity
and ambiguity
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
27
speech comprehension

Stages in comprehension:
– Perception: perceiving sounds and mapping sounds onto
words
– Interpretation
– Evaluation
– From a processing point of view perception and
interpretation are closely interlinked
keywords: on-line processing, hypothesis testing and
integrating
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
28
backchannelling

“utterances” and other communicative signals guiding the
progress of the communication, i.e., serving a control function
– nodding
– yes, hm-hm
– “madrid?”
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
29
error recovery

on-line property allows fast and efficient recovery:
– problems noticed immediately
– cause of problem usually easily identified
“what did you say?”
“did you say …”
“what do you mean”
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
30
common ground


Grounding
agreeing about current state of the dialogue
Important role of feedback
Shared knowledge
+ knowledge that speaker knows is available to hearer
+ world knowledge (“living memory”)
+ knowledge concerning context of communication and what
has been established by previous part of discourse
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
31
Contents







basic scheme for communication
the structure of messages
the structure of communication
communication as process
implications for man-machine interaction
key points
human-human communication as model for human-computer
interaction
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
32




human communication no neat succession of turns: overlap in
time: provision for speech recognition needed
Indirect communication: message function and form no 1:1
relation
interpretation dependent on domain knowledge: requires
domain modelling: successful for restricted domains
interpretation dependent on context: requires discourse model
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
33
supporting the interaction



Prompting: system initiative
Zooming: increase amount of feedforward in system prompt
– How may I help you
– Which connection
– Are you interested in connections or prices
– Please respond with yes or no
Cooperative replies
– User: “Do you have flights between 7:15 and 7:45?”
– S: “No, but there is a flight at 7:55”
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
34
robustness issue

Lack of robustness creates need for verifcation

Explicit verification:
– Did you say “amsterdam”?
Implicit verification
– U: “I want to travel from .. to rotterdam”
– S: “when do you want to travel to rotterdam?”

– Cf: S: “when do you want to travel to amsterdam?”
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
35




Verification is costly when you got it right
Feedback (positive feedback or implicit verification) is
misleading when you good it wrong
Use verfication when uncertain
Use feedback when confident
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
36
confidence-based verification




High confidence: just accept or implicit verification
Medium confidence: explicit verification
Low confidence: explicit request for repetition
Precision/recall issue for correctness and confidence measure:
– Confidence measure may not be right: either too high or too
low
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
37
Contents







basic scheme for communication
the structure of messages
the structure of communication
communication as process
implications for man-machine interaction
key points
human-human communication as model for human-computer
interaction
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
38




communication as joint action (cooperation)
– taking into consideration listener needs
– listening for intention
rule-governed behaviour at many different levels
– structure of messages
– structure of conversations (e.g. turn-taking behaviour)
grounding
feedback (back-channelling)
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
39
Contents







basic scheme for communication
the structure of messages
the structure of communication
communication as process
implications for man-machine interaction
key points
human-human communication as model for human-computer
interaction
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
40
cooperative assistant

Characteristics of cooperative assistant
– Interprets messages in terms of intentions not in terms of
form
(“c.a. is not a politician or lawyer”)
– Cooperative replies
(not blunt “no” but offers alternatives, e.g. through
constraint relaxation)
– Context-sensitive replies and help
No global help manual
– Proactive, triggered by implicit cues
Does not await explicit request for help but anticipates
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
41
discussion


Is this a good model for human computer interaction?
Considerations:
– Schneiderman c.s.: user should stay in control
– Maes/Sidner:
• computers should be able to understand our intentions
so that we don’t have to specify all the actions explicitly
ourselves
• Computers should monitor our actions so that they can
learn how to anticipate our needs and preferences
SAI User-System Interaction U1, Speech in the interface: 6. Human communication
42