Martin J. Pickering, Simon Garrod, Behavioral and Brain Sciences,
2012.
An Integrated Theory of Language
Production and Comprehension
Computer Science & Engineering
2012-20835
Sang-Woo Lee
Background - Aphasia
Broca’s Aphasia
 Expressive aphasia
 Agrammatic aphasia
 Understand what other people say, but cannot speak the
sentence well.
 Caused by damage to, or developmental issues in the anterior
regions of the brain
 Including (but not limited to) the Broca’s area
3
Wernicke’s Aphasia
 Also known as Receptive Aphasia
 Fluent apahsia, or sensory aphasia
 Speak the sentence fluently, but not well-organized sense in their
speech
 Traditionally associated with neurological damage to Wernicke’s area
in the brain

(Actually it is not just simply associated to Wernicke’s area in current experimental
result, but anyway…)
4
There are modules which specify some function
perception
action
The “classical Lichtheim-Broca-Wernicke” Model
5
Contents
 Background - Aphasia
 Traditional independence of production and comprehension
 Interweaving in action and action perception
 Perception process in action
 Predict next action of other
 Joint Action
 Interweaving in Production and Comprehension
 Comprehension process in Production
 Predict next speech of other
 Interactive Language
 Professor’s Question
6
Traditional independence of
production and comprehension
Traditional model of communication
 Discrete stages
 A produces, B comprehends
 B produces, A comprehends
8
Horizontal Split
 Assumes “horizontal split” between production
and comprehension
 Arrows-within-arrows indicate feedback (in
interactive accounts)
 But this feedback is internal to production or
comprehension
 It may involve “general knowledge”
 But production does not involve comprehension processes
 And comprehension does not involve production processes
9
Example of predict in Behavioral Instance
 Interlocutors are not static, as the traditional model assumes, but
are “moving targets” performing a joint activitiy (Garrod &
Pickering, 2009)
10
Example of predict in Neuroscience
(Pickering & Garrod, 2007)
Big(neuter)
Painting
(neuter)
große Gemälde
Big (common)
Bookcase (common)
großen Bücherschrank
11
Other Counterexample
 Also, many experiments demonstrate effects of
one on the other
 Picture-word interference (Schriefers et al., 1990)
 Word identification affected by externally controlled
cheek movement (Ito et al., 2009)
 And strongly overlapping neural circuits for
production and comprehension (e.g.,
Pulvermüller & Fadiga, 2010; Scott et al., 2009)
12
Result
 Prediction process could be naturally understood with production
module
 When they comprehend the utterance, they also use production
model internally.
- There is forward model
To predict perception
caused by their own
utterance
- Fast alert when you say
something wrong
13
Interweaving in action and action perception



Perception process in action
Predict next action of other
Joint Action
Interweaving in action and action perception
 Close links between action and action perception,
e.g.
 participants’ arm movements affected by observing anot
her person’s arm movements (Kilner et al., 2003)
 And making hand movements can facilitate concurrent vi
sual discrimination of deviant hand postures (Miall et al.,
2006)
 Such links could have various purposes
 Supporting overt imitation
 facilitating memory or understanding (“postdictively”)
 But authors propose that they aid prediction of ow
n and others’ actions, by use of a forward model
 Based on computational neuroscience (Wolpert, 1997; se
e Grush, 2004)
Forward modelling in action
 In our terms, the action command causes the action implementer
to move the hand and the perceptual implementer to construct t
he percept
 And the efference copy causes the forward action model to gener
ate the predicted hand movement and the forward perceptual m
odel to construct the predicted percept
Forward Modeling in Action
- Just act
- Feel involved percepts
of your own act
e.g. Own coordination
info
Feeling of wind blowing
to your arm
Gravity info …
Efference Copy
18
Forward Modeling in Action
- Predict perception
caused by their own action
e.g. Own coordination info
Feeling of wind blowing to your arm
Gravity info …
Prediction-by-Simulation
- Predict perception of
other’s next action
by seeing other’s current
action
Joint Action
 People are highly adept at joint activities (Seba
nz et al., 2006).
 ballroom dancing, playing a duet, carrying a large o
bject together
 Precise timing is crucial
 To succeed, A predicts B’s action and B predict
s B’s action
Joint Action
22
Interweaving in Production and Comprehension



Comprehension process in Production
Predict next speech of other
Interactive Language
Forward modeling in language production
 Action implementer  production
implementer
 Perceptual implementer  comprehension
implementer
 Action command  production command
 Drives the production implementer
 Efference copy drives the forward models
 Comparator  monitor
 compares the utterance percept and the predicted
utterance percept
Unifying production and comprehension
 Production and comprehension are interwoven
 Tight coupling in dialogue (Clark, 1996; Pickering &
Garrod, 2004)
 Behavioural experiments show effects of
comprehension processes on production and vice
versa (e.g., Schriefers et al., 1990)
 Overlap of brain circuits for production and
comprehension (e.g., Pulvermuller & Fadiga, 2010)
 Such interweaving facilitates prediction of self
and other’s utterances
Classical modeling in language production
production command
i t 
Production
implementer
Utterance
- Just say utterance,
- Listen what you say.
psem, syn, phont 
Comprehension
implementer
Utterance percept
csem, syn, phont 
Efference
copy
monitor
Predicted utterance percept
cˆsem, syn, phont 
Forward
comprehension
model
Forward
production
model
Predicted utterance
pˆ sem, syn, phont 
26
Forward modeling in language production
production command
i t 
Production
implementer
- Predict perception
caused by their own
utterance
- Fast alert when you say
something wrong
Utterance
psem, syn, phont 
Comprehension
implementer
Utterance percept
csem, syn, phont 
Efference
copy
monitor
Predicted utterance percept
cˆsem, syn, phont 
Forward
comprehension
model
Forward
production
model
Predicted utterance
pˆ sem, syn, phont 
27
Self-monitoring
 Speaker wishes to say kite
 In the past, she has always constructed the
kite-concept and then uttered /k/
 She therefore constructs forward model
p^[phon](t) = /k/
 If she then incorrectly constructs p[phon] = /g/,
the monitor notices the mismatch
 If she believes the forward model, she will
detect an error (and perhaps reformulate)
 Otherwise, she will alter her forward model
Prediction-by-simulation
B’s utterance
Person B
B’s utterance
psem, syn, phonB t 
psem, syn, phonB t 1
Derived intentional
act of
communication
iComprehension
A(t+1)
implementer
Comprehension
implementer
- Predict perception of
other’s next utterance
by listening other’s current
utterance
Utterance percept
Covert
imitation
csem, syn, phonB t 
Utterance percept
csem, syn, phonB t  1
monitor
Inverse
model +
context
Predicted utterance percept
cˆsem.syn, phonB t  1
Person A
Derived production
command
iB t 
Overt
Responses
Forward
comprehension
model
Derived production
command
Big(neuter)
Painting
(neuter)
iB t  1
Efference
copy
Forward
production
model
Predicted utterance
pˆ sem, syn, phonB t  1
29
Interactive Language
 Joint action involves combining accounts of action
and action perception
 Similarly, interactive language involves combining a
ccounts of production and comprehension
 Facilitates coordination (e.g., short intervals between spea
kers; Wilson & Wilson, 2005)
 Facilitates alignment (developing same representations; Pi
ckering & Garrod, 2004)
 Alignment in turn facilitates comprehension (better predic
tion of others)
30
Interactive Language
31
Conclusion
 We propose that language production and comprehen
sion are interwoven
 It assumes a central role to prediction in production, c
omprehension, and dialogue
 Speakers construct forward models to predict aspects
of their upcoming utterances
 Listeners covertly imitate speakers and use forward mo
dels to predict the speakers
 Our account helps explain the efficiency of production
and comprehension and the remarkable fluidity of dial
ogue
Thank you
Professor’s Question



Comprehension process in Production
Predict next speech of other
Interactive Language
Question 1
 Q1: Give the evidence for how language production and
comprehension are tightly interwoven. How does this relate to
the perception-action cycle theory of cognitive systems?
35
Question 2
 Q2: Explain and give the evidence for how action, action
perception, and joint action are interwoven. Explain how the
authors use this to develop accounts of production,
comprehension, and interactive language.
 participants’ arm movemen
ts affected by observing an
other person’s arm movem
ents (Kilner et al., 2003)
 And making hand
movements can facilitate
concurrent visual
discrimination of deviant
hand postures (Miall et al.,
2006)
 Action implementer 
production implementer
 Perceptual implementer 
comprehension
implementer
 Action command 
production command
 Comparator  monitor
36
Question 3
 Q3: Give examples of what behavioral and neuroscientific data on
language processing can be explained by the integrated theory of
language production and comprehension explains, while modular
theory does not.
- Neuroscientific data
(Pickering & Garrod, 2007)
- Behavioral data
37