Embodied Language, Cognition, and
Communication (E LCC):
Spontaneous gesture with speech
Alan Cienki and Irene Mittelberg
Engelse taalkunde
Afdeling Taal en Communicatie
Faculteit der Letteren
LCC presentation, 6 November 2008
Some background: gesture studies
• Spoken language studied in fields such
as anthropology, sociology (e.g., CA),
ethnography of communication
• Gesture sometimes studied in these
contexts
• The study of “non-verbal
communication” in clinical psychology
Gesture studies as its own field
• E.g., Adam Kendon’s work since 1970s
– Gesture: Visible action as utterance (2004)
• Landmark – Hand and mind: What gestures
reveal about thought (David McNeill 1992)
• International Society for Gesture Studies (ISGS)
• ISGS conference series
• Founding of the journal Gesture in 2001
What are we calling a gesture?
• A visible, distinct, effortful movement
of part of the body (Kendon)
• Special status of manual gestures
Background on gesture
• Range of conventionality of
form-meaning associations –
from highly idiosyncratic to highly
conventionalized
“Kendon’s continuum” (Kendon 1982; McNeill 1992)
gesticulation - pantomime - emblems - sign language
Gesturing is not usually just
handwaving
• Not random, structureless or
unsystematic
• Basic structuring principles of gesture at
work:
– schematic patterns
– abstractions from physical actions, objects,
and relations
– roles of metonymy and metaphor
On the coding of gestural forms
• Four parameters are often used (based
on research on sign languages):
– hand shape
– palm orientation
– location in gesture space
– movement
Notation of form features in
gestures
•
status of fingers differentiated
(Bressem 2006, in prep.)
Notation of form features in
gestures
•
form class “flat hand”
(Bressem 2006, in prep.)
Models of gesture space
etc.
-1
0
1
2
3
upper
extreme periphery
periphery
center
cc
lower
right
left
(Fricke 2005/ in prep.)
(McNeill 1992)
Some functions of gestures
• Discourse structuring function
• Pragmatic/interactional function
• Referential function
• These can overlap
– Gestures can have multiple functions
Some functions of gestures
• Referential function
– Concrete reference
– Abstract reference
Some functions of gestures
Concrete reference
Oh,
an essay.
You have to `write fast!
{video clip}
(video clips have not been
included in this PowerPoint
since we do not have
permission to publish them)
Some functions of gestures
• Abstract reference:
Representation of the abstract
(non-physical) in terms of the concrete
e.g., an idea as a space, form, or
motion
 Metaphor
Example: metaphors in gestures
that,
there’s never a situation,
that is,
ideally r- -where there’s,
an ideal,
<@well@>,
<at least there isn’t in my life>,
where there’s something right,
and something absolutely wrong.
What you have to do is draw your line,
and figure out on which side of it you fall.
Example: metaphors in gestures
{video clip}
Example: metaphors in gestures
Possible analysis might include:
• SITUATIONS ARE LOCATIONS
• MAKING A DECISION IS DRAWING A LINE
• EVALUATING YOUR BEHAVIOR IS
POSITIONING YOURSELF
• Note timing of onset of gestures with
respect to speech
Metaphor
• Metaphor as not just a matter of language,
but of mapping one conceptual domain (with
imagery!) onto another (Lakoff & Johnson 1980, 1999;
and many other works…)
• Therefore, we should see evidence of
conceptual metaphors in different forms of
human expression
• Potential for multimodal expression of
metaphor in language use (Cienki & Müller 2008;
Forceville & Urios-Aparisi, forthcoming; Müller 2008)
Side note: perspective in
gesture studies
• If studying gesture can help in meaningmaking, whose meaning-making are we
talking about?
– the speaker’s?
– the addressee’s?
– another observer’s? (the researcher’s?)
Interaction of metaphor and metonymy
in co-verbal gestures
Irene Mittelberg
points of departure:
How do speech and gesture share the semiotic work of rendering abstract knowledge
domains more graspable?
Previous research has shown that spontaneous co-speech gestures may reveal:
•
embodied and situated aspects of abstract reasoning
•
materializations of spatial metaphor
•
interaction of iconicity and metaphor (also in signed languages)
Bouvet 2001; Calbris 2003; Cienki 1998, 2005; Cienki & Müller 2008; McNeill 1992, 2005;
Müller 1998, 2003, 2008; Nuñez 2004; Nuñez & Sweetser 2006; Sweetser 1998, 2007;
Taub 2001; P. Wilcox 2000, 2004; S. Wilcox 2004; Williams 2008; inter alia
While metaphor is central to accessing abstract
domains
and the gestures in the data of this study are essentially
metaphorical in nature metaphorical modes do not suffice
to account for the semantic / pragmatic
processes in the multimodal data.
=> not necessarily a direct iconic link
between
the form of the gesture and
the form of its referent.
example:
palm-up open hand gesture on mention of “noun”
as sitting on palm-up open hand
(Müller 2004)
contiguity relation between
hand and imaginary object
Grammatical categories (noun, verb)
contiguity relations between
hands & imaginary objects
{video clip}
cognitive-semiotic approach to co-speech gesture
Charles Sanders Peirce’s (1931, 1955)
pragmaticist theory of signs, focus on interpretation
perspective for analysis: addressee
-------------Roman Jakobson’s (1956, 1960 1961,1963, 1966)
theory of metaphor & metonymy
-------------Cognitive linguistics: metaphor, metonymy, image schemas
(Dirven 2002; Gibbs 1994, 2006; Johnson 1987; Lakoff 1987, 1993;
Lakoff & Johnson 1980, 1999; Radden 2000; Panther & Thornburg 2003, 2004;
Sweetser 1990; Taub 2001; Wilcox 2004)
Gesture research
(Calbris 1991; Cienki 1998; Kendon 2000, 2004; McNeill 1992, 2005;
Müller 1998, 2004, 2007; Webb 1996; Streeck 2003; Sweetser 1998;
reseach project “Towards a grammar of gesture,“ Berlin / Viadrina FFO)
--------Data:
genre:
academic discourse / introductory linguistics courses
subject matter: linguistic form, grammar, & linguistic theory
gesture types: “referential gestures” of abstract concepts and structures (Müller 1998)
meta-linguistic & meta-grammatical co-speech gestures
inventory of gestural sign carriers
in meta-linguistic & meta-grammatical gestures
data-driven approach resulted in a set of prominent
hand configurations & motion patterns
recurrent forms and recurrent form-meaning mappings
– manipulation of imaginary objects of different sizes and dimensions (solid
objects and containers with an inside)
– geometrical shapes
basic shapes: squares, triangles, regtangles, (semi-) circles, etc.
lines:
•
straight, curved, wavy along horizontal, vertical und diagonal axes
other patterns evoke some of image-schematic patterns suggested in the
cognitive linguistics literature
=> ‘material’ basis for metaphorical and metonymic projections
Mittelberg (2006, 2008, fc)
Johnson‘s original definition of “image schemas”:
“recurring, dynamic patterns of our perceptual interactions and
motor programs that give coherence and structure to our
experience.”
(Johnson 1987, XIV)
Johnson (2005, 31):
“But let us not forget that the truly significant work done by image
schemas is tied to the fact that they are not merely skeletons or
abstractions. They are recurring patterns of organism-environment
interactions that exist in the felt qualities of our experience,
understanding and thought.”
image schemata instantiated in the gesture data
SUPPORT
CONTAINMENT
OBJECT
SOURCE-PATH-GOAL
BALANCE
SCALE
CYCLE
ITERATION
FRONT-BACK
FORCE
ADJACENCY
CONTACT
PART-WHOLE
(“puoh-tray,” “puoh-cup”)
(“puoh-cup,” “fist,” “pcoh-box”)
(“puoh-tray,” “puoh-cup,” “pcoh-box,”, “fist”)
(“hori-trace,” “vert-trace,” “diag-trace; )
(“puoh-tray-bh,” “puoh-cup-bh,” “fist-bh,” “sym-offshoot”)
(“scale”)
(“circle-bh,” “wrist-rotation,” “rotation lateral”)
(“wrist-rotation,” “rotation lateral”)
(“push,” “pull”)
(“push,” “pull”, “hori-join,” “sym-offshoot”)
(Johnson 1987; Gibbs 2005; Hampe 2005; Lakoff & Johnson 1999; Mandler 1996; Talmy 1988)
Gesture research: iconic and metaphoric gestures
(Bouvet 2001; Calbris 2003; Cienki 1998, 2005; Cienki & Müller 2008; Mittelberg 2008, fc.;
Müller 1998, 2007, 2008, fc.; Sweetser 1998; Williams 2004, 2008; Zlatev 2005; inter alia)
metonymic pathways
as emphasized by relevance theorists (e.g., Sperber & Wilson 2002) and
cognitive linguists (Gibbs 1994; Lakoff 1987; Fauconnier & Turner 2002),
“metonymic pathways” are part of the cognitive competence of speakers and listeners
=> provide natural inference schemata
(Panther & Thornburg 2003, 2004)
One definition of metonymy:
“Metonymy is a cognitive process in which one conceptual entity, the vehicle,
provides mental access to another conceptual entity, the target, within the same
domain or ICM.”
“Proust is on the top shelf.”
(Kövecses and Radden 1998: 39)
Jakobson (1956): “Two types of language and
two types of aphasic disturbances”
inspired by Peirce’s (1955) concepts of similarity & contiguity
theory of
metaphor & metonymy
as two major modes of association & signification
Jakobson

devotes equal attention to both tropes and those who use his work posit a continuum between
the two.
(cf. Croft 1993; Dirven 2002; Lodge 1977; Radden 2000; Waugh 1998)

no absolute categories: they interact in any process of meaning-making to various degrees

hierarchy: the function of a given sign depends on the preponderance of one mode over the
others.
The aim here is to show how metonymic principles lead into the interpretation of
metaphoric gestures: metonymy first, metaphor second
Jakobson further distinguished between
internal metonymy:
contiguity relationships between sign & object
synecdoche (pars pro toto)
“all hands on deck” (hands stand for entire bodies)
external metonymy:
contiguity relationships between signs or between entities in extra-linguistic,
discursive, or conceptual space
“metonymy proper” (adjacency, contact)
“The White House remains silent.”
“The White House” stands for U.S. president / spokesperson (place for person)
I. internal metonymy (within metaphoric gestures)
the formation of the gestural sign
gestures exhibit the basic principle of semiotic representation
=> that is, that all semiotic representations are partial
these abstraction processes are driven by synecdoche:
by picking out the locally pragmatically salient aspects of an object / action
e.g., shape, dimensions, texture, manner of movement
for example, a gesture representing a picture frame
(Bouvet 2001; Cienki & Müller 2006; Gibbs 1994; Müller 1998, fc.; Sweetser 2001;
Taub 2001; Wilcox 2004; Wilcox & Morford 2007)
most work on metonymy in co-speech gesture has focused on this kind of
metonymic motivation
internal metonymy (within metaphoric gestures)
a horizontal line
stands for a sentence
“a sentence is a string of words”
external metonymy (within metaphoric gestures)
space
extending between
fingers is
assigned meaning
(adjacency/contact)
here:
the word “Diana”
in a sentence
external metonymy / metonymy of place
“subcategory”
{video clip}
Conotiguity relations between body and objects
“there is”
pointing gesture
(“shifters”; Jakobson)
“the main verb”
CONTAINMENT (cup)
metaphor
[object manipulation &
external metonymy]
“there is what‘s called the main verb”
Contiguity relations holding between hands & object
“sub-category”
imaginary OBJECT seemingly
being held between two hands
[external metonymy]
comparably low position
in gesture space &
in relation to body
[metonymy of place]
metonymy first, metaphor second
(Mittelberg & Waugh fc.)
1. Metonymy (external):
source: hands
target: object
(physical)
(imaginary)
2. Metaphor:
source: OBJECT
target: IDEA
(imaginary)
(conceptual)
co-speech: “sub-category“ (technical)
 metonymic target = metaphoric source
(imaginary object)
=> PLACE FOR FUNCTION
“sub-category”
some conclusions
The assumption that some metaphors are grounded in metonymy
seems to hold in the gestures discussed.
(Barcelona 2000; Dirven 2002; Fauconnier & Turner 2002;
Geeraerts 2002; Goossens 1995; Jakobson 1956, 1960;
Lakoff 1987; Lodge 1977; Radden 2000; inter alia)
 figures of thought may manifest themselves in gesture even if the concurrent speech is nonfigurative
(“noun”, “Diane”, “teach-”, “infinitive”, “sentence”, etc.)
(cf. Cienki 1998, 2008; Cienki & Müller 2008,fc.; Müller 2008)
Gesture and cognitive linguistics
Alan Cienki
• Given:
– the goal in cognitive linguistics to characterize language in a
way that is coherent with what is known about cognitive
processing in general
– the cog.ling. focus on actual linguistic behavior
– the universality of gesture with speech
 Gesture can provide additional insight into
conceptualizations underlying semantic and
grammatical structures
Gesture in relation to the study of
several topics in cognitive linguistics
•
•
•
•
•
metaphor
cognitive models
mental spaces
metonymy and cognitive reference points
image schemas
Metaphor in words and
gestures (Cienki 1998, 2008a)
• Gestures can provide evidence of activation
(on some level) of an image being used to
characterize a given topic
• (a mapping from one domain to another, i.e.,
[conceptual] metaphor, on some level)
• The inclusion of gesture data provides a way
out of the criticism of circularity of conceptual
metaphor theory
• (that is: language provides evidence of conceptual
metaphors, and we find evidence of conceptual
metaphors in language)
Metaphor in words and
gestures (Cienki 1998, 2008a)
• Metaphor can appear in words without accompanying
gestures
• Metaphor can appear in gestures without
accompanying metaphorically used words
• The ‘same’ metaphor can appear in words and
gestures in an utterance
• Different metaphors can be expressed
simultaneously in words and gestures
Cognitive models
• A cognitive model as the way an
individual characterizes knowledge for a
given domain
(compare schema, script, frame, etc.)
• Cultural models as shared cognitive
models
• These models can be constituted by
metaphors to varying degrees
Russian model of chestnost’
Dlia menia chestnost’
kategoriia
For me
chestnost’
category.
eto
nekaia abso`liutnaia
is a kind of absolute
Kogda vot iest’
situatsiia,
When there’s this situation,
seichas
postupit’ chestno `tak.
then [you need] to act honestly like this.
{video clip}
Cognitive models and gesture
(Cienki 1999)
• Metaphors can highlight aspects of
cognitive/cultural models
• Gesture data can provide evidence of
the speaker’s online use of a cognitive
model
• This can be valuable for the study of
(aspects of) cognitive models which are
not expressed in words
Mental spaces
• Developed from work by Gilles Fauconnier (1985)
• Refers to “small conceptual ‘packets’
constructed as we think and talk”
• Linguistic forms
– reveal the speaker’s use of various mental spaces,
– and they provide cues to the listener for how to
construct the mental spaces used by the speaker
• Examples:
hypothetical constructions (if…, then…)
Mental spaces: gestural example
(Cienki 2008b)
It depends on the
student,
but it also depends
on the teacher.
{video clip}
Mental spaces, metaphor, and gesture
• IDEAS AS SPACES
(a kind of ontological metaphor, i.e.,
ABSTRACT AS CONCRETE)
Metonymy and reference points
• Langacker (1993) notes how we
regularly make use of one (known)
entity to invoke mental contact with
another (less well-known) entity:
– > cognitive reference point
• Metonymy as one means of doing this
Metonymy and reference points
(Cienki 2007)
• Appears in gesture in many ways:
– The hand enacts a function it would actually
perform (possibly with an object)
(e.g., you have to write fast!)
– Or the hand represents (embodies) part of an
object
(viz. Darstellungsweisen [modes of representation], Müller 1998)
– Or the hand points to a reference point
Metonymy in a referential gesture
– hand points
`That was dishonest,
’cause she didn’t…
• abstract target
grounded on
concrete referent
{video clip}
Image schemas
“An image schema is a recurring, dynamic
pattern of our perceptual interactions and
motor programs that gives coherence and
structure to our experience.” (Johnson 1987: xiv)
E.g.:
CONTAINER
PATH
CYCLE
PART-WHOLE
SURFACE
BALANCE
COUNTERFORCE
LINK
CONTACT
CENTER-PERIPHERY
If we accept that...
• image schemas constitute significant patterns
in our embodied experience
• most research on image schemas has been
based on linguistic data
• manual gesture is a behavior involving a
different kind of embodied experience than
speech
• What role might image schemas play in
our production, or comprehension, of
gestures?
Image schemas and gesture
(Cienki 2005)
• compared the categorization of two types of
gestures in terms of image schemas:
– Abstract Referential (metaphoric) gestures
– combined group of discourse-structuring and
performative gestures (= “Other” gestures)
• two conditions:
– gestures with no sound
– gestures with speech
Image schemas and gesture
(Cienki 2005)
• Can people reliably characterize the forms of
gestures using image schemas as
descriptors?
• Will this vary if they hear the accompanying
speech or not?
Image schemas and gesture:
findings (Cienki 2005)
• both groups of gestures were characterized with
reliable agreement using a set of image schemas as
descriptors
– (this was true with and without sound)
• Abstract Referential (metaphoric) gestures were
categorized with a slightly greater degree of
agreement than the group of “Other” gestures (in
both conditions)
Conclusions
• Cognitive linguistics claims to be a
usage-based theory of language;
gesture with speech is part of spoken
language usage
• Taking this into account can lead to
multimodal analyses of grammar (e.g.,
Fricke 2008)
Conclusions
• The study of gesture with speech
prioritizes research on the dynamic
processes of language and cognitive
processing, including mental simulation
• Can help move cognitive linguistics
beyond its focus on the individual
speaker and connect it to research on
interaction and intersubjectivity
Looking ahead
• Creativity in gesture (IM & AC)
• Open-class and closed-class systems in
co-speech gesture (IM)
• Spoken language semantics (AC)
• Creativity in gesture (IM & AC)
Creativity in gesture: forms
– use of more extended gesture space
– greater dynamicity
– coordinated with other conspicuous
embodied expressions
(e.g., body shifts, facial expression)
• Differences in scale between
non-creative and creative gestures
Creativity in gesture: functions
– graphic representation of something
concrete
– or graphic representation of an (abstract)
idea via a physical image
–> metaphoric gesturing
– or clearly performing a comment reflecting
the speaker’s attitude toward the topic
• Creative gestures appear to serve
discrete functions (versus greater
multifunctionality of less creative gestures)
Creativity in gesture:
contexts of production
• Impetus to express a complex idea
concisely
–> High communicative pressure
– e.g., pedagogic context, or shared problem
solving
• ongoing creative work
• the “object” being presented needs to be
presented
• Open-class and closed-class systems in
co-speech gesture (IM)
open-class & closed-class systems
in co-speech gesture
In language, there is a fundamental difference between
 content words (e.g., nouns, verbs, adjectives);
open-class system (Talmy 2000)
 function words (propositions, personal & demonstrative pronouns, conjunctions,
etc.) expressing spatial, temporal, logical and functional relations;
closed-class system
 How does the difference between substance and relations (structure) play
out in co-speech gesture?
open-class & closed-class systems
in co-speech gesture
two lines of inquiry:
A) Do gestures accompanying linguistic expressions of either kind differ in form
and function?
B) Is there a difference between gestures depicting contents and gestures
depicting relations - without considering the concurrent speech.
Since gestures are inherently indexical a focus will be on how they support or
function as “shifters” (Jakobson 1960), e.g. those linguistic forms that take on a
different meaning in a given moment of speech
• Spoken language semantics (AC)
Spoken language semantics
• Premises:
– Inherent bias in formal linguistic research towards
language as it is written
– Cognitive linguistic approach to semantics as
conceptualization
– Different processes of conceptualization during
speaking versus writing, listening versus reading
• Thesis: The semantics of spoken language is
different in form and content than that of
written language
The semantics of spoken language as different in
form and content than that of written language
• Spoken language functions differently than
written language in terms of
–
–
–
–
–
time frame
grammatical structures used
embodied aspects of use
involvement of imagery
dynamic processes of expressing meaning
• Proposal: This calls for a distinct approach to
analyzing the semantics of spoken language
Thank you!