University of Sao Paulo
Workshop: Gesture Theory
Applications to Second Language
Development
Aug. 9, 2012
Steve McCafferty, University of Nevada, Las Vegas, Dept. of
Educational Psychology and Higher Education
Clark, P. B. (1986). Mandelbrot, Comment j'ai decouvert les fractales,
La Recherche, 420-424. http://www.gapsystem.org/~history/Biographies/Mandelbrot.html. Retrieved 3/10/
2010
About the Discovery of Fractals by Mandelbrot:
“What he saw was that the overwhelming
smoothness paradigm with which
mathematical physics had attempted to
describe Nature was radically flawed and
incomplete. Fractals and pre-fractals once
noticed were everywhere.”
Gesture and the origins of language
• Gesture first: Some researchers suggest that gesture
was how language began, that is, before speech there
was gesture (after bipedalism), language developing
later
• Gesture and speech co-development: Other
researchers (notably McNeill, 2012) argue that
gesture developed only in modern humans together
with language, about 150-200 thousand years ago,
after the expansion of the pre-frontal lobes was
complete (Deacon, 1997) (Growth Point Hypothesis)
Gesture and the origins of language
• In any case, Streeck (2010), Streeck and
LeBaron (2000) and Müller (1998) have
proposed that gestures derive from practical,
world-directed actions. However, as McNeill
(2012) points out, gestures can be
multidimensional with regard to meaningmaking, encompassing abstract as well as
concrete information at the same time
Embodiment: Philosophy and Neurology
• Spinoza: The mind body connection (opposite
of de Carte)
• Vygotsky: no separation of cognition/affect or
social/psychological in development
• The World on Paper (2004) by David Olsen
(literacy)
• Luria, Sacks, Ramachandran, Damasio
(neurology)
Embodiment: Philosophy and Neurology
• Mirror neurons (when we watch others move our
mirror neurons fire in the brain in the same way as
the person doing the activity) (see Arbib, 2012)
• McNeill: The Growth Point Hypothesis (Brodmann’s
area 44 and Broca’s area 45 in the brain are adjacent
• Goldman & Gallese (1998), on the basis of mirror
neurons, put forth their Simulation Theory – that we
“read” the minds of others if we have been in similar
states (Mead’s Loop suggests that people become
aware of their own actions [not others] this way)
Signs: Ecological perspectives
• Von Uexkull (1902) was the first to study signmediated relations with the environment and these
were non-linguistic (social/biological)
• Vygotsky (1930s) recognized the importance of
ecosocial, environmental contexts on development
• Gibson’s (1979) theory of affordances is grounded in
body and mind functioning as a whole for the
purposes of the organism
Signs: Ecological perspectives
• Thibault (2004) takes an ecosocial semiotic
view of language in contexts
• All of these researchers note that the
relationship to the environment is one in which
the organisms that live there both are shaped
by it as well as shape it in a dynamic, on-going
process
Embodiment: Linguistics
• (1926) Saussure: lange vs. parole
• (1933) Bloomfield: AM structualism
• (1950s) Birdwhistell (kinesics), Hall
(proxemics), Ruesch and Kees (NVC)
• (1950s) Chomsky: Competence not
performance (Structural model undermined by
the generative approach [Scollon & Scollon,
2009, p. 175])
Embodiment: Linguistics
• Other modes of communication were not found to
function like language (Scollon & Scollon, 2009)
• “Language and linguistic analysis may be an
important source of concepts, but it is fatal to the
[multimodality] research endeavor to simply transport
linguistic analysis over into the analysis of other
modes” (Scollon & Scollon, 2009, p. 177)
• Harris (1998; 2003) at Oxford Univ. is one linguist
who does not isolate language from communicative
and situational contexts (Integrated Linguistics)
Gesture Studies: Language and Linguistics
• McNeill’s (1992; 2005; 2012) Growth Point Hypothesis:
Speaking is more than uttering speech sounds with meaning;
more deeply it is also “inhabiting” language (a term from
Merleau-Ponty 1962).
• The GP can be regarded as a hypothesis of how this
multilayered inhabitance takes place, using vocal, manual and
body movements, all orchestrated by imagery and
incorporating context, purpose and thought
• How speech and gesture organize is a dynamic process
depending on contexts
Gesture Studies: Language and Linguistics
• Language and gesture operate differently in the
making of meaning
• Language is a linear, hierarchical, sequential, abstract
contribution to thought and communication
• Gesture is a holistic, imagistic, concrete contribution
to thought and communication
• There is a dialectical synthesis of these two opposite
modes of semiotic packaging in relation to contexts in
order to make meaning spontaneously
Gesture Studies : Language and Linguistics
• Krauss, Chen, and Gottesman (2000) view
gesture as auxiliary to speech in their Lexical
Retrieval Hypothesis
• Kita’s (2000) Information Packaging
Hypothesis: Spatio-motoric thinking
“organizes information with action schemas”
and is typical of people’s “interaction with an
object, locomotion, and imitating somebody
else’s action” (p. 164)
Gesture Studies : Language and Linguistics
• De Ruiter (2007) sees gesture as driven by
communicative intent in a parallel but
unconnected coordination with speech (the
Sketch Model)
• Alibali and Heath’s (2003) Semantic
Information Hypothesis argues that The
visibility between speaker and listener should
influence speakers’ production of gestures that
convey semantic information
Gesture Studies : Language and Linguistics
• Kendon (2004) views gesture as primarily
communicative and as a function of language
• Goldin-Meadow (2008) in part, views gesture as
reducing the cognitive load for both thinking and
communicating. She also views gesture mismatches
as indicative of cognitive development in children.
• Roth (2002): highschool science students first
manipulated materials while conducting physics
experiments, then explained the experiment using
gestures, eventually being able to verbalize them
Vygotsky and development
• “Vygotsky denied the existence of human beings
without culture – after all, he considered culture and
culturally mediated thought/action to be the hallmark
of the emergence of human beings as a distinct
species” (Cole & Gajdamaschko, 2007, p. 201)
• Towards the end of his life Vygotsky moved from a
focus on word to a focus on meaning, taking into
account the larger realm of semiosis
Vygotsky and development
• However, culture vs. nature, culture vs. biology are
dualisms not accepted by Vygotsky, Mead, or James
• Regardless, at some point Western psychology
decided that the material world is the domain of
scientists and that psychologists deal with the
internal, non-material world (Costall, 2007, p. 119);
in some current iterations psychology is entirely
biological
Elaboration of Vygotsky and development
• Timescales intermix, so that a trajectory
though time-space involves encountering new
aspects of things experienced earlier that
become linked in dynamic and interdependent
ways (Lemke, 2000). True for second language
development as well.
Gesture Studies: Social, Cultural Signs
• “Ape gestural signals work in ways more
similar to human communication than do ape
vocal signals , particularly in terms of flexible
production and attention to the attentional
states of the partner” (Tomasello & Call, 2007,
p. 222)
• We mirror the gestures and other nonverbal
behaviors of others as an aspect of showing
social solidarity (Tannen, 1990)
Gesture Studies: Social, Cultural Signs
• “[Nonverbal] imitation/conformity/
solidarity/affiliation … has important
consequences for the evolution of human
cooperative communication” (Tomasello, 2008,
p. 152)
Gesture Studies: Social, Cultural Signs
• Cultural forms of nonverbal behavior
constitute “the primary public dimension that
defines our personal identity, and in it, style
and tradition matter, to the degree that these
things establish who we are, who our friends
are, and where we stand in society” (Donald,
2001, p. 240)
Gesture Studies: Social, Cultural Signs
• Cultural forms of nonverbal behavior
constitute “the primary public dimension that
defines our personal identity, and in it, style
and tradition matter, to the degree that these
things establish who we are, who our friends
are, and where we stand in society” (Donald,
2001, p. 240)
Gesture Studies: Social, Cultural Signs
• “Indeed, when a young child pretends to drink
from an empty cup and looks playfully to the
adult’s face, one could say that in addition to
pretense for the self this is also an iconic
gesture to share this representation with the
adult, communicatively” (Tomasello, 2008, p.
152)
An Ecosocial, Semiotic Approach to
Meaning-Making
• How people shape their environment
semiotically and how this in turn shapes
them as part of the complex, dynamic,
and nonlinear process “in which the
world and its meanings are actively
produced by us” (Thibault 2004, p. 184)
An Ecosocial, Semiotic Approach to
Meaning-Making
• In turn, this conceptualization suggests, as
Thibault (2004, p. 234) further argues, that
“[m]eaning is not … something which is
isolable in just one part of the system. Rather it
is a result of the relation between the agentobserver and the overall system of relations on
all levels.”
An Ecosocial, Semiotic Approach to
Meaning-Making
• As such, it is further argued by Thibault (2004,
p. 176), that “… meaning is stored, not at the
level of the individual per se, but at the level of
… contextual configurations … which
integrate individuals to their ecosocial
environment and therefore, to the systems of
interpretance that are embedded in these.”
An Ecosocial, Semiotic Approach to
Meaning-Making
• Newcomers are subjected to mastering the
“use of objects and symbols, pragmatically,
semiotically, and semantically, something that
can only be done by participating in sociocultural practice” (Rosa, 2007, p. 305)
• Newcomers construct a sense of self “as an
object among others, as an agent and as an
actor” (ibid, p. 308)
An Ecosocial, Semiotic Approach to
Meaning-Making
• Language is grounded upon forms of life, not
something that can be considered in isolation,
independent of the multiple functions it fulfils
in the lives of its users (Wittgenstein,
1953/1958)
• “Speaking is more than uttering speech sounds
with meaning; more deeply it is also
‘inhabiting’ language (a term from MerleauPonty, 1962 as found in McNeill, 2012)
An Ecosocial, Semiotic Approach to
Meaning-Making
• Figured worlds are socially and culturally constructed
or interpretive realms where people come together to
construct joint meanings and activities (Boaler &
Greeno, 2000)
• Performing is the act of children as they dress up to
pretend to be their parents (ZPD – the act of
becoming) (Newman and Holzman, 1993; Holzman,
2011). This is dramaturgical as well.
• Teachers prolept students beyond what is known in
the present to what they will know (van Lier, 2004)
An Ecosocial, Semiotic Approach to
Meaning-Making
• Bakhtin (1934/5, as found in Rosa, 2007, p. 307)
suggests how a word becomes populated with
sense and reaches a fully conventional stage:
“The word of language – is half alien. It becomes
‘one’s own’ when the speaker inhabits it with his
intention, his accent, masters the word, brings it to
bear upon his meaningful and expressive strivings.
Until that moment of appropriation … the word
exists on the lips of others, in alien contexts, in
service of other’s intentions.”
An Ecosocial, Semiotic Approach to
Meaning-Making
• “Consciousness became … a result of the
internalization of (social) communication with
semiotic materials (cultural), accumulated
along the (historical) past of the cultural
group” (Valsiner & Rosa, 2007, p. 8)
• As an aspect of inner speech, L2 learners may
also develop an L2 inner voice, which includes
gestures as well as gesturing (Shigematsu,
unpublished dissertation, Univ. of Nevada)
Actuation
• How organisms come to act on affordances in
the environment is the process of actuation
• Mediational Actuations of Communication:
Three stages: (Rosa, 2007, p. 303).
• 1) exposure to an aspect of communication
that
is the object to be understood
• 2) engaging in the act itself
• 3) conventional use of object in social settings
Actuation
• Newcomers are subjected to mastering the
“use of objects and symbols, pragmatically,
semiotically, and semantically, something that
can only be done by participating in sociocultural practice” (Rosa, 2007, p. 305)
• Newcomers construct a sense of self “as an
object among others, as an agent and as an
actor” (ibid, p. 308)
L2 Gestures, Gesturing (naturalistic
contexts)
• A comparative study on the gestures of immigrant
and assimilated Southern Italians (Sicilians and
Neapolitans) and Eastern European Jews (Polish and
Lithuanian) in New York City found that the gestures
produced by the assimilated groups differed from
those produced by the immigrant groups. The
gestures of the assimilated groups had become both
similar to one another and generally more American
(Effron, 1941/1972)
L2 Gestures, Gesturing (naturalistic
contexts)
• L1 Anglophone children deploy L2 Francophone
emblematic gestures when speaking to age-matched
L1 French speakers (Von Raffler-Engler, 1976)
• Mohen &Helmer (1988): L2 children acquire L2
emblematic gestures at very young ages
• Jungheim (2006) studied the actuation of emblems in
naturalistic contexts
• McCafferty (1998; 2008), McCafferty & Ahmed
(2000): acquisition of L2 gestures and gesturing
L2 Gestures, Gesturing (naturalistic
contexts)
• Nicoladis (2007) suggests that with bilinguals
the weaker language includes the use of more
deictic gestures than with the stronger
language, although she offers no real
explanation for this
L2 Gestures, Gesturing (classroom
contexts)
• Jungheim (1991) found that comprehension of
selected emblems was increased if students were
explicitly taught through watching the teacher
perform the gestures in the L2 classroom (adult
Japanese L2 learners of English in Japan)
•Allen (1995), although researching lexical retention,
found that when coupled with emblems (teacher
gesturing), students remembered the items tested
significantly better than without such coupling
L2 Gestures, Gesturing (classroom
contexts)
• Tellier (2008) found that when students
enacted gestures as portrayed by the teacher
along with new vocabulary they retained the
vocabulary better than when they did not
L2 Gestures, Gesturing (classroom
contexts)
• Haught and McCafferty (2008) focused on L2
university students in the U.S., finding that
taking a direct dramaturgical approach –
verbally rehearsing short scenes of a highly
cultural nature – led students to imitate
gestures first inadvertently produced by the
instructor, a former actor, and then to use them
on their own (the scenes were practiced
recursively). Only his gestures were copied
L2 Gestures, Gesturing (mixed contexts)
• Nardotto Peltier & McCafferty (2010) found
that university instructors of Italian in the U.S.
who had spent time in Italy engaged in an
almost constant use of Italian emblematic
gestures as well as Italian ways of gesturing
(growth point acquisition as suggested by
McNeill, personal communication, October,
2011)
L2 Gestures, Gesturing (mixed contexts)
• In follow-up interviews, all of the instructors
noted that communicating in Italian goes
beyond just speaking, emphasizing the
embodied nature of this practice. Students in
the study were found to mirror their
instructors’ Italian gestures and to gesture in
an Italian fashion. Also, instructors had not
been video recorded previously and all were
surprised at the extent of their Italian gestures.
L2 Gestures, Gesturing (classroom
contexts)
• Zhao (2007) found that instructors in university ESL
classes in the U.S. used a number of verbal metaphors
and metaphoric gestures as a pedagogical element in
the teaching of writing for academic purposes
• Students used these same gestures when interacting
with each other, for example, drawing a virtual
vertical line in space from top to bottom to illustrate a
sequence of connected ideas as formulated in a
logical, hierarchical, linear order (internalization)
L2 gesture and motion events
• A number of researchers have investigated the
possible acquisition of motion events from one
language typology to another in which gesture
plays a significant role for verb frame as
opposed to satellite frame languages, for
example Spanish and English. Results to date
are inconclusive.
Foreigner talk and gesture
• It appears that some native speakers, when
they feel that the L2 interlocutor is of
beginning or intermediate proficiency in the
L2 will resort to concretizing their gestures to
provide explicit reference (sense) for the
language learner (McCafferty & Rosborough,
2007; Stam & Tellier, forthcoming)
Generalizations for L2 gesture
• Overall, it appears that there is greater
frequency of gesturing for speakers in the L2
than in the L1, but that this diminishes with
increasing levels of L2 proficiency
Generalizations for L2 gesture
• There has been a concern that L2 gestures
operate as “compensation” for L2 linguistic
deficiency. However, this may not be a valid
concern in the sense that people gesture as an
aspect of meaning-making, a process which
may require greater work in the gestural
channel in an L2 (see McCafferty 2008, a &
b), but at the same time, is a feature of L1
gesture too, as dependent on contexts.
Generalizations for L2 gesture
• Compensation does not consider the use of
gestures of identity in the case of naturalistic
learners as well, that is, the effort to
accommodate the L2 surrounds both intrapersonally and inter-personally