The Brain and Multimodal
Communication
Elisabeth Ahlsén
University of Gothenburg
SSKKII Interdisciplinary Center & Department
of Linguistics
Nationella Forskarskolan i Kognitionsvetenskap ht 2008
Contents
1. Communication on different levels of
awareness and control corresponding brain functions
2. What models cover - view of cognition
and communication
3. Multimodal communication - some
controversies
4. Examples of current research trends
5. Brain damage - cognition and
communication
1. Communication on different levels
of awareness and control corresponding brain functions
Communicative Body Movements:
1)Facial gestures (nose, eyebrows, cheek, forehead, chin etc)
2)Head movements
3)Gaze direction, mutual gaze
4)Lip movement
5)Hand and arm movements
6)Leg and foot movements
7)Body posture
8)Distance btw communicators
9) Spatial orientation
10) Touch
11) Non-linguistic sounds
Homunculus
Types of representation
3 ways of carrying, conveying and
sharing information
• Index
• Icon
• Symbol
Variation in awareness and intentionality
Bodily communication often more
spontaneous, unaware, unintended (index,
icon) than words (symbols).
Three levels of intentionality:
indicate
display
signal
index
x
icon symbol
x
x
indicate -> display -> signal
conventional signal -> automatic
Dimensions of content in
communication
Words
Factual information
Emotions and
Attitudes
Physiological states
Identity
Social relations
Communication Management
Interactive communication management
Own communication management
Body
Interactive view on embodied
communication
Two communicators form one dynamic system by
establishing stabilizing communication links
Feedback
employs all
of these
levels, too.
Feedback
Max
Feedback + Max
Outlook
Communication - Cognition
Memory (WM, LTM)
Attention (sustained attention)
Central executive
…
Right hemisphere lesions
•
•
•
•
•
Left neglect, spatial disorder
Prosody disorders
Lexical-semantic disorders
Problems with emotion information
Discourse disorder (complexity, ToM,
metaphor, humor, irony, etc)
Primitive brain - Limbic system Neocortex
James-Lange vs Cannon-Bard
Formation of emotions
Amygdala
Mediation and control of major affective activities like friendship, love and
affection, on the expression of mood and, mainly, on fear, rage and aggression
The center for identification of danger. Fundamental for self preservation.
Humans with marked lesions of the amygdala, loose the affective meaning of
the perception of an outside information, like the sight of a well known person.
The subject knows, exactly, who the person is, but is not capable to decide
whether he likes or dislikes him (or her).
Hippocampus
Formation of long-term memory.
When both hippocampi (right and left) are destroyed, nothing can
be retained in the memory.
The subject quickly forgets any recently received message.
The intact hippocampus allows the animal to compare the conditions
of a present threat with similar past experiences, thus enabling it to
choose the best option.
Fornix & parahippocampal
gyrus
Connecting pathways
Thalamus
Lesion or stimulation of the medial dorsal and anterior nuclei of the thalamus are
associated with changes in emotional reactivity.
The medial dorsal nucleus makes connections with cortical zones of the pre-frontal
area and with the hypothalamus. The anterior nuclei connect with the mamillary
bodies, and through them, via fornix, with the hippocampus and the cingulate
gyrus.
Hypothalamus
Lesions of the hypothalamic nuclei interfere with several vegetative
functions and some of the so-called motivated behaviors.
Specifically, its lateral parts seem to be involved with pleasure and rage,
while the median part is like to be involved with aversion, displeasure and a
tendency to uncontrollable and loud laughing. However, in general terms,
the hypothalamus has more to do with the expression (symptomatic
manifestations) of emotions
Cingulate gyrus
(gyrus cinguli)
Its frontal part coordinates smells and sights with pleasant memories of previous
emotions. This region also participates in the emotional reaction to pain and in
the regulation of aggressive behaviour
Brain stem
Responsible for the "emotional reactions", (indeed, they are just reflex answers)
of inferior vertebrates
Remains active, not only as alerting mechanisms, vital for survival, but in the
maintenance of the sleep-awake cycle.
Septum
Pleasant sensations, mainly those related to sexual experiences
Anterior to the thalamus
Prefrontal area
Its intense bi-directional connections with thalamus, amygdala and other
subcortical structures, account for the important role it plays in the genesis and,
specially, in the expression of affective states.
When the pre-frontal cortex suffers a lesion, the subject looses his sense of social
responsibility as well as the capacity for concentration and abstraction. In some
cases, although consciousness and some cognitive functions, like speech, remain
intact, the subject can no longer solve problems, even the most elementary ones.
2. What models cover - view of
cognition and communication
A. Localist-associationist ”classical” model
Language - cortical centers + subcortical white
fibres
-Broca’s area - production
-Wernicke’s area - perception-comprehension
-Connections
-Concept center
Geschwind
B. Dynamic localization of function
A. R. Luria
Functional systems of many subfunctions in
different areas, not always language
specific, (e.g. rapid alternations, inertia)
Dynamic systems, can be restored-changed
3 blocks: brain stem, posterior PTO, anterior
C. Hierarchical / evolution based models
(”holistic”)
J.H. Jackson- J. Brown: Microgenesis
”from deep to surface” in phylogenesis,
ontogenesis and microgenesis”
Focus on role of subcortical, older structures,
emotion etc.
Automatic vs. Controlled behavioron
Levels of function for language, emotion,
perception
3. Multimodal communication
- brain and cognition
- some controversies
• Localism - Associationism - Holism
• Modularity
- Interaction
• Symbol manipulation
- Distributed activation
/representation
• Linear models
- Interactive Activation
Models (IAM,ANN)
• Language/speech and gesture intertwined or
separate/compementary - related how?
• Definition/delineation of language / communication
system
• Different types of body movements - how are they related
and how do they relate to spoken language?
4. Examples of current research trends
- Embodiment - many aspects, gesture one of
them, role of body for development of
language another
- ”Growth point” (McNeill)
- Cultural influence vs universality of gesture +
relation to speech
- Building embodied artificial agents
- Studying coordination-communication on
lower levels of awareness and control
- Neuroimaging?
- Action-Perception link - localization
5. Brain damage & multimodal
communication
From language/words sentences to
communication - multimodal
Widening of the concept of aphasia (cf.
language) - controversy
From left hemisphere cortex to any languagecommunication disorder, including, RH,
subcortical-cerebellar, distributed, e.g.
Alzheimer etc., TBI
New approaches needed and substantial
research
New models needed
Next episode…
On
evolution,
mirror neurons,
the role of imitation and pantomime
and possible revisions of theories about
communication disorders…
Assignment
Choose one of the controversies
mentioned above and write one page
argumenting for each of the two sides or
suggesting two different views and
approaches to answering one of the
questions.
Literature:
Arbib M (2005). From monkey-like action
recognition to human language: an
evolutionary framework for neurolinguistics.
Brain and Behavioral Sciences, 28, 105–24.
Ahlsén, E. 2008. Neurological disorders of
embodied communication. In.I. Wachsmuth,
G. Knoblich & M. Lenzen (eds.) Embodied
Communication in Humans and Machines.
Oxford: Oxford University. Press.