Embodied Cognition and Thinking
with External representations
How the body shapes the mind
Kenny Skagerlund 2015-09-03
Boundaries of cognition?
Embodied Cognition
The body as part of the thinking process
Embodied Cognition
Experiment:
Think about the word ”future”, what that concept means to
you, and keep that word in your mind.
One finding from an experiment (Miles, Lind & Macrae, 2010):
Subjects leaned forward while thinking about the future.
Why?
Embodied Cognition

Experiment

Setup: Subjects were faced with a moral dilemma and
judgement about a crime. Some of the subjects were
asked to wash their hands before the experiment.
Finding: Subjects who had
washed their hands prior, gave
more lenient and humane
judgements
Embodied Cognition

Experiment

Setup: A traditional mental rotation task of increasing
difficulty.

Findings: Gesture with hands more prevalent on hard
problems.
Preliminary conclusion

The body seems to shape our…






…decision making process…
…problem-solving…
…knowledge representation…
…perception…
Etc.
Bodily processes are tied to cognitive processes


How, why and to what extent?’
Is there a separation between body and mind?
Embodied Cognition

What is it?





The body is part of our mind
Ago Ergo Cogito - "I act, therefore I think"
Response to disappointments of traditional CogSci
Embodied cognition is a research program
Embodied vs. Situated vs. Distributed
Embodied Cognition – A Prologue

Traditional Cognitivism

Dualistic thought – Mind and body as distinct (Descartes)





Body as physical entity – deterministic
Mind as ”soul” and immaterial – free will
Reason and representations – detached from physical world
Sensing and acting doesn’t require thinking – animals can do it
Therefore, the mind is unique and detached from world
Embodied Cognition – A prologue

Traditional cognitivism


From Descartes to Newell & Simon
Cognition = Formal manipulation of symbols according to rules





The physical symbol system hypothesis
Symbols are representations (propositions)
The form of the symbols are what matters, not the semantics
Neurons can act as symbols
Good Old Fashioned Artificial Intelligence (GOFAI)



CYC
Turing Machines
…
19
Embodied Cognition – A prologue

Traditional cognitivism – Some concerns:

Searle’s Chinese Room


Syntax is not semantics
How do representations get their meaning?
Embodied Cognition – A prologue

Traditional cognitivism – Some concerns:

Searle’s Chinese Room



Evolutionary concerns




Syntax is not semantics
How do representations get their meaning?
The heritage from lower animals: bottom-up rather than top-down
(Brooks)
Rules and explicit representations are expensive
Evolutionary theories controversial at the time
20
The brain is not like a computer


Parallel rather than sequential processing
Interactivity with environment rather than internalization

”The world is its own best model”
Embodied Cognition – A prologue

Moving from cognitivism to embodied cognition

Classical computational view of vision





Algorithmic
Inferential
Impoverished data
…
Is there an alternative?
Embodied Cognition – A prologue

Moving from cognitivism to embodied cognition

Gibson’s Ecological theory of perception



We move around to see ”invariant” features of the environment
Action provides information extended in time
We don’t have to infer from impoverished data (traditional view)


The body is part of the visual process



Complex representations and algorithms not necessary
Action is input
Input vs. Output?
Information is ”hunted” actively
Embodied Cognition – A prologue

Moving from Cognitivism to Embodied cognition

Connectionism

We dont need formal symbols for computation – nor rules/inferences




Like Gibson’s theory
Processes carried out in parallel rather than serially
Biologically plausible – neurons
We don’t necessarily need representations
for every aspect of cognition
Embodied Cognition – a prologue

Gibson’s theory and Connectionism

Paved the way for embodied cognition

Actions and cognitive activity are central to cognition

Representations are not necessary


Inferences not necessary


Patterns of activity instead (connectionism)
Active exploration of environment instead (Gibson)
The brain can be seen as a controller for activity to extract
information
Embodied Cognition

Some characteristics (Finally!)

Replacement


Constitution


The body and world plays a constitutive role in cognitive processing
Conceptualization


Bodily interactions with the environment replaces the need for complex
algorithmic processes and representations
Our bodies shape the conceptualizations about the world. Our concepts
depend on the properties of our bodies.
Note: Some theorists emphasize these characteristics to a different
degree
Embodied Cognition

Replacement

Formal rules, symbols and algorithms



Explicit planning and internalization
Expensive and effortful
Evolution has shaped our bodies



Simple processes – energy efficient
Use external resources (gravity, friction etc)
Goes for cognitive processes as well
Embodied Cognition - Replacement

Example: ”The Gaze Heuristic”

Traditional Cognitivism and AI:

1 Frame problem and compute variables



Distance,Height, velocity, angle, friction etc.
Make prediction
2 Form a plan

Plan a motor sequence to intercept

3 Execute motor action

4 Catch ball
Embodied Cognition - Replacement

Example: ”The Gaze Heuristic”
Embodied Cognition - Replacement

Example: ”The Gaze Heuristic”
• Visually fixate the ball and approach – adjust running speed to keep ball at fixed
and constant point.
• Internalizing the world and computing the variables is rendered superfluous
Embodied Cognition

Constitution

The body and external resources in the environment are part
of the cognitive processes



Epistemic actions
Counting on fingers
Computations performed in the world or
through bodily activity
Embodied Cognition - Constitution

Example: (Chu & Kita, 2011)

Experiment 1:


Participants gestured more on harder problems
Experiment 2:

Encouraging gesture during problem solving
enhanced performance

Experiment 3:


Gesturing enhanced performance of subsequent
experiment – even when gesturing was prohibited
Conclusion:

Spatial computations become internalized, then gesture frequency
decrease.
Embodied Cognition - Conceptualization

Recall Searle’s Chinese Room



Syntax and formal rules not enough for semantics and
understanding
”The Grounding Problem”

(A.k.a ”Physical grounding hypothesis”/”Symbol grounding problem)

If Cognition is a form of computation and transformation of
symbols and representations according to rules, independent of
meaning, how is meaning acquired?
One of Embodied Cognition’s defining projects
(Anderson, 2003)
Embodied Cognition - Conceptualization

Solution:



Representations and symbols are grounded in bodily experience
Symbols get meaning – concepts
One account (Lakoff & Johnson)


21
”The peculiar nature of our bodies shapes our very possibilities for
conceptualization and categorization” –(Lakoff & Johnson, 1999)
Meaning through metaphor – and metaphor is grounded in
embodiment and its properties



Even the most abstract words get meaning from embodied properties
i.e. ”Future is Forward”, ”Relationship is a Journey”,
i.e ”up” and ”down” get meaning from our spatial bodily property
Embodied Cognition - Conceptualization
Higher concept
Lower concept
Embodied property
”Future”
”Forward”
”Front”
”Approach”
”Backward”
”Back”
• All concepts ultimately get ”grounded” in ”fundamental” embodied
properties
• Embodiment enables abstract representation
• AI and Robotics
• Embodiment as a prerequisite for intelligence!
”Withdraw”
Embodied Cognition

Some defining characteristics (Recap)

Replacement


Constitution


Bodily interactions with the environment replaces the need for complex
algorithmic processes and representations
The body and world plays a constitutive role in cognitive processing
Conceptualization


Our bodies shape the conceptualizations about the world. Our
concepts depend on the properties of our bodies.
Unique theme for Embodied Cognition
Embodied Cognition
Experiment:
Think about the word ”future”, what that concept means to
you, and keep that word in your mind.
One finding from an experiment (Miles, Lind & Macrae, 2010):
Subjects leaned forward while thinking about the future.
Why?
One answer:The concept of ”front” and ”approach”
mediated the behavior (Conceptualization)
Embodied Cognition

Experiment

Setup: A traditional mental rotation task of increasing
difficulty.

Findings: Gesture with hands more prevalent on hard
problems.

Embodied actions are part of
cognitive computation (constitution)
Embodied Cognition

Embodied vs. Situated vs. Distributed



Embodied Cognition – Focus on the body and how the body
shapes the mind (e.g. Grounding problem)
Situated Cognition – Focus on the individual and cognitive
activity with artifacts
Distributed Cognition – Focus on several individuals and
artifacts as a socio-cultural cognitive system
Embodied Cognition
Low level of
abstraction
Situated Cognition
23
Distributed Cognition
High level of
abstraction
Embodied Cognition – conclusions

The body is part of our mind

Involved in cognitive processes – constitution


The body shapes the mind


Problem solving, decision making, perception etc.
Determines, to some extent, what we can represent and think
about – conceptualization
Implications for CogSci:


AI and intelligence
Embodiment and evolution
24
”Cognition”
Embodied Cognition – conclusions
”The body and brain divide the labor of cognition between them,
sharing in processes that neither of them could do on its own”
”Cognition”
Thinking with external
representations
Thinking with things

Prove: all three medians of a triangle always intersect at a
single point.
Why make a diagram like this?
Thinking with things

Prove: all three medians of a triangle always intersect at a
single point.
Why make a diagram like this?
Because it’s too hard to do it in your head!
Interactive strategy

Project  Create  Project structure
Create figure
Project 1st
construction
Project 2nd
construction if you
can, otherwise
create again
Projection, perception and imagination

Perception



Imagination



Seeing what is there.
Purely dependent on external stimulus
Internal representation
Pure fabrication of what is NOT
Projection



Seeing what could be the case. Augmenting what is there.
Depends on both external structure and imagination
External structure as ”anchor”
Project  Create  Project
• Creating external structure
• Mental (internal) structure is offloaded
• Interactive strategy to increase mental power
•
Chess masters vs. Novices
• Occurs all the time!
• Annotations
• Gestures
• …
• Epistemic function (but may be pragmatic too)
Project  Create  Project

When problems are simple



No use of externalising
Can actually ”cost more” to create and adapt to anchors
When problems are complex


Lean on external structure
Amplifies cognitive ability
Project  Create  Project (examples)
• Thumb as a symbolic marker
• Projects his location on map…
• …finds where he is…
• …externalises that information to free up resources
External vs. Internal representations

Everyone benefits from external representations

External representations have properties that internal
doesn’t



Materiality – constraints and affordances – primes, visual hints and
insights – creating serially and evaluate as you go, mental structures
decay
Memory and perception are not the same thing
Internal representations are not stable
External vs. Internal representations
• External representations provide affordances that internal lack
External vs. Internal representations

Scrabble
Rearrange external structure
External vs. Internal representations
• Interaction Designer
•
•
•
•
•
Project  create  project
Generate new ideas
Generate deeper ideas
Share ideas
…
Cognitive coupling

Agent and artifact form a tight cognitive coupling

Interaction and transformation between internal and external
representations

Cognitive processes
migrate to wherever
costs are lowest
Cost of coupling/anchoring
must be factored in.
Tighter coupling – lower cost
Registering an artifact help us form tight cognitive coupling
Some conclusions

Project-create-project cycle





Cognitive coupling



Part of our everyday activity and problem solving
Offload working memory resources
External representations have other affordances than our internal
representations
Deeper and wider cognitive activity
Cognitive activity ”flow” wherever costs are lower
Good artifacts ”afford” cheap and fast interaction and cognitive
process
External representations help us think the unthinkable!