Attention and emotion:
From data to conceptual issues
Luiz Pessoa
Department of Psychology
University of Maryland, College Park
Background
•
1990s: work showing limitations of visual
processing and the need for attention
Change blindness
 Attentional blink

Background
•
Processing of emotion-laden information is
prioritized

Independent of awareness
L
Whalen et al. (1998)
R
Morris et al. (1998)
“Automatic” Processing
Amygdala
LeDoux
Research goal
•
Understand the role of attention and
awareness during the processing of
emotional visual items

Employ strong attentional manipulations

Evaluate awareness with Signal Detection Theory
Role of spatial attention
•
Is activity evoked by emotional faces
automatic?
OR
•
Does activity evoked by emotional faces
require attention?
Spatial attention
Unattended
Faces
Attended Faces
Not drawn to scale
200 ms
Male/female
Easy: 91% correct
200 ms
Same/different
Difficult: 64% correct
Attention is required for the
expression of valence (N = 21)
L
R
Response Amplitude
Right Amygdala
Fear ATT
Neutral ATT
Happy ATT
Fear UNATT
Neutral UNATT
Happy UNATT
0.25
0.2
0.15
0.1
0.05
0
X
-0.05
-0.1
0
2
4
6
8
10
12
Seconds
•
Strong valence X attention interaction: Effect
of valence depends on attention
Pessoa et al. (2002): PNAS
Emotional perception requires attention
•
Attention parametrically manipulated within the
same task
•
Affective significance enhanced via conditioning
Easy
Hard
Task: find X
Pessoa et al. (2005): Neuroimage
Hsu and Pessoa (2007): Neuropsychologia
Lim et al. (2008): Neuropsychologia
Role of visual awareness
Target
Mask
Yes/No
Confidence? 1-3 scale
Pessoa et al. (2005): Emotion
Visual awareness: Signal Detection
67 ms
33 ms
Amygdala responses
VS.
67 ms
33 ms
AWARE
UNAWARE
L
R
Y = -4
L
R
Y = -4
Pessoa et al. (2006): Cerebral Cortex
Behavioral results: Individual differences
•
Many participants can detect fearful faces
even at 17 ms
17 ms
Szczepanowski and Pessoa et al. (2007): Journal of Vision
Fear stimulus > Neutral stimulus
Amygdala
67 ms
33 ms
AWARE
UNAWARE
“Normals”
N = 19
L
R
L
Y = -4
x
R
Y = -4
AWARE
AWARE
“Detecters”
N=8
L
R
L
Y = -6
.
Pessoa et al. (2006): Cerebral Cortex
R
Y = -6
Role of temporal attention/awareness
CS+ vs. CS–
T1
T2
100 ms
100 ms
2s
Lim, Padmala, and Pessoa (2009): PNAS
Attentional blink: Behavior (N = 30)
Enhanced perception of CS+: Reduced blink
80
T1
T2
Building or
House?
T2 accuray (%)
•
75
70
65
60
CS+
CS–
55
50
CS+
CS-
scene category
Role of attention/awareness
T2
Parahippocampal
gyrus
Miss trials
T2
Parahippocampal
gyrus
Miss trials:
no differences
observed between
CS+ and CS- trials
% signal change
•
% signal change
Role of attention/awareness
0.20
Visual ctx
0.15
CS+
CS-
0.10
0.05
0.00
-0.05
0
2
4
6
8 10 12 14 16
0.10
0.08
0.05
0.03
0.00
-0.03
-0.05
-0.08
Amygdala
0
2
4
6
8 10 12 14 16
Time
Conceptual issues
Impasse
• While a great deal has been learned about the extent
and limits of affective visual processing, two camps
have opposing and entrenched views
Impasse
• While a great deal has been learned about the extent
and limits of affective visual processing, two camps
have opposing and entrenched views
Capacity-limited
Capacity-Unlimited
Impasse
• While a great deal has been learned about the extent
and limits of affective visual processing, two camps
have opposing and entrenched views
Capacity-limited
Capacity-Unlimited
Not too surprising…
•
Emotional stimuli are sufficiently potent that
they exhibit a host of properties that do not
appear to occur with neutral items

•
They are processed when unattended
Affective processing is subject to capacity
limitations, as revealed by several
experimental manipulations

Attentional blink
Impasse: will it go away?
•
Advocates of limited processing can claim
that processing resources have not been
consumed
 “If
the manipulation were stronger, the impact
of affective items would go away…”
Impasse
•
Showing that the emotional effect has
disappeared is always subject to the
“null problem”

Arguing for the absence of an effect
Power vs. strength of manipulation
Response Amplitude
Left Amygdala
Right Amygdala
Fear att
Fear unatt
Happy att
Happy unatt
Neutral att
Neutral unatt
0.25
0.2
0.15
0.1
0.05
0
-0.05
-0.1
0
2
4
6
8
Seconds
Pessoa et al. (2002): PNAS
10
12
0
2
X
4
6
8
Seconds
10
12
Processing resources
Given the limited capacity of mental
processes, performance is impaired if
demands are greater than available resources
performance
•
Easy/efficient
Hard/inefficient
processing resources
Norman and Bobrow (1975)
performance
Processing resources
Nakayama and
Joseph (1998)
performance
processing resources
“automatic”
processing resources
performance
Processing resources
Nakayama and
Joseph (1998)
performance
processing resources
Capacity
limitation
processing resources
Dual-tasks
Processing resources
•
Moors and De Houwer (2006):
Every process is uncontrolled, efficient,
unconcious, and fast
Processing resources
•
Moors and De Houwer (2006):
Every process is uncontrolled, efficient,
unconcious, and fast, to some degree…
•
•
Relative to what?
Affective processing: relative to neutral not
enough
Fine comparisons needed (e.g., abrupt onsets,
search, etc.)
 Broad set of comparison tasks

Two camps
Capacity-limited
Capacity-Unlimited
Preattentive-attentive model
t1
t2
Stage 2:
Attentive
boundary
Stage 1:
Preattentive
Preattentive-attentive model
•
Some features are processed pre-attentively
in virtue of the fact that they are optimally
matched to properties of the early visual
system (e.g., orientation)
•
Affective processing: Sub-cortical pathway

Superior colliculus  pulvinar  amygdala
Dynamic model
Multiple
interactive
“stages”
Dynamic model
•
Processing is not pre-attentive or attentive,
but a gradient of processing efficiency is
hypothesized to exist
•
Gradient based on the properties of early
visual areas
•
But critically, gradient is dynamically
configured based on task demands
•
Configuring is suggested to depend on
parietal and frontal cortex
Dynamic model
•
•
More susceptible to
capacity limitations
Multiple “gates”
“bottlenecks”
Variable permeability
Less susceptible to
capacity limitations
Dynamic model
•
Hierarchical and “short-cut” connections
Multiple waves
•
Initial processing of visual
information proceeds
simultaneously along parallel
channels
•
“Multiple waves” of activation
across visual cortex and beyond
•
The multiple waves are engaged
dynamically based on task
requirements
Subcortical processing
Subcortical processing
“passive”
“integrative”
Pessoa and Adolphs, Nat. Rev. Neurosci (2010)
Subcortical processing
Pessoa and Adolphs, Nat. Rev. Neurosci (2010)
Processing architecture and attention
Task 2
Task 1
Collaborators
Ralph Adolphs
 Jan Engelmann
 Shruti Japee
 Shen-Mou Hsu
 Seung-Lark Lim
 Srikanth Padmala
 Remik Szczepanowski
 Leslie Ungerleider

National Institute of
Mental Health
emotioncognition.org