J Nonverbal Behav (2007) 31:137–152
DOI 10.1007/s10919-007-0026-6
Exploring the Role of Working Memory in Dynamic
Social Cue Decoding Using Dual Task Methodology
Louise H. Phillips Æ Mary Tunstall Æ Shelley Channon
Published online: 10 March 2007
Ó Springer Science+Business Media, LLC 2007
Abstract Decoding nonverbal social cues involves skills such as understanding
emotions and discerning kin relationships. Social cue decoding has been described as
an automatic process which does not tax cognitive resources. However, clinical
deficits in social decoding are often interpreted in terms of deficits in attention and
working memory. Two experiments are described which used dual task methodology
to investigate the role of working memory in dynamic social cue decoding tasks: the
Interpersonal Perception Task (IPT, Experiment 1) and the Profile of Nonverbal
Sensitivity (PONS, Experiment 2). Results revealed that decoding social cues from
the IPT did not require working memory resources, while social perception in the
PONS made heavy demands on working memory. Implications for theoretical
interpretation of these tasks and their use in clinical populations are discussed.
Keywords Attention Æ Nonverbal Æ Social cue decoding Æ Working memory
Social cue decoding comprises abilities such as perceiving facial and auditory
expressions of emotion, understanding what other people think or feel from verbal
and nonverbal cues, deciding when someone is lying or being sarcastic, or judging
the nature of the relationship between two people from their interactions. These
skills have been extensively studied in clinical conditions such as schizophrenia (e.g.,
Mary Tunstall is now at the Department of Psychology, University of Durham.This research was
funded by the UK Economic and Social Research Council.
L. H. Phillips (&) Æ M. Tunstall
School of Psychology, College of Life Sciences and Medicine, University of Aberdeen,
Aberdeen, Scotland AB24 2UB, UK
e-mail: louise.phillips@abdn.ac.uk
S. Channon
Department of Psychology, University College London, London, UK
M. Tunstall
Department of Psychology, University of Durham, Durham, UK
123
138
J Nonverbal Behav (2007) 31:137–152
Calkins, Gur, Ragland, & Gur, 2003; Corrigan & Addis, 1995; Quintana, Wong,
Ortiz-Portillo, Marder, & Mazziotta, 2003), and patients with focal brain lesions
(e.g., Mah, Arnold, & Grafman, 2004; Stone, Baron-Cohen, & Knight, 1998; Stuss,
Gallup, & Alexander, 2001). Impaired ability to decode social cues is likely to result
in impaired social functioning, disturbed behavior, and poor quality of life (e.g.,
Bozikas Kosmidis, Anezoulaki, Giannakou, & Karavatos, 2004; Carton, Kessler, &
Pape, 1999; Mah et al., 2004).
There are a number of standardized tests which investigate the ability to decode
social cues. While many studies have used static posed photographs of facial
expression of emotion, such measures do not provide a naturalistic assessment of the
ability to decode social cues in real settings. In more realistic situations expressions
are likely to be subtle, dynamic social cues are available, the body as well as the face
can be viewed, and social cues are usually displayed in interaction with other people.
This has resulted in the development of more naturalistic tasks which aim to assess
social cue decoding involving dynamic stimuli, such as the Interpersonal Perception
Test (IPT) and the Profile of Nonverbal Sensitivity (PONS), which will be the focus
of the current study. The IPT and PONS require social perception that goes beyond
tests of emotion recognition, requiring also that a perceiver can assess interpersonal
cues such as intimacy or status (Sergi & Green, 2002).
There has been some debate as to whether social cue decoding is dependent on
cognitive resources such as attention (i.e., conscious and resource-demanding
monitoring of the environment) and working memory (i.e., the simultaneous and
effortful weighing of competing information placing a demand on both processing
and memory resources). Three possibilities can be contrasted: (i) social cue decoding
is an automatic process which can be carried out without conscious attentional
resources, (ii) social decoding requires conscious attention to proceed effectively,
but does not heavily load working memory resources, and (iii) social decoding
requires not only conscious attention, but also working memory resources in order to
simultaneously process complex stimuli and weigh up competing interpretations of
social cues. Social decoding comprises a wide spectrum of processes – e.g., emotion
recognition, status detection, discerning relationships between people, interpreting
another’s mental state – and these processes may differ in their need for attentional
and working memory resources.
Some aspects of social cue decoding have been interpreted as automatic processes
which are not dependent on attentional resources (e.g., Ambady & Rosenthal, 1992;
Bargh, 1997; Barker, Andrade, & Romanowski, 2004; Oschner & Lieberman, 2001).
For example, Patterson, Foster, and Bellmer (2001) argue that ‘‘social judgments are
not typically the result of deliberate, controlled inferences... but rather are often
relatively automatic in nature and beyond awareness’’ (p. 208). Ambady and
Rosenthal (1992) argue that people can make accurate social judgments based on
very minimal cognitive processing. Further, it has been argued that when decoding
nonverbal channels of social information, people may actually make better social
judgments under attentional load, because intuition may be a better guide to the
meaning of social cues than controlled, in-depth cognitive processing (Ambady &
Rosenthal, 1992; Patterson & Stockbridge, 1998).
In accordance with this idea of social cue decoding as making limited demands on
attentional resources, correlational studies of healthy adult populations have tended to
find that measures of social cue decoding do not correlate with measures of working
memory and attentional control (e.g., MacPherson, Phillips, & Della Sala, 2002;
123
J Nonverbal Behav (2007) 31:137–152
139
Maylor, Moulson, Muncer, & Taylor, 2002; Sullivan & Ruffman, 2004; Toomey,
Seidman, Lyons, Faraone, & Tsuang, 1999). However, correlational methods do not
directly test whether two processes depend on the same cognitive resources: dual task
studies can provide more insight into this issue. The logic of dual task studies is that any
combination of tasks which depend on the same cognitive resources should interfere
with each other when performed in synchrony. There is some evidence that dynamic
social cue decoding can survive intact despite a working memory preload (Patterson &
Stockbridge, 1998; Patterson et al., 2001), providing more direct evidence that some
aspects of social processing may not be dependent on working memory.
In contrast, studies of patients with brain injury indicate overlapping brain networks being involved in both working memory and social perception. Lesions to the
frontal lobes of the brain disrupt both working memory tasks and the ability to
decode dynamic social stimuli (Mah et al., 2004). There are a number of studies of
patients with brain injury which indicate that their deficits in social cue decoding,
attentional control, and working memory are correlated (Bryson, Bell, & Lysaker,
1997; Channon & Crawford, 2000; Mah et al., 2004). This indicates that similar
anatomical and functional brain networks may be involved in social perception and
working memory. Also, dual task studies indicate that at least some nonverbal
processes of person perception (such as interpreting another’s emotions from a
videotape in the context of situational cues) can be disrupted by concurrent cognitive load (Gilbert, Pelham, & Krull, 1988).
There is agreement that the range of social judgments that we carry out in everyday
life is likely to depend on both automatic processes which do not make demands on
attention and more attentionally demanding processes which load working memory
(see, e.g., Wegner & Bargh, 1997, for a comprehensive review). However, the literature
described above indicates that there is some ambiguity about the cognitive mechanisms underlying the detection of social cues from dynamic nonverbal sources. If some
aspects of social cue decoding are largely automatic, beyond conscious control and
intuitive (e.g., Ambady & Rosenthal, 1992; Barker et al., 2004) then they should not be
substantially disrupted by concurrent cognitive load. In contrast, if aspects of social cue
decoding are dependent on attentional and working memory resources (e.g., Bozikas
et al., 2004; Mah et al., 2004) then concurrent load should substantially interfere with
performance. The current study used dual task manipulations to investigate whether
social cue decoding depends upon the same cognitive resources as a working memory
demanding secondary task.
The social cue decoding tasks selected (IPT and PONS) were chosen on the grounds
that they presented relatively naturalistic dynamic presentations of social situations,
and have been widely used in research into social perception in both healthy and
clinical populations. This study therefore addresses the theoretically interesting issue
of the extent of working memory involvement in two different social cue decoding
tasks, but also has implications for the interpretation of deficits in social perception,
particularly where an individual also has attentional or memory problems.
Experiment 1: Effects of Cognitive Load on the Interpersonal Perception Task
The Interpersonal Perception Task (IPT, Archer & Costanzo, 1988; Costanzo &
Archer, 1989) consists of video clips of social scenes, and the viewer has to decode an
123
140
J Nonverbal Behav (2007) 31:137–152
aspect of the social behavior portrayed in the scenes. The categories of social
relationships which must be decoded include: the nature of family relationships,
decisions about relative status, detection of deception, and deciding who has won a
competition. The scenes shown are relatively long clips of unrehearsed, spontaneous
behavior, and Archer and Costanzo argue that this means that the task is much
closer to the everyday demand to decode interactions. However, the people in the
videos knew they were being filmed, and the scenes were set up and played to the
camera, which does limit how naturalistic the stimuli are. Patients suffering from
schizophrenia, closed head injury, and focal frontal lobe lesions perform poorly on
video tasks requiring social decoding such as the IPT (e.g., Bryson et al., 1997;
Corrigan & Addis, 1995; Mah et al., 2004; Sergi & Green, 2002). An important
question is whether patients’ problems in decoding dynamic social cues are correlated with cognitive processes such as maintaining attention or updating working
memory which are also impaired in the same populations (e.g., Bozikas et al., 2004;
Bryson et al., 1997; Sergi & Green, 2002).
There is some evidence that social cue decoding as assessed by the IPT can
survive intact despite a working memory preload in healthy adults (Patterson &
Stockbridge, 1998; Patterson et al., 2001). This suggests relatively little involvement
of working memory in the IPT, although Patterson et al. (2001) concede that the
memory load involved in their task may not have been particularly taxing. In contrast to this evidence from a healthy population, clinical studies suggest that the IPT
may make demands on working memory. In a study of brain injury, Mah et al. (2004)
found that both orbitofrontal and dorsolateral frontal lesions impaired performance
on the IPT task, and in particular impaired the detection of deception. As the frontal
lobes are a key part of the brain network which controls attention and working
memory, Mah et al. argue there is likely to be overlap between the social processing
load of the IPT and working memory. Indeed, performance on the IPT was significantly correlated with measures of working memory.
In the current experiment, dual task methodology was used to look at the common resources involved in the IPT and tasks loading attention and working memory
in a healthy population. The cognitively-loading secondary tasks were variants on
the n-back paradigm (e.g., Braver et al,, 1997): a 0-back task which loads attentional
resources to monitor stimuli, and a 2-back task which loads attention and also makes
additional demands on working memory. In the 0-back task, participants make a
specific response to a single pre-specified target (e.g., the letter ‘X’) from a stream of
different letters. In the 2-back task the target must be identical to the item that was
presented two trials previously to provoke the specified response. The design of the
n-back is such that in both tasks (0-, and 2-back), participants have to respond to all
stimuli so that the task requires constant monitoring and attention. The information
that is presented during each trial is the same within both tasks, as are the demands
made on response selection and execution (Jansma, Ramsey, Coppola, & Kahn,
2000). However, the 0-back task makes minimal demands on working memory, while
the 2-back task demands constant updating of working memory for accurate performance.
This experiment therefore contrasts the following predictions: (1) If the social
cue decoding carried out in the IPT is mostly dependent on automatic processes
there should be little dual task cost to IPT performance from concurrent n-back
performance, (2) if social cue decoding depends on the maintenance of attention,
but not on updating within working memory, dual task interference levels should
123
J Nonverbal Behav (2007) 31:137–152
141
be significant and similar when the IPT is combined with either the 0-back or the
2-back secondary tasks compared to single task conditions, (3) if social cue
decoding depends heavily on updating functions in working memory there should
be substantially more disruption to the IPT from concurrent 2-back performance
as compared to concurrent 0-back performance. As it is possible that making
different kinds of social decision involves cognitive resources to different degrees,
we also investigated whether type of social decoding (e.g., detecting deception
versus making kinship judgments) interacts with cognitive load. In particular,
because there is evidence of greater frontal lobe involvement in deception
detection compared to other items on the IPT (Mah et al., 2004), this suggests
greater working memory load in these items.
In order to check for the possibility of dual-task trade-off effects, the performance
of n-back tasks under single and dual task conditions was also investigated. It is
possible that the IPT task could be successfully carried out during concurrent n-back
performance, indicating that social perception can be conducted during cognitive
load. However, this may reflect a prioritization of social cue decoding over the
performance on n-back tasks, and this will be investigated through analysis of the
secondary (n-back) task performance.
Method
Participants
Fifty-four participants were tested. All were first year psychology undergraduates at
the University of Aberdeen (27 male and 27 female; aged between 18 and 36; mean
age = 20.17, SD = 6.19). Their participation was as part of a scheme where they
gained credits for their course.
Design
A between-participants design was employed because there were insufficient stimuli
to allow multiple conditions to be given to individual participants. This resulted in
three conditions with 18 participants randomly assigned to each: (1) A single task
condition where the IPT was performed alone, (2) A dual task with 0-back condition,
where the IPT was performed concurrently with the attentionally-loading 0-back
task, and (3) A dual task with 2-back condition where the IPT was carried out
concurrently with the working memory loading 2-back task. Each participant was
randomly assigned to one of these conditions and completed the IPT task for their
assigned condition: either on its own (single task) or in a dual task situation. The
0-back and 2-back tasks were also completed as single tasks by all participants. In
order to check that the three experimental groups did not differ in terms of basic
social cue decoding, a facial emotion labeling task, using a subset of the Facial
Expressions of Emotion, Stimuli and Test (FEEST; Young, Perrett, Calder, Sprengelmeyer, & Ekman, 2002) stimuli, was used as a baseline comparison measure
between the groups. The experimental tasks were counterbalanced to take into
account which video task and letter sets were used for each task condition, and which
order the dual and single tasks, and the 0-back and 2-back, were completed.
123
142
J Nonverbal Behav (2007) 31:137–152
Materials and procedure
The IPT. Twenty four of the IPT video clips (Archer & Costanzo, 1988) were
presented using MediaLab (Jarvis, 2004) on a 17’’ computer screen without sound.
Six clips from the original thirty in the IPT were excluded following pilot work
because performance was indistinguishable from chance. It was necessary to exclude
sound in order to allow participants to simultaneously complete the n-back tasks.
Previous research (e.g., Iizuka, Patterson, & Matchen, 2002) had suggested although
IPT accuracy was lower in the visual-only modality, this method of presentation
resulted in above-chance performance and was concluded to be a valid test of social
decoding.
For each IPT item, a question appeared on the screen, followed by the video clip
relating to it. This was then followed by a screen which presented the question again,
a still from the video clip, and three possible answers to the question. For example
one question asked ‘Who is the child of the two adults?’ with the response options
‘only the little boy,’ ‘only the little girl,’ or ‘neither the boy or the girl.’ The still was
shown to clarify who the questions referred to, and reduce memory load (e.g., for
some questions participants had to decide whether the correct answer was a person
on the left or right hand side). The next item appeared once a response had been
made. The original IPT clips were between 30 and 90 seconds in length, but in the
current study each clip was cut to approximately 30 seconds, after pilot work indicated that this had negligible effects on accuracy. Some of the original IPT items
provided only two possible response choices. In the current study, a third choice was
added to these questions to reduce guessing effects and keep response format the
same for all items. The breakdown of types of IPT clips were: 4 about kinship, 4
about intimacy, 4 about the relative social status of two people, 6 about deception,
and 6 about who won a competition.
N-back tasks. The letter sets for the n-back tasks were aurally presented in a
pseudorandom sequence, either alone or in dual task conditions during the presentation of the social decoding videos. Figure 1 provides an example of both 0-back
and 2-back sequences. Three sets of four letters were used; Q,J,Z,B; X,R,T,N; and
L,Y,H,F. The letters were chosen to ensure that they were acoustically distinct.
These were heard through the computer’s built-in speakers at a rate of 3 seconds
between the onset of one letter and the onset of the next letter. Participants were
told to respond to letters by repeating the letter aloud, except for specified target
letters when they were asked instead to respond ‘‘snap.’’ The letters were presented
0-back (target ‘Z’)
JBQZ QBZJJQBZQBJ
2-back
RXTXNRRNRXTTTNXN
Fig. 1 Sample stimulus streams from the 0-back and 2-back tasks. All letters were repeated out loud
by participants, except for target letters (indicated by arrows). Participants were instructed to
respond ‘snap’ to target stimuli
123
J Nonverbal Behav (2007) 31:137–152
143
Table 1 Performance on the IPT Task under Single and Dual Task Conditions in Study 1
Condition
Single task
Dual with 0-back
Dual with 2-back
IPT proportion correct
IPT reaction time (ms)
M
SD
M
SD
0.437
0.470
0.410
0.113
0.118
0.116
4056
4407
4217
974
1586
1670
so that the target appeared 25% of the time. In the 0-back condition the target letter
was one letter (e.g., X), and each time the participant heard that letter the response
‘‘snap’’ was expected. In the 2-back condition a target response of ‘‘snap’’ had to be
made on hearing any letter which was identical to the letter heard two trials previously (i.e., the letter before last; for details see Figure 1).
Data Analysis
Errors and response times (RTs) were recorded for the social cue decoding tasks and
the n-back tasks. RTs were recorded in order to monitor any speed-accuracy tradeoff which might occur in the dual task situation. An audio tape-recorder was used to
record the n-back responses. These recordings were subsequently processed through
a voice key to obtain the response times for the n-back, from the onset of the
stimulus to the onset of the response. The response times were averaged to produce
a mean response time for each participant for each task – two participants were
excluded from the n-back RT analysis because of problems in filtering their
responses through the voice key.
Results
There was no difference between the three experimental groups (single task, dual
with 0-back, and dual with 2-back) in terms of their performance on the FEEST task
of facial emotion recognition, F(2, 53) = 0.88, g2p = .033,1 or the number of errors
made on the single 2-back task, F(2, 53) = 1.16, g2p = .044. This indicates that
participants in the three conditions did not differ in terms of emotion recognition or
working memory abilities.
Performance on the IPT task under single and dual task conditions is shown in
Table 1. A one-way ANOVA was carried out to investigate the effects of dual task
condition (single, 0-back, 2-back) on accuracy levels on the IPT task. There was no
significant dual task effect on accuracy, F(2, 53) = 1.22, g2p = .046. Accuracy rates
were relatively low, so to check whether performance was significantly different
from chance (i.e., 33.3% correct responding) in each dual task condition a series of
one sample t tests were calculated. These showed that in each condition (single task,
t(17) = 4.05, p < .01, dual with 0-back, t(17) = 5.04, p < .001, dual with 2-back,
t(17) = 2.91, p = .01), IPT performance was significantly above chance. To investigate further the different types of social decision made in the IPT, the effects of dual
task condition and decision type (with five levels: kinship relationships, intimacy,
1
SPSS produces partial eta-squared measures of effect size. For the differences between classical
and partial eta-squared as effect size measures see Pierce, Block, and Aguinis (2004).
123
144
J Nonverbal Behav (2007) 31:137–152
status analysis, deception detection, competition) on accuracy rates was examined.
There was no effect of dual task condition on overall accuracy, F(2, 51) = 1.49,
g2p = .055, but also there was no effect of decision type, F(4, 204) = 2.04, g2p = .039,
and no interaction between dual task condition and decision type, F(8, 204) = 0.65,
g2p = .025. Dual task effects on latencies to make decisions about the IPT items were
also investigated: there was no effect of dual task condition on mean RT, F(2, 53) =
0.27, g2p = .010.
N-back performance is described in Table 2. Error rates on the 0-back task were
rather low, and so for inferential analyses, all of the n-back task error scores were
transformed using square root transformations to achieve greater parity of variance
across conditions. However, because of the low level of errors on the 0-back task,
this analysis should be treated with caution. An ANOVA was carried out to
investigate the effects of dual task condition (single vs. dual, within-participants) and
n-back task type (0-back vs. 2-back, between-participants) on n-back errors. There
was a significant effect of dual task condition, F(1, 34) = 57.27, p < .0001, g2p = .627,
with more errors made in the dual task situation. There was also a significant difference between the 0- and 2-back tasks, F(1, 34) = 142.47, p < .0001, g2p = .808,
with more errors being made in the 2-back task. There was no significant interaction
between dual task condition and n-back task, F(1, 34) = 3.03, p = .08, g2p = .087.
A similar ANOVA was carried out to investigate RT to n-back stimuli. There was
an overall effects of dual task, F(1,32) = 34.20, p < .0001, g2p = .517, such that
n-back performance was slowed under dual task conditions. No difference was found
between 0-back and 2-back RT, F(1,32) = 3.10, p = .08, g2p = .088. The interaction
between dual task condition and type of n-back task approached significance,
F(1,32) = 3.80, p = .06, g2p = .106, with the magnitude of the dual task slowing being
slightly higher for the 0-back than the 2-back task.
Discussion
Considering only performance on the IPT task, there was no effect of either type of
dual task on accuracy or latency of social cue decoding. Further, there was no
interaction between the type of social decision made and dual task condition, contrary to the prediction that deception detection might make stronger demands on
working memory. This indicates that social cue decoding as measured by the IPT can
be carried out despite considerable load on attentional resources. Thinking about an
analogous real-life situation: people may be able to effectively decode social information from interactions in their visual field while simultaneously carrying out a
telephone conversation. Dual task effects are generally large in magnitude and so a
Table 2 Performance on the N-back Tasks under Single and Dual Task Conditions in Study 1
Condition
Single 0-back
Dual 0-back
Single 2-back
Dual 2-back
123
N-back errors
N-back
(ms)
reaction
time
M
SD
M
SD
0.029
0.074
0.274
0.528
0.047
0.082
0.166
0.126
831
1061
999
1114
198
218
182
206
J Nonverbal Behav (2007) 31:137–152
145
relatively low sample size was used in the current experiment, in common with
general methodology in this field (e.g., Phillips, Wynn, Gilhooly, Della Sala & Logie,
1999). Looking at the accuracy levels for the IPT in Table 1, it is of interest to note
that performance was actually better during dual task 0-back performance (47.0%
accuracy) compared to single task performance (43.7% accuracy). However, it still
remains possible that non-significant results were found because of inadequate
power.
Although performance on the IPT itself was not disrupted by concurrent n-back
performance, there were significant dual task costs in terms of slower and less
accurate performance on both 0-back and 2-back tasks. Carrying out the IPT placed
a general load on attentional functioning, but did not place additional load on the
updating function in working memory specifically tapped by the 2-back task.
Continuing the real-life analogy of having a telephone conversation while decoding
social cues from surrounding interactions, these results suggest that while social
information from the visual field could be detected, the telephone conversation
might become stilted or slowed.
There has been some debate about the extent to which social perception is
automatic or requires working memory resources (e.g., Bargh, 1997; Mah et al., 2004;
Oschner & Lieberman, 2001; Patterson et al., 2001). The current results indicate that
the dynamic social cue detection required by the IPT can be carried out effectively
during a complex working memory task - this suggests relatively automatic performance of the task. The fact that dual task interference was largely picked up on
secondary task performance indicates that participants were prioritizing social
decoding over the n-back tasks. This pattern indicates that the IPT may be resourcedemanding (in the sense that it interferes with concurrent attentional tasks) but it is
also relatively obligatory (in the sense of being beyond conscious control, and
impervious to the effects of load). Social decoding is thus somewhat like the process
of familiarity-based memory retrieval (Craik, Govani, Naveh-Benjamin, & Anderson, 1996), which is relatively unaffected by concurrent load, and unaffected by
instructions prioritizing particular task demands, but causes substantial decrements
in secondary task performance. Further, the similar degree of interference levels
with 0-back and 2-back tasks suggests that social judgments are not highly dependent
on the updating function of working memory.
The IPT presents lengthy clips for social decoding, and it is possible that the
amount of time available to sample from the videos may minimize the effects of
secondary task performance. Also, the IPT has relatively weak psychometric properties, e.g., relatively low accuracy rates and reliability, which might limit the chances
of finding dual task effects. To investigate whether the results found for the IPT
extend to a task of dynamic cue decoding involving more rapid processing, the same
dual-task methodology was applied to the Profile of Nonverbal Sensitivity.
Experiment 2: Effects of Cognitive Load on the Profile of Nonverbal Sensitivity
The Profile of Nonverbal Sensitivity (PONS, Rosenthal, Hall, DiMatteo, Rogers, &
Archer, 1979) is a widely used task of interpreting dynamic nonverbal social cues,
and has been used to measure accuracy of social decoding in both clinical and nonclinical populations. Dynamic displays of face, body, and voice tone are shown on
123
146
J Nonverbal Behav (2007) 31:137–152
video clips. The portrayals are all by the same woman, who is not a professional
actress. The major domains of social cue explored in this test are dominance and
emotional tone. An example of a submissive positive scene is expressing gratitude,
submissive negative includes talking about divorce, dominant positive includes
expressing motherly love, and dominant negative includes nagging a child.
PONS performance has been found to be impaired in schizophrenia (Sergi &
Green, 2002; Toomey, Schuldberg, Corrigan, & Green, 2002). There is evidence that
relatively automatic early visual processes may be related to poor ability to decode
social cues from the PONS among patients with schizophrenia (e.g., Wynn, Sergi,
Dawson, Schell, & Green, 2005); however less is known of the role of more controlled attentional processes in performing the task. A recent neuroimaging study of
the PONS indicated that performing the task activated a network of areas including
two regions of the frontal lobes (Lawrence et al., 2006) – these regions are also those
most consistently involved in working memory tasks such as the n-back (Owen,
McMillan, Laird, & Bullmore, 2005). This anatomical overlap in brain circuits involved in the PONS and n-back tasks suggests that carrying out both simultaneously
in a dual task situation is likely to cause considerable disruption.
Compared to the IPT, the PONS involves relatively short video clips of only a
single person, and the decisions to be made are about the social and emotional status
of that individual rather than about a social relationship between two people. Two
potential arguments might be made about the relative involvement of working
memory in the IPT and PONS. The PONS clips are shorter and include less information than the IPT and so might require more focused control of attention in order
to decode the information accurately. If this is the case then higher levels of interference between social decoding and the n-back tasks would be predicted. However,
the PONS also requires judgments to be made about only one individual rather than
the relationship between at least two people, suggesting that fewer points of information need to be gathered in order to make the social judgments – which might
result in low levels of interference between the PONS and n-back tasks.
As in Experiment 1, the PONS was performed simultaneously with 0-back and 2back tasks. Given the evidence for overlapping brain regions involvement in both
PONS and n-back tasks, and the brevity of the video clips, it was hypothesized that
there would be significant interference between the PONS and n-back tasks.
Method
Participants
Fifty-four participants were tested. All were first year psychology undergraduates at
the University of Aberdeen (27 male and 27 female; aged between 18 and 32; mean
age = 21.54, SD = 5.24). Their participation was as part of a scheme where they
gained credits for their course.
Design
As before there were three conditions with 18 participants randomly assigned to
each: single task, dual task with 0-back, and dual task with 2-back. All participants
123
J Nonverbal Behav (2007) 31:137–152
147
completed single task 0-back and 2-back tasks, and the FEEST facial emotion
labeling task. Task order was counterbalanced.
Materials and Procedure
The PONS. Fifty four of the visual-only PONS video clips (Rosenthal et al., 1979)
were presented on a 17’’ computer screen. For each item, participants were initially
shown the choice of potential answers, before being shown the clip, followed by the
list of potential answers again. This method has been used in previous studies using
the PONS (e.g., Sergi & Green, 2002; Toomey, Wallace, Corrigan, Schuldberg, &
Green, 1997), and matches the procedure for the IPT in Experiment 1. Participants
then had to select which of the three statements best described the video clip. For
example, response choices might be ‘leaving on a trip,’ ‘expressing deep affection,’
or ‘asking forgiveness.’ The next item appeared once a response had been made. The
original PONS stimuli contained only two response choices, but we added a third
statement to decrease the probability that a correct answer would be chosen by
chance, and to make the task more similar to the IPT. Pilot work established that
accuracy levels with three response choices were satisfactory. Each PONS clip was
two seconds long, and featured either the face, or body, or both face and body of the
same female actress.
N-back tasks. The procedure for these was identical to Study 1. In this experiment
data from three participants’ RT to n-back stimuli could not be processed through
the voice key.
Results
There was no difference between the three experimental groups (single task, 0-back,
and 2-back) in static facial emotion recognition, F(2, 53) = 0.16, g2p = .006, or errors
on the 2-back task, F(2, 53) = 1.05, g2p = .040.
Performance on the PONS under single and dual task conditions is shown in
Table 3. A one-way ANOVA was carried out to investigate the effects of dual task
condition (single, 0-back, 2-back) on accuracy in the PONS task. There was a highly
significant effect of dual task condition on PONS performance, F(2, 53) = 7.21,
p < .01, g2p = .220, and a Tukey’s LSD post-hoc test indicated that the 2-back dual
task condition resulted in substantially lower PONS performance compared to single
task. The 2-back condition differed from the 0-back condition, while there was no
difference between single task PONS and dual task 0-back PONS. An ANOVA
investigating the effect of dual task condition on latency to respond to PONS stimuli
revealed no dual task effect, F(2, 53) = 0.07, g2p = .003.
Table 3 Performance on the PONS Task under Single and Dual Task Conditions in Study 2
Condition
Single Task
Dual with 0-back
Dual with 2-back
PONS proportion correct
PONS
(ms)
reaction
time
M
SD
M
SD
0.614
0.566
0.486
0.093
0.098
0.116
3324
3489
3330
750
937
1062
123
148
J Nonverbal Behav (2007) 31:137–152
Table 4 Performance on the N-back Tasks under Single and Dual Task Conditions in Study 2
Condition
Single 0-back
Dual 0-back
Single 2-back
Dual 2-back
N-back errors
N-back
(ms)
reaction
time
M
SD
M
SD
0.051
0.121
0.324
0.557
0.072
0.105
0.158
0.092
855
1199
1010
1197
264
249
179
173
N-back performance is described in Table 4. As before, n-back task error scores
were transformed using square root transformations. For both transformed errors,
and n-back RT an ANOVA was carried out to investigate the effects of dual task
condition (single vs. dual, within-participants) and n-back task type (0-back vs.
2-back, between-participants). For n-back errors, there was a significant difference
between single and dual tasks, F(1, 34) = 67.96, p < .001, g2p = .667. As can be seen
from Table 4 there were much higher numbers of errors in n-back performance
during dual compared to single task conditions. There was also an effect of n-back
task on errors, F(1, 34) = 136.63, p < .001, g2p = .801, with a greater number of
errors in the 2-back compared to the 0-back task. There was no interaction between
dual task condition and type of n-back task, F(1, 34) = 0.38, g2p = .011, indicating
that similar amount of dual task interference was seen for both 0-back and 2-back
tasks.
In terms of the RT to n-back stimuli, there was a significant effect of dual task
condition, F(1, 31) = 60.35, p < .001, g2p = .661, with slower performance in dual
compared to single task conditions. There were similar response times for 0-back
and 2-back tasks, F(1, 31) = 2.11, g2p = .064. An interaction between dual task
condition and type of n-back task, F(1, 31) = 6.44, p < .05, g2p = .172, was further
explored using Tukey’s LSD test, and this indicated that while the 0-back task was
performed more quickly than the 2-back in single task conditions, both tasks were
carried out much more slowly in the dual task condition, where the RT to both
0-back and 2-back was equivalent.
Discussion
The results from this dual task study of the PONS task indicate that the social
decoding required by this task depends heavily on working memory resources.
Performing the PONS at the same time as a 2-back task resulted in considerably
poorer accuracy as compared to performing the PONS alone. Also, there was not a
significant effect of the 0-back task on PONS performance, indicating that any
interference between the PONS and 2-back is unlikely to be due to the general
attentional load of carrying out two tasks at once, but reflects more specific
involvement of working memory updating in the PONS. There was also a substantial
effect of carrying out the PONS on the accuracy and speed of performing the n-back
tasks.
Previous evidence indicates that deficits in both social perception and working
memory are seen in patients with schizophrenia (Sergi & Green, 2002; Honey &
Fletcher, 2006); also neuroimaging studies indicate that the social decoding
demanded by the PONS and n-back tasks result in activation in a remarkably similar
123
J Nonverbal Behav (2007) 31:137–152
149
brain network (Lawrence et al., 2006; Owen et al., 2005). The current study provides
direct evidence that social perception and working memory tasks make demands on
the same cognitive resources by showing substantial dual task interference between
the PONS and n-back. The social decoding required in the PONS test is dependent
on effortful working memory processes.
General Discussion
The extent to which social cue decoding depends on working memory resources
clearly depends on the methods by which social perception is tested, and the complexity of social decoding required. In the current experiments, we looked at two
widely used tasks of social decoding in order to investigate whether these tasks were
disrupted when working memory was loaded. Greater dual-task interference was
seen in the PONS task (a 21% decline in performance from single task to dual task
2-back condition) as compared to the IPT (a 6% decline from single task to 2-back
condition). This suggests that the PONS makes higher demands on working memory
resources than the IPT, possibly because of the much shorter nature of the video
clips or the type of social judgments to be made. The lack of dual task effects on
performance of the IPT task suggests that participants were able to compensate for
attentional load by accumulating brief snapshots of information across the lengthier
course of the clip (Ambady & Rosenthal, 1992). Although the dual task situation
generated here used laboratory-based tasks, it may not differ too markedly from
commonly encountered situations in the real world, where for instance one must
attend to and monitor a conversation with one person while observing social relationships and emotional cues in a different part of the room. Although it might often
be the case that a continuous stream of visual information is available from which to
decode social cues (as in the IPT), it is also possible to envisage situations more like
the PONS in which only brief sequences of information may be available (e.g., when
people move in and out of view, turn away, or disguise emotions).
The current tasks of social perception were dynamic and visual-only. It would also
be of interest to explore the role of attention and working memory in other measures
of social decoding such as emotion recognition from static faces or prosodic information. In the current experiments it was necessary to limit the social stimuli to the
visual channel in order to avoid perceptual interference with the auditory secondary
tasks. However future studies could explore the involvement of working memory in
prosodic decoding through the use of visually presented secondary tasks. As suggested above, an obvious explanation for why the PONS showed greater levels of
interference than the IPT is that the PONS clips were considerably shorter (2s
compared to 30s). It would be of interest to produce stimuli for which exactly
equivalent social decisions needed to be made, but which only differed in length of
presentation in order to further investigate dual task effects. Varying the number of
people viewed in the clip, and the types of social judgment to be made would also be
of interest. A final issue that could be addressed in future work is the role of prioritization in determining patterns of dual task effects. In experiment 1, IPT accuracy was unaffected by concurrent n-back while working memory performance
deteriorated in the dual task condition: it was argued that at least some aspects of
social cue decoding may be relatively obligatory in that they capture attention away
123
150
J Nonverbal Behav (2007) 31:137–152
from other tasks. This could be tested by differentially weighting performance on
social decoding and working memory in dual task situations.
We have begun exploring the role of attentional and working memory resources
in decoding social information from static visual cues. The results so far indicate that
the type of decisions to be made strongly influence the involvement of working
memory. Bull, Phillips, and Conway (2006) found that a concurrent working memory
task did not significantly disrupt the ability to decode mental states from pictures of
eyes. In contrast, Phillips, Channon, Tunstall, Hedenstrom, and Lyons (2006) found
that concurrent working memory caused substantial dual task interference when
identifying emotional expressions from photographs of faces. Future experiments
should help to clarify what types of social decision make greatest demands on
working memory by matching other aspects of task demands (e.g., number of
decision choices, length of presentation of stimuli).
The stimuli used to measure social cue decoding in the current studies, IPT and
PONS, were selected because (1) they have been widely used in previous studies of
healthy and clinical populations to measure social functioning, and (2) both involve
dynamic video stimuli which are more naturalistic than still photographs. The results
of the current study indicate that while the IPT makes relatively few demands on
working memory, the PONS places much stronger demands on working memory.
This may be useful information for anyone using these tests, particularly to assess
people who may have concurrent cognitive and social deficits. In terms of the theoretical debate about the role of attention in processing social cues, the results
suggest that where people are asked to decode dynamic social stimuli with limited
time to process the information (i.e., in the PONS), working memory resources are
loaded. However, the PONS has a number of artificial elements compared to the
social cue decoding which occurs in everyday environments: participants must
choose from predetermined labels, and the answers must be given explicitly. In
everyday social interaction the processes involved are likely to be more automatic
and implicit, and might be less susceptible to dual task interference.
In conclusion, the current results suggest that the PONS task of dynamic social
cue decoding depends on working memory resources, while in contrast the IPT task
of dynamic social decoding makes minimal demands on working memory. These two
tasks are widely used in assessing social perception in healthy and clinical populations, and the current results have important implications for the interpretation of
task deficits. Patients suffering from traumatic brain injury or schizophrenia who
show social skill deficits usually display concurrent problems with working memory,
and the current evidence suggests that those deficits may be interdependent, particularly where social decoding is assessed by the PONS task.
References
Ambady, N., & Rosenthal, R. (1992). Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. Psychological Bulletin, 111, 256–274.
Archer, D., & Costanzo, M. (1988). A guide to the Interpersonal Perception Task (IPT) for instructors
and researchers. Berkeley, CA: University of California Center for Media.
Bargh J. A. (1997). The automaticity of everyday life. In R. S. J. Wyer (Ed.), Automaticity of
everyday life: Advances in social cognition (pp. 1–61). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
123
J Nonverbal Behav (2007) 31:137–152
151
Barker, L. A., Andrade, J. K., & Romanowski, C. A. J. (2004). Impaired implicit cognition with
intact executive function after extensive bilateral prefrontal pathology: A case study. Neurocase,
10, 233–248.
Bozikas, V. P., Kosmidis, M. H., Anezoulaki, D. , Giannakou, M., & Karavatos, A. (2004). Relationship of affect recognition with psychopathology and cognitive performance in schizophrenia.
Journal of the International Neuropsychological Society, 10, 549–558.
Braver, T. S., Cohen, J. D., Nystrom, L. E., Jonides, J., Smith, E. E., & Noll, D. C. (1997). A
parametric study of prefrontal cortex involvement in human working memory. Neuroimage, 5,
49–62.
Bull, R., Phillips, L. H., & Conway, C. (2006). Examining the relationship between Theory of Mind
and executive functioning using dual-task studies. Manuscript submitted for publication.
Bryson, G., Bell, M., & Lysaker, P. (1997). Affect recognition in schizophrenia: a function of global
impairment or a specific cognitive deficit. Psychiatry Research, 71, 105–113.
Calkins, M. E., Gur, R. C., Ragland, J. D., & Gur, R. E. (2003). Face recognition memory deficits in
schizophrenia patients and their relatives: Comparison with visual object memory performance.
Schizophrenia Research, 60, 127–128.
Carton, J. S., Kessler, E. A., & Pape, C. L. (1999). Nonverbal decoding skills and relationship wellbeing in adults. Journal of Nonverbal Behavior, 23, 91–100.
Channon, S., & Crawford, S. (2000). The effects of anterior lesions on performance on a story
comprehension test: left anterior impairment on a theory of mind-type task. Neuropsychologia,
38, 1006–1017.
Corrigan, P. W., & Addis, I. B. (1995). Effects of extraneous stimuli on social cue perception in
schizophrenia. Psychiatry Research, 56, 111–120.
Costanzo, M., & Archer, D. (1989). Interpreting the expressive behavior of others: The Interpersonal
Perception Task. Journal of Nonverbal Behavior, 134, 225–245.
Craik, F. I. M., Govoni, R., Naveh-Benjamin, M., & Anderson, N. D. (1996). The effects of divided
attention on encoding and retrieval processes in human memory. Journal of Experimental
Psychology: General, 125, 159–180.
Gilbert, D. T., Pelham, B. W., & Krull, D. S. (1988). On cognitive busyness: When person perceivers
meet persons perceived. Journal of Personality and Social Psychology, 54, 733–740.
Honey, G. D., & Fletcher, P. C. (2006). Investigating principles of human brain functioning
underlying working memory: What insights from schizophrenia? Neuroscience, 139, 59–71.
Iizuka, Y., Patterson, M. L., & Matchen, J. C. (2002). Accuracy and confidence on the interpersonal
perception test: A Japanese-American comparison. Journal of Nonverbal Behavior, 26, 159–174.
Jansma, J. M., Ramsey, N. F., Coppola, R., & Kahn, R. S. (2000). Specific versus nonspecific brain
activity in a parametric N-Back task. NeuroImage, 12, 688–697.
Jarvis, B. (2004) MediaLab v 2004.1.39. Empirisoft Corporation.
Lawrence, E. J., Shaw, P., Giampietro, V. P., Surguladze, S., Brammer, M. J., & David, A.S. (2006).
The role of ‘shared representations’ in social perception and empathy: An fRMI study. NeroImage, 29, 1173–1184.
MacPherson, S. E., Phillips, L. H., & Della Sala, S. (2002). Age, executive function, and social
decision making: a dorsolateral prefrontal theory of cognitive aging. Psychology and Aging, 17,
598–609.
Mah, L., Arnold, M. C., & Grafman, J. (2004). Impairment of social perception associated with
lesions of the prefrontal cortex. American Journal of Psychiatry, 161, 1247–1255.
Maylor, E. A., Moulson, J. M., Muncer, A., & Taylor, L. A. (2002). Does performance on theory of
mind tasks decline in old age? British Journal of Psychology, 93, 465–485.
Ochsner, K. N., & Lieberman, M. D. (2001). The emergence of social cognitive neuroscience.
American Psychologist, 56, 717–734.
Owen, A. M., McMillan, K. M., Laird, A. R., & Bullmore, E. (2005). N-back working memory
paradigm: A meta-analysis of normative functional imaging studies. Human Brain Mapping, 25,
46–59.
Patterson, M. L., Foster, J. L., & Bellmer, C. D. (2001). Another look at accuracy and confidence in
social judgments. Journal of Nonverbal Behavior, 25, 207–219.
Patterson, M., & Stockbridge, E. (1998). Effects of cognitive demand and judgment strategy on
person perception accuracy. Journal of Nonverbal Behavior, 22, 253–263.
Phillips, L. H., Channon, S., Tunstall, M., Hedenstrom, A., & Lyons, K. (2006). Working memory and
emotional expression identification. Manuscript in preparation.
Phillips, L. H., Wynn, V., Gilhooly, K. J., Della Sala, S., & Logie, R. H. (1999). The role of memory
in the Tower of London task. Memory, 7, 209–231.
123
152
J Nonverbal Behav (2007) 31:137–152
Pierce, C. A., Block, R. A., & Aguinis, H. (2004). Cautionary note on reporting eta-squared values
from multifactor ANOVA designs. Educational and Psychological Measurement, 64, 916–924.
Rosenthal, R., Hall, J. A., DiMatteo, M. R., Rogers, P. L., & Archer, D. (1979). Sensitivity to
Nonverbal Communication: The PONS Test. Baltimore: The Johns Hopkins University Press.
Quintana, J., Wong, T., Ortiz-Portillo, E., Marder, S. R., & Mazziotta, J. C. (2003). Right lateral
fusiform gyrus dysfunction during facial information processing in schizophrenia. Biological
Psychiatry, 53, 1099–1112.
Sergi, M. J., & Green, M.F. (2002). Social perception and early visual processing in schizophrenia.
Schizophrenia Research, 59, 233–241.
Sullivan, S., & Ruffman, T. (2004). Social understanding: How does it fare with advancing years?
British Journal of Psychology, 95, 1–18.
Stone, V. E., Baron-Cohen, S., & Knight, R. T. (1998). Frontal lobe contributions to theory of mind.
Journal of Cognitive Neuroscience, 10, 640–656.
Stuss, D. T., Gallup, G. G., & Alexander, M. P. (2001) The frontal lobes are necessary for theory of
mind. Brain, 124, 279–286.
Toomey, R., Schuldberg, D., Corrigan, P.W., & Green, M.F. (2002). Nonverbal social perception and
symptomatology in schizophrenia. Schizophrenia Research, 53, 83–91.
Toomey, R., Seidman, L. J., Lyons, M. J., Faraone, S. V., & Tsuang, M. T. (1999). Poor perception of
nonverbal social-emotional cues in relatives of schizophrenic patients. Schizophrenia Research,
40, 121–130.
Toomey, R., Wallace, C. J., Corrigan, P. W., Schuldberg, D., & Green, M. F. (1997). Social processing correlates of nonverbal social perception in schizophrenia. Psychiatry: Interpersonal and
Biological Processes, 60, 292–300.
Wynn, J. K., Sergi, M. J., Dawson, M. E., Schell, A. M., & Green, M. F. (2005). Sensorimotor gating,
orienting and social perception in schizophrenia. Schizophrenia Research, 73, 319–325.
Wegner, D. M., & Bargh, J. A. (1997). Control and automaticity in social life. In D. T. Gilbert, S. L.
Fiske, & L. Gardner (Eds.) Handbook of Social Psychology (4th Edition). (pp. 446–498). NY:
McGraw Hill.
Young, A., Perrett, D., Calder, A., Sprengelmeyer, R., & Ekman, P. (2002). Facial Expressions of
Emotion - Stimuli and Tests (FEEST) ( Software Manual v2.1 ed.). Bury St Edmunds, England:
Thames Valley Test Company.
123