✚ A neuroscience
perspective on
Perception
using music to of
achieveProblems
emotional
Music:
objectives
and Prospects
Frank Russo
Ryerson University
Frank Russo
SMARTLab
Ryerson University
Room 217
February 10, 2016
1 of 41
The importance of emotion regulation
and emotional communication
• Emotions affect our cognition and behavior
• Emotions are primarily conveyed nonverbally
– 7%-38%-55% rule (lexical, prosodic, facial)
• Necessity of facial and vocal expressivity is evident
throughout the lifespan
– Still-face paradigm (Tronick and colleagues)
– “Botox effect” (Davis et al., 2010, Emotion)
2 of 40
2
Emotional mimicry in song
3 of 40
3
zygomaticus (smiling)
0.6
0.4
0.2
happy
0
neutral
perception
imagery
sad
-0.2
-0.4
-0.6
Chan, Livingstone & Russo (2013), Music Perception
4 of 40
4
corrugator (frowning)
0.6
0.4
0.2
happy
0
perception
-0.2
imagery
neutral
sad
-0.4
-0.6
-0.8
5 of 40
5
Emotion Induction
(Stress Regulation | Adults)
Sandstrom & Russo (2009), Music and Medicine
6 of 40
6
7 of 40
7
1
0.8
0.6
Sad
(Grieg)
0.2
0
Agitated
(Shostakovich)
-0.2
-0.4
Peaceful
(Bizet)
-0.6
Happy
(Strauss)
-0.8
-1
-1.2
Time
Sandstrom & Russo (2010), Music & Medicine
Peaceful music
Recovery from stress
GSR (Z-score)
0.4
White noise
2.5
2.5
2
2
1.5
1.5
1
1
0.5
0.5
0
0
40
60
80
100
Absorption (TAS)
120
40
60
80
100
120
Absorption (TAS)
Sandstrom & Russo (2009), Psychology of Music | AIMS Scale
8 of 40
8
Emotion Induction
(Stress Regulation | Infants)
Ghazban, Russo, Trehub & Boudreau (in progress)
9 of 40
9
10 of 40
10
Music and Emotional Communication in
Deaf children who use CIs
11 of 40
11
CIs and Music
• Children who use CIs generally take pleasure in
listening to music and many participate in musical
activities (Gfeller et al., 1999; Nakata et al., 2005)
• Early work suggested that perception of rhythm is in
tact but perception of melody impaired (Cooper,
Tobey, & Loizou, 2008)
12 of 40
12
Measures: Music Perception
• MBEA for Children (Peretz & Béland, 2013)
– Uses 5 subscales of music perception including Scale,
Different Contour, Interval, Rhythm, and incidental
Memory
– Examples:
13
CIs and Music Perception
Hopyan, Peretz, Chan, Papsin, Gordon, 2012
14 of 40
14
Experimental findings in normal hearing children
 Children randomly assigned to receive keyboard
lessons for one year were more successful at
determining the correct emotion in semantically
neutral spoken sentences than a control group
(Thompson et al., 2004)
15 of 40
15
Stress
patterns
Music
Phrasing
Emotional
speech
Pitch
16 of 40
16
Regular Practice/
Weekly Lessons
6 months
VS.
17 of 40
17
Music Perception
18 of 40
18
Music Perception: Subscales
100
90
90
*
80
*
*
70
60
50
40
30
20
10
Percentage of correct responses
Percentage of correct responses
100
80
70
60
Pre
50
Mid
40
Post
30
20
10
0
Scale
Contour
Interval
Rhythm
Memory
0
Scale
Contour
Interval
Rhythm
Memory
19 of 40
19
Speech Perception
Audio-visual
Audio-only
20 of 40
20
Possible mechanisms
• Changes in auditory cortex and the structural
connectivity of cortical and subcortical networks
Schlaug et al., 2010; Strait & Kraus, 2011
21 of 40
21
Possible mechanisms (contd).
• Increased auditory attention and/or working
memory
Strait & Kraus, 2011
22 of 40
22
Summary | Deaf Children who use CI’s
o Participants in the music condition improved on music
perception tasks to a greater extent than children in the
art condition.
o Participants in the music condition transferred skills to
the detection of emotion in speech
o Future work should consider mechanisms, the modality
of training (voice, keyboard, ear training), and
contributions of multimodal feedback (visual,
vibrotactile)
23 of 40
23
Emotional Communication in
People living with Parkinson’s
• PD exhibit deficits in spontaneous facial expressions,
termed the Masked face
24 of 40
24
Rhythmic Auditory Stimulation
• Rhythmic auditory stimulation while walking can
improve gait velocity, stride length, step
cadence1-2
– Dance therapy may also improve measures of gait and
balance 3-5
• Can rhythmically supported intentional
mimicry assist rehabilitation of spontaneous
facial function in PD?
Baseline Assessment
Participants
• 27 PD patients
• 28 aged-matched controls
Procedure
• Observe videos of emotional speech and song1-2
• Spontaneous facial movements recorded with EMG3-5
Placement of electrodes for EMG
Green = zygomaticus (smiling muscles for positive emotions)
Red = Corrugator (frowning muscles for negative emotions)
Purple = Frontalis major (frowning muscle for negative emotions)
26 of 40
26
delayed response
27 of 40
27
Singing Intervention
A 13-week imitative facial expressive singing program
Participants
• 26 Parkinson’s Disease patients
– 13 Treatment
– 13 Control
Procedure
• Observe videos of emotional speech and song1-2
• Spontaneous facial movements recorded with EMG3-5
Assessment
• Week 1, Week13, and 2 month follow-up
28 of 40
28
Results
29 of 40
29
Results
30 of 40
30
Emotional Communication in Autism
31 of 40
31
Deficits in mimicry
• Decreased congruent
automatic mimicry of
facial expressions
(McIntosh et al., 2006)
• Delayed mimicry of
facial expressions
(Oberman et al., 2009)
32 of 40
32
Abnormal functioning of actionobservation network
• Individuals with
autism: diminished
activation of this
network in response
to emotion displays
(Oberman et al., 2005)
33 of 40
33
Imitation therapy using song:
Video game for autism
34 of 40
34
35 of 40
35
Autism video game study: DV’s
•
•
•
•
•
EEG mu wave (index of action observation network)
EMG (corrugator supercilli and zygomaticus major)
Basic Empathy Scale (Jolliffe & Farrington, 2006)
Accuracy on emotion judgments
Social Responsiveness Scale (Bölte et al., 2008)
36 of 40
36
Preliminary results (N = 9)
Accuracy of Emotion Detection
1
0.9
0.8
0.7
0.6
0.5
Imitation group
0.4
Observation group
0.3
0.2
0.1
0
Pre-Training
Post-Training
37 of 40
37
Preliminary results (N = 9)
Social Responsiveness Scale
140
120
*
100
80
Imitation Group
60
Observation Group
40
20
0
Pre-Training
Post-Training
38 of 40
38
russo@ryerson.ca
smartlaboratory.org
@RyersonSMARTLab
39 of 40
39