KMSF- sunny
親： namachan さん
1

 Music – Emotion: mutual relationship
 Users choose songs based on current feelings
 Playlist constantly expanding
 difficulty in picking appropriate song
2

 Track User Emotion
 Recommend by Sorting playlist based on user’s current emotion
 Sort songs by 2 factors
 Relevancy to User Preference
 Effect on User Emotion
3

 Matching Music Mood and User Emotion
(-) Emotion: declared by user
(-) Subjectiveness on
Music Mood?
(-) Music Mood and User
Emotion are always the same?
 EmuPlayer
 Emotion: automatically detected
 User Preference under each emotion is studied
 Recommendation implying emotional effect’s feedback
4

 Before
 NoRPulse = 76.73, NoRTemp = 33.6
 Pulse = 70.04, Temp = 34.0  relax/~sleepy
 Song Information
 SongNo = 3
 SongID = 4, Title = “So Close”
 After
 “Like”
 Pulse = 79.0, Temp = 34.39  pleasure
 Score = 1
5

Vital
Information
Emo
Detector
User
Emotion
Recommender
Sorted
Playlist
EmuPlayer
 Emotion Recognition
 Music Recommendation
6

Emotion Recognition
Merit of Vital-sensor
 Requirements
 Portability if integrated in Music Player
 Continuity of output data
 Sensitiveness towards changes in emotion
 Vital-sensor meets all those requirement
7

Emotion Recognition
Vital-sensor v.s Other Methods
Requirement regard for the use in MRS
Method
Portability if integrated in
Music Player
Continuity of output data
Facial Expression
Speech
Eyes movement
Brainwave
Gesture
Vital-sensor
X
X
○
X
△
△
X
X
△
○
X
○
Sensitiveness towards changes in emotion while listening to music
△
△
△
◎
△
○
8

Emotion Recognition
Russell model and Two Biosignals
 Russell’s model
 Horizontal axis:
Pleasure
 SkinTemp
 Vertical axis:
Arousal
 Heart Rate
180 °
135 °
90 °
225 °
270 °
45 °
315 °
0 °
9

Emotion Recognition
Mapping in EmuPlayer
 Define Emotion Region
 Based on Theory of the
Fuzziness of words
 8 equal regions
 Mapping
 Based on angle
 (1,1)  45 °  Excitement
10

 2 factors to evaluate a song
 Relevancy to User Preference
 Mental Effect on User Emotion
 2 subjects
 Study User Preference
 Study Emotional Effect of songs
11

Music Recommendation
Study User Preference
 Rating Like/Dislike
 Record listening history
12

Music Recommendation
Study songs’ emotional effect
 Define emotional effect: Good-Bad
 to avoid potentially harmful recommendations to user emotion good bad
13

Music Recommendation
Effect Definition Survey
Matching point = 42/48*100 = 87.5%
14

Music Recommendation
Rating songs
 Better songs rank higher
15

1
User
Data Receiver
2
8
Data
Pre
Processor
9
3
Emo
Detector
10
7’
11
Database
4
Evaluator
Recommender
5
5
RF-ECG sensor
7
6
10’
Interface
16

17

Number of
Participant
10~
Gender Average Age
Male 21
 Experiment 1: Testing Accuracy of Emotion
Recognition through arranged situation
 Survey: (1) if they experienced the emotion expressed through the situation,
 and (2) if not, what emotions rather than the one in (1) they experienced.
18

Output
(Result from
Engine)
Input
(Verified
Experimenting Emo)
Arousal
Relaxation
Exciteme nt
Pleasure Relaxatio n
4.25% 7.96% 22.77
%
55.58
%
Excitement 5%
Pleasure
10% 63.34
%
1.4% 10.87
%
21.66
%
66.86
%
20.87
%
Arousal
Depression
81.68
%
5%
8.33% 0
0
0
6.66% 6.66% 0
0
Sleepine ss
0
0
Disples ure
9.44% 0
0
0
Distress Depressi on
0
0
0
0
0
0
0
0
6.66% 20% 10% 55.01
%
Accuracy = 64.5%
19

Evaluation on Emotion Recognition
Experiment 2
 Change User Emotion by music
 Purpose
 Verify whether the system can realize user’s emotional changes
 Verify songs’ influence on listeners’ emotions
20

Case Experiment
1
2
Arousal
 Pleasure/Rela xation
Normal
 Pleasure
Mean
Classical music
Music participants like
Result
Arousal:81.68%  1.41%
Pleasure:0%  54.91%
Relaxation:6.66%  38.94%
Pleasure: 66.86%  93.33%
3 Normal
 Excitement
Fast beat
Music
Excitement: 10.87%  62.12%
4 Normal
 Depression
Loud Heavy
Music played in long time
Pleasure:66.86%  Depression:
80.02%
21

Emotion Recognition
Conclusion from 2 experiments
 Accuracy of Extracting Emotion: 64.5%
 Strong at detecting bad emotions
 Detect precisely regarding to changes of user emotion
 Hypothesis of music influencing on user emotion is true
22

 Observing high-rating songs
 % being “dislike” after being listened
 % paying “bad influence” on user’s emotion after being listened
 % being reduced in score after being listened
 Observing “like” song
 Emotional change
23

EmuPlayer Performance
Observing high-rating songs
24

EmuPlayer Performance
Observing “like” song
Emo Before
Arousal
Arousal
Excitement
Excitement
Pleasure
Pleasure
Relaxation
Relaxation
Relaxation
Sleepiness
Displeasure
Emo After
Relaxation
Sleepiness
Arousal
Pleasure
Excitement
Pleasure
Arousal
Pleasure
Relaxation
Sleepiness
Displeasure
Percentage
1%
1%
1%
2%
2%
77%
1%
2%
7%
4%
2%
 No song influencing badly on users’ emotion
25

Conclusion of EmuPlayer Performance
 EmuPlayer algorithm ensures recommendation of songs meeting proposed two requirements
 Songs influencing badly on user emotion: 0%
 Songs being “dislike” in later listening time:
6.66%
26

 Are you interested in such a MRS system?
 Yes: 90%
 Scale your satisfaction of EmuPlayer’s work
 Average point = 3.6/5
 Do you feel uncomfortable wearing RF-ECG?
 Yes: 40%
 Did you experience bad emotion after listening to highrating song
 Yes: 10%
 Reflect the truth: proposing ER method responds to only clear and strong emotions. Slight changes in emotion felt by users may not be recognized by the system
27

 Concept of EmuPlayer is essential
 Evaluate song through 2 factors
 Employ User Emotion as crucial input for MRS
 Accuracy of extracting emotion is not very high:
64.5%
 Strong at detecting bad emotion
 Applicable in giving alert when playing music influences badly on listener’s emotional state
 EmuPlayer’s efficiency in suggesting songs meeting the two requirements
28

 Enhance the accuracy of detecting emotion by
 Employing other means than Heart Rate and Skin
Temperature
 Alternate RF-ECG
 Enhance the work of Recommending music by combining proposed method with songs’ content analyzing
 Enhance reasoning user’s state by combining
User Emotion with context analyzing
29

Thank you for listening
30