Human-Computer Interface Course – Final Project Lecturer: Dr. 連震杰 Student ID: P78971252 Student Name: 傅夢璇 Mail address: [email protected] Department: Information Engineering Laboratory: ISMP Lab Supervisor: Dr. 郭耀煌 Title: Real-time classification of evoked emotions using facial feature tracking and physiological responses Jeremy N. Bailenson, Emmanuel D. Pontikakis, Iris B. Mauss, James J. Gross, Maria E. Jabon, Cendri A.C. Hutcherson, Clifford Nass, Oliver John. International Journal of Human-Computer Studies. Volume 66 Issue 5, May, pp. 303-317, 2008. Keywords: Facial tracking; Emotion; Computer vision Abstract They propose real-time models to predict people emotion through subjects watching videotapes and their physiological responses. The real emotion is extracted from the facial expressions while watching videotapes instead posed. There are two emotion types that are amusement and sadness to be exploited. The emotion intensity is also observed. The experiment is split into three aspects which are performance measurement, model choose comparison and various specific tests. Introduction There are many applications and technologies of facial tacking are achieved by camera and image technologies. In Japan, some cars are installed cameras in the dashboard to insure the driving security. Besides, the cameras are also used to detect the drivers’ emotion statuses which are angry, drowsy or drunk drivers. First goal of facial emotion system development is to assist the human-computer interface applications. The other goal is to understand people emotion from their facial expressions. Related work There are at least three main ways in which psychologists assess facial expressions of emotions (Rosenberg and Ekman, 2000) 1) naı¨ve coders view images or videotapes, and then make holistic judgments concerning the degree on target faces in the images. But the limitation is in that the coders may miss subtle facial movements, and in that the coding may be biased by idiosyncratic morphological features of various faces. 2) use componential coding schemes in which trained coders use a highly regulated procedural technique to detect facial actions such as the Facial Action Coding System (Ekman and Friesan, 1978). The advantage is this technique including the richness of the dataset. The disadvantage is the frame-by-frame 3) coding of the points is extremely laborious. To obtain more direct measures of muscle movement via facial electromyography (EMG) with electrodes attached on the skin of the face. This allows for sensitive measurement of features, the placement of the electrodes is difficult and also relatively constraining for subjects who wear them. However, his approach is also not helpful for coding archival footage. Approach Part 1- Actual facial emotion The stimuli are used as the inputs which elicit intense emotions from people watch videotapes. The actual emotional behavior is higher accessed than in studies that used deliberately posed faces. Paul Ekman el. Propose the Duchenne Smiles that is the automatic smiles involving crinkling of the eye corners. The automatic facial expressions appear to be more informative about underlying mental states than posed ones (Nass and Brave, 2005). Part 2- Opposite emotions and intensity The emotions were coded second-by-second by using a linear scale for oppositely valenced emotions of the amusement and sadness. The learning algorithms are trained by both binary set of data and linear set of data spanning a full scale of emotional intensity. Part 3- Three model types Hundreds of video frames rated individually for amusement and sadness are collected from each person enable to create three model types. The first type is a ‘‘universal model’’ which predicts how amused any face is by using one set of subjects’ faces as training data and another independent set of subjects’ faces as testing data. The model would be useful for HCI applications in bank automated teller machines, traffic light cameras, and public computers with webcams. The second one is an ‘‘idiosyncratic model’’ predicts how amused or sad by using training and testing data from the same subject for each model. It is useful for HCI applications in the same person who uses the same interface. For example, driving in an owned car, using the same computer with a webcam, or any application with a camera in a private home are applications of the idiosyncratic model. Thirdly, a gender-specific model that trained and tested using only data from subjects in same gender is proposed. This model is useful for HCI applications target a specific gender. For example, make-up advertisements directed at female consumers, or home repair advertisements targeted at males. Part 4- Features The data is not only extracted from facial expressions but also from subjects’ physiological responses including Cardiovascular activity, Electro dermal responding and Somatic activity. The facial features from a camera and the heart rate obtains from the hands gripping the steering wheel. Part 5- Real-time algorithm The real-time algorithm that is designed to predict emotion from computer vision algorithms of facial features detecting and real-time physiological measures extracted is proposed. There are applications on respond to a user’s emotion such as cars seek to avoid accidents for drowsy drivers or advertisements seek to match their content to the mood of a person walking. The amusement and sadness are targeted in order to sample positive and negative emotions. The 15 physiological measures were monitored. The selected films induced dynamic changes in emotional states over the 9-min period which enables varying levels of emotional intensity across participants. Data collection Training data: It was taken from 151 Stanford undergraduates watched movies pretested to elicit amusement and sadness while they watch videotapes and physiological responses were also assessed. In the laboratory session, firstly, the participants should watch a 9-min film clip which was composed of an amusing, a neutral, a sad, and another neutral segment (each segment was approximately 2min long). From the larger dataset of 151, randomly chose 41 to train and test the learning algorithms Expert ratings of emotions It was anchored at 0 with neutral and 8 with strong laughter for amusement and strong sadness expression. Average inter-rater reliabilities were satisfactory, with Cronbach’s alphas= 0.89 (S.D.= 0.13) for amusement behavior and 0.79 (S.D.= 0.11) for sadness behavior. Physiological measures The 15 physiological measures were monitored including heart rate, systolic blood pressure, diastolic blood pressure, mean arterial blood pressure, pre-ejection period, skin conductance level, finger temperature, finger pulse amplitude, finger pulse transit time, ear pulse transit time, ear pulse amplitude, composite of peripheral sympathetic activation, composite cardiac activation, and somatic activity shown as the figure. System architecture The videos of the 41 participants were analyzed at a resolution of 20 frames per second. The level of amusement/sadness of every person for every second was measured from 0 (less amused/sad) to 8 (more amused/sad). The goal is to predict at every individual second the level of amusement or sadness for every person. The emotion recognition system architecture from facial tracking output and physiological responses is shown as the figure. For measuring the facial expression of the person at every frame, the NEVEN Vision Facial Feature Tracker is used. 22 points are tracked on a face with four blocks which are mouth, nose, eyes and eyebrow. eyebrow eyes nose mouth Predicting emotion intensity The WEKA software package is chosen as the statistical tool, the linear regression function using the Akaike criterion for mode selection. Two-fold cross-validation was performed on each dataset using two non-overlapping sets of subjects. The separate tests are performed both sadness and amusement. Three test are using face video alone, physiological features alone, and using both of them to predict the expert ratings. The table below demonstrates the intensity of amusement is easier predicted than that of sadness. The correlation coefficients of the sadness neural nets were consistently 20–40% lower than those forthe amusement classifiers. The table shows the facial features performed better than the classifiers only using the physiological ones. The dataset is processed to discrete the expert ratings for amusement and sadness. In the amusement datasets, all the expert ratings less than or equal to 0.5 were set to neutral and 3 or higher were discretized to amused. On the other hand, in the sadness datasets, all the expert ratings less than or equal to 0.5 were discretized to neutral and 1.5 or higher were discretized to sad. Emotion classification A Support Vector Machine classifier with a linear kernel and a Logitboost with a decision stump weak classifier using 40 iterations (Freund and Schapire, 1996; Friedman et al., 2000) is applied. Each dataset uses the WEKA machine learning software package (Witten and Fank, 2005). The data is split into two non-overlapping datasets and performed a two-fold cross-validation on all classifiers. The precision, the recall and the F1 measure is defined as the harmonic mean between the precision and the recall. For a multi-class classification problem with classes Ai, i=1,..,M and each class Ai having a total of Ni instances in the dataset, if the classifier predicts correctly Ci instances for Ai and predicts C’I i instances to be in Ai where in fact those belong to other classes (misclassifies them), then the former measures are defined as The discrete classification results for all-subject datasets is shown as Experimental results The linear classification results for the individual subjects is shown as The discrete classification results for the individual subjects is shown as These two tables show that the performance of amusement is better than sadness. Linear classification results for gender-specific datasets as table Discrete classification results for gender-specific datasets as the table These two tables indicate in the gender of male emotional responses than female. Conclusion A real-time system for emotion recognition is presented. A relatively large number of subjects watched videos through facial and physiological to recognize the feel of amused or sad responses. A second-by second ratings of the intensity with expressed amusement and sadness to train coders. The results demonstrated better fits in the performance of emotion categories than intensity, the amusement ratings than sadness, a full model using both physiological measures and facial tracking than use alone and person-specific models than others. Discussion • The emotion recognition through facial expressions while watching • videotapes is not strong prove become of the limitations in content of videotapes Otherwise, the correctness of the devices of physiological features collection • are also considered The statistics in the aspects of emotion intensity is not significant improve References  Bradley, M., 2000. Emotion and motivation. In: Cacioppo, J., Tassinary, L., Brenston, G. (Eds.), Handbook of Psychophysiology. Cambridge University Press, New York.  Ekman, P., Friesan, W., 1978. The Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press.  Ekman, P., Davidson, R.J., Friesen, W.V., 1990. The Duchenne smile: emotional expression and brain physiology II. Journal of Personality and Social Psychology 58, 342–353.  el Kaliouby, R., Robinson, P., Keates, S., 2003. Temporal context and the recognition of emotion from facial expression. In: Proceedings of HCI International, vol. 2, Crete, June 2003, pp. 631–635. ISBN 0-8058-4931-9.  Kring, A.M., 2000. Gender and anger. In: Fischer, A.H. (Ed.), Gender and Emotion: Social Psychological Perspectives. Cambridge University Press, New York, pp. 211–231.  Lyons, M.J., Bartneck, C., 2006. HCI and the face. In: CHI ’06 Extended Abstracts on Human Factors in Computing Systems. ACM, pp. 1671–1674.  Pantic, M., Patras, I., 2006. Dynamics of Facial Expression: Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences. IEEE Transactions on Systems, Man and Cybernetics—Part B, vol. 36, no. 2, pp. 433–449, April 2006.  Witten, H.I., Fank, E., 2005. Data Mining: Practical Machine Learning Tools and Techniques, second ed. Morgan Kaufmann, San Francisco.  Zlochower, A., Cohn, J., Lien, J., Kanade, T., 1998. A computer vision based method of facial expression analysis in parent–infant interaction. In: International Conference on Infant Studies.