FACE EXPRESSION RECOGNITION USING ANN Ramya Avabhrith M Tech, Dept of CSE, Nitte Meenakshi Institute of Technology,Yelahanka, Bangalore, Karnataka, India ramyaavabrath@gmail.com Dr. Jharna Majumdhar Shipla Ankalaki Dean R&D, Professor and Head, Dept of CSE (PG) Nitte Meenakshi Institute of Technology,Yelahanka, Bangalore, Karnataka, India M Tech, Dept of CSE, Nitte Meenakshi Institute of Technology,Yelahanka, Bangalore, Karnataka, India classification of human facial expression by computer is an important issue to develop automatic facial expression recognition system in vision community. This paper explains about an approach to the problem of facial feature extraction from a still frontal posed image and classification and recognition of facial expression and hence emotion and mood of a person. Feed forward back propagation neural network [3] is used as a classifier for classifying the expressions of supplied face into four basic categories like happy, sad, surprise, and neutral. For face portion segmentation basic image processing operation like projection and skin color polynomial model. Facial features like eye, eyebrow and lip are extracted based on the projection and fine search method. For most efficient classifier neural network is used for the facial expression classification. ABSTRACT Facial Expression Recognition (FER) is a rapidly growing and ever green research field in the area of Computer Vision, Artificial Intelligent and Automation. There are many application which uses Facial Expression to evaluate human nature, feelings, judgment, opinion. Recognizing Human Facial Expression is not a simple task because of some circumstances due to illumination, facial occlusions, face color/shape etc. Our goal is to categorize the facial expression in the given image into four basic emotional states – Happy, Sad, Neutral and Surprise. Our method consists of three steps, namely face detection, facial feature extraction and emotion recognition. First, face detection is performed using a novel skin-color extraction and extraction of location of facial component features such as the eye and the mouth analysis by using a knowledge based approach and projection. Next, the extraction of facial features distance is performed by employing an iterative search algorithm, on the edge information of the localized face region in gray scale. Finally, emotion recognition is performed by giving the extracted eleven facial features as inputs to a feed-forward neural network trained by back-propagation. PREVIOUS WORK Facial expressions provide the building blocks with which to understand emotion. In order to effectively use facial expressions, it is necessary to understand how to interpret expressions, and it is also important to study what others have done in the past. Fasel and Luettin performed an in depth study in an attempt to understand the sources that drive expressions [4]. Their results indicate that using FACS may incorporate other sources of emotional stimulus including non-emotional mental and physiological aspects used in generating emotional expressions. Much of previous work has used FACS as a framework for classification. In addition to this, previous studies have traditionally taken two approaches to emotion classification according to Fasel and Luettin [4]: a judgment based approach and a sign-based approach. The judgment approach develops the categories of emotion in advance such as the traditional six universal emotions. The sign-based approach uses a FACS system [5], encoding action units in order to categorize an expression based on its constituents. This approach assumes no categories, but rather assigns an emotional value to a face using a combination of the key action units that create the expression. Keywords ANN, Expression classification, Facial features, Projection, perceptron, Skin colour extraction. Shilpaa336@gmail.com FACS, INTRODUCTION Computer vision is the branch of artificial intelligence that focuses on making computers to emulate human vision, including learning, making inferences and performing cognitive actions based on visual inputs, i.e. Images. Computer vision also plays a major role in Human Computer Intelligent Interaction (HCII) which provides natural ways for humans to use computer as aids. HCI will be much more effective and useful if computer can predict about emotional state of human being and hence mood of a person from supplied images on the basis of facial expressions. Mehrabian [1] pointed out that 7% of human communication information is communicated by linguistic language (verbal part), 38% by paralanguage (vocal part) and 55% by facial expression. Therefore facial expressions are the most important information for emotions perception in face to face communication. Ekman and Friesen [2] represent 6 basic face expressions (emotions), hence recognition of facial expressions became a major modality. For example, Smart Devices like computer/robots can sense/understand the human’s intension from their expression then it will helpful to the system to assist them by giving suggestions or proposals as per their needs. Automatic facial expression and facial AU (Action Unit) recognition have attracted much attention in the recent years due to its potential applications. For classifying facial expressions into different categories, it is necessary to extract important facial features which contribute in identifying proper and particular expressions. Recognition and DATA COLLECTION Data required for experimentation is done taking our collage NMIT database for training and testing of classifiers. The database contains 40 color images of 4 facial expressions (Happy, Sad, Surprise, and Neutral) posed by ten different NMIT students have rated each image on 4 emotion adjectives. Few samples are shown in Figure 1. 1 Neutral Smile Sad The skin-color pixels, which fall within the area between these two polynomials, can be extracted by the following rule checking on their r,g values: Surprise R1: g > Neutral Smile Sad and g> Surprise In r_g plane to exclude the bright white pixels, which falling around the point (r, g) = (0.33, 0.33). Figure 1. Few samples of facial expressions of persons PROPOSED METHOD R2: W = Real-time face detection algorithm for locating faces in images and videos. The algorithm starts from the extraction of skin pixels based upon rules derived from a simple quadratic polynomial model [6]. First, much computation time are saved. Second, both extraction processes can be performed simultaneously in one scan of the image or video frame. Happy Sad Surprise Neutral Classification 0.0004. R1 and R2 simultaneously for skin pixel extraction. To improve the extraction results, the two extra simple rules are used. R3: R > G > B; R4: R – G ≥ 45: An overview of the proposed work contains three major modules: (i) face detection, (ii) facial feature extraction and (iii) emotion recognition from the extracted features. The first step, namely face detection is performed on a color image containing the frontal view of a human subject. Preprocessing + Rule R3 is derived from the observation that the skin pixels tend to be red and yellow With Rule R3; the pixels tending to be blue can be effectively removed. Rule R4 is defined in order to remove the yellow-green pixels. Face detection S= { 1 if all R1; R2; R3 and R4 are true 0 otherwise Where S=1 means the examined pixel is a skin pixel. To improve the compactness of the formed skin-color regions, apply the projection to extract the correct face part. And crop the face part based on detected face centroid. Feature extraction Facial Feature Extraction Lip detection: Figure 2. Block Diagram For The Expression Recognition System Observing the pixels of lips, we find that the colors of lips range from dark red to purple under normal light condition. Two discriminate functions for lip pixels as Face detection Instead of using probabilistic models, here we use a quadratic polynomial model for the color model of skin pixels to reduce the computation time [6]. Skin-color region extraction: For chromatic color space, each pixel is represented by two values, denoted by (r, g). The conversion from conventional RGB color space to chromatic color space is defined as follows: During the discrimination using l(r), the darker pixels on faces, which are usually the eyes, eyebrows or nose holes, also need to be excluded from the discrimination. The rule for detecting lip pixels. L Where R, G, B denotes the intensities of pixels in red, green and blue channels, respectively. The skin locus, formed by the skin pixel distribution on the r_g plane. These two polynomials are: = { 1 if 0 otherwise and R≥20 and G≥20 and B≥20 L = 1 indicates the examined pixel is a lip pixel. The above rule will may fail for sometimes due to other darker features like eye. For the testing of lip part can be done by cropping [7] the face image into equally two parts. The lower part 2 contains the lip, and apply the above same rules will get only lip part. There may be full of noise in lip region, to extract the accurate lip distance apply the projections [8]. The projection points are thicker only when there is more concentration of pixels. The distance between two corners of lip can be identified by applying Horizontal projection. Similarly, distance between the upper and lower lip region can be identified by Vertical projection eye brow midpoint and right eye midpoint is considered (H). G Ratio LE is: v. a. H b. G G+H Ratio of distance between right eye and eye brows to both(left and right) eye and eye brows: The distance between left eye brow midpoint and left eye midpoint is considered (G). The distance between right eye brow midpoint and right eye midpoint is considered (H). G H Figure 3. face components projection. a) lip horizontal and verticle projection, b) eye horizontal and verticle projection. Ratio RE is: Eye detection: The eye components are extracted through a threshold operation (e.g. threshold =20) on the histogram-equalized grayscale image converted from the original color image. Ratio of Areas of features: The midpoint of a left eye brow is considered (A1), the midpoint of a right eye brow is considered (A2) and lower lip point is considered (A3).Find the area of a triangle A for the points A1, A2, A3. The upper peak point of a left eye is considered (B1), the upper peak point of a right eye is considered (B2) and lower lip point is considered (B3).Find the area of a triangle B for the points B1, B2, B3. The lower peak point of a left eye is considered (C1), the lower peak point of a right eye is considered (C2) and lower lip point is considered (C3).Find the area of a triangle C for the points C1, C2, C3. For the testing of correct eye part can be done by cropping the eye part. i.e after detection of face part apply the Edge detector i.e convert the image into binary form. For the binary image apply Horizontal and Vertical projection. The peak value of projection will gives the location of an eye in a face. Crop the eye region [8] [9] based on the peak value which is shown in Figure 3.b). So we can find the distance between eye and eyebrow, and iris. Facial Feature distance Ratio: Total eleven features extracted based on the projection mechanism. i. Ratio of eye brow distance a b c vi. Ratio of area of eye brows to eyes, eyebrows, and lip is R1 = A A+B+C vii. Ratio of area of upper peak point of eye to eyes, eyebrows, and lip is R2 = B A+B+C viii. Ratio of area of lower peak point of eye to eyes, eyebrows, and lip is R3 = C A+B+C Distance between upper and lower lips. Distance between two corners of lip Distance between only eye part i.e upper peak of an eye and lower peak of an eye. d Distance between point b &c Distance between point a &d ii. H G+H Ratio of number of skin and hair pixel is No. of skin pixel No. of hair pixel iii. The distance between the corner of left eye and right corner eye is considered (E). The distance between corners of mouth (F). Ratio of eye and lip distance is: Distance E Distance F iv. Ratio of distance between left eye and eye brows to both(left and right) eye and eye brows: The distance between left eye brow midpoint and left eye midpoint is considered (G). The distance between right ix. x. xi. Emotion Recognition After extracting the facial feature we can classify the expression based on the following measures- 3 calculated in the output layer are then back propagated to the hidden layers where the synaptic weights are updated to reduce the error. This learning process is repeated until the output error value, for all patterns in the training set, are below a specified value. The Back Propagation, however, has two major limitations: a very long training process, with problems such as local minima and network paralysis; and the restriction of learning only static input -output mappings. ANN requires large processing time for large neural networks. The network is trained in supervised manner using input and target. Here Targets are user defined for three expressions. We used [1000] for neutral, [0100] for happy, [0010] for sad, and [0001] for surprise. Eleven features of face values are given as an input to the neural network. Model uses an input layer, 11 hidden layers and 4 output layers as shown in Figure 4. a. Happiness (smile): Corners of the lips are drawn back and up. Mouth may or may not be parted, teeth exposed. Cheeks are raised. Crow’s feet near the outside of the eyes b. Sadness: Inner corners of the eyebrows are drawn up. Corner of the lips are drawn down. Lower lip pouts out c. Surprise: The brows are raised and curved. Eyelids are opened, white of the eye showing above and below. Jaw drops open and teeth are parted but there is not tension or stretching of the mouth. d. Neutral: Inner corners of the eyebrows are normal. Corner of the lips is starched. Eyelids are partially opened. Artificial Neural Network classifier NEUTRAL An Artificial Neural Network often just called a neural network is mathematical model inspired by biological neural networks. A neural network consists of an interconnected group of artificial neurons, and it processes information using a connectionist approach to computation. In most cases a neural network is an adaptive system that changes its structure during a learning phase. Neural networks are used to model complex relationships between inputs and outputs or to find patterns in data. Neural computing has re-emerged as an important programming paradigm that attempts to mimic the functionality of the human brain. This area has been developed to solve demanding pattern processing problems, like speech and image processing. These networks have demonstrated their ability to deliver simple and powerful solutions in areas that for many years have challenged conventional computing approaches. A neural network is represented by weighted interconnections between processing elements (PEs). These weights are the parameters that actually define the non-linear function performed by the neural network. Perceptron: The Perceptron is a type of artificial neural network. It can be seen as the simplest of feed forward neural network; a linear classifier. The perceptron is a binary classifier that maps its input x (a real valued vector) to an output value f(x) (a single binary value) across the matrix. F(x) = 1, if w.x+b>0 = 0, else where w is a vector of real valued weights and w.x is the dot product (which compute the weighted sum) .b is a “bias” a constant term that does not depend on any input value. The value of f(x)(0 or 1) is used to classify x as either a positive or a negative instance in the case of a binary classification problem .The bias can be thought of as offsetting the activation function ,or giving the output neuron a base level of activity. If b is negative, then the weighted combination of inputs must produce a positive value which is greater than –b in order to, push the classifier neuron over the 0 threshold. Back-Propagation Networks is most widely used neural network algorithm than other algorithms due to its simplicity, together with its universal approximation capacity. The back-propagation algorithm defines a systematic way to update the synaptic weights of multi-layer perceptron (MLP) networks [3][10]. The supervised learning is based on the gradient descent method, minimizing the global error on the output layer. The learning algorithm is performed in two stages feed-forward and feedbackward. In the first phase the inputs are propagated through the layers of processing elements, generating an output pattern in response to the input pattern presented. In the second phase, the errors SMILE Facial features SAD SURPRISE Figure 4. Neural network EXPERIMENTAL RESULTS AND ANALYSIS In this paper expression is detected for NMIT collage Face Database for frontal view facial images. Fig. 5 shows the results of facial feature extraction. Initially Face portion segmentation is done by skin polynomial model and for the accurate face part extracted by projection Cropped Facial image is divided into two regions based on the centre of an image and location of permanent facial feature by projection mechanism. Figure 4 shows localized permanent facial features which are used as input to classification. For ANN classification, Training Phase: In this work, supervised learning is used to train the back propagation neural network. The training samples are taken from the NMIT collage database. This work has considered 18 training samples for all expressions. After getting the samples, supervised learning is used to train the network. Testing Phase: This proposed system is tested with NMIT collage database. It was taken totally 40 sample images for all of the facial expressions. Fig. 6 shows the GUI for displaying the results of face expression. Success rate is given in TABLE I. a 4 b c enormously beneficial for fields as diverse as medicine, law, communication, education, and computing. The currently, using the skin based polynomial model reduces the computation time, features are every effective for classification of the expression. And also back propagation neural network is efficient for recognition facial expression. This aspect of machine analysis of facial expressions forms the main focus of the current and future research in the field. d e f REFERENCES [1] Mehrabian.A,"Communication without Words", Psychology Today, Vo1.2, No.4, pp 53-56. 1968. [2] Ekman, P, Friesen, “Constants across Cultures in the Face and Emotion”, J. Pers. Psycho. WV, 1971, vol. 17, no. 2, pp. 124-129. [3] S.P.Khandait, Dr. R.C.Thool and P.D.Khandait. “Automatic Facial Feature Extraction and Expression Recognition based on Neural Network” (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 2, No.1 (January 2011). g Figure 5. Face feature extraction: a) input face part, b) face part extraction, c)cropped face part, d)left eye part, e)right eye part, f)lip part, g) final feature part. [4] B. Fasel and J. Luettin. Automatic facial expression analysis: A survey. Pattern Recognition, 2003. [5] Ying-li Tian, Member,Takeo Kanade,Fellow, and Jeffrey F. Cohn, Member, “Recognizing Action Units for Facial Expression Analysis”, IEEE Transactions on pattern analysis and machine intelligence,2001. [6] Cheng-Chin Chiang ,Wen-Kai Tai,Mau-Tsuen Yang,Yi-Ting Huang,Chi-Jaung Huang “A novel method for detecting lips,eyes and faces in real time”, Elsevier 2003. [7] Rajesh A Patil, Vineet Sabula, A. S. MandaI ”Automatic Detection of Facial Feature Points in Image Sequences” , IEEE. [8] W.K. Teo , Liyanage C De Silva and Prahlad Vadakkepat , “FACIAL EXPRESSION DETECTION AND RECOGNITION SYSTEM”, Journal of The Institution of Engineers, Singapore Vol. 44 Issue 3 2004 Figure 6. GUI for classification [9] Yepeng Guan, “Robust Eye Detection from Facial Image based on Multi-cue Facial Information”, IEEE Guangzhou, China, 2007. TABLE I. ANN classification Expression Number of image Experimented Number of correct Recognition Success rate Happy 10 10 100% Sad 10 9 90% Surprise 10 10 100% Neutral 10 9 90% Total 40 38 95% [10] Le Hoang Thai, Nguyen Do Thai Nguyen and Tran Son Hai, Member, “A Facial Expression Classification System Integrating Canny, Principal Component Analysis and Artificial Neural Network”, International Journal of Machine Learning and Computing, Vol. 1, No. 4, October 2011 [11] Akshat Garg, Vishakha Choudhary, “FACIAL EXPRESSION RECOGNITION USING PRINCIPAL COMPONENT ANALYSIS”, International Journal of Scientific Research Engineering &Technology (IJSRET) Volume 1 Issue4 pp 039-042 July 2012. [12] Aryuanto Soetedjo “ Eye Detection Based-on Color and Shape Features” (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 3, No. 5.,2011. Implementation tools is used as Microsoft Visual C++ 6.0 This is installed on Windows 7 OS. CONCLUSION: [13] S.-H. Leung, S.-L. Wang, and W.-H. Lau, “Lip image segmentation using fuzzy clustering incorporating an elliptic shape function,” IEEE Transactions on Image Processing, vol. 13, no. 1, pp. 51–62, 2004. Faces are tangible projector panels of the mechanisms which govern our emotional and social behaviors. The automation of the entire process of facial expression recognition is, therefore, a highly intriguing problem, the solution to which would be 5