[1] Bartlett, M.S., Littlewort, G.C., Lainscsek, C., Fasel, I., Frank, M.G., Movellan, J.R., “Fully automatic facial action
recognition in spontaneous behavior”, In 7th International Conference on Automatic Face and Gesture Recognition, 2006,
p. 223-228.
Bartlett et al. [1] proposed a system that can automatically recognize frontal views of the image from a video stream,
in which 20 AUs are detected for each frame. Here context-independent training has been used. One binary classifier
has been used for each of the 20 AUs. These classifiers have been trained to recognize the occurrence of AUs
regardless of their co-occurrence (a specific AU occurring alone or with others). They also compared the
performance analysis of AdaBoost and linear SVM. Paired t-tests showed a slight advantage of AdaBoost over
linear SVM. One interesting feature that the authors tried to measure is the intensity of specific AUs. They used the
output margin of the system which actually describes the distance to the separating hyperplane as an interpretation
for the intensity of the AU.
[2]Braathen, B., Bartlett, M.S., Littlewort-Ford, G., Smith, E. and Movellan, J.R. (2002). An approach to automatic
recognition of spontaneous facial actions. Fifth International Conference on automatic face and gesture recognition, pg.
231-235.
[3] F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. H. Salesin, “Synthesizing realistic facial expressions from
photographs”, Computer Graphics, 32(Annual ConferenceSeries):75–84, 1998.
Braathan et al. [2] address a natural problem with image collection and shift the paradigm from 2D to 3D facial
images. Older automated facial expression recognition systems have relied on posed images where images clearly
show the frontal view of the face. But this is impractical. Many times the head is in an out-of-image-plane (turned or
nodded) in spontaneous facial images. Braathan et al. tackled three issues sequentially in their project. First the face
geometry was estimated. Thirty images were taken for each subject and the position of eight special features was
identified (ear lobes, lateral and nasal corners of the eyes, nose tip, and base of the center upper teeth). 3D locations
of these eight features were then recovered. These eight 3D location points were fitted in the canonical view of the
subject. Later a scattered data interpolation technique [3] was used to generate and fit the other unknown points in
the face model. Second, a 3D pose estimation technique known as Markov Chain Monte-Carlo method or particle
filtering method was used. This generated a sequence of 3D poses of the head. Canonical face geometry was then
used to warp these images on to a face model and rotate these into a frontal view. Then this image was projected
back to the image plane. Finally, Braatthan et al. used Support Vector Machines (SVM) and Hidden Markov Models
(HMM) for the training and learning of their spontaneous facial expression detection system. Since normally the
head is in a slanted position during severe pain, this system is especially useful for analyzing real time painful
expressions.
[4] Jagdish Lal Raheja, Umesh Kumar, “Human facial expression detection from detected in captured image using back
propagation neural network”, In International Journal of Computer Science & Information Technology (IJCSIT), Vol. 2,
No. 1, Feb 2010,116-123.
[5] Paul Viola, Michael Jones, ”Rapid Object Detection using a Boosted Cascade of Simple features”, Conference on
computer vision and pattern recognition, 2001.
Jagdish and Umesh proposed a simple architecture for facial expression recognition based on token finding and
standard error-based back propagation neural network [48]. After capturing the image from a webcam, they used the
face detection technique devised by Viola and Jones [11]. In order to process the image, histogram equalization was
done to enhance image quality followed by edge detection and thinning. Tokens, which denote the smallest unit of
information, were generated from the resultant image and passed into the neural network. The network was trained
with 100 samples and could classify three expressions. The provided report did not say anything about the number
of nodes in the input layer and hidden layer of the network. It also did not mention about the training samples.
[6] A. B. Ashraf, S. Lucey, J. F. Cohn, T. Chen, K. M. Prkachin, and P. E. Solomon. The painful face II-- Pain expression
recognition using active appearance models. International Journal of Image and Vision Computing, 27(12):1788-1796,
November 2009.
Active Appearance Model (AAM) [6] has been proposed as an innovative way for pain recognition from facial
expressions. AAMs were used to develop the automated machine learning-based system. Use of a Support Vector
Machine (SVM) and leave-one-out procedures lead to a hit rate of 81%. The main advantage of AAMs is the feature
of decoupling appearance and shape parameters from facial images. In AAM, shape s is expressed as a 2D
triangulated mesh. Location of the mesh vertices is related to the original image from which the shape s was derived.
In AAMs, a shape s can be expressed as a combination of a base shape s0 and a set of shape vectors si . Three AAMderived representations are pointed: Similarity Normalized Shape (sn), Similarity Normalized Appearance (an), and
Shape Normalized Appearance (a0). The authors developed their own representations using AAM-derived
representations that they used in painful face detection. Temporal information was not used in recognition of pain
which may increase the accuracy rate.
[7] Md. Maruf Monwar, Siamak Rezaei and Dr. Ken Prkachin, “Eigenimage Based Pain Expression Recognition”, In
IAENG International Journal of Applied Mathematics, 36:2, IJAM_36_2_1. (online vversion available 24 May 2007)
[8] Md. Maruf Monwar, Siamak Rezaei: Appearance-based Pain Recognition from Video Sequences. IJCNN 2006: 24292434
Monwar et al. proposed an automatic pain expression detection system using Eigenimage [7]. Skin color modeling
was used to detect the face from video sequence. Mask image technique was used to extract the appropriate portion
of the face for detecting pain. Each resultant image from masking was projected into a feature space to form an
Eigenspace based on training samples. When a new image arrived, its position in the feature space was compared
with that of the training samples and based on that a decision is drawn (pain or no pain). For this experiment, 38
subjects of different ethnicity, age, and gender were videotaped for two expressions – normal and painful. First the
chromatic color space was used to find the distribution of skin color. Later the Gaussian model was used to represent
this distribution. Then the probability of a pixel being skin was obtained. After segmenting the skin region, a
meaningful portion of the face was detected using mask image and later a bitwise ‘AND’ operation was used with
the mask image and original image to develop the resultant image. These resultant images were used as samples for
training the Eigenfaces method and M number of Eigenfaces with highest Eigenvalues being sorted out. When
detecting a new face, the facial image was projected in the Eigenspace, and the Euclidian distance between the new
face and all the faces in the Eigenspace was measured. The face that represents the closest distance was assumed to
be a match for the new image. Average hit rate was recorded to be 90-92%. Later, the researchers extended their
model [8] to create two more feature spaces – Eigeneyes and Eigenlips. Portions of eyes and lips were used from
facial images for Eigeneyes and Eigenlips methods. All possible combinations of Eigenfaces, Eigeneyes, and
Eigenlips (alone or in combination with others) were used to find pain from images. A combination of all three
together provided the best result in terms of accuracy (92.08%). Here skin pixels have been sorted out using
chromatic color space. However skin color varies a lot based on race and, ethnicity, and detailed elaboration of the
subjects’ skin colors is missing in these studies.
[9] Mayank Agarwal, Nikunj Jain, Manish Kumar, and Himanshu Agrawal, “Face recognition using principle component
analysis,eigneface, and neural network”, In International Conference on Signal Acquisition and Processing, ICSAP, 310314.
Here the authors proposed a methodology for face recognition based on information theory. Principle component
analysis and feed forward back propagation Neural Network has been used for feature extraction and face
recognition respectively. The algorithm has been used on 400 images of 40 different people taken form Olivetti and
Oracle Research Laboratory (ORL) face database. Training dataset is composed of 60% of the images and the rest
are left for test database. The Artificial Neural Network (ANN) had 3 layers – input, output, and one hidden layer.
Though this method achieved 97% accuracy this had two major drawbacks. The method incorporates one ANN for
each person in the face database which is impractical in terms of scalability. Another issue is they have tested only
with different images of the same person whose picture has been used in the training of the ANN which is not
practical for a real life scenario.
[10] Murthy, G. R. S. and Jadon, R. S. (2009). Effectiveness of eigenspaces for facial expression recognition. International
Journal of Computer Theory and Engineering, Vol. 1, No. 5, pp. 638-642.
Here the authors proposed a modified methodology for eigenface based facial expression recognition. The authors
formed a separate eigenspace for each of the six basic emotions from the training images. When a new image comes,
the new image is projected in each of the eigenspaces and a reconstructed image is formed from each eigenspaces. A
classification decision is taken by measuring the mean squeare error between the input image and the reconstructed
image. Using Cohn-Kanade and JAFFE (Japanese Female Facial Expression) database, the method received a
maximum of 83% accuracy for ‘happiness’ and 72% accuracy for ‘disgust’. The paper does not say anything about
facial contour selection. The classification accuracy rate is also quite low.
[11] Singh. S. K., Chauhan D. S., Vatsa M., and Singh R. (2003). A robust skin color based face detection algorithm.
Tamkang Journal of Science and Engineering, Vol. 6, No. 4, pp. 227-234.
Here the authors compared algorithms based on 3 different color spaces RGB, YCbCr, and HSI. Later they
combined them to derive a modified skin color based face detection algorithm. An algorithm based on the venn
diagram of set theory has been used to detect the skin color. A face portion from the original image is taken apart by
detecting 3 points – midpoint of eyes and leaps. This method showed 95% accuracy rate on IITK database. But this
method only detects the facial portion from any image. It does not say anything about expression classification.
[12] Rimé, B., Finkenauera, C., Lumineta, O., Zecha, E., and Philippot, P. 1998. Social Sharing of Emotion: New Evidence
and New Questions. In European Review of Social Psychology, Volume 9.
We have also done an analysis on the mood related applications of Facebook. Rimé et al. [12] argued that everyday
life emotional experiences create an urge for social sharing. Authors also showed that most emotional experiences
are shared with others shortly after they occurred. These research findings show that mood sharing can be an
important area in social networks. At present there are many applications in Facebook which claim to do mood
detection for the user. Here, we have listed top 10 such applications based on the number of active users.
Name
of
Application
My Mood
SpongeBob Mood
The Mood Weather
Report
Name and Mood
Analyzer
Mood Stones
My Friend's Mood
My Mood Today!
How's your mood
today?
Your Mood of the
day
Patrick’s Mood
No. of users
1,822,571
646,426
Mood Categories
56
42
611,132
174
29,803
14,092
15,965
11,224
13
9
6,694
39
4,349
3329
15
Table 1: Mood applications in Facebook
Here user chooses a mood category (symbol) that he/she thinks is the most suitable one. Once the choice is made by
the user, the application gives him the option of sharing the mood with his friends on Facebook. The mood is then
displayed on user’s wall. It is encouraging to see that a large number of users are interested in using the mood
related application. This is the case when users are manually selecting one of the mood categories. If we give them
the mood detection application which works on the principle of image processing and then gives the context aware
results instead of just manually selecting one of the categories, it will really prove revolutionary.