Face Emotion Detection by Neural Network
Alfiya Mahat
Student, SPWEC
Aurangabad, Maharashtra
alfiya704@gmail.com
Prof. Ganesh Sable
Professor, SPWEC
Aurangabad, Maharashtra
Sable.eesa@gmail.com
Abstract
Emotions are important part of human
communication.There has been a huge interest in
automatic recognition of human emotion because of
its wide spread application in security, surveillance,
marketing, advertisement, and human-computer
interaction. To communicate with a computer in a
natural way, it will be desirable to use more natural
modes of human communication based on voice,
gestures and facial expressions. In this paper,
consists of implementation of simple algorithm for
Face emotion detection. In spite of the most complex
algorithms developed, robust detection and location
of faces or their constituent is difficult to attain in
many cases. Several classifier based method have
been developed in this area but facial expression
recognition system remains a hot area of research
due to Environmental cluster illumination and
miscellaneous source of facial variability. In the
present work, four emotions namely angry, sad,
happy along with neutral is tested from user defined
database and real time database. The proposed
method uses virtual face model method for feature
extraction and Neural Networks for training and
classification.
Keywords
Adaptive Median filter, Virtual face model , Face
emotion detection, neural network, database.
1. Introduction
Biometric is the science and technology of
recording and authenticating identity using
physiological or behavioral characteristics of the
subject. A biometric representation of an individual,
it is a measurable characteristic, whether
physiological or behavioral, of a living organism that
can be used to differentiate that organism as an
individual. Biometric data is captured when the user
makes an attempt to be authenticated by the system.
This data is used by the biometric system for realtime comparison against biometric samples.
Biometrics offers the identity of an individual may be
viewed as the information associated with that person
in a particular identity management system [3]. There
are many ways that humans can express their
emotions. The most natural way to express emotions
is using facial expressions. A human can express
his/her emotion through lip and eye. A category of
emotions which universally developed by Ekmen are:
Dipali Rindhe
Student, SPWEC
Aurangabad, Maharashtra
dipalirindhe8943@gmail.com
sadness, angry, joy, fear, disgust and surprise without
consider natural emotion [2].
Humans are capable of producing thousands
of facial actions during communication that vary in
complexity, intensity and meaning. Emotion or
intention is often communicated by subtle changes in
one or several discrete features. The addition or
absence of one or more facial actions may alter its
interpretation.
2. Proposed Method
There are number of Algorithms for the face
emotion detection. But as seen, all have some
disadvantages or they have less accuracy. The
design and implementation of the Facial Expression
Recognition System can be subdivided into three
main parts. The first part is Pre-processing then
post-processing which includes feature extraction
and training of the images and the third part is
testing. The image processing part consists of image
acquisition of noisy image as input through scanning
The second part consists of the feature extraction by
VFM and training by Back Propagation Neural
Network[4].
Figure 1: Block diagram of face motion detection
system
Above figure 1 shows the block diagram of
face emotion detection system. This whole system is
divided into three steps. pre-processing, training and
testing.
2.1 Preprocessing
According to the need of the next level the
pre processing step convert the image. The
different steps in pre-processing stage are
shown below.
Step 1: RGB to gray conversion, Step 2: Adaptive
Median Filtering, Step 3: Histogram of Filtered
image.
Input Image of different emotion is read and
stored in variable in matrix form, and RGB to grey
conversion is required for images because the
processing takes place on gray scale image. This
conversion is carried out by eliminating hue and
saturation information while retaining the luminance
information of image.
(a)
(b)
Figure 2 (a) Input Image ,(b) RGB to Gray
Step 2: Adaptive Median Filtering
Filtering of noise and other artifacts in the
image and sharpening the edges in the image. It
includes adaptive median filter for noise removal.
But for the complete system it needs the process of
noise removal. For filtering a mask with center pixel
is rotated over image.
(a)
(b)
(c)
Figure 3 a) Input Image, b)RGB to Gray Image, c)
Filtered Image
Step 3: Histogram of Filtered Image
The Histogram block computes the
frequency distribution of the elements in the input.
Where, the following output shows the Histogram of
filtered image.
(a)
(b)
Figure 4: (a) Filtered Image (b) Histogram of Filtered
image
After taking the image for testing, it shows
the pre-processing output. Statistically analysis is
also done by that mean, median and standard
deviation of the input image, Filtered by adaptive
median filter and. The significance of these shows
that mean median and standard deviation should be
less in enhanced frame. These extracted features of
image are then fed into back propagation neural
network for the training and then emotions are
detected. Following table shows the mean, median
and standard deviation of the input image, filtered
image, also there PSNR value and the elapsed time.
Table 1 Preprocessing Output
Sr
No
1
Name of
person
Abu
2
Adiba
3
Alfiya
Mean
0.5449
0.5452
0.5881
0.5889
0.6343
0.6348
Media
n
0.5530
0.5524
0.6094
0.6111
0.6721
7614
Standa
rd
0.2271
0.2259
0.2169
0.2159
0.2710
0.2693
PSNR
72.766
Elapsed
Time
0.45760
66.109
0.47089
65.298
0.49934
2.2 Post Processing
The post processing is divided into two steps that is
Feature Extraction by virtual face model and training
by back propagation neural network.
a) Feature Extraction
For face emotion detection first extracts the whole
face from the given image after that features are
extracted from the selected face by using virtual face
model, then these features are given as the input for
the training of back propagation algorithm. From that
neural network classifies the output.
By using coordinate system of MATLAB
we are designed the different features. First we are
selecting the different part of face such as front face,
left eye, right eye, mouth and nose. After selecting
the face part we are assigning the coordinate system
of in terms of x and y coordinates of all features then
finding the absolute, for example calculating the
fourth feature point by taking the absolute of fourth y
coordinate of left eye and first y coordinate of left
eye, then we got the absolute value of these points
such as fe_pt4=abs(b4(2)-b1(2))/2, and for template
feature Tw, Th, and Tp are width, height, and the
number of pixel in template. After extracting the
features and finding the feature points from the
images of different emotions are fed to the back
propagation neural network for training.
(a)
(b)
Figure 5: (a) Input Image, (b) Output of Feature
Extraction
The feature extraction of image consists of
different features are selected from face such as left
eye, right eye, lips, nose, mouth and eye pair. Initially
the coordinates of all features are considered along
with whole face coordinated. In selection of
coordinates there is an X and Y coordinates are
selected of left eye, right eye, lips, nose, and mouth
and eye pair along with whole face. After selection of
coordinates mean of all features of face are
calculated.
Table 2(a) Output of Postprocessing
Input
imag
e
Abu
Adi
ba
Alfi
ya
fe_p
t1
fe_p
t2
fe_p
t3
fe_p
t4
fe_p
t5
fe_p
t6
fe_p
t7
fe_p
t8
71
71
25
17
17
4.5
51
24.5
74
74
10
6.5
6.5
28
7.50
24
75.5
75.5
22
15
15
7.5
54
25.5
input-output mappings. The aim is to train the
network to achieve a balance between the network’s
ability to respond and the ability to give a reasonable
response to the input that is similar, but not identical
to the one used in the training. The training of a back
propagation network involves the three stages. The
feed forward of the input training pattern, the
calculation and the back propagation of the
associated error and the weighted adjustment. A
weight in a neural network is a segment of the
information about the input signal that has to be
stored.
Table 2(b) Output of Postprocessing
Input
Image
fe_
pt9
fe_p
t10
fe_p
t11
Abu
Adiba
46.
50
50
30.5
0
33
40.5
0
41
Alfiya
50
27
48.5
0
fe_
pt1
2
18
fe_
pt1
3
25
20
10
16
22
fe_p
t14
fe_pt
15
21.5
0
25.5
0
24.5
0
21.5
0
25.5
0
24.5
0
fe_
pt1
6
6
(a)
(b)
9
1.5
00
b) Training of neural network
This part shows the results for training of
neural network. The Training of neural network is
done by using back propagation neural network. It
consists of a single frame of one subject which
includes total of eighty images of different emotions,
which consists of happy, angry, sad, and normal.
Two parameters epochs and errors are considered for
the detection of expression by face feature. Epochs
specify maximum number of iterations that are to be
taken for the detection of feature.
The back-propagation algorithm defines a
systematic way to update the synaptic weights of
multi-layer perceptron (MLP) networks. The
supervised learning is based on the gradient descent
method, minimizing the global error on the output
layer. The learning algorithm is performed in two
stage. feed-forward and feed- backward. In the first
phase the inputs are propagated through the layers of
processing elements, generating an output pattern in
response to the input pattern presented. In the second
phase, the errors calculated in the output layer are
then back propagated to the hidden layers where the
synaptic weights are updated to reduce the error. This
learning process is repeated until the output error
value, for all patterns in the training set, are below a
specified value. The Back Propagation, however, has
two major limitations: a very long training process,
with problems such as local minima and network
paralysis; and the restriction of learning only static
(c)
(d)
Figure 6(a) Neural network training state (b)
Training Regression (c) Training error histogram
(d) Training performance
2.3 Testing
The testing part consists of final step of
emotion detection For testing here we are considering
the four emotions such as happy, sad, angry and
neutral. For testing feature extraction of input image
is done. The different emotions are defined into
different classes. So the output of testing is obtained
in terms of class. This is the final step in that we got
the output. The values of final data feature test is
compare with final features of the input test image if
both are same then it shows the accurate output
otherwise it will not shows the proper output.
Input image
Table 4 Output of testing
Testing output
2.4 Database
For our experiments a real time database of
any person is used. The database consists of total 50
persons with of four emotions i.e. 50*4=200. In our
experiments, we have taken four emotions for a
single person from that number of pairs are trained in
the training phase that are trained with different
emotions. The images of original database which are
acquired from mobile camera and digital camera.
3. Sad: Following table 6 shows the confusion matrix
of sad emotion
Table 6 Confusion Matrix for sad
1
2
1
1
0
4
39
23
1
0
0
1
103
11
1
0
2
16
70
1
0
0
2
1
0
4. Angry:Following table 7 shows the confusion
matrix for the angry emotion.
Table 7 Confusion Matrix for Angry
1
2
2
0
0
0
41
25
1
0
0
1
108
7
0
0
0
27
62
0
0
0
2
1
0
3.2 Experimentation
For our experimentation we are tested the 50
images randomly taking from the database of any
emotion. Table shows the experimental result of face
emotion detection. Here, we are considering the
different parameter of neural network such as time,
number of iteration, performance, gradient, mean
square error and accuracy.
Table 8 Experimental Result of face emotion
detection
Figure. 7 Samples of input images from Database
3.1 Results
The result of this face emotion detection system
consists of testing of different facial images. So result
of this system is given with the help of confusion
matrixes of different emotions.
1. Happy: Following table shows the output of happy
emotion in terms of confusion matrix.
Table 4 Confusion Matrix for happy
2
1
2
0
0
0
14
25
1
0
0
41
98
4
0
0
1
45
43
0
0
0
0
3
0
2. Neutral: Following table shows the result of
neutral image in terms of the confusion matrix. It
drives the matrix of 5X5 dimension.
Table 5 Confusion Matrix for neutral
1
1
3
0
0
2
40
24
1
0
0
3
103
10
0
0
0
21
66
2
0
0
0
3
0
Name
of
person
Epochs
(Iterati
on)
Time
(sec)
Performa
nce
MSE
Gradi
ent
Accuracy
Abu
10
01
0.109
0.100
0.109
78.57%
Adiba
11
01
0.091
0.010
0.101
81.90%
Afliy
12
04
0.111
0.100
0.213
79.50%
The Overall Accuracy is calculated by testing of 50
images. In that average accuracy of all 50 images are
calculated. So we have got the exact accuracy for the
system.
Accuracy = ∑ Neural Network accuracy
50
=78.27%
3.3 System requirements
Table 9 consist of system requirements. It
shows the MATLAB version, types of database,
processor so that it will easy for the testing.
Parameter
Processor
System
Type
Table 9 System Requirement
Description
Intel(R) Core(TM) i3-3227U CPU
1.73GHz
32- bit operating system, celeronbased processor
Database
Facial images of happy, sad, angry and
normal emotions of 50 persons i.e. total
50*4=200 images in jpg format.
Tools
MATLAB/ R2012a
Algorithm
Back propagation Algorithm
4.1 Conclusion
From this work, it can be concluded that the
Back propagation Neural Network is used for
emotion detection in two steps such as training and
testing. In physical biometrics generally parameters
like fingerprint, palmprint iris and retina but face also
a good biometric solution for identification. The
proposed system requires less computational time
and an automated approach is used to extraction and
detection. In this study a front face is extracted then a
multiple feature such as eyes and lips and nose are
considered in terms of the coordinate values of these
features. This technique can be used as robust face
emotion detection algorithm. The average efficiency
of this algorithm is 78.27% in case of large database
emotion detection. Classification of emotions such as
sad, Angry, Normal and Happy are considered by
using back propagation neural network.
REFERENCES
[1] r. Ezhilarasi, r. I. Minu “automatic emotion
recognition and classification”proceeding of
international conference on modelling, optimisation
and computing (icmoc – 2012)
[2] A. Habibizad navin, Mir Kamal Mirnia, “A New
Algorithm to Classify Face Emotions through Eye
and Lip Features by Using Particle Swarm
Optimization”. 2012 4th International Conference on
Computer Modeling and Simulation (ICCMS 2012).
[3] Renu Nagpal, Pooja Nagpal ,Sumeet Kaur
“Hybrid Technique for Human Face Emotion
Detection”, Proceeding on (IJACSA) International
Journal of Advanced Computer Science and
Applications,Vol. 1, No. 6, December 2010
[4] H.S.M. Al-Khaffaf , A.Z. Talib, R. Abdul Salam,
“A Study on the Effects of noise level, cleaning
method, and vectorization software on the Quality of
vector data”, Lecture Notes Computer Science 299309.
[5] L. Beaurepaire, K.Chehdi, B.Vozel (1997),
“Identification of the nature Of the noise and
estimation of its statistical parameters by analysis of
Local histograms”, Proceedings of ICASSP-97,
Munich, 1997.
[6] J. Y. Chang and J. L. Chen, “A facial Expression
recognition system using neural
Networks,” Journal of IEEE, pp. 3511-3516, 1999.
[7]Meng Joo,Shiqian Wu,Juwei Lu,Hock Lye
Toh(2002),”Face Recognition with Radial Basis
Function Neural Networks”,IEEE Transactions on
Neural Networks,Vol. 13,No.3
[8] P.Y. Simard, H.S. Malvar (2004), “An efficient
binary image activity Detector based on connected
components”, Proceedings of. IEEE International
Conference on Acoustics, Speech, and Signal
Processing, 229–232, 2004.
[9] Thaahirah S.M. Rasied, Othman O. Khalifa and
Yuslina Binti Kamarudin, “Human Face Recognition
based on singular Valued Decomposition and Neural
Network”, GVIP 05 Conference, 19-21 December
2005, CICC, Cairo, Egypt.
[10]Rizon, M., Yaacob, S., Desa H., Karthigayan M.,
Hashim F., Saad P., Shakaff, A., Saad, A., Mamat M.
(2006),” Face Recognition using Eigenfaces and
Neural Networks”.
American Journal of Applied Sciences 2 (6): 18721875, 2006.
[11]Praseeda Lekshmi.V and M. Sasikumar (2008),
“A Neural Network Based Facial Expression
Analysis using Gabor Wavelet,” World Academy of
Science, Engineering and Technology.
[12] Sarawat Anam,Md. Shohidul Islam, M.A.
Kashem,
M.N.Islam,
M.R.Islam
and
M.S.Islam(2009), “Face Recognition using Genetic
Algorithm
and
Back
Propagation
Neural
Network”,Proceedings
of
the
International
Multiconference of Engineers and Computer Scientist
2009.Vol.1.IMCSE 2009.
[13]Ruba Soundar Kathavarayan and Murugesan
Karuppasamy,(2010)“Preserving Global and Local
Features for Robust Face Recognition under Various
Noisy Environments” , International Journal of
Image Processing (IJIP) Volume(3), Issue(6)
[14]Jagdish Lal Raheja, Umesh Kumar ,(2010),
“Human Facial Expression Detection from detected
in Captured image using Back Propagation Neural
Network”, International Journal of Computer Science
and
Information
Technology(IJCSIT),Vol
2,No.1,February 2010.
[15] G.Sofia ,M. Mohamed Sathik ,(2010)“An
Iterative approach For Mouth Extraction In Facial
Expression Recognition”, Proceedings of the Int.
Conf. On Information Science and Applications
ICISA 2010 6 February 2010, Chennai, India.
[16] C.R Vimal chand (2010), “Face and gender
Recognition Using Genetic Algorithm and Hopfield
Neural Network”, Global Journal of Computer