Eigen Value Based Facial Emotion Classification
Sheily Verma, Vilas H. Gaidhane & Asha Rani
ICE Division, Netaji Subhas Institute of Technology, University of Delhi, Sector-3
Dwarka, New Delhi, 11078, India
E-mail : er.sheily15@gmail.com, vilasgla98@gmail.com, ashansit@gmail.com
Abstract – One of the most important way for human to display emotion is through facial expression. Recognition of facial emotions
represent an important aspect of interpersonal communication. This paper proposes a digital hardware system with field
programmable gate array (FPGA) for facial emotion classification which makes use of power method algorithm used to calculate the
Eigen value of a matrix. The work describes a real time automatic facial expression recognition system using webcam input. The
digital hardware is designed for emotion recognition system using verilog and implemented on Xilinx Spartan 3E FPGA. The
presented algorithm has less mathematical calculation and complexity therefore the recognition is fast.
Keywords – Power method algorithm, Emotion recognition, Eigen value, Verilog, FPGA
I.
are complex and cannot be simply matched. Most of the
time it is a combination of six basic emotion known as
Happy, Surprise, Angry, Sad, Fear, Disgust. In this
paper, the concept of power method is used and
implemented for such applications on a computer
software but not very time efficient due to additional
constraints of memory. Hardware implementation offers
much greater speed than software implementation hence
implementing on hardware become an attractive
alternative. Here the image requires less storage space
because only one of the features is stored in the
memory. In the present work, digital hardware
architecture for real time emotion recognition system is
implemented. The emotion recognition system is
designed by Verilog and implemented using FPGA.
INTRODUCTION
In the field of image processing it is very interesting
to recognize human gesture. Nowadays it has become an
active research topic, many theories on emotion
recognition have been proposed but there is a little
agreement on definition of emotions. In general it is
defined as changes in physiological signals, Emotion is
a response to a particular situation. It is an integral part
of our existence. Automatic emotion recognition is
considered as one of the important task in computer
vision, security, law enforcement, clinic, education,
psychiatry and telecommunication [1].
Since the early 1970, Paul Ekman and his
colleagues have performed extensive studies on human
facial expressions. They developed the well known
facial action coding system (FACS) for facial expression
description. They found evidence to support universality
in facial expressions. These "universal facial expression
are those representing happiness, sadness, anger, fear,
surprise and disgust". Ekman work inspired many
researchers to analyze facial expressions by means of
image and video processing. Researchers nowadays
focus on development of machine emotion recognition
algorithm and their implementation using FPGA.
II. EMOTION RECOGNITION-AN OVERVIEW
In psychology of emotions facial expressions play
an important role. Emotion recognition is a process of
matching human emotion to its database. It is a process
of classifying different emotions of a particular image
[4]. The main factor used in building a facial expression
recognition system is face detection, alignment, image
normalization, feature extraction and classification. The
approach of this system can be adapted to real time.
This paper introduces an effective emotion
recognition algorithm by making use of power method
generally used to calculate the eigen value. Emotion
classification or recognition is a process that
demonstrate and convey a lot of evident information
visually rather than verbally [2] [3]. Human emotions
Several approaches have been taken in the literature
for learning classifier for emotion recognition. In the
static approach the classifier defines each frame in the
video to one of the facial expression categories. Bayesian
network classifiers are commonly used in this approach,
Naive bayes classifier and Gaussian classifier are also
ISSN (Print) : 2319 – 2526, Volume-2, Issue-5, 2013
151
International Journal on Advanced Computer Theory and Engineering (IJACTE)
v and  , respectively.
lim k  X k   and
used often [7] [8]. In the dynamic approach the classifier
try to capture the temporal pattern in displaying facial
expressions, and Hidden Markov model (HMM) is used
in this approach.
The largest Eigen value is calculated using above
steps and the algorithm is implemented on FPGA. Since
a big image of 255×255 cannot be stored in FPGA at a
time due to small memory space, the image is converted
in the Eigen vale as a feature of image. The above
algorithm is selected for this purpose.
Neural networks make use of back propagation
method. They first extract some features from the images
than these features are used as input into a classification
system. These methods are complicated and time
consuming hence to reduce complexity the concept of
power method is used, which is generally applied in
control system for Eigen value calculations. The general
emotion recognition system is given in figure 1.
IV. DATA PREPARATION
Data preparation is an important phase since the
prepared dataset becomes input to the verilog training
and testing. Once the image has been acquired and
extracted using webcam the image processing
techniques are needed for image processing. First of all
image filtering is performed in preprocessing of an
image, then the gray scale transformation is performed.
Threshold technique is used to convert an intensity
image to a text image. All these phases are completed
using MATLAB toolbox. The next phase comprises of
Emotion recognition algorithm. The steps taken are
summarized as follows:
Image
acquisition
Emotion
Eigen value
dataset
Preprocessing
Eigen
value face
extraction
Largest
Eigen
value
Compare
Eigen
value
Emotion
classification
Step 1: Image is acquired in the form of text image.
Step 2: Image Cropped to16×16 pixel matrix. Each pixel
is in the range of 0-255
Fig. 1: The Emotion Recognition System
Step 3: Power method is applied and implemented on
verilog to calculate the largest Eigen value.
III. EMOTION RECOGNITION ALGORITHM
Emotion Recognition can be achieved by
calculating the highest Eigen value. In this approach the
Eigen vector corresponding to dominant (largest in
absolute value) Eigen value is calculated. Each of the
image is available in pixel matrix form. Here the system
matches the highest Eigen value of the input image and
the reference image. The highest Eigen value is
calculated by power method. Suppose matrix A has
dominant Eigen value   and that there is a unique
normalized Eigen vector  that corresponds to  . This
Eigen pair  , v can be obtained by the following
iterative procedure called power method [9]. The steps
used in power iterations are summarized as follows:
Step 4: Simulation is performed.
Step5:The file (.bit file) is generated to observe the
simulation results are acquired.
Step6: This file is stored on FPGA ROM.
Step7:The emotion is recognized corresponding to the
simulated Eigen value
V. SIMULATION RESULTS
The highest Eigen value is calculated both by the
MATLAB software as well as from Verilog making use
of power method algorithm. It is observed from the
results that Eigen values calculated from both the
methods are quite similar. It is also observed that Verilog
provides the results earlier as compared to MATLAB.
Figure 2 represents the Eigen value calculated by
MATLAB software.
Step 1: Choose a random vector X 0  1,1,1,,1 R n
Step 2: Calculate Yk  AX k
Step 3: Calculate X k 
1
Yk
C k 1
limk  Ck  

 
where, C k 1  X jk , and X jk   max 1  j  n X jk 

Step 4: The sequences X k  and C k 1 will converge to
ISSN (Print) : 2319 – 2526, Volume-2, Issue-5, 2013
152
International Journal on Advanced Computer Theory and Engineering (IJACTE)
Fig. 2 : Eigen value of different Emotions
Emotions
Eigen
value
A face image shown in Fig.2 is considered for the
experimentation. MATLAB software is used for
preparing the database. The image is being read in
MATLAB from which the pixels are generated then
these pixels are used as inputs to the verilog on which
power method algorithm is applied. The values which are
generated can have a slight difference and are based on
the processor. Pixels are transferred to (UART) universal
asynchronous receiver transmitter serial communication
which is used for communication between software and
hardware (FPGA). Xilinx ISE 13.4 has been used as a
Verilog simulation and synthesis tool to test the design.
Happy
0.88526
Angry
0.87546
Sad
0.89295
Fig. 5 : Simulation result for sad image
A comparative study is performed between
MATLAB and Verilog. It observed from the simulation
results that Verilog is much faster as compared
MATLAB. The decision is made on the basis of
comparison of calculated largest Eigen value stored in
database. Implementation of algorithm is on FPGA
platform as FPGA based systems are far more practical,
mobile and can be easily integrated onto other systems.
The FPGA implementation of Emotion recognition
system is realized using FPGA kit (Spartan 3E) with the
Xilinx ISE integrated design environment (IDE)v13.4.
VI. HARDWARE DESCRIPTION
The algorithm is implemented on FPGA kit. The
board used is Nexys2 circuit board which is based on
Xilinx Spartan 3E FPGA. It's on board high speed USB
(Universal serial bus) 2 port, 16Mbytes of RAM and
ROM and several input-output devices and ports make it
an ideal platform for digital systems of all kinds. The
FPGA on the Nexys2 board must be configured. During
configuration a bit file is transferred into memory cells
within the FPGA. The FPGA can be programmed in two
ways: directly from a PC using the on board platform
flash ROM or through the USB port. A jumper on the
Nexys2 board determines which source the FPGA will
use to load its configuration. The FPGA will
automatically load a configuration from the platform
flash ROM. Eight LED (Light emitting diode) are
provided for circuit output.
Fig.3 : Simulation result for happy image
VII. CONCLUSION
Fig. 4 : Simulation result for angry image
In this paper, the power method algorithm is used
for Emotion recognition which is implemented on Xilinx
Spartan 3E FPGA kit. The benefits of FPGA based
system are in terms of size, mobility, flexibility,
configurability and has been utilized in this work. The
largest Eigen value is obtained using power method and
it is compared with the desired Eigen value stored in the
dataset. It is concluded that the simulated results both in
MATLAB as well as in Verilog for Emotions like happy,
angry, sad are quite similar. However the Verilog
provides the results faster as compared to MATLAB.
ISSN (Print) : 2319 – 2526, Volume-2, Issue-5, 2013
153
International Journal on Advanced Computer Theory and Engineering (IJACTE)
REFERENCES
[1]
S. Morishima and H. Harashima, “Emotion space
for analysis and synthesis of facial expression,”
IEEE Trans. Pattern Anal. Mach . Intell., vol.15,
no.10 , pp. 188-193 , 1993.
[2]
L. Leon, G. Clarke, F. Sepulveda, and V.
Callaghan, “Optimized attribute selection for
emotion classification using physiological
signals,” in Proc. IEEE Int. Conf. Autom. Face
Gesture Recog., pp. 184-187, Mar. 2005.
[3]
G. Donato, M. Bartlett, J. Hager, P. Ekman, and
T. Sejnowski “Classifying facial actions,” IEEE
Trans. Pattern Anal. Mach. Intell ., vol.21, no.10,
pp. 974-989 , Oct.1999.
[4]
G. Littlewort, J. Whitehill, T. Wu, I. Fasel, M.
Frank, J. Movellan, and M. Bartlett, “Computer
expression recognition toolbox,” in Proc. IEEE
Int. Conf. Autom. Face Gesture Recog., pp. 298–
305, Mar. 2011.
[5]
K.N Chia, H. J Kim, Shane Lansing ,W. H. Smith
and John Villa senor ,“ High performance
Automatic Target Recognition Through Data
Specific VLSI," IEEE transactions on very large
scale integration (VLSI) systems, vol 6, no.3, pp.
364-371 , September 1998.
[6]
Y.Yacoob and L.S. Davis , “Recognizing human
facial expression from long image sequences
using optical flow," IEEE Transactions on
pattern Analysis and machine intelligence , pp
636-642, June 1996.
[7]
P. Ekman, and E.L Rosenberg , “What the face
Reveals: Basic and applied studies of spontenous
expression using the facial coding system
(FACS)," New York: Oxford University Press ,
1997.
[8]
L. Ma and K. Khorsani, “Facial expression
recognition using constructive feed forward
neural networks," IEEE Trans. Pattern Anal .
Mach . Intell., vol.34, no. 3, pp.1588-1595, 2004.
[9]
A. Dhall, A. Asthana , R. Goecke , and T. Gedon
, “Emotion recognition using LPQ features,” in
proc. IEEE Int. Conf. Autom. Face Gesture
Recog. Workshop facial expression Recog. Anal.
Challenge , pp.878-883, Mar.2011.

ISSN (Print) : 2319 – 2526, Volume-2, Issue-5, 2013
154