CONCEPT DESIGN
OUR APPROACH
Development of the architecture can be classified in three broad categories. In the first phase the facial portion
has to be extracted out from an image. Then in the second phase the extracted image has to be analyzed for facial
features and classification follows to identify one of the several mood categories. Finally we need to integrate the
mobile application with Facebook. Figure 1 depicts the overview of the architecture.
Figure 1. Architecture of EMS
Face Detection
Pixels corresponding to skin have difference with other pixels in an image. Skin color modeling in chromatic
color space [5] has shown the clustering of skin pixels in a specific region. Though the skin color of persons vary
widely based on different ethnicity, research [4] shows that still they form a cluster in the chromatic color space.
After taking the image of the subject we first crop the image and take only the head portion of the image. Then we
use skin color modeling for extracting the required facial portion from the head image.
Facial Expression Detection
For this part we plan to use a combination of Eigenfaces, Eigeneyes, and Eigenlips methods based on Principal
Component Analysis (PCA) [6,7]. This analysis method includes only the characteristic features of the face
corresponding to a specific facial expression and leaves other features. This strategy reduces the amount of training
sample and helps us to make our system computationally inexpensive. These resultant images will be used as
samples for training Eigenfaces method and M Eigenfaces with highest Eigenvalues will be sorted out. When
detecting a new face, the facial image will be projected in the Eigenspace and the Euclidian distance between the
new face and all the faces in the Eigenspace will be measured. The face that represents the closest distance will be
assumed as a match for the new image. Similar process will be followed for finding results using Eigenlips and
Eigeneyes methods. Here is a step by step break down of the whole process.
1. The first step is to obtain a set S with M face images. Each image is transformed into a vector of size N2 and
placed into the set, S={Г1,Г2,Г3.. ,ГM}
2. Second step is to obtain the mean image Ψ
𝑀
1
Ψ = ∑ Г𝑛
𝑀
𝑛=1
3. We find the difference Φ between the input image and the mean image, Φi= Гi- Ψ
4. Next we seek a set of M orthonormal vectors, uM, which best describes the distribution of the data. The kth vector,
uk, is chosen such that
𝑀
1
λ𝑘 = ∑ (𝑢𝑘𝑇 Φ𝑛 )2
𝑀
𝑛=1
5. λk is a maximum, subject to
𝑢𝑙𝑇 u𝑘 = δ𝑙𝑘 =
1 , if l  k

0 , otherwise
where uk and λk are the eigenvectors and eigenvalues of the covariance matrix C
6. The covariance matrix C has been obtained in the following manner Ω
1
𝑇
𝑇
C = ∑𝑀
𝑛=1 Φ𝑛 Φ𝑛 = 𝐴𝐴 Where, 𝐴 = [Φ1 Φ2 Φ3 . . Φ𝑚 ]
𝑀
7. To find eigenvectors from the covariance matrix is a huge computational task. Since M is far less than N 2 by N2,
we can construct the M by M matrix,
L = 𝐴𝑇 𝐴, where L𝑚𝑛 = Φ𝑇𝑚 Φ𝑛
8. We find the M Eigenvectors, vl of L.
9. These vectors (vl) determine linear combinations of the M training set face images to form the Eigenfaces u l
𝑀
u𝑙 = ∑
𝑘=1
v𝑙𝑘 Φ𝑘 ,
l = 1,2, … , M
10. After computing the Eigenvectors and Eigenvalues on the covariance matrix of the training images


M eigenvectors are sorted in order of descending Eigenvalues
Some top eigenvectors are chosen to represent Eigenspace
11. Project each of the original images into Eigenspace to find a vector of weights representing the contribution of
each Eigenface to the reconstruction of the given image.
When detecting a new face, the facial image will be projected in the Eigenspace and the Euclidian distance between
the new face and all the faces in the Eigenspace will be measured. The face that represents the closest distance will
be assumed as a match for the new image. Similar process will be followed for Eigenlips and Eigeneyes methods.
The mathematical steps are as followed:
 Any new image is projected into Eigenspace and find the face-key
ω𝑘 = 𝑢𝑘𝑇 (Г − Ψ)𝑎𝑛𝑑 Ω𝑇 = [ω1 , ω2 , … . , ω𝑀 ]
where, uk is the kth eigenvector and ωk is the kth weight in the weight vector Ω𝑇 = [ω1 , ω2 , … . , ω𝑀 ]
 The M weights represent the contribution of each respective Eigenfaces. The vector Ω, is taken as the ‘facekey’ for a face’s image projected into Eigenspace.
 We compare any two ‘face-keys’ by a simple Euclidean distance measure
€ = ‖Ω𝑎 − Ω𝑏 ‖2
 An acceptance (the two face images match) or rejection (the two images do not match) is determined by
applying a threshold.
Integration to Facebook
The application uses device camera to capture facial images of the user and recognize and report their mood. The
mobile version, for iPhone and Android powered devices, uses the devices’ built in camera to capture images and
transfer the image to a server. The server extracts the face, the facial feature points, and classifies the mood using a
classifier, and sends the recognized mood back to the mobile application. The mobile application can connect to
Facebook allowing the user to publish their recognized mood.