International Journal of Engineering Trends and Technology (IJETT) – Volume 29 Number 3 - November 2015
Driver Drowsiness Detection System Using Automatic
Facial Gestures
Regina Antony#1, G. S. Ajith*2
#P. G Scholar & Computer Science & Engineering & Amal Jyothi College of Engineering Kanjirapally, Erumely Rd,
Koovapally, Kerala, 686518,India
*Assistant Professor, Amal Jyothi College of Engineering, Kanjirappally, Erumely Rd, Koovapally, Kerala, 686518 India
Abstract-Drowsy driverdetection and recognize
drivers state with high performance is the objectiveof
this work. Drowsy driving is one of the main reasons
of traffic accidentsin which many people die or get
injured. Drowsy driver detection methodsare divided
into two main groups: methods focusing on drivers
performanceand methods focusing on drivers
state.Furthermore, methods focusing on drivers state
are divided into two groups: first identify whether the
driver is drowsy or not by processing the
facialexpression of the driver and comparing the best
classification method. Here, driver data are video
segments captured by a camera. There are twomain
states of a driver, those are alert and drowsy states.
Video segmentscaptured are analysed by making use
of image processing techniques.Thisuses various
images of driver to detect drowsiness states using
his/her eyesstates, mouth state and head poses.
Keywords—Driver fatigue, Drowsiness detection, Invehicle monitoring, Driver warning system.
I.
INTRODUCTION
The increasing growth of population leads to the
increasing number of vehicles in the road. As a result
the number of vehicle accidents also increases.
Detailed studies shows that around half million
accidents occur in a year, and thousands of people die
in this accidents in India alone. Driver drowsiness is
one of the main reasons for these traffic accidents. The
reports of national sleep foundation shows that 60% of
adult drivers drive while feeling drowsy and 37% have
even actually fallen asleep during driving [1]. The
following are the signs shown by a drowsy driver:
•
Driver cannot keep eyes open
•
Frequent yawning
•
Unable to keeping head up
•
Driver is hurry and impatient
Different statistics in different countries were
reported that accidents happened mainly due to driver
fatigue and distraction. Driver drowsiness and lack of
attention are the main reason of about 30% of crashes
and 20% of deaths [1]. The only method for
controlling these accidents is development of
technologies for drowsiness detection and prevention.
There are two main approaches for detecting driver
drowsiness. First and the commonly used method are
video recognition techniques using camera images. By
using this approach the images captured by camera is
ISSN: 2231-5381
analysed and detect the physical changes in drivers,
such as eyelid Movement, eye gaze, yawning and head
nodding. The second one is the measurement of
biomedical signals. This can give significant
information such as fatigue in addition to drowsiness
of the driver.
Monitoring the brain activity is the best method
for measuring the drowsiness. But in this method, the
brain Activity is measured using an electrode that is
placed in the Head of the driver that makes this an
intrusive approach. Another significant feature where
symptoms of drowsiness appeared is eye. There is a
close relation between drowsiness and the percentage
of eyelid closure. This paper mainly focuses on the
driver face monitoring system that investigates the
driver physical condition based on the processing of
drivers eye, head and mouth images.
II. PREVIOUS WORKS
Processing the face region is the best method for
detecting whether a person is drowsy or not. There are
many researches based on this particular method. The
reason for this is that the first symptoms of drowsiness
and distraction are appearing in the face of the driver.
Face detection is the foremost part in the
drowsiness detection system [2]. There are mainly two
categories for face detection: (1) feature-based and (2)
learning-based methods [2].By applying certain
heuristic rules on features, the face in the image can be
detected. This is the basic principle behind featurebased methods. The rules are based on colour of the
face, shape of the face etc. In the case of learningbased face detection method, to learn the
discriminative features it uses statistical learning
methods and the training samples. This method has
less error rates but it leads to more computational
complexity. Viola-Jones [3] algorithm is the
commonly used for object detection. Almost in all
drowsiness detection system, because of the symptoms
related to face, the face regions are always processed
for detecting the state of the driver. Mainly, there are
three face detection methods are available: (1) based
on the imaging in the infrared spectrum, (2) featurebased methods, and (3) other methods.
Imaging in the infrared spectrum is one of the fast
and relatively accurate methods for face detection [4].
http://www.ijettjournal.org
Page 123
International Journal of Engineering Trends and Technology (IJETT) – Volume 29 Number 3 - November 2015
The part selected for imaging includes lighting and
camera. Lighting and camera selection is one of the
important stages when designing the system because
the system should work in all light conditions. The use
of lighting devices not only provides enough light in
environment, but also should not disturb driver’s
vision. Thus, near infrared (IR) spectrum is usually
used in lighting.
Module Description
Second method for face detection is feature-based
approach. It includes methods such as: Image
binarization and projection. Smith et al. [5] based on
the skin colour, binarization of face region was
performed which causes the needed parts appear black,
while other parts of the face appear white. Then, the
use of connected component analysis increases the
accuracy of face detection. In projection also, assume
that the region we need is darker than other skin
region.. As a result, most of the driver drowsiness
detection systems detect driver drowsiness and
distraction based on the symptoms extracted from the
face. This is the main focus of this paper.
There are number of techniques available for
detecting the drowsiness of aperson. They are i) Image
processing based ii) EEG based iii) ECG basediv)
ANN based techniques. In this the most widely used
technique to detectthe drowsiness of a person is the
image processing based technique. Thisuses various
images of a drowsy person to identify changes in
his/her facialexpression while driving.
In this method, first the images of the driver while
driving is extracted fromthe video using the webcam
sensor placed in the dashboard of the vehicle.Face
detection and tracking are important in the image
processing basedtechnique.
There are 2 modules in here which are detailed below:
1. Facial feature extraction
2. Drowsiness detection according to extracted
features
A. Facial feature extraction
Face and feature detection
III. PROPOSED SYSTEM
In the proposed system, drivers face images are
used for processing so thatone can find its states. From
the face image one can see that driver is awakeor
sleeping. Using same images, they can define
drowsiness of driver becausein face image if driver is
sleeping or dozing then his/her eyes are closed
inimage.And also we can predict their drowsiness with
the help of the symptomslike yawning, blinking and
head position like nodding, tilting and shaking.And
other symptoms of drowsiness can also detected from
the face image.
The first step in this process is to detect the face.
The face is detectedusing Viola-Jones algorithm of
Haar features with Adaboost learning. Thisis
commonly used for face detection. The method
proposed by Viola andJones is composed of weak
classifiers cascaded and their output is a
strongclassifier which detects the target object. For
each stage in the cascade, aweak classifier is trained to
reject a certain fraction of the non-target
objectpatterns and not rejecting any part of the target
object. The classifiers useHaar features in order to
encode facial features. For each feature, the value is
the difference between the sum of the pixels in black
regions and the sumof the pixels in white regions.
Fig. 2 Haar features
Fig. 1 Overview of proposed system
ISSN: 2231-5381
Rectangle features are computed very rapidly using
the integral image whichis an intermediate
representation. The integral image at location pixels (x,
y)contains the sum of the upper left pixels of the
original image, inclusively. Thevalue of the integral
image at location 1 is the sum of the pixels in
rectangleA, the value at location 2 is A+B, location 3
is A+C and at location 4is A+B+C+D. Then, the sum
within D is computed as 4 + 1 2 3 thatmeans the sum
http://www.ijettjournal.org
Page 124
International Journal of Engineering Trends and Technology (IJETT) – Volume 29 Number 3 - November 2015
of the pixels within rectangle D is computed with four
arrayreferences. Integral image provides the advantage
of fast feature evaluation.
Fig. 3 Integral image
Adaboost is used to select a small set of features
and train the classifiers. The learning algorithm for
weak classifiers is designed to select the single
rectangle feature best separating the positive and
negative examples. For each feature, the weak
classifier determines the optimal threshold.
Track Points
Using the Kanade Lucas Tomasi algorithm the
feature points in the detected face are identified. The
identified points in the detected face are tracked using
vision point tracker. Point tracker tracks each point in
the previous frame with the corresponding points in
the current frame.
Eye Detection
Detecting the eye region from the detected face is
the next step in this technique. Eye detection phase is
a module in which detected face is the input and the
eye regions are the outputs. The eye regions include
both the images of left and right eye region. Image of
the right eye region is the eye region cropped from the
original image and the image of the left eye region is
the eye region cropped from the original image.
Before given the input to the artificial neural network
for further training and classification, the eye regions
cropped from the detected face is need to be modified.
Yawning Detection
Yawn is one of the symptoms of drowsiness. The
yawn is assumed to be modelled with a large vertical
mouth opening. Using face tracking and then mouth
trackingone can detect yawn. The mouth is searched in
the lower most part of the detectedface region, thus
further decreasing the computation cost of the
system.If themouth opened beyond a particular
threshold for a long time, then we cansay that yawning
and driver is about to sleep.
Head pose is estimated by calculating optic flow of
the facial features. Headmovement is one of the
symptoms that shows the drowsiness. If the headis
moving either towards left or right or his head is
moving down, then wecan say that he is drowsy.
When the head moves down or lean towards leftor
right, the average distance between both the eyes
going to decrease andif the distance reaches a
particular threshold, then we can conclude that
thedriver is going to sleep.
B. Drowsiness Detection According To
TheExtracted Features
After the detection of all the features are completed
and the selected featureimages are cropped from
frames, we need to modify those images. The
framesextracted from video segments are in RGB
format.
When image in RGB format and a gray-level are
compared, RGB format image has negligible
advantage in displaying the image. This is verified
bytests on several networks. In addition, the size of
RGB images are threetimes the size of gray-level
images, that’s why we work with gray-level
imagesinstead of images in RGB format.
Training Neural Network
The time and memory consumption for training
neural network increases with the increasing number
of neurons. Therefore, we need to minimize
thenumber of neurons. Theneural networks we use are
feed forward networks which suit the objective,
deciding the state of an eye from eye region image.
The number of theeye regions obtained from the
feature extraction stage is given as input fortraining
the neural network. The output obtained from the
training regionis the state of the driver.
Driver State Estimation
Two cases are considered for detecting the state of
the driver. First one is drowsiness and the second one
is distraction. For drowsiness detection,closing and
opening of the eyes, continuous yawning and head
pose are considered.If the driver keeps the eye closed
for a certain amount of time (2seconds in our system),
the eye state will be considered as closed. If hekeeps
opening his mouth continuously and leaning head
forward for a certainamount of time, then also he is
going to sleep.
Then considering the distraction, if the eye of the
driver keeps focusing either to left or to right again for
a particular time (2 seconds in our system) thenwe can
conclude that the driver is distracted. After
considering these twoconditions we can conclude that
the driver is drowsy and distracted.
Head Movement detection
ISSN: 2231-5381
http://www.ijettjournal.org
Page 125
International Journal of Engineering Trends and Technology (IJETT) – Volume 29 Number 3 - November 2015
IV. EXPERIMENTAL RESULTS
The proposed method was tested on different group
of varying ages. The main purpose of this experiment
is to acquire the date for analysis and to compare the
validity and accuracy of the system. The web cam
placed in the dashboard of the car captured the video
of the driver while driving. From the video, frames are
acquired and using that frames, the face of the driver
is detected using Viola Jones algorithm. Then the eye
and mouth regions are cropped and head node is
calculated. Then it is given to the ANN for training.
number of neurons. That is why; only 81 neurons in
the input layerof neural networks are used. The neural
network here used is feed forwardnetwork and which
is reliably good for estimating the state of
driver.Theneural networks here use are feed forward
networks which suit the objective,deciding the state of
the driver from various states. Tried to train neural
networks by the back propagation methods. But the
disadvantage is that, itsmemory requirement is more
than the other methods. Since my problem canbe
categorized as a nonlinear problem, here need to use
nonlinear activationfunctions.
Here, first the eye states are considered as open,
half closed and closed andtrain each frames based on
artificial neural network. Then head pose andmouth
movements are considered and they are also trained.
The graph showsthe prediction accuracy of actual and
the predicted values of the featuresshowing the
drowsiness. For that, eye movements, yawning and
head movements are taken and given them to an ANN
classifier for classification. Andit produces an ROC
curve showing the accuracy of the drowsiness
detectionsystem proposed here.
Fig. 4 Detected face
First, the video is input to the Frame Extractor
module followed by the analysis of the output 150
frames one by one and then each frame is labelled
according to eye states, head pose and mouth states.
These valuesform the ground truth for all the features
for each frame. After that theseare converted to gray
level images. The intensity values of each pixel in
theobtained images with size [10 16] are transferred
into a matrix with size [814]. Reshaped feature image
matrices with size [81 4] are concatenated and amatrix
with size [81 150] is generated for a 30 second video
segment.
Fig. 6 Driver safety analysis graph
A confusion matrix is also obtained from ANN
after training. Confusion matrices for training, testing,
and validation, and the three kinds of data combined
and showed in the figure. The network outputs are
very accurate, asyou can see by the high numbers of
correct responses in the green squaresand the low
numbers of incorrect responses in the red squares. The
lowerright blue squares illustrate the overall
accuracies.
Fig. 5 Neural Network
The time and memory consumption for training
neural network increaseswith the increasing number of
neurons. Therefore, there is a need to minimize the
ISSN: 2231-5381
http://www.ijettjournal.org
Page 126
International Journal of Engineering Trends and Technology (IJETT) – Volume 29 Number 3 - November 2015
he closes his eyes for long time which is giving
information that thedriver might have slept.
In this work, a method to locate and track a drivers
mouth and eye usingcascade of classifiers training and
detection of eyes and mouth. Training of eye
movement and mouth images are done by using ANN.
Finally, ANN isused to classify the mouth regions to
detect yawning,eye region to detect eyemovement and
head node then alert drowsiness. The experimental
resultsshow that proposed method gives better results
than methods using geometric features. The proposed
method detects yawning alert fatigue earlier, andwill
facilitate to make drive safer.
ACKNOWLEDGMENT
We thank computer science department of Amal
Jyothi College of Engineering for providing us with
relevant data. This work was supported as part of
thesis project.
Fig. 7 Confusion matrix
REFERENCES
[1]
V. CONCLUSION
Various Techniques for drowsiness detection has
been reviewed with variousconditions. In this system,
the eye movement of the driver, head movementand
the yawning of the driver are used as various features
used to review thedrowsiness. A non-intrusive visual
based system is developed to locate eyesand mouth
and determines the driver’s drowsiness level through
horizontalaverage intensities of the eyes and mouth
region at face. During monitoringthe system is able to
detect when the eyes are closed, head movement
andmouth open and simultaneously for too long and
again and again in less period of time thus giving an
alert to the driver. Also the system alerts thedriver if
ISSN: 2231-5381
[2]
[3]
[4]
[5]
N. L. Haworth, T. J. Triggs and E. M. Grey, Driver Fatigue:
Concepts, Measurement and Crash Countermeasures, Human
Factors Group, Department of Psychology, Monash
University, 1988 June.
M. H. Yang, D. J. Kriegman and N. Ahuja, Detecting Faces
in Images: A Survey, IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol. 24, no. 1, 2002.
P. a. M. J. Viola, Rapid Object Detection Using a Boosted
Cascade of Simple Features, Proc. IEEE Conf. Computer
Vision and Pattern Recognition, Kauai, HI, USA ,2001.
R. Grace, V. E. Byme, D. M. Bierman, J. M. Legrand, D.
Gricourt, R. K. Davis, J. J. Staszewski and B. Carnahan, A
Drowsy Driver Detection System for Heavy Vehicles,
Proceedings of 17th AIAA/IEEE/SAE Digital Avionics
Systems Conference (DASC), Washington, USA , November,
1998.
P. Smith, M. Shah and N. V. Lobo, Monitoring Head/Eye
Motion for Driver Alertness with One Camera, Proceeding of
15th IEEE International Conference on Pattern Recognition,
Barcelona, Spain, 2000.
http://www.ijettjournal.org
Page 127