International Journal of Application or Innovation in Engineering & Management (IJAIEM)
Web Site: www.ijaiem.org Email: editor@ijaiem.org, editorijaiem@gmail.com
Volume 2, Issue 3, March 2013
ISSN 2319 - 4847
Human Computer Interaction using Hand
Gesture Recognition with Neural Network: A
Review
Sujeet D.Gawande 1, Prof. Nitin R. Chopde2
1
M.E.Scholar, 2M.E. (Computer Engineering)
Department of Computer Science and Engineering,
1,2
G.H. Raisoni College of Engineering and Management, Amravati.
1,2
ABSTRACT
The aim of the hand gesture recognition is to develop the system to recognise the gesture, for control devices by providing
command. In this paper we are discussing the researches done on the hand gesture recognition using neural network. Several
hand gesture recognition researches that use Neural Networks are discussed in this paper, comparisons between these methods
were presented, advantages and drawbacks of the discussed methods also included, and implementation tools for each method
were presented as well.
Keywords: Neural Networks, Human Computer Interaction, Gesture Recognition System, Gesture Features, Static
Gestures, Dynamic Gestures.
1. INTRODUCTION
With the development of information technology in our Society, one can expect that computer systems to a larger extent
will be embedded into our daily life [Murthy R. S. &. Jadon. R. S.:2009].These environment leads to the new types of
human computer interaction (HCI). The use of hand gestures provides an attractive alternative to cumbersome interface
devices for human-computer interaction (HCI). , the existing HCI techniques may become a bottleneck in the effective
utilization of the available information flow. For example, the most popular mode of HCI is based on simple
mechanical devices—keyboards and mice. These devices have grown to be familiar but inherently limit the speed and
naturalness with which human can interact with the computer. The development of user interface requires a good
understanding of the structure of human hands to specify the kinds of postures and gestures [Garg. P., Aggarwal. N,
and Sofat. S.:2009]. Feelings and thoughts can also be expressed by the gesture. Users generally use hand gestures for
expression of their feelings and notifications of their thoughts. Hand gesture and hand posture are the two terms related
to the human hands in hand gesture recognition. The difference between hand gesture and hand posture, hand posture
is considered to be a static form of hand poses [Garg. P., Aggarwal. N and Sofat. S.:2009]. Gestures can be classified
into static gestures and dynamic gestures. Static gestures are usually described in terms of hand shapes, and dynamic
gestures are generally described according to hand movements. Gesture can be defined as a meaningful physical
movement of the fingers, hands, arms [Mitra.S. and, Acharya. T: 2007], or other parts of the body [gesture Wikipedia
website] [Mitra.S. and, Acharya. T: 2007], with the purpose to convey information or meaning for the environment
interaction [5]. Gesture recognition, needs a good interpretation of the hand movement as effectively Meaningful
commands [Murthy R. S. &. Jadon. R. S.:2009].]. For human computer interaction (HCI) interpretation system there
are two commonly approaches [Murthy R. S. &. Jadon. R. S.:2009].
(A). Data Gloves Approaches
These methods employ mechanical or optical sensors. In gesture recognition, it is more common to use a camera in
combination with an image recognition system .These systems have the disadvantage that the image/gesture
recognition is very sensitive to illumination, hand position, hand orientation etc. In order to circumnavigate these
problems we decided to use a data glove as input device. [Weissmann J, Salomon.R: 1999].
(B). Vision Based Approaches
These techniques based on the how person realize information about the environment. These methods usually done by
capturing the input image using camera(s) [Meena Sanjay: 2011].
Volume 2, Issue 3, March 2013
Page 332
International Journal of Application or Innovation in Engineering & Management (IJAIEM)
Web Site: www.ijaiem.org Email: editor@ijaiem.org, editorijaiem@gmail.com
Volume 2, Issue 3, March 2013
ISSN 2319 - 4847
Figure 1: design of low cost data glove approach [www.academic.education]
Figure 2: vision based approaches.
2. ARTIFICIAL NEURAL NETWORK AN OVERVIEW
According Haykin [Haykin .S:1999], and Marcus [Lamar Marcus Vinicius: 2001], Artificial Neural Networks are one
of the technologies that solved a broad range of Problems in an easy and convenient manner. The working concept of
Artificial Neural Networks (ANNs) is similar to human nervous system, hence it has synonym with the word neural
networks, as in illustrated in Figure. Neural network is also known as Artificial Neural Network (ANN), is an artificial
intelligent system which is based on biological neural network. Neural networks able to be trained to perform a
particular function by adjusting the values of the connections (weight) between these elements.
Figure 3: block diagram of neural network
Volume 2, Issue 3, March 2013
Page 333
International Journal of Application or Innovation in Engineering & Management (IJAIEM)
Web Site: www.ijaiem.org Email: editor@ijaiem.org, editorijaiem@gmail.com
Volume 2, Issue 3, March 2013
ISSN 2319 - 4847
Table 1: application of neural network
Industry
Automobile
Financial
Defense
Business application
Automobile warranty analysis and automatic
guidance system.
Real estate appraisal, loan advising, corporate
bond rating, credit line use analysis, credit card
activity tracking
Weapon steering, target tracking, object
discrimition, facial recognition, feature extraction
and noise suppression.
Breast cancer cell analysis, EEG and ECG
analysis, and prosthesis design.
Medical
(A) Advantages of Neural Network
Neural Network has variety of advantages especially for those analysts. Below are some the of Neural Network’s
advantages:
a) Neural Network system is developed through learning rather than programming. This will definitely save some time
for programmer to do programming because programming is much more time consuming and require them to specify
the exact behavior of the model. This allows programmer/analyst focus more on result rather than design the program.
b) [Maraqa Manar, Zaiter. Raed Abu: 2008] Neural Network is flexible in a changing environment. Normal
programmed systems are limited to certain situation. When the condition changed, the program no longer functions
well.
c) Neural Network able to build informative models where most conventional methods fail. Neural Network can easily
model data which is very difficult to model because Neural Network works by through traditional approaches such as
programming logic and inferential statistics.
d) Neural Network pattern recognition is a powerful and robust approach for harnessing the information in the data.
Neural Network learns to recognize patterns from the data set that presented to it.
(B) Limitations of Neural Network
Even though Neural Network has variety of benefits, but every system also has their own limitation. Below are some of
the Neural Network’s limitations:
a) Neural Network unable to explain the model or network that it has built in a useful way. Neural Network always get
better results but have a hard time to explain how it’s got here. This explanation is important especially for analysts
who want to know how the model behaves [Murakami Kouichi and Taguchi Hitomi: 1999].
b) Neural Network won’t produce good results if the input data are not representative of the problem. This situation
classified as ‘garbage in’ produce ‘garbage out’. So analyst has to spend time to understanding the problem or the
outcome that expected. And, analyst must select appropriate data used to train the system and are measured in a way
that reflects the behavior of the factors.
c) Neural Network takes time to train a model when very complex data set present to it. This technique will slow down
on low end computer or machine that without math coprocessors. Because nowadays, most computers’ processor is fast
enough to train this Neural Network.
An artificial neural network involves a network of simple processing elements (artificial neurons) which can exhibit
complex global behaviour, determined by the connections between the processing elements and element parameters. It
consists of an interconnected group of artificial neurons and processes information using a connectionist approach to
computation [9]. In most cases an ANN is an adaptive system that changes its structure based on external or internal
information that flows through the network during the learning phase. The utility of artificial neural network models
lies in the fact that they can be used to infer a function from observations. This is particularly useful in applications
where the complexity of the data or task makes the design of such a function by hand impractical. The tasks to which
artificial neural network share applied. The supervised learning paradigm is also applicable to sequential data (e.g., for
speech and characters recognition).
(C) Feed forward Multilayer Perception Network
The feed forward neural network was the first and arguably simplest type of artificial neural network devised. In this
network, the information moves in only one direction, forward, from the input nodes, through the hidden nodes (if any)
and to the output nodes. There are no cycles or loops in the network. In computing, feed-forward normally refers to a
Volume 2, Issue 3, March 2013
Page 334
International Journal of Application or Innovation in Engineering & Management (IJAIEM)
Web Site: www.ijaiem.org Email: editor@ijaiem.org, editorijaiem@gmail.com
Volume 2, Issue 3, March 2013
ISSN 2319 - 4847
multi-layer perception network in which the outputs from all neurons go to following but not preceding layers, so there
are no feedback loops. Fig 3 below shows a representation of a simple feed-forward Neural Network with four inputs,
one hidden layer and four outputs. Neural networks learn by changing their weights [www.engineersgarage.com]
3. COMPARISON FACTORS
Comparisons between the selected methods have been concluded according some important factors, table 1 shows these
factors. For simplicity the name of the method will be pointed as the name of work used in that paper. I.e. Kouichi
[Murakami Kouichi and Taguchi Hitomi: 1999.] will be referred as Japanese language recognition. Manar [Maraqa
Manar, Zaiter Raed Abu: 2008] Arabic language recognition. Hninn [Tin Hninn H. Maung: 2009] as Myanmar
language recognition. Gonzalo [Bailador Gonzalo, Roggen Daniel, and Tröster Gerhard: 2007] as signal Gesture and
Stergiopoulou [Stergiopoulou E., Papamarkos N: 2009] as shape fitting gesture.
Table2: Comparison between recognition methods in neural network parameters
[Ibraheem Noor A. & Khan Rafiqul Z: 2012]
Method
Neural
Neural
Activation
Learning
network
Network
Function
Time
Type
Japanese
Two
Back
Sigmoid
Several
Language
propagation
Hours
network
Arabic
Two
Elman
Sigmoid
N
language
recurrent
recognition
network
Myanmar
One
Supervised
Hard-limit
N
language
neural
recognition
network
Signal
One
Continuous
Differential
N
Gesture
Time
Equation
Recurrent
Neural
Networks
shape fitting
One
Self-Growing
N
gesture
and SelfOrganized
Neural Gas
TABLE 3: Comparison between recognition methods in hand gesture recognition approach used
[Ibraheem Noor A. & Khan Rafiqul Z: 2012]
Method
Name
Type of
input
device
Segmentation
operation
Japanese
language
recognitio
n
Data glove
threshold
Arabic
language
recognitio
n
Myanmar
language
recognitio
n
Colored glove,
Digital camera
HSI colour
model
Digital
camera
threshold
Volume 2, Issue 3, March 2013
Feature
vector
Representat
ion
13 data
item (10 for
bending, 3
for
coordinate
angles)
Available
Features
from
resource
Orientation
histogram
Neural
network
Type
sampl
e
gestur
es
Recogniti
on Rate
Recognition
Time
back
propagation
network
42
71.4%
Several
seconds
Elman
recurrent
Network
30
81.96%
N
33
90%
N
supervised
neural
network
Page 335
International Journal of Application or Innovation in Engineering & Management (IJAIEM)
Web Site: www.ijaiem.org Email: editor@ijaiem.org, editorijaiem@gmail.com
Volume 2, Issue 3, March 2013
ISSN 2319 - 4847
signal
Gesture
shape
fitting
gesture
accelerometer
sensor,
wireless mouse
Digital
camera
Automatically
(magnitude
acceleration
signal) /
manually
(wireless
mouse
button)
YCbCr color
Space
do not
require in
signal
predictors
Two angles
of
the hand
shape,
compute
palm
distance
Continuous
Time
Recurrent
Neural
Networks
SelfGrowing
and SelfOrganized
Neural Gas
160
94%
N
31
90.45%
1.5 seconds
4. IMPLEMENTATION TOOLS
MATLAB programming language wit`h image processing toolbox was used for implementing the recognition system
and C, and C++ language were used less [Chaudhary Ankit,Raheja, J. L., Das Karen, and Raheja Sonia: 2011]. Hninn
[Tin Hninn H. Maung: 2009] use MATLAB hand tracking and gesture recognition. Manar [Maraqa Manar, Zaiter.
Raed Abu: 2008] use MATLAB6 and C language, MATLAB6 used for image segmentation while C language for HGR
system. Kouichi [Murakami Kouichi and Taguchi Hitomi: 1999] use SUN/4 workstation for Japanese Character and
word recognition. Also Stergiopoulou [Stergiopoulou E., Papamarkos N: 2009] used Delphi language with 3GHs CPU
to implement hand gesture recognition system using SGONG network.
5. DISCUSSION AND CONCLUSION
In this paper the idea about the hand gesture recognition and artificial neural network are presented. Artificial neural
network is one of the most effective software computing techniques. The advantages and the disadvantages of the
neural network are also discussed in this paper. For human computer interaction (HCI) interpretation system there are
two commonly approaches discuss that is vision based approaches and data gloves approaches. Neural Networks system
can be applied for extracted features from the input image gestures after applying segmentation, as in [Stergiopoulou
E., Papamarkos N: 2009] to extract the shape of the hand. Comparison between recognition methods in hand gesture
recognition approach used is also discuss according to the different parameters.
REFERENCES
[1] A Low Cost Data Glove for Virtual Reality Pablo Temoche Esmitt, Ramirez Omaira Rodiguez
www.academic.education.
[2] Bailador Gonzalo, Roggen Daniel, and Tröster Gerhard: 2007 “Real time gesture recognition using Continuous
Time Recurrent Neural Networks”, Proceedings of the ICST, 2nd international conference on Body area networks.
[3] Chaudhary Ankit,Raheja, J. L., Das Karen, and Raheja Sonia: 2011 “Intelligent Approaches to interact with
Machines using Hand Gesture Recognition in Natural way A Survey” International Journal of Computer Science &
Engineering Survey (IJCSES), vol. 2 (1).
[4] Engineers Garage.”Artificial Neural Networks (ANN): Introduction, Details & Applications.
Available”http://www.engineersgarage.com/articles/artificial-neural-networks.
[5] Garg. P., Aggarwal. N and Sofat. S: 2009 “Vision Based Hand Gesture Recognition,” World Academy of Science,
Engineering and Technology vol. 49, pp. 972-977.
[6] Gesture Wikipedia website.
[7] Haykin .S: 1999 “Neural Networks - A Comprehensive Foundation”, Englewood Cliffs, NJ: Prentice-Hall, Second
Edition. Available: http://www.amazon.de/Neural-Networks-Comprehensive Simon- Haykin/dp/0132733501.
[8] Ibraheem Noor A. & Khan Rafiqul Z: 2012 Vision Based Gesture Recognition Using Neural Networks
Approaches: A Review. International Journal of human Computer Interaction (IJHCI), Volume (3): Issue (1):
[9] Lamar Marcus Vinicius: 2001 “Hand Gesture Recognition using T-CombNET a Neural Network Model dedicated
to Temporal Information Processing,” Doctoral Thesis, Institute of Technology, Japan.
[10] Maraqa Manar, Zaiter. Raed Abu: 2008 “Recognition of Arabic Sign Language (ArSL) Using Recurrent Neural
Networks,” IEEE First International Conference on the Applications of Digital Information and Web
Technologies, (ICADIWT 2008), pp. 478-48.
Volume 2, Issue 3, March 2013
Page 336
International Journal of Application or Innovation in Engineering & Management (IJAIEM)
Web Site: www.ijaiem.org Email: editor@ijaiem.org, editorijaiem@gmail.com
Volume 2, Issue 3, March 2013
ISSN 2319 - 4847
[11] Meena Sanjay: 2011 “A Study on Hand Gesture Recognition Technique,” Master thesis, Department of Electronics
and Communication Engineering, National Institute of Technology, India.
[12] Mitra.S. and Acharya. T: 2007 “Gesture Recognition: A Survey” IEEE Transactions On systems, Man and
Cybernetics, Part C: Applications and reviews, vol. 37 (3), pp. 311-324.
[13] Murakami Kouichi and Taguchi Hitomi: 1999 “Gesture Recognition using Recurrent Neural Networks. ”ACM
Proceedings of the SIGCHI conference on Human factors in computing Systems”: Reaching through technology
(CHI '91), pp.237-242.
[14] Murthy R. S. &. Jadon. R. S: 2009. “A Review of Vision Based Hand Gestures Recognition,” International Journal
of Information Technology and Knowledge Management, vol. 2(2), pp. 405-410.
[15] Stergiopoulou E., Papamarkos N: 2009 “Hand gesture recognition using a neural Network shape fitting technique,”
Elsevier Engineering Applications of Artificial Intelligence, Vol. 22(8), pp.1141-158.
[16] Tin Hninn H. Maung: 2009. “Real-Time Hand Tracking and Gesture Recognition System Using Neural
Networks,” World Academy of Science, Engineering and Technology 50,pp. 466- 470.
[17] Weissmann J, Salomon.R: 1999 - Neural Networks. IJCNN'99. 1999 - Ieeexplore.ieee.org.
Volume 2, Issue 3, March 2013
Page 337