AN EYE MOUSE SYSTEM USING
FPGA
Professor: Yih-Ran Sheu
Student: Dinh Viet Thang(M992B206)
CONTENT
I. Abstract
II. Introduction
III. System Design
IV. Experimental Result
V. Conclusions
I. Abstract
- The objective of this paper present a set of techniques integrated
into an eye-mouse system using a field programmable gate array
(FPGA).
- Our system requires a camera (TRDB-D5M), FPGA-DE2 board
and a personal computer. A field programmable gate array
(FPGA) is used to control the CCD camera and execute the
image processing operations and control eye-mouse of computer
via RS232 protocol.
- The coordinate of eye-mouse in our system is determined by the
system based on FPGA imaging analysis. No computer is
necessary in image processing. Good efficiency can be seen
using FPGA hardware approach to determine the eye-mouse
coordinate per image frame.
- The color information of eye is used to estimate the direction of
pupil movement and then uses the direction information to
control the mouse of computer.
II. Introduction
•Nowadays, eye tracking and eye mouse system more and more are
widely. There are many eye tracking system using image
processing by computer supported by many image processing tools
such as Matlab, Visual C, Visual Basic…etc, but these systems
reduce the speed of the computer and sometimes cause unexpected
situations.
• Thus, processing image to detect eye movements using FPGA
hardware bring an alternative solution. In this paper, we practically
design and develop the improved eye-mouse device with TRDD5M camera, FPGA hardware, and computer. An eye tracking
systems is a system that can track the movements of the user's eyes.
•Eye mouse system is a very good approach for people who do not
have the ability to perform the movements of common control on
computer such as mouse drag and mouse click…etc.
•In this paper, we perform image processing to detect pupil
movement in FPGA hardware then sends control information to the
computer to control a cursor on the screen. There are several
different ways to track the directions of eye movements. Each
approach has its own advantages, disadvantages, and limitations. We
use information about eye color to detect the movement of the eye.
•There are many color spaces such as RGB, HSI, YUV, YIQ ... We
choose the YUV color space because it has many advantages and
simple.
•The image processing in FPGA is not as good as image
processing in personal computer but this process is totally
independent, thus increasing the processing speed of the system.
The eye mouse system is used to access computers via eye
movements.
III. System Design
•The eye-mouse system consists of two modules. The first module
including second -stage algorithm is proposed to implement image
processing and pupil movement detection on FPGA hardware.
•An efficient pupil detection based on the pupil color information is
employed to find the pupil and corner of user’s eye. After this, some
algorithms is used to detect the direction information of pupil/corner
of eye then transmit to computer via RS232 communication protocol.
•The second module is software on computer including cursor control
block, user interface block, and application. The computed direction
information of pupil movements is used to control the mouse of
computer.
•Through the graphic user interface, users can implement some
application.
•The component of eye mouse system is introduced as follows.
a. Software architecture system
b. Figure Hardware architecture system
2.1 HARWARE SPECCIFICATIONS
•The system consists of a personal computer, FPGA hardware with
TRD-D5M camera and RS232 cable, FPGA hardware with a video
capture, zoom buttons and adjust contrast.
•A real-time eye-tracking system requires an image capturing
device operating at an extremely high frame rate to be able to
effectively track the pupil in the presence of saccadic motion.
•Therefore, a high-frame rate image capture device is necessary in
order to build a general eye tracker system for various humancomputer interaction applications [1].
•The TRDB_D5M Kit satisfies most of the above requirements to
develop a 5 Mega Pixel Digital Camera on the Altera DE2-70 / DE2
boards.
•2.2 THE ALGORITHM SYSTEM
• The image input through the camera is transformed by FPGA
hardware to obtain the image data of the plane coordinate.
• Every point (i,j) on the coordinate plane has its own image
data. With this image information, the system is able to identify
the pupil of eye from the image[5].
• Our methods based on color information about the pupil of
eye to determine the pixel according to the definition given.
• The first problem is to represent images in the appropriate
color space that allows to distinguish two components:
composition luminance, luminous intensity and color
components.
• RGB color space generally inappropriate because it
mixes the two components.
2.2.1 Image processing of an eye-mouse system
•- Although there are other color models such as CMY, YIQ and
HSI, in this paper YUV color model is chose because of its
advantages. In YUV color model, the luminance information is
represented with Y component while the color information is
represented by the U and V components.
•- The main idea of using color-space information is to increase the
information available about an image. The pupil feature detection
system based on eigenfeature approach is used for gray-level Y
and U, V color information of image. YUV color components can
be obtained by using the RGB color coefficients as follows [6].
Y   0.299 0.587 0.114  R 
U   - 0.148 - 0.289 0.437  G 
  
 
V   0.615 - 0.515 0.100   B 
•a. The definition of thresholding
255, f (i, j )  T
g(i,j) = 
0, f (i, j )  T
b. Calculation for the coordinate of the pupil center:
1
1
 X C ,Y C    K  X , K  Y
1
1
K
K
Where K is the total number of the dark spot at which the gray level is lower than the threshold after
dynamic binarizing of the eye image.
X and Y are coordinates of the black pixel. [5]
2.2.2 Eye movement detection
After we have located the pupil center point from the eye we
can proceed to extract the direction information from the pupil
movements.
The directions of the pupil movements are quantized into three
directions: right, left, and center.
The third different position of pupil center point corresponding
to the third region on the picture with size 320x240 pixel
is illustrated as follows.
Figure 6. The 3 different regions of pupil movement corresponding to the 3 directions of mouse.
•2.2.3 Communication between FPGA hardware and computer
• The direction information of pupil is transmitted to computer by
RS232 or USB protocol communication.
• This paper we use RS232 protocol communication. Visual Basic
software is used to receive the direction information of the eye
movements from FPGA hardware.
• Although the baud rate of RS232 port is lower than USB, this
speed is enough to satisfy our requirement.
•3. EXPERIMENTAL RESULTS
•3.1 Eye Tracking Experiment
- In this paper, we only perform a simple mouse eye system. The obtained results are quite accurate because
only three regions of the window is active.
- For other complicated applications which require the pupil of the eye moving to more precise location of the
window, only information detected from the color of the eye of people is not enough.
- In addition, changes in light conditions also reduce the accuracy of the system. - Therefore, in these cases we
need to combine information about color and other methods as detect edges, Hough Transform to upgrade
the system. Some exact results are shown in the picture.
•3.2 Mouse Control and communication Aid
- We can use the eye tracking system combined with the communication aid, as shown in Figure.7.
- The communication aid can be designed to contain 3 cells for each window to express our needs. A mouse
click is required when a cell is active.
- The mouse click can be started after the cursor has occupied the cell for more than a specified period (2
second). The center of the window is used to rest the cursor.
Figure 7. Communication Aid a) The Home application cell b) The Emergency cell
IV. Conclusions
- In this paper we propose an automatic and real time detecting pupil movement direction
method and then implement a simple eye-mouse system.
- The particular feature of this method is using the color information to detect the pupil
movement. In this paper, we only use the color information so the results have some problem
when the lighting conditions change.
- Furthermore, the eye’s pupil is detected with a predefined camera position only, so this
method has some limitations. However, the algorithm is efficient because it is only based on
low level signals and thresholds for simple image processing.
- So, in a limited application, this algorithm can work well. Currently, we have simulated our
process by Visual Basic and then convert to Verilog program for FPGA hardware.
- In our future work, we plan to use more information of eye, edge information, and the
Hough transform to get better results.
References
[1]. Ashlt talukder, john-michael morookian, s. monacos, r. lam, c. lebaw, a. bond
Intelligent Instruments and Systems Technology Group, In-Situ Instruments
Section, Real-time Non-Invasive Eye tracking and Gaze-point Determination
for Human-Computer Interaction and Biomedicine.
[2]. Mu-chun su, kuo-chung wang, gwo-dong chen Department of Computer
Science and Information Engineering,National Central University, Chung Li,
Taiwan, An eye tracking system and its application in aids for people with
severe disabilities.
[3]. Hong LIU, Yuwen WU, Hongbin ZHA National Lab. On Machine
Perception, Peking University, 100871, Beijing, China, Eye states detection from
color facial image sequence
[4]. Hasan Demirel, Department of Electrical & Electronic Engineering Eastern
Mediterrnean University, Color spase analysis for facial feature detection.
[5].Chern-Sheng Lin, Hsien- Tse Chen, Tien-Gern Lin, Mau- Shiun Yeh, ChuenLin Tien, Development and Application of An infrared eye-mouse control
system