eyeCU Eye Tracking System



A wearable device
(glasses) with a
mounted camera
Images of the eye
are captured with a
digital camera
Images are
processed, and
mouse movement
commands are sent
to the computer to
control the cursor
Camera
Camera
Control
Image Data
Processing
Module
Wireless Data
Transfer
Host
Computer

Primary:
 Locate the pupil, assign it to one of four quadrants, send
movement commands to the computer, move the cursor
 Transfer the images from the Beagle Bone to the computer
via Ethernet
 Design 4 custom PCBs: camera board, power board,
interface board, and MSP430 board
Secondary:
 Support the eye tracker interface with computer
applications
 Display images that the camera captures on the computer
with overlays that indicate how the images are being
processed
 Refine algorithm to support a variety of users
 Interactive calibration
 Tertiary:
 DSP algorithm appropriate for a variety of lighting
conditions
 Develop point of sight control


Key Points
 IRB approval is a lengthy and rigorous process
 IR LEDs can inflict damage
 Due to potential hazard IR poses: probability of
IRB approval = 0%
 Completely Dropped IR illumination from project

Coded in Python

Uses the Windows API to get and set cursor position

Receives commands indicating the direction of movement
 R = (1,0), L = (-1,0), U = (0,1), D = (0,-1), S = (0,0)

Cursor position = (Previous cursor position) +
(Speed)(Command Vector)

For point of sight control, pupil coordinates are received
and mapped to screen coordinates.

Input: video of moving eye

Output: coordinates of pupil centroid, cursor
movement command

Current datasets are taken on a webcam with the IR
filter removed for enhanced contrast
A: aspect ratio for current region D: set of properly sized regions
C: set of connected regions
Si: pixel count for region i
Start
Threshold
C=
{Connected
Regions}
D={Ci|
0.8R < Si <
1.2R}
Compute
Aspect Ratio
Yes
Is D
empty?
No
Yes
Output Blink
Compute
Centroid
Yes
No
Max
adaptations?
Remove
Tails
0.5 < A < 2
No
End
Output
Command
Start
Threshold
C=
{Connected
Regions}
D={Ci|
0.8R < Si <
1.2R}
Compute
Aspect Ratio
Yes
Is D
empty?
No
Yes
Output Blink
Compute
Centroid
Yes
No
Max
adaptations?
Remove
Tails
0.5 < A < 2
No
End
Output
Command

Reference Pupil Area: 554

Current Threshold = 34

Number of pixels in connected
region:
 Eyebrow: 2416
 Pupil area: 1044
Start
Threshold
C=
{Connected
Regions}
D={Ci|
0.8R < Si <
1.2R}
Compute
Aspect Ratio
Yes
Is D
empty?
No
Yes
Output Blink
Compute
Centroid
Yes
No
Max
adaptations?
Remove
Tails
0.5 < A < 2
No
End
Output
Command
• Q: Queue of vectors
containing points
• p: current point
• D: matrix containing
locations and labels
of connected regions
Start
Q=(p0)
Pop item
from Q, call
it p
No
Write label
to D(p),
increment
region size
Add dark,
unprocessed
neighbors of
p to Q
Animation from Wikipedia’s flood fill algorithm
page
Q empty?
Yes
End

Reference Pupil Number
of pixels: 554

Current Threshold =34

Number of pixels in
connected
region:
 Eyebrow: 2416
 Pupil area: 1044
 Hair area: 48

Automated method for changing the threshold value for
each frame to accurately detect the pupil.

Adjust the threshold to find a connected region that meets
the following requirement.
 𝑆𝑖 ≈ 𝑅 (Discrepancy is the deciding factor on which
direction to change threshold value).

The change in the threshold value is decreased after each
iteration (idea of convergence).

Maximum of n adaptations. If pupil can not be detected
before maximum is reached, identify user as blinking.

Reference Pupil Area: 554

Current Threshold =31

Number of pixels
in connected region:
 Eyebrow 1: 564
 Eyebrow 2: 196
 Pupil area: 512
Start
Threshold
C=
{Connected
Regions}
D={Ci|
0.8R < Si <
1.2R}
Compute
Aspect Ratio
Yes
Is D
empty?
No
Yes
Output Blink
Compute
Centroid
Yes
No
Max
adaptations?
Remove
Tails
0.5 < A < 2
No
End
Output
Command

Aspect ratio: vertical length/horizontal length of the
connected region.

Assumption: aspect ratio of the pupil must be
approximately one.

Adjust the threshold until:
 a:
0.5 < 𝐴𝑖 < 2
OR
 b: We have reached the maximum threshold adaptations,
indicating that the user is blinking.

Eyebrow Aspect Ratio = 0.49

Pupil Aspect Ratio = 1.07
Start
Threshold
C=
{Connected
Regions}
D={Ci|
0.8R < Si <
1.2R}
Compute
Aspect Ratio
Yes
Is D
empty?
No
Yes
Output Blink
Compute
Centroid
Yes
No
Max
adaptations?
Remove
Tails
0.5 < A < 2
No
End
Output
Command

Use statistics to get rid of unwanted aberrations.

Scan the region horizontally, compute number of
pixels in each row.

Find mean and standard deviation of the pixel count.

Eliminate the row of pixels that fall outside of a certain
number of standard deviations of the mean.

Repeat the process for a vertical scan.
30
25
Number of Pixels
20
15
10
5
0
450
460
470
480
490
Discarded Columns
Kept Columns
500
510
X Position
520
530
540
550
Start
Threshold
C=
{Connected
Regions}
D={Ci|
0.8R < Si <
1.2R}
Compute
Aspect Ratio
Yes
Is D
empty?
No
Yes
Output Blink
Compute
Centroid
Yes
No
Max
adaptations?
Remove
Tails
0.5 < A < 2
No
End
Output
Command
Center
Right
Left
Up
Down
Blinking

Several parameters in the image processing algorithm,
such as initial threshold, are best set manually.

We propose to allow for interactive calibration, in which the
user will be able to modify algorithm parameters via a GUI
on the computer.

Video from the camera will be transferred via Ethernet to
the computer so that the user can view the effect of these
parameters in real time and choose suitable values.

Values obtained in calibration
 Initial threshold
 Processing region
 Reference points (center, corners)
CURRENT STATUS


Cursor movement
implemented in Python
Initial pupil detection
completed in MATLAB
NEXT STEP





Develop drivers to
facilitate the transfer of
command codes from the
Beagle Bone to the
computer.
Port Matlab code to C
Optimize algorithm
Test algorithm on Beagle
Bone
Experiment with different
lighting conditions
NO OS
OS (LINUX)
TI StarterWare
 Provides abstraction
layer
 Faster (no OS overhead)
 More control
 Use with CCS5
 Possibility of needing to
write additional board
level drivers


Linux as backup
 Several available
modules
 Additional overhead
 Less control
 Need Linux utilities to
compile software
http://www.ti.com/tool/starterware-sitara
Power
ARM Cortex A8
10X2
Header
0
2
9
1
C1_DOUT7
C1_DVSS
C1_IOVSS
C1_DVSS
8
1
7
1
C1_DCLK
C1_DOUT6
6
1
5
1
C1_HD
C1_DOUT5
F
u
1
F
u
1
F
u
1
4
1
3
1
C1_VD
C1_DOUT4
Semi
Cap
Semi
Cap
Semi
Cap
2
1
1
1
C1_VDSS
C1_IOVDD
5
C
C
4
C
C
3
C
C
0
1
9
C1_DVDD
C1_IOVSS
8
7
C1_SDA
C1_DOUT3
6
5
C1_SCL
C1_DOUT2
4
3
C1_RST1
C1_DOUT1
C1_DVDD
C1_IOVDD
C1_PVDD
2
1
C1_PVDD
C1_DOUT0
PC1
C1_DVSS
TCM8230MD_1
D
V
IOVSS
C1_VD
8
5
1
C1_IOVSS
OPEN
D
H
DVSS
C1_HD
9
7
C1_DVSS
Res3
3
C
R
K
L
C
D
RESET
C1_DVSS
C1_DVSS
C1_DCLK
0
1
3
7
T
U
O
D
SDA
C1_DOUT7
0
2
5
C1_SDA
F
u
1
F
p
2
3
F
p
2
3
F
u
1
6
T
U
O
D
SCL
C1_DOUT6
9
1
4
C1_SCL
Semi
Cap
Semi
Cap
Semi
Cap
Semi
Cap
5
T
U
O
D
C1_DOUT5
8
1
2
C
C
1
C
C
6
C
C
4
T
U
O
D
0
C1_DVSS
C1_DVSS
C1_DVSS
C1_DOUT4
7
1
3
T
U
O
D
Res3
7
C
C
24.54MHz
C1_DOUT3
4
1
2
T
U
O
D
SN74LVC1404
C1_DOUT2
3
1
C1_RST1
L
A
T
X
2
C
R
1
T
U
O
D
C1_DOUT1
2
1
F
p
0
3
D
N
G
XIN
0
T
U
O
D
K
L
C
T
X
E
4
3
2
1
C1_DOUT0
1
1
2
C1_CLK
0
0
5
Semi
Cap
K
0
1
T
U
O
X
1
C
Y
2
Res3
8
C
C
K
1
Res3
A
IOVDD
6
6
1
C1_IOVDD
4
C
R
Res3
1
C
R
Y
L
R
T
C
D
D
V
D
PVDD
2.2M
5
1
C1_DVDD
6
1
C1_PVDD
S
C
R
Res3
1
C
U
OSCOUT
C
C
V
C1_CLK
7
8
C1_PVDD
F
C
R
2
C
U
C1_PVDD
Oscillator driver
Camera
Header to daughterboard
MOSFET-N
3
D
Q
SDA
C_SDA
FIFO Buffer
D
N
10X2
Header
Res3
Res3
VCC-2.8
4
D
R
3
D
R
8
1
0
2
7
1
9
1
C_DOUT7
MOSFET-N
C_DCLK
C_DOUT6
D
N
G
D
N
G
6
1
5
1
1
D
Q
C_HD
C_DOUT5
VCC-3.3
4
1
3
1
C_VD
C_DOUT4
Cam_RST
C_RST
2
1
1
1
C_VDSS
C_IOVDD
IDT72V241L10PF
0
1
9
MOSFET-N
C_DVDD
C_IOVSS
8
7
D
N
C_SDA
C_DOUT3
6
5
E
G
S
R
5
9
\RS
O
2
D
Q
C_SCL
C_DOUT2
5
2
SCL
4
3
C_RST
C_DOUT1
VCC-2.8
D
N
G
2
1
WEN1
\WEN1
C_PVDD
C_DOUT0
4
2
WCLK
PD3
3
2
C_sDCLK
k
2
k
2
WEN2/LD
WEN2
2
2
SN74LVCH16T245
Res3
Res3
F
F
1
1
F
F
VCC-2.8
2
D
R
1
D
R
1OE
F
E
0
1
F
E
2
VCC-1.5
K
L
C
R
N
VCC-2.8
8
K
L
C
R
D
N
G
E
R
9
3
5
4
\REN2
D
N
G
D
N
G
2OE
8
4
5
2
7
4
3
VCC-3.3
VCC-2.8
1
N
E
R
\REN1
D
N
G
2DIR
2DIR
6
8
2
4
2
PAE
D
N
G
1DIR
1DIR
5
6
1
\EF
E
8
D
R
4
8
Q
8
D
7
1
9
2
C_DATA4
4
Q
4
D
5
6
1
E
L
F
2
C_DATA5
5
Q
5
D
0
2
2B5
2A5
D
VCC-3.3
8
1
8
2
2
2
2B6
2A6
9
2
C_DOUT5
2
C_DATA6
6
Q
6
D
2B7
2A7
7
2
C_DOUT6
D
C_DATA7
9
1
7
2
3
2
0
7
Q
7
D
2B8
2A8
6
2
C_DOUT7
2
6
2
F
D
N
G
0
1
D
N
G
E
Res3
3
PAF
5
1
D
N
G
E
PAF
D
N
G
L
4
PAE
1
2
1
I2C Shifter
G
k
2
k
2
VCC-2.8
C_SCL
6
1
0
3
9
1
0
3
C_DOUT4
C_DATA3
3
Q
3
D
2B4
2A4
Res3
5
1
1
3
7
1
2
3
C_DOUT3
\FF
C_DATA2
2
Q
2
D
2B3
2A3
F
F
4
1
2
3
6
1
3
3
C_DOUT2
7
D
R
C_DATA1
1
Q
1
D
2B2
2A2
3
1
1
4
1
5
3
C_DOUT1
C_DATA0
0
Q
0
D
2B1
2A1
2
1
2
3
1
6
3
C_DOUT0
F
F
D
N
G
D
N
G
C
C
V
1B8
1A8
VCC-3.3
1
2
2
1
7
3
1B7
1A7
2
D
U
1
1
8
3
1B6
1A6
5
6
1
2
D
E
L
9
0
4
F
u
1
F
u
1
1B5
1A5
Res3
VCC-3.3
8
1
4
D
N
G
Semi
Cap
Semi
Cap
\PAE
1B4
1A4
PAE
6
3
4
2
D
C
1
D
C
6
D
R
C_VerticalSync
1B3
1A3
5
4
4
C_VD
C_HorizontalSync
1B2
1A2
3
6
4
C_HD
F
u
1
PAE
1B1
1A1
C_sDCLK
2
7
4
C_DCLK
Semi
Cap
VCC-3.3
VCC-2.8
VCC-3.3
3
D
C
1
5
6
L
2
D
E
B
C
C
B
C
C
V
V
8
1
Res3
7
\PAF
PAF
VCC-3.3
5
D
R
A
C
C
V
Level Shifter
2
4
A
C
C
V
1
3
PAF
1
D
U
VCC-3.3
VCC-2.8
VCC-3.3
Indicator LEDs
Connector to camera
D
N
G
XBee
Connectors to Beagle Bone
0
6
6
G
Res3
LED_Blink
RD10
D
N
G
D
N
6
D
D
2
D
E
L
2x23
Header
D
N
G
D
N
G
D
D
N
G
D
4
4
6
6
4
5
4
4
4
3
4
4
3
4
5
D
4
D
N
G
D
N
G
D
D
N
G
\EF
2
4
1
4
2
4
1
4
0
4
9
3
0
4
9
3
2x23
Header
F
u
1
8
3
7
3
8
3
7
3
Semi
Cap
6
3
5
6
3
5
C
D
E
2
5
D
D
4
3
L
4
D
4
3
3
3
3
6
3
4
4
4
3
6
4
3
4
4
3
4
5
3
4
5
4
\FF
4
K
L
C
R
2
3
1
3
2
4
1
4
C_VerticalSync
2
3
1
3
2
4
1
4
0
3
9
2
0
4
9
3
0
3
9
2
0
4
9
3
8
2
7
2
8
3
7
3
8
2
7
2
8
3
7
3
D
N
G
6
2
5
2
6
3
5
3
VCC-3.3
x
R
6
2
5
2
6
3
5
3
5
6
1
4
2
3
2
4
3
3
3
x
T
4
2
3
2
4
3
3
3
Res3
2
2
1
2
2
3
1
3
XBee
2
2
1
2
2
3
1
3
9
D
R
0
2
9
1
0
3
9
2
0
2
9
1
0
3
9
2
ON/SLEEP
3
T
U
O
D
DIN/CONFIG
4
1
3
1
2
3
x
T
4
1
3
1
2
1
1
1
C_DATA5
2
1
1
1
2
2
V
0
1
9
1DIR
0
2
9
1
x
R
C
6
1
5
1
C_DATA1
4
2
2
2
1
2
1
2
C
6
1
5
1
C_HorizontalSync
\PAF
6
2
4
2
3
2
3
2
\PAE
D
1
0
1
SDA
8
1
7
1
SCL
8
2
6
2
5
2
5
2
C_DATA4
N
G
8
1
7
1
8
2
7
2
7
2
C_DATA0
1
0
1
9
0
2
9
1
D
N
G
D
N
G
SYS_5V
8
7
SYS_5V
2DIR
8
1
7
1
\REN1
3
D
U
8
7
8
1
7
1
VDD_5V
6
5
VDD_5V
\REN2
6
1
5
1
\WEN1
6
5
6
1
5
1
VDD_3V3EXP
4
3
VDD_3V3EXP
WEN2
4
1
3
1
4
3
4
1
3
1
2
1
\RS
2
1
1
1
VCC-3.3
2
1
2
1
1
1
Cam_RST
0
1
9
PD2
0
1
9
D
N
G
D
N
G
8
7
LED_Blink
8
7
C_DATA3
6
5
C_DATA2
6
5
C_DATA7
4
3
C_DATA6
4
3
2
1
2
1
VCC-3.3
VCC-3.3
PD1
G
N
D
G
N
D
G
N
D
1
6
5
1
6
5
L
E
D
2
L
E
D
2
Res3
Res3
1
6
5
L
E
D
2
RP4
RP5
Res3
VDD_5V
VCC-1.5
VCC-2.8
1.5V
2.8V
RP3
5
V
G
N
D
G
N
D
G
N
D
G
N
D
TLV1117
1
0
0
p
F
Semi
Cap
1
1
0
u
F
1
0
0
u
F
1
K
CP4
Semi
Cap
Semi
Cap
Res3
ADJ/GND
CP1
CP2
RP2
1
0
0
u
F
Semi
Cap
TLV1117
CP5
1
OUTPUT
4
VDD_5V
INPUT
3
1
0
u
F
ADJ/GND
OUTPUT
1
K
2
Semi
Cap
Res3
UP1
VCC-1.5
CP3
RP1
1N4002
Diode
OUTPUT
4
VDD_5V
INPUT
3
OUTPUT
DP1
2
UP2
VCC-2.8
1N4002
Diode
Linear Regulators
DP2
Power
FIFO Buffer
I2C
XBee
XBee
Comment
Description
Designator
Footprint
LibRef
LED2
Typical RED, GREEN,
YELLOW, AMBER GaAs LED 1.5V, 2.8V, 5V, DD5, DD6, EF, FF, PAE, PAF
DIODE1608 LED2
Cap Semi
Capacitor (Semiconductor SIM CC1, CC2, CC3, CC4, CC5, CC6, CC7, CC8, CD1, CD2, CD3, CD4,
Model)
CP1, CP2, CP3, CP4, CP5
CAPC1608
L
Cap Semi
Quantity Source
9 Digikey
17 Digikey
1 Amp General Purpose
Diode 1N4002 Rectifier
DP1, DP2
DIO10.46- Diode
5.3x2.8
1N4002
Header 10X2
Header, 10-Pin, Dual row
PC1, PD3
HDR2X10
Header
10X2
Samtec
2 samples
Header 2x23
Header 2x23
PD1, PD2
HDR2X23
Header
2x23
Samtec
2 samples
MOSFET-N
N-Channel MOSFET
QD1, QD2, QD3
SOT95P24 MOSFET0-3N
N
Resistor
RC1, RC2, RC3, RC4, RCF, RCS, RD1, RD2, RD3, RD4, RD5, RD6,
RD7, RD8, RD9, RD10, RP1, RP2, RP3, RP4, RP5
RESC1608
L
Res3
TCM8230MD_ VGA CAMERA MODULE
1
640x480
UC1
TCM8230M TCM8230
D
MD_1
1 Sparkfun
SN74LVC1404 Oscillator driver.
UC2
SN74LVC1 SN74LVC
404
1404
TI
1 samples
SN74LVCH16T 16-bit bus transceiver with
245
voltage regulation.
UD1
SN74LVCH SN74LVC
16T245
H16T245
TI
1 samples
IDT72V241L10
PF
36K (4K x 9) FIFO Buffer
UD2
IDT72V241 IDT72V24
L10PF
1L10PF
1 Digikey
XBee
XBee wireless
transmitter/reveiver
UD3
XBee
XBee
1 Sparkfun
TLV1117
Linear Regulator
UP1, UP2
TLV1117DCY
TLV1117
TI
2 samples
XTAL
Crystal Oscillator
YC1
BCYW2/D3.1
XTAL
1 Digikey
Res3
2 Digikey
3 Digikey
21 Digikey
CURRENT STATUS

Daughterboard
NEXT STEPS

 Daughterboard populated
 Schematic and layout
and tested working
 Revision of daughterboard
underway
 MSP430 host controller
board populated
 Testing underway
finished
 Board on order from
Advanced Circuits

Beagle Bone
 2 development boards on-
hand

MSP430
 Not yet started
Milestone 1:

Milestone 2:
 Integrate battery power
 Final revisions of boards
done
 TCM8230MD CMOS Camera
 Ideal for a wearable device
 640 x 480 Pixel Resolution (VGA)
 30 FPS
 Externally clocked by 24.5MHz
clock
 Data Output: RGB 5:6:5
 Camera Control: I2C
▪ 0-2.8V
▪ SCL Max Frequency 400kHz

Signal Integrity
CURRENT STATUS



Learned Stellaris
development board
architecture
Interfacing camera to
Stellaris development
board through I2C
Learning about Beagle
Bone architecture
NEXT STEP




MSP430 board design
and layout
Interfacing the camera to
the Beagle Bone
Test camera to Beagle
Bone interface
Getting LCD screen
functional
5V
3.3V
LiFePO4
Battery
6.6V
Power Supply
(4 Buck
Converters)
2.8V
1.5V
Processing
Module
6.6V
6.6V
LiFePO4
Battery
6.6V
TPS6214x
TPS6214x
TPS6214x
5V
3.3V
Beagle
Bone
XBee
2.8V
Camera
6.6V
TPS6214x
1.5V

LiFePO4
 6.6V and 2.3Ah

4 Buck Converters
 Texas Instruments
TPS6214x
 4 Voltage Rails
▪ 5V
▪ 3.3V
▪ 2.8V
▪ 1.5V

XBEE
 Series 1 (802.15.4)

Purpose
 Relay cursor commands
from Beagle Bone to
MSP430 Host Controller
 Send algorithm
parameters from
MSP430 Host Controller
to Beagle Bone
CURRENT STATUS


Wireless and MSP430
 Working with MSP430
development board
 Integrating an FTDI
breakout board from
Sparkfun with the
development board
Power
 Power design finished
and components on
order
NEXT STEPS


Wireless and MSP430
 Test USB interfacing
with MSP430
 Integrate XBEE with
development board
Power
 Test proof of concept
for power design
 Design PCB for power

Software
 Get code to compile on Beagle Bone
 Be able to dump data via Ethernet from the Stellaris to
the host computer
 Port Matlab to C

Hardware
 Get XBEE wireless running
 Get pictures off the camera placed into the correct
memory
 PCBs populated and tested
 Have Beagle Bone up and running with as many
drivers as possible
 Integration between XBEE and computer

Software
 Have Beagle Bone generating cursor
commands
 Optimization of code

Hardware
 Camera operational
 Have necessary Beagle Bone drivers
operational
 Have wireless operational
 Possibly have a battery recharging circuit




Duck Hunt
Integration of full system
Documentation
User manual
Section
Component
Quantity
Cost (each)
Cost to Team
Extended cost
Source
Wireless
XBee
2
11.75
11.75
23.5 Sparkfun.com
USB XBee Explorer
1
24.95
24.95
24.95 Sparkfun.com
XBee Breakout Board
2
2.95
2.95
5.9 Sparkfun.com
Processing
0
Beagle Bone Evaluation Board
2
89
Mechanical
0
0 Capstone
0
Lens-less Glasses
1
5.99
Camera
5.99
5.99
0
640x480 CMOS Camera
5
Test Cameras
3 Donated
FTDI to USB
1
9.99
9.95
49.75 Sparkfun.com
0 Capstone
10
Manufacturing
10
10 Sparkfun.com
0
PCB Fabrications (2 at 4 layer)
3
66
66
198 4pcb.com (Advanced Circuits)
PCB Fabrications (2 at 4 layer, 2
image)
3
112
112
336 4pcb.com (Advanced Circuits)
9.95
1
2
25
40
2
Poster
1
55
55
Shipping, miscellanous parts
1
200
200
Components
LCD Screen
Hardware Buffer
Daughterboard components
0 Adafruit.com
25
50 Digikey.com
0 Digikey.com
Presentation
55 ITLL
Misc.
Total Cost
200
959.09
Tasks
Armeen
Taeb
Nick
Bertrand
Computer
Applications
S
P
Lighting
P
S
DSP
P
S
Code Optimization
S
P
Arielle
Blum
Mike
Mozingo
Khashi
Xiong
Camera Module
P
Wireless
Communication
S
P
Physical Setup
S
P
Firmware/Drivers
P
S
S
Power
S
P
PCB Layout
P
S
Mascot/Cheerleader
Bruce
Chen
P,S,T