Advanced Topics on Internet Of Things
It Starts with iGaze: Visual Attention
Driven Networking with Smart Glasses
Lan Zhang, Xiang-Yang Li, Wenchao Huang, Kebin Liu,
Shuwei Zong, Xuesi Jian, Puchun Feng, Taeho Jung,
Yunhao Liu
ACM. New York, NY, USA ©2014
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
1/35
Content
1. Background
2. iGaze Idea
3. Challenges and Solutions
4. System Design
5. Empirements and Evaluation
6. Future Work
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
2/35
Background Smart Glasses
 Collect information sensors
 Can run Mobile App
 Control, or retrieve data from, other
 Support wireless technologies Bluetooth, WiFi, and GPS
 May also have all the features of a smartphone
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
3/35
Background Existing Technologies
Image Recognition High Computation Cost
QR code Scanning Distance < 3
Scanning Angle <30°
Hand gesture Displacement > 20cm
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
4/35
Background Human Computer Interface
Voice Commands
Head Gesture Control
TouchPad or Buttons
Smart glasses are still looking for revolutionary applications
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
5/35
iGaze Idea New Networking Mode
 Can understand the user's visual attention
using an eye camera
.
 Camera Can automatically connect to the
target of interest
 Can run on top of existing networking
protocols, e.g., Wi-Fi.
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
6/35
iGaze Idea
Museum
.
Obtain the detailed descriptions of art works by glazing at it instead.
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
7/35
iGaze Idea
Device Control
.
A house can turn on a smart appliance only by taking a glance at them
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
8/35
iGaze Idea
Advertising
.
When people look at a signboard or product, information can be sent to the potential
customer immediately.
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
9/35
iGaze Idea
.
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
10/35
iGaze Idea iGaze Principales
Unidirectional
Given an observer and his/her
gaze vector, only the device vector
to the correct visual target’s device
is consistent with the gaze vector.
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
11/35
iGaze Idea iGaze Principales
Bidirectional
Given a pair of users who are
looking at each other, their gaze
vectors have opposite directions.
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
12/35
iGaze Idea
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
13/35
Challenges and Solutions
Question 1 : How to accurately capture the vision plane attention of a user in real time ?
Answer :
• Develop a low-cost glasses hardwarembedded
with attention capture software
• Capture the gaze direction using the orientation
sensor and eye camera.
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
14/35
Challenges and Solutions Visual Attention Detection
• Visual attention detection  Fixation
• A fixation duration window threshold for
visual attention
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
15/35
Challenges and Solutions Determine Gaze Direction
 Projection on the image plane is elliptical
 Look ahead: the projection looks more like a circle
 Look off to one side: it looks more close to an
ellipse
 estimate the pose of the iris circle by backprojecting the ellipse onto a circle in 3D space
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
16/35
Challenges and Solutions
Question 2 : How to match the target on device plane according to user’s visual attention?
Answer :
 Capture the direction between glasses
 Leveraging Doppler effect caused by arbitrary mild movement of head-
mounted speakers.
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
17/35
Challenges and Solutions Determine Device Direction
Head Node or Shake
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
18/35
System Design
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
19/35
System Design Gaze Vector Acquisition
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
20/35
System Design Device Vector Estimation
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
21/35
System Design Matching and Networking
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
22/35
System Design Application Layer
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
23/35
System Design Visual Attention Networking (VAN) Protocol
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
24/35
System Design Visual Attention Networking (VAN) Protocol
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
25/35
System Design Visual Attention Networking (VAN) Protocol
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
26/35
System Design Visual Attention Networking (VAN) Protocol
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
27/35
Empirements and Evaluation
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
28/35
Empirements and Evaluation Attention & Gaze Direction Accuracy
Attention capture accuracy: ~95%
Gaze direction accuracy:
< 5° (91 %)
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
< 10° (99 %)
Presentor: Aouaidjia Kamel
29/35
Empirements and Evaluation Device Direction Accuracy
Direction estimation is robust against different head gestures and patterns.
Only one mild head gesture is sufficient for highly accurate estimation.
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
30/35
Empirements and Evaluation Device Direction Accuracy
When the distance gets larger, the accuracy decreases
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
32/35
Empirements and Evaluation Computation Cost & Energy Consumption
Computation Cost
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Energy Cost
Presentor: Aouaidjia Kamel
33/35
Future Work

Multiple objects in the gaze direction.

Multiple users may be interested in a same object.

User may be interested in a moving object.

How to avoid insecure connection.

Aesthetic design.
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
34/35
Thank you for listening
It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses
Presentor: Aouaidjia Kamel
35/35