The World in an Eye
Ko Nishino
Shree K. Nayar
Columbia University
IEEE CVPR Conference
June 2004, Washington DC, USA
Supported by NSF
Ophthalmology
Computer Vision
• Anatomy, physiology
•Iris recognition
[Helmholtz 09;…]
[Daugman 93;…]
HCI
Computer Graphics
• Gaze as an interface
• Vision-realistic Rendering
[Bolt 82;…]
[Barsky et al. 02;…]
Corneal Imaging System
Geometric Model of the Cornea
Iris
Pupil Sclera
Cornea
R=7.6mm
tb  2.18mm
rL  5.5mm
eccentricity = 0.5
Finding the Limbus
limbus parameters

max g rx  *
e
rx
Gaussian
e : radii r , r  center c , c 
x
y
x

e I x, y ds  g ry * ry
intensity value
y
tilt 
 I x, y ds
e
Self-calibration:
3D Coordinates, 3D Orientation
How does the World Appear in an Eye?
Imaging Characteristics: Viewpoint Locus
P
Vr
N
Vi
Camera Pupil
X
rc
S
Vt , , r   St ,   rVi t , 
Z
det J Vt , , rc   0
[Burkhard and Shealy 73; Swaminathan et al. 01]
Viewpoint Loci
camera pupil
X
cornea
Z
Viewpoint Loci
camera pupil
X
cornea
Z
Viewpoint Loci
camera pupil
cornea
Imaging Characteristics: Resolution
surface point distance
from viewpoint locus
S rc
 cos  t ,   2
 f
 rc t , 
 cos  t ,  
2
cos  t , S z t ,   Pz 
2
P
X
viewpoint locus
Z
cornea
surface point distance
from camera pupil
Imaging Characteristics: Field of View
camera pupil
V i tb , 
FOV  
2
0
 V t ,   1d closed loop of limbus
i
z
b
incident light rays
Resolution and Field of View
gaze direction
X
cornea
Z
Resolution and Field of View
resolution
gaze direction
X
FOV
cornea
Z
Resolution and Field of View
resolution
gaze direction
X
FOV
cornea
Z
27
person’s FOV
45
What does the Eye Reveal?
Environment Map from an Eye
What Exactly You are Looking At
Eye Image:
Computed Retinal Image:
Corneal Stereo System
left cornea
right cornea
epipolar plane
camera pupil
right/left
effective viewpoint
 C R  C L S Lp  C L

 L
R
L
L
 C C
S

C
p

epipolar curve

  S R t ,   C R  0

epipolar curve in 3D



From Two Eyes in an Image …
Y
X
Z
Y
X
Curvesand side view)
Reconstructed Epipolar
Structure (frontal
Eyes Reveal …
– Where the person is
– What the person is looking at
– The structure of objects
Implications
Human Affect Studies: Social Networks
Security: Human Localization
Advanced Interfaces: Robots, Computers
Computer Graphics: Relighting [SIGGRAPH 04]
VisualEyesTM
http://www.cs.columbia.edu/CAVE/
with Akira Yanagawa