Ronen Basri David Jacobs
Weizmann NEC

Lighting affects appearance

Lighting => infinite DOFs.
Set of possible images infinite dimensional
(Belhumeur and Kriegman)
But, in many cases, lighting => 9 DOFs.

Convex, Lambertian objects: 9D linear space captures >98% of reflectance.
Explains previous empirical results.
(Epstein, Hallinan and Yuille; Hallinan;
Belhumeur and Kriegman)
For lighting, justifies low-dim methods.
Simple, analytic form.
=> New recognition algorithms.

Lambertian
No cast shadows
Lights far and isotropic n q l l l max (cos q
, 0)

Lighting
Reflectance
Images
...

1
0.5
r
2
1.5
1
0.5
0
0
0
0
1
1
2
2
3
3

+
+
+
(See D’Zmura, ‘91; Ramamoorthi and Hanrahan ‘00)

Orthonormal basis for functions on the sphere
Funk-Hecke convolution theorem
Rotation = Phase Shift n
’th order harmonic has
2 n +1 components.

1.5
A n
0.5
1 0.886
0.591
0.222
0
0 1 2 3
0.037
4 n
5
0.014
6 7
0.007
8

Reflectance functions near low-dimensional linear subspace r
 k
 l
 n

 

0
 n
 m n
( K nm
L nm
) h nm
 n
2  
  
0 m n n
( K nm
L nm
) h nm
Yields 9D linear subspace.

Accuracy depends on lighting.
For point source: 9D space captures 99.2% of energy
For any lighting: 9D space captures >98% of energy.

b nm
( p )
 l r nm
( X , Y , Z ) l l Z l
X l
Y
2 l
( Z
2 
X
2 
Y
2
) l
( X
2 
Y
2
) l
XY l
XZ l
YZ

Normals present to varying amounts.
Albedo makes some pixels more important.
Worst case approximation arbitrarily bad.
“Average” case approximation should be good.

Models
Query
Find Pose
Compare
Vector: I
Harmonic Images
Matrix: B

Linear: min a
Ba

I
Non-negative light: min a
T
BH a
(See Georghides, Belhumeur and Kriegman)

I , a

0
Non-negative light, first order approximation: min a
Ba

I , 4 a
0
2  a
1
2 + a
2
2
+ a
3
2

Shashua. With no shadows, i=l l n min a
|| B a

I || with B = [
First harmonic, no DC l
X, l
Y, l
Z].
Koenderink & van Doorn heuristically suggest using l too.

PCA on many images
Amano, Hiura, Yamaguti, and Inokuchi; Atick and Redlich; Bakry, Abo-Elsoud, and Kamel; Belhumeur, Hespanha, and Kriegman; Bhatnagar, Shaw, and Williams;
Black and Jepson; Brennan and Principe; Campbell and Flynn; Casasent, Sipe and
Talukder; Chan, Nasrabadi and Torrieri; Chung, Kee and Kim; Cootes, Taylor,
Cooper and Graham; Covell; Cui and Weng; Daily and Cottrell; Demir, Akarun, and Alpaydin; Duta, Jain and Dubuisson-Jolly; Hallinan; Han and Tewfik; Jebara and Pentland; Kagesawa, Ueno, Kasushi, and Kashiwagi; King and Xu; Kalocsai,
Zhao, and Elagin; Lee, Jung, Kwon and Hong; Liu and Wechsler; Menser and
Muller; Moghaddam; Moon and Philips; Murase and Nayar; Nishino, Sato, and
Ikeuchi; Novak, and Owirka; Nishino, Sato, and Ikeuchi; Ohta, Kohtaro and
Ikeuchi; Ong and Gong; Penev and Atick; Penev and Sirivitch; Lorente and
Torres; Pentland, Moghaddam, and Starner; Ramanathan, Sum, and Soon; Reiter and Matas; Romdhani, Gong and Psarrou; Shan, Gao, Chen, and Ma; Shen, Fu,
Xu, Hsu, Chang, and Meng; Sirivitch and Kirby; Song, Chang, and Shaowei;
Torres, Reutter, and Lorente; Turk and Pentland; Watta, Gandhi, and Lakshmanan;
Weng and Chen; Yuela, Dai, and Feng; Yuille, Snow, Epstein, and Belhumeur;
Zhao, Chellappa, and Krishnaswamy; Zhao and Yang.

Space built analytically
Size and accuracy known
More efficient
O ( ps
2
) vs.
O ( pr
2
) time, s

100 , r

9
When pose unknown, rendering and PCA done at run time.

3-D Models of 42 faces acquired with scanner.
30 query images for each of
10 faces ( 300 images).
Pose automatically computed using manually selected features (Blicher and Roy).
Best lighting found for each model; best fitting model wins.

9D Linear Method: 90% correct.
9D Non-negative light: 88% correct.
Ongoing work: Most errors seem due to pose problems. With better poses, results seem near 100% .

We characterize images object produces.
Useful for recognition with 3D model.
Also tells us how to generalize from images.