Introduction to Computer Vision
Week 9, Fall 2010
Instructor: Prof. Ko Nishino
Radiometry and Reflectance
Image Intensities
source
image intensity
surface
normal
sensor
surface element
Image intensity understanding is an under-constrained problem!
Radiometric Concepts
Solid Angle
: Solid Angle subtended by
: Foreshortened Area
What is the solid angle subtended by a hemisphere?
Radiometric Concepts
flux
Radiant Intensity
Light flux (power) emitted per unit solid angle
: Flux
Surface Irradiance
Light flux (power) incident per unit surface area
Does not depend on where the light is coming from
Radiometric Concepts
Surface Radiance (Brightness)
Light flux (power) emitted
per unit foreshortened area per unit solid angle
d"
L=
(dA cos# r )d$
2
W/m
str )
(
foreshortened area
!
•  L depends on direction
•  Surface can radiate into whole hemisphere
•  L is proportional to irradiance E
•  Depends on reflectance properties of surface
: Flux
Radiometric Image Formation
(Scene Radiance to Image Irradiance)
image plane
Image Irradiance:
Solid angles:
lens
scene
Scene Radiance:
Radiometric Image Formation
(Scene Radiance to Image Irradiance)
image plane
Image Irradiance:
Solid Angle subtended by lens
lens
scene
Scene Radiance:
Radiometric Image Formation
(Scene Radiance to Image Irradiance)
image plane
lens
Image Irradiance:
Flux received by lens from
Scene Radiance:
= Flux projected onto
(image)
d" = L( dAs cos# ) d$ L
Image Irradiance
!
scene
Radiometric Image Formation
(Scene Radiance to Image Irradiance)
We have
Solid Angles:
Flux:
Solid Angle subtended by lens:
Image Irradiance:
We get
image irradiance
scene radiance
•  Image irradiance is proportional to scene radiance
•  Telephoto lenses have narrow field of view
effects of
are small
Bi-directional Reflectance Distribution
Function (BRDF)
!  We saw Surface Radiance to Image Irradiance
!  Let’s see Surface Irradiance to Surface Radiance!
: Irradiance due to source in direction
: Radiance of surface in direction
BRDF:
Rotationally symmetric BRDF:
(isotropic)
Isotropic and Anisotropic BRDF
real image
isotropic BRDF model
anisotropic BRDF model
by Ward
BRDF Properties
BRDF: f (" i , # i ;" r , # r ) =
L(" r , # r )
E (" i , # i )
!  Conservation of energy
!
% f (" ,# ;" ,# )d$
i
i
r
r
i
&1
hemisphere
!  Helmholtz Reciprocity
!
!
f (" i , # i ;" r , # r ) = f (" r , # r ;" i , # i )
Reflectance Model
Reflection: An Electromagnetic Phonomenon
Two approaches to derive Reflectance Models
"  Physical Optics (Wave Optics)
"  Geometrical Optics (Ray Optics)
Geometrical models are approximations to physical models
But they are easier to use!
Reflectance that Require Wave Optics
Reflection
surface reflection
surface
non-homogeneous medium
(pigments)
Body Reflection
!  Diffuse Reflection
!  Matte Appearance
!  Non-Homogeneous Medium
body reflection
scattering &
absorption
Surface Reflection
!  Specular Reflection (Highlights)
!  Glossy Appearance
!  Dominant for Metals
Image Intensity = Diffuse Component + Specular Component
Diffuse and Specular Reflection
diffuse
specular
diffuse+specular
Reflectance Models
!  Lambertian Model (Diffuse Component)
Surface appears equally bright from ALL directions
Jean Henri Lambert
Surface Radiance is proportional to Surface Irradiance
albedo
Note:
source radiance
is independent of
!
Reflectance Models
!  Ideal Specular Model (Mirror Reflection)
"  Very SMOOTH surface
"  All incident energy reflected in a single direction
perfect mirror direction
(
L = k" (# i $ # e )" (% i + & ) $ % e
Viewer receives light only when
!
s = 2(n" r )n # r
)
Reflectance Models
!  Torrance-Sparrow Model
"  Specular reflection from rough surfaces
incident light
reflection
Micro-facet Orientation Model (example)
Gaussian Model (isotropic)
p(" ) =
1
e
2#$ n
%
1"2
2 $ 2n
reflection
micro-facet
: roughness parameter
micro-facet’s
incident
light
Reflectance Models
!  Torrance-Sparrow Model
"  Masking and Shadowing Effects
masked
perfect mirror direction
shadowed
geometric factor
(masking and shadowing)
"s
L=
p( $)G(s,n,v)
n# v
: angle between
(half-vector of
and ) and
Unified Reflectance Model
!  Primary Reflection Components
"  Diffuse Lobe (Lambertian)
"  Specular Lobe (Torrance-Sparrow)
"  Specular Spike (Beckmann-Spizzichino)
Lambertian + Torrance-Sparrow
(G.O)
(G.O)
(P.O)
& "d
)
"s
L = k( n$ s +
p( %)G(s,n,v)+
'#
*
n$ v
Reflectance Model
real image
isotropic BRDF model
anisotropic BRDF model
by Ward
Dichromatic Reflectance Model
surface reflection
surface
body reflection
non-homogeneous medium
(pigments)
Color of body (diffuse) component
= Color of Object X Color of Illumination
Color of surface (specular) component
= Color of Illumination
Dichromatic Reflectance Model
diffuse
specular
diffuse+specular
Photometric Stereo
Image Intensity and Geometry
!  Shading as a cue for shape reconstruction
!  What is the relation of intensity and shape?
"  Reflectance Map
Surface Normal
surface normal
Equation of plane
or
Let
(note that –Horn)
Surface normal
Surface Normal
Gradient Space
Normal vector
Source vector
plane is called the Gradient Space (pq plane)
•  Every point in it corresponds to a particular surface orientation
Reflectance Map
!  Relates image irradiance I(x,y) to surface orientation (p,q)
for given source direction and surface reflectance
!  Lambertian case
: source brightness
: surface albedo (reflectance)
: constant (optical system)
Image irradiance:
Let
then
Reflectance Map
!  Lambertian case
Iso-brightness contour
Reflectance Map
(Lambertian)
cone of constant
Reflectance Map
!  Lambertian case
iso-brightness
contour
Note:
is maximum when
Reflectance Map
!  Glossy surfaces (Torrance-Sparrow reflectance model)
"d
" skc
I=
kc cos $ i +
p( %)G = R( p,q)
#
cos $ r
diffuse term
specular term
Diffuse peak
!
Specular peak
Single-Image Shape Recovery?
!  Given a single image of an object with known surface
reflectance taken under a known light source, can we
recover the shape of the object?
!  Given R(p,q) ( (pS,qS) and surface reflectance) can we
determine (p,q) uniquely for each image point?
NO
Solution
!  Take more images
"  Photometric stereo
!  Add constraints
"  Shape-from-shading
Photometric Stereo
Photometric Stereo
Lambertian :
Image irradiance:
!  Can write this as a matrix equation:
Solving the Equations
inverse
More than Three Lights
!  Get better results by using more lights
!  Least squares solution:
"1
n˜ = (S S) ST I
T
!  Solve for
!
as before
Moore-Penrose pseudo inverse
Color Images
!  The case of RGB images
"  get three sets of equations, one per color channel:
"  Simple solution: first solve for
"  Then substitute known
using one channel
into above equations to
get
"  Or combine three channels and solve for
Computing light source directions
!  Trick: place a chrome sphere in the scene
"  the location of the highlight tells you where the
light source is
Recall the Rule for Specular Reflection
!  For a perfect mirror, light is reflected about N
!  We see a highlight when
!  then
is given as follows:
s = 2(n" r )n # r
Computing the Light Source Direction
Chrome sphere that has a highlight at position h in the image
N
H
h
rN
C
c
sphere in 3D
image plane
!  Can compute N by studying this figure
"  Hints:
"  use this equation:
"  can measure c, h, and r in the image
Chrome Sphere Images and Mask
Depth from Normals
V2
V1
N
!  Get a similar equation for V2
"  Each normal gives us two linear constraints on z
"  compute z values by solving a matrix equation (project 3)
Limitations
!  Big problems
"  Doesn’t work for shiny things, semi-translucent things
"  Shadows, inter-reflections
!  Smaller problems
"  Camera and lights have to be distant
"  Calibration requirements
"  measure light source directions, intensities
"  camera response function
Trick for Handling Shadows
!  Weight each equation by the pixel brightness:
!  Gives weighted least-squares matrix equation:
!  Solve for
as before
Project 3: Photometric Stereo
1.
2.
3.
4.
5.
Estimate light source directions
Compute surface normals
Compute albedo values
Estimate depth from surface normals
Relight the object (with original texture and uniform albedo)
Extra Credits
!  Better integration of surface normals
"  Frankot & Chellapa
!  Novel views
!  Robustness
"  Better handling of shadows and specularities
Shape-from-Shading
Single-Image Shape Recovery?
!  Given a single image of an object with known surface
reflectance taken under a known light source, can we
recover the shape of the object?
!  Given R(p,q) ( (pS,qS) and surface reflectance) can we
determine (p,q) uniquely for each image point?
q
p
NO
Ambiguity in Human Perception
by V. Ramachandran!
Solution
!  Take more images
"  Photometric stereo
!  Add constraints
"  Shape-from-shading
Stereographic Projection
(p,q)-space (gradient space)
(f,g)-space
z
S
s
p
1
N
!
z =1
ŝ
q
n
z =1
n̂
g
n
y
x
x
Problem
(p,q) can be infinite when
1
s
f
y
z
!1
! = 90!
f =
2p
1+ 1+ p2 + q2
g=
2q
1+ 1+ p2 + q2
Redefine reflectance map as
R( f , g )
Occluding Boundaries
e
n
v
n
n # e, n # v " n = e ! v
e and v are known
The n values on the occluding boundary can be used as
the boundary condition for shape-from-shading
e
Smoothness Constraint
!  Used to constrain shape-from-shading
!  Relates orientations (f,g) of neighboring surface points
Minimize
es =
2
2
2
2
f
+
f
+
g
+
g
"" ( x y ) ( x y )dxdy
image
( f , g ) : surface orientation under stereographic projection
!
fx =
!f
!f
!g
!g
, f y = , gx =
, gy =
!x
!y
!x
!y
(penalize rapid changes in surface orientation f and g over the image)
Image Irradiance Constraint
!  Image irradiance should match the reflectance map
Minimize
ei =
2
## (I( x, y ) " R( f ,g)) dxdy
image
(minimize errors in image irradiance in the image)
!
Shape-from-Shading
!  Find surface orientations (f,g) at all image points that
minimize
weight
e = es + "ei
smoothness
constraint
image irradiance
error
!
Minimize
e=
$$ ( f
image
2
x
+f
2
y
) + (g
2
x
)
2
+ g + " ( I ( x, y ) # R( f ,g)) dxdy
2
y
Calculus of Variations
Minimize
e=
"" F ( f ,g, f , f ,g ,g )dxdy
x
y
x
y
image
(
2
x
F= f + f
2
y
) + (g
2
x
+g
2
y
) + #(I( x, y ) $ R( f ,g))
Euler equations for F
#
#
F f " F f x " F f y = 0,
#x
#y
!
2
(read Robot Vision A.6)!
#
#
Fg " Fg x " Fg y = 0
#x
#y
Euler equations for shape-from-shading
%R
" f = #$ ( I ( x, y ) # R( f ,g)) ,
%f
2
!
%R
" g = #$ ( I ( x, y ) # R( f ,g))
%g
2
Solve this coupled pair of second-order partial differential equations
with the occluding boundary conditions!
!
Numerical Shape-from-Shading (Ikeuchi & Horn 89)
!  Smoothness error at image point (i,j)
2
2
1#
si, j = % f i+1, j " f i, j + f i, j +1 " f i, j + gi+1, j " gi, j
4$
(
) (
) (
2
) + (g
i, j +1
" gi, j
Of course you can consider more neighbors (smoother results)
!  Image irradiance error at image point (i,j)
(
(
ri, j = Ii, j " R f i, j ,gi, j
!
Find
!
))
2
{ f i, j } and {gi, j } that minimize
(
e = # # si, j + "ri, j
i
!
!
j
)
)
2
&(
'
Numerical Shape-from-Shading (Ikeuchi & Horn 89)
Find
{ f i, j } and {gi, j } that minimize
e = # # ( si, j + "ri, j )
i
If f k ,l and g k ,l minimize e , then
!
!
!
"e
"e
= 0,
=0
"f k,l
"gk,l
"e
"R
= 2( f k,l # f k,l ) # 2 $ Ik,l # R( f k,l ,gk,l )
"f k,l
"f
(
)
!
=0
f k,l
"e
"R
= 2( gk,l # gk,l ) # 2 $ Ik,l # R( f k,l ,gk,l )
=0
"gk,l
"g g k,l
(
!
j
)
where f k ,l and g k ,l are 4-neighbors average around image point (k,l)
1
f k,l = f i+1, j + f i, j +1 + f i"1, j + f i, j "1
8
1
gk,l = gi+1, j + gi, j +1 + gi"1, j + gi, j "1
8
(
(
!
!
)
)
Numerical Shape-from-Shading (Ikeuchi & Horn 89)
Minimize
"e
"R
= 2( f k,l # f k,l ) # 2 $ Ik,l # R( f k,l ,gk,l )
"f k,l
"f
(
)
=0
f k,l
"e
"R
= 2( gk,l # gk,l ) # 2 $ Ik,l # R( f k,l ,gk,l )
=0
"gk,l
"g g k,l
(
)
Update rule
(
)
$R
$f
)
$R
$g g
f k,l = f k,l + " Ik,l # R( f k,l ,gk,l )
!
n +1
n
!
(
g k,l = g k,l + " Ik,l # R( f k,l ,gk,l )
n +1
!
n
f k,l
k,l
!  Use known ( f , g ) values on the occluding boundary to constrain
the solution (boundary conditions)
!
n +1
n +1
n
n
!  Compare (f k ,l , g k ,l ) with (f k ,l , g k ,l ) for convergence test
!  As the solution converges, increase ! to remove the smoothness
constraint
Results
by Ikeuchi and Horn!
Results
Scanning Electron Microscope image!
(inverse intensity)!
by Ikeuchi and Horn!
Results: Tsai and Shah
SFS w/ 1/r^2 term
Emmanuel Prados and Olivier Faugerus