776 Computer
Vision
Jan-Michael Frahm
Spring 2012
Scheduling
• January 20th is the first Friday class
• 3:30pm to 4:45 pm in SN 115
Capturing light
Source: A. Efros
Light transport
slide: R. Szeliski
What is light?
Electromagnetic radiation (EMR) moving along rays in space
• R(l) is EMR, measured in units of power (watts)
– l is wavelength
Light field
• We can describe all of the light in the scene by specifying the
radiation (or “radiance” along all light rays) arriving at every point
in space and from every direction
slide: R. Szeliski
slide: R. Szeliski
What is light?
Electromagnetic radiation (EMR) moving along rays in space
• R(l) is EMR, measured in units of power (watts)
– l is wavelength
Perceiving light
• How do we convert radiation into “color”?
• What part of the spectrum do we see?
slide: R. Szeliski
The visible light spectrum
• We “see” electromagnetic radiation in a range of
wavelengths
slide: R. Szeliski
Light spectrum
• The appearance of light depends on its power spectrum
o How much power (or energy) at each wavelength
daylight
tungsten bulb
Our visual system converts a light spectrum into “color”
• This is a rather complex transformation
slide: R. Szeliski
Brightness contrast and constancy
• The apparent brightness depends on the surrounding region
o brightness contrast: a constant colored region seem lighter or
darker depending on the surround:
• http://www.sandlotscience.com/Contrast/Checker_Board_2.htm
o brightness constancy: a surface looks the same under widely
varying lighting conditions.
slide: R. Szeliski
Light response is nonlinear
• Our visual system has a large dynamic range
o We can resolve both light and dark things at the same time
o One mechanism for achieving this is that we sense light intensity on a
logarithmic scale
• an exponential intensity ramp will be seen as a linear ramp
o Another mechanism is adaptation
• rods and cones adapt to be more sensitive in low light, less sensitive in
bright light.
After images
• Tired photoreceptors
o Send out negative response after a strong stimulus
http://www.sandlotscience.com/Aftereffects/Andrus_Spiral.htm
Light transport
slide: R. Szeliski
Light sources
• Basic types
o point source
o directional source
• a point source that is infinitely far away
o area source
• a union of point sources
slide: R. Szeliski
from Steve Marschner
from Steve Marschner
The interaction of light and matter
• What happens when a light ray hits a point on an object?
o Some of the light gets absorbed
• converted to other forms of energy (e.g., heat)
o Some gets transmitted through the object
• possibly bent, through “refraction”
o Some gets reflected
• as we saw before, it could be reflected in multiple directions at once
• Let’s consider the case of reflection in detail
o In the most general case, a single incoming ray could be
reflected in all directions. How can we describe the amount of
light reflected in each direction?
slide: R. Szeliski
The BRDF
• The Bidirectional Reflection Distribution Function
o Given an incoming ray
and outgoing ray
what proportion of the incoming light is reflected along
outgoing ray?
surface normal
Answer given by the BRDF:
slide: R. Szeliski
BRDFs can be incredibly complicated…
slide: S. Lazebnik
Constraints on the BRDF
• Energy conservation
o Quantity of outgoing light ≤ quantity of incident light
• integral of BRDF ≤ 1
• Helmholtz reciprocity
o reversing the path of light produces the same reflectance
=
slide: R. Szeliski
Diffuse reflection
• Diffuse reflection
o Dull, matte surfaces like chalk or latex paint
o Microfacets scatter incoming light randomly
o Effect is that light is reflected equally in all directions
slide: R. Szeliski
Diffuse reflection
Diffuse reflection governed by Lambert’s law
• Viewed brightness does not depend on viewing direction
• Brightness does depend on direction of illumination
• This is the model most often used in computer vision
Lambert’s Law:
L, N, V unit vectors
Ie = outgoing radiance
Ii = incoming radiance
BRDF for Lambertian surface
slide: R. Szeliski
Specular reflection
For a perfect mirror, light is reflected about N
I i
Ie  
0
if V  R
otherwise
Near-perfect mirrors have a highlight around R
• common model:
slide: R. Szeliski
Specular reflection
Moving the light source
Changing ns
slide: R. Szeliski
Phong illumination model
• Phong approximation of surface reflectance
o Assume reflectance is modeled by three components
• Diffuse term
• Specular term
• Ambient term (to compensate for inter-reflected light)
L, N, V unit vectors
Ie = outgoing radiance
Ii = incoming radiance
Ia = ambient light
ka = ambient light reflectance factor
(x)+ = max(x, 0)
slide: R. Szeliski
BRDF models
• Phenomenological
o
o
o
o
Phong [75]
Ward [92]
Lafortune et al. [97]
Ashikhmin et al. [00]
• Physical
o Cook-Torrance [81]
o Dichromatic [Shafer 85]
o He et al. [91]
• Here we’re listing only some well-known examples
slide: R. Szeliski
Measuring the BRDF
traditional
design by Greg Ward
• Gonioreflectometer
o Device for capturing the BRDF by moving a camera + light source
o Need careful control of illumination, environment
slide: R. Szeliski
BRDF databases
• MERL (Matusik et al.): 100 isotropic, 4 nonisotropic, dense
• CURET (Columbia-Utrect): 60 samples, more sparsely sampled,
but also bidirectional texure functions (BTF)
slide: R. Szeliski
Image formation
• How bright is the image of a scene point?
slide: S. Lazebnik
Radiometry: Measuring light
• The basic setup: a light source is sending radiation to
a surface patch
• What matters:
o How big the source and the patch “look” to each other
source
patch
slide: S. Lazebnik
Solid Angle
• The solid angle subtended by a region at a point is
the area projected on a unit sphere centered at
that point
o Units: steradians
• The solid angle dw subtended by a patch of area dA
is given by:
dAcosq
dw =
2
r
A
slide: S. Lazebnik
Radiance
• Radiance (L): energy carried by a ray
o Power per unit area perpendicular to the direction of travel,
per unit solid angle
o Units: Watts per square meter per steradian (W m-2 sr-1)
n
dω
P
L
dA cos  dw
θ
dA
P  L dA cos  dw
dA cos
slide: S. Lazebnik
Radiance
• The roles of the patch and the source are essentially
symmetric
dA2
θ2
P  L dA1 cos 1 dw2
 L dA2 cos  2 dw1
r
θ1
L dA1 dA2 cos 1 cos  2

r2
dA1
slide: S. Lazebnik
Irradiance
• Irradiance (E): energy arriving at a surface
o Incident power per unit area not foreshortened
o Units: W m-2
o For a surface receiving radiance L coming in from dw the
corresponding irradiance is
n
dω
θ
dA
P L dA cos  dw
E

dA
dA
 L cos  dw
slide: S. Lazebnik
Radiometry of thin lenses
• L: Radiance emitted from P toward P’
• E: Irradiance falling on P’ from the lens
What is the relationship between E and L?
slide: S. Lazebnik
Radiometry of thin lenses
z
| OP |
cos 
z'
| OP ' |
cos 
dA
dA’
o
Area of the lens:
d2
4
d2 
The power δP received by the lens from P is P  L
 4  cos  w


P
L
The radiance emitted from the lens towards P’ is
2
d 

 cos  w
The irradiance received at P’ is
 4 
2

  d 2 cos      d 
4
     cos   L
E  L cos  
2 

 4 ( z ' / cos  )   4  z ' 
Solid angle subtended by the lens at P’
slide: S. Lazebnik
Radiometry of thin lenses
  d 2

4
E     cos   L
 4  z ' 

• Image irradiance is linearly related to scene
radiance
• Irradiance is proportional to the area of the lens
and inversely proportional to the squared distance
between the lens and the image plane
• The irradiance falls off as the angle between the
viewing ray and the optical axis increases
slide: S. Lazebnik
From light rays to pixel values
X  E  t
  d 2

4
E     cos   L
 4  z ' 

Z  f E  t 
• Camera response function: the mapping f from
irradiance to pixel values
o Useful if we want to estimate material properties
o Enables us to create high dynamic range images
Source: S. Seitz, P. Debevec
From light rays to pixel values
X  E  t
  d 2

4
E     cos   L
 4  z ' 

Z  f E  t 
• Camera response function: the mapping f from
irradiance to pixel values
For more info
• P. E. Debevec and J. Malik. Recovering High Dynamic Range Radiance Maps
from Photographs. In SIGGRAPH 97, August 1997
Source: S. Seitz, P. Debevec
Photometric stereo (shape from shading)
• Can we reconstruct the shape of an object based
on shading cues?
Luca della Robbia,
Cantoria, 1438
Photometric stereo
• Assume:
o A Lambertian object
o A local shading model (each point on a surface receives light only from
sources visible at that point)
o A set of known light source directions
o A set of pictures of an object, obtained in exactly the same
camera/object configuration but using different sources
o Orthographic projection
• Goal: reconstruct object shape and albedo
S2
Sn
S1
???
Forsyth & Ponce, Sec. 5.4
slide: S. Lazebnik
Surface model: Monge patch
Forsyth & Ponce, Sec. 5.4
Image model
• Known: source vectors Sj and pixel values Ij(x,y)
• We also assume that the response function of the
camera is a linear scaling by a factor of k
• Combine the unknown normal N(x,y) and albedo
ρ(x,y) into one vector g, and the scaling constant k
and source vectors Sj into another vector Vj:
I j ( x, y )  k B ( x, y )
 k   x, y N  x, y   S j 
   x, y N  x, y   ( k S j )
 g ( x, y )  V j
slide: S. Lazebnik
Least squares problem
• For each pixel, we obtain a linear system:
T

I
(
x
,
y
)
 1

V1 
 I ( x, y )   T 
 2
  V2  g ( x, y )

   


  T
 I n ( x, y )  Vn 
(n × 1)
known
(n × 3)
known
(3 × 1)
unknown
• Obtain least-squares solution for g(x,y)
• Since N(x,y) is the unit normal, (x,y) is given by the
magnitude of g(x,y) (and it should be less than 1)
• Finally, N(x,y) = g(x,y) / (x,y)
slide: S. Lazebnik
Example
Recovered albedo
Recovered normal field
Forsyth & Ponce, Sec. 5.4
Recovering a surface from normals
•Recall the surface is
written as
(x, y, f (x, y))
•This means the normal
has the form:
 f x 



1
N(x, y)   2
 fy 

2
 f x  f y  1  
 1 
•If we write the estimated
vector g as
g1 (x, y)
g(x, y)  g2 (x, y)


g3 (x, y)
•Then we obtain values
for the partial derivatives
of the surface:
f x (x, y)  g1 (x, y) g3 (x, y)
f y (x, y)  g2 (x, y) g3(x, y)
slide: S. Lazebnik
Recovering a surface from normals
•Integrability: for the
surface f to exist, the
mixed second partial
derivatives must be
equal:
g1 (x, y) g3 (x, y)

y
g2 (x, y) g3 (x, y)
x
(in practice, they should at least
be similar)
•We can now recover the
surface height at any
point by integration along
some path,
x e.g.
f (x, y)   f x (s, y)ds 
0
y
 f (x,t)dt  c
y
0
(for robustness, can take integrals
over many different paths and
average the results)
slide: S. Lazebnik
Surface recovered by integration
Forsyth & Ponce, Sec. 5.4