SHAPE FROM SHADING
Mica Arie-Nachimson
May 2011
What is Shading?

Well… not shadow…

We can’t reconstruct shape from one shadow…
Image from www.moolf.com
What is Shading?

Variable levels of darkness

Gives a cue for the actual 3D shape
There is a relation between intensity and shape

Images from S. Narasimhan, Carnegie Mellon; www.erco.com; www.alesmav.com;
H. Wang, University of California
Shading Examples

These circles differ only in grayscale intensity

Intensities give a strong “feeling” of scene
structure
Talk Outline

Introduction and Basics
Notations
 Reflectance map
 Photometric stereo



Main Approaches
Propagation Solutions
Basic Solution by Horn [85]
 Using Fast Marching, Kimmel & Sethian [2001]


Global energy minimization solution using Belief
Propagation – only if time permits
Introduction and Basics
Problem definition
Notations
Reflectance Map
Photometric Stereo
What determines scene radiance?



The amount of light that falls on the surface
The fraction of light that is reflected (albedo)
Geometry of light reflection
 Shape
of surface
 Viewpoint
n

𝐼 = 𝜌𝑛 ∙ 𝑙
(𝜃𝑖 , 𝜑𝑖 )
(𝜃𝑒 , 𝜑𝑒 )
𝜌
Albedo 𝜌

A property of the surface



Can be modeled as a single scalar for the entire surface
or dependent on location
Usually assumed to be a known scalar


Usually normalized 1 ≥ 𝜌 ≥ 0
Is this a ball with uniform albedo or is this a 2D circle with
varying albedo?
We’re not going to talk about the albedo
Surface Normal
Convenient notation for surface orientation
 A smooth surface has a tangent plane at every
point

We can model the surface using the normal at
every point
The Shape From Shading Problem

Given a grayscale image
 And
albedo
 And light source direction

Reconstruct scene geometry
 Can
be modeled by surface normals
Lambertian Surface




Appears equally bright from all viewing directions
Reflects all light without absorbing
Matte surface, no “shiny” spots
Brightness of the surface as seen from camera is
linearly correlated to the amount of light falling on
the surface
Today we will discuss only
Lambertian surfaces under
point-source illumination
n
(𝜃𝑖 , 𝜑𝑖 )
(𝜃𝑒 , 𝜑𝑒 )
Some Notations:
Surface Orientation

A smooth surface has a tangent plane at every
point
 Mark

𝑝=
𝜕𝑧
𝜕𝑥
;𝑞 =
𝜕𝑧
𝜕𝑦
Parametrize surface orientation by first partial
derivatives of 𝑧
𝑞𝛿𝑦
𝑝𝛿𝑥
Some Notations:
Surface Orientation
𝑧
Surface normal
 𝑟𝑥 = 𝑝, 0,1 , 𝑟𝑦 = 0, 𝑞, 1


𝑟𝑥
𝑛 = 𝑟𝑥 × 𝑟𝑦 = 𝑝, 𝑞, −1
Normalize 𝑛 =
𝑛
𝑛
=
𝑟𝑦
(𝑝,𝑞,−1)
𝜕𝑧
𝜕𝑧
𝑝=
;𝑞 =
𝜕𝑥
𝜕𝑦
𝑝2 +𝑞2 +1
𝑞𝛿𝑦
𝑝𝛿𝑥
Reflectance Map
Relationship between surface orientation and
brightness
Lambertian surface
1 + 𝑝𝑠 𝑝 + 𝑞𝑠 𝑞
𝑅 𝑝, 𝑞 = 𝑛 ∙ 𝑙 =
1 + 𝑝2 + 𝑞2 1 + 𝑝𝑠 2 + 𝑞𝑠 2


Image irradiance (brightness) is proportional to 𝑅
𝐼 𝑥, 𝑦 = 𝑅(𝑝, 𝑞)
Reflectance Map Example

Brightness as a function of surface orientation
Lambertian surface
iso-brightness
q
contour
0 .8
0 .9
R p, q   0.7
1 .0
 pS , qS 
p
i  90
 ppS  qqS 1  0
𝑅 𝑝, 𝑞 =
1 + 𝑝𝑠 𝑝 + 𝑞𝑠 𝑞
0 .3
1 + 𝑝2 + 𝑞 2 1 + 𝑝𝑠 2 + 𝑞𝑠 2
Illustration from S. Narasimhan, Carnegie Mellon
0 .0
Reflectance Map of a Glossy Surface

Brightness as a function of surface orientation
Surface with diffuse and
glossy components
Reflectance Map Examples

Brightness as a function of surface orientation
Graphics with a 3D Feel



Given a 3D surface 𝑧 𝑥, 𝑦 , lighting and viewing
direction, we can compute the gray level of pixel
𝐼(𝑥, 𝑦) of the surface
Find the gradient of the surface (𝑝, 𝑞)
Use I x, y = 𝑅 𝑝, 𝑞 = 𝑛 ∙ 𝑙 =
determine the gray level
1+𝑝𝑠 𝑝+𝑞𝑠 𝑞
1+𝑝2 +𝑞2
1+𝑝𝑠 2 +𝑞𝑠 2
to
Shape From Shading?

Can we reconstruct shape from a single image?
q
p

Two variables 𝑝, 𝑞 with one equation… So what do
we do?
Illustration from S. Narasimhan, Carnegie Mellon
Shape From Shading!

Use more images
 Photometric

stereo
Shape from shading
 Introduce
constraints
 Solve locally
 Linearize problem
Images from
R. Basri et al. “Photometric Stereo with General, Unknown Lighting“, IJCV 2006
Photometric Stereo

Take several pictures of same object under same
viewpoint with different lighting
q
p , q 
1
1
S
S
p
Photometric Stereo

Take several pictures of same object under same
viewpoint with different lighting
q
p , q 
p , q 
2
2
S
S
1
1
S
S
p
Photometric Stereo

Take several pictures of same object under same
viewpoint with different lighting
q
p , q 
p , q 
2
2
S
S
1
1
S
S
p
p , q 
3
3
S
S
Main Approaches for
Shape From Shading
A Quick Review
Main Approaches

Minimization: Solve for a global error function while
introducing constraint(s):
Brightness (𝐼 − 𝑅)2 𝑑𝑥𝑑𝑦
 Smoothness
(𝑝𝑥 2 +𝑞𝑦 2 )𝑑𝑥𝑑𝑦
 Intensity gradient
((𝐼𝑥 − 𝑅𝑥 )2 +(𝐼𝑦 − 𝑅𝑦 )2 )𝑑𝑥𝑑𝑦
…

Main Approaches

Minimization: Solve for a global error function while
introducing constraint(s):
Brightness (𝐼 − 𝑅)2 𝑑𝑥𝑑𝑦
 Smoothness
(𝑝𝑥 2 +𝑞𝑦 2 )𝑑𝑥𝑑𝑦
 Intensity gradient
((𝐼𝑥 − 𝑅𝑥 )2 +(𝐼𝑦 − 𝑅𝑦 )2 )𝑑𝑥𝑑𝑦
…




Propagation: Grow a solution around an initial known
point or boundary
Local: Assume local surface type
Linear: Make problem linear by a linearization of
reflectance function
Main Approaches

Minimization: Solve for a global error function while
introducing constraint(s):
Brightness (𝐼 − 𝑅)2 𝑑𝑥𝑑𝑦
 Smoothness
(𝑝𝑥 2 +𝑞𝑦 2 )𝑑𝑥𝑑𝑦
 Intensity gradient
((𝐼𝑥 − 𝑅𝑥 )2 +(𝐼𝑦 − 𝑅𝑦 )2 )𝑑𝑥𝑑𝑦
…




Propagation: Grow a solution around an initial known
point or boundary
Local: Assume local surface type
Linear: Make problem linear by a linearization of
reflectance function
Main Approaches

Minimization: Solve for a global error function while
introducing constraint(s):
Brightness (𝐼 − 𝑅)2 𝑑𝑥𝑑𝑦
 Smoothness
(𝑝𝑥 2 +𝑞𝑦 2 )𝑑𝑥𝑑𝑦
 Intensity gradient
((𝐼𝑥 − 𝑅𝑥 )2 +(𝐼𝑦 − 𝑅𝑦 )2 )𝑑𝑥𝑑𝑦
…




Propagation: Grow a solution around an initial known
point or boundary
Local: Assume local surface type
Linear: Make problem linear by a linearization of
reflectance function
Basic Propagation Solution
Horn [85]
Solution by Characteristic Curves
Propagating Solution


Suppose that we know the depth of some point on
the surface
Extend the solution by taking a small step 𝛿𝑧 =
𝑝𝛿𝑥 + 𝑞𝛿𝑦
𝑝 and 𝑞 are unknown, and the image irradiance
equation gives only one constraint
 But

If we knew 𝑝 and 𝑞, can compute changes in 𝑝, 𝑞
using second partial derivatives
 𝛿𝑝

=
𝜕2 𝑧
𝜕2 𝑧
𝛿𝑥+ 𝜕𝑥𝜕𝑦𝛿𝑦
𝜕𝑥2
𝛿𝑝
𝛿𝑥
=𝐻
𝛿𝑦
𝛿𝑞
and 𝛿𝑞 =
𝐻=
𝜕2 𝑧
𝜕2 𝑧
𝛿𝑥+ 2 𝛿𝑦
𝜕𝑥𝜕𝑦
𝜕𝑦
𝜕2 𝑧
𝜕𝑥 2
𝜕2 𝑧
𝜕𝑥𝜕𝑦
𝜕2 𝑧
𝜕𝑥𝜕𝑦
𝜕2 𝑧
𝜕𝑦 2
Propagating Solution
𝛿𝑝
𝛿𝑥
=𝐻
, what else can we use?
𝛿𝑦
𝛿𝑞
 Image irradiance equation 𝑅 𝑝, 𝑞 = 𝐼 𝑥, 𝑦
𝜕𝐼 𝜕𝑅 𝜕𝑅 𝜕𝑝
𝜕𝑅 𝜕𝑞
𝐼𝑥 = 𝜕𝑥
= =
+
𝜕𝑥 𝜕𝑝 𝜕𝑥
𝜕𝑞 𝜕𝑥
𝐼𝑦 =
𝜕𝐼 𝜕𝑅 𝜕𝑅 𝜕𝑝
= =
𝜕𝑦 𝜕𝑦 𝜕𝑝 𝜕𝑦
𝑅𝑝
𝐼𝑥
=
𝐻
𝐼𝑦
𝑅𝑞
𝜕𝑞
+ 𝜕𝑅
𝜕𝑞 𝜕𝑦
Propagating Solution
𝑅𝑝
𝐼𝑥
𝛿𝑝
𝛿𝑥
=𝐻
;
=𝐻
𝐼
𝑅𝑞
𝛿𝑦
𝛿𝑞
𝑦
 We note that for a small step 𝛿𝜏
𝑅𝑝
𝛿𝑥
=
𝛿𝜏
𝑅𝑞
𝛿𝑦
 For these specific δ𝑥, 𝛿𝑦
𝑅𝑝
𝐼𝑥
𝛿𝑝
𝛿𝑥
=𝐻
=𝐻
𝛿𝜏 = 𝐼 𝛿𝜏
𝑅𝑞
𝛿𝑦
𝛿𝑞
𝑦

If you take a small step in the image plane parallel to
the direction of the gradient of 𝑅, then can compute
change in (𝑝, 𝑞)
Propagating Solution
𝐼𝑥
𝛿𝑝
= 𝐼 𝛿𝜏
𝛿𝑞
𝑦

;
Get ODEs:
𝜕𝑥
𝜕𝜏
𝜕𝑧
= 𝑅𝑝 ; 𝜕𝑦
=
𝑅
;
𝑞 𝜕𝜏 = 𝑝𝑅𝑝 + 𝑞𝑅𝑞 ;
𝜕𝜏
𝜕𝑝
𝜕𝜏

𝑅𝑝
𝛿𝑥
=
𝛿𝜏
𝑅𝑞
𝛿𝑦
= 𝐼𝑥 ; 𝜕𝑞
= 𝐼𝑦
𝜕𝜏
These equations describe a characteristic curve
(𝑥, 𝑦, 𝑧, 𝑝, 𝑞)
( 𝑥 + 𝛿𝑥, 𝑦 + 𝛿𝑦, 𝑧 + 𝛿𝑧, 𝑝 + 𝛿𝑝, 𝑞 + 𝛿𝑞 )
Propagating Characteristic Curve

Characteristic curve
(𝑥, 𝑦, 𝑧, 𝑝, 𝑞)
(𝑥 + 𝛿𝑥, 𝑦 + 𝛿𝑦, 𝑧 + 𝛿𝑧, 𝑝 + 𝛿𝑝, 𝑞 + 𝛿𝑞)

Need to initialize every curve at some known point
 Singular
points
 Occluding boundaries
Image by Dejan Todorović
Propagating Characteristic Curve

Characteristic curve
(𝑥, 𝑦, 𝑧, 𝑝, 𝑞)
(𝑥 + 𝛿𝑥, 𝑦 + 𝛿𝑦, 𝑧 + 𝛿𝑧, 𝑝 + 𝛿𝑝, 𝑞 + 𝛿𝑞)

Need to initialize every curve at some known point
 Singular
points
 Occluding boundaries

Curves are “grown” independently, very instable
Image by Dejan Todorović
Shape From Shading by Fast
Marching
“Optimal Algorithm for Shape from Shading and
Path Planning”, R. Kimmel and J. A. Sethian
[2001]
Vertical Light Source Case

Recall reflectance map
1 + 𝑝𝑠 𝑝 + 𝑞𝑠 𝑞
𝐼 𝑥, 𝑦 = 𝑅 𝑝, 𝑞 = 𝑛 ∙ 𝑙 =



1 + 𝑝2 + 𝑞2 1 + 𝑝𝑠 2 + 𝑞𝑠 2
Assume light source located near the viewer

𝑙=(0,0,1)

𝐼 𝑥, 𝑦 = 𝑅 𝑝, 𝑞 =
𝛻𝑧 =
1
𝐼2
1
1+𝑝2 +𝑞2
=
1
1+𝛻2 𝑧
−1
This is an Eikonal equation

Can be solved by an 𝑂 𝑁𝑙𝑜𝑔𝑁 numerical algorithm
Fast Marching
Fast Marching

Expanding Dijkstra’s shortest path algorithm for
general surfaces
 Represented
 Flat

as triangulated mesh
domains
Many computer vision problems can be set into
flat domains
 Every
pixel is a node, edges
between adjacent nodes
Dijkstra’s Shortest Paths
Connected graphs
 Start with the source node
9
6
2
14
11
10
9
s
7
15
Dijkstra’s Shortest Paths
Connected graphs
 Start with the source node
 Update all neighbors
9
14
14
6
2
9
9
s
11
10
15
7
7
Dijkstra’s Shortest Paths
Connected graphs
 Start with the source node
 Update all neighbors
 Go to the closest neighbor
 Set
9
14
14
6
2
9
9
its value
s
11
10
15
7
7
Dijkstra’s Shortest Paths
Connected graphs
 Start with the source node
 Update all neighbors
 Go to the closest neighbor
its value
 Compute its neighbors
 Update smaller scores
9
14
14
6
2
9
9
 Set
s
23
11
10
15
7
7
Dijkstra’s Shortest Paths
Connected graphs
 Start with the source node
 Update all neighbors
 Go to the closest neighbor
9
14
14
its value
 Compute its neighbors
 Update smaller scores
6
2
9
9
 Set

Continue with smallest value node
 Remember
path
s
23
11
10
15
7
7
Flat Domains: Why Does Dijkstra Fail?


Dijkstra will not find the
diagonal path
Need to examine triangles
Fast Marching: Problem Definition




Suppose there is a forest fire with multiple sources
Every point that was touched by the fire is burnt
and will not be visited by the fire again
Firemen register the time T(x) at which the fire
arrives to location x.
The fast marching algorithm simulates this
scenario
Fast Marching: Problem Definition




Multiple sources
Advancing forward
Advances either at a constant rate 𝑑𝑇(𝑥,𝑦)
= 1 or at
𝑑𝑥
varying rate 𝑑𝑇(𝑥,𝑦)
= 𝑓(𝑥, 𝑦)
𝑑𝑥
What is the arrival time at every location?
Image from G. Rosman, Technion
Fast Marching Algorithm






Set 𝑇 𝑥0 = 0 and mark the
points as done
Set the rest of the points as
∞ and mark them as far
All far points adjacent to
done points become verify
points
Update all verify points using
𝑇 of the done set
The verify with the smallest
𝑇 becomes done
Continue until all points are
done
Illustrations from selected publications of J. A. Sethian
Fast Marching on a Grid
Update step
Tij
?
Ti,j-1
Ti-1,j
Slide based on slides by R. Kimmel, Technion
Ti+1,j
i+1,j
i,j-1
ij
i-1,j
i,j+1
Ti,j+1
Fast Marching on a Grid
Update step
𝑇𝑖𝑗 − min 𝑇𝑖−1,𝑗 , 𝑇𝑖+1,𝑗 , 𝑇𝑖𝑗
Tij
2
+ 𝑇𝑖𝑗 − min 𝑇𝑖,𝑗−1 , 𝑇𝑖,𝑗+1 , 𝑇𝑖𝑗
2
=
ℎ2 𝑓𝑖𝑗 2
Ti,j-1
Ti-1,j
Slide based on slides by R. Kimmel, Technion
?
Ti+1,j
i+1,j
i,j-1
ij
i-1,j
i,j+1
Ti,j+1
Fast Marching on a Grid
Update step
𝑇1
𝑇𝑖𝑗 − min 𝑇𝑖−1,𝑗 , 𝑇𝑖+1,𝑗 , 𝑇𝑖𝑗
Tij
2
+ 𝑇𝑖𝑗 − min 𝑇𝑖,𝑗−1 , 𝑇𝑖,𝑗+1 , 𝑇𝑖𝑗
𝑇2
Slide based on slides by R. Kimmel, Technion
2
=
ℎ2 𝑓𝑖𝑗 2
?
Ti,j-1
Ti-1,j
Ti+1,j
i+1,j
i,j-1
ij
i-1,j
i,j+1
Ti,j+1
Fast Marching on a Grid

Initialization: all 𝑇𝑖𝑗 = ∞ or known initial value

Update step:
𝑇𝑖𝑗 − 𝑇1
 Fit
2
+ 𝑇𝑖𝑗 − 𝑇2
2
= ℎ2 𝑓𝑖𝑗 2
Tij
a triangle with gradient 𝑓𝑖𝑗
and two determined values at
neighboring grid points
Ti,j-1
Ti-1,j
Slide based on slides by R. Kimmel, Technion
Ti+1,j
i+1,j
i,j-1
ij
i-1,j
i,j+1
Ti,j+1
Update Step Details

Update step is a quadratic equation:
𝑇𝑖𝑗 − 𝑇1

Solution: 𝑇𝑖𝑗 =
1
2
2
+ 𝑇𝑖𝑗 − 𝑇2
2
= ℎ2 𝑓𝑖𝑗 2
2
𝑇1 + 𝑇2 + 2ℎ2 𝑓𝑖𝑗 − 𝑇1 − 𝑇2
Disregard the “minus” solution
which yields 𝑇𝑖𝑗 < 𝑇1 , 𝑇2
2
Update Step Details

Update step is a quadratic equation:
𝑇𝑖𝑗 − 𝑇1
1
2
2
+ 𝑇𝑖𝑗 − 𝑇2
Solution: 𝑇𝑖𝑗 =

If 𝑇1 − 𝑇2 > ℎ𝑓𝑖𝑗 , then 𝑇𝑖𝑗 <
𝑇1 > 𝑇2 and 𝑇1 > ℎ𝑓𝑖𝑗 + 𝑇2

𝑇𝑖𝑗 < 𝑇1
= ℎ2 𝑓𝑖𝑗 2
2
𝑇1 + 𝑇2 + 2ℎ2 𝑓𝑖𝑗 − 𝑇1 − 𝑇2


2
1
2
𝑇1 + 𝑇2 + ℎ𝑓𝑖𝑗
2
Update Step Details

Update step is a quadratic equation:
𝑇𝑖𝑗 − 𝑇1
1
2
2
+ 𝑇𝑖𝑗 − 𝑇2
Solution: 𝑇𝑖𝑗 =

If 𝑇1 − 𝑇2 > ℎ𝑓𝑖𝑗 , then 𝑇𝑖𝑗 <
𝑇1 > 𝑇2 and 𝑇1 > ℎ𝑓𝑖𝑗 + 𝑇2

𝑇𝑖𝑗 < 𝑇1
= ℎ2 𝑓𝑖𝑗 2
2
𝑇1 + 𝑇2 + 2ℎ2 𝑓𝑖𝑗 − 𝑇1 − 𝑇2


2
1
2
𝑇1 + 𝑇2 + ℎ𝑓𝑖𝑗
2
Update Step Details

Update step is a quadratic equation:
𝑇𝑖𝑗 − 𝑇1
1
2
2
+ 𝑇𝑖𝑗 − 𝑇2
Solution: 𝑇𝑖𝑗 =

If 𝑇1 − 𝑇2 > ℎ𝑓𝑖𝑗 , then 𝑇𝑖𝑗 <
𝑇1 > 𝑇2 and 𝑇1 > ℎ𝑓𝑖𝑗 + 𝑇2


2
1
2
2
𝑇1 + 𝑇2 + ℎ𝑓𝑖𝑗
𝑇𝑖𝑗 < 𝑇1
𝑇2 > 𝑇1 and 𝑇2 > ℎ𝑓𝑖𝑗 + 𝑇1


= ℎ2 𝑓𝑖𝑗 2
𝑇1 + 𝑇2 + 2ℎ2 𝑓𝑖𝑗 − 𝑇1 − 𝑇2


2
𝑇𝑖𝑗 < 𝑇2
This means that the wave front propagation comes from outside
the triangle

Only choice is to update 𝑇𝑖𝑗 = min 𝑇1 , 𝑇2 + ℎ𝑓𝑖𝑗
Update Step Details

Update step is a quadratic equation:
𝑇𝑖𝑗 − 𝑇1

+ 𝑇𝑖𝑗 − 𝑇2
2
= ℎ2 𝑓𝑖𝑗 2
If 𝑇1 − 𝑇2 < ℎ𝑓𝑖𝑗


2
𝑇𝑖𝑗 =
1
2
Else
 𝑇𝑖𝑗
2
𝑇1 + 𝑇2 + 2ℎ2 𝑓𝑖𝑗 − 𝑇1 − 𝑇2
= min 𝑇1 , 𝑇2 + ℎ𝑓𝑖𝑗
2
Shape From shading
Vertical Light Source Case

Eikonal equation
𝛻𝑧 =

− 1 = 𝑓(𝑥, 𝑦)
Numerical approximation of this equation
max
𝑧𝑖𝑗 −𝑧𝑖−1,𝑗 𝑧𝑖𝑗 −𝑧𝑖+1,𝑗
, ∆𝑥 , 0
∆𝑥
+ max

1
𝐼(𝑥,𝑦)2
2
𝑧𝑖𝑗 −𝑧𝑖,𝑗−1 𝑧𝑖𝑗 −𝑧𝑖,𝑗+1
, ∆𝑦 , 0
∆𝑦
Assume w.l.o.g. that ∆𝑥 = ∆𝑦 = 1
2
= 𝑓𝑖𝑗 2
Shape From Shading
Vertical Light Source Case

Eikonal equation
𝛻𝑧 =

1
𝑅2
− 1 = 𝑓(𝑥, 𝑦)
Numerical approximation of this equation
max 𝑧𝑖𝑗 − 𝑧𝑖−1,𝑗 , 𝑧𝑖𝑗 − 𝑧𝑖+1,𝑗 , 0
+ max 𝑧𝑖𝑗 − 𝑧𝑖,𝑗−1 , 𝑧𝑖𝑗 − 𝑧𝑖,𝑗+1 , 0

Assume w.l.o.g. that ∆𝑥 = ∆𝑦 = 1
2
2
= 𝑓𝑖𝑗 2
Vertical Light Source Case
Derive Update Step
2
max 𝑧𝑖𝑗 − 𝑧𝑖−1,𝑗 , 𝑧𝑖𝑗 − 𝑧𝑖+1,𝑗 , 0
+ max 𝑧𝑖𝑗 − 𝑧𝑖,𝑗−1 , 𝑧𝑖𝑗 − 𝑧𝑖,𝑗+1 , 0
𝑧1
2
2
𝑧𝑖𝑗 − min 𝑧𝑖−1,𝑗 , 𝑧𝑖+1,𝑗 , 𝑧𝑖𝑗
+ 𝑧𝑖𝑗 − min 𝑧𝑖,𝑗−1 , 𝑧𝑖,𝑗+1 , 𝑧𝑖𝑗
2
= 𝑓𝑖𝑗
𝑧2
𝑧𝑖𝑗 − 𝑧1
2
+ 𝑧𝑖𝑗 − 𝑧2
= 𝑓𝑖𝑗 2
2
= 𝑓𝑖𝑗
2
2
Vertical Light Source Case
Derive Update Step
𝑧𝑖𝑗 − 𝑧1
2
+ 𝑧𝑖𝑗 − 𝑧2
2
= 𝑓𝑖𝑗 2
Where:
𝑧1 = min 𝑧𝑖−1,𝑗 , 𝑧𝑖+1,𝑗
𝑧2 = min 𝑧𝑖,𝑗−1 , 𝑧𝑖,𝑗+1

If 𝑓𝑖𝑗 > 𝑧1 − 𝑧2
𝑧𝑖𝑗 =
𝑧1 + 𝑧2 + 2𝑓𝑖𝑗 2 − 𝑧1 − 𝑧2
Otherwise 𝑧𝑖𝑗 = min 𝑧1 , 𝑧2
2
+ 𝑓𝑖𝑗
2
Vertical Light Source Case
Fast Marching



Fast marching introduces order to these update
steps
Points are updated from small to large
Total complexity: 𝑂(𝑁 log 𝑁)
 log 𝑁
for the selection of smallest point and update of
neighboring points
 𝑁 points (pixles)
General Light Source

Recall reflectance map
𝑅 𝑝, 𝑞 = 𝑛 ∙ 𝑙 =


1 + 𝑝𝑠 𝑝 + 𝑞𝑠 𝑞
1 + 𝑝2 + 𝑞2 1 + 𝑝𝑠 2 + 𝑞𝑠 2
In this case 𝛻𝑧 = 𝑓 𝑥, 𝑦, 𝑝, 𝑞 = 𝑓 𝑥, 𝑦, 𝑧 𝑥, 𝑦
 The right hand side depends on 𝑧
This is not an Eikonal equation anymore
 How can we solve this?
Solve in the light source coordinates, then
change the variables back
General Light Source
Change Coordinate System

Before moving to light source coordinate system,
choose w.l.o.g light source direction to be
2
2
𝑙 = 𝑙1 , 0, 𝑙3 , where 𝑙1 +𝑙3 = 1
 This will simplify our equations a bit
General Light Source
Change Coordinate System


𝑙3
𝑥
𝑦 = 0
𝑧
−𝑙1
0 𝑙1
1 0
0 𝑙3

𝛻𝑧 =
− 1 = 𝑓(𝑥, 𝑦)
Eikonal equation!
𝒍𝟏
𝟎
𝒍𝟑
𝑥
𝑙1
𝑦
𝑧
𝐼 𝑥, 𝑦 = 𝐼 𝑥, 𝑦 = 𝐼(𝑙3 𝑥 + 𝑙1 𝑧, 𝑦)
1
𝐼(𝑥,𝑦)2
𝒍=
𝑙3
𝑥
𝑦
𝑧
Light source coordinates marked with ~

𝑧
Change coordinate system
𝒍
𝒛
𝑥
𝑦
𝒙=
𝒍𝟑
𝟎
−𝒍𝟏
General Light Source
Solution in New Coordinate System

But…
𝐼 𝑥, 𝑦 = 𝐼 𝑥, 𝑦 = 𝐼(𝑙3 𝑥 + 𝑙1 𝑧, 𝑦)
 We need to find 𝑧 in order to find the intensity
Use the smallest 𝑧 value from all neighbors of 𝑧𝑖𝑗
General Light Source Solution
Update Step
𝑧𝑖𝑗 − 𝑧1
2
+ 𝑧𝑖𝑗 − 𝑧2
2
= 𝑓𝑖𝑗 2
Where:
𝑧1 = min 𝑧𝑖−1,𝑗 , 𝑧𝑖+1,𝑗 , 𝑧𝑖𝑗
𝑧2 = min 𝑧𝑖,𝑗−1 , 𝑧𝑖,𝑗+1 , 𝑧𝑖𝑗

𝑘 = 𝑙3 𝑖 + 𝑙1 min 𝑧1 , 𝑧2

If 𝑓𝑘𝑗 > 𝑧1 − 𝑧2
𝑧𝑖𝑗 =
𝑧1 + 𝑧2 + 2𝑓𝑘𝑗 2 − 𝑧1 − 𝑧2
2
Otherwise 𝑧𝑖𝑗 = min 𝑧1 , 𝑧2 + 𝑓𝑘𝑗
2
Results


Results shown on
synthetic shading
image
𝑙 = (0.2,0,0.96)
Minimization Using Efficient
Belief Propagation
“Efficient Belief Propagation for Vision Using
Linear Constraint Nodes”, B. Potetz [2007]
Belief Propagation

Estimate the marginals of a multivariate probability
distribution

𝑝 𝑋 =
𝜑𝑖 (𝑥𝑖 ),
𝑥𝑖 ⊂ 𝑋
 𝜑𝑖 (𝑥𝑖 ) potential function
 𝑥𝑖 consists of several elements

Usually represented as a factor graph
Factor Graph




𝑝 𝑋 = 𝜑𝑖 (𝑥𝑖 ),
𝑥𝑖 ⊂ 𝑋
Bipartite graph
Every element of 𝑋 is represented by a variable
node
Every potential function 𝜑𝑖 𝑥𝑖 represented by a
factor node 𝑓, connected to all variable nodes of 𝑥𝑖
circles: variable nodes
squares: factor nodes
Belief Propagation on a Factor Graph


𝑝 𝑋 = 𝜑𝑖 (𝑥𝑖 ),
𝑥𝑖 ⊂ 𝑋
Estimate the marginals
𝑏

𝑥𝑖 =
𝑋∖𝑥𝑖 𝑝(𝑋)
Iteratively compute messages along edges of the
factor graph
 Every
variable node computes its marginal probability
according to the probabilities its neighbors sent

Usually converges to a good enough local
minimum
Shape From Shading Using a
Factor Graph

Recall I x, y = 𝑅 𝑝, 𝑞 = 𝑛 ∙ 𝑙
1+𝑝𝑠 𝑝+𝑞𝑠 𝑞
=
2
2
2
2
1+𝑝 +𝑞

1+𝑝𝑠 +𝑞𝑠
Constraint I: Consistent integral
𝐷𝑖𝑗 −𝑥 𝑧 − 𝐷𝑖𝑗 −𝑦 𝑧 + 𝐷𝑖𝑗 +𝑥 𝑧 − 𝐷𝑖𝑗 +𝑦 𝑧 = 0
Where 𝐷𝑖𝑗 −𝑥 𝑧 = 𝑧 𝑖, 𝑗 − 𝑧 𝑖 − 1, 𝑗
Integral of the surface gradient along a closed curve
should be zero
Illustration from Klaus & Horn, “Robot Vision”, MIT Press, 1986
Shape From Shading Using a
Factor Graph

Constraint II: Lambertian constraint
𝜑𝐿 𝑝, 𝑞 = 𝑃 𝑝, 𝑞 𝐼
Where 𝑃(𝑝, 𝑞|𝐼) =
𝑐𝑜𝑛𝑠𝑡 𝐼 𝑥, 𝑦 = 𝑅(𝑝, 𝑞)
0
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
 𝜑𝐿
𝑝, 𝑞 could be changed for other lighting
conditions

Constraint III: Shape prior
 Modeled
by a Laplace distribution
𝑒𝑥𝑝(− 𝜎∆𝑝 ), 𝑒𝑥𝑝(− 𝜎∆𝑞 )
1
2
Shape From Shading Using a
Factor Graph

Constraint I: Consistent integral
𝐷𝑖𝑗 −𝑥 𝑧 − 𝐷𝑖𝑗 −𝑦 𝑧 + 𝐷𝑖𝑗 +𝑥 𝑧 − 𝐷𝑖𝑗 +𝑦 𝑧 = 0

Constraint II: Lambertian constraint
𝜑𝐿 𝑝, 𝑞 = 𝑃 𝑝, 𝑞 𝐼
Constraint III: Shape prior

𝑒𝑥𝑝(− 𝜎∆𝑝 ), 𝑒𝑥𝑝(− 𝜎∆𝑞 )
1
2
Variable nodes
Factor node, consistent integral
Factor node, Lambertian constraint
Factor node, shape prior
one pixel one pixel
one pixel one pixel one pixel
Shape From Shading Using a
Factor Graph

Constraint I: Consistent integral
𝐷𝑖𝑗 −𝑥 𝑧 − 𝐷𝑖𝑗 −𝑦 𝑧 + 𝐷𝑖𝑗 +𝑥 𝑧 − 𝐷𝑖𝑗 +𝑦 𝑧 = 0

Constraint II: Lambertian constraint
𝜑𝐿 𝑝, 𝑞 = 𝑃 𝑝, 𝑞 𝐼
Constraint III: Shape prior

𝑒𝑥𝑝(− 𝜎∆𝑝 ), 𝑒𝑥𝑝(− 𝜎∆𝑞 )
1
2
Variable nodes
Factor node, consistent integral
Factor node, Lambertian constraint
Factor node, shape prior
Results

Input:
original surface

Rendering of original
surface with 𝑙 = (1,0,1)
Input image
Result:
recovered surface
Rendering of recovered
surface with 𝑙 = (1,0,1)
Conclusion

Shape from shading problem
 Definition
 Main
difficulties
 Main approaches

Propagating solutions
 Horn
[85]
 Kimmel [2001]

Minimization with BP while introducing constraints
Thank You!