Inverse Volume Rendering with Material
Dictionaries
Ioannis Gkioulekas1 Shuang Zhao2
Todd Zickler1
1Harvard
1
Kavita Bala2
Anat Levin3
2Cornell
3Weizmann
Most materials are translucent
2
food
skin
jewelry
architecture
Photo credit: Bei Xiao, Ted Adelson
We know how to render them
Monte-Carlo
rendering
?
material
parameters
3
rendered image
Veach 1997, Dutré et al. 2006
We show how to measure them
inverse
rendering
material
parameters
rendered
image
captured
photograph
4
Our contributions
1. exact inverse volume rendering
material
• with arbitrary phase functions!
2. validation with calibration materials
known
parameters
thin
thick
nondilutable
solids
3. database of broad range of materials
5
Why is inverse rendering so hard?
random walk of
photons inside
volume
radiative transfer
• volume light transport has
very complex dependence
material parameters
material sample
6
thin
thick
nondilutable
solids
Light transport approximations
random walk of
single-bounce
photons inside
random walk
volume
Previous approach:
single-scattering
Narasimhan et al. 2006
thin
7
nondilutable
 
 
thick
solids
Light transport approximations
isotropic
random
walk of
distribution
of
photons
inside
photons
volume
Previous approach:
diffusion
…
…
…
…
Jensen et al. 2001
parameter ambiguity
material 1
material 2
8
≈ ≠
thin
nondilutable
Papas et al. 2013
 
 
thick
solids
Inverse rendering without approximations
random walk of
exact inversion
photons inside
of random walk
volume
thin
9
nondilutable
 
 
thick
solids
Our approach
appearance matching
i. material representation
ii. operator-theoretic analysis
iii. stochastic optimization
10
Background
random walk of
photons inside
medium
θ
m = (σt σs p(θ))
11
extinction coefficient σt
scattering coefficient σs
phase function p(θ)
Phase function parameterization
Henyey-Greenstein lobes
Previous approach:
single-parameter families
Chen et al. 2006
Donner et al. 2008
Fuchs et al. 2007
g ∈ −1,1
Goesele et al. 2004
Gu et al. 2008
Hawkins et al. 2005
not general enough
Gkioulekas et al. 2013
Holroyd et al. 2011
Jensen et al. 2001
McCormick et al. 1981
Narasimhan et al. 2006
Papas et al. 2013
Pine et al. 1990
Prahl et al. 1993
12
Wang et al. 2008
Dictionary parameterization
tent phase functions
dictionary of materials
phase functions
DD=={m
{p11,, m
p22,, …,
…, pmQQ} }
pD11
1
2
3
4
5
6
7
8
9
10
π6 5π4 π3
π
π
π
7
8
π
π10
π11
9
p
• arbitrary materials
phase functions
π2
π1
m
p = Σ q πq m
pqq
• similarly for σt and σs
σt = Σq πq σt,q σs = Σq πq σs,q
13
Our approach
appearance matching
m = Σq π q mq
i. material representation
ii. operator-theoretic analysis
iii. stochastic optimization
14
Operator-theoretic analysis
random walk of
photons inside
medium
discretized random walk paths
• propagation step τ
τ τ τ
m = (σt σs p(θ))
15
Operator-theoretic analysis
radiance at all
medium points
and directions
discretized random walk paths
• propagation step τ
Ln+1(x, θ) = K Ln(x, θ)
• total radiance radiance
radiance
after n
after n+1
L = n Ln = (I - K)-1 Linput
steps
steps
Σ
L(x, θ)
m = (σt σs p(θ))
• rendering operator R
L(x, θ) = R Linput(x, θ)
dictionary representation:
m = Σq πq mq
K(π) = Σq πq Kq
R(π)= (I - Σ q πq Kq)-1
16
Our approach
appearance matching
m = Σq π q mq
i. material representation
R(π)= (I - Σ q πq Kq)-1 ii. operator-theoretic analysis
iii. stochastic optimization
17
Stochastic optimization
appearance matching
min ǁ photo - render(π) ǁ2
π
analytic operator expression for gradient!
𝜕loss π
𝜕πq
= render(π) · single-stepq · render(π)
R(π)
Kq
R(π)
• gradient descent optimization for inverse rendering
18
Stochastic optimization
• exact gradient descent
for k = 1, …, N,
πk
end
19
=
πk - 1
-
ak
𝜕loss π
𝜕πq
𝜋𝑘−1
N = a few hundreds
*
exact several CPU hours
=
intractable
Stochastic optimization
Monte-Carlo rendering to compute
102 samples
noisy + fast
20
104 samples
𝜕loss π
𝜕πq
𝜋𝑘−1
106 samples
accurate + slow
Stochastic optimization
• exact gradient descent
for k = 1, …, N,
πk
=
πk - 1
-
ak
𝜕loss π
𝜕πq
𝜋𝑘−1
end
N = a few hundreds
*
exact several CPU hours
=
intractable
• stochastic gradient descent
for k = 1, …, N,
πk
end
21
=
πk - 1
-
𝜕loss π
k
a
𝜕πq
𝜋𝑘−1
N = a few hundreds
*
noisy few CPU seconds
=
solvable
Theory wrap-up
appearance matching
min ǁ photo - render(π) ǁ2
π
m = Σq π q mq
i. material representation
R(π)= (I - Σ q πq Kq)-1 ii. operator-theoretic analysis
noisy
22
𝜕loss π
𝜕πq
iii. stochastic optimization
Our contributions
1. exact inverse volume rendering
material
• with arbitrary phase functions!
2. validation with calibration materials
known
parameters
thin
thick
nondilutable
solids
3. database of broad range of materials
23
Measurements
appearance matching
min ǁ photo - render(π) ǁ2
π
multiple lighting
multiple viewpoints
24
Acquisition setup
material sample
frontlighting
camera
25
backlighting
Acquisition setup
material
sample backlighting
sample
frontlighting material
frontlighting
camera
backlighting
top rotation stage
camera
26
stage
bottom rotationbottom rotation
top rotation
stage
stage
Validation
calibration materials
medium material
particle material
%
Mie
theory
known
parameters
size
very precise dispersions
(NIST Traceable Standards)
polystyrene
monodispersions
27
aluminum oxide
polydispersions
Frisvad et al. 2007
Parameter accuracy
comparison of ground-truth and measured parameters
p(θ)
θ
-π
polystyrene 1
polystyrene 2
all parameters estimated
within 4% error
28
0
π
polystyrene 3
aluminum oxide
ground-truth
measured
Henyey-Greenstein fit
Matching novel measurements
comparison of captured and rendered images
captured
rendered
rendered with HG
polystyrene 3
images under unseen geometries
predicted within 5% RMS error
29
profiles
ground-truth
measured
Henyey-Greenstein fit
Our contributions
1. exact inverse volume rendering
material
• with arbitrary phase functions!
2. validation with calibration materials
known
parameters
thin
thick
nondilutable
solids
3. database of broad range of materials
30
Measured materials
hand cream
shampoo
olive oil
robitussin
curacao
mixed soap
whole milk
mustard
31
material
R
whole milk
100.92
reduced milk 57.291
mustard
16.447
wine
shampoo
8.111
hand cream
20.82
coffee
liquid clay
37.544
milk soap
7,625
mixed soap
3.923
glycerine soap 0.201
robitussin
0.009
coffee
0.054
olive oil
0.041
blue curacao
0.01
red wine
0.015
extinction
absorption
G
B
R
G
B
105.345 102.768 0.013 0.013 0.041
62.46 63.757 0.007 0.007 0.024
milk 0.061
soap 0.451
18.536
6.457 0.057
9.919 10.575 0.178
0.328 0.439
liquid clay
32.353 41.798 0.011 0.011 0.012
reduced
48.25 67.949
0.004milk
0.004 0.005
8.004
8.557 0.003 0.004 0.015
4.018
4.351 0.003 0.005 0.013
0.202
0.221 0.001 0.001 0.002
0.001
0.001 0.012 0.197 0.234
0.051
0.049 0.275 0.309 0.406
0.039
0.012 0.062 0.047 0.353
0.012
0.021 0.083 0.048 0.011
0.013
0.011 0.122 0.351 0.402
thin
nondilutable
first moment
R
G
B
0.954 0.963 0.946
0.954 0.957 0.942
0.155 0.173 0.351
0.907 0.882 0.874
0.188 0.247 0.265
0.312 0.442 0.512
0.164 0.167 0.155
0.33 0.322 0.316
0.955 0.949 0.943
0.906 0.977 0.98
0.911 0.899 0.906
0.946 0.954 0.975
0.955 0.973 0.979
0.947 0.975 0.977
thick
solids
Measured phase functions
whole milk
reduced milk
mustard
shampoo
hand cream
liquid clay
milk soap
mixed soap
glycerine soap
robitussin
p(θ)
θ
coffee
33
olive oil
curacao
wine
-π
0
π
measured
Henyey-Greenstein fit
Synthetic images
mixed soap
glycerine soap
olive oil
curacao
rendered image
34
whole milk
Synthetic images
chromaticity
35
Synthetic images
mixed soap
glycerine soap
olive oil
curacao
rendered image
36
whole milk
Effect of phase function
measured phase function
Henyey-Greenstein fit
chromaticity
rendered image
p(θ)
mixed soap
θ
37
-π
0
π
measured
Henyey-Greenstein fit
Discussion
• more interesting materials: more general solids,
heterogeneous volumes, fluorescing materials
• other setups: alternative lighting (basis, adaptive, highfrequency), geometries, or imaging (transient imaging)
• faster capture and convergence: trade-offs between
accuracy, generality, mobility, and usability
38
Take-home messages
1. exact inverse volume rendering
material
• with arbitrary phase functions!
2. validation with calibration materials
known
parameters
thin
thick
nondilutable
solids
3. database of broad range of materials
39
Acknowledgements
• Henry Sarkas (Nanophase)
• Wenzel Jakob (Mitsuba)
Funding:
•
•
•
•
•
•
National Science Foundation
European Research Council
Binational Science Foundation
Feinberg Foundation
Intel
Amazon
Database of measured materials:
http://tinyurl.com/sa2013-inverse
40
Error surface
appearance matching
41
min ǁ photo - render(π) ǁ2
π
Light generation
MEMS light switch
RGB combiner
42
green (535 nm)
laser
blue (480 nm) laser
red (635 nm)
laser