What can be Known about the
Radiometric Response from Images?
Michael Grossberg and Shree Nayar
CAVE Lab, Columbia University
Partially funded by NSF ITR Award
ECCV Conference
May, 2002, Copenhagen, Denmark
Radiometric Response Function
Response
function:
f(I)=u
Scene Radiance: R
Image Plane
Irradiance: I
Response: u
0 255
g(u)=I
I
Irradiance
Inverse
response
function: g
u
Response = Gray-level
Response Recovery from Images
What is recovered?
Images at different
exposures
Inverse Radiometric
Response, g
Correspondence of graylevels between images
Exposure
Ratios
Gray-levels:
Image D
k3
Gray-levels:
k2
Image C
k1
Gray-levels:
uB
Image B
Gray-levels:
uA
Image A
Irradiance
What is measured? What is needed?
I
u
Response
Exposure Ratios
k1
k2
k3
Recovery Algorithms: S. Mann and R. Picard, 1995, P. E. Debevec, and
J. Malik, 1997, T. Mitsunaga S. K. Nayar 1999, S. Mann 2001,
Y. Tsin, V. Ramesh and T. Kanade 2001
How is Radiometric Calibration Done?
Images at Different Exposures
Corresponding Gray-levels
Inverse Response g,
Exposure Ratio k
Irradiance
I
Geometric
Correspondences
Response
Recovery
Algorithms
We eliminate the need for geometric
correspondences:
Static Scenes
Dynamic Scenes
k1
k2
u
k3
We find:
• All ambiguities in recovery
• Assumptions that break them
Constraint Equations
IB
 Constraint on irradiance I:
Brighter
image
 IB= kIA
 Constraint on g:
 g(uB)=kg(uA)
Filter
IA
Darker
image
 Brightness Transfer Function T:
 uB=T(uA)
 Constraint on g in terms of T
g(T(uA))=kg(uA)
T
How Does the Constraint Apply?
I
kg(uA)
=
g(T(uA))
g(T(uA))
kg(uA)
Irradiance
 Exposure ratio k
known
 Constraint makes
curve self-similar
1
1/k
g(uA)
u
0
uA T-1(1)
Gray-levels
T(uA)
1
Self-Similar Ambiguity:
Can We Recover g?
I
Constraint gives
no information in
[T-1(1),1]
Regularity
assumptions break
ambiguity
Known k: only
Self-similar
ambiguity
1
and
copy
Irradiance
 Conclusions:
Choose anything here
1/k
1/k2
1/k3
u
0
T-1(1)
Gray-levels
1
Exponential Ambiguity:
Can We Recover g and k ?
Exposure
ratio
k=21/3
I
Irradiance
k=21/2
γ=1/3
γ=1/2
γ=1
γ=2
γ=3
Response
k=2
k=22
k=23
U
Gray-level Image B
Inverse Response Function gγ
Brightness Transfer Function T
T(M)=2M
Gray-level Image A
T(u) = g -1(kg(u)) = g -γ(k- γg γ(u)) = T(u)
We cannot disambiguate (gγ, kγ) from (g, k) using T!
Obtaining the Brightness Transfer
Function (S. Mann, 2001)
2D-Gray-level
Histogram
Regression
Gray-level Image A
Scenes must be static.
Brightness Transfer
Function
Gray-level Image B
Registered Static Images at Different
Exposures
Gray-level Image A
Brightness Transfer Function without
Registration
Brightness Histograms
Different Exposures
Gray-level Image B
Histogram
Specification
Brightness Transfer
Function
Gray-level Image B
Unregistered Images at
Gray-level Image A
Gray-level Image A
Scenes may have motion.
How does Histogram Specification
Work?
Cumulative Area
(Fake Irradiance)
Histogram Equalization
Histogram Equalization
Histogram Specification
Gray-levels in Image A
Gray-levels in Image B
Histogram Specification = Brightness Transfer Function
Results: Object Motion
0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Red Response
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Recovered Response
Macbeth Chart Data
0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Green Response
Irradiance
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Irradiance
Irradiance
Recovered Inverse Radiometric Response Curves
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Blue Response
Results:
Object and Camera
Motion
Irradiance
Recovered Inverse
Radiometric Response Curves
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Irradiance
Red Response
Recovered
Response
Macbeth Chart
Data
Green Irradiance
Blue Irradian
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Green Response
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Irradiance
Red Irradiance
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Blue Response
Conclusions: What can be Known about
Inverse Response g from Images?
 A1:
Exposure ratio
k known
Recovery
of g from T
Exposure ratio
k unknown
Self-similar Ambiguity
Self-similar Ambiguity
+
Exponential Ambiguity
Need assumptions
on g and k to
recover g
 A2: In theory, we can recover exposure ratio directly
from Brightness Transfer Function T
 A3: Geometric correspondence step eliminated allowing
recovery in dynamic scenes: