CS4670: Computer Vision
Noah Snavely
Lecture 22: Computational photography 1: HDR
The ultimate camera
What does it do?
The ultimate camera
Infinite resolution
Infinite zoom control
Desired object(s) are in focus
No noise
No motion blur
Infinite dynamic range (can see dark and bright things)
...
Creating the ultimate camera
The “analog” camera has changed very little in >100 yrs
• we’re unlikely to get there following this path
More promising is to combine “analog” optics with
computational techniques
• “Computational cameras” or “Computational photography”
This lecture will survey techniques for producing higher
quality images by combining optics and computation
Common themes:
• take multiple photos
• modify the camera
Noise reduction
Take several images and
average them
Why does this work?
Basic statistics:
•
variance of the mean
decreases with n:
Field of view
We can artificially increase the field of view by
compositing several photos together (project 2).
Improving resolution: Gigapixel images
Max Lyons, 2003
fused 196 telephoto shots
A few other notable examples:
• Obama inauguration (gigapan.org)
• HDView (Microsoft Research)
What do we see?
Vs.
Problem: Dynamic Range
1
The real world is
high dynamic range.
1500
25,000
400,000
2,000,000,000
Long Exposure
10-6
Real world
High dynamic range
10-6
106
106
Picture
0 to 255
Short Exposure
10-6
Real world
High dynamic range
10-6
106
106
Picture
0 to 255
Image
pixel (312, 284) = 42
42 photons?
Dynamic Range
Typical cameras have limited dynamic range
HDR images — merge multiple inputs
Pixel count
Scene Radiance
HDR images — merged
Pixel count
Radiance
Camera is not a photometer!
Limited dynamic range
• 8 bits captures only 2 orders of magnitude of light intensity
• We can see ~10 orders of magnitude of light intensity
Unknown, nonlinear response
• pixel intensity  amount of light (# photons, or “radiance”)
Solution:
• Recover response curve from multiple exposures, then
reconstruct the radiance map
Camera pipeline
Camera pipeline
Camera pipeline
Camera pipeline
Camera pipeline
Camera pipeline
Z  f ( E  t )
8 bits
Imaging system response function
255
Pixel
value
0
log Exposure = log (Radiance * t)
(CCD photon count)
Real-world response functions
In general, the response function is not provided
by camera makers who consider it part of their
proprietary product differentiation. In addition,
they are beyond the standard gamma curves.
Camera is not a photometer
• Limited dynamic range
 Perhaps use multiple exposures?
• Unknown, nonlinear response
 Not possible to convert pixel values to radiance
• Solution:
– Recover response curve from multiple exposures,
then reconstruct the radiance map
Varying exposure
• Ways to change exposure
– Shutter speed
– Aperture
– Neutral density filters
Shutter speed
• Note: shutter times usually obey a power series –
each “stop” is a factor of 2
• ¼, 1/8, 1/15, 1/30, 1/60, 1/125, 1/250, 1/500,
1/1000 sec
Usually really is:
¼, 1/8, 1/16, 1/32, 1/64, 1/128, 1/256, 1/512,
1/1024 sec
HDRI capturing from multiple exposures
• We want to obtain the response curve
HDRI capturing from multiple exposures
Image series
•
1 •
• 2
•
1 •
• 2
•
1 •
• 2
•
1 •
• 2
3
t =
2 sec
3
t =
1 sec
3
3
t =
1/2 sec
t =
1/4 sec
Z ij  f ( E i t j )
f 1 (Zij )  Ei t j
g (Z ij )  ln Ei  ln t j , where g  ln f 1
•1
•2
•
3
t =
1/8 sec
Idea behind the math
g ( Z ij )  ln Ei  ln t j
ln Ei  ln t j
Z ij
Idea behind the math
g ( Z ij )  ln Ei  ln t j
ln Ei  ln t j
Z ij
Idea behind the math
g ( Z ij )  ln Ei  ln t j
ln Ei  ln t j
Z ij
Math for recovering response curve
Recovering response curve
• The solution can be only up to a scale, add a
constraint
• Add a hat weighting function
Sparse linear system




n×p




1 



254 

256
n



 g(0)

:
 g(255)
 lnE  
: 
: 
 lnE 




Ax=b
1
n


























Matlab code
Matlab code
function [g,lE]=gsolve(Z,B,l,w)
n = 256;
A = zeros(size(Z,1)*size(Z,2)+n+1,n+size(Z,1));
b = zeros(size(A,1),1);
k = 1;
%% Include the data-fitting equations
for i=1:size(Z,1)
for j=1:size(Z,2)
wij = w(Z(i,j)+1);
A(k,Z(i,j)+1) = wij; A(k,n+i) = -wij; b(k,1) = wij * B(i,j);
k=k+1;
end
end
A(k,129) = 1;
k=k+1;
%% Fix the curve by setting its middle value to 0
for i=1:n-2
%% Include the smoothness equations
A(k,i)=l*w(i+1); A(k,i+1)=-2*l*w(i+1); A(k,i+2)=l*w(i+1);
k=k+1;
end
x = A\b;
%% Solve the system using SVD
g = x(1:n);
lE = x(n+1:size(x,1));
Recovered response function
Constructing HDR radiance map
combine pixels to reduce noise and obtain a
more reliable estimation
Varying shutter speeds
Reconstructed radiance map
What is this for?
• Human perception
• Vision/graphics applications
Assorted pixel (Single Exposure HDR)
Assorted pixel
Assorted pixel
Pixel with Adaptive Exposure Control
light
attenuator
element
detector
element
Tt+1
controller
It
Tone Mapping
• How can we do this?
Linear scaling?, thresholding? Suggestions?
10-6
Real World
Ray Traced
World (Radiance)
High dynamic range
10-6
106
106
Display/
Printer
0 to 255
The
Radiance
Map
Linearly scaled to
display device
Simple Global Operator
• Compression curve needs to
– Bring everything within range
– Leave dark areas alone
• In other words
– Asymptote at 255
– Derivative of 1 at 0
Global Operator (Reinhart et al)
Lworld
Ldisplay 
1  Lworld
Global Operator Results
Reinhart Operator
Darkest 0.1% scaled
to display device
Questions?