HDR Image Construction from Multi-exposed Stereo
LDR Images
Ning Sun, Hassan Mansour, Rabab Ward
Proceedings of 2010 IEEE 17th International Conference on Image Processing
September 26-29, 2010, Hong Kong
Andy {andrey.korea@gmail.com}
Algorithm description
Two LDR images with
different exposures
Camera
response
function
Radiance
maps of LDR
images
Refined
disparity map
HDR image
Initial disparity map
Main concept:
1. Multi-exposed stereo images are captured using identical cameras placed adjacent to each other
on a horizontal line.
2. Stereo matching is then used to find a disparity map that matches each pixel in one image to the
corresponding pixel in another image.
3. A subset of the matched pixels is used to generate the camera response function which in turn is
used to generate the scene radiance map for each view with an expanded dynamic range.
4. The disparity map is refined by performing a second stereo matching stage using the radiance
maps
2
Intelligent Systems Lab.
Imaging models
Imaging models are used to determine the scene radiance from the measured pixel data
Gamma-correction model
Left image
Il  R
Polynomial camera response
Right image

Scene radiance
Correction factor
I r  R e

Left image
Right image


n
n
J cn     cn I l  p   e cn I r  p 
pP  n
n

Exposure ration
between images
Scene radiance
Exposure ration
between images
cn  arg min J cn 
3
Intelligent Systems Lab.
Computing the disparity map
f *  arg min Ed  f   ES  f , N 
Best disparity map
f F
Set of feasible disparities
Dissimilarity term
Pixel dissimilarity
Disparity smoothness
Ed  f    Dp  f p    1  NCC  p, f p 
p
Smoothing term
Es  f , N   
   p, qV
p qN p
p
p,q
Used for initial disparity estimation
4
Intelligent Systems Lab.
Pixel dissimilarity
NCC  p, f p  
W  p  - Search window centered on p
~ ~
w
w
I
 l r l q I r q  f p 
qW  p 
2
~
wl I l  p 
wt 
2
~
wr I r  p  f p 
- Bilateral weight
fp
- displacement
 p  t 2 I ' t   I '  p  2 
wt   exp


2
2
2

2



d
s
Spatial smoothing
Intensity smoothing
I’ - intensity in log space defined as:
I '  log I   log e   log R
 s  2.6  r  14.0
5
Intelligent Systems Lab.
Pixel dissimilarity
 w t I  
 w t log R 
  log R 
 w t  
 w t  
~
tW
I l  I l 
j
tW p
p
tW p


tW p


wt log e  log R 
wt  log R 


~




tW  p 
tW  p 
I r   log e  log R  


log
R



wt 
wt  






tW  p 
tW  p 




6
Intelligent Systems Lab.
Disparity smoothness
Es  f , N   
   p, qV
p qN p

V p ,q  f p , f q   min f p  f q ,Vmax
2
p,q

 p  q 2 I L  p   I L q  2 I a  p   I a q  2 I b  p   I b q  2 
  p, q   exp




2
2
2
2
2 s
2 r
2 r
2 r


 s  2.4  r  16.0
Initial disparity and camera response
1. Minimize
f *  arg min Ed  f   ES  f , N 
f F
using graph cut algorithm
2. Compute polynomial coefficients for camera response function
7
Intelligent Systems Lab.
Error correction
Minimize energy function one more time with different dissimilarity function
f *  arg min Ed  f   ES  f , N 
f F
Ed  f    Dp  f p 
p
Convert images to radiance space
~
R
(results should be same for both images)
For valid pixels
initial

0, if f p  f p
Dp  f p   

 K , othervise
For erroneous pixels

~
~
~ ~
D p  f p   Rl  p   Rr  p  f p   C p f p ,W  p , Rl , Rr

Hamming distance between pixels p and
p+fp after applying Census transform
8
Intelligent Systems Lab.
Input LDR images
9
Intelligent Systems Lab.
Disparity maps
Reference disparity map
Initial disparity estimation
10
Final map
Intelligent Systems Lab.
HDR images
11
Intelligent Systems Lab.
Experimental results
Image name
Exposure Ratio
RMSE Error
Error pixels (%)
Statue
4
16
0.9943
0.9976
8.23
8.82
Dolls
4
16
0.8454
0.8591
4.77
5.58
Clothes
4
16
1.5459
1.1556
7.43
8.15
Baby
4
16
1.432
1.4642
9.42
10.13
12
Intelligent Systems Lab.
Conclusions
Disparity map computation algorithm is proposed
Proposed method is able to compute disparity between differently exposed images
Can deal with saturated regions in the image
Can be used for capturing motion scenes with different exposures
Disadvantages
- High computational costs
- Generated images are slightly blurred
- No rotation is considered
13
Intelligent Systems Lab.
Ideal image formation system
From optics
Radiometric response
Aperture
ER
Image radiance
 d 
4
  cos 
4h
Angle from ray to optical axis
Scene radiance
Shutter speed
L  Et
2
or
L  Rke
Focal length
Where
1
k  2 cos 4 
h
Camera exposure
I  f L  e
Image brightness
Sensor response
L f
L
Irradiance
Camera response function
e
1
 d2
4
t
N
B  gB   c I
cn  0
n
n
Reverse camera response function
I
Response = Gray-level
14
Intelligent Systems Lab.
Response function examples
L
I
Response functions of a few popular cameras provided by their
manufacturers
15
Intelligent Systems Lab.
Graph-cut algorithm
1. Start with an arbitrary labeling f
2. Set success := 0
3. For each label 2 L
3.1. Find f* = arg min E(f’) among f’ within one α-expansion of f
3.2. If E(f*) < E(f), set f := f* and success := 1
4. If success = 1 goto 2
5. Return f
16
Intelligent Systems Lab.
Census transform
If (CurrentPixelIntensity<CentrePixelIntensity) boolean bit=0
else boolean bit=1
Input image
3x3 transform
17
5x5 transform
Intelligent Systems Lab.