Introduction of Saliency Map
Presenter: Chien-Chi Chen
Advisor: Jian-Jiun Ding
1
Outline
• Introduction of saliency map
• Button-up approach
–
–
–
–
–
–
L. Itti’s approach
Frequency-tuned
Multi-scale contrast
Depth of field
Spectral Residual approach
Global contrast based
• Top-down approach
– Context-aware
• Information maximum
– Measuring visual saliency by site entropy rate
2
Outline
• Introduction of saliency map
• Button-up approach
–
–
–
–
–
–
L. Itti’s approach
Frequency-tuned
Multi-scale contrast
Depth of field
Spectral Residual approach
Global contrast based
• Top-down approach
– Context-aware
• Information maximum
– Measuring visual saliency by site entropy rate
3
Introduction of saliency map
• Low-level(contrast)
– Color
Important!
– Orientation
– Size
– Motion
– Depth
• High-level
– People
– Context
Lowlevel
With
face
detectio
n
Judd et
al, 2009
4
Outline
• Introduction of saliency map
• Button-up approach
–
–
–
–
–
–
L. Itti’s approach
Frequency-tuned
Multi-scale contrast
Depth of field
Spectral Residual approach
Global contrast based
• Top-down approach
– Context-aware
• Information maximum
– Measuring visual saliency by site entropy rate
5
L. Itti’s approach
• Architecture:
Gaussian
Pyramids
R,G,B,Y
Gabor pyramids
for q = {0º, 45º,
90º, 135º}
L. Itti’s approach
• Center-surround Difference
• Achieve center-surround difference through across-scale difference
• Operated denoted by Q: Interpolation to finer scale and point-to-point
subtraction
• One pyramid for each channel: I(s), R(s), G(s), B(s), Y(s)
where s  [0..8] is the scale
L. Itti’s approach
• Center-surround Difference
– Intensity Feature Maps
I(c, s) = | I(c) Q I(s)|
c  {2, 3, 4}
s = c + d where d  {3, 4}
So I(2, 5) = | I(2) Q I(5)|
I(2, 6) = | I(2) Q I(6)|
I(3, 6) = | I(3) Q I(6)|
…
•  6 Feature Maps
•
•
•
•
L. Itti’s approach
• Center-surround
Difference
Center-surround Difference

Orientation Feature Maps
•Color Feature Maps

• O(c, s,q )  O(c,q )  O(s,q )
Red-Green and Yellow-Blue
Same c and s as with intensity
+B-Y
+R-G
+G-R
+G-R
+R-G
+B-Y
+Y-B
+Y-B
+B-Y
RG(c, s) = | (R(c) - G(c)) Q (G(s) - R(s)) |
BY(c, s) = | (B(c) - Y(c)) Q (Y(s) - B(s)) |
L. Itti’s approach
•
•
•
Normalization Operator
Promotes maps with few strong peaks
Surpresses maps with many comparable
peaks
1.
2.
3.
4.
Normalization of map to range [0…M]
Compute average m of all local maxima
Find the global maximum M
Multiply the map by (M – m)2
L. Itti’s approach
Example of
Operation:
Inhibition of return
Frequency-tuned
Image Average
 L 
 
I    a 
 
b 
S ( x, y)  I   Ihc ( x, y )
Gaussian blur
 L 
 hc 
Ihc ( x, y )   ahc 


bhc 
12
Multi-scale contrast
• Saliency algorithm
Multi-scale
contrast
Image
Centersurround
histogram
Conditional
Random
Field
Saliency
map
Color
spatialdistribution
13
Multi-scale contrast
Multi-scale contrast
• Local summation of
laplacian pyramid
L
fc ( x, I )    || I l ( x)  I l ( x) ||2
l 1 xN ( x)
Center-surround histogram
• Distance between histograms
of RGB color:
1 ( Ri  Rsi )2
 ( R, Rs )   i
2 ( R  Rsi )
2
R* ( x)  arg max  2 ( R( x), Rs ( x))
R( x)
fh ( x, I ) 
xx  2 ( R* ( x), Rs* ( x))

{x| xR* ( x)}
14
Multi-scale contrast
• Color spatial-distribution
The variance of
Coordinate of
pixel x and y
Image(RGB)
GMM
Distance from
pixel x to
image center
f s ( x, I )   p(c | I x )  (1  V (c))  (1  D(c))
c
15
Multi-scale contrast
• Energy term:
K
E ( A | I )    k Fk (ax , I )   S (ax , ax , I )
x k 1
x, x 
• Saliency object:
 f k ( x, I ), ax  0
Fk (ax , I )  
1  f k ( x, I ), ax  1
• Pairwise feature:
S (ax , ax , I ) | ax  ax |  exp( dx, x )
d x, x || I x  I x ||, L2norm
  (2 || I x  I x ||2 ) 1
16
Multi-scale contrast
• CRF:
1
P( A | I )  exp( E ( A | I ))
Z
 *  arg max  log P ( An | I n ;  )

n
• The derivative of the log-likelihood with
respect to k
17
Depth of field
• As the spread of single lens reflex camera, more
and more low depth of field(DOF) images are
captured.
• However, current saliency detection methods
work poorly for the low DOF images.
18
Depth of field
• Algorithm:
19
Depth of field
• Classification:
• Focal Point: In a low DOF image
Rectangle with the highest
DOG
edge density, and center is
initial focal point
d

S (i, j )  S (i, j ) Ae 2s
• Composition Analysis:
segmentation
Sr  Sr e
Region

Ar
n
d


Ais1 ms 2 s 3
20
Spectral Residual Approach
• First scaling image to 64x64.
• Then we smoothed the saliency map with a
gaussian filter g(x) (s = 8).
21
Global contrast-based
• Histogram based contrast(Lab):
O( N 2 )
O( N )  O( n 2 )
Quantization
of Lab
Each channel to have
12 different value
85
123  1728
22
Global contrast-based
• Region based contrast
– Segment the Image
– [Efficient graph-based image segmentation]
23
Outline
• Introduction of saliency map
• Button-up approach
–
–
–
–
–
–
L. Itti’s approach
Frequency-tuned
Center-surround
Depth of field
Spectral Residual approach
Global contrast based
• Top-down approach
– Context-aware
• Information maximum
– Measuring visual saliency by site entropy rate
24
Context-Aware
• Goal: Convey the image content
Liu et al, 2007
25
Context-Aware
• Distance between a pair of patches:
d ( pi , p j ) 
dcolor ( pi , p j )
1  c  d position ( pi , p j )
High
j
salient
Context-Aware
• Distance between a pair of patches:
K
1

r
r
r 
Si  1  exp   d ( pi , q j ) 
 K k 1

r
qk  K most similar patches at scale r
High for K most similar
Saliency
Context-Aware
• Salient at:
– Multiple scales  foreground
– Few scales  background
Scale 1
1
Si 
M
Scale 4
rM
S
r  r1
r
i
Context-Aware
• Foci =
Si  0.8
• Include distance map 1  d (i)
foci
Sˆi  Si 1  d foci (i ) 
Si
X
Outline
• Introduction of saliency map
• Button-up approach
–
–
–
–
–
–
L. Itti’s approach
Frequency-tuned
Center-surround
Depth of field
Spectral Residual approach
Global contrast based
• Top-down approach
– Context-aware
• Information maximum
– Measuring visual saliency by site entropy rate
30
Measuring visual saliency by site
entropy rate
1
31
Measuring visual saliency by site
entropy rate
2
A fully-connected graph representation is adopted for each
32
Sub-band graph representation
33
Sub-band graph representation
34
Measuring visual saliency by site
entropy rate
3
A random walk is adopted on each sub-band graph. And Site entropy
rate(SER) is measured the average information from a node to the other
35
The site entropy rate
•
i 
Wi
,Wi   ij ,W   ij
2W
j
i , j : j i
Pij 
ij
 ij
j
•
36
Conclusion
• Image processing is funny
• Unusual in its neighborhood will correspond
to high saliency weight
• Contrast is the key of saliency
37
Reference
[1] R. Achanta, F. Estrada, P. Wils, and S. S¨usstrunk. Salient region detection and
segmentation. In ICVS, pages 66–75. Springer, 2008. 410, 412, 414
[2] R. Achanta, S. Hemami, F. Estrada, and S. S¨usstrunk. Frequency-tuned salient
region detection. In CVPR, pages 1597–1604, 2009. 409, 410, 412, 413, 414, 415
[3] L. Itti, C. Koch, and E. Niebur. A model of saliency based visual attention for rapid
scene analysis. IEEE TPAMI, 20(11):1254–1259, 1998. 409, 410, 412, 414
[4] X. Hou and L. Zhang. Saliency detection: A spectral residual approach. In CVPR,
pages 1–8, 2007. 410, 412, 413, 414
[5] S. Goferman, L. Zelnik-Manor, and A. Tal. Context-aware saliency detection. In CVPR,
2010. 410, 412, 413, 414, 415
[6] MM Cheng, GX Zhang, N. J. Mitra, X. Huang, S.M. Hu. Global Contrast based Salient
Region Detect. In CVPR, 2011 .
[7] T. Liu, Z. Yuan, J. Sun, J.Wang, N. Zheng, T. X., and S. H.Y. Learning to detect a salient
object. IEEE TPAMI, 33(2):353–367, 2011. 410
[8] W. Wang, Y. Wang, Q. Huang, W. Gao, Measuring Visaul Saliency by Site Entropy
Rate, In CVPR, 2010.
38