Shift-Map Image Editing
Shift-Map Image Editing
Yael Pritch, Eitam Kav-Venaki, Shmuel Peleg
Computer Science and Engineering
The Hebrew University of Jerusalem, Israel
ICCV 2009
Outline
Introduction
Image Editing as Graph Labeling
Hierarchical Solution for Graph Labeling
Shift-Map Application
Concluding Remarks
Outline
Introduction
Image Editing as Graph Labeling
Hierarchical Solution for Graph Labeling
Shift-Map Application
Concluding Remarks
Introduction
Geometric image rearrangement is becoming more popular
◦ Image resizing (a.k.a. retargeting)
◦ Object rearrangement and removal
Early methods manipulation mostly crop and scale
◦ For image resizing, examining image content and removing “ less important ” regions
Introduction
Seam carving [2, 13]
Continuous image warping [19, 16]
Shift-map editing
◦ Avoids scaling and mostly remove or shift image regions
(a)
Original image
(b)
Video-retargeting [19]
(c)
Optimized scale-and-stretch [16]
(d)
Improved Seam Carving[13]
(e) Our shift-map editing
(a) Original image
(b)
Our shift-map editing
(c) Video-retargeting [19]
(d)
Optimized scale-and-stretch [16]
(e)
Improved Seam Carving[13]
Outline
Introduction
Image Editing as Graph Labeling
Hierarchical Solution for Graph Labeling
Shift-Map Application
Concluding Remarks
Image Editing as Graph Labeling
Shift-map
◦ The relative shift of every pixel in the output image from its source in an input image
◦ Represents the selected label for each output pixel
Two terms are used in computing the optimal shift-map
◦ Data term
◦ Smoothness term
Image Editing as Graph Labeling
Input image I(x, y)
Output image R(u, v)
The relationship between input image and output image is defined by
◦ t t
Each output pixel can be labeled by a shift
Image Editing as Graph Labeling
The optimal shift-map M minimizes the cost function :
◦
E d
: data term
◦
E s
: smoothness term
◦
◦ N : neighboring pixels
= 1
Single pixel data term
Pixel rearrangement
Pixel saliency and removal
◦ S : saliency map, very high for pixels to be removed, very low for pixels not to be removed
Smoothness term for pixels pair
represents discontinuities added to the output image by discontinuities in the shift-map in the output image R if
M
u
1
, v
1
M
( u
2
,
2 v
2
)
The smoothness term account color difference and gradient difference
Smoothness term for pixels pair
◦ e i
: four unit vectors - four spatial neighbors
◦ Color differences are Euclidean distances in
RGB
◦
I
R
◦ gradients at these locstion
= 2
Outline
Introduction
Image Editing as Graph Labeling
Hierarchical Solution for Graph Labeling
Shift-Map Application
Concluding Remarks
Hierarchical Solution for Graph
Labeling
Finding the optimal graph labeling , the number of possible labels is the number of pixels in the input image
Use heuristic hierarchical approach reduces the memory and computational
◦ First solved in a coarse resolution
◦ Higher resolution level
Hierarchical Solution for Graph
Labeling
Example : 4th pyramid level
◦ The number of pixels and number of labels are reduce by a factor of 64
Coarse shift-map
Coarse level
4x4
16x16
32x32
Input image
64x64
Nearest neighbor interpolation
Coarse level
Output image
Hierarchical Solution for Graph
Labeling
Use three to five pyramid levels
The coarsest level contains up to
100 x 100 pixels
Outline
Introduction
Image Editing as Graph Labeling
Hierarchical Solution for Graph Labeling
Shift-Map Application
Concluding Remarks
Shift-Map Application
Image retargeting
Image rearrangement
Inpainting
Image composition
Image retargeting
Label order constraint
◦ The shift-map will retain the spatial order
◦ In the case of reducing width
M ( u , v )
( t x
, t y
) ( u 1 , v ) ( t t
' x
t t x
0
◦ In the case of increasing width
M ( u , v )
( t x
, t y
) ( u 1 , v ) ( t t
' x
t
0
Image retargeting
Controlling object removal
◦ It is possible to control the size and number of removed objects by performing several steps of resizing
◦ Also possible to control object removal by marking objects as salient
◦ The number of steps becomes the number of removed columns
(a) Original image
(b) Resizing in single step
(c) Six smaller resizing steps
(d) Ten smaller resizing steps
Shift-map retargeting :
(a) Original image
(b)(c)(e) No saliency
(d) Child was marked salient
Original image [13] [19] [16] shift-map
Image rearrangement
Moving an object to a new image location
Deleting part of the image
Specified in two parts using the data term
◦ Force pixels to appear in a new location using
Eq. 2
◦ Marks these pixels for removal from their original location using Eq. 3
Example – 1 :
◦ move the person and a part of the temple to the right, and keep the tourists at their original location
Example – 2 :
◦ Kid on the left should move to the center, baby should move to the left, kid on the right should remain in place
Inpainting
Unwanted pixels are given an infinitely high data term as described in Eq. 3
Maps pixels inside the hole to other locations in the input image
Inpainting
A good complition with no user intervention
Image composition
In the shift-map framework the input can consist of either a single image, or of a set of images
◦
M ( u , v ) image
( t x
, t y
, t ind
)
, is the index of the input ind
Tolerate misalignments between the input images
Outline
Introduction
Image Editing as Graph Labeling
Hierarchical Solution for Graph Labeling
Shift-Map Application
Concluding Remarks
Concluding Remarks
Shift-maps are proposed as a new framework to describe various geometric rearrangement problems
Images generated by the shift map are natural looking
Minimal and intuitive user interaction
Distortions that may be introduced by stitching are minimized
Large regions can be synthesized
