Improved Seam Carving for
Video Retargeting
Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan
Source: ACM Transactions on Graphics (TOG), Volume 27 Issue 3, August 2008
Presenter: Hon-Hang Chang
Date: 2011/ 09/ 30
1
Outline
•
•
•
•
•
•
Introduction
Preliminaries
Seam Carving using Graph Cut
Forward Energy
Results
Conclusion
2
• Because of human perception, the temporal may even be more disturbing than
spatial in video, as the human eye is highly sensitive to movement.
Introduction
• 2D manifolds
To treat video as 3D
cube
1 D path on 2D image
(Seam)
2D manifolds in a 3D
volume
• In this paper the author define a new formulation of
seam carving using graph cuts.
3
•
Introduction
A whole seam
A seam is a monotonic and connected path of pixels going
from the top of the image to the bottom
Or from left to right
• The image size is reduced by one either in the horizontal or the vertical
dimension.
• video should support retargeting capabilities as it is displayed on TVs,
computers, cellular phones.
4
Temporal term
•
Preliminaries
Spatiotemporal L1-norm
Spatial term

Etemporal (i, j )  max{| I t (i, j ) |}
t 1
t


Espatial (i, j )  max{| I t (i, j ) |  | I t (i, j ) |}
t 1
x
y
N
N
Among this, {It} t=1~N
Eglobal (i, j)    Espatial  (1   )Etemporal
Where,
α ∈ [0, 1]
Extend the spatial L1-norm to a spatiotemporal L1-norm:
• Motion artifacts are more noticeable
• Taking α= 0.3
5
Seam Carving using Graph Cuts
• Graph cut
S (Source)
arc1
T (Sink)
arc2
Source
Sink
arc3
arc4
6
Seam Carving using Graph Cuts
• Optimal seam in graph cut
An optimal seam must be:
1. 1.Valid
- A simple cut cannot define a seam carving
2. Monotonic
- Including only one pixel in each row (or column)
3. Connected
-The seam must be connected
• The proposed formulation extend seam carving to video and define
a monotonic and connected 2D manifold seam inside the video
cube.
7
Seam Carving using Graph Cuts
• Graph cut for image (Non-monotonic)
•
Every internal node P is connected to its four neighbors Nbr(Pi,j) = {Pi−1,j , Pi+1,j , Pi,j−1,
Pi,j+1}.
∂y(i, j) = |I(i+1, j ) − I(i, j)|
Vertical direction
Pi-1,j
Pi,j-1
Pi,j
Pi,j+1
Pi+1,j
Neighbors pixels
∂x(i, j) = |I(i, j + 1) − I(i, j)|
Horizontal direction
8
Seam Carving using Graph Cuts
• Graph cut for image (Unconnected)
• Prove_1:
9
Seam Carving using Graph Cuts
• Graph cut for image (Original)
E1(i, j) =∂x(i, j) + ∂y(i, j)
• Prove_2:
10
Seam Carving using Graph Cuts
• Graph cut for image
• We can use any energy function defined on the pixels as the
weight of the forward horizontal arcs .
• Achieve the same results as the original dynamic programming
based seam carving
11
Seam Carving using Graph Cuts
• Graph cut for video
• The extension to video is straightforward.
12
Seam Carving using Graph Cuts
• Graph cut for video
• Computation time is quadratic in the number of voxels. O(mn2)
- n #of nodes ; m #of arcs
• Performance issues are encountered already for high
resolution images
• The graph cut approach to seam carving allows us to extend the
benefits of content-aware resizing to video. Still, the method is not
perfect and no single energy function was shown to perform properly
in all cases. Therefore, they introduce a new energy function that
better protects media content, and improves video results.
13
Forward Energy
• Remove the seam with the least amount of energy from the
image
• The inserted energy is due to new edges created by previously
non adjacent pixels that become neighbors once the seam is
removed
• To measure the real change in energy after a
removal of a seam
They measure the difference in the energy of the image after the
removal (It=i+1) and the energy of only those parts that were not
removed in the previous image
14
Forward Energy
15
•
Forward Energy
Forward Energy in Graph Cut
+LR = | I(i, j+1)− I(i, j−1)| (arc weight between pi,j and pi,j+1)
difference between the Left and Right neighbors
+LU = | I(i-1, j)− I(i, j−1)| (arc weight between pi,j and pi-1,j)
difference between the Left and Up neighbors
-LU = | I(i+1, j)− I(i, j−1)| (arc weight between pi,j and pi+1,j)
difference between the Left and Up neighbors with respect to the end
16
point of the arrow
•
Forward Energy
Forward Energy in Graph Cut
+LR = | I(i, j+1)− I(i, j−1)| (arc weight between pi,j and pi,j+1)
difference between the Left and Right neighbors
+LU = | I(i-1, j)− I(i, j−1)| (arc weight between pi,j and pi-1,j)
difference between the Left and Up neighbors
-LU = | I(i+1, j)− I(i, j−1)| (arc weight between pi,j and pi+1,j)
difference between the Left and Up neighbors with respect to the end
point of the arrow
17
Results
18
Results
19
Conclusion
Video retargeting is achieved using graph cuts and we have
shown a construction that is consistent with the dynamic
programming approach.
20