Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume 27 Issue 3, August 2008 Presenter: Hon-Hang Chang Date: 2011/ 09/ 30 1 Outline • • • • • • Introduction Preliminaries Seam Carving using Graph Cut Forward Energy Results Conclusion 2 • Because of human perception, the temporal may even be more disturbing than spatial in video, as the human eye is highly sensitive to movement. Introduction • 2D manifolds To treat video as 3D cube 1 D path on 2D image (Seam) 2D manifolds in a 3D volume • In this paper the author define a new formulation of seam carving using graph cuts. 3 • Introduction A whole seam A seam is a monotonic and connected path of pixels going from the top of the image to the bottom Or from left to right • The image size is reduced by one either in the horizontal or the vertical dimension. • video should support retargeting capabilities as it is displayed on TVs, computers, cellular phones. 4 Temporal term • Preliminaries Spatiotemporal L1-norm Spatial term Etemporal (i, j ) max{| I t (i, j ) |} t 1 t Espatial (i, j ) max{| I t (i, j ) | | I t (i, j ) |} t 1 x y N N Among this, {It} t=1~N Eglobal (i, j) Espatial (1 )Etemporal Where, α ∈ [0, 1] Extend the spatial L1-norm to a spatiotemporal L1-norm: • Motion artifacts are more noticeable • Taking α= 0.3 5 Seam Carving using Graph Cuts • Graph cut S (Source) arc1 T (Sink) arc2 Source Sink arc3 arc4 6 Seam Carving using Graph Cuts • Optimal seam in graph cut An optimal seam must be: 1. 1.Valid - A simple cut cannot define a seam carving 2. Monotonic - Including only one pixel in each row (or column) 3. Connected -The seam must be connected • The proposed formulation extend seam carving to video and define a monotonic and connected 2D manifold seam inside the video cube. 7 Seam Carving using Graph Cuts • Graph cut for image (Non-monotonic) • Every internal node P is connected to its four neighbors Nbr(Pi,j) = {Pi−1,j , Pi+1,j , Pi,j−1, Pi,j+1}. ∂y(i, j) = |I(i+1, j ) − I(i, j)| Vertical direction Pi-1,j Pi,j-1 Pi,j Pi,j+1 Pi+1,j Neighbors pixels ∂x(i, j) = |I(i, j + 1) − I(i, j)| Horizontal direction 8 Seam Carving using Graph Cuts • Graph cut for image (Unconnected) • Prove_1: 9 Seam Carving using Graph Cuts • Graph cut for image (Original) E1(i, j) =∂x(i, j) + ∂y(i, j) • Prove_2: 10 Seam Carving using Graph Cuts • Graph cut for image • We can use any energy function defined on the pixels as the weight of the forward horizontal arcs . • Achieve the same results as the original dynamic programming based seam carving 11 Seam Carving using Graph Cuts • Graph cut for video • The extension to video is straightforward. 12 Seam Carving using Graph Cuts • Graph cut for video • Computation time is quadratic in the number of voxels. O(mn2) - n #of nodes ; m #of arcs • Performance issues are encountered already for high resolution images • The graph cut approach to seam carving allows us to extend the benefits of content-aware resizing to video. Still, the method is not perfect and no single energy function was shown to perform properly in all cases. Therefore, they introduce a new energy function that better protects media content, and improves video results. 13 Forward Energy • Remove the seam with the least amount of energy from the image • The inserted energy is due to new edges created by previously non adjacent pixels that become neighbors once the seam is removed • To measure the real change in energy after a removal of a seam They measure the difference in the energy of the image after the removal (It=i+1) and the energy of only those parts that were not removed in the previous image 14 Forward Energy 15 • Forward Energy Forward Energy in Graph Cut +LR = | I(i, j+1)− I(i, j−1)| (arc weight between pi,j and pi,j+1) difference between the Left and Right neighbors +LU = | I(i-1, j)− I(i, j−1)| (arc weight between pi,j and pi-1,j) difference between the Left and Up neighbors -LU = | I(i+1, j)− I(i, j−1)| (arc weight between pi,j and pi+1,j) difference between the Left and Up neighbors with respect to the end 16 point of the arrow • Forward Energy Forward Energy in Graph Cut +LR = | I(i, j+1)− I(i, j−1)| (arc weight between pi,j and pi,j+1) difference between the Left and Right neighbors +LU = | I(i-1, j)− I(i, j−1)| (arc weight between pi,j and pi-1,j) difference between the Left and Up neighbors -LU = | I(i+1, j)− I(i, j−1)| (arc weight between pi,j and pi+1,j) difference between the Left and Up neighbors with respect to the end point of the arrow 17 Results 18 Results 19 Conclusion Video retargeting is achieved using graph cuts and we have shown a construction that is consistent with the dynamic programming approach. 20