Introduction to Motion
Estimation for Video Codec
Advisor:Jian-Jiun Ding
Speaker:Jian-Hwu Wang
Date:05/06/2011
DISP Lab, GICE, NTU
1
Outline
• Introduction to Motion Estimation
– Motion Vector
– Cost Function
• Video Codec
• The Methods for Motion Estimation
• Conclusion
DISP Lab, GICE, NTU
2 /33
Introduction to Motion Estimation
• Using motion estimation to find the motion
vectors
Motion vector
(u,v)
Motion
Vector
Current
block
(x+u,y+v)
(x,y)
Search
range
Best
matching
location
– Motion Vector
• A 2D point that represents the consistent between the
current block and space of previous frame.
DISP Lab, GICE, NTU
3 /33
Introduction to Motion Estimation
• Motion Estimation
– Cost function
• Sum of Absolute Difference(SAD)
• Sum of Squared Difference(SSD)
DISP Lab, GICE, NTU
4 /33
Introduction to Motion Estimation
• Motion Estimation
– Cost function
• Mean Absolute Error(MAE)
• Mean Square Error(MSE)
DISP Lab, GICE, NTU
5 /33
Outline
• Introduction to Motion Estimation
• Video Codec
– Video encoder
– Video decoder
• The Methods for Motion Estimation
• Conclusion
DISP Lab, GICE, NTU
6 /33
Video Codec
• Video Encoder[1]
– Intra-prediction
DISP Lab, GICE, NTU
7 /33
Video Codec
• Video Encoder[1]
– Inter-prediction
DISP Lab, GICE, NTU
8 /33
Video Codec
• Video Decoder[1]
DISP Lab, GICE, NTU
9 /33
Outline
• Introduction to Motion Estimation
• Video Codec
• The Methods for Motion Estimation
– Full motion search
– Fast motion search
– True motion search
• Conclusion
DISP Lab, GICE, NTU
10 /33
The Methods for Motion Estimation
• Full motion search
(A) Top-to-bottom scan
DISP Lab, GICE, NTU
(B) Spiral search[2]
11 /33
The Methods for Motion Estimation
• Fast motion search
(A) Three step search[3]
(B) Diamond search[4]
DISP Lab, GICE, NTU
12 /33
The Methods for Motion Estimation
• Motion Analysis
– Traditional motion estimation
• Its goal is found for high compression rate.
• There are wrong motion vectors while blocks in several
situations.
– True motion estimation(TME)
• The goal of TME is described the meaningful
information of moving object in video sequence.
DISP Lab, GICE, NTU
13 /33
The Methods for Motion Estimation
• The feature of motion vector field(MVF) of TME
• The consistency in spatial domain
• The dependent in time domain
DISP Lab, GICE, NTU
14 /33
The Methods for Motion Estimation
• True Motion Estimation
• Overlapped block-based motion estimation[5]
–
–
–
Take a block that block size bigger than normal block size
8x8 -> 16x16
Perform motion estimation after sampling 16x16 block to 8x8
–
Post smoothness – Motion vector median filter
DISP Lab, GICE, NTU
15 /33
The Methods for Motion Estimation
• True Motion Estimation
• Overlapped block bi-directional motion estimation[6][7]
–
8x8 -> 12x12
SAD(mvx , mvy ) 
–
9
9
 f
j  2i  2
n
( xc  mvx  i, yc  mvy  j )  f n1 ( xc  mvx  i, yc  mvy  j )
Post smoothness – Motion vector median filter
DISP Lab, GICE, NTU
16 /33
The Methods for Motion Estimation
• True Motion Estimation
• 3-D recursive search[8]
• 1-D recursive search
– Small search region
» Candidate motion vector
– Update motion vector by using recursive research
» Update motion vector
–
Stop condition is necessary
DISP Lab, GICE, NTU
17 /33
The Methods for Motion Estimation
• True Motion Estimation
• 3-D recursive search


X 
CS a ( X , t )  C  CS max C  Da  X    , t   US a X , t
Y  



 

Sa
 
X
 
  Da  X  2    , t  T , 0 
Y 
 
 


X  
CSb ( X , t )  C  CS max C  Db  X  
 , t   USb X , t
 Y  


 
 
X 
  Db  X  2  
 , t  T , 0 
 Y 
 
 
 
Sb
C



Tb
Ta
 0   0   0   0   0   1    1  3    3 
USn   ,  ,  ,  ,  ,  ,  ,  ,  
 0   1    1  2    2   0   0   0   0 
DISP Lab, GICE, NTU
18 /33
The Methods for Motion Estimation
•
True motion estimation based on reliable motion decision unit
•
•
λa = 0.25
12 ≦λb ≦ 20
DISP Lab, GICE, NTU
19 /33
The Methods for Motion Estimation
• True motion estimation based on reliable motion decision unit
– The types of motion vector
• Unrelated MV
• Matched MV
DISP Lab, GICE, NTU
20 /33
The Methods for Motion Estimation
• True motion estimation based on reliable motion decision unit
– The types of motion vector
• Unmatched MV
Frame 39
Frame 40
• Uncertain MV
Repeative Pattern
The MVF of Uncertain MV
DISP Lab, GICE, NTU
21 /33
The Methods for Motion Estimation
• True motion estimation based on reliable motion decision unit
– Introduction to reliable motion decision unit
• The drawbacks of conventional motion estimation
– Can’t classify the reliabilities of MVs
– It would make the blocks of consecutive frames produce wrong MVs.
• Reliable/Unreliable Motion Vector
Reliable MV
Unreliable MV
Matched MV
DISP Lab, GICE, NTU
Unrelated MV
Unmatched MV
Uncertain MV
22 /33
The Methods for Motion Estimation
• True motion estimation based on reliable motion decision unit
– Introduction to reliable motion decision unit
• Reliable motion decision unit based on Early-stop search.
– Early-stop search and Early-stop point(ESP)
(a)ESP distribution
of reliable MV
(b)ESP distribution
of unreliable MV
DISP Lab, GICE, NTU
23 /33
The Methods for Motion Estimation
• True motion estimation based on reliable motion decision unit
– ESP classification
(c)The CDF of ESP
of reliable MV
(d)The CDF of ESP
of unreliable MV
DISP Lab, GICE, NTU
24 /33
The Methods for Motion Estimation
 True motion estimation based on reliable motion decision unit
 Smoothness constraint (SC)
5
SC (m, n)   (m  mvxi )2  (n  mv yi ) 2 
1
2
4
C
3
i 1
Cost (m, n)  SAD(m, n)    SC (m, n)
5
(u, v)  arg min Cost (m, n)
 M  m, n M
• The comparison of smoothness constraint and post-smoothness(PS)
The MVF of SC
The MVF of PS
DISP Lab, GICE, NTU
25 /33
The Methods for Motion Estimation
• The comparison of method
– Downsample frame rate 30Hz to 15Hz or 10Hz
– Make Bi-directional Motion Compensation
Interpolation (BMCI) frames by its before and
after frames
– Compare BMCI frames with extracted frame
DISP Lab, GICE, NTU
26 /33
The Methods for Motion Estimation
• Experiment result :
– Compare PSNR between BMCI frame and extracted frame.
– 30Hz -> 15Hz
PSNR(dB)
SS
Akiyo
Bream
Bus
Carphone
Football
Foreman
Hall
monitor
Mobile
Mother
daughter
News
Silent
Stefan
Average
38.88 26.21 22.46 28.48 21.78 28.52 31.9 22.11 32.84 31.93 32.89 24.44 28.54
3DRS[9]
44.29 30.77 23.4 30.77 21.71 29.53 33.98 25.8 36.55 34.17 34.17 23.35 30.71
Zhai[7]
41.13 30.05 24.48 29.24 22.26 30.1 34.44 24.75 34.17 33.82 34.05 25.78 30.36
SC(λ = 12)
43.22 29.01 23.21 30.87 22.19 30.73 35.57 25.02 37.64 34.1 34.47 26.01 31.01
SC(λ = 16)
43.24 29.12 23.09 30.87 22.17 30.76 35.58 25.03 37.68 34.15 34.42 26.01 31.02
SC(λ = 20)
43.26 29.18
23.08
30.84 22.17 30.72 35.59 25.05 37.66 34.15 34.42 25.91 31.01
7
DISP Lab, GICE, NTU
27 /33
The Methods for Motion Estimation
• Experiment result :
– Compare PSNR between BMCI frame and extracted frame.
– 30Hz -> 10Hz
PSNR(dB)
Akiyo
Bream
Bus
Carphone
Football
Foreman
Hall
monitor
Mobile
Mother
daughter
News
Silent
Stefan
Average
SS
35.45 23.41 17.41 26.64 19.84 24.51 29.87 19.15 31.3 29.49 29.61 18.34 25.42
3DRS[9]
37.58 24.05 17.45 27.7 19.91 24.86 30.92 19.99 33.26 30.3 29.78 18.56 26.2
Zhai[7]
36.33 24.11 17.45 26.86 19.81 24.9 30.65 19.78 31.83 30.04 29.86 18.4 25.84
SC(λ = 12)
37.01 23.99 17.53 27.85 19.91 25.1 31.21 19.9 33.58 30.21 30.12 18.52 26.24
SC(λ = 16)
37.01 23.99 17.54 27.86 19.91 25.09 31.26 19.92 33.59 30.18 30.11 18.54 26.25
SC(λ = 20)
37.02 23.99 17.54 27.87 19.92 25.1 31.24 19.93 33.59 30.2 30.09 18.55 26.25
DISP Lab, GICE, NTU
28 /33
The Methods for Motion Estimation
• Elapsed time:
– 8x8 block, there are 64 subtraction operators and 63 addition operators.
– 12x12 block, there are 144 subtraction operators and 143 addition
operators.
Method
SS
3DRS[10]
Zhai[8]
Maximum operational complexity of motion estimation in each frame
(64 +63)x(33x33)x((352x288)/64)
= 219,071,952
(64 +63)x5x30x((352x288)/64)
= 30,175,200
(144+143)x(33x33)x((352x288)/64)
+ ((352x288)/64)x(9x8x(2+1)) = 495,410,256
Proposed
2x(64+63)x(33x33)x((352x288)/64) = 438,143,904
DISP Lab, GICE, NTU
29 /33
The Methods for Motion Estimation
• Elapsed time:
(ms)/Frame
Hall
Mobile
Mother
daughter
688
768
694
359
469
814
715
448
390
437
310
416
417
395
372
2510 1789 1427 2396 1285 1338 1257 1395
622
826 1496 1393
446
771
1173
738
1289
766
762
897
626
520
583
957
794
442
747
1166
739
1283
753
744
889
621
513
582
947
786
443
746
1161
738
1284
754
743
889
620
511
581
950
785
Akiyo
Bream
Bus
Carphone
Football
Foreman
241
1159
916
725
1076
671
243
449
384
432
418
401
monitor
News Silent Stefan
Average
SS
3DRS[9]
Zhai[7]
SC(λ = 12)
SC(λ = 16)
SC(λ = 20)
DISP Lab, GICE, NTU
30 /33
Conclusion
• The motion estimation(ME) of video processing
was a popular research in recently decade years.
• It must be taken other side-information by ME to
apply another applications, such as : object
detection, tracking, video stabilization and frame
rate up conversion…
DISP Lab, GICE, NTU
31 /33
Reference
• [1] Iain E. G. Richardson, “H.264 and MPEG-4 Video Compression: Video
Coding for Next-Generation Multimedia” John Wiley & Sons Inc, 2003
• [2] C. Chok-Kwan, P. Lai-Man, “Normalized Partial Distortion Search
Algorithm for Block Motion Estimation,” IEEE Transactions on Circuits
and Systems for Video Technology, Vol. 10, No. 3, Apr 2000, pp.417-422.
• [3] T. Koga, K. Iinuma, A. Hirano, Y. Iijima, and T. Ishiguro, “Motion
compensated interframe coding for video conferencing,” in Proc. NTC81,
pp. C9.6.1-9.6.5, New Orleans, LA, Nov./Dec. 1981.
• [4] J. Y. Tham, S. Ranganath, M. Ranganath, and A. A. Kassim, "A novel
unrestricted center-biased diamond search algorithm for block motion
estimation", IEEE Trans. Circuits Syst. Video Technol., vol. 8, pp.369 377 , 1998.
• [5] Taehyeun Ha, Seongjoo Lee and Jaeseok Kim, “Motion Compensated
Frame Interpolation by new Block-based Motion Estimation Algorithm,”
IEEE Transactions on Consumer Electronics, Volume 50, Issue 2, pp.752759, May 2004.
DISP Lab, GICE, NTU
32 /33
Reference
• [6]J. Zhai, K. Yu, J. Li, and S. Li, “A low complexity motion compensated
frame interpolation method,” in Proc.IEEE ISCAS, May 2005, pp. 23–26.
• [7]Ya-Ting Yang, Yi-Shin Tung, and Ja-LingWu, “Quality enhancement of
frame rate up-converted video by adaptive frame skip and reliable motion
extraction,” IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 12,
pp.1700–1713, Dec. 2007.
• [8]G. de Haan, P.W.A.C. Biezen, H. Huijgen, and O.A. Ojo, “True-Motion
Estimation with 3-D Recursive Search Block Matching”, IEEE
Transactions on Circuits and Systems for Video Technology, VOL.3, NO.5,
OCTOBER 1993.
DISP Lab, GICE, NTU
33 /33