Digital Video Compression
Fundamentals and Standards
Web Technology
Outline
Introduction
Video Compression Standards
Simulation Reference Software
Future Work and Conclusions
2008/12/26
Digital Video Compression Fundamentals and Standards
2
Outline
Introduction
Video Compression Standards
Simulation Reference Software
Future Work and Conclusions
2008/12/26
Digital Video Compression Fundamentals and Standards
3
Introduction (1/2)
Why video compression technique is
important ?
One movie video without compression
720 x 480 pixels per frame
30 frames per second
Total 90 minutes
Full color
The total quantity of data = 167.96 G Bytes !!
2008/12/26
Digital Video Compression Fundamentals and Standards
4
Introduction (2/2)
What is the difference between video
compression and image compression?
Temporal Redundancy
Coding method to remove redundancy
Intraframe Coding
Remove spatial redundancy
Interframe Coding
Remove temporal redundancy
2008/12/26
Digital Video Compression Fundamentals and Standards
5
The most intuitive method to remove
Spatiotemporal redundancy
3-Dimensional DCT
Remove spatiotemporal correlation
Good for low motion video
Bad for high motion video
N 1 N 1 N 1
8
  (2 x  1)u 
  (2 y  1)v 
  (2t  1)w 
F ( x, y, t )  3 C (u)C (v)C (w) ( x, y, t ) cos 
cos
cos

 2 N

 2 N

N
 2N
t 0 x 0 y 0
for u  0,..., N  1 ,v  0,..., N  1 and w  0,..., N  1
1/ 2 for k  0
where N  8 and C (k )  
 1 otherwise
2008/12/26
Digital Video Compression Fundamentals and Standards
6
The most popular method to remove
temporal redundancy
The Block-Matching Algorithm
2008/12/26
Digital Video Compression Fundamentals and Standards
7
Matching Function
The dissimilarity D ( s, t ) between two blocks  n and  n1
p
q
D(s, t )    M [n( x, y), n  1( x  Vx, y  Vy)]
Vy 1 Vx , 1
The matching criteria
Mean square error (MSE)
M (u, v)  (u - v)2
High precision is needed
Mean absolute difference (MAD)
M (u, v) | u - v |
Low precision is enough
2008/12/26
Digital Video Compression Fundamentals and Standards
8
The Exhaustive Block-Matching
Algorithm
Reference Frame
Current Frame
Motion
Vector
|A|=12
12 22 33 44 55 66 77 88
11 22 33 44 55 66 77 88
1 0
12 22 33 44 55 66 77 88
11 22 33 44 55 66 77 88
1 0
12 22 33 44 55 66 77 88
11 22 33 44 55 66 77 88
1 0
12 22 33 44 55 66 77 88
11 22 33 44 55 66 77 88
1 0
11 23 34 44 55 66 77 88
11 22 33 44 55 66 77 88
0 1
11 23 34 44 55 66 77 88
11 22 33 44 55 66 77 88
0 1
Search
11 23 34 44 55 66 77 88
11 22 33 44 55 66 77 88
0 1
Range 11 22 33 44 55 66 77 88
11 23 34 44 55 66 77 88
0 1
Digital Video Compression Fundamentals and Standards
2008/12/26
0
0
0
0
1
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
9
Fast Block-Matching Algorithms
EBMA needs Intensive computation
 720 480 
 16  16   128 128  16 16  30  170 G instructions/sec
Fast Algorithm is needed
Find the possible local optimal
Reference Frame
2008/12/26
Digital Video Compression Fundamentals and Standards
Current Frame
10
Fast Block-Matching Algorithms
The characteristics of fast algorithm
Not accurate as EBMA
Save large computation
Two famous fast algorithm
2-D logarithm Search Method
Three Steps Search Method
2008/12/26
Digital Video Compression Fundamentals and Standards
11
2-D logarithm Search Method
1
1
2
1
1
1
2
2
4
4
4
3
4
4
3
2008/12/26
Digital Video Compression Fundamentals and Standards
12
Three Step Search Method
1
2008/12/26
1
2
2
2
1 3 2 3
2
3 3 3
2
2
1
1
1
1
1
1
2
3 3 3
Digital Video Compression Fundamentals and Standards
13
Multiresolution Motion Estimation (1/3)
The number of levels is L
l-th level images of the target frames
t ,l (x), x   l , t  1, 2,...
where  l is set of pixels at level L
At the l-th level, the MV is d( x)
At the l-th level, the estimated MV is dl (x)  U (dl 1 (x))
Determine update ql (x) such that error is minimized
error 
 | 2,l (x  dl (x)  ql (x))  1,l (x)) |
p
xl
The new motion vector is dl (x)  dl (x)  ql (x)
2008/12/26
Digital Video Compression Fundamentals and Standards
14
Multiresolution Motion Estimation (2/3)
Variable block size method
2008/12/26
Digital Video Compression Fundamentals and Standards
15
Multiresolution Motion Estimation (3/3)
2008/12/26
Digital Video Compression Fundamentals and Standards
16
Outline
Introduction
Video Compression Standards
Simulation Reference Software
Future Work and Conclusions
2008/12/26
Digital Video Compression Fundamentals and Standards
17
The Development of Video
Compression Standards
……
……
5-point 1D DCT
6-point 1D DCT
5-point 1D DCT
6-point 1D DCT
5-point 1D DCT
6-point 1D DCT
……
……
……
……
2008/12/26
Digital Video Compression Fundamentals and Standards
18
The MPEG-1 Standard
Group of Pictures
Motion Estimation
Motion Compensation
Differential Coding
DCT
Quantization
Entropy Coding
2008/12/26
Digital Video Compression Fundamentals and Standards
19
Group of Pictures (1/2)
I-frame (Intracoded Frame)
Coded in one frame such as DCT.
This type of frame do not need previous frame
P-frame (Predictive Frame)
One directional motion prediction from a previous frame
The reference can be either I-frame or P-frame
Generally referred to as inter-frame
B-frame (Bi-directional predictive frame)
Bi-directional motion prediction from a previous or future frame
The reference can be either I-frame or P-frame
Generally referred to as inter-frame
2008/12/26
Digital Video Compression Fundamentals and Standards
20
Group of Pictures (2/2)
The distance between two nearest P-frame or P-frame and I-frame
denoted by M
The distance between two nearest I-frames
denoted by N
Forward Motion Compensation
GOP
N=9
M=3
I
B
B
P
B
B
P
B
B
I
Bidirectional Motion Compensation
2008/12/26
Digital Video Compression Fundamentals and Standards
21
The MPEG-1 Encoder (1/4)
Intra-frame
Residue
DCT
Q
Entropy
Coding
Q-1
Motion
Compensation
Inter-frame
Motion Vector
IDCT
Motion
Estimation
Frame
Memory
2008/12/26
Digital Video Compression Fundamentals and Standards
22
The MPEG-1 Encoder (2/4)
Differential Coding
^
D(t )  (t )  (t )
 (t ) is the input image
^
 (t ) is the predictive image
DCT
N 1 N 1
2
  (2 x  1)u 
  (2 y  1)v 
F (u , v)  C (u )C (v) f ( x, y ) cos 
cos



N
2N
2N

x 0 y 0
for u  0,..., N  1 and v  0,..., N  1
1/ 2 for k  0
where N  8 and C (k )  
 1 otherwise
2008/12/26
Digital Video Compression Fundamentals and Standards
23
The MPEG-1 Encoder (3/4)
Quantization
Intra quantization matrix
Inter quantization matrix
Qintra
8

 16
 19

22


 22

 26
 26

 27
2008/12/26
16 19 22 26 27 29 34 
 16 17 18 19


16 22 24 27 29 34 37 
 17 18 19 20
 18 19 20 21
22 26 27 29 34 34 38 


22 26 27 29 34 37 40 
19 20 21 22

Qintra 
 20 21 22 23
26 27 29 32 35 40 48 


27 29 32 35 40 48 58 
 21 22 23 24
 22 23 24 26
27 29 34 38 46 56 69 


29 35 38 46 56 69 83 
 23 24 35 27
Digital Video Compression Fundamentals and Standards
20 21 22 23 

21 22 23 24 
22 23 24 25 

23 24 26 27 
25 26 27 28 

26 27 28 30 
27 28 30 31 

28 30 31 33 
24
The MPEG-1 Encoder (4/4)
Motion Compensation
Exploit motion vector and the previous reconstructed frame to generate
the predictive frame
 n ( x, y )   n ( x  vx ( p, q), y  v y ( p, q)), ( x, y )  MB( p, q)
 n ( x, y) is the compensated image
n ( x, y) is the previous image
v  (vx ( p, q), v y ( p, q)) is the motion vector
Reference frame
2008/12/26
Target frame
Digital Video Compression Fundamentals and Standards
25
The MPEG-2 Standard
Field/Frame DCT Coding
Field/Frame Prediction Mode Selection
Alternative Scan Order
Various Picture Sampling Formats
User Defined Quantization Matrix
2008/12/26
Digital Video Compression Fundamentals and Standards
26
Progressive Scan and Interlaced
Scan
Progressive Scan
2008/12/26
Interlaced Scan
Digital Video Compression Fundamentals and Standards
27
Field/Frame DCT Coding
The field type DCT
Fast motion video
The frame type DCT
Slow motion video
Field DCT Coding
2008/12/26
Luminance MB
Frame DCT Coding
Digital Video Compression Fundamentals and Standards
28
Alternative Scan Order
Zigzag scan order
Frame DCT
Alternative scan order
Field DCT
1
0
1
0
1
0
1
0
2008/12/26
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
Zig-zag
1 1 1
0 0 0
1 1 1
0 0 0
1 1 1
0 0 0
1 1 1
0 0 0
scan
1 1
0 0
1 1
0 0
1 1
0 0
1 1
0 0
2D DCT
4
0.72
0
0.85
0
1.27
0
3.62
Alternate
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
Digital Video Compression Fundamentals and Standards
scan
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0
0
0
0
0
0
0
0
29
The MPEG-2 Encoder (2/2)
Quantization
User can change the quantization if necessary
Intra quantization matrix
Inter quantization matrix
Qintra
8

 16
 19

22

 22

 26
 26

 27
16 16 16 16
16 19 22 26 27 29 34 


16 22 24 27 29 34 37 
16 16 16 16
16 16 16 16
22 26 27 29 34 34 38 


16 16 16 16
22 26 27 29 34 37 40 
Qinter  
16 16 16 16
26 27 29 32 35 40 48 


27 29 32 35 40 48 58 
16 16 16 16
16 16 16 16
27 29 34 38 46 56 69 


29 35 38 46 56 69 83 
16 16 16 16
16 16 16 16 

16 16 16 16 
16 16 16 16 

16 16 16 16 
16 16 16 16 

16 16 16 16 
16 16 16 16 

16 16 16 16 
Various picture sampling formats
4:4:4
4:2:2
4:2:0
2008/12/26
Digital Video Compression Fundamentals and Standards
30
The MPEG-2 Encoder (1/2)
SNR Enhanced Layer
Entropy
Coding
Q
+
Base Layer
Bits Enhance
Basic quality requirement
For SDTV
Q-1
+
Enhanced Layer
High quality service
For HDTV
DCT
Entropy
Coding
Q
Bits Base
Q-1
Q-1
+
Motion
Compensation
+
Base
Layer
IDCT
Motion
Estimation
+
+
Frame
Memory
2008/12/26
Digital Video Compression Fundamentals and Standards
31
H.264/AVC
Variable Block Size
Multiple Reference Frames
Integer Transform
Intra Prediction
In-loop Deblocking Filtering
1/4-pel Resolution Motion Estimation
CAVLC
2008/12/26
Digital Video Compression Fundamentals and Standards
32
Variable Block Size
The fixed block size may not be suitable for all
motion objects
Improve the flexibility of comparison
Reduce the error of comparison
7 types of blocks for selection
16 x 16
0
8x8
0
2008/12/26
16 x 8
0
1
8x4
0
1
8 x 16
0
1
4x8
0
1
8x8
0
1
2
3
4x4
0
1
2
3
Digital Video Compression Fundamentals and Standards
33
Multiple Reference Frames
The neighboring frames are not the most similar in some cases
The B-frame can be reference frame
B-frame is close to the target frame in many situations
2008/12/26
Digital Video Compression Fundamentals and Standards
34
Integer Transform for Reducing The
Spatial Redundancy (1/2)
The transform matrix C
1 1 1 1 


2
1

1

2

C 
 1 1 1 1 


 1 2 2 1 
4×4 Block Size
Separable Integer Transform
The transform coefficients are CXCT
2008/12/26
Digital Video Compression Fundamentals and Standards
35
Integer Transform for Reducing The
Spatial Redundancy (2/2)
0.5
0.5
0.5   x00
 0.5
 0.6325 0.3162 0.3162 0.6325   x
  10
Y  AXAT  
 0.5
0.5
0.5
0.5   x20


 0.3162 0.6325 0.6325 0.3162   x30
 1 1 1 1   x00

2 1 1 2   x10


=
 1 1 1 1   x20
 

 1 2 2 1  x30
 1 1 1 1   x00

2 1 1 2   x10


=
 1 1 1 1   x20
 

 1 2 2 1  x30
C
2008/12/26
x01
x02
x11
x12
x21
x22
x31
x32
x01
x02
x11
x12
x21
x22
x31
x32
X
x01
x02
x11
x12
x21
x22
x31
x32
x03  0.5 0.6325 0.5 0.3162 
x13  0.5 0.3162 0.5 0.6325 


x23  0.5 0.3162 0.5 0.6325 


x33  0.5 0.6325 0.5 0.3162 
x03  0.5 0.6325 0.5 0.3162    0.5
0.5
0.5
0.5 
 




x13  0.5 0.3162 0.5 0.6325 

  0.3162 0.3162 0.3162 0.3162 
x23  0.5 0.3162 0.5 0.6325    0.5
0.5
0.5
0.5 

  

x33  0.5 0.6325 0.5 0.3162   0.3162 0.3162 0.3162 0.3162 
x03  1 2 1 1    0.25 0.1581 0.25 0.1581

x13  1 1 1 2    0.1581
0.1
0.1581
0.1 

 
x23  1 1 1 2    0.25 0.1581 0.25 0.1581

 

x33  1 2 1 1   0.1581
0.1
0.1581
0.1 
CT
E
Digital Video Compression Fundamentals and Standards
36
Intra Prediction
Predict the similarity between the neighboring pixels in one
frame in advance, and exploit differential coding transform
coding to remove the redundancy.
Intra
Prediction
Vertical
Transform/
Quantization
DC
+ + + +
Horizontal
+
+
+
+
Vertical right
2008/12/26
Entropy
Coding
Diagonal down-left
Diagonal right
Mean
Horizontal down
Vertical left
Horizontal up
Digital Video Compression Fundamentals and Standards
37
Remove Perceptual Redundancy
In-loop deblocking filtering
Remove blocking artifact
Result from block based motion compensation
Result from block based transform coding
q0 q1 q2 q3
QP
p3 p2 p1 p0
2008/12/26
Digital Video Compression Fundamentals and Standards
38
1/4-pel Resolution Motion
Estimation
Integer search positions
Best integer match
Half-pel search positions
Best half-pel match
Quarter-pel search positions
Best quarter-pel match
2008/12/26
Digital Video Compression Fundamentals and Standards
39
The H.264/AVC Encoder
Control Data
Coder
Controller
Transform/
Quantization
Motion
Compensation
Intra-frame
Prediction
Residue
Entropy
Coding
Inverse
Transform/
De-Quantization
Motion Vector
De-blocking
Filter
Motion
Estimation
2008/12/26
Digital Video Compression Fundamentals and Standards
40
Outline
Introduction
Video Compression Standards
Simulation Reference Software
Future Work and Conclusions
2008/12/26
Digital Video Compression Fundamentals and Standards
41
H.264 Reference Software
JM http://iphome.hhi.de/suehring/tml/
Current software version: JM 15
Benchmark
2008/12/26
Digital Video Compression Fundamentals and Standards
42
Reference Software Demo
2008/12/26
Digital Video Compression Fundamentals and Standards
43
Outline
Introduction
Video Compression Standards
Simulation Reference Software
Future Work and Conclusions
2008/12/26
Digital Video Compression Fundamentals and Standards
44
Future Work
Fast Mode Decision Algorithm
Interpolation Filter Design
Deblocking Filter Design
DCT-Based Motion Estimation
Implementation Based on TI DSP
2008/12/26
Digital Video Compression Fundamentals and Standards
45
Fast Mode Decision Algorithm
The computational cost of H.264 is large
Variable block-size ME
Multiple reference frames
Fast mode decision is needed for reducing the computation
time
2008/12/26
Digital Video Compression Fundamentals and Standards
46
Interpolation Filter Design
In order to estimate and compensate the fractional-pel
displacements
Adaptive Interpolation filter for replacing the fixed coefficient
filter
2008/12/26
Digital Video Compression Fundamentals and Standards
47
Deblocking Filter Design
Block based ME and Transform result in the annoying
blocking artifact
Reduce the blocking artifact can increase the quality of
compressed video
2008/12/26
Digital Video Compression Fundamentals and Standards
48
DCT Based Motion Estimation
Robust even in noisy environment
Complexity comparison
DCT Based ME
O(M2)
Block Based ME
O(N2×M2)
Concept
Pseudo Phase
Similar to F{x(t  )}  e j F{x(t )}
2008/12/26
Digital Video Compression Fundamentals and Standards
49
Implementation Based on TI DSP
TMS320C6416
TI DM642
2008/12/26
Digital Video Compression Fundamentals and Standards
50
Conclusions
Thank You
Q&A
2008/12/26
Digital Video Compression Fundamentals and Standards
51
References (1/2)
[1] Yun Q.Shi and Huifang Sun, “Image and Video Compression for Multimedia
Engineering: Fundamentals, Algorithms, and Standards”, CRC press, 2000.
[2] Yao Wand, Jorn Ostermann and Ya-Qin Zhang, “Video Processing and
Communications”, Prentice Hall, 2007.
[3] Richardson, Lain E. G., “Video Codec Design: Developing Image and Video
Compression Systems”, John Wiley & Sons Inc, 2002.
[4] Barry G, Haskell, Atul Puri and Arun N. Netravali, “Digital Video : An
Introduction to MPEG-2”, Boston : Kluwer Academic, 1999.
[5] T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, “Overview of the
H.264/AVC video coding standard”, IEEE Trans. on Circuits and systems for
video Technology, vol. 13, no. 7, pp. 560-576, July 2003.
[6] G. Sullivan and T. Wiegand, “Video Compression - From Concepts to the
H.264/AVC Standard”, Proceedings of the IEEE, Special Issue on Advances in
Video Coding and Delivery, December 2004.
[7] 酒井善則、吉田俊之 共著，白執善 編譯，“影像壓縮技術”，全華，
2004.
2008/12/26
Digital Video Compression Fundamentals and Standards
52
References (2/2)
[8] Thomas Wedi, “Adaptive Interpolation Filters and High-Resolution
Displacements For Video Coding”, IEEE Trans. on Circuits and Systems For
Video Technology, vol.
[9] Dong-Hwan Kim, Hwa-Yong Oh, O˘guzhan Urhan, Sarp Ertürk and Tae-Gyu
Chang, “Optimal Post-Process/In-Loop Filtering for Improved Video
Compression Performance”, IEEE Trans. on Consumer Electronics, vol. 53, no.
4, Nov. 2007.
[10] Shu-Fa Lin, Meng-Ting Lu, and Homer Chen, ” Fast Multi-Frame Motion
Estimation for H.264 and Its Applications to Complexity-Aware Streaming”,
IEEE International Symposium on Circuits and Systems, 2005.
[11] Kai-Ting Cheng and Hsueh-Ming Hang, “Acceleration and Implementation
of H.264 Encoder and Scalable Extension of H.264 Decoder on TI DSP
Platform”, master thesis, June 2007
2008/12/26
Digital Video Compression Fundamentals and Standards
53