Speaker: Wei-Yi Wei
Advisor: Prof. Jian-Jung Ding
Digital Image and Signal Processing Lab
GICE, National Taiwan University

Introduction
Video Compression Standards
Simulation Reference Software
Future Work and Conclusions
2008/12/26 Digital Video Compression Fundamentals and Standards 2

Introduction
Video Compression Standards
Simulation Reference Software
Future Work and Conclusions
2008/12/26 Digital Video Compression Fundamentals and Standards 3

Why video compression technique is important ?
One movie video without compression
720 x 480 pixels per frame
30 frames per second
Total 90 minutes
Full color
The total quantity of data = 167.96 G Bytes !!
2008/12/26 Digital Video Compression Fundamentals and Standards 4

What is the difference between video compression and image compression?
Temporal Redundancy
Coding method to remove redundancy
Intraframe Coding
Remove spatial redundancy
Interframe Coding
Remove temporal redundancy
Digital Video Compression Fundamentals and Standards 2008/12/26 5

The most intuitive method to remove
Spatiotemporal redundancy
3-Dimensional DCT
Remove spatiotemporal correlation
Good for low motion video
Bad for high motion video
2008/12/26

8
N
3
N

1 N

1 N

1  t

0 x

0 y

0




(2 x

1) u
2 N

 cos



(2 y

1) v
2 N

 cos



(2 t

1) w
2 N

 for u

0,..., N

1 , v

0,..., N

1 and w

0,..., N

1 where N

C k


1/ 2 for k

0
1 otherwise
Digital Video Compression Fundamentals and Standards 6

The most popular method to remove temporal redundancy
The Block-Matching Algorithm
2008/12/26 Digital Video Compression Fundamentals and Standards 7

 p q  
Vy

1 Vx

M [
 n
( , ),
 n

1
(
The matching criteria
Mean square error (MSE)
( , )

( - )
2
High precision is needed
Mean absolute difference (MAD)
 x
 y
)]
2008/12/26
Low precision is enough
Digital Video Compression Fundamentals and Standards
 n

1
8

The Exhaustive Block-Matching
Algorithm
Reference Frame
Motion
Vector
Current Frame
|A|=12
12 22 33 44 55 66 77 88
12 22 33 44 55 66 77 88
12 22 33 44 55 66 77 88
12 22 33 44 55 66 77 88
11 23 34 44 55 66 77 88
11 23 34 44 55 66 77 88
2008/12/26
Search
Range
11 22 33 44 55 66 77 88
11 22 33 44 55 66 77 88
11 22 33 44 55 66 77 88
11 22 33 44 55 66 77 88
11 22 33 44 55 66 77 88
11 22 33 44 55 66 77 88
11 22 33 44 55 66 77 88
11 22 33 44 55 66 77 88
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 1 1 0 0 0 0 0
0 1 1 0 0 0 0 0
0 1 1 0 0 0 0 0
0 1 1 0 0 0 0 0
Digital Video Compression Fundamentals and Standards 9

EBMA needs Intensive computation


 16 16



 
128 128
 
16 16
   
170 G instructions/sec
Fast Algorithm is needed
Find the possible local optimal
Reference Frame Current Frame
2008/12/26 Digital Video Compression Fundamentals and Standards 10

Fast Block-Matching Algorithms
The characteristics of fast algorithm
Not accurate as EBMA
Save large computation
Two famous fast algorithm
2-D logarithm Search Method
Three Steps Search Method
2008/12/26 Digital Video Compression Fundamentals and Standards 11

2008/12/26
1 2
1 1 2
1
4
4
4
3
Digital Video Compression Fundamentals and Standards
4
12

2008/12/26
Three Step Search Method
1
1
1
1
2
2
2
1
2 2
3 3 3
3 3
2
3 3 3
2
1
1 1
Digital Video Compression Fundamentals and Standards 13

Multiresolution Motion Estimation (1/3)
The number of levels is L l -th level images of the target frames
   l
, t

1, 2,...
where is set of pixels at level L
At the l -th level, the MV is d ( )
At the l -th level, the estimated MV is ( )

U ( d l

1 x
Determine update such that error is minimized error
  x
 l
|

2, l
(
The new motion vector is x d x l
( )

 q x l
( )
 
1, l
 l
( ) p
2008/12/26 Digital Video Compression Fundamentals and Standards 14

Multiresolution Motion Estimation (2/3)
Variable block size method
2008/12/26 Digital Video Compression Fundamentals and Standards 15

Multiresolution Motion Estimation (3/3)
2008/12/26 Digital Video Compression Fundamentals and Standards 16

Introduction
Video Compression Standards
Simulation Reference Software
Future Work and Conclusions
2008/12/26 Digital Video Compression Fundamentals and Standards 17

The Development of Video
Compression Standards
5-point 1D DCT
6-point 1D DCT
5-point 1D DCT
6-point 1D DCT
……
……
……
……
5-point 1D DCT
6-point 1D DCT
……
……
2008/12/26 Digital Video Compression Fundamentals and Standards 18

Group of Pictures
Motion Estimation
Motion Compensation
Differential Coding
DCT
Quantization
Entropy Coding
Digital Video Compression Fundamentals and Standards 2008/12/26 19

I-frame (Intracoded Frame)
Coded in one frame such as DCT.
This type of frame do not need previous frame
P-frame (Predictive Frame)
One directional motion prediction from a previous frame
The reference can be either I-frame or P-frame
Generally referred to as inter-frame
B-frame (Bi-directional predictive frame)
Bi-directional motion prediction from a previous or future frame
The reference can be either I-frame or P-frame
Generally referred to as inter-frame
Digital Video Compression Fundamentals and Standards 2008/12/26 20

The distance between two nearest P-frame or P-frame and I-frame denoted by M
The distance between two nearest I-frames denoted by N
Forward Motion Compensation
GOP N=9
M=3
I B B P B B P B B I
Bidirectional Motion Compensation
Digital Video Compression Fundamentals and Standards 2008/12/26 21

Intra-frame
Residue
Inter-frame
Motion Vector
2008/12/26 Digital Video Compression Fundamentals and Standards 22

Differential Coding t
^
( ) ( ) t
 is the input image
^
 is the predictive image
DCT

2
N
N

1 N

1  x

0 y

0 for u

0,..., N

1 and v

0,..., N

1 where N

C k



(2 x

1) u
2 N
 
 cos 

(2 y

1) v
2 N


 

1/ 2 for k

0
1 otherwise
2008/12/26 Digital Video Compression Fundamentals and Standards 23

Quantization
Intra quantization matrix
Inter quantization matrix
Q intra
 











8
16
16
16
19
22
22
24
26
27
27
29
29
34
34
37
19 22 26 27 29 34 34 38
22 22 26 27 29 34 37 40
22 26 27 29 32 35 40 48
26
26
27
27
29
29
32
34
35
38
40
46
48
56
58
69










27 29 35 38 46 56 69 83 
Q intra
 











16 17
17 18
18 19
19
20
20
21 22
21 22
18
19
20
19
20
21
21 22
23
23
24
20 21 22
21 22
22
23
25
26
23
24
26
27
23
24
26
27
28
23
24
25
27
28
30
22 23 24 26 27 28 30 31










23 24 35 27 28 30 31 33 
2008/12/26 Digital Video Compression Fundamentals and Standards 24

Motion Compensation
Exploit motion vector and the previous reconstructed frame to generate the predictive frame
 n
  n
( x
 x
( , ),
 v y
( , )), ( , )

( , )
 n is the compensated image
 n is the previous image v

( ( , ), x y
( , )) is the motion vector
Reference frame Target frame
2008/12/26 Digital Video Compression Fundamentals and Standards 25

Field/Frame DCT Coding
Field/Frame Prediction Mode Selection
Alternative Scan Order
Various Picture Sampling Formats
User Defined Quantization Matrix
2008/12/26 Digital Video Compression Fundamentals and Standards 26

Progressive Scan and Interlaced
Scan
Progressive Scan Interlaced Scan
2008/12/26 Digital Video Compression Fundamentals and Standards 27

The field type DCT
Fast motion video
The frame type DCT
Slow motion video
Field DCT Coding Luminance MB Frame DCT Coding
2008/12/26 Digital Video Compression Fundamentals and Standards 28

Zigzag scan order
Frame DCT
Alternative scan order
Field DCT
2008/12/26
0 0 0 0 0 0 0 0
1 1 1 1 1 1 1 1
0 0 0 0 0 0 0 0
1 1 1 1 1 1 1 1
0 0 0 0 0 0 0 0
1 1 1 1 1 1 1 1
0 0 0 0 0 0 0 0
2D DCT
0.72 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0.85 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
1.27 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
3.62 0 0 0 0 0 0 0
Digital Video Compression Fundamentals and Standards 29

Quantization
User can change the quantization if necessary
Intra quantization matrix
Inter quantization matrix
Q intra
 











8 16 19 22 26 27 29 34
16 16 22 24 27 29 34 37
19 22 26 27 29 34 34 38
22 22 26 27 29 34 37 40
22 26 27 29 32 35 40 48
26 27 29 32 35 40 48 58
26 27 29 34 38 46 56 69










27 29 35 38 46 56 69 83 
Q inter








16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16

 16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16












Various picture sampling formats
4:4:4
4:2:2
2008/12/26
4:2:0
Digital Video Compression Fundamentals and Standards 30

SNR Enhanced Layer
+
Q
Entropy
Coding
Bits Enhance
Base Layer
Basic quality requirement
For SDTV
+
Enhanced Layer
High quality service
For HDTV
Motion
Compensation
DCT Q
Q -1
Q -1
Bits Base
Q -1
+
+
Entropy
Coding
Base
Layer
Motion
Estimation
IDCT
+
+
Frame
Memory
Digital Video Compression Fundamentals and Standards 2008/12/26 31

Variable Block Size
Multiple Reference Frames
Integer Transform
Intra Prediction
In-loop Deblocking Filtering
1/4-pel Resolution Motion Estimation
CAVLC
Digital Video Compression Fundamentals and Standards 2008/12/26 32

The fixed block size may not be suitable for all motion objects
Improve the flexibility of comparison
Reduce the error of comparison
7 types of blocks for selection
16 x 16 16 x 8 8 x 16 8 x 8
0
8 x 8
0
0
1
8 x 4
0
1
0
0
4 x 8
1
1
0 1
2 3
4 x 4
0 1
2 3
2008/12/26 Digital Video Compression Fundamentals and Standards 33

The neighboring frames are not the most similar in some cases
The B-frame can be reference frame
B-frame is close to the target frame in many situations
2008/12/26 Digital Video Compression Fundamentals and Standards 34

Integer Transform for Reducing The
Spatial Redundancy (1/2)
The transform matrix C
C







1
2
1

1
1
1


1
1
1

1
1
2
1

2 2

1






4 × 4 Block Size
Separable Integer Transform
The transform coefficients are CXC T
Digital Video Compression Fundamentals and Standards 2008/12/26 35

Integer Transform for Reducing The
Spatial Redundancy (2/2)
Y

AXA
T 




0.5
0.5

0.5
0.5
0.6325
0.3162

0.3162

0.6325
0.5

0.5
0.5
0.5
0.3162

0.6325
0.6325

0.3162








 x
00 x
01 x
02 x
03 x x x
10
20
30 x x x
11
21
31 x x x
12
22
32 x x x
13
23
33










0.5
0.5
0.3162

0.5

0.6325
0.5

0.6325
0 .3162

0.5
0.5
0.3162
0.6325




0.5

0.6325
0.5

0.3162
=








1 1 1 1
2 1

1

2
1

1

1 1




 1

2 2

1





 x x x x
00 01 02 03 x
10 x
11 x
12 x
13 x x x x
20 21 22 23 x x x x
30 31 32 33







0.5
0.6325
0.5
0.3162



0.5
0.3162

0.5

0.6325
0.5
0.3162
0.5
0.6325
 0.5


0.6325

0.5

0.3162












0.5
0.3162
0.5
0.3162
=









1
2
1

1
1
1


1
1
1

1
1
2
1

2 2

1
 
 







 x x x x
00 01 02 03 x
10 x
11 x
12 x
13 x x x x
20 21 22 23 x x x x
30 31 32 33







1 2 1 1




1 1

1

2
1

1

 1

2 1
1

2
1
  
  







 


0.25
0.1581
0.25
0.1581

0.1581
0.1
0.1581
0.1


0.25
0.1581
0.25
0.1581

0.1581
0.1
0.1581
0.1
0.5
0.3162
0.5
0.3162
0.3162
0.3162
0.3162
0.5
0.5
0.5
0.5
0.3162




C X C T E
Digital Video Compression Fundamentals and Standards 2008/12/26 36

Predict the similarity between the neighboring pixels in one frame in advance, and exploit differential coding transform coding to remove the redundancy.
Vertical
Vertical right
Intra
Prediction
Horizontal
Transform/
Quantization
Entropy
Coding
Diagonal down-left
+
+
+
+
DC
+ + + +
Mean
Horizontal down Vertical left
Diagonal right
Horizontal up
Digital Video Compression Fundamentals and Standards 2008/12/26 37

Remove Perceptual Redundancy
In-loop deblocking filtering
Remove blocking artifact
Result from block based motion compensation
Result from block based transform coding q
0 q
1 q
2 q
3
QP p
3 p
2 p
1 p
0
Digital Video Compression Fundamentals and Standards 2008/12/26 38

2008/12/26
1/4-pel Resolution Motion
Estimation
Integer search positions
Best integer match
Halfpel search positions
Best halfpel match
Quarterpel search positions
Best quarter-pel match
Digital Video Compression Fundamentals and Standards 39

2008/12/26
Control Data
Coder
Controller
Residue
Transform/
Quantization
Entropy
Coding
Inverse
Transform/
De-Quantization
Motion Vector
Motion
Compensation
Intra-frame
Prediction
Motion
Estimation
De-blocking
Filter
Digital Video Compression Fundamentals and Standards 40

Introduction
Video Compression Standards
Simulation Reference Software
Future Work and Conclusions
2008/12/26 Digital Video Compression Fundamentals and Standards 41

JM http://iphome.hhi.de/suehring/tml/
Current software version: JM 15
Benchmark
2008/12/26 Digital Video Compression Fundamentals and Standards 42

2008/12/26 Digital Video Compression Fundamentals and Standards 43

Introduction
Video Compression Standards
Simulation Reference Software
Future Work and Conclusions
2008/12/26 Digital Video Compression Fundamentals and Standards 44

Fast Mode Decision Algorithm
Interpolation Filter Design
Deblocking Filter Design
DCT-Based Motion Estimation
Implementation Based on TI DSP
2008/12/26 Digital Video Compression Fundamentals and Standards 45

The computational cost of H.264 is large
Variable block-size ME
Multiple reference frames
Fast mode decision is needed for reducing the computation time
2008/12/26 Digital Video Compression Fundamentals and Standards 46

In order to estimate and compensate the fractional-pel displacements
Adaptive Interpolation filter for replacing the fixed coefficient filter
2008/12/26 Digital Video Compression Fundamentals and Standards 47

Block based ME and Transform result in the annoying blocking artifact
Reduce the blocking artifact can increase the quality of compressed video
2008/12/26 Digital Video Compression Fundamentals and Standards 48

Robust even in noisy environment
Complexity comparison
DCT Based ME
O(M 2 )
Block Based ME
O(N 2 × M 2 )
Concept
Pseudo Phase
Similar to { ( )} e { ( )}
Digital Video Compression Fundamentals and Standards 2008/12/26 49

Implementation Based on TI DSP
TMS320C6416
TI DM642
2008/12/26 Digital Video Compression Fundamentals and Standards 50

2008/12/26
Digital Video Compression Fundamentals and Standards 51

2008/12/26
[1] Yun Q.Shi and Huifang Sun, “
Image and Video Compression for Multimedia
Engineering: Fundamentals, Algorithms, and Standards
”, CRC press, 2000.
[2] Yao Wand, Jorn Ostermann and Ya-Qin Zhang, “ Video Processing and
Communications
”, Prentice Hall, 2007.
[3] Richardson, Lain E. G., “ Video Codec Design: Developing Image and Video
Compression Systems ”, John Wiley & Sons Inc, 2002.
[4] Barry G, Haskell, Atul Puri and Arun N. Netravali, “
Digital Video : An
Introduction to MPEG-2
”, Boston : Kluwer Academic, 1999.
[5] T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, “Overview of the
H.264/AVC video coding standard”,
IEEE Trans. on Circuits and systems for video Technology , vol. 13, no. 7, pp. 560-576, July 2003.
[6] G. Sullivan and T. Wiegand, “Video Compression - From Concepts to the
H.264/AVC Standard”, Proceedings of the IEEE , Special Issue on Advances in
Video Coding and Delivery, December 2004.
[7] 酒井善則、吉田俊之 共著，白執善 編譯，“ 影像壓縮技術 ”，全華，
2004.
Digital Video Compression Fundamentals and Standards 52

[8] Thomas Wedi, “Adaptive Interpolation Filters and High-Resolution
Displacements For Video Coding”,
IEEE Trans. on Circuits and Systems For
Video Technology , vol.
[9] Dong-Hwan Kim, Hwa-Yong Oh, O˘guzhan Urhan, Sarp Ertürk and Tae-Gyu
Chang, “Optimal Post-Process/In-Loop Filtering for Improved Video
Compression Performance”,
IEEE Trans. on Consumer Electronics , vol. 53, no.
4, Nov. 2007.
[10] Shu-Fa Lin, Meng-Ting Lu, and Homer Chen, ” Fast Multi-Frame Motion
Estimation for H.264 and Its Applications to Complexity-Aware Streaming”,
IEEE International Symposium on Circuits and Systems , 2005.
[11] Kai-Ting Cheng and Hsueh-Ming Hang, “Acceleration and Implementation of H.264 Encoder and Scalable Extension of H.264 Decoder on TI DSP
Platform”, master thesis, June 2007
2008/12/26 Digital Video Compression Fundamentals and Standards 53