New Techniques for
Improved Video Coding
Thomas Wiegand
Fraunhofer Institute for Telecommunications
Heinrich Hertz Institute
Berlin, Germany
wiegand@hhi.de
Outline
§ Inter-frame Encoder Optimization
§ Texture Analysis and Synthesis
§ Model-Aided Coding
T. Wiegand: New Techniques for Improved Video Coding
1
Scope of Video Coding Standardization
Only Restrictions on the Bitstream, Syntax, and Decoder
are standardized:
• Permits optimization beyond the obvious
• Permits complexity reduction for implementability
• Provides no guarantees of quality
Source
Sink
Pre-Processing
Encoding
Post-Processing
& Error Recovery
Decoding
Scope of Standard
T. Wiegand: New Techniques for Improved Video Coding
2
Hybrid Video Encoder
Input
Video
Signal
Coder
Control
Split into
Macroblocks
16x16 pixels
Control
Data
Transform/
Scal./Quant.
Decoder
Quant.
Transf. coeffs
Scaling & Inv.
Transform
Entropy
Coding
Intra-frame
Prediction
De-blocking
Filter
MotionIntra/Inter Compensation
Output
Video
Signal
Motion
Data
Motion
Estimation
T. Wiegand: New Techniques for Improved Video Coding
3
Inter-frame Optimization of Video Encoding
§ Most video encoders select locally optimal transform
coefficients as they ignore temporal dependencies
Time
Motion Compensation
§ New approach that takes inter-frame dependencies into
account
T. Wiegand: New Techniques for Improved Video Coding
4
Linear Signal Model for Decoder
motion data
transform coefficients
s = Ms + Tc + p
reconstructed sample
values
static predictor
inverse transform & scaling
§ Very large size of matrices
§ Numbers when optimizing 4 QCIF pictures jointly:
M, T: 100,000 x 100,000 entries
T. Wiegand: New Techniques for Improved Video Coding
5
Quadratic Program
§ Use Mean Squared Error distortion
§ Select transform coefficient levels by solving:
original video signal
minimize
subject to:
(s − v) (s − v) + λ ∑ abs(c i )
T
s = Ms + Tc + p
i
§ Equivalent to a quadratic program
§ Can be solved by standard solution methods!
T. Wiegand: New Techniques for Improved Video Coding
6
Results
§
§
§
§
§
§
Used H.264/MPEG4-AVC
Compare looking at 2 or 3 frames at a time
Compare to H.264/MPEG4-AVC Test Model
Constant QP
Motion estimation on original frames
Only optimized inter-frames
T. Wiegand: New Techniques for Improved Video Coding
7
FlowerGarden 30Hz IPPP…..
Average Y PSNR [dB]
39
38
37
36
35
34
33
32
TM2
2 Frames Jointly
3 Frames Jointly
31
30
29
0
100
200
300
Bit-Rate [kbits/sec]
400
500
16% Bit-rate improvement
T. Wiegand: New Techniques for Improved Video Coding
8
FlowerGarden 30Hz IbPbPbP…..
37
Average Y PSNR [dB]
36
35
34
33
32
TM2
3 Frames Jointly
31
30
0
50
100
150
200
Bit-Rate [kbits/sec]
250
300
15% Bit-rate improvement
T. Wiegand: New Techniques for Improved Video Coding
9
Summarizing Inter-frame Optimization
§Very complex: has only recently become possible
§Problems with H.264/MPEG4-AVC: rounding
§Adjust design to support efficient encoding methods
§Heavy incorporation of encoding methods has been
used in H.264/MPEG4-AVC development with
Lagrangian encoder optimization
§Do not standardize the encoder!
T. Wiegand: New Techniques for Improved Video Coding
10
Texture Analysis and Synthesis
§Textures with large amount of visible details are difficult to
code e.g., grass, sand, clouds, water ...
§Viewer often does not perceive large differences between
different versions of grass, sand, clouds, water ...
Where is the original texture ?
T. Wiegand: New Techniques for Improved Video Coding
11
Video Analysis and Synthesis
§ Approach: Exact reproduction of details is irrelevant if
• Textures are shown with limited spatial accuracy
• Viewer does not know the original video
§ Consequence: Mean Squared Error (MSE) distortion criterion is
not suitable for efficient coding of detail-irrelevant textures
§ Issue: Could we use similarity criteria (color histograms, etc)
instead of MSE as coding distortion ?
§ Scheme: If information needed for approximate reproduction of
detail-irrelevant textures requires less bit-rate than using MSE
Ł bit-rate savings
T. Wiegand: New Techniques for Improved Video Coding
12
Video Codec Architecture
Video In
Encoder
TA
Bits
Side Info
Decoder
Video
Out
TS
Video coding using a
texture analyzer (TA) and a texture synthesizer (TS)
T. Wiegand: New Techniques for Improved Video Coding
13
Texture Analyzer Principle
Ref. mask
Ref.
frame
Split & Merge
Segmentation
Motion
Compensation
Texture Analyzer
Current mask
Current frame
T. Wiegand: New Techniques for Improved Video Coding
14
Texture Synthesizer
Stuffing procedure
T. Wiegand: New Techniques for Improved Video Coding
15
Texture Synthesizer
§ Match surrounding samples between current picture and warped
pictures to find synthesis sample for the current sample
§ Decoder processing required!
§ Volume matching for water and temporal consistence
T. Wiegand: New Techniques for Improved Video Coding
16
Flowergarden, CIF, 30 Hz, QP=16, 41 frames
Bit-rate savings: 19.4%
T. Wiegand: New Techniques for Improved Video Coding
17
Canoe, CIF, 30 Hz, QP=16, 73 frames
Bit-rate savings: 8.8 %
T. Wiegand: New Techniques for Improved Video Coding
18
Summarizing Texture Analysis and Synthesis
§ Textures with large amount of visible details are
difficult to code e.g., grass, sand, clouds, water ...
§ Errors in these textures are barely visible
§ Texture analysis and synthesis can help to
represent these textures efficiently
§ Method requires processing at the decoder side
§ Other methods require processing at the decoder
side to achieve coding efficiency improvements
through backward adaptation (CABAC)
T. Wiegand: New Techniques for Improved Video Coding
19
Model-Aided Coding
§ Model-based coding (MBC):
• Extreme low bit-rates possible
• Missing generality
§ Hybrid video coding (H.263/4, MPEG-2/4,...)
• Robust against scene changes
• Higher bit-rates
§ Combination of MBC with hybrid video coding
⇒ Model-aided codec
• High coding efficiency
• No restriction to scene content
T. Wiegand: New Techniques for Improved Video Coding
20
Architecture of the Model-Aided Codec
video
signal
S [ x,y,t ]
e
CoderControl
control
data
Intraframe
DCT Coder
DCTcoefficients
Modelbased
Coder
Intraframe
Decoder
Motion
Compensation
Model-based
Decoder
Model
Frame
Reconst.
Frame
motion
data
FAPs
Decoder
T. Wiegand: New Techniques for Improved Video Coding
21
Coding Results
§ Sequenz Clapper Board: 8.33 Hz, CIF resolution, 260
frames
§ Same average bit-rate (≈12 kbit/s)
§ H.263 annexes D, F, I, J, T
H.263 (TMN-10)
Model-Aided Codec (MAC)
T. Wiegand: New Techniques for Improved Video Coding
22
Reconstruction Quality vs. Bit-rate
3.5 dB
45 %
§ Gain of 3.5 dB PSNR compared to H.263 Coder (TMN-10)
§ Bit-rate reduction ≈ 45 %
T. Wiegand: New Techniques for Improved Video Coding
23
Model-Aided Coding with Approximate Geometry
H.264
(TML-8)
MAC
T. Wiegand: New Techniques for Improved Video Coding
24
Summarizing Model-Aided Coding
§Waveform-coding and 3-D model-based coding
can be combined to efficiently exploit statistical
dependencies in the video signal
§The “trick” is to generate a model/representation of
the video signal and to animate the central
representation to predict the video signal:
§3D computer graphic models
§Mosaics
§MCTF
T. Wiegand: New Techniques for Improved Video Coding
25
Concluding Remarks
Three Techniques are presented that improve
against existing H.264/MPEG4-AVC coding
1. Inter-frame encoder optimization
§ Exploit dependencies over multiple pictures
§ Incorporate/anticipate efficient encoding into decoder
standardization process (do not standardize encoder)
2. Video analysis and synthesis (VAS)
§ Consider subjective methods in video compression
(investigate alternative distortion measures)
§ Allow more processing at the decoder side for backward
adaptive coding
3. Model-aided coding
§ Further enhance exploitation of statistical depenedencies
§ Incorporate more semantic ideas – combination with VAS
T. Wiegand: New Techniques for Improved Video Coding
26
Acknowledgements
§ Peter Eisert
§ Bernd Girod
§ Patrick Ndjiki-Nya
§ Brad Schumitsch
§ Heiko Schwarz
§ Aljoscha Smolic
T. Wiegand: New Techniques for Improved Video Coding
27