Ch. 6- H.264/AVC Part I (pp.160~199) Sheng-kai Lin Shengkai Lin@CMLAB, NTUCSIE 2004 1 Outline • Introduction • Baseline profile – – – – – – Reference Picture Management Slice Inter-prediction Intra-prediction Deblocking filter Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004 2 H.264/AVC Introduction (I) • 4:2:0 • Define VCL and NAL – VCL: Video Coding Layer – NAL: Network abstraction • Variable block size • Multiple reference frame • ¼ sample motion compensation Shengkai Lin@CMLAB, NTUCSIE 2004 3 H.264/AVC Introduction (II) • CAVLC • CABAC Shengkai Lin@CMLAB, NTUCSIE 2004 4 Terms (I) • Picture (frame) – one or more slices • Slice – group of macroblock (1~ ) – minimal inter-dependency between coded slices can limit the error propagation • Macroblock Shengkai Lin@CMLAB, NTUCSIE 2004 5 Terms (II) • I slice – I macroblock only • P slice – P, I macroblock • B slice – B,I macroblock • SI slice • SP slice Shengkai Lin@CMLAB, NTUCSIE 2004 6 Terms (III) • I macroblock – intra prediction from decoded samples in the current slice • for entire macroblock • for each 4*4 block • P macroblock – inter prediction using past references • 16*16, 16*8, 8*16, 8*8, 4*8, 8*4, 4*4 Shengkai Lin@CMLAB, NTUCSIE 2004 7 Terms (IV) • B macroblock – inter prediction using past/ future references • list 0 – past reference frame buffer • list 1 – future reference frame buffer • RBSP (Raw Byte Sequence Payload) – NAL payload Shengkai Lin@CMLAB, NTUCSIE 2004 8 H.264/AVC profiles Shengkai Lin@CMLAB, NTUCSIE 2004 9 H.264/AVC Encoder Shengkai Lin@CMLAB, NTUCSIE 2004 10 H.264/AVC Decoder Shengkai Lin@CMLAB, NTUCSIE 2004 11 Outline • Introduction • Baseline profile – – – – – – Reference Picture Management Slice Inter-prediction Intra-prediction Deblocking filter Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004 12 Reference Picture Management (I) • Reference pictures are stored in decoded picture buffer (DPB) • short/long term reference picture • A decoded frame may be marked as – unused for reference – short term picture – long term picture Shengkai Lin@CMLAB, NTUCSIE 2004 13 Reference Picture Management (II) • “Sliding Window” memory management – Keep #(long_term_pic+ short_term_pic) – Remove short term picture if lack of space • Adaptive memory control – issued by encoder – change the type of the ref frame • IDR (Instantaneous Decoder Refresh) – clear ref buffer – I frame Shengkai Lin@CMLAB, NTUCSIE 2004 14 Outline • Introduction • Baseline profile – – – – – – Reference Picture Management Slice Inter-prediction Intra-prediction Deblocking filter Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004 15 Redundant coded picture • Send the duplicated part or all of a coded picture • Error resilience Shengkai Lin@CMLAB, NTUCSIE 2004 16 Arbitrary Slice Order (ASO) • The decoding order of the slices could be arbitrary • Application example – reduce end-end transmission delay in RT app Shengkai Lin@CMLAB, NTUCSIE 2004 17 Slice Group (I) • former called “FMO” – Flexible Macroblock Ordering • a subset of the macroblocks and may contain one or more slices • Application example – Error resilience Shengkai Lin@CMLAB, NTUCSIE 2004 18 Slice Group (II) Shengkai Lin@CMLAB, NTUCSIE 2004 19 Outline • Introduction • Baseline profile – – – – – – Reference Picture Management Slice Inter-prediction Intra-prediction Deblocking filter Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004 20 Inter Prediction • Variable block size • ¼ pixel motion compensation Shengkai Lin@CMLAB, NTUCSIE 2004 21 Interpolation Samples (I) 1. Generate ½ samples by six tap FIR • • b, h, m, s FIR para :(1/32, -5/32, 5/8, 5/8, 5/32, 1/32) 2. Generate center ½ sample by FIR • j 3. Generate ¼ sample by linear interpolation Shengkai Lin@CMLAB, NTUCSIE 2004 22 Interpolation Samples (II) Shengkai Lin@CMLAB, NTUCSIE 2004 23 Motion vector prediction (I) • Encoding MV can cost a lot of bits – especially small partition sizes • Neighboring MV is highly correlated – predictable – encode the diff MVD and transmit it Shengkai Lin@CMLAB, NTUCSIE 2004 24 Motion vector prediction (II) • Partitions excluding 16 × 8 and 8 × 16, MVp = (MVA+ MVB + MVC) /3 • For 16 × 8 partitions, MVp of the upper 16 × 8 = MVB ;MVp for the lower 16 × 8 = MVA Shengkai Lin@CMLAB, NTUCSIE 2004 25 Motion vector prediction (III) • For 8 × 16 partitions, MVp for the left 8 × 16 = MVA ;MVp for the right 8 × 16 = MVC • For skipped macroblocks, do as 16 × 16 Inter mode Shengkai Lin@CMLAB, NTUCSIE 2004 26 Motion vector prediction (IV) Shengkai Lin@CMLAB, NTUCSIE 2004 27 Outline • Introduction • Baseline profile – – – – – – Reference Picture Management Slice Inter-prediction Intra-prediction Deblocking filter Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004 28 Intra Prediction • For Luma samples – 4*4 block • 9 prediction modes – 16*16 block • 4 modes – I_PCM • transmit the encoded samples w/o pred. & trans Shengkai Lin@CMLAB, NTUCSIE 2004 29 4*4 luma prediction modes Shengkai Lin@CMLAB, NTUCSIE 2004 30 16*16 prediction modes Shengkai Lin@CMLAB, NTUCSIE 2004 31 8*8 Chroma Prediction Modes • Similar to 16*16 luma intraprediction except the different numbering. – – – – DC(mode 0) horizontal (mode 1) vertical(mode 2) plane (mode 3). Shengkai Lin@CMLAB, NTUCSIE 2004 32 Signaling Intra Prediction Modes (I) • The choices of the mode should be signaled to the decoder. – might cost lots of bits. • Intra modes for neighboring 4 × 4 blocks are often correlated – predict the parameters ! • The prediction mode for luma coded in Intra16×16 mode or chroma coded in Intra mode is signalled in the macroblock header and the following method is not used. Shengkai Lin@CMLAB, NTUCSIE 2004 33 Signaling Intra Prediction Modes (II) B A C • If A and B are available, C = min (A,B) • else if (neither A nor B are available) C = 2 (DC) • else C = available (A,B) Shengkai Lin@CMLAB, NTUCSIE 2004 34 Signaling Intra Prediction Modes (III) • Flag prev_intra4×4_pred_mode – 1: most probable prediction mode is used. – 0: Flag rem_intra4×4_pred_mode indicates the change of the mode Shengkai Lin@CMLAB, NTUCSIE 2004 35 Signaling Intra Prediction Modes (IV) • If rem_intra4×4_pred_mode < most_probable_mode, the prediction mode = rem intra4×4 pred mode • else the prediction mode = (rem_intra4×4_pred_mode+1) Shengkai Lin@CMLAB, NTUCSIE 2004 36 Outline • Introduction • Baseline profile – – – – – – Reference Picture Management Slice Inter-prediction Intra-prediction Deblocking filter Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004 37 Deblocking filter • • • • Filter 4 vertical/horizontal boundaries of luma Filter 2 vertical/horizontal boundaries of chroma Affect up to 3 samples on the either side. The filter is stronger at places where there is likely to be significant blocking distortion – e.g.: such as the boundary of an intra coded macroblock or a boundary between blocks that contain coded coefficients. Shengkai Lin@CMLAB, NTUCSIE 2004 38 Outline • Introduction • Baseline profile – – – – – – Reference Picture Management Slice Inter-prediction Intra-prediction Deblocking filter Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004 39 Transform and Quantisation • 3 transforms – DCT-base transform for all 4*4 residual block – Hadamard transform for 4*4 luma DC coefficient (in 16*16 intra) – Hadamard transform for 2*2 chroma DC coefficient Shengkai Lin@CMLAB, NTUCSIE 2004 40 DCT-base transform (I) • Integer transform • Might be zero mismatch • The code part of the algo need add and shift only • Scale multiplication could be integrated into quantiser. • Inverse operation could be down by 16-bit arithmetic. Shengkai Lin@CMLAB, NTUCSIE 2004 41 DCT-base transform (II) • a=1/2, b = (2/5)1/2, d = 1/2 Shengkai Lin@CMLAB, NTUCSIE 2004 42 Quantisation • Combine with the scale of the DCT • Skip the rest details Shengkai Lin@CMLAB, NTUCSIE 2004 43 4*4 DC luma Transform and Quantisatoion • |ZD(i, j)| = (|YD(i, j)| MF(0,0) + 2f ) >> (qbits +1) • sign (ZD(i, j)) = sign (YD(i, j)) Shengkai Lin@CMLAB, NTUCSIE 2004 44 2*2 DC chroma Transform and Quantisatoion • |ZD(i, j)| = (|YD(i, j)| MF(0,0) + 2f ) >> (qbits +1) • sign (ZD(i, j)) = sign (YD(i, j)) Shengkai Lin@CMLAB, NTUCSIE 2004 45 • Thank you • Next time, we’ll start at CAVLC • Bye… Shengkai Lin@CMLAB, NTUCSIE 2004 46