Topics in Signal Processing Course ID: EE5359 Project Proposal: HEVC Lossless Coding and Improvements SUBMITTED BY: SUJATHA GOPALAKRISHNAN STUDENT ID: 1001024145 Table of Contents ACRONYMS: ................................................................................................................................. 3 HEVC .............................................................................................................................................. 5 Block Diagram HEVC .................................................................................................................... 5 HEVC Lossless Coding .................................................................................................................. 6 Block Diagram HEVC Lossless Coding ......................................................................................... 7 Basic Definitions ............................................................................................................................. 7 H.264[1][2] ................................................................................................................................. 7 Inter Frame .................................................................................................................................. 7 Intra Frame .................................................................................................................................. 7 Loop Filters ................................................................................................................................. 7 De-blocking Filter ................................................................................................................... 8 Sample Adaptive Offset .......................................................................................................... 8 Block-Based Angular Intra Prediction ........................................................................................ 8 Sample-Based Angular Intra Prediction ..................................................................................... 8 Coding Tools ............................................................................................................................... 8 LCU/CTU ............................................................................................................................... 9 Parallel Processing .................................................................................................................. 9 Entropy Coding ....................................................................................................................... 9 Motion Estimation .................................................................................................................. 9 Motion Compensation........................................................................................................... 10 Algorithm of Sample Based Angular Intra Prediction .................................................................. 10 Proposed Project ........................................................................................................................... 12 Projected Enhancement in the Project .......................................................................................... 12 References ..................................................................................................................................... 12 ACRONYMS: Two dimension 3D Three dimension ACM MoVid Association for Computer Machinery Mobile Video AHG Ad Hoc Groups AIF Adaptive Interpolation Filter ALF Adaptive Loop Filter Advanced Motion Vector Prediction AMVP APIF Adaptive Pre-Interpolation Filter ASIC Application-Specific Integrated Circuit Advanced Video Coding AVC AVS Audio Video Standard BBC British Broadcasting Corporation BD Bjontegaard Distortion BL Base Layer bpp Bits per pixel BS Boundary Strength CU Coding Unit CI Confidence Interval CABAC Context Adaptive Binary Arithmetic Coding Central Processing Unit CPU CRA Clean Random Access CSVT Circuits and Systems for Video Technology CU Coding Unit Discrete Cosine Transform DCT DCTIF Discrete Cosine Transform Interpolation Filters DDCT Directional Discrete Cosine Transform DSP Digital Signal Processing DST Digital Sine Transform EC Error Concealment Finite Impulse Response FIR FPGA Field Programmable Gate Array fps Frames per second GPU Graphics Processing Unit HDR High Definition Range High efficiency video coding HEVC HEVStream High Efficiency Video Stream HTTP Hyper Text Transfer Protocol IEEE Conference on Industrial Electronics and Applications ICIEA Institute of Electrical and Electronics Engineers IEEE INTDCT Integer Discrete Cosine Transform intra HE Intra high efficiency IPTV Internet Protocol Television IS & T Information Systems and Technology 2D ISO ITU-T IVMSP JCTVC JM JPEG JPEG-XR JSVM JTC LR Mbit/s MC MDDCT MDDT ME MJPEG MMSP MPEG Mpixel Mpm MV NAB NAL PCM PSNR PU QP RD RDOQ RDPCM ROT RTP SAO SHVC SVC SELC SIP SSVC TB TU International Organization for Standardization Telecommunication Standardization Sector of the International Telecommunications Union Image, Video, and Multidimensional Signal Processing Joint Collaborative Team on Video Coding Joint Model Joint Photographic Experts Group JPEG extended range Joint Scalable Video Model Joint Technical Committee Low Resolution Megabit per second Motion Compensation Modified Directional Discrete Cosine Transform Mode-Dependent Directional Transform Motion Estimation Motion JPEG Multimedia Signal Processing Moving Picture Experts Group Megapixel Most Probable Modes Motion Vector National Association of Broadcasters Network Abstraction Layer Pulse Code Modulation Peak-to-peak signal to noise ratio Prediction Unit Quantizer parameter Rate Distortion Rate-distortion optimized quantization Residual Differential Pulse Code Modulation Rotational Transform Real-time Transport Protocol Sample adaptive offset Scalable High Efficiency Video Coding Scalable Video Coding Sample based weighted prediction for Enhancement Layer Coding Signal and Image Processing Spatially Scalable Video Coding Transform Block Transform Unit HEVC High Efficiency Video Coding (HEVC) [1] [2] is a video compression standard, a successor to H.264/MPEG-4 AVC [22]. HEVC is said to double the data compression ratio compared to H.264/MPEG-4 AVC [1] at the same level of video quality [2]. The design of most video coding standards is primarily aimed at having the highest coding efficiency HEVC benefits from the use of larger Coding Tree Unit (CTU) sizes. The HEVC video coding layer uses the same "hybrid" approach used in all modern video standards, starting from H.261 [1], in that it uses inter-/intra-picture prediction and 2D transform coding. The main goal of the HEVC standardization effort is to enable significantly improved compression performance relative to existing standards, in the range of 50% bit rate reduction for equal perceptual video quality [10] [11]. Block Diagram HEVC Figure 1: HEVC Encoder [2] Figure 2: HEVC Decoder Block Diagram [3] Some differences in HEVC [1][2] are coding tree units instead of macro blocks, single entropy coding methods-Context Adaptive Binary Arithmetic Coding (CABAC) [15] method and features like tiles , wave front parallel processing and dependent slices to enhance parallel processing. HEVC Lossless Coding The lossless coding mode of HEVC main profile bypasses transform quantization and in-loop filters as shown in the fig.2 [4][19]. Comparing it with non-lossless coding mode, it has smallest quantization parameter value. Lossless coding mode provides perfect fidelity and average bit rate reduction. Outperforms the existing lossless compression solution such as JPEG-2000 [22] and JPEG-LS [22]. It employs Sample Angular-based Intra-Prediction (SAP) [4]. Same prediction mode signaling method. Same interpolation method of HEVC. Uses adjacent neighbors as reference shown in fig.7. Block Diagram HEVC Lossless Coding Figure 3: HEVC lossless Algorithm Block Diagram [4] The blocks that are marked bypass are not being used when implementing a HEVC [1][2] lossless algorithm, thereby providing average bit rate reduction. Basic Definitions H.264 [1][2] H.264/MPEG-4 AVC [1] [2] [22] is a block-oriented, motion-compensation based video compression standard. Inter Frame An inter frame is a frame in a video compression stream which is expressed in terms of one or more neighboring frames. The "inter" part of the term refers to the use of Inter frame prediction. Intra Frame The term intra-frame refers to the various lossless and lossy compression techniques that happens relative to information which is contained only within the current frame and not relative to any other frame in the video sequence. Loop Filters HEVC [1] specifies two loop filters that are applied sequentially; the de-blocking filter (DBF) [4] applied first and the sample adaptive offset (SAO) filter applied afterwards. Both loop filters are applied in the inter-picture prediction loop, i.e. the filtered image is stored in the decoded picture buffer (DPB) as a reference for inter-picture prediction. De-blocking Filter The DBF is similar to the one used by H.264/MPEG-4 AVC [1] [2], but with a simpler design and better support for parallel processing. DBF first apply horizontal filtering for vertical edges to the picture and only after that does, it apply vertical filtering for horizontal edges to the picture. This allows for multiple parallel threads to be used for the DBF [1]. Sample Adaptive Offset The SAO filter is applied after the DBF and is designed to allow for better reconstruction of the original signal amplitudes by applying offsets stored in a lookup table in the bit stream. Block-Based Angular Intra Prediction It is a method of computing predicted samples produced by PU when lossless coding is not enabled. It is defined to exploit spatial sample redundancy in intra coded CUs. As shown in the fig.4, a total of 33 angles are defined for the angular prediction, which can be categorized into two classes: vertical and horizontal angular predictions as illustrated [14]. Figure 4: Block Based Angular Intra Prediction in HEVC [4] Sample-Based Angular Intra Prediction It is a method of computing predicted samples produced by PU when lossless coding is enabled. It is explained detail in the following. Coding Tools Coding efficiency is the ability to encode video at the lowest possible bit rate while maintaining a certain level of video quality. This could be achieved with the following coding tools. LCU/CTU Coding tree unit (CTU) is the basic processing unit of the HEVC video standard and conceptually corresponds in structure to macroblock units, which were used in several previous video standards. CTU is also referred to as largest coding unit (LCU) [1] [2]. In the HEVC, one frame is divided into a series of non-overlapped Coding Tree Unit (CTU) [9] [12]. Figure 5: Division of a CTB into CBs and transform blocks TB [2] Parallel Processing A picture is divided into tiles. Main purpose of these tiles is that, they can be decoded /encoded individually in a simultaneous way called parallel processing. Parallel computing is basically a technique in which multiple computation tasks are assigned to multiple processes and process the job simultaneously. The basic approach for parallel processing is to break the task into multiple smaller tasks and further assign each task to each of the thread which performs required operations in parallel. Parallelization can sometimes get complicated due to race conditions, data dependency, synchronization and communication among different threads [13]. Entropy Coding HEVC uses a context-adaptive binary arithmetic coding (CABAC) algorithm that is fundamentally similar to CABAC in H.264/MPEG-4 AVC. CABAC is the only entropy encoder method that is allowed in HEVC while there are two entropy encoder methods allowed by H.264/MPEG-4 AVC. CABAC and the entropy coding of transform coefficients in HEVC are designed for a higher throughput than H.264 while maintaining higher compression efficiency for larger transform block sizes relative to simple extensions [15][16]. These techniques include reducing context coded bins, grouping bypass bins, grouping bins with the same context, reducing context selection dependencies, reducing total bins, and reducing parsing dependencies. It also describes reductions to memory requirements that benefit both throughput and implementation costs. Motion Estimation Motion estimation [5] is an essential process in many video coding standards like MPEG-2, H.264/AVC and HEVC [1] [2]. Motion estimation has been used at the encoder. Motion Estimation itself consumes more than 50% coding complexity or time to encode. To reduce the computation time, many fast motion estimation Algorithms were proposed and implemented [5]. HEVC allows for two MV modes which are Advanced Motion Vector Prediction (AMVP) and merge mode. AMVP uses data from the reference picture and can also use data from adjacent prediction blocks. The merge mode allows for the MVs to be inherited from neighboring prediction blocks. Merge mode in HEVC is similar to "skipped" and "direct" motion inference modes in H.264. Motion estimation process in HEVC consumes more than 50% coding complexity or time to encode with equal perceptual quality [6] [7]. Many block based motion estimation algorithms [8] [9] are proposed and also implemented to reduce the computation time. Figure 6: Illustration of Motion Estimation process [5] Motion Compensation The interpolation of fractional luma sample positions HEVC uses separable application of one-dimensional half-sample interpolation, with an 8-tap filter or quarter-sample interpolation with a 7-tap filter [5]. While H.264/MPEG-4 AVC[1][2] uses a two-stage process that first derives values at half-sample positions, using separable one-dimensional 6-tap interpolation followed by integer rounding; then applies linear interpolation between values at nearby half-sample positions to generate values at quarter-sample positions. HEVC has improved precision due to the longer interpolation filter and the elimination of the intermediate rounding error. As in H.264/MPEG-4 AVC [1] [2], a scaling and offset operation may be applied to the prediction signal(s) in a manner known as weighted prediction [1]. Algorithm of Sample Based Angular Intra Prediction The SAP [4] is designed to better exploit the spatial redundancy in the lossless coding mode by generating intra prediction samples from adjacent neighbors. The design principle here is very similar to the sample-based DPCM in [21][4] H.264/MPEG-4 AVC [20][4] lossless coding, but SAP[4] is fully harmonized with the HEVC block-based angular intra prediction, and can be applied to all the angular intra prediction modes specified in HEVC [4]. As shown in the fig.7 SAP is performed sample by sample. The adjacent neighboring samples , of the current sample in the current PU are used for prediction. That is, the reference samples used for prediction are not limited to those boundary reference samples from the left and upper neighboring PUs. The SAP has to be processed in a predefined order to ensure the availability of these adjacent neighbors for prediction. Figure 7: Algorithm of SAP [4] Proposed Project The aim of this proposal is to analyze HEVC lossless coding. Hence if a scheme is designed that will reduce the computation time using HEVC lossless coding it will further increase its performance and reduce encoding time without much increase in complexity [21]. The simulation will be conducted using HM Software 16.0 [18] [26], with different video sequences [29], search range, block sizes and number of frames using GPU multicore computing. Bit rate differences, Compression ratio, Distribution of intra-coded ratio (ratio between intra-coded and coded blocks), and results based on it, will be calculated for different algorithms using various search patterns. Projected Enhancement in the Project Performance analysis of HEVC Lossless Mode using HM16.0 [18] software. Performance analysis of SAP [4] using HM16.0 [18] software. Complete Study of HEVC Lossless Coding [4]. References [1] G.J. Sullivan et al, “Overview of the high efficiency video coding (HEVC) standard”, IEEE Trans. CSVT, vol. 22, pp.1649-1668, Dec 2012. [2] G.J. Sullivan et al, “Standardized Extensions of High Efficiency Video Coding (HEVC)”, IEEE Journal of selected topics in Signal Processing, vol.7, pp.1001-1016, Dec 2013. [3] C. Fogg, “Suggested figures for the HEVC specification”, ITU-T/ISO/IEC Joint Collaborative Team on Video Coding (JCT-VC) document JCTVC- J0292r1, July 2012. [4] M. Zhou et al, “HEVC lossless coding and improvements”, IEEE Trans. CSVT, vol.22, pp.1839-1843, Dec 2013. [5] N. Purnachand et al, "Fast Motion Estimation Algorithm for HEVC", IEEE Second International Conference on Consumer Electronics-Berlin (ICCE-Berlin), vol.11, pp.34-37, Sep 2012. [6] P. Hanhart et al, “ Subjective quality evaluation of the upcoming HEVC video compression standard”, SPIE Optical Engineering+ Applications. International Society for Optics and Photonics, vol. 8499, pp.84990v-84990v, Aug 2012. [7] M. Horowitz et al, “Informal subjective quality comparison of video compression performance of the HEVC and H.264/MPEG - 4 AVC standards for low delay applications”, SPIE Optical Engineering+ Applications. International Society for Optics and Photonics, vol.84990, pp.84990w-84990w, Aug 2012. [8] A. Abdelazim, W. Masri and B. Noaman., "Motion estimation optimization tools for the emerging high efficiency video coding (HEVC)", IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, vol. 9029, pp. 902905-902905, Feb 17 2014. [9] M. Jakubowski and G. Pastuszak, “Block-based motion estimation algorithms-a survey”, Journal of Opto-Electronics Review, vol. 21, pp.86-102, Mar 2013. [10] B. Bross et al, “High Efficiency Video Coding (HEVC) Text Specification Draft 10”, Document JCTVC-L1003, ITU-T/ISO/IEC Joint Collaborative Team on Video Coding (JCT-VC), Jan. 2013, available on http://phenix.it-sudparis.eu/jct/doc_end_user/current_document.php?id=7243 [11] J. Ascenso et al, "Improving Frame Interpolation with Spatial Motion Smoothing for Pixel Domain Distributed Video Coding", 5th EURASIP Conference on Speech and Image Processing, Multimedia Communications and Services, pp.1-6, Smolenice, Slovak Republic, July 2005. [12] X. Wang et al, “Paralleling Variable Block Size Motion Estimation of HEVC on Multicore CPU plus GPU platform”, IEEE International Conference on Image Processing (ICIP), vol.22, pp. 1836-1839, Sep. 2013. [13] Introduction to parallel computing https://computing.llnl.gov/tutorials/parallel_comp/#Whatis [14] L. Zhao et al, “Group-Based Fast mode decision algorithm for intra prediction in HEVC”, IEEE Eighth international Conference on Signal Image Technology and Internet based Systems. Article no.6115979, pp. 225-229, Nov 2011. [15] V. Sze and M. Budagavi, "High Throughput CABAC Entropy Coding in HEVC", IEEE Transactions on Circuits and Systems for Video Technology, vol.22, no.12, pp.1778-1791, Dec 2012. [16] T.Nguyen et al, "Transform Coding Techniques in HEVC", IEEE Journal of Selected Topics in Signal Processing, vol.7, pp.978–989, Dec 2013. [17] HEVC tutorial by I.E.G. Richardson: http://www.vcodex.com/h265.html [18] HEVC Reference Software HM16.0. https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-16.0rc1/ [19] B. Bross et al,“High Efficiency Video Coding (HEVC)Text Specification Draft 8”, JCT-VC document, JCTVC-J1003, Stockholm, Sweden, Jul. 2012. http://phenix.it-sudparis.eu/jct/doc_end_user/current_document.php?id=6465 [20] Joint Video Team, “Advanced Video Coding for Generic Audiovisual Services”, ITU-T Rec. H.264 and ISO/IEC, 14496-10 (MPEG-4) AVC, pp.H.100-H.869, Feb 2014. [21] Y.L. Lee et al, "Improved lossless intra coding for H.264/MPEG-4 AVC", IEEE Trans. Image Process., vol.15, no.9, pp.2610-2615, Sep 2006. [22]K.R. Rao, D.N. Kim and J.J Hwang, “High Efficiency Video Coding (HEVC) Revised/Updated Chapter from the book Video Coding Standards”–Springer 2014. [23] ITU-T website: http://www.itu.int/ITU-T/index.html [24] JCT-VC documents are publicly available at http://ftp3.itu.ch/av-arch/jctvc-site and http://phenix.it-sudparis.eu/jct/ [25] V.Sze, M.Budagavi, and G.J. Sullivan “High Efficiency Video Coding (HEVC)” Springer, 2014. [26] Software reference manual for HM: https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/branches/HM-9.2-dev/doc/software-man ual.pdf [27] M. Wien, “High efficiency video coding: Tools and specification”, Springer, 2015. [28] I.E. Richardson, “Coding video: A practical guide to HEVC and beyond”, Wiley, 11 May 2015 [29] Video Sequences: http://ultravideo.cs.tut.fi/