advertisement

ECEN 1200 Telecommunications 1 10-16-06 Fall 2006 P. Mathys Huffman Code for “Fun” Image The “Fun” image shown below is a bitmap graphic with 32×16 = 512 pixels using 6 different colors: White, yellow, magenta, blue, black, and red. If the different pixel colors are encoded using a fixed length binary representation, such as the one shown in the following table: Color Code White Yellow Magenta Blue Black Red 000 001 010 011 100 101 then 3 bits are needed for each pixel. This results in a total size of 32 × 16 × 3 = 1536 bits when no data compression is used. One data compression strategy to reduce the number of bits needed to represent the image is to use short strings to encode colors that occur frequently and longer strings for colors that are occur less frequently. Counting the number of pixels for each color in the “Fun” image yields: 1 Color # of Pixels Probability White Yellow Magenta Blue Black Red 394 42 30 22 18 6 394/512 = 0.770 42/512 = 0.082 30/512 = 0.058 22/512 = 0.043 18/512 = 0.035 6/512 = 0.012 Sum 512 1.000 Huffman coding is a recursive procedure to build a prefix-free variable length code with shortest average length from the above table. It works as follows. Start by writing down all the colors, together with their probabilities or the number of times they occur, in increasing numerical order, e.g., from right to left as shown below. 394 white 42 yellow 30 magenta 22 blue 18 black 6 red Huffman’s algorithm regards each of the quantities listed above as a leaf of a (upside-down) tree. In a first step the two leaves that occur least frequently are joined by branches to an intermediate (or parent) node which gets labeled with the sum of the weights of the joined leaves as shown next. 24 394 white 42 yellow 30 magenta 22 blue 18 black 6 red Now the procedure is repeated, working with the remaining leaves and the newly created intermediate node. Note that the problem size has now been reduced by one, i.e., from creating a code for 6 quantities to creating a code for 5 quantities in this example. These two features, reduction of the size of the problem in each step, and repetition of the same procedure on the smaller problem, is what characterizes a recursive algorithm. The next step of the Huffman coding algorithm, again combining the two quantities with the lowest weight into an intermediate node, is shown in the figure below. 46 24 394 white 42 yellow 30 magenta 22 blue 2 18 black 6 red The following three figures show the next steps of building the tree for the Huffman code. 118 46 72 394 white 42 yellow 46 24 30 magenta 22 blue 18 black 72 6 red 512 42 yellow 394 white 24 30 magenta 22 blue 18 black 6 red 118 46 72 42 yellow 394 white 24 30 magenta 22 blue 18 black 6 red Once the tree is completed, label the two branches emanating from each parent node with different binary symbols, e.g., using 0 for all left branches and using 1 for all right branches, as shown in the next figure 512 0 394 white 1 118 1 0 72 0 42 yellow 46 0 1 30 magenta 22 blue 1 0 18 black 24 1 6 red The code for each leaf in the tree is now obtained by starting at the root (the node at the top of the upside-down tree) and writing down the branch labels while travelling to the leaf. The result is shown in the figure below. 3 Note that, because all codes are for quantities which are tree leaves (and not intermediate nodes), the code is automatically prefix-free, i.e., no codeword is a prefix of another codeword. This is important because it guarantees unique decodability of the variable length code. The codes for each color are shown again in the following table: Color # of Pixels Code 394 42 30 22 18 6 0 100 101 110 1110 1111 White Yellow Magenta Blue Black Red Thus, the code for white (which occurs most frequently) has the shortest length and the codes for black and red (which occur least frequently) have the longest lengths. The total number of bits required to represent the image using this Huffman code is computed as (number of occurrences of color times number of bits in corresponding code, added over all colors) 394 × 1 + 42 × 3 + 30 × 3 + 22 × 3 + 18 × 4 + 6 × 4 = 772 bits . Dividing the uncompressed size (1536 bits) by the compressed size (772 bits) yields x = 1536/772 = 1.99 and therefore the compression ratio is 1.99 : 1, almost a factor of 2, without losing any information. Another way to characterize the coding scheme is to compute the average number of bits needed per pixel, in this case 772/512 = 1.51 bits/pixel, down from 3 bits/pixel without compression. An interesting question to ask is “What is the smallest number of bits per pixel needed for lossless coding?” The answer to this question was found in 1950 by Claude E. Shannon. He introduced the notion of the entropy of a source as a measure of the amount of uncertainty about the output that a source generates. If that uncertainty is large, then the output of a source contains a lot of information (in an information theoretic sense), otherwise its output is (partially) predictable and contains less information. Shannon proved that the minimum number of bits necessary for a faithful representation of the source output is equal to the entropy (in bits per symbol) of the source. The entropy of a source is computed from its probabilistic characterization. In the simplest case the output symbols from the source, denoted by the random variable X, are i.i.d. (independent and identically distributed) with a probability mass function pX (x). In this case the entropy H(X) of a source that produces M -ary symbols, 0, 1, . . . M −1, is computed as H(X) = − M −1 X pX (x) log2 pX (x) [bits/symbol] . x=0 Note that the logarithm to the base 2 of some number n can be computed as log2 (n) = log10 (n)/ log10 (2) or as log2 (n) = ln(n)/ ln(2). 4 Binary Entropy function. The entropy function of a binary random variable X ∈ {0, 1} with probability mass function pX (x=1) = 1−pX (x=0) is computed as H(X) = −pX (x=0) log2 (pX (x=0)) − pX (x=1) log2 (pX (x=1)) [bits/bit] The graph of this function is shown below. From the graph it is easy to see that the entropy (or uncertainty) is maximized when pX (x = 0) = pX (x = 1) = 1/2, with a maximum value of 1 bit per source symbol (which is also a bit in this case). This would correspond, for example, to the uncertainty that one has about the outcome (heads or tails) of the flip of an unbiased coin. On the other hand, as pX (x = 0) approaches either 0 or 1 (and thus pX (x = 1) approaches either 1 or 0), the entropy (or uncertainty) goes to zero. Using the example of flipping a coin, this would correspond to flipping a heavily biased coin (or even one that has heads or tails on both sides). For the “Fun” image, the entropy (based on the i.i.d. assumption and single pixels) is computed as H(X) = −0.77 log2 0.77 − 0.082 log2 0.082 − 0.058 log2 0.058 − 0.043 log2 0.043+ − 0.035 log2 0.035 − 0.012 log2 0.012 = 1.2655 [bits/pixel] Thus, between the 1.51 bits/pixel achieved by the Huffman code and the theoretical minimum given by the entropy H(X) = 1.27 bits/pixel there is still some room for improvement, but not by a very large amount. To get closer to H(X), one would have to use a larger (Huffman) coding scheme that combines several pixels (e.g., pairs or triplets of pixels) into each coded binary representation. c 1996–2006, P. Mathys. Last revised: 10-17-06, PM. 5