advertisement

ELEC 444 – Digital Communications, Tutorial 1, February 2018 Problem 1 Suppose that equal numbers of letter grades A, B, C, D, F are given in a certain class. How many information in bits you have received if the instructor has told you that your grade is not an F? How much information you still need to determine your grade? Solution: Information in bits you have received if the instructor has told you that your grade is not an F: or Pnot F Inot F log 2 ( Pnot F ) 1 PF 1 1 5 4 5 , then I not F log 2 4 5 0.322 bits To know your grade, 4 choices are left: I needed log 2 ( Pgrade ) log 2 (1 / 4) 2 bits Problem 2. Consider a random variable X with alphabet X = {a1, a2, a3, a4} and probabilities PX(a1) = 1/2, PX(a2) = 1/4, PX(a3) = 1/8, PX(a4) = 1/8. What is the entropy of this random variable? Suppose independent trials of the random variable occur at rate r = 100 trials/second. What is then the rate of the source. Can you devise a coder that exactly achieves the rate of the source? Solution: In general, the entropy H(X) of a random variable X is defined as H(X ) x X PX ( x )log 2 PX ( x ) So, now we can write the solution directly as 1 1 1 1 log 2 (2) log 2 (4) log 2 (8) log 2 (8) 2 4 8 8 1 2 3 3 2 4 8 8 4 4 3 3 8 8 8 8 14 8 1.75 bits/symbol H(X ) where we have used the facts that logy(yx) = x and logy(1/x) = –logy(x). Then, the rate of the source is given by R rH ( X ) 175 bits/sec Following encoding "rule" will achieve the entropy limit (and is also easy to decode since no codeword is a prefix of some other codeword): Outcome a1 a2 a3 a4 Probability 1/2 1/4 1/8 1/8 Codeword 0 10 110 111 The average number of bits produced by the coder is now 1 1 1 1 1 2 3 3 1.75 bits/sec H ( X ) 2 4 8 8 meaning that we indeed achieved the entropy limit. It is, however, not obvious when it is possible to reach the entropy limit. This issue will be considered in the Problem 2. Problem 3 Consider an unfair coin that produces heads with probability 1/4. 1. What is the entropy of the coin flip outcome? 2. Suppose the coin is flipped once per second. What is then the rate of this source? 3. Devise a coder to encode successive coin flip outcomes so that the average number of bits per flip is less than one. 4. How does your encoder compare with the rate of the source? Solution: 1. The entropy of the unfair coin flip outcome is given by H ( X ) 0.25 log 2 0.25 0.75 log 2 0.75 0.81 bits . 2. When the coin is flipped once per second, the rate of the source is R rH ( X ) 0.81 bits / sec . 3. If we encode each outcome separately (with bits 0 and 1), the average number of bits produced by the coder is 0.25 1 0.75 1 1. We can also have a more efficient coding scheme that always takes pair of outcomes and encodes them. For example, the following encoder will have smaller average number of bits per flip than the previous trivial coder. Here T = tail, H = head, and P(T) = 3/4, P(H) = 1/4 and consecutive flips are assumed independent. Outcome Probability Codeword TT 9/16 0 TH 3/16 10 HT 3/16 110 HH 1/16 111 So, the average number of bits produced by the coder is 9 3 3 1 27 1 2 3 3 1.6875 bits/pair 16 16 16 16 16 meaning that the average number of bits per flip is then 1.6875 0.8438 bits . 2 This is, of course, still above the entropy of the source but already much closer to it than the trivial coding efficiency. Problem 4 Suppose a source emits r = 2000 Symbols/Sec from an alphabet of size M = 4 with symbol probability and codewords listed in the follwing table: xi Pi Codewords A 1/2 0 B 1/4 1 C 1/8 10 D 1/8 11 1. Calculate the infomation rate. 2. The average code length of the source and the Kraft factor ? 3. Draw a conclusion ? Solution: The Source information rate is equal: R = r H(X) 1 1 1 1 log 2 (1 / 2) log 2 (1 / 4) log 2 (1 / 8) log 2 (1 / 8) 2 4 8 8 1.75bits/sym H(X ) R = 2000 x 1.75 = 1750 bits/sec. The average code length (average number of binary digits per source symbol) is defined by: rb M N Pi N i , r i1 where rb is the signalling rate, r is the symbol rate, and Ni is the codeword length for the ith symbol. N 1/ 2 1 1/ 4 1 1/ 8 2 1/ 8 2 1.25 H ( X ) . • For uniquely decipherable binary code the word length Ni must satisfy M K 2 Ni 1 i 1 Kraft Inequality. Kraft Factor: K 2 1 2 1 2 2 2 2 1.5 1 . • The result N H (X ) is meaningless because K > 1, which tells us that this code is not uniquely decipherable. For example, the code sequence 10011 could be decoded as BAABB or CABB or CAD.