Project 8: The concept of Entropy and its applications to information
and coding
Table of Contents
1.Abstract ....................................................................................................................2
2.Introduction ..............................................................................................................3
2.1.Historical Remarks:................................................................................................3
2.2.Topic High-level Overview.....................................................................................4
2.2.1.Information theory...............................................................................................4
2.2.2.Coding ...............................................................................................................7
2.2.3.Entropy ..............................................................................................................7
1. Abstract
TO BE DONE AT THE END – BRIEF DESCRIPTION OF WHAT HAS BEEN DONE BY US
IN THE PROJECT.
2. Introduction
Much of current technology is concerned with the transmission and storage of information.
Most of the devices that we use daily either transmit/receive information (telephones, radios,
TV sets), store it (MP3 players, CDs, DVDs, memory sticks) or do both (computers, mobile
phones). This technology has been made possible by great advances in physics and
engineering, but also by the formulation of a mathematical theory of communication by Claude
Shannon in 1948.
The concept of entropy is central to the mathematical description of information. Given a
source that generates a sequence of symbols (e.g. a text, a numerical code, or the sequence of
nucleotides in the human genome), its entropy is a quantitative measure of how much
information it produces per unit time, and of how much memory is needed to store that
information. In most cases it is possible to encode the sequence of symbols in such a way as
to reduce the amount of memory needed, a process known as ``data compression''. The
entropy of the sequence tells us how much it can be compressed without distortion.
In this project we will define the notion of entropy and discuss some of its properties. The
theorems relevant to its applications to coding and data compression will be presented,
motivated with examples and proved. Students will solve a few problems in which the entropy
of sequences of symbols is estimated and the sequences coded and compressed.
The purpose of this section is to give an introduction into the project of “The concept of
Entropy and its applications to information and coding” and will be mainly based on the paper
published by Claude Shannon in the Bell Systems Technical Journal in 1948. This should
give the reader the necessary grounding in the subject and prepare for more advanced
discussions on the matter.
2.1. Historical Remarks:
To start with, the term entropy was coined in 1865 by the German physicist Rudolf Clausius, from the
Greek words en - "in" and trope - "a turning", in analogy with energy. The entropy in the context of
thermodynamics can provide a measure of the amount of energy in a physical system that cannot be
used to do work.
The work of Shannon was inspired by some of the developments in electrical communications at that
time (e.g. Pulse-code or Pulse-position modulation). Abstracting away from the means we use for
communication, its fundamental problem is that of reproducing at one point either exactly or
approximately a message selected at another point. E.g.:
Diagram 1
Thus, the natural question to ask was how to make this information transfer more efficient.
Among the efficiency considerations that motivated Shannon were:
~ What is the most effective way to encode a message given some information on its
statistical structure? Can we make use this structure to make some savings in the length of
the resulting code?
~ What is the effect of noise in the channel on the reproduction of the original message at the
recipient end of communication? Is it possible to minimise such impact?
It's also necessary to say that the Theory of Mathematical Communication came as the first
systematics framework of its kind.
2.2. Topic High-level Overview
The 3 key important aspects of this topic are:
Information theory
Coding
Entropy
2.2.1. Information theory
Information is measured in different units like characters, in the case of describing the length of an
email message, digits in the length of a telephone number, etc. The convention in information theory
is to measure information in bits. A bit is a contraction of the term binary digit which is either a zero or
a one. For example there are 8 possible configurations of 3 bits, i.e. 000, 001, 010, 011, 100, 101,
110 and 111.
Information can be described in a number of forms such as:
A message received and understood
Knowledge acquired through study or experience or instruction
Formal accusation of a crime
Data: a collection of facts from which conclusions may be drawn; statistical data
An important observation made by Shannon with respect to the message being transmitted is that
what the message means semantically is irrelevant to the engineering task of communication. What is
essential though is that the message selected is one from the set of possible messages that the
communication system should be able to transmit. If this set has a finite number of elements, say N,
then this number or some function of this number can be used to determine the information given
when a particular message from the set is selected. It was first R.V.L Hartley who pointed out that the
most natural choice would be a logarithmic function.
In order to illustrate this quite abstract proposition, let’s consider the following example:
Given an alphabet of K = {a, b}, a message is a string made up of symbols from K. Having received
part of the message,
A
A
A
A
A
We naturally, by just observing the pattern of symbols in the partly received text, expect the next
symbol to be an A as well. If it happens to be so, then the information gained by us receiving the next
symbol is very little, since we anticipated the outcome beforehand. Conversely, if the next symbol is a
B, then we gain a lot of information by receiving it as it is very unexpected.
This example highlights the relation between our expectation, or probability in the mathematical
sense, and the amount of information, as well as demonstrates that this relation is ‘inversely
proportional’ (not used in the strict mathematical sense here).
Information
Probability
High
Low
Low
High
Table 1
Trying now to formalise the above example, in order to draw more mathematically precise conclusions
from it, we introduce the following definitions.
Let a source of information output each symbol 𝑆𝑖 from some alphabet K (|𝐾| = 𝑖) with a probability 𝑃𝑖 .
We define the information gained upon reading 𝑆𝑖 as 𝐼(𝑆𝑖 ).
Example:
𝐾 = {𝐴, 𝐵} such that 𝑆1 = 𝐴 and 𝑆2 = 𝐵. We also know that the probability distribution of symbols is
𝑃1 = 1 and 𝑃2 = 0 i.e. we expect the message only to contain symbol 𝐴 and no occurrences of symbol
𝐵.
E.g.: 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴..
From our discussion above, we require that the information gained from reading another symbol 𝐴
from the message is zero, as we are sure that’s the symbol we are going to read. Hence, 𝑃1 = 1 =>
𝐼(𝑆𝑖 ) = 𝑈. Using the same argument, the information gained from encountering symbol B somewhere
in the message should necessarily tend to infinity. Thus, 𝑃𝑖 = 0 => 𝐼(𝑆𝑖 ) = ∞. Consolidating these
conclusions in the form of a diagram, the function relating probability and information gain should look
qualitatively similar to the one below:
I(si)
I(si) = f(Pi)
Pi
1
0
Diagram 2
Also we will assume for simplicity that symbols in the message occur independently, i.e. we will not
allow any conditional probabilities at this stage (this more advanced topic will be discussed in detail in
later sections). In this case, we can expect with reason that the information gained by reading two
symbols, say 𝑆𝑖 and 𝑆𝑗 , will be the same as the information gained from reading symbol 𝑆𝑖 plus that of
𝑆𝑗 . This is a very intuitively anticipated result, as each time we receive an extra symbol from the
message, we add to our already existent knowledge received previously. Hence, information is
incremental in nature and this statement can be formally written as
𝐼(𝑆𝑖 𝑆𝑗 ) = 𝐼(𝑆𝑖 ) + 𝐼(𝑆𝑗 )
*
Now, in the light of the above conclusions and assumptions we have made, we can try to guess what
𝐼(𝑆𝑖 ) = 𝑓(𝑃𝑖 ) is. The simplest example that might spring to one’s mind when seeing the somewhat
‘inversely proportional’ link between the two would be the function of the form
𝐼(𝑆𝑖 ) = 𝑓(𝑃𝑖 ) =
1
𝑃𝑖
However, it’s very easy to see that this is not defined for 𝑃𝑖 = 0 and 𝐼(𝑆𝑖 ) = 0, and it would be
unreasonable to exclude such basic cases from our theory. Also,
𝐼(𝑆𝑖 𝑆𝑗 ) =
1
1 1
≠ + = 𝐼(𝑆𝑖 ) + 𝐼(𝑆𝑗 )
𝑃𝑖 ∗ 𝑃𝑗 𝑃𝑖 𝑃𝑗
Remark: We have interpreted the probability of 𝑆𝑖 and 𝑆𝑗 appearing one after another in the message
as𝑃𝑖 ∗ 𝑃𝑗 , based on our assumption that these are independent events. Otherwise, by Bayes’ theorem
𝑃𝑖∗𝑗 = 𝑃𝑖 ∗ 𝑃𝑗|𝑖 , where 𝑃𝑗|𝑖 is the conditional probability of 𝑆𝑗 given 𝑆𝑖 . (The property in question would
still apply in this case as well)
Next we try out Hartley’s suggestion to use the logarithmic function, in order to quantise information
and see that logically it does make a lot of sense. Suppose, 𝐼(𝑆𝑖 ) = 𝑓(𝑃𝑖 ) = log(𝑃𝑖 ). Then, using the
properties of the logarithmic function,
𝐼(𝑆𝑖 𝑆𝑗 ) = log(𝑃𝑖 𝑃𝑗 ) = log(𝑃𝑖 ) + log(𝑃𝑗 ) = 𝐼(𝑆𝐼 ) + 𝐼(𝑆𝑗 )
Which matches property (*). It’s also qualitatively equivalent to the function we depicted in Diagram 1
earlier, apart from the fact that 𝑃𝑖 and 𝐼(𝑆𝑖 ) should be ‘inversely’ related, with 𝑆𝑖 < 𝑆𝑗 → 𝐼(𝑆𝑖 ) > 𝐼(𝑆𝑗 ),
while the logarithmic function is clearly increasing:
Diagram 3
In order to make our 𝑓(𝑃𝑖 ) non-increasing / decreasing, we simple reflect it about the x-axis by
introducing the minus sign, and 𝑓(𝑃𝑖 ) becomes:
(𝑆𝑖 ) = 𝑓(𝑃𝑖 ) = −log(𝑃𝑖 )
There is some ambiguity with information theory as there are many different types of logarithms. A
particular one has to be chosen to see how many units of information are received. 𝑙𝑜𝑔2 is called a bit,
𝑙𝑜𝑔10 is called a Hartley, etc.
Example:
Another example is tossing a coin. The 2 possibilities when tossing a coin are heads(𝐻) and tails(𝑇).
1
1
1
In a fair tossing experiment, we know the probabilities will be i.e. 𝑃(𝐻) = and 𝑃(𝑇) = . So if we
2
2
2
toss the coin once and get heads, we would receive the following information:
1
𝐼(𝐻) = −𝑙𝑜𝑔2 = 1
2
Remark: We have interpreted the probability of a heads occurring when a coin is tossed once. hence
we can see that one toss gives us 1 bit of information.
The information in the whole message can be calculated using the formula 𝐼(𝐿) = − ∑𝑘𝑖=1 𝑛𝑖 𝑙𝑜𝑔𝑃𝑖
where 𝑘 is the number of characters in the alphabet, 𝐿 is the length of string of characters, and 𝑛𝑖 is
the frequency of character𝑖. 𝑛𝑖 depends on L i.e. longer L is, larger 𝑛𝑖 is. Information density is the
𝐼(𝐿)
information per character =
.
𝐿
2.2.2. Coding
The importance of coding and its relation to information theory arises from the need to transmit
information over some medium. And it is very unlikely that the source of information would produce
the message in exactly the form that is suitable for the medium to transmit. That is it's not generally
the case that the source feeding the channel and the channel itself have the same alphabets. Hence,
the message is first encoded into a signal by a transmitter and then the inverse operation is done by
the receiver, in order to transform the received data into the form accepted by the recipient. Thus,
when coding, it is vital to ensure code invertibility (reversibility).
Coding theory also helps design efficient and reliable data transmission methods so that a suitable
coding with the shortest possible length can be created, with data redundancies omitted and errors
corrected.
The 3 classes of coding are:
1. Source coding – also known as data compression
2. Channel coding – also known as forward error correction
3. Joint source and channel coding
A simple example of coding could be ASCII, which is a character-encoding scheme based on the
ordering of the English alphabet. ASCII codes represent text in computers, communications
equipment, and other devices that use text:
Diagram 4
As can be seen from the code chart above, each ANSII character is encoded as a 7-bit integer (thus,
there are 2^7 = 128 characters in the set). E.g. character 'A' would correspond to 01000001 and 'a' –
to 01100001.
However, ANSII codes is just an illustrative example of a character-encoding scheme used in most
modern digital computers. This scheme does not have any notion of probability distribution or other
statistical information incorporated in it, while we are going to take a slightly different perspective on
code.
At this stage, it is feasible to introduce the term entropy again, but this time in the context of
information theory. Coding will be dealt with in more detail in section XXX
It is a measure of the uncertainty associated with a random variable and quantifies the information
contained in a message.
2.2.3. Entropy
The entropy of a random variable is a measure of randomness and can be shown to be a good
measure of the information contained in a variable. It also tells us the information density of the
particular variable, i.e. the average information per character. Equivalently, entropy quantifies the
average information content one is missing, when one does not know the value of the random
variable. Types of variables could be messages, a story book, the size of a file that needs to be sent
via email etc. The more random it is, the higher the entropy.
Very random  HOT  high entropy
Not random  FROZEN  low entropy
The entropy is the limit when 𝐿 → ∞ for
below:
𝐼(𝐿)
𝐿
. Entropy is denoted S and calculated using the formula
S= lim
𝐼(𝐿)
𝐿→∞ 𝐿
𝑘
𝑛𝑖
= lim (− ∑
𝑛→∞
lim
𝐿→∞
𝑖=1
𝑛𝑖
𝐿
𝐿
log 𝑃𝑖
= 𝑃𝑖
In this context, we will be referring to the Shannon entropy which quantifies the information contained
in a message, usually in units such as bits.
S= − ∑𝑘𝑖=1 𝑃𝑖 𝑙𝑜𝑔𝑃𝑖
Where 𝑃𝑖 is the average information per character and 𝑙𝑜𝑔𝑃𝑖 is the information of character “𝑖”.