Tutorial 5: Information, Nanoscience and Complexity
Question1: [Binary and Decimal Numbering System]
Write the number 2500 in binary.
Read in this direction!
0
Conversion from 0
base 10 to 2
1
0
0
0
1
1
1
0
0
1
2500
1250
625
312
156
78
39
19
9
4
2
1
0
2
2
2
2
2
2
2
2
2
2
2
2
This reads as we have the
divisor 1250 and the
remainder 0 when 2500 is
divided by 2.
Stop until you get 0!
The number 2500 in binary is 100111000100.
Question1: [Continue]
Also, what is the number 1100100 (binary code) in decimal?
Conversion from base 2 to 10
1
26
1
25
0
24
0
1
0
0
23
Right to left!
22
21
20
0 x 20 + 0 x 21 + 1 x 22 + 0 x 23 + 0 x 24 + 1 x 25 + 1 x 26
= 0 x 1 + 0 x 2 + 1 x 4 + 0 x 8 + 0 x 16 + 1 x 32 + 1 x 64
= 0 + 0 + 4 + 0 + 0 + 32 + 64
= 100
Question2: [Number of bits]
How many bits does one need to encode any position in a string of 800
symbols?
The position of the symbols goes like 1, 2, 3, 4, 5, ….. 800. To determine the
number of bits needed to encode any position, we only need to determine the
number of bits required to encode the highest position. The highest position in
this case is 800.
The smallest power of 2 greater than 800 is 210 = 1,024. Hence the minimal
number of bits needed to encode a position in a string of 800 symbols is 10.
Question3: [Shannon entropy and Optimal Coding]
Suppose you are given a five-sided biased die with a probability of 1/8 to
obtain a 1 or 2, and probability of ¼ to obtain a 3, 4, or 5. What is an optimal
code for transmitting the throws of this die?
To solve this problem, we have to first determine the number of bits required
to encode each of the symbols respectively according to the formula –log2P.
Next, based on these, design the coding scheme wisely without ambiguity.
These will be the optimal code to transmit this information.
Average number of bits per throw,
Dice
Probability
Length of code
Example of
code
1
1/8
-log2(1/8) =3
111
2
1/8
-log2(1/8) =3
110
3
1/4
-log2(1/4) =2
10
4
1/4
-log2(1/4) =2
01
5
1/4
-log2(1/4) =2
00
n   pi log 2  pi 
i
1 
1 
 2    3   3    2   2.25
8 
4 
The coding scheme is NOT unique. We can
swap the “0” and “1” globally and this
doesn’t not change the code. Also, which
face belongs to which code of the same
length is arbitrary so we can just choose.
Question4: [Information Theory (past year exam)]
(i) How many bits will I minimally need to transmit the outcomes of 300
throws of an 8 sided die? (2 marks)
There are eight different sides, one could use three bits per throw (since 23=8,
we can count from 1 to 8). So 300 throws would require 900 bits. BUT, this is the
minimal bits needed when the dices and throw are unbiased! (Of course we can
do better if we assume the dices are biased).
Suppose the dice/throw are biased with In this question, we assume the throw are
unbiased since probability is not given.
following frequencies:
Dice
Probability
Length of Code
1
1/2
1
2
1/4
2
3
1/8
3
4
1/16
4
5
1/32
5
6
1/64
6
7
1/128
7
8
1/128
7
n   pi log 2  pi 
i
1
1
1
 1 
 1    2     3     4   
2
4
8
 16 
  1 
 1 
 1 
 5     6     2 7  

128
 32 
 64 



 1.984
N  300 n
 3001.984  595.2  900
Question4: [continue]
(ii) How many bits does one need to encode any position in a string of 60,000
symbols? (2 marks)
The smallest power of 2 greater than 60,000 is 216 = 65,536. Hence the minimal
number of bits needed to encode a position in a string of 60,000 symbols is 16.
(iii) Suppose you are given a four-sided biased die with a probability of 1/8 to
obtain a “1” or a “2”, a probability of 1/4 to obtain a “3” and a probability of
1/2 to obtain a “4”. What is the optimal code for transmitting the throws of this
die? (4 marks)
Dice Probabilit
Length of
y
Code
1
1/8
3
2
1/8
3
3
1/4
2
Based4 on these,
we know that the
1/2
1 optimal
codes for the symbol “1”, “2”, “3” and “4” are
3 bits, 3 bits, 2 bits and 1 bit respectively.
Question4: [continue]
Optimal Coding
To design the optimal coding scheme, we
always start from the simplest one, i.e. the
one requires the minimum number of
bits.
Dic
e
Probabili
ty
Length of
Code
1
1/8
3
2
1/8
3
3
1/4
2
We could choose the following coding scheme: 4
1/2
1
“4” = 0, “3” = 10, “2” = 110, “1” = 111
“4” = 1, “3” = 01, “2” = 001, “1” = 000
Of course, the coding scheme is not
or
unique. We could also choose:
“4” = 1, “3” = 01, “2” = 000, “1” = 001
However, the following scheme is not correct:
“4” = 0, “3” = 11, “2” = 110, “1” = 111
Let’s say we have the string 110. Should be decode it to be the symbol
“2” or “34”?
Therefore, the design of the optimal code must be careful so that no ambiguity
would be caused!
Question5: [Turing Machine]
What is a Turing machine? Design a Turing machine that can write the word
“Life” its tape.
A Turing machine is a very simple machine, but, logically speaking, has all the
power of any digital computer. It may be described as follows:
1.
2.
3.
4.
5.
There is an infinite tape with squares on it.
Each square may contain a symbol from a finite alphabet but there can be
only finitely many non-blank square on the tape.
There is a read-write head which is positioned somewhere on the tape.
The machine is in one of a finite number of states.
There is a set of rules that tell the machine what to do given a certain state
and a certain symbol read from the tape.
Question5: [continue]
The rule is of the following form:
(current state, current symbol, new state, new symbol, left/right)
This rule means that if the Turing machine is now in current state,
and the symbol under the read/write head is current symbol,
change its internal state to new state, replace the symbol on the
tape at its current position by new symbol, and move the
read/write head one square in the given direction (left or right).
Question5: [Turing Machine (continue)]
Start with the symbol S on the tape and in state zero.
Then apply the rules until the machine stops.
S
Head
State
0
> = Right, - = write or do nothing
Rules are defined in this format: (current state, current symbol, new state,
new symbol, left/right)
Question5: [Turing Machine (continue)]
Let us go through this algorithm step by step
In the beginning (this is also called the initial condition) we have:
Apply rule 1.
Step 0 - Read
1) 0 , S , 1 , L , R
S
Head
Rules:
2) 1, - , 2 , I , R
State
3) 2, - , 3 , F, R
0
4) 3, - , 4 , E ,R
5) 4, -, - , -, -
R = Right, - = write or do nothing
Rules are defined in this format: (current state, current symbol, new state, new
symbol, left/right)
Question5: [Turing Machine (continue)]
Step 1 - Move
Right
L
Head
Rules:
State
1
1) 0 , S , 1 , L , R
2) 1, - , 2 , I , R
3) 2, - , 3 , F, R
4) 3, - , 4 , E ,R
R = Right, - = write or do nothing
5) 4, -, - , -, -
Rules are defined in this format: (current state, current symbol, new state,
new symbol, left/right)
Question5: [Turing Machine (continue)]
Apply rule 2
Step 2 - Read
L
Head
Rules:
State
1) 0 , S , 1 , L , R
1
2) 1, - , 2 , I , R
3) 2, - , 3 , F, R
4) 3, - , 4 , E ,R
5) 4, -, - , -, -
> = Right, - = write or do nothing
Rules are defined in this format: (current state, current symbol, new state, new
symbol, left/right)
Question5: [Turing Machine (continue)]
Step 3 - Move
Right
L I
Head
Rules:
State
1) 0 , S , 1 , L , R
2
2) 1, - , 2 , I , R
3) 2, - , 3 , F, R
4) 3, - , 4 , E ,R
5) 4, -, - , -, -
> = Right, - = write or do nothing
Rules are defined in this format: (current state, current symbol, new state, new
symbol, left/right)
Question5: [Turing Machine (continue)]
Apply rule 3
Step 4 - Read
L I
Head
Rules:
State
1) 0 , S , 1 , L , R
2
2) 1, - , 2 , I , R
3) 2, - , 3 , F, R
4) 3, - , 4 , E ,R
> = Right, - = write or do nothing
5) 4, -, - , -, -
Rules are defined in this format: (current state, current symbol, new state,
new symbol, left/right)
Question5: [Turing Machine (continue)]
Step 5-Move
Right
L I F
Head
State
3
Rules:
1) 0 , S , 1 , L , R
2) 1, - , 2 , I , R
3) 2, - , 3 , F, R
R = Right, - = write or do nothing
4) 3, - , 4 , E ,R
5) 4, -, - , -, -
Rules are defined in this format: (current state, current symbol, new state,
new symbol, left/right)
Question5: [Turing Machine (continue)]
Step 6 - Read
Apply rule 4
L I F E
Head
State
4
Rules:
1) 0 , S , 1 , L , R
2) 1, - , 2 , I , R
3) 2, - , 3 , F, R
R = Right, - = write or do nothing
4) 3, - , 4 , E ,R
5) 4, -, - , -, -
Rules are defined in this format: (current state, current symbol, new state,
new symbol, left/right)
Question5: [Turing Machine (continue)]
Step 6 - Read
Apply rule 5
L I F E
Head
Rules:
State
4
1) 0 , S , 1 , L , R
2) 1, - , 2 , I , R
3) 2, - , 3 , F, R
R = Right, - = write or do nothing
4) 3, - , 4 , E ,R
5) 4, -, - , -, -
The head will stay there without doing anything at this position after
applying Rule 5.
Rules are defined in this format: (current state, current symbol, new state,
new symbol, left/right)
Question6: [Turing Machine (past year exam)]
(i) Look at the figure: In which direction will the Turing machine above move
next? (1 mark)
Since the current state is ‘1’ and current symbol is ‘2’, we apply rule 4 and the
head moves to left.
(ii) Change the rules such that when starting from S in state 0, the machine will
forever keep on moving to the right. (2 marks)
Rules:
Rules:
Rules:
1) 0, S or All, 0 , - , R (*)
1) 0, S , 0 , - , R (*)
2) 0, 2 , 0 , - , R
2) 0, 2 , 0 , - , R
2) 0, 0 or 1 or 2 , 0 , - , R (*)
3) 2, 1 , 1 , 0 , R
3) 2, 1 , 1 , 0 , R
3) 2, 1 , 1 , 0 , R
4) 1, 1 or 2 , 0 , - , R (*)
4) 1, 1 or 2 , 0 , - , L
4) 1, 1 or 2 , 0 , - , L
Some
1) 0, S or 0 or 1 , 1 , - , R (*)
possibilities
Question6: [Continue]
(iii)What is the minimum number of rules that we need to implement (ii)?
(1 mark) Rules:
Rules:
1) 0, S or All, 0 , - , R (*)
1) 0, S or 0 or 1 , 1 , - , R
2) 0, 2 , 0 , - , R
2) 0, 2 , 0 , - , R
3) 2, 1 , 1 , 0 , R
3) 2, 1 , 1 , 0 , R
4) 1, 1 or 2 , 0 , - , L
4) 1, 1 or 2 , 0 , - , R
non-trivial cases, we
need 2 rules
We need 1 rules minimally, namely modify rule 1. But, this is absolutely
trivial case! (acceptable generally speaking).
(iv) Design a Turing machine that can write the word ‘NUS’ and give its rules? (4
marks)
N U S
S
Head
State
0
Rules:
1) 0, S , 1 , N , R
2) 1, - , 2 , U , R
3) 2, - , 3 , S , R
4) 3, - , - , - , -
Head
State
3
Question11: [Numbering Scheme in Cellular Automata]
Consider the elementary cellular automata with only next nearest neighbour (i.e. figure
20.24 in text book), discuss the numbering scheme in cellular automata.
7
6
5
4
The first thing to
notice is that the
top is always the
same.
Value 1
If we say that black is one and grey is zero,
then we can see that the TOP is just counting
from 0 to 7.
=4
Value 1
0
Value 2
1
Value 2
2
Value 4
3
Value 4
This is the part
that changes.
=3
Question11: [Continue]
How about the bottom? We can do exactly the same thing but
since we have 8 boxes on the bottom it’s counting from 0 to 255.
Value 128 Value 64 Value 32
Value 8
Value 4
Value 2
“Rule 255”
Value 16
Value 1
“Rule 0”
64+16+8+2= “Rule 90”
Question11: [Continue]
Now, it would be easier to determine the numbering scheme if we
are given the rules.
What is the numbering scheme for the following rule?
Value 128
Value 64
Value 32
Value 16
numbering scheme is equivalent to
the “rule”. (but normally can be
given in either binary or decimal).
(see next slide)
2+16+64+128 = “Rule 210”
Value 8
Value 4
Value 2
Value 1
http://cell-auto.com/wolframnumbers/
Question11: [Continue]
How to design the rule from the given numbering scheme (say in decimal)?
First, we convert the
numbering scheme (i.e. 210)
into its binary.
Add the binary number ‘0’
from the leftmost till the
number of bits is 8!
Color the bottom boxes according to the binary numbers.
‘0’  grey
‘1’  black
Question11: [Continue]
Example 1: Design the rule for the numbering scheme 100.
0 1 1 0 0 1 0 0
the extra ‘0’ bit
0
1
1
0
0
100
2
0
50
2
1
25
2
0
12
2
0
6
2
1
3
2
1
1
2
0
0
1
0
0
Question11: [Continue]
Example 2: Design the rule for the numbering scheme
210.
0
1 1 0 1 0 0 1 0
1
0
1
0
0
1
1
0
210
2
1
105
2
0
52
2
0
26
2
1
13
2
0
6
2
1
1
3
1
2
2
0
Exercise: Can you work out the elementary automata with second
nearest neighbor with 2 possible states?
Hint: There are 25=32 of them, allow us to count from 0-31. How
about the lower box? We have (20=1, 21=2,……. 231=2,147,483,648)
Question13: [Cellular Automata (past year exam)]
We have the following cellular automaton rule:
(i) What rule is this? (i.e. give the number such as “rule 90”).
value 128
value 64
value 32 value 16 value 8
value 4
value 2
1128  1 4  132
Answer: “Rule 132”
(ii) Draw the next four steps in figure 1 using the rule from step (i).
Rule 132
value 1
Question13: [Continue]
(iii)What rule was used to make figure 2?
value 128
value 8
value 64 value 32 value 16
value 4
value 2
value 1
1 64 1 32 1 8 1 4 1 2  110 So, we have Rule 110.
(iv)What is notable about the rule used in figure 2?
This is an universal Turning machine and thus can act perform universal
computation. (the proof is difficult!)
(v) If, for the cellular automaton considered here, we allow the cells
to have 3 states instead of 2 states, how many rules will there be?
(5 marks in total)
For 3 states and nearest neighbors case,
N rules
 33   7.6255 1012
3