1. Steady state Distribution of a Markov Chain (with Markov Chain matrix properties)
1. A salesman’s territory consists of three cities A, B and C. He never sells in the same city on
successive days. If he sells in A, then the next day he sells in city B. However, if he sells in either B or
C, then the next day he is twice as likely to sell in city A as in the other city. In the long run, how often
does he sell in each of the cities?
A = [0, 1, 0; 2/3, 0, 1/3; 2/3, 1/3, 0];
A = A';
B = A^50
B = 3×3
0.4000
0.4500
0.1500
0.4000
0.4500
0.1500
0.4000
0.4500
0.1500
disp('In the long run, how often does he sell in each of the cities..')
In the long run, how often does he sell in each of the cities..
disp(B(:, 1));
0.4000
0.4500
0.1500
2.
A = [0, 1; 1/2, 1/2];
A = A';
B = A^50;
disp('Steady state distribution of chain')
disp(B(:, 1))
3. The three- state Markov chain is given by the transition probability matrix P= [ / /; / /; / / ]. Find the
steady state distribution of the chain
A = [0, 2/3, 1/3; 1/2, 0, 1/2; 1/2, 1/2, 0];
A = A';
B = A^50;
disp('Steady state distribution of chain')
disp(B(:, 1))
2. Generation of random state sequences
1
1. Three Boys A, B, C are throwing a ball each other. A always throws the ball to B and B always throws
the ball to C but C is just as likely to throw the ball to B as to A. Show the process is Markovian. Find
the transition probability matrix and generate 700 the state sequence. Also reestimate the parameters.
clc;clear all;
disp(['The person receiving the ball depends ONLY on the person ' ...
'holding the ball in the previous state or previously'])
The person receiving the ball depends ONLY on the person holding the ball in the previous state or previously
disp('Hence, Process is a markovian')
Hence, Process is a markovian
Find the transition probability matrix
T=zeros(3,3);
T(1,1)=0;
T(1,2)=1;
T(2,1)=0;
T(2,2)=0;
T(3,1)=0.5; T(3,2)=0.5;
T(1,3)=0;
T(2,3)=1;
T(3,3)=0;
disp('Transition probability matrix')
Transition probability matrix
T
T = 3×3
0
0
0.5000
1.0000
0
0.5000
0
1.0000
0
disp('All Rows have a sum equal to 1')
All Rows have a sum equal to 1
disp('It is a stochastic matrix')
It is a stochastic matrix
generate 700 the state sequence
s(1)=randi(3,1); % randomly choose a state
n=700;
for i = 1:n-1
cs = s(i);
r = rand();
if r<T(cs, 1)
s(i+1)=1;
elseif r<T(cs,1)+T(cs,2)
s(i+1)=2;
else
s(i+1)=3;
end
2
end
disp('Part of state sequence')
Part of state sequence
display(s(1:12))
Columns 1 through 4
3
2
3
2
Columns 5 through 8
3
1
2
3
Columns 9 through 12
2
3
1
2
Reestimate the parameters
% Counting transition
A=zeros(3,3);
n=length(s);
for i=1:n-1
k=s(i);
l=s(i+1);
A(k,l)=A(k,l)+1;
end
disp('Estimated transition probabilities')
Estimated transition probabilities
rowsum=sum(A,2);
div=repmat(rowsum,[1,3]);
A=A./div;
disp(A)
Columns 1 through 2
0
0
0.4823
1.0000
0
0.5177
Column 3
0
1.0000
0
disp('Original transition probabilities')
Original transition probabilities
disp(T)
Columns 1 through 2
3
0
0
0.5000
1.0000
0
0.5000
Column 3
0
1.0000
0
2. Consider the random experiment of rolling a die once. Generate 800 random state sequences for the
appearance on the die and reestimate the parameters. (you can determine the Markov chain on your
own).
clear all; clc;
% Transition probability matrix or Markov Chain
T = [3/15, 4/15, 2/15, 2/15, 1/15, 3/15;
2/15, 1/15, 3/15, 4/15, 3/15, 2/15;
2/15, 2/15, 3/15, 3/15, 3/15, 2/15;
3/15, 4/15, 2/15, 1/15, 1/15, 4/15;
1/15, 3/15, 2/15, 1/15, 4/15, 4/15;
3/15, 3/15, 5/15, 2/15, 1/15, 2/15];
Generate 800 random state sequences for the appearance on the die
s(1)=randi(6,1); % randomly choose a state
n=800;
for i = 1:n-1
cs = s(i); % current state
r = rand();
if r<T(cs, 1)
s(i+1)=1;
elseif r<T(cs,1)+T(cs,2)
s(i+1)=2;
elseif r<T(cs,1)+T(cs,2)+T(cs,3)
s(i+1)=3;
elseif r<T(cs,1)+T(cs,2)+T(cs,3)+T(cs,4)
s(i+1)=4;
elseif r<T(cs,1)+T(cs,2)+T(cs,3)+T(cs,4)+T(cs,5)
s(i+1)=5;
else
s(i+1)=6;
end
end
disp('Part of state sequence')
display(s(1:12))
Reestimate the parameters
% Counting transition
A=zeros(6,6);
n=length(s);
for i=1:n-1
4
k=s(i);
l=s(i+1);
A(k,l)=A(k,l)+1;
end
disp('Estimated transition probabilities')
rowsum=sum(A,2);
div=repmat(rowsum,[1,6]);
A=A./div;
disp(A)
disp('Original transition probabilities')
disp(T)
3. In the case of tossing a coin once, generate 2000 random state sequences for the appearance on
the die. Display any subset of the sequence with length 30 and reestimate its parameters. (you can
determine the Markov chain on your own).
% Transition probability matrix or Markov Chain
T = [0.3, 0.7;
0.6, 0.4];
generate 2000 random state sequences for the appearance on the die. Display any subset of the sequence with
length 30
s(1)=randi(2,1); % randomly choose a state
n=2000;
for i = 1:n-1
cs = s(i);
r = rand();
if r<T(cs, 1)
s(i+1)=1;
else
s(i+1)=2;
end
end
disp('Part of state sequence')
r = randi(1900, 1)
display(s(r+1:r+30))
Reestimate its parameters
% Counting transition
A=zeros(2,2);
n=length(s);
for i=1:n-1
k=s(i);
l=s(i+1);
A(k,l)=A(k,l)+1;
end
disp('Estimated transition probabilities')
5
rowsum=sum(A,2);
div=repmat(rowsum,[1,2]);
A=A./div;
disp(A)
disp('Original transition probabilities')
disp(T)
3. Generate Symbol and State sequences
1. Generate observation sequence and state sequence of the climate problem which we have discussed
in the class. Also re estimate the parameters.
%HMM data Generation
clear all;
T=[0.5, 0.3, 0.2; 0.4, 0.2, 0.4; 0, 0.3, 0.7]
T = 3×3
0.5000
0.4000
0
0.3000
0.2000
0.3000
0.2000
0.4000
0.7000
state_rainy.sad=0.9;
state_rainy.happy = 0.1;
state_cloudy.sad = 0.6;
state_cloudy.happy = 0.4;
state_sunny.sad = 0.2;
state_sunny.happy = 0.8;
x=rand();
if x<1/3
start_state=1;
elseif x<2/3
start_state=2;
else
start_state=3;
end
current_state=start_state;
num_of_observations = 200;
% s = zeros([1, num_of_samples])
s(1)=current_state;
Obs='';
for i=1:num_of_observations
x=rand();
if current_state==1
if x<state_rainy.sad
Obs(i)='S';
else
Obs(i)='H';
end
elseif current_state == 2
if x<state_cloudy.sad
6
else
end
Obs
Obs(i)='S';
else
Obs(i)='H';
end
if x<state_sunny.sad
Obs(i)='S';
else
Obs(i)='H';
end
end
x=rand();
if x<T(current_state,1)
current_state=1;
elseif x<T(current_state,1)+T(current_state,2)
current_state=2;
else
current_state=3;
end
s(end+1)=current_state;
Obs =
'HHSSSSSHSSSHHHSHSHSSSHHHSHHHHSHHSSSSSHHSSHHHHHSSSSSSSHSSSSSSHHSSSHHSSSSHSHHHSHHSSHHHSHSHHSSHSSHHHHSHSHHSHSSSHHHHHSH
s=s(1:end-1);
% s=s(1:end-1);
state_rainy_computed.sad=0;
state_rainy_computed.happy=0;
state_cloudy_computed.sad=0;
state_cloudy_computed.happy=0;
state_sunny_computed.sad=0;
state_sunny_computed.happy=0;
for i=1:length(s)
e=Obs(i);
if and(s(i)==1, e=='S')
state_rainy_computed.sad = state_rainy_computed.sad+1;
elseif and(s(i)==1, e=='H')
state_rainy_computed.happy = state_rainy_computed.happy+1;
elseif and(s(i)==2, e=='S')
state_cloudy_computed.sad = state_cloudy_computed.sad+1;
elseif and(s(i)==2, e=='H')
state_cloudy_computed.happy = state_cloudy_computed.happy+1;
elseif and(s(i)==3, e=='S')
state_sunny_computed.sad = state_sunny_computed.sad+1;
else
state_sunny_computed.happy = state_sunny_computed.happy+1;
end
end
total=state_rainy_computed.sad+state_rainy_computed.happy+...
state_cloudy_computed.sad+state_cloudy_computed.happy+...
state_sunny_computed.sad+state_sunny_computed.happy;
7
state_rainy_computed_sad = state_rainy_computed.sad/(state_rainy_computed.happy+state_rainy_com
state_rainy_computed_sad = 0.8333
state_rainy_computed_happy = state_rainy_computed.happy/(state_rainy_computed.happy+state_rainy
state_rainy_computed_happy = 0.1667
state_cloudy_computed_sad = state_cloudy_computed.sad/(state_cloudy_computed.happy+state_cloudy
state_cloudy_computed_sad = 0.6591
state_cloudy_computed_happy = state_cloudy_computed.happy/(state_cloudy_computed.happy+state_cl
state_cloudy_computed_happy = 0.3409
state_sunny_computed_sad = state_sunny_computed.sad/(state_sunny_computed.happy+state_sunny_com
state_sunny_computed_sad = 0.1944
state_sunny_computed_happy = state_sunny_computed.happy/(state_sunny_computed.happy+state_sunny
state_sunny_computed_happy = 0.8056
%% Reestimation of the parameters
% Counting transition
A=zeros(3,3);
n=length(s);
for i=1:n-1
k=s(i);
l=s(i+1);
A(k,l)=A(k,l)+1;
end
% Estimated transition probabilities
rowsum=sum(A,2);
div=repmat(rowsum,[1,3]);
A=A./div;
disp('Estimated transition probabilities')
Estimated transition probabilities
disp(A)
Columns 1 through 2
0.5417
0.5000
0
0.2708
0.2045
0.2056
Column 3
0.1875
0.2955
0.7944
8
2. Generate observation sequence and state sequence of the climate problem (which I already sent you
in teams). Also re estimate the parameters.
3. Generate 5 observation of the model which we have discussed today’s class.
clear all; clc;
%HMM data Generation
T=[0.7, 0.3; 0.2, 0.8];
state1.head
state1.tail
state2.head
state2.tail
=
=
=
=
0.8;
0.2;
0.3;
0.7;
x=rand();
if x<0.5
start_state = 1;
else
start_state = 2;
end
current_state = start_state;
num_of_observations = 5;
% s = zeros([1, num_of_samples])
s(1)=current_state;
Obs='';
for i=1:num_of_observations
x=rand();
if current_state==1
if x < state1.head
Obs(i)='H';
else
Obs(i)='T';
end
else
if x < state2.head
Obs(i)='H';
else
Obs(i)='T';
end
end
x=rand();
if x<T(current_state,1)
current_state=1;
9
else
end
Obs
current_state=2;
end
s(end+1)=current_state;
Obs =
'HTTTT'
s=s(1:end-1)
s = 1×5
2
2
2
2
2
4. Generate 6 observation of the model which we have discussed today’s class. Also re estimate the
parameters.
clear all; clc;
%HMM data Generation
T=[0.7, 0.3; 0.2, 0.8]
T = 2×2
0.7000
0.2000
0.3000
0.8000
state1.head = 0.8;
state1.tail = 0.2
state1 = struct with fields:
head: 0.8000
tail: 0.2000
state2.head = 0.3;
state2.tail = 0.7
state2 = struct with fields:
head: 0.3000
tail: 0.7000
x=rand();
if x<0.5
start_state = 1;
else
start_state = 2;
end
current_state = start_state;
num_of_observations = 6;
% s = zeros([1, num_of_samples])
s(1)=current_state;
Obs='';
for i=1:num_of_observations
10
end
Obs
x=rand();
if current_state==1
if x < state1.head
Obs(i)='H';
else
Obs(i)='T';
end
else
if x < state2.head
Obs(i)='H';
else
Obs(i)='T';
end
end
x=rand();
if x<T(current_state,1)
current_state=1;
else
current_state=2;
end
s(end+1)=current_state;
Obs =
'HHTHTT'
s=s(1:end-1);
% s=s(1:end-1);
state1_computed.head
state1_computed.tail
state2_computed.head
state2_computed.tail
=
=
=
=
0;
0;
0;
0;
for i=1:length(s)
e=Obs(i);
if and(s(i)==1, e=='H')
state1_computed.head = state1_computed.head+1;
elseif and(s(i)==1, e=='T')
state1_computed.tail = state1_computed.tail+1;
elseif and(s(i)==2, e=='H')
state2_computed.head = state2_computed.head+1;
else
state2_computed.tail = state2_computed.tail+1;
end
end
total=state1_computed.head+state1_computed.tail+...
state2_computed.head+state2_computed.tail;
state1_computed_head = state1_computed.head/(state1_computed.head+state1_computed.tail)
state1_computed_head = 0.7500
state1_computed_tail = state1_computed.tail/(state1_computed.head+state1_computed.tail)
11
state1_computed_tail = 0.2500
state2_computed_head = state2_computed.head/(state2_computed.head+state2_computed.head)
state2_computed_head = NaN
state2_computed_tail = state2_computed.tail/(state2_computed.head+state2_computed.tail)
state2_computed_tail = 1
%% Reestimation of the parameters
% Counting transition
A=zeros(2,2);
n=length(s);
for i=1:n-1
k=s(i);
l=s(i+1);
A(k,l)=A(k,l)+1;
end
% Estimated transition probabilities
rowsum=sum(A,2);
div=repmat(rowsum,[1,2]);
A=A./div;
disp('Estimated transition probabilities')
Estimated transition probabilities
disp(A)
0.7500
0
0.2500
1.0000
5. Generate all possible state sequence and display nth row of it.(n is the last two digit of your
registration number)
n=64;
Seq=[];
for i=0:n-1
x=dec2base(i,2,6);
x= strrep( x , '0' , '2' );
Seq=[Seq;x];
end
Seq(45, :)
ans =
'121122'
4) Finding the probability of Symbol sequence, State sequence
1. Generate transition probability matrix of brand(soap) switching problem and find its joint probability.
Also determine P(sym_sequence,state_sequence).
12
disp('Transition Probability Matrix')
Transition Probability Matrix
T = [0.8, 0.1, 0.1;
0.2, 0.6, 0.2;
0.1, 0.2, 0.7]
T = 3×3
0.8000
0.2000
0.1000
0.1000
0.6000
0.2000
0.1000
0.2000
0.7000
% Find P(sym_sequence,state_sequence)
% It is joint probability
% Let us take T's elements as [ T11 T12 ; T21 T22]
% actual transition prob is transpose of T
T=[0.5 0.5; 0.4 0.6];
state1.h=0.7;state1.t=0.3;state2.h=0.6;
state2.t=0.4;
x='HTTTH';
n=length(x); % length of symbol sequence
N=2^n;
Seq=[];
for i=0:N-1
x=dec2base(i,2,5);
x= strrep( x , '0' , '2' );
Seq=[Seq;x];
end
Seq;
% Get sym_sequence probabiity
prodx=1;
stat_seq=Seq(10,:); % take 10th row
for i=1:length(x)
if stat_seq(i)==1
if x(i)=='H'
prodx=prodx*state1.h;
else
prodx=prodx*state1.t;
end
else
if x(i)=='H'
prodx=prodx*state2.h;
else
prodx=prodx*state2.t;
end
end
end
% Get trans_sequence probabiity
prods=1;
for i=1:n-1
13
j=str2num(stat_seq(i));
k=str2num(stat_seq(i+1));
prods=prods*T(j,k);
end
reqd_prob=vpa(prodx*prods)
reqd_prob =
2. P(x = HTTTH,
= 12121)
3. Generate x with length 128 and Find P(X = row corresponds to your last two digit of your reg.
number | Pi = )
14