Theory of Computations III
CS-6800 |SPRING -2014
Markov Models
By
Sandeep Ravikanti
Contents
 Introduction
 Definition
 Examples
 Types
of Markov Models
 Applications
 References
Introduction

Origin
Mid 20th cent.: named after Andrei A. Markov (1856–
1922), Russian mathematician.

Markov model
It’s a stochastic model describing a sequence of
possible events in which the probability of each event
depends only on the state attained in the previous
event
Formal Definition

Markov model is a NDFSM.

Each step of state is predicted through probability distribution with the current
state.

Steps here usually corresponds as time intervals, they can also correspond to
any ordered discrete sequence.

Formally a Markov Model is said to be triple

M = (K, 𝜋, A)
K
is finite set of States
𝜋
is a vector that contains the intial probabilities of each of the states.
A
is Matrix that represents transitions probabilities

A[ p , q] = Pr (state q at time t | state p at time t -1)
Formal Definition Cont.…..
Examples of Markov models
0.75

Simple Markov Model for weather

Influence of weather on day T is
determined by weather on day T-1

The transition matrix for weather
model

M = 0.75 0.25
𝜋 = 0.4
Sunny
0.25
0.3
0.7
𝜋 = 0.6

Rainy
0.7
0.3
Transition matrix rows sum up to 1.0
since its transition matrix
Markov Model For Weather



Probability of being Sunny for five days in a row.

Sunny , Sunny , Sunny , Sunny , Sunny

0.4 * (0.75)4 = 0.1266
Probability of being Rainy for five days in a row.

Rainy , Rainy , Rainy, Rainy, Rainy

0.6 * (0.7)4 = 0.2401
Given its sunny today ,probability of staying sunny for four more days

(0.75)4 = 0.316
Modelling of Markov Models

Link Structure of World Wide Web.

Performances of a System.

Population Genetics.
Types of Markov Models
Cont.….

Markov chain
•
Defined as sequences of outputs produced by Markov model.
•
Simplest Markov model.
•
System state at time t+1 depends on state at t.
•
Markov chain that describes some random process helps :

In answering probability of sequences of states S1,S2,S3,………Sn.

To compute probability by multiplying probabilities of each transition using probability of S1.

Pr (s1 s2. . . Sn) = 𝜋

Knowing the result for an arbitrarily long sequences of steps
𝑠1 . 𝜋ni=2 A[si-1,si ].
Cont.….
Markov Chain for predicting the weather

The matrix M represents the weather model.

M=

Sunny
0.3 0.7
is a transition matrix.
0.6 0.4
Weather on day “0” is known to be sunny.

Vector representation in which its sunny.

X (0) = [1 0]

Weather on day “1” can be predicted by:

0.3 0.7
= [0.3 0.7]
0.6 0.4
Weather on day “2” can be predicted by:

X (1) = X (0) M = [1 0]
X (2) = X (1) M = X (0) M2 = [1 0]
Rainy

0.3 0.7
0.6 0.4
General rules for day “n”

X (n) = X (n-1) M = = X (n) = X (0) Mn
2=
[0.51 0.49].
Properties of Markov chains

Reducibility

Accessible

Essential or final

Irreducible

Periodicity

Recurrence


Transient

Recurrent
Ergodicity

Hidden Markov model

Defined as non deterministic finite state transducer.

States of the system are not directly observable

Derived from two key properties.

They are Markov models. Their state at time t is a functions solely of their time at
t-1.

Actual Progression of machine is hidden. Only output string is observed.


Markov Decision Process
•
It is a Markov chain in which state transitions depend on the current state and
an action vector that is applied to the system.
•
Used to compute a policy of actions that will maximize some utility
•
Can be solved with value iteration and related methods.
Partially observable Markov decision process
•
It is a Markov decision process in which the state of the system is only partially
observed.
•
Pomdps are known to be NP complete.
•
Recent approximation techniques have made them useful for A variety of
applications, such as controlling simple agents or robots
Applications

PHYSICS

CHEMISTRY

TESTING

INFORMATION SCIENCES

QUEUING THEORY

INTERNET APPLICATIONS

STATISTICS

ECONOMICS AND FINANCE

SOCIAL SCIENCE

MATHEMATICAL BIOLOGY

GAMES

MUSIC
References

Elaine A Rich, Automata, Computability And Complexity, Theory And Application
.1st Edition

Wikipedia-Markov Model," [Online]. Available:
Http://En.Wikipedia.Org/Wiki/Markov_chain. [Accessed 28 10 2012].

P. Xinhui Zhang, "Dtmarkovchains," [Online]. Available:
Http://Www.Wright.Edu/~Xinhui.Zhang/. [Accessed 28 10 2012].

http://www.dartmouth.edu/~chance/teaching_aids/books_articles/probability_b
ook/chapter11.pdf
Questions…..?