Markov Analysis
Jørn Vatn
NTNU
1
Introduction
 Markov analysis is used to model systems which have
many different states
 These states range from “perfect function” to a total fault
state
 The migration between the different states may often be
described by a so-called Markov-model
 The possible transitions between the states may further be
described by a Markov diagram
2
Purpose
 Markov analysis is well suited for deciding reliability
characteristics of a system
 Especially the method is well suited for small systems with
complicated maintenance strategies
 In a Markov analysis the following topics will be of interest
 Estimating the average time the system is in each state. These
numbers might further form a basis for economic considerations.
 Estimating how frequent the system in average “visits” the various
states. This information might further be used to estimate the need
for spare parts, and maintenance personnel.
 Estimate the mean time until the system enters one specific state,
for example a critical state.
3
Markov Analysis procedure
1.
2.
3.
4.
5.
Make a sketch of the system
Define the system states
Draw the Markov diagram with the transition rates
Quantitative assessment
Compilation and presentation of the result from the
analysis
4
Make a sketch of the system
 Pump system wit active pump and a spare pump in
standby
Active
pump
Standby
pump
5
Definition of system states
 x1 = state of active pump
 x2 = state of standby pump
1 if component i is functioning
xi  
0 if component i is in a fault state
System state
xS
Component state Comments
x1
x2
2
1
1
Both pumps functioning
1
0
1
The active pump is in a fault state, the
standby pump is functioning
0
0
0
Both pumps in a fault state
6
State transitions



1 =
2 =
1 =
B =
For this system we have assumed that if the active pump fails, the
standby pump could always be started
Further we assume that if both pumps have failed, they will both be
repaired before the system is put into service again
The following transition rates are defined
failure rate of the active pump
failure rate of the standby pump (while running, 2 = 0 in standby
position)
repair rate of the active pump (1/1 = Mean Down Time when the
active pump has failed)
repair rate when both pumps are in a fault state. I.e. we assume
that if the active pump has failed, and a repair with repair rate 1 is
started, one will ”start over again” with repair rate B, if the standby
pump also fails, independent of “how much” have been repaired on
the active pump.
7
Markov state space diagram
 The circles represent the system states, and the arrows
represent the transition rates between the different system
states
 The Markov diagram and the description of states
represent the total qualitative description of the system
1
2
1
2
1
0
B
8
Quantitative assessment
 We want to assess the following quantities
 Average time the system remain in the various
system states
 The visiting frequencies to each system state
1
2
9
1
2
1
0
B
 a00
a
Transition matrix
 10
A

 The indexing starts on 0, and moves

to r, e.g. there are r +1 system states
 ar 0
 Each cell in the matrix has two indexes,
a01
a11
aij
a r1
where the first (row index) represent the
”from” state, whereas the second
(column index) represent the “to” state.
 The cells represent transition rates from one state to
another
 aij is thus the transition rate from state i to state j
 The diagonal elements are a kind of ”dummy”-elements,
which are filled in at the end, and shall fulfil the condition
that all cells in a row adds up to zero
10
a0 r 
a1r 




arr 
Example transition matrix: (From , To )
0
0 B
A

1  2
2  0
1
2
0
B 

1 
1 
2  1
1
1
2
11
1
2
1
0
B
State probabilities
 Let Pi(t) represent the probability that the system is in
state i at time t
 Now introduce vector notation, i.e.
 P(t) = [P0(t), P1(t),…,Pr(t)]
 From the definition of the matrix diagram it might be
shown that the Markov state equations are given by:
P(t) A = d P(t)/d t
 These equations may be used to establish both the steady
state probabilities, and the time dependent solution
12
Steady state probabilities
 Let the vector P = [P0, P1,…,Pr] represent the average
time the system is in the various system states in the long
time run
 For example, P0 is average fraction of the time the system
is in state 0, P1 is average fraction of the time the system
is in state 1
 The elements P = [P0, P1,…,Pr] are also denoted steady
state probabilities to indicate that in the stationary situation
Pi represents the probability that the system is in state i.
13
The steady state solution
 In the long run when the system has stabilized we must
have that d P(t)/d t = 0, hence
 PA = 0
 This system of equations is over-determined, hence we
may delete one column, and replace it with the fact that
P0+ P1+…+Pr = 1
 Hence, we have
14
The steady state solution
P A1 = b
where
 a00
a
A1   10


 ar 0
a01
a11
a r1
1
1



1
and
b = [0,0, …,0,1]
15
Example
 P0
P1
  B
P2   2

 0
0
2  1
1
1
1   0 0 1

1
which gives
12
P0 
(2   B )1  (2  1 )  B
 B 1
P1 
( 2   B )1  ( 2  1 ) B
 B ( 2  1 )
P2 
( 2   B )1  ( 2  1 ) B
16
Numerical solution
 To solve the steady state equations P A1 = b is a tedious
task
 Often we therefore solve these equations by numerical
methods
 The Markov.xls program does this, where we have to:
 Define the transition rates
 Assign numerical values to the transition rates
 Specify the Markov state space matrix
17
Program for simple Markov analysis
Transition matrix
Parameter Value
Dim
Init
SystFail
3
2
0
1
1.00E-03
2
1
B
5.00E-03
Steady state pr.
P0
0.000915
P1
0.007627
P2
0.991458
Visit frequencies
v0
3.81317E-05
v1
0.000991458
v2
0.000991458
From 0
From 1
From 2
To 0
To 1
To 2
-0.04167
0 0.041667
0.005
-0.13
0.125
0
0.001
-0.001
-0
0.125
0.0416667
0
MTTFS 26200.02
Parameter Numeric values
names
of the parameters
(Give the cells
names)
1
2
1
2
1
0
B
0
A
1
2
18
  B
 
 2
 0
1
0
2  1
1
2
B 
1 

1 
Visiting frequencies
 Often we are interested in evaluating how many times the
system enters the various states, i.e. the visiting
frequencies
 The visiting frequency for state j is denoted j, and could
be obtained by:
 j = -Pjajj
 From our example we obtain the “system failure rate”
 0   P0a00 
B12
(2  B )1  (2  1 ) B
19
Time dependent solution
 Up to now we have investigated the steady state situation
 In some situations we also want to investigate the time
dependent solution, i.e. the probability that the system is
in e.g. state 0 at time t
 We now let Pi(t) be the probability that the system is in
state i at time t
 The time dependent solution may be found by:
 P(t) A = d P(t)/d t
 Which could be solved by Laplace methods, or numerical
methods
 For numerical methods we apply Markov.xls
20
Spare parts – Simple model
 Assume a spare part regime where there is only one
spare in the stock
 Upon a demand (with demand rate ) a new spare is
ordered
 The intensity of arrival of a new spare is  = 1/ MTA
(Mean Time to Arrival)
  
A







1
0

21
Spare parts 2 spares in stock
 Assume a spare part regime where there are two
spares in the stock
 Upon a demand (with demand rate ) a new spare is
ordered
 The intensity of arrival of a new spares is  = 1/ MTA
(Mean Time to Arrival) independent of how many in
order
0
1
2

2


1
A
0

22
0
1
2



 0

    

 
0