Iterative Transient State
Distribution Calculation in
Semi-Markov Processes
Nick Dingle
njd200@doc.ic.ac.uk
Department of Computing, Imperial College London,
South Kensington Campus, London SW7 2AZ.
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.1
Motivation
Have focused mainly on passage time density and
quantile calculation in previous work
Other measures of interest, e.g. transient state
distributions
System may never reach steady-state due to resets
etc,
May be interested in knowing the probability of being
in a certain state at a given moment in time
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.2
Passage Times
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.3
Transient State Distributions
0.04
Transient solution
Steady-state solution
0.035
0.03
Probability
0.025
0.02
0.015
0.01
0.005
0
0
20
40
60
80
100
Time, t
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.4
Motivation
Existing transformation of passage times → transient
distributions computational expensive
To overcome this, we seek to apply our iterative
passage time algorithm directly to transient state
distribution calculation
Work presented here is the contribution of the SI of our
PMEO-PDS 2003 paper recently submitted to FGCS
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.5
Semi-Markov Processes
A more expressive generalisation of Markov processes
where state sojourn times can be generally distributed
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.6
Semi-Markov Processes
A more expressive generalisation of Markov processes
where state sojourn times can be generally distributed
An N -state SMP is characterised by:
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.6
Semi-Markov Processes
A more expressive generalisation of Markov processes
where state sojourn times can be generally distributed
An N -state SMP is characterised by:
an N × N matrix P where pij is the probability of
moving from state i to state j ,
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.6
Semi-Markov Processes
A more expressive generalisation of Markov processes
where state sojourn times can be generally distributed
An N -state SMP is characterised by:
an N × N matrix P where pij is the probability of
moving from state i to state j ,
an N × N matrix H where hij (t) is the distribution
function of the sojourn time of the process in state i,
given that its going to state j
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.6
Semi-Markov Processes
A more expressive generalisation of Markov processes
where state sojourn times can be generally distributed
An N -state SMP is characterised by:
an N × N matrix P where pij is the probability of
moving from state i to state j ,
an N × N matrix H where hij (t) is the distribution
function of the sojourn time of the process in state i,
given that its going to state j
We def ine the kernel R(i, j, t) of an SMP as:
R(i, j, t) = pij Hij (t)
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.6
Transient State Distributions I
The transient state distribution πij (t) is:
πij (t) = IP(Z(t) = j | Z(0) = i)
where Z(t) is the state of the SMP at time t
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.7
Transient State Distributions I
The transient state distribution πij (t) is:
πij (t) = IP(Z(t) = j | Z(0) = i)
where Z(t) is the state of the SMP at time t
There is a translation from passage time densities to
transient state distributions in LT form (Pyke):
∗ (s)
1
−
h
1
∗
i
πij
(s) =
if i = j
s 1 − Lii (s)
∗
∗
πij
(s) = Lij (s)πjj
(s) if i 6= j
∗ (s) is the Laplace transform of π (t) and
where πij
ij
P
∗ (s)
h∗i (s) = k rik
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.7
Transient State Distributions II
For multiple target states:
πi∗~j (s) =
X
∗
πik
(s)
k∈~j
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.8
Transient State Distributions II
For multiple target states:
πi∗~j (s) =
X
∗
πik
(s)
k∈~j
To construct πi∗~j (s) using the translation is
computationally expensive:
for a vector of target states ~j , need 2|~j| − 1 passage
time quantities Lik (s),
requires the solution of |~j| linear systems
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.8
Transient State Distributions II
For multiple target states:
πi∗~j (s) =
X
∗
πik
(s)
k∈~j
To construct πi∗~j (s) using the translation is
computationally expensive:
for a vector of target states ~j , need 2|~j| − 1 passage
time quantities Lik (s),
requires the solution of |~j| linear systems
This motivates our development of a new transient state
distribution formula for multiple target states
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.8
Iterative Transient Algorithm I
From Pyke we derive the following expression for πij (t):
πij (t) = δij F i (t) +
N Z
X
k=1 0
t
R(i, k, t − τ ) πkj (τ ) dτ
where:
δij = 1 if i = j and 0 otherwise
F i (t) is the reliability function of the sojourn time
distribution in state i
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.9
Iterative Transient Algorithm I
From Pyke we derive the following expression for πij (t):
πij (t) = δij F i (t) +
N Z
X
k=1 0
t
R(i, k, t − τ ) πkj (τ ) dτ
where:
δij = 1 if i = j and 0 otherwise
F i (t) is the reliability function of the sojourn time
distribution in state i
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.9
Iterative Transient Algorithm I
From Pyke we derive the following expression for πij (t):
πij (t) = δij F i (t) +
N Z
X
k=1 0
t
R(i, k, t − τ ) πkj (τ ) dτ
where:
δij = 1 if i = j and 0 otherwise
F i (t) is the reliability function of the sojourn time
distribution in state i
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.9
Iterative Transient Algorithm II
Transforming the convolution into the Laplace domain
and generalising to multiple target states ~j :
πi∗~j (s)
=
∗
δi∈~j F i (s) +
N
X
∗
rik
(s)πk∗~j (s)
k=1
where δi∈~j = 1 if i ∈ ~j and 0 otherwise
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.10
Iterative Transient Algorithm IV
E.g. when ~j = {1, 3}:

















∗ (s)
1 − r11
∗ (s)
−r12
···
∗ (s)
−r1N
∗ (s)
−r21
∗ (s)
1 − r22
···
∗ (s)
−r2N
∗ (s)
−r31
∗ (s)
−r32
···
∗ (s)
−r3N
..
.
..
.
..
..
.
∗ (s)
−rN
2
∗ (s)
−rN
2
···
PASTA 2004 Edinburgh
.
∗
1 − rN
N (s)


















njd200@doc.ic.ac.uk
π ∗~ (s)
1j
π ∗~ (s)
2j
π ∗~ (s)
3j
.
..
π ∗ ~ (s)
Nj


 
 
 
 
 
 
 
 
=
 
 
 
 
 
 
 

June 2004
∗
F 1 (s)
0
∗
F 3 (s)
..
.
0

















– p.11
Iterative Transient Algorithm V
In general:
(r)
(I − U) π~ (s)
j
=v
where:
∗ (s)
matrix U has elements upq = rpq
(r)
vector π~ (s) has elements
j
³
´
π ∗~ (s), π ∗~ (s), . . . , π ∗ ~ (s)
1j
2j
Nj
vector v has elements vi =
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
∗
δi∈~j F i (s)
June 2004
– p.12
Iterative Transient Algorithm VI
Therefore:
(r)
π~ (s) = (I − U)−1 v
j
´
³
= I + U + U 2 + U3 · · · v
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.13
Iterative Transient Algorithm VI
Therefore:
(r)
π~ (s) = (I − U)−1 v
j
´
³
= I + U + U 2 + U3 · · · v
Approximate this as:
´
³
(r)
π~ (s) = I + U + U2 + · · · + Ur v
j
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.13
Iterative Transient Algorithm VI
Therefore:
(r)
π~ (s) = (I − U)−1 v
j
´
³
= I + U + U 2 + U3 · · · v
Approximate this as:
´
³
(r)
π~ (s) = I + U + U2 + · · · + Ur v
j
Can generalise to multiple start states by employing an
initial vector α:
´
³
(r)
π~~ (s) = α I + U + U2 + · · · + Ur v
ij
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.13
Example
where X ∼ exp(2.0) and Y ∼ det(2.0)
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.14
Example
0.8
Analytic solution
0.7
0.6
Probability
0.5
0.4
0.3
0.2
0.1
0
0
2
4
6
8
10
12
Time, t
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.15
Example
0.8
Analytic solution
1 iteration
0.7
0.6
Probability
0.5
0.4
0.3
0.2
0.1
0
0
2
4
6
8
10
12
Time, t
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.15
Example
0.8
Analytic solution
1 iteration
2 iterations
0.7
0.6
Probability
0.5
0.4
0.3
0.2
0.1
0
0
2
4
6
8
10
12
Time, t
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.15
Example
0.8
Analytic solution
1 iteration
2 iterations
4 iterations
0.7
0.6
Probability
0.5
0.4
0.3
0.2
0.1
0
0
2
4
6
8
10
12
Time, t
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.15
Example
0.8
Analytic solution
1 iteration
2 iterations
4 iterations
6 iterations
0.7
0.6
Probability
0.5
0.4
0.3
0.2
0.1
0
0
2
4
6
8
10
12
Time, t
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.15
Example
0.8
Analytic solution
1 iteration
2 iterations
4 iterations
6 iterations
8 iterations
0.7
0.6
Probability
0.5
0.4
0.3
0.2
0.1
0
0
2
4
6
8
10
12
Time, t
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.15
Example
0.8
Analytic solution
1 iteration
2 iterations
4 iterations
6 iterations
8 iterations
10 iterations
0.7
0.6
Probability
0.5
0.4
0.3
0.2
0.1
0
0
2
4
6
8
10
12
Time, t
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.15
Conclusion
Presented an iterative transient state distribution
calculation method which is more eff icient than
previous passage time-based transformations
Builds on our existing iterative passage time algorithm
Similarities with Markovian uniformization
Applicable to both Markov and semi-Markov processes
Applied to large SMPs generated from SM-SPNs
(PMEO) and SPAs (PASTA)
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.16
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.17
First Passage Times
First passage time from state i to target states ~j in an
SMP:
Pi~j = inf{u > 0 : Z(u) ∈ ~j, N (u) > 0, Z(0) = i}
Z(t) is the state of the SMP at time t
N (u) is the number of state transitions by time u
Laplace transform of Pi~j density can be computed as:
Li~j (s) =
X
∗
rik
(s)Lk~j (s) +
k∈
/~j
X
∗
rik
(s)
k∈~j
∗ (s) is the LST of R(i, k, t)
where rik
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.18
Iterative PT Algorithm I
Generates successively better approximations to the
passage time quantity Pi~j
(r)
ij
P ~ is the conditional passage time for the system to
reach any state in ~j in r transitions:
(r)
P ~ = inf{u > 0 : Z(u) ∈ ~j, 0 < N (u) ≤ r, Z(0) = i}
ij
(r)
P~
ij
→ Pi~j as r → ∞
PASTA 2004 Edinburgh
njd200@doc.ic.ac.uk
June 2004
– p.19
Iterative PT Algorithm II
(r)
P~ :
ij
Calculate Laplace transform of rth iteration,
r
X
(r)
L ~ (s) =
contribution from mth transition matrix
ij
=
m=1
r
X
UU
0 m−1
e~j
m=1
∗ (s) and U0 has
where matrix U has elements upq = rpq
states in ~j made absorbing
Can generalise to multiple source states ~i:
r
X
(r)
0 m−1
L~~ (s) =
αUU
e~j
ij
PASTA 2004 Edinburgh
m=1
njd200@doc.ic.ac.uk
June 2004
– p.20
F i(t)
The LT of F i (t),
∗
F i (s),
is generated from h∗i (s):
∗
F i (s)
PASTA 2004 Edinburgh
1 − h∗i (s)
=
s
njd200@doc.ic.ac.uk
June 2004
– p.21