Intro. to Stochastic Processes
Cheng-Fu Chou
Cheng-Fu Chou, CMLab, CSIE, NTU
Outline
 Stochastic Process
 Counting Process
 Poisson Process
 Markov Process
 Renewal Process
P. 2
Cheng-Fu Chou, CMLAB, CSIE, NTU
Stochastic Process
 A stochastic process N= {N(t), t T} is a collection of r.v., i.e.,
for each t in the index set T, N(t) is a random variable
– t: time
– N(t): state at time t
– If T is a countable set, N is a discrete-time stochastic
process
– If T is continuous, N is a continuous-time stoc. proc.
P. 3
Cheng-Fu Chou, CMLAB, CSIE, NTU
Counting Process
 A stochastic process {N(t) ,t  0} is said to be a counting
process if N(t) is the total number of events that occurred up
to time t. Hence, some properties of a counting process is
– N(t)  0
– N(t) is integer valued
– If s < t, N(t)  N(s)
– For s < t, N(t) – N(s) equals number of events occurring in
the interval (s, t]
P. 4
Cheng-Fu Chou, CMLAB, CSIE, NTU
Counting Process
 Independent increments
– If the number of events that occur in disjoint time intervals
are independent
 Stationary increments
– If the dist. of number of events that occur in any interval of
time depends only on the length of time interval
P. 5
Cheng-Fu Chou, CMLAB, CSIE, NTU
Poisson Process
 Def. A: the counting process {N(t), t0} is said to be Poisson
process having rate l, l>0 if
– N(0) = 0;
– The process has independent-increments
– Number of events in any interval of length t is Poisson dist.
with mean lt, that is for all s, t 0.
P[ N (t  s)  N ( s)  n]  e  lt
(l t ) n
n!
n = 0,1, 2,...
P. 6
Cheng-Fu Chou, CMLAB, CSIE, NTU
Poisson Process
 Def. B: The counting process {N(t), t 0} is said to be a Poisson
process with rate l, l>0, if
– N(0) = 0
– The process has stationary and independent increments
– P[N(h) = 1] = lh +o(h)
– P[N(h)  2] = o(h)
f (h)
0
– The func. f is said to be o(h) if lim h 0
h
– Def A  Def B, i.e,. they are equivalent.
– We show Def B  Def A
– Def A  Def B is HW
P. 7
Cheng-Fu Chou, CMLAB, CSIE, NTU
Important Properties
 Property 1: mean number of event for any t 0, E[N(t)]=lt.
 Property 2: the inter-arrival time dist. of a Poisson process
with rate l is an exponential dist. with parameter l.
 Property 3: the superposition of two independent Poisson
process with rate l1 and l2 is a Poisson process with rate
l1+l2
P. 8
Cheng-Fu Chou, CMLAB, CSIE, NTU
Properties (cont.)
 Property 4: if we perform Bernoulli trials to make independent
random erasures from a Poisson process, the remaining
arrivals also form a Poisson process
 Property 5: the time until rth arrival , i.e., tr is known as the
rth order waiting time, is the sum of r independent
experimental values of t and is described by Erlan pdf.
P. 9
Cheng-Fu Chou, CMLAB, CSIE, NTU
Ex 1
 Suppose that X1 and X2 are independent exponential random
variables with respective means 1/l1 and 1/l2;What is P{X1 <
X2}
P. 10
Cheng-Fu Chou, CMLAB, CSIE, NTU

P{ X 1  X 2 }   P{ X 1  X 2 | X 1  x}l1e l1x dx
0

  P{x  X 2 }l1e l1x dx
0

  e l2 x l1e l1x dx
0

  l1e ( l1  l2 ) x dx
0

l1
l1  l2
P. 11
Cheng-Fu Chou, CMLAB, CSIE, NTU
Conditional Dist. Of the Arrival Time
 Suppose we are told that exactly one event of a Poisson
process has taken place by time t, what is the distribution of
the time at which the event occurred?
P. 12
Cheng-Fu Chou, CMLAB, CSIE, NTU
P{x  s, N (t )  1}
P{N (t )  1}
P{1 eventin [0,s), 0 eventsin [s, t)}

P {N(t) 1}
P{1 eventin [0,s)}P {0 eventsin [s, t)}

P {N(t) 1}
P{x  s | N (t )  1} 
lse ls e l (t  s )

lte lt
s
t
So, the timeof theeventis uniformlydistributed over[0,t]

P. 13
Cheng-Fu Chou, CMLAB, CSIE, NTU
Ex 2
 Consider the failure of a link in a communication network.
Failures occur according to a Poisson process with rate 4.8 per
day. Find
– P[time between failures  10 days]
– P[5 failures in 20 days]
– Expected time between 2 consecutive failures
– P[0 failures in next day]
– Suppose 12 hours have elapsed since last failure, find the
expected time to next failure
P. 14
Cheng-Fu Chou, CMLAB, CSIE, NTU
1. 1  e
4.8*10
5
(4.8* 20)  4.8*20
2.
e
5!
3. 5 hours
4. e
 4.8
5. 5 hours
P. 15
Cheng-Fu Chou, CMLAB, CSIE, NTU
Poisson, Markov, Renewal Processes
Poisson Process:
Counting process
iid exponential
times between
arrivals
Relax
counting
process
Relax
exponentia
l
interarrival
times
Continuous Time
Markov Chain:
Exponential times
between transitions
Renewal Process:
Counting process
iid times between
arrivals
P. 16
Cheng-Fu Chou, CMLAB, CSIE, NTU
Markov Process
Cheng-Fu Chou, CMLab, CSIE, NTU
Markov Process
 P[X(tn+1)  Xn+1| X(tn)= xn, X(tn-1) = xn-1,…X(t1)=x1] =
P[X(tn+1)  Xn+1| X(tn)=xn]
– Probabilistic future of the process depends only on the
current state, not on the history
– We are mostly concerned with discrete-space Markov
process, commonly referred to as Markov chains
– Discrete-time Markov chains
– Continuous-time Markov chains
P. 18
Cheng-Fu Chou, CMLAB, CSIE, NTU
DTMC
 Discrete Time Markov Chain:
– P[Xn+1 = j | Xn= kn, Xn-1 = kn-1,…X0= k0]
= P[Xn+1 = j | Xn = kn]
 discrete time, discrete space
 A finite-state DTMC if its state space is finite
 A homogeneous DTMC if P[Xn+1 = j | Xn= i ] does not depend
on n for all i, j, i.e., Pij = P[Xn+1 = j | Xn= i ], where Pij is one
step transition prob.
P. 19
Cheng-Fu Chou, CMLAB, CSIE, NTU
Definition
 P = [ Pij] is the transition matrix
 p00
p
 10
P   ...

 pi 0
 ...
p01 ...
p11 ...
p0 j
p1 j
...
...
...
...
...
pij
...
...
...
where pij  0 and
p
ij
...
...
...

...
...
1
j
– A matrix that satisfies those conditions is called a stochastic matrix
– n-step transition prob.
pijn  P[ xn  j | x0  i ]
i, j, n  0, pijn is the prob. of going from state i
to j in n step
P. 20
Cheng-Fu Chou, CMLAB, CSIE, NTU
Chapman-Kolmogorov Eq.
 Def.
For all n  0, m  0, i , j  I
pij( n  m )   pikn pkjm
kI
in matrix form P n  m  P n P m where P n =[pijn ]
 Proof:
P. 21
Cheng-Fu Chou, CMLAB, CSIE, NTU
Question
 We have only been dealing with conditional prob. but what
we want is to compute the unconditional prob. that the
system is in state j at time n, i.e.
 n ( j )  p( xn  j )
So, given the initial dist. of x0 ,i.e.,
 0 (i)  p( x0  i ) and   0  1
iI
we can get
p[ xn  j ]   p( xn  j | x0  i ) 0 (i )
iI
  pijn 0 (i)
iI
P. 22
Cheng-Fu Chou, CMLAB, CSIE, NTU
Result 1
 For all n  1, n = 0Pn, where m = (m(0),m(1),…) for all m 
0. From the above equ., we deduce that n+1 = nP. Assume
that limn n(i) exists for all i, and refer it as (i). The
remaining question is how to compute 
– Reachable: a state j is reachable from i. if
pijn  0 for some n  1
– Communicate: if j is reachable from i and if i is reachable
form j, then we say that i and j communicate (i  j)
P. 23
Cheng-Fu Chou, CMLAB, CSIE, NTU
Result 1 (cont.)
 Irreducible:
– A M.C. is irreducible if i  j for all i,j I
 Aperiodic:
– For every state iI, define d(i) to be largest common
divisor of all integer n, s.t.,
pijn  0 if d (i)  1 then the state is aperiodic
P. 24
Cheng-Fu Chou, CMLAB, CSIE, NTU
Result 2
 Invariant measure of a M.C., if a M.C. with transition matrix P
is irreducible and aperiodic and if the system of equation =P
and 1=1 has a strict positive solution then (i) = limn n(i)
independently of initial dist.
– Invariant equ. : =P
– Invariant measure 
P. 25
Cheng-Fu Chou, CMLAB, CSIE, NTU
Gambler’s Ruin Problem
 Consider a gambler who at each play of game has probability
p of winning one unit and probability q=1-p of losing one unit.
Assuming that successive plays of the game are independent,
what is the probability that, starting with i units, the gambler’s
fortune will reach N before reaching 0?
P. 26
Cheng-Fu Chou, CMLAB, CSIE, NTU
Ans
 If we let Xn denote the player’s fortune at time n, then the
process {Xn, n=0, 1,2,…} is a Markov chain with transition
probabilities:
– p00 =pNN =1
– pi,i+1 = p = 1-pi,i-1
 This Markov chain has 3 classes of states: {0},{1,2,…,N-1}, and
{N}
P. 27
Cheng-Fu Chou, CMLAB, CSIE, NTU
 Let Pi, i=0,1,2,…,N, denote the prob. That, starting with i, the
gambler’s fortune will eventually reach N.
 By conditioning on the outcome of the initial play of the game
we obtain
– Pi = pPi+1 + qPi-1, i=1,2, …, N-1
Since p+q =1
Pi+1 – Pi = q/p(Pi-Pi-1),
Also, P0 =0, so
P2 – P1 = q/p*(P1-P0) = q/p*P1
P3 - P2 =q/p*(P2-P1)= (q/p)2*P1
P. 28
Cheng-Fu Chou, CMLAB, CSIE, NTU
1  (q / p)i
p
P1 if  1

q
Pi  1  (q / p )
p

iP1 if  1

q
Now, using PN  1, we obtain
 1  (q / p)
if p  1 / 2

N
P1  1  (q / p )
1

if p  1 / 2

N
Note that , as N  
  q i

Pi  1 -  p  if p  1/2
 
 0 if p  1/2

P. 29
Cheng-Fu Chou, CMLAB, CSIE, NTU
 If p > ½, there is a positive prob. that the gambler’s fortune
will increase indefinitely
 Otherwise, the gambler will, with prob. 1, go broke against an
infinitely rich adversary.
P. 30
Cheng-Fu Chou, CMLAB, CSIE, NTU
CTMC
 Continuous-time Markov Chain
– Continuous time, discrete state
– P[X(t)= j | X(s)=i, X(sn-1)= in-1,…X(s0)= i0]
= P[X(t)= j | X(s)=i]
– A continuous M.C. is homogeneous if
o P[X(t+u)= j | X(s+u)=i] = P[X(t)= j | X(s)=i] = Pij[t-s],
where t > s
– Chapman-Kolmogorov equ.
For all t > 0, s > 0, i , j  I
pij (t  s)   pik (t ) pkj ( s)
kI
P. 31
Cheng-Fu Chou, CMLAB, CSIE, NTU
CTMC (cont.)
(t)=(0)eQt
– Q is called the infinitesimal generator
– Proof:
P. 32
Cheng-Fu Chou, CMLAB, CSIE, NTU
Result 3
 If a continuous M.C. with infinitesimal generator Q is
irreducible and if the system of equations Q = 0, and 1=1,
has a strictly positive solution then (i)= limt p(x(t)=i) for all
iI, independently of the initial dist.
P. 33
Cheng-Fu Chou, CMLAB, CSIE, NTU
Renewal Process
Cheng-Fu Chou, CMLab, CSIE, NTU
Renewal Process
A counting process {N(t), t  0} is a renewal process if for each n,
Xn is the time between the (n-1)st and nth arrivals and {Xn, n  1}
are independent with the same distribution F.
The time of the nth arrival is
n
with S0 = 0.
Sn   i 1 X i , n  1,
Can write
and if m = E[Xn], n  1, then the strong law of large numbers says
that
N t   max n : S  t
n
 Sn

P   m as n     1
n

Note: m is now a
time interval, not a rate;
1/ m will be called
the rate of the r. p.
Cheng-Fu Chou, CMLAB, CSIE, NTU
Fundamental Relationship
It follows that
N t   n  Sn  t
P  N  t   n  P  N  t   n  P N  t   n  1 
P Sn  t  P Sn 1  t  Fn  t   Fn 1  t 
where Fn(t) is the n-fold convolution of F with itself.
The mean value of N(t) is



n 1
n 1
n 1
m  t   E  N  t     P N  t   n   P Sn  t   Fn  t 
Condition on the time of the first renewal to get the renewal
equation:
m  t   F  t    m  t  x  f  x dx
t
0
Cheng-Fu Chou, CMLAB, CSIE, NTU
Exercise 1
Is it true that:
N t   n  Sn  t ?
N t   n  Sn  t ?
N t   n  Sn  t ?
Cheng-Fu Chou, CMLAB, CSIE, NTU
Exercise 2
If the mean-value function of the renewal process {N(t), t  0} is
given by m t   t 2, t  0
Then what is P{N(5) = 0} ?
Cheng-Fu Chou, CMLAB, CSIE, NTU
Exercise 3
Consider a renewal process {N(t), t  0} having a gamma (r,l)
interarrival distribution with density
(a) Show that
(b) Show that
f  x 
le
l x
 r  1!
l x
r 1
,x 0

P  N  t   n  
i  nr

i e
m t     
i r  r 
 lt
 lt  , where
e
 lt
 lt 
i
i!
i
i!
 x  is the largest integer  x
Hint: use the relationship between the gamma (r,l) distribution and the sum
of r independent exponentials with rate l to define N(t) in terms of a Poisson
process with rate l.
Cheng-Fu Chou, CMLAB, CSIE, NTU
Limit Theorems
N t 
1
 as t  
t
m
 Elementary renewal theorem:
m t 
1
 as t  
t
m
 Central limit theorem: For large t, N(t) is approximately
normally distributed with mean t/m and variance ts 2 m 3
 With probability 1,
where s2 is the variance of the time between arrivals;
in particular,
Var  N  t 
t
s2
 3 as t  
m
Cheng-Fu Chou, CMLAB, CSIE, NTU
Exercise 4
A machine in use is replaced by a new machine either when it
fails or when it reaches the age of T years. If the lifetimes of
successive machines are independent with a common
distribution F with density f, show that
(a) the long-run rate at which machines are replaced is
1
 xf  x   T 1  F T   
 0

(b) the long-run rate at which machines in use fail equals
F T 
T
 T xf  x   T 1  F T   
 0

Hint: condition on the lifetime of the first machine
Cheng-Fu Chou, CMLAB, CSIE, NTU
Renewal Reward Processes
Suppose that each time a renewal occurs we receive a reward.
Assume Rn is the reward earned at the nth renewal and {Rn, n  1}
are independent and identically distributed (Rn may depend on
Xn). The total reward up to time t is
N t 
R  t    n 1 Rn
If E  R   and E  X   
then
and


E  R
 R t 

P

as t     1
EX 
 t



E  R  t  
t

E  R
EX 
as t  
Cheng-Fu Chou, CMLAB, CSIE, NTU
Age & Excess Life of a Renewal Process
The age at time t is A(t) = the amount of time elapsed since the
last renewal. The excess life Y(t) is the time until the next
renewal:
SN(t)
t
A(t)
Y(t)
A  t   t  S N t 
A  t  dt

What is the average value of the age lim
s
0
t 
s
Cheng-Fu Chou, CMLAB, CSIE, NTU
Average Age of a Renewal Process
Imagine we receive payment at a rate equal to the current age of
the renewal process. Our total reward up to time s is s
A  t  dt

0
and the average reward up to time s is
 A t  dt  E  reward during a renewal cycle
s
0
s
E  length of a renewal cycle
If X is the length of a renewal cycle, then the total reward during
the cycle is X
X2
0 tdt  2
So, the average age is
E  X 2 
2E  X 
Cheng-Fu Chou, CMLAB, CSIE, NTU
Average Excess or Residual
Now imagine we receive payment at a rate equal to the current
excess of the renewal process. Our total reward up to time s is
 Y t  dt
s
0
and the average reward up to time s is
 Y t  dt  E  reward during a renewal cycle
s
0
s
E  length of a renewal cycle
If X is the length of a renewal cycle, then the total reward during
the cycle is X
X2
0  X  t  dt  2
So, the average excess is (also)
E  X 2 
2E  X 
Cheng-Fu Chou, CMLAB, CSIE, NTU
Inspection Paradox
Suppose that the distribution of the time between renewals, F,
is unknown. One way to estimate it is to choose some
sampling times t1, t2, etc., and for each ti, record the total
amount of time between the renewals just before and just
after ti. This scheme will overestimate the inter-renewal
times – Why?
For each sampling time, t, we will record X N t 1  SN t 1  SN t 
Find its distribution by conditioning on the time of the last
renewal prior to time t
Cheng-Fu Chou, CMLAB, CSIE, NTU
Inspection Paradox (cont.)
SN(t)+1
SN(t)

0
t-s


t

P X N t 1  x  E  P X N t 1  x S N t   t  s 



If s  x then P  X

 t  s  P  X
If s  x then P X N t 1  x S N t   t  s  1


PX
N  t  1
 x S N t 
N  t  1
  1 F  x  1 F  x
 s 1  F  s 
P X N t 1  x
N  t  1

 x X N t 1  s
Cheng-Fu Chou, CMLAB, CSIE, NTU
Inspection Paradox (cont.)
SN(t)+1
SN(t)
0
t
t-s


For any s, P X N t 1  x S N t   t  s  1  F  x 
so P X
P X


x

E

x
S

t

s
N  t  1
N  t  1
N t 






 1  F  x   P  X  x

where X is an ordinary inter-renewal time.
Intuitively, by choosing “random” times, it is more likely
we will choose a time that falls in a long time interval.
Cheng-Fu Chou, CMLAB, CSIE, NTU