Intro. to Stochastic Processes
Cheng-Fu Chou
Cheng-Fu Chou, CMLab, CSIE, NTU
Outline
 Stochastic Process
 Counting Process
 Poisson Process
 Markov Process
P. 2
Cheng-Fu Chou, CMLAB, CSIE, NTU
Stochastic Process
 A stochastic process N= {N(t), t T} is a collection of
r.v., i.e., for each t in the index set T, N(t) is a random
variable
– t: time
– N(t): state at time t
– If T is a countable set, N is a discrete-time
stochastic process
– If T is continuous, N is a continuous-time stoc.
proc.
P. 3
Cheng-Fu Chou, CMLAB, CSIE, NTU
Counting Process
 A stochastic process {N(t) ,t  0} is said to be a
counting process if N(t) is the total number of events
that occurred up to time t. Hence, some properties of
a counting process is
– N(t)  0
– N(t) is integer valued
– If s < t, N(t)  N(s)
– For s < t, N(t) – N(s) equals number of events
occurring in the interval (s, t]
P. 4
Cheng-Fu Chou, CMLAB, CSIE, NTU
Counting Process
 Independent increments
– If the number of events that occur in disjoint time
intervals are independent
 Stationary increments
– If the dist. of number of events that occur in any
interval of time depends only on the length of time
interval
P. 5
Cheng-Fu Chou, CMLAB, CSIE, NTU
Poisson Process
 Def. A: the counting process {N(t), t0} is said to be
Poisson process having rate l, l>0 if
– N(0) = 0;
– The process has independent-increments
– Number of events in any interval of length t is
Poisson dist. with mean lt, that is for all s, t 0.
P[ N (t  s)  N ( s)  n]  e  lt
(l t ) n
n!
n = 0,1, 2,...
P. 6
Cheng-Fu Chou, CMLAB, CSIE, NTU
Poisson Process
 Def. B: The counting process {N(t), t 0} is said to be
a Poisson process with rate l, l>0, if
– N(0) = 0
– The process has stationary and independent
increments
– P[N(h) = 1] = lh +o(h)
– P[N(h)  2] = o(h)
f (h)
0
– The func. f is said to be o(h) if lim h 0
h
– Def A  Def B, i.e,. they are equivalent.
– We show Def B  Def A
– Def A  Def B is HW
P. 7
Cheng-Fu Chou, CMLAB, CSIE, NTU
Important Properties
 Property 1: mean number of event for any t 0,
E[N(t)]=lt.
 Property 2: the inter-arrival time dist. of a Poisson
process with rate l is an exponential dist. with
parameter l.
 Property 3: the superposition of two independent
Poisson process with rate l1 and l2 is a Poisson
process with rate l1+l2
P. 8
Cheng-Fu Chou, CMLAB, CSIE, NTU
Properties (cont.)
 Property 4: if we perform Bernoulli trials to make
independent random erasures from a Poisson process,
the remaining arrivals also form a Poisson process
 Property 5: the time until rth arrival , i.e., tr is known
as the rth order waiting time, is the sum of r
independent experimental values of t and is described
by Erlan pdf.
P. 9
Cheng-Fu Chou, CMLAB, CSIE, NTU
Ex 1
 Suppose that X1 and X2 are independent exponential
random variables with respective means 1/l1 and
1/l2;What is P{X1 < X2}
P. 10
Cheng-Fu Chou, CMLAB, CSIE, NTU

P{ X 1  X 2 }   P{ X 1  X 2 | X 1  x}l1e l1x dx
0

  P{x  X 2 }l1e l1x dx
0

  e l2 x l1e l1x dx
0

  l1e ( l1  l2 ) x dx
0

l1
l1  l2
P. 11
Cheng-Fu Chou, CMLAB, CSIE, NTU
Conditional Dist. Of the Arrival Time
 Suppose we are told that exactly one event of a
Poisson process has taken place by time t, what is the
distribution of the time at which the event occurred?
P. 12
Cheng-Fu Chou, CMLAB, CSIE, NTU
P{x  s, N (t )  1}
P{N (t )  1}
P{1 eventin [0,s), 0 eventsin [s, t)}

P {N(t) 1}
P{1 eventin [0,s)}P {0 eventsin [s, t)}

P {N(t) 1}
P{x  s | N (t )  1} 
lse ls e l (t  s )

lte lt
s
t
So, the timeof theeventis uniformlydistributed over[0,t]

P. 13
Cheng-Fu Chou, CMLAB, CSIE, NTU
Ex 2
 Consider the failure of a link in a communication
network. Failures occur according to a Poisson process
with rate 4.8 per day. Find
– P[time between failures  10 days]
– P[5 failures in 20 days]
– Expected time between 2 consecutive failures
– P[0 failures in next day]
– Suppose 12 hours have elapsed since last failure,
find the expected time to next failure
P. 14
Cheng-Fu Chou, CMLAB, CSIE, NTU
1. 1  e
4.8*10
5
(4.8* 20)  4.8*20
2.
e
5!
3. 5 hours
4. e
 4.8
5. 5 hours
P. 15
Cheng-Fu Chou, CMLAB, CSIE, NTU
Markov Process
 P[X(tn+1)  Xn+1| X(tn)= xn, X(tn-1) = xn-1,…X(t1)=x1] =
P[X(tn+1)  Xn+1| X(tn)=xn]
– Probabilistic future of the process depends only on
the current state, not on the history
– We are mostly concerned with discrete-space
Markov process, commonly referred to as Markov
chains
– Discrete-time Markov chains
– Continuous-time Markov chains
P. 16
Cheng-Fu Chou, CMLAB, CSIE, NTU
DTMC
 Discrete Time Markov Chain:
– P[Xn+1 = j | Xn= kn, Xn-1 = kn-1,…X0= k0]
= P[Xn+1 = j | Xn = kn]
 discrete time, discrete space
 A finite-state DTMC if its state space is finite
 A homogeneous DTMC if P[Xn+1 = j | Xn= i ] does not
depend on n for all i, j, i.e., Pij = P[Xn+1 = j | Xn= i ], where Pij
is one step transition prob.
P. 17
Cheng-Fu Chou, CMLAB, CSIE, NTU
Definition
 P = [ Pij] is the transition matrix
 p00
p
 10
P   ...

 pi 0
 ...
p01 ...
p11 ...
p0 j
p1 j
...
...
...
...
...
pij
...
...
...
where pij  0 and
p
ij
...
...
...

...
...
1
j
– A matrix that satisfies those conditions is called a stochastic
matrix
– n-step transition prob.
pijn  P[ xn  j | x0  i ]
i, j, n  0, pijn is the prob. of going from state i
to j in n step
P. 18
Cheng-Fu Chou, CMLAB, CSIE, NTU
Chapman-Kolmogorov Eq.
 Def.
For all n  0, m  0, i , j  I
pij( n  m )   pikn pkjm
kI
in matrix form P n  m  P n P m where P n =[pijn ]
 Proof:
P. 19
Cheng-Fu Chou, CMLAB, CSIE, NTU
Question
 We have only been dealing with conditional prob. but
what we want is to compute the unconditional prob.
that the system is in state j at time n, i.e.
 n ( j )  p( xn  j )
So, given the initial dist. of x0 ,i.e.,
 0 (i)  p( x0  i ) and   0  1
iI
we can get
p[ xn  j ]   p( xn  j | x0  i ) 0 (i )
iI
  pijn 0 (i)
iI
P. 20
Cheng-Fu Chou, CMLAB, CSIE, NTU
Result 1
 For all n  1, n = 0Pn, where m = (m(0),m(1),…) for all
m  0. From the above equ., we deduce that n+1 = nP.
Assume that limn n(i) exists for all i, and refer it
as (i). The remaining question is how to compute 
– Reachable: a state j is reachable from i. if
pijn  0 for some n  1
– Communicate: if j is reachable from i and if i is reachable
form j, then we say that i and j communicate (i  j)
P. 21
Cheng-Fu Chou, CMLAB, CSIE, NTU
Result 1 (cont.)
 Irreducible:
– A M.C. is irreducible if i  j for all i,j I
 Aperiodic:
– For every state iI, define d(i) to be largest common
divisor of all integer n, s.t.,
pijn  0 if d (i)  1 then the state is aperiodic
P. 22
Cheng-Fu Chou, CMLAB, CSIE, NTU
Result 2
 Invariant measure of a M.C., if a M.C. with transition
matrix P is irreducible and aperiodic and if the system
of equation =P and 1=1 has a strict positive
solution then (i) = limn n(i) independently of initial
dist.
– Invariant equ. : =P
– Invariant measure 
P. 23
Cheng-Fu Chou, CMLAB, CSIE, NTU
Gambler’s Ruin Problem
 Consider a gambler who at each play of game has
probability p of winning one unit and probability q=1-p
of losing one unit. Assuming that successive plays of
the game are independent, what is the probability
that, starting with i units, the gambler’s fortune will
reach N before reaching 0?
P. 24
Cheng-Fu Chou, CMLAB, CSIE, NTU
Ans
 If we let Xn denote the player’s fortune at time n,
then the process {Xn, n=0, 1,2,…} is a Markov chain
with transition probabilities:
– p00 =pNN =1
– pi,i+1 = p = 1-pi,i-1
 This Markov chain has 3 classes of states:
{0},{1,2,…,N-1}, and {N}
P. 25
Cheng-Fu Chou, CMLAB, CSIE, NTU
 Let Pi, i=0,1,2,…,N, denote the prob. That, starting
with i, the gambler’s fortune will eventually reach N.
 By conditioning on the outcome of the initial play of
the game we obtain
– Pi = pPi+1 + qPi-1, i=1,2, …, N-1
Since p+q =1
Pi+1 – Pi = q/p(Pi-Pi-1),
Also, P0 =0, so
P2 – P1 = q/p*(P1-P0) = q/p*P1
P3 - P2 =q/p*(P2-P1)= (q/p)2*P1
P. 26
Cheng-Fu Chou, CMLAB, CSIE, NTU
1  (q / p)i
p
P1 if  1

q
Pi  1  (q / p )
p

iP1 if  1

q
Now, using PN  1, we obtain
 1  (q / p)
if p  1 / 2

N
P1  1  (q / p )
1

if p  1 / 2

N
Note that , as N  
  q i

Pi  1 -  p  if p  1/2
 
 0 if p  1/2

P. 27
Cheng-Fu Chou, CMLAB, CSIE, NTU
 If p > ½, there is a positive prob. that the gambler’s
fortune will increase indefinitely
 Otherwise, the gambler will, with prob. 1, go broke
against an infinitely rich adversary.
P. 28
Cheng-Fu Chou, CMLAB, CSIE, NTU
CTMC
 Continuous-time Markov Chain
– Continuous time, discrete state
– P[X(t)= j | X(s)=i, X(sn-1)= in-1,…X(s0)= i0]
= P[X(t)= j | X(s)=i]
– A continuous M.C. is homogeneous if
o P[X(t+u)= j | X(s+u)=i] = P[X(t)= j | X(s)=i] = Pij[t-s],
where t > s
– Chapman-Kolmogorov equ.
For all t > 0, s > 0, i , j  I
pij (t  s)   pik (t ) pkj ( s)
kI
P. 29
Cheng-Fu Chou, CMLAB, CSIE, NTU
CTMC (cont.)
(t)=(0)eQt
– Q is called the infinitesimal generator
– Proof:
P. 30
Cheng-Fu Chou, CMLAB, CSIE, NTU
Result 3
 If a continuous M.C. with infinitesimal generator Q is
irreducible and if the system of equations Q = 0, and
1=1, has a strictly positive solution then (i)= limt
p(x(t)=i) for all iI, independently of the initial dist.
P. 31
Cheng-Fu Chou, CMLAB, CSIE, NTU