An Introduction
To Stochastic
Modeling
Howard M.Taylor
Samuel Karlin
An Introduction to
Stochastic Modeling
Third Edition
An Introduction to
Stochastic Modeling
Third Edition
Howard M. Taylor
Statistical Consultant
Onancock, Vi ginia
Samuel Karlin
Department of Mathematics
Stanford University
Stanford, California
O
Academic Press
San Diego
London
New York
Sydney
Boston
Tokyo
Toronto
This book is printed on acid-free paper.
Copyright © 1998, 1994, 1984 by Academic Press
All rights reserved.
No part of this publication may be reproduced or
transmitted in any form or by any means, electronic
or mechanical, including photocopy, recording, or
any information storage and retrieval system, without
permission in writing from the publisher.
Permissions may be sought directly from Elsevier's Science and Technology Rights Department in
Oxford, UK. Phone: (44) 1865 843830, Fax: (44) 1865 853333, c-mail: permissions@elsevier.co.uk.
You may also complete your request on-line via the Elsevier homepage: httpJ/www.elseviercom by
selecting 'Customer Support' and then 'Obtaining Permissions'.
ACADEMIC PRESS
An Imprint of Elsevier
525 B St., Suite 1900, San Diego, California 92101-4495, USA
1300 Boylston Street, Chestnut Hill, MA 02167, USA
http://www.apnet.com
Academic Press Limited
24-28 Oval Road, London NW 1 7DX, UK
http://www.hbuk.co.uk/ap/
Library of Congress Cataloging-in-Publication Data
Taylor, Howard M.
An introduction to stochastic modeling / Howard M. Taylor, Samuel
Karlin. - 3rd ed.
p.
cm.
Includes bibliographical references (p.
ISBN-13: 978-0-12-684887-8
1. Stochastic processes.
QA274.T35 1998
003'.76--dc2l
-
) and index.
ISBN-10: 0-12-684887-4
I. Karlin, Samuel.
ISBN-13: 978-0-12-684887-8
ISBN-10: 0-12-684887-4
PRINTED IN THE UNITED STATES OF AMERICA
05060708 IP 987654
II. Title.
Contents
Preface
I
II
III
Introduction
ix
1
1. Stochastic Modeling
2. Probability Review
3. The Major Discrete Distributions
4. Important Continuous Distributions
5. Some Elementary Exercises
6. Useful Functions, Integrals, and Sums
6
24
33
43
53
Conditional Probability and Conditional
Expectation
57
1. The Discrete Case
2. The Dice Game Craps
3. Random Sums
57
64
70
4. Conditioning on a Continuous Random Variable
5. Martingales*
79
87
Markov Chains: Introduction
95
1. Definitions
1
95
2. Transition Probability Matrices of a Markov Chain
100
3. Some Markov Chain Models
4. First Step Analysis
5. Some Special Markov Chains
105
116
135
151
6. Functionals of Random Walks and Success Runs
*Stars indicate topics of a more advanced or specialized nature.
vi
Contents
7. Another Look at First Step Analysis*
8. Branching Processes*
9. Branching Processes and Generating Functions*
169
177
184
IV The Long Run Behavior of Markov Chains
199
1. Regular Transition Probability Matrices
2. Examples
199
3. The Classification of States
4. The Basic Limit Theorem of Markov Chains
5. Reducible Markov Chains*
V Poisson Processes
VI
VII
215
234
245
258
267
1. The Poisson Distribution and the Poisson Process
2. The Law of Rare Events
3. Distributions Associated with the Poisson Process
4. The Uniform Distribution and Poisson Processes
5. Spatial Poisson Processes
6. Compound and Marked Poisson Processes
267
279
290
297
Continuous Time Markov Chains
333
1. Pure Birth Processes
2. Pure Death Processes
3. Birth and Death Processes
4. The Limiting Behavior of Birth and Death
Processes
5. Birth and Death Processes with Absorbing States
6. Finite State Continuous Time Markov Chains
7. A Poisson Process with a Markov Intensity*
333
345
355
Renewal Phenomena
419
1. Definition of a Renewal Process and
Related Concepts
2. Some Examples of Renewal Processes
3. The Poisson Process Viewed as a Renewal
419
426
Process
*Stars indicate topics of a more advanced or specialized nature.
311
318
366
379
394
408
432
vii
Contents
4. The Asymptotic Behavior of Renewal Processes
437
5. Generalizations and Variations on Renewal
Processes
VIII
447
6. Discrete Renewal Theory*
457
Brownian Motion and Related Processes
473
1. Brownian Motion and Gaussian Processes
473
2. The Maximum Variable and the Reflection Principle 491
IX
3. Variations and Extensions
4. Brownian Motion with Drift
5. The Ornstein-Uhlenbeck Process*
498
508
524
Queueing Systems
541
1. Queueing Processes
2. Poisson Arrivals, Exponential Service Times
3. General Service Time Distributions
4. Variations and Extensions
5. Open Acyclic Queueing Networks
6. General Open Networks
541
547
558
567
581
592
Further Reading
601
Answers to Exercises
603
Index
625
*Stars indicate topics of a more advanced or specialized nature.
Preface to the First Edition
Stochastic processes are ways of quantifying the dynamic relationships of
sequences of random events. Stochastic models play an important role in
elucidating many areas of the natural and engineering sciences. They can
be used to analyze the variability inherent in biological and medical
processes, to deal with uncertainties affecting managerial decisions and
with the complexities of psychological and social interactions, and to provide new perspectives, methodology, models, and intuition to aid in other
mathematical and statistical studies.
This book is intended as a beginning text in stochastic processes for students familiar with elementary probability calculus. Its aim is to bridge
the gap between basic probability know-how and an intermediate-level
course in stochastic processes-for example, A First Course in Stochastic
Processes, by the present authors.
The objectives of this book are three: (1) to introduce students to the
standard concepts and methods of stochastic modeling; (2) to illustrate the
rich diversity of applications of stochastic processes in the sciences; and
(3) to provide exercises in the application of simple stochastic analysis to
appropriate problems.
The chapters are organized around several prototype classes of stochastic processes featuring Markov chains in discrete and continuous
time, Poisson processes and renewal theory, the evolution of branching
events, and queueing models. After the concluding Chapter IX, we provide a list of books that incorporate more advanced discussions of several
of the models set forth in this text.
Preface to the Third Edition
The purposes, level, and style of this new edition conform to the tenets set
forth in the original preface. We continue with our objective of introducing some theory and applications of stochastic processes to students hav-
ing a solid foundation in calculus and in calculus-level probability, but
who are not conversant with the "epsilon-delta" definitions of mathematical analysis. We hope to entice students towards the deeper study of
mathematics that is prerequisite to further work in stochastic processes by
showing the myriad and interesting ways in which stochastic models can
help us understand the real world.
We have removed some topics and added others. We added a small section on martingales that includes an example suggesting the martingale
concept as appropriate for modeling the prices of assets traded in a perfect
market. A new chapter introduces the Brownian motion process and includes several applications of it and its variants in financial modeling. In
this chapter the Black-Scholes formula for option pricing is evaluated and
compared with some reported prices of options. A Poisson process whose
intensity is itself a stochastic process is described in another new section.
Some treatments have been updated. The law of rare events is presented
via an inequality that measures the accuracy of a Poisson approximation
for the distribution of the sum of independent, not necessarily identically
distributed, Bernoulli random variables. We have added the shot noise
model and related it to a random sum.
The text contains more than 250 exercises and 350 problems. Exercises
are elementary drills intended to promote active learning, to develop familiarity with concepts through use. They often simply involve the substitution of numbers into given formulas, or reasoning one or two steps
away from a definition. They are the kinds of simple questions that we, as
instructors, hope that students would pose and answer for themselves as
they read a text. Answers to the exercises are given at the end of the book
so that students may gauge their understanding as they go along.
Problems are more difficult. Some involve extensive algebraic or calculus manipulation. Many are "word problems" wherein the student is
asked, in effect, to model some described scenario. As in formulating a
model, the first step in the solution of a word problem is often a sentence
of the form "Let x = ...." A manual containing the solutions to the problems is available from the publisher.
A reasonable strategy on the part of the teacher might be to hold students responsible for all of the exercises, but to require submitted solutions only to selected problems. Every student should attempt a representative selection of the problems in order to develop his or her ability to
carry out stochastic modeling in his or her area of interest.
A small number of problems are labeled "Computer Challenges." These
call for more than pencil and paper for their analyses, and either simulation, numerical exploration, or symbol manipulation may prove helpful.
Computer Challenges are meant to be open-ended, intended to explore
what constitutes an answer in today's world of computing power. They
might be appropriate as part of an honors requirement.
Because our focus is on stochastic modeling, in some instances we have
omitted a proof and contented ourselves with a precise statement of a
result and examples of its application. All such omitted proofs may be
found in A First Course in Stochastic Processes, by the present authors.
In this more advanced text, the ambitious student will also find additional
material on martingales, Brownian motion, and renewal processes, and
presentations of several other classes of stochastic processes.
To the Instructor
If possible, we recommend having students skim the first two chapters, re-
ferring as necessary to the probability review material, and starting the
course with Chapter III, on Markov chains. A one quarter course adapted
to the junior-senior level could consist of a cursory (one-week) review of
Chapters I and II, followed in order by Chapters III through VI. For interested students, Chapters VII, VIII, and IX discuss other currently active
areas of stochastic modeling. Starred sections contain material of a more
advanced or specialized nature.
Acknowledgments
Many people helped to bring this text into being. We gratefully acknowl-
edge the help of Anna Karlin, Shelley Stevens, Karen Larsen, and
Laurieann Shoemaker. Chapter IX was enriched by a series of lectures on
queueing networks given by Ralph Disney at The Johns Hopkins University in 1982. Alan Karr, Ivan Johnstone, Luke Tierney, Bob Vanderbei,
and others besides ourselves have taught from the text, and we have profited from their criticisms. Finally, we are grateful for improvements suggested by the several generations of students who have used the book over
the past few years and have given us their reactions and suggestions.
Chapter I
Introduction
1.
Stochastic Modeling
A quantitative description of a natural phenomenon is called a mathematical model of that phenomenon. Examples abound, from the simple
equation S = Zgt2 describing the distance S traveled in time t by a falling
object starting at rest to a complex computer program that simulates a
biological population or a large industrial system.
In the final analysis, a model is judged using a single, quite pragmatic,
factor, the model's usefulness. Some models are useful as detailed quantitative prescriptions of behavior, as for example, an inventory model that
is used to determine the optimal number of units to stock. Another model
in a different context may provide only general qualitative information
about the relationships among and relative importance of several factors
influencing an event. Such a model is useful in an equally important but
quite different way. Examples of diverse types of stochastic models are
spread throughout this book.
Such often mentioned attributes as realism, elegance, validity, and
reproducibility are important in evaluating a model only insofar as they
bear on that model's ultimate usefulness. For instance, it is both unrealistic and quite inelegant to view the sprawling city of Los Angeles as a geometrical point, a mathematical object of no size or dimension. Yet it is
quite useful to do exactly that when using spherical geometry to derive a
minimum-distance great circle air route from New York City, another
"point."
2
I
Introduction
There is no such thing as the best model for a given phenomenon. The
pragmatic criterion of usefulness often allows the existence of two or
more models for the same event, but serving distinct purposes. Consider
light. The wave form model, in which light is viewed as a continuous flow,
is entirely adequate for designing eyeglass and telescope lenses. In con-
trast, for understanding the impact of light on the retina of the eye, the
photon model, which views light as tiny discrete bundles of energy, is
preferred. Neither model supersedes the other; both are relevant and
useful.
The word "stochastic" derives from the Greed
to aim, to
guess) and means "random" or "chance." The antonym is "sure," "deterministic," or "certain." A deterministic model predicts a single outcome
from a given set of circumstances. A stochastic model predicts a set of
possible outcomes weighted by their likelihoods, or probabilities. A coin
flipped into the air will surely return to earth somewhere. Whether it lands
heads or tails is random. For a "fair" coin we consider these alternatives
equally likely and assign to each the probability 12.
However, phenomena are not in and of themselves inherently stochas-
tic or deterministic. Rather, to model a phenomenon as stochastic or deterministic is the choice of the observer. The choice depends on the observer's purpose; the criterion for judging the choice is usefulness. Most
often the proper choice is quite clear, but controversial situations do arise.
If the coin once fallen is quickly covered by a book so that the outcome
"heads" or "tails" remains unknown, two participants may still usefully
employ probability concepts to evaluate what is a fair bet between them;
that is, they may usefully view the coin as random, even though most people would consider the outcome now to be fixed or deterministic. As a less
mundane example of the converse situation, changes in the level of a large
population are often usefully modeled deterministically, in spite of the
general agreement among observers that many chance events contribute
to their fluctuations.
Scientific modeling has three components: (i) a natural phenomenon
under study, (ii) a logical system for deducing implications about the phenomenon, and (iii) a connection linking the elements of the natural system
under study to the logical system used to model it. If we think of these
three components in terms of the great-circle air route problem, the natural system is the earth with airports at Los Angeles and New York; the
logical system is the mathematical subject of spherical geometry; and the
1.
Stochastic Modeling
3
two are connected by viewing the airports in the physical system as points
in the logical system.
The modern approach to stochastic modeling is in a similar spirit. Nature does not dictate a unique definition of "probability," in the same way
that there is no nature-imposed definition of "point" in geometry. "Probability" and "point" are terms in pure mathematics, defined only through
the properties invested in them by their respective sets of axioms. (See
Section 2.8 for a review of axiomatic probability theory.) There are, however, three general principles that are often useful in relating or connecting the abstract elements of mathematical probability theory to a real or
natural phenomenon that is to be modeled. These are (i) the principle of
equally likely outcomes, (ii) the principle of long run relative frequency,
and (iii) the principle of odds making or subjective probabilities. Historically, these three concepts arose out of largely unsuccessful attempts to
define probability in terms of physical experiences. Today, they are relevant as guidelines for the assignment of probability values in a model, and
for the interpretation of the conclusions of a model in terms of the phenomenon under study.
We illustrate the distinctions between these principles with a long experiment. We will pretend that we are part of a group of people who decide to toss a coin and observe the event that the coin will fall heads up.
This event is denoted by H, and the event of tails, by T.
Initially, everyone in the group agrees that Pr{H} = ;. When asked why,
people give two reasons: Upon checking the coin construction, they believe that the two possible outcomes, heads and tails, are equally likely;
and extrapolating from past experience, they also believe that if the coin
is tossed many times, the fraction of times that heads is observed will be
close to one-half.
The equally likely interpretation of probability surfaced in the works of
Laplace in 1812, where the attempt was made to define the probability of
an event A as the ratio of the total number of ways that A could occur to
the total number of possible outcomes of the experiment. The equally
likely approach is often used today to assign probabilities that reflect some
notion of a total lack of knowledge about the outcome of a chance phenomenon. The principle requires judicious application if it is to be useful,
however. In our coin tossing experiment, for instance, merely introducing
the possibility that the coin could land on its edge (E) instantly results in
Pr(H) = Pr{T} = Pr{E} = ;.
4
I
Introduction
The next principle, the long run relative frequency interpretation of
probability, is a basic building block in modern stochastic modeling, made
precise and justified within the axiomatic structure by the law of large
numbers. This law asserts that the relative fraction of times in which an
event occurs in a sequence of independent similar experiments approaches, in the limit, the probability of the occurrence of the event on any
single trial.
The principle is not relevant in all situations, however. When the surgeon tells a patient that he has an 80-20 chance of survival, the surgeon
means, most likely, that 80 percent of similar patients facing similar
surgery will survive it. The patient at hand is not concerned with the long
run, but in vivid contrast, is vitally concerned only in the outcome of his,
the next, trial.
Returning to the group experiment, we will suppose next that the coin is
flipped into the air and, upon landing, is quickly covered so that no one can
see the outcome. What is Pr{H} now? Several in the group argue that the
outcome of the coin is no longer random, that Pr{H} is either 0 or 1, and
that although we don't know which it is, probability theory does not apply.
Others articulate a different view, that the distinction between "random" and "lack of knowledge" is fuzzy, at best, and that a person with a
sufficiently large computer and sufficient information about such factors
as the energy, velocity, and direction used in tossing the coin could have
predicted the outcome, heads or tails, with certainty before the toss.
Therefore, even before the coin was flipped, the problem was a lack of
knowledge and not some inherent randomness in the experiment.
In a related approach, several people in the group are willing to bet with
each other, at even odds, on the outcome of the toss. That is, they are willing to use the calculus of probability to determine what is a fair bet, without considering whether the event under study is random or not. The usefulness criterion for judging a model has appeared.
While the rest of the mob were debating "random" versus "lack of
knowledge," one member, Karen, looked at the coin. Her probability for
heads is now different from that of everyone else. Keeping the coin covered, she announces the outcome "Tails," whereupon everyone mentally
assigns the value Pr{H} = 0. But then her companion, Mary, speaks up
and says that Karen has a history of prevarication.
The last scenario explains why there are horse races; different people
assign different probabilities to the same event. For this reason, probabil-
1.
Stochastic Modeling
5
ities used in odds making are often called subjective probabilities. Then,
odds making forms the third principle for assigning probability values in
models and for interpreting them in the real world.
The modern approach to stochastic modeling is to divorce the definition
of probability from any particular type of application. Probability theory
is an axiomatic structure (see Section 2.8), a part of pure mathematics. Its
use in modeling stochastic phenomena is part of the broader realm of science and parallels the use of other branches of mathematics in modeling
deterministic phenomena.
To be useful, a stochastic model must reflect all those aspects of the
phenomenon under study that are relevant to the question at hand. In addition, the model must be amenable to calculation and must allow the deduction of important predictions or implications about the phenomenon.
1.1.
Stochastic Processes
A stochastic process is a family of random variables X where t is a parameter running over a suitable index set T. (Where convenient, we will
write X(t) instead of X,.) In a common situation, the index t corresponds
to discrete units of time, and the index set is T = {0, 1, 2, . . .}. In this
case, X, might represent the outcomes at successive tosses of a coin, repeated responses of a subject in a learning experiment, or successive ob-
servations of some characteristics of a certain population. Stochastic
processes for which T = [0, c) are particularly important in applications.
Here t often represents time, but different situations also frequently arise.
For example, t may represent distance from an arbitrary origin, and X, may
count the number of defects in the interval (0, t] along a thread, or the
number of cars in the interval (0, t] along a highway.
Stochastic processes are distinguished by their state space, or the range
of possible values for the random variables X by their index set T, and by
the dependence relations among the random variables X,. The most widely
used classes of stochastic processes are systematically and thoroughly
presented for study in the following chapters, along with the mathematical techniques for calculation and analysis that are most useful with these
processes. The use of these processes as models is taught by example.
Sample applications from many and diverse areas of interest are an integral part of the exposition.
6
2.
I
Introduction
Probability Review*
This section summarizes the necessary background material and establishes the book's terminology and notation. It also illustrates the level of
the exposition in the following chapters. Readers who find the major part
of this section's material to be familiar and easily understood should have
no difficulty with what follows. Others might wish to review their probability background before continuing.
In this section statements frequently are made without proof. The
reader desiring justification should consult any elementary probability
text as the need arises.
2.1.
Events and Probabilities
The reader is assumed to be familiar with the intuitive concept of an event.
(Events are defined rigorously in Section 2.8, which reviews the axiomatic
structure of probability theory.)
Let A and B be events. The event that at least one of A or B occurs is
called the union of A and B and is written A U B; the event that both occur
is called the intersection of A and B and is written A fl B, or simply AB.
This notation extends to finite and countable sequences of events. Given
events A A2, ... , the event that at least one occurs is written A, U A, U
= U i A;, the event that all occur is written A, fl A, fl
= f1;=, A;.
The probability of an event A is written Pr{A }. The certain event, denoted by (1, always occurs, and Pr{ Cl } = 1. The impossible event, denoted by 0, never occurs, and Pr{O} = 0. It is always the case that 0 s
Pr(A) : 1 for any event A.
Events A, B are said to be disjoint if A fl B = 0; that is, if A and B
cannot both occur. For disjoint events A, B we have the addition law
Pr{A U B) = Pr(A) + Pr{B}. A stronger form of the addition law is as
follows: Let A,, A,, . . . be events with A; and A; disjoint whenever i 0 j.
Then Pr(U , A; ) = Ex , Pr{A;}. The addition law leads directly to the law
* Many readers will prefer to omit this review and move directly to Chapter III, on
Markov chains. They can then refer to the background material that is summarized in the
remainder of this chapter and in Chapter II only as needed.
2.
Probability Review
7
be disjoint events for which fl = A, U A,
U .... Equivalently, exactly one of the events A A,. . . will occur. The law
of total probability asserts that Pr{B} = 7-;_, Pr(B fl A;} for any event B.
of total probability: Let A,, A,, .
. .
The law enables the calculation of the probability of an event B from
the sometimes more easily determined probabilities Pr{B fl A;}, where i
= 1, 2, .... Judicious choice of the events A; is prerequisite to the profitable application of the law.
Events A and B are said to be independent if Pr{A fl B) = Pr{A} X
Pr{B}. Events A,, A,, ... are independent if
Pr{A;1 fl A;, fl ... fl A; } = Pr(A;d Pr{A;,} ... Pr{A; }
for every finite set of distinct indices i i,, .
2.2
.
.
, i,,.
Random Variables
An old-fashioned but very useful and highly intuitive definition describes
a random variable as a variable that takes on its values by chance. In Section 2.8, we sketch the modern axiomatic structure for probability theory
and random variables. The older definition just given serves quite adequately, however, in virtually all instances of stochastic modeling. Indeed,
this older definition was the only approach available for well over a century of meaningful progress in probability theory and stochastic
processes.
Most of the time we adhere to the convention of using capital letters
such as X, Y, Z to denote random variables, and lowercase letters such as
x, y, z for real numbers. The expression (X:5 x) is the event that the random variable X assumes a value that is less than or equal to the real number x. This event may or may not occur, depending on the outcome of the
experiment or phenomenon that determines the value for the random variable X. The probability that the event occurs is written Pr{X <- x). Allowing x to vary, this probability defines a function
F(x) = Pr{X
x},
-oo < x < +o,
called the distribution function of the random variable X. Where several
random variables appear in the same context, we may choose to distinguish their distribution functions with subscripts, writing, for example,
FX(6) = Pr(X --- ) and F,.(6) = Pr{ Y
6}, defining the distribution
8
I
Introduction
functions of the random variables X and Y, respectively, as functions of
the real variable 6.
The distribution function contains all the information available about a
random variable before its value is determined by experiment. We have,
for instance, Pr{X > a} = 1 - F(a), Pr{a < X <_ b} = F(b) - F(a), and
Pr{X = x} = F(x) - lim.10 F(x - e) = F(x) - F(x -).
A random variable X is called discrete if there is a finite or denumerable
set of distinct values x,, x2, ... such that a, = Pr{X = x; } > 0 for i = 1, 2,
... and Y a; = 1. The function
p(x,) = px(x;) = a;
for i = 1, 2,
...
(2.1)
is called the probability mass function for the random variable X and is related to the distribution function via
p(x;) = F(x,) - F(x; -) and F(x)
p(x;).
xi x
The distribution function for a discrete random variable is a step function,
which increases only in jumps, the size of the jump at x; being p(x;).
If Pr{X = x} = 0 for every value of x, then the random variable X
is called continuous and its distribution function F(x) is a continuous
function of x. If there is a nonnegative function f(x) = fx(x) defined for
-- < x < - such that
n
Pr{a<X<_b} = Jf(x)dx
for -oo<a<b< co,
(2.2)
a
then f(x) is called the probability density function for the random variable
X. If X has a probability density function f(x), then X is continuous and
F(x)
f(6) de,
-o-<x<-.
If F(x) is differentiable in x, then X has a probability density function
given by
f(x) = d F(x) = F'(x),
-00 < x < 00..
(2.3)
In differential form, (2.3) leads to the informal statement
Pr { x < X < x + dx } = F(x + dx) - F(x) = dF(x) = f (x) dx. (2.4)
2.
Probability Review
9
We consider (2.4) to be a shorthand version of the more precise statement
Pr{x<X<-x+Ax} =f(x)L x+o(Ox),
Ax 10,
(2.5)
where o(Ax) is a generic remainder term of order less than Ox as Ax 10.
That is, o(1x) represents any term for which lima,10 o(Ax)/Ax = 0. By the
fundamental theorem of calculus, equation (2.5) is valid whenever the
probability density function is continuous at x.
While examples are known of continuous random variables that do not
possess probability density functions, they do not arise in stochastic models
of common natural phenomena.
2.3.
Moments and Expected Values
If X is a discrete random variable, then its mth moment is given by
E[X",]
x' Pr(X = x;},
(2.6)
[where the x; are specified in (2.1)] provided that the infinite sum converges absolutely. Where the infinite sum diverges, the moment is said not
to exist. If X is a continuous random variable with probability density
functionf(x), then its mth moment is given by
E[X"]
= J x";f(x) dx,
(2.7)
provided that this integral converges absolutely.
The first moment, corresponding to m = 1, is commonly called the
mean or expected value of X and written m, or µX. The mth central moment of X is defined as the mth moment of the random variable X - tt,
provided that p exists. The first central moment is zero. The second central moment is called the variance of X and written oX or Var[X]. We have
the equivalent formulas Var[X] = E[(X - µ)2] = E[X2] - Al.
The median of a random variable X is any value v with the property that
Pr{X ? v) ?
and
Pr{X :s v} ? Z.
If X is a random variable and g is a function, then Y = g(X) is also a
10
I
Introduction
random variable. If X is a discrete random variable with possible values
then the expectation of g(X) is given by
x x2,
....
E[g(X)] _ Y, g(x;) Pr{X = x;},
(2.8)
provided that the sum converges absolutely. If X is continuous and has the
probability density function fx, then the expected value of g(X) is evaluated from
E[g(X)] = J g(x)fx(x) dx.
(2.9)
The general formula, covering both the discrete and continuous cases, is
E[g(X)] = f g(x) dFx(x),
(2.10)
where Fx is the distribution function of the random variable X. Technically
speaking, the integral in (2.10) is a Lebesgue-Stieltjes integral. We do not
require knowledge of such integrals in this text, but interpret (2.10) to sig-
nify (2.8) when X is a discrete random variable, and to represent (2.9)
when X possesses a probability density fx.
Let F,(y) = Pr{ Y <_ y} denote the distribution function for Y = g(X).
When X is a discrete random variable, then
E[Y] _
_
y; Pr{Y = y;}
g(x;) Pr{X = x;}
if y; = g(x;) and provided that the second sum converges absolutely. In
general,
E[Y] = JydF)(y)
(2.11)
J g(x) dFx(x)
If X is a discrete random variable, then so is Y = g(X). It may be, however, that X is a continuous random variable while Y is discrete (the reader
should provide an example). Even so, one may compute E[Y] from either
form in (2.11) with the same result.
2.
Probability Review
2.4.
11
Joint Distribution Functions
Given a pair (X, Y) of random variables, their joint distribution function
is the function Fxr of two real variables given by
Fx,.(x, y) = F(x, y) = Pr(X <- x and Y:5 y).
Usually, the subscripts X, Y will be omitted, unless ambiguity is possible.
A joint distribution function Fxr is said to possess a (joint) probability density if there exists a functionfxr of two real variables for which
Fxr(x, y) = xf
f
'v
-l) di? d6
for all x, y.
The function Fx(x) = lim,.,x F(x, y) is a distribution function, called the
marginal distribution function of X. Similarly, F,.(y) = lim, F(x, y) is
the marginal distribution function of Y. If the distribution function F possesses the joint density function f, then the marginal density functions for
X and Y are given, respectively, by
+x
fx(x) = J f(x, y) dy and f(y)
+x
f f(x, y) dx.
If X and Y are jointly distributed, then E[X + Y] = E[X] + E[Y], provided only that all these moments exist.
Independence
If it happens that F(x, y) = FF(x) X F,.(y) for every choice of x, y, then the
random variables X and Y are said to be independent. If X and Y are inde-
pendent and possess a joint density function f(x, y), then necessarily
f(x, y) = fx(x)f,.(y) for all x, y.
Given jointly distributed random variables X and Y having means µx
and µ, and finite variances, the covariance of X and Y, written o or
Cov[X, Y], is the product moment o-xr = E[(X - µx)(Y - Ixy)] =
E[XY] - µxµy, and X and Y are said to be uncorrelated if their covariance
is zero, that is, o-xy = 0. Independent random variables having finite variances are uncorrelated, but the converse is not true; there are uncorrelated
random variables that are not independent.
12
I
Introduction
Dividing the covariance o-, by the standard deviations ox and o-, dep < + 1.
fines the correlation coefficient p = o-x,I oxo'r for which -1
The joint distribution function of any finite collection X ... , X,, of random variables is defined as the function
F(x,, ... , x,,) = Fx,.....x,,(x1, .
. .
, x,.)
= Pr(X1 :5 x1,...,X.:x
If F(x,,... , x,,) = Fx,(x,)
Fx (x,,) for all values of x,, ... , x,,, then the
random variables X . . . , X,, are said to be independent.
A joint distribution function F(x...... x,,) is said to have a probability
density function f(6,.....
if
x,
x
F(x,,...,x,) = J ... J .f( .,...,
-x
for all values of x,,
. . .
x
, x,,.
Expectation
For jointly distributed random variables X ... , X,, and arbitrary functions h,,
. . .
,
h,, of n variables each,
In
Ill
E1
h;(X,,
l=
... ,
7, E[h;(X......
l=I
provided only that all these moments exist.
2.5.
Sums and Convolutions
If X and Y are independent random variables having distribution functions
Fx and F, respectively, then the distribution function of their sum Z = X
+ Y is the convolution of Fx and F,.:
+x
Fz(z) =-xJ Fx(z - )
+x
J Fr(z
-x
- 71) dFx(71).
(2.12)
If we specialize to the situation where X and Y have the probability densi-
2.
Probability Review
13
tiesfx and f,., respectively, then the density function fz of the sum Z = X +
Y is the convolution of the densities fx and f,.:
+x
x
fz(z) = J fx(z - 17)fr(71) dq = f ff(z - 6)fx(6) d6.
(2.13)
Where X and Y are nonnegative random variables, the range of integration
is correspondingly reduced to
z
Z
fz(z) = f fx(z - 71)fr(71) d'rl = Jfr(z
0
0
d
-
for z ? 0.
(2.14)
If X and Y a r e independent and have respective variances o- and o ,
then the variance of the sum Z = X + Y is the sum of the variances: o =
o + 0-2. More generally, if X ... , X,, are independent random variables
having variances o , ... , c,2, respectively, then the variance of the sum
Z=X, +... +X is QZ= 0-; +... +o,2.
2.6.
Change of Variable
Suppose that X is a random variable with probability density function fx
and that g is a strictly increasing differentiable function. Then Y = g(X)
defines a random variable, and the event ( Y -- y) is the same as the event
{X : g-'(y)}, where g-' is the inverse function to g; i.e., y = g(x) if and
only if x = g-'(y). Thus we obtain the correspondence F,.(y) = Pr{Y : y)
= Pr{X s g-'(y)} = Fx(g-'(y)) between the distribution function of Yand
that of X. Recall the differential calculus formula
dg-'
1
1
where y = g(x),
= g(x) = dgldx
and use this in the chain rule of differentiation to obtain
dy
=
dFy(y)
dy
'
-
dFx(g-'(y))
dy
= fx(x)
1
OX),
where y = g(x)..
The formula
1
where y = g(x).
(2.15)
fx(x),
g,(x)
expresses the density function for Y in terms of the density for X when g
is strictly increasing and differentiable.
.fr(y) =
14
2.7.
I
Introduction
Conditional Probability
For any events A and B, the conditional probability of A given B is written Pr{AIB} and defined by
Pr{AIB} =
Pr{A fl B)
Pr{B}
if Pr{B} > 0,
(2.16)
and is left undefined if Pr(B) = 0. [When Pr{B} = 0, the right side of
(2.16) is the indeterminate quantity o.]
In stochastic modeling, conditional probabilities are rarely procured via
(2.16) but instead are dictated as primary data by the circumstances of the
application, and then (2.16) is applied in its equivalent multiplicative form
Pr{A fl B) = Pr{AIB} Pr{B}
(2.17)
to compute other probabilities. (An example follows shortly.) Central in
this role is the law of total probability, which results from substituting
Pr{A fl B.) = Pr{AIB;} Pr{ B;} into Pr{A} = E;=, Pr{A fl B;}, where fl =
B, U B, U
and B; fl B; = 0 if i 0 j (cf. Section 2.1), to yield
x
Pr{A} = 7, Pr{AIB;} Pr {B;}.
(2.18)
Example Gold and silver coins are allocated among three urns labeled
I, II, III according to the following table:
Urn
I
II
III
Number of
Gold Coins
Number of
Silver Coins
4
3
8
6
9
6
An urn is selected at random, all urns being equally likely, and then a coin
is selected at random from that urn. Using the notation I, II, III for the
events of selecting urns I, II, and III, respectively, and G for the event of
selecting a gold coin, then the problem description provides the following
probabilities and conditional probabilities as data:
2.
Probability Review
15
Pr{I} = ,
Pr(Gli) =
,
Pr(II) = ,
Pr{ GIII } = ,,_,
Pr{III} = ,
Pr{G1III} = 162,
and we calculate the probability of selecting a gold coin according to
(2.18), viz.
Pr{G} = Pr{GII} Pr//{I} + /Pr{GIII} Pr{II} + Pr{GjIII} Pr{III}
I_/'3)
+
I?l3\\1
+
l})
36
As seen here, more often than not conditional probabilities are given as
data and are not the end result of calculation.
Discussion of conditional distributions and conditional expectation
merits an entire chapter (Chapter II).
2.8.
Review of Axiomatic Probability Theory*
For the most part, this book studies random variables only through their
distributions. In this spirit, we defined a random variable as a variable that
takes on its values by chance. For some purposes, however, a little more
precision and structure are needed.
Recall that the basic elements of probability theory are
1. the sample space, a set fl whose elements w correspond to the possible outcomes of an experiment;
2. the family of events, a collection
of subsets A of fl: we say that
the event A occurs if the outcome w of the experiment is an element
of A; and
3. the probability measure, a function P defined on c!; and satisfying
(a)
0 = P[Q ] <_ P[A]
P[fl] = 1
for A E
(0 = the empty set)
* The material included in this review of axiomatic probability theory is not used in the
remainder of the book. It is included in this review chapter only for the sake of completeness.
16
I
Introduction
and
P[u
A,,]
r,=1
(b)
if the events A,, A2,
The triple (Il,
,
=n=1
7
(2.19)
... are disjoint; i.e., if A. fl A; = 0 when i 0 j.
P) is called a probability space.
Example When there are only a denumerable number of possible outcomes, say ft = {w,, w,, ...}, we may take 9i to be the collection of all
subsets of fl. If p,, p2, ... are nonnegative numbers with Y-,, p,, = 1, the
assignment
P[A] = Z, p,
w,EA
determines a probability measure defined on.
It is not always desirable, consistent, or feasible to take the family of
events as the collection of all subsets of (Z. Indeed, when fl is nondenumerably infinite, it may not be possible to define a probability measure on
the collection of all subsets maintaining the properties of (2.19). In whatever way we prescribe such that (2.19) holds, the family of events
should satisfy
(a) 0 is in
and f is in
;
(b) A` is in whenever A is in 9, where A`
w E fl; w (4 A) (2.20)
is the complement of A; and
A,, is in whenever A,, is in
for n = 1, 2, ... .
(c)
A collection of subsets of a set fl satisfying (2.20) is called a o-algebra. If is a o --algebra, then
_(
i
C
J A,-,
n1
....
whenever A,, is in
for n = 1, 2,
in
Manifestly, as a consequence we find that finite unions and finite intersections of members of ?
are maintained in .
In this framework, a real random variable X is a real-valued function
defined on fl fulfilling certain "measurability" conditions given here. The
distribution function of the random variable X is formally given by
Probability Review
2.
17
Pr{a<X<_b} =P[{co;a<X(c)):!5 b}].
(2.21)
In words, the probability that the random variable X takes a value in (a, b]
is calculated as the probability of the set of outcomes w for which a <
X(w) : b. If relation (2.21) is to have meaning, X cannot be an arbitrary
function on fl, but must satisfy the condition that
{ w; a < X(w) : b) is in
for all real a < b,
since embodies the only sets A for which P[A] is defined. In fact, by exploiting the properties (2.20) of the o-algebra , we find that it is enough
to require
{ w; X(w) <_ x} is in g for all real x.
Let s4 be any o--algebra of subsets of f1. We say that X is measurable with
respect to .si, or more briefly .A-measurable, if
{ w; X(w) < x} is in
.
for all real x.
Thus, every real random variable is by definition 9;-measurable. There
may, in general, be smaller o-algebras with respect to which X is also
measurable.
The a -algebra generated by a random variable X is defined to be the
smallest o--algebra with respect to which X is measurable. It is denoted by
i(X) and consists exactly of those sets A that are in every o--algebra A for
which X is A-measurable. For example, if X has only denumerably many
possible values x,, x2, . . . , the sets
i= 1,2,...,
A;= {w;X(w)=x;},
form a countable partition of f1, i.e.,
f1 = U A;,
i=I
and
A;fA;=0
ifi*j,
and then P(X) includes precisely 0, f1, and every set that is the union of
some of the A;'s.
Example For the reader completely unfamiliar with this framework,
the following simple example will help illustrate the concepts. The exper-
18
I
Introduction
iment consists in tossing a nickel and a dime and observing "heads" or
"tails." We take SZ to be
fl = { (H, H), (H, T), (T, H), (T, T)),
where, for example, (H, T) stands for the outcome "nickel = heads, and
dime = tails." We will take the collection of all subsets of fl as the family of events. Assuming each outcome in ft to be equally likely, we arrive
at the probability measure:
AEI
0
AE?
n
P[A]
{ (H, T), (T, H), (T, T))
((H, H), (T, H), (T, T))
{ (H, H), (H, T), (T, T))
((H, H), (H, T), (T, H))
{ (T, H), (T, T))
{(H, T), (T, T)}
{ (H, T), (T, H))
{(H, H)}
{(H, T))
{ (T, H))
{(T, T)}
{ (H, H), (H, T) }
{ (H, H), (T, H))
{ (H, H), (T, T))
The event "nickel is heads" is ((H, H), (H, T)) and has, according to the
table, probability , as it should.
Let X,, be 1 if the nickel is heads, and 0 otherwise; let X, be the corresponding random variable for the dime; and let Z = X,, + Xd be the total
number of heads. As functions on fl, we have
Z
CO E'l
(H, H)
(H, T)
(T, H)
(T, T)
X1(w)
Z(w)
1
1
2
1
0
1
0
0
1
1
0
0
X"(w)
Finally, the a=algebras generated by X,, and Z are
0, Il, {(H, H), (H, T)), {(T, H), (T, T)},
and
9,(Z) _ 0, fl, ((H, H)), {(H, T), (T, H)), {(T, T)},
{(H, T), (T, H), (T, T)), {(H, H), (T, T)},
((H, H), (H, T), (T, H) }.
Exercises
19
contains four sets and &(Z) contains eight. Is X,, measurable with
respect to 9;(Z), or vice versa?
Every pair X, Y of random variables determines a o--algebra called the
o --algebra generated by X, Y. It is the smallest a-algebra with respect to
which both X and Y are measurable. This or-algebra comprises exactly
those sets A that are in every or algebra si for which X and Y are both .measurable. If both X and Y assume only denumerably many possible values, say x,, x,, . . . and y,, y,, ... , respectively, then the sets
A;; = {w; X(co) = xi, Y(w) = y;},
i, j = 1, 2, ... ,
present a countable partition of 11, and (X, Y) consists precisely of 0,
fl, and every set that is the union of some of the A,1's. Observe that X is
measurable with respect to 5;(X, Y), and thus (X) C &(X, Y).
More generally, let {X(t ); t E T J be any family of random variables.
Then the o --algebra generated by (X(t); t E T) is the smallest
with respect to which every random variable X(t), t E T, is measurable. It
is denoted by {X(t); t E T).
A special role is played by a distinguished o--algebra of sets of real
numbers. The o --algebra of Borel sets is the o--algebra generated by the
identity function f(x) = x, for x E (-cc, cc). Alternatively, the o --algebra
of Borel sets is the smallest o--algebra containing every interval of the
form (a, b], -x < a :5 b < +cc. A real-valued function of a real variable
is said to be Borel measurable if it is measurable with respect to the o,--algebra of Borel sets.
Exercises
2.1. Let A and B be arbitrary, not necessarily disjoint, events. Use the
law of total probability to verify the formula
Pr{A} = Pr{AB} + Pr{AB`},
where B` is the complementary event to B. (That is, B` occurs if and only
if B does not occur.)
2.2. Let A and B be arbitrary, not necessarily disjoint, events. Establish
the general addition law
Pr(A U B} = Pr{A} + Pr{B} - Pr{AB}.
20
I
Introduction
Hint: Apply the result of Exercise 2.1 to evaluate Pr{AB'} =
Pr(A) - Pr{AB}. Then apply the addition law to the disjoint events AB
and AB`*, noting that A = (AB) U (AB`).
2.3.
(a) Plot the distribution function
for x s 0,
for 0 < x < 1,
10
F(x) = x;
forx? 1.
1
(b) Determine the corresponding density function f(x) in the three re-
gions (i) x : 0, (ii) 0 < x < 1, and (iii) 1 < x.
(c) What is the mean of the distribution?
(d) If X is a random variable following the distribution specified in (a),
evaluate Pr{; X <_}.
2.4. Let Z be a discrete random variable having possible values 0, 1, 2,
and 3 and probability mass function
P(0) = ;,
p(2) =
P(l) = Z,
p(3) = x.
(a) Plot the corresponding distribution function.
(b) Determine the mean E[Z].
(c) Evaluate the variance Var[Z].
2.5. Let A, B, and C be arbitrary events. Establish the addition law
Pr{A U B U C} = Pr{A} + Pr{B} + Pr{C} - Pr{AB}
- Pr{AC} - Pr{BC} + Pr{ABC}.
2.6. Let X and Y be independent random variables having distribution
functions F, and F, respectively.
(a) Define Z = max{X, Y} to be the larger of the two. Show that
F,(z) = FX(z)FY(z) for all z.
(b) Define W = min{X, Y} to be the smaller of the two. Show that
F,,,(w) = 1 - [I - FX(w)] [ 1 - F,.(w)] for all w.
Problems
2.7.
21
Suppose X is a random variable having the probability density
function
f (X)
- 10
elsewhere,
where R > 0 is a fixed parameter.
(a) Determine the distribution function FX(x).
(b) Determine the mean E[X].
(c) Determine the variance Var[X].
2.8.
A random variable V has the distribution function
10
F(v)= 1-(1-v)A
forv < 0,
for 0<vs1,
forv > 1,
1
where A > 0 is a parameter. Determine the density function, mean, and
variance.
2.9. Determine the distribution function, mean, and variance corresponding to the triangular density.
X
,f(x)= 2-x
10
2.10.
for 0 <- x
1,
forl
elsewhere.
Let 1(A) be the indicator random variable associated with an
event A, defined to be one if A occurs, and zero otherwise. Define A`, the
complement of event A, to be the event that occurs when A does not occur.
Show
(a) 1{A`} = 1 - 1{A}.
(b) 1{A fl B) = 1(A)1{B} = min{1{A}, 1{B}}.
(c) 1{A U B) = max(1{A}, 11B}).
Problems
Thirteen cards numbered 1, ... , 13 are shuffled and dealt one at a
time. Say a match occurs on deal k if the kth card revealed is card number
2.1.
22
I
Introduction
k. Let N be the total number of matches that occur in the thirteen cards.
Determine E[N].
Hint: Write N = 1 (A,) +
match occurs on deal k.
+1
where Ak is the event that a
2.2. Let N cards carry the distinct numbers x ... , x,,. If two cards are
drawn at random without replacement, show that the correlation coefficient p between the numbers appearing on the two cards is - lI(N - 1).
2.3. A population having N distinct elements is sampled with replacement. Because of repetitions, a random sample of size r may contain
fewer than r distinct elements. Let S, be the sample size necessary to get
r distinct elements. Show that
E[S,]=N(1 +
1
N N-1
+...+
N - r+l ).
1
2.4. A fair coin is tossed until the first time that the same side appears
twice in succession. Let N be the number of tosses required.
(a) Determine the probability mass function for N.
(b) Let A be the event that N is even and B be the event that N : 6.
Evaluate Pr{A}, Pr{B}, and Pr{AB}.
2.5. Two players, A and B, take turns on a gambling machine until one
of them scores a success, the first to do so being the winner. Their probabilities for success on a single play are p for A and q for B, and successive
plays are independent.
(a) Determine the probability that A wins the contest given that A plays
first.
(b) Determine the mean number of plays required, given that A wins.
2.6. A pair of dice is tossed. If the two outcomes are equal, the dice are
tossed again, and the process repeated. If the dice are unequal, their sum
is recorded. Determine the probability mass function for the sum.
2.7. Let U and W be jointly distributed random variables. Show that U
and W are independent if
Pr{U>uand W>w} =Pr{U>u} Pr{W>w}
forallu,w.
23
Problems
2.8. Suppose X is a random variable with finite mean µ and variance 0-',
and Y = a + bX for certain constants a, b ± 0. Determine the mean and
variance for Y.
2.9.
Determine the mean and variance for the probability mass function
2(n - k)
p(k) = n(n - 1 )
for k = 1, 2, ... , n.
Random variables X and Y are independent and have the probability mass functions
2.10.
Px(0) =
Pv(l) = b,
Px(3) = >,
Pr(2) = 3,
Ps-(3) = ;.
Determine the probability mass function of the sum Z = X + Y.
2.11. Random variables U and V are independent and have the probability mass functions
Pu(0) = ,,
Pu(l) = ,
P,(l) = i,
p,(2) = .
pu(2) = ,
Determine the probability mass function of the sum W = U + V.
2.12. Let U, V, and W be independent random variables with equal variances o.'-. Define X = U + W and Y = V - W. Find the covariance between X and Y.
2.13.
Let X and Y be independent random variables each with the uniform probability density function
f(x)
_
-
I01
for0<x<l,
elsewhere.
Find the joint probability density function of U and V, where U =
max{X, Y} and V = min{X, Y}.
24
3.
I
Introduction
The Major Discrete Distributions
The most important discrete probability distributions and their relevant
properties are summarized in this section. The exposition is brief, since
most readers will be familiar with this material from an earlier course in
probability.
3.1.
Bernoulli Distribution
A random variable X following the Bernoulli distribution with parameter
p has only two possible values, 0 and 1, and the probability mass function
is p(l) = p and p(O) = 1 - p, where 0 < p < 1. The mean and variance
are E[X] = p and Var[X] = p(1 - p), respectively.
Bernoulli random variables occur frequently as indicators of events.
The indicator of an event A is the random variable
1(A) = 1A = {0
if A occurs,
if A does not occur.
(3.1)
Then 1A is a Bernoulli random variable with parameter p = E[lA] _
Pr{A}.
The simple expedient of using indicators often reduces formidable calculations into trivial ones. For example, let a a2, ... , a,, be arbitrary real
numbers and A, , A2, ... , A,, be events, and consider the problem of showing that
n
it
17 a;aj Pr(A; fl Aj) ? 0.
(3.2)
i=tj=t
Attacked directly, the problem is difficult. But bringing in the indicators
1(A) and observing that
ail(A) }
0
It
n
a.1(A)}{
lI
ajl(A)}
n
_ Y Y a aj1(A;)1(A) = Y 7, a;aj1(A, fl Aj)
1=1j=I
i=1j=1
3.
25
The Major Discrete Distributions
gives, after taking expectations,
it
n
0
-,
E{{>
_
a,1(A)} ] = 17 ara;E[uA, n A;)]
i=i;=I
a;a; Pr{A; f1 A;},
t=I j=I
and the demonstration of (3.2) is complete.
3.2.
Binomial Distribution
Consider independent events A A2, .... A,,, all having the same probability p = Pr{A;} of occurrence. Let Y count the total number of events
among A ... , A that occur. Then Y has a binomial distribution with parameters n and p. The probability mass function is
py(k) = Pr{Y = k}
n!
= k!(n - k)!
(3.3)
for k = 0,1,..., n.
Pk(1 - P)11-k
Writing Y as a sum of indicators in the form Y = 1(A,) +
+
makes it easy to determine the moments
+
E[Y] = E[1(A,)] +
np,
and using independence, we can also determine that
Var[Y] = Var[1(A,)] +
+
np(1 - p).
Briefly, we think of a binomial random variable as counting the number
of "successes" in n independent trials where there is a constant probability p of success on any single trial.
3.3.
Geometric and Negative Binomial Distributions
Let A A,, . .. be independent events having a common probability p =
Pr{A;} of occurrence. Say that trial k is a success (S) or failure (F)
26
I
Introduction
according as A, occurs or not, and let Z count the number of failures prior
to the first success. To be precise, Z = k if and only if 1(A,) = 0, . . . ,1(Ak)
= 0, and 1(Ak+0 = 1. Then Z has a geometric distribution with parameter
p. The probability mass function is
pz(k) = p(1 - p)k
for k = 0, 1,
... ,
(3.4)
and the first two moments are
E[Z] =
Var[Z] = 1 , P
P;
P
P"
Sometimes the term "geometric distribution" is used in referring to the
probability mass function
1
for k = 1, 2.....
Pz,(k) = p(1 - p)k_I
(3.5)
This is merely the distribution of the random variable Z' = 1 + Z, the
number of trials until the first success. Hence E[Z'] = 1 + E[Z] = 11p,
and Var[Z'] = Var[Z] = (1 - p)lp2.
Now fix an integer r ? I and let W, count the number of failures observed before the rth success in A, A,, .... Then W has a negative binominal distribution with parameters r and p. The event W,. = k calls for
(A) exactly r - 1 successes in the first k + r - 1 trials, followed by, (B)
a success on trial k + r. The probability for (A) is obtained from a binomial distribution, and the probability for (B) is simply p, which leads to
the following probability mass function for W,.:
p(k) = Pr{ W, = k) _
(r - -
1)!
p,(l - p)k,
k = 0, 1,
....
(3.6)
+ Z where Z,
Another way of writing W, is as the sum W,. = Z, +
Z,_ are independent random variables each having the geometric distribution of (3.4). This formulation readily yields the moments
E[W,.] _
r(1 - p)
,
P
3.4.
Var[W,.] =
r(1 - p)
,
(3.7)
"
The Poisson Distribution
If distributions were graded on a scale of one to ten, the Poisson clearly
merits a ten. It plays a role in the class of discrete distributions that paral-
3.
The Major Discrete Distributions
27
lels in some sense that of the normal distribution in the continuous class.
The Poisson distribution occurs often in natural phenomena, for powerful
and convincing reasons (the law of rare events, see later in this section).
At the same time, the Poisson distribution has many elegant and surprising mathematical properties that make analysis a pleasure.
The Poisson distribution with parameter k > 0 has the probability mass
function
Ake-a
p(k) =
fork = 0, 1,
ki
....
(3.8)
Using this series expansion
A2
A3
+
e" = 1 + A +
2!
+
(3.9)
3!
we see that Ekzo p(k) = 1. The same series helps calculate the mean via
x
x
kp(k)=>k
k=0
k=1
kke-T
k!
Ak-1
=Ae -k
k=1 (k - 1)!
A.
The same trick works on the variance, beginning with
x
x
k=0
Ake-.1
k(k - 1)
k(k - 1)p(k)
k=2
k!
= Ate-a Y
/ll k-2
= A2.
k=2 (k - 2)!
Written in terms of a random variable X having the Poisson distribution
with parameter A, we have just calculated E[X] = A and E[X(X - 1)] =
A2, whence E[X 2] = E[X(X - 1)] + E[X] = A2 + A and Var[X] = E[X 2]
- {E[X]}2 = A. That is, the mean and variance are both the same and
equal to the parameter A of the Poisson distribution.
The simplest form of the law of rare events asserts that the binomial
distribution with parameters n and p converges to the Poisson with parameter A if n - oo and p -+ 0 in such a way that A = np remains constant.
In words, given an indefinitely large number of independent trials, where
success on each trial occurs with the same arbitrarily small probability,
then the total number of successes will follow, approximately, a Poisson
distribution.
The proof is a relatively simple manipulation of limits. We begin by
writing the binomial distribution in the form
28
Introduction
I
Pr{X = k} =
n!
p)"-k
k!(n - k)i pk(1 -
=n(n-1)...(n-k+1)pk(l-p)
k!(1 - p)k
and then substitute p = a/n to get
-
A)"
(n)k(1
Pr{X=k} =n(n-
1)
1
k
n
n
n
n
nll
(1
)
-n
->
n
1
k!(1
-)
n
asn-4 oo ;
e-x
as n -> 00;
and
k
asn-3
to obtain the Poisson distribution
Ake`
Pr{X=k} =
k!
;
fork= 0, 1,.:.
in the limit. Extended forms of the law of rare events are presented in V.
Example
You Be the Judge In a purse-snatching incident, a woman
described her assailant as being seven feet tall and wearing an orange hat,
red shirt, green trousers, and yellow shoes. A short while later and a few
blocks away a person fitting that description was seen and charged with
the crime.
3.
The Major Discrete Distributions
29
In court, the prosecution argued that the characteristics of the assailant
were so rare as to make the evidence overwhelming that the defendant
was the criminal.
The defense argued that the description of the assailant was rare, and
that therefore the number of people fitting the description should follow a
Poisson distribution. Since one person fitting the description was found,
the best estimate for the parameter is A = 1. Finally, they argued that the
relevant computation is the conditional probability that there is at least
one other person at large fitting the description given that one was observed. The defense calculated
Pr{X?2IX? 1} =
1 -Pr(X=0) - Pr{X= 1 )
1 - Pr{X = 01
1-e-'-e-'
1 - e-'
= 0.4180,
and since this figure is rather large, they argued that the circumstantial evidence arising out of the unusual description was too weak to satisfy the
"beyond a reasonable doubt" criterion for guilt in criminal cases.
3.5.
The Multinomial Distribution
This is a joint distribution of r variables in which only nonnegative integer values 0, . . . , n are possible. The joint probability mass function is
Pr{X1 =k...... X,.=k,.}
n!
=
k!
1
10
r
if k, +...+kr=n,
(3.10)
otherwise,
where
Some moments are E[X;] = np;, Var[X,] = np;(1 - p;), and Cov[X;X;] _
-np,pi.
The multinomial distribution generalizes the binomial. Consider an experiment having a total of r possible outcomes, and let the corresponding
probabilities be p, , ... , Pr, respectively. Now perform n independent
30
1
Introduction
replications of the experiment and let X; record the total number of times
that the ith type outcome is observed in the n trials. Then X...... X, has
the multinomial distribution given in (3.10).
Exercises
Consider tossing a fair coin five times and counting the total
3.1.
number of heads that appear. What is the probability that this total is
three?
3.2. A fraction p = 0.05 of the items coming off a production process
are defective. If a random sample of 10 items is taken from the output of
the process, what is the probability that the sample contains exactly one
defective item? What is the probability that the sample contains one or
fewer defective items?
3.3. A fraction p = 0.05 of the items coming off of a production process
are defective. The output of the process is sampled, one by one, in a random manner. What is the probability that the first defective item found is
the tenth item sampled?
3.4. A Poisson distributed random variable X has a mean of A = 2.
What is the probability that X equals 2? What is the probability that X is
less than or equal to 2?
3.5.
The number of bacteria in a prescribed area of a slide containing a
sample of well water has a Poisson distribution with parameter 5. What is
the probability that the slide shows 8 or more bacteria?
3.6. The discrete uniform distribution on { 1,
... , n) corresponds to the
probability mass function
1
p(k) =
n
0
for k = 1,...,n,
elsewhere.
(a) Determine the mean and variance.
(b) Suppose X and Y are independent random variables, each having
Problems
31
the discrete uniform distribution on 10, . . . , n}. Determine the
probability mass function for the sum Z = X + Y.
(c) Under the assumptions of (b), determine the probability mass function for the minimum U = min{X, Y).
Problems
3.1. Suppose that X has a discrete uniform distribution on the integers
0, 1, ... , 9, and Y is independent and has the probability distribution
Pr{Y = k} = ak for k = 0, 1, .... What is the distribution of Z = X + Y
(mod 10), their sum modulo 10?
3.2.
The mode of a probability mass function p(k) is any value k* for
which p(k*) >_ p(k) for all k. Determine the mode(s) for
(a) The Poisson distribution with parameter A > 0.
(b) The binomial distribution with parameters n and p.
3.3.
Let X be a Poisson random variable with parameter A. Determine
the probability that X is odd.
3.4. Let U be a Poisson random variable with mean µ. Determine the
expected value of the random variable V = 1/(1 + U).
3.5. Let Y = N - X where X has a binomial distribution with parameters N and p. Evaluate the product moment E[XY] and the covariance
Cov[X, Y].
3.6. Suppose (X,, X,, )(,) has a multinomial distribution with parameters
M and ir, > 0 for i = 1, 2, 3, with 7r,+7r,+ir,= 1.
(a) Determine the marginal distribution for X,.
(b) Find the distribution for N = X, + X,.
(c) What is the conditional probability Pr{X1 = kIN = n} for
0<_k--- n?
3.7. Let X and Y be independent Poisson distributed random variables
having means µ and v, respectively. Evaluate the convolution of their
mass functions to determine the probability distribution of their sum Z =
X+Y.
32
I
Introduction
3.8.
Let X and Y be independent binomial random variables having parameters (N, p) and (M, p), respectively. Let Z = X + Y.
(a) Argue that Z has a binomial distribution with parameters (N + M, p)
by writing X and Y as appropriate sums of Bernoulli random variables.
(b) Validate the result in (a) by evaluating the necessary convolution.
3.9. Suppose that X and Y are independent random variables with the
geometric distribution
p(k) = (1 - ir)iTk
for k = 0, 1,
... .
Perform the appropriate convolution to identify the distribution of Z = X
+ Y as a negative binomial.
3.10. Determine numerical values to three decimal places for
Pr{X = k},k= 0, 1,2,when
(a) X has a binomial distribution with parameters n = 10 and p = 0.1.
(b) X has a binomial distribution with parameters n = 100 and p =
0.01.
(c) X has a Poisson distribution with parameter A = 1.
3.11. Let X and Y be independent random variables sharing the geometric distribution whose mass function is
p(k) = (1 - ir)irk
for k = 0, 1,
... ,
where 0<n<1.Let U=min{X,Y},V=max{X,Y}, and W = V - U.
Determine the joint probability mass function for U and W and show that
U and W are independent.
3.12.
Suppose that the telephone calls coming into a certain switch-
board during a one-minute time interval follow a Poisson distribution with
mean A = 4. If the switchboard can handle at most 6 calls per minute,
what is the probability that the switchboard will receive more calls than it
can handle during a specified one-minute interval?
3.13. Suppose that a sample of 10 is taken from a day's output of a machine that produces parts of which 5 percent are normally defective. If 100
Important Continuous Distributions
4.
33
percent of a day's production is inspected whenever the sample of 10
gives 2 or more defective parts, then what is the probability that 100 percent of a day's production will be inspected? What assumptions did you
make?
3.14.
Suppose that a random variable Z has the geometric distribution
Pz(k) = p(1 - p)k
for k = 0, 1, ...
,
where p = 0.10.
(a) Evaluate the mean and variance of Z.
(b) What is the probability that Z strictly exceeds 10?
3.15.
Suppose that X is a Poisson distributed random variable with
mean A = 2. Determine Pr(X <- A).
3.16. Consider the generalized geometric distribution defined by
Pk = b(l - p)k
for k = 1, 2, ...
,
and
x
Po=1-7Pk,
k=1
where 0 < p < 1 and p <- b <- p/(1 - p).
(a) Evaluate po in terms of b and p.
(b) What does the generalized geometric distribution reduce to when
b=p?When b=pl(1 -p)?
(c) Show that N = X + Z has the generalized geometric distribution
when X is a Bernoulli random variable for which Pr{X = 11 = a,
0 < a < 1, and Z independently has the usual geometric distribution given in (3.4).
4.
Important Continuous Distributions
For future reference, this section catalogs several continuous distributions
and some of their properties.
34
I
4.1.
Introduction
The Normal Distribution
The normal distribution with parameters µ and o' > 0 is given by the familiar bell-shaped probability density function
(,xµ, Q-') =
µ, / 2?'
e ,.
,
1
-x < x < x.
(4.1)
The density function is symmetric about the point F.c, and the parameter o'
is the variance of the distribution. The case µ = 0 and o' = 1 is referred
to as the standard normal distribution. If X is normally distributed with
mean µ and variance o', then Z = (X - µ)/o- has a standard normal distribution. By this means, probability statements about arbitrary normal
random variables can be reduced to equivalent statements about standard
normal random variables. The standard normal density and distribution
functions are given respectively by
b() =
e-F212
-oc < 6 < 00,
(4.2)
c(x) = j (P(6) d6,
-x < x < x.
(4.3)
1
and
The central limit theorem explains in part the wide prevalence of the
normal distribution in nature. A simple form of this aptly named result con-
+ ,, of independent and identically
distributed summands ,, 62, ... having finite means µ = E[&k] and finite
In this case, the central limit theorem asserts that
variances cr- =
cerns the partial sums S,, = , +
lim Prj
II- %
I
S,,
7L
n
x}
=
I (x)
for all x.
(4.4)
The precise statement of the theorem's conclusion is given by equation
(4.4). Intuition is sometimes enhanced by the looser statement that, for
large n, the sum S,, is approximately normally distributed with mean nµ
and variance no-'.
In practical terms we expect the normal distribution to arise whenever
the numerical outcome of an experiment results from numerous small additive effects, all operating independently, and where no single or small
group of effects is dominant.
4.
35
Important Continuous Distributions
The Lognormal Distribution
If the natural logarithm of a nonnegative random variable V is normally
distributed, then V is said to have a lognormal distribution. Conversely, if
X is normally distributed with mean µ and variance cr2, then V = e' defines a lognormally distributed random variable. The change-of-variable
formula (2.15) applies to give the density function for V to be
1
exp{-
V ov
1(ln v - µ )'J'
2
o,
v ? 0.
j
(4.5)
The mean and variance are, respectively,
E[V] = exp{µ + to },
(4.6)
Var[V] = exp{2(µ +
4.2.
1].
The Exponential Distribution
A nonnegative random variable T is said to have an exponential distribution with parameter A > 0 if the probability density function is
(oe
At
.fT(t) =
for t ? 0,
for t > 0.
(4.7)
The corresponding distribution function is
FT(t) = 1 - e- At
{0
fort ? 0,
fort > 0,
(4.8)
and the mean and variance are given, respectively, by
1
E[T] _
1
and
Var[T] = ,.
Note that the parameter is the reciprocal of the mean and not the mean
itself.
The exponential distribution is fundamental in the theory of continuous-time Markov chains (see V), due in major part to its memoryless prop-
erty, as now explained. Think of T as a lifetime and, given that the unit
36
1
Introduction
has survived up to time t, ask for the conditional distribution of the remaining life T - t. Equivalently, for x > 0 determine the conditional
probability Pr{T - t > xIT > t}. Directly applying the definition of conditional probability (see Section 2.7), we obtain
Pr{T-t>xIT>t} = Pr{T>t+x,T>t}
Pr{T > T}
Pr{T>t+x}
Pr{T > t}
(because x > 0)
(4.9)
e-xcr+.
[from (4.8)]
There is no memory in the sense that Pr{T - t > xIT > t} = e-a` _
Pr IT > x), and an item that has survived fort units of time has a remaining lifetime that is statistically the same as that for a new item.
To view the memoryless property somewhat differently, we introduce
the hazard rate or failure rate r(s) associated with a nonnegative random
variable S having continuous density g(s) and distribution function G(s) <
1. The failure rate is defined by
r(s) =
1
g(G(s)
for s > 0.
(4.10)
We obtain the interpretation by calculating (see Section 2.2)
s+As)
Pr{s<S5s+Osls<S} _ Pr{s<S:5
Pr{s < S}
g(s) As
+ o(As)
1 - G(s)
[from (2.5)]
= r(s) As + o(Os).
An item that has survived to time s will then fail in the interval (s, s + Os]
with conditional probability r(s) As + o(1s), thus motivating the name
"failure rate."
4.
Important Continuous Distributions
37
We can invert (4.10) by integrating
-r(s) =
_ d[ I - G(s)]Ids
-g(s)
- d{ ln[ 1 - G(s)] }
1 - G(s)
1 - G(s)
ds
to obtain
- f r(s) ds = In[ 1
- G(t)],
0
or
G(t) = 1 -expj - f r(s) ds},
l
t ? 0,
0
which gives the distribution function explicitly in terms of the hazard rate.
The exponential distribution is uniquely the continuous distribution
with the constant failure rate r(t) = A. (See Exercise 4.8 for the discrete
analog.) The failure rate does not vary in time, another reflection of the
memoryless property.
Section 5 contains several exercises concerning the exponential distribution. In addition to providing practice in relevant algebraic and calculus manipulations, these exercises are designed to enhance the reader's intuition concerning the exponential law.
4.3.
The Uniform Distribution
A random variable U is uniformly distributed over the interval [a, b],
where a < b, if it has the probability density function
1
fi (u) =
b-a
for a --- u :5 b,
0
elsewhere.
(4.11)
The uniform distribution extends the notion of "equally likely" to the continuous case. The distribution function is
10
(x)=
for a
a,
x-a
for a<x:! b,
1
for x > b,
b-a
(4.12)
38
I
Introduction
and the mean and variance are, respectively,
E[U] = 2 (a + b) and Var[U] _ (b
12 a)'
The uniform distribution on the unit interval [0, 1], for which a = 0 and
b = 1, is most prevalent.
4.4.
The Gamma Distribution
The gamma distribution with parameters a > 0 and A > 0 has probability
density function
f(x) = rA (Ax)"-'e-'
for x > 0.
(4.13)
Given an integer number a of independent exponentially distributed random variables Y,, ... , Ya having common parameter A, then their sum
XX = Y, +
+ Y. has the gamma density of (4.13), from which we obtain the moments
E[Xa] = Ci
and
Var[Xa] =
z
these moment formulas holding for noninteger a as well.
4.5.
The Beta Distribution
The beta density with parameters a > 0 and /3 > 0 is given by
+ J3) x«-
f( x)
x)
x)a- I
for0<x<1,
F(a)r(R)
(4.14)
0
elsewhere.
The mean and variance are, respectively,
E[X] =
a
a
and
Var[X] = (a +
al+
+1
(The gamma and beta functions are defined and briefly discussed in Section 6.)
4.
Important Continuous Distributions
4.6.
39
The Joint Normal Distribution
Let ox, o-v, lxx, Ar, and p be real constants subject to ox > 0, o,. > 0, and
-1 < p < 1. For real variables x, y define
f(X -
x)2
-
-
1
-µY)
(4.15)
)2}
+ (Y
The joint normal (or bivariate normal) distribution for random variables
X, Y is defined by the density function
ik(x1J ) =
x. v
1
27roxo,
1 --p2
(4.16)
x exp{- 2 Q(x, Y)}, - 00 < x, Y < 00.
The moments are
E[X]=µx,
Var[X] = oil,
E[Y]=p,
Var[Y] = Cry.
and
Cov[X, Y] = E[(X - µx)(Y - Ay)] = poxor
The dimensionless parameter p is called the correlation coefficient. When
p is positive, then positive values of X are (stochastically) associated with
positive values of Y. When p is negative, then positive values of X are associated with negative values of Y. If p = 0, then X and Y are independent
random variables.
Linear Combinations of Normally Distributed Random Variables
Suppose X and Y have the bivariate normal density (4.16), and let Z =
aX + by for arbitrary constants a, b. Then Z is normally distributed with
mean
E[Z] = aµx + bµ}
40
I
Introduction
and variance
Var[X] = a2oX + 2abpoxo + b2oy.
A random vector X...... X,,, is said to have a multivariate normal distribution, or a joint normal distribution, if every linear combination
+
a univariate normal distribution. Obvia,X, +
ously, if X,..... X has a joint normal distribution, then so does the random vector Y ... , Y,,,, defined by the linear transformation in which
forj= 1,...,m,
for arbitrary constants a;;.
Exercises
4.1. The lifetime, in years, of a certain class of light bulbs has an exponential distribution with parameter A = 2. What is the probability that a
bulb selected at random from this class will last more than 1.5 years?
What is the probability that a bulb selected at random will last exactly 1.5
years?
4.2. The median of a random variable X is any value a for which
Pr{X s a} >_ and Pr{X ? a) ? Z. Determine the median of an exponen2
tially distributed
random variable with parameter A. Compare the median
to the mean.
4.3. The lengths, in inches, of cotton fibers used in a certain mill are exponentially distributed random variables with parameter A. It is decided to
convert all measurements in this mill to the metric system. Describe the
probability distribution of the length, in centimeters, of cotton fibers in
this mill.
4.4. Twelve independent random variables, each uniformly distributed
over the interval (0, 1], are added, and 6 is subtracted from the total. Determine the mean and variance of the resulting random variable.
4.5. Let X and Y have the joint normal distribution described in equa-
tion (4.16). What value of a minimizes the variance of Z = aX +
(1 - a)Y? Simplify your result when X and Y are independent.
Problems
41
4.6.
Suppose that U has a uniform distribution on the interval [0, 1]. Derive the density function for the random variables
(a) Y = -ln(1 - U).
(b) W,, = U" for n > 1.
Hint:
Refer to Section 2.6.
4.7.
Given independent exponentially distributed random variables S
and T with common parameter A, determine the probability density function of the sum R = S + T and identify its type by name.
4.8.
Let Z be a random variable with the geometric probability mass
function
p(k) = (1 - ir)#rk,
k = 0, 1,
... ,
where 0 < ?r < 1.
(a) Show that Z has a constant failure rate in the sense that
Pr{Z=kIZ?k}=1- 7r fork=0,1,....
(b) Suppose Z' is a discrete random variable whose possible values
are 0,
.... and for which Pr (Z' = kIZ' ? k} = 1 - 7r for
.... Show that the probability mass function for Z' is p(k).
1,
k = 0, 1,
Problems
4.1. Evaluate the moment E[e '], where A is an arbitrary real number
and Z is a random variable following a standard normal distribution, by
integrating
1
1-e
212 dz.
Hint: Complete the square -Z z2 + Az = -[(z - A)2 - A2] and use
the fact that
r I e-(z -a)-/2 dz = 1.
4.2. Let W be an exponentially distributed random variable with parameter 0 and mean p. = 1/0.
(a) Determine Pr{ W > p.).
(b) What is the mode of the distribution?
42
Introduction
I
4.3.
Let X and Y be independent random variables uniformly distributed
over the interval [6 -Z, 0 + Z] for some fixed 0. Show that W = X - Yhas
a distribution that is independent of 0 with density function
1+w
fw(w)= 1-w
10
for-law<0,
for0 w<1,
forlwl> 1.
4.4. Suppose that the diameters of bearings are independent normally
distributed random variables with mean µ$ = 1.005 inch and variance o
= (0.003)2 inch2. The diameters of shafts are independent normally distributed random variables having mean µs = 0.995 inch and variance os
= (0.004)2 inch2.
w
6
Shaft
Bearing
Let S be the diameter of a shaft taken at random and let B be the diameter
of a bearing.
(a) What is the probability Pr(S > B) of interference?
(b) What is the probability of one or fewer interferences in 20 random
shaft-bearing pairs?
Hint: The clearance, defined by C = B - S, is normally distributed
(why?), and interference occurs only if C < 0.
4.5.
If X follows an exponential distribution with parameter a = 2, and
independently, Y follows an exponential distribution with parameter 8 =
3, what is the probability that X < Y?
5. Some Elementary Exercises
5.
43
Some Elementary Exercises
We have collected in this section a number of exercises that go beyond
what is usually covered in a first course in probability.
5.1.
Tail Probabilities
In mathematics, what is a "trick" upon first encounter becomes a basic
tool when familiarity through use is established. In dealing with nonnegative random variables, we can often simplify the analysis by the trick of
approaching the problem through the upper tail probabilities of the form
Pr{X > x}. Consider the following example.
A jar has n chips numbered 1, 2, . . . , n. A person draws a chip, returns
it, draws another, returns it, and so on, until a chip is drawn that has been
drawn before. Let X be the number of drawings. Find the probability distribution for X.
It is easier to compute Pr{X > k} first. Then, Pr{X > 1 } = 1, since at
least two draws are always required. The event (X > 2) occurs when dis-
tinct numbers appear on the first two draws, whence Pr{X > 2) _
(nln)[(n - 1)/n]. Continuing in this manner, we obtain
Pr(X>k)=1(1-n)(1
n...(1-kn1
)
/
fork= 1,...,n- 1.
(5.1)
Finally,
Pr{X = k} = Pr{X > k - 1) - Pr{X > k}
=r(1_n)...(1
_L(1_n)...(1
_(1
kn2)J
kn1)1
kn2)\1
n)...(1_kn2)L1-(1
kn1(1_n)...(1- k
kn1)J
n 2
fork=2,...,n+ 1.
44
1
Introduction
Now try deriving Pr(X = k) directly, for comparison with the "trick"
approach.
The usefulness of the upper tail probabilities is enhanced by the formula
x
x
E[X] _ 7, Pr{X > k} = 7, Pr{X ? k},
k=0
(5.2)
k=1
valid for nonnegative integer-valued random variables X. To establish
(5.2), abbreviate the notation by using p(k) = Pr{X = k}, and rearrange
the terms in E[X] = E,z, kp(k) as follows:
E[X] = Op(O) + lp(l) + 2p(2) + 3p(3) +
=p(1)+p(2)+p(3)+p(4)+
+ p(2) + p(3) + p(4) +
+p(3)+p(4)+
+p(4)+
=Pr{X? 1)+Pr{X?2}+Pr{X? 3}+
Pr{X ? k},
k=1
thus establishing (5.2).
For the chip drawing problem, the mean number of draws required,
then, is
E[X] = Pr{X > 0} + Pr{X > 11 +
+ Pr{X > n},
since Pr{X > k) = 0 fork > n. Then substituting (5.1) into (5.2) leads directly to
E[X]=2+(1- 1) +(1
+1
nl(1
n)(l
n) ...
(1
n)
-nn
+...
1).
Now let X be a nonnegative continuous random variable with densityf(x)
and distribution function F(x). The analog to (5.2) is
E[X]
f [I - F(z)] dz,
n
(5.3)
5. Some Elementary Exercises
45
obtained by interchanging an order of integration as follows:
E[X] = J xf(x) dx
f (J dz)f(x) dx
x
=
r[if(X) dx]dz=
J [I -F(z)]dz
Interchanging the order of integration where the limits are variables
often proves difficult for many students. The trick of using indicator functions to make the limits of integration constant may simplify matters. In
the preceding interchange, let
1(z < x) =
11
if 0<-z<x,
to
otherwise,
and then
1[J dz]f(x) dx = J [ J 1(z < x)f(x) dz] dx
x x
x x
f [ f 1(z < x)f(x) dx] dz = J [f f(x) dx] dz.
As an application of (5.3), let XX = min (c, X) for some positive constant c. For example, suppose X is the failure time of a certain piece of
equipment. A planned replacement policy is put in use that calls for replacement of the equipment upon its failure or upon its reaching age c,
whichever occurs first. Then
X
X, =min(c, X) = {c
if X -- c,
ifX>c
is the time for replacement.
Now
05 z,
Pr{X,>z}_11-F(z)
if G
whence we obtain
E[Xc] = J [1 - F(z)] dz,
0
<c,
46
1
Introduction
which is decidedly shorter than
E[XX] = J xf(x) dx + c[1 - F(c)].
0
Observe that X, is a random variable whose distribution is partly continuous and partly discrete, thus establishing by example that such distributions do occur in practical applications.
5.2.
The Exponential Distribution
This exercise is designed to foster intuition about the exponential distribution, as well as to provide practice in algebraic and calculus manipulations relevant to stochastic modeling.
Let X0 and X, be independent exponentially distributed random variables with respective parameters A0 and A,, so that
fort?O,i=0,1.
Pr{X;>t}=e-A,`
Let
ifX0-X
_ 0
N-{1
if X,
X0,
U = min{X0, X, } = XN;
M
1 - N;
V = max{X0, X,} = XM;
and
W=V-U=jX0-X,j.
In this context, we derive the following:
(a)
Pr(N = 0 and U > t} = e-cAo
)tA0
(a0
+ A, ).
The event (N = 0 and U > t } is exactly the event { t < X0 < X, }, whence
Pr{N = 0, U > t} = Pr{t < X0 < X, j
=
ff
Kxp<x,
A0e-AoxoA,e-Aix1 dx, dx0
47
Some Elementary Exercises
5.
=
)A0e_A0x0o
f (rAie'x'
xo
+
=f e
(AO +
A0
A0+A,,J
=
A0
A,)e-(,+A,)r0
dx0
e ( AO+A
AO + A,
(b)
Pr{N = 0 } =
and
+o
Ao
A,
Pr{N = 1) =
+' A, .
AO
We use the result in (a) as follows:
Pr{N = 0) = Pr{N = 0, U > 0) =
from (a).
A0
AO + A,
Obviously, Pr{N = 1) = 1 - Pr{N = 0) = A,/(AO + A,).
(c)
t ? 0.
Pr{ U > t} = e (AO+a,'',
Upon adding the result in (a),
Pr{N = 0 and U > t} = e
1
,(,
+A , "
A0
A,,
A0 +
to the corresponding quantity associated with N = 1,
Pr{N = I and U > t} = e-('t) +A,),
A
Ao + A '
)
we obtain the desired result via
Pr{ U > t} = Pr{N = 0, U > t} + Pr{N = 1, U > t}
= e-cAO+A,"
e-(A+A,),
Ao
IAO+A,
+
A,
A0+A,
48
I
Introduction
At this point observe that U and N are independent random variables.
This follows because (a), (b), and (c) together give
Pr{N = 0 and U > t} = Pr{N = 0} X Pr{ U > t}.
Think about this remarkable result for a moment. Suppose X0 and X,
represent lifetimes, and A0 = 0.001, while A, = 1. The mean lifetimes are
E[Xo] = 1000 and E[X,] = 1. Suppose we observe that the time of the first
death is rather small, say, U = min{Xo, X, } = z. In spite of vast disparity
between the mean lifetimes, the observation that U = 2 provides no information about which of the two units, 0 or 1, was first to die! This apparent paradox is yet another, more subtle, manifestation of the memoryless
property unique to the exponential density.
We continue with the exercise.
(d)
t?0.
Pr{W>tIN=0} =e-A",
The event { W > t and N = 0) for t ? 0 corresponds exactly to the
event It < X, - X0). Thus
Pr{W> tandN= 0} = Pr{X, - Xo> t}
I
X, -Xo>r
J
Aoe AOXoA,e
A'x'
= J ( f A,e A'x' dx )Aoe
0
dxo dx,
°x° dxo
Xo+r
(
= J e-A,(Xo+n
Aoe-A,yo dX
0
0
e-r(0 + A,)e-(,k.+A,
Ao + A,
=
0
Ao
A0 + A,
= Pr{N = 0}e-A,`
[from (b)].
o dxo
5. Some Elementary Exercises
49
Then, using the basic definition of conditional probability (Section 2.7),
we obtain
Pr{W>t,N=0} =e-Ai',
Pr{W>tIN=O} =
t?0,
Pr{N = 0}
as desired.
Of course a parallel formula holds conditional on N = 1:
t?0,
Pr{W>tIN= 1} =e-/,
and using the law of total probability we obtain the distribution of W in
the form
Pr{W>t} =Pr{W>t,N=O} +Pr{W>t,N= 1}
A
-Aor
> 0,
+
e
-a
r
+
t
AO
A0 +
U and W = V - U are independent random variables.
AO
,
(e)
)
To establish this final consequence of the memoryless property, it suffices to show that
Pr{U>uand W> w) =Pr{U>u} Pr{W>w}
forall u ?0, w ?0.
Determining first
Pr{N=0, U>u,W>w} =Pr{u<X0<X, -w)
fr
=
o
I
u<xo<x, -w
=
J
J
U xo+w
Ale-"l x I dr,)A0e-Aoxo dro
_ f e-A,(x(-+w) 1We-Aoxu
_ (A o
)e-A,u I (A0 + A')e-cAo+A,Iro
A,
AO
(A0
+
e
)e -A,w-(Ap+A,)e,
Xo
50
1
Introduction
and then, by symmetry,
and finally adding the two expressions, we obtain
Pr( U > u, W > w} =
Rko
t
o
+
)e-","
+
(
AO
+'
=Pr{W>w}Pr{U>u},
1l
u,w?0.
The calculation is complete.
Exercises
5.1. Let X have a binomial distribution with parameters n = 4 and p = ;.
Compute the probabilities Pr{X ? k} for k = 1, 2, 3, 4, and sum these to
verify that the mean of the distribution is 1.
5.2. Ajar has four chips colored red, green, blue, and yellow. A person
draws a chip, observes its color, and returns it. Chips are now drawn repeatedly, without replacement, until the first chip drawn is selected again.
What is the mean number of draws required?
5.3. Let X be an exponentially distributed random variable with parameter A. Determine the mean of X
(a) by integrating by parts in the definition in equation (2.7) with
m =1
(b) by integrating the upper tail probabilities in accordance with equation (5.3).
Which method do you find easier?
5.4. A system has two components: A and B. The operating times until
failure of the two components are independent and exponentially distributed random variables with parameters 2 for component A, and 3 for B.
The system fails at the first component failure.
(a) What is the mean time to failure for component A? For component
B?
Problems
51
(b) What is the mean time to system failure?
(c) What is the probability that it is component A that causes system
failure?
(d) Suppose that it is component A that fails first. What is the mean remaining operating life of component B?
5.5.
Consider a post office with two clerks. John, Paul, and Naomi enter
simultaneously. John and Paul go directly to the clerks, while Naomi must
wait until either John or Paul is finished before she begins service.
(a) If all of the service times are independent exponentially distributed
random variables with the same mean 1/A, what is the probability
that Naomi is still in the post office after the other two have left?
(b) How does your answer change if the two clerks have different service rates, say A, = 3 and A, = 47?
(c) The mean time that Naomi spends in the post office is less than that
for John or Paul provided that max { A A2) > c min( A, , A, j for a
certain constant c. What is the value of this constant?
Problems
5.1. Let X, , X2, ... be independent and identically distributed random
variables having the cumulative distribution function F(x) = Pr{X < x}.
For a fixed number , let N be the first index k for which Xk > . That is,
N = 1 if X, > 6; N = 2 if X, < and X, > ; etc. Determine the probability mass function for N.
5.2. Let X X,, ... , X,, be independent random variables, all exponentially distributed with the same parameter A. Determine the distribution
function for the minimum Z = min{X......
X is a discrete random variable having the geometric
distribution whose probability mass function is
p(k) = p(1 - p)k
for k = 0, 1,
... .
(a) Determine the upper tail probabilities Pr{X > k} for k = 0, 1, ...
(b) Evaluate the mean via E[X] = Ek>o Pr{X > k }.
.
52
I
Introduction
5.4. Let V be a continuous random variable taking both positive and
negative values and whose mean exists. Derive the formula
0
E[V] = J [1 - F,,(v)] dv - J F,(v) dv.
-x
0
5.5.
Show that
E[W2] = J 2y[I - F,,,(y)] dy
0
for a nonnegative random variable W.
Determine the upper tail probabilities Pr{V > t} and mean E[V]
for a random variable V having the exponential density
5.6.
f'-(v) _
I-
A,
for v < 0,
for v ? 0,
where A is a fixed positive parameter.
5.7. Let X,, X,, ... , X,, be independent random variables that are exponentially distributed with respective parameters A A...... A,,. Identify
the distribution of the minimum V = min (X,, X,, .. .
Hint:
5.8.
,
For any real number v, the event ( V > v) is equivalent to
Let U,, U,,
... ,
U be independent uniformly distributed random
variables on the unit interval
min{U U,,
[0, 1].
Define the minimum V _
... ,
(a) Show that Pr { V,, > v } = (1 - v)" for 0 : v <_ 1.
1 - (w/n)]" for 0 <- w
(b) Let W = nV,,,. Show that Pr { W > w)
and thus
w} = e-
n,
forw?0.
5.9. A flashlight requires two good batteries in order to shine. Suppose,
for the sake of this academic exercise, that the lifetimes of batteries in use
6.
Useful Functions, Integrals, and Sums
53
are independent random variables that are exponentially distributed with
parameter h = 1. Reserve batteries do not deteriorate. You begin with five
fresh batteries. On average, how long can you shine your light?
6.
Useful Functions, Integrals, and Sums
Collected here for later reference are some calculations and formulas that
are especially pertinent in probability modeling.
We begin with several exponential integrals, the first and simplest being
f ex dx = -e-x.
(6.1)
When we use integration by parts, the second integral that we introduce
reduces to the first in the manner
f
-xe-x + J ex dx = -e-x(1 + x).
xe-x dx =
(6.2)
Then (6.1) and (6.2) are the special cases of a = 1 and a = 2, respectively, in the general formula, valid for any real number a for which the
integrals are defined, given by
J
x°-'e-.,. dx
= -x°-'e-x + (a - 1)
J x°-2e-x dx.
(6.3)
Fixing the limits of integration leads to the gamma function, defined by
I'(a) = J
xa-'e-x dx,
for a > 0.
(6.4)
0
From (6.3) it follows that
['(a) = (a - 1)1'(a- 1),
(6.5)
and therefore, for any integers k,
I'(k) _ (k - 1)(k - 2)
2 I'(1).
(6.6)
54
I
Introduction
An easy consequence of (6.1) is the evaluation I'(1) = 1, which with (6.5)
shows that the gamma function at integral arguments is a generalization
of the factorial function, and
for k = 1, 2,
IF(k) = (k - 1)!
....
(6.7)
A more difficult integration shows that
I'() = V,
(6.8)
which with (6.6) provides
forn=0,1.....
I(n+;)=
(6.9)
Stirling's formula is the following important asymptotic evaluation of
the factorial function:
n! =
n"e-"(2irn)1/2er(n)112",
(6.10)
in which
1
12n+ 1
< r(n) < 1.
(6.11)
We sometimes write this in the looser form
n! - n"e-"(27m)1/2
as n - oo,
(6.12)
the symbol "-" signifying that the ratio of the two sides in (6.12) approaches 1 as n - oo. For the binomial coefficient (k) = n!/[k!(n - k)!] we
then obtain
(n\
(n -
k)
k!
k)k
as n*c,
(6.13)
as a consequence of (6.12) and the exponential limit
k
e-k = Jim (1 - k
pl-.
n
"
.
The integral
B(m, n) = f x"'-'(1 - x)"-' dx,
0
(6.14)
6.
Useful Functions, Integrals, and Sums
55
which converges when m and n are positive, defines the beta function, related to the gamma function by
r(m)r(n)
B(m, n )=
form > 0, n > 0.
r(m+n)
(6.15)
For nonnegative integral values m, n,
B(m+1,n+1)= f x(1-x)"dx=
m!n!
(m + n + 1)! '
(6.16)
... , the binomial theorem provides the evaluation
For n = 1, 2,
(1 - x)" _
(-1)"(n)xk,
for -oo < x < oo.
k
k=0
(6.17)
The formula may be generalized for nonintegral n by appropriately generalizing the binomial coefficient, defining for any real number a,
_
1a(a- 1)
k!
(a-k+ 1)
for k = 1, 2,
(Ok')
... ,
(6.18)
fork = 0.
1
As a special case, for any positive integer n,
r
\k
n)
(-1)k n(n +
(n + k - 1)
k!
= (-1)k( n + k - 1
k
(6.19)
)
The general binomial theorem, valid for all real a, is
-1)k(k)xk
for -1 < x < 1.
(6.20)
k=0
When a = -n for a positive integer n, we obtain a group of formulas useful in dealing with geometric series. For a positive integer n, in view of
(6.19) and (6.20), we have
(n + k1)
(1 - x) " =
k=0
k
xk
for jxj < 1.
(6.21)
56
I
Introduction
The familiar formula
=
7, xk = 1 + x + x2 +
for jxl < 1
1
(6.22)
1 - x
k=O
for the sum of a geometric series results from (6.21) with n = 1. The cases
n = 2 and n = 3 yield the formulas
x
7- (k+ 1)xk= I +2x+3x2+...
k=0
1
(1 -x)22
7
(k
+ 2 )( k + 1 )xk =
k=O
2
(1
- x)
for l < 1 ,
(6 . 23)
for Ixl <
( 6 . 24 )
1.
3
Sums of Numbers
The following sums of powers of integers have simple expressions:
n(n + 1)
2
'
n(n + 1)(2n + 1)
6
n22(n + 1)2
4
Chapter II
Conditional Probability and
Conditional Expectation
1.
The Discrete Case
The conditional probability Pr{AJB} of the event A given the event B is
defined by
Pr{A and B}
Pr{B)
Pr{A1B} =
if Pr{B} > 0,
(1.1)
and is not defined, or is assigned an arbitrary value, when Pr{B} = 0. Let
X and Y be random variables that can attain only countably many differ-
ent values, say 0, 1, 2, .... The conditional probability mass function
pxr(xl y) of X given Y = y is defined by
Pr{X=xand Y=y}
Pxr(xl y) =
Pr{ Y = y}
and is not defined, or
is
if Pr{ Y = y} > 0,
assigned an arbitrary value, whenever
Pr{ Y = y} = 0. In terms of the joint and marginal probability mass functions pxr(x, y) and py(y) = E, pxr(x, y), respectively, the definition is
Pxr(xlY)
=
Pxt(x> y)
pr(y)
if pr(y) > 0; x, y = 0, 1.....
(1.2)
Observe that pxr(xl y) is a probability mass function in x for each fixed
y, i.e., pxr(xl y) ? 0 and If px r(6l y) = 1, for all x, y.
58
Conditional Probability and Conditional Expectation
II
The law of total probability takes the form
Y_
Pr{X = x) _ I Px r(xIY)Pv(Y)
(1.3)
Notice in (1.3) that the points y where px y(xl y) is not defined are exactly
those values for which p.(y) = 0, and hence, do not affect the computation. The lack of a complete prescription for the conditional probability
mass function, a nuisance in some instances, is always consistent with
subsequent calculations.
Example Let X have a binomial distribution with parameters p and N,
where N has a binomial distribution with parameters q and M. What is the
marginal distribution of X?
We are given the conditional probability mass function
pxrc(kln)
-
()p(1
=
k=0, 1.... ,n,
p),,-k,
and the marginal distribution
(M)q,
pN(n) = n
(1 -
n = 0, 1, ... , M.
q)",
We apply the law of total probability in the form of (1.3) to obtain
M
Pr{X = k} = 7, px,(kI n)p,(n)
=0
M
12!
px (1
k!(n - k)!
-
p)"-k
M!
n!(M
- n)! q"(1 -
q)M
1
- k)!(M - n)! (1 -
X(1gq
(pq)"(1 _
= k!(M-k)!
M!
q),ti,
x(1
L
+ q(1
1 -q J
p)1M-k
1.
59
The Discrete Case
M!
k = 0, 1,
= k!(M - k)! (pq)k(1 - pq)M-k,
... , M.
In words, X has a binomial distribution with parameters M and pq.
Example Suppose X has a binomial distribution with parameters p and
N where N has a Poisson distribution with mean A. What is the marginal
distribution for X?
Proceeding as in the previous example but now using
Aue-A
PN(n)=
n!
n=0,1,...,
,
we obtain
Pr{X = k} = > PXN(kjn)pN(n)
,r=0
n!
n_k k!(n _ k)! P
Ake-Apk
.
[A(1
k(1 -
A"e -A
rr
p)
k
n!
p)],,-k
=k (n - k)!
k!
(Ap)k e--'
A(] -p)
k!
(AP)ke-A,,
fork = 0, 1, ... .
k!
In words, X has a Poisson distribution with mean Ap.
Example Suppose X has a negative binomial distribution with parameters p and N, where N has the geometric distribution
Pv(n) _ (1 -
for n = 1, 2,
... .
What is the marginal distribution for X?
We are given the conditional probability mass function
pxN(kl n)
n+k-1
=(
k
7
)prr(1
- p)k, k = 0, 1 , ... .
60
II
Conditional Probability and Conditional Expectation
Using the law of total probability, we obtain
x
Pr{X = k} = 7 pXN(kln)pN(n)
n=o
(n + k - 1)!
k!(n - 1)!
n=
(1 - MO - P)kP
P)k(l n
>(
nl
= (1 - J3)(1 - P)*P(l (p
- QI
-P)
05 lGP
k
P)le»-]
k
k
8p)-k-I
fork = 0, 1, ... .
We recognize the marginal distribution of X as being of geometric form.
Let g be a function for which the expectation of g(X) is finite. We define the conditional expected value of g(X) given Y = y by the formula
E[g(X )I Y = y] = 7 g(x)Pr(xly)
if Pr(y) > 0,
(1.4)
and the conditional mean is not defined at values y for which py(y) = 0.
The law of total probability for conditional expectation reads
E[g(X )] = 7 E[g(X )I Y = y]Pr(y).
(1.5)
Y
The conditional expected value E[g(X)IY = y] is a function of the real
variable y. If we evaluate this function at the random variable Y, we obtain a random variable that we denote by E[g(X )I Y]. The law of total probability in (1.5) now may be written in the form
E[g(X)] =
(1.6)
Since the conditional expectation of g(X) given Y = y is the expectation
with respect to the conditional probability mass function pxr(xl y), conditional expectations behave in many ways like ordinary expectations. The
following list summarizes some properties of conditional expectations. In
Exercises
61
this list, with or without affixes, X and Y are jointly distributed random
variables; c is a real number; g is a function for which E[Ig(X)I] < CO ; h
is a bounded function; and v is a function of two variables for which
E[Iv(X, Y)I ] < oo. The properties are
(1) E[c,g,(X,) + c2g2(X2)IY = y]
= cIE[gi(X1)IY = y] + c,E[g,(X,)I Y = y].
(1.7)
(2) if g?0, then E[g(X)IY = y] ?0.
(1.8)
(3) E[v(X, Y)I Y = y] = E[v(X, y)IY = y].
(1.9)
(4) E[g(X)I Y = y] = E[g(X)]
if X and Y are independent.
(5) E[g(X)h(Y)I Y = y] = h(y)E[g(X )I Y = y].
(1.10)
(1.11)
(6) E[g(X)h(Y)j = I h(y)E[g(X )I Y = y]pr(y)
Y
(1.12)
= E{h(Y)E[g(X)IY]}.
As a consequence of (1.7), (1.11), and (1.12), with either g = 1 or h ° 1,
we obtain
E[cI Y = y] = c,
(1.13)
E[h(Y)I Y = y] = h(y),
(1.14)
E[g(X)] = 7, E[g(X)I Y = y]pr(y) = E{E[g(X)IY]}.
(1.15)
Y
Exercises
1.1. 1 roll a six-sided die and observe the number N on the uppermost
face. I then toss a fair coin N times and observe X, the total number of
heads to appear. What is the probability that N = 3 and X = 2? What is
the probability that X = 5? What is E[X], the expected number of heads
to appear?
1.2. Four nickels and six dimes are tossed, and the total number N of
heads is observed. If N = 4, what is the conditional probability that exactly two of the nickels were heads?
Ii
62
Conditional Probability and Conditional Expectation
1.3. A poker hand of five cards is dealt from a normal deck of 52 cards.
Let X be the number of aces in the hand. Determine Pr{X > lix ? 1).
This is the probability that the hand contains more than one ace, given that
it has at least one ace. Compare this with the probability that the hand contains more than one ace, given that it contains the ace of spades.
1.4. A six-sided die is rolled, and the number N on the uppermost face
is recorded. From ajar containing 10 tags numbered 1 , 2, ... , 10 we then
select N tags at random without replacement. Let X be the smallest number on the drawn tags. Determine Pr(X = 2}.
1.5.
Let X be a Poisson random variable with parameter A. Find the conditional mean of X given that X is odd.
1.6.
Suppose U and V are independent and follow the geometric distri-
bution
p(k) = p(1 - p)k
fork = 0, 1, ... .
Define the random variable Z = U + V.
(a) Determine the joint probability mass function
pU,Z(u,
z) _
Pr{U=u,Z=z}.
(b) Determine the conditional probability mass function for U given
that Z = n.
Problems
1.1. Let M have a binomial distribution with parameters N and p. Conditioned on M, the random variable X has a binomial distribution with parameters M and ir.
(a) Determine the marginal distribution for X.
(b) Determine the covariance between X and Y = M - X.
1.2. A card is picked at random from N cards labeled 1, 2, ... , N, and
the number that appears is X. A second card is picked at random from
cards numbered 1, 2, ... , X and its number is Y. Determine the conditional distribution of X given Y = y, for y = 1, 2. ... .
Problems
63
1.3. Let X and Y denote the respective outcomes when two fair dice are
thrown. Let U = min(X, Y}, V = max{X, Y}, and S = U + V, T = V - U.
(a) Determine the conditional probability mass function for U given
V=v.
(b) Determine the joint mass function for S and T.
1.4. Suppose that X has a binomial distribution with parameters p =
and N, where N is also random and follows a binomial distribution with
parameters q = ; and M = 20. What is the mean of X?
1.5. A nickel is tossed 20 times in succession. Every time that the nickel
comes up heads, a dime is tossed. Let X count the number of heads appearing on tosses of the dime. Determine Pr{X = 0).
1.6. A dime is tossed repeatedly until a head appears. Let N be the trial
number on which this first head occurs. Then a nickel is tossed N times.
Let X count the number of times that the nickel comes up tails. Determine
Pr{X = 0), Pr{X = 11, and E[X].
1.7. The probability that an airplane accident that is due to structural
failure is correctly diagnosed is 0.85, and the probability that an airplane
accident that is not due to structural failure is incorrectly diagnosed as
being due to structural failure is 0.35. If 30 percent of all airplane accidents are due to structural failure, then find the probability that an airplane
accident is due to structural failure given that it has been diagnosed as due
to structural failure.
1.8. Initially an urn contains one red and one green ball. A ball is drawn
at random from the urn, observed, and then replaced. If this ball is red,
then an additional red ball is placed in the urn. If the ball is green, then a
green ball is added. A second ball is drawn. Find the conditional probability that the first ball was red given that the second ball drawn was red.
1.9. Let N have a Poisson distribution with parameter A = 1. Conditioned on N = n, let X have a uniform distribution over the integers
0, 1,
... , n+ 1. What is the marginal distribution for X?
64
11
Conditional Probability and Conditional Expectation
1.10. Do men have more sisters than women have? In a certain society, all married couples use the following strategy to determine the number of children that they will have: If the first child is a girl, they have no
more children. If the first child is a boy, they have a second child. If the
second child is a girl, they have no more children. If the second child is a
boy, they have exactly one additional child. (We ignore twins, assume
sexes are equally likely, and the sex of distinct children are independent
random variables, etc.) (a) What is the probability distribution for the
number of children in a family? (b) What is the probability distribution for
'the number of girl children in a family? (c) A male child is chosen at random from all of the male children in the population. What is the probability distribution for the number of sisters of this child? What is the probability distribution for the number of his brothers?
2.
The Dice Game Craps
An analysis of the dice game known as craps provides an educational ex-
ample of the use of conditional probability in stochastic modeling. In
craps, two dice are rolled and the sum of their uppermost faces is observed. If the sum has value 2, 3, or 12, the player loses immediately. If
the sum is 7 or 11, the player wins. If the sum is 4, 5, 6, 8, 9, or 10, then
further rolls are required to resolve the game. In the case where the sum
is 4, for example, the dice are rolled repeatedly until either a sum of 4
reappears or a sum of 7 is observed. If the 4 appears first, the roller wins;
if the seven appears first, he or she loses.
Consider repeated rolls of the pair of dice and let Z,, for n = 0, 1.... be
the sum observed on the nth roll. Then Z0, Z,, . . . are independent identically distributed random variables. If the dice are fair, the probability mass
function is
Pz(2) = 36,
Pz(8) = b,
Pz(3) = 36,
Pz(9) = 6,
Pz(4) = 36,
Pz(10) = 6
Pz(5) = 6,
Pz(11) = 6
Pz(6) = 36,
Pz(12) = 36
(2.1)
Pz(7)
36,
2.
65
The Dice Game Craps
Let A denote the event that the player wins the game. By the law of total
probability,
12
Pr{A} _ 7, Pr{AIZo = k}pz(k).
(2.2)
k=2
Because Zo = 2, 3, or 12 calls for an immediate loss, then
Pr{AIZ0 = k) = 0 for k = 2, 3, or 12. Similarly, Zo = 7 or 11 results in an
immediate win, and thus Pr(Alto = 7) = Pr{AIZ4 = 111 = 1. It remains
to consider the values 4 = 4, 5, 6, 8, 9, and 10, which call for additional
rolls. Since the logic remains the same in each of these cases, we will
argue only the case in which Zo = 4. Abbreviate with a = Pr{AI4 = 4}.
Then a is the probability that in successive rolls Z,, Z2, ... of a pair of
dice, a sum of 4 appears before a sum of 7. Denote this event by B, and
again bring in the law of total probability. Then
12
a = Pr{B} = I Pr{BIZ1 = k}pz(k).
(2.3)
k=2
Now Pr{BIZ, = 4) = 1, while Pr(BIZ, = 7) = 0. If the first roll results in
anything other than a 4 or a 7, the problem is repeated in a statistically
identical setting. That is, Pr{BIZ1 = k} = a for k # 4 or 7. Substitution
into (2.3) results in
a=pz(4)X1+pz(7)X0+ pz(k)Xa
k#4,7
= pz(4) + [1 - pz(4) - pz(7)]a,
or
a =
pz(4)
pz(4) + pz(7)
(2.4)
The same result may be secured by means of a longer, more computational, method. One may partition the event B into disjoint elemental
events by writing
B= {Z, =4} U {Z, 04or7,Z2=4}
U {Z, 0 4 or 7,Z2* 4 or 7, Z., =4) U -
66
II
Conditional Probability and Conditional Expectation
and then
Pr{B} = Pr{Z1 = 4} + Pr{Z1 * 4 or 7, Z, = 4}
+Pr{Z1 04 or7,Z,04or7,Z3=4}
Now use the independence of Z,, Z,,
cure
... and sum a geometric series to se-
Pr{B} = pz(4) + [1 - pz(4) - pz(7)]pz(4)
+ [1 - pz(4) - pz(7)]2pz(4) + ...
pz(4)
pz(4) + pz(7)
in agreement with (2.4).
Extending the result just obtained to the other cases having more than
one roll, we have
Pr{AIZ0=k} =
pz(k)
pz(k) + pz(7)
fork=4,5,6,8,9, 10.
Finally, substitution into (2.2) yields the total win probability
pz(k),
Pr(A) = p7(7) + p,(il) +
(2.5)
k=4.5.6.89.10 pz(k) + pz(7)
The numerical values for p,(k) given in (2.1), together with (2.5), determine the win probability
Pr(A) = 0.49292929
Having explained the computations, let us go on to a more interesting
question. Suppose that the dice are not perfect cubes but are shaved so as
to be slightly thinner in one dimension than in the other two. The numbers
that appear on opposite faces on a single die always sum to 7. That is, 1 is
opposite 6, 2 is opposite 5, and 3 is opposite 4. Suppose it is the 3-4 dimension that is smaller than the other two. See Figure 2.1. This will cause
3 and 4 to appear more frequently than the other faces, 1, 2, 5, and 6. To
see this, think of the extreme case in which the 3-4 dimension is very thin,
leading to a 3 or 4 on almost all tosses. Letting Y denote the result of tossing a single shaved die, we postulate that the probability mass function is
given by
2.
67
The Dice Game Craps
Pr(3) = PY(4) = b + 2s = p+,
Pr(l) = pr(2) = pr(5) = pr(6) _ 6 -s = p-,
where s > 0 is a small quantity depending on the amount by which the die
has been biased.
A Cubic Die
A Shaved Die
Figure 2.1 A cubic die versus a die that has been shaved down in one dimension.
If both dice are shaved in the same manner, the mass function for their
sum can be determined in a straightforward manner from the following
joint table:
Die #1
Die
1
2
3
4
5
6
#2
P-
P-
P+
P+
P-
P-
P-
P?
P?
PP-
P+P-
P?
PZ
2
p-
P2
P?
P+P-
P+P-
P?
3
p+
P+P-
P+P-
p+
P+
P+P-
P+P-
4
P+
P+P-
P+P-
1
5
p-
6 p-
PZ
P2
P1
-
P+
P+,
P+P-
P+P-
P2
P+P-
P+P-
P-
P-
PZ
P+P-
P+P-
P'
P!
68
11
Conditional Probability and Conditional Expectation
It is easily seen that the probability mass function for the sum of the dice
is
p(2) = p' = p(12),
p(3) =
2p2
= p(11),
p(4) = p-(p- + 2p+) = p(10),
p(5) = 4p+p- = p(9),
p(6) = p2 + (p+ + p-)2 = p(8),
p(7) = 4p + 2p+.
To obtain a numerical value to compare to the win probability
associated with fair dice, let us arbitrarily set 8 = 0.02, so
. Then routine substituand p+ = 0.206666
that p- = 0.146666
0.492929
tions according to the table lead to
p(2) = p(12) = 0.02151111,
p(5) = p(9) = 0.12124445,
p(3) = p(11) = 0.04302222,
p(6) = p(8) = 0.14635556,
p(4) = p(10) = 0.08213333,
p(7) = 0.17146667,
(2.6)
and the win probability becomes Pr{A } = 0.5029237.
The win probability of 0.4929293 with fair dice is unfavorable, that is,
is less than With shaved dice, the win probability is favorable, now being
0.5029237. What appears to be a slight change becomes, in fact, quite significant when a large number of games are played. See III, Section 5.
2.
Exercises
2.1.
A red die is rolled a single time. A green die is rolled repeatedly.
The game stops the first time that the sum of the two dice is either 4 or 7.
What is the probability that the game stops with a sum of 4?
2.2.
Verify the win probability of 0.5029237 by substituting from (2.6)
into (2.5).
2.3.
Determine the win probability when the dice are shaved on the 1-6
faces and p+ = 0.206666
and p_ = 0.146666
.
69
Problems
Problems
Let X X,, . . . be independent identically distributed positive random variables whose common distribution function is F. We interpret X
2.1.
X ... as successive bids on an asset offered for sale. Suppose that the
policy is followed of accepting the first bid that exceeds some prescribed
number A. Formally, the accepted bid is X,, where
N= min{k? 1;Xk>A}.
Set a = Pr(X, > A) and M = E[X,,,].
(a) Argue the equation
x
M=
J x dF(x) + (1 - a)M
A
by considering the possibilities, either the first bid is accepted, or it
is not.
(b) Solve for M, thereby obtaining
x
M = a-' J x dF(x).
A
(c) When X, has an exponential distribution with parameter A, use the
memoryless property to deduce M = A + A'.
(d) Verify this result by calculation in (b).
2.2. Consider a pair of dice that are unbalanced by the addition of
weights in the following manner: Die #1 has a small piece of lead placed
near the four side, causing the appearance of the outcome 3 more often
than usual, while die #2 is weighted near the three side, causing the outcome 4 to appear more often than usual. We assign the probabilities
Die #1
p(l) = p(2) = p(5) = p(6) = 0.166667,
p(3) = 0.186666,
p(4) = 0.146666;
70
II
Conditional Probability and Conditional Expectation
Die #2
p(l) = p(2) = p(5) = p(6) = 0.166667,
p(3) = 0.146666,
p(4) = 0.186666.
Determine the win probability if the game of craps is played with these
loaded dice.
3.
Random Sums
+ N, where N is random, arise frequently
Sums of the form X = , +
and in varied contexts. Our study of random sums begins with a crisp
definition and a precise statement of the assumptions effective in this section, followed by some quick examples.
We postulate a sequence 6 Z, ... of independent and identically dis-
tributed random variables. Let N be a discrete random variable, independent of ,, 6,, ... and having the probability mass function pN(n) _
Pr{N = n} for n = 0, 1, .... Define the random sum X by
ifN = 0,
0
X16,+...+6N
ifN>0.
We save space by abbreviating (3.1) to simply X = 6, +
(3.1)
+
understanding that X = 0 whenever N = 0.
Examples
(a) Queueing Let N be the number of customers arriving at a service
facility in a specified period of time, and let 6; be the service time
required by the ith customer. Then X = t;, +
+ eN is the total
demand for service time.
(b) Risk Theory Suppose that a total of N claims arrives at an insurance company in a given week. Let t;; be the amount of the
ith claim. Then the total liability of the insurance company is X =
(c) Population Models Let N be the number of plants of a given
species in a specified area, and let ; be the number of seeds produced by the ith plant. Then X = , +
+ fN gives the total
number of seeds produced in the area.
(d) Biometrics A wildlife sampling scheme traps a random number N
of a given species. Let 6; be the weight of the ith specimen. Then
X=6+
+ G is the total weight captured.
3.
Random Sums
71
When 6 62, ... are discrete random variables, the necessary background
in conditional probability was covered in Section 1. In order to study the
+ 6n, when 6 ,, ... are continuous random
random sum X = 6, +
variables, we need to extend our knowledge of conditional distributions.
3.1.
Conditional Distributions: The Mixed Case
Let X and N be jointly distributed random variables and suppose that the
possible values for N are the discrete set n = 0, 1, 2,
Then the elementary definition of conditional probability (1.1) applies to define the
conditional distribution function FxN(xIn) of the random variable X given
that N = n to be
....
FF. (xIn) =
Pr{X<xandN=n}
if Pr{N = n} > 0,
Pr{N=n}
(3.2)
and the conditional distribution function is not defined at values of n for
which Pr{N = n} = 0. It is elementary to verify that Fx,v(xln) is a probability distribution function in x at each value of n for which it is defined.
The case in which X is a discrete random variable was covered in Section 1. Now let us suppose that X is continuous and that FF,v(xln) is differentiable in x at each value of n for which Pr{N = n} > 0. We define the
conditional probability density function fx N(xl n) for the random variable X
given that N = n by setting
ftN(xln) = d Fx (xln)
if Pr(N = n) > 0.
(3.3)
Again, fx,. (xI n) is a probability density function in x at each value of n for
which it is defined. Moreover, the conditional density as defined in (3.3)
has the appropriate properties, for example,
v
Pr{ a < X < b, N = n }
=f
(3.4)
a
for a < b and where p\,(n) = Pr{N = n). The law of total probability
leads to the marginal probability density function for X via
.fx(x) = Y, fx,v(xIn)pn,(n).
11=0
(3.5)
72
II
Conditional Probability and Conditional Expectation
Suppose that g is a function for which E[Ig(X)I] < oo. The conditional
expectation of g(X) given that N = n is defined by
E[g(X)IN = n] = J g(x)fXN(xln) dx.
(3.6)
Stipulated thus, E[g(X)IN = n] satisfies the properties listed in (1.7) to
(1.15) for the joint discrete case. For example, the law of total probability is
00
E[g(X)] = 7, E[g(X)IN= n]PN(n) = E{E[g(X)IN]}.
(3.7)
n=o
3.2. The Moments of a Random Sum
Let us assume that
k
and N have the finite moments
E[lk] =
Var[ek] = o,',
I-L,
(3.8)
E[N] = v,
Var[N] = 7-,
and determine the mean and variance for X = 6, +
+ 6N as defined in
(3.1). The derivation provides practice in manipulating conditional expectations, and the results,
Var[X] = vo + µ2T,
E[X] = µv,
(3.9)
are useful and important. The properties of conditional expectation listed
in (1.7) to (1.15) justify the steps in the determination.
If we begin with the mean E[X], then
x
E[X] _ > E[XIN = n]pN(n)
[(by (1.15)]
n=0
x
E[, +
+NIN = n]pN(n)
(definition of X)
n=1
E[e, + ... + e,JN = n]PN(n)
n=1
[by (1.9)]
x
6,,JpN(n)
n=
x
lu
7, nPN(n) = µµv
,t=1
[by (1.10)]
3.
73
Random Sums
To determine the variance, we begin with the elementary step
Var[X] = E[(X - µv)2] = E[(X - Nµ + Nµ - vµ)2]
= E[(X - Nµ)2] + E[µ2(N - v)2]
(3.10)
+ 2E[p(X - Nµ)(N - v)].
Then
x
E[(X - Nµ)22] = Y E[(X - Nµ)2 IN = n]PN(n)
_
E[(6, +...+fn-nµ)2N=n]PN(n)
n=1
npN(n) = va-22,
n=1
and
E[A2(N - v)22] = µ2E[(N - v)2] = A2 T2,
while
x
E[p(X - Nµ)(N - v)] = µ7 E[(X - np)(n - v)IN = n]p,(n)
,1=0
- v)E[(X - nµ)I N = n] pN(n)
=0
(because E[(X - nµ)IN = n] =
nµ] = 0). Then (3.10)
with the subsequent three calculations validates the variance of X as stated
in (3.9).
Example The number of offspring of a given species is a random variable having probability mass function p(k) for k = 0, 1. .... A population
begins with a single parent who produces a random number N of progeny,
each of which independently produces offspring according to p(k) to form
a second generation. Then the total number of descendants in the second
+ ,,,, where k is the number of
generation may be written X = 6, +
progeny of the kth offspring of the original parent. Let E[N] = E[6k] = µ
and Var[N] = Var[ k] = 0,22. Then
E[X] = µ2
and
Var[X] = µw2(1 + µ).
74
11
3.3.
Conditional Probability and Conditional Expectation
The Distribution of a Random Sum
Suppose that the summands ,, 62, ... are continuous random variables
having a probability density functionf(z). For n ? 1, the probability den+ is the n-fold convolution of
sity function for the fixed sum , +
the density f(z), denoted by fl'°(z) and recursively defined by
f f `(z) = f(z)
and
f'"'(z) = f f"'-"(z - u)f(u) du
for n > 1.
(3.11)
(See I, Section 2.5 for a discussion of convolutions.) Because N and
62.... are independent, then fl"'(z) is also the conditional density func-
tion for X = 6, +
+ 6, given that N = n ? 1. Let us suppose that
Pr{N = 0} = 0. Then, by the law of total probability as expressed in (3.5),
X is continuous and has the marginal density function
.ff(x) = , f O(x)pN(n)
(3.12)
n
Remark When N = 0 can occur with positive probability, then X = ,
+
+ N is a random variable having both continuous and discrete
components to its distribution. Assuming that
with probability density function f(z), then
62,
... are continuous
Pr{X=0} =Pr{N=0) =p,,,(0),
while for 0 < a < b or a < b < 0, then
G
x
Pr(a < X < b} = f { f("'(z)pN(n)I dz.
n-i
(3.13)
Example A Geometric Sum of Exponential Random Variables In the
following computational example, suppose
Ae-A=
f (z) = {0
for z ? 0,
for z < 0,
and
pN(n) = /3(1 - f3)"-'
for n = 1, 2, ... .
3.
75
Random Sums
For n ? 1, the n-fold convolution of f(z) is the gamma density
111
1)!z"-le-Az
f c"o(z)
(n
/--
for z ? 0,
forz<0.
0
(See I, Section 4.4 for discussion.)
The density for X = , +
+
N
is given, according to (3.5), by
W
fc")(z)PN(n)
fx(z) =
n=1
All
1
=
= FA(1
tie -Az\
-
Azeau pr
= Age-ARz,
z >_ 0.
Surprise! X has an exponential distribution with parameter A/3.
Example Stock Price Changes Stochastic models for price fluctuations of publicly traded assets were developed as early as 1900.
Let Z denote the difference in price of a single share of a certain stock
between the close of one trading day and the close of the next. For an
actively traded stock, a large number of transactions take place in a single
day, and the total daily price change is the sum of the changes over these
individual transactions. If we assume that price changes over successive
transactions are independent random variables having a common finite
variance,' then the central limit theorem applies. The price change over
a large number of transactions should follow a normal, or Gaussian,
distribution.
' Rather strong economic arguments in support of these assumptions can be given. The
independence follows from concepts of a "perfect market," and the common variance from
notions of time stationarity.
76
II
Conditional Probability and Conditional Expectation
A variety of empirical studies have supported this conclusion. For the
most part, these studies involved price changes over a fixed number of
transactions. Other studies found discrepancies in that both very small and
very large price changes occurred more frequently in the data than sug-
gested by normal theory. At the same time, intermediate-size price
changes were underrepresented in the data. For the most part these studies examined price changes over fixed durations containing a random
number of transactions.
A natural question arises: Does the random number of transactions in a
given day provide a possible explanation for the departures from normality that are observed in data of daily price changes? Let us model the daily
price change in the form
o+X,
(3.14)
where to, t; ... are independent normally distributed random variables
with common mean zero and variance o r2, and N has a Poisson distribution with mean v.
We interpret N as the number of transactions during the day, ; for
i ? 1 as the price change during the ith transaction, and o as an initial
price change arising between the close of the market on one day and the
opening of the market on the next. (An obvious generalization would
allow the distribution of o to differ from that of t,, .Z..... )
Conditioned on N = n, the random variable Z = io + , + ... + is
normally distributed with mean zero and variance (n + 1)o . The conditional density function is
z
,(z) =
2-Tr(n + 1)o-
exp{
2 (n + 1)o-2J.
Since the probability mass function for N is
Xle-A
p,,,(n)=
,
n!.
n=0,1,...,
using (3.12) we determine the probability density function for the daily
price change to be
Alle-A
(P.(z)
.fZ(z)
,1=o
n.
.
77
Exercises
The formula for the density ff(z) does not simplify. Nevertheless, numerical calculations are possible. When A = 1 and 0.2 = 1, then (3.9) shows
that the variance of the daily price change Z in the model (3.14) is
Var[Z] _ (1 + A)o-2 = 1. Thus comparing the density ff(z) when A = 1
and o.2 = ; to a normal density with mean zero and variance, one sheds
some light on the question at hand.
The calculations were carried out and are shown in Figure 3.1.
The departure from normality that is exhibited by the random sum in
Figure 3.1 is consistent with the departure from normality shown by stock
price changes over fixed time intervals. Of course, our calculations do not
prove that the observed departure from normality is caused by the random
number of transactions in a fixed time interval. Rather, the calculations
show only that such an explanation is consistent with the data and is,
therefore, a possible cause.
2
Figure 3.1
A standard normal density (solid line) as compared to a density for
a random sum (dashed line). Both densities have zero mean and unit variance.
Exercises
3.1. A six-sided die is rolled, and the number N on the uppermost face is
recorded. Then a fair coin is tossed N times, and the total number Z of heads
to appear is observed. Determine the mean and variance of Z by viewing Z
as a random sum of N Bernoulli random variables. Determine the probability mass function of Z, and use it to find the mean and variance of Z.
78
11
Conditional Probability and Conditional Expectation
3.2. Six nickels are tossed, and the total number N of heads is observed.
Then N dimes are tossed, and the total number Z of tails among the dimes
is observed. Determine the mean and variance of Z. What is the probability that Z = 2?
Suppose that upon striking a plate a single electron is transformed
into a number N of electrons, where N is a random variable with mean p.
and standard deviation o. Suppose that each of these electrons strikes a
second plate and releases further electrons, independently of each other
and each with the same probability distribution as N. Let Z be the total
number of electrons emitted from the second plate. Determine the mean
and variance of Z.
3.3.
3.4. A six-sided die is rolled, and the number N on the uppermost face
is recorded. From a jar containing 10 tags numbered 1 , 2, ... , 10 we then
select N tags at random without replacement. Let X be the smallest number on the drawn tags. Determine Pr (X = 2) and E[X].
3.5. The number of accidents occurring in a factory in a week is a Poisson random variable with mean 2. The number of individuals injured in
different accidents is independently distributed, each with mean 3 and
variance 4. Determine the mean and variance of the number of individuals injured in a week.
Problems
3.1. The following experiment is performed: An observation is made of
a Poisson random variable N with parameter A. Then N independent
Bernoulli trials are performed, each with probability p of success. Let Z
be the total number of successes observed in the N trials.
(a) Formulate Z as a random sum and thereby determine its mean and
variance.
(b) What is the distribution of Z?
3.2. For each given p, let Z have a binomial distribution with parameters p and N. Suppose that N is itself binomially distributed with parameters q and M. Formulate Z as a random sum and show that Z has a binomial distribution with parameters pq and M.
4.
79
Conditioning on a Continuous Random Variable
3.3. Suppose that f,, ,, ... are independent and identically distributed
with Pr(6R = ± 11 = ;. Let N be independent of 6, ,2, ... and follow the
geometric probability mass function
p,,(k) = all - a)'
for k = 0, 1,
... ,
where 0 < a < 1. Form the random sum Z = , +
+
(a) Determine the mean and variance of Z.
(b) Evaluate the higher moments m, = E[Z'] and m,, = E[Z°].
Hint:
Express Z' in terms of the ;'s where C; = 1 and E[6; ;] = 0.
3.4. Suppose 6 62, ... are independent and identically distributed random variables having mean µ and variance 0-2 . Form the random sum
SN=6i +...+6N
(a) Derive the mean and variance of S, when N has a Poisson distribution with parameter A.
(b) Determine the mean and variance of S, when N has a geometric distribution with mean A = (1 - p)lp.
(c) Compare the behaviors in (a) and (b) as A - .
3.5. To form a slightly different random sum, let &, ,, . . . be independent identically distributed random variables and let N be a nonnegative
integer-valued random variable, independent of ,
.... The first two
moments are
E[6,] = µ,
Var[ R] _ 0-2,
E[N] = v,
Var[N] = T'-,
Determine the mean and variance of the random sum Z =o +
4.
+
Conditioning on a Continuous Random Variable2
Let X and Y be jointly distributed continuous random variables with joint
probability density function ff.,.(x, y). We define the conditional probability density function fx,,(xl y) for the random variable X given that Y = y by
the formula
2 The reader may wish to defer reading this section until encountering VII, on renewal
processes, where conditioning on a continuous random variable first appears.
80
11
Conditional Probability and Conditional Expectation
fxr(xI y) = f fr(y))
(4.1)
if ff (y) > 0,
and the conditional density is not defined at values y for which fy(y) = 0.
The conditional distribution function for X given Y = y is defined by
Fx r(xjy) =
f fx r(61y) d
if fr(y) > 0.
(4.2)
-z
Finally, given a function g for which E[I g(X )I ] < oo, the conditional expectation of g(X) given that Y = y is defined to be
E[g(X)I Y = yl =
Jg(x)fxy(xy) dx
if fr(y) > 0.
(4.3)
The definitions given in (4.1) to (4.3) are a significant extension of our
elementary notions of conditional probability because they allow us to
condition on certain events having zero probability. To understand the distinction, try to apply the elementary formula
Pr{AIB} =
Pr{A and B}
Pr{B}
if Pr{B} > 0
(4.4)
to evaluate the conditional probability Pr{a < X <_ bIY = y}. We set
A = (a < X:5 b) and B = {Y = y}. But Y is a continuous random variable, and thus Pr{B} = Pr( Y = y) = 0, and (4.4) cannot be applied. Equation (4.2) saves the day, yielding
b
Pr{a
< X : bIY = y)
= Fx r(bly) - Fx
r(
I y)
(4.5)
a
provided only that the density fr(y) is strictly positive at the pointy.
To emphasize the important advance being made, we consider the following simple problem. A woman arrives at a bus stop at a time Y that is
uniformly distributed between 0 (noon) and 1. Independently, the bus arrives at a time Z that is also uniformly distributed between 0 and 1. Given
that the woman arrives at time Y = 0.20, what is the probability that she
misses the bus?
On the one hand, the answer Pr(Z < YIY = 0.20) = 0.20 is obvious.
On the other hand, this elementary question cannot be answered by the
4.
Conditioning on a Continuous Random Variable
81
elementary conditional probability formula (4.4) because the event
(Y = 0.20) has zero probability. To apply (4.1), start with the joint density function
r1
AY(z' y) = 0
elsewhere,
and change variables according to X = Y - Z. Then
fx.r(x,y)= 1
for05y<l,y- 1 :! x:y,
and, applying (4.1), we find that
fx r(xI0.20) =
fx,r(x, 0.20)
fr(0.20)
for -0.80 <- x :5 0.20.
=1
Finally,
00
Pr(Z<YIY=0.20) =Pr(X>OIY=0.20) = Jfx (x0.20) dx = 0.20.
0
We see that the definition in (4.1) leads to the intuitively correct answer.
The conditional density function that is prescribed by (4.1) possesses
all of the properties that are called for by our intuition and the basic concept of conditional probability. In particular, one can calculate the probability of joint events by the formula
d b
Pr{a < X < b, c < Y < d} = J(Jfx y(xl y) dx} fy(y) dy,
c
(4.6)
a
which becomes the law of total probability by setting c = -oo and
d=+oo:
+00
Pr(a < X < b} =
b
f {Jf(xy) dx}fy(y) dy.
xa
(4.7)
For the same reasons, the conditional expectation as defined in (4.3) sat-
isfies the requirements listed in (1.7) to (1.11). The property (1.12),
adapted to a continuous random variable Y, is written
E[g(X)h(Y)] = E{h(Y)E[g(X)IY])
= Jh(y)E[g(X)Y = y]ff(y) dy,
(4.8)
82
II
Conditional Probability and Conditional Expectation
valid for any bounded function h, and assuming E[Ig(X)I] < x. When
h(y) = 1, we recover the law of total probability in the form
E[g(X)] = E{E[g(X)I Yl }= JE[g(X)Y = y].fr(y) dy.
(4.9)
Both the discrete and continuous cases of (4.8) and (4.9) are contained
in the expressions
E[g(X)h(Y)] = E{h(Y)E[g(X)IY]}= Jh(y)E[g(X)Y= y] dFF(y),
(4.10)
dnd
E[g(X)] = E{E[g(X)IY]}= JE[g(X)Y = y] dFF(y).
(4.11)
[See the discussion following (2.9) in I for an explanation of the symbolism in (4.10) and (4.11).]
The following exercises provide practice in deriving conditional probability density functions and in manipulating the law of total probability.
Example Suppose X and Y are jointly distributed random variables
having the density function
.fxr(x, y) =
1e
for x, y > 0.
y
We first determine the marginal density for y, obtaining
x
.fr(y) = Jfxy(X, y) dx
0
X
= e-, f Y
dx = e
for t'>0.
0
Then
fx r(xl y) =
f.f(c')) = y- 'e-for
x, y > 0.
That is, conditional on Y = y, the random variable X has an exponential
distribution with parameter 1/y. It is easily seen that E[XI Y = y] = Y.
Example For each given p, let X have a binomial distribution with parameters p and N. Suppose that p is uniformly distributed on the interval
[0, 1]. What is the resulting distribution of X?
We are given the marginal distribution for p and the conditional distri-
4.
Conditioning on a Continuous Random Variable
83
bution for X. Applying the law of total probability and the beta integral I,
(6.16) we obtain
Pr{X = k} = JPr{X = klp = }f,(O de
0
_
_
kl-
N!
0
k!(N - k)!
Nkd
k!(N - k)!
k!(N - k)! (N + 1)!
N!
1
N+1
fork= 0, 1,...,N.
That is, X is uniformly distributed on the integers 0, 1, . . . , N.
When p has the beta distribution with parameters r and s, then similar
calculations give
I
Pr{X = k} =
k!(N - k)!
Y)r(S f f.-.(1
-
- 6)"-k d
_ (N) I'(r + s)r(r + k)I'(s + N - k)
- kI'(r)I'(s)I'(N + r + s)
fork=0, 1,...,N.
Example A random variable Y follows the exponential distribution
with parameter 0. Given that Y = y, the random variable X has a Poisson
distribution with mean y. Applying the law of total probability then yields
x
Pr{X = k} =
Joedy
yKe-
0
6 f yke-(,+0)`' dy
k! o
x
e
k!(1+ O)k+1 0
uke " du
0
0
(1 + O)k+I
for k= 0, 1, ... .
84
II
Conditional Probability and Conditional Expectation
Suppose that Y has the gamma density
y ? 0.
r(a)(ey)a-ie-m,
fr(Y) =
Then similar calculations yield
Pr{X = k} =
f
0
yke-'
B
p,,
k I'0
(OY
Ba
k!1'(a)(1 + 9)k+a
)a-Ie-° dy
f uk+a- I e-" du
0
0
k!F(a) )
( l+0)1+0)
k = 0,1,....
k
This is the negative binomial distribution.
Exercises
4.1. Suppose that three contestants on a quiz show are each given the
same question, and that each answers it correctly, independently of the
others, with probability p. But the difficulty of the question is itself a random variable, so let us suppose, for the sake of illustration, that p is uniformly distributed over the interval (0, 1]. What is the probability that exactly two of the contestants answer the question correctly?
4.2. Suppose that three components in a certain system each function
with probability p and fail with probability 1 - p, each component operating or failing independently of the others. But the system is in a random
environment, so that p is itself a random variable. Suppose that p is uniformly distributed over the interval (0, 1]. The system operates if at least
two of the components operate. What is the probability that the system
operates?
4.3. A random variable T is selected that is uniformly distributed over
the interval (0, 1]. Then a second random variable U is chosen, uniformly
distributed on the interval (0, T]. What is the probability that U exceeds 1?
Problems
85
4.4. Suppose X and Y are independent random variables, each exponentially distributed with parameter A. Determine the probability density
function for Z = X/Y.
4.5.
Let U be uniformly distributed over the interval [0, L] where L follows the gamma density fL(x) = xe for x >_ 0. What is the joint density
function of U and V = L - U?
Problems
4.1. Suppose that the outcome X of a certain chance mechanism depends on a parameter p according to Pr{X = 1) = p and Pr{X = 0} =
1 - p, where 0 < p < 1. Suppose that p is chosen at random, uniformly
distributed over the unit interval [0, 1], and then that two independent outcomes X, and X, are observed. What is the unconditional correlation coefficient between X, and X,?
Note:
Conditionally independent random variables may become dependent if they share a common parameter.
4.2. Let N have a Poisson distribution with parameter A > 0. Suppose
that, conditioned on N = n, the random variable X is binomially distributed with parameters N = n and p. Set Y = N - X. Show that X and Y
have Poisson distributions with respective parameters Ap and A(1 - p)
and that X and Y are independent.
Note:
Conditionally dependent random variables may become independent through randomization.
4.3. Let X have a Poisson distribution with parameter A > 0. Suppose A
itself is random, following an exponential density with parameter 0.
(a) What is the marginal distribution of X?
(b) Determine the conditional density for A given X = k.
4.4.
Suppose X and Y are independent random variables having the
same Poisson distribution with parameter A, but where A is also random,
being exponentially distributed with parameter 0. What is the conditional
distribution for X given that X + Y = n?
86
II
Conditional Probability and Conditional Expectation
4.5.
Let X and Y be jointly distributed random variables whose joint
probability mass function is given in the following table:
x
-1
0
1
2
9
9
1
0
2
y
1
9
9
9
0
91
p(x,y)=Pr{x=x,Y=y}
Show that the covariance between X and Y is zero even though X and Y
are not independent.
4.6. Let Xo, x, X2,
. . . be independent identically distributed nonnegative random variables having a continuous distribution. Let N be the first
index k for which Xk > Xo. That is, N = 1 if X, > Xo, N = 2 if X, X0 and
X2 > X0, etc. Determine the probability mass function for N and the mean
E[N]. (Interpretation: X0, X,.... are successive offers or bids on a car that
you are trying to sell. Then N is the index of the first bid that is better than
the initial bid.)
4.7. Suppose that X and Y are independent random variables, each having the same exponential distribution with parameter a. What is the conditional probability density function for X, give that Z = X + Y: = z?
4.8. Let X and Y have the joint normal density given in I, (4.16). Show
that the conditional density function for X, given that Y = y, is normal
with moments
AXr=!'fi'x+
Pox
PY
(y - A v)
and
axr=cTxV1-Pl.
5.
87
Martingales
Martingales*
5.
Stochastic processes are characterized by the dependence relationships
that exist among their variables. The martingale property is one such relationship that captures a notion of a game being fair. The martingale
property is a restriction solely on the conditional means of some of the
variables, given values of others, and does not otherwise depend on the
actual distribution of the random variables in the stochastic process. Despite the apparent weakness of the martingale assumption, the consequences are striking, as we hope to suggest.
5.1.
The Definition
We begin the presentation with the simplest definition.
Definition A stochastic process {X,,; n = 0, 1.... } is a martingale if for
n=0,1,...,
(a) E[IX Il < 00,
and
(b)
X
Taking expectations on both sides of (b),
Xo,
. . . ,
E{X,,},
and using the law of total probability in the form
X ... ,
shows that
and consequently, a martingale has constant mean:
E[X0] = E[XA] =
0 : k <_ n.
(5.1)
* Some problems scattered throughout the text call for the student to identify certain stochastic processes as martingales. Otherwise, the material of this section is not used in the
sequel.
88
II
Conditional Probability and Conditional Expectation
A similar conditioning (see Problem 5.1) verifies that the martingale
equality (b) extends to future times in the form
E[X,,,IXO,
. . . ,
X,,
for m ? n.
(5.2)
To relate the martingale property to concepts of fairness in gambling,
consider X to be a certain player's fortune after the nth play of a game.
The game is "fair" if on average, the player's fortune neither increases nor
decreases at each play. The martingale property (b) requires the player's
fortune after the next play to equal, on average, his current fortune and not
be otherwise affected by previous history. Some early work in martingale
theory was motivated in part by problems in gambling. For example, martingale systems theorems consider whether an astute choice of betting
strategy can turn a fair game into a favorable one, and the name "martingale" derives from a French term for the particular strategy of doubling
one's bets until a win is secured. While it remains popular to illustrate
martingale concepts with gambling examples, today, martingale theory
has such broad scope and diverse applications that to think of it purely in
terms of gambling would be unduly restrictive and misleading.
Example Stock Prices in a Perfect Market Let X,, be the closing price
at the end of day n of a certain publicly traded security such as a share of
stock. While daily prices may fluctuate, many scholars believe that, in a
perfect market, these price sequences should be martingales. In a perfect
market freely open to all, they argue, it should not be possible to predict
with any degree of accuracy whether a future price
will be higher or
lower than the current price X,,. For example, if a future price could be expected to be higher, then a number of buyers would enter the market, and
their demand would raise the current price X,,. Similarly, if a future price
could be predicted as lower, a number of sellers would appear and tend to
depress the current price. Equilibrium obtains where the future price cannot be predicted, on average, as higher or lower, that is, where price sequences are martingales.
5.2.
The Markov Inequality
What does the mean of a random variable tell us about its distribution?
For a nonnegative random variable X, Markov's inequality is APr{X ? A)
s E[X], for any positive constant A. For example, if E[X] = 1, then
S.
Martingales
89
Pr(X ? 4} <_ , no matter what the actual distribution of X is. The proof
uses two properties: (i) X > 0 (X is a nonnegative random variable); and
(ii) E[X1{X ? A}] ? APr{X ? A}. (Recall that 1(A) is the indicator of an
event A and is one if A occurs and zero otherwise. See I, Section 3.1.)
Then by the law of total probability,
E[X] = E[X1{X ? A}] + E[X1{X < A)]
E[X1{X ? All
(by (i))
APr{X ? A),
(by (ii))
and Markov's inequality results.
5.3.
The Maximal Inequality for Nonnegative Martingales
Because a martingale has constant mean, Markov's inequality applied to
a nonnegative martingale immediately yields
A) <_
E[X0],A>0.
We will extend the reasoning behind Markov's inequality to achieve an inequality of far greater power:
X ? A}
E[X °] .
(5.3)
Instead of limiting the probability of a large value for a single observation
X,,, the maximal inequality (5.3) limits the probability of observing a large
value anywhere in the time interval 0, . . . , m, and since the right side of
(5.3) does not depend on the length of the interval, the maximal inequality limits the probability of observing a large value at any time in the infinite future of the martingale!
In order to prove the maximal inequality for nonnegative martingales,
we need but a single additional fact: If X and Y are jointly distributed random variables and B is an arbitrary set, then
E[X1{Yin B}] = E[E{XIY}1{Yin B}].
(5.4)
But (5.4) follows from the conditional expectation property (1.12),
E[g(X)h(Y)] = E{h(Y)E[g(X)IY]}, with g(x) = x and h(y) = 1(y in B).
90
II
Conditional Probability and Conditional Expectation
We will have need of (5.4) with X = X,,, and Y = (X0,
(5.4) followed by (5.2) then justifies
... , X, ), whereupon
E[X,,1{X0<A,...,X,,.
=E[E{X,,IX0,...,X,,}1{X0<
<A,X,,_A}].
A,
Theorem 5.1 The maximal inequality for nonnegative martingales
... be a martingale with nonnegative values; i.e., Pr(X ? 0)
= 1 for n = 0, 1. .... For any A > 0,
Let X0, X
X ? A) s
Pr
E[X0]
for
(5 . 6)
for all n.
(5.7)
A
and
Pr{
X > A)
E[X °]
,
Proof Inequality (5.7) follows from (5.6) because the right side of (5.6)
does not depend on m. We begin with the law of total probability, as in I,
Section 2.1. Either the {X0, ... , X,,, } sequence rises above A for the first
time at some index n, or else it remains always below A. As these
possibilities are mutually exclusive and exhaustive, we apply the law of
total probability to obtain
,,,
E[X ,] = >, E[X,,,1 {X < A,...
,
A, X,, ? A}
!t=
,X,,,<A}]
m
0)
11 =
M
E[X,,1(X0 < A__,
A>,Pr{X0 < A_
A, X,, ? A)]
X,,-, < A, X,,
(using 5.5))
Al
,1=o
= A Pr f
X,, ? A).
Example A gambler begins with a unit amount of money and faces a
series of independent fair games. Beginning with X = 1, the gambler bets
S.
Martingales
91
the amount p, 0 < p < 1. If the first game is a win, which occurs with
probability Z, the gambler's fortune is X, = 1 + pXo = 1 + p. If the first
game is a loss, then X, = 1 - pXo = 1 - p. After the nth play and with a
current fortune of X,,, the gambler wagers pX,,, and
((1 + p)X,,
with probability Z,
- p)X
with probability Z.
1(1
is a nonnegative martingale, and the maximal inequality (5.6)
Then
with A = 2, for example, asserts that the probability that the gambler ever
doubles his money is less than or equal to z, and this holds no matter what
the game is, as long as it is fair, and no matter what fraction p of his fortune is wagered at each play. Indeed, the fraction wagered may vary from
play to play, as long as it is chosen without knowledge of the next
outcome.
As amply demonstrated by this example, the maximal inequality is a
very strong statement. Indeed, more elaborate arguments based on the
maximal and other related martingale inequalities are used to show that a
nonnegative martingale converges: If
is a nonnegative martingale,
then there exists a random variable, let us call it X., for which lim,,. X,, =
X.. We cannot guarantee the equality of the expectations in the limit, but
the inequality E[X0] ? E[X] ? 0 can be established.
Example In III, Section 8, we will introduce the branching process
model for population growth. In this model, X,, is the number of individuals in the population in the nth generation, and µ > 0 is the mean family
size, or expected number of offspring of any single individual. The mean
population size in the nth generation is Xoµ". In this branching process
model, X,,/µ" is a nonnegative martingale (see III, Problem 8.4), and the
maximal inequality implies that the probability of the actual population
ever exceeding ten times the mean size is less than or equal to 1/10. The
nonnegative martingale convergence theorem asserts that the evolution of
such a population after many generations may be described by a single
random variable X. in the form
X -== X,,µ",
for large n.
Example How NOT to generate a uniformly distributed random variable An urn initially contains one red and one green ball. A ball is drawn
at random and it is returned to the urn, together with another ball of the
92
11
Conditional Probability and Conditional Expectation
same color. This process is repeated indefinitely. After the nth play there
will be a total of n + 2 balls in the urn. Let R,, be the number of these balls
that are red, and X,, =
2) the fraction of red balls. We claim that
is a martingale. First, observe that
(R + 1 with probability X,,,
R with probability 1 - X,,,
so that
n + 1),
R,, + X,, =
and finally,
2
E[X,,,,I X,,]
= n+3
E[R,,+,IX,,]
=
n+3
1
X" = X.
This verifies the martingale property, and because such a fraction is always nonnegative, indeed, between 0 and 1, there must be a random variable X. to which the martingale converges. We will derive the probability
distribution of the random limit. It is immediate that R, is equally likely to
be 1 or 2, since the first ball chosen is equally likely to be red or green.
Continuing,
Pr{R2 = 3) _ //Pr//{R, = 31R, = 2}Pr{R1 = 21
l3)(2)
3,
Pr{R, = 2} = Pr{R2 = 2IR, = 1 }Pr{R, = 1)
(+ Pr{R2//=((21R, = 2}Pr{R1 = 21
(3)(/(2) + (3)\2) =
3+
and since the probabilities must sum to one,
Pr{R, = 1) =
.
By repeating these simple calculations, it is easy to see that
1
n+1
fork=1,2,...,n+1,
and that therefore X,, is uniformly distributed over the values 1/(n + 2),
Exercises
93
2/(n + 2), . . . , (n + 1)/(n + 2). This uniform distribution must prevail in
the limit, which leads to
Pr{X,,-x} =x
for0<x<1.
Think about this remarkable result for a minute! If you sit down in front
of such an urn and play this game, eventually the fraction of red balls in
your urn will stabilize in the near vicinity of some value, call it U. If I play
the game, the fraction of red balls in my urn will stabilize also, but at another value, U'. Anyone who plays the game will find the fraction of red
balls in the urn tending towards some limit, but everyone will experience
a different limit. In fact, each play of the game generates a fresh, uniformly distributed random variable, in the limit. Of course, there may
be faster and simpler ways to generate uniformly distributed random
variables.
Martingale implications include many more inequalities and convergence theorems. As briefly mentioned at the start, there are so-called systems theorems that delimit the conditions under which a gambling system,
such as doubling the bets until a win is secured, can turn a fair game into
a winning game. A deeper discussion of martingale theory would take us
well beyond the scope of this introductory text, and our aim must be limited to building an enthusiasm for further study. Nevertheless, a large variety of important martingales will be introduced in the Problems at the
end of each section in the remainder of the book.
Exercises
5.1. Let X be an exponentially distributed random variable with mean
E[X] = 1. For x = 0.5, 1, and 2, compare Pr{X > x} with the Markov inequality bound E[X]/x.
5.2. Let X be a Bernoulli random variable with parameter p. Compare
Pr{X ? 1) with the Markov inequality bound.
5.3. Let 6 be a random variable with mean µ and standard deviation a-.
Let X = (6 - µ)Z. Apply Markov's inequality to X to deduce Chebyshev's
inequality:
Pr {
- µl ? e } s 0Z8
for any e > 0.
94
II
Conditional Probability and Conditional Expectation
Problems
5.1. Use the law of total probability for conditional expectations
E[E{XIY, Z}IZ] = E[XIZ] to show
E[E{X,,+2IXO,
... ,
I Xo,
... ,
X is a martingale.
XO, ... ,
X.
5.2.
Let U,, U2, . . . be independent random variables each uniformly
distributed over the interval (0, 1]. Show that X0 = 1 and X,, = 2"U,
U
for n = 1, 2.... defines a martingale.
5.3
Let So = 0, and for n ? 1, let S,, = s, +
+ s be the sum of n
independent random variables, each exponentially distributed with mean
E[e] = 1. Show that
defines a martingale.
5.4. Let
62, ... be independent Bernoulli random variables with
parameter p, 0 < p < 1. Show that X. = 1 and X,, = p" 6, ... ,,, n = 1,
2,
. . . ,
defines a nonnegative martingale. What is the limit of X,, as n-oo?
5.5. Consider a stochastic process that evolves according to the following laws: If X,, = 0, then X,,+, = 0, whereas if X,, > 0, then
IX,, + 1
with probability 2
-1
with probability Z .
X",
X
(a) Show that X,, is a nonnegative martingale.
(b) Suppose that X0 = i > 0. Use the maximal inequality to bound
Pr{X ? N for some n ? OIXO = i}.
Note: X,, represents the fortune of a player of a fair game who wagers $1
at each bet and who is forced to quit if all money is lost (X,, = 0). This
gambler's ruin problem is discussed fully in III, Section 5.3.
Chapter III
Markov Chains: Introduction
1.
Definitions
A Markov process {X,} is a stochastic process with the property that, given
the value of X the values of X, for s > t are not influenced by the values
of X for u < t. In words, the probability of any particular future behavior
of the process, when its current state is known exactly, is not altered by
additional knowledge concerning its past behavior. A discrete-time
Markov chain is a Markov process whose state space is a finite or countable set, and whose (time) index set is T = (0, 1, 2, ...). In formal terms,
the Markov property is that
Pr{X,,+ = jIXO = i0,
... ,
X = i}
(1.1)
for all time points n and all states i0 . ,
i, j.
It is frequently convenient to label the state space of the Markov chain
by the nonnegative integers {0, 1, 2, ...}, which we will do unless the
contrary is explicitly stated, and it is customary to speak of X,, as being in
state i if X,, = i.
The probability of X,,+, being in state j given that X,, is in state i is called
the one-step transition probability and is denoted by P;"+'. That is,
P;;"+' = Pr{
jIX,, = i}.
(1.2)
96
III
Markov Chains: Introduction
The notation emphasizes that in general the transition probabilities are
functions not only of the initial and final states, but also of the time of
transition as well. When the one-step transition probabilities are independent of the time variable n, we say that the Markov chain has stationary
transition probabilities. Since the vast majority of Markov chains that we
shall encounter have stationary transition probabilities, we limit our discussion to this case. Then P ,j',"" = Pij is independent of n, and Pij is the
conditional probability that the state value undergoes a transition from
i to j in one trial. It is customary to arrange these numbers Pij in a matrix,
i'n the infinite square array
P=
PO0
Poi
P02
P03
P10
Pn
P12
P13
P20
P,,
P22
P23
PO
P,,,
P, P,
and refer to P = (IP;jJJ as the Markov matrix or transition probability matrix of the process.
The ith row of P, for i = 0, 1, . . . , is the probability distribution of the
under the condition that X,, = i. If the number of states is
values of
finite, then P is a finite square matrix whose order (the number of rows)
is equal to the number of states. Clearly, the quantities Pij satisfy the
conditions
P, ?0
=1
fori,j=0,1,2,...,
for i = 0, 1, 2,
....
(1.3)
(1.4)
j=0
The condition (1.4) merely expresses the fact that some transition occurs
at each trial. (For convenience, one says that a transition has occurred
even if the state remains unchanged.)
A Markov process is completely defined once its transition probability
matrix and initial state X0 (or, more generally, the probability distribution
of X0) are specified. We shall now prove this fact.
1.
Definitions
97
Let Pr{Xo = i} = pi. It is enough to show how to compute the quantities
Pr{Xo = i0, X, = i X2 = i,, ... , X =
(1.5)
since any probability involving X;., ... , Xj,, for j, < ... < jR, can be obtained, according to the axiom of total probability, by summing terms of
the form (1.5).
By the definition of conditional probabilities we obtain
Pr{Xo = i0, X, = i,, X, = i,..... X = i,r}
(1.6)
X Pr{X =
i0, X, = i,,
. . . ,
I.
Now, by the definition of a Markov process,
Pr{X,, = i I XO = i0, X, = i
. . . ,
(1.7)
= Pr{X,r = i,r1 X,_ = i,r_, } = Pi,r-
Substituting (1.7) into (1.6) gives
Pr{XO=io, X,=i......
Pr{X0 = i0, X, = i,,
. .
Then, upon repeating the argument n - 1 additional times, (1.5) becomes
Pr{Xo = io, X, = i...... X,, =
(1.8)
= pi Po.i, ... pill-
This shows that all finite-dimensional probabilities are specified once the
transition probabilities and initial distribution are given, and in this sense
the process is defined by these quantities.
Related computations show that (1.1) is equivalent to the Markov property in the form
Pr{X,,, = ji,
=
io,
. . . ,
j,, .
i X"
. . ,
for all time points n, m and all states io,
... , X" = ill)
. . . ,
i,,,
1,
(1.9)
ill)
, j,,,.
In other words,
once (1.9) is established for the value m = 1, it holds for all m ? 1 as
well.
98
III
Markov Chains: Introduction
Exercises
1.1. A Markov chain X,,, X ... on states 0, 1, 2 has the transition probability matrix
0
1
2
0
0.1
0.2
0.7
P=1
0.9
0.1
0
2
0.1
0.8
0.1
and initial distribution p = Pr{Xo = 0) = 0.3, p, = Pr{Xp = 1) = 0.4, and
p, = Pr{X = 2) = 0.3. Determine Pr{X = 0, X, = 1, X, = 2).
1.2.
A Markov chain X, X, X,, . . . has the transition probability matrix
0
1
2
0
0.7
0.2
0.1
P=1
0
0.6
0.4
2
0.5
0
0.5
Determine the conditional probabilities
Pr{X, = 1, )(, = 1IX, = 0)
1.3.
and
Pr{X1 = 1, X, = 1IX = 01.
A Markov chain X,,, X,, X2.... has the transition probability matrix
0
1
2
0.6
0.3
0.1
P=1 0.3
0.3
0.4
2
0.1
0.5
0
0.4
If it is known that the process starts in state X = 1, determine the probability Pr{X = 1, X, = 0, X, = 2}.
99
Problems
1.4. A Markov chain X0, X,, X2, .
. .
has the transition probability matrix
0
1
2
0
0.1
0.1
0.8
P=1
0.2
0.2
0.6
2
0.3
0.3
0.4
Determine the conditional probabilities
Pr{X, = 1,X2= 1IX0 =0}
1.5.
and
A Markov chain Xo, X,, X2,
. . .
Pr{X2= 1,X3= IJX, =0}.
has the transition probability matrix
0
1
2
0
0.3
0.2
0.5
P=1
0.5
0.1
0.4
2
0.5
0.2
0.3
and initial distribution po = 0.5 and p, = 0.5. Determine the probabilities
Pr(X0 = 1, X, = 1, X2 = 0}
and
Pr{X1 = 1, X2 = 1, X3 = 0}.
Problems
1.1. A simplified model for the spread of a disease goes this way: The
total population size is N = 5, of which some are diseased and the remainder are healthy. During any single period of time, two people are selected at random from the population and assumed to interact. The selec-
tion is such that an encounter between any pair of individuals in the
population is just as likely as between any other pair. If one of these persons is diseased and the other not, then with probability a = 0.1 the disease is transmitted to the healthy person. Otherwise, no disease transmission takes place. Let X,, denote the number of diseased persons in the
population at the end of the nth period. Specify the transition probability
matrix.
100
III
Markov Chains: Introduction
1.2. Consider the problem of sending a binary message, 0 or 1, through
a signal channel consisting of several stages, where transmission through
each stage is subject to a fixed probability of error a. Suppose that X0 = 0
is the signal that is sent and let X,, be the signal that is received at the nth
is a Markov chain with transition probabilities
stage. Assume that
a andP0,=P10=a,where 0<a<1.
Poo
(a) Determine Pr{Xo = 0, X, = 0, X, = 0}, the probability that no error
occurs up to stage n = 2.
(b) Determine the probability that a correct signal is received at stage
2.
Hint: This is Pr{Xo=O,X,=0,X,=0}+Pr{Xo=0,X,=1,X,=0}.
1.3. Consider a sequence of items from a production process, with each
item being graded as good or defective. Suppose that a good item is followed by another good item with probability a and is followed by a defective item with probability 1 - a. Similarly, a defective item is followed
by another defective item with probability /3 and is followed by a good
item with probability 1 - /3. If the first item is good, what is the probability that the first defective item to appear is the fifth item?
1.4. The random variables ,, 6z, ... are independent and with the common probability mass function
k
Pr{6 = k} =
0
1
0.1
0.3
2
0.2
3
0.4
be the largest observed to date.
Set X0 = 0, and let X,, = max
Determine the transition probability matrix for the Markov chain {X,,}.
2.
Transition Probability Matrices of a Markov Chain
A Markov chain is completely defined by its one-step transition probability matrix and the specification of a probability distribution on the state of
the process at time 0. The analysis of a Markov chain concerns mainly the
calculation of the probabilities of the possible realizations of the process.
2.
Transition Probability Matrices of a Markov Chain
101
Central in these calculations are the n-step transition probability matrices
P'"' = IIP°;'M1. Here P;] denotes the probability that the process goes from
state i to state j in n transitions. Formally,
i).
P' = Pr(X,,,+,, =
(2.1)
Observe that we are dealing only with temporally homogeneous processes
having stationary transition probabilities, since otherwise the left side of
(2.1) would also depend on m.
The Markov property allows us to express (2.1) in terms of JJP I as
stated in the following theorem.
Theorem 2.1
The n-step transition probabilities of a Markov chain
satisfy
P;;
PA P1;
,
(2.2)
k=0
where we define
1
P;°' = {0
ifi=j,
ifi0j.
From the theory of matrices we recognize the relation (2.2) as the formula for matrix multiplication, so that P'"' = P x P("-''. By iterating this
formula, we obtain
P'"'=PXPX... XP=P";
(2.3)
n factors
in other words, the n-step transition probabilities P,] are the entries in the
matrix P", the nth power of P.
Proof The proof proceeds via a first step analysis, a breaking down, or
analysis, of the possible transitions on the first step, followed by an application of the Markov property. The event of going from state i to state j in
n transitions can be realized in the mutually exclusive ways of going to
some intermediate state k (k = 0, 1, ...) in the first transition, and then
going from state k to state j in the remaining (n - 1) transitions. Because
of the Markov property, the probability of the second transition is P(;-"
and that of the first is clearly P, . If we use the law of total probability, then
(2.2) follows. The steps are
102
Markov Chains: Introduction
III
M
P;;)=Pr{X,, =jjXo=i} =7 Pr{X =j, X. =kjXo=i}
k=0
x
Pr{X1 = kiX0 = i} Pr(X = jjX0 = i, X, = k)
k=0
X
Pk
Pkj(n-1)
k=0
If the probability of the process initially being in state j is pj, i.e., the
distribution law of X0 is Pr{Xo = j} = pj, then the probability of the
process being in state k at time n is
pR.° =
j=0 p-P;'k = Pr{X = k}.
(2.4)
Exercises
2.1. A Markov chain (X,,) on the states 0, 1, 2 has the transition probability matrix
0
1
2
0.1
0.2
0.7
0.2
0.2
0.6
0.6
0.1
0.3
(a) Compute the two-step transition matrix P2.
(b) What is Pr {)(, = 1I X, = 0}?
(c) What is Pr{X3 = 1 Xo = 0)?
2.2. A particle moves among the states 0, 1, 2 according to a Markov
process whose transition probability matrix is
0
P=1
2
0
1
2
0
2
z
2
0
1
2
2
1
2
0
Let X,, denote the position of the particle at the nth move. Calculate
Pr(X,, =OJXo= 0)forn=0, 1,2,3,4.
Exercises
103
2.3. A Markov chain X0, X,, X,.... has the transition probability matrix
0
1
2
0
0.7
0.2
0.1
P=1
0
0.6
0.4
2
0.5
0
0.5
Determine the conditional probabilities
Pr{X1 = 1IXo=01
Pr{X4 = 1jXo=01.
and
2.4. A Markov chain X0, X, X2.... has the transition probability matrix
0
1
2
0
0.6
0.3
0.1
P=1
0.3
0.3
0.4
2
0.4
0.1
0.5
If it is known that the process starts in state X0 = 1, determine the proba-
bility Pr(X, = 2).
2.5. A Markov chain X0, X,, X,,
. . .
has the transition probability matrix
0
1
2
0
0.1
0.1
0.8
P=1
0.2
0.2
0.6
2
0.3
0.3
0.4
Determine the conditional probabilities
Pr{(3 = 1JX, = 0)
2.6.
1X0 = 0).
and
A Markov chain X0, X,, X,.... has the transition probability matrix
0
1
2
0.3
0.2
0.5
P=1 0.5
0.1
0.4
2
0.2
0.3
0
0.5
104
Markov Chains: Introduction
III
and initial distribution po = 0.5 and p, = 0.5. Determine the probabilities
Pr{X7 = 0) and Pr{X3 = 01.
Problems
2.1. Consider the Markov chain whose transition probability matrix is
given by
0
1
2
3
0.4
0.3
0.2
0.1
1
0
1
0 4
03
.
02
2
0.3
0.2
0.1
0.4
3
0.2
0.1
0.4
0.3
0
P=
.
.
.
Suppose that the initial distribution is p; = 4 for i = 0, 1, 2, 3. Show that
Pr(X,, = k} = ;, k = 0, 1, 2, 3, for all n. Can you deduce a general result
from this example?
2.2. Consider the problem of sending a binary message, 0 or 1, through
a signal channel consisting of several stages, where transmission through
each stage is subject to a fixed probability of error a. Let X0 be the signal
that is sent and let X be the signal that is received at the nth stage. Suppose X,, is a Markov chain with transition probabilities P® = P = 1 - a
and Po, = PO = a, (0 < a < 1). Determine Pr{XS = OIXO = 0), the probability of correct transmission through five stages.
2.3. Let X,, denote the quality of the nth item produced by a production
system with X,, = 0 meaning "good" and X,, = 1 meaning "defective."
Suppose that X,, evolves as a Markov chain whose transition probability
matrix is
0
0.99
0.12
1
0.01
0.88
What is the probability that the fourth item is defective given that the first
item is defective?
3.
Some Markov Chain Models
105
2.4.
Suppose X,, is a two-state Markov chain whose transition probability matrix is
0
1
1-a
f
P=0 a
1-10
is a Markov chain having the four states (0, 0),
Then Z,, =
(0, 1), (1, 0), and (1, 1). Determine the transition probability matrix.
2.5. A Markov chain has the transition probability matrix
0
1
2
0
0.7
0.2
0.1
P=1
0.3
0.5
0.2
2
0
0
1
The Markov chain starts at time zero in state X0 = 0. Let
T=min{n? 0; X,, =2}
be the first time that the process reaches state 2. Eventually, the process
will reach and be absorbed into state 2. If in some experiment we observed
such a process and noted that absorption had not yet taken place, we
might be interested in the conditional probability that the process is in
state 0 (or 1), given that absorption had not yet taken place. Determine
Pr(X3=OJX0,T>3).
Hint:
The event IT > 3) is exactly the same as the event
{)(0 2) = {X3 = 0) U {X3 = I).
3.
Some Markov Chain Models
The importance of Markov chains lies in the large number of natural physical, biological, and economic phenomena that can be described by them
and is enhanced by the amenability of Markov chains to quantitative manipulation. In this section we give several examples of Markov chain models that arise in various parts of science. General methods for computing
certain functionals on Markov chains are derived in the following section.
106
3.1.
111
Markov Chains: Introduction
An Inventory Model
Consider a situation in which a commodity is stocked in order to satisfy a
continuing demand. We assume that the replenishment of stock takes
place at the end of periods labeled n = 0, 1, 2, . . . , and we assume that
the total aggregate demand for the commodity during period n is a random
variable ,, whose distribution function is independent of the time period,
fork=0,1,2,...,
Pr{6,=k} =a,
(3.1)
where a, ? 0 and I. _o a, = 1. The stock level is examined at the end of
each period. A replenishment policy is prescribed by specifying two nonnegative critical numbers s and S > s whose interpretation is, If the endof-period stock quantity is not greater than s, then an amount sufficient to
increase the quantity of stock on hand up to the level S is immediately procured. If, however, the available stock is in excess of s, then no replenishment of stock is undertaken. Let X,, denote the quantity on hand at the
end of period n just prior to restocking. The states of the process {X,,} consist of the possible values of stock size
S,S- 1,..., +1,0, -1, -2,...,
where a negative value is interpreted as an unfilled demand that will be
satisfied immediately upon restocking.
The process {X } is depicted in Figure 3.1.
S
X()
X,
x;
2
Figure 3.1
3
The inventory process.
Period
Some Markov Chain Models
3.
107
According the the rules of the inventory policy, the stock levels at two
consecutive periods are connected by the relation
X,+ =
where
,
JX - +,
is - 611
ifs < X < S,
(3.2)
if X --- s,
is the quantity demanded in the nth period, stipulated to follow
the probability law (3.1). If we assume that the successive demands
f 2.... are independent random variables, then the stock values X0, X,,
... constitute a Markov chain whose transition probability matrix can
be calculated in accordance with relation (3.2). Explicitly,
X2,
P,j = Pr{X,,+1 = jIX,, = i}
i - j}
Pr(,+S-j)
if s<i:S,
if i<s.
Consider a spare parts inventory model as a numerical example in
which either 0, 1, or 2 repair parts are demanded in any period, with
Pr(,, =0} = 0.5,
=2} =0.1,
1} = 0.4,
Pr
and suppose s = 0, while S = 2. The possible values for X,, are S = 2, 1, 0,
and -1. To illustrate the transition probability calculations, we will consider first the determination of PO = Pr{X,,+, = OIX,, = 1). When X,, = 1,
then no replenishment takes place and the next state
0 results when
the demand
1, and this occurs with probability P,o = 0.4. To illus-
trate another case, if X = 0, then instantaneous replenishment to S = 2
ensues, and a next period level of
0 results from the demand quantity +, = 2. The corresponding probability of this outcome yields
Poo = 0.1. Continuing in this manner, we obtain the transition probability
matrix
P=
-1
0
+1
+2
-1
0
0.1
0.4
0.5
0
0
0.1
0.4
0.5
+1
0.1
0.4
0.5
0
+2
0
0.1
0.4
0.5
108
III
Markov Chains: Introduction
Important quantities of interest in inventory models of this type are the
long-term fraction of periods in which demand is not met (X,, < 0) and
long-term average inventory level. Using the notation p(;' = Pr{X = j},
we give these quantities respectively as
l;,o p°° and lim,,,,,
1;,o jp1;''. This illustrates the importance of determining conditions under
which the probabilities p';) stabilize and approach limiting probabilities ir;
as n - x and of determining methods for calculating the limiting probabilities ar; when they exist. These topics are the subject of IV.
3.2.
The Ehrenfest Urn Model
A classical mathematical description of diffusion through a membrane is
the famous Ehrenfest urn model. Imagine two containers containing a
total of 2a balls (molecules). Suppose the first container, labeled A, holds
k balls and the second container, B, holds the remaining 2a - k balls. A
ball is selected at random (all selections are equally likely) from the totality of the 2a balls and moved to the other container. (A molecule diffuses
at random through the membrane.) Each selection generates a transition
of the process. Clearly, the balls fluctuate between the two containers with
an average drift from the urn with the excess numbers to the one with the
smaller concentration.
Let Y be the number of balls in urn A at the nth stage, and define
X = Y - a. Then {X,,} is a Markov chain on the states i = -a,
-a + 1, . . . , -1, 0, + 1, . . , a with transition probabilities
.
a-i
2a
a+i
2a
0
ifj=i+1,
ifj=i - 1,
otherwise.
An important quantity in the Ehrenfest urn model is the long-term, or
equilibrium, distribution of the number of balls in each urn.
3.3.
Markov Chains in Genetics
The following idealized genetics model was introduced by S. Wright to
investigate the fluctuation of gene frequency under the influence of muta-
3.
109
Some Markov Chain Models
tion and selection. We begin by describing the so-called simple haploid
model of random reproduction, disregarding mutation pressures and selective forces. We assume that we are dealing with a fixed population size
of 2N genes composed of type-a and type-A individuals. The makeup of
the next generation is determined by 2N independent Bernoulli trials as
follows: If the parent population consists of j a-genes and 2N - j A-genes,
then each trial results in a or A with probabilities
p;=
2
N,
q;= 1
- 2N'
respectively. Repeated selections are done with replacement. By this pro-
cedure we generate a Markov chain
where X is the number of
a-genes in the nth generation among a constant population size of 2N individuals. The state space contains the 2N + 1 values {0, 1, 2, ... , 2N).
The transition probability matrix is computed according to the binomial
distribution as
Pr{X+,==j}=P= (2N)pkq2Nk
(j, k = 0, 1, ... , 2N).
For some discussion of the biological justification of these postulates
we refer the reader to Fisher.'
Notice that states 0 and 2N are completely absorbing in the sense that
once X,, = 0 (or 2N) then X,,+k = 0 (or 2N, respectively) for all k ? 0. One
of the questions of interest is to determine the probability, under the condition X0 = i, that the population will attain fixation, i.e., that it will become a pure population composed only of a-genes or A-genes. It is also
pertinent to determine the rate of approach to fixation. We will examine
such questions in our general analysis of absorption probabilities.
A more complete model takes account of mutation pressures. We assume that prior to the formation of the new generation each gene has the
possibility to mutate, that is, to change into a gene of the other kind.
Specifically, we assume that for each gene the mutation a - A occurs
with probability a, and A -3 a occurs with probability /3. Again we assume that the composition of the next generation is determined by 2N
R. A. Fisher, The Genetical Theory of Natural Selection, Oxford (Clarendon) Press,
London and New York, 1962.
110
III
Markov Chains: Introduction
independent binomial trials. The relevant values of p; and q; when the parent population consists of j a-genes are now taken to be
p'
2N
(1 - a) +
(1
2N)R
and
(3.4)
q'
2N a+
(I
2N)(1-0).
The rationale is as follows: We assume that the mutation pressures operate first, after which a new gene is chosen by random selection from the
population. Now, the probability of selecting an a-gene after the mutation
forces have acted is just 1/(2N) times the number of a-genes present;
hence the average probability (averaged with respect to the possible mutations) is simply 1/(2N) times the average number of a-genes after mutation. But this average number is clearly j(1 - a) + (2N - j)/3, which leads
at once to (3.4).
The transition probabilities of the associated Markov chain are calculated by (3.3) using the values of p, and q, given in (3.4).
If a/3 > 0, then fixation will not occur in any state. Instead, as n - c,
the distribution function of X will approach a steady-state distribution
of a random variable f, where Pr{ = k} = Irk (k = 0, 1, 2, ... , 2N)
(Y.k=o Irk = 1, irk > 0). The distribution function of is called the steadystate gene frequency distribution.
We return to the simple random mating model and discuss the concept
of a selection force operating in favor of, say, a-genes. Suppose we wish
to impose a selective advantage for a-genes over A-genes so that the relative number of offspring have expectations proportional to 1 + s and 1,
respectively, where s is small and positive. We replace p! = j/(2N) and
q; = 1 - jl(2N) by
(1+s)j
p'
2N+sj'
q;= 1
p;'
and build the next generation by binomial sampling as before. If the parent population consisted of j a-genes, then in the next generation the expected population sizes of a-genes and A-genes, respectively, are
2N
(1 + s)j
2N + sj '
2N(2N - j)
2N + Si
Some Markov Chain Models
3.
111
The ratio of expected population size of a-genes to A-genes at the
(n + 1)st generation is
1+s
1
X
j
_ (1 + s 1( number of a-genes in the nth generation
2N - j
l
1
J number of A-genes in the nth generation),
which explains the meaning of selection.
3.4. A Discrete Queueing Markov Chain
Customers arrive for service and take their place in a waiting line. During
each period of time, a single customer is served, provided that at least one
customer is present. If no customer awaits service, then during this period
no service is performed. (We can imagine, for example, a taxi stand at
which a cab arrives at fixed time intervals to give service. If no one is present, the cab immediately departs.) During a service period new customers
may arrive. We suppose that the actual number of customers that arrive
during the nth period is a random variable , whose distribution is independent of the period and is given by
Pr{k customers arrive in a service period} = Pr{6,, = k} = ak,
. , where a, ? 0 and
a, = 1.
We also assume that ,, ,, . . . are independent random variables. The
for k = 0, 1,
.
.
state of the system at the start of each period is defined to be the number
of customers waiting in line for service. If the present state is i, then after
the lapse of one period the state is
c-1+6
-{6
ifi?1,
ifi=0,
(3.5)
where is the number of new customers having arrived in this period
while a single customer was served. In terms of the random variables of
the process we can express (3.5) formally as
X.+' = (X,, - 1)+ +,, ,
where Y+ = max{Y, 0}. In view of (3.5), the transition probability matrix
may be calculated easily, and we obtain
112
III
P=
Markov Chains: Introduction
ao
a,
a,
a3
a4
ao
a,
a,
a3
a4
0
a a,
a,
a3
0
0
a,
a,
a2
0
0
0
a a,
It is intuitively clear that if the expected number of new customers,
JA_, kak, that arrive during a service period exceeds one, then with the passage of time the length of the waiting line increases without limit. On the
other hand, if Ik=0 kak <1, then the length of the waiting line approaches
a statistical equilibrium that is described by a limiting distribution
7rk>0,
fork=0, 1,...,
where Jk=o Irk = 1. Important quantities to be determined by this model
include the long run fraction of time that the service facility is idle, given
by iro, and the long run mean time that a customer spends in the system,
given by Ik= (1 + k) ark .
Exercises
3.1. Consider a spare parts inventory model in which either 0, 1, or 2 repair parts are demanded in any period, with
0.4,
Pr{,, = 1} = 0.3,
=0.3,
and suppose s = 0 and S = 3. Determine the transition probability matrix
for the Markov chain
where X,, is defined to be the quantity on hand
at the end of period n.
3.2. Consider two urns A and B containing a total of N balls. An experiment is performed in which a ball is selected at random (all selections
equally likely) at time t (t = 1, 2, . . .) from among the totality of N balls.
Then an urn is selected at random (A is chosen with probability p and B
is chosen with probability q) and the ball previously drawn is placed in
this urn. The state of the system at each trial is represented by the number
of balls in A. Determine the transition matrix for this Markov chain.
113
Problems
3.3. Consider the inventory model of Section 3.1. Suppose that S = 3.
Set up the corresponding transition probability matrix for the end-ofperiod inventory level X,,.
3.4. Consider the inventory model of Section 3.1. Suppose that S = 3
and that the probability distribution for demand is Pr{6 = 01 = 0.1,
Pr{ = 1) = 0.4, Pr{ = 21 = 0.3, and Pr{ = 3) = 0.2. Set up the corresponding transition probability matrix for the end-of-period inventory
level X,,.
3.5.
An urn initially contains a single red ball and a single green ball. A
ball is drawn at random, removed, and replaced by a ball of the opposite
color, and this process repeats so that there are always exactly two balls in
the urn. Let X be the number of red balls in the urn after n draws, with
Xo = 1. Specify the transition probabilities for the Markov chain
Problems
3.1. An urn contains six tags, of which three are red and three green.
Two tags are selected from the urn. If one tag is red and the other is green,
then the selected tags are discarded and two blue tags are returned to the
urn. Otherwise, the selected tags are returned to the urn. This process repeats until the urn contains only blue tags. Let X,, denote the number of red
tags in the urn after the nth draw, with Xo = 3. (This is an elementary
model of a chemical reaction in which red and green atoms combine to
form a blue molecule.) Give the transition probability matrix.
3.2. Three fair coins are tossed, and we let X, denote the number of
heads that appear. Those coins that were heads on the first trial (there were
X, of them) we pick up and toss again, and now we let X, be the total number of tails, including those left from the first toss. We toss again all coins
showing tails, and let X3 be the resulting total number of heads, including
those left from the previous toss. We continue the process. The pattern is,
Count heads, toss heads, count tails, toss tails, count heads, toss heads,
etc., and Xo = 3. Then
ability matrix?
is a Markov chain. What is the transition prob-
114
III
Markov Chains: Introduction
3.3. Consider the inventory model of Section 3.1. Suppose that unfulfilled demand is not back ordered but is lost.
(a) Set up the corresponding transition probability matrix for the endof-period inventory level X,,.
(b) Express the long run fraction of lost demand in terms of the demand distribution and limiting probabilities for the end-of-period
inventory.
3.4.
Consider the queueing model of Section 3.4. Now suppose that at
most a single customer arrives during a single period, but that the service
time of a customer is a random variable Z with the geometric probability
distribution
Pr{Z = k} = all - a)k-'
for k = 1, 2.
... .
Specify the transition probabilities for the Markov chain whose state is the
number of customers waiting for service or being served at the start of
each period. Assume that the probability that a customer arrives in a period is /3 and that no customer arrives with probability 1 - /3.
3.5. You are going to successively flip a quarter until the pattern HHT
appears; that is, until you observe two successive heads followed by a
tails. In order to calculate some properties of this game, you set up a
Markov chain with the following states: 0, H, HH, and HHT, where 0 represents the starting point, H represents a single observed head on the last
flip, HH represents two successive heads on the last two flips, and HHT is
the sequence that you are looking for. Observe that if you have just tossed
a tails, followed by a heads, a next toss of a tails effectively starts you over
again in your quest for the HHT sequence. Set up the transition probability matrix.
3.6. Two teams, A and B, are to play a best of seven series of games.
Suppose that the outcomes of successive games are independent, and each
is won by A with probability p and won by B with probability 1 - p. Let
the state of the system be represented by the pair (a, b), where a is the
number of games won by A, and b is the number of games won by B.
Specify the transition probability matrix. Note that a + b s 7 and that the
series ends whenever a = 4 or b = 4.
Problems
115
3.7. A component in a system is placed into service, where it operates
until its failure, whereupon it is replaced at the end of the period with a
new component having statistically identical properties, and the process
repeats. The probability that a component lasts for k periods is ak, for
k = 1, 2, .... Let X,, be the remaining life of the component in service at
the end of period n. Then X,, = 0 means that X,, will be the total operating life of the next component. Give the transition probabilities for the
Markov chain
3.8. Two urns A and B contain a total of N balls. Assume that at time t
there were exactly k balls in A. At time t + 1 an urn is selected at random
in proportion to its contents (i.e., A is chosen with probability k/N and B
is chosen with probability (N - k)/N). Then a ball is selected from A with
probability p or from B with probability q and placed in the previously
chosen urn. Determine the transition matrix for this Markov chain.
3.9.
Suppose that two urns A and B contain a total of N balls. Assume
that at time t there are exactly k balls in A. At time t + 1 a ball and an urn
are chosen with probability depending on the contents of the urn (i.e., a
ball is chosen from A with probability k1N or from B with probability
(N - k)/N). Then the ball is placed into one of the urns, where urn A is
chosen with probability k/N or urn B is chosen with probability (N - k)/N.
Determine the transition matrix of the Markov chain with states represented by the contents of A.
3.10. Consider a discrete-time, periodic review inventory model and let
6, be the total demand in period n, and let X,, be the inventory quantity on
hand at the end-of-period n. An (s, S) inventory policy is used: If the endof-period stock is not greater than s, then a quantity is instantly procured
to bring the level up to S. If the end-of-period stock exceeds s, then no replenishment takes place.
(a) Suppose that s = 1, S = 4, and X0 = S = 4. If the period demands
turn out to be
2,62
2,
2, what are the end-of-period stock levels X,, for
periods n = 1, 2, ... , 8?
(b) Suppose 6,, 62, . . . are independent random variables where
Pr{6,,=0) =0.1,Pr(, = 1) =0.3,Pr{e,,=2} =0.3,Pr{6,,=3} =
116
Markov Chains: Introduction
III
0.2, and Pr{,, = 4} = 0.1. Then X0, X
.
.
.
is a Markov chain. De-
termine P41 and P.
4.
First Step Analysis
A surprising number of functionals on a Markov chain can be evaluated
by a technique that we call first step analysis. This method proceeds by
analyzing, or breaking down, the possibilities that can arise at the end of
the first transition, and then invoking the law of total probability coupled
with the Markov property to establish a characterizing relationship among
the unknown variables. We first applied this technique in Theorem 2.1. In
this section we develop a series of applications of the technique.
4.1.
Simple First Step Analyses
whose transition probability matrix is
Consider the Markov chain
0
1
2
1
0
0
P=1 a
/3
y
0
0
1
0
2
where a > 0, /3 > 0, y > 0, and a + /3 + y = 1. If the Markov chain begins in state 1, it remains there for a random duration and then proceeds either to state 0 or to state 2, where it is trapped or absorbed. That is, once in
state 0 the process remains there for ever after, as it also does in state 2. Two
questions arise: In which state, 0 or 2, is the process ultimately trapped, and
how long, on the average, does it take to reach one of these states? Both
questions are easily answered by instituting a first step analysis.
We begin by more precisely defining the questions. Let
T=min{n? 0;X,, =OorX,, =2}
be the time of absorption of the process. In terms of this random absorption time, the two questions ask us to find
4.
First Step Analysis
117
u = Pr{XT = OIX0 = 1}
and
v=E[TIXO= 1].
We proceed to institute a first step analysis, considering separately the
three contingencies X, = 0, X, = 1, and X, = 2, with respective probabilities a, /3, and y. Consider u = Pr{XT = OIXO = 1}. If X, = 0, which occurs with probability a, then T = 1 and XT = 0. If X, = 2, which occurs
with probability y, then again T = 1, but XT = 2. Finally, if X, = 1, which
occurs with probability /3, then the process returns to state 1 and the problem repeats from the same state as before. In symbols, we claim that
Pr{XT = OIX, = 0) = 1,
Pr{XT = OIX, = 2) = 0,
Pr{XT = OIX, = 1 } = u,
which inserted into the law of total probability gives
u = Pr{XT = 0jXo = 1 }
2
_ > Pr{XT = 01Xo = 1, X, = k} Pr{X, = k1X0 = 1)
k=0
2
Pr{XT = O1X, = k} Pr{X1 = kjX0 = 1)
k=O
(by the Markov property)
= 1(a) + u(/3) + 0(y).
Thus we obtain the equation
u=a+/3u,
(4.1)
which gives
u=
a
1-
_
a
a+ y
Observe that this quantity is the conditional probability of a transition to
0, given that a transition to 0 or 2 occurred. That is, the answer makes
sense.
118
III
Markov Chains: Introduction
We turn to determining the mean time to absorption, again analyzing
the possibilities arising on the first step. The absorption time T is always
at least 1. If either X, = 0 or X, = 2, then no further steps are required. If,
on the other hand, X, = 1, then the process is back at its starting point, and
on the average, v = E[TIXo = 1] additional steps are required for absorption. Weighting these contingencies by their respective probabilities, we
obtain for v = E[TI Xo = 1],
v = 1 + a(0) + /3(v) + y(0)
(4.2)
which gives
v=
1
1-/3
In the example just studied, the reader is invited to verify that T has the
geometric distribution in which
fork=0, 1,...,
Pr{T>klXo= 1} =/31
and therefore
W
1
E[TIXO = 1] = > Pr(T > klX0 = 1) _ 1 - R .
k=O
That is, a direct calculation verifies the result of the first step analysis. Unfortunately, in more general Markov chains a direct calculation is rarely
possible, and first step analysis provides the only solution technique.
A significant extension occurs when we move up to the four-state
Markov chain whose transition probability matrix is
0
1
2
3
0
1
0
0
0
P=1
P,o
P11
'I2
'l3
2
P20
P21
P22
P23
3
0
0
0
1
Absorption now occurs in states 0 and 3, and states 1 and 2 are "transient."
The probability of ultimate absorption in state 0, say, now depends on the
4.
First Step Analysis
119
transient state in which the process began. Accordingly, we must extend
our notation to include the starting state. Let
T=min{n? 0;
for i = 1, 2,
and
for i = 1, 2.
v; = E[TI Xo = i]
We may extend the definitions for u; and v; in a consistent and commonsense manner by prescribing u0 = 1, u, = 0, and v = v1 = 0.
The first step analysis now requires us to consider the two possible
starting states X0, = I and Xo = 2 separately. Considering X. = I and applying a first step analysis to u, = Pr{XT = OIX = 1), we obtain
it, = P,a +
(4.3)
P12u,.
The three terms on the right correspond to the contingencies X, = 0,
X, = 1, and X, = 2, respectively, with the conditional probabilities
Pr{XT = OIX, = 0} = 1,
Pr{XT = OIX, = 1) = u,,
and
Pr (XT = OI X, = 2) = u,.
The law of total probability then applies to give (4.3), just as it was used
in obtaining (4.1). A similar equation is obtained for u,:
U, = P,0 + P,,u, +
(4.4)
The two equations in u, and u, are now solved simultaneously. To give a
numerical example, we will suppose
0
1
2
3
1
0
0
0
1
0.4
0.3
0.2
0.1
2
0.1
0.3
0.3
0.3
3
0
0
0
1
0
P
(4.5)
120
III
Markov Chains: Introduction
The first step analysis equations (4.3) and (4.4) for u, and u, are
u, = 0.4 + 0.3u, + 0.2u,,
u, = 0.1 + 0.3u, + 0.3u,,
or
0.7u, - 0.2u, = 0.4,
-0.3u, + 0.7u, = 0.1.
The solution is u, = ° 4and u, _9. Note that one cannot, in general, solve
for u, without bringing in u,, and vice versa. The result u, = ',; tells us that
described by (4.5) will
once begun in state Xo = 2, the Markov chain
ultimately end up in state 0 with probability u, = 13 , and alternatively, will
be absorbed in state 3 with probability I - u, =41.
The mean time to absorption also depends on the starting state. The first
step analysis equations for v; = E[TI XO = i] are
P,,v,,
v, = I +
v, = 1 + -',v, +
(4.6)
The right side of (4.6) asserts that at least one step is always taken. If the
first move is to either X, = 1 or X, = 2, then additional steps are needed,
and on the average, these are v, and v,, respectively. Weighting the contingencies X, = 1 and X, = 2 by their respective probabilities and summing according to the law of total probability results in (4.6).
For the transition matrix given in (4.5), the equations are
v, = 1 + 0.3v, + 0.2v,,
v, = I + 0.3v, + 0.3v,,
and their solutions are v, _43and v,-
4.1A gain, v, cannot be obtained without also considering v,, and vice versa. For a process that begins in state Xo
= 2, on the average v, = 43= 2.33 steps will transpire prior to absorption.
be a finite-state
To study the method in a more general context, let
Markov chain whose states are labeled 0, 1, . . . , N. Suppose that states
0, 1,...,r- 1aretransient2inthat P:.7- 0asn-4oc for0:5 i,j<r,
while states r, . . . , N are absorbing (P; = I for r < i !5; N). The transition
matrix has the form
' The definition of a transient state is different for an infinite-state Markov chain. See IV,
Section 3.
4.
First Step Analysis
121
P=
(4.7)
'1 [
I '0
where 0 is an (N - r + 1) X r matrix all of whose entries are zero, I is an
(N - r + 1) X (N - r + 1) identity matrix, and Q;j = Pj for 0 : i, j < r.
Started at one of the transient states X0 = i, where 0 << i < r, such
a process will remain in the transient states for some random duration,
but ultimately the process gets trapped in one of the absorbing states
i = r,
. . . , N. Functionals of importance are the mean duration until absorption and the probability distribution over the states in which absorption takes place.
Let us consider the second question first and fix a state k among the absorbing states (r : k s N). The probability of ultimate absorption in state
k, as opposed to some other absorbing state, depends on the initial state
Xo = i. Let Uk = u, denote this probability, where we suppress the target
state k in the notation for typographical convenience.
We begin a first step analysis by enumerating the possibilities in the
first transition. Starting from state i, then with probability Pk the process
immediately goes to state k, thereafter to remain, and this is the first possibility considered. Alternatively, the process could move on its first step
to an absorbing state j : k, where r <_ j : N, in which case ultimate absorption in state k is precluded. Finally, the process could move to a transient state j < r. Because of the Markov property, once in state j, then the
probability of ultimate absorption in state k is uj = Uk by definition.
Weighting the enumerated possibilities by their respective probabilities
via the law of total probability, we obtain the relation
U. = Pr{Absorption in klX0 = i}
Pr{Absorption in kIX0 = i, X, = j}pj
j=o
N
r-
j=r
j=o
=Pi
P
j*k
To summarize, for a fixed absorbing state k, the quantities
u; = Uk = Pr{Absorption in klX0 = i }
for 0 < i < r
122
Markov Chains: Introduction
III
satisfy the inhomogeneous system of linear equations
I.- I
i=0, 1,...,r- 1.
UR=PA+7' P;jUk,
j=0
(4.8)
Example A Maze A white rat is put into the maze shown:
__F_77
1
0
1
food
2
8
3
4
5
6
'
shock
In the absence of learning, one might hypothesize that the rat would move
through the maze at random; i.e., if there are k ways to leave a compartment, then the rat would choose each of these with probability 1/k. Assume that the rat makes one change to some adjacent compartment at each
unit of time and let X,, denote the compartment occupied at stage n. We
suppose that compartment 7 contains food and compartment 8 contains an
electrical shocking mechanism, and we ask the probablity that the rat,
moving at random, encounters the food before being shocked. The appropriate transition probability matrix is
0
0
1
1
2
2 3 4 5 6 7 8
2
3
;
3
3
3
3
3
3
1
1
4.
First Step Analysis
123
Let u; = u;(7) denote the probability of absorption in the food compartment 7, given that the rat is dropped initially in compartment i. Then equations (4.8) become, in this particular instance,
+ lug,
U0 =
Zu,
1
I
1
ul - 3 + 3u0
+ 3u3,
I
1
U2 =
+ 3u31
3uo
4U1 +
U3 =
+ 4U4 + 4U5,
4u2
I
I
1
u4 = 3
+ 3u3
+ 3u6+
1
U5 =
+
3u3
U6 =
I3u6e
?u4 + 12U5-
Turning to the solution, we see that the symmetry of the maze implies that
uo = u6, u2 = u5, and u, = u4. We also must have u3 =!. With these simplifications the equations for uo, u and u2 become
U0 =
?ul + ;U2,
u1 = z +
13uo+
U,
2
=6+luo,
and the natural substitutions give uo = ;Z + lu0) +
z anduz = 3.
U, _-3,
1
Z(6
+ luo), or uo = ;,
One might compare these theoretical values under random moves with
actual observations as an indication of whether or not learning is taking
place.
4.2.
The General Absorbing Markov Chain
Let (X,, } be a Markov chain whose transition probability matrix takes the
form (4.7). We turn to a more general form of the first question by introducing the random absorption time T. Formally, we define
T=min{n? 0;X?r}.
124
III
Markov Chains: Introduction
Let us suppose that associated with each transient state i is a rate g(i) and
that we wish to determine the mean total rate that is accumulated up to absorption. Let w, be this mean total amount, where the subscript i denotes
the starting position X0 = i. To be precise, let
T-I
w; = E[
g(X,,)IX0 = i
,1=0
The choice g(i) = 1 for all i yields ,T=o g(X,,) _ T=o 1 = T, and then
w; is identical to v, = E[TIXO = i], the mean time until absorption. For a
transient state k, the choice
ifi=k,
ifi*k,
1
g(i)={0
gives w, = Wk, the mean number of visits to state k (0 <- k < r) prior to
absorption.
We again proceed via a first step analysis. The sum YT=p
always
includes the first term g(X0) = g(i). In addition, if a transition is made
from i to a transient state j, then the sum includes future terms as well. By
invoking the Markov property we deduce that this future sum proceeding
from state j has an expected value equal to w;. Weighting this by the transition probability P, and then summing all contributions in accordance
with the law of total probability, we obtain the joint relations
r-1
for i=0,...,r- 1.
w;=g(i)+Y, P,w;
(4.9)
i=o
The special case in which g(i) = 1 for all i determines v, = E[TI`Xo = i]
as solving
1.- 1
v;=1+7, P;v;
fori=0,1,...,r-1.
l=o
The case in which
1
S(i)=Sk={0
ifi=k,
ifi0k,
(4.10)
4.
First Step Analysis
125
determines Wk, the mean number of visits to state k prior to absorption
starting from state i, as solving
,.-1
Wk= 6;k+7 P,Wk
fori=0, 1,...,r- 1.
(4.11)
j=0
Example A Model of Fecundity Changes in sociological patterns
such as increase in age at marriage, more remarriages after widowhood,
and increased divorce rates have profound effects on overall population
growth rates. Here we attempt to model the life span of a female in a population in order to provide a framework for analyzing the effect of social
changes on average fecundity.
The general model we propose has a large number of states delimiting
the age and status of a typical female in the population. For example, we
begin with the twelve age groups 0-4 years, 5-9 years, . . . , 50-54 years,
55 years and over. In addition, each of these age groups might be further
subdivided according to marital status: single, married, separated, divorced, or widowed, and might also be subdivided according to the number of children. Each female would begin in the (0-4, single) category and
end in a distinguished state A corresponding to death or emigration from
the population. However, the duration spent in the various other states
might differ among different females. Of interest is the mean duration
spent in the categories of maximum fertility, or more generally a mean
sum of durations weighted by appropriate fecundity rates.
When there are a large number of states in the model, as just sketched,
the relevant calculations require a computer. We turn to a simpler model
which, while less realistic, will serve to illustrate the concepts and approach. We introduce the states
E0: Prepuberty,
E,: Divorced,
E,: Single,
E,: Widowed,
E,: Married,
Es: A,
and we are interested in the mean duration spent in state E,: Married, since
this corresponds to the state of maximum fecundity. To illustrate the computations, we will suppose the transition probability matrix is
126
III
Markov Chains: Introduction
Eo
E,
E2
E3
E,
E5
Eo
0
0.9
0
0
0
0.1
E,
0
0.5
0.4
0
0
0.1
E2
0
0
0.6
0.2
0.1
0.1
E3
0
0
0.4
0.5
0
0.1
E,
0
0
0.4
0
0.5
0.1
E5
0
0
0
0
0
1.0
P =
In practice, such a matrix would be estimated from demographic data.
Every person begins in state Eo and ends in state E5, but a variety of intervening states may be visited. We wish to determine the mean duration
spent in state E2: Married. The powerful approach of first step analysis begins by considering the slightly more general problem in which the initial
state is varied. Let w; = W,,2 be the mean duration in state E2 given the initial state X, = E; for i = 0, 1, . . . , 5. We are interested in wo, the mean duration corresponding to the initial state E0.
First step analysis breaks down, or analyzes, the possibilities arising in
the first transition, and using the Markov property, an equation that relates
w0, . . . , w5 results.
We begin by considering wo. From state E0, a transition to one of the
states E, or E5 occurs, and the mean duration spent in E2 starting from Eo
must be the appropriately weighted average of w, and w5. That is,
wa = 0.9w, + 0.1w5.
Proceeding in a similar manner, we obtain
w, = 0.5w, + 0.4w2 + O.lw5.
The situation changes when the process begins in state E2, because in
counting the mean duration spent in E2, we must count this initial visit
plus any subsequent visits that may occur. Thus for E2 we have
w2= 1 +0.6w2+0.2w3+0.1w,+0.1w5.
The other states give us
W3=0.4w,+0.5W3+O.IW51
w4 =0.4w2+0.5w,+0.1w5,
w5=w5.
Exercises
127
Since state E., corresponds to death, it is clear that we must have w, = 0.
With this prescription, the reduced equations become, after elementary
simplification,
-1.0w + 0.9w,
= 0,
-0.5w, + 0.4w,
= 0,
-0.4w,+0.2w1+0.1w4= -1,
0.4w, - 0.5w,
= 0,
-0.5w,=0.
0.4w,
The unique solution is
wo=4.5,
w,=5.00,
w3=w,=5.00.
w, = 6.25,
Each female, on the average, spends w, = W02 = 4.5 periods in the childbearing state E, during her lifetime.
Exercises
4.1. Find the mean time to reach state 3 starting from state 0 for the
Markov chain whose transition probability matrix is
0
1
2
3
0.4
0.3
0.2
0.1
0
0.7
0.2
0.1
2
0
0
0.9
0.1
3
0
0
0
1
0
P-
1
4.2.
Consider the Markov chain whose transition probablity matrix is
given by
0
1
2
0
1
0
0
P=1
0.1
0.6
0.3
2
0
0
1
128
Markov Chains: Introduction
III
(a) Starting in state 1, determine the probability that the Markov chain
ends in state 0.
(b) Determine the mean time to absorption.
4.3. Consider the Markov chain whose transition probability matrix is
given by
P_
0
1
2
3
1
0
0
0
1
02
1
0
1
0 6
0
2
0.2
0.3
0.4
0.1
3
0
0
0
1
.
.
.
.
(a) Starting in state 1, determine the probability that the Markov chain
ends in state 0.
(b) Determine the mean time to absorption.
4.4. A coin is tossed repeatedly until two successive heads appear. Find
the mean number of tosses required.
Hint: Let X,, be the cumulative number of successive heads. The state
space is 0, 1, 2 and the transition probability matrix is
Determine the mean time to reach state 2 starting from state 0 by invoking a first step analysis.
4.5. A coin is tossed repeatedly until either two successive heads appear
or two successive tails appear. Suppose the first coin toss results in a head.
Find the probability that the game ends with two successive tails.
Exercises
129
Consider the Markov chain whose transition probability matrix is
given by
4.6.
0
1
2
3
1
0
0
0
0
P_
1
0
1
04
0
1
04
2
0.2
0.1
0.6
0.1
3
0
0
0
1
.
.
.
.
(a) Starting in state 1, determine the probability that the Markov chain
ends in state 0.
(b) Determine the mean time to absorption.
4.7.
Consider the Markov chain whose transition probability matrix is
given by
0
P=
0
1
2
3
1
0
0
0
1
0
1
02
.
05
0 2
2
0.1
0.2
0.6
0.1
3
0
0
0
1
.
.
.
Starting in state 1, determine the mean time that the process spends in
state 1 prior to absorption and the mean time that the process spends in
state 2 prior to absorption. Verify that the sum of these is the mean time
to absorption.
4.8. Consider the Markov chain whose transition probability matrix is
given by
0
1
2
3
1
0
0
0
0.5
0.2
0.1
0.2
2
0.2
0.1
0.6
0.1
3
0
0
0
1
0
P
130
Markov Chains: Introduction
III
Starting in state 1, determine the mean time that the process spends in
state 1 prior to absorption and the mean time that the process spends in
state 2 prior to absorption. Verify that the sum of these is the mean time
to absorption.
4.9.
Consider the Markov chain whose transition probability matrix is
given by
0
P_
0
1
2
3
1
0
0
0
1
0
1
0 2
.
0 5
02
2
0.1
0.2
0.6
0.1
3
0
0
0
1
.
.
.
Starting in state 1, determine the probability that the process is absorbed
into state 0. Compare this with the (1,0)th entry in the matrix powers P-,
P4, P8, and P's.
Problems
Which will take fewer flips, on average: successively flipping a
quarter until the pattern HHT appears, that is, until you observe two successive heads followed by a tails; or successively flipping a quarter until
the pattern HTH appears? Can you explain why these are different?
4.1.
4.2. A zero-seeking device operates as follows: If it is in state m at time
n, then at time n + 1, its position is uniformly distributed over the states
0, 1, . . . , m - 1. Find the expected time until the device first hits zero
starting from state m.
Note:
This is a highly simplified model for an algorithm that seeks a
maximum over a finite set of points.
4.3. A zero-seeking device operates as follows: If it is in state j at time
n, then at time n + 1, its position is 0 with probability 1/j, and its position
is k (where k is one of the states 1, 2, . . . , j - 1) with probability 2k/j2.
Find the expected time until the device first hits zero starting from state m.
Problems
131
4.4. Consider the Markov chain whose transition probability matrix is
given by
0
1
2
3
1
0
0
0
0.1
0.2
0.5
0.2
0.1
0.2
0.6
0.1
0.2
0.2
0.3
0.3
0
P
2
Starting in state X0 = 1, determine the probability that the process never
visits state 2. Justify your answer.
4.5. A white rat is put into compartment 4 of the maze shown here:
3
2
1
4
food
5
6
7
shock
He moves through the compartments at random; i.e., if there are k ways to
leave a compartment, he chooses each of these with probability 1/k. What
is the probability that the rat finds the food in compartment 3 before feeling the electric shock in compartment 7?
4.6.
Consider the Markov chain whose transition matrix is
P=
0
1
2
3
4
0
q
p
0
0
0
1
q
0
p
0
0
g
0
0
p
0
3
q
0
0
0
p
4
0
0
0
0
1
2
132
III
Markov Chains: Introduction
where p + q = 1. Determine the mean time to reach state 4 starting from
state 0. That is, find E[TIXO = 0] where T = min{n ? 0; X,, = 4}.
Hint: Let v, = E[TIXO = i] for i = 0, 1, . . . , 4. Establish equations
for v0, v...... v4 by using a first step analysis and the boundary condition v4 = 0. Then solve for v0.
4.7. Let X,, be a Markov chain with transition probabilities P,. We are
given a "discount factor" 8 with 0 < P < 1 and a cost function c(i), and
we wish to determine the total expected discounted cost starting from state
i, defined by
rx
h; = EI
l
mcm')!Xa =
it
Using a first step analysis, show that h, satisfies the system of linear
equations
h; = c(i) + 0
P;h;
for all states i.
4.8.
An urn contains five red and three green balls. The balls are chosen
at random, one by one, from the urn. If a red ball is chosen, it is removed.
Any green ball that is chosen is returned to the urn. The selection process
continues until all of the red balls have been removed from the urn. What
is the mean duration of the game?
4.9. An urn contains five red and three yellow balls. The balls are chosen
at random, one by one, from the urn. Each ball removed is replaced in the
urn by a yellow ball. The selection process continues until all of the red balls
have been removed from the urn. What is the mean duration of the game?
4.10.
You have five fair coins. You toss them all so that they randomly
fall heads or tails. Those that fall tails in the first toss you pick up and toss
again. You toss again those that show tails after the second toss, and so on,
until all show heads. Let X be the number of coins involved in the last
toss. Find Pr(X = 1).
4.11.
An urn contains two red and two green balls. The balls are chosen
at random, one by one, and removed from the urn. The selection process
Problems
133
continues until all of the green balls have been removed from the urn.
What is the probability that a single red ball is in the urn at the time that
the last green ball is chosen?
4.12. A Markov chain X0, X, , X,.... has the transition probability
matrix
0
1
2
0
0.3
0.2
0.5
P=1
0.5
0.1
0.4
2
0
0
1
and is known to start in state X0 = 0. Eventually, the process will end up
in state 2. What is the probability that when the process moves into state
2, it does so from state 1?
Hint:
Let T = min(n ? 0; X,, = 2), and let
for i = 0, 1.
z; = Pr{XT_, = 1IX0 = i)
Establish and solve the first step equations
0.3zo + 0.2z
zo =
z,=0.4+0.5z()+O.lz,.
4.13.
A Markov chain X,,, X,, X,.... has the transition probability
matrix
0
1
2
0
0.3
0.2
0.5
P=1
0.5
0.1
0.4
2
0
0
1
and is known to start in state X = 0. Eventually, the process will end up
in state 2. What is the probability that the time T = min{n > 0; X,, = 2}
is an odd number?
134
III
Markov Chains: Introduction
4.14. A single die is rolled repeatedly. The game stops the first time that
the sum of two successive rolls is either 5 or 7. What is the probability that
the game stops at a sum of 5?
4.15. A simplified model for the spread of a rumor goes this way: There
are N = 5 people in a group of friends, of which some have heard the
rumor and the others have not. During any single period of time, two people are selected at random from the group and assumed to interact. The selection is such that an encounter between any pair of friends is just as
likely as between any other pair. If one of these persons has heard the
rumor and the other has not, then with probability a = 0.1 the rumor is
transmitted. Let X,, denote the number of friends who have heard the
rumor at the end of the nth period.
Assuming that the process begins at time 0 with a single person knowing the rumor, what is the mean time that it takes for everyone to hear it?
4.16.
An urn contains 5 tags, of which 3 are red and 2 are green. A tag
is randomly selected from the urn and replaced with a tag of the opposite
color. This continues until only tags of a single color remain in the urn.
Let X denote the number of red tags in the urn after the nth draw, with
Xo = 3. What is the probability that the game ends with the urn containing only red balls?
4.17.
The damage X of a system subjected to wear is a Markov chain
with the transition probability matrix
0
1
2
0
0.7
0.3
0
P=1
0
0.6
0.4
2
0
0
1
The system starts in state 0 and fails when it first reaches state 2. Let
T = min( n ? 0; X = 2) be the time of failure. Use a first step analysis to
evaluate ¢(s) = E[sn ] for a fixed number 0 < s < 1. (This is called the
generating function of T. See Section 9.)
4.18.
A well-disciplined
man, who smokes exactly one half of a cigar each day, buys a box conTime-dependent transition probabilities.
Some Special Markov Chains
5.
135
taining N cigars. He cuts a cigar in half, smokes half, and returns the other
half to the box. In general, on a day in which his cigar box contains w
whole cigars and h half cigars, he will pick one of the w + h smokes at
random, each whole and half cigar being equally likely, and if it is a half
cigar, he smokes it. If it is a whole cigar, he cuts it in half, smokes one
piece, and returns the other to the box. What is the expected value of T,
the day on which the last whole cigar is selected from the box?
Hint: Let X,, be the number of whole cigars in the box after the nth
smoke. Then X is a Markov chain whose transition probabilities vary
with n. Define v (w) = E[T I X,, = w]. Use a first step analysis to develop
a recursion for v (w) and show that the solution is
2Nw+n+2w
IA I
w + 1
whence
E[T] = vo(N) = 2N Computer Challenge. Let N be a positive integer and let
Z be independent random variables, each having the geometric
distribution
4.19.
Z
. . . ,
Pr{Z=k}
111
=(2I,
fork= 1,2.....
Since these are discrete random variables, the maximum among them may
be unique, or there may be ties for the maximum. Let p,,, be the probability that the maximum is unique. How does p,,, behave when N is large? (Al-
ternative formulation: You toss N dimes. Those that are heads you set
aside; those that are tails you toss again. You repeat this until all of the
coins are heads. Then p, is the probability that the last toss was of a single coin.)
5.
Some Special Markov Chains
We introduce several particular Markov chains that arise in a variety of
applications.
136
5.1.
Markov Chains: Introduction
III
The Two-State Markov Chain
Let
0
1
P=0 1-a
b
a
where 0 < a, b < 1,
1-b
(5.1)
be the transition matrix of a two-state Markov chain.
When a = 1 - b, so that the rows of P are the same, then the states
. are independent identically distributed random variables with
Pr(X,, = 0) = b and Pr(X = 1) = a. When a 0 1 - b, the probability dis-
X, X 2 ,..
at the previous stage.
For the two-state Markov chain, it is readily verified by induction that
the n-step transition matrix is given by
tribution for X,, varies depending on the outcome
b a + (1
1
P"=
a+b
- a - b)"
b a
a -a
(5.2)
a+b
-b
b
To verify this general formula, introduce the abbreviations
A=
b a
and B =
b a
a -a
-b
b
so that (5.2) can be written
(a+b)-'[A+(1 -a-b)"B)
P
Next, check the multiplications
b a
AP =
x
b a
1-a
a
b a
b
1-b
b a
=A
and
a -a x 1-a
BP =
-
1 -b
a-a2-ab a2-a+ab
-b+ab+bz -ab+b-b''11
-b
_
I
a
b
b
=(1-a-b)B.
5.
Some Special Markov Chains
137
Now (5.2) is easily seen to be true when n = 1, since then
P' =
1
b a
a+b
b a
+(1-a-b)
a -a
a+b -b b
b+a-a2-ab a-a+a2+ab
b-b+ab+b22 a+b-ab-b2
1-a
a
b
1-b
= P.
To complete an induction proof, assume that the formula is true for n.
Then
P"P = (a + b)-'[A + (1 - a - b)"B]P
= (a + b)-'[AP + (1 - a - b)"BP]
=(a+b)-'[A+(1 -a-b)"+'B]
= P"
We have verified that the formula holds for n + 1. It therefore is established for all n.
Note that 11 -a-bj<1 when 0 < a, b < 1, and thus 11 -a-b("- 0
as n -3
and
lim
P" =
"-Ix
(5.3)
This tells us that such a system, in the long run, will be in state 0 with
probability b/(a + b) and in state 1 with probability a/(a + b), irrespective of the initial state in which the system started.
For a numerical example, suppose that the items produced by a certain
worker are graded as defective or not, and that due to trends in raw material quality, whether or not a particular item is defective depends in part
on whether or not the previous item was defective. Let X,, denote the quality of the nth item with X,, = 0 meaning "good" and X,, = 1 meaning "defective." Suppose that {X } evolves as a Markov chain whose transition
matrix is
138
Markov Chains: Introduction
III
P=0
0
0.99
0.12
1
0.01
0.88
Defective items would tend to appear in bunches in the output of such a
system.
In the long run, the probability that an item produced by this system is
defective is given by al(a + b) = 0.01/(0.01 + 0.12) = 0.077.
Markov Chains Defined by
Independent Random Variables
5.2.
denote a discrete-valued random variable whose possible values are
the nonnegative integers and where Pr{4 = i } = a; ? 0, for i = 0, 1, ... ,
and
U a; = 1. Let 6,, 6,, ... , 6,,, ... represent independent observaLet
,
tions of .
We shall now describe three different Markov chains connected with
the sequence ,, 6,..... In each case the state space of the process is the
set of nonnegative integers.
Example Independent Random Variables Consider the process X,,,
n = 0, 1, 2, ... , defined by X,,
(X,, _ o prescribed). Its Markov matrix has the form
P=
(5.4)
That all rows are identical plainly expresses the fact that the random variable
is independent of X,,.
Example Successive Maxima The partial maxima of 6 ,,
a second important Markov chain. Let
for n = 1, 2,.
..,
... define
5.
139
Some Special Markov Chains
with 00 = 0. The process defined by X,, = 0,, is readily seen to be a Markov
allows the transition probamax { X,,,
chain, and the relation
bilities to be computed to be
P=
Ao
a,
a2
a3
0
A,
a2
a3
0
0
A2
a3
0
0
0
A3
(5.5)
+ a, for k = 0, 1, ... .
Suppose ,, 2, ... represent successive bids on a certain asset that is offered for sale. Then X = max { . . , &) is the maximum that is bid up
where Ak = ao +
.
to stage n. Suppose that the bid that is accepted is the first bid that equals
or exceeds a prescribed level M. The time of sale is the random variable
T = min{n >_ 1; X,, ? M}. A first step analysis shows that the mean
µ = E[T] satisfies
A=1+A
(5.6)
M},
or µ = 1/Pr{e, >_ M} = 1/(a,,, + a,,,+, + ). The first step analysis invoked in establishing (5.6) considers the two possibilities {6, < M} and
{6, > M}. With this breakdown, the law of total probability justifies the
sum
E[T] =
MI
M) + E[Tle, < M]
M}.
(5.7)
Clearly, E[TI , ? MI = 1, since no further bids are examined in this case.
On the other hand, when , < M we have the first bid, which was not accepted, plus some future bids. The future bids 2, 3, . . . have the same probabilistic properties as in the original problem, and they are examined until
the first acceptable bid appears. This reasoning leads to E[TI , < M] _
1 + µ. Substitution into (5.7) then yields (5.6) as follows:
E[T] = 1 X Pr{6, ? M} + (1 + µ)
M}
= I +µPr{6, <M}.
To restate the argument somewhat differently, one always examines the
first bid 6,. If , < M, then further bids are examined in a future that is
140
III
Markov Chains: Introduction
probabilistically similar to the original problem. That is, when 6, < M,
then on the average µ bids in addition to 6, must be examined before an
acceptable bid appears. Equation (5.6) results.
Example Partial Sums Another important Markov chain arises from
consideration of the successive partial sums i' of the ;, i.e.,
n1,2,...,
and by definition, 77, = 0. The process X,, = rl is readily seen to be a
Markov chain via
jIX, = i ... ,
i)
= Pr{ +I - .l - 4 1 = il, 62 = l2 - i,, . . , 6,, = i - i,,_1}
j - i}
(independence of 6,, 2, ...)
=
.
=Pr{Xi+1=jlX,, =i}.
The transition probability matrix is determined by
Pr{
,+, = j - i}
for j ? i,
for j < i,
_ jaj-i
0
where we have used the independence of the 6;.
Schematically, we have
P=
ao
a,
a,
a3
0
ao
a,
a2
0
0
ao
a,
(5.8)
If the possible values of the random variable 6 are permitted to be the
positive and negative integers, then the possible values of rl,, for each n
will be contained among the totality of all integers. Instead of labeling the
states conventionally by means of the nonnegative integers, it is more convenient to identify the state space with the totality of integers, since the
S.
Some Special Markov Chains
141
transition probability matrix will then appear in a more symmetric form.
The state space consists then of the values ... -2, -1, 0, 1, 2, ...
The transition probability matrix becomes
P=
a_,
ao
a,
a,
a-,
a-,
ao
a,
a,
a-3
a_,
a_,
ao
a,
where Pr{ = k} = a k for k = 0, ±1, ±2,
5.3.
... ,and ak L-0, 1:k=_= ak = 1.
One-Dimensional Random Walks
When we discuss random walks, it is an aid to intuition to speak about the
state of the system as the position of a moving "particle."
A one-dimensional random walk is a Markov chain whose state space
is a finite or infinite subset a, a + 1, . . . , b of the integers, in which the
particle, if it is in state i, can in a single transition either stay in i or move
to one of the neighboring states i - 1, i + 1. If the state space is taken as
the nonnegative integers, the transition matrix of a random walk has the
form
P=
i-1
0
1
2
0
ro
Po
0
1
q,
r,
P,
0
q2
r2
0
0
q;
2
i
...
...
0
0
i
i+l
r;
p,
...
...
...
0
(5.9)
where p,>O,q;>0,r,?O,and q;+r;+p;= 1,i= 1,2,...(i? 1),
po ? 0, ro ? 0, ro + po = 1. Specifically, if X,, = i, then for i ? 1,
142
Markov Chains: Introduction
III
i + 1 X,, = i} = p;,
=q;,
with the obvious modifications holding for i = 0.
The designation "random walk" seems apt, since a realization of the
process describes the path of a person (suitably intoxicated) moving randomly one step forward or backward.
The fortune of a player engaged in a series of contests is often depicted
by a random walk process. Specifically, suppose an individual (player A)
with fortune k plays a game against an infinitely rich adversary and has
probability pk of winning one unit and probability qk = 1 - p, (k ? 1) of
losing one unit in the next contest (the choice of the contest at each stage
may depend on his fortune), and ro = 1. The process X,,, where X,, represents his fortune after n contests, is clearly a random walk. Note that once
the state 0 is reached (i.e., player A is wiped out), the process remains in
that state. The event of reaching state k = 0 is commonly known as the
"gambler's ruin."
If the adversary, player B, also starts with a limited fortune l and player
A has an initial fortune k (k + l = N), then we may again consider the
Markov chain process X,, representing player A's fortune. However, the
states of the process are now restricted to the values 0, 1, 2, . . . , N. At any
trial, N - X,, is interpreted as player B's fortune. If we allow the possibility of neither player winning in a contest, the transition probability matrix
takes the form
0
1
2
3
1
0
0
0
q,
r,
PI
0
0
q2
r,
P2
qN-I
N
0
0
N
...
...
...
(5.10)
rN-I PN-1
0
1
Again p; (q;), i = 1, 2, . . . , N - 1, denotes the probability of player A's
fortune increasing (decreasing) by 1 at the subsequent trial when his
5.
Some Special Markov Chains
143
present fortune is i, and r, may be interpreted as the probability of a draw.
Note that, in accordance with the Markov chain given in (5.10), when
player A's fortune (the state of the process) reaches 0 or N it remains in
this same state forever. We say player A is ruined when the state of the
process reaches 0, and player B is ruined when the state of the process
reaches N.
The probability of gambler's ruin (for player'A) is derived in the next
section by solving a first step analysis. Some more complex functionals on
random walk processes are also derived there.
The random walk corresponding to Pk = p, qk = 1 - p = q for all
k ? 1 and ro = I describes the situation of identical contests. There is a
definite advantage to player A in each individual trial if p > q, and conversely, an advantage to player B if p < q. A "fair" contest corresponds to
p = q = ;. Suppose the total of both players' fortunes is N. Then the corresponding walk, where X,, is player A's fortune at stage n, has the transition probability matrix
0
1
2
1
0
0 0
0
2
g
N-1 N
3
0
0
0
0
0 p ..
0
0
0
p
0
1
g0p0
1
P=
0
N-1
0
0
00
N
0
0
0
0
..
...
(5.11)
Let u; = U,0 be the probability of gambler's ruin starting with the initial
fortune i. Then u, is the probability that the random walk reaches state 0
before reaching state N, starting from X. = i. The first step analysis of Section 4, as used in deriving equation (4.8), shows that these ruin probabilities satisfy
u; = pu,+, + qu;_,
for i = 1,
... , N - 1
(5.12)
together with the obvious boundary conditions
uo = 1
and
UN = 0.
These equations are solved in the next section following a straightforward but arduous method. There it is shown that the gambler's ruin
144
III
Markov Chains: Introduction
probabilities corresponding to the transition probability matrix given in
(5.11) are
u; = Pr{X reaches state 0 before state IVIX, = i}
when p=q=Z,
(5.13)
whenp 0 q.
The ruin probabilities u; given by (5.13) have the following interpretation. In a game in which player A begins with an initial fortune of i units
and player B begins with N - i units, the probability that player A loses
all his money before player B goes broke is given by u,, where p is the
probability that player A wins in a single contest. If player B is infinitely
rich (N - cc), then passing to the limit in (5.13) and using (q/p)" -4 00 as
N -+ oo if p < q, while (q/p)" -4 0 if p > q, we see that the ruin probabilities become
if psq,
if p > q.
(5.14)
(In passing to the limit, the case p = q = z must be treated separately.) We
see that ruin is certain (u; = 1) against an infinitely rich adversary when
the game is unfavorable (p < q), and even when the game is fair (p = q).
In a favorable game (p > q), starting with initial fortune i, then ruin occurs (player A goes broke) with probability (q/p)'. This ruin probability decreases as the initial fortune i increases. In a favorable game against an in-
finitely rich opponent, with probability 1 - (q/p)' player A's fortune
increases, in the long run, without limit.
More complex gambler's-ruin-type problems find practical relevance in
certain models describing the fluctuation of insurance company assets
over time.
Random walks are not only useful in simulating situations of gambling
but frequently serve as reasonable discrete approximations to physical
processes describing the motion of diffusing particles. If a particle is sub-
5.
Some Special Markov Chains
145
jected to collisions and random impulses, then its position fluctuates randomly, although the particle describes a continuous path. If the future position (i.e., its probability distribution) of the particle depends only on the
present position, then the process X where X, is the position at time t, is
Markov. A discrete approximation to such a continuous motion corresponds to a random walk. A classical discrete version of Brownian motion
(VIII) is provided by the symmetric random walk, By a symmetric random walk on the integers (say all the integers) we mean a Markov chain
with state space the totality of all integers and whose transition probability matrix has the elements
P; _
P
lt]=t+1,
p
r
ifj=i - 1,
ifj=i,
0
otherwise,
i,j=0,1,2,...,
where p > 0, r ? 0, and 2p + r = 1. Conventionally, "simple random
walk" refers only to the case r = 0, p = Z.
The classical simple random walk in n dimensions admits the following formulation. The state space is identified with the set of all integral
lattice points in E' (Euclidean n space); that is, a state is an n-tuple k =
(k,, k,, ... ,
of integers. The transition probability matrix is defined by
I4-k,l = 1,
i=o
otherwise.
Analogous to the one-dimensional case, the simple random walk in E'
represents a discrete version of n-dimensional Brownian motion.
5.4.
Success Runs
Consider a Markov chain on the nonnegative integers with transition
probability matrix of the form
146
III
Markov Chains: Introduction
0
1
2
3
4
0
po
qo
0
0
0
1
p,
r,
q,
0
0
P= 2
P2
0
r,
q2
0
3
p3
0
0
r3
q3
(5.15)
where q; > 0, p; > 0, and p, + q, + r; = 1 for i = 0, 1, 2,
.... The zero
state plays a distinguished role in that it can be reached in one transition
from any other state, while state i + 1 can be reached only from state i.
This example arises surprisingly often in applications and at the same
time is very easy to compute with. We will frequently illustrate concepts
and results in terms of it.
A special case of this transition matrix arises when one is dealing with
success runs resulting from repeated trials each of which admits two possible outcomes, success S or failure F. More explicitly, consider a sequence of trials with two possible outcomes S or F. Moreover, suppose
that in each trial, the probability of S is a and the probability of F is
0 = 1 - a. We say a success run of length r happened at trial n if the outcomes in the preceding r + 1 trials, including the present trial as the last,
were respectively F, S, S, . . . , S. Let us now label the present state of the
process by the length of the success run currently under way. In particular, if the last trial resulted in a failure, then the state is zero. Similarly,
when the preceding r + 1 trials in order have the outcomes F, S, S, ... ,
S, the state variable would carry the label r. The process is clearly Markov
(since the individual trials were independent of each other), and its transition matrix has the form (5.15), where
p,, /3, r,,= 0, and q = a
for n = 0, 1, 2,
... .
A second example is furnished by the current age in a renewal process.
Consider a light bulb whose lifetime, measured in discrete units, is a ran-
dom variable , where
Pr( =k}=a,>0
fork= 1,2,...,
a,=1.
k=1
5.
Some Special Markov Chains
147
Let each bulb be replaced by a new one when it bums out. Suppose the
first bulb lasts until time the second bulb until time , + 62, and the nth
2, ... are
bulb until time 6, + . + ,,, where the individual lifetimes
independent random variables each having the same distribution as Let
X,, be the age of the bulb in service at time n. This current age process is
depicted in Figure 5.1.
By convention we set X,, = 0 at the time of a failure.
The current age is a success run Markov process for which
ak+l
Pk =
ak+ I+ ak+2 + ...
rk=0,qk= 1
Pk+
fork=0,1.....
(5.16)
We reason as follows: The age process reverts to zero upon failure of the
item in service. Given that the age of the item in current service is k, then
failure occurs in the next time period with conditional probability
Pk = ak+1/(ak+I + ak+2 +
). Given that the item has survived k periods,
it survives at least to the next period with the remaining probability
qk = 1 - PkRenewal processes are extensively discussed in VII.
Figure 5.1
The current age X,, in a renewal process. Here f, = 3, 2 = 2,
ands,=3.
148
III
Markov Chains: Introduction
Exercises
5.1. The probability of the thrower winning in the dice game called
"craps" is p = 0.4929. Suppose Player A is the thrower and begins the
game with $5, and Player B, his opponent, begins with $10. What is the
probability that Player A goes bankrupt before Player B? Assume that the
bet is $1 per round.
Hint:
Use equation (5.13).
5.2. Determine the gambler's ruin probability for Player A when both
players begin with $50, bet $1 on each play, and where the win probability for Player A in each game is
(a) p = 0.49292929
(b) p = 0.5029237
(See II, Section 2.)
What are the gambler's ruin probabilities when each player begins with
$500?
5.3. Determine P" for n = 2, 3, 4, 5 for the Markov chain whose transition probability matrix is
P=
0.4
0.6
0.7
0.3
5.4. A coin is tossed repeatedly until three heads in a row appear. Let X,,
record the current number of successive heads that have appeared. That is,
X = 0 if the nth toss resulted in tails; X,, = 1 if the nth toss was heads and
the (n - 1)st toss was tails; and so on. Model X,, as a success runs Markov
chain by specifying the probabilities p, and q;.
5.5. Suppose that the items produced by a certain process are each
graded as defective or good, and that whether or not a particular item is
defective or good depends on the quality of the previous item. To be specific, suppose that a defective item is followed by another defective item
with probability 0.80, whereas a good item is followed by another good
item with probability 0.95. Suppose that the initial (zeroth) item is good.
Using equation (5.2), determine the probability that the eighth item is
149
Exercises
good, and verify this by computing the eighth matrix power of the transition probability matrix.
5.6. A baseball trading card that you have for sale may be quite valuable. Suppose that the successive bids ,, i2.... that you receive are independent random variables with the geometric distribution
for k = 0, 1....
Pr( = k) = 0.01(0.99)k
.
If you decide to accept any bid over $100, how many bids, on the average, will you receive before an acceptable bid appears?
Hint:
Review the discussion surrounding equation (5.6).
5.7. Consider the random walk Markov chain whose transition probability matrix is given by
0
1
2
3
1
0
0
0
0.3
0
0.7
0
2
0
0.3
0
0.7
3
0
0
0
1
0
P
Starting in state 1, determine the probability that the process is absorbed
into state 0. Do this first using the basic first step approach of equations
(4.3) and (4.4), and second using the particular results for a random walk
given in equation (5.13).
5.8.
As a special case, consider a discrete-time queueing model in
which at most a single customer arrives in any period and at most a single
customer completes service. Suppose that in any single period, a single
customer arrives with probability a, and no customers arrive with proba-
bility 1 - a. Provided that there are customers in the system, in a single
period a single customer completes service with probability f3, and no customers leave with probability 1 - (3. Then X,,, the number of customers in
the system at the end of period n, is a random walk in the sense of Section
5.3. Referring to equation (5.9), specify the transition probabilities p;, q;,
and r; for i = 0,1.....
150
III
Markov Chains: Introduction
5.9. In a simplified model of a certain television game show, suppose
that the contestant, having won k dollars, will at the next play have k + 1
dollars with probability q and be put out of the game and leave with nothing with probability p = 1 - q. Suppose that the contestant begins with
one dollar. Model her winnings after n plays as a success runs Markov
chain by specifying the transition probabilities p;, q;, and r; in equation
(5.15).
Problems
5.1. As a special case of the successive maxima Markov chain whose
transition probabilities are given in equation (5.5), consider the Markov
chain whose transition probability matrix is given by
P=
0
1
2
0
ao
a,
a,
a3
1
0
ao + a,
a2
a3
2
0
0
ao + a, + a2
a3
3
0
0
0
3
1
Starting in state 0, show that the mean time until absorption is vo = 1/a3.
5.2. A component of a computer has an active life, measured in discrete
units, that is a random variable T, where Pr{T = k} = a,, for k = 1, 2.... .
Suppose one starts with a fresh component, and each component is replaced by a new component upon failure. Let X,, be the age of the component in service at time n. Then
is a success runs Markov chain.
(a) Specify the probabilities p; and q,.
(b) A "planned replacement" policy calls for replacing the component
upon its failure or upon its reaching age N, whichever occurs first.
Specify the success runs probabilities p; and q; under the planned
replacement policy.
5.3. A Batch Processing Model.
Customers arrive at a facility and
wait there until K customers have accumulated. Upon the arrival of the
Kth customer, all are instantaneously served, and the process repeats. Let
6.
Functionals of Random Walks and Success Runs
eo, e
151
denote the arrivals in successive periods, assumed to be independent random variables whose distribution is given by
.
. .
1} = 1 -a,
Pr{6,=0}=a,
where 0 < a < 1. Let X,, denote the number of customers in the system at
is a Markov chain on the states 0, 1, . . . , K - 1. With
time n. Then
K = 3, give the transition probability matrix for
Be explicit about
any assumptions you make.
5.4. Martha has a fair die with the usual six sides. She throws the die
and records the number. She throws the die again and adds the second
number to the first. She repeats this until the cumulative sum of all the
tosses first exceeds 10. What is the probability that she stops at a cumulative sum of 13?
5.5 Let
be a random walk for which zero is an absorbing state and
such that from a positive state, the process is equally likely to go up or
down one unit. The transition probability matrix is given by (5.9) with
r0 = 1 and p, = q; = ; for i ? 1. (a) Show that
is a nonnegative martingale. (b) Use the maximal inequality II, (5.7) to limit the probability
that the process ever gets as high as N > 0.
6.
Functionals of Random Walks and Success Runs
Consider first the random walk on N + 1 states whose transition probability matrix is given by
3
N
000
0
g0p0
0
P= 2 0 g 0 p
0
NII0 0 0 0
1
0
0
1
1
1
2
152
III
Markov Chains: Introduction
"Gambler's ruin" is the event that the process reaches state 0 before reach-
ing state N. This event can be stated more formally if we introduce the
concept of hitting time. Let T be the (random) time that the process first
reaches, or hits, state 0 or N. In symbols,
T=min{n?O;X,, =0or
The random time T is shown in Figure 6.1 in a typical case.
Figure 6.1 The hitting time to 0 or N. As depicted here, state 0 was
reached first.
In terms of T, the event written as X, = 0 is the event of gambler's ruin,
and the probability of this event starting from the initial state k is
uk = Pr{X, = OjXO = k}.
Figure 6.2 shows the first step analysis that leads to the equations
fork= 1,...,N- 1,
uk=Puk+, +quk_
(6.1)
N
k+1
k
k-1
1
I
0
1
I
I
I
I
`'
V
I
I
n
Figure 6.2 First step analysis for the gambler's ruin problem.
6.
Functionals of Random Walks and Success Runs
153
with the obvious boundary conditions
uN = 0.
uo = 1,
Equations (6.1) yield to straightforward but tedious manipulations. Because the approach has considerable generality and arises frequently, it is
well worth pursuing in this simplest case.
We begin the solution by introducing the differences xk = uk - u,-, for
k = 1,
. . . ,
N. Using p + q = 1 to write uk = (p + q)uk = PUk + quk,
equations (6.1) become
k= 1;
0=p(u2-u1)-q(u,-uo)=px2-qx1;
k = 2;
0 = Plus - u2) - q(u2 - u) = Px3 - qx2;
k = 3;
0 = P(u4 - u) - q(u3 - u2) = Px4 - qx3;
k = N - 1;
0 = P(UN - UN-1)
q(UN-1 - UN-2) = PXN
qxN-1;
or
x2 = (q/P)x1,
x3 = (q/P)x2 = (q/P)2x1,
x4 = (q/p)x3 = (q/p)3x1
Xk = (q/p)xk-1 =
(glp)k-1x1,
XN = (q/p)xN-1 =
(q/P)N-1x1
We now recover u0, u ... , uN by invoking the conditions uo = 1, UN = 0
and summing the xk's:
X,=U1-UO=U1-1,
x2=u,-u1,
x1+x2=u2-1,
X3=U3- U2,
X, +X2+X3 =U3- 1,
X, +... +Xk=Uk - 1,
Xk=Uk-Uk_1,
XN = UN - UN-1
- UN-1,
X, + ... + XN = UN - 1 = -1-
154
Markov Chains: Introduction
III
The equation for general k gives
Uk=1+x,+x,+ +xk
= 1 + x, + (q/p)x, +
= 1 + [I + (qlp) + .
+ (q/p)k-'x,
+
(glp)A-
(6.2)
I ]xi,
which expresses uA in terms of the as yet undetermined x,. But u,,, = 0
gives
0 = I + [1 +(qlp)+...
+(q/p)"-,lx,,
or
x, =
1
I + (qlp) + ... + (qlp) -I ,
which substituted into (6.2) gives
u=1A
1 + (qlp) + ... + (glp)k-1
1 + (qlp) + ... + (qlp)'
The geometric series sums to
k
1 + (qlp) + ... +
(glp)k-
i=
ifp=q=j,
1 - (glp)k
1 - (qlp)
if p
q,
whence
1 - (k/N) = (N - k)/N
ux =
when p = q =4i21
1 - 1 - (glp)k _ (qlp)" - (qlp)"
1 - (glp)N
1-
when p 0 q.
(6.3)
(q/p)N
A similar approach works to evaluate the mean duration
v; = E[TIXO = i).
(6.4)
The time T is composed of a first step plus the remaining steps. With prob-
ability p the first step is to state i + 1, and then the remainder, on the
6.
Functlonals of Random Walks and Success Runs
155
average, is v,,., additional steps. With probability q the first step is to
i - 1 and then, on the average, there are v;_, further steps. Thus for the
mean duration a first step analysis leads to the equation
v; = 1 + pv;+I + gvi_1
ford = 1,
. . .
,
N - 1.
(6.5)
Of course, the game ends in states 0 and N, and thus
V0=O,
VN=O.
We will solve equations (6.5) when p = q = 2T he solution for other
values of p proceeds in a similar manner, and the solution for a general
random walk is given later in this section.
Again we introduce the differences xk = vk - vk_1 for k = 1, . . . , N,
writing (6.5) in the form
k = 1;
-1 =2(V2- VI) - i(VI - Vo)=12x2-12X1;
k=2;
-1 =2(V3- V2)-12(V2-V0 =12X3-12X2,
k = 3;
- 1 = 2(V4 - V3)
k = N - 1;
-1 = 2(VN - VN-1)
2(V3 - V2) = 2X4
2x3;
2(VN-I - VN-2) = 2XN
2XN-I
The right side forms a collapsing sum. Upon adding, we obtain
k=1;
k=2;
-1=2x2-2x1;
-2=2x3-2x1;
k = 3;
-3 = 12x4 - 2x1;
k=N- 1;
-(N- 1)=2xN-12x,.
The general line gives xk = x, - 2(k - 1) for k = 2, 3, ... , N. We return
to the vk's by means of
x1=v,-vo= v,;
x2=v2-v,;
X3=V3- V2;
Xk = Vk - Vk-1;
x,+x2=v2;
X1+X2+x3=V3;
XI + ... + Xk = Vk;
156
III
Markov Chains: Introduction
or
vk=kv, - 2[1 + 2 +
+ (k - 1)] =kv, - k(k - 1),
(6.6)
which gives Vk in terms of the as yet unknown v,. We impose the boundary condition vN, = 0 to obtain 0 = Nv, - N(N - 1) or v, _ (N - 1). Substituting this into (6.6) we obtain
vk = k(N - k),
k = 0, 1,
... , N,
(6.7)
for the mean duration of the game. Note that the mean duration is greatest for initial fortunes k that are midway between the boundaries 0 and N,
as we would expect.
6.1.
The General Random Walk
We give the results of similar derivations on the random walk whose transition matrix is
3
N
0
0
0
1
0
0
P= 2
0
N
0
0
0
0
1
where q, >0 andpk >0 fork = 1,...,N- 1. LetT=min(n? 0;
X = 0 or X = N) be the hitting time to states 0 and N.
Problems
6.1. The probability of gambler's ruin
u;=Prix, =OIXo=i]
(6.8)
satisfies the first step analysis equation
u;=q;u;_,+r;u;+p;u;+,
for i = 1,
and
U0= 1,
uN=0.
.
.
.
,
N - 1,
6.
Functionals of Random Walks and Success Runs
157
The solution is
i= 1,...,N- 1,
Pr + ... + PN-I
u'
1 + PI + P2 + ... + PN-I
(6.9)
where
_
glg2
.. qk
PIP2
Pk
k= 1,...,N- 1.
Pk
(6.10)
6.2. The mean hitting time
vk = E[TIXQ = k]
(6.11)
satisfies the equations
Vk = 1 + gkVk-I + rkvk + PkVk+I
and
vo = VN = 0.
(6.12)
The solution is
Vk-(
(DI +... +4)N-1
1+p,+...+PN-I)(1+PI+...+Pk-1)
fork= 1,...,N- 1,
+ Ik-
- (4), +
(6.13)
where p; is given in (6.10) and
q1
g2PI
q,P,-I
q.+...+
=q2...q_+q3
PI ... Pi
P,
P2
q,
P,-1P,
+
1
(6.14)
Pr
fori = 1,...,N- 1.
6.3.
Fix a state k, where 0 < k < N, and let Wk be the mean total visits
to state k starting from i. Formally, the definition is
rT-1
Wk = EI
n-0
1{X,, = k}IX0 = iJ,
where
1
={0
if X,, =k,
if
k.
(6.15)
158
Markov Chains: Introduction
III
Then Wik satisfies the equations
for i = 1, ... , N - I
Wk = Sik + q,W-I.k + riWk + PiW+I.k
and
Wok=WNk=0,
where
1
ifi=k,
{0
if i * k.
_
Sik
The solution is
( (1 + ... + Pi-1)(A + ... + PN-1) (
1 + -r
I
Wk =
+ PN-1
1
\gkPk- I
(1 + . . . + Pi-1)(Pk + - - - + PN-1)
1+
L
+pN_1
1
A-1)
-)
gkPk-I
Example As a sample calculation of these functionals, we consider the
special case in which the transition probabilities are the same from row to
row. That is, we study the random walk whose transition probability matrix is
0
1
2
3
N
0
1
0
0
0
0
1
g
r p 0
0
0 g r p
0
P= 2
N 0
0
0 0
1
with p > 0, q > 0, and p + q + r = 1. Let us abbreviate by setting
0 = (q/p), and then pk, as defined in (6.10), simplifies according to
Pk=
glgzgk
= 9
' Pk
P
PI Pz
.
k
=Ok
fork= 1,...,N-1.
6.
Functionals of Random Walks and Success Runs
159
The probability of gambler's ruin, as defined in (6.8) and evaluated in
(6.9), becomes
uk = Pr{XT = OIXO = k}
ek -
ON
e
if e = (qlp) = 1.
This, of course, agrees with the answer given in (6.3).
We turn to evaluating the mean time
fork= 1,...,N= 1
vk=E[TIXo=k]
by first substituting p; = 0' into (6.14) to obtain
1q +
=
1 +...+
qO
qe;-1 )e
1
1(ei+e'-,+...+6)
q
=
1(1+6+ +e'-')
p
when p = q(e = 1),
when p 0 q (e 0 1).
Now observe that
pq,,
160
111
Markov Chains: Introduction
so that (6.13) reduces to
Vk =
k (4:D, + ... + (DN-1)
- ((')I + ...
+ (I)k-1).
(6.17)
N
In order to continue, we need to simplify the terms of the form
(D, +
+ (D,-,. We consider the two cases B = (qlp) = 1 and
0 = (qlp) 0 1 separately.
When p = q, or equivalently, 0 = 1, then (D; = ilp, whence
1+...+(I-1)I(I-1)
2p
P
which inserted into (6.17) gives
v; = E[TlXo = it
=
N
=
1)1
N(N
i
J
L
-
2p
i(N - i)
i(i2 1)
(6.18)
P
if p = q.
2p
When p = 2, then v, = i(N - i) in agreement with (6.7).
When p * q, so that 0 = ql p # 1, then
1(1 - 6`
=p1-0
whence
[(1-1)-(0+02+...+61-1)l
P(1-6)
= p(1
1
0)
[(J
1) - 6(11
I
-e0
1
11
1J'
and
v;=E[TIXO=il
p(11 0)[N(1-0)
Al - 0)
H
c
1 - ON
p(11 0)['-(1-6)
Functionals of Random Walks and Success Runs
6.
161
when 0 = (qlp) 0 1.
Finally, we evaluate Wk, expressed verbally as the mean number of visits to state k starting from X0 = i and defined formally in (6.15). Again we
consider the two cases 0 = (qlp) = 1 and 0 = (qlp) 0 1.
When 0 = 1, then p;=0'=1
and (6.16) simplifies to
i(N - k)
for 0 < i:5 k < N,
Wk.
_
- k)
9
N
-(
-k)
_ k(N - i)
i(N - k)
max{o, i - k}
qN
q
for O < k < i < N,
qN
(6.19)
When 0 = (qlp) * 1, then p, = 0' and
Pk +
- + PN-1 =
1-0'
and
qPk-( = PPk =
p0k.
In this case, (6.16) simplifies to
Wk- (1 - 0')(0k- ON)(_1 l
(1-0)(1-0")\p0kl
and
Wk- (1- 0k)(0i- 0")
1
forO<k<i<N.
(1 - O)(1 - ON) \pokl
We may write the expression for Wk in a single line by introducing the
notation (i - k) + = max{0, i - k}. Then
Wk =
(1 - 0')(1 - ON-k)
P(1-0)(1-ON)
-
I - 0(i-k)+
P(1-0)
(6.20)
162
6.2.
III
Markov Chains: Introduction
Cash Management
Short-term cash management is the review and control of a corporation's
cash balances, short-term loan balances, and short term marketable security holdings. The objective is to maintain the smallest cash balances that
are adequate to meet future disbursements. The corporation cashier tries
to eliminate idle cash balances (by reducing short-term loans or buying
treasury bills, for example) but to cover potential cash shortages (by selling treasury bills or increasing short-term loans). The analogous problem
for an individual is to maintain an optimal balance between a checking
and a savings account.
In the absence of intervention, the corporation's cash level fluctuates
randomly as the result of many relatively small transactions. We model
this by dividing time into successive, equal length periods, each of short
duration, and by assuming that from period to period, the cash level
moves up or down one unit, each with probability one-half. Let X be the
cash on hand in period n. We are assuming that
is the random walk
in which
k ± 1I X,, = k} =
;.
The cashier's job is to intervene if the cash level ever gets too low or too
high. We consider cash management strategies that are specified by two pa-
rameters, s and El, where 0 < s < 9. The policy is as follows: If the cash
level ever drops to zero, then sell sufficient treasury bills to replenish the
cash level up to s. If the cash level ever increases up to `l, then invest in
treasury bills in order to reduce the cash level to s. A typical sequence of
cash levels
when s = 2 and 9' = 5 is depicted in Figure 6.3.
First
Second
Cycle
Cycle
Third
Cycle
n
Figure 6.3 Several typical cycles in a cash inventory model.
6.
Functionals of Random Walks and Success Runs
163
We see that the cash level fluctuates in a series of statistically similar
cycles, each cycle beginning with s units of cash on hand and ending at
the next intervention, whether a replenishment or reduction in cash. We
begin our study by evaluating the mean length of a cycle and the mean
total unit periods of cash on hand during a cycle. Later we use these quantities to evaluate the long run performance of the model.
Let T denote the random time at which the cash on hand first reaches
the level 0' or 0. That is, T is the time of the first transaction. Let v, =
E[TIXO = s] be the mean time to the first transaction, or the mean cycle
length. From (6.7) we have
vs = s(w - s).
(6.21)
Next, fix an arbitrary state k (0 < k < 0') and let Wk be the mean number of visits to k up to time T for a process starting at Xo = s. From (6.19)
we have
Wk = 21
(9' - k) - (s - k)+
J.
(6.22)
Using this we obtain the mean total unit periods of cash on hand up to
time T starting from X0 = s by weighting Wk by k and summing according
to
c3-1
W=7, kWk
k=1
-1
s-1
k=1
k=1
=2{ 1 k(9'-k)-I k(s-k){
- 2{ s [W(9' - 1)(`,' + 1)1 3l
=
3 [SP2
6
J
(6.23)
s(s - 1)(s + 1){
6
- SZ].
Having obtained these single cycle results, we will use them to evaluate the long run behavior of the model. Note that each cycle starts from
the cash level s, and thus the cycles are statistically independent. Let K be
the fixed cost of each transaction. Let T; be the duration of the ith cycle
*Use the sum Y.r: k(a - k) = b a(a + 1)(a - 1).
Markov Chains: Introduction
III
164
and let R; be the total opportunity cost of holding cash on hand during that
time. Over n cycles the average cost per unit time is '
Average cost =
nK+R, +
T, +
+R
+T
Next, divide the numerator and denominator by n, let n -4 co, and invoke
the law of large numbers to obtain
Long run average cost =
K + E[R;]
E[T;]
Let r denote the opportunity cost per unit time of cash on hand. Then
E[R;] = rW while E[T;] = v,. Since these quantities were determined in
(6.21) and (6.23), we have
K + (1/3)rs(SP2 - s2)
s(`.f - s)
Long run aversga cost =
(6.24)
In order to use calculus to determine the cost-minimizing values for i and
s, it simplifies matters if we introduce the new variable x = s/`f. Then
(6.24) becomes
Longrun aversga cost =
K + (1/3)rg'x(1 - x2)
S2x(I - x)
whence
d(average cost)
dx
K(1 - 2x)
W2X2(l - X)2
0
`,3x(1 - x) +
d(average cost)
dSP
I
+ 3r`.f,
0=
r(1 + x)
2K
3
which yield
x0,, =
3
and
_Y.M
= 3s0P, = 3
,
4
Implementing the cash management strategy with the values sP, and J,,,
results in the optimal balance between transaction costs and the opportunity cost of holding cash on hand.
6.
Functionals of Random Walks and Success Runs
6.3.
165
The Success Runs Markov Chain
Consider the success runs Markov chain on N + 1 states whose transition
matrix is
P=
0
1
2
3
N
0
1
0
0
0
0
1
A
r1
q1
0
P2
0
r,
q,
0
0
0
0
qN-1
0
0
0
1
2
N-1 PN-1
N
0
...
0
Note that states 0 and N are absorbing; once the process reaches one of
these two states it remains there.
Let T be the hitting time to states 0 or N,
T=min{n? 0; X,, = 0 or X,, = N).
Problems
6.1. The probability of absorption at 0 starting from state k
(6.25)
u, = Pr{XT = OIXO = k)
satisfies the equation
uA=pA+rkuk+gAuA+,,
fork= 1,...,N- I
and
uo= 1, uN=0.
The solution is
uk = 1 - (Pk qA
I
... I
+ qA
qN-1
pN-1 + qN-1
fork= 1,...,N- 1.
(6.26)
6.2. The mean hitting time
Vk=E[T Xo
= k]
(6.27)
166
III
Markov Chains: Introduction
satisfies the equation
Vk = 1 + rkVk + gkVk+I
fork= 1,...,N- 1 andvo= vN= 0.
The solution is
Vk =
+
1
Pk + qk
+ ... +
ITk.k+I
ITk.N-I
(6.28)
PN- I + qN- I
Pk+ I + qk+ l
where
1(
qk
... (
qk+l
(Pk + gk/II\Pk+I + gx+1
g%-1
Pj-1 + q%-1
(6.29)
for k <j.
6.3. Fix a state j (0 <j < N) and let W % be the mean total visits to state
j starting from state i [see equation (6.15)]. Then
1
for j = i,
P,+q1
W; _
l ...
g,
(
(
p;+q1)
1
+gi-1
P%-1
1
for i <j,
(6.30)
q%-1) Pi+q%
0
for i > j.
Exercises
6.1.
A rat is put into the linear maze as shown:
0
shock
2
1
I
I
3
I-I
4
5
food
(a) Assume that the rat is equally likely to move right or left at each
step. What is the probability that the rat finds the food before getting shocked?
(b) As a result of learning, at each step the rat moves to the right with
probability p > z and to the left with probability q = 1 - p <
Exercises
167
What is the probability that the rat finds the food before getting
shocked?
6.2. Customer accounts receivable at Smith Company are classified
each month according to
0: Current
1: 30 to 60 days past due
2: 60 to 90 days past due
3: Over 90 days past due
Consider a particular customer account and suppose that it evolves month
to month as a Markov chain {X } whose transition probability matrix is
P_
0
1
2
3
0
0.9
0.1
0
0
1
05
.
0
0 5
0
2
0.3
0
0
0.7
3
0.2
0
0
0.8
.
Suppose that a certain customer's account is now in state 1: 30 to 60 days
past due. What is the probability that this account will be paid (and thereby
enter state 0: Current) before it becomes over 90 days past due? That is, let
T = min In ? 0; X = 0 or X,, = 3}. Determine Pr{XT = OIXO = 11.
6.3. Players A and B each have $50 at the beginning of a game in which
each player bets $1 at each play, and the game continues until one player
is broke. Suppose there is a constant probability p = 0.492929... that
Player A wins on any given bet. What is the mean duration of the game?
6.4. Consider the random walk Markov chain whose transition probability matrix is given by
P_
0
1
2
3
0
1
0
0
0
1
0 3
0
0 7
0
2
0
0.3
0
0.7
3
0
0
0
1
.
.
168
Markov Chains: Introduction
III
Starting in state 1, determine the mean time until absorption. Do this first
using the basic first step approach of equations (4.6), and second using the
particular formula for v; that follows equation (6.18), which applies for a
random walk in which p 0 q.
Problems
Consider the random walk Markov chain whose transition probability matrix is given by
6.1.
0
1
2
3
1
0
0
0
0.3
0
0.7
0
2
0
0.1
0
0.9
3
0
0
0
1
0
P_
1
Starting in state 1, determine the mean time until absorption. Do this first
using the basic first step approach of equations (4.6), and second using the
particular results for a random walk given in equation (6.13).
6.2.
whose transition matrix is
Consider the Markov chain
P_
0
1
2
3
a
0
/3
0
1
a
0
0
/3
2
a
/3
0
0
3
0
0
0
1
where a > 0, /3 > 0, and a + /3 = 1. Determine the mean time to reach
state 3 starting from state 0. That is, find E[TIXo = 0], where T =
min{n ? 0; X = 3}.
6.3. Computer Challenge. You have two urns: A and B, with a balls in
A and b in B. You pick an urn at random, each urn being equally likely,
and move a ball from it to the other urn. You do this repeatedly. The game
ends when either of the urns becomes empty. The number of balls in A at
7.
Another Look at First Step Analysis
169
the nth move is a simple random walk, and the expected duration of the
game is E[T] = ab [see equation (6.7)]. Now consider three urns, A, B,
and C, with a, b, and c balls, respectively. You pick an urn at random, each
being equally likely, and move a ball from it to one of the other two urns,
each being equally likely. The game ends when one of the three urns becomes empty. What is the mean duration of the game? If you can guess
the general form of this mean time by computing it in a variety of particular cases, it is not particularly difficult to verify it by a first step analysis.
What about four urns?
7.
Another Look at First Step Analysis*
In this section we provide an alternative approach to evaluating the functionals treated in Section 4. The nth power of a transition probability matrix having both transient and absorbing states is directly evaluated. From
these nth powers it is possible to extract the mean number of visits to a
transient state j prior to absorption, the mean time until absorption, and the
probability of absorption in any particular absorbing state k. These functionals all depend on the initial state X. = i, and as a by-product of the derivation, we show that, as functions of this initial state i, these functionals
satisfy their appropriate first step analysis equations.
Consider a Markov chain whose states are labeled 0, 1, . . . , N. States
0, 1, . . . , r - 1 are transient in that P;;' - 0 as n - oo for 0 -< i, j < r,
while states r, . . . , N are absorbing, or trap, and here I , = 1 for r : i : N.
The transition matrix has the form
R
P= Q
0
I
(7.1)
where 0 is an (N - r + 1) X r matrix all of whose components are zero,
I is an (N - r + 1) X (N - r + 1) identity matrix, and Q;; = P; for
0:i,j<r.
To illustrate the calculations, begin with the four-state transition matrix
* This section contains material at a more difficult level. It is not prerequisite to what
follows.
170
Markov Chains: Introduction
III
2
3
Qoi
R0,
R03
Qio
Qu
R,,
R13
2
0
0
1
0
3
0
0
0
1
0
P
0
1
Qoo
(7.2)
Straightforward matrix multiplication shows the square of P to be
Q2 R + QR
P2 =
(7 3)
.
0
I
Continuing on to the third power, we have
P3 =
Q R
0
x
I
Q2
R+QR
0
I
Q3 R + QR + Q2R
0
I
and for higher values of n,
P" =
Qn (I+Q+
+Q^-')R
(7.4)
I
0
The consideration of four states was for typographical convenience only.
It is straightforward to verify that the nth power of P is given by (7.4)
for the general (N + 1)-state transition matrix of (7.1) in which states 0,
i, j < r) while states
1, ... , r - I are transient (P;;'-3 0 as n -4 cc for 0
r, ... , N are absorbing (P; = 1 for r : i :5 N).
We turn to the interpretation of (7.4). Let M' be the mean number of
visits to state j up to stage n for a Markov chain starting in state i.
Formally,
W,'= E[
1{X, = j}IXo
(7.5)
1=o
where
1
1(X,=j}={0
if X, = j,
if X,0 j.
(7.6)
Now, E[1{X, = j}IXo = i] = Pr{X, = jlXO = i} = P;;', and since the expected value of a sum is the sum of the expected values, we obtain from
(7.5) that
7.
Another Look at First Step Analysis
171
n
W;;)=7, E[1(X,=j}IX0=i]
i=o
(7.7)
n
_
P(!).
1=0
Equation (7.7) holds for all states i, j, but it has the most meaning when i and
j are transient. Because (7.4) asserts that P;;' = Q;;' when 0 s i, j < r, then
a)
*I
(o)
YY)-ij- Qij+Qij+
Oc hj<r,
(.>
+Qij+
where
if i = j,
ifi*j.
In matrix notation, Q(0) = I, and because Q1") = Q", the nth power of Q,
then
W(")=I+Q+Q2+...+Q"
=I+Q(I+Q+ +Q"-')
=I+
(7.8)
QW("-u.
Upon writing out the matrix equation (7.8) in terms of the matrix entries,
we recognize the results of a first step analysis. We have
r-1
will,)
= Vij + I
QikWkj-1)
k=0
r-1
= Sij + I PkW(k'-').
k=0
In words, the equation asserts that the mean number of visits to state j in
the first n stages starting from the initial stage i includes the initial visit if
i = j (S,j) plus the future visits during the n - 1 remaining stages weighted
by the appropriate transition probabilities.
We pass to the limit in (7.8) and obtain for
Wj = lim W°;' = E[Total visits to jl Xo = i],
the matrix equations
W=I+Q+Q2+
0 < i, j < r,
172
111_ Markov Chains: Introduction
W=I+QW.
(7.9)
In terms of its entries, (7.9) is
r- I
W j = S ; j + L P, W,,
for i, j = 0,
... , r - 1.
(7.10)
1=0
Equation (7.10) is the same as equation (4.11), which was derived by a
first step analysis.
Rewriting equation (7.9) in the form
W-QW=(I-Q)W=I,
(7.11)
we see that W = (I - Q)-', the inverse matrix to I - Q. The matrix W is
often called the fundamental matrix associated with Q.
Let T be the time of absorption. Formally, since states r, r + 1,
are the absorbing ones, the definition is
X,
. . . ,
N
N}.
Then the (i, j)th element Wj of the fundamental matrix W evaluates
T-1
Wj=E
ln=O
1
for0:i,j<(7.12)
1{X=j}IX0=i]
Let v, = E[TI Xo = i] be the mean time to absorption starting from state i.
The time to absorption is composed of sojourns in the transient states.
Formally,
T-1 r-1
r- I T- I
17, 1{X =j} =1n=07,j=0'(X" =j)
j=0 n=0
T-1
T.
n=0
It follows from (7.12), then, that
I
I
j=0
j=0
E[T-7,I
n=0
=E[TIX0=i]=v;
(7.13)
for0:5i<r.
7.
Another Look at First Step Analysis
173
Summing equation (7.10) over transient states j as follows,
r-1
r-1
r-1 r-1
j=0
j=0 k=0
PJW j
I Wj =
j=0
for i = 0, 1, .
. .
,r-
1,
and using the equivalence v, = Ij-o W, leads to
r-1
v;= 1 +7, Pjvk
for i = 0, 1,...,r- 1.
(7.14)
k=0
This equation is identical with that derived by first step analysis in (4.10).
We turn to the hitting probabilities. Recall that states k = r, . . . , N are absorbing. Since such a state cannot be left once entered, the probability of
absorption in a particular absorbing state k up to time n, starting from initial state i, is simply
=Pr{T<_n and XT=kIXo=i}
(7.15)
fori =0, ..,r- 1;k= r,...,N,
where T = min{n > 0: r:5 X,, <_ N} is the time of absorption. Let
U;' = Pr f T
n and XT =kIXo = i }
for 0:i<randr-< k:5 N.
(7.16)
Referring to (7.4) and (7.15), we give the matrix UI"' by
Ul"'=(I+Q+...+Q°-')R
(7.17)
=
W('-11R
[by (7.8)].
If we pass to the limit in n, we obtain the hitting probabilities
Uk=lirU;''=Pr{XT=kIXo=i}
for0:! i<randr-< k:5 N.
Equation (7.17) then leads to an expression of the hitting probability matrix U in terms of the fundamental matrix W as simply U = WR, or
r-1
Uk=X W, R,,
j=o
for0:5 i<randr<-k:5 N.
(7.18)
174
Markov Chains: Introduction
III
Equation (7.18) may be used in conjunction with (7.10) to verify the first
step analysis equation for Uk. We multiply (7.10) by Rjk and sum, obtaining thereby
r-1r-I
r-
r-I
ijRjk + 7,7. `i1W1jRjk,
ijRjk =
j=0
j=0 1=0
j=0
which with (7.18) gives
r-1
Uk=Rik+2., PlUk
1=0
r-i
for 0:i<rand r<_k<-N.
=Pik+7, P,U,k
1=0
This equation was derived earlier by first step analysis in (4.8).
Exercises
7.1. Consider the Markov chain whose transition probability matrix is
given by
P-
0
1
2
3
1
0
0
0
1
0.1
0.2
0.5
0.2
2
0.1
0.2
0.6
0.1
3
0
0
0
1
The transition probability matrix corresponding to the nonabsorbing states
is
0 0.2
2
0.5
0.2
0.6
1
Q
1
Calculate the matrix inverse to I - Q, and from this determine
(a) the probability of absorption into state 0 starting from state 1;
(b) the mean time spent in each of states 1 and 2 prior to absorption.
175
Problems
Consider the random walk Markov chain whose transition probability matrix is given by
7.2.
0
1
2
3
1
0
0
0
1
0 .3
0
0.7
0
2
0
0.3
0
0.7
3
0
0
0
1
0
P_
The transition probability matrix corresponding to the nonabsorbing states
is
1
0
0.3
011
l
Q
2
0.7
0
Calculate the matrix inverse to I - Q, and from this determine
(a) the probability of absorption into state 0 starting from state 1;
(b) the mean time spent in each of states 1 and 2 prior to absorption.
Problems
7.1. A zero-seeking device operates as follows: If it is in state m at time
n, then at time n + 1 its position is uniformly distributed over the states
0, 1, . . . , m - 1. State 0 is absorbing. Find the inverse of the I - Q matrix for the transient states 1, 2,
.
.
.
, m.
7.2. A zero-seeking device operates as follows: If it is in state j at time
n, then at time n + I its position is 0 with probability 1/j, and its position
is k (where k is one of the states 1, 2, . . . , j - 1) with probability
State 0 is absorbing. Find the inverse of the I - Q matrix.
2k/j2.
7.3. Let X, be an absorbing Markov chain whose transition probability
matrix takes the form given in equation (7.1). Let W be the fundamental
matrix, the matrix inverse of I - Q. Let
T=min{n?0;rSn -5 N}
176
III
Markov Chains: Introduction
be the random time of absorption (recall that states r, r + 1,
the absorbing states). Establish the joint distribution
Pr{XT_, = j, XT = kIXO = i} = W;Pk
... , N are
for 0 <- i, j < r; r:5 k : N,
whence
N
Pr{XT_, = j1X0 = i} = 7, W,.;Pk
for 0:i,j<r.
k=r
7.4. The possible states for a Markov chain a r e the integers 0, 1, ... ,
N, and if the chain is in state j, at the next step it is equally likely to be in
any of the states 0, 1, . . . , j - 1. Formally,
1,
P;
0
1/i,
ifi=j=0,
if 0<i<-j<_N,
if 0<-j<i-- N.
(a) Determine the fundamental matrix for the transient states
1,2,...,N.
(b) Determine the probability distribution for the last positive integer
that the chain visits.
7.5.
Computer Challenge.
Consider the partial sums:
So=k and
where
12, ... are independent and identically distributed as
Pr{6=0}=1-2ir
and
2
±j} = 1r(4j 2- 1),j= 1,2,....
Can you find an explicit formula for the mean time Vk for the partial sums
starting from So = k to exit the interval [0, N] = {0, 1, . . . , N}? In another context, the answer was found by computing it in a variety of special cases.
Branching Processes
8.
177
(Note: A simple random walk in the integer plane moves according to
(i, j), then the next position is equally likely to be
the rule, If (X,,,
any of the four points (i + 1, j), (i - 1, j), (i, j + 1), or (i, j - 1). Let us
suppose that the process starts at the point (X0, Yo) = (k, k) on the diagonal, and we observe the process only when it visits the diagonal. Formally,
we define
T,=min{n> 0; X,,
and
T = min{n > T,,,-.;
X,, =
It is not hard to show that
So = k, S,,, = XT., = YT,., m
> 0,
is a version of the above partial sum process.
8.
Branching Processes*
Suppose an organism at the end of its lifetime produces a random number
of offspring with probability distribution
Pr{6 = k} = pk
for k = 0, 1, 2,
... ,
(8.1)
where as usual, pk ? 0 and Irk- pk = 1. We assume that all offspring act
independently of each other and at the ends of their lifetimes (for simplicity, the lifespans of all organisms are assumed to be the same) individually have progeny in accordance with the probability distribution
(8.1), thus propagating their species. The process
where X,, is the
population size at the nth generation, is a Markov chain of special structure called a branching process.
The Markov property may be reasoned simply as follows. In the
nth generation the X,, individuals independently give rise to numbers of
* Branching processes are Markov chains of a special type. Sections 8 and 9 are not prerequisites to the later chapters.
178
III
offspring e;'', 421"',
Markov Chains: Introduction
... , e' and hence the cumulative number produced for
the (n + 1)st generation is
,i,
8.1.
(8.2)
Examples of Branching Processes
Thcre are numerous example of Markov branching processes that arise
naturally in various scientific disciplines. We list some of the more prominent cases.
Electron Multipliers
An electron multiplier is a device that amplifies a weak current of electrons. A series of plates are set up in the path of electrons emitted by a
source. Each electron, as it strikes the first plate, generates a random number of new electrons, which in turn strike the next plate and produce more
electrons, and so forth. Let X. be the number of electrons initially emitted,
X, the number of electrons produced on the first plate by the impact due
to the Xo initial electrons; in general, let X,, be the number of electrons
emitted from the nth plate due to electrons emanating from the (n - I)st
plate. The sequence of random variables X0, X, X,..... X,,, ... constitutes
a branching process.
Neutron Chain Reaction
A nucleus is split by a chance collision with a neutron. The resulting fission yields a random number of new neutrons. Each of these secondary
neutrons may hit some other nucleus, producing a random number of additional neutrons, and so forth. In this case the initial number of neutrons
is X. = 1. The first generation of neutrons comprises all those produced
from the fission caused by the initial neutron. The size of the first generation is a random variable X,. In general, the population X at the nth generation is produced by the chance hits of the
individual neutrons of
the (n - 1)st generation.
g.
Branching Processes
179
Survival of Family Names
The family name is inherited by sons only. Suppose that each individual
has probability pk of having k male offspring. Then from one individual
there result the 1st, 2nd, ... , nth, ... generations of descendants. We may
investigate the distribution of such random variables as the number of descendants in the nth generation, or the probability that the family name
will eventually become extinct. Such questions will be dealt with beginning in Section 8.3.
Survival of Mutant Genes
Each individual gene has a chance to give birth to k offspring, k =
. . , which are genes of the same kind. Any individual, however, has
a chance to transform into a different type of mutant gene. This gene may
become the first in a sequence of generations of a particular mutant gene.
We may inquire about the chances of survival of the mutant gene within
the population of the original genes. In this example, the number of offspring is often assumed to follow a Poisson distribution.
The rationale behind this choice of distribution is as follows. In many
populations a large number of zygotes (fertilized eggs) are produced, only
a small number of which grow to maturity. The events of fertilization and
maturation of different zygotes obey the law of independent binomial trials. The number of trials (i.e., number zygotes) is large. The law of rare
events then implies that the number of progeny that mature will approximately follow the Poisson distribution. The Poisson assumption seems
quite appropriate in the model of population growth of a rare mutant gene.
If the mutant gene carries a biological advantage (or disadvantage), then
1, 2, .
the probability distribution is taken to be the Poisson distribution with
mean A > 1 or (<1).
All of the preceding examples possess the following structure. Let Xo
denote the size of the initial population. Each individual gives birth to k
new individuals with probability pk independently of the others. The total-
ity of all the direct descendants of the initial population constitutes the
first generation, whose size we denote by X,. Each individual of the first
generation independently bears a progeny set whose size is governed by
the probability distribution (8.1). The descendants produced constitute the
second generation, of size X2. In general, the nth generation is composed
180
III
Markov Chains: Introduction
of descendants of the (n - 1)st generation, each of whose members independently produces k progeny with probability px, k = 0, 1, 2. .... The
population size of the nth generation is denoted by X,,. The X,, forms a sequence of integer-valued random variables that generate a Markov chain
in the manner described by (8.2).
8.2.
The Mean and Variance of a Branching Process
Equation (8.2) characterizes the evolution of the branching process as successive random sums of random variables. Random sums were studied in
II, Section 3, and we can use the moment formulas developed there to
compute the mean and variance of the population size X. First some notation. Let µ = E[fl and U2 = Var[f] be the mean and variance, respectively, of the offspring distribution (8.1). Let M(n) and V(n) be the mean
and variance of X,, under the initial condition Xo = 1. Then direct application of II, (3.9) with respect to the random sum (8.2) gives the recursions
M(n + 1) = µM(n)
(8.3)
V(n + 1) = v2M(n) + µ2V(n).
(8.4)
and
The initial condition X,, = 1 starts the recursions (8.3) and (8.4) at M(0) = 1
and V(0) = 0. Then, from (8.3) we obtain M(1) = µl = µ, M(2) _
µM(1) = tL2, and, in general,
M(n) = µ"
for n = 0, 1,
....
(8.5)
Thus the mean population size increases geometrically when µ > 1, decreases geometrically when µ < 1, and remains constant when µ = 1.
Next, substitution of M(n) = µ" into (8.4) gives V(n + 1) = 0-2g, +
A2V(n), which with V(0) = 0 yields
V(1) = 0.2,
V(2) = 0-2µ + µ2V(1) = 0'21, + a''-µ22,
V(3) = 0.2µe + µ'-V(2)
= a'2µ` + v22µ3 + v2µ4,
S.
Branching Processes
181
and, in general,
v'-[µ"-I + g, + ... + µz, 2
V(n) =
v-µ-1[l +
=
=v'µ' X
µ + ... +
n
if µ = 1,
1-µ
if
(8.6)
1.
Thus the variance of the population size increases geometrically if
µ > 1, increases linearly if µ = 1, and decreases geometrically if µ < 1.
8.3.
Extinction Probabilities
Population extinction occurs when and if the population size is reduced to
zero. The random time of extinction N is thus the first time n for which
X = 0, and then, obviously, Xk = 0 for all k > N. In Markov chain terminology 0 is an absorbing state, and we may calculate the probability of
extinction by invoking a first step analysis. Let
u,, = Pr(N <_ n) = Pr{X = 01
(8.7)
be the probability of extinction at or prior to the nth generation, beginning
with a single parent Xo = 1. Suppose that the single parent represented by
Xo = 1 gives rise to e;0) = k offspring. In turn, each of these offspring will
generate a population of its own descendants, and if the original popula-
tion is to die out in n generations, then each of these k lines of descent
must die out in n - 1 generations. The analysis is depicted in Figure 8.1.
Now, the k subpopulations generated by the distinct offspring of the
original parent are independent, and they have the same statistical properties as the original population. Therefore, the probability that any particuby definition, and the
lar one of them dies out in n - 1 generations is
probability that all k subpopulations die out in n - 1 generations is the kth
power
because they are independent. Upon weighting this factor by
the probability of k offspring and summing according to the law of total
probability, we obtain
U,, = >,
k=0
n = 1, 2, ....
(8.8)
III _Markov Chains: Introduction
182
Generation
X,= 1
0
X1 - (°) = k
1
f
I
n -
j
I
I
I
1
I
1
I
I
I
\
j
k subsequent subpopulations
Figure 8.1 The diagram illustrates that if the original population is to die out
by generation n, then the subpopulations generated by distinct
initial offspring must all die out in n - 1 generations.
Of course uo = 0, and u, = po, the probability that the original parent had
no offspring.
Example Suppose a parent has no offspring with probability and two
offspring with probability 4. Then the recursion (8.8) specializes4 to
1
U11 =
4
3
,
+ 4(u, 1)- =
1+
4
Beginning with uo = 0 we successively compute
u, = 0.2500,
u6 = 0.3313,
u2 = 0.2969,
u, = 0.3323,
u3 = 0.3161,
u8 = 0.3328,
U4= 0.3249,
u9 = 0.3331,
u, = 0.3292,
u,o = 0.3332.
Problems
183
We see that the chances are very nearly that such a population will die
out by the 10th generation.
Exercises
8.1. A population begins with a single individual. In each generation,
each individual in the population dies with probability ; or doubles with
probability ;. Let X,, denote the number of individuals in the population in
the nth generation. Find the mean and variance of X,,.
8.2. The number of offspring of an individual in a popualtion is 0, 1, or
2 with respective probabilities a > 0, b > 0, and c > 0, where a + b +
c = 1. Express the mean and variance of the offspring distribution in
terms of b and c.
8.3. Suppose a parent has no offspring with probability ; and has two
offspring with probability . If a population of such individuals begins
with a single parent and evolves as a branching process, determine u,,, the
probability that the population is extinct by the nth generation, for n = 1,
2,3,4,5.
8.4. At each stage of an electron multiplier, each electron, upon striking
the plate, generates a Poisson distributed number of electrons for the next
stage. Suppose the mean of the Poisson distribution is A. Determine the
mean and variance for the number of electrons in the nth stage.
Problems
8.1. Each adult individual in a population produces a fixed number M of
offspring and then dies. A fixed number L of these remain at the location
of the parent. These local offspring will either all grow to adulthood,
which occurs with a fixed probability 6, or all will die, which has probability 1 - P. Local mortality is catastrophic in that it affects the entire
local population. The remaining N = M - L offspring disperse. Their successful growth to adulthood will occur statistically independently of one
another, but at a lower probability a = p/3, where p may be thought of as
the probability of successfully surviving the dispersal process. Define
the random variable to be the number of offspring of a single parent that
184
III
Markov Chains: Introduction
survive to reach adulthood in the next generation. According to our assumptions, we may write as
= v, + v, +
+ v, + (M - N)O,
where 0, v v...... v, are independent with Pr{vk = I } = a, Pr{vk = 0)
1 - a, and with Pr{O = 1) = /8 and Pr{/3 = 0) = 1 - 8. Show that the
mean number of offspring reaching adulthood is E[f] = aN + /3(M - N),
and since a < /3, the mean number of offspring is maximized by dispersing none (N = 0). Show that the probability of having no offspring surviving to adulthood is
Pr{6=0} =(1 -a)"(1 -/3),
and that this probability is made smallest by making N large.
8.2. Let Z = Y,; =o X,, be the total family size in a branching process
whose offspring distribution has a mean µ = E[6] < 1. Assuming that
Xo = 1, show that E[Z] = 1/(1 - µ).
8.3. Families in a certain society choose the number of children that
they will have according to the following rule: If the first child is a girl,
they have exactly one more child. If the first child is a boy, they continue
to have children until the first girl, and then cease childbearing.
(a) For k = 0, 1, 2,
. . . , what is the probability that a particular family will have k children in total?
(b) For k = 0, 1, 2, . . . , what is the probability that a particular family will have exactly k male children among their offspring?
be a branching process with mean family size µ. Show
is a nonnegative martingale. Interpret the maximal inequality as applied to
8.4.
Let
that Z =
9.
Branching Processes and Generating Functions*
Consider a nonnegative integer-valued random variable
bility distribution is given by
* This topic is not prerequisite to what follows.
whose proba-
9.
Branching Processes and Generating Functions
185
fork = 0, 1, ....
Pr{ = k} = pk
(9.1)
The generating function 4(s) associated with the random variable 6 (or
equivalently, with the distribution {pk }) is defined by
OC
for 0 s s :5 1.
O(s) = E[s1 } = >, pkS k
(9.2)
k=0
Much of the importance of generating functions derives from the following three results.
First, the relation between probability mass functions (9.1) and generating functions (9.2) is one-to-one. Thus knowing the generating function
is equivalent, in some sense, to knowing the distribution. The relation that
expresses the probability mass function {pk} in terms of the generating
function O(s) is
1
d'¢(s)
k!
dsk
(9.3)
Pk -
(f(s)
= po + p,s + p2S2 + ...'
Po = 0(0),
d¢(s)
=p
,
+ 2p2s + 3p3s2 +
,
ds
whence
d¢(s)
ds
Second, if,,
... ,
ating functions 4,(s), ...
+
of their sum X
L=o
are independent random variables having gener,
respectively, then the generating function
+ is simply the product
OX(s) = 4 (s)(Pz(s) ... 4",(s).
(9.4)
This simple result makes generating functions extremely helpful in dealing with problems involving sums of independent random variables. It is
to be expected, then, that generating functions might provide a major tool
in the analysis of branching processes.
III _Markov Chains: Introduction
186
Third, the moments of a nonnegative integer-valued random variable
may be found by differentiating the generating function. For example, the
first derivative is
do(s)
= PI
d s
+ 2p,s + 3p,s' + ,
whence
d4(s)
= p, + 2p2 + 3p3 +
= E[f],
(9.5)
ds
and the second derivative is
d''-(p(s) =
2P2 + 3(2)p3s + 4(3)pas2 +
ds'-
whence
d2((s)
ds2
= 2p, + 3(2)p; + 4(3)p. + ..
s=I
x
k(k - I )p, =
1)]
k=2
=
E[fl.
Thus
d'4(s)
+ E[fl
ds-
+ d4(s)
d 20(s)
ds'
ds
_,
L,
and
Var[6] = E[62] - {E[ f ] )'-
= d2 b(s)
ds-
+ d4(s)
Ll
ds
(d4(s)
=,
ds
Example If has a Poisson distribution with mean A for which
(9.6)
9.
Branching Processes and Generating Functions
187
Ake-A
pk=Pr{e=k} =
fork=0, 1,...,
k!
then
x
Ake-A
4(s) = E[sc] = k=o
k, sk k!
(As)k
k=0
k!
= e-AeAs = e-A('-S'
for Isl < 1.
Then
d4(s)
=
d4(s)
Ae-A(I-S);
= A;
ds
ds
d'4(s)
S-,
dz4'(s)
= Aze-A(I-S)
ds2
dsz
- Az.
IS=,
From (9.5) and (9.6) we verify that
E[f] = A,
Var[f]=Az+A-(A)2=A.
9.1.
Generating Functions and Extinction Probabilities
Consider a branching process whose population size at stage n is denoted
by X,,. Assume that the offspring distribution pk = Pr{ = k} has the gen-
erating function 4(s) = E[sc] = 1k spk. If u = Pr(X,, = 0} is the probability of extinction by stage n, then the recursion (8.8) in terms of generating functions becomes
pk(un-,)k
= due-1)
un =
k=0
That is, knowing the generating function 4(s), we may successively compute the extinction probabilities u,, beginning with u,, = 0 and then u,
4(uo), u, = 4(u), and so on.
188
III
Markov Chains: Introduction
Example The extinction probabilities when there are no offspring with
probability po = 4 and two offspring with probability P2 = 4 were computed in the example in Section 8.3. We now reexamine this example
using the offspring generating function e(s) = 14 + is2.
This generating
4
function is plotted as Figure 9.1. From the picture it is clear that the extinction probabilities converge upward to the smallest solution of the
equation u = ¢(u). This, in fact, occurs in the most general case. If u,. denotes this smallest solution to u = 4(u), then ux gives the probability that
the population eventually becomes extinct at some indefinite, but finite,
time. The alternative is that the population grows infinitely large, and this
occurs with probability 1 - ux.
For the example at hand, O(s) = 4 + as2, and the equation u = 4 (u) is
the simple quadratic u = a + 2U2, which gives
4±
U,
U,
1
6
1
1,3
4(u1)
0(uo)
U,
Figure 9.1
The generating function corresponding to the offspring distribution po =4' and p2 = ;. Here u,, = Pr{XA = 0} is the probability of
extinction by generation k.
9.
Branching Processes and Generating Functions
189
The smaller solution is ux = , which is to be compared with the apparent
limit of the sequence u,, computed in the example in Section 8.3.
It may happen that u,, = 1, that is, the population is sure to die out at
some time. An example is depicted in Figure 9.2: The offspring distribu-
tion is po = and p2 = ;. We solve u = (u) _ +
4±
16- 12
2
;u2 to obtain
= 1, 3.
The smaller solution is ux = 1, the probability of eventual extinction.
In general, the key is whether or not the generating function 4(s)
crosses the 45° line ¢(s) = s, and this, in turn, can be determined from the
slope
01(1) =
do(s)
ds
of the generating function at s = 1. If this slope is less than or equal to
one, then no crossing takes place, and the probability of eventual extinction is ux = 1. On the other hand, if the slope 0'(1) exceeds one, then the
0 (u)
U, = O(u,)
ui = 4(uo)
ui
u,
u
Figure 9.2 The generating function corresponding to the offspring distribulion po - and p, = 4
190
III
_
Markov Chains: Introduction
equation u = ¢(u) has a smaller solution that is less than one, and extinction is not a certain event.
But the slope 0'(1) of a generating function at s = 1 is the mean E[fl
of the corresponding distribution. We have thus arrived at the following
important conclusion: If the mean offspring size
1, then u,. = 1
and extinction is certain. If E[f] > 1, then ux < 1 and the population may
grow unboundedly with positive probability.
The borderline case E[fl = 1 merits some special attention. Here
1] = 1 for all n, so the mean population size is constant. Yet the
population is sure to die out eventually! This is a simple example in which
the mean population size alone does not adequately describe the population behavior.
9.2.
Probability Generating Functions and
Sums of Independent Random Variables
Let and i be independent nonnegative integer-valued random variables
having the probability generating functions (p.g.f.s)
O(s) = E[sc]
and
i/i(s) = E[s"]
for IsI < 1.
The probability generating function of the sum + q is simply the product O(s)4i(s) because
E[s4]E[s'']
=
(because
and rl are independent) (9.7)
41(s)
The converse is also true. Specifically, if the product of the p.g.f.s of
two independent random variables is a p.g.f. of a third random variable,
then the third random variable equals (in distribution) the sum of the other
two.
Let
....
be independent and identically distributed nonnegative
integer-valued random variables with p.g.f. 4(s) = E[sf]. Direct induction
of (9.7) implies that the sum , +
+ ,, has p.g.f.
E[sf +...+',.] = [4(s)]'".
(9.8)
9.
Branching Processes and Generating Functions
191
We extend this result to determine the p.g f. of a sum of a random number
of independent summands Accordingly, let N be a nonnegative integer-
valued random variable, independent of C,, e,, ... , with p.g.f. gN(s) _
E[sN], and consider the random sum (see II, Section 3).
X= ,+...+6N
Let hx(s) = E[sx] be the p.g.f. of X. We claim that hx(s) takes the simple
form
(9.9)
hx(s) = M[O(s)].
To establish (9.9), consider
hx(s) _ ' Pr{X = k}s"
k=0
_
(
k=0 n=0
Pr{X = kIN = n} Pr{N = n}Is"
/
k-0 (n=0
X
D
_7, 1
+6,,=k}Pr{N=n}s"
k=0n=0
[because N is independent of ,, 62, ...]
n-0 UO
//
X
_ , O(s)" Pr{N = n}
[using (9.8)]
n=0
=
[by the definition of
With the aid of (9.9), the basic branching process equation
XI,
0,, + ... +
:,
ex"
(9.10)
can be expressed equivalently and succinctly by means of generating
functions. To this end, let
E[sx,] be the p.g.f. of the population size
X,, at generation n, assuming that X. = 1. Then easily, O0(s) = E[s] = s,
and O,(s) = O(s) = E[sc]. To obtain the general expression, we apply (9.9)
to (9.10) to yield
192
111
Markov Chains: Introduction
(9.11)
This expression may be iterated in the manner
.
4 [ca(s) }
(9.12)
(n + 1) iterations
That is, we obtain the generating function for the population size X,, at
generation n, given that X0 = 1, by repeated substitution in the probability generating function of the offspring distribution:
For general initial population sizes X0 = k, the p.g.f. is
00
Pr{X =JIX0 = k}s' =
(9.13)
j=o
exactly that of a sum of k independent lines of descents. From this perspective, the branching process evolves as the sum of k independent
branching processes, one for each initial parent.
Example Let ¢(s) = q + ps, where 0 < p < 1 and p + q = 1. The associated branching process is a pure death process. In each period each
individual dies with probability q and survives with probability p. The
in this case are readily determined, e.g., 42(s) = q + p,
(q + ps) = 1 - p2 + p'-s, and generally, O,(s) = 1 - p" + p"s. If we foliterates
low (9.13), the nth generation p.g.f. starting from an initial population size
[1 - p" + p"s]k.
of k is
The probability distribution of the time T to extinction may be determined from the p.g.f. as follows:
=OIXo=k}
Pr{T=nIX(O)=k}
[4,(O)]k - [(P"_,(0)]k
_ (I - p )k - (I 9.3.
)k
Multiple Branching Processes
Population growth processes often involve several life history phases
(e.g., juvenile, reproductive adult, senescence) with different viability and
9.
Branching Processes and Generating Functions
193
behavioral patterns. We consider a number of examples of branching
processes that take account of this characteristic.
For the first example, suppose that a mature individual produces offspring according to the p.g.f. ¢(s). Consider a population of immature individuals each of which grows to maturity with probability p and then re-
produces independently of the status of the remaining members of the
population. With probability 1 - p an immature individual will not attain
maturity and thus will leave no descendants. With probability p an individual will reach maturity and reproduce a number of offspring determined according to the p.g.f. -O(s). Therefore the progeny size distribution
(or equivalently the p.g.f.) of a typical immature individual taking account
of both contingencies is
(1 - p) + PO(s).
(9.14)
If a census is taken of individuals at the adult (mature) stage, the aggregate number of mature individuals contributed by a mature individual
will now have p.g.f.
4(1 - p + ps).
(9.15)
(The student should verify this finding.)
It is worth emphasis that the p.g.f.s (9.14) and (9.15) have the same
mean p¢'(1) but generally not the same variance, the first being
p[f(l) + 4'(1) - (4'(l))']
as compared to
pz4"(1) + p4'(1) - p2(4'(1))2.
Example A second example leading to (9.15), as opposed to (9.14),
concerns the different forms of mortality that affect a population. We appraise the strength (stability) of a population as the probability of indefinite survivorship = 1 - probability of eventual extinction.
In the absence of mortality the offspring number X of a single individual has the p.g.f. 4(s). Assume, consistent with the postulates of a branching process, that all offspring in the population behave independently governed by the same probability laws. Assume also an adult population of
size X = k. We consider three types of mortality:
(a) Mortality of Individuals Let p be the probability of an offspring surviving to reproduce, independently of what happens to others. Thus the
194
III
Markov Chains: Introduction
contribution of each litter (family) to the adult population of the next generation has a binomial distribution with parameters (N, p), where N is the
progeny size of the parent with p.g.f. 4(s). The p.g.f. of the adult numbers
contributed by a single grandparent is therefore 4(q + ps), q = 1 - p, and
for the population as a whole is
'
tp.(s) = [4(q + ps)]'.
(9.16)
This type of mortality might reflect predation on adults.
(b) Mortality of Litters Independently of what happens to other litters,
each litter survives with probability p and is wiped out with probability q
= 1 - p. That is, given an actual litter size , the effective litter size is
with probability p, and 0 with probability q. The p.g.f. of adults in the following generation is accordingly
4z(s) _ [q + p4(s)1'.
(9.17)
This type of mortality might reflect predation on juveniles or on nests and
eggs in the case of birds.
(c) Mortality of Generations An entire generation survives with probability p and is wiped out with probability q. This type of mortality might
represent environmental catastrophes (e.g., forest fire, flood). The p.g.f. of
population size in the next generation in this case is
F,(s) = q + p[4(s)]'.
(9.18)
All the p.g.f.s (9.16) through (9.18) have the same mean but usually different variances.
It is interesting to assess the relative stability of these three models.
That is, we need to compare the smallest positive roots of 4,i(s) = s,
i = 1, 2, 3, which we will denote by s*, i = 1, 2, 3, respectively.'
We will show by convexity analysis that
4"(s) :5 u2(s) !5; y1,(s)
A function f(x) is convex in x if for every x, and x, and 0 < A < 1, then
f[Ax, + (1 - A)x,] s Af(x,) + (1 - A)f(x,). In particular, the function
QC(s) = G =o pks, for 0 < s < 1 is convex in s, since for each positive integer k, [(As,) + (1 - A)s,]' As; + (1 - A)s; for 0 < A, s,, s, < 1. Now,
qi,(s) = [4(q + ps)]' < [gca(1) + p4(s)]k = [q + pcb(s)]' = i/i,(s), and then
s* < s*. Thus the first model is more stable than the second.
195
Problems
Observe further that due to the convexity of f(x) = x', x > 0, i/r,(s) =
[p4(s) + q}' < p[4(s)]k + q X P = /i,(s), and thus s*2 < s*,3 implying that
the second model is more stable than the third model. In conjunction we
get the ordering s* < s* < s*.
Exercises
9.1. Suppose that the offspring distribution is Poisson with mean
A = 1.1. Compute the extinction probabilities u = Pr (X,, = 01X0 = I) for
n = 0, 1, . . . , 5. What is ux, the probability of ultimate extinction?
9.2 Determine the probability generating function for the offspring distribution in which an individual either dies, with probability po, or is replaced by two progeny, with probability P21 where po + P2 = 1.
9.3. Determine the probability generating function corresponding to the
offspring distribution in which each individual produces 0 or N direct descendants, with probabilities p and q, respectively.
9.4. Let ¢(s) be the generating function of an offspring random variable
6. Let Z be a random variable whose distribution is that of , but conditional on > 0. That is,
Pr{Z=k}
fork= 1,2.....
Express the generating function for Z in terms of 4.
Problems
9.1.
One-fourth of the married couples in a far-off society have no children at all. The other three-fourths of families have exactly three children,
each child equally likely to be a boy or a girl. What is the probability that
the male line of descent of a particular husband will eventually die out?
9.2.
One-fourth of the married couples in a far-off society have exactly
three children. The other three-fourths of families continue to have children until the first boy and then cease childbearing. Assume that each
child is equally likely to be a boy or girl. What is the probability that the
male line of descent of a particular husband will eventually die out?
196
III
Markov Chains: Introduction
9.3. Consider a large region consisting of many subareas. Each subarea
contains a branching process that is characterized by a Poisson distribution with parameter A. Assume, furthermore, that the value of A varies
with the subarea, and its distribution over the whole region is that of a
gamma distribution. Formally, suppose that the offspring distribution is
given by
e-ASR
7r(kj1k)
=
k!
for k = 0, 1,\\.,
where A itself is a random variable having the density function
6«A«ie0A
for A > 0,
f(A)
F(a)
where 6 and a are positive constants. Determine the marginal offspring
distribution pA = f ir(kl A)f(A) dA.
Hint:
Refer to the last example of II, Section 4.
9.4. Let 4(s) = 1 - p(1 - s),', where p and /3 are constants with 0 < p,
/3 < 1. Prove that O(s) is a probability generating function and that its
iterates are
0,,(s) = 1 -
pl+a+...+1.,-I(1 - s)a'
for n = 1,2,....
9.5. At time 0, a blood culture starts with one red cell. At the end of one
minute, the red cell dies and is replaced by one of the following combinations with the probabilities as indicated:
2 red cells
1 red, 1 white
2 white
Each red cell lives for one minute and gives birth to offspring in the same
way as the parent cell. Each white cell lives for one minute and dies without reproducing. Assume that individual cells behave independently.
(a) At time n + 1 minutes after the culture begins, what is the probability that no white cells have yet appeared?
(b) What is the probability that the entire culture eventually dies out
entirely?
Problems
197
9.6. Let ¢(s) = as- + bs + c, where a, b, c are positive and 4(l) = 1.
Assume that the probability of extinction is u., where 0 < u,, < 1. Prove
that u = cla.
9.7. Families in a certain society choose the number of children that
they will have according to the following rule: If the first child is a girl,
they have exactly one more child. If the first child is a boy, they continue
to have children until the first girl and then cease childbearing. Let 6 be
the number of male children in a particular family. What is the generating
function of V. Determine the mean of 6 directly and by differentiating the
generating function.
9.8. Consider a branching process whose offspring follow the geometric distribution pk = (1 - c)ck for k = 0, 1, ... , where 0 < c < 1. Determine the probability of eventual extinction.
9.9. One-fourth of the married couples in a distant society have no children at all. The other three-fourths of families continue to have children
until the first girl and then cease childbearing. Assume that each child is
equally likely to be a boy or girl.
(a) For k = 0, 1, 2,
what is the probability that a particular husband will have k male offspring?
(b) What is the probability that the husband's male line of descent will
cease to exist by the 5th generation?
. . . ,
9.10. Suppose that in a branching process the number of offspring of an
initial particle has a distribution whose generating function is f(s). Each
member of the first generation has a number of offspring whose distribu-
tion has generating function g(s). The next generation has generating
function f, the next has g, and the distributions continue to alternate in this
way from generation to generation.
(a) Determine the extinction probability of the process in terms of f(s)
and g(s).
(b) Determine the mean population size at generation n.
(c) Would any of these quantities change if the process started with the
g(s) process and then continued to alternate?
Chapter IV
The Long Run Behavior of
Markov Chains
1.
Regular Transition Probability Matrices
Suppose that a transition probability matrix P =j on a finite number of
states labeled 0, 1, . . . , N has the property that when raised to some
power k, the matrix Pk has all of its elements strictly positive. Such a tran-
sition probability matrix, or the corresponding Markov chain, is called
regular. The most important fact concerning a regular Markov chain is the
existence of a limiting probability distribution it = (iro, 1r,, . . . , TrN),
where i r, > 0 for j = 0, 1, ... , N and I; 7rj = 1, and this distribution is
independent of the initial state. Formally, for a regular transition probability matrix P = P, we have the convergence
limP;;'=
Tr;>0
/1-)r.
forj=0, 1,...,N,
or, in terms of the Markov chain
lim Pr{X = jjX0 = i } = 7 r ; > 0
I)--
for j = 0, 1,
... , N.
This convergence means that, in the long run (n -4 co), the probability
of finding the Markov chain in state j is approximately Tr; no matter in
which state the chain began at time 0.
IV The Long Run Behavior of Markov Chains
200
Example The Markov chain whose transition probability matrix is
0
1
_0I 1-a
P
a
1-b.
b
1
is regular when 0 < a, b < 1, and in this case the limiting distribution is
1r = (b/(a + b), a/(a + b)). To give a numerical example, we will suppose
that
0.67
0.25
0.33
0.75
P
The first several powers of P are given here:
_ 0.6114
0.3886
0.5650
P3 =
0.4932
0.5673
0.506811
0.4572
0.4883
P5 =
0.5117
0.5220
0.5350
0.4780
0.4560 '
p6 = 0.5307
0.4693
0.5271
0.5294
0.4729
0.4706
P'
II
0.4350
P4= 0.5428
0.5253
0.4747 '
P
I
0.4327 '
By n = 7, the entries agree row-to-row to two decimal places. The limiting probabilities are b/(a + b) = 0.5282 and a/(a + b) = 0.4718.
Example Sociologists often assume that the social classes of successive generations in a family can be regarded as a Markov chain. Thus, the
occupation of a son is assumed to depend only on his father's occupation
and not on his grandfather's. Suppose that such a model is appropriate and
that the transition probability matrix is given by
Son's Class
Lower Middle Upper
Lower
Father ' s Middle
Class
Upper
0 . 40
0 . 50
0 . 10
0 . 05
0 . 70
0 . 25
0.05
0.50
0.45
1.
Regular Transition Probability Matrices
201
For such a population, what fraction of people are middle class in the long
run?
For the time being, we will answer the question by computing sufficiently high powers of P' A better method for determining the limiting
distribution will be presented later in this section.
We compute
0.1900
0.6000
0.2100
0.0675
0.6400
0.2925
0.0675
0.6000
0.3325
0.0908
0.6240
0.2852
0.0758
0.6256
0.2986
0.0758
0.6240
0.3002
0.0772
0.6250
0.2978
P8= P4XP4= 0.0769 0.6250
0.2981
P2=PxP=
P2=PxP=
0.0769
0.6250
0.2981
Note that we have not computed P" for consecutive values of n but have
speeded up the calculations by evaluating the successive squares P2,
P4, P8.
In the long run, approximately 62.5 percent of the population are middle class under the assumptions of the model.
Computing the limiting distribution by raising the transition probability
matrix to a high power suffers from being inexact, since n = 00 is never
attained, and it also requires more computational effort than is necessary.
Theorem 1.1 provides an alternative computational approach by asserting
that the limiting distribution is the unique solution to a set of linear equations. For this social class example, the exact limiting distribution, computed using the method of Theorem 1.1, is aro = ,=; = 0.0769, ir, = s =
0.6250, andir2=;O',=0.2981.
PNZ
will
If a transition probability matrix P on N states is regular, then
have no zero elements. Conversely, if PN' is not strictly positive, then the
Markov chain is not regular. Furthermore, once it happens that Pk has no
zero entries, then every higher power P'`+", n = 1, 2, . . . , will have no zero
entries. Thus it suffices to check the successive squares P, P2, P4, P8. ... .
IV The Long Run Behavior of Markov Chains
202
7
Finally, to determine whether or not the square of a transition probability
matrix has only strictly positive entries, it is not necessary to perform the
actual multiplication, but only to record whether or not the product is
nonzero.
Example Consider the transition probability matrix
P=
0.9
0.1
0
0
0
0
0
0.9
0
0.1
0
0
0
0
0.9
0
0
0.1
0
0
0
0.9
0
0
0
0.1
0
0
0.9
0
0
0
0
0.1
0
0.9
0
0
0
0
0
0.1
0.9
0
0
0
0
0
0.1
We recognize this as a success runs Markov chain. We record the nonzero
entries as + and write P X P in the form
++ 0 0
+ 0+ 0
+
0
0
0
0
0
0
0 0+ 0
0
0
PxP= +
0
0
0
+
0
0
+
0
0
0
0
+
0
+
0
0
0
0 0+
+
0
0
0
0
++ 0 0 0
+ 0+ 0 0
+ 0 0+ 0
x
0
+
0
0
0
0
0
0
+
0
0
0
+
0
0
+
0
0
0
0
+
0
+
0
0
0
+
0
0
0
0 0+
0 0+
1.
Regular Transition Probability Matrices
P4 =
203
+
+
+
0
0
0
0
+
+
0
+
0
0
0
+
+
0
0
+
0
0
+
+
0
0
0
+
0
+
+
0
0
0
0
+
+
+
0
0
0
0
+
+
+
0
0
0
0
+
+
+
+
+
+
0
0
+
+
+
+
0
+
0
+
+
+
+
0
0
+
+
+
+
+
0
0
+
+
+
+
+
0
0
+
+
+
+
+
0
0
+
+
+
+
+
0
0
+
= P2
,
P8 =
We see that P8 has all strictly positive entries, and therefore P is regular.
The limiting distribution for a similar matrix is computed in Section 2.2.
Every transition probability matrix on the states 0, 1, . . . , N that satisfies the following two conditions is regular:
1. For every pair of states i, j there is a path k
P,k>...P;>0.
2. There is at least one state i for which P; > 0.
... , kr for which Pk,
204
IV The Long Run Behavior of Markov Chains
V
Theorem 1.1
Let P be a regular transition probability matrix on the
states 0, 1, . . N. Then the limiting distribution err = (iro, Tr
the unique nonnegative solution of the equations
. ,
1TkPkj,
j=0,1,...,N,
...
ire,) is
(1.2)
k=0
N
1k = 1.
(1.3)
k=0
Proof Because the Markov chain is regular, we have a limiting distribution, lim,,. P;' j= .7r j, for which Ik= 0 irk = 1. Write P" as the matrix product P"-' P in the form
P1"-"P
k
kj
if
j = 0, .
.
,
N,
k=0
and now let n - oo. Then P;;' -* irj, while P,k"-" -4 4rk, and (1.4) passes
into Trj = Jk o ITkPkj as claimed.
It remains to show that the solution is unique. Suppose that x0, x,, ... ,
x1 solves
N
xj=7, xkPkj
forj=0,...,N
(1.5)
k=0
and
N
(1.6)
k=0
We wish to show that xj = if,, the limiting probability. Begin by multiplying (1.5) on the right by P, and then sum over j to get
N N
N
N
P,=Y_ 7, xkPjI,_
j=0
j=0 k=0
But by (1.5) we have x, _ J
o
(1.7)
k=0
xjPj,, whence (1.7) becomes
N
k(
Ip(2)
X, _
for 1= 0, ...,N.
k=0
Repeating this argument n times we deduce that
N
x,xkP(")
k=0
fort=0,...,N,
1.
Regular Transition Probability Matrices
205
and then passing to the limit in n and using that P(k'; - 7r, we see that
N
XI
1=0,...,N.
ZXk7r
k=0
But by (1.6) we have lk Xk = 1, whence x, = 7r, as claimed.
Example For the social class matrix
0
1
2
0
0.40
0.50
0.10
P=1
0.05
0.70
0.25
2
0.05
0.50
0.45
the equations determining the limiting distribution ('7ro, 7x,, 7r2) are
0.40770 + 0.057x, + 0.057x2 = 7ro,
(1.8)
0.507x0 + 0.707r, + 0.507x2 = 7r,,
(1.9)
0.107x0 + 0.257r, + 0.457x2 = 7x2,
(1.10)
7r,+
(1.11)
1r0+
7x2=1.
One of the equations (1.8), (1.9), and (1.10) is redundant, because of the
linear constraint Ik Pk = 1. We arbitrarily strike out (1.10) and simplify
the remaining equations to get
-607ra+57x,+57r,=0,
(1.12)
57ro - 37r, + 57x, = 0,
(1.13)
7r, +
(1.14)
7ro +
7x2 = 1.
We eliminate 7x2 by subtracting (1.12) from (1.13) and five times (1.14) to
reduce the system to
657ro-87x,=0,
657x0
= 5.
Then Ira = bs = ,=,, 7r, _ $, and then 7r, = 1 - 7ro - 7r, _ ;0, as given earlier.
206
1.1
IV The Long Run Behavior of Markov Chains
V
Doubly Stochastic Matrices
A transition probability matrix is called doubly stochastic if the columns
sum to one as well as the rows. Formally, P = IP;I is doubly stochastic if
P;
0
and
Pk1 = 1
7, Pk =
k
for all i, j.
k
Consider a doubly stochastic transition probability matrix on the N
states 0, 1, . . . , N - 1. If the matrix is regular, then the unique limiting
distribution is the uniform distribution ?r = (1/N, ... , 1/N). Because there
is only one solution to 7r; = 1 k irkPk1 and Ik 7rk = 1 when P is regular, we
need only check that 'rr = (1/N, ... , 1/N) is a solution where P is doubly
stochastic in order to establish the claim. By using the doubly stochastic
feature I; Pi, = 1 we verify that
As an example, let Y,, be the sum of n independent rolls of a fair die and
consider the problem of determining with what probability Y is a multiple of 7 in the long run. Let X,, be the remainder when Y is divided by 7.
Then X,, is a Markov chain on the states 0, 1, ... , 6 with transition probability matrix
2
3
1
I
6
6
1
4
I
6
1
6
6
0
1
6
6
6
0
6
I
6
I
The matrix is doubly stochastic, and it is regular (P2 has only strictly positive entries), hence the limiting distribution is 'rr = (17 , . . . , 71). Furthermore, Y is a multiple of 7 if and only if X,, = 0. Thus the limiting probability that Y is a multiple of 7 is ;.
Regular Transition Probability Matrices
1.
1.2
207
Interpretation of the Limiting Distribution
Given a regular transition matrix P for a Markov process {X,,} on the
N + 1 states 0, 1, . . . , N, we solve the linear equations
N
for i = 0, 1,
TTkPkr
. . . ,
N
k=O
and
7ro+ IF, +
+ 7r,,,= 1.
The primary interpretation of the solution (7ro, ... , 7r,,,) is as the limiting
distribution
7r; = lim P;;) = lim Pr{X = jl Xo = i }.
In words, after the process has been in operation for a long duration, the
probability of finding the process in state j is 7r;, irrespective of the starting state.
There is a second interpretation of the limiting distribution 'rr =
(7ro, 7r . . . , 7rN) that plays a major role in many models. We claim that
is in
7r; also gives the long run mean fraction of time that the process
state j. Thus if each visit to state j incurs a "cost" of c;, then the long run
mean cost per unit time associated with this Markov chain is
N
Long run mean cost per unit time =
i=o
To verify this interpretation, recall that if a sequence a,,, a,, . . . of real
numbers converges to a limit a, then the averages of these numbers also
converge in the manner
1 ,,,-I
lim - 7 ak = a.
11- ' m k=O
P;' = 7r; to conclude that
We apply this result to the convergence
I ,,,-
lim - 7
Prk>
;; _
7r1.
208
IV The Long Run Behavior of Markov Chains
7
Now, (1/m) Y,k o
is exactly the mean fraction of time during steps
0, 1, ... , m - 1 that the process spends in state j. Indeed, the actual (random) fraction of time in state j is
1 in-1
-7, 1{Xk=j},
m k=0
where
11Xk= }=
1
ifXk =j,
fo
if X *f.
Therefore the mean fraction of visits is obtained by taking expected values according to
f1 in-1
1m-i
M k=0
m k=0
EL->2 1{Xk=j}IXo=ij=- E[1{Xk=j}IXo=i]
m-1
7 Pr(Xk = AXo = I}
m k=0
1 in-1
p;k1.
m k=0
Because lim,,. P;'' = n , the long run mean fraction of time that the
process spends in state j is
r 1m-1
lim El -
m k=0
1m-1
1
11Xk = j}IXo = iJ = 1im - I P;;' = 7rj,
m k=0
independent of the starting state i.
Exercises
1.1. A Markov chain X0, X X,,
. . .
has the transition probability matrix
0
1
0
0.7
0. 2
0.1
P=1
0
0. 6
0.4
2
0.5
0
0.5
Determine the limiting distribution.
2
209
Exercises
1.2.
A Markov chain X0, X X2.... has the transition probability matrix
0
2
1
0.6
0.3
0.1
P=1 0.3
0.3
0.4
2
0.1
0.5
0
0.4
Determine the limiting distribution.
1.3.
A Markov chain X0, X,, X2, ... has the transition probability matrix
0
1
2
0
0.1
0.1
0.8
P = 1
0.2
0.2
0.6
2
0.3
0.3
0.4
What fraction of time, in the long run, does the process spend in state I?
1.4. A Markov chain X0, X X,.... has the transition probability matrix
0
1
2
0
0.3
0.2
0.5
P = 1
0.5
0.1
0.4
2
0.5
0.2
0.3
Every period that the process spends in state 0 incurs a cost of $2. Every
period that the process spends in state 1 incurs a cost of $5. Every period
that the process spends in state 2 incurs a cost of $3. What is the long run
cost per period associated with this Markov chain?
1.5. Consider the Markov chain whose transition probability matrix is
given by
0
1
2
3
0.1
0.5
0
0.4
1
0
0
1
0
2
0
0
0
1
3
1
0
0
0
0
P
Determine the limiting distribution for the process.
IV The Long Run Behavior of Markov Chains
210
1.6.
Compute the limiting distribution for the transition probability
matrix
0
0
P=1
2
1
2
2
'
2'
0
1
1
1
3
3
3
6
2
3
1.7. A Markov chain on the states 0, 1, 2, 3 has the transition probability matrix
0
P=
1
2
3
0
0.1
0.2
0.3
0.4
1
0
03
0 3
.
04
2
0
0
0.6
0.4
3
1
0
0
0
.
.
Determine the corresponding limiting distribution.
1.8. Suppose that the social classes of successive generations in a family follow a Markov chain with transition probability matrix given by
Son's Class
Lower Middle Upper
Lower
Father ' s Middle
Cl ass
Upper
07
02
0
02
.
0 6
02
0.1
0.4
0.5
.
.
.
.
1
.
What fraction of families are upper class in the long run?
1.9. Determine the limiting distribution for the Markov chain whose
transition probability matrix is
2
0
6
3
1.10. A bus in a mass transit system is operating on a continuous route
with intermediate stops. The arrival of the bus at a stop is classified into
one of three states, namely
211
problems
1. Early arrival;
2. On-time arrival;
3. Late arrival.
Suppose that the successive states form a Markov chain with transition
probability matrix
1
2
3
0.5
0.4
0.1
P=2
0.2
0.5
0.3
3
0.1
0.2
0.7
1
Over a long period of time, what fraction of stops can be expected to be late?
Problems
1.1. Five balls are distributed between two urns, labeled A and B. Each
period, an urn is selected at random, and if it is not empty, a ball from that
urn is removed and placed into the other urn. In the long run, what fraction of time is urn A empty?
1.2. Five balls are distributed between two urns, labeled A and B. Each
period, one of the five balls is selected at random, and whichever urn it's
in, it is moved to the other urn. In the long run, what fraction of time is
urn A empty?
1.3.
A Markov chain has the transition probability matrix
P=
where a; ? 0, i = 1,
0
1
2
3
4
5
0
C11
a,
a3
a4
a5
a6
1
1
0
0
0
0
0
2
0
1
0
0
0
0
3
0
0
1
0
0
0
4
0
0
0
1
0
0
5
0
0
0
0
1
0
... , 6, and a, +
iting probability of being in state 0.
+ a6 = 1. Determine the lim-
212
IV The Long Run Behavior of Markov Chains
1.4. A finite state regular Markov chain has transition probability matrix
P = NP;j and limiting distribution IT = PIT;j. In the long run, what fraction
of the transitions are from a presribed state k to a prescribed state m?
1.5. The four towns A, B, C, and D are connected by railroad lines as
shown in the following diagram:
Figure 1.1
A graph whose nodes represent towns and whose arcs represent
railroad lines.
Each day, in whichever town it is in, a train chooses one of the lines out
of that town at random and traverses it to the next town, where the process
repeats the next day. In the long run, what is the probability of finding the
train in town D?
1.6.
Determine the following limits in terms of the transition probabil-
ity matrix P = II1 J0 and limiting distribution 'rr = I7r 11 of a finite state regular Markov chain
(a)
(b)
(c) lim,,.
jjX0 = i}.
Pr{X = k,
jXo = i}.
k, X, = jjX0 = i}.
1.7. Determine the limiting distribution for the Markov chain whose
transition probability matrix is
P=
0
1
2
3
0
2
0
0
_
1
1
0
0
0
2
0
2
'
3
6
3
0
0
1
0
'
'
213
Problems
1.8.
Show that the transition probability matrix
0
0
0
2
3
4
i
0
0
0
0
'
2
2
0
1
P=2
3
1
3
3
0
0
0
2
0
0
4
0
0
2
0
is regular and compute the limiting distribution.
1.9. Determine the long run, or limiting, distribution for the Markov
chain whose transition probability matrix is
0
1
2
3
0
0
1
0
0
0
0
1
4
4
4
4
0
0
2
2
'
'
1.10. Consider a Markov chain with transition probability matrix
P=
Po
A
P2
PN
PN
Po
Pi
PN-1
PN-I
PN
Po
PN-2
PA
P2
P3
where 0 < po < 1 and po + p, +
" '
Po
+ PN = 1. Determine the limiting dis-
tribution.
1.11. Suppose that a production process changes state according to a
Markov process whose transition probability matrix is given by
IV The Long-Run Behavior of Markov Chains
214
0
1
2
3
0.3
0.5
0
0.2
0.5
0.2
0.2
0.1
2
0.2
0.3
0.4
0.1
3
0.1
0.2
0.4
0.3
P
It is known that ?r, = 3;-9 = 0.3140 and ar, = ;;9 = 0.2137.
(a) Determine the limiting probabilities .7ro and it .
(b) Suppose that states 0 and 1 are "In-Control" while states 2 and 3 are
deemed "Out-of-Control." In the long run, what fraction of time is
the process Out-of-Control?
(c) In the long run, what fraction of transitions are from an In-Control
state to an Out-of-Control state?
1.12
Let P be the transition probability matrix of a finite state regular
Markov chain, and let n be the matrix whose rows are the stationary distribution ir. Define Q = P - 11.
(a) Show that P" = H + Q".
(b) When
P=
0
2'
'
2
obtain an explicit expression for Q' and then for P".
1.13. A Markov chain has the transition probability matrix
0
1
2
0.4
0.4
0.2
P=1 0.6
0.2
0.2
2
0.2
0.4
0
0.4
2.
215
Examples
After a long period of time, you observe the chain and see that it is in
state 1. What is the conditional probability that the previous state was state
2? That is, find
lim
21X = 1}.
Examples
2.
Markov chains arising in meteorology, reliability, statistical quality control, and management science are presented next, and the long run behavior of each Markov chain is developed and interpreted in terms of the phenomenon under study.
2.1.
Including History in the State Description
Often a phenomenon that is not naturally a Markov process can be modeled as a Markov process by including part of the past history in the state
description. To illustrate this technique, we suppose that the weather on
any day depends on the weather conditions for the previous two days. To
be exact, we suppose that if it was sunny today and yesterday, then it will
be sunny tomorrow with probability 0.8; if it was sunny today but cloudy
yesterday, then it will be sunny tomorrow with probability 0.6; if it was
cloudy today but sunny yesterday, then it will be sunny tomorrow with
probability 0.4; if it was cloudy for the last two days, then it will be sunny
tomorrow with probability 0.1.
Such a model can be transformed into a Markov chain, provided we say
that the state at any time is determined by the weather conditions during
both that day and the previous day. We say the process is in
State (S, S) if it was sunny both today and yesterday,
State (S, C) if it was sunny yesterday but cloudy today,
State (C, S) if it was cloudy yesterday but sunny today,
State (C, C) if it was cloudy both today and yesterday.
Then the transition probability matrix is
216
IV The Long Run Behavior of Markov Chains
Today's State
(S, S)
(S, C)
(S,S)
Yesterday's (S, C)
0.8
0.2
(C, S)
0.6
State
(Cl S)
(C, C)
0.4
0.6
0.1
0.9
0.4
(C, C)
The equations determining the limiting distribution are
0.81ro
+ 0.67x,
= 7ro,
0.27ro
+ 0.47r,
= 7r
Ira +
0.47x,
+ 0.17x, = 7r,,
0.67r,
0.9'7, = 7"
7r, +
7r, +
7r3 = 1.
Again, one of the top four equations is redundant. Striking out the first
equation and solving the remaining four equations gives 7ro = ;,, 7r,
7r,=,, and 7r,=.
We recover the fraction of days, in the long run, on which it is sunny by
summing the appropriate terms in the limiting distribution. It can be sunny
today in conjunction with either being sunny or cloudy tomorrow. There-
fore, the long run fraction of days in which it is sunny is 7ro + 7r, =
7r(S, S) + 7r(S, C)
=S}
2.2.
Formally,
Pr{X = S} = lim,,
,.
[Pr{X = S,
=C}] = ITO+7r,.
Reliability and Redundancy
An airline reservation system has two computers, only one of which is in
operation at any given time. A computer may break down on any given
day with probability p. There is a single repair facility that takes 2 days to
restore a computer to normal. The facilities are such that only one computer at a time can be dealt with. Form a Markov chain by taking as states
the pairs (x, y) where x is the number of machines in operating condition
at the end of a day and y is 1 if a day's labor has been expended on a machine not yet repaired and 0 otherwise. The transition matrix is
Examples
2.
217
To State
P=
-+ (2, 0)
(1, 0)
(1, 1)
(0, 1)
From State
1
(2, 0)
q
p
0
0
(1,0)
0
0
q
p
(1, 1)
q
p
0
0
(0, 1)
0
1
0
0
wherep + q = I.
We are interested in the long run probability that both machines are inoperative. Let (4ro, ir,, 7r,, ir3) be the limiting distribution of the Markov
chain. Then the long run probability that neither computer is operating is
1r3, and the availability, the probability that at least one computer is oper-
ating,is1-7r3=fro+7r,+,TZ.
The equations for the limiting distributions are
giro
+ qir,
PIro
+pIr,+Ira=?r,,
_ iro,
qIT,
=ITz,
PITS
= I3
and
7ro+1r,+9r,+Ir3=1.
The solution is
Ir, _
qp
q2
110
p2'
P
l
+P 2'
ir,
1+pP2
7r3 =
l
+p
2.
The availability is R, = 1 - 7T3 = 1/(1 + p2).
In order to increase the system availability, it is proposed to add a duplicate repair facility so that both computers can be repaired simultaneously. The corresponding transition matrix is now
218
IV The Long Ru n Behavior o f Markov Chains
To State
- (2, 0)
(1, 0)
(1, 1)
(0, 1)
0
0
q
p
From State
1
(2, 0)
q
p
P_(1,0)
0
0
(1, 1)
q
p
0
0
(0, 1)
0
0
1
0
.
and the limiting distribution is
ITO
=
q
P
,,
l +p+p -
1 +p+p P
1 +p+p2
,
P_
73 _
1+p+p2
*
Thus availability has increased to R, = 1 - TT, = (1 + p)/(1 + p + p2).
2.3. A Continuous Sampling Plan
Consider a production line where each item has probability p of being
defective. Assume that the condition of a particular item (defective or
nondefective) does not depend on the conditions of other items. The following sampling plan is used.
Initially every item is sampled as it is produced; this procedure continues until i consecutive nondefective items are found. Then the sampling
plan calls for sampling only one out of every r items at random until a defective one is found. When this happens the plan calls for reverting to 100
percent sampling until i consecutive nondefective items are found. The
process continues in the same way.
State Ek (k = 0, 1, . . . , i - 1) denotes that k consecutive nondefective
items have been found in the 100 percent sampling portion of the plan,
while state E; denotes that the plan is in the second stage (sampling one
out of r). Time m is considered to follow the mth item, whether sampled
or not. Then the sequence of states is a Markov chain with
219
Examples
2.
pi, = Pr{in state Ek after m + 1 itemslin state E; after m items}
p
1-p
P
fork= 0,0-j<i,
for k=j+1<i,
fork=O,j=i,
r
1- p-r
fork=j=i,
0
otherwise.
Let ark be the limiting probability that the system is in state Ek for
i. The equations determining these limiting probabilities are
k = 0, 1, .
. . ,
(0)
PITo+
par,+ +
(plr) ar; = ITo,
(1) (1 - p)'ro
(2)
= IT,,
(1 - P)irI
(1 - p)'T, + (1 - plr)ar, = 'Ti
(i)
together with
(*)
1T0+IT1+...+ar=1.
From equations (1) through (i) we deduce that ark = (1 - park-,, so that
ark = (1 - p)karo for k = 0, . . . , i - 1, while equation (i) yields ar; =
(r1p)(1 - p)ar;_, or ar, _ (rlp)(1 - p)'aro. Having determined ark in terms
of aro for k = 0, . . . , i, we place these values in (*) to obtain
{[1 + ... + (1 -
p)'-,} +
p (1 - p)'}iro = 1.
The geometric series simplifies, and after elementary algebra the solution
is
1To =
p
1 + (r - 1)(1 - p);,
IV The Long Run Behavior of Markov Chains
220
whence
Trk
P(1 - P)k
fork=0,...,i-1,
1 + (r - 1)(1 - p)'
while
ir'
r(1 - p)'
1+(r-1)(1-p)"
Let AFI (Average Fraction Inspected) denote the long run fraction of
items that are inspected. Since each item is inspected while in states
E0, ... , E;_, but only one out of r is inspected in state E;, we have
AFI = (7ro +
+ iri-,) + (1/r)ir,
_ (1 - ?r;) + (1/r)ir,
1
1 + (r - 1)(1 - p)'
Let us assume that each item found to be defective is replaced by an
item known to be good. The average outgoing quality (AOQ) is defined to
be the fraction of defectives in the output of such an inspection scheme.
The average fraction not inspected is
(r - 1)(1 - p)'
1 - AFI= 1 + (r- 1)(1 -p)"
and of these on the average p are defective. Hence
AOQ =
1)(1
P)P
(r 1 + (r - 1)(1 - p)'
This average outgoing quality is zero if p = 0 orp = 1, and rises to a maximum at some intermediate value, as shown in Figure 2.1. The maximum
AOQ is called the average outgoing quality limit (AOQL), and it has been
determined numerically and tabulated as a function of i and r.
This quality control scheme guarantees an output quality better than the
AOQL regardless of the input fraction defective, as shown in Figure 2.2.
2.
Examples
221
a
0
AOQL
0
p
1
Figure 2.1 The average outgoing quality (AOQ) as a function of the input
quality level p
INPUT
Arbitrary Quality
CONTINUOUS
SAMPLING
INSPECTION
p
OUTPUT
Guaranteed Quality
AOQL
Figure 2.2 Black-box picture of a continuous inspection scheme as a method
of guaranteeing outgoing quality
2.4.
Age Replacement Policies
A component of a computer has an active life, measured in discrete units,
that is a random variable T, where Pr[T = k] = ak for k = 1, 2,
Suppose one starts with a fresh component, and each component is replaced
by a new component upon failure. Let X,, be the age of the component in
service at time n. Then
is a success runs Markov chain. (See III, Section 5.4.)
In an attempt to replace components before they fail in service, an age
replacement policy is instituted. This policy calls for replacing the component upon its failure or upon its reaching age N, whichever occurs first.
has the transiUnder this age replacement policy, the Markov chain
tion probability matrix
....
222
IV The Long Run Behavior of Markov Chains
0
1
2
3
N- 1
Po
1 -PO
0
0
0
A
0
1 - p,
0
0
2
P2
0
0
1 - P2
N-1
1
0
0
0
0
1
P=
...
0
0
where
fork=0,1,...,N-2.
ak+l
Pk
ak+I + ak+2 +
State 0 corresponds to a new component, and therefore the limiting
probability 7ro corresponds to the long run probability of replacement during any single time unit, or the long run replacement per unit time. Some
of these replacements are planned or age replacements, and some correspond to failures in service. A planned replacement occurs in each period
for which X,, = N - 1, and therefore the long run planned replacements
per unit time is the limiting probability 7TN-,. The difference 7ro - 7TN-, is
the long run rate of failures in service. The equations for the limiting distribution 7r = (7ro, 7T,, ... , 7TN-,) are
P1n1 +
P0ir0 +
+
PN-2ITN-2 + irN-1 = TAO,
(1 - P0)iro
= IT1,
= 72,
(1 - PI)Tr,
(1
7ro +
PN-2)ITN-2
7r, + .. .
= IrN-1,
+ 7TN-I = 1.
Solving in terms of 7ro, we obtain
ITo = 1T0,
I, = (1 - Po) 7T0,
IT2 = (1 - P1)7r1 = (1 - PI)(1 - PO)ITo,
1Tk = (1
Pk-1)lTk-1 = (1
'TN-1 = (1
PN-2)7rN-2 = (1
Pk-l)(1
PN-2)(l
Pk-2) ... (1 - PO)n0,
PN-3) ... (1 - P0)ir0,
223
Examples
2.
and since iro + 7T, +
+ 7rv-, = 1, we have
1 = [1 +(I -Po)+(I -Po)(I
-Pi)+...
+ (1 - NO - Pi) ... (I - PN_2)]moo,
or
I
P) + ... + (I - Po)(1 - Pi) ... (I - PN-2)
1 + (1 - Po) + (I - Pp)(I
for j = 1, 2, ... , where A, = 1, then Pk = ak+,/Ak+,
If A; = a; + a;+, +
and 1 - Pk = Ak+,/Ak+,, which simplifies the expression for 7rp to
ITo=A,+A,+...+Av'
1
and then
A,
Trv-1
=A 7r = A, + A, + ... + A,*
,v
In practice, one determines the cost C of a replacement and the additional cost K that is incurred when a failure in service occurs. Then the
and the replacelong run total cost per unit time is Ciro + K(Iro ment age N is chosen so as to minimize this total cost per unit time.
Observe that
N-I
= A, + A, +
ITO
+ AN
Pr{T ? j} _
Pr{T > k}
k=0
J=1
X
Pr{min{T, N} > k} = E[min(T, N}].
k=0
In words, the reciprocal of the mean time between replacements
E[min{T, N}] yields the long run replacements per unit time iro. This relation will be further explored in the chapter on renewal processes.
2.5.
Optimal Replacement Rules
A common industrial activity is the periodic inspection of some system as
part of a procedure for keeping it operative. After each inspection, a decision must be made whether or not to alter the system at that time. If the
224
IV The Long Run Behav r of Markov Chains
inspection procedure and the ways of modifying the system are fixed, an
important problem is that of determining, according to some cost criterion,
the optimal rule for making the appropriate decision. Here we consider the
case in which the only possible act is to replace the system with a new one.
Suppose that the system is inspected at equally spaced points in time
and that after each inspection it is classified into one of the L + 1 possible states 0, 1, . . . , L. A system is in state 0 if and only if it is new and is
in state L if and only if it is inoperative. Let the inspection times be n =
0, 1, . . . , and let X,, denote the observed state of the system at time n. In
the absence of replacement we assume that
is a Markov chain with
transition probabilities p;; =
jl X,, = i} for all i, j, and n.
It is possible to replace the system at any time before failure. The motivation for doing so may be to avoid the possible consequences of further
deterioration or of failure of the system. A replacement rule, denoted by
R, is a specification of those states at which the system will be replaced.
Replacement takes place at the next inspection time. A replacement rule R
modifies the behavior of the system and results in a modified Markov
chain
n = 0, 1, ... }. The corresponding modified transition probabilities p;;(R) are given by
p,(R) = p;;
if the system is not replaced at state i,
p,o(R) = 1,
and
p;; (R) = 0,
j00
if the system is replaced at state i.
It is assumed that each time the equipment is replaced, a replacement
cost of K units is incurred. Further it is assumed that each unit of time the
system is in state j incurs an operating cost of a;. Note that a. may be interpreted as failure (inoperative) cost. This interpretation leads to the one
period cost function c;(R) given for i = 0, . . . , L by
a,
c, (R) = K + a;
if p,o(R) = 0,
if p;o(R) = 1.
We are interested in replacement rules that minimize the expected long
run time average cost. This cost is given by the expected cost under the
limiting distribution for the Markov chain
Denoting this average
cost by 4(R), we have
225
Examples
2.
(k(R) _ , 7r;(R)c;(R),
i=O
i}. The limiting distribution 7r;(R) is de-
where ir;(R) = lim,,.,
termined by the equations
L
irk(R)pk;(R),
iT;(R) _
i = 0, ... , L,
k=O
and
lTo(R) + ir(R) + ... + lrL(R) = 1.
We define a control limit rule to be a replacement rule of the form
"Replace the system if and only if X,, ? k,"
where k, called the control limit, is some fixed state between 0 and L. We
let Rk denote the control limit rule with control limit equal to k. Then Ro is
the rule "Replace the system at every step," and RL is the rule "Replace
only upon failure (State L)."
Control limit rules seem reasonable provided that the states are labeled
monotonically from best (0) to worst (L) in some sense. Indeed, it can be
shown that a control limit rule is optimal whenever the following two conditions hold:
(1) ao:5 a, :5 ...<aL.
L
L
(2) Ifi<- j,thenY p;pd,,, for every k=0,...,L.
,n=k
ni=k
Condition (1) asserts that the one stage costs are higher in the "worse"
states. Condition (2) asserts that further deterioration is more likely in the
"worse" states.
Let us suppose that conditions (1) and (2) prevail. Then we need only
check the L + 1 control limit rules R,,, . . . , RL in order to find an optimal
rule. Furthermore, it can be shown that a control limit k* satisfying
O(Rk.-1) ? (A(R,<) < 4(Rk,+,) is optimal, so that not always do all L + 1
control limit rules need to be checked.
Under control limit k we have the cost vector
c(Rk) = (an, .
. .
and the transition probabilities
, ak-1,
K + ak,
.
. . ,
K + aL),
226
IV The Long Run Behavior of Markov Chains
0
1
k-1
k
L
0
0
Poi
Po.k-,
Pok
POL
1
0
Pn
P1.k-1
Pik
PAL
0
Pk-II
...
Pk- Lk- ,
Pk-Lk
Pk-I.L
k
1
0
..
0
0
0
L
1
0
0
0
0
P(Rk) = k - 1
To look at a numerical example, we will find the optimal control limit
k* for the following data: L = 5 and
P=
ao =
0
1
2
3
4
5
0
0
0.2
0.2
0.2
0.2
0.2
1
0
0.1
0.2
0.2
0.2
0.3
2
0
0
0.1
0.2
0.3
0.4
3
0
0
0
0.1
0.4
0.5
4
0
0
0
0
0.4
0.6
5
0
0
0
0
0
1
= aL-, = 0, aL = 5, and K = 3. When k = 1, the transition matrix
is
0
P(R,) =
1
2
3
4
5
0
0
0.2
0.2
0.2
0.2
0.2
1
1
0
0
0
0
0
2
1
0
0
0
0
0
1
0
0
0
0
0
4
1
0
0
0
0
0
5
1
0
0
0
0
0
3
2.
Examples
227
which implies the following equations for the stationary distribution:
7T, + 7T2 + 7T3 + 7T4 + 7T5 = IT0,
0.27x0
0.27x0
0.21T0
0.27T0
= 1s,
0.27ro
7T0 +
7T,+7T,+ 7T3+
V'a
+7T5= 1.
The solution is 7T0 = 0.5 and 7T, = 7T2 = 7T3 = 7T4 = 7T5 = 0.1. The aver-
age cost associated with k = 1 is
4, = 0.5(0) + 0.1(3) + 0.1(3) + 0.1(3) + 0.1(3) + 0.1(3 + 5)
= 2 .0 .
When k = 2, the transition matrix is
0
1
2
3
4
5
0
0
0.2
0.2
0.2
0.2
0.2
1
0
0.1
0.2
0.2
0.2
0.3
2
1
0
0
0
0
0
1
0
0
0
0
0
4
1
0
0
0
0
0
5
1
0
0
0
0
0
P(R2) = 3
and the associated stationary distribution is 7To = 0.450, 7r, = 0.100, 7x2 =
7T3 = 7T, = 0.110, 7T5 = 0.120. We evaluate the average cost to be
4)2 = 0.45(0) + 0.10(0) + 0.11(3) + 0.11(3) + 0.11(3) + 0.12(8)
= 1.95.
Continuing in this manner, we obtain the following table:
IV The Long Run Behavior of Markov Chains
228
Control
Limit
Average
Stationary Distribution
k
1
2
3
4
5
Tr0
'TTZ
?r3
7r4
IT5
Y'k
0.1000
0.1100
0.1089
0.0961
0.0756
0.1000
0.1100
0.1198
0.1175
0.0925
0.1000
0.1100
0.1307
0.1623
0.2139
0.1000
0.1200
0.1505
0.1916
0.2785
2.0000
1.9500
IT,
0.5000 0.1000
0.4500 0.1000
0.4010 0.0891
0.3539 0.9786
0.2785 0.06189
Cost
1.9555
2.0197
2.2280
The optimal control limit is k* = 2, and the corresponding minimum
average cost per unit time is 02= 1.95.
Exercises
2.1. On a Southern Pacific island, a sunny day is followed by another
sunny day with probability 0.9, whereas a rainy day is followed by another rainy day with probability 0.2. Supposing that there are only sunny
or rainy days, in the long run on what fraction of days is it sunny?
2.2.
In the reliability example of Section 2.2, what fraction of time is the
repair facility idle? When a second repair facility is added, what fraction
of time is each facility idle?
2.3. Determine the average fraction inspected, AFI, and the average outgoing quality, AOQ, of Section 2.3 for p = 0, 0.05, 0.10, 0.15,. .. , 0.50
when
(a) r = 10 and i = 5.
(b) r = 5 and i = 10.
2.4.
Section 2.2 determined the availability R of a certain computer system to be
I
R = 1+p2
R,
l+p
I +p+p22
for one repair facility,
for two repair facilities,
where p is the computer failure probability on a single day. Compute and
compare R, and R2 for p = 0.01, 0.02, 0.05, and 0.10.
229
Exercises
2.5. From purchase to purchase, a particular customer switches brands
among products A, B, and C according to a Markov chain whose transition probability matrix is
A
B
C
A
0.6
0.2
0.2
P=B
0.1
0.7
0.2
C
0.1
0.1
0.8
In the long run, what fraction of time does this customer purchase brand
A?
2.6. A component of a computer has an active life, measured in discrete
units, that is a random variable T where
Pr{T = 1} = 0.1,
Pr{T = 3} = 0.3,
Pr{T = 2} = 0.2,
Pr{T = 4} = 0.4.
Suppose one starts with a fresh component, and each component is replaced by a new component upon failure. Determine the long run probability that a failure occurs in a given period.
2.7.
Consider a machine whose condition at any time can be observed
and classified as being in one of the following three states:
State 1: Good operating order
State 2: Deteriorated operating order
State 3: In repair
We observe the condition of the machine at the end of each period in a sequence of periods. Let X denote the condition of the machine at the end
of period n for n = 1, 2..... Let X0 be the condition of the machine at the
start. We assume that the sequence of machine conditions is a Markov
chain with transition probabilities
P,=0.9,
p2, = 0,
and that the process starts in state Xo = 1.
P,,=0.1,
230
IV The Long Run Behavior of Markov Chains
(a) Find Pr(X4 = 1).
(b) Calculate the limiting distribution.
(c) What is the long run rate of repairs per unit time?
2.8. At the end of a month, a large retail store classifies each receivable
account according to
0: Current
1: 30-60 days overdue
2: 60-90 days overdue
3: Over 90 days
Each such account moves from state to state according to a Markov chain
with transition probability matrix
0
1
2
3
0
0.95
0.05
0
0
1
0.50
0
0.50
0
2
0.20
0
0
0.80
3
0.10
0
0
0.90
In the long run, what fraction of accounts are over 90 days overdue?
Problems
2.1.
Consider a discrete-time periodic review inventory model (see III,
Section 3.1), and let , be the total demand in period n. Let X,, be the inventory quantity on hand at the end of period n. Instead of following an
(s, S) policy, a (q, Q) policy will be used: If the stock level at the end of
a period is less than or equal to q = 2 units, then Q = 2 additional units
will be ordered and will be available at the beginning of the next period.
Otherwise, no ordering will take place. This is a (q, Q) policy with q = 2
and Q = 2. Assume that demand that is not filled in a period is lost (no
back ordering).
(a) Suppose that X0 = 4, and that the period demands turn out to be
61 = 3, C2 = 4, 6s = 0, 4 = 2. What are the end-of-period stock levels for periods n = 1, 2, 3, 4?
231
Problems
(b) Suppose that Z Z,,
. are independent random variables, each
having the probability distribution where
.
.
=
0
1
2
3
4
Pr{e = k} =
0.1
0.3
0.3
0.2
0.1
k
Then X0, X,, ... is a Markov chain. Determine the transition probability
distribution, and the limiting distribution.
(c) In the long run, during what fraction of periods are orders placed?
2.2.
A system consists of two components operating in parallel: The
system functions if at least one of the components is operating. In any single period, if both components are operating at the beginning of the period, then each will fail, independently, during the period with probability
a. When one component has failed, the remaining component fails during
a period with a higher probability /3. There is a single repair facility, and
it takes two periods to repair a component.
(a) Define an appropriate set of states for the system in the manner of
the Reliability and Redundancy example, and specify the transition
probabilities in terms of a and /3.
(b) When a = 0.1 and /i = 0.2, in the long run what fraction of time
is the system operating?
2.3.
Suppose that a production process changes state according to a
Markov process whose transition probability matrix is given by
0
1
2
3
0
0.2
0.2
0.4
0.2
1
0.5
0.2
0.2
0.1
2
0.2
0.3
0.4
0.1
3
0.1
0.2
0.4
0.3
P
(a) Determine the limiting distribution for the process.
(b) Suppose that states 0 and 1 are "in-control," while states 2 and 3 are
deemed "out-of-control." In the long run, what fraction of time is
the process out-of-control?
IV The Long Run Behav!"f Markov Chains
232
(c) In the long run, what fraction of transitions are from an in-control
state to an out-of-control state?
2.4. A component of a computer has an active life, measured in discrete
units, that is a random variable , where
=
1
2
3
4
Pr{f = k} =
0.1
0.3
0.2
0.4
k
Suppose that one starts with a fresh component, and each component is replaced by a new component upon failure. Let X,, be the remaining life of
the component in service at the end of period n. When X,, = 0, a new item
is placed into service at the start of the next period.
(a) Set up the transition probability matrix for
(b) By showing that the chain is regular and solving for the limiting
distribution, determine the long run probability that the item in service at the end of a period has no remaining life and therefore will
be replaced.
(c) Relate this to the mean life of a component.
2.5. Suppose that the weather on any day depends on the weather conditions during the previous two days. We form a Markov chain with the
following states:
State (S, S) if it was sunny both today and yesterday,
State (S, C) if it was sunny yesterday but cloudy today,
State (C, S) if it was cloudy yesterday but sunny today,
State (C, C) if it was cloudy both today and yesterday,
and transition probability matrix
Today's State
(C, C)
(S, S)
(S, C)
(S,S)
0.7
0.3
0
0
P= (S,C)
0
0
0.4
0.6
(C, S)
0.5
0.5
0
0
(Cl C)
0
0
0.2
0.8
(Cl S)
233
Problems
Given that it is sunny on days 0 and 1, what is the probability it is
sunny on day 5?
(b) In the long run, what fraction of days are sunny?
-(a)
2.6. Consider a computer system that fails on a given day with probability p and remains "up" with probability q = 1 - p. Suppose the repair
time is a random variable N having the probability mass function p(k) =
/3(1
fork= 1,2,...,where0<8< 1.LetX,, = 1 ifthecom-
puter is operating on day n and X,, = 0 if not. Show that {X,, } is a Markov
chain with transition matrix
0
1
and a = 1 - /3. Determine the long run probability that the computer is
operating in terms of a, /3, p, and q.
2.7. Customers arrive for service and take their place in a waiting line.
There is a single service facility, and a customer undergoing service at the
beginning of a period will complete service and depart at the end of the
period with probability (3 and will continue service into the next period
with probability a = 1 - /3, and then the process repeats. This description
implies that the service time q of an individual is a random variable with
the geometric distribution,
Pr{rl = k} = /3ak-'
for k = 1, 2,
... ,
and the service times of distinct customers are independent random variables.
At most a single customer can arrive during a period. We suppose that
the actual number of arrivals during the nth period is a random variable
taking on the values 0 or 1 according to
=p
and
Pr(,= 1}=q=1-p
forn = 0, 1,
. . . .
The state X,, of the system at the start of period n is defined to be the
number of customers in the system, either waiting or being served. Then
234
IV The Long Run Behavior of Markov Chains
is a Markov chain. Specify the following transition probabilities in
terms of a, /3, p, and q: P. PO , P02, P,p, P, and P,,. State any additional
assumptions that you make.
2.8. An airline reservation system has a single computer, which breaks
down on any given day with probability p. It takes two days to restore a
failed computer to normal service. Form a Markov chain by taking as
states the pairs (x, y), where x is the number of machines in operating condition at the end of a day and y is 1 if a day's labor has been expended on
a machine, and 0 otherwise. The transition probability matrix is
To State
-4 (1, 0)
(0, 0)
(0, 1)
From State
I
(1,0)
P = (0, 0)
(0, 1)
q
p
0
0
0
1
1
0
0
Compute the system availability iro for p = 0.01, 0.02, 0.05, and 0.10.
3.
The Classification of States
Not all Markov chains are regular. We consider some examples.
The Markov chain whose transition probability matrix is the identity
matrix
P=0
0
1
1
0
0
1
remains always in the state in which it starts. Since trivially P" = P for all
n, the Markov chain X,, has a limiting distribution, but it obviously depends on the initial state.
The Markov chain whose transition probability matrix is
0 1
P=010
10
3.
The Classification of States
235
oscillates deterministically between the two states. The Markov chain is
periodic, and no limiting distribution exists. When n is an odd number,
then P" = P, but when n is even, then P" is the 2 X 2 identity matrix.
When P is the matrix
0
0
P
1
2
1110z
1'
P" is given by
0
P" =
1
01 (z)"
1
1 - (z ),f
0
1
and the limit is
lim P" =
0
1
0110
1
0
1
1
Here state 0 is transient; after the process starts from state 0, there is a
positive probability that it will never return to that state.
The three matrices just presented illustrated three distinct types of behavior in addition to the convergence exemplified by a regular Markov
chain. Various and more elaborate combinations of these behaviors are
also possible. Some definitions and classifications of states and matrices
are needed in order to sort out the variety of possibilities.
3.1.
Irreducible Markov Chains
Since j is said to be accessible from state i if P;' > 0 for some intege
n ? 0; i.e., state j is accessible from state i if there is positive probability
that state j can be reached starting from state i in some finite number of
transitions. Two states i and j, each accessible to the other, are said to com-
municate, and we write i H j. If two states i and j do not communicate,
then either
P;;' = 0
for all n >- 0
IV The-Long Run Behavior of Markov Chains
236
or
for all n ::,-0
P%;') = 0
or both relations are true. The concept of communication is an equivalence relation:
(i) i H i (reflexivity), a consequence of the definition of
=j,
(1
P1)=S;;=10
i
j
(ii) If i t--* j, then j t4 i (symmetry), from the definition of communication.
(iii) If i H j and j t-4 k, then i H k (transitivity).
The proof of transitivity proceeds as follows: i <-4j and j H k imply that
there exist integers n and m such that P;) > 0 and P;'k)> 0. Consequently,
by the nonnegativity of each P;,), we conclude that
x
P('?) P('11)
it
k
ii
rk
p(,n)
> 0.
ik
r=0
A similar argument shows the existence of an integer v such that Pki°) > 0,
as desired.
We can now partition the totality of states into equivalence classes. The
states in an equivalence class are those that communicate with each other.
It may be possible starting in one class to enter some other class with positive probability; if so, however, it is clearly not possible to return to the
initial class, or else the two classes would together form a single class. We
say that the Markov chain is irreducible if the equivalence relation induces
only one class. In other words, a process is irreducible if all states communicate with each other.
To illustrate this concept, we consider the transition probability matrix
0 0 0
a
P
a
00
00
0 0 0
1
0
0
z
0 0.0 1
0
0
P
0
0.0
P°I
3.
The Classification of States
237
where P, is an abbreviation for the matrix formed from the initial
two rows and columns of P, and similarly for P2. This Markov chain
clearly divides into the two classes composed of states (1, 2) and states
{3,4,5}.
If the state of Xo lies in the first class, then the state of the system thereafter remains in this class, and for all purposes the relevant transition matrix is P. Similarly, if the initial state belongs to the second class, then the
relevant transition matrix is R. This is a situation where we have two
completely unrelated processes labeled together.
In the random walk model with transition matrix
states
P=
1
0
0
0
0
0
0
0
q
0p
0
0
0
0
1
0
q
p
0
0
0
2
0
(3.1)
0
...
q
0
0
...
0
0
p a-1
1
a
we have the three classes (0), 11, 2, . . . , a - 1), and {a}. In this example
it is possible to reach the first class or third class from the second class,
but it is not possible to return to the second class from either the first or
the third class.
3.2.
Periodicity of a Markov Chain
We define the period of state i, written d(i), to be the greatest common di-
visor (g.c.d.) of all integers n ? 1 for which P;;'' > 0. [If P;;' = 0 for all
n ? 1, define d(i) = 0.] In a random walk (3.1), every transient state
1, 2, ... , N - 1 has period 2. If P; > 0 for some single state i, then that
state now has period 1, since the system can remain in this state any length
of time.
In a finite Markov chain of n states with transition matrix
238
IV The Long Run Behavior of Markov Chains
0
1
n
0 0
0
0
1
0
0
1
0
0
0
P=
1
0
0
each state has period n.
Consider the Markov chain whose transition probability matrix is
P=
0
1
2
3
0
0
1
0
0
1
0
0
1
0
2
0
0
0
1
3
z
0
2
0
We evaluate P.=0,P'.1) =0,P'1'=0,P11=1,P1,10) = 0,P1,1) =;. The set
of integers n ? 1 for which P > 0 is {4, 6, 8, ... }. The period of state 0
is d(0) = 2, the greatest common divisor of this set.
Example Suppose that the precipitation in a certain locale depends on
the season (Wet or Dry) as well as on the precipitation level (High, Low)
during the preceding season. We model the process as a Markov chain
whose states are of the form (x, y), where x denotes the season (W = Wet,
D = Dry) and y denotes the precipitation level (H = High, L = Low).
Suppose the transition probability matrix is
(W, H)
(W, H)
P=
(W, L)
(D, H)
(D, L)
0
0
0.8
0.2
(W,L)
(D, H)
0
0
0.4
0.6
0.7
0.3
0
0
(D, L)
0.2
0.8
0
0
All states are periodic with period d = 2.
3.
The Classification of States
239
A situation in which the demand for an inventory item depends on the
month of the year as well as on the demand during the previous month
would lead to a Markov chain whose states had period d = 12.
The random walk on the states 0, ± 1, ±2, . . . with probabilities
P;.i+, = p, P,, = q = I - p is periodic with period d = 2.
We state, without proof, three basic properties of the period of a state:
1. If i t-- j then d(i) = d(j).
This assertion shows that the period is a constant in each class of
communicating states.
2. If state i has period d(i), then there exists an integer N depending on
i such that for all integers n ? N,
p;ud<<» > 0.
This asserts that a return to state i can occur at all sufficiently large
multiples of the period d(i).
3. If P;' > 0, then P;"+"
> 0 for all n (a positive integer) sufficiently
large.
A Markov chain in which each state has period 1 is called aperiodic.
The vast majority of Markov chain processes we deal with are aperiodic.
Results will be developed for the aperiodic case, and the modified conclusions for the general case will be stated, usually without proof.
3.3.
Recurrent and Transient States
Consider an arbitrary, but fixed, state i. We define, for each integer n ? 1,
i, v= 1,2,...,n- 1X0=i}.
In other words, f,;" is the probability that starting from state i, the first return to state i occurs at the nth transition. Clearly f,!;" = P;;, and f °i' may be
calculated recursively according to
P(n) =
n ? 1,
(3.2)
k=O
where we define f ' = 0 for all i. Equation (3.2) is derived by decomposing the event from which P;;" is computed according to the time of the first
240
IV The Long Run Behavior of Markov Chains
return to state i. Indeed, consider all the possible realizations of the
process for which X0 = i, X,, = i, and the first return to state i occurs at the
kth transition. Call this event Ek. The events Ek (k = 1, 2, ... , n) are
clearly mutually exclusive. The probability of the event that the first return is at the kth transition is by definition f k). In the remaining n - k tran-
sitions, we are dealing only with those realizations for which X,, = i.
Using the Markov property, we have
Pr{Ek} = Pr{first return is at kth transitionlX0 = i} Pr{X = iIXk = i}
-Jii{(k)iiin-k)
1<k-<n
(recall that P° = 1). Hence
,r
Pr(X
= i10
X = i }l
rr
k=1
Pr(E.
l k} -
(k)
f ii
u
k=1
-k) _
(k)P(n-k)
k=0
f ii
n
since by definition f ;°j = 0. The verification of (3.2) is now complete.
When the process starts from state i, the probability that it returns to
state i at some time is
N-' =o f ii
f ii
Jii
=o
(3.3)
We say that a state i is recurrent if fi = 1. This definition says that a state
i is recurrent if and only if, after the process starts from state i, the probability of its returning to state i after some finite length of time is one. A
nonrecurrent state is said to be transient.
Consider a transient state i. Then the probability that a process starting
from state i returns to state i at least once is fi < 1. Because of the Markov
property, the probability that the process returns to state i at least twice is
(fi )2, and repeating the argument, we see that the probability that the
process returns to i at least k times is (fi)k for k = 1, 2, .... Let M be the
random variable that counts the number of times that the process returns
to i. Then we have shown that M has the geometric distribution in which
Pr{M?kjX0=i} =(fii)k
fork= 1,2,...
(3.4)
and
E[
0
i]
=
f,
1 -,fi
( 3 .5 )
3.
The Classification of States
241
Theorem 3.1 establishes a criterion for the recurrence of a state i in
terms of the transition probabilities P(i"'.
Theorem 3.1 A state i is recurrent if and only if
x
`
"ii)
n=1
=
00.
Equivalently, state i is transient if and only if
P;") < oo.
Proof Suppose first that state i is transient so that, by definition, fi < 1,
and let M count the total number of returns to state i. We write M in terms
of indicator random variables as
1(X = i},
where
1
={0
if
X 0 i.
Now, equation (3.5) shows that E[MIXo = i] < oo when i is transient. But
then
x
E[1(X = i) I Xo = i]
oo > E[MI Xo = i] =
x
`ii"),
11=1
as claimed.
Conversely, suppose
Pi;"' < oo. Then M is a random variable whose
mean is finite, and thus M must be finite. That is, starting from state i, the
process returns to state i only a finite number of times. Then there must be
a positive probability that, starting from state i, the process never returns
to that state. In other words I - fii > 0 or fi < 1, as claimed.
Corollary 3.1 If i H j and if i is recurrent, then j is recurrent.
242
IV The Long Run Behavior of Markov Chains
Proof Since i <-4j, there exists m, n ? 1 such that
P;") > 0
and
P,',") > 0.
Let v > 0. We obtain, by the usual argument (see Section 3.1),
P("'+"+°)
P1 ;"')P;°)Pf") and, on summing,
x
x+
P(m+it+v)
v=o
) m)P(OP(n) = P(m)P(n)
P)
V=o
P;") also diverges.
Hence if V'=o P;'> diverges, then
11
This corollary proves that recurrence, like periodicity, is a class prop-
erty; that is, all states in a equivalence class are either recurrent or
nonrecurrent.
Example Consider the one-dimensional random walk on the positive
and negative integers, where at each transition the particle moves with
probability p one unit to the right and with probability q one unit to the
left (p + q = 1). Hence
p
(3.6)
n
n!n!
We appeal now to Stirling's formula (see 1, (6.10)),
n! ,r
W+in-e-'
.
(3.7)
Applying (3.7) to (3.6), we obtain
It is readily verified that p(1 - p) = pq <_ , with equality holding if and
only if p = q = 2. Hence
Pte' = oc if and only if p = ;. Therefore, from
Theorem 3.1, the one-dimensional random walk is recurrent if and only if
p = q = Remember that recurrence is a class property. Intuitively, if
p q, there is positive probability that a particle initially at the origin will
drift to +cc if p > q (to -o if p < q) without ever returning to the origin.
243
Exercises
Exercises
3.1.
A Markov chain has a transition probability matrix
234
000
567
000
0 1
0
00 000
1
00
000 1 0 000
2
0000
000
3
4 0.5 0 0 0 0 0.5 0 0
0
1
1
1
5
6
7
00000
00000
10000
01 0
00
1
000
For which integers n = 1, 2, ..., 20 is it true that
P%) > 0?
What is the period of the Markov chain?
Hint:
One need not compute the actual probabilities. See Section 1.1.
3.2. Which states are transient and which are recurrent in the Markov
chain whose transition probability matrix is
0
1
2
0
3
4
0
0
0
1
1
1
2
4
4
0
0
0
1
1
1
4
4
4
0
0
1
0
0
0
0
0
0
0
0
0
1
7
0
0
244
IV The Long Run Behavior of Markov Chains
3.3.
A Markov chain on states 10, 1, 2, 3, 4, 5} has transition probability matrices
2
0
0
43
0
0
0
45
0
0
4
4
(a)
1
1
6
6
1
0
0
0
0
0
0
0
0
0
0
0
I
3
0
8
8
4
4
0
0
0
1
(b)
0
Find all communicating classes.
3.4. Determine the communicating classes and period for each state of
the Markov chain whose transition probability matrix is
0
1
2
3
4
5
0
0
0
;
0
0
0
1
0
0
0
2 0
0
0
1
0
0
3 0
0
0
0
1
0
4 0
0
0
0
0
1
5 0
0
3
1
3
0
3
0
1
4.
The Basic Limit Theorem of Markov Chains
245
Problems
3.1.
A two state Markov chain has the transition probability matrix
0
1
P=0
a
1-b
I1 b a
(a) Determine the first return distribution
(b) Verify equation (3.2) when i = 0. (Refer to III, (5.2).)
Show that a finite state aperiodic irreducible Markov chain is regular and recurrent.
3.2.
3.3.
Recall the first return distribution (Section 3.3)
J"'=Pr{X,
0 by convention. Using equation (3.2), determine f', n = 1, 2,
3, 4, 5, for the Markov chain whose transition probability matrix is
4.
0
1
2
3
0
0
2'
0
2'
1
0
0
1
0
2
0
0
0
1
3
z
0
0
z
The Basic Limit Theorem of Markov Chains
Consider a recurrent state i. Then
f;;''= Pr{X = i,X,
i for v= 1,...,n - 11X°= i}
(4.1)
is the probability distribution of the first return time
R; = min( n ? 1; X = i).
(4.2)
246
IV
The Long Run Behavior of Markov Chains
This is
f = Pr{R; = njXo = i }
for n = 1, 2,
....
(4.3)
Since state i is recurrent by assumption, then f; _ I", f;'' = 1, and R; is
a finite-valued random variable. The mean duration between visits to state
i is
m; = E[R;IXO = i] = 7, nfj'.
(4.4)
/1 = 1
After starting in i, then, on the average, the process is in state i once every
m; = E[R;IXO = i] units of time. The basic limit theorem of Markov chains
states this result in a sharpened form.
Theorem 4.1 The basic limit theorem of Markov chains
(a) Consider a recurrent irreducible aperiodic Markov chain. Let
P' be the probability of entering state i at the nth transition, n = 0, 1,
given that X0 = i (the initial state is i). By our earlier convention
1. Let f;," be the probability of first returning to state i at the nth
transition, n = 0, 1 , 2, ... , where f ;;°' = 0. Then,
2,
. . .
,
lim
1t" =
I
1
1
',=
fc
(b) under the same conditions as in (a),
states j.
mr
Pi;'ll =
P;"'for all
Remark Let C be a recurrent class. Then P,"' = 0 for i E C, j (4 C, and
every n. Hence, once in C it is not possible to leave C. It follows that the
submatrix JIPJI, i, j E C, is a transition probability matrix and the associated Markov chain is irreducible and recurrent. The limit theorem, therefore, applies verbatim to any aperiodic recurrent class.
If lim,,. P,;"I > 0 for one i in an aperiodic recurrent class, then 9T; > 0
for all j in the class of i. In this case, we call the class positive recurrent
or strongly ergodic. If each ?r; = 0 and the class is recurrent, we speak of
the class as null recurrent or weakly ergodic. In terms of the first return
time R; = min { n ? 1; X = i } , state i is positive recurrent if m; _
E[R;IXQ = i] < c and null recurrent if m; = °. This statement is immediate from the equality lim,,. P;;"' = ir; = 1/m;. An alternative method for
4.
The Basic Limit Theorem of Markov Chains
247
determining the limiting distribution ?r, for a positive recurrent aperiodic
class is given in Theorem 4.2.
Theorem 4.2 In a positive recurrent aperiodic class with states j = 0,
1,2,...,
limPJ"='7r,=2.
= 1,
'TiPip
i=0
i=0
and the ir's are uniquely determined by the set of equations
x
x
ir,?0, , ir,= 1,
ir,P; forj = 0, 1,
and
. . . .
(4.6)
r=0
1=0
Any set (i71);_0 satisfying (4.6) is called a stationary probability distribution of the Markov chain. The term "stationary" derives from the property that a Markov chain started according to a stationary distribution will
follow this distribution at all points of time. Formally, if Pr{Xo = i} = 1r
then Pr{X = i} = 1r, for all n = 1, 2, .... We check this for the case
n = 1; the general case follows by induction. We write
Pr{X1 = i}
> Pr{Xo = k) Pr{X1 = iIX0 = k}
k=0
X
1r,P = IT,,
k=0
where the last equality follows because 7r = (1r, 1r ...) is a stationary
distribution. When the initial state X is selected according to the stationis given
ary distribution, then the joint probability distribution of (X,,,
by
Pr (X,, = i, X,,, = j) = 7r,P,,.
The reader should supply the proof.
A limiting distribution, when it exists, is always a stationary distribution, but the converse is not true. There may exist a stationary distribution
but no limiting distribution. For example, there is no limiting distribution
for the periodic Markov chain whose transition probability matrix is
P=°011'
IV The Long Run Behavior of Markov Chains
248
but it = (;,
;)
is a stationary distribution, since
0
1
1
0
Example Consider the class of random walks whose transition matrices are given by
P = IIP;II =
0
1
0
q,
0
P,
0
q2
0
P2
This Markov chain has period 2. Nevertheless we investigate the existence of a stationary probability distribution; i.e., we wish to determine
the positive solutions of
i = 0, 1,
x; = Z, x;P;, = P,-,x,-1 + q,+,x,+,,
i=o
... ,
(4.7)
under the normalization
i=o
where p-, = 0 and po = 1, and thus xo = q,x,. Using equation (4.7) for
i = 1, we could determine x2 in terms of xo. Equation (4.7) for i = 2 determines x3 in terms of xo, and so forth. It is immediately verified that
P,-,P,-2
q,q,-,
r-i
xo = xo
q,
Pk
k=O qk+,
is a solution of (4.7), with xo still to be determined. Now, since
i-I
1 =xo+IxoF1
Pk ,
i=1
we have
1
xo =
1 + E;_, IIk o
Pk
qk+,
The Basic Limit Theorem of Markov Chains
4.
249
and so
x i-l
xo>Oifandonly if> f
Pk
<oo.
i=1 k=o qk+l
In particular, if Pk = p and qk = q = 1 - p fork ? 1, the series
x (-i
i=1
1
x
r
P=\q/
k oqk+l
converges only when p < q, and then
1Z(p)1/
plq
Pr=i q
Pq
P
q-P
and
xo =
xk =
l+l/(q-p)
p 1q1 xo = Zp (1
1+q-p
- q) q )k
2(q)'
for k = 1, 2,
... .
Example Consider now the Markov chain that represents the success
runs of binomial trials. The transition probability matrix is
Po
1-pa
0
0
P,
0
1-p,
0
A
0
0
1-P2
(0<pk<1).
The states of this Markov chain all belong to the same equivalence class
(any state can be reached from any other state). Since recurrence is a class
property (see Corollary 3.1), we will investigate recurrence for the zeroth
state.
Let R. = min{n ? 1; X = 0} be the time of first return to state 0. It is
easy to evaluate
IV The Long Run Behavior of Markov Chains
250
Pr {Ro > 11X0 = 01 = (1 - Po),
Pr(R0 > 21X0 = 0) = (1 - Po)(l - Pi),
Pr{Ro > 31X0 = 0) = (1 - NO - Pi)(1 - P2),
k-1
Pr{R0 > k1 Xo = 0) = (1 - Po)(1 - Pi) ... (I - Pk-t) = 11 (I - Pr).
i=o
In terms of the first return distribution
foo =Pr(RO=nIX0=0),
we have
Pr{Ro > k1Xo = 0) =
f 11,0E
0
,
or
k-1
k
foo = I - Pr(Ro > klX0 = 0) = 1 - fl (1 - p;).
,,=1
;=O
By definition, state 0 is recurrent provided I,,_, ff' = 1. In terms of
po, pl, ... then, state 0 is recurrent whenever "Mk.. IIk=o (1 - p) _
I1;=, (1 - p;) = 0. Lemma 4.1 shows that II o (I - p;) = 0 is equivalent,
in this case, to the condition Y,;_ p; = oo.
Lemma4.1 If0<p;<1,i=0, 1,2,...,then u,, =H o(1 -p;)-0
as in - 00 if and only if t, =o p; = oo.
Proof Assume J o p; = co. Since the series expansion for exp(-p;) is
an alternating series with terms decreasing in absolute value, we can write
1 - P, < 1 - P; +
P
3l
2!
3!
+ ... = exp(-Pr), i = 0, 1, 2..... (4.8)
Since (4.8) holds for all i, we obtain II;"=o (1 - p;) < exp(-I;"o p;). But
by assumption,
'II
1im
,,,_*t ;=o
4.
The Basic Limit Theorem of Markov Chains
251
hence
in
lim 7, (1 - pi) = 0.
m-,- 1=0
To prove necessity, observe that from a straightforward induction,
n:
Fl(1-pi)>(1-pi-pi.,-...-P.)
i=i
for any j and all m = j + 1, j + 2..... Assume now that J;_, pi < oo; then
p, < 1 for some j > 1. Thus
0<
m
lim F1 (1 - p;) > lim (1
-
m
p)
,
which contradicts u,,, -* 0.
State 0 is recurrent when II;=o (1 - pi) = 0, or equivalently, when
Ji=o p; = oo. The state is positive recurrent when mo = E[RoIXQ = 0] < oo.
But
mo = I Pr(R0 > kIX0 = 0)
k=0
x k-1
=1
+Ifl(1-pi).
Thus positive recurrence requires the stronger condition that Ik=, IIk=o
(1 - pi) < oo, and in this case, the stationary probability 7r0 is given by
IT0
m0
1 + Yk=1 llk=0 (1
Pi)
From the equations for the stationary distribution we have
(1 - P0)1T0 = IT1,
(1-P1)7r,=7r,,
(1 - P2)7r2 = 1rs,
IV The Long Run Behavior of Markov Chains
252
or
IT, = (I - Po) lo,
172 = (I - P1)1Ti = (I - Pi)(' - Po)lTo,
177 = (I - P2)1r2 = (I - P2)(1 - PA)(I - Po)17o,
and, in general,
k-1
Irk=7ro F1 (1 -p)
fork? 1.
i=o
In the special case where p, = p = 1 - q for i = 0, 1, ... , then
II;=o (I -Pi)=q,k
x
1
qk=-,
mo= 1 +
P
k=1
so that irk = pqk for k = 0, 1, ... .
Remark Suppose ao, a,, a2i ... is a convergent sequence of real numbers where a - a as n - oo. Then it can be proved by elementary methods that the partial averages of the sequence also converge in the form
1 J, - 1
lim - 7 ak = a.
(4.9)
n k=0
Applying (4.9) with a,, = P,;'', where i is a member of a positive recurrent
aperiodic class, we obtain
1 n-,
I
Jim -7 P,o"'=7r,=->0,
,t-," n,,=o
m,
(4.10)
where 7r = (7ro, 7r,, ...) is the stationary distribution and where m, is the
mean return time for state i. Let M;"' be the random variable that counts
the total number of visits to state i during time periods 0, 1, ... , n - 1.
We may write
n-I
M;"'=> 1(Xk=i},
k=0
where
(4.11)
4.
The Basic Limit Theorem of Markov Chains
253
if Xk = i,
if Xk = i,
1
Ilk = i } = 10
(4.12)
and then see that
n-i
n-'
E[M;")IXo = i] = 7 E[1{Xk = i}IX0 = i] _
P,;k).
(4.13)
k=0
k=0
Then, referring to (4.10), we have
1
1
lim - E[M;")I XO = i] -
"- n
m;
(4.14)
In words, the long run (n -> c) mean visits to state i per unit time equals
7r;, the probability of state i under the stationary distribution.
Next, let r(i) define a cost or rate to be accumulated upon each visit to
state i. The total cost accumulated during the first n stages is
n-I
n-I
Rc"-"
= 7- r(Xk) = 7, 7 1 {X4. = i } r(i)
k=0
k=0
i
(4.15)
W
_
M;")r(i).
i=0
This leads to the following derivation showing that the long run mean cost
per unit time equals the mean cost evaluated over the stationary distribution:
I
lim 1 E[R°'-')IX0 = i] = lim
E[M;")IXO = i]r(i)
n
"-Ix l=() n
(4.16)
ir,r(i).
i=0
(When the Markov chain has an infinite number of states, then the
derivation requires that a limit and infinite sum be interchanged. A sufficient condition to justify this interchange is that r(i) be a bounded function of i.)
254
IV The Long Run Behavior of Markov Chains
Remark The Periodic Case If i is a member of a recurrent periodic irreducible Markov chain with period d, one can show that P;;' = 0 if m is
not a multiple of d (i.e., if m : nd for any n), and that
d
limP;nd=
m,
n-ax
.
These last two results are easily combined with (4.9) to show that (4.10)
also holds in the periodic case. If m; < oo, then the chain is positive recurrent and
1im"-'" n,,,=o
(4.17)
where 'rr = (iro, Ir ...) is given as the unique nonnegative solution to
71kPkj,
j=0,1,...,
k=0
and
= 1.
j=0
That is, a unique stationary distribution 'rr = (Iro, ir,, ...) exists for a positive recurrent periodic irreducible Markov chain, and the mean fraction
of time in state i converges to ir; as the number of stages n grows to infinity.
The convergence of (4.17) does not require the chain to start in state i.
Under the same conditions,
-,
lim-7
Pk;"'=;_-,= n ,,,=o
mi
1
holds for all states k = 0, 1.... as well.
Exercises
4.1.
Determine the limiting distribution for the Markov chain whose
transition probability matrix is
255
Exercises
1
2
3
4
0
gp
0
0
0
1
g
0
p
0
0
p=2 g
0
3
g
0
0 p 0
0 0 p
4
1
0
0
0
0
0
wherep > 0, q > 0, andp + q = 1.
Consider the Markov chain whose transition probability matrix is
given by
4.2.
P=
0
1
2
3
0
0
1
0
0
1
0.1
0.4
0.2
0.3
2
0.2
0.2
0.5
0.1
3
0.3
0.3
0.4
0
(a) Determine the limiting probability .7ro that the process is in state 0.
(b) By pretending that state 0 is absorbing, use a first step analysis (III,
Section 4) and calculate the mean time m,0 for the process to go
from state 1 to state 0.
(c) Because the process always goes directly to state 1 from state 0, the
mean return time to state 0 is m = 1 + m,,,. Verify equation (4.5),
ir = 1/NO.
4.3. Determine the stationary distribution for the periodic Markov chain
whose transition probability matrix is
0
1
2
3
0110
2
0
2
0
0
0
0
0
2
0
11
256
IV The Long Run Behavior of Markov Chains
Problems
4.1. Consider the Markov chain on (0, 1} whose transition probability
matrix is
0
1
Oll a
0<a,/3<1.
a/ill'
l
(a) Verify that (Iro, ir,) = ((3/(a + /3), al(a + (3)) is a stationary distribution.
(b) Show that the first return distribution to state 0 is given by f.(') _
(1 - a) and f . " ' ) = a/3(1 - /3)i-2 f o r n = 2, 3, ... .
(c) Calculate the mean return time mo = Y,'_, nff'' and verify that
Trp = 1 /mo.
4.2. Determine the stationary distribution for the Markov chain whose
transition probability matrix is
0
P
2
3
0
1
2
3
0
0
2
2
0
0
I
3
4
4
1
2
3
3
3
2
3
0
0
0
0
1
Consider a random walk Markov chain on state 0, 1, ...
transition probability matrix
4.3.
0
1
2
3
4
5
N- 1
N
0
0
1
0
0
0
0
0
0
1
q,
0
p,
0
0
0
0
q2
0
P2
0
3
0
0
0
q3
0
P3
0
0
0
0
0
0
N-1
0
0
0
0
0
0
0
P N- I
N
0
0
0
0
0
0
2
...
...
0
, N with
257
Problems
where pi +q; = l,pi> O,q;> 0foralli.
The transition probabilities from state 0 and N "reflect" the process
back into state 1, 2, . . . , N - 1. Determine the limiting distribution.
4.4. Let {a; : i = 1, 2, ...} be a probability distribution, and consider
the Markov chain whose transition probability matrix is
0
1
2
3
4
5
0
a,
a,
a,
a4
a5
a6
1
1
0
0
0
0
0
2
0
1
0
0
0
0
3
0
0
1
0
0
0
4
0
0
0
1
0
0
What condition on the probability distribution { a; : i = 1, 2, ... } is necessary and sufficient in order that a limiting distribution exist, and what is
this limiting distribution? Assume a, > 0 and a2 > 0, so that the chain is
aperiodic.
4.5. Let P be the transition probability matrix of a finite state regular
Markov chain. Let M = Ilm;jll be the matrix of mean return times.
(a) Use a first step argument to establish that
mj=1+
k*j
Pkmk j.
(b) Multiply both sides of the preceding by ir, and sum to obtain
7r,Pkmkj.
k*j
i
Simplify this to show (see equation (4.5))
Trjm,, = 1,
4.6.
or irj = 1/m;;.
Determine the period of state 0 in the Markov chain whose transition probability matrix is
IV The Long Run Behavior of Markov Chains
258
P=
-1
-2
-3
-4
1
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
2
0
2
0
0
0
-1
0
0
0
0
0
1
0
0
-2
0
0
0
0
0
0
1
0
-3
0
0
0
0
0
0
0
1
-4
0
0
0
1
0
0
0
0
3
2
1
0
3
0
0
0
2
1
0
1
0
0
4.7. An individual either drives his car or walks in going from his home
to his office in the morning, and from his office to his home in the afternoon. He uses the following strategy: If it is raining in the morning, then
he drives the car, provided it is at home to be taken. Similarly, if it is rain-
ing in the afternoon and his car is at the office, then he drives the car
home. He walks on any morning or afternoon that it is not raining or the
car is not where he is. Assume that, independent of the past, it rains during successive mornings and afternoons with constant probability p. In the
long run, on what fraction of days does our man walk in the rain? What if
he owns two cars?
4.8.
A Markov chain on states 0, 1,
. .
.
has transition probabilities
forj=0, 1,...,i,i+ 1.
Find the stationary distribution.
5.
Reducible Markov Chains*
Recall that states i and j communicate if it is possible to reach state j starting from state i, and vice versa, and a Markov chain is irreducible if all
* The overwhelming majority of Markov chains encountered in stochastic modeling are
irreducible. Reducible Markov chains form a specialized topic.
5.
259
Reducible Markov Chains
pairs of states communicate. In this section we show, mostly by example,
how to analyze more general Markov chains.
Consider first the Markov chain whose transition probability matrix is
which we write in the form
P-
OI
I
P" 11,
where
I
1
2
2
P, _
I
3
4
4
P, =
and
1
2
3
3
2
,
3
3
The chain has two communicating classes, the first two states forming one
class and the last two states forming the other. Then
P2 =
I
I
2
2
1
3
4
4
0
0
0
0
0
3
5
8
8
5
,I
16
16
0
0
0
0
0
0
0
1
2
3
3
2
I
3
3
x
0
0
0
5
4
9
4
9
I
2
I
3
4
4
0
0
0
0
0
9
I
2
0
0
0
0
1
2
3
3
2
I
3
3
P;
0
0
Pz
5
9
and, in general,
P" =
P;t
0
0
P"
2
n ? 1.
(5.1)
IV The Long Run Behavior of Markov Chains
260
Equation (5.1) is the mathematical expression of the property that it is not
possible to communicate back and forth between distinct communicating
classes; once in the first class the process remains there thereafter; and
similarly, once in the second class, the process remains there. In effect,
two completely unrelated processes have been labeled together. The transition probability matrix P is reducible to the irreducible matrices PI and
P2. It follows from (5.1) that
limp"
=
-+x
where
7r0')
limp;' _
?To')
lro'
?Ti' )
and limp" _
ITl 1)
I0(2)
We solve for 7T(" = (7r."), ?T(,") and ir(2' _ (aro22), 7T 2)) in the usual way:
27T0
1
+47T1
7T0
(1)
7TO
+ - 7T(I)
i
=
(1)
7Ti
IT0(1) + IT( 11)= 1,
or
1
0
3s
,71)=2
1
3s
(5.2)
and because P2 is doubly stochastic (see Section 1.1), it follows that
1, X12) = 1
The basic limit theorem of Markov chains, Theorem 4.1, referred to an
irreducible Markov chain. The limit theorem applies verbatim to any
aperiodic recurrent class in a reducible Markov chain. If i, j are in the same
aperiodic recurrent class, then P"' - 1/m; > 0 as n - cc. If i, j are in the
same periodic recurrent class, then n-'
-40 11 =0 P' -4 1/m; ? 0 as n -' oo.
If j is a transient state, then P")
as n -4 oc, and, more generally,
P;'' -4 0 as n -4 oo for all initial states i.
5.
Reducible Markov Chains
261
In order to complete the discussion of the limiting behavior of P;"', we
still must consider the case where i is transient and j is recurrent. Consider
the transition probability matrix
P=
0
1
2
3
0
2
2
0
0
1
4
4
0
0
2
4
4
4
4
3
0
0
0
1
There are three classes: 10, 1}, {2), and {3 }, an d of these (0, 11 and {3) are
recurrent, while (2) is transient. Starting from state 2, the process ultimately gets absorbed in one of the other classes. The question is, Which
one? or more precisely, What are the probabilities of absorption in the two
recurrent classes starting from state 2?
A first step analysis answers the question. Let u denote the probability
of absorption iris class 10, 1) starting from state 2. Then 1 - u is the probability of absorption in class (3 1. Conditioning on the first step, we have
u= (4+')I +qu+,-(0)=2+;u,
or u =
. With probability
the process enters 10, 1) and remains there
ever after. The stationary distribution for the recurrent class (0, 1), computed in (5.2), is ar _ ;, Ir, _ . Therefore,
P;o' = X ; _ 2
P',; _ X -_ 1. That is, we multiply the probability of entering the
class 10, 11 by the appropriate probabilities under the stationary distribution for the various states in the class. In matrix form, the limiting behavior of P" is given by
0
3
lime" =
1
2
3
3
2
4
9
9
0
0
0
0
0
0
1
To firm up the principles, consider one last example:
IV The Long Run Behavior of Markov Chains
262
0
1
2
3
4
5
I
2
0
0
0
0
3
0
0
0
0
13
6
6
I.
1
3
0
2
P"=.
0
6
6
0
3
6
0
0
0
0
1
0
0
0
1
0
I
3
6
40
50
1
1
There are three classes: C, = {0, 11, C2 = {2, 3), and C3 = {4, 5). The stationary distribution in C, is (Tro, ir,), where
1r0+3IT1 = IT0>
'T0 + 3 IT1 = IT1 >
Ir0 + IT, = 1.
and ?r, =5.
Then
Class C3 is periodic, and P;"' does not converge for it j in C3 = (4, 5).
The time averages do converge, however; and lim nix n- I"-'m=0P(')ij _ = 2for
>
i = 3, 4 andj = 3, 4.
For the transient class C2 = {2, 3}, let u; be the probability of ultimate
absorption in class C, = {0, 1) starting from state i for i = 2, 3. From a
first step analysis, then
u2=3(1)+0(1)+Ou2+3u3+6(0)+6(0),
U3 = 6(1) + 6(1) + 6U2 + 0U3 + 3(0) + 6(0),
or
U2 -3+A;
1
The solution is u2 =
trix form, we have
1
I
U3 =3+6U2.
and u3 = ,;. Combining these partial answers in ma-
S.
Reducible Markov Chains
2
3
4
5
0
0
0
0
0
0
0
0
(17)(5)
0
(7)(5)
(17)(5)
0
0 X X
0 X X
4
0
0
0
5
0
0
0
0
1
2
3
5
2
5
3
5
s
2
(I7)(5)
3
0
1
limp =
263
0XX
0XX
where X denotes that the limit does not exist. For the time average, we
have
1
2
3
4
5
5
0
0
0
0
5
0
0
0
0
(17)(S)
(17)(5)
0
0
(17)(2)
(17)(2)
(17)(5)
(17)(5)
0
0
(17)(2)
(17)(2)
0
0
0
0
z
2
0
0
0
0
12
2
0
2
2
1'
lim -
I
One possible behavior remains to be illustrated. It can occur only when
there are an infinite number of states. In this case it is possible that all
states are transient or null recurrent and
P;'I = 0 for all states i, j.
For example, consider the deterministic Markov chain described by
X = X + n. The transition probability matrix is
0
1
2
3
0
0
1
0
0
1
0
0
1
0
P=2 0
0
0
1
0
0
0
0
3
IV The Long Run Behavior of Markov Chains
264
Then all states are transient, and lim,,,. P;"' =
0 for all states i, j.
Pr{X = jjX0 = i } _
If there is only a finite number M of states, then there are no null
recurrent states, and not all states can be transient. In fact, since
1;'=o
P;"' = 1 for all n, it cannot happen that lim,,, P;"' = 0 for all j.
Exercises
5.1. Given the transition matrix
0
1
2
3
4
4
4
0
0
0
2
2
0
0
0
0
0
1
0
0
0
0
3
3
0
1
0
0
0
0
determine the limits, as n --4 oo, of P,") for i = 0, 1, ... , 4.
5.2. Given the transition matrix
1
2
3
4
5
6
7
1
3
.3
0
0
0
0
0
2
4
0
0
0
0
0
3
0
0
0
1
0
0
P=4
0
0
1
0
0
0
0
5
0
0
1
0
0
0
0
6
'6
0
=6
=
6
0
4'-
=
4
7
0
0
0
0
0
0
1
derive the following limits, where they exist:
(a)
(e)
(b) lim,,, P ;''
(f) lim ,x P; '
(c)
(g)
(h)
PP;''
(d) lim,,., P3")
Al")
P67'
Pte'
265
problems
Problems
5.1. Describe the limiting behavior of the Markov chain whose transition probability matrix is
Hint:
0
1
2
3
4
5
0
0.1
0.1
0.2
0.2
0.1
0.1
0.1
0.1
1
0
0.1
0.1
0.1
0
0.3
0.2
0.2
2
0.6
0
0
0.1
0.1
0.1
0.1
0.0
3
0
0
0
0.3
0.7
0
0
0
4
0
0
0
0.7
0.3
0
0
0
5
0
0
0
0
0
0.3
0.4
0.3
6
0
0
0
0
0
0.1
0
0.9
7
0
0
0
0
0
0.8
0.2
0
6
7
First consider the matrices
0
1
2
3-4
5-7
0
0.1
0.1
0.2
0.3
0.3
1
0
0.1
0.1
0.1
0.7
PA= 2
0.6
0
0
0.2
0.2
3-4
0
0
0
1
0
5-7
0
0
0
0
1
and
P8
3
4
3
0.3
0.71
4
0.7
0.3
5
6
7
0.3
0.4
0.3
0.1
0
0.9
0.8
0.2
0
PC =
The Long Run Behavior of Markov Chains
IV
266
5.2. Determine the limiting behavior of the Markov chain whose transition probability matrix is
0
1
2
3
4
5
6
7
0
0.1
0.2
0.1
0.1
0.2
0.1
0.1
0.1
1
0
0.1
0.2
0.1
0
0.3
0.1
0.2
2
0.5
0
0
0.2
0.1
0.1
0.1
0
3
0
0
0.3
0.7
0
0
0
0
4
0
0
0.6
0.4
0
0
0
0
5
0
0
0
0
0
0.3
0.4
0.3
6
0
0
0
0
0
0.2
0.2
0.6
7
0
0
0
0
0
0.9
0.1
0
P=
Chapter V
Poisson Processes
1.
The Poisson Distribution and
the Poisson Process
Poisson behavior is so pervasive in natural phenomena and the Poisson
distribution is so amenable to extensive and elaborate analysis as to make
the Poisson process a cornerstone of stochastic modeling.
1.1.
The Poisson Distribution
The Poisson distribution with parameter µ > 0 is given by
fork=0,1.....
Let X be a random variable having the Poisson distribution in (1.1). We
evaluate the mean, or first moment, via
k=0
P = k=
= µe-µ
= µ.
ke-µµk
µ(k-1)
k!
V Poisson Processes
268
To evaluate the variance, it is easier first to determine
E[X(X - 1)] = 7, k(k - 1)pk
k=2
µ(k-2)
(k - 2)!
=
2
Then
E[X2] = E[X(X - 1)] + E[X]
= µ2 + µ,
while
ax 2= Var[X] = E[X2] - {E[X]}2
=µ2+µ-µ2=µ,
Thus the Poisson distribution has the unusual characteristic that both the
mean and the variance are given by the same value µ.
Two fundamental properties of the Poisson distribution, which will
arise later in a variety of forms, concern the sum of independent Poisson
random variables and certain random decompositions of Poisson phenomena. We state these properties formally as Theorems 1.1 and 1.2.
Theorem 1.1
Let X and Y be independent random variables having
Poisson distributions with parameters µ and v, respectively. Then the sum
X + Y has a Poisson distribution with parameter µ + v.
Proof By the law of total probability,
Pr{X=k,Y=n-k}
Pr{X+Y=n}
k=0
_
Pr{X = k} Pr{Y = n - k}
k=0
(X and Y are independent)
1.
The Poisson Distribution and the Poisson Process
{µkk
µ}l(n
269
- k)!}
(1.2)
e-(µ+V)
n!
n!
k=o k!(n - k)!
µkvx-k
The binomial expansion of (µ + v)" is, of course,
?1
k=
n!
µkvn-k
k!(n - k)!
and so (1.2) simplifies to
e cµ+")(! + v)
Pr(X+Y=n)n!
n=0,1,...,
the desired Poisson distribution.
To describe the second result, we consider first a Poisson random variable N where the parameter is µ > 0. Write N as a sum of ones in the form
N ones
and next, considering each one separately and independently, erase it with
probability 1 - p and keep it with probability p. What is the distribution
+ 1?
of the resulting sum M, of the form M = 1 + 0 + 0 + 1 +
The next theorem states and answers the question in a more precise
wording.
Theorem 1.2 Let N be a Poisson random variable with parameter µ,
and conditional on N, let M have a binomial distribution with parameters
N and p. Then the unconditional distribution of M is Poisson with parameter µp.
Proof The verification proceeds via a direct application of the law of
total probability. Then
V Poisson Processes
270
Pr{M = k} = 7 Pr{M = kIN = n) Pr{N = n}
n=0
l µ-{k!(n'1
k)!
e-"(pp)k
pk(1 -
n!
}
[µ(1 (n - k)!
k!
= e-"(µp)k
P)"-kl1
e,O -n>
k!
e "''(µp)k
for k = 0, 1, ... ,
k!
which is the claimed Poisson distribution.
1.2.
The Poisson Process
The Poisson process entails notions of both independence and the Poisson
distribution.
Definition A Poisson process of intensity, or rate, A > 0 is an integervalued stochastic process (X(t); t ? 0) for which
(i) for any time points to = 0 < t, < t2 <
< t,,, the process
increments
X(t,) - X(t0), X(t2) - X(t1)1
. . .
,
are independent random variables;
(ii) for s ? 0 and t > 0, the random variable X(s + t) - X(s):has the
Poisson distribution
Pr{X(s
+ t) - X(s) = k} =
(At)
ki
ke-a,
fork = 0, 1....
(iii) X(0) = 0.
In particular, observe that if X(t) is a Poisson process of rate A > 0, then
the moments are
E[X(t)] = At and
Var[X(t)] = oX(,, = At.
1.
The Poisson Distribution and the Poisson Process
271
Example Defects occur along an undersea cable according to a Poisson
process of rate A = 0.1 per mile. (a) What is the probability that no defects appear in the first two miles of cable? (b) Given that there are no
defects in the first two miles of cable, what is the conditional probability
of no defects between mile points two and three? To answer (a) we observe that X(2) has a Poisson distribution whose parameter is (0.1)(2) =
0.2. Thus Pr{X(2) = 0) = e-02 = 0.8187. In part (b) we use the independence of X(3) - X(2) and X(2) - X(0) = X(2). Thus the conditional probability is the same as the unconditional probability, and
Pr{X(3) - X(2) = 0) = Pr{X(1) = 0) = e-01 = 0.9048.
Example Customers arrive in a certain store according to a Poisson
process of rate A = 4 per hour. Given that the store opens at 9:00 A.M.,
what is the probability that exactly one customer has arrived by 9:30 and
a total of five have arrived by 11:30 A.M.?
Measuring time t in hours from 9:00 A.M., we are asked to determine
Pr{X(;) = 1, X(2 ) = 5). We use the independence of X()
2 - X(;) and X(;)
to reformulate the question thus:
Pr{X(;) = 1, X(2) = 5) = Pr{X(;) = 1, X() - X(Z) = 4)
= r e a(ir)4(2) I r e -4(2[4(2) ]4
)
!l
1!
!l
4!
= (2e-'-)(5;'e-8) = 0.0154965.
1.3.
Nonhomogeneous Processes
The rate A in a Poisson process X(t) is the proportionality constant in the
probability of an event occurring during an arbitrarily small interval. To
explain this more precisely,
u
Pr{X(t + h) - X(t) = 1) _ (Ah)e
1!
_ (Ah)(1 - Ah + ' A2h2 - ... )
= Ah + o(h),
V Poisson Processes
272
where o(h) denotes a general and unspecified remainder term of smaller
order than h.
It is pertinent in many applications to consider rates A = A(t) that vary
with time. Such a process is termed a nonhomogeneous or nonstationary
Poisson process to distinguish it from the stationary, or homogeneous,
process that we primarily consider. If X(t) is a nonhomogeneous Poisson
process with rate A(t), then an increment X(t) - X(s), giving the number
of events in an interval (s, t], has a Poisson distribution with parameter
f,` A(u) du, and increments over disjoint intervals are independent random
variables.
Example Demands on a first aid facility in a certain location occur according to a nonhomogeneous Poisson process having the rate function
for 0 <- t < 1,
for l <- t < 2,
2t
A(t) = 2
4-t
for 2<-t-:c 4,
where t is measured in hours from the opening time of the facility. What is
the probability that two demands occur in the first two hours of operation
and two in the second two hours? Since demands during disjoint intervals
are independent random variables, we can answer the two questions separately. The mean for the first two hours is µ = f o 2t dt + f ; 2dt = 3, and
thus
Pr{X(2) = 2} =
e
2!
= 0.2240.
For the second two hours, µ = J'2 (4 - t) dt = 2, and
Pr{X(4) - X(2) = 2) =
e
2!
= 0.2707.
Let X(t) be a nonhomogeneous Poisson process of rate A(t) > 0 and define A(t) = fo A(u) du. Make a deterministic change in the time scale and
define a new process Y(s) = X(t), where s = A(t). Observe that As =
A(t)At + o(Ot). Then
Pr{Y(s + As) - Y(s) = 1 } = Pr{X(t + At) - X(t) = 1 }
= A(t)At + o(At)
_ As + o(As),
1.
The Poisson Distribution and the Poisson Process
273
so that Y(s) is a homogeneous Poisson process of unit rate. By this means,
questions about nonhomogeneous Poisson processes can be transformed
into corresponding questions about homogeneous processes. For this reason we concentrate our exposition on the latter.
1.4 Cox Processes
Suppose that X(t) is a nonhomogeneous Poisson process, but where the
rate function {,1(t), t > 01 is itself a stochastic process. Such processes
were introduced in 1955 as models for fibrous threads by Sir David Cox,
who called them doubly stochastic Poisson processes. Now they are most
often referred to as Cox processes in honor of their discoverer. Since their
introduction, Cox processes have been used to model a myriad of phenomena, e.g., bursts of rainfall, where the likelihood of rain may vary with
the season; inputs to a queueing system, where the rate of input varies
over time, depending on changing and unmeasured factors; and defects
along a fiber, where the rate and type of defect may change due to varia-
tions in material or manufacture. As these applications suggest, the
process increments over disjoint intervals are, in general, statistically dependent in a Cox process, as contrasted with their postulated independence in a Poisson process.
Let {X(t); t ? 0) be a Poisson process of constant rate A = 1. The very
simplest Cox process, sometimes called a mixed Poisson process, involves choosing a single random variable 0, and then observing the
process X'(t) = X(Ot). Given 0, then X' is, conditionally, a Poisson
process of constant rate A = 0, but 0 is random, and typically, unobservable. If 0 is a continuous random variable with probability density
function f(9), then, upon removing the condition via the law of total probability, we obtain the marginal distribution
x
Pr{X'(t) = k} =
f
(dt)0e
k!
f(O) d9.
(1.3)
0
Problem 1.12 calls for carrying out the integration in (1.3) in the particular instance in which 0 has an exponential density.
VI, Section 7 develops a model for defects along a fiber in which a
Markov chain in continuous time is the random intensity function for a
V Poisson Processes
274
Poisson process. A variety of functionals are evaluated for the resulting
Cox process.
Exercises
1.1. Defects occur along the length of a filament at a rate of A = 2 per
foot.
(a) Calculate the probability that there are no defects in the first foot of
the filament.
(b) Calculate the conditional probability that there are no defects in the
second foot of the filament, given that the first foot contained a single defect.
1.2. Let pk = Pr{X = k} be the probability mass function corresponding
to a Poisson distribution with parameter A. Verify that p0 = exp{-A}, and
that pk may be computed recursively by pk = (A/k)pk_ I.
1.3 Let X and Y be independent Poisson distributed random variables
with parameters a and 0, respectively. Determine the conditional distribution of X, given that N = X + Y = n.
1.4
Customers arrive at a service facility according to a Poisson process
of rate A customer/hour. Let X(t) be the number of customers that have arrived up to time t.
(a) What is Pr{X(t) = k} for k = 0, 1, ... ?
(b) Consider fixed times 0 < s < t. Determine the conditional probability Pr(X(t) = n + kI X(s) = n) and the expected value E[X(t)X(s)].
1.5. Suppose that a random variable X is distributed according to a Poisson distribution with parameter A. The parameter A is itself a random vari-
able, exponentially distributed with density f(x) = 0e-°' for x ? 0. Find
the probability mass function for X.
1.6. Messages arrive at a telegraph office as a Poisson process with
mean rate of 3 messages per hour.
(a) What is the probability that no messages arrive during the morning
hours 8:00 A.M. to noon?
275
Problems
(b) What is the distribution of the time at which the first afternoon message arrives?
1.7. Suppose that customers arrive at a facility according to a Poisson
process having rate A = 2. Let X(t) be the number of customers that have
arrived up to time t. Determine the following probabilities and conditional
probabilities:
(a) Pr{X(1) = 2).
(b) Pr{X(1) = 2 and X(3) = 6}.
(c) Pr{X(1) = 2X(3) = 6).
(d) Pr{X(3) = 6X(1) = 2).
1.8. Let {X(t); t ? 0) be a Poisson process having rate parameter A = 2.
Determine the numerical values to two decimal places for the following
probabilities:
(a) Pr{X(1) s 2}.
(b) Pr{X(1) = 1 and X(2) = 3).
(c) Pr{X(1) >- 21X(1) ? I).
1.9. Let {X(t); t ? 0) be a Poisson process having rate parameter A = 2.
Determine the following expectations:
(a) E[X(2)].
(b) E[{X(1)}2].
(c) E[X(1) X(2)].
Problems
1.1. Let ,, 62.... be independent random variables, each having an exponential distribution with parameter A. Define a new random variable
X as follows: If 6, > 1, then X = 0; if 6, <_ 1 but 6, + 2 > 1, then set
X = 1; in general, set X = k if
1
+ ... +
k
< < 61 + ... + 6k + 61. +
Show that X has a Poisson distribution with parameter A. (Thus the
method outlined can be used to simulate a Poisson distribution.)
V Poisson Processes
276
Hint:
+
6, +
k
has a gamma density
Akxk_I
fk(x) _ (k
Condition on 6, +
+
k
1)ie_x,
-
forx > 0.
and use the law of total probability to show
1
Pr{X = k) =
f [I - F(l - x)] f(x) dx,
0
where F(x) is the exponential distribution function.
1.2. Suppose that minor defects are distributed over the length of a
cable as a Poisson process with rate a, and that, independently, major defects are distributed over the cable according to a Poisson process of rate
/3. Let X(t) be the number of defects, either major or minor, in the cable
up to length t. Argue that X(t) must be a Poisson process of rate a + /3.
1.3. The generating function of a probability mass function pk =
Pr{X = k}, for k = 0, 1, ... , is defined by
x
gx(s) = E[sx] _ 7, pksk
for lsl < 1.
k=0
Show that the generating function for a Poisson random variable X with
mean µ is given by
gx(s) = e µ"")
1.4. (Continuation) Let X and Y be independent random variables,
Poisson distributed with parameters a and /3, respectively. Show that the
generating function of their sum N = X + Y is given by
9N(s) = e (a+)3ui
s>
Hint: Verify and use the fact that the generating function of a sum of
independent random variables is the product of their respective generating functions. See III, 9.2.
1.5. For each value of h > 0, let X(h) have a Poisson distribution with
parameter Ah. Let pk(h) = Pr{X(h) = k} for k = 0, 1, .... Verify that
277
Problems
urn
1
- po(h) = A,
or po(h) = 1 - Ah + o(h);
h
h-,o
li m
p'(h) =
d->o
h
li m p-(h)
ii--*o
h
or p,( h ) = Ah + o( h);
A,
= 0,
or p2( h )
= o(h ).
Here o(h) stands for any remainder term of order less than h as h - 0.
1.6. Let (X(t); t ? 0) be a Poisson process of rate A. For s, t > 0, determine the conditional distribution of X(t), given that X(t + s) = n.
1.7 Shocks occur to a system according to a Poisson process of rate A.
Suppose that the system survives each shock with probability a, independently of other shocks, so that its probability of surviving k shocks is a'.
What is the probability that the system is surviving at time t?
1.8.
Find the probability Pr(X(t) = 1, 3, 5, ... } that a Poisson process
having rate A is odd.
1.9. Arrivals of passengers at a bus stop form a Poisson process X(t)
with rate A = 2 per unit time. Assume that a bus departed at time t = 0
leaving no customers behind. Let T denote the arrival time of the next bus.
Then the number of passengers present when it arrives is X(T). Suppose
that the bus arrival time T is independent of the Poisson process and that
T has the uniform probability density function
1
fr(t) = I0
for0-- t-< 1,
elsewhere
(a) Determine the conditional moments E[X(T)T =
t]
and
E[{X(T)}2 T = t].
(b) Determine the mean E[X(T)] and variance Var[X(T)].
1.10. Customers arrive at a facility at random according to a Poisson
process of rate A. There is a waiting time cost of c per customer per unit
V Poisson Processes
278
time. The customers gather at the facility and are processed or dispatched
in groups at fixed times T, 2T, 3T,
There is a dispatch cost of K. The
process is depicted in the following graph.
....
w
0
E
T
0
2T
E
Time
Figure 1.1 The number of customers in a dispatching system as a
function of time.
(a) What is the total dispatch cost during the first cycle from time 0 to
time Ti
(b) What is the mean total customer waiting cost during the first cycle?
(c) What is the mean total customer waiting + dispatch cost per unit
time during the first cycle?
(d) What value of T minimizes this mean cost per unit time?
1.11. Assume that a device fails when a cumulative effect of k shocks
occurs. If the shocks happen according to a Poisson process of parameter
A, what is the density function for the life T of the device?
1.12. Consider the mixed Poisson process of Section 1.4, and suppose
that the mixing parameter 0 has the exponential density f(O) = e-0 for
0>0.
(a) Show that equation (1.3) becomes
Pr{X'(t)=j}=\1+t1J\l+t),
forj=0,1,....
2.
The Law of Rare Events
279
(b) Show that
Pr{X'(t)=j,X'(t+s)=j+k} =I
+k
.
)t's"(l
I
1;+k+i
+s+t/
so that X'(t) and the increment X'(t + s) - X'(t) are not independent random variables, in contrast to the simple Poisson process as defined in Section 1.2.
2.
The Law of Rare Events
The common occurrence of the Poisson distribution in nature is explained
by the law of rare events. Informally, this law asserts that where a certain
event may occur in any of a large number of possibilities, but where the
probability that the event does occur in any given possibility is small, then
the total number of events that do happen should follow, approximately,
the Poisson distribution.
A more formal statement in a particular instance follows. Consider a
large number N of independent Bernoulli trials where the probability p of
success on each trial is small and constant from trial to trial. Let X,,,,, denote the total number of successes in the N trials, where X,,.4, follows the
binomial distribution
Pr{X1,.,, = k} =
k!(N
- k)! p'(1 - p)'-"
for k = 0,
... , N.
(2.1)
Now let us consider the limiting case in which N - x and p -4 0 in
such a way that Np = µ > 0 where p is constant. It is a familiar fact (see
I, Section 3) that the distribution for X,,.,, becomes, in the limit, the Poisson distribution
e-'`µ1.
for k = 0, 1,
Pr{X,A = k} =
....
(2.2)
k!
This form of the law of rare events is stated as a limit. In stochastic
modeling, the law is used to suggest circumstances under which one
might expect the Poisson distribution to prevail, at least approximately.
For example, a large number of cars may pass through a given stretch of
highway on any particular day. The probability that any specified car is in
an accident is, we hope, small. Therefore, one might expect that the actual
V Poisson Processes
280
number of accidents on a given day along that stretch of highway would
be, at least approximately, Poisson distributed.
While we have formally stated this form of the law of rare events as a
mathematical limit, in older texts, (2.2) is often called "the Poisson approximation" to the binomial, the idea being that when N is large and p is
small, the binomial probability (2.1) may be approximately evaluated by
the Poisson probability (2.2) with .t = Np. With today's computing
power, exact binomial probabilities are not difficult to obtain, so there is
little need to approximate them with Poisson probabilities. Such is not the
case if the problem is altered slightly by allowing the probability of
success to vary from trial to trial. To examine this proposal in detail, let
E,, EZ, ... be independent Bernoulli random variables, where
Pr {E; = 1) = p;
Pr {E; = 0) = I - pi,
and
and let S = E, +
+ E,,. When p, = P2 _
= p, then S,, has a binomial distribution, and the probability Pr{S = k} for some k = 0, 1.... is
easily computed. It is not so easily evaluated when the p's are unequal, the
binomial formula generalizing to
p)'(2.3)
!i
Pr{S = k} = I(k'
where I'"' denotes the sum over all 0, 1 valued x;'s such that x, +
+
x,, = k.
Fortunately, Poisson approximation may still prove accurate and allow
the computational challenges presented by equation (2.3) to be avoided.
Theorem 2.1
Let E, e2.... be independent Bernoulli random vari-
ables, where
Pr{E; = 1} = p,
and let S,, = E, +
Pr{E; = 0} = I - p;,
and
+ E,,. The exact probabilities for S,,, determined using
(2.3), and Poisson probabilities with µ = p, +
µ
+ p differ by at most
Ae-
Pr (S,, = k} -
(2.4)
k!
Not only does the inequality of Theorem 2.1 extend the law of rare events
to the case of unequal p's, it also directly confronts the approximation
2.
The Law of Rare Events
281
issue by providing a numerical measure of the approximation error. Thus
the Poisson distribution provides a good approximation to the exact probabilities whenever the pi's are uniformly small as measured by the right
side of (2.4). For instance, when p, = P2 = ... = µ/n, then the right side
of (2.4) reduces to µ2/n, which is small when n is large, and thus (2.4) provides another means of obtaining the Poisson distribution (2.2) as a limit
of the binomial probabilities (2.1).
We defer the proof of Theorem 2.1 to the end of this section, choosing
to concentrate now on its implications. As an immediate consequence, for
example, in the context of the earlier car accident vignette, we see that the
individual cars need not all have the same accident probabilities in order
for the Poisson approximation to apply.
2.1
The Law of Rare Events and the Poisson Process
Consider events occurring along the positive axis [0, oc) in the manner
shown in Figure 2.1. Concrete examples of such processes are the time
points of the x-ray emissions of a substance undergoing radioactive decay,
the instances of telephone calls originating in a given locality, the occurrence of accidents at a certain intersection, the location of faults or defects
along the length of a fiber or filament, and the successive arrival times of
customers for service.
Let N((a, b]) denote the number of events that occur during the interval
(a, b]. That is, if t, < t,-< t3 <
denote the times (or locations, etc.) of
N((a, b]) = 3
ti
t,
a
Figure 2.1
b
A Poisson point process.
t
V Poisson Processes
282
successive events, then N((a, b]) is the number of values t; for which
a<t,
b.
We make the following postulates:
(1) The numbers of events happening in disjoint intervals are independent random variables. That is, for every integer m = 2, 3, ... and
< t,,,, the random variables
time points t0 = 0 < t, < t, <
N((to, ti]), N((t t,]), ... , N((t,,,-,, t])
are independent.
(2) For any time t and positive number h, the probability distribution
of N((t, t + h]), the number of events occurring between time t and
t + h, depends only on the interval length h and not on the time t.
(3) There is a positive constant A for which the probability of at least
one event happening in a time interval of length h is
Pr{N((t,t+ h]) ? 1) = Ah + o(h)
ash
0.
(Conforming to a common notation, here o(h) as h .L 0 stands for a
general and unspecified remainder term for which o(h)/h - 0 as
h 10. That is, a remainder term of smaller order than h as h vanishes.)
(4) The probabilty of two or more events occurring in an interval of
length h is o(h), or
Pr{N((t, t + h]) ? 2} = o(h),
h 10.
Postulate 3 is a specific formulation of the notion that events are rare.
Postulate 4 is tantamount to excluding the possibility of the simultaneous
occurrence of two or more events. In the presence of Postulates 1 and 2,
Postulates 3 and 4 are equivalent to the apparently weaker assumption that
events occur singly and discretely, with only a finite number in any finite
interval. In the concrete illustrations cited earlier, this requirement is usually satisfied.
Disjoint intervals are independent by 1, and 2 asserts that the distribution of N((s, t]) is the same as that of N((0, t - s]). Therefore, to describe
the probability law of the system, it suffices to determine the probability
distribution of N((0, t]) for an arbitrary value of t. Let
Pk(t) = Pr{N((0, t]) = k}.
2.
The Law of Rare Events
283
We will show that Postulates 1 through 4 require that Pk(t) be the Poisson
distribution
Pk(t) =
(At)ke-Ar
k!
for k = 0, 1,
....
(2.5)
To establish (2.5), divide the interval (0, t] into n subintervals of equal
length h = t/n, and let
11
0
if there is at least one event in the interval ((i - 1)t/n, it/n],
otherwise.
+ E counts the total number of subintervals that conThen S,, = E, +
tain at least one event, and
p, = Pr{E; = 1) = Atln + o(tin)
(2.6)
according to Postulate 3. Upon applying (2.4), we see that
We
Pr(S,, = k} - µ
µ
<_ n[At/n + o(tln)]2
I
k!
(n
+211to(n) +no(nl,
n
/
where
'I
µ=
p, = At + no(t/n).
(2.7)
Because o(h) = o(t/n) is a term of order smaller than h = tin for large n,
it follows that
n o(tln) = t
o(tln)
tin
= t o(h)
h
vanishes for arbitrarily large n. Passing to the limit as n - oo, then, we deduce that
µkelim Pr{S = k} =
,
k!
with µ = At.
To complete the demonstration, we need only show that
lim Pr{S = k} = Pr{N((0, t]) = k} = Pk(t).
x
V Poisson Processes
284
But S,, differs from N((0, t]) only if at least one of the subintervals contains two or more events, and Postulate 4 precludes this because
IPk(t)
- Pr{S =
k}l
Pr{N((0, t])
r=i
Pr(N(((i - 1)t it11
n
nil
tl l
<_ n o(t/n)
2l
J
(by Postulate 4)
as n - co.
-3 0
By taking n arbitrarily large, or equivalently, by dividing the interval
(0, t] into arbitrarily small subintervals, we see that it must be the case that
(At) xe Al
Pr{N((0, t]) = k) =
ki
fork ? 0,
and Postulates 1-4 imply the Poisson distribution.
Postulates 1 through 4 arise as physically plausible assumptions in
many circumstances of stochastic modeling. The postulates seem rather
weak. Surprisingly, they are sufficiently strong to force the Poisson behavior just derived. This motivates the following definition.
Definition Let N((s, t]) be a random variable counting the number of
events occurring in an interval (s, t]. Then N((s, t]) is a Poisson point
process of intensity A > 0 if
(i) for every m = 2, 3.... and distinct time points to = 0 < t, < t2 <
< t,,,, the random variables
N((to, t.l ), N((t,, t,]), ... ,
t,,,])
are independent; and
(ii) for any times s < t the random variable N((s, t]) has the Poisson
distribution
Pr{N((s, t]) = k} =
Wt - s)]ke a(Is)
ki
k = 0,1,....
Poisson point processes often arise in a form where the time parameter
is replaced by a suitable spatial parameter. The following formal example
illustrates this vein of ideas. Consider an array of points distributed in a
The Law of Rare Events
2.
285
space E (E is a Euclidean space of dimension d ? 1). Let N(A) denote the
number of points (finite or infinite) contained in the region A of E. We postulate that N(A) is a random variable. The collection {N(A)} of random
variables, where A varies over all possible subsets of E, is said to be a homogeneous Poisson process if the following assumptions are fulfilled:
(i) The numbers of points in nonoverlapping regions are independent
random variables.
(ii) For any region A of finite volume, N(A) is Poisson distributed with
mean AIAI, where IAI is the volume of A. The parameter A is fixed
and measures in a sense the intensity component of the distribution,
which is independent of the size or shape. Spatial Poisson
processes arise in considering such phenomena as the distribution
of stars or galaxies in space, the spatial distribution of plants and
animals, and the spatial distribution of bacteria on a slide. These
ideas and concepts will be further studied in Section 5.
2.2
Proof of Theorem 2.1
First, some notation. Let e(p) denote a Bernoulli random variable with
success probability p, and let X(O) be a Poisson distributed random variable with parameter 0. We are given probabilities p ... , p,, and let
µ = p, +
+ p,,. With
( p 1 ),
.
. . ,
e(p,,) assumed to be independent, we
have S,, = e(p,) +
+ e(,,), and according to Theorem 1.1, we may
write X(µ) as the sum of independent Poisson distributed random variables in the form X(µ) = X(p,) +
+ X(,,). We are asked to compare
Pr{S = k} with Pr{X(µ) = k}, and, as a first step, we observe that if S,,
and X(µ) are unequal, then at least one of the pairs e(pk) and X(pk) must
differ, whence
'I
lPr{S = k) - Pr{X(µ) = k}I <
Pr{e(pJ
X(pA)).
(2.8)
k=1
As the second step, observe that the quantities that are compared on the
left of (2.8) are the marginal distributions of S and X(µ), while the bound
on the right is a joint probability. This leaves us free to choose the joint
distribution that makes our task the easiest. That is, we are free to specify
the joint distribution of each e(pp) and X(pk), as we please, provided only
that the marginal distributions are Bernoulli and Poisson, respectively.
V Poisson Processes
286
To complete the proof, we need to show that Pr{e(p) 0 X(p) } <_ p2 for
some Bernoulli random variable E(p) and Poisson random variable X(p),
since this reduces the right side of (2.8) to that of (2.4). Equivalently, we
want to show that 1 - p2 S Pr{E(p) = X(p) } = Pr{E(p) = X(p) = 01 +
Pr{E(p) = X(p) = I), and we are free to choose the joint distribution, provided that the marginal distributions are correct.
Let U be a random variable that is uniformly distributed over the interval (0, 1]. Define
if0<U5p,
1
E(p) =
0
and for k = 0, l , ... , set
k-1
X(p) = k
'e-v
p'e-''
<U<_
when
r=0
it
i=0
i!
It is elementary to verify that e(p) and X(p) have the correct marginal dis-
tributions. Furthermore, because 1 - p < e", we have E(p) = X(p) = 0
only for U:5 1 - p, whence Pr{E(p) = X(p) = 0) = 1 - p. Similarly,
E(p) = X(p) = 1 only when e -P < U <- (1 + p)e "', whence Pr{E(p) _
X(p) = I) = pe ''. Upon summing these two evaluations, we obtain
-p2
Pr {E(p)=X(p)} = 1 - p+peP= 1
as was to be shown. This completes the proof of (2.4).
Problem 2.10 calls for the reader to review the proof and to discover the
single line that needs to be changed in order to establish the stronger result
/1
IPr{S in I) - Pr(X(µ) in I) I
Y
k=1
for any set of nonnegative integers I.
Exercises
2.1. Determine numerical values to three decimal places for Pr{X = k},
k = 0, 1, 2, when
(a) X has a binomial distribution with parameters n = 20 and p = 0.06.
(b) X has a binomial distribution with parameters n = 40 and p = 0.03.
(c) X has a Poisson distribution with parameter A = 1.2.
287
problems
2.2. Explain in general terms why it might be plausible to assume that
the following random variables follow a Poisson distribution:
(a) The number of customers that enter a store in a fixed time period.
(b) The number of customers that enter a store and buy something in a
fixed time period.
(c) The number of atomic particles in a radioactive mass that disintegrate in a fixed time period.
2.3. A large number of distinct pairs of socks are in a drawer, all mixed
up. A small number of individual socks are removed. Explain in general
terms why it might be plausible to assume that the number of pairs among
the socks removed might follow a Poisson distribution.
2.4. Suppose that a book of 600 pages contains a total of 240 typographical errors. Develop a Poisson approximation for the probability that three
particular successive pages are error-free.
Problems
2.1. Let X(n, p) have a binomial distribution with parameters n and p.
Let n --> x and p - 0 in such a way that np = A. Show that
lim Pr{ X(n, p) = 01 = e'
and
lim
,I---
Pr{X(n, p) = k + I)
Pr {X (n, p) = k}
A
k+1
for k = 0, 1,
... .
2.2. Suppose that 100 tags, numbered 1 , 2, ... , 100, are placed into an
urn, and 10 tags are drawn successively, with replacement. Let A be the
event that no tag is drawn twice. Show that
Pr(A) = (1 -
ioo)
.
(1 - iw) = 0.6282.
Use the approximation
1 -x-e-`
forx-0
V Poisson Processes
288
to get
Pr{A} - exp{-100 (1 + 2 +
+ 9)} = e°-45 = 0.6376.
Interpret this in terms of the law of rare events.
2.3. Suppose that N pairs of socks are sent to a laundry, where they are
washed and thoroughly mixed up to create a mass of unmatched socks.
Then n socks are drawn at random without replacement from the pile. Let
A be the event that no pair is among the n socks so selected. Show that
2"(n)
Pr{A} =
(2N)
-
; (1
2N-il
n
Use the approximation
1
forx- 0
-x-e-.Y
to get
Pr{A} = exp
i
_
2N-i}
exp{
n(n-1)
4N
'
the approximations holding when n is small relative to N, which is large.
Evaluate the exact expression and each approximation when N = 100 and
n = 10. Is the approximation here consistent with the actual number of
pairs of socks among the n socks drawn having a Poisson distribution?
Answer: Exact 0.7895; Approximate 0.7985.
2.4. Suppose that N points are uniformly distributed over the interval
[0, N). Determine the probability distribution for the number of points in
the interval [0, 1) as N - c.
2.5. Suppose that N points are uniformly distributed over the surface of
a circular disk of radius r. Determine the probability distribution for the
number of points within a distance of one of the origin as N -* co, r - co,
N/(7rr2)_A.
2.6. Certain computer coding systems use randomization to assign
memory storage locations to account numbers. Suppose that N = MA dif-
289
Problems
ferent accounts are to be randomly located among M storage locations.
Let X; be the number of accounts assigned to the ith location. If the accounts are distributed independently and each location is equally likely to
be chosen, show that Pr{X; = k} - e-"Ak/k! as N - oo. Show that X; and
X; are independent random variables in the limit, for distinct locations
i j. In the limit, what fraction of storage locations have two or more accounts assigned to them?
2.7. N bacteria are spread independently with uniform distribution on a
microscope slide of area A. An arbitrary region having area a is selected for
observation. Determine the probability of k bacteria within the region of
oc and a -3 0 such that (alA)N -3 c (0 < c < oo),
area a. Show that as N
then p(k) -3 e-`c'lk!.
2.8. Using (2.3), evaluate the exact probabilities for S,, and the Poisson
approximation and error bound in (2.4) when n = 4 and p, = 0.1, p2 = 0.2,
p., = 0.3, and p4= 0.4.
2.9. Using (2.3), evaluate the exact probabilities for S,, and the Poisson
approximation and error bound in (2.4) when n = 4 and p, = 0.1, P2 = 0.1,
p3=0.1,andp4=0.2.
2.10.
Review the proof of Theorem 2.1 in Section 2.2 and establish the
stronger result
Pr{S in 1 } - Pr{X(µ) in I }
k=1
for any set of nonnegative integers I.
2.11. Let X and Y be jointly distributed random variables and B an
arbitrary set. Fill in the details that justify the inequality JPr{X in B} Pr{Yin B}1
Hint:
Pr [X * Y}.
Begin with
{XinB} = {XinBandYinB)
C(Yin B}
or
or
{X/Y}.
{XinBand YnotinB}
290
V Poisson Processes
2.12. Computer Challenge Most computers have available a routine
for simulating a sequence U,,, U . . . of independent random variables,
each uniformly distributed on the interval (0, 1). Plot, say, 10,000 pairs
on the unit square. Does the plot look like what you would expect? Repeat the experiment several times. Do the points in a fixed number of disjoint squares of area 1/10,000 look like independent unit Poisson random variables?
'3. Distributions Associated with the Poisson Process
A Poisson point process N((s, t]) counts the number of events occurring
in an interval (s, t]. A Poisson counting process, or more simply a Poisson
process X(t), counts the number of events occurring up to time t. Formally,
X(t) = N((0, t]).
Poisson events occurring in space can best be modeled as a point
process. For Poisson events occurring on the positive time axis, whether
we view them as a Poisson point process or Poisson counting process is
largely a matter of convenience, and we will freely do both. The two descriptions are equivalent for Poisson events occurring along a line. The
Poisson process is the more common and traditional description in this
case because it allows a pictorial representation as an increasing integervalued random function taking unit steps.
Figure 3.1 shows a typical sample path of a Poisson process where W is
the time of occurrence of the nth event, the so-called waiting time. It is often
convenient to set W, = 0. The differences S,, = W,,,, - W are called sojourn
times; S,, measures the duration that the Poisson process sojourns in state n.
X(t)
3
2
I
J
1
01
H- S o
Figure 3.1
4-
11
Wo
WI
IS
W2
i
Wv
Sz
+{
A typical sample path of a Poisson process showing the waiting
times W and the sojourn times S,,.
3.
Distributions Associated with the Poisson Process
291
In this section we will determine a number of probability distributions
associated with the Poisson process X(t), the waiting times W,,, and the sojourn times S.
The waiting time W has the gamma distribution whose
probability density function is
Theorem 3.1
1
(n -
n = 1,2,...,t?0.
1)!e
(3.1)
In particular, W,, the time to the first event, is exponentially distributed:
t ? 0.
Ae ",
(3.2)
Proof The event W,, < t occurs if and only if there are at least n events
in the interval (0, t], and since the number of events in (0, t] has a Poisson
distribution with mean At we obtain the cumulative distribution function
of W via
F,,(t) = Pr{W : t} = Pr{X(t) ? n}
(At)ke"t
&k
=1-
n= 1,2,...,t?0.
k!
k=O
We obtain the probability density function
mulative distribution function. Then
by differentiating the cu-
d
d
dt
I
e
1a--+
At
(At)'-
1!
2!
El
"
+Ae
A+
A(At)
A(At)'-
+
2!
1!
"
(n - 1)!
11+
-+
e"`,
+...+
+
+A
(At)"-11
(n - 1)! Jr
(At)"-2
(n - 2)!
At
(At)2
(At)"-1
1!
2!
(n - 1)!
n = 1, 2, .
. .
,t
?0.
1
J
V Poisson Processes
292
There is an alternative derivation of the density in (3.1) that uses the
Poisson point process N((s, t]) and proceeds directly without differentiation. The event t < W t + At corresponds exactly to n - 1 occurrences
in (0, t] and one in (t, t + At], as depicted in Figure 3.2.
N((t,t+At])= I
N((O,t])=n- 1
t
t+At
W.
Figure 3.2
Then
Pr(t < W,,:!:-: t + At) + o(At)
[see I, (2.5)]
= Pr{N((0, t]) = n - 1) Pr{N((t, t + At]) = 1) + o(At)
(,kt),-yea,
A(At) + o(Ot).
(n - 1)!
Dividing by At and passing to the limit as At 0 we obtain (3.2).
Observe that Pr{N((t, t + At]) ? 1 } = Pr{N((t, t + At]) = 1 } + o(At) _
A(At) + o(Ot).
Theorem 3.2 The sojourn times So, S ... ,
are independent random variables, each having the exponential probability density function
A(s) _ Ae .',
s ? 0.
(3.3)
Proof We are being asked to show that the joint probability density
function of So, S ... ,
is the product of the exponential densities
given by
(Ae-x,,,_,).
si, ... ,
(3.4)
We give the proof only in the case n = 2, the general case being entirely
similar. Referring to Figure 3.3 we see that the joint occurrence of
s, < S, < s, + As,
and
s, < S, < s, + Os2
3.
Distributions Associated with the Poisson Process
293
<---- S,
.0--- S1
F-- S1
- I s1 I<
S2 -- LS?
Figure 3.3
corresponds to no events in the intervals (0, s,] and (s, + As,, s, + As, + s,]
and exactly one event in each of the intervals (s s, + Os,] and (s, + As, +
s s, + As, + s, + Os,]. Thus
fs,.s,(S,, s,) As,
As, = Pr{s, < S, < s, + As,, s, < S, < s, + As,)
+ O(1s, Os,)
= Pr{N((0, s,]) = 0)
X Pr{N((s1 + As,, s, + As, + s,]) = 0)
X Pr{N((s1, s, + Os,]) = 1)
X Pr {N((s, + As, + s,, s, + As, + s, + Os,]) = 1 }
+ o(Ls, Os,)
=
e-as,e-x.,,e-as.,,e A,,,,k(Os1)A(Os,) + o(Os, Os,)
= (Ae-A8)(Ae-A',)(Os1)(Os,) + o(Os, Os,).
Upon dividing both sides by (As1)(As,) and passing to the limit as As, ---> 0
and As, - 0, we obtain (3.4) in the case n = 2.
The binomial distribution also arises in the context of Poisson
processes.
Theorem 3.3 Let {X(t)} be a Poisson process of rate A > 0. Then for
0<u<tand 0:k--:: n,
Pr{X(u) = k X(t) = n) =
n!
(ulkr
k!(n - k)! t 111
u -k
t)
.
(3.5)
V Poisson Processes
294
Proof Straightforward computations give
Pr{X(u) = klX(t) = n} =
Pr{X(u) = k and X(t) = n }
Pr{X(t) = n)
Pr{X(u) = k and X(t) - X(u) = n - k}
Pr{X(t) = n}
{e-""(Au)Vk! } { e x(r_"'[A(t - u)]"-"/(n - k)!)
e "'(At)"/n!
n!
u"(t -
k! (n - k)!
which establishes (3.5).
Exercises
3.1.
A radioactive source emits particles according to a Poisson process
of rate A = 2 particles per minute. What is the probability that the first particle appears after three minutes?
3.2.
A radioactive source emits particles according to a Poisson process
of rate A = 2 particles per minute.
(a) What is the probability that the first particle appears some time
after three minutes but before five minutes?
(b) What is the probability that exactly one particle is emitted in the interval from three to five minutes?
3.3. Customers enter a store according to a Poisson process of rate A = 6
per hour. Suppose it is known that but a single customer entered during
the first hour. What is the conditional probability that this person entered
during the first fifteen minutes?
3.4. Let X(t) be a Poisson process of rate 6 = 3 per hour. Find the conditional probability that there were two events in the first hour, given that
there were five events in the first three hours.
295
Problems
3.5. Let X(t) be a Poisson process of rate 6 per hour. Find the conditional
probability that there were m events in the first t hours, given that there
were n events in the first T hours. Assume 0 s m <_ n and 0 < t < T.
3.6. For i = 1, . . ., n, let {X;(t); t ? 0} be independent Poisson
processes, each with the same parameter A. Find the distribution of the
first time that at least one event has occurred in every process.
3.7. Customers arrive at a service facility according to a Poisson
process of rate A customers/hour. Let X(t) be the number of customers that
have arrived up to time t. Let W W, . . . be the successive arrival times
of the customers. Determine the conditional mean E[WSIX(t) = 3].
3.8. Customers arrive at a service facility according to a Poisson
process of rate A = 5 per hour. Given that 12 customers arrived during the
first two hours of service, what is the conditional probability that 5 customers arrived during the first hour?
3.9. Let X(t) be a Poisson process of rate A. Determine the cumulative
distribution function of the gamma density as a sum of Poisson probabilities by first verifying and then using the identity W,.
t if and only if
X(t) ? r.
Problems
3.1.
Let X(t) be a Poisson process of rate A. Validate the identity
{W,>W,W,>w,}
if and only if
{X(w,) = 0, X(w,) - X(w,) = 0 or 1).
Use this to determine the joint upper tail probability
Pr{W, > w, W, > w,} = Pr{X(w1) = 0, X(w,) - X(w,) = 0 or 1 }
=
A(w, - w )]e-Ao,,
Finally, differentiate twice to obtain the joint density function
f(w w,) = A2 exp{-Aw,}
for 0 < w, < w,.
296
V Poisson Processes
3.2. The joint probability density function for the waiting times W, and
W, is given by
f(w w,) = A2 exp{-,1w,}
for 0 < w, < w,.
Determine the conditional probability density function for W given that
W, = w,. How does this result differ from that in Theorem 3.3 when
n = 2 and k = 1?
3.3. The joint probability density function for the waiting times W, and
W, is given by
f(w,, w,) = A2 exp{-Aw,}
for 0 < w, < w,.
Change variables according to
So=W,
and S,=W-W,
and determine the joint distribution of the first two sojourn times. Compare with Theorem 3.2.
3.4. The joint probability density function for the waiting times W, and
W, is given by
f(w W,) = A2 exp{-Aw,}
for 0 < w, < w,.
Determine the marginal density functions for W, and W2, and check your
work by comparison with Theorem 3.1.
3.5. Let X(t) be a Poisson process with parameter A. Independently, let
T be a random variable with the exponential density
fT(t) = Oe
for t > 0.
Determine the probability mass function for X(T).
Hint: Use the law of total probability and I, (6.4). Alternatively, use
the results of I, Section 5.2.
3.6.
Customers arrive at a holding facility at random according to a
Poisson process having rate A. The facility processes in batches of size Q.
That is, the first Q - 1 customers wait until the arrival of the Qth customer. Then all are passed simultaneously, and the process repeats. Ser-
4.
The Uniform Distribution and Poisson Processes
297
vice times are instantaneous. Let N(t) be the number of customers in the
holding facility at time t. Assume that N(0) = 0 and let T = min (t ? 0:
N(t) = Q} be the first dispatch time. Show that E[T] = Q/A and E[ f N(t)
dt] = [1 + 2 +
+ (Q - 1)]/A = Q(Q - 1)/2A.
o
A critical component on a submarine has an operating lifetime that
is exponentially distributed with mean 0.50 years. As soon as a component
fails, it is replaced by a new one having statistically identical properties.
3.7.
What is the smallest number of spare components that the submarine
should stock if it is leaving for a one-year tour and wishes the probability
of having an inoperable unit caused by failures exceeding the spare inventory to be less than 0.02?
3.8. Consider a Poisson process with parameter A. Given that X(t) = n
events occur in time t, find the density function for W the time of occurrence of the rth event. Assume that r I n.
3.9. The following calculations arise in certain highly simplified models of learning processes. Let X,(t) and X2(t) be independent Poisson
processes having parameters A, and k,, respectively.
(a) What is the probability that XI(t) = 1 before X2(t) = 1?
(b) What is the probability that X,(t) = 2 before X,(t) = 2?
3.10. Let (W,} be the sequence of waiting times in a Poisson process
defines a nonnegative
of intensity A = 1. Show that X = 2"
martingale.
4.
The Uniform Distribution and Poisson Processes
The major result of this section, Theorem 4.1, provides an important tool
for computing certain functionals on a Poisson process. It asserts that,
conditioned on a fixed total number of events in an interval, the locations
of those events are uniformly distributed in a certain way.
After a complete discussion of the theorem and its proof, its application
in a wide range of problems will be given.
In order to completely understand the theorem, consider first the following experiment. We begin with a line segment t units long and a fixed
V Poisson Processes
298
number n of darts and throw darts at the line segment in such a way that
each dart's position upon landing is uniformly distributed along the segment, independent of the location of the other darts. Let U, be the position
of the first dart thrown, U, the position of the second, and so on up to U,,.
The probability density function is the uniform density
.fU(u) =
1
for 0 :5 u -< t,
0
elsewhere.
t
< W denote these same positions, not in the order
Now let W, : W, <
in which the darts were thrown, but instead in the order in which they
appear along the line. Figure 4.1 depicts a typical relation between
U, U,,..., U and W,,W,..... W,,.
The joint probability density function for W, W..... W is
f,......1 (w......
for0<w, <w,<... < w,, <t.
w,,)=n!t-"
(4.1)
For example, to establish (4.1) in the case n = 2 we have
.fV,.w,(w,, w,) Ow, Ow,
=Pr{w, < W, <w, +Awl, w,< W,<_w,+sw,}
=Pr{w,< U,:w,+Awl, w,<U,<w,+Ow,}
+Pr{w, < U,<_w, +Ow,,w,< U, mow,+Ow,}
w,
=2(
t X
w)=2t
,
,
-iw,Sw,.
t
Dividing by Awl IXw, and passing to the limit gives (4.1). When n = 2,
there are two ways that U, and U, can be ordered; either U, is less. than U,,
or U, is less than U,. In general there are n! arrangements of U,, ... , U
that lead to the same ordered values W, 5 . :!5 W,,, thus giving (4.1).
1
0
Figure 4.1
1U,
W,
U
U,
W,
W,
U,
W
W, W,, ..., W,, are the values U, U,,
increasing order.
.i
U-,
W
t
..., U arranged in
4.
The Uniform Distribution and Poisson Processes
299
Let W, W,, . . . be the occurrence times in a Poisson
process of rate A > 0. Conditioned on N(t) = n, the random variables
W, W,, ... , W have the joint probability density function
Theorem 4.1
w) =
fV,.....w
for O < w, < ... < w,, C t.
n!t-n
(4.2)
Proof The event w; < W < w; + Ow; for i = 1, ... , n and N(t) = n
corresponds to no events occurring in any of the intervals (0, w,], (w, +
(w + Ow,,, t], and exactly one event in
Awl, w,], . . . , (w,, +
each of the intervals (w1, w, + Ow,], (w,, W, + pw,], . . . , (w,,, w +
These intervals are disjoint, and
Pr{N((0, w,]) = 0,
= e A1[eACa
N((w,, + Awl,, t]) = 0}
. . . ,
]
= e A`[1 + o(max{Aw;})],
while
Pr{N((w1, w, + Awl]) = 1, .
,
. .
= A(iw) . . .
N((w,,, w,, +
1}
o(max{Ow;})].
Thus
fw,..... 14, X(,)= (w,,
... ,
Awl?
=Pr{w,<W,<w,+Ow......
o(Ow, ..
Awl,)
Pr{w;<W:5 w;+Iw;,i= 1,...,n,N(t)=n}
Pr{N(t) = n}
+ o(Ow, ... Awl,)
e-Ark(Owl) ...
eA'(At)"/n!
[1 + o(max{Aw;})]
= n!t-"(iw,)
Dividing both sides by (Ow,) ...
establishes (4.2).
o(max{Aw;})].
and letting Awl -4 0, ..., Aw -4 0
300
V Poisson Processes
Theorem 4.1 has important applications in evaluating certain symmetric functionals on Poisson processes. Some sample instances follow.
Example Customers arrive at a facility according to a Poisson process
of rate A. Each customer pays $1 on arrival, and it is desired to evaluate
the expected value of the total sum collected during the interval (0, t] discounted back to time 0. This quantity is given by
Er
X(t)
M=
I ep`yk],
... are the arrival times, and X(t) is
the total number of arrivals in (0, t]. The process is shown in Figure 4.2.
where f3 is the discount rate, W,, WZ,
$1
e-awi
e-sW2
W,
0
Wz
W.
Figure 4.2 A dollar received at time WR. is discounted to a present value at
time 0 of
We evaluate the mean total discounted sum M by conditioning on
X(t) = n. Then
x
r
M = I EI > e l`iJX(t) = nJ Pr{X(t) = n}.
n=1
(4.3)
k=1
Let U,, ... , U denote independent random variables that are uniformly
distributed in (0, t]. Because of the symmetry of the functional
1% exp{-/3W} and Theorem 4.1, we have
,
4.
The Uniform Distribution and Poisson Processes
r
301
,r
ELI
=E
eQuA
k=1
k=1
= nE[e-, ""]
Substitution into (4.3) then gives
M=
-
t[1 -
lit [1
nPr{X(t)=n}
- eI`]E[X(t)l
Example Viewing a fixed mass of a certain radioactive material, suppose that alpha particles appear in time according to a Poisson process of
intensity A. Each particle exists for a random duration and is then annihilated. Suppose that the successive lifetimes Y,, Y,, . . . of distinct particles
are independent random variables having the common distribution function G(y) = Pr{Yk <- y}. Let M(t) count the number of alpha particles existing at time t. The process is depicted in Figure 4.3.
Y
-
,
Y,
0
W.
W,
W..._
.
W.
t
Figure 4.3 A particle created at time WA <- t still exists at time t if WA + YA ? t.
302
V Poisson Processes
We will use Theorem 4.1 to evaluate the probability distribution of M(t)
under the condition that M(O) = 0.
Let X(t) be the number of particles created up to time t, by assumption
a Poisson process of intensity A. Observe that M(t) S X(t); the number of
existing particles cannot exceed the number of particles created. Condi-
tion on X(t) = n and let W...... W,, - t be the times of particle creation.
Then particle k exists at time t if and only if Wk + Yk ? t. Let
1{Wk+Y?t} _
I
if W1 +Yk?t,
I0
if W+Y<t.
Then 1(Wk + Yk ? t) = 1 if and only if the kth particle is alive at time t.
Thus
Pr{M(t) = mIX(t) = n} = Pr{ 7 1{W + Yk. ? t} = mIX(t) = ni.
k=1
Invoking Theorem 4.1 and the symmetry among particles, we have
Pr1{W +Yk?t} =mIX(t)=nt
k=1
(4.4)
=Pr1{Uk+Y>-t)=m
k=1
where U, U2, ... , U are independent and uniformly distributed on (0, t].
The right-hand side of (4.4) is readily recognized as the binomial distribution in which
lr
p=Pr{Uk.+Yk?t} =- IPr{Yk?t-u}du
t0
= 1r
t f [I - G(t - u)] du
0
I
=
1
r
f [I - G(z)] dz.
0
Thus, explicitly writing the binomial distribution, we have
(4.5)
4,
The Uniform Distribution and Poisson Processes
t
303
p(1 - p)
Pr{M(t) = mIX(t) = n} = m!(nn_ m)!
with p given by (4.5). Finally,
X
Pr{M(t) = m} = 7' Pr{M(t) = MIX(t) = n} Pr{X(t) = n}
-
n!.
f
_,,, m!(n - m)!
p(1 - p)
-
(Apt)"
n!
(4.6)
p)°-,,,(At)
A,
(n - m)!
m!
The infinite sum is an exponential series and reduces according to
(1 - p)
,,,(At)
[At(' - p)]'
(n - m) !
eARO -n,,
l=0
and this simplifies (4.6) to
Pr {M(t) = m} =
e A"' (Apt)",
m!
form = 0, 1. ... .
In words, the number of particles existing at time t has a Poisson distribution with mean
r
Apt =
A
f [I
- G(y)] dy.
(4.7)
0
It is often relevant to let t --> cc in (4.7) and determine the corresponding long run distribution. Let µ = E[Yk] = f,, [1 - G(y)] dy be the mean
lifetime of an alpha particle. It is immediate from (4.7) that as t - x, the
distribution of M(t) converges to the Poisson distribution with parameter
AA. A great simplification has taken place. In the long run, the probability
distribution for existing particles depends only on the mean lifetime µ,
and not otherwise on the lifetime distribution G(y). In practical terms this
statement implies that in order to apply this model, only the mean lifetime
µ need be known.
304
4.1
V Poisson Processes
Shot Noise
The shot noise process is a model for fluctuations in electrical currents
that are due to chance arrivals of electrons to an anode. Variants of the
phenomenon arise in physics and communication engineering. Assume:
(1) Electrons arrive at an anode according to a Poisson process
(X(t); t ? 0) of constant rate A;
(2) An arriving electron produces a current whose intensity x time
units after arrival is given by the impulse response function h(x).
The intensity of the current at time t is, then, the shot noise
X(t)
1(t) _ Y, h(t - Wk),
(4.8)
k=1
where W W, are the arrival times of the electrons.
Common impulse response functions include triangles, rectangles, and
decaying exponentials of the form
h(x) = e
x > 0,
where 0 > 0 is a parameter, and power law shot noise for which
h(x) = x °,
for x > 0.
We will show that for a fixed time point t, the shot noise 1(t) has the same
probability distribution as a certain random sum that we now describe. Independent of the Poisson process X(t), let U,, U,, . . . be independent random variables, each uniformly distributed over the interval (0, t], and de-
fine Ek = h(Uk) for k = 1, 2,
....
The claim is that 1(t) has the same
probability distribution as the random sum
S(t) = E, + ... + EX(".
(4.9)
With this result in hand, the mean, variance, and distribution of the shot
noise 1(t) may be readily obtained using the results on random sums developed in II, Section 3. II, (3.9), for example, immediately gives us
E[I(t)] = E[S(t)] = AtE[h(U,)] = A f h(u) du
0
and
Var[1(t)] = Jet{Var[h(U,)] + E[h(U,)]2)
t
= At E[h(U,)Z] = A f h(u)2 du.
4,
The Uniform Distribution and Poisson Processes
305
In order to establish that the shot noise 1(t) and the random sum S(t) share
the same probability distribution, we need to show that Pr(I(t) <_ x) _
Pr{S(t) : x} for a fixed t > 0. Begin with
X(t)
Pr{I(t) : x) = Pr{ I h(t - W)
x}
k=1
X(t)
_
Pr{ 7, h(t - W) < xl X(t) = n} Pr{X(t) = n)
n=0
k=1
Pr{ 7, h(t - W)
n =O
xI X(t) = n} Pr{X(t) = n},
k=1
and now invoking Theorem 4.1,
=
h(t - Uk) !5 x{ Pr{X(t) = n}
Pr
n =O
x
k =I
n=0
l
rt
Pr{>
_
h(Uk)
k=1
x} Pr{X(t) = n}
J
(because Uk and t - Uk have the same distribution)
Pr{E1 +
n=o
Pr(S(t)s x),
which completes the claim.
4.2. Sum Quota Sampling
A common procedure in statistical inference is to observe a fixed number
n of independent and identically distributed random variables X1, ..., X
and use their sample mean
X _
+ X
X, +
n
V Poisson Processes
306
as an estimate of the population mean or expected value E[X,]. But suppose we are asked to advise an airline that wishes to estimate the failure
rate in service of a particular component, or, what is nearly the same thing,
to estimate the mean service life of the part. The airline monitors a new
plane for two years and observes that the original component lasted 7
months before failing. Its replacement lasted 5 months, and the third component lasted 9 months. No further failures were observed during the remaining 3 months of the observation period. Is it correct to estimate the
mean life in service as the observed average (7 + 5 + 9)/3 = 7 months?
This airline scenario provides a realistic example of a situation in which
the sample size is not fixed in advance but is determined by a preassigned
quota t > 0. In sum quota sampling, a sequence of independent and identically distributed nonnegative random variables X X,.... is observed sequentially, the sampling continuing as long as the sum of the observations
is less than the quota t. Let this random sample size be denoted by N(t).
Formally,
N(t)=max{n?0;X,+
+ X,, <t}.
The sample mean is
W"(
X,v<<, _N(t) -
+
X, +
N(t)
Of course it is possible that X, ? t, and then N(t) = 0, and the sample
mean is undefined. Thus, we must assume, or condition on, the event that
N(t) ? 1. An important question in statistical theory is whether or not this
sample mean is unbiased. That is, how does the expected value of this
sample mean relate to the expected value of, say, X,?
In general, the determination of the expected value of the sample mean
under sum quota sampling is very difficult. It can be carried out, however,
in the special case in which the individual X summands are exponentially
distributed with common parameter A, so that N(t) is a Poisson process.
One hopes that the results in the special case will shed some light on the
behavior of the sample mean under other distributions.
The key is the use of Theorem 4.1 to evaluate the conditional
expectation
E[W,, ';N(t) = n] = E[max{U
n
l
n + 11'
... ,
U,,)]
4.
The Uniform Distribution and Poisson Processes
307
where U, ... , U are independent and uniformly distributed over the
interval (0, t]. Note also that
(At)"e"
Pr{N(t) = nIN(t) > 0) =
n!(1 - e-")'
Then
E N(t)
N(t) > 0] = > ELW' N(t) = nJ Pr{N(t) = nIN(t) > 0)
n
)(I)j ( At)"e "`
t\ n + 1nn!(1 - e-") J
=
n
(
x
1(
(,kt)u+l
l
A \e"' 1 1),,
(n + 1)
1/(e"'- 1 - At)
1l\e"'1
At
1
We can perhaps more clearly see the effect of the sum quota sampling if
we express the preceding calculation in terms of the ratio of the bias to the
true mean E[X,] = 1/A. We then have
E[X,] E[XI]
-
E[N(t)]
At
e"' _ 1
= ee1NUn
-1
The left side is the fraction of bias, and the right side expresses this fraction bias as a function of the expected sample size under sum quota sampling. The following table relates some values:
Fraction Bias
0.58
0.31
0.16
0.17
0.03
0.015
0.0005
E[N(t)]
1
2
3
4
5
6
10
V Poisson Processes
308
In the airline example, we observed N(t) = 3 failures in the two-year
period, and upon consulting the above table, we might estimate the fraction bias to be something on the order of - 16%. Since we observed
X,,,, = 7, a more accurate estimate of the mean time between failures
(MTBF = E[X,]) might be 7/.84 = 8.33, an estimate that attempts to correct, at least on average, for the bias due to the sampling method.
Looking once again at the table, we may conclude in general, that the
bias due to sum quota sampling can be made acceptably small by choosing the quota t sufficiently large so that, on average, the sample size so selected is reasonably large. If the individual observations are exponentially
distributed, the bias can be kept within 0.05% of the true value, provided
that the quota t is large enough to give an average sample size of 10 or
more.
Exercises
4.1. Let [X(t); t ? 0} be a Poisson process of rate A. Suppose it is
known that X(1) = n. For n = 1, 2, ... , determine the mean of the first
arrival time W,.
4.2. Let [X(t); t > 0) be a Poisson process of rate A. Suppose it is
known that X(1) = 2. Determine the mean of W, W,, the product of the first
two arrival times.
4.3. Customers arrive at a certain facility according to a Poisson process
of rate A. Suppose that it is known that five customers arrived in the first
+ WS].
hour. Determine the mean total waiting time E[W, + W, +
4.4. Customers arrive at a service facility according to a Poisson
process of intensity A. The service times Y,, Y,, . . . of the arriving customers are independent random variables having the common probability
distribution function G(y) = Pr{Yx < y}. Assume that there is no limit to
the number of customers that can be serviced simultaneously; i.e., there is
an infinite number of servers available. Let M(t) count the number of customers in the system at time t. Argue that M(t) has a Poisson distribution
with mean Apt, where
p = t-' J[1 - G(y)] dy.
0
309
Problems
4.5. Customers arrive at a certain facility according to a Poisson process
of rate A. Suppose that it is known that five customers arrived in the first
hour. Each customer spends a time in the store that is a random variable,
exponentially distributed with parameter a and independent of the other
customer times, and then departs. What is the probability that the store is
empty at the end of this first hour?
Problems
4.1. Let W, W2, ... be the event times in a Poisson process {X(t);
t ? 0) of rate A. Suppose it is known that X(1) = n. For k < n, what is the
conditional density function of W, ... , W _ W,,,,, ... , W,,, given that
W = w?
4.2. Let {N(t); t ? 0} be a Poisson process of rate A, representing the arrival process of customers entering a store. Each customer spends a duration in the store that is a random variable with cumulative distribution
function G. The customer durations are independent of each other and of
the arrival process. Let X(t) denote the number of customers remaining in
the store at time t, and let Y(t) be the number of customers who have arrived and departed by time t. Determine the joint distribution of X(t) and
Y(t).
...
4.3. Let W,, W,,
be the waiting times in a Poisson process {X(t);
t ? 01 of rate A. Under the condition that X(1) = 3, determine the joint
distribution of U = W1IW, and V = (1 - W3)/(1 - W,).
4.4. Let W,, W,
. . . be the waiting times in a Poisson process {X(t);
t ? 0} of rate A. Independent of the process, let Z, ZZ, . . . be independent
and identically distributed random variables with common probability
density function f(x), 0 < x < oo. Determine Pr{Z > z1, where
Z=min {W,+Z,,W,+Z,,...}.
4.5. Let W,, W,, . . . be the waiting times in a Poisson process {N(t);
t ? 0) of rate A. Determine the limiting distribution of W under the condition that N(t) = n as n -* - and t - oo in such a way that nit = (3 > 0.
V Poisson Processes
310
4.6.
Customers arrive at a service facility according to a Poisson
process of rate A customers/hour. Let X(t) be the number of customers that
have arrived up to time t. Let W, W,, . . . be the successive arrival times
of the customers.
(a) Determine the conditional mean E[W X(t) = 2].
(b) Determine the conditional mean E[W, X(t) = 5].
(c) Determine the conditional probability density function for W,,
given that X(t) = 5.
4.7. Let W W,, . . . be the event times in a Poisson process (X(t);
t ? 0) of rate A, and letf(w) be an arbitrary function. Verify that
X(t)
t
E[> .f(W )] = A Jf(w) dw.
i=1
0
4.8. Electrical pulses with independent and identically distributed
random amplitudes 6 62, . . . arrive at a detector at random times W,,
W, ... according to a Poisson process of rate A. The detector output 0k(t)
for the kth pulse at time t is
0k(t) =
0
fort < Wk,
k exp{-a(t - W)}
fort ? W.
That is, the amplitude impressed on the detector when the pulse arrives is
ek, and its effect thereafter decays exponentially at rate a. Assume that the
detector is additive, so that if N(t) pulses arrive during the time interval
[0, t], then the output at time t is
N(t)
Z(t) _ 7 Ok(t).
k=l
Determine the mean output E[Z(t)] assuming N(0) = 0. Assume that the
amplitudes 4, 62, . . . are independent of the arrival times W,, W . ... .
4.9. Customers arrive at a service facility according to a Poisson process
of rate k customers per hour. Let N(t) be the number of customers that
have arrived up to time t, and let W, , W,, . . . be the successive arrival times
of the customers. Determine the expected value of the product of the waiting times up to time t. (Assume that W, W,
W,,,,, = 1 when N(t) = 0.)
5.
Spatial Poisson Processes
311
Compare and contrast the example immediately following Theorem 4.1, the shot noise process of Section 4.1, and the model of Problem
4.8. Can you formulate a general process of which these three examples
are special cases?
4.10.
Computer Challenge Let U0, U,, . . . be independent random
variables, each uniformly distributed on the interval (0, 1). Define a stochastic process {S,,} recursively by setting
4.11.
So = 0
for n > 0.
and
(This is an example of a discrete-time, continuous-state, Markov process.)
When n becomes large, the distribution of S,, approaches that of a random
variable S = SF, and S must have the same probability distribution as
U(1 + S), where U and S are independent. We write this in the form
2b
S = U(1 + S),
from which it is easy to determine that E[S] = 1, Var[S]
Laplace transform)
6
E[e-oS] = exp (-J
and even (the
1 _ e.,
l 0+
dull
0 > 0.
The probability density functionf(s) satisfies
f(s) = 0 for s : 0,
df
ds
=
l f(s- 1),
s
and
fors>0.
What is the 99th percentile of the distribution of S? (Note: Consider the shot
noise process of Section 4.1. When the Poisson process has rate A = I and
the impulse response function is the exponential h(x) = exp{-x}, then the
shot noise 1(t) has, in the limit for large t, the same distribution as S.)
5.
Spatial Poisson Processes
In this section we define some versions of multidimensional Poisson
processes and describe some examples and applications.
V Poisson Processes
312
Let S be a set in n-dimensional space and let si be a family of subsets
of S. A point process in S is a stochastic process N(A) indexed by the sets
A in , si and having the set of nonnegative integers (0, 1 , 2, ... ) as its possible values. We think of "points" being scattered over S in some random
manner and of N(A) as counting the number of points in the set A. Because
N(A) is a counting function, there are certain obvious requirements that it
must satifsy. For example, if A and B are disjoint sets in whose union
A U B is also in s4, then it must be that N(A U B) = N(A) + N(B). In
words, the number of points in A or B equals the number of points in A
plus the number of points in B when A and B are disjoint.
The one-dimensional case, in which S is the positive half line and a
comprises all intervals of the form A = (s, t], for 0 <- s < t, was introduced in Section 3. The straightforward generalization to the plane and
three-dimensional space that is now being discussed has relevance when
we consider the spatial distribution of stars or galaxies in astronomy, of
plants or animals in ecology, of bacteria on a slide in medicine, and of defects on a surface or in a volume in reliability engineering.
Let S be a subset of the real line, two-dimensional plane, or threedimensional space; let sl be the family of subsets of S and for any set A
in si; let IAI denote the size (length, area, or volume, respectively) of A.
Then {N(A); A in sl} is a homogeneous Poisson point process of intensity
A>0if
(i) for each A in A, the random variable N(A) has a Poisson distribution with parameter AN;
(ii) for every finite collection {A......
of disjoint subsets of S, the
random variables N(A,), . . . ,
are independent.
In Section 2, the law of rare events was invoked to derive the Poisson
process as a consequence of certain physically plausible postulates. This
implication serves to justify the Poisson process as a model in those situations where the postualtes may be expected to hold. An analogous result
is available in the multidimensional case at hand. Given an arbitrary point
process {N(A); A in .si}, the required postulates are as follows:
(1) The possible values for N(A) are the nonnegative integers
{0, 1, 2, ...} and 0 < Pr(N(A) = 0) < I if 0 < 1AJ < oo.
5.
Spatial Poisson Processes
313
(2) The probability distribution of N(A) depends on the set A only
through its size (length, area, or volume) JAI, with the further property that Pr{N(A) ? 1 } = AJAJ + o(JAI) as JAI L 0.
(3) For m = 2, 3, . . . , if A,, A...... A. are disjoint regions, then N(A,),
N(A, U
. ,
N(A,),
+
N(A,,,).
U A,,,) = N(AI) + N(A,) +
A2 U
(4)
Pr{N(A) ? 1) = 1
lim
A'o Pr{N(A) = 1}
The motivation and interpretation of these postulates is quite evident.
Postulate 2 asserts that the probability distribution of N(A) does not depend on the shape or location of A, but only on its size. Postulate 3 requires that the outcome in one region not influence or be influenced by the
outcome in a second region that does not overlap the first. Postulate 4 precludes the possibility of two points occupying the same location.
If a random point process N(A) defined with respect to subsets A of
Euclidean n-space satisfies Postulates 1 through 4, then N(A) is a homogeneous Poisson point process of intensity A > 0, and
AA(AIAI)k
Pr(N(A) = k} =
fork = 0, 1, ... .
e
kd
(5.1)
As in the one-dimensional case, homogeneous Poisson point processes
in n-dimensions are highly amenable to analysis, and many results are
known for them. We elaborate a few of these consequences next, beginning with the uniform distribution of a single point. Consider a region A
of positive size JAI > 0, and suppose it is known that A contains exactly
one point; i.e., N(A) = 1. Where in A is this point located? We claim that
the point is uniformly distributed in the sense that
Pr{N(B) = 1IN(A) = 1) =
LB'
for any set B C A.
(5.2)
In words, the probability of the point being in any subset B of A is proportional to the size of B; that is, the point is uniformly distributed in A.
The uniform distribution expressed in (5.2) is an immediate consequence
of elementary conditional probability manipulations. We write A = B U C,
where B is an arbitrary subset of A and C is the portion of A not included
V Poisson Processes
314
in B. Then B and C are disjoint, so that N(B) and N(C) are independent
Poisson random variables with respective means A BI and AICI. Then
Pr{N(B) = IIN(A) = 1 } =
Pr{N(B) = 1, N(C) = 0}
r{ () -
}
A'B1e-axe-Ac
AIAIe-A A
=
IBI
I
(because
Al
Cj = A ),
and the proof is complete.
The generalization to n points in a region A is stated as follows. Consider
a set A of positive size IAI > 0 and containing N(A) = n ? 1 points. Then
these n points are independent and uniformly distributed in A in the sense
U A,,, = A,
that for any disjoint partition A...... A,,, of A, where A, U
+ k,,, = n, we have
and any positive integers k ... , k,,,, where k, +
Pr{N(A1= k...... N(A,,,) = k,,, N(A) = n }
n!
A,
k,! ... k,,,!
A
JA
(5.3)
J
IAI
Equation (5.3) expresses the multinomial distribution for the conditional
distribution of N(A,), . . . , N(A,,,) given that N(A) = n.
Example An Application in Astronomy Consider stars distributed in
space in accordance with a three-dimensional Poisson point process of intensity A > 0. Let x and y designate general three-dimensional, vectors,
and assume that the light intensity exerted at x by a star located at y is
f(x, y, a) = a/IIx - yII2 = al[(x, - y1)2 + (x, - y2)2 + (x3 - y3)2], where
a is a random parameter depending on the intensity of the star at y. We assume that the intensities a associated with different stars are independent,
identically distributed random variables possessing a common mean µ
and variance o<;. We also assume that the combined intensity exerted at the
point x due to light created by different stars accumulates additively. Let
Z(x, A) denote the total light intensity at the point x due to signals emanating from all sources located in region A. Then
315
Exercises
N(A)
f(x, y,., a)
Z(x, A) _
(5.4)
N(A)
where y, is the location of the rth star in A. We recognize (5.4) as a random sum, as discussed in II, Section 3.2. Accordingly, we have the mean
intensity at x given by
E[Z(x, A)] = (E[N(A)])(E[ f(x, y, a)]).
(5.5)
Note that E[N(A)] = .1JAJ, while because we have assumed a and y to be
independent,
E[f(x, y, a)] = E[a]E[II x But as a consequence of the Poisson distribution of stars in space, we may
take y to be uniformly distributed in A. Thus
E[Ijx
-
1
JA 1
J
x
A
With µa = E[a], then (5.5) reduces to
E[Z(x, A)] = ' Pa J
A
dy
x-
Exercises
5.1. Bacteria are distributed throughout a volume of liquid according to
a Poisson process of intensity 0 = 0.6 organisms per mm'. A measuring
device counts the number of bacteria in a 10 mm' volume of the liquid.
What is the probability that more than two bacteria are in this measured
volume?
5.2. Customer arrivals at a certain service facility follow a Poisson
process of unknown rate. Suppose it is known that 12 customers have arrived during the first three hours. Let Ni be the number of customers who
arrive during the ith hour, i = 1, 2, 3. Determine the probability that
N, = 3, N, = 4, and N; = 5.
V Poisson Processes
316
5.3. Defects (air bubbles, contaminants, chips) occur over the surface of
a varnished tabletop according to a Poisson process at a mean rate of one
defect per top. If two inspectors each check separate halves of a given
table, what is the probability that both inspectors find defects?
Problems
5.1. A piece of a fibrous composite material is sliced across its circular
cross section of radius R, revealing fiber ends distributed across the circular area according to a Poisson process of rate 100 fibers per cross section.
The locations of the fibers are measured, and the radial distance of each
fiber from the center of the circle is computed. What is the probability
density function of this radial distance X for a randomly chosen fiber?
5.2.
Points are placed on the surface of a circular disk of radius one according to the following scheme. First, a Poisson distributed random vari-
able N is observed. If N = n, then n random variables 0...... 0,, are independently generated, each uniformly distributed over the interval
[0, 2ir), and n random variables R . . . , R,, are independently generated,
each with the triangular density f(r) = 2r, 0 < r < 1. Finally, the points
are located at the positions with polar coordinates (R;, 9;), i = 1, . . . , n.
What is the distribution of the resulting point process on the disk?
5.3. Let {N(A); A E R2) be a homogeneous Poisson point process in the
plane, where the intensity is A. Divide the (0, t] X (0, t] square into n2
boxes of side length d = t/n. Suppose there is a reaction between two or
more points whenever they are located within the same box. Determine
the distribution for the number of reactions, valid in the limit as t 7 00and
d-*0insuch awaythat td-
> 0.
5.4. Consider spheres in three-dimensional space with centers distributed according to a Poisson distribution with parameter AIAJ, where CAI
now represents the volume of the set A. If the radii of all spheres are distributed according to F(r) with densityf(r) and finite third moment, show
that the number of spheres that cover a point t is a Poisson random variable with parameter ;Air f o r'f(r) dr.
317
problems
5.5. Consider a two-dimensional Poisson process of particles in the
plane with intensity parameter v. Determine the distribution Fo(x) of the
distance between a particle and its nearest neighbor. Compute the mean
distance.
5.6.
Suppose that stars are distributed in space following a Poisson
point process of intensity A. Fix a star alpha and let R be the distance from
alpha to its nearest neighbor. Show that R has the probability density
function
fR(x) _ (4A7rx'-) expj
4A 7rx'
1,
x > 0.
Consider a collection of circles in the plane whose centers are distributed according to a spatial Poisson process with parameter AJAI, where
5.7.
J denotes the area of the set A. (In particular, the number of centers (A)
in the set A follows the distribution law Pr(e(A) = k} = e AA[(A[AJ)'1k!].)
The radius of each circle is assumed to be a random variable independent
of the location of the center of the circle, with density functionf(r) and finite second moment.
(a) Show that C(r), defined to be the number of circles that cover the
origin and have centers at a distance less than r from the origin, determines a variable-time Poisson process, where the time variable
is now taken to be the distance r.
Hint: Prove that an event occurring between r and r + dr (i.e., there is
a circle that covers the origin and whose center is in the ring of radius r
to r + dr) has probability A27rr dr f ;f(p) dp + o(dr), and events occurring over disjoint intervals constitute independent r.v.'s. Show that C(r)
is a variable-time (nonhomogeneous) Poisson process with parameter
x
A(r) = 27rAr f f (p) d p.
(b) Show that the number of circles that cover the origin is a Poisson
random variable with parameter A f o 7rr2f(r) dr.
V Poisson Processes
318
Compound and Marked Poisson Processes
6.
Given a Poisson process X(t) of rate A > 0, suppose that each event has
associated with it a random variable, possibly representing a value or a
cost. Examples will appear shortly. The successive values Y Y,, ... are
assumed to be independent, independent of the Poisson process, and random variables sharing the common distribution function
G(y) = Pr{Yk <_ y}.
A compound Poisson process is the cumulative value process defined by
X(t)
Z(t) = 7 Y
for t ? 0.
(6.1)
k=I
A marked Poisson process is the sequence of pairs (W, Y,), (W,, Y,), ...,
where W,, W,, ... are the waiting times or event times in the Poisson
process X(t).
Both compound Poisson and marked Poisson processes appear often as
models of physical phenomena.
6.1.
Compound Poisson Processes
Consider the compound Poisson process Z(t) 111; Yk. If A > 0 is the rate
for the process X(t) and µ = E[Y] and v2 = Var[Y,] are the common mean
and variance for Y,, Y,, ... , then the moments of Z(t) can be determined
from the random sums formulas of II, Section 3.2 and are
E[Z(t)] = Apt;
Var[Z(t)] = A(v'- + µ-)t.
(6.2)
Examples
(a) Risk Theory Suppose claims arrive at an insurance company in
accordance with a Poisson process having rate A. Let Yk be the magnitude
of the kth claim. Then Z(t) _ 110", Yk represents the cumulative amount
claimed up to time t.
(b) Stock Prices Suppose that transactions in a certain stock take
place according to a Poisson process of rate A. Let Yk denote the change in
market price of the stock between the kth and (k - 1)st transaction.
The random walk hypothesis asserts that Y, Y,, . . . are independent random variables. The random walk hypothesis, which has a history dating
Compound and Marked Poisson Processes
6.
319
back to 1900, can be deduced formally from certain assumptions describing a "perfect market."
Then Z(t) _ IA=I'I,, Y, represents the total price change up to time t.
This stock price model has been proposed as an explanation for why
stock price changes do not follow a Gaussian (normal) distribution.
The distribution function for the compound Poisson process Z(t) _
can be represented explicitly after conditioning on the values of
IM11,,
X(t). Recall the convolution notation
G"(y)=Pr{Y,+...+Y<_y}
+f
(6.3)
J G")(y - z) dG(z)
-x
with
G(01(y)
{1
= 0
for y ? 0,
for y < 0.
Then
r X(r)
Pr(Z(t)
z} = Prj 2 Y < z}
k=I
x(n
_
Pr
n=o
_
7, Y
( 1 t)'
zjX(t) = n l/l
k=1
J
(At)ne-At
=o
n!
GOO (Z)
-A,
(6.4)
n!
(since X(t) is independent
of
Example A Shock Model Let X(t) be the number of shocks to a system up to time t and let Yk be the damage or wear incurred by the kth
shock. We assume that damage is positive, i.e., that Pr{Y. ? 0} = 1, and
that the damage accumulates additively, so that Z(t) = 111', Y, represents
the total damage sustained up to time t. Suppose that the system continues
to operate as long as this total damage is less than some critical value a
and fails in the contrary circumstance. Let The the time of system failure.
Then
{T > t}
if and only if
{Z(t) < a}.
(Why?)
(6.5)
V Poisson Processes
320
In view of (6.4) and (6.5), we have
x
(At)"e-A,
Pr[T > t} =
G("'(a).
n!
n=0
All summands are nonnegative, so we may interchange integration and
summation to get the mean system failure time
x
E[T] =
f
Pr[T> t} dt
0
x
G",,(a).
=0
This expression simplifies greatly in the special case in which Y Y2, .. .
are each exponentially distributed according to the density gy(y) = µe F"
for y > 0. Then the sum Y, +
+ Y has the gamma distribution
(µz)ke ,<<
k!
k=0
and
x
7
n=0
x
G'(a)=7,
(pz)ke "<
_
k=n
x
k!
IM)keµa
Y
n=0k=n
k!
(Fa)ke-µa
k!
x
(1 + k) (I-)ke_µa
ki
k=0
= 1 + µa.
When Y, Y,,
... are exponentially distributed, then
E[T]=
1+µa
A
6.
Compound and Marked Poisson Processes
6.2.
321
Marked Poisson Processes
Again suppose that a random variable Y is associated with the kth event in a
Poisson process of rate A. W e stipulate that Y Y,, ... are independent, independent of the Poisson process, and share the common distribution function
G(y) = Pr{Y" : y}.
The sequence of pairs (W, Y,), (W,, Y,), ... is called a marked Poisson
process.
We begin the analysis of marked Poisson processes with one of the simplest cases. For a fixed value p (0 < p < 1) suppose
Pr(Y"=0) = q = 1 -p.
Pr{Y"= 1} = p,
Now consider separately the processes of points marked with ones and of
points marked with zeros. In this case we can define the relevant Poisson
processes explicitly by
and
Y
X. (t) =
X0(t) = X(t) - X,(t).
k=1
Then nonoverlapping increments in X,(t) are independent random variables, X,(0) = 0, and finally, Theorem 1.2 applies to assert that X,(t) has a
Poisson distribution with mean Apt. In summary, X,(t) is a Poisson process
with rate Ap, and the parallel argument shows that X0(t) is a Poisson
process with rate A(1 - p). What is even more interesting and surprising
is that X0(t) and X,(t) are independent processes! The relevant property to
check is that Pr{X0(t) = j and X,(t) = k) = Pr{X0(t) = j) X Pr{X1(t) = k}
We establish this independence by writing
for j, k = 0, 1,
....
Pr{X0(t) = j, X,(t) = k} = Pr{X0(t) = j + k, XI(t) = k}
= Pr{X1(t) = kIX(t) = j + k} Pr{X(t) = j + k)
(j + k)!
j!k!
pk (
- P)
(At)f+"e-A7
J
(j + k)!
A('-n"(A(1
[eAr(Apt)kI[e
k!
- p)t)'I
j!
= Pr{X1(t) = k} Pr{X0(t) = j}
forj, k = 0, 1,....
V Poisson Processes
322
Example Customers enter a store according to a Poisson process of rate
A = 10 per hour. Independently, each customer buys something with probability p = 0.3 and leaves without making a purchase with probability q =
1 - p = 0.7. What is the probability that during the first hour 9 people
enter the store and that 3 of these people make a purchase and 6 do not?
Let X, = X,(1) be the number of customers who make a purchase during the first hour and X0 = X(,(1) be the number of people who do not. Then
X, and Xo are independent Poisson random variables having respective
rates 0.3(10) = 3 and 0.7(10) = 7. According to the Poisson distribution,
Pr{X, = 3) =
3'e 3
3!
= 0.2240,
lee-7
= 0 . 1490 ,
Pr {X = 6) =
6!
and
Pr{X1 = 3, X0 = 6} = Pr{X1 = 3) Pr{X = 61 = (0.2240)(0.1490) = 0.0334.
In our study of marked Poisson processes, let us next consider the case
where the value random variables Y,, Y,, ... are discrete, with possible val-
ues 0, 1, 2, ... and
Pr{Y = k} = ax > 0
for k = 0, 1, ... ,with Lak = 1.
In Figure 6.1, the original Poisson event times W, k, ... are shown on
Y, = 3
Y,=2
I
W,
Figure 6.1
W,
W,
W4
A marked Poisson process. W, W,, . . are the event times in
a Poisson process of rate A. The random variables Y, Y,, . . . are
the markings, assumed to be independent and identically distributed, and independent of the Poisson process.
.
6.
Compound and Marked Poisson Processes
323
the bottom axis. Then a point is placed in the (t, y) plane at (W,,, Y,,) for
every n. For every integer k = 0, 1, 2.... one obtains a point process that
corresponds to the times W for which Y = k. The same reasoning as in
the zero-one case applies to imply that each of these processes is Poisson,
the rate for the kth process being Aak, and that processes for distinct values of k are independent.
To state the corresponding decomposition result when the values
Y Y,, . . . are continuous random variables requires a higher level of sophistication, although the underlying ideas are basically the same. To set
the stage for the formal statement, we first define what we mean by a nonhomogeneous Poisson point process in the plane, thus extending the homogeneous processes of the previous section. Let 0 = 0(x, y) be a nonnegative function defined on a region S in the (x, y) plane. For each subset
A of S, let µ(A) = f fA 6(x, y) dx dy be the volume under 9(x, y) enclosed
by A. A nonhomogeneous Poisson point process of intensity function
0(x, y) is a point process {N(A); A C S) for which
(i) for each subset A of S, the random variable N(A) has a Poisson distribution with mean µ(A);
(ii) for disjoint subsets A,, . . ., A,,, of S, the random variables N(A,), ... ,
N(A,,,) are independent.
It is easily seen that the homogeneous Poisson point process of intensity A corresponds to the function 9(x, y) being constant, and 6(x, y) = A
for all x, y.
With this definition in hand, we state the appropriate decomposition result for general marked Poisson processes.
...
Theorem 6.1
Let (W Y,), (W,, Y,),
be a marked Poisson process
where W W,,
. are the waiting times in a Poisson process of rate A
and Y Y,, ... are independent identically distributed continuous random
variables having probability density function g(y). Then (W Y),
(W,, Y2), . . . form a two-dimensional nonhomogeneous Poisson point
process in the (t, y) plane, where the mean number of points in a region A
is given by
µ(A) = JJAg(y)dydt.
A
(6.6)
V Poisson Processes
324
Figure 6.2 diagrams the scene.
Y
oW,
oW3
6
0
W4
W5
Figure 6.2 A marked Poisson process.
Theorem 6.1 asserts that the numbers of points in disjoint intervals are
independent random variables. For example, the waiting times corresponding to positive values Y Y,, . . . form a Poisson process, as do the times associated with negative values, and these two processes are independent.
Example Crack Failure The following model is proposed to describe
the failure time of a sheet or volume of material subjected to a constant
stress o. The failure time is viewed in two parts, crack initiation and crack
propagation.
Crack initiation occurs according to a Poisson process whose rate per
unit time and unit volume is a constant A > 0 depending on the stress
level a-. Crack initiation per unit time then is a Poisson process of rate
A,IVJ, where IVI is the volume of material under consideration. We let
W W,, . . . be the times of crack initiation.
Once begun, a crack grows at a random rate until it reaches a critical
size, at which instant structural failure occurs. Let Yk be the time to reach
critical size for the kth crack. The cumulative distribution function
G, (y) = Pr{Yk s y} depends on the constant stress level a-.
We assume that crack initiations are sufficiently sparse as to make
Y Y,, . . . independent random variables. That is, we do not allow two
small cracks to join and form a larger one.
The structural failure time Z is the smallest of W, + Y,, W, + Y,..... It
is not necessarily the case that the first crack to appear will cause system
failure. A later crack may grow to critical size faster.
6.
Compound and Marked Poisson Processes
325
Figure 6.3 A crack failure model.
In the (t, y) plane, the event {min { Wk + Y,.) > z) corresponds to no
points falling in the triangle A = ((t, y): t + y z, t ? 0, y ? 0), as shown
in Figure 6.3.
The number of points N(O) falling in the triangle A has a Poisson distribution with mean µ(0) given, according to (6.6), by
µ(A) = JJAIVjds g?(u) du
.s
= f AoIVI
g,(u) du ds
0
=AvIVI f Go(z
- s)ds
0
Z
= A,1VI
f GQ(v) dv.
0
From this we obtain the cumulative distribution function for structural
failure time,
Pr(Z<-z) = I -Pr{Z>z} = 1 -Pr(N(L)=0}
= 1 - exp -,QV1 f GQ(v) dv
.
0
Observe the appearance of the so-called size effect in the model,
wherein the structure volume IVI affects the structural failure time even at
326
V Poisson Processes
constant stress level The parameter A and distribution function G,,(y)
would require experimental determination.
Example The Strength Versus Length of Filaments
It was noted that
the logarithm of mean tensile strength of brittle fibers, such as boron filaments, in general varies linearly with the logarithm of the filament length,
but that this relation did not hold for short filaments. It was suspected that
the breakdown in the log linear relation might be due to testing or measurement problems, rather than being an inherent property of short filaments. Evidence supporting this idea was the observation that short filaments would break in the test clamps, rather than between them as desired,
more often than would long filaments. Some means of correcting observed
mean strengths to account for filaments breaking in, rather than between,
the clamps was desired. It was decided to compute the ratio between the
actual mean strength and an ideal mean strength, obtained under the assumption that there was no stress in the clamps, as a correction factor.
Since the molecular bonding strength is several orders of magnitude
higher than generally observed strengths, it was felt that failure typically
was caused by flaws. There are a number of different types of flaws, both
internal flaws such as voids, inclusions, and weak grain boundaries, and
external, or surface, flaws such as notches and cracks that cause stress
concentrations. Let us suppose that flaws occur independently in a Poisson manner along the length of the filament. We let Yk be the strength of
the filament at the kth flaw and suppose Yk has the cumulative distribution
function G(y), y > 0. We have plotted this information in Figure 6.4. The
flaws reduce the strength. Opposing the strength is the stress in the filament. Ideally, the stress should be constant along the filament between the
clamp faces and zero within the clamp. In practice, the stress tapers off to
zero over some positive length in the clamp. As a first approximation it is
reasonable to assume that the stress decreases linearly. Let l be the length
of the clamp and t the distance between the clamps, called the gauge
length, as illustrated in Figure 6.4 on the next page.
The filament holds as long as the stress has not exceeded the strength
as determined by the weakest flaw. That is, the filament will support a
stress of y as long as no flaw points fall in the stress trapezoid of Figure
6.4. The number of points in this trapezoid has a Poisson distribution with
6.
Compound and Marked Poisson Processes
327
kthflaw
Distance along filament
y*
Ideal stress
Actual
B
' stress
A
-H
l
1Filament
Clamp
Figure 6.4 The stress versus strength of a filament under tension. Flaws reducing the strength of a filament below its theoretical maximum
y* are distributed randomly along its length. The stress in the filament is constant at the level y between clamps and tapers off to
zero within the clamps. The filament fails if at any point along its
length a flaw reduces its strength below its stress.
mean µ(B) + 2µ(A). In particular, no points fall there with probability
e-I"(B)+'a`A)1. If
we let S be the strength of the filament, then
Pr(S > y) =
e-?µ(A)-µ mf)
We compute
/t(A)
= f G(
l
llAdx
and
µ(B) = AtG(y).
V Poisson Processes
328
Finally, the mean strength of the filament is
x
x
E[S] = J Pr{S > y} dy = J exp -A tG(y) + 2 JG() dx
dy.
0
For an ideal filament we use the same expression but with 1 = 0.
Exercises
6.1. Customers demanding service at a central processing facility arrive
according to a Poisson process of intensity 6 = 8 per unit time. Independently, each customer is classified as high priority with probability
a = 0.2, or low priority with probability 1 - a = 0.8. What is the probability that three high priority and five low priority customers arrive during the first unit of time?
6.2. Shocks occur to a system according to a Poisson process of intensity A. Each shock causes some damage to the system, and these damages
accumulate. Let N(t) be the number of shocks up to time t, and let Y be
the damage caused by the ith shock. Then
is the total damage up to time t. Determine the mean and variance of the
total damage at time t when the individual shock damages are exponentially distributed with parameter 0.
6.3.
Y,, YZ,
Let {N(t); t ? 01 be a Poisson process of intensity A, and let
... be independent and identically distributed nonnegative random
variables with cumulative distribution function G(y) = Pr{Y <_ y}. Determine Pr{Z(t) > zjN(t) > 0}, where
Z(t) = min{Y
6.4.
Y,,
... ,
YN,,,}
Men and women enter a supermarket according to independent
Poisson processes having respective rates of two and four per minute.
(a) Starting at an arbitrary time, what is the probability that at least two
men arrive before the first woman arrives?
329
Problems
(b) What is the probability that at least two men arrive before the third
woman arrives?
6.5. Alpha particles are emitted from a fixed mass of material according
to a Poisson process of rate A. Each particle exists for a random duration
and is then annihilated. Suppose that the successive lifetimes Y Y,, . . . of
distinct particles are independent random variables having the common
distribution function G(y) = Pr{Yk s y}. Let M(t) be the number of particles existing at time t. By considering the lifetimes as markings, identify
the region in the lifetime, arrival-time space that corresponds to M(t), and
thereby deduce the probability distribution of M(t).
Problems
6.1. Suppose that points are distributed over the half line [0, 00) according to a Poisson process of rate A. A sequence of independent and identically distributed nonnegative random variables Y Y2, . . . is used to reposition the points so that a point formerly at location WA is moved to the
location Wk + Yk. Completely describe the distribution of the relocated
points.
6.2.
Suppose that particles are distributed on the surface of a circular region according to a spatial Poisson process of intensity A particles per unit
area. The polar coordinates of each point are determined, and each angular coordinate is shifted by a random amount, the amounts shifted for distinct points being independent random variables following a fixed proba-
bility distribution. Show that at the end of the point movement process,
the points are still Poisson distributed over the region.
6.3.
Shocks occur to a system according to a Poisson process of intensity A. Each shock causes some damage to the system, and these damages
accumulate. Let N(t) be the number of shocks up to time t, and let Y; be
the damage caused by the ith shock. Then
X(t)=Y,+...+YN(,)
is the total damage up to time t. Suppose that the system continues to operate as long as the total damage is strictly less than some critical value a,
and fails in the contrary circumstance. Determine the mean time to system
V Poisson Processes
330
failure when the individual damages Y have a geometric distribution with
Pr{Y= k} = p(1 - p)k,k= 0, 1,....
6.4. Let {X(t); t ? 0} and {Y(t); t ? 0} be independent Poisson
processes with respective parameters A and µ. For a fixed integer a, let
T = min { t ? 0; Y(t) = a) be the random time that the Y process first
k} for k = 0, 1, ... .
reaches the value a. Determine
Hint: First consider = X(T,) in the case in which a = 1. Then has
a geometric distribution. Then argue that X(T,) for general a has the
same distribution as the sum of a independent 's and hence has a negative binomial distribution.
6.5. Let (X(t); t ? 0) and (Y(t); t ? 0) be independent Poisson
processes with respective parameters A and µ. Let T = min { t ? 0;
Y(t) = I) be the random time of the first event in the Y process. Determine
Pr{X(T/2) = k) fork = 0, 1....
.
be the event times in a Poisson process {X(t); t ? 0}
of rate A. A new point process is created as follows: Each point WA is replaced by two new points located at W,, + Xk and Wk + YA, where X, Y,
X,, Y,, . . . are independent and identically distributed nonnegative random
variables, independent of the Poisson process. Describe the distribution of
the resulting point process.
6.6
Let W,, W,,
. . .
6.7. Let {N(t); t ? 01 be a Poisson process of intensity A, and let
Y Y,, . . . be independent and identically distributed nonnegative random
variables with cumulative distribution function
G(y) = y°
for 0 < y < 1.
Determine Pr{Z(t) > zIN(t) > 0), where
Z(t) = min{Y,,
Describe the behavior for large t.
6.8. Let {N(t); t ? 0) be a nonhomogeneous Poisson process of intensity A(t), t > 0, and let Y Y,, . . . be independent and identically distributed nonnegative random variables with cumulative distribution function
G(y)=y"
for0<y<1.
331
problems
Suppose that the intensity process averages out in the sense Oat
lrr
lim
- J A(u) du = 0.
I-+,
t
n
Let
Z(t) = min{Y,, Y,..... Y,,,,)}.
Determine
lim Pr{t"°Z(t) > z}.
6.9. Let W,, W,, ... be the event times in a Poisson process of rate A,
and let N(t) = N((0, t]) be the number of points in the interval (0, t].
Evaluate
N(,)
E7
Note:
(4Y.)'-].
Y.°_, KY = 0.
6.10. A Bidding Model Let U,, U,,
be independent random variables, each uniformly distributed over the interval (0, 1]. These random
variables represent successive bids on an asset that you are trying to sell,
and that you must sell by time t = 1, when the asset becomes worthless.
As a strategy, you adopt a secret number 0, and you will accept the first
offer that is greater than 0. For example, you accept the second offer if
U, <_ 0 while U, > 0. Suppose that the offers arrive according to a unit
rate Poisson process (A = 1).
. .
.
(a) What is the probability that you sell the asset by time t = 1?
(b) What is the value for 0 that maximizes your expected return? (You get
nothing if you don't sell the asset by time t = 1.)
(c) To improve your return, you adopt a new strategy, which is to accept
an offer at time t if it exceeds 0(t) = (1 - t)/(3 - t). What are your
new chances of selling the asset, and what is your new expected
return?
Chapter VI
Continuous Time Markov Chains
1.
Pure Birth Processes
In this chapter we present several important examples of continuous time,
discrete state, Markov processes. Specifically, we deal here with a family
of random variables {X(t); 0 < t < co} where the possible values of X(t)
are the nonnegative integers. We shall restrict attention to the case where
(X(t)) is a Markov process with stationary transition probabilities. Thus,
the transition probability function for t > 0,
P;(t)=Pr{X(t+u)=jjX(u)=i},
i,j=0,1,2,...,
is independent of u ? 0.
It is usually more natural in investigating particular stochastic models
based on physical phenomena to prescribe the so-called infinitesimal
probabilities relating to the process and then derive from them an explicit
expression for the transition probability function. For the case at hand, we
will postulate the form of P;(h) for h small, and, using the Markov property, we will derive a system of differential equations satisfied by P;(t) for
all t > 0. The solution of these equations under suitable boundary conditions gives P;(t).
By way of introduction to the general pure birth process, we review
briefly the axioms characterizing the Poisson process.
334
1.1.
VI
Continuous Time Markov Chains
Postulates for the Poisson Process
The Poisson process is the prototypical pure birth process. Let us point out
the relevant properties. The Poisson process is a Markov process on the
nonnegative integers for which
(i) Pr{X(t + h) - X(t) = 1JX(t) = x} = Ah + o(h) as h 10
(x = 0, 1, 2, ...).
(ii) Pr{X(t + h) - X(t) = 0IX(t) = x} = 1 - Ah + o(h) as h 10.
(iii) X(O) = 0.
The precise interpretation of (i) is the relationship
Pr{X(t + h) - X(t) = lIX(t) = x}
lira
h
= A.
The o(h) symbol represents a negligible remainder term in the sense that
if we divide the term by h, then the resulting value tends to zero as h tends
to zero. Notice that the right side of (i) is independent of x.
These properties are easily verified by direct computation, since the explicit formulas for all the relevant properties are available. Problem 1.13
calls for showing that these properties, in fact, define the Poisson process.
1.2.
Pure Birth Process
A natural generalization of the Poisson process is to permit the chance of
an event occurring at a given instant of time to depend upon the number
of events that have already occurred. An example of this phenomenon is
the reproduction of living organisms (and hence the name of the process),
in which under certain conditions-sufficient food, no mortality, no migration, for example-the infinitesimal probability of a birth at a given instant is proportional (directly) to the population size at that time. This example is known as the Yule process and will be considered in detail later.
Consider a sequence of positive numbers, { AA } . We define a pure birth
process as a Markov process satisfying the following postulates:
(1) Pr{X(t + h) - X(t) = 1IX(t) = k} = AA.h + o,,k(h) (h -4 0+).
1.
Pure Birth Processes
335
(2) Pr{X(t + h) - X(t) = 0 X(t) = k) = 1 - 1tkh + o,.,(h).
(3) Pr{X(t + h) - X(t) < 0 X(t) = k} = 0 (k >_ 0).
(1.1)
As a matter of convenience we often add the postulate
(4) X(0) = 0.
With this postulate X(t) does not denote the population size but, rather, the
number of births in the time interval (0, t}.
Note that the left sides of Postulates (1) and (2) are just P.k+,(h) and
PA,x(h), respectively (owing to stationarity), so that o,,k(h) and o,.,(h) do not
depend upon t.
We define P(t) = Pr{X(t) = n}, assuming X(0) = 0.
By analyzing the possibilities at time t just prior to time t + h (h small),
we will derive a system of differential equations satisfied by P(t) for
t ? 0, namely
Pa(t) = -A0P0(t),
forn ? 1,
P111(t) = -
(1.2)
with initial conditions
PO(0) = 1,
0,
n > 0.
Indeed, if h > 0, n ? 1, then by invoking the law of total probability, the
Markov property, and Postulate (3) we obtain
x
h) =
PR.(t) Pr(X(t + h) = nI X(t) = k}
k=0
PA.(t) Pr{X(t + h) - X(t) = n - kJX(t) = k}
k=0
PA(t) Pr{X(t + h) - X(t) = n - k X(t) = k}.
k=0
Now for k = 0, 1,...,n-2wehave
Pr{X(t + h) - X(t) = n - klX(t) = k}
Pr{X(t + h) - X(t) ? 21X(t) = k}
= o1.k(h) + o,.k(h),
336
VI
Continuous Time Markov Chains
or
Pr(X(t + h) - X(t) = n - kjX(t) = k} = 03.,,,k(h),
k= 0, ... , n - 2.
Thus
h) = P(t)[1 -
02.,,(h)] +
o,.,,_,(h)]
n-2
k=0
.or
h) -
(1.3)
=
o,,(h)] +
oj.,,_1(h)l + o,,(h),
where, clearly, lim,,10
0 uniformly in t ? 0, since o,,(h) is
bounded by the finite sum Jk=0 o3.n.k(h), which does not depend on t.
Dividing by h and passing to the limit h 10, we validate the relations
(1.2), where on the left side we should, to be precise, write the derivative
from the right. With a little more care, however, we can derive the same
relation involving the derivative from the left. In fact, from (1.3) we see
are continuous functions of t. Replacing t by t - h in
at once that the
(1.3), dividing by h, and passing to the limit h 10, we find that each
has a left derivative that also satisfies equation (1.2).
The first equation of (1.2) can be solved immediately and yields
Po(t) = exp{-A0t}
for t > 0.
(1.4)
Define Sk as the time between the kth and the (k + 1)st birth, so that
n-1
S, :!s: t<S;
The random variables Sk are called the "sojourn times" between births,
and
k-1
W=
= the time at which the kth birth occurs.
i=O
We have already seen that Po(t) = exp{-Aot}. Therefore,
Pr{S0 <- t} = 1 - Pr(X(t) = 0) = I - exp(-A0t);
1.
Pure Birth Processes
337
i.e., So has an exponential distribution with parameter Ao. It may he deduced
from Postulates (1) through (4) that Sk, k > 0, also has an exponential distribution with parameter Ak and that the S;'s are mutually independent.
This description characterizes the pure birth process in terms of its sojourn times, in contrast to the infinitesimal description corresponding to
(1.1).
To solve the differential equations of (1.2) recursively, introduce Q,,(t)
=
for n = 0, 1, .... Then
Q;,(t) = A,,eA,1F,(t) +
P'(t)]
=
[using (1.2)].
Integrating both sides of these equations and using the boundary condition
0 for n ? 1 gives
r
f en°AA
.P,-.(x) dx,
0
or
1
JeI-1(x) dx,
P,(t)
n = 1, 2 , ....
(1.5)
0
It is now clear that all Pk(t) ? 0, but there is still a possibility that
7,
=0
1.
To secure the validity of the process, i.e., to assure that
all t, we must restrict the Ak according to the following:
x
=0
P(t) = 1
if and only if
1
1 for
(1.6)
The intuitive argument for this result is as follows: The time Sk between
consecutive births is exponentially distributed with a corresponding parameter Ak. Therefore, the quantity I 1/A,, equals the expected time before
is the
the population becomes infinite. By comparison, 1 - Y.;,=o
probability that X(t) = oo.
338
Continuous Time Markov Chains
VI
If I A < oc, then the expected time for the population to become infinite is finite. It is then plausible that for all t > 0 the probability that
X(t) = 00 is positive.
When no two of the birth parameters A,,, A,.... are equal, the integral
equation (1.5) may be solved to give the explicit formula
PO(t) =
1
I
e- Ai',
AO - A,
Ao
A,
1
e - A"' +
AO
/
and
Pr{X(t) = nI X(0) = 0)
for n > 1,
= AO ... A,,_I[Bo.,,e-A,,' + ... + B,,.,,e-A'']
(1.8)
where
1
Bo.,, = (AI _ AO) ...
(All - AX
(1.9)
1
Bk',,
(Ao
AA.) ...A._(I
AA.)(AA.+I
AA.)
- - -
l(A,, - A.A)
for 0 < k < n,
and
B,i.,, =
1
(Ao -
(A,,-I Ak when j 0 k by assumption, the denominator in (1.9) does
Because A;
not vanish, and Bk.,, is well-defined.
We will verify that P,(t), as given by (1.7), satisfies (1.5). Equation
(1.4) gives Po(t) = e-'°'. We next substitute this in (1.5) when n = 1,
thereby obtaining
P, (t) =
Aoe-A
, J eA,` e-A°.e a
0
=
= AO
AI)
1
A, - A.
in agreement with (1.7).
e
I[1 _ e
All, +
1
A - AI
I,
e-
1.
Pure Birth Processes
339
The induction proof for general n involves tedious and difficult algebra.
The case n = 2 is suggested as a problem.
1.3.
The Yule Process
The Yule process arises in physics and biology and describes the growth
of a population in which each member has a probability f3h + o(h) of giving birth to a new member during an interval of time of length h (/3 > 0).
Assuming independence and no interaction among members of the population, the binomial theorem gives
Pr(X(t + h) - X(t) = 1I X(t) = n} = (n)[/3h + o(h)][l - f3h + o(h)]"-'
= n f3h +
i.e., for the Yule process the infinitesimal parameters are A,, = no. In
words, the total population birth rate is directly proportional to the population size, the proportionality constant being the individual birth rate /3.
As such, the Yule process forms a stochastic analogue of the determinis-
tic population growth model represented by the differential equation
dy/dt = ay. In the deterministic model, the rate dy/dt of population growth
is directly proportional to population size y. In the stochastic model, the
infinitesimal deterministic increase dy is replaced by the probability of a
unit increase during the infinitesimal time interval dt. Similar connections
between deterministic rates and birth (and death) parameters arise frequently in stochastic modeling. Examples abound in this chapter.
The system of equations (1.2) in the case that X(0) = 1 becomes
(n -
P,',(t) _ -
n = 1, 2, ...
under the initial conditions
P,(0) = 1,
0,
n = 2, 3. ... .
Its solution is
P(t) = e-1(1 - e-1)",
n
1,
(1.10)
as may be verified directly. We recognize (1.10) as the geometric distribution 1, (3.5) with p = e-'*.
340
Continuous Time Markov Chains
VI
The general solution analogous to (1.8) but for pure birth processes
starting from X(0) = 1 is
n > 1.
P,ft) = A, ... A,,-. [B1.,,e-A'' + ...
(1.11)
When A,, = /3n, we will show that (1.11) reduces to the solution given in
(1.10) for a Yule process with parameter P. Then
1
Bl.u =
(A,-AI)(A3-A1)...(A,,
- AI)
1
R I(1)(2) ... (n - 1)
P"-'(n - 1)!'
1
B,,,, = (A,
- A,)(A3 - A) ... (A,, - A,)
I(-1)(1)(2) ... (n - 2)
-1
2)!'
and
1
(AI - Ak) ... (Ak-I - Ak)(Ak+l - Ak) ... (An - Ak)
(_ 1)k-I
P"-'(k - 1)!(n - k)!
Thus, according to (1.11),
P"-'(n - 1)!(B,.,,e-l1 + ... + B,,.,,e
P,(t) =
11
_
k=1
(n - 1)!
k - 1)!(n - k)i (-1)k le k/tr
(n - 1)!
j=0 J!(n - 1 - J)!
11-
= e -O(1 - e')" -I
which establishes (1.10).
(-e-13r)
[see I, (6.17)],
341
Exercises
Exercises
1.1. A pure birth process starting from X(O) = 0 has birth parameters
for n = 0, 1, 2, 3.
k = 1, A = 3, A, = 2, A, = 5. Determine
1.2. A pure birth process starting from X(O) = 0 has birth parameters
,, = 1, A, = 3, A, = 2, A, = 5. Let W, be the random time that it takes the
process to reach state 3.
(a) Write W, as a sum of sojourn times and thereby deduce that the
mean time is E[W] = 6.
(b) Determine the mean of W, + W, + W.
(c) What is the variance of W,?
1.3. A population of organisms evolves as follows. Each organism exists, independent of the other organisms, for an exponentially distributed
length of time with parameter 0, and then splits into two new organisms,
each of which exists, independent of the other organisms, for an exponentially distributed length of time with parameter 0, and then splits into
two new organisms, and so on. Let X(t) denote the number of organisms
existing at time t. Show that X(t) is a Yule process.
1.4. Consider an experiment in which a certain event will occur with
probability ah and will not occur with probability 1 - ah, where a is a
fixed positive parameter and h is a small (h < 1/a) positive variable. Suppose that n independent trials of the experiment are carried out, and the
total number of times that the event occurs is noted. Show that
(a) The probability that the event never occurs during the n trials is
1 - nah + o(h);
(b) The probability that the event occurs exactly once is nah + o(h);
(c) The probability that the event occurs twice or more is o(h).
Hint:
Use the binomial expansion
(1 - ah)" = 1 - nah +
n(n
- 1)
2
(ah)2 -
1.5. Using equation (1.10), calculate the mean and variance for the Yule
process where X(0) = 1.
342
VI
Continuous Time Markov Chains
1.6. Operations 1, 2, and 3 are to be performed in succession on a major
piece of equipment. Operation k takes a random duration Sk that is exponentially distributed with parameter Ak for k = 1, 2, 3, and all operation
times are independent. Let X(t) denote the operation being performed at
time t, with time t = 0 marking the start of the first operation. Suppose
that A, = 5, A, = 3, and A, = 13. Determine
(a) P,(t) = Pr{X(t) = 1).
(b) P,(t) = Pr{X(t) = 2).
(c) P,(t) = Pr{X(t) = 3}.
Problems
1.1. Let X(t) be a Yule process that is observed at a random time U,
where U is uniformly distributed over [0, 1). Show that Pr{X(U) = k} _
pk/(/3k) for k = 1, 2,
Hint:
. . .
,
with p = I - e-a.
Integrate (1.10) over t between 0 and 1.
1.2. A Yule process with immigration has birth parameters Ak = a + k/3
for k = 0, 1, 2,
Here a represents the rate of immigration into the
....
population, and 6 represents the individual birth rate. Supposing that
X(0) = 0, determine P(t) for n = 0, 1, 2, ... .
1.3. Consider a population comprising a fixed number N of individuals.
Suppose that at time t = 0 there is exactly one infected individual and
N - 1 susceptible individuals in the population. Once infected, an individual remains in that state forever. In any short time interval of length h,
any given infected person will transmit the disease to any given susceptible person with probability ah + o(h). (The parameter a is the individual
infection rate.) Let X(t) denote the number of infected individuals in the
population at time t ? 0. Then X(t) is a pure birth process on the states
0, 1, . . . , N. Specify the birth parameters.
1.4. A new product (a "Home Helicopter" to solve the commuting problem) is being introduced. The sales are expected to be determined by both
media (newspaper and television) advertising and word-of-mouth advertising, wherein satisfied customers tell others about the product. Assume
that media advertising creates new customers according to a Poisson
process of rate a = 1 customer per month. For the word-of-mouth adver-
343
Problems
tising, assume that each purchaser of a Home Helicopter will generate
sales to new customers at a rate of 0 = 2 customers per month. Let X(t)
be the total number of Home Helicopter customers up to time t.
(a) Model X(t) as a pure birth process by specifying the birth parameters Ak, for k = 0, 1, ... .
(b) What is the probability that exactly two Home Helicopters are sold
during the first month?
1.5. Let W. be the time to the kth birth in a pure birth process starting
from X(0) = 0. Establish the equivalence
Pr{W1 > t, W, > t.+ s} =
P,(s)}.
From this relation together with equation (1.7), determine the joint den-
sity for W, and W,, and then the joint density of S = W, and S, _
W,-W,.
1.6. A fatigue model for the growth of a crack in a discrete lattice
proposes that the size of the crack evolves as a pure birth process with
parameters
Ak = (1 + k)''
for k = 1, 2.
... .
The theory behind the model postulates that the growth rate of the crack
is proportional to some power of the stress concentration at its ends, and
that this stress concentration is itself proportional to some power of
1 + k, where k is the crack length. Use the sojourn time description to deduce that the mean time for the crack to grow to infinite length is finite
when p > 1, and that therefore the failure time of the system is a welldefined and finite-valued random variable.
1.7. Let A0, A, and A, be the parameters of the independent exponentially distributed random variables S,, S, and S,. Assume that no two of
the parameters are equal.
(a) Verify that
Pr{So > t} = e-",,,
Pr{S + S, > t} _
'
e a',
344
VI
Continuous Time Markov Chains
and evaluate in similar terms
Pr{Sa+S,+S2>t}.
(b) Verify equation (1.8) in the case that n = 2 by evaluating
P,(t)=Pr{X(t)=2}=Pr{So+S,+S,>t}-Pr{S0+S,>t}.
1.8.
Let N(t) be a pure birth process for which
Pr{an event happens in (t, t + h)IN(t) is odd} = ah + o(h),
Pr{an event happens in (t, t + h)IN(t) is even} _ 6h + o(h),
where o(h)/h -* 0 as h 1 0. Take N(O) = 0. Find the following
probabilities:
Po(t) = Pr{N(t) is even};
Hint:
P,(t) = Pr(N(t) is odd}.
Derive the differential equations
Po(t) = aP,(t) - /3Po(t) and
P,(t) = -aP,(t) + /3Po(t),
and solve them by using Po(t) + P,(t) = 1.
1.9.
Under the conditions of Problem 1.8, determine E[N(t)].
1.10. Consider a pure birth process on the states 0, 1, . . . , N for which
A, = (N - k)A for k = 0, 1, . . . , N. Suppose that X(0) = 0. Determine
Pr{X(t) = n} for n = 0, 1, and 2.
1.11. Beginning with Pa(t) = e-a°' and using equation (1.5), calculate
P,(t), P,(t), and P,(t) and verify that these probabilities conform with equation (1.7), assuming distinct birth parameters.
1.12 Verify that P,(t), as given by (1.8), satisfies (1.5) by following the
calculations in the text that showed that P,(t) satisfies (1.5).
1.13.
when all birth parameters are the same
Using (1.5), derive
constant A and show that
(At)"e-a'
P (t) =
n=0,1, ...
n!
Thus, the postulates of Section 1.1 serve to define the Poisson processes.
2.
2.
Pure Death Processes
345
Pure Death Processes
Complementing the increasing pure birth process is the decreasing pure
death process. It moves successively through states N, N - 1, . . . , 2, 1
and ultimately is absorbed in state 0 (extinction). The process is specified
by the death parameters µk > 0 for k = 1, 2, . . . , N, where the sojourn
time in state k is exponentially distributed with parameter µk, all sojourn
times being independent. A typical sample path is depicted in Figure 2.1.
X(t)
N
SN
SN- 1
N-1
N-2
2
SI
1
WI
Figure 2.1
W,
t
A typical sample path of a pure death process, showing the
sojourn times S , ,.. . , SI and the waiting times W, W2,. . . , WN.
Alternatively, we have the infinitesimal description of a pure death
process as a Markov process X(t) whose state space is 0, 1, ... , N and for
which
(i) Pr(X(t + h) = k - 1I X(t) = k} = µkh + o(h), k = 1, ... , N;
(ii) Pr{X(t + h) = kjX(t) = k} = 1 - µkh + o(h), k = 1, ... , N; (2.1)
(iii) Pr(X(t + h) > kjX(t) = k} = 0, k = 0, 1, . . . , N.
The parameter I-µ, is the "death rate" operating or in effect while the
process sojourns in state k. It is a common and useful convention to assign
AO=0.
346
Continuous Time Markov Chains
VI
When the death parameters µ,, µ,, ... , µv are distinct, that is, p 0 g,
if j k, then we have the explicit transition probabilities
P,,(t) = e-",';
and for n < N,
Pr{X(t) = nI X(0) = N)
(2.2)
where
1
µk) ... (AA +I - P-)(P- - I 2.1.
Ak) ... (A - -k)
The Linear Death Process
As an example, consider a pure death process in which the death rates are
proportional to population size. This process, which we will call the linear death process, complements the Yule, or linear birth, process. The parameters are µk = ka, where a is the individual death rate in the population. Then
1
A») ... (A", - A")
(!L"' -
1
aN-"-'(N - n)(N - n - 1) ... (2)(1)'
1
(Av -
(A,,12 -
A,,+])
1
n - 1) ... (1)(-1),
1
(AN - Pk) ... (11k+1 - Pk)(P-k-I - N-k) ... (A - Pk)
1
= ai(N-k)...(1)(-1)(-2)...(n-k)
1
k)!(k - n)!
2.
Pure Death Processes
347
Then
N
µ4l
aA`-,1
I
N!
I
k=n
N
n! k=,,
e-ka1
aN-n
)(-1)k -(N - k)!(k - n)!
-n
j=0 (N - 1ye-j°'
n -,)!j!
N!
= n!
N!
n!(N - n)!
e"'(1 -
(2.3)
e-a1)N-
n = 0, ... , N.
Let T be the time of population extinction. Formally, T = min{t ? 0;
X(t) = 0). Then T:5 t if and only if X(t) = 0, which leads to the cumulative distribution function of T via
FT(t) = Pr{T < t} = Pr{X(t) = 0)
= P0(t) _ (1 - e-tt1)N,
t ? 0.
(2.4)
The linear death process can be viewed in yet another way, a way that
again confirms the intimate connection between the exponential distribution and a continuous time parameter Markov chain. Consider a population consisting of N individuals, each of whose lifetimes is an independent
exponentially distributed random variable with parameter a. Let X(t) be
the number of survivors in this population at time t. Then X(t) is the linear pure death process whose parameters are µk = ka for k = 0, 1, ... ,
N. T o help understand this connection, let 6 6,, ... , ,,, denote the times
of death of the individuals labeled 1, 2, . . . , N, respectively. Figure 2.2,
on the next page, shows the relation between the individual lifetimes
,, ... , 6N and the death process X(t).
The sojourn time in state N, denoted by S, equals the time of the earliest death, or S, = min( 6,, . . . , 6A. } . Since the lifetimes are independent
and have the same exponential distribution,
N}>t}
Pr{S,,,>t}
=Pr{6,>t,...,6, >t}
_ [Pr{6, > t}]'
e Na t
348
VI
Continuous Time Markov Chains
N=6
{
EN
5
I
4
3
2
I
I
Time of
Death
X 0)
N=6
S
5
S4
4
S3
3
S2
2
S,
1
W,
W2 W3
W4
W5
t
WN
Figure 2.2 The linear death process. As depicted here, the third individual is
the first to die, the first individual is the second to die, etc.
That is, SN has an exponential distribution with parameter Na. Similar rea-
soning applies when there are k members alive in the population. The
memoryless property of the exponential distribution implies that the remaining lifetime of each of these k individuals is exponentially distributed
with parameter a. Then the sojourn time Sk is the minimum of these k remaining lifetimes and hence is exponentially distributed with parameter
ka. To give one more approach in terms of transition rates, each individual in the population has a constant death rate of a in the sense that
<t+h)
Pr(t<6, <t+hit< ,} =
ear - e-«(r+h)
e ar
=1-eah
= ah + o(h)
as h 1 0.
Pure Death Processes
2.
349
If each of k individuals alive in the population at time t has a constant
death rate of a, then the total population death rate should be ka, directly
proportional to the population size. This shortcut approach to specifying
appropriate death parameters is a powerful and often-used tool of stochastic modeling. The next example furnishes another illustration of its
use.
2.2.
Cable Failure Under Static Fatigue
A cable composed of parallel fibers under tension is being designed to
support a high-altitude weather balloon. With a design load of 1000 kg
and a design lifetime of 100 years, how many fibers should be used in the
cable?
The low-weight, high-strength fibers to be used are subject to static
fatigue, or eventual failure when subjected to a constant load. The higher
the constant load, the shorter the life, and experiments have established a
linear plot on log-log axes between average failure time and load that is
shown in Figure 2.3.
The relation between mean life p, and load I that is illustrated in Figure 2.3 takes the analytic form
log, A, = 2 - 40 log,ol.
log (time)
-+--slope = -P= -40
log (100 yrs)
log (1 kg)
log (load)
Figure 2.3 A linear relation between log mean failure time and log load.
350
VI
Continuous Time Markov Chains
Were the cable to be designed on the basis of average life, to achieve
the 100 year design target each fiber should carry 1 kg. Since the total load
is 1000 kg, N = '-°,°-° = 1000 fibers should be used in the cable.
One might suppose that this large number (N = 1000) of fibers would
justify designing the cable based on average fiber properties. We shall see
that such reasoning is dangerously wrong.
Let us suppose, however, as is the case with many modern high-performance structural materials, that there is a large amount of random scatter
of individual fiber lifetimes about the mean. How does this randomness
affect the design problem?
Some assumption must be made concerning the probability distribution
governing individual fiber lifetimes. In practice, it is extremely difficult to
gather sufficient data to determine this distribution with any degree of certainty. Most data do show, however, a significant degree of skewness, or
asymmetry. Because it qualitatively matches observed data and because it
leads to a pure death process model that is accessible to exhaustive analysis, we will assume that the probability distribution for the failure time T
of a single fiber subjected to the time-varying tensile load l(t) is given by
t
Pr{T <_ t} = I - exp 1-f K[l(s)] ds
,
t ? 0.
0
This distribution corresponds to a failure rate, or hazard rate, of
r(t) = K[l(t)] wherein a single fiber, having not failed prior to time t and
carrying the load 1(t), will fail during the interval (t, t + At] with
probability
Pr{t < T < t + WIT > t} = K[l(t)] At + o(At).
The function K[l], called the breakdown rule, expresses how changes in
load affect the failure probability. We are concerned with the power law
breakdown rule in which K[l] = 1'3/A for some positive constants A and /3.
Assuming power law breakdown, under a constant load l(t) = 1, the single
fiber failure time is exponentially distributed with mean µ, = E[TJl] _
1/K[l] = Al-1. A plot of mean failure time versus load is linear on log-log
axes, matching the observed properties of our fiber type. For the design
problem we have /3 = 40 and A = 100.
2.
Pure Death Processes
351
Now place N of these fibers in parallel and subject the resulting bundle
or cable to a total load, constant in time, of NL, where L is the nominal
load per fiber. What is the probability distribution of the time at which the
cable fails? Since the fibers are in parallel, this system failure time equals
the failure time of the last fiber.
Under the stated assumptions governing single-fiber behavior, X(t), the
number of unfailed fibers in the cable at time t, evolves as a pure death
process with parameters A, = kK[NLIk] for k = 1, 2, ... , N. Given
X(t) = k surviving fibers at time t and assuming that the total bundle load
NL is shared equally among them, then each carries load NL/k and has a
corresponding failure rate of K[NL/k]. As there are k such survivors in the
bundle, the bundle, or system, failure rate is µk = kK[NL/k] as claimed.
It was mentioned earlier that the system failure time was W,,,, the waiting time to the Nth fiber failure. Then Pr{W,,, S t} = Pr{X(t) = 0} = P,(t),
where
is given explicitly by (2.2) in terms of µ,, ... , µ,,,. Alternatively, we may bring to bear the sojourn time description of the pure death
process and, following Figure 2.1, write
+5,,
S, are independent exponentially distributed random
variables and Sk has parameter µk = kK[NL/k] = k(NLIk)R/A. The mean
system failure time is readily computed to be
where SN,
+ E[S,]
E[W,] = E[S,,,] +
= AL-R
N(2.5)
(k)"
x-,
k
)R '( )
The sum in the expression for E[W,,] seems formidable at first glance, but
a very close approximation is readily available when N is large. Figure 2.4
compares the sum to an integral.
From Figure 2.4 on the next page we see that
(k)(i)N
k_, N
N
xR- 'dx=
352
VI
Continuous Time Markov Chains
llR - '
(N/
0
1
2
3
NNN
H1H
N-11
x
N
N
Figure 2.4 The sum Yk , (kIN)a-'(1/N) is a Riemann approximation
to f xs-' dx = 1//3.
Indeed, we readily obtain
1
/3
=f1
N (kl1
(-) :5
xa'dx I
()E( 1 +
N)R
-
1+r1/N
xR-'dx
(N)°I.
When N = 1000 and /3 = 40, the numerical bounds are
k
(40)(1.0408),
N) (N)
which shows that the integral determines the sum to within about 4
percent.
Substituting 1/$ for the sum in (2.5) gives the average cable life
A
E[WN]
to be compared with the average fiber life of
A
µrLp.
353
Exercises
That is, a cable lasts only about 1//3 as long as an average fiber under an
equivalent load. With A = 100, /3 = 40, and N = 1000, the designed cable
would last, on the average, 100/[40(1)40] = 2.5 years, far short of the desired life of 100 years. The cure is to increase the number of fibers in the
cable, thereby decreasing the per fiber load. Increasing the number
of fibers from N to N' decreases the nominal load per fiber from
L to L' = NL/N'. To achieve parity in fiber-cable lifetimes we equate
A _
L[3
A
/3(NLIN' )s'
or
N' = N/3'
.
For the given data, this calls for N' = 1000(40)" = 1097 fibers. That is,
the design lifetime can be restored by increasing the number of fibers in
the cable by about 10 percent.
Exercises
2.1. A pure death process starting from X(0) = 3 has death parameters
µ0 = 0, µ, = 3, p, = 2, µ, = 5. Determine
for n = 0, 1, 2, 3.
2.2. A pure death process starting from X(0) = 3 has death parameters
Ao = 0, µ, = 3,A2= 2,A3 = 5. Let W, be the random time that it takes
the process to reach state 0.
(a) Write W, as a sum of sojourn times and thereby deduce that the
mean time is E[W,] = o.
(b) Determine the mean of W, + W, + W,.
(c) What is the variance of W,?
2.3.
Give the transition probabilities for the pure death process described by X(0) = 3, µ, = 1, p2 = 2, and µ, = 3.
2.4. Consider the linear death process (Section 2.1) in which
X(0) = N = 5 and a = 2. Determine Pr(X(t) = 2}.
Hint:
Use equation (2.3).
354
Continuous Time Markov Chains
VI
Problems
2.1. Let X(t) be a pure death process starting from X(O) = N. Assume
that the death parameters are µ,, µ,, ... , µ,,,. Let T be an independent exponentially distributed random variable with parameter 0. Show that
N
Pr{X(T) = 0}
i=1 µ,+ 0
2.2. Let X(t) be a pure death process with constant death rates µ,. = 0
for k = 1, 2, . . ., N. If X(O) = N, determine
Pr(X(t) = n} for
n=0,1,...,N.
A pure death process X(t) with parameters AI, p2, ... starts at
X(O) = N and evolves until it reaches the absorbing state 0. Determine the
mean area under the X(t) trajectory.
2.3.
Hint:
This is E[W, + W +
+ WA,J.
2.4. A chemical solution contains N molecules of type A and M molecules of type B. An irreversible reaction occurs between type A and B
molecules in which they bond to form a new compound AB. Suppose that
in any small time interval of length h, any particular unbonded A molecule
will react with any particular unbonded B molecule with probability
Oh + o(h), where 0 is a reaction rate. Let X(t) denote the number of unbonded A molecules at time t.
(a) Model X(t) as a pure death process by specifying the parameters.
(b) Assume that N < M, so that eventually all of the A molecules become bonded. Determine the mean time until this happens.
2.5. Consider a cable composed of fibers following the breakdown rule
K[1] = sinh(l) = ;(e' - e-') for l L- 0. Show that the mean cable life is
given by
E[W^,J = 7, {k sinh(NL/k)} ' =
k=I
j
N sinh(k/N)1
I
k=1 l
- J {x sinh(L/x) } -' dx.
0
`
N
3.
Birth and Death Processes
355
2.6. Let T be the time to extinction in the linear death process with parameters X(O) = N and a (see Section 2.1).
(a) Using the sojourn time viewpoint, show that
aLN+N1
E[T]
+...+
1
1J
(b) Verify the result of (a) by using equation (2.4) in
x
E[T] = J Pr{T > t) dt = J [I - FT(t)] dt.
0
Hint:
3.
0
Let y = 1 - e-°".
Birth and Death Processes
An obvious generalization of the pure birth and pure death processes discussed in Sections 1 and 2 is to permit X(t) both to increase and to decrease. Thus if at time t the process is in state n, it may, after a random sojourn time, move to either of the neighboring states n + 1 or n - 1. The
resulting birth and death process can then be regarded as the continuoustime analogue of a random walk (III, Section 5.3).
Birth and death processes form a powerful tool in the kit of the stochastic modeler. The richness of the birth and death parameters facilitates
modeling a variety of phenomena. At the same time, standard methods of
analysis are available for determining numerous important quantities such
as stationary distributions and mean first passage times. This section and
later sections contain several examples of birth and death processes and
illustrate how they are used to draw conclusions about phenomena in a variety of disciplines.
3.1.
Postulates
As in the case of the pure birth processes, we assume that X(t) is a Markov
process on the states 0, 1, 2, ... and that its transition probabilities Pi(t)
are stationary; i.e.,
P;(t) = Pr{X(t + s) = jjX(s) = i}
for all s ? 0.
VI
356
Continuous Time Markov Chains
In addition, we assume that the Pj(t) satisfy
(1) P,;,.,(h) = A;h + o(h) as h L 0, i ? 0;
(2) P,;_,(h) = µ;h + o(h) as h .1 0, i ? 1;
(3) P.;(h) = 1 - (A; + µ)h + o(h) as h 10, i ? 0;
(4) Pj(O) = 5;;;
(5) Ao=O,Ao>O,µ;,A;>O,i= 1,2,....
The o(h) in each case may depend on i. The matrix
_AO
µ,
A=
0
0
Ao
0
-(A, + µ,)
A,
µ,
-(A,+µ,)
/0
µ,
0
0
...
...
A,
,
(3.1)
-(A, + µ,) .. .
is called the infinitesimal generator of the process. The parameters A; and
µ; are called, respectively, the infinitesimal birth and death rates. In Postulates (1) and (2) we are assuming that if the process starts in state i, then
in a small interval of time the probabilities of the population increasing or
decreasing by 1 are essentially proportional to the length of the interval.
Since the Pj(t) are probabilities, we have Pj(t) ? 0 and
Pj(t) <- 1.
(3.2)
j=0
Using the Markov property of the process we may also derive the socalled Chapman-Kolmogorov equation
Pj(t + s) = Z, PR(t)PR;(s).
(3.3)
k=0
This equation states that in order to move from state i to state j in time
t + s, X(t) moves to some state k in time t and then from k to j in the
remaining time s. This is the continuous-time analogue of formula (2.2)
in III.
So far we have mentioned only the transition probabilities Pj(t). In
order to obtain the probability that X(t) = n, we must specify where the
process starts or more generally the probability distribution for the initial
state. We then have
3. Birth and Death Processes
357
Pr{X(t) = n} _
q;
3.2.
=
Pr{X(0) = i}.
Sojourn Times
With the aid of the preceding assumptions we may calculate the distribution of the random variable Si, which is the sojourn time of X(t) in state i;
that is, given that the process is in state i, what is the distribution of the
time Si until it first leaves state i? If we let
Pr{S, ? t} = G;(t),
it follows easily by the Markov property that as h 10,
Gi(t + h) = G;(t)G,(h) = G;(t)[R,;(h) + o(h)]
= G,(t)[1 - (A, + µ;)h] + o(h),
or
Gi(t + h) - G,(t)
h
_ -(A, + µ)G;(t) + o(1),
so that
G(t) _ -(A1 + µ,)G,(t).
(3.4)
If we use the conditions G;(0) = 1, the solution of this equation is
G;(t) = exp[-(A, + µ)t];
i.e., Si follows an exponential distribution with mean (A1 + µ;)-'. The
proof presented here is not quite complete, since we have used the intuitive relationship
G,(h) = Pi(h) + o(h)
without a formal proof.
According to Postulates (1) and (2), during a time duration of length h
a transition occurs from state i to i + 1 with probability A,h + o(h) and
from state i to i - 1 with probability µ;h + o(h). It follows intuitively that,
given that a transition occurs at time t, the probability that this transition
358
VI
Continuous Time Markov Chains
is to state i + I is A;/(A; + µ;) and to state i - 1 is µ;l(A; + A j). The rigorous demonstration of this result is beyond the scope of this book.
It leads to an important characterization of a birth and death process,
however, wherein the description of the motion of X(t) is as follows: The
process sojourns in a given state i for a random length of time whose distribution function is an exponential distribution with parameter (A; + µ;).
When leaving state i the process enters either state i + I or state i - 1 with
probabilities A;/(A; + µ;) and µ;/(A; + µ;), respectively. The motion is
analogous to that of a random walk except that transitions occur at random times rather than at fixed time periods.
The traditional procedure for constructing birth and death processes is
to prescribe the birth and death parameters { A;, µ;) =o and build the path
structure by utilizing the preceding description concerning the waiting
times and the conditional transition probabilities of the various states. We
determine realizations of the process as follows. Suppose X(O) = i; the
particle spends a random length of time, exponentially distributed with
parameter (A; + A,), in state i and subsequently moves with probability
A;/(A; + p,) to state i + 1 and with probability p,/(A; + µ;) to state i - 1.
Next the particle sojourns a random length of time in the new state and
then moves to one of its neighboring states, and so on. More specifically,
we observe a value t, from the exponential distribution with parameter
(A; + p) that fixes the initial sojourn time in state i. Then we toss a coin
with probability of heads p; = A;/(A; + µ;). If heads (tails) appears we
move the particle to state i + 1 (i - 1). In state i + 1 we observe a value
t, from the exponential distribution with parameter (A;+, + µ;+,) that fixes
the sojourn time in the second state visited. If the particle at the first transition enters state i - 1, the subsequent sojourn time t; is an observation
from the exponential distribution with parameter (A;_, + µ;_,). After the
second wait is completed, a Bernoulli trial is performed that chooses the
next state to be visited, and the process continues in the same way.
A typical outcome of these sampling procedures determines a realization of the process. Its form might be, for example,
X(t)
i,
for 0<t<t
i + 1,
fort, <t<t,+t,,
i,
fort,
+t,<t<t,+t,+t,,
3.
Birth and Death Processes
359
Thus by sampling from exponential and Bernoulli distributions appropriately, we construct typical sample paths of the process. Now it is possible
to assign to this set of paths (realizations of the process) a probability
measure in a consistent way so that P;(t) is determined satisfying (3.2) and
(3.3). This result is rather deep, and its rigorous discussion is beyond the
level of this book. The process obtained in this manner is called the minimal process associated with the infinitesimal matrix A defined in (3.1).
The preceding construction of the minimal process is fundamental,
since the infinitesimal parameters need not determine a unique stochastic
process obeying (3.2), (3.3), and Postulates 1 through 5 of Section 3.1. In
fact, there could be several Markov processes that possess the same infin-
itesimal generator. Fortunately, such complications do not arise in the
modeling of common phenomena. In the special case of birth and death
processes for which A0 > 0, a sufficient condition that there exists a
unique Markov process with transition probability function P;(t) for
which the infinitesimal relations (3.2) and (3.3) hold is that
0,,
(3.5)
where
00= 1,
µiµ2 ... An
n= 1,2,....
In most practical examples of birth and death processes the condition
(3.5) is met, and the birth and death process associated with the prescribed
parameters is uniquely determined.
3.3.
Differential Equations of Birth and Death Processes
As in the case of the pure birth and pure death processes, the transition
probabilities P;(t) satisfy a system of differential equations known as the
backward Kolmogorov differential equations. These are given by
Poi(t) _
AoP,,j(t) + k P,j(t),
P;;(t) = I-p,P-'.;(t) - R + WPM) + A,P+1.;(t),
and the boundary condition P;(0) = 5;;.
(3.6)
i ? 1,
360
VI
Continuous Time Markov Chains
To derive these we have, from equation (3.3),
x
Pj(t + h) = 7, Pk(h)Pj(t)
k=0
= P.i-.(h)P-..j(t) + P.i(h)Pj(t) +.P.i+.(h)P+..j(t)
(3.7)
+ Y-kPik(h)Pkj(t),
where the last summation is over all k 0 i - 1, i, i + 1. Using Postulates
1, 2, and 3 of Section 3.1, we obtain
II F (h)Pj(t) C 4k Pk(h)
= 1 - [P,.i(h) + P.i-.(h) + P.i+.(h)]
= I - [1 - (Ai + µi)h + o(h) + µih + o(h) + Aih + o(h)]
= o(h),
so that
Pj(t + h) = µihP_,,j(t) + [1 - (A, + p.)h]Pj(t) + AihPi+,.j(t) + o(h).
Transposing the term P1
(t) to the left-hand side and dividing the equation
letting
by h, we obtain, after
h 10(,
pp
P;;(t) = /L.F-..j(t) - (Ai + p..)rij(t) + AiP+1.j(t)
The backward equations are deduced by decomposing the time interval
(0, t + h), where h is positive and small, into the two periods
(0, h),
(h, t + h)
and examining the transition in each period separately. In this sense the
backward equations result from a "first step analysis," the first step being
over the short time interval of duration h.
A different result arises from a "last step analysis," which proceeds by
splitting the time interval (0, t + h) into the two periods
(0, t),
(t, t + h)
and adapting the preceding reasoning. From this viewpoint, under mor
stringent conditions we can derive a further system of differential
equations
3.
Birth and Death Processes
361
Pio(t) = -AoI.o(t) + p.1Pj.1(t),
(3.8)
P,j(t) =
(Aj + µj)Pj(t) + Aj+IP.j+1(t),
with the same initial condition Pj(0) = S;j. These are known as the forward
Kolmogorov differential equations. To derive these equations we interchange t and h in equation (3.7), and under stronger assumptions in addi-
tion to Postulates (1), (2), and (3) it can be shown that the last term is
again o(h). The remainder of the argument is the same as before. The usefulness of the differential equations will become apparent in the examples
that we study in this and the next section.
A sufficient condition that (3.8) hold is that [Pkj(h)]/h = o(l) for k = j,
j - 1, j + 1, where the o(l) term apart from tending to zero is uniformly
bounded with respect to k for fixed j as h - 0. In this case it can be proved
that 14. Pk(t)Pkj(h) = o(h).
Example Linear Growth with Immigration A birth and death process
is called a linear growth process if A,, = An + a and µ = An with A > 0,
µ > 0, and a > 0. Such processes occur naturally in the study of biological reproduction and population growth. If the state n describes the current population size, then the average instantaneous rate of growth is
An + a. Similarly, the probability of the state of the process decreasing by
one after the elapse of a small duration of time h is pnh + o(h). The factor An represents the natural growth of the population owing to its current
size, while the second factor a may be interpreted as the infinitesimal rate
of increase of the population due to an external source such as immigration. The component µn, which gives the mean infinitesimal death rate of
the present population, possesses the obvious interpretation.
If we substitute the above values of A,, and µ in (3.8), we obtain
Pia(t) = -aJo(t) + µl'1(t),
P',.j(t) = [A(j - 1) + a]P.j-1(t) - [(A + p)j + a]Pj(t)
+ pA + 1)I'.j+1(t),
j ? 1.
Now, if we multiply the jth equation by j and sum, it follows that the expected value
E[X(t)] = M(t) = L jPj(t)
j=1
satisfies the differential equation
362
VI
Continuous Time Markov Chains
M'(t) = a + (A - µ)M(t),
with initial condition M(0) = i, if X(O) = i. The solution of this equation
is
M(t)=at+i
ifA=µ,
and
M(t) =
a
A-µ
{e'a-"" - 1} + ie`A-'A)t
if A 0 µ.
(3.9)
The second moment, or variance, may be calculated in a similar way. It is
interesting to note that M(t) - oo as t - x if A ? µ, while if A < µ, the
mean population size for large t is approximately
a
µ-11
These results suggest that in the second case, wherein A < µ, the population stabilizes in the long run in some form of statistical equilibrium.
Indeed, it can be shown that a limiting probability distribution {7r;} exists
for which lim, P, (t) _ i7 , j = 0, 1,
Such limiting distributions for
general birth and death processes are the subject of Section 4.
....
Example The Two State Markov Chain Consider a Markov chain
{X(t)} with state 10, 1) whose infinitesimal matrix is
AR
0
1
R
(3.10)
The process alternates between states 0 and 1. The sojourn times in state
0 are independent and exponentially distributed with parameter a. Those
in state 1 are independent and exponentially distributed with parameter P.
This is a finite state birth and death process for which A, = a, A, = 0,
µo = 0 and µ, _ /3. The first Kolmogorov forward equation in (3.8)
becomes
P ,(t) = -aP.(t) + $P0,(t).
Now, PP,(t) = 1 - P,(t), which placed in (3.11) gives
(3.11)
3.
363
Birth and Death Processes
Pm(t) = /3 - (a + /3)A(t).
Let Q00(t) = e(a+p)'POO(t). Then
dQ.O(t)
=
(a + R)e(a+R),POO(t)
dt
= e(a+R),[Pa)(t) + (a + l3)Poo(t)]
= ae(a+p),
,
N
which can be integrated immediately to yield
QO,(t) = 0 J e(`9)` dt + C
(
/
\a+/3)
e(a+p), + C.
The initial condition Q,(0) = 1 determines the constant of integration to
be C = a/(a + /3). Thus
and
Pm(t)
+
a
f3
+
Re-c«+p,
(3.13a)
a
Since P0#) = 1 - Pm(t), we have
PA)(t) =
a
a
a+/3
a+/3
'
(3.13b)
and by symmetry,
a
+
pi(t) = a 0
R-
p)(t) =
a+(3
R
a+
a 0
e
(a+p),
(3.13c)
Re-(a+p),.
(3.13d)
Vf
364
Continuous Time Markov Chains
These transition probabilities assume a more succinct form if we reparametrize according to 7r = a/(a + /3) and r = a + P. Then
Pm(t) _ (1 - IT) + 7re-7',
(3.14a)
Pa, (t) = Ir -
(3.14b)
ITe-71,
P10(t) _ (1 - Ir) - (1 - ir)e-T',
(3.14c)
and
IT + (1 - 7r)e-r'.
(3.14d)
Observe that
lim P01(t) = lim
Ir,
so that ir is the long run probability of finding the process in state 1 independently of where the process began. The long run behavior of general
birth and death processes is the subject of the next section.
Exercises
3.1.
Particles are emitted by a radioactive substance according to a Poisson process of rate A. Each particle exists for an exponentially distributed
length of time, independent of the other particles, before disappearing. Let
X(t) denote the number of particles alive at time t. Argue that X(t) is a birth
and death process, and determine the parameters.
3.2. Patients arrive at a hospital emergency room according to° a Poisson process of rate A. The patients are treated by a single doctor on a first
come, first served basis. The doctor treats patients more quickly when the
number of patients waiting is higher. An industrial engineering time study
suggests that the mean patient treatment time when there are k patients in
the system is of the form mk =a - /3k/(k + 1), where a and /3 are constants with a > /3 > 0. Let N(t) be the number of patients in the system
at time t (waiting and being treated). Argue that N(t) might be modeled as
a birth and death process with parameters A,, = A for k = 0, 1, . . . and
µ,. = k/m, for k = 0, 1..... State explicitly any necessary assumptions.
365
Problems
3.3. Let {V(t) } be the two state Markov chain whose transition
probabilities are given by (3.14a-d). Suppose that the initial distribution
is (1 - ir, 7r). That is, assume that Pr{ V(0) = 01 = 1 - 7r and
Pr{V(0) = 11 = 7r. In this case, show that Pr{V(t) = 1 } = 7r for all times
t > 0.
Problems
3.1. Let 6,,, n = 0, 1, ... , be a two state Markov chain with transition
probability matrix
F=0
Let {N(t); t ? 0) be a Poisson process with parameter A. Show that
X(t) =
t j 0,
Sev(r),
is a two state birth and death process and determine the parameters Ao and
µ, in terms of a and A.
3.2.
Collards were planted equally spaced in a single row in order to
provide an experimental setup for observing the chaotic movements of the
flea beetle (P. cruciferae). A beetle at position k in the row remains on that
plant for a random length of time having mean mk (which varies with the
"quality" of the plant) and then is equally likely to move right (k + 1) or
left (k - 1). Model the position of the beetle at time t as a birth and death
process having parameters Ak = µk = 1/(2mk) for k = 1, 2, ... , N - 1,
where the plants are numbered 0, 1,
plausible at the ends 0 and N?
. . . ,
N. What assumptions might be
3.3. Let {V(t) } be the two state Markov chain whose transition probabilities are given by (3.14a-d). Suppose that the initial distribution is
(1 - ir,
7r). That is, assume that Pr { V(0) = 0) =
Pr(V(0) = I) = nr. For 0 < s < t, show that
E[V(s)V(t)] = 1r - arPa(t - s),
whence
Cov[V(s), V(t)] = 1r(1 - 7r)e-(°+'3' ` '
1
- rr and
366
VI
Continuous Time Markov Chains
3.4. A Stop-and-Go Traveler The velocity V(t) of a stop-and-go traveler is described by the two state Markov chain whose transition probabilities are given by (3.14a-d). The distance traveled in time t is the integral of the velocity:
f V(u) du.
S(t) =
0
Assuming that the velocity at time t = 0 is V(0) = 0, determine the mean
of S(t). Take for granted the interchange of integral and expectation in
t
f E[V(u)] du.
E[S(t)] =
0
4.
The Limiting Behavior of Birth and
Death Processes
For a general birth and death process that has no absorbing states, it can
be proved that the limits
lim P,.;(t) = ?r; ? 0
!fix
(4.1)
exist and are independent of the initial state i. It may happen that 1r; = 0
for all states j. When the limits 17) are strictly positive, however, and satisfy
= 1,
(4.2)
I=0
they form a probability distribution that is called, naturally enough, the
limiting distribution of the process. The limiting distribution is also a stationary distribution in that
x
1; = X. ir,P,(t),
(4.3)
i=0
which tells us that if the process starts in state i with probability ir;, then
at any time t it will be in state i with the same probability ir;. The proof of
x and use the fact that
(4.3) follows from (3.3) and (4.1) if we let t
x
4.
The Limiting Behavior of Birth and Death Processes
367
The general importance of birth and death processes as models derives
in large part from the availability of standard formulas for determining if
a limiting distribution exists and what its values are when it does. These
formulas follow from the Kolmogorov forward equations (3.8) that were
derived in Section 3.3:
(4.4)
Pi.o(t) _ -A0P.o(t) + Ail'.i(t),
P/i.j(t) = Aj-IPj.j-I(t) - (Aj + Aj)Pij(t) + 1jj+Irr.j+I(t),
1,
oo in
with the initial condition i,(0) = Sij. Now pass to the limit as t
(4.4) and observe first that'\the limit of the right side of (4.4) exists according to (4.1). Therefore, the limit of the left side, the derivatives P;j(t),
exists as well. Since the probabilities are converging to a constant, the
limit of these derivatives must be zero. In summary, passing to the limit in
(4.4) produces
0 = -AoiTo + µ11T1,
(4.5)
0 = Aj-i'7Tj-i - (Aj +
j
/j'j).7Tj + !-_j+i1Tj+i,
1.
The solution to (4.5) is obtained by induction. Letting
00 = 1
and
-
AoA,... A3_,
/LI IJ.2 ... µj
for j ? 1,
(4.6)
we have ir, = Aoirolµ, = O,,7ro. Then, assuming that l'k = 6k7ro for
k = 1,...,j,weobtain
µj+iij+i = (Aj + N-j)Ojir0 - A1_101 ,Tro
= A03ir() + (µ3e3 - AA_,O,_,)iro
= A303iro,
and finally
1Tj+1 = Oj+i7To
In order that the sequence { 7rj } define a distribution, we must have
I.3 7r3 = 1. If 1 03 < x, then we may sum the following,
368
V1
Continuous Time Markov Chains
Ira = 007TO
IT, = 0,1xO
?T2 = 029r0
1 = (I 0k) IT0
0k, and then
to see that IT0 =
e;
7r; = 0;7r0 =
for j = 0, 1,
x
Ik=0 dk
....
(4.7)
If E dk = oo, then necessarily vr0 = 0, and then Tr; = 0;ir0 = 0 for all j, and
there is no limiting distribution (lim,,, P,.;(t) = 0 for all j).
Example Linear Growth with Immigration As described in the
example at the end of Section 3.3, this process has birth parameters
A,, = a + An and death parameters µ = µn for n = 0, 1, . . . , where
A > 0 is the individual birth rate, a > 0 is the rate of immigration into the
population, and µ > 0 is the individual death rate.
Suppose A < µ. It was shown in Section 3.3 that the population mean
M(t) converges to a/(µ - A) as t - oo. Here we will determine the limiting distribution of the process under the same condition A < µ.
Then 00 = 1, 0, = altk, 0, = a(a + A)l[A(21k)], 03 = a(a + A)(a + 2A)/
[µ(2µ)(31x)], and, in general,
a(a + A) . . . [a + (k - 1)A]
Bk
µk(k)!
-
(alA)[(alA) + 1] ... [(alA) + k - 1] / A\k
µ
k!
((a/A)
k
k - l1(µlk
Now use the infinite binomial formula (I, equation (1.71)),
IN+k- 1\
(1 - x)-" _
k
k=0
Ixk
for Ixl < 1,
4.
The Limiting Behavior of Birth and Death Processes
369
to determine that
x
k
k= 0
- 1/\µ) -
(1
-Y
",A)
when A < A. Thus iro = (1 - .Vµ)"', and
,A
R (a/A)[(a/A) + 1 ] ... [(alA) + k - 1 ]
fork > 1.
Example Repairman Models A system is composed of N machines,
of which at most M <_ N can be operating at any one time. The rest are
"spares." When a machine is operating, it operates a random length of
time until failure. Suppose this failure time is exponentially distributed
with parameter A.
When a machine fails, it undergoes repair. At most R machines can be
"in repair" at any one time. The repair time is exponentially distributed
with parameter A. Thus a machine can be in any of four states: (i) operating; (ii) "up," but not operating, i.e., a spare; (iii) in repair; (iv) waiting for
repair. There is a total of N machines in the system. At most M can be operating. At most R can be in repair.
The action is diagrammed in Figure 4.1 on the next page.
Let X(t) be the number of machines "up" at time t, either operating or
spare. Then (we assume) the number operating is min{X(t), M}, and the
number of spares is max{0, X(t) - M}. Let Y(t) = N - X(t) be the number of machines "down." Then the number in repair is min{Y(t), R}, and
the number waiting for repair is max{0, Y(t) - R). The foregoing formulas permit us to determine the number of machines in any category, once
X(t) is known.
Then X(t) is a finite state birth and death process' with parameters
A,,=AXmin{N-n,R}
AR
{A(N-n)
for n = 0, 1,
. . . ,
N - R,
forn =N - R+ 1,...,N,
' The definition of birth and death processes was given for an infinite number of states.
The adjustments in the definitions and analyses for the case of a finite number of states are
straightforward and even simpler than the original definitions, and are left to the reader.
370
VI
Continuous Time Markov Chains
FACTORY
"Spares"
Capacity = M
Failure Rate = p.
000
000000
k
X(t) = Number of machines "up"
Waiting
for
repair
REPAIR SHOP
Capacity = R
Repair Rate = K
000
0000
Y (t) = Number of machines "down"
Figure 4.1
Repairman model.
and
min{n, M}
µn
={µM
forn=0,1,... ,M,
forn=M+ 1,...,N.
It is now a routine task to determine the limiting probability distribution for any values of A, µ, N, M, and R. (See Problems 4.1 and 4.7.) In
terms of the limiting probabilities iro, 7r ... , arv, some quantities of interest are the following:
Average Machines Operating = 7r, + 21r, +
+ M(lrM+
Long Run Utilization =
I
+ Mir
+ ... + 'TV);
Average Machines Operating
Capacity
Tr, + 27r, +
+ Ma
M
+ (IT,r+, + ... + Irv);
4.
The Limiting Behavior of Birth and Death Processes
371
Average Idle Repair Capacity = 1,7r,-,,, + 21rN-R+2 +
+ R7rN.
These and other similar quantities can be used to evaluate the desirability of adding additional repair capability, additional spare machines,
and other possible improvements.
The stationary distribution assumes quite simple forms in certain special cases. For example, consider the special case in which M = N = R.
The situation arises, for instance, when each machine's operator becomes
its repairman upon its failure. Then A,, = A(N - n) and µ = µn for
n = 0, 1, ... , N, and following (4.6), we determine 00 = 1, 0, = AN/µ,
02 _ (AN)A(N - 1)/µ(2µ), and, in general,
N(N-1)
(N
k+1) A"
(1)(2 ... (k)
-)
A`
-\k1
\-)
The binomial formula (1 + x)" _ Ik=o (k)x' applies to yield
N
N
k=O
=O
'k )N.
M%)'
Thus ir0 = [I + (A/µ)] ' _ [µl(A + µ)]N, and
'77i
-
(k
)(
l.t)]N
(4.8)
(k)\A+µ)k(A+µ)vk
We recognize (4.8) as the familiar binomial distribution.
Example Logistic Process Suppose we consider a population whose
size X(t) ranges between two fixed integers N and M (N < Al) for all
t
0. We assume that the birth and death rates per individual at time t are
given by
A = a(M - X(t)) and µ = f3(X(t) - N),
and that the individual members of the population act independently of
each other. The resulting birth and death rates for the population then become
A,, = an(M - n) and µ = f3n(n - N).
372
VI
Continuous Time Markov Chains
To see this we observe that if the population size X(t) is n, then each of the
n individuals has an infinitesimal birth rate A, so that A = an(M - n). The
same rationale applies in the interpretation of the µ,,.
Under such conditions one would expect the process to fluctuate between the two constants N and M, since, for example, if X(t) is near M, the
death rate is high and the birth rate low, and then X(t) will tend toward N.
Ultimately the process should display stationary fluctuations between the
two limits N and M.
The stationary distribution in this case is
Mm
c
N+,» - N + m (
)(a l
/n
where c is an appropriate constant determined so that
this, we observe that
ANA,N+,
.
ITN+,,, = 1. To see
AN+m-I
ON+m =
AN+il.LN+2
AN +m
of"N(N+
1)
1)
... (N + m)m!
N+Jm\MmNI(R)m
Example Some Genetic Models Consider a population consisting of
N individuals that are either of gene type a or gene type A. The state of
the process X(t) represents the number of a-individuals at time t. We assume that the probability that any individual dies and is replaced by another during the time interval (t, t + h) is Ah + o(h) independent of the
values of X(t) and that the probability of two or more changes occurring
in a time interval h is o(h).
The changes in the population structure are effected as follows. An
individual is to be replaced by another chosen randomly from the population; i.e., if X(t) = j, then an a-type is selected to be replaced with probability j/N and an A-type with probability 1 - j/N. We refer to this stage as
death. Next, birth takes place by the following rule. Another selection is
made randomly from the population to determine the type of the new in-
4.
The Limiting Behavior of Birth and Death Processes
373
dividual replacing the one that died. The model introduces mutation pressures that admit the possibility that the type of the new individual may be
altered upon birth. Specifically, let y, denote the probability that an a-type
mutates to an A-type, and let y, denote the probability of an A-type mutating to an a-type.
The probability that the new individual added to the population is of
type a is
-yO+(1-N)y,.
(4.9)
We deduce this formula as\follows: The probability that we select an atype and that no mutation occurs is (j/N)(1 - y,). Moreover, the final type
may be an a-type if we select an A-type that subsequently mutates into an
a-type. The probability of this contingency is (1 - j/N)y2. The combination of these two possibilities gives (4.9).
We assert that the conditional probability that X(t+) - X(t) = 1 when
a change of state occurs is
N)[N(l
(1
y,) + (1
- N)y,J,
where X(t) = j.
(4.10)
In fact, the a-type population size can increase only if an A-type dies (is
replaced). This probability is 1 - (j/N). The second factor is the probability that the new individual is of type a as in (4.9).
In a similar way we find that the conditional probability that
X(t+) - X(t) = -1 when a change of state occurs is
N
N)(1 - y2) + N y,
where X(t) = j.
The number of type a individuals in the population is thus a birth and
death process with a finite number of states and infinitesimal birth and
death rates
1t; = A(I
and
fl-.(l
Y') + (1
- N)y,JN
Continuous Time Markov Chains
VI
374
- Yz)]N,
µ, = lNCNY, + (1 - N)(1
j < N.
0
Although these parameters seem rather complicated, it is interesting to
see what happens to the stationary measure {lrk}k_o if we let the population size N --> oo and the probabilities of mutation per individual y, and y,
tend to zero in such a way that y,N - K, and y,N -4 K,, where 0 < K,,
K, < c. At the same time, we shall transform the state of the process to
the interval [0, 1] by defining new states j/N, i.e., the fraction of a-types
in the population. To examine the stationary density at a fixed fraction x,
where 0 < x < 1, we shall evaluate ark as k -4 oo in such a way that
k = [xN], where [xN] is the greatest integer less than or equal to xN.
Keeping these relations in mind we write
a)
A.=- N-y,)J +, whereas
A(N
J)
NYz
.(
(1
YI
YI-Yz
J
and
µj _
A(N
N J) ( 1
(
- YI - YzJ) ' 1 +
b
N_j
)
where b =
NY
1_y
Then
k-1
log A -
log 0k =
log µj
j=o
j=1
k-1
a
j=1
k-1
b
J=1
N -J
- log(N - k)k(1 +
b
N
k
)+logNa
).
Now, using the expansion
2
log(l + x) = x - 2 +
33
,
it is possible to write
(
k-1
log
j=1
k-I
1+-a)
=a-+ck,
j=I J
J
1
IxI < 1,
-
Y-2,
4.
The Limiting Behavior of Birth and Death Processes
375
where c, approaches a finite limit as k - x. Therefore, using the relation
k-I 1
7',----log k
ask
l=1
we have
k-I
log k"+c,.
7, log (1 +
In a similar way we obtain
k-I
b
)
log (1 +
N
j
j=1
ask-4 oo.
N'
log
(N
k),,
+ d,
ask - x,
where d, approaches a finite limit as k - x. Using the above relations we
have
log 9, - log (C,
k" (N - k)''Na l
as k
I'(N - k)k)
x,
(4.11)
where log C, = c, + d which approaches a limit, say C, as k - x. Notice
that a - K, and b -3 K, as N -3 x. Since k = [Nx], we have, for N -+ x,
0, -
CK,NK,-IxK,-1(1 -
x)K,-'.
Now, from (4.11) we have
0, - aC,k '(I
k )b- i
N.
Therefore,
Since C, ---> C as k tends to x, we recognize the right side as a Riemann
sum approximation of
K,C f x"2-'(1 - x)KH-' dx.
n
Thus
N
1
9;
r=o
N"2K,C f x"2-1(1
o
- x)"'-' dx,
376
VI
Continuous Time Markov Chains
so that the resulting density on [0, 11 is
1
e4.
10,
since dx
N fo
J
XKi-'(I - X) X'-1
XK2-i(1 - X)K,
XKi-I(1 - X)K,-I dX
dx
fo XK,-i(1 - X)K,_, dx'
1/N. This is a beta distribution with parameters K, and K_
Exercises
4.1. In a birth and death process with birth parameters A,, = A for
n = 0, 1, ... and death parameters µ = An for n = 0, 1, ... , we have
=
(Ap);e-a"
Po,(t)
J!
where
1
P = -[I - e-"1µ
Verify that these transition probabilities satisfy the forward equations
(4.4), with i = 0.
4.2. Let X(t) be a birth and death process where the possible states
are 0, 1, . . . , N, and the birth and death parameters are, respectively,
A,, = a(N - n) and µ = On. Determine the stationary distribution.
4.3. Determine the stationary distribution for a birth and death process
having infinitesimal parameters A,, = a(n + 1) and µ = /3n2 for
n = 0, 1, ...,where 0<a</i.
4.4. Consider two machines, operating simultaneously and independently, where both machines have an exponentially distributed time to
failure with mean 1/µ (p. is the failure rate). There is a single repair facility, and the repair times are exponentially distributed with rate A.
(a) In the long run, what is the probability that no machines are operating?
(b) How does your answer in (a) change if at most one machine can operate, and thus be subject to failure, at any time?
377
Problems
4.5. Consider the birth and death parameters A,, = 0 < 1, and
nl(n + 1) for n = 0, 1. .... Determine the stationary distribution.
4.6. A birth and death process has parameters A,, = A and µ = nµ, for
n = 0, 1. .... Determine the stationary distribution.
Problems
4.1. For the repairman model of the second example of this section,
suppose that M = N = 5,R = 1, A = 2, and µ = 1. Using the limiting
distribution for the system, determine
(a) The average number of machines operating.
(b) The equipment utilization.
(c) The average idle repair capacity.
How do these system performance measures change if a second repairman
is added?
4.2. Determine the stationary distribution, when it exists, for a birth and
death process having constant parameters A,, = A for n = 0, 1, ... and
n= 1,2.....
4.3. A factory has five machines and a single repairman. The operating
time until failure of a machine is an exponentially distributed random
variable with parameter (rate) 0.20 per hour. The repair time of a failed
machine is an exponentially distributed random variable with parameter
(rate) 0.50 per hour. Up to five machines may be operating at any given
time, their failures being independent of one another, but at most one machine may be in repair at any time. In the long run, what fraction of time
is the repairman idle?
4.4. This problem considers a continuous time Markov chain model for
the changing pattern of relationships among members in a group. The
group has four members: a, b, c, and d. Each pair of the group may or may
not have a certain relationship with each other. If they have the relationship, we say that they are linked. For example, being linked may mean that
the two members are communicating with each other. The following
378
VI
Continuous Time Markov Chains
graph illustrates links between a and b, between a and c, and between
b and d:
Figure 4.2
Suppose that any pair of unlinked individuals will become linked in a
small time interval of length h with probability ah + o(h). Any pair of
linked individuals will lose their link in a small time interval of length h
with probability /3h + o(h). Let X(t) denote the number of linked pairs of
individuals in the group at time t. Then X(t) is a birth and death process.
(a) Specify the birth and death parameters Ak and /j,, for k = 0, 1.... .
(b) Determine the stationary distribution for the process.
4.5.
A chemical solution contains N molecules of type A and an equal
number of molecules of type B. A reversible reaction occurs between type
A and B molecules in which they bond to form a new compound AB. Suppose that in any small time interval of length h, any particular unbonded
A molecule will react with any particular unbonded B molecule with prob-
ability ah + o(h), where a is a reaction rate of formation. Suppose also
that in any small time interval of length h, any particular AB molecule disassociates into its A and B constituents with probability j3h + o(h), where
,6 is a reaction rate of dissolution. Let X(t) denote the number of AB molecules at time t. Model X(t) as a birth and death process by specifying the
parameters.
4.6.
A time-shared computer system has three terminals that are at-
tached to a central processing unit (CPU) that can simultaneously handle
at most two active users. If a person logs on and requests service when
two other users are active, then the request is held in a buffer until it can
receive service. Let X(t) be the total number of requests that are either active or in the buffer at time t. Suppose that X(t) is a birth and death process
with parameters
Birth and Death Processes with Absorbing States
5.
_
Ak
(A
0
379
for k = 0, 1, 2,
fork ? 3;
and
µk-Jkµ
2µ
for k = 0, 1, 2,
fork=3.
Determine the long run probability that the computer is fully loaded.
4.7. A system consists of three machines and two repairmen. At most
two machines can operate at any time. The amount of time that an operating machine works before breaking down is exponentially distributed
with mean 5. The amount of time that it takes a single repairman to fix a
machine is exponentially distributed with mean 4. Only one repairman
can work on a failed machine at any given time. Let X(t) be the number
of machines in operating condition at time t.
(a) Calculate the long run probability distribution for X(t).
(b) If an operating machine produces 100 units of output per hour,
what is the long run output per hour of the system?
4.8. A birth and death process has parameters
Ak = a(k + 1)
for k = 0, 1, 2,
... ,
and
µk = 6(k + 1)
for k = 1, 2,
... .
Assuming that a < (3, determine the limiting distribution of the process.
Simplify your answer as much as possible.
5.
Birth and Death Processes with Absorbing States
Birth and death processes in which A0 = 0 arise frequently and are correspondingly important. For these processes, the zero state is an absorbing
state. A central example is the linear-growth birth and death process without immigration (cf. Section 3.3). In this case A,, = nit and µ = nµ. Since
growth of the population results exclusively from the existing population,
VI
380
Continuous Time Markov Chains
it is clear that when the population size becomes zero it remains zero
thereafter; i.e., 0 is an absorbing state.
5.1.
Probability of Absorption into State 0
It is of interest to compute the probability of absorption into state 0 starting from state i (i ? 1). This is not, a priori, a certain event, since conceivably the particle (i.e., state variable) may wander forever among the
states (1, 2, ...) or possibly drift to infinity.
Let u, (i = 1, 2, ...) denote the probability of absorption into state 0
from the initial state i. We can write a recursion formula for u, by considering the possible states after the first transition. We know that the first
transition entails the movements
i --' i + 1
A
with Probabilit J
lili
i-i-1
with probability
+A
µ`
/L1+A
Invoking the familiar first step analysis, we directly obtain
u, =
A,
u;+, +
A + Ar
lxi + k;
u;_
i -=-:t
1,
(5.1)
where uo = 1.
Another method for deriving (5.1) is to consider the "embedded random walk" associated with a given birth and death process. Specifically,
we examine the birth and death process only at the transition times. The
discrete time Markov chain generated in this manner is denoted by {Y,,),' ,
where Yo = X0 is the initial state and Y (n ? 1) is the state at the,nth transition. Obviously, the transition probability matrix has the form
0
0
q1
0 P,
0
0
q2
1
P=
0
.. .
0 P2 ...
5.
Birth and Death ProcWses with Absorbing States
381
where
P
A;+µ,
=1-q;
fori?1.
The probability of absorption into state 0 for the embedded random walk
is the same as for the birth and death processes, since both processes execute the same transitions. A closely related problem (gambler's ruin) for
a random walk was examined in III, Section 6.1.
We turn to the task of solving (5.1) subject to the conditions uo = 1 and
0 <_ u; <_ 1 (i ? 1). Rewriting (5.1), we have
(u; - u;-I),
(u;+I - u,) _
Defining v; = u,+, - u;, we obtain
Vi Al
i_ll
Iteration of the last relation yields the formula v; = p; vo, where
Po=1
and
AI
2
Ai
P; = A,11, ... J1;
form 1;
and with u;+, - u; = v;,
u;+i - u;= vi=P;vo=P;(u,-uo)=P,(u,- 1)
fori?1.
Summing these last equations from i = 1 to i = m - 1, we have
u,,, - u, = (u, - 1)
p;,
m > 1.
(5.2)
Since u,,, by its very meaning is bounded by 1, we see that if
x
(5.3)
i=1
then necessarily u, = I and u,,, = 1 for all m > 2. In other words, if (5.3)
holds, then ultimate absorption into state 0 is certain from any initial state.
382
VI
Continuous Time Markov Chains
Suppose 0 < u, < 1; then, of course,
x
i < 00.
i=1
Obviously, u,,, is decreasing in m, since passing from state m to state 0 requires entering the intermediate states in the intervening time. Furthermore, it can be shown that u,,, - 0 as m
. Now letting m -4 oo in (5.2)
permits us to solve for u,; thus
11=1 Pi
1+i ,Pi'
u'
and then from (5.2), we obtain
l,_,,. P.
1.=i P;
5.2.
m
1.
Mean Time Until Absorption
Consider the problem of determining the mean time until absorption,
starting from state m.
We assume that condition (5.3) holds, so that absorption is certain. Notice that we cannot reduce our problem to a consideration of the embedded random walk, since the actual time spent in each state is relevant for
the calculation of the mean absorption time.
Let w; be the mean absorption time starting from state i (this could be
infinite). Considering the possible states following the first transition, instituting a first step analysis, and recalling the fact that the mean waiting
time in state i is (Ai + µi)-' (it is actually exponentially distributed with
parameter Ai + µ;), we deduce the recursion relation
1
w; =
Al+µ.
+
Jt'
A.+µ.
wi+, +
µi
A.+µ.
w;_,
i ? 1,
(5.4)
where w = 0. Letting z; = wi - w;+, and rearranging (5.4) leads to
1
µ;
Zi- 1,
Iterating this relation gives
(5.5)
5.
Birth and Death Proggses with Absorbing States
1
Z,
+
Zo,
A,
+
1
+
µ,It
µ,
P,
1
z, =
µ,
Z,
A,
+
A,
A,
µ,µ,µi
+
µ'3
+
A,A
z3
A3
383
A3A,
A,A,A,
A3A,A,
Z0'
z
°'
and finally
(The product II;;;+, p,/A; is interpreted as 1.) Using the notation
µ.µ_'
and
1
µ`
the expression for z,,, becomes
_
M
Z=
!
1
r=1
i
m
+ P,,,Zo,
pi
or, since z,,, = is,, - w,,,+, and zo = wo - w, = -w,, then
1
"'
An
j=1
1
- VV'.
(5.6)
ipi
If G'i=, (1/A;p,) = oo, then inspection of (5.6) reveals that necessarily
w, = x. Indeed, it is probabilistically evident that w< w,,,+, for all m, and
this property would be violated for m large if we assume to the contrary
that w, is finite.
Now suppose I;_, (1/A;p;) < oo; then letting m - x in (5.6) gives
1
j=1 A;P,
1
- lim -(w- u;+.)
An
It is more involved but still possible to prove that
1
lim -(w,,, - w,,,+,) = 0,
384
Continuous Time Markov Chains
VI
and then
x
1
Aipi
We summarize the discussion of this section in the following theorem:
Theorem 5.1 Consider a birth and death process with birth and death
parameters A,, and p.,,, n ? 1, where A0 = 0 so that 0 is an absorbing state.
The probability of absorption into state 0 from the initial state m is
m'=m Pi
1 + I;-, pi
i=1
X
un =
(5.7)
00
i
i=1
The mean time to absorption is
X
1
00
i=
w
Ai pi
(5.8)
I
k=1
j=k+1 AJpi
- < oo
i=
Aipi
where pa = 1 and p, = (µ1µ2 ... p'i)l(A1A2 ... Ai)
Example Population Processes Consider the linear growth birth and
death process without immigration (cf. Section 3.3) for which µ = nµ
and A,, = nil, n = 0, 1..... During a short time interval of length h, a single individual in the population dies with probability µh + o(h) and gives
birth to a new individual with probability Ah + o(h), and thus µ > 0 and
A > 0 represent the individual death and birth rates, respectively.
Substitution of a = 0 and i = m in equation (3.9) determines the mean
population size at time t for a population starting with X(0) = m individuals. This mean population size is M(t) = me' " 1, exhibiting exponential
growth or decay according as A > µ or A < A.
Let us now examine the extinction phenomenon and determine the
probability that the population eventually dies out. This phenomenon corresponds to absorption in state 0 for the birth and death process.
When A,, = nA and µ = nµ, a direct calculation yields pi = (µ/ 1)i, and
then
5.
Birth and Death Proses with Absorbing States
(µ/A)' =
1 - (A/A)
i =,n
385
when A > µ,
when A :5 A.
Co
From Theorem 5.1, the probability of eventual extinction starting with m
individuals is
Pr{ExtinctionIX(0) = m} _ (µ"/A)n
1
when A >
when A
_
µ.
(5.9)
When A = µ, the process is sure to vanish eventually. Yet in this case
the mean population size remains constant at the initial population level.
Similar situations where mean values do not adequately describe population behavior frequently arise when stochastic elements are present.
We turn attention to the mean time to extinction assuming that extinction is certain, that is, when A
A. For a population starting with a single individual, then, from (5.8) with m = 1 we determine this mean time
to be
1
Aipi
_
i
1
i1µ
A
x
1
(A/µ)
If
Ai=1
xi_'
dx
o
(5.10)
when
Co
when
386
VI
Continuous Time Markov Chains
When the birth rate A exceeds the death rate µ, a linear growth birth and
death process can, with strictly positive probability, grow without limit. In
contrast, many natural populations exhibit density-dependent behavior
wherein the individual birth rates decrease or the individual death rates increase or both changes occur as the population grows. These changes are
ascribed to factors including limited food supplies, increased predation,
crowding, and limited nesting sites. Accordingly, we introduce a notion of
environmental carrying capacity K, an upper bound that the population
size cannot exceed.
Since all individuals have a chance of dying, with a finite carrying ca-
pacity, all populations will eventually become extinct. Our measure of
population fitness will be the mean time to extinction, and it is of interest
to population ecologists studying colonization phenomena to examine
how the capacity K, the birth rate A, and the death rate µ affect this mean
population lifetime.
The model should have the properties of exponential growth (on the average) for small populations, as well as the ceiling K beyond which the
population cannot grow. There are several ways of approaching the population size K and staying there at equilibrium. Since all such models give
more or less the same qualitative results, we stipulate the simplest model,
in which the birth parameters are
forn=0,1,...,K- 1,
nA
"-I0
forn>K.
Theorem 5.1 yields w,, the mean time to population extinction starting
with a single individual, as given by
A A,
1
i ArA,
A,
i== I
. . .
;- I
!-'I/-k? ... At
1
K
1
A
(5.11)
IL
i\p
Equation (5.11) isolates the distinct factors influencing the mean time
to population extinction. The first factor is 1/µ, the mean lifetime of an individual, since µ is the individual death rate. Thus, the sum in (5.11) represents the mean generations, or mean lifespans, to population extinction,
a dimensionless quantity that we denote by
le,
I
where 0 =
-µ
.
(5.12)
5.
Birth and Death Proc4gses with Absorbing States
387
Next we examine the influence of the birth-death, or reproduction, ratio
0 = A/g and the carrying capacity K on the mean time to extinction. Since
,1 represents the individual birth rate and 1/µ is the mean lifetime of a sin-
gle member in the population, we may interpret the reproduction ratio
0 = A(1/µ) as the mean number of offspring of an arbitrary individual in
the population. Accordingly, we might expect significantly different behavior when 0 < 1 as opposed to when 0 > 1, and this is indeed the case.
A carrying capacity of K = 100 is small. When K is on the order of 100
or more, we have the following accurate approximations, their derivations
being sketched in Exercises 5.1 and 5.2:
for 0 < 1,
for 0 = 1,
Ms
(5.13)
for 0 > 1.
The contrast between 0 < 1 and 0 > 1 is vivid. When 0 < 1, the mean
generations to extinction Mx is almost independent of carrying capacity K
and approaches the asymptotic value 0' ln(1 - 0)-' quite rapidly. When
0 > 1, the mean generations to extinction Mg grows exponentially in K.
Some calculations based on (5.13) are given in Table 5.1.
Table 5.1
K
10
100
1000
Mean generations to extinction for
a population starting with a single
parent and where 0 is the reproduction rate and K is the environmental capacity.
0=0.8
0= 1
1.96
2.01
2.01
2.88
5.18
7.48
0= 1.2
3.10
4.14 x 104
7.59 x 1076
388
VI
Continuous Time Markov Chains
Example Sterile Male Insect Control The screwworm fly, a cattle
pest in warm climates, was eliminated from the southeastern United States
by the release into the environment of sterilized adult male screwworm
flies. When these males, artificially sterilized by radiation, mate with native females, there are no offspring, and in this manner part of the reproductive capacity of the natural population is nullified by their presence. If
the sterile males are sufficiently plentiful so as to cause even a small decline in the population level, then this decline accelerates in succeeding
generations even if the number of sterile males is maintained at approximately the same level, because the ratio of sterile to fertile males will increase as the natural population drops. Because of this compounding effect, if the sterile male control method works at all, it works to such an
extent as to drive the native population to extinction in the area in which
it is applied.
Recently, a multibillion-dollar effort involving the sterile male technique has been proposed for the control of the cotton boll weevil. In this
instance, it was felt that pretreatment with a pesticide could reduce the
natural population size to a level such that the sterile male technique
would become effective. Let us examine this assumption, first with a deterministic model and then in a stochastic setting.
For both models we suppose that sexes are present in equal numbers,
that sterile and fertile males are equally competitive, and that a constant
number S of sterile males is present in each generation. In the deterministic case, if No fertile males are in the parent generation and the No fertile
females choose mates equally likely from the entire male population, then
the fraction No/(N0 + S) of these matings will be with fertile males and
will produce offspring. Letting 9 denote the number of male offspring that
results from a fertile mating, we calculate the size N of the next generation according to
N, = BNo(N
,,
+o S)
(5.14)
For a numerical example, suppose that there are No = 100 fertile males
(and an equal number of fertile females) in the parent generation of the
native population, and that S = 100 sterile male insects are released. If
6 = 4, meaning that a fertile mating produces four males (and four females) for the succeeding generation, then the number of both sexes in the
first generation is
5.
Birth and Death Processes with Absorbing States
N, = 4(100)(100100100)
=
389
200;
the population has increased, and the sterile male control method has
failed.
Table 5.2 The trend of an insect population subject to sterile male releases.
Number of Insects
Generation Natural Population
Number of
Sterile Insects
Ratio Sterile Number of
to Fertile
Progeny
20
100
5:1
13.33
F,
13.33
7.5:1
6.27
F,
6.27
F3
1.48
0.09
16:1
67.5:1
1156:1
0.09
F4
100
100
100
100
Parent
1.48
-
On the other hand, if a pesticide can be used to reduce the initial population size to No = 20, or 20 percent of its former level, and S = 100 sterile males are released, then
and the population is declining. The succeeding population sizes are given
in Table 5.2, above. With the pretreatment, the population becomes extinct
by the fourth generation.
Often deterministic or average value models will adequately describe
the evolution of large populations. But extinction is a small population
phenomenon, and even in the presence of significant long term trends,
small populations are strongly influenced by the chance fluctuations that
determine which of extinction or recolonization will occur. This fact motivates us to examine a stochastic model of the evolution of a population
in the presence of sterile males. The factors in our model are
the inidividual birth rate;
the inidividual death rate;
0 = AIA, the mean offspring per individual;
A,
µ,
390
K,
S,
in,
Continuous Time Markov Chains
VI
the carrying capacity of the environment;
the constant number of sterile males in the population;
the initial population size.
We assume that both sexes are present in equal numbers in the natural
population and that X(t), the number of either sex present at time t,
evolves as a birth and death process with parameters
A=
n<K,
fAn(n+S
forn?K,
0
and
(5.15)
forn=0,1.....
This is the colonization model of the Population Processes example,
modified in analogy with (5.14) by including in the birth rate the factor
n/(n + S) to represent the probability that a given mating will be fertile.
To calculate the mean time to extinction w,,, as given in (5.8), we first
use (5.15) to determine
Pk =
(µlk(k + S)!
P'_P"
Ak = \ A /
A,A,
fork= 1,...,K- 1,
k!S!
Po= 1, and pK=oo,or1/px=0,
and then substitute these expressions for pk into (5.8) to obtain
In- 1
+ kL=1
win =
A
,-
A,P,
_ In - I
j=k+I A,P,
In- I
- =2_ P j=k+I IkiPi-'
= kj=k+Ij AiPi
1
,
-
k
k=O
1 1m-1 K-1
1
µO=j+1
e,-k j!(S + k)1
k!(S + j)!
I
(5.16)
S.
Birth and Death Processes with Absorbing States
391
Because of the factorials, equation (5.16) presents numerical difficulties
when direct computations are attempted. A simple iterative scheme works
to provide accurate and effective computation, however. We let
K-'
j!(S + k)!
k!(S + j)!'
-k j + 1
so that w,,, _ (a. +
+
But it is easily verified that
ak_, = k + 0(S
+k
)C".
Beginning with aK = 0, one successively computes aK_,, aK_21 ... , a0,
and then w,,, = (a0 +
+ a,,,- )11.k.
Using this method, we have computed the mean generations to extinction in the stochastic model for comparison with the deterministic model
as given in Table 5.2. Table 5.3 lists the mean generations to extinction for
various initial population sizes m when K = S = 100, A = 4, and µ = 1,
so that 0 = 4. Instead of the four generations to extinction as predicted by
the deterministic model when m = 20, we now estimate that the population will persist for over 8 billion generations!
Table 5.3 The mean lifespans to
extinction in a birth and
death model of a population containing a constant
number S = 100 of sterile
males.
Initial
Population Size
Mean Lifespans
to Extinction
5
8,101,227,748
4,306,531
3,822
4
566
3
65
20
10
2
6.3
1
1.2
392
Continuous Time Markov Chains
VI
What is the explanation for the dramatic difference between the predic-
tions of the deterministic model and the predictions of the stochastic
model? The stochastic model allows the small but positive probability that
the population will not die out but will recolonize and return to a higher
level near the environmental capacity K, and then persist for an enormous
length of time.
While both models are qualitative, the practical implications cannot be
dismissed. In any large-scale control effort, a wide range of habitats and
microenvironments is bound to be encountered. The stochastic model suggests the likely possibility that some subpopulation in some pocket might
persist and later recolonize the entire area. A sterile male program that de-
pends on a pretreatment with an insecticide for its success is chancy at
best.
Exercises
5.1. Assuming 0 < 1, verify the following steps in the approximation to
Mg, the mean generation to extinction as given in (5.12):
K
Mg=
i=1
=9
-0 =e1
6
K
fx;- ' dx
i=1 0
1
a1xK
1_x dxB'J
f
c
c
dx
1-x-9c
1
(e'
x"
1-xdx
e
In
B
1
l
0
1
1
1
_
1
OK+1
OK+2
1-0 0\K+l +K+2+
0
=
1-B -0-' f0
In
1
-
6K
(I+K+Ie+K+102+...
1-0 K+I\
1
eln1-e
K+2
OK
(K+1)(1-9)
K+3
/
Problems
393
5.2. Assume that 0 > 1 and verify the following steps in the approximation to Mg, the mean generation to extinction as given in (5.12):
j K-j+ 1
01l
)2
I
K I1+K`K11-1
+K IOIKIJ
+
1
_
OK- If
l_
1
OK
K(0 - 1)'
K L 1 - (1/0)J
Problems
5.1. Consider the sterile male control model as described in the example entitled "Sterile Male Insect Control" and let u,,, be the probability that
the population becomes extinct before growing to size K starting with
X(O) = m individuals. Show that
um =
Pi
form= 1,...,K,
K-I
l1=o Pr
where
P1 =
0(S
+ i)!
i.
5.2.
Consider a birth and death process on the states 0, 1,
parameters
AO =N1o=As =N-5=0,
11, = 1,
1t2 = 2,
A3 = 3,
A4 - 4,
µ1=4,
N2=3,
µ3=2,
µ= 1.
... , 5 with
394
Continuous Time Markov Chains
VI
Note that 0 and 5 are absorbing states. Suppose the process begins in state
X(0) = 2.
(a) What is the probability of eventual absorption in state 0?
(b) What is the mean time to absorption?
6.
Finite State Continuous Time Markov Chains
A continuous time Markov chain X(t) (t > 0) is a Markov process on the
states 0, 1, 2. .... We assume as usual that the transition probabilities are
stationary; i.e.,
P,;(t) = Pr{X(t + s) = jjX(s) = i}.
(6.1)
In this section we consider only the case where the state space S is finite,
labeled as 10, 1, 2, . . . , N).
The Markov property asserts that P;;(t) satisfies
(a) P;(t) ? 0,
P;(t) = 1,
(b)
i, j = 0, 1,
... , N, and
J=O
(c) PA(s + t) =
P;(s)Pjl(t)
j=o
for t, s ? 0
(Chapman-Kolmogorov relation),
and we postulate in addition that
(d)
1,
lamP;(t)={0,
i=j,
ij.
If P(t) denotes the matrix (;P;(t) ";_,, then property (c) can be written
compactly in matrix notation as
P(t + s) = P(t)P(s),
t, s ? 0.
(6.2)
Property (d) asserts that P(t) is continuous at t = 0, since the fact P(O) = I
(= identity matrix) is implied by (6.2). It follows simply from (6.2) that
P(t) is continuous for all t > 0. In fact, ifs = h > 0 in (6.2), then because
of (d), we have
6.
Finite State Continuou,ime Markov Chains
395
lim P(t + h) = P(t) lim P(h) = P(t)I = P(t).
(6.3)
On the other hand, for t > 0 and 0 < h < t we write (6.2) in the form
P(t) = P(t - h)P(h).
(6.4)
But P(h) is near the identity when h is sufficiently small, and so P(h)-'
[the inverse of P(h)] exists and also approaches the identity I. Therefore,
P(t) = P(t) lim (P(h))-' = lim P(t - h).
(6.5)
The limit relations (6.3) and (6.5) together show that P(t) is continuous.
Actually, P(t) is not only continuous but also differentiable in that the
limits
lim
1 -P;(h)
h
k-,0+
-= q,
(6.6)
p;(h)
,1-,m
h
i 0 j,
= qi;,
exist, where 0 < q,, < ° (i 0 j) and 0 :5- q; < oc. Starting with the relation
I-
i=o,j*i
P,;(h),
dividing by h, and letting h decrease to zero yields directly the relation
N
q, = L,
q,;.
j=o,j*i
The rates q; and q;; furnish an infinitesimal description of the process
with
Pr{X(t + h) = jJX(t) = i} = q;;h + o(h)
for i 0 j,
Pr{X(t + h) = it X(t) = i} = I - q;h + o(h).
In contrast to the infinitesimal description, the sojourn description of
the process proceeds as follows: Starting in state i, the process sojourns
there for a duration that is exponentially distributed with parameter q;.
The process then jumps to state j 0 i with probability p;, = q;;/q;; the sojourn time in state j is exponentially distributed with parameter q;, and so
VI
396
Continuous Time Markov Chains
on. The sequence of states visited by the process, denoted by fo, 6......
is a Markov chain with discrete parameter, called the embedded Markov
chain. Conditioned on the state sequence o, .... , the successive sojourn times So, S ... are independent exponentially distributed random
variables with parameters q,,,, qF
, . . . ,
respectively.
Assuming that (6.6) has been verified, we now derive an explicit expression for P;(t) in terms of the infinitesimal matrix
A=
-qo
q01
qio
-q1
NO
qON
IN
...
qN i
- qN
The limit relations (6.6) can be expressed concisely in matrix form:
lim
la-+0+
P(h) - I = A,
(6.7)
h
which shows that A is the matrix derivative of P(t) at t = 0. Formally,
A = P'(0).
With the aid of (6.7) and referring to (6.2), we have
P(t + h) - P(t) = P(t)[P(h) - I] = P(h) - I
P(t).
h
h
h
(6.8)
The limit on the right exists, and this leads to the matrix differential equation
P'(t) = P(t)A = AP(t),
(6.9)
where P(t) denotes the matrix whose elements are P'(t) = dP; (t)/dt. The
existence of P'(t) is an obvious consequence of (6.7) and (6.8). The differential equations (6.9) under the initial condition P(0) = I can be solved
by standard methods to yield the formula
A" t"
P(t)=eA,=I+
If=
n!
(6.10)
Example The Two State Markov Chain Consider a Markov chain
{X(t)} with states 10, 1) whose infinitesimal matrix is
6.
Finite State Continuous Time Ma"rkov Chains
397
A=0
The process alternates between states 0 and 1. The sojourn times in state
0 are independent and exponentially distributed with parameter a. Those
in state 1 are independent and exponentially distributed with parameter /3.
We carry out the matrix multiplication
-a
a
-R X
18
-a
a
a2 + a/3
- a22 - a/3
R -R
-R2-a/3
/3222+a13
-a
= -(a + /3)
a
R -R
to see that A2 = -(a + /3)A. Repeated multiplication by A then yields
which when inserted into (6.10) simplifies the sum according to
P(t)=I-
[-(a + /3)t]" A
1
a+
=I- a+R
=I+
n!
[e-("+")'
- I IA
Aa+a+/3
1
1
Ae-l°+g".
And with 7r = al(a + /i) and r = a + /3,
P(t) =
1 - 7r
7r
1 - it
IT
IT
- 7r
e-TI
-(1-IT) (1-IT)
which is the matrix expression for equations (3.14a-d).
Returning to the general Markov chain on states (0, 1, ... , N), when
the chain is irreducible (all states communicate) then P; (t) > 0 for i,
398
VI
Continuous Time Markov Chains
j = 0, 1, ... , N and lim,.. P;(t) = ir; > 0 exists independently of the
initial state i. The limiting distribution may be found by passing to the
limit in (6.9), noting that lim,,,. P'(t) = 0. The resulting equations for
7 T = (?ro, ?T,, ... , ITN) are
0 = 'nA = (ir0. ,7r1.
.
. .
- qo
q01
qio
-qi
qNo
qN1
.
q0N
q1N
, 7TN)
9N
which is the same as
i*j
Equation (6.11) together with
qij,
ITo + ITi +
j=0,1,...,N.
+ ITN = 1
(6.11)
(6.12)
determines the limiting distribution.
Equation (6.11) has a mass balance interpretation that aids us in understanding it. The left side TT;q; represents the long run rate at which particles executing the Markov process leave state j. This rate must equal the
long run rate at which particles arrive at state j if equilibrium is to be
maintained. Such arriving particles must come from some state i * j, and
a particle moves from state i 0 j to state j at rate q;;. Therefore, the right
side I.;#; 7rq;; represents the total rate of arriving particles.
Example Industrial Mobility and the Peter Principle Let us suppose
that a draftsman position at a large engineering firm can be occupied by a
worker at any of three levels: T = Trainee, J = Junior Draftsman, and
S = Senior Draftsman. Let X(t) denote the level of the person in the position at time t, and suppose that X(t) evolves as a Markov chain whose infinitesimal matrix is
T
A=J
S
6.
Finite State Continuous Time Markov Chains
399
Thus a Trainee stays at that rank for an exponentially distributed time having parameter aT and then becomes a Junior Draftsman. A Junior Draftsman stays at that level for an exponentially distributed length of time hav-
ing parameter a, = a,T + a,s. Then the Junior Draftsman leaves the
position and is replaced by a Trainee with probability a,T/a, or is promoted
to a Senior Draftsman with probability a,/a,, and so on.
Alternatively, we may describe the model by specifying the movements
during short time intervals according to
Pr{X(t + h) = JI X(t) = T} = aTh + o(h),
Pr{X(t + h) = TIX(t) = J} = a,Th + o(h),
Pr{X(t + h) = SIX(t) = J} = a,sh + o(h),
Pr{X(t+h)=TIX(t)=S}=ash+o(h),
and
Pr(X(t + h) = ijX(t) = i} = I - a;h + o(h)
for i = T, J, S.
The equations for the equilibrium distribution (7T, ir,, 7TS) are, according to (6.11),
aJT7rJ + asIr ,
aJ,7T
1=
7T +
7T, +
and the solution is
IrT
a, a,
a,a, + a5a, + aTaJS'
asaT
J
7r5 =
a,a, + a,aT +
aTa1s
asa, + a,aT + aTa,s
7T5,
VI
400
Continuous Time Markov Chains
Let us consider a numerical example for comparison with an alternative
model to be developed later. We suppose that the mean times in the three
states are
State
Mean Time
T
0.1
J
0.2
S
1.0
and that a Junior Draftsman leaves and is replaed by a Trainee with probability s and is promoted to a Senior Draftsman with probability These
suppositions lead to the prescription aT =I 0, a,T = 2, a,s = 3, as = 1. The
equilibrium probabilities are
1(5)
ITT =
1(5) + 1(10) + 10(3)
.7r, =
=
5
45
=
0.11,
10 45= 0.22,
mss= 405 =0.67.
But the duration that people spend in any given position is not exponentially distributed in general. A bimodal distribution is often observed
in which many people leave rather quickly, while others persist for a substantial time. A possible explanation for this phenomenon is found in the
"Peter Principle," which asserts that a worker is promoted until finally
reaching a position in which he or she is incompetent. When this happens,
the worker stays in that job until retirement. Let us modify the industrial
mobility model to accommodate the Peter Principle by considering two
types of Junior Draftsmen, Competent and Incompetent. We suppose that
a fraction p of Trainees are Competent, and q = 1 - p are Incompetent.
We assume that a competent Junior Draftsman stays at that level for an ex-
2
Laurence J. and Raymond Hull, The Peter Principle, Buccaneer Books.
Cutchogue, New York, 1969.
6.
Finite State Continuous Time Markov Chains
401
ponentially distributed duration with parameter a, and then is promoted to
Senior Draftsman. Finally, an incompetent Junior Draftsman stays in that
position until retirement, an exponentially distributed sojourn with parameter a,, and then he or she is replaced by a Trainee. The relevant infinitesimal matrix is given by
T
I
C
-ar
qar
paT
a,
-a,
S
-ac
ac
-a,
S
The duration in the Junior Draftsman position now follows a probability law that is a mixture of exponential densities. To compare this model
with the previous model, suppose that p = 3, q = 5, a, = 2.86, and ac = 10.
These numbers were chosen so as to make the mean duration as a Junior
Draftsman,
PI
+ ql 1 I = (3)(0.10) + (5)(0.35) = 0.20,
\1)a,a,
the same as in the previous calculations. The probability density of this
duration is
f(t) = 3(10)e
+
for t ? 0.
5(2.86)e-z.aer
This density is plotted in Figure 6.1, next page, for comparison with the
exponential density g(t) = 5e 5r, which has the same mean. The bimodal
tendency is indicated in thatf(t) > g(t) when t is near zero and when t is
very large.
With the numbers as given and ar = 10 and a, = I as before, the stationary distribution (Irr, Tr irc, ir,) is found by solving
101rr =
1 ar
2.86or,,
2.861r, = 41Tr,
101rc = 61rr,
l0arc,
17r, =
1 = Trr +
Tr, +
7T
+
7Ts.
402
Continuous Time Markov Chains
VI
4.0
2.0
1.0
.60
.40
.20
.10
.06
.04
.02
1
0
.2
1
.4
1
1
6
8
I
1.0
Figure 6.1 The exponential density (straight line) versus the mixed exponential density (curved line). Both distributions have the same
mean. A logarithmic scale was used to accentuate the differences.
The solution is
7r,. = 0.111,
ir, = 0.155,
ors = 0.667,
7r, = 0.067.
Let us make two observations before leaving this example. First, the
limiting probabilities 7r, , Tr and ir, = 'Ir, + Trc agree between the two
models. This is a common occurrence in stochastic modeling, wherein the
limiting behavior of a process is rather insensitive to certain details of the
model and depends only on the first moments, or means. When this happens, the model assumptions can be chosen for their mathematical convenience with no loss.
The second observation is specific to the Peter Principle. We have assumed that p = s of Trainees are competent Junior Draftsmen and only
q = , are Incompetent. Yet in the long run, a Junior Draftsman is found to
be Incompetent with probability Mr,/(ar, + Ire) = 0.155/(0.155 + 0.067) _
0.70!
6.
Finite State Continuous Time Markov Chains
403
Example Redundancy and the Burn-In Phenomenon An airlines
reservation system has two computers, one on-line and one backup. The
operating computer fails after an exponentially distributed duration having parameter µ and is replaced by the standby. There is one repair facility, and repair times are exponentially distributed with parameter A. Let
X(t) be the number of computers in operating condition at time t. Then
X(t) is a Markov chain whose infinitesimal matrix is
0
A=1
2
1
2
A
0
-(A + µ)
A
µ
-µ
ro
The stationary distribution (7ro, IT,, 7r,) satisfies
A7ro =
(A + µ)7r, = A7ro,
A7r,,
A1'2 =
1 =
1TO +
7r1 +
7T2,
and the solution is
1
1 + (Al p.) + (A
7r,
IT,
µ)
Alµ
1 + (A1µ) + (A/µ.)',
(Alp)2
1 + (Alµ) + (Alµ)'
The availability, or probability that at least one computer is operating, is
1 -7rp=7T,+7r,.
Often in practice the assumption of exponentially distributed operating
times is not realistic because of the so-called burn-in phenomenon. This
idea is best explained in terms of the hazard rate r(t) associated with a
probability density function f(t) of a nonnegative failure time T. Recall
404
Vi
Continuous Time Markov Chains
that r(t) At measures the conditional probability that the item fails in the
next time interval (t, t + At) given that it has survived up to time t, and
therefore we have
r(t) =
f (t)
1 - F(t)
fort ? 0,
where F(t) is the cumulative distribution function associated with the
probability density function f(t).
A constant hazard rate r(t) = A for all t corresponds to the exponential
density function f(t) = Ae-A' for t ? 0. The burn-in phenomenon is described by a hazard rate that is initially high and then decays to a constant
level, where it persists, possibly later to rise again (aging). It corresponds
to a situation in which a newly manufactured or newly repaired item has
a significant probability of failing early in its use. If the item survives this
test period, however, it then operates in an exponential or memoryless
manner. The early failures might correspond to incorrect manufacture or
faulty repair, or might be a property of the materials used.
Anyone familiar with automobile repairs has experienced the burn-in
phenomenon.
One of many possible ways to model the burn-in phenomenon is to use
a mixture of exponential densities
f(t) = pae' + q&-O',
t ? 0,
(6.13)
r(t)
2
1
0
0.2
0.4
0.6
0.8
t
Figure 6.2 The hazard rate corresponding to the density given in (6.13). The
higher hazard rate at the initial t values represents the bum-in
phenomenon.
405
Exercises
where 0 < p = 1 - q < 1 and a, /3 are positive. The density function for
whichp = 0.1, a = 10, q = 0.9, and 8 = 0.909
= 1/1.1 has mean one.
Its hazard rate is plotted in Figure 6.2 on the previous page, where the
higher initial burn-in level is evident.
We may incorporate the burn-in phenomenon corresponding to the
mixed exponential density (6.13) by expanding the state space according
to the following table:
State
Notation
Both computers down
One operating computer, current up time has parameter a
One operating computer, current up time has parameter /3
Two operating computers, current up time has parameter a
Two operating computers, current up time has parameter /3
OA
1B
2A
28
Equation (6.13) corresponds to a probability p that a computer beginning operation will have an exponentially distributed up time with parameter a and a probability q that the parameter is /3. Accordingly, we have
the infinitesimal matrix
0
lA
1B
-A
pA
qA
lA
a
-(A + a)
A = 18
/i
0
2A
2A
A
-(A+/3)
2A
pa
qa
2Q
p13
q13
A
-a
-J6
The stationary distribution can be determined in the usual way by applying (6.11).
Exercises
6.1. A certain type component has two states: 0 = OFF and 1 = OPERATING. In state 0, the process remains there a random length of time,
which is exponentially distributed with parameter a, and then moves to
406
VI
Continuous Time Markov Chains
state 1. The time in state 1 is exponentially distributed with parameter 13,
after which the process returns to state 0.
The system has two of these components, A and B, with distinct parameters:
Component
Operating Failure Rate'
Repair Rate
A
NA
aA
B
13B
aB
In order for the system to operate, at least one of components A and B
must be operating (a parallel system). Assume that the component stochastic processes are independent of one another. Determine the long run
probability that the system is operating by
(a) Considering each component separately as a two-state Markov
chain and using their statistical independence;
(b) Considering the system as a four-state Markov chain and solving
equations (6.11).
6.2.
Let X,(t) and X,(t) be independent two-state Markov chains having
the same infinitesimal matrix
A =
0
011 -A
1
1
µ µI
jj_
Argue that Z(t) = X,(t) + XZ(t) is a Markov chain on the state space
S = (0, 1, 2) and determine the transition probability matrix P(t) for Z(t).
Problems
6.1. Let Y,,, n = 0, 1, . . . , be a discrete time Markov chain with transition probabilities P = lPA, and let {N(t); t ? 0) be an independent Poisson process of rate A. Argue that the compound process
X(t) = Y,N(,),
t ? 0,
is a Markov chain in continuous time and determine its infinitesimal parameters.
407
Problems
6.2. A certain type component has two states: 0 = OFF and 1 =
OPERATING. In state 0, the process remains there a random length of
time, which is exponentially distributed with parameter a, and then moves
to state 1. The time in state 1 is exponentially distributed with parameter
J3, after which the process returns to state 0.
The system has three of these components, A, B, and C, with distinct
parameters:
Component
Operating Failure Rate
Repair Rate
A
3A
aA
B
NB
a6
C
ac
ac
In order for the system to operate, component A must be operating, and at
least one of components B and C must be operating. In the long run, what
fraction of time does the system operate? Assume that the component stochastic processes are independent of one another.
6.3.
Let X, (t), X, (t),
... , XN(t) be independent two-state Markov chains
having the same infinitesimal matrix
A=0
Determine the infinitesimal matrix for the Markov chain Z(t) _
XI(t) +
+ XN(t).
6.4. A system consists of two units, both of which may operate simultaneously, and a single repair facility. The probability that an operating
system will fail in a short time interval of length At is µ(4t) + o(Ot). Repair times are exponentially distributed, but the parameter depends on
whether the failure was regular or severe. The fraction of regular failures
is p, and the corresponding exponential parameter is a. The fraction of se-
vere failure is q = 1 - p, and the exponential parameter is /3 < a.
Model the system as a continuous time Markov chain by taking as
states the pairs (x, y), where x = 0, 1, 2 is the number of units operating
408
VI
Continuous Time Markov Chains
and y = 0, 1, 2 is the number of units undergoing repair for a severe failure. The possible states are (2, 0), (1, 0), (1, 1), (0, 0), (0, 1), and (0, 2).
Specify the infinitesimal matrix A. Assume that the units enter the repair
shop on a first come, first served basis.
7.
A Poisson Process with a Markov Intensity*
Consider "points" scattered in some manner along the semi-infinite interval [0, 00), and for an interval of the form I = (a, b], with 0 <_ a < b < cc,
let N(1) count the number of "points" in the interval I. Then N(1), as I
ranges over the half-open intervals I = (a, b], is a point process. (See V,
Section 5 for generalizations to higher dimensions.) Suppose that, conditional on a given intensity, N(I) is a nonhomogeneous Poisson process,
but where the intensity function (A(t), t ? 0) is itself a stochastic process.
Such point processes were introduced in V, Section 1.4, where they were
called Cox processes in honor of their discoverer. While Cox processes
are sufficiently general to describe a plethora of phenomena, they remain
simple enough to permit explicit calculation, at least in some instances. As
an illustration, we will derive the probability of no points in an interval for
a Cox process in which the intensity function is a two-state Markov chain
in continuous time. The Cox process alternates between being "ON" and
"OFF." When the underlying intensity is "ON," points occur according to
a Poisson process of constant intensity A. When the underlying process is
"OFF," no points occur. We will call this basic model a (0, A) Cox process
to distinguish it from a later generalization. The (0, A) Cox process might
describe bursts of rainfall in a locale that alternates between dry spells and
wet ones, or arrivals to a queue from a supplier that randomly shuts down.
Rather straightforward extensions of the techniques that we will now use
in this simple case can be adapted to cover more complex models and
computations, as we will subsequently show.
We assume that the intensity process (A(t); t ? 0) is a two-state Markov
chain in continuous time for which
Pr(A(t + h) = AIA(t) = 0) = ah + o(h),
*Starred sections contain material of a more specialized or advanced nature.
(7.1)
7. A Poisson Process with a Markov Intensity
409
and
Pr{A(t + h) = OIA(t) = A) = /3h + o(h).
(7.2)
This intensity is merely the constant A times the two-state birth and death
process of Section 3. As may be seen by allowing t - ° in (3.13), such a
process has the limiting distribution Pr(A(c) = 0) = /3/(a + /3) and
Pr{A(°°) = A) = a/(a + /3). We will assume that the intensity process be-
gins with this limiting distribution, or explicitly, that Pr(A(0) = 0) =
/3/(a + /3) and Pr{A(0) A) = a/(a + (3). With this assumption, the intensity process is stationary in the sense that Pr{A(t) = 0) = /3/(a + /3)
and Pr{A(t) = A] = a/(a +`,/3) for all t ? 0. This stationarity carries over
to the Cox process N(I) to imply that Pr{N((0, t]) = k) = Pr{N((s, s + t])
= k) for all s, t ? 0 and k = 0, 1..... We are interested in determining
f(t; A) = Pr{N((0, t]) = 0).
Let
I
A(t) = f A(s) ds
(7.3)
0
and note the conditional Poisson probability
Pr {N((0, t]) = OIA(s) for s < t} = e
so that upon removing the conditioning via the law of total probability we
obtain
At; A) =
E[e-All)] =
.f0(t) + f (t),
(7.4)
where
fo(t) = Pr{N((0, t]) = 0 and A(t) = 0),
(7.5)
fl(t) = Pr{N((0, t]) = 0 and A(t) = Al.
(7.6)
and
Using an infinitesimal "last step analysis" similar to that used to derive the
Kolmogorov forward equations, we will derive a pair of first-order linear
differential equations for fo(t) and f,(t). To this end, by analyzing the'possibilities at time t and using the law of total probability, we begin with
410
VI
Continuous Time Markov Chains
fo(t + h) =
fa(t) Pr{N((t, t + h]) = OIA(t) = 0} Pr{A(t + h) = OIA(t) = 0)
+ fl(t) Pr{N((t, t + h]) = OJA(t) = A) Pr{A(t + h) = OIA(t) = A)
= fo(t)[1 - ah + o(h)] + f(t)e-A''/ih + o(h),
and
f(t+h)=
fl(t) Pr{N((t, t + h]) = OIA(t) = A) Pr{A(t + h) = AIA(t) = A)
+ fo(t) Pr(N((t, t + h]) = OIA(t) = 01 Pr(A(t + h) = AJA(t) = 0)
= f,(t)e-A''[1 - /3h + o(h)] + fo(t)ah + o(h).
We rearrange the terms and use e-A" = 1 - Ah + o(h) to get
fo(t + h) - fo(t) = -afo(t)h + /3f,(t)h + o(h)
and
f, (t + h) - f, (t) = -(/3 + A)f,(t)h + afa(t)h + o(h),
which, after dividing by h and letting h tend to zero, become the differential equations
dfo(t)
_ -afo(t) + f3ff(t)
(7.7)
= -(/3 + A)f (t) + af0(t).
(7.8)
dt
and
df, (t)
dt
The initial conditions are
fo(0) = Pr(A(0) = 0) = /3/(a + /3)
(7.9)
f(0) = Pr{A(0) = A} = a/(a + /3).
(7.10)
and
Such coupled first-order linear differential equations are readily solved. In
our case, after carrying out the solution and simplifying the result, the an-
swer is Pr{N((0, t]) = 0) = fo(t) + f(t) = f(t; A), where
f(t; A) = c+ exp{-µ+t} + c_ exp{-µ_t)
(7.11)
7. A Poisson ProceM with a Markov Intensity
411
with
p
=;{(A+a+/3)±
(A+a+R)'-4aA},
[aA/(a + /3)] - µ_
(7.12)
(7.13)
c+ =
and
c_ =
µ+ - [aAI(a + /3)]
(7.14)
A Geheralization Let N be a Cox process driven by a 0-1 Markov
chain A(t), but now suppose that when the intensity process is in state 0,
the Cox process is, conditionally, a Poisson process of rate A0, and when
the intensity process is in state 1, then the Cox process is a Poisson
process of rate A,. The earlier Cox process had A. = 0 and A, = A. Without loss of generality, we assume 0 < A. < A,.
In order to evaluate Pr{N((0, t]) = 01, we write N as the sum N =
N, + N, of two independent processes, where N, is a Poisson process of
constant rate A. and N, is a (0, A) Cox process with A = A, - A0. Then N
is zero if and only if both N, and N, are zero, whence
Pr{N((0, t]) = 0) = Pr{N,((0, t]) =
=
e_Ao'
t]) = 01
f(t, A, - A0).
(7.15)
Example The tensile strength S(t) of a single fiber of length t is often
assumed to follow a Weibull distribution of the form
Pr{S(t) > x) = exp{-tox&},
for x > 0,
(7.16)
where 6 and o are positive material constants. The explicit appearance of
the length t in the exponent is an expression of a weakest-link size effect,
in which the fiber strength is viewed as the minimum strength of independent sections. This theory suggests that the survivorship probability of
strength for a fiber of length t should satisfy the relation
Pr{S(t) > x} = [Pr{S(1) > x}]',
t > 0.
(7.17)
The Weibull distribution is the only type of distribution that is concentrated on 0 x < oo and satisfies (7.17).
412
VI
Continuous Time Markov Chains
However, a fiber under stress may fail from a surface flaw such as a
notch or scratch, or from an internal flaw such as a void or inclusion.
Where the diameter d of the fiber varies along its length, the relative magnitude of these two types of flaws will also vary, since the surface of the
fiber is proportional to d, while the volume is proportional to d 2. As a simple generalization, suppose that the two types of flaws alternate and that the
changes from one flaw type to the other follow a two-state Markov chain
along the continuous length of the fiber. Further, suppose that a fiber of
constant type i flaw, for i = 0, 1, will support the load x with probability
Pr(S(t) > x} = exp{-to;x%
x > 0,
where o-; and S; are positive constants.
We can evaluate the survivorship probability for the fiber having
Markov varying flaw types by bringing in an appropriate (A0, A,) Cox
process. For a fixed x > 0, suppose that flaws that are weaker than x will
occur along a fiber of constant flaw type i according to a Poisson process
of rate A,(x) =
for i = 0, 1. A fiber of length t and having Markov
varying flaw types will carry a load of x if and only if there are no flaws
weaker than x along the fiber. Accordingly, for the random flaw type fiber,
using (7.15) we have
Pr(S(t) > x) = e-V'".f(t; 11,(x) - Ao(x)).
(7.18)
Equation (7.18) may be evaluated numerically under a variety of assumptions for comparison with observed fiber tensile strengths. Where
fibers having two flaw types are tested at several lengths, (7.18) may be
used to extrapolate and predict strengths at lengths not measured.
is sometimes more meaningful to reparametrize according to
it = a/(a + /3) and r = a + P. Here IT is the long run fraction of fiber
It
length for which the applicable flaw distribution is of type 1, and 1 - IT
is the similar fraction of type 0 flaw behavior. The second parameter T is
a measure of the rapidity with which the flaw types alternate. In particular, when T = 0, the diameter or flaw type remains in whichever state it
began, and the survivor probability reduces to the mixture
Pr{S(t) > x} = ire-",c' + (1 -
(7.19)
On the other hand, at r = x, the flaw type process alternates instantly, and
the survivor probability simplifies to
Pr{S(t) > x} = exp{-t[irA,(x) + (1 - ir)Ao(x)]}.
(7.20)
413
Exercises
The probability distribution for N((0, t]) Let A(t) be the cumulative intensity for a Cox process and suppose that we have evaluated
0 < 0 < 1.
g(t; 0) = E[e-('-I)A("],
For a (0, A) Cox process, for instance, g(t, 0) = f(t; (1 - 0)A), where f is
defined in (7.11). Upon expanding as a power series in 0, according to
A(t)'
g(t; 0) = E[e-A(')
k, ek]
k=0
-A(r)
A(')k
] 0k
k!
Pr{N((0, t]) = k} ak,
k=0
we see that the coefficient of 0k in the power series is Pr(N((0, t]) = k). In
principle then, the probability distribution for the points in an interval in
a Cox process can be determined in any particular instance.
Exercises
7.1. Suppose that a (0, A) Cox process has a = /3 = 1 and A = 2. Show
that µ. = 2 ± \, and c_ =
;(2
+ V) = 1 - c+, whence
Pr{N((0, t]) = 0) = e- 2 '[cosh(V2-t) +
2 sinh(\t)].
7.2. Suppose that a (0, A) Cox process has a = /3 = 1 and A = 2. Show
that
fO(t) =
1 + V2_
+ l - V Le-(2+N2),
4
4
and
f(t) =
qe-(2-%2)r +
4e-(2+)2)r
satisfy the differential equations (7.7) and (7.8) with the initial conditions
(7.9) and (7.10).
414
VI
Continuous Time Markov Chains
Problems
7.1. Consider a stationary Cox process driven by a two-state Markov
chain. Let it = a/(a + 3) be the probability that the process begins in
state A.
(a) By using the transition probabilities given in (3.14a-d), show that
Pr{A(t) = A } = 7r for all t > 0.
(b) Show that E[N((O, t])] = TTAt for all t > 0.
7.2. The excess life y(t) in a point process is the random length of the
duration from time t until the next event. Show that the cumulative distri-
bution function for the excess life in a Cox process is given by
Pr{y(t) : x} = 1 - Pr{N((t, t + x]) = 0}.
7.3.
Let T be the time to the first event in a stationary (0, A) Cox
process. Find the probability density function 4)(t) for T. Show that when
a = /3 = 1 and A = 2, this density function simplifies to ¢(t) = exp{ -2t}
cosh(Vt).
7.4. Let T be the time to the first event in a stationary (0, A) Cox process.
Find the expected value E[T]. Show that E[T] = 1 when a = 6 = 1 and
A = 2. What is the average duration between events in this process?
7.5. Determine the conditional probability of no points in the interval
(t, t + s], given that there are no points in the interval (0, t] for a stationary Cox process driven by a two-state Markov chain. Establish the limit
lim Pr{N((t, t + s]) = 0IN((0, t]) = 0} = e-'"-5,
7.6. Show that the Laplace transform
x
(P(s; A) =
f e-"f(t; A) dt
0
is given by
s+(1-ir)A+T
Cs; A) = s2 + (T + A)s + 9TTA '
s > 0.
415
Problems
where T = a + /3 and IT = a/(a + /3). Evaluate the limit (a) as T -* 00,
and (b) as T -3 0.
7.7. Consider a (0, A) stationary Cox process with a = /3 = 1 and
A = 2. Show that g(t; 0) = f(t; (1 - 0)A) is given by
g(t; 0) = e-`--m' cosh(Rt) +
1
R
sinh(Rt)
,
where
0'-20+2.
R=
Use this to evaluate Pr{N((0, 1]) = 11.
7.8. Consider a stationary (0, A) Cox process. A long duration during
which no events were observed would suggest that the intensity process is
in state 0. Show that
Pr{A(t) = 0IN((0, t]) = 0) =
fo(t)
.f(t) '
where fo(t) is defined in (7.5).
a.
'(1 -
IT)rl
fo(t) =
a+e--' +
a_e_µ-'
f(t) =
b+e-µ
'+
b_e_µ-'
+
(a+/3+A)jR= (a+ 9+A)2
R
4aA
J
L
R
J
satisfy the differential equations (7.7) and (7.8) subject to the initial conditions (7.9) and (7.10).
416
VI
Continuous Time Markov Chains
7.10. Consider a stationary (0, A) Cox process.
(a) Show that
Pr{N((0, h]) > 0, N((h, h + t]) = 0) = f(t; A) - f(t + h; A),
whence
Pr{N((h h + t]) = 0IN((0 h]) > 01 =
'
f(t; A) - f(t + h; A)
1 - f(h; A)
(b) Establish the limit
lim Pr(N((h, h + t]) = 0IN((0, h]) > 0) =
-10
11
f '(t; A)
f'(0, A)'
where
_ dftt; A)
dt
(c) We interpret the limit in (b) as the conditional probability
Pr (N((O, t]) = 01 Event occurs at time 0).
Show that
Pr{N((0, t]) = 01 Event at time 01 =
p+e-K+' +
p_e-,U-',
where
p+ =
C+µ+
c+µ+ + c-µ-'
c-M--
p- = e+µ+ +
c-lk(d) Let The the time to the first event in (0, c) in a stationary (0, A) Cox
process with a = /3 = 1 and A = 2. Show that
E[rIEvent at time 0] = 1.
Why does this differ from the result in Problem 7.4?
7.11. A Stop-and-Go Traveler The velocity V(t) of a stop-and-go traveler is described by a two-state Markov chain. The successive durations in
which the traveler is stopped are independent and exponentially distributed with parameter a, and they alternate with independent exponentially
417
Problems
distributed sojourns, parameter /3, during which the traveler moves at unit
speed. Take the stationary case in which Pr{V(0) = 1) = Tr = a/(a + /3).
The distance traveled in time t is the integral of the velocity:
I
S(t) =
J V(u) du.
0
Show that
E[e "(')] = f(t; 0),
0
real.
(This is the Laplace transform of the probability density function of S(t).)
7.12. 'Let T be the time of the first event in a (O, A) Cox process. Let the
0 and A states represent "OFF" and "ON," respectively.
(a) Show that the total duration in the (0, T] interval that the system is
ON is exponentially distributed with parameter A and does not depend on a, f3 or the starting state.
(b) Assume that the process begins in the OFF state. Show that the total
duration in the (0, T] interval that the system is OFF has the same
distribution as
is exponentially distributed with parameter A, N(t) is a Poisson
process with parameter /3, and qo, -q,, ... are independent and exponentially distributed with parameter a.
where
Chapter VII
Renewal Phenomena
1.
Definition of a Renewal Process
and Related Concepts
Renewal theory began with the study of stochastic systems whose evolution through time was interspersed with renewals or regeneration times
when, in a statistical sense, the process began anew. Today, the subject is
viewed as the study of general functions of independent, identically distributed, nonnegative random variables representing the successive intervals between renewals. The results are applicable in a wide variety of both
theoretical and practical probability models.
A renewal (counting) process {N(t), t > 0} is a nonnegative integervalued stochastic process that registers the successive occurrences of an
event during the time interval (0, t], where the times between consecutive
events are positive, independent, identically distributed random variables.
Let the successive durations between events be {Xk}A , (often representing
the lifetimes of some units successively placed into service) such that X;
is the elapsed time from the (i - 1)st event until the occurrence of the ith
event. We write
F(x)=Pr{Xksx},
k= 1,2,3,...,
Renewal Phenomena
VII
420
for the common probability distribution of X X2..... A basic stipulation
for renewal processes is F(O) = 0, signifying that X,, X,, ... are positive
random variables. We refer to
n?1
(W0 = 0 by convention),
as the waiting time until the occurrence of the nth event.
The relation between the interoccurrence times {Xk} and the renewal
counting process {N(t), t ? 0) is depicted in Figure 1.1. Note formally that
N(t) = number of indices n for which 0 < W :5 t.
(1.2)
N(t)
010
X1-
WO
Figure 1.1
W,
X, -0 -X3
W,
{
t
W3
The relation between the interoccurrence times Xk and the
renewal counting process N(t).
In common practice the counting process {N(t), t ? 0) and the partial
sum process { W,,, n ? 0) are interchangeably called the "renewal
process." The prototypical renewal model involves successive replacements of light bulbs. A bulb is installed for service at time WO = 0, fails at
time W, = X,, and is then exchanged for a fresh bulb. The second bulb
fails at time W, = X, + X, and is replaced by a third bulb. In general, the
nth bulb burns out at time W = X, +
+ X and is immediately replaced, and the process continues. It is natural to assume that the successive lifetimes are statistically independent, with probabilistically identical
1.
Definition of a Renewal Process and-Related Concepts
421
characteristics in that
Pr{XA : x} = F(x)
for k = 1, 2,
... .
In this process N(t) records the number of light-bulb replacements up to
time t.
The principal objective of renewal theory is to derive properties of certain random variables associated with { N(t) } and { W,,) from knowledge of
the interoccurrence distribution F. For example, it is of significance and
relevance to compute the expected number of renewals for the time duration (0, t]:
E[N(t)] = M(t)
is called the renewal function. To this end, several pertinent relationships
and formulas are worth recording. In principle, the probability law of
+ X,, can be calculated in accordance with the convolution
W = X, +
formula
Pr{ W <_ x} =
where F,(x) = F(x) is assumed known or prescribed, and then
x
F,,(x) =
f
0
X
y) dF(y) = fF,,-,(x - y) dF(y).
0
Such convolution formulas were reviewed in I, Section 2.5.
The fundamental connecting link between the waiting time process
{ W,} and the renewal counting process {N(t)} is the observation that
N(t) ? k if and only if W
t.
(1.3)
In words, equation (1.3) asserts that the number of renewals up to time t
is at least k if and only if the kth renewal occurred on or before time t.
Since this equivalence is the basis for much that follows, the reader should
verify instances of it by referring to Figure I.I.
422
VII
Renewal Phenomena
It follows from (1.3) that
Pr{N(t)? k) =Pr(Wkst}
(1.4)
t?O,k= 1,2,...,
=Fk(t),
and consequently,
Pr{N(t) = k} = Pr{N(t) ? k} - Pr (N(t) ? k + 1 }
(1.5)
t?O,k= 1,2,....
Fk(t)-Fk+I(t),
For the renewal function M(t) = E[N(t)] we sum the tail probabilities in
the manner E[N(t)] Ek., Pr{N(t) ? k}, as derived in I, (5.2), and then
use (1.4) to obtain
x
M(t) = E[N(t)] = 7, Pr{N(t) ? k}
k=1
F(t).
Pr{Wkst}
(1.6)
k=1
k=1
There are a number of other random variables of interest in renewal
theory. Three of these are the excess life (also called the excess random
variable), the current life (also called the age random variable), and the
total life, defined, respectively, by
y, = WN(6)+I - t
(excess or residual lifetime),
6, = t - WN(,)
(current life or age random variable),
/3, = y, + 6,
(total life).
A pictorial description of these random variables is given in Figure 1.2.
N(t)
,
'Y,
WO
t
WN(,) + I
t
Figure 1.2 The excess life y,, the current life S and the total life 0,.
1.
Definition of a Renewal Process and Related Concepts
423
An important identity enables us to evaluate the mean of
in terms
of the mean lifetime µ = E[X,] of each unit and the renewal function
M(t). Namely, it is true for every renewal process that
E[Wv(n+i] = E[X, + ... + X,v(,)+i]
= E[X,]{E[N(t) + 1]},
or
{M(t) + 1).
(1.7)
At first glance, this identity resembles the formula given in II, (3.9) for the
mean of a random sum, which asserts that E[X, +
+ X,,,] = E[X,]E[N]
when N is an integer-valued random variable that is independent of X,
X,..... The random sum approach does not apply in the current context,
however, the crucial difference being that the random number of summands N(t) + 1 is not independent of the summands themselves. Indeed,
in Section 3, on the Poisson process viewed as a renewal process, we will
show that the last summand X,N,,,+, has a mean that approaches twice the
unconditional mean µ = E[X,] for t large. For this reason, it is not correct,
in particular, that E[W,v,,,,] can be evaluated as the product of E[X,] and
E[N(t)]. In view of these comments, the identity expressed in equation
(1.7) becomes more intriguing and remarkable.
To derive (1.7), we will use the fundamental equivalence (1.3) in the
form
N(t)?j- 1
if and only if
t,
which expressed in terms of indicator random variables becomes
1{N(t)?j- 1} =1{X, +- +X,_,:5 t}.
Since this indicator random variable is a function only of the random variables X ... , X;_ , it is independent of X,, and thus we may evaluate
t}]
(1.8)
µF-1(t).
With (1.8) in hand, the evaluation of the equivalence expressed in (1.7)
424
VII
Renewal Phenomena
becomes straightforward. We have
E[WN(,)+l] = E[X, + ... + 40+i]
IN(t)+I
= E[X,] + EL 7,
11
Xi]
j=2
µ+E[ >X;1{N(t)+ 1?j
LJ-2
i=2
=µ+µ
F_#)
(using (1.8))
i=2
= µ[ 1 + M(t)]
(using (1.6)).
Some examples of the use of the identity (1.7) will appear in the exercises,
and an alternative proof in the case of a discrete renewal process can be
found in Section 6.
Exercises
1.1. Verify the following equivalences for the age and the excess life in
a renewal process N(t):
y, > x if and only if N(t + x) - N(t) = 0;
and for 0 < x < t,
S,>x if and only if N(t)-N(t-x)=0.
Why is the condition x < t important in the second case but not the first?
1.2.
Consider a renewal process in which the interoccurrence times
have an exponential distribution with parameter A:
f(x) = Ae-',
and
F(x) = 1 - e-a"
for x > 0.
Calculate F(t) by carrying out the appropriate convolution [see the
equation just prior to (1.3)] and then determine Pr(N(t) = 1 } from equation (1.5).
425
Problems
1.3.
Which of the following are true statements?
(a) N(t) < k if and only if Wk > t.
(b) N(t) :5 k if and only if W4. ?t.
(c) N(t) > k if and only if Wk < t.
1.4. Consider a renewal process for which the lifetimes X,, X,, ... are
discrete random variables having the Poisson distribution with mean A.
That is,
e-An
Pr{Xk=n}_
forn=0,1,....
A
n!
(a) What is the distribution of the waiting time Wk?
(b) Determine Pr{N(t) = k).
Problems
1.1. Verify the following equivalences for the age and the excess life in
a renewal process N(t): (Assume t > x.)
Pr{S,?x,y,>y} =Pr{N(t-x)=N(t+y)}
Pr{Wk<t-x,Wk+i>t+y}
k=0
=[1-F(t+y)]
t-x
f
+
k=1
[I - F(t + y - z)] dFk(z).
0
Carry out the evaluation when the interoccurrence times are exponentially
distributed with parameter A, so that dFk is the gamma density
Akzk-1
dFk(z) =
1.2.
(k
- 1)! e AZ dz
for z > 0.
From equation (1.5), and for k ? 1, verify that
Pr{N(t)=k} =Pr{Wk:t<Wk+,}
= J[1 - F(t - x)] dFk(x),
426
VII
Renewal Phenomena
and carry out the evaluation when the interoccurrence times are exponentially distributed with parameter A, so that dFk is the gamma density
AkZk-i
dFA(z) =
(k _ 1) e
A= dz
for z > 0.
1.3. A fundamental identity involving the renewal function, valid for all
renewal processes, is
E[Wv(,,+i] = E[X1](M(t) + 1).
See equation (1.7). Using this identity, show that the mean excess life can
be evaluated in terms of the renewal function via the relation
E[y] = E[X,](1 + M(t)) - t.
1.4. Let y, be the excess life and S, the age in a renewal process having
interoccurrence distribution function F(x). Determine the conditional
probability Pr{ y, > yes, = x} and the conditional mean E[y,IS, = x].
2.
Some Examples of Renewal Processes
Stochastic models often contain random times at which they, or some part
of them, begin afresh in a statistical sense. These renewal instants form
natural embedded renewal processes, and they are found in many diverse
fields of applied probability including branching processes, insurance risk
models, phenomena of population growth, evolutionary genetic mechanisms, engineering systems, and econometric structures. When a renewal
process is discovered embedded within a model, the powerful results of
renewal theory become available for deducing implications.
2.1.
Brief Sketches of Renewal Situations
The synopses that follow suggest the wide scope and diverse contexts in
which renewal processes arise. Several of the examples will be studied in
more detail in later sections.
2.
Some Examples of Fnewal Processes
427
(a) Poisson Processes A Poisson process (N(t), t ? 01 with parameter
A is a renewal counting process having the exponential interoccurrence
distribution
F(x) = I - e-',
x ? 0,
as established in V, Theorem 3.2. This particular renewal process possesses a host of special features, highlighted later in Section 3.
(b) Counter Processes The times between successive electrical impulses or signals impinging on a recording device (counter) are often assumed to form a renewal process. Most physically realizable counters
lock for some duration immediately upon registering an impulse and will
not record impulses arriving during this dead period. Impulses are
recorded only when the counter is free (i.e., unlocked). Under quite reasonable assumptions, the sequence of events of the times of recorded impulses forms a renewal process, but it should be emphasized that the re-
newal process of recorded impulses is a secondary renewal process
derived from the original renewal process comprising the totality of all arriving impulses.
(c) Traffic Flow The distances between successive cars on an indefinitely long single-lane highway are often assumed to form a renewal
process. So also are the time durations between consecutive cars passing
a fixed location.
(d) Renewal Processes Associated with Queues In a single-server
queueing process there are embedded many natural renewal processes. We
cite two examples:
(i) If customer arrival times form a renewal process, then the times of
the starts of successive busy periods generate a second renewal
process.
(ii) For the situation in which the input process (the arrival pattern of
customers) is Poisson, the successive moments in which the server
passes from a busy to a free state determine a renewal process.
(e) Inventory Systems In the analysis of most inventory processes it is
customary to assume that the pattern of demands forms a renewal process.
Most of the standard inventory policies induce renewal sequences, e.g.,
the times of replenishment of stock.
428
VII
Renewal Phenomena
(f) Renewal Processes in Markov Chains Let Z0, Z,, . . . be a recurrent
Markov chain. Suppose Z = i, and consider the durations (elapsed number of steps) between successive visits to state i. Specifically, let W,, = 0,
W,=min[ n>O;Z,, =i},
and
W+, =min{n>W;Z,, =i},
k= 1,2.....
Since each of these times is computed from the same starting state i, the
Markov property guarantees that Xk = W - W,, are independent and
identically distributed, and thus {XA} generates a renewal process.
2.2.
Block Replacement
Consider a light bulb whose life, measured in discrete units, is a random
variable X, where Pr{X = k} = p, for k = 1, 2, .... Assuming that one
starts with a fresh bulb and that each bulb is replaced by a new one when
it burns out, let M(n) = E[N(n)] be the expected number of replacements
up to time n.
Because of economies of scale, in a large building such as a factory or
office it is often cheaper, on a per bulb basis, to replace all the bulbs, failed
or not, than it is to replace a single bulb. A block replacement policy attempts to take advantage of this reduced cost by fixing a block period K
and then replacing bulbs as they fail during periods 1, 2, ... , K - 1, and
replacing all bulbs, failed or not, in period K. This strategy is also known
as "group relamping." If c, is the per bulb block replacement cost and
c, is the per bulb failure replacement cost (c, < c,), then the mean total
cost during the block replacement cycle is c, + c,M(K - 1), where
M(K - 1) = E[N(K - 1)] is the mean number of failure replacements.
Since the block replacement cycle consists of K periods, the mean total
cost per bulb per unit time is
+ c,M(K - 1)
0(K) = c,
K
If we can determine the renewal function M(n) from the life distribution
{ pj, then we can choose the block period K = K* so as to minimize the
cost rate 0(K). Of course, this cost must be compared to the cost of replacing only upon failure.
The renewal function M(n), or expected number of replacements up to
2.
Some Examples of Renewal Processes
429
time n, solves the equation
I
pkM(n - k)
M(n) = FX(n) +
for n = 1, 2,
... .
k=I
To derive this equation, condition on the life X, of the first bulb. If it fails
after time n, there are no replacements during periods [1, 2, ... , n]. On
the other hand, if it fails at time k < n, then we have its failure plus, on
the average, M(n - k) additional replacements during the interval [k + 1,
k + 2, ... , n]. Using the law of total probability to sum these contributions, we obtain
x
n
pk(0) + L pk[ 1 + M(n - k)]
M(n) =
k=n+1
k=1
n-I
= FX(n) + 7, pkM(n - k)
[because M(0) = 0],
k=1
as asserted.
Thus we determine
M(1) = FX(1),
M(2) = FX(2) + p,M(1),
M(3) = FX(3) + p,M(2) + p,M(1),
and so on.
To consider a numerical example, suppose that
p, = 0.1,
p2 = 0.4,
p, = 0.3,
and
p4 = 0.2,
and
c,=2 and c,=3.
Then
M(1) = p, = 0.1,
M(2)=(p,+P2)+p,M(1)=(0.1 +0.4)+0.1(0.1)=0.51,
M(3)=(p, +p2+p,)+p,M(2)+p2M(1)
= (0.1 +0.4+0.3)+0.1(0.51)+0.4(0.1)=0.891,
M(4)=(p, +p2+p,+ p4)+p,M(3)+p,M(2)+p,M(l)
= 1 + 0.1(0.891) + 0.4(0.51) + 0.3(0.1) = 1.3231.
430
VII
Renewal Phenomena
The average costs are shown in the following table:
Bl oc k Per i od
K
Cost =
+ c,M(K - 1)
K
C1
= 0(K)
2.00000
1.15000
1.17667
1
2
3
4
1.16825
1.19386
5
The minimum cost block period is K* = 2.
We wish to elicit one more insight from this example. Forgetting about
block replacement, we continue to calculate
M(5) = 1.6617,
M(6) = 2.0647,
M(7) = 2.4463,
M(8) = 2.8336,
M(9) = 3.2136,
M(10) = 3.6016.
Let u,, be the probability that a replacement occurs in period n. Then M(n)
= M(n - 1) + u asserts that the mean replacements up to time n is the
mean replacements up to time n - 1 plus the probability that a replacement
occurs in period n. The calculations are shown in the following table:
n
1
2
3
4
5
6
7
8
9
10
u,, = M(n) - M(n - 1)
0.1000
0.4100
0.3810
0.4321
0.3386
0.4030
0.3816
0.3873
0.3800
0.3880
431
Exercises
The probability of a replacement in period n seems to be converging.
This is indeed the case, and the limit is the reciprocal of the mean bulb
lifetime:
1
E[X,]
_
1
0.1(1) + 0.4(2) + 0.3(3) + 0.2(4)
This calculation makes sense. If a light bulb lasts, on the average, E[X,]
time units, then the probability that it will need to be replaced in any period should approximate 1/E[X,]. Actually, the relationship is not as simple as just stated. Further discussion takes place in Sections 4 and 6.
Exercises
Let {X,,; n = 0, 1, ...) be a two-state Markov chain with the transition probability matrix
2.1.
0
P=0
l-a
b
1
a
1 -b'
State 0 represents an operating state of some system, while state 1 represents a repair state. We assume that the process begins in state X0 = 0,
and then the successive returns to state 0 from the repair state form a
renewal process. Determine the mean duration of one of these renewal
intervals.
2.2. A certain type component has two
states:
0 = OFF and
1 = OPERATING. In state 0, the process remains there a random length
of time, which is exponentially distributed with parameter a, and then
moves to state 1. The time in state 1 is exponentially distributed with parameter 0, after which the process returns to state 0.
The system has two of these components, A and B, with distinct
parameters:
Component
Operating Failure Rate
Repair Rate
A
B
Al
a,
Nf3
aR
432
VII
Renewal Phenomena
In order for the system to operate, at least one of components A and B
must be operating (a parallel system). Assume that the component stochastic processes are independent of one another. Consider the successive
instants that the system enters the failed state from an operating state. Use
the memoryless property of the exponential distribution to argue that these
instants form a renewal process.
2.3. Calculate the mean number of renewals M(n) = E[N(n)] for the renewal process having interoccurrence distribution
for n = 1, 2,
... , 10. Also calculate u = M(n) - M(n - 1).
Problems
2.1. For the block replacement example of this section for which
p, = 0.1, P2 = 0.4, p3 = 0.3, and p, = 0.2, suppose the costs are c, = 4
and c2 = 5. Determine the minimal cost block period K* and the cost of
replacing upon failure alone.
2.2. Let X,, X2, ... be the interoccurrence times in a renewal process.
Suppose Pr(Xk = 1) = p and Pr{Xk = 2} = q = 1 - p. Verify that
M(n) = E[N(n)] =
n
-
1+q
qz
(1 + q)2
rl
-(
q) ]
forn=2,4,6.....
2.3.
Determine M(n) when the interoccurrence times have the geo-
metric distribution
Pr{X,=k)=pk=(3(l-I3) k-'
fork=1,2,...,
where 0</3< 1.
3.
The Poisson Process Viewed as a Renewal Process
As mentioned earlier, the Poisson process with parameter A is a renewal
process whose interoccurrence times have the exponential distribution
F(x) = 1 - e-', x ? 0. The memoryless property of the exponential dis-
3.
The Poisson Process Viewed as a Renewal Process
433
tribution (see Sections 4.2, 5.2 of I, and V) serves decisively in yielding
the explicit computation of a number of functionals of the Poisson renewal process.
The Renewal Function Since N(t) has a Poisson distribution, then
Pr{N(t)=k]
(At)ke-rt
ki
,
k=0,1,...,
and
M(t) = E[N(t)] = At.
Excess Life Observe that the excess life at time t exceeds x if and only
if there are no renewals in the interval (t, t + x] (Figure 3.1). This event
has the same probability as that of no renewals in the interval (0, x], since
a Poisson process has stationary independent increments. In formal terms,
we have
Pr{y, > x} = Pr{N(t + x) - N(t) = 0}
(3.1)
= Pr(N(x) = 0) = e-'.
Thus, in a Poisson process, the excess life possesses the same exponential
distribution
Pr{y, : x) = 1 - e-1,
x ? 0,
(3.2)
as every life, another manifestation of the memoryless property of the exponential distribution.
N(t)
t
t+x
Figure 3.1 The excess life y, exceeds x if and only if there are no renewals
in the interval (t, t + x].
434
VII
Renewal Phenomena
Current Life The current life S of course, cannot exceed t, while for
x < t the current life exceeds x if and only if there are no renewals in
(t - x, t], which again has probability e". Thus the current life follows
the truncated exponential distribution
Pr I{ o, < x }
for0<x<t,
=
for t s x.
1
(3.3 )
Mean Total Life Using the evaluation of I, (5.3) for the mean of a non- negative random variable, we have
E[t3,] = E[y,] + E[8,]
=
+JPr{S,>x}dx
0
.+ Jedx
0
=I+
I
(I - e-a,).
Observe that the mean total life is significantly larger than the mean life
1/A = E[Xk] of any particular renewal interval. A more striking expression
of this phenomenon is revealed when t is large, where the process has
been in operation for a long duration. Then the mean total life E[/3,] is approximately twice the mean life. These facts appear at first paradoxical.
Let us reexamine the definition of the total life /3, with a view to explaining on an intuitive basis the seeming discrepancy. First, an arbitrary
time point t is fixed. Then /3, measures the length of the renewal interval
containing the point t. Such a procedure will tend with higher likelihood
to favor a lengthy renewal interval rather than one of short duration. The
phenomenon is known as length-biased sampling and occurs, well disguised, in a number of sampling situations.
Joint Distribution of y, and St The joint distribution of y, and S, is
determined in the same manner as the marginals. In fact, for any x > 0 and
0 < y < t, the event { y, > x, 5, > y) occurs if and only if there are no renewals in the interval (t - y, t + x], which has probability e-A``+`". Thus
435
Problems
Pr{y,>x, S,>y} =
ifx>0,0<y<
{e_MY)
ify>t.
0
(3.4)
For the Poisson process, observe that y, and S, are independent, since their
joint distribution factors as the product of their marginal distributions.
Exercises
3.1. Let W,, W,, ... be the event times in a Poisson process (X(t); t ? 0)
of rate A. Evaluate
Pr{WN(,)+, > t + s}
and
E[WN(,)+I]
3.2. Particles arrive at a counter according to a Poisson process of rate
A. An arriving particle is recorded with probability p and lost with probability 1 - p independently of the other particles. Show that the sequence
of recorded particles is a Poisson process of rate Ap.
3.3. Let W,, W,, ... be the event times in a Poisson process (N(t); t ? 0)
of rate A. Show that
N(t)
and
WN(,)+I
are independent random variables by evaluating
Pr{N(t) = n and
WN(,)+, > t + s).
Problems
3.1. In another form of sum quota sampling (see V, Section 4.2), a sequence of nonnegative independent and identically distributed random
variables X XZ, ... is observed, the sampling continuing until the first
time that the sum of the observations exceeds the quota t. In renewal
process terminology, the sample size is N(t) + 1. The sample mean is
WN(,)+I
N(t) + 1
X, + . . . + VN(,)+I
N(t) + 1
An important question in statistical theory is whether or not this sample
mean is unbiased. That is, how does the expected value of this sample
436
VII
Renewal Phenomena
mean relate to the expected value of, say, X,? Assume that the individual
X summands are exponentially distributed with parameter A, so that N(t)
is a Poisson process, and evaluate the expected value of the foregoing
sample mean and show that
Wv(n+,
EN(t)+1
Hint:
1l
=-L[1
Use the result of the previous exercise, that
WN(,)+,
and
N(t)
are independent, and then evaluate separately
E[WNU,+,I
and
E(
1
L N(t) + 1
3.2. A fundamental identity involving the renewal function, valid for all
renewal processes, is
E[%(,)+1] = E[X,](M(t) + 1).
See equation (1.7). Evaluate the left side and verify the identity when the
renewal counting process is a Poisson process.
3.3.
Pulses arrive at a counter according to a Poisson process of rate A.
All physically realizable counters are imperfect, incapable of detecting all
signals that enter their detection chambers. After a particle or signal ar-
rives, a counter must recuperate, or renew itself, in preparation for the
next arrival. Signals arriving during the readjustment period, called dead
time or locked time, are lost. We must distinguish between the arriving
particles and the recorded particles. The experimenter observes only the
particles recorded; from this observation he desires to infer the properties
of the arrival process.
Suppose that each arriving pulse locks the counter for a fixed time T.
Determine the probability p(t) that the counter is free at time t.
3.4. This problem is designed to aid in the understanding of lengthbiased sampling. Let X be a uniformly distributed random variable on
[0, 1]. Then X divides [0, 1] into the subintervals [0, X] and (X, 1]. By
symmetry, each subinterval has mean length ;. Now pick one of these
subintervals at random in the following way: Let Y be independent of
4.
The Asymptotic Behavior of Renewal Processes
437
X and uniformly distributed on [0, 1], and pick the subinterval [0, X] or
(X, 1] that Y falls in. Let L be the length of the subinterval so chosen.
Formally,
L=
X
if Y<X,
1 -X
ifY>X.
Determine the mean of L.
3.5.
Birds are perched along a wire as shown according to a Poisson
process of rate A per unit distance:
t
0
D(t)-I
At a fixed point t along the wire, let D(t) be the random distance to the
nearest bird. What is the mean value of D(t)? What is the probability density functionf(x) for D(t)?
4.
The Asymptotic Behavior of Renewal Processes
A large number of the functionals that have explicit expressions for
Poisson renewal processes are far more difficult to compute for other renewal processes. There are, however, many simple formulas that describe
the asymptotic behavior, for large values of t, of a general renewal
process. We summarize some of these asymptotic results in this section.
4.1.
The Elementary Renewal Theorem
The Poisson process is the only renewal process (in continuous time)
whose renewal function M(t) = E[N(t)] is exactly linear. All renewal
functions are asymptotically linear, however, in the sense that
lim M(t) = lim
1-,t
f-1=
E[N(t)] = 1
t
A
,
(4.1)
438
VII
Renewal Phenomena
where µ = E[XA] is the mean interoccurrence time. This fundamental
result, known as the elementary renewal theorem, is invoked repeatedly
to compute functionals describing the long run behavior of stochastic
models having renewal processes associated with them.
The elementary renewal theorem (4.1) holds even when the interoccurrence times have infinite mean, and then lim, M(t)/t = 1/ce = 0.
The elementary renewal theorem is so intuitively plausible that it has
often been viewed as obvious. The left side, lim, M(t)/t, describes the
long run mean number of renewals or replacements per unit time. The
right side, 1/µ, is the reciprocal of the mean life of a component. Isn't it
obvious that if a component lasts, on the average, µ time units, then in the
long run these components will be replaced at the rate of 1/µ per unit
time? However plausible and convincing this argument may be, it is not
obvious, and to establish the elementary renewal theorem requires several
steps of mathematical analysis, beginning with the law of large numbers.
As our main concern is stochastic modeling, we omit this derivation, as
well as the derivations of the other asymptotic results summarized in this
section, in order to give more space to their application.
Example Age Replacement Policies Let X, X,, .. . represent the
lifetimes of items (light bulbs, transistor cards, machines, etc.) that are
successively placed in service, the next item commencing service immediately following the failure of the previous one. We stipulate that {Xk } are
independent and identically distributed positive random variables with finite mean µ = E[XA]. The elementary renewal theorem tells us to expect
to replace items over the long run at a mean rate of 1/µ per unit time.
In the long run, any replacement strategy that substitutes items prior to
their failure will use more than 1/µ items per unit time. Nonetheless,
where there is some benefit in avoiding failure in service, and where units
deteriorate, in some sense, with age, there may be an economic or reliability advantage in considering alternative replacement strategies. Telephone or utility poles serve as good illustrations of this concept. Clearly,
it is disadvantageous to allow these poles to fail in service because of the
damage to the wires they carry, the damage to adjoining property, overtime wages paid for emergency replacements, and revenue lost while service is down. Therefore, an attempt is usually made to replace older utility poles before they fail. Other instances of planned replacement occur in
preventative maintenance strategies for aircraft, where "time" is now
measured by operating hours.
4.
The Asymptotic Behavior of Renewal Processes
439
An age replacement policy calls for replacing an item upon its failure
or upon its reaching age T, whichever occurs first. Arguing intuitively, we
would expect that the long run fraction of failure replacements, items that
fail before age T, will be F(T), and the corresponding fraction of (conceivably less expensive) planned replacements will be I - F(T). A renewal interval for this modified age replacement policy obviously follows
a distribution law
for x < T,
for x ? T,
FT(x) _ IF(x)
1
and the mean renewal duration is
f
T
AT = f (I - FT (x) } dC = f { 1 - F(x) } dr < A.
0
0
The elementary renewal theorem indicates that the long run mean replacement rate under age replacement is increased to 1/µ T.
Now, let Y,, Y,, ... denote the times between actual successive failures.
The random variable Y, is composed of a random number of time periods
of length T (corresponding to replacements not associated with failures),
plus a last time period in which the distribution is that of a failure conditioned on failure before age T; that is, Y, has the distribution of NT + Z,
where
Pr{N ? k} = { 1 - F(T)}",
k = 0, 1,
... ,
and
0<z<T.
Pr{Z<_z} = F(z)
F(T)'
Hence,
T
E[Y,] = F(T) T[1 - F(T)] + f(F(T) - F(x)) dx
0
1
=F(T)f{1-F(x)}dx=F(T).
0
The sequence of random variables for interoccurrence times of the bona
fide failure {Y} generates a renewal process whose mean rate of failures
per unit time in the long run is 1/E[Y,]. This inference again relies on the
440
VII
Renewal Phenomena
elementary renewal theorem. Depending on F, the modified failure rate
1/E[Y,] may possibly yield a lower failure rate than 1/µ, the rate when replacements are made only upon failure.
Let us suppose that each replacement, whether planned or not, costs K
dollars, and that each failure incurs an additional penalty of c dollars.
Multiplying these costs by the appropriate rates gives the long run mean
cost per unit time as a function of the replacement age T:
_
K
C
C(T) lur
-+
E[YE]
K + cF(T )
f o [ l - F(x)] dx .
In any particular situation, a routine calculus exercise or recourse to numerical computation produces the value of T that minimizes the long run
cost rate. For example, if K = 1, c = 4, and lifetimes are uniformly distributed on [0, 1], then F(x) = x for 0 :5 x : 1, and
f [F(x)] dx = T I 1-
2 T)
\\
0
and
C(T) =
1 + 4T
T(1 - T/2)
To obtain the cost minimizing T, we differentiate C(T) with respect to T
and equate to zero, thereby obtaining
dC(T)
dT
-- 0-- 4T(1 - T/2) - (1 + 4T)(1 - T)
[T(1
- T/2)]2
0=4T-2T2- 1 +T-4T+4T2,
0=2T2+T- 1,
T=
-1±V1_ _+8
1
4
and the optimal choice is T* = 2. Routine calculus will verify that this
choice leads to a minimum cost, and not a maximum or inflection point.
4.
The Asymptotic Behavior of Renewal Processes
4.2.
441
The Renewal Theorem for Continuous Lifetimes
The elementary renewal theorem asserts that
lim
!-x
M(t)
t
-
1
= /I.
It is tempting to conclude from this that M(t) behaves like t/µ as t grows
large, but the precise meaning of the phrase "behaves like" is rather subtle. For example, suppose that\all of the lifetimes are deterministic, say
X, = 1 for k = 1, 2, .... Then it is straightforward to calculate
M(t) = N(t) = 0
for 0
t < 1,
=1
for I
t < 2,
k
1.
That is, M(t) = [t], where [t] denotes the greatest integer not exceeding t.
Since A = 1 in this example, then M(t) - t/µ = [t] - t, a function that oscillates indefinitely between 0 and -1. While it remains true in this illustration that M(t)/t = [t]/t- 1 = l/µ, it is not clear in what sense M(t) "behaves like" t/µ. If we rule out the periodic behavior that is exemplified in
the extreme by this deterministic example, then M(t) behaves like t/µ in
the sense described by the renewal theorem, which we now explain. Let
M(t, t + h] = M(t + h) - M(t) denote the mean number of renewals in
the interval (t, t + h]. The renewal theorem asserts that when periodic behavior is precluded, then
limM(t, t + h] = h/µ
for any fixed h > 0.
(4.2)
In words, asymptotically, the mean number of renewals in an interval is
proportional to the interval's length, with proportionality constant 1/µ.
A simple and prevalent situation in which the renewal theorem (4.2) is
valid occurs when the lifetimes X X2, ... are continuous random variables having the probability density function f(x). In this circumstance,
the renewal function is differentiable, and
m(t) =
dM(t)
dt
= ,t_j
7 x (t),
where f(t) is the probability density function for W = X, +
Now (4.2) may be written in the form
(4.3)
+ X,,.
442
VII
M(t+h)-M(t)
h
- -µ
1
ast -->
Renewal Phenomena
00,
which, when h is small, suggests that
lim
m(t) = lim
,mar
ty-
dM(t) =
1
(4.4)
dt
and indeed, this is the case in all but the most pathological of circumstances when X1, X,, . . . are continuous random variables.
If in addition to being continuous, the lifetimes X, X,.... have a finite
mean µ and finite variance 0.2, then the renewal theorem can be refined to
include a second term. Under the stated conditions, we have
lim I
M(t) -
t
Cr2
µ
2p-
Example When the lifetimes X,, X,,
...
µ,
(4.5)
have the gamma density
function
f(x) = xe-'
for x > 0,
+ X have the gamma density
then the waiting times W,, = X, +
f (x) =
x
(2n - 1)!
(4.6)
e-`
for x > 0,
as may be verified by performing the appropriate convolutions. (See I,
Section 2.5.) Substitution into (4.3) yields
.f (x) _
m(x)
and
M(t) =
Jm(x)dx__t- 4 [1 - e--`].
0
Since the gamma density in (4.6) has moments µ = 2 and 0.2 = 2, we
4.
The Asymptotic Behavior of Renewal Processes
443
verify that m(t)-*1/µ as t--->oo and M(t) - tlµ--1 = (v.2 - µ2)12µ2, in
agreement with (4.4) and (4.5).
4.3.
The Asymptotic Distribution of N(t)
The elementary renewal theorem
--
E[N(t)]
1
lim-' t
(4.7)
A
implies that the asymptotic mean of N(t) is approximately t/µ. When
E[Xj -and 0.2 = Var[XA ] = E[(Xk - µ)'-] are finite, then the asymptotic variance of N(t) behaves according to
lim
t--
Var[N(t)]
t
=
V
,
(4.8)
.
µ
That is, the asymptotic variance of N(t) is approximately to-'/µ3. If we
standardize N(t) by subtracting its asymptotic mean and dividing by its
asymptotic standard deviation, we get the following convergence to the
normal distribution:
lim Pr
(N(t) - tlµ
tW21µ3
}
x =
X
1
J e
dy.
2
In words, for large values of t the number of renewals N(t) is approximately normally distributed with mean and variance given by (4.7) and
(4.8), respectively.
4.4.
The Limiting Distribution of Age and Excess Life
Again we assume that the lifetimes X,, X2, . . . are continuous random
variables with finite mean µ. Let y, = WV,,,+, - t be the excess life at time
t. The excess life has the limiting distribution
lim Pr{y, 5 x} =
1
r
J
co
[1 - F(y)] dy.
(4.9)
VII
444
Renewal Phenomena
The reader should verify that the right side of (4.9) defines a valid distribution function, which we denote by H(x). The corresponding probability
density function is h(y) = µ ' [ 1 - F(y)]. The mean of this limiting distribution is determined according to
x
00
f yh(y) dy = 1 Jy[1 - F(y)] dy
µ'o
o
x
x
= 1 JY{Jf(t) dt dy
J f(t) J y \dy dt
µv
o
x
1
f t2f(t) dt
2µ
o
a.2+µ'=
2µ
'
where u2 is the common variance of the lifetimes X,, XZ....
.
{y,?xand6,?y} ifandonlyif {y,_y._x+y}.
It follows that
limPr{y, a: x, 3,?y} = limPr{y,_,?x + y)
x
= µ-' J [1 - F(z)] dz,
x+y
exhibiting the joint limiting distribution of (y S). In particular,
limPr{S, ? y} = limPr{y, ? 0, S, ? y}
x
= µ ' J [1 - F(z)] dz
y
= 1 - H(y).
(4.10)
445
Exercises
The limiting distribution for the current life, or age, S, = t - W,(,) can
be deduced from the corresponding result (4.9) for the excess life. With
the aid of Figure 4.1, corroborate the equivalence
N(t)
r-J
_X + y
-
fE- Y -)FF- x _0
I
Figure4.1
'Y,
t
{5,>--sand y, ?x) if and only if {y,_,.>_x+y}.
Exercises
4.1. Consider the triangular lifetime density f(x) = 2x for 0 < x < 1.
Determine an asymptotic expression for the expected number of renewals
up to time t.
Hint:
Use equation (4.5).
4.2. Consider the triangular lifetime density f(x) = 2x for 0 < x < 1.
Determine an asymptotic expression for the probability distribution of excess life. Using this distribution, determine the limiting mean excess life
and compare with the general result following equation (4.9).
4.3. Consider the triangular lifetime density function f(x) = 2x, for
0 < x < 1. Determine the optimal replacement age in an age replacement
model with replacement cost K = 1 and failure penalty c = 4 (cf. the example in Section 4.1).
446
VII
Renewal Phenomena
4.4.
Show that the optimal age replacement policy is to replace upon
failure alone when lifetimes are exponentially distributed with parameter
A. Can you provide an intuitive explanation?
4.5. What is the limiting distribution of excess life when renewal lifetimes have the uniform density f(x) = 1, for 0 < x < 1?
4.6. A machine can be in either of two states: "up" or "down." It is up
at time zero and thereafter alternates between being up and down. The
lengths X,, X,.... of successive up times are independent and identically
distributed random variables with mean a, and the lengths Y,, Y,, ... of
successive down times are independent and identically distributed with
mean /3.
(a) In the long run, what fraction of time is the machine up?
(b) If the machine earns income at a rate of $13 per unit time while up,
what is the long run total rate of income earned by the machine?
(c) If each down time costs $7, regardless of how long the machine is
down, what is the long run total down time cost per unit time?
Problems
4.1. Suppose that a renewal function has the form M(t) = t +
[1 - exp(-at)]. Determine the mean and variance of the interoccurrence
distribution.
4.2. A system is subject to failures. Each failure requires a repair time
that is exponentially distributed with rate parameter a. The operating time
of the system until the next failure is exponentially distributed with rate
parameter /3. The repair times and the operating times are all statistically
independent. Suppose that the system is operating at time 0. Using equation (4.5), determine an approximate expression for the mean number of
failures up to time t, the approximation holding for t >> 0.
4.3.
Suppose that the life of a light bulb is a random variable X with
hazard rate h(x) = Ox for x > 0. Each failed light bulb is immediately replaced with a new one. Determine an asymptotic expression for the mean
age of the light bulb in service at time t, valid for t >> 0.
5.
Generalizations and Variations on Renewal Processes
447
4.4. A developing country is attempting to control its population growth
by placing restrictions on the number of children each family can have.
This society places a high premium on female children, and it is felt that
any policy that ignores the desire to have female children will fail. The
proposed policy is to allow any married couple to have children up to the
first female baby, at which point they must cease having children. Assume
that male and female children are equally likely. The number of children
in any family is a random variable N. In the population as a whole, what
fraction of children are female? Use the elementary renewal theorem to
justify your answer.
4.5.
A Markov chain X,,, X,, X,, ... has the transition probability matrix
0
0
1
2
0.7
0
P=1 0.6
0
0.4
0
0.5
0.5
2
0.3
A sojourn in a state is an uninterrupted sequence of consecutive visits to
that state.
(a) Determine the mean duration of a typical sojourn in state 0.
(b) Using renewal theory, determine the long run fraction of time that
the process is in state 1.
5.
5.1.
Generalizations and Variations
on Renewal Processes
Delayed Renewal Processes
W e continue to assume that {XR) are all independent positive random variables, but only X,, X,, ... (from the second on) are identically distributed
with distribution function F, while X, has possibly a different distribution
function G. Such a process is called a delayed renewal process. We have
all the ingredients for an ordinary renewal process except that the initial
time to the first renewal has a distribution different from that of the other
interoccurrence times.
448
VII
Renewal Phenomena
A delayed renewal process will arise when the component in operation
at time t = 0 is not new, but all subsequent replacements are new. For example, suppose that the time origin is taken y time units after the start of
an ordinary renewal process. Then the time to the first renewal after the
origin in the delayed process will have the distribution of the excess life
at time y of an ordinary renewal process.
+ X,,, and let N(t) count the
As before, let Wo = 0 and W = X, +
number of renewals up to time t. But now it is essential to distinguish between the mean number of renewals in the delayed process
MD(t) = E[N(t)],
(5.1.)
and the renewal function associated with the distribution F,
M(t)
(5.2)
Fk(t).
k=1
For the delayed process the elementary renewal theorem is
lim
t-4=
MD(t)
t
=
-,A
1
where µ = E[XZ],
(5.3)
and the renewal theorem states that
h
lim [MD(t) - MD(t - h)] = -,
provided XZ, X,,
5.2.
. .
. are continuous random variables.
Stationary Renewal Processes
A delayed renewal process for which the first life has the distribution
function
X
G(x) = IL'
Ji - F(y) } dy
0
is called a stationary renewal process. We are attempting to model a renewal process that began indefinitely far in the past, so that the remaining
life of the item in service at the origin has the limiting distribution of the
excess life in an ordinary renewal process. We recognize G as this limiting distribution.
Generalizations and Variations on Ranewal Processes
S.
449
It is anticipated that such a process exhibits a number of stationary, or
time-invariant, properties. For a stationary renewal process,
t
MD(t) = E[N(t)] = -
(5.4)
A
and
Pr{ y'D
x) = G(x),
for all t. Thus, what is in general\,only an asymptotic renewal relation bea stationary renewal process.
comes an identity, holding for
5.3.
Cumulative and Related Processes
Suppose associated with the ith unit, or lifetime interval, is a second random variable Y ({Y } identically distributed) in addition to the lifetime X;.
We allow X; and Y to be dependent but assume that the pairs (X,, Y,),
(X,, K,).... are independent. We use the notation F(x) = Pr{X; <_ x},
G(y)=Pr{Y;sy},IL =E[X;], and v=E[Y].
A number of problems of practical and theoretical interest have a natural formulation in those terms.
Renewal Processes Involving Two Components to Each
Renewal Interval
Suppose that Y represents a portion of the duration X;. Figure 5.1 illustrates the model. There we have depicted the Y portion occurring at the
-.*- YZ -
X,
`14X2
X3H
Figure 5.1 A renewal process in which an associated random variable Y,_
represents a portion of the ith renewal interval.
450
VII
Renewal Phenomena
beginning of the interval, but this assumption is not essential for the results that follow.
Let p(t) be the probability that t falls in a Y portion of some renewal interval. When X,, X,, ... are continuous random variables, the renewal theorem implies the following important asymptotic evaluation:
limp(t) =
E[Y]
E[X,]'
(5.5)
Here are some concrete examples.
A Replacement Model Consider a replacement model in which replacement is not instantaneous. Let Y be the operating time and Z; the lag
period preceding installment of the (i + 1)st operating unit. (The delay
in replacement can be conceived as a period of repair of the service unit.)
We assume that the sequence of times between successive replacements
X, = Y, + Z., k = 1, 2, . . . , constitutes a renewal process. Then p(t), the
probability that the system is in operation at time t, converges to
E[Y,]/E[X,].
A Queueing Model A queueing process is a process in which customers arrive at some designated place where a service of some kind is
being rendered, for example, at the teller's window in a bank or beside the
cashier at a supermarket. It is assumed that the time between arrivals, or
interarrival time, and the time that is spent in providing service for a given
customer are governed by probabilistic laws.
If arrivals to a queue follow a Poisson process of intensity A, then the
successive times X, from the commencement of the kth busy period to the
start of the next busy period form a renewal process. (A busy period is an
uninterrupted duration when the queue is not empty.) Each X, is composed
of a busy portion Z,, and an idle portion Y. Then p(t), the probability that
the queue is empty at time t, converges to E[Y,]/E[X,]. This example is
treated more fully in IX, which is devoted to queueing systems.
The Peter Principle The "Peter Principle" asserts that a worker will
be promoted until finally reaching a position in which he or she is incompetent. When this happens, the person stays in that job until retirement.
Consider the following single job model of the Peter Principle: A person
is selected at random from the population and placed in the job. If the person is competent, he or she remains in the job for a random time having
5.
Generalizations and Variations on Renewal Processes
451
cumulative distribution function F and mean µ and is promoted. If
incompetent, the person remains for a random time having cumulative
distribution function G and mean v > µ and retires. Once the job is vacated, another person is selected at random and the process repeats.
Assume that the infinite population contains the fraction p of competent
people and q = 1 - p incompetent ones.
In the long run, what fraction of time is the position held by an incompetent person?
A renewal occurs every time that the position is filled, and therefore the
mean duration of a renewal cycle is
E[Xk] = pp. + (1 - p)v.
To answer the question, we let Yk = Xk if the kth person is incompetent,
and Yk = 0 if the kth person is competent. Then the long run fraction of
time that the position is held by an incompetent person is
E[Y1]
E[X1]
_
(1 - p)v
pµ + (1 - p)v'
Suppose that p = i of the people are competent, and that v = 10, while
µ = 1. Then
E[Y,]
E[X1]
-
(1/2)(10)
_ 10 _
0 . 91 .
(1/2)(10) + (1/2)(1)
11
-
Thus, while half of the people in the population are competent, the job is
filled by a competent person only 9 percent of the time!
Cumulative Processes
Interpret Y; as a cost or value associated with the ith renewal cycle. A class
of problems with a natural setting in this general context of pairs (X,, Y),
where X, generates a renewal process, will now be considered. Interest
here focuses on the so-called cumulative process
N(t)+ I
W(t) _ 7.
Yk,
k=1
the accumulated costs or value up to time t (assuming that transactions are
made at the beginning of a renewal cycle).
452
V11
Renewal Phenomena
The elementary renewal theorem asserts in this case that
1
lim E[W(t)] =
1- - t
E[I]
(5.6)
This equation justifies the interpretation of E[Y,]/µ as a long run mean
cost or value per unit time, an interpretation that was used repeatedly in
the examples of Section 2.
Here are some examples of cumulative processes.
Replacement Models Suppose Y; is the cost of the ith replacement. Let
us suppose that under an age replacement strategy (see Section 3 and the ex-
ample entitled "Age Replacement Policies" in Section 4) a planned replacement at age T costs c, dollars, while a failure replaced at time x < T
costs c2 dollars. If Yk is the cost incurred at the kth replacement cycle, then
Yk =
1C2
with probability 1 - F(T),
with probability F(T),
and E[Yk] = c,[l - F(T)] + c2F(T). Since the expected length of a replacement cycle is
T
E[min{Xk, T}] =
f [1 - F(x)] dx,
0
we have that the long run cost per unit time is
c,[1 - F(T)] + c,F(T)
fa[l -F(x)]dx
and in any particular situation a routine calculus exercise or recourse to
numerical computation produces the value of T that minimizes the long
run cost per unit time.
Under a block replacement policy, there is one planned replacement
every T units of time and, on the average, M(T) failure replacements, so
the expected cost is E[Yk] = c, + c,M(T), and the long run mean cost per
unit time is {c, + CAT)}/T.
Risk Theory Suppose claims arrive at an insurance company according to a renewal process with interoccurrence times X,, X,..... Let Yk be
5.
Generalizations and Variations on Renewal Processes
453
the magnitude of the kth claim. Then W(t) = k '; Y, represents the cumulative amount claimed up to time t, and the long run mean claim rate is
lim t E[W(t)] =
E[Yj
E[X,]
Maintaining Current Control of a Process A production process
produces items one by one. At any instance, the process is in one of two
possible states, which we label iris control and out-of-control. These states
are not directly observable. Production begins with the process in-control,
and it remains in-control for a random and unobservable length of time
before a breakdown occurs, after which the process is out-of-control. A
control chart is to be used to help detect when the out-of-control state occurs, so that corrective action may be taken.
To be more specific, we assume that the quality of an individual item is
a normally distributed random variable having an unknown mean and a
known variance a2. If the process is in-control, the mean equals a standard
target, or design value, A, Process breakdown takes the form of shift in
mean away from standard to µ, = I± Sv, where S is the amount of the
shift in standard deviation units.
The Shewhart control chart method for maintaining process control
calls for measuring the qualities of the items as they are produced and then
plotting these qualities versus time on a chart that has lines drawn at the
target value A, and above and below this target value at A0 ± ko, where k
is a parameter of the control scheme being used. As long as the plotted
qualities fall inside these so-called action lines at p ± kcr, the process is
assumed to be operating in-control, but if ever a point falls outside these
lines, the process is assumed to have left the in-control state, and investigation and repair are instituted. There are obviously two possible types of
errors that can be made while thus controlling the process: (1) needless investigation and repair when the process is in-control yet an observed quality purely by chance falls outside the action lines and (2) continued operation with the process out-of-control because the observed qualities are
falling inside the action lines, again by chance.
Our concern is the rational choice of the parameter k, that is, the rational spacing of the action lines, so as to balance, in some sense, these two
possible errors.
454
VII
Renewal Phenomena
The probability that a single quality will fall outside the action lines
when the process is in-control is given by an appropriate area under the
normal density curve. Denoting this probability by a, we have
a = (D(-k) + 1 - I(k) = 2c(-k),
where 1(x) = (21r)-"2 fT, exp(-y2/2) dy is the standard cumulative normal distribution function. Representative values are given in the following table:
k
a
1.645
1.96
0.10
0.05
Similarly, the probability, denoted by p, that a single point will fall outside the action lines when the process is out-of-control is given by
p=(D(-6-k)+ 1 -4)(-6+k).
Let S denote the number of items inspected before an out-of-control signal
arises assuming that the process is out-of-control. Then Pr{S = 1) = p,
Pr(S = 2) = (1 - p)p, and in general, Pr(S = n) = (1 - p)"-'p. Thus, S
has a geometric distribution, and
E[S]
p
Let The the number of items produced while the process is in-control. We
suppose that the mean operating time in-control E[T] is known from past
records.
The sequence of durations between detected and repaired out-ofcontrol conditions forms a renewal process because each such duration
begins with a newly repaired process and is a probabilistic replica of all
other such intervals. It follows from the general elementary renewal
theorem that the long run fraction of time spent out-of-control (O.C.) is
O.C. =
_
E[S]
E[S] + E[T]
1
1 + pE[T]
The long run number of repairs per unit time is
R
_
1
E[S] + E[T]
_
p
1 + pE[T]
455
Exercises
Let N be the random number of "false alarms" while the process is incontrol, that is, during the time up to T, the first out-of-control. Then, con-
ditioned on T, the random variable N has a binomial distribution with
probability parameter a, and thus E[NIT} = aT and E[N] = aE[T].
Again, it follows from the general elementary renewal theorem that the
long run false alarms per unit time'(F.A.) is
F.A. =
apE[T]
=
E[S] + E['] 1 + pE[T]
E[N]
If each false alarm costs c dollars, each repair cost K dollars, and the
cost rate while operating out-of-control is C dollars, then we have the long
run average cost per unit time of
A.C. = C(O.C.) + K(R) + c(F.A.)
_ C + Kp + capE[T]
1 + pE[T]
By trial and error one may now choose k, which determines a and p, so
as to minimize this average cost expression.
Exercises
5.1. Jobs arrive at a certain service system according to a Poisson
process of rate A. The server will accept an arriving customer only if it is
idle at the time of arrival. Potential customers arriving when the system is
busy are lost. Suppose that the service times are independent random variables with mean service time A. Show that the long run fraction of time
that the server is idle is 1/(1 + Aµ). What is the long run fraction of potential customers that are lost?
5.2. The weather in a certain locale consists of alternating wet and dry
spells. Suppose that the number of days in each rainy spell is Poisson distributed with parameter 2, and that a dry spell follows a geometric distribution with a mean of 7 days. Assume that the successive durations of
rainy and dry spells are statistically independent random variables. In the
long run, what is the probability on a given day that it will be raining?
456
VII
Renewal Phenomena
5.3.
Consider a light bulb whose life is a continuous random variable X
with probability density function f(x), for x > 0. Assuming that one starts
with a fresh bulb and that each failed bulb is immediately replaced by a
new one, let M(t) = E[N(t)] be the expected number of renewals up to
time t. Consider a block replacement policy (see Section 2.1) that replaces
each failed bulb immediately at a cost of c per bulb and replaces all bulbs
at the fixed times T, 2T, 3T, .... Let the block replacement cost per bulb
be b < c. Show that the long run total mean cost per bulb per unit time is
O(T)= b+cM(T)
T
Investigate the choice of a cost minimizing value T* when M(t) _
t + 1 - exp(-at).
Problems
5.1. A certain type component has two states: 0 = OFF and 1 =
OPERATING. In state 0, the process remains there a random length of
time, which is exponentially distributed with parameter a, and then moves
to state 1. The time in state 1 is exponentially distributed with parameter
/3, after which the process returns to state 0.
The system has two of these components, A and B, with distinct
parameters:
Component
Operating Failure Rate
Repair Rate
A
NA
/3s
aA
B
ag
In order for the system to operate, at least one of components A and B
must be operating (a parallel system). Assume that the component stochastic processes are independent of one another.
(a) In the long run, what fraction of time is the system inoperational
(not operating)?
(b) Once the system enters the failed state, what is the mean duration
there prior to returning to operation?
6.
Discrete Renewal Theory
457
(c) Define a cycle as the time between the instant that the system first
enters the failed state and the next such instant. Using renewal theory, find the mean duration of a cycle.
(d) What is the mean system operating duration between successive
system failures?
5.2. The random lifetime X of an item has a distribution function F(x).
What is the mean total life E[XI X >\x] of an item of age x?
5.3. At the beginning of each period, customers arrive at a taxi stand at
times of a renewal process with distribution law F(x). Assume an unlimited supply of cabs, such as might occur at an airport. Suppose that each
customer pays a random fee at the stand following the distribution law
G(x), for x > 0. Write an expression for the sum W(t) of money collected
at the stand by time t, and then determine the limit expectation
E[W(t)]
lim
rax
t
5.4. A lazy professor has a ceiling fixture in his office that contains two
light bulbs. To replace a bulb, the professor must fetch a ladder, and being
lazy, when a single bulb fails, he waits until the second bulb fails before
replacing them both. Assume that the length of life of the bulbs are independent random variables.
(a) If the lifetimes of the bulbs are exponentially distributed, with the
same parameter, what fraction of time, in the long run, is our professor's office half lit?
(b) What fraction of time, in the long run, is our professor's office half
lit if the bulbs that he buys have the same uniform (0, 1) lifetime
distribution?
6.
Discrete Renewal Theory*
In this section we outline the renewal theory that pertains to nonnegative
integer-valued lifetimes. We emphasize renewal equations, the renewal
argument, and the renewal theorem (Theorem 6.1).
* The discrete renewal model is a special case in the general renewal theory presented
in sections 1-5 and does not arise in later chapters.
458
VII
Renewal Phenomena
Consider a light bulb whose life, measured in discrete units, is a random variable X where Pr{X = k} = pk for k = 0, 1, .... If one starts with
a fresh bulb and if each bulb when it burns out is replaced by a new one,
then M(n), the expected number of renewals (not including the initial
bulb) up to time n, solves the equation
11
pkM(n - k),
M(n) = FX(n) +
(6.1)
k=0
where FX(n) = p +
+ p,, is the cumulative distribution function of the
random variable X. A vector or functional equation of the form (6.1) in the
unknowns M(0), M(1), ... is termed a renewal equation. The equation is
established by a renewal argument, a first step analysis that proceeds by
conditioning on the life of the first bulb and then invoking the law of total
probability. In the case of (6.1), for example, if the first bulb fails at time
k:5 n, then we have its failure plus, on the average, M(n - k) additional
failures in the interval [k, k + 1, . . . , n]. We weight this conditional mean
by the probability pk = Pr{X1 = k} and sum according to the law of total
probability to obtain
M(n) _
[ 1 + M(n - k)]Pk
k=0
F,(n)=LppM(n-k).
k=0
Equation (6.1) is only a particular instance of what is called a renewal
equation. In general, a renewal equation is prescribed by a given bounded
sequence {bk } and takes the form
,1
for n = 0, 1,
v =
....
(6.2)
k=0
The unknown variables are vo, v,, ... , and po, pi.... is a probability distribution for which, to avoid trivialities, we always assume p < 1.
Let us first note that there is one and only one sequence vo, v,, . . . satisfying a renewal equation, because we may solve (6.2) successively to
get
6.
Discrete Renewal Weory
459
bo
I - po ,
vo =
v,=
(6.3)
b, + p, vo
I
- Po
and so on.
Let u,, be the mean number of renewals that take place exactly in period
n. When po = 0, so that the lifetime's are strictly positive and at most one
renewal can occur in any period, thei u,, is the probability that a single renewal occurs in period n. The sequence u,,, u ... satisfies a renewal equation that is of fundamental importance in the general theory. Let
for n = 0,
for n > 0.
(6.4)
Then {u } satisfies the renewal equation
Pkun-k
for n = 0, 1,
....
(6.5)
k=0
Again, equation (6.5) is established via a renewal argument. First, observe that S counts the initial bulb, the renewal at time 0. Next, condition
on the lifetime of this first bulb. If it fails in period k < n, which occurs
with probability pk, then the process begins afresh and the conditional
probability of a renewal in period n becomes
Weighting the contingency represented by
by its respective probability pk and summing according to the law of total probability then yields (6.5).
The next lemma shows how the solution {v,,} to the general renewal
equation (6.2) can be expressed in terms of the solution
to the particular equation (6.5).
Lemma 6.1
If { v } satisfies (6.2) and { u } satisfies (6.5), then
forn=0, 1,....
k=0
Proof In view of our remarks on the existence and uniqueness of solutions to equation (6.2), we need only verify that v,, = Yk=o b-kuk satisfies
(6.2). We have
460
VII
Renewal Phenomena
v_
k=0
=b
if
k=0 1=0
=b
b kPk-rui
1=0 k=!
=b
pj
ur
1=o j=o
=b
j=0 1=0
=b
j=0
Example Let X, X,.... be the successive lifetimes of the bulbs and let
Wo = 0 and W = X, +
+ X,, be the replacement times. We assume
that p0, = Pr{X1 = 0) = 0. The number of replacements (not including the
initial bulb) up to time n is given by
N(n) = k
for Wk <- n < Wk,,.
The M(n) = E[N(n)] satisfies the renewal equation (6.1)
n
M(n) = po + ... + p + Z. PkM(n - k),
k=0
and elementary algebra shows that m,, = E[N(n) + I] = M(n) + 1 satisfies
it
for n = 0, 1,
M" = 1 +
k=0
....
(6.6)
Then (6.6) is a renewal equation for which b,, = I for all n. In view of
Lemma 6.1 we conclude that
6.
Discrete Renewal Theory
461
77
Mn = > . lug = up +
+ u,,.
k=0
Conversely, u,, = m,, - m,,_, = M(n) - M(n - 1).
To continue with the example, let g,, = E[WN(,,,+,] The definition is illustrated in Figure 6.1. We will argue that g,, satisfies a certain renewal
always includes the first renewal
equation. As shown in Figure 6.1,
duration X,. In addition, if X, = k : n, which occurs with probability pk,
then the conditional mean of the added lives constituting WN(,,)+, is
Weighting these conditional means by, their respective probabilities and
summing according to the law of total 'probability then gives
g = E[XI] +
pk.
k=0
Hence, by Lemma 6.1,
g,,
k=0
E[X,luk=E[X,lm
We get the interesting formula [see (1.7)]
E[X, +
+
X E[N(n) + 1].
(6.7)
Note that N(n) is not independent of {Xk}, and yet (6.7) still prevails.
N(n)
,
X,
X;
X,
Wo
Figure 6.1
W,
n
n
WN(,,)+I always contains X, and contains additional durations when
X, =k<--n.
462
6.1.
VII
Renewal Phenomena
The Discrete Renewal Theorem
The renewal theorem provides conditions under which the solution { v } to
a renewal equation will converge as n grows large. Certain periodic behavior, such as failures occurring only at even ages, must be precluded,
and the simplest assumption assuring this preclusion is that p, > 0.
Theorem 6.1
Suppose that 0 < p, < 1 and that
and
are the
solutions
to
the
renewal
equations
(6.5)
and
(6.2),
respectively.
Then
.
(a)
u,, = 1/j okpk; and (b) if
<00, then
{Y,A obk}/{T1A okPk}.
We recognize that Yk=okp, = E[X,] is the mean lifetime of a unit. Thus
(a) in Theorem 6.1 asserts that in the long run, the probability of a renewal
occurring in a given interval is one divided by the mean life of a unit.
Remark Theorem 6.1 holds in certain circumstances when p, = 0. It
suffices to assume that the greatest common divisor of the integers k for
which pk > 0 is one.
Example Let y,, = WN,,,,+, - n be the excess life at time n. For a fixed
integer m, let f(m) = Pr{y = m}. We will establish a renewal equation
for f(m) by conditioning on the first life X. For m > 1,
if 0:5 k <- n,
1
10
ifk=n+m,
otherwise.
(The student is urged to diagram the alternatives
Pr{y = mIX, = k}.) Then, by the law of total probability,
=7'
arising in
=k}pk
k=o
it
P,,,+ + L f -k(m)Pk
k=O
We apply Theorem 6.1 with b,, = p,, + to conclude that
(6.8)
6.
Discrete Renewal Theory
lim Pr{y = m} _
463
"k=0 Pw+k
7-i=o kPk
Pr{X,? m}
E[X,] '
m= 1,2,....
The limit is a bona fide probability mass function, since its terms sum
to one:
Pr{X1 ? m}
_
E[XI]
6.2.
E[X,]
= 1.
E[XI]
Deterministic Population Growth with Age Distribution
In this section we will discuss a simple deterministic model of population
growth that takes into account the age structure of the population. Surprisingly, the discrete renewal theorem (Theorem 6.1) will play a role in
the analysis. As the language will suggest, the deterministic model that we
treat may be viewed as describing the mean population size in a more
elaborate stochastic model that is beyond our scope to develop fully.
A Simple Growth Model Let us set the stage by reviewing a simple
model that has no age structure. We consider a single species evolving in
discrete time t = 0, 1, 2, ... , and we let N, be the population size at time
t. We assume that each individual present in the population at time t gives
rise to a constant number A of offspring that form the population at time
t + 1. (If death does not occur in the model, then we include the parent as
one of the offspring, and then necessarily A ? 1.) If No is the initial population size, and each individual gives rise to X offspring, then
N,=AN,
N,=AN,=A'NQ,
and in general,
N, = A'N0.
(6.9)
464
VII
Renewal Phenomena
If A > 1, then the population grows indefinitely in time; if A < 1, then the
population dies out; while if A = 1, then the population size remains constant at N, = No for all t = 0, 1, ... .
The Model with Age Structure We shall now introduce an age
structure in the population. We need the following notation:
n,,,, = the number of individuals of age u in the population at time t;
N, = Y,;;=o n,,., = the total number of individuals in the population at
time t;
b, = the number of new individuals created in the population at time
t, the number of births;
/3 = the expected number of progeny of a single individual of age u
in one time period;
l = the probability that an individual will survive, from birth, at
least to age u.
The conditional probability that an individual survives at least to age u,
given that is has survived to age u - 1, is simply the ration
The net
maternity function is the product
M.
and is the birth rate adjusted for the death of some fraction of the population. That is, m is the expected number of offspring at age u of an individual now of age 0.
Let us derive the total progeny of a single individual during its lifespan.
An individual survives at least to age u with probability 1,,, and then during the next unit of time gives rise to /3 offspring. Summing
m
over all ages u then gives the total progeny of a single individual:
u=0
1"A' = . m,,.
(6.10)
N=0
If M > 1, then we would expect the population to increase over time; if
M < 1, then we would expect the population to decrease; while if M = 1,
then the population size should neither increase nor decrease in the long
run. This is indeed the case, but the exact description of the population
evolution is more complex, as we will now determine.
6.
Discrete Renewal Theory
465
In considering the effect of age structure on a growing population, our
interest will center on b,, the number of new individuals created in the
population at time t. We regard 6,,, 1,,, and n,,.0 as known, and the problem
is to determine b, for t ? 0. Once b, is known, then n,,., and N, may be determined according to, for example,
no., = bl,
(6.11)
1
nu.l = nu-I,0
for u
1,
(6.12)
lu-1
and
N, =
(6.13)
uJ*
u=0
In the first of these simple relations, no.,, is the number in the population
at time 1 of age 0, which obviously is the same as b those born in the
population at time 1. For the second equation, n,,., is the number in the
population at time 1 of age u. These individuals must have survived from
individuals in the population at time 0 of age u - 1; the condithe
which explains the second
tional probability of survivorship is
equation. The last relation simply asserts that the total population size results by summing the numbers of individuals of all ages. The generalizations of (6.11) through (6.13) are
(6.14)
no., = b,,
for u ? 1,
n,,,, =
(6.15)
and
x
N, _
n,,,,
fort ? 1.
(6.16)
1,=0
Having explained how n,,,, and N, are found once b, is known, we turn
to determining b,. The number of individuals created at time t has two
components. One component, a,, say, counts the offspring of those individuals in the population at time t who already existed at time 0. In the
simplest case, the population begins at time t = 0 with a single ancestor
of age u = 0, and then the number of offspring of this individual at time t
is a, = m,, the net maternity function. More generally, assume that there
466
VII
Renewal Phenomena
were n,,.o individuals of age u at time 0. The probability that an individual
of age u at time 0 will survive to time t (at which time it will be of age
t + u) is
Hence the number of individuals of age u at time 0 that
survive to time t is
and each of these individuals, now of age
t + u, will produce (3,+ new offspring. Adding over all ages we obtain
L P,+,,n,,.o-
1,+
1
(6.17)
m,+,,n,,.o
_
1
11=0
The second component of b, counts those individuals created at time t
whose parents were not initially in the population but were born after time
0. Now, the number of individuals created at time T is b_. The probability
that one of these individuals survives to time t, at which time he will be
of age t - T, is 1,-,. The rate of births for individuals of age t - 'r is
The second component results from summing over r and gives
b, = a, +>. 01-711-:br
(6.18)
= a,
r=O
Example Consider an organism that produces two offspring at age 1,
and two more at age 2, and then dies. The population begins with a single
organism of age 0 at time 0. We have the data
no.o = 1,
n,,.o = 0
for u ? 1,
b,=b,=2,
10=1,=1,=1
and
1,,=0 foru>2.
We calculate from (6.17) that
a,=O, a, = 2, a,=2,
Finally, (6.18) is solved recursively as
and
a, = 0, for t > 2.
6.
Discrete Renewal Theory
467
bo = 0,
b, = a, + mob, + m,bo
=2+ 0 +
0
=2,
b, = a, + mob, + m,b, + m,bo
= 2 + 0 + (2)(2) +
0
= 6,
b3 = a, + mob, + m,b, + m,b, +
m,bo
= 0 + 0 + (2)(6) + (2)(2) + 0 = 16.
Thus, for example, an individual of age 0(at time 0 gives rise to 16 new
individuals entering the population at time 3.
The Long Run Behavior
Somewhat surprisingly, since no "renewals" are readily apparent, the discrete renewal theorem (Theorem 6.1) will be invoked to deduce the long
run behavior of this age-structured population model. Observe that (6.18)
(6.19)
a,
has the form of a renewal equation except that {m,.} is not necessarily a
bona fide probability distribution in that, typically, {m,.} will not sum to
one. Fortunately, there is a trick that overcomes this difficulty. We introduce a variable s, whose value will be chosen later, and let
m,,,, = m,.s',
b,," = b,.s
and
a" = a,.s'.
Now multiply (6.19) by s' and observe that s'm,.b,-,, = (m,.s'')(b,_,,s'-'')
m"br"_, to get
b # = a" + > m,",b,"_,..
(6.20)
468
VII
Renewal Phenomena
This renewal equation holds no matter what value we choose for s. We
therefore choose s such that (m,". } is a bona fide probability distribution.
That is, we fix the value of s such that
x
MI.
vO
There is always a unique such s whenever 1 < J;,=o m, < oo. We may now
apply the renewal theorem to (6.20), provided that its hypothesis concerning nonperiodic behavior is satisfied. For this it suffices, for example,
that m, > 0. Then we conclude that
x
lim b," = lim bs' =
!fix
!fix
y
".
2.,.=o vm,.
(6.21)
We set A = 1/s and K = x=o a,"/E=o vm," to write (6.21) in the form
b, - KA'
for t large.
In words, asymptotically, the population grows at rate A where A = 1/s is
the solution to
tc
,.A-" = 1.
=0
When t is large (t > u), then (6.15) may be iterated in the manner
l
n,,.,
rl
= no.,
l
1101 =
This simply expresses that those of age u at time t were born t - u time
units ago and survived. Since for large t we have b,- =- KA'-", then
469
Exercises
n,,.,
K(l,k-u)n',
K.. (l,, lc )A',
N, = >, nu.,
and
lim
n,,.,
- N,
l"A-U
_ Y,xV-v*
This last expression furnishes the asymptotic, or stable, age distribution
in the population.
Example Continuing the example in which m, = m2 = 2 and mk = 0
otherwise, then we have
x
I m,,s"=2s+2s22= 1,
v=o
which we solve to obtain
+8
s= -2±V4
4
=
-1 2±v
_ (0.366, -1.366).
The relevant root is s = 0.366, whence A = 1/s = 2.732. Thus asymptotically, the population grows geometrically at rate A = 2.732
, and the
stable age distribution is as shown in the following table:
Age
Fraction of Population
0
1/(1 + s + s2) = 0.6667
s/(1 + s + s2) = 0.2440
s2/(1 + s + s2) = 0.0893
1
2
Exercises
6.1. Solve for v,, for n = 0, 1,
... ,
PkV,,-k
10 in the renewal equation
forn=0, 1,...,
k=0
where b0=b, =2, b2=b3=... =0,andpo=4, p, =i, and P2=4.
470
6.2.
VII
Renewal Phenomena
(Continuation of Exercise 6.1)
(a) Solve for u,, for n = 0, 1, ... , 10 in the renewal equation
)!
forn=0,1,...,
u=S
k=0
where So = 1, S, = S, _
= 0, and {pk} is as defined in Exercise
6.1.
(b) Verify that the solution v,, in Exercise 6.1 and u,, are related according to v,, =
6.3.
Using the data of Exercises 6.1 and 6.2, determine
(a)
u,,.
(b) lim,,, v,,.
Problems
6.1.
Suppose the lifetimes X, X2, ... have the geometric distribution
Pr{X1 = k} = a(l - a)k-'
fork = 1, 2,
... ,
where 0<a<1.
(a) Determine u,, for n = 1, 2, ... .
(b) Determine the distribution of excess life y by using Lemma 6.1
and (6.8).
6.2. Marlene has a fair die with the usual six sides. She throws the die
and records the number. She throws the die again and adds the second
number to the first. She repeats this until the cumulative sum of all the
tosses first exceeds a prescribed number n. (a) When n = 10, what is the
probability that she stops at a cumulative sum of 13? (b) When n is large,
what is the approximate probability that she stops at a sum of n + 3?
6.3. Determine the long run population growth rate for a population
whose individual net maternity function is m, = m; = 2, and mk = 0
otherwise. Why does delaying the age at which offspring are first produced cause a reduction in the population growth rate? (The population
Problems
471
growth rate when m, = m, = 2, and mk = 0 otherwise was determined in
the last example of this section.)
6.4. Determine the long run population growth rate for a population
whose individual net maternity function is mo = m, = 0 and m2 =
= a > 0. Compare this with the population growth rate when
m, =
m,= a, andm,=0fork
2.
Chapter VIII
Brownian Motion and
Related Processes
1.
Brownian Motion and Gaussian Processes
The Brownian motion stochastic process arose early in this century as an
attempt to explain the ceaseless irregular motions of tiny particles suspended in a fluid, such as dust motes floating in air. Today, the Brownian
motion process and its many generalizations and extensions occur in numerous and diverse areas of pure and applied science such as economics,
communication theory, biology, management science, and mathematical
statistics.
1.1. A Little History
The story begins in the summer of 1827, when the English botanist Robert
Brown observed that microscopic pollen grains suspended in a drop of
water moved constantly in haphazard zigzag trajectories. Following the
reporting of his findings, other scientists verified the strange phenomenon.
Similar Brownian motion was apparent whenever very small particles
were suspended in a fluid medium, for example, smoke particles in air.
Over time, it was established that finer particles move more rapidly, that
the motion is stimulated by heat, and that the movement becomes more
active with a decrease in fluid viscosity.
474
VIII
Brownian Motion and Related Processes
A satisfactory explanation had to wait until the next century, when in
1905, Einstein would assert that the Brownian motion originates in the
continual bombardment of the pollen grains by the molecules of the surrounding water, with successive molecular impacts coming from different
directions and contributing different impulses to the particles. Einstein argued that as a result of the continual collisions, the particles themselves
had the same average kinetic energy as the molecules. Belief in molecules
and atoms was far from universal in 1905, and the success of Einstein's
explanation of the well-documented existence of Brownian motion did
much to convince a number of distinguished scientists that such things as
atoms actually exist. Incidentally, 1905 is the same year in which Einstein
set forth his theory of relativity and his quantum explanation for the photoelectric effect. Any single one of his 1905 contributions would have
brought him recognition by his fellow physicists. Today, a search in a university library under the subject heading "Brownian motion" is likely to
turn up dozens of books on the stochastic process called Brownian motion
and few, if any, on the irregular movements observed by Robert Brown.
The literature on the model has far surpassed and overwhelmed the literature on the phenomenon itself!
Brownian motion is complicated because the molecular bombardment
of the pollen grain is itself a complicated process, so it is not surprising that
it took more than another decade to get a clear picture of the Brownian motion stochastic process. It was not until 1923 that Norbert Wiener set forth
the modem mathematical foundation. The reader may also encounter
"Wiener process" or "Wiener-Einstein process" as names for the stochastic process that we will henceforth simply call "Brownian motion."
Predating Einstein by several years, in 1900 in Paris, Louis Bachelier
proposed what we would now call a "Brownian motion model" for the
movement of prices in the French bond market. While Bachelier's paper
was largely ignored by academics for many decades, his work now stands
as the innovative first step in a mathematical theory of stock markets that
has greatly altered the financial world of today. Later in this chapter, we will
have much to say about Brownian motion and related models in finance.
1.2.
The Brownian Motion Stochastic Process
In terms of our general framework of stochastic processes (cf. I, Section
1.1), the Brownian motion process is an example of a continuous-time,
1.
Brownian Motion and Gaussian Processes
475
continuous-state-space Markov process. Let B(t) be the y component (as a
function of time) of a particle in Brownian motion. Let x be the position
x. Let p(y, t1x) be the probability denof the particle at time to; i.e.,
x. We postulate that the
sity function, in y, of B(to + t), given that
probability law governing the transitions is stationary in time, and therefore p(y, tlx) does not depend on the initial time to.
Since p(y, t1 x) is a probability density function in y, we have the
properties
x
p(y, t1x) ? 0
and
J
p(y, t1x) dy = 1.
(1.1)
Further, we stipulate that B(to ± t) is likely to be near B(to) = x for small
values of t. This is done formally by requiring that
limp(y, t1 x)
r
= 0 for y * x.
(1.2)
)0
From physical principles, Einstein showed that p(y, t1x) must satisfy the
partial differential equation
ap__ 0,,a2p
at
(1.3)
ax'-'
This is called the diffusion equation, and 0-22 is the diffusion coefficient,
which Einstein showed to be given by Q'- = RTINf, where R is the gas constant, T is the temperature, N is Avogadro's number, and f is a coefficient
of friction. By choosing the proper scale, we may take o-' = 1. With this
choice we can verify directly (See Exercise 1.3) that
p(Y> tax) =
expl
2t
-x)21
(1.4)
is a solution of (1.3). In fact, it is the only solution under the conditions
(1.1) and (1.2). We recognize (1.4) as a normal probability density function whose mean is x and whose variance is t. That is, the position of the
particle t time units after observations begin is normally distributed. The
mean position is the initial location x, and the variance is the time of observation t.
Because the normal distribution will appear over and over in this chapter, we are amply justified in standardizing some notation to deal with it.
476
VIII
Brownian Motion and Related Processes
Let
be the standard normal probability density function, and let
41(z) = J O(x) dx
(1.6)
-x
be the corresponding cumulative distribution function. A small table of the
cumulative normal distribution appears at the end of this section (see
p. 486). Let
1
¢(z/N/t-),
(1.7)
_Vrt
and
Z
INz) =
f ¢,(x) dx = (D(zl V)
(1.8)
-x
be the probability density function and cumulative distribution function,
respectively, for the normal distribution with mean zero and variance t. In
this notation, the transition density in (1.4) is given by
(1.9)
P(y, tlx) = 4,(y - x),
and
Pr{B(t) <_ yIB(O) = x} _
y
X
The transition probability density function in (1.4) or (1.9) gives only the
probability distribution of B(t) - B(O). The complete description of the
Brownian motion process with diffusion coefficient QZ is given by the following definition.
Definition Brownian motion with diffusion coefficient v2 is a stochastic process {B(t); t ? 0} with the properties:
(a) Every increment B(s + t) - B(s) is normally distributed with mean
zero and variance o 2t; o > 0 is a fixed parameter.
Brownian Motion and Gaussian Processes
1.
(b) For every pair of disjoint time intervals (t t2],
477
t,], with
0 : t, < t2 C t3 < t,, the increments B(t,) - B(t3) and B(t) - B(t,)
(t3i
are independent random variables, and similarly for n disjoint time
intervals, where n is an arbitrary positive integer.
(c) B(0) = 0, and B(t) is continuous as a function of t.
The definition says that a displacement B(s + t) - B(s) is independent
of the past, or alternatively, if we know B(s) = x, then no further knowl-
edge of the values of B(T) for past times T < s has any effect on
our knowledge of the probability law governing the future movement
B(s + t) - B(s). This is a statement of the Markov character of the
process. We emphasize, however, that the independent increment assumption (b) is actually more restrictive than the Markov property. A typical Brownian motion path is illustrated in Figure 1.1.
B(t)
Figure 1.1
A typical Brownian motion path.
478
Vlll Brownian Motion and Related Processes
The choice B(O) = 0 is arbitrary, and we often consider Brownian motion starting at x, for which B(O) = x for some fixed point x. For Brownian motion starting at x, the variance of B(t) is oat, and 0-2 is termed the
variance parameter in the stochastic process literature. The process b(t)
= B(t)lo- is a Brownian motion process whose variance parameter is one,
the so-called standard Brownian motion. By this device, we may always
reduce an arbitrary Brownian motion to a standard Brownian motion; for
the most part, we derive results only for the latter. By part (a) of the definition, for a standard Brownian motion (v2 = 1), we have
Pr{B(s + t) : yIB(s) = x} = Pr(B(s + t) - B(s) : y - x}
yx
Remark Let us look, for the moment, at a Brownian displacement
B(it) after a small elapsed time it. The mean displacement is zero, and
the variance of the displacement is it itself. It is much more common in
practical work to use a standard deviation, the square root of the variance,
to measure variability. For the normal distribution, for instance, the probability of an observation more than 2 standard deviations away from the
mean is about 5%, and the standard deviation is in the same units as the
original measurement, and not (units)'-. The standard deviation of the
Brownian displacement is Vlt, which is much larger than it itself
when it is small. Indeed, StdDev[B(it)]/it = V/it = 1/V - oo as
it - 0. This is simply another manifestation of the erratic movements of
the Brownian particle, yet it is a point that Bachelier and others had difficulty in handling. But the variance, being linear in time, and not the
standard deviation, is the only possibility if displacements over disjoint
time intervals are to be stationary and independent. Write a total displacement B(s + t) - B(0) as the sum of two incremental steps in the form
B(s + t) - B(0) = {B(s) - B(0) } + {B(s + t) - B(s)). The incremental
steps being statistically independent, their variances must add. The stationary assumption is that the statistics of the second step B(t + s) - B(t)
do not depend on the time t when the step began, but only on the duration
s of the movement. We must have, then, Var[B(t + s)] = Var[B(t)] +
Var[B(s)], and the only nonnegative solution to such an equation is to have
the variance of the displacement a linear function of time.
1.
Brownian Motion and.Gaussian Processes
.479
The Covariance Function
Using the independent increments assumption (b), we will determine the
covariance of the Brownian/motion. Recall that E[B(t)] = 0 and that
s < t,
E[B(t)2] = 0-2 t. Then, for 0
Cov[B(s), B(t)] = E[P(s)B(t)]
= E{B(s) {B(t) - B(s) + B(s) }]
= E[B(s)2] + E[B(s)(B(t) - B(s)}]
= 0-2 s + E[B(s)]E[B(t) - B(s)]
= u2s
(by (b))
(since E[B(s)] = 0).
Similarly, if 0 <_ t < s, we obtain Cov[B(s), B(t)] = 0-'t. Both cases may
be treated in a single expression via
for s, t ? 0.
Cov[B(s), B(t)] = Q.2 min{s, t},
1.3.
(1.10)
The Central Limit Theorem and the
Invariance Principle
Let S,, _ , +
+ 6 be the sum of n independent and identically distributed random variables 6 ... , having zero means and unit variances. In this case, the central limit theorem asserts that
lim Prj
11--
n
x}
_ fi(x)
for all x.
The central limit theorem is stated as a limit. In stochastic modeling, it is
used to justify the normal distribution as appropriate for a random quantity whose value results from numerous small random effects, all acting
independently and additively. It also justifies the approximate calculation
of probabilities for the sum of independent and identically distributed
summands in the form Pr{S : x} c(x/V), the approximation known
to be excellent even for moderate values of n in most cases in which the
distribution of the summands is not too skewed.
In a similar manner, functionals computed for a Brownian motion can
often serve as excellent approximations for analogous functionals of a
480
All Brownian Motion and Related Processes
partial sum process, as we now explain. As a function of the continuous
variable t, define
B,,(t) = S
Vn_
,
t ? 0,
where [x] is the greatest integer less than or equal to x. Observe that
Sk V Sk
k
k
forn:5
t<n+n.
1
N/k'
Because SklNIrk has unit variance, the variance of B (t) is [nt]ln, which
converges to t as n - co. When n is large, then k = [nt] is large, and Sk/v-k
is approximately normally distributed by the central limit theorem, and, finally, B (t) inherits the independent increments property (b) from the postulated independence of the summands. It is reasonable, then, to believe
that B (t) should behave much like a standard Brownian motion process,
at least when n is large. This is indeed true, and while we cannot explain
the precise way in which it holds in an introductory text such as this, the
reader should leave with some intuitive feeling for the usefulness of the
result and, we hope, a motivation to learn more about stochastic
processes. The convergence of the sequence of stochastic processes defined in (1.11) to a standard Brownian motion is termed the invariance
principle. It asserts that some functionals of a partial sum process of independent and identically distributed zero mean and unit variance random
variables should not depend too heavily on (should be invariant of) the actual distribution of the summands, but be approximately given by the
analogous functional of a standard Brownian motion, provided only that
the summands are not too badly behaved.
Example Suppose that the summands have the distribution in which
1, each with probability Then the partial sum process S,, is a simple random walk for which we calculated in III, Section 5.3 (using a different notation)
Pr{S reaches -a < 0 before b > O1So = 0}
_
b
a+b
(1.12)
1.
Brownian Motion and Gaussian Processes
481
Upon changing the scale in accordance with (1.11), we have
reaches -a < 0 before b > OISO = 0)
_
b
bV_n
=
a+b'
and invoking the invariance principle, it should be, and is, the case that for
a standard Brownian motion we have
Pr{B(t) reaches -a < 0 before b > 01B(0) = 0)
_
b
(1.13)
a+b'
Finally, invoking the invariance principle for a second time, the evaluation
in (1.12) should hold approximately for an arbitrary partial sum process,
provided only that the independent and identically distributed summands
have zero means and unit variances.
1.4.
Gaussian Processes
A random vector X ... , X is said to have a multivariate normal distribution, or a joint normal distribution, if every linear combination
+
a,X, +
a univariate normal distribution. Obviously, if X ... , X,, has a joint normal distribution, then so does the random vector Y . . . , Ym, defined by the linear transformation in which
Y=a1,X,+
+a;,,X,,,
forj= 1,...,m,
for arbitrary constants a;;.
The multivariate normal distribution is specified by two parameters, the
mean values µ; = E[X;] and the covariance matrix whose entries are
I;; = Cov[X;, X,]. In the joint normal distribution, I;; = 0 is sufficient to
imply that X; and X, are independent random variables.
Let T be an abstract set and {X(t); t in T J a stochastic process. We call
{X(t); tin T} a Gaussian process if for every n = 1, 2, ... and every finite subset {t,, . . . ,
of T, the random vector (X(t,), . . . , X(t,,)) has a
multivariate normal distribution. Equivalently, the process is Gaussian if
every linear combination
a,X(t,) +
+
a; real,
482
VIII
Brownian Motion and Related Processes
has a univariate normal distribution. Every Gaussian process is described
uniquely by its two parameters, the mean and covariance functions, given
respectively by
µ(t) = E[X(t)],
tin T,
and
r(s, t) = E[(X(s) - µ(s) } (X(t) - µ(t) }],
s, tin
T.
The covariance function is positive definite in the sense that for every
n = 1, 2, . . . , real numbers a...... a,,, and elements t...... t,, in T,
7,
a,a;r(t,, t) ? 0.
i=lj=l
One need only evaluate the expected value of
a;(X(t) - µ(t;)})'- >_ 0
in terms of the covariance function in order to verify this.
Conversely, given an arbitrary mean value function µ(t) and a positive
definite covariance function F(s, t), there then exists a corresponding
Gaussian process. Brownian motion is the unique Gaussian process having continuous trajectories, zero mean, and covariance function (1.10).
We shall use this feature, that the mean value and covariance functions define a Gaussian process, several times in what follows.
We have seen how the invariance principle leads to the Gaussian
process called Brownian motion. Gaussian processes also arise as the limits of normalized sums of independent and identically distributed random
functions. To sketch out this idea, let 1(t), 62(t), ... be independent and
identically distributed random functions, or stochastic processes. Let
µ(t) = E[e(t)] and r(s, t) =
fi(t)] be the mean value and covariance functions, respectively. Motivated by the central limit theorem,
we define
XN(t) _ 1=1 ( r^(t)- p(t)I
NN
The central limit theorem tells us that the distribution of X,,,(t) converges
to the normal distribution for each fixed time point t. A multivariate extension of the central limit theorem asserts that for any finite set of time
points (tl, ... , t,,), the random vector
(X, (tl), ... , XN(t,))
1.
Brownian Motion and Gaussian Processes
483
has, in the limit for large N, a multivariate normal distribution. It is not
difficult to believe, then, that under ordinary circumstances, the stochastic
processes {XN(t); t ? 0} would converge, in an appropriate sense, to a
Gaussian process {X(t); t ? 0) whose mean is zero and whose covariance
function is F(s, t). We call this the central limit principle for random functions. Several instances of its application appear in this chapter, the first of
which is next.
Example Cable Strength Under Equal Load Sharing Consider a
cable constructed from N wires in parallel. Suspension bridge cables are
usually built this way. A section of the cable is clamped at each end and
elongated by increasing the distance between the clamps. The problem is
to determine the maximum tensile load that the cable will sustain in terms
of the probabilistic and mechanical characteristics of the individual wires.
Let Lo be the reference length of an unstretched unloaded strand of wire,
and let L be the length after elongation. The nominal strain is defined to
be t = (L - Lo)/Lo. Steadily increasing t causes the strand to stretch and
exert a force fi(t) on the clamps, up to some random failure strain , at
which point the wire breaks. Hooke's law of elasticity asserts that the wire
force is proportional to wire strain, with Young's modulus K as the proportionality constant. Taken all together, we write the force on the wire as
a function of the nominal strain as
Kt,
0
for 0 - t<
for
t.
(1.14)
A typical load function is depicted in Figure 1.2 on the next page.
We will let F(x) = Pr{C <_ x) be the cumulative distribution function of
the failure strain. We easily determine the mean load on the wire to be
µ(t) = E[a(t)] = E[Kt1{t < }] = Kt[1 - F(t)].
(1.15)
The higher moments are, for 0 < s < t,
K2stE[1{s < }1{t < fl]
= K2st[1 - F(t)]
and
F(s, t) = E[6(s)e(t)] -
K2stF(s)[1 - F(t)],
for 0 < s < t.
(1.16)
484
Brownian Motion and Related Processes
VIII
Load
t (t)
Strain t
t
Figure 1.2 The load on an elastic wire as a function of nominal strain. At a
strain of the wire fails, and the load carried drops to zero.
Turning to the cable, if it is clamped at the ends and elongated, then each
wire within it is elongated the same amount. The total force SN(t) on the
cable is the sum of the forces exerted by the individual wires. If we assume that the wires are independent and a priori identical, then these wire
forces 6,(t), ,(t), ... are independent and identically distributed random
functions, and
SN(t) =
i=1
i(t)
is the random load experienced by the cable as a function of the cable
strain. An illustration when N = 5 is given in Figure 1.3.
We are interested in the maximum load that the cable could carry without failing. This is
QN = max{SN(t); t ? 0).
1.
Brownian Motion and Gaussian Processes
485
Maximum sustainable
cable load
I
Cable Strain t
Figure 1.3 The load experienced by a cable composed of five elastic wire
strands as a function of the cable strain t.
To obtain an approximation to the distribution of QN we apply the central
limit principle for random functions. This leads us to believe that
XN(t) =
SN(t)
VNY
should, for large N, be approximately a Gaussian process X(t) with mean
zero and covariance function given by (1.16). We write this approximation in the form
SN(t) -- Np.(t) + NI"N-X(t).
(1.17)
When N is large, the dominant term on the right of (1.17) is Nµ(t). Let t*
be the value oft that maximizes µ(t). We assume that t* is unique and that
486
VIII
Brownian Motion and Related Processes
the second derivative of µ(t) is strictly negative at t = t*. We would then
expect that
QN = max SN(t) - Nµ(t*) + VNX(t*).
(1.18)
That is, we would expect that the cable strength would be approximately
normally distributed with mean Nµ(t*) and variance NF(t*, t*). To carry
out a numerical example, suppose that F(x) = 1 - exp{-x5}, a Weibull
distribution with shape parameter 5. It is easily checked that t* = 1/502 =
0.7248, that µ(t*) = 0.5934K, and r(t*, t*) = (0.2792)22K2. A cable composed of N = 30 wires would have a strength that is approximately normally distributed with mean 30(0.5934)K = 17.8K and standard deviation
0.2792K = 1.5292K.
The above heuristics can be justified, and indeed, significant refinements
in the approximation have been made. We have referred to the approach as
the central limit principle for random functions because we have not supplied
sufficient details to label it a theorem. Nevertheless, we will see several more
applications of the principle in subsequent sections of this chapter.
Table 8.1
The cumulative normal distribution
z
(D(x) =
J
v-e
z du
-x
X
-3
-2
-1
0
1
2
3
-2.326
-1.96
-1.645
-1.282
1.282
1.645
1.96
2.236
c(x)
0.00135
0.02275
0.1587
0.5000
0.8413
0.97725
0.99865
0.01
0.025
0.05
0.10
0.90
0.95
0.975
0.99
Exercises
487
Exercises
1.1.
Let (B(t); t ? 0) be a standard Brownian motion.
(a) Evaluate Pr{B(4) : 3IB(0) = 1).
(b) Find the number c for which Pr(B(9) > cl B(0) = 1) = 0.10.
1.2. Let {B(t); t j 0) be a standard Brownian motion and c > 0 a constant. Show that the process defined by W(t) = cB(tlc2) is a standard
Brownian motion.
1.3.
(a) Show that
d4(x)
dx
= 4'(x) = -x4'(x),
where ¢(x) is given in (1.5).
(b) Use the result in (a) together with the chain rule of differentiation
to show that
P(Y, tI x) = ',(y - x) =
o(Y
_otx
)
satisfies the diffusion equation (1.3).
Consider a standard Brownian motion {B(t); t >_ 0) at times
0 < u < u + v < u + v + w, where u, v, w > 0.
1.4.
(a) Evaluate the product moment E[B(u)B(u + v)B(u + v + w)].
(b) Evaluate the product moment
E[B(u)B(u + v)B(u + v + w)B(u + v + w + x)]
where x > 0.
1.5.
Determine the covariance functions for the stochastic processes
(a) U(t) = e-'B(e2i), for t:::, 0.
(b) V(t) = (1 - t)B(tl(1 - t)), for 0 < t < 1.
(c) W(t) = tB(1/t), with W(0) = 0.
B(t) is standard Brownian motion.
488
VIII
Brownian Motion and Related Processes
Consider a standard Brownian motion {B(t); t ? 0} at times
0 < u < u + v < u + v + w, where u, v, w > 0.
1.6.
(a) What is the probability distribution of B(u) + B(u + v)?
(b) What is the probability distribution of B(u) + B(u + v) +
B(u + v + w)?
1.7. Suppose that in the absence of intervention, the cash on hand for a
certain corporation fluctuates according to a standard Brownian motion
{B(t); t ? 0 }. The company manages its cash using an (s, S) policy: If the
cash level ever drops to zero, it is instantaneously replenished up to level
s; If the cash level ever rises up to S, sufficient cash is invested in longterm securities to bring the cash-on-hand down to level s. In the long run,
what fraction of cash interventions are investments of excess cash? Hint:
Use equation (1.13).
Problems
1.1.
Consider the simple random walk
So=0,
S
in which the summands are independent with Pr{ = ± 1 } = 2I n III, Section 5.3, we showed that the mean time for the random walk to first reach
-a < 0 or b > 0 is ab. Use this together with the invariance principle to
show that E[T] = ab, where
T = Ta.h = min(t ? 0; B(t) = -a or B(t) = b},
and B(t) is standard Brownian motion. Hint: The approximate Brownian
motion (1.11) rescales the random walk in both time and space.
1.2.
Evaluate E[eA°"'] for an arbitrary constant A and standard Brownian
motion B(t).
1.3.
For a positive constant e, show that
PrjIB(t)I
t
>e2{1
I=
How does this behave when t is large (t - x)? How does it behave when
t is small (t - 0)?
Problems
1.4.
489
Let a,, ... , a,, be real constants. Argue that
i=1
is normally distributed with mean zero and variance
rt
n
17, a;a; min (ti, tj
i=Ij=1
1.5. Consider the simple random walk
so=O,
in which the summands are independent with Pr{
11 _ 2W e are
going to stop this random walk when it first drops a units below its maximum to date. Accordingly, let
M,, = max Sk,
Osksn
Y = M,, - S,,,
and
T=Ta
(a) Use a first step analysis to show that
1
Pr{M, = 0} = 1 +
a
(b) Why is Pr{M,
*
2} = Pr{M, ? 1 }2, and
Pr{M, >- k)
a
)k?
+a
Identify the distribution of M,.
(c) Let B(t) be standard Brownian motion, M(t) = max{B(u);
0 < u < t}, Y(t) = M(t) - B(t), and T = min{t ? 0; Y(t) = a). Use
the invariance principle to argue that M(r) has an exponential distribution with mean a. Note: r is a popular strategy for timing the
sale of a stock. It calls for keeping the stock as long as it is going
up, but to sell it the first time that it drops a units from its best price
to date. We have shown that E[M(T)] = a, whence E[B(T)]
E[M(T)] - a = 0, so that the strategy does not gain a profit, on average, in the Brownian motion model for stock prices.
490
VIII
Brownian Motion and Related Processes
1.6. Manufacturers of crunchy munchies such as cheese crisps use
compression testing machines to gauge product quality. The crisp, or
whatever, is placed between opposing plates, which then move together.
As the crisp is crunched, the force is measured as a function of the distance that the plates have moved. The output of the compression testing
machine is a graph of force versus distance that is much like Figure 1.3.
What aspects of the graph might be measures of product quality? Model
the test as a row of tiny balloons between parallel plates. Each single balloon might follow a force-distance behavior of the form o- = Ke(1 - q(e)),
where o is the force, K is Young's modulus, e is strain or distance, and
q(e) is a function that measures departures from Hooke's law, to allow for
soggy crisps. Each balloon obeys this relationship up until the random
strain at which it bursts. Determine the mean force as a function of
strain. Use F(x) for the cumulative distribution function of failure strain.
1.7. For n = 0, 1, ... , show that (a) B(n) and (b) B(n)2 - n are martingales (see II, Section 5).
1.8. Computer Challenge A problem of considerable contemporary
importance is how to simulate a Brownian motion stochastic process. The
invariance principal provides one possible approach. An infinite series expression that N. Wiener introduced may provide another approach. Let Z,,,
Z,, ... be a series of independent standard normal random variables. The
infinite series
B(t) =
l
t
Zo + _ -
sin mt
m
Z
0 <_ t <_ 1,
is a standard Brownian motion for 0 <_ t 1. Try to simulate a Brownian
motion stochastic process, at least approximately, by using finite ;sums of
the form
B, (t) _
t
Z+
0:5 t<_ 1.
V -7T
If B(t), 0 < t
1, is a standard Brownian motion on the interval [0, 1],
then B' (t) = (1 + t)B(1/(1 + t)), 0 :!5; t < x, is a standard Brownian motion on the interval [0, x). This suggests
2.
The Maximum Variable and the Reflection Principle
BN (t) = (1 + t)BN (1 +1
t),
491
0 : t < oo,
as an approximate standard Brownian motion. In what ways do these finite approximations behave like Brownian motion? Clearly, they are zero
mean Gaussian processes. What is the covariance function, and how does
it compare to that of Brownian motion? Do the gambler's ruin probabili-
ties of (1.13) accurately describe their behavior? It is known that the
squared variation of a Brownian motion stochastic process is not random,
but constant:
,7
lim
f,-,x
k=1
B(n) - BI
kn l)
= 1.
This is a further consequence of the variance relation E[(OB)2] = At (see
the remark in Section 1.2). To what degree do the finite approximations
meet this criterion?
2.
The Maximum Variable and the
Reflection Principle
Using the continuity of the trajectories of Brownian motion and the symmetry of the normal distribution, we will determine a variety of interesting probability expressions for the Brownian motion process. The starting
point is the reflection principle.
2.1.
The Reflection Principle
Let B(t) be a standard Brownian motion. Fix a value x > 0 and a time
t > 0. Bearing in mind the continuity of the Brownian motion, property
(c) of the definition, consider the collection of sample paths B(u) for
u ? 0 with B(0) = 0 and for which B(t) > x. Since B(u) is continuous and
B(0) = 0, there exists a time 'r, itself a random variable depending on the
particular sample trajectory, at which the Brownian motion B(u) first attains the value x.
VIII
492
Brownian Motion and Related Processes
We next describe a new path B*(u) obtained from B(u) by reflection.
For u > T, we reflect B(u) about the horizontal line at height x > 0 to
obtain
B(u),
B*(u) _ {x - [B(u) - x],
for u <- T,
for u > T.
Figure 2.1 illustrates the construction. Note that B*(t) < x because
B(t) > x.
Because the conditional probability law of the path for u > r, given that
B(t) = x, is symmetric with respect to the values y > x and y < x, and in-
Figure 2.1
The path B(u) is reflected about the horizontal line at x, showing
that for every path ending at B(t) > x, there are two paths, B(u)
and B*(u), that attain the value x somewhere in the interval
0<_u-t.
2.
The Maximum VariaWA and the Reflection Principle
493
dependent of the history prior to T,' the reflection argument displays for
every sample path with B(t) > x two equally likely sample paths, B(u) and
B*(u), for which both
max B(u) > x and
Out
max B*(u) > x.
Ouust
Conversely, by the nature of this correspondence, every sample path B(u)
for which max",,,, B(u) > x results from either of two equally likely sam-
ple paths, exactly one of which is such that B(t) > x. The two-to-one
correspondence fails only if B(t) = x, but because B(t) is a continuous ran-
dom variable (normal distribution), we have Pr{B(t) = x} = 0, and this
case can be safely ignored. Thus we conclude that
Pr{max B(u)
Ouust
> x} = 2Pr{B(t) > x}.
In terms of the maximum process defined by
M(t) = max B(u),
(2.1)
Ouust
and using the notation set forth in (1.8), we have
Pr{M(t) > x} = 2[1 - c,(x)].
2.2.
(2.2)
The Time to First Reach a Level
With the help of (2.2) we may determine the probability distribution of the
random time Tx at which the Brownian motion first attains a prescribed
value x > 0 starting from B(O) = 0. Formally, define the hitting time
TT = min{u ? 0; B(u) = x}.
(2.3)
Clearly, Tx :5 t if and only if M(t) ? x. In words, the Brownian motion attains the level x > 0 before time t if and only if at time t the maximum of
' The argument is not quite complete because the definition asserts that an increment in
a Brownian motion after a fixed time t is independent of the past, whereas here we are
restarting from the random time r. While the argument is incomplete, the assertion is true:
A Brownian path begins afresh from hitting times such as r.
494
VIII
Brownian Motion and Related Processes
the process is at least x. If the two events are equivalent, then their probabilities must be the same. That is,
Pr{T,. s t} = Pr{M(t) > x} = 2[l - (D,(x)]
x
2
r
e-f2n2'
J
d
(2.5)
x
The change of variable 6 = ijVt leads to
Pr{Tx < t} =
-
/`Je-
"- dry.
(2.6)
x/Vt
The probability density function of the random time T is obtained by differentiating (2.6) with respect to t, giving
fT,(t) =
2.3.
sn
xxt
for 0 < t < oo.
(2.7)
The Zeros of Brownian Motion
As a final illustration of the far-reaching consequences of the reflection
principle and equation (2.7), we will determine the probability that a standard Brownian motion B(t), with B(0) = 0, will cross the t axis at least
once in the time interval (t, t + s] for t, s > 0. Let us denote this quantity
by O(t, t + s). The result is
t + s) = Pr{B(u) = 0 for some u in (t, t + s] }
=
=
2
arctan
(2.8)
IT
2
arccos tt/(.
IT
First, let us define some notation concerning the hitting time T, defined in
(2.3). Let
H,(z, x) = Pr{T,
tIB(0) = z}
be the probability that a standard Brownian motion starting from B(0) = z
2.
The Maximum Variable and the Reflection Principle
495
will reach the level x before time t. In equation (2.5) we gave an integral
that evaluated
for x > 0.
H,(0, x) = Pr{T,. < tIB(0) = 0),
The symmetry and spatial homogeneity of the Brownian motion make it
clear that H,(0, x) = H,(x, 0). That is, the probability of reaching x > 0
starting from B(0) = 0 before time t is the same as the probability of
reaching 0 starting from B(0) = x. Consequently, from (2.7) we have
H,(0, x) = H,(x, 0) = Pr{T < tjB(0) = x}
(2.9)
d.
e
We will condition on the value of the Brownian motion at time t and use
the law of total probability to derive (2.8). Accordingly, we have
x
19(t, t + s) =
f
Pr{B(u) = 0 for some u in (t, t + s] I B(t) = x}4,(x) dx,
where 4,(x) is the probability density function for B(t) as given in (1.7).
Then, using (2.9),
x
1S(t, t + s) = 2
f H,(x, 0)0,(x) dx
0
x
=2f
0
s
f0
1 ,ne-x',af) d}
f[J x
0
e-x'1(2£)-x21(21) d x
0
To evaluate the inner integral, we let
v=
x-( 1
1
21+ t
whence
+ - dx,
)
1
(dv = x
t
e
sn d
dx
496
Brownian Motion and Related Processes
VIII
and so
;Vt fo ( 1+
v e-`' d+_ 3n
1
_ \
_7r
o
It
d
)1f
f (t + )\'
df
o
The change of variable 71 = N16-t gives
i(t, t + s) =
2
r
f
d
1
,=
2
2 arccos
t/(t + s).
o
Finally, Exercise 2.2 asks the student to use standard trigonometric identities to show the equivalence arctan s/t = arccos 't/(t+ s).
Exercises
2.1. Let {B(t); t ? 0} be a standard Brownian motion, with B(0) = 0,
and let M(t) = max{B(u); 0 s u < t}.
(a) Evaluate Pr(M(4) <_ 2).
(b) Find the number c for which Pr{M(9) > c} = 0.10.
2.2. Show that
arctan
s/t = arccos tl(s + t).
2.3. Suppose that net inflows to a reservoir are described by a standard
Brownian motion. If at time 0 the reservoir has x = 3.29 units of water on
hand, what is the probability that the reservoir never becomes empty in
the first t = 4 units of time?
2.4. Consider the simple random walk
S,.=6,+...+6,..
So=0,
in which the summands are independent with Pr{ = ±1) = z. Let
Mn = maxo<k<,, Sk. Use a reflection argument to show that
a) =2Pr(S,, > a) +Pr{S,, = a),
a>0.
497
Problems
2.5. Let ro be the largest zero of a standard Brownian motion not exceeding a > 0. That is, To = max {u ? 0; B(u) = 0 and u < a}. Show that
2
- aresin tla.
Pr{To < t} =
Ir
2.6. Let T, be the smallest zero of a standard Brownian motion that exceeds b > 0. Show that
Pr{T1 < t} =
2
- arccos
ir
blt.
Problem's
2.1. Find the conditional probability that a standard Brownian motion is
not zero in the interval (t, t + b] given that it is not zero in the interval
(t, t + a], where 0 < a < b and t > 0.
2.2. Find the conditional probabiiity that a standard Brownian motion is
not zero in the interval (0, b] given that it is not zero in the interval (0, a],
where 0 < a < b. Hint: Let t -* 0 in the result of Problem 2.1.
2.3.
For a fixed t > 0, show that M(t) and I B(t)I have the same marginal
probability distribution, whence
fMU)(Z)
(Here M(t) =
V - 0( V
for
z > 0.
Show that
E[M(t)] =
V-2-dir.
For 0 < s < t, do (M(s), M(t)) have the same joint distribution as
(IB(s)I, IB(t)I )?
2.4. Use the reflection principle to obtain
Pr{M(t) ? z, B(t)
x} = Pr{B(t) ? 2z - x}
=1-
2z - x
OD(
N/t )
for O < x < m.
498
VIII
Brownian Motion and Related Processes
(M(t) is the maximum defined in (2.1).) Differentiate with respect to x, and
then with respect to z, to obtain the joint density function for M(t) and
B(t):
fM(,)
x) =
2z-x 2
Vt
t
'0(
2z-x
"fit ).
2.5.
Show that the joint density function for M(t) and Y(t) = M(t) - B(t)
is given by
_
y)
z+y 2
t
Vt
z+y
0
V t )-
2.6. Use the result of Problem 2.5 to show that Y(t) = M(t) - B(t) has
the same distribution as JB(t)I.
3.
Variations and Extensions
A variety of processes derived from Brownian motion find relevance and
application in stochastic modeling. We briefly describe a few of these.
3.1
Reflected Brownian Motion
Let {B(t); t ? 0} be a standard Brownian motion process. The stochastic
process
B(t),
R(t) = B(t)I = -B(t),
if B(t) ? 0,
if B(t) < 0,
is called Brownian motion reflected at the origin, or, more briefly, reflected Brownian motion. Reflected Brownian motion reverberates back to
positive values whenever it reaches the zero level and, thus, might be used
to model the movement of a pollen grain in the vicinity of a container
boundary that the grain cannot cross.
Since the moments of R(t) are the same as those of B(t)I, the mean and
variance of reflected Brownian motion are easily determined. Under the
condition that R(0) = 0, for example, we have
3.
Variations and Extensions
499
E[R(t)] = f Jxjcb,(x) dx
_x
x
=2
exp(-x2/2t) dx
(3.1)
0
=
V-2-dir.
The integral was evaluated through the change of variable y = x/\. Also,
Var[R(t)] = E[R(t)2] - {E[R(t)]}2
= E[B(t)2] - 2t/ir
2
(3.2)
t.
7T)
Reflected Brownian motion is a second example of a continuous-time,
continuous-state-space Markov process. Its transition density p(y, tjx) is
derived from that of Brownian motion by differentiating
Pr(R(t) :5 yJR(0) = x} = Pr{-y < B(t) : yjB(0) = x}
Y
=
f 0,(z - x) dz
_y
with respect to y to get
P(Y, t1x) = 4,(Y - x) +
x)
= O,(Y - x) + O,(Y + x).
3.2.
Absorbed Brownian Motion
Suppose that the initial value B(O) = x of a standard Brownian motion
process is positive, and let T be the first time that the process reaches zero.
The stochastic process
B(t),
A(t)_10
for t -- T,
fort>T
VIII
500
Brownian Motion and Related Processes
is called Brownian motion absorbed at the origin, which we will shorten
to absorbed Brownian motion. Absorbed Brownian motion might be used
to model the price of a share of stock in a company that becomes bankrupt at some future instant. We can evaluate the transition probabilities for
absorbed Brownian motion by another use of the reflection principle introduced in Section 2. For x > 0 and y > 0, let
G,(x, y) = Pr(A(t) > yjA(0) = x}
0IB(0) = x}.
= Pr{B(t) > y,
(3.3)
To determine (3.3), we start with the obvious relation
Pr{B(t) > yIB(0) = x} =
0IB(0) = x}.
G,(x, y) + Pr(B(t) > y,
B(u)
y
x
0
/-
V,.. , I
B *(u)
_y
N
11
t
Figure 3.1
For every path B(u) starting at x, ending at B(t) > y, and reaching
zero in the interval, there is another path B*(u) starting at x and
ending at B*(t) < -y.
3.
Variations and Extensions
501
The reflection principle is applied to the last term; Figure 3.1, previous
page, is the appropriate picture to guide the analysis. We will argue that
Pr{B(t) > y, mino51S,B(u) <_ OIB(O) = x}
= Pr(B(t) < -y,
OIB(O)
= x}
(3.4)
= Pr{B(t) < -ylB(O) = x} = 4),(-y - x).
The reasoning behind (3.4) goes as follows: Consider a path starting at
x > 0, satisfying B(t) > y, and which reaches zero at some intermediate
time T. By reflecting such a path about zero after time r, we obtain an
equally likely path starting from x and assuming a value below -y at time
t. This implies the equality of the first two terms of (3.4). The equality of
the last terms is clear from their meaning, since the condition that the min-
imum be below zero is superfluous in view of the requirement that the
path end below -y (y > 0). Inserting (3.4) into (3.3) yields
G,(x, y) = Pr{B(t) > yIB(0) = x) - Pr(B(t) < -ylB(O) = x}
= 1 - INy - x) - 4),(-(y + x))
= I NY + x) - q,(y - x)
yf X0,(z)dz=(D(y
y-x
N/i
(3.5)
)(P(yVtx).
From (3.3) and (3.5), we obtain the transition distribution for absorbed
Brownian motion:
Pr(A(t) > yjA(0) = x} = c)(y
x) - c)(yN/tx).
(3.6)
Under the condition that A(0) = x > 0, A(t) is a random variable that has
both discrete and continuous parts. The discrete part is
Pr(A(t) = OJA(O) = x} = 1 - G,(x, 0)
X
=1- f P,(z) dz
-x
= 2[1 - (D,(x)l.
502
VIII
Brownian Motion and Related Processes
In the region y > 0, A(t) is a continuous random variable whose transition
density p(y, tlx) is obtained by differentiating with respect to y in (3.6) and
suitably changing the sign:
p(y, t1 x) _ 4Xy - x) - 4 (Y + x).
3.3.
The Brownian Bridge
The Brownian bridge (B°(t); t ? 0) is constructed from a standard Brownian motion {B(t); t ? 0) by conditioning on the event {B(0) = B(1) = 0) .
The Brownian bridge is used to describe certain random functionals aris-
ing in nonparametric statistics, and as a model for the publicly traded
prices of bonds having a specified redemption value on a fixed expiration
date.
We will determine the probability distribution for B°(t) by using the
conditional density formula for jointly normally distributed random variables derived in II, Problem 4.8. First, for 0 < t < 1, the random variables
B(t) and B(1) - B(t) are independent and normally distributed according
to the definition of Brownian motion. It follows that X = B(t) and
Y = B(1) = B(t) + {B(I) - B(t)) have a joint normal distribution (See I,
Section 4.6) for which we have determined µx = µ,. = 0, o-x = \, o = 1,
and p = Cov[X, Y]/a o = \. Using the results of II, Problem 4.8, it then
follows that given Y = B(1) = y, the conditional distribution of X = B(t)
is normal, with
La-x
lax r = !-fix +
o.r
(y - Ay) = YN/'-t = 0
when y = 0,
and
o'x r = o'x
=
t(1 - t).
For the Brownian bridge, B°(t) is normally distributed with E[B°(t)] = 0
and Var[B°(t)] = t(1 - t). Notice how the condition B(0) = B(1) = 0
causes the variance of B°(t) to vanish at t = 0 and t = 1.
The foregoing calculation of the variance can be extended to determine
the covariance function. Consider times s, t with 0 < s < t < 1. By first
obtaining the joint distribution of (B(s), B(t), B(1)), and then the conditional joint distribution of (B(s), B(t)), given that B(1) = 0, one can verify
that the Brownian bridge is a normally distributed stochastic process with
3.
Variations and Extensions
503
mean zero and covariance function IF(s, t) = Cov[B°(s), B°(t)] = s(1 - t),
for 0 < s < t < 1. (See Problem 3.3 for an alternative approach.)
Let X, X,, ... be independent and identically distributed random variables. The empirical
cumulative distribution function corresponding to a sample of size N is
Example The Empirical Distribution Function
defined by
Fn,(t) =
N#{X;
---- t for i = 1, ... , N}
(3.7)
where
_
'(t)
1,
ifX;<_t,
10,
if X; > t.
The empirical distribution function is an estimate, based on the observed
sample, of the true distribution function F(t) = Pr{X < t). We will use the
central limit principle for random functions (Section 1.4) to approximate
the empirical distribution function by a Brownian bridge, assuming that
the observations are uniformly distributed over the interval (0, 1). (Problem 3.9 calls for the student to explore the case of a general distribution.)
In the uniform case, F(t) = t for 0 < t < 1, and µ(t) = E[t;(t)] = F(t) = t.
For the higher moments, when 0 < s < t < 1,
and I'(s, t) =
fi(t)] = E[6(s)4(t)] -
F(s) = s,
s - st =
s(1 - t).
In view of (3.7), which expresses the empirical distribution function in
terms of a sum of independent and identically distributed random functions, we might expect the central limit principle for random functions to
yield an approximation in terms of a Gaussian limit. Following the guidelines in Section 1.4, we would expect that
XN(t)
_" {;(t) - A(t))
VN-
NF,,, (t) - Nt
VN
= \{FN(t) - t}
504
VIII
Brownian Motion and Related Processes
would converge, in an appropriate sense, to a Gaussian process with zero
mean and covariance I'(s, t) = s(1 - t), for 0 < s < t < 1. As we have
just seen, this process is a Brownian bridge. Therefore, we would expect
the approximation
Such approximations are heavily used in the theory of nonparametric
statistics.
3.4.
Brownian Meander
Brownian meander {B+(t); t ? 0} is Brownian motion conditioned to be
positive. Recall (3.3) and (3.5):
B(u) > OIB(0) = x}
G,(x, y) = Pr{B(t) > y,
- D(,)
V 4)(Y-X),
so that
G,(x, 0) =
B(u) > OIB(O) = x}
_(D(Vt)(D(
It N-X)
The transition law for Brownian meander is
Pr{B+(t) > yIB+(0) = x) = Pr(B(t) >
B(u) > 0, B(0) = x},
whence
Pr{B+(t) > yIB+(0) = x} =
G,(x, y)
G,(x, 0)
4)(y + x)
(D(
-
4)(y
-Vx)
t) - (D( t)
3.
Variations and Extensions
505
Of most interest is the limiting case as x -> 0,
(D(y+x)
0(y
Pr{B+(t) > yIB+(0) = 0}X0= lim (x
1
-(
X)
xl
v
0(0)
=e
A simple integration yields the mean
00
E[B+(t)IB+(0) = 01 = J Pr{B+(t) > YIB+(0) = 0} dy
0
x
=J
dy
0
00
= zf e dy
_x
1
-+ '
Exercises
3.1. Show that the cumulative distribution function for reflected Brownian motion is
Pr{R(t) < yIR(0) = x} _
(YX)
X)
- (D
_ (D(yN/t-xl +
x+Y
(-7t-)
(D(Yx
l-1
(X_ Y
- 4 -7t-).
Evaluate this probability when x = 1, y = 3, and t = 4.
VIII
506
Brownian Motion and Related Processes
3.2.
The price fluctuations of a share of stock of a certain company are
well described by a Brownian motion process. Suppose that the company
is bankrupt if ever the share price drops to zero. If the starting share price
is A(0) = 5, what is the probability that the company is bankrupt at time
t = 25? What is the probability that the share price is above 10 at time
t = 25?
3.3. The net inflow to a reservoir is well described by a Brownian motion. Because a reservoir cannot contain a negative amount of water, we
suppose that the water level R(t) at time t is a reflected Brownian motion.
What is the probability that the reservoir contains more than 10 units of
water at time t = 25? Assume that the reservoir has unlimited capacity and
that R(0) = 5.
Suppose that the net inflows to a reservoir follow a Brownian motion. Suppose that the reservoir was known to be empty 25 time units ago
but has never been empty since. Use a Brownian meander process to evaluate the probability that there is more than 10 units of water in the reservoir today.
3.4.
3.5. Is reflected Brownian motion a Gaussian process? Is absorbed
Brownian motion (cf. Section 1.4)?
Problems
3.1.
Let B,(t) and B,(t) be independent standard Brownian motion
processes. Define
R(t) =
B,(t)2 + B,(t)2,
t ? 0.
R(t) is the radial distance to the origin in a two-dimensional Brownian motion. Determine the mean of R(t).
3.2. Let B(t) be a standard Brownian motion process. Determine the
conditional mean and variance of B(t), 0 < t < 1, given that B(1) = b.
3.3. Let B(t) be a standard Brownian motion. Show that B(u) - uB(1),
0 < u < 1, is independent of B(1).
Problems
507
(a) Use this to show that B°(t) = B(t) - tB(1), 0 s t < 1, is a Brownian bridge.
(b) Use the representation in (a) to evaluate the covariance function for
a Brownian bridge.
3.4. Let B(t) be a standard Brownian motion. Determine the covariance
function for
WI(s) = (1 - s)B(
s
1-
s),
0 < s < 1,
and compare it to that for a Brownian bridge.
3.5. Determine the expected value for absorbed Brownian motion A(t)
at time t = 1 by integrating the transition density (3.6) according to
x
E[A(1)IA(0) = x] = f y p(y, 1Ix) dy
0
x
=
f
y[(k(y - x) - 4(y + x)] dy.
0
The answer is E[A(1)IA(0) = x] = x. Show that E[A(t)IA(0) = x] = x for
all t > 0.
3.6. Let M = max{A(t); t 0} be the largest value assumed by an absorbed Brownian motion A(t). Show that Pr{M > zIA(0) = x} = x/z for
0<x<z.
3.7. Let to = 0 < t, < t2 < ... be time points, and define X,, = A(t,,),
where A(t) is absorbed Brownian motion starting from A(0) = x. Show
that
is a nonnegative martingale. Compare the maximal inequality
(5.7) in II with the result in Problem 3.6.
3.8. Show that the transition densities for both reflected Brownian motion and absorbed Brownian motion satisfy the diffusion equation (1.3) in
the region 0 < x < oo.
3.9. Let F(t) be a cumulative distribution function and B°(u) a Brownian bridge.
508
VIII
Brownian Motion and Related Processes
(a) Determine the covariance function for B°(F(t)).
(b) Use the central limit principle for random functions to argue that
the empirical distribution functions for random variables obeying
F(t) might be approximated by the process in (a).
4.
Brownian Motion with Drift
Let {B(t); t ? 0) be a standard Brownian motion process, and let µ and
o, > 0 be fixed. The Brownian motion with drift parameter µ and variance
parameter 0-22 is the process
X(t) = µt + o-B(t)
for
t ? 0.
(4.1)
Alternatively, Brownian motion with drift parameter µ and variance parameter o-2 is the process whose increments over disjoint time intervals are
independent (property (b) of the definition of standard Brownian motion)
and whose increments X(t + s) - X(t), t, s > 0, are normally distributed
with mean ps and variance o-2s. When X(0) = x, we have
Pr{X(t) : yIX(0) = x) = Pr{µt + oB(t) : ylo-B(0) = x}
= Pr j B(t)
c
Y
o µt I
= o-I
=4CDr(y
Cr
µt)=(D(y
µt).
Brownian motion with drift is not symmetric when µ 0 0, and the reflection principle cannot be used to compute the distribution of the maximum
of the process. We will use an infinitesimal first step analysis to determine
some properties of Brownian motion with drift. To set this up, let us introduce some notation to describe changes in the Brownian motion with
drift over small time intervals of length At. We let OX = X(t + At) - X(t)
and AB = B(t + At) - B(t). Then AX = µ0t + o-AB, and
X(t + At) = X(t) + AX = X(t) + µ0t + o-AB.
(4.2)
We observe that the conditional moments of AX, given X(t) = x, are
E[AXIX(t) = x] = µ0t + o-E[AB] = µ0t,
(4.3)
Var[OXIX(t) = x] = 0-2E[(AB)2] = o-2At,
(4.4)
4.
Brownian Motion with Drift
509
and
E [( AX)2I X(t) = x ]
_ o2Ot + ( µ0t)22 = o 20t + o( t1t) ,
(4 . 5 )
for c > 2.
(4.6)
while
E[(1 X)`] = o(At)
4.1.
The Gambler's Ruin Problem
Let us suppose that X(O) = x, and that a < x and b > x are fixed quantities. We will be interested in some properties of the random time T at
which the process first assumes one of the values a or b. This so-called hitting time is formally defined by
T=Tub=min(t? 0;X(t)=aorX(t)=b}.
Analogous to the gambler's ruin problem in a random walk (III, Section
5.3), we will determine the probability that when the Brownian motion
exits the interval (a, b), it does so at the point b. The solution for a standard Brownian motion was obtained in Section 1 by using the invariance
principle. Here we solve the problem for Brownian motion with drift by
instituting an infinitesimal first step analysis.
Theorem 4.1
For a Brownian motion with drift parameter µ and variance parameter Q2, and a < x < b,
e-2gda2 - e-2"'°2
u(x) = Pr{X(T,,,) = bjX(0) = x} =
e-2,.bla22 _
e-
2p,,/o2 .
(4.7)
Proof Our proof is not entirely complete in that we will assume (i) that
u(x) is twice continuously differentiable, and (ii) that we can choose a
time increment At so small that exiting the interval (a, b) prior to time At
can be neglected. With these provisos, at time At the Brownian motion
will be at the position X(0) + .X = x + OX, and the conditional probability of exiting at the upper point b is now u(x + AX). Invoking the law
of total probability, it must be that u(x) = Pr{X(T) = b1X(0) = x) =
E[Pr{X(T) = bIX(0) = x, X(At) = x + OX}IX(0) = x] = E[u(x + OX)],
where a < x < b.
Vlll
510
Brownian Motion and Related Processes
The next step is to expand u(x + AX) in a Taylor series, whereby
u(x +
= u(x) + u'(x)AX + ZU''(x)(AX)2 + o(AX)2. Then
u(x) = E[u(x + AX)]
= u(x) + u'(x)E[AX] + 2'u'"(x)E[(OX)2] + E[o(At)].
We use (4.3), (4.5), and (4.6) to evaluate the moments of AX, obtaining
u(x) = u(x) + u'(x)AAt + ;u'"(x)O'At + o(At),
which, after subtracting u(x), dividing by At, and letting At -* 0, becomes
the differential equation
0 = µu'(x) + ;o_2u"(x) for a < x < b.
(4.8)
The solution to (4.8) is
z
u(x) = Ae_21) ° + B,
where A and B are constants of integration. These constants are determined by the conditions u(a) = 0 and u(b) = 1. In words, the probability
of exiting at b if the process starts at a is zero, while the probability of exiting at b if the process starts at b is one. When these conditions are used
to determine A and B, then (4.7) results.
Example Suppose that the fluctuations in the price of a share of stock
in a certain company are well described by a Brownian motion with drift
µ = 1/10 and variance 0-22 = 4. A speculator buys a share of this stock at
a price of $100 and will sell if ever the price rises to $110 (a profit) or
drops to $95 (a loss). What is the probability that the speculator sells at a
profit? We apply (4.7) with a = 95, x = 100, b = 110, and 2µ/o22 _
2(0.1)/4 = 1/20. Then
e-100/20 - e-95/20
Pr(Sell at profit) = e-na2o - e-9sao = 0.419.
The Mean Time to Exit an Interval
Using another infinitesimal first step analysis, the mean time to exit an interval may be determined for Brownian motion with drift.
4.
Brownian Motion with Drift
511
Theorem 4.2 For a Brownian motion with drift parameter /.Land variance parameter o-Z, and a < x < b,
E[T,,,>I X(0)
= x] =
1A [u(x)(b - a) - (x - a)],
(4.9)
where u(x) is given in (4.7).
Proof Let v(x) = E[TT,,,IX(0) = x]. As in the proof of Theorem 4.1, we
will assume (i) that v(x) is twice continuously differentiable, and (ii) that
we can choose a time increment At so small that exiting the interval (a, b)
prior to time At can be neglected. With these provisos, after time At the
Brownian motion will be at the position X(O) + AX = x + AX, and the
conditional mean time to exit the interval is now At + v(x + AX). Invoking the law of total probability, it must be that v(x) = E[TI X(0) = x] =
E[At + E{T - AtIX(0) = x, X(At) = x + AX}IX(0) = x] = At +
E[v(x + AX)], where a < x < b.
The next step is to expand v(x + .X) in a Taylor series, whereby
v(x + X) = v(x) + v'(x)OX + ;v''(x)(OX)' + o(OX)'. Then
v(x) = At + E[v(x + AX)]
= At + v(x) + v'(x)E[AX] + Zv'"(x)E[(zXX)2] + E[o(OX)2].
We use (4.3), (4.5), and (4.6) to evaluate the moments of AX, obtaining
v(x) = At + v(x) + v'(x)AAt + ;v"(x)o-2At + o(At),
which, after subtracting v(x), dividing by At, and letting At - 0, becomes
the differential equation
-1 = pv'(x) + ;cr2v"(x) for a < x < b.
(4.10)
Since it takes no time to reach the boundary if the process starts at the
boundary, the conditions are v(a) = v(b) = 0. Subject to these conditions,
the solution to (4.10) is uniquely given by (4.9), as is easily verified (Problem 4.1).
Example A Sequential Decision Procedure A Brownian motion X(t)
either (i) has drift p. = +;5 > 0, or (ii) has drift µ = - S < 0, and it is
desired to determine which is the case by observing the process. The
512
VIII
Brownian Motion and Related Processes
process will be monitored until it first reaches the level b > 0, in which
case we will decide that the drift is µ = +18, or until it first drops to the
level a < 0, which occurrence will cause us to decide in favor of µ = -ZS.
This decision procedure is, of course, open to error, but we can evaluate
these error probabilities and choose a and b so as to keep the error probabilities acceptably small. We have
a = Pr{Decide µ =
-ZS}
= Pr{X(T) = bI E[X(t)] = -2'5t}
1 - e+woz
(using (4.7))
=
_ e+&,' 2
e+s'
(4.11)
2
and
1 - /3 = Pr{Decide µ = +2Sµ = +25}
= Pr{X(T) = bIE[X(t)] =+2 6t}
(4.12)
_ 1 - e-&-''p2
- e-w-2 e-&,/v
-
If acceptable levels of the error probabilities a and /3 are prescribed, then
we can solve in (4.11) and (4.12) to determine the boundaries to be used
in the decision procedure. The reader should verify that these boundaries
are
a
8
log(1
aand b = 8
log(1
a
R).
(4.13)
For a numerical example, if 0,2 = 4 and we are attempting to decide between µ = -2 and µ = +2, and the acceptable error probabilities are chosen to be a = 0.05 and /3 = 0.10, then the decision boundaries that should
be used are a = -4 log(0.95/0.10) = -9.01, and b = 4 log(0.90/0.05)
11.56.
In the above procedure for deciding the drift of a Brownian motion, the
observation duration until a decision is reached will be a random variable
whose mean will depend upon the true value of the drift. Using (4.9) with
x = 0 and µ replaced by ±15 gives us the mean observation interval, as a
function of the true mean µ:
4.
Brownian Motion with Drift
513
E[TI p. = -ZS] = 2(S) [(1 - a) log(' -
a
- a log
and
2(S)z[(1
E[TItk = +ZS]
=
2
- R)
log(1
a R)
- R log('
a
We have developed a sequential decision procedure for evaluating the
drift of a Brownian motion. However, invoking the invariance principle
leads us to believe that similar results should maintain, at least approximately, in analogous situations in which the Brownian motion is replaced
by a partial sum process of independent and identically distributed summands. The result is known as Wald's approximation for his celebrated sequential probability ratio test of a statistical hypothesis.
The Maximum of a Brownian Motion with Negative Drift
Consider a Brownian motion with drift {X(t)}, where the drift parameter
µ is negative. Over time, such a process will tend toward ever lower values, and its maximum M = max { X(t) - X(0); t ? 0) will be a well-defined and finite random variable. Theorem 4.1 will enable us to show that
M has an exponential distribution with parameter 21µI/o.2. To see this, let
us suppose that X(0) = 0 and that a < 0 < b are constants. Then Theorem
4.1 states that
1 - e-zpd°z
Pr{X(Tb) = bI X(0) = x} = e-10m2 _ e-z,°i°z,
(4.14)
where
is the random time at which the process first reaches a < 0 or
b > 0. That is, the probability that the Brownian motion reaches b > 0 be-
fore it ever drops to a < 0 is given by the right side of (4.14). Because
both µ<0and a<0,then aµ>0and
e`W,1J ' = e-'- ''°` --> 0 as a - -oo,
lim Pr{X(Tnb) = b) =
1 -0
e-zpei°2
_0
and then
= e -zpbi°z
514
VIII
Brownian Motion and Related Processes
But as a - -oo, the left side of (4.14) becomes the probability that the
process ever reaches the point b, that is, the probability that the maximum
M of the process ever exceeds b. We have deduced, then, the desired exponential distribution
Pr(M>b) =e'µ1°'
4.2.
b > 0.
(4.15)
Geometric Brownian Motion
A stochastic process (Z(t); t ? 0) is called a geometric Brownian motion
with drift parameter a if X(t) = log Z(t) is a Brownian motion with drift
a - io2 and variance parameter a-2. Equivalently, Z(t) is geometric
Brownian motion starting from Z(0) = z if
Z(t) = Zex(t) = ze(n- v-)r+oe(n
l1
,
(4.16)
where B(t) is a standard Brownian motion starting from B(0) = 0.
Modern mathematical economists usually prefer geometric Brownian
motion over Brownian motion as a model for prices of assets, say shares
of stock, that are traded in a perfect market. Such prices are nonnegative
and exhibit random fluctuations about a long-term exponential decay or
growth curve. Both of these properties are possessed by geometric
Brownian motion, but not by Brownian motion itself. More importantly,
if to < t, <
< t are time points, then the successive ratios
Z(t,) Z(t)
Z(t)
Z(t,,-,)
are independent random variables, so that crudely speaking, the percentage changes over nonoverlapping time intervals are independent.
We turn to determining the mean and variance of geometric Brownian
motion. Let 6 be a normally distributed random variable with mean zero
and variance one. We begin by establishing the formula
E[e"{]=e="2,
which results immediately from
-x<A<oo,
4.
Brownian Motion with Drift
515
x
1=
j
1
(area under normal density)
2 du
-x
=
e av'-zau+A' du
1
J
-x
11
X
= e-
AZ r e""
1
-x
1
N /5_1
e- 112 du
=
To obtain the mean of geometric Brownian motion Z(t) = ze"' _
we use the fact that = B(t)/ is normally distributed with
mean zero and variance one, whence
zee°-=°2''+°ac",
E[Z(t)I Z(0) = z] =
zE[e(«_°-)1+°e(I)]
= Ze(«
a2),E[e°"I"']
7
( = B(t)lN17)
(4.17)
at
Equation (4.17) has interesting economic implications in the case where
a is positive but small relative to the variance parameter .2 On the one
hand, if a is positive, then the mean E[Z(t)] = z exp(at) -4 oo as t - oo.
On the other hand, if a is positive but a < ZO.2, then a - ;Q.2 < 0, and
X(t) = (a - 20 2)t + QB(t) is drifting in the negative direction. As a consequence of the law of large numbers, it can be shown that X(t) - -00 as
t - oo under these circumstances, so that Z(t) = z exp[X(t)] - exp(-cc)
= 0. The geometric Brownian motion process is drifting ever closer to
zero, while simultaneously, its mean or expected value is continually increasing! Here is yet another stochastic model in which the mean value
function is entirely misleading as a sole description of the process.
The variance of the geometric Brownian motion is derived in much the
same manner as the mean. First
E[Z(t)2IZ(0) = Z] = z2E[e211t) ] = z2E[e2c«
= zzezc«+_°=,t
°'-1,+z°ao,]
(as in (4.17))
516
Brownian Motion and Related Processes
VIII
and then
Var[Z(t)] = E[Z(t)2] - { E[Z(t)] } 2
= Z2e 2(a+-a2), Z2e2at(e(' 2,
=
(4.18)
z2e2at
- 1).
Because of their close relation as expressed in the definition (4.16), many
results for Brownian motion can be directly translated into analogous results for geometric Brownian motion. For example, let us translate the
gambler's ruin probability in Theorem 4.1. For A < 1 and B > 1, define
T=TA.a=minjt?0; Z(t) =AorZ(t) =BI.
Theorem 4.3 For a geometric Brownian motion with drift parameter
a and variance parameter or, and A < 1 < B,
1-
1
PrIZ(T) = B =
B1-2aha2
A'-2ada:
- A'-2aia2
.19)
(4.19)
Example Suppose that the fluctuations in the price of a share of stock
in a certain company are well described by a geometric Brownian motion
with drift a = 1/10 and variance o2 = 4. A speculator buys a share of this
stock at a price of $100 and will sell if ever the price rises to $110
(a profit) or drops to $95 (a loss). What is the probability that the speculator sells at a profit? We apply (4.19) with A = 0.95, B = 1.10 and
1 - 2a/o-2 = 1 - 2(0.1)/4 = 0.95. Then
Pr{Sell at profit} =
I - 0.95095
1.100.91 -
= 0.3342.
0.950.95
Example The Black-Scholes Option Pricing Formula A call, or warrant, is an option entitling the holder to buy a block of shares in a given
company at a specified price at any time during a stated interval. Thus the
call listed in the financial section of the newspaper as
Hewlett
Aug
$60
$6
means that for a price of $6 per share, one may purchase the privilege (option) of buying the stock of Hewlett-Packard at a price of $60 per share at
4.
Brownian Motion with Drift
517
any time between now and August (by convention, always the third Friday
of the month). The $60 figure is called the striking price. Since the most
recent closing price of Hewlett was $59, the option of choosing when to
buy, or not to buy at all, carries a premium of $7 = $60 + $6 - $59 over
a direct purchase of the stock today.
Should the price of Hewlett rise to, say, $70 between now and the third
Friday of August, the owner of such an option could exercise it, btlying at
the striking price of $60 and immediately selling at the then current market price of $70 for a $10 profit, less, of course, the $6 cost of the option
itself. On the other hand, should the price of Hewlett fall, the option
owner's loss is limited to his $6 cost of the option. Note that the seller
(technically called the "writer") of the option has a profit limited to the $6
that he receives for the option but could experience a huge loss should the
price of Hewlett soar, say to $100. The writer would then either have to
give up his own Hewlett shares or buy them at $100 on the open market
in order to fulfill his obligation to sell them to the option holder at $60.
What should such an option be worth? Is $6 for this privilege a fair
price? While early researchers had studied these questions using a geometric Brownian motion model for the price fluctuations of the stock, they
all assumed that the option should yield a higher mean return than the
mean return from the stock itself because of the unlimited potential risk to
the option writer. This assumption of a higher return was shown to be false
in 1973 when Fisher Black, a financial consultant with a Ph.D. in applied
mathematics, and Myron Scholes, an assistant professor in finance at MIT,
published an entirely new and innovative analysis. In an idealized setting
that included no transaction costs and an ability to borrow or lend limitless amounts of capital at the same fixed interest rate, they showed that an
owner, or a writer, of a call option could simultaneously buy or sell the underlying stock ("program trading") in such a way as to exactly match the
returns of the option. Having available two investment opportunities with
exactly the same return effectively eliminates all risk, or randomness, by
allowing an investor to buy one while selling the other. The implications
of their result are many. First, since writing an option potentially carries
no risk, its return must be the same as that for other riskless investments
in the economy. Otherwise, limitless profit opportunities bearing no risk
would arise. Second, since owning an option carries no risk, one should
not exercise it early, but hold it until its expiration date, when, if the market price exceeds the striking price, it should be exercised, and otherwise
518
VIII
Brownian Motion and Related Processes
not. These two implications then lead to a third, a formula that established
the worth, or value, of the option.
The Black-Scholes paper spawned hundreds, if not thousands, of further academic studies. At the same time, their valuation formula quickly
invaded the financial world, where soon virtually all option trades were
taking place at or near their Black-Scholes value. It is remarkable that the
valuation formula was adopted so quickly in the real world in spite of the
esoteric nature of its derivation and the ideal world of its assumptions.
In order to present the Black-Scholes formula, we need some notation.
Let S(t) be the price at time t of a share of the stock under study. We assume that S(t) is described by a geometric Brownian motion with drift parameter a and variance parameter o=. Let F(z, T) be the value of an option,
where z is the current price of the stock and r is the time remaining until
expiration. Let a be the striking price. When r = 0, and there is no time
remaining, one exercises the option for a profit of z - a if z > a (market
price greater than striking price) and does not exercise the option, but lets
it lapse, or expire, if z : a. This leads to the condition
F(z,0)=(z-a)+=max{z-a,0}.
The Black-Scholes analysis resulted in the valuation
F(z, T) = e-ME[(Z(T) - a)+IZ(0) = z],
(4.20)
where r is the return rate for secure, or riskless, investments in the economy, and where Z(t) is a second geometric Brownian motion having drift
parameter r and variance parameter o . Looking at (4.20), the careful
reader will wonder whether we have made a mistake. No, the worth of the
option does not depend on the drift parameter a of the underlying stock.
In order to put the valuation formula into a useful form, we write
Z(T) = zee`-
B(T)/N//T,
(4.21)
and observe that
ze
O )T+O\'Tf >a
is the same as
> vo =
20' T
dVI-T
(4.22)
4.
Brownian Motion with Drift
519
Then
erTF(z, T) = E[(Z(T)
=
- a)+IZ(0) = z]
E[(ze'r-'?2)T+OV'Tf - a)+l
W
[ze`,.-' u',-+ox/-n_ - al O(v) dv.
= J
Vo
x
x
e'2 'T J e°","O(v) dv
- a J 4(v) dv.
V0
V0
Completing the square in the form
-;v'+0-
-![(v-0-)Z20-27]
v=
N/IT
shows that
ean4(v) = e+°'-4(v - 0-v"T) ,
when ce
x
errF(z, T) =
ze(r
J 4(v - 0-V T) dv - a[1 - (D(vo)l
0-o
= zer-[1 - 4 (vo - 0-V T)l - a[1 - F(vo)].
Finally, note that
//-0-o- UVT=
""
log(a/z) - (r + ;0-22)7
0-V
and that
1 - I(x) = I(-x)
and
log(a/z) = -log(z/a)
to get, after multiplying by e-r-, the end result
log(z/a) + (r + Zoz)T
F(z, r) = z4lf
l
/
log(z/a) + (r O'
(4.23)
Z0-')
520
VIII
Brownian Motion and Related Processes
Equation (4.23) is the Black-Scholes valuation formula. Four of the five
factors that go into it are easily and objectively evaluated: The current
market prize z, the striking price a, the time r until the option expires, and
the rate r of return from secure investments such as short-term government securities. It is the fifth factor, a., sometimes called the volatility, that
presents problems. It is, of course, possible to estimate this parameter
based on past records of price movements. However, it should be emphasized that it is the volatility in the future that will affect the profitability of
the option, and when economic conditions are changing, past history may
not accurately indicate the future. One way around this difficulty is to
work backwards and use the Black-Scholes formula to impute a volatility
from an existing market price of the option. For example, the
Hewlett-Packard call option that expires in August, six months or, T = 2
year, in the future, with a current price of Hewlett-Packard stock of $59,
a striking price of $60, and secure investments returning about
r = 0.05, a volatility of a. = 0.35 is consistent with the listed option price
Striking
Price
Time to
Expiration
(Years)
Offered
a
T
Price
Valuation
F(z, T)
1/12
$17.00
$17.45
2/12
19.25
13.50
15.13
18.87
13.09
14.92
8.50
12.00
5.50
9.13
9.26
130
130
135
135
140
140
145
145
145
150
150
155
155
155
1/12
2/12
1/12
2/12
1/12
2/12
5/12
Black-Scholes
1/12
13.63
3.13
2/12
6.38
1/12
1.63
2/12
5/12
4.00
9.75
11.46
6.14
8.52
13.51
3.80
6.14
2.18
4.28
9.05
Exercises
521
of $6. (When o = 0.35 is used in the Black-Scholes formula, the resulting valuation is $6.03.) A volatility derived in this manner is called an imputed or implied volatility. Someone who believes that the future will be
more variable might regard the option at $6 as a good buy. Someone who
believes the future to be less variable than the imputed volatility of
o, = 0.35 might be inclined to offer a Hewlett-Packard option at $6.
The table on the previous page compares actual offering prices on February 26, 1997, for options in IBM stock with their Black-Scholes valuation using (4.23). The current market price of IBM stock is $146.50, and
in all cases, the same volatility a = 0.30 was used.
The agreement between the actual option prices and their BlackScholes valuations seems quite good.
Exercises
4.1. A Brownian motion {X(t)} has parameters µ = -0.1 and a- = 2.
What is the probability that the process is above y = 9 at time t = 4, given
that it starts at x = 2.82?
4.2. A Brownian motion {X(t)} has parameters µ = 0.1 and a- = 2.
Evaluate the probability of exiting the interval (a, b] at the point b starting from X(0) = 0 for b = 1, 10, and 100 and a = -b. Why do the probabilities change when alb is the same in all cases?
4.3. A Brownian motion {X(t)} has parameters µ = 0.1 and a = 2.
Evaluate the mean time to exit the interval (a, b] from X(0) = 0 for b =
1, 10, and 100 and a = -b. Can you guess how this mean time varies with
b for b large?
4.4. A Brownian motion X(t) either (i) has drift p. = +16 > 0, or (ii) has
drift µ = -ZS < 0, and it is desired to determine which is the case by observing the process for a fixed duration T. If X(T) > 0, then the decision
will be that µ = +ZS; If X(T) :5 0, then µ = -ZS will be stated. What
should be the length T of the observation period if the design error probabilities are set at a = 8 = 0.05? Use S = 1 and a- = 2. Compare this fixed
duration with the average duration of the sequential decision plan in the
example of Section 4.1.
522
VIII
Brownian Motion and Related Processes
4.5.
Suppose that the fluctuations in the price of a share of stock in a
certain company are well described by a geometric Brownian motion with
drift a = -0.1 and variance
= 4. A speculator buys a share of this
stock at a price of $100 and will sell if ever the price rises to $110 (a
profit) or drops to $95 (a loss). What is the probability that the speculator
sells at a profit?
cr-2
4.6.
be a standard normal random variable.
Let
(a) For an arbitrary constant a, show that
E[(e - a)+] = 4(a) - a[1 - 4)(a)]
(b) Let X be normally distributed with mean µ and variance cr2. Show
that
E[(X - b)+] =
+(1) -
(b
0"
o
µ)L1
- (D(b
v
.1
Problems
4.1. What is the probability that a standard Brownian motion {B(t) }
ever crosses the line a + bt (a > 0, b > 0)?
4.2. Show that
Prmax
4.3.
b + B(t)
1+t
>a =
e-
,
a
>0 , b<
a.
If B°(s), 0 < s < 1, is a Brownian bridge process, then
B(t) = (1 + t)B°(
t )
1+t
is a standard Brownian motion. Use this representation and the result of
Problem 4.2 to show that for a Brownian bridge B°(t),
Pr{max B°(u) > a} = e-°'.
°:511s1
4.4. A Brownian motion X(t) either (i) has drift µ = µ0i or (ii) has drift
µ = µ,, where µ.° < µ, are known constants. It is desired to determine
which is the case by observing the process. Derive a sequential decision
Problems
523
procedure that meets prespecified error probabilities a and 0. Hint: Base
;(µo + µ,).
your decision on the process X'(t) = X(t) Change a Brownian motion with drift X(t) into an absorbed Brownian motion with drift X(t) by defining
4.5.
XA(t) =
(X(t),
for t < r,
0,
fort? T,
where
T = min{t ? 0; X(t) = 0).
(We suppose that X(0) = x > 0 and that A < 0, so that absorption is sure
to occur eventually.) What is the probability that the absorbed Brownian
motion ever reaches the height b > x?
4.6. What is the probability that a geometric Brownian motion with drift
parameter a = 0 ever rises to more than twice its initial value? (You buy
stock whose fluctuations are described by a geometric Brownian motion
with a = 0. What are your chances to double your money?)
4.7. A call option is said to be "in the money" if the market price of the
stock is higher than the striking price. Suppose that the stock follows a
geometric Brownian motion with drift a, variance a2 , and has a current
market price of z. What is the probability that the option is in the money
at the expiration time T? The striking price is a.
4.8. Verify the Hewlett-Packard option valuation of $6.03 stated in the
text when T = ;, z = $59, a = 60, r = 0.05 and a- = 0.35. What is the
Black-Scholes valuation if o- = 0.30?
4.9. Let r be the first time that a standard Brownian motion B(t) starting
from B(0) = x > 0 reaches zero. Let A be a positive constant. Show that
w(x) = E[e-ATIB(0) = x] = e-'-21*.
Hint: Develop an appropriate differential equation by instituting an
infinitesimal first step analysis according to
w(x) = E[E{e-ATjB(Ot)}!B(0) = x] = E[e-A°'w(x + AB)].
VIII
524
Brownian Motion and Related Processes
4.10. Let to = 0 < t, < t, < ... be time points, and define X = Z(t,,)
where Z(t) is geometric Brownian motion with drift parameters
r and variance parameter 022 (see the geometric Brownian motion in the
Black-Scholes formula (4.20)). Show that
is a martingale.
5.
The Ornstein-Uhlenbeck Process*
The Ornstein-Uhlenbeck process {V(t); t ? 0 } has two parameters, a drift
coefficient (3 > 0 and a diffusion parameter 0-22. The process, starting from
V(0) = v, is defined in terms of a standard Brownian motion {B(t) } by
scale changes in both space and time:
V(t) = ve-O' +
ae
` B(e2a' - 1),
fort ? 0.
(5.1)
The first term on the right of (5.1) describes an exponentially decreasing
trend towards the origin. The second term represents the fluctuations
about this trend in terms of a rescaled Brownian motion. The Ornstein-Uhlenbeck process is another example of a continuous-state-space,
continuous-time Markov process having continuous paths, inheriting
these properties from the Brownian motion in the representation (5.1). It
is a Gaussian process (see the discussion in Section 1.4), and (5.1) easily
shows its mean and variance to be
E[V(t)I V(0) = v] = ve-a',
(5.2)
and
Var[ V(t)V(0) = x] =
v2
2R Var[B(e2a' - 1)]
(5.3)
=
2( 1
ZR
Knowledge of the mean and variance of a normally distributed random
variable allows its cumulative distribution function to be written in terms
of the standard normal distribution (1.6), and by this means we can imThis section contains material of a more specialized nature.
S.
The Ornstein-Uhlenbeck Process
525
mediately express the transition distribution for the Ornstein-Uhlenbeck
process as
V -2,8(y
Pr(V(t)
- xe 0`) l
y1V(0) = x} = ID( aV-l - e_20,
I.
(5.4)
The Covariance Function
Suppose that 0 < u < s, and that V(0) = x. Upon subtracting the mean as
given by (5.2), we obtain
Cov[V(u), V(s)] = E[{V(u) - xe-Al) {V(s) - xe-g''}]
= 2Re-r,,+'''E[{B(e''-m" - 1)}{B(e'' - 1) }]
(5.5)
0,2
2
0,_
enc., a_e
216
5.1. A Second Approach to Physical Brownian Motion
The path that we have taken to introduce the Ornstein-Uhlenbeck process
is not faithful to the way in which the process came about. To begin an explanation, let us recognize that all models of physical phenomena have deficiencies, and the Brownian motion stochastic process as a model for the
Brownian motion of a particle is no exception. If B(t) is the position of a
pollen grain at time t and if this position is changing over time, then the
pollen grain must have a velocity. Velocity is the infinitesimal change in
position over infinitesimal time, and where B(t) is the position of the
pollen grain at time t, the velocity of the grain would be the derivative
dB(t)/dt. But while the paths of the Brownian motion stochastic process
are continuous, they are not differentiable. This remarkable statement is
difficult to comprehend. Indeed, many elementary calculus explanations
implicitly tend to assume that all continuous functions are differentiable,
and if we were to be asked to find an example of one that wasn't, we might
526
VIII
Brownian Motion and Related Processes
consider it quite a challenge. Yet each path of a continuous Brownian motion stochastic process is (with probability one) differentiable at no point.
We have encountered yet another intriguing facet of stochastic processes
that we cannot treat in full detail but must leave for future study. We will
attempt some motivation, however. Recall that the variance of the Brownian increment AB is At. But variations in the normal distribution are not
scaled in terms of the variance, but in terms of its square root, the standard
deviation, so that the Brownian increment AB is roughly on the order of
\, and the approximate derivative
AB
AB
At V
1
VA-t
is roughly on the order of 1/\. This, of course, becomes infinite as
At -, 0, which suggests that a derivative of Brownian motion, were it to
exist, could only take the values ±c. As a consequence, the Brownian
path cannot have a derivative. The reader can see from our attempt at explanation that the topic is well beyond the scope of an introductory text.
Although its movements may be erratic, a pollen grain, being a physi-
cal object of positive mass, must have a velocity, and the OrnsteinUhlenbeck process arose as an attempt to model this velocity directly.
Two factors are postulated to affect the particle's velocity over a small
time interval. First, the frictional resistance or viscosity of the surround-
ing medium is assumed to reduce the magnitude of the velocity by a
deterministic proportional amount, the constant of proportionality being
R > 0. Second, there are random changes in velocity caused by collisions
with neighboring molecules, the magnitude of these random changes
being measured by a variance coefficient o-'-. That is, if V(t) is the velocity at time t, and AV is the change in velocity over (t, t + At], we might
express the viscosity factor as
E[zVIV(t) = v] = -,6v1t + o(At)
(5.6)
and the random factor by
Var[i VI V(t) = v] = o-2At + o(Ot).
(5.7)
The Ornstein-Uhlenbeck process was developed by taking (5.6) and (5.7)
together with the Markov property as the postulates, and from them deriving the transition probabilities (5.4). While we have chosen not to follow this path, we will verify that the mean and variance given in (5.2) and
5.
The Ornstein-Uhlenbeck Process
527
(5.3) do satisfy (5.6) and (5.7) over small time increments. Beginning with
(5.2) and the Markov property, the first step is
E[V(t + ot)IV(t) = v] = ve-O-" = v[1 - 80t + o(Ot)],
and then
E[OVI V(t) = v] = E[V(t + At)I V(t) = v] - v
= -/3vAt + o(Ot),
and over small time intervals the mean change in velocity is the proportional decrease desired in (5.6). For the variance, we have
-
Var[OVIV(t) = v] = Var[V(t + ot)IV(t) = v]
0-2(1-e
2/3
= 0,2A t + o(Ot),
and the variance of the velocity increment behaves as desired in (5.7). In
fact, (5.6) and (5.7) together with the Markov property can be taken as the
definition of the Ornstein-Uhlenbeck process in much the same way, but
involving far deeper analysis, that the infinitesimal postulates of V, Section 2.1 serve to define the Poisson process.
Example Tracking Error Let V(t) be the measurement error of a radar
system that is attempting to track a randomly moving target. We assume
V(t) to be an Ornstein-Uhlenbeck process. The mean increment E[OVI V(t)
= v] = -/3vtt + o(At) represents the controller's effort to reduce the current error, while the variance term reflects the unpredictable motion of the
target. If /3 = 0.1, 0- = 2, and the system starts on target (v = 0), the probability that the error is less than one at time t = 1 is, using (5.4),
Pr(IV(t)j<1)= (
_
1- e
20(1
I- e-)
'-a
211,
I- e-0 2)) -
q)(
20(1
= D(0.53) - 1(-0.53) = 0.4038.
1
e 12))
528
VIII
Brownian Motion and Related Processes
As time passes, this near-target probability drops to I(1/N/,2-0-)
(D(-I/ N/-2-0) = F(0.22) - 4(-0.22) = 0.1742.
-
Example Genetic Fluctuations under Mutation In VI, Section 4, we
introduced a model describing fluctuations in gene frequency in a population of N individuals, each either of gene type a or gene type A. With X(t)
being the number of type a individuals at time t, we reasoned that X(t)
would be a birth and death process with parameters
A' -1
Y) + (1
Nl LN(1
N)YZ]
and
µj=WNLN'Y] +(I-N)(I-Y2),.
The parameters y, and y2 measured the rate of mutation from a-type to Atype, and A-type to a-type, respectively. Here we attempt a simplified description of the model when the population size N is large. The steadystate fluctuations in the relative gene frequency X(t)/N are centered on the
mean
Ir =
Y2
Y1 +y,
Accordingly, we define the rescaled and centered process
VN(t) = N/N_ (XN(t) - -rr).
With
AV= VN(t + At) - VN(t),
and AX = X(t + At) - X(t),
we have
E[OXX(t) = j] = (A; - µ)0t + o(Ot),
which becomes, after substitution and simplification,
E[.X(t) = j] = NA[(1
- j)
- N y,l At + o(Ot).
5.
The Ornstein-Uhlenbeck Process
529
More tedious calculations show that
NA2y,Y2
E[AX2IX(t) = j] =
L (y, + y,)2
+0
(!N)1At-
Our next step is to rescale these in terms of v, using
J
N=1r+c.
V
In the rescaled variables,
E[AVIVN(t) = v] =
I
EI AX
j
v
=a+V
N
X(t)
N
/
v
= AVNV I 1 - IT -
I yZ - (IT
K
+
v -) y, Ai + o(At)
= -A(y, + y2)VAt + o(Ot).
A similar substitution shows that
E[AV 2I VN(t) = v] =
2AY,Y2
(Y, +
y2)2
At + o(Ot).
Similar computations show that the higher moments of AV are negligible.
Since the relations (5.6) and (5.7) serve to characterize the OrnsteinUhlenbeck process, the evidence is compelling that the rescaled
gene processes (VN(t)) will converge in some appropriate sense to an
Ornstein-Uhlenbeck process V(t) with
16
=A(Y,+Y2) and o--_
2AY,Y2
(Y, + Y2)
and
X(t) ~ Nir + \/NV(t)
for large N.
This is indeed the case, but it represents another topic that we must leave
for future study.
530
5.2.
Brownian Motion and Related Processes
VIII
The Position Process
If V(t) is the velocity of the pollen grain, then its position at time t would be
S(t) = S(O) +
f V(u) du.
(5.8)
0
Because the Ornstein-Uhlenbeck process is continuous, the integral in
(5.8) is well-defined. The Ornstein-Uhlenbeck process is normally distributed, and so is each approximating sum to the integral in (5.8). It must
be, then, that the position process S(t) is normally distributed, and to describe it, we need only evaluate its mean and covariance functions. To
simplify the mathematics without losing any essentials, let us assume that
S(0) = V(0) = 0. Then
11
E[S(t)] = E Iff V(s) ds] =
f E[V(s)] ds = 0.
0
0
(The interchange of integral and expectation needs justification. Since the
expected value of a sum is always the sum of the expected values, the in-
terchange of expectation with Riemann approximating sums is clearly
valid. What is needed is justification in the limit as the approximating
sums converge to the integrals.)
Var[S(t)] = E[S(t)2] =
= EI j
rrr
z1
0
J
E[ f V(s) ds}
f V(u) du 11 f0 V(s) ds }J
J
0
Ir
=
f f E[V(s)V(u)] du ds
00
I
=2
S
f f E[V(s)V(u)] du ds
(5.9)
00
f
= v2
f
S
e-kk'+10) du ds
X00f
a12
p2
f e-A'(O'
It
_ zlt -
(1
(Using (5.5))
- 1- I + e-P) ds
11
- e F') + R(1 - e '-R')J.
5.
The Ornstein-Uhlenbeck Process
531
This variance behaves like that of a Brownian motion when t is large in
the sense that
Var[S(t)]
t
-
v2
as
t -
.
/3-
That is, observed over a long time span, the particle's position as n\odeled
by an Ornstein-Uhlenbeck velocity behaves much like a Brownian motion with variance parameter o-2//32. In this sense, the Ornstein-Uhlenbeck
model agrees with the Brownian motion model over long time spans and
improves upon it for short durations. Section 5.4 offers another approach.
Example - Stock Prices It is sometimes assumed that the market price
of a share of stock follows the position process under an OrnsteinUhlenbeck velocity. The model is consistent with the Brownian motion
model over long time spans. In the short term, the price changes are not
independent but have an exponentially decreasing correlation meant to
capture some notion of a market momentum. Suppose a call option is to
be exercised, if profitable at a striking price of a, at some fixed time t in
the future. If V(0) = 0 and S(0) = z is the current stock price, then the expected value of the option is
E[(S(t) - a)+] = T{O\a
µ)
T
- (a
T
µ)L1 -
(D(a
T
µ)]}'
(5.10)
where
and
0,'2
TZ =
9`
[t - ? 1 J6
e-O` +
213
Note that µ and r' are the mean and variance of S(t). The derivation is left
for Problem 5.4.
5.3.
The Long Run Behavior
It is easily seen from (5.2) and (5.3) that for large values of t, the mean of
the Ornstein-Uhlenbeck process converges to zero and the variance to
(72/2/3. This leads to a limiting distribution for the process in which
532
VIII
Brownian Motion and Related Processes
lim Pr{ V(t) < Y1 WO) = x} = (D (
yx
y
(5.11)
That is, the limiting distribution of the process is normal with mean zero
and variance 0.-22/(2/3). We now set forth a representation of a stationary
Ornstein-Uhlenbeck process, a process for which the limiting distribution
in (5.11) holds for all finite times as well as in the limit. The stationary
Ornstein-Uhlenbeck process {Vs(t); -oo < t < -} is represented in terms
of a Brownian motion by
Vs(t) = (e-I3'B(e221),
-00 < t < 00.
(5.12)
The stationary Ornstein-Uhlenbeck process is Gaussian (see Section 1.4)
and has mean zero. The covariance calculation is
r(s, t) = Cov[VS(s), VS(t)]
_ %3e sc.,+n Cov[B(e2,`), B(e2a`)]
(5.13)
_
2,6 e
s+" min{e'-R'', e2'}
0' 2
2f3
The stationary Ornstein-Uhlenbeck process is the unique Gaussian
process with mean zero and covariance (5.13).
The independence of the Brownian increments implies that the stationary Ornstein-Uhlenbeck process is a Markov process, and it is straightforward to verify that the transition probabilities are given by (5.4).
Example An Ehrenfest Urn Model in Continuous Time A single particle switches repeatedly between urn A and urn B. Suppose that the duration it spends in an urn before moving is an exponentially distributed
random variable with parameter /3, and that all durations are independent.
Let fi(t) = 1 if the particle is in urn A at time t, and 6(t) = -1 if in urn B.
Then 16t); t ? 01 is a two-state Markov process in continuous time for
which (See III, (3.14))
S.
The Ornstein-Uhlenbeck Process
533
Pr{at + s) = 116(t) = 1) = 2 + 2 e(5.14)
Let us further stipulate that the particle is equally likely to be in either urn
at time zero. It follows, then, that it is equally likely to be in either urn at
all times, and that therefore, E[f(t)] = 0 for all t. Using (5.14) and the
symmetry of the process, we may derive the covariance. We have
E[6t)e(t + s)] = 2
s) = 116(t) = 1)
+ 2 Pr 16(t + s) _ - l 16(t) _
- 2 Pr{e(t + s) = -
1)
1}
(5.15)
- 2 Pr{f(t + s) = l16(t) _ -1}
Now consider N of these particles, each alternating between the urns independently of the others, and let e;(t) track the position of the ith particle
at time t. The disparity between the numbers of particles in the two urns
is measured by
N
SN(t) = 7 ext)
i=1
If SN(t) = 0, then the urns contain equal numbers of particles. If SN(t) = k,
then there are (N + k)12 particles in urn A. The central limit principle for
random functions suggests that
VN(t) =
ITN SN(t)
should, for large N, behave similarly to a Gaussian process with mean
zero and covariance F(s, t) = exp( -2I3lt - sl). This limiting process is a
stationary Ornstein-Uhlenbeck process with .2 = 2(3. Thus we have derived the approximation
534
VIII
Brownian Motion and Related Processes
SN,(t) ~ \/NVs(t),
t > 0,
for the behavior of this continuous-time urn model when the number o
particles is large.
5.4.
Brownian Measure and Integration*
We state, in the form of a theorem without proof, an exceedingly useful
formula for computing certain functionals of Gaussian processes. This
theorem provides a tiny glimpse into a vast and rich area of stochastic
process theory, and we included it in this elementary text in a blatant attempt to entice the student towards further study.
Theorem 5.1
Let g(x) be a continuous function and let {B(t); t ? 0} be
a standard Brownian motion. For each fixed value oft > 0, there exists a
random variable
t
(g) = f g(x) dB(x)
(5.16)
0
that is the limit of the approximating sums
211
J.(g) =
I g ( 2" t) B( 2" t) -
k-1
B(k
L
2,
1
t)]
( 5 .17)
as n - oo. The random variable J(g) is normally distributed with mean
zero and variance
t
Var[J(g)] = Jg2(u) du.
(5.18)
0
Iff(x) is another continuous function of x, then J (f) and ,9(g) have a joint
normal distribution with covariance
t
E[,9(f).(g)] = Jf(x)g(x) dx.
(5.19)
0
* This subsection is both more advanced and more abstract than those that have preceeded it.
S.
The Ornstein-Uhlenbeck Process
535
The proof of the theorem is more tedious than difficult, but it does require
knowledge of facts that are not included among our prerequisites. The theorem asserts that a sequence of random variables, the Riemann approxi-
mations, converge to another random variable. The usual proof begins
with showing that the expected mean square of the difference between
distinct Riemann approximations satisfies the Cauchy criterion for convergence, and then goes on from there.
When g(x) is differentiable, then an integration by parts may be validated, which shows that
t
t
J g(x) dB(x) = g(t)B(t) -
J B(x)g'(x) dx,
0
0
(5.20)
and this approach may yield a concrete representation of the integral in
certain circumstances. For example, if g(x) = 1, then g'(x) = 0, and
t
Ji dB(x) = g(t)B(t) - 0 = B(t),
0
as one would hope. When g(x) = t - x, then g'(x) _ -1, and
t
t
J(t_x)dB(x)__JB(x)dx.
(5.21)
0
0
The process on the right side of (5.21) is called integrated Brownian motion. Theorem 5.1 then asserts that integrated Brownian motion is normally distributed with mean zero and variance
Var[ J B(x) dx] =
0
J (t - x)2 dx =
t3
.
0
The calculus of the Brownian integral of Theorem 5.1 offers a fresh and
convenient way to determine functionals of some Gaussian processes. For
example, in the case of the Ornstein-Uhlenbeck process, we have the integral representation
t
V(t) = ve-9' + o- J e-g"-"' dB(u).
(5.22)
The second term on the right of (5.22) expresses the random component
of the Ornstein-Uhlenbeck process as an exponentially weighted moving
VIII
536
Brownian Motion and Related Processes
average of infinitesimal Brownian increments. According to Theorem 5.1,
this random component has mean zero and is normally distributed. We use
Theorem 5.1 to determine the covariance. For 0 < s < t,
Cov[V(s), V(t)] = E({V(s) - ve-1) {V(t) - ve-a'}]
= 0-2EI f e-a(S-14) dB(u) f
0
dB(w)1
0
J
I
= o-2
f 1(u <
du
0
=
o'ze-a(s+t) f eza" du
0
0,
2
2
e(e)' _ 1)
2
in agreement with (5.5).
Example The Position Process Revisited
Let us assume that V(0) =
v = 0. The integral of the Ornstein-Uhlenbeck velocity process gives the
particle's position S(t) at time t. If we replace the integrand by its representation (5.22) (v = 0), we obtain
t
S(t) =
r
s
f V(s) ds = o- f f e-a(S-"' dB(u) ds
00
0
t
I
o- f
f e-a(s-") ds dB(u)
0u
t
o-
t
f ea" f e-R` ds dB(u)
o
It
t
o-
eau(e-a'
- e-a') dB(u)
0
I
_ 0' f (I - e-a('-')) dB(u).
go
(5.23)
S.
The Ornstein-Uhlenbeck Process
537
Theorem 5.1 applied to (5.23) tells us that the position S(t) at time t is normally distributed with mean zero and variance
Var[S(t)] = 02
f [I - e-jS(`-" )]2 du
0
-
t
_ 0' f (1
e-s"')2
dw
o
F3
'
2
911 t -
(1 - e
') +
2f3(1 _
e-,sr)]
in agreement with (5.9). Problem 5.4 calls for using Theorem 5.1 to determine the covariance between the velocity V(t) and position S(t).
The position process under an Ornstein-Uhlenbeck velocity behaves
like a Brownian motion over large time spans, and we can see this more
clearly from the Brownian integral representation in (5.23). If we carry
out the first term in the integral (5.23) and recognize that the second part
is V(t) itself, we see that
r
S(t) _ 0, f (1 - e-P('-'4) dB(u)
0
B(t) -
fe
air-u)
l
dB(u)Q[
]
(5.24)
0
=
1
[UB(t) - V(t)].
R
Let us introduce a rescaled position process that will allow us to better see
changes in position over large time spans. Accordingly, for N > 0, let
1
SN(t) = -S(Nt)
(5.25)
_ , [O-J§(t) + ,V(t)],
538
VIII
Brownian Motion and Related Processes
wherea(t) = B(Nt)/\/N remains a standard Brownian motion. (See Exercise 1.2.) Because the variance of V(t) is always less than or equal to
0.21(2/3), the variance of V(t)/HIV becomes neglible for large N. Equation
(5.25) then shows more clearly in what manner the position process becomes like a Brownian motion: For large N,
S\,(t)
o,
a
B(t).
Exercises
5.1. An Ornstein-Uhlenbeck process V(t) has 0-2 = 1 and /3 = 0.2. What
is the probability that V(t) <_ 1 for t = 1, 10, and 100? Assume that
V(0) = 0.
5.2. The velocity of a certain particle follows an Ornstein-Uhlenbeck
process with .2 = 1 and /3 = 0.2. The particle starts at rest (v = 0) from
position S(0) = 0. What is the probability that it is more than one unit
away from its origin at time t = 1. What is the probability at times t = 10
and t = 100?
5.3. Let ,, 62,
... be independent standard normal random variables
and /3 a constant, 0 < /3 < 1. A discrete analogue to the OrnsteinUhlenbeck process may be constructed by setting
for n?1.
Vo=v and
(a) Determine the mean value function and covariance function for
V,,. Determine the conditional mean and variance
(b) Let AV =
of AV, given that V = v.
Problems
5.1.
Let e,, e,,
...
be independent standard normal random variables
and /3 a constant, 0 < /3 < 1. A discrete analogue to the OrnsteinUhlenbeck process may be constructed by setting
V. = v
and V = (1 -
6,,
for
n ? 1.
Problems
539
(a) Show that
Vr = (1 - N)"v +
, (1 k=1
3)n _k6k-
N/
Comment on the comparison with (5.22).
S = v + V, +
(b) Let AV,, = V 6, +
+ , Show that
+ V,,, and B,,
B
Compare and contrast with (5.24).
5.2. Let _S(t) be the position process corresponding to an OrnsteinUhlenbeck velocity V(t). Assume that S(0) = V(0) = 0. Obtain the covariance between S(t) and V(t).
5.3.
Verify the option valuation formulation (5.10).
Hint:
Use the result of Exercise 4.6.
5.4. In the Ehrenfest urn model (see III, Section 3.2) for molecular diffusion through a membrane, if there are i particles in urn A, the probability that there will be i + 1 after one time unit is 1 - i/(2N), and the probability of i - 1 is i/(2N), where 2N is the aggregate number of particles in
both urns. Following III, Section 3.2, let Y be the number of particles in
urn A after the nth transition, and let X,, = Y - N. Let OX =
X,, be
the change in urn composition. The probability law is
We anticipate a limiting process in which the time between transitions becomes small and the number of particles becomes large. Accordingly, let
At = 1/N and measure fluctuations of a rescaled process in units of order
1/N/"N-. The definition of the rescaled process is
VN(t) =
XjNj
Note that in the duration t = 0 to t = 1 in the rescaled process, there
are N transitions in the urns, and a unit change in the rescaled process
540
VIII
Brownian Motion and Related Processes
corresponds to a fluctuation of order 1l'"N- in the urn composition. Let
AV = VN(t + 1/N) - VN(t) be the displacement in the rescaled process
over the time interval At = 1/N. Show that
E[OV IVN(t) = v] _
\2
-v(N) =
2V /
V "N(2
+ 2\/N)
VAt,
and that (AV)2 = 1/N, whence
Var[AVIVN(t) = v] = E[(OV)Z] - {E[OV])2
= N + o(N) = At + o(Ot).
Chapter IX
Queueing Systems
1.
Queueing Processes
A queueing system consists of "customers" arriving at random times to
some facility where they receive service of some kind and then depart. We
use "customer" as a generic term. It may refer, for example, to bona fide
customers demanding service at a counter, to ships entering a port, to
batches of data flowing into a computer subsystem, to broken machines
awaiting repair, and so on. Queueing systems are classified according to
1. The input process, the probability distribution of the pattern of arrivals of customers in time;
2. The service distribution, the probability distribution of the random
time to serve a customer (or group of customers in the case of batch
service); and
3. The queue discipline, the number of servers and the order of customer service.
While a variety of input processes may arise in practice, two simple and
frequently occurring types are mathematically tractable and give insights
into more complex cases. First is the scheduled input, where customers arrive at fixed times T, 2T, 3T, .... The second most common model is the
"completely random" arrival process, where the times of customer arrivals form a Poisson process. Understanding the axiomatic development
of the Poisson process in V may help one to evaluate the validity of the
542
IX
Queueing Systems
Poisson assumption in any given application. Many theoretical results are
available when the times of customer arrivals form a renewal process. Ex-
ponentially distributed interarrival times then correspond to a Poisson
process of arrivals as a special case.
We will always assume that the durations of service for individual customers are independent and identically distributed nonnegative random
variables and are independent of the arrival process. The situation in which
all service times are the same fixed duration D, is, then, a special case.
The most common queue discipline is first come, first served, where
customers are served in the same order in which they arrive. All of the
models that we consider in this chapter are of this type.
Queueing models aid the design process by predicting system performance. For example, a queueing model might be used to evaluate the
costs and benefits of adding a server to an existing system. The models enable us to calculate system performance measures in terms of more basic
quantities. Some important measures of system behavior are
1. The probability distribution of the number of customers in the system. Not only do customers in the system often incur costs, but in
many systems, physical space for waiting customers must be
planned for and provided. Large numbers of waiting customers can
also adversely affect the input process by turning potential new customers away. (See Section 4.1 on queueing with balking.)
2. The utilization of the server(s). Idle servers may incur costs without
contributing to system performance.
3. System throughput. The long run number of customers passing
through the system is a direct measure of system performance.
4. Customer waiting time. Long waits for service are annoying in the
simplest queueing situations and directly associated with major costs
in many large systems such as those describing ships waiting to
unload at a port facility or patients awaiting emergency care at a
hospital.
1.1.
The Queueing Formula L = A W
Consider a queueing system that has been operating sufficiently long to
have reached an appropriate steady state, or a position of statistical equilibrium. Let
1.
Queueing Processes
543
L = the average number of customers in the system;
A = the rate of arrival of customers to the system; and
W = the average time spent by a customer in the system.
The equation L = AW is valid under great generality for such systems and
is of basic importance in the theory of queues, since it directly relates two
of our most important measures of system performance, the mean queue
size and the mean customer waiting time in the steady state, that is, mean
queue size and mean customer waiting time evaluated with respect to a
limiting or stationary distribution for the process.
The validity of L = A W does not rest on the details of any particular
model, but depends only upon long run mass flow balance relations. To
sketch this reasoning, consider a time T sufficiently long so that statistical
fluctuations have averaged out. Then the total number of customers to
have entered the system is AT, the total number to have departed is A(T W), and the net number remaining in the system L must be the difference
L= AT - [A(T- W)] = AW.
Figure 1.1 depicts the relation L = AW.
Of course, what we have done is by no means a proof, and indeed, we
shall give no proof. We shall, however, provide several sample verifications of L = A W where L is the mean of the stationary distribution of
time
(a) Random arrivals, departures
Figure 1.1
(b) Smoothed values
The cumulative number of arrivals and departures in a queueing
system. The smoothed values in (b) are meant to symbolize long
run averages. The rate of arrivals per unit time is A, the mean
number in the system is L, and the mean time a customer spends
in the system is W.
544
IX
Queueing Systems
customers in the system, W is the mean customer time in the system determined from the stationary distribution, and A is the arrival rate in a
Poisson arrival process.
Let Lo be the average number of customers waiting in the system who
are not yet being served, and let Wo be the average waiting time in the system excluding service time. In parallel to L = AW, we have the formula
Lo = AW0
(1.1)
The total waiting time in the system is the sum of the waiting time before
service and the service time. In terms of means, we have
W = Wo + mean service time.
1.2.
(1.2)
A Sampling of Queueing Models
In the remainder of this chapter we will study a variety of queueing systems. A standard shorthand is used in much of the queueing literature for
identifying simple queueing models. The shorthand assumes that the arrival times form a renewal process, and the format A/B/c uses A to describe the interarrival distribution, B to specify the individual customer
service time distribution, and c to indicate the number of servers. The
common cases for the first two positions are G = GI for a general or arbitrary distribution, M (memoryless) for the exponential distribution, Ek
(Erlang) for the gamma distribution of order k, and D for a deterministic
distribution, a schedule of arrivals or fixed service times.
Some examples discussed in the sequel are the following:
The M/M/1 queue Arrivals follow a Poisson process; service times
are exponentially distributed; and there is a single server. The number
X(t) of customers in the system at time t forms a birth and death
process. (See Section 2.)
The M/M/oo queue There are Poisson arrivals and exponentially distributed service times. Any number of customers are processed simultaneously and independently. Often self-service situations may be de-
scribed by this model. In the older literature this was called the
"telephone trunking problem."
1.
Queueing Processes
545
The MIG11 queue In this model there are Poisson arrivals but arbitrarily distributed service times. The analysis proceeds with the help of
an embedded Markov chain.
More elaborate variations will also be set forth. Balking is the refusal
of new customers to enter the system if the waiting line is too long. More
generally, in a queueing system with balking, an arriving customer enters
the system with a probability that depends on the size of the queue. Here
it is important to distinguish between the arrival process and the input
process, as shown in Figure 1.2. A special case is a queue with overflow,
in which an arriving customer enters the queue if and only if there is at
least one server free to begin service immediately.
In a priority queue, customers are allowed to be of different types. Both
the service discipline and the service time distribution may vary with the
customer type.
A queueing network is a collection of service facilities where the departures from some stations form the arrivals of others. The network is
closed if the total number of customers is fixed, these customers continuously circulating through the system. The machine repair model (see the
example entitled "Repairman Models" in VI, Section 4) is an example of
a closed queueing network. In an open queueing network, customers may
arrive from, and depart to, places outside the network, as well as move
from station to station. Queueing network models have found much recent
application in the design of complex information processing systems.
Departs
system
with
probability
Waiting
line
1 -Pn
X
0
Arrival
process
Enters system
with probability
pn
Input
process
XXXX
n customers
waiting
Service
facility
Figure 1.2 If n customers are waiting in a queueing system with balking, an
arriving customer enters the system with probability pn and does
not enter with probability 1 - p,.
546
IX
Queueing Systems
Exercises
1.1. What design questions might be answered by modeling the following queueing systems?
The Customer
(a) Arriving airplanes
(b) Cars
(c) Broken TVs
(d) Patients
(e) Fires
The Server
The runway
A parking lot
Repairman
Doctor
Fire engine company
What might be reasonable assumptions concerning the arrival process,
service distribution, and priority in these instances?
1.2. Consider a system, such as a barber shop, where the service required is essentially identical for each customer. Then actual service times
would tend to cluster near the mean service time. Argue that the exponential distribution would not be appropriate in this case. For what types
of service situations might the exponential distribution be quite plausible?
1.3. Oil tankers arrive at an offloading facility according to a Poisson
process whose rate is A = 2 ships per day. Daily records show that there
is an average of 3 ships unloading or waiting to unload at any instant in
time. On average, what is the duration of time that a ship spends in port?
Assume that a ship departs immediately after unloading.
Problems
1.1. Two dump trucks cycle between a gravel loader and a gravel unloader. Suppose that the travel times are insignificant relative to the load
and unload times, which are exponentially distributed with parameters µ
and A, respectively. Model the system as a closed queueing network. Determine the long run gravel loads moved per unit time.
Hint:
4ofVI.
Refer to the example entitled "Repairman Models" in Section
Poisson Arrivals, Exponential Service Times
2.
547
Poisson Arrivals, Exponential Service Times
2.
The simplest and most extensively studied queueing models are those
having a Poisson arrival process and exponentially distributed service
times. In this case the queue size forms a birth and death process (see Sections 3 and 4 of VI), and the corresponding stationary distribution is readily found.
We let A denote the intensity, or rate, of the Poisson arrival process and
assume that the service time distribution is exponential with parameter A.
The corresponding density function is
g(x) = Ae--
for x > 0.
(2.1)
For the Poisson arrival process we have
Pr{An arrival in [t, t + h)} = Ah + o(h)
(2.2)
Pr{No arrivals in [t, t + h) } = 1 - Ah + o(h).
(2.3)
and
Similarly, the memoryless property of the exponential distribution as expressed by its constant hazard rate (see I, Section 4.2) implies that
Pr(A service is completed in [t, t + h)IService in progress at time t}
= Ah + o(h),
(2.4)
and
Pr{Service not completed in [t, t + h)IService in progress at time t}
= 1 - µh + o(h).
(2.5)
The service rate p applies to a particular server. If k servers are simultaneously operating, the probability that one of them completes service in a
time interval of duration h is (kµ)h + o(h), so that the system service rate
is kµ. The principle used here is the same as that used in deriving the infinitesimal parameters of the Yule process (VI, Section 1).
We let X(t) denote the number of customers in the system at time t,
counting the customers undergoing service as well as those awaiting service. The independence of arrivals in disjoint time intervals together with
the memoryless property of the exponential service time distribution
548
IX
Queueing Systems
implies that X(t) is a time homogeneous Markov chain, in particular, a
birth and death process. (See Sections 3 and 4 of VI.)
2.1. The MIM/1 System
We consider first the case of a single server and let X(t) denote the number of customers in the system at time t. An increase in X(t) by one unit
corresponds to a customer arrival, and in view of (2.2) and (2.5) and the
postulated independence of service times and the arrival process, we have
Pr{X(t + h) = k + 1IX(t) = k) = [Ah + o(h)] X [1 - µh + o(h)]
for k = 0, 1,
= Ah + o(h)
... .
Similarly, a decrease in X(t) by one unit corresponds to a completion of
service, whence
fork= 1,2.....
Pr{X(t+h)=k- lIX(t)=k} =µh+o(h)
Then X(t) is a birth and death process with birth parameters
Ak - A
fork=0,1,2,...
Ak=1h
fork= 1,2.....
and death parameters
Of course, no completion of service is possible when the queue is empty.
We thus specify µ0 = 0.
Let
I r k = lim Pr{X(t) = k}
for k = 0, 1, .. .
be the limiting, or equilibrium, distribution of queue length. Section 4 of
VI describes a straightforward procedure for determining the limiting distribution lTk from the birth and death parameters Ak and µk. The technique
is to first obtain intermediate quantities 8; defined by
00 = 1
and
0; =
AOA,
A,_,
for j ? 1,
(2.6)
Poisson Arrivals, Exponential Service Times
2.
549
and then
1
?r0 =
and
el
irk = 0kir0 =
Y"'=
o
fork 1, 1.
o el
(2.7)
When 1 o 0, _ 00, then lim,.x Pr{X(t) = k) = 0 for all k, and the queue
length grows unboundedly in time.
For the M/M/1 queue at hand we readily compute 00 = 1 and 8; = (A/µy
for j = 1, 2, .... Then
x
1
j=o
J=O
(I
A)
ifA<
A//J,)
ifA?µ.
= 00
Thus, no equilibrium distribution exists when the arrival rate A is equal to
or greater than the service rate A. In this case the queue length grows without bound.
When A < µ, a bona fide limiting distribution exists, given by
7r0 =f0 =I --A
1
(2.8)
Ii-
and
7Fk= 1'00k=1 -
A )(- A
IL
µ
fork=0, 1,
)
(2.9)
The equilibrium distribution (2.9) gives us the answer to many questions
involving the limiting behavior of the system. We recognize the form of
(2.9) as that of a geometric distribution, and then reference to I, Section
3.3 gives us the mean queue length in equilibrium to be
L=
A
(2.10)
µ - A
The ratio p = A/µ is called the traffic intensity,
P=
arrival rate
=
system service rate
A
µ
(2.11)
550
As the
IX
Queueing Systems
intensity approaches one, the mean queue length
traffic
L = p/(1 - p) becomes infinite. Again using (2.8), the probability of being
served immediately upon arrival is
Iro = 1 - -,
µ
the probability, in the long run, of finding the server idle. The server
utilization, or long run fraction of time that the server is busy, is
1 - iro = A/µ.
We can also calculate the distribution of waiting time in the stationary
case when k < µ. If an arriving customer finds n people in front of him,
his total waiting time T, including his own service time, is the sum of the
service times of himself and those ahead, all distributed exponentially
with parameter µ, and since the service times are independent of the
queue size, T has a gamma distribution of order n + 1 with scale parameter µ,
+iT,ne-kT
r
Pr{T
ahead} = f µI'(n + 1)
dT.
(2.12)
By the law of total probability, we have
t1n ahead) X (A (1 -
Pr(T
Pr{T :5 t}
n=o
since (A/µ)"(1 - A/µ) is the probability that in the stationary case a customer on arrival will find n ahead in line. Now, substituting from (2.12),
we obtain
x
r
Pr{T s t} =
µn+1 Tne -µT
r(n + 1)
r
=
µe-µT(1
0
= J (1
-
A)dT
A1µ/
(p.)(
),L
n
r(n + 1)
dT
- µ)µ expj-Tµ(1 - µ)} dT = 1 - exp[-t(µ - A)],
A
which is also an exponential distribution.
A
2.
Poisson Arrivals
cponential Service Times
551
The mean of this exponential waiting time distribution is the reciprocal
of the exponential parameter, or
1
W= µ - A
(2.13)
Reference to (2.10) and (2.13) verifies the fundamental queueing formula
L = A.W.
A queueing system alternates between durations when the servers are
busy and durations when the system is empty and the servers are idle. An
idle period begins the instant the last customer leaves, and endures until
the arrival of the next customer. When the arrival process is Poisson of
rate A, then an idle period is exponentially distributed with mean
A busy period is an uninterrupted duration in which the system is not
empty. When arrivals to a queue follow a Poisson process, then the successive durations Xk from the commencement of the kth busy period to the
start of the next busy period form a renewal process (see Figure 2.1.) Each
Xk is composed of a busy portion Bk and an idle portion Ik. Then the renewal theorem (see "A Queueing Model" in VII, Section 5.3) applies to
X(t)
cB,< /i>
IE
Figure 2.1
-XI
0
H
<
B,
0F 1,
---
r
X,
The busy periods Bk and idle periods 1k of a queueing system.
When arrivals form a Poisson process, then Xk = Bk + Ik,
k = 1, 2, ... , are independent identically distributed nonnegative
random variables, and thus form a renewal process.
552
IX
Queueing Systems
tell us that po(t), the probability that the system is empty at time t, converges to
E[II l
lim po(t) = lro =
E[I,] + E[B,]
We substitute the known quantities ir,, = 1 - Alµ and E[I,] = 1/A to
obtain
1/A + E[B,]'
1
which gives
1
E[B,] _
A
for the mean length of a busy period.
In Section 3, in studying the M/G/1 system we will reverse this reasoning, calculate the mean busy period directly, and then use renewal theory
to determine the server idle fraction Tro.
2.2 The M/M/cc System
When an unlimited number of servers are always available, then all customers in the system at any instant are simultaneously being served. The
departure rate of a single customer being µ, the departure rate of k customers is kµ, and we obtain the birth and death parameters
Ak = A
and µk = kµ
for k = 0, 1.
... .
The auxiliary quantities of (2.6) are
AOA, ... Ak-I
ek=
µiµ2
µk
1
Ak
=k(µ)
fork=0,1,...,
which sum to
= e"''`
2.
Poisson Arrivals, Exponential Service Times
553
whence
1
aT0 =
Ik0ek=C
-N µ
and
(Aµ)ke-uµ
ark= Okaro=
fork= 0, 1,...,
k!
(2.14)
a Poisson distribution with mean queue length
Since a customer in this system begins service immediately upon arrival, customer waiting time consists only of the exponentially distributed
service time, and the mean waiting time is W = 1/p.. Again, the basic
queueing formula L = A W is verified.
The M/G/cG queue will be developed extensively in the next section.
2.3.
The MMIs System
When a fixed number s of servers are available and the assumption is
made that a server is never idle if customers are waiting, then the appropriate birth and death parameters are
Ak=A
fork= 1,2,...,
Jkµ
fork=0,
µk - sµ
fork > s.
If X(t) is the number of customers in the system at time t, then the number undergoing service is min(X(t), s), and the number waiting for service
is max{X(t) - s, 0}. The system is depicted in Figure 2.2.
554
IX
Queueing Systems
s = 5 parallel servers
X
A common
waiting line
XXX
Arrivals
Departures
X
X
X
X 7
X
/
Figure 2.2 A queueing system with s servers.
The auxiliary quantities are given by
fork=0,1,...,s,
ek-AoAI...Ak_I
tL
S!
Sjtk)
for k ? s,
and when A < sµ, then
j=0
z
j=0 j!(-l
A
+j=S
z S!,
(
/-t )'( SA
(2.15)
'
r=c
1 (A 1' +
j! N J
(Alto
s!(1 - A/s N-)
for A < sµ
The traffic intensity in an MIMIs system is p = Al(sli). Again, as the
traffic intensity approaches one, the mean queue length becomes unbounded. When A < sµ, then from (2.8) and (2.15),
=
,To
(A )i
Ij'=()
j! A
+ s!(1 - Alsµ)1
Exercises
555
and
1
Ak
fork=0,1,...,s,
ki(µ)ITO
(2.16)
s! \ul s (su)
fork? s,
IT0
k
We evaluate L0, the mean number of customers in the system waiting for,
and not undergoing, service. Then
Lo=7,
(J-s)lrj=7' kir,
k
k=0
j=s
k=0
YJAIAS
k(
= so(µ).,
kµlk
(2.17)
k
7,
7ro
A ls
(A/sµ)
= s! (µl (1 - Alsp)2'
Then
W = W0 +
and
L=AW=A(W0+ 1 I=LO+ A
µ/
µ
Exercises
2.1. Customers arrive at a tool crib according to a Poisson process of
rate A = 5 per hour. There is a single tool crib employee, and the individual service times are exponentially distributed with a mean service time of
556
IX
Queueing Systems
10 minutes. In the long run, what is the probability that two or more workers are at the tool crib being served or waiting to be served?
2.2. On a single graph, plot the server utilization 1 - Tro = p and the
mean queue length L = p/(1 - p) for the M/M/l queue as a function of
the traffic intensity p = A/µ for 0 < p < 1.
2.3. Customers arrive at a checkout station in a market according to a
Poisson process of rate A = 1 customer per minute. The checkout station
can be operated with or without a bagger. The checkout times for customers are exponentially distributed, and with a bagger the mean checkout time is 30 seconds, while without a bagger this mean time increases to
50 seconds. Compare the mean queue lengths with and without a bagger.
Problems
2.1.
Determine explicit expressions for 7r0 and L for the MIMIs queue
when s = 2. Plot I - iro and L as a function of the traffic intensity
p = A/2µ.
Determine the mean waiting time W for an M/M12 system when
A = 2 and µ = 1.2. Compare this with the mean waiting time in an M/M11
system whose arrival rate is A = 1 and service rate is µ = 1.2. Why is there
a difference when the arrival rate per server is the same in both cases?
2.2.
2.3.
Determine the stationary distribution for an MIM12 system as a
function of the traffic intensity p = A/2µ, and verify that L = AW.
2.4. The problem is to model a queueing system having finite capacity.
We assume arrivals according to a Poisson process of rate A, independent
exponentially distributed service times having mean 1/µ, a single server,
and a finite system capacity N. By this we mean that if an arriving customer finds that there are already N customers in the system, then that customer does not enter the system and is lost.
Let X(t) be the number of customers in the system at time t. Suppose
that N = 3 (2 waiting, 1 being served).
(a) Specify the birth and death parameters for X(t).
(b) In the long run, what fraction of time is the system idle?
(c) In the long run, what fraction of customers are lost?
Problems
557
2.5. Customers arrive at a service facility according to a Poisson
process having rate A. There is a single server, whose service times are exponentially distributed with parameter µ. Let N(t) be the number of peo-
ple in the system at time t. Then N(t) is a birth and death process with
parameters A = A for n ? 0 and µ = µ for n ? 1. Assume A < A. Then
lrk = (1 - A/p)(Alµ)k, k ? 0, is a stationary distribution for N(t); cf. equation (2.9).
Suppose the process begins according to the stationary distribution.
That is, suppose Pr{N(0) = k} _ ffk for k = 0, 1, .... Let D(t) be the
number of people completing service up to time t. Show that D(t) has a
Poisson distribution with mean At.
Hint: Let Pk;(t) = Pr{D(t) = jIN(0) = k} and P(t) _ I 1rkPk;(t) =
Pr{D(t) = j}. Use a first step analysis to show that P0;(t + At) =
A(Ot)P,;(t) + [1 - A(Ot)]Po;(t) + o(Ot), and for k = 1, 2,
... ,
Pk;(t + At) = µ(Ot)Pk ,.;-I(t) +
[1 - (A + µ)(Ot)]PkJ(t) + 0(t).
Then use P;(t) _ Ik irrPk;(t) to establish a differential equation. Use the explicit form of Irk given in the problem.
2.6.
Customers arrive at a service facility according to a Poisson
process of rate A. There is a single server, whose service times are exponentially distributed with parameter µ. Suppose that "gridlock" occurs
whenever the total number of customers in the system exceeds a capacity
C. What is the smallest capacity C that will keep the probability of gridlock, under the limiting distributing of queue length, below 0.001? Express your answer in terms of the traffic intensity p = A/µ.
2.7. Let X(t) be the number of customers in an M/MIcc queueing system
at time t. Suppose that X(0) = 0.
(a) Derive the forward equations that are appropriate for this process
by substituting the birth and death parameters into VI, (3.8).
(b) Show that M(t) = E[X(t)] satisfies the differential equation
M'(t) = A - AM(t) by multiplying the jth forward equation by j
and summing.
(c) Solve for M(t).
558
3.
IX
Queueing Systems
General Service Time Distributions
We continue to assume that the arrivals follow a Poisson process of rate
A. The successive customer service times Y,, Y2, ... , however, are now allowed to follow an arbitrary distribution G(y) = Pr{ Yk y } having a finite mean service time v = E[1 ]. The long run service rate is µ = 1/v.
Deterministic service times of an equal fixed duration are an important
special case.
3.1.
The MIGI1 System
If arrivals to a queue follow a Poisson process, then the successive durations Xk from the commencement of the kth busy period to the start of the
next busy period form a renewal process. (A busy period is an uninterrupted duration when the queue is not empty. See Figure 2.1.) Each Xk is
composed of a busy portion Bk and an idle portion Ik. Then po(t), the probability that the system is empty at time t, converges to
lim po(t) = pro =
E[I l]
E[XI]
(3.1)
E[11]
E[111 + E[B, ]
by the renewal theorem (see "A Queueing Model" in VII, Section 5.3).
The idle time is the duration from the completion of a service that empties the queue to the instant of the next arrival. Because of the memoryless property that characterizes the interarrival times in a Poisson process,
each idle time is exponentially distributed with mean E[l,] = 1/A.
The busy period is composed of the first service time Y,, plus busy periods generated by all customers who arrive during this first service time.
Let A denote this random number of new arrivals. We will evaluate the
conditional mean busy period given that A = n and Y, = y. First,
E[BJA = 0, Y, = y] = y,
because when no customers arrive, the busy period is composed of the
3.
General Service Time Distributions-
559
first customer's service time alone. Next, consider the case in which
A = 1, and let B' be the duration from the beginning of this customer's
service to the next instant that the queue is empty. Then
E[B, I A = 1, Y, = y] = y + E[B']
= y + E[B, ],
because upon the completion of service for the initial customer, the single
arrival begins a busy period B' that is statistically identical to the first, so
that E[B'] = E[B,]. Continuing in this manner we deduce that
E[B,I A = n, Y, = y] = y + nE[B,]
and then, using the law of total probability, that
x
E[B,IY, = y] = 7, E[B,IA = n, Y, = y] Pr{A = nIY, = y)
n=0
v + nE[B 11
(Ay)'e'
v
n.
n=0
= y+ 1lyE[B,].
Finally,
E[BI] = J E[B,I Y, = y] dG(y)
0
X
= J {y + AyE[B,] } dG(y)
(3.2)
0
= v{ 1 + AE[B,] }.
Since E[B,] appears on both sides of (3.2), we may solve to obtain
E[B,] =
v
1
_ Av,
provided that Av < 1.
To compute the long run fraction of idle time, we use (3.3) and
(3.3)
560
IX
Queueing Systems
E[I1l
E[lll + E[B,l
1/A
(3.4)
1/A + v/(1 - AV)
=1-Av
ifAv<1.
Note that (3.4) agrees, as it must, with the corresponding expression
(2.8) obtained for the M/M11 queue where v = 1/µ. For example, if
arrivals occur at the rate of A = 2 per hour and the mean service time is
20 minutes, or v = ; hours, then in the long run the server is idle
1 - 2(3) _ ; of the time.
The Embedded Markov Chain
The number X(t) of customers in the system at time t is not a Markov
process for a general M/GI1 system, because if one is to predict the future
behavior of the system, one must know, in addition, the time expended in
service for the customer currently in service. (It is the memoryless property of the exponential service time distribution that makes this additional
information unnecessary in the M/M/1 case.)
Let X,,, however, denote the number of customers in the system imme-
diately after the departure of the nth customer. Then {X } is a Markov
chain. Indeed, we can write
1+A
X"=
if X,,-, > 0,
if
All
0,
(3.5)
_(X,,-,-1)++ A,
where A,, is the number of customers that arrive during the service of the
nth customer and where x+ = max( x, 0). Since the arrival process is Poisson, the number of customers A,, that arrive during the service of the nth
customer is independent of earlier arrivals, and the Markov property follows instantly. We calculate
f
Pr{A,,
0
(3.6)
_
(
J
0
(Ay)ke-z,.
k!
dG(y),
3.
General Service Time Distributions
561
and then, for j = 0, 1, ... ,
=i}
P;=Pr{X,,
1)+}
a;_;+,
for i ? 1, j ? i + 1,
a;
for i = 0.
(3.7)
The Mean Queue Length in Equilibrium L
The embedded Markov chain is of special interest in the M/G/1 queue be-
cause in this particular instance, the stationary distribution fir) for the
Markov chain {X } equals the limiting distribution for the queue length
process {X(t)}. That is, lim, Pr{X(t) = j) =
Pr{X = j}. We will
use this helpful fact to evaluate the mean queue length L.
The equivalence between the stationary distribution for the Markov
and that for the non-Markov process {X(t) } is rather subtle. It
chain
is not the consequence of a general principle and should not be assumed
to hold in other circumstances without careful justification. The equivalence in the case at hand is sketched in an appendix to this section.
We will calculate the expected queue length in equilibrium
L = lim, E[X(t)] by calculating the corresponding quantity in the embedded Markov chain, L = lim, ,,. E[X ]. If X = X,, is the number of customers in the system after a customer departs and X' is the number after
the next departure, then in accordance with (3.5),
X'=X-S+N,
(3.8)
where N is the number of arrivals during the service period and
I1
ifX> 0,
0
ifX=0.
In equilibrium, X has the same distribution as does X', and in particular,
L = E[X] = E[X'],
(3.9)
and taking expectation in (3.8) gives
E[X'] = E[X] - E[S] + E[N],
and, by (3.9) and (3.4), then
E[N] = E[S] = 1 - iro = Av.
(3.10)
562
IX
Queueing Systems
Squaring (3.8) gives
(X')22=X22+52 +N2-26X+2N(X- 6),
and since 6 = 6 and X6 = X, then
(X')2 = X2 + S + N2 - 2X + 2N(X - 6).
(3.11)
Now N, the number of customers that arrive during a service period, is independent of X, and hence of 6, so that
E[N(X - 6)] = E[N]E[X - 6],
(3.12)
and because X and X' have the same distribution, then
E[(X')2] = E[X2].
(3.13)
Taking expectations in (3.11) we deduce that
E[(X')2] = E[X2] + E[6] + E[N22] - 2E[X] + 2E[N]E[X - 6],
and then substituting from (3.10) and (3.13), we obtain
0 = Av + E[N2] - 2L + 2Av(L - Av),
or
AP + E[N22] - 2(A V)2
L
(3.14)
2(1-Av)
It remains to evaluate E[N22], where N is the number of arrivals during
a service time Y. Conditioned on Y = y, the random variable N has a Pois-
son distribution with a mean (and variance) equal to Ay [see (3.6)],
whence E[N2I Y = y] = Ay +
gives
E[N2] =
(Ay)2. Using the law of total probability then
[E[NIY = y] dG(y)
0
X
= A J y dG(y) + A2
0
= AY + A2 (.T2 + u2),
f y2 dG(y)
0
(3.15)
3.
General Service Time Distributions
563
where TZ is the variance of the service time distribution G(y). Substituting
(3.15) into (3.14) gives
L=
2Av + A2T2 - (AV)2
2(1 - Av)
A2T2 + p2
V
(3.16)
2(1-p)'
where p = Av is the traffic intensity.
Finally, W = L/lt, which simplifies to
W=v+
A(T2 + V2)
2(1 - p)
(3.17)
The results (3.16) and (3.17) express somewhat surprising facts. They
say that for a given average arrival rate A and mean service time v, we can
decrease the expected queue size L and waiting time W by decreasing the
variance of service time. Clearly, the best possible case in this respect corresponds to constant service times, for which T2 = 0.
Appendix
We sketch a proof of the equivalence between the limiting queue size distribution and the limiting distribution for the embedded Markov chain in
an M/G/1 model. First, beginning at t = 0 let -q denote those instants when
the queue size X(t) increases by one (an arrival), and let denote those instants when X(t) decreases by one (a departure). Let Y = X(rl -) denote
the queue length immediately prior to an arrival and let X,, = X(6, +) denote the queue length immediately after a departure. For any queue length
i and any time t, the number of visits of Y,, to i up to time t differs from
the number of visits of X,, to i by at most one unit. Therefore, in the long
run the average visits per unit time of Y,, to i must equal the average visits of X,, to i, which is ir,, the stationary distribution of the Markov chain
Thus we need only show that the limiting distribution of {X(O) is the
same as that of
which is X(t) just prior to an arrival. But because the
arrivals are Poisson, and arrivals in disjoint time intervals are independent, it must be that X(t) is independent of an arrival that occurs at time t.
IX Queueing Systems
564
It follows that {X(t)} and
have the same limiting distribution, and
therefore {X(t) } and the embedded Markov chain {X } have the same limiting distribution.
3.2.
The M/G/x System
Complete results are available when each customer begins service immediately upon arrival independently of other customers in the system. Such
situations may arise when modeling customer self-service systems. Let
W1, W2, ... be the successive arrival times of customers, and let V, V2, .. .
be the corresponding service times. In this notation, the kth customer is in
the system at time t if and only if W, s t (the customer arrived prior to t)
and W + V > t (the service extends beyond t).
The sequence of pairs (W,, V,), (W,, V2), ... forms a marked Poisson
process (see V, Section 6.2), and we may use the corresponding theory to
quickly obtain results in this model. Figure 3.1 illustrates the marked Pois-
V
4
(Wk, Vk)
x
W, W, I W W,
W6
i
0
Figure 3.1
W
t
Wk
For the M/G/o queue the number of customers in the system at
time t corresponds to the number of pairs (W V,) for which
W, <_ t and W, + V, > t. In the sample illustrated here, the number of customers in the system at time t is 3.
3.
General Service Time Distributions
565
son process. Then X(t), the number of customers in the system at time t,
is also the number of points (W, Vk) for which Wk < t and Wk + Vk > t.
That is, it is the number of points (Wk, Vk) in the unbounded trapezoid described by
A,= {(w,v):O_w<_tand v>t-w}.
(3.18)
According to V, Theorem 6.1, the number of points in A, follows a Poisson distribution with mean
µ(A) = Jj. A(dw) dG(v)
Ai
= A J I J dG(v)} dw
0 t-w
(3.19)
t
=A J[1 -G(t-w)]dw
0
t
= A J [1 - G(x)] dx.
0
In summary,
pk(t) = Pr{X(t) = k}
A(Ar)ke-," >
for k = 0, 1,
... ,
k!
where µ(A) is given by (3.19). As t - oo, then
x
I i m µ(A) = A
J
[1 - G(x)] dx = Av,
0
where v is the mean service time. Thus we obtain the limiting distribution
(Av)ke-aP
1Tk =
k!
fork=0,1,....
566
IX
Queueing Systems
Exercises
3.1.
Suppose that the service distribution in a single server queue is exponential with rate p.; i.e., G(v) = 1 - e-µY for v > 0. Substitute the mean
and variance of this distribution into (3.16) and verify that the result
agrees with that derived for the M/M/1 system in (2.10).
3.2.
Consider a single-server queueing system having Poisson arrivals
at rate A. Suppose that the service times have the gamma density
Laya- ' e - {1y
g(Y)
I^
r(a)
for y >_ 0,
where a > 0 and µ > 0 are fixed parameters. The mean service time is
a/µ and the variance is a/µZ. Determine the equilibrium mean queue
length L.
3.3. Customers arrive at a tool crib according to a Poisson process of
rate A = 5 per hour. There is a single tool crib employee, and the individual service times are random with a mean service time of 10 minutes and
a standard deviation of 4 minutes. In the long run, what is the mean number of workers at the tool crib either being served or waiting to be served?
3.4. Customers arrive at a checkout station in a market according to a
Poisson process of rate A = 1 customer per minute. The checkout station
can be operated with or without a bagger. The checkout times for customers are random. With a bagger the mean checkout time is 30 seconds,
while without a bagger this mean time increases to 50 seconds. In both
cases, the standard deviation of service time is 10 seconds. Compare the
mean queue lengths with and without a bagger.
3.5.
Let X(t) be the number of customers in an M/G/co queueing system
at time t. Suppose that X(0) = 0. Evaluate M(t) = E[X(t)], and show that
it increases monotonically to its limiting value as t - -.
Problems
3.1. Let X(t) be the number of customers in an M/GIc queueing system
at time t, and let Y(t) be the number of customers who have entered the
4.
Variations and Extensions
567
system and completed service by time t. Determine the joint distribution
of X(t) and Y(t).
3.2. In operating a queueing system with Poisson arrivals at a rate of
A = I per unit time and a single server, you have a choice of server mechanisms. Method A has a mean service time of v = 0.5 and a variance in
service time of r 2 = 0.2, while Method B has a mean service time of
v = 0.4 and a variance of r2 = 0.9. In terms of minimizing the waiting
time of a typical customer, which method do you prefer? Would your answer change if the arrival rate were to increase significantly?
4.
Variations and Extensions
In this section we consider a few variations on the simple queueing mod-
els studied so far. These examples do not exhaust the possibilities but
serve only to suggest the richness of the area.
Throughout we restrict ourselves to Poisson arrivals and exponentially
distributed service times.
4.1.
Systems with Balking
Suppose that a customer who arrives when there are n customers in
the systems enters with probability p,, and departs with probability
g = 1 - p,,. If long queues discourage customers, then p,, would be a decreasing function of n. As a special case, if there is a finite waiting room
of capacity C, we might suppose that
{1
forn<C,
0
for n ? C,
indicating that once the waiting room is filled, no more customers can
enter the system.
Let X(t) be the number of customers in the system at time t. If the arrival process is Poisson at rate A and a customer who arrives when there
are n customers in the system enters with probability p,,, then the appropriate birth parameters are
for n = 0,1,....
568
IX
Queueing Systems
In the case of a single server, then µ = µ for n = 1, 2,
. . . , and we may
evaluate the stationary distribution Irk of queue length by the usual means.
In systems with balking, not all arriving customers enter the system,
and some are lost. The input rate is the rate at which customers actually
enter the system in the stationary state and is given by
The rate at which customers are lost is A 5:,',-o 7r, ,q,,, and the fraction of cus-
tomers lost in the long run is
x
fraction lost = I
n=0
Let us examine in detail the case of an MIMIs system in which an arriving customer enters the system if and only if a server is free. Then
fork=0,1,...,s-1,
Jl
Ak={0
fork=s,
and
fork= 0, 1,...,s.
µk=k1x
To determine the limiting distribution, we have
k
9k=
k = 0, 1, .. . , s.
(4.1)
The long run fraction of customers lost is 1rsgs = 7rs, since q, = 1 in this
case.
4.
Variations and Extensions
4.2.
569
Variable Service Rates
In a similar vein, one can consider a system whose service rate depends
on the number of customers in the system. For example, a second server
might be added to a single-server system whenever the queue length exceeds a critical point . If arrivals are Poisson and service rates are memoryless, then the appropriate birth and death parameters are
Ak=A
fork=0,1,..., and µk=
12g
fork:
fork >.
More generally, let us consider Poisson arrivals Ak = A f o r k = 0, 1, ... ,
and arbitrary service rates µk for k = 1, 2,
The stationary distribution
in this case is given by
....
'rk =
aoAk
fork ? 1,
(4.2)
where
W
Ak
moo=
-1
k=1 µ'l'1'2
.
(4.3)
µ'k
4.3. A System with Feedback
Consider a single-server system with Poisson arrivals and exponentially
distributed service times, but suppose that some customers, upon leaving
the server, return to the end of the queue for additional service. In particular, suppose that a customer leaving the server departs from the system
with probability q and returns to the queue for additional service with
probability p = I - q. Suppose that all such decisions are statistically independent, and that a returning customer's demands for service are statistically the same as those of a customer arriving from outside the system.
Let the arrival rate be A and the service rate be A. The queue system is depicted in Figure 4.1.
570
Queueing Systems
IX
Feedback
a,
X
XXXI
X
Input
Arrivals
Figure 4.1
Lq
X
Server
Output
Departures
A queue with feedback.
Let X(t) denote the number of customers in the system at time t. Then
X(t) is a birth and death process with parameters A,, = A for n = 0, 1, .. .
and µ = qp. for n = 1, 2..... It is easily deduced that the stationary distribution in the case that A < qA is
lTk _ (1
- q )(q )X
for k = 0, 1,
....
(4.4)
4.4. A Two-Server Overflow Queue
Consider a two-server system where server i has rate µ, for i = 1, 2. Arrivals to the system follow a Poisson process of rate A. A customer arriving when the system is empty goes to the first server. A customer arriving
when the first server is occupied goes to the second server. If both servers
are occupied, the customer is lost. The flow is depicted in Figure 4.2.
Poisson Arrivals
Rate = X_
T
if #1 and #2 busy
if #1 is busy
I
Server # 2
Rate µ2
Server #1
Rate p.
l
Overflow lost
Overflow
Output
1
Figure 4.2 A two-server overflow model.
Variations and Extensions
4.
571
The system state is described by the pair (X(t), Y(t)), where
X(t) =
if Server #1 is busy,
if Server #1 is idle.
I
I0
and
Y(t) =
if Server #2 is busy,
if Server #2 is idle.
{1
0
The four states of the system are {(0, 0), (1, 0), (0, 1), (1, 1)), and transitions among these states occur at the rate given in the following table:
From
To
State
State
Transition
Rate
(0,0)
(1,0)
A
(1, 0)
(1, 0)
(1, 1)
(1, 1)
(0, 0)
(1, 1)
(1, 0)
A
(0,1)
µ1
(0,1)
(0,1)
(1, 1)
A
(0,0)
R2
Description
Arrival when system is empty
Service completion by #1 when #2 is free
Arrival when #1 is busy
Service completion by #2 when #1 is busy
Service completion by #1 when #2 is busy
Arrival when #2 is busy and #1 is free
Service completion by #2 when #1 is free
The process (X(t), Y(t)) is a finite-state, continuous-time Markov chain
(see VI, Section 6), and the transition rates in the table furnish the infinitesimal matrix of the Markov chain:
A=
(0, 0)
(0, 1)
(1, 0)
(1, 1)
(0, 0)
-A
0
A
0
(0, 1)
/L2
-(A + µ,)
0
A
(1,0)
µ,
0
-(A+µ,)
A
(1, 1)
0
µ1
1..x.2
-(11 + N-2)
From VI, (6.11) and (6.12), we find the stationary distribution 'rr = (ir(o.o),
IT(o.1), iT(i.o)1T1.n) by solving TrA = 0, or
572
IX
-A7r(o.0)
+ 1111r(i.o)
+ /L1T(O.1)
- (A + /.L2)Ir(O.1)
+ /L)IT().)
- (A + µ))iT().o)
ATi(o.o)
Queueing Systems
+ /.2IT(,.,)
+ A1T0.0) - (µ) + /L2)1(,.))
ATr(o.,)
together with
1r(O.o) + V(o.)) + 7rO.o) + 1(i.i) = 1.
Tedious but elementary algebra yields the solution:
µ1µ,(2A + µ) + N2)
D
A2µ,
1T(O.))=
D
(4.5)
IT(i.o) =
Aµ,(A + µ) + N2)
D
A2(A + p,2)
'n'(I.1) =
D
where
D=µ,µ2(2A+µ,+µ2)+A2µ,+Aµ2(A+Al +µi)
+ A2(A + µ).
The fraction of customers that are lost, in the long run, is the same as the
fraction of time that both servers are busy, 7r(,.,) = A2(A + µ2)/D.
4.5.
Preemptive Priority Queues
Consider a single-server queueing process that has two classes of customers, priority and nonpriority, forming independent Poisson arrival
processes of rates a and (3, respectively. The customer service times are
independent and exponentially distributed with parameters y and 5, respectively. Within classes there is a first come, first served discipline, and
the service of priority customers is never interrupted. If a priority cus-
4.
Variations and Extensions
573
tomer arrives during the service of a nonpriority customer, then the latter's
service is immediately stopped in favor of the priority customer. The interrupted customer's service is resumed when there are no priority customers present.
Let us introduce some convenient notation. The system arrival rate is
= a + /3, of which the fraction p = a/A are priority customers and
q = /i/A are nonpriority customers. The system mean service time is given
by the appropriately weighted means 1/y and 1/8 of the priority and nonpriority customers, respectively, or
(a
-p\l +q\Sl
+
(4.6)
where µ is the system service rate. Finally, we introduce the traffic intensities p = /1/µ for the system, and 0 = a/y and T = /3/S for the priority and
nonpriority customers, respectively. From (4.6) we see that p = cr + T.
The state of the system is described by the pair (X(t), Y(t)), where X(t)
is the number of priority customers in the system and Y(t) is the number
of nonpriority customers. Observe that the priority customers view the
system as simply an M/M/1 queue. Accordingly, we have the limiting distribution from (2.9) to be
for m = 0, 1, ...
lim Pr(X(t) = m) _ (1 - v)o"'
(4.7)
provided o = a/y < 1.
Reference to (2.10) and (2.13), gives us, respectively, the mean queue
length for priority customers
LPP =
a
y-a
=
Q
1-o,
(4.8)
and the mean wait for priority customers
W, =
y-a
l
(4.9)
To obtain information about the nonpriority customers is not as easy,
since these arrivals are strongly affected by the priority customers. Nevertheless, (X(t), Y(t)) is a discrete-state, continuous-time Markov chain,
574
IX
Queueing Systems
and the techniques of VI, Section 6 enable us to describe the limiting distribution, when it exists. The transition rates of the (X(t), Y(t)) Markov
chain are described in the following table:
From
To
State
State
Transition
Rate
Description
(m, n)
(m, n)
(0, n)
(m + 1, n)
(m, n + 1)
a
(0, n - 1)
8
Arrival of priority customer
Arrival of nonpriority customer
Completion of nonpriority service
(m, n)
(m - 1, n)
y
Completion of priority service
nil
R
m?1
Let
rm,,, = lim Pr(X(t) = m, Y(t) = n}
,mix
be the limiting distribution of the process. Reasoning analogous to that of
VI, (6.11) and (6.12) (where the theory was derived for a finite-state
Markov chain) leads to the following equations for the stationary distribution:
(a + /3)1T0.0 = Yi'1.0
(a + /3 + y) 1T
+ air,-1.0,
= Yirn,+1.0
(a + /3 + S)1r0.,. = Yir1
(a + /3 +
(4.10)
+ 87r0,1,
m ? 1,
(4.11)
n ? 1,
(4.12)
+ S1Ta,+1 + t31To.,,-1+
N1Tm.,-1 + air,,-1,n,
m, n
1.
(4.13)
The transition rates leading to equation (4.13) are shown in Figure 4.3.
In principle, these equations, augmented with the condition
Y.,,, I 7r,,,.,, = 1, may be solved for the stationary distribution, when it ex-
4.
Variations and Extensions
575
Rate out
Rate in
lTm+I.n(Y)
Figure 4.3 In equilibrium, the rate of flow into any state must equal the rate
of flow out. Illustrated here is the state (m, n) when m ? 1 and
n ? 1, leading to equation (4.13).
ists. We will content ourselves with determining the mean number Ln of
nonpriority customers in the system in steady state, given by
Ln = >
nwm.n
(4.14)
?n=On=0
We introduce the notation
x
x
M. = >. nlTmn = >. nlTm.n,
n=0
(4.15)
n=1
so that
(4.16)
Using (4.7), let
p. = Pr{X(t) = m} _ >. ?rm,n = (I - c)om
n=
(4.17)
576
IX
Queueing Systems
and
00
IT,, =Pr{Y(t)=n}=
(4.18)
M=0
We begin by summing both sides of (4.10) and (4.11) for m = 0, 1, .. .
to obtain
W
x
(a + N)?rO + y Y irin.O = y Y 1Tm.O + (S?T0,1 + a?TO,
m=1
m=1
which simplifies to give
(4.19)
Giro = 51TO.l.
Next, we sum (4.12) and (4.13) over m = 0, 1.... to obtain
x
(a + N)ir, + S1ro,n + y 7 1Tm,,, =
M=1
00
s7T0.+,+1 + PTT,,-1 + air,,,
M=1
which simplifies to
/31r + 81ro,n =
SiTO.n+1,
and inductively with (4.19), we obtain
/37r = 87TO.n+1
for n = 0, 1,
....
(4.20)
Summing (4.20) over n = 0, 1, ... and using I iT = 1, we get
x
/3 = S Z. Tro,,,+, = S Pr{X(t) = 0, Y(t) > 0},
n=0
or
Pr(X(t) = 0, Y(t) > 0}
irO,,, =
n=1
R
= T.
(4.21)
s
Since (4.17) asserts that Pr{X(t) = 0} = 1 - (a/y) = 1 - o,, we have
7ro.o = Pr(X(t) = 0, Y(t) = 0} = Pr{X(t) = 0} - Pr{X(t) = 0, Y(t) > 0}
=1-a-
1-v-T
when o, +r<1.
(4.22)
4.
Variations and Extensions
577
With these preliminary results in hand, we turn to determining
Mm = Y,," , n'Tm,n. Multiplying (4.12) by n and summing, we derive
00
00
(a + /3 + S)Mo = TM, + S 7, niro,,,+1 + /3 7, n1To.n-1
n=1
n=1
z
00
= 7M1 + SMo - 81 ITo.u+I + (3Mo +'0 7 iro"-1
n=0
n=1
=yM,+SMo-S(S)+/3Mo+13(1-x),
where the last line results from (4.17) and (4.21). After simplification and
rearrangement, the result is
M1 = UMo +
R Q.
(4.23)
Y
We next multiply (4.13) by n and sum to obtain
00
(a + 13 + Y)Mm = YMm+1 + $ > nwm,,,-1 + aMm-1
n=1
00
YMm+I + W. + l3
lTm.n-1 + aMm-1
n=1
Again, referring to (4.17) and simplifying, we see that
(a+Y)M,n=YMm+1+aM,n-1+/3(1 -
a.)am
for m = 1, 2,.... (4.24)
Equations (4.23) and (4.24) can be solved inductively to give
Mm=Moo- + mo-m
form= 0,
1,...,
Y
which we sum to obtain
M. = 1
L. _
m
1
U
[(1
a)
](4.25)
578
IX
Queueing Systems
This determines L. in terms of Mo. To obtain a second relation, we multiply (4.20) by n and sum to obtain
(3L"
SMo - 8 Z, '7ro.,,+
1
/7=0
n=0
= SMa -
S(A3
[see (4.21)],
)
or
(L" + 1) = r(L,, + 1).
MO =
(4.26)
S
We substitute (4.26) into (4.25) and simplify, yielding
L
T
1-U
11
)L" =
0-1
1-Q
1
r(L" + 1) + '6
yl - 0-Qall
[T+ R
o*
yl-0- ],
and finally,
_(
L
T
\ 1-Q-T)[1 +
S
o
1y)l-
J
(4.27)
The condition that L" be finite (and that a stationary distribution exist) is
that
p=Q+T<1.
That is, the system traffic intensity p must be less than one.
Since the arrival rate for nonpriority customers is 8, we have that the
mean waiting time for nonpriority customers is given by W" = L"//3.
Some simple numerical studies of (4.8) and (4.27) yield surprising results concerning adding priority to an existing system. Let us consider first
a simple M/M/1 system with traffic intensity p whose mean queue length
is given by (2.10) to be L = pl(1 - p). Let us propose modifying the system in such a way that a fraction p = ; of the customers have priority. We
Exercises
579
assume that priority is independent of service time. These assumptions
lead to the values a = /3 = ZA and y = S = µ, whence o- = r = p/2. Then
the mean queue lengths for priority and nonpriority customers are given
by
LP _
_
Q
1-o
_
p'2
1-(p12)
p
2-p
and
p'2
_
p
L=I(_'2- p)[I +(2-p)(1
- p)
The mean queue lengths L, LP, and L,, were determined for several values of the traffic intensity p. The results are listed in the following table:
p
0.6
0.8
0.9
0.95
L
1.50
4.00
9.00
19.00
LP
0.43
0.67
0.82
0.90
L,
1.07
3.34
8.19
18.10
It is seen that the burden of increased queue length, as the traffic intensity increases, is carried almost exclusively by the nonpriority customers!
Exercises
4.1. Consider a two-server system in which an arriving customer enters
the system if and only if a server is free. Suppose that customers arrive according to a Poisson process of rate A = 10 customers per hour, and that
service times are exponentially distributed with a mean service time of six
minutes. In the long run, what is the rate of customers served per hour?
4.2. Customers arrive at a checkout station in a small grocery store according to a Poisson process of rate A = 1 customer per minute. The
checkout station can be operated with or without a bagger. The checkout
times for customers are exponentially distributed, and with a bagger the
580
IX
Queueing Systems
mean checkout time is 30 seconds, while without a bagger this mean time
increases to 50 seconds. Suppose the store's policy is to have the bagger
help whenever there are two or more customers in the checkout line. In
the long run, what fraction of time is the bagger helping the cashier?
4.3.
Consider a two-server system in which an arriving customer enters
the system if and only if a server is free. Suppose that customers arrive according to a Poisson process of rate A = 10 customers per hour, and that
service times are exponentially distributed. The servers have different experience in the job, and the newer server has a mean service time of six
minutes, while the older has a mean service time of four minutes. In the
long run, what is the rate of customers served per hour? Be explicit about
any additional assumptions that you make.
4.4. Suppose that incoming calls to an office follow a Poisson process
of rate A = 6 per hour. If the line is in use at the time of an incoming call,
the secretary has a HOLD button that will enable a single additional caller
to wait. Suppose that the lengths of conversations are exponentially dis-
tributed with a mean length of 5 minutes, that incoming calls while a
caller is on hold are lost, and that outgoing calls can be ignored. Apply the
results of Section 4.1 to determine the fraction of calls that are lost.
Problems
4.1. Consider the two-server overflow queue of Section 4.4 and suppose
the arrival rate is A = 10 per hour. The two servers have rates 6 and 4 per
hour. Recommend which server should be placed first. That is, choose between
and
and justify your answer. Be explicit about your criterion.
4.2. Consider the preemptive priority queue of Section 4.5 and suppose
that the arrival rate is k = 4 per hour. Two classes of customers can be
identified, having mean service times of 12 minutes and 8 minutes, and it
is proposed to give one of these classes priority over the other. Recom-
5.
Open Acyclic Queueing Networks
581
mend which class should have priority. Be explicit about your criterion
and justify your answer. Assume that the two classes appear in equal proportions and that all service times are exponentially distributed.
Balking refers to the refusal of an arriving customer to enter the
queue. Reneging refers to the departure of a customer in the queue before
obtaining service. Consider an M/M11 system with reneging such that the
probability that a specified single customer in line will depart prior to service in a short time interval (t, t + At] is
o(At) when n is the number of customers in the system. (Note that ro = r, = 0.) Assume Poisson
arrivals at rate A and exponential service times with parameter µ, and determine the stationary distribution when it exists.
4.3.
4.4. A small grocery store has a single checkout counter with a full-time
cashier. Customers arrive at the checkout according to a Poisson process
of rate A per hour. When there is only a single customer at the counter, the
cashier works alone at a mean service rate of a per hour. Whenever there
is more than one customer at the checkout, however, a "bagger" is added,
increasing the service rate to /3 per hour. Assume that service times are exponentially distributed and determine the stationary distribution of the
queue length.
4.5. A ticket office has two agents answering incoming phone calls. In
addition, a third caller can be put on HOLD until one of the agents becomes available. If all three phone lines (both agent lines plus the hold
line) are busy, a potential caller gets a busy signal, and is assumed lost.
Suppose that the calls and attempted calls occur according to a Poisson
process of rate A, and that the length of a telephone conversation is exponentially distributed with parameter µ. Determine the stationary distribution for the process.
5.
Open Acyclic Queueing Networks
Queueing networks, composed of groups of service stations, with the departures of some stations forming the arrivals of others, arise in computer
and information processing systems, manufacturing job shops, service in-
582
IX
Queueing Systems
dustries such as hospitals and airport terminals, and in many other contexts. A remarkable result often enables the steady-state behavior of these
complex systems to be analyzed component by component.
5.1.
The Basic Theorem
The result alluded to in the preceding paragraph asserts that the departures
from a queue with Poisson arrivals and exponentially distributed service
times in statistical equilibrium also form a Poisson process. We give the
precise statement as Theorem 5.1. The proof is contained in an appendix
at the end of this section. See also Problem 2.5.
Theorem 5.1 Let {X(t), t ? 0) be a birth and death process with constant birth parameters A,, = A for n = 0, 1, ... , and arbitrary death
parameters µ for n = 1, 2, .... Suppose there exists a stationary
distribution lTk ? 0 where Y.k irk = 1 and that Pr{X(0) = k} = ITk for
k = 0, 1. .... Let D(t) denote the number of deaths in (0, t]. Then
Pr{X(t) = k, D(t) = j} = Pr{X(t) = k} Pr{D(t) = j}
(Atye-"'
Ilk
j!
for k,j?0.
Remark The stipulated conditions are satisfied, for example, when X(t)
is the number of customers in an M/M/s queueing system that is in steady
state wherein Pr(X(0) = j) = Trj, the stationary distribution of the process.
In this case, a stationary distribution exists provided that A < sµ, where
µ is the individual service rate.
To see the major importance of this theorem, suppose that X(t) represents the number of customers in some queueing system at time t. The theorem asserts that the departures form a Poisson process of rate A. Furthermore, the number D(t) of departures up to time t is independent of the
number X(t) of customers remaining in the system at time t.
We caution the reader that the foregoing analysis applies only if the
processes are in statistical equilibrium where the stationary distribution
Irk = Pr{X(t) = k} applies. In contrast, under the condition that X(0) = 0,
5.
Open Acyclic Queueing Networks
583
then neither will the departures form a Poisson process, nor will D(t) be
independent of X(t).
5.2.
Two Queues in Tandem
Let us use Theorem 5.1 to analyze a simple queueing network composed
of two single-server queues connected in series as shown in Figure 5.1.
Let Xk(t) be the number of customers in the kth queue at time t. We assume steady state. Beginning with the first server, the stationary distribution (2.9) for a single server queue applies, and
1--
Pr{X,(t) = n} =
(_'
for n = 0, 1.
... .
Theorem 5.1 asserts that the departure process from the first server, denoted by D,(t), is a Poisson process of rate A that is statistically independent of the first queue length X,(t). These departures form the arrivals to
the second server, and therefore the second system has Poisson arrivals
and is thus an M/M/1 queue as well. Thus, again using (2.9),
Pr(X1(t) = m) = (I
-)()
Server 1
Poisson Arrivals
Rate X
i
form = 0, 1. ... .
Server 2
Rate
Rate
ILi
1L?
Figure 5.1 Two queues in series in which the departures from the first form
the arrivals for the second.
584
IX
Queueing Systems
Furthermore, because the departures D,(t) from the first server are independent of X,(t), it must be that X2(t) is independent of X,(t). We thus obtain the joint distribution
Pr{X1(t) = n and X2(t) = m} = Pr{X1(t) = n} Pr{X2(t) = m}
-
().)m
\1
for n, m = 0,1.....
We again caution the reader that the foregoing analysis applies only
when the network is in its limiting distribution. In contrast, if both queues
are empty at time t = 0, then neither will the departures D,(t) form a Poisson process nor will D,(t) and X,(t) be independent.
5.3.
Open Acyclic Networks
The preceding analysis of two queues in series applies to more general
systems. An open queueing network (see Figure 5.2) has customers arriving from and departing to the outside world. (The repairman model in VI,
Section 4 is a prototypical closed queueing network.) Consider an open
network having K service stations, and let Xk(t) be the number of customers in queue k at time t. Suppose
1. The arrivals from outside the system to distinct servers form independent Poisson processes.
Arrivals
Figure 5.2 An open queueing network.
S.
Open Acyclic Aueueing Networks
585
2. The departures from distinct servers independently travel instantly
to other servers, or leave the system, with fixed probabilities.
3. The service times for the various servers are memoryless in the sense
that
Pr{Server #k completes a service in (t, t + Ot)IXk(t) = n}
=
o(Ot)
for n = 1, 2, ... ,
(5.1)
and does not otherwise depend on the past.
4. The system is in statistical equilibrium (steady state).
5. The network is acyclic in that a customer can visit any particular
server at most once. (The case where a customer can visit a server
more than once is more subtle, and is treated in the next section.)
Then
(a) X,(t), X2(t), ... , XK(t) are independent processes, where
Pr{X1(t) = n,, X2(t) = n2,
... , XK(t) = nK}
= Pr{X1(t) = n.) Pr{XZ(t) = n2} .. Pr{XK(t) = nK}.
(5.2)
(b) The departure process Dk(t) associated with the kth server is a Poisson process, and Dk(t) and Xk(t) are independent.
(c) The arrivals to the kth station form a Poisson process of rate Ak.
(d) The departure rate at the kth server equals the rate of arrivals to that
server.
Let us add some notation so as to be able to express these results more
explicitly. Let
A,)k = rate of arrivals to station k from outside the system,
Ak = rate of total arrivals to station k,
Pk; = probability that a customer leaving station k next visits station j.
Then, the arrivals to station k come from outside the system or from some
other station j. The departure rate from j equals the arrival rate to j, the
fraction Pk of which go to station k, whence
(5.3)
586
IX
Queueing Systems
Since the network is acyclic, (5.3) may be solved recursively, beginning
with stations having only outside arrivals. The simple example that follows will make the procedure clear.
The arrivals to station k form a Poisson process of rate Ak. Let
A
k
41k(n) = ITko X
for n = 1, 2,
PkIPk2 ... Ak
... ,
(5.4)
where
All
4'40) = ITko =
{1
(
+ ,7=i
7, \AAiAA-2
... Ab,
(5.5)
Referring to (4.2) and (4.3) we see that (5.4) and (5.5) give the stationary distribution for a queue having Poisson arrivals at rate Ak and memoryless service times at rates µk for n = 1, 2.
Accordingly, we may
now express (5.2) explicitly as
....
Pr(X1(t) = n X2(t) = n2,
. . .
,
XK = nk}
= 41,(n.)412(n2) ... 4K(nK)
(5.6)
Example Consider the three station network as shown in Figure 5.3.
#2
P12-;
µ2 = 2
JRateX2= (;) 4
Poisson Arrivals
#1
Rate = Xo, = 4
µu=3
µ,,,=6,n?2
Poisson,
I
Rate (;) 4
#3
Poisson Departures
µs = 6
Poisson, Rate = () 4
Rate = 4
Figure 5.3 A three station open acyclic network. Two servers, each of rate 3,
at the first station give rise to the station rates µ = 3 and µ,, =
6 for n ? 2. Stations 2 and 3 each have a single server of rate 2
and 6, respectively.
Open Acyclic Queueing Networks
5.
587
The first step in analyzing the example is to determine the arrival rates
at the various stations. In equilibrium, the arrival rate at a station must
equal its departure rate, as asserted in (d). Accordingly, departures from
state 1 occur at rate A, = 4, and since these departures independently travel
to stations 2 and 3 with respective probabilities P,, = and P13 = , we determine the arrival rate A2 = (104. At station 3 the arrivals include both
those from station 1 and those from station 2. Thus A3 = ()4 + (1)4 = 4.
Having determined the arrival rates at each station, we turn to determining the equilibrium probabilities. Station I is an M/M/2 system with
A = 4 and µ = 3. From (2.16), or (5.4) and (5.5), we obtain
2
ll
Pr{X,(t)=0}=7ro={1+(31+
-i
=0.2
2(1/3)
and
Pr{X1(t) = n} =
(4
. 2)
3/ (0
(0.4)
for n = 1 ,
2
for n ? 2 .
( 3 )"
Station 2 is an M/M/1 system with A = and µ = 2. From (2.9) we obtain
Pr(X2(t) = n) = (--)(--)
for n = 0, 1,
... .
Similarly station 3 is an M/M/1 system with A = 4 and µ = 6, so that (2.9)
yields
Pr{X3(t)=n}
=(3)(3)
for n = 0,1.....
Finally, according to Property (a), the queue lengths X#), K,(t), and
X3(t) are independent, so that
Pr{X1(t) = n,, X2(t) = n2, X3(t) = n3)
= Pr{X1(t) = n,} Pr(X2(t) = n,} Pr{X3(t) = n3}.
588
5.4.
IX
Queueing Systems
Appendix: Time Reversibility
Let {X(t), -oo < t < +co} be an arbitrary countable-state Markov chain
having a stationary distribution Tr; = Pr{X(t) = j} for all states j and all
times t. Note that the time index set is the whole real line. We view the
process as having begun indefinitely far in the past, so that it now is
evolving in a stationary manner. Let Y(t) = X(- t) be the same process, but
with time reversed. The stationary process {X(t)} is said to be time reversible if { X(t) } and { Y(t) } have the same probability laws. Clearly,
Pr { X(0) = j) = Pr { Y(O) = j) = ir;, and it is not difficult to show that both
processes are Markov. Hence, in order to show that they share the same
probability laws it suffices to show that they have the same transition
probabilities. Let
P;(t) = Pr{X(t) = jIX(0) = i},
Qi;(t)=Pr{Y(t)=jlY(0)=i).
The process {X(t)} is reversible if
/
Pj (t) = Qij (t)
(5.7)
for all states i, j and all times t. We evaluate Qi;(t) as follows:
Qi; (t) = Pr{Y(t) = jI Y(0) = i }
= Pr{X(-t) = i1 X(0) = i }
= Pr{X(0) = jIX(t) = i}
(by stationarity)
Pr{X(0) = j, X(t) = i}
Pr{X(t) = i}
IT; Pi(t)
.
IT,
In conjunction with (5.7) we see that the process {X(t)} is reversible if
F,(t) = Q11(t) =
'Pi(t)
iTi
or
ii I;(t) = ;Pi(t),
(5.8)
S.
Open Acyclic oueueing Networks
589
for all states i, j and all times t.
As a last step, we determine a criterion for reversibility in terms of the
infinitesimal parameters
a;; = lim
110
1
I
t
i 0 j.
Pr{X(t) = AX(o) = i },
It is immediate that (5.8) holds when i = j. When i 0 j,
P;(t) = at + o(t),
(5.9)
which substituted into (5.8) gives
1rJa;;t + o(t)] = ir;[a;;t + o(t)],
and after dividing by t and letting t vanish, we obtain the criterion
'zr,a;j = ir1a;;
for all i 0 j.
(5.10)
When the transition probabilities are determined by the infinitesimal parameters, we deduce that the process {X(t)} is time reversible whenever
(5.10) holds.
All birth and death processes satisfying VI, (3.5) and having stationary
distributions are time reversible! Because birth and death processes have
a,.1+I = A.,
a;.;-1 = µr,
and
a;.l=0
ifli
- jj> 1,
in verifying (5.10) it suffices to check that
or
for i = 0, 1,
....
But [see VI, (4.6) and (4.7)],
1r;=aro
Jy0A,
A;-,
fori=l,2,...,
(5.11)
590
Queueing Systems
IX
whence (5.11) becomes
AOA, ... A,
1
1TO (
Alp-- , , * A
)Ai = 'To
N+ I
Al P-2
which is immediately seen to be true.
5.5.
Proof of Theorem 5.1
Let us consider a birth and death process {X(t)} having the constant birth
rate Ak = A for k = 0, 1, ... , and arbitrary death parameters lµk > 0 for
k = 1, 2,
This process corresponds to a memoryless server queue
having Poisson arrivals. A typical evolution is illustrated in Figure 5.4.
The arrival process for {X(t) } is a Poisson process of rate A. The reversed
time process Y(t) = X(-t) has the same probabilistic laws as does {X(t)},
so the arrival process for {Y(t)} also must be a Poisson process of rate A.
But the arrival process for {Y(t)} is the departure process for {X(t)} (see
Figure 5.4). Thus it must be that these departure instants also form a Poisson process of rate A. In particular, if D(t) counts the departures in the
process over the duration (0, t], then
....
(At)'e-A,
for j = 0,
Pr{D(t) = j} =
1,
....
(5.12)
j!
X(t)
Future
for
,
,
,
i
t
Arrivals for
i
Departures for
1
e
+
+
Departures for
+
1
Arrivals for
Figure 5.4 A typical evolution of a queueing process. The instants of arrivals
and departures have been isolated on two time axes below the graph.
Exercises
591
Moreover, looking at the reversed process Y(- t) = X(t), the "future" ar-
rivals for Y(-t) in the Yduration [-t, 0) are independent of Y(-t) = X(t).
(See Figure 5.4.) These future arrivals for Y(-t) are the departures for
in the interval (0, t}. Therefore, these departures and X(t) = Y(-t) must be
independent. Since Pr{X(t) = k} = lrk, by the assumption of stationarity,
the independence of D(t) and X(t) and (5.12) give
Pr{X(t) = k, D(t) = j} = Pr (X(t) = k} Pr ( D(t)
j)
1rke-`(At)!
and the proof of Theorem 5.1 is complete.
Exercises
5.1. Consider the three server network pictured here:
P21 = .2
P12 =
Server #2
=3,n?1
Arrivals
Poisson, rate
ip.
Departs with
probability .8
Server #1
1
X=2
Server # 3
1
In the long run, what fraction of time is server #2 idle while, simultaneously, server #3 is busy? Assume that all service times are exponentially
distributed.
5.2. Refer to the network of Exercise 5.1. Suppose that Server #2 and
Server #3 share a common customer waiting area. If it is desired that the
total number of customers being served and waiting to be served not exceed the waiting area capacity more than 5% of the time in the long run,
how large should this area be?
592
IX
Gueueing Systems
Problems
5.1. Suppose three service stations are arranged in tandem so that the
departures from one form the arrivals for the next. The arrivals to the first
station are a Poisson process of rate A = 10 per hour. Each station has a
single server, and the three service rates are µ, = 12 per hour, p, = 20 per
hour, and p, = 15 per hour. In-process storage is being planned for station 3. What capacity C3 must be provided if in the long run, the probability of exceeding C3 is to be less than or equal to 1 percent? That is, what
is the smallest number C3 = c for which lim,, Pr{X3(t) > c} : 0.01?
6.
General Open Networks
The preceding section covered certain memoryless queueing networks in
which a customer could visit any particular server at most once. With this
assumption, the departures from any service station formed a Poisson
process that was independent of the number of customers at that station in
steady state. As a consequence, the numbers X,(t), X2(t), ... , XK(t) of customers at the K stations were independent random variables, and the product form solution expressed in (5.2) prevailed.
The situation where a customer can visit a server more than once is
more subtle. On the one hand, many flows in the network are no longer
Poisson. On the other hand, rather surprisingly, the product form solution
of (5.2) remains valid.
Example To begin our explanation, let us first reexamine the simple
feedback model of Section 4.3. The flow is depicted in Figure 6.1. The arrival process is Poisson, but the input to the server is not. (The distinction
between the arrival and input processes is made in Figures 1.2 and 4.1.)
Feedback
Not Poisson
Feedback with
probability p
Depart with probability q
Arrivals
Input
Output
Poisson Departures
Poisson X
Not Poisson
Not Poisson
Rate = X
Figure 6.1
A single server with feedback.
6.
General Open Networks
593
The output process, as shown in Figure 6.1, is not Poisson, nor is it independent of the number of customers in the system. Recall that each customer in the output is fed back with probability p and departs with probability q = 1 - p. In view of this non-Poisson behavior, it is remarkable
that the distribution of the number of customers in the system is the same
as that in a Poisson M/M/1 system whose input rate is A/q and whose service rate is p,, as verified in (4.4).
Example Let us verify the product form solution in a slightly more
complex two server network, depicted in Figure 6.2.
Server #2
P2
Feedback with
probability p
Arrivals
Server #1
Poisson, rate X
Wi
Depart with
probability q
Figure 6.2 A two server feedback system. For example, server #2 in this system might be an inspector returning a fraction p of the output for
rework.
If we let X;(t) denote the number of customers at station i at time t, for
i = 1, 2, then X(t) = [X,(t), X2(t)] is a Markov chain whose transition rates
are given in the following table:
Transition
Rate
From
State
State
(m, n)
(m, n)
(m + 1, n)
A
(m + 1, n - 1)
µ,
Arrival of new customer
Input of feedback customer
(m - 1, n)
qµ,
Departure of customer
(m - 1, n + 1)
pµ,
Feedback to server #2
To
Description
n?1
(m, n)
m?1
(m, n)
m?1
594
IX
Queueing Systems
Let ir,,,,, = lim,,,. Pr(X1(t) = m, X2(t) = n} be the stationary distribution of the process. Reasoning analogous to that of (6.11) and (6.12) of VI
(where the theory was developed for finite-state Markov chains) leads to
the following equations for the stationary distribution:
(6.1)
Air00 = g1.1ir1.0,
(A + jx2)ir0.,, = PP'Iii.,,-1 + gp1iT1.,,,
(A +
n ? 1,
Air,,,-1.0 + gj 1ir,,+1.0 +
1.1,
m ? 1,
(A + A, +
(6.2)
(6.3)
AlT,,-i.,, + p1x11;m+1.,,-1 + qp 1 irn,+;.,,
m, n ? 1.
(6.4)
+
The mass balance interpretation as explained following (6.12) in VI
may help motivate (6.1) through (6.4). For example, the left side in (6.1)
measures the total rate of flow out of state (0, 0) and is jointly proportional
to ir0.0, the long run fraction of time the process is in state (0, 0), and A,
the (conditional) transition rate out of (0, 0). Similarly, the right side of
(6.1) measures the total rate of flow into state (0, 0).
Using the product form solution in the acyclic case, we will "guess" a
solution and then verify that our guess indeed satisfies (6.1) through (6.4).
First we need to determine the input rate, call it A1, to server #1. In equilibrium, the output rate must equal the input rate, and of this output, the
fraction p is returned to join the new arrivals after visiting server #2. We
have
Input Rate = New Arrivals + Feedback,
which translates into
A, = A + pA,
or
A,=1-pq
A
The input rate to server #2 is
A
(6.5)
6.
General Open Networks
595
A
A2 = pA, = q
(6.6)
The solution that we guess is to treat server #1 and server #2 as independent M/M/1 systems having intput rates A, and A2, respectively (even
though we know from our earlier discussion that the input to server #2,
while of rate A2, is not Poisson). That is, we attempt a solution of the form
A,
- ')(' ")
A
AI
P12
P12
(6.7)
In
form,n? 1.
- (1
It is immediate that
W
W
77 1Tirr.n = 1
,n=0n=0
provided that A, = (A/q) < µ, and A2 = pAlq < p.2.
We turn to verifying (6.1) through (6.4). Let
(Algµ,)m X
(pA1gµ2)". It suffices to verify that 0,,,,, satisfies (6.1) through (6.4), since
and 6,,,,, differ only by the constant multiple iro,,, = 1 - A,/µ,) X
(1 - A2/µ2). Thus we proceed to substitute 9,,,,,, into (6.1) through (6.4) and
verify that equality is obtained.
We verify (6.1):
A=q,.l
qµ,
=A.
We verify (6.2):
(A
+
11
+
-
or after dividing by (pA/gµ2)" and simplifying,
A+µ2=\pq
1\pAI
+A=A+µ2.
qk
596
IX
Queueing Systems
We verify (6.3):
)
(A
_ AI
I,,,+i
+
+
'(gp-z
P,( qlu,)
+µ)(qµ,
1'
which, after dividing by (A/qµ,)"', becomes
(A+µ0_A(q
+gA,(
)
gµj +µ\gJ11)(q
)'
or
A+µ, =glxl +A+plL,, = A+ µ,.
The final verification, that 9satisfies (6.4), is left to the reader as Exercise 6.1.
6.1.
The General Open Network
Consider an open queueing network having K service stations, and le
Xk(t) denote the number of customers at station k at time t. We assume that
1. The arrivals from outside the network to distinct servers form independent Poisson processes, where the outside arrivals to station k
occur at rate A.
2. The departures from distinct servers independently travel instantly
to other servers, or leave the system, with fixed probabilities, where
the probability that a departure from station j travels to station k is
Pik
3. The service times are memoryless, or Markov, in the sense that
Pr{Server #k completes a service in (t, t + ot]!Xk(t) = n)
=
o(Ot)
for n = 1, 2,
... ,
and does not otherwise depend on the past.
4. The system is in statistical equilibrium (stationary).
(6.8)
General Open Networks
6.
597
5. The system is completely open in that all customers in the system
eventually leave.
Let Ak be the rate of input at station k. The input at station k is composed
of customers entering from outside the system, at rate Aok, plus customers
traveling from (possibly) other stations. The input to station k from station
j occurs at rate A;Pk, whence, as in (5.3),
x
fork= 1,...,K.
Ak= AOk+ Z, A/P;k
(6.9)
j=1
Condition 5 above, that all entering customers eventually leave, ensures
that (6.9) has a unique solution.
With A...... A, given by (6.9), the main result is the product form
solution
Pr{X1(t) = n,, X2(t) = n,, .
. .
Xk(t) = ng}
,
(6.10)
= i/i1(n1)Y'2(n2) ... Y'K(nK)+
where
TlkOAk
t/ik(n)=
for n = 1, 2,
I
I /2'k2
*
... ,
(6.11)
* * Pin
and
All
410) _ 1rkO =
k
{ 1+
n=I l-'i'k1P'k2
Alai
t
(6.12)
.
Example The example of Figure 6.1 (see also Section 4.3) corresponds
to K = 1 (a single service station) for which P = p < 1. The external arrivals are at rate A,, = A, and (6.9) becomes
A, = Ao, + AP,,,
A, = A + A,p,
or
which gives A, = A/(1 - p) = A/q. Since the example concerns a single
server, then
= µ for all n, and (6.11) becomes
\n
41(n) = ?l10(µ
)n'
=
1110
`
qlA
598
IX
Oueueing Systems
where
I - -),
9µ
IT,o =
in agreement with (4.4).
Example Consider next the two server example depicted in Figure 6.2.
The data given there furnish the following information:
A01=A,
A02=0,
P11 = 0,
P12 = p,
P21 = 1,
P22 = 0,
which substituted into (6.9) gives
A, = A + A2(1),
A2=0+A1(p),
which readily yields
A, _
A
q
and
AZ =
PA
q
in agreement with (6.5) and (6.6). It is readily seen that the product solution of (6.10) through (6.12) is identical with (6.7), which was directly
verified as the solution in this example.
Exercises
6.1.
In the case m ? 1, n ? 1, verify that 9as given following (6.7)
satisfies the equation for the stationary distribution (6.4).
Problems
6.1. Consider the three server network pictured here:
Problems
599
P21 = .2
P12 =
Server # 2
1L2n=3,n?1
Arrivals
j
Poisson, rate
1.
Departs with
probability .8
Server #1
µIn=6,n? 1
1=2
LPI
Server #3
µ3n=2,n?1
In the long run, what fraction of the time is server #2 idle while, simultaneously, server #3 is busy? Assume that the system satisfies assumptions
(1) through (5) of a general open network.
Further Reading
Elementary Textbooks
Breiman, L. Probability and Stochastic Processes with a View Toward
Applications. Boston: Houghton Mifflin, 1969.
Cinlar, E. Introduction to Stochastic Processes. Englewood Cliffs, N.J.:
Prentice-Hall, 1975.
Cox, D. R., and H. D. Miller. The Theory of Stochastic Processes. New
York: John Wiley & Sons, 1965.
Hoel, R. G., S. C. Port, and C. J. Stone. Introduction to Stochastic Processes.
Boston: Houghton Mifflin, 1972.
Kemeny, J. G., and J. L. Snell. Finite Markov Chains. New York: Van Nostrand Reinhold, 1960.
Intermediate Textbooks
Bhat, U. N. Elements of Applied Stochastic Processes. New York: John
Wiley & Sons, 1972.
Breiman, L. Probability. Reading, Mass.: Addison-Wesley, 1968.
Dynkin, E. B., and A. A. Yushkevich. Markov Processes: Theorems and
Problems. New York: Plenum, 1969.
Feller, W. An Introduction to Probability Theory and Its Applications. 2 vols.
New York: John Wiley & Sons, 1966 (vol. 2), 1968 (vol. 1, 3rd ed.).
Karlin, S., and H. M. Taylor. A First Course in Stochastic Processes. New
York: Academic Press, 1975.
Karlin, S., and H. M. Taylor. A Second Course in Stochastic Processes. New
York: Academic Press, 1981.
Further Readings
602
Kemeny, J. G., J. L. Snell, and A. W. Knapp. Denumerable Markov Chains.
New York: Van Nostrand Reinhold, 1966.
Ross, S. M. Stochastic Processes. New York: John Wiley & Sons, 1983.
Ross, S. M. Introduction to Probability Models, 5/e New York: Academic
Press, 1993.
Renewal Theory
Kingman, J. F. C. Regenerative Phenomena. New York: John Wiley & Sons,
1972.
Queueing Processes
Kleinrock, L. Queueing Systems. Vol. 1, Theory. Vol. 2, Computer Applications. New York: John Wiley & Sons, Interscience, 1976.
Branching Processes
Athreya, K. B., and P. Ney. Branching Processes. New York: SpringerVerlag, 1970.
Harris, T. The Theory of Branching Processes. New York: Springer-Verlag,
1963.
Stochastic Models
Bartholomew, D. J. Stochastic Models for Social Processes. New York: John
Wiley & Sons, 1967.
Bartlett, M. S. Stochastic Population Models in Ecology and Epidemiology.
New York: John Wiley & Sons, 1960.
Goel, N. S., and N. Richter-Dyn. Stochastic Models in Biology. New York:
Academic Press, 1974.
Point Processes
Lewis, P. A. Stochastic Point Processes: Statistical Analysis, Theory, and
Applications. New York: John Wiley & Sons, Interscience, 1972.
Answers to Exercises
Chapter I
2.1.
Because B and B` are disjoint events whose union is the whole sample space, the law of total probability (Section 2.1) applies to give the desired formula.
2.3.
(b)Ax)
f(x) = 3x2
0
(c) E[X] - 1
(d) Pr(
X<_1}
2.4.
(b) E[Z] _ $.
(c) Var[Z] = 6;.
2.7.
(a)
0
FX(x) = xR
1
for x s 0;
for 0 < x < 1;
for x > 1.
- 26
64,
for x < 0;
for O s x s l;
for 1 < x.
(b) E[X] = R/(1 + R).
(c) Var[X] = R/[(R + 2)(R + 1)2].
2.8. f(v) = A(1 - v)A-' for O s v s 1;
E[V]=1/(A+ 1);
Var[V] = A/[(A + 2)(A + 1)2].
Answers to Exercises
604
2.9.
for x < 0;
0
x2
FX(x)
for x
1; Var[X] = 16-
3.1.
Pr{X = 3} = ;2.
3.2.
Pr {0 defective) = 0.3151.
Pr{0 or 1 defective) = 0.9139.
3.3.
Pr{N = 10) = 0.0315.
3.4.
Pr{X=2} =2e-2=0.2707.
Pr(Xs 2) = 5e-22= 0.6767.
3.5.
Pr{X ? 81 = 0.1334.
3.6.
(a) Mean = n 2 1; Variance =
m+1
(b)
I
Pr{Z=m}=
n2 - 1
12
form=0,...,n;
n-
2n+1-m
for m = n + 1, ..., 2n.
n2
(c) Pr{U = k} =
4.1.
1 +2(n-k)
(n + 1)2
for
k=0,...,n.
Pr{X>1.5}=e3=0.0498.
Pr{X = 1.5) = 0.
4.2.
Median =
4.3.
Exponential distribution with parameter A/2.54.
4.4.
Mean = 0; Variance = 1.
log 2; Mean =
.
605
Answers to Exercises
Qy - PO'xQY
4.5.
a*
4.6.
(a) f,.(y) = e-,
uX + 02
for
2po-xv,.
p
±1.
for y,::> 0.
c- iv
1
1
(b) f,,(w) = (w)
for 0 < w < 1.
n
4.7. ^ as the gamma density fR(r) =
5.1.
Pr{X ? 1 } = 0.6835938
Pr{X > 2) = 0.2617188
Pr{X? 31 = 0.0507812
Pr { X ? 4) = 0.0039062.
5.2.
Mean = ;.
5.3.
E[X] _
5.4.
(a) E[X,,] =
A2re-Ar
.
E[XB] = 3
(b) E[min{XA, XB}] = s;
(c) Pr{X,, < XB) = 5;
(d) E[XB - XAjXA < XB] _ ;.
5.5.
(a) Pr{Naomi is last) = ;;
(b) Pr{Naomi is last) _ 2 = 0.1128;
.,j'
(c) c = 2 + N/-3.
Chapter II
1.1.
Pr{N=3,X=2} =,=6;
Pr{X=5} =48;
E[X] = '-.
1.2.
Pr(2 nickel heads N = 4}
1.3.
Pr{X > 1IX ? 1 } = 0.122184;
Pr { X > 1 lAce of spades) = 0.433513.
for
r > 0.
Answers to Exercises
606
1.4.
Pr{X = 21 = 0.2204.
1.5.
E[XI X is odd] = A eA
1.6.
Pr{U=u,Zz}p2(1-p)z,
A
Pr{U = uIZ = n} =
CA
+ e-A)'
1
n
1,
1
2.1.
Pr{Game ends in a 4} = o.
2.3.
Pr(Win) = 0.468984.
3.1.
k
0
1
2
3
4
5
6
Pr{Z = k}
0.16406
0.31250
0.25781
0.16667
0.07552
0.02083
0.00260
O
u:!:-: z;
O: u :5 n.
E[Z] =
;;
Var[Z] = 1.604167.
3.2.
E[Z] = Var[Z] =
8;
Pr{ Z = 2) = 0.29663.
3.3.
E[Z] = A2; Var[Z] = µ(1 + µ)c72.
3.4.
Pr{X = 2) = 0.2204;
E[X] = 2.92024.
3.5.
E[Z] = 6; Var[Z] = 26.
4.1.
Pr{X= 2} _;.
4.2.
Pr {System operates l = ;.
4.3.
Pr{U > ;) = 1 - ;(1 + log 2) = 0.1534.
4.4.
f,(z)
(1
+
for
z)2
0 < z < oo.
Answers to Exercises
607
5.1.
u > 0, v > 0.
for
4.5. f,.,,(u, v) = e-"""'
X
2
0.14
1
Pr{X > x}
0.61
0.37
2
1
1
- E[X]
x
5.2.
Pr{X ? 1 } = E[X] = p.
Chapter III
1.1.
0.
1.2.
0.12, 0.12.
1.3.
0.03.
1.4.
0.02, 0.02.
1.5.
0.025, 0.0075.
2.1.
(a)
P2 _
0.47
0.13
0.40
0.42
0.14
0.44
0.26
0.17
0.57
(b) 0.13.
(c) 0.16.
n
0
Pr{X,, =OIX)=O}
1
2.2.
2.3.
0.264, 0.254.
2.4.
0.35.
2.5.
0.27, 0.27.
1
2
3
0
'
'
4
;
2
4
s
Answers to Exercises
608
2.6.
0.42, 0.416.
-1
0
1
2
3
-1
0
0
0.3
0.3
0.4
0
0
0
0.3
0.3
0.4
1
0.3
0.3
0.4
0
0
2
0
0.3
0.3
0.4
0
3
0
0
0.3
0.3
0.4
3.1.
P=
3 2
.
.
(
NIP + N N l )q;
P'
(N_ i)
P'', =
(-)q.
-1
0
1
2
3
-1
0
0
0.1
0.4
0.5
0
0
0
0.1
0.4
0.5
0.1
0.4
0.5
0
0
2
0
0.1
0.4
0.5
0
3
0
0
0.1
0.4
0.5
3.3.
P=
1
-2
-1
0
1
2
3
0
0
0.2
0.3
0.4
0.1
0
0
0.2
0.3
0.4
0.1
0
0
0
0.2
0.3
0.4
0.1
1
0.2
0.3
0.4
0.1
0
0
2
0
0.2
0.3
0.4
0.1
0
31 1
0
0
0.2
0.3
0.4
0.1
3.4.
-12I
I
Answers to Exercises
1
2
1
0
2
'
0
'
2
0
1
0
0
3.5.
609
I
4.1.
v03 = I*
4.2.
(a) u,o=1;
4.3.
(a) u,o- = 1o05 .
-s
- ,o
(b) v, T-
4.4.
v0 = 6.
4.5.
uH,TT
4.6.
4.7.
= '3
6.
(a) uI0 = 23
(b)- '
vl = 23
W
- zo.
-
W. 2
= 2s
as
V, - ,,.
4.8. w = 1.290;w12=0.323
v,=1.613.
4.9.
u10
= zz = 0.40909
Y,o = 0.17;
P;o = 0.2658;
P;o = 0.35762 . . .;
P;0 = 0.40245
.
5.1.
0.71273.
5.2.
(a) 0.8044; 0.99999928
(b) 0.3578; 0.00288
.
Answers to Exercises
610
5.3.
p, =
0.42
0.58
0.49
0.51
0.474
0.447
P3 =
0.526
0.553
P4_
0.5422
0.5341
0.4578
0.4659
P5 = 0.53734
0.53977
5.4.
0.46266
0.46023
0
1
2
3
0
0
0
1
5.5.
PcO = 0.820022583.
5.6.
2.73.
5.7.
u,o = 0.3797468.
5.8.
po = a, ro = 1 - a;
pi = a(l - /3), q, = /3(1 - a),
r;=a(3+(1 -a)(1 -(3),
5.9.
Po= 1,qo=0,
p;=p,q;=q,r;=0
6.1.
for
i? 1.
for
i?1.
(a) u3s = ;
(p)2J/L1
(b) uss = I
1
-
- Ip
].
Answers to Exercises
611
6.2.
u,o=0.65.
6.3.
v = 2152.777
6.4.
v, = 2.1518987.
7.1.
w
(a) u10
20
25
7t
tJ
1o
40
I1
11
--22;
9 .
20.
(/b1
\u ) WII
11; W12
7.2.
70
79
100
w=
- 25
71
79
30
79
100
79
a) ulo 30
79;
- 100.
_ 70
(b) w - 79; W12 - 79
(.
8.1.
M(n) = 1, V(n) = n.
8.2.
p.=b+2c;a2=b+4c-(b+2c)2.
8.3.
n
1
2
3
UPI
0.5
0.625
0.695
4
0.742
8.4.
M(n)=A",V(n)=A"(1-' ,JIB 1.
9.1.
n
2
1
0.480
0.333
UPI
ux = 0.82387.
9.2.
cp(s) = Po +
9.3.
cp(s)=p+qs".
9.4.
(P(0)
1 - c4(0)
(P(S)
P2S2
3
0.564
4
0.619
5
0.775
5
0.658
Answers to Exercises
612
Chapter IV
1.1
.
IT0
10,
-?,
7T 1 =
5
21
6
, 1T,
31
16
66
66
-
_ 19
66,
1.3.
1T I1
1.4.
2.94697.
1G
.5.
790 - 2 9 , 7r1 - 2 9 7 T 2 = 29, 1T 3 - 29
1.6.
1TO
1.7.
- 40
- 135
L40
7r0-441,1
441,192-441,193-
1.8.
7r = 177.
1.9.
1r0
1'3
10
_9
5
S
14,1f1
=3
A,
2
- 14-
-
_
2
3
,
I
71 72
126
441
7
17
1.10. irlatc
40
2.1.
1T,. = 9.
2.2.
One facility: Pr (Idle) =
q2
1 + p2'
Two facilities: Pr(Idle) =
2.3.
(a)
p
0
0.02
AFI
0.10
0
0.11
AFI
0
0.20
AOQ
0
AOQ
(b)
2.4.
p
R,
R,
p
0.05
0.998
0.998
0.10
0.990
0.991
1
l+p+p2'
0.018
0.04
0.12
0.036
0.06
0.13
0.054
0.08
0.14
0.072
0.10
0.16
0.090
0.02
0.23
0.016
0.04
0.27
0.032
0.06
0.32
0.048
0.08
0.37
0.064
0.10
0.42
0.080
0.15
0.978
0.981
0.20
0.962
0.968
0.25
0.941
0.952
Answers to Exercises
2.5.
ITA=5.
2.6.
ao = .
2.7.
(a) 0.6831;
613
(b) ir,=ar,=-0°,ir,=,',;
(c) ?r3 = ,!,.
2.8.
3.1.
?r 3 = 8SI
{n ? 1; P`"" > 01 = 15, 8, 10, 13, 15, 16, 18, 20, 21, 23, 24, 25, 26,
28, ...)
d(0) = 1, P`57' = 0, P;38' > 0
3.2.
Transient states: (0, 1, 31.
Recurrent states: (2, 4, 5).
3.3.
(a) (0, 2), { 1, 3), 14, 5);
(b) (0), (5 ), ( 1, 2), (3, 4).
3.4.
{0), d = 1;
for all i, j.
{1},d=0;
{2,3,4,5},d= 1.
4.1.
lr,=pkl(1 +p+p-+p;+p4) for k=0,...,4.
4.2.
,4i9
(a) ?r0 - - 9999
8550
(b) m ,o10 - 1449
4.3.
Iro=Ir,=0.2,7,=7r,=0.3.
5.1.
lim P`"" = lim P`,"o' = 0.4;
limP;o'= limPo'= 0;
lim Pa'o = 0.4.
5.2.
(a)
(b) 0,
(c)
3I,
(d) 9,
(e) 3,
(f) X,
(g) 5,
(h) 27,
Answers to Exercises
614
Chapter V
1.1.
(a)
a-22;
(b) e2.
k =0, 1, ... .
1.2.
(pklpk-1) = A/k,
1.3.
Pr{X = kI N = n} _ ()p*(1
0,1,...;
"e a1.4.
(a) ( )
a
p= a+/3
- p)"kAt
k
k!
(b) Pr{X(t) = n + kIX(s) = n) =
Wt - s)]
E[X(t)X(s)] = Alts + As.
k!
1.5.
Pr{X = k} = (1 - p)pk for k = 0, 1, ... where p = 1/(1 + 0).
1.6.
(a) a-12;
(b) Exponential, parameter A
1.7.
= 3.
(a) 2e-2;
(b) 64 e -6.,
(6,)(1)2(2)4;
(c)
3
3
(d) 32e-4
1.8.
(a)
5e-22;
(b) 4e-4;
1 - 3e-2
(c) 1 - e-2
1.9.
2.1.
(a) 4;
(b) 6;
(c) 10.
k
0
1
(a)
0.290
0.296
0.370
0.366
0.301
0.361
(b)
(c)
2
0.225
0.221
0.217
Answers to Exercises
615
2.2.
Law of rare events, e.g. (a) Many potential customers who could
enter store, small probability for each to actually enter.
2.3.
The number of distinct pairs is large; the probability of any particular pair being in sample is small.
2.4.
Pr{Three pa9f s error free) - e-''-.
3.1.
a-6.
3.2.
(a) e-' - e-";
(b) 4e-'.
3.3.
;.
3.4.
()()'(3)' = xs
TI
m=0,1,
3.5.
3.6.
F(t) = (1
3.7.
t+
2.
A
3.8.
(i5)(2)S( )'.
I
3.9.
Pr{W,.<_t}=1-
4.1.
n+l
4.2.
4.
4.3.
.
7,
4.4.
See equation (4.7).
4.5.
[1-
enl5
a
JI
(At)Ae-Ar
k!
,n.
Answers to Exercises
616
5.1.
0.9380.
5.2.
0.05216.
5.3.
0.1548.
6.1.
0.0205.
6.2.
Mean=
t, Variance =
2At
02
e-AC(z)t - e- At
6.3.
6.4.
1 - e-At
(a) 1;
(b) zr
I-II
A(t)ke-ACn
6.5.
Pr{M(t) = k} =
where A(t) =A
k!
J
[1 - G(u)] du.
0
Chapter VI
Po(t) = e-';
P1e-' - I -3t.
,
I(t) = z
ze
3[1e-' +
Ze-3r - 1e-2t]
P2(t) =
P3(t) =
1.2.
6[8e-t +
;e-3t -
3e-zr -
z4e-St].
(a) 6i
(b) 2s.
6+
36
C
1.3.
36-
X(t) is a Markov process (memoryless property of exponential dis-
tribution) for which the sojourn time in state k is exponentially
distributed with parameter Ak.
1.4.
(a) Pr {X = 01 = (1 - ah)" = 1 -nah + o(h);
(b) Pr{X = 1) = nah(1 - ah)" = nah + o(h);
(c) Pr{X ? 2) = 1 - Pr{X = 0) - Pr(X - 11 = o(h).
617
Answers to Exercises
-, Var[X(t)] =
1.5.
E[X(t)] _
1.6.
(a) P,(t) = e-S;
2
P
P
where P = e
5[Ze-3r - lie-5']
(b) P2(t) =
i6e-5' +
15[zoe-3r -
(c) P3(t) =
2.1.
1
-Le-13r].
80
P3(t) = e-Sr;
P2(t) =
[3e-2r -
PI(t) =
3e-S7;
;e-2r -
10[be-5r +
Ze-3r]
Pa(t) = 1 - P,(t) - NO - NO2.2.
(a) io;
(b) 29'
15+
(c)
361
900
2.3.
P3(t)=e ;
P2(t) = [e_, - e-2,];
e_' - e-2r + 1e-31];
pi(t) = 2[i
Po(t) = 1 - PI(t) - NO - P3(t).
2.4.
P2(t) = 10e-41(l - e-2')3
3.1.
3.2.
n=0,1,....
for
Assume that Pr{ Particular patient exits in [t, t + h)lk patients }
1h + o(h).
Mk
4.2.
ar.k =-
4.3.
1k
4 .4 .
(a)
(Npk(l
Ik
=
where p =
P)"-k
where
A
-a
aro=1/(1+µ+2(µ)I;
(b) am =
4.5.
-
1/(1 +
(-)2).
+
ark = (k + 1)(1 - 0)29'`
a
a+
Answers to Exercises
618
A
9Ae-o
0 = -.
where
4.6.
7r, =
5.1.
Use log
5.2.
Use
6.1.
1
6.2.
P,(t) =
A+µ + A+µ
7.1.
k!
1x
1
K
K-i
f(t) =
2
=1
µ
= 1 + x + x2 +
x < 1.
K >> i.
for
- e a+\2u +
e-(A+µ)r1
J
2+
e-(2-'2),
4
4
-e --(2-\ 2)1 +
7.2.
4
°
- 2 -4
_
4
e-c-\2)t -
2+
e-(,+v2,r
4
Chapter VII
1.1.
The age S, of the item in service at time t cannot be greater than t.
1.2.
F,(t) = 1 - e-A, - Ate-A,
1.3.
(a) True;
(b) False;
(c) False.
1.4.
(a) W has a Poisson distribution with parameter Ak.
I
e-"A(Ak)"
n=0
n!
(b) Pr(N(t) = k} _
2.1.
a + b.
e-A(A+I)(Ak + A)
,
n!
619
Answers to Exercises
2.2.
The system starts from an identical condition at those instants
when first ;both components are OFF.
2.3.
M(n)
n
u(n)
1
0.4
0.4
2
0.66
1.104
1.6276
2.0394
2.4417
2.8897
3.3374
3.7643
4.1947
0.26
0.444
0.5236
0.41184
0.4023
0.44798
0.44769
0.42693
0.4304
3
4
5
6
7
8
9
10
1
3.1.
a-";t +
A
3.2.
The inter-recording time is a random sum with a geometric number
of exponential terms, and is exponential with parameter Ap. (See
the example following II, (3.13)).
3.3.
Pr f N(t)
= , WN(,) , > t +
= {_(At)"e'
n!
S)
le-As.
4.1.
M(t) = Zt - 6.
4.2.
8.
4.3.
T* =
4.4.
c(T) is a decreasing function of T.
4.5.
h(x) = 2(1 - x) for 0 : x < 1.
4.6.
(a)
8
a
a+p
;
- 15
Answers to Exercises
620
/LA
5 .1 .
1
5.2.
9
5.3.
T* = cc.
6.1.-6.2.
n
0
1
1.1111
2
0.96296
1.01235
0.99588
1.00137
0.99954
1.00015
0.99995
1.00002
0.99999
3
4
5
6
7
8
9
10
6.3.
0.6667
1.33333
0.88888
1.03704
0.98765
1.00412
0.99863
1.00046
0.99985
1.00005
0.99998
1.00001
E[X] = 1; 1 bk = 1.
Chapter VIII
1.1.
(a) 0.8413.
(b) 4.846.
1.2.
Cov[W(s), W(t)] = min{s, t}.
1.3.
ap = ',_cp(z)t-'[z2 - 1];
ax
= r zcp(z).
Answers to Exercises
1.4.
621
(a) 0.
(b) 3u2 + 3iuv + uw.
1.5.
(a) a-'-`;
(b) t(1 -s)
for
0 < t < s < 1;
(c) min{s, t}.
1.6.
(a) Normal, .t = 0,
U-z
= 4u + v;
(b) Normal, A=0,o-'-=9u+4v+w.
S
1.7.
S
2.1.
(a) 0.6826.
(b) 4.935.
2.2.
If tan 0 = V, then cos 0 = t/(.
2.3.
0.90.
2.4.
Reflection principle: Pr(M,, ? a, S,, < a) _
Pr{M ? a, S,, > a}(= Pr(S,, > a)).
Also, Pr{M ? a, S = a) = Pr(S = a).
2.5.
Pr(,r0<t) =Pr{B(u) 00
2.6.
Pr{z, <t} = Pr(B(u) = 0
3.1.
Pr(R(t) < yIR(0) = x)
forallt<-u<a} = 1 - t9(t,a).
for some u, b < u < t).
= Pr(-y < B(t) < yIB(0) = x}
= Pr{B(t) < yIB(0) = x} - Pr{B(t) < -yIB(0) = x).
3.2.
0.3174, 0.15735.
3.3.
0.83995.
3.4.
0.13534.
3.5.
No, No.
Answers to Exercises
622
4.1.
0.05.
4.2.
0.5125, 0.6225, 0.9933.
4.3.
0.25, 24.5, 986.6.
4.4.
r = 43.3
4.5.
0.3325.
versus
E[T] = 21.2
x
4.6.
(a) E[( - a)+] = J xcp(x) dx - a f cp(x) dx
a
a
= cp(a) - a[ I - c1(a)]
(b) (X - b)+ = v(
-
b
5.1.
0.8643, 0.7389, 0.7357.
5.2.
0.03144, 0.4602, 0.4920.
5.3.
(a)
Q
µ)+.
(,1 - S)"v
CoV[Vro i+k] _ (1 - 6)k.
(b) E[# V,, = v] _ -(3v
Var[OVIV = v] = 1.
Chapter IX
1.1.
(a) Probability waiting planes exceed available air space.
(b) Mean number of cars in lot.
1.2.
The standard deviation of the exponential distribution equals the
mean.
1.3.
11 days.
2.1.
25
36
Answers to Exercises
623
L = 5.
versus
2.3.
L = 1
3.1.
,
v=-,z-=-,
L
3.2.
L= p+
3.3.
3.
3.4.
;-' versus 3.
3.5.
M(t) = A
1
1
_
p
1-p
p 2(1 + a)
2a(1 - p)
J
[1 - G(y)] dy - Av.
0
4.1.
8 per hour.
4.2.
16
4.3.
8.55.
4.4.
0.0647.
5.1.
15(4)
5.2.
Pr{Z :5 201 = 0.9520
5
Index
Absorbed Brownian motion 499-502, 507
Absorbing Markov chain 123-125
Absorbing state 105, 120, 170
in birth and death process 379-384
Absorption probability 165
Absorption time 173
in birth and death process 382-384
Accounts receivable model 167, 230
Addition law 6
Age 45, 422, 424, 425, 426
in a Poisson process 434
in a renewal process 146
limiting distribution of 443
Age replacement 45, 221-223, 438, 446
Airline reservation model 234
Aperiodic 239, 246
Arrival process 590
Astronomy application 314
Availability 217, 228, 234, 403
Average fraction inspected 220, 228
Average outgoing quality 220, 228
Backward equations 359
Bacteria 30, 289, 315
Balking 567, 581
Basic limit theorem of Markov chain 246
Batch processing 150-151, 296
Bernoulli distribution 24
Bernoulli random variables sum of 280, 289
Beta distribution 38, 83
Beta function 55
Bid model 69, 86, 149, 331
Binary message 100, 104
Binomial distribution 25, 293
convolution of 32
Binomial expansion 341
Binomial theorem 55
Bird problem 437
Birth and death process 547
limiting distribution of 366-368
postulates for 355-356
with absorbing states 379-384
Birth process 333-339
linear 346
Black-Scholes 516-521
Block replacement 428-430, 432, 456
Borel set 19
Branching process 91, 177-195
multiple 192-195
Brand switching model 229
Breakdown rule 350, 354
Brownian bridge 502, 522
Brownian meander 504
Brownian measure 534
Brownian motion 474-479
absorbed 499-502
geometric 514-516
integrated 535
maximum of 491-493, 513
of pollen grain 473, 526
reflected 498-499
standard 478
two dimensional 506
with drift 508-511
Bum-in 403-405
Busy period 551
Index
626
Cable fatigue model 349-353, 354
Cable strength model 483
Call option 516, 523, 531
Car problem 258
Carrying capacity 386, 390
Cash management 162-164, 488
Central limit principle 483, 508, 533
Central limit theorem 34, 75, 479
Certain event 6
Chapman-Kolmogorov equation 356, 394
Chebyshev inequality 93
Chemical reaction 354, 378
Cigar problem 134-135
Collards 365
Communicate 235, 258
Compound Poisson process 318-320
Compression testing model 490
Computer challenge 135, 168-169, 176,
290, 311, 490
Computer coding 288
Conditional
density function 71, 79
dependence 85
distribution function 71
expectation 81
expected value 60
independence 85
mass function 57
probability 14
Continuous random variable 8
Continuous sampling 218-221
Control limit rule 225
Convolution 12
of binomial distribution 32
of geometric distribution 32
of Poisson 268
Correlation coefficient 12, 22, 39, 85
Counter model 427, 436
Covariance 11, 86
matrix 481
Covariance function 482
for Brownian bridge 507
for Brownian motion 479
of Ornstein-Uhlenbeck process 525
Cox process 273-274, 408-411
Crack failure model 64-68, 324, 343
Craps 148
Cumulative process 449, 451
Current life, See age
Customer waiting time 542
Damage 134
Death process 345-349
linear 346-349
Defects 274, 316
Delayed renewal process 447
Departure process 582, 590
Detector model 310
Dice
five before seven 134
four before seven 65, 68
shaved 66-68
weighted 69-70
Differential equations 335, 337, 344, 415,
523, 557
and limiting distribution 367
backward 359
for Cox process 409-410
for M(t) 361
for two state Markov chain 363
forward 361, 367
Kolmogorov 359, 361
Diffusion equation 475
Discount factor 132, 300
Discrete random variable 8
Disease spread 99, 342
Disjoint 6
Dispatch model 278
Dispersal of offspring 183
Distribution function 7
Do men have more sisters 64
Doubly stochastic 206, 260
Poisson process 273
Ehrenfest urn model 108, 532-533, 539
Electron multiplier model 78, 178
Elementary renewal theorem 437
Embedded Markov chain 396, 560, 563
Index
Embedded random walk 380
Empirical distribution function 503
Equal load sharing 351, 483
Equally likely outcomes 3
Excess life 115, 414, 422, 424, 425, 426,
462
for Poisson process 433
limiting distribution of 443
Expected value 9
of sum 12
Exponential distribution 35-37, 46-50
Extinction 345, 347, 355, 388-392
mean time to 385, 390-393
Extinction probability 181-183, 384, 393
and generating function 187-190
Failure 446
crack model for 324
Failure model 115
Failure rate 36, 41, 350
Fair game 88, 143
Family name survival 179
Family size 184, 195, 197
Fatigue 324, 343, 349-353
Fecundity model 125-127
Feedback 569, 592
Fiber, See filament
Fibrous composite 316
Filament, strength of 326-328, 411-413
First come-first served 542, 239
First return 240, 245
First return
mean 255, 257
First step analysis 101, 116-125, 169-174,
255,261-262,360,380
infinitesimal 508, 509, 510
Five before seven 134
Flashlight problem 52-53
Flea beetle model 365, 557
Forward equations 361, 367, 376, 409
Four before seven 65, 68
Fundamental matrix 172, 175
Gauge length 326
627
Gambler 90-91
Gambler's ruin 94, 142, 144, 148, 152,
156, 509
mean duration of 167
Gambling 22, 88, 94
Game show 150
Gamma distribution 38, 276, 291
Gamma function 53
Gaussian process 481, 506
Gene frequency model 108-111, 179,
372-376,528-529
Generating function 185, 276
and extinction probability 187-190
and sums of rv's. 185, 190-192
Geometric Brownian motion 514-516
Geometric distribution 25-26, 33, 339
convolution of 32
generalized 33
Geometric series 56
Hazard rate 36, 350
History 215-216
Hitting probability 173
Hitting time 152, 493, 509
mean of 165
Home helicopter 342
Idle period 551
Immigration 342, 361, 368
Impossible event 6
Impulse response function 304
Imputed volatility 520
Independent events 7
Independent increments 270, 477
Independent Poisson processes 321
Independent random variables 11
defining a Markov chain 138-141
Indicator
function 45
of event 24
random variable 21
Industrial mobility 398-402
Infinitesimal first step analysis 508, 509,
510
Index
628
Infinitesimal generator 356, 396
Infinitesimal matrix 359
Infinitesimal parameter 339
Infinitesimal probability 333, 334, 395
Input process 541
Input rate 570
Integral equation 338
Integrated Brownian motion 535
Intersection 6
Invariance principle 480, 482, 488
Inventory model 106-108, 112-113, 114,
115, 230
Irreducible 246, 258
Irreducible Markov chain 236
Joint distribution I I
L = A W 542-544
Laplace transform 311, 414
Last step analysis 360, 409
Law of large numbers 4
Law of rare events 27, 28, 279-286, 288
Law of total probability 6-7, 14, 49, 58,
60, 65, 71, 72, 82, 87, 94, 101, 119,
120, 121, 268, 335, 409, 458, 461,
559
Lazy professor 457
Learning model 297, 434
Length biased sampling 436
Limiting distribution 199, 254, 257, 377
and differential equations 367
for birth and death process 466-468
interpretation of 207
of age and excess life 443-445
Limiting transition probability 212, 232
Linear birth process, See Yule process
Linear death process 346-349, 353, 355
Linear growth 346
with immigration 361, 368
Loaded dice 69-70
Logistic process 371-372
Lognormal distribution 35
Long run relative frequency 4
M/G/l queue 545, 558-563
M/G/c queue 564-565
M/M/l queue 544, 548-552
M/M/S queue 553-555, 556
M/M/e queue 544,552-553,557
Marginal distribution 1 I
Marked Poisson process 321-324, 564
Markov chain 95
embedded 396
in continuous time 394-396
two state 136-138, 200, 362-364,
396-398,407,431
Markov inequality 88-89, 93
Markov process 95
Marlene, see also Martha 470
Martha, see also Marlene 151
Martingale 87-94, 151, 184, 297, 490,
507, 524
systems theorems 88
Match 21
Maximal inequality 89-90
Maximum of Brownian motion 491-493,
513
Maze 122-123, 131, 166
Mean 9
Measurable 17
Median 9, 40
Memoryless property 35, 48, 348, 585,
596
Mode 31
Moment 9
Multinomial distribution 29-30
Naomi 51
Negative binomial distribution 25-26, 59,
84
Neutron chain reaction 178
Normal distribution 34
bivariate 39
conditional 86
multivariate 40, 481
notation for 475
table of 486
Null recurrent 246
o(.) notation 9, 271-272, 276-277, 334,
344
Offspring 73
Index
Optimal replacement model 224-228
Option pricing 516-521
Ornstein-Uhlenbeck process 524-538
and Brownian motion 525, 537
stationary 532
Overflow 545, 570, 580
Partial sums 140-141, 176, 480
Pattern search 114, 130
Period of state 237
Periodic 235, 254
Peter principle 398-402, 450
Planned replacement, See Age replacement
Poisson
decomposition 268-269
sum of 268
Poisson counting process 290
Poisson distribution 26-29, 196, 258,
267-268
and uniform distribution 63
convolution of 31
Poisson point process 290
definition 284
nonhomogeneous 323
postulates for 282
Poisson process 270-273
as a renewal process 427, 432-435
compound 318-320
doubly stochastic 273
marked 321-324, 564
mixed 273, 278
nonhomogeneous 271-273
postulates for 334, 344
repositioning of 329
spatial 311-314
with Markov intensity 408-411
Population size model 70, 335, 339-341,
341,346-349,371-372,384
with age structure 463-469
Population stability 193
Position process 530, 536
Positive recurrent 246
Preemptive priority 572-579, 580
Premium 517
629
Priority, preemptive 572-579, 580
Priority queue 545
Probability density function 8
Probability mass function 8
Probability measure 15
Probability of absorption 165
Probability of extinction 181-183
Probability space 16
Process control 453
Product form 592
Production process 30, 100, 104, 148, 213,
218-221, 231
Pure birth process, See birth process
Pure death process, See death process
Queue discipline 531
Queueing model 70, 114, 450
discrete 111-112, 149
Queueing network 545
general open 592-598
open acyclic 581-591
Quiz show 84
Radioactive decay 294, 301, 364
alpha particles 329
Random environment 84, 70-77
Random sum 180
distribution of 74-75
moments of 72
Random variable 7
continuous 8
discrete 8
Random walk 141-145, 151, 156-161,
162, 480, 488, 489, 496
simple 145, 177
Random walk hypothesis 318
Recurrent 246
Recurrent state 240, 241
Reducible Markov chain 258-264
Redundancy 216-218, 403-405
Reflected Brownian motion 498-499
Reflection principle 491-493, 497
Regular Markov chain 199
Regular transition matrix 199-205, 257
Reliability model 216-218, 228, 234
630
Reneging 581
Renewal argument 458
Renewal equation 458
Renewal function 421
for Poisson process 433
Renewal process 146
definition 419
delayed 447-448
in Markov chain 428
in queues 427
stationary 448
Renewal theorem 441, 462, 551, 558
elementary 439
Repair model 407
Repairman model 369-371, 376, 377, 545,
546
Replacement model 150, 221-228, 438,
450, 452
optimal 224-228
Reservoir model 496, 506
Risk theory 70, 318, 452-453
Rumor spread 134
or-algebra 16
Sample space 15
Sampling
continuous 218-221
length biased 434, 436
sum quota 305-308, 435
wildlife 71
Sequential decisions 511-513, 521, 522
Sequential hypothesis test 513
Server utilization 542
Service distribution 531
Shaved dice 66-68
Shewart control chart 453
Shock model 277, 278, 319, 328, 329
Shot noise 304-305, 310, 311
Sigma-algebra 16
Size effect 325
Social mobility 200, 210
Sojourn 336, 357-359, 395, 447
in Poisson process 290, 292
Spatial point process 311-314
Index
Spread
of disease 99, 342
of rumor 134
Stable age distribution 469
State space 5
Stationary
Cox process 414
distribution 247, 252, 256, 258, 366,
376-378
intensity 409
Ornstein-Uhlenbeck process 532
renewal process 448-449
transition probability 96
Sterile male insect control 388-392
Stirling's formula 54
Stochastic 2
Stochastic process 5
Stock prices 75-77, 88, 318, 474, 489,
506, 510, 514, 522, 531
Stop and go traveler 366, 416-417
Strength
compression 490
of cable 483
of filaments 326-328
Striking price 517
Subjective probability 5
Success runs 145-147, 165-166
Successive maxima 138-140
Sum quota sampling 305-308, 435
Sums of numbers 56
System 50-51, 231, 378, 379, 406, 407,
431,446, 456
System throughput 542
Tail probability 43-46, 52, 422
Tandem queues 583
Tensile strength 411-413
Time reversal 588-591
Total life 422, 457
for Poisson process 434
Tracking error 527
Traffic flow 427
Traffic intensity 549, 563, 573
Transient state 120, 170, 235, 240, 241
Index
Transition probability 95
for two state Markov
limit of 212, 232
matrix 96
n-step 101
regular 199-205
stationary 96
631
363-364
Waiting time
in Poisson process 290, 296, 309
in renewal process 420
Wald's approximation 513
Warrant 516
Wear 134
Weather model 215-216, 232, 238,
455
Uncorrelated II
Uniform distribution 37, 93, 290, 311
and 1 sisson process 297-299
discrete 30
Union 6
Weibull distribution 486
Weighted dice 69-70
Weiner process 474
Wildlife sampling 71
Urn 63, 91-93, 108, 113, 115, 132, 134,
168-169, 211
You Be the Judge 28-29
Yule process 339-341, 342, 346
with immigration 342
Variable service rates 569
Variance 9
Velocity 538
Volatility 520
Zero-seeking device 130, 175
Zeros of Brownian motion 494-496,
497