Journal
and Stochastic Analysis, 10:4
of Applied Mathematics
(1997), 423-430.
A HIDDEN MARKOV MODEL FOR AN INVENTORY
SYSTEM WITH PERISHABLE ITEMS
L. AGGOUN, L. BENKHEROUF and L. TADJ
King Saud University, Dept. of Statistics and Operations Research
P.O. Box 255, Riyadh 1151, Saudi Arabia
(Received April, 1996; Revised May, 1997)
This paper deals with a parametric multi-period integer-valued inventory
model for perishable items. Each item in the stock perishes in a given period of time with some probability. Demands are assumed to be random.
and the probability that an item perishes is not known with certainty. Expressions for various parameter estimates of the model are established and
the. problem of finding an optimal replenishment schedule is formulated as
an optimal stochastic control problem.
Key words: Inventory Control, Perishable Items, Markov Models,
Measure Chain Techniques.
AMS subject classifications: 60K30, 60J10, 90B05.
1. Introduction
This paper deals with a parametric multi-period integer valued inventory model for
perishable items. Each item in the stock is assumed to perish in period n with
probability (1-an) where 0 < a n < 1. The sequence {an} is not known with
certainty but it is known to belong to one of a finite set of states of a Markov chain
to be specified below. The above model is inspired from a special type of time series
models called First Order Integer-Valued Autoregressive process (INAR(1)). These
models were introduced independently by hl-Osh and il-Zaid [1] and McKenzie [6]
for modeling counting processes consisting of dependent variables.
Let X be a nonnegative integer-valued random variable. Then for any a E (0, 1),
define the operator o by:
x
ao
E Yi,
X
(1.1)
i=1
where Y is a sequence of i.i.d, random variables, independent of X, such that:
P(Yi O) a.
0, + 1, + 2,...} is given by:
1)
P(Yi
Then the
INAR(1)
process
{X,:n
Xn
1-
ao
Xn
1
at- Vn,
(1.2)
(1.3)
where V n is a sequence of uncorrelated nonnegative integer-valued random variables
Printed in the U.S.A.
()1997 by North
Atlantic Science Publishing Company
423
L. AGGOUN, L. BENKHEROUF and L. TADJ
424
with mean #n and variance r 2
One possible application of model (1.3) is in its monitoring the number of
patients in a hospital. Here the number X n of patients at epoch n is composed of
patients left from the previous epoch (each patient stays in the hospital with probability a or leaves with probability (1- a)), and a new set V n of arriving patients.
Now let X n represent the number of items at the end of period n in our inventory and consider the following extension of (1.3):
x=oX_-v+u,
(1.4)
x
with X 0 constant (integer). Here
an Xn_ 1
o
E Y as defined in (1.1) for non-
i--1
negative integer random variables X n_ 1 and we allow X n_ 1 to take on negative
values in which case a n o X n 1 0. Note that a negative X n is interpreted as shortage. V n is a sequence of 7/----valued independent random variables with probability
The variable U n is a 7/+-valued sequence
and independent of
distribution
representing the replenishment quantity at time n (which later on will be considered
as a control variable). Further we assume that the sequence {an} is a homogeneous
Markov chain with finite state space S. For instance, {an} expresses changes in a
caused by changes in the environment (temperature, humidity, air pressure, etc.) at
different epochs n.
=-
en
Y.
The objective of this paper is to:
find the conditional probability distribution of {an} given the information
accumulated about the level of inventory {Xn} up to time n,
(ii) estimate the transition probabilities of the Markov chain {an}
(iii) formulate the optimal stochastic control problem related to (1.4).
The proposed model is an example of a hidden Markov model. Our approach
shall use measure change techniques (see Elliot, Aggoun, and Moore [5]). A reference
probability measure is introduced under which processes of interest are independent of
each other. This greatly simplifies computations and allows derivations of interesting
results.
Note that in analogy with engineering applications of hidden Markov models, we
may think of {an} as a signal hidden in some noisy environment and can only be observed through noisy observations or measurements given by {Xn}.
Most of existing inventory models for perishable items in the literature are concerned with determining suitable ordering policies that optimize some of their performance measures. According to Nahmias [7], all perishable models fall in two main
(i)
categories:
(a)
(b)
fixed life perishability models,
random lifetime models.
Our model is closely related to (b). However, due to the complexity inherent to
the analysis of models where units have a random lifetime, very little progress has
been made on that front, apart from suggesting equivalent deterministic or using
queueing models for their analysis. Our paper proposes a new model for perishable
items which we believe is useful. For related deterministic models see Benkherouf
and Mahmoud [2] and Benkherouf [3].
The paper is organized as follows" In Section 2 recursive unnormalized conditional distributions of {an} given the information accumulated about the inventory level-
A Hidden Markov Model for
an
Inventory System with Perishable Items
425
surviving items processes are derived. Section 3 contains estimates of the probability
transitions of the Markov chain {an}. Finally, in Section 4, an optimal control formulation for finding the replenishment schedule is proposed. The last section contains
a summary and some open problems.
2. Conditional Probability Distribution of
{an}
Let I1,...,I M be a partition of the interval (0,1). Also, let 81 be any point in I1, s 2
be any point in I2,...,s M be any point in IM, and S- {81,...,8M}.
Suppose that the sequence {an} is a homogeneous Markov chain with state space
S describing the evolution of the parameter c n of the inventory model.
All random variables are initially defined on a probability space ([2,,P). For
1_<i, j_<M, write
Pij- P(n G I iloz n 1 E I j)
P(ctn-- 8ilOn-1-- 8j),
and
M
P-
(Pij),
Pij- 1.
i-1
lecall that X n represents the inventory level at time n. Now let
n
cr{Xk, Y,k <- n,i >_ 1),
G, cr{Xk, Y,ck,l <_ n,i >_ 1},
(2.2)
(2.3)
be the complete filtrations generated by the parameter process {an} the inventory
level-surviving items processes {Xn, Y]}, and the inventory level-surviving items and
the parameter processes respectively.
Note that A n and
C G n for n- 1,
Following the techniques of Elliot,
Aggoun, and Moore [5], we introduce a new probability measure P under which {c}
is a sequence of i.i.d, random variables uniformly distributed on the set S{Sl,...,SM} and {Xn} is a sequence of independent -valued random variables with
probability distributions Cn independent of {a}. For this, define the factors"
n
0
An
(Ma) I<n
1,
Si) n(Xn
n o X n --1 Un)
.(x.)
where I( si)is the indicator function of the event (a si) and
(2.4)
M
a.
j=l
Remark 1: Note
that
I( n
si)- a n"
Now define
that. {a}
A
is an
is
An_ 1
adapted and
E[I( -si)[A_l]-an,
A-martingale increment.
k=0
so
L. AGGOUN, L. BENKHEROUF and L. TADJ
426
Then {An} is a Gn-martingale such that E[An]- 1. Here E denotes the expectation
under P. We can define a new probability measure P on (f2,
Gn) by setting
an
dP
The existence of
on
h=lP,Gn){Xn}
(a
is due to Kolmogorov’s extension theorem
e.g., Chung [4]). Then under
with probability distributions
uniformly distributed over S.
(see,
is a sequence of independent random variables
is a sequence of i.i.d, random variables
Cn
and
{c%}
Pn(Si): P(an
is given by (2.2), and
si
r)
Write
where
si) Yn},
E{I(cn
:{I(.- s)a- }.
q.(si)"
(2.8)
(2.9)
Here E denotes the expectation under P and A n is given by (2.6).
Theorem 1: For n >_ 1, we have
q.()
Pn(8i )-
M
and
k=l
qn(si)
n(Xn
Xn
sid
1
Un)
M
E Pijqn- 1(8j)"
-I
Proofi Recall that dP
Now using
a
(2.11)
A n is equivalent to dP
A 1.
d
an
version of Bayes theorem (see Elliot, Aggoun, and Moore [5]),
an
we
can write:
p.(s)
E{(.
,) .}
q(i)
M
E
(2.12)
q()
k=l
Working with P and using
(2.4), (2.5), (2.6), and (2.9),
(x
qn(i)
o
x
it is easily seen that
M
Pijqn
(x)
(2.13)
as required.
It is worth noting at this stage tha if model (1.4) is replaced by
Xn n- o X n
Vn + Un,
then
(2.11) becomes:
qn(Si)
Now, if 0
f
n(--
(2.14)
M
6n(Xn
sj X n_ l
Un)pijqn_ l(sj)"
(2.15)
j
is fixed, then using
(2.11)
can be obtained using the relation
the first updated conditional distribution of %
A Hidden Markov Model for
an
Inventory System with Perishable Items
l(Xl
ql(Si)
8 o
1 (Xl)
with
P(l
si] 1)
X 0 U1)
M
ql(Sk)
k=l
Further updates follow by using
M
E
Pi, c O,
1(X1
ql(Si)
Pl(Si)
1(X1
k=l
(2.11).
427
si o X 0
sk o
U 1)pi, a
0
X 0 -U1)Pk,
co
3. Estimation of the Transition Probabilities Pij
were assumed to be known. However, in practice
seldom know Pij exactly and consequently, estimates of the transition
probabilities would be reasonable requirements. These estimates are calculated each
time new information regarding the behavior of the system becomes available. In
this model, the state space S-{Sl,...,SM} is fixed a priori. This is done by
partitioning the open unit interval into M subintervals I1,...,I M according to some
prior information about the behavior of a. When, at time n, the new information is
made available through Yn-measurable estimates {ij(n)}, a new partition of I can
replace the initial one updating the state space of {an}"
To replace, at time n, the parameter {pij(n)} by {ij(n)} in the Markov chain
In Section 2, the probabilities Pij
we
{an},
define
n
Xn-
M
M
k=l i=1 j
and set
1
(ij())l(akli)l(ak-llj)
Pij
dP
dPG n
_Xn
(3.1)
(3.2)
Again, the existence of P is due to Kolmogorov’s extension theorem.
It is easy to see that under P, the Markov chain {n} has transition probabilities
given by {ij(n)}.
Theorem 2: The new estimates {ij(n)} at lime n of lhe probability transitions
given lhe observalion of the inventory level from lime 0 to lime n are given by"
ij ()-
(3.3)
E1
where
v(X)
and
7.(N#)
M
J{A- 1X },
Nn
counts the number of transitions from slate j to state
Proof: Note that from (3.1),
LgXn-
n
M
M
up to time n.
E E E I(Ck e Ii)I(Crk_ e Ij){Logij(n -Logpij }
E E NiLgij(n) + R(pij)’
1
k=li=lj=l
M
M
=1 j=l
L. AGGOUN, L. BENKHEROUF and L. TADJ
428
where R(Pij does not contain
Conditioning on fin,
ij(n)terms.
M
M
E E E(NJni
E(Log X n [fin)
f n )Lg’ij (n) + E{R(PiJ)
V,}"
(3.4)
i=1j=1
We require that
M
i=1
We wish to choose
{ij(n)}
that maximizes
(3.4)
subject to
iary problem:
Consider the auxil-
M
L(ij(n),A
IVy) + (
E(Log
L(ij(n),
Maximizing
(3.5).
ij(n)- 1).
(3.6)
gives
E(NJni f n) +A-O,
M
Fij(n)
1.
i--1
Hence,
Next,
we derive a recursive equation for the unnormalized conditional probability
we derive first recurHowever, in order to obtain a recursion for
sive equations for the process {I(c n
intermediate
a
as
step"
necessary
}
7n(NJni).
7n(NJni),
sl)NJni
sl)N jin A n-1 fin):
E {l(a n
Note that
7n(NJ)
7n{l(an sl)NJni)"
(3.7)
M
E n{I(cn
Sl)NJni}"
/=1
>_ 1,
Theorem 3: For n
"n{I(an- sl)NJi }
Cn(Xn on o Xn
l
Un)
[rn=l?n
{ I(cn- 1
sm)Nn’- 1}P/m + q(nj)
lPljPij
where qn is given recursively by (2.11).
Proof: The proof follows if we note that
NJn
NJin- 1 + I(cn- 1
sj)I(c n si)
and use Remark 1, Section 2.
The revised pard.meters {’ij(n)} give new probability measures for the model.
The quantities 7n(Na) can then be reestimated using the new probabilities.
A Hidden Markov Model for
an
Inventory System with Perishable Items
429
4. Optimal Control
In this section we formulate an optimal control problem for model (1.4). However,
{an}
is not fully observed. Using results of Section 2 we transform the problem into
fully observed one. Dynamic programming results and minimum principle are obtained in terms of separated controls (i.e., controls which depend on the Markov
chain only through qn)"
Recall from (1.4) that our inventory model has the form:
a
Xn
cn o
X n- 1
Vn + Un"
(4.1)
Assume that at each time n, 0 <_ n <_ T, the control U n takes on values in some finite
subset
of 7/+ and that U n is 5n-measurable where 5 n is given by (2.2). Also
,
assume the existence of bounded measurable functions
For each admissible control Upected costs as:
(Uo,...,Un)E
cn, dn, and fn.
we define the associated ex-
T
J(Xo, U)
E[
{cn(Xn) + dn(Un) + f n(n)}].
(4.2)
T
[1
{c(X) + e(v) +/())]
n:0
{cn(Xn + dn(Un)}[A
1
in] +
n=0
fn(Sk)[i(n Sk)A
1
n]
k=l
{c(X) + e(u)}
n=0
()+
(4.a)
f()()
k=l
/=1
qn(" are given by Theorem 1.
So, the expected cost is expressed in terms of the conditional
measures
The value function for this control problem is as follows for 0
_< t _< T"
where
Vt(qIc, inf UT) 7
qn(l) +
{cn(Xn) -t- dn(Un)}
n
1
f n(k)qn(k)
k
qn(" )"
qt
q
1
inf
Vt(q, U).
(u
ur)
The following dynamic programming result can easily be established.
Theorem 4:
Vt(q)
{ct(Xt) -t- dt(Ut)} E qt (1) + E f t(k)qt(k) + Vt + l(qt + 1)
inf:
ut
/=1
qt
q
k=l
(4.4)
L. AGGOUN, L. BENKHEROUF and L. TADJ
430
A minimum principle has the following form.
Theorem 5: Suppose U* is a control such that for each positive
S-{Sl,...,SM} U(q)
U*
achieves the minimum in
(4.4).
Then
measure q on
Yt(q,U* )-yt(q)
and
is an. optimal control.
5. Summary and Open Problems
In this paper, an integer-valued inventory model for perishable items was introduced.
Various parameters estimators of the model were suggested using measure change
techniques. Further an optimal control formulation was set for the model. Possible
extensions of the paper include:
1. Extending model (1.4) to take into account the age of the items. Here difficulties
arise regarding the choice of the issuing policies.
2. The current paper departed from the classical approach of determining suitable
ordering policies. It would be of interest to investigate Economic Order Quantity
models.
3. The parameter process {an} is assumed to be a Markov chain in this paper. More
general processes for {an} could be taken.
References
[1]
A1-Osh, M.A. and A1-Zaid, A.A., First-order integer-valued autoregressive
(INAR(1))
process,
Benkherouf, L., On
[4]
[a]
[6]
Time Series Anal. 8
(1987),
261-275.
an inventory model with deteriorating items and
decreasing
time-varying demand and shortages, European J. of Operations Research 86
(1995), 293-299.
Benkherouf, L. and Mahmoud, M.G., On an inventory model for deteriorating
items with increasing time-varying demand and shortages, J. Oper. Res.
47 (1996), 188-200.
Chung, K.L., A Course in Probability Theory, Academic Press, New York 1974.
Elliot, R.J., Aggoun, L. and Moore, J.B., Hidden Markov Models: Estimation
and Control, Applications of Mathematics 29, Springer-Verlag, New York 1995.
McKenzie, E., Some simple models for discrete variate time series, Water Res.
Bull. 21
[7]
a.
(1985),
645-650.
Nahmias, S., Perishable inventory theory: A review, Operations Res. 30 (1982),
680-707.