This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
MODEL CLASSES
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
1
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Classes of stationary stochastic processes for the description of
time series (MA, AR, ARMA)
e(t )
MODEL
y (t )
Process y (t ) is the output of a given model fed by a white noise e(t )
The relationship y(t )  e(t ) is given by linear difference equations
that in fact define the model.
MA (Moving Average) Processes
A process y (t ) is an MA process if:
y(t )  c0 e(t )  c1e(t  1)  c2 e(t  2)  ...  cn e(t  n) , where
e(t )  WN (  , 2 ) .
Terminology:
c0 , c1 , c2 , ... , cn
MA process (model) coefficients;
n
process (model) order;
MA(n)
MA process (or model) of order n.
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
2
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
AR (Auto Regressive) processes
A process y (t ) is an AR process if it is generated as:
y(t )  a1 y(t  1)  a2 y(t  2)  ...  am y(t  m)  e(t ) ,
where e(t )  WN (  , 2 ) .
Terminology:
a1 , a2 ,..., am
AR process (model) coefficients;
m
process (model) order;
AR(m)
AR process of order m.
Hence, the output y (t ) of an AR process is recursively defined as the
linear combination of last m past values of the process itself plus the
input e(t ) at the same time instant.
Observation. The difference equation generating the AR process
admits non-unique solution unless we specify an initial condition.
Which solution do we consider as the AR process?
By AR process we mean the solution obtained by taking the initial
condition y (t 0 )  0 and letting the initial time instant tends to minus
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
3
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
infinity, t0   (in short, we will write y ()  0 )). In other words,
the AR process is the steady-state solution.
Example
y (t )  ay (t  1)  e(t ) , where e(t )  WN (  , 2 ) (AR(1) process)
What is the steady state solution?
y (t )  ay (t  1)  e(t )
y (t  1)  ay (t  2)  e(t  1)
 e(t )  ae(t  1)  a 2 y (t  2)
y (t  2)  ay (t  3)  e(t  2)
…
 e(t )  ae(t  1)  a 2 e(t  2)    a tt y (t 0 )
y(t0 )  0
…
t0  
0

 e(t )  ae(t  1)  a e(t  2)    a e(t  n)     a i e(t  i )
2
n
i 0
The steady state solution is an MA(∞) process with coefficients
c0  1, c1  a, c2  a 2 , , ci  a i ,  .
In general, AR processes are MA(∞) processes with coefficients
determined by the AR model coefficients by recursively apply the
difference equation.
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
4
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
ARMA (Auto Regressive Moving Average) Processes
A process y (t ) is an ARMA process if it is generated as:
y(t ) 
 a1 y(t  1)  a2 y(t  2)  ...  am y(t  m) 
AR(m) part
 c0 e(t )  c1e(t  1)  ...  cn e(t  n) .
MA(n) part
where e(t )  WN (  , 2 ) .
Again by ARMA process we mean the steady-state solution obtained
by letting y ()  0 . Similarly to AR processes, the steady-state
solution is an MA(∞) process whose coefficients are obtained from
the ARMA model coefficients by recursively apply the difference
equation.
Terminology:
m
AR part order
n
MA part order
ARMA(m, n)
ARMA process of orders m and n
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
5
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Observation
ARMA processes generalize the AR and MA processes:
- an MA(n) process is also an ARMA (0, n) process
- an AR(m) process is also an ARMA (m, 0) process
Main question: under which conditions (on the model coefficients)
an ARMA process is well defined and stationary?
Starting from the MA(∞) is very difficult to give an answer (the
relation between the MA(∞) coefficients and the ARMA model
coefficients is too complex)
The answer will be given later, after introducing the operatorial
representation of ARMA processes.
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
6
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Model classes for I/O systems (ARMAX)
e(t)
u(t)
MODEL
+
MODEL
+
y(t)
I/O systems are obtained from time series models by adding a model
for the relationship between output y (t ) and input u (t ) . The
relationships y(t )  {e(t ), u (t )} is given by linear difference
equations which in fact define the models.
ARMAX processes
(Auto Regressive Moving Average with eXogeneous input)
A process y (t ) , generated by a remote white noise input e(t ) and by
an exogenous (measurable) input u (t ) , is an ARMAX process if:
y (t ) =
 a1 y(t  1)  a2 y(t  2)  ...  am y(t  m) 
AR(m) part
 c0e(t )  c1e(t  1)  ...  cn e(t  n) 
MA(n) part
+ b0u (t  k )  b1u (t  k  1)  ...  b p u (t  k  p)
X(k,p) part
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
7
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Terminology:
m
AR part order
n
MA part order
p
X part order
k
exogenous input delay
ARMAX(m, n,p,k) ARMAX process of orders m, n,p with input delay
equal to k.
Observation
ARMAX (ARMA) models are time-invariant and linear. They are
very general, and can be used to describe many processes of interest
(clearly, a suitable selection of the model orders is required).
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
8
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Extension to the nonlinear case: N-ARMAX models (N: Non linear)
y(t ) 
f  y (t  1),  y (t  2), ..., y (t  m),
e(t ), e(t  1), ..., e(t  n),
u (t  k ), u (t  k  1), ..., u (t  k  p )

non-linear combination of y (t ) , e(t ) , and u (t ) past values.
f usually belongs to a class of:

non linear

parametric

with good approximation properties
functions.
Some examples:

polynomials

splines

wavelets

neural networks

RBF (Radial basis neural networks)

…
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
9
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Arma/Armax processes OPERATIORIAL REPRESENTATION
Definition (backward and forward shift operators)
The backward shift operator z 1 (from the space of discrete-time
signals to the same space) is defined as: z 1 x(t )  x(t  1) .
Similarly, z is the forward shift operator and: z x(t )  x(t  1)
Properties of operators z 1 and z
z 1 and z are linear:
z 1 a  x(t )  b  y (t )   a  x(t  1)  b  y (t  1)
z a  x(t )  b  y (t )   a  x(t  1)  b  y (t  1)
z 1 and z can be recursively applied:
z 1 z 1 z 1  x(t )  
 z 1 z 1  x(t  1)  z 1  x(t  2)   x(t  3) 
 z 3 x(t ) (compact notation)
(similarly for z )
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
10
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
z 1 and z can be linearly composed:
(az 1  bz  cz 3  dz 2 ) x(t ) 
 a( z 1 x(t ))  b( z x(t ))  c( z 3 x(t ))  d ( z 2 x(t )) 
 ax(t  1)  bx(t  2)  cx(t  3)  dx(t  2)
ARMA processes operatorial representation (transfer function)
Given an ARMA (m, n) process:
y(t )  a1 y(t  1)  ...  am y(t  m)  c0 e(t )  c1e(t  1)  ...  cn e(t  n)
let’s use the backward shift operator z 1
y (t )  a1 z 1 y (t )  ...  am z  m y (t )  c0e(t )  c1 z 1e(t )  ...  cn z  n e(t )
i.e.
1  a z
1
1
 a2 z 2  ...  am z  m y (t )  c0  c1 z 1  ...  cn z  n e(t )
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
11
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Even more compact notation:
c
C( z)
 c1 z 1  ...  cn z  n 
,
i.e.
y
(
t
)

e(t )
y (t ) 
e
(
t
)
1  a1 z 1  a2 z 2  ...  am z m 
A( z )
0
where
C ( z )  c0  c1 z 1  ...  cn z  n 
A( z )  1  a1 z 1  a2 z 2  ...  am z  m ,
C ( z)
is called discrete time transfer function and it simply says that
A( z )
y (t ) is generated as the steady-state output of a linear digital filter fed
by e(t )
C ( z)
can be seen as a new operator which returns the output process
A( z )
y (t ) when applied to the remote input e(t ) and defines completely the
digital filter through which y (t ) is generated.
e(t )
C ( z)
A( z )
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
y (t )
12
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
TRANSFER FUNCTION COMBINATIONS
Let W (z ) and M (z ) be transfer functions of linear digital filters.
Case 1. (series)
Let y (t ) be the process generated as the output of filter W (z ) fed by
the output of filter M (z ) fed by an input u (t ) :
u (t )
W (z )
x(t )
M (z )
y (t )
 y (t )  W ( z ) x(t )
that is, y (t )  W ( z )M ( z )u (t ) , or equivalently 
 x(t )  M ( z )u (t )
Then, the process y (t ) is also the output of a new filter having transfer
function W ( z )  M ( z ) fed by u (t ) . That is,
y (t )  W ( z )  M ( z )u (t )
Perform symbolic
product here as if W(z)
and M(z) were function
of the variable z
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
13
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Case 2. (parallel)
Let y (t ) be the process generated as the sum of the output of filter
W (z ) fed by u (t ) and the output of filter M (z ) fed by u (t ) :
W (z )
 y (t )
u (t )

M (z )
that is, y(t )  W ( z )u (t )  M ( z )u (t ) .
Then, the process y (t ) is also the output of a new filter having transfer
function W ( z )  M ( z ) fed by u (t ) . That is,
y (t )  W ( z )  M ( z )u (t )
Perform symbolic sum
here as if W(z) and
M(z) were function of
the variable z
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
14
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
ARMAX processes operatorial representation
Given an ARMAX(m, n, p,k):
y (t )  a1 y (t  1)  a2 y (t  2)  ...  am y (t  m) 
 c0 e(t )  c1e(t  1)  ...  cn e(t  n) 
 b0 u (t  k )  b1u (t  k  1)  ...  b p u (t  k  p )
let’s use the backward shift operator z 1
y (t )  a1 z 1 y (t )  a2 z 2 y (t )  ...  am z  m y (t ) 
 c0 e(t )  c1 z 1e(t )  ...  cn z n e(t ) 
 b0 z k u (t )  b1 z k 1u (t )  ...  b p z k  p u (t )
i.e.
1  a z
1
1
 a2 z 2  ...  am z  m y (t ) 
 c0  c1 z 1  ...  cn z  n e(t )  b0  b1 z 1  ...  b p z  p z  k u (t )
Then, thanks to the transfer function combination properties and
thanks to linearity:
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
15
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
y (t ) 
b
0
 b1 z 1  ...  b p z  p z  k
u (t ) 
1  a z  a z ...  a z 
c  c z  ...  c z  e(t )

1  a z  a z  ...  a z 
1
2
1
m
2
m
1
0
n
1
1
n
1
2
2
m
m
i.e.
B( z ) z  k
C( z)
y (t ) 
u (t ) 
e(t )
A( z )
A( z )
where
B( z )  b0  b1 z 1  ...  b p z  p 
C ( z )  c0  c1 z 1  ...  cn z  n 
A( z )  1  a1 z 1  a2 z 2  ...  am z  m 
C ( z)
B( z ) z  k
and
are transfer functions.
A( z )
A( z )
e(t)
u(t)
C ( z)
A( z )
+
B( z ) z  k
A( z )
+
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
y(t)
16
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Observation
Transfer function in ARMA /ARMAX models can be both in
negative or positive powers of z
Es ( m  n ):

c0  c1 z 1  ...  cn z n z  m
c0 z  m  c1 z m1  ...  cn z mn 


1  a1 z 1  ...  am z m z  m z m  a1 z m1  a2 z m2  ...  am 
Please, do NOT use mixed negative/positive powers of z !!!
Why? Example...
Let’s consider an ARMA(1,1) process y(t )  e(t )
1 
y (t ) 
1 
z 1 
e(t )
1 1
3z 
1
2
where e(t ) ~WN (0,1)
This means that
 1 1 
 1 1 
1  z  y (t )  1  z e(t )
 3 
 2 
1
1
y (t )  z 1 y (t )  e(t )  z 1e(t )
3
2
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
17
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
i.e. by recalling the definition of z 1
1
1
y(t )  y(t  1)  e(t )  e(t  1)
3
2
This relationship must be valid for any t , even at time
1
1
y (t  1)  y (t )  e(t  1)  e(t )
3
2
By using the forward shift operator
z  13 y(t )  z  12 e(t )
so obtaining y (t ) 
z  12
e(t )
z  13
Hence,
z
y (t ) 
e(t )
z
1
2
1
3
is equivalent to
1 
y (t ) 
1 
z 1 
e(t )
1 1
3z 
1
2
(same result by multiplying the numerator and the denominator by z )
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
18
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
POLES / ZEROES
Let us consider a linear digital filter and its transfer function W (z ) .
W (z ) is an operator, but let us see it as a (rational) function of a
complex variable z .
Definition

The zeroes of W (z ) are all values of z such that W ( z )  0

The poles of W (z ) are all values of z such that W ( z ) 1  0
If W (z ) is written with positive powers of z :

The zeroes are the numerator roots

The poles are the denominator roots
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
19
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Definition
A linear digital filter having transfer function W (z ) is asymptotically
stable if and only if all its poles are strictly inside the circle with
unitary radius in the complex domain (the boundary is not allowed)
If in addition all zeroes are strictly inside the circle with unitary
radius in the complex domain, the filter is called minimum phase
filter, otherwise non-minimum phase filter.
1
Im
1
Re
Asymptotically stable
The boundary is excluded
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
20
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Example.
Find the poles and zeros of the filter having the following transfer
function:
1  12 z 1
W ( z) 
1  13 z 1
First, let us rewrite the transfer function by means of positive powers
of z
1  12 z 1 1  12 z 1 z z  12
W ( z) 

 
1 1
1 1
1 3 z
1  3 z z z  13
Zeroes = roots of the numerator, i.e. z 
1
1
 0z 
2
2
1
1
Poles = roots of the denominator, i.e. z   0  z  
3
3
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
21
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
1
Img
1

1
3
1
2
Re
zeroes
poles
The pole as well as the zero are strictly inside the unit circle so that
the filter is asymptotically stable and minimum phase.
Observation
Letting y (t ) be an ARMA process we will say “the poles and zeroes
of process y (t ) ” instead of “the poles and zeroes of the transfer
function of the filter through which y (t ) is generated”.
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
22
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Example.
Consider the MA(2) process generated as
1
1
y (t )  e(t )  e(t  1)  e(t  2) , where e(t ) ~ WN ( 0 ,1) ,
2
4
Computed the process poles and zeroes
By using the backward shift operator:
1
1
y (t )  e(t )  z 1e(t )  z 2 e(t )
2
4
and
1
 1

y (t )  1  z 1  z 2 e(t )
4 
 2
(that is y (t ) 
1
C ( z)
 1

e(t ) where C ( z )  1  z 1  z 2  and A( z )  1.
4
A( z )
 2

Positive powers of z representation:
 z 2 1  12 z 1  14 z 2 
e(t )
y (t )   2 
1
z

 z 2  12 z 1  14 
e(t )
y (t )  
2
z


Model Identification and Data Analysis (MIDA) – MODEL CLASSES
23
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
zeroes:
1
3
1
1
numerator roots: z 2  z   0 , that is z1, 2    i
4
4
2
4
poles:
denominator roots: z 2  0 , that is z1, 2  0
1
Img
3
4
1

1
4
Re

3
4
zeroes
poles
The filter generating y (t ) is minimum phase and asymptotically stable
The zeroes are conjugate and complex
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
24
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Observation
An MA(n) process has:

n nontrivial zeroes;

n poles, all lying at the origin  MA processes are always
generated by asymptotically stable digital filters.
(MA processes are also called “ALL-ZEROES” processes being their
poles trivial).
An AR(m) process has:

m zeroes all lying at the origin  AR process always generated
through a minimum phase filter

m nontrivial poles.
(AR processes are also called “ALL-POLES” processes being their
zeroes trivial).
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
25
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Es.
z 1  13 z 2
W ( z) 
1  2 z 1
find poles and zeroes.
First, rewrite the transfer functionas
z  13
z 2 z 1  13 z 2
W ( z)  2 
 2
1
z 1 2z
z  2z
zeroes: roots of the numerator: z 
1
0
3
z 
z1  0 , z 2  2
poles: roots of the denominator z 2  2 z  0
1
1
3
Img
1
1

3
2
Re
zeroes
poles
Minimum phase but unstable system (there’s a pole outside the unit
circle)
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
26
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
When an ARMA process is stationary?
Let us consider a stochastic process y (t ) obtained as the steady-state
output of a digital filter F (z ) fed by a stochastic process v(t ) as input
v(t )
F (z )
y (t )
y ()  0
Theorem. The steady-state output y (t ) is stationary if and only if:

v(t ) is stationary;

F (z ) is asymptotically stable.
That is:
The steady state output of an asymptotically stable digital filter fed by
a stationary stochastic process is stationary as well.
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
27
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Now, let us consider the stochastic processes y (t ) obtained as output
of an asymptotically stable digital filter F (z ) fed by a stationary
stochastic process v(t ) as input, but with a generic initialization (not
steady-state output)
v(t )
F (z )
y (t )
y (t0 )  y 0
Theorem. There is just one stationary output which corresponds to the
steady-state solution. However, if F (z ) is asymptotically stable, then
all possible outputs obtained for different initialization of the digital
filter F (z ) tends asymptotically (as t  ) to the steady-state
solution, i.e. to the stationary output.
v(t )
F (z )
y (t )
y ()  0
v(t )
F (z )
y (t )
y (t0 ) )  y 0
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
28
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Example. (MA processes redux)
Let y(t )  c0 e(t )  c1e(t  1)  c2 e(t  2)  ...  cn e(t  n)
where e(t ) ~ WN ( 0 , 2 ) .
 Is y (t ) an S.S.P.?
We already know that MA(n) processes are always stationary. Let us
verify this fact through the theorem.
Let consider the operatorial representation of y (t ) :
y (t )  c0  c1 z 1  c2 z 2  ...  cn z  n e(t )
with positive powers:
c0 z n  c1 z n1  c2 z n2  ...  cn
y (t ) 
e(t )
zn
There are n poles all lying at the origin ( z n  0 ), so that the digital
filter generating y (t ) is asymptotically stable.
Moreover, e(t ) is a White Noise which is S.S.P.
Therefore y (t ) is S.S.P. as well
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
29
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Example. (ARMA process)
Let us consider the ARMA process:
y (t ) 
1
y (t  1)  e(t )  8e(t  1) , with e(t ) ~ WN ( 0 , 4 ) ,
2
Is y (t ) sationary?
Rewrite y (t ) as:
1  8 z 1
y (t ) 
e(t )
1  12 z 1
There’s just one pole equal to

y (t ) 
z 8
e(t )
z  12
1
. Hence, the filter is asymptotically
2
stable. Moreover, the input is a White Noise (which is stationary by
definition).
Thanks to the previous theorem, the steady state output y (t ) of filter
z 8
when fed by e(t ) is a stationary stochastic process, and all
1
z2
possible outputs tends asymptotically to the stationary solution.
ARMA process simulation:
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
30
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
input e(t)
5
0
-5
0
10
20
30
40
50
60
70
80
90
100
60
70
80
90
100
output y(t)
50
0
-50
0
10
20
30
40
50
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
31
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Example. (non-stationary process)
y (t ) 
1
e(t ) ,
1  z 1
e  WN ( 0 ,1 )
5
0
-5
-10
-15
-20
-25
-30
-35
0
50
100
150
200
250
300
350
400
450
500
(random-walk or Brownian motion)
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
32
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Another example of a non-stationary process
1  0.5 z 1
y (t ) 
e(t ) ,
1
1  1.1z
e  WN ( 0 ,1 )
10
0
-10
-20
-30
-40
-50
-60
0
5
10
15
20
25
30
35
40
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
45
50
33
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
MEAN VALUE AND COVARIANCE FUNCTION OF ARMA
PROCESSES
AR(1) processes
Let us consider the AR(1) (or equivalently ARMA(1,0)) process
generating according to
y(t )  a  y(t  1)  e(t )
where
e(t ) ~ WN ( 0 ,  )
2
 Is y (t ) stationary?
 compute m y and  y ( ) for   0,1,2,
Operatorial representation for y (t ) :
y (t )  z 1 a y (t )  e(t )
1  z
1
y (t ) 
a y (t )  e(t )
1
e(t )
1
1 z a
Transfer function with positive powers (to spot out zeroes and poles):
y (t ) 
z
e(t )
z a
there’s just one pole: z  a .
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
34
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
The process generating system is asymptotically stable if a  1.
Since e(t ) is a S.S.P. (by definition of white noise), when a  1 the
steady-state output process y (t ) is in turn a S.S.P.
MA () representation for the AR(1) process:
- by recursively applying the difference equation (already done)
- by division of the numerator and denominator transfer function
polynomials (alternative method)
Operatorial representation
y (t ) 
1
e(t )
1
1  az
1
 1  az 1  a 2 z 2  a 3 z 3  a 4 z 4  ...
1
1  az
1  az 1
1
 1  az 1 
1  az 1  a 2 z 2  a3 z 3  
az 1
 az
1
2 2
a z

1
1  az 1
a 2 z 2
 a 2 z 2  a 3 z 3 
a 3 z 3




az 1
1
1  az 1
az  2
1
1  az 
1  az 1
az  3
1
2
1  az  az 
1  az 1

Model Identification and Data Analysis (MIDA) – MODEL CLASSES
35



This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Hence, y(t )  1  az 1  a 2 z 2  a 3 z 3  a 4 z 4  ...e(t ) and by recalling
the meaning of the backward shift operator:
y(t )  e(t )  a  e(t  1)  a 2  e(t  2)  a 3  e(t  3)  a 4  e(t  4)  ...
That’s the MA () representation of the AR(1) process.

The MA () is well defined and stationary if
 a   . Since this is a
i
0
geometric series, the necessary and sufficient condition is that a  1.
Let us compute m y .
We could use the MA () representation of y (t ) . However, this
approach is hard in the general ARMA case.
Alternative (easiest) method.
Start from the time-domain representation and apply expectation to
both sides:
E  y(t )  E a  y(t  1)  e(t )
and thanks to linearity:
E  y(t )  a  E  y(t  1)]  Ee(t ).
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
36
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Thanks to stationarity E  y (t )  E  y (t  1)  m y , so that
m y  a  m y  me
my 
me
1 a
me  0  m y  0
Let us compute  y (0)  E[ y(t )  m y  ]
2
(since m  0 , we have that  y (0)  E[ y(t )  ] )
2
y
Start from y(t )  a  y(t  1)  e(t ) , take the square and apply operator
E[] to both side:
E [ y (t )  ]  E [a  y (t  1)  e(t )  ]
2
2
Thanks to linearity
 y (0)  a 2 E[ y(t  1) 2 ]  E[e(t ) 2 ]  2aE y(t  1)e(t )
Mid-terms evaluation :
2aE y (t  1)e(t )  0 (we will show this later)
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
37
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.


E e(t )    
E  y(t  1)   y (0) (thanks to stationarity)
2
2
2
Hence,  y (0)  a 2 y (0)  2
and  y (0) 
2
1 a2
Let us compute  y (1)  E y (t )  m y   y (t  1)  m y  
 E y (t ) y (t  1) (since m y  0 )
Start from y(t )  a  y(t  1)  e(t ) and multiply both sides for y (t  1) .
Apply operator E[] to both side:
E  y(t ) y(t  1)  E a  y(t  1)  e(t ) y(t  1)
Thanks to linearity:


 y (1)  a  E  y(t  1)2  Ee(t ) y(t  1)
Mid-terms evaluation:
Ee(t ) y (t  1)  0
(we will show this later)
E  y(t  1)    y (0)
(we have already computed it!)

2

Model Identification and Data Analysis (MIDA) – MODEL CLASSES
38
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
 y (1)  a   y (0)  a 
2
1 a2
Similar rationale for  y (2)
 y (2)  E y (t )  m y  y (t  2)  m y   E y (t ) y (t  2).
E  y(t ) y(t  2)  E a  y(t  1)  e(t ) y(t  2)
 y (2)  a  E y (t  1) y (t  2)  Ee(t ) y (t  2)
Since Ee(t ) y (t  2)  0 (we will show this later)
 y (2)  a   y (1)  a 
2
2
1 a2
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
39
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Summary:
 y (0) 
2
1 a2
 y (1)   y (1)  a  y (0)

 y (2)   y (2)  a  y (1)
...

  y ( )  a   y (  1) con  1
Recursive expression for  y ( )
 y ( )  a 

2
1 a2
This result has been established for a generic AR(1) process
Those equations are called “Yule-Walker equations”.
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
40
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Graphical representation
Case 1: a  1 and a  0 (i.e. 0  a  1)
 y ( )
2
1 a2
-5
-4
-3
-2
-1
1
2
3
4
4
5

5
Case 2: a  1 and a  0 (i.e.  1  a  0 )
 y ( )
2
1 a2
-5
-4
-3
-2
-1
1

2
3

2
1 a2
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
41
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Proof: Ee(t ) y (t  1)  0 , Ee(t ) y (t  2)  0 …
Basic property: MA () representation of the AR process:
y(t )  e(t )  a  e(t  1)  a 2  e(t  2)  a 3  e(t  3)  a 4  e(t  4)  ...
By using the MA () representation for y (t  1) (same expression as
before with t  1 in place of t though):
y(t  1)  e(t  1)  a  e(t  2)  a 2  e(t  3)  a 3  e(t  4)  ...
we have that
Ee(t ) y (t  1) 
 E e(t )  e(t  1)  a  e(t  2)  a 2  e(t  3)  a 3  e(t  4)  ...  0
(all products give null contribution, never )
Similarly,
Ee(t ) y (t  2) 
 E e(t )  e(t  2)  a  e(t  3)  a  e(t  4)  a  e(t  5)  ...  0
2
3
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
42
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
ARMA processes (GENERAL CASE)
y(t )  a1 y(t  1)  ...  am y(t  m)  c0 e(t )  ...  cn e(t  n)
where e(t ) ~ WN ( 0 , 2 )
Mean
E[ y(t )]  E[a1 y(t  1)  ...  am y(t  m)  c0 e(t )  ...  cn e(t  n)]
 a1E[ y(t  1)]  ...  am E[ y(t  m)]  c0 E[e(t )]  ...  cn E[e(t  n)]
m y  a1m y  ...  am m y  c0  0  ...  cn  0
i.e. m y  0
Covariance function
E[ y(t ) 2 ]  E[a1 y (t  1)  ...  am y (t  m)  c0 e(t )  ...  cn e(t  n)  ] 
2
 a1 E[ y(t  1) 2 ]  a2 E[ y (t  2) 2 ]  2a1a2 E[ y(t  1) y (t  2)]  ...
2
2
 c0 E[e(t ) 2 ]  2a1c0 E[ y(t  1)e(t )]  ...
2
Hence
 y (0)  a12 y (0)  a2 2 y (0)  2a1a2 y (1)  ...
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
This term and others
similar can be
evaluated by means
of the MA(∞)
representation of the
ARMA process
43
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Then
E[ y (t ) y (t  1)] 
 E[a1 y(t  1)  ...  am y(t  m)  c0 e(t )  ...  cn e(t  n) y(t  1)] 
 a1E[ y (t  1) 2 ]  ...  c0 E[e(t ) y (t  1)]  ...
Hence
 y (1)  a1 y (0)    c0 E[e(t ) y (t  1)]  
This term and others
similar can be
evaluated by means
of the MA(∞)
representation of the
ARMA process
Proceeding this way:
 y (0)  a1 2 y (0)  a2 2 y (0)  2a1a2 y (1)  ...

 y (1)  a1 y (0)    c0 E[e(t ) y (t  1)]  


 (m  1)  a  (m  2)  
1 y
 y
m variables – m linear equations (YULE-WALKER equations for an
ARMA process)
 y (0),  y (1),,  y (m  1)
Then,  y (m),  y (m  1), can be recursevely computed from
 y (0),  y (1),,  y (m  1)
 y (m)  E[ y (t ) y (t  m)] 
 E[a1 y(t  1)  ...  am y(t  m)  c0 e(t )  ...  cn e(t  n) y(t  m)]  
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
44
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
PROCESSES WITH NON-NULL MEAN
y(t )  a1 y(t  1)  ...  am y(t  m)  c0 e(t )  ...  cn e(t  n)
where e(t ) ~ WN (  , 2 )
  0 in general
Operatorial representation
c0  c1 z 1    cn z  n
y (t ) 
e(t )  W ( z )e(t ) ,
1  a1 z 1    am z m
e(t ) ~ WN (  , 2 )
Mean
E[ y(t )]  E[a1 y(t  1)  ...  am y(t  m)  c0 e(t )  ...  cn e(t  n)]
 a1E[ y(t  1)]  ...  am E[ y(t  m)]  c0 E[e(t )]  ...  cn E[e(t  n)]
m y  a1m y  ...  am m y  c0    ...  cn  
i.e. m y 
c0  c1    cn
   W (1)  
1  a1    am
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
45
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Covariance function
 y ( )  E[ y (t )  m y   y (t   )  m y ]
we cannot drop m y  0 !!!
 y ( )  E[ y (t )  y (t   )]
Moreover in this case: E[e(t ) 2 ]  2 and E[e(t )  e(t  k )]  0 , k  0
Indeed:
2  E[e(t )   2 ]  E[e(t ) 2 ]  E[ 2 ]  2E[e(t )   ]  E[e(t ) 2 ]   2
Hence, E[e(t ) 2 ]  2   2
0  E[e(t )   e(t  k )   ]  E[e(t )e(t  k )]   2
Hence, E[e(t )e(t  k )]   2
Easiest way to deal with processes with non-null mean
Define two new processes (unbiased processes)
y (t )  y (t )  m y
E ~
y (t )  E y (t )  m y  0
~

~
Ee~ (t )  Ee(t )  me  0
e (t )  e(t )  me
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
46
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Which type of process is ~
y (t ) ?
~
y (t )  y (t )  m y 
 a1 y (t  1)  ...  am y (t  m)  c0 e(t )  ...  cn e(t  n)  m y
 a1 ( ~
y (t  1)  m y )  ...  am ( ~
y (t  m)  m y ) 
 c0 (e~ (t )  me )  ...  cn (e~ (t  n)  me )  m y
 a1 ~
y (t  1)  ...  am ~
y (t  m)  c0 e~(t )  ...  cn e~(t  n)
 (1  a1  ...  am )m y  (c0  cn )me
This term is null, remember that m y 
c0  c1    cn
me
1  a1    am
Hence,
~
y (t )  a1 ~
y (t  1)  ...  am ~
y (t  m)  c0 e~(t )  ...  cn e~(t  n)
e(t ) ~ WN (0 , 2 )
Standard zero mean ARMA process
Moreover,
 y ( )  E y (t )  m y   y (t   )  m y   E ~y (t )    ~y (t   )    ~y ( )
Hence:
 ( )   ( ) 
~
y
y
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
47
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Alternative interpretation of the bias removal procedure

 e(t )

W (z )
y (t )
e~ (t )
Thanks to linearity

W (z )
my

y (t )

e~ (t )
W (z )
~
y (t )
Recall that the stationary process y (t ) is the steady-state output
m y is the steady-state output of W (z ) fed by a constant input signal
equal to 
Gain theorem: the steady-state output is constant and it holds that:
my  W ( z ) z 1  
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
48
This material is protected by copyright and is intended for students’ use only. Sell and distribution are strictly forbidden. Final
exam requires integrating this material with teacher explanations and textbooks.
Observation
Given a S.P. generated as the output of a digital filter fed by a S.S.P.
v(t)
F(z)
y(t)
If F (z ) is asymptotically stable (i.e. y (t ) is S.S.P.) , and if
E[v(t )]  mv  0 , then E[ y (t )]  m y  0 .
Model Identification and Data Analysis (MIDA) – MODEL CLASSES
49