Stationary Stochastic Processes, 2020
1
Stationary Stochastic Processes
Table of Formulas, 2020
Basics of probability theory
• The following is valid for probabilities:
⇤ P(W) = 1, where W is all possible outcomes
⇤ 0  P(A)  1, where A is some event
⇤ P(Ac ) = 1
P(A), where Ac is the complement of A
⇤ P(A [ B) = P(A) + P(B), if the events A and B are mutually exclusive
• The addition law of probability: P(A [ B) = P(A) + P(B)
• The conditional probability: P(B | A) =
P(A \ B)
P(A \ B)
P(A)
• A and B are independent () P(A \ B) = P(A) P(B)
Stochastic variables
8 X
>
pX (k)
(X discrete)
>
<
kx0
Z x0
• Distribution functions: FX (x0 ) = P(X  x0 ) =
>
>
:
fX (x) dx (X continuous)
1
8 X
>
k pX (k)
(X discrete)
>
<
Zk 1
• Expected value: E[X] = mX =
>
>
:
x fX (x) dx (X continuous)
1
• Variance: V[X] = E[X2 ]
8 X
>
(k mX )2 pX (k)
(X discrete)
>
<
k
Z 1
m2X =
>
>
:
(x mX )2 fX (x) dx (X continuous)
1
• Rules for expected value and variance (a and b constants):
⇤ E[aX + b] = aE[X] + b
⇤ V[aX] = a2 V[X]
⇤ V[X + b] = V[X]
⇤ E[X + Y] = E[X] + E[Y]
⇤ V[X + Y] = V[X] + V[Y] + 2C[X, Y]
2
Stationary Stochastic Processes, 2020
• Covariance: C[X, Y] = E[(X
mX )(Y
mY )] = E[XY]
mX mY
C[X, Y]
• Correlation coefficient: r[X, Y] = p
V[X] V[Y]
• Taylor series expansions (”Gauss’ approximations”):
E[g(X1 , . . . , Xn )] ⇡ g(E[X1 ], . . . , E[Xn ])
n
X
X
V[g(X1 , . . . , Xn )] ⇡
c2i V[Xi ] + 2
ci cj C[Xi , Xj ]
i=1
where ci =
@g
(x1 , . . . , xn )
@xi
i<j
xk =E[Xk ], 8k
• The two-dimensional probability density function of a jointly Gaussian random
variable, (X, Y), with E[X] = E[Y] = 0, V[X] = V[Y] = 1 and C[X, Y] = r is
⇢
1
1
p
fX,Y (x, y) =
exp
(x2 2rxy + y2 )
2
2(1 r2 )
2p 1 r
Trigonometric relations
cos(a + b) = cos a cos b
cos(a
sin a sin b
b) = cos a cos b + sin a sin b
sin(a + b) = sin a cos b + cos a sin b
sin(a
b) = sin a cos b
cos a sin b
Stationary stochastic processes
• Estimation of expected value:
n
1X
m̂n =
Xt
n t=1
n 1
1 X
V [m̂n ] = 2
(n
n t= n+1
1
1 X
V [m̂n ] ⇡
rX (t)
n t= 1
|t|)rX (t)
for large n
Stationary Stochastic Processes, 2020
3
• If m̂n 2 N(m, V[m̂n ]), the confidence interval for m is
p
p
Im : {m̂n la/2 V[m̂n ], m̂n + la/2 V[m̂n ]}
with confidence level 1
1.96.
a. For confidence level 0.95, a = 0.05 and la/2 = l0.025 =
• Estimation of covariance function:
n t
1X
(Xt
r̂n (t) =
n t=1
mX )(Xt+t
t
mX ) for
0
where mX is replaced by m̂n if mX is unknown.
The Poisson process and the Wiener process
• A simply increasing process {X(t), t
0} is a homogeneous Poisson process, if
X(0) = 0 and X(t) has stationary, independent increments. If the intensity is l,
⇤ E[X(t)] = lt
⇤ V[X(t)] = lt
⇤ rX (s, t) = l min(s, t)
The interarrival times are independent and exponentially distributed with mean
value 1/l.
• A Gaussian process {X(t), t 0} is a Wiener process, if X(0) = 0, and X(t) has
independent increments, where X(t) X(t + h) 2 N(0, s2 h),
⇤ E[X(t)] = 0
⇤ V[X(t)] = s2 t
⇤ rX (s, t) = s2 min(s, t)
Spectral representations
• Relations between covariance function rX (t) and spectral density RX (f):
Continuous time
rX (t) =
RX (f) =
R1
1
R1
Discrete time
RX (f)ei2pft df
r (t)e
1 X
i2pft
dt
rX (t) =
RX (f) =
R 1/2
1/2
P1
RX (f)ei2pft df
t= 1 rX (t)e
i2pft
4
Stationary Stochastic Processes, 2020
• Folding (aliasing): Let {Zt , t = 0, ±d, ±2d, . . . } be the continuous time process
Y(t) sampled with time interval d and sampling frequency fs = 1/d:
RZ (f) =
1
X
RY (f + kfs )
for
k= 1
fs /2 < f  fs /2
• Sum of harmonic components with random phase and amplitude:
X(t) = A0 +
n
X
Ak cos(2pfk t + fk )
k=1
where fk 2 Rect(0, 2p), Ak , k = 0, . . . , n, are independent and E[A0 ] = 0.
⇤ Covariance function:
rX (t) =
s20
+
n
X
s2k cos 2pfk t
k=1
where s20 = E [A20 ] and s2k = E [A2k ] /2.
⇤ Spectral density:
RX (f) =
n
X
bk dfk (f),
k= n
where b0 = s20 = E [A20 ], and bk = s2k /2 = E [A2k ] /4.
Linear filters - general theory
• Impulse response h(u):
8 R1
< 1 h(u)X(t u) du
Y(t) =
: P1
u)
u= 1 h(u)X(t
(continuous time)
(discrete time)
• Relation between covariance functions:
8 R1 R1
< 1 1 h(u)h(v) rX (t + u v) du dv
rY (t) =
P1
: P1
v)
u= 1
v= 1 h(u)h(v) rX (t + u
(continuous time)
(discrete time)
• Relation between spectral densities:
RY (f) = |H(f)|2 RX (f)
where H(f) is the frequency function corresponding to the impulse response h(n).
Stationary Stochastic Processes, 2020
5
• Di↵erentiation:
X0 (t) exists (in quadratic mean) if r00X (t) exists. This is equivalent
R1
2
to 1 (2pf) R(f)df < 1. If X0 (t) exists, the following relations hold:
r00X (t)
rX0 (t) =
RX0 (f) = (2pf)2 RX (f)
Z
0
V [X (t)] =
1
(2pf)2 RX (f) df
1
rX,X0 (t) = r0X (t)
(j+k)
rX(j) ,X(k) (t) = ( 1)j rX
• Integration:
E
C
Z
Z
g(s)X(s) ds,
Z
g(s)X(s) ds
=
h(t)Y(t) dt
=
Z
(t)
g(s)E[X(s)] ds
Z Z
• Cross-covariance and cross-spectrum:
rX,Y (t) = C[X(t), Y(t + t)] =
g(s)h(t) C[X(s), Y(t)] ds dt
Z
ei2pft RX,Y (f) df
RX,Y (f) = H(f)RX (f) = AX,Y (f)eiFX,Y (f)
where AX,Y (f) is the amplitude spectrum and FX,Y (f) the phase spectrum. The
squared coherence spectrum is
k2X,Y (f) =
A2X,Y (f)
RX (f)RY (f)
AR- MA- and ARMA-models
• White noise in discrete time: {et , t = 0, ±1, . . .}, E[et ] = 0 and V[et ] = s2 :
Re (f) = s2
for
1/2  f  1/2
• AR(p)-process: (a0 = 1)
Xt + a1 Xt
1
+ a2 Xt
2
+ . . . + ap Xt
p
= et
6
Stationary Stochastic Processes, 2020
? Yule-Walker equations for covariance function:
rX (k) + a1 rX (k
1) + . . . + ap rX (k
p) =
1
RX (f) = Pp
| k=0 ak e
i2pfk |2
⇢
s2 for k = 0
0 for k = 1, 2, . . .
? Spectral density:
• MA(q)-process: (b0 = 1)
Xt = et + b1 et
? Covariance function:
rX (t) =
⇢
+ b 2 et
2
P
bj bk for |t|  q
for |t| > q
1
s2
0
s2
j k=t
+ . . . + b q et
q
? Spectral density:
RX (f) = |
q
X
bk e
i2pfk 2 2
|s
k=0
Matched filter and Wiener filter
• Matched filter:
? with white noise:
h(t) = s(T
RT
0
SNRopt =
? with colored noise:
s(T
t) =
SNRopt =
Z
0tT
s2 (T u) du
s2N
T
h(u)rN (t
u) du
0
Z TZ
0
• Wiener filter:
t),
T
h(u)h(v)rN (u
0
RS (f)
RS (f) + RN (f)
R
RS (f) df
SNR = R R (f)R (f)
S
N
df
RS (f)+RN (f)
H(f) =
v) du dv
Stationary Stochastic Processes, 2020
7
Spectral estimation
• Periodogram of the sequence {x(t), t = 0, 1, 2, . . . n 1},
1
R̂x (f) = |X (f)|2
n
Pn 1
i2pft
where X (f) = t=0 x(t)e
.
1
h
i
X
E R̂x (f) =
kn (t)rX (t)e i2pft
t= 1
Z
=
where kn (t) = 1
1/2
Kn (f
u)RX (u)du
1/2
P
for n + 1  t  n 1 and Kn (f) = nt= 1 n+1 kn (t)e
⇢ 2
h
i
RX (f) for 0 < |f| < 1/2
V R̂x (f) ⇡
2R2X (f) for f = 0, ±1/2
|t|
n
i2pft
.
The distribution of the periodogram estimate is
R̂x (f)
q2 (2)
⇡
for 0 < f < 1/2
RX (f)
2
• Modified periodogram
2
n 1
1 X
R̂w (f) =
x(t)w(t)e i2pft
n t=0
Z 1/2
1
X (n)W(f n)dn
=
n
1/2
• Lag-windowing
R̂lw (f) =
1
X
kLn (t)r̂x (t)e
2
i2pft
t= 1
=
• Averaging of spectrum
Z
1/2
KLn (f
n)R̂x (n)dn
1/2
K
1 X
R̂av (f) =
R̂x,j (f)
K j=1
where K di↵erent spectrum estimates, R̂x,j (f), j = 1 . . . K, are used. The distribution is
R̂av (f)
q2 (2K)
⇡
for 0 < f < 1/2
RX (f)
2K
8
Stationary Stochastic Processes, 2020
Fourier transforms
g(t)
e a|t|
(a > 0)
G(f) =
p
a
a|t|
e
a|t| cos(2pf
e
a|t| sin(2pf
p
p/a exp(
0 t)
0 t)
a
if
if
sin(2pat)
2pt
1
0
t=0
t 6= 0
a|t| if |t| 
if |t| >
1
a
1
a
g(t) ⇤ h(t) =
R
g(t)h(t
a
a2 +(2pf0 2pf)2
+
a
a2 +(2pf0 +2pf)2
2pf0 2pf
a2 +(2pf0 2pf)2
+
2pf0 +2pf
a2 +(2pf0 +2pf)2
⇢
1/2
0
(
1
a
if
if
2a
(2pf)2
1
R
2pf
a
G(n)H(f
i2pf G(f)
1
f
a G( a )
1
t
a g( a )
G(af)
t0 )
g(t)ei2pf0 t
G(f)e
i2pft0
G(f
f0 )
Parseval’s theorem:
n 1
X
t=0
|x(t)|2 =
where X(f) =
Z
1/2
1/2
Pn
i2pf)k+1 }
|f|  a
|f| > a
cos
t)dt G(f)H(f)
g(at)
g(t
+ i2pf)k+1 + (a
(2pf)2
4a )
G(f) ⇤ H(f) =
g(t)h(t)
g0 (t)
2
k!
{(a
(a2 +(2pf)2 )k+1
at2
⇢
2pa|f|
e
2
e
⇢
i2pft g(t) dt
(2pf) )
2 (a(a2 +(2pf)
2 )2
a|t|
|t|k e
1e
2a
a2 +(2pf)2
1
a2 +t2
|t|e
R1
|X(f)|2 df,
1
i2pft .
t=0 x(t)e
if
if
f=0
f 6= 0
n)dn
Stationary Stochastic Processes, 2020
9
Gaussian distribution table
F(x) = F(x)
x
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2.0
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
3.0
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.9
4.0
0.00
0.50000
0.53983
0.57926
0.61791
0.65542
0.69146
0.72575
0.75804
0.78814
0.81594
0.84134
0.86433
0.88493
0.90320
0.91924
0.93319
0.94520
0.95543
0.96407
0.97128
0.97725
0.98214
0.98610
0.98928
0.99180
0.99379
0.99534
0.99653
0.99744
0.99813
0.99865
0.99903
0.99931
0.99952
0.99966
0.99977
0.99984
0.99989
0.99993
0.99995
0.99997
0.01
0.50399
0.54380
0.58317
0.62172
0.65910
0.69497
0.72907
0.76115
0.79103
0.81859
0.84375
0.86650
0.88686
0.90490
0.92073
0.93448
0.94630
0.95637
0.96485
0.97193
0.97778
0.98257
0.98645
0.98956
0.99202
0.99396
0.99547
0.99664
0.99752
0.99819
0.99869
0.99906
0.99934
0.99953
0.99968
0.99978
0.99985
0.99990
0.99993
0.99995
0.99997
0.02
0.50798
0.54776
0.58706
0.62552
0.66276
0.69847
0.73237
0.76424
0.79389
0.82121
0.84614
0.86864
0.88877
0.90658
0.92220
0.93574
0.94738
0.95728
0.96562
0.97257
0.97831
0.98300
0.98679
0.98983
0.99224
0.99413
0.99560
0.99674
0.99760
0.99825
0.99874
0.99910
0.99936
0.99955
0.99969
0.99978
0.99985
0.99990
0.99993
0.99996
0.99997
0.03
0.51197
0.55172
0.59095
0.62930
0.66640
0.70194
0.73565
0.76730
0.79673
0.82381
0.84849
0.87076
0.89065
0.90824
0.92364
0.93699
0.94845
0.95818
0.96638
0.97320
0.97882
0.98341
0.98713
0.99010
0.99245
0.99430
0.99573
0.99683
0.99767
0.99831
0.99878
0.99913
0.99938
0.99957
0.99970
0.99979
0.99986
0.99990
0.99994
0.99996
0.99997
0.04
0.51595
0.55567
0.59483
0.63307
0.67003
0.70540
0.73891
0.77035
0.79955
0.82639
0.85083
0.87286
0.89251
0.90988
0.92507
0.93822
0.94950
0.95907
0.96712
0.97381
0.97932
0.98382
0.98745
0.99036
0.99266
0.99446
0.99585
0.99693
0.99774
0.99836
0.99882
0.99916
0.99940
0.99958
0.99971
0.99980
0.99986
0.99991
0.99994
0.99996
0.99997
0.05
0.51994
0.55962
0.59871
0.63683
0.67364
0.70884
0.74215
0.77337
0.80234
0.82894
0.85314
0.87493
0.89435
0.91149
0.92647
0.93943
0.95053
0.95994
0.96784
0.97441
0.97982
0.98422
0.98778
0.99061
0.99286
0.99461
0.99598
0.99702
0.99781
0.99841
0.99886
0.99918
0.99942
0.99960
0.99972
0.99981
0.99987
0.99991
0.99994
0.99996
0.99997
0.06
0.52392
0.56356
0.60257
0.64058
0.67724
0.71226
0.74537
0.77637
0.80511
0.83147
0.85543
0.87698
0.89617
0.91309
0.92785
0.94062
0.95154
0.96080
0.96856
0.97500
0.98030
0.98461
0.98809
0.99086
0.99305
0.99477
0.99609
0.99711
0.99788
0.99846
0.99889
0.99921
0.99944
0.99961
0.99973
0.99981
0.99987
0.99992
0.99994
0.99996
0.99998
0.07
0.52790
0.56749
0.60642
0.64431
0.68082
0.71566
0.74857
0.77935
0.80785
0.83398
0.85769
0.87900
0.89796
0.91466
0.92922
0.94179
0.95254
0.96164
0.96926
0.97558
0.98077
0.98500
0.98840
0.99111
0.99324
0.99492
0.99621
0.99720
0.99795
0.99851
0.99893
0.99924
0.99946
0.99962
0.99974
0.99982
0.99988
0.99992
0.99995
0.99996
0.99998
0.08
0.53188
0.57142
0.61026
0.64803
0.68439
0.71904
0.75175
0.78230
0.81057
0.83646
0.85993
0.88100
0.89973
0.91621
0.93056
0.94295
0.95352
0.96246
0.96995
0.97615
0.98124
0.98537
0.98870
0.99134
0.99343
0.99506
0.99632
0.99728
0.99801
0.99856
0.99896
0.99926
0.99948
0.99964
0.99975
0.99983
0.99988
0.99992
0.99995
0.99997
0.99998
0.09
0.53586
0.57535
0.61409
0.65173
0.68793
0.72240
0.75490
0.78524
0.81327
0.83891
0.86214
0.88298
0.90147
0.91774
0.93189
0.94408
0.95449
0.96327
0.97062
0.97670
0.98169
0.98574
0.98899
0.99158
0.99361
0.99520
0.99643
0.99736
0.99807
0.99861
0.99900
0.99929
0.99950
0.99965
0.99976
0.99983
0.99989
0.99992
0.99995
0.99997
0.99998