Stochastic processes Lecture 7 Linear time invariant systems 1 Random process 2 1st order Distribution & density function First-order distribution First-order density function 3 2end order Distribution & density function 2end order distribution 2end order density function 4 EXPECTATIONS • Expected value • The autocorrelation 5 Some random processes • • • • • • • • Single pulse Multiple pulses Periodic Random Processes The Gaussian Process The Poisson Process Bernoulli and Binomial Processes The Random Walk Wiener Processes The Markov Process 6 Sxx(f) Recap: Power spectrum density f 7 Power spectrum density • Since the integral of the squared absolute Fourier transform contains the full power of the signal it is a density function. • So the power spectral density of a random process is: 𝑆𝑥𝑥 𝑓 = 𝑙𝑖𝑚 𝐸 𝑇→∞ 𝑇 𝑠 −𝑇 𝑡 𝑒 −𝑗2𝜋𝑓𝑡 𝑑𝑡 2 2𝑇 • Due to absolute factor the PSD is always real 8 Power spectrum density • The PSD is a density function. – In the case of the random process the PSD is the density function of the random process and not necessarily the frequency spectrum of a single realization. • Example – A random process is defined as X 𝑡 = sin(𝜔𝑟 𝑡) – Where ωr is a unifom distributed random variable wiht a range from 0-π – What is the PSD for the process and – The power sepctrum for a single realization 9 Properties of the PSD 1. Sxx(f) is real and nonnegative 2. The average power in X(t) is given by: 𝐸 𝑋 2 (𝑡) = 𝑅𝑥𝑥 0 = ∞ 𝑆𝑥𝑥 𝑓 𝑑𝑓 −∞ 3. If X(t) is real Rxx(τ) and Sxx(f) are also even 𝑆𝑥𝑥 −𝑓 = 𝑆𝑥𝑥 𝑓 4. If X(t) has periodic components Sxx(f)has impulses 5. Independent on phase 10 Wiener-Khinchin 1 • If the X(t) is stationary in the wide-sense the PSD is the Fourier transform of the Autocorrelation 11 Wiener-Khinchin Two method for estimation of the PSD Fourier Transform X(f) |X(f)|2 X(t) Sxx(f) X(t) f Fourier Transform t Rxx() Sxx(f) Autocorrelation f 12 The inverse Fourier Transform of the PSD • Since the PSD is the Fourier transformed autocorrelation • The inverse Fourier transform of the PSD is the autocorrelation 13 Cross spectral densities • If X(t) and Y(t) are two jointly wide-sense stationary processes, is the Cross spectral densities • Or 14 Properties of Cross spectral densities 1. Since is 2. Syx(f) is not necessary real 3. If X(t) and Y(t) are orthogonal Sxy(f)=0 4. If X(t) and Y(t) are independent Sxy(f)=E[X(t)] E[Y(t)] δ(f) 15 Cross spectral densities example • 1 Hz Sinus curves in white noise 𝑋 𝑡 = sin 2𝜋 𝑡 + 3 𝑤(𝑡) 𝑌 𝑡 = sin 2𝜋 𝑡 + 𝜋 2 + 3 𝑤(𝑡) Welch Cross Power Spectral Density Estimate Where w(t) is Gaussian noise 5 0 X(t) 10 0 -10 0 5 10 t (s) Signal Y(t) 15 20 Y(t) 10 -5 -10 -15 -20 -25 0 -10 Power/frequency (dB/Hz) Signal X(t) -30 0 5 10 t (s) 15 20 0 5 10 15 Frequency (Hz) 20 25 16 The periodogram The estimate of the PSD • The PSD can be estimate from the autocorrelation 𝑁−1 𝑅𝑥𝑥 [𝑚]𝑒 −𝑗ω𝑚 𝑆𝑥𝑥 ω = 𝑚=−𝑁+1 • Or directly from the signal 𝑆𝑥𝑥 ω = 1 𝑁 2 𝑁−1 𝑥 [𝑛]𝑒 −𝑗ω𝑛 𝑛=0 17 Bias in the estimates of the autocorrelation N=12 𝑁− 𝑚 −1 𝑅𝑥𝑥 𝑚 = 𝑥 𝑛 𝑥[𝑛 + 𝑚] 𝑛=0 Autocorrelation Autocorrelation M=-10 M=0 M=4 222 888 111 666 000 444 -1-1 -1 -2-2 -2 -10 -10 -10 -5 -5 -5 000 555 nnn 10 10 10 15 15 15 20 20 20 222 000 222 111 -2 -2 -2 000 -4 -4 -4 -1-1 -1 -2-2 -2 -10 -10 -10 -5 -5 -5 000 555 n+m n+m n+m 10 10 10 15 15 15 20 20 20 -6 -6 -6 -15 -15 -15 -10 -10 -10 -5 -5 0 5 10 15 18 Variance in the PSD • The variance of the periodogram is estimated to the power of two of PSD 𝑉𝑎𝑟 𝑆𝑥𝑥 𝜔 = 𝑆𝑥𝑥 (𝜔) 2 Realization 1 10 0 5 -5 True PSD 1 0 0.8 Sxx(f) PSD: Realization 1 5 5 t (s) Realization 2 10 0 5 10 0 5 0 50 100 150 f (Hz) PSD: Realization 2 200 50 200 0.6 0.4 -5 0.2 0 0 50 100 f (Hz) 150 200 0 5 t (s) Realization 3 10 0 5 10 0 5 -5 0 5 t (s) 10 0 0 0 100 150 f (Hz) PSD: Realization 3 50 100 f (Hz) 150 200 19 Averaging • Divide the signal into K segments of M length 𝑥𝑖 = 𝑥 𝑖 − 1 𝑀 + 1: 𝑖 𝑀 1≤𝑖≤𝐾 • Calculate the periodogram of each segment 𝑆𝑖𝑥𝑥 1 ω = 𝑀 2 𝑀−1 𝑥 𝑖 [𝑛]𝑒 −𝑗ω𝑛 𝑛=0 • Calculate the average periodogram 1 𝑆𝑥𝑥 [ω] = 𝐾 𝐾 𝑆𝑖𝑥𝑥 [ω] 𝑖=0 20 Illustrations of Averaging X(t) 2 0 -2 -4 0 1 2 10 3 4 5 4 6 7 10 8 9 0 100 10 6 4 5 2 5 2 0 0 100 200 0 0 100 200 0 0 100 200 0 200 10 5 0 0 50 100 f (Hz) 150 200 21 PSD units • Typical units: 2/Hz or dB V/Hz • Electrical measurements: V . • Sound: Pa2/Hz or dB/Hz • How to calculate dB I a power spectrum: PSDdB(f) = 10 log10 { PSD(f) } 22 Agenda (Lec. 7) • Recap: Linear time invariant systems • Stochastic signals and LTI systems – – – – Mean Value function Mean square value Cross correlation function between input and output Autocorrelation function and spectrum output • Filter examples • Intro to system identification 23 Focus continuous signals and system Continuous signal: x(t) 1 0 -1 0 20 40 60 80 100 8 10 t (s) Discrete signal: 1 x[n] 0.5 0 -0.5 -1 0 2 4 6 n 24 Systems 25 Recap: Linear time invariant systems (LTI) • What is a Linear system: – The system applies to superposition T a x1 (t ) b x2 (t ) a T x1 (t ) b T x2 (t ) Nonlinear systems Linear system 20 25 18 20 16 15 14 10 y(t) y(t) 12 10 5 8 0 6 -5 4 -10 x[n] 2 -15 x[n] 2 0 20 log(x[n]) 0 1 2 3 x(t) 4 5 -20 0 1 2 3 x(t) 4 5 26 Recap: Linear time invariant systems (LTI) • Time invariant: • A time invariant systems is independent on explicit time – (The coefficient are independent on time) • That means If: Then: y2(t)=f[x1(t)] y2(t+t0)=f[x1(t+t0)] The same to Day tomorrow and in 1000 years A non Time invariant 20 years 45 years 70 years 27 Examples • A linear system y(t)=3 x(t) • A nonlinear system y(t)=3 x(t)2 • A time invariant system y(t)=3 x(t) • A time variant system y(t)=3t x(t) 28 The impulse response The output of a system if Dirac delta is input (t ) h[n] f [ (t )] h(t ) Impuls response Impuls x(t) T{∙} y(t) inf 0 0 -10 -5 0 5 t 10 15 20 -10 -5 0 5 t 10 15 20 Convolution • The output of LTI system can be determined by the convoluting the input with the impulse response 30 Fourier transform of the impulse response • The Transfer function (System function) is the Fourier transformed impulse response • The impulse response can be determined from the Transfer function with the invers Fourier transform 31 Fourier transform of LTI systems • Convolution corresponds to multiplication in the frequency domain Time domain Output Impuls response Input 3 2 1 1 0 * -1 -2 -3 -10 -5 0 5 t 10 15 = 0 y(t) 2 y(t) x(t) 3 -2 -3 -10 20 -10 0 -1 -5 0 5 t 10 15 20 -5 0 5 t 10 15 20 Frequency domain 1.5 3000 1200 2500 1000 1 1000 = 600 400 0.5 200 500 0 -2 |Y(f)| x 1500 800 |H(f)| |X(f)| 2000 -1 0 f (Hz) 1 2 0 -2 -1 0 f (Hz) 1 2 0 -2 -1 0 f (Hz) 1 2 32 Causal systems • Independent on the future signal y(t) Impuls response 0 -10 -5 0 5 t 10 15 20 33 Stochastic signals and LTI systems • Estimation of the output from a LTI system when the input is a stochastic process Α is a delay factor like τ 34 Statistical estimates of output • The specific distribution function fX(x,t) is difficult to estimate. Therefor we stick to – Mean – Autocorrelation – PSD – Mean square value. 35 Expected Value of Y(t) (1/2) • How do we estimate the mean of the output? ∞ ∞ 𝐸𝑌 𝑡 =𝐸 𝑋 𝑡 − 𝛼 ℎ 𝛼 𝑑𝛼 −∞ 𝑌(𝑡) = 𝑋 𝑡 − 𝛼 ℎ 𝛼 𝑑𝛼 −∞ ∞ 𝐸𝑌 𝑡 = 𝐸 𝑋 𝑡 − 𝛼 ℎ 𝛼 𝑑𝛼 −∞ If mean of x(t) is defined as mx(t) ∞ 𝐸𝑌 𝑡 = 𝑚𝑥 (𝑡 − 𝛼)ℎ 𝛼 𝑑𝛼 −∞ 36 Expected Value of Y(t) (2/2) If x(t) is wide sense stationary 𝑚𝑥 𝑡 − 𝛼 = 𝑚𝑥 𝑡 = 𝑚𝑥 (𝑚𝑥 𝑖𝑠 𝑎 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡) ∞ 𝑚𝑦 = 𝐸 𝑌 𝑡 = ∞ 𝑚𝑥 (𝑡 − 𝛼)ℎ 𝛼 𝑑𝛼 −∞ 𝑚𝑦 = 𝐸 𝑌 𝑡 ∞ = 𝑚𝑥 𝑚𝑥ℎ 𝛼 𝑑𝛼 −∞ (𝑡 − 𝛼)ℎ 𝛼 𝑑𝛼 −∞ Alternative estimate: At 0 Hz the transfer function is equal to the DC gain ∞ ℎ 𝛼 𝑑𝛼 = 𝐻(0) −∞ Therefor: 𝑚𝑦 = 𝐸 𝑌 𝑡 = 𝑚𝑥 𝐻(0) 37 Expected Mean square value (1/2) ∞ 𝐸𝑌 𝑡 2 𝑌(𝑡) = =𝐸 𝑌 𝑡 𝑌 𝑡 −∞ ∞ 𝐸𝑌 𝑡 2 =𝐸 ∞ 𝑋 𝑡 − 𝛼1 ℎ 𝛼1 𝑑𝛼1 −∞ ∞ 𝐸𝑌 𝑡 2 𝑋 𝑡 − 𝛼 ℎ 𝛼 𝑑𝛼 𝑋 𝑡 − 𝛼2 ℎ 𝛼2 𝑑𝛼2 −∞ ∞ =𝐸 𝑋 𝑡 − 𝛼1 𝑋 𝑡 − 𝛼2 ℎ 𝛼1 ℎ 𝛼2 𝑑𝛼1𝑑𝛼2 −∞ −∞ ∞ 𝐸𝑌 𝑡 2 ∞ = 𝐸 𝑋 𝑡 − 𝛼1 𝑋 𝑡 − 𝛼2 ℎ 𝛼1 ℎ 𝛼2 𝑑𝛼1𝑑𝛼2 −∞ −∞ ∞ 𝐸𝑌 𝑡 2 ∞ = 𝑅𝑥𝑥 (𝑡 − 𝛼1, 𝑡 − 𝛼2) ℎ 𝛼1 ℎ 𝛼2 𝑑𝛼1𝑑𝛼2 −∞ −∞ ∞ 𝐸𝑌 𝑡 2 = ∞ 𝑅𝑥𝑥(𝛼1, 𝛼2) ℎ 𝑡 − 𝛼1 ℎ 𝑡 − 𝛼2 𝑑𝛼1𝑑𝛼2 −∞ −∞ 38 Expected Mean square value (2/2) ∞ 𝐸𝑌 𝑡 2 ∞ = 𝑅𝑥𝑥(𝛼1, 𝛼2) ℎ 𝑡 − 𝛼1 ℎ 𝑡 − 𝛼2 𝑑𝛼1𝑑𝛼2 −∞ −∞ By substitution: 𝛼 = 𝑡 − 𝛼1 𝛽 = 𝑡 − 𝛼2 ∞ 𝐸𝑌 𝑡 2 ∞ = 𝑅𝑥𝑥(𝑡 − 𝛼, 𝑡 − 𝛽)ℎ 𝛼 ℎ 𝛽 𝑑𝛼1𝑑𝛼2 −∞ −∞ If X(t)is WSS ∞ 𝐸𝑌 𝑡 2 = ∞ 𝑅𝑥𝑥 (𝛼 − 𝛽) ℎ 𝛼 ℎ 𝛽 𝑑𝛼1𝑑𝛼2 −∞ −∞ Thereby the Expected Mean square value is independent on time 39 Cross correlation function between input and output • Can we estimate the Cross correlation between input and out if X(t) is wide sense stationary ∗ Input 𝑅𝑦𝑥 𝑡 + 𝜏, 𝑡 = 𝐸 𝑌 𝑡 + 𝜏 𝑋 (𝑡) 3 2 x(t) 1 ∞ 0 𝑅𝑦𝑥 𝑡 + 𝜏, 𝑡 = 𝐸 -1 -2 -3 -10 -5 0 5 t 10 15 −∞ 20 ∞ Output 3 𝑅𝑦𝑥 𝑡 + 𝜏, 𝑡 = 𝐸 2 y(t) 1 𝑋 𝑡 − 𝛼 + 𝜏 ℎ 𝛼 𝑑𝛼 𝑋 ∗ (𝑡) 0 𝑋 𝑡 − 𝛼 + 𝜏 𝑋 ∗ (𝑡)ℎ 𝛼 𝑑𝛼 −∞ -1 -2 -3 -10 -5 0 5 t 10 15 𝑅𝑥𝑥 𝜏 = 𝐸 𝑋 𝑡 + 𝜏 𝑋 (𝑡) 20 ∞ Cross-correlation between y(t) and x(t) 1500 𝑅𝑦𝑥 𝜏 = 1000 Rxy() 500 𝑅𝑥𝑥 𝜏 − 𝛼 ℎ 𝛼 𝑑𝛼 = 𝑅𝑥𝑥 𝜏 ∗ ℎ(𝜏) −∞ 0 -500 -1000 -1500 -30 -20 -10 0 (s) 10 20 30 Thereby the cross-correlation is the convolution between 40 the auto-correlation of x(t) and the impulse response Autocorrelation of the output (1/2) 𝑅𝑦𝑦 𝜏 = 𝑅𝑦𝑦 𝑡 + 𝜏, 𝑡 = 𝐸 𝑌 𝑡 + 𝜏 𝑌(𝑡) ∞ Y(t) and Y(t+τ) is : 𝑌(𝑡 + 𝜏) = 𝑋 𝑡 + 𝜏 − 𝛼 ℎ 𝛼 𝑑𝛼 −∞ ∞ 𝑌(𝑡) = 𝑋 𝑡 − 𝛽 ℎ 𝛽 𝑑𝛽 −∞ ∞ 𝑋 −∞ 𝑅𝑦𝑦 𝜏 = 𝐸 𝑅𝑦𝑦 𝜏 = 𝐸 ∞ ∞ 𝑋 −∞ −∞ ∞ 𝑅𝑦𝑦 𝜏 = 𝑡 + 𝜏 − 𝛼 ℎ 𝛼 𝑑𝛼 ∞ 𝑋 −∞ 𝑡 − 𝛽 ℎ 𝛽 𝑑𝛽 𝑡 + 𝜏 − 𝛼 𝑋 𝑡 − 𝛽 ℎ 𝛼 ℎ 𝛽 𝑑𝛼𝑑𝛽 ∞ 𝐸[𝑋 𝑡 + 𝜏 − 𝛼 𝑋 𝑡 − 𝛽 ]ℎ 𝛼 ℎ 𝛽 𝑑𝛼𝑑𝛽 −∞ −∞ 41 Autocorrelation of the output (2/2) ∞ 𝑅𝑦𝑦 𝜏 = ∞ 𝐸[𝑋 𝑡 + 𝜏 − 𝛼 𝑋 𝑡 − 𝛽 ]ℎ 𝛼 ℎ 𝛽 𝑑𝛼𝑑𝛽 −∞ −∞ By substitution: α=-β ∞ 𝑅𝑦𝑦 𝜏 = ∞ 𝐸[𝑋 𝑡 + 𝜏 − 𝛼 𝑋 𝑡 + 𝛼 ]ℎ 𝛼 ℎ −𝑎 𝑑𝛼𝑑𝛼 −∞ −∞ 𝑅𝑦𝑦 𝜏 = 𝑅𝑦𝑥 𝜏 ∗ ℎ(−𝜏) Remember: 𝑅𝑦𝑥 𝜏 = 𝑅𝑥𝑥 𝜏 ∗ ℎ 𝜏 = ∞ 𝑅 𝜏 − 𝛼 ℎ 𝛼 𝑑𝛼 −∞ 𝑥𝑥 Autocorrelation of y(t) 1000 𝑅𝑦𝑦 𝜏 = 𝑅𝑥𝑥 𝜏 ∗ ℎ(𝜏) ∗ ℎ(−𝜏) Rxy() 500 0 -500 -1000 -30 -20 -10 0 (s) 10 20 30 42 Spectrum of output • Given: 𝑅𝑦𝑦 𝜏 = 𝑅𝑥𝑥 𝜏 ∗ ℎ 𝜏 ∗ ℎ(−𝜏) |𝐻 𝑓 |2 = 𝐻 𝑓 𝐻 ∗ (𝑓) • The power spectrum is 𝑆𝑦𝑦 𝑓 = 𝑆𝑥𝑥 𝑓 𝐻 𝑓 𝐻 ∗ (𝑓) 𝑆𝑦𝑦 𝑓 = 𝑆𝑥𝑥 𝑓 |𝐻 𝑓 |2 5 6 10 x 10 12 x 10 1.5 10 x 4 = 6 4 0.5 2 0 -2 Syy(f) 8 1 6 |H(f)|2 Sxx(f) 8 2 -1 0 f (Hz) 1 2 0 -2 -1 0 f (Hz) 1 2 0 -2 -1 0 f (Hz) 1 43 2 Filter examples 44 Typical LIT filters • FIR filters (Finite impulse response) • IIR filters (Infinite impulse response) – Butterworth – Chebyshev – Elliptic 45 Ideal filters • Highpass filter • Band stop filter • Bandpassfilter Filter types and rippels 47 Analog lowpass Butterworth filter • Is ”all pole” filter – Squared frequency transfer function H( f ) 2 1 2N 1 f / fc • N:filter order • fc: 3dB cut off frequency • Estimate PSD from filter 1 S yy ( f ) S xx ( f ) 2N 1 f / fc Chebyshev filter type I • Transfer function Hf 2 1 1 2TN2 f / f p • Where ε is relateret to ripples in the pass band • Where TN is a N order polynomium cos(N cos1 x) x 1 TN ( x) 1 cosh(N cosh x) x 1 Transformation of a low pass filter to other types (the s-domain) Filter type Lowpas>Lowpas Transformation p s s ' p Lowpas>Highpas p ' p Lowpas>Highpas Lowpas>Stopband p Old Cutoff frequency ' p New Cutoff frequency s s s 2 l u s p s ( u l ) s ( u l ) s p 2 s l u New Cutoff frequency ' p ' p l , u l , u l : Lowest Cutoff frequency u : Highest Cutoff frequency Discrete time implantation of filters • A discrete filter its Transfer function in the zdomain or Fourier domain b0 b1 z 1 b2 z 2 ´.......bm z M Y ( z) H ( z) X ( z) a0 a1 z 1 a2 z 2 ´.......am z M – Where bk and ak is the filter coefficients • In the time domain: y[n] b0 x[n] b1 x[n 1] b2 x[n 2] ...... aM x[n M ] a1 y[n 1] a2 y[n 2] ...... aM y[n M ] 51 Filtering of a Gaussian process • Gaussian process – X(t1),X(t2),X(t3),….X(tn) are jointly Gaussian for all t and n values • Filtering of a Gaussian process y[n] b0 w[n] b1w[n 1] b2 w[n 2] ...... aM w[n M ] a1 y[n 1] a2 y[n 2] ...... aM y[n M ] – Where w[n] are independent zero mean Gaussian random variables. 52 The Gaussian Process • X(t1),X(t2),X(t3),….X(tn) are jointly Gaussian for all t and n values • Example: randn() in Matlab Histogram of Gaussian process Gaussian process 700 5 4 600 3 500 2 1 400 0 300 -1 200 -2 100 -3 -4 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 0 -4 -3 -2 -1 0 1 2 3 4 5 The Gaussian Process and a linear time invariant systems • Out put = convolution between input and impulse response Gaussian input Gaussian output Example • x(t): Gaussian process 5 Histogram of Gaussian process 4 700 3 600 2 500 1 400 0 -1 300 -2 200 -3 100 -4 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 0 -4 • h(t): Low pass filter • y(t): -2 -1 0 1 2 3 4 5 Histogram of y(t) 600 1.5 500 1 400 0.5 300 0 200 -0.5 100 0 -1.5 -1 -1.5 -3 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 -1 -0.5 0 0.5 1 1.5 Filtering of a Gaussian process example 2 White noise 4 x(t) 2 0 -2 0 100 200 Magnitude (dB) -4 Phase (degrees) Band pass filter 300 400 500 t (ms) 600 700 800 900 1000 Transfere function of filter 0 -50 -100 0 100 200 300 Frequency (Hz) 400 500 0 100 200 300 Frequency (Hz) 400 500 0 -500 -1000 Output 1 y(t) 0.5 0 -0.5 -1 0 100 200 300 400 500 t (ms) 600 700 800 900 1000 56 Intro to system identification • Modeling of signals using linear Gaussian models: • Example: AR models y[n] a1 y[n 1] a2 y[n 2] ...... aM y[n M ] w[n] • The output is modeled by a linear combination of previous samples plus Gaussian noise. 57 Modeling example • Estimated 3th order model y[n] -2.6397 y[n 1] 2.3903 y[n 2] - 0.7299 y[n 3] w[n] Output 1 Output 0.5 -0.5 signal points used for prediction Prediction True point y(t) -1 0.35 0 100 200 300 400 500 t (ms) 600 700 800 900 1000 0.3 Output 0.25 451 0.294 451.5 452 452.5 t (ms) 453 453.5 454 0.292 0.29 y(t) y(t) 0.4 0 0.288 w[n] 0.286 0.284 0.282 58 453.98 453.99 454 t (ms) 454.01 454.02 Agenda (Lec. 7) • Recap: Linear time invariant systems • Stochastic signals and LTI systems – – – – Mean Value function Mean square value Cross correlation function between input and output Autocorrelation function and spectrum output • Filter examples • Intro to system identification 59