Environmental Data Analysis with MatLab 2nd Edition Lecture 20: Coherence; Tapering and Spectral Analysis SYLLABUS Lecture 01 Lecture 02 Lecture 03 Lecture 04 Lecture 05 Lecture 06 Lecture 07 Lecture 08 Lecture 09 Lecture 10 Lecture 11 Lecture 12 Lecture 13 Lecture 14 Lecture 15 Lecture 16 Lecture 17 Lecture 18 Lecture 19 Lecture 20 Lecture 21 Lecture 22 Lecture 23 Lecture 24 Lecture 25 Lecture 26 Using MatLab Looking At Data Probability and Measurement Error Multivariate Distributions Linear Models The Principle of Least Squares Prior Information Solving Generalized Least Squares Problems Fourier Series Complex Fourier Series Lessons Learned from the Fourier Transform Power Spectra Filter Theory Applications of Filters Factor Analysis Orthogonal functions Covariance and Autocorrelation Cross-correlation Smoothing, Correlation and Spectra Coherence; Tapering and Spectral Analysis Interpolation Linear Approximations and Non Linear Least Squares Adaptable Approximations with Neural Networks Hypothesis testing Hypothesis Testing continued; F-Tests Confidence Limits of Spectra, Bootstraps Goals of the lecture Part 1 Finish up the discussion of correlations between time series Part 2 Examine how the finite observation time affects estimates of the power spectral density of time series Part 1 “Coherence” frequency-dependent correlations between time series Scenario A in a hypothetical region windiness and temperature correlate at periods of a year, because of large scale climate patterns but they do not correlate at periods of a few days wind speed 1 2 3 temperature time, years 1 2 time, years 3 summer hot and windy wind speed winters cool and calm 1 2 3 temperature time, years 1 2 time, years 3 wind speed heat wave not especially windy 1 cold snap not especially calm 2 3 temperature time, years 1 2 time, years 3 in this case times series correlated at long periods but not at short periods Scenario B in a hypothetical region plankton growth rate and precipitation correlate at periods of a few weeks but they do not correlate seasonally growth rate 1 2 3 precipitation time, years 1 2 time, years 3 plant growth rate growth rate has no seasonal signal 1 summer drier than winter 2 3 precipitation time, years 1 2 time, years 3 plant growth rate growth rate high at times of peak precipitation 1 2 3 precipitation time, years 1 2 time, years 3 in this case times series correlated at short periods but not at long periods Coherence a way to quantify frequency-dependent correlation strategy band pass filter the two time series, u(t) and v(t) around frequency, ω0 compute their zero-lag cross correlation (large when time series are similar in shape) repeat for many ω0’s to create a function c(ω0) band pass filter f(t) has this p.s.d. 2Δω |f(ω)|2 2Δω ω -ω0 0 ω0 evaluate at zero lag t=0 and at many ω0’s Short Cut Fact 1 A function evaluates at time t=0 is equal to the integral of its Fourier Transform Fact 2 the Fourier Transform of a convolution is the product of the transforms integral over frequency integral over frequency assume ideal band pass filter that is either 0 or 1 negative frequencies positive frequencies integral over frequency assume ideal band pass filter that is either 0 or 1 negative frequencies positive frequencies c is real so real part is symmetric, adds imag part is antisymmetric, cancels integral over frequency assume ideal band pass filter that is either 0 or 1 negative frequencies positive frequencies c is real so real part is symmetric, adds imag part is antisymmetric, cancels interpret intergral as an average over frequency band integral over frequency can be viewed as an average over frequency (indicated with the overbar) Two final steps 1. Omit taking of real part in formula (simplifying approximation) 2. Normalize by the amplitude of the two time series and square, so that result varies between 0 and 1 the final result is called Coherence precipitation A) 5 0 0 200 400 600 800 1000 1200 1400 1600 T-air B) 20 10 0 new dataset: 0 200 400 600 800 1000 1200 1400 1600 T-water C) 10 salinity 0 0 turbidity D) 200 400 600 800 1000 1200 1400 1600 200 400 600 800 1000 time, days 1200 1400 1600 200 400 600 800 1000 time, days 1200 1400 1600 200 400 600 800 1000 time, days 1200 1400 1600 34 32 30 28 0 chlorophyll Water Quality 20 E) 20 0 0 F) 20 0 0 Reynolds Channel, Coastal Long Island, New York precipitation precipitation A) 5 0 0 200 400 600 800 1000 1200 1400 1600 air temperature T-air B) 20 10 0 new dataset: 0 200 400 600 800 1000 1200 1400 1600 water temperature T-water C) 10 salinity 0 0 turbidity D) E) 400 600 800 1000 1200 1400 1600 200 400 600 800 1000 time, days 1200 1400 1600 600 800 1000 time, days 1200 1400 1600 800 1000 time, days 1200 1400 1600 turbidity 20 0 200 salinity 34 32 30 28 0 chlorophyll Water Quality 20 0 F) 200 400 chlorophyl 20 0 0 200 400 600 Reynolds Channel, Coastal Long Island, New York B) periods near 5 days 800 1000 1200 1400 1600 1 0 -1 -2 400 450 500 550 800 1000 1200 1400 1600 4 2 0 -2 -4 400 450 500 550 1 0 -1 400 450 500 550 0.6 0.4 0.2 0 -0.2 -0.4 400 450 500 time, days 550 450 500 time, days 550 450 500 time, days 550 0.05 0 -0.05 precip precip A) periods near 1 year 0 200 400 600 T-air T-air 10 0 -10 -10 200 400 600 800 1000 1200 1400 1600 10 salinity salinity 600 0 0 -10 0 200 400 600 800 1000 1200 1400 1600 time, days 1 0 -1 -2 -3 turbidity turbidity 400 10 0 0 200 400 600 4 2 0 -2 -4 -6 0 200 400 600 800 1000 1200 1400 1600 time, days 2 0 -2 -4 400 800 1000 1200 1400 1600 time, days chlorophyl chlorophyl 200 T-water T-water 0 2 0 -2 400 Fig, 9.18. Band-pass filtered water quality measurements from Reynolds Channel (New York) for several years starting January 1, 2006. A) Periods near one year; and B) periods near 5 days. MatLab script eda09_16. B) periods near 5 days 800 1000 1200 1400 1600 1 0 -1 -2 400 450 500 550 800 1000 1200 1400 1600 4 2 0 -2 -4 400 450 500 550 1 0 -1 400 450 500 550 0.6 0.4 0.2 0 -0.2 -0.4 400 450 500 time, days 550 450 500 time, days 550 450 500 time, days 550 0.05 0 -0.05 precip precip A) periods near 1 year 0 200 400 600 T-air T-air 10 0 -10 -10 200 400 600 800 1000 1200 1400 1600 10 salinity salinity 600 0 0 -10 0 200 400 600 800 1000 1200 1400 1600 time, days 1 0 -1 -2 -3 turbidity turbidity 400 10 0 0 200 400 600 4 2 0 -2 -4 -6 0 200 400 600 800 1000 1200 1400 1600 time, days 2 0 -2 -4 400 800 1000 1200 1400 1600 time, days chlorophyl chlorophyl 200 T-water T-water 0 2 0 -2 400 Fig, 9.18. Band-pass filtered water quality measurements from Reynolds Channel (New York) for several years starting January 1, 2006. A) Periods near one year; and B) periods near 5 days. MatLab script eda09_16. 0.5 one year 0 C) precipitation and salinity 1 one year coherence coherence 1 0 B) air-temp and water-temp water-temp and chlorophyll 1 coherence A) 0.5 0.5 one week 0.05 0.1 0.15 0.2 frequency, cycles per day 0 0 0.05 0.1 0.15 0.2 frequency, cycles per day 0 0 0.05 0.1 0.15 0.2 frequency, cycles per day high coherence at periods of 1 year 0.5 one year 0 C) precipitation and salinity 1 one year coherence coherence 1 0 B) air-temp and water-temp water-temp and chlorophyll 1 coherence A) 0.5 0.5 one week 0.05 0.1 0.15 0.2 frequency, cycles per day 0 0 0.05 0.1 0.15 0.2 frequency, cycles per day moderate coherence at periods of about a month 0 0 0.05 0.1 0.15 0.2 frequency, cycles per day very low coherence at periods of months to a few days Part 2 windowing time series before computing powerspectral density scenario: -1 0 0 indefinitely 200 long phenomenon 400 you are studying an … 0 0.1 time t, s ASD of d ASD of W 0 0 -1 -10 0 but you only observe a 1 200 400 200 400 time t, s time t, s of it … short portion W(t) W(t)*d(t) 1 0 0 -1 -10 0 200 200 time t, s time t, s 400 400 ASD of W ASD of Wd) d(t) W(t) 1 1 10 5 5 0 00 0 0.1 0.1 4 5 2 0 00 0 0.1 0.1 how does the power spectral density of the short piece differ from the p.s.d. of the indefinitely long phenomenon (assuming stationary time series) -1 0 200 time t, s 00 0 200 200 200 time t, ss time time t, t, s 400 400 400 200 200 200 time time time t,t, t, ss s 400 400 400 200 200 time time t,t, ss 400 400 00 0 0.2 0.3 0.4 frequency f, Hz 00 000 0 0.1 0.1 0.1 0.2 0.3 0.4 0.2 0.4 0.2 0.3 0.3 0.4 frequency f, Hz frequency frequency f, f, Hz Hz 00 00 0.1 0.1 0.2 0.3 0.4 frequency f, Hz 55 5 We might suspect that the 4 difference will be 55 increasingly significant as 2 the window of observation 00 00 becomes so0.2short that it 0.1 0.3 0 0.1 0 0.1 0.2 0.3 0.4 0.4 frequency f,f, Hz frequency Hz includes just a few 44 oscillations of the period of 22 interest. ASD of of Wd) Wd) ASD W(t)*d(t) W(t)*d(t) 11 00 -1 -1 00 0.1 ASD of W ASD ASD ofof Wd) W W(t) W(t)*d(t) W(t) 11 1 -1 -1 -10 0 0 0 10 10 ASDofofdd ASD ASD of W d(t) W(t) d(t) 11 1 -1 -1 -100 0 0 400 starting point short piece is the indefinitely long time series multiplied by a window function, W(t) ASD of d d(t) 1 0 -1 0 200 time t, s 10 5 0 400 0 0.1 0.2 0.3 frequency f, 0 0.1 0.2 0.3 frequency f, 0 0.1 0.2 0.3 frequency f, ASD of W W(t) 1 0 -1 0 200 time t, s ASD of Wd) W(t)*d(t) 0 -1 0 400 1 0 200 time t, s 400 5 4 2 0 by the convolution theorem Fourier Transform of short piece is Fourier Transform of indefinitely long time series convolved with Fourier Transform of window function so Fourier Transform of short piece exactly Fourier Transform of indefinitely long time series when Fourier Transform of window function is a spike ASD of d d(t) 1 0 -1 10 5 0 0 200 400 boxcar window function 0 time t, s 0.2 0.3 0.4 its0.1Fourier Transform frequency f, Hz ASD of W W(t) 1 0 -1 0 200 time t, s ASD of Wd) W(t)*d(t) 0 -1 0 400 1 0 200 time t, s 400 5 0 0.1 0.2 0.3 0.4 frequency f, Hz 0 0.1 0.2 0.3 0.4 frequency f, Hz 4 2 0 ASD of d d(t) 1 0 -1 10 5 0 0 200 400 boxcar window function 0 time t, s ASD of W W(t) 1 0 -1 0 200 time t, s ASD of Wd) W(t)*d(t) 0 -1 0 400 1 0 200 time t, s 400 5 0.2 0.3 0.4 its0.1Fourier Transform frequency f, Hz sinc() function sort of spiky but has side lobes 0 0.1 0.2 0.3 0.4 frequency f, Hz 0 0.1 0.2 0.3 0.4 frequency f, Hz 4 2 0 ASD of d d(t) 1 0 -1 0 200 time t, s 10 5 0 400 0 0.1 0.2 0.3 0.4 frequency f, Hz 0 0.1 0.2 0.3 0.4 frequency f, Hz 0 0.1 0.2 0.3 0.4 frequency f, Hz ASD of W W(t) 1 0 -1 0 200 time t, s ASD of Wd) W(t)*d(t) 0 -1 0 400 1 0 200 time t, s 400 5 4 2 0 Effect 1: broadening of spectral peaks ASD of d d(t) 1 0 -1 0 200 time t, s 5 0 400 narrow spectral peak 10 0 ASD of W W(t) 1 0 -1 0 200 time t, s ASD of Wd) W(t)*d(t) 1 0 -1 0 200 time t, s 400 0.2 0.3 0.4 frequency f, Hz wide central spike 5 0 400 0.1 0 0.1 4 0.2 0.3 0.4 frequency f, Hz wide spectral peak 2 0 0 0.1 0.2 0.3 0.4 frequency f, Hz Effect 2: spurious side lobes ASD of d d(t) 1 0 -1 0 200 time t, s 5 0 400 only one spectral peak 10 0 ASD of W W(t) 1 0 -1 0 200 time t, s ASD of Wd) W(t)*d(t) 1 0 -1 0 200 time t, s 400 0.2 0.3 0.4 frequency f, Hz side lobes 5 0 400 0.1 0 0.1 4 spurious spectral peaks 2 0 0.2 0.3 0.4 frequency f, Hz 0 0.1 0.2 0.3 0.4 frequency f, Hz Q: Can the situation be improved? A: Yes, by choosing a smoother window function more like a Normal Function (which has no side lobes) but still zero outside of interval of observation -1 5 0 -1 0 0 200 400 boxcar window function 200 time t, s ASD of Wd) W(t)*d(t) W(t)*d(t) 0 -1 0 200 time t, s 400 5 0 0-1 00 400 1 1 ASD of W W(t) ASD of W W(t) 0 5 frequency f, Hz time t, s 1 -1 10 0.1 0.2window 0.3 0.4function 0 Hamming 0 200 400 0 time t, s 0 0 0.1 41 20 0-1 00 0.1 200 0.3 400 0.2 0.4 time t,f,sHz frequency 4 2 0 0.2 0.4 200 0.3 400 frequency time t,f,sHz ASD of Wd) 0 ASD of d 10 1 ASD of d d(t) d(t) 1 0 2 1 0 0 ASD of d d(t) 1 0 -1 0 200 time t, s ASD of W W(t) 1 -1 0 200 time t, s ASD of Wd) W(t)*d(t) 0 -1 0 200 time t, s 400 0 0.1 0.2 0.3 0.4 frequency f, Hz 0 0.1 0.2 0.3 0.4 frequency f, Hz 0 0.1 0.2 0.3 0.4 frequency f, Hz 4 2 0 400 1 5 0 400 0 10 2 1 0 ASD of d d(t) 1 0 -1 0 200 time t, s ASD of W W(t) 1 -1 0 200 time t, s ASD of Wd) W(t)*d(t) 0 -1 0 200 time t, s 400 0 0.1 0.2 0.3 0.4 frequency f, Hz 0 0.1 0.2 0.3 0.4 frequency no f, Hzside 4 2 0 400 1 5 0 400 0 10 2 1 0 0 0.1 lobes but central peak wider than with boxcar 0.3 0.4 0.2 frequency f, Hz Hamming Window Function Q: Is there a “best” window function? A: Only if you carefully specify what you mean by “best” (notion of best based on prior information) “optimal”window function maximize ratio of power in central peak (assumed to lie in range ±ω0 ) to overall power The parameter, ω0, allows you to choose how much spectral broadening you can tolerate Once ω0 is specified, the problem can be solved by using standard optimization techniques One finds that there are actually several window functions, with radically different shapes, that are “optimal” (t) W3(t) v w3(t) W2(t) v w2(t) W1(t) v w1(t) Family of three “optimal” window functions 0.2 0 -0.2 0 10 20 0 10 20 0 10 20 30 timev t, s 40 50 60 30 time, s time v t, s 40 50 60 30 timev t, s 40 50 60 time, s 0.2 0 -0.2 0.2 0 -0.2 0.2 time, s a common strategy is to compute the power spectral density with each of these window functions separately and then average the result technique called Multi-taper Spectral Analysis |d(f)| -0.5 400 w1(t)d(t) 400 400 0.4 0.2 0 500 0 400 0.2 0 500 0 v w2(t)d(t) W1(t)d(t) 0.2 0.1 0 -0.1 0 100 200 300 time t, s v time t, s w3(t)d(t) W2(t)d(t) 0.1 0 -0.1 0 100 200 300 time t, s v time t, s W3(t)d(t) 0.1 0 -0.1 0 100 200 300 time t, s time t, vs 0.1 0.2 0.3 0.4 frequency v f, Hz |d0(f)| 200 300 time t, s t,vs time 100 50 0 500 0 |d1(f)| d(t) 0 -0.5 100 0 frequency, Hz 1 B(t)d(t)0.5 0 500 400 0.1 0.1 0.2 0.3 0.4 frequency v f, Hz frequency, Hz 0.5 0.2 0.3 0.4 v f, Hz frequency frequency, Hz |d2(f)| 200 300 time t, s t,vs time 0.1 0.2 0.3 0.4 frequency v f, Hz frequency, Hz |d3(f)| 100 0.2 0 500 0 |d(f)|avg 0 20 10 0 v d(t) 1 d(t) 0.5 0 0.2 0.1 0 0 0.1 0.1 0.2 0.3 0.4 frequency v f, Hz frequency, Hz 0.2 0.3 0.4 frequency v f, Hz frequency, Hz 0.5 v |d(f)| -0.5 400 w1(t)d(t) 400 400 0.4 0.2 0 500 0 400 0.2 0 500 0 v w2(t)d(t) W1(t)d(t) 0.2 0.1 0 -0.1 0 100 200 300 time t, s v time t, s w3(t)d(t) W2(t)d(t) 0.1 0 -0.1 0 100 200 300 time t, s v time t, s W3(t)d(t) 0.1 0 -0.1 0 100 200 300 time t, s time t, vs 0.1 0.2 0.3 0.4 frequency v f, Hz |d0(f)| 200 300 time t, s t,vs time 100 50 0 500 0 |d1(f)| d(t) 0 -0.5 100 0 frequency, Hz 1 B(t)d(t)0.5 0 500 400 0.1 0.1 0.2 0.3 0.4 frequency v f, Hz frequency, Hz 0.5 0.2 0.3 0.4 v f, Hz frequency frequency, Hz |d2(f)| box car tapering 200 300 time t, s t,vs time 0.1 0.2 0.3 0.4 frequency v f, Hz frequency, Hz |d3(f)| 100 0.2 0 500 0 |d(f)|avg 0 20 10 0 v d(t) 1 d(t) 0.5 0 0.2 0.1 0 0 0.1 0.1 0.2 0.3 0.4 frequency v f, Hz frequency, Hz 0.2 0.3 0.4 frequency v f, Hz frequency, Hz 0.5 v |d(f)| -0.5 0 100 200 300 time t, s t,vs time 400 d(t) 0 -0.5 100 200 300 time t, s t,vs time 0 0.1 0.2 0.3 0.4 frequency v f, Hz frequency, Hz 1 B(t)d(t)0.5 0 500 20 10 0 400 |d0(f)| d(t) 1 d(t) 0.5 0 100 50 0 500 0 0.1 0.2 0.3 0.4 400 0.2 0 500 0 w3(t)d(t) W2(t)d(t) 0.1 0 -0.1 0 100 200 300 time t, s v time t, s W3(t)d(t) 0.1 0 -0.1 0 100 200 300 time t, s time t, vs 400 0.1 0.2 0.3 0.4 v f, Hz frequency frequency, Hz |d2(f)| 200 300 time t, s v time t, s 0.5 0.1 0.2 0.3 0.4 frequency v f, Hz frequency, Hz |d3(f)| 100 0.2 0 500 0 |d(f)|avg 0 -0.1 0 |d1(f)| 400 0.4 0.2 0 500 0 v w2(t)d(t) W1(t)d(t) 0.2 0.1 v w1(t)d(t) Hz tapering with three “optimal” windowfrequency, functions frequency v f, Hz 0.2 0.1 0 0 0.1 0.1 0.2 0.3 0.4 frequency v f, Hz frequency, Hz 0.2 0.3 0.4 frequency v f, Hz frequency, Hz 0.5 v |d(f)| -0.5 400 w1(t)d(t) 400 400 0.4 0.2 0 500 0 400 0.2 0 500 0 v w2(t)d(t) W1(t)d(t) 0.2 0.1 0 -0.1 0 100 200 300 time t, s v time t, s w3(t)d(t) W2(t)d(t) 0.1 0 -0.1 0 100 200 300 time t, s v time t, s W3(t)d(t) 0.1 0 -0.1 0 100 200 300 time t, s time t, vs 0.1 0.2 0.3 0.4 frequency v f, Hz |d0(f)| 200 300 time t, s t,vs time 100 50 0 500 0 |d1(f)| d(t) 0 -0.5 100 0 frequency, Hz 1 B(t)d(t)0.5 0 500 400 p.s.d. produced by averaging 0.1 0.1 0.2 0.3 0.4 frequency v f, Hz frequency, Hz 0.5 0.2 0.3 0.4 v f, Hz frequency frequency, Hz |d2(f)| 200 300 time t, s t,vs time 0.1 0.2 0.3 0.4 frequency v f, Hz frequency, Hz |d3(f)| 100 0.2 0 500 0 |d(f)|avg 0 20 10 0 v d(t) 1 d(t) 0.5 0 0.2 0.1 0 0 0.1 0.1 0.2 0.3 0.4 frequency v f, Hz frequency, Hz 0.2 0.3 0.4 frequency v f, Hz frequency, Hz 0.5 v Summary always taper a time series before computing the p.s.d. try a simple Hamming taper first it’s simple use multi-taper analysis when higher resolution is needed e.g. when the time series is very short