3. Massaging and Manipulating Data
15000
10000
Amplitude
5000
0
0
0.2
0.4
0.6
0.8
-5000
-10000
-15000
Time (s)
Sample
Sample
Prep
Instrument
Instrument
Out put
A signal can
contain noise at a different frequency
drift with time
Distorts the “real” information about the sample
Signal (Data)
1
1.2
Consider
a system where our signal is a sin wave
15000
10000
Amplitude
5000
0
0
0.2
0.4
0.6
-5000
-10000
-15000
Time (s)
It might have “noise” and it might “drift” with time
0.8
1
1.2
Consider a Sin wave as the signal
Add some noise
Composite signal contains noise
A signal can
contain noise at a different frequency
drift with time
0.25
0.2
Signal
0.15
Amplitude
0.1
0.05
0
0
0.05
0.1
0.15
0.2
0.25
-0.05
-0.1
-0.15
-0.2
-0.25
Time
Drift
0.3
0.35
0.4
0.45
0.5
The signal is superimposed on slowly changing background and
Slowly decays
0.25
0.2
0.15
Amplitude
0.1
0.05
0
0
0.05
0.1
0.15
0.2
0.25
-0.05
-0.1
-0.15
-0.2
-0.25
Time
0.3
0.35
0.4
0.45
0.5
Enhancing the Signal of the desired Frequency
Make use of:
Summing “in phase” signals
Statistics (std decreases as the sq rt of the Number of samples)
Std = standard deviation, s
Sq rt = square root
What does “phase” mean?
180 degrees out of phase
90 degrees out of phase
15000
15000
10000
10000
Amplitude
Amplitude
5000
5000
00
00
0.2
0.2
0.4
0.4
0.6
0.6
-5000
-5000
-10000
-10000
-15000
-15000
Time
Time (s)
(s)
0.8
0.8
11
1.2
1.2
Summing in phase sin waves leads to signal amplification
25000
25000
20000
20000
15000
15000
Amplitude
Amplitude
10000
10000
5000
5000
00
00
0.1
0.1
0.2
0.2
0.3
0.3
0.4
0.4
0.5
0.5
-5000
-5000
-10000
-10000
-15000
-15000
-20000
-20000
-25000
-25000
time
time (s)
(s)
Will make use of this concept to enhance Signal
0.6
0.6
0.7
0.7
0.8
0.8
0.9
0.9
11
This is the desired signal enhancement that comes from summing the entire
waveform
25000
25000
If the summed waveforms are
Slightly offset (out of phase)
20000
20000
The “enhanced” signal is distorted
15000
15000
Amplitude
10000
10000
5000
5000
00
00
0.2
0.2
0.4
0.4
0.6
0.6
0.8
0.8
11
1.2
1.2
-5000
-5000
-10000
-10000
-15000
-15000
-20000
-20000
-25000
-25000
tt
This means that we design instruments to “trigger” the data signal acquisition timing
to avoid distortion
Sin wave Plus 180o out of phase wave = Destructive Summation
15000
15000
10000
10000
Amplitude
5000
5000
00
00
0.2
0.2
0.4
0.4
0.6
0.6
0.8
0.8
11
1.2
1.2
-5000
-5000
-10000
-10000
-15000
-15000
tt (s)
(s)
Will make use of this concept to reduce noise. In noise if it is random ½ of time it will
Fluctuate positive and ½ time will fluctuate negative, thus is should sum “destructively)
 pp  12892  7690
N s~  
 867
 6
6
x sample  xblank x sample  xblank
S


N
s
 pp 
 
 6
15000
12892
10000
7690
amplitude
5000
0
0
100
200
300
400
500
600
-5000
-10000
-15000
time or point
S  x sample  xblank
S  10000   10000  20000
S 20000

 23
N
867
Sum of 2, 4, and 8 waveforms
100000
100000
91435
80000
60000
80000
40000
amplitude
73756
60000
20000
0
0
100
200
300
400
500
600
-20000
-40000
-60000
40000
-80000
-100000
amplitude
time or point
20000
0
0
100
200
300
400
500
600
-20000
-40000
-60000
-80000
-100000
S
160000
160000


 54.3
91435

73756
N 
2946





6
time or point
We went from S/N of 23 with 1 waveform to
54.2 with 8 summed waveforms
# summed
waveforms
S/N
Sqrt # summed
waveforms
1
S 20000

 23
N
867
1
4
S
 46.4
N
2
8
S
160000
160000


 54.3
91435

73756
N 
2946





6
4/1 = 4
46.4/23=2.01
8/1=8
54.3/23 = 2.35
2.828
2/1=2
2.828/1=2.82
Observations?
What implications does this have for designing an instrument?
designing a method (quality control)?
Build instruments with
computers
fast data acquisition time (micro to milli-seconds)
Sum in phase signals over and over and over and over again
Massage the Data after acquisition (Digitally)
D
time
4
0.1
0.11
0.12
0.13
0.14
0.15
0.16
0.17
0.18
0.19
0.2
0.21
0.22
0.23
0.24
"signal"
-0.03185
-0.03503
-0.03821
-0.0414
-0.04458
-0.04776
-0.05094
-0.05412
-0.0573
-0.06048
-0.06366
-0.06684
-0.07002
-0.0732
-0.07637
noise
0.174321
-0.01688
-0.18818
-0.14037
0.070015
0.199355
0.099605
-0.11379
-0.19752
-0.05792
0.146468
0.187003
0.020172
-0.16849
-0.17265
sum
5 box
0.142473
-0.05191
-0.22639 -0.05843
-0.18177 -0.05661
0.025435 -0.03649
0.151594
-0.0248
0.048663 -0.03941
-0.16791 -0.06818
-0.25483 -0.08194
-0.11841 -0.06764
0.082805 -0.04402
0.120162 -0.04139
-0.04985 -0.06752
-0.24168 -0.09792
-0.24902 -0.10216
7
-0.01313
-0.05747
-0.08646
-0.07103
-0.03324
-0.0197
-0.04848
-0.08996
-0.10155
-0.07503
-0.04398
-0.04536
9
=AVERAGE(D4:D8)
-0.05718
-0.08617
-0.0712
-0.0327
-0.01804
-0.04772
-0.09223
-0.10533
-0.07567
-0.03924
-0.03949
=AVERAGE(D5:D9)
The “box” slides down=
Sliding box car filter
Raw signal
0.2
0.2
5 point
0.1
0.1
Sliding
boxcar
00
Amplitdue
Amplitdue
00
0.1
0.1
0.2
0.2
0.3
0.3
0.4
0.4
0.5
0.5
-0.1
-0.1
-0.2
-0.2
-0.3
-0.3
-0.4
-0.4
7 point
Sliding box
car
-0.5
-0.5
time
time (s)
(s)
9 sliding box car gets much closer to the “real” signal.
0.6
0.6
0.7
0.7
0.8
0.8
Now that you have tried it I can tell you how to do it automatically
Right click the data
Add trendline
Specify the number
Of points in the
Moving average
Fourier Transform –
eliminate high frequencies
Consider a series of sin waves of
Differing amplitudes and frequencies
1.5
Sin(f)
Base frequency
1/3sin(3f)
1/5sin(5f)
Multiples of the base
frequency
1/7(sin(7f)
1
Voltage
0.5
0
0
0.2
0.4
0.6
-0.5
-1
-1.5
Time
Sum the sin waves and what do you get?
0.8
1
1.2
What do you observe?
90%
Sin(f)
Sin(f) +1/3sin(3f)
Sin(f) +1/3sin(3f) +1/5sin(5f)
Sin(f) +1/3sin(3f) +1/5sin(5f)+
1/7(sin(7f)
10%
Rise time
The fact that any waveform can be considered to be the sum of a large number of
Frequencies allows us to do some nifty tricks to massage out noise
5000
4000
3000
Amplitude
2000
1000
0
0
0.2
0.4
0.6
0.8
1
1.2
-1000
-2000
-3000
-4000
-5000
Time
Measurement  A1 sin2f s,1t 
 N 1 sin215 f N ,1t 
Signal
Noise
Noise
160
Digitally filter the high frequency
140
120
5000
100
Amplitude
4000
3000
60
1000
0
0.2
0.4
0.6
0.8
1
40
FT
0
1.2
20
-1000
-2000
0
1
-3000
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Frequency
-4000
-5000
Time
subtract
noise
Signal wave
10000
4000
9000
3000
8000
2000
Anti-FT
0
0
0.2
0.4
0.6
-1000
0.8
1
1.2
relative amplitude
7000
1000
Amplitude
Amplitude
2000
80
6000
5000
4000
3000
2000
-2000
1000
-3000
0
1
-4000
Time
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20
frequency
Running an FT in excel
F f  
j



f  t  exp
 2jft
dt
1
Euler’s theorem
exp
 cos x  j sin x
 jx
F f  



N
f  t  cos2ft dt  j 

f  t  sin2ft dt


Discrete Fourier transform, k must be an integer power of 2:
Allowed values are 2,4,8,16,32,64,128,256,512,1024)
F f  
k 0
N
 2k 
f  t  cos
 dt  j  k  0 f  t  sin2ft dt
 N 
The digital FT performed by excel does not make reference to your measured time, but
treats your data as a string of data from n to 2n.
In cell a1 type a title: base frequency
In cell b1 base frequency = 1
In cell a2 type a title: amplitude
In cell b2 assign a value to the amplitude
In cell c1 type a title: noise frequency
In cell c2 type a title: noise amplitude
In cell d1 assign a value to noise frequency
In cell d2 assign a value to noise amplitude
2. Create a counter from 1 to 512 in
cell A7 (label cell a5= time)
3. Create the actual normalized
time in seconds so that your
base frequency will just
complete one full cycle in the
512 time points.
=A7/512
4000
3000
2000
4. Create the sin wave starting in cell
C7 =$B$2*SIN(2*PI()*$B$1*B7)
Amplitude
1000
0
0
0.2
0.4
0.6
-1000
-2000
 V o sin2ft 
-3000
-4000
Time
0.8
1
1.2
6.
Create some noise to filter out
In cell D7: =$D$2*sin(2*PI()*$D$1*B7)
7. Sum the signal (column C) and the
noise (column D). Here I have it in
column E.
In cell 76: =C7+D7
5000
4000
3000
Amplitude
2000
1000
0
0
0.2
0.4
0.6
-1000
-2000
-3000
-4000
-5000
Time
0.8
1
1.2
Open the tools pack and go to Fourier transform. Specify the 512 data points associated
with the signal plus noise in column E. Tell it to place the transformation in column F.
signal
The base frequency does not
Always represent the signal
Frequency, see next example
Noise
Lg amplitude
At 15x
.
The data placed into column F will be complex number representation of the amplitude
of the frequency. It consists of the base frequency and the noise frequency at 15x the
base. The first point in column E will represent 0 frequency, or DC. Each
sequential box will represent the amplitude of frequency 1 (1cycle/512 second),
frequency 2 and so forth up to point 256 at which point the numbers display the
amplitude of each decreasing frequency
The data displayed is a complex number. To convert to something you can graph create
a column J ( =imreal(F7) )
Create a histogram of column J.
You do not need to use the
Histogram package in Excel or
to create bins here . Simply
highlight the data in column J
including only the 2nd through the
~ 30 data points (base frequency
to frequency multiple 30). Using
that single column insert a bar
graph. The computer will
automatically assign numbers 1,
2..... to your x axis.
160
140
120
Amplitude
100
80
60
40
20
0
1
2
3
4
5
6
7
8
9
10
11
Frequency
12
13
14
15
16
17
18
19
To filter the data create column H. Every number in column H should be zero except for
the point representing the base frequency (point 8 in column F). Copy the complex
number in column F over to column H. All other values should be zero.
10000
9000
8000
relative amplitude
7000
6000
5000
4000
3000
2000
1000
0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20
frequency
We made the amplitude
Of the 15x frequency
Go to zero
The second half of the FFT
output is
A mirror image of the first
half, so
You have to make the 15x
frequency
Amplitude go to zero at the
bottom too!
Now re-create a time string of data by performing the inverse fft on column H. To do this
go to tools, Fourier Transform. In the Fourier Transform command box you click inverse.
Tell the computer to place the data in column I.
This data needs to be
converted from the complex
notation.
Type in column j
=IMREAL(i7)
Now plot the data in column M as an xy scatter plot. You should see a beautiful noise
free sin wave identical to the original noise free sin wave you generated or acquired.
5000
4000
3000
Amplitude
2000
1000
0
0
0.2
0.4
0.6
-1000
-2000
-3000
-4000
Fft filtered
-5000
Time
0.8
1
1.2
Another Example in which the Signal waveform is the sum of several sin waves
15000
10000
Amplitude (V)
5000
0
0
0.2
0.4
0.6
0.8
1
1.2
-5000
-10000
-15000
Measurement  A1 sin2f s,1t   As sin2f S ,Time
A sin2f S ,n t 
2 t ....
(s) n
 N 1 sin2f N ,1t   N 2 sin2f N ,2 t ..... N n ,1 sin2f N ,n t 
Signal
Noise
50
Noise
45
Digitally filter the high frequency
40
35
Amplitude
15000
10000
30
25
20
15
0
0
0.2
0.4
0.6
0.8
1
FT
1.2
10
5
-5000
0
1
-10000
7
13
19 25
31
37
43 49
55
61
67 73
79
85
91 97 103 109 115 121
85
91
97 103 109 115 121
frequency
-15000
Time (s)
subtract
noise
Square wave
2
1.5
1
3.5
0.5
3
0
2.5
0
0.2
0.4
0.6
-0.5
0.8
1
1.2
Anti-FT
-1
-1.5
Amplitude
Amplitude
Amplitude (V)
5000
2
1.5
1
-2
Time (s)
0.5
0
1
7
13
19
25
31
37
43
49
55
61
67
Frequency
73
79
The next example is very complex time
Varying signal.
The example also shows what can happen if
You cut too many high frequency components
This part of the response
Requires high frequency
To define the response well
2048 Data points = 211
Total time = 18.5 s
0.12
0.1
0.08
1.4
0.12
Current, uA
0.06
0.04
0.02
0.7
1.2
0.1
0
0.8
0.9
1
1.1
1.2
1.3
-0.02
-0.04
0.08
-0.06
-0.08
V vs Ag/AgCl
0.06
0.04
0.8
0.02
0.6
0
-0.02
0.4
-0.04
0.2
-0.06
0
-0.08
0
2
4
6
8
10
12
14
Time (s)
Signal  A1 sin2f s,1t   As sin2f S ,2 t .... An sin2f S ,n t 
 N 1 sin2f N ,1t   N 2 sin2f N ,2 t ..... N n ,1 sin2f N ,n t 
Signal
Noise
16
18
20
Current (microAmps)
V vs Ag/AgCl
1
1.4
Measurement  A1 sin2f s,1t   As sin2f S ,2 t .... An sin2f S ,n t 
30
 N 1 sin2f N ,1t   N 2 sin2f N ,2 t ..... N n,1 sin2f N ,n t 
20
30
Amplitude
10
20
0
1
-10
2
3
4
5
6
7
8
9
10
11
12
DC
-20
-30
10
-40
Amplitude
Frequency
0
1
10 19 28 37 46 55 64 73 82 91 100 109 118 127 136 145 154 163 172 181
-10
-20
-30
cycles
f 
s
1cycle  18.5s
1cycle
f base 
 0.05405Hz
18.5s
-40
Frequency
Signal is composed of
Frequency multiples of
The base up to about
28x base frequency
Base frequency is defined
By the length of the data
String subjected to FFT
Measurement  A1 sin2f s,1t   As sin2f S ,2 t .... An sin2f S ,n t 
 N 1 sin2f N ,1t   N 2 sin2f N ,2 t ..... N n,1 sin2f N ,n t 
30
20
0
1
10 19 28 37 46 55 64 73 82 91 100 109 118 127 136 145 154 163 172 181
30
-10
20
-20
10
-30
-40
Amplitude
Amplitude
10
0
1
10 19 28 37 46 55 64 73 82 91 100 109 118 127 136 145 154 163 172 181
-10
Frequency
-20
0.12
Note distortion where there is
Is a rapid change in the voltage signal
Distortion is due to high frequency removal
0.1
0.08
Current, uA
0.06
0.04
0.02
0
0.7
0.8
0.9
1
1.1
1.2
1.3
-0.02
Remainder of signal
Is well matched with
Information content
retained
-0.04
-0.06
-0.08
V vs Ag/AgCl
1.4
Measurement  A1 sin2f s,1t   As sin2f S ,2 t .... An sin2f S ,n t 
 N 1 sin2f N ,1t   N 2 sin2f N ,2 t ..... N n,1 sin2f N ,n t 
30
20
0
1
10 19 28 37 46 55 64 73 82 91 100 109 118 127 136 145 154 163 172 181
-10
30
-20
20
Clipped all but 5xbase frequency
-30
-40
10
Amplitude
Amplitude
10
0
1
-10
10 19 28 Frequency
37 46 55 64 73 82 91 100 109 118 127 136 145
When we leave only 5x base frequency we have completely
Distorted the underlying signal
0.12
0.1
0.08
Current, uA
0.06
0.04
0.02
0
0.7
0.8
0.9
1
1.1
-0.02
-0.04
-0.06
-0.08
V vs Ag/AgCl
1.2
1.3
1.4
Electronically Filter before digitization
q
 C
V

I c  V C2f cos2ft 
dq
I
dt
o
V  IR
dq d  CV 
 dV 
I

 C

 dt 
dt
dt
V  V sin(2ft )

V  IC Xc
o
o
V
IC 
 V o 2fC cos2ft 
Xc

d sin(2ft )
dV
o
V
dt
dt
1

dV
o
 V 2f cos2ft 
dt
 2fC cos2ft 
Xc
Amplitude only:
1
Xc 
2fC
V
V1  IR1
Vout
Voltage dropped from
Here to ground
V2  IR2
ground
V2
IR2
R2


Vtotal IR1  IR2 R1  R2

V
V1  IR1
Vout
I Xc
Xc

 

Vtotal IR  I X
R1  X c
1
c
R1 
V2  IX c
ground

 X  
 c

R12   X c 

1
Xc 
2fC
Vo

Vtotal
1
2fC
 1 
R 

 2fC 
2
1
2
2
1
2fC
V
V1  IR1
V2  IX c
ground
Vo

Vtotal
Vo

Vtotal
Vo

Vtotal
 1 
R 

 2fC 
2
2
1
1
 1 
2fC R  

 2fC 
2
1
1
2fCR 
2
1
Vo

Vtotal
 2fC 


 2fC 
1
2fCR 
1
2
1
2
2
15000
15000
10000
10000
Amplitude
Amplitude
5000
5000
After analog filtering
0
00
-5000
-5000
0.2
0
0.4
0.2
0.6
0.4
0.8
0.6
Vo
-10000
-10000
-15000
-15000
Time (s)
Time (s)
1
0.8
1.2
1
1.2
This is a “Bode Plot”
Cut off frequency