The Best Method of Noise Filtering
Yuri Kalambet, Sergey Maltsev,
Ampersand Ltd., Moscow, Russia;
Yuri Kozmin,
Shemyakin Institute of Bioorganic Chemistry,
Moscow, Russia
kalambet@ampersand.ru
1
History: Adaptive peak approximation
2
Rough slope width estimate
• Evaluate baseline using
default gap (minimum peak
width Integration
parameter)
• Evaluate peak height using
default gap
• Count all points from peak
apex to slope end with
height bigger than halfheight of the peak. Count
obtained is an estimate of
the slope width.
3
Properties of adaptive peak
approximation
•
•
•
•
Good noise suppression at each slope
Minimal peak shape disturbances
All peak parameters are resistant to oversampling
Baseline approximation may be poor – either
noisy (small gap) or disturbed (large gap).
• No approximation outside of peaks
• Does not improve formal signal/noise ratio
• Baseline position is one of the most important
sources of error
4
Improvement 1:
Non-central approximation
x*
G2
G1
5
Height
Confidence intervals
8
9
7
8
6
4
2
3
1
2
6
4
5
Concentration
0
0
1
2
3
4
5
6
7
8
9
10
6
Confidence interval estimate
CY  t n(1/p2)  S  u*
where
1
(Y  Xβˆ )  (Y  Xβˆ )


2
u*  x* (X  X) x*
S 
n p
n - number of data points used for polynomial approximation (gap of the
filter);
p - power of the polynomial;
X - matrix of x power values on independent axis (time);
Y - vector of detector response values;
x*  {1, x* ,...,x*p }
βˆ  ( X X) 1 X  Y
tm - Student’s coefficient for confidence probability (1-δ) and m degrees of
freedom
x* - position at which smoothed (approximated) value is estimated.
7
Approximation using confidence
intervals
G1
G2
x*
x*
G 2 confidence interval
8
Algorithm of simple Confidence filter
approximation
• Evaluate points and confidence intervals for new (shifted)
window
9
Algorithm of simple Confidence filter
approximation
• Evaluate points and confidence intervals for new (shifted)
window
• Compare new confidence interval with that for previously
evaluated point. If the new one is smaller than previous,
replace approximated point and its confidence interval.
10
Algorithm of simple Confidence filter
approximation
• Evaluate points and confidence intervals for new (shifted)
window
• Compare new confidence interval with that for previously
evaluated point. If the new one is smaller than previous,
replace approximated point and its confidence interval.
• Computational complexity of Confidence filter is comparable
to that of simple convolution, (e.g. Savitzky-Golay) and
linearly depends on the product
gap∙ (degree of the polynomial).
11
Bonus #1: Correct handling of baseline steps
and array boundaries
mv
Original
SG
ASG
2000
1500
1000
500
0
340
350
360
370
380
390
400
410
420
430
440
450
460
470
480
490
500
Nmeas
dotted – raw data; thick line – Confidence Filter;
thin line – Savitzky-Golay filter
12
Confidence filter algorithm improvement:
Adaptive gap of the polynomial
• Repeat confidential filter algorithm for
approximations with different windows (gaps)
• Computational complexity:
degree∙gap∙(gap-1)/2
• Logarithmic step: next gap is k times smaller,
than previous, e.g. gap2 = gap1/k, k>1;
Computational complexity: degree∙gap∙k/(k-1)
13
Confidence interval estimate
CY  t n(1/p2)  S  u*
where
1
(Y  Xβˆ )  (Y  Xβˆ )


2
u*  x* (X  X) x*
S 
n p
n - number of data points used for polynomial approximation (gap of the
filter);
p - power of the polynomial;
X - matrix of x power values on independent axis (time);
Y - vector of detector response values;
x*  {1, x* ,...,x*p }
βˆ  ( X X) 1 X  Y
tm - Student’s coefficient for confidence probability (1-δ) and m degrees of
freedom
x* - position at which smoothed (approximated) value is estimated.
14
t(df) for confidence probability 0.975
15
Confidence interval profiles for different slits
(degree = 3)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
-15
-14
-13
-12
-11
-10
-9
-8
-7
-6
-5
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Confidence Interval profiles, 31 points, 0…5 degrees
17
σ evaluation problems:
• Small gaps: accidental perfect fit
• Large gaps: treating small peaks as a noise due to
large number of degrees of freedom
• Is pump pulsation a noise or a signal?
• Small gaps: confidence interval depends on
confidence level
σ evaluation solutions:
• Evaluate in advance using the whole data array
• Use the estimate for evaluation of confidence
intervals
18
Handling σ estimate
CY  tn(1/p2)  S  u*
CY  tn½p    u*
S 2   2 , Form ula(a ) 
CY   2

2
S


,
Form
ula
(
b
)


SR2   2   2 ( , n  p)
19
Noise Filtering: How it works 1
23
AU
-0.004
280nm
-0.005
22
26
S moo280
27
25
21
20
21
22
23
24
25
min
20
22
26
23
25
10
Gap
21
24
23
0
20
27
21
22
24
25
min
20
27
22
0
S hift
21
23
25
24
26
-20
20
21
22
23
24
25
min
20
Noise Filtering: How it works 2
21
0 .0 7 8 4
mV
ch 1
33
0 .0 7 3 3
36
37
38
39
40
41
min
Raw
34
35
34
35
36
37
38
39
40
41
m in
34
35
36
37
38
39
40
41
m i n
34
35
36
37
38
39
40
41
m in
34
35
36
37
38
39
40
41
m in
34
35
36
37
38
39
40
41
min
m V
ch 1
33
0 .0 7 5
0…3
Automatic selection of
degree and gap of
approximating
polynomial
m V
ch 1
33
0 .0 7 7 3
1…3
m V
ch 1
33
0 .0 7 8 4
2...3
m V
ch 1
33
0 .0 7 8 4
3
mV
ch 1
33
3…10
22
Is pump pulsation a noise or a signal?
23
Conclusions:
• Confidence filter introduces a measure of
approximation quality
• Confidence filter helps to select the best set of
functions that approximate the data set
• Confidence filter is metrologically the best noise
filtering method and can be used in the fight with
legal metrology
Patent pending
24
Thank you!
25