Minimal bounds on the Mean Square Error: A Tutorial
Alexandre Renaux
Washington University in St. Louis
Friday 23rd February 2007
1/70
Alex
Outline
Minimal bounds on the Mean Square Error
1
Framework and motivations
2
Minimal bounds on the MSE: unification
3
Minimal bounds on the MSE: application
4
Conclusion and perspectives
Friday 23rd February 2007
2/70
Alex
Outline
1
Framework and motivations
2
Minimal bounds on the MSE: unification
3
Minimal bounds on the MSE: application
4
Conclusion and perspectives
Friday 23rd February 2007
3/70
Alex
1
Framework and motivations
Statistical signal processing
Extract informations
(estimation)
Applications:
Radar/Sonar
Digital communications
Medical imaging
Astrophysic
…
Friday 23rd February 2007
4/70
Alex
1
Framework and motivations
Statistical framework
Parameters
space
Friday 23rd February 2007
Observations
space
5/70
Alex
1
Framework and motivations
Performances
’’Distance’’ between
r.v.
and
Estimates distibution
Mean Square Error
Friday 23rd February 2007
Bias
Variance
6/70
Alex
1
Framework and motivations
Performances: Cramér-Rao inequality
Estimates distribution
For
unbiased
with
Fisher Information
Matrix
Cramér-Rao
If equality, then
efficient estimator
Friday 23rd February 2007
7/70
Alex
1
Framework and motivations
Context
Maximum Likelihood estimators
Direction of Arrivals estimation
Frequency estimation
The parameters support is finite
Friday 23rd February 2007
8/70
Alex
1
Framework and motivations
Mean Square Error (dB)
MSE behavior of ML estimator: 3 areas
Non-Information
SNRT
Friday 23rd February 2007
Rife and Boorstyn 1974
Van Trees 1968
9/70
Signal to Noise Ratio (dB)
Alex
1
Framework and motivations
Single frequency estimation (100 observations)
SNR = 10 dB
MSE
Run 1
SNR
Normalized frequency
Friday 23rd February 2007
10/70
Alex
1
Framework and motivations
Single frequency estimation (100 observations)
SNR = 10 dB
MSE
Run 2
SNR
Normalized frequency
Friday 23rd February 2007
11/70
Alex
1
Framework and motivations
Single frequency estimation (100 observations)
SNR = 10 dB
MSE
Run 20
SNR
Normalized frequency
Friday 23rd February 2007
12/70
Alex
1
Framework and motivations
Single frequency estimation (100 observations)
SNR = -6 dB
MSE
Run 1
SNR
Normalized frequency
Friday 23rd February 2007
13/70
Alex
1
Framework and motivations
Single frequency estimation (100 observations)
SNR = -6 dB
MSE
Run 2
SNR
Normalized frequency
Friday 23rd February 2007
14/70
Alex
1
Framework and motivations
Single frequency estimation (100 observations)
SNR = -6 dB
Run 7
MSE
Outlier
SNR
Normalized frequency
Friday 23rd February 2007
15/70
Alex
1
Framework and motivations
Single frequency estimation (100 observations)
SNR = -20 dB
Outlier
MSE
Run 1
SNR
Normalized frequency
Friday 23rd February 2007
16/70
Alex
1
Framework and motivations
Single frequency estimation (100 observations)
SNR = -20 dB
Run 2
MSE
Outlier
SNR
Normalized frequency
Friday 23rd February 2007
17/70
Alex
1
Framework and motivations
Single frequency estimation (100 observations)
SNR = -20 dB
Run 20
MSE
Outlier
SNR
Normalized frequency
Friday 23rd February 2007
18/70
Alex
Mean Square Error (dB)
1
Framework and motivations
-
Non-Information
SNRT
Friday 23rd February 2007
19/70
Asymptotic MSE
Asymptotic efficiency
Threshold prediction
Global MSE
Ultimate performances
Signal to Noise Ratio (dB)
Alex
Outline
1
Framework and motivations
2
Minimal bounds on the MSE: unification
3
Minimal bounds on the MSE: application
4
Conclusion and prospect
Friday 23rd February 2007
20/70
Alex
Mean Square Error (dB)
2 Minimal bounds on the MSE: unification
Insuffisancy of the CramérRao bound
Non-Information
- Optimistic
- Bias
- Threshold
Other minimal
Bounds
(tightest)
SNRT
Friday 23rd February 2007
21/70
Signal to Noise Ratio (dB)
Alex
2 Minimal bounds on the MSE: unification
Two categories
Deterministic
parameters
Random
parameters
Determininstic Bounds
Bayesian Bounds
Bound the local MSE
Friday 23rd February 2007
Bound the global MSE
22/70
Alex
2 Minimal bounds on the MSE: unification
Two categories
Deterministic
parameters
Random
parameters
Determininstic Bounds
Bayesian Bounds
Bound the local MSE
Friday 23rd February 2007
Bound the global MSE
23/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
Glave IEEE IT 1973
Barankin Approach
In a class of unbiased estimator
, we want to find the
particular estimator for which the variance is minimal
at the true value of the parameter
Constrained optimization problem
Class
Friday 23rd February 2007
of unbiased estimator ????
24/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
Barankin (1949)
Friday 23rd February 2007
25/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
Barankin
Needs the resolution of an integral equation
Sometimes,
doesn’t exist
Friday 23rd February 2007
26/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
Cramér-Rao
Friday 23rd February 2007
27/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
Cramer
Friday 23rd February 2007
Rao
Fisher
Frechet
28/70
Darmois
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
Bhattacharyya
(1946)
Friday 23rd February 2007
29/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
Bhattacharyya
Friday 23rd February 2007
30/70
Barankin
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
?
Bhattacharyya
Friday 23rd February 2007
Guttman
Fraser
31/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
McAulay-Seidman
(1969)
(Barankin)
Test points
Friday 23rd February 2007
32/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
McAulay-Seidman
Barankin
Test points
Friday 23rd February 2007
33/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
How to choose test points ?
Friday 23rd February 2007
34/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
Chapman-Robbins (1951)
1 test point
Friday 23rd February 2007
35/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
Chapman
Friday 23rd February 2007
Robbins
Hammersley
Kiefer
36/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
Abel (1993)
Test points
Friday 23rd February 2007
37/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
Friday 23rd February 2007
38/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds unification
Quinlan-Chaumette-Larzabal (2006)
Test points
Friday 23rd February 2007
39/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds
f0=0, K=32 observtions
Don’t take into accout the
support of the parameter
Friday 23rd February 2007
40/70
Alex
2 Minimal bounds on the MSE: unification
Deterministic bounds
Already used in Signal Processing
CRB for wide range of topics
ChRB and Barankin (McAulay-Seidman version)
Time delay estimation
DOA estimation
Digital communications (synchronization parameters)
Abel bound
Digital communications (synchronization parameters)
Friday 23rd February 2007
41/70
Alex
2 Minimal bounds on the MSE: unification
Two categories
Deterministic
parameters
Random
parameters
Determininstic Bounds
Bayesian Bounds
Bound the local MSE
Friday 23rd February 2007
Bound the global MSE
42/70
Alex
2 Minimal bounds on the MSE: unification
Bayesian bounds unification
Best Bayesian bound: MSE of the conditional mean estimator
(MMSEE)
is the solution of
Friday 23rd February 2007
43/70
Alex
2 Minimal bounds on the MSE: unification
Bayesian bounds unification
For your information
Friday 23rd February 2007
44/70
Alex
2 Minimal bounds on the MSE: unification
Bayesian bounds unification
Best Bayesian
bound
Friday 23rd February 2007
Minimal
bound
45/70
Alex
2 Minimal bounds on the MSE: unification
Bayesian bounds unification
Constrained optimization problem
Degres of freedom
Friday 23rd February 2007
46/70
Alex
2 Minimal bounds on the MSE: unification
Bayesian bounds unification
s
1
h
Best Bayesian bound
Friday 23rd February 2007
47/70
Alex
2 Minimal bounds on the MSE: unification
Bayesian bounds unification
s
1
h
Bayesian Cramér-Rao bound (Van Trees 1968)
Friday 23rd February 2007
48/70
Alex
2 Minimal bounds on the MSE: unification
Bayesian bounds unification
Van Trees
Friday 23rd February 2007
49/70
Alex
2 Minimal bounds on the MSE: unification
Bayesian bounds unification
s
1
…
Test points
h
Reuven-Messer bound (1997) (Bayesian Barankin bound)
Bobrovsky-Zakaï bound (1976) (1 test point)
Friday 23rd February 2007
50/70
Alex
2 Minimal bounds on the MSE: unification
Bayesian bounds unification
Reuven
Friday 23rd February 2007
Messer
Bobrovsky
Zakai
51/70
Alex
2 Minimal bounds on the MSE: unification
Bayesian bounds unification
1
h
Bayesian Bhattacharyya bound (Van Trees 1968)
Friday 23rd February 2007
52/70
Alex
2 Minimal bounds on the MSE: unification
Bayesian bounds unification
Van Trees
Friday 23rd February 2007
53/70
Alex
2 Minimal bounds on the MSE: unification
Bayesian bounds unification
s
1
h
Weiss-Weinstein bound (1985)
Friday 23rd February 2007
54/70
Alex
2 Minimal bounds on the MSE: unification
Bayesian bounds unification
Weiss
Friday 23rd February 2007
Weinstein
55/70
Alex
2 Minimal bounds on the MSE: unification
Relationship between deterministic and Bayesian bounds
Deterministic bounds
Bayesian bounds
Constrained optimization problem
Friday 23rd February 2007
56/70
Alex
2 Minimal bounds on the MSE: unification
Relationship between deterministic and Bayesian bounds
Deterministic bounds
Bayesian bounds
Cramér-Rao
Bayesian Cramér-Rao
Bhattacharyya
Bayesian Bhattacharyya
Chapman-Robbins
Bobrovsky-Zakaï
McAulay-Seidman
Reuven-Messer
Abel
???
???
Friday 23rd February 2007
Weiss-Weinstein
57/70
Alex
2 Minimal bounds on the MSE: unification
Relationship between deterministic and Bayesian bounds
Deterministic bounds
Bayesian bounds
Cramér-Rao
Bayesian Cramér-Rao
Bhattacharyya
Bayesian Bhattacharyya
Chapman-Robbins
Bobrovsky-Zakaï
McAulay-Seidman
Reuven-Messer
Abel
Bayesian Abel
???
Weiss-Weinstein
Friday 23rd February 2007
58/70
Alex
2 Minimal bounds on the MSE: unification
Relationship between deterministic and Bayesian bounds
Deterministic bounds
Bayesian bounds
Cramér-Rao
Bayesian Cramér-Rao
Bhattacharyya
Bayesian Bhattacharyya
Chapman-Robbins
Bobrovsky-Zakaï
McAulay-Seidman
Reuven-Messer
Abel
Bayesian Abel
Deterministic Weiss-Weinstein
Weiss-Weinstein
Friday 23rd February 2007
59/70
Alex
2 Minimal bounds on the MSE: unification
Bayesian bounds
Already used in Signal Processing
BCRB and Bayesian Barankin bound ???
Bobrovsky-Zakai and Bayesian Abel bounds
Digital communications (synchronization parameters)
Weiss-Weinstein bounds
Spectral Analysis
Underwater acoustic
Array Processing
Digital communications
Physics (gravitational waves)
Friday 23rd February 2007
60/70
Alex
Outline
1
Framework and motivations
2
Minimal bounds on the MSE: unification
3
Minimal bounds on the MSE: application
4
Conclusion and perspectives
Friday 23rd February 2007
61/70
Alex
3 Minimal bounds on the MSE: application
Synchronization problem (spectral analysis)
Observations (complex)
Pilot symbols (known)
Additive noise, Gaussian, circular, iid
Parameter of interest
deterministic or random
Friday 23rd February 2007
62/70
Alex
3 Minimal bounds on the MSE: application
Synchronization problem (spectral analysis)
QPSK Modulation
20 observations
Maximum
Likelihood
Local MSE
Friday 23rd February 2007
63/70
Alex
3 Minimal bounds on the MSE: application
Synchronization problem (spectral analysis)
Bayesian bounds for deterministic parameters estimators
The support of the parameter is taken into
account
Friday 23rd February 2007
64/70
Alex
3 Minimal bounds on the MSE: application
Synchronization problem (spectral analysis)
QPSK Modulation
20 observations
Maximum
Likelihood
Global MSE
Friday 23rd February 2007
65/70
Alex
Outline
1
Framework and motivations
2
Minimal bounds on the MSE: unification
3
Minimal bounds on the MSE : application
4
Conclusion and perspectives
Friday 23rd February 2007
66/70
Alex
4
Conclusion and perspectives
Contributions
Deterministic bounds unification
Bayesian bounds unification
Bayesian Abel bound and deterministic Weiss-Weinstein bound
Closed-form expressions of the minimal bounds in a
synchronization framework (+ Gaussian observation model with
parameterized mean)
Friday 23rd February 2007
67/70
Alex
4
Conclusion and perspectives
BTW
-Understanding the CRB
- CRB for Singular FIM
- Ziv-Zakai Family (Bayesian)
- Minimal bounds for discret time filtering (Bayesian)
- Some global class of Cramér-Rao bound (Bayesian)
- Hybrid bounds (mixing non-random and random parameters)
Friday 23rd February 2007
68/70
Alex
4
Conclusion and perspectives
Perspectives
Estimators set for Bayesian bounds
Interpretation in terms of bias of the Weiss-Weinstein bound
Closed form expressions for the multiple parameters case
Friday 23rd February 2007
69/70
Alex
Minimal bounds on the Mean Square Error
Thank you
Friday 23rd February 2007
70/70
Alex