System Identification
6.435
SET 5
– Least Squares
– Statistical Properties
Munther A. Dahleh
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
1
Least Squares
• Linear regressions
• LS Estimates: Statistical properties
• Bias, variance, covariance
• Noise-variance estimation
• Introduction to model structure determination:
Statistical analysis and hypothesis testing
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
2
•
unknown parameters
measured
known
• Multivariable
• Examples:
–
–
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
3
• In a matrix form
• In general, this system is perturbed by noise
e – noise, stochastic
• For N > n, the system is overdetermined.
• Define
, then
• LS:
θ
Lecture 5
θ
6.435, System Identification
Prof. Munther A. Dahleh
4
Solution to LS
•
•
• Proof: Standard pseudo-inverse formula
by substitution
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
5
• Equivalently
• Example
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
6
Analysis of LS
•
• Assume that
∴ white noise, zero mean,
variance
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
7
• Theorem:
1)
is unbiased estimate of
2)
3) an unbiased estimate of
• Basic assumption:
is given by
is fixed.
• Proof:
1)
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
8
2)
3)
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
9
Best Linear Unbiased Estimate
•
correlated noise
• Analysis of LS estimate:
1)
2)
• Consider general linear estimators:
• Want to find Z such that the estimate is unbiased and
is minimized.
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
10
Solution of BLUE
• Solution:
•
for any unbiased estimate.
• Proof:
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
11
• To show that
However,
result follows.
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
12
• If
and is equal to the least
squares estimate. Hence LS is the BLUE when e is white.
• Can nonlinear estimates help? Not if the distribution of e is
Gaussian
• BLUE of
Lecture 5
for any constant matrix is
.
6.435, System Identification
Prof. Munther A. Dahleh
13
• Example:
BLUE of
Lecture 5
is
6.435, System Identification
Prof. Munther A. Dahleh
14
Maximum Likelihood Estimate
is unknown
Suppose: Maximum likelihood estimate
θ
The parameter that makes y the “most likely event”.
Result: Suppose that y is a random variable with a distribution
that depends on
. Let
be the likelihood function,
and
is an unbiased estimate of . Then:
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
15
Least Squares
Differentiate:
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
16
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
17
• Remarks:
– LS estimate for is efficient, i.e.
= lower bound (termed efficient)
– LS estimate of
Lecture 5
(guess) is asymptotically efficient.
6.435, System Identification
Prof. Munther A. Dahleh
18
To show that:
(assuming Gaussian dist)
Cramer-Rao bdd.
• Result: BLUE for Gaussian distribution is best estimate over
Nonlinear estimators.
Lecture 5
6.435, System Identification
Prof. Munther A. Dahleh
19