EE103 (Fall 2011-12)
8. Linear least-squares
• definition
• examples and applications
• solution of a least-squares problem, normal equations
8-1
Definition
overdetermined linear equations
Ax = b
(A is m × n with m > n)
if b 6∈ range(A), cannot solve for x
least-squares formulation

1/2
n
m X
X
aij xj − bi)2
minimize kAx − bk =  (
i=1 j=1
• r = Ax − b is called the residual or error
• x with smallest residual norm krk is called the least-squares solution
• equivalent to minimizing kAx − bk2
Linear least-squares
8-2
Example


2 0
A =  −1 1  ,
0 2


1
b= 0 
−1
least-squares solution
minimize (2x1 − 1)2 + (−x1 + x2)2 + (2x2 + 1)2
to find optimal x1, x2, set derivatives w.r.t. x1 and x2 equal to zero:
10x1 − 2x2 − 4 = 0,
−2x1 + 10x2 + 4 = 0
solution x1 = 1/3, x2 = −1/3
(much more on practical algorithms for LS problems later)
Linear least-squares
8-3
r22 = (−x1 + x2)2
r12 = (2x1 − 1)2
30
20
15
20
10
10
5
0
2
0
2
2
0
2
0
0
x2
−2 −2
x2
x1
0
−2 −2
x1
r12 + r22 + r32
r32 = (2x2 + 1)2
30
60
20
40
10
20
0
2
0
2
2
0
x2
Linear least-squares
0
−2 −2
x1
2
0
x2
0
−2 −2
x1
8-4
Outline
• definition
• examples and applications
• solution of a least-squares problem, normal equations
Data fitting
fit a function
g(t) = x1g1(t) + x2g2(t) + · · · + xngn(t)
to data (t1, y1), . . . , (tm, ym), i.e., choose coefficients x1, . . . , xn so that
g(t1) ≈ y1,
g(t2) ≈ y2,
...,
g(tm) ≈ ym
• gi(t) : R → R are given functions (basis functions)
• problem variables: the coefficients x1, x2, . . . , xn
• usually m ≫ n, hence no exact solution with g(ti) = yi for all i
• applications: developing simple, approximate model of observed data
Linear least-squares
8-5
Least-squares data fitting
compute x by minimizing
m
X
(g(ti) − yi)2 =
m
X
(x1g1(ti) + x2g2(ti) + · · · + xngn(ti) − yi)
2
i=1
i=1
in matrix notation: minimize kAx − bk2 where


g1(t1) g2(t1) g3(t1) · · · gn(t1)
 g1(t2) g2(t2) g3(t2) · · · gn(t2) 
,
A=
.
.
.
.


.
.
.
.
g1(tm) g2(tm) g3(tm) · · · gn(tm)
Linear least-squares


y1
 y2 

b=
.
 . 
ym
8-6
Example: data fitting with polynomials
g(t) = x1 + x2t + x3t2 + · · · + xntn−1
basis functions are gk (t) = tk−1, k = 1, . . . , n

t21
t22
tn−1
1
n−1
t2

1 t1
···
 1 t2

···

,
A= . .
.
.
. .
.
. 
1 tm t2m · · · tn−1
m


y1
 y2 

b=
 .. 
ym
interpolation (m = n): can satisfy g(ti) = yi exactly by solving Ax = b
approximation (m > n): make error small by minimizing kAx − bk
Linear least-squares
8-7
example. fit a polynomial to f (t) = 1/(1 + 25t2) on [−1, 1]
• pick m = n points ti in [−1, 1], and calculate yi = 1/(1 + 25t2i )
• interpolate by solving Ax = b
n = 15
n=5
8
1.5
6
1
4
0.5
2
0
−0.5
−1
0
−0.5
0
0.5
1
−2
−1
−0.5
0
0.5
1
(dashed line: f ; solid line: polynomial g; circles: the points (ti, yi))
increasing n does not improve the overall quality of the fit
Linear least-squares
8-8
same example by approximation
• pick m = 50 points ti in [−1, 1]
• fit polynomial by minimizing kAx − bk
n=5
n = 15
1
1
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0
0
−0.2
−1
−0.5
0
0.5
1
−0.2
−1
−0.5
0
0.5
1
(dashed line: f ; solid line: polynomial g; circles: the points (ti, yi))
much better fit overall
Linear least-squares
8-9
Least-squares estimation
y = Ax + w
• x is what we want to estimate or reconstruct
• y is our measurement(s)
• w is an unknown noise or measurement error (assumed small)
• ith row of A characterizes ith sensor or ith measurement
least-squares estimation
choose as estimate the vector x̂ that minimizes
kAx̂ − yk
i.e., minimize the deviation between what we actually observed (y), and
what we would observe if x = x̂ and there were no noise (w = 0)
Linear least-squares
8-10
Navigation by range measurements
find position (u, v) in a plane from distances to beacons at positions (pi, qi)
beacons
(p1, q1)
(p4, q4)
ρ1
ρ4
ρ2
(p2, q2)
(u, v)
unknown position
ρ3
(p3, q3)
four nonlinear equations in two variables u, v:
p
(u − pi)2 + (v − qi)2 = ρi
for i = 1, 2, 3, 4
ρi is the measured distance from unknown position (u, v) to beacon i
Linear least-squares
8-11
linearized distance function: assume u = u0 + ∆u, v = v0 + ∆v where
• u0, v0 are known (e.g., position a short time ago)
• ∆u, ∆v are small (compared to ρi’s)
p
(u0 + ∆u − pi)2 + (v0 + ∆v − qi)2
p
(u0 − pi)∆u + (v0 − qi)∆v
2
2
≈
(u0 − pi) + (v0 − qi) + p
(u0 − pi)2 + (v0 − qi)2
gives four linear equations in the variables ∆u, ∆v:
p
(u0 − pi)∆u + (v0 − qi)∆v
p
≈ ρi − (u0 − pi)2 + (v0 − qi)2
(u0 − pi)2 + (v0 − qi)2
for i = 1, 2, 3, 4
Linear least-squares
8-12
linearized equations
Ax ≈ b
where x = (∆u, ∆v) and A is 4 × 2 with
b i = ρi −
ai1 =
ai2 =
p
(u0 − pi)2 + (v0 − qi)2
(u0 − pi)
p
(u0 − pi)2 + (v0 − qi)2
p
(u0 − pi)2 + (v0 − qi)2
(v0 − qi)
• due to linearization and measurement error, we do not expect an exact
solution (Ax = b)
• we can try to find ∆u and ∆v that ‘almost’ satisfy the equations
Linear least-squares
8-13
numerical example
• beacons at positions (10, 0), (−10, 2), (3, 9), (10, 10)
• measured distances ρ = (8.22, 11.9, 7.08, 11.33)
• (unknown) actual position is (2, 2)
linearized range equations (linearized around (u0, v0) = (0, 0))




−1.00
0.00 −1.77
 0.98 −0.20  ∆u
 1.72 




 −0.32 −0.95  ∆v ≈  −2.41 
−0.71 −0.71
−2.81
least-squares solution: (∆u, ∆v) = (1.97, 1.90) (norm of error is 0.10)
Linear least-squares
8-14
Least-squares system identification
measure input u(t) and output y(t) for t = 0, . . . , N of an unknown system
unknown
system
u(t)
y(t)
example (N = 70):
4
5
y(t)
u(t)
2
0
0
−2
−4
0
20
40
t
60
−5
0
20
40
60
t
system identification problem: find reasonable model for system based
on measured I/O data u, y
Linear least-squares
8-15
moving average model
ymodel(t) = h0u(t) + h1u(t − 1) + h2u(t − 2) + · · · + hnu(t − n)
where ymodel(t) is the model output
• a simple and widely used model
• predicted output is a linear combination of current and n previous inputs
• h0, . . . , hn are parameters of the model
• called a moving average (MA) model with n delays
least-squares identification: choose the model that minimizes the error
E=
N
X
t=n
Linear least-squares
(ymodel(t) − y(t))2
!1/2
8-16
formulation as a linear least-squares problem:
E
N
X
=
(h0u(t) + h1u(t − 1) + · · · + hnu(t − n) − y(t))2
t=n
!1/2
= kAx − bk



A = 





x = 


Linear least-squares
u(n)
u(n − 1) u(n − 2)
u(n + 1)
u(n)
u(n − 1)
u(n + 2) u(n + 1)
u(n)
..
..
..
u(N )
u(N − 1) u(N − 2)



h0
y(n)
 y(n + 1) 
h1 



h2 
y(n + 2) 
,
b=



.. 
..


hn
y(N )
···
···
···
u(0)
u(1)
u(2)
..
· · · u(N − n)






8-17
example (I/O data of page 8-15) with n = 7: least-squares solution is
h0 = 0.0240,
h4 = 0.2425,
h1 = 0.2819,
h5 = 0.4873,
5
h2 = 0.4176,
h6 = 0.2084,
h3 = 0.3536,
h7 = 0.4412
solid: y(t): actual output
4
dashed: ymodel(t)
3
2
1
0
−1
−2
−3
−4
0
10
20
30
40
50
60
70
t
Linear least-squares
8-18
model order selection: how large should n be?
relative error E/kyk
1
0.8
0.6
0.4
0.2
0
0
20
40
n
• suggests using largest possible n for smallest error
• much more important question: how good is the model at predicting
new data (i.e., not used to calculate the model)?
Linear least-squares
8-19
model validation: test model on a new data set (from the same system)
4
5
ȳ(t)
ū(t)
2
0
0
−2
−4
0
20
40
60
relative prediction error
t
−5
0
20
40
60
t
1
0.8
• for n too large the predictive
ability of the model becomes
worse!
0.6
0.4
validation data
modeling data
0.2
0
0
20
• validation data suggest n = 10
40
n
Linear least-squares
8-20
for n = 50 the actual and predicted outputs on system identification and
model validation data are:
I/O set used to compute model
5
model validation I/O set
5
solid: y(t)
dashed: ymodel(t)
0
−5
0
solid: ȳ(t)
dashed: ȳmodel(t)
0
20
40
t
60
−5
0
20
40
60
t
loss of predictive ability when n is too large is called overfitting or
overmodeling
Linear least-squares
8-21
Outline
• definition
• examples and applications
• solution of a least-squares problem, normal equations
Geometric interpretation of a LS problem
minimize kAx − bk2
A is m × n with columns a1, . . . , an
• kAx − bk is the distance of b to the vector
Ax = x1a1 + x2a2 + · · · + xnan
• solution xls gives the linear combination of the columns of A closest to b
• Axls is the projection of b on the range of A
Linear least-squares
8-22
example
b


1 −1
2 ,
A= 1
0
0

a2

1
b= 4 
2
a1
Axls = 2a1 + a2
least-squares solution xls


1
Axls =  4  ,
0
Linear least-squares
xls =
2
1
8-23
The solution of a least-squares problem
if A is left-invertible, then
xls = (AT A)−1AT b
is the unique solution of the least-squares problem
minimize kAx − bk2
• in other words, if x 6= xls, then kAx − bk2 > kAxls − bk2
• recall from page 4-25 that AT A is positive definite and that
(AT A)−1AT
is a left-inverse of A
Linear least-squares
8-24
proof
we show that kAx − bk2 > kAxls − bk2 for x 6= xls:
kAx − bk2 = kA(x − xls) + (Axls − b)k2
= kA(x − xls)k2 + kAxls − bk2
> kAxls − bk2
• 2nd step follows from A(x − xls) ⊥ (Axls − b):
(A(x − xls))T (Axls − b) = (x − xls)T (AT Axls − AT b) = 0
• 3rd step follows from zero nullspace property of A:
x 6= xls
Linear least-squares
=⇒
A(x − xls) 6= 0
8-25
The normal equations
(AT A)x = AT b
if A is left-invertible:
• least-squares solution can be found by solving the normal equations
• n equations in n variables with a positive definite coefficient matrix
• can be solved using Cholesky factorization
Linear least-squares
8-26