1
Least Squares
We frequently want to fit a straight
line to a series of data points.
i
xi
yi
1
x1
y1
2
y2
x2
3
y3
x3
…
…
…
i
xi
yi
…
…
…
N
yN
xN
Hence we want to fit y ( x)  ax  b . For each the point the error is
Ei  [ y ( xi )  yi ]2  [axi  b  yi ]2 .
The total error is
N
N
i 1
i 1
E   Ei   [axi  b  yi ]2 .
Minimizing the error with respect to a and b
N
E
 0  2  [axi  b  yi ]xi
a
i 1
N
E
 0  2  [axi  b  yi ] .
b
i 1
Then
N
a

i 1
xi2
b
N
N
xi   xi y i ,

i 1
i 1
and
N
N
N
i 1
i 1
i 1
y
1
4
7
10
22
x2
0
1
4
9
14
a  xi  b  1   y i .
N
Note that
1 N

i 1
Example: y  3 x  1
Total
x
0
1
2
3
6
xy
0
4
14
30
48
2
Then the two equations for a and b are:
14a  6b  48
.
6a  4b  22
Multiplying the second by 3/2 and subtracting from the first yields 5a  15 , or a  3 .
The second equation is 4b  4 , or b  1 , then y  3 x  1 which checks.
Generally the solution for the coefficients (ai) when n  3 or 4 and is inaccurate for
larger n.
Other functional forms can be made linear. Some popular examples are:
1) y  be ax can be written as ln y  ax  ln b which is linear in ln y and x, and
2) y  bx a can be written as ln y  a ln x  ln b which is linear in ln y and ln x.
A measure of the scatter is the variance
N
 Ei
 
2
i 1
N  n 1
,
where n is the degree of the polynomial.
For higher degree polynomials it is better to use the orthogonal Chebyshev (or
Tchebyshev) polynomials, Tn (x) , of degree n. The polynomials are given by
Tn ( x)  cos(n )
  cos 1 x
(i.e., x  cos )
The first few polynomials are:
T0 ( x)  1,
T1( x)  cos  x,
T2 ( x)  cos 2  cos 2   sin 2   2 cos 2   1  2 x 2  1,
and
T3 ( x)  cos 3  cos( 2   )  cos 2 cos   sin 2 cos   (cos 2   sin 2  ) cos   (2 sin  cos  ) sin 
T3 ( x)  cos 3   sin 2  cos   2 sin 2  cos   cos 3   3 sin 2  cos   cos 3   3(1  cos 2  ) cos 
T3 ( x)  cos 3   3 cos   3 cos 3  cos   4 cos 4   3 cos   4 x 3  3x .
3
Similarly,
T4 ( x)  8 x 4  8 x 2  1 .
Since
TN 1 ( x)  cos( N  1)   cos( N ) cos   sin( N ) sin  and
TN 1 ( x)  cos( N  1)   cos( N ) cos   sin( N ) sin  ,
Adding TN 1 ( x)  TN 1 ( x)  2 cos  cos( N )  2 xTN ( x) results in the recursive relation
TN 1 ( x)  2 xTN ( x)  TN 1 ( x) .
This means that the leading coefficient doubles every time N increase by one, that is
TN ( x)  2 N 1 x N  ... . Hence it is better to use t N ( x)  TN ( x) / 2 N 1  x N  ...
It can be shown that of all the polynomials of degree N with a N  1 that TN ( x) / 2 N 1
has a smaller upper bound in magnitude on [-1,1] than any other polynomial of degree N.
Since we know that TN ( x)  cos( N ) then TN ( x)  1 and TN ( x) / 2 N 1  1 / 2 N 1 .
This can be proved by contradiction. If Pn ( x)  x n  ... is a polynomial of degree  n
and has a maximum in [-1,1] less than Tn ( x) / 2 n 1 . The polynomial
Tn ( x) / 2 n 1  Pn ( x)  a n 1 Pn 1 ( x)
is of degree n-1. But note that, including the end points, Tn(x) has n+1 relative extremes
in [-1,1] equal to  1 / 2 n 1 and they alternate in sign. Since relative extremes of Pn(x)
are less than 1 / 2 n 1 then Pn-1(x) crosses zero at least once for every Tn(x) extreme which
is N times. Hence Pn-1(x) has n zeros but is a polynomial of degree n-1 and the
assumption must be wrong, and the theorem is correct.