Physics 114: Lecture 17
Least Squares Fit to
Polynomial
Dale E. Gary
NJIT Physics Department
Reminder, Linear Least Squares

We start with a smooth line of the form
y ( x)  a  bx
which is the “curve” we want to fit to the data. The chi-square for this
situation is
2
2




y

y
(
x
)
1
2   i
     yi  a  bx  

i


 i


To minimize any function, you know that you should take the derivative
and set it to zero. But take the derivative with respect to what?
2
Obviously, we want to find constants a and b that minimize  , so we will
form two equations:
2
1

 1

 2 
     yi  a  bxi    2  2  yi  a  bxi    0,
a
a   i

 i

2
1

x

 2 
     yi  a  bxi    2  i2  yi  a  bxi    0.
b
b   i

 i

Apr 12, 2010
Polynomial Least Squares



Let’s now allow a curved line of polynomial form
y( x)  a  bx  cx 2  dx3  ...
which is the curve we want to fit to the data.
For simplicity, let’s consider a second-degree polynomial (quadratic). The
chi-square for this situation is
2
2




y  y ( x)
1
2
2   i

y

a

bx

cx





i


i


 i

Following exactly the same approach as before, we end up with three
equations in three unknowns (the parameters
a, b and c):
2
1

 1

 2 
     yi  a  bxi  cxi2    2  2  yi  a  bxi  cxi2    0,
a
a   i

 i

2
1

x

 2 
     yi  a  bxi  cxi2    2  i2  yi  a  bxi  cxi2    0,
b
b   i

 i

2
1
 xi2

 2 
2 
     yi  a  bxi  cxi    2  2  yi  a  bxi  cxi2    0.
c
c   i

 i

Apr 12, 2010
Second-Degree Polynomial

The solution, then, can be found from the same determinant technique we
used before, except now we have 3 x 3 determinants:

1
a 


yi
2
i
xi yi
2
i
xi2 yi
2
i
 
 
 
xi2
xi
2
i
2
i
xi2
xi3
2
i
2
i
xi3
xi4
2
i
2
i
  
1
c     

   
xi
yi
2
i
2
i
xi2
xi yi
2
i
2
i
2
i
xi2
xi3
2
i
2
i
1
2
i
xi
xi2 yi
2
i
 
1
b    

  
2
i
,
2
i
xi
xi yi
2
i
2
i
xi2
2
i
xi3
xi2 yi
2
i
2
i
xi4
2
i
2
i
  
where        
  
1
2
i
,



xi2
yi
1
xi
xi2
2
i
2
i
xi2
xi3
2
i
2
i
2
i
xi2
xi3
xi4
2
i
2
i
2
i
xi
You can see that extending to arbitrarily high powers is straightforward, if
tedious.
 We have already seen the MatLAB command that allows polynomial fitting. It
is just p = polyfit(x,y,n), where n is the degree of the fit. We have used
n = 1 so far.

Apr 12, 2010
MatLAB Example:
2nd-Degree Polynomial Fit
First, create a set of points that follow a second degree polynomial, with
some random errors, and plot them:




Now use polyfit to fit a second-degree polynomial:



25
3.0145 -2.5130
hold on
plot(x,polyval(p,x),'r')
And the original function


p = polyfit(x,y,2)
prints p = 1.5174
20
Now overplot the fit


x = -3:0.1:3;
y = randn(1,61)*2 - 2 + 3*x + 1.5*x.^2;
plot(x,y,'.')
plot(x,-2 + 3*x + 1.5*x.^2,'g')
Notice that the points scatter about
the fit. Look at the residuals.
data1
Polyfit
y(x)
15
y = 1.5x 2 + 3x - 2

10
5
0
-5
-10
-3
-2
-1
0
x
1
Apr 12, 2010
2
3
MatLAB Example (cont’d):
2nd-Degree Polynomial Fit
The residuals are the differences between the points and the fit:




The residuals appear flat and random, which is good. Check the standard
10
deviation of the residuals:


resid = y – polyval(p,x)
figure
plot(x,resid,'.')
std(resid)
prints ans = 1.9475
This is close to the value of 2 we
used when creating the points.
5
Residuals

0
-5
-10
-3
-2
-1
0
x
Apr 12, 2010
1
2
3
MatLAB Example (cont’d):
Chi-Square for Fit

We could take our set of points, generated from a 2nd order polynomial, and
fit a 3rd order polynomial:




The fit looks the same, but there is a subtle difference due to the use of an
additional parameter. Let’s look at the standard deviation of the new



resid2 = y – polyval(x,p2)
std(resid2)
prints ans = 1.9312
Is this a better fit? The residuals are slightly smaller BUT check chi-square.



p2 = polyfit(x,y,3)
hold off
plot(x,polyval(x,p2),'.')
chisq1 = sum((resid/std(resid)).^2)
% prints 60.00
chisq2 = sum((resid2/std(resid2)).^2) % prints 60.00
They look identical, but now consider the reduced chi-square.
sum((resid/std(resid)).^2)/58.
% prints 1.0345
sum((resid2/std(resid2)).^2)/57. % prints 1.0526
=> 2nd-order fit is preferred
Apr 12, 2010
Linear Fits, Polynomial Fits,
Nonlinear Fits
When we talk about a fit being linear or nonlinear, we mean linear in the
coefficients (parameters), not in the independent variable. Thus, a
polynomial fit is linear in coefficients a, b, c, etc., even though those
coefficients multiply non-linear terms in independent variable x, (i.e. cx2).
 Thus, polynomial fitting is still linear least-squares fitting, even though we are
fitting a non-linear function of independent variable x. The reason this is
considered linear fitting is because for n parameters we can obtain n linear
equations in n unknowns, which can be solved exactly (for example, by the
method of determinants using Cramer’s Rule as we have done).
 In general, this cannot be done for functions that are nonlinear in the
parameters (i.e., fitting a Gaussian function f(x) = a exp{[(x  b)/c]2}, or sine
function f(x) = a sin[bx +c]). We will discuss nonlinear fitting next time, when
we discuss Chapter 8.
 However, there is an important class of functions that are nonlinear in
parameters, but can be linearized (cast in a form that becomes linear in
coefficients). We will now take a look at that.

Apr 12, 2010
Linearizing Non-Linear Fits

Consider the equation
y( x)  aebx ,
where a and b are the unknown parameters. Rather than consider a and b,
we can take the natural logarithm of both sides and consider instead the
function
ln y  ln a  bx.

This is linear in the parameters ln a and b, where chi-square is
2


1
 2     ln yi  ln a  bx   .
  i


Notice, though, that we must use uncertainties i′, instead of the usual i
to account for the transformation of the dependent variable:
2
  (ln yi )  2 1 2
 i 2  
 i  2 i
yi
 y 
  i 
1
i.
yi
Apr 12, 2010
MatLAB Example:
Linearizing An Exponential
First, create a set of points that follow the exponential, with some random
0.25
errors, and plot them:






0.15
0.1
0.05
0
logy = log(y+dev);
plot(x,logy,’.’)
As predicted, the points now make a pretty good
straight line. What about the errors. You might
think this will work:


0.2
Now convert using log(yi) – MatLAB for ln(yi)


x = 1:10;
y = 0.5*exp(-0.75*x);
sig = 0.03*sqrt(y); % errors proportional to sqrt(y)
dev = sig.*randn(1,10);
errorbar(x,y+dev,sig)
y

errorbar(x, logy, log(sig))
Try it! What is wrong?
2
4
6
8
10
6 6
88
10
10
x
-1
2
-2
0
-3
-2
-4
-4
ln(y)

-6
-5
-8
-6
-10
-7
-12
-8
-14
-9
0
2
2
4 4
xx
Apr 12, 2010
MatLAB Example (cont’d):
Linearizing An Exponential
The correct errors are as noted earlier:


This now gives the correct plot. Let’s go ahead
and try a linear fit. Remember, to do a weighted
linear fit we use glmfit().





0.25
0.2
0.15
0.1
0.05
p = glmfit(x,logy,’normal’,’weights’,logsig);
p = circshift(p,1);
% swap order of parameters
hold on
plot(x,polyval(p,x),’r’)
0



hold off
errorbar(x,y+dev,sig)
hold on
plot(x,exp(polyval(p,x)),’r’)
Note parameters a′ = ln a = 0.6931, b′ = b = 0.75
2
4
66
88
10
10
6
88
10
10
x
-2
-3
To plot the line over the original data:


logsig = sig./y;
errorbar(x, logy, logsig)
y

-4
ln(y)

-5
-6
-7
-8
-9
2
4
x
Apr 12, 2010
Summary
Use polyfit() for polynomial fitting, with third parameter giving the degree of
the polynomial. Remember that higher-degree polynomials use up more
degrees of freedom (an nth degree polynomial takes away n + 1 DOF).
 A polynomial fit is still considered linear least-squares fitting, despite its
dependence on powers of the independent variable, because it is linear in the
coefficients (parameters).
 bx
 For some problems, such as exponentials, y( x)  ae ,, one can linearize the
problem. Another type that can be linearized is a power-law expression,
y( x)  axb ,

as you will do in the homework.
 When linearizing, the errors must be handled properly, using the usual error
propagation equation, e.g.
2
  (ln yi )  2 1 2
 i 2  
 i  2 i

y
yi


  i 
1
i.
yi
Apr 12, 2010