MA22S6 Homework 6 Solutions
ˆ Question 1: We are interested in X̄ =
1
n
n
∑i=1 Xi . Firstly, we say it is unbiased if has an
expected value equal to the true mean, µ. Recall that E[aY ] = aE[Y ] if a is a constant
and Y is some variable, and that expectation is a linear operator, i.e., that the expectation
of a sum is the sum of the expectations. We see
1 n
E[X̄] = E[ ∑ Xi ]
n i=1
n
1
E[∑ Xi ]
=
n i=1
1
(E[X1 ] + ⋅ ⋅ ⋅ + E[Xn ])
=
n
1
=
(µ + . . . + µ)
n
1
=
nµ
n
= µ
Hence X̄ is an unbiased estimator. Nowe we must find its variance. Recall that Var[aY ] =
a2 Var[Y ] if a is a constant and Y is some variable, and that the variance of a sum of
i ndependent random variables is the sum of the variances. Hence
1 n
Var[X̄] = Var[ ∑ Xi ]
n i=1
n
1 2
= ( ) Var[∑ Xi ]
n
i=1
1
=
(Var[X1 ] + ⋅ ⋅ ⋅ + Var[Xn ])
n2
1 2
=
(σ + . . . + σ 2 )
n2
1
=
nσ 2
n2
σ2
=
n
Finally we wish to show that SB2 =
1
n
n
∑i=1 (Xi − X̄)2 is a biased estimator for σ 2 , i.e., that
its expectation tends to σ 2 in the limit n → ∞ but differs from σ 2 at finite values of n.
1
We can expand SB2 as follows:
SB2 =
=
=
=
=
=
1 n
∑(Xi − X̄)2
n i=1
1 n
∑(X 2 + X̄ 2 − 2X̄Xi )
n i=1 i
1 n 2 n 2 n
( ∑ X + ∑ X̄ − ∑ 2X̄Xi )
n i=1 i i=1
i=1
n
1 n 2
( ∑ Xi + nX̄ − 2X̄ ∑ Xi )
n i=1
i=1
n
n
1
1 n
( ∑ Xi2 + nX̄ 2 − 2nX̄ 2 ) as X̄ = ∑ Xi implies ∑ Xi = nX̄
n i=1
n i=1
i=1
n
1
( ∑ X 2 − nX̄ 2 )
n i=1 i
Lets find the expectation of this:
1 n
E[SB2 ] = E[ ( ∑ Xi2 − nX̄ 2 )]
n i=1
n
1
=
E[( ∑ Xi2 − nX̄ 2 )]
n
i=1
n
1
σ2
=
[ ∑(σ 2 + µ2 ) − n( + µ2 )] as σ 2 = E[Xi2 ] − µ2 implies E[Xi2 ] = σ 2 + µ2
n i=1
n
2
σ
σ2
and
= E[X̄ 2 ] − µ2 implies E[X̄ 2 ] =
+ µ2
n
n
1
σ2
=
[n(σ 2 + µ2 ) − n( + µ2 )]
n
n
1
(nσ 2 − σ 2 )
=
n
n−1 2
1
=
σ = σ 2 − σ 2 → σ 2 for n → ∞.
(1)
n
n
Hence the estimator is biased. An unbiased estimator is
1
n−1
n
∑i=1 (Xi − X̄)2 .
ˆ Question 2: Fitting to a constant corresponds to the very simple model of ŷi = α. To
find the optimal α we solve
dχ2 (α)
= 0,
dα
d n (yi − α)2
= 0
∑
dα i=1
σi2
2
n
yi − α
= 0
2
i=1 σi
n
yi − α
= 0
∑
2
i=1 σi
n
n
yi
1
−
α
∑ 2
∑ 2 = 0
i=1 σi
i=1 σi
n y
∑i=1 σ2i
α = n 1i
∑i=1 σ2
−2 ∑
i
We see that this is simply a weighted arithmetic average ∑i ci yi , with weights ci determined by the errors on the data points and ∑i ci = 1. The smaller the error σi , the higher
the weight ci .
ˆ Question 3:
a) We can write our model, assuming direct proportionality as ŷi = αxi .
b) Now to find the best fit for α we solve the one parameter model as follows:
d 2
χ (α) = 0
dα
d 3 (yi − αxi )2
= 0
∑
dα i=1
σi2
3
−2 ∑ xi
i=1
3
yi − αxi
= 0
σi2
yi − αxi
= 0,
σi2
i=1
3
3
3
3
x2
x2
xi y i
xi yi
∑ 2 − α ∑ i2 = 0 ⇒ ∑ 2 = α ∑ i2
i=1 σi
i=1 σ1
i=1 σi
i=1 σ1
∑ xi
Thus we have a one dimensional linear system which can be written in the form
Ax = b as follows:
3
∑i=1
xi yi
σ12
b
x2
= ∑3i=1 σi2
α
=
x
i
A
To find x we need to find the inverse A−1 . In this case it is simply
models of higher dimension we need to find the inverse matrix.
Plugging in the values from the given table we can solve:
x = A−1 b
3
1
∑3i=1
x2
i
σ2
i
but for
b = 24897.67901; A−1 = 4.51646 × 10−6 ; and so α = 0.11245
We have already calculated A−1 so our one-sigma confidence estimate for α∗ is:
α∗ = 0.11245 ± 0.00213
where σ =
√
A−1 .
c) To evaluate the quality of the model we need to find estimates for each of the Yi .
i.e. We need to find ŷi for each i:
ŷ1 = α∗ x1 = 0.11245 × 40 = 4.498
ŷ2 = α∗ x2 = 0.11245 × 80 = 8.996
ŷ3 = α∗ x3 = 0.11245 × 100 = 11.245
Using these estimates we can substitute in to the equation below and find the χ2
statistic. From class we know that the χ2 ≈ n for a good model.
3
(yi − ŷi )2
,
σi2
i=1
χ2 (α∗ ) = ∑
= 13.021778.
This is much larger than n so it is not a very good model.
ˆ Question 4:
a) Plotting our points from the model y = αx + β and the data table we have been
given we see that a line through the points is roughly appropriate and hence a
priori it seems linear modelling would be the correct choice. (Intuitively though you
would think that only 4 data points is not enough data about a system to make this
judgement.)
b) In this example we have a one dimensional system that is linear in two parameters.
Hence to find the estimates of our parameters we need to solve the equations:
∂χ2 (α,β)
∂α
= 0 and
4
∂χ2 (α,β)
∂β
= 0.
i.e. find the minimum for each parameter. We proceed as follows:
∂χ2 (α, β)
=0
∂β
∂ 4 (yi − β − αxi )2
=0
∑
∂β i=1
σi2
∂χ2 (α, β)
=0
∂α
∂ 4 (yi − β − αxi )2
=0
∑
∂α i=1
σi2
4
4
(yi − β − αxi )xi
=0
σi2
i=1
4
yi xi − βxi − αx2i
=0
∑
σi2
i=1
yi − β − αxi
−2 ∑
=0
σi2
i=1
−2 ∑
4
yi − β − αxi
=0
σi2
i=1
∑
4
4
x2i
y i xi
xi
∑ 2 −β ∑ 2 −α∑ 2 = 0
i=1 σi
i=1 σi
i=1 σi
4
4
4
x2
yi xi
xi
∑ 2 = β ∑ 2 + α ∑ i2
i=1 σi
i=1 σi
i=1 σi
4
4
4
4
yi
1
xi
∑ 2 −β ∑ 2 −α∑ 2 = 0
i=1 σi
i=1 σi
i=1 σi
4
4
4
yi
1
xi
=
β
+
α
∑ 2
∑ 2
∑ 2
i=1 σi
i=1 σi
i=1 σi
As before we have a linear system but this time there are two parameters so:
b
=
A
x
⎡ ⎤
⎡
⎤ ⎡ ⎤
⎢b ⎥
⎢A
⎥ ⎢ ⎥
⎢ 1⎥
⎢ 11 A12 ⎥ ⎢β ⎥
⎢ ⎥ = ⎢
⎥ ⎢ ⎥
⎢ ⎥
⎢
⎥ ⎢ ⎥
⎢b ⎥
⎢A
⎥ ⎢ ⎥
⎢ 2⎥
⎢ 21 A22 ⎥ ⎢α⎥
⎣ ⎦
⎣
⎦ ⎣ ⎦
To solve we can plug in from the given data table where each of the matrix and
vector entries can be calculated as follows:
b1
= ∑4i=1 σy2i
= 2988.91895
b2
= ∑4i=1 yσi x2i
= 20271.00247
A11
= ∑4i=1 σ12
= 169.856698
A12 = A21
= ∑4i=1 σx2i
= 994.801031
= ∑4i=1
= 8842.442613
A22
i
i
i
i
x2i
σi2
To find the vector of parameters we need to find the inverse A−1 of A to solve
A−1 b = x. This can be done using simple 2 × 2 matrix inverse calculations. The
resulting parameters are α∗ = 0.91698 and β ∗ = 12.22623.
We still need to find the one-sigma interval for α∗ and β ∗ . From the slides we know
that the variance of each parameter is σα∗ =
√
1
(A−1 )22
and σβ ∗ =
us:
α∗ = 0.91698 ± 0.01821
β ∗ = 12.22623 ± 0.13138
5
√ 1
(A−1 )11
which gives
c) To evaluate the quality of the model we need to find estimates for each of the Yi .
i.e. We need to find ŷi for each i:
ŷ1 = α∗ x1 + β ∗ = 0.91698 × 3 + 12.22623 = 14.977
ŷ2 = α∗ x2 + β ∗ = 0.91698 × 7 + 12.22623 = 18.645
ŷ3 = α∗ x3 + β ∗ = 0.91698 × 10 + 12.22623 = 21.396
ŷ4 = α∗ x4 + β ∗ = 0.91698 × 16 + 12.22623 = 26.898
as before, using these estimates we can substitute for ŷi in to the equation below
and find the χ2 statistic.
4
(yi − ŷi )2
,
σi2
i=1
χ2 (α∗ , β ∗ ) = ∑
= 37.757
From class we know that the χ2 ≈ n for a good model. Our result for χ2 = 37.8 is
much larger than n = 4 so it is not a very good model.
6