SAMPLE MOMENTS
1. P OPULATION M OMENTS
1.1. Moments about the origin (raw moments). The rth moment about the origin of a random variable X,
denoted by µ0r , is the expected value of Xr ; symbolically,
µ0r =E(X r )
X
=
xr f(x)
(1)
x
for r = 0, 1, 2, . . . when X is discrete and
µ0r =E(X r )
Z ∞
=
xr f(x) dx
(2)
−∞
when X is continuous. The rth moment about the origin is only defined if E[ Xr ] exists. A moment about
the origin is sometimes called a raw moment. Note that µ01 = E(X) = µX , the mean of the distribution of
X, or simply the mean of X. The rth moment is sometimes written as function of θ where θ is a vector of
parameters that characterize the distribution of X.
If there is a sequence of random variables, X1, X2 , . . . Xn , we will call the rth population moment of the
ith random variable µ0i, r and define it as
µ0i,r = E (Xir )
(3)
1.2. Central moments. The rth moment about the mean of a random variable X, denoted by µr , is the
expected value of ( X − µX )r symbolically,
µr =E[ ( X − µX )r ]
X
r
=
( x − µX ) f(x)
(4)
x
for r = 0, 1, 2, . . . when X is discrete and
µr =E[ (X − µX )r ]
Z ∞
r
=
(x − µX ) f(x) dx
(5)
−∞
when X is continuous. The rth moment about the mean is only defined if E[ (X - µX )r ] exists. The rth
moment about the mean of a random variable X is sometimes called the rth central moment of X. The rth
central moment of X about a is defined as E[ (X - a)r ]. If a = µX , we have the rth central moment of X about
µX .
Note that
Date: December 7, 2005.
1
2
SAMPLE MOMENTS
µ1 = E[X − µX ] =
Z
∞
(x − µX ) f(x) dx = 0
−∞
2
µ2 = E[(X − µX ) ] =
Z
(6)
∞
2
(x − µX ) f(x) dx = V ar(X) = σ
2
−∞
Also note that all odd moments of X around its mean are zero for symmetrical distributions, provided
such moments exist.
If there is a sequence of random variables, X1, X2, . . . Xn, we will call the rth central population moment
of the ith random variable µi,r and define it as
µi,r = E Xir − µ0i,1
r
(7)
When the variables are identically distributed, we will drop the i subscript and write
µ0r
and µ r .
2. S AMPLE M OMENTS
2.1. Definitions. Assume there is a sequence of random variables, X1, X2 , . . . Xn . The first sample moment,
usually called the average is defined by
n
1 X
X̄n =
Xi
n i= 1
(8)
Corresponding to this statistic is its numerical value, x̄n, which is defined by
n
1 X
x̄n =
xi
n i= 1
(9)
where xi represents the observed value of Xi . The rth sample moment for any t is defined by
n
1 X r
X
n i=1 i
(10)
1X r
=
xi
n
(11)
X̄nr =
This too has a numerical counterpart given by
n
x̄rn
i=1
2.2. Properties of Sample Moments.
2.2.1. Expected value of X̄nr . Taking the expected value of equation 10 we obtain
n
n
1X
1X 0
E X̄nr = E X̄nr =
E Xir =
µi,r
n
n
i=1
(12)
i=1
If the X’s are identically distributed, then
n
1X 0
E X̄nr = E X̄nr =
µ = µ0r
n i=1 r
(13)
SAMPLE MOMENTS
3
2.2.2. Variance of X̄nr . First consider the case where we have a sample X1, X2, . . . ,Xn .
V ar
X̄nr
= V ar
n
1X r
X
n i=1 i
!
1
= 2 V ar
n
n
X
Xir
!
(14)
i= 1
If the X’s are independent, then
n
1 X
V ar (Xir )
V ar X̄nr = 2
n i=1
(15)
If the X’s are independent and identically distributed, then
1
V ar X̄nr = V ar (X r )
(16)
n
where X denotes any one of the random variables (because they are all identical). In the case where r =1,
we obtain
1
σ2
V ar X̄n = V ar ( X ) =
n
n
(17)
3. S AMPLE C ENTRAL M OMENTS
3.1. Definitions. Assume there is a sequence of random variables, X1, X2, . . . ,Xn . We define the sample
central moments as
Cnr =
n
r
1 X
Xi − µ0i,1 , r = 1, 2, 3, . . .,
n
i=1
⇒ Cn1
n
1 X
Xi − µ0i,1
=
n
⇒ Cn2 =
1
n
i=1
n
X
Xi − µ0i,1
(18)
2
i=1
These are only defined if µ0i , 1 is known.
3.2. Properties of Sample Central Moments.
3.2.1. Expected value of Cnr . The expected value of Cnr is given by
E (Cnr ) =
n
1 X
E
n
Xi − µ0i,1
r
i=1
=
n
1 X
µi,r
n
(19)
i=1
The last equality follows from equation 7.
If the Xi are identically distributed, then
E ( Cnr ) =µ r
E Cn1 =0
(20)
4
SAMPLE MOMENTS
3.2.2. Variance of C nr . First consider the case where we have a sample X1, X2, . . . ,Xn .
!
!
n
n
X
r
r
1
1X
r
0
0
V ar ( Cn ) = V ar
= 2 V ar
Xi − µi,1
Xi − µi,1
n i=1
n
i=1
(21)
If the X’s are independently distributed, then
V ar ( Cnr ) =
n
r 1 X
V ar Xi − µ0i,1
2
n i=1
(22)
If the X’s are independent and identically distributed, then
1
r
(23)
V ar ( X − µ01 )
n
where X denotes any one of the random variables (because they are all identical). In the case where r =1,
we obtain
V ar ( Cnr ) =
V ar Cn1
1
V ar [ X − µ01 ]
n
1
= V ar [ X − µ ]
n
1
= σ2 − 2 Cov [ X , µ ] + V ar [ µ ]
n
1
= σ2
n
=
4. S AMPLE A BOUT
THE
(24)
AVERAGE
4.1. Definitions. Assume there is a sequence of random variables, X1 , X2, . . . Xn . Define the rth sample
moment about the average as
Mnr =
n
r
1 X
Xi − X̄n ,
n i=1
r = 1, 2, 3, . . .,
(25)
This is clearly a statistic of which we can compute a numerical value. We denote the numerical value by,
mrn , and define it as
mrn =
n
1 X
r
( xi − x̄n )
n i= 1
(26)
In the special case where r = 1 we have
Mn1 =
n
1 X
Xi − X̄n
n
i=1
=
n
1 X
Xi − X̄n
n
i=1
= X̄n − X̄n = 0
4.2. Properties of Sample Moments about the Average when r = 2.
(27)
SAMPLE MOMENTS
5
4.2.1. Alternative ways to write Mnr . We can write Mn2 in an alternative useful way by expanding the squared
term and then simplifying as follows
Mnr =
n
r
1 X
Xi − X̄n
n
i=1
⇒ Mn2 =
n
2
1 X
Xi − X̄n
n
i=1
!
n
X
2
Xi − 2 Xi X̄n + X̄n2
1
=
n
1
=
n
=
1
n
i=1
n
X
i=1
n
X
n
2X̄n Xn
1 X 2
X̄n
Xi2 −
Xi +
i=1
n
n
(28)
i= 1
Xi2 − 2X̄n2 + X̄n2
i=1
n
X
1
=
n
Xi2
!
− X̄n2
i=1
4.2.2. Expected value of Mn2. The expected value of Mnr is then given by
2
E Mn
1
= E
n
=
1
n
"
n
X
n
X
Xi2
#
− E X̄n2
i=1
2
E Xi2 − E X̄n
− V ar(X̄n )
(29)
i=1
n
1 X 0
=
µ
−
n i=1 i,2
n
1 X 0
µ
n i=1 i,1
!2
− V ar(X̄n )
The second line follows from the alternative definition of variance
2
V ar ( X ) =E X 2 − [ E ( X ) ]
⇒ E X 2 = [ E ( X ) ]2 + V ar ( X )
2
⇒ E X̄n2 = E X̄n
+ V ar(X̄n )
(30)
and the third line follows from equation 12. If the Xi are independent and identically distributed, then
6
SAMPLE MOMENTS
E
Mn2
#
" n
X
1
2
=
Xi − E X̄n2
E
n
i=1
n
1 X 0
µi,2 −
=
n
1X 0
µi,1
n
n
i=1
− V ar(X̄n )
i=1
(31)
σ2
n
2
= µ02 − (µ01 ) −
1 2
σ
n
= σ2 −
=
!2
n− 1 2
σ
n
where µ01 and µ02 are the first and second population moments, and µ2 is the second central population
moment for the identically distributed variables. Note that this obviously implies
E
"
n
X
Xi − X̄
2
#
= n E Mn2
i=1
=n
n− 1
n
σ2
(32)
= (n − 1) σ2
4.2.3. Variance of Mn2 . By definition,
V ar Mn2
= E
h
Mn2
2i
− E Mn2
2
(33)
The second term on the right on equation 33 is easily obtained by squaring the result in equation 31.
n − 1 2
E Mn2 =
σ
n
2
2 (n − 1)2 4
⇒ E Mn2
= E Mn2 =
σ
n2
(34)
Now consider the first term on the right hand side of equation 33. Write it as
E
h
Mn2
2 i

n
2
1 X
=E 
Xi − X̄n
n
i=1
Now consider writing
1
n
Pn
i=1
Xi − X̄n
2
as follows
!2

(35)
SAMPLE MOMENTS
7
n
n
2 1 X
2
1 X
Xi − X̄ =
(Xi − µ) − (X̄ − µ)
n i=1
n i=1
n
2
1 X
Yi − Ȳ
=
n i=1
(36)
where Yi =Xi − µ
Ȳ =X̄ − µ
Obviously,
n
X
Xi − X̄
2
=
i= 1
n
X
Yi − Ȳ
2
, where Yi = Xi − µ , Ȳ = X̄ − µ
(37)
i= 1
Now consider the properties of the random variable Yi which is a transformation of Xi . First the expected
value.
Yi =Xi − µ
E (Yi) = E (Xi ) − E ( µ)
=µ − µ
(38)
=0
The variance of Yi is
Y i = Xi − µ
V ar (Yi ) = V ar (Xi )
(39)
= σ2 if Xi are independently and identically distributed
Also consider E(Yi 4). We can write this as
4
E( Y ) =
Z
∞
y4 f ( x ) d x
−∞
=
Z
∞
( x − µ)4 f(x) d x
−∞
= µ4
Now write equation 35 as follows
(40)
8
SAMPLE MOMENTS
E
h
i
2 2
Mn

n
2
1 X
=E 
Xi − X̄n
n
i=1

n
1 X
=E 
n i=1
2
Xi − X̄

n
2
1 X
=E 
Yi − Ȳ
n i=1

=
Ignoring
1
n2
n
X
1
E
n2
Yi − Ȳ
i=1
!2 

(41a)
!2 

(41b)
!2

2
(41c)
!2 

(41d)
for now, expand equation 41 as follows

E
n
X
i= 1
Yi − Ȳ
2

!2 
n
X
 =E 
i= 1

=E
n
X
!2 

Yi2 − 2 YiȲ + Ȳ 2
Yi2 − 2 Ȳ
i=1
n
X
Yi +
i=1
n
X
(42a)
Ȳ 2
i=1
!2 

 (
)
!2 
n
X

=E
Yi2 − 2 n Ȳ 2 + n Ȳ 2
(42b)
(42c)
i=1
 (
)
!2 
n
X
=E
Yi2 − nȲ 2 
(42d)
i=1

=E

=E
n
X
i=1
n
X
i=1
Yi2
!2
− 2 n Ȳ 2
n
X
i=1

Yi2 + n2 Ȳ 4
"
!2 
#
n
X
Yi2  − 2 n E Ȳ 2
Yi2 + n2 E Ȳ 4
Now consider the first term on the right of 42 which we can write as
i=1
(42e)
(42f)
SAMPLE MOMENTS

E
n
X
Yi2
!2 
i=1

 =E 
=E
"
n
X
n
X
Yi2
i=1
n
X
Yi4
j=1
+
9

Yj2 
XX
i=1
=
n
X
E Yi4 +
i=1
XX
(43a)
Y2
i=
6 j i
i 6= j
Yj2
#
(43b)
E Yi2 E Yj2
(43c)
=n µ4 + n (n − 1 ) µ22
(43d)
=n µ4 + n (n − 1 ) σ4
(43e)
Now consider the second term on the right of 42 (ignoring 2n for now) which we can write as
"
E Ȳ 2
n
X
#
Yi2 =
i=1

1
E
n2
n
X
j=1
Yj
n
X
k=1
Yk
n
X
i=1

Yi2 

(44a)





n


X
X
X
X
X
X
1

4
2
2
2
= 2 E
Yi 
Yi +
Yi Yj +
Yj Yk
i 6= j
j 6= k


n

i = 1
i 6= j


i 6= k

(44b)





n


X
X
X
X
X
X
1 

= 2 
E Yi2 (44c)
E Yi4 +
E Yi2 E Yj2 +
E Yj E Yk
i
=
6
j
j
=
6
k


n
i = 1

i
=
6
j


i 6= k
1 n µ4 + n (n − 1) µ22 + 0
2
n
1
= µ4 + (n − 1 )σ4
n
=
The last term on the penultimate line is zero because E(Yj ) = E(Yk ) = E(Yi ) = 0.
(44d)
(44e)
10
SAMPLE MOMENTS
Now consider the third term on the right side of 42 (ignoring n2 for now) which we can write as


n
n
n
n
X
X
X
X
4 1
E Ȳ
= 4 E
Yi
Yj
Yk
Y` 
n
i=1
j=1
k=1
(45a)
`=1


n
X
XX
XX
X
1
= 2 E
Yi4 +
Yi2 Yk2 +
Yi2 Yj2 +
Yi2 Yj2 + · · · 
i 6= k
i 6= j
n
i=1
(45b)
i 6= j
where for the first double sum (i = j 6= k = `), for the second (i = k 6= j = `), and for the last (i = ` 6= j =
k) and ... indicates that all other terms include Yi in a non-squared form, the expected value of which will
be zero. Given that the Yi are independently and identically distributed, the expected value of each of the
double sums is the same, which gives


n
X
XX
XX
X
4 1
E Ȳ = 4 E 
Yi4 +
Yi2 Yk2 +
Yi2 Yj2 +
Yi2 Yj2 + · · · 
i 6= k
i 6= j
n
i= 1
(46a)
i 6= j
" n
#
XX
1 X
4
2
2
= 4
E Yi + 3
Yi Yj + terms containing EXi
i 6= j
n
i=1
" n
#
X
XX
1
4
2
2
= 4
E Yi + 3
Yi Yj
i 6= j
n
(46b)
(46c)
i=1
1
n4
1
= 4
n
1
= 3
n
=
n µ4 + 3 n (n − 1) ( µ2 )2
(46d)
n µ4 + 3 n (n − 1) σ4
(46e)
µ4 + 3 (n − 1 ) σ4
(46f)
Now combining the information in equations 44, 45, and 46 we obtain
SAMPLE MOMENTS

E
n
X
i=1
Yi − Ȳ
2

!2 
 =E 

=E 
n
X
i= 1
n
X
11
!2 

Yi2 − 2 YiȲ + Ȳ 2
Yi
i=1
2
!2 
"
 − 2 n E Ȳ 2
n
X
(47a)
Yi2
#
+ n2 E Ȳ
4
(47b)
i=1
E
1
µ4 + (n − 1 ) µ22
n
+ n2
=
n2
2n
1
n2(n − 1 ) 2 2 n (n − 1 ) 2
3(n − 1 ) 2
µ4 −
µ4 + µ4 +
µ2 −
µ2 +
µ2
n
n
n
n
n
n
(47e)
=
n2 − 2 n + 1
(n − 1 ) (n2 − 2 n + 3 ) 2
µ4 +
µ2
n
n
(47f)
=
n2 − 2 n + 1
(n − 1 ) (n2 − 2 n + 3 ) 4
µ4 +
σ
n
n
(47g)
Now rewrite equation 41 including
h
1 2
µ
+
3
(n
−
1
)
µ
4
2
n3
(47c)
1
=n µ4 + n (n − 1 ) µ22 − 2 µ4 + (n − 1 ) µ22 +
µ4 + 3 (n − 1 ) µ22
n
(47d)
=nµ4 + n(n − 1 )µ22 − 2n
Mn2
2 i
1
n2
as follows

!2 
n
2
1  X

= 2E
Yi − Ȳ
n
i= 1
1
= 2
n
n2 − 2 n + 1
(n − 1) (n2 − 2 n + 3 )
µ4 +
σ4
n
n
(48a)
(48b)
=
n2 − 2 n + 1
(n − 1 ) (n2 − 2 n + 3)
µ
+
σ4
4
n3
n3
(48c)
=
( n − 1 )2
(n − 1 ) (n2 − 2 n + 3)
µ4 +
σ4
3
n
n3
(48d)
Now substitute equations 34 and 48 into equation 33 to obtain
h
2 i
2
V ar Mn2 =E Mn2
− E Mn2
(n − 1 )2
(n − 1 ) (n2 − 2 n + 3 ) 4
( n − 1 )2 4
=
µ
+
σ
−
σ
4
n3
n3
n2
We can simplify this as
(49)
12
SAMPLE MOMENTS
h
2i
2
− E Mn2
V ar Mn2 =E Mn2
(50a)
(n − 1 ) (n2 − 2 n + 3 ) 4
n(n − 1)2 4
( n − 1 )2
µ
+
σ
−
σ
4
n3
n3
n3
µ4 ( n − 1) 2 + ( n − 1 ) σ4
n2 − 2 n + 3 − n ( n − 1)
=
n3
µ4 ( n − 1) 2 + ( n − 1 ) σ4
n2 − 2 n + 3 − n2 + n
=
n3
µ4 ( n − 1) 2 + ( n − 1 ) σ4 (3 − n )
=
n3
µ4 ( n − 1) 2 − ( n − 1 ) σ4 (n − 3 )
=
n3
=
=
( n − 1) 2 µ4
( n − 1 ) (n − 3 ) σ4
−
n3
n3
(50b)
(50c)
(50d)
(50e)
(50f)
5. S AMPLE VARIANCE
5.1. Definition of sample variance. The sample variance is defined as
Sn2 =
n
1 X
n− 1
Xi − X̄n
2
(51)
i=1
We can write this in terms of moments about the mean as
Sn2 =
n
X
1
n − 1
Xi − X̄n
2
i=1
n
2
n
1 X
Xi − X̄n
=
Mn2 where Mn2 =
n−1
n
(52)
i=1
5.2. Expected value of S2 . We can compute the expected value of S2 by substituting in from equation 31 as
follows
E Sn2
=
n
E Mn2
n − 1
=
n n − 1 2
σ
n − 1 n
= σ2
5.3. Variance of S2 . We can compute the variance of S2 by substituting in from equation 50 as follows
(53)
SAMPLE MOMENTS
n2
V ar Mn2
( n − 1 )2
n2
( n − 1) 2 µ4
( n − 1) (n − 3) σ4
=
−
( n − 1 )2
n3
n3
13
V ar Sn2 =
=
(54)
µ4
(n − 3)σ4
−
n
n(n − 1 )
5.4. Definition of σ̂2 . One possible estimate of the population variance is σ̂2 which is given by
σ̂2 =
n
2
1X
Xi − X̄n
n i=1
(55)
= Mn2
5.5. Expected value of σ̂2 . We can compute the expected value of σ̂2 by substituting in from equation 31 as
follows
E σ̂2 =E Mn2
=
n − 1 2
σ
n
(56)
5.6. Variance of σ̂2 . We can compute the variance of σ̂2 by substituting in from equation 50 as follows
V ar σ̂2
= V ar Mn2
=
(n − 1)2 µ4
(n − 1) (n − 3) σ4
−
n3
n3
(57)
µ4 − µ22
µ4 − 3µ22
2 (µ4 − 2µ22 )
+
−
2
n
n
n3
We can also write this in an alternative fashion
=
V ar σ̂2 =V ar
Mn2
=
( n − 1) 2 µ4
( n − 1 ) (n − 3 ) σ4
−
3
n
n3
=
( n − 1) 2 µ4
( n − 1 ) (n − 3 ) µ22
−
n3
n3
=
n 2 µ 4 − 2 n µ4 + µ 4
n2 µ22 − 4 n µ22 + 3 µ22
−
n3
n3
=
n2 ( µ4 − µ22 ) − 2 n ( µ4 − 2 µ22 ) + µ4 − 3 µ22
n3
=
µ4 − µ22 2 (µ4 − 2 µ22 ) µ4 − 3 µ22
+
−
n
n2
n3
(58)
14
SAMPLE MOMENTS
6. N ORMAL
POPULATIONS
6.1. Central moments of the normal distribution. For a normal population we can obtain the central moments by differentiating the moment generating function. The moment generating function for the central
moments is as follows
MX (t) = e
t2 σ 2
2
.
(59)
The moments are then as follows. The first central moment is
d t2 σ 2 |t = 0
e 2
dt
t2 σ 2 |t = 0
= t σ2 e 2
E(X − µ) =
(60)
=0
The second central moment is
d2 t2 σ2 e 2
|t = 0
dt2
d 2 t2 σ 2 |t = 0
=
tσ e 2
dt
t2 σ 2 t2 σ2 + σ2 e 2
|t = 0
= t2 σ 4 e 2
E(X − µ )2 =
(61)
= σ2
The third central moment is
d3 t2 σ2 e 2
|t = 0
dt3
t2 σ2 d 2 4 t22σ2 e
+ σ2 e 2
|t = 0
=
t σ
dt
t2 σ 2 t2 σ 2 t2 σ 2 + 2 t σ4 e 2
+ t σ4 e 2
|t = 0
= t3 σ 6 e 2
E(X − µ )3 =
=
=0
The fourth central moment is
t2 σ 2 t2 σ2 + 3 t σ4 e 2
|t = 0
t3 σ 6 e 2
(62)
SAMPLE MOMENTS
d4 t2 σ2 e 2
|t = 0
dt4
t2 σ2 d 3 6 t2 σ 2 + 3 t σ4 e 2
|t = 0
=
t σ e 2
dt
t2 σ 2 t2 σ 2 t2 σ 2 t2 σ2 + 3 t2 σ 6 e 2
+ 3 t2 σ 6 e 2
+ 3 σ4 e 2
|t = 0
= t4 σ 8 e 2
15
E(X − µ )4 =
(63)
t2 σ 2 t2 σ 2 t2 σ2 + 6 t2 σ 6 e 2
+ 3 σ4 e 2
|t = 0
= t4 σ 8 e 2
= 3σ4
6.2. Variance of S2 . Let X1 , X2, . . . Xn be a random sample from a normal population with mean µ and
variance σ2 < ∞.
We know from equation 54 that
V ar
Sn2
n2
V ar Mn2
2
(n − 1)
n2
( n − 1) 2 µ4
( n − 1) (n − 3) σ4
=
−
( n − 1)2
n3
n3
=
=
(64)
µ4
(n − 3 ) σ4
−
n
n (n − 1)
If we substitute in for µ4 from equation 63 we obtain
V ar Sn2
=
µ4
(n − 3 ) σ4
−
n
n (n − 1 )
=
3 σ4
(n − 3 ) σ4
−
n
n (n − 1)
=
( 3 ( n − 1 ) − (n − 3 )) σ4
n (n − 1)
( 3 n − 3 − n + 3 )) σ4
=
n (n − 1)
=
2 n σ4
n (n − 1)
=
2 σ4
(n − 1)
(65)
6.3. Variance of σ̂2 . It is easy to show that
V ar (σ̂2 ) =
2 σ4 ( n − 1 )
n2
(66)