PROBABILITY AND STATISTICS
FOR ENGINEERING
Mean, Variance, Moments and
Characteristic Functions
Hossein Sameti
Department of Computer Engineering
Sharif University of Technology
Mean and Variance
 p.d.f of a r.v represents complete information about it.
 quite often it is desirable to characterize the r.v in terms of its average behavior.
 We introduce two parameters
- Mean
- Variance
 represent the overall properties of the r.v and its p.d.f.
Sharif University of Technology
2
Mean
 Mean or the Expected Value of a r.v X is defined as

 X  X  E ( X )   x f X ( x)dx.

 If X is a discrete-type r.v,
 X  X  E ( X )   x  pi ( x  xi )dx   xi pi   ( x  xi )dx
i
i
  xi pi   xi P( X  xi ) .
i

 

1
i
 Mean represents the average value of the r.v in a very large number of trials.
X ~ U ( a, b)
Example: Uniform Distribution
E( X ) 

b
a
x
1 x2
dx 
ba
ba 2
b
a
b2  a 2
ab


2( b  a )
2
This is the midpoint of the interval (a,b).
Sharif University of Technology
3
Mean
Example: Exponential Distribution
X ~  ( )
 1 x / 
 e
, x  0,
f X ( x)   

 0, otherwise.
E( X )  

x
0

e
x /

dx    ye  y dy  
0
Sharif University of Technology
4
Mean
X ~ P ( )
Example: Poisson Distribution
P( X  k )  e 
E( X ) 

 kP( X
e

, k  0,1,2,, .

 ke

k
 e

k
k!
k 0
 (k  1)!
k 1
k!
 k) 
k 0

k

i
 i!
e


k
k 1
k!
k
 e   e    .
i 0
Sharif University of Technology
5
Mean
X ~ B(n, p)
Example: Binomial Distribution
 n  k n k
P( X  k )  
,
k 
p q
 
k  0,1,2,, n.
n
E ( X )   kP( X  k )
k 0
n
 n  k nk
n!
  k   p q   k
p k q nk
(n  k )! k!
k 0  k 
k 1
n
n 1
n!
(n  1)!
k nk

p q  np 
p i q n i 1
k 1 ( n  k )! ( k  1)!
i  0 ( n  i  1)!i!
n
 np( p  q) n 1  np.
Sharif University of Technology
6
Mean
X ~ N ( , 2 )
Example: Normal Distribution
f X ( x) 
E( X ) 


1
2
1
2
2
1
2


2
xe( x   )


1
2



( y   )e
2
e ( x   )
/ 2 2
2
/ 2 2
.
dx
 y 2 / 2 2
dy
1

 y / 2
ye
dy



e
dy
2  
2  

2 

2

0
 y 2 / 2 2
2
2
1
 .
Sharif University of Technology
7
Functions of r.vs
 Given X ~ f X ( x),
 suppose Y  g ( X ) defines a new r.v with p.d.f fY ( y ).
 Y has a mean Y  E (Y ) 



y fY ( y )dy.
 To determine E (Y ), we don’t need to determine fY ( y ).
 Recall that for any y,
P y  Y  y  y    Pxi  X  xi  xi  ,
y  0
i
where xi represent the multiple solutions of the equation y  g ( xi ).
 This can be rewritten as
fY ( y )y   f X ( xi )xi ,
i
where the xi , xi  xi  terms form nonoverlapping intervals.
Sharif University of Technology
8
Functions of r.vs – continued
 Hence
y fY ( y )y   y f X ( xi )xi   g ( xi ) f X ( xi )xi ,
i
i
 and hence as y covers the entire y-axis,
 the corresponding Δx’s are nonoverlapping, and they cover the entire x-axis.
 As y  0,

E (Y )  E g ( X )   y

fY ( y )dy  


g ( x) f X ( x)dx.
 Which in the discrete case, reduces to:
E (Y )   g ( xi )P( X  xi ).
i
 So, fY ( y ) is not required to evaluate E (Y ) for Y  g ( X ).
Sharif University of Technology
9
Functions of r.vs – continued
Example
 Determine the mean of Y  X 2 , where X is a Poisson r.v.
E X

2
 k

2
k 0
e

P( X  k )   k e

2
k 0
k

k
k!
e
k 1

k
k 1

i 1
i 0
i!
 k (k  1)!  e  (i  1)


2
k
k!


  i

i 
i
 
 e   i     e   i  e  
 i 0 i! i 0 i! 
 i 1 i!



 

i
m 1
 
 
 e  
 e   e  
 e  
 i 1 (i  1)!

 m 0 m!

 e  e   e    2   .

E X k  is known as the kth moment of r.v X
Sharif University of Technology
10
Another Measure
 Mean alone will not be able to truly represent the p.d.f of any r.v.
 Consider two Gaussian r.vs X1 ~ N (0,1) and X 2 ~ N (0,10).
 Despite having the same mean   0 , their p.d.fs are quite different as one is
more concentrated around the mean.
 We need an additional parameter to measure this spread around the mean.
f X 2 ( x2 )
f X 1 ( x1 )
x1
x2
(b)  2  10
(a)  2  1
Sharif University of Technology
11
Variance and Standard Deviation
 For a r.v X with mean  , X   (positive or negative) represents the deviation
of the r.v from its mean.
 So, consider the quantity E [ X    2 ] and its average value  X    2 ,
represents the average mean square deviation of X around its mean.
 Define
 2  E[ X    2 ]  0.

X
 Using E g ( X )  



variance of the r.v X
2
g ( x) f X ( x)dx, with g ( X )  ( X   ) we have:

   ( x   )2 f X ( x)dx  0.
2
X
 X  E ( X   )2

standard deviation of X
 The standard deviation represents the root mean square spread of the r.v X
around its mean.
Sharif University of Technology
12
Variance and Standard Deviation

   ( x   ) 2 f X ( x)dx
2
X

Var( X )  

2
X



x


2
 2 x   2  f X ( x )dx

x f X ( x )dx  2   x f X ( x )dx   2
 E X
2

2
 
2
 E X
2
  E ( X )
2
___
2
X X .
2
This can be used to compute  X .
2
X ~ P ( )
Example: Poisson Distribution
  X  X  2     2  .
2
___
2
2
X
Sharif University of Technology
13
Variance and Standard Deviation
X ~ N ( , 2 )
Example: Normal Distribution
Var ( X )  E [( X   ) ] 
2

 x   
2

1
2
e
2
( x   ) 2 / 2 2
dx.
 To simplify this, we can make use




f X ( x )dx 

1
2

2
e ( x   )
2
/ 2 2
dx  1
 For a normal p.d.f. This gives



e ( x   )
2
/ 2 2
dx 
2  .
 Differentiating both sides of this equation with respect to  , we get
 or




( x   )2

   x   
3
2
e ( x   )
1
2 2
e
2
/ 2 2
2
dx 
( x   ) 2 / 2 2
dx   2 ,
Sharif University of Technology
Var( X )   2
14
Moments
 in general
___
n
mn  X  E ( X n ),
n 1
are known as the moments of the r.v X, and
n  E [( X   ) n ]
are known as the central moments of X.
 Clearly,   m1 , and   2 .
 It is easy to relate mn and  n .
2
 n n k
n k 

 n  E [( X   ) ]  E     X (   ) 
 k 0  k 

n
n
n
n
k
n k
   E X  (   )
    mk (   ) n  k .
k 0  k 
k 0  k 
n
 generalized moments of X about a,
 absolute moments of X.
E[( X  a )n ]
E[| X |n ]
Sharif University of Technology
15
Variance and Standard Deviation
Example: Normal Distribution
X ~ N (0,  2 )
 It can be shown that
0,
n odd,

E( X )  
n
1  3(n  1) , n even.
n
n

1

3

(
n

1
)

,
n even,
n
E (| X | )   k
2 k 1
2
k
!

2 /  , n  (2k  1), odd.

Var( X )   2
Sharif University of Technology
16
Characteristic Function
 The characteristic function of a r.v X is defined as
 X ( )  E e

jX
 


  X (0)  1, and  X ( )  1 for all
e jx f X ( x)dx.
.
 For discrete r.vs:
 X ( )   e jk P( X  k ).
k
Sharif University of Technology
17
Example
X ~ P ( )
Example: Poisson Distribution
P( X  k )  e 

 X ( )   e
k 0
jk
e

k
k!
, k  0,1,2,, .
j
j
(e j )k
e 
 e ee  e ( e 1) .
k!
k!
k 0
k


Sharif University of Technology
18
Example
Example: Binomial Distribution
X ~ B(n, p)
 n  k n k
P( X  k )  
,
k 
p q
 
n
 X ( )   e
k 0
jk
k  0,1,2,, n.
 n  k n k n  n 
  p q    ( pe j )k q n k ( pe j  q)n .
k 0  k 
k 
Sharif University of Technology
19
Characteristic Function and Moments
  ( jX )k   k E ( X k ) k
 X ( )  E e   E 
j


k!  k  0
k!
 k 0
2
k
2 E( X )
2
k E( X )
 1  jE( X )  j
  j
 k  .
2!
k!
jX
 Taking the first derivative of this equation with respect to , and letting it to be
equal to zero, we get
 X ( )
1  X ( )
 jE( X ) or E ( X ) 
.
  0
j   0
 Similarly, the second derivative gives
1  2 X ( )
E( X )  2
,
j
 2  0
2
Sharif University of Technology
20
Characteristic Function and Moments – continued
 repeating this procedure k times, we obtain the kth moment of X to be
1  k  X ( )
E( X )  k
, k  1.
k
j

 0
k
 We can use these results to compute the mean, variance and other higher order
moments of any r.v X.
Sharif University of Technology
21
Example
Example: Poisson Distribution
 X ( )  e
 ( e j 1)
X ~ P ( )
.


j
 X ( )
 e  e e je j ,

 2  X ( )

e j
j 2
e j
2 j

e
e
(

je
)

e

j
e .
2

1  X ( )
E( X ) 
,
j   0
1  2 X ( )
E( X )  2
.
2
j

 0
So,
2
2
2
E ( X )   , E ( X )    .
These results agree with previous ones, but the efforts involved in the calculations
are very minimal.
Sharif University of Technology
22
Example
Example: Mean and Variance of Binomial Distribution
 X ( )  ( pe j  q) n .
 X ( )
 jnpe j ( pe j  q) n 1

So, E ( X )  np
 2 X ( )
 j 2np e j ( pe j  q)n 1  (n  1) pe j 2 ( pe j  q)n 2 
2

1  2 X ( )
E( X )  2
,
j
 2  0
2
E ( X 2 )  np1  (n  1) p   n 2 p 2  npq.
 X2  E ( X 2 )  E ( X ) 2  n 2 p 2  npq  n 2 p 2  npq.
Sharif University of Technology
23
Example
X ~ N ( , 2 )
Example: Normal(Guassian) Distribution
 X ( )  
e

1

2
e jx
j
1

2
e ( x   )

e
jy
e
2
/ 2 2
dx (Let x    y )
 y 2 / 2 2
dy  e
j
1


2
2
(Let y  j 2  u so that y  u  j 2 )

1
 ( u  j 2 )( u  j 2 ) / 2 2
 e j
e
du
2  
2

1
j  2 2 / 2
 u 2 / 2 2
( j  2 2 / 2 )
e e
e
du

e
.
2  
2
2

2

e  y / 2
2
( y  j 2 2 )
dy
The characteristic function of a Gaussian r.v itself has the “Gaussian” bell shape.
Sharif University of Technology
24
Example - continued
For X ~ N (0,  ), we have f X ( x ) 
2
 X ( )  e
e x
2
1
2
 2 2 / 2
2
e
 x 2 / 2 2
, and therefore:
.
/ 2 2
e 

x
(b)
(a)
 Note the reverse roles of
2 / 2
2
 2 in f X (x ) and  X ().
Sharif University of Technology
25
Example
Example: Cauchy Distribution
f X ( x) 
( /  )
,
 2  x2

E( X ) 


2 
 1   2  x 2 dx  ,
  x
Which diverges to infinity. Similarly: E ( X )  
dx.
2
2



 x
2
x2

dx

   2  x 2



With x   tan  ,


0
 / 2 sin 
 tan 
2

sec

d


 0  2 sec2 
 0 cos  d
 / 2 d (cos  )

 /2
 
  log cos  0   log cos  ,
0
cos 
2
x
dx 
2
2
 x
 /2
So the double sided integral for mean is undefined.
It concludes that the mean and variance of a Cauchy r.v are undefined.
Sharif University of Technology
26
Chebychev Inequality
 Consider an interval of width 2ε symmetrically centered around its mean μ.
 What is the probability P| X   |   that X falls outside this interval?
X
 
 
X

2
we can start with the definition of
  E ( X   )
2


| x   |
2
 
 2.


( x   ) 2 f X ( x)dx  
| x   |
 2 f X ( x)dx  2 
So the desired probability is:
| x   |
( x   ) 2 f X ( x)dx
f X ( x)dx   2 P | X   |   .
2
P | X   |    2 .

Sharif University of Technology
Chebychev inequality
27
Chebychev Inequality
 So, the knowledge of f X (x ) is not necessary. We only need  2 .
 With   k we obtain P  | X   | k  
1
.
2
k
 Thus with k  3, we get the probability of X being outside the 3σ interval around
its mean to be 0.111 for any r.v.
 Obviously this cannot be a tight bound as it includes all r.vs.
 For example, in the case of a Gaussian r.v, from the Table, (   0,   1)
P | X | 3   0.0027.
 Chebychev inequality always underestimates the exact probability.
Sharif University of Technology
28
Moment Identities
 Suppose X is a discrete random variable that takes only nonnegative integer
values. i.e.,
P( X  k )  pk  0,
k  0, 1, 2,
 Then



 P( X  k )   
k 0
k  0 i  k 1

i 1
i 1
k 0
P( X  i )   P( X  i ) 1

  i P( X  i)  E ( X )
i 0
 Similarly,


i 1
k 0
i 1
k 0
i (i  1)
E{ X ( X  1)}
P( X  i ) 
2
2
i 1

 k P( X  k )  P( X  i )  k  
 Which gives:


E ( X )   i P( X  i )   (2k  1) P( X  k ).
2
i 1
2
k 0
These equations are at times quite useful in simplifying calculations.
Sharif University of Technology
29
Example: Birthday Pairing
 In a group of n people, what is
A) The probability that two or more persons will have the same birthday?
B) The probability that someone in the group will have the birthday that matches
yours?
Solution
A)
C = “no two persons have the same birthday”
N ( N  1)  ( N  n  1) n 1
k
P (C ) 

(
1

)

N
Nn
k 1
n 1
n 1
 k / N
k
P (C )  1  P (C )  1   (1  )  1  e k 1
 1  e  n ( n 1) / 2 N
N
k 1
(e  x  1  x )
N  365
For n=23, P(C )  0.5
Sharif University of Technology
30
Example – continued
B)
D = “a person not matching your birthday”
P( D)  (
N 1 n
)  (1  1 / N ) n  e  n / N
N
P( D )  1  e  n / N
• let X represent the minimum number of people in a group for a birthday pair
to occur.
• The probability that “the first n people selected from that group have different
birthdays” is given by
n 1
pn   (1  Nk )  e n ( n 1) / 2 N .
k 1
Sharif University of Technology
31
Example – continued
 But the event the “the first n people selected havedifferent birthdays” is the same
as the event “ X > n.”
 Hence
 n ( n 1) / 2 N
P( X  n )  e
.
 Using moment identities, this gives the mean value of X to be

E ( X )   P( X  n ) 
n 0
e
(1/ 8 N )

 1/ 2 e
  N /2 

e
 n ( n 1) / 2 N
n 0
 x2 / 2 N
dx  e
(1/ 8 N )


1
2

 1/ 2 e
 ( x 2 1/ 4) / 2 N
1/ 2
2 N   0 e
dx
 x2 / 2 N
dx

1
 24.44.
2
Sharif University of Technology
32
Example – continued
 Similarly,

E ( X )   (2n  1) P( X  n )
2
n 0

  (2n  1)e
n 0
 2e
(1/ 8 N )
 n ( n 1) / 2 N



2 ( x  1)e  ( x
2
1/ 4) / 2 N
dx
1/ 2
1/ 2

  x2 / 2 N

 x2 / 2 N
 ( x 2 1/ 4) / 2 N
dx   xe
dx   2  e
dx
  xe
0
 1/ 2
0
 2 N 2
1
 2
N    2E( X )
2

8

1
5
 2 N   2 N  1  2 N  2 N 
4
4
 779.139.
 Thus, Var ( X )  E ( X 2 )  ( E ( X ))2  181.82 and
 X  13.48.
 Since the standard deviation is quite high compared to the mean value, the
actual number of people required could be anywhere from 25 to 40.
Sharif University of Technology
33
Moment Identities for Continuous r.vs
if X is a nonnegative random variable with density function fX (x) and distribution
function FX (X), then


E{ X }   0 x f X ( x )dx   0

 0


y

  dy  f ( x)dx
x
X
0


f X ( x )dx dy   0 P( X  y )dy   0 P( X  x )dx


  0 {1  FX ( x )}dx   0 R( x )dx,
where
R( x )  1  FX ( x )  0,
Similarly
x  0.
E{ X 2 }   0 x 2 f ( x )dx   0   0 2 ydy  f ( x )dx


x
X
X
 2 0   y f ( x )dx  ydy


X

 2 0 x R( x )dx.
Sharif University of Technology
34