Order Statistics
• The order statistics of a set of random variables X1, X2,…, Xn are the same
random variables arranged in increasing order.
• Denote by
X(1) = smallest of X1, X2,…, Xn
X(2) = 2nd smallest of X1, X2,…, Xn

X(n) = largest of X1, X2,…, Xn
• Note, even if Xi’s are independent, X(i)’s can not be independent since
X(1) ≤ X(2) ≤ … ≤ X(n)
• Distribution of Xi’s and X(i)’s are NOT the same.
week 10
1
Distribution of the Largest order statistic X(n)
• Suppose X1, X2,…, Xn are i.i.d random variables with common distribution
function FX(x) and common density function fX(x).
• The CDF of the largest order statistic, X(n), is given by
FX  n  x   PX n   x  
• The density function of X(n) is then
f X  n  x  
d
FX  n   x  
dx
week 10
2
Example
• Suppose X1, X2,…, Xn are i.i.d Uniform(0,1) random variables. Find the
density function of X(n).
week 10
3
Distribution of the Smallest order statistic X(1)
• Suppose X1, X2,…, Xn are i.i.d random variables with common distribution
function FX(x) and common density function fX(x).
• The CDF of the smallest order statistic X(1) is given by
FX 1 x   PX 1  x   1  PX 1  x  
• The density function of X(1) is then
f X 1 x  
d
FX x  
dx 1
week 10
4
Example
• Suppose X1, X2,…, Xn are i.i.d Uniform(0,1) random variables. Find the
density function of X(1).
week 10
5
Distribution of the kth order statistic X(k)
• Suppose X1, X2,…, Xn are i.i.d random variables with common distribution
function FX(x) and common density function fX(x).
• The density function of X(k) is
f X  n  x  
n!
FX x k 1 1  FX x nk f X x 
k  1!n  k !
week 10
6
Example
• Suppose X1, X2,…, Xn are i.i.d Uniform(0,1) random variables. Find the
density function of X(k).
week 10
7
Some facts about Power Series

• Consider the power series

a t
k 0
k
k
with non-negative coefficients ak.
 a t converges for any positive value of t, say for t = r, then it
k
• If
k 0
converges for all t in the interval [-r, r] and thus defines a function of t on
that interval.
k

• For any t in (-r, r), this function is differentiable at t and the series
converges to the derivatives.
 ka t
k 0
k 1
k
• Example:
For k = 0, 1, 2,… and -1< x < 1 we have that
1  x 
k 1
 k  m m
 x
  
k 
m 0 

(differentiating geometric series).
week 10
8
Generating Functions
• For a sequence of real numbers {aj} = a0, a1, a2 ,…, the generating function
of {aj} is

At    a j t j
j 0
if this converges for |t| < t0 for some t0 > 0.
week 10
9
Probability Generating Functions
• Suppose X is a random variable taking the values 0, 1, 2, … (or a subset of
the non-negative integers).
• Let pj = P(X = j) , j = 0, 1, 2, …. This is in fact a sequence p0, p1, p2, …
• Definition: The probability generating function of X is

 X t   p0  p1t  p2t     p j t j
2
• Since p j t j  p j if |t| < 1 and
for |t| < 1.
j 0

p
j 0
j
 1 the pgf converges absolutely at least
• In general, πX(1) = p0 + p1 + p2 +… = 1.
• The pgf of X is expressible as an expectation:
 X t    p j t j  E t X 

j 0
week 10
10
Examples
• X ~ Binomial(n, p),
 n
j 0  j 
n
 X t      p j q n j t j   pt  q n
converges for all real t.
• X ~ Geometric(p),

 X t    pq j1t j 
j 1
converges for |qt| < 1 i.e. t 
pt
1  qt
1
1

q 1 p
Note: in this case pj = pqj for j = 1, 2, …
week 10
11
PGF for sums of independent random variables
• If X, Y are independent and Z = X+Y then,
 Z t   E t Z   E t X Y   E t X t Y   E t X E t Y    X t  Y t 
• Example
Let Y ~ Binomial(n, p). Then we can write Y = X1+X2+…+ Xn . Where Xi’s
are i.i.d Bernoulli(p). The pgf of Xi is
 X t   t 0 1  p   t 1 p  tp  q.
i
The pgf of Y is then
 Y t   E t X  X
1
2  X n
  E t E t  E t   tp  q  .
week 10
X1
X2
Xn
n
12
Use of PGF to find probabilities
• Theorem
Let X be a discrete random variable, whose possible values are the
nonnegative integers. Assume πX(t0) < ∞ for some t0 > 0. Then
πX(0) = P(X = 0),
 ' X 0  P X  1,
 ' ' X 0  2P X  2,
etc. In general,
 Xk  0  k! P X  k ,
where  Xk  is the kth derivative of πX with respect to t.
• Proof:
week 10
13
Example
• Suppose X ~ Poisson(λ). The pgf of X is given by
e  j j
 X t   
t 
j
!
j 0

•
Using this pgf we have that
week 10
14
Finding Moments from PGFs
• Theorem
Let X be a discrete random variable, whose possible values are the
nonnegative integers. If πX(t) < ∞ for |t| < t0 for some t0 > 1. Then
 ' X 1  E X ,
etc. In general,
 ' ' X 1  EX  X 1,
 Xk  1  E  X  X  1 X  2 X  K  1,
Where  Xk  is the kth derivative of πX with respect to t.
• Note: E(X(X-1)∙∙∙(X-k+1)) is called the kth factorial moment of X.
• Proof:
week 10
15
Example
• Suppose X ~ Binomial(n, p). The pgf of X is
πX(t) = (pt+q)n.
Find the mean and the variance of X using its pgf.
week 10
16
Uniqueness Theorem for PGF
• Suppose X, Y have probability generating function πX and πY respectively.
Then πX(t) = πY(t) if and only if P(X = k) = P(Y = k) for k = 0,1,2,…
• Proof:
Follow immediately from calculus theorem:
If a function is expressible as a power series at x=a, then there is only one
such series.
A pgf is a power series about the origin which we know exists with radius
of convergence of at least 1.
week 10
17
Moment Generating Functions
• The moment generating function of a random variable X is
 
mX t   E etX
mX(t) exists if mX(t) < ∞ for |t| < t0 >0
• If X is discrete
mX t    etx p X x .
x
• If X is continuous

mX t    etx f X x dx.

• Note: mX(t) = πX(et).
week 10
18
Examples
•
X ~ Exponential(λ). The mgf of X is
  
mX t   E e
tX

0
etxex dx 
• X ~ Uniform(0,1). The mgf of X is
    e dx 
mX t   E e
tX
1
tx
0
week 10
19
Generating Moments from MGFs
• Theorem
Let X be any random variable. If mX(t) < ∞ for |t| < t0 for some t0 > 0. Then
mX(0) = 1
m' X 0  E X ,
 
m' ' X 0  E X 2 ,
etc. In general,
 
m Xk  0  E X k ,
Where m Xk  is the kth derivative of mX with respect to t.
• Proof:
week 10
20
Example
• Suppose X ~ Exponential(λ). Find the mean and variance of X using its
moment generating function.
week 10
21
Example
• Suppose X ~ N(0,1). Find the mean and variance of X using its moment
generating function.
week 10
22
Example
• Suppose X ~ Binomial(n, p). Find the mean and variance of X using its
moment generating function.
week 10
23
Properties of Moment Generating Functions
• mX(0) = 1.
• If Y=a+bX, a, b  R then the mgf of Y is given by
 


 
mY t   E etY  E eatbtX  eat E ebtX  eat mX bt .
• If X,Y independent and Z = X+Y then,
 




  
mZ t   E etZ  E etX tY  E etX etY  E etX E etY  mX t mY t 
week 10
24
Uniqueness Theorem
• If a moment generating function mX(t) exists for t in an open interval
containing 0, it uniquely determines the probability distribution.
week 10
25
Example
• Find the mgf of X ~ N(μ,σ2) using the mgf of the standard normal random
variable.



• Suppose, X 1 ~ N 1 ,  12 , X 2 ~ N 2 ,  22

independent.
Find the distribution of X1+X2 using mgf approach.
week 10
26