Order Statistics
• The order statistics of a set of random variables X1, X2,…, Xn are the same
random variables arranged in increasing order.
• Denote by
X(1) = smallest of X1, X2,…, Xn
X(2) = 2nd smallest of X1, X2,…, Xn

X(n) = largest of X1, X2,…, Xn
• Note, even if Xi’s are independent, X(i)’s can not be independent since
X(1) ≤ X(2) ≤ … ≤ X(n)
• Distribution of Xi’s and X(i)’s are NOT the same.
week 2
1
Distribution of the Largest order statistic X(n)
• Suppose X1, X2,…, Xn are i.i.d random variables with common distribution
function FX(x) and common density function fX(x).
• The CDF of the largest order statistic, X(n), is given by
FX  n  x   PX n   x  
• The density function of X(n) is then
f X  n  x  
d
FX  n   x  
dx
week 2
2
Example
• Suppose X1, X2,…, Xn are i.i.d Uniform(0,1) random variables. Find the
density function of X(n).
week 2
3
Distribution of the Smallest order statistic X(1)
• Suppose X1, X2,…, Xn are i.i.d random variables with common distribution
function FX(x) and common density function fX(x).
• The CDF of the smallest order statistic X(1) is given by
FX 1 x   PX 1  x   1  PX 1  x  
• The density function of X(1) is then
f X 1 x  
d
FX x  
dx 1
week 2
4
Example
• Suppose X1, X2,…, Xn are i.i.d Uniform(0,1) random variables. Find the
density function of X(1).
week 2
5
Distribution of the kth order statistic X(k)
• Suppose X1, X2,…, Xn are i.i.d random variables with common distribution
function FX(x) and common density function fX(x).
• The density function of X(k) is
f X  n  x  
n!
FX x k 1 1  FX x nk f X x 
k  1!n  k !
week 2
6
Example
• Suppose X1, X2,…, Xn are i.i.d Uniform(0,1) random variables. Find the
density function of X(k).
week 2
7
Statistical Model
• A statistical model for some data is a set of distributions,  f :   
one of which corresponds to the true unknown distribution that
produced the data.
• The statistical model corresponds to the information a statistician
brings to the application about what the true distribution is or at least
what he or she is willing to assume about it.
• The variable θ is called the parameter of the model, and the set Ω is
called the parameter space.
• From the definition of a statistical model, we see that there is a
unique value    , such that fθ is the true distribution that generated
the data. We refer to this value as the true parameter value.
week 2
8
Examples
• Suppose there are two manufacturing plants for machines. It is
known that the life lengths of machines built by the first plant have
an Exponential(1) distribution, while machines manufactured by the
second plant have life lengths distributed Exponential(1.5). You
have purchased five of these machines and you know that all five
came from the same plant but do not know which plant. Further, you
observe the life lengths of these machines, obtaining a sample
(x1, …, x5) and want to make inference about the true distribution of
the life lengths of these machines.
• Suppose we have observations of heights in cm of individuals in a
population and we feel that it is reasonable to assume that the
distribution of height is the population is normal with some
unknown mean and variance. The statistical model in this case is
f  , : ,  2   where Ω = R×R+, where R+ = (0, ∞).
2
week 2
9
Point Estimate
• Most statistical procedures involve estimation of the unknown value
of the parameter of the statistical model.
• A point estimate, ˆ  ˆx1 ,..., xn  , is an estimate of the parameter θ.
It is a statistic based on the sample and therefore it is a random
variable with a distribution function.
• The standard deviation of the sampling distribution of an estimator
is usually called the standard error of the estimator.
• For a given statistical model with unknown parameter θ there could
be more then one point estimate.
• The parameter θ of a statistical model can have more then just one
element.
week 2
10
Properties of Point Estimators
• Let ˆ be a point estimator for a parameter θ. Then ˆ is an unbiased
estimator if E ˆ    .
• The bias of a point estimator is given by
 
B ˆ  E ˆ  
• The variance of a point estimator is
      
2
var ˆ  E ˆ 2  E ˆ .
Ideally we would like our estimator to have minimum variance.
week 2
11
Mean Square Error of Point Estimators
• The mean square error (MSE) of a point estimator is



2
MSE ˆ  E  ˆ   .



    
• Claim: MSE ˆ  Var ˆ  B ˆ .
2
week 2
12
Common Point Estimators
• A natural estimate for the population mean μ is the sample mean (in
any distribution). The sample mean is an unbiased estimator of the
population mean.
• A common estimator for the population variance is the sample
variance s2.
week 2
13
Claim
• Let X1, X2,…, Xn be random sample of size n from a normal
population. The sample variance s2 is an unbiased estimator of the
population variance σ2.
• Proof…
week 2
14
Example
• Suppose X1, X2,…, Xn is a random sample from U(0, θ) distribution.
Let ˆ  X n  . Find the density of ˆ and its mean. Is ˆ unbiased?
week 2
15
Asymptotically Unbiased Estimators
Bˆ   0
• An estimator is asymptotically unbiased if lim
n
• Example:
week 2
16
Relative Efficiency
• Given two estimators ˆ1 and ˆ2 of a parameter θ with variances
Var ˆ1 and Var ˆ2 respectively. The efficiency of ˆ2 relative to ˆ1
 
 
is the ratio
 
 
Var ˆ`1
Var ˆ2
• Interpretation…
week 2
17
Example
• In the uniform example above let ˆ1  X n  and ˆ2  n  1 X n  .
n
Which point estimate is more efficient?
week 2
18