Chapters:
1. Unbiased Estimator
2. Mean Square Error
3. Sufficient Statistics
4. Complete Statistics
5. Exponential family
1.
Unbiasedness
Definition 1.1: - A statistics 𝑇 = 𝑇(𝑋1 , 𝑋2 , … … , 𝑋𝑛 ) is called an unbiased estimator
of 𝜃 if 𝐸(𝑇) = 𝜃. If 𝐸(𝑇) ≠ 𝜃 then T is called a biased estimator of 𝜃 and bias is
defined as,
𝐵𝑖𝑎𝑠(𝜃̂) = 𝐸(𝑇 − 𝜃)
Definition 1.2: - An estimator 𝑇 = 𝑇(𝑋1 , 𝑋2 , … … , 𝑋𝑛 ) is called asymptotically
unbiased estimator of 𝜃 if lim 𝐸(𝑇) = 𝜃 or,
𝑛⟶∞
lim 𝐵𝑖𝑎𝑠(𝜃̂) = 0
𝑛⟶∞
Definition 1.3: - The mean square error (MSE) of an estimator 𝜃̂ of a parameter
2
𝜃 is the function of 𝜃 defined by 𝐸(𝜃̂ − 𝜃) , and this is denoted as 𝑀𝑆𝐸𝜃̂ .
2
2
2
𝑀𝑆𝐸𝜃̂ = 𝐸(𝜃̂ − 𝜃) = 𝑉(𝜃̂ ) + (𝐸(𝜃̂) − 𝜃) = 𝑉(𝜃̂) + (𝐵𝑖𝑎𝑠 𝑜𝑓 𝜃̂)
2
2
If an estimator is unbiased then we have, 𝑀𝑆𝐸𝜃̂ = 𝐸(𝜃̂ − 𝜃) = 𝐸 (𝜃̂ − 𝐸(𝜃)) = 𝑉(𝜃̂)
Definition 1.4: - Let T and T’ be two unbiased estimators of the population
parameter 𝜃, such that 𝐸(𝑇) = 𝜃 and 𝐸(𝑇′) = 𝜃. Then the estimator T is efficient
relative to the estimator T’ if the variance of the finite-sample distribution of T
is less than or at most equal to the variance of the finite-sample distribution of
T’;
That is, if
𝑉(𝑇) ≤ 𝑉(𝑇′) for all finite n where 𝐸(𝑇) = 𝜃 = 𝐸(𝑇′)
o Both the estimators T and T’ must be unbiased, since the efficiency
property refers only to the variances of unbiased estimators.
1
o An efficiency, 𝑒θ of an estimator T* for estimating 𝜃 is defined as 𝑉(𝑇 ∗).
Example 1.1:- If T is an unbiased estimator of 𝛉. Then show that 𝐓 𝟐 is a biased
estimator of 𝛉𝟐 .
Sol: It is given that 𝐸(𝑇) = 𝜃.
We know that,
𝑉(𝑇) > 0
⟹ 𝐸(𝑇 2 ) − 𝐸 2 (𝑇) > 0
⟹ 𝐸(𝑇 2 ) − 𝜃 2 > 0
[∵ 𝐸(𝑇) = 𝜃]
⟹ 𝐸(𝑇 2 ) > 𝜃 2
This proves that 𝑇 2 is a biased estimator of 𝜃 2 .
Example 1.2:- If 𝐗 𝟏 , 𝐗 𝟐 , … . . , 𝐗 𝐧 ~𝐁𝐞𝐫(𝐩). Then find an unbiased estimator of 𝐩𝟐
based on 𝐗 𝟏 , 𝐗 𝟐 , … . . , 𝐗 𝐧 .
Sol: We know that if 𝑋1 , 𝑋2 , … . . , 𝑋𝑛 ~𝐵𝑒𝑟(𝑝) then 𝐸(𝑋𝑖 ) = 𝑛𝑝.
Consider a simple statistics ‘T’ based on 𝑋1 , 𝑋2 , … . . , 𝑋𝑛 as 𝑇 = ∑𝑛𝑖=1 X i then
𝑇~𝐵𝑖𝑛(𝑛, 𝑝) and hence 𝐸(𝑇) = 𝑛𝑝 𝑎𝑛𝑑 𝑉(𝑇) = 𝑛𝑝𝑞
Now,
𝐸(𝑇 2 ) − 𝐸 2 (𝑇) = 𝑛𝑝𝑞
⟹ 𝐸(𝑇 2 ) − (𝑛𝑝)2 = 𝑛𝑝𝑞 = 𝑛𝑝 − 𝑛𝑝2
⟹ 𝐸(𝑇 2 ) = (𝑛𝑝)2 + 𝑛𝑝 − 𝑛𝑝2 = 𝑛𝑝2 (𝑛 − 1) + 𝑛𝑝
⟹ 𝐸(𝑇 2 ) − 𝑛𝑝 = 𝑛𝑝2 (𝑛 − 1)
⟹ 𝐸(𝑇 2 ) − 𝐸(𝑇) = 𝑛𝑝2 (𝑛 − 1)
⟹ 𝐸(𝑇 2 − 𝑇) = 𝑛𝑝2 (𝑛 − 1)
𝑇(𝑇−1)
⟹ 𝐸 [𝑛(𝑛−1)] = 𝑝2
∴
𝑇(𝑇−1)
𝑛(𝑛−1)
is an unbiased estimator of 𝑝2 .
Example 1.3:- If 𝐗 𝟏 , 𝐗 𝟐 , … . . , 𝐗 𝐧 are a random sample from the uniform distribution
i.e. 𝐔(𝟎, 𝛉). Show that [
𝐧+𝟏
𝐧
𝐗 (𝐧) ] is an unbiased estimator of 𝛉.
Sol: Given that 𝑋1 , 𝑋2 , … . . , 𝑋𝑛 ~𝑈(0, 𝜃).
Transform Yi =
Xi
Therefore,
𝐸(𝑌(𝑗) ) = 𝑛+1
θ
then Y1, Y2, Y3………… Yn ~U(0,1) and Y(j) ~Beta(j, n − j + 1)
𝑗
𝑋(𝑗)
⟹𝐸(
𝜃
𝑗
[∵ 𝑌(𝑛) =
) = 𝑛+1
𝑋(𝑛)
𝜃
𝑗
⟹ 𝐸(𝑋(𝑗) ) = 𝑛+1 𝜃
𝑛
⟹ 𝐸(𝑋(𝑛) ) = 𝑛+1 𝜃
𝑛+1
⟹𝐸(
This proves that
𝑛
n+1
n
𝑋(𝑛) ) = 𝜃
X (n) is an unbiased estimator of θ.
Note: There are many other ways to this. One of them is to first find the pdf of
X(n) , where 𝑋(𝑛) = 𝑚𝑎𝑥(𝑋1 , 𝑋2 , … . . , 𝑋𝑛 ) and compute the expectation.
Example 1.4:- If X1 , X2 , … . . , Xn ~Poi(λ). Then find an unbiased estimator of 𝐞−λ
based on T = ∑ni=1 Xi .
Sol: We know that if 𝑋1 , 𝑋2 , … . . , 𝑋𝑛 ~𝑃𝑜𝑖(𝜆), then 𝑇 = ∑ni=1 Xi ~𝑃𝑜𝑖(𝑛𝜆).
Let ℎ(𝑇) be an unbiased estimator of e−λ
Therefore we must have,
E(ℎ(𝑇)) = 𝐞−λ
⟹
−nλ
∑∞
𝑡=0 ℎ(𝑡)𝐞
⟹
𝐞−nλ ∑∞
𝑡=0 ℎ(𝑡)
⟹
∑∞
𝑡=0 ℎ(𝑡)
By equating we have,
⟹
(nλ)t
t!
(nλ)t
t!
(nλ)t
t!
= 𝐞−λ
= 𝐞−λ
= 𝐞nλ 𝐞−λ = 𝐞(n−1)λ = ∑∞
𝑡=0
ℎ(𝑡)nt = (n − 1)t
[(n−1)λ]t
t!
]
⟹
ℎ(𝑡) =
n−1 t
(n−1)t
=(
nt
n
)
n−1 T
Hence, (
n
) is an unbiased estimator of e−λ .
Example 1.5:- Suppose 𝐗 𝟏 , 𝐗 𝟐 , … . . , 𝐗 𝐧 are i.i.d. random variables with probability
density function
1
f(x|σ) = 2σ exp (−
∑ |𝐗|
Show that an estimator 𝛔
̂=
𝐧
|x|
σ
is unbiased.
∞
1
) ; −∞ < 𝑥 < ∞; 𝜎 > 0
Sol: We have 𝐸(|𝑋|) = 2𝜎 ∫−∞ |𝑥| 𝑒𝑥𝑝 (−
∞
2
|𝑥|
𝜎
) 𝑑𝑥
𝑥
= 2𝜎 ∫0 𝑥 𝑒𝑥𝑝 (− 𝜎) 𝑑𝑥
∞
1
𝑥
= 𝜎 ∫0 𝑥 𝑒𝑥𝑝 (− 𝜎) 𝑑𝑥
[Having an even function]
[Mean of exponential dist]
=𝜎
Hence, 𝐸(𝜎̂) = 𝐸 (
∑𝑛
𝑖=1 |𝑋𝑖 |
𝑛
This shows that 𝜎̂ =
1
) = 𝑛 ∑𝑛𝑖=1 𝐸(|𝑋𝑖 |) = 𝜎
∑𝑛
𝑖=1 |𝑋𝑖 |
𝑛
is an unbiased estimator of 𝜎.
Example 1.6:- Let 𝐗 𝟏 , 𝐗 𝟐 , … . . , 𝐗 𝐧 be a random sample from a Bernoulli distribution
with parameter 𝐩; 𝟎 ≤ 𝐩 ≤ 𝟏. Find the bias of the estimator
estimating 𝐩.
for
[JAM 2006]
Sol: - Since Xi ~ Ber(p);
i = 1, 2, … . , n
Therefore, ∑ Xi ~Bin(n, p);
Now,
𝐧
√𝐧+𝟐 ∑𝐢=𝟏 𝐗 𝐢
𝟐(𝐧+√𝐧)
n
√n+2 ∑i=1 Xi
)
2(n+√n)
E(
⟹ E(∑ Xi ) = np
√n+2np
√n)
= 2(n+
Hence, Bias =
𝑛
√𝑛+2 ∑𝑖=1 𝑋𝑖
2(𝑛+√𝑛)
𝐸(
√𝑛+2𝑛𝑝
−
√𝑛)
− 𝑝) = 2(𝑛+
√𝑛(1−2𝑝)
√𝑛(1+√𝑛)
𝑝=2
1
= (1+
√
1
( − 𝑝)
𝑛) 2
Example 1.7:- Let X1 , X2 , … . . , Xn be a random sample from Ber(p); 𝟎 ≤ 𝐩 ≤ 𝟏. Show
1
that there doesn’t exists any unbiased estimator of p.
Sol: We know that if 𝑋1 , 𝑋2 , … . . , 𝑋𝑛 ~𝐵𝑒𝑟(𝑝) then 𝑇 = ∑𝑛𝑖=1 𝑋𝑖 ~𝐵𝑖𝑛(𝑛, 𝑝). Let ℎ(𝑇) be
1
a function of T which is an unbiased estimator of .
𝑝
Therefore we have,
1
𝐸[ℎ(𝑇)] = ∑𝑛𝑡=1 ℎ(𝑡)(𝑛𝑡)𝑝𝑡 (1 − 𝑝)𝑛−𝑡 = 𝑝
⟹
∑𝑛𝑡=1 ℎ(𝑡)(𝑛𝑡)𝑝𝑡+1 (1 − 𝑝)𝑛−𝑡 − 1 = 0
In the left hand side, we have a polynomial of degree n+1 in the variable p and it
must be zero for all p ∈ (0, 1). That is the polynomial must be identically equal
to zero and hence we must have −1 ≡ 0 which is a contradiction.
This shows that there doesn’t exists any unbiased estimator of
1
𝑝
based on T.
𝛉𝟐
̂ ) = , where
Example 1.8:- For a random variable X, we have 𝐄(𝐗) = 𝛉, 𝛉 > 0 𝐚𝐧𝐝 𝐯𝐚𝐫(𝛉
𝟐𝟓
̂ is an estimate of 𝛉 and 𝛉
̂ = 𝐤 𝐗. Also it is given that 𝐌𝐒𝐄(𝛉) = 𝟐[𝐛𝐢𝐚𝐬(𝛉)]𝟐 . Find k.
𝛉
𝐤+𝟏
Sol:
𝑘
𝑘
E(𝜃̂ ) = 𝐸 (𝑘+1 𝑋) = 𝑘+1 𝜃
as 𝐸(𝑋) = 𝜃
𝑘
−𝜃
∴
𝐵𝑖𝑎𝑠 = 𝐸(𝜃̂ − 𝜃) = 𝑘+1 𝜃 − 𝜃 = 𝑘+1
and
𝑀𝑆𝐸𝜃̂ = 𝑉(𝜃̂) + (𝐵𝑖𝑎𝑠 𝑜𝑓 𝜃̂)
2
by definition
2
2
2
−𝜃
2𝜃
It is also given that 𝑀𝑆𝐸𝜃̂ = 2[𝑏𝑖𝑎𝑠(𝜃̂)] = 2 (𝑘+1) = (𝑘+1)2
2
2
Hence 𝑉(𝜃̂) + (𝐵𝑖𝑎𝑠 𝑜𝑓 𝜃̂) = 𝑀𝑆𝐸𝜃̂ = 2(𝐵𝑖𝑎𝑠 𝑜𝑓 𝜃̂ )
2
⟹ 𝑉(𝜃̂) = (𝐵𝑖𝑎𝑠 𝑜𝑓 𝜃̂ )
⟹
𝜃2
−𝜃
2
= (𝑘+1)
25
⟹ 𝑘=4
Example 1.9:- Let X and Y are random variables with finite means. Find the
𝟐
function 𝐠(𝐗), for which 𝐄 [(𝐘 − 𝐠(𝐗)) ] is minimum, where 𝐠(𝐗) ranges over all
functions.
Sol: - WE have,
2
𝐸(𝑌 − 𝑔(𝑋)) = 𝐸(𝑌 − 𝐸(𝑌|𝑋) + 𝐸(𝑌|𝑋) − 𝑔(𝑋))
2
2
2
= 𝐸(𝑌 − 𝐸(𝑌|𝑋)) + 𝐸(𝐸(𝑌|𝑋) − 𝑔(𝑋))
+2𝐸[(𝑌 − 𝐸(𝑌|𝑋))(𝐸(𝑌|𝑋) − 𝑔(𝑋))]
The product term can be shown to zero by iterating the expectation. Thus
2
2
2
𝐸(𝑌 − 𝑔(𝑋)) = 𝐸(𝑌 − 𝐸(𝑌|𝑋)) + 𝐸(𝐸(𝑌|𝑋) − 𝑔(𝑋))
2
≥ 𝐸(𝑌 − 𝐸(𝑌|𝑋)) ,
for all 𝑔(. )
2
Therefore, 𝐸(𝑌 − 𝑔(𝑋)) is minimum when 𝑔(𝑋) = 𝐸(𝑌|𝑋).
Example 1.10:- Let 𝐗 𝟏 , 𝐗 𝟐 , … . . , 𝐗 𝐧 be a random sample from a 𝐍(𝛍, 𝛔𝟐 ) distribution,
where both µ and 𝛔𝟐 are unknown. Find the value of b that minimizes the mean
𝐛
̅)𝟐 for estimating 𝛔𝟐 , where 𝐗
̅=
squared error of the estimator 𝐓𝐛 = 𝐧−𝟏 ∑𝐧𝐢=𝟏(𝐗 𝐢 − 𝐗
𝟏
𝐧
∑𝐧𝐢=𝟏 𝐗 𝐢 .
[JAM 2006]
b
Sol: Given that, 𝑇𝑏 = n−1 ∑ni=1(𝑋𝑖 − 𝑋̅)2 = b𝑆 2
⟹
𝐸(𝑇𝑏 ) = E(b𝑆 2 ) = b𝜎 2 and 𝑉(𝑇𝑏 ) = V(b𝑆 2 ) =
Also,
𝑀𝑆𝐸(𝑇𝑏 ) = 𝑉(𝑇𝑏 ) + [𝐵𝑖𝑎𝑠(𝑇𝑏 )]2 =
2𝑏 2 𝜎4
n−1
2𝑏 2 𝜎4
n−1
.
2𝑏 2
+ (b − 1)2 𝜎 4 = 𝜎 4 {𝑏 2 − 2𝑏 + 1 + n−1}
Therefore, for minimum/maximum we have
𝑑
𝑑𝑏
𝑀𝑆𝐸(𝑇𝑏 ) = 0
4𝑏
⟹
𝜎 4 {2𝑏 − 2 + n−1} = 0
⟹
2𝑏(n − 1) − 2(n − 1) + 4b = 0
⟹
2𝑏(n − 1 + 2) = 2(n − 1)
⟹
𝑏 = (n+1)
(n−1)
(n−1)
Hence, 𝑀𝑆𝐸(𝑇𝑏 ) is minimum when 𝑏 = (n+1).
Example 1.11:- Let 𝐗 𝟏 , 𝐗 𝟐 , … . . , 𝐗 𝐧 be a random sample from a 𝐏(𝛌) distribution.
Give two different estimates of 𝜆 and determine which estimate is better.
Sol: Let 𝑋̅ and 𝑆 2 be the sample mean and sample variance, respectively.
1
Therefore, 𝐸(𝑋̅) = 𝜆 and 𝐸(𝑆 2 ) = E {n−1 ∑ni=1(𝑋𝑖 − 𝑋̅)2 } = 𝜎 2 = λ.
2
4
2
𝜎
𝜆
2𝜎
2λ
Also, 𝑉(𝑋̅) = 𝑛 = 𝑛 and 𝑉(𝑆 2 ) = 𝑛−1 = 𝑛−1.
Hence for some value of 𝜆, 𝑋̅ is better, 𝑆 2 is better.
[prove this]
2.
Sufficient Statistics
A sufficient statistic is one that contains all the necessary information about the
unknown parameters in a given population.
Definition 2.1: A statistic T(X) is sufficient for a given population parameter 𝜃 if
and only if the distribution of the data 𝑋1 , 𝑋2 , … . . , 𝑋𝑛 given 𝑇 = 𝑡 does not depend
on the unknown parameter 𝜃.
Example 2.1:- Let X1 , X2 , … . . , Xn ~Poi(λ). Find sufficient statistics for 𝛌.
Sol: Let, 𝑇 = ∑ Xi
⟹
𝑇~𝑃𝑜𝑖(𝑛𝜆)
Therefore,
P(𝑋1 = 𝑥1 , 𝑋2 = 𝑥2 , … . , 𝑋𝑛 = 𝑥𝑛 |𝑇 = 𝑡)
=
=
=
=
=
P(𝑋1 =𝑥1 ,𝑋2 =𝑥2 ,….,𝑋𝑛 =𝑥𝑛 ,𝑇=𝑡)
P(𝑇=𝑡)
P(𝑋1 =𝑥1 ,𝑋2 =𝑥2 ,….,𝑋𝑛 =𝑥𝑛 )
when ∑ xi = t
P(𝑇=𝑡)
P(𝑋1 =𝑥1 ) 𝑃(𝑋2 =𝑥2 ) …. 𝑃(𝑋𝑛 =𝑥𝑛 )
P(𝑇=𝑡)
because each Xi′ s are independent
e−λ (λ)𝑥1 e−λ (λ)𝑥2 ,…….e−λ (λ)𝑥𝑛
𝑥1 !𝑥2 !……𝑥𝑛 !
e−nλ (nλ)t
e−nλ (λ)∑ xi
𝑥1 !𝑥2 !……𝑥𝑛 !
e−nλ (nλ)t
t!
t!
=𝑥
n−t t!
,
1 !𝑥2 !……𝑥𝑛 !
independent of .
As, the conditional distribution 𝑋1 , 𝑋2 , … . . , 𝑋𝑛 |𝑇 = 𝑡 is independent of 𝜆, hence 𝑇 =
∑ni=1 Xi is sufficient statistics for 𝛌.
Example 2.2:- Let X1, X 2 ~Poi(λ). Then show that T = X1, +2X2 is not sufficient
statistics for 𝛌.
Sol:
𝑃(𝑇 = 2) = 𝑃(𝑋1 +2𝑋2 = 2) = 𝑃(𝑋1 = 0, 𝑋2 = 1) + 𝑃(𝑋1 = 2, 𝑋2 = 0)
= 𝑃(𝑋1 = 0)𝑃(𝑋2 = 1) + 𝑃(𝑋1 = 2)𝑃(𝑋2 = 0)
λ2
= e−λ . λe−λ + e−λ 2! . e−λ
λ
= λe−2λ (1 + 2)
Now,
P(𝑋1 = 𝑥1 , 𝑋2 = 𝑥2 |𝑇 = 𝑡) =
=
=
P(𝑋1 =0,𝑋2 =1,𝑇=2)
P(𝑇=2)
e−λ .λe−λ
λ
λe−2λ (1+ )
2
=
P(𝑋1 =𝑥1 ,𝑋2 =𝑥2 ,𝑇=𝑡)
P(𝑋1 =0,𝑋2 =1)
P(𝑇=𝑡)
P(𝑇=𝑡)
=
P(𝑋1 =0)𝑃(𝑋2 =1)
P(𝑇=𝑡)
λ
not independent of λ
= (1 + 2),
As, the conditional distribution 𝑋1 , 𝑋2 |𝑇 = 𝑡 is not independent of 𝜆, hence
𝑇 = 𝑋1, +2𝑋2 is not sufficient for λ.
Note: To prove not a sufficient statistics, you just need to give only 1 counter
example.
Example 2.3:- If X1, X2 ~N(µ, 1). Then show that T = X1 +2X2 is not sufficient for µ.
Sol: Here, we can’t compute P(T=t) because we are dealing with continuous
distribution.
However if we can show that the distribution of 𝑋1 |𝑇 depends on µ, then we can
say that T is not sufficient statistics for µ.
as
𝑋1, 𝑋2 ~𝑁(µ, 1) ⟹ 𝐸(𝑋1 ) = 𝐸(𝑋2 ) = µ and 𝑉(𝑋1 ) = 𝑉(𝑋2 ) = 1
Now,
𝐸(𝑇) = 𝐸(𝑋1 + 2𝑋2 ) = 𝐸(𝑋1 ) + 2𝐸(𝑋2 ) = 3µ
𝑉(𝑇) = 𝑉(𝑋1 + 2𝑋2 ) = 𝑉(𝑋1 ) + 4𝑉(𝑋2 ) + 4𝐶𝑜𝑣(𝑋1 , 𝑋2 ) = 5
𝐶𝑜𝑣(𝑋1 , 𝑇) = 𝐶𝑜𝑣(𝑋1 , 𝑋1 +2𝑋2 ) = 𝑉(𝑋1 ) + 2𝐶𝑜𝑣(𝑋1 , 𝑋2 ) = 𝑉(𝑋1 ) = 1
Hence,
𝜌=
𝐶𝑜𝑣(𝑇,𝑋1 )
√𝑉(𝑇)𝑉(𝑋1 )
=
1
√5
Therefore,
(𝑋1 , 𝑇)~𝐵𝑁 (µ, 3µ, 1,5,
1
)
√5
and hence,
(𝑋1 |𝑇)~𝑁 (µ +
1 1
√5 √5
1
(𝑇 − 3µ), (1 − ))
5
That is,
(𝑋1 |𝑇)~𝑁 (
𝑇+2µ 4
5
, 5)
As, the conditional distribution 𝑋1 |𝑇 = 𝑡 is not independent of µ, hence 𝑇 =
𝑋1, +2𝑋2 is not sufficient for µ.
The method we learn above involves computation of conditional probability and
sometimes it may be much difficult to compute the conditional probability. So
another method is to compute the sufficient statistics is given by Neyman.
The Neyman Factorization Theorem
Definition 2.2: A real valued statistic 𝑇 = 𝑇(𝑋1 , 𝑋2 , … . . , 𝑋𝑛 ) is sufficient for the
unknown parameter 𝜃 if and only if the following factorization holds:
𝐿(𝜃) = 𝑔(𝑇(𝑥1 , 𝑥2 , … . . , 𝑥𝑛 ); 𝜃)ℎ(𝑥1 , 𝑥2 , … . . , 𝑥𝑛 )
for all 𝑥1 , 𝑥2 , … . . , 𝑥𝑛 ;
where the two functions 𝑔(. ; 𝜃) and ℎ(. ) are both nonnegative, ℎ(𝑥1 , 𝑥2 , … . . , 𝑥𝑛 ) is
independent of 𝜃, and 𝑔(𝑇(𝑥1 , 𝑥2 , … . . , 𝑥𝑛 ); 𝜃) depends on 𝑥1 , 𝑥2 , … . . , 𝑥𝑛 only through
the observed value 𝑇(𝑥1 , 𝑥2 , … . . , 𝑥𝑛 ) of the statistic T with 𝜃.
Note:
o In the above statement it is not mandatory that 𝑔(𝑇(𝑥1 , 𝑥2 , … . . , 𝑥𝑛 ); 𝜃) must
be the pmf or the pdf of 𝑇(𝑥1 , 𝑥2 , … . . , 𝑥𝑛 ). However, it is essential that the
function ℎ(𝑥1 , 𝑥2 , … . . , 𝑥𝑛 ) must be entirely free from 𝜃.
o It should be noted that the splitting of likelihood function 𝐿(𝜃) may not be
unique, there may be more than one way to determine the function ℎ(. ) so
that the above relation holds. That’s why there can be different versions
of the sufficient statistics.
Example 2.4: If X1 , X2 , … . . , Xn ~Bin(p). Then find sufficient statistics for p.
Sol: Here, 𝐿(𝑝) = ∏𝑛𝑖=1 𝑝 𝑥𝑖 (1 − 𝑝)1−𝑥𝑖 = 𝑝∑ 𝑥𝑖 (1 − 𝑝)𝑛−∑ 𝑥𝑖 ,
Now consider 𝑔(∑𝑛𝑖=1 𝑥𝑖 ; 𝑝) = 𝑝∑ 𝑥𝑖 (1 − 𝑝)𝑛−∑ 𝑥𝑖 𝑎𝑛𝑑 ℎ(𝑥1 , 𝑥2 , … , 𝑥𝑛 ) = 1
Therefore, 𝐿(𝑝) =
𝑔(∑𝑛𝑖=1 𝑥𝑖
for all 𝑥1 , 𝑥2 , … , 𝑥𝑛 ∈ {0,1}
; 𝑝)ℎ(𝑥1 , 𝑥2 , … , 𝑥𝑛 )
Hence 𝑇 = 𝑇(𝑋1 , 𝑋2 , … . . , 𝑋𝑛 ) = ∑𝑛𝑖=1 𝑋𝑖 is sufficient for p.
Example 2.5:- If 𝐗 𝟏 , 𝐗 𝟐 , … . . , 𝐗 𝐧 ~𝐍(µ, 𝛔𝟐 ) where µ and 𝛔𝟐 are both unknown. Find
the joint sufficient statistics for 𝛉 = (µ, 𝛔).
Sol: Here,
𝐿(𝜃) = {𝜎√2𝜋}
−𝑛
1
𝑒𝑥𝑝 {− 2𝜎2 (∑𝑛𝑖=1 𝑥𝑖2 − 2µ ∑𝑛𝑖=1 𝑥𝑖 + 𝑛µ2 )},
1
Now consider, 𝑔(∑𝑛𝑖=1 𝑥𝑖 , ∑𝑛𝑖=1 𝑥𝑖2 ; 𝜃) = 𝜎 −𝑛 𝑒𝑥𝑝 {− 2𝜎2 (∑𝑛𝑖=1 𝑥𝑖2 − 2µ ∑𝑛𝑖=1 𝑥𝑖 + 𝑛µ2 )},
and
ℎ(𝑥1, 𝑥2, … . . , 𝑥𝑛 ) = {√2𝜋}
−𝑛
for all 𝑥1, 𝑥2, … . . , 𝑥𝑛 ∈ 𝑅 𝑛
Therefore, 𝐿(𝑝) = 𝑔(∑𝑛𝑖=1 𝑥𝑖 , ∑𝑛𝑖=1 𝑥𝑖2 ; 𝜃)ℎ(𝑥1, 𝑥2, … . . , 𝑥𝑛 )
Hence 𝑇 = 𝑇(𝑋1 , 𝑋2 , … . . , 𝑋𝑛 ) = (∑𝑛𝑖=1 𝑋𝑖 , ∑𝑛𝑖=1 𝑋𝑖2 ) is jointly sufficient for (µ, 𝜎).
Note:
o The sample itself 𝑋 = (𝑋1 , … … . . , 𝑋𝑛 ) is always sufficient for 𝜃, but this
provides no data reduction.
o The order statistics 𝑋(1) , … … . . , 𝑋(𝑛) are always sufficient for 𝜃.
o If T is a sufficient statistic for 𝜃, then 𝑔(𝑇), which is a one-to-one function
of T, is also sufficient for 𝜃.
̅ = 1 ∑𝑛𝑖=1 𝑋𝑖 , 𝑆 2 = 1 {∑𝑛𝑖=1 𝑋𝑖2 −
o So in above example if we consider,𝑋
1
(∑𝑛𝑖=1 𝑋𝑖 )2 },
𝑛
(𝑋̅ 2 )
𝑛
then it is one-to-one function of 𝑇 =
𝑛−1
𝑛
(∑𝑖=1 𝑋𝑖 , ∑𝑛𝑖=1 𝑋𝑖2 ).
Hence
𝑇 = , 𝑆 is also jointly sufficient for(µ, 𝜎 2 ).
o If 𝑇 = (𝑇1 , 𝑇2 ) is jointly sufficient for 𝜃 = (𝜃1 , 𝜃2 ). Then it doesn’t imply that
𝑇1 is sufficient for 𝜃1 or 𝑇2 is sufficient for 𝜃2 . So in above example we
can’t say that alone 𝑋̅ is sufficient for µ and 𝑆 2 is sufficient for 𝜎 2 because
𝑋̅ has some information about both µ and 𝑆 2 , whereas 𝑆 2 has information
about 𝜎 2 alone.
o If T be a sufficient statistic for 𝜃 then any arbitrary function T’ of T is not
necessarily sufficient for 𝜃.
Example 2.6:- Suppose the statistic T is sufficient for 𝜃. Then any monotonic
function 𝛏(𝐓) will also be sufficient for 𝜃.
Sol: Let 𝑌 = 𝜉(𝑇). Since 𝜉(𝑇) is monotonic therefore 𝑇 = 𝜉 −1 (𝑌) exists.
Now,
for all 𝑥1 , 𝑥2 , … . . , 𝑥𝑛
𝐿(𝜃) = 𝑔(𝑇; 𝜃)ℎ(𝑥1 , 𝑥2 , … . . , 𝑥𝑛 )
𝐿(𝜃) = 𝑔(𝜉 −1 (𝑌); 𝜃)ℎ(𝑥1 , 𝑥2 , … . . , 𝑥𝑛 )
𝐿(𝜃) = 𝜓(𝑌; 𝜃)ℎ(𝑥1 , 𝑥2 , … . . , 𝑥𝑛 )
Hence, Y is sufficient for 𝜃.
Example 2.7:- Suppose that X is distributed as 𝐍(𝛉, 𝟏) where −∞ < 𝛉 < ∞ is the
unknown parameter. Obviously, T = X is sufficient for 𝛉. One should check that
the statistic T’ = |X|, a function of T, is not sufficient for 𝛉.
Sol: Let 𝐺(. ) be the distribution function of T=|X|.
Therefore,
𝐺(𝑡) = 𝑃(|𝑋| ≤ 𝑡) = 𝑃(−𝑡 ≤ 𝑋 ≤ 𝑡) = 𝛷(𝑡 − 𝜃) − 𝛷(−𝑡 − 𝜃)
𝐹(𝑥, 𝑡) = 𝑃(𝑋 ≤ 𝑥, 𝑇 ≤ 𝑡) = 𝑃(𝑋 ≤ 𝑥, |𝑋| ≤ 𝑡) = 𝑃(∞ < 𝑋 ≤ 𝑥, −𝑡 ≤ 𝑋 ≤ 𝑡)
= 𝑃{𝑚𝑎𝑥(∞, −𝑡) < 𝑋 < 𝑚𝑖𝑛(𝑥, 𝑡)} = 𝑃{−𝑡 < 𝑋 < 𝑚𝑖𝑛(𝑥, 𝑡)}
= 𝛷{𝑚𝑖𝑛(𝑥, 𝑡) − 𝜃} − 𝛷(−𝑡 − 𝜃).
𝑓(𝑥, 𝑡) =
Example 2.8:- Suppose that 𝐗 𝟏 , 𝐗 𝟐 , … . , 𝐗 𝐧 be a random sample from 𝐔(𝟎, 𝛉)
distributed, where 𝛉(> 0) is unknown. Find sufficient statistics for 𝜃.
Sol: Here,
1
𝑓(𝑥𝑖 ) = 𝜃 𝐼(0 < 𝑥𝑖 < 𝜃),
where I is an indicator function defines as 𝐼 = {
1, 𝑥 < 𝜃
0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
1
1
𝐿(𝜃) = ∏𝑛𝑖=1 𝜃 𝐼(0 < 𝑥𝑖 < 𝜃) = 𝜃𝑛 𝐼(0 < 𝑥(𝑛) < 𝜃)
1
Now consider, 𝑔(𝑥(𝑛) ; 𝜃) = 𝜃𝑛 𝐼(0 < 𝑥(𝑛) < 𝜃) 𝑎𝑛𝑑 ℎ(𝑥1 , 𝑥2 , … , 𝑥𝑛 ) = 1
for all 𝑥1 , 𝑥2 , … , 𝑥𝑛 ∈ (0, 𝜃)
Therefore, 𝐿(𝜃) = 𝑔(𝑥(𝑛) ; 𝜃)ℎ(𝑥1 , 𝑥2 , … , 𝑥𝑛 )
Hence 𝑇 = 𝑇(𝑋1 , 𝑋2 , … . , 𝑋𝑛 ) = 𝑋(𝑛) is sufficient for 𝜃.
Example 2.9:- Let 𝐗 𝟏 , 𝐗 𝟐 , 𝐗 𝟑 be independent random variables with 𝐗 𝐤 (k = 1,2,3)
having the probability density function
−kθx
,
fk (x) = {kθe
0,
0<𝑥<∞
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
where 𝛉 > 𝟎. Then find a sufficient statistics for 𝜃.
[JAM 2008]
Sol: The likelihood function is given by
𝐿(𝜃) = 𝑓1 (x1 )𝑓2 (x2 )𝑓3 (x3 )
= 𝜃e−θx1 . 2𝜃e−2θx2 . 3𝜃e−3θx3 ,
0< 𝑥1, x2 , x3 < ∞
= 6𝜃 3 e−θ(x1+2x2+3x3)
This shows that x1 + 2x2 + 3x3 is sufficient for θ.
Note:
o The above example illustrate that we can also used Neyman Factorization
Theorem to prove our sufficient statistics for non-iid case.
Example 2.10:- The distribution of 𝑋 = (X1 , X 2 , X3 , X4 ) is given by
1
θ
P(X1 = x1 ) = 2 + 4 ,
P(X2 = x2 ) = P(X3 = x3 ) =
Show that (X1 , X2 + X3 , X4 ) is sufficient statistics for 𝜃.
Sol: The joint distribution of (X1 , X2 , X3 , X4 ) is
1−θ
4
θ
, P(X4 = x4 ) = 4
P(X = x) = x
=x
n!
1
1 !x2 !x3 !x4
n!
1 !x2 !x3 !x4
θ x1
( + 4) (
! 2
1
θ x1
( + 4) (
! 2
1−θ x2
1−θ x3
) (
4
1−θ x2 +x3
4
)
4
θ x4
) (4 )
θ x4
(4 )
= 𝑔(x1 , x2 + x3 , x4 , θ)ℎ(𝑥1 , 𝑥2 , 𝑥3 , 𝑥𝑛 )
Therefore, 𝑇 = (X1 , X2 + X3 , X4 ) is sufficient statistics for 𝜃.
Example 2.11:- Let the probability mass function of a random variable X be
P(X = 0) = θ,
P(X = 1) = 2θ,
P(X = 2) = 1 − 3θ;
1
0 < 𝜃 < 3.
Let (𝐓𝟏 , 𝐓𝟐 , 𝐓𝟑 ) be the vector of observed frequency of 0,1,2 in a sample of size n.
Show that (𝐓𝟏 , 𝐓𝟐 , 𝐓𝟑 ) is a sufficient statistics. Further assume that 𝐓𝟒 = 𝐓𝟏 + 𝐓𝟐 .
Then using the result that T4 ~Bin(n, 3θ), show that 𝐓𝟒 is also sufficient for 𝜃.
Sol: The joint distribution of (X1 , X2 , … … , Xn ) is
P(X = x) = (θ)t1 (2θ)t2 (1 − 3θ)t3
[where t1 + t 2 + t 3 = n]
Now consider, 𝑔(t1 , t 2 , t 3 , θ) = (θ)t1 (2θ)t2 (1 − 3θ)t3 𝑎𝑛𝑑 ℎ(𝑥1 , 𝑥2 , … , 𝑥𝑛 ) = 1
for all 𝑥1 , 𝑥2 , … , 𝑥𝑛
Therefore, P(X = x) = 𝑔(t1 , t 2 , t 3 , θ)ℎ(𝑥1 , 𝑥2 , … . , 𝑥𝑛 )
Hence, 𝑇 = (T1 , T2 , T3 ) is sufficient statistics for 𝜃.
Also, P(X = x|T4 ) = 𝑃(𝑋1 = 𝑥1 , … . , 𝑋𝑛 = 𝑥𝑛 |T4 = t)
=
=
𝑃(𝑋1 =𝑥1 ,….,𝑋𝑛 =𝑥𝑛 )
𝑃(T4 =t)
(θ)t1 (2θ)t2 (1−3θ)t3
(nt)(3θ)t (1−3θ)n−t
2t2 (θ)t1 +t2 (1−3θ)n−t1 +t2
= (n)(3θ)t1+t2 (1−3θ)n−t1+t2
t
2t2
= (n)3t
t
[as 𝑡 = t1 + t 2 and t1 + t 2 + t 3 = n]
As the conditional distribution of 𝑋|T4 is independent of 𝜃, hence T4 is also
sufficient for 𝜃.
Example 2.12:- Let X1, X2, … … , Xn be a random sample from a distribution with
density
𝜃1
𝑓(𝑥|𝜃1 𝜃2 ) = {
𝑒 −𝑥/𝜃2 , 𝑥 > 0
𝜃2
𝜃1
(1 − 𝜃 ) 𝑒 𝑥/𝜃2 , 𝑥 < 0
,
2
where 0 < 𝜃1 < 1 and 𝜃2 > 0. Show that (𝑺, 𝑲) is sufficient for (𝜽𝟏 , 𝜽𝟐 ), where K is
the number of positive 𝑿𝒊 ′𝒔, 𝑲 = ∑𝒏𝒊=𝟏 𝑰(𝑿𝒊 > 𝟎), and 𝑺 = ∑𝒏𝒊=𝟏 |𝑿𝒊 |.
Sol: Let k 𝑿𝒊 ′𝒔, out of n, are positive and n-k 𝑿𝒊 ′𝒔 are negative.
The likelihood function is given by
𝑘
𝜃
𝜃
𝑓(𝑥|𝜃1 𝜃2 ) = [𝜃1 𝑒 −𝑥/𝜃2 ] [(1 − 𝜃1 ) 𝑒 𝑥/𝜃2 ]
2
𝜃
𝑛−𝑘
2
𝑘
𝜃
𝑓(𝑥|𝜃1 𝜃2 ) = (𝜃1 ) (1 − 𝜃1 )
2
2
= 𝜃1 𝑘
(𝜃2 −𝜃1 )𝑛−𝑘
𝜃2
𝑛
𝑛−𝑘
𝑒
−
𝑒 − ∑1 𝑥/𝜃2 𝑒 ∑2 𝑥/𝜃2
1
(∑ 𝑥−∑2 𝑥)
𝜃2 1
, where ∑1 𝑥 is the sum of all 𝑋𝑖 ′𝑠
which are positive and ∑2 𝑥 is the sum of all 𝑋𝑖 ′𝑠 which are negative.
= 𝜃1 𝑘
(𝜃2 −𝜃1 )𝑛−𝑘
𝜃2
𝑛
𝑒
−
1
(∑𝒏 |𝑿 |)
𝜃2 𝒊=𝟏 𝒊
,
[ We note that ∑1 𝑥 − ∑2 𝑥 = ∑𝒏𝒊=𝟏 |𝑿𝒊 |; Suppose X=-10, -5, 2, 3, 5, then ∑1 𝑥 = 2 +
3 + 5 = 10 and ∑2 𝑥 = −10 − 5 = −15 and ∑𝒏𝒊=𝟏 |𝑿𝒊 | = 𝟏𝟎 + 𝟓 + 𝟐 + 𝟑 + 𝟓 = 𝟐𝟓.
That is ∑1 𝑥 − ∑2 𝑥 = 10 − (−15) = 25 = ∑𝒏𝒊=𝟏 |𝑿𝒊 | ]
Now consider, 𝑔(𝑆, 𝐾; 𝜃1 , 𝜃2 ) = 𝜃1 𝑘
(𝜃2 −𝜃1 )𝑛−𝑘
𝜃2
𝑛
𝑒
−
1
(∑𝒏 |𝑿 |)
𝜃2 𝒊=𝟏 𝒊
𝑎𝑛𝑑 ℎ(𝑥1 , 𝑥2 , … , 𝑥𝑛 ) = 1
for all 𝑥1 , 𝑥2 , … , 𝑥𝑛 ∈ (0, 𝜃)
Therefore, 𝐿(𝜃) = 𝑔(𝑆, 𝐾; 𝜃1 , 𝜃2 )ℎ(𝑥1 , 𝑥2 , … , 𝑥𝑛 )
Hence 𝑇 = 𝑇(𝑋1 , 𝑋2 , … . , 𝑋𝑛 ) = (𝑺, 𝑲) is sufficient for (𝜽𝟏 , 𝜽𝟐 ).
3.
Ancillary Statistics
We have learned that a sufficient statistic T preserves all the information about
𝜃 contained in the data X. However, an ancillary statistic T by itself provides no
information about the unknown parameter 𝜃. But still we learn them because
they can play an important role in statistical methodologies.
Definition 3.1: A statistic T is called ancillary for 𝜃 provided that the pmf or pdf
of T, denoted by f(t) for t ∈ T, does not involve the unknown parameter 𝜃 ∈ Θ.
Example 3.1: Suppose that 𝑿𝟏, 𝑿𝟐, 𝑿𝟑………… 𝑿𝒏 are iid 𝑵(𝜽, 𝟏) where 𝜃 ∈ 𝑹𝒏 is
unknown. Find few ancillary statistics.
Sol:
(1) A statistic 𝑇1 = 𝑋1 −𝑋2 is distributed as 𝑁(0,2) whatever be the value of the
unknown parameter 𝜃. Hence 𝑇1 is ancillary for 𝜃.
(2) Another statistic 𝑇2 = 𝑋1 +𝑋2 + ⋯ +𝑋𝑛−1 − (𝑛 − 1)𝑋𝑛 is distributed as
𝑁(0, 𝑛(𝑛 − 1)) whatever be the value of the unknown parameter 𝜃. Hence
𝑇2 is also ancillary for 𝜃.
1
2
(3) The sample variance 𝑆 2 is distributed as 𝑛−1 𝜒𝑛−1
whatever be the value of
the unknown parameter 𝜃 and hence 𝑆 2 is ancillary too for 𝜃.
Example 3.2: Suppose that 𝑿𝟏, 𝑿𝟐, 𝑿𝟑………… 𝑿𝒏 are iid 𝒆𝒙𝒑(𝜽), where 𝜃 (>0) is the
unknown parameter with n = 2. Find ancillary statistics.
Sol: Define 𝑌𝑖 = 𝜃𝑋𝑖 for 𝑖 = 1, … . 𝑛, then the joint distribution of 𝑌 = 𝑌1, 𝑌2………… 𝑌𝑛
does not involve the unknown parameter 𝜃.
𝑌
𝑋
Define 𝑇1 = 𝑌1 = 𝑋1 . We can observe that the distribution of 𝑇1 is independent of 𝜃
𝑛
𝑛
and so 𝑇1 is ancillary.
𝑌2
Also, define the statistic 𝑇2 = ∑1
𝑖=1 𝑌𝑛
𝑋2
= ∑1
𝑖=1 𝑋𝑖
and we can observe that the
distribution of 𝑇2 is independent of 𝜃 and so 𝑇2 is ancillary.
Note:
o Defining Ƒ1 , Ƒ2 𝑎𝑛𝑑 Ƒ3 be the location, scale and location-scale family
defines as
Ƒ1 = {𝑓(𝑥; 𝜃) = 𝑔(𝑥 − 𝜃): 𝜃 ∈ R, 𝑥 ∈ R };
𝑥
Ƒ2 = {𝑓(𝑥; 𝛿) = 𝛿 −1 𝑔 (𝛿) : 𝛿 ∈ R+ , 𝑥 ∈ R } ;
Ƒ3 = {𝑓(𝑥; 𝜃, 𝛿) = 𝛿 −1 𝑔 (
𝑥−𝜃
𝛿
) : 𝜃 ∈ R, 𝛿 ∈ R+ , 𝑥 ∈ R } ;
where 𝜃, 𝛿 may belong to some appropriate subsets of R and R+
o Let the common pdf of 𝑋1, 𝑋2, 𝑋3………… 𝑋𝑛 belongs to the location family Ƒ1 .
Then the statistics U = (𝑋1 − 𝑋𝑛 , 𝑋2 − 𝑋𝑛 , … … , 𝑋𝑛−1 − 𝑋𝑛 ) is ancillary.
o Let the common pdf of 𝑋1, 𝑋2, 𝑋3………… 𝑋𝑛 belongs to the scale family Ƒ2 . Then
𝑋
𝑋
the statistics V = ( 𝑋1 , 𝑋2 , … … ,
𝑛
𝑛
𝑋𝑛−1
𝑋𝑛
) is ancillary.
o Let the common pdf of 𝑋1, 𝑋2, 𝑋3………… 𝑋𝑛 belongs to the location-scale family
𝑋1 −𝑋𝑛 𝑋2 −𝑋𝑛
Ƒ3 . Then the statistics V = (
is the sample variance.
𝑆
,
𝑆
,……,
𝑋𝑛−1 −𝑋𝑛
𝑆
) is ancillary, where S2
Complete Statistics
4.
Definition 4.1: - A statistic T, is called complete if and only if for any real valued
function 𝒉(𝑡) defined for t ∈ T, having finite expectation, such that
𝐸[𝒉(𝑡)] = 0 for all 𝜃 ∈ Θ implies 𝑃[𝒉(𝑡) = 0] = 𝟏.
Note:
o
o
o
o
If
If
If
If
a non-constant function of T is ancillary, then T(X) is not complete.
E(T) does not depend on 𝜃, then T(X) is not complete.
T is complete then (T), is also complete.
T is complete, then only one unbiased estimator based on T is possible.
Example 4.1: Suppose that a statistic T is distributed as 𝐵𝑒𝑟(𝑝), 0 < 𝑝 < 1. Let us
examine if T is a complete statistic.
Sol: The pmf of T is given by
𝑔(𝑡; 𝜃) = 𝑝𝑡 (1 − 𝑝)1−𝑡 ,
𝑡 = 0, 1 𝑎𝑛𝑑 0 < 𝑝 < 1.
Consider any real valued function ℎ(𝑡) such that 𝐸[ℎ(𝑡)] = 0 for all 0 < 𝑝 < 1.
𝐸[ℎ(𝑡)] = ∑1𝑡=0 ℎ(𝑡) 𝑔(𝑡; 𝜃) = ℎ(0)(1 − 𝑝) + ℎ(1)𝑝 = 0
⟹ ℎ(0) + [ℎ(0) − ℎ(1)]𝑝 = 0
As above equation is linear in p and so it may be zero for exactly one value of p
between 0 and 1. But we need that ℎ(0) + [ℎ(0) − ℎ(1)]𝑝 must be zero for
infinitely many values of p in (0, 1).
Hence, this expression must be identically equal to zero which means that
ℎ(0) = 0 and ℎ(0) − ℎ(1) = 0.
⟹ ℎ(0) = 0 𝑎𝑛𝑑 ℎ(1) = 0
In other words, we have ℎ(𝑡) = 0 for 𝑡 = 0, 1. Thus,T is a complete statistic.
Definition 4.2: A statistic T is called complete sufficient for an unknown
parameter 𝜃 if and only if (i) T is sufficient for 𝜃 and (ii) T is complete.
Problem 4.2: Suppose that 𝐗 𝟏, 𝐗 𝟐, 𝐗 𝟑………… 𝐗 𝐧 be iid Poi(λ), λ > 0. Show that 𝐓 =
∑ni=1 Xi is a complete statistic.
Sol: We already have shown that T = ∑ni=1 Xi is a sufficient statistics for 𝜆. So
need to prove that T is also complete. We also know that T = ∑ni=1 X i ~𝑃𝑜𝑖(𝑛𝜆).
Consider any real valued function ℎ(𝑡) such that 𝐸[ℎ(𝑡)] = 0 for all 𝜆 > 0.
∞
−𝑛𝜆
𝐸[ℎ(𝑡)] = ∑∞
𝑡=0 ℎ(𝑡) 𝑔(𝑡; 𝜆) = ∑𝑡=0 ℎ(𝑡) 𝑒
(𝑛𝜆)𝑡
𝑡!
∀ 𝑡 ∈ 𝑇 = {0,1, … }
= 0;
𝑛𝑡
𝑡
⟹ ∑∞
𝑡=0 𝑘(𝑡) 𝜆 = 0,
where 𝑘(𝑡) = ℎ(𝑡) 𝑡!
We can see that the above expression can be expressed in an infinite power of 𝜆
(> 0) and if the infinite power series is equal to zero then each term will be
equal to zero.
Hence,
𝑘(𝑡) = 0 ∀ 𝑡 = 0,1, …
⟹ ℎ(𝑡) ≡ 0 ∀ 𝑡 = 0,1, …
Therefore, T is complete statistics.
Example 4.3: Suppose that X1, X2, X3………… Xn be iid N(θ, θ2 ), θ > 0. Show that 𝐓 =
̅, 𝐒 𝟐 ), is jointly sufficient for 𝜃 but not complete.
(𝐗
Sol:
Here,
𝐿(𝜃) = {√2𝜋θ2 }
−𝑛
1
𝑒𝑥𝑝 {− 2θ2 (∑𝑛𝑖=1 𝑥𝑖2 − 2𝜃 ∑𝑛𝑖=1 𝑥𝑖 + 𝑛𝜃 2 )},
1
Now consider, 𝑔(∑𝑛𝑖=1 𝑥𝑖 , ∑𝑛𝑖=1 𝑥𝑖2 ; 𝜃) = 𝜃 −𝑛/2 𝑒𝑥𝑝 {− 2θ2 (∑𝑛𝑖=1 𝑥𝑖2 − 2𝜃 ∑𝑛𝑖=1 𝑥𝑖 + 𝑛𝜃 2 )},
and
ℎ(𝑥1, 𝑥2, … . . , 𝑥𝑛 ) = {√2𝜋}
−𝑛
for all 𝑥1, 𝑥2, … . . , 𝑥𝑛 ∈ 𝑅 𝑛
Therefore, 𝐿(𝜃) = 𝑔(∑𝑛𝑖=1 𝑥𝑖 , ∑𝑛𝑖=1 𝑥𝑖2 ; 𝜃)ℎ(𝑥1, 𝑥2, … . . , 𝑥𝑛 )
Hence 𝑇′ = 𝑇′(𝑋1 , 𝑋2 , … . . , 𝑋𝑛 ) = (∑𝑛𝑖=1 𝑋𝑖 , ∑𝑛𝑖=1 𝑋𝑖2 ) is jointly sufficient for 𝜃.
Also 𝑇 = (𝑋̅, 𝑆 2 ) is jointly sufficient for 𝜃 because it is one-to-one function of
𝑇 ′ = (∑𝑛𝑖=1 𝑋𝑖 , ∑𝑛𝑖=1 𝑋𝑖2 ).
Now aim is to show that ‘T’ is not complete.
We know that, 𝐸(𝑋̅) = 𝜃 and also 𝐸(𝑆 2 ) = θ2 for all 𝜃>0.
Thus we have, 𝐸(𝑋̅ 2 − 𝑆 2 ) = 0.
Consider ℎ(𝑇) = 𝑋̅ 2 − 𝑆 2 and then we have 𝐸[ℎ(𝑇)] ≡ 0 for all 𝜃>0 but ℎ(𝑡) = 𝑥̅ 2 − 𝑠 2
is not identically zero ∀ 𝑡 ∈ 𝑅 × 𝑅 + .
Therefore, T is jointly sufficient for 𝜃 but not complete.
Example 4.4: Suppose that X1, X2, X3………… Xn be iid with the common pdf given by
1
f(x) = θ exp {−
(x−θ)
θ
},
where x > 𝜃
Check whether the sufficient statistics is complete or not.
Sol:
1
Here,
𝑓(𝑥) = 𝜃 𝑒𝑥𝑝 {−
(𝑥−𝜃)
𝜃
} 𝐼(𝜃 < 𝑥 < ∞),
where I is an indicator function defined as 𝐼 = {
1
𝐿(𝜃) = ∏𝑛𝑖=1 𝜃 𝑒𝑥𝑝 {−
(𝑥−𝜃)
𝜃
1
} 𝐼(𝜃 < 𝑥𝑖 < ∞) = 𝜃𝑛 𝑒𝑥𝑝 {−
1
Now consider 𝑔(∑𝑛𝑖=1 𝑥𝑖 , 𝑥(1) ; 𝜃) = 𝜃𝑛 𝑒𝑥𝑝 {−
Therefore, 𝐿(𝜃) =
1, 𝑥 > 𝜃
0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
∑(𝑥−𝜃)
𝜃
} 𝐼(𝑥(1) > 𝜃)
∑(𝑥−𝜃)
𝜃
} 𝐼(𝑥(1) > 𝜃) 𝑎𝑛𝑑 ℎ(𝑥1, 𝑥2, … . . , 𝑥𝑛 ) = 1
𝑔(∑𝑛𝑖=1 𝑥𝑖 , 𝑥(1) ; 𝜃)ℎ(𝑥1, 𝑥2, … . . , 𝑥𝑛 )
for all 𝑥1, 𝑥2, … . . , 𝑥𝑛 ∈ (𝜃, ∞)
Hence 𝑇′ = 𝑇′(𝑋1, 𝑋2, 𝑋3………… 𝑋𝑛 ) = (∑𝑛𝑖=1 𝑋𝑖 , 𝑋(1) ) is jointly sufficient for 𝜃. Also 𝑇 =
(∑𝑛𝑖=1(𝑋𝑖 − 𝑋(1) ), 𝑋(1) ) is one to one function of T’ and hence T is also jointly
sufficient for 𝜃.
Now let 𝑧 =
𝑥−𝜃
𝜃
⟹ 𝜃𝑑𝑧 = 𝑑𝑥.
Also 𝜃𝑍(1) = 𝜃𝑚𝑖𝑛(𝑍1 , … 𝑍𝑛 ) = 𝑚𝑖𝑛(𝑋1 , … 𝑋𝑛 ) − 𝜃 = 𝑋(1) − 𝜃
Therefore,
𝑓(𝑧) = 𝑒 −𝑧 ;
𝑧>0
⟹ 𝑍~𝑒𝑥𝑝(1) and hence,
𝑋(1) −𝜃
⟹ 𝐸(𝑍(1) ) = 𝐸 (
𝜃
𝜃
⟹ 𝐸(𝑋(1) ) = 𝑛 + 𝜃 =
𝑍(1) = 𝑚𝑖𝑛(𝑍1 , … 𝑍𝑛 )~𝑒𝑥𝑝(𝑛)
1
)=𝑛
𝜃(1+𝑛)
.
𝑛
We also have 𝐸(∑𝑛𝑖=1 𝑋𝑖 ) = ∑𝑛𝑖=1 𝐸(𝑋𝑖 ) = ∑𝑛𝑖=1 𝐸(𝜃 + 𝜃𝑍𝑖 ) = 2𝑛𝜃
Therefore,
𝐸(∑𝑛𝑖=1(𝑋𝑖 − 𝑋(1) )) = 2𝑛𝜃 − 𝜃(1 + 𝑛) = (𝑛 − 1)𝜃
⟹
1
𝐸 [(𝑛−1) ∑𝑛𝑖=1(𝑋𝑖 − 𝑋(1) )] = 𝜃
Let us define,
1 −1
ℎ(𝑡) = (1 + )
𝑛
⟹
𝑥(1) − (𝑛 − 1)−1 ∑𝑛𝑖=1(𝑥𝑖 − 𝑥(1) )
1 −1
𝐸[ℎ(𝑇)] = 𝐸 [(1 + 𝑛)
1 −1
⟹
= (1 + 𝑛)
⟹
= (1 + 𝑛)
⟹
=𝜃−𝜃 =0
1 −1
∀ 𝑡 ∈ 𝑇 = {𝜃, ∞} × 𝑅 +
𝑋(1) − (𝑛 − 1)−1 ∑𝑛𝑖=1(𝑋𝑖 − 𝑋(1) )]
1
𝐸(𝑋(1) ) − 𝑛−1 𝐸[∑𝑛𝑖=1(𝑋𝑖 − 𝑋(1) )]
1
𝐸(𝑋(1) ) − 𝑛−1 𝐸[∑𝑛𝑖=1(𝑋𝑖 − 𝑋(1) )]
1 −1
Therefore, 𝐸[ℎ(𝑇)] ≡ 0 for all 𝜃>0 but ℎ(𝑡) = (1 + 𝑛)
𝑥(1) − (𝑛 − 1)−1 ∑𝑛𝑖=1(𝑥𝑖 − 𝑥(1) )
is not identically zero ∀ 𝑡 ∈ 𝑇 = {𝜃, ∞} × 𝑅 + .
Therefore, 𝑇 = (∑𝑛𝑖=1(𝑋𝑖 − 𝑋(1) ), 𝑋(1) ) is sufficient for 𝜃 but not complete.
Example 4.5:- Suppose that 𝐗 𝟏, 𝐗 𝟐, 𝐗 𝟑………… 𝐗 𝐧 ~𝐔(𝟎, 𝛉) with 𝛉 > 0 being the unknown
parameter. Show that 𝐓 = 𝐗 (𝐧) is complete sufficient statistics for 𝜃.
Sol: We already have shown that 𝑇 = 𝑋(𝑛) = 𝑚𝑎𝑥(𝑿𝟏, 𝑿𝟐, … … . , 𝑿𝒏 ) is a sufficient
statistic for 𝜃. In this case and the pdf of T is given by
𝑓(𝑡) = 𝑛
𝑡 𝑛−1
𝜃𝑛
;
0<𝑡<𝜃
Let ℎ(𝑡) be any arbitrary real valued function such that 𝐸[ℎ(𝑡)] = 0 for all 𝜃> 0.
Therefore we have,
θ
E[h(t)] = ∫0 h(t) n
tn−1
θn
dt = 0
θ
⟹ ∫0 h(t) t n−1 dt = 0
⟹ h(θ)θn−1 = 0
[using derivatives]
for all θ > 0
⟹ h(θ) ≡ 0,
This shows that T = X (n) is complete.
Hence 𝑇 = 𝑋(𝑛) is complete sufficient statistics for 𝜃.
Example 4.6:- Suppose that 𝐗 𝟏 , 𝐗 𝟐 , … … , 𝐗 𝐧 are iid with the density function
𝑓(x; θ) = e−(x−θ) ;
𝑥 > θ,
Show that 𝑻 = 𝑿(𝟏) is complete statistic for 𝜃.
Sol: Let 𝐹𝑋 (. ) be the distribution function of X,
𝐹𝑋 (𝑡) = 1 − e−(t−θ)
Now,
𝑓𝑇 (𝑡) = 𝑛𝑓𝑋 (𝑡)(1 − 𝐹𝑋 (𝑡))
⟹ 1 − 𝐹𝑋 (𝑡) = e−(t−θ)
𝑛−1
= ne−(t−θ) [e−(t−θ) ]
𝑓(𝑡) = ne−n(t−θ) ;
n−1
= ne−n(t−θ)
𝑡>𝜃
Let ℎ(𝑡) be any arbitrary real valued function such that 𝐸[ℎ(𝑡)] = 0 for all 𝜃> 0.
Therefore we have,
∞
E[h(t)] = ∫θ h(t) ne−n(t−θ) dt = 0
∞
⟹ ∫θ h(t) e−nt dt = 0
[∵ ne−nθ > 0]
⟹ −h(θ)enθ = 0
This shows that T = X (1)
[using derivatives]
for all θ > 0
⟹ h(θ) ≡ 0,
is complete.
Example 4.7:- If there exists an unbiased estimator of 𝜃 which is complete
sufficient statistics, then it is unique.
Sol: Suppose T is complete sufficient statistics. Further assume that 𝑔1 (T) and
𝑔2 (T) be two unbiased estimator of 𝜃.
Therefore,
𝐸[𝑔1 (T)] = 𝜃 = 𝐸[𝑔2 (T)]
⟹
𝐸[𝑔1 (T) − 𝑔2 (T)] = 0
Hence, 𝑔1 (T) ≡ 𝑔2 (T) with probability 1.
Example 4.8:- Consider the probability mass function of X is given by
𝑃(X = x) = (1 − θ)2 θx ;
𝑥 = 0,1,2,3 ….
= θ;
𝑥 = −1
= 0;
otherwise
Show that the family is not complete.
Sol: Let 𝜓(X) be any real valued function such that 𝐸[𝜓(X)] = 0.
⟹
∑∞
𝑥=−1 𝜓(X)𝑃(X = x) = 0
⟹
𝜓(−1)𝑃(X = −1) + ∑∞
𝑥=0 𝜓(X)𝑃(X = x) = 0
⟹
2 x
𝜓(−1)θ + ∑∞
𝑥=0 𝜓(X)(1 − θ) θ = 0
⟹
x
−2
∑∞
𝑥=0 𝜓(X)θ = − (1−θ)2 𝜓(−1) = −𝜓(−1)θ(1 − θ)
⟹
∞
x
2
x
∑∞
𝑥=0 𝜓(X)θ = −𝜓(−1)θ[1 + 2θ + 3θ + ⋯ . . ] = −𝜓(−1) ∑𝑥=1 xθ
⟹
∞
∞
x
x
x
∑∞
𝑥=0 𝜓(X)θ = − ∑𝑥=1 𝜓(−1)xθ = ∑𝑥=0 −𝜓(−1)xθ
θ
By equating we have, 𝜓(x) = −𝑥𝜓(−1),
This shows that the family is not complete.
Exponential Parameter Family:
(1) One parameter form: Suppose that 𝐗 𝟏, 𝐗 𝟐, … … , 𝐗 𝐧 are iid with the common
pmf/pdf belonging to the 1-parameter exponential family if the likelihood
is defined by 𝑓(𝑥; 𝜃) = 𝑎(𝑥)𝑏(𝜃)𝑒𝑥𝑝{𝑐(𝜃)𝑇(𝑥)}. Then 𝑇(𝑥) is complete
sufficient statistics for 𝜃.
(2) Two parameter form: Suppose that X1, X2, … … , Xn are iid with the common
pmf/pdf belonging to the 2-parameter exponential family if the likelihood
is defined by 𝑓(𝑥; 𝜃) = 𝑎(𝑥)𝑏(𝜃)𝑒𝑥𝑝{𝑐1 (𝜃1 )𝑇1 (𝑥) + 𝑐2 (𝜃2 )𝑇2 (𝑥)}. Then 𝑇 =
(𝑇1 , 𝑇2 ) is complete sufficient statistics for 𝜃.
Bernoulli, Binomial, Poisson, Normal, Exponential, Gamma, Beta, Geometric etc.
are the example of exponential family.
o For Bernoulli: 𝑓(𝑥; 𝑝) = 𝑝
∑𝑛
𝑖=1 𝑥𝑖
(1 − 𝑝)
𝑛−∑𝑛
𝑖=1 𝑥𝑖
𝑛
𝑝
∑𝑛
𝑖=1 𝑥𝑖
= (1 − 𝑝) (1−𝑝)
𝑝
= (1 − 𝑝)𝑛 𝑒𝑥𝑝 [∑𝑛𝑖=1 𝑥𝑖 𝑙𝑛 (1−𝑝)]
Therefore, 𝑇 = ∑𝑛𝑖=1 𝑋𝑖 is complete sufficient for 𝑝.
𝑛
o For Poisson: 𝑓(𝑥; 𝜆) = 𝑒 −𝑛𝜆 𝑥
𝜆∑𝑖=1 𝑥𝑖
1 !𝑥2 !……𝑥𝑛
= 𝑒 −𝑛𝜆
!
𝑒𝑥𝑝(∑𝑛
𝑖=1 𝑥𝑖 𝑙𝑛𝜆)
𝑥1 !𝑥2 !……𝑥𝑛 !
Therefore, 𝑇 = ∑𝑛𝑖=1 𝑋𝑖 is complete sufficient for 𝜆.
o For Normal: 𝑓(𝑥; 𝜃) =
𝑥𝑖 −µ 2
1
− (∑𝑛
1
𝑖=1( 𝜎 ) )
2
𝑒
(2𝜋𝜎)𝑛/2
=
𝑛
2
1 ∑𝑖=1 𝑥𝑖
𝜎2
− (
1
2
𝑒
(2𝜋𝜎)𝑛/2
−
2µ ∑𝑛
𝑖=1 𝑥𝑖
𝜎2
Therefore, 𝑇 = (∑𝑛𝑖=1 𝑋𝑖 , ∑𝑛𝑖=1 𝑋𝑖2 ) is complete sufficient for θ= (µ, 𝜎 2 ).
𝑛
o For Exponential: 𝑓(𝑥; 𝜆) = 𝜆𝑛 𝑒 −𝜆 ∑𝑖=1 𝑥𝑖 = 𝜆𝑛 𝑒𝑥𝑝(𝜆 ∑𝑛𝑖=1 𝑥𝑖 )
Therefore, 𝑇 = ∑𝑛𝑖=1 𝑋𝑖 is complete sufficient for 𝜆.
𝛼𝛽
𝑛
𝛽−1
o For Gamma: 𝑓(𝑥; 𝛼, 𝛽) = 𝛤𝛽 𝑒 −𝛼 ∑𝑖=1 𝑥𝑖 ∏𝑛𝑖=1 𝑥𝑖
𝛼𝛽
= 𝛤𝛽 𝑒𝑥𝑝(−𝛼 ∑𝑛𝑖=1 𝑥𝑖 + (𝛽 − 1) ∑𝑛𝑖=1 𝑙𝑛𝑥𝑖 )
+
𝑛µ2
)
𝜎2
Therefore, 𝑇 = (∑𝑛𝑖=1 𝑋𝑖 , ∑𝑛𝑖=1 𝑙𝑛𝑋𝑖 ) is complete sufficient for (𝛼, 𝛽).
𝑛
o For Geometric: 𝑓(𝑥; 𝑝) = 𝑝(1 − 𝑝)∑𝑖=1 𝑥𝑖 = 𝑝 𝑒𝑥𝑝(∑𝑛𝑖=1 𝑥𝑖 𝑙𝑛(1 − 𝑝))
Therefore, 𝑇 = ∑𝑛𝑖=1 𝑋𝑖 is complete sufficient for p.
Basu’s Theorem: Suppose that we have two statistics, T = T(X) which is complete
sufficient for 𝜃 and W = W(X) which is ancillary for 𝜃. Then, T and W are
independently distributed.
Proof: To prove this theorem, we need to show P(W ≤ w|T = t) = P(W ≤ w).
Then we can say that W and T are independently distributed.
Since, T is complete sufficient statistics, therefore 𝑔(𝑡) = P(W ≤ w|T = t) is
independent of 𝜃 and as W is ancillary statistics, P(W ≤ w) is independent of 𝜃.
Now, let us define
1,
W≤w
Iw = {
0,
𝑊>𝑤
E[E(Iw |T)] = E(Iw ) = P(W ≤ w)
⟹
E[P(W ≤ w|T)] = P(W ≤ w)
⟹
E[g(T) − P(W ≤ w)] = 0
As T is complete statistics, so we have if E[Ψ(T)] = 0 then Ψ(T) = 0.
Hence g(T) = P(W ≤ w) or P(W ≤ w|T) = P(W ≤ w).
Example 4.9:- Let 𝐗 𝟏, 𝐗 𝟐, … … , 𝐗 𝐧 ~𝐍(µ, 𝛔𝟐 ) with n ≥ 2, (µ, σ) ∈ ℜ × ℜ+, where µ is
̅ and 𝐒 𝟐 are independently distributed.
unknown, but σ is known. Show that, 𝐗
Sol: We know that 𝑈 = ̅
X is complete sufficient statistic for µ. Also it can be
shown that W = S 2 is an ancillary statistic for µ because the distribution of W
doesn’t involve µ.
̅ and 𝐒 𝟐 are then independently
Now, by Basu’s Theorem, the two statistics 𝐗
distributed.
o
o
o
o
̅
X and (Xi − ̅
X) are also independently distributed.
̅ and (Xi − M) are also independently distributed (M is the median).
X
̅ and (X(n) − X
̅) are also independently distributed.
X
(X1 − X2 ) and (X1 + X2 ) are independently distributed because (X1 + X2 ) is
sufficient statistics and(X1 − X2 ) is ancillary, when X1 , X2 ~N(µ, σ2 ).
4.10:- Suppose that 𝐗 𝟏, 𝐗 𝟐, … … , 𝐗 𝐧 ~𝐔(𝟎, 𝛉) with n ≥ 2, 𝜃 (< 0) being the unknown
parameter. Show that 𝐗 (𝐧) and
𝐗 (𝟏)
𝐗 (𝐧)
are independently distributed.
Sol: We know that U = X(n) is a complete sufficient statistic for 𝜃. It can be
X(1)
shown that 𝑊 = X
X(1)
X(n)
(n)
which is ancillary for 𝜃. Hence, by Basu.s Theorem, X(n) and
are independently distributed.
X(1)
o E(X(1) ) = E (X
(n)
X(1)
X(1)
E(X(1) )
(n)
(n)
(n) )
. X(n) ) = E (X ) E(X(n) ) ⟹ E (X ) = E(X
̅
X
o X(n) and (S ) are also independently distributed.
4.11:- Suppose that 𝐗 𝟏, 𝐗 𝟐, … … , 𝐗 𝐧 ~𝐍(𝛉, 𝟏). Find the covariance between sample
mean and sample median.
̅ and M be the sample mean and sample median. Then it is seen that X
̅
Sol: Let X
̅
be complete sufficient statistics for 𝜃 and (X − M) is ancillary. Then Basu says
̅ − M) are independently distributed.
that ̅
X and (X
1
̅, X
̅ − M) = 𝑉(X
̅) − 𝑐𝑜𝑣(X
̅, M) = − 𝑐𝑜𝑣(X
̅, M)
Therefore, 0 = 𝑐𝑜𝑣(X
n
1
̅, M) =
⟹ 𝑐𝑜𝑣(X
n
4.12:- Let X1, X2, … … , X n ~exp(θ). Define 𝒈(X) = 𝐗
𝐗𝟏
𝟏 +𝐗 𝟐 +⋯+𝐗 𝐧
. Find 𝑬[𝒈(X)].
Sol: Let 𝑍𝑖 = 𝜃𝑋𝑖 , then 𝑍𝑖′ 𝑠 are exp(1) and hence independent of 𝜃. Then we have
Z1
𝑔(X) = Z +Z +⋯+Z
= 𝑔(Z) (say). Now 𝑔(Z) ~𝑔(X) and hence 𝑔(X) is ancillary
1
statistics.
2
n
Therefore 𝜃 = 𝐸(X1 ) = E [(X
X1
1 +X2 +⋯+Xn
) (X1 + X 2 + ⋯ + Xn )]
= E[𝑔(X)T(X)]
= E[𝑔(X)]E[T(X)]
= E[𝑔(X)]nθ
Hence,
1
E[𝑔(X)] = 𝑛
[As T(X) is complete sufficient statistics]
[by Basu’s Theorem]
[As E[T(X)] = 𝑛𝜃]
5.
Consistency
An estimator 𝑇(= 𝜃̂𝑛 ) is said to be a consistent estimator of 𝜃 if, for all 𝜖 > 0,
lim 𝑃(|𝜃̂𝑛 − 𝜃| > 𝜖) = 0;
𝑛→∞
that is, the sequence of real numbers 𝑃(|𝜃̂𝑛 − 𝜃| > 𝜖) → 0 as 𝑛 → ∞.
Consistency is a desirable large-sample property. If 𝜃̂𝑛 is consistent, then the
probability that the estimator 𝜃̂𝑛 differs from the true 𝜃 becomes small as the
sample size n increases. On the other hand, if you have an estimator that is not
consistent, then no matter how many data you collect, the estimator 𝜃̂𝑛 may
never “converge” to 𝜃.
Definition 5.1: If an estimator 𝜃̂𝑛 is consistent, we say that 𝜃̂𝑛 converges in
𝑝
probability to 𝜃 and write 𝜃̂𝑛 ⟶
𝜃.
Suppose that 𝜃̂𝑛 is an estimator of 𝜃. If both 𝐸(𝜃̂𝑛 ) ⟶ θ and 𝑉(𝜃̂𝑛 ) ⟶ 0, as 𝑛 ⟶ ∞,
then 𝜃̂𝑛 is a consistent estimator for 𝜃. In many problems, it will be much easier
to show that 𝐸(𝜃̂𝑛 ) ⟶ θ and 𝑉(𝜃̂𝑛 ) ⟶ 0, as 𝑛 ⟶ ∞, rather than showing
𝑃(|𝜃̂𝑛 − 𝜃| > 𝜖) ⟶ 0 i.e., appealing directly to the definition of consistency.
Example 5.1: Show that the estimator T is consistent for 𝜃 if 𝐸(𝑇) ⟶ θ and
𝑉(𝑇) ⟶ 0, as 𝑛 ⟶ ∞.
Sol: For consistency we need to prove 𝑃(|𝑇 − 𝜃| > 𝜖) ⟶ 0 as 𝑛 ⟶ ∞.
We know that, 𝑃(|𝑇 − 𝜃| > 𝜖) = 𝑃((𝑇 − 𝜃)2 > 𝜖 2 ) ≤
=
=
𝐸(𝑇−𝐸(𝑇)+𝐸(𝑇)−𝜃)2
𝜖2
𝑉(𝑇)+[𝐸(𝑇)−𝜃]2
𝜖2
𝐸(𝑇−𝜃)2
𝜖2
2
=
𝐸(𝑇−𝐸(𝑇)) +[𝐸(𝑇)−𝜃]2
𝜖2
⟶ 0 as 𝑛 ⟶ ∞
because 𝐸(𝑇) ⟶ θ ⇔ 𝐸(𝑇) − 𝜃 ⟶ 0 and also 𝑉(𝑇) ⟶ 0, as 𝑛 ⟶ ∞.
Hence, T is consistent for 𝜃.
Example 5.2: If T is consistent for 𝜃 and g is a function continuous at 𝜃, then 𝒈(𝑇) is
consistent for 𝒈(𝜃).
Sol: For any continuous function g, we have
{|𝑔(𝑋(𝑛) ) − 𝑔(𝜃)| ≥ 𝜖} ⟹ {|𝑋(𝑛) − 𝜃| ≥ 𝛿}
Hence, 𝑃{|𝑔(𝑋(𝑛) ) − 𝑔(𝜃)| ≥ 𝜖} ≤ 𝑃{|𝑋(𝑛) − 𝜃| ≥ 𝛿} ⟶ 0, as 𝑛 ⟶ ∞
Hence 𝑔(𝑇) is consistent for 𝑔(𝜃).
Example 5.1: Suppose that X1 , … . , Xn is an iid sample from U(0,𝜃), 𝜃>0. Show that
𝑿(𝒏) is consistent for 𝜃.
Sol: We have, 𝑃(|𝑋(𝑛) − 𝜃| > 𝜖) = 𝑃(𝑋(𝑛) − 𝜃 > 𝜖) + 𝑃(𝑋(𝑛) − 𝜃 < −𝜖)
= 𝑃(𝑋(𝑛) > 𝜃 + 𝜖) + 𝑃(𝑋(𝑛) < 𝜃 − 𝜖)
= 𝑃(𝑋(𝑛) < 𝜃 − 𝜖)
[since 𝑃(𝑋(1) > 𝜃 + 𝜖) = 0]
= [𝑃(𝑋 < 𝜃 − 𝜖)]𝑛
[when 𝜃 > 𝜖]
𝜖 𝑛
= (1 − 𝜃) ⟶ 0, as 𝑛 ⟶ ∞
𝜖
[since 0 < (1 − 𝜃) < 1]
Hence, 𝜃̂𝑛 = 𝑋(𝑛) is a consistent estimator of 𝜃.
Example 5.1: Suppose that X1 , … . , Xn is an iid sample from a shifted-exponential
distribution
−(𝑥−𝜃)
,
𝑥>𝜃
𝑓(𝑥; 𝜃) = {𝑒
0,
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒.
Show that the first order statistic 𝜃̂𝑛 = 𝑋(1) is a consistent estimator of 𝜃.
Sol: We have, 𝑃(|𝑋(1) − 𝜃| > 𝜖) = 𝑃(𝑋(1) − 𝜃 > 𝜖) + 𝑃(𝑋(1) − 𝜃 < −𝜖)
= 𝑃(𝑋(1) > 𝜃 + 𝜖) + 𝑃(𝑋(1) < 𝜃 − 𝜖)
= 𝑃(𝑋(1) > 𝜃 + 𝜖)
= [𝑃(𝑋 > 𝜃 + 𝜖)]𝑛
= 𝑒 −𝑛𝜖 ⟶ 0, as 𝑛 ⟶ ∞
[since 𝑃(𝑋(1) < 𝜃 − 𝜖) = 0]
Hence, 𝜃̂𝑛 = 𝑋(1) is a consistent estimator of 𝜃.
Example 5.2:- Let 𝐗 𝟏 , 𝐗 𝟐 , … . . , 𝐗 𝐧 be a random sample from a Bernoulli distribution
with parameter 𝐩; 𝟎 ≤ 𝐩 ≤ 𝟏. Check whether the estimator
p is consistent or not.
for estimating
[JAM 2011]
Sol: - Since Xi ~ Ber(p);
i = 1, 2, … . , n
Therefore, ∑ Xi ~Bin(n, p);
n
Now,
Xi
√n+2 ∑
E ( 2(n+ i=1
)
√n)
Also,
V(
n
𝐧
√𝐧+𝟐 ∑𝐢=𝟏 𝐗 𝐢
𝟐(𝐧+√𝐧)
√n+2 ∑i=1 Xi
)
2(n+√n)
=
=
⟹ E(∑ Xi ) = np and V(∑ Xi ) = npq
√n+2np
2(n+√n)
=
4
4(n+√n)
2
1
+2p
√n
1
2(1+ )
√n
⟶ 𝑝 as 𝑛 ⟶ ∞
V(∑ni=1 Xi ) =
npq
(n+√n)
2
=(
pq
√n+1)
⟶ 0 as 𝑛 ⟶ ∞
𝐧
𝐗𝐢
√𝐧+𝟐 ∑
Hence, 𝜃̂𝑛 = 𝟐(𝐧+ 𝐢=𝟏
is a consistent estimator of p.
𝐧)
√
Example 5.3:- Let X1 , … . , Xn be a random sample from 𝑁(𝜇, 𝜎 2 ). Show that 𝑆 2 =
1 n
∑i=1(𝑋𝑖 − 𝑋̅)2 is a consistent estimator of 𝜎2.
𝑛
Sol: - We know that
𝑛𝑆 2
𝜎2
=
̅ 2
∑n
i=1(𝑋𝑖 −𝑋 )
𝜎2
2
~𝜒𝑛−1
𝑛𝑆 2
𝑛−1
Therefore, 𝐸 ( 𝜎2 ) = 𝑛 − 1 ⟹ 𝐸(𝑆 2 ) = (
𝑛𝑆 2
𝑛
𝑛−1
1
) 𝜎 2 = (1 − 𝑛) 𝜎 2 ⟶ 𝜎 2 as 𝑛 ⟶ ∞
1
1
And 𝑉 ( 𝜎2 ) = 2(𝑛 − 1) ⟹ 𝑉(𝑆 2 ) = 2 ( 𝑛2 ) 𝜎 4 = 2𝜎 4 (𝑛 − 𝑛2 ) ⟶ 0 as 𝑛 ⟶ ∞
Hence, S2 is a consistent estimator of 𝜎2.
Example 5.3:- Let X1 , … . , Xn be a random sample from 𝑁(𝜇, 𝜇), 𝜇 > 0. Find a
consistent estimator of µ2.
Sol: Let 𝑇 = 𝑋̅𝑆 2 , then 𝐸(𝑇) = 𝐸(𝑋̅𝑆 2 ) = 𝐸(𝑋̅)𝐸(𝑆 2 ) = 𝜇. 𝜇 = 𝜇 2
Now, 𝐸(𝑇 2 ) = 𝐸[(𝑋̅𝑆 2 )2 ] = 𝐸(𝑋̅ 2 )𝐸(𝑆 2 )2
= {𝑉(𝑋̅) + 𝐸 2 (𝑋̅)}{𝑉(𝑆 2 ) + 𝐸 2 (𝑆 2 )}
2𝜇 2
𝜇
2
2𝜇
1
= {𝑛 + 𝜇 2 } {𝑛−1 + 𝜇 2 } = 𝜇 3 {𝑛(𝑛−1) + (𝑛−1) + 𝑛 + µ}
Therefore,
2
2𝜇
1
𝑉(𝑇) = 𝐸(𝑇 2 ) − 𝐸 2 (𝑇) = 𝜇 3 {𝑛(𝑛−1) + (𝑛−1) + 𝑛 + µ} − 𝜇 4
2
2𝜇
1
= 𝜇 3 {𝑛(𝑛−1) + (𝑛−1) + 𝑛}
⟶ 0 as 𝑛 ⟶ ∞
Hence, 𝑇 = 𝑋̅𝑆 2 is a consistent estimator of µ2.
Example 5.3:- Let X1 , … . , Xn be a random sample from 𝑈[0, 𝜃]. Examine the
consistency of the estimators 𝑇1 = 𝑋(𝑛) , 𝑇2 = 𝑋(1) + 𝑋(𝑛) , 𝑇3 = (𝑛 + 1)𝑋(1) and 𝑇4 =
2𝑋̅ for estimating 𝜃.
[ISS 2012]
Sol: To check this, first we need to find the mean and variance of each Ti’s.
Given that 𝑋1 , 𝑋2 , … . . , 𝑋𝑛 ~𝑈(0, 𝜃).
Transform Yi =
Xi
Therefore,
𝐸[𝑌(𝑗) ] = 𝑛+1 and 𝑉[𝑌(𝑗) ] = (𝑛+1)2 (𝑛+2)
(1)
θ
then Y1, Y2, Y3………… Yn ~U(0,1) and Y(j) ~Beta(j, n − j + 1)
𝑗
𝑗
𝑛𝜃
𝜃
𝐸(𝑇1 ) = 𝐸[𝑋(𝑛) ] = 𝐸[𝜃𝑌(𝑛) ] = 𝑛+1 = 1+1/𝑛 ⟶ 𝜃 as 𝑛 ⟶ ∞
𝑛𝜃2
𝜃2
𝑉(𝑇1 ) = 𝑉[𝑋(𝑛) ] = 𝑉[𝜃𝑌(𝑛) ] = (𝑛+1)2 (𝑛+2) = (𝑛+1)2 (1+2/𝑛) ⟶ 0 as 𝑛 ⟶ ∞
Hence, 𝑇1 is consistent.
(2)
𝜃
𝑛𝜃
𝐸(𝑇2 ) = 𝐸[𝑋(1) + 𝑋(𝑛) ] = 𝐸[𝜃𝑌(1) + 𝜃𝑌(𝑛) ] = 𝑛+1 + 𝑛+1 = 𝜃
𝑛𝜃2
𝜃2
𝑉(𝑇2 ) = 𝑉[𝑋(1) + 𝑋(𝑛) ] = 𝑉[𝜃𝑌(1) + 𝜃𝑌(𝑛) ] = (𝑛+1)(𝑛+2) = (1+1/𝑛)(𝑛+2) ⟶ 0 as 𝑛 ⟶
∞
Hence, 𝑇1 is consistent.
(3)
𝐸(𝑇3 ) = 𝐸[(𝑛 + 1)𝑋(1) ] = (𝑛 + 1)𝐸[𝜃𝑌(1) ] =
(𝑛+1)𝜃
𝑛+1
=𝜃
(𝑛+1)2 𝜃2
𝜃2
𝑉(𝑇3 ) = 𝑉[(𝑛 + 1)𝑋(1) ] = (𝑛 + 1)2 𝑉[𝜃𝑌(1) ] = (𝑛+1)2 (𝑛+2) = (𝑛+2) ⟶ 0 as 𝑛 ⟶ ∞
Hence, 𝑇3 is consistent.
(4)
𝜃
𝐸(𝑇4 ) = 𝐸[2𝑋̅] = 2𝐸[𝑋̅] = 2 2 = 𝜃
2
2
4𝜃
𝜃
𝑉(𝑇4 ) = 𝑉[2𝑋̅] = 4𝑉[𝑋̅] = 12𝑛 = 3𝑛 ⟶ 0 as 𝑛 ⟶ ∞
Hence, 𝑇4 is consistent.
Example 5.4:- Let X1 , … . , Xn (n > 4) be a random sample from a population with
mean µ and variance 𝜎2. Consider the following estimators of µ
n
1
1
3
1
(𝑋2 +. . . +𝑋𝑛−1 ) + 𝑋𝑛
𝑈 = ∑ 𝑋𝑖 , W = 𝑋1 +
𝑛
8
4(n − 2)
8
i=1
(1) Examine whether the estimators U and W are unbiased
(2) Examine whether the estimators U and W are consistent
(3) Which of these two estimators is more efficient?
Sol: - We are given that 𝐸(𝑋𝑖 ) = 𝜇 and 𝑉(𝑋𝑖 ) = 𝜎 2
1
(1) 𝐸(𝑈) = 𝐸 (𝑛 ∑ni=1 𝑋𝑖 ) = 𝜇
1
[JAM 2012]
∀ 𝑖 = 1,2, …
and
3
1
1
3
1
𝐸(W) = E [8 𝑋1 + 4(n−2) (𝑋2 +. . . +𝑋𝑛−1 ) + 8 𝑋𝑛 ] = 8 µ + 4(n−2) (n − 2)µ + 8 µ = µ
Hence, U and W both are unbiased estimator of µ
1
(2) Also, 𝑉(𝑈) = 𝑉 (𝑛 ∑ni=1 𝑋𝑖 ) =
1
𝜎2
𝑛
⟶ 0 as 𝑛 ⟶ ∞
3
1
and 𝑉(W) = V [8 𝑋1 + 4(n−2) (𝑋2 +. . . +𝑋𝑛−1 ) + 8 𝑋𝑛 ]
1
9
1
= 64 𝜎 2 + 16(n−2)2 (n − 2)𝜎 2 + 64 𝜎 2
9
1
= 16(n−2) 𝜎 2 + 32 𝜎 2
18+n−2
= 32(n−2) 𝜎 2
16+n
= 32(n−2) 𝜎 2
↛ 0 as 𝑛 ⟶ ∞
Hence, U is a consistent estimator but W is not.
(3) Suppose, 𝑉(𝑈) < 𝑉(𝑊) ⟹
𝜎2
𝑛
16+n
< 32(n−2) 𝜎 2 ⟹ 𝑛2 + 16𝑛 > 32𝑛 − 64
⟹ 𝑛2 − 16𝑛 + 64 > 0 ⟹ (n − 8)2 > 0 which is true.
So U is more efficient than W because 𝑉(𝑈) < 𝑉(𝑊).
Exercise
1.1:- Let 𝑋1 , 𝑋2 , … . . , 𝑋𝑛 are a random sample from geometric distribution
𝑃(𝑋 = 𝑥) = 𝑝𝑞 𝑥 , 𝑥 = 0,1,2 … ….
Find the unbiased estimator of 𝑝 based on 𝑇 = ∑𝑛𝑖=1 𝑋𝑖 .
1.2:- Let 𝑋1 , 𝑋2 , … . . , 𝑋𝑛 are a random sample from geometric distribution
𝑃(𝑋 = 𝑥) = 𝑝𝑞 𝑥−1 , 𝑥 = 1,2 … ….
𝟏
Find the unbiased estimator of 𝑝 based on 𝑇 = ∑𝑛𝑖=1 𝑋𝑖 and find its distribution.
1.2:- Let 𝑋1 , 𝑋2 , … . . , 𝑋𝑛 be iid 𝐵𝑒𝑟(𝑝). Then show that
𝑇(𝑛−𝑇)
𝑛(𝑛−1)
is an unbiased
estimator of 𝑝(1 − 𝑝) and has smaller variance than the estimator
∑𝑛𝑖=1 𝑋𝑖 .
𝟏 𝐧−𝟏
1.3:- If 𝑋1 , 𝑋2 , … . . , 𝑋𝑛 ~𝐸𝑥𝑝(𝜆). Then show that [𝟏 − 𝐧𝐗̅]
̅ > 1 and zero when 𝐧𝐗
̅ < 1.
when 𝐧𝐗
𝑇
𝑇
(1 − 𝑛), where 𝑇 =
𝑛
is used to estimate 𝐞−𝛉
1.4:- Let 𝑋1 , 𝑋2 , … . . , 𝑋𝑛 are a random sample from the population with probability
density function,
𝑓(𝑥, 𝜃) = (𝜃 + 1)𝑥 𝜃 ; 0 < 𝑥 < 1; 𝜃 > −1
−(n−1)
Show that [ ∑ logx − 1] is an unbiased estimator of 𝛉.
i
1.5:- Let 𝑋1 , 𝑋2 , … . . , 𝑋𝑛 be a random sample from Laplace distribution,
1
𝑓(𝑥, 𝜃) = 2 𝑒 −|𝑥−𝜃| ;
−∞ < 𝑥, 𝜃 < ∞;
Let 𝑋(1) ≤. . . . ≤ 𝑋(𝑛) be the order statistic. Show that 𝑻 =
𝑋(1) +𝑋(𝑛)
2
is unbiased for 𝛉.
1.6:- Let 𝑿𝟏 , 𝑿𝟐 , … . . , 𝑿𝒏 be a random sample from 𝑵(𝝁, 𝝈𝟐 ), with known mean μ.
Find a constant C such that 𝐶𝑇𝑐 is an unbiased estimator of σ, where 𝑇𝑐 =
𝟏 𝐧
∑ |𝑿𝒊 − µ|.
𝐧 𝐢=𝟏
2
Hints: 𝐸(𝑇𝑐 ) = mean deviation about mean = √𝜋 𝜎
1.7 Let 𝑿𝟏 , 𝑿𝟐 , 𝑿𝟑 be a random sample of size 3 from a population with mean μ and
variance 𝜎2. Let 𝑻𝟏 , 𝑻𝟐 and 𝑻𝟑 are the estimators used to estimate the mean µ,
𝟏
where 𝑻𝟏 = 𝑿𝟏 + 𝑿𝟐 − 𝑿𝟑
𝑻𝟐 = 𝟐𝑿𝟏 + 𝟑𝑿𝟐 − 𝟒𝑿𝟑 𝑻𝟑 = 𝟑 (𝝀𝑿𝟏 + 𝑿𝟐 + 𝑿𝟑 )
(a) Are 𝑻𝟏 and 𝑻𝟐 unbiased estimates?
(b) Find the value of 𝜆 such that 𝑻𝟑 is an unbiased estimator for µ.
(c) Which is the best estimator?
[MHPCS 2007]
1.8 Suppose that 𝑿𝟏 , 𝑿𝟐 , … . . , 𝑿𝒏 are a random sample from the uniform distribution
̅ is an unbiased estimator of θ.
on (θ − 1, θ + 1). Show that the sample mean 𝑿
Let 𝑿(𝟏) and 𝑿(𝒏) be the smallest and largest order statistics derived from
𝑿𝟏 , … . . , 𝑿𝒏 . Show also that random variable 𝑴 = (𝑿(𝟏) + 𝑿(𝒏) )/𝟐 is an unbiased
estimator of θ.
Hints: Let 𝑊 =
(𝑋−𝜃+1)
2
~𝑈(0,1). Then 𝑊(𝑖) ~𝐵𝑒𝑡𝑎(𝑖, 𝑛 − 𝑖 + 1) and 2𝑊(𝑖) + 𝜃 − 1 =
𝑋(𝑖) ⟹ 𝑋(1) + 𝑋(𝑛) = 2(𝑊(1) + 𝑊(𝑛) ) + 2𝜃 − 2 and 𝐸[𝑊(𝑖) ] =
𝑖
𝑛+1
1.9:- Let 𝑿𝟏 , 𝑿𝟐 , … . . , 𝑿𝒏 be a random sample from a 𝑵(𝝁, 𝝈𝟐 ) distribution, where
̅ )𝟐 will
both µ and 𝝈𝟐 are unknown. Then for what value of 𝛼, 𝑻𝜶 = 𝛂 ∑𝐧𝐢=𝟏(𝑿𝒊 − 𝑿
have minimum mean squared error for estimating 𝝈𝟐 ,
1.10:- Let 𝑋1, 𝑋2 , 𝑋3 be iid 𝑵(0, 𝜃 2 ) random variables, 𝜃>0. Then find the value of k
such that the estimator 𝒌 ∑𝟑𝒋=𝟏 |𝑿𝒋 | is an unbiased estimator of 𝜃.
[JAM 2011]
𝜋
2
Hints: Ans is √18. Use the fact that MD about mean for 𝑁(µ, 𝜎 2 ) is √𝜋 𝜎.
1.11:- Let the random variable 𝑿~𝑼(5,5 + 𝜃). Based on a random sample of size 1,
say 𝑋1, find an unbiased estimator of 𝜽𝟐 is
[JAM 2012]
Hints: Let Y=X-5, then 𝑌~𝑈(0, 𝜃). Compute 𝐸(𝑌 2 ) =
𝜃2
3
. Now adjust.
Jam 2005
2.1:- Let 𝑋1, 𝑋2 ~𝐵𝑒𝑟(𝑝). Show that 𝑇 = 𝑋1, +2𝑋2 is not sufficient for p.
2.2:- Let 𝑋1, 𝑋2 , 𝑋3 ~𝐵𝑒𝑟(𝑝). Examine whether the following are sufficient statistics
for p or not?
1
(a) 𝑇 = 6 (𝑋1, +2𝑋2 + 3𝑋3 )
(b) 𝑇 = 𝑋1 𝑋2 + 𝑋3
(c) 𝑇 = 𝑋1 + 2𝑋2 + 𝑋3
(d) 𝑇 = 2𝑋1 + 3𝑋2 + 4𝑋3
Hints: (a) No (b) No
2.3:- Let 𝑋1 , … , 𝑋4 ~𝐵𝑒𝑟(𝑝). Show that 𝑇 = 𝑋1 (𝑋3 + 𝑋4 ) + 𝑋2 is not sufficient for p.
2.4:- If 𝑋1, 𝑋2 ~𝑁(µ, 1). Then show that 𝑇 = 𝑋1 +𝑋2 is sufficient for µ.
Hints: Find the conditional distribution of (𝑋1, 𝑋2 )|𝑇.
2.5:- Let 𝑋1 , 𝑋2 , … . . , 𝑋𝑚 are iid from 𝐵𝑒𝑟(𝑝), 𝑌1 , 𝑌2 , … . . , 𝑌𝑛 are iid from 𝑩𝒆𝒓(𝒒), and
𝑿′𝒔 and independent of 𝒀′𝒔 where 𝟎 < 𝒑 < 1 is unknown parameter with 𝒒 = 𝟏 − 𝒑.
𝒏
Using the conditional distribution approach show that, 𝑻 = ∑𝒎
𝒊=𝟏 𝑿𝒊 − ∑𝒋=𝟏 𝒀𝒋 is
sufficient for p.
Hints: Write T as, 𝑇 = 𝑋1 + ⋯ +𝑋𝑚 + (1 − 𝑌1 ) + ⋯ + (1 − 𝑌𝑛 ) − 𝑛 and then show that
P(T = t) = (m+n
)pn+t (1 − p)m−t . Now use conditional distribution approach to
n+t
prove this.
2.6:- Suppose 𝐗 𝟏 , 𝐗 𝟐 are iid 𝐍(µ, 𝟏) , − ∞ < 𝜇 < ∞. Define 𝑻𝟏 = 𝟐𝐗 𝟏 + 𝐗 𝟐 and
𝐓𝟐 = 𝐗 𝟏 + 𝟐𝐗 𝟐 . Find the conditional distribution of 𝑻𝟏 |𝑻𝟐 . Hence comment on 𝑻𝟐 .
4
9
Hints: 𝑇1 |𝑇2 ~𝑁 (3𝜇 + (𝑇2 − 3𝜇), )
5
5
2.7:-If T is sufficient for 𝜃, Show that T is also sufficient for 𝜸(𝛉).
2.8:- Suppose that X is 𝑵(𝟎, 𝝈𝟐 ) where 0 < 𝜎 < ∞ is the unknown parameter. By
means of the conditional distribution approach, show that | X | is sufficient for 𝜎2.
Hints: Write the pdf in terms of |X| and use factorization theorem.
2.9:- Let 𝑿𝟏 , 𝑿𝟐 , … . . , 𝑿𝒎 are iid 𝑷𝒐𝒊(𝝀), 𝒀𝟏 , 𝒀𝟐 , … . . , 𝒀𝒏 are iid 𝑷𝒐𝒊(𝟐𝝀), and 𝑿′𝒔 are
independent of 𝒀′𝒔 where 𝝀(> 0) is unknown. Using the conditional distribution
𝒏
approach show that, 𝑻 = ∑𝒎
𝒊=𝟏 𝑿𝒊 + ∑𝒋=𝟏 𝒀𝒋 is sufficient for 𝜆.
Hints: 𝑇~𝑷𝒐𝒊(𝒎𝝀 + 𝟐𝒏𝝀). Use conditional distribution approach to prove this.
2.10:- Let X1 , X2 , … … , Xn be independent random samples with densities
Prove that 𝐓 = 𝐦𝐢𝐧𝐢 (𝐗 𝐢 /𝐢) is a sufficient statistics for 𝜃.
Hints: Use factorization theorem to prove this.
2.11:- Suppose that 𝐗 𝟏 , 𝐗 𝟐 , … … , 𝐗 𝐧 are iid with 𝑼[𝐤𝛉, (𝐤 + 𝟏)𝛉] where k is an
integer and 𝛉(> 0) is the unknown. Find the sufficient statistics for 𝜃.
Hints: Use factorization theorem to prove 𝑇 = (X(1) , X(n) ) is sufficient statistics.
2.12:- Suppose that 𝐗 𝟏 , 𝐗 𝟐 , … … , 𝐗 𝐧 are iid with 𝑼[−𝐢(𝛉 − 𝟏), 𝐢(𝛉 + 𝟏)] where i is an
integer and 𝛉(> 0) is the unknown. Show that 𝑇 = (X(1) /i, X(n) /i) is sufficient
statistics for 𝜃.
Hints: Use factorization theorem to prove this.
2.13:- Suppose that 𝐗 𝟏 , 𝐗 𝟐 , … … , 𝐗 𝐧 be a random sample from the distribution
having the density function
1
𝑓(x; θ) = 2 e−|x−θ| ;
−∞ < 𝑥 < ∞,
Show that order statistics, 𝐗 (𝟏) , 𝐗 (𝟐) , … … , 𝐗 (𝐧) are the only sufficient statistics for 𝜃.
Hints: Use factorization theorem to prove this.
2.14:- Let 𝐗 𝟏 , 𝐗 𝟐 , … … , 𝐗 𝐧 be the random samples from the distribution having pmf
𝑃(X = x) = (1 − p)px−θ ;
0 < 𝑝 < 1,
𝑥 = 𝜃, 𝜃 + 1, … …,
Find the joint sufficient statistics for (𝜃, p).
Hints: Use factorization theorem to prove 𝑇 = (X(1) , ∑ni=1 Xi ) is sufficient.
2.15:- Let 𝐗 𝟏 , 𝐗 𝟐 , … … , 𝐗 𝐧 be the random samples from logNormal(µ, 𝜎2). Show that
𝑻 = (∑ni=1 𝑙𝑜𝑔2 Xi , ∑ni=1 𝑙𝑜𝑔Xi ) is the joint sufficient statistics for 𝜃=(µ, 𝜎2).
Hints: Use factorization theorem to prove this.
2.16:- Let 𝐗 𝟏 and 𝐗 𝟐 be independent discrete uniform random variable on
{𝟏, 𝟐, … . . , 𝐍} where N is an unknown integer. Obtain the conditional distribution of
(𝐗 𝟏 , 𝐗 𝟐 ) given 𝐓 = 𝐦𝐚𝐱(𝐗 𝟏 , 𝐗 𝟐 ). Hence show that T is sufficient statistics for N but
not 𝐗 𝟏 + 𝐗 𝟐 .
Hints: First show that
2t−1
P(max(X1 , X2 ) = t) = P(X1 = t, X2 < 𝑡) + P(X1 < 𝑡, X2 = t) + P(X1 = t, X 2 = t) = N2 .
Then show that conditional distribution of (X1 , X2 ) given T is independent of N.
2.17:- Let 𝐗 𝟏 , 𝐗 𝟐 , … … , 𝐗 𝐦 are iid Gamma(𝛼, 𝛽), 𝒀𝟏 , 𝒀𝟐 , … . . , 𝒀𝒏 are iid Gamma(𝛼, k𝛽),
and 𝑿′𝒔 are independent of 𝒀′𝒔 where 𝟎 < 𝛼, 𝛽 < ∞ and 𝛽 is only unknown
parameter. Assume that k (>0) is known. Then what is sufficient for 𝛽.
Hints: In the joint likelihood of X and Y, apply factorization theorem.
2.18:- Show that both the geometric mean 𝐧√𝐗 𝟏 , 𝐗 𝟐 , … … , 𝐗 𝐧 of the observations and
the arithmetic average of the logarithms of the observations are sufficient
statistics for 𝛽 when sampling from the Gamma(𝛼, 𝛽) distribution with known 𝛼.
1
Hints: 𝑛 ∑ni=1 𝑙𝑜𝑔Xi = log[(∏ni=1 Xi )1/n ] and log(.) is a 1-1 function.
2.19:- Let 𝐗 𝟏 , 𝐗 𝟐 , … … , 𝐗 𝐧 be iid with common density function is given by
1
𝑓(x; θ) = 2θ e−|x|/θ ;
−∞ < 𝑥 < ∞,
1
1
Define, 𝑺 = n ∑ni=1 Xi and 𝑇 = n ∑ni=1 |Xi |. Show that in each case, if it is unbiased
estimator and sufficient statistics for 𝜃.
Hints: T is both unbiased estimator and sufficient whereas S is not.
1
2.20:- Suppose 𝐗 𝟏 has density 𝑓1 (x) = θ e−x/θ , x > 0 and 𝐗 𝟐 has density 𝑓2 (x) =
2 −2x/θ
θ
e
, x > 0 and 𝐗 𝟏 , 𝐗 𝟐 are independent. Then find a sufficient statistics for 𝜃.
2.21:- Suppose 𝐗 𝟏 and 𝐗 𝟐 are iid random variables each following exponential
distribution with mean �. Then which of the following is true?
Conditional distribution of 𝐗 𝟐 | 𝐗 𝟏 + 𝐗 𝟐 = 𝐭 is:
𝐭
(a) Exponential with mean 𝟐 and hence 𝐗 𝟏 + 𝐗 𝟐 is sufficient for 𝜃.
(b) Exponential with mean
𝐭𝛉
𝟐
and hence 𝐗 𝟏 + 𝐗 𝟐 is not sufficient for 𝜃.
(c) Uniform(0,t) and hence 𝐗 𝟏 + 𝐗 𝟐 is sufficient for 𝜃.
(d) Uniform(0,t𝜃) and hence 𝐗 𝟏 + 𝐗 𝟐 is not sufficient for 𝜃.
4.1:- Let X be a random variable having probability mass function
2θ
if x = −1
𝑝(𝑥) = {θ2
if x = 0
2
1 − 2θ − θ if x = 1
where 𝛉 ∈ [𝟎, √𝟐 − 𝟏]. Show that there is only one, and only one, unbiased
estimator of (𝛉 + 𝟏)𝟐 based on single observation.
[JAM 2009]
Hints:
4.2:- Suppose that X1 , X2 , … … , Xn are iid with 𝑼(𝛉 − 𝟏, 𝛉 + 𝟏), where 𝛉(> 0) is the
unknown parameter. Show that 𝑻 = (𝐗 (𝟏) , 𝐗 (𝐧) ) is jointly sufficient for 𝜃 but T is not
complete.
2𝑛
Hints: Use factorization theorem to prove sufficiency. 𝐸(X(1) ) = (1 + 𝜃) − 𝑛+1 and
2𝑛
𝐸(X(n) ) = 𝑛+1 − (1 − 𝜃)
⟹
𝐸(X(n) − X(1) ) =
2(𝑛−1)
𝑛+1
exactly.
2(𝑛−1)
𝑛+1
⟹ 𝐸 (X(n) − X(1) −
2(𝑛−1)
𝑛+1
) = 0 where as X(n) − X(1) ≠
4.3:- Let the pdf of X be
𝑓(x; θ) = Q(θ)M(x);
0 < 𝑥 < θ, 0 < θ < ∞,
M(x)(> 0) being a continuous function of x. Show that 𝑻 = 𝑿(𝒏) is complete
sufficient statistics for 𝜃.
Hints: Use the same methodology as we applied for U(0, 𝜃).
4.4:- Check whether the following family is complete or not.
(a) 𝑃(X = 0) = p, 𝑃(X = 1) = 3p,
𝑃(X = 2) = 1 − 4p;
2
(b) 𝑃(X = 0) = p, 𝑃(X = 1) = p ,
𝑃(X = 2) = 1 − p − p2 ;
p
p
(c) 𝑃(X = −1) = 2 , 𝑃(X = 0) = 1 − p, 𝑃(X = 1) = 2 ;
|x|
θ
(d) 𝑓(X; θ) = (1 − θ) {2(1−θ)}
;
0 < 𝑝 < 0.25
0<𝑝<𝑐
0<𝑝<1
−∞ < 𝑥 < ∞
(e) 𝐗 𝟏 , 𝐗 𝟐 , … … , 𝐗 𝐧 are iid 𝑵(θ, 1), θ = {1,2}
1
1
(f) 𝑋~𝐵𝑖𝑛(2, θ), θ = {2 , 4}
1
(g) 𝑋~𝐵𝑖𝑛(2, θ), θ = {2 ,
1
3
1
, 4}
(h) 𝑋~𝑈(θ, θ + 1) , − ∞ < 𝜃 < ∞
(i) 𝐗 𝟏 , 𝐗 𝟐 , … … , 𝐗 𝐧 are iid 𝑵(θ, θ2 )
(j) 𝐗 𝟏 , 𝐗 𝟐 , … … , 𝐗 𝐧 are iid 𝑵(θ, θ2 )
Hints: (a) (c) and (d) (g) are complete, (b), (e), (f), (h) (i) is not.
5.1:- Let that X1 , X2 , … … , Xn be iid random variables with probability density
function
1
𝑓(x) = 2 λ3 x 2 e−λx ;
𝑥 > 0, 𝜆 > 0
Then which of the following are true?
𝟐
𝟏
(a) 𝒏 ∑𝐧𝐢=𝟏 𝐗 is an unbiased estimator of 𝜆.
(b)
(c)
(d)
𝟑𝐧
𝐢
is an unbiased estimator of 𝜆.
∑𝐧
𝐢=𝟏 𝐗 𝐢
𝟐 𝐧 𝟏
∑
𝒏 𝐢=𝟏 𝐗 𝐢
𝟑𝐧
is
∑𝐧
𝐢=𝟏 𝐗 𝐢
is a consistent estimator of 𝜆.
a consistent estimator of 𝜆
5.2:- Let that X1 , X2 , … … , Xn (𝑛 ≥ 3) be a random sample from U(𝜃-5, 𝜃-3) random
variables. Let X(1) and X(n) denote the smallest and largest of the sample values.
Then which of the following are always true?
(a) (X(1) , X(n) ) is complete sufficient for 𝜃.
(b) X1 + X2 − 2X3 is an ancillary statistics.
(c) X(n) + 3 is unbiased for 𝜃.
(d) X(1) + 5 is consistent for 𝜃.
5.3:- If X1 , X2 , … … , Xn are iid from N(0,1). Specify which of the following are
unbiased, consistent and sufficient.
(a) 𝑻𝟏 =
X1 +2X2
3
(b) 𝑻𝟐 =
∑n
i=1 Xi
n
(c) 𝑻𝟑 =
∑n
i=1 Xi
n+1
[ISS 2010]