STATISTICAL INFERENCE PART II POINT ESTIMATION 1 SUFFICIENT STATISTICS • X, f(x;), • X1, X2,…,Xn be a sample rvs • Y=U(X1, X2,…,Xn ) is a statistic. • A sufficient statistic, Y is a statistic which contains all the information for the estimation of . 2 SUFFICIENT STATISTICS • Given the value of Y, the sample contains no further information for the estimation of . • Y is a sufficient statistic (ss) for if the conditional distribution h(x1,x2,…,xn|y) does not depend on for every given Y=y. • A ss for is not unique. • If Y is a ss for , then a 1-1 transformation of Y, say Y1=fn(Y) is also a ss for . 3 SUFFICIENT STATISTICS • The conditional distribution of sample rvs given the value of y of Y, is defined as f x1 , x 2 , , x n , y ; h x1 , x 2 , , x n y g y ; h x1 , x 2 , , xn y L ; x1 , x 2 , g y ; • If Y is a ss for , then h x1 , x 2 , , xn y ss for Not depend on for every given y. L ; x1 , x 2 , g y ; , xn , xn H x1 , x 2 , , xn may include y or constant. 4 • Also, the conditional range of Xi given y not depend on . SUFFICIENT STATISTICS EXAMPLE: X~Ber(p). For a r.s. of size n, show that X is a ss for p. n i i 1 5 SUFFICIENT STATISTICS • Neyman’s Factorization Theorem: Y is a ss for iff L k 1 y ; k 2 x1 , x 2 , The likelihood function , xn Does not contain any other xi Not depend on for every given y (also in the conditional range of xi.) where k1 and k2 are non-negative functions and k2 does not depend on or y. 6 EXAMPLES 1. X~Ber(p). For a r.s. of size n, find a ss for p if exists. 7 EXAMPLES 2. X~Beta(θ,2). For a r.s. of size n, find a ss for θ. 8 SUFFICIENT STATISTICS • A ss may not exist. • Jointly ss Y1,Y2,…,Yk may be needed. Example: Example 10.2.5 in Bain and Engelhardt (page 342 in 2nd edition), X(1) and X(n) are jointly ss for • If the MLE of exists and unique and if a ss for exists, then MLE is a function of a ss for . 9 EXAMPLE X~N(,2). For a r.s. of size n, find jss for and 2. 10 MINIMAL SUFFICIENT STATISTICS • If S ( x ) ( s1 ( x ),..., s k ( x )) is a ss for θ, then, ~ ~ ~ * S ( x ) ( s 0 ( x ), s1 ( x ),..., s k ( x )) ~ ~ ~ is also a SS ~ for θ. But, the first one does a better job in data reduction. A minimal ss achieves the greatest possible reduction. 11 MINIMAL SUFFICIENT STATISTICS • A ss T(X) is called minimal ss if, for any other ss T’(X), T(x) is a function of T’(x). • THEOREM: Let f(x;) be the pmf or pdf of a sample X1, X2,…,Xn. Suppose there exist a function T(x) such that, for two sample points x1,x2,…,xn and y1,y2,…,yn, the ratio f x1 , x 2 , , x n ; f y1 , y 2 , , y n ; is constant as a function of iff T(x)=T(y). Then, T(X) is a minimal sufficient statistic for . 12 EXAMPLE • X~N(,2) where 2 is known. For a r.s. of size n, find minimal ss for . Note: A minimal ss is also not unique. Any 1-to-1 function is also a minimal ss. 13 RAO-BLACKWELL THEOREM • Let X1, X2,…,Xn have joint pdf or pmf f(x1,x2,…,xn;) and let S=(S1,S2,…,Sk) be a vector of jss for . If T is an UE of () and (T)=E(TS), then i) (T) is an UE of () . ii) (T) is a fn of S, so it is also jss for . iii) Var((T) ) Var(T) for all . • (T) is a uniformly better unbiased estimator of () . 14 RAO-BLACKWELL THEOREM • Notes: • (T)=E(TS) is at least as good as T. • For finding the best UE, it is enough to consider UEs that are functions of a ss, because all such estimators are at least as good as the rest of the UEs. 15 Example • Hogg & Craig, Exercise 10.10 • X1,X2~Exp(θ) • Find joint p.d.f. of ss Y1=X1+X2 for θ and Y2=X2. • Show that Y2 is UE of θ with variance θ². • Find φ(y1)=E(Y2|Y1) and variance of φ(Y1). 16 ANCILLARY STATISTIC • A statistic S(X) whose distribution does not depend on the parameter is called an ancillary statistic. • An ancillary statistic contains no information about . 17 Example • Example 6.1.8 in Casella & Berger, page 257: Let Xi~Unif(θ,θ+1) for i=1,2,…,n Then, range R=X(n)-X(1) is an ancillary statistic because its pdf does not depend on θ. 18 COMPLETENESS • Let {f(x; ), } be a family of pdfs (or pmfs) and U(x) be an arbitrary function of x not depending on . If E U X 0 for all requires that the function itself equal to 0 for all possible values of x; then we say that this family is a complete family of pdfs (or pmfs). E U X 0 for all U x 0 for all x. 19 EXAMPLES 1. Show that the family {Bin(n=2,); 0<<1} is complete. 20 EXAMPLES 2. X~Uniform(,). Show that the family {f(x;), >0} is not complete. 21 BASU THEOREM • If T(X) is a complete and minimal sufficient statistic, then T(X) is independent of every ancillary statistic. • Example: X~N(,2). X : th e m ss fo r 2 X ~ N ( , / n ) X is a (n-1)S2/ and complete 2 ~ family of 2 N ( , / n ) is complete family . statistic 2 n 1 S2 Ancillary statistic for By Basu theorem, X and S2 are independent. 22 COMPLETE AND SUFFICIENT STATISTICS (css) • Y is a complete and sufficient statistic (css) for if Y is a ss for and the family g y ; ; is complete. The pdf of Y. 1) Y is a ss for . 2) u(Y) is an arbitrary function of Y. E(u(Y))=0 for all implies that u(y)=0 23 for all possible Y=y. THE MINIMUM VARIANCE UNBIASED ESTIMATOR • Rao-Blackwell Theorem: If T is an unbiased estimator of , and S is a ss for , then (T)=E(TS) is – an UE of , i.e.,E[(T)]=E[E(TS)]= and – the MVUE of . 24 LEHMANN-SCHEFFE THEOREM • Let Y be a css for . If there is a function Y which is an UE of , then the function is the unique Minimum Variance Unbiased Estimator (UMVUE) of . • Y css for . • T(y)=fn(y) and E[T(Y)]=. T(Y) is the UMVUE of . So, it is the best estimator of . 25 THE MINIMUM VARIANCE UNBIASED ESTIMATOR • Let Y be a css for . Since Y is complete, there could be only a unique function of Y which is an UE of . • Let U1(Y) and U2(Y) be two function of Y. Since they are UE’s, E(U1(Y)U2(Y))=0 imply W(Y)=U1(Y)U2(Y)=0 for all possible values of Y. Therefore, U1(Y)=U2(Y) for all Y. 26 Example • Let X1,X2,…,Xn ~Poi(μ). Find UMVUE of μ. • Solution steps: n – Show that S X i is css for μ. i 1 – Find a statistics (such as S*) that is UE of μ and a function of S. – Then, S* is UMVUE of μ by Lehmann-Scheffe Thm. 27 Note • The estimator found by Rao-Blackwell Thm may not be unique. But, the estimator found by Lehmann-Scheffe Thm is unique. 28