advertisement

Chapter 3: The Multiple Linear Regression Model Advanced Econometrics - HEC Lausanne Christophe Hurlin University of Orléans November 23, 2013 Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 1 / 174 Section 1 Introduction Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 2 / 174 1. Introduction The objectives of this chapter are the following: 1 De…ne the multiple linear regression model. 2 Introduce the ordinary least squares (OLS) estimator. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 3 / 174 1. Introduction The outline of this chapter is the following: Section 2: The multiple linear regression model Section 3: The ordinary least squares estimator Section 4: Statistical properties of the OLS estimator Subsection 4.1: Finite sample properties Subsection 4:2: Asymptotic properties Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 4 / 174 1. Introduction References Amemiya T. (1985), Advanced Econometrics. Harvard University Press. Greene W. (2007), Econometric Analysis, sixth edition, Pearson - Prentice Hil (recommended) Pelgrin, F. (2010), Lecture notes Advanced Econometrics, HEC Lausanne (a special thank) Ruud P., (2000) An introduction to Classical Econometric Theory, Oxford University Press. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 5 / 174 1. Introduction Notations: In this chapter, I will (try to...) follow some conventions of notation. fY ( y ) probability density or mass function FY ( y ) cumulative distribution function Pr () probability y vector Y matrix Be careful: in this chapter, I don’t distinguish between a random vector (matrix) and a vector (matrix) of deterministic elements. For more appropriate notations, see: Abadir and Magnus (2002), Notation in econometrics: a proposal for a standard, Econometrics Journal. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 6 / 174 Section 2 The Multiple Linear Regression Model Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 7 / 174 2. The Multiple Linear Regression Model Objectives 1 De…ne the concept of Multiple linear regression model. 2 Semi-parametric and Parametric multiple linear regression model. 3 The multiple linear Gaussian model. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 8 / 174 2. The Multiple Linear Regression Model De…nition (Multiple linear regression model) The multiple linear regression model is used to study the relationship between a dependent variable and one or more independent variables. The generic form of the linear regression model is y = x1 β1 + x2 β2 + .. + xK βK + ε where y is the dependent or explained variable and x1 , .., xK are the independent or explanatory variables. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 9 / 174 2. The Multiple Linear Regression Model Notations 1 y is the dependent variable, the regressand or the explained variable. 2 xj is an explanatory variable, a regressor or a covariate. 3 ε is the error term or disturbance. IMPORTANT: do not use the term "residual".. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 10 / 174 2. The Multiple Linear Regression Model Notations (cont’d) The term ε is a random disturbance, so named because it “disturbs” an otherwise stable relationship. The disturbance arises for several reasons: 1 Primarily because we cannot hope to capture every in‡uence on an economic variable in a model, no matter how elaborate. The net e¤ect, which can be positive or negative, of these omitted factors is captured in the disturbance. 2 There are many other contributors to the disturbance in an empirical model. Probably the most signi…cant is errors of measurement. It is easy to theorize about the relationships among precisely de…ned variables; it is quite another to obtain accurate measures of these variables. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 11 / 174 2. The Multiple Linear Regression Model Notations (cont’d) We assume that each observation in a sample fyi , xi 1 , xi 2 ..xiK g for i = 1, .., N is generated by an underlying process described by yi = xi 1 β1 + xi 2 β2 + .. + xiK βK + εi Remark: xik = value of the k th explanatory variable for the i th unit of the sample xunit,variable Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 12 / 174 2. The Multiple Linear Regression Model Notations (cont’d) Let the N 1 column vector xk be the N observations on variable xk , for k = 1, .., K . Let assemble these data in an N K data matrix, X. Let y be the N 1 column vector of the N observations, y1, y2 , .., yN . Let ε be the N 1 column vector containing the N disturbances. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 13 / 174 2. The Multiple Linear Regression Model Notations (cont’d) 0 B B B y =B B N 1 B @ y1 y2 .. yi .. yN 0 1 C C C C C C A xk N 1 Christophe Hurlin (University of Orléans) B B B =B B B @ x1k x2k .. xik .. xNk 1 C C C C C C A 0 B B B ε =B B N 1 B @ Advanced Econometrics - HEC Lausanne ε1 ε2 .. εi .. εN 1 C C C C C C A 0 1 β1 B β C 2 C β =B @ .. A K 1 βK November 23, 2013 14 / 174 2. The Multiple Linear Regression Model Notations (cont’d) X = (x1 : x2 : .. : xK ) N K or equivalently 0 B B B X =B B N K B @ Christophe Hurlin (University of Orléans) x11 x12 x21 x22 .. .. xi 1 xi 2 .. xN 1 xN 2 .. x1k .. x2k .. .. .. xik .. .. .. xNk .. x1K .. x2K .. .. .. xiK .. .. .. xNK Advanced Econometrics - HEC Lausanne 1 C C C C C C A November 23, 2013 15 / 174 2. The Multiple Linear Regression Model Fact In most of cases, the …rst column of X is assumed to be a column of 1s so that β1 is the constant term in the model. x1 N 1 0 B B B X =B B N K B @ Christophe Hurlin (University of Orléans) 1 x12 1 x22 .. .. 1 xi 2 .. 1 xN 2 = 1 N 1 .. x1k .. x2k .. .. .. xik .. .. .. xNk .. x1K .. x2K .. .. .. xiK .. .. .. xNK Advanced Econometrics - HEC Lausanne 1 C C C C C C A November 23, 2013 16 / 174 2. The Multiple Linear Regression Model Remark More generally, the matrix X may as well contain stochastic and non stochastic elements such as: Constant; Time trend; Dummy variables (for speci…c episodes in time); Etc. Therefore, X is generally a mixture of …xed and random variables. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 17 / 174 2. The Multiple Linear Regression Model De…nition (Simple linear regression model) The simple linear regression model is a model with only one stochastic regressor: K = 1 if there is no constant yi = β1 xi + εi or K = 2 if there is a constant: yi = β1 + β2 xi 2 + εi for i = 1, .., N, or y = β1 + β2 x2 + ε Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 18 / 174 2. The Multiple Linear Regression Model De…nition (Multiple linear regression model) The multiple linear regression model can be written y = X N 1 Christophe Hurlin (University of Orléans) β + ε N KK 1 N 1 Advanced Econometrics - HEC Lausanne November 23, 2013 19 / 174 2. The Multiple Linear Regression Model One key di¤erence for the speci…cation of the MLRM: Parametric/semi-parametric speci…cation Parametric model: the distribution of the error terms is fully characterized, e.g. ε N (0, Ω) Semi-Parametric speci…cation: only a few moments of the error terms are speci…ed, e.g. E (ε) = 0 and V (ε) = E εε> = Ω. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 20 / 174 2. The Multiple Linear Regression Model This di¤erence does not matter for the derivation of the ordinary least square estimator But this di¤erence matters for (among others): 1 The characterization of the statistical properties of the OLS estimator (e.g., e¢ ciency); 2 The choice of alternative estimators (e.g., the maximum likelihood estimator); 3 Etc. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 21 / 174 2. The Multiple Linear Regression Model De…nition (Semi-parametric multiple linear regression model) The semi-parametric multiple linear regression model is de…ned by y = Xβ + ε where the error term ε satis…es E ( ε j X) = 0 N 1 V ( ε j X ) = σ 2 IN N N and IN is the identity matrix of order N. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 22 / 174 2. The Multiple Linear Regression Model Remarks 1 If the matrix X is non stochastic (…xed), i.e. there are only …xed regressors, then the conditions on the error term u read: E (ε) = 0 V ( ε ) = σ 2 IN 2 If the (conditional) variance covariance matrix of ε is not diagonal, i.e. if V ( ε j X) = Ω the model is called the Multiple Generalized Linear Regression Model Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 23 / 174 2. The Multiple Linear Regression Model Remarks (cont’d) The two conditions on the error term ε E ( εj X) = 0N 1 V ( ε j X ) = σ 2 IN are equivalent to E ( yj X) = Xβ V ( y j X ) = σ 2 IN Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 24 / 174 2. The Multiple Linear Regression Model De…nition (The multiple linear Gaussian model) The (parametric) multiple linear Gaussian model is de…ned by y = Xβ + ε where the error term ε is normally distributed ε N 0, σ2 IN As a consequence, the vector y has a conditional normal distribution with yj X Christophe Hurlin (University of Orléans) N Xβ, σ2 IN Advanced Econometrics - HEC Lausanne November 23, 2013 25 / 174 2. The Multiple Linear Regression Model Remarks 1 The multiple linear Gaussian model is (by de…nition) a parametric model. 2 If the matrix X is non stochastic (…xed), i.e. there are only …xed regressors, then the vector y has marginal normal distribution: y Christophe Hurlin (University of Orléans) N Xβ, σ2 IN Advanced Econometrics - HEC Lausanne November 23, 2013 26 / 174 2. The Multiple Linear Regression Model The classical linear regression model consists of a set of assumptions that describes how the data set is produced by a data generating process (DGP) Assumption 1: Linearity Assumption 2: Full rank condition or identi…cation Assumption 3: Exogeneity Assumption 4: Spherical error terms Assumption 5: Data generation Assumption 6: Normal distribution Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 27 / 174 2. The Multiple Linear Regression Model De…nition (Assumption 1: Linearity) The model is linear with respect to the parameters β1 , .., βK . Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 28 / 174 Remarks The model speci…es a linear relationship between the dependent variable and the regressors. For instance, the models y = β0 + β1 x + u y = β0 + β1 cos (x ) + v y = β0 + β1 1 +w x are all linear (with respect to β). In contrast, the model y = β0 + β1 x β2 + ε is non linear Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 29 / 174 2. The Multiple Linear Regression Model Remark The model can be linear after some transformations. Starting from y = Ax β exp (ε) , one has a log-linear speci…cation: ln (y ) = ln (A) + β ln (x ) + ε Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 30 / 174 2. The Multiple Linear Regression Model De…nition (Log-linear model) The loglinear model is ln (yi ) = β1 ln (xi 1 ) + β2 ln (xi 2 ) + .. + βK ln (xiK ) + εi This equation is also known as the constant elasticity form as in this equation, the elasticity of y with respect to changes in x does not vary with xik : ∂yi xik ∂ ln (yi ) = βk = ∂ ln (xik ) ∂xik yi Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 31 / 174 2. The Multiple Linear Regression Model The classical linear regression model consists of a set of assumptions that describes how the data set is produced by a data generating process (DGP) Assumption 1: Linearity Assumption 2: Full rank condition or identi…cation Assumption 3: Exogeneity Assumption 4: Spherical error terms Assumption 5: Data generation Assumption 6: Normal distribution Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 32 / 174 2. The Multiple Linear Regression Model De…nition (Assumption 2: Full column rank) X is an N K matrix with rank K . Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 33 / 174 2. The Multiple Linear Regression Model Interpretation 1 There is no exact relationship among any of the independent variables in the model. 2 The columns of X are linearly independent. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 34 / 174 2. The Multiple Linear Regression Model Example Suppose that a cross-section model satis…es: yi = β0 + β1 non labor incomei + β2 salaryi + β3 total incomei + εi The identi…cation condition does not hold since total income is exactly equal to salary plus non labor income (exact linear dependency in the model). Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 35 / 174 2. The Multiple Linear Regression Model Remarks 1 Perfect multi-collinearity is generally not di¢ cult to spot and is signalled by most statistical software. 2 Imperfect multi-collinearity is a more serious issue (see further). Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 36 / 174 2. The Multiple Linear Regression Model De…nition (Identi…cation) The multiple linear regression model is identi…able if and only if one the following equivalent assertions holds: (i) rank (X) = K (ii) The matrix X> X is invertible (iii) The columns of X form a basis of L (X) (iv) Xβ1 = Xβ2 =) β1 = β2 8 ( β1 , β2 ) 2 RK RK (v) Xβ = 0 =) β = 0 8 β 2 RK (vi) ker (X) = f0g Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 37 / 174 2. The Multiple Linear Regression Model The classical linear regression model consists of a set of assumptions that describes how the data set is produced by a data generating process (DGP) Assumption 1: Linearity Assumption 2: Full rank condition or identi…cation Assumption 3: Exogeneity Assumption 4: Spherical error terms Assumption 5: Data generation Assumption 6: Normal distribution Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 38 / 174 2. The Multiple Linear Regression Model De…nition (Assumption 3: Strict exogeneity of the regressors) The regressors are exogenous in the sense that: E ( εj X) = 0N 1 or equivalently for all the units i 2 f1, ..N g E ( εi j X) = 0 or equivalently E ( εi j xjk ) = 0 for any explanatory variable k 2 f1, ..K g and any unit j 2 f1, ..N g . Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 39 / 174 2. The Multiple Linear Regression Model CommentsComments 1 The expected value of the error term at observation i (in the sample) is not a function of the independent variables observed at any observation (including the i th observation). The independent variables are not predictors of the error terms. 2 The strict exogeneity condition can be rewritten as: E ( y j X) = Xβ 3 If the regressors are …xed, this condition can be rewritten as: E (ε) = 0N Christophe Hurlin (University of Orléans) 1 Advanced Econometrics - HEC Lausanne November 23, 2013 40 / 174 2. The Multiple Linear Regression Model Implications The (strict) exogeneity condition E ( εj X) = 0N 1 1 has two implications: The zero conditional mean of ε implies that the unconditional mean of u is also zero (the reverse is not true): E (ε) = EX (E ( εj X)) = EX (0) = 0 2 The zero conditional mean of ε implies that (the reverse is not true): E (εi xjk ) = 0 8i, j, k or Cov (εi , X) = 0 Christophe Hurlin (University of Orléans) 8i Advanced Econometrics - HEC Lausanne November 23, 2013 41 / 174 2. The Multiple Linear Regression Model The classical linear regression model consists of a set of assumptions that describes how the data set is produced by a data generating process (DGP) Assumption 1: Linearity Assumption 2: Full rank condition or identi…cation Assumption 3: Exogeneity Assumption 4: Spherical error terms Assumption 5: Data generation Assumption 6: Normal distribution Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 42 / 174 2. The Multiple Linear Regression Model De…nition (Assumption 4: Spherical disturbances) The error terms are such that: V ( εi j X) = E ε2i X = σ2 for all i 2 f1, ..N g and Cov ( εi , εj j X) = E ( εi εj j X) = 0 for all i 6= j The condition of constant variances is called homoscedasticity. The uncorrelatedness across observations is called nonautocorrelation. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 43 / 174 2. The Multiple Linear Regression Model Comments 1 Spherical disturbances = homoscedasticity + nonautocorrelation 2 If the errors are not spherical, we call them nonspherical disturbances. 3 The assumption of homoscedasticity is a strong one: this is the exception rather than the rule! Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 44 / 174 2. The Multiple Linear Regression Model Comments Let us consider the (conditional) variance covariance matrix of the error terms: V ( εj X) = E εε> X | {z } | {z } N N 0 B B B =B B B @ E ( ε1 ε2 j X) E ε21 X E ( ε2 ε1 j X) E ε22 X .. E ( εi ε1 j X) .. E ( εN ε1 j X) Christophe Hurlin (University of Orléans) N N .. E ( ε1 εj j X) .. E ( ε2 εj j X) .. .. .. E ( εi εj j X) .. .. .. E ( εN εj j X) Advanced Econometrics - HEC Lausanne .. E ( ε1 εN j X) .. E ( ε2 εN j X) .. .. .. E ( εi εN j X) .. .. .. E ε2N X November 23, 2013 1 C C C C C C A 45 / 174 2. The Multiple Linear Regression Model Comments The two assumptions (homoscedasticity and nonautocorrelation) imply that: V ( εj X) = E εε> X = σ2 IN | {z } | {z } N N 0 B B B =B B B @ Christophe Hurlin (University of Orléans) N N σ2 0 0 σ2 .. .. .. 0 .. .. .. .. .. .. 0 0 .. .. .. 0 .. 0 .. 0 .. .. .. 0 .. .. .. σ2 Advanced Econometrics - HEC Lausanne 1 C C C C C C A November 23, 2013 46 / 174 2. The Multiple Linear Regression Model The classical linear regression model consists of a set of assumptions that describes how the data set is produced by a data generating process (DGP) Assumption 1: Linearity Assumption 2: Full rank condition or identi…cation Assumption 3: Exogeneity Assumption 4: Spherical error terms Assumption 5: Data generation Assumption 6: Normal distribution Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 47 / 174 2. The Multiple Linear Regression Model De…nition (Assumption 5: Data generation) The data in (xi 1 xi 2 ...xiK ) may be any mixture of constants and random variables. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 48 / 174 2. The Multiple Linear Regression Model Comments 1 Analysis will be done conditionally on the observed X, so whether the elements in X are …xed constants or random draws from a stochastic process will not in‡uence the results. 2 In the case of stochastic regressors, the unconditional statistical properties of are obtained in two steps: (1) using the result conditioned on X and (2) …nding the unconditional result by ”averaging” (i.e., integrating over) the conditional distributions. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 49 / 174 2. The Multiple Linear Regression Model Comments Assumptions regarding (xi 1 xi 2 ...xiK yi ) for i = 1, .., N is also required. This is a statement about how the sample is drawn. In the sequel, we assume that (xi 1 xi 2 ...xiK yi ) for i = 1, .., N are independently and identically distributed (i.i.d). The observations are drawn by a simple random sampling from a large population. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 50 / 174 2. The Multiple Linear Regression Model The classical linear regression model consists of a set of assumptions that describes how the data set is produced by a data generating process (DGP) Assumption 1: Linearity Assumption 2: Full rank condition or identi…cation Assumption 3: Exogeneity Assumption 4: Spherical error terms Assumption 5: Data generation Assumption 6: Normal distribution Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 51 / 174 2. The Multiple Linear Regression Model De…nition (Assumption 6: Normal distribution) The disturbances are normally distributed. εi j X N 0, σ2 or equivalently εj X Christophe Hurlin (University of Orléans) N 0N 1, σ 2 IN Advanced Econometrics - HEC Lausanne November 23, 2013 52 / 174 2. The Multiple Linear Regression Model Comments 1 Once again, this is a convenience that we will dispense with after some analysis of its implications. 2 Normality is not necessary to obtain many of the results presented below. 3 Assumption 6 implies assumptions 3 (exogeneity) and 4 (spherical disturbances). Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 53 / 174 2. The Multiple Linear Regression Model Summary The main assumptions of the multiple linear regression model A1: linearity The model is linear with β A2: identi…cation X is an N A3: exogeneity E ( εj X) = 0N K matrix with rank K 1 A4: spherical error terms V ( ε j X) = A5: data generation X may be …xed or random A6: normal distribution εj X Christophe Hurlin (University of Orléans) σ2 I N 0N Advanced Econometrics - HEC Lausanne N 1, σ 2I N November 23, 2013 54 / 174 2. The Multiple Linear Regression Model Key Concepts 1 Simple linear regression model 2 Multiple linear regression model 3 Semi-parametric multiple linear regression model 4 Multiple linear Gaussian model 5 Assumptions of the multiple linear regression model 6 Linearity (A1), Identi…cation (A2), Exogeneity (A3), Spherical error terms (A4), Data generation (A5) and Normal distribution (A6) Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 55 / 174 Section 3 The ordinary least squares estimator Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 56 / 174 3. The ordinary least squares estimator Introduction 1 The simple linear regression model assumes that the following speci…cation is true in the population: y = Xβ + ε where other unobserved factors determining y are captured by the error term ε. 2 3 Consider a sample fxi 1 , xi 2 , .., xiK , yi gN i =1 of i.i.d. random variables (be careful to the change of notations here) and only one realization of this sample (your data set). How to estimate the vector of parameters β? Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 57 / 174 3. The ordinary least squares estimator Introduction (cont’d) 1 If we assume that assumptions A1-A6 hold, we have a multiple linear Gaussian model (parametric model), and a solution is to use the MLE. The MLE estimator for β coincides to the ordinary least squares (OLS) estimator (cf. chapter 2). 2 If we assume that only assumptions A1-A5 hold, we have a semi-parametric multiple linear regression model, the MLE is unfeasible. 3 In this case, the only solution is to use the ordinary least squares estimator (OLS). Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 58 / 174 3. The ordinary least squares estimator Intuition Let us consider the simple linear regression model and for simplicity denote xi = xi 2 : yi = β1 + β2 xi + εi The general idea of the OLS consists in minimizing the ”distance” between the points (xi , yi ) and the regression line ybi = b β1 + b β2 xi or the points (xi , ybi ) for all i = 1, .., N Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 59 / 174 3. The ordinary least squares estimator Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 60 / 174 3. The ordinary least squares estimator Intuition (cont’d) Estimates of β1 and β2 are chosen by minimizing the sum of the squared residuals (SSR): N ∑ bε2i i =1 This SSR can be written as: N ∑ bε2i = i =1 yi b β1 b β2 xi 2 Therefore, b β1 and b β2 are the solutions of the minimization problem N b β1 , b β2 = arg min ∑ (yi ( β1 ,β2 ) i =1 Christophe Hurlin (University of Orléans) β1 Advanced Econometrics - HEC Lausanne β2 xi )2 November 23, 2013 61 / 174 3. The ordinary least squares estimator De…nition (OLS - simple linear regression model) In the simple linear regression model yi = β1 + β2 xi + εi , the OLS estimators b β1 and b β2 are the solutions of the minimization problem N b β1 , b β2 = arg min ∑ (yi ( β1 ,β2 ) i =1 The solutions are: b β1 = y N β1 β2 xi )2 b β2 x N ∑N (xi x N ) (yi y N ) b β 2 = i =1 N 2 ∑i =1 (xi x N ) 1 N x respectively denote the where y N = N 1 ∑N ∑ i =1 i i =1 yi and x N = N sample mean of the dependent variable y and the regressor x. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 62 / 174 3. The ordinary least squares estimator Remark The OLS estimator is a linear estimator (cf. chapter 1) since it can be expressed as a linear function of the observations yi : b β2 = with ωi = N ∑ ωi yi i =1 ( xi x N ) (xi x N )2 ∑N i =1 in the case where y N = 0. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 63 / 174 3. The ordinary least squares estimator De…nition (Fitted value) The predicted or …tted value for observation i is: ybi = b β1 + b β2 xi with a sample mean equal to the sample average of the observations ybN = Christophe Hurlin (University of Orléans) 1 N N ∑ ybi = y N i =1 = 1 N N ∑ yi i =1 Advanced Econometrics - HEC Lausanne November 23, 2013 64 / 174 3. The ordinary least squares estimator De…nition (Fitted residual) The residual for observation i is: b β1 bεi = yi b β 2 xi with a sample mean equal to zero by de…nition. bεN = Christophe Hurlin (University of Orléans) 1 N N ∑ bεi = 0 i =1 Advanced Econometrics - HEC Lausanne November 23, 2013 65 / 174 3. The ordinary least squares estimator Remarks 1 2 ε2i (or SSR) is The …t of the regression is ”good” if the sum ∑N i =1 b ”small”, i.e., the unexplained part of the variance of y is ”small”. The coe¢ cient of determination or R2 is given by: ∑N (ybi R = iN=1 ∑i =1 (yi 2 Christophe Hurlin (University of Orléans) y N )2 y N )2 2 =1 εi ∑N i =1 b 2 N ∑i =1 (yi y N ) Advanced Econometrics - HEC Lausanne November 23, 2013 66 / 174 3. The ordinary least squares estimator Orthogonality conditions Under assumption A3 (strict exogeneity), we have E ( εi j xi ) = 0. This condition implies that: E ( εi ) = 0 E (εi xi ) = 0 Using the sample analog of this moment conditions (cf. chapter 6, GMM), one has: 1 N β1 b β2 xi = 0 yi b N i∑ =1 1 N N ∑ i =1 yi b β1 b β2 xi xi = 0 This is a system of two equations and two unknowns b β1 and b β2 . Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 67 / 174 3. The ordinary least squares estimator De…nition (Orthogonality conditions) The ordinary least squares estimator can be de…ned from the two sample analogs of the following moment conditions: E ( εi ) = 0 E (εi xi ) = 0 The corresponding system of equations is just-identi…ed. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 68 / 174 3. The ordinary least squares estimator OLS and multiple linear regression model Now consider the multiple linear regression model y = Xβ + ε or K yi = ∑ βk xik + εi k =1 Objective: …nd an estimator (estimate) of β1 , β2 , .., βK and σ2 under the assumptions A1-A5. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 69 / 174 3. The ordinary least squares estimator OLS and multiple linear regression model Di¤erent methods: 1 Minimize the sum of squared residuals (SSR) 2 Solve the same minimization problem with matrix notation. 3 Use moment conditions. 4 Geometrical interpretation Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 70 / 174 3. The ordinary least squares estimator 1. Minimize the sum of squared residuals (SSR): As in the simple linear regression, N b = arg min ∑ β β i =1 ε2i K N = arg min ∑ β yi i =1 ∑ βk xik k =1 !2 One can derive the …rst order conditions with respect to βk for k = 1, .., K and solve a system of K equations with K unknowns. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 71 / 174 3. The ordinary least squares estimator 2. Using matrix notations: De…nition (OLS and multiple linear regression model) In the multiple linear regression model yi = xi> β + εi , with b is the solution of xi = (xi 1 , .., xiK )> , the OLS estimator β N b = arg min ∑ yi β β xi> β 2 i =1 The OLS estimators of β is: b= β Christophe Hurlin (University of Orléans) N ∑ i =1 xi xi> ! 1 N ∑ xi yi i =1 Advanced Econometrics - HEC Lausanne ! November 23, 2013 72 / 174 3. The ordinary least squares estimator 2. Using matrix notations: De…nition (Normal equations) Under suitable regularity conditions, in the multiple linear regression model yi = xi> β + εi , with xi = (xi 1 : .. : xiK )> , the normal equations are N ∑ xi i =1 Christophe Hurlin (University of Orléans) yi b = 0K xi> β Advanced Econometrics - HEC Lausanne 1 November 23, 2013 73 / 174 3. The ordinary least squares estimator 2. Using matrix notations: De…nition (OLS and multiple linear regression model) b In the multiple linear regression model y = Xβ + ε, the OLS estimator β is the solution of the minimization problem b = arg min ε> ε = arg min (y β β Xβ)> (y Xβ) β The OLS estimators of β is: b = X> X β Christophe Hurlin (University of Orléans) 1 X> y Advanced Econometrics - HEC Lausanne November 23, 2013 74 / 174 3. The ordinary least squares estimator 2. Using matrix notations: De…nition b of β minimizes the following criteria The ordinary least squares estimator β s ( β) = k(y Christophe Hurlin (University of Orléans) Xβ)k2IN = (y Xβ)> (y Advanced Econometrics - HEC Lausanne Xβ) November 23, 2013 75 / 174 3. The ordinary least squares estimator 2. Using matrix notations: The FOC (normal equations) are de…ned by: ∂s ( β) ∂β = b β b = 0K 2|{z} X> y X β {z } K N| 1 N 1 The second-order conditions hold: ∂s ( β) ∂β∂β > b β > =2 X | {zX} is de…nite positive K K since by de…nition X> X is a positive de…nite matrix. We have a minimum. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 76 / 174 3. The ordinary least squares estimator 2. Using matrix notations: De…nition (Normal equations) Under suitable regularity conditions, in the multiple linear regression model y = Xβ + ε, the normal equations are given by: X> (y Xβ) = 0K |{z} | {z } K N Christophe Hurlin (University of Orléans) 1 N 1 Advanced Econometrics - HEC Lausanne November 23, 2013 77 / 174 3. The ordinary least squares estimator De…nition (Unbiased variance estimator) In the multiple linear regression model y = Xβ + ε, the unbiased estimator of σ2 is given by: b2 = σ Christophe Hurlin (University of Orléans) N 1 N K ∑ bε2i i =1 SSR N K Advanced Econometrics - HEC Lausanne November 23, 2013 78 / 174 3. The ordinary least squares estimator 2. Using matrix notations: b2 can also be written as: The estimator σ b2 = σ N b2 = σ 2 (y b = σ Christophe Hurlin (University of Orléans) N 1 K ∑ yi i =1 Xβ)> (y N K k(y b xi> β 2 Xβ) Xβ)k2IN N K Advanced Econometrics - HEC Lausanne November 23, 2013 79 / 174 3. The ordinary least squares estimator 3. Using moment conditions: Under assumption A3 (strict exogeneity), we have E ( εj X) = 0. This condition implies: E (εi xi ) = 0K 1 with xi = (xi 1 : .. : xiK )> . Using the sample analogs, one has: 1 N N ∑ xi i =1 yi b = 0K xi> β 1 We have K (normal) equations with K unknown parameters b β1 , .., b βK . The system is just identi…ed. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 80 / 174 3. The ordinary least squares estimator 4. Geometric interpretation: 1 2 The ordinary least squares estimation methods consists in determining the adjusted vector, b y, which is the closest to y (in a certain space...) such that the squared norm between y and b y is minimized. Finding b y is equivalent to …nd an estimator of β. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 81 / 174 3. The ordinary least squares estimator 4. Geometric interpretation: De…nition (Geometric interpretation) The adjusted vector, b y, is the (orthogonal) projection of y onto the column space of X. The …tted error terms, b ε, is the projection of y onto the orthogonal space engendered by the column space of X. The vectors b y and b ε are orthogonal. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 82 / 174 3. The ordinary least squares estimator 4. Geometric interpretation: Source: F. Pelgrin (2010), Lecture notes, Advanced Econometrics Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 83 / 174 3. The ordinary least squares estimator 4. Geometric interpretation: De…nition (Projection matrices) The vectors b y and b ε are de…ned to be: b y=P y b ε=M y where P and M denote the two following projection matrices: P = X X> X M = IN Christophe Hurlin (University of Orléans) P = IN 1 X> X X> X Advanced Econometrics - HEC Lausanne 1 X> November 23, 2013 84 / 174 3. The ordinary least squares estimator Other geometric interpretations: Suppose that there is a constant term in the model. 1 The least squares residuals sum to zero: N ∑ bεi = 0 i =1 2 The regression hyperplane passes through the point of means of the data (xN , y N ) . 3 The mean of the …tted (adjusted) values of y equals the mean of the actual values of y : ybN = y N Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 85 / 174 3. The ordinary least squares estimator De…nition (Coe¢ cient of determination) The coe¢ cient of determination of the multiple linear regression model (with a constant term) is the ratio of the total (empirical) variance explained by model to the total (empirical) variance of y: R2 = bi ∑N i =1 ( y N ∑i =1 (yi Christophe Hurlin (University of Orléans) y N )2 y N )2 2 =1 εi ∑N i =1 b 2 N ∑i =1 (yi y N ) Advanced Econometrics - HEC Lausanne November 23, 2013 86 / 174 3. The ordinary least squares estimator Remark 1 2 The coe¢ cient of determination measures the proportion of the total variance (or variability) in y that is accounted for by variation in the regressors (or the model). Problem: the R2 automatically and spuriously increases when extra explanatory variables are added to the model. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 87 / 174 3. The ordinary least squares estimator De…nition (Adjusted R-squared) The adjusted R-squared coe¢ cient is de…ned to be: N 2 R =1 N 1 p 1 1 R2 where p denotes the number of regressors (not counting the constant term, i.e., p = K 1 if there is a constant or p = K otherwise). Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 88 / 174 3. The ordinary least squares estimator Remark One can show that 1 2 3 2 R <R2 2 if N is large R 'R2 2 The adjusted R-squared R can be negative. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 89 / 174 3. The ordinary least squares estimator Key Concepts 1 OLS estimator and estimate 2 Fitted or predicted value 3 Residual or …tted residual 4 Orthogonality conditions 5 Normal equations 6 Geometric interpretations of the OLS 7 Coe¢ cient of determination and adjusted R-squared Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 90 / 174 Section 4 Statistical properties of the OLS estimator Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 91 / 174 4. Statistical properties of the OLS estimator In order to study the statistical properties of the OLS estimator, we have to distinguish (cf. chapter 1): 1 The …nite sample properties 2 The large sample or asymptotic properties Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 92 / 174 4. Statistical properties of the OLS estimator But, we have also to distinguish the properties given the assumptions made on the linear regression model 1 Semi-parametric linear regression model (the exact distribution of ε is unknown) versus parametric linear regression model (and especially Gaussian linear regression model, assumption A6). 2 X is a matrix of random regressors versus X is a matrix of …xed regressors. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 93 / 174 4. Statistical properties of the OLS estimator Fact (Assumptions) In the rest of this section, we assume that assumptions A1-A5 hold. A1: linearity The model is linear with β A2: identi…cation X is an N A3: exogeneity E ( εj X) = 0N K matrix with rank K 1 A4: spherical error terms V ( ε j X ) = σ 2 IN A5: data generation X may be …xed or random Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 94 / 174 Subsection 4.1. Finite sample properties of the OLS estimator Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 95 / 174 4.1. Finite sample properties Objectives The objectives of this subsection are the following: 1 2 3 4 Compute the two …rst moments of the (unknown) …nite sample b and σ b2 distribution of the OLS estimators β b and Determine the …nite sample distribution of the OLS estimators β b under particular assumptions (A6). σ Determine if the OLS estimators are "good": e¢ cient estimator versus BLUE. Introduce the Gauss-Markov theorem. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 96 / 174 4.1. Finite sample properties First moments of the OLS estimators Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 97 / 174 4.1. Finite sample properties Moments In a …rst step, we will derive the …rst moments of the OLS estimators 1 2 b and V β b Step 1: compute E β b2 Step 2: compute E σ Christophe Hurlin (University of Orléans) b2 and V σ Advanced Econometrics - HEC Lausanne November 23, 2013 98 / 174 4.1. Finite sample properties De…nition (Unbiased estimator) In the multiple linear regression model y = Xβ0 + ε, under the assumption b is unbiased: A3 (strict exogeneity), the OLS estimator β b =β E β 0 where β0 denotes the true value of the vector of parameters. This result holds whether or not the matrix X is considered as random. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 99 / 174 4.1. Finite sample properties Proof Case 1: …xed regressors (cf. chapter 1) b = X> X β 1 X> y = β 0 + X> X 1 X> ε So, if X is a matrix of …xed regressors: b = β + X> X E β 0 1 X> E ( ε ) Under assumption A3 (exogeneity), E ( εj X) = E (ε) = 0. Then, we get: b =β E β 0 The OLS estimator is unbiased. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 100 / 174 4.1. Finite sample properties Proof (cont’d) Case 2: random regressors b = X> X β 1 X> y = β 0 + X> X 1 X> ε If X is includes some random elements: b X = β + X> X E β 0 1 X> E ( ε j X ) Under assumption A3 (exogeneity), E ( εj X) = 0. Then, we get: b X =β E β 0 Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 101 / 174 4.1. Finite sample properties Proof (cont’d) Case 2: random regressors b is conditionally unbiased. The OLS estimator β Besides, we have: b X =β E β 0 b = EX E β b X E β = EX ( β 0 ) = β 0 where EX denotes the expectation with respect to the distribution of X. b is unbiased. So, the OLS estimator β b =β E β 0 Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 102 / 174 4.1. Finite sample properties De…nition (Variance of the OLS estimator, non-stochastic regressors) In the multiple linear regression model y = Xβ + ε, if the matrix X is non-stochastic, the unconditional variance covariance matrix of the OLS b is: estimator β 1 b = σ 2 X> X V β Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 103 / 174 4.1. Finite sample properties Proof b = X> X β 1 X> y = β 0 + X> X 1 X> ε So, if X is a matrix of …xed regressors: b V β = E = E = Christophe Hurlin (University of Orléans) b β β0 X> X X> X 1 > 1 b β β0 X> εε> X X> X X> E εε> X X> X Advanced Econometrics - HEC Lausanne 1 1 November 23, 2013 104 / 174 4.1. Finite sample properties Proof (cont’d) Under assumption A4 (spherical disturbances), we have: V (ε) = E εε> = σ2 IN The variance covariance matrix of the OLS estimator is de…ned by: b V β = X> X = X> X = σ 2 X> X = σ 2 X> X Christophe Hurlin (University of Orléans) 1 1 X> E εε> X X> X X > σ 2 IN X X > X 1 X> X X> X 1 1 1 1 Advanced Econometrics - HEC Lausanne November 23, 2013 105 / 174 4.1. Finite sample properties De…nition (Variance of the OLS estimator, stochastic regressors) In the multiple linear regression model y = Xβ0 + ε, if the matrix X is stochastic, the conditional variance covariance matrix of the OLS b is: estimator β 1 b X = σ 2 X> X V β The unconditional variance covariance matrix is equal to: b = σ2 EX V β X> X 1 where EX denotes the expectation with respect to the distribution of X. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 106 / 174 4.1. Finite sample properties Proof b = X> X β 1 X> y = β 0 + X> X 1 X> ε So, if X is a stochastic matrix: b X V β = E = E = Christophe Hurlin (University of Orléans) b β β0 X> X X> X 1 > 1 b β β0 X X> εε> X X> X 1 X> E εε> X X X> X Advanced Econometrics - HEC Lausanne X 1 November 23, 2013 107 / 174 4.1. Finite sample properties Proof (cont’d) Under assumption A4 (spherical disturbances), we have: V ( εj X) = E εε> X = σ2 IN The conditional variance covariance matrix of the OLS estimator is de…ned by: b X V β = X> X = X> X = σ 2 X> X Christophe Hurlin (University of Orléans) 1 1 X> E εε> X X X> X X > σ 2 IN X X > X 1 1 1 Advanced Econometrics - HEC Lausanne November 23, 2013 108 / 174 4.1. Finite sample properties Proof (cont’d) We have: b X = σ 2 X> X V β 1 The (unconditional) variance covariance matrix of the OLS estimator is de…ned by: b = EX V β b X V β = σ2 EX X> X 1 where EX denotes the expectation with respect to the distribution of X. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 109 / 174 4.1. Finite sample properties Summary Case 1: X stochastic b =β E β 0 Mean Variance Cond. mean Cond. var b = σ 2 EX V β X> X b X =β E β 0 b X = σ 2 X> X V β Christophe Hurlin (University of Orléans) Case 2: X nonstochastic 1 b =β E β 0 b = σ 2 X> X V β 1 — 1 Advanced Econometrics - HEC Lausanne — November 23, 2013 110 / 174 4.1. Finite sample properties Question How to estimate the variance covariance matrix of the OLS estimator? b V β OLS b V β OLS = σ2 = σ 2 EX Christophe Hurlin (University of Orléans) X> X 1 X> X if X is nonstochastic 1 if X is stochastic Advanced Econometrics - HEC Lausanne November 23, 2013 111 / 174 4.1. Finite sample properties Question (cont’d) De…nition (Variance estimator) An unbiased estimator of the variance covariance matrix of the OLS estimator is given: 1 2 > b b β b V = σ X X OLS b 2 = (N K ) 1 b where σ ε>b ε is an unbiased estimator of σ2 . This result holds whether X is stochastic or non stochastic. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 112 / 174 4.1. Finite sample properties Summary Case 1: X stochastic Variance Estimator b = σ 2 EX V β b b β V OLS Christophe Hurlin (University of Orléans) X> X b 2 X> X =σ Case 2: X nonstochastic 1 1 b = σ 2 X> X V β b b β V OLS Advanced Econometrics - HEC Lausanne 1 b 2 X> X =σ November 23, 2013 1 113 / 174 4.1. Finite sample properties De…nition (Estimator of the variance of disturbances) Under the assumption A1-A5, in the multiple linear regression model b2 is unbiased: y = Xβ + ε, the estimator σ b2 = σ2 E σ where b2 = σ N 1 N K ∑ bε2i = i =1 b ε>b ε N K This result holds whether or not the matrix X is considered as random. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 114 / 174 4.1. Finite sample properties Proof We assume that X is stochastic. Let M denotes the projection matrix (“residual maker”) de…ned by: M = IN X X> X 1 X> with b ε = M (N ,1 ) The N 1 2 y (N ,N )(N ,1 ) N matrix M satis…es the following properties: if X is regressed on X, a perfect …t will result and the residuals will be zero, so M X = 0 The matrix M is symmetric M> = M and idempotent M M = M Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 115 / 174 4.1. Finite sample properties Proof (cont’d) The residuals are de…ned as to be: Since y = Xβ + ε, we have b ε=My b ε = M (Xβ + ε) = MXβ + Mε Since MX = 0, we have b ε = Mε Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 116 / 174 4.1. Finite sample properties Proof (cont’d) b2 is based on the sum of squared residuals (SSR) The estimator σ b2 = σ ε> Mε b ε>b ε = N K N K The expected value of the SSR is E b ε>b ε X = E ε> Mε X The scalar ε> Mε is a 1 tr E ε> Mε X 1 scalar, so it is equal to its trace. = tr E εε> M X = tr E Mεε> X since tr (AB ) = tr (AB ) . Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 117 / 174 4.1. Finite sample properties Proof (cont’d) Since M = IN X X> X 1 X> depends on X, we have: E b ε>b ε X = tr E Mεε> X = tr M E εε> X Under assumptions A3 and A4, we have E εε> X = σ2 IN As a consequence E b ε>b ε X = tr σ2 M IN = σ2 tr (M) Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 118 / 174 4.1. Finite sample properties Proof (cont’d) E b ε>b ε X = σ2 tr (M) 1 = σ2 tr IN X X> X X> = σ2 tr (IN ) σ2 tr X X> X = σ2 tr (IN ) σ2 tr X> X X> X 1 X> 1 = σ2 tr (IN ) σ2 tr (IK ) = σ 2 (N K ) Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 119 / 174 4.1. Finite sample properties Proof (cont’d) b2 , we have: By de…nition of σ 2 b X = E σ E b ε>b ε X N K = σ 2 (N K ) = σ2 N K b2 is conditionally unbiased. So, the estimator σ b2 = EX E σ b2 X E σ = EX σ2 = σ2 b2 is unbiased: The estimator σ b2 = σ2 E σ Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 120 / 174 4.1. Finite sample properties Remark b2 . Given the same principle, we can compute the variance of the estimator σ b2 = σ As a consequence, we have: ε> Mε b ε>b ε = N K N K b2 X = V σ 1 (N K) V ε> Mε X b 2 = EX V σ b2 X V σ But, it takes... at least ten slides..... Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 121 / 174 4.1. Finite sample properties b2 ) De…nition (Variance of the estimator σ In the multiple linear regression model y = Xβ0 + ε, the variance of the b2 is estimator σ b2 = V σ 2σ4 N K where σ2 denotes the true value of variance of the error terms. This result holds whether or not the matrix X is considered as random. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 122 / 174 4.1. Finite sample properties Summary Case 1: X stochastic Case 2: X nonstochastic b2 = σ2 E σ b2 = σ2 E σ Mean b2 = V σ Variance Cond. mean Cond. var 2σ4 N K b2 X = σ2 E σ b2 X = V σ Christophe Hurlin (University of Orléans) b2 = V σ 2σ4 N K Advanced Econometrics - HEC Lausanne 2σ4 N K — — November 23, 2013 123 / 174 4.1. Finite sample properties Finite sample distributions Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 124 / 174 4.1. Finite sample properties Summary 1 2 3 Under assumptions A1-A5, we can derive the two …rst moments of b i.e. the (unknown) …nite sample distribution of the OLS estimator β, b and V β b . E β Are we able to characterize the …nite sample distribution (or exact b? sampling distribution) of β For that, we need to put an assumption on the distribution of ε and use a parametric speci…cation. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 125 / 174 4.1. Finite sample properties Finite sample distribution Fact (Finite sample distribution, I) (1) In a parametric multiple linear regression model with stochastic b is known. The regressors, the conditional …nite sample distribution of β unconditional …nite sample distribution is generally unknown: b X β D where D is a multivariate distribution. Christophe Hurlin (University of Orléans) b β ?? Advanced Econometrics - HEC Lausanne November 23, 2013 126 / 174 4.1. Finite sample properties Finite sample distribution Fact (Finite sample distribution, II) (2) In a parametric multiple linear regression model with nonstochastic b is regressors, the marginal (unconditional) …nite sample distribution of β known: b D β Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 127 / 174 4.1. Finite sample properties Finite sample distribution Fact (Finite sample distribution, III) (3) In a semi-parametric multiple linear regression model, with …xed or b is unknown random regressors, the …nite sample distribution of β b X β Christophe Hurlin (University of Orléans) ?? b β ?? Advanced Econometrics - HEC Lausanne November 23, 2013 128 / 174 4.1. Finite sample properties Example (Linear Gaussian regression model) Under the assumption A6 (normality - linear Gaussian regression model), N 0,σ2 IN ε b = X> X β 1 X> y = β + X> X 1 X> ε b is If X is a stochastic matrix, the conditional distribution of β b X β N 1 β,σ2 X> X b is If X is a nonstochastic matrix, the marginal distribution of β Christophe Hurlin (University of Orléans) b β N β,σ2 X> X 1 Advanced Econometrics - HEC Lausanne November 23, 2013 129 / 174 4.1. Finite sample properties Theorem (Linear gaussian regression model) b and σ b2 have a Under the assumption A6 (normality), the estimators β …nite sample distribution given by: b β N b2 σ (N σ2 β,σ2 X> X K) χ2 (N 1 K) b and σ b2 are independent. This result holds whether or not the Moreover, β b is matrix X is considered as random. In this last case, the distribution of β 1 b X = σ 2 X> X conditional to X with V β . Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 130 / 174 4.1. Finite sample properties E¢ cient estimator or BLUE? Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 131 / 174 4.1. Finite sample properties Question: OLS estimator = "good" estimator of β? The question is to know if there this estimator is preferred to other unbiased estimators? b V β OLS b 7V β other In general, to answer to this question we use the FDCR or Cramer-Rao bound and study the e¢ ciency of the estimator (cf. chapters 1 and 2). Problem: the computation of the FDCR bound requires an assumption on the distribution of ε. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 132 / 174 4.1. Finite sample properties Two cases: 1 In a parametric model, it is possible to show that the OLS estimator is e¢ cient - its variance reaches the FDCR bound. The OLS is the Best Unbiased Estimator (BUE) 2 In a semi-parametric model: 1 2 3 we don’t known the conditional distribution of y …ven X and the likelihood of the sample. The FDCR bound is unknown: so it is impossible to show the e¢ ciency of the OLS estimator. In a semi-parametric model, we introduce another concept: the Best Linear Unbiased Estimator (BLUE) Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 133 / 174 4.1. Finite sample properties Problem (E¢ ciency of the OLS estimator) In a semi-parametric multiple linear regression model (with no assumption on the distribution of ε), it is impossible to compute the FDCR or Cramer Rao bound and to show that the e¢ ciency of the OLS estimator. The e¢ ciency of the OLS estimator can only be shown in a parametric multiple linear regression model. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 134 / 174 4.1. Finite sample properties Theorem (E¢ ciency - Gaussian model) In a Gaussian multiple linear regression model y = Xβ0 + ε, with b is e¢ cient. Its ε N 0,σ2 IN (assumption A6), the OLS estimator β variance attains the FDCR or Cramer-Rao bound: b = I 1 (β ) V β 0 N This result holds whether or not the matrix X is considered as random. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 135 / 174 4.1. Finite sample properties Proof Let us consider the case where X is nonstochastic. Under the assumption A6, the distribution is known, since y N Xβ,σ2 IN . The log-likelihood > of the sample fyi , xi gN i =1 , with xi = (xi 1 : .. : xiK ) , is then equal to: `N ( β; y, x) = N ln σ2 2 If we denote θ = β> : σ2 I N (θ0 ) = Eθ Christophe Hurlin (University of Orléans) > N ln (2π ) 2 (y Xβ)> (y 2σ2 Xβ) , we have (cf. chapter 2): ∂2 `N (θ; y, x) ∂θ∂θ> = 1 > X X σ2 Advanced Econometrics - HEC Lausanne 01 2 02 T 2σ4 1 ! November 23, 2013 136 / 174 4.1. Finite sample properties Proof (cont’d) I N (θ0 ) = 1 > X X σ2 01 2 The inverse of this block matrix is given by 1 0 1 > 2 σ X X 02 1 A= I N 1 (θ0 ) = @ 2σ4 01 2 T 02 1 T 2σ4 ! I N 1 ( β0 ) 01 2 02 IN 1 1 σ2 ! Then we have: b = I 1 ( β ) = σ 2 X> X V β 0 N Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne 1 November 23, 2013 137 / 174 4.1. Finite sample properties Problem 1 In a semi-parametric model (with no assumption on the distribution of ε), it is impossible to compute the FDCR bound and to show the e¢ ciency of the OLS estimator. 2 The solution consists in introducing the concept of best linear unbiased estimator (BLUE): the Gauss-Markov theorem... Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 138 / 174 4.1. Finite sample properties Theorem (Gauss-Markov theorem) In the linear regression model under assumptions A1-A5, the least squares b is the best linear unbiased estimator (BLUE) of β whether estimator β 0 X is stochastic or nonstochastic. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 139 / 174 4.1. Finite sample properties Comment The estimator b βk for k = 1, ..K is the BLUE of βk Best = smallest variance Linear (in y or yi ) : b βk = ∑N i =1 ω ki yi Unbiased: E b βk = βk Estimator: b βk = f (y1 , .., yN ) Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 140 / 174 4.1. Finite sample properties Remark The fact that the OLS is the BLUE means that there is no linear b unbiased estimator with a smallest variance than β BUT, nothing guarantees that a nonlinear unbiased estimator may b have a smallest variance than β Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 141 / 174 4.1. Finite sample properties Summary Model Case 1: X stochastic Case 2: X nonstochastic Semi-parametric b is the BLUE β (Gauss-Markov) b is the BLUE β (Gauss-Markov) Parametric b is e¢ cient and BUE β (FDCR bound) b is e¢ cient and BUE β (FDCR bound) BUE: Best Unbiased Estimator Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 142 / 174 4.1. Finite sample properties Other remarks Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 143 / 174 4.1. Finite sample properties Remarks 1 Including irrelevant variables (i.e., coe¢ cient = 0) does not a¤ect the unbiasedness of the estimator. 2 Irrelevant variables decrease the precision of OLS estimator, but do not result in bias. 3 Omitting relevant regressors obviously does a¤ect the estimator, since the zero conditional mean assumption does not hold anymore. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 144 / 174 4.1. Finite sample properties Remarks 1 2 Omitting relevant variables cause bias (under some conditions on these variables). 1 The direction of the bias can be (sometimes) characterized. 2 The size of the bias is more complicated to derive. More generally, correlation between a single regressor and the error term (endogeneity) usually results in all OLS estimators being biased Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 145 / 174 4.1. Finite sample properties Key Concepts 1 OLS unbiased estimator under assumption A3 (exogeneity) 2 Mean and variance of the OLS estimator under assumptions A3-A4 (exogeneity - spherical disturbances) 3 Finite sample distribution 4 Estimator of the variance of the error terms 5 BLUE estimator and e¢ cient estimator (FDCR bound) 6 Gauss-Markov theorem Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 146 / 174 Subsection 4.2. Asymptotic properties of the OLS estimator Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 147 / 174 4.2. Asymptotic properties Objectives The objectives of this subsection are the following 1 2 3 Study the consistency of the OLS estimator. b Derive the asymptotic distribution of the OLS estimator β. b Estimate the asymptotic variance covariance matrix of β. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 148 / 174 4.2. Asymptotic properties Theorem (Consistency) Let consider the multiple linear regression model yi = xi> β0 + εi , with xi = (xi 1 : .. : xiK )> . Under assumptions A1-A5, the OLS estimators b and σ b2 are ( weakly) consistent: β p b! β β0 p b2 ! σ2 σ Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 149 / 174 4.2. Asymptotic Properties Proof (cf. chapter 1) We have: b= β N ∑ i =1 xi xi> ! 1 N ∑ xi yi i =1 ! N = β0 + ∑ xi xi> 1 N N i =1 ! 1 N ∑ xi εi i =1 ! By multiplying and dividing by N, we get: b = β+ β Christophe Hurlin (University of Orléans) 1 N N ∑ i =1 xi xi> ! 1 ∑ xi εi i =1 Advanced Econometrics - HEC Lausanne ! November 23, 2013 150 / 174 4.2. Asymptotic Properties Proof (cf. chapter 1) 1 N b = β+ β 1 ∑ xi xi> i =1 ! 1 1 N N ∑ xi εi i =1 ! By using the (weak) law of large number (Kitchine’s theorem), we have: 1 N 2 N N 1 N p ∑ xi xi> ! E xi xi> i =1 N p ∑ xi εi ! E (xi εi ) i =1 By using the Slutsky’s theorem: p b! β+E β Christophe Hurlin (University of Orléans) 1 xi xi> E (xi εi ) Advanced Econometrics - HEC Lausanne November 23, 2013 151 / 174 4.2. Asymptotic Properties Proof (cf. chapter 1) p b! β β+E 1 xi xi> E (xi µi ) Under assumption A3 (exogeneity): E ( εi j xi ) = 0 ) We have E (εi xi ) = 0K E ( εi ) = 0 1 p b! β β b is (weakly) consistent. The OLS estimator β Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 152 / 174 4.2. Asymptotic properties Theorem (Asymptotic distribution) Consider the multiple linear regression model yi = xi> β0 + εi , with b is xi = (xi 1 : .. : xiK )> . Under assumptions A1-A5, the OLS estimator β asymptotically normally distributed: p or equivalently d b N β b β Christophe Hurlin (University of Orléans) β0 ! N 0, σ2 EX 1 xi xi> asy N β0 , σ2 E 1 xi xi> N X Advanced Econometrics - HEC Lausanne November 23, 2013 153 / 174 4.2. Asymptotic properties Comments 1 2 3 Whatever the speci…cation used (parametric or semi-parametric), it is always possible to characterize the asymptotic distribution of the OLS estimator. b is asymptotically Under assumptions A1-A5, the OLS estimator β normally distributed, even if ε is not normally distributed. This result (asymptotic normality) is valid whether X is stochastic or nonstochastic. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 154 / 174 4.2. Asymptotic properties Proof (cf. chapter 1) This proof has be shown in the chapter 1 (Estimation theory) 1 Let us rewrite the OLS estimator as: b=β + β 0 2 b Normalize the vector β p b N β β0 = AN = Christophe Hurlin (University of Orléans) 1 N N ∑ xi xi> i =1 ! 1 N ∑ xi εi i =1 ! β0 1 N N ∑ xi xi> i =1 ! 1 p N ∑ xi xi> bN = 1 N N p N i =1 Advanced Econometrics - HEC Lausanne N ∑ xi εi i =1 1 N ! = AN 1 bN N ∑ xi εi i =1 November 23, 2013 155 / 174 4.2. Asymptotic properties Proof (cf. chapter 1) 3. Using the WLLN (Kinchine’s theorem) and the CMP (continuous mapping theorem). Remark: if xi this part is useless. ! 1 1 N p xi xi> ! EX 1 xi xi> ∑ N i =1 4. Using the CLT: BN = p N 1 N N ∑ xi εi i =1 E (xi εi ) ! d ! N (0, V (xi εi )) Under assumption A3 E ( εi j xi ) = 0 =) E (xi εi ) = 0 and V (xi εi ) = E xi εi εi xi> = EX E xi εi εi xi> xi = EX xi V ( εi j xi ) xi> = σ2 EX xi xi> Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 156 / 174 4.2. Asymptotic properties Proof (cf. chapter 1) So we have with p b N β β 0 = A N 1 bN p AN 1 ! A 1 d bN ! N (Eb , Vb ) A 1 = EX 1 xi xi> Christophe Hurlin (University of Orléans) Eb = 0 Vb = σ2 EX xi xi> = σ2 A Advanced Econometrics - HEC Lausanne November 23, 2013 157 / 174 4.2. Asymptotic properties Proof (cf. chapter 1) p AN 1 ! A 1 d bN ! N (Eb , Vb ) A 1 =E xi xi> 1 Eb = 0 Vb = σ2 EX xi xi> = σ2 A By using the Slusky’s theorem (for a convergence in distribution), we have: d AN 1 bN ! N (Ec , Vc ) Ec = A Vc = A 1 Vb Christophe Hurlin (University of Orléans) Eb = A 1 A 1 =A 1 1 0=0 2 σ A Advanced Econometrics - HEC Lausanne A 1 = σ2 A 1 November 23, 2013 158 / 174 4.2. Asymptotic properties Proof (cf. chapter 1) p b N β d β0 = AN 1 bN ! N 0, σ2 A 1 with A 1 = EX 1 xi> xi (be careful: it is the inverse of the expectation and not the reverse). Finally, we have: p b N β Christophe Hurlin (University of Orléans) d β0 ! N 0, σ2 EX 1 xi xi> Advanced Econometrics - HEC Lausanne November 23, 2013 159 / 174 4.2. Asymptotic properties Remark 1 In some textbooks (Greene, 2007 for instance), the asymptotic variance covariance matrix of the OLS estimator is expressed as a function of the plim of N 1 X> X denoted by Q. Q = p lim 2 1 > X X N Both notations are equivalent since: 1 1 > X X= N N N ∑ xi xi> i =1 By using the WLLN (Kinchine’s theorem), we have: 1 > p X X ! Q = EX xi xi> N Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 160 / 174 4.2. Asymptotic properties Theorem (Asymptotic distribution) Consider the multiple linear regression model y = Xβ0 + ε. Under b is asymptotically normally assumptions A1-A5, the OLS estimator β distributed: p d b β ! N β N 0, σ2 Q 1 0 where Q = p lim 1 > X X = EX xi> xi N or equivalently b β Christophe Hurlin (University of Orléans) asy N β0 , σ2 Q N 1 Advanced Econometrics - HEC Lausanne November 23, 2013 161 / 174 4.2. Asymptotic properties De…nition (Asymptotic variance covariance matrix) b is The asymptotic variance covariance matrix of the OLS estimators β equal to 2 b = σ E 1 xi xi> Vasy β N X or equivalently 2 b = σ Q 1 Vasy β N where Q = p lim N 1 X> X. Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 162 / 174 4.2. Asymptotic properties Remark Finite sample variance Asymptotic variance b = σ 2 EX V β X> X 1 2 b = σ E 1 xi xi> Vasy β N X Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 163 / 174 4.2. Asymptotic properties Remark How to estimate the asymptotic variance covariance matrix of the OLS estimator? 2 b = σ E 1 xi xi> Vasy β N X Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 164 / 174 4.2. Asymptotic properties De…nition (Estimator of the asymptotic variance matrix) A consistent estimator of the asymptotic variance covariance matrix is given by: 1 b =σ b asy β b 2 X> X V b2 is consistent estimator of σ2 : where σ b2 = σ Christophe Hurlin (University of Orléans) b ε>b ε N K Advanced Econometrics - HEC Lausanne November 23, 2013 165 / 174 4.2. Asymptotic properties Proof 2 b = σ E 1 xi xi> Vasy β N X We known that: p X> X 1 = N N N b2 ! σ2 σ p ∑ xi xi> ! EX i =1 xi xi> = Q By using the CMP and the Slutsky theorem, we get: b σ 2 Christophe Hurlin (University of Orléans) X> X N ! 1 p ! σ2 EX 1 xi xi> Advanced Econometrics - HEC Lausanne November 23, 2013 166 / 174 4.2. Asymptotic properties Proof (cont’d) As consequence 2 b = σ E 1 xi x> Vasy β i N X ! 1 X> X p 2 b σ ! σ2 EX 1 xi xi> N b2 σ N or equivalently Christophe Hurlin (University of Orléans) X> X N ! b 2 X> X σ 1 p ! σ2 E 1 xi xi> N X 1 p b ! Vasy β p b ! b b asy β V Vasy β Advanced Econometrics - HEC Lausanne November 23, 2013 167 / 174 4.2. Asymptotic properties Remark Under assumptions A1-A5, for a stochastic matrix X, we have Finite sample Variance Estimator b = σ 2 EX V β b =σ b β b 2 X> X V Christophe Hurlin (University of Orléans) Asymptotic X> X 1 1 b = σ2 N Vasy β 1Q 1 with Q = p lim N1 X> X b =σ b asy β b 2 X> X V Advanced Econometrics - HEC Lausanne November 23, 2013 1 168 / 174 4.2. Asymptotic properties Example (CAPM) The empirical analogue of the CAPM is given by: e rit = αi + βi e rmt + εt e rit = rit rft | {z } excess return of security i at time t rft ) } market excess return at time t where εt is an i.i.d. error term with: E ( εt ) = 0 e rmt = (rmt | {z V ( εt ) = σ 2 E ( εt j e rmt ) = 0 Data …le: capm.xls for Microsoft, SP500 and Tbill (closing prices) from 11/1/1993 to 04/03/2003 Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 169 / 174 4.2. Asymptotic sample properties Example (CAPM, cont’d) Question: write a Matlab code in order to …nd (1) the estimates of α and β for Microsoft, (2) the corresponding standard errors, (3) the SE of the regression, (4) the SSR, (5) the coe¢ cient of determination and (6) the adjusted R 2 . Compare your results to the following table (Eviews) Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 170 / 174 4.2. Asymptotic sample properties Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 171 / 174 4.2. Asymptotic sample properties Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 172 / 174 4.2. Asymptotic sample properties Key Concepts 1 The OLS estimator is weakly consist 2 The OLS estimator is asymptotically normally distributed 3 Asymptotic variance covariance matrix 4 Estimator of the asymptotic variance Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 173 / 174 End of Chapter 3 Christophe Hurlin (University of Orléans) Christophe Hurlin (University of Orléans) Advanced Econometrics - HEC Lausanne November 23, 2013 174 / 174