Lecture 1 Mark Hendricks University of Chicago September 2012 Outline OLS OLS Inference OLS Asymptotics Hendricks, Spring 2012 Time Series Statistics Lecture 1 2/42 Linear regression model Consider a linear regression model involving two variables, y and x. y =β0 + βx + Hendricks, I y is referred to as the regressand, or explained variable. I x is referred to as the regressor, covariate, or explanatory variable. I β0 and β are the (constant) parameters of the model. Spring 2012 Time Series Statistics Lecture 1 3/42 Multivariate linear regression In a multivariate regression model with k regressors, y =β0 + β1 x1 + β2 x2 + . . . βk xk + =β0 + k X βj xj + j=1 =x0 β + Hendricks, I The last line defines x such that the first element is the constant 1, and the first element of β is β0 . I Including the regression constant in the vector notation will simplify the algebra, as we will always consider the case where the first regressor is a constant. Spring 2012 Time Series Statistics Lecture 1 4/42 Data from the regression model A sample of n observations is denoted as (yi , xi ) for i = 1, 2, . . . n. yi =x0i β + i where 1 xi,1 xi ≡ xi,2 .. . xi,k Hendricks, Spring 2012 β0 β1 β ≡ β2 .. . βk Time Series Statistics Lecture 1 5/42 Sample regression Consider a sample estimate of β, denoted by b. I Define ei = yi − x0i b Thus, yi = x0i b + ei I Hendricks, This is the sample counterpart to the population regression equation. Spring 2012 Time Series Statistics Lecture 1 6/42 Ordinary least squares The ordinary least squares estimator of β minimizes the sum of squared sample errors: b ≡ arg min bo = arg min bo Hendricks, Spring 2012 n X (i )2 i=1 n X yi − x0i bo 2 i=1 Time Series Statistics Lecture 1 7/42 OLS problem Rewrite the OLS problem in matrix notation, b ≡ arg min e0 e bo = arg min (Y − Xbo )0 (Y − Xbo ) bo where 0 x1 x0 2 X ≡ . , .. x0n Hendricks, Spring 2012 y1 y2 Y ≡ . , .. e1 e2 e ≡ . , .. yn Time Series Statistics en Lecture 1 8/42 Assumption: Full-rank Assumption 1: X0 X is full rank. Equivalently, assume that there is no exact linear relationship among any of the regressors. Hendricks, I Clearly, the existence of OLS estimator requires that this assumption be satisfied. I Multicollinearity refers to the case where this assumption fails. Spring 2012 Time Series Statistics Lecture 1 9/42 OLS estimate Solving the minimization problem above gives the OLS estimate: b = X0 X Hendricks, −1 X0 Y I This estimate yields sample residuals of −1 0 e =Y − X X0 X XY −1 0 = I − X X0 X X Y I Thus e is orthogonal to X. I Equivalently, the in-sample correlation between xi and ei is zero. Spring 2012 Time Series Statistics Lecture 1 10/42 OLS as linear projection Using the derivation of b and e, we can write the regression equation as a linear projection. Y = PY + MY where M and P are defined as P ≡ X X0 X −1 X0 M ≡ I n − X X0 X Hendricks, −1 X0 I P is an n × n matrix which projects data into the space spanned by the column vectors of X. I M is an n × n matrix which projects data into the space orthogonal to X. Spring 2012 Time Series Statistics Lecture 1 11/42 OLS as method of moments Suppose the population correlation between x and is zero. 0 =E [x] 0 =E x y − x0 β Thus, −1 β = E xx0 E [xy ] If we estimate β by simply replacing population moments with sample moments, −1 0 b = X0 X XY which is the same as the OLS estimator above. Hendricks, Spring 2012 Time Series Statistics Lecture 1 12/42 Assumption: Exogeneity Assumption 2: is exogenous to the regressors, x. E [ |x] = 0 The exogeneity assumption, Hendricks, I implies that is uncorrelated with x. I implies that is uncorrelated with any function of x. I does NOT imply that is independent of x. Spring 2012 Time Series Statistics Lecture 1 13/42 Conditional expectation It is generally true that given any variable y along with a vector of variables, x, we can write y =E [y | x] + where is simply the prediction error, = y − E [y | x] Under the exogeneity assumption, E [y | x] =x0 β Hendricks, Spring 2012 Time Series Statistics Lecture 1 14/42 Projection and conditional expectation I Using just the assumption of full-rank, linear projection still gives the best linear conditional forecast. h 2 i b = arg min E y − x0 γ γ That is, the linear projection x0 b minimizes the mean-squared-error. I Contrast with assuming linearity and exogeneity. Then the linear projection equals the conditional expectation: E [y |x] = x0 b Hendricks, Spring 2012 Time Series Statistics Lecture 1 15/42 Unbiased statistics An estimate is unbiased if its expectation equals the population value. I Consider the sample estimator, x̄, for a sample of n. I Suppose we have a variable x with population mean µx .; I Verify that x is unbiased: " E [x] =E n 1X xi n # i=1 n 1X = E [xi ] n i=1 =µx Hendricks, Spring 2012 Time Series Statistics Lecture 1 16/42 Is OLS estimate unbiased? Check if the OLS estimator is unbiased: h −1 0 i XY E [b] =E X0 X i h −1 0 0 X (Xβ + ) =E X X i h −1 0 X =β + E X0 X But E [b] = β if, and only if, h −1 0 i E X0 X X =0 which is guaranteed by the exogeneity assumption but not simply by orthogonality. Hendricks, Spring 2012 Time Series Statistics Lecture 1 17/42 Variance of estimator Continue with the example of the sample mean estimator, x̄ for sample size n. I Suppose var [x] = σ 2 . I What is the variance of the sample mean estimator, x̄? var [x̄] = Hendricks, Spring 2012 σ2 n Time Series Statistics Lecture 1 18/42 Variance of OLS estimator The variance of the OLS estimator is −1 −1 0 X ΣX X0 X var [b |x] = X0 X where Σ = E [0 |x]. Hendricks, I So far, no assumptions have been made regarding the distribution of the residuals, , beyond their orthogonality. I Inference regarding regression estimates is heavily influenced by the covariance of the residuals. Spring 2012 Time Series Statistics Lecture 1 19/42 Heteroscedasticity and autocorrelation of residuals I I Hendricks, Heteroscedasticity refers to the have distinct variances, 2 σ1 0 0 σ2 2 Σ= 0 0 case where the residuals but ··· ··· .. . ··· 0 0 σn2 Autocorrelation refers to the case where residuals are correlated. In this case, Σ is not diagonal. Spring 2012 Time Series Statistics Lecture 1 20/42 Assumption: Homoscedastic and orthogonal residuals Assumption 3: The residuals are uncorrelated across observations, with identical variances, Σ = E 0 |x = σ 2 In Hendricks, Spring 2012 Time Series Statistics Lecture 1 21/42 Gauss-Markov Theorem With these assumptions, the OLS estimator, b, is the minimum variance linear unbiased estimator of β. Hendricks, I The assumption on Σ simplifies the variance of the OLS estimator, −1 0 2 −1 var [b |x] = X0 X X σ In X X0 X −1 =σ 2 X0 X I Any other linear, unbiased estimator of β will have larger variance. I This is known as the Gauss-Markov theorem. It depends on the above assumptions regarding linearity, exogeneity, full-rank, and residual covariance structure. Spring 2012 Time Series Statistics Lecture 1 22/42 Outline OLS OLS Inference OLS Asymptotics Hendricks, Spring 2012 Time Series Statistics Lecture 1 23/42 OLS Inference The distribution of the OLS estimates is required in order to assess statistical significance. I Hendricks, Above the mean and variance of b were derived without making any distributional assumptions. Spring 2012 Time Series Statistics Lecture 1 24/42 Assumption: Normality of residuals Assumption 4: The residuals, are normally distributed. |x ∼ N (0, Σ) Hendricks, Spring 2012 Time Series Statistics Lecture 1 25/42 Distribution of OLS estimator Assumptions 1, 2, 3, 4 imply b |x ∼ N (β, Ω) where Ω = σ2 X 0X −1 Often, these 4 assumptions are referred to as the classical regression model. I Hendricks, Note that many results were derived without any distributional assumptions. Spring 2012 Time Series Statistics Lecture 1 26/42 OLS Z-test Testing the significance of an element of b, would simply be a z-test: bj − βj ∼Z σ ωjj where ωjj2 is the (j, j) element of X0 X element of Ω. . Thus, σ 2 ωij2 is the (i, j) I However, this statistic depends on σ, which is unknown. I Instead one must use the sample estimate of the variance, s2 = Hendricks, −1 Spring 2012 e0 e n−k −1 Time Series Statistics Lecture 1 27/42 OLS t-test The t-test assesses statistical significance of an element of b, bj − βj ∼ t(n − k − 1) s ωjj which depends only on observable data as well as the hypothesized value, βj . Hendricks, Spring 2012 Time Series Statistics Lecture 1 28/42 R-squared The R-squared, or coefficient of determination, in a regression is defined as Ry2,x = regression sum of squares total sum of squares =1 − error sum of squares total sum of squares Algebraically, this is P b n (xi − x̄)2 Ry2,x = Pni=1 2 i=1 (yi − ȳ ) 0 ee =1 − Pn 2 i=1 (yi − ȳ ) Hendricks, Spring 2012 Time Series Statistics Lecture 1 29/42 R-squared versus correlation Intuitively, the R-squared is the square of the correlation between y and the projection of y onto x. Ry2,x = [corr (Y, PY)]2 In a univariate regression of y on x, Ry2,x = [corr(y , x)]2 Hendricks, Spring 2012 Time Series Statistics Lecture 1 30/42 Caveat: Regressing on a constant The interpretation and formula for R-squared does not hold if there is no constant regressor. I Recall that the first element of x is the constant 1, and the first column of X is a column of 1’s. I Including a constant in the regression is equivalent to running a regression with demeaned data. I Running a regression on just a constant regressor and nothing else, would simply pick up the mean in the data. I Thus, simply shifting the sample data will improve the fit as it lines up closer to the sample mean. Note that throughout the derivations above, the constant regressor guarantees that the mean of e will be zero. Hendricks, Spring 2012 Time Series Statistics Lecture 1 31/42 OLS F-test An F-test will determine the joint significance of the linear regression: Ry2,x 1 − Ry2,x n−k −1 k −2 ∼ F (k − 2, n − k − 1) Namely, this tests whether all coefficients are jointly equal to zero. Hendricks, I Note the use of the R-squared stat. I It is simple to generalize this to test whether b is jointly equal to a non-zero hypothesis vector, β ∗ . Spring 2012 Time Series Statistics Lecture 1 32/42 Problems with multicollinearity If regressor j is highly correlated with the other regressors, then the variance of the coefficient estimate can be written as var [bj |x] = 1 1 − Rx2j ,x[−k] ! σ2 Pn 2 i=1 (xi,j − x̄j ) where Rx2j ,x[−k] denotes the R-squared from regressing xj on the remaining regressors. Hendricks, Spring 2012 Time Series Statistics Lecture 1 33/42 Variance inflation factor The term 1 1 − Rx2j ,x[−k] is known as the variance inflation factor. Hendricks, I The VIF is closely related to the condition number of X0 X. I The condition number in linear algebra measures the sensitivity of inverting a matrix. I It compares the largest and smallest eigenvalues. I Most software packages will warn the user if the condition number (VIF) is too big. Spring 2012 Time Series Statistics Lecture 1 34/42 Variation in regressors The sample variance of xj reduces the variance of the OLS estimate bj . I Again write var [bj |x] = Hendricks, 1 1 − Rx2j ,x[−k] ! σ2 Pn 2 i=1 (xi,j − x̄j ) I The denominator of the second term is the scaled sample variance. I If there is not enough variation in the regressor data, then the OLS estimation can not precisely estimate βj . Spring 2012 Time Series Statistics Lecture 1 35/42 Time Series Statistics Lecture 1 36/42 Outline OLS OLS Inference OLS Asymptotics Hendricks, Spring 2012 Non-normality Applications often do not satisfy Assumption 4, upon which the inference results relied. Hendricks, I However, the asymptotic distribution of the OLS estimate is still normally distributed under much weaker assumptions. I In practice, inference often relies on having large data sets and appealing to the asymptotic results. Spring 2012 Time Series Statistics Lecture 1 37/42 Assumption: Orthogonality of population residuals Assumption 5: the regressors. The population residuals are uncorrelated with E x0 = 0 Hendricks, I This assumption is much weaker than Assumption 2. I This is a restriction on the population variables, not the fitted estimates, which have zero correlation by construction. Spring 2012 Time Series Statistics Lecture 1 38/42 Consistency A sample statistic is consistent if it converges to the true population value in probability. I Suppose that Assumptions 1, 5 hold. I Then the OLS estimator, b is consistent, plim b = β I Hendricks, In practice, more attention is paid to having a consistent estimator than an unbiased estimator, due to the weaker assumption. Spring 2012 Time Series Statistics Lecture 1 39/42 Asymptotic distribution of OLS Under Assumptions 1, 5, 3, the OLS estimate is asymptotically normal, b |x ∼asym N (β, Ω) where Ω = σ2 X 0X Hendricks, Spring 2012 −1 Time Series Statistics Lecture 1 40/42 Heteroscedastic and autocorrelated inference For many applications, particularly in time-series, Assumption 3 is clearly false. Hendricks, I For practical purposes, this is not a big problem for inference. I Standard errors which correct for heteroscedasticity and autocorrelation are widely employed. I We come back to these later. Spring 2012 Time Series Statistics Lecture 1 41/42 References I Hendricks, Cochrane, John. Asset Pricing. 2001. I Greene, William. Econometric Analysis. 2011. I Hamilton, James. Time Series Analysis. 1994. I Wooldridge, Jeffrey. Econometric Analysis of Cross Section and Panel Data. 2011. Spring 2012 Time Series Statistics Lecture 1 42/42