Basic Regression – Model, Assumptions, and Estimation I. Modeling the Regression from Theory We now turn to the subject of regression in econometrics. To understand this we begin with an economic theory that says that X and Y are related. We can write this as 𝑌 = 𝑓(𝑋) Our economic theory tells us the qualitative relation between X and Y. Namely, our theory tells us whether 𝑓 ′ (𝑋) > 0 or 𝑓 ′ (𝑋) < 0. However, we are now interested in assessing any possible stable quantitative relation between X and Y. We want to know how much X affects Y and not merely the sign of the effect. Given the basic functional relation between X and Y , our next step is to convert this general function relation into a model which is linear in the parameters. This can be done in a number of ways. Here are some examples. 𝒀 = 𝒇(𝑿) Linear (in parameters) Model (1) 𝑌 = 𝛽𝑜 + 𝛽1 𝑋 + 𝜀 (1') 𝑌 = 𝛽𝑜 + 𝛽1 𝑋 + 𝜀 (2) 𝑌 = 𝐴𝑒 𝛼𝑋 𝑒 𝜀 (2') 𝑙𝑛(𝑌) = 𝑙𝑛(𝐴) + 𝛼𝑋 + 𝜀 (3) 𝑌 = 𝐴𝑋 𝛼 𝑒 𝜀 (3') 𝑙𝑛(𝑌) = 𝑙𝑛(𝐴) + 𝛼𝑙𝑛(𝑋) + 𝜀 In the table above note how that each equation on the right hand side is linear in the parameters. If we let 𝑙𝑛(𝐴) = 𝛽𝑜 and 𝛼 = 𝛽1 , then it is clear that the equations on the right hand side are linear in the parameters. The last term in each of the models is ε which we take as a random variable having a particular PDF. This means that ε has a mean, E[ε] and a variance, var(ε). We usually assume that E[ε] = 0 and that var(ε) = σ2, particularly when we are discussing basic regression. Now suppose that we have a model like (1') above. We also have a random sample on X and Y. We can write this as X1, X2, ..., XN and Y1, Y2,...,YN. This means that we have N equations which we can write as 𝑌1 = 𝛽𝑜 + 𝛽1 𝑋1 + 𝜀1 𝑌2 = 𝛽𝑜 + 𝛽1 𝑋2 + 𝜀2 𝑌3 = 𝛽𝑜 + 𝛽1 𝑋3 + 𝜀3 ⋮ 𝑌𝑁 = 𝛽𝑜 + 𝛽1 𝑋𝑁 + 𝜀𝑁 This system of equations can be written in a concise way as 𝑌𝑡 = 𝛽𝑜 + 𝛽1 𝑋𝑡 + 𝜀𝑡 for t = 1,2,...,N The goal of our study is to use data on X and Y to estimate or guess the values of 𝛽𝑜 , 𝛽1 , σ2, and 𝜀𝑡 for t = 1,2,...,N. This lecture is concerned with finding a good way to estimate these parameters. In order to do this we need to make some basic assumptions about the model. II. Major Assumptions used in Basic Regression Our basic regression model uses certain assumptions which we cannot prove to be true, but which we can later statistically test for their reliability. These assumptions can be written as follows: Assumptions: (1) 𝐸[𝜀𝑡 ] = 0 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑡 = 1,2, … , 𝑁 (2) var(𝜀𝑡 ) = 𝜎 2 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑡 = 1,2, … , 𝑁 (3) cov(𝜀𝑡 , 𝜀𝑠 ) = 0 𝑤ℎ𝑒𝑛 𝑡 ≠ 𝑠 (4) the X's and the ε's are independent The first assumption is easy. It just says that the error is stable and does not have a mean that is moving over the sample. Since the mean of ε is stable, we can just assume that it is zero. The second assumption says that the variance is constant and is equal to a number called σ2 . The third assumption says that any random disturbance εt will be uncorrelated with any other random disturbance in the sample εs. These random errors cannot affect each other linearly. The fourth and last assumption is that the X's and ε's have no relation to each other. The last assumption is vital if we are to make a good estimation. Unfortunately, the assumption is violated in many cases that we consider in economics. But, at a deeper level, our regression model assumes that Y is on the left hand side and X is on the right hand side. Y is clearly influenced by ε. X is assumed to be independent of ε, but not of Y. Thus, there is a fundamental asymmetry in basic regression that X and Y cannot change places across the equality sign. If X is on the right, then we are assuming the error is independent of X. If we then change places, putting X on the left and Y on the right, we will be making a clear mistake in modeling, since it is impossible for ε to be independent of both X and Y. Regression is therefore more than correlation, which does not distinguish between such right hand variables and left hand side variables. We now turn to the issue of estimation of the model. III. Estimation of the Basic Regression Model Our model can be written as 𝑌𝑡 = 𝛽𝑜 + 𝛽1 𝑋𝑡 + 𝜀𝑡 for t = 1,2,...,N Our first step is to use the random sample on X and Y to create estimators for the coefficients βo and β1. These estimators are called method of moment estimators, but if the error term ε is distributed as a normal PDF then the method of moment estimator will be equal to the famous ordinary least squares estimator than we will discuss later. Here is how we use our assumptions to guide the creation of good estimators. To start with we assume that our estimators of βo and β1 are written as 𝛽̂𝑜 and 𝛽̂1 . In addition, our estimator of 𝜀𝑡 will be written as 𝜀̂𝑡 . The predicted value of 𝒀𝒕 will be written as 𝑌̂𝑡 , and is equal to 𝑌̂𝑡 = 𝛽̂𝑜 + 𝛽̂1 𝑋𝑡 which means that the predicted error, 𝜀̂𝑡 , (called the residual) is equal to 𝜀̂𝑡 = 𝑌𝑡 − 𝑌̂𝑡 which implies that 𝑌𝑡 = 𝛽̂𝑜 + 𝛽̂1 𝑋𝑡 + 𝜀̂𝑡 Step 1: Add the 𝑌𝑡 ′𝑠 together, divide by N, and subtract from 𝑌𝑡 to get the following: (𝑌𝑡 − 𝑌̅ ) = 𝛽̂1 (𝑋𝑡 − 𝑋̅) + (𝜀̂𝑡 − 𝜀̂)̅ for t = 1, 2,...,N Note that we next force 𝜀̂ ̅ = 0 in the above in order to closely follow Assumption 1. Step 2: Multiply both sides by (𝑋𝑡 − 𝑋̅) to get (𝑌𝑡 − 𝑌̅)(𝑋𝑡 − 𝑋̅) = 𝛽̂1 (𝑋𝑡 − 𝑋̅)2 + 𝜀̂𝑡 (𝑋𝑡 − 𝑋̅) for t = 1, 2,...,N Step 3: We next sum these together to get the following 𝑁 𝑁 𝑁 ∑(𝑋𝑡 − 𝑋̅)(𝑌𝑡 − 𝑌̅) = 𝛽̂1 ∑(𝑋𝑡 − 𝑋̅)2 + ∑ 𝑡=1 𝑡=1 (𝑋𝑡 − 𝑋̅) 𝜀̂𝑡 𝑡=1 ̅ After this, we use assumption 4 to force ∑𝑁 𝑡=1(𝑋𝑡 − 𝑋 ) 𝜀̂𝑡 = 0. ̅ 2 Step 4: Finally, we divide both sides by ∑𝑁 𝑡=1(𝑋𝑡 − 𝑋 ) to get the estimator for 𝛽1 as 𝛽̂1 = ̅ ̅ ∑𝑁 𝑡=1(𝑋𝑡 − 𝑋 )(𝑌𝑡 − 𝑌 ) ̅ 2 ∑𝑁 𝑡=1(𝑋𝑡 − 𝑋 ) Step 5: Having developed the estimator for 𝛽1 , we now consider the estimator for 𝛽𝑜 in the following way – we know that 𝑌𝑡 = 𝛽̂𝑜 + 𝛽̂1 𝑋𝑡 + 𝜀̂𝑡 and 𝜀̂ ̅ = 0 and therefore taking the average of the Y's gives 𝑌̅ = 𝛽̂𝑜 + 𝛽̂1 𝑋̅ and we can easily solve for our estimator of 𝛽𝑜 as 𝛽̂𝑜 = 𝑌̅ − 𝛽̂1 𝑋̅ We now have successfully calculated two of our estimators -- one for βo and one for β1. We still must estimate the error terms, as well as the variance of the error. Here is how we can do this. Step 6: We can easily create an estimator for the error term, since we have previously shown how this can be done. We simply let 𝜀̂𝑡 = 𝑌𝑡 − 𝑌̂𝑡 = 𝑌𝑡 − 𝛽̂𝑜 − 𝛽̂1 𝑋𝑡 for t = 1, 2, ..., N We refer to these 𝜀̂𝑡 's as the regression residuals. We often make a graph of these residuals much like the following taken from GRETL A careful study of the residuals can often tell us whether our regression is healthy and valuable or not. It is a diagnostic tool much like an X-ray or a blood test is a diagnostic tool for the doctor. The residuals are extremely important and therefore we will be spending much time trying to understand how to correctly evaluate them. Step 7: Finally we can guess σ2 (which we call s2) using the following estimator – 2 ∑𝑁 𝑡=1 𝜀̂𝑡 𝜎̂ = 𝑠 = 𝑁−2 2 2 We divide this estimator by N-2 since two degrees of freedom of our data have been lost calculating the estimators of βo and β1. The estimator s2 shows us the rough band about zero that the residuals fluctuate. Typically, we can say that 95% of the residuals will lie in a band which is 2 standard deviations above and below zero in size. The standard deviation, called the standard error of the regression is equal to Standard Error of the Regression = √𝑠 2 Problems: (1) Explain why we assume 𝜀̂ ̅ = 0 in deriving our estimators. ̅ (2) Explain why we assume ∑𝑁 𝑡=1(𝑋𝑡 − 𝑋 ) 𝜀̂𝑡 = 0 in deriving our estimators. (3) On page 1 above, why can we say that the right hand side of the table are "linear in parameters". (4) Explain why that 𝑌𝑡 = 𝛽̂𝑜 + 𝛽̂1 𝑋𝑡 + 𝜀̂𝑡 , given that 𝑌𝑡 = 𝛽𝑜 + 𝛽1 𝑋𝑡 + 𝜀𝑡 . (5) Prove that 𝑁 ∑ (𝑋𝑡 − 𝑋̅)(𝑌𝑡 − 𝑌̅) 𝑡=1 𝑁 =∑ 𝑡=1 𝑋𝑡 (𝑌𝑡 − 𝑌̅) = ∑ 𝑁 (𝑋𝑡 − 𝑋̅)𝑌𝑡 𝑡=1 (6) Suppose that our regression model is 𝑌𝑡 = 𝛽1 𝑋𝑡 + 𝜀𝑡 for t = 1, 2,..., N We can therefore skip Step 1 in the derivation of the estimator for β1. Sow that the estimator for β1 is equal to 𝛽̂1 = ∑𝑁 𝑡=1 𝑋𝑡 𝑌𝑡 2 ∑𝑁 𝑡=1 𝑋𝑡 . (7) Suppose that our regression model is 𝑌𝑡 = 𝛽𝑜 + 𝜀𝑡 for t = 1, 2,..., N. Derive an estimator for βo. (8) Suppose that our regression model is 𝑌𝑡 = 𝛽𝑜 + 𝛽1 𝑋𝑡 + 𝜀𝑡 for t = 1, 2,..., N. Prove that the average of 𝑌̂𝑡 is equal to the average of 𝑌𝑡 . (9) Suppose that our regression model is 𝑌𝑡 = 𝛽𝑜 + 𝛽1 𝑋𝑡 + 𝜀𝑡 for t = 1, 2,..., N. Explain why that the estimator 𝛽̂1 is a random variable. Find the expectation of 𝛽̂1 . (10) Suppose that our regression model is 𝑌𝑡 = 𝛽𝑜 + 𝛽1 𝑋𝑡 + 𝜀𝑡 for t = 1, 2,..., N. Find the variance of 𝛽̂1 .