advertisement

Ordinary Least Squares Regression PO7001: Quantitative Methods I Kenneth Benoit 24 November 2010 Independent and Dependent variables I I A dependent variable represents the quantity we wish to explain variation in, or the thing we are trying to explain Typical examples of a dependent variable in political science: I I I I I An independent variable represents a quantity whose variation will be used to explain variation in the dependent variable Typical examples of independent variables in political science: I I I I I I votes received by a governing party support for a referendum result like Lisbon party support for European integration demographic: gender, national background, age economic: socioeconomic status, income, national wealth political: party affiliation of one’s parents institutional: district magnitude, electoral system, presidential v. parliamentary behavioural: campaign spending levels Using this language implies causality: X → Y The importance of variation I Variation in outcomes – of the dependent variable – are what we seek to explain in social and political research I We seek to explain these outcomes using (independent) variables I The very language – here, the term “variable” suggests that the quantity so named has to vary I Conversely, a quantity that does not vary is impossible to study in this way. This also applies to samples that do not vary: these will not help us in research. I Typically when we collect data, we wish to have as much variation in our sample as possible I Example: In the returns from schooling example, it would be best to have as much variation in schooling as possible, to maximize the leverage of our research Different functional relationships Linear A linear relationship, also known as a straight-line relationship, exists if a line drawn through the central tendency of the points is a straight-line Curvilinear Exists if the relationship between variables is not a straight-line function, but is instead curved Example: Television viewing (Y ) as a function of age (X ) (More on this in Quant 2) Correlations I The central idea behind correlation is that two variables have a relationship such that there is a systematic relationship between the their values I This relationship can be positive or negative I This relationship varies in strength I Bivariate associations are usually depicted graphically by use of a scatterplot I We can also summarize (as per an earlier week) the correlation using Pearson’s r to show the direction and strength of the bivariate relationship Pearson’s r revisited r = = = P (X − X̄ )(Yi − Ȳ ) qPi i 2 2 i (Xi − X̄ ) (Yi − Ȳ ) Sum of Products p Sum of Squaresx Sum of Squaresy SP p SSx SSy Testing the significance of Pearson’s r I Pearson’s r measures correlation in the sample – but the association we are interested in exists in the population I Question: what is the probability that any correlation we measure in a sample really exists in the population, and is not merely due to sampling error I H0 : ρx,y = 0 – No correlation exists in the population I HA : ρx,y 6= 0 I Significance can be tested by the t-ratio: √ r N −2 t= √ 1 − r2 Pearson’s r significance testing: Example I Let’s assume we have data such that: r = .24, N = 8 I So computation of t is: t = = = √ .24 8 − 2 p 1 − (.24)2 .24(2.45) √ 1 − .0576 .59 .59 √ = = .61 .97 .9424 I From R (using qt()), we know that the critical value for t with df=6, α = .05 is 2.447 – so we do not reject H0 I In R there is a test for this called cor.test(x,y) Pearson’s r significance testing using R > x <- c(12,10,6,16,8,9,12,11) > y <- c(12,8,12,11,10,8,16,15) > cor(x,y) [1] 0.2420615 > # compute the empirical t-value > (t.calc <- (.24*sqrt(8-2) / sqrt(1-.24^2))) [1] 0.6055768 > # compute the critical t-value > (t.crit <- qt(1-.05/2, 6)) [1] 2.446912 > # compare > t.calc > t.crit [1] FALSE > # using R’s built-in correlation significance test > cor.test(x,y) Pearson’s product-moment correlation data: x and y t = 0.6111, df = 6, p-value = 0.5636 alternative hypothesis: true correlation is not equal to 0 95 percent confidence interval: -0.5577490 0.8087778 sample estimates: cor 0.2420615 Regression analysis I Recall the basic linear model: yi = a + bXi I I Here the relationship is determined by two parameters: a the intercept: this refers to the value of Y when X is zero b the slope: or the rate of change in Y for a one-unit change in X . Also known as the regression coefficient Note that this implies a straight line, perfect relationship however. Because this is never the case in real research, we also have an error term or residual ei : Yi = β0 + β1 xi + εi and we use the β and ε terminology instead of the high-school math quantities a and b Example 35 30 25 20 15 Sentence length (Y) 40 45 par(mar=c(4,4,1,1)) x <- c(0,3,1,0,6,5,3,4,10,8) y <- c(12,13,15,19,26,27,29,31,40,48) plot(x, y, xlab="Number of prior convictions (X)", ylab="Sentence length (Y)", pch=19) abline(h=c(10,20,30,40), col="grey70") 0 2 4 6 Number of prior convictions (X) 8 10 Least squares formulas For the three parameters (simple regression): I the regression coefficient: P (xi − x̄)(yi − ȳ ) P β̂1 = (xi − x̄)2 I the intercept: β̂0 = ȳ − β̂1 x̄ I and the residual variance σ 2 : 1 X σ̂ 2 = [yi − (β̂0 + β̂1 xi )]2 n−2 Least squares formulas continued Things to note: I the prediction line is ŷ = β̂0 + β̂1 x I the value ŷi = β̂0 + β̂1 xi is the predicted value for xi I the residual is ei = yi − ŷi I The residual sum of squares (RSS) = I The estimate for σ 2 is the same as 2 i ei P σ̂ 2 = RSS/(n − 2) Example to show fomulas in R > x <- c(0,3,1,0,6,5,3,4,10,8) > y <- c(12,13,15,19,26,27,29,31,40,48) > (data <- data.frame(x, y, xdev=(x-mean(x)), ydev=(y-mean(y)), + xdevydev=((x-mean(x))*(y-mean(y))), + xdev2=(x-mean(x))^2, + ydev2=(y-mean(y))^2)) x y xdev ydev xdevydev xdev2 ydev2 1 0 12 -4 -14 56 16 196 2 3 13 -1 -13 13 1 169 3 1 15 -3 -11 33 9 121 4 0 19 -4 -7 28 16 49 5 6 26 2 0 0 4 0 6 5 27 1 1 1 1 1 7 3 29 -1 3 -3 1 9 8 4 31 0 5 0 0 25 9 10 40 6 14 84 36 196 10 8 48 4 22 88 16 484 > (SP <- sum(data$xdevydev)) [1] 300 > (SSx <- sum(data$xdev2)) [1] 100 > (SSy <- sum(data$ydev2)) [1] 1250 > (b1 <- SP / SSx) [1] 3 From observed to “predicted” relationship I In the above example, β̂0 = 14, β̂1 = 3 I This linear equation forms the regression line I The regression line always passes through two points: I I the point (x = 0, y = β̂0 ) the point (x̄, ȳ ) (the average X predicts the average Y ) 2 i ei P I The residual sum of squares (RSS) = I The regression line is that which minimizes the RSS Plot of regression example 45 Regression line 30 25 20 15 Sentence length (Y) 35 40 Yhat = 14 + 3X Y intercept 0 2 4 6 Number of prior convictions (X) 8 10 Requirements for regression 1. Both variables should be measured at the interval level 2. The relationship (in the population) between X and Y is linear I I A transformation may fix non-linearity in some cases Outliers or influential points may need special treatment 3. Sample is randomly chosen (necessary for inference) 4. Both variables must be normally distributed, unless we have a very large sample Regression “terminology” I y is the dependent variable I I referred to also (by Greene) as a regressand X are the independent variables I I also known as explanatory variables also known as regressors I y is regressed on X I The error term is sometimes called a disturbance Notation I Independent variables X I Dependent variable Y I Yi is a random variable (not directly observed) I yi is a realised value of Yi (observed) I So dependent variable sometimes refers to a set of numbers in your dataset (y ) and sometimes to a random variable at each i (Yi ). Interpreting the regression results I The Y -intercept corresponds to the expected value of Y when X = 0 (may or may not be meaningful) I The regression coefficient β1 refers to the expected change in Y resulting from a one-unit change in X I Typically we are more interested in β1 than β0 , but β0 forms a vital part of any regression estimation I We can make a prediction yˆi for any i (and xi , although we should be careful to choose reasonable values of xi ) Example: For xi =13, yˆi = β̂0 + β̂1 (13) = 14 + 3(13) = 14 + 39 = 53 Sums of squares (ANOVA) TSS Total sum of squares P (yi − ȳ )2 P ESS Estimation or Regression sum of squares (ŷi − ȳ )2 P 2 P RSS Residual sum of squares ei = (ŷi − yi )2 The key to remember is that TSS = ESS + RSS R2 How much of the variance did we explain? PN PN 2 (yˆi − ȳ )2 RSS i=1 (yi − yˆi ) = 1 − PN = Pi=1 R =1− N 2 2 TSS i=1 (yi − ȳ ) i=1 (yi − ȳ ) 2 Can be interpreted as the proportion of total variance explained by the model. R 2 and Pearson’s r For simple linear regression (i.e. one independent variable), R 2 is the same as the correlation coefficient, Pearson’s r , squared. R2 I Note that computing the regression line comes from the same sums of squares (SP, SSx , SSY ) used in computing the correlation r I The squared correlation R 2 is an important quantity in regression (sometimes called the coefficient of determination) I R 2 is the proportion of variance in Y determined by variation in X I 0 ≤ R 2 ≤ 1.0 I But remember that R 2 is not a regression parameter I Computation: SP r=p SSx SSy R 2 computation example r = p = p = = = SP SSx SSy 300 (100)(1250) 300 √ 125000 300 353.55 .85 r 2 is then (.85)2 = .72 This means that 72% of the sentence length is explained by the number of priors R2 I A much over-used statistic: it may not be what we are interested in at all I Interpretation: the proportion of the variation in y that is explained lineraly by the independent variables I Defined in terms of sums of squares: ESS TSS RSS = 1− TSS P (yi − ŷi )2 = 1− P (yi − ȳ )2 R2 = I Alternatively, R 2 is the squared correlation coefficient between y and ŷ R 2 continued I When a model has no intercept, it is possible for R 2 to lie outside the interval (0, 1) I R 2 rises with the addition of more explanatory variables. For n−1 this reason we often report “adjusted R 2 ”: 1 − (1 − R 2 ) n−k−1 where k is the total number of regressors in the linear model (excluding the constant) I Whether R 2 is high or not depends a lot on the overall variance in Y I To R 2 values from different Y samples cannot be compared R 2 continued I Solid arrow: P variation in y when X is unknown (TSS Total Sum of Squares (yi − ȳ )2 ) I Dashed arrow: variation in y when X is known (ESS Estimation P Sum of Squares (ŷi − ȳ )2 ) R 2 decomposed y = ŷ + Var(y ) = Var(ŷ ) + Var(e) + 2Cov(ŷ , e) Var(y ) = Var(ŷ ) + Var(e) + 0 X X (yi − ȳ )2 /N = (ŷi − ŷ¯ )2 /N + (ei − ê)2 /N X X X (yi − ȳ )2 = (ŷi − ŷ¯ )2 + (ei − ê)2 X X X (yi − ȳ )2 = (ŷi − ŷ¯ )2 + ei2 X TSS TSS/TSS = ESS + RSS = ESS/TSS + RSSTSS 1 = R 2 + unexplained variance Example from height-weight data (of course) there is a direct way to compute regression statistics in R: > # regression models in R > regmodel <- lm(y ~ x) > summary(regmodel) Call: lm(formula = y ~ x) Residuals: Min 1Q Median -5.2121 -2.5682 -0.6515 3Q 1.5303 Max 8.2424 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 49.0909 21.6202 2.271 0.0636 . x 0.7576 0.3992 1.898 0.1065 --Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 4.587 on 6 degrees of freedom Multiple R-squared: 0.375, Adjusted R-squared: 0.2709 F-statistic: 3.601 on 1 and 6 DF, p-value: 0.1065 Illustration using the Anscombe dataset [show R analysis here] ## Illustration using Anscombe dataset data(anscombe) attach(anscombe) round(coef(lm(y1~x1)), 2) # same b0, b1 round(coef(lm(y2~x2)), 2) round(coef(lm(y3~x3)), 2) round(coef(lm(y4~x4)), 2) round(summary(lm(y1~x1))$r.squared, 2) # same R^2 round(summary(lm(y2~x2))$r.squared, 2) round(summary(lm(y3~x3))$r.squared, 2) round(summary(lm(y4~x4))$r.squared, 2) # plot the four x-y pairs par(mfrow=c(2,2), mar=c(4, 4, 1, 1)+0.1) # 4 plots in one graphic window plot(x1,y1) abline(lm(y1~x1), col="red", lty="dashed") plot(x2,y2) abline(lm(y2~x2), col="red", lty="dashed") plot(x3,y3) abline(lm(y3~x3), col="red", lty="dashed") plot(x4,y4)1 abline(lm(y4~x4), col="red", lty="dashed") detach(anscombe) 6 y2 4 3 5 4 6 5 7 y1 8 7 9 8 10 9 11 Anscombe plots 4 6 8 10 12 14 4 6 8 10 12 14 x2 y4 6 6 8 8 y3 10 10 12 12 x1 4 6 8 10 x3 12 14 8 10 12 14 x4 16 18 Extensions of the regression model I The regression model can be generalized to multiple regression, which involves “regressing Y ” on several independent variables X1 , X2 , etc. I Regression allows us to isolate the linear contribution of each unit of each Xk on Y , by “holding everything else constant” I This is the most common and most powerful basic technique in social science statistics, and something that is used in virtually every analysis that attempts to establish any sort of cause and effect I The additional X variables are typically considered to be control variables I Some variables may also be categorical, especially when they are “dummy variables” (0 or 1) Linear model The linear model can be written as: Yi = Xi β + εi εi ∼ N(0, σ 2 ) Or, alternatively, as: Yi ∼ N(µi , σ 2 ) µi = Xi β Linear model Two components of the model: Yi ∼ N(µi , σ 2 ) µi = Xi β Stochastic Systematic Generalised version: Yi ∼ f (θi , α) θi = g (Xi , β) Stochastic Systematic Model Yi ∼ f (θi , α) θi = g (Xi , β) Stochastic Systematic Stochastic component: varies over repeated (hypothetical) observations on the same unit. Systematic component: varies across units, but constant given X . Model Yi ∼ f (θi , α) θi = g (Xi , β) Stochastic Systematic Two types of uncertainty: Estimation uncertainty: lack of knowledge about α and β; can be reduced by increasing N. Fundamental uncertainty: represented by stochastic component and exists independent of researcher. Inference from regression In linear regression, the sampling distribution of the coefficient estimates form a normal distribution, which is approximated by a t distribution due to approximating σ by s. Thus we can calculate a confidence interval for each estimated coefficient. Or perform a hypothesis test along the lines of: H0 :β1 = 0 H1 :β1 6= 0 Inference from regression To calculate the confidence interval, we need to calculate the standard error of the coefficient. Rule of thumb to get the 95% confidence interval: β − 2SE < β < β + 2SE Thus if β is positive, we are 95% certain it is different from zero when β − 2SE > 0. (Or when the t value is greater than 2 or less than −2.) In R, we get the standard errors by using the summary command on the model output. Simple regression model: example Does infant mortality get lower as GDP per capita increases, measured in 1976? > m <- lm(INFMORT ~ LEVEL, data=aclp, subset=(YEAR == 1976 & INFMORT >= 0)) > summary(m) Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 33.3028647 2.7222356 12.234 9.44e-13 *** LEVEL -0.0018998 0.0003038 -6.253 9.29e-07 *** F -test In simple linear regression, we can do an F -test: H0 :β1 = 0 H1 :β1 6= 0 F = ESS/1 ESS = 2 RSS/(n − 2) σ̂ with 1 and n − 2 degrees of freedom. Example > require(foreign) > dail <- read.dta("dailcorrected.dta") > summary(lm(votes1st ~ spend_total + incumb + electorate + minister, data=dail)) Call: lm(formula = votes1st ~ spend_total + incumb + electorate + minister, data = dail) Residuals: Min 1Q -4934.1 -1038.8 Median -347.6 3Q 1054.0 Max 6900.3 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 7.966e+02 4.172e+02 1.909 0.0569 . spend_total 1.737e-01 1.095e-02 15.862 <2e-16 *** incumbIncumbent 2.522e+03 2.207e+02 11.424 <2e-16 *** electorate -4.827e-04 5.404e-03 -0.089 0.9289 minister -1.303e+02 3.965e+02 -0.329 0.7425 --Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 1847 on 458 degrees of freedom (1 observation deleted due to missingness) Multiple R-squared: 0.6478, Adjusted R-squared: 0.6447 F-statistic: 210.6 on 4 and 458 DF, p-value: < 2.2e-16 CLRM: Basic Assumptions 1. Specification: I I I I Relationship between X and Y in the population is linear: E(Y ) = X β No extraneous variables in X No omitted independent variables Parameters (β) are constant 2. E() = 0 3. Error terms: I I Var() = σ 2 , or homoskedastic errors E(ri ,j ) = 0, or no auto-correlation CLRM: Basic Assumptions (cont.) 4. X is non-stochastic, meaning observations on independent variables are fixed in repeated samples I I I implies no measurement error in X implies no serial correlation where a lagged value of Y would be used an independent variable no simultaneity or endogenous X variables 5. N > k, or number of observations is greater than number of independent variables (in matrix terms: rank(X ) = k), and no exact linear relationships exist in X 6. Normally distributed errors: |X ∼ N(0, σ 2 ). Technically however this is a convenience rather than a strict assumption Ordinary Least Squares (OLS) I Objective: minimize I I P P (Yi − Ŷi )2 , where Ŷi = b0 + b1 Xi error ei = (Yi − Ŷi ) b = = I ei2 = P (Xi − X̄ )(Yi − Ȳ ) P (Xi − X̄ ) P XY P i 2i Xi The intercept is: b0 = Ȳ − b1 X̄ OLS rationale I Formulas are very simple I Closely related to ANOVA (sums of squares decomposition) I Predicted Y is sample mean when Pr(Y |X ) =Pr(Y ) I I I In the special case where Y has no relation to X , b1 = 0, then OLS fit is simply Ŷ = b0 Why? Because b0 = Ȳ − b1 X̄ , so Ŷ = Ȳ Prediction is then sample mean when X is unrelated to Y I Since OLS is then an extension of the sample mean, it has the same attractice properties (efficiency and lack of bias) I Alternatives exist but OLS has generally the best properties when assumptions are met OLS in matrix notation I Formula for coefficient β: Y 0 = Xβ + XY = X 0X β + X 0 X 0Y = X 0X β + 0 (X 0 X )−1 X 0 Y = β+0 β = (X 0 X )−1 X 0 Y I Formula for variance-covariance matrix: σ 2 (X 0 X )−1 I I In simple P case where y = β0 + β1 ∗ x, this gives σ 2 / (xi − x̄)2 for the variance of β1 Note how increasing the variation in X will reduce the variance of β1 The “hat” matrix I The hat matrix H is defined as: β̂ = (X 0 X )−1 X 0 y X β̂ = X (X 0 X )−1 X 0 y ŷ I I I = Hy H = X (X 0 X )−1 X 0 is called the hat-matrix P 2 Other important quantities, such as ŷ , ei (RSS) can be expressed as functions of H Corrections for heteroskedastic errors (“robust” standard errors) involve manipulating H Some important OLS properties to understand Applies to y = α + βx + I If β = 0 and the only regressor is the intercept, then this is the same as regressing y on a column of ones, and hence α = ȳ — the mean of the observations I If α = 0 so that therePis no intercept and one explanatory variable x, then β = P xy x2 I If there is an intercept and one explanatory variable, then P − x̄)(yi − ȳ ) i (x Pi β = (xi − x̄)2 P (xi − x̄)yi = Pi (xi − x̄)2 Some important OLS properties (cont.) I If the observations are expressed as deviations from P ∗ their P means, y ∗ = y − ȳ and x ∗ = x − x̄, then β = x y ∗ / x ∗2 I The intercept can be estimated as ȳ − βx̄. This implies that the intercept is estimated by the value that causes the sum of the OLS residuals to equal zero. I The mean of the ŷ values equals the mean y values – together with previous properties, implies that the OLS regression line passes through the overall mean of the data points Normally distributed errors OLS in R > dail <- read.dta("dail2002.dta") > mdl <- lm(votes1st ~ spend_total*incumb + minister, data=dail) > summary(mdl) Call: lm(formula = votes1st ~ spend_total * incumb + minister, data = dail) Residuals: Min 1Q -5555.8 -979.2 Median -262.4 3Q 877.2 Max 6816.5 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 469.37438 161.54635 2.906 0.00384 ** spend_total 0.20336 0.01148 17.713 < 2e-16 *** incumb 5150.75818 536.36856 9.603 < 2e-16 *** minister 1260.00137 474.96610 2.653 0.00826 ** spend_total:incumb -0.14904 0.02746 -5.428 9.28e-08 *** --Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 1796 on 457 degrees of freedom (2 observations deleted due to missingness) Multiple R-squared: 0.6672, Adjusted R-squared: 0.6643 F-statistic: 229 on 4 and 457 DF, p-value: < 2.2e-16 OLS in Stata . use dail2002 (Ireland 2002 Dail Election - Candidate Spending Data) . gen spendXinc = spend_total * incumb (2 missing values generated) . reg votes1st spend_total incumb minister spendXinc Source | SS df MS -------------+-----------------------------Model | 2.9549e+09 4 738728297 Residual | 1.4739e+09 457 3225201.58 -------------+-----------------------------Total | 4.4288e+09 461 9607007.17 Number of obs F( 4, 457) Prob > F R-squared Adj R-squared Root MSE = = = = = = 462 229.05 0.0000 0.6672 0.6643 1795.9 -----------------------------------------------------------------------------votes1st | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------spend_total | .2033637 .0114807 17.71 0.000 .1808021 .2259252 incumb | 5150.758 536.3686 9.60 0.000 4096.704 6204.813 minister | 1260.001 474.9661 2.65 0.008 326.613 2193.39 spendXinc | -.1490399 .0274584 -5.43 0.000 -.2030003 -.0950794 _cons | 469.3744 161.5464 2.91 0.004 151.9086 786.8402 ------------------------------------------------------------------------------ Sums of squares (ANOVA) TSS Total sum of squares P (yi − ȳ )2 P ESS Estimation or Regression sum of squares (ŷi − ȳ )2 P 2 P RSS Residual sum of squares ei = (ŷi − yi )2 The key to remember is that TSS = ESS + RSS Examining the sums of squares > yhat <- mdl$fitted.values # uses the lm object mdl from previous > ybar <- mean(mdl$model[,1]) > y <- mdl$model[,1] # can’t use dail$votes1st since diff N > TSS <- sum((y-ybar)^2) > ESS <- sum((yhat-ybar)^2) > RSS <- sum((yhat-y)^2) > RSS [1] 1473917120 > sum(mdl$residuals^2) [1] 1473917120 > (r2 <- ESS/TSS) [1] 0.6671995 > (adjr2 <- (1 - (1-r2)*(462-1)/(462-4-1))) [1] 0.6642865 > summary(mdl)$r.squared # note the call to summary() [1] 0.6671995 > RSS/457 [1] 3225202 > sqrt(RSS/457) [1] 1795.885 > summary(mdl)$sigma [1] 1795.885 Regression model return values Here we will talk about the quantities returned with the lm() command and lm class objects. Next (last) week I ANOVA - Analysis of variance I Regression with categorical independent variables I Regression with interaction terms (Brambor, Clark, and Golder) I Preview of Quant 2 with non-linear/normal models