Statistics for Managers using Microsoft Excel 3rd Edition Chapter 11 Simple Linear Regression © 2002 Prentice-Hall, Inc. Chap 11-1 Chapter Topics Types of regression models Determining the simple linear regression equation Measures of variation Assumptions of regression and correlation Residual analysis Measuring autocorrelation Inferences about the slope © 2002 Prentice-Hall, Inc. Chap 11-2 Chapter Topics (continued) Correlation - measuring the strength of the association Estimation of mean values and prediction of individual values Pitfalls in regression and ethical issues © 2002 Prentice-Hall, Inc. Chap 11-3 Purpose of Regression Analysis Regression analysis is used primarily to model causality and provide prediction Predicts the value of a dependent (response) variable based on the value of at least one independent (explanatory) variable Explains the effect of the independent variables on the dependent variable © 2002 Prentice-Hall, Inc. Chap 11-4 Types of Regression Models Positive Linear Relationship Negative Linear Relationship © 2002 Prentice-Hall, Inc. Relationship NOT Linear No Relationship Chap 11-5 Simple Linear Regression Model Relationship between variables is described by a linear function The change of one variable causes the change in the other variable A dependency of one variable on the other © 2002 Prentice-Hall, Inc. Chap 11-6 Population Linear Regression Population regression line is a straight line that describes the dependence of the average value (conditional mean) of one variable on the other Population Slope Coefficient Population Y intercept Dependent (Response) Variable © 2002 Prentice-Hall, Inc. Random Error Yi X i i Population Regression YX Line (conditional mean) Independent (Explanatory) Variable Chap 11-7 Population Linear Regression (continued) Y (Observed Value of Y) = Yi X i i i = Random Error YX X i (Conditional Mean) X Observed Value of Y © 2002 Prentice-Hall, Inc. Chap 11-8 Sample Linear Regression Sample regression line provides an estimate of the population regression line as well as a predicted value of Y Sample Y Intercept Yi b0 b1 X i ei Sample Slope Coefficient Residual Sample Regression Line ˆ Y b0 b1 X (Fitted Regression Line, Predicted Value) © 2002 Prentice-Hall, Inc. Chap 11-9 Sample Linear Regression (continued) b0 and b1 are obtained by finding the values of b0 and b that minimizes the sum of the 1 squared residuals n i 1 Yi Yˆi e 2 n i 1 2 i b0 provides an estimate of b1 provides and estimate of © 2002 Prentice-Hall, Inc. Chap 11-10 Sample Linear Regression (continued) Yi b0 b1 X i ei Y ei Yi X i i b1 i YX X i b0 Y i b0 b1 X i X Observed Value © 2002 Prentice-Hall, Inc. Chap 11-11 Interpretation of the Slope and the Intercept E Y | X 0 is the average value of Y when the value of X is zero. E Y | X 1 measures the change in the X average value of Y as a result of a one-unit change in X. © 2002 Prentice-Hall, Inc. Chap 11-12 Interpretation of the Slope and the Intercept (continued) b Eˆ Y | X 0 is the estimated average value of Y when the value of X is zero. Eˆ Y | X b1 X is the estimated change in the average value of Y as a result of a oneunit change in X. © 2002 Prentice-Hall, Inc. Chap 11-13 Simple Linear Regression: Example You want to examine the linear dependency of the annual sales of produce stores on their size in square footage. Sample data for seven stores were obtained. Find the equation of the straight line that fits the data best. © 2002 Prentice-Hall, Inc. Store Square Feet Annual Sales ($1000) 1 2 3 4 5 6 7 1,726 1,542 2,816 5,555 1,292 2,208 1,313 3,681 3,395 6,653 9,543 3,318 5,563 3,760 Chap 11-14 Scatter Diagram: Example Annua l Sa le s ($000) 12000 10000 8000 6000 4000 2000 0 0 Excel Output © 2002 Prentice-Hall, Inc. 1000 2000 3000 4000 5000 6000 S q u a re F e e t Chap 11-15 Equation for the Sample Regression Line: Example Yˆi b0 b1 X i 1636.415 1.487 X i From Excel Printout: C o e ffi c i e n ts I n te r c e p t 1 6 3 6 .4 1 4 7 2 6 X V a ria b le 1 1 .4 8 6 6 3 3 6 5 7 © 2002 Prentice-Hall, Inc. Chap 11-16 Graph of the Sample Regression Line: Example Annua l Sa le s ($000) 12000 10000 8000 6000 4000 2000 0 0 1000 2000 3000 4000 5000 6000 S q u a re F e e t © 2002 Prentice-Hall, Inc. Chap 11-17 Interpretation of Results: Example Yˆi 1636.415 1.487 Xi The slope of 1.487 means that for each increase of one unit in X, we predict the average of Y to increase by an estimated 1.487 units. The model estimates that for each increase of one square foot in the size of the store, the expected annual sales are predicted to increase by $1487. © 2002 Prentice-Hall, Inc. Chap 11-18 Simple Linear Regression in PHStat In excel, use PHStat | regression | simple linear regression … EXCEL spreadsheet of regression sales on footage © 2002 Prentice-Hall, Inc. Chap 11-19 Measure of Variation: The Sum of Squares SST = Total = Sample Variability © 2002 Prentice-Hall, Inc. SSR Explained Variability + SSE + Unexplained Variability Chap 11-20 Measure of Variation: The Sum of Squares SST = total sum of squares Measures the variation of the Yi values around their mean Y SSR = regression sum of squares (continued) Explained variation attributable to the relationship between X and Y SSE = error sum of squares Variation attributable to factors other than the relationship between X and Y © 2002 Prentice-Hall, Inc. Chap 11-21 Measure of Variation: The Sum of Squares (continued) SSE =(Yi - Yi )2 Y _ SST = (Yi - Y)2 _ SSR = (Yi - Y)2 Xi © 2002 Prentice-Hall, Inc. _ Y X Chap 11-22 Venn Diagrams and Explanatory Power of Regression Variations in store sizes not used in explaining variation in sales Sizes Sales Variations in sales explained by the error term SSE Variations in sales explained by sizes or variations in sizes used in explaining variation in sales SSR © 2002 Prentice-Hall, Inc. Chap 11-23 The ANOVA Table in Excel ANOVA df Regression p SS MS SSR MSR =SSR/p Residuals n-p-1 SSE Total n-1 © 2002 Prentice-Hall, Inc. F Significance F MSR/MSE P-value of the F Test MSE =SSE/(n-p-1) SST Chap 11-24 Measures of Variation The Sum of Squares: Example Excel Output for Produce Stores Degrees of freedom ANOVA df SS MS Regression 1 30380456.12 30380456 Residual 5 1871199.595 374239.92 Total 6 32251655.71 F 81.17909 Regression (explained) df Error (residual) df Total df © 2002 Prentice-Hall, Inc. SSE SSR Significance F 0.000281201 SST Chap 11-25 The Coefficient of Determination SSR Regression Sum of Squares r SST Total Sum of Squares 2 Measures the proportion of variation in Y that is explained by the independent variable X in the regression model © 2002 Prentice-Hall, Inc. Chap 11-26 Venn Diagrams and Explanatory Power of Regression r 2 Sales Sizes © 2002 Prentice-Hall, Inc. SSR SSR SSE Chap 11-27 Coefficients of Determination (r 2) and Correlation (r) Y r2 = 1, r = +1 Y r2 = 1, r = -1 ^=b +b X Y i ^=b +b X Y i 0 1 i 0 X Yr2 = .8, r = +0.9 X © 2002 Prentice-Hall, Inc. X Y ^=b +b X Y i 0 1 i 1 i r2 = 0, r = 0 ^=b +b X Y i 0 1 i X Chap 11-28 Standard Error of Estimate n SYX SSE n2 i 1 Y Yˆi 2 n2 The standard deviation of the variation of observations around the regression line © 2002 Prentice-Hall, Inc. Chap 11-29 Measures of Variation: Produce Store Example Excel Output for Produce Stores R e g r e ssi o n S ta ti sti c s M u lt ip le R 0 .9 7 0 5 5 7 2 R S q u a re 0 .9 4 1 9 8 1 2 9 A d ju s t e d R S q u a re 0 .9 3 0 3 7 7 5 4 S t a n d a rd E rro r 6 1 1 .7 5 1 5 1 7 O b s e r va t i o n s 7 Syx r2 = .94 94% of the variation in annual sales can be explained by the variability in the size of the store as measured by square footage © 2002 Prentice-Hall, Inc. Chap 11-30 Linear Regression Assumptions Normality Y values are normally distributed for each X Probability distribution of error is normal 2. Homoscedasticity (Constant Variance) 3. Independence of Errors © 2002 Prentice-Hall, Inc. Chap 11-31 Variation of Errors around the Regression Line f(e) • Y values are normally distributed around the regression line. • For each X value, the “spread” or variance around the regression line is the same. Y X2 X1 X © 2002 Prentice-Hall, Inc. Sample Regression Line Chap 11-32 Residual Analysis Purposes Examine linearity Evaluate violations of assumptions Graphical Analysis of Residuals Plot residuals vs. Xi , Yi and time © 2002 Prentice-Hall, Inc. Chap 11-33 Residual Analysis for Linearity Y Y X e X X e X Not Linear © 2002 Prentice-Hall, Inc. Linear Chap 11-34 Studentized Residual X X X X 2 SRi SYX ei 1 hi where 1 hi n i n i 1 2 i Residual divided by its standard error Standardized residual adjusted for the distance from the average X value Allow us to normalize the magnitude of the residuals in units reflecting the variation around the regression line © 2002 Prentice-Hall, Inc. Chap 11-35 Residual Analysis for Homoscedasticity Y Y X SR X SR X Heteroscedasticity © 2002 Prentice-Hall, Inc. X Homoscedasticity Chap 11-36 Residual Analysis:Excel Output for Produce Stores Example Observation 1 2 3 4 5 6 7 Excel Output Predicted Y 4202.344417 3928.803824 5822.775103 9894.664688 3557.14541 4918.90184 3588.364717 Residuals -521.3444173 -533.8038245 830.2248971 -351.6646882 -239.1454103 644.0981603 171.6352829 Residual Plot 0 1000 © 2002 Prentice-Hall, Inc. 2000 3000 4000 Square Feet 5000 6000 Chap 11-37 Residual Analysis for Independence The Durbin-Watson Statistic Used when data is collected over time to detect autocorrelation (residuals in one time period are related to residuals in another period) Measures violation of independence assumption n D 2 ( e e ) i i 1 i2 n e i 1 © 2002 Prentice-Hall, Inc. 2 i Should be close to 2. If not, examine the model for autocorrelation. Chap 11-38 Durbin-Watson Statistic in PHStat PHStat | regression | simple linear regression … Check the box for Durbin-Watson Statistic © 2002 Prentice-Hall, Inc. Chap 11-39 Obtaining the Critical Values of Durbin-Watson Statistic Table 13.4 Finding critical values of Durbin-Watson Statistic 5 p=1 p=2 n dL dU dL dU 15 1.08 1.36 .95 1.54 16 1.10 1.37 .98 1.54 © 2002 Prentice-Hall, Inc. Chap 11-40 Using the Durbin-Watson Statistic H0 : H1 No autocorrelation (error terms are independent) : There is autocorrelation (error terms are not independent) Reject H0 (positive autocorrelation) 0 © 2002 Prentice-Hall, Inc. dL Inconclusive Accept H0 (no autocorrelatin) dU 2 4-dU Reject H0 (negative autocorrelation) 4-dL 4 Chap 11-41 Residual Analysis for Independence Graphical Approach Not Independent e Independent e Time Cyclical Pattern Time No Particular Pattern Residual is plotted against time to detect any autocorrelation © 2002 Prentice-Hall, Inc. Chap 11-42 Inference about the Slope: t Test t test for a population slope Null and alternative hypotheses Is there a linear dependency of Y on X ? H0: 1 = 0 H1: 1 0 (no linear dependency) (linear dependency) Test statistic b1 1 t where Sb1 Sb1 d. f . n 2 © 2002 Prentice-Hall, Inc. SYX n (X i 1 i X) 2 Chap 11-43 Example: Produce Store Data for Seven Stores: Store Square Feet Annual Sales ($000) 1,726 1,542 2,816 5,555 1,292 2,208 1,313 3,681 3,395 6,653 9,543 3,318 5,563 3,760 1 2 3 4 5 6 7 © 2002 Prentice-Hall, Inc. Estimated Regression Equation: Yi = 1636.415 +1.487Xi The slope of this model is 1.487. Is square footage of the store affecting its annual sales? Chap 11-44 Inferences about the Slope: t Test Example Test Statistic: H0: 1 = 0 From Excel Printout b Sb1 t H1: 1 0 1 Coefficients Standard Error t Stat P-value .05 Intercept 1636.4147 451.4953 3.6244 0.01515 df 7 - 2 = 5 Footage 1.4866 0.1650 9.0099 0.00028 Critical Value(s): Reject .025 Reject .025 -2.5706 0 2.5706 © 2002 Prentice-Hall, Inc. Decision: Reject H0 t Conclusion: There is evidence that square footage affects annual sales. Chap 11-45 Inferences about the Slope: Confidence Interval Example Confidence Interval Estimate of the Slope: b1 tn2 Sb1 Excel Printout for Produce Stores L o w er 95% I n te r c e p t U p p er 95% 475.810926 2797.01853 X V a r i a b l e 11 . 0 6 2 4 9 0 3 7 1.91077694 At 95% level of confidence, the confidence interval for the slope is (1.062, 1.911). Does not include 0. Conclusion: There is a significant linear dependency of annual sales on the size of the store. © 2002 Prentice-Hall, Inc. Chap 11-46 Inferences about the Slope: F Test F Test for a population slope Null and alternative hypotheses Is there a linear dependency of Y on X ? H0: 1 = 0 H1: 1 0 (No Linear Dependency) (Linear Dependency) Test statistic SSR 1 F SSE n 2 Numerator d.f.=1, denominator d.f.=n-2 © 2002 Prentice-Hall, Inc. Chap 11-47 Relationship between a t Test and an F Test Null and alternative hypotheses H0: 1 = 0 H1: 1 0 t n2 © 2002 Prentice-Hall, Inc. 2 (No linear dependency) (Linear dependency) F1,n 2 Chap 11-48 Inferences about the Slope: F Test Example H0: 1 = 0 H1: 1 0 .05 numerator df = 1 denominator df 7 - 2 = 5 Test Statistic: From Excel Printout ANOVA df Regression Residual Total 1 5 6 Reject = .05 0 © 2002 Prentice-Hall, Inc. 6.61 F1,n2 SS MS F Significance F 30380456.12 30380456.12 81.179 0.000281 1871199.595 374239.919 32251655.71 Decision: Reject H0 Conclusion: There is evidence that square footage affects annual sales. Chap 11-49 Purpose of Correlation Analysis Correlation analysis is used to measure strength of association (linear relationship) between two numerical variables Only concerned with strength of the relationship No causal effect is implied © 2002 Prentice-Hall, Inc. Chap 11-50 Purpose of Correlation Analysis (continued) Population correlation coefficient (Rho) is used to measure the strength between the variables Sample correlation coefficient r is an estimate of and is used to measure the strength of the linear relationship in the sample observations © 2002 Prentice-Hall, Inc. Chap 11-51 Sample of Observations from Various r Values Y Y Y X r = -1 X r = -.6 Y © 2002 Prentice-Hall, Inc. X r=0 Y r = .6 X r=1 X Chap 11-52 Features of and r Unit free Range between -1 and 1 The closer to -1, the stronger the negative linear relationship The closer to 1, the stronger the positive linear relationship The closer to 0, the weaker the linear relationship © 2002 Prentice-Hall, Inc. Chap 11-53 Test for a Linear Relationship Hypotheses H0: = 0 (no correlation) H1: 0 (correlation) Test statistic t r where r n2 2 n r r2 © 2002 Prentice-Hall, Inc. X i 1 n X i 1 i i X Yi Y X 2 n Y Y i 1 2 i Chap 11-54 Example: Produce Stores From Excel Printout Is there any evidence of a linear relationship between the annual sales of a store and its square footage at .05 level of significance? © 2002 Prentice-Hall, Inc. r R e g r e ssi o n S ta ti sti c s M u lt ip le R R S q u a re 0 .9 7 0 5 5 7 2 0 .9 4 1 9 8 1 2 9 A d ju s t e d R S q u a re 0 . 9 3 0 3 7 7 5 4 S t a n d a rd E rro r O b s e rva t io n s 6 1 1 .7 5 1 5 1 7 7 H0: = 0 (No association) H1: 0 (Association) .05 df 7 - 2 = 5 Chap 11-55 Example: Produce Stores Solution r .9706 t 9.0099 2 1 .9420 r 5 n2 Critical Value(s): Reject .025 Reject .025 -2.5706 0 2.5706 © 2002 Prentice-Hall, Inc. Decision: Reject H0 Conclusion: There is evidence of a linear relationship at 5% level of significance The value of the t statistic is exactly the same as the t statistic value for test on the slope coefficient Chap 11-56 Estimation of Mean Values Confidence interval estimate for Y | X X : i The mean of Y given a particular Xi Standard error of the estimate Size of interval varies according to distance away from mean, X (Xi X ) 1 ˆ Yi tn2 SYX n n 2 (Xi X ) t value from table with df=n-2 © 2002 Prentice-Hall, Inc. 2 i 1 Chap 11-57 Prediction of Individual Values Prediction interval for individual response Yi at a particular Xi Addition of one increases width of interval from that for the mean of Y 1 ( Xi X ) ˆ Yi tn2 SYX 1 n n 2 ( Xi X ) 2 i 1 © 2002 Prentice-Hall, Inc. Chap 11-58 Interval Estimates for Different Values of X Y Confidence Interval for the mean of Y Prediction Interval for a individual Yi X © 2002 Prentice-Hall, Inc. X A given X Chap 11-59 Example: Produce Stores Data for seven stores: Store Square Feet Annual Sales ($000) 1 2 3 4 5 6 7 1,726 1,542 2,816 5,555 1,292 2,208 1,313 3,681 3,395 6,653 9,543 3,318 5,563 3,760 © 2002 Prentice-Hall, Inc. Predict the annual sales for a store with 2000 square feet. Regression Model Obtained: Yi = 1636.415 +1.487Xi Chap 11-60 Estimation of Mean Values: Example Confidence Interval Estimate for Y | X X i Find the 95% confidence interval for the average annual sales for a 2,000 square-foot store. Predicted Sales Yi = 1636.415 +1.487Xi = 4610.45 ($000) X = 2350.29 SYX = 611.75 1 ˆ Yi tn2 SYX n tn-2 = t5 = 2.5706 ( X i X )2 n 2 ( X X ) i 4610.45 612.66 i 1 © 2002 Prentice-Hall, Inc. Chap 11-61 Prediction Interval for Y : Example Prediction Interval for Individual Y Find the 95% prediction interval for the annual sales of a 2,000 square-foot store Predicted Sales Yi = 1636.415 +1.487Xi = 4610.45 ($000) X = 2350.29 Yˆi tn2 SYX SYX = 611.75 tn-2 = t5 = 2.5706 1 ( X i X )2 1 n 4610.45 1687.68 n 2 ( Xi X ) i 1 © 2002 Prentice-Hall, Inc. Chap 11-62 Estimation of Mean Values and Prediction of Individual Values in PHStat In excel, use PHStat | regression | simple linear regression … Check the “confidence and prediction interval for X=” box EXCEL spreadsheet of regression sales on footage © 2002 Prentice-Hall, Inc. Chap 11-63 Pitfalls of Regression Analysis Lacking an awareness of the assumptions underlying least-squares regression Not knowing how to evaluate the assumptions Not knowing the alternatives to least-squares regression if a particular assumption is violated Using a regression model without knowledge of the subject matter © 2002 Prentice-Hall, Inc. Chap 11-64 Strategies for Avoiding the Pitfalls of Regression Start with a scatter plot of X on Y to observe possible relationship Perform residual analysis to check the assumptions Use a histogram, stem-and-leaf display, box-and-whisker plot, or normal probability plot of the residuals to uncover possible non-normality © 2002 Prentice-Hall, Inc. Chap 11-65 Strategies for Avoiding the Pitfalls of Regression (continued) If there is violation of any assumption, use alternative methods (e.g.: least absolute deviation regression or least median of squares regression) to least-squares regression or alternative least-squares models (e.g.: Curvilinear or multiple regression) If there is no evidence of assumption violation, then test for the significance of the regression coefficients and construct confidence intervals and prediction intervals © 2002 Prentice-Hall, Inc. Chap 11-66 Chapter Summary Introduced types of regression models Discussed determining the simple linear regression equation Described measures of variation Addressed assumptions of regression and correlation Discussed residual analysis Addressed measuring autocorrelation © 2002 Prentice-Hall, Inc. Chap 11-67 Chapter Summary (continued) Described inference about the slope Discussed correlation -- measuring the strength of the association Addressed estimation of mean values and prediction of individual values Discussed possible pitfalls in regression and recommended a strategy to avoid them © 2002 Prentice-Hall, Inc. Chap 11-68