Lecture 8: Hypothesis Testing (Chapter 7.1–7.2, 7.4) Distribution of Estimators (Chapter 5.1–5.2, Chapter 6.4) Copyright © 2006 Pearson Addison-Wesley. All rights reserved. Agenda for Today • Hypothesis Testing (Chapter 7.1) • Distribution of Estimators (Chapter 5.2) • Estimating s 2 (Chapter 5.1, Chapter 6.4) • t-tests (Chapter 7.2) • P-values (Chapter 7.2) • Power (Chapter 7.2) Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-2 What Sorts of Hypotheses to Test? • To test a hypothesis, we first need to specify our “null hypothesis” precisely, in terms of the parameters of our regression model. We refer to this “null hypothesis” as H0. • We also need to specify our “alternative hypothesis,” Ha , in terms of our regression parameters. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-3 What Sorts of Hypotheses to Test? (cont.) • Claim: The marginal propensity to consume is greater than 0.70 : Ci b0 b1Incomei i • Conduct a one-sided test of the null hypothesis • H0 : b1 > 0.70 against the alternative, • Ha : b1 = 0.70 Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-4 What Sorts of Hypotheses to Test? (cont.) • Claim: The marginal propensity to consume equals the average propensity to consume: Ci b0 b1Incomei i , with b0 0 • Conduct a two-sided test of • H0 : b0 = 0 against the alternative, • Ha : b0 ≠0 Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-5 What Sorts of Hypotheses to Test? (cont.) • The CAPM model from finance says that the E(excess return on portfolio k) b ·(Excess return on market portfolio) • Regress E(excess return on portfolio k) b0 b1 (excess return on market portfolio) for a particular mutual fund, using data over time. Test H0 : b0 > 0. • If b0 > 0, the fund performs better than expected, said early analysts. If b0 < 0, the fund performs less well than expected. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-6 What Sorts of Hypotheses to Test? (cont.) • H0 : b0 > 0 • Ha : b0 = 0 • What if we run our regression and find bˆ 0 0.012 • Can we reject the null hypothesis? What if bˆ 0 0.012? Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-7 Hypothesis Testing: Errors • In our CAPM example, we are testing – H0 : b0 > 0, against the alternative – Ha : b0 = 0 • We can make 2 kinds of mistakes. – Type I Error: We reject the null hypothesis when the null hypothesis is “true.” – Type II Error: We fail to reject the null hypothesis when the null hypothesis is “false.” Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-8 Hypothesis Testing: Errors (cont.) • Type I Error: Reject the null hypothesis when it is true. • Type II Error: Fail to reject the null hypothesis when it is false. • We need a rule for deciding when to reject a null hypothesis. To make a rule with a lower probability of Type I error, we have to have a higher probability of Type II error. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-9 Hypothesis Testing: Errors (cont.) • Type I Error: Reject the null hypothesis when it is true. • Type II Error: Fail to reject the null hypothesis when it is false. • In practice, we build rules to have a low probability of a Type I error. Null hypotheses are “innocent until proven guilty beyond a reasonable doubt.” Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-10 Hypothesis Testing: Errors (cont.) • Type I Error: Reject the null hypothesis when it is true. • Type II Error: Fail to reject the null hypothesis when it is false. • We do NOT ask whether the null hypothesis is more likely than the alternative hypothesis. • We DO ask whether we can build a compelling case to reject the null hypothesis. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-11 Hypothesis Testing • What constitutes a compelling case to reject the null hypothesis? • If the null hypothesis were true, would we be extremely surprised to see the data that we see? Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-12 Hypothesis Testing: Errors • In our CAPM example H 0 : b0 0 H a : b0 0 • What if we run our regression and find bˆ 0 0.012? • Could a reasonable jury reject the null hypothesis if the estimate is “just a little lower” than 0? Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-13 Hypothesis Testing: Errors (cont.) • Type I Error: Reject the null hypothesis when it is true. • Type II Error: Fail to reject the null hypothesis when it is false. • In our CAPM example, our null hypothesis is b0 > 0. Can we use our data to amass overwhelming evidence that this null hypothesis is false? Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-14 Hypothesis Testing: Errors (cont.) • Note: if we “fail to reject” the null, it does NOT mean we can “accept” the null hypothesis. • “Failing to reject” means the null has “reasonable doubt.” • The null hypothesis could still be fairly unlikely, just not overwhelmingly unlikely. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-15 Hypothesis Testing: Strategy • Our Strategy: Look for a Contradiction. • Assume the null hypothesis is true. • Calculate the probability that we see the data, assuming the null hypothesis is true. • Reject the null hypothesis if this probability is just too darn low. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-16 How Should We Proceed? 1. Ask how our estimates of b0 and b are distributed if the null hypothesis is true. 2. Determine a test statistic. 3. Settle upon a critical region to reject the null hypothesis if the probability of seeing our data is too low. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-17 How Should We Proceed? (cont.) • The key tool we need is the probability of seeing our data if the null hypothesis is true. • We need to know the distribution of our estimators. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-18 Distribution of a Linear Estimator (from Chapter 5.2) If the true value of b 0 0, what is the probability that we observe data with estimated intercept bˆ ? 0 Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-19 Distribution of a Linear Estimator (cont.) • Perhaps the most common hypothesis test is H0 : b = 0 against Ha : b ≠ 0 • This hypothesis tests whether a variable has any effect on Y • We will begin by calculating the variance of our estimator for the coefficient on X1 Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-20 Hypothesis Testing • Add to the Gauss–Markov Assumptions • The disturbances are normally distributed i ~ N(0, s ) 2 Yi ~ N(b Xi , s ) 2 Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-21 DGP Assumptions Yi b0 b1 X1i b2 X 2i bk X ki i E( i ) 0 Var( i ) s 2 Cov( i , j ) 0, for i j Each explanator is fixed across samples i ~ N (0, s ) 2 Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-22 How Should We Proceed? • Ask how our guesses of b0 and b1 are distributed. • Since the Yi are distributed normally, all linear estimators are, too. bˆ0 ~ N ( b 0 ,?) bˆ1 ~ N ( b1 ,?) What are the variances of bˆ0 and bˆ1? Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-23 What is the Variance of bˆ 1? Var(wiYi ) Var(wiYi ) 0 wi2Var(Yi ) s 2 wi2 Let xi Xi - X and yi Yi - Y Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-24 What is the Variance of bˆ 1? (cont.) Var(wiYi ) s 2 wi2 ˆ b1 xiyi x i wi 2 xi xi 2 xi 2 2 2 2 ˆ Var(b1) s wi s ( ) 2 xi s 2 ( x xi 2 i Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 2 2 ) s2 x i 2 8-25 What is the Variance of bˆ 1? (cont.) Var ( bˆ 1) s 2 x i 2 where xi Xi X Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-26 What is the Variance of bˆ 1? (cont.) Var ( bˆ 1) s2 x 2 i Thus... bˆ 1 ~ N ( b 1, s2 x 2 ) i Suppose we have as our null hypothesis H 0 : b 1 b 1* Under the null hypothesis bˆ 1 ~ N ( b 1 , * s2 x 2 ) i Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-27 Distribution of a Linear Estimator • We have a formula for the distribution of our estimator. However, this formula is not in a very convenient form. We would really like a formula that gives a distribution for which we can look up the probabilities in a common table. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-28 Test Statistics • A “test statistic” is a statistic: 1. Readily calculated from the data 2. Whose distribution is known (under the null hypothesis) • Using a test statistic, we can compute the probability of observing the data given the null hypothesis. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-29 Test Statistics (cont.) bˆ 1 ~ N ( b 1 , * s2 x 2 ) i If we subtract the mean and divide by the standard error, we can transform a Normal Distribution into a Standard Normal Distribution. Z bˆ 1 - b 1 * s 2 xi Copyright © 2006 Pearson Addison-Wesley. All rights reserved. ~ N (0,1) 2 8-30 Test Statistics (cont.) We can easily look up the probability that we observe a given value of Z bˆ 1 - b 1 * s2 x 2 i on a Standard Normal table. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-31 Test Statistics (cont.) Z bˆ 1 - b 1 * s 2 xi ~ N (0,1) 2 Suppose we want to test H 0 : b1 0.70, against the alternative H a : b1 0.70 We could replace b1 * with 0.70 and calculate Z If Z -1.64, then we know there is less than a 5% chance we would observe this data if b1 really were greater than 0.70 Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-32 Estimating s2 (from Chapter 5.1, Chapter 6.4) One Problem: We cannot observe bˆ 1 - b 1* because we cannot observe s 2 . s2 x 2 i Solution: Estimate s . 2 Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-33 Estimating s2 (cont.) • We need to estimate the variance of the error terms, i Yi - b0 - b1 X1i - ...- bk Xki • Problem: we do not observe i directly. • Another Problem: we do not know b0…bk, so we cannot calculate i either. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-34 Estimating s2 (cont.) • We need to estimate the variance of the error terms, i Yi - b0 - b1 X1i - ...- bk Xki • We can proxy for the error terms using the residuals. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-35 Estimating s2 (cont.) ˆ ˆ ˆ ei Yi b 0 b1 X 1i ... b k X ki Y Yˆ i i Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-36 Estimating s2 (cont.) ei Yi bˆ0 bˆ1 X 1i ... bˆk X ki Y Yˆ i i • Once we have an estimate of the error term, we can calculate an estimate of the variance of the error term. We need to make a “degrees of freedom” correction. 1 2 s e i n k 1 2 Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-37 Estimating s2 (cont.) s2 Recall Var ( bˆ 1) xi 2 Thus... bˆ 1 ~ N ( b 1, s2 x 2 ) i Suppose we have as our null hypothesis H 0 : b 1 b 1* Under the null hypothesis bˆ 1 ~ N ( b 1 , * s2 x 2 ) i Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-38 Estimating s2 (cont.) Suppose we have as our null hypothesis H 0 : b1 b * 1 Under the null hypothesis 2 s * ˆ b1 ~ N b1 , 2 x i Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-39 Estimating s2 (cont.) Plug in our estimate for s : 2 2 s * ˆ b1 ~ N b1 , 2 x i 2 e * i ˆ b1 ~ N b1 , 2 ( n k 1) x i Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-40 Standard Error (from Chapter 5.2) • Remember, the standard deviation of the distribution of our estimator is called the “standard error.” • The smaller the standard error, the more closely your estimates will tend to fall to the mean of the distribution. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-41 Standard Error (from Chapter 5.2) • If your estimate is unbiased, a low standard error implies that your estimate is probably “close” to the true parameter value. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-42 t-statistic (from Chapter 7.2) tˆ bˆ b * s2 2 x i ~ tn k 1 • Because we need to estimate the standard error, the t-statistic is NOT distributed as a Standard Normal. Instead, it is distributed according to the t-distribution. • The t-distribution depends on n-k-1. For large n-k-1, the t-distribution closely resembles the Standard Normal. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-43 t-statistic (cont.) tˆ * ˆ b b sbˆ 2 ~ tn k 1 • Under the null hypothesis H0 : b1 = b1* t ~ tn-2 • In our earlier example, we could: 1. Replace b1* with 0.70 2. Compare t to the “critical value” for which the tn-2 distribution has .05 of its probability mass lying to the left, 3. There is less than a 5% chance of observing the data under the null if t < “critical value.” Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-44 Figure 7.1 Critical Regions for Two-Tailed and One-Tailed t-Statistics with a 0.05 Significance Level and 10 Degrees of Freedom Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-45 Significance Level • We can now calculate the probability of observing the data IF the null hypothesis is true. • We choose the maximum chance we are willing to risk that we accidentally commit a Type I Error (reject a null hypothesis when it is true). • This chance is called the “significance level.” Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-46 Significance Level (cont.) • We choose the probability we are willing to accept of a Type I Error. • This probability is the “Significance Level.” • The significance level gives operational meaning to how compelling a case we need to build. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-47 Significance Level (cont.) • The significance level denotes the chance of committing a Type I Error. • By historical convention, we usually reject a null hypothesis if we have less than a 5% chance of observing the data under the null hypothesis. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-48 Significance Level (cont.) • 5% is the conventional significance level. Analysts also often look at the 1% and 10% levels. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-49 Critical Region • We know the distribution of our test statistic under the null hypothesis. • We can calculate the values of the test statistic for which we would reject the null hypothesis (i.e., values that we would have less than a 5% chance of observing under the null hypothesis). Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-50 Critical Region (cont.) • We can calculate the values of the test statistic for which we would reject the null hypothesis. • These values are called the “critical region.” Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-51 Figure 7.1 Critical Regions for Two-Tailed and One-Tailed t-Statistics with a 0.05 Significance Level and 10 Degrees of Freedom Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-52 Critical Region • Regression packages routinely report estimated coefficients, their estimated standard errors, and the t-statistics associated with the null hypothesis that an individual coefficient is equal to zero. • Some programs also report a “p-value” for each estimated coefficient. • This reported p-value is the smallest significance level for a two sided test at which one would reject the null that the coefficient is zero. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-53 One-Sided, Two-Sided Tests • t-tests come in two flavors: 1-sided and 2-sided. • 2-sided tests are much more common: – H0 : b = b* – Ha : b ≠b* • 1-sided tests look at only one-side: – H0 : b > b* – Ha: b = b* Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-54 One-Sided, Two-Sided Tests (cont.) • The procedure for both 1-sided and 2-sided tests is very similar. • For either test, you construct the same t-statistic: * ˆ b b tˆ ˆ s.e.(b ) Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-55 One-Sided, Two-Sided Tests (cont.) • Once you have your t-statistic, you need to choose a “critical value.” The critical value is the boundary point for the critical region. You reject the null hypothesis if your t-statistic is greater in magnitude than the critical value. • The choice of critical value depends on the type of test you are running. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-56 Critical Value for 1-Sided Test • For a 1-sided test, you need a critical value such that a of the distribution of the estimator is greater than (or less than) the critical value. a is our significance level (for example, 5%). Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-57 Critical Value for 1-Sided Test (cont.) • In our CAPM example, we want to test: H 0 : b0 0 H a : b0 0 • We need a critical value t* such that a of the distribution of our estimator is less than t* Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-58 Figure 7.1 Critical Regions for Two-Tailed and One-Tailed t-Statistics with a 0.05 Significance Level and 10 Degrees of Freedom Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-59 Critical Value of a 1-Sided Test (cont.) • For a 5% significance level and a large sample size, t* = -1.64 • We reject the null hypothesis if: ˆt 1.64 Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-60 Critical Value for a 2-Sided test • For a 2-sided test, we need to spread our critical region over both tails. We need a critical value t* such that – a/2 of the distribution is to the right of t* – a/2 of the distribution is to the left of –t* • Summing both tails, a of the distribution is beyond either t* or -t* Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-61 Figure 7.1 Critical Regions for Two-Tailed and One-Tailed t-Statistics with a 0.05 Significance Level and 10 Degrees of Freedom Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-62 Critical Value for 2-Sided Test • For a large sample size, the critical value for a 2-sided test at the 5% level is 1.96 • You reject the null hypothesis if: tˆ 1.96 or tˆ 1.96 Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-63 P-values • The p-value is the smallest significance level for which you could reject the null hypothesis. • The smaller the p-value, the stricter the significance level at which you can reject the null hypothesis. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-64 P-values (cont.) • Many statistics packages automatically report the p-value for a two-sided test of the null hypothesis that a coefficient is 0 • If p < 0.05, then you could reject the null that b= 0 at a significance level of 0.05 • The coefficient “is significant at the 95% confidence level.” Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-65 Statistical Significance • A coefficient is “statistically significant at the 95% confidence level” if we could reject the null that b = 0 at the 5% significance level. • In economics, the word “significant” means “statistically significant” unless otherwise qualified. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-66 Performing Tests • How do we compute these test statistics using our software? Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-67 Power • Type I Error: reject a null hypothesis when it is true • Type II Error: fail to reject a null hypothesis when it is false • We have devised a procedure based on choosing the probability of a Type I Error. • What about Type II Errors? Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-68 Power (cont.) • The probability that our hypothesis test rejects a null hypothesis when it is false is called the Power of the test. • (1 – Power) is the probability of a Type II Error. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-69 Power (cont.) • If a test has a low probability of rejecting the null hypothesis when that hypothesis is false, we say that the test is “weak” or has “low power.” • The higher the standard error of our estimator, the weaker the test. • More efficient estimators allow for more powerful tests. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-70 Power (cont.) • Power depends on the particular Ha you are considering. The closer Ha is to H0, the harder it is to reject the null hypothesis. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-71 Figure 7.2 Distribution of bˆs for bs = -2, 0, and a Little Less Than 0 Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-72 Figure 7.3 Power Curves for Two-Tailed Tests of H0 : bs = 0 Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-73 Figure SA.12 The Distribution of the t-Statistic Given the Null Hypothesis is False and = + 5 Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-74 Figure SA.13 The t-Statistic’s Power When the Sample Size Grows Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-75 Review • To test a null hypothesis, we: – Assume the null hypothesis is true; – Calculate a test statistic, assuming the null hypothesis is true; – Reject the null hypothesis if we would be very unlikely to observe the test statistic under the null hypothesis. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-76 Six Steps to Hypothesis Testing 1. State the null and alternative hypotheses 2. Choose a test statistic (so far, we have learned the t-test) 3. Choose a significance level, the probability of a Type I Error (typically 5%) Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-77 Six Steps to Hypothesis Tests (cont.) 4. Find the critical region for the test (for a 2-sided t-test at the 5% level in large samples, the critical value is t*=1.96) 5. Calculate the test statistic * ˆ b b tˆ s.e.(bˆ ) 6. Reject the null hypothesis if the test statistic falls within the critical region Is tˆ t * or tˆ t *? Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 8-78