advertisement

Before dropping so-called insignificant variables from a regression, we should be careful to consider (1) what we mean by “insignificant” (2) why the variable was included in the first place and (3) what potential costs arise if the variable is excluded. To practitioners, insignificance arises if the p-value is less than some “critical” level, like .05. In probability theory, this is the level of Type I error; it is the risk of falsely rejecting the null hypothesis (Ho: the coefficient on the variable is insignificantly different from zero). Lower p-values mean lower probabilities of committing Type I errors. In a capital murder trial, for example, the judge typically instructs the jury to consider the evidence in such a way as to minimize this risk – that is, to minimize the risk of convicting an innocent defendant (the null is that the defendant is not guilty). In statistical tests, the 5% level sets a limit on the amount of risk we are willing to take. In five out of one-hundred random samples, we’d reject the null incorrectly, i.e., five percent of the time. If we set the p-value lower, say at one percent, then we are making it even harder to reject the null. That is, we are making it harder to commit a Type I error. So, the first question is, what p-value should we select? Well, there’s no answer to that but suppose it is 5%. Then in a standard t-test, we’d conclude that the variable has no explanatory power in the model if the p-value is 0.05001? Probably not. Rather, we’d conclude that the probability of making a Type I error (rejecting the null of zero when we shouldn’t) is slightly higher than we’d otherwise like. But in no way does the decision to reject or not, mean that the variable is somehow uninformative. That’s lesson number one. Now consider why the variable is in there to begin with. Suppose I have a scientific experiment for which the “theory” states that X and Y cause Z. I collect a sample of data on all three and regress Z on X and Y jointly. Suppose the estimated coefficient on X is deemed insignificant because its p-value is 0.0499. Theoretically, this can’t happen (it’s like rejecting gravity because the apple that fell from the tree onto Isaac Newton’s head didn’t fall fast enough). X is supposed to cause Z. But in random samples, there is measurement error and coefficients are necessarily measured imprecisely. Noisy data cause standard errors to rise and t-statistics to fall. So, should we reject X? Or should we say that the contribution X has in explaining Z is measured imprecisely? That’s lesson number two. X was included because it should be there. Even rejecting X’s contribution has risk – in this case 0.0499 risk of Type I error. Because this number is less than 0.05 is not a legitimate reason to exclude X. Which finally brings us to our third consideration on what potential costs there are to excluding a so-called “insignificant” variable. Suppose we go ahead and exclude X. In the statistics literature, X is formally an erroneously excluded variable and so the equation governing Z is now misspecified. Forget that though. Think instead of where X is. X now resides in the error term (it is now part of the unobserved, albeit random, part of Z. The error term in any regression is the composite of all excluded unobserved influences on Z. We usually assume this term is zero on average and uncorrelated with any of the included regressors. Now, it includes X too. Well, if X is correlated with Y, and there is no reason to think otherwise, then excluding X will make the estimate on the coefficient on Y biased. And that’s lesson number three. In sum, the decision to say a variable is insignificant is a subjective choice and practitioners falsely think p-values are somehow objectively applied. They are not. Secondly, so-called insignificance is a statement about precision in estimation and not about causality. So there is absolutely no reason to drop variables whose coefficients are somehow deemed insignificant. Thirdly, dropped variables are now part of the error term and to the extent these dropped variables are correlated with any included regressors then coefficients on those regressors will be estimated with bias. This is a much more serious problem than leaving in regressors with imprecisely estimated coefficients.