Methodological Variation in
Empirical Corporate Finance
Todd Mitton
14th March 2022
Background and Research Question
Background
• Selective publication of statistically significant results
•1) Researchers may selectively report desired significant results-- P- hacking
•2) Publishers may select statistically significant results by maintaining a bias toward
publishing papers that report significant findings.
Research Question
• Assess the impact of methodological variation on research in corporate
finance
•1) To what degree are different methodologies employed and accepted in the literature
•2) How methodological variation affects the statistical significance of coefficients
•3) To what degree can methodological variation enable statistically significant findings
•4) Discuss remedies for the excessive reporting of statistical significance
2
A Survey of Methodological Variation
Six of the most common categories of regressions in corporate finance:
• Profitability, firm value, leverage, investment, payouts, and cash holdings.
• In the Journal of Finance, Journal of Financial Economics, and Review of
Financial Studies from 2000 to 2018.
Variability in dependent variables
3
A Survey of Methodological Variation
Variability in dependent variables
• The correlations between alternative dependent variables are not
particularly high, with a median (average) across all correlations of 0.33
(0.38)
4
A Survey of Methodological Variation
Variability in dependent variables
Variability in control variables
Lack of consistency of control variable usage; lack of standardization in size
control usage.
5
A Survey of Methodological Variation
Variability in dependent variables
Variability in control variables
Variability in other methodological decisions
6
A Survey of Methodological Variation
Variability in dependent variables
Variability in control variables
Variability in other methodological decisions
Motivation for methodological decisions
• For many methodological decisions, a large majority articles provide no
reason for the decision.
7
A Survey of Methodological Variation
Variability in dependent variables
Variability in control variables
Variability in other methodological decisions
Motivation for methodological decisions
Most common methodology
1) Use the most common dependent variable.
2) Include all control variables that are used in a majority of regressions.
3) Use log(total assets) as the size control.
4) Include all industries in the sample.
5) Do not log the explanatory variable.
6) Use the contemporaneous explanatory variable.
7) Winsorize non-indicator variables at the 1st/99th percentiles.
8) Do not log the dependent variable.
9) Use the end-of year denominator on flow/stock dependent variables.
10) when controlling for industry, use two-digit SIC dummies.
8
The impact of Methodology on Statistical Significance
Changes in t-statistics
Examine how the t-statistic on β changes when changing methods
• Purely random explanatory variables
• Random variable U as the t-statistic on β using the most common methodology, and
the random variable V as the t-statistic on β using the same explanatory variable
but an alternative method.
• Random variable W =V −U; is the difference in the t-statistic on β when using an
alternative method compared to the most common methodology
• U and V each having the standard normal distribution
[0, 1.13]
•When U and V are identical, σUV = 1, then σW is 0 and W is 0;
•When U and V are independent σUV = 1, σW is 2, the mean absolute deviation (MAD) of W is 1.13
9
The impact of Methodology on Statistical Significance
Changes in t-statistics
10
The impact of Methodology on Statistical Significance
Cumulative effects of methodological flexibility
11
Proposed Remedies
Robustness checks
12
Proposed Remedies
Specification checks
perform regressions with all possible combinations of methods and report in
graphical form the distribution of results obtained.
13
Proposed Remedies
Robustness checks
Specification checks
Economic significance
14