Approaches to testing statistical
significance of interactions
Jane E. Miller, PhD
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Overview
• Review: Testing statistical significance of individual
model terms
• The TEST statement
• All-interaction-term dummies model specification
• Introduction to post-hoc probing of interaction
effects
– Simple slopes calculations for compound coefficients
– Changing the reference category
• Presenting the results of statistical tests of
interaction pattern
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Estimated coefficients from an OLS model of birth weight in grams
Model A: Without
Model B: With
interactions
interactions
ttβ
β
statistic
statistic
Main effects terms
Race (ref. = non-Hisp. white)
Non-Hispanic Black (NHB)
–172.6** –9.86
Mexican American (MA)
–23.1 –1.02
Mother’s ed. (ref. = >HS)
Less than high school (<HS)
–55.5** –2.88
High school graduate (=HS)
–53.9** –3.64
Interactions: race & education
NHB_<HS
MA_<HS
NHB_=HS
MA_=HS
F-statistic
94.08
Degrees of freedom (df)
9
–168.1** –5.66
–104.2** –2.16
–54.2** –2.35
–62.0** –3.77
–38.5 –0.88
99.4 1.72
18.4 0.47
93.7 1.49
65.59
13
Statistical significance of βs on
individual interaction terms
• Statistical significance of coefficients on each of the
interaction terms is assessed as for any other
independent variable in a multivariate regression
model
• In the example from the previous slide, none of the
βs on the interaction terms between race/ethnicity
and mother’s education achieve statistical
significance as assessed by their t-statistics
– E.g., βNHB_<HS = –38.5, with a t-statistic of –0.88
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
What don’t inferential statistics for
individual terms in a model tell us?
• Based on the separate test statistics for each of the
individual main effect and interaction s alone cannot
assess statistical significance of differences in predicted
birth weight
– For example, for non-Hispanic blacks born to mothers with < HS
compared to non-Hispanic whites born to mothers with > HS
(the reference category)
– across racial/ethnic groups within the < HS group
– across education levels among non-Hispanic blacks
• Remember: each of these comparisons involves
comparing values calculated from more than one 
 <HS + NHB + <HS_NHB
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Substantive question behind the interaction model:
“Does race modify the association between education
and birth weight?”
• The bar for each
race/education combination
involves the sum of the
intercept and one to three
other coefficients
• t-tests for individual βs
won’t tell us about
statistical significance of
differences in those sums
Non-Hispanic white
Mexican-American
Predicted birth weight (grams)
Non-Hispanic black
3,450
3,400
3,350
3,300
3,250
3,200
3,150
3,100
3,050
3,000
2,950
<HS
=HS
>HS
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Testing differences between s
• To formally test statistical significance of differences
between coefficients, e.g., H0: βj = βk, calculate the
test statistic:
– Divide the difference between the estimated coefficients
(j − i) by the standard error of the difference
– Compare the value of the test statistic against the critical
value with one degree of freedom
• See podcast on testing statistical significance of
differences across coefficients
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Standard error of the difference
• The standard error of the difference is calculated:
Square root
√[var(j) + 2 × cov(j, k) + var(k) ]
– var(j) and var(k) are the variances of j and k,
respectively
– cov(j, k) is the covariance between j and k
• The complete variance-covariance matrix for
regression coefficients can be requested as part of
the output
• The variance of each coefficient can be calculated
from its standard error (s.e.): var(j) = [s.e.(j)]2
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
TEST statement
• The test statistics for each β tests whether it is
statistically significantly different from the reference
category
• To test other contrasts among categories, request the
test statistic for equality of coefficients for pairs of βs
– E.g., to test whether predicted birth weight is statistically
significantly different for infants born to mothers with < HS
than for those born to mothers with = HS education
• Specify “TEST <HS = =HS” in your SAS syntax
• Compare the value of that test statistic against the critical value
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Evaluating whether interaction
specification could be simplified
• For a specification that involves several main effects
and interaction terms, might be able to simplify the
specification if some terms can be omitted
• E.g., for a three-category variable, might it be
possible to
– Combine one of the modeled categories with the
reference category?
– Combine the two modeled categories with one another?
• Use standard approaches for comparing fit of models
with different specifications based on model GOF
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Respecify the model with simplified
classifications of categorical variable
• Create a dummy variable for ≤HS
– Combines former categories of < HS and = HS
– Specification will compare those two education levels
together against the reference category (> HS)
• Compare the overall fit of models
– With two separate dummies for <HS and =HS
– With one dummy for ≤HS
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Caveat about combining categories
• Only combine categories for which it makes
substantive sense to do so. E.g.,
– <HS and >HS aren’t adjacent ordinal categories, so you
would NOT combine them with one another to compare
against =HS
– For some research questions, you could combine nonHispanic blacks with Mexican-Americans because both are
considered racial/ethnic minority groups in the United
States
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Simplifying the interaction terms
• If there is no statistically significant difference in
model GOF for
– the model with collapsed education main effects
– the model with detailed education classification
• Could estimate a model with simplified interaction
terms:
o ≤HS_NHB instead of <HS_NHB and =HS_NHB
o ≤HS_MA instead of <HS_MA and =HS_MA
• Compare the fit of that model against the fit of the
model with detailed education by race interactions.
• Choose parsimonious model based on model GOF
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Model specification with individual
dummies for all interaction combinations
• An alternative for categorical-by-categorical
interactions is to
– Create a dummy variable for each interaction
combination except the reference category
– Estimate a model with all of those dummies but
NOT main effects terms for the variables involved
in the interaction
– Then request a formal inferential test of
differences among pairs of interaction terms
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Example: Race by education interaction
Mother’s educational attainment
< HS
= HS
> HS
Non-Hispanic black
<HS_NHB
=HS_NHB
>HS_NHB
Mexican-American
<HS_MA
=HS_NHW
>HS_MA
<HS_NHW
=HS_MA
Reference
category
Race
Non-Hispanic white
• The all-interaction-term specification for a model of birth
weight can be written:
y = β0 + β1<HS_NHB + β2<HS_NHW + β3<HS_MA + β4=HS_NHB +
β5HS_NHW + β6=HS_MA + β7>HS_NHB + β8>HS_MA + βiXi,
• No main effects terms needed because these eight dummies
distinguish among all nine possible combinations of race and
education
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Example: Testing differences across groups
other than the reference category
• The test statistics for β<HS_NHB and β<HS_MA each test
whether predicted birth weight is statistically
significantly different for that group compared to
> HS & NHW (the reference category)
• To contrast other combinations of race and
education, request the test statistic for equality of βs
– E.g., to test whether predicted birth weight is statistically
significantly different for non-Hispanic black with < HS than
for Mexican American infants of mothers with < HS
• In SAS: “TEST <HS_NHB = <HS_MA”
• Compare the value of that test statistic against the critical value
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Presenting results
of statistical tests for interactions
• Much of the work for testing statistical significance of
interactions should be done behind the scenes
• Create a table of is, standard errors, and model
goodness-of-fit statistics
• Supplement it with a table or chart reporting the
predicted values of the dependent variable for
pertinent values of the IVs in the interaction,
calculated from the is
– Use symbols to denote which values are statistically
significantly different from one another
Chart to present results of statistical
significance testing of interactions
Non-Hispanic white
* denotes statistically
Mexican-American
Non-Hispanic black
Predicted birth weight (grams)
3,450
3,400
†
£
¥
3,350
3,300
3,250
*
†
*
†
£
*
£
*
†
3,200
3,150
*
†
¥
*
¥
3,100
*
3,050
¥
3,000
2,950
<HS
=HS
>HS
significantly different at p < 0.05
from non-Hispanic white > HS
† denotes statistically significantly
different at p < 0.05 from nonHispanic black < HS
£ denotes statistically significantly
different at p < 0.05 from nonHispanic black = HS
¥ denotes statistically significantly
different at p < 0.05 from
Mexican-American = HS
Summary
• In models using main effects and interaction terms,
calculating the overall shape of an interaction requires
summing several βs
– Tests of the βs don’t address statistical significance of differences in
the overall interaction pattern
• Approaches to addressing statistical significance of
interactions include
– Comparing goodness-of-fit of models with and without interactions
– Using an all-interaction-dummy specification and test of difference
across coefficients
– Using simple slopes techniques for post-hoc comparisons
– Changing the reference category to test different contrasts
Separate podcasts
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Suggested resources
• Miller, J. E. 2013. The Chicago Guide to Writing about
Multivariate Analysis, 2nd Edition. University of Chicago Press,
chapters 11, 15, and 16.
• Cohen, Jacob, Patricia Cohen, Stephen G. West, and Leona S.
Aiken. 2003. Applied Multiple Regression/Correlation Analysis
for the Behavioral Sciences, 3rd Edition. Florence, KY:
Routledge, chapters 8 and 9.
• Figueiras, Adolfo, Jose Maria Domenech-Massons, and
Carmen Cadarso. 1998. Regression Models: Calculating the
Confidence Interval of Effects in the Presence of Interactions.
Statistics in Medicine 17: 2099–2105.
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Suggested online resources
• Podcasts on
– Testing statistical significance of differences
between coefficients
– Testing whether a multivariate specification can
be simplified
– Calculating the overall shape of an interaction
from regression coefficients
– Conducting post-hoc tests of compound
coefficients using simple slopes
– Using alternative reference categories to test
statistical significance of an interaction
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Suggested exercises: Simplifying an
interaction specification
• Estimate an OLS model with an interaction between
a three-category independent variable (IV1) and a
two-category independent variable (IV2)
• Test whether the βs on the main effects terms for the
two included categories of IV1 are statistically
significantly different
– From the reference category
– From each other
• Consider those results in conjunction with
theoretical criteria to decide whether it makes sense
to combine categories
Exercise: Simplifying an interaction, cont.
• Re-specify the model combining the two IV1
categories
• Compare GOF with your initial model
• Calculate revised interaction terms with the
simplified IV1 specification
• Estimate a model with simplified interaction terms
• Compare GOF with the more detailed interaction
specification
Suggested exercises: All-interactiondummies specification
• For an interaction between a three-category
independent variable (IV1) and a two-category
independent variable (IV2), create dummy variables
for each possible combination of IV1 and IV2 except
the reference category
• Specify a model with those terms
– Remember NOT to include main effects terms
• Use the TEST statement to test statistical significance
of differences in βs
– Across categories of IV1 within strata defined by IV2
– Across categories of IV2 within strata defined by IV1
Contact information
Jane E. Miller, PhD
jmiller@ifh.rutgers.edu
Online materials available at
http://press.uchicago.edu/books/miller/multivariate/index.html
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.