Supplemental Materials
Friends and Foes in the Lexicon: Homophone Naming in Aphasia
by E. L. Middleton et al., 2014, Journal of Experimental Psychology: Learning,
Memory, and Cognition
http://dx.doi.org/10.1037/a0037778
Appendix S1
Statistical Model Fitting Procedure for Pairwise Mixed Models of Accuracy
In this appendix, we lay out our strategy for model criticism. Fitting a generalized
linear mixed model is a nontrivial problem in statistical computing, and it is important to
check a number of assumptions to ensure that the model is sensible. In particular, we
must be confident that the resulting estimates are numerically stable, that the solution is
at a regular point in the parameter space (defined below), and that it fits the data.
We considered a total of four specific models, two for each group, Stage 1 and
Stage 2. These models had the same basic structure:
1. Fixed effects for the relevant condition compared to homophones (LF versus
homophones for the Stage 1 group, HF versus homophones for the Stage 2
group).
2. Fixed effect for session (not of substantive interest).
3. Random intercepts for participant and for item (i.e., simplified model).
4. Random slope correlated with random intercept for condition on participant
(removed in the simplified model).
To foreshadow, several diagnostics favored the simplified models, which are
reported in the paper. To summarize the outcomes of model assessment: (a) the
parameters for the random intercepts models were stable across different number of
quadrature nodes, with higher quadrature providing more accurate estimates; (b)
information regarding the correlation among the random intercept and slope in each
model indicated the random slope models were overparameterized for the data; (c) model
selection using the Akaike information criterion (AIC) favored the simplified models; (d)
goodness of fit as revealed by plots of deviance residuals indicated the simplified models
captured the data well.
The parameter estimates, associated confidence intervals, random effects standard
deviations, and correlations along with the log-likelihoods, model df, and AIC are
reported in Table SA1 below. The confidence intervals for the random effects standard
deviations and correlations are computed on a transformed metric that is more accurate,
and then back transformed. This is why they are asymmetric.
Estimation was done using the QR decomposition for random effects
implemented in Stata 13.2 (StataCorp, 2013), which is believed to be more accurate and
numerically stable in the presence of small random effects terms than alternative
parameterizations. In practice, estimation using the QR decomposition helps ensure that
regularity conditions for likelihood estimation are met. These conditions ensure that the
log-likelihood has properties that justify the use of the chi-square statistic. The most
relevant condition is that the parameter estimates cannot be on the boundary of the
parameter space. A boundary solution involves estimated random effect variances that
equal 0 or random effect correlations that are 1. These are not the interior of the
parameter space, which means that statistics based on the log-likelihood, such as the
likelihood ratio or AIC should be interpreted with caution.
The random intercepts only model agrees with the model fit in R using lmer 1.1-6
(Bates, 2014), which uses a different numerical approach. R output is reported in the
paper, but it agrees with Stata output. This is important to note because a well-behaved
model should give essentially the same answer given any reasonable numerical
estimation. Varying the estimation methods and software provides useful evidence that
the model is a good one (McCulloch, Searle, & Neuhaus, 2009).
The Laplace approximation was used for numerical quadrature. We attempted to
increase the number of quadrature nodes to improve the accuracy. This was possible with
the simplified models but failed for the random slopes models. Further suggestive that the
simplified models appeared to function well, increasing the number of quadrature nodes
resulted in the same estimated parameters.
In the Stage 2 group, the random effects variances were overall quite modest, and
arguments could be made for using ordinary logistic regression for this model. However,
to be consistent with all models in the manuscript, random intercepts were retained for
participants and items. The random slopes models for both groups appeared to be
overparameterized because the correlations between random effects had extraordinarily
wide confidence intervals, essentially spanning 1 to 1, and the random slope estimate in
the Stage 1 group had a very wide confidence interval. In other words, in each model the
data contained very little information about variation by participant. By AIC, inclusion of
the random slope led to a more poorly fit model, although, as mentioned above, it needs
to be used only heuristically due to the boundary solution.
To inspect goodness of fit to the data, we examined the deviance residuals for
each of the simplified models. Deviance residuals are designed to have mean 0 and
variance 1 and represent an individual observation’s contribution to the model misfit. The
Stage 1 group’s model’s deviance residuals behaved quite well. This is unsurprising
given the fact that there was much more item- and participant-heterogeneity observed,
which generally improves the residuals of a logistic regression. The Stage 2 model’s
deviance residuals were more structured, but there were no clear outliers. Given that the
random effects were estimated to be quite modest for this group, the patterns observed
suggested modeling with ordinary logistic regression would not have been inappropriate.
Neither set of residuals is indicative of a poorly performing model. Quantile plots of the
deviance residuals are shown below as Figures SA1 and SA2.
References
Bates, D. (2014). The lme4 Package. Retrieved from http://cran.rproject.org/web/packages/lme4/lme4.pdf
McCulloch, C., Searle, S. R., & Neuhaus, J. M. (2009). Generalized, linear, and mixed
models (2nd ed.). New York, NY: Wiley-Interscience.
Pawitan, Y. (2001). In all likelihood. New York, NY: Oxford University Press.
StataCorp. (2013). Stata Version 13.2 reference manual. College Station, TX: Author.
Table SA1
Parameter Estimates, Confidence Intervals (in Parentheses), Log-Likelihood, Model df,
and AIC for the Simplified and Maximal Pairwise Models of Accuracy
Parameter
Random
intercepts
Random slope
Stage 2 group
Stage 2 group
Session
HF
Random
intercepts
Random slopes
Stage 1 group
Stage 1 group
0.47
0.51
0.83
0.35
(0.084,0.847)
(0.18,0.84)
(0.15,1.52)
(0.20,0.90)
—
—
0.94
0.81
(0.035,1.84)
(0.014,1.60)
—
—
0.47
0.43
(0.90,0.048)
(.93,0.068)
0.36
0.68
1.41
1.33
(0.13,0.94)
(0.47,0.98)
(1.00,1.98)
(0.99,1.77)
0.16
0.14
0.86
0.96
(0.022,1.12)
(0.021,0.95)
(0.42,1.78)
(0.48, 1.94)
SD
(HF:participant)
—
—
—
0.19
SD
(LF:participant)
—
Cor
(HF:participant)
—
LF
SD (item)
SD (participant)
(0.0016,22.94)
0.21
—
—
—
0.24
(0.025,1.81)
—
(0.99,0.98)
Cor
—
1
—
—
(LF:participant)
logL
df
AIC
(1,1)
341.43
483.94
227.47
337.38
5
7
5
7
692.86
981.88
464.94
688.76
Figure SA1. Deviance residuals for Stage-1 participant group (simplified model).
Figure SA2. Deviance residuals for Stage-2 participant group (simplified model).