Lecture 7
Model Checking for Linear
Mixed Models for
Longitudinal Data
Ziad Taib
Biostatistics, AZ
MV, CTH
May 2011
Name, department
1
Date
Outline of lecture 7
1. Model checking for the linear model
2. Model checking for the linear mixed models for
longitudinal data
Name, department
2
Date
1. Introductio to model checking
 The process of statistical analysis might take the form
Select
Model Class
Data
Summarize
Some Models
Conclusions
Stop
 In the above process, however, even after a careful selection of
model class, the data themselves may indicate that the particular
model is unsuitable. Thus, it seems to be reasonable to introduce
model checking to the original process. The new process of
statistical analysis is
Select
Model Class
Some Models
Conclusions
Summarize
Data
Model Checking
Stop
A statistical model, whether of the fixed-effects or mixedeffects variety, represents how you think your data were
generated. Following model specification and estimation,
it is of interest to explore the model-data agreement by
raising questions such as
 Does the model-data agreement support the model
assumptions?
 Should model components be refined, and if so,
which components? For example, should regressors
be added or removed, and is the covariation of the
observations modeled properly?
 Are the results sensitive to model and/or data? Are
individual data points or groups of cases particularly
influential on the analysis?
In classical linear models, this examination of modeldata agreement has traditionally revolved around
 the informal, graphical examination of estimates of
model errors to assess the quality of distributional
assumptions: residual analysis
 overall measures of goodness-of-fit
 the quantitative assessment of the inter-relationship
of model components; for example, collinearity
diagnostics
 the qualitative and quantitative assessment of
influence of cases on the analysis: influence
analysis.
The inadequacy indicated by model checking could
thus take two forms and is part of the technique of
model checking.
1. The detection of systematic discrepancies. It may be e.g. that
the data as a whole show some systematic departure from the
fitted model. An example of this type is informal checking using
residuals.
2. The detection of isolated discrepancies. It may be that a few
data values are discrepant from the rest. This can be done using
measures of leverage or measures of influence
2. The linear model
Yij  X i   Zi bi   ij , i  1, ,2,...,n
Model checking for linear models uses mainly the following statistics:
 The fitted values:
E[Y ]   ; ˆ n1  X n p ˆ p1
 The mean residual sum of square:
 The residual:
ˆ  Y  ˆ


t
ˆ
Y  X Y  Xˆ
2
s 
n p

Residual checking
0
-1
-2
-3
Residuals
1
2
3
 Plot residuals against mean
0
20
40
60
ˆ  Y  ˆ
80
100
3. Model checking in linear mixed
models
Influence diagnostics
Linear models for uncorrelated data have well established
measures to assess the influence of one or more
observations on the analysis. For such models, closedform update expressions allow efficient computations
without refitting the model. When similar notions of
statistical influence are applied to mixed models, things are
more complicated.
Removing data points affects fixed effects and covariance
parameter estimates. Update formulas for “leave-one-out”
estimates typically fail to account for changes in
covariance parameters. Moreover, in repeated measures
or longitudinal studies, one is often interested in
multivariate influence, rather than the impact of
isolated points.
Checks for Isolated Departures from the
Model
 Cook’s distance can be used to assess the
influence of observation i, by considering the parameter
estimate without the contribution from the i’th observation:

ˆ  ˆ  X X ˆ  ˆ  ˆ  ˆ  Vaˆr ˆ  ˆ  ˆ 
C 

t
(i )
(i )
i
(i )
ps2

Yˆ  Yˆ  Yˆ  Yˆ  Yˆ  Yˆ


(i )
(i )
2
ps
(i )
p
t
(i )
1
t
t
2
ps
2
…, n
, i  1,
3. Model checking in linear mixed models
 The PROC MIXED uses three fit criteria:
 -2 times the residual log-likelihood (-2RLL),
 Akaike’s Information Criterion (AIC) (Akaike, 1974) or its
corrected version for finite samples (AICC) (Hurvich &
Tsai, 1989),
 Bayesian Information Criterion (BIC) (Schwarz, 1978).
 These criteria are indices of relative goodness-of-fit and may
be used to compare models with different covariance
structures and the same fixed effects (Bozdogan, 1987;
Keselman, Algina, Kowalchuk, & Wolfinger, 1998; Littell et
al., 1996; Wolfinger, 1993, 1996, 1997).
3.1 Model selection: likelihood
 When choosing between different models we want to be
able to decide which model fits our data best. If the models
compared are nested within each other it is possible to do
a likelihood ratio test where the test statistic has an
approximate distribution. The test statistic for the likelihood
statistic is,
2
2log(L1 )  log(L2 ) ~  DF
 where DF are the degrees of freedom defined as the
difference in number of parameters for the models and L1
and L2 are the likelihoods for the first and second model
respectively.
Name, department
16 Date

If the two models compared are not nested within each other but
contain the same number of parameters they can be compared
directly by looking at the log likelihood and
the model with the biggest likelihood value wins

If the two models are not nested and contain different number of
parameters the likelihood cannot be used directly. It is still possible to
compare these models with some of the methods described below.
 The bigger the likelihood is the better the model fits data and we use this
when we compare different models

Name, department
17 Date
Since we are interested in getting as simple models as possible we also
have to consider the number of parameters in the structures. A model
with many parameters usually fits data better than a model with less
number of parameters
Information.
3.2 Model selection: Information criteria
 It is possible to compute so called information criteria and there are
different ways to do that and here we show two of these, Akaikes
information criteria (AIC) and Bayesian information criteria (BIC). The
idea with both of these is to punish models with many parameters in
some way. We present the information criteria the way they are
computed in SAS.
 The AIC value is computed as below where q is the number of parameters
in the covariance structure. Formulated this way, a smaller value of AIC
indicates a better model.
AIC  2 LL  2q
 The BIC value is computed using the following formula where q is the
number of parameters in the covariance structure and n is the number of
effective observations, which means the number of individuals. Like for
AIC a smaller value of BIC is better than a larger.
BIC  2 LL 
Name, department
18 Date
q
2 log(n)
AGE 1
AGE 2
AGE 3
AGE 4
AGE 5
0.8938
0.8921
1.0098
1.2798
1.6207
0.7632
1.5286
1.4229
1.7209
2.1504
0.7163
0.7718
2.2236
2.8210
3.3660
0.6600
0.6790
0.9229
4.2017
5.2374
0.6106
0.6195
0.8040
0.9100
7.8834
Note: Variances on diagonal, covariances above
diagonal, correlations below diagonal
Structure
CS
AR(1)
UN
Cov Par
2
2
15
-2RLL
2438.60
2150.40
1857.50
AIC
2442.60
2154.40
1887.50
BIC
2448.50
2160.30
1931.60
Chi-Square
387.94
676.17
969.12
Pr>Chi-Square
<.0001
<.0001
<.0001
Model fit
 It is possible to define a goodness of fit measure similar to, R, the
coefficient of determination often used for linear regression. It is called
Concordance Correlation coefficient (CCC). Unlike the AIC or the BIC,
the CCC does not compare the model at hand to other models, thus it
does not require that other models be fitted.
 For simple linear regression we have
3.3 Residuals for linear mixed models
 In model selection, we accept the model with the best
likelihood value in relation to the number of parameters
but we still do not know if the model chosen is a good
model or even if the normality assumption we have made
is realistic.To check this we can look at two types of plots
for our data,
 normal plots
 residual plots to check
1. normality of the residuals and the random effects
2. if the residuals seem to have a constant variance
3. outliers
Name, department
22 Date
 The predicted values and residuals can be computed in
many different ways. Some of these are accounted for in
the in what following.
 Recall that the general linear mixed model is of the form:
Yij  X i   Zi bi   ij , i  1, ,2,...,n
 Assuming we have ML estimates of the fixed parameters
and EB predictions of the random parameters
Name, department
23 Date
 We can “estimate” the residuals according to the
following three methods:
1. The marginal residual
2. The conditional residual
3. The best linear unbiased predictor
 Each type of residual is useful to evaluate some of the
assumptions of the model.
Name, department
24 Date
Can be used to assess linearity of the response w.r.t
explanatory variables. A random behviour around zero
is a sign of linearity.
Name, department
25 Date
 Plots of /s against Y can be used to assess
homogeneity of the variances as well as normality.
Name, department
26
Date
 Plots of bi against subject indices can be used to find
outliers. Plot elements in bi to assess normality and check
for outliers.
Name, department
3.4 An example
To illustrate the above procedures, we analyze data from a
study conducted at the School of Dentistry of the University
of Sao Paulo, Brazil, designed to compare a low cost
toothbrush (monoblock) with a conventional toothbrush
with respect to the maintenance of the capacity to remove
bacterial plaque under daily use. The data in the table
correspond to bacterial plaque indices obtained from 32
children aged 4 to 6 before and after tooth brushing in four
evaluation sessions.
Name, department
28 Date
Following Singer et al. (2004) who analyze a different data
set from the same study, we considered fitting models of
the form
Name, department
29 Date
i subject
d session
j type of toothbrush
Three possible models
(3.1)
(3.2)
(3.3)
post
pre
The model reduction procedure can be based on likelihood
ratio tests (LRT) and AIC and BIC:
 The LRT p-values corresponding to the reduction of (3.1) to (3.2)
and of (3.2) to (3.3) were, respectively 0.3420 and 0.1623.
 The AIC (BIC) for the three models are
(3.1)
(3.2)
(3.3)
AIC
95.0
102.8
105.6
BIC
68.6
86.7
92.1
Based on these results, we adopt (3.3) to illustrate the use
of the proposed diagnostic procedures.
To check for the linearity of effects, we plot the marginal
residuals versus the logarithms of the pretreatment
bacterial plaque index in Figure 2. The figure supports the
regression model for the transformed response (log of the
bacterial plaque index)
Name, department
33 Date
Figure 2
 The figure suggests something is wrong with
observations #12.2 and #29.4
Name, department
35 Date
References
1. Atkinson, C. A. (1985). Plots, transformations, and regression: an
introduction to graphical methods of diagnostic regression analysis.
Oxford University Press, Oxford.
2. Cook, R. D. and Weisberg, S. (1982). Residuals and influence
regression. Chapman & Hall, New York.
3. Cox, D. R. and Snell, E. J. (1968). A general definition of residuals
(with discussion). Journal Royal Statistical Society B 30, 248–275.
4. Fei, Y. and Pan, J. (2003). Influence assessments for longitudinal
data in linear mixed models. In 18th international workshop on
Statistical Modelling. G. Verbeke, G. Molenberghs, M. Aerts and S.
Fieuws (eds.). Leuven: Belgium, 143–148.
5. Grady, J. J. and Helms, R.W. (1995). Model selection techniques for
the covariance matrix for incomplete longitudinal data. Statistics in
Medicine 14, 1397–1416.
Name, department
38
Date
References
6. Jiang, J. (2001). Goodness-of-fit tests for mixed model diagnostics.
The Annals of Statistics 29, 1137–1164.
7. Lange, N. and Ryan, L. (1989). Assessing normality in random effects
models. The Annals of Statistics 17, 624– 642.
8. Longford, N. T. (2001). Simulation-based diagnostics in randomcoefficient models. Journal of the Royal Statistical Society A 164,
259–273.
9. Nobre, J. S. and Singer, J. M. (2006). Fixed and random effects
leverage for influence analysis in linear mixed models. (Submitted;
http://www.ime.usp.br/jmsinger).
10. Oman, S. D. (1995). Checking the assumptions in mixed-model
analysis of variance: a residual analysis approach. Computational
Statistics and Data Analysis 20, 309–330.
References
11. Verbeke, G. and Lesaffre, E. (1997). The effect of misspecifying the
random-effects distributions in linear mixed models for longitudinal
data. Computational Statistics and Data Analysis 23, 541–556.
12. Waternaux, C., Laird, N. M., and Ware, J. H. (1989). Methods for
analysis of longitudinal data: blood-lead concentrations and cognitive
development. Journal of the American Statistical Association 84, 33–
41.
13. Weiss, R. E. and Lazaro, C. G. (1992). Residual plots for repeated
measures. Statistics in Medicine 11, 115–124.
14. Wolfinger, R. (1993). Covariance structure selection in general mixed
models. Communications in Statistics-Simulation 22, 1079–1106.
Any Questions
Name, department
41 Date
?