Testing factorial invariance in
multilevel data: A Monte Carlo
study
Eun Sook Kim
Oi-man Kwok
Myeongsun Yoon
The purpose of the present study

Investigate the influence of data dependency
on the test of factorial invariance in multilevel
data.

Comparisons between multilevel SEM (the modelbased or the designed-based multiple-group
multilevel CFA) and multiple-group CFA
Measurement invariance studies in
multilevel data

Multilevel SEM



The groups of interest are treated as random samples.
Good for measurement invariance testing in individual
and organizational levels.
Multiple-group CFA


The number of groups is small or relatively finite (e.g.,
<20 or so).
Can be utilized to test measurement invariance over
groups at the organizational level.
Multilevel measurement model

A conventional measurement model

A multilevel measurement model
In a multilevel measurement model

The variance of the factor

The variance of residual

The variance of observed scores
Examine measurement invariance in
multilevel data

The general multilevel CFA model
Examine measurement invariance in
multilevel data

Test at the between level
Study 1

Test at the within level
Study 2
Data generation
Design


Noninvariance in the organizational (study 1)
and the individual level (study 2).
Manipulated factors




The number of clusters (CN):30, 50, 80 and 160
Cluster size (CS): 10 and 20
The size of ICC: .09, .20, and .33
The total sample size was between 300 and
3,200, and 1,000 replications were in a
condition.

Data dependency on group membership:

Intraclass correlation (ICC)
Analysis





Tested a configural and a metric models.
The LRT was used to test factorial invariance
in multiple-group CFA .
The Satorra-Bentler scaled chi-squared test
was used in multiple-group multilevel CFA.
Examined power and type I error.
Conducted ANOVA to examine the influence
of 3 manipulated factors on power and type I
error.
Results (Study 1)
Low admissible rates.
Analysis in study 2

Examined factorial invariance at the individual
level using the design-based multilevel CFA,
instead of the model-based one, due to the
limitation of the SEM software.
Results (study 2)
Limitations and conclusions



The explanations of the discrepant findings
between study 1 and study 2 might be
confounded by the type of analysis.
It is necessary to take account of the data
dependency in multilevel data.
A substantially large number of clusters is
required in model-based multiple-group
multilevel modeling.
Comments

Information of the accuracy of estimated
parameters is not available.


MLR provides accurate factor loadings but
underestimates the residual variances and standard
errors at the between level. Having more clusters
or higher ICC does not effectively compensate for
this.
Groups of interest might occur at different
levels simultaneously, and how to model
them?
Comments (Cont.)


It is not clear how the model was constrained.
This study seemed to constrain the variance of
the latent factor to be equal across groups,
which is not practical and reasonable in real
data.
The large effect size in the factorial invariance
with a large sample size yielded very high
power, and it is recommended reducing the
effect size to medium.
Comments (Cont.)

Contrast to the findings of Jones-Farmer
(2010), the inflated type I error occurred in
CFA even though data dependency was
considered small. Further examinations are
needed to understand the controversial
findings.
Future studies

Further examinations in



The issue of sample sizes using other
estimation methods


Scalar and error variance models
Cross-level invariance
In MLR, the min. sample size of the between level
is 100. How about in FIML?
The assessment of model fit in ML SEM.
Future studies (Cont.)




Examine more than 1 items with different
factor loadings.
Implement purification in this study.
Conduct equal means for the latent factor or
equal intercepts of indicators to examine
factorial invariance.
Examine scalar invariance to see if the
findings are similar to the findings of French
and Finch (2010).
Future studies (Cont.)


The possibility to develop a new indicator in
SEM, such as adding person fit (as IRT
approaches do) to SEM approaches, instead of
model fit only.
Conduct the model-based multilevel CFA
using other software to overcome the
limitation in current SEM software.