Supplemental Material
Cero, Witte, Jones-Farmer, Kistner (in review). Misspecification of correlated errors and SEM
estimates of reliability: A simulation study
Table S1. Population-level reliabilities for generation models used in Study 1
Six-Item Scales
12-Item Scales
Population
Zero Pop.
One Pop.
Zero Pop.
One Pop.
Scale
Correlated
Correlated
Correlated
Correlated
Strength
Errors
Errors
Errors
Errors
λ’s = .80
.914
.907
.955
.953
λ’s = .50
.667
.645
.800
.796
Congeneric
.731
.718
.845
.840
λ’s = .30
.372
.350
.543
.531
Note. Pop. = population. All population correlated errors were set to r = .30. Scales with all
loadings set to λ = .80 represent ‘strong scales,’ those with λ = .50 represent scales of ‘moderate
strength,’ congeneric scales have variable loadings with median loading set to λ = .50. ‘Weak
scales’ are represented by loadings set to λ = .30.
Table S2. Bias in populations with zero correlated errors by specification, scale strength,
and sample size.
Scale Strength
Six-Item Scales
12-Item Scales
Est. Model
n = 50
n = 150
n = 300
n = 50
n = 150
n = 300
λ’s = .80
Coefficient α
-.004
-.001
.000
-.002
.000
.000
Correct
-.002
-.001
.000
-.001
.000
.000
Over
-.001
.000
.000
.000
.000
.000
λ’s = .50
Coefficient α
-.021
-.010
-.003
-.009
-.004
-.001
Correct
-.008
-.001
.000
-.004
-.001
.000
Over
.001
.000
-.002
-.001
.000
-.001
Congeneric
Coefficient α
-.033
-.026
-.022
-.016
-.012
-.011
Correct
-.004
-.004
.002
-.004
-.002
.000
Over
.034
.000
.000
-.003
-.001
.000
λ’s = .30
Coefficient α
-.069
-.012
-.009
-.045
-.010
-.005
Correct
.239
.103
.010
-.030
-.009
-.003
Over
.269
.093
.015
-.040
-.010
-.005
Note. Under-specified models are not included because it is not possible to under-specify a
correlated error for a population that includes none. Scales with all loadings set to λ = .80
represent ‘strong scales,’ those with λ = .50 represent scales of ‘moderate strength,’
congeneric scales have variable loadings with median loading set to λ = .50. ‘Weak scales’
are represented by loadings set to λ = .30. Labels within each scale strength refer to the
estimated model specification (e.g., ‘over’ denotes an over-specified correlated error).
Table S3. Standard errors in populations with zero correlated errors by specification, scale
strength, and sample size.
Scale Strength
Six-Item Scales
12-Item Scales
Est. Model n = 50
n = 150
n = 300
n = 50
n = 150
n = 300
λ’s = .80
Coefficient α
.021
.012
.008
.010
.005
.004
Correct
.020
.011
.008
.011
.006
.004
Over
.020
.012
.008
.010
.006
.004
λ’s = .50
Coefficient α
.080
.045
.031
.046
.025
.017
Correct
.076
.042
.030
.044
.026
.017
Over
.085
.046
.032
.045
.025
.017
Congeneric
Coefficient α
.064
.037
.026
.111
.059
.039
Correct
.059
.034
.024
.119
.063
.041
Over
.216
.039
.027
.139
.064
.038
λ’s = .30
Coefficient α
.155
.081
.058
.037
.020
.014
Correct
.343
.416
.057
.035
.020
.014
Over
.351
.220
.070
.034
.018
.013
Note. Under-specified models are not included because it is not possible to under-specify a
correlated error for a population that includes none. Scales with all loadings set to λ = .80
represent ‘strong scales,’ those with λ = .50 represent scales of ‘moderate strength,’ congeneric
scales have variable loadings with median loading set to λ = .50. ‘Weak scales’ are represented by
loadings set to λ = .30. Labels within each scale strength refer to the estimated model specification
(e.g., ‘over’ denotes an over-specified correlated error).
Table S4. Confidence coverage in populations with zero correlated errors by specification, scale
strength, and sample size.
Scale Strength
Six-Item Scales
12-Item Scales
Est. Model
n = 50
n = 150
n = 300
n = 50
n = 150
n = 300
λ’s = .80
Alpha
.940
.922
.950
.944
.958
.946
Correct
.950
.939
.950
.937
.928
.942
Over
.936
.937
.953
.928
.948
.952
λ’s = .50
Alpha
.949
.949
.943
.940
.948
.958
Correct
.929
.947
.948
.940
.925
.950
Over
.922
.946
.947
.938
.948
.949
Congeneric
Alpha
.969
.949
.935
.956
.943
.916
Correct
.942
.936
.940
.936
.939
.945
Over
.905
.937
.944
.943
.964
.953
λ’s = .30
Alpha
.944
.953
.954
.942
.938
.956
Correct
.889
.937
.937
.926
.944
.937
Over
.865
.918
.951
.898
.928
.967
Note. Under-specified models are not included because it is not possible to under-specify a
correlated error for a population that includes none. Scales with all loadings set to λ = .80 represent
‘strong scales,’ those with λ = .50 represent scales of ‘moderate strength,’ congeneric scales have
variable loadings with median loading set to λ = .50. ‘Weak scales’ are represented by loadings set
to λ = .30. Labels within each scale strength refer to the estimated model specification (e.g., ‘over’
denotes an over-specified correlated error).
Table S5. Percentage of study 1 models converging in populations with zero correlated errors
by specification, scale strength, and sample size.
Scale Strength
Six-Items Scales
12-Item Scales
Est. Model
n = 50
n = 150
n = 300
n = 50
n = 150
n = 300
λ’s = .80
Coefficient α
100
100
100
100
100
100
Correct
100
100
100
100
100
100
Over
100
100
100
100
100
100
λ’s = .50
Coefficient α
100
100
100
100
100
100
Correct
99
100
100
100
100
100
Over
98
100
100
100
100
100
Congeneric
Coefficient α
100
100
100
100
100
100
Correct
100
100
100
100
100
100
Over
98
100
100
100
100
100
λ’s = .30
Coefficient α
100
100
100
100
100
100
Correct
73
95
99
93
100
100
Over
71
92
99
91
100
100
Note. Under-specified models are not included because it is not possible to under-specify a
correlated error for a population that includes none. Scales with all loadings set to λ = .80
represent ‘strong scales,’ those with λ = .50 represent scales of ‘moderate strength,’
congeneric scales have variable loadings with median loading set to λ = .50. ‘Weak scales’
are represented by loadings set to λ = .30. Labels within each scale strength refer to the
estimated model specification (e.g., ‘over’ denotes an over-specified correlated error).
Table S6. Fit statistics for six item scales in populations with zero correlated errors by specification, scale strength, and sample size
Scale Strength
Est. Model
λ's = .80
Correct
Over
λ's = .50
Correct
Over
Congeneric
Correct
Over
λ's = .30
Correct
Over
n = 50
% χ2 Rej. RMSEA SRMR
n = 150
% χ2 Rej. RMSEA SRMR
n = 300
% χ2 Rej. RMSEA SRMR
Model
df
χ2
9
8
9.93
8.64
.084
.068
.046
.045
.030
.028
9.25
8.35
.059
.066
.023
.024
.017
.016
9.22
8.07
.060
.063
.016
.015
.012
.011
9
8
9.25
8.29
.050
.054
.039
.040
.060
.057
8.9
8.14
.054
.053
.020
.022
.034
.033
8.88
7.92
.050
.047
.014
.015
.024
.023
9
8
9.86
8.64
.088
.072
.045
.044
.058
.055
9.24
8.2
.047
.058
.022
.022
.033
.031
9.39
8.16
.068
.053
.016
.016
.023
.022
9
8
8.12
6.86
.040
.018
.028
.023
.066
.062
8.31
7.5
.033
.032
.016
.019
.040
.038
8.67
7.69
.047
.040
.013
.014
.029
.027
χ2
χ2
Note. Fit statistics in this table represent the average value a given fit statistic achieved in a condition (i.e., over 1,000 simulations of the same type). All generation
models were free of any correlated errors. Scales with all loadings set to λ = .80 represent strong scales; those with λ = .50 represent scales of moderate strength.
Congeneric scales have variable loadings with median loading set to λ = .50. Weak scales are represented by loadings set to λ = .30. Labels within each scale
strength refer to the estimated model specification (e.g., ‘over’ denotes an over-specified correlated error). % χ2 Rej. = percentage of models with significant χ2
values in a condition - if all assumptions were met, this would approximate .05. RMSEA = Root Mean Square Error of Approximation. SRMR = Standardized Root
Mean Square Residual.
Table S7. Fit statistics for 12 item scales in populations with zero correlated errors by specification, scale strength, and sample size.
Scale Strength
Est. Model
λ's = .80
Correct
Over
λ's = .50
Correct
Over
Congeneric
Correct
Over
λ's = .30
Correct
Over
n = 50
% χ2 Rej. RMSEA SRMR
n = 150
% χ2 Rej. RMSEA SRMR
n = 300
% χ2 Rej. RMSEA SRMR
Model
df
χ2
54
53
62.84
61.22
.200
.189
.051
.049
.041
.040
56.27
55.28
.076
.076
.018
.018
.023
.023
55.26
54.64
.068
.063
.011
.012
.016
.016
54
53
62.32
60.86
.212
.180
.049
.048
.083
.082
56.12
55.39
.071
.077
.018
.018
.047
.047
55.36
54.73
.069
.069
.011
.012
.033
.033
54
53
61.73
61.67
.209
.214
.046
.050
.075
.076
55.90
55.29
.079
.071
.017
.018
.057
.057
55.64
54.47
.070
.073
.012
.012
.031
.030
54
53
60.00
58.41
.161
.133
.042
.041
.098
.097
56.29
56.11
.090
.091
.018
.020
.043
.043
55.18
53.25
.055
.049
.011
.010
.041
.040
χ2
χ2
Note. Fit statistics in this table represent the average value a given fit statistic achieved in a condition (i.e., over 1,000 simulations of the same type). All generation
models were free of any correlated errors. Scales with all loadings set to λ = .80 represent strong scales; those with λ = .50 represent scales of moderate strength.
Congeneric scales have variable loadings with median loading set to λ = .50. Weak scales are represented by loadings set to λ = .30. Labels within each scale
strength refer to the estimated model specification (e.g., ‘over’ denotes an over-specified correlated error). % χ2 Rej. = percentage of models with significant χ2
values in a condition - if all assumptions were met, this would approximate .05. RMSEA = Root Mean Square Error of Approximation. SRMR = Standardized Root
Mean Square Residual.
Table S8. Fit statistics for 12 item scales by sample size, scale strength, and model specification for study 1 (one population error
correlation).
Scale Strength Model
n = 50
n = 150
n = 300
df
Est. Model
χ2 % χ2 Rej. RMSEA SRMR
χ2 % χ2 Rej. RMSEA SRMR
χ2 % χ2 Rej. RMSEA
λ's = .80
Correct
53
61.37 .207
.049
.040
55.71 .093
.019
.023
53.92 .066
.011
Over
52
59.63 .185
.048
.040
54.85 .078
.019
.022
52.98 .051
.011
Under
54
67.00 .323
.062
.042
68.44 .365
.038
.025
79.21 .656
.037
λ's = .50
Correct
53
61.54 .207
.049
.082
55.45 .095
.018
.047
54.06 .066
.011
Over
52
59.68 .184
.048
.081
53.84 .072
.017
.046
53.50 .072
.012
Under
54
66.28 .301
.060
.085
67.16 .309
.036
.051
78.55 .662
.037
Congeneric
Correct
53
61.31 .198
.049
.075
55.62 .073
.018
.043
53.88 .056
.011
Over
52
59.84 .187
.048
.074
54.49 .087
.019
.042
53.21 .067
.011
Under
54
65.48 .280
.058
.077
65.67 .277
.033
.045
73.89 .531
.033
λ's = .30
Correct
53
58.64 .137
.041
.097
55.25 .087
.018
.057
54.39 .070
.012
Over
52
57.19 .130
.040
.096
53.93 .073
.017
.056
52.49 .066
.011
Under
54
63.51 .229
.052
.101
66.41 .316
.035
.062
75.79 .591
.034
SRMR
.016
.016
.019
.033
.033
.039
.030
.030
.034
.040
.039
.047
Note. Fit statistics in this table represent the average value a given fit statistic achieved in a condition (i.e., over 1,000 simulations of the same type). All generation
models included a single correlated error between the first and second items of the scale. Scales with all loadings set to λ = .80 represent strong scales; those with λ =
.50 represent scales of moderate strength. Congeneric scales have variable loadings with median loading set to λ = .50. Weak scales are represented by loadings set to
λ = .30. Labels within each scale strength refer to the estimated model specification (e.g., ‘over’ denotes an over-specified correlated error). % χ2 Rej. = percentage
of models with significant χ2 values in a condition - if all assumptions were met, this would approximate .05. RMSEA = Root Mean Square Error of Approximation.
SRMR = Standardized Root Mean Square Residual.
Table S9. Population reliabilities for generation models used in Study 2
Zero Pop.
One Pop.
Two Pop.
Three Pop.
Population
Correlated
Correlated
Correlated
Correlated
Model
Errors
Errors
Errors
Errors
λ’s = .80
.914
.907
.899
.891
λ’s = .50
.667
.645
.625
.606
Congeneric
.731
.718
.694
.677
Note. All over-specified models were estimated using populations that contained zero
correlated errors. All population correlated errors were set to r = .30. All scales contained
six items.
.90
.30
Panel A: Six Items
n = 50
n = 150
n = 300
Panel B: 12 Items
.70
.20
.50
.10
.30
.10
.00
-.10
-.10
-.30
-.20
-.50
Correct
Over
Under
Alpha
Correct
Over
Under
Alpha
Figure S1. Bias ± 1 standard error of reliability estimates for weakly defined scales (λ’s = .30) by model type.
Note, the height of each bar represents the average bias for the condition specified. The line through each bar
represents ± one standard error of the reliability estimate, with the center of the line located at the average bias.
Thus, the spread of the bar represents the average bias ± one standard error of the reliability estimate. Dashed
lines indicate that the standard error extends beyond the range of the graph. In such cases, the numeric values of
the standard errors are indicated near their respective dashed lines. The first three conditions on each graph are
based on CR estimates, and the labels on the horizontal axis refer to the different model specifications (g., ‘over’
denotes one over-specified correlated error). The fourth condition on each graph is based on α. All results are
from populations with one correlated error.