Borrowing the Strength of Unidimensional
Scaling to Produce Multidimensional
Educational Effectiveness Profiles
P R E S E N T A T I O N A T T H E 1 2 TH A N N U A L
MARYLAND ASSESSMENT CONFERENCE
COLLEGE PARK, MD
OCTOBER 18, 2012
JOSEPH A. MARTINEAU
JI ZENG
MICHIGAN DEPARTMENT OF EDUCATION
Background
2
 Prior research showing that using unidimensional measures of
multidimensional achievement constructs can distort value-added

Martineau, J. A. (2006). Distorting Value Added: The Use of Longitudinal,
Vertically Scaled Student Achievement Data for Value-Added Accountability.
Journal of Educational and Behavioral Statistics, 31(1), 35-62.

Construct irrelevant variance can become considerable in value-added
measures when a construct is multidimensional, but is modeled in valueadded as unidimensional.
Common misunderstanding is that if the multiple constructs are highly
correlated, value-added should not be distorted.
Correct understanding is that if value-added on the multiple constructs is
highly correlated, value-added should not be distorted


Background
3
 Prior research showing that the choice of dimension/domain within
construct changes value-added significantly

Lockwood, J.R et al. (2007). The Sensitivity of Value-Added Teacher Effect
Estimates to Different Mathematics Achievement Measures. Journal of
Educational Measurement, 44(1), 47-67.

Depending on choices made in value-added modeling, the correlation
between teacher value-added on Procedures and Problem Solving ranged
from 0.01 to 0.46.
This gives a surprisingly low correlation in value-added that indicates that at
least in this situation, one needs to be concerned about modeling valueadded in both dimensions rather than unidimensionally.
Only work I am aware of to date that has inspected inter-construct valueadded correlations.


Background
4
 Prior research showing that commonly used factor analytic techniques
underestimate the number of dimensions in a multidimensional
construct

Zeng, J. (2010) . Development of a Hybrid Method for Dimensionality
Identification Incorporating an Angle-Based Approach. Unpublished
doctoral dissertation, University of Michigan.

Common dimensionality identifications procedures make the unwarranted
assumption that all shared variance among indicator variables arise because
the indicator variables measure the same construct (shared variance can also
arise because the indicator variables are influenced by a common exogenous
variable)
Because of this unwarranted assumption, commonly used dimensionality
identification techniques underestimate the number of dimensions in a data
set.

Background
5
 Scaling constructs as multidimensional is a
difficult task




Multidimensional Item Response Theory (MIRT) is timeconsuming and costly to run
Replicating MIRT analyses can be challenging (there are multiple
subjective decision points along the way)
Identifying the number of dimensions in MIRT can be
challenging
Once the number of dimensions is identified, identifying which
items load in which dimensions in MIRT can also be challenging

The factor analysis techniques underlying MIRT are techniques for
data reduction, not dimension identification
Background
6
 Short of resolving the considerable difficulties in
analytically identifying dimensions within a construct (and
replicating such analyses), can another approach be used?
 Propose using/trusting content experts’ identifications of
dimensions within constructs (e.g., the divisions agreed
upon by the writers of content standards) as the best
currently available identification of dimensions, for
example…


Within English language proficiency, producing reading, writing,
listening, and speaking scales.
Within Mathematics, producing number & operations, algebra,
geometry, measurement, and data analysis/statistics scales.
Background
7
 However, separately scaling each dimension can also be difficult and
costly compared to running a traditional unidimensional IRT
calibration



Confirmatory MIRT
Bi-factor IRT model
Separate unidimensional calibration and year-to-year equating of each
dimension score
 Another option:




Unidimensionally calibrate the total score
Unidimensionally equate the total score from year to year
Use (fixed) item parameters from the unidimensional calibration to create
the multiple dimension scores as specified by content experts
Use of this method needs to be investigated
 Practical necessity for Smarter Balanced Assessment Consortium
Purpose
8
 Investigate the feasibility and validity of relying on
unidimensional total score calibration as a basis for
creating multidimensional profile scores…


For reporting multidimensional student achievement scores
For reporting multidimensional value-added measures
 Investigate the impact of separate versus fixed calibration
of multidimensional achievement scores in terms of impact
on…


Student achievement scores
Value-added scores
 …as compared to the impact of other common decisions in
scaling, outcome selection, and value-added modeling
Methods
9
 Decisions Modeled in the Analyses

Psychometric decisions



Choice of outcome metric


Choice of psychometric model
 1-PL vs. 3-PL
 PCM vs. GPCM
Estimation of sub-scores
 Separate calibration for each dimension vs. fixed calibration based on
unidimensional parameters
Which sub-score is modeled
Value-added modeling decisions


Inclusion of demographics in models
Number of pre-test covariates (for covariate adjustment models)
Methods
10
 Outcomes
 Correlations in student achievement metrics compared across each
psychometric choice and outcome choice
 Correlations in value-added modeling compared across each choice
 Classification consistency in value-added compared across each
choice for


Three-category classification decisions
 Based on confidence intervals around point-estimates placing
programs/schools into three categories: (1) above average, (2)
statistically indistinguishable from the average, and (3) below average
Four-category classification decisions
 Based on sorting programs’/schools’ point estimates into quartiles,
representing arbitrary cut points for classification
Methods
11
 Data
 Michigan English Language Proficiency Assessment (ELPA)
 Level III (Grades 3-5)
 3391 students each with 3 measurement occasions (10,173 total scores)
 Measures







Total
Reading
Writing
Listening
Speaking
(domain)
(domain)
(domain)
(domain)
Calibrated the ELPA as a unidimensional measure using both 1PL/Partial Credit Model and 3-PL/Generalized Partial Credit Model
Created domain scores both from fixed parameters from unidimensional
calibration and in separate calibrations for each domain
Methods
12
 Data




Michigan Educational Assessment Program (MEAP) Mathematics
Grades 7 and 8 (not on a vertical scale)
Over 110,000 students per grade
Measures
Total
 Number & Operations
 Algebra



(using items from the two domains)
(domain)
(domain)
Calibrated the MEAP Math tests as unidimensional measures using
both 1-PL and 3-PL models
Created domain scores both from fixed parameters from
unidimensional calibration and in separate calibrations for each
domain
Methods
13
 Value-added modeling the ELPA
 3-level
HLM nesting test occasion within student
within English language learner program to
obtain program value-added
 𝑦𝑖𝑗𝑘
= 𝜋0𝑗𝑘 + 𝜋1𝑗𝑘 𝑡𝑖𝑚𝑒𝑖𝑗𝑘 + 𝑒𝑖𝑗𝑘
 𝜋0𝑗𝑘
= 𝛽00𝑘 + β′0 𝐗𝑗𝑘 + 𝑟0𝑗𝑘
 𝜋1𝑗𝑘
= 𝛽10𝑘 + β′1 𝐗𝑗𝑘 + 𝑟1𝑗𝑘
 𝛽00𝑘
= 𝛾000 + 𝛄′00 𝐖𝑘 + 𝑢00𝑘
= 𝛾100 + 𝛄′10 𝐖𝑘 + 𝑢10𝑘
 𝛽10𝑘
Methods
14
 Value-added modeling the ELPA
 VAMs
were run in a fully-crossed design with…
 All
outcomes (R, W, L, S)
 PCM- and GPCM-calibrated outcomes
 Fixed and separately calibrated outcomes
 With and without demographics in the VAMs
 32
real-data applications across design
factors
Methods
15
 Value-added modeling MEAP mathematics
 2-level
HLM covarying grade-8 outcomes on
grade-7 outcomes with students nested within
schools
 𝑦𝑖𝑗𝑘
= 𝛽0𝑘 + 𝛽1𝑘 𝑦
 𝛽0𝑘
= 𝛾00 + 𝛄′0 𝐖𝑘 + 𝑢0𝑘
= 𝛾10 + 𝑢1𝑘
= 𝛾20 + 𝑢2𝑘
 𝛽1𝑘
 𝛽2𝑘
𝑖−1 𝑗𝑘
+ 𝛽2𝑘 𝑧
𝑖−1 𝑗𝑘
+ 𝛃′𝐗𝑗𝑘 + 𝐞𝑗𝑘
Methods
16
 Value-added modeling MEAP mathematics
 VAMs
were run in a fully-crossed design with…
 Both
outcomes (algebra and number & operations)
 1-PL and 3-PL calibrated outcomes
 Fixed and separately calibrated outcomes
 With and without demographics
 With either one or two pre-test covariates
 32
real-data applications across design factors
Results
17
ELPA
Results: ELPA Student-Level Outcomes
18
 Correlations across fixed vs. separate calibrations
Model choice
PCM
GPCM
Content Area
Reading
Writing
Listening
Speaking
Reading
Writing
Listening
Speaking
Correlation
0.997
0.995
0.997
1.000
0.997
0.997
0.994
1.000
Results: ELPA Student-Level Outcomes
19
 Correlations across model choice (PCM vs. GPCM)
Calibration choice Content Area
Reading
Writing
Fixed
Listening
Speaking
Reading
Writing
Separate
Listening
Speaking
Correlation
0.972
0.983
0.967
0.982
0.978
0.983
0.977
0.982
Results: ELPA Student-Level Outcomes
20
 Correlations across content areas
Model
choice
Calibration
choice
Fixed
PCM
Separate
Fixed
GPCM
Separate
Content
Area
Reading
Writing
Listening
Speaking
Reading
Writing
Listening
Speaking
Reading
Writing
Listening
Speaking
Reading
Writing
Listening
Speaking
Reading
-
Content Area
Writing Listening Speaking
0.636
0.627
0.371
0.537
0.385
0.368
0.622
0.626
0.373
0.519
0.375
0.365
0.655
0.662
0.402
0.559
0.407
0.405
0.639
0.648
0.395
0.543
0.400
0.394
-
Low to moderate
inter-dimension
correlations
However, Rasch
dimensionality
analysis from
WINSTEPS
identified the total
score as a
unidimensional
score
Results: ELPA Program District-Level Value-Added
Outcomes
21
 Impact of fixed versus separate calibration
Correlations
3-Category
Consistency
4-Category
Consistency
Content Area
Reading
Writing
Listening
Speaking
No Demos
PCM
GPCM
1.000
0.987
1.000
0.997
1.000
0.987
1.000
1.000
Content Area
Reading
Writing
Listening
Speaking
No Demos
PCM
GPCM
0.996
0.996
1.000
0.996
1.000
1.000
1.000
1.000
Content Area
Reading
Writing
Listening
Speaking
No Demos
PCM
GPCM
0.982
0.875
0.973
0.946
0.991
0.897
1.000
1.000
Demos
PCM
1.000
1.000
1.000
1.000
GPCM
0.992
0.997
0.987
1.000
min
max
mean
SD
0.987
1.000
0.997
0.005
GPCM
0.991
0.991
0.996
1.000
min
max
mean
SD
0.991
1.000
0.998
0.003
GPCM
0.902
0.946
0.906
1.000
min
max
mean
SD
0.875
1.000
0.961
0.043
Demos
PCM
1.000
1.000
1.000
1.000
Demos
PCM
0.982
0.982
0.991
1.000
Results: ELPA Program District-Level Value-Added
Outcomes
22
 Correlations between Listening and Reading VA
Reading
No Demos
Demos
Listening
No Demos
Fixed
Separate
Fixed
GPCM
Separate
Fixed
PCM
Separate
Fixed
GPCM
Separate
PCM
PCM
Fixed Separate
0.371 0.371
0.372 0.371
0.360 0.361
0.376 0.377
0.330 0.330
0.329 0.330
0.304 0.305
0.328 0.329
 Min = 0.228, Max = 0.397
 Mean = 0.322, SD = 0.037
GPCM
Fixed Separate
0.301 0.327
0.303 0.328
0.387 0.392
0.389 0.397
0.292 0.308
0.294 0.309
0.341 0.342
0.346 0.350
Demos
PCM
Fixed Separate
0.303 0.302
0.304 0.303
0.301 0.302
0.327 0.328
0.318 0.317
0.318 0.318
0.307 0.309
0.333 0.335
GPCM
Fixed Separate
0.228 0.245
0.230 0.247
0.316 0.321
0.320 0.329
0.261 0.275
0.263 0.277
0.329 0.332
0.332 0.339
Results: ELPA Program District-Level Value-Added
Outcomes
23
 Correlations between Listening and Writing VA
Writing
No Demos
Demos
Listening
No Demos
Fixed
Separate
Fixed
GPCM
Separate
Fixed
PCM
Separate
Fixed
GPCM
Separate
PCM
PCM
Fixed Separate
0.358 0.359
0.359 0.360
0.403 0.403
0.368 0.368
0.362 0.362
0.363 0.364
0.395 0.395
0.364 0.364
 Min = 0.342, Max = 0.420
 Mean = 0.373, SD = 0.019
GPCM
Fixed Separate
0.369 0.366
0.370 0.367
0.420 0.412
0.383 0.376
0.373 0.371
0.374 0.372
0.410 0.405
0.378 0.373
Demos
PCM
Fixed Separate
0.342 0.343
0.343 0.344
0.385 0.385
0.354 0.355
0.361 0.362
0.362 0.363
0.397 0.397
0.365 0.365
GPCM
Fixed Separate
0.353 0.353
0.354 0.354
0.401 0.396
0.370 0.364
0.372 0.371
0.374 0.372
0.412 0.407
0.379 0.374
Results: ELPA Program District-Level Value-Added
Outcomes
24
 Correlations between Listening and Speaking VA
Speaking
No Demos
Demos
Listening
No Demos
Fixed
Separate
Fixed
GPCM
Separate
Fixed
PCM
Separate
Fixed
GPCM
Separate
PCM
PCM
Fixed Separate
0.002 0.002
0.004 0.004
0.068 0.068
0.051 0.051
-0.005 -0.005
-0.004 -0.004
0.065 0.065
0.047 0.047
 Min = -0.005, Max = 0.108
 Mean = 0.046, SD = 0.035
GPCM
Fixed Separate
0.026 0.026
0.028 0.028
0.102 0.102
0.080 0.080
0.025 0.025
0.027 0.027
0.097 0.097
0.076 0.076
Demos
PCM
Fixed Separate
0.005 0.005
0.007 0.007
0.081 0.081
0.061 0.061
0.001 0.001
0.002 0.002
0.075 0.075
0.056 0.056
GPCM
Fixed Separate
0.032 0.032
0.033 0.033
0.108 0.108
0.086 0.086
0.028 0.028
0.029 0.029
0.101 0.101
0.080 0.080
Results: ELPA Program District-Level Value-Added
Outcomes
25
 Correlations between Reading and Writing VA
Writing
No Demos
Demos
Reading
No Demos
Fixed
Separate
Fixed
GPCM
Separate
Fixed
PCM
Separate
Fixed
GPCM
Separate
PCM
PCM
Fixed Separate
0.389 0.390
0.392 0.393
0.466 0.464
0.455 0.454
0.365 0.365
0.369 0.369
0.453 0.450
0.440 0.438
 Min = 0.335, Max = 0.491
 Mean = 0.412, SD = 0.047
GPCM
Fixed Separate
0.393 0.386
0.396 0.389
0.480 0.466
0.468 0.455
0.370 0.365
0.374 0.369
0.465 0.454
0.452 0.442
Demos
PCM
Fixed Separate
0.335 0.336
0.338 0.339
0.442 0.440
0.420 0.419
0.374 0.374
0.379 0.379
0.478 0.476
0.464 0.462
GPCM
Fixed Separate
0.341 0.338
0.344 0.341
0.455 0.443
0.432 0.422
0.379 0.372
0.384 0.377
0.491 0.477
0.476 0.461
Results: ELPA Program District-Level Value-Added
Outcomes
26
 Correlations between Reading and Speaking VA
Speaking
No Demos
Demos
Reading
No Demos
Fixed
Separate
Fixed
GPCM
Separate
Fixed
PCM
Separate
Fixed
GPCM
Separate
PCM
PCM
Fixed Separate
0.121 0.121
0.122 0.122
0.129 0.129
0.134 0.134
0.122 0.122
0.125 0.125
0.163 0.163
0.162 0.162
 Min = 0.121, Max = 0.205
 Mean = 0.151, SD = 0.026
GPCM
Fixed Separate
0.132 0.132
0.134 0.134
0.174 0.174
0.172 0.172
0.136 0.136
0.139 0.139
0.205 0.205
0.199 0.199
Demos
PCM
Fixed Separate
0.131 0.131
0.132 0.132
0.152 0.152
0.154 0.154
0.125 0.125
0.128 0.128
0.171 0.171
0.168 0.168
GPCM
Fixed Separate
0.136 0.136
0.138 0.138
0.179 0.179
0.177 0.177
0.134 0.134
0.138 0.138
0.203 0.203
0.197 0.197
Results: ELPA Program District-Level Value-Added
Outcomes
27
 Correlations between Speaking and Writing VA
Writing
No Demos
Demos
Speaking
No Demos
Fixed
Separate
Fixed
GPCM
Separate
Fixed
PCM
Separate
Fixed
GPCM
Separate
PCM
PCM
Fixed Separate
0.151 0.150
0.151 0.150
0.207 0.205
0.207 0.205
0.173 0.172
0.173 0.172
0.216 0.215
0.216 0.215
 Min = 0.150, Max = 0.246
 Mean = 0.199, SD = 0.029
GPCM
Fixed Separate
0.169 0.180
0.169 0.180
0.225 0.236
0.225 0.236
0.192 0.202
0.192 0.202
0.235 0.246
0.235 0.246
Demos
PCM
Fixed Separate
0.158 0.157
0.158 0.157
0.209 0.208
0.209 0.208
0.167 0.165
0.167 0.165
0.212 0.210
0.212 0.210
GPCM
Fixed Separate
0.180 0.189
0.180 0.189
0.231 0.240
0.231 0.240
0.189 0.197
0.189 0.197
0.233 0.243
0.233 0.243
Results: ELPA Program District-Level Value-Added
Outcomes
28
 Impact of choice of psychometric model
Correlations
3-Category
Consistency
4-Category
Consistency
Content Area
Reading
Writing
Listening
Speaking
No Demos
Fixed
Sep
0.837
0.900
0.988
0.987
0.929
0.945
0.975
0.975
Demos
Fixed
Sep
0.834
0.887
0.988
0.986
0.942
0.955
0.980
0.980
min
max
mean
SD
0.834
0.988
0.943
0.052
Content Area
Reading
Writing
Listening
Speaking
No Demos
Fixed
Sep
0.973
0.982
0.996
0.991
0.987
0.987
0.964
0.964
Demos
Fixed
Sep
0.978
0.987
0.996
0.996
0.982
0.987
0.969
0.969
min
max
mean
SD
0.964
0.996
0.982
0.011
Content Area
Reading
Writing
Listening
Speaking
No Demos
Fixed
Sep
0.567
0.634
0.902
0.866
0.728
0.728
0.795
0.795
Demos
Fixed
Sep
0.580
0.634
0.920
0.893
0.768
0.754
0.839
0.839
min
max
mean
SD
0.567
0.920
0.765
0.113
Results: ELPA Program District-Level Value-Added
Outcomes
29
 Impact of Including/Not Including Demographics
PCM
Correlations
Content Area
Reading
Writing
Listening
Speaking
Fixed
0.915
0.978
0.982
0.993
GPCM
Sep
0.915
0.978
0.982
0.993
Fixed
0.931
0.979
0.980
0.997
PCM
3-Category
Consistency
Content Area
Reading
Writing
Listening
Speaking
Fixed
0.991
0.987
0.991
0.991
4-Category
Consistency
Fixed
0.808
0.830
0.924
0.902
min
max
mean
SD
0.915
0.997
0.969
0.030
Sep
0.982
0.973
0.982
0.996
min
max
mean
SD
0.973
0.996
0.988
0.006
Sep
0.741
0.839
0.915
0.911
min
max
mean
SD
0.741
0.924
0.859
0.060
GPCM
Sep
0.987
0.987
0.991
0.991
Fixed
0.987
0.987
0.987
0.996
PCM
Content Area
Reading
Writing
Listening
Speaking
Sep
0.922
0.982
0.981
0.997
GPCM
Sep
0.817
0.821
0.911
0.902
Fixed
0.750
0.848
0.911
0.911
Results
30
MEAP Mathematics
Results: MEAP Math Student-Level Outcomes
31
 Correlations among variables based on psychometric
decisions
3-PL 1-PL 3-PL 1-PL
Number &
Operations
Algebra
Grade 7 above
diagonal/Grade 8
below
Fixed
Sep
Fixed
Sep
Fixed
Sep
Fixed
Sep
Algebra
1-PL
Fixed
Sep
1.000
1.000
0.900 0.901
0.891 0.893
0.684 0.685
0.684 0.685
0.670 0.671
0.667 0.668
3-PL
Fixed
Sep
0.943 0.941
0.943 0.941
0.996
0.984
0.677 0.666
0.676 0.665
0.691 0.682
0.688 0.679
Number & Operations
1-PL
3-PL
Fixed
Sep Fixed
Sep
0.775 0.775 0.775 0.743
0.775 0.775 0.775 0.742
0.748 0.748 0.748 0.751
0.746 0.745 0.746 0.748
1.000 1.000 0.941
1.000
1.000 0.941
0.936 0.935
0.941
0.935 0.934 0.998
-
Results: MEAP Math Student-Level Outcomes
32
 Very high correlations based on fixed versus separate
calibrations
3-PL 1-PL 3-PL 1-PL
Number &
Operations
Algebra
Grade 7 above
diagonal/Grade 8
below
Fixed
Sep
Fixed
Sep
Fixed
Sep
Fixed
Sep
Algebra
1-PL
Fixed
Sep
1.000
1.000
0.900 0.901
0.891 0.893
0.684 0.685
0.684 0.685
0.670 0.671
0.667 0.668
3-PL
Fixed
Sep
0.943 0.941
0.943 0.941
0.996
0.984
0.677 0.666
0.676 0.665
0.691 0.682
0.688 0.679
Number & Operations
1-PL
3-PL
Fixed
Sep Fixed
Sep
0.775 0.775 0.775 0.743
0.775 0.775 0.775 0.742
0.748 0.748 0.748 0.751
0.746 0.745 0.746 0.748
1.000 1.000 0.941
1.000
1.000 0.941
0.936 0.935
0.941
0.935 0.934 0.998
-
Results: MEAP Math Student-Level Outcomes
33
 Very high correlations based on fixed versus separate
calibrations
3-PL 1-PL 3-PL 1-PL
Number &
Operations
Algebra
Grade 7 above
diagonal/Grade 8
below
Fixed
Sep
Fixed
Sep
Fixed
Sep
Fixed
Sep
Algebra
1-PL
Fixed
Sep
1.000
1.000
0.900 0.901
0.891 0.893
0.684 0.685
0.684 0.685
0.670 0.671
0.667 0.668
3-PL
Fixed
Sep
0.943 0.941
0.943 0.941
0.996
0.984
0.677 0.666
0.676 0.665
0.691 0.682
0.688 0.679
Number & Operations
1-PL
3-PL
Fixed
Sep Fixed
Sep
0.775 0.775 0.775 0.743
0.775 0.775 0.775 0.742
0.748 0.748 0.748 0.751
0.746 0.745 0.746 0.748
1.000 1.000 0.941
1.000
1.000 0.941
0.936 0.935
0.941
0.935 0.934 0.998
-
Results: MEAP Math Student-Level Outcomes
34
 Not as high correlations based on 1-PL versus 3-PL
calibrations
3-PL 1-PL 3-PL 1-PL
Number &
Operations
Algebra
Grade 7 above
diagonal/Grade 8
below
Fixed
Sep
Fixed
Sep
Fixed
Sep
Fixed
Sep
Algebra
1-PL
Fixed
Sep
1.000
1.000
0.900 0.901
0.891 0.893
0.684 0.685
0.684 0.685
0.670 0.671
0.667 0.668
3-PL
Fixed
Sep
0.943 0.941
0.943 0.941
0.996
0.984
0.677 0.666
0.676 0.665
0.691 0.682
0.688 0.679
Number & Operations
1-PL
3-PL
Fixed
Sep Fixed
Sep
0.775 0.775 0.775 0.743
0.775 0.775 0.775 0.742
0.748 0.748 0.748 0.751
0.746 0.745 0.746 0.748
1.000 1.000 0.941
1.000
1.000 0.941
0.936 0.935
0.941
0.935 0.934 0.998
-
Results: MEAP Math Student-Level Outcomes
35
 Moderate to high correlations across dimensions
3-PL 1-PL 3-PL 1-PL
Number &
Operations
Algebra
Grade 7 above
diagonal/Grade 8
below
Fixed
Sep
Fixed
Sep
Fixed
Sep
Fixed
Sep
Algebra
1-PL
Fixed
Sep
1.000
1.000
0.900 0.901
0.891 0.893
0.684 0.685
0.684 0.685
0.670 0.671
0.667 0.668
3-PL
Fixed
Sep
0.943 0.941
0.943 0.941
0.996
0.984
0.677 0.666
0.676 0.665
0.691 0.682
0.688 0.679
Number & Operations
1-PL
3-PL
Fixed
Sep Fixed
Sep
0.775 0.775 0.775 0.743
0.775 0.775 0.775 0.742
0.748 0.748 0.748 0.751
0.746 0.745 0.746 0.748
1.000 1.000 0.941
1.000
1.000 0.941
0.936 0.935
0.941
0.935 0.934 0.998
-
Results: MEAP Mathematics School-Level Value-Added
Outcomes
36
Correlations
1 pre-test covariate
No Demos
Demos
Content Area
1-PL 3-PL
1-PL 3-PL
Algebra
1.000 0.995
1.000 0.992
Number & Operations 1.000 0.977
1.000 0.956
2 pre-test covariates
No Demos
Demos
1-PL 3-PL
1-PL 3-PL
1.000 0.985
1.000 0.985
1.000 0.988
1.000 0.983
3-Cat
Consistency
1 pre-test covariate
No Demos
Demos
Content Area
1-PL 3-PL
1-PL 3-PL
Algebra
0.989 0.968
0.987 0.973
Number & Operations 0.989 0.923
0.994 0.935
2 pre-test covariates
No Demos
Demos
1-PL 3-PL
1-PL 3-PL
0.987 0.935
0.989 0.960
0.990 0.946
0.989 0.966
4-Cat
Consistency
 Impact of fixed versus separate calibration
1 pre-test covariate
No Demos
Demos
Content Area
1-PL 3-PL
1-PL 3-PL
Algebra
0.995 0.926
0.993 0.883
Number & Operations 0.989 0.827
0.984 0.712
2 pre-test covariates
No Demos
Demos
1-PL 3-PL
1-PL 3-PL
0.992 0.856
0.986 0.848
0.993 0.875
0.983 0.817
Results: MEAP Mathematics School-Level Value-Added
Outcomes
37
Correlations
1 pre-test covariate
Multidimensional
No Demos
Demos
Calibration Type
1-PL 3-PL
1-PL
3-PL
Fixed Parameter 0.548 0.608
0.361 0.391
Separate
0.549 0.649
0.366 0.436
2 pre-test covariates
No Demos
Demos
1-PL
3-PL
1-PL
3-PL
0.652 0.697
0.574 0.609
0.653 0.711
0.576 0.614
3-Cat
Consistency
1 pre-test covariate
Multidimensional
No Demos
Demos
Calibration Type
1-PL 3-PL
1-PL
3-PL
Fixed Parameter 0.637 0.667
0.649 0.703
Separate
0.637 0.691
0.650 0.726
2 pre-test covariates
No Demos
Demos
1-PL
3-PL
1-PL
3-PL
0.703 0.751
0.716 0.774
0.705 0.749
0.713 0.784
4-Cat
Consistency
 Impact of choice of outcome (Algebra vs. Number)
1 pre-test covariate
Multidimensional
No Demos
Demos
Calibration Type
1-PL 3-PL
1-PL
3-PL
Fixed Parameter 0.399 0.424
0.322 0.337
Separate
0.397 0.429
0.322 0.350
2 pre-test covariates
No Demos
Demos
1-PL
3-PL
1-PL
3-PL
0.447 0.475
0.404 0.412
0.444 0.484
0.405 0.436
Results: MEAP Mathematics School-Level Value-Added
Outcomes
38
Correlations
1 pre-test covariate
Multidimensional
No Demos
Demos
Calibration Type
Alg
Num
Alg
Num
Fixed Parameter 0.939 0.963
0.883 0.934
Separate
0.938 0.962
0.876 0.937
2 pre-test covariates
No Demos
Demos
Alg
Num
Alg
Num
0.918 0.961
0.925 0.962
0.925 0.962
0.873 0.938
3-Cat
Consistency
1 pre-test covariate
Multidimensional
No Demos
Demos
Calibration Type
Alg
Num
Alg
Num
Fixed Parameter 0.890 0.901
0.851 0.912
Separate
0.886 0.907
0.841 0.918
2 pre-test covariates
No Demos
Demos
Alg
Num
Alg
Num
0.867 0.921
0.837 0.915
0.876 0.918
0.839 0.915
4-Cat
Consistency
 Impact of choice of psychometric model
1 pre-test covariate
Multidimensional
No Demos
Demos
Calibration Type
Alg
Num
Alg
Num
Fixed Parameter 0.732 0.763
0.611 0.673
Separate
0.717 0.775
0.604 0.685
2 pre-test covariates
No Demos
Demos
Alg
Num
Alg
Num
0.679 0.773
0.602 0.677
0.701 0.770
0.610 0.670
Results: MEAP Mathematics School-Level Value-Added
Outcomes
39
Correlations
1 pre-test covariate
Multidimensional
1-PL
3-PL
Calibration Type
Alg
Num
Alg
Num
Fixed Parameter 0.964 0.815
0.813 0.717
Separate
0.962 0.819
0.806 0.780
2 pre-test covariates
1-PL
3-PL
Alg
Num
Alg
Num
0.984 0.822
0.895 0.775
0.983 0.825
0.877 0.793
3-Cat
Consistency
1 pre-test covariate
Multidimensional
1-PL
3-PL
Calibration Type
Alg
Num
Alg
Num
Fixed Parameter 0.880 0.772
0.771 0.713
Separate
0.875 0.767
0.774 0.724
2 pre-test covariates
1-PL
3-PL
Alg
Num
Alg
Num
0.928 0.774
0.841 0.771
0.927 0.775
0.831 0.756
4-Cat
Consistency
 Impact of Including/Not Including Demographics
1 pre-test covariate
Multidimensional
1-PL
3-PL
Calibration Type
Alg
Num
Alg
Num
Fixed Parameter 0.775 0.551
0.572 0.464
Separate
0.774 0.556
0.544 0.522
2 pre-test covariates
1-PL
3-PL
Alg
Num
Alg
Num
0.864 0.557
0.646 0.508
0.858 0.552
0.635 0.547
Results: MEAP Mathematics School-Level Value-Added
Outcomes
40
Correlations
No Demographics
Multidimensional
1-PL
3-PL
Calibration Type
Alg
Num
Alg
Num
Fixed Parameter 0.937 0.965
0.923 0.964
Separate
0.937 0.965
0.937 0.962
Includes Demographics
1-PL
3-PL
Alg
Num
Alg
Num
0.941 0.947
0.930 0.951
0.941 0.948
0.941 0.942
3-Cat
Consistency
No Demographics
Multidimensional
1-PL
3-PL
Calibration Type
Alg
Num
Alg
Num
Fixed Parameter 0.855 0.884
0.851 0.889
Separate
0.859 0.889
0.878 0.883
Includes Demographics
1-PL
3-PL
Alg
Num
Alg
Num
0.889 0.918
0.872 0.744
0.885 0.922
0.885 0.755
4-Cat
Consistency
 Impact of covarying on one vs. two pre-test scores
No Demographics
Multidimensional
1-PL
3-PL
Calibration Type
Alg
Num
Alg
Num
Fixed Parameter 0.734 0.764
0.696 0.753
Separate
0.729 0.768
0.727 0.754
Includes Demographics
1-PL
3-PL
Alg
Num
Alg
Num
0.715 0.687
0.704 0.713
0.716 0.693
0.714 0.698
Conclusions
41
 Practically important impacts on value-added
metrics and value-added classifications



Choice of psychometric model
Including/not including demographics
Including/not including multiple pre-test values
 Prohibitive impacts on value-added metrics and
value-added classifications

Choice of outcome (i.e., domain within construct)
 Practically negligible impacts on value-added metrics
and value-added classifications

Separate versus fixed calibrations of domains within construct
Conclusions, continued…
42
 Need to pay attention to modeling domains within
constructs if constructs can reasonably be considered
multidimensional


Of the common psychometric and statistical modeling decisions one
can make, the choice of which subscore to use as an outcome is the
most influential
Because subscores give different profiles of both student
achievement and program/school value-added, each subscore should
be modeled to the degree possible
 4-category (i.e., quartile) classifications on value-added
are appreciably impacted by every psychometric and
statistical modeling choice evaluated here, but 3-category
classifications are not


Discourage more than three categories
RTTT requires at least four categories
Conclusions, continued…
43
 3- vs. 4-category distinction is actually a proxy for
 Statistical decision categories (3-categories)
 Arbitrary cut point categories (4-categories)
 Can leverage unidimensional calibrations of
multidimensional achievement scales to create
multidimensional profiles of value-added

Except where using four categories of classifications
Limitations
44
 Inductive reasoning
 Results are likely to hold in similar circumstances
 Still will need to investigate feasibility of using fixed
parameters from unidimensional calibration for specific
circumstances if those circumstances are high stakes
 This is a proof of concept
 PCM and GPCM models were run using different
software (WINSTEPS vs. PARSCALE)
Contact Information
45
 Joseph A. Martineau, Ph.D.
 Executive Director
 Bureau of Assessment & Accountability
 Michigan Department of Education
 martineauj@michigan.gov
 Ji Zeng, Ph.D.
 Psychometrician
 Bureau of Assessment & Accountability
 Michigan Department of Education
 zengj@michigan.gov