A procedure for dimensionality
analyses of response data from
various test designs
Jinming Zhang
William Stout
Introduction

Dimension




Dimensional structure of the test (e.g., algebra
and geometry)
statistical dimensional structure of response
data
Incorporate both Judgments about test
content and evidence from statistical
analyses
Missing data: CAT & multistage testing

Missing item pair measurement
DETECT index
Modified DETECT index
Example 3


Stage 1 booklet: 2 dim & Stage 2 booklet 1: 2
dim, Stage 2 booklet 2: 1 dim (additional dim),
Stage 2 booklet 3: 1 dim (additional dim).
Bridge item (in Stage 1)


E(1,2) and E(2,3), so E(1,3)
First-stage booklet should


measure all of the constructs/contents the whole
test aim to measure, though it is unknown.
classify examinees into different proficiency levels
Item 1 and 2 are measuring the same dimension
1. If E(1,2), E(2,3), and E(1,3)


D(P*) is maximized in P* partition if item 1 and 2 are in
the same cluster, other than in other P
2. If only E(1,2) and E(2,3), and if item 2 is a bridge
item

D(P*) is maximized in P* partition if item 1 and 2 are in
the same cluster when item 2 is in the same cluster or
not, other than in other P
Whether a test has an approximate
simple structure or not
Discordance

Resulted from



Not a approximate simple structure
Inaccuracy of conditional covariance estimation
Given a partition of items,

What made Dd(P*) and PropD(P*) large



Unidimensionality
Violation of approximate simple structure
Inaccuracy of conditional covariance estimation
polyDETECT


Obtain the composite theta score:
unidimensinoal approximation & simple
structure approach.
Use percentiles of composite theta scores as
cut-off points in forming AHGs.


Between 25 to 100 in each group
Cross-validation:
polyDETECT

Evidence of multidimensionality
Condition 3 is hold or not?




Each sizable cluster contains at least one
stage-1 item.
Exist at least one sizable cluster that does not
contain any Stage-1 items, and all items in such
cluster belong to the same booklet.
Exist at least one sizable cluster that does not
contain any Stage-1 items, and all items in such
cluster belong to at least two booklet.
Exist at least two sizable cluster that does not
contain any Stage-1 items, items in each such
cluster come from the same booklet but
different clusters belong to different booklets.
Dealing with CAT data




Estimates of item parameters were obtained
before conducting CAT. Why do
dimensionality assessment on CAT data?
Sparse data set of CAT
100,000 responses are required at least
Item selection, item exposure control,
content balance to satisfy the condition 3
Simulation study










M2PL
Each booklet has 30 items
Dimension: 1, 2, 3
Number of examinees: 750, 1500, 3000, 4000
Theta: MVN(0,sigma), correlation = 0.8
Cut-off points for low-, moderate-, and highscoring group: <10, 11~18, >18
About 37.82% unestimable item pairs
Cross-validation
Replication: 100
Composite theta: use unidimensional IRT model
Results
Real data analyses

Missing values are large (55%~71%)
Real data analyses



Weak dimensionality (M value)
High PropD(P*) indicates a large amount of
spurious information to form partition P*
Confirmatory analyses: D3 >D2

But DETECT tends to underestimate, so twocluster partition solution may be preferred.
Real data analyses

High correlation indicates a weak degree of
multidimensionality
Concluding remarks

Moderate violation of approximate simple
structure is still hold