Closing the loop: Providing test
developers with performance level
descriptors so standard setters can
do their job
Amanda A. Wolkowitz Alpine Testing Solutions,
James C. Impara Psychometric Inquiries
Chad W. Buckendahl Psychometric Consultant
S
What do standard setters do &
how do they do it?
S
Standard setters recommend cut scores
S
An early step in the process is defining what the “borderline” examinee is
expected to be able to do in terms of the test content. Specifically, they
examine the performance level descriptors (PLDs) and define the borderline
examinee at each performance level.
S
Using modifications of the Angoff or Bookmark methods, they review test
items and judge the difficulty of the item for examinees who are at the
borderline of one or more performance categories.
S
Typically they estimate item difficulty one or more times (rounds), sometimes
with item or other data provided after their first round of item difficulty
estimation.
How are tests developed?
S Item writers, typically content “experts,” draft items that are
responsive to the test specifications (or test blueprint).
S The test blueprint may or, most often, may not include a
description of the various performance levels.
S The test blueprint virtually never provides a description of
the “borderline” examinee at each performance level.
The study design
S This was not a designed study, but an ad hoc study. That is,
we developed the research question and looked for data that
would provide some answers. Thus, there are some
limitations.
S The first data collection was in 2009 and the second was in
2013. Both in the same southeastern state in the USA.
S Both related to the same assessment.
The study design - 2
S 2009
S Performance level descriptors (PLDs) defined initially.
S Borderline performance described for each PLD
S Standard setting done for alternative assessments (for students
with severe cognitive disabilities) in:
S
S
S
S
S
English Language Arts (ELA) grades 4 – 8
Mathematics grades 3 – 8
All tests had 15 items scored dichotomously (0 or 2 for each item)
Four performance levels were defined, thus three cut scores
There were separate panels for each content area.
The study design - 3
S 2013 Standard Setting was the same as 2009 except:
S PLDs developed in 2009 were examined and refined. The
original PLDs were known to test developers and drove the
development process.
S Scoring was modified from dichotomous to three-point scoring
for each item – partial credit was permitted, so scores were 0,
1, or 2.
S Slightly fewer panelists (17 – 20 for each grade span in 2009, 14
– 15 in 2013).
Study design - 4
S A final difference in the two standard setting activities was the
method used in the standard setting.
S 2009 used the Modified Angoff method as described by Impara &
Plake (1997), often characterized as the Yes/No method.
S 2013 used the Extended Angoff method as described by
Hambleton & Plake, 1995 and Plake & Hambleton, 2001
S The reason for this difference was because of the change from
dichotomous scoring to giving partial credit (3-points)
S Both methods rely on item level judgments.
PLDs
S
There were four performance levels:
S
S
S
S
S
The PLDs further defined each level
S
S
S
S
S
Achievement Level 1 (limited command),
Achievement Level 2 (partial command),
Achievement Level 3 (solid command), and
Achievement Level 4 (superior command).
Level 1 students would need academic support,
Level 2 students would likely need academic support,
Level 3 students would be prepared, and
Level 4 students would be well prepared to be successful in further studies in that content
area.
PLDs also contained specific abilities that students at that given level could demonstrate.
Study Expectations
S The principal research question was:
Will the consistency of ratings at the end of round 1 of the
standard setting process increase? That is: Will developing items
with known PLDs help panelists be more consistent with their
initial ratings and more congruent with the item p-values prior to
any feedback.
Why?
S Why is this an important question?
S If panelists are more consistent in their round 1 ratings, then
they may come to closure faster in subsequent rounds, perhaps
reducing the number of rounds (sometimes 3 rounds are used),
thus making the process more efficient.
S Panelists often become frustrated if there are no or too few
items at a performance level, thus causing them to question the
validity of the process.
How?
S How will we know if there is greater consistency among panelists?
S Distribution of students across levels will be consistent with expectations –
most students will be classified at Levels 2 and 3.
S There should be greater congruence between actual item difficulty and the
panelists’ estimate of item difficulty
S The correlation between actual item difficulty and panelists’ item
difficulty estimate will be higher
S The range of panelists’ cut scores will be lower.
S Percentage of panelists who were within one point of the recommended
cut score at the end of round 1 will be higher.
S The standard deviation of the panelists’ cut scores at each level will be
lower.
Result – 1
S Distribution of students across levels
Distribution of students - ELA
Distribution of students - ELA
S In virtually every grade in the 2009 standard setting many
students were assigned to achievement level 4, the highest
level.
S In 2013, the distribution was much more appropriate, with
most students assigned to levels 2 and 3.
Distribution of students - Math
Distribution of students - Math
S In 2009, several of the grades showed appropriate
distributions, but many still have many students assigned to
levels 1 and 4.
S In 2013, relatively few students were assigned to levels 1 and
4 and the preponderance of students were classified as level
2 or 3.
Congruence of actual and
panelist’s item difficulty
S It was expected that the actual item difficulty value for an item (i.e.,
the percent of students in the population who get the item correct)
would be greater than or equal to the corresponding Level 2 cut score
rating and less than the corresponding Level 4 cut score rating.
S Hence, the actual p-value would be between the Level 2 cut score and
the Level 4 cut score.
S Except there should be a relatively small number of items that have
difficulties that are outside this range (those items that virtually all
examinees answer (for the Level 1 targeted items) and those that
virtually no one answers correctly (those targeted at Level 4).
Congruence of actual and
panelist’s item difficulty
Congruence - summary
ELA4
ELA5
ELA6
ELA7
ELA8
MATH3
MATH4
MATH5
MATH6
MATH7
MATH8
2009
10 (67%)
7 (47%)
10 (67%)
8 (53%)
13 (87%)
11 (73%)
9 (60%)
10 (67%)
10 (67%)
12 (80%)
9 (60%)
2013
15 (100%)
14 (93%)
14 (93%)
11 (73%)
13 (87%)
11 (73%)
14 (93%)
15 (100%)
15 (100%)
14 (93%)
13 (87%)
Percent of p-values Between the Item Level Cut Scores for Achievement Level 2 and Achievement Level 4
(i.e., item Level 2 cut score ≤ p-value ≤ item Level 4 cut score)
Correlation Analysis
S A correlation analysis compared the relationship between
actual item difficulty values and the average item rating at
each achievement level.
S Expectation: a direct relationship was expected between the
item’s difficulty value and average item rating.
S As the item difficulty value increases (i.e., the item becomes
easier), the greater the chance a borderline student will
correctly respond to the item correctly.
S This trend was expected for all three cut scores.
Correlation Analysis
S Results – the reverse of expectations.
S 2009 item ratings generally had moderate to strong positive
correlations with the corresponding item difficulty values.
S 2013 ratings tended to have only moderate correlations at best.
S The 2009 ratings correlated higher with the p-values than did
the 2013 ratings
Correlation Analysis
S Why were 2009 correlations higher?
S One possible explanation: the 2009 panel only had to make
Yes/No judgments whereas the 2013 panel had to make a
judgment as to whether a student would score 0, 1, or 2 points on
the item
S Another possible explanation: is that the items on the 2013 exams
may have had more similar difficulty values around the intended
PLDs than the 2009 items.
S Also, it was learned that in 2013 few students were assigned the 0
score, resulting in a restriction of range of p-values.
Internal Consistency – range
S Internal consistency was evaluated several ways
S First by comparing the range of recommended cut scores
following round 1 for each level and panel.
S Thus, a smaller range would indicate that the given year’s
panel was more consistent with their ratings than the other
year’s panel.
Internal Consistency – Range
S Range –ELA
Exam
ELA - 4
ELA – 5
ELA - 6
ELA - 7
ELA – 8
2009 2013 2009 2013 2009 2013
Level Level Level Level Level Level
2
2
3
3
4
4
12
21
14
18
12
9
12
7
12
12
10
7
12
14
12
8
12
7
10
13
12
9
18
5
16
16
18
8
12
6
Internal Consistency - Range
Range – Math
Exam
Math - 3
Math - 4
Math - 5
Math - 6
Math - 7
Math - 8
2009 2013 2009 2013 2009 2013
Level Level Level Level Level Level
2
2
3
3
4
4
12
9
20
14
14
10
12
9
14
13
14
8
8
12
12
15
16
9
10
19
12
15
12
9
8
16
12
10
12
6
10
9
14
8
8
7
Internal Consistency –
Proximity to the median
S Internal consistency was evaluated several ways:
S Second, calculating the percent of panelists whose ratings were
within one point (plus or minus) of their panel’s recommended
Round 1 median cut score (all median cut scores ended up as
possible scores, i.e., no median cut score ended in “.5”).
S Thus, if the percent of panelists’ cut scores were all
relatively close together, they would be close to the median.
S For example, the median Level 2 cut score recommendation for
the “Math – 4” exam was 6 out of 30 points for 2009 and 7 out
of 30 points for 2013.
Internal Consistency –
Proximity to the median
ELA
Exam
ELA - 4
ELA – 5
ELA - 6
ELA - 7
ELA – 8
2009
Level
2
40%
30%
47%
35%
35%
2013
Level
2
33%
47%
40%
47%
33%
2009
Level
3
25%
25%
29%
18%
12%
2013
Level
3
47%
40%
27%
40%
60%
2009
Level
4
35%
65%
24%
24%
35%
2013
Level
4
67%
53%
40%
47%
47%
Internal Consistency –
Proximity to the median
Math
Exam
Math - 3
Math - 4
Math - 5
Math - 6
Math - 7
Math - 8
2009
Level
2
15%
40%
25%
29%
18%
29%
2013
Level
2
47%
53%
47%
40%
27%
20%
2009
Level
3
20%
35%
45%
18%
29%
24%
2013
Level
3
40%
53%
27%
47%
40%
47%
2009
Level
4
40%
45%
40%
29%
41%
29%
2013
Level
4
33%
60%
33%
67%
60%
20%
Internal Consistency – Standard
Deviation
S Internal consistency was evaluated several ways:
S The third way to look at internal consistency is to compare the
standard deviations of the panelists’ ratings across years.
S Thus, if the panelists are more consistent, then the standard
deviations will be smaller.
Internal Consistency – Standard
Deviation
ELA
Exam
ELA - 4
ELA – 5
ELA - 6
ELA - 7
ELA – 8
2009
Level
2
2.74
2.93
2.85
2.60
3.89
2013
Level
2
5.18
2.14
3.63
3.70
4.33
2009
Level
3
3.73
3.18
3.82
3.31
4.56
2013
Level
3
4.35
3.51
2.82
2.66
2.29
2009
Level
4
2.62
2.95
3.28
4.03
3.10
2013
Level
4
2.44
2.21
3.72
1.58
1.87
Internal Consistency – Standard
Deviation
Math
Exam
Math - 3
Math - 4
Math - 5
Math - 6
Math - 7
Math - 8
2009
Level
2
3.43
2.91
2.62
2.92
2.93
2.54
2013
Level
2
2.72
2.67
3.19
4.91
4.51
3.02
2009
Level
3
4.82
3.08
3.06
2.83
3.06
3.94
2013
Level
3
4.13
3.31
3.83
4.00
2.72
2.34
2009
Level
4
3.34
3.33
4.01
2.96
3.09
2.73
2013
Level
4
3.14
2.30
2.90
2.35
1.83
2.26
Recall this slide
S How will we know if there is greater consistency among panelists?
S
Distribution of students across levels will be consistent with expectations –
most students will be classified at Levels 2 and 3.
S
There will be greater congruence between actual item difficulty and the
panelists estimate of item difficulty
S
The correlation between actual item difficulty and panelists’ item difficulty
estimate will be higher
S
The range of panelists’ cut scores will be lower.
S
Percentage of panelists who were within one point of the recommended cut
score at the end of round 1 will be higher.
S
The standard deviation of the panelists’ cut scores at each level will be lower.
How did we do in terms of
distribution of students?
S Expected result:
S Distribution of students across levels will be consistent with
expectations – most students will be classified at Levels 2 and
3.
S Actual result:
S In both ELA and Math the results were as expected in virtually
every grade and performance level.
S Thus, positive results
How did we do in terms of
congruence of item difficulty?
S Expected result:
S There will be greater congruence between actual item difficulty
and the panelists estimate of item difficulty
S Actual result:
S There was greater congruence between actual item difficulties and
panelists’ estimation of item difficulty at all levels and grades.
S However, there were few cases in which the actual p-values were
outside the cut score boundaries.
S Thus, somewhat positive results, but somewhat problematic.
How did we do in terms of correlation of
actual and estimated item difficulty?
S Expected result:
S The correlation between actual item difficulty and panelists’
item difficulty estimate will be higher in 2013.
S Actual result:
S The 2009 ratings correlated higher with the p-values than did
the 2013 ratings
S Thus, negative results
How did we do in terms of the
ranges of panelists’ cut scores?
S Expected results:
S The range of panelists’ cut scores will be lower in 2013.
S Actual results:
S In the majority of grades and levels the range of cut scores was
lower in 2013, particularly at Levels 3 and 4.
S Thus, mostly positive results
How did we do in terms of the proximity
of panelists’ cut scores to the median?
S Expected result:
S Percent of panelists who were within one point of the
recommended cut score at the end of round 1 will be higher in
2013.
S Actual result:
S In the majority of comparisons, the percent of panelists ratings
that were within one point of the median was higher in 2013.
S Thus, mostly positive.
How did we do in terms of the standard
deviations of panelists’ cut scores?
S Expected results:
S The standard deviation of the panelists’ cut scores at each level
will be lower in 2013.
S Actual results:
S In the majority of comparisons the 2013 panels had lower
standard deviations than the 2009 panels.
S Thus, mostly positive
Overall & Conclusion
S The results overall supported providing test developers with the
PLDs.
S More specifically designed research is needed. Many limitations to
this study.
S If future studies are supportive of providing the test developers
with the PLDs and they are instructed to target item development
to these PLDs, it could result in more efficiency in the standard
setting process and in greater levels of satisfaction among
panelists.
Questions?
S Thank you for your attention.
S Are there any questions?