Chapter 5
Alignment Among Secondary and Post-Secondary Assessments
in Maryland
The Maryland Assessment Environment
The Maryland State Department of Education (MSDE) has been actively involved
in a major educational reform initiative for more than ten years. The Governor’s 1989
Commission on School Performance reviewed the State’s public school systems and
recommended that it establish comprehensive state accountability, assessment and
reporting systems. As a result, the Maryland State Board for Education adopted the
reform initiative entitled Schools for Success, one of the first state reform movements to
“hold schools accountable for a high quality education and measurable results for all
students.” This initiative is based on the premise that “all children can learn” and should
be exposed to “equally rigorous content.”
The major high school components of the Schools for Success reform program,
the High School Improvement Program and the High School Assessments (HSA), are
presently being developed. The assessments will build on the current core academic area
requirements in English, mathematics, science and social studies and, eventually, may
include a series of 12 tests. The tests, which will be administered at tenth grade, will
assess students’ knowledge of state standards (i.e., Core Learning Goals) that were
developed by teams of content experts appointed by the State Superintendent of Schools.
The HSAs are intended to reflect high expectations and standards, and will replace the
Maryland Functional Tests, a series of minimum competency graduation tests that
assesses basic reading, writing, and math skills. Currently, there are HSAs for English I,
algebra, geometry, biology, and government. As originally approved, students will be
required to pass three subject matter tests (English I, government, and either algebra or
geometry) to graduate from high school.1
1
Starting in fall of 2003, students entering grade 9 are required to pass the English I, algebra, government,
and biology assessments in order to graduate from high school. Passing the geometry assessment will be
eventually be added as a graduation requirement.
93
Maryland Assessments Included in this Study
For this study, we examined the HSA prototype items in English, algebra, and
geometry. All three tests will assess student knowledge with both multiple-choice and
open-ended items, but to date, no final decisions have been made with respect to mix of
item types, test length, or time limit. Because prototype items may not be representative
of the final set of items, interpretations regarding the technical features, content coverage,
and cognitive demands of the HSAs are tentative.
In addition to taking the HSAs, students applying to a higher education institution
may need to take a placement test that ensures students enroll in the appropriate course
for their proficiency level. Because the kinds of placement tests given are likely to vary
by the selectivity of the institution, we obtained placement tests from both a highly
selective university institution (University of Maryland, College Park), and a less
selective institution (Towson State University). We also obtained placement tests used at
the community college level.
At Towson State University, the Nelson-Denny Reading Test and ASSET Writing
Skills exam are used to place students in an appropriate English course.2 The NelsonDenny Reading exam contains 80 items measuring vocabulary, and 38 questions
assessing reading comprehension. Students are allowed approximately one hour to
complete both sections. The ASSET Writing Skills test, which measures knowledge of
grammar and other conventions of written English, consists of 30 multiple-choice items
administered in 25 minutes. The University of Maryland, College Park, does not require
its students take a placement exam in English.
Both Towson State University and the University of Maryland, College Park
administer math placement tests. At both universities, all entering students who do not
have AP Calculus credit or who do not transfer college-level calculus credit from another
university are required to take a placement test to determine which math course is most
appropriate given their preparation. At Towson State University, students are allowed
one hour to complete a computer-adaptive multiple-choice exam, in which the number of
items administered is contingent upon the students’ demonstrated proficiency level. At
the University of Maryland, College Park, students are allowed 70 minutes to complete
2
Nelson-Denny is published by Riverside Publishing and ASSET is published by ACT.
94
63 multiple-choice items. Despite the apparent differences between the two placement
tests, both institutions give students the same sample exam to prepare. Hereafter, we will
refer to this sample exam as the Towson/UOM Math Placement Test.3
Many of the community colleges in Maryland also administer placement tests to
ensure that students enroll in the appropriate course for their ability level. One of the
most commonly used assessments is the ACCUPLACER, which is a series of computeradaptive assessments that measure student readiness for a broad range of mathematical
and English courses.4 Depending upon the math test taken and their performance on this
test, students are then placed into the appropriate math course. For our study, we
examined the Elementary Algebra, Reading Comprehension, and Sentence Skills tests.5
Tables 5.1 and 5.2, organized by test function, list these testing programs and the
type of information we were able to obtain for this study. For most tests, we used a
single form from a recent administration or a full-length, published sample test. In a few
instances where full-length forms were unavailable, we used published sets of sample
items. This was the case for Towson/UOM Math Placement Test and the HSAs. For the
ELA tests, Table 5.2 specifies whether the test includes each of three possible skills:
reading, editing, and writing.
3
Because Towson State University administers a computer-adaptive exam, students of different proficiency
levels may encounter different item content mixes. Thus, interpretations about the content and cognitive
demands elicited by the sample Towson Math Placement Test should be interpreted cautiously.
4
ACCUPLACER is published by the Educational Testing Service.
5
We were unable to obtain items from the ACCUPLACER pool of items. Instead, we examined a fulllength copy of the Companion exams, which is based on the same test specifications as the
ACCUPLACER, and is the paper-and-pencil version of the ACCUPLACER. Again, ACCUPLACER is a
computer-adaptive measure so students may encounter different kinds of items, depending upon their
proficiency. Interpretations about the ACCUPLACER’s content and cognitive demands should be made
cautiously.
95
96
Test Type
State achievement
End-of-course
State achievement
End-of-course
College
admissions
College
admissions
College
admissions
Test
High School
Assessment
Algebra (HSA
Algebra)
High School
Assessment
Geometry (HSA
Geometry)
ACT
SAT I
SAT II
Mathematics Level
IC
Full sample
form
Full sample
form
Full sample
form
Prototype
items
Prototype
items
Materials
Examined
96
60 minutes
75 minutes
60 minutes
To be
determined
To be
determined
Time Limit
Table continues
50 MC
35 MC
15 QC
10 GR
60 MC
MC
OE
MC
OE
Number and
Type of Items
Calculator
Calculator
Calculator
Calculator
Calculator
Tools
Selection of
students for higher
education
Selection of
students for higher
education
Selection of
students for higher
education
Monitor student
achievement
toward stateapproved content
standards
Monitor student
achievement
toward stateapproved content
standards
Purpose
Table 5.1
Technical Characteristics of the Mathematics Assessments
Elementary and intermediate
algebra (30%), geometry (38%,
specifically plane Euclidean
(20%), coordinate (12%), and
three-dimensional (6%)),
trigonometry (8%), functions
(12%), statistics and probability
(6%), and miscellaneous (6%)
Arithmetic (13%), algebra
(35%), geometry, (26%), and
other (26%)
Prealgebra (23%), elementary
algebra (17%), intermediate
algebra (15%), coordinate
geometry (15%), plane geometry
(23%) and trigonometry (7%)
Geometry
Elementary algebra
Content as Specified in Test
Specifications
97
Notes.
MC = multiple-choice
OE = open-ended
GR = grid-in
QC = quantitative comparison
College placement
University of
Maryland Math
Placement Test
College placement
End-of-Course
ACCUPLACER
Elementary
Algebra
College placement
College
admissions
SAT II
Mathematics Level
IIC
Towson State
University Math
Placement Test
Test Type
Test
Sample items
Sample items
Full sample
paper-andpencil form
Full sample
form
Materials
Examined
97
70 minutes
60 minutes
Varies by
examinee
60 minutes
Time Limit
63 MC
Number of MC
items given
depends on
demonstrated
proficiency
level
Number of MC
items given
depends on
demonstrated
proficiency
level
50 MC
Number and
Type of Items
Calculator
Calculator
None
Calculator
Tools
Placement of
students into
appropriate
mathematics
course
Placement of
students into
appropriate
mathematics
course
Assess student
proficiency in
elementary algebra
Selection of
students for higher
education
Purpose
Arithmetic, elementary algebra,
advanced algebra, and
trigonometry
Arithmetic, elementary algebra,
advanced algebra, and
trigonometry
Signed numbers and rational
numbers, algebraic expressions,
equations, inequalities, and word
problems
Algebra (18%), geometry (20%,
specifically coordinate (12%)
and three-dimensional (8%)),
trigonometry (20%), functions
(24%), statistics and probability
(6%), and miscellaneous (12%)
Content as Specified in Test
Specifications
98
Test Function
State achievement
College
admissions
College
admissions
College
admissions
College
admissions
College
admissions
Test
High School
Assessment English
(HSA English)
ACT
AP Language and
Composition
SAT I
SAT II Literature
SAT II Writing
Full sample
form
Full sample
form
Full sample
form
Full sample
form
Full sample
form
Prototype
items
Materials
Examined
98
60 minutes
-- 40 minutes
editing
-- 20 minutes
writing
60 minutes
75 minutes
180 minutes
--60 minutes
reading
-- 120 minutes
writing
80 minutes
--35 minutes
reading
--45 minutes
editing
To be
determined
Time Limit
Table continues
60 MC editing
1 OE writing
60 MC reading
40 MC reading
38 MC editing
Selection of students for
higher education
Selection of students for
higher education
Selection of students for
higher education
Provide opportunities for
HS students to receive
college credit and
advanced course
placement
Selection of students for
higher education
40 MC reading
75 MC editing
52 MC reading
1 OE reading
2 OE writing
Monitor student
achievement toward stateapproved content
standards
Purpose
MC reading
MC editing
1 OE writing
Number and
Type of Items
N
Y
Y
Y
Y
Y
Reading
Section?
Table 5.2
Technical Characteristics of the English/Language Arts Assessments
Y
N
Y
N
Y
Y
Editing
Section?
Y
N
N
Y
N
Y
Writing
Section?
99
Notes.
MC = multiple-choice
OE = open-ended
College placement
Nelson-Denny
Reading Test
College placement
ACCUPLACER
Sentence Skills
College placement
College placement
ACCUPLACER
Reading
Comprehension
ASSET Writing
Skills
Test Function
Test
Full sample
form
Full sample
form
Full sample
paper-andpencil form
Full sample
paper-andpencil form
Materials
Examined
99
56 minutes
--24 minutes
vocabulary
section
--32 minutes
comprehension
section
25 minutes
Varies by
examinee
Varies by
examinee
Time Limit
80 MC
vocabulary
items
38 MC
comprehension
items
32 MC
Number of MC
items given
depends on
demonstrated
proficiency
level
Number of MC
items given
depends on
demonstrated
proficiency
level
Number and
Type of Items
Assess student proficiency
in vocabulary
development and reading
comprehension
Placement of students into
appropriate English course
Placement of students into
appropriate English course
Placement of students into
appropriate English course
Purpose
Y
N
N
Y
Reading
Section?
N
Y
Y
N
Editing
Section?
N
N
N
N
Writing
Section?
Alignment Among Maryland Math Assessments
In this section, we describe the results of our alignment exercise for the math
assessments. The results are organized so that alignment among tests with the same
function is presented first, followed by a discussion of alignment among tests with
different functions. In some instances, there are only two tests that share the same
purpose, so it is important to recognize that patterns or comparisons between these tests
may not be indicative of more general trends within this category of tests.
Alignment is described by highlighting similarities and differences with respect to
technical features, content, and cognitive demands. That is, we first present how the
assessments vary on characteristics such as time limit, format, contextualized items,
graphs, diagrams, and formulas. We then document differences with respect to content
areas, and conclude with a discussion of discrepancies in terms of cognitive requirements.
Table 5.3 presents the alignment results for the math assessments. The numbers
in Table 5.3 represent the percent of items falling into each category. As an example of
how to interpret the table, consider the SAT I results; 58% of its items are multiplechoice, 25% are quantitative comparisons, and 17% are grid-in items. With respect to
contextualization, 25% of the SAT I questions are framed as a real-life word problem.
Graphs are included within the item-stem on 7% of the questions, but graphs are not
included within the response options (0%), and students are not asked to produce any
graphs (0%). Similarly, diagrams are included within the item-stem on 18% of the
questions, but diagrams are absent from the response options (0%), and students are not
required to produce a diagram (0%). With respect to content, the SAT I does not include
trigonometry (0%), and assesses elementary algebra (37%) most frequently. In terms of
cognitive demands, procedural knowledge (53%) is the focus of the test, but conceptual
understanding (32%) and problem solving (15%) are assessed as well. Results for the
other tests are interpreted in an analogous manner.
100
101
MC
QC
53
HSA
Geometry
0
0
0
0
SAT II Math
100
Level IC
SAT II Math
100
Level IIC
100
0
0
0
0
0
0
17
0
12
11
GR
0
0
0
0
0
0
35
58
OE
Notes.
Format
MC = multiple-choice items
QC = quantitative comparison items
GR = fill-in-the-grid items
OE = open-ended items
Towson/
UOM Math
Placement
Test
ACCUPLACER
(Elementary 100
Algebra)
College Placement Tests
25
58
SAT I
0
100
ACT
College Admissions Tests
32
HSA
Algebra
State Achievement Tests
Test
Format
0
25
12
18
25
22
65
89
C
Context
6
0
12
8
7
5
24
37
S
0
0
0
0
0
0
6
16
P
13
13
2
26
18
13
65
11
S
0
0
0
0
0
0
6
0
RO
0
0
0
0
0
0
0
5
P
Diagrams
25
13
10
12
1
15
0
0
M
Table continues
0
0
0
0
8
0
41
0
G
Formulas
Contextualization
C = contextualized items
101
0
0
2
0
0
2
0
0
RO
Graphs
19
0
2
2
13
17
0
5
PA
31
0
22
10
2
5
0
0
IA
13
0
12
12
6
15
24
16
CG
6
13
14
28
19
25
71
0
PG
19
0
18
4
0
8
6
0
0
0
6
8
13
3
0
16
TR SP
0
0
12
6
11
5
0
11
MISC
25
0
26
34
32
40
6
0
75
100
54
58
53
53
71
68
PK
Cognitive
Demands
CU
Diagrams
S
= graph/diagram within item-stem
RO = graph/diagram within response options
P
= graph/diagram needs to be produced
13
88
14
30
37
22
0
53
EA
Content
Table 5.3
Alignment Among the Technical, Content, and Cognitive Demands Categories for the Math Assessments
0
0
20
8
15
7
24
32
PS
102
Formulas
M = formula needs to be memorized
G = formula is provided
102
Content Areas
PA = prealgebra
EA = elementary algebra
IA = intermediate algebra
CG = coordinate geometry
PG = plane geometry
TR = trigonometry
SP = statistics and probability
MISC = miscellaneous topics
Cognitive Demands
CU = conceptual understanding
PK = procedural knowledge
PS = problem-solving
Alignment Among Tests With the Same Function
State Achievement Tests
Two state achievement tests are included in this analysis: the HSA Algebra and
HSA Geometry. Both assessments are end-of-course exams that include multiple-choice
and open-ended items. The HSA Algebra and HSA Geometry contain many
contextualized questions (89% and 65%, respectively) as well as many questions that
contain graphs within the item-stem (37% and 24%, respectively). Questions with
diagrams in the item-stem comprise a large proportion of the HSA Geometry (65%), but
only a small proportion of the HSA Algebra (11%). Neither test requires includes items
that require a memorized formula, but 41% of the HSA Geometry items requires a
formula that has been provided.
In terms of content areas, the HSA Algebra emphasizes elementary algebra (53%)
whereas the HSA Geometry focuses on planar geometry (71%). With respect to
cognitive demands, both tests assess procedural knowledge most frequently (68%-71%),
but a moderate proportion of items measure problem solving as well (24%-32%).
College Admissions Tests
We examined four college admissions tests: the ACT, SAT I, SAT II Math Level
IC, and SAT II Math Level IIC. All tests, except the SAT I, have a one-hour time limit.
SAT I has a 75-minute time limit. All four exams are also predominantly multiplechoice, although the SAT I includes quantitative comparison (25%) as well as grid-in
(17%) items. Contextualized questions are most prevalent on the SAT I (25%) and least
prevalent on the SAT II Math Level IIC (12%). Students are rarely asked to work with
graphs, and questions that contain graphs within the item-stem constitute no more than
12% of items on the college admissions measures. Questions that include diagrams
within the item-stem are more prevalent, comprising 26%, 18%, and 13% of items on the
SAT II Math Level IC, SAT I, and ACT, respectively. However, questions with
diagrams are infrequent on the SAT II Math Level IIC (2%). Formulas are also
uncommon, but there are differences with respect to the extent to which formulas are
necessary. Whereas the ACT, SAT II Math Level IC, and SAT II Math Level IIC include
103
some items in which a memorized formula is needed (15%, 12%, and 10%, respectively),
these items are largely absent from the SAT I (1%).
Although the college admissions exams generally sample from the same content
areas, they do not do so to the same extent. Elementary algebra comprises most of the
SAT I items (37%). The SAT II Math Level IC also emphasizes elementary algebra
(30%), but focuses on planar geometry as well (28%). The ACT shows a similar content
emphasis as that of the SAT II Math Level IC; 22% of its items assess elementary algebra
and 25% assess planar geometry. The SAT II Math Level IIC, on the other hand, draws
from more advanced content areas, such as intermediate algebra (22%) and trigonometry
(18%).
In terms of cognitive demands, all four tests assess procedural knowledge to a
similar degree. Procedural knowledge items constitute between 54% and 58% of the
items found on college admissions measures. However, there is more variation among
the exams with respect to emphasis on problem solving. The SAT I and SAT II Math
Level IIC place relatively greater emphasis on problem solving (20% and 15%,
respectively) than do the ACT and SAT II Math Level IC (7% and 8%, respectively).
College Placement Tests
We examined two college placement tests, the ACCUPLACER Elementary
Algebra and Towson/UOM Math Placement Test, both of which are multiple-choice
measures. Contextualized questions comprise a moderate proportion of the
ACCUPLACER Elementary Algebra (25%), but are absent from the Towson/UOM Math
Placement Test. Students are rarely asked to work with graphs or diagrams on either test.
Questions that contain graphs within the item-stem constitute 6% of the Towson/UOM
Math Placement Test items and 0% of the ACCUPLACER Elementary Algebra items.
Questions that contain diagrams within the item-stem comprise 13% of items on both
tests. There is more variation with respect to formulas, as items requiring a memorized
formula comprise 13% of items on the ACCUPLACER Elementary Algebra, but 25% on
the Towson/UOM Math Placement Test.
In terms of content areas, elementary algebra comprises the majority of the
ACCUPLACER Elementary Algebra (88%), but only a small share of the Towson/UOM
104
Math Placement Test (13%). Instead, the Towson/UOM Math Placement Test focuses on
intermediate algebra (31%), and to a lesser extent, prealgebra and trigonometry (19% on
each content area). The differences in content emphasis between these two measures
reflect the fact that the ACCUPLACER Elementary Algebra is used to determine whether
students need to enroll in remedial math courses, whereas the Towson/UOM Math
Placement Test is used to place students into a wider range of math courses, including
higher-level courses such as calculus.
With respect to cognitive demands, both measures focus on procedural knowledge
(75%-100%). However, the Towson/UOM Math Placement Test also includes a
moderate proportion of conceptual understanding items (25%).
End-of-Course Tests
There are two end-of-course measures that focus on the same content area, the
HSA Algebra and ACCUPLACER Elementary Algebra.6 Both are multiple-choice tests,
but the HSA Algebra is a paper-and-pencil test, whereas the ACCUPLACER Elementary
Algebra is a computer-adaptive measure. The HSA Algebra contains a higher proportion
of problems framed in a real-life context (89%) than does the ACCUPLACER
Elementary Algebra (25%). Questions that contain graphs within the item-stem are
prevalent on the HSA Algebra (37%), but missing from the ACCUPLACER Elementary
Algebra (0%). However, the proportion of questions with diagrams within the item-stem
is comparable across both tests (11% for the HSA Algebra and 13% for the
ACCUPLACER Elementary Algebra). With respect to formulas, 13% of the
ACCUPLACER Elementary Algebra items require a memorized formula but these types
of items are absent from the HSA Algebra.
Although both tests focus on elementary algebra, the ACCUPLACER Elementary
Algebra contains a higher proportion of such items (88%) than does the HSA Algebra
(53%). Similarly, a majority of items on both tests assess procedural knowledge, but
these items comprise a larger proportion of the ACCUPLACER Elementary Algebra
(100%) than the HSA Algebra (68%).
6
Although HSA Geometry is also an end-of-course exam, it is eliminated from this analysis because it does
not focus on the same content area as the two other end-of-course exams.
105
Alignment Among Tests with Different Functions
With the exception of the SAT I, HSA Algebra, and HSA Geometry, none of the
math assessments requires students to generate their own answers. Questions framed
within a realistic context represent a small to moderate proportion of college admissions
(12%-25%) and college placement tests (0%-25%), but represent a large proportion of
state achievement tests (65%-89%). Questions that contain graphs within the item-stem
are typically uncommon, comprising less than 12% of college admissions and college
placement tests. However, they comprise 24% of the HSA Geometry and 37% of the
HSA Algebra items. Diagrams are included on every measure that we examined, but
usually constitute only a small or moderate fraction of a test. Questions that contain
diagrams within the item-stem represent 2%-26% of college admissions items and 13%
of college placement items. On state achievement measures, questions that contain
diagrams within the item-stem are included on 11% of the HSA Algebra questions and
65% of the HSA Geometry questions. Items calling for formulas (either memorized or
provided) are also relatively infrequent, usually comprising less than 25% of a given test.
The exception is the HSA Geometry, where 41% of the items require a formula that has
been provided.
With respect to the content category, HSA Algebra, HSA Geometry, and
ACCUPLACER Elementary Algebra are course-specific exams and therefore narrowly
focus their content on a single area (namely elementary algebra or geometry). In
contrast, college admissions exams and the Towson/UOM Math Placement Test are more
broadly distributed across multiple content areas. Excluding the SAT I, every content
area included in our content category is represented on the college admissions measures,
and 6 of the 8 content areas are assessed on the Towson/UOM Math Placement Test. The
SAT I does not assess trigonometry and the Towson/UOM Math Placement Test does not
assess statistics or miscellaneous topics.
Although college admissions exams and the Towson/UOM Math Placement Test
generally assess the same content areas, there are differences in emphasis. Excluding the
SAT II Math Level IIC, college admissions exams focus on elementary algebra and
planar geometry most often. In contrast, the Towson/UOM Math Placement Test, like
the SAT II Math Level IIC, emphasizes intermediate algebra and trigonometry.
106
In terms of cognitive requirements, all tests emphasize procedural knowledge,
although there is variation with respect to extent. Procedural knowledge items are most
common on college placement tests (75%-100%), followed by state achievement tests
(68%-71%), and least common on college admissions tests (53%-58%). Problemsolving items are most prevalent on state achievement measures (24%-32%), whereas
conceptual understanding items are most prevalent on college admissions exams (26%40%).
Discussion
Below, we discuss the implications of the discrepancies among the math
assessments. We begin by highlighting instances in which differences are justifiable,
then address whether there were any misalignments that may send students confusing
signals. We also explore the possibility that state achievement tests can inform
postsecondary decisions.
Which Discrepancies Reflect Differences in Test Use?
As noted in Chapter 1, content discrepancies may reflect differences in intended
test use. To illustrate, consider the Towson/UOM Math Placement Test and HSA
Algebra. The Towson/UOM Math Placement Test includes topics from a wide variety of
courses, whereas the HSA Algebra contains items primarily from a single content area.
In this particular case, the two tests have disparate functions, and content differences
reflect variations in purpose. The Towson/UOM Math Placement Test is used to
determine the courses that students are prepared to enroll in, given their prior
background. Because this test must accommodate students who are very well prepared,
as well as students who may not have a strong math background, the test must include
items from an array of content areas, from basic (i.e., prealgebra) to advanced (i.e.,
trigonometry). The HSA Algebra, on the other hand, is a measure of proficiency of one
specific course. Consequently, it is warranted that the HSA Algebra limits its content to
a narrow area of math.
The above example represents justifiable discrepancies across tests with different
purposes. However, there are also instances in which discrepancies within tests of
107
similar purposes are warranted as well. Although diagrams comprise 11% of the HSA
Algebra items, but 65% of the HSA Geometry items, this discrepancy is not a
misalignment. Instead, it is indicative of the latter exam’s content focus, which
emphasizes figural relations more so than any other math content area (Fischbein, 1993).
Likewise, that the SAT I places greater emphasis on problem-solving and non-routine
logic problems, whereas the ACT places greater emphasis on procedural knowledge and
textbook-like items is warranted given that the SAT I is intended to be a reasoning
measure, and the ACT is intended to assess content knowledge found in high-school math
courses.
Is There Evidence of Misalignment?
In our analysis of the math tests, we could not find any examples of
misalignments, as discrepancies among college admissions, college placement, and state
achievement measures are either small or moderate, and could generally be predicted a
priori. To illustrate, consider that open-ended items are included on the HSAs but are
absent from college admissions measures. As noted in Chapter 1, the inclusion of openended items on state achievement exams is indicative of attempts to use these tests as
levers of instructional reform. College admissions exams, on the other hand, exclude
open-ended items because such items can potentially undermine the public’s perceptions
of these tests as “objective” measures in which to make fair comparisons of student
proficiency.7 Format differences, in this case, are not misalignments.
Can State Achievement Tests Inform Postsecondary Admissions and Course Placement
Decisions?
Although there are many discrepancies among exams of different functions, it
may still be possible that one test can serve multiple purposes satisfactorily. Currently,
some measures are used for more than one purpose. Many postsecondary institutions, for
example, allow students to submit scores from college admissions exams such as the SAT
I or ACT as a means of exemption from a remedial college placement test. Potentially,
state achievement tests, such as the HSAs, can be used for similar purposes.
7
Open-ended items are also excluded because they are more costly than multiple-choice items.
108
Policymakers have advocated using HSA scores for purposes beyond monitoring student
achievement (Schmidt, 2000) because such a policy change would not only reduce testing
burden, but it would also motivate students to focus on state standards rather than on
external tests like the SAT I or ACT. (Healy, 2001; Olson, 2001a; Standards for Success,
2001). Below, we discuss the potential of the HSA Algebra and HSA Geometry for
college placement and admissions decisions. We also discuss the possibility that testing
burden can be eased by using scores from the SAT II Math Level IIC for other purposes,
namely placement into an appropriate math course.
By virtue of their narrow test content, neither the HSA Algebra nor the HSA
Geometry can be used to inform admissions decisions. For the same reason, the HSAs
cannot be used for broad course placement decisions, such as determining whether an
examinee has the necessary proficiency to enroll in a high-level math course.
However, the HSAs hold more promise as alternatives to remedial college
placement measures. The HSA Algebra, for example, can be used in place of the
ACCUPLACER Elementary Algebra. Both focus on elementary algebra, but the HSA
Algebra contains a higher proportion of problem-solving items. Conceivably, a
sufficiently high score on the HSA Algebra should logically excuse students from having
to take the ACCUPLACER Elementary Algebra.
It may be possible to ease students’ testing burden in other ways beyond
expanding the use of HSA scores. For instance, the SAT II Math Level IIC can be used
to place students into an appropriate math course up to calculus. The SAT II Math Level
IIC has roughly the same content distribution as the Towson/UOM Math Placement Test,
and contains approximately the same proportion of trigonometry items. Potentially,
academic counselors can use scores from the SAT II Math Level IIC to advise students
which math course is most appropriate. To determine the feasibility of the SAT II Math
Level IIC as a measure that informs placement decisions, more research is needed to
explore the relationship between the SAT II Math Level IIC and the Towson/UOM Math
Placement Test.
109
Alignment Among Maryland ELA Assessments
Below we present the ELA results. As with math, we discuss discrepancies both
within and across test functions. The results are also organized by skill, namely reading,
editing, and writing. In some instances, there are only two tests in a given category, so it
is important to keep in mind that patterns or comparisons may not be representative of
more general trends within this category of tests.
Alignment is characterized by describing differences with respect to technical
features, content, and cognitive demands. Specifically, we discuss differences in time
limit and format, then document discrepancies with respect to topic, voice, and genre of
the reading passages, before concluding with variations in cognitive processes.
The alignment results for tests that measure reading skills are presented in Tables
5.4-5.5. Tables 5.6-5.7 provide the results for exams that assess editing skills, and Tables
5.8-5.9 provide the findings for exams that assess writing skills. For each table, the
numbers represent the percent of items falling in each category. To provide a concrete
example of how to interpret the findings, consider the content category results for the AP
Language and Composition, presented in Table 5.4. With respect to topic, 50% of the
reading passages included on the AP Language and Composition are personal accounts,
whereas 25% of the topics are about humanities, and the remaining 25% are about natural
science. It does not include topics from fiction or social science (0% each). In terms of
the author’s voice, 75% of the passages are written in a narrative style, whereas the other
25% are written in an informative manner. With respect to genre, only essays (100%) are
used; passages on the AP Language and Composition are not presented as letters, poems,
or stories (0% each). Results for the other tests are interpreted in a similar manner.
110
111
0
20
63
AP Language and
Composition
SAT I
SAT II Literature
0
0
ACCUPLACER Reading
Comprehension
Nelson-Denny Reading Test
College Placement Tests
25
0
Fiction
ACT
College Admissions Tests
HSA English
State Achievement Test
Test
43
60
0
40
25
25
67
29
35
0
20
25
25
0
111
29
0
13
20
0
25
0
Humanities Natural Social
Science Science
Topic
0
5
25
0
50
0
33
Personal
Accounts
0
50
100
40
75
50
100
0
0
0
0
0
0
0
0
5
0
0
0
0
0
Narrative Descriptive Persuasive
Voice
100
45
0
60
25
50
0
Informative
Table 5.4
Alignment Within the Content Category for the Reading Passages
0
0
13
0
0
0
33
Letter
100
100
25
80
100
75
0
Essay
0
0
50
0
0
0
33
Poem
Genre
0
0
13
20
0
25
33
Story
Reading Measures
Alignment Among Tests of the Same Function
State Achievement Tests
The only state achievement ELA test included in our sample is the HSA English,
which assesses reading proficiency solely with multiple-choice items (see Table 5.2).
Most of the reading passages on the HSA English are about humanities (67%), written in
a narrative style (100%) (see Table 5.4). With respect to genre, passages are split evenly
among letters, poems, and stories (33% on each). The HSA English reading items assess
inference skills (75%) most frequently, although a moderate proportion of questions also
assess recall skills (25%) (see Table 5.5).
Table 5.5
Alignment Within the Cognitive Demands Category for Tests
Measuring Reading Skills
Test
Recall
Inference
Evaluate Style
25
75
0
ACT
58
42
0
AP Language and Composition
23
77
0
SAT I
18
83
0
SAT II Literature
13
80
7
ACCUPLACER Reading Comprehension
11
89
0
Nelson-Denny Reading Test
86
15
0
State Achievement Test
HSA English
College Admissions Tests
College Placement Tests
College Admissions Tests
Four college admissions exams assess reading proficiency: the ACT, AP
Language and Composition, SAT I, and SAT II Literature. With the exception of the AP
Language and Composition, no college admissions test assesses reading skills with openended items. Testing time devoted to measuring reading skills is 60 minutes for both the
SAT II Literature and AP Language and Composition. Because the SAT I does not
contain separate sections for editing and reading items, we cannot determine testing time
112
earmarked specifically for assessing reading proficiency, although testing time devoted to
assessing both types of skills is 75 minutes (see Table 5.2).
Reading passage topics also vary from one measure to the next (see Table 5.4).
The SAT II Literature emphasizes fiction (63%) whereas the AP Language and
Composition emphasizes personal accounts (50%). The SAT I favors humanities (40%),
but the ACT is evenly distributed among fiction, humanities, natural science, and social
science (25% each). Narrative pieces are included on all college admissions measures,
and range from 40% of the SAT I passages to 100% of the SAT II Literature passages.
Essay is generally the most common genre, appearing on 75% of the ACT, 80% of the
SAT I, and 100% of the AP Language and Composition passages. However, the SAT II
Literature is more likely to include poems (50%) than essays (25%). With the exception
of the ACT, college admission exams place most emphasis on interpretation and analysis
of the reading passages. Inference items range from 42% of the ACT questions to 83%
of the SAT I questions (see Table 5.5).
College Placement Tests
Two college placement tests, the ACCUPLACER Reading Comprehension and
Nelson-Denny Reading Test, contain reading items. Both assess reading proficiency
solely with multiple-choice items. The ACCUPLACER Reading Comprehension is
computer-adaptive measure, so testing time varies by examinee. In contrast, the NelsonDenny Reading Test is a paper-and-pencil exam that is administered in approximately
one hour (see Table 5.2).
Both the ACCUPLACER Reading Comprehension and Nelson-Denny Reading
Test favor reading passages about humanities (60% and 43%, respectively), but the
ACCUPLACER Reading Comprehension passages are written in an informative voice
(100%), whereas the Nelson-Denny Reading Test passages are written in either a
narrative or informative voice (50% and 45%, respectively) (see Table 5.4). There are no
differences with respect to genre, as both exams present passages only as essays. For the
cognitive demands category, the ACCUPLACER Reading Comprehension emphasizes
inference to a great extent (89%), but the Nelson-Denny Reading Test focuses on recall
(86%) (see Table 5.5).
113
Alignment Among Tests of Different Functions
Across all measures, reading skills are assessed primarily with multiple-choice
items. Testing time devoted specifically to assessing reading skills ranges from 35
minutes for the ACT to 60 minutes for the SAT II Literature and AP Language and
Composition. All assessments contain reading passages on two or more topics, and
every reading assessment includes a topic from humanities except the SAT II Literature.
Every test also contains either a narrative passage or an informative passage, and the
majority includes both. Essay is the most prevalent genre, comprising the majority of
college placement (100%) and college admissions tests (75%-100%, the SAT II
Literature exam notwithstanding). Essays, however, are absent from the HSA English.
Instead, the HSA English uses letters, poems, and stories (33% each).
Of the reading measures, only the Nelson-Denny Reading Test does not include a
significant proportion of inference items (15%). Items assessing inference comprise 75%
of the HSA English, 89% of the ACCUPLACER Reading Comprehension, and 42%-83%
of college admissions measures.
Editing Measures
Alignment Among Tests of the Same Function
State Achievement Test
The HSA English is the only measure that falls within this category. As with
reading, editing proficiency is assessed only with multiple-choice items (see Table 5.2).
All of its reading passages are social science topics written in an informative style, and
presented as essays (see Table 5.6). In terms of cognitive demands, the vast majority of
its items measures evaluate style skills (73%) (see Table 5.7).
114
115
ACCUPLACER Sentence
Skills
ASSET Writing Skills
College Placement Tests
SAT II Writing
SAT I
ACT
College Admissions Tests
HSA English
State Achievement Test
Test
0
0
0
0
Fiction
100
100
60
0
N/A
0
0
N/A
20
0
115
0
0
0
100
Humanities Natural Social
Science Science
Topic
0
0
20
0
Personal
Accounts
67
50
40
0
0
0
0
0
N/A
N/A
0
0
0
0
Narrative Descriptive Persuasive
Voice
33
50
60
100
Informative
Table 5.6
Alignment Within the Content Category for the Editing Passages
0
0
0
0
Letter
100
100
100
100
N/A
0
0
0
0
Poem
N/A
Essay
Genre
0
0
0
0
Story
Table 5.7
Alignment Within the Cognitive Demands Category for Tests
Measuring Editing Skills
Test
Recall
Inference
Evaluate Style
13
13
73
ACT
48
4
48
SAT I
0
100
0
SAT II Writing
50
3
47
ACCUPLACER Sentence Skills
91
0
9
ASSET Writing Skills
89
11
0
State Achievement Test
HSA English
College Admissions Tests
College Placement Tests
College Admissions Tests
Items measuring editing skills are included on three college admissions tests, the
ACT, SAT I, and SAT II Writing. The exams are predominantly multiple-choice, with
testing time ranging from 40 minutes for the ACT to 45 minutes for the SAT II Writing
(see Table 5.2). As mentioned earlier, the SAT I does not specify the specific amount of
testing time devoted to measuring editing skills.
The SAT I does not include a reading passage, but instead uses a few sentences as
prompts. In contrast, the ACT and SAT II Writing include reading passages. These
reading passages are typically essays about humanities, and written in either a narrative
or informative voice (see Table 5.6). The ACT and SAT II Writing items are equally
distributed among recall (48% and 50%, respectively) and evaluate style items (48% and
47%, respectively), but the SAT I assesses only inference skills (100%) (see Table 5.7).
College Placement Tests
Two college placement tests, the ASSET Writing Skills and ACCUPLACER
Sentence Skills, assess editing skills via multiple-choice items. The ASSET Writing
Skills allow students 25 minutes to complete the test, but the ACCUPLACER Sentence
Skills is a computer-adaptive measure that does not specify a time limit (see Table 5.2).
The ACCUPLACER Sentence Skills uses sentences as prompts, whereas the ASSET
116
Writing Skills includes reading passages. Reading passes on the ASSET Writing Skills
are typically narrative essays about a humanities topics. In terms of cognitive demands,
both tests focus heavily on recall, and to approximately the same extent (89%-91%).
However, the ASSET Writing Skills also includes inference items (11%), whereas the
ACCUPLACER Sentence Skills contains evaluate style items (9%).
Alignment Among Tests of Different Functions
Across all tests, editing skills are assessed solely with multiple-choice items. All
measures, except the SAT I and ACCUPLACER Sentence Skills, include reading
passages as a prompt. (The SAT I and ACCUPLACER Sentence Skills use sentences as
prompts). Of those measures that include a reading passage, all presented passages are
essays, written in either a narrative or informative voice. More variation is observed with
respect to reading passage topics. The HSA English favors social science (100%)
whereas humanities topics are more likely to appear on the ACT, SAT II Writing, and
ASSET Writing Skills (60%, 100%, and100%, respectively). The ACT also draws from
natural science and personal accounts (20% each), topics that are not represented on any
other editing measure.
None of the editing exams assesses the full spectrum of the cognitive demands
category. The ACT and SAT II Writing emphasize recall and evaluate style items, but
are generally devoid of inference items, whereas the reverse is true for the SAT I.
College placement measures focus heavily on recall items (89%-91%) and rarely include
inference and evaluate style items. The HSA English includes a large proportion of
evaluate style items (73%), but does not contain many recall or inference items (13%
each).
Writing Measures
Alignment Among Tests of the Same Function
State Achievement Tests
The HSA English requires students to provide a writing sample (see Table 5.2).
Topics are typically drawn from fiction or personal accounts (see Table 5.8). With
117
respect to scoring criteria, the HSA English requires students to demonstrate mechanics,
word choice, style, organization, and insight (see Table 5.9).
Table 5.8
Alignment Among the Writing Prompt Topics
Topic
Test
Fiction
Humanities
Natural Science
Social Science
Personal Accounts
State Achievement Test
HSA English
X
X
College Admissions Tests
AP Language and Composition
X
SAT II Writing
X
X
Table 5.9
Alignment Among the Scoring Criteria for Tests Measuring Writing Skills
Scoring Criteria Elements
Test
Mechanics
Word Choice
Organization
Style
Insight
X
X
X
X
X
AP Language and Composition
X
X
X
X
X
SAT II Writing
X
X
X
X
State Achievement Test
HSA English
College Admissions Tests
College Admissions Tests
Of the college admissions measures, only the SAT II Writing and AP Language
and Composition require a writing sample. The SAT II Writing provides students with a
one- or two-sentence writing prompt on a topic (usually humanities), and allows 20
minutes for students to respond (see Tables 5.2 and 5.8). In contrast, prompts on the AP
Language and Composition are typically reading passages, and students are required to
provide a total of three writing samples in over two hours (see Table 5.2).8 Topics can
8
The AP Language and Composition requires a total of three writing samples, two of which are produced
during the 120-minute writing session, and one during the 60-minute reading session. However, because
examinees also respond to a set of multiple-choice items during the reading session, it is unknown the
amount of time students devote specifically to the writing sample.
118
vary, but are usually about humanities or personal accounts (see Table 5.8). The AP
Language and Composition emphasizes all elements of the scoring criteria, but the SAT
II Writing downplays the importance of insight (see Table 5.9).
College Placement Tests
None of the college placement tests analyzed for this case study site requires a
writing sample.
Alignment Among Tests of Different Functions
Writing measures can vary from 20 minutes for a single writing sample (SAT II
Writing) to over 2 hours for three writing samples (AP Language and Composition).
Humanities and personal accounts are the most common topics, and every test includes a
writing prompt from at least one of these areas. The HSA English and AP Language and
Composition measures emphasize mechanics, word choice, organization, style, and
insight, but the SAT II Writing omits insight from its scoring criteria.
Discussion
Our discussion of the discrepancies among ELA assessments parallels that of the
math discussion. We first identify examples of discrepancies that are justifiable, then
discuss the implications of the misalignments. We also discuss the feasibility of using
state achievement tests to inform admission decisions.
Which Discrepancies Reflect Differences in Test Use?
As in math, some discrepancies among the ELA assessments reflect differences in
purpose. Consider, for instance, discrepancies between the scoring standards of the HSA
and the AP Language and Composition. For the former test, maximum scores are
awarded to writing samples that have minor diction errors, mechanics lapses, and
underdeveloped paragraphs. Under the AP Language and Composition guidelines, such
compositions might receive adequate scores, but would not be viewed as exemplary
papers. Because the AP Language and Composition is used to award academic credit to
students who demonstrate college-level proficiency, whereas the HSA is used to monitor
119
the achievement of all students within the state, including those not planning to attend a
postsecondary institution, discrepancies between their scoring criteria are warranted.
Even when two measures have similar test functions, discrepancies may still be
warranted. For example, the large discrepancy between the SAT I (100%) and the ACT
(4%) and the SAT I and SAT II Writing (3%) with respect to inference items is
attributable to subtleties in purpose. The SAT I is intended to be a measure of reasoning
proficiency, so great emphasis on inference questions is justifiable. The ACT and SAT II
Writing, on the other hand, are curriculum-based measures, so relatively greater focus on
skills learned within English classes (i.e., recall and evaluate style skills) is to be
expected.
Is There Evidence of Misalignment?
Although the vast majority of the ELA discrepancies is small or moderate, or
stems from variations in test function, two instances of misalignment were observed. The
first misalignment pertains to the scoring criteria of the SAT II Writing. Insight is
included within the scoring criteria of the HSA English and AP Language and
Composition, but is omitted from the scoring rubrics of the SAT II Writing. Given that
insight is included in the standards of most English courses, it appears that the SAT II
Writing standards are incongruent with those that are typically expressed. Potentially,
this misalignment can send students mixed messages about the importance of insight with
respect to writing skills. If the developers of the SAT II Writing were to add insight to
the scoring criteria, or provided a clear rationale of why insight has been omitted from the
scoring rubrics, students would receive a more consistent signal about the importance of
insight with respect to writing proficiency.
The other misalignment involves the large discrepancies in cognitive demands
elicited by the Nelson-Denney Reading Test and the ACCUPLACER Reading
Comprehension. The former test assesses recall most frequently (86%), whereas the
latter exam emphasizes inference (89%). Given that the two tests are both placement
120
measures used to determine the most appropriate course for students given their prior
background, it is unclear why this difference arises.9
Can State Achievement Tests Inform Postsecondary Admissions Decisions?
As mentioned earlier, policymakers are exploring the possibility that scores on
graduation tests can be used to inform college admissions decisions. In reading, that the
HSA English assesses inference skills to the same extent as college admission measures
may suggest that the HSA English might be a viable alternative to college admissions
tests. However, inference items can vary with respect to cognitive sophistication elicited.
A previous study by Education Trust (1999) showed that ELA inference items could vary
greatly with respect to nuance of interpretations. Given the differences in the intended
test uses, it is very likely that inference items on the HSA English may not be as complex
as that elicited by college admissions exams. More research needs to be conducted to
determine whether the HSA English can discriminate among higher-achieving examinees
as well as college admissions exams.
With respect to assessing writing skills, however, the HSA English is more viable
as an alternative measure to college admissions exams. Neither the ACT nor SAT I
requires a writing sample, and the SAT II Writing allows 20 minutes for a writing
sample. Given the short time limit, the SAT II Writing composition represents a very
limited indicator of writing proficiency. In contrast, the HSA English requires a writing
sample and will likely allow students more than 20 minutes to produce their
composition.10 Arguably, the HSA English would allow admissions officers to better
judge applicants’ writing skill than college admissions exams. However, as discussed
earlier, the current scoring rubrics for the HSA English may not be rigorous enough to be
of use for some institutions, especially the selective ones. Therefore, changes to the
scoring guidelines may need to be implemented if the HSA writing samples were to be
used to inform admissions decisions at these higher-selectivity schools. Again, any
policy changes regarding the use of the HSA English to inform placement or admissions
9
These exams are used at two different institutions, so students are not likely to take both tests.
Although no final decision has been made regarding the time limit of the writing sample, field studies
have allowed at least one hour for students to compose their essay.
10
121
decisions will require more research, particularly the relationship between the HSA
English scores and first-year college grade point average in English courses.
122