2005–2009 CLA INSTITUTIONAL REPORT
Winston-Salem State
University
Acknowledgements and Notes
We would like to thank the
Lumina Foundation for Education
for funding the fall 05 – spring 09
CLA longitudinal study.
We would also like to thank your
institution for participating.
This report presents the results of
your institution across the three
phases of this longitudinal study.
Additional summary findings from
of this study will be published in
the coming months.
Institutional Report
1 Purpose of the CLA (1)
2 Your Results (2-7)
3 Longitudinal Cohort (8-12)
4 Diagnostic Guidance (13)
5 Moving Forward (14)
Appendices
A Task Overview (16-19)
B Task Development (20)
C Scoring Criteria (21-23)
D Scoring Process (24-25)
E Scaling Procedures (26-27)
F Student Data File (28)
G CAE Board of Trustees and Officers (29)
Institutional Report
1: Purpose of the CLA
The Collegiate Learning Assessment
The continuous improvement model
The CLA uses detailed scoring
(CLA) offers an authentic approach
also requires multiple assessment
guides to precisely and reliably
to assessment and improvement
indicators beyond the CLA because
evaluate student responses. It
of teaching and learning in higher
no single test to benchmark student
also encourages institutions to
education. Over 400 institutions and
learning in higher education is
compare their student learning
180,000 students have participated
feasible or desirable.
results on the CLA with learning
to date. Growing commitment on
at other institutions and on other
the part of higher education to assess
This, however, does not mean certain
student learning makes this a good
skills judged to be important by most
time to review the distinguishing
faculty and administrators across
The signaling quality of the CLA
features of the CLA and how it
virtually all institutions cannot be
is important because institutions
connects to improving teaching and
measured; indeed, the higher order
need to benchmark (have a frame
learning on your campus.
skills the CLA focuses on fall into
of reference for) where they stand
this measurable category.
and how much progress their
The CLA is intended primarily to
assessments.
students have made relative to the
assist faculty, department chairs,
The CLA presents realistic problems
progress of students at other colleges.
school administrators and others
that require students to analyze
Otherwise, how do they know how
interested in programmatic change
complex materials. Several different
well they are doing?
to improve teaching and learning,
types of materials are used that vary
particularly with respect to
in relevance to the task, credibility,
Yet, the CLA is not about ranking
strengthening higher order skills.
and other characteristics. Students’
institutions. Rather, it is about
written responses to the task are
highlighting differences between
The CLA helps campuses follow a
graded to assess their abilities to
them that can lead to improvements
continuous improvement model that
think critically, reason analytically,
in teaching and learning.
positions faculty as central actors.
solve problems, and communicate
clearly and cogently.
CLA Education (described on
While the CLA is indeed an
assessment instrument, it is
page 14) does just that by focusing
The institution—not the student—is
deliberately designed to contribute
on curriculum and pedagogy and
the initial primary unit of analysis.
directly to the improvement of
the link between assessment and
The CLA is designed to measure
teaching and learning. In this respect
teaching and learning.
an institution’s contribution, or
it is in a league of its own.
value added, to the development of
these competencies, including the
effects of changes to curriculum and
pedagogy.
1
2: Your Results
Table 1 below provides the number of your students
with both Entering Academic Ability (EAA) scores
and CLA scores in different phases of the longitudinal
assessment.
EAA scores represent SAT Math + Verbal, ACT
Composite, or Scholastic Level Exam (SLE) scores on
the SAT scale.
Per sampling restrictions of Phase 1 of the assessment,
the sample of students throughout the report also have
EAA scores.
1
Number of your students with CLA scores in different longitudinal phases
Phase 1
and
Phase 2
Phase 2
and
Phase 3
Phase 1
and
Phase 3
Performance Task
15
8
41
Analytic Writing Task
4
2
32
Make-an-Argument
5
2
33
Critique-an-Argument
5
3
33
4
2
31
F 2005
F 2005
F 2005
S 2007
S 2007
S 2007
S 2009
S 2009
S 2009
Total Score
Phases referenced
in columns above
2
2: Your Results
For each longitudinal phase, Table 2 below presents
summary statistics for your students including counts,
means, 25th and 75th percentiles, and standard
deviations.
We also calculated these statistics across all students
and all schools. See Tables 7 and 8 on pages 10 and 11.
2
Summary statistics for your students participating in the longitudinal assessment
Number of
Students
25th
Percentile
Mean
Score
75th
Percentile
Standard
Deviation
Performance Task
186
875
983
1099
159
Analytic Writing Task
131
906
983
1055
124
Make-an-Argument
138
801
985
1084
154
Critique-an-Agument
138
869
976
1018
147
Total Score
129
923
994
1061
117
EAA Score
183
830
911
970
109
Number of
Students
25th
Percentile
Mean
Score
75th
Percentile
Standard
Deviation
Performance Task
15
910
1029
1155
151
Analytic Writing Task
5
906
951
980
83
Make-an-Argument
5
942
942
942
100
Critique-an-Agument
Phase 1
Phase 2
6
869
968
1018
122
Total Score
5
908
1014
1066
124
EAA Score
15
910
970
1050
125
Number of
Students
25th
Percentile
Mean
Score
75th
Percentile
Standard
Deviation
Performance Task
42
946
1058
1192
158
Analytic Writing Task
42
952
1045
1127
136
Make-an-Argument
42
911
1042
1163
186
Critique-an-Agument
42
905
1047
1198
158
Total Score
42
953
1052
1139
112
EAA Score
42
830
951
1050
137
Phase 3
3
2: Your Results
Table 3 provides summary statistical data
The larger the positive effect size, the
The last column shows the correlation
on your students across all three phases.
greater the improvement. Effect sizes
between students’ scores in different
We present these data in three groups to
greater than 0.50 are generally considered
phases. A high positive correlation
examine change across different phases.
“large.” Negative effect sizes would indicate
indicates that the students who scored
that the students scored higher during an
relatively highly in one phase (relative
The “effect size” column of Table
earlier phase. To place your performance
to their classmates) also tended to score
3 is particularly important. This
in context, Figure A on page 5 plots these
relatively highly in another phase. In other
column indicates how much change
effect sizes versus all other participating
words, the improvement in scores was fairly
occurred between two different phases.
institutions.
consistent across students.
3
Comparison of your students’ scores in Phase 1 (Fall 2005), Phase 2 (Spring 2007), and Phase 3 (Spring 2009)
Phase 1
Phase 2
Difference
Summary Statistics
Number of
Students
Mean
Score
Standard
Deviation
Mean
Score
Standard
Deviation
Mean
Score
Standard
Deviation
Effect
Size*
Mean Score
Correlation
Performance Task
15
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Analytic Writing Task
4
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Make-an-Argument
5
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Critique-an-Agument
5
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
4
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Number of
Students
Mean
Score
Standard
Deviation
Mean
Score
Standard
Deviation
Mean
Score
Standard
Deviation
Effect
Size
Mean Score
Correlation
Performance Task
8
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Analytic Writing Task
2
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Make-an-Argument
2
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Critique-an-Agument
3
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
2
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Number of
Students
Mean
Score
Standard
Deviation
Mean
Score
Standard
Deviation
Mean
Score
Performance Task
41
1005
163
1055
159
Analytic Writing Task
32
1000
116
1041
140
Make-an-Argument
33
1002
150
1039
Critique-an-Agument
33
986
152
31
1021
113
Phases 1 and 2
Total Score
Phase 2
Phases 2 and 3
Total Score
Phase 3
Phase 1
Phases 1 and 3
Total Score
Difference
Phase 3
Summary Statistics
Difference
Summary Statistics
Standard
Deviation
Effect
Size
Mean Score
Correlation
50
-4
0.31
0.27
41
24
0.35
0.22
194
37
44
0.25
0.13
1053
159
67
7
0.44
0.32
1045
117
24
4
0.21
0.42
* A different method was used previously to calculate effect sizes, so these results differ slightly from those in the 2005–2007 Institutional Report.
Previously, we divided mean differences between phases by the CLA score standard deviation of all participating students in the earlier phase,
which would tend to underestimate effect sizes. Now, we divide by the standard deviation of your participating students in the earlier phase.
4
2: Your Results
Mean Score Changes between Phase 1, Phase 2, and Phase 3 by CLA Measure
1
A
Effect Size Box and Whisker Plots
CLA Measure
Between Phases
Effect Size
-.75
-.50
-.25
0
.25
.50
.75
1.00
1.25
1.50
1.75
-.75
-.50
-.25
0
.25
.50
.75
1.00
1.25
1.50
1.75
Performance Task
1
Analytic Writing Task
2
Make-an-Argument
Critique-an-Argument
Total Score
Performance Task
Analytic Writing Task
2
3
Make-an-Argument
Critique-an-Argument
Total Score
Performance Task
Analytic Writing Task
1
3
Make-an-Argument
Critique-an-Argument
Total Score
Key for Phases
When
Phase
The Figure A
1 “box and whisker” plots show effect size distributions across
CLA measures for longitudinal schools. The top cluster displays effect sizes
between Phase 1 (fall 2005 first-year students) and Phase 2 (spring 2007
rising juniors); the middle cluster repeats this for Phases 2 and 3 (spring 2009
seniors); and the bottom cluster does the same for Phases 1 and 3.
Who
1
Fall 2005
First-year
2
Spring 2007
Rising Junior
3
Spring 2009
Senior
In each plot, the extreme left hand vertical bar shows the 5th percentile. The
“box” itself shows the 25th (left face), 50th (internal vertical line), and 75th
(right face) percentile points. The extreme right hand vertical line shows the
95th percentile. The horizontal x-axis shows the effect size (in standard
deviation units).
Key for Box and Whisker Plots
5th
50th
Your School
Effect sizes are one way to measure change between two time points. Effect
sizes were calculated at a school by taking the difference in mean (or average)
CLA scores of the same students from two different phases (e.g., Phase 1 and
Phase 3). This difference is then divided by the spread of scores (standard
deviation) from the first phase (e.g., Phase 1) to produce an effect size.
95th
Circles
25th
identify effect sizes for your school (see Table 53 for values).
75th
5
2: Your Results
Table 4 below indicates whether students
Colleges with scores between -1.00 and
Figure B on the next page displays your
scored well above, above, at, below, or well
+1.00 standard errors from their expected
deviation scores in a scatterplot and
below what would be expected given their
scores are categorized as At Expected.
documents the regression equations from
scores as first-year students. Results are
Institutions with scores greater than one
which they were derived.
expressed in the form of value-added scores
standard error (but less than two standard
that correspond to standard errors.
errors) from their expected scores are in
The two regression equations (Phase 1 to
the Above Expected or Below Expected
Phase 2 in blue; Phase 1 to Phase 3 in red)
categories (depending on the direction
are each based on data from all institutions
of the deviation). The schools with scores
where at least 25 students received CLA
greater than two standard errors from
total scores in both phases in the model.
their expected scores are in the Well
Therefore, the sample of institutions is not
Above Expected or Well Below Expected
identical across the testing phases reported
categories.
here, but the use of inclusive samples
maximizes the strength of each equation.
4
Value added results for your school
Deviation
Performance
Deviation
Performance
Score
Level
Score
Level
Performance Task
N/A
N/A
-0.2
At
Analytic Writing Task
N/A
N/A
-0.8
At
Make-an-Argument
N/A
N/A
-0.6
At
Critique-an-Argument
N/A
N/A
-0.8
At
N/A
N/A
-0.9
At
F 2005
F 2005
F 2005
F 2005
S 2007
S 2007
S 2007
S 2007
S 2009
S 2009
S 2009
S 2009
Total Score
Phases referenced
in columns above
6
2: Your Results
B
Relationship between Spring 2009, Spring 2007, and Fall 2005 CLA Performance
1500
First-year (Phase 1) to Rising Junior (Phase 2)
First-year (Phase 1) to Senior (Phase 3)
CLA Total Score (Phase 2 or Phase 3)
1400
1300
1200
1100
1000
900
Regression
Intercept
Slope
R-square
800
800
900
1000
1100
1200
1300
55 -85
0.97 1.15
0.73 0.80
1400
1500
CLA Total Score (Phase 1)
The vertical y-axis in Figure B above
The horizontal x-axis shows a school’s
A school’s data point is above its
shows a school’s mean CLA Total score
mean CLA Total score for Phase 1.
corresponding line if its students in
for both Phase 2 (in blue) and Phase 3
Phase 2 (or Phase 3) performed better
(in red).
on the CLA than what would be
expected relative to their Phase 1 scores.
7
3: Longitudinal Cohort
During the spring 2009 testing cycle, 31 institutions
tested a sufficient number of students to provide the
school-level analyses and results presented in this
report.
Table 5 shows CLA longitudinal schools grouped
by Basic Carnegie Classification. The spread of
schools differs slightly from that of the 1,713 fouryear institutions across the nation, with Doctorategranting Universities constituting a higher percentage
among CLA schools than nationally. Accordingly,
representation among both Master’s Colleges and
Universities as well as Baccalaureate Colleges is lower
among CLA longitudinal schools than it is nationally.
5
Nation
Carnegie Classification
CLA
Number
Percentage
Number
Percentage
Doctorate-granting Universities
282
16%
12
39%
Master’s Colleges and Universities
664
39%
9
29%
Baccalaureate Colleges
767
45%
10
32%
1,713
31
Source: Carnegie Foundation for the Advancement of Teaching, Carnegie Classifications
Data File, June 11, 2008.
8
3: Longitudinal Cohort
Table 6 provides comparative statistics for colleges and
universities across the nation and CLA longitudinal
schools in terms of some important characteristics.
These statistics suggest that CLA longitudinal
schools are fairly representative of institutions
nationally. Percentage public, percentage HBCU, and
undergraduate student body size are exceptions.
6
School Characteristic
Nation
CLA
Percentage public
37%
52%
Percentage Historically Black College or University (HBCU)
5%
13%
Mean percentage of undergraduates receiving Pell grants
34%
31%
Mean four-year graduation rate
36%
40%
Mean six-year graduation rate
52%
61%
Mean first-year retention rate
73%
82%
Mean Barron’s selectivity rating
3.4
3.8
1067
1106
Mean estimated median SAT score
Mean number of FTE undergraduate students (rounded)
Mean student-related expenditures per FTE student (rounded)
4,320
9,040
$12,365
$14,180
Source: College Results Online dataset, managed by and obtained with permission from the Education Trust, covers
most 4-year Title IV-eligible higher-education institutions in the United States. Data were constructed from IPEDS
and other sources. Because all schools did not report on every measure in the table, the averages and percentages
may be based on slightly different denominators.
9
3: Longitudinal Cohort
For each longitudinal phase, Table 7 below presents
summary statistics for all students including counts,
means, 25th and 75th percentiles, and standard
deviations.
7
Student
Count
25th
Percentile
Mean
Score
75th
Percentile
Standard
Deviation
Performance Task
11437
971
1087
1209
189
Analytic Writing Task
9221
977
1085
1196
165
Make-an-Argument
9879
942
1080
1225
188
Critique-an-Agument
9627
869
1079
1167
186
Total Score
9168
990
1096
1198
149
EAA Score
11360
950
1093
1230
195
Student
Count
25th
Percentile
Mean
Score
75th
Percentile
Standard
Deviation
Performance Task
3327
1012
1157
1277
207
Analytic Writing Task
3161
980
1119
1267
161
Make-an-Argument
3227
942
1114
1225
181
Critique-an-Agument
3202
1018
1120
1316
180
Total Score
3141
1025
1141
1249
160
EAA Score
3329
1010
1134
1270
188
Student
Count
25th
Percentile
Mean
Score
75th
Percentile
Standard
Deviation
Performance Task
2374
1093
1216
1337
194
Analytic Writing Task
2308
1091
1221
1363
186
Make-an-Argument
2318
1051
1199
1354
214
Critique-an-Agument
Phase 1 (student level)
Phase 2 (student level)
Phase 3 (student level)
2317
1083
1240
1393
210
Total Score
2290
1115
1222
1339
163
EAA Score
2380
1030
1153
1300
184
10
3: Longitudinal Cohort
For each longitudinal phase, Table 8 below presents
summary statistics for all schools including counts,
means, 25th and 75th percentiles, and standard
deviations.
8
School
Count
25th
Percentile
Mean
Score
75th
Percentile
Standard
Deviation
Performance Task
48
1027
1094
1168
104
Analytic Writing Task
47
1021
1089
1164
98
Make-an-Argument
48
1018
1081
1159
103
Critique-an-Agument
48
1006
1084
1153
103
Total Score
47
1038
1100
1170
93
EAA Score
48
1011
1104
1209
152
School
Count
25th
Percentile
Mean
Score
75th
Percentile
Standard
Deviation
Performance Task
32
1068
1147
1194
103
Analytic Writing Task
32
1057
1114
1170
93
Make-an-Argument
32
1062
1111
1166
94
Critique-an-Agument
32
1051
1114
1160
94
Total Score
32
1073
1132
1185
96
EAA Score
32
1044
1116
1193
129
School
Count
25th
Percentile
Mean
Score
75th
Percentile
Standard
Deviation
Performance Task
31
1126
1193
1256
107
Analytic Writing Task
30
1120
1201
1267
109
Make-an-Argument
30
1095
1181
1251
109
Critique-an-Agument
Phase 1 (school level)
Phase 2 (school level)
Phase 3 (school level)
30
1152
1219
1281
111
Total Score
30
1138
1202
1267
104
EAA Score
32
1049
1126
1217
137
11
3: Longitudinal Cohort
The institutions listed here in alphabetical order
agreed to be identified as participating schools. All
participated in Phase 3 of the longitudinal assessment,
but some may not have tested enough students to be
included in all comparative analyses.
Auburn University
Belmont University
Bowling Green State University
California State Polytechnic University-Pomona
California State University-Northridge
Carleton College
Central Michigan University
City University of New York City College
City University of New York Herbert H. Lehman College
Cleveland State University
Colorado College
Earlham College
Fayetteville State University
Grand Valley State University
Kalamazoo College
Loyola University of Chicago
Macalester College
Morgan State University
North Carolina Agricultural and Technical State University
Northern Arizona University
Pace University
Saint Xavier University
Spelman College
St. Olaf College
Syracuse University
The George Washington University
The Ohio State University
University of California, Riverside
University of Charleston
University of North Carolina at Charlotte
University of Saint Thomas
University of the Pacific
University of Wyoming
Wagner College
Winston-Salem State University
Wofford College
12
4: Diagnostic Guidance
CLA results operate as a signaling tool of overall institutional performance on tasks that measure
higher order skills holistically. However, the three types of CLA tasks—Performance, Make-anArgument, and Critique-an-Argument—differ slightly in the combination of skills necessary to
perform well.
Indeed, some schools score significantly lower on one type than on another. Examining
performance across CLA task types can serve as an initial diagnostic exercise. Specifically, cases of
performance Well Below or Below Expected on a particular task type indicate that students are
not demonstrating the expected level of skill at analyzing complex, realistic scenarios; writing a
persuasive, analytic essay to support a position on an issue; and/or critiquing written arguments.
Performance Task
Make-an-Argument
Critique-an-Argument
Analyzing
complex, realistic scenarios
Writing
a persuasive, analytic essay
Critiquing
written arguments
Synthesizing information from multiple
Establishing a thesis or a position on an
Identifying a variety of logical flaws or
sources; recognizing conflicting
issue; maintaining the thesis throughout
fallacies in a specific argument; explaining
evidence, weighing the credibility of
the essay; supporting the thesis with
how or why the logical flaws affect
different sources of evidence; identifying
relevant and persuasive examples (e.g.,
the conclusions in that argument; and
logical fallacies, interpreting data,
from personal experience, history,
presenting their critique in a written
tables, and figures correctly; drawing
art, literature, pop culture, or current
response that is a grammatically correct,
reasonable and logical inferences from
events); anticipating and countering
organized, well-developed, logically
the available information; developing
opposing arguments to the position,
sound, and neutral in tone.
sound conclusions based on all available
fully developing ideas, examples, and
evidence; and utilizing the most relevant
arguments; crafting an overall response
and credible evidence available to justify
that generates interest, provokes thought,
their conclusion.
and persuades the reader; organizing the
structure of the essay (e.g., paragraphing,
ordering of ideas and sentences within
paragraphs); employing transitions and
varied sentence structure to maintain
the flow of the argument; and utilizing
sophisticated grammar and vocabulary.
13
5: Moving Forward
We encourage institutions to examine
CLA Education focuses on curriculum
Through the steps noted above we
performance across CLA tasks and
and pedagogy and embraces the crucial
encourage institutions to move toward
communicate results across campus,
role that faculty play in the process of
a continuous system of improvement in
link student-level CLA results with
assessment.
teaching and learning stimulated by the
other data sources, pursue in-depth
CLA. Without your contributions, the
sampling, stay informed through the
The flagship program of CLA
CLA would not be on the exciting path
CLA Spotlight, and participate in CLA
Education is the Performance Task
that it is today. We look forward to your
Education offerings.
Academy, which shifts the focus from
continued involvement!
general assessment to the course-level
Student-level CLA results are provided
work of faculty. The Performance Task
for you to link with other data sources
Academy provides an opportunity for
(e.g., course-taking patterns, grades,
faculty members to learn to diagnose
portfolios, student satisfaction and
their individual students’ work and to
engagement, major-specific tests, etc.).
receive guidance in creating their own
performance tasks, which are designed
These internal analyses can help you
to supplement the educational reform
generate hypotheses for additional
movement toward a case and problem
research, which you can pursue through
approach in learning and teaching.
CLA in-depth sampling in experimental
areas (e.g., programs or colleges within
A CLA Education web site also has
your campus) in subsequent years or
been formed as a clearing house for
simultaneously.
performance tasks developed by
faculty. For more information, visit
We welcome and encourage your
www.claintheclassroom.org, or contact
participation in the CLA Spotlight—a
Director of CLA Education, Dr. Marc
series of free informational web
Chun at mchun@cae.org.
conferences. Each CLA Spotlight
features campuses doing promising work
using the CLA, guest-speakers from the
larger world of assessment, and/or CLA
staff members who provide updates or
insights to CLA-related programs and
projects.
14
Appendices
A: Task Overview
Introduction
The CLA is comprised of three types of prompts
within two types of task: the Performance Task
and the Analytic Writing Task. Your students
were asked to take all three prompts. The Analytic
Writing Task includes a pair of prompts called
Make-an-Argument and Critique-an-Argument.
The CLA uses direct measures of skills in which
students perform cognitively demanding tasks
from which quality of response is scored. All CLA
measures are administered online and contain
open-ended prompts that require constructed
responses. There are no multiple-choice questions.
The CLA tasks require students to integrate critical
thinking, analytic reasoning, problem solving,
and written communication skills. The holistic
integration of these skills on the CLA tasks mirrors
the requirements of serious thinking and writing
tasks faced in life outside of the classroom.
16
A: Task Overview
Performance Task
Each Performance Task requires
No two Performance Tasks assess
Performance Tasks often require
students to use an integrated set of
the exact same combination of skills.
students to marshal evidence from
critical thinking, analytic reasoning,
Some ask students to identify and then
different sources; distinguish rational
problem solving, and written
compare and contrast the strengths and
from emotional arguments and fact
communication skills to answer
limitations of alternative hypotheses,
from opinion; understand data in tables
several open-ended questions about a
points of view, courses of action, etc. To
and figures; deal with inadequate,
hypothetical but realistic situation. In
perform these and other tasks, students
ambiguous, and/or conflicting
addition to directions and questions,
may have to weigh different types of
information; spot deception and holes
each Performance Task also has its
evidence, evaluate the credibility of
in the arguments made by others;
own document library that includes a
various documents, spot possible bias,
recognize information that is and is not
range of information sources, such as
and identify questionable or critical
relevant to the task at hand; identify
letters, memos, summaries of research
assumptions.
additional information that would help
to resolve issues; and weigh, organize,
reports, newspaper articles, maps,
photographs, diagrams, tables, charts,
Performance Tasks also may ask
and synthesize information from several
and interview notes or transcripts.
students to suggest or select a course
sources.
Students are instructed to use these
of action to resolve conflicting or
materials in preparing their answers to
competing strategies and then provide
the Performance Task’s questions within
a rationale for that decision, including
the allotted 90 minutes.
why it is likely to be better than one or
more other approaches. For example,
The first portion of each Performance
students may be asked to anticipate
Task contains general instructions and
potential difficulties or hazards that are
introductory material. The student is
associated with different ways of dealing
then presented with a split screen. On
with a problem, including the likely
the right side of the screen is a list of the
short- and long-term consequences and
materials in the Document Library. The
implications of these strategies. Students
student selects a particular document
may then be asked to suggest and
to view by using a pull-down menu. On
defend one or more of these approaches.
the left side of the screen are a question
Alternatively, students may be asked
and a response box. There is no limit
to review a collection of materials or
on how much a student can type. Upon
a set of options, analyze and organize
completing a question, students then
them on multiple dimensions, and then
select the next question in the queue.
defend that organization.
17
A: Task Overview
Analytic Writing Task
Make-an-Argument
Critique-an-Argument
Students write answers to two types of
A “Make-an-Argument” prompt
A “Critique-an-Argument” prompt
essay prompts, namely: a “Make-an-
typically presents an opinion on some
asks students, in 30 minutes, to critique
Argument” question that asks them to
issue and asks students to write, in 45
an argument by discussing how well
support or reject a position on some
minutes, a persuasive, analytic essay to
reasoned they find it to be (rather than
issue; and a “Critique-an-Argument”
support a position on the issue. Key
simply agreeing or disagreeing with the
question that asks them to evaluate
elements include: establishing a thesis
position presented). Key elements of
the validity of an argument made
or a position on an issue; maintaining
the essay include: identifying a variety
by someone else. Both of these tasks
the thesis throughout the essay;
of logical flaws or fallacies in a specific
measure a student’s skill in articulating
supporting the thesis with relevant and
argument; explaining how or why the
complex ideas, examining claims and
persuasive examples (e.g., from personal
logical flaws affect the conclusions
evidence, supporting ideas with relevant
experience, history, art, literature, pop
in that argument; and presenting a
reasons and examples, sustaining a
culture, or current events); anticipating
critique in a written response that is a
coherent discussion, and using standard
and countering opposing arguments
grammatically correct, organized, well-
written English.
to the position, fully developing ideas,
developed, logically sound, and neutral
examples, and arguments; crafting an
in tone.
overall response that generates interest,
provokes thought, and persuades the
reader; organizing the structure of the
essay (e.g., paragraphing, ordering of
ideas and sentences within paragraphs);
employing transitions and varied
sentence structure to maintain the
flow of the argument; and utilizing
sophisticated grammar and vocabulary.
18
A: Task Overview
Example Performance Task
Example Document Library
You advise Pat Williams, the president
‚‚ Newspaper article about the accident
of DynaTech, a company that makes
‚‚ Federal Accident Report on in-flight
breakups in single-engine planes
precision electronic instruments and
navigational equipment. Sally Evans,
a member of DynaTech’s sales force,
recommended that DynaTech buy a
small private plane (a SwiftAir 235)
that she and other members of the
sales force could use to visit customers.
Pat was about to approve the purchase
when there was an accident involving
a SwiftAir 235. Your document library
Example Questions
‚‚ Do the available data tend to support
or refute the claim that the type of
wing on the SwiftAir 235 leads to
more in-flight breakups?
‚‚ Internal Correspondence (Pat’s e-mail
to you and Sally’s e-mail to Pat)
‚‚ What is the basis for your conclusion?
‚‚ Charts relating to SwiftAir’s
performance characteristics
‚‚ What other factors might have
contributed to the accident and
should be taken into account?
‚‚ Excerpt from magazine article
comparing SwiftAir 235 to similar
planes
‚‚ Pictures and descriptions of SwiftAir
Models 180 and 235
‚‚ What is your preliminary
recommendation about whether
or not DynaTech should buy the
plane and what is the basis for this
recommendation?
contains the following materials:
Example Make-an-Argument
Example Critique-an-Argument
There is no such thing as “truth” in the
A well-­respected professional journal
sample group were more likely to be
media. The one true thing about the
with a readership that includes
overweight––relative to the national
information media is that it exists only
elementary school principals recently
average. Based on this study, the
to entertain.
published the results of a t­ wo-­year
principal of Jones Elementary School
study on childhood obesity. (Obese
decided to confront her school’s obesity
individuals are usually considered to
problem by opposing any fast food
be those who are 20 percent above
restaurant openings near her school.
their recommended weight for
height and age.) This study sampled
50 schoolchildren, ages 5-11, from
Smith Elementary School. A fast food
restaurant opened near the school just
before the study began. After two years,
students who remained in the
19
B: Task Development
Iterative Development Process
A team of researchers and writers
While developing a Performance Task,
After several rounds of revision, the
generate ideas for Make-an-Argument
a list of the intended content from each
most promising of the Performance
and Critique-an-Argument prompts,
document is established and revised.
Tasks and the Make-an-Argument
and Performance Task storylines, and
This list is used to ensure that each piece
and Critique-an-Argument prompts
then contribute to the development
of information is clearly reflected in the
are selected for pre-piloting. Student
and revision of the prompts and
document and/or across documents,
responses from the pilot test are
Performance Task documents.
and to ensure that no additional pieces
examined to identify what pieces
of information are embedded in the
of information are unintentionally
For Analytic Writing Tasks, multiple
document that were not intended. This
ambiguous, what pieces of information
prompts are generated, revised and
list serves as a draft starting point for
in the documents should be removed,
pre-piloted, and those prompts that
the analytic scoring items used in the
etc. After revision and additional pre-
elicit good critical thinking and writing
Performance Task scoring rubrics.
piloting, the best functioning tasks (i.e.,
those that elicit the intended types and
responses during pre-piloting are further
revised and submitted to more extensive
During revision, information is either
ranges of student responses) are selected
piloting.
added to documents or removed from
for full piloting.
documents to ensure that students could
During the development of
arrive at approximately three or four
During piloting, students complete
Performance Tasks, care is taken to
different conclusions based on a variety
both an operational task and one of the
ensure that sufficient information is
of evidence to back up each conclusion.
new tasks. At this point, draft scoring
provided to permit multiple reasonable
Typically, some conclusions are designed
rubrics are revised and tested in grading
solutions to the issues present in
to be supported better than others.
the pilot responses, and final revisions
are made to the tasks to ensure that the
the Performance Task. Documents
are crafted such that information is
Questions for the Performance Task
task is eliciting the types of responses
presented in multiple formats (e.g.,
are also drafted and revised during the
intended.
tables, figures, news articles, editorials,
development of the documents. The
letters, etc.).
questions are designed such that the
initial questions prompt the student
to read and attend to multiple sources
of information in the documents, and
later questions require the student to
evaluate the documents and then use
their analysis to draw conclusions and
justify those conclusions.
20
C: Scoring Criteria
Introduction
Assessing Critical Thinking,
Analytic Reasoning and
Problem Solving
Assessing Writing
This section summarizes the
Applied in combination, critical
Analytic writing skills invariably
types of questions addressed
thinking, analytic reasoning
depend on clarity of thought.
by CLA scoring of all task
and problem solving skills are
Therefore, analytic writing
types. Because each CLA
required to perform well on
and critical thinking, analytic
task and their scoring rubrics
CLA tasks. We define these
reasoning, and problem
differ, not every item listed is
skills as how well students can
solving are related skills sets.
applicable to every task. The
evaluate and analyze source
The CLA measures critical
tasks cover different aspects
information, and subsequently
thinking performance by asking
of critical thinking, analytic
to draw conclusions and
students to explain in writing
reasoning, problem solving, and
present an argument based
their rationale for various
writing and in doing so can, in
upon that analysis. In scoring,
conclusions. In doing so, their
combination, better assess the
we specifically consider the
performance is dependent
entire domain of performance.
following items to be important
on both writing and critical
aspects of these skills.
thinking as integrated rather
See page 22 for detail.
than separate skills. We evaluate
writing performance using
holistic scores that consider
several aspects of writing
depending on the task. The
following are illustrations of the
types of questions we address in
scoring writing on the various
tasks.
See page 23 for detail.
21
C: Scoring Criteria
Assessing Critical Thinking,
Analytic Reasoning and
Problem Solving
Evaluation of evidence
Analysis and synthesis of evidence
How well does the student assess the quality and relevance
How well does the student analyze and synthesize data and
of evidence, including:
information, including:
‚‚ Determining what information is or is not pertinent to
the task at hand
‚‚ Presenting his/her own analysis of the data or
information (rather than “as is”)
‚‚ Distinguishing between rational claims and emotional
ones, fact from opinion
‚‚ Committing or failing to recognize logical flaws (e.g.,
distinguishing correlation from causation)
‚‚ Recognizing the ways in which the evidence might be
limited or compromised
‚‚ Breaking down the evidence into its component parts;
‚‚ Spotting deception and holes in the arguments of others
‚‚ Considering all sources of evidence
‚‚ Drawing connections between discrete sources of data
and information
‚‚ Attending to contradictory, inadequate or ambiguous
information
Drawing conclusions
Acknowledging alternative explanations/viewpoints
How well does the student form a conclusion from their
How well does the student acknowledge additional
analysis, including:
perspectives and consider other options, including:
‚‚ Constructing cogent arguments rooted in data/
information rather than speculation/opinion
‚‚ Recognizing that the problem is complex with no clear
answer
‚‚ Selecting the strongest set of supporting data
‚‚ Proposing other options and weighing them in the
decision
‚‚ Prioritizing components of the argument
‚‚ Avoiding overstated or understated conclusions
‚‚ Identifying holes in the evidence and subsequently
suggesting additional information that might resolve the
issue
‚‚ Considering all stakeholders or affected parties in
suggesting a course of action
‚‚ Qualifying responses and acknowledging the need
for additional information in making an absolute
determination
22
C: Scoring Criteria
Interest
How well does the student maintain the reader’s interest?
Does the...
‚‚ Student use creative and engaging examples or
descriptions
Assessing Writing
‚‚ Structure, syntax and organization add to the interest of
their writing
‚‚ Student use colorful but relevant metaphors, similes, etc.
‚‚ Writing engage the reader
‚‚ Writing leave the reader thinking
Presentation
Development
How clear and concise is the argument? Does the student…
How effective is the structure? Does the student…
‚‚ Clearly articulate the argument and the context for that
argument
‚‚ Correctly and precisely use evidence to defend the
argument
‚‚ Comprehensibly and coherently present evidence
‚‚ Logically and cohesively organize the argument
‚‚ Avoid extraneous elements in the argument’s
development
‚‚ Present evidence in an order that contributes to a
persuasive and coherent argument
Persuasiveness
Mechanics
How well does the student defend the argument? Does the
What is the quality of the student’s writing?
student…
‚‚ Effectively present evidence in support of the argument
‚‚ Draw thoroughly and extensively from the available
range of evidence
‚‚ Analyze the evidence in addition to simply presenting it
‚‚ Consider counterarguments and address weaknesses in
his/her own argument
‚‚ Is vocabulary and punctuation used correctly
‚‚ Is the student’s understanding of grammar strong
‚‚ Is the sentence structure basic, or more complex and
creative
‚‚ Does the student use proper transitions
‚‚ Are the paragraphs structured logically and effectively
23
D: Scoring Process
Score Sheet
There are two types of items that appear
Performance Task holistic items are
Make-an-Argument score sheets contain
on a CLA score sheet: analytic and
scored on four or seven-point scales
only holistic items scored on four or
holistic. Analytic scoring items are
(i.e., 1-4 or 1-7). There are multiple
seven-point scales (i.e., 1-4 or 1-7). The
particular to each prompt and holistic
holistic items per Performance Task that
holistic items include ratings for various
items refer to general dimensions, such
require graders to provide an evaluation
aspects of writing (e.g., organization,
as evaluation of evidence, drawing
of different aspects of critical thinking
mechanics, etc.) and critical thinking
conclusions, acknowledging alternative
and reasoning in the student responses.
(e.g., reasoning and logic, sophistication
explanations and viewpoints, and overall
These holistic items include areas
and depth of treatment of the issues
writing. We compute raw scores for each
such as the student’s use of the most
raised in the prompt) as well as two
task by adding up all points on all items
relevant information in the Performance
overall assessments of writing and
(i.e., calculating a unit-weighted sum).
Task, their recognition of strengths
critical thinking.
and weaknesses of various pieces of
Performance Task scoring is tailored
information, overall critical thinking,
For all task types, blank responses or
to each specific prompt and includes
and overall writing.
responses that are entirely unrelated to
a combination of both holistic and
the task (e.g., writing about what they
analytic scoring items. Though there
Critique-an-Argument score sheets also
had for breakfast) are assigned a 0 and
are many types of analytic items on the
include a combination of analytic and
are flagged for removal from the school-
Performance Task score sheets, the most
holistic scores. Critique-an-Argument
level results.
common represent a list of the possible
analytic items are a list of possible
pieces of information a student could
critiques of the argument presented in
or should raise in their response. These
the prompt. In addition, a few holistic
cover the information presented in the
items are used to rate the overall quality,
Performance Task documents as well
critical thinking and writing over the
as information that can be deduced
entire response.
from comparing information across
documents. The analytic items are
generally given a score of 0 if the student
did not use the information in their
response, or 1 if they did. The number
of analytic items varies by prompt.
24
D: Scoring Process
Scoring Procedure
All scorer candidates undergo rigorous training in
order to become certified CLA scorers. Training
includes an orientation to the prompt and score sheet,
instruction on how to evaluate the scoring items,
repeated practice grading a wide range of student
responses, and extensive feedback and discussion after
scoring each response.
After participating in training, scorers complete a
reliability check where they score the same set of
student responses. Scorers with low agreement or
reliability (determined by comparisons of raw score
means, standard deviations and correlations among the
scorers) are either further coached or removed from
scoring.
In fall 2008 and spring 2009, a combination of
machine and human scoring was used for the Analytic
Writing Task.
The CLA utilizes Pearson Knowledge Technology’s
Intelligent Essay Assessor program for evaluating
responses to the Make-an-Argument and Critique-anArgument prompts.
The machine scoring engine was developed and tested
using scores from a broad range of responses that were
previously scored by humans (often double scored).
In some cases the automated scoring engine is unable
to score off-topic or abnormally short/long responses.
These student responses are scored by humans.
25
E: Scaling Procedures
To facilitate reporting results across
schools, ACT scores were converted
Standard ACT to SAT
Conversion Table
(using the ACT-SAT crosswalk to the
right) to the scale of measurement used
ACT to SAT
to report SAT scores.
36
1600
35
1580
34
1520
33
1470
32
1420
31
1380
open admission schools), we make
30
1340
available the Scholastic Level Exam
29
1300
28
1260
27
1220
26
1180
25
1140
and Recentered SAT I Sum Scores” by
were converted to SAT scores using data
24
1110
N.J. Dorans, C.F. Lyu, M. Pommerich,
from 1,148 students participating in
23
1070
and W.M. Houston (1997), College and
spring 2006 that had both SAT and SLE
22
1030
21
990
University, 73, 24-31; “Concordance
20
950
19
910
18
870
N.J. Dorans, Research Notes (RN-07),
17
830
College Entrance Examination Board:
16
780
1999; “Correspondences between ACT
15
740
14
680
and SAT I Scores” by N.J. Dorans, College
13
620
12
560
Entrance Examination Board: 1999; ETS
11
500
Research Report 99-2, Educational Testing
For institutions where a majority of
students did not have ACT or SAT
scores (e.g., two-year institutions and
(SLE), a short-form cognitive ability
measure, as part of the CLA. The SLE is
produced by Wonderlic, Inc. SLE scores
scores. These converted scores (both
ACT to SAT and SLE to SAT) are
referred to simply as entering academic
ability (EAA) scores.
Students receive a single score on a
CLA task because each task assesses
an integrated set of critical thinking,
analytic reasoning, problem solving, and
written communication skills.
Sources:
“Concordance Between ACT Assessment
between SAT I and ACT Scores for
Individual Students” by D. Schneider and
Board Research Report 99-1, College
Service: 1999.
26
E: Scaling Procedures
Each Performance Task and Analytic
A linear scale transformation is used
the same as the lowest EAA score of any
Writing Task has a unique scoring
to convert reader-assigned raw scores
freshman who took that task. On very
rubric, and the maximum number of
to scale scores. This process results
rare occasions, a student may achieve an
reader assigned raw score points differs
in a scale score distribution with the
exceptionally high or low raw score (i.e.,
across tasks. Consequently, a given
same mean and standard deviation as
well above or below the other students
reader-assigned raw score, such as 15
the Entering Academic Ability (EAA)
taking that task). When this occurs,
points, may be a relatively high score on
scores of the freshmen who took that
it results in assigning a student a scale
one task but a low score on another task.
measure. This type of scaling preserves
score that is outside of the normal EAA
the shape of the raw score distribution
range. Prior to the spring of 2007, scores
To adjust for such differences, reader-
and maintains the relative standing of
were capped at 1600. Capping was
assigned raw scores on the different
students. For example, the student with
discontinued starting in fall 2007.
tasks are converted to a common scale
the highest raw score on a task will also
of measurement. This process results
have the highest scale score on that task,
In the past, CAE revised its scaling
in scale scores that reflect comparable
the student with the next highest raw
equations each fall. However, many
levels of proficiency across tasks. For
score will be assigned the next highest
institutions would like to make year-
example, a given CLA scale score
scale score, and so on.
to-year comparisons (i.e., as opposed
indicates approximately the same
to just fall to spring). To facilitate this
percentile rank regardless of the task
This type of scaling generally results in
activity, in fall 2007 CAE began using
on which it was earned. This feature of
the highest raw score earned on a task
the same scaling equations it developed
the CLA scale scores allows combining
receiving a scale score of approximately
for the fall 2006 administration and
scores from different tasks to compute
the same value as the maximum EAA
has done so for new tasks introduced
a school’s mean scale score for each task
score of any freshman who took that
since then. As a result of this policy, a
type as well as a total average scale score
task. Similarly, the lowest raw score
given raw score on a task will receive the
across types.
earned on a task would be assigned a
same scale score regardless of when the
scale score value that is approximately
student took the task.
27
F: Student Data File
We provide a CLA Student Data
We provide student-level information
Student-level scores are not designed to
File, which includes variables across
to link with other data you collect (e.g.,
be diagnostic at the individual level and
three categories: student self-reported
from NSSE, CIRP, portfolios, local
should be considered as only one piece of
information from their CLA on-line
assessments, course-taking patterns,
evidence about a student’s skills.
profile; CLA scores and identifiers; and
participation in specialized programs,
information provided/verified by the
etc.) to help you hypothesize about
registrar.
campus-specific factors related to overall
institutional performance.
Self-Reported Data
‚‚ Student ID, E-mail address,
and Name (first, middle
initial, last)
‚‚ Age
‚‚ Gender
‚‚ Race/Ethnicity
‚‚ Primary and Secondary
Academic Major (34
categories)
CLA Scores and Identifiers
‚‚ CLA scores for Performance Task,
Analytic Writing Task, Make-anArgument, Critique-an-Argument,
and Total CLA Score (depending
on the number of tasks taken and
completeness of responses):
‚‚ Transfer Student Status
‚‚ Program ID and Name (for
classification of students into
different schools, majors,
studies, programs, etc.)
CLA scale scores;
‚‚ Entering Academic Ability
(EAA) Score
--
Student Performance Level categories (i.e., well below expected,
below expected, at expected,
above expected, well above
expected) if CLA scale score and
entering academic ability (EAA)
scores are available;
‚‚ SAT I - Math
Percentile Rank in the CLA
(among students in the same class
year; based on scale score); and
‚‚ SAT I - Writing (Essay subscore)
‚‚ English as primary language
--
‚‚ Attended school as Freshman,
Sophomore, Junior, Senior
‚‚ CLA Local Survey Responses
‚‚ Class Standing
--
‚‚ Field of Study (6 categories;
based on primary academic
major)
‚‚ Total years at school
Registrar Data
--
Percentile Rank at School
(among students in the same class
year; based on scale score).
‚‚ SAT I - Verbal / Critical
Reading
‚‚ SAT Total (Math + Verbal)
‚‚ SAT I - Writing
‚‚ SAT I - Writing (Multiplechoice subscore)
‚‚ ACT - Composite
‚‚ Unique CLA numeric identifiers
‚‚ ACT - English
‚‚ Year, Administration (Fall or Spring),
Type of Test (90 or 180-minute), Date
of test
‚‚ ACT - Reading
‚‚ ACT - Mathematics
‚‚ ACT - Science
‚‚ ACT - Writing
‚‚ GPA
28
G: CAE Board of Trustees and Officers
Roger Benjamin
President & CEO
James Hundley
Executive Vice President & COO
Benno Schmidt
Chairman, CAE
Richard Atkinson
President Emeritus, University of California System
Doug Bennett
President, Earlham College
Michael Crow
President, Arizona State University
Russell C. Deyo
Vice President & General Counsel, Johnson & Johnson
Richard Foster
Managing Partner, Millbrook Management Group, LLC
Ronald Gidwitz
Chairman, GCG Partners
Lewis B. Kaden
Vice Chairman, Citigroup Inc.
Michael Lomax
President, United Negro College Fund
Katharine Lyall
President Emeritus, University of Wisconsin System
Eduardo Marti
President, Queensborough Community College, CUNY
Ronald Mason
President, Jackson State University
Diana Natalicio
President, University of Texas at El Paso
Charles Reed
Chancellor, California State University
Michael D. Rich
Executive Vice President, RAND Corporation
Farris W. Womack
Executive Vice President and Chief Financial Officer, Emeritus
Professor, Emeritus
The University of Michigan
29
pb
3030