Report – General Education Assessment Review (GEAR) Sub-Group
Evaluation of Collegiate Learning Assessment (CLA)/Community College Learning
Assessment (CCLA) for Use in Strengthened Campus-Based Assessment (SCBA)
May 2009
I. Introduction/Background Information
SCBA and GEAR
After the SUNY Board of Trustees passed its Resolution on Strengthened Campus-Based
Assessment (SCBA) in June 2004, SUNY System Administration assigned significant
responsibility to the SUNY-wide General Education Assessment Review Group (GEAR) in
implementing this initiative. One major task was to review and identify nationally-normed
measures that campuses could use, if they so chose, to assess the student learning outcomes
included under the SUNY GER Knowledge and Skills Areas of Mathematics and Basic
Communication [Written] and the Critical Thinking [Reasoning] Competency.
During the 2004-05 academic year, GEAR reviewed many existing nationally-normed tests in
these three outcome areas, including the ACT CAAP, the ETS MAPP, the Graduate Record
Examination, AccuPlacer, and COMPASS. Ultimately, GEAR found that only the ACT CAAP
test for Critical Thinking and two ACT CAAP Writing modules (one for the first Written
Communication outcome and one for the second outcome) met its review criteria.1 Since that
time, no additional nationally-normed tests have been identified by GEAR as appropriate for use
for either the Critical Thinking [Reasoning] or Basic Communication [Written] outcomes
In recent years, several SUNY campuses have expressed interest in using the Collegiate Learning
Assessment (CLA) or the Community College Learning Assessment (CCLA) for SCBA. The
CLA/CCLA, which has only become widely available and used by colleges and universities
since 2004, is intended to assess writing, critical thinking, and problem solving skills.2 During
the Spring 2009 semester, a GEAR sub-group consisting of Robert Axelrod, Patty Francis, Tina
Good, Milton Johnson, Rose Rudnitski, and Melanie Vainder reviewed the CLA/CCLA
regarding its appropriateness for use as part of SCBA. This report summarizes the sub-group’s
major conclusions from its review and offers recommendations with respect to this issue.
CLA/CCLA
The CLA/CCLA, developed and administered by the Council for Aid to Education (CAE),
differs substantively from other existing nationally-normed tests in a number of ways. Perhaps
the most important difference is that the CLA/CCLA is not a multiple-choice test. Instead, it
asks students to evaluate a complex set of materials or “prompts” and to construct written
responses. The CLA/CCLA must be administered under controlled conditions in a computer lab,
and students have two hours to take the test; typical test-taking time (depending on the particular
task a student receives) ranges from 75-90 minutes. Answers are scored by trained evaluators,
1
These tests can be found at http://www.act.org/caap/suny/.
A full description of the CLA can be found at http://www.cae.org/content/pro_collegiate.htm; the CCLA is
described at http://www.cae.org/content/pro_communcollege.htm.
2
1
although recently CAE has piloted machine scoring for some tasks. An explanation of
CLA/CCLA scoring can be found at
http://www.cae.org/content/pdf/CLA_Scoring_Criteria_(Jan%202008).pdf.
Another difference is that CAE’s analysis of results corrects statistically for students’ incoming
SAT/ACT scores, providing a “purer” estimate of student performance (i.e., student CLA/CCLA
scores cannot be attributed simply to the fact that they entered the institution with more or less
pre-collegiate ability as assessed by the SAT/ACT).3 Similarly, for last-semester students
CLA/CCLA scores are corrected statistically for student’s college GPA.
Yet another difference is that the CLA/CCLA is “inherently” value-added, meaning that
campuses receive three sets of scores: 1) those for first-semester students; 2) those for finalsemester students; and 3) those indicating differences, or value-added, between the two student
cohorts.4 Campuses may choose between a cross-sectional or a longitudinal approach. The
cross-sectional approach tests first-semester students in the fall semester and final-semester
students in the spring semester of the same academic year. The longitudinal approach tests the
same students at the two different testing points (i.e., after four years in four-year schools and
after two years in community colleges). For obvious reasons, most institutions have elected to
utilize the cross-sectional approach, so the current discussion will focus on that particular design.
With respect to sampling, the participating institution tests 100 first-year and 100 final-year
students. Prior to testing, the institution must submit to CAE a sampling plan detailing how
students will be selected for participation in order to assure samples are representative, and
institutions cannot proceed with testing until their sampling plan is approved. Other safeguards
in this respect include the fact that campuses, subsequent to testing, must report the average
SAT/ACT scores and GPA (for last-semester students) for the population from which the sample
was obtained. Most important, the fact that the analyses of results correct for student SAT/ACT
scores and GPA makes it unlikely that campuses would intentionally select students with higher
SAT/scores or GPA’s.
II. Evaluation Process
In reviewing the CLA/CCLA, the GEAR sub-group used the same criteria that were employed
when the GEAR Group conducted its original reviews of existing tests in 2004-05. These
criteria are explained below:
a. Does the test map to the SUNY SCBA outcomes and, if so, which ones?
b. Does the report on student test performance received by the institution provide sub-scores
for specific SCBA outcomes, making it possible for institutions to determine strengths
and weaknesses in that performance?
c. Is the test psychometrically sound (i.e., in terms of reliability, validity, and
standardization)?
3
Students who do not have ACT/SAT scores are asked to complete a 15-minute Scholastic Level Evaluation (SLE),
which correlates positively with ACT/SAT measures.
4
The CLA compares freshmen and seniors whereas the CCLA compares freshmen and second-year students.
2
d. Can the test be administered within the context of a class session?5
During its initial conference call, sub-group members agreed to do preliminary research and
share articles/sources with each other electronically, establishing a deadline of about a month, to
be followed by three weeks in which members would review all materials that had been received.
At that time, members participated in another conference call in order to discuss their
impressions and reach final conclusions.
III. Sub-Group Conclusions
The following presentation of sub-group conclusions is organized by the four evaluative criteria
described above, and also includes a discussion of other issues that were raised during group
discussion.
1. Mapping of CLA/CCLA Items to Outcomes
The GEAR sub-group concluded that the CLA/CCLA does map satisfactorily to the two
Critical Thinking outcomes (i.e., “Identify, analyze, and evaluate arguments as they occur in
their own or other’s work” and “Develop well-reasoned arguments”) and to the first Basic
Communication [Written] outcome (i.e., “Produce coherent texts within common college-level
written forms”).
2. Reporting of Sub-scores for Individual SCBA Outcomes
A review of the institutional report received from CAE confirms that separate sub-scores are provided
for the two Critical Thinking outcomes and one Writing outcome described immediately above. To
be specific, this report provides summary information for students’ performance on the “Critique-anArgument” task (i.e., Critical Thinking Outcome #1), the “Make-an-Argument” task (i.e., Critical
Thinking Outcome #2), and the Analytic Writing task (i.e., Writing Outcome #1).
3. Psychometric Properties
Because the CCLA was offered for the first time in 2008-09, the psychometric data
summarized in this section were gathered on the CLA. These data suggest that the CLA is
characterized by reasonable reliability and validity. For inter-rater reliability, reported
estimates range from .76 to .87 for performance tasks (mean = .81), and mean estimates of
.65 and .80 have been reported for the Make-an-Argument and Critique-an-Argument tasks,
respectively. Internal consistency is very good (i.e., ranging from .82 to .91 for Writing and
from .84 to .92 for Performance Tasks).
Concurrent validity has been demonstrated between the CLA and SAT/ACT scores, with
correlations ranging between .73 - .88 for Writing Tasks and between .78 - .92 for
Performance Tasks. In addition, institutions typically report correlations between students’
CLA performance and their GPAs. At present, as part of a large FIPSE grant, construct and
5
Unlike the other three criteria, this last criterion is more “preferred” than required, as GEAR does allow campuses
to administer measures in stand-alone fashion, as long as the campuses clarify how they will address possible
problems related to student motivation and representativeness.
3
concurrent validity of the CLA, ACT CAAP, and EST MAPP are being investigated. When
complete, this study should provide much more useful information about the validity of all
three of these instruments.
With respect to standardization, the CLA has primarily been administered to college-age
students from a wide variety of institution types, yielding significant normative data on
student performance. At present, more than 350 institutions overall have participated in the
CLA. A listing of institutions participating in the CLA during the 2008-09 academic year
can be found at http://www.cae.org/content/pdf/CLA_School_List_0809.pdf.
A more thorough description of the CLA’s psychometric properties can be found at
http://www.cae.org/content/pdf/CLA_Technical_FAQs.pdf.
4. Test Administration
Because the CLA/CCLA must be administered in controlled conditions in a computer
laboratory, and because up to two hours must be allowed for testing, GEAR sub-group
members noted that it would be difficult for campuses to administer the test totally within the
context of a class session. As a result, campuses proposing to use the CLA/CCLA as part of
SCBA, and receive funding for its use, would need to provide detailed information to GEAR
as to how they would motivate students to participate in the testing and perform to the best of
their ability.
5. Other Issues
In addition to concerns that have already been described, GEAR sub-group members
expressed reservations regarding four other issues related to the administration of the
CLA/CCLA. First, the prescribed sample size of 100 for each student cohort struck some
members as possibly problematic, and perhaps inappropriate for institutions with very high
or very low enrollments.6 Related to this concern, despite the fact that CAE has procedures
in place to help assure that testing samples are representative of the student body overall, it is
important for GEAR itself to be satisfied with a campus’ sampling strategy. A third concern
revolved around the CLA/CCLA’s assessment of “value-added,” based on the observation
that such information might easily be over- or under-estimated if rigorous sampling
procedures are not followed. Finally, the GEAR sub-group noted that, compared to other
nationally-normed tests, the CLA/CCLA is complex and somewhat difficult to understand
and explain, and has administration requirements not all campuses will be able to meet.
6
Institutions are free to over-sample, but must pay for each additional student tested.
4
IV. Sub-Group Recommendations
Based on its review of materials and subsequent discussion, the GEAR sub-group offers the
following recommendations:
1. Campuses should be given the option to use the CLA/CCLA on a pilot basis (i.e., for one
administration) to measure the following SCBA outcomes: a) Critical Thinking Outcome
#1; b) Critical Thinking Outcome #2; and/or c) Basic Communication [Written] Outcome
#1.
2. Campuses that choose this option should submit to GEAR a revised general education
assessment plan describing these changes to their existing, approved plan.
3. These revised plans must address all nine evaluative criteria, as appropriate, included in
GEAR’s Review Process Guidelines.7 In particular, the campus must describe in detail
how it will assure:
a. That the student samples to be used in both the first-year and senior groups are
representative of the student groups from which they are selected. Strategies for
achieving this objective include conducting analyses to demonstrate that the
samples do not differ significantly from the overall first-year or senior student
groups with respect to factors such as ACT/SAT scores, GPA’s, or course-taking
patterns.
b. That students participating in the testing are adequately motivated to perform well
on the CLA/CCLA.
c. That potential problems associated with value-added designs are adequately
controlled. Campuses administering the CLA/CCLA cross-sectionally would
need to address concerns related to sample comparability while those
administering the CLA/CCLA longitudinally would have to address concerns
related to student attrition.
4. GEAR should provide campuses with a thorough description of the CLA/CCLA during
the summer of 2009, to include specific suggestions regarding sample size, “valueadded,” administration procedures, and student motivation.
5. System Administration, working with GEAR, should ask CAE to sponsor a special
Webinar during the summer of 2009 for SUNY campuses interested in using the
CLA/CCLA.
6. Subsequent to the assessment, campuses participating in the pilot should be required to
provide two separate reports:
7
These guidelines can be found at http://www.cortland.edu/gear/GEAR%20Guidelines%2002-01-06.pdf.
5
a. A report on results to System Administration using the standard Summary Report
form required by System Administration and available at
http://www.suny.edu/provost/Assessmentinit.cfm?navLevel=5. (Note that the
form is scheduled to be updated this spring.)
b. A report to GEAR, using a special form developed by GEAR for this purpose.
This form will request feedback from the campus regarding the adequacy of the
CLA/CCLA in mapping to the outcomes of interest and their procedures in
assuring student representativeness and motivation and sample comparability as
well as controlling against attrition. GEAR will use this information to determine
whether the CLA/CCLA is appropriate for longer-term use.
6