Assessing Student
Achievement in the
Core Curriculum
Office of Assessment and Program Improvement
Report to the University Community
Fall 2003
Office of the Provost
The Assessment
Process
 Define specific learning outcomes that should generally be
evident in students who complete a program of study
 Identify sources of evidence about how well students are
typically achieving these outcomes
 Set performance targets as criteria for program success
 Conduct studies and use the results to improve achievement
2
Office of the Provost
Assessing a Major vs.
Assessing the Core
MAJOR
CORE
What an AU student
What any AU student
majoring in a given subject
should know
should know
and be able to do
and be able to do
no matter what major
Defined by
specialists
Defined
Cooperatively
3
Office of the Provost
Working Assumptions
 We can collect reasonably good evidence about how well
students are achieving the desired “core outcomes” for an AU
graduate, but it is difficult to know what led to this achievement
 Assessing students’ core competencies is more feasible than
determining the exact contribution of the Core Curriculum itself
 Using several methods and measurement points will provide
fuller information than relying on any one “snapshot” method
 The “core outcomes” are everybody’s business
4
Office of the Provost
Core Curriculum
Oversight Committee
 Created together with the Core Curriculum
• Original charge was to oversee implementation of Core
• Began assessment in 1995 with reviews of social science
and Great Books courses
• Reviews focused on teaching – syllabi, assignments,
textbooks, grade distributions
• Assessment best practice now focuses on student learning
5
Office of the Provost
Assessment Process
 Define intended outcomes
6
Office of the Provost
From Broad Purpose
to Specific Plans
In 2001 the Core Curriculum Oversight Committee
 Studied the broad purpose statement for the Core Curriculum
 Formulated 12 more specific intended learning outcomes
 Selected 4 outcomes for assessment in 2002 and 2003
7
Office of the Provost
Starting Points
for Assessment
Students who have completed the Core
Curriculum will be able to
 Read critically and
analytically
 Comprehend the basic
concepts of algebra
 Communicate in writing at a
sophisticated level
 Gather, interpret, and
synthesize information in
accordance with
contemporary scholarly
standards
8
Office of the Provost
Future Assessment
Outcomes for Future Study
 Scientific Reasoning*
 Oral Communication
 Art Appreciation
 Awareness of Issues
 History, Culture, Values
 Critical Thinking
 Social Sciences
 Problem Solving
9
Office of the Provost
Assessment Process
 Define intended outcomes
 Identify means of assessment
10
Office of the Provost
Two Methods Used
 Local methods to estimate the abilities of students who were
still taking Core courses


Analysis of papers and exams
Pre- and post-testing
 ACT’s Collegiate Assessment of Academic Proficiency (CAAP)
tests to estimate the abilities of students who had fulfilled all
Core Curriculum requirements

Writing, Reading, Math, Critical Thinking, Science Reasoning
11
Office of the Provost
Assessment Methods
LOCAL
STANDARD
Review of work samples
ACT CAAP Tests
Pre- and Post-Testing
Project SAILS
Taking Core
Fulfilled Core
12
Office of the Provost
Assessment Process
 Define intended outcomes
 Identify means of assessment
 Set targets for program success
13
Office of the Provost
Standards Chosen:
Local Methods
 Critical Reading
• 70% of portfolios reviewed should rate satisfactory or better
when evaluated by two independent judges
 Written Communication
• 70% of portfolios reviewed should rate satisfactory or better
when evaluated by two independent judges
 College Algebra
• Improvement should be evident on post-test
 Information Literacy
• Both standards – 70% satisfactory and post-test improvement
14
Office of the Provost
Standards Chosen:
CAAP Test
 AU students who have completed the Core Curriculum should
score at or above the 70th percentile nationally on each CAAP
module
 No more than 20% of AU students tested should perform below
the 50th percentile nationally on any CAAP module
 National comparison group was sophomores at about 80 public
4-year colleges
15
Office of the Provost
Assessment Process
 Define intended outcomes
 Identify means of assessment
 Set targets for program success
 Collect and analyze assessment data
16
Outcome 1
Critical Reading
Students who have completed the
Core Curriculum will be able to read
critically and analytically at a
sophisticated level
Office of the Provost
Critical Reading
Analysis of papers and exams from Great Books II showed
 Students were able to find, remember, and interpret specific
details from their readings
 They were able to discuss themes found in their readings
 Assessment could not determine whether students were able to
discuss formal literary features of the works they had read or
place them in their historical and social contexts
18
Critical Reading
Office of the Provost
Percent of Great Books II Portfolios Rated
Excellent/Satisfactory
100
90
86
88
75
61
%
49
38
33
0
Details
Themes
Papers
Forms
Exams
Can't Tell
Contexts
19
Office of the Provost
Critical Reading
In formal papers for Great Books II, students met but did not
exceed the performance target for three aspects writing about
texts that may imply critical reading ability
 Formulating a thesis about a text they had read
 Using textual evidence to support that thesis
 Reasoning soundly to connect evidence and thesis
20
Critical Reading
Office of the Provost
GB2 Essays rated Excellent or Satisfactory
100
73
73
Thesis
Evidence
Reasoning
%
71
0
Percent Rated Excellent or Satisfactory
21
Office of the Provost
Critical Reading
Results from the two CAAP tests for Critical Reading were mixed
 Reading
• Median score was at the 73rd percentile nationally
• But 22% of those tested scored below the 50th percentile
 Critical Thinking
• Median score was at the 82nd percentile nationally
• Only 11% scored below the 50th percentile
22
Critical Reading
Office of the Provost
Median Percentile Scores and “Failure Rates”
CAAP Reading and Critical Thinking Tests
100
82
%
73
22
11
0
Reading
Median Score as National Percentile Rank
Critical Thinking
% Below 50th Percentile
23
Outcome 2
Written Communication
Students who have completed the
Core Curriculum will be able to
communicate in writing at a
sophisticated level
Office of the Provost
Written
Communication
Analysis of argument papers from Composition II showed
 Students were clearly able
• to formulate a thesis
• to use language that was appropriate to the writing situation
 Students had more trouble
• Supporting their thesis with sound evidence
• Organizing their arguments effectively
25
Written Communication
Office of the Provost
Four Aspects of Rhetorical Effectiveness
ENGL1120 Papers with Satisfactory Ratings
100
90
89
73
%
66
0
Thesis
Evidence
Organization
Rated Satisfactory or Better
Language
26
Office of the Provost
Written
Communication
The two CAAP tests for writing were given in 2002 only
 Students “maxed out” the Essay Writing test – nearly all scored
at or above the 70th national percentile
 Scores on the multiple-choice Writing Skills test were also
strongly positive
 It is not clear that there is good alignment between this test and
the goals of Composition I and II
27
Written Communication
Office of the Provost
Median Percentile Scores and “Failure Rates”
CAAP Writing Skills and Essay Writing Tests
100
97
%
86
7
16
0
Writing Skills (2002 only)
Median Score as National Percentile Rank
Essay Writing (2002 only)
% Below 50th Percentile
28
Outcome 3
College Algebra
Students who have completed the
Core Curriculum will comprehend
the basic concepts of algebra
Office of the Provost
College Algebra
Local Assessment (2003 Only)
 Students took Math Placement Exam at Camp War Eagle
 Some items from this exam were embedded in the final exams
for pre-calculus Core courses (not including MATH 1100)
 On average, the 916 students tested scored 30% higher on
these items on the final exam than they had done on the
placement test
30
Office of the Provost
College Algebra
Results for the CAAP Mathematics test must be treated separately
for 2002 and 2003 because of a change in the test design
 2002 version (4 calculus items)
students met the performance targets for the algebra subscore
 2003 version (no calculus items)
students met the performance targets for the whole test
 For both years, students’ performance just cleared the bar of
the 70th national percentile, but concerns remain
31
College Algebra
Office of the Provost
Median Percentile Scores for CAAP
Mathematics Test
100
73
%
76
18
5
0
Algebra Subscore (2002)
Median Score as National Percentile Rank
Mathematics Test (2003)
% Below 50th Percentile
32
Office of the Provost
College Algebra
The CAAP results are less positive when students are grouped
 Those who had taken Math courses beyond the Core met
performance targets
. . . But . . .
 Those who had taken only one Math course failed to meet
performance targets
33
College Algebra
Office of the Provost
Median Percentile Scores and “Failure Rates”
“More Math” vs. “Core Math” Students
100
82
%
64
24
11
0
"More Math" (2003)
Median Score as National Percentile Rank
"Core Math" (2003)
% Below 50th Percentile
34
Experimental Outcome
Science Reasoning
Office of the Provost
Science Reasoning
The CAAP results for Science Reasoning (not formally assessed
in either year) resemble those for Mathematics
 Those who had taken only one Math course scored poorly in
Science Reasoning
 Those who had taken Math courses in addition to the Core did
much better in Science Reasoning
36
Science Reasoning
Office of the Provost
Median Percentile Scores and “Failure Rates”
CAAP Science Reasoning Test
100
87
72.5
%
70
20
14
4
0
All Students
"More Math" Students
Median Score as National Percentile Rank
"Core Math" Students
% Below 50th Percentile
37
Outcome 4
Information Literacy
Students who have completed the
Core Curriculum will be able to
gather, synthesize, and interpret
information in accordance with
contemporary scholarly standards
Office of the Provost
Information Literacy
 Before receiving bibliographic
instruction (BI), ENGL 1120
students were tested on how to
conduct library searches
AU Bibliographic Instruction Quiz
(Summer 2001)
100
61
 Sample: “You wish to find
current articles dealing with
drug use on U.S. campuses.
Your best “place” to begin?
 Students were retested after BI
and had higher scores
48
0
Before BI
After BI
% Correct
39
Office of the Provost
Information Literacy
 165 UNIV1000 students used
the online Texas Information
Literacy Tutorial (TILT) and took
the accompanying quizzes
Texas Information Literacy Test
100
96
86
 The overall score on the TILT
was 96%
 Their lowest score was on an
item that tested whether they
thought the library’s whole
collection was available online
0
Total
Availability Question
% Correct
40
Office of the Provost

Information Literacy
Project SAILS
In 2003, 247 ENGL1120 students
took part in Project SAILS, a
national information literacy
test recognized by the
Association of Research
Libraries
Project SAILS
100
66
57

Multiple choice questions
yielded information about 4 key
information literacy skills

On each skill, AU students did
somewhat better than the
average for all 10 participating
institutions
50
50
Assess
Needs
Find
Evaluate Understand
Information Information
Issues
0
% Correct
41
Information Literacy
Work Samples
Office of the Provost
Aspects of Using Sources Effectively
ENGL1120 Papers with Satisfactory Ratings
100
91
63
68
66
%
47
0
Appropriateness
Balance*
Integration
Citation*
Rated Satisfactory or Better (* = 2003 only)
Reference List*
42
Office of the Provost
Assessment Process
 Define intended outcomes
 Identify means of assessment
 Set targets for program success
 Collect and analyze assessment data
 Act on the results to improve student achievement
43
Office of the Provost
Planned Actions
 Remedy focused deficiencies by additional instruction and
practice in appropriate courses – e.g., documenting sources
 Guide instruction, assessment, and improvement by clarifying
some broad intended outcomes – e.g., critical reading ability
 Continue to get good evidence about our students’ core
competencies through dialogue and collaboration
 Benefit from experience with four outcomes to guide
assessment planning for the intended learning outcomes that
have not been studied so far
44
Office of the Provost
Further Information
Office of Assessment and Program Improvement
Drew Clark, Director
209 Samford Hall
844-5802
clarkj3@auburn.edu
http://www.auburn.edu/academic/provost/assessment