Critical Thinking Assessment Test (CAT) Scoring Session
SPC EPI Tech Campus, Room 1-324
April 23, 2010
Attendees: Janice Thiel, Jesse Coraggio, Maggie Tymms, Gail Lancaster, Larry
Goldsmith, Linda Kelsey, Darlene Westberg, Steven Hardt, Gary Brown, Arlene
Gillis, Ginny Price, David Allen
“The CAT instrument is a unique tool designed to assess and promote the
improvement of critical thinking and real-world problem solving skills. The
instrument is the product of extensive development, testing, and refinement with
a broad range of institutions, faculty, and students across the country. The
National Science Foundation has provided support for many of these activities.
The CAT Instrument is designed to assess a broad range of skills that faculty
across the country feel are important components of critical thinking and real
world problem solving. The test was designed to be interesting and engaging for
students. All of the questions are derived from real world situations. Most of the
questions require short answer essay responses and a detailed scoring guide
helps insure good scoring reliability.” (Tennessee Tech University, Critical
Thinking Assessment Test Overview).
In collaboration with Tennessee Technological University and with support from
the National Science Foundation, St. Petersburg College (SPC) received a grant
to administer the Critical Thinking Assessment Test (CAT) instrument to a
representative sample of approximately 100 students enrolled in the College
during 2008. SPC conducted a second administration in 2009, which resulted in
administering the CAT instrument to sixty-six students enrolled in randomly
selected classes. Beginning in 2010, SPC standardized the process by
identifying one general education discipline in which to conduct future CAT
administrations. In Spring 2010, three College Algebra sections, and three
Elementary Statistics sections were randomly selected, and the CAT instrument
was administered to a representative sample of students enrolled in the six
courses listed in Table 1.
Critical Thinking Assessment Test (CAT) Scoring Session
1
Table 1
Distribution of Students by Course
Course
Discipline
MAC 1105 (2462)
College Algebra
Scored CAT
Assessments
12
MAC 1105 (571)
College Algebra
22
MAC 1105 (591)
College Algebra
11
STA 2023 (1390)*
Elementary Statistics
14
STA 2023 (1051)
Elementary Statistics
15
STA 2023 (1052)
Elementary Statistics
6
*STA 2023 section 1390 was incorrectly recorded on the CAT assessments as section 1280.
SPC Faculty members were invited to participate in the CAT scoring session
during various meetings held between January and April.
The CAT Scoring Session was held on April 23, 2010 at the EPI Tech Campus of
St. Petersburg College. Although one-hundred and twenty CAT assessments
were administered during Spring 2010, only eighty were scored due to the
number of scorers that were available. The remainder may be scored during a
future workshop as time allows. Copies of the CAT Scoring Session agenda and
non-disclosure consent form are located in Appendix A and B, respectively.
The majority of the scoring faculty (9) and facilitators (2) arrived by 8:15a.m., and
participated in a continental breakfast. An additional faculty member arrived at
noon, and assisted with the scoring during the afternoon sessions. When most
participants had arrived, Lead Facilitator, Maggie Tymms welcomed everyone,
thanked them for participating, and asked for individual introductions. The CAT
overview was presented using an overhead projector, and nine assessments and
a scoring rubric were provided to each scorer.
The CAT overview consisted of a history and synopsis of the CAT development
process, the purpose of creating the assessment as a tool for improving student
success, best practices, and the importance of assessing Critical Thinking skills.
Please see figures 1, 2 and 3 below from the CAT Overview presented to the
scoring faculty. Figure 1 presented the History of CAT Development, Figure 2
presented the Development of the CAT Instrument, and Figure 3 presented Best
Practices for Improving Critical Thinking.
Critical Thinking Assessment Test (CAT) Scoring Session
2
Figure 1. History of CAT Development
Source: CAT Overview, Center for Assessment & Improvement of Learning, Tennessee Tech
University 2008.
Figure 2. Developing the CAT instrument.
Source: CAT Overview, Center for Assessment & Improvement of Learning, Tennessee Tech
University 2008.
Critical Thinking Assessment Test (CAT) Scoring Session
3
Figure 3. Best practices for improving critical thinking.
Source: CAT Overview, Center for Assessment & Improvement of Learning, Tennessee Tech
University 2008.
Critical Thinking Assessment Test (CAT) Scoring Session
4
Following the CAT Overview presentation and some group discussions, the CAT
scoring sessions began. During each CAT scoring session, the procedure listed
below was followed for each question, beginning with test item number one.
1. The CAT Training Module, presented on a projection screen, provided the
criterion and scoring rubric for a specific test item.
2. Next, a sample test item was presented on the screen, and various
responses were discussed and scored based on the scoring rubric given
for the specific item by the presenter on the training module.
3. Lastly, each scorer reviewed the response provided for the specific item
on his/her first assessment, and scored it based on the scoring rubric.
4. Scorers who encountered a response which did not clearly follow the
rubric discussed the response with the group for clarification.
5. Each scorer then passed the scored assessment to the person on their
right, and the same test item was scored by a second scorer.
6. In the event that two scores differed, the assessment was provided to a
third scorer, and a third score was recorded.
7. When all scoring for the specific test item on all assessments was
completed, the assessments were collected and redistributed randomly.
8. Finally, steps 1 through 7 were repeated for each test item until all
assessments were completely scored.
A fifteen minute break was offered in the morning, and a buffet lunch was
provided during a one-hour break close to noon. Lunch was followed by a
review of the morning scoring sessions, during which discussions ensued,
and scorers suggested ways to share the lessons learned with faculty at their
home campuses.
Once the scoring of all assessments was complete, the assessments were
reviewed to ensure that all items had been scored accurately. The day came
to a close at approximately 3:30 p.m.
The eighty graded assessments were returned to Tennessee Tech University,
together with the required scoring materials.
Critical Thinking Assessment Test (CAT) Scoring Session
5
Results:
The results of the eighty scored assessments show a mean score of 17.6 with a
possible range of scores from 0 to 38. This administration resulted in a maximum
score of 32, and a standard deviation of 5.1. There were 33 males and 46
females, varying in age from 16 to 49. The students reported having earned
between 0 and 125 credits, and were enrolled in six different course sections.
The assessments were aggregated by gender, age, number of credits, course,
and grade point average (GPA).
There was a slight difference between the mean score of male and female
students, as seen in Table 2. This difference may be attributed to the smaller
number of males assessed coupled with the fact that one of the 33 males
received only four points for the assessment.
Table 2
CAT Score by Gender
CAT Score 2010 by Gender
Total
Mean
Standard
Deviation
Minimum
Maximum
Male
33
17.74
4.96
4
26
Female
46
17.42
5.24
9
32
1
19.0
0.0
19
19
Gender
Not
Indicated
Students were divided into three categories by age, and the results were
calculated. The categories were selected based on standard college student age
categories. They included ‘16 to 25’, ‘26 to 44’, and ‘Over 44’. Slight differences
were noted in the mean score of students based on age, as seen in Table 3.
Critical Thinking Assessment Test (CAT) Scoring Session
6
Table 3
CAT Score by Age
CAT Score 2010 by Age
Total
Mean
Standard
Deviation
Minimum
Maximum
16 to 25
61
17.55
5.08
9
32
26 to 44
15
17.60
5.72
4
26
Over 44
4
17.75
2.63
15
20
Age
Range
Students were also divided into categories based on number of credits earned.
The division of these groups was decided based on the number of students
within each group. An attempt was made to have groups close in size. The
groups included less than 13 credits earned, which made up 30.0% of the group;
13 to 29 credits earned, 27.5% of the group; 30 to 40 credits earned, 15.0% of
the group; and more than 40 credits, 27.5% of the group. Differences in scores
were noted in the students with more than 40 credits, as seen in Table 4. In
addition to having a higher mean score, this group also had a higher minimum
score.
Table 4
CAT Score by Credits
CAT Score 2010 by Credits Earned
Total
Mean
Standard
Deviation
Minimum
Maximum
Less than 13
24
17.08
4.58
10
25
13 to 29
22
17.47
6.08
9
32
30 to 40
12
17.00
5.97
4
26
More than 40
22
18.52
4.07
12
26
Credits
Critical Thinking Assessment Test (CAT) Scoring Session
7
Student scores were also aggregated by the course section they were enrolled in
for the administration of the CAT. Students enrolled in the MAC 1105 (2462)
course attained a higher mean score than students enrolled in the other five
courses. The mean score for this group was 19.64 with a minimum of 10, and a
maximum of 32, as shown in Table 5.
Table 5
CAT Scores by Course
CAT Score 2010 by Course Section
Total
Mean
Standard
Deviation
Minimum
Maximum
MAC 1105
(2462)
12
19.64
6.21
10
32
MAC 1105
(571)
22
17.21
4.35
9
25
MAC 1105
(591)
11
16.09
5.67
4
25
STA 2023
(1390)*
14
17.74
4.48
10
26
STA 2023
(1051)
15
17.20
5.35
9
29
STA 2023
(1052)
6
18.00
5.33
10
26
Course
*STA 2023 section 1390 was incorrectly recorded on the CAT assessments as section 1280.
Although the mean scores were not notably different between courses, an
analysis was conducted to understand the relationship between the number of
credits earned and the course in which the student was enrolled. Half of the
students enrolled in the STA 2023 (1052) course had earned more than 40
credits, and this course had the second highest mean score, however this course
had very few students (6). The higher mean score may have been attributed to
the credits earned rather than the course the student was enrolled in during the
administration of the CAT. The course MAC 1105 (591) had the lowest mean
score and the fewest students with more than 40 credits, as shown in Table 6.
Critical Thinking Assessment Test (CAT) Scoring Session
8
Table 6
Relationship Between The Courses and Number of Credits Earned
Course Type
Credits
Earned
MAC 1105
MAC 1105
MAC 1105
STA 2023
STA 2023
STA 2023
(2462)
(571)
(591)
(1390)*
(1051)
(1052)
Less than 13
5
6
6
2
3
2
24
13 to 29
1
11
1
5
3
1
22
30 to 40
3
0
2
2
5
0
12
More than 40
3
5
2
5
4
3
22
Total
12
22
11
14
15
6
80
Total
*STA 2023 section 1390 was incorrectly recorded on the CAT assessments as section 1280.
The scores from the CAT were also compared by GPA. The student without a
recorded GPA had the lowest mean score (13.00). Students (34) with a GPA
between 2.0 and 2.9 had mean scores that ranged from 16.94 to 17.35. As
might be expected, students (37) with a GPA between 3.0 and 4.0 had the
highest mean scores, ranging between 18.13 and 18.81.
Table 7
CAT Scores by GPA
GPA
CAT Score 2010 by GPA
Standard
Total
Mean
Deviation
Minimum Maximum
No GPA reported
1
13.00
0.0
13.00
13.00
Greater than 0 and
Less than 2.0
8
16.13
7.14
9
26
2.0 to 2.4
18
17.35
5.40
4
26
2.5 to 2.9
16
16.94
4.40
10
25
3.0 to 3.4
23
18.13
4.37
10
29
3.5 to 4.0
14
18.81
5.47
10
32
Critical Thinking Assessment Test (CAT) Scoring Session
9
Conclusion:
There are some indications that there was a relationship between the number
of credits earned and the student’s score on the CAT. This could have positive
implications as an indicator for the college. The second major requirement in
meeting the accreditation requirement standards of the Southern Accreditation
of Colleges and Schools (SACS) is a quality enhancement plan (QEP). The QEP is
a significant issue related to student learning that is faculty-driven, and has a
broad-based involvement. Critical thinking has been the QEP focus at SPC.
This measure will assist the institution as one of multiple measures assessing
SPC’s ability to carry out the QEP.
These results suggest an increase in critical thinking skills for students who
have completed more than 40 credits of coursework. There are, however,
some limitations in the analysis. The students with more than 40 credits who
were given the assessment not only had a large number of credits, but were
continuing their education. This factor may make it difficult to draw the
conclusion that the number of credits is the primary cause for the increased
score. There were fewer males (33) than females (46) in the tested group, and
the age distribution is younger (76% under 26) than the overall population at
SPC at which according to the 2010-11 Fact Book, 50% of the students are under
the age of 25.
Despite these limitations in the data collection, the overall CAT administration
and scoring process were highly beneficial to St. Petersburg College. The
faculty who received the training and had the opportunity to utilize the scoring
rubric will have transferable skills they can use in the future with their
students. The administrators and faculty who conducted the training are able
to continue to provide professional development to faculty. The continued use
of quantifiable instruments to determine St. Petersburg College’s effective
implementation of the critical thinking initiative is another example of SPC’s
Institutional Effectiveness model for continuous improvement at the college.
Critical Thinking Assessment Test (CAT) Scoring Session
10
Reference:
Tennessee Tech University, Critical Thinking Assessment Test Overview
Retrieved on July 17, 2008, from http://www.tntech.edu/cat/)
Critical Thinking Assessment Test (CAT) Scoring Session
11
Appendix A: CAT Scoring Session Agenda
CAT Scoring Session Agenda
SPC EpiCenter - 1-324
April 23, 2010
Start Time
Activity
8:00
8:15
8:30
9:00
10:00
10:15
11:15
12:30
2:30
2:45
5:00
Breakfast
Welcome and Introductions
CAT Overview
CAT Scoring Session
Break
CAT Scoring Session
Lunch, Discussion, and Review
CAT Scoring Session
Break
CAT Scoring Session
End CAT Scoring Session
Critical Thinking Assessment Test (CAT) Scoring Session
12
Appendix B: Non-Disclosure Agreement
Critical Thinking Assessment Test (CAT) Scoring Session
13