Using Grades over 26 Years to
Evaluate a Career Course:
Two Studies
Robert C. Reardon, Ph.D.
Stephen J. Leierer, Ph.D.
Donghyuck Lee, M.Ed.
Florida State University
Course History
•
•
•
•
•
1973: The career course developed
1980: The team-teaching of 3 or 4 instructors
1981: From the quarter to semester system
1984: Plus/minus grading system
1984 – 1988: Catalyst materials introduced.
Course History
• 1994: Cognitive Information Processing (CIP)
theory introduced.
• 1996 – 2000: Once per week sections offered.
• 1998: Friday section offered.
• 1999: CIP text and student manual introduced.
• 1999: Internet used in students’ research
Course Objectives
1. Perceive the individual as purposefully responsible
and active in the life/career planning process and to
develop skills for increasing such behavior in others
and oneself.
2. Understand how personal characteristics influence
career development.
3. Become oriented to the socioeconomic world of
work as it impacts individual and family career
systems.
4. Identify appropriate academic major and/or
occupational alternatives in relation to personal
characteristics.
Course Objectives
5. Learn about and use a variety of information
resources to explore academic major or
occupational options.
6. Understand career development theories and use
decision-making skills for life/career planning and
management.
7. Learn about and use job-hunting skills needed for
employment.
8. Formulate action plans and strategies for moving
oneself or other persons toward implementation of
life/career goals.
Course Activities
Activities & Assignments
Total Points
Related Course Objective(s)
Participation
30
1
Attendance
58
1
Chapter 1 Quiz
10
6
Autobiography
60
2
SDS Interpretive Report
10
2,4,6
2 Occupational Interviews
40
1,3,5
Career Field Analysis
200
3,4,5
Skills Assessment Activity
10
2,4
SIGI PLUS or Discover Computer Feedback Form
10
2,4
Choices Computer Feedback Form
10
2,4
Career Thought Inventory Profile
10
1,6
Individual Action Plan
10
1,5,8
Academic/Career Plan Project
100
1,8
Resume
15
7
Cover Letter
5
7
Performance Tests
75
2,3,5,6,7
Course Structure
• Unit I: “Career Concepts and Applications”
– Focuses on self-knowledge, knowledge about
options, and decision making
– Assignments: Writing an autobiography,
completing the Self-Directed Search, and a skills
assessment activity.
Course Structure
• Unit II: “Social Conditions Affecting Career
Development”
– Focuses on current social, economic, family, and
organizational changes affecting the career
planning process and the need for students to
develop more complex cognitive schema to solve
career problems.
– Assignments: Complete autobiography, Career
Field Analysis (CFA) paper, and two information
interview reports
Course Structure
• Unit III: “Implementing a Strategic Career
Plan”
– Focuses on employability skills and strategies for
implementing academic/career plans.
– Assignments: Two information interview reports,
the completion of a resume and cover letter, and a
strategic/academic career plan paper
Instructional Methods
• A mixture of lecture, panel presentations, and
small and large group activities.
• Each instructor works with a small group of
students in breakout sessions and evaluates
their work.
• Instructors meet individually with the students
to discuss their assessments and progress.
Course Grading Procedures
• Grades are based on the successful execution
of a performance contract (PC) by the student.
• The PC includes 16 different graded activities
spread across the three units of the course.
• 28 different activities are graded in the 3-credit
version of the course.
Population
• 6,176 undergraduate students who completed
the course, “Introduction to Career
Development.”
• 15% to 25% of the class composed of students
with officially undeclared majors.
• 60% unsure, dissatisfied, or undecided with
current career situation.
• 75 academic periods (semester/quarter)
studied.
Gender Distribution
Male
40%
Female
60%
Grade Distribution
Seniors
20%
Freshmen
15%
Juniors
20%
Sophomores
45%
Ethnic Distribution
74%
12%
4%
American
Indian
3%
African
American
Asian
7%
Hispanic
American
4%
Caucasian
Other
Satisfaction with
Current Career Situations
40%
60%
Satisfied
Unsure, Dissatisfied, or undecided
Data Collection Procedures
• Study #1: Archived course grade data 19782004 by academic term obtained from the
university registrar.
• Study #2: Archived course grade data and
student evaluation of teaching (SET) data
1999-2004 by course section obtained from
registrar and instructors.
Working with Archival Data
• Retrieving electronic grades by historical term
– Aggregate group records by quarter/semester
– Precluded analysis of student characteristics
• Issues in retrieving SET from faculty
– Confidentiality and security
– Course section data
• Large dataset: 6,176 students over 75 terms
Research Questions Study #1
1. Did grades provide evidence of students
meeting the course learning objectives?
2. How did changes in course structures and
procedures affect student learning?
3. Did grades differ by semester?
4. Did the mean grade point average earned in
the course change over time?
Results
• Evidence of Students Meeting the Course
Learning Objectives
– 74% of the students earned a B+ or better
– Mean GPA was 3.44 (SD = .84)
– Negative correlation between mean semester GPA
and the semester identification number (r = -.38, p
= .002)
Results
45.0%
44.4%
40.0%
35.0%
30.0%
25.0%
19.2%
20.0%
13.4%
10.0%
15.0%
10.0%
5.0%
3.7%
2.8%
1.8% 2.5% 0.9%
0.3% 0.6% 0.3%
B-
C+
0.0%
A
A-
B+
B
C
C-
D+
D
D-
F
Results
• Eight course objectives are connected to the 16
graded course learning objectives.
• 3% receiving ‘F’ grade provides evidence of
the course demands.
Results
• Course Structures and Procedures
M
SD
F (1, 6174)
Quarter system
Semester system
3.47
3.44
.88
.84
.28
5-point grading system
11-point grading system
3.48
3.44
.92
.84
1.09
Results
• Course Structures and Procedures
M
SD
F (1, 6174)
Semester without Catalyst materials
Semester with Catalyst materials
3.41
3.53
.88
.74
25.30*
Before CIP integrated into the course
After CIP integrated into the course
3.56
3.33
.75
.92
115.06*
Before textbook used
After textbook used
3.51
3.24
.80
.90
127.75*
Results
• The changes in semester system and grading system
did not produce a difference in course grades.
• The intensive infusion of the work-family life balance
materials associated with higher grades.
• The Infusion of CIP theory and textbook made the
course more challenging and lowered grades.
– Using internet-based sources in researching occupations
resulted in lower grades on the career field analysis (CFA)
research paper.
Results
• Grade Distribution by Semester
– Significant difference in the aggregated GPA by semester
(F = 6280.86, p < .0005)
3.58
3.6
Aggregated GPA
3.55
3.5
3.48
3.45
3.39
3.4
3.35
3.3
3.25
Spring
Summer
Fall
Results
• Course grades varied by semester.
• Grades in the summer were significantly higher than
in other terms.
– The summer term provides a more intensive course
experience (intensive class schedule; students’ study load).
• Grades in the fall were significantly lower than in
other semesters.
– Four-month intermission after registration (perhaps lowed
motivation).
Results
• Grade Distribution over Time
Aggregated GPA
– Significant difference in the aggregated GPA by time
= 23.69, p < .0005)
4
3.9
3.8
3.7
3.6
3.5
3.4
3.3
3.2
3.1
3
(F
1: Fall 1978 - Spring 1981
2: Fall 1981 - Summer 1985
3.63
3.5
3.47
3: Fall 1985 - Summer 1990
3.54
4: Fall 1990 - Summer 1995
3.36
5: Fall 1995 - Summer 2006
3.23
1
2
3
4
Time Period
5
6
6: Fall 2000 - Summer 2004
Results
• Students in the latest time period (fall 2000
through summer 2004) had significantly lower
grades than those in any other time period.
• Grade inflation was not the case with this
career course.
Conclusions
• The career course investigated in this study
appears to be an effective intervention as
evident in student grades.
• Grade inflation was not the case for this
course.
• However, grades were affected by historical
events, temporal conditions, and course
modifications.
Limitations
• Using aggregate grades across academic terms
rather than individual student grades.
• This precluded an examination of ethnicity,
gender, or other learner characteristics in this
research.
Research Questions Study #2
1. What was the nature of students’ evaluation of
teaching (SET) in this career course?
2. Were earned or expected grades in class sections
different across semesters or class meeting times?
3. Were earned or expected grades related to SET
ratings?
Sample for Study #2
•
Fall 1999 – Summer 2004
•
62 course sections led by 12 different
instructors who taught from 1 to 10 times
•
74% of sections reported expected grades (we
found no bias pattern in missing data)
•
92% of sections reported SET ratings
Results
Overall Student Evaluation of Teaching
(SET) very positive
Data Analysis
1 Day
2 Days
3 Days
4 Days
Fall




Spring




Summer




•
The ideal design to examine the research questions about SET, Earned
Grade Point Average, and Expected Grade Point Average would be a
Split-Plot design.
The data were archival, there are no observations for some Semester X
Days combinations. There are two possible solutions to this problem
•
–
–
Split-plot design using ANOVA for unbalanced data (paper)
Create a 7-level variable of Semester-Days (NCDA presentation)
Results
Student Evaluation
of Teaching (SET)
Students’ Evaluation of Teaching
Results
Student Evaluation
of Teaching (SET)
Earned Grades across Class Semesters and Times
Results
Student Evaluation
of Teaching (SET)
Expected Grades across Class Semesters and Times
Student Evaluation
of Teaching (SET)
Results
Earned-Expected Grades across Class Semesters & Times
Results
Student Evaluation of Teaching (SET)
Earned and Expected Grades with SET
Student Evaluation
of Teaching (SET)
Summary
Fall-Spring 1, 3
Fall-Spring 2
Summer 4
Student
Evaluation
of Teaching
Less than
the Mean
Greater than
the Mean
Less than
the Mean
-
+
-
Earned
Grade
Less than
the Mean
Less than
the Mean
Greater than
the Mean
-
-
+
Less than
the Mean
Greater than
the Mean
Greater than
the Mean
-
+
+
Expected
Grade
Student Evaluation
of Teaching (SET)
Summary
• The aggregated earned grade in the Summer semester
was significantly higher than the aggregated earned
grade during the Fall or Spring semesters.
• The aggregated expected grade was significantly
different across semesters.
Summer > Fall-Spring-2 > Fall-Spring 1, 3
• The aggregated expected grade was significantly higher
than the aggregated earned grade.
• The difference between earned and expected grades was
influenced by the Semester-Days variable.
Summer > Fall-Spring-2 > Fall-Spring 1, 3
Student Evaluation
of Teaching (SET)
Summary
• SET ratings on the difference of aggregated earned and
expected grade varies according to semester-class times
Summer > Fall-Spring-2 > Fall-Spring 1, 3
• Difference in EGPA and XGPA influenced by SET
Student Evaluation
of Teaching (SET)
Conclusions
• When controlling for SET, there was no significant
difference between earned GPA and expected GPA.
• Aggregated EGPA and XGPA were influenced by
variables related to the process of teaching, e.g.,
number of class meetings per week (1, 2, 3) and
length of the semester (6 vs. 17 wks.).
• A significant amount of variability that exists between
earned and expected grades can be traced to Student
Evaluation of Teaching (SET).
Student Evaluation
of Teaching (SET)
Limitations
• Archival data analysis can become complex.
• Most of the variables in these studies are aggregate
measures.
• Using aggregated grades as individual performance
indicators can be misleading.
• SET is voluntarily collected, making for poor coverage and
probably poor utility for an ongoing archival project.
• It is often unclear whether SET came as a result of the
outcome (high or low expected and earned grades) or it led
to the outcome.
• These data are based on archived records collected by the
University in multiple classes for the purpose of grading
students in a class.
Student Evaluation
of Teaching (SET)
Contact Information
Presentation available at:
www.career.fsu.edu/techcenter
Robert Reardon, PhD
rreardon@admin.fsu.edu
Steve Leierer, PhD
sleierer@memphis.edu
Donghyuck Lee, MEd
ryan_dhl@yahoo.com