Assessing Learning of Student
Employees
Douglas S. Franklin, Ph.D.,
Assistant Dean of Students
Kevin A. Smith, Assistant Director for Leadership &
Community Service
Learning Objectives
• Participants will:
1. gain and apply knowledge of theories relevant to
student employment;
2. learn how to develop learning outcomes relevant
to a student employee work;
3. learn how to construct a rubric for use in
employee performance management;
4. learn how to develop a survey to assess student
employee perceptions of growth.
Project Goal
• Define, establish learning objectives,
and assess the achievement of learning
through student employment as it
relates to the institutional core values of
commitment, community, citizenship,
civility, and character.
Why Student Employment?
• 6 Years of Student
Employment Experience at
University of Arkansas
• 20 Years as U.S. Naval
Reserve Officer
Our responsibilities!
• Empower students through intellectual and
practical skills
• Inform students about conditions that affect their
lives.
• Recognize student learning occurs in the active
part of student‟s lives.
“The conception of mind as a purely isolated possession of
the self is at the very antipodes of the truth…the self is not
a separate mind building up knowledge anew on its own
account. (Dewey, 1958 1929, p. 317)
Common Perceptions of Student Employees
• “Many students in work-study programs seek library
positions because they think they will be able to study while
on the job. …"We're not looking for people just to pay
students to study” (Sharon Welsh, director of student employment at
Rutgers University)
• “I cleaned out grease traps for minimum wage. I received no
guidance, took home my paycheck and bought beer” (Student
employee, Ohio University
• “Students can‟t even answer the telephone correctly”
(Anonymous School Director, Ohio University)
Reality
• About 41 % of full-time and 76 percent of
part-time college students ages 16–24 were
employed (NCES, 2009)
“postsecondary education has become our
(nation‟s) core workforce-development
system” Carnevale, A.P (2008)
Percentage of 16- to 24-year-old college students who were
employed, by attendance status and hours worked per week:
October 1970 through October 2009
Source: NCES, 2011
Academic Impact: Mixed Bag
•
•
•
No overall negative relationship between
student employment and educational
performance, although the author
suggested a negative link for African
American and Hispanic students. Canabal
(„98)
Students working less than 15 hours
weekly did not demonstrate a statistically
significant negative impact on GPA (Henry,
„67; Kaiser & Bergen, „68;Merritt, „70).
Students working 1–15 hours weekly had
significantly higher GPA than students
working 16 or more hours. Students
working 1–15 hours weekly had a higher
GPA than nonworking students, NCES
(„94)
•
•
•
•
Little impact on GPA between working
and nonworking groups but student
grades declined with increasing hours of
work (Gleason, „93; Hay & Lindsay,‟69)
Employment had little impact on cognitive
growth, reading comprehension,
mathematics, and critical thinking,
(Pascarella et al., „94)
Increasing hours of student employment
associated with decreased likelihood of
regular class attendance and less regular
review of class materials (Lammers et al.
,2001)
Student employment increased amount of
time required to graduate (Canabal,
1998; Gleason, 1993).
Career Development: Mostly Positive
•
•
•
•
Working students were as persistent, and had as high of a professional
commitment, as nonworking students” (Fjortoft, „95)
Work or internship experiences during college …have a positive net
influence on the development of career-related skills and the likelihood of
being employed immediately after college” (Pascarella & Terenzini, „05)
Graduates with work or internship experience produced higher quality
work, accepted supervision and direction more willingly, demonstrated
better time management skills, and were better able to interact with
coworkers on team projects (Casella & Brougham, „95)
Student employees demonstrated a lack of Moral and Alienative
Commitment which is indicative of a lack of commitment to the
organization (Pereles, „07)
Learning Through Work
• Billett (2001) examined student employment concluded
learning through workplace experiences include:
– access to authentic work activities (i.e. authentic activities,
novel and routine);
– observation and listening (cues and clues – indirect guidance);
– access to more experienced co-workers (direct guidance –
development of heuristics);
– practice (opportunities to reinforce, refine and hone).
“Learning that is personally transformative turns
out to be the learning that involves membership in a
community of practice” Wenger, 1999, p. 32
Developing educational worth from work
(practiced based) experiences (Adapted from Billett, S. 2009)
• Identify experiences to develop, sustain and utilize
students‟ personal epistemologies, including their
critical engagement and reflection.
• Identify and acknowledge the pedagogic potential of
practice experiences,
– How can experiences be integrated into the curricula to
maximize students‟ learning?
• Prepare, position, sequence and identify the most
appropriate duration of experiences
All Employment Not the Same
• Astin (1993)
– observed off-campus work (both full-time and part-time) is
“uniformly negative” but noted on-campus employment has a
completely different impact on students.
– Effects of on-campus employment were characterized as positive in
nature and had the same positive outcomes as other student
involvement.
• Anderson (1981)
– compared with nonworking students who lived on campus, the
author concluded that students who worked on campus (work
study) and lived at home were more likely to persist, while students
who lived and worked off campus were less likely to remain in
school.
Theoretical Framework
• Astin‟s Theory of Involvement
– “students learn more the more they are
involved in both the academic and social
aspects of the collegiate experience.”
“The world‟s problems are not going to be solved by
math and science and technology; they are human
problems, problems of beliefs and values and feelings.
It would behoove higher education to begin to attend
more to these aspects of student development” (Astin,
2003)
Theoretical Framework
• Bridges Transition Theory
1. Ending, Losing, Letting Go – “Letting go of old ways and old
identities”
2. The Neutral Zone – “Going through an in-between time when
the old is gone but the new isn‟t fully operational”
3. The New Beginning – “Coming out of the transition and
making a new beginning”
“It isn‟t the changes that do you in,
it‟s the transitions” Bridges, 2003
Theoretical Framework
• Bloom's Taxonomy of Learning (Domains)
– Cognitive: mental skills (Knowledge)
– Affective: growth in feelings or emotional areas
(Attitude)
– Psychomotor: manual or physical skills (Skills)
“There is more than one type of learning”
Bloom, 1956
Student Employment at OHIO
Future Research? What is the effect of campus location (urban verses rural)
and type (commuter verse residential) on student employment?
Assessing the 5C‟s
• Creating definitions
– Used current higher education literature
– Sample learning outcomes developed from
learning outcome domains and dimensions
from the Council for the Advancement of
Standards (CAS).
5C‟s- Character
• Definition- The moral and ethical qualities
of persons as well as the demonstration of
those qualities in their emotional
responses, reasoning, and behavior.
– Associated with such qualities as integrity,
responsibility, trustworthiness.
• CAS Domain- Interpersonal
• SLO- Establishes mutually beneficial
relationships with others
5C‟s-Civility
• Definition: Implies respect for others, a
willingness to hear and seek others views,
and the exercise of restraint in criticizing.
– Creates an environment where the views and
actions of others are tolerated.
• CAS: Domain: Cognitive Complexity
• CAS: Dimension: Critical Thinking
• SLO: Assesses assumptions and considers
alternative perspectives and solutions
5C‟s: Community
• Definition: Supportive environment where
members with shared values work
collaboratively toward a common purpose or
engage in a common experience.
• CAS Domain: Practical Competence
• CAS Dimension: Social Responsibility
• SLO: Appropriately challenges the unfair,
unjust, or uncivil behavior of other individuals
or groups
5C‟s-Citizenship
• Definition: Process whereby an individual and
the collaborative group become responsibly
connected to the community and society; to be
a good citizen is to work for positive change on
the behalf of others and the community
• CAS Domain: Knowledge acquisition,
constraint, interjection, or application
• CAS Dimension: Constructing Knowledge
• SLO: Makes meaning from text, instruction,
and/or experience
5C‟s: Commitment
• Definition: the level of one‟s personal
investment
– Associated with high levels of intention,
persistence and resolve.
• CAS: Domain: Intrapersonal Development
• CAS: Dimension: Identity Development
• SLO: Identifies and commits to important
aspects of self.
Assessing the 5C‟s
• Measuring Commitment
• Self Assessment (Indirect measure)
– Pre and post testing (Rate the impact
employment had on your ability to:
• Work to overcome obstacles that hamper goal
achievement (Very Low to Very High)
Assessing the 5C‟s
• Testing (Direct measure)
– How many times was student late or noshow? (not implemented)
• Rubric (Example)
SLO
Approaches
Meets
Exceeds
Establishes a
work ethic
Is tardy less than once
a month and misses
work less than once
per quarter
Not tardy and
does not miss
work
Identifies scheduling
problems and seeks to fill
operational weakness
Limitations and Delimitations
• Scope is limited by:
– Lack of cohesive support (ad hoc)
– Lack of experience
– Inconsistent distribution of survey
– Use of indirect measures of proxy pre-post
instruments
• Scope is limited to:
– Ohio University Division of Student Affairs
Common Ground: Developing
Learning Outcomes
• As a result of (describe work experience) the student
will (action verb) (knowledge, skill or attitude).
• As a result of working as a supervisor at the Ping
Center, the student will demonstrate effective verbal
communication skills with patrons.
Bloom‟s Taxonomy
“Objectives indicate what we want our students to learn;
they are explicit formulations of the ways in which
students are expected to be changed by the educative
process” (A taxonomy for learning, teaching, and
assessing . Editors Anderson & Krathwohl, „01)
Multipronged Approach
Personal
Reflection
Performance
Rubric
(Peer & Pro)
Perception
Surveys
Multipronged Approach
Personal
Reflection
Performance
Rubric
(Peer & Pro)
Perception
Surveys
What‟s a Rubric?
Dimension
Scale Level 1 Scale Level 2 Scale Level 3
1
2
3
4
“…a rubric is a scoring tool that lays out the specific
expectations for an assignment” Stevens (2005),
Introduction to Rubrics, p. 3
Learning from the Samurai?
Learning outcome
from participation /
employment
Supportive
developme
ntal or
educationa
l theory
Leadership
Development:
Supervisor
demonstrates
a thorough
understanding
of complex
facility and
program
operations
Social
Change
Model:
See
FALDO,
pg. 94
Outcome Measures
Departmental
Vision and
philosophy
“Learningcentered
environment
that
promotes
the growth
of all
students”
Activity
Description
Facility or
Program
Supervisor
Approaching
Meets
Exceeds
Seeks to
broaden
understan
ding and
acquires
knowledge
Analyzes
and
applies
knowledge
Seamlessly
integrates
actions
into daily
life.
“…a life of service, discipline and compassion…what could be more necessary?”
(The Last Samurai, 2003)
Learning Outcomes Rubric
• Response
– 10 Res Housing
– 70 Campus Recreation
• Data not analyzed yet
.
Performance Management Rubric
• Rubric for Campus Involvement Center
Multipronged Approach
Personal
Reflection
Performance
Rubric
Peer and Pro
Perception
Surveys
Proxy Pre-Post Test
• Distributed Electronically (Survey Monkey)
• Adapted from the Measuring Outcomes from Recsports
Experiences (M.O.R.E.) (Haines & Fortman, 2008)
– Demonstrated construct validity through parallel model design
(Carini, Kuh, & Klein, 2006; Cheln, Li, Gliklich, Chu, Liang, & Wang, 2002;
Elliot, Smith, Smith, & Chambers, 2000; Gonyea, Kish, Kuh, Muthia & Thomas,
2003, Gordon, Ludlum, & Hoey, 2006)
Descriptive Statistics
Area Worked
Campus Involvement Center
Distribution
Response
% Response
Response
Rate
21
5
5.5%
23.8%
Campus Recreation
426
50
55.5%
11.7%
Event Services
291
18
20%
6.1%
Career Services
NA
4
4.4%
Residential Housing
14
14
15.5%
100%
6
3
3.3%
50%
University Judiciaries (DOS)
Responded
106
13.9%
Answered All Questions
90
12.3%
Skipped Questions
16
Descriptive Statistics
Current Job
Count
% of Response
Resident , Facility or Program Assistant
28
29.8%
Supervisor
20
21.2%
Manager or Student Director
17
18%
Specialists (Sports Official, Life Guard, Instructor, Personal Trainer
11
11.7%
Support (Technical Assistants and Receptionists)
16
17%
Graduate Assistants
4
4.2%
Answered Question
94
Skipped Question
12
Descriptive Statistics
Ethnicity
Count
% of
Response
Institutional
Data 2009
African American
4
4.25%
5%
Native American
1
1%
.4%
Asian American
0
0.0%
1.3%
Hispanic
1
1%
1.9%
84
89.3%
84.6%
Multiple
1
1%
Other or unknown
1
1%
Prefer not to respond
4
4.25%
Caucasian
Answered Question
94
Skipped Question
12
Descriptive Statistics
Gender
Response
Count
% of
Response
Institutional
Data 2009
Male
36
40%
48.3%
Female
58
60%
51.7%
Answered Question
94
Skipped Question
12
Relationship to major
Response Count
% of Response
Directly
21
23.1%
Not Directly
72
79.1%
Answered Question
91
Skipped Question
15
Major Areas (Domains)
•
•
•
•
•
•
•
•
Knowledge of Policies and Procedures
Decision Making
Customer Service
Teamwork (Community, Citizenship, Civility)
Pursuing Goals (Commitment)
Effective Communication (Civility)
Managing Personal Affairs (Commitment)
Commitment to Ethics and Integrity (Character)
Knowledge of Policies and Procedures
Learning Outcome
Pre
Ave
Post
Ave
Significance
Ability to use work related policies
105
4.38
90
5.36
***
Ability to find information
105
4.68
90
5.42
***
Adherence to policies
105
4.89
90
5.44
***
Ability to offer helpful information w/o provocation
104
4.42
89
5.46
***
Decision Making
Learning Outcome
Pre
Ave.
Post
Ave
Significance
Making decisions alone
105
4.50
90
5.36
***
Understanding the „big picture‟
105
4.70
90
5.47
***
Demonstrating good judgment
104
4.98
90
5.57
***
Asking superiors for help when necessary
105
4.88
88
5.47
***
Customer Service
Learning Outcome
Pre
Ave.
Post
Ave
Significance
Demonstrating respect when interacting with others
105
5.30
90
5.60
***
Answering questions without hesitation
105
4.55
90
5.39
***
Understanding the reasoning behind rules
104
4.92
90
5.46
***
Teamwork (Community, Citizenship
and Civility)
Learning Outcome
Pre
Ave.
Post
Ave
Significance
Giving constructive feedback
105
4.37
90
5.23
***
Receiving constructive feedback
105
4.53
90
5.23
***
Ability to handle problems w/o arguments
105
4.90
90
5.47
***
Supporting others decisions
105
4.91
90
5.47
***
Raising concerns when necessary
105
4.60
89
5.40
***
Pursuing Goals (Commitment)
Learning Outcome
Pre
Ave.
Post
Ave
Significance
Pursuing individual goals
105
4.91
90
5.46
***
Articulates rationale for personal or
educational goals
105
4.73
90
5.38
***
Makes plans to achieve goals
105
4.80
90
5.42
***
Works to overcome obstacles that hamper
goal achievement
105
4.83
89
5.43
***
Effective Communication (Civility)
Learning Outcome
Pre
Ave.
Post
Ave
Significance
Conveys meaning to others through writing
105
4.54
90
5.20
***
Articulates meaning to others
104
4.59
90
5.32
***
Responds after reflection
105
4.60
90
5.24
***
Demonstrates the ability to persuade
105
4.50
89
5.17
***
Demonstrates the ability to explain abstract ideas
105
4.50
90
5.21
***
Uses appropriate grammar
105
4.89
90
5.46
***
Appears comfortable when presenting to others
103
4.46
89
5.25
***
Managing Personal Affairs
(Commitment)
Learning Outcome
Pre
Ave.
Post
Ave
Significance
Demonstrates self-reliance
105
4.88
89
5.51
***
Manages time effectively
105
4.61
90
5.46
Meets routine obligations w/o assistance
104
4.96
90
5.62
Avoids conflict between work and personal
obligations
105
4.93
90
5.43
Is productive in personal affairs
105
***
***
***
4.90
89
5.48
***
Commitment to Ethics and Integrity
(Character)
Learning Outcome
Pre
Ave.
Post
Ave
Significance
Incorporating ethical reasoning into actions
104
4.88
90
5.48
***
Considering personal values in decision -making
103
4.95
90
5.50
***
Aligning personal values with actions
104
4.85
90
5.51
***
Demonstrates dependability
104
5.06
90
5.61
***
Demonstrates honesty
103
5.15
90
5.66
***
Accepting responsibility for actions
104
5.10
90
5.62
***
Top 9 Learning Outcomes
Learning Outcome
Ave
Significance
Demonstrates honesty (Character)
5.66
Meets routine obligations w/o assistance (Commitment)
5.62
***
Accepting responsibility for actions (Character)
5.62
***
Demonstrates dependability (Commitment)
5.61
Demonstrating respect when interacting with others (Civility and
Community)
Demonstrating good judgment (Character)
5.60
***
***
5.57
***
Demonstrates self-reliance (Commitment)
5.51
***
Aligning personal values with actions (Character)
5.51
***
Considering personal values in decision -making (Character)
5.50
***
***
Job Satisfaction: Ping Exit Survey n=20
•
•
•
•
•
•
•
Goal Setting, Time
management
Adjustment to change
Mentor training
Appropriate training &
supervision
Responsiveness (Pro staff, GA,
Sup, Mgr
Fit with whole experience
Scheduling
•
•
•
•
•
•
•
Fairness of evaluation
Appropriateness of orientation
and staff meetings
Equipment condition
Respect (Pro staff, GA, Sup,
Mgr.)
Co-worker interactions
Ability to express opinions
Value of input
Learning Outcomes Ping Exit Survey CAS Learning Domain (2007)
n=15/14
4 pt scale
Communication
3.53
Collaboration
3.36
Interpersonal Relationships
3.20
Social Responsibility
3.20
Satisfying and Productive Lifestyles
3.14
Leadership Development
3.13
Independence
3.11
Self-esteem
3.07
Healthy Behaviors
3.07
Intellectual growth
2.86
Career Choices
2.80
Self appraisal
2.73
Personal and Educational Goals
2.67
Appreciation for Diversity
2.64
Clarification of Values
2.57
Spiritual Awareness
2.00
Multipronged Approach
Personal
Reflections
Performance
Rubric
(Peer & Pro)
Perception
Surveys
Student Voices
• “Time management, problem solving, „handy‟
knowledge, communication”
• “Leadership skills, organizational skills,
professional skills”
• “Communication, adapting to changing situations,
first hand first aid experience”
Students employees from the Ping Center when answering
the question “List 3 transferable skills learned during
employment in Campus Recreation”
Student Voices: Informal Learning
“I …appreciated the growing responsibilities I acquired
throughout the four years that I‟ve worked here… I‟ve been
entrusted with information …and I‟m relied upon by the
accountants to help complete deposits. It‟s certainly helped to
make me feel more adult and responsible… something that helps
the transition from Athens to the real world easier…On the whole,
I can say I‟ve gotten more satisfaction out of this job than other
part-time jobs, and I think that has everything to do with the
responsibility and trust I was given.”
Student employee : Ohio University Campus
Recreation Business Office
Student Voices: Connecting to the
Classroom
“I had a slight idea of what marketing was in taking
this position, but the job taught a lot about real-world
application and the challenges that arise in
developing and implementing marketing programs. I
learned the job as I performed the job, not only
through hands-on working, but in the classroom as
well.”
Student employee : Ohio University
Amanda J. Cunningham Leadership
Center
Student Voices
• “This job has provided me the opportunity to be heard by
hundreds of students on campus and impact their lives in a
positive way through the content and messages in the
Center‟s presentations. After taking on three quarters of
restless students, bored organization members, and test
faculty, I feel like I could conquer the world.”
Student employee : Ohio University Amanda
J. Cunningham Leadership Center
Student Voices
• “Over the course of the year, I have developed greatly as a
professional and as a student leader. I have greatly
enhanced my skills of professional communication,
presentation, public-speaking, event planning, and
coaching.”
Student employee : Ohio
University, Amanda J.
Cunningham Leadership Center
Student Voices
•
“I have learned so much about myself. …how to take responsibility and how to
create new routines for myself and figure out how to do my job in the best and
most efficient manor [sic]. It was overwhelming at first having so many people
come to me asking questions; I initially would freeze up and get nervous. At
this point I can proudly say that I am much more confident and I enjoy
answering questions and guiding anyone who comes to me in the correct
direction. I have also learned quite a bit about working with others. I have
worked on my collaborative skills, active listening skills, and I have realized
that we all have had something great to contribute to the CIC, and when we
combine all of our thoughts and ideas, we can really do impressive things.
This experience has also helped me confirm that I want to eventually work for
a non profit [sic] organization that in general has the goal to give back.”
Student employee : Ohio University Campus
Involvement Center Center
Student Voices
• “Adaptability is the most important lesson I have learned from this
job.”
Student employee : Ohio University
Amanda J. Cunningham Leadership
Center
Questions
• Doug Franklin franklin@ohio.edu
• Kevin Smith- smithk3@ohio.edu