Quality Assessment –
Changes in Behavior Following Career Advising
Walden University
Career Services Center
Presenters: Denise Pranke, Career Services Advisor
Lisa Cook, Director
Why a Presentation on Metrics Anyway?
“In God we trust; all others bring
data.”
-W. Edwards Deming
© 2013 Walden University, LLC
2
“Leaning In” to Metrics
• Higher Learning Commission Re-accreditation
Visit
• Obama College Scorecard
• Baldrige Internal Review
© 2013 Walden University, LLC
3
We Need to Look at Process, Not Just Outcome
© 2013 Walden University, LLC
4
Topics
• Developing the assessment process
• Implementing our survey with a positive
student experience
• Compiling and reporting the results
• Other metrics fun!
©2013 Walden University, LLC
5
Why Assessment?
“We define assessment as a continuous process
of gathering and interpreting evidence to tell a
story about the effectiveness of a career
intervention for the dual purposes of continuous
improvement and celebrating successes.”
- Makela, J. P., & Rooney, G. S. (2012). Learning Outcomes Assessment Step by Step: Enhancing EvidenceBased Practice in Career Services (p.1). Broken Arrow, OK: National Career Development Association.
© 2013 Walden University, LLC
6
WHAT are you
Assessing?
© 2013 Walden University, LLC
7
Who We Are:
Walden University
Career Services Mission and Team
Lisa Cook, JD
Director of Career Services
Dina Bergren, MA
Career Services Advisor
Nicolle Skalski, MM
Career Services Advisor
Denise Pranke, MEd
Career Services Advisor
Andrea Obrycki, MA
Career Services Advisor
The Walden Career Services Center aspires to be a cutting edge, global career
services center preparing our learners to navigate career transition over a
lifetime. To achieve this vision, we educate, coach, and advise our learners
to proactively manage their careers.
© 2013 Walden University, LLC
8
What We Assess
© 2013 Walden University, LLC
9
Quantity - Counting Usage
Who uses the Career Services Center?
What reason or topic?
How do they contact us?
© 2013 Walden University, LLC
10
Who uses our one-on-one advising services?
0.5%
March 2013 by School
0.5%
Education
11.7%
15.4%
Health Sciences
Nursing
Management
16.0%
12.2%
Information Systems & Technology
Counseling and Social Service
2.1%
Psychology
Public Policy and Administration
13.8%
College of Undergraduate Studies
3.2%
24.5%
© 2013 Walden University, LLC
11
Center for Continuing Education
What topic?
5%
1%
1 st Appointment
Orientation
3%
One-on-One Appointment Topic 2012
1 st Appointment Orientation
Resume
Job Search
Career Options for Degree
Networking
Interviewing
Self Assessment
Resume
6%
50%
25%
9%
6%
5%
3%
1%
Job Search
9%
50%
Career Options for
Degree
Networking
25%
One-on-One Appointment Topic
60%
50%
40%
30%
20%
10%
0%
© 2013 Walden University, LLC
12
Interviewing
Self Assessment
How do they contact us?
Contact Method 2012
Contact Method 2012
One-on-One Advising
29%
Group Advising
6%
Phone & Email
39%
26%
One-on-One Advising
Group Advising
6%
26%
Webinar Attendees
29%
Phone & Email
39%
Webinar Attendees
Contact Method
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
One-on-One
Advising
Group Advising Phone & Email
© 2013 Walden University, LLC
13
Webinar
Attendees
Quantity: 57,791 Touch Points in 2012
2012 Touch Points
Advising
Phone & Email
Social Media
Webinar
Attendees
Website
Optimal
Resume
Total
3,074
2,937
7,926
1,986
39,753
2,115
57,791
Compared to What
• Started weekly group advising option to orient students to Career Services – saw 25%
increase in advising over 2011
• Increased monthly career webinars attendance by 73% over 2011
• Collaborated with other departments to promote career services with undergrads;
undergrad advising comprised 17% of appointments (matching their overall percentage
of student body)
© 2013 Walden University, LLC
14
More Counting
• Time:
–Response time
– Staff time on specific services
• Cost $$$$$$$$
© 2013 Walden University, LLC
15
Sometimes we have to shift our focus….
“Connect the Dots” Exercise
Connect all 9 dots with 4 straight lines without lifting
your pen/pencil from the paper.
●
●
●
●
●
●
●
●
●
© 2013 Walden University, LLC
16
Hmmm…….. it can be done!
●
●
●
●
●
●
●
●
●
© 2013 Walden University, LLC
17
Quality
“Soon the thought interrupted again. Quality?
There was something irritating, even angering,
about that question. He thought about it, and
then thought some more, and then looked out
the window, and then thought about it some
more. Quality? ”
- Pirsig, R. M., (1974). Zen and the Art of Motorcycle Maintenance: An Inquiry
into Values. (p.177). New York, NY: Bantam Books.
© 2013 Walden University, LLC
18
Developing a Quality Assessment Project
Plan
Conclusions &
Decisions
Design
Compile Data &
Present Results
Implement
& Collect Data
© 2013 Walden University, LLC
19
Quality Assessment Project
Plan
Design
• Entire team effort with a project lead
• Determine what service to assess
• Determine what impact to assess:
Satisfaction, Learning, Behavior, Results
• Who are the stakeholders
• What is the time frame
• Keep it simple
•
•
•
•
Surveys
Rubrics
Focus Groups
Observations
© 2013 Walden University, LLC
20
Design of 1st Appointment
Quality Assessment Project
• During 1st Appointment, Career Services Advisor guides
student on setting two priority action steps based on the
student’s goals.
• After appointment, an e-mail is sent to the student listing the
two priority action steps.
• After one month, Advisor calls student to check on the
completion of steps and ask student: If we were to offer
additional services, what would you like to see offered?
© 2013 Walden University, LLC
21
Implementing the Project
Data
Collection
•
•
•
•
Phone call
Who to call: all or sample
Survey call talking points
Tracking & logging data
© 2013 Walden University, LLC
22
Follow Up Calls
•During the appointment we outlined a
couple of priority action steps, such as…
•I am following up to see how you are
progressing on these steps.
• Have you taken any other steps?
• If we were to offer additional services,
what would you like to see offered?
© 2013 Walden University, LLC
23
So What?
Presenting Results
• Share results
• What to include
• What visual presentation
© 2013 Walden University, LLC
24
1st Appointment Quality Assessment: July – December 2012
Results
Percentage of Students Completing
Action Steps after 1st Appointment
100
96%
82%
80%
88%
84%
88
88%
Number of Students
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Action Steps Completed
80
60
40
52
46
35
19
20
7
0
July
n = 34
Aug
Sept
Oct
n = 45
n = 66
n = 40
Nov
n = 38
Dec
n = 24
0
1
2
3
4
5
Number of Action Steps Completed
July – Dec n = 247
86% of students contacted completed one or more action steps
Mean of steps completed = 1.93
St. Dev. = 1.23
The main reason given for not completing any action steps was time constraints
© 2013 Walden University, LLC
25
Response to Question: If we were to offer additional
services, what would you like to see offered? n = 112
86 students stated that the appointment and/or resources were very helpful, or they could not think
of any additional services. Selected comments:
• You were very helpful. Before meeting with you, I had no callbacks. Since then, I was hired for
an accounting position and have an interview for a college instructor position.
• Such great information! I was pointed in the right direction, improved my CV, and applied at 3
different online colleges.
• "I am really, really glad you called”
• "Very, very helpful"
• Really liked OR. Was easy to use and could do everything I needed
26 students provided suggestions. Selected comments:
• More insider tips on LinkedIn would be great
• Assistance finding field experience & Internships
• Assistance with connecting directly with employers
• Webinar on jobs in psychology
• Job openings & job placement
• Job postings and more networking strategies
• 24 hour chat w/ Career Services
© 2013 Walden University, LLC
26
The Big Picture
Conclusions &
Decisions
• Results were consistent
over 6 months
• Found 86% of students
were taking action steps
• Had a solid measure of
advising impact
• Gained pride and feeling
of accomplishment as a
team
• Decided to change
assessment direction
© 2013 Walden University, LLC
27
MORE METRICS FUN!
© 2013 Walden University, LLC
28
Webinar Quality Assessment
Webinar: Building Your Professional Brand
Do you have any other comments about the webinar presentation?
- The information presented was helpful and I may get more involved with
the different social media programs that are out there. I will also take
advantage of the Career Services Center. Thanks.
What topics would you like to see in future Career Services presentations?
- Healthcare Topics; LinkedIn focused webinar; How to transition up in your
career
© 2013 Walden University, LLC
29
Doctoral Employment Outcomes:
Top 10 Careers in Higher Education
© 2013 Walden University, LLC
30
Doctoral Employment Outcomes:
Top 10 Career Alternatives
© 2013 Walden University, LLC
31
Questions
32
A Final Thought…
“Not everything that counts can be counted. Not
everything that can be counted counts.“
– Albert Einstein
© 2013 Walden University, LLC
33