Assessment of COGS Academic
Support Service Programs
AY 2012 – 2013
Liaison- Susan L. Pocotte, Ph.D.
Associate Dean for Academic Affairs
Why Assess Academic Support Services?
• Student success includes more than
achievement of student learning outcomes
– Student experience
– Preparation for career
– Supplemental procedure responsibilities
• Students quit due to poor services
• Academic support services can have direct
impact on student success
• Service is part of student-centeredness
Stakeholder participation in
COGS Assessment
• COGS Administration
– Conduct assessment and prepare reports
•
•
•
•
•
•
Council of Associate Graduate Deans
Graduate Council
Graduate Council Executive Committee
Graduate Faculty
Graduate Students
University Assessment Committee
Graduate Student Involvement in the
Assessment Process
• Graduate student involvement in COGS
assessment process is primarily through
the survey response.
• Graduate students are voting members of
Graduate Council and GCEC
– Opportunity to provide feedback to
assessment reports and processes
Best Practice #1
Strategically Aligned COGS
Assessment Plan
Strategic Alignment of Academic SSOs
• UT and COGS Missions and Strategic Plans
• COGS services
• HLC six fundamental questions of student
learning assessment
• Incorporate CGS-ETS 2012 CommissionRecommendations for Universities
• Program specific accreditation criteria
COGS Assessment Plan Template
Service Categories
+
Measurable Outcomes
COGS Service Categories
A. Recruitment of Graduate Student applicants
B. Application and Admission processes
C. Orientation processes: graduate students
D. Student progress and compliance
E. Graduate Student Development Workshops,
Research forums
F. Graduate Student Academic Appeals and
Grievances
COGS Service Categories
G. Graduate student financial support
H. Graduation, audits, clearances
I. Graduate faculty membership
J. Graduate faculty communications
K. Graduate program, course approvals
L. Graduate program review process
A. Recruitment of Graduate Student applicants
Data collection
Applicable service
methods, metrics and
outcomes
sources
Data collected is the
designation of
complete or
incomplete.
Data collected is the
designation of
complete or
incomplete.
Data collected is the
designation of
complete or
incomplete.
Data collected is
number of applicants.
Data collected is the
designation of
complete or
incomplete.
1. Collaborate with UT
administration on the
adaptation of the
“recruiter” electronic
system that tracks
communication with
prospective applicants
for utilization in COGS.
2. Develop/revise
recruitment materials
to promote UT
graduate academic
programs to
prospective applicants.
Direct or Indirect
Measure of service
Frequency of data
Person(s) responsible
collection and review for reviewing data
Assistant Dean for
Graduate Admissions
Direct
Annually
Graduate Orientation
and Enrollment
Specialist
Assistant Dean for
Graduate Admissions
Direct
3. Develop a
recruitment plan to
attract TRIO
undergraduate
Direct
students to apply to UT
Graduate programs and
COGS scholarships.
4. Establish
relationships with UT
enrollment
management and
external organizations
Direct
to identify processes
for identifying potential
applications to UT
Annually
Graduate Orientation
and Enrollment
Specialist
Assistant Dean for
Graduate Admissions
Annually
Graduate Orientation
and Enrollment
Specialist
Assistant Dean for
Graduate Admissions
Annually
Graduate Orientation
and Enrollment
Specialist
H. Graduation, audits, clearances
Data collection methods, Applicable service
metrics and sources
outcomes
Data collected from
COGS staff reports;
COGS graduate
student exit survey.
Data collected from
COGS staff reports;
COGS graduate
student exit survey.
Data collected from
COGS staff reports;
COGS graduate
student exit survey.
Direct or Indirect
Measure of service
1. Review
mechanisms/workflows
to enhance efficiency
and timely processes.
Successful
communication as
Direct and Indirect
indicated by applicants
as indicated by
satisfaction scores >
85%.
2. Track student issues
that delay or prevent
clearance to inform
students earlier.
Successful
Direct and Indirect
communication as
indicated by satisfaction
scores > 85%.
3. COGS staff and
communication
processes are
responsive, effective
and efficient for
Direct
students as indicated by
satisfaction scores >
85%.
Frequency of data
collection and review
Person(s) responsible for
reviewing data
Dean
Assistant Dean for Academic
Affairs
End of Fall, Spring,
Summer semesters.
Thesis and Dissertation Services
Specialist
Manager of Administrative
Services- HSC
Graduation Services Specialist
Dean
Assistant Dean for Academic
Affairs
End of Fall, Spring,
Summer semesters
Thesis and Dissertation Services
Specialist
Manager of Administrative
Services- HSC
Graduation Services Specialist
Dean
Assistant Dean for Academic
Affairs
End of Fall, Spring,
Summer semesters
Thesis and Dissertation Services
Specialist
Manager of Administrative
Services- HSC
Graduation Services Specialist
Best Practice #2
Strategic and Flexible
Metrics/Measurements of Academic
Support Service Outcomes
Data Collection:
Metrics, Methods and Sources

Direct measure of overall student success is difficult
– Measure individual indicators and extrapolate to
overall success
– Use indirect methods (e.g. student survey)
• Primary measurement tools
–
–
–
–
Survey results
Metrics of academic service delivery
Keep it simple and efficient!
Avoid temptation to survey “everyone” for
“everything”
– Allow anecdotal observations
Best Practice #3
Measurement tool(s) to collect both
centralized and unit-specific data.
Graduate Student Exit Survey
• COGS has conducted an exit survey of all
graduating graduate students since Spring
2006.
- Questions address services provided by COGS,
other academic service units, and the
College/Department that provides the graduate
program. (centralized data!)
• Beginning Spring 2011, COGS required
students to complete the survey.
-Response rate 99%
Trended Data from the Graduate
Student Exit Surveys
•
•
•
•
Aggregate data
Students assured responses anonymous
27 specific questions
One open-ended question
– Please make any comments you wish to share
regarding your UT graduate experience.
To what extent do you agree/disagree with the
following statements about your graduate experience:
• Overall, the quality of my graduate educational
experience was high.
• My program was intellectually challenging and
stimulating.
• My academic program prepared me well for my
professional career goals.
• My advisor and I met at appropriate intervals to discuss
my program of study.
• My dissertation/thesis/project advisor discussed my
research with me on a regular basis.
To what extent do you agree/disagree with the
following statements about your graduate experience:
• My dissertation/thesis/project advisor critiqued
my work in ways that helped my work progress.
• Information about academic policies and
procedures was communicated sufficiently.
• Equipment and facilities were available when I
needed them.
• There were adequate university services to assist
with non-academic issues (health needs,
personal, family).
Please indicate your level of satisfaction with the
quality of information/advice/service received from the
following sources:
• Advisor
• Members of my doctoral/thesis/project
committee
• Staff in my department's office
• Department's Web-site
• Financial Aid
• Registrar
• Career Development
• Information Technology
COGS specific
• Please indicate your level of satisfaction with
the quality of information/advice/service
received from COGS Web-site.
• Please indicate your level of satisfaction with
the quality of information/advice/service
received from the following sources:
– College of Graduate Studies Staff
To what extent do you agree/disagree
with each of the following statements?
• The College of Graduate Studies was accessible to
me.
• The College of Graduate Studies was responsive.
• The College of Graduate Studies performed its
functions effectively.
• The College of Graduate Studies performed its
functions efficiently.
• The College of Graduate Studies was consistent in
service delivery.
General
• What was your employment status for most of
your graduate education?
• Which of the following statements BEST
describes your post-degree employment
status?
• Location of employer or anticipated location
of employer.
• Please make any comments you wish to share
regarding your UT graduate experience.
Trended Data from the Graduate
Student Exit Surveys
• Indicates sustained achievement of COGS
academic support service outcomes related to
students.
– 80-90 % of the respondents were satisfied or very
satisfied
• Indicates sustained satisfaction with other
service units and academic depts.
– 80-90 % of the respondents were satisfied or very
satisfied
Q. Which of the following statements BEST describes your post-degree
employment status?
I expect to continue with my current
employer in my current position.
I expect to continue with my current
employer in a new position.
I expect to return to a previous
employer in a new or previous
position.
I will begin a new position with a new
employer.
I will be self-employed
I will continue my graduate education.
I will look for employment
I will not be employed.
I will not look for employment.
No Response
Spring
2011
Summer
2011
Fall
2011
Spring
2012
Summer
2012
Fall
2012
Spring Sum
2013 mer
2013
16.9%
18%
18%
13%
20%
24%
16%
21%
9.1%
5%
7.5%
8%
6%
11%
10%
6%
0.7%
2%
2%
1%
0.5%
0%
1%
2%
28.6%
27%
31%
32%
26%
30%
32%
31%
1.1%
32.2%
9.5%
0.6%
0.2%
1.1%
1%
23%
23%
1%
1%
0%
1.5%
34%
5%
0.5%
0%
0.5%
1%
32%
10%
1%
1%
1%
0.5%
24.5%
21%
0.5%
1%
0%
2%
24%
7%
1%
0%
1%
0%
31%
9%
1%
0%
0%
1%
17%
20%
0%
2%
0%
Location of employer or anticipated location of employer.
Spring
2011
Summer
2011
Fall 2011
Spring
2012
Summer
2012
Fall 2012
Spring
2013
Summer
2013
Ohio
65.6%
64%
57 %
63%
54%
56%
62%
54%
Out of State
USA
29%
31%
38%
33%
31%
35%
30%2%
29%
Outside USA
2.9%
3%
5%
3%
3%
3%
2%
4%
No Response
2.6%
2%
0%
1%
12%
7%
6%
0%
Best Practice #4
• COGS has a seat on the UAC
– Dialogue
– Academic units continuously learn about COGS
support services
• Break down barriers
• Leadership in improvement of UT assessment
– New synergies and efficiencies
• Prevent duplication of efforts
• Reduce cost of assessment
Best Practice # 5
• Leverage assessment to adapt and survive
external and internal pressures
– Demonstrate relevance
– Advocate for graduate students
– Provide leadership in implementing new strategic
initiatives in support service in graduate education
– Collaboration between COGS and Academic Units
– Inform UT and COGS Strategic Planning
Best Practice #6
COGS
Indicators of Continuous
Improvement
• Assessment data analysis informs decision
making
– Academic Service Outcomes
– Assessment Plan and Process
COGS Annual Assessment Plan
and Process
Establish
Outcomes
and
Metrics
Revise the
process
and
template if
needed
Does the
evidence
support the
changes?
Systematic
Continuous
Improvement
Collect and
analyze the
Data
Does the
evidence
support the
outcomes?
Indicators of Continuous
Improvement
• April 2, 2014: Participate in Workshop with Dr.
Barbara Walvoord- National assessment
expert and author consulting at UT
Indicators of Continuous
Improvement
• UAC feedback to COGS report AY 2012-2013
• Strengths
– Recognition of and the commitment to
assessment as a part of the core mission and
values of the college
– Assessment results are shared with a variety of
constituents
– Promoting UT’s assessment process by presenting
at external conferences
• Areas for improvement
– Continue to work with GSA to identify a graduate
student to participate on the COGS Assessment
Committee
COGS Leadership in Professional
Organizations
Council of Graduate Schools
• Annual Meeting December 2013
• Dr. Pocotte - Presenter and co-facilitator at
pre-conference workshop
• “Assessment and Review of Graduate
Programs-Master’s”
•
Assuring quality is central to effective management of graduate programs. This workshop will
address the evaluation of master’s level programs and emphasize outcomes-based
assessment. A best practices summary will be followed by descriptions of outcomes
assessment, representing various stages in the development of the assessment process. The
facilitators will discuss strategies for making program assessment meaningful and
manageable, dealing with faculty resistance, assessing student learning, integrating outcomes
assessment with program review, and meeting accreditation requirements.
Strategic Assessment of Graduate
College Academic Support Services as
Part of University Assessment
Susan L. Pocotte, PhD
Associate Dean for Academic Affairs
College of Graduate Studies (COGS)
The University of Toledo
Toledo, Ohio
CGS December 4, 2013