Annual Outcomes Assessment
Report Workshop:
Support, Research and Service Units
Office of Institutional Planning, Research and Effectiveness (OIPRE)
October 6 – 8, 2015
Drake LRC
1
Workshop Objectives
1.
2.
3.
4.
5.
Provide a rationale for engaging in outcomes assessment
Describe the concept of institutional effectiveness
Outline steps in the outcomes assessment process
Describe AAMU outcomes assessment process requirements
Outline annual outcomes assessment report timeline
2
Rationale for Assessment
• Outcomes assessment is a formative process
• Outcomes assessment is not an end; rather, it is a means for improving unit
processes/functions
• Motivation to engage in outcomes assessment
• Intrinsic value
• External accountability demands
3
Rationale for Assessment
• External accountability demands
• Outcomes assessment is a requirement of our regional accreditor – SACSCOC
• Without regional accreditation our students are not eligible for federal
student aid
• Intrinsic value
• Outcomes assessment allows us to determine how well we are fulfilling our
role at the university and where appropriate, make the necessary changes to
our operations to ensure students reach the expected performance levels
4
Institutional Effectiveness
• Institutional effectiveness is the systematic, explicit, and documented
process of measuring institutional performance against mission in all
aspects of an institution (p. 16).
• A commitment to continuous improvement is at the heart of an ongoing planning and evaluation process. It is a continuous, cyclical
process that is participative, strategic, flexible, relevant and
responsive (p. 16).
Source: Southern Association of Colleges and Schools Commission on Colleges, Resource Manual for the Principles of Accreditation
5
University Mission Statement
Alabama Agricultural and Mechanical University is a public, comprehensive 1890 Land-Grant institution, committed to access and opportunity, of knowledge
and excellence in teaching, research and service is responsive to the needs of a diverse student population and the social and economic needs of the state and
region. The university offers contemporary baccalaureate, master’s educational specialist and doctoral level degrees to prepare student for careers in the arts,
sciences, business, engineering, education, agriculture and technology. As a center of excellence, the University is dedicated to providing a student-centered
educational environment for the emergence of scholars, scientist, leaders and critical thinkers, who are equipped to excel through their contribution and
leadership in a 21st century national and global society.
Strategic Priority 2:
Strategic Priority 1:
Strengthen
Structures,
Operations, and
Systems to
Promote and
Support
Organizational
Excellence and
Stewardship
Enhance AAMU’s
Distinctiveness
and
Competitiveness
UNIT 1
MISSION
UNIT 2
MISSION
Strategic Priority 3:
Strategic Priority 4:
Strategic Priority 5:
Strategic Priority 6:
Upgrade
University
Infrastructure and
Facilities
Secure the
University’s
Financial Future
Enhance the
University’s Image
and Recognition
Enhance university
Engagement
through Expanded
Outreach
UNIT 3
MISSION
UNIT 4
MISSION
UNIT 5
MISSION
6
Institutional Effectiveness
• Comprehensive Standard 3.3.1 (Institutional Effectiveness – unit level)
The institution 1) identifies expected outcomes*, 2) assesses the extent
to which it achieves these outcomes, and 3) provides evidence of
improvement based on analysis of results in the following areas…
*For academic programs this includes student learning outcomes.
7
Institutional Effectiveness
… in each of the following areas:
• 3.3.1.1 Educational Programs
• 3.3.1.2 Administrative Support Services
• 3.3.1.3 Academic and Student Support Services
• 3.3.1.4 Research
• 3.3.1.5 Community/Public Service
8
Assessment Steps
1.
2.
3.
4.
5.
6.
Develop unit mission statement/purpose
Develop outcomes
Develop appropriate assessment measures
Develop desired performance targets
Analyze results
Develop and implement improvement plans
9
Assessment Steps
Mission
Statement
Step 1
Improvement
Plans
Outcomes
Step 2
Step 6
Assessment
Measures
Analysis
Step 5
Step 3
Performance
Targets
Step 4
Unit Mission Statements
• Mission statements should…
•
•
•
•
•
•
Be specific, distinctive
Clearly state the purpose of the unit
Identify the unit’s stakeholders
State the values and principles that guide the unit
Speak to what the unit is currently doing
Align with AAMU’s strategic goals
11
Mission Statement Template
• The mission of <<enter unit name>> is to <<unit’s primary
purpose(s)>> by providing <<unit’s primary functions or activities>>
to <<unit’s primary stakeholders>>.
12
Sample Mission Statements
• The Accounting Office seeks (1) to provide administrators with
accurate and timely financial data to assist them in the management
of the institution’s resources, and (2) to ensure that financial records
are maintained in accordance with generally accepted accounting
principles and guidelines as established by State and Federal
agencies.
• The Office of Assessment serves to promote excellence in student
learning, educational practices and quality of service by establishing
and maintaining a campus culture of self evaluation and improvement
all levels of the institution.
13
Sample Mission Statements
• The library strives to support and stimulate teaching and learning by
providing an environment in which instruction and research can
flourish. The college’s libraries aim to acquire, preserve, provide
access to, and disseminate recorded knowledge in all its
forms. Access will be provided through traditional and technological
methods. The library will provide bibliographic, reference, and
instructional support to student faculty, staff and the community.
• The Registrar’s Office is responsible for processing applications for
admission, processing and reporting student enrollment, maintaining
registered student records, receiving grades, and the processing of
transcript information.
14
Unit Outcomes
Unit outcomes should…
• Focus on current services, processes or instruction
• Be clearly and succinctly stated
• Be under the control or responsibility of the unit
• Be meaningful, not trivial
• Be measurable
• Be actionable; lead to improvements
• Lead to non “yes/no” results
15
Unit Outcomes
Tips for developing unit outcomes:
1. Develop a master list of what the unit does (services or functions)
2. From this list, identify the three most important services or
functions
16
Outcomes and the Annual Assessment Report
• For the annual assessment report, support, research and service units
must report on:
• Three (3) outcomes
17
Types of Outcomes
• Efficiency
• Accuracy
• Effectiveness
• Client satisfaction
• Quality
• Comprehensiveness
• Compliance with standards
18
Unit Outcome Example
Efficiency Example
The Registrar’s Office processes transcript requests in a timely manner.
19
Unit Outcome Example
Accuracy Example
Purchasing accurately processes purchase orders.
20
Unit Outcome Example
Effectiveness Example
The Human Resources office provides effective new employee
orientation services.
21
Unit Outcome Example
Quality Example
The Honors College provides high quality advising to its students.
22
Unit Outcome Example
Comprehensiveness Example
Financial Aid provides comprehensive customer service.
23
Unit Outcome Example
Compliance with Standards Example
The Finance Department complies with standard accounting practices.
24
Critiquing Unit Outcomes
• Information Technology will
procure new networking
equipment.
• Information Technology provides
reliable networking service to
the campus.
• This is not a current
function/process of the office.
Also, this outcome cannot lead
to improvements.
25
Critiquing Unit Outcomes
• The Career Center will ensure
that 90% of graduates obtain
jobs or pursue a higher degree.
• The Career Center effectively
informs students about career or
higher education opportunities.
• This outcome is not under the
unit’s control. As written, the
outcome cannot lead to
improvements.
26
Critiquing Unit Outcomes
• Athletics will ensure fans enjoy
an attractive and entertaining
football schedule.
• Athletics provides athletic
facilities that are safe.
• This outcome is not under the
unit’s control. Also, this outcome
is difficult to measure.
27
Critiquing Unit Outcomes
• Payroll will always issue
paychecks on time.
• Payroll accurately reconciles and
remits benefit deductions.
• This outcome leaves no room for
improvement (100% target).
28
Critiquing Unit Outcomes
• The Library will offer prompt,
courteous, professional service
to students, faculty, and staff
regarding books, periodicals,
ebooks, and interlibrary loans.
• The Library offers satisfactory
service to its constituents.
• This outcome is bundled – the
statement includes multiple
outcomes.
29
Assessment Measures
• Now that you have developed sound outcomes, you must
1. Indicate how the outcome will be assessed
30
Assessment Measures
• Assessment measures should be designed to yield meaningful,
actionable data that can lead to improvements.
31
Assessment Measures
• Should…
•
•
•
•
•
Be directly related to the outcome
Address all aspects of the outcome over time
Provide measurable results
Provide adequate data for analysis
Provide actionable data
32
Assessment Measures
• Should…
• Outline a systematic way to assess the outcome
• Include the following details:
•
•
•
•
What will be collected
When the data will be collected
What assessment tool will be used
How will the data be collected
33
Choosing Assessment Measures
• Direct Assessments
• Are based on direct analysis of the service, process or function in question
• Indirect Assessments
• Are based on stakeholder attitudes and perceptions of the service, process or
function of interest
34
Choosing Assessment Measures
• Where possible, multiple direct assessment methods should be used.
35
Assessment Measures
• Quantitative Assessment Measures
• Response time
• Accuracy
• Cost savings
• Efficiency
36
Assessment Measures
• Client Satisfaction Assessment Methods
• Surveys
• Comment cards
• Focus groups
• Observation of behavior
37
Assessment Measures
• External or Peer Comparison Assessment Methods
• Auditors
• Fire Marshall
• Peer institutions
• Other outside agencies
38
Performance Targets
• Why do we need performance targets?
• Allows the unit to identify desired performance levels
• Provides standard for determining success
• Puts data analysis in perspective
39
Performance Targets
• Targets should…
• Use component (not respondent) as unit of analysis
• Relate to outcome and assessment method
• Be specific (“x% of responses will be correct for each question” or “y% of
scores for each item on rubric or survey will be at least 4”
• Avoid vague words such as “most” or “majority”
• Avoid “all” or “100%” targets
40
Critiquing Measures and Targets
• The Travel Office will conduct a
training session each month on the
institution’s travel policies. At least
75% of all offices will send a
representative to the training.
• As stated, this outcome cannot lead to
improvements. Also, attendance at
the training sessions is out of the
unit’s control.
• The Travel Office will conduct a
training session each month on
the institution’s travel policies
and administer a five-question
survey at the end.
• At least 75% of all responses to
each question will be correct.
41
Critiquing Measures and Targets
• The University Police Office will collect
comment cards from its walk-up service
window. The average score for each card
will be at least a 3.5 out of a possible
score of 5.
• As stated, this outcome cannot lead to
improvements. Also, the outcome uses
averages which focus on the respondent
rather than components. Instead, we
want to know what proportion of
respondents reached the target score for
each item on the card.
• The University Police Office will
collect comment cards from its
walk-up service window.
• For each item on the card, at least
80% of responses will be a 4 of 5
(Satisfied or Very Satisfied).
• The items on the comment card
represent this measure’s
components.
42
Critiquing Measures and Targets
• Custodial Services will use a checklist
to monitor the cleanliness of the
Administration Building’s conference
room for a one-month period. No
more than four of the ten items on
the checklist will be deficient.
• This measure doesn’t assess overall
effectiveness of custodial services.
Need to look more broadly. Also, the
time frame for the assessment is too
limited. Finally, the measure as stated
doesn’t readily lend itself to targeting
improvement efforts.
• Custodial Services will use a
checklist to monitor the
cleanliness of classrooms for a
two-month period.
• No item on the checklist will be
found to be deficient more than
10% of the time.
43
Critiquing Measures and Targets
• The Cafeteria plans to expand its
salad bar substantially in
December. The Cafeteria will count
the number of people who visit its
salad bar before and after the
expansion. The target is an
increase of 10%.
• This measure will not yield
actionable data. Because the
performance target is based on
attendance it may measure things
other than the impact of the new
salad bar (may reflect seasonal
changes in attendance patterns).
• The Cafeteria plans to expand its
salad bar substantially in
December. At the end of the spring
semester, a survey will be
conducted to assess satisfaction
with the expansion. Also, three
focus groups will be conducted to
identify additional potential
improvements.
• For each item on the survey, at
least 75% of responses will be
Satisfied or Very Satisfied.
44
Critiquing Measures and Targets
• Student Support Services has
created a new website. The
number of hits to the website will
be tracked for two months. The
target is an average of ten hits per
day.
• This measure doesn’t provide
actionable data as it simply tracks
website traffic, not the quality of
the website.
• Student Support Services has
created a new website. Several
focus groups representing various
constituencies on campus will be
conducted to determine how
useful, attractive, and functional
the new website is.
• Since focus groups do not yield
quantitative data, no target is set.
45
Analysis of Results
• Here we move from planning the assessment to conducting it
• Provides data on which you can base changes
46
Analysis of Results
Analysis should…
• Provide detailed data (avoid “a majority” or “most”)
• Use specific numbers (no rounding)
• Avoid technical language
• Be clearly and succinctly presented
• Align with outcome and target
• Support actions taken later to foster improvement
47
Improvement Plans
• Should…
• Address outcomes where the performance target was not met
• State what future actions will take place to address the outcome
48
Types of Improvement Plans
• Procedure, process change
• Technology update
• Assessment method change
• Target adjustment
49
Class Example
Outcome
• Institutional Planning, Research and Effectiveness (OIPRE) provides a
website that is informative and useful.
50
Class Example
Assessment Measure
• OIPRE staff will develop a series of 12 questions representing
information that should be readily available from the OIPRE website.
Fifty people from outside of OIPRE will be asked to use the website to
answer the 12 questions and make any suggestions for enhancing the
site.
• OIPRE will create a spreadsheet to record the number of respondents
who found the correct answer to each of the 12 questions. They will
also note any suggestions for enhancing the website.
51
Class Example
Target
• For each of the 12 questions, at least 40 of the 50 respondents will
find the answer.
52
Class Example
Analysis
Question
1
2
3
4
5
6
7
8
9
10
11
12
Topic Area
Answers Found
Enrollment
48
Retention
42
Graduation
45
Majors Offered
34
Academic Leadership
32
Administrative Officers
41
Faculty and Staff
44
Freshmen Profile
47
Salary Information
40
General Campus Info
48
Peer Group
35
OIPRE Contact Information
33
Total Respondents
50
50
50
50
50
50
50
50
50
50
50
50
53
Class Example
Analysis
50 respondents were tasked with finding twelve pieces of information
from the OIPRE website. At least 40 respondents were able to identify
eight of the twelve items. Fewer than forty respondents were able to
identify four of the twelve items (majors offered, academic leadership,
peer group and OIPRE contact information).
54
Class Example
Improvement plans
Respondents indicated difficulty finding OIPRE contact information on
the website. A ‘Contact Us’ banner will be placed at the top of each
page so site visitors are always one click away from OIPRE contact
information. Respondents indicated having to search through multiple
pages to find Deans and Department Chairs and majors offered within
each school. OIPRE will re-format the web page to display a
comprehensive listing of academic leaders and majors on a single page
rather than showing this information for the respective colleges on
college-specific pages.
55
Annual Assessment Report Template
University Mission: Alabama Agricultural and Mechanical University is a public, comprehensive 1890 Land-Grant institution, committed to
access and opportunity, and dedicated to intellectual inquiry. The application of knowledge and excellence in teaching, research and service is
responsive to the needs of a diverse student population and the social and economic needs of the state and region. The University offers
contemporary baccalaureate, master’s, educational specialist and doctoral level degrees to prepare students for careers in the arts, sciences,
business, engineering, education, agriculture and technology. As a center of excellence, the University is dedicated to providing a studentcentered educational environment for the emergence of scholars, scientists, leaders and critical thinkers, who are equipped to excel through
their contributions and leadership in a 21st century national and global society.
Unit Mission/Purpose: Step 1
Unit Outcome
Assessment Methods
Step 2
Step 3
Performance
Targets
Step 4
Assessment Results
Improvement Plans
Step 5
Step 6
Improvements
Implemented
Step 6
56
Assessment Dates
• Assessment Plans Due
• Steps 1 – 4
• Mission, outcomes, measures and targets
• Friday, November 13, 2015
• Assessment Reports Due
• Steps 5 – 6
• Findings, improvement plans
• May ??, 2016
57
OIPRE Contact Information
Office of Institutional Planning, Research and Assessment
306 Patton Hall
James Walke, Director
james.walke@aamu.edu
256-372-8876
Dianne Bozeman, Assessment Coordinator
dianne.bozeman@aamu.edu 256-372-8644
58