INSTITUTIONAL EFFECTIVENESS
REQUIREMENTS
Alexei G. Matveev, Ph.D.
NORFOLK STATE UNIVERSITY
Southern University at New Orleans // Workshop on SACS Reaffirmation Preparations
New Orleans, LA // October 16-17, 2008

Two boys are walking down the street. The first boy says, “I’ve been really busy this summer. I’ve been teaching my dog to talk.”
His friend responds, “Wow! I can’t wait to have a conversation with your dog.”
The first boy shakes his head. “I said I’ve been teaching him. I didn’t say he learned anything.” from Mary J. Allen (2004).
Assessing Academic Programs in Higher Education
Anker Publishing, Inc.: Bolton, MA.
2

• Does the University achieve its stated goals?
• How do we know that the University achieves its stated goals?
• What do we do to ensure that the University continues to achieve its stated goals with excellence?
3

1. Importance of Demonstrating Institutional
Effectiveness
1.
Higher education context
2.
SACS Principles of Accreditation
2. Components of Institutional Effectiveness
1.
Characteristics
2.
Evidence
4

• Systematic, explicit, and documented process of measuring performance against the mission in all aspects of an institution
(SACS, 2005, p. 9)
• Intentional process of ascertaining the level of achievement of an institution’s mission and its educational goals and objectives
(SACS, 2005, p. 8)
5

• Continuous Quality Enhancement
– “The concept of quality enhancement is at the heart of the
Commission’s philosophy of accreditation. . .
– Although evaluation of an institution’s educational quality and its effectiveness in achieving its mission is a difficult task. . . , an institution is expected to document quality and effectiveness in all its major aspects”
(SACS, 2001/2004, p. 5)
• Purpose of IE Requirements
– Establish an approach to ensure and facilitate continuous quality enhancement
– Provide evidence of continuous quality enhancement
6

7

Importance of Demonstrating IE
• Changing Student Population
– Students as consumers/customers
– Education as investment
– “Too many decisions about higher education—from those made by policymakers to those made by students and families—rely heavily on reputation and rankings derived to a large extent from inputs such as financial resources rather than outcomes.”
(The Secretary of Education Commission on the Future of Higher
Education, 2006)
8

Importance of Demonstrating IE
• Academic Quality
– The higher education community accepts student achievement of expected learning outcomes as the key indicator of quality
(AAC&U, 2004, p.1; Ewell/CHEA,
2002; Massy, 2003; Ratcliff, 1997; SACS, 2001/2004; Tagg, 2003)
– Whereas “quality” was once defined in terms of inputs and resources – what the institution has— it’s now defined in terms of processes and outcomes—what the institution does with what it has (
Wergin, 2005, p. 36)
9

Importance of Demonstrating IE
• Accountability
– “Postsecondary education institutions should measure and report meaningful student learning outcomes.”
– “The federal government should provide incentives for … institutions to develop interoperable outcomesfocused accountability systems designed to be accessible and useful for students, policymakers, and the public, as well as for internal management and institutional improvement.”
(The Secretary of Education Commission on the Future of Higher
Education, 2006)
10

• Core Requirement 2.5:
Institution-Wide Approach to Continuous Quality Enhancement
– The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that
(1) incorporate a systematic review of institutional mission, goals, and outcomes;
(2) result in continuing improvement in institutional quality; and
(3) demonstrate the institution is effectively accomplishing its mission.
11

(Cont’d)
• Comprehensive Standard 3.3.1:
Program-Level
Evidence of Continuous Quality Enhancement / Outcomes
Assessment. The institution
1. identifies expected outcomes for
– Educational programs
(including student learning outcomes for educational program)
– Administrative support services
– Educational support services
– Research
– Public/community service
2. assesses whether it achieves these outcomes; and
3. provides evidence of improvement based on those results.
12

• CR 2.5 requires that an institution have an effective process for producing improvement and accomplishing its mission.
• CS 3.3.1 requires that an institution identify outcomes (resulting from the process required in CR 2.5), evaluate achievement of those outcomes, and demonstrate improvement based on the results of that evaluation.
13

“Even though the concept of institutional effectiveness may not be explicitly referenced in all of the comprehensive standards, the accreditation process assumes that all programs and services wherever offered within the context of the institution’s activity are reviewed as part of the institutional effectiveness process.”
(SACS,
2005, p. 9)
14

(Cont’d)
• CR 2.9
Learning Resources and Services
• CR 2.10 Student Support Services
• CR 2.11.1 Financial Resources
• CR 2.12 Quality Enhancement Plan
• CS 3.2.8 Qualified Administrative Officers
• CS 3.4.10 Responsibility for Curriculum
• CS 3.8.3
Qualified Staff: Learning Resources
• CS 3.9.3
Qualified Staff: Student Affairs
• CS 3.11.3 Physical Facilities
• FR 4.1 Student Achievement
15

“Support documentation and assessment data is lacking to indicate that student support programs, services, and activities support the institution's mission to promote student learning and development. Throughout the narrative reference is made to using assessment of services utilizing the CAS Professional Standards for Higher
Education and Assessment Practices in Student
Affairs but no examples of assessment instruments or results are given”
(SACS reviewer’s comment)
16

“Sufficient evidence is lacking regarding job placement data, graduate school acceptance, certification/licensures, etc., for those programs not represented by an external accrediting body, as well as some programs with external accreditation”
(SACS reviewer’s comment)
17

The institution engages in ongoing, integrated, and institutionwide research-based planning and evaluation processes that (1) incorporate a systematic review of institutional mission, goals, and outcomes; (2) result in continuing improvement in institutional quality; and (3) demonstrate the institution is effectively accomplishing its mission
.

• Donna Wilkinson, VP SACS-COC
–
Conditions for Effective Planning and Evaluation
Processes
–
Characteristics of Sound Planning and Evaluation
Processes
19

• Mission-Driven IE Process
– “The review process by the Council for
Assessment and Planning covers the number of reports completed and submitted, but it does not serve to assess the value of the reports with regard to assessment of institutional mission. The process lacks a link to the institutional mission, stated or implied, at any level -college/department or upperlevel administration.”
(SACS reviewer’s comment)
20

(Cont’d)
• Systematically Reviewed IE Process
– “While the recent approval of the Mission and
Goals by the Board provides evidence that the college periodically reviews its mission and goals, it does not provide evidence of a systematic review of the mission and goals.”
(SACS reviewer’s comment)
21

(Cont’d)
• Aligned/Coherent IE Process
“While there are institution-wide, integrated, databased planning processes built on a well-defined strategic plan with appropriate goals and objectives that are adhered to by all the units on campus, there is no immediately obvious linkage between a particular goal, an assessment method, specific criteria for success, and actions taken to improve.”
(SACS reviewer’s comment)
22

(Cont’d)
• Systemic and Documented IE Process
“An institutional effectiveness process began in 2005. Not all units have completed the process or if they have, evidence is not shown.”
(SACS reviewer’s comment)
23

(Cont’d)
• IE Process Explicitly Linked to Analysis,
Decision-Making, and Budgeting
“The extensive assessment reports from all the units on campus contain a wealth of specific indicators of success and a lot of raw data, but there appears to be no attempt to interpret the meaning of the data and link them to actions in a way that closes the assessment loop. Neither does the narrative address how funding decisions are linked to the assessments.”
(SACS reviewer’s comment)
24

(Cont’d)
• Improvement-Focused IE Process
– “The [name of the institution’s] plan, however, is an accountability plan, requiring annual reports of enrollment, retention, graduation rates, etc. with no clear evaluation process and no description of how the results of the reports will be used.”
(SACS reviewer’s comment)
– “The report did not provide explicit description of use of results for continual improvement. A “model program review system” was in place, but again, use of results was not specified.”
(SACS reviewer’s comment)
25

(Cont’d)
• Mature IE Process
– “The institution’s processes have been strengthened greatly over the past four years and the institution documents a commitment to continuing improvement of its processes and the use of their results. However, the institution is still developing those processes and has not yet compiled results that may be used to document continuing improvements. The institution needs to document that it has attained results from its institutional effectiveness efforts and used the results for continuous improvements.”
(SACS reviewer’s comment)
26

27

1. Planning
2. Budgeting
3. Accountability and Assessment
4. Feedback Mechanisms
5. Coordination
6. IE Program Support
29

• Structural Element
– Characteristics
– Evidence
30

Planning
• 1.1 Environmental scans
– Characteristics:
• Periodic
• External and Internal
• Triangulated
• Mission-based interpretation
– Evidence:
• Participants and data collection methods
• Findings and analyses
• Communication/distribution
31

Planning (Cont’d)
• 1.2. Strategic Plan development and implementation
– Characteristics:
• Explicitly mission-driven (mission statements are typically reviewed during the strategic planning processes)
• Environmental scan data-based
• Enforced, integrated in annual planning and reporting
– Evidence:
• Strategic Plan
• Strategic Plan flowchart
• Strategic Plan Committee meeting minutes
• Annual Implementation Reports
32

Planning (Cont’d)
• 1.3 Alignment between internal plans and external mandates
– Characteristics:
• Alignment, not hierarchy
– Evidence:
• Crosswalks between plans
• Common indicators
33

Budgeting
• 2. Budgeting
– Characteristics:
• Mission-driven and explicitly aligned with Strategic Plan
• Assessment and evaluation data-based
• Open and participatory process
– Evidence:
• Budget
• Budget development process flowchart
• Budget request forms and instructions
• Budget Committee meeting minutes
34

Accountability and Assessment
• 3.1.1 Annual Reporting
– Characteristics:
• Standardized Annual Report template based on the
Strategic Plan goals
• Activities and outputs-based
• Unit/provider-focused
– Evidence:
• Annual Reports (Units, Divisions, University)
• Annual Report templates and instructions
35

Accountability and Assessment
• 3.1.2 Personnel Evaluation
– Characteristics:
• Based on the Strategic Plan objectives
• Transparent process
– Evidence:
• Personnel Handbooks
• Job descriptions
• Personnel evaluation process flowchart
• Evaluation forms and samples
36

Accountability and Assessment
• 3.2.1 Unit/Program Assessment
– Characteristics:
• Formative
• Mission-driven and Outcomes-based
• Open
• Student/customer/client-focused
– Evidence:
• Matrix of submitted and accepted reports and sample reports
• Assessment report templates and instructions
• Departmental/Committee meeting minutes
37

Accountability and Assessment
• 3.2.2 Core Competency/Gen Ed Assessment
– Characteristics:
• Based on coherent Gen Ed rationale and well-articulated outcomes aligned with the University Mission
• Assigned to designated individual or group (e.g., Gen Ed
Council) but Participatory
• Comprehensive
• Representative
– Evidence:
• General education assessment process flowchart
• General Education
Program assessment report
38

Accountability and Assessment
• 3.2.3 University Assessment
– Characteristics:
• Representative
• Focused on campus climate/environment
• Based on perceptions, satisfaction and self-reported growth
– Evidence:
• Matrix of survey administration
• Reports analyzing and interpreting survey data
• Summary reports, displays, and presentations
39

Accountability and Assessment
• 3. 3 Program Review
– Characteristics:
• Comprehensive and summative
• Periodic
• Based on perceptions, satisfaction and self-reported growth
– Evidence:
• Schedule of program reviews and specialized accreditation visits
• Program review criteria, process flowchart, and instructions
• Program review reports and self-studies
40

Feedback Mechanisms
• Strategic Plan
– University Annual Theme
– President and VP’s annual objectives
• Annual Reports
– Unit Plan for upcoming year
• Personnel Evaluations
– Improvements Needed section
• Assessment Reports
– Improvement Plans and Improvements Made sections
41

Coordination
President’s Cabinet / Strategic Planning Council
• Institutional Research
– Enrollment data analysis and reporting
• Planning and Budget
– Financial planning and reporting
• Institutional Effectiveness and Assessment
– Outcomes assessment and quality assurance
42

Support
• Senior Leadership support and guidance
• Designated individuals/offices to coordinate IE process
• Policies
• Procedures, process flowcharts, forms and templates
• Handbooks and Manuals
• Faculty and Staff Workshops
• Software, technology
• Websites
43

1.
Introduction
2.
Planning
1.
Strategic planning process
2.
Budgeting
3.
Feedback Mechanisms
1.
Evaluation
1.
Annual reports
2.
Personnel evaluation
2.
Assessment
1.
Unit/program
2.
Core learning areas
3.
University
3.
State accountability program reviews and specialized accreditations
5. Coordination of IE process
4. Continuous Quality
Enhancement: Mission-Critical
Indicators
1. Student success
2. Management structures
3. Funding
4. Improvements in IE process
6. Conclusion
44

(from Green Book)
• What are the institution’s processes for systematic, ongoing, integrated, research-based reviews that result in continuous improvement?
• How does the institution demonstrate a sustained, documented history of planning evaluation cycles, including the use of results for improvement, to accomplish the institution’s mission?
• Is there appropriate institutional research and budgetary support for assessment programs throughout the institution?
45

(from Green Book)
• What is the evidence that data from various sources concerning the effectiveness of programs and services are being used to make decisions for improvement?
• How is the institutional effectiveness process related to the budget?
• Are appropriate internal and external constituents and stakeholders involved in the planning and assessment process?
46

47

INSTITUTIONAL EFFECTIVENESS
REQUIREMENTS
Alexei G. Matveev, Ph.D.
NORFOLK STATE UNIVERSITY
Southern University at New Orleans // Workshop on SACS Reaffirmation Preparations
New Orleans, LA // October 16-17, 2008