Using Institutional Effectiveness to
Build a Culture of Performance
Improvement
Revised version for
the 2011 SACS
Working Group
Department of Institutional Research and Effectiveness
St. Petersburg College
P.O. Box 13489
St. Petersburg, FL 33733
(727) 341-3059
FAX (727) 341-5411
2007 SACS-COC Annual Meeting
Issues with IE
Using IE to Build a Culture of Performance Improvement
 SACS’s Peer Review Research Project
revealed that in the area of Institutional
Effectiveness (IE)
 62% of institutions in off-site reviews were
deemed non-compliant and
 27% of institutions in on-site visits were
deemed non-compliant
 IE was the second most identified area
behind faculty qualifications
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
2
SPC Experience with IE
Using IE to Build a Culture of Performance Improvement
 As part of the recent SACS reaccreditation
process, SPC spent a considerable amount of
time and effort documenting and detailing their
IE processes
 The result…SPC had “zero” compliance issues
in the area of IE during the off-site reviews
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
3
SACS Requirements for IE
Using IE to Build a Culture of Performance Improvement
 Core Requirement 2.5: Integrated Planning & Evaluation Processes
Institutional Effectiveness
 “…ongoing, integrated, and institution-wide research-based
planning and evaluation processes…”
 Comprehensive Standard 3.3.1: Institutional Effectiveness
 “…identifies expected outcomes…,assesses whether the extent
to which it achieves these outcomes, and provides evidence of
improvement…”
 Comprehensive Standard 3.5.1: General Education Core College
Level Competency
 “… identifies college-level general education competencies and
the extent to which graduates have attained them. within the
general education core and provides evidence that graduates
have attained those competencies.”
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
4
SACS Requirements for IE
Using IE to Build a Culture of Performance Improvement
 Additional IE related requirements include:
 Comprehensive Standard 3.1.1: Institutional Mission
 Federal Requirement 4.1: Evaluating Success
Student Achievement
 Federal Requirement 4.2: Program Curriculum
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
5
IE Evidence
Using IE to Build a Culture of Performance Improvement
 SACS Suggested Documentation for CR 2.5
 Evidence of linkage of IE to institutional mission
 Institutional plans and budgets that demonstrate linkage of
assessment findings to planning at all levels
 Minutes of appropriate (IE related) unit, committee, task
force meetings…
 Documentation that relates to IE, such as budget
preparation instructions, minutes of budget presentation
meetings, annual reports, annual assessment updates, IE
reports
 Samples of specific actions taken to improve the IE process
and/or results from that process
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
6
Performance Improvement
Using IE to Build a Culture of Performance Improvement
 From Compliance to Performance Improvement
Community
College Leadership
Program Directors
Faculty
Students
Assessment
Data
Reports
SPC
Data
Information
Knowledge
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
Improved Outcomes
Improvement
7
SPC’s Definition of IE
Using IE to Build a Culture of Performance Improvement
Mission
Evaluations/ Strategic & Unit
Assessments
Planning
Budget Planning Operating Results
 “At SPC, Institutional Effectiveness is the integrated,
systematic, explicit, and documented processes of
measuring performance against the SPC mission for
purposes of continuous improvement of academic
programs, administrative services, and educational
services offered by the college.”
“Closing the Loop”
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
8
IE Phases
Using IE to Build a Culture of Performance Improvement
 IE Processes are yearly processes that
consist of three distinct phases:
 Evaluation/assessment phase
 Strategic/budget planning phase
 Implementation/operation phase
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
9
Evaluation/Assessment Phase
Using IE to Build a Culture of Performance Improvement
 Reflective look back at the prior year’s assessment and
evaluation results.
 educational outcomes
 institutional surveys
 departmental-specific assessments
 Individual assessments results are reviewed by the
respective oversight group which in turn aggregates the
information and presents recommendations to the
President’s Cabinet.
 These recommendations are evaluated and are routinely
summarized in the form of the Institutional Objectives
(IOs).
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
10
Strategic/Budget Planning Phase
Using IE to Build a Culture of Performance Improvement
 Transition from assessment/evaluation results to
planning for the upcoming operational year
 Includes assessment-driven unit planning
 Unit managers state their intent for the upcoming year
in measurable objectives
 Unit managers ensure alignment with the institution’s
mission and goals, Strategic Directions and
Institutional Objectives (SD-IOs)
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
11
Implementation/Operation Phase
Using IE to Build a Culture of Performance Improvement
 Execution of the developed unit plans
 Unit managers are required to provide periodic
reports of their progress to ensure the fidelity
and stability of these unit plans
 Periodic reports include a “mid-year status
report” and an “end-of-year final results report.”
The end-of-year report includes assessment
results in addition to the unit’s use of these
results in improving their unit.
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
12
Individual IE Processes
Using IE to Build a Culture of Performance Improvement
 Individual IE processes are designed to result in
continuous improvement and comprise the following:
a. Mission statement process
b. Evaluation and assessment processes for academic
programs, student services, and administrative services
c. Strategic Directions and Institutional Objectives (SDIOs) process
d. Unit / budget planning process
e. Collegewide operating budget development process
f. Budget approval, implementation, and annual operating
results
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
13
a. Mission Statement Process
Mission Statement
Using IE to Build a Culture of Performance Improvement
 IE process begins and ends with the College
Mission statement
 College’s Mission is formally reviewed every five
years by a committee representing faculty,
administrators, students, community leaders, and
members of the Board of Trustees (BOT)
 Mission is also reviewed annually by President’s
Cabinet before the new strategic planning cycle
begins
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
14
a. Mission Statement Process
Assessing the Mission
Using IE to Build a Culture of Performance Improvement
SPC Mission – Assessments - Closing the Loop
Evaluations / Assessments
SPC Mission Components
Accessible Education






BS, BAS, AA, AS Degrees
Technical Certificates
Applied Tech. Diplomas
Distance Education
Developmental Education
Continuing Education
Partnerships / Services



Student Services
Community Services
Other Institutions
1.
2.
3.
4.
5.
1.
2.
3.
4.
6.
7.
8.
9.
10.
11.


AS Degrees & PSAV Programs
WorkNet One-Stop Ops.
December 10, 2007
(Revised 04-27-2011)
1.
2.
3.
4.
5.
5.
Economic & Workforce
Development
Education Oversight Group (EOG) Reports
Academic Program Assessment Reports (APARs)
Academic Program Viability Reviews (APVRs)
Comprehensive Academic Program Reviews (CAPRs)
End-of-year SD-IOs Assessment Report
Recurring Cycle
1.
2.
3.
Entering Students Survey
Enrolled Students Survey
Graduating Students Survey
Employer Survey
Recent Alumni Survey
Budget. Planning, IR Survey
Technology Resources Allocation Survey
Other Department-specific Surveys
Administrative Services Oversight (ASOG) Reports
Student Services Oversight (SSO) Report
Education Support Services (ESS) Reports (Libraries,
Learning Support Centers)
1.
Environmental Scans
Trends Analysis
WorkNet Surveys & Exit Interviews
1.
2007 SACS-COC Annual Meeting
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
2.
3.
Annual
Every 3-years /
program
Annual
Every 3-years /
program
Annual
Annual
Annual
Annual
Annual
Annual
Annual
Annual
Annual
Annual
Annual
Annual
On-going
On-going
Annual
15
b. Evaluation and Assessment Processes
Evaluation and Assessment
Using IE to Build a Culture of Performance Improvement
 Evaluation and assessment processes at SPC are
centered on mission-driven outcomes in the following
program and service areas:
 (i) 26 Organizational units comprising 40 Academic programs
(lower division);
 (ii) 8 Colleges and Schools comprising 20 Majors (upper
division), and
 (iii) key administrative and educational support services.
 Analysis of outcome results is on-going and captured in
various assessment reports.
 SPC evaluates all student services to ensure students
are provided the best support possible to ensure student
success.
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
16
b. Evaluation and Assessment Processes
Assessment Areas
Using IE to Build a Culture of Performance Improvement
 Academic Outcomes
 General Education
 BAS/BS
 AAS/AS
 Key Administrative and Educational Support
Services
 Viability
 Importance
 Satisfaction
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
17
b. Evaluation and Assessment Processes
Assessment Focus
Using IE to Build a Culture of Performance Improvement
 Continually evaluate assessment instruments
and current processes
 Integrate assessment with educational process
 Focus on fewer, yet meaningful improvements (action
items)
 Use multiple assessment methods
E Assessments
G
d
e
 Involve faculty in process
n
 Integrate timing of assessments
Reviews
Surveys
(3-year cycle)
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
18
b. Evaluation and Assessment Processes
Direct and Indirect Measures
Using IE to Build a Culture of Performance Improvement
 Direct Measures





Internally Developed General Education
Externally Developed General Education (e.g., MAPP, iSkills)
AAS/AS Academic Program Assessment Report (APAR)
BAS/BS Program Assessment
Comprehensive Academic Program Review (CAPR)
 Indirect measures






Entering Student Survey
Enrolled Student Survey
Graduating Student Survey
Recent Alumni Survey
Employer Survey
Community College Survey of Student Engagement (CCSSE)
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
19
b. Evaluation and Assessment Processes
Three-year Assessment Cycle
Using IE to Build a Culture of Performance Improvement
APARs
Academic Program
Viability Reviews (APVRs)
conducted yearly
CAPRs
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
20
b. Evaluation and Assessment Processes
Key Assessments
Using IE to Build a Culture of Performance Improvement
Academic Program
Viability Review
(APVR)
Academic Program
Assessment Review (3)
(APAR)
1) Program Description
(1)
1) Introduction w/ use of past
results (1,2)
2) Number of Graduates –
by year
2) Major Learning Outcomes –
program specific (1, 2, 3)
3) Actual Enrollments – by
last six semesters
3) Assessment Methodology (3)
4) SSH Productivity – by
last six semesters
5) Program Profitability –
RPI Index by year
4) Criteria for success (3)
5) Summary of assessment
findings
1) Key APAR findings
2) Key APVR findings
3) Environmental scan of
stakeholders – program
specific (1)
4) Enrollment / Graduation
trends
5) Employment / wages trends
(1)
6) Discussion & analysis
7) Action plan & time table
8) Budgetary & planning
implications
9) Follow-up report
December 10, 2007
(Revised 04-27-2011)
Comprehensive
Academic Program
Review (3) (CAPR)
Notes:
1)
2)
3)
6) Discussion & analysis
7) Action plan & time table
8) Annual follow-up report
9) Cabinet Review & Approval
SACS Compliance Requirement
Florida DOE Recommendations
BOT Rule 6Hx23-3.04 Requirement
Oversight Group
Report
(Multiple Programs)
1) Outcomes Asmt.
Sources (1)
2) Previous Years Plans
& Results (1,2)
3) Current Year Asmt.
Results (1,2)
4) Key Areas Requiring
Improvements (not
meeting criteria /
expectations)
5) Recommended Action
Plans
6) Cabinet Review &
Acknowledgement
21
b. Evaluation and Assessment Processes
Academic Program Assessment
Using IE to Build a Culture of Performance Improvement
 Academic Program Assessment Reports
(APARs)
 Prepared for the Gen Ed, A.S., and A.A.S. programs
according to the three-year cycle
 Include a summary of assessment findings and a
discussion and analysis of those assessment findings
(including use of results, action plan items, and
budgetary implications)
 Process was initiated college-wide in 1999
 Program data is gathered and analyzed on a
continuous basis
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
22
b. Evaluation and Assessment Processes
Academic Program Viability Review
Using IE to Build a Culture of Performance Improvement
 Academic Program
Viability Review (APVR)
 Published yearly
 Included measures:
 Program Graduates
 Course Enrollment
 Student Semester Hour
(SSH) productivity
 Relative Profitability Index
(RPI-T)
 Economic Trend Data
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
23
b. Evaluation and Assessment Processes
Program Review Process
Using IE to Build a Culture of Performance Improvement
 Comprehensive Academic Program Review
(CAPR) is a summative evaluation that includes
multiple measures such as:




program-specific performance
profitability measures
academic outcomes, and
stakeholder perceptions
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
24
b. Evaluation and Assessment Processes
Program Review Objectives
Using IE to Build a Culture of Performance Improvement
 CAPR was designed to meet three objectives:
 To provide a comprehensive report that summaries all elements
of the program’s viability and productivity from a 360-degree
perspective,
 To provide comprehensive and relevant program specific
information to key College stakeholders such as the President
Cabinet members in order to make critical decisions regarding
the continued sustainability of a program, and
 To provide program leadership a vehicle to support and
document actionable change for the purposes of performance
improvement.
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
25
b. Evaluation and Assessment Processes
Ed Outcomes Online
Using IE to Build a Culture of Performance Improvement
 To provide a medium for completing the draft
educational assessment reports as well as establishing a
repository for program specific information, SPC
developed an Educational Assessment Web site
(https://it.spcollege.edu/edoutcomes/)
 College administration and instructional staff are
provided access to “completed” assessment reports
including the APAR (Program & Gen Ed) and the CAPR
 Online access further encourages the use of assessment
data as well as highlighting “best practices” across the
college
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
26
b. Evaluation and Assessment Processes
College Access
Using IE to Build a Culture of Performance Improvement
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
27
b. Evaluation and Assessment Processes
Program Director Access
Using IE to Build a Culture of Performance Improvement
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
28
b. Evaluation and Assessment Processes
Student Survey Process
Using IE to Build a Culture of Performance Improvement
 Data collection
 Increased student awareness of survey process and
accessibility
 Minimized use of class time to complete surveys
 Use multiple points of student contact
 Online survey submission and scanning of paper
forms
 Quality of survey forms
 Professional “look and feel” of survey format
 Items designed to gather pertinent data
 Included online branching models to reduce survey
length
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
29
c. Strategic Directions and Institutional Objectives Process
Strategic Directions
Using IE to Build a Culture of Performance Improvement
 College’s SD-IOs are comprised of longer-term
directional statements and shorter-term,
annually measurable objectives.
 Strategic Directions are:
 Updated on a three-year cycle and are based on
achieving the College’s mission.
 Developed to make improvements in learning
outcomes, quality, productivity, and other
environmental-scan considerations, as well as inputs
from IE Team, President’s Cabinet, faculty and
student representatives, the BOT, and
external/community stakeholders.
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
30
c. Strategic Directions and Institutional Objectives Process
Institutional Objectives
Using IE to Build a Culture of Performance Improvement
 Institutional Objectives are
 Reviewed and updated annually, with key on-going
initiatives updated based on progress made during
the prior year and carried forward, where appropriate.
 Designed for the purpose of achieving the
corresponding longer-term strategic direction.
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
31
c. Strategic Directions and Institutional Objectives Process
SD-IO Alignment
Using IE to Build a Culture of Performance Improvement
Mission – Assessments - Closing the Loop Examples
Key Components
SPC Mission
Accessible
Education
BS, BAS, AA, AS
Degrees
Technical Certificates
Applied Tech.
Diplomas
Distance Education
Developmental
Education
Continuing Education
December 10, 2007
(Revised 04-27-2011)
Evaluation/
Assessments
1.Education
Oversight
Group (EOG) Reports
2.Academic Program
Assessment Reports
(APARs)
3.Academic Program
Viability Reviews
(APVRs)
4.Comprehensive
Academic Program
Reviews (CAPRs)
2005-2006 Strategic Plan
SD & IOs (87 Total; A few
selected below)
A.2: Use educational outcome data
to improve General Education
Outcomes
A.11: Incorporate service learning
into Honors Program courses
I.4: Increase baccalaureate degree
opportunities w/ targeted new
programs
2007 SACS-COC Annual Meeting
Operating / Results
(See full details: 200506 Year-End Report)
A.2: Gen. Ed. outcomes for
all A.S. programs
implemented
A.11: Service learning now
required for Honors students
I.4: Int’l Business, O&P,
Paralegal Studies added
32
d. Unit / Budget Planning Process
Unit / Budget Planning
Using IE to Build a Culture of Performance Improvement
 SPC’s unit/budget planning is centered on:
 more than 150 Unit / Budget Managers,
 development of departmental unit plans and budget
requests, and
 use of the national award-winning Strategic Planning
Online (SPOL) system.
 Unit / Budget Managers are urged and
encouraged to access, review, and make use of
all available assessment outcome data (i.e.,
College’s IE website, oversight group reports,
etc.)
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
33
d. Unit / Budget Planning Process
Unit / Budget Planning
Using IE to Build a Culture of Performance Improvement
 Budget guidelines are prepared by the Collegewide
Budget Committee (CBC), which provides
 Information about the College’s expectations for State
appropriations,
 Collegewide budgetary outlook, and
 Salary package and other key strategic expectations.
 Budget guidelines also provide details on how capital
expenditure requests are to be entered into the SPOL
system
 SPOL system requires the unit / budget managers to
select one of the SD-IOs for each of the unit plan
objectives entered
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
34
d. Unit / Budget Planning Process
Unit / Budget Planning
Using IE to Build a Culture of Performance Improvement
 Supporting tasks, expected outcomes, and related
budget funding are also captured in the SPOL system
 After all unit plans and budget requests are entered, they
undergo management review in which refinements and
adjustments may be made based on funding availability
and overall College priorities
 Approved Unit plans (by College President’s direct
reports) are aggregated and submitted for further review
and evaluation by the CBC
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
35
e. Collegewide Operating Budget Development Process
Collegewide Budget Committee
Using IE to Build a Culture of Performance Improvement
 Each year, the President confirms the membership of a
CBC (SVPs, VPs, provosts, and other key functional
leaders)
 Certain members of the committee are rotated on an
annual basis for cross-training purposes.
 The charge of the CBC includes the following:
 Develop budget guidelines,
 Review of all budget requests approved by department heads,
vice presidents, and senior vice presidents,
 Ensure that the SD-IOs are funded according to their priorities
and funding availability, and
 Develop the College’s Operating Budget.
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
36
f. Budget Approval, Implementation, and Annual Operating Results
Budget Approval
Using IE to Build a Culture of Performance Improvement
 Operating Budget is approved by the President’s
Cabinet and recommended to the BOT
 Once approved by BOT, the budget is
communicated and uploaded into the College’s
central financial system
 College financials and operating results are
monitored on a monthly basis and reported
quarterly to the BOT
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
37
f. Budget Approval, Implementation, and Annual Operating Results
Operating Performance and Results
Using IE to Build a Culture of Performance Improvement
 Operating performance and results are centered on:
 Executing SD-IOs
 Supporting unit plans and quality improvement initiatives
 Adhering to the operating budget
 Operating performance and results are accomplished
through the following key steps:




Implementing budgets and action plans
Monitoring achievements and related circumstances
Evaluating mid-year unit plan reviews
Review end-of-year unit plan reports
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
38
IE Next Steps
Using IE to Build a Culture of Performance Improvement
 Ultimate goal is to provide stakeholders
‘timely’, ‘relevant’, ‘accurate’, and
‘interpretable’ data through:
 Formatted (dashboard) style reports, and
 On-demand customizable reporting, with
 Valid, reliable, and standardized measures.
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
39
Links to Additional IE Materials
Using IE to Build a Culture of Performance Improvement
http://www.spcollege.edu/Central/IE/2007_SACS_Presentation.htm
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
40
Key Points
Using IE to Build a Culture of Performance Improvement
 IE is more than assessment
 A lot has changed since 2007
 IE is less centralized
 New planning system has been implemented
 Assessment activities have expanded
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
41
Next Steps
Using IE to Build a Culture of Performance Improvement
 Need to capture/define the current
processes
 Need to ensure alignment among current
processes
 Need to incorporate the new Student
Achievement Committee Structure
 Finally, need to tell the story…
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
42
Questions
Using IE to Build a Culture of Performance Improvement
December 10, 2007
(Revised 04-27-2011)
2007 SACS-COC Annual Meeting
43
Using Institutional Effectiveness to
Build a Culture of Performance
Improvement
Revised version for
the 2011 SACS
Working Group
Department of Institutional Research and Effectiveness
St. Petersburg College
P.O. Box 13489
St. Petersburg, FL 33733
(727) 341-3059
FAX (727) 341-5411
2007 SACS-COC Annual Meeting