Assessing the Program Improvement Objective (PIO) Process
Written by Kathleen Schoenecker, PIO Committee Chair
September 29, 2011
Description of the Process
The first step of the Program Improvement Objective (PIO) process involves all departments developing objectives to improve a specific function of the department, as part of the department’s annual program review and assessment of student outcomes and impacts. PIOs are comprised of the following elements:

Objective Statement – a sentence starting with an action verb describing what the PIO will accomplish and why.

Action Plan – a plan outlining the action steps to accomplish the PIO as well as requests for the staff, equipment, technology, facilities, and other resources required to complete the objective.

Assessment Plan – a plan to assess the success and effectiveness of the objective after it is implemented.

Rationale – a clear description of how the PIO relates to the college as a whole in one or more of the following ways: o The achievement of one or more of the college goals and objectives o Impact beyond the individual department, and o Contribution to student learning or student impacts.
This process was developed by the President’s Task Force, a shared governance group in Fall
2008. During the Spring 2009 pilot, each department developed one PIO, which was entered into a computer program developed as a class assignment by a CS faculty member and his students.
Prioritization of the pilot PIOs was completed by the Resource Allocation Team. Following the pilot year, a PIO Resource Allocation Process was developed, which outlined the various groups who evaluate the worthiness of PIOs and consider them for resource allocation. Part of this process included the establishment a shared governance committee to evaluate PIOs, as well as the development of a standardized rubric to assist management and the PIO Committee with the evaluation process. Additionally, the PIO computer program was adapted and included in the
CurricUNET Program Review module.
The PIO Resource Allocation process was piloted during the 2010-11 academic year.
Departments could submit their previous year’s PIO and/or write a new PIO to be formally evaluated for resource allocation. By February 1, all PIOs were to be entered into the
CurricUNET program review module to be considered for evaluation. As it was a pilot year, departments who wrote less than satisfactory PIOs had the opportunity to revise them prior to being formally evaluated.
The first few rounds of formal evaluation consist of sequential reviews by directors, deans, and vice presidents who narrowed down the PIOs written by over 90 departments and divisions to the top 50 PIOs, by a deadline of March 30. The PIO Committee takes the 50 PIOs recommended by senior management and uses the rubric to conduct a deeper analysis of the PIOs and their
1 | Page

worthiness to be considered for resources. Some PIOs with a smaller, department-specific impact are referred to the Mini Grant process, an auxiliary resource allocation process overseen by the Budget Committee, which developed its own rubric and selection criteria.
By March 15, the committee chooses 20 PIOs to be evaluated by Faculty Senate and the
Resource Allocation Team who more closely identify the exact resources needed to implement each PIO. The Budget Committee then examines available funding sources to support the top 20
PIOs. The President makes final recommendations and presents the PIO resource allocation to the Board by June 30.
Involvement and Roles
The PIO Committee is a shared governance committee, chaired by a member of College Council and comprised of 1 manager, 1 SEIU, 1 CSEA, 2 faculty members (one from Chemistry, one from Nursing), 1 confidential employee, 1 student, and the Program Review Coordinator (exofficio). Representatives from each group were fully involved in the committee’s review process; both evaluating and prioritizing the top PIOs, and making recommendations for improving the process in future years. The only exception is the SEIU representative, who became unavailable for meetings and, due to the short time frame for accomplishing the committee’s charge, could not participate in the process.
While all participants to the process performed their duties satisfactorily, it was difficult for duties to be performed in the prescribed time frame. The existing time frame may not be realistic for allowing all the involved parties to have adequate involvement in the process.
Research and Data
One of the recommendations of the 2008 WASC/ACCJC visit was that Ohlone work on
“codifying the links between program review, budget decision-making, facilities planning, and information technology decisions that support student learning outcomes” (Accreditation
Commission Evaluation Report for Ohlone College, March 2008). Out of this imperative, the
President’s Task Force was formed to review the program and services review process, including the ties to strategic planning and budget development. The task force developed a framework for integrating program review within institutional and departmental planning and budget development, incorporating an assessment of student learning outcomes. Part of this framework was a process to utilize data from program review and student learning outcomes assessment to inform college and departmental budget planning processes through Program Improvement
Objectives.
Following the 2009 pilot year, feedback from the President’s Task Force and the Resource
Allocation Team was incorporated into the process, and modifications were made to how PIOs would be evaluated (including the development of a standardized rubric) and what process PIOs would go through before being considered for resources.
Additionally, College Council conducted a study of the relationship between PIOs from the pilot year and college goals and objectives. The results of this study showed that college goals and
2 | Page

objectives were not always reflected in departments’ PIOs and were sometimes more of an afterthought. This has resulted in greater efforts directed to educating departments on the connection between college goals and objectives and the PIO process.
Opportunities for Input and Vetting
The President’s Planning Task Force, a shared governance committee, developed the initial framework of the PIO process. College Council was closely involved in the development of the
PIO process and approved the resource allocation process and timeline, the PIO evaluation rubric, and the PIO Committee charge. Additionally, as a shared governance committee, the PIO
Committee considers the needs of and impact on all representative groups when evaluating PIOs.
Some departments expressed concern that there was insufficient information and support provided to departments regarding the PIO evaluation process and the expectations for PIO content. This was reflected in the fact that many departments were asked to re-write their PIOs to better align with the requested format.
Integration with Other Plans and Processes
The PIO process is a component of the now annual Program Review process and must be entered into the Program Review module within CurricUNET. As a part of each department’s program review, the departments identify specific, measurable objectives to improve their department, which are developed into PIOs.
The Mini Grant process is a facet of the PIO process that the Budget Committee oversees. A
PIO may be re-directed to the Mini Grant process if any of the evaluating entities believe that, while important, the PIO lacks the college-wide impact to be considered for the primary PIO resource allocation process. The Mini Grant process is an adapted version of the PIO process, with a rubric that is similar but was modified by the Budget Committee to reflect the different criteria for Mini Grants established by the Budget Committee and the Ohlone Foundation.
The PIO process is strongly connected to the college’s goals and objectives and is designed to be a pathway for supporting and achieving many of them. Connection to college goals and objectives is part of the rubric for evaluating PIOs. College Council has conducted a study of the pilot year’s PIOs to identify which goals and objectives can be or ought to be achieved through
PIOs. It is anticipated this process will start to play a greater role in supporting college goals and objectives education, as efforts are currently in place to encourage departments to write PIOs that directly address college objectives that are not being addressed by other means.
Product of the Planning Process
As the result of the 2010-11 pilot year, 20 PIOs were recommended by the PIO Committee, which are still being examined by the Resource Allocation Team and senior management to determine what resources are available to support their implementation. It is anticipated that several of the top PIOs can be accomplished as part of Measure G, particularly PIOs from the
Information Technology and Science departments.
3 | Page

Of the PIOs referred to the Mini Grant process, 13 were awarded the full money requested, with one additional PIO being partially funded. These PIOs are in the process of being implemented, and reports will be given to College Council and the Foundation office later this academic year.
In addition to the list of prioritized PIOs, another outcome of last year’s process was the establishment of a general procedure that the PIO Committee used to evaluate PIOs. The stages of this process are as follows:
1.
The PIO Committee chair organizes and formats all PIO data extracted from
CurricUNET into a readable spreadsheet format for all members to review and disseminates this to the committee members.
2.
Committee members individually read, evaluate, and prioritize the PIOs.
3.
The PIO Committee comes together as a group to discuss their individual recommendations and create a unified priority list.
This process allows for the most efficient use of committee members’ time given the short time period in which to complete this analysis. This process was also adopted by the Budget
Committee in their evaluation of PIOs in the Mini Grant process.
Planning for Sustainability
The PIO process will take place annually as a component of the program review process. By building this process into CurricUNET, it is easier for departments to enter their information and the information can be readily available to all entities who are involved in the evaluation process.
The goal of the pilot years of this process has been to develop and refine a process for evaluating
PIOs and allow for a more efficient process in the future. Part of the PIO Committee’s charge for this year was to advise the college on the effectiveness of the current iteration of the PIO process and make recommendations for improvement. Each year of this process has involved assessment which has been incorporated into the subsequent year’s process.
Improving the Process
The first improvement that is already underway is a revision of the rubric used to evaluate and score PIOs. The PIO Committee had a number of recommendations based on their experience using the rubric, which have been compiled by the PIO Committee chair and the Program
Review Coordinator into a new PIO rubric that is currently awaiting College Council approval.
Major modifications include a modified point system to provide greater scoring precision, as well as more specific descriptions and examples of characteristics that are expected of each score classification. The language of the criteria was also revised to better align with the language used elsewhere in college planning documents. The committee had concerns that the rubric may have a bias toward instructional programs and modified its language to be more inclusive of noninstructional departments, whose impact on student learning may be indirect.
4 | Page

There were technical issues with the PIO Extract Utility and the format in which PIOs were presented once extracted from CurricUNET. The extracted information was somewhat poorly formatted and it was difficult for committee members to clearly identify which information was associated with which PIOs. As a result, the committee spent much time formatting the PIO content into an easily readable format. In some cases, the extracted information contained error messages, and it was unclear whether the error resulted from the entering of information by departments into CurricUNET or an error in the extraction process. This impacted several departments whose additional information was unavailable for consideration by the committee.
It may be worthwhile to look at improving the PIO Extract Utility to avoid these problems in the future.
It is also imperative that departments are privy to the rubric’s information as they are developing their PIOs. The fact that college objectives, not just goals, were a key component of the scoring criteria was not conveyed to departments until late in the Fall 2010 semester, and this information could not be incorporated into CurricUNET except through a help box, which most departments did not look at. Several departments expressed concern about this, because many
PIOs were penalized for their omission of college objectives. In addition, there were significant variations in the level of detail provided in PIO action plans, indicating a need for more guidance to departments about the expectations for PIOs.
In response to concerns from departments, more guidance is being provided to those who are in the process of writing PIOs, so that departments can better align their PIOs with college goals and objectives and ensure that all pertinent information is included in the PIO. Members of the
PIO committee gave a Flex Week workshop on creating more effective PIOs; this workshop will be repeated later on in the Fall semester.
5 | Page