Chabot College Office of Institutional Research (OIR): 2015-16 Program Review, Year Two. Updated 10/24/14
Chabot College Administrative Services Program Review Report 2015 -­‐2016 Year Two of Program Review Cycle Office of Institutional Research Submitted on October 24, 2014 Contact: Carolyn Arnold Final Forms, 9/29/14
page 1 of 6 Chabot College Office of Institutional Research (OIR): 2015-16 Program Review, Year Two. Updated 10/24/14
Table of Contents __ X__ Year 2 Section A: What Progress Have We Made? Section B: What Changes Do We Suggest? Required Appendices: A: Budget History B: Service Area Outcomes C: Office of Institutional Research 2013-­‐14 Research Agenda D: Office of Institutional Research 2014-­‐15 Research Agenda E: New Initiatives F2: Classified Staffing Requests F5: Supplies and Services Requests F6: Conference/Travel Requests F7: Technology and Other Equipment Requests F8: Facilitie
page 2 of 6 Chabot College Office of Institutional Research (OIR): 2015-16 Program Review, Year Two. Updated 10/24/14
YEAR TWO A. What Progress Have We Made? 1) What were your year one Program Review goals? The OIR year one goals were pulled from an ambitious research agenda that continues from year to
year. The overall goals are to provide research for accreditation, strategic planning and program review,
Basic Skills and Title III projects, learning support services, grants, mandated reporting requirements,
assessment validation, and other major research projects such as nursing and distance education, respond to
adhoc research requests, disseminate research in useable formats, maintain and upgrade the IR database and
technology, develop the IR staff, and serve on state-wide, district, and college committees. The specific
objectives under these goals are in the research agenda (see Appendix C and D).
Although all goals need to be addressed each year, the highlighted goals for 2013-14 were:
1) Reinstate the Research Analyst position in order to accomplish the Research Agenda.
2) Monitor achievement of the Strategic Plan Goal using student educational goal groups,
3) Produce ongoing environmental scans.
4) Continue to provide targeted research for faculty research and strategic planning.
5) Continue measuring our College-wide Learning Goals (CWLG),
6) Provide answers to routine requests,
7) Make the IR research more visible and used by the college community.
8) Obtain our transfer data from the national database,
2) Did you achieve those goals? Specifically describe your progress on the goals you set for Strategic Plan achievement. All of the year one goals were achieved, except for the last one, which was partially achieved.
1) The Research Analyst position was reinstated, and Jeremy Wilson was hired, for which I
am very grateful. His presence has made many of the accomplishments possible.
2) Measuring progress towards the achievement of our strategic plan goal was top priority,
and the Coordinator reported to PRBC on milestones to show we were on track in year 1.
3) J. Wilson completed the draft of the accreditation environmental scan during the summer.
4) We provided targeted research all year, and once Jeremy Wilson started, we were also able
to provide targeted and intricate grant evaluation and grant development research.
5) We measured the College-wide learning goals on the student Accreditation survey, which
contributes to meeting our SLO goals for accreditation.
6) We answered countless routine requests.
7) The Coordinator gave a record 15 presentations at Chabot or the District, which made
research more visible to the college community. She also sent out eight college Emails
highlighting research and reminding the staff about the available IR research on the web.
8) The Coordinator became adept at using the limited statewide database for transfer data, but
did not have the time to learn how to use the national database.
3) What are you most proud of? a. Finding a way to measure progress on a very general Strategic Plan Goal, and using the educational goal groups that I created for this purpose to target students for new initiatives, such as the Title V grant proposal and the FYE pilot. b. Maintaining a high level of research output throughout all the staff transitions. In Fall 2013 I trained 3 new student assistants, who conducted the student survey and did many other valuable research tasks. I started Fall 2013 with one experienced programmer and ended with another. In Spring 2014, two experienced students performed all the assistant work. We started with one programmer, and ended up with a new Research Analyst/programmer. Despite people coming and going, everyone got trained and we did tremendous amounts of work. I am proud of that. page 3 of 6 Chabot College Office of Institutional Research (OIR): 2015-16 Program Review, Year Two. Updated 10/24/14
4) What challenges did you face that may have prevented achieving your goals? a. A staff of students: For most of the academic year, I had three and then two of the most talented students possible and a very experienced and capable programmer, and together we all accomplished an incredible amount of work. However, the time I had to spend training and supervising them cut down on the forward movement I could make on many pressing research tasks. Among many other things, validating our assessment instruments and obtaining national transfer data did not get done. b. Backlog of research. After three years without a Research Analyst, I had a backlog of unfinished or un-­‐started research projects. Although I started catching up when J. Wilson was hired in April, there was so much left undone, such as assessment validation for Chemistry, English, and Math, that I am still catching up. I still have not gotten to the transfer data. The effects of under-­‐staffing are immense! c. Frequent Groupwise crashes. About 20 times a day, Groupwise freezes and/or crashes, and it takes time to reestablish a connection. This slows me down a lot. 5) Cite relevant data in your narrative (e.g., efficiency, persistence, success, external accreditation demands, etc.). These data show how the OIR is meeting its goals. a. The main IRO service area outcomes are research/programming requests filled. The year 2013-­‐14 almost matched 2012-­‐13 in productivity. In 2014-­‐15, we filled 68 (Fall) + 90 (Spring) = 158 research requests. This was above the average of 100 a year from previous years, but was less than the record 200 requests we filled in 2012-­‐13. This reflects both the accomplishments and the turmoil of this year. b. In 2013-­‐14, the Coordinator gave a record 15 research presentations at Chabot or at the District. These presentations were about defining and using the educational goal groups to the Basic Skills Committee, the Student Services Advisory Committee, and Flex Day, an introduction and a progress report on the strategic plan using those groups to PRBC, trends in local HS graduates, assessment and equity, and repeaters to DEMC, programs that work to the Presidential Task Force, Title III grant outcomes to the Basic Skills committee, Title III objectives met to PRBC, bottlenecks to CEMC, English cut scores to the DCC, and a presentation to the Board on student success and equity as seen in the Scorecard. c.
In 2013-­‐14, the Coordinator also gave three presentations at conferences, either on the ed goal groups or on the bottleneck process at Chabot, wrote one article for the RP newsletter, Perspectives, and developed one factsheet for the CLPCCD Chancellor. d. The Faculty/Staff Accreditation survey, which is only conducted every six years, was conducted in Spring 2014 (by the students in the Institutional Research Office!). Several questions about the use of Institutional Research (IR) data showed that more staff feel that IR data is being provided and used for the evaluation of courses and programs than in years past. Eighty-­‐five percent of the staff feel that the IRO provides data for college and program evaluation (up from 76% in 2008) and 51% of staff now use IRO data in the planning and evaluation of their courses or programs, up from 38% in 2008. Overall, in the design, development, and evaluation of programs and services, 62% of staff feel that institutional research results are used, up from 53% in 2008, 45% in 2001, and only 37% in 1995. These results show that the OIR continues to facilitate a culture of inquiry and data-­‐driven decisions at Chabot College. page 4 of 6 Chabot College Office of Institutional Research (OIR): 2015-16 Program Review, Year Two. Updated 10/24/14
B. What Changes Do We Suggest? Limit your narrative to two pages. Be very specific about what you hope to achieve, why, how. 1) Given your experiences over the past year, what changes do you suggest to your course/program improvement plan? Our research agenda is our program improvement plan. While it is updated yearly to reflect current research objectives, the goals do not change, so we do not need to change it. We just need to do more of it! The following new initiatives will allow us to do that. 2) What new initiatives might you begin to support the achievement of our Strategic Plan goal? What are your specific, measurable goals? How will you achieve them? The OIR supports the Strategic Plan goal directly by measuring Chabot’s progress on that goal. The entire OIR research agenda supports the strategic plan goal by providing research for programs to improve student success and help students achieve their educational goal. Major, specific, measurable goals for the OIR, over & above the 2015-­‐16 Research Agenda, are: 1) Obtain state-of-the-art Institutional Research programming software that also provides
dashboards and other data displays for other college users.
a. What: New software needs to be purchased and installed by the beginning of Fall 2015 in order to avoid disrupting the Fall semester of research requests. b. Why: At both Chabot and LPC, we have been programming using the same version of our
current query software, BrioQuery, for over 15 years. It will be obsolete on the next
upgrade of Windows. We need a more modern and versatile tool that allows for our current
complex programming as well as fulfills the new expectations for dashboard data displays.
c. How: Whatever software we obtain, we would share the licenses and cost between
Chabot and Las Positas. With Las Positas OIR, we have explored several options:
1) Argos (The district owns and has given us access to it). We have been testing it but it
is not powerful enough to do what we need to do and it is inefficient to use, so we would be less
productive. It also requires us to completely rewrite our hundreds of existing Brio programs.
2) Oracle (Oracle owns the latest version of BrioQuery). The latest version is called
Oracle Hyperion IR. This is the most expensive option. However, we feel that this is the best
product to meet the colleges' IR needs now and in the future. Many of our colleagues at other
colleges are using it and praise its power and productivity. It would convert all our Brio
programs automatically to the new version, so we would not lose any time or ability to serve
the colleges. Oracle is giving us a substantial discount on an advanced version, called Business
Intelligence, based on our previous Brio license. It usually costs $40,000 and they are offering
it to us for $19,000/with $4,000/yr for support. This would be split between Chabot and LPC.
3) Tableau. We are currently testing this software, but it does not appear to be as
powerful for programming as BrioQuery. It is more like a more flexible version of Excel, and
would require a complete rewrite of all our programs to use it alone. However, it would be a
good stopgap software to own so we could produce some dashboards (see next goal) before we
can replace BrioQuery. It costs much less – For Chabot only: $2,400 and $600/yr for support.
2) Create dashboard reports so college community and public can easily see our progress on achieving our strategic plan goal and other college outcomes
a. What: Dashboard reports are easy-­‐to-­‐understand charts of major college or program indicators that show whether the college or program is meeting its goals. They would be created for the strategic plan goal and for other college outcomes. Eventually, this
would be done for individual programs.
b. Why: In these days of accountability, institutional research needs to be even more visible and accessible to our campus community and the public. page 5 of 6 Chabot College Office of Institutional Research (OIR): 2015-16 Program Review, Year Two. Updated 10/24/14
c. How: These would be created in whatever software we have by next year – Excel,
Hyperion, or Tableau.
3) Obtain and produce transfer data from the national database for Chabot students by
program in an expedited, routine way.
a. Why: This continues from 2014-­‐15, because is necessary to show transfer outcomes. 4) Make the IR research more visible and used by the college community.
a. Why: This continues from 2014-­‐15, because visibility and use can always be increased. 5) Produce and maintain up-to-date environmental scans on labor force trends that inform
divisions and disciplines about which programs to add, maintain, or discontinue.
a. Why: Our disciplines need more detailed forecasts for their program review. 3) Would any of these require collaboration with other disciplines or areas of the college? How will make that collaboration occur? 1) We would collaborate in the selection and purchase of the programming software with Las Positas. Then we can increase efficiency of both offices by sharing programs.
2) If we had the same software, we could also collaborate on developing dashboards at a district level.
4) Over the next 6 years, what are your long-­‐term vision(s) and goals? (Ed.Mas.Plan) Vision: We envision an Institutional Research Office that uses the latest software to provide on-­‐
demand, relevant data and analysis to the campus community. These analyses support data-­‐driven decision-­‐making at all levels of the college, from program-­‐level development and evaluation to institutional-­‐level goals and outcomes. The IR staff is actively involved in the asking of research questions and analysis of data with the faculty, staff, and administrators who request it. We provide a high level of accessibility to research, promote deeper understanding of data, and continue to maintain a high level of satisfaction with our services for all members of our campus community. We assist the college community in having more accessibility to and more participation in research so that they can evaluate and modify their own programs in order to meet the strategic plan goal of the college. Faculty inquiry and data-­‐driven decisions are the norm for both program review and college planning and evaluation activities, as staff are conversant with the use of data. The IRO is very visible to faculty and staff, and the IR office staff members are catalysts and facilitators for a wide variety of discussions of research, including teaching college staff how to think about data. This results in more data-­‐driven decisions and an ongoing culture of inquiry. Goals: 1) Use state-­‐of-­‐the-­‐art software for data extraction, analysis, and visualization. 2) Use state-­‐of-­‐the-­‐art survey software in both paper and online form 3) Provide more detailed narrative reports that provide in-­‐depth analysis of issues or programs of interest, that contribute to important college discussions, outreach, accessibility, and use of services. 4) Increase the time of IR staff in interaction with faculty or staff about their data 5) Provide a way for on-­‐campus users to complete drill-­‐down data requests. 6) Provide more guidance about the use of data in decision-­‐making. 7) Increase the number of research requests and completed requests 8) Provide a service completion survey (ongoing evaluation of our services) page 6 of 6