DAVID F. TRUJILLO
Program Development, Proposal-Writing and Evaluation Consulting
SITE VISIT MEMORANDUM TO: Richard K. Hogrefe Dean, Arts & Sciences/STEM Project Director Crafton Hills College 11711 Sand Canyon Road Yucaipa, CA 92399 FROM: David F. Trujillo David F. Trujillo & Associates P.O. Box 1192 Taos, NM 87571 RE: HSI‐STEM Project External Evaluation – Year 3 DATE: October 12, 2014 Overview: This memorandum follows my initial site visit to the Crafton Hills College’s Title III Part F (HSI‐STEM & Articulation) STEM Pathways Project, and is meant to provide a “point in time” assessment of the Project. The site visit took place on September 15, 2014 and was my first site visit, even though the grant has completed nearly 36 months of its 60‐month funding period. The timing of the site visit is an important consideration, naturally. At this point in the grant’s “life cycle,” the appropriate emphasis of the assessment is not just on accomplishments to date, measuring achievements against the Project’s funded objectives and assessing the status of the expenditure pattern, but on the Project’s more “environmental” aspects – implementation challenges, potential fore sustainability and institutionalization, and plans for Year 4. The focus shifts from formative toward summative evaluation, and it is also the appropriate time to begin thinking about the next HSI STEM grant. Consistent with these efforts and their intended outcomes, the site visit entailed open‐
ended, informal but focused discussions with the Project team ‐ Project Director Dean Richard Hogrefe; Ginger Sutphin, Administrative Secretary; Patricia Menchaca, STEM Pathways Coordinator; Ernesto Rivera, STEM Transfer Services Coordinator; Benjamin Gamboa, Research Analyst; Daniel O’Hare, Alternative Learning Strategies Coordinator; and also with Keith Wurtz, Dean of Institutional Effectiveness, Research & Planning; and President Cheryl Marshall. These conversations provide a basis for the following observations, findings and P.O. Box 1192 Taos NM 87571 505.795.6645
1
DAVID F. TRUJILLO
Program Development, Proposal-Writing and Evaluation Consulting
recommendations. The points offered here are not meant to be all‐inclusive, nor are they presented by importance or priority. They are meant to supplement what was mentioned on site as well as to provide an objective external perspective on the Project’s value to the College. Observations and Findings: The eventual success of any institutional development grant project depends on several factors, or “ingredients” being in place; i.e., having a clear plan of action (as laid out in the approved application) that addresses a significant challenge or issue affecting the institution’s effectiveness; having the necessary financial resources – both in the grant and from the institution – to carry out that plan; having a qualified, skilled and committed staff; and, critically, having the support of the administration. Certainly, the application itself addresses a set of issues/dynamics of importance to the institution. The major purpose of the Project is to increase STEM degrees and transfer rates, which is significant both for external (e.g., fiscal stability) and internal reasons, and which address directly the potential for Crafton Hills College (CHC) to be a better, stronger HSI – and particularly as a producer of Hispanic STEM graduates and transfer students. The comprehensiveness of the Project’s approach to the issue is a systemic one, with strategies ranging from outreach initiatives to curricular improvements (new courses, articulation with California State University campuses), to Student Success initiatives (STEM Trek, STEM Academy, learning communities and Supplemental Instruction), to key investments in science laboratory equipment and instrumentation, and to professional development activities. The most valuable asset of the Project is its skilled, talented and committed team. Project leadership has put in place, across the board, individuals who not only have the ability to carry out discrete activities but who also have the personal motivation to make the Project as a whole meaningful, successful and sustainable. The members of the team provided overviews of their activities in cogent, professional and understandable ways. The Project has at its disposal sufficient funding from the U.S. Department of Education to carry out its approved activities, but I am also impressed by the College’s commitment of human resources to the Project. It is obvious that the Project team has built and/or built upon positive relationships with numerous offices and departments that support Project activities and thereby deepen the impact of those activities. The Project also enjoys the support of the President, who sees the initiative as vital, the activities (for the most part) as relevant and the outcomes as positive for the continued evolution of the campus. (Interestingly, the one “criticism” from Dr. Marshall is that the Project should publicize itself more, for both campus and District audiences, even develop a “glossy marketing piece.”) The discussions about gathering, analyzing and communicating data provided insights about various challenges faced by the Project (such as the difficulty in tracking STEM majors after they transfer) and about the ability of the Project team to gather and utilize data. The limitation in getting valid transfer data from CSU is somewhat troubling, since that is a key measure of Project effectiveness, but it should be kept in perspective. First, it is beyond the control of the Project or the College; second, it creates no real problem with reporting to the Department of Education. As noted on site, the purpose of the application P.O. Box 1192 Taos NM 87571 505.795.6645
2
DAVID F. TRUJILLO
Program Development, Proposal-Writing and Evaluation Consulting
objectives is to focus the efforts of the Project, as well as to provide performance targets that are meaningful to the College. The Department of Education has very little real concern with the specific metrics identified by grantees, and typically only cares that efforts are being made to conduct activities that will create broad impact upon institutional capacity, efficiency and effectiveness. Again, data are best used for internal purposes – to drive change efforts, to validate the Project’s approach, and to demonstrate for the purposes of institutionalization that the grant‐funded activities make a real difference. Generally speaking, as evidenced in the Institutional Effectiveness office’s Research Briefs, the College demonstrates the value of the HSI‐STEM grant at CHC. The data illuminates student satisfaction with STEM activities (Supplemental Instruction, Learning Communities), and the IERP can paint a compelling picture of the impact the grant is having on course completion rates and persistence rates of participating STEM students. The data sets are disaggregated by ethnicity, and I believe it is significant that the success of Hispanic and non‐Hispanic students in STEM courses is roughly comparable. While this parity could be validated by a study that isolates the progress of students who started in 095 (since the strength of most feeder schools may be accounting for the Hispanic/non‐Hispanic results), it is still noteworthy that the data demonstrate a high persistence rate and slow but steady improvement in several metrics. As pointed out by the Project Director, however, it is probably not all that important to measure the Project’s impact to an exact degree or with the rigor of a research study. Rather, what is more significant is the growing enthusiasm for STEM programs, building a “culture” of supporting STEM programs, and the role the Project is playing in transforming the College’s mission and reputation in the community beyond CTE programs. The Project’s Outreach activities are comprehensive, wide‐ranging, opportunistic, creative and well managed. Such activities as the Sci‐Fri program and the STEM Explorer Camp, as well as the numerous “little outreach” activities being conducted, obviously create a positive presence for Crafton Hills among secondary students and their families, and strengthen ties with the public school district. The Project’s ability to track participating students is highly important, obviously. While these activities are pointedly not recruitment, it will nevertheless be of benefit to the College in the long run to determine which of the outreach participants subsequently enrolled at the College, and which activities produce the best results. As the College attempts over the next two years to absorb the cost of these activities, such data would be useful. STEM students at the College are clearly benefited by such programs as the STEM Academy and STEM Trek, by visits to various universities (UCLA, UC Riverside, Cal Poly‐Pomona), by activities at ESRI, the Jet Propulsion Lab and the Rocky Mountain Biological Laboratory, by participation in a wind turbine competition, and by a variety of other programs that are coordinated by Project staff. The Project’s direct academic and advising services are also effective and well received. While the facilities available for STEM advising are less than adequate, the Project is partnering with the Transfer Center to establish appropriate office space. Finally, the tour of STEM labs and classrooms demonstrated that the Title III Part F funds that have been invested in the “instructional infrastructure” have made significant improvements in the ability of STEM faculty to provide high quality instruction using P.O. Box 1192 Taos NM 87571 505.795.6645
3
DAVID F. TRUJILLO
Program Development, Proposal-Writing and Evaluation Consulting
current materials and content. The equipment and supplies purchased through the grant have certainly given STEM faculty more tools and have just as certainly enhanced the STEM educational experience for students. Summary/Next Steps: As the Project enters Year 4, it is important to note that the Project has enough time and resources remaining to make adjustments to its strategies and activities. The Title III Part F regulations, being the same as Title V, provide substantial opportunity to shift activities, methods, staffing and expenditure patterns as long as these efforts continue to address the funded objectives. The idea of conducting “pilot programs,” for instance, is a reasonable one. Unlike many HSI‐STEM projects, the grant doesn’t have a great deal of carryover – the Project carried over $171,000 from Year 2 to Year 3 but reduced that figure to only $80,000 from Year 3 to Year 4. Still, that amount still represents an opportunity for the Project team to make improvements in approach or methods. As noted on site, the carryover situation is primarily significant in light of the need to close out the Project by September 30, 2016, so that the College is able to secure another HSI‐STEM grant in the 2016 competition. As any institutional development project enters its fourth year, the grantee should be focusing on the sustainability of those grant‐funded activities that are demonstrating impact, and on the institutionalization of Project staff. This aspect of the Project is highly positive, with the key programmatic positions (the STEM Pathways Coordinator, the STEM Transfer Services Coordinator, and the Research Analyst) slated to be institutionalized and/or partially absorbed onto the College budget. This not only demonstrates the College’s support for the Project but also provides assurance that crucial Project activities will continue. In this context, I found President Marshall’s perspective on the Project to be particularly thoughtful, and useful at gaining a “big picture” perspective. Though her appraisal of the Project and its impact was positive, as noted above, she did question whether the Learning Communities initiative will prove to be worth the investment of institutional funds that will be needed to sustain it once the grant expires. Questions about the economies of scale for such an effort are certainly fair, and point out that the Project’s data collection and analysis needs to address the “return on investment” dimension. Similarly, we discussed the Supplemental Instruction program in light of actual impact. While the data at this point indicates that SI does in fact increase passing rates and persistence among STEM students, the overt purpose of SI (to improve passing rates within targeted courses) raises a question about whether this approach actually adds value when considering the needs of “marginal” students, who tend not to volunteer. As with the Learning Communities issue, a finer grained look at the data is called for. I would recommend that the institution begin preparing to apply for the 2016 HSI‐STEM & Articulation competition, which will probably be in the spring of 2016 and provide an significant opportunity to further develop the College’s STEM program capabilities. Naturally, the theme, emphases and major activities of that grant should build upon the capacities created and the knowledge gained during the current grant, and should address strengths, challenges and opportunities that are central to achieving the College’s vision and mission. P.O. Box 1192 Taos NM 87571 505.795.6645
4