DAVID F. TRUJILLO
Program Development, Proposal-Writing and Evaluation Consulting
SITE VISIT MEMORANDUM
TO:
Richard K. Hogrefe
Dean, Arts & Sciences/STEM Project Director
Crafton Hills College
11711 Sand Canyon Road
Yucaipa, CA 92399
FROM:
David F. Trujillo
David F. Trujillo & Associates
P.O. Box 1192
Taos, NM 87571
RE:
HSI-STEM Project External Evaluation – Year 4
DATE:
January 4, 2016
Overview:
This memorandum represents a "snapshot" assessment of Crafton Hills College’s
HSI-STEM Pathways Project, based on discussions held and observations made
during my recent (December 1, 2015) site visit to the Project. This site visit, my
second to the campus, took place in the first months of the Project's fifth and final
year. Timing is, of course, a critical dimension of this assessment. The Project has
only nine additional months remaining in the original funding period, and the
anticipated deadline for the next HSI-STEM (Title III Part F) grant is only four
months away (early May).
As such, the relevant questions that guided the site visit discussions revolved
around: the Project's accomplishments during Year 4; any challenges or barriers
faced by the Project; either during Year 4 or going forward; the relative
sustainability of the Project's functions/initiatives; the degree to which the Project
has met or will meet its explicit objectives; the impact that the Project has had on
the institution's ability to strengthen the "STEM pipeline"; and, how the Project has
prepared the College to compete in the 2016 HSI-STEM competition.
Appropriately, then, the site visit involved several open-ended, informal
conversations with the core Project team - Project Director and Dean Richard
Hogrefe; Ginger Sutphin, Administrative Secretary; Patricia Menchaca, STEM
Pathways Coordinator; Ernesto Rivera, STEM Transfer Services Coordinator; and
Benjamin Gamboa, Research Analyst. These conversations engendered the following
P.O. Box 1192 Taos NM 87571 505.795.6645
1
DAVID F. TRUJILLO
Program Development, Proposal-Writing and Evaluation Consulting
observations, findings and recommendations. The points offered here are not
meant to be all-inclusive, nor are they presented by importance or priority. They
are meant to supplement what was mentioned on site as well as to provide an
objective external perspective on the Project’s value to the College.
Observations and Findings:
The days' first discussion was a breakfast meeting that provided a general overview
of the Project's implementation and operation. This conversation set a baseline for
the day (i.e., agenda and expectations) and also reinforced my sense that the Project
is well managed by an experienced, committed and skilled team of professionals. As
is usually the case with effective institutional development projects (both Title III
and Title V), the makeup of the Project's core team has been stable and their efforts
consistent.
The site visit's second conversation was with Mr. Gamboa, the Project's Research
Analyst, who has been with the College for approximately two and a half years.
Essentially, the conversation provided a look at the Project and its impact through
the lens of outcomes data and institutional effectiveness. The conversation
illustrated the Project's and the College's use of continuous quality improvement
principles - that is, constantly gathering and analyzing data about key aspects of the
Project in order to measure efficiency and effectiveness, and thereby to improve the
implementation process. The College clearly perceives the value of this function, as
it will have fully absorbed the cost of this position by the completion of the grant
period.
The Year 4 research focus moved away from last year's emphasis on studying the
impact of Supplemental Instruction (SI) and Learning Communities efforts. That
analysis, as noted in my earlier site visit memo, established a solid understanding of
the impact and efficacy of those programs by the end of Year 3. Generally speaking,
both programs had good outcomes, both quantifiably and qualitatively,
demonstrating that students provided these supports were able to move toward
completing their degree programs and prepare for transfer to a four-year institution
in a STEM field. The analysis of student input and data on student progress led to
improvements to the implementation of both SI and Learning Communities. A key
finding was that students attending at least two SI sessions were statistically more
likely to succeed in the course. The successes of the SI program will likely lead to its
institutionalization by the College - at some level - at the conclusion of the current
grant.
The Learning Communities program, with only 8-10 sections per semester, has
proven too small to effectively track or demonstrate impact. In the past year, the
Project team found that the learning communities were not being properly linked to
the curriculum, thus making them not as effective as anticipated. Even though the
implementation of the Learning Communities was refocused in 2014-2015 in
P.O. Box 1192 Taos NM 87571 505.795.6645
2
DAVID F. TRUJILLO
Program Development, Proposal-Writing and Evaluation Consulting
response, the outcomes data is still lacking. As a result, and factoring in the cost of
the innovation, the administration is not convinced about their effectiveness and
therefore their affordability.
An overall challenge from the data analysis/research perspective is that the College
in Year 2 lost the ability to track students longitudinally as they move through the
pipeline. The College's STEM graduates/transfers predominately transfer to CSU
San Bernardino, and 80% of student information is not tracked through that
transfer process. The University does not track majors and the data from the
National Student Clearinghouse is incomplete. In responses, a survey was created
this year designed to elicit from graduated upon exit to self-report their intentions
to transfer, their acceptance to an institution, and their choice of major. While
limited, this survey should provide some basis for assessing the long-term impact of
Project services. What is crucial, however, is that the College's outcome measures
have shown improvement even though clear, specific cause-and-effect connections
can't be made; in 2015, 60 Hispanic students transferred to four-year schools in
STEM majors, which exceeded the Project objectives for Years 4 and 5.
(Anecdotally, as reported by students, a significant factor in this increase has been
the work of the Project's STEM Transfer Services Coordinator, Mr. Rivera. This
points up the crucial role that advising and counseling plays in STEM student
success, persistence and transfer, especially for low-income/first generation
students, and has implications for the design of the next HSI-STEM application.)
In response to CHC's plans to expand in the next few years, Mr. Gamboa has created
a visual "dashboard" to track marketing data and its effects on applications to the
College. Information gathered about what works with different segments of the
service area populations will be used to guide disaggregated campaigns. For
instance, among the lessons learned is that alumni outreach efforts resulted in
applications that were fairly evenly split between Hispanics and White/nonHispanics, while high school visit result in applications that are 60% Hispanic and
36% White/non-Hispanic applications, when the demographics of the area high
schools are only 52% Hispanic. All aspects of the College's outreach target lowincome students, unlike traditional marketing campaigns. Specifically, this effort
tracks Ethnicity, Age, Gender and STEM interests in order to disaggregate the
different student demographics and monitor the effectiveness of marketing in
regards to HSI-STEM metrics.
Overall, as the Project moves toward its completion this fall, and prepares to file its
last annual performance report (as noted on site, the Final Performance Report will
be compiled by the Department of Education from the Project's five APRs), I see no
significant challenges in accomplishing the Project's funded objectives. There are
certainly no issues or concerns with data compilation and analysis. The
institutionalization of the Project data gathering and data analysis functions speaks
P.O. Box 1192 Taos NM 87571 505.795.6645
3
DAVID F. TRUJILLO
Program Development, Proposal-Writing and Evaluation Consulting
volumes about the College's commitment to the Project and to the use of
institutional research/institutional effectiveness to improve outcomes for students.
As Mr. Gamboa points out, “the Grant has been phenomenal in creating a clear
pathway for students that are here to successfully complete their program and
transfer on. Our numbers clearly support this scenario.” In response to a question
about what lessons the current project has for the next HSI-STEM application, he
commented on the need for better outreach to local school districts and better
working relationships with CHC's feeder schools. While there have been marked
improvements and College/K-12 relationships are improving, outreach to the
community is a generational and diffuse process that should continue to be a strand
in the College's larger HSI-STEM initiatives.
The discussion with the STEM Transfer Coordinator, Mr. Rivera, underscored the
importance of personal attention and commitment as a central element in STEM
advising and counseling services. Again, the College sees the value of his efforts and
has committed to absorbing him into the Counseling unit. The impact of these
efforts has been demonstrated in terms of student retention in STEM courses and
significant differences in the success rates of students who have been assisted and
those who haven't. Further, Mr. Rivera has created a spreadsheet of course
sequences/scheduling in STEM programs that supports the use of individual student
education plans; this could (and should) lead to changes in how the College
schedules STEM course sequences.
In terms of what services matter in improving students success and transfer ("best
practices"), Mr. Rivera emphasized the importance of focusing on the student's goals
and needs, taking a holistic view of the student, and proceeding with both empathy
and sympathy for that student. With the high numbers of underprepared, first
generation/low-income students entering STEM fields of study at CHC, this holistic
approach to advising/counseling will be even more important in the future. Looking
to the next HSI-STEM application, there are clear implications in Mr. Rivera's efforts
for implementing some iteration of "intrusive" or "inescapable" advising; i.e.,
moving toward establishing non-voluntary, mandatory strategies.
The discussion with the Project's Pathways Coordinator (Ms. Menchaca) also
highlighted successful efforts that can be expanded and/or revised for inclusion in
the next HSI-STEM application. For instance, the 80-student cohort in Pathways is
currently capped by limitations on human and other resources. It has been
successful in providing students with research experiences and internships with
NASA, the Rocky Mountain Biological Laboratory (RMBL) and UC Riverside. As the
College prepares the next HSI-STEM grant application, it makes sense to consider
mechanisms for engaging some of the College's more successful STEM students to
provide online support to other STEM students, and to organize the Pathways
students as more of a cohort, where possible scheduling them into the same courses
in a learning communities-type strategy.
P.O. Box 1192 Taos NM 87571 505.795.6645
4
DAVID F. TRUJILLO
Program Development, Proposal-Writing and Evaluation Consulting
Ms. Menchaca provided an overview of the Project's outreach efforts, which
comprise several effective programs and initiatives: SciFri, the Explorer Camp, the
high school programs in the spring semester (which focus on seniors). She also
noted the benefits provided by the relationship with RMBL and the provision of
faculty workshops for encouraging literacy in science, particularly by addressing the
teaching of reading strategies. STEM faculty members have proven receptive to
these workshops. With this in mind, considering that professional development is a
critical and central part of an effective HSI-STEM initiative, and given that the
College has less than 10 fulltime STEM faculty, it makes sense for the College to look
at incorporating into the next application a robust program of professional
development that engages full-time and part-time faculty in activities designed to
improve pedagogy and student learning outcomes.
The culminating meeting with Dr. Hogrefe and Ms. Sutphen covered both the
current project and the anticipated 2016-2021 project. As noted at the time, it is
very possible that fully expending the current grant budget will not be an issue this
year, since the HSI Division director, Ms. Ceja, is seeking approval for allowing
current grantees to implement a no-cost extension in 2016-2017 while also
receiving a new HSI-STEM award. Another item discussed was whether the College
should pursue either an individual grant or a cooperative arrangement grant. Again,
the College can theoretically apply for and receive both awards; in terms of focus
and quality, my advice is to prioritize an individual over the cooperative. In my
experience with both Title III Part F and Title V, individual applications are easier to
write, easier for readers to understand and easier to administer than cooperatives.
Additionally, an individual award provides up to $800,000 per year to the grantee,
while the cooperative requires sharing $1,000,000 per year with one or more
partners.
In addition to the other potential proposal elements/initiatives noted in this memo
and in our onsite discussions, I would recommend the inclusion of a comprehensive
STEM Success Center that would serve as a "one-stop shop" for STEM students.
Summary/Next Steps:
The beneficial impacts of the current Project over the last four years are apparent in
the quantitative outcomes noted above. The qualitative impacts on the institution
and its community are probably more significant. As a cumulative result of the
outreach activities, and provision of support services to STEM students and the
investments made in STEM program capacity, there has been a "cultural" shift in the
broad awareness of science fields and careers. The College's efforts to sustain
critical elements of the Project speak to that awareness.
As the Project moves through Year 5, the focus should naturally be on bringing
current effort to completion, on transitioning Project staff and resources onto
institutional and other funds, and on preparing an application for the 2016 HSI-
P.O. Box 1192 Taos NM 87571 505.795.6645
5
DAVID F. TRUJILLO
Program Development, Proposal-Writing and Evaluation Consulting
STEM & Articulation competition, which represents another highly significant
opportunity for further developing the College’s STEM program capabilities and for
infusing resources and innovation into the College's evolution as a HSI.
Finally, I suggest that we schedule a final/Year 5 external evaluation site visit in the
late summer/early fall of 2016. While no institution is guaranteed to receive an
award under the next Title III Part F competition, the College is extremely well
positioned (via experience and talent) to submit a competitive application. I
anticipate that the Project team will be implementing start-up processes for a new
award by that time. For that final visit, the focus will be largely summative,
naturally, with an emphasis on closeout, impact, outcomes, transitions and
sustainability.
P.O. Box 1192 Taos NM 87571 505.795.6645
6