A Culture of Evidence: What Is It? Do We Have One?
by Kay M. McClenney, Byron N. McClenney, and Gregory F. Peterson
Do you really know your students' needs and the reality of their matriculation experiences?
Introduction
Since the inception of the American community college, access has remained the
cornerstone. Enrolling a disproportionate number of first-generation students, low-income
students, and students of color, these institutions have been instrumental in providing
underserved student populations the opportunity to pursue higher education. Community
colleges enroll approximately 42 percent of all undergraduates, including 56 percent of all
Hispanic students and 45 percent of all African-American students (Bailey, Jacobs, and
Leinbach 2003; Bailey, Jacobs, and Leinbach 2005). They also serve more than half of firsttime undergraduates from the lowest two socioeconomic status (SES) quartiles (Bailey,
Jenkins, and Leinbach 2005). Clearly, community colleges have made the opportunity to
attend college a reality for many who otherwise would have been left out of the halls of
higher education.
While many students of color and low-income students enter community college, an
unacceptably low proportion of them persist to degree completion. For example, only 11
percent of African-American and 21 percent of Hispanic students entering community
colleges in 1995 completed a degree within five years, as compared with 28 percent of White
students (Bailey et al. 2004). These degree completion rates for African-American and
Hispanic students were much lower at community colleges than at four-year institutions,
where completion rates were 43 percent and 45 percent respectively. Degree completion
rates for first-time undergraduates in the lowest SES quartiles were also much lower than
overall student completion rates (Bailey, Jenkins, and Leinbach 2005). These statistics are
consistent with the U.S. Department of Education’s (2002) findings that only about one in
10 students who enters community college completes a bachelor’s degree within five years,
and that one is less likely to be a low-income student or a student of color.
As more attention is given to accountability in higher education, community colleges face the
challenge of maintaining their commitment to access while increasing the success of their
students, particularly those from minority and lower-income populations. To accomplish
this, an increasing number of advocates and critics assert that colleges must pursue efforts
aimed at transformational change at the heart of the institution, not boutique programs on
the institutional margins (McClenney 2004). In other words, community colleges must
change how they do business to create environments and systems in which all students
succeed. Assertions that transformational change can improve student completion rates are
based on findings that show similar groups of students perform differently at different
institutions, implying that characteristics and practices of an institution can directly affect
student success (Bailey et al. 2004).
A “Culture of Evidence”
The first step in promoting transformational change requires institutions to understand their
students’ patterns of persistence and degree completion. Only when community colleges
know and use data on their students’ experiences for making decisions about how to serve
those students will they be capable of removing barriers to student success and building
programs and services that help students succeed (MDC 2006). For colleges to know and
use data on their students’ experiences, they must learn to foster an institutional
environment in which planning and decision-making processes at all levels of the institution
are data-driven. Such an environment—one in which “institutional and individual reflection
and action are typically prompted and supported by data about student learning and
institutional performance”—is known as a “culture of evidence” (McClenney and
McClenney 2003, p. 3).
Although the use of data to inform decisions made at higher education institutions has been
asserted in the literature for many years, this process has rarely been directly studied or
systematically integrated into institutional practice (Kuh et al. 2005). A general level of
resistance to accountability and assessment among higher education institutions has also
limited the experience of many community colleges with developing analysis-based planning
and decision-making processes (Cohen and Brawer 2002; Johnston and Kristovich 2000;
Miller and Malandra n.d.). With this limited exposure to a culture of evidence, models are
needed to guide community college leaders in integrating the use of data into an institution’s
cultural norms. This is particularly true in the face of increased requirements for evidentialbased planning and the use of data on academic programs to drive accreditation (Hollowell,
Middaugh, and Sibolski 2006).
The Community College Inventory: Focus on Persistence, Learning,
and Attainment
One such model exists in McClenney and McClenney’s (2003) Community College
Inventory: Focus on Persistence, Learning, and Attainment. The inventory “provides
descriptions of eleven characteristics of colleges that are strongly focused on student
success—that is, student persistence, learning and attainment” (p. 1). One of the sections,
“Part II: The Culture of Evidence,” lists eight characteristics indicative of the presence of a
culture of evidence. While the inventory is not intended to serve as a test or checklist for
institutions developing a culture of evidence, it can be very useful in “prompting institutional
review, reflection, [and] discussion” as a college looks for ways to infuse the analysis and use
of data into its institutional processes (p. 1).
An example of how the Community College Inventory can be used to guide institutions in
fostering a culture of evidence is shown in the efforts of Achieving the Dream: Community
Colleges Count, a national initiative funded by the Lumina Foundation for Education,
KnowledgeWorks Foundation, Nellie Mae Education Foundation, The Houston
Endowment, The Heinz Endowments, and College Spark Washington. Achieving the
Dream encompasses a “national effort to increase the success of community college
students, particularly those groups that have been underserved in higher education” (Lumina
Foundation for Education 2005, p. 1). Currently involving 58 colleges in nine states, the
initiative develops mechanisms to sustain five strands of work: institutional change, policy,
public engagement, new knowledge, and partners’ capacities (Lincoln 2006). An underlying
component of the work is the development of a culture of evidence at each of the
participating institutions (Achieving the Dream 2005). To accomplish this, many of the
colleges have used the Community College Inventory as a tool to inform their efforts to
make data-based planning and decision making a cultural norm (see figure 1).
This internal assessment and review process can be illustrated by looking at each of the eight
items in the inventory that address the indicators of a culture of evidence. The response
categories (0 through 4) allow respondents to estimate the extent to which each
characteristic is evident at their own college. The following discussion amplifies the
description of each characteristic and provides illustrative examples of work undertaken by
community colleges that have substantial efforts underway to build and sustain a culture of
evidence.
Indicator 1: Institutional research and information systems provide systematic, timely, useful,
and user-friendly information about student persistence, learning, and attainment. For an
institution’s planning and decision-making processes to be informed by data on its students’
experiences, these data must be consistently available to constituent groups across campus at
exactly the time they are needed to inform planning for program and service evaluation,
development, and other efforts. Often, though, only a few members of the campus
community are involved in the analysis and use of student data because these data are not
presented in a user-friendly and timely way (Bensimon 2004).
Richland College in Dallas, Texas, a recent winner of the Malcolm Baldrige National Quality
Award, is an example of a college with systematic and user-friendly data processes. To
provide timely, user-friendly information across the institution, Richland College distributes
a monthly report on the progress made on key performance indicators (KPIs). In the report,
the month’s progress score (1 through 10) is listed for each KPI and compared to the
previous month’s scores and the overall score for the current year-to-date. KPIs are also
color-coded to show if the score is within target range, between 89.99 percent and 85
percent of target range, or less than 85 percent of target range. The color tags make the
document user-friendly, allowing readers to quickly assess which KPIs are being met and
which areas need improvement. In this way, Richland College is able to maintain awareness
of institutional improvement efforts across the college.
Indicator 2: The institutional culture promotes willingness of governing board members,
administrators, faculty, staff, and students to rigorously examine and openly discuss
institutional performance regarding student persistence, learning, and attainment. When data
are made available and user-friendly, members of the college community can be engaged in
discussions surrounding student learning and success. Bensimon (2004) posits that
in order to bring about change in an institution, individuals must see, on their own,
and as clearly as possible, the magnitude of inequities (awareness). Then they must
analyze and integrate the meaning of these inequities (interpretation), so that they are
moved to act upon them (action). (p. 46)
For individual board members, administrators, faculty, staff, and students to become
“aware,” which Bensimon suggests is the first step toward change, all constituent groups
must have the opportunity and support to critically analyze institutional data on the student
experience and confront “the brutal facts” of reality (Collins 2001, p. 70). This requires
colleges to create an environment that tolerates “bad news” about institutional performance
and tough-minded reviews of instructional and student services and strategies. For example,



To promote broad-based participation in increasing the success of low-income
students and students of color, Valencia Community College in central Florida held a
college-wide session—a “Big Meeting”—and presented the findings on student
persistence and attainment outcomes. Table groups discussed the findings, proposed
potential Achieving the Dream strategies, and then rated which strategies they were
most interested in pursuing. The meeting was followed up with student focus
groups, multiple large-group work sessions, and a daylong training session for college
“learning leaders.”
The president of El Paso Community College in Texas conducted Achieving the
Dream “town hall meetings” at each of the college’s five campuses and a sixth
meeting with a community advisory group that included the president and provost of
The University of Texas at El Paso, four school superintendents, and community
business leaders. These meetings were in addition to 13 student focus groups and
other college-wide planning and informational meetings.
The Houston Community College System in Texas engaged its faculty and staff in an
All-College Professional Development Day on “Helping HCC Students Achieve
Their Dreams” and created “involvement” committees responsible for seeking out
faculty, staff, and student input.
Indicator 3: The institution is committed to cohort tracking of entering students to
determine rates of attainment and to identify areas for improvement. According to Sharp
(1989), “progress of individuals through their educational programs can best be shown when
various measures of their performance are collected and maintained across successive time
periods” (p. 85–86). Cohort tracking allows institutions to identify patterns or trends that
emerge over time in the experiences of students at the college. Once these patterns have
been identified, the college can then implement strategies targeted at the points where
students struggle the most. Since cohort tracking also captures the progress made by
individual students, colleges can measure the effectiveness of the improvement strategies
they implement as well.
As Achieving the Dream colleges join the initiative, they are required to provide student
cohort data for the current year and from two prior years. Each college then tracks the
incoming-student cohort for each successive year of the initiative. The cohort includes all
degree/certificate-seeking undergraduate students (full- and part-time) entering the
institution for the first time during the fall term, including students enrolled in courses that
are part of a vocational or occupational program, at off-campus centers, and in distance
learning or home study programs. The student remains a member of the cohort even if he or
she transfers to another institution, drops out of the institution, or skips one or more
enrollment periods (JBL Associates 2004). Not only are cohort data gathered at the
institution, but the data also are collected in a national database for analysis and
benchmarking purposes across all participating institutions.
Indicator 4: The institution regularly collects, analyzes, and reports data pertaining to
successful completion and persistence in developmental and college-level courses. Students
who attend community colleges enroll in courses for many different reasons—to transfer to
a four-year institution, earn a certificate, brush up on their job skills, or for personal
enrichment—and this can make it difficult for institutions to measure whether students are
reaching the goals they have set. However, there are indicators that, when regularly analyzed,
can provide key insights into student success. For example, it could be argued that no
student enrolls in a course with the intent to fail; thus, successful completion of courses,
including developmental coursework, can be used to measure student progress. Given that,
one indicator for students taking developmental courses would be their success in qualifying
for and completing college-level classes with a C or better. Also, the persistence and
completion rate of students who indicate they plan to complete a degree or certificate
program are important outcomes to monitor. These data can provide a vivid picture of
where students are in achieving their goals and where their progress is blocked—information
that can inform a college’s improvement efforts. For example,


In analyzing student completion and participation rates in its developmental
education programs, El Paso Community College found that 98 percent of first-time
college students needed at least one developmental course. On average, it took these
students up to four semesters to move into credit courses, and many students did
not persist that long. With this knowledge, the college has begun to design strategies
that reduce the time it takes students to complete developmental coursework and to
align its English as a Second Language program more closely with college-level
courses.
The Alamo Community College District in San Antonio, Texas created research
briefs displaying district and college data on developmental education (specifically
developmental mathematics), gatekeeper courses, persistence and academic success,
and graduation rates. These briefs were used to inform the strategies the district
adopted to increase the success of its low-income students and students of color.
The Alamo Community College District also used data from the research briefs to
encourage individuals and colleges to apply for mini-grants supporting innovation in
addressing student persistence and attainment rates.
Indicator 5: Data depicting student persistence, learning, and attainment are routinely
disaggregated and reported by student characteristics, including gender, race/ethnicity, and
income level. The Achieving the Dream initiative is founded on the belief that “more
students succeed when colleges demonstrate commitment to excellent education for all”
(Achieving the Dream n.d., p. 4). If an institution only knows what it measures, then it is
critical that community colleges measure differences in student persistence and completion
rates associated with gender, race/ethnicity, and income level. Bensimon (2004) contends
that this evidence about the state of equity in educational outcomes for underrepresented
students presented in the form of graphically displayed quantitative data can have a powerful
effect in mobilizing institutional attention and action. (p. 45) When an institution is able to
see the differences in student performance that exist among different groups of students, it
can more readily focus its efforts on students with the greatest need. In doing this, the
college’s intervention strategies will have the greatest likelihood of affecting its students’
success.
After disaggregating student outcomes data, Tallahassee Community College in Florida
identified the lack of academic preparation among African-American students, the lack of
cultural preparation among faculty, and the lack of evaluation and coordination of student
services as the most salient contributors to low student performance rates. In response, the
college made closing the achievement gap between African-American and White students
and moving more students beyond developmental courses into college-level courses leading
to degrees the two main priorities of its institutional change efforts. This has led to the
implementation of a pilot study of first-time-in-college students using Individual Learning
Plans to increase academic readiness, personal development, and social integration at the
college.
Indicator 6: The institution regularly assesses its performance and progress in implementing
educational practices which evidence shows will contribute to higher levels of student
persistence and learning. Many colleges, after identifying gaps in student performance,
conclude their improvement efforts by implementing intervention programs and strategies
based on the “best practices” of other institutions. A culture of evidence expands upon this
process by first verifying the success of acknowledged best practices through reviewing
available outcome data on these programs and strategies and then by requiring ongoing
assessment of all interventions to measure progress made. By continuously monitoring its
improvement efforts, the institution is able to better adapt interventions to meet the unique
needs of its students and community. For example,


To improve the first-year experience of its students, Durham Technical Community
College in North Carolina held its first pre-enrollment orientation program in fall
2005, followed by a second orientation in spring 2006. The college compared
attendance levels and student surveys from both orientations to uncover orientation
components and institutional policies for new students that contributed to higher
persistence and success rates. Findings from the analysis of these orientations will
influence the design of new-student orientations in the future.
Drawing from research on course redesign, Tallahassee Community College
modified the design of its first-year composition course to integrate appropriate
technologies for diagnostic assessment of students and to focus more on learningcentered writing activities. The college then measured the success rates of students in
redesigned and traditional course sections to assess the impact of the changes. With
these data, the college was able to expand the breadth of the redesign by highlighting
effective pedagogical techniques that were transferable across campus.
Indicator 7: The results of student and institutional assessments are used routinely to inform
institutional decisions regarding strategic priorities, resource allocation, faculty and staff
development, and improvements to programs and services for learners.
The Achieving the Dream initiative pushes participating colleges to “make decisions and
allocate resources based on evidence of what is working and what is not” (Achieving the
Dream n.d., p. 4). For the initiative to have a long-term impact, the colleges are expected to
“take successful innovations to scale and integrate them into institutional strategic plans and
budgets to ensure sustainability” (p. 4). This merging of Achieving the Dream efforts and
results into its core work drives the institution to measure all of its programs and services
against key indicators of student success. As this happens, the analysis of data on students’
experiences becomes central to the day-to-day activities of all the institution’s constituents.
For example,



To link the assessment efforts of its Achieving the Dream work to its institutional
mission, Durham Technical Community College blended its Achieving the Dream
strategies with the college’s three-year strategic initiatives and grouped them into
three larger categories: student success, access, and infrastructure enhancement. The
college’s Teaching-Learning Center also began offering professional development
opportunities aligned with these categories.
Tidewater Community College in Virginia and Tallahassee Community College both
embedded their Achieving the Dream strategies into their Quality Enhancement
Plan, a key component of the Southern Association of Colleges and Schools’
accreditation process.
Capital Community College, Housatonic Community College, and Norwalk
Community College, all in Connecticut, have linked student engagement data from
the Community College Survey of Student Engagement (CCSSE) to strategic
planning goals to address gaps in student performance. For example, Capital
Community College has designed a new advising model to improve the first-year
experience of its students, integrating one of its Achieving the Dream strategies with
its strategic planning goals. Following implementation, the effectiveness of the
advising model is being assessed each semester, using CCSSE scores and student
focus groups.
Indicator 8: Beliefs and assertions about “what works” in promoting student learning and
attainment are evidence-based. When community college constituents approach the
experiences of students by asking, “What leads to student success? And how do we know?,”
these institutions will have successfully transitioned from a “culture of anecdote” to a culture
of evidence (McClenney 2004). As community colleges implement processes and programs
built on data analysis and the evidence of outcomes, they are developing an environment in
which measurement is natural and non-intimidating. Gone will be the days when things are
done a certain way because “that’s how they’ve always been done” or because the latest fad
suggested that a certain strategy would work. Instead, the work of these community colleges
will be grounded in their students’ experiences, a foundation that will firmly plant them on
the pathway toward diminishing gaps in student performance and promoting increased
success for all students.
Conclusion
Data on the experiences of students, particularly students of color and low-income students,
must be a critical component of future efforts to increase overall persistence and degree
completion rates at community colleges across the nation. The Community College
Inventory provides a framework for community colleges to begin the discussion of how
analyzing student data can lead to the creation of an institutional environment in which
everyone in the college knows who is succeeding and why. Using this inventory, colleges can
promote transformational change at the institution’s core and avoid programs or projects
limited to the periphery of the college’s work. As these core institutional changes are made,
institutions will be better able to uphold their promise to help all students to succeed.
References
Achieving the Dream. 2005. What is Achieving the Dream? Retrieved December 19, 2006, from the World
Wide Web: www.achievingthedream.org.
———. n.d. Round Three: Investment Grant Request for Proposals. Retrieved December 19, 2006, from the
World
Wide
Web:
www.achievingthedream.org/_images/_index03/Application_for_Round_Three_Colleges_final.pdf.
Bailey, T. R., M. Alfonso, J. C. Calcagno, D. Jenkins, G. S. Kienzl, and T. Leinbach. 2004. Improving Student
Attainment in Community Colleges: Institutional Characteristics and Policies. New York: Community College
Research
Center.
Retrieved
December
19,
2006,
from
the
World
Wide
Web:
ccrc.tc.columbia.edu/ContentByType.asp?t=1&ContentItemTypeID=0&PagePos=2.
Bailey, T. R., D. Jenkins, J. Jacobs, and T. Leinbach. 2003. Community Colleges and the Equity Agenda: What
the Record Shows. Paper presented at the AACC National Conference, Dallas.
Bailey, T. R., D. Jenkins, and T. Leinbach. 2005. Graduation Rates, Student Goals, and Measuring Community
College Effectiveness. CCRC Brief, no. 28, September.
Bensimon, E. M. 2004. The Diversity Scorecard: A Learning Approach to Institutional Change. Change 36 (1):
45–52.
Cohen, A. M., and F. B. Brawer. 2002. The American Community College. 4th ed. San Francisco: Jossey-Bass.
Collins, J. 2001. Good to Great: Why Some Companies Make the Leap and Others Don’t. New York:
HarperCollins.
Hollowell, D., M. F. Middaugh, and E. Sibolski. 2006. Integrating Higher Education Planning and Assessment:
A Practical Guide. Ann Arbor, MI: Society for College and University Planning.
JBL Associates. 2004. Achieving the Dream: Community Colleges Count. Instructions for Data Submissions.
Retrieved
December
19,
2006,
from
the
World
Wide
Web:
www.aacc.nche.edu/Content/NavigationMenu/ResourceCenter/Projects_Partnerships/Current/Achieving_th
e_Dream/Data_Manual.pdf#search='achieving%20the%20dream%20instructions%20for%20data%20submiss
ion'.
Johnston, G. H., and S. A. R. Kristovich. 2000. Community College Alchemists: Turning Data into
Information. New Directions for Community Colleges, no. 109: 63–73.
Kuh, G. D., J. Kinzie, J. H. Schuh, E. J. Whitt, and associates. 2005. Student Success in College: Creating
Conditions that Matter. San Francisco: Jossey-Bass.
Lincoln, C. 2006. Achieving the Dream. Web cast, May 2. Retrieved December 19, 2006, from the World Wide
Web: www.achievingthedream.org/ABOUTATD/ARCHIVEDNEWS/atdwebcast.tp.
Lumina Foundation for Education. 2005. Achieving the Dream to Serve Community College Students. Student
Access and Success News, January: 1–2.
McClenney, K. M. 2004. Keeping America’s Promise: Challenges for Community Colleges. Keeping America’s
Promise, 7–18. A joint publication of the Education Commission of the States and the League for Innovation
in the Community College. Denver: Education Commission of the States.
McClenney, K. M., and B. N. McClenney. 2003. Community College Inventory: Focus on Student Persistence,
Learning and Attainment. Retrieved December 19, 2006, from the World Wide Web:
www.ccsse.org/publications/Community %20College%20Inventory.pdf.
MDC. 2006. Increasing Student Success at Community Colleges: Institutional Change in Achieving the Dream:
Community Colleges Count. Indianapolis: Lumina Foundation for Education.
Miller, C., and G. Malandra. n.d. Accountability/Assessment. A National Dialogue: The Secretary of
Education’s Commission on the Future of Higher Education, Issue Paper. Washington, D.C.: Department of
Education.
Retrieved
December
19,
2006,
from
the
World
Wide
Web:
www.ed.gov/about/bdscomm/list/hiedfuture/reports/miller-malandra.pdf.
Sharp, B. H. 1989. Assessment-Related Information from Institutional Data Systems. In Institutional
Effectiveness and Outcomes Assessment Implementation on Campus: A Practitioner’s Handbook, ed. J. O.
Nichols, 81–93. New York: Agathon.
U.S. Department of Education. 2002. Descriptive Summary of 1995–96 Beginning Postsecondary Students: Six
Years Later. NCES 2003-151. Washington, D.C.: U.S. Department of Education National Center for
Education Statistics.