The Research & Planning Group for California Community Colleges Curricular Redesign and
Gatekeeper Completion:
A Multi-College Evaluation of the
California Acceleration Project
April 2014
Authors:
Craig Hayward, Ph.D.
Terrence Willett, M.S.
Senior Researchers, RP Group
Prepared for:
Deborah Harrington
Executive Director, 3CSN
Katie Hern, Director
California Acceleration Project
This study was jointly funded by California Community College Success Network (3CSN) and
a private grant from the Walter S. Johnson Foundation.
Table of Contents
Executive Summary ...................................................................................................................................... 2 Acknowledgements ...................................................................................................................................... 4 Introduction ................................................................................................................................................. 5 Methods ..................................................................................................................................................... 11 Data sources ........................................................................................................................................... 13 Comparison groups ................................................................................................................................ 16 Descriptive statistics: Students characteristics ...................................................................................... 17 Descriptive statistics: Implementation survey ....................................................................................... 21 Description of Logistic Regression and Multivariate Model ................................................................... 26 Results ........................................................................................................................................................ 27 Overall English and math multivariate models ...................................................................................... 27 English model ..................................................................................................................................... 27 Math model ........................................................................................................................................ 29 Marginal means ...................................................................................................................................... 30 College-­‐ and pathway-­‐specific acceleration effects ............................................................................... 35 Acceleration and ethnicity ...................................................................................................................... 41 Restrictions ............................................................................................................................................. 43 Discussion .................................................................................................................................................. 45 References .................................................................................................................................................. 50 Appendix A. Definition of variables included in the logistic regression models. ........................................ 53 Appendix B. Change in pseudo R2 between covariates-­‐only logistic regression model and model including the acceleration independent variable. ..................................................................................................... 55 Appendix C. Average values used in regression marginal means. .............................................................. 56 Appendix D. English logistic regression marginal means. .......................................................................... 57 Appendix E. Math logistic regression marginal means. .............................................................................. 58 Appendix F. Qualitative analysis of the relationship between design principles and college-­‐level acceleration effects. ................................................................................................................................... 59 Appendix G. English multivariate model for high-­‐acceleration colleges only. ........................................... 61 Appendix H. English multivariate logistic regression on sequence completion with comparison group restricted to those with sufficient primary terms to complete sequence by spring 2013. ........................ 62 Appendix I. Math multivariate logistic regression on sequence completion with comparison group restricted to those with sufficient primary terms to complete sequence by spring 2013. ........................ 63 Appendix J. English multivariate model of completion of the gatekeeper pre-­‐requisite (CB 21=A). ......... 64 Appendix K. Math multivariate model of completion of the gatekeeper pre-­‐requisite (CB 21=A). ........... 65 1 California Acceleration Project Evaluation Report
Executive Summary
The problem – Large numbers of students are being placed into long remedial or basic skills sequences from which few emerge. Across the California Community College system, only 19% of students beginning at three levels below transfer-­‐level successfully complete transferable English within three years. The comparable number for the math sequence is only 7%. The intervention – This study examines student outcomes from 16 colleges offering redesigned English and math pathways in 2011-­‐12 during their first year of implementation as part of the California Acceleration Project (CAP), an initiative of the California Community Colleges’ Success Network (3CSN). While there was variation in the specific models implemented, all participating colleges reduced students’ time in remediation by at least a semester; made no changes to the transferable college-­‐level course (only remediation was redesigned); and aligned remediation with the college-­‐level requirements of students’ intended pathways. Most also employed a set of CAP instructional design principles for creating “high-­‐challenge, high-­‐support classrooms.” Implementation mattered – For the 2,489 students in an accelerated pathway, the overall effect of curricular redesign was robust and significant. Although, CAP colleges shared many features, there was also considerable variation in the specifics of how the 16 participating colleges implemented acceleration, particularly among the English pathways. English pathways that articulated directly with the transfer-­‐level gatekeeper course tended to show large increases in sequence completion. Pathways with additional requirements such as extra courses and/or strong institutional filtering processes such as tests, waivers, or challenge applications tended to show little or no acceleration effect. Acceleration effects were large and robust – After controlling for an array of potentially confounding demographic and academic variables, students’ odds of completing a transferable college-­‐level course were 1.5 times greater in accelerated English models overall and 2.3 times greater in high-­‐acceleration models. Students’ odds of completing a transferable math course were 4.5 times greater in accelerated pathways than for students in traditional remediation. Students’ progress was followed through spring 2013, at that time the estimated math sequence completion rate for students in accelerated pathways was 38%, while the completion rate for the comparison group in the traditional sequence was 12%. For students in accelerated English pathways, the overall estimated English sequence completion rate was 30%, while the estimated English sequence completion for students in the comparison group was 22%, after controlling for relevant demographic and academic variables. For students in the high-­‐
2 acceleration English pathways, the estimated English sequence completion rate was 38%, relative to 20% for students in the traditional sequence. Acceleration worked for students at all placement levels – Accelerated pathways increased the odds of completing transfer-­‐level gatekeeper courses for students placed at all levels of the basic skills sequence in math and in English, relative to comparably-­‐placed students in the traditional sequence. Acceleration worked for students of all backgrounds – Students of all ethnic backgrounds benefited from effective acceleration pathways. For example, after adjusting for control variables, Hispanic students’ estimated completion of the English gatekeeper course was 33%, versus 26% for Hispanic students in the comparison group. The difference was even greater in the math sequence where the estimated gatekeeper completion rate for Hispanics was 40% versus 15% for Hispanic students in the comparison group. 3 Acknowledgements
The authors of this report would like to acknowledge the assistance received from the California Community College Chancellor’s Office, in particular, System Software Analyst, Vinod Verma’s excellent work supporting the special collection of assessment and placement data from the 16 participating colleges, as well as support and advice from Myrna Huffman, Director of MIS Services, and Vice Chancellor of Technology Research and Information Services, Patrick Perry. We would also like to thank Dr. Darla Cooper for her valuable review and suggestions. We are deeply grateful to the numerous campus faculty, matriculation office personnel, researchers and information technology staff who supplied data and information about their accelerated pathways, placement policies and institutional practices. We would especially like to thank Katie Hern and Myra Snell for their providing us with background and context for the California Acceleration Project. It was our pleasure to work with such passionate faculty leaders. Their insight, humor and support were welcome as we worked through the myriad challenges associated with a complex, multi-­‐site project such as this. Finally, we would like to acknowledge our families who missed us for many hours over many evenings and weekends during the two-­‐year course of this project. Without their support and understanding, we would not have been able to accomplish nearly so much, nearly so well. Suggested citation (authors in alphabetical order): Hayward, C. & Willett, T. (2014). C urricular R edesign a nd G atekeeper C ompletion: A M ulti-­‐
College E valuation o f t he C alifornia A cceleration P roject. Berkeley, CA: The Research and Planning Group for California Community Colleges. 4 Introduction
For community college students, earning a solid grade in transfer-­‐level statistics or college composition is a major milestone. These achievements demonstrate that a student has the ability to succeed in challenging collegiate courses and is more likely to transfer to a four-­‐year university and earn a bachelor’s degree.1, 2, 3 Achieving this milestone is challenging for many students, however. National statistics indicate that 68% of students begin their community college English and math trajectory somewhere below transfer-­‐level.4 This alarming percentage should be taken as a minimum estimate, A Note on Terminology This paper uses the term “transfer-­‐level” to refer to gatekeeper, end-­‐of-­‐sequence courses and “below transfer-­‐
level” to refer to developmental or basic skills courses. A transfer-­‐
level course is a California Community College (CCC) course that is accepted by a California State University (CSU) and/or a University of California (UC). These courses typically but not always transfer to private and out-­‐of-­‐state universities as well. because many students assigned to below transfer-­‐level coursework never even enroll in a single English or math course. In one study of over 50 community colleges, 79% of students tested into the remedial sequence in English, math, or both.5 Perhaps not surprisingly, a large proportion of these so-­‐called basic skills or remedial students fail to advance and complete critical gatekeeper courses. In a study of 57 Achieving the Dream colleges across America, it was found that only one in five students who began the remedial math sequence at three or more levels below successfully completed the highest level of the remedial sequence and only one in ten completed the gatekeeper transfer-­‐level math course.6 This national pattern also holds for California community colleges. 1
Adelman (2006) Hayward (2011) 3
Moore & Shulock (2009) 4
Jaggars & Stacey (2014) 5
Scott-­‐Clayton, Crosta & Belfield (2012) 6
Bailey, Jeong, & Cho (2010) 2
5 When examining statewide progression using the Basic Skills Cohort Tracker, attrition is readily apparent for both English and math.7 Looking in detail at English, one can see how the volume of students progressing declines precipitously despite reasonably strong success rates at each course level (Figure 1). Figure 1. Statewide progression of students from three levels below transfer to transfer-­‐level English from fall 2010 through spring 2013. With somewhat lower per class success rates, the outcome for students starting from three levels below in the math sequence is even starker, as shown in Figure 2. Figure 2. Statewide progression of students from three levels below transfer to transfer-­‐level math from fall 2010 through spring 2013. 7
http://datamart.cccco.edu/Outcomes/BasicSkills_Cohort_Tracker.aspx 6 The observation that each level of a course provides an opportunity for a student to fail or simply fail to enroll in the subsequent course has shifted attention from success rates in single courses to a focus on the overall performance of the developmental sequence. Explicit descriptions of this pattern of exponential attrition have made the disconcerting implications clear.8,9,10 The proportion of students who fail to complete the remedial sequence has an inverse relationship to the number of levels a student must traverse before reaching the transfer-­‐level gatekeeper course. Growing awareness of the reality of exponential attrition has led to a burst of experimentation with programs designed to shorten the sequence, decrease the number of exit points, and increase completion of transfer-­‐level English and math. One such effort to take acceleration to scale, known as the California Acceleration Project (CAP), is the focus of this paper. There are a number of strategies that can be described as “acceleration.” 11 Modularization breaks the sequence into many low-­‐unit modules with the goal of focusing instruction on those areas where it is most needed and avoiding unnecessary coursework. Fast-­‐track courses provide intensive instruction in shorter time periods, potentially allowing students to pass multiple sequence levels in a single term.12 This approach is sometimes also referred to as a compression model. Mainstreaming is a form of acceleration that allows community college students to enroll in a transfer-­‐level course, typically with additional supports such as additional classes, tutoring, or supplemental instruction.13 Another form that mainstreaming takes is to extend the transfer-­‐level course over two terms, though this approach reduces the structural benefit of acceleration. The form of acceleration promoted by California Acceleration Project (CAP) is known as curricular redesign.14,15 Curricular redesign is a term for acceleration strategies that replace 8
Bahr, 2008 Snell in Bond, 2009 10
Hern & Snell, 2010 11
Zachry Rutschow & Shneider (2011) 12
Cf. Edgecombe, Jaggars, Baker, & Bailey (2013) 13
Cf. Jenkins, Speroni, Belfield, Jaggars, & Edgecombe (2010) 14
Edgecombe (2013) 9
7 multiple levels of remedial coursework by focusing the accelerated course’s curriculum on just those skills and abilities that are explicitly required for success at the transfer level – a principle known as backwards design. It resembles fast-­‐tracking, in that some students may be surmounting the equivalent of two or more semesters of the traditional remedial sequence in a single term but it differs in that it comprises only a single course in which all accelerated students are enrolled. Unlike mainstreaming, curricular redesign does not necessarily include additional supports or co-­‐requisites. The most aggressive form of curricular redesign shortens the remedial sequence to just one pre-­‐transfer-­‐level course that students can take regardless of their placement test score or prior course-­‐taking history. Overall developmental unit loads are typically reduced with this approach. Additionally, instead of simply repackaging the same content into a shorter timeframe, curricular redesign asks faculty to reconsider both content and pedagogy in developmental courses (e.g. increasing their emphasis on quantitative reasoning, decreasing emphasis on algebra for students in non-­‐STEM paths). While evidence of the relative effectiveness of the various acceleration strategies is developing, there is still a dearth of rigorous experimental and quasi-­‐experimental research capable of ruling out competing hypotheses for observed effects.16 The focus of this paper is on providing a rigorous, multivariate evaluation of the efficacy of the curricular redesign model of acceleration as implemented among multiple California community colleges. CAP is part of the California Community Colleges Success Network17 (3CSN) and the project promotes and supports a community of practice centered on accelerated pathways for English as well as math. CAP provides training, advice and support to faculty who are interested in implementing accelerated pathways at their local community college. While CAP colleges are free to implement acceleration in a locally appropriate way, they received guidance in two major aspects of implementing an accelerated pathway. The first aspect concerns the structure of the below transfer sequence. A common approach promoted by CAP involves creating a single pre-­‐transfer-­‐level English or math course that replaces two or more levels of the 15
http://cap.3csn.org/why-­‐acceleration/ Zachry Rutschow & Shneider (2011) 17
http://3csn.org/ 16
8 traditional sequence. Students that are successful in the accelerated course are expected to subsequently enroll in transfer-­‐level college composition or statistics/general education math, as appropriate.18 As with Carnegie Foundation's national efforts, Quantway and Statway, CAP’s math acceleration focuses on students in general education/statistics math pathways (as distinct from STEM pathways leading to pre-­‐calculus and calculus).19, 20 The second area in which CAP colleges received training and guidance involved the design principles of accelerated coursework.21 These principles include: •
Backwards design: Engage students in the same kinds of work required in the transfer-­‐
level course •
Higher levels of challenge: Assignments and assessments require a higher level of critical thinking than is typical in traditional developmental courses •
Just-­‐in-­‐time remediation: Students address foundational skills in the context of more challenging tasks •
Intentional support for affective issues: Classroom strategies for keeping students productively engaged (e.g., cultivating a growth mindset, reducing student fear, intrusive advising when students struggle) •
Contextualized teaching and learning: Teaching in a context that is meaningful and relevant to students' lives, including career preparation and social justice •
Increased reading: Students do more reading and/or more challenging reading than in the traditional sequence 18
CAP colleges were free to create variations on this theme. For example, two colleges in this evaluation developed a two-­‐step acceleration design in which the courses four and three levels below transfer were combined. We refer to these lower level accelerated pathways as ‘low-­‐acceleration’. The low-­‐accelerated course then either fed into a second accelerated course that articulated with the transfer-­‐level gatekeeper course (i.e., a ‘high-­‐acceleration’ course), or successful students could move back into the traditional sequence and take the two-­‐
level below course (and subsequently the one-­‐level below course and the transfer-­‐level course, if they persisted and were successful). The result was that two lowest courses in the sequence were combined into a low-­‐
acceleration pathway that required additional below transfer-­‐level coursework before students were eligible to enroll in the transfer-­‐level gatekeeper course. 19
http://www.carnegiefoundation.org/quantway 20
http://www.carnegiefoundation.org/statway 21
Hern & Snell (2013) 9 •
Increased writing demands: Students do more writing and/or more challenging writing than in the traditional sequence •
Thematic approach: Course assignments are connected by a relevant theme or driving question As described in more detail later, these design principles were, for the most part, widely adopted by both English and math faculty members involved in designing the accelerated courses. CAP occurs against a background of wide experimentation in strategies meant to improve the likelihood that students will complete transfer-­‐level English and math. This evaluation provides a better understanding of how the CAP acceleration principles fit into this zeitgeist. For instance, one group of complimentary strategies focus on preventing students from entering remediation in the first place, thereby increasing the volume of students whose first community college English and math coursework is at the transfer-­‐level. These strategies include support services in high school such as tutoring, parent outreach, and financial literacy such as high school advanced placement (AP) honors programs, the California Student Opportunity and Access Program22 (Cal-­‐SOAP), and Achievement Via Individual Determination23 (AVID). Other efforts involve partnerships between high schools and universities such as the Early Assessment Program24 (EAP) sponsored by the California State University (CSU). EAP provides diagnostic testing in 11th and 12th grade courses as well as workshops to help students address any deficiencies prior to enrolling in college. Yet another prevention-­‐based approach, given currency by the institutionalization of the Long Beach College Promise,25 focuses on improving assessment and placement systems by making better use of high school transcripts in placement and via collaborative curricular alignment between high schools and community colleges.26, 27, 28 This paper describes the outcomes of students from colleges that participated 22
http://www.csac.ca.gov/doc.asp?id=38 http://www.avid.org/ 24
http://www.calstate.edu/eap/ 25
http://www.longbeachcollegepromise.org/reports/ 26
Willett, Hayward, & Dahlstrom (2008) 23
10 in the first year of CAP implementation. It should provide an understanding of the performance of CAP’s acceleration principles so that policy analysts, administrators, faculty and students can make informed decisions about the relative value of CAP’s curricular redesign model of acceleration as a tool to improve completion of English and math sequences.29 We also examine the performance of student subgroups as well as variability in college-­‐level effects. In addition to descriptions of the various statistical models used to test the primary research hypothesis, the results section provides descriptive statistics about the student cohorts and accelerated pathways. Readers intrigued by the results section will find the technical appendices provide even more detailed reporting on outcomes and control variables. The discussion section of this paper is intended to be accessible to all audiences. It provides a review of the major findings of the study and also includes a discussion of the potential for acceleration to be brought to scale in terms of fidelity of implementation, ownership, and sustainability.30 Methods
The primary research hypothesis driving this evaluation of acceleration is as follows: Students who participate in accelerated pathways will complete the transfer-­‐
level gatekeeper course at a rate higher than comparable students who participate in the traditional sequence. This research hypothesis was examined by contrasting the completion of the transfer-­‐level gatekeeper course by accelerated students relative to comparable students who were enrolled in the traditional English and math basic skills sequences in the 2011-­‐2012 academic year. There were two cohorts of English accelerated students – those who took their first accelerated class in fall 2011 and those who took their first accelerated class in spring 2012. In a similar 27
Fuenmayor, Hetts, & Rothstein (2012) Willett (2013) 29
For more detail see Snell & Huntsman (2013) 30
Coburn (2003) 28
11 fashion, there were two accelerated math cohorts, one for fall 2011 and one for spring 2012.31 Students’ outcomes were tracked through spring 2013, allowing students up to two years to complete the transfer-­‐level gatekeeper course, depending on the term of their initial enrollment. Note that although these cohorts consisted of students taking their first accelerated course, this was not necessarily their first course in the sequence. In fact, the accelerated course was the first English course for only 68% of the English cohort and the first math course for only 41% of the math cohort. Four comparison groups were also created, one for each accelerated cohort as described later in this section. Faculty who participated in CAP training and implemented some form of acceleration at their college were invited to participate in a multi-­‐college evaluation of acceleration. Twenty six colleges total responded, 18 of which had implemented English acceleration and 13 of which had implemented math acceleration (five of these colleges had both English and math responses). Many of these colleges implemented their acceleration projects after the 2011-­‐
2012 academic year, which was the time frame used to establish cohorts for the current study. Eight CAP colleges had active math acceleration pathways and nine CAP colleges had active English acceleration pathways in the 2011-­‐2012 academic year. One college had both English and math accelerated pathways while another college had two distinct accelerated English pathways (one ‘low-­‐acceleration’ pathway for students at three and four levels below and one ‘high-­‐acceleration’ pathway for students at one and two levels below transfer). Thus, the evaluation includes 18 accelerated pathways at 16 colleges. These 16 colleges represented a broad spectrum, with the smallest college having an annual full time equivalent student (FTES) value of about 4,000 and the largest college having over 25,000 annual FTES. Half of the colleges were in multi-­‐campus districts. Three quarters were in urban or suburban areas, with the remaining quarter in rural areas. The percent of underrepresented 31
Two accelerated math sections held in the summer of 2012 were included in the second accelerated math cohort. Both for purposes of greater representation and in order to improve statistical power for the math sample, it was desirable to include these additional two sections. Since the summer sections fell in between the spring 2012 and fall 2012 primary terms, the timing was considered to be neither an advantage nor a consequential disadvantage over similar students who took their math classes in spring 2012. 12 minority (URM) students ranged from 20% to over 90% with an average (mean) of about 50%. The study nominally includes 3,197 accelerated students: 2,316 students were in English pathways and 881 students were in math pathways. However, 708 of the students were missing at least one of the required data points, such as placement level or demographic information, and therefore were excluded from the analysis. The descriptive statistics and multivariate analyses presented in this report are based on the 2,489 accelerated students (1,836 English + 653 math) with complete data profiles. Data sources
Data for this evaluation come primarily from the centralized administrative database maintained by the Chancellor’s Office for the California Community system. Comprising administrative data from 112 colleges, the Chancellor’s Office Management Information System (COMIS) database holds student unit records with enrollment, grade, course, and demographic information.32 For this evaluation, the COMIS data are supplemented by assessment and placement data submitted by each participating CAP college. Student-­‐level placement information is critical in order to properly match students who enter the flattened accelerated sequence with comparable students in the traditional, multi-­‐level remedial sequence. Finally, faculty from each CAP college completed an implementation survey that provided information on the specific ways in which acceleration was implemented at each site. Each data source is described in more detail below. The administrative COMIS data were used to identify enrollments in English and math courses for the accelerated students as well as for a comparison group in the traditional sequence. The primary dependent variable, successful completion of the gatekeeper transfer-­‐level math or English course, was derived from the COMIS data. Additionally, data were collected on each student’s initial English and math course, the number of prior successes and failures in the sequence, cumulative grade point average (GPA) (excluding the accelerated course or comparable traditional sequence course), age, gender, ethnicity, receipt of the Pell grant, 32
http://extranet.cccco.edu/Divisions/TechResearchInfoSys/MIS/DED.aspx 13 disability status,33 English as a Second Language (ESL) status, and Extended Opportunity Programs and Services (EOPS) status for use as covariates and for providing a method for better understanding how the effects of acceleration might vary among specific student sub-­‐
populations. Age was determined at the first term of enrollment in the 2011-­‐2012 academic year. Variables that can change from one term to the next such as receiving a Pell grant, disability status, and EOPS participation had broad inclusion criteria where a single term of receiving a Pell grant or disability services or participation in EOPS would result in a student being flagged as having that attribute. This broader inclusion allows for the recognition of students with additional challenges even if they stopped receiving grants or services. The CAP implementation survey was distributed via email to faculty contacts at each participating college for each accelerated pathway at that college. The response rate for the eight math colleges was 100% and the response rate for the 9 English colleges was 78%. The implementation survey allowed for the identification of accelerated students by gathering sufficient information on course identifiers and, if necessary, section identifiers, of accelerated classes. In instances where the implementation survey was not completed, accelerated pathways could readily be identified from the class schedule and/or course catalog because of distinct course identifiers. The implementation survey also provided information on any distinct practices in the areas of recruitment, pedagogy, student support, and sequencing that might set the accelerated courses apart from the classes offered in the traditional sequence. Additionally, there were two questions about the climate at the college regarding interest in (or opposition to) accelerated courses. Responses to these questions are useful for a qualitative analysis of inter-­‐college differences. 33
Cognitive disabilities were initially analyzed separately from non-­‐cognitive (e.g., hearing or mobility) disabilities. However, both types of disabilities were found to have similar relationships to sequence completion and were therefore combined into a single variable. 14 Each participating college was required to upload assessment and placement data for English and math because the COMIS does not include the placement data that would be required for matching students on assessed ability level. The collected assessment and placement data were matched to students in the COMIS data files. A Chancellor’s Office programmer received these data through secure transmission and encrypted the identifiers so they would match with the COMIS research data set. Referential checks ensured that each college had at least a 95% match rate between the placement data and COMIS data with most colleges easily exceeding that threshold. Even so, not all students in COMIS had assessment or placement data available, most likely due to the variety of options that students have for determining placement locally at each college, including equivalencies from other colleges, AP credit, transcript evaluations, and the absence of prerequisites for low-­‐level courses. The assessment to COMIS match rate for the accelerated English students was 80% (1,989/2,489) while the match rate for accelerated students in math was 63% (556/881). The key variable used from the placement data set was the students’ highest placement level in the college’s English or math sequence. The assessment data were used to provide a ‘current level’ for students with no prior coursework in the sequence, thereby allowing both students with relevant academic histories and first-­‐time students to be included in the analysis. Including both first-­‐time students and students with prior relevant coursework enhances the ecological validity of the analysis – an important consideration since many of the students in the accelerated cohorts were not first-­‐time. The current level variable allowed for the control of a student’s current level or place in the traditional sequence as of the term in which they were entered into either an accelerated or comparison cohort. A student’s current level was determined by the highest level of the sequence attempted or, if the student had no prior enrollments in the sequence, by the student’s placement level. This control variable is particularly important because, by design, the course level34 of the accelerated course does not necessarily coincide with the placement or preparation level of enrolled students. Yet, because completion of prerequisite coursework (or, lacking that, placement level) is presumed to be related to a 34
COMIS data element CB21, COURSE-­‐PRIOR-­‐TO-­‐COLLEGE-­‐LEVEL. 15 student’s skill level in English or math, it is important to evaluate and control for any systematic differences between the accelerated students and the comparison students on this factor.35 Comparison groups
Comparison students were drawn from the pool of students who had enrollments in English and/or math in the 2011-­‐2012 academic year. Students with a prior record of having successfully completed transfer-­‐level math were excluded from the math comparison group while students who had previously passed transfer-­‐level English were excluded from the English comparison group. Four comparison groups were formed: two for English and two for math. Students were assigned to either comparison cohort 1 or comparison cohort 2 depending on their enrollment patterns. Students with enrollments in the traditional English and math sequences fell into three approximately equal groups: (1) those who had a qualifying enrollment (in either English or math) in fall 2011, but not in spring 2012; (2) those who had a qualifying enrollment in spring 2012, but not in fall 2011; and (3) those who had a qualifying enrollment in both terms. Students who had enrollments in the appropriate subject in fall 2011, but not in spring 2012 were assigned to cohort 1, students with appropriate enrollments in spring 2012, but not in fall 2011 were assigned to cohort 2, and those with appropriate enrollments in both terms were randomly assigned to either cohort 1 or cohort 2. Comparison cohorts 1 and 2 for math include 12,086 and 11,521 students, respectively, for a total of 23,607 students in the math comparison group. Comparison cohorts 1 and 2 for English include 11,830 and 10,524 students, respectively, for a total English comparison group of 22,354 students. Cohort term was included as a control variable in the multivariate analyses since students in cohort 1 had an additional primary term to progress. Note that students were assigned to a cohort based on their observed enrollments in the 2011-­‐2012 academic year, the qualifying enrollment was not necessarily their first enrollment in the sequence. This is an 35
While Bahr (2013) presents a case that the effect of initial enrollment level on sequence completion is not linear and therefore should be considered as a categorical variable, we found that treating current level as a categorical variable did not significantly improve our model and that the observed effects of current level were generally linear, perhaps because current level is not limited to only first-­‐time in the sequence students. In any case, we included current level as a continuous variable and modeled its effect on sequence completion as linear. 16 important consideration because many accelerated students attempt the accelerated course only after initially enrolling in courses in the traditional sequence. As shown in the Results section, accelerated and comparison group students were matched on current level as well as prior successes and non-­‐successes in the sequence. A student with no prior successes or non-­‐
successes is effectively a first-­‐time (in the sequence) student. Descriptive statistics: Students characteristics
Tables 1, 2, 3, and 4 provide descriptive statistics of the students in the accelerated and comparison groups. Compared to the English comparison group, accelerated English students were: •
more likely to have a lower starting level, •
equally likely to be female, •
more likely to be African American or Hispanic, •
more likely to have received a Pell grant, •
equally likely to have been in EOPS, •
slightly more likely not to have graduated from high school, and •
more likely to have been identified with a disability (Table 1). Accelerated and comparison group students had very similar GPAs and ages while accelerated students had fewer prior English successes and non-­‐successes than comparison group student (Table 2).36 36
Variable definitions may be found in Appendix A. 17 Table 1. Descriptive statistics for English acceleration and comparison group students: Categorical variables. Characteristics Total English sample Cohort 1 (Fall 2011) Cohort 2 (Spring 2012) Current level Four levels below Three levels below Two levels below One level below Transfer-­‐level Female Ethnicity African American Asian Hispanic White All other grant recipient Pell EOPS participant Not a high school graduate Any disability Accelerated Cohort Count Percent 1,836 100% 856 47% 980 53% 678 37% 276 15% 707 39% 161 9% 14 1% 961 52% 278 15% 185 10% 1,015 55% 205 11% 153 8% 1161 63% 309 17% 82 4% 202 11% Comparison Group Count Percent 22,354 100% 11,830 53% 10,524 47% 2,036 9% 2,792 12% 8,760 39% 6,357 28% 2,409 11% 11,727 52% 2,566 11% 2,209 10% 10,658 48% 4,530 20% 2,391 11% 13,143 59% 3,796 17% 661 3% 1,892 8% 18 Table 2. Descriptive statistics for English acceleration and comparison group students: Continuous variables. Accelerated Cohort Mean SD 1.72 1.057 0.37 0.746 0.33 0.828 21.26 6.311 Continuous Variables GPA control Prior English successes Prior English non-­‐successes Age* Comparison Group Mean SD 1.70 0.969 0.65 0.957 0.53 0.964 22.39 7.160 SD = Standard Deviation * Age was included as a control variable in early models. It had no net effect on sequence completion, so, for parsimony, it was excluded from the final models. Compared to the math comparison group, accelerated math students were: •
more likely to have a lower starting level, •
more likely to be female, •
more likely to be African American or White, •
more likely to have received a Pell grant, •
more likely to have been in EOPS, •
slightly less likely not to have graduated from high school, and •
more likely to have been identified with a disability (Table 3). Accelerated math students had higher GPAs, about the same number of prior math successes, more prior math non-­‐successes and similar ages compared to the math comparison group students (Table 4). 19 Table 3. Descriptive statistics for math acceleration and comparison group students: Categorical variables. Accelerated Cohort Count Percent 653 100% 144 22% 509 78% Characteristics Total math sample Cohort 1 (Fall 2011) Cohort 2 (Spring 2012) Current level Four levels below Three levels below Two levels below One level below Transfer-­‐level Female Ethnicity African American Asian Hispanic White All other grant recipient Pell EOPS participant Not a high school graduate Any disability 86 199 285 69 14 399 88 28 230 235 72 355 123 13 115 13% 30% 44% 11% 2% 61% 13% 4% 35% 36% 11% 54% 19% 2% 18% Comparison Group Count Percent 23,607 100% 12,086 51% 11,521 49% 3,089 4,864 8,447 5,619 1,588 12,639 2,634 2,564 8,859 6,324 3,226 11,009 2,654 690 2,268 13% 21% 36% 24% 7% 54% 11% 11% 38% 27% 14% 47% 11% 3% 10% 20 Table 4. Descriptive statistics for math acceleration and comparison group students: Continuous variables. Accelerated Cohort Mean SD 2.28 0.868 0.76 0.981 1.04 1.593 24.3 9.111 Continuous Variables GPA control Prior math successes Prior math non-­‐successes Age* Comparison Group Mean SD 2.06 0.945 0.77 0.960 0.79 1.280 23.70 8.011 SD = Standard Deviation * Age was included as a control variable in early models. It had no net effect on sequence completion, so, for parsimony, it was excluded from the final models. Descriptive statistics: Implementation survey
According to the responses of key informants at each college, accelerated math implementations were more likely to target students with lower likelihoods of success than accelerated English implementations (Table 5). For example, only three of the seven respondent colleges in the English analysis targeted students “at risk for academic failure” as compared with all eight of the colleges in the math analysis. Similarly, most math implementations targeted students who had been previously unsuccessful in the traditional sequence or who had low confidence in their skills; by contrast, very few of the English implementations targeted this population. To help control for these differences, the analysis included the number of prior successes and non-­‐successes in the sequence and an overall GPA control. The colleges that indicated they focused on students of color tended to have high proportions of underrepresented minorities in their student body. This focus was more prevalent for math implementations than English. The other specific populations of first-­‐time students, honors students, students with high confidence in their skills, and learning community participants were targeted by a minority of colleges in both disciplines. 21 Table 5. Percent of college implementations targeting specific populations of students by discipline (populations not mutually exclusive). Target Population At risk for academic failure Unsuccessful in traditional sequence Students with low confidence in their skills Students of color First-­‐time college students Honors students Students with high confidence in their skills Learning community participants Count of Responses English 43% 43% 29% 29% 43% 0% 0% 29% 7 Math 100% 88% 75% 50% 38% 13% 13% 13% 8 The recruitment patterns for accelerated courses indicate an emphasis on under-­‐prepared students and those most at risk for failure, particularly for math. In the one case where recruitment targeted honors students, high confidence students and/or students with additional supports such as those in learning communities, those recruitment efforts were complemented by additional recruitment efforts targeting at-­‐risk populations and students with histories of prior failure in the subject area. The majority of colleges utilized counselors as a primary recruitment strategy followed by faculty of prerequisite courses and class schedules and flyers (Table 6). Math implementations had higher rates of specific recruitment strategies than English implementations, particularly for at-­‐risk and low-­‐confidence students. Inclusion in the logistic regression model of statistical controls for ethnicity as well as for prior successes and non-­‐successes in the sequence alleviate concerns about biasing the estimate of the acceleration effect due to systematic differences between the accelerated and comparison groups due to differences in recruitment practices. 22 Table 6. Recruitment strategies for accelerated courses by subject (strategies not mutually exclusive). Recruitment Strategy College counselors Faculty of prerequisite courses Class schedule or flyers Open, no particular recruiting During assessment Non-­‐counselor college personnel Via application or other formal process Integrated into specialized pathway High school personnel Count of Responses English 71% 43% 43% 43% 29% 14% 14% 0% 0% 7 Math 88% 75% 75% 50% 38% 13% 13% 13% 0% 8 While the majority of accelerated sections were not team taught (Table 7), three colleges did team-­‐teach some of their accelerated sections and one college team-­‐taught all of its math sections. Table 7. Team teaching of accelerated sections by subject. Team-­‐taught No sections Some sections All sections Count of Responses English 71% 29% 0% 7 Math 71% 14% 14% 7 The majority of accelerated implementations had faculty collaborate on the design of curriculum and assessment methods for the accelerated courses (Table 8). Table 8. Faculty collaboration in building curriculum and assessments for accelerated courses. Faculty collaboration No Yes Count of Responses English 0% 100% 7 Math 29% 71% 7 23 Most colleges used a variety of strategies to shape and implement accelerated curricula (Table 9). These design principles were promoted by the training that CAP colleges received via their participation in the California Acceleration Project. Just-­‐in-­‐time remediation was the most widely implemented design principle among accelerated English pathways, while contextualized teaching and learning was the most popular design principle among the accelerated math pathways. The adoption of design principles was fairly similar across English and math CAP colleges, though accelerated English courses were more likely than math courses to include a thematic approach. 37 Table 9. Design principles adopted in accelerated pathways by subject. Design principles Just in time remediation: Students address foundational skills in the context of more challenging tasks. This approach is in contrast to frontloaded instruction in sub skills (e.g., completing isolated grammar exercises devoid of context) Backwards design: Engaged in the same kinds of reading, thinking, and writing required in the transfer-­‐level course Intentional support for affective issues: Classroom strategies for keeping students productively engaged (e.g., cultivating a growth mindset, reducing student fear, intrusive advising when students struggle) Increased reading: Students do more reading and/or more challenging reading than in the traditional sequence Contextualized teaching and learning: Teaching in a context that is meaningful and relevant to students' lives, including career preparation, community service and social justice Higher levels of challenge: In class tasks, assignments, and assessments require a higher level of critical thinking than is typically required in traditional developmental courses Increased writing demands: Students do more writing and/or more challenging writing than in the traditional sequence Thematic approach: The course’s assignments are connected by a relevant theme or driving question Average number of design principles implemented Count of responses English Math 100% 75% 86% 75% 86% 75% 86% 75% 71% 88% 71% 75% 86% 63% 86% 38% 6.7 7 5.6 8 37
CAP did not provide training in the ‘thematic approach’ design principle to math faculty. 24 According to the local stakeholders who responded to the implementation survey, the majority of English implementations received wide support from both department faculty and administrators (Table 10). Math acceleration tended to receive a mixed reception among department faculty, but had stronger support from administrators. Table 10. Support for implementing acceleration from department and administration by subject. English Math Support for Acceleration Department Department Administration Administration Faculty Faculty Not much interest 0% 14% 14% 0% Substantial opposition 0% 0% 0% 0% 71% Mixed reception 29% 0% 29% 71% 86% 71% Wide support 14% Total Percent 100% 100% 100% 100% Count of Responses 7 7 7 7 Most acceleration implementations did not provide any additional supports (Table 11). Of those that did, supplemental instruction was the most common. One college paired their acceleration with Puente and Academy for College Excellence (ACE) programs as a source of additional support. Five colleges, two in English and three in math, had at least some of their accelerated sections as part of learning communities. Table 11. Additional supports provided to accelerated students above and beyond support services available to all students. Additional Support* Supplemental instruction Tutoring Counseling Financial support Text books Count of Responses * Supports are not mutually exclusive English 43% 29% 14% 0% 0% 7 Math 14% 0% 0% 14% 0% 7 25 Description of Logistic Regression and Multivariate Models
Logistic regression is typically used to evaluate situations where two possible outcomes may arise. In this case, students may either complete the remedial sequence and the gatekeeper transfer-­‐level course, or not. In order to understand any unique effect of acceleration, it is critical to first remove the influence of any student characteristics that are not randomly distributed between the accelerated students and comparison students in the traditional sequence. By using multivariate logistic regression, marginal means and other statistical procedures, we are able to equate the two groups on 13 potentially confounding variables. The use of extensive academic, socioeconomic and demographic controls provides a rigorous test of the research hypothesis. The presence of multiple controls allows us to clearly evaluate the unique effect of accelerated classes on gatekeeper completion, relative to traditional English and math pathways. Other analysis methods were considered as well. Poisson regressions with robust variance can have advantages over logistic regression (Barros & Hirakata, 2003), but showed findings equivalent to logistic regression in this case. Only logistic regression outputs have been shown due to their greater familiarity for most audiences. Another method considered was propensity score matching (PSM), however due to the large number of data points and control variables, logistic regression should provide a more accurate estimate of effect sizes (Soledad Cepada, Boston, Farrar & Strom 2003). 26 Results
Overall English and math multivariate models
English model
The full multivariate logistic regression model for English students indicated that the odds (see sidebar next page) of students in accelerated English pathways completing the transfer-­‐level English gatekeeper course were approximately 1.5 times higher than the odds of completion for students in the traditional sequence (Table 12). Moreover, inclusion of the acceleration term improved the overall fit of the model as shown by comparing the covariate-­‐only model to the full regression model (Appendix B). Table 12 presents the overall model for English, however, acceleration effects tended to vary from one implementation to another and the odds ratio for each participating college and pathway are presented in a later section Table 12. Logistic regression coefficients predicting completion of English gatekeeper course showing acceleration effect net effect of covariates (N=24,190). Variable Accelerated Cohort 1 Not a high school graduate Asian Black Hispanic Other ethnicity Female Any Disability EOPS participant Pell Grant recipient ESL coursework (ever) Current level Prior English non-­‐successes Prior English successes GPA control Constant B 0.427 -­‐0.059 -­‐0.941 0.523 -­‐0.398 -­‐0.005 0.043 0.186 -­‐0.099 0.224 -­‐0.063 -­‐0.266 0.302 -­‐0.338 0.365 0.825 -­‐3.41 SE 0.063 0.033 0.128 0.062 0.067 0.043 0.061 0.033 0.061 0.045 0.035 0.077 0.017 0.022 0.018 0.018 0.076 Wald 46.585 3.251 53.700 71.406 35.430 0.014 0.507 32.845 2.571 25.035 3.172 11.912 332.378 227.896 416.319 2030.804 2029.335 df 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Sig. <0.005 0.071 <0.005 <0.005 <0.005 0.907 0.477 <0.005 0.109 <0.005 0.075 <0.005 <0.005 <0.005 <0.005 <0.005 <0.005 Odds Ratio 1.533 0.943 0.390 1.688 0.672 0.995 1.044 1.205 0.906 1.251 0.939 0.766 1.352 0.713 1.440 2.282 0.033 SE = Standard Error; df = degrees of freedom; Sig. = significance level 27 The covariate effects described below are not associated with acceleration, per se. They are general effects observed in the student population that are congruent with findings from decades of similar work in educational research. The effect sizes for these covariates provide a useful context for interpreting the relative importance of the unique acceleration effect. The strongest covariates were the academic controls, including GPA control, current level, prior English non-­‐successes, and prior English successes. The odds of students completing the sequence increased by 2.282 for each point of increase in concurrent GPA. For example, the odds of students with a 3.0 GPA completing the English sequence were Interpreting Odds Ratios The primary output of the multivariate logistic regression models used in this evaluation are odds ratios. An odds ratio (OR) is given for each factor or variable in the model. Most of the factors in the model are considered to be control variables and are of secondary interest. The primary goal of this evaluation was to investigate whether participation in accelerated pathways increased the odds of students completing the transfer-­‐level gatekeeper course. An odds ratio equal to one (OR = 1.0) indicates the students with a given characteristic (e.g., participated in an accelerated pathway) have about the same odds of completing the gatekeeper course as do students without that characteristic. An odds ratio that is significantly greater than one (OR > 1.0) indicates that students with the characteristic are more likely to complete the gatekeeper course while odds ratios that are significantly less than one (OR < 1.0) indicate that the presence of that characteristic is associated with lower odds of completing the gatekeeper course. 2.282 times higher than the odds of completion for students with a 2.0 GPA. Similarly, the odds ratio increased by 1.440 for each successful prior English class completion while each prior non-­‐success (a “D,” “F,” or “W”) in an English class significantly reduced the odds of sequence completion. Current level, which indicates the highest English level attempted for students with an academic history or the highest placement level for students with no prior English coursework, was a significant predictor of successful sequence completion. For example, the odds of students with a current level of two levels below transfer were 1.352 greater than for students with a current level of three levels below. The effect of increasing a student’s current level is equivalent (coincidentally) to the greater odds of English gatekeeper course completion due to participation EOPS (odds ratio = 1.352). Disability status was not a significant predictor of English gatekeeper course completion. Similarly, having received a Pell grant was not a significant predictor of English gatekeeper course completion. Relative to the odds of completion for White students, Asian students are more likely to complete the English gatekeeper course while African Americans are less likely to complete, 28 Hispanic students are equally as likely to complete the gatekeeper course as are White students. Math model
Results for the full math logistic regression indicated a large, positive effect on completion of transfer-­‐level math associated with participation in an accelerated math pathway (Table 13). The odds ratio (OR) indicated that the odds of students in the accelerated math pathway completing transfer-­‐level math were about 4.5 times greater than the odds for students in the traditional sequence. It should be noted that students in the accelerated math sequence were completing statistics as their transfer-­‐level math while students in the traditional pathway could complete any transfer-­‐level math. Regression models comparing the covariate-­‐only model to the full regression model with the accelerated independent variable showed a significant increase in effect size (pseudo R2) for math (Appendix B). While Table 13 presents the overall model, acceleration effects tended to vary by specific pathway and the odds ratio for each participating college and pathway are presented in a later section. Table 13. Logistic regression coefficients predicting completion of math gatekeeper course showing acceleration effect net effect of covariates (N=24,260). Variable Accelerated Cohort 1 Current level Female Asian Black Hispanic Other ethnicity Not a high school graduate Any Disability EOPS participant Pell recipient Prior math nonsuccesses Prior math successes GPA control Constant B 1.515 -­‐0.065 0.556 -­‐0.047 0.435 -­‐0.157 -­‐0.105 0.080 -­‐0.118 -­‐0.184 0.204 0.087 -­‐0.020 0.322 0.773 -­‐4.86 SE 0.092 0.038 0.019 0.037 0.061 0.075 0.047 0.060 0.118 0.069 0.061 0.040 0.016 0.019 0.023 0.095 Wald 273.614 2.985 822.107 1.600 50.239 4.405 5.003 1.782 1.002 7.215 11.096 4.743 1.626 287.430 1105.906 2617.700 SE = Standard Error; df = degrees of freedom; Sig. = significance level df 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Sig. <0.005 0.084 <0.005 0.206 <0.005 0.036 0.025 0.182 0.317 0.007 0.001 0.029 0.202 <0.005 <0.005 <0.005 OR 4.549 0.937 1.744 0.954 1.546 0.854 0.900 1.084 0.888 0.832 1.226 1.091 0.980 1.379 2.167 0.008 29 As with the English model, it is important to note that the covariate effects described here are not associated with acceleration, per se. They are general effects observed in the student population that are congruent with findings from decades of similar work in educational research. The effect sizes for these covariates do, however, provide a useful context for interpreting the relative importance of the unique acceleration effect. Among the covariates, the most potent were again the academic controls, including GPA control, current level, and prior math successes. The odds of students completing the sequence increased by 2.167 for each point of increase in concurrent GPA. In other words, the odds of students with a 3.0 GPA completing the math sequence were 2.167 times higher than the odds of completion for students with a 2.0 GPA. Similarly, the odds ratio increased by 1.379 for each successful prior math completion. Current level, which indicates the highest math level attempted for students with an academic history or the highest placement level for students with no prior math, was strongly associated with increased odds of successful sequence completion. For example, the odds of completion for students with a current level of two levels below transfer were 1.744 greater than for students with a current level of three levels below. Participation in the EOPS program was associated with greater odds of math gatekeeper course completion, whereas having any type of disability was associated with a reduction in the odds of completing the math gatekeeper course. Having received a Pell grant had a small, positive association with math gatekeeper course completion. Relative to the odds of completion for White students, Asian students were more likely to complete the math gatekeeper course, while African-­‐American and Hispanic students were somewhat less likely to complete the math gatekeeper course. Marginal means
Marginal means provide another way to present and discuss the effect size of key variables in the model. To create marginal means, first each covariate in the regression model has its average (mean) value entered into the regression equation (Appendix C). For example, as 52% of the students in the English regression were female, a value of 0.52 would be entered into the 30 female variable. Likewise, as 48% of the students in the English regression were Hispanic, a value of 0.48 would be entered into the Hispanic variable. After all of these average values have been entered for the covariates, a value of “0” is entered into the independent variable to represent an “average” student being in the comparison group. The regression equation then outputs an estimate of what percent of the comparison is predicted or estimated to successfully complete transfer-­‐level coursework. This process is conducted again with a value of “1” entered into the independent variable to estimate outcomes for an “average” student in an accelerated pathway. In Figure 3, we see that for a cohort of average students in a non-­‐
accelerated or comparison pathway, 22% were predicted to successfully complete transfer-­‐
level English as compared to 30% of average students in an accelerated pathway. For math, 12% of a cohort of average students in a non-­‐accelerated or comparison pathway were predicted to successfully complete transfer-­‐level math as compared to 38% of a cohort of average students in an accelerated pathway. 31 100%
Percent of Students Successfully
Completing Transfer-Level Course in
Sequence
Comparison
Accelerated
80%
60%
38%
40%
30%
22%
20%
12%
0%
English
Math
Figure 3. Marginal means for the percentage of students completing transfer-­‐level English and math for accelerated and comparison sequences for all students. Readers may notice that the magnitude of the acceleration effect appears to vary somewhat between the odds ratio and the marginal means representations. This apparent difference is a normal artifact of a scaling difference between the two approaches to representing effect size. Marginal means express effect size in terms of relative risk, which is bounded because values may not exceed 100%. So, for small effect sizes, odds ratios and relative risk statistics are fairly congruent, but for larger effect sizes, odds ratios scale up much more dramatically. Both types of statistics are in common use and presenting both provides a sense of perspective regarding the effect size. 32 Percent of Students Successfully
Completing Transfer-Level Course
in Sequence
Comparison
Accelerated
100%
80%
60%
40%
20%
22%
17%
27%
21%
32%
25%
38%
30%
0%
Starting Place Starting Place Starting Place Starting Place
4 or More 3 Levels Below 2 Levels Below 1 Level Below
Levels Below
English Starting Place
Figure 4. Marginal means for the percentage of students completing transfer-­‐level English for accelerated and comparison sequences by current level. In Figures 4 and 5, the marginal means are applied to the current levels for English and math students, respectively. Students at all levels show higher throughput rates in accelerated pathways. Although students with a higher current level show a larger absolute gap in percentage points, the relative gain of students at the lower levels (i.e., [accelerated rate – traditional rate]/traditional rate) is in fact larger than the relative gain of students who start at the higher levels. For instance, the five point difference (22-­‐17) between accelerated English students placed four levels below represents a 29% increase relative to the throughput rate of students in the traditional sequence (17%). The 8 point gain for students starting at one level below, while larger in absolute terms, represents only a 26% increase. Similarly, for math the 15 point difference between accelerated and non-­‐accelerated students at four levels below 33 represents a 250% increase while the 30 point gain for students at one level below represents a Percent of Students Successfully
Completing Transfer-Level Course
in Sequence
130% increase relative to the completion rate of students in traditional pathways. 38 Comparison
100%
Accelerated
80%
60%
53%
41%
40%
30%
21%
20%
6%
10%
23%
15%
0%
Starting Place 4 Starting Place 3 Starting Place 2 Starting Place 1
or More Levels Levels Below
Levels Below
Level Below
Below
Math Starting Place
Figure 5. Marginal means for the percentage of students completing transfer-­‐level math by spring 2013 for accelerated and comparison sequences by current level. A benefit of marginal means analysis is that the procedure incorporates all of the effects of the statistical controls and covariates while presenting an intuitively accessible comparison that can be represented as percentages. However, these predicted outcomes are for a theoretical “average” student who is 52% female, 48% Hispanic, and so on for the other variables. It is also possible to estimate outcomes for specific values of other covariates such as for female 38
Additional regressions (not shown) were run to test for the significance of interaction effects of each level of current level by acceleration status. Significant interactions were found for the overall math model and for the English pathways identified as “high acceleration” in Figure 7. While the overall acceleration effects (main effects) were similar to those in Table 12 and Figure 7, the interactions showed that, in general, acceleration had its strongest effects for students starting at the lower levels of the sequence. 34 students, holding all other covariates to their average values. Appendices D and E display marginal means for specific values of each covariate. Figures 6 and 7 present these values for the ethnicity covariates below, in the section on acceleration and ethnicity. College- and pathway-specific acceleration effects
Participation in an accelerated pathway showed a robust, significant effect on gatekeeper completion. However, there was considerable variation in how colleges implemented acceleration, particularly for English pathways. College level effects for English are shown in Figure 6. Each pathway is denoted by a pseudonyms based on the Greek alphabet in order to maintain anonymity. Darker bars are used to portray acceleration odds ratios that did not reach significance in the individual college-­‐specific regression models. 3.5 3 2.81 2.96 2.5 2.34 2.16 2 1.5 1.35 Zeta* Alpha.2* Beta†*‡ Epsilon* Gamma* 0 Omega† 0.5 0.90 1.16 Pi‡ 0.82 Rho 1 Iota† 1.04 Alpha.1† Acceleratoin Odds Raco 2.53 College Pseudonym † These pathways require a challenge/waiver process to allow enrollment in transfer-­‐level English ‡ Did not complete the implementation survey * p < 0.01 35 Figure 6. Acceleration effect size (odds ratio) by college-­‐specific English pathways (lighter bars with asterisks (*) are significant at p < 0.01). A qualitative analysis of the college-­‐level effects revealed that there was a tendency for those English pathways that did not directly articulate with the transfer-­‐level English course by default to have low, non-­‐significant odds ratios. Three of the five English acceleration pathways with non-­‐significant English acceleration effects imposed restrictions on the advancement of students to the transfer-­‐level. This restriction primarily took the form of requiring those students who successfully completing the accelerated pathway to go through a challenge or waiver process to allow enrollment into the transfer-­‐level course; without completing the challenge process they would be directed into additional below transfer level coursework, by default. This pattern of smaller, non-­‐significant effects being primarily associated with longer, more restricted acceleration pathways, led us to hypothesize that less permeable pathways, such as those created when a challenge/waiver process imposes a substantial barrier, do not lead to robust acceleration effects. Conversely, in programs where the challenge/waiver process is streamlined or automatic, the pathways are relatively open and acceleration effects are similar to those colleges where the accelerated course is a default prerequisite to the transfer-­‐level course. This distinction is illustrated by the presence of two distinct pathways at one college (Alpha). The first pathway, Alpha.1 is a pathway that replaces courses at four and three levels below. While successful students can petition to subsequently enroll in transfer-­‐level English, by default the Alpha.1 pathway articulates either with the English course at two levels below or with a second accelerated pathway (i.e., Alpha.2). We labeled this type of accelerated pathway ‘low-­‐acceleration’. Students in the low-­‐acceleration, Alpha.1 pathway do not show a significant acceleration effect. The second acceleration pathway, Alpha.2, is a more typical acceleration pathway that combines students at one or two levels below and articulates directly with the 36 transfer-­‐level English. This type of pathway is labeled ‘high-­‐acceleration’. In contrast to Alpha.1, the high-­‐acceleration pathway (Alpha.2) shows a large, significant acceleration effect. Further exploration of the high-­‐acceleration pathways relative to low-­‐acceleration pathways found that high-­‐acceleration pathways had an odds ratio of 2.3 (see Appendix G). A logistic regression (not shown) based only on the four low-­‐acceleration colleges yielded an odds ratio of only 1.2 for the acceleration effect size. These estimates are useful for providing a general sense of the potential impact of going to scale with only high-­‐acceleration pathways in English across a variety of colleges (Figure 7). Accelerazon Odds Razo (Effect Size) for English CAP Colleges 2.3 2.5 2 1.5 1.5 1.2 1 0.5 0 All English CAP pathways High-­‐accelerazon pathways Low-­‐accelerazon pathways Figure 7. Acceleration effects for all English pathways, high-­‐acceleration English pathways, and low-­‐acceleration English pathways. An alternative way to compare the effect of high-­‐acceleration pathways relative to low-­‐
acceleration pathways is to present the marginal means. As explained earlier, the marginal means procedure presents an estimated completion rate for a specific group in the specific situation where all other covariate values are held to their average valuable, thus creating an estimated completion rate for the hypothetical average student. Figure 8 shows that students in high-­‐acceleration English pathways had an estimated gatekeeper completion rate of 38% 37 relative to 20% for students in the traditional sequence. Students in the low-­‐acceleration pathways had gatekeeper completion rates (22%) that were equivalent to those of similar Percent of Students Successfully Completing
Transfer- Level Gatekeeper Course
students in the traditional sequence (23%) 100%
80%
60%
38%
40%
20%
20%
23%
22%
English Low
Acceleration
Model
Comparison
Students
English Low
Acceleration
Model
Accelerated
Students
0%
English High
Acceleration
Model
Comparison
Students
English High
Acceleration
Model
Accelerated
Students
Figure 8. Estimated percent of accelerated and comparison group students successfully completing transfer-­‐level gatekeeper English by spring 2013 for high-­‐acceleration and for low-­‐
acceleration pathways. In addition, a qualitative analysis of the implementation survey data revealed that there was a tendency for the colleges that implemented fewer of the CAP design principles to have lower acceleration effects. While there was not a great deal of variability among English CAP colleges in implementing CAP design principles (one college implemented three principles and the others implemented either seven or eight principles), the college which implemented only 38 three design principles did evince a lower acceleration effect than the average effect among colleges with seven design principles. Similarly, the average acceleration effect of colleges that implemented seven design principles was lower than that of the colleges that implemented eight design principles. Even though these differences did not reach statistical significance, the regularity and linearity of the pattern is noteworthy and suggests a potential area for future research. Interested readers may see Appendix F for additional qualitative information on the relationship between design principle implementation and acceleration effect size. It may be that waivers and challenge processes do not necessarily present barriers to students in all implementations. The processes surrounding waiver application may be simple and easy or complex and difficult. For example, while waiver processes in English appeared to impose a barrier to acceleration, all but two of the math acceleration pathways required waiver processes, yet the acceleration effects for these pathways were large and statistically significant with only one exception. For the math pathways, the waiver processes appear to have been streamlined and therefore it did not impose a significant barrier to student advancement. Among the math colleges, seven of the eight accelerated math pathways showed significant acceleration effects (Figure 9). With an odds ratio of nearly 18, Kappa College was a clear outlier among the math colleges, in terms of acceleration effect size. The math logistic regression model was run again without the outlier college (Kappa) in order to confirm that the significance of the acceleration effect in the overall model was not solely due to the one outlier college. When the outlier is excluded, the overall effect size drops from 4.5 to 4.0 – still a large and statistically significant effect. 39 20.00 17.76 18.00 16.00 14.00 12.00 10.00 7.25 8.00 6.00 4.00 2.00 0.94 0.00 Lambda 2.77 3.06 3.12 Nu* Mu* Delta* 5.03 5.11 Theta* Beta* Eta* Kappa* * p < 0.01 Figure 9. Acceleration effect size (odds ratio) by college-­‐specific math pathways (lighter bars with asterisks (*) are significant at p < 0.01). It was not immediately apparent why Lambda College's accelerated math pathway was the only math pathway not to have a statistically significant acceleration effect. One hypothesis is that, because the college in question had few accelerated cohorts, the lack of effect could simply be due to random chance. It could have just been an effect of a particularly negative set of circumstances with those particular students. Alternatively (or in addition), the lack of an acceleration effect at Lambda College could perhaps represent issues with implementation at that particular site. Lack of adherence to the design principles advocated by CAP could be responsible for the lack of a significant acceleration effect at Lambda College. The implementation survey revealed that Lambda College did not implement any of the CAP design principles. It was the only college not to do so. This result parallels the qualitative finding among English acceleration pathways where those colleges that implemented more design principles tended to have larger acceleration effects. Indeed, a similar pattern of linear increase in acceleration effect is found among math colleges when they are grouped into low, medium and high design principle implementation groups. There is more 40 variability in the number of design principles implemented among the math colleges than among the English colleges and the linear pattern is, if anything, more pronounced among the math colleges (see Table 9 and Appendix F). In summary, while Lambda College’s acceleration pilot represents a departure from the norm in math acceleration, it is clear that, in general, math acceleration had a strong and positive association with completion of the math sequence across a variety of colleges and implementations. It did appear that the efficacy of the accelerated math pathways was affected by the degree to which CAP design principles were implemented. The one math pathway that did not show a statistically significant effect for acceleration was one that did not implement any of the CAP design principles. Thus, it seems that in order to reap the full benefit of acceleration, not only must the structure of the pathway be minimal and streamlined, but implementation of at least some of the CAP design principles is perhaps requisite as well. Acceleration and ethnicity
This section explores whether there is evidence that acceleration reduces the achievement gap. Comparing the ORs for ethnic group membership from the acceleration-­‐only model provides some clues as to whether acceleration addresses the achievement gap. The key question is whether there is a differential positive (or negative) effect for underrepresented ethnic groups that would have a bearing on the achievement gap. In the math regression model for just students in the traditional sequence (not shown), African American ethnicity is associated with an odds ratio that is significantly less than 1 (OR < 1.0), indicating that African American students’ odds of completing the sequence are lower than those of White students. In the regression model that is restricted to accelerated students only, however, African American students have a non-­‐significant odds ratio that is very close to 1.0, indicating that they have the same odds of completing the math sequence as White students. While this evidence is not conclusive, it does suggest that application of the CAP design principles may be part of an effective program to address the achievement gap. Further research that could unambiguously address the effect of acceleration on the achievement gap is certainly warranted. 41 An analysis of marginal means for students by ethnic group (Figures 10 and 11) show a pattern of improvement for accelerated students of all ethnicities that closely mirrors the overall Percent of Students Successfully Completing
Transfer-Level Course in Sequence
improvement shown in Figure 3 (See Appendices C, D and E for marginal means analysis). 100%
Comparison
80%
Accelerated
60%
42%
40%
34%
27%
21%
33%
26%
33%
26%
34%
27%
20%
0%
African
American
Asian
Hispanic
White
Other
Ethnicity
Figure 10. Estimated marginal means for percent of students completing transfer-­‐level English within tracking period by ethnic group for accelerated and comparison cohorts. 42 Percent of Students Successfully Completing
Transfer-Level Course in Sequence
100%
Comparison
80%
Accelerated
60%
49%
38%
40%
40%
41%
21%
20%
15%
15%
16%
42%
17%
0%
Asian
Black
Hispanic
White
Other
Ethnicity
Figure 11. Estimated marginal means for percent of students completing transfer-­‐level math within tracking period by ethnic group for accelerated and comparison cohorts. As Figure 10 and Figure 11 illustrate, Hispanic, Asian and African American students in accelerated classes all benefit from the overall improvement associated with acceleration. In other words, acceleration provides a significant main effect benefit for all students. Note that students in this study were tracked for either one and a half years or for two years, depending on cohort. Completion rates for both accelerated and comparison group students would be expected to continue to increase if the students were followed for a longer period of time. Restrictions
Students were tracked for either one or one and a half years after they had completed their cohort term. This timeframe may not have been sufficient to allow all students, particularly those placed very low in the traditional sequence in cohort 2 (spring 2012), to complete 43 transfer-­‐level coursework. Given the established minimal throughput rates of students placed very low in the remedial sequence and given that a third of the colleges had a floor of three levels below for their developmental pathways, the timeframe of the study most likely does not create a strong bias in the results. Even so, additional regression models were run to test the main hypothesis in additional ways. The first alternative regression model excluded students from the comparison group if they would not have had a sufficient number of primary terms (i.e., fall and spring) to complete the gatekeeper course within the time frame of the study. This modification excluded comparison group students with a current level of four levels below transfer from cohort 1 (fall 2011) and students with a current level of three or four levels below transfer from cohort 2 (spring 2012). The estimated effects of acceleration in the restricted English and math models were similar to what was observed in the full English and math models (see Appendix H and Appendix I). The convergence of the two approaches increases confidence that the results from the full models are not unduly biased due to the limited timeframe of the study (two years). Due to interest in comparing students at all levels of placement, the full models are preferable to the models that restrict the composition of the comparison group. One additional approach was taken to confirm the validity of the findings in the full models. Eligibility for enrollment in the transfer-­‐level course (i.e., completion of a course that is one-­‐
level below the accelerated course) was calculated for all students and used as an alternative dependent variable. Since this outcome takes less time to attain, only students in cohort 2 with a current level four levels below needed to be excluded. The results of those regressions were also congruent with the full models presented in the main findings section of this paper, with significant, positive effects of accelerated English and math pathways on the odds of completing the prerequisite to the transfer-­‐level gatekeeper course (see Appendix J and Appendix K). Even with these confirmatory analyses, it would still be prudent to track these students (or similar students) for at least two additional primary terms in order to confirm the findings of this evaluation. 44 Finally, it should be noted that this study did not look at actual evidence of student learning. Neither did we perform an analysis of distributions of transfer-­‐level grades. There was no effort to look at the content of courses or to perform a curricular analysis beyond the survey of CAP design principles and use of those course traits found in the COMIS, such as levels below transfer. It bears reiterating that no college made changes to the existing transferable course. Students in both accelerated and non-­‐accelerated paths met the same criteria of rigor by passing that transferable course. The increases in the throughput of students in English and math basic skills coursework do not necessarily indicate the quality or content of learning is equivalent between accelerated and comparison group students except inasmuch as passing the transfer-­‐level course (or not, as the case may be) provides a broad indication of a baseline adequacy of student learning. Discussion
This study found support for the hypothesis that accelerated pathways in English and math can improve student completion of transfer-­‐level gatekeeper courses. After controlling for an array of demographic and academic variables, we saw that, on average, accelerated pathways in the colleges studied had higher rates of throughput to transfer-­‐level coursework than did the traditional pathways. The observed effect was present even after controlling for a large number of potentially confounding variables such as current level, performance in other courses, ethnicity and other demographic characteristics. The acceleration effect tended to be stronger for math than for English, possibly because fewer students complete the transfer-­‐level gatekeeper course in the traditional math pathway than in the traditional English pathway (Figures 1 and 2). Accelerated pathways increased the odds of sequence completion for students at all levels of the basic skills sequence in English and math. The implication is that students from an array of skill ranges can be prepared for success in transfer-­‐level English or statistics via an effective acceleration implementation. Further, no specific level was associated with negative outcomes indicating that these accelerated pilots adhered to a “do no harm” principle. 45 Students in the analysis showed higher outcomes in accelerated pathways regardless of demographics such as ethnicity, gender, and financial need. Given the diversity of students and colleges in this study, it appears that effective acceleration implementations do not benefit just a particular subset of students but rather all students. In addition, no demographic group showed significantly lower outcomes in the accelerated pathway relative to the traditional pathway. This finding suggests that accelerated pathways may be a useful component in a set of strategies to reduce the significance of achievement gaps. Although acceleration’s effect size was large, it is possible that, because effect size is a relative measure vis a vis the comparison group, the acceleration effect size could diminish over time. Cohorts were tracked for either one and a half or two years which, even allowing for summer session, may not have given the lowest level comparison group students sufficient time to complete the sequence. With additional time to track comparison group students, they might be able to “catch-­‐up” to the accelerated students, at least partially. Any catch-­‐up effect is likely to be small, however, given that many of the colleges do not place students lower than three levels below and given the low throughput rate of students starting at three or four levels below in the traditional sequence. A follow-­‐up study that allowed additional time for tracking students would address these questions as well as questions about performance in subsequent coursework and whether curricular redesign continues to confer advantages all the way to terminal outcomes such as degree receipt and transfer. Given that acceleration has shown promising effects, the question of scalability comes to the fore. The concept of scalability has become a key criteria for the evaluation of successful educational interventions. For an intervention to be considered scalable it must be shown to be effective and it must also be capable of reaching a large proportion of the relevant student population (i.e., that group of students that could potentially benefit from the intervention). Public Agenda and Achieving the Dream have provided guidelines for evaluating the scale of an intervention depending on the percentage of the eligible target population that is affected: small scale projects reach between 1% to 9% of eligible students; medium from 10% to 25%; and interventions that reach more than 25% of eligible students are considered to be large 46 scale interventions.39 By this rubric, the colleges in the current study are currently operating at a small to medium scale, with the percentage of affected students ranging from 1% to 11% for math and from 4% to 15% for English. This range of scale is in keeping with that observed in a study of 13 acceleration interventions in which scale ranged from 0% to 15% during the early, pilot years of the acceleration interventions. Within two years, the acceleration interventions had ramped up to affect from 1% to 91% of eligible students, depending on the institution.40 One of the strengths of the particular form of acceleration promoted by the California Acceleration Project is that it is based on a structural, curricular redesign paired with professional development for faculty. Some CAP design principles, such as just-­‐in-­‐time remediation and contextualization, were widely adopted. While it is clear that, mathematically and logically, the compressed accelerated pathway causes a structural reduction in the number of potential loss points, it is not yet clear which design principles are necessary to support to the effective implementation of curricular redesign acceleration. It appears that at least some of the design principles must be implemented in order for effects to accrue. Indeed, there is a qualitative suggestion in the data that the greater the number of design principles employed, the larger the average acceleration effect. This study suggests that participation in an accelerated pathway leads to reliable increases in student completion of transfer-­‐level gatekeeper courses. However, there was considerable variation in the specifics of how the 16 participating colleges implemented acceleration. For English acceleration pathways in particular, those that articulated directly with the transfer-­‐
level “gatekeeper” tended to show large increases in sequence completion. English acceleration pathways that placed additional requirements such as extra courses and/or institutional filtering processes tended to show little or no acceleration effect. The scaling of English accelerated pathways may have been hampered by lack of what Coburn (2003) refers to as "fidelity to the innovation". This threat to scalability manifests itself in the large number of English implementations with no significant acceleration effect. As acceleration 39
http://www.publicagenda.org/files/CuttingEdge2.pdf Quint, Jaggars, Byndloss, & Magazinnik, (2013) 40
47 is interpreted and implemented locally, there is a risk that what is being labeled as acceleration has, in fact, lost essential characteristics of the original acceleration concept (e.g., streamlined pathways that articulate smoothly with the transfer-­‐level gatekeeper course). While creation of low-­‐acceleration pathways may have planted seeds at some colleges for participation in more effective high-­‐acceleration pathways, it is also possible that the lack of significant effects for low-­‐accelerated pathways during the pilot phase of intervention may result in decreased interest in further work in the acceleration paradigm. Math, on the other hand, showed large, positive acceleration effects with near uniformity. However, results of the implementation survey showed that acceleration had a mixed reception in the majority of math departments. In this sense, math acceleration is much more likely to suffer from a crisis of "ownership" which can affect an innovation's sustainability and spread.41 Even with good results, an innovation may not be able to survive long-­‐term without support of the departmental faculty. From an administrative or policy perspective that is concerned with the efficacious use of scarce resources in order to increase college completion, improving English and math sequence completion rates is of great interest. From the perspective of a student or a family member, the ability to move quickly and smoothly through the required English and math sequences is a boon. Effective acceleration should reduce the costs of college education in terms of both time (opportunity cost) and money (fees, books, etc.). This evaluation found strong evidence that accelerated curriculum can be developed at multiple college sites in a short period of time with good results, particularly for those accelerated pathways that articulate directly with transfer-­‐
level gatekeeper courses. And while the scale of these accelerated pathways is currently only small to medium, growth to include a high proportion of eligible students seems quite feasible, given recently noted patterns of growth of other acceleration efforts.42 The streamlined structure of accelerated pathways has been demonstrated to combat the draining effects of sequence attrition. While not all accelerated pathways showed significant, 41
Coburn (2003) Quint, Jaggars, Byndloss, & Magazinnik, (2013) 42
48 positive effects, no pathways showed significant negative effects. Seven of eight math accelerated pathways had large and significant effects on completion of the gatekeeper math course. For English, five of ten accelerated pathways (and four of the six high-­‐acceleration English pathways) showed positive and significant effects on completion of the gatekeeper English course. Taken as a whole, the effects of acceleration in both English and math were robust and large. However, the degree of acceleration (low vs. high) was a critical factor in understanding the effectiveness of the accelerated pathway, particularly in English. It also appears that, where implemented with fidelity, the California Acceleration Project design principles effectively support students with their affective and remediation needs. 49 References
Adelman, C. (2009). The toolbox revisited: Paths to degree completion from high school through college. Washington, DC: US Department of Education. Bahr, P. R. (2008). Does mathematics remediation work?: A comparative analysis of academic attainment among community college students. Research in Higher Education, 49, 420–
450. Bahr, P. R. (2013). The Deconstructive Approach to Understanding Community College Students’ Pathways and Outcomes. Community College Review, 41:2,137-­‐153. Bahr, P., Hom, W., & Perry, P. (2004). Student Readiness for Postsecondary Coursework: Developing a College-­‐Level Measure of Student Average Academic Preparation. The Journal of Applied Research in the Community College, 12:1, 7-­‐16. Bailey, T., Jeong, D., & Cho, S. (2010). Student progression through developmental sequences in community colleges (CCRC Brief). New York, NY: Columbia University, Teachers College, Community College Research Center. Barros, A. & Hirakata, V. (2003). Alternatives for logistic regression in cross-­‐sectional studies: an empirical comparison of models that directly estimate the prevalence ratio. BMC Medical Research Methodology, 3:21. Bond, L. (2009). “Toward Informative Assessment and a Culture of Evidence.” A report from Strengthening Pre-­‐collegiate Education in Community Colleges (SPECC). Stanford, California: The Carnegie Foundation for the Advancement of Teaching. Coburn, C. (2003). Rethinking scale: Moving beyond numbers to deep and lasting change. Educational Researcher, 32:6, 3-­‐12. Edgecombe, N. (2013). Accelerating the Academic Achievement of Students Referred to Developmental Education. (CCRC Brief #55). New York, NY: Community College Research Center Teachers College, Columbia University. Edgecombe, N., Jaggars, S. S., Baker, E. D., & Bailey, T. (2013). Acceleration through a holistic support model: An implementation and outcomes analysis of FastStart@CCD. New York, NY: Columbia University, Teachers College, Community College Research Center. 50 Fuenmayor, A., Hetts, J., & Rothstein, K. (2011). Assessing assessment: Evaluating models of assessment and placement. Long Beach, CA: Long Beach City College. Hayward, C. (2011). The Transfer Velocity Project: A comprehensive look at the transfer function. The Journal of Applied Research in the Community College, 18:2, 21-­‐32. Hern, K., & Snell, M. (2010). Exponential attrition and the promise of acceleration in developmental English and math. Perspectives. Berkeley, CA: Research and Planning Group. Retrieved from http://www.rpgroup.org/resources/accelerated-­‐developmental-­‐
english-­‐and-­‐math. Hern, K. & Snell, M. (2013). Toward a vision of accelerated curriculum & pedagogy: High challenge, high support classrooms for underprepared students. Oakland, CA: LearningWorks. Jaggars, S. & Stacey, G. (2014). What we know about developmental education outcomes (Research Overview). New York, NY: Columbia University, Teacher’s College, Community College Research Center. Jenkins, D., Speroni, C., Belfield, C., Jaggars, S. & Edgecombe, N. (2010). A Model for Accelerating Academic Success of Community College Remedial English Students: Is the Accelerated Learning Program (ALP) Effective and Affordable? (CCRC Working Paper No. 21). New York, NY: Columbia University, Teacher’s College, Community College Research Center. Moore, C, & Shulock, N. (2009). Student progress toward completion: Lessons from the research literature. Sacramento, CA: Institute for Higher Education Leadership & Policy. Quint, J.C., Jaggars, S., Byndloss, D.C., Magazinnik, A. (2013). Bringing developmental education to scale: Lessons from the developmental education initiative. New York, NY: MDRC. Scott-­‐Clayton, J., Crosta, P.M., & Belfield, C.R. (2012). Improving the targeting of treatment: Evidence from college remediation (NBER Working Paper No. 18457). Cambridge, MA: National Bureau of Economic Research. Soledad Cepeda, M., Boston, R., Farrar, J., & Strom, B. (2003). Comparison of Logistic Regression versus Propensity Score When the Number of Events Is Low and There Are Multiple Confounders. American Journal of Epidemiology, 158, 280-­‐287. 51 Snell, M. & Huntsman, S. (2013, February). Opening the algebra gate: A pre-­‐statistics path to transfer-­‐level math. Paper presented at the meeting of the California Mathematics Council Community Colleges – South. Anaheim, CA. Retrieved March 30, 2014 from: http://www.cmc3s.org/conferences/Spring2013/presentations/MyraSnellOpeningTheAl
gebraGate.pdf Willett, T., Hayward, C., & Dahlstrom, E. (2008). An Early Alert System for Remediation Needs of Entering Community College Students: Leveraging the California Standards Test, Report 2007036. Encinitas, CA: California Partnership for Achieving Student Success. Willett, T. (2013). Student Transcript Enhanced Project Technical Report. Berkeley, CA: The Research and Planning Group for California Community Colleges. Retrieved March 30, 2014 from http://www.rpgroup.org/sites/default/files/STEPSTechnicalReport.pdf. Zachry Rutschow, E. & Schneider, E. (2011). Unlocking the gate: What we know about improving developmental education. New York, NY: MDRC. 52 Appendix A. Definition of variables included in the logistic regression
models.
Gatekeeper completion (dependent variable): A student is considered to have completed the sequence (in math or English) when they have successfully completed (grade of “C” or better) the appropriate transfer-­‐level, gatekeeper course. Accelerated (independent variable): Students enrolled in an accelerated course in their cohort term were given a code of “1” on this variable. Students in the traditional course were coded as “0.” Current level: For students with an academic history, current level indicates the highest level attempted in the math or English sequence (as appropriate) as of the cohort term. For students without prior enrollments in the sequence, the highest placement level in English or math was used. To facilitate the interpretation of the odds ratio statistics, current level was reverse-­‐
coded: Transfer-­‐level courses were coded as “4,” with one level below coded as “3,” two levels below as ”2,” three levels below as “1,” and four levels below as “0.” GPA control: GPA control is the Grade Point Average that students received for their non-­‐
sequence related courses in the cohort term. Since a number of students were first-­‐time college students with no prior GPA, we used the GPA for courses taken concurrently in the cohort term – exclusive of grade points associated with the accelerated course or other courses in the traditional sequence. Prior successes in the related sequence: The number of English or math Taxonomy of Program43 (TOP)-­‐coded courses successfully completed with a “C” or higher, prior to entering the cohort term. First-­‐time students receive a zero in this field. 43
http://extranet.cccco.edu/Portals/1/AA/BasicSkills/TopTax6_rev0909.pdf 53 Prior non-­‐successes in the related sequence: The number of English or math TOP-­‐coded courses for which the student received a “D,” “F,” “NP,” or “W” prior to entering the cohort term. First-­‐time students receive a zero in this field. Female: Students who identified as female on their college application were coded as “1” for this variable; others were coded as “0.” Ethnicity: Comprising four binary variables – African American, Asian, Hispanic and Other – each ethnicity category is interpreted in relation to the White ethnic category, which is not explicitly included in the model. Pell grant recipient: Students who had ever received a Pell grant were given a “1” for this variable; others were coded as “0.” EOPS: Students who had ever participated in EOPS were coded as “1” for this field; others were coded as “0.” Any disability: Students who had ever been institutionally identified as possessing a disability of any type were coded as “1;” others were coded as “0.” ESL coursework: For the English completion analysis only, if a student ever took an ESL class, they received a “1” in this variable; others were coded as “0.” Not a high school graduate: If at the time of application, students indicated that they had never received a high school diploma or equivalent, they received a “1” for this field; others received a “0.” Cohort Term: Students in cohort 1 (fall 2011) received a value of “1” and those in cohort 2 (spring 2012) received a value of “2.” 54 Appendix B. Change in pseudo R2 between covariates-only logistic
regression model and model including the acceleration independent
variable for successful completion of transfer level English or math.
Discipline English Math Models Covariates only pseudo R2 Including Acceleration (IV) R2 Wald χ2 df p Covariates Only Pseudo R2 Including Acceleration (IV) R2 Wald χ2 df p Value 0.1470 0.1486 46.59 1 < 0.0005 0.1296 0.1408 273.61 1 < 0.0005 IV = Independent Variable 55 Appendix C. Average values used in regression marginal means for
successful completion of transfer level English or math.
Covariate Comparison Term Current level Female Asian Black Hispanic Other Ethnicity Not High School Graduate Disability EOPS Low Income (Pell Recipient) Any ESL Courses Prior English Nonsuccesses Prior English Successes GPA English 1.48 2.12 0.52 0.10 0.12 0.48 0.11 0.03 0.09 0.17 0.59 0.04 0.51 0.63 1.70 Math 1.50 1.90 0.54 0.11 0.11 0.37 0.14 0.03 0.10 0.11 0.47 NA 0.80 0.77 2.06 56 Appendix D. Logistic regression marginal means for successful
completion of transfer level English.
Accelerated Mean SE 30% 1.2% Variable Overall Comparison Mean SE 22% 0.3% Difference 8% Fall 2011 Cohort Spring 2012 Cohort 34% 33% 1.1% 1.1% 27% 26% 0.4% 0.4% 7% 7% Current level 4 or More Levels Below Current level 3 Levels Below Current level 2 Levels Below Current level 1 Level Below 22% 27% 32% 38% 0.9% 1.0% 1.1% 1.2% 17% 21% 25% 30% 0.5% 0.4% 0.3% 0.4% 6% 7% 7% 8% Female Male 35% 32% 1.1% 1.1% 28% 25% 0.4% 0.4% 8% 7% Asian Black Hispanic White Other Ethnicity 42% 27% 33% 33% 34% 1.6% 1.4% 1.1% 1.3% 1.5% 34% 21% 26% 26% 27% 1.1% 0.9% 0.4% 0.6% 0.9% 8% 7% 7% 7% 7% Not High School Graduate High School Graduate 19% 34% 1.8% 1.1% 14% 26% 1.3% 0.3% 5% 7% Disability No Disability 32% 34% 1.5% 1.1% 25% 26% 0.9% 0.3% 7% 7% EOPS Not EOPS 37% 33% 1.3% 1.1% 29% 25% 0.7% 0.3% 8% 7% Low Income (Pell Recipient) Not Low Income 33% 34% 1.1% 1.2% 26% 27% 0.3% 0.4% 7% 7% Any ESL Courses No ESL Courses 29% 34% 1.6% 1.1% 22% 26% 1.1% 0.3% 7% 7% One Prior English Non-­‐success No Prior English Non-­‐successes 30% 37% 1.1% 1.2% 23% 29% 0.3% 0.3% 7% 8% One Prior English Success No Prior English Successes 36% 29% 1.1% 1.0% 28% 22% 0.3% 0.3% 8% 7% GPA = 1.00 GPA = 2.00 GPA = 3.00 GPA = 4.00 21% 36% 55% 72% 1.0% 1.3% 1.4% 1.3% 15% 28% 45% 64% 0.3% 0.3% 0.6% 0.9% 6% 9% 10% 9% SE = Standard Error 57 Appendix E. Logistic regression marginal means for successful
completion of transfer level math.
Variable Overall Accelerated Mean SE 38% 2.1% Comparison Mean SE 12% 0.2% Difference 26% Fall 2011 Cohort Spring 2012 Cohort Current level 4 or More Levels Below Current level 3 Levels Below Current level 2 Levels Below Current level 1 Level Below 41% 40% 1.8% 1.7% 17% 16% 0.3% 0.3% 25% 24% 21% 1.4% 6% 0.3% 15% 30% 41% 53% 1.7% 1.9% 2.0% 10% 15% 23% 0.3% 0.2% 0.4% 20% 26% 30% Female Male 40% 41% 1.8% 1.8% 16% 17% 0.3% 0.3% 24% 25% Asian Black Hispanic White Other Ethnicity 49% 38% 40% 41% 42% 2.1% 2.1% 1.8% 1.8% 2.0% 21% 15% 15% 16% 17% 0.8% 0.8% 0.4% 0.4% 0.7% 27% 24% 24% 25% 25% Not High School Graduate High School Graduate 39% 41% 2.8% 1.7% 15% 16% 1.3% 0.2% 24% 25% Disability No Disability 38% 41% 2.0% 1.7% 14% 16% 0.7% 0.2% 23% 25% EOPS Not EOPS 44% 40% 2.0% 1.7% 18% 16% 0.7% 0.2% 26% 24% Low Income (Pell Recipient) Not Low Income 42% 40% 1.8% 1.8% 17% 16% 0.3% 0.3% 25% 24% One Prior Math Non-­‐success No Prior Math Non-­‐successes 41% 41% 1.7% 1.8% 16% 16% 0.2% 0.3% 25% 25% One Prior Math Success No Prior Math Successes 42% 36% 1.8% 1.7% 17% 13% 0.2% 0.3% 25% 23% GPA = 1.00 GPA = 2.00 GPA = 3.00 GPA = 4.00 24% 39% 55% 71% 1.6% 1.9% 2.0% 1.8% 7% 14% 24% 39% 0.2% 0.2% 0.4% 0.9% 17% 25% 31% 32% SE = Standard Error 58 Appendix F. Qualitative analysis of the relationship between design
principles and college-level acceleration effects for completing transfer
level English and math.
A qualitative analysis of the relationship between design principle use and college-­‐level acceleration effects suggests that colleges with higher use of CAP design principles realized larger gains from their acceleration pilots. This qualitative analysis is only suggestive and does not demonstrate a causal relationship. Limitations include: small sample size; small cluster sizes (the “Low principle use” group only contains one college for both the math and English group). Moreover, for English, the “Mid principle use” used seven design principles while the “High principle use” group used eight principles, indicating relatively little variability in implementation among the English pathways. Mean Accelerazon Effect by Design Principle Cluster -­‐ English Accelerazon effect 3 2.10 1.81 2 2 1.04 1 1 0 Low principle use Mid principle use High principle use 59 Mean Accelerazon Effect by Design Principle Cluster -­‐ Math 8 7.26 Accelerazon effect 7 6 5 3.90 4 3 2 1 0.94 0 Low principle use Mid principle use High principle use 60 Appendix G. Logistic regression coefficients for predicting successful
completion of transfer level English for high-acceleration colleges only.
B 0.823 -­‐0.126 -­‐0.941 0.252 -­‐0.561 -­‐0.110 0.142 0.260 -­‐0.110 0.232 -­‐0.059 -­‐0.787 0.264 -­‐0.188 0.244 0.913 -­‐3.660 Accelerated Cohort 1 Not a high school graduate Asian Black Hispanic Other ethnicity Female Any Disability EOPS participant Pell Grant recipient ESL coursework (ever) Current level Prior English non-­‐successes Prior English successes GPA control Constant S.E. 0.094 0.052 0.191 0.137 0.105 0.065 0.095 0.053 0.095 0.065 0.060 0.144 0.031 0.035 0.031 0.031 0.130 SE = Standard Error; df = degrees of freedom; Sig. = significance level Wald 76.741 5.744 24.283 3.378 28.537 2.821 2.233 23.979 1.343 12.667 0.959 29.931 74.788 29.036 61.007 889.360 788.764 df 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Sig. < 0.005 .017 < 0.005 .066 < 0.005 .093 .135 < 0.005 .246 < 0.005 .328 < 0.005 < 0.005 < 0.005 < 0.005 < 0.005 < 0.005 Odds Ratio 2.278 0.882 0.390 1.287 0.571 0.896 1.152 1.297 0.895 1.261 0.943 0.455 1.302 0.829 1.276 2.491 0.026 61 Appendix H. Logistic regression coefficients for predicting successful
completion of transfer level English with comparison group restricted to
those with sufficient primary terms to complete sequence by spring
2013.
B 0.427 -­‐0.112 -­‐0.988 0.337 -­‐0.432 -­‐0.011 0.150 0.194 -­‐0.139 0.180 -­‐0.038 -­‐0.830 0.350 -­‐0.331 0.402 0.827 -­‐3.592 Accelerated Cohort 1 Not a high school graduate Asian Black Hispanic Other ethnicity Female Any Disability EOPS participant Pell Grant recipient ESL coursework (ever) Current level Prior English non-­‐successes Prior English successes GPA control Constant S.E. 0.075 0.041 0.159 0.101 0.081 0.053 0.080 0.041 0.080 0.056 0.045 0.132 0.022 0.030 0.023 0.023 0.099 SE = Standard Error; df = degrees of freedom; Sig. = significance level Wald 32.258 7.350 38.523 11.040 28.412 0.041 3.500 22.047 3.056 10.421 0.686 39.294 242.461 120.853 307.593 1308.094 1326.518 df 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Sig. < 0.005 0.007 < 0.005 0.001 < 0.005 0.839 0.061 < 0.005 0.080 0.001 0.407 < 0.005 < 0.005 < 0.005 < 0.005 < 0.005 < 0.005 Odds Ratio 1.532 0.894 0.372 1.400 0.649 0.989 1.162 1.215 0.870 1.197 0.963 0.436 1.419 0.718 1.494 2.286 0.028 62 Appendix I. Logistic regression coefficients for predicting successful
completion of transfer level math with comparison group restricted to
those with sufficient primary terms to complete sequence by spring
2013.
B 1.274 -­‐0.007 0.396 -­‐0.034 0.462 -­‐0.157 -­‐0.072 0.139 0.010 -­‐0.113 0.073 0.177 -­‐0.027 0.329 0.724 -­‐4.348 Accelerated Cohort 1 Current level Female Asian Black Hispanic Other ethnicity Not a high school graduate Any Disability EOPS participant Pell recipient Prior math nonsuccesses Prior math successes GPA control Constant S.E. 0.107 0.053 0.037 0.045 0.073 0.095 0.058 0.072 0.139 0.087 0.077 0.049 0.018 0.023 0.028 0.144 SE = Standard Error; df = degrees of freedom; Sig. = significance level Wald 141.404 0.018 113.305 0.558 39.868 2.715 1.537 3.703 0.005 1.688 0.905 13.164 2.156 208.854 656.456 906.180 df 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Sig. <0.005 0.893 <0.005 0.455 <0.005 0.099 0.215 0.054 0.942 0.194 0.342 <0.005 0.142 <0.005 <0.005 <0.005 Odds Ratio 3.576 0.993 1.486 0.967 1.588 0.855 0.931 1.150 1.010 0.893 1.076 1.193 0.974 1.389 2.062 0.013 63 Appendix J. Logistic regression coefficients for predicting successful
completion of the English gatekeeper pre-requisite (CB 21=A).
B 0.155 0.293 -­‐0.548 0.469 -­‐0.110 0.085 0.211 0.191 -­‐0.207 0.128 -­‐0.031 -­‐0.422 0.317 -­‐0.257 0.979 0.703 -­‐2.699 Accelerated Cohort 1 Not a high school graduate Asian Black Hispanic Other ethnicity Female Any Disability EOPS participant Pell Grant recipient ESL coursework (ever) Current level Prior English non-­‐successes Prior English successes GPA control Constant S.E. 0.059 0.030 0.094 0.062 0.057 0.040 0.057 0.030 0.056 0.043 0.032 0.083 0.016 0.018 0.021 0.017 0.069 SE = Standard Error; df = degrees of freedom; Sig. = significance level Wald 7.009 92.920 34.291 57.762 3.641 4.568 13.769 40.047 13.670 8.877 0.925 26.127 401.334 204.992 2079.971 1767.911 1549.097 df 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Sig. 0.008 <0.005 <0.005 <0.005 0.056 0.033 <0.005 <0.005 <0.005 0.003 0.336 <0.005 <0.005 <0.005 <0.005 <0.005 <0.005 Odds Ratio 1.168 1.340 0.578 1.599 0.896 1.089 1.235 1.210 0.813 1.137 0.969 0.655 1.374 0.773 2.661 2.019 0.067 64 Appendix K. Logistic regression coefficients for predicting successful
completion of the math gatekeeper pre-requisite (CB 21=A).
Accelerated Cohort 1 Current level Female Asian Black Hispanic Other ethnicity Not a high school graduate Any Disability EOPS participant Pell recipient Prior math nonsuccesses Prior math successes GPA control Constant B 1.412 0.235 0.625 -­‐0.039 0.247 -­‐0.494 -­‐0.154 -­‐0.079 -­‐0.081 -­‐0.209 0.169 0.023 -­‐0.105 0.922 0.711 -­‐3.634 S.E. 0.098 0.032 0.017 0.032 0.055 0.060 0.039 0.051 0.095 0.056 0.053 0.034 0.013 0.020 0.018 0.073 SE = Standard Error; df = degrees of freedom; Sig. = significance level Wald 209.207 54.107 1428.218 1.553 19.923 67.243 15.189 2.365 0.720 13.857 10.075 0.469 60.910 2052.837 1485.307 2445.765 df 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Sig. <0.005 <0.005 <0.005 0.213 <0.005 <0.005 <0.005 0.124 0.396 <0.005 0.002 0.494 <0.005 <0.005 <0.005 <0.005 Odds Ratio 4.103 1.265 1.869 0.961 1.280 0.610 0.858 0.924 0.922 0.811 1.185 1.023 0.900 2.514 2.036 0.026 65