A Mixed-Method Evaluation of a Series of Strategies
Designed to Improve Student Success in Mathematics
and English Developmental Education Courses at
Essex County College
Dr. Leigh Bello de Castro
Essex County College
Dr. Charyl Yarbrough
The Heldrich Center for Workforce Development at
Rutgers
The Center for Academic Foundations
ECC Incoming Freshman
• Accuplacer and Companion Tests
• Results
• 85% - Math 086
• 10% - Math 092
• 5% - Math 100 or higher
ECC Center for Academic Foundations
Objectives
• Established in 2007
• Target Group: students scoring between
20 -29 Accuplacer
• 25% pass rate in Math 086 (Target Group)
• Develop Learning Communities for students in need of
Math and English developmental/transitional courses.
• Provide an array of support services.
•
•
•
•
Counseling
Early Intervention
Career Counseling
Intensive Tutoring
• Implement new teaching strategies into the curriculum.
ECC Developmental Sequence
Mth 100
Mth 092
AFM 083
Mth 086
Eng 101
Eng 092
AFE 083
*** AFM and AFE Divisional Midterms
and Finals
Eng 085
ECC Center for Academic Foundations Staff
•
•
•
•
•
•
•
Full time Coordinator
Full time Teacher Advisor Math
Full time Teacher Advisor English
Adjunct Math and English Faculty
Supplemental Instruction Leaders
Resource Specialists
Math and English Faculty Advisors
ECC Center for Academic Foundations
•
Implement new teaching strategies into the curriculum.
•
•
•
•
•
•
Block Scheduling
•
•
•
•
Customized Book
ALEKS
My Skills Lab
Early Intervention
Supplemental Instructional Leaders
Math, English and Computer Course (CSS 101)
Tutoring with SI Leader after class
2 class periods of lecture, 1 class period in Lab with ALEKS.
Stand Alone Math Sections
ESSEX COUNTY COLLEGE
INSTITUTIONAL RESEARCH
BASIC SKILL TRACKING - CAF and Regular
Spring 2009, Spring 2010 and Fall 2010 Cohorts
Criteria
Cohort
Original Cohort
Return Following Sem.
Success* Rate in
Percent W
Return Following Sem.
Success* Rate in
Size/ N Who
took MTH 092
MTH 092
MTH 092
took ENG 096
ENG 096
Percent W
ENG 096
Passed Course
FALL 2010
N
original/ passed
%
N
%
N
original/ passed
%
N
%
%
original/ passed
AFM 083
609/ 335
270
44.3%/ 80.6%
104
38.5%
25.9%
MTH 086
2202/ 1146
896
40.6%/ 78.2%
417
46.5%
19.0%
AFE 083
444/ 278
233
52.5%/ 83.8%
156
67.0%
11.6%
ENG 085
957/ 708
578
60.4%/ 81.6%
395
68.3%
11.1%
SPRING 2010
AFM 083
470/ 220
99
21.1%/ 45.0%
12
12.1%
41.4%
MTH 086
1528/ 670
348
22.8%/ 51.9%
120
34.5%
24.1%
AFE 083
305/ 159
73
23.9%/ 45.9%
34
46.6%
23.3%
ENG 085
426/ 237
92
25.6%/ 38.8%
54
58.7%
9.8%
SPRING 2009
AFM 083
302/ 91
42
13.9%/ 42.6%
9
21.4%
19.0%
MTH 086
1400/ 577
278
19.9%/ 48.1%
104
37.4%
15.5%
AFE 083
183/ 112
42
23%/ 37.5%
19
45.2%
23.8%
ENG 085
321/ 200
73
22.7%/ 36.5%
45
61.6%
11.0%
*Success rate = A,B+,B,C+,C/All Grades
The numbers in RED refer to the number or percent of students who successfully completed their first remedial course and would be
allowed to take the next level course.
For Example: For Fall 2010 AFM 083 - originally there were 609 students registered for this course. Of these, 335 passed the course and
would qualify to take
The next level mathematics course. In fact, 270, or 44.3 % of the original cohort took MTH 096. This increases to 80.6% of those qualified to
take MTH 092.
Of the 270 who took MTH 092, 104 or 38.5% successfully completed MTH 092 and 25.9% withdrew.
Center for Academic Foundations
Math 092
• Students were not succeeding in next course
• Spring 2011
• PBI Funds for CAF 09 level
•
•
•
•
•
Block scheduling
Intrusive Support
Resource Specialist
Mandatory Tutoring
Supplemental Instructional Leaders
• MTH 092 and ENG 096
• Curriculum
• Books
• Instructional Software
• Implemented the original CAF model
• Additional Staffing
• Evaluation
Purpose of Evaluation
• Describe ECC’s PBI Formula Grant Activities
• Generate actionable information on the implementation of
the PBI Formula Grant initiatives
• Assess the overall effectiveness of ECC’s PBI Formula Grant
initiatives in achieving positive outcomes
Key Research Questions
1. Do the ECC project services have the intended positive
effects on student outcomes (e.g., higher percentage of
students passing mathematics and English courses)?
2. Which of the project services have worked well and which
have not been as effective in helping students perform
better in subject courses?
3. In what ways can the services be modified to maximize their
effectiveness?
Evaluation Design
• Consisted of formative and summative components
• Utilized qualitative and quantitative methodologies
• Evaluators from multiple academic disciplines participated in the
evaluation.
• Submit interim reports
• Focused on understanding the culture and politics of the school as
part of the evaluation process
• Considered the views of all stakeholders
Qualitative Data Collection
(Spring 2011)
Method
Time Frame
Number of Participants/Sessions
Course Observations
2-3 times per course
per semester
sixteen observations
Instructor Interviews
Pre-Post Semester
Five interviews
Student Focus
Groups
Post Semester
Four groups consisting of 5-8
students
Self-Efficacy and
Subject Matter
Confidence Surveys
Pre-Post Semester
N = 42-67
Competing Priorities
and Barriers Scales
Pre Semester
Quantitative Data Collection
Administrative Data
• The Heldrich Center collects class level data from the program
administrator.
Program Description
Profile of Students
• 121 students in five developmental math sections
• 109 students in five developmental English sections
• Most students were comfortable with use of computers at
baseline
• 67% of students worked 20 hours per week or less
• Moderate levels of confidence at the beginning of the
semester
RQ 1. Do the ECC project services have the intended
positive effects on student outcomes?
90.0%
80.0%
Spring 2011
70.0%
60.0%
50.0%
Average College Wide
Success Rate
40.0%
CAF Program Success
Rate Goal
30.0%
CAF Overall Success rate
(withdrawals included)
20.0%
10.0%
0.0%
Math092
ENG096
Success rate is the number of students who passed the course divided by the number of students enrolled.
RQ 1. Do the ECC project services have the intended
positive effects on student outcomes?
Table 3. MTH-090 Grade Distribution and Success and Withdrawal Rates, by Section
Total
Total
Success Rate
Enrolle
Completing Withdrawals Withdrawal
Section
d
Withdrawals Course
Included
Rate
AF1
30
0
96.7%
0.0%
30
AF2
26
4
53.8%
15.4%
22
AF4
21
3
76.2%
14.3%
18
AF5
29
4
75.9%
13.8%
25
AF6
15
5
13.3%
33.3%
10
Table 4. ENG-090 Grade Distribution and Success and Withdrawal Rates, by Section
Total
Total
Success Rate
Enrolle
Completing Withdrawals Withdrawal
Section
d
Withdrawals Course
Included
Rate
AF1
24
1
87.5%
4.2%
23
AF2
25
8
48.0%
32.0%
17
AF4
23
2
73.9%
8.7%
21
AF5
21
0
81.0%
0.0%
21
AF6
16
2
62.5%
12.5%
14
Spring 2011
The charts below representthe grade distributionfor CAF and the Division of Mathematicsand Physics. Data reflectsa 68% successrate for
those studentstakingthe Mathcourse in comparison to their counterpartstakingthe traditionalMTH 092 whose successrate is 40%. In the
ENG 096, the CAF courseshad 69% successrate as comparedto 60% in the traditionalENG 096 courses.
MTH 092
Sect.
AF 1
AF 2
AF 4
AF 5
AF 6
TOTAL
Total
30
26
22
29
15
122
A's
4
4
2
6
0
16
B+
7
1
4
5
0
17
B
2
3
4
5
1
15
C+
5
4
0
1
0
10
C
11
2
6
5
1
25
D
0
0
0
0
0
0
F
1
8
2
3
8
22
I
0
0
0
0
0
0
W
0
4
4
4
5
17
A-C
29
14
16
22
2
83
A-F
30
22
18
25
10
105
All
30
26
22
29
15
122
Pass*
97%
64%
89%
88%
20%
79%
Success
97%
54%
73%
76%
13%
68%
25
26
23
21
16
111
3
0
1
2
0
6
2
2
3
4
7
18
5
2
3
4
2
16
4
4
0
1
0
9
7
4
10
6
1
28
0
0
0
0
0
0
2
5
4
4
4
19
0
0
0
0
0
0
2
9
2
0
2
15
21
12
17
17
10
77
23
17
21
21
14
96
25
26
23
21
16
111
91%
71%
81%
81%
71%
80%
84%
46%
74%
81%
63%
69%
1944
122
1822
170
16
154
129
17
112
140
15
125
136
10
126
232
25
207
80
0
80
634
22
612
16
0
16
407
17
390
807
83
724
1521
105
1416
1944
122
1822
53%
79%
51%
42%
68%
40%
1504
111
1393
168
6
162
152
18
134
212
16
196
141
9
132
244
28
216
15
0
15
363
19
344
18
0
18
191
15
176
917
77
840
1295
96
1199
1504
111
1393
71%
80%
70%
61%
69%
60%
ENG 096
AF 1
AF 2
AF 4
AF 5
AF 6
TOTAL
ALL
MTH 092
CAF
less CAF
ALL
ENG 096
CAF
less CAF
* Pass rate = A,B+,B,C+,C/A,B+, B, C+, C, D, and F
* Success Rate = A, B+, B, C+, C/ all grades including W and I.
RQ 2. Which of the project services have worked well
and which have not been as effective in helping
students perform better in subject courses?
• Scheduling
The CAF program successfully grouped the components of the
courses to support block scheduling.
• Each CAF course required the same group of students to meet
in three different session types (i.e., classroom, tutorial, and
lab).
• Students in CAF Math were not necessarily in CAF English with
the same group of students
RQ2. Which of the project services have worked well and which
have not been as effective in helping students perform better in
subject courses?
Supplemental Instructors
• Typically advanced ECC students or recent graduates attending
four year college
• Varied in experience and skill
• Established positive rapport with students
• Administrators matched inexperienced supplemental
instructors with lead teachers
RQ2. Which of the project services have worked well
and which have not been as effective in helping
students perform better in subject courses?
Inclusion of technology:
• Students and instructors reported that the math lectures, computer
programs, and tutorials aligned rather seamlessly
in the MTH-100 course.
• Students interpreted the grammatical issues presented by the webbased tutorial program and the supplemental instructors (in tutorial
sessions) as distinct and discrete topics, unrelated to writing and
reading comprehension topics presented by the course instructor.
• Students were more motivated to complete computer-aided
assignments in math than in English
RQ 2. Which of the project services have worked well and
which have not been as effective in helping students perform
better in subject courses?
• Program Culture
• The CAF program built a trusting environment in which
students felt comfortable seeking help from instructors and
supplemental instructors.
Table 8. CAF ENG-096 Student Survey Responses Related to General Perceptions of School Support
English - How much does this college emphasize each of the following? Please
select the most appropriate answer.
Very Little Some Quite a Bit Very Much
Encouraging you to spend significant
3.2% 12.7%
46.0%
38.1%
amounts of time studying
Providing the support you need to
help you succeed at this college
3.2%
11.1%
38.1%
47.6%
Encouraging contact among students
from different economic, social, and
racial or ethnic backgrounds
3.2%
25.8%
25.8%
45.2%
Helping you cope with your nonacademic responsibilities (work,
family, etc.)
6.5%
21.0%
33.9%
38.7%
Providing the support you need to
thrive socially
3.2%
11.3%
37.1%
48.4%
Providing the financial support you
need to afford your education
4.8%
11.1%
33.3%
50.8%
Using computers in academic work
4.8%
11.3%
24.2%
59.7%
Table 9. CAF MTH-092 Student Survey Responses Related to General Perceptions of School Support
Math - How much does this college emphasize each of the following? Please select
the most appropriate answer.
Very Little Some Quite a Bit Very Much
Encouraging you to spend significant
4.1% 16.3%
34.7%
44.9%
amounts of time studying
Providing the support you need to
2.1% 16.7%
37.5%
43.8%
help you succeed at this college
Encouraging contact among students
from different economic, social, and
racial or ethnic backgrounds
6.1%
14.3%
38.8%
40.8%
Helping you cope with your nonacademic responsibilities (work,
family, etc.)
8.2%
16.3%
36.7%
38.8%
Providing the support you need to
thrive socially
Providing the financial support you
need to afford your education
Using computers in academic work
4.2%
18.8%
35.4%
41.7%
0.0%
18.4%
30.6%
51.0%
2.0%
16.3%
32.7%
49.0%
RQ 2. Which of the project services have worked well and
which have not been as effective in helping students perform
better in subject courses?
Program Culture
• The students did not develop the sense of community that
learning communities are designed to foster and that have
been shown to have positive effects on student achievement.
• CAF instructors did not provide substantive guidance to
students related to collaborative learning.
Table 10. CAF Math and English Pre-Post Confidence and Self-Efficacy Mean
Pre-test
sample size
Pre-test
mean
Post-test
sample size
Post-test
mean
Math Confidence
English Confidence
67
71
3.18
3.51
46
59
3.41
3.55
Math Self-Efficacy
English Self-Efficacy
75
63
7.58
7.76
55
42
7.74
7.7
RQ 2. Which of the project services have worked well and
which have not been as effective in helping students perform
better in subject courses?
General Teaching Approach
• Across math and English courses, the instructors implemented
procedural teaching techniques.
• The teaching methods did not encourage students to engage
proactively in the “process” of learning.
RQ3. In what ways can the services be modified to maximize
their effectiveness?
• Administrators should provide professional development
training in “scaffolding”
• Instructors should emphasize analysis and critical thinking
• Instructors should ensure that all components of a course are
clearly connected
• Instructors should lead activities that foster interaction and
collaboration between students
• Ensure that instructors are punctual and present at each class
session
Fall 2011 – Success Rate Update
90.0%
80.0%
Average College Wide
Success Rate
70.0%
60.0%
CAF Program Success
Rate Goal
50.0%
40.0%
CAF Overall Success rate
Spring 2011
(withdrawals included)
30.0%
20.0%
CAF Overall Success rate
Fall 2011 (withdrawals
included)
10.0%
0.0%
Math092
ENG096
Success rate is the number of students who passed the course divided by the number of students enrolled.
Fall 2011 Comparison Data
course
AFE083
ENG085
AFM083
MTH086
ENG096AF
ENG096MTH092AF
MTH092-
NO_
NO_A NO_B+ NO_B NO_C+ NO_C NO_D NO_F NO_I W
A-C A-F
A-W Previous Enrolled Success Rate
45
43
85
46
96
5 109
2
74 315 429 505 431
505
62.4%
114
103
202
108
132
1 230
2
58 659 890 950 950
957
69.4%
105
54
82
39
95
13 148 33 118 375 536 687 603
694
54.6%
250
136
213
122
217
57 701 19 342 938 1696 2057 2057
2164
45.6%
20
27
17
17
19
0
21
0
17 100 121 138 138
138
72.5%
155
123
194
134
189
9 323
4 168 795 1127 1299 1299
1307
61.2%
18
4
12
8
15
1
31
2
24 57
89 115 115
115
49.6%
1390
106
48
98
71
134
67 504 19 343 457 1028 1390
1403
32.9%
Spring 2012 – Evaluation Update
Method
Time Frame
Administrative Data – Success Rates
and Withdrawal Rates
Post Semester
Course Observations
Pre-Post Semester
Instructor Interviews
Pre-Post Semester
Supplemental Instructor Interviews
Post Semester
Student Focus Groups
Post Semester
Self-Efficacy and Subject Matter
Confidence Surveys
Pre-Post Semester
Competing Priorities and Barriers
Scales
Pre Semester