Planning & Research Office
Report 20150185
Requestor: Math Department
Researcher(s): Steve Blohm and Terrence Willett
Date: 6/30/15
Title: Effects of Math Tutoring
Effects of Math Tutoring
Introduction
The purpose of this study is to measure the effects of math tutoring at Cabrillo College. This is an
observational study in which we looked at student grades, success rates, and completion rates for
students who received tutoring and those who did not in all math classes in which at least one student
was tutored.
From a research point of view, it would have been better to use an experiment where we could examine
the effects of tutoring while holding other variables constant. In order to perform an experiment to
measure the effect of math tutoring, we would need to randomly select students to be tutored from a
pool of students who seek tutoring, while denying other students tutoring. However, due to practical,
ethical, and regulatory concerns about denying tutoring students to students, an experimental approach
was not possible. The direct comparisons between those receiving tutoring and those not receiving
tutoring while useful are subject to self-selection bias. To help control for bias from students selfselecting to participate in tutoring, regression and propensity score matching (PSM) techniques were
employed to equate participants and non-participants on a variety of background variables. The results
are of this study are consistent with the belief that tutoring is helpful for students.
Sample
The sample analyzed consists of students who used tutoring at the Math Learning Center (MLC) from
summer 2009 through summer 2013 along with those who were enrolled in a course where at least one
of the students used tutoring. The MLC is overseen by a math faculty member and provides free, drop in
math tutoring for Cabrillo students in a study hall setting including closed study rooms and computer
stations1. Math tutors go through a training class with regular workshops to provide general tutoring
skills and to ensure students with learning disabilities and other special needs can be accommodated.
The MLC allows students to check out math textbooks, graphing and scientific calculators, laptops,
1
https://www.cabrillo.edu/services/mlc/
1
Planning & Research Office
Report 20150185
software, audio visual materials, and “learning manipulatives” such as blocks to aid understanding of
fractions and proportions.
We looked at six courses at Cabrillo where enrollment and tutoring use were both relatively high
compared to other courses. The courses we looked at were Statistics (Math 12), Intermediate Algebra
(Math 152), Basic Algebra (Math 154), Essential Mathematics (Math 254A), Pre-calculus (Math 4), and
Calculus (Math 5A).
Summary Findings
Many students at Cabrillo have had difficulty succeeding in Intermediate Algebra. We look in detail at
the relationship between hours tutored and success in this course while controlling for other factors
that are predictive of success. We came up with an equation that indicates the number of tutoring hours
that are required on average for a particular probability of success for a given course given certain
demographic indicators. While our study indicated that students in general will have a greater chance of
success if they receive tutoring, using this equation we can come up with some concrete numbers. For
example, to have at least a 75% chance of success in intermediate algebra, an underrepresented
minority (URM) male would need, on average, 29.5 hours of tutoring in a semester or about two hours
of tutoring per week. A URM female would need 23.24 hours of tutoring for the same probability of
success.
Detailed Findings
The following figure looks at the success rates in each class for these three groups
1. Students who did not log tutoring time or log time at the MLC - Nothing
2. Students who used the MLC, but did not use tutoring – MLC Only
3. Student who used some tutoring – Tutoring
2
Planning & Research Office
Report 20150185
Figure 1: Percentage of Math Students Earning a C or Better
In every case, those who were tutored outperform those who were not. The sample sizes were large so
even small difference would show up as significant. However, in many cases the differences are
substantial. For example, pre-calculus students who were tutored succeed 58% of the time and those
who were not succeeded 48% of the time. Below we will investigate the effect of tutoring among
specific among different gender, ethnic groups, and other special populations.
3
Planning & Research Office
Report 20150185
Figure 2: Percentage of Math Students Earning a C or Better by Gender
Both female and male students perform better with tutoring than without. Female students tend to
outperform male students in all cases except for those not tutored in Calculus I. In every case other than
Math 154, Elementary Algebra, the higher success rate for female tutored students was statistically
significant at the 0.05 level of significance. For male students the higher success rate for tutored
students was statistically significant in all cases other than Math 12 (Statistics) and Math 5A (Calculus).
Chi-Square details are in Appendix A.
4
Planning & Research Office
Report 20150185
Figure 3: Percentage of Math Students Earning a C or Better by URM Status
While under-represented minority students (URM) do tend to have lower success rates compared to
other students, those URM students who receive tutoring perform better than URM students who do
not receive tutoring. For Math 152, Math 254A and Math 5A the higher success rates are statistically
significant at the 0.05 level significance. This information is based on a two-tailed significance level
where the hypothesis is whether or not success rates are different for tutored students. If we were to
instead consider these one-tailed tests where our hypothesis is whether or not tutored students have a
higher success rate then the results would be significant for all six courses2. Chi-Square details are in
Appendix A.
2
While a chi-square test is generally only a two-tailed test because the chi-square statistic is not symmetrical, in
this case our chi-square is mathematically equivalent to a Z-statistic. We can therefor take half the p-value of the
two tailed test and compare that to the 0.05 level of significance for a one-tailed test.
5
Planning & Research Office
Report 20150185
Percent of Students Enrolled in Intermediate
Algebra that Used Tutoring by Ethnicity
29.82%
25.63%
23.68%
23.34%
18.68%
24.14%
18.52%
13.45%
8.11%
American
Indian,
Alaskan
Nativ
Asian
Black NonHispanic
Filipino
Latino
Multiple
Ethnicities
Pacific
Islander
Unknown
White NonHispanic
Figure 4: Tutoring Use By Ethnicity
Of the 8,082 Students enrolled in math 152 over the years studied, 1,757 (21.74%) used tutoring.
19.25% of under-represented minority (URM) students used tutoring for intermediate algebra while
23.95% of non-URM students used tutoring. The table below shows tutoring use by ethnicity for Math
152, Intermediate Algebra. While URM students in general used less tutoring than other students, black
students used tutoring at the highest rate.
6
Planning & Research Office
Report 20150185
Figure 5: propensity score matching analysis of tutoring use
The above chart shows performance differences between students who were tutored and a comparison
group of students who were not tutored but have similar attributes based on one to many propensity
score matching (PSM). The criteria used for matching were gender, ethnicity (as a binary variable
indicating URM or not), age, and prior overall grade point average (GPA). The chart shows that on
average those who were tutored had better success rates than those who were not. The chart also
shows a 95% confidence interval for the difference. If the confidence interval does not include zero then
we can be at least 95% confident that the difference did not occur by random chance. In every case
except for Math 154, Elementary Algebra the results were statistically significant. The largest difference
in success rates between the two groups ids for Math 254, Pre-algebra.
7
Planning & Research Office
Report 20150185
Figure 6: Distribution of Hours Tutored for Math 152 Intermediate Algebra
For Math 152, Intermediate Algebra, the maximum number of hours someone was tutored for a
semester was 72 hours and the minimum was less than an hour. The mean time for tutoring was about 3
hours per semester with a standard deviation of 5.5 hours. But as can be seen in the histogram above,
the data are not normally distributed. The median for hours tutored was 1.15 hours; 50% of students
tutored received 1.15 or less hours of tutoring in the semester. Seventy five percent of students tutored
received 3.16 or less hours of tutoring during the semester.
Looking at these numbers separately for those who succeeded vs those who did not succeed we see the
average number of tutoring hours for those who succeed was 3.58 and the average number of tutoring
hours for those who do not succeed was 2.29. This difference is statistically significant at p < 0.001
8
Planning & Research Office
Report 20150185
Logistic Regression
We used a Logistic Regression to estimate the number of tutoring hours required to succeed
in Intermediate algebra and to attempt to at least partly control for self-selection bias.
Below we estimate the ln(odds of succeeding) using hours tutored, gender, URM status, tutoring use
and overall GPA .
𝑝 = 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑜𝑓 𝑠𝑢𝑐𝑐𝑒𝑒𝑑𝑖𝑛𝑔 𝑖𝑛 𝑀𝑎𝑡ℎ 152
𝑝
𝑙𝑛 (
) = 𝛼 + 𝛽1 𝑥1 + 𝛽2 𝑥2 + 𝛽3 𝑥3 + 𝛽4 𝑥4 + 𝛽5 𝑥5 + 𝜀
1−𝑝
𝑝
𝑙𝑛 (
) = 𝛼 + 𝑇𝑢𝑡𝑜𝑟𝑖𝑛𝑔𝐻𝑜𝑢𝑟𝑠𝑥1 + 𝑈𝑅𝑀𝑥2 + 𝐺𝑒𝑛𝑑𝑒𝑟𝑥3 + 𝐺𝑃𝐴𝑥4 + 𝑈𝑠𝑒𝑇𝑢𝑡𝑜𝑟𝑥5 + 𝜀
1−𝑝
Variables in the
Equation
Constant
Tutoring Hours
URM (Y = 1)
Gender (F = 1)
Overall GPA
Use Tutoring (Y = 1)
Beta
-0.541
0.046
-0.420
0.339
0.229
0.104
Std.
Error
0.055
0.012
0.046
0.046
0.017
0.063
Wald
95.802
16.288
83.445
54.537
172.508
2.668
df
1
1
1
1
1
1
P-Value
P < 0.001
P < 0.001
P < 0.001
P < 0.001
P < 0.001
P = 0.102
Because we are looking at the log odds of succeeding, it is difficult to interpret the Betas. However,
positive values mean that the variable has a positive influence on succeeding and negative values mean
the variable has a negative impact on succeeding. Of the variables coded dichotomously, URM status
has the largest influence.
We could use the equation below to predict the probability of success for a particular student that
student’s information for each of the variable in the equation.
𝑙𝑛 (
𝑝
) = − 0.541 + 0.046𝑥1 − 0.420𝑥2 + 0.339𝑥3 + 0.229𝑥4 + 0.104𝑥5 + 𝜀
1−𝑝
9
Planning & Research Office
Report 20150185
Alternatively we could use the equation to estimate the Number of Tutoring Hours Required to Succeed
in Intermediate Algebra
The equation below can be used to predict the ln(odds of succeeding) using hours tutored, gender, URM
status and overall GPA as a covariate.
𝑝 = 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑜𝑓 𝑠𝑢𝑐𝑐𝑒𝑒𝑑𝑖𝑛𝑔 𝑖𝑛 𝑀𝑎𝑡ℎ 152
𝑝
𝑙𝑛 (
) = 𝛼 + 𝛽1 𝑥1 + 𝛽2 𝑥2 + 𝛽3 𝑥3 + 𝛽4 𝑥4 + 𝜀
1−𝑝
𝑝
𝑙𝑛 (
) = 𝛼 + 𝑇𝑢𝑡𝑜𝑟𝑖𝑛𝑔𝐻𝑜𝑢𝑟𝑠𝑥1 + 𝑈𝑅𝑀𝑥2 + 𝐺𝑒𝑛𝑑𝑒𝑟𝑥3 + 𝐺𝑃𝐴𝑥4 + 𝜀
1−𝑝
Variables in the
Equation
Constant
Tutoring Hours
URM (Y = 1)
Gender (F = 1)
Overall GPA
𝑙𝑛 (
Beta
-0.142
0.048
-0.610
0.303
0.148
Std.
Error
0.133
0.012
0.100
0.100
0.040
Wald
1.15
17.39
37.15
9.21
13.43
df
1
1
1
1
1
P-Value
P = 0.284
P < 0.001
P < 0.001
P = 0.002
P < 0.001
𝑝
) = − 0.142 + 0.048𝑥1 − 0.610𝑥2 + 0.303𝑥3 + 0.148𝑥4 + 𝜀
1−𝑝
Note: Sample only includes those who used tutoring.
10
Planning & Research Office
Report 20150185
If we want to estimate the number of tutoring hours required for success for a group, it is simpler to
exclude GPA (a continuous variable) as a covariate. Here we estimate the ln(odds of succeeding) using
hours tutored, gender, and URM status.
𝑝 = 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑜𝑓 𝑠𝑢𝑐𝑐𝑒𝑒𝑑𝑖𝑛𝑔 𝑖𝑛 𝑀𝑎𝑡ℎ 152
𝑝
𝑙𝑛 (
) = 𝛼 + 𝛽1 𝑥1 + 𝛽2 𝑥2 + 𝛽3 𝑥3 + 𝜀
1−𝑝
𝑝
𝑙𝑛 (
) = 𝛼 + 𝑇𝑢𝑡𝑜𝑟𝑖𝑛𝑔𝐻𝑜𝑢𝑟𝑠𝑥1 + 𝑈𝑅𝑀𝑥2 + 𝐺𝑒𝑛𝑑𝑒𝑟𝑥3 + 𝜀
1−𝑝
Variables in the Equation
Constant
Tutoring Hours
URM (Y = 1)
Gender (F = 1)
𝑙𝑛 (
Beta
0.213
0.052
-0.645
0.322
Std. Error
0.091
0.012
0.099
0.099
Wald
5.499
20.32
42.169
10.52
df
1
1
1
1
P-Value
p = 0.019
p < 0.001
p < 0.001
p = 0.001
𝑝
) = 0.213 + 0.052𝑥1 − 0.645𝑥2 + 0.322𝑥3 + 𝜀
1−𝑝
Note: Sample only includes those who used tutoring.
11
Planning & Research Office
Report 20150185
Now we can estimate the number of tutoring hours required for success in Math 152 for a
particular population of interest. For example, if we want to know the number of hours of
tutoring estimated for under-represented minority male success in Intermediate Algebra we
could use the equation below.
The equation for an under-represented minority male simplifies to:
𝑝
𝑙𝑛 (
) = − 0.432 + 0.052𝑥1 + 𝜀
1−𝑝
•
•
•
•
•
•
Among the 1,467 Male URM students who were not tutored, 560 succeeded (38%)
Among the 316 Male URM students who were tutored at all, 129 succeeded (41%)
For p = 0.5 we estimate that a URM male needs 8.3 hours of tutoring (about 0.5 hours
per week)
29 Male URM students were tutored for at least 8.3 hours and 17 succeeded (59%)
For p = 0.75 we estimate that a URM male needs 29.5 hours of tutoring (about 2 hours
per week)
The two URM male students who had at least 29.5 hours of tutoring both succeeded.
12
Planning & Research Office
Report 20150185
English Placement Scores
Other covariates such as English Placement scores could be used. However, comparing English
Placement scores for those who used tutoring and those who do not use tutoring we see no significant
difference.
Used
Tutoring
N
No
Yes
5,134
1,271
Mean
Std.
English
Deviation
Placement
Score
45.85
8.143
45.48
8.183
Std.
Error
Mean
0.114
0.230
The difference of 0.37 is not significant (p = 0.149). Given that the sample size is over 6,000, this is
evidence that English placement scores are not related to the decision to use tutoring for Math 152,
Intermediate Algebra.
There is a significant difference between English placement scores for those who succeed in Math 152,
Intermediate Algebra vs those who do not succeed in Math 152; English placement is a significant
predictor of success in Math 152.
Success
in
Math152
N
No
Yes
3,314
3,091
Mean
Std.
English
Deviation
Placement
Score
44.57
8.111
47.07
7.996
Std.
Error
Mean
0.141
0.144
The difference of 2.5 is statistically significant (p < 0.001)
When we add English placement scores to the equation it is a significant predictor of success. However,
the fit of the equation does not change in any practical way and the sample size is reduced by almost
1,700 students when we use this information.
The primary implication is that more tutoring did appear to lead to greater success in math classes in
general. Even those students who use tutoring tend to use very little. Encouraging students to use
tutoring and to use it often would likely lead to greater success in math courses. Some of the
underrepresented minority groups used tutoring at a disproportionately lower rate than their white
counterparts. In particular, Latino, Filipino, and Pacific Islander students used tutoring at less than 80%
of the rate of use by White students. Other non-URM Asian students tended to use less tutoring as well.
No more than 30% of any ethnic group used tutoring and this speaks to the fact that all students,
regardless of their background, could probably benefit by using more tutoring.
13
Planning & Research Office
Report 20150185
Appendix A
Female Students
Course Name
Chi-Square N
df Sig. (2-sided)
MATH-12
14.661
2,593 1
0
MATH-152
17.304
4,139 1
0
MATH-154
3.089
2,936 1
0.079
MATH-254A
11.353
1,000 1
0.001
MATH-4
9.816
723
1
0.002
MATH-5A
12.188
578
1
0
Table 1: Chi-Square analysis tutored vs non-tutored female students.
Male Students
Course Name
Chi-Square N
df Sig. (2-sided)
MATH-12
1.254
1,659 1 0.263
MATH-152
18.303
3,943 1 0.000
MATH-154
13.847
2,730 1 0.000
MATH-254A
21.522
712
1 0.000
MATH-4
5.644
1,053 1 0.018
MATH-5A
2.790
1,067 1 0.095
Table 2: Chi-Square analysis tutored vs non-tutored male students.
URM Students
Course Name
Chi-Square N
df Sig. (2-sided)
MATH-12
3.146
1,637 1 0.076
MATH-152
4.842
3,802 1 0.028
MATH-154
3.174
2,901 1 0.075
MATH-254A
26.060
1,103 1 0.000
MATH-4
3.481
634
1 0.062
MATH-5A
5.446
550
1 0.020
Table 3: Chi-Square analysis tutored vs non-tutored URM students.
14