RESEARCH REPORT
February 2023
The Impact of IXL on Math
Learning in Minnesota
Christina Schonberg, Ph.D.
IXL LEARNING
777 Mariners Island Blvd., Suite 600, San Mateo, CA 94404
650-372-4040 | www.ixl.com
Research Report
Executive Summary
_____________________________________________________________________________
IXL is an end-to-end teaching and learning solution that engages learners in grades Pre-K through 12
with a comprehensive curriculum and personalized recommendations for meeting learning goals.
Previous research has shown that IXL can have a significant positive impact on students’ academic
performance (Bashkov, 2021; Empirical Education, 2013; IXL Learning, 2017).
The goal of this study was to examine IXL usage among third- through eighth-grade students in
Minnesota and its impact on math achievement, as measured by the Minnesota Comprehensive
Assessments (MCA). Using a pretest-posttest design, we found1:
•
IXL implementation improves student achievement, especially in Title I schools. Grade
cohorts that used IXL performed better on the MCA than grade cohorts that did not use IXL.
Specifically, IXL Math cohorts saw proficiency rates2 between one and four percentage points
higher than those of cohorts not using IXL.
•
Higher levels of IXL usage are associated with better MCA Math performance. Students
performed better on the assessment when they answered more questions, reached proficiency
in more skills (SP)3, and/or spent more time on IXL.
In all figures, asterisks indicate statistical significance at the following levels: * = p < .05, ** = p < .01
Proficiency rate: percentage of students in a cohort classified as “Meets” or “Exceeds” standards on the MCA
3
SP/week = skills proficient (i.e., SmartScore ≥ 80) per week
1
2
1
Research Report
The Impact of IXL on Math Learning in Minnesota
Background
_____________________________________________________________________________
IXL is an end-to-end teaching and learning solution that engages learners in grades Pre-K through 12
with a comprehensive curriculum and personalized recommendations for meeting learning goals. It
covers four main subject areas: mathematics, English language arts (ELA), science, and social studies.
As of this writing, over 40% of students in Minnesota and over 14 million students worldwide use
IXL. IXL is deeply rooted in learning sciences research (see Bashkov et al., 2021) and engages each
student in a personalized learning experience tailored to their working level. As a result, students
work through problems that are neither too easy nor too difficult, which in turn supports their selfefficacy and motivation for continued learning.
The goal of the present study was to examine the efficacy of IXL Math across public schools in
Minnesota, as well as the effects of increased IXL usage on academic achievement. Specifically, we
investigated the following research questions:
1. Overall efficacy of IXL Math: Controlling for baseline performance, grade level, and
demographic characteristics, were math proficiency rates on the Minnesota Comprehensive
Assessments (MCA) higher among grade-level cohorts that used IXL Math, relative to comparable
cohorts that did not use IXL?
2. Efficacy of IXL Math in Title I schools: Controlling for the same covariates as in question 1, did
grade-level cohorts in Title I schools that used IXL Math perform better on the MCA Math test
than comparable cohorts in Title I schools that did not use IXL Math?
3. Usage effects of IXL Math: Controlling for the same covariates as in question 1, how did the
amount of IXL Math usage relate to grade-level cohorts’ performance on the MCA Math test?
Study Design and Methodology
_____________________________________________________________________________
DATA SOURCES
Assessment Data
All assessment and demographic data were obtained from the Minnesota Department of
Education. Math performance at pretest (2021) and posttest (2022) were measured using the MCA,
a summative assessment program that includes end-of-grade assessments in math for students in
grades 3 through 8 (the MCA is administered in high school as well; this study focuses on elementary
and middle school performance). The outcome measure was the percentage of students within a
grade-level cohort reaching proficiency in math (i.e., the proficiency rate on the math exam). More
information about the MCA can be found at https://education.mn.gov/mde/fam/tests/.
IXL Usage Data
IXL usage data were obtained from IXL’s database. When students use IXL, they complete practice
problems organized within “skills,” or specific topic areas within a subject. IXL uses a proprietary
SmartScore to indicate a student’s proficiency within a skill. The SmartScore ranges from 0-100
and increases as students answer questions correctly. However, it is not a percent correct score;
2
Research Report
a SmartScore of 100 is always possible. A SmartScore of 80 indicates proficiency in a skill, and a
SmartScore of 100 indicates mastery. IXL recommends that students should aim to reach proficiency
in at least two skills per week (SP/week; An et al., 2022).
RESEARCH QUESTIONS 1 AND 2: EFFICACY OVERALL AND IN TITLE I SCHOOLS
Study Design
To answer research question 1 (efficacy overall) and research question 2 (efficacy in Title I schools),
we used a quasi-experimental pretest-posttest control group design to compare the proficiency
rates of grade-level cohorts that used IXL during the 2021-22 school year to the proficiency rates of
cohorts that did not use IXL at all during this time (Figure 1). To control for baseline performance
and demographic characteristics, we used one-to-one propensity score matching (described in more
detail below) to match each IXL cohort to a similar cohort that did not use IXL. Matching procedures
were carried out separately for the analyses that included all schools and the analyses that included
only Title I schools.
Figure 1. Study design
Participants
Our criteria for inclusion in the treatment (i.e., IXL Math) groups were based on student IXL Math
usage rates—specifically, the number of skills in which students reached proficiency or “skills
proficient” per week (SP/week) during the study period (i.e., the 2021-22 school year). Because there
was a wide range of IXL Math usage among grade-level cohorts, we examined the impact of IXL Math
at two different usage thresholds: ≥ .5 SP/week and ≥ 1 SP/week. We defined comparison cohorts
as those in which students did not use IXL at all during the study period. After applying these criteria
and conducting propensity score matching, analytic sample sizes ranged from 1,556 to 488 study
cohorts (see Table 1).
Table 1. Efficacy Analysis Sample Sizes
3
Research Report
Propensity Score Matching
We conducted one-to-one propensity score matching without replacement using the MatchIt
package in R (Ho et al., 2011) as a preprocessing step prior to analysis. A propensity score is
the probability that a school cohort would be assigned to the treatment (i.e., IXL) group over
the comparison group and is calculated using a combination of covariates (e.g., demographic
characteristics). In the absence of random assignment, propensity scores can be used to match
comparison cohorts to treatment cohorts and create equivalent treatment and comparison groups.
In a comparison of unmatched groups (e.g., IXL cohorts compared to all non-IXL cohorts), non-IXL
cohorts could be very different from IXL cohorts on some dimensions. In contrast, using propensity
score matching allows us to compare the performance of pairs of IXL and non-IXL cohorts that
are very similar to each other. This comparison allows for a clearer attribution of the effect of IXL
and broader generalization of the findings to other cohorts that are not yet using IXL. Comparison
cohorts for the overall efficacy analysis were identified from 2,601 non-treatment cohorts (or in the
Title I analysis, 1,717 non-treatment cohorts) in the state that had non-missing pretest, posttest, and
demographic data. After matching, the resulting treatment and comparison groups had extremely
similar baseline performance and demographic characteristics (see Table A, Appendix A).
Analysis
We specified and tested separate multilevel models for each analysis (all schools and Title I schools)
to account for clustering at the school and district levels (i.e., grade cohorts within a school tend to
be more similar to each other than grade cohorts in other schools, and schools within a district tend
to be more similar to each other than schools in other districts). In the models assessing the impact
of IXL in all schools, we regressed the 2022 MCA grade-level proficiency rate on IXL cohort status
(treatment or comparison) and covariates (baseline performance and demographic characteristics).
We included covariates in these models because the absolute standardized mean differences (SMDs)
for some covariates were greater than .05 after matching, indicating that these differences needed
to be accounted for statistically. Specifically, we controlled for the following grade-level demographic
characteristics: grade level, percentage of male students, and percentage of White students. In
addition, at the school level, we controlled for percentage of economically disadvantaged students,
school size, student-teacher ratio, locale (city, suburb, town, or rural), and school Title I status. In
the models assessing the impact of IXL in Title I schools, we regressed the 2022 MCA grade-level
proficiency rate on IXL cohort status alone, as the SMDs for all covariates named above were less
than .05 after matching.
Following What Works Clearinghouse guidelines (WWC, 2020), each effect is accompanied by a test
of statistical significance using a probability (p) value and a measure of effect size. The p-value is
the probability of observing the current or more extreme data, assuming the effect is zero (Cohen,
1994). The smaller the p-value, the less likely it is that the result occurred at random; p-values less
than .05 are considered statistically significant. Effect size is reported using Hedges’ g and indicates
the difference between treatment and control groups on an outcome measure in standard deviation
units. For broad-scope educational assessments, moderate effect sizes range from about 0.10–0.20,
and effect sizes of about 0.20 or higher are considered large (Kraft, 2020; Lipsey et al., 2012). Where
applicable, we also report percentile gain, which is the expected change in IXL cohorts’ percentile
rank relative to non-IXL cohorts at the 50th percentile and is based on the effect size. Given that
these analyses are at the grade cohort level, the effect sizes should be interpreted at the grade
cohort level as well.
4
Research Report
RESEARCH QUESTION 3: USAGE EFFECTS
The goal of this set of analyses was to investigate the relationship between increased IXL usage and
MCA Math performance. We specifically examined the weekly averages of three IXL usage metrics:
questions answered, skills proficient, and time spent (in minutes).
Participants
In these analyses, we included all grade-level cohorts with any amount of IXL usage during the
study period. Prior to analysis, we identified cohorts that had IXL usage further than ±3 SD from the
mean as outliers and removed them from the sample (outlier n = 44, or 3.8% of the initial sample).
The final sample consisted of 1,122 grade-level cohorts. Descriptive statistics for cohorts’ IXL usage
during the study period can be found in Table 2.
Table 2. IXL Math Usage During the 2021-22 School Year
Analysis
As in the previous analyses, we specified and tested multilevel models, which accounted for the fact
that cohorts were clustered within schools and schools were clustered within districts. The outcome
variable—2022 MCA proficiency rate—and covariates were the same as those of the first set of
efficacy analyses (i.e., all schools). The predictors of interest (questions answered, skills proficient,
and time spent) were highly intercorrelated (smallest r = .85). Thus, to avoid multicollinearity, it was
necessary to model each variable separately. As there was no control or comparison group, Hedges’
g is not applicable; however, we report a standardized regression coefficient for each analysis to
gauge the practical significance of IXL usage relative to the effects of the covariates. As with the
previous analyses, effects should be interpreted at the grade cohort level.
Results
_____________________________________________________________________________
OVERALL EFFICACY
We found that, at both the .5 SP/week and 1 SP/week usage thresholds, grade-level cohorts that
used IXL Math outperformed comparable non-IXL cohorts on the 2022 MCA Math assessment
(Figure 2). The estimated treatment effects for IXL Math were positive and statistically significant. At
the .5 SP/week threshold, the proficiency rate was about one percentage point higher for IXL Math
cohorts relative to cohorts not using IXL (b = 1.31, p = .022); the effect size (Hedges’ g) was 0.06, which
corresponds to a percentile gain of two points. IXL Math cohorts that met the 1 SP/week threshold
saw a proficiency rate that was about two percentage points higher than that of cohorts not using
5
Research Report
IXL (b = 2.06, p = .009). The effect size was 0.10, which corresponds to a percentile gain of four
points. Full results for each model are reported in Appendix B.
Figure 2. The overall efficacy of IXL Math
TITLE I EFFICACY
Relative to the analysis of all schools, we found an even larger impact of IXL Math among grade-level
cohorts in Title I schools. At both the .5 SP/week and 1 SP/week usage thresholds, Title I grade-level
cohorts that used IXL Math outperformed comparable Title I non-IXL cohorts on the 2022 MCA
Math assessment (Figure 3). As in the analysis of all schools, the estimated treatment effects for
IXL Math were positive and statistically significant. At the .5 SP/week threshold, the proficiency rate
was close to four percentage points higher for IXL Math cohorts relative to cohorts not using IXL (b
= 3.72, p = .008); the effect size (Hedges’ g) was 0.16, which corresponds to a percentile gain of six
points. IXL Math cohorts that met the 1 SP/week threshold had a proficiency rate that was over four
percentage points higher than that of cohorts not using IXL (b = 4.20, p = .038); the effect size was
0.19, which corresponds to a percentile gain of eight points. Full results for each model are reported
in Appendix C.
Figure 3. The efficacy of IXL Math in Title I schools
6
Research Report
USAGE EFFECTS
We found that all three IXL Math usage metrics were positively and statistically significantly
associated with 2022 MCA Math proficiency rates (see Figure 4): the more that students within a
cohort used IXL Math, the better the cohort’s performance on the MCA. Based on model coefficients
and typical usage amounts, an average cohort’s MCA proficiency rate would be expected to increase
by 1.20 points for every additional 30 questions each student answered on IXL each week (p = .011,
β = 0.04); 1.41 points for every additional skill each student reached proficiency in each week (p =
.006, β = 0.04); or 2.70 points for every additional 30 minutes each student spent on IXL Math each
week (p = .016, β = 0.03). Full model results are presented in Appendix D.
Figure 4. Predicted usage effects of IXL Math
Note: SP/week = skills proficient per week.
Discussion and Recommendations
_____________________________________________________________________________
In this study, we investigated the impact of IXL Math on academic achievement in public schools in
Minnesota. We found that grade-level cohorts that used IXL Math outperformed comparable cohorts
that did not use IXL, controlling for baseline performance and demographics. This effect was even
larger among grade-level cohorts in Title I schools. Furthermore, we found that increased IXL usage
was associated with larger achievement gains. These results add to the large body of research
showing that IXL is a highly effective way to boost student learning (e.g., An, 2022a, 2022b; IXL
Learning, 2017; Schonberg, 2022; Xiong, 2022).
In the samples analyzed here, particularly those in the usage analyses, students’ usage of IXL was
slightly lower than IXL’s recommendation of reaching proficiency in two skills per week (An et al.,
2022). When grade-level cohorts reached proficiency in at least .5 skills per week (or one skill every
two weeks), they performed better than grade-level cohorts that did not use IXL, showing that IXL is
a powerful educational tool even in smaller doses. At a higher usage threshold—one skill proficient
7
Research Report
per week—IXL cohorts outperformed non-IXL cohorts by a larger amount in both the full sample
and the Title I subgroup. IXL had a larger impact when a higher amount of usage was used as the
threshold for inclusion in the treatment group, and these results are consistent with other research
showing that interventions are more effective when they are carried out with fidelity (see Finney et
al., 2021; Noell et al., 2002). We anticipate that students would experience even greater gains when
IXL is used as recommended (i.e., 2 SP/week).
Early on in the COVID-19 pandemic, education researchers identified the widening achievement gap
between high-income and low-income students as a particular area of concern (Bailey et al., 2021).
The results of the present study show that, as an end-to-end teaching and learning solution, IXL can
play a key role in helping all students recover from the educational disruptions of the past three
years. With its personalized guidance and first-of-its-kind assessment suite, IXL recognizes content
areas where students may be struggling and engages them with material at the appropriate level.
IXL meets students where they are and helps them “catch up” by providing support for relearning
missed or forgotten material. This combination of personalized learning and remediation has been
suggested as a highly effective approach for both recovering from pandemic-related learning loss
as well as boosting subsequent learning gains (Kaffenberger, 2021). In sum, IXL continues to be
a powerful platform in supporting student learning, helping students both to recover from the
pandemic and to unlock their future academic potential.
8
Research Report
References
_____________________________________________________________________________
An, X. (2022a). The impact of IXL on math and ELA learning in Arkansas (pp. 1–12).
https://www.ixl.com/materials/us/research/The_Impact_of_IXL_in_Arkansas.pdf
An, X. (2022b). The impact of IXL on math and ELA learning in South Dakota (pp. 1–12).
https://www.ixl.com/materials/us/research/The_Impact_of_IXL_in_South_Dakota.pdf
An, X., Schonberg, C., & Bashkov, B. M. (2022). IXL implementation fidelity and usage recommendations
(pp. 1–17). https://www.ixl.com/materials/us/research/IXL_Implementation_Fidelity_and_Usage_
Recommendations.pdf
Bailey, D. H., Duncan, G. J., Murnane, R. J., & Au Yeung, N. (2021). Achievement gaps in the wake of
COVID-19. Educational Researcher, 50(5), 266–275. https://doi.org/10.3102/0013189X211011237
Bashkov, B. M. (2021). Assessing the impact of IXL Math over three years: A quasi-experimental study (pp.
1–11). https://www.ixl.com/materials/us/research/IXL_Math_3-Year_QED_ESSA_Tier_2.pdf
Bashkov, B. M., Mattison, K., & Hochstein, L. (2021). IXL design principles: Core features grounded in
learning science research (pp. 1–16). https://www.ixl.com/research/IXL_Design_Principles.pdf
Cohen, J. (1994). The Earth is round (p < .05). American Psychologist, 49, 997-1003.
Empirical Education. (2013). A study of student achievement, teacher perceptions, and IXL Math (pp.
1–12). https://www.empiricaleducation.com/pdfs/IXLfr.pdf
Finney, S. J., Wells, J. B., & Henning, G. W. (2021). The Need for Program Theory and Implementation
Fidelity in Assessment Practice and Standards (Occasional Paper No. 52; pp. 1–19). University of
Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
Ho, D., Imai, K., King, G., & Stuart, E. A. (2011). MatchIt: Nonparametric preprocessing for parametric
causal inference. Journal of Statistical Software, 42(8), 1–28. https://doi.org/10.18637/jss.v042.i08
IXL Learning. (2017). Measuring the impact of IXL Math and IXL Language Arts in Minnesota schools (pp.
1–11). https://www.ixl.com/research/Impact-of-IXL-in-Minnesota.pdf
Kaffenberger, M. (2021). Modelling the long-run learning impact of the Covid-19 learning shock:
Actions to (more than) mitigate loss. International Journal of Educational Development, 81, 102326.
https://doi.org/10.1016/j.ijedudev.2020.102326
Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4),
241–253. https://doi.org/10.3102/0013189X20912798
9
Research Report
Lipsey, M. W., Puzio, K., Yun, C., Hebert, M. A., Steinka-Fry, K., Cole, M. W., Roberts, M., Anthony,
K. S., & Busick, M. D. (2012). Translating the Statistical Representation of the Effects of Education
Interventions Into More Readily Interpretable Forms (NCSER 2013-3000; pp. 1–54). National Center
for Special Education Research, Institute of Education Sciences. http://ies.ed.gov/ncser
Noell, G., Gresham, F., & Gansle, K. (2002). Does treatment integrity matter? A preliminary
investigation of instructional implementation and mathematics performance. Journal of
Behavioral Education, 11, 51–67.
Schonberg, C. (2022). The impact of IXL on Math and ELA learning in Wyoming (pp. 1–14). https://www.
ixl.com/materials/us/research/The_Impact_of_IXL_on_Math_and_ELA_Learning_in_Wyoming.pdf
What Works ClearinghouseTM Procedures Handbook, Version 4.1. (2020). What Works Clearinghouse,
Institute of Education Sciences. https://ies.ed.gov/ncee/wwc/handbooks
Xiong, Y. (2022). The impact of IXL on ELA learning in Iowa (pp. 1–10). https://www.ixl.com/materials/
us/research/Impact_of_IXL_in_Iowa.pdf
10
Research Report
Appendix A: Demographics
_____________________________________________________________________________
Table A. Demographics for IXL Math Overall Efficacy Analysis (≥ .5 SP/week)
Note. Numbers in parentheses show standard deviations. Sample demographics for the other efficacy analyses were quite
similar to the demographics reported here; therefore, we report the demographics for the sample used in the .5 SP/week
analysis across all schools as it was the broadest sample analyzed.
11
Research Report
Appendix B: Efficacy Analysis Results - All Schools
_____________________________________________________________________________
Table B1. Full IXL Math Efficacy Model: ≥ .5 SP/week
Note. Dependent variable: percentage of students reaching proficiency on the 2022 MCA Math assessment. b =
unstandardized regression coefficient, SE = standard error, CI = confidence interval, β = standardized regression coefficient.
1
4
Dummy coded; grade 3 as reference group. 2 Grand-mean centered. 3 Dummy coded; non-Title I schools as reference group.
Dummy coded; city as reference group.
12
Research Report
Table B2. Full IXL Math Efficacy Model: ≥ 1 SP/week
Note. Dependent variable: percentage of students reaching proficiency on the 2022 MCA Math assessment. b =
unstandardized regression coefficient, SE = standard error, CI = confidence interval, β = standardized regression coefficient.
1
4
Dummy coded; grade 3 as reference group. 2 Grand-mean centered. 3 Dummy coded; non-Title I schools as reference group.
Dummy coded; city as reference group.
13
Research Report
Appendix C: Efficacy Analysis Results - Title I Schools
_____________________________________________________________________________
Table C1. Full IXL Math Efficacy Model for Title I Schools: ≥ .5 SP/week
Note. Dependent variable: percentage of students reaching proficiency on the 2022 MCA Math assessment. b =
unstandardized regression coefficient, SE = standard error, CI = confidence interval, β = standardized regression coefficient.
Table C2. Full IXL Math Efficacy Model for Title I Schools: ≥ 1 SP/week
Note. Dependent variable: percentage of students reaching proficiency on the 2022 MCA Math assessment. b =
unstandardized regression coefficient, SE = standard error, CI = confidence interval, β = standardized regression coefficient.
14
Research Report
Appendix D: Usage Analysis Results
_____________________________________________________________________________
Table D1. Full IXL Math Usage Model: Questions Answered
Note. Dependent variable: percentage of students reaching proficiency on the 2022 MCA Math assessment. b =
unstandardized regression coefficient, SE = standard error, CI = confidence interval, β = standardized regression coefficient.
1
4
Dummy coded; grade 3 as reference group. 2 Grand-mean centered. 3 Dummy coded; non-Title I schools as reference group.
Dummy coded; city as reference group. 5 Average weekly amount.
15
Research Report
Table D2. Full IXL Math Usage Model: Skills Proficient
Note. Dependent variable: percentage of students reaching proficiency on the 2022 MCA Math assessment. b =
unstandardized regression coefficient, SE = standard error, CI = confidence interval, β = standardized regression coefficient.
1
4
Dummy coded; grade 3 as reference group. 2 Grand-mean centered. 3 Dummy coded; non-Title I schools as reference group.
Dummy coded; city as reference group. 5 Average weekly amount.
16
Research Report
Table D3. Full IXL Math Usage Model: Time Spent
Note. Dependent variable: percentage of students reaching proficiency on the 2022 MCA Math assessment. b =
unstandardized regression coefficient, SE = standard error, CI = confidence interval, β = standardized regression coefficient.
1
4
Dummy coded; grade 3 as reference group. 2 Grand-mean centered. 3 Dummy coded; non-Title I schools as reference group.
Dummy coded; city as reference group. 5 Average weekly amount.
17