Collaborative Conference for Student Achievement
2014 Poster Session
Intelligent Tutor Systems Differentiate Math Lesson
Dr. Karen K. Lucas
Assistant Professor of Teacher Education
Catawba College – Salisbury, NC
Abstract
This study investigated academic gains made by students who learned Algebra 1 via a blended program that integrated an online intelligent tutor system (ITS) with face-toface instruction. Flow theory was used to explain students’ improved achievement.
Analyses of data suggested that the ITS contributed to the closing of achievement gaps.
Introduction
Background
•
Online learning has entered the K-12 classrooms in response to shrinking budgets, teacher shortages, and pressure for results.
•
Blended Learning is “any time a student learns at least in part at a supervised brick-andmortar location away from home and at least in part through online delivery with some element of student control over time, place, path, and/or pace” (Horn & Staker, 2011, p.
3).
•
Intelligent Tutor Systems (ITS) are computer applications that “use artificial intelligence
(AI) software technologies and cognitive psychology models to provide one-on-one instruction. They evaluate student performance, assess the student knowledge and skills, provide instructional feedback and select appropriate next exercises for the student” (D.
L. Johnson, 2005, p. 17).
•
Apangea Math is an ITS that was developed based on research by the U.S. Air Force
Research Laboratory and the National Science Foundation to assist students who struggle with learning mathematics.

INTELLIGENT TUTOR SYSTEMS DIFFERENTIATE MATH
Prior Research on ITS Programs
•
Practical Algebra Tutor (PAT) & the Pittsburg Urban Mathematics Project
(Koedinger, Anderson, Hadley, & Mark, 1997)
The PAT group students were online for 25 out of 180 days in the 1993-94 school year.
The PAT group scored 1 sigma better on standardized tests than the control group.
•
Computer Assisted Instruction (CAI) along with ITS compared to CAI alone
(Chien, Md.Yunus, Ali, & Baker, 2008)
Both groups were online 1 hr/day for 8 days; pre and post-test were administered.
The CAI + ITS group learned significantly more than the CAI alone group.
Difference were attributed to the personalized feedback from the ITS.
• Carnegie Learning’s Cognitive Tutor compared to ALEKS
(Barrus, Sabo, Joseph, & Atkinson, 2011)
Both groups used off-the-shelf ITS programs for 4 hrs/day for 14 days of summer school.
Both groups’ scores improved significantly over time (day 1, day 7, day 13).
Gains did not differ significantly based on which ITS program was used.
Problem
•
Achievement in Math has been a concern in America (National Mathematics Advisory
Panel, 2008).
•
In our knowledge-based economy, algebra is seen as the gatekeeper to high-level mathematics and high-paying career opportunities (Capraro & Joffrion, 2006; Ladson-
Billings, 1997; Stein, Kaufman, Sherman, & Hillen, 2011; Wu, 2001). “The high failure rate in algebra, especially among minority students, has rightfully become an issue of general social concern” (Wu, 2001, p. 1).
•
Studies have been called for regarding the effects of blended learning in K-12 education
(Chen, Lim, & Tan, 2011; Means, Toyama, Murphy, Bakie, & Jones, 2010; Patrick,
2011).
Theory
In an effort to understand motivation, Mihaly Csikszentmihalyi proposed Flow
Theory in 1975. He described flow as a state of deep concentration; for example, when artists, musicians, or athletes become intensely involved in painting or performing, they are enjoying the process of what they do (Liao, 2006; Scherer, 2002).
2

INTELLIGENT TUTOR SYSTEMS DIFFERENTIATE MATH
Flow theory says when there is a balance between challenging tasks and the skills required to meet those challenges, flow or student engagement can occur. The result of flow is an experience that combines concentration, interest, and enjoyment simultaneously
(Shernoff, Csikszentmihalyi, Schneider, & Shernoff, 2003).
ITS programs use artificial intelligence (AI), which adapts lessons to individualize instruction ensuring that the challenge of the tasks closely matches the students’ skill levels and students have opportunities to experience success (Fleisher, 2006; Huffmyer, 2008;
Johnson, 2005). Based on Flow Theory, using an ITS program is expected to improve learning flow, increasing student engagement and improving achievement (Lucas, 2012).
Purpose & Significance
•
Examine the effects of integrating an Intelligent Tutor System (ITS), Apangea Math, with face-to-face instruction on student achievement in Algebra 1.
•
Analyze the effect of the ITS on different groups of students based on their initial skill levels?
•
Document the practices teachers used and challenges they encountered while implementing this blended learning program.
• Inform administrators’ and educators’ instructional decisions regarding the implementation of ITS programs.
•
Contribute to a growing body of knowledge about instructional technology.
Research Questions
•
Does student achievement differ based on instructional program (face-to-face plus
Apangea or face-to-face alone)?
• Do achievement gains differ based on students’ initial aptitude levels?
•
What challenges do teachers encounter, and what practices do they use when implementing Apangea Math?
Method
Participants
Intervention Group
•
75 Freshmen Algebra 1 students
•
Spring 2012
•
Urban high school
•
Blended learning
•
85% Black, 12% White, 1%
Hispanic, and 1% Other
•
91% Economically disadvantaged
Comparison Group
•
99 Freshmen Algebra 1 students
•
Spring 2011
•
Same urban high school
•
Face-to-face instruction
•
89% Black, 10% White, 1%
Hispanic, and 1% Other
•
87% Economically disadvantaged
3

INTELLIGENT TUTOR SYSTEMS DIFFERENTIATE MATH
The Blended Learning Program
•
The intervention group experienced a “Rotation Model” of Blended Learning where they rotated between face-to-face and online instruction facilitated by the same teacher on a fixed schedule, working online for 40 min (out of 90) for 4 days per week.
•
The Comparison group experienced face-to-face instruction alone.
Achievement Data
•
Discovery Assessments (DA) - The intervention and comparison groups both took pre and post DA tests. Analysis of this data will reveal change in achievement over time with and without access to Apangea.
•
EOC Exams - Both groups also took the Algebra 1 EOC final exam.
Implementation Data
•
Online Apangea Statistics recorded students’ ITS usage time and lessons passed.
•
The teachers responded to open ended questions weekly regarding challenges encountered and practices used.
•
The researcher conducted observations and facilitated meetings with teachers.
Research Design
•
Quasi-Experimental research that took place in a natural school setting with intact classes
•
Mixed Methods
Quantitative data – Student achievement scores and ITS usage statistics
Qualitative data – analyzed with QDA Miner
•
Model: (2011) N -- O
1
------------- O
1
--------- O
3
(2012) N -- O
1,
---- X ----- O
1
, O
2
--- O
3
N = non-equivalent group
X = implementation of Blended Learning
O
1
= pre and post Discovery Assessments
O
2
= collection of Apangea usage statistics
O
3
= Algebra 1 EOC exams
4

INTELLIGENT TUTOR SYSTEMS DIFFERENTIATE MATH
Results
Algebra 1 EOC Exam Scores
The Intervention Group students scored significantly higher than the Comparison Group students, t(170.96) = 5.92, p < .001, d = .88
Intervention Group
M = 82.7, SD = 10.95
Comparison Group
M = 71.2, SD = 14.45
Advanced
5%
Below
Basic
10%
Advanced
17%
Proficient
11%
Below
Basic
31%
Basic
37%
Proficient
36%
Basic
53%
5
53% scored Proficient or Advanced
Discovery Assessments
16% scored Proficient or Advanced
Means and Standard Deviation for Discovery Assessments by Time and Group
Pre-test Post-test Total
DA
Scores
Comparison Group n
Intervention Group 67 42.54 (13.50) 51.99 (15.74) 47.26 (15.36)
92
M
33.67
( SD )
(10.55)
M ( SD )
38.32 (13.21)
M ( SD )
35.99 (12.15)
Total 159 37.41 (12.63) 44.08 (15.81) 40.74 (14.67)
The interaction effect between time and group on DA scores was significant,
F(1,157) = 5.25 , p < .05, partial eta
2
= .032. The intervention group’s gains (9.45 points)

INTELLIGENT TUTOR SYSTEMS DIFFERENTIATE MATH were significantly greater than the comparison group’s gains (4.65 points). The eta for this interaction effect was about .18, which is a small to medium effect size (Leech, Barrett, &
Morgan, 2008).
In order to analyze the effect of the ITS on different groups of students based on their initial aptitude, initial levels were determined by the students’ DA pre-test scores. The DA test makers group students into four categories based on their scores: less than 28% correct was considered below basic , 28 to 42% correct was basic , 43to 58% correct was proficient , and 59% or more correct was advanced .
Intervention Group Means of Correlated Variables by Initial DA Skill Level
DA Pre-test
Apangea Time (hours)
Apangea Lessons Passed
DA Gains
EOC Scores
Below Basic n = 8
20.50
15.31
17.88
24.50
77.63 n
Basic
= 25
36.12
14.80
21.27
7.46
80.04
Proficient n = 26
48.50
13.00
27.38
7.46
87.42
Advanced n = 8
67.00
15.65
32.63
5.25
94.38
The mean amount of time that students spent on Apangea during the semester based on initial aptitude level ranged from 13.00 hours to 15.65. Apangea time did not differ significantly based on initial aptitude level, F (3,62) = 1.53, p = .215. The mean number of
Apangea lessons passed per student based on initial aptitude level ranged from 17.88 to
32.63. Even though there appeared to be a consistent trend indicating that as students’ initial aptitude level went up, the number of Apangea lessons they passed increased, the ANOVA results showed Apangea lessons passed did not differ significantly based on initial aptitude,
F (3,62) = 2.30, p = .087. Thus, these aspects of the treatment (Apangea time and Apangea lessons passed) did not differ significantly based on initial aptitude.
However, DA gains did differ significantly by initial aptitude level, F (3,62) = 5.11, p
< .01. The Bonferroni post hoc test showed that the students who had an initial aptitude level of below basic had statistically greater DA gains ( M = 24.50, SD = 11.46) than those with initial aptitude levels of basic ( M = 7.46, SD = 13.67), proficient ( M = 7.46, SD = 9.74) and advanced ( M = 5.25, SD = 11.41).
6

INTELLIGENT TUTOR SYSTEMS DIFFERENTIATE MATH
Intervention Group DA Gains by Students’ Initial Skill Level
Advanced
Proficient
Pre-test
Gains
Basic
Below Basic
0 10 20 30 40 50 60 70 80
Teachers’ Challenges and Practices
The qualitative data collected about challenges and practices were coded and recoded in an iterative process to reveal themes that emerged. The data was also categorized as firstorder (external) or second-order (internal) according to the Snoeyink and Ertmer (2001) framework.
Challenges encountered and Practices used by Teachers implementing the ITS program
Challenges
First Order Challenges
•
Network issues
•
Software Limitations
•
Limited Adherence
•
Help Abuse
Second Order Challenges
•
Lack of time
•
Disbelief in the program
•
Off-task behavior
•
Student Burn-out
Practices
First Order Practices
•
Establish Protocols
•
Provide Incentives
•
Provide Personal Instruction
•
Create Learning Pathways
Second Order Practices
•
Self-Directed Learning
•
Developing Technological
Pedagogical Content Knowledge
(TPACK)
7

INTELLIGENT TUTOR SYSTEMS DIFFERENTIATE MATH
Discussion
Outcome Highlights
•
Intervention group students scored significantly higher than comparison group students on the final Algebra 1 EOC Exam.
•
The intervention group’s gains measured by DA pre and post-tests were significantly greater than the comparison group’s gains.
•
Those gains displayed by the intervention group where driven primarily by the gains made by the students who demonstrated initial aptitude levels that was below basic.
•
In this study, the students with low initial skill levels benefited more than those with higher initial skill levels from the implementation of a blended learning program with an
ITS.
•
Teachers indicated that well established network connectivity and easy access to technical support were critical elements needed to support the successful implementing of blended learning.
•
Teachers emphasized that it was important to have clear communication between administration and teachers and to define program protocols, meaning it was a good practice to explicitly spell out expectations for when and how the ITS was to be used.
•
Providing time for ongoing professional development and reflection increased teachers’
Technological Pedagogical Content Knowledge (TPACK), which increased their acceptance of and belief in the blended learning program.
•
Teachers described the ITS program as the best differentiation tool they had available to them.
Recommendations for Future Research
•
Conducting a similar study with groups that are randomly selected would represent a true experimental design, which would improve internal validity.
•
Conducting a similar study with larger groups of more diverse students would improve generalizability and generate more data regarding aptitude treatment interaction.
•
Conducting research on blended learning models of instruction that provide face-to-face and the ITS program in different proportions of time may result in different outcomes.
Conclusion
The notion that many mathematical concepts are considered prerequisites to learning
Algebra is supported in Foundations for success: The final report of the National
Mathematics Advisory Panel (2008), which states that “The coherence and sequential nature of mathematics dictate the foundational skills that are necessary for the learning of algebra”
8

INTELLIGENT TUTOR SYSTEMS DIFFERENTIATE MATH
(p. 18). This indicates that by the very nature of the discipline of mathematics, filling students’ prior learning gaps is essential prior to them learning higher level mathematical concepts. Thus, teachers are tasked with finding efficient ways to identify individual student’s learning gaps and provide differentiated instruction to address each student’s instructional needs.
When Apangea, the ITS program used in this study, employed artificial intelligence to differentiate instruction, adjusting online responses and lessons according to students’ input, students’ instructional needs were identified and addressed. The ITS program’s ability to fill learning gaps was particularly effective for the students who needed instruction in concepts that were considerably more basic than the mathematical content that teachers typically teach in Algebra 1 courses. Analyses of data in this study suggested that the ITS contributed to the closing of achievement gaps for those students whose initial aptitude level was below basic. The increase in students’ achievement seen in this study can be described as an example of Flow Theory, which says when there is a balance between challenging tasks and the skills required to meet those challenges (e.g., when a student interacts with an ITS) flow or student engagement can occur.
References
Barrus, A. L., Sabo, K. E., Joseph, S. S., & Atkinson, R. K. (2011). Evaluating adaptive, computer-based mathematics tutoring systems: A math improvement and feasibity study . Paper presented at the AERA conference. Paper retrieved from http://lsrl.lab.asu.edu/site/wp-content/uploads/2011/08/AERA-2011-Evaluating-
Adaptive-Mathematics-Tutoring-Systems.pdf
Capraro, M. M., & Joffrion, H. (2006). Algebraic equations: Can middle-school students meaningfully translate from words to mathematical symbols? Reading Psychology,
27 (2-3), 147-164.
Chen, W., Lim, C., & Tan, A. (2011). From many-to-one to one-to-many: The evolution of ubiquitous computing in education. Educational Technology, 51 (1), 21-25.
Chien, T. C., Md.Yunus, A. S., Ali, W. Z. W., & Baker, A. R. (2008). The effect of an intelligent tutoring system (ITS) on student achievement in algebraic expression.
International Journal of Instruction, 1 (2), 25-38. Retrieved from http://eiji.net/dosyalar/iji_2008_2_2.pdf
Fleisher, P. (2006). Help Is on the Way. Technology & Learning, 26 (9), 14, 16, 18.
Horn, M., & Staker, H. (2011). The rise of K-12 blended learning: Innosight Institute.
Retrieved from http://www.innosightinstitute.org/innosight/wpcontent/uploads/2011/01/The-Rise-of-K-12-Blended-Learning.pdf
Huffmyer, S. (2008). Succeeding at math. Learning and Leading with Technology, 36 (3), 34-
35.
9

INTELLIGENT TUTOR SYSTEMS DIFFERENTIATE MATH
Johnson, D. L. (2005). Computer tutors get personal. Learning & Leading with Technology,
33 (3), 14-19.
Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in
Education, 8 (1), 30-43. Retrieved from http://telearn.archivesouvertes.fr/docs/00/19/73/83/PDF/koedinger97.pdf
Ladson-Billings, G. (1997). It doesn't add up: African American students' mathematics achievement. Journal for Research in Mathematics Education, 28 , 697-708. doi:
10.2307/749638
Leech, N. L., Barrett, K. C., & Morgan, G. A. (2008). SPSS for intermediate statistics (3rd ed.). New York, NY: Taylor & Francis Group.
Lucas, K. K. (2012). Teaching and learning Algebra 1 via an intelligent tutor system: Effects on student engagement and achievement.
(Doctoral dissertation). Retrieved from http://trace.tennessee.edu/utk_graddiss/1539
Means, B., Toyama, Y., Murphy, R., Bakie, M., & Jones, K. (2010). Evaluation of evidencebased practices on online learning: A meta-analysis and review of online learning studies . Washington, DC: Center for Technology in Learning, Office of Planning,
Evaluation, and Policy Development, U.S. Department of Education.
National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National Mathematics Advisory Panel . Washington, DC: U.S. Department of
Education.
Patrick, S. (2011). New learning models: The evolution of online learning into innovative K-
12 blended programs. Educational Technology, 51 (6), 19-26.
Scherer, M. (2002). Do students care about learning? Educational Leadership, 60 (1), 12-17.
Shernoff, D. J., Csikszentmihalyi, M., Schneider, B., & Shernoff, E. S. (2003). Student engagement in high school classrooms from the perspective of Flow Theory. School
Psychology Quarterly, 18 (2), 158-157.
Snoeyink, R., & Ertmer, P. A. (2001). Thrust into technology: how veteran teachers respond.
Journal of Educational Technology Systems, 30 (1), 85-111.
Stein, M. K., Kaufman, J. H., Sherman, M., & Hillen, A. F. (2011). Algebra: A challenge at the crossroads of policy and practice. Review of Educational Research, 81 (4), 453-
492. doi: 10.3102/0034654311423025
Wu, H. (2001). How to prepare students for algebra. American Educator, 25 (2), 10-17.
10