Does the Use of Technological Interventions Improve Student Academic Achievement in Mathematics and Language Arts for an Identified Group of At-risk Middle School Students? Mark Neill Jerry Mathews Abstract: Computer-assisted instructional programs offer another avenue of support for student achievement. This study was conducted to investigate the influence of two computer-assisted instructional programs on the math and language arts academic achievement of two groups of academic at-risk middle school students compared to a group of students who were not at-risk in a traditional instructional program. This study utilized correlation statistics to analyze student achievement by these subjects as measured by Rausch Unit (RIT) scores on the state mandated assessment. The results of the study indicated only a small gap in academic achievement between the at-risk students receiving computer-assisted learning interventions compared to those students engaged in the traditional instructional strategies. There was a 22% increase in the number of students who met or exceeded the state-mandated growth targets in language arts and math after the first year of the computer-assisted learning interventions. About the authors: Dr. Mark Neill is an assistant professor at Idaho State University. He is coordinator of the Educational Leadership program. Dr. Jerry Mathews is an associate professor at Mississippi State University. He is the graduate coordinator and program coordinator of Educational Administration in the department of Leadership and Foundations. Introduction The implications established by the provisions of the No Child Left Behind (NCLB) legislation has left many school districts searching for avenues to address the specific learning needs of select groups of learners. Several student groups are typically identified as at-risk of failing to meet the proficiency requirements of state mandated tests. These groups usually include English as a Second Language (ESL), special needs, or economically disadvantaged students. An institutional goal of a middle school in a western state was to improve student performance in two areas on this western state’s achievement test (WSAT). The WSAT is produced and managed by the Northwest Evaluation Association (NWEA, 2005) and is utilized, in part, to assess the Adequate Yearly Progress (AYP) requirements of NCLB. This research project analyzed Rausch Units (RIT) subscale score academic data of 7th and 8th grade students from three testing periods in language arts and math. The data were analyzed and reported primarily using correlation measures. Theoretical Framework: Research Question The proliferation of computer technology in American schools has resulted in a need for more rigorous research into the effectiveness of this technology in support of instruction and student achievement. Research correlating teachers’ technology skills and their use of technology in classroom instruction with higher academic achievement has been conducted (Attewell, & Battle, 1999; Mann, Shakeshaft, Becker, & Kottkampk, 1999; Wenglinsky, 1998). Meaningful data are needed to inform decision makers regarding the capacity of technological interventions to increase student achievement and guide instructional practice. In an effort to better assess what technology does, rather than what is, this study focused on the impact of technological interventions on improving student achievement. Specifically, this research assessed the impact of selected educational technological interventions on student achievement in middle school language arts and mathematics. The key research question guiding this study was: Does the use of technological interventions improve student achievement in mathematics and language arts for an identified group of at-risk students? The theoretical framework supporting the design of this study was based on one research question. The research question was: “Is there a statistically significant correlation in the relationship between the RIT subscales in language arts and math scores of middle school students and the predictor variables: (a) grade level [7th and 8th], (b) instructional intervention [1 = Bridges, 2 = Fast ForWord, 3 = traditional instruction], (c) economic status [economically disadvantaged and noneconomically disadvantaged], (d) gender, and (e) ethnicity [white and non- white]. Even though technological interventions were the focus of this study, four other predictor variables were included in the statistical analysis. Methodology Research Design A causal-comparative research design was used in this predictive study. The independent variables were not manipulated by the researchers. In the typical public school environment learning interventions are not purposefully withheld from student participants. Therefore, this study did not utilize a true experimental design with random selection and assignment of participants. Rather, this correlation study provided results that permitted the researchers to determine the strength and direction of the relationship between different sets of data or to predict scores on one variable based on the knowledge of scores on another (Gall, Gall, & Borg, 2007). In order to address the research question, the researchers employed a multiple linear regression analysis (MLR), which according to Hair, Black, Babin, Anderson, and Tatham (2005) is an appropriate analysis when multiple independent variables are used for the purpose of predicting variation in a single dependent variable. Correlations were determined using RIT mean scale scores of the Western State Achievement Test (WSAT). In order to determine statistically significant differences in student achievement, MLR analysis was used to calculate independent t-tests and analysis of variance (ANOVA) to compare the mean scores of the 7th and 8th grade students. Data Source This research project included measures of scale scores in RIT language arts and math of 7th grade and 8th grade students from testing scheduled during the fall 2004, winter 2005, and spring 2005 WSAT testing periods. The WSAT was produced and managed by the Northwest Evaluation Association (NWEA, 2005) and was utilized, in part, to meet the federal requirements for reporting the states’ standardized student achievement data to the U.S. Department of Education. Three intact groups of students were included in the study. Two groups were identified as academically at-risk students who participated in two different technological learning interventions. The third intact group was students participating in a traditionally delivered instructional strategy. See Table 1 for the definitions of the variables used in the study. Table 1 Summary of Variables in Study Variables/Levels Description Data Obtained From WSAT math RIT Scores Mean scores of the Western State Achievement Test Western State Department of Education WSAT language arts RIT scores Mean Growth Target scores Mean scores of the Western State Achievement Test Pre-set mean scores that indicate successfully passing the WSAT subscales Western State Department of Education Western State Department of Education 7th Grade Level Students enrolled in the 7th Grade who had math and language achievement test scores Western State City Middle School 8th Grade Level Students enrolled in the 8th Grade who had math and language achievement test scores Western State City Middle School Bridges Instruction Bridges' online Learning Styles Inventory (LSI) developed by the Bridges Transitions Company Fast ForWord Instruction Educational products were developed by Scientific Learning, Inc. designed to promote academic learning Western State City Middle School Western State City Middle School Traditional academic program, delivered through regular class and coursework, exclusive of the specified technological interventions. Western State City Middle School Economic Status Students defined as economically disadvantaged or who were listed as qualifying for free or reduced lunches. Students defined as noneconomically disadvantaged were listed as students who did not qualify for free or reduced lunches. Western State City Middle School Gender Male and female students Western State City Middle School Traditional Instruction Ethnicity Students identified as minority (AfricanAmerican, Hispanic and Native American.) Students identified as non-minority (students who were not African-American, Hispanic and Native American) Participants and Data Collection Procedures Rausch Unit Scale Scores Students in the school who did not meet the individual mean growth target were identified as at-risk for the purpose of this study and were selected to participate in the Fast ForWord and Bridges computer assisted interventions. Mean Growth Target is defined as the average amount of the Rausch Unit (RIT) growth observed for students in the latest Northwest Evaluation Association (NWEA, 2005) normed study. The RIT scale can be compared, in theory, to a meter stick which is made up of equal units of measure, for example, centimeters. RIT scores are considered to be reliable and accurate indicators of achievement growth over time. Because these units do not change, the RIT score can be used with confidence to compare a student’s academic growth from one year to the next. Therefore, since the units of the RIT scale are equal in value, reliable comparisons and conclusions can be made about the academic growth of a child or a group of children (NWEA, 2005). RIT Scores Are Grade Independent Since WSAT tests are adaptive and the test items are based on student performance, not age or grade, identical scores across grades mean the same thing. For example, a seventh grader who received a score of 210 and an eighth grader who received a score of 210 are learning at the same level. This Western State City Middle School allows growth to be measured independent of grade level (NWEA, 2005). A primary objective of the research study was to determine the impact of technological interventions on the academic achievement of middle school students as determined by their RIT scores on state mandated assessments. As a result of their previous performance on state mandated assessments, approximately 100 seventh and eighth grade students who failed to meet the Mean Growth Target on the WSAT RIT language arts and math subscales were identified as at-risk. These students participated in the Fast ForWord and Bridges technology program. FastForWord Learning Intervention Program A group of at-risk students were selected to participate in the FastForWord learning intervention program. Fast ForWord products were developed by Scientific Learning, Inc. to strengthen the cognitive skills of brainbased learning maps including, (a) memory, (b) attention, (c) processing, and (d) sequencing. According to the developers, Fast ForWord products are designed to promote academic learning success. Scientific Learning, Inc. (2004) promotes the concept that strengthening these skills results in improved critical language and reading. Bridges Learning Intervention Program This research project also included a different group of at-risk students who participated in a second technological intervention called Bridges. This program intended to provide assistance to students in the development and improvement of study skills and academic dispositions. Bridges' online Learning Styles Inventory (LSI) improves study habits, attitudes and behavior, motivation, and helps students get on a successful academic track (Bridges Transition Company, 2004). The LSI is used in the education community to diagnose middle school students’ unique learning styles based on an analysis of their personal preferences. Traditional Academic Program A third group of students who met or exceeded the Mean Growth Target scores and were not identified as at-risk students, participated in the traditional academic program as defined by the state standards in math and language arts. The traditional academic program was delivered through regular class and coursework, exclusive of the specified technological interventions. Approximately 400 students participated in the traditional academic program. Data Analysis Several steps were taken with the data a priori to help ensure the integrity of the study. Data were adequately inspected for any missing values with no problems noted. The following assumptions, as recommended by Hair et al. (2005), were checked: linearity of the phenomena measured, constant variance of the error terms, independence of the error terms, collinearity, and normality of the error term distribution. Collinearity, according to Hair et al. (2005), refers to the fact that a predictor variable is highly correlated with another predictor variable. The variance inflation factor (VIF) was examined to ensure correlation models did not exceed a VIF value of 10. Reported as Pearson’s R2, correlation coefficients were produced using the simultaneous solution multiple linear regression (MLR) analysis procedure. A simultaneous solution was used because the researchers wanted to predict the dependent variable based on all independent variables of interest. According to Hair et al. (2005), MLR allows researchers to determine a correlation between a dependent variable and the best linear combination of two or more predictor or independent variables. The correlation coefficients indicate the strength of the correlation. An F statistic from an ANOVA tests the significance of the R2. The threshold for determining significance was set a priori at an alpha level of 0.05. Beta weights of the standardized coefficients were examined to determine the unique importance of independent variables in the model. A priori assumptions of normality and homogeneity for the ANOVA were examined and satisfied (Hair et al., 2005). According to Fraenkel and Wallen (2006), when comparing only two groups, the F statistic is satisfactory to reveal whether a level of significance exists between the comparisons. An alpha level of 0.05 was set a priori for the ANOVA. The MLR analysis, ANOVA, and descriptive statistics were run using the Statistical Package for Social Sciences 12.0® (Norusis, 2003). Results The population of middle school students was examined for the testing periods fall 2004, winter 2005, and spring 2005. Descriptive statistics for the dependent variables of WSAT RIT language arts and math subscale test scores by intervention are shown in Table 2. The population of testing subjects changed from one testing cycle to another, due to subject migration (e.g., Hispanic subjects). Student enrollment fluctuated during the agricultural harvest Table 2 cycle in the western state. Therefore, subject participation differed in each of the testing cycles with the highest enrollment occurring during the peak harvest time in the fall, dropping dramatically during the winter testing period, and increasing again during the spring planting season. Descriptive Statistics of RIT Scores by Intervention Intervention Bridges Fast ForWord Traditional Subject Language Math Language Math Language Math Fall 2004 Winter 2005 Spring 2005 n M SD n M SD n M SD 79 212.10 10.88 63 212.46 8.18 61 215.62 8.86 79 218.87 13.16 63 221.74 11.52 61 225.67 11.09 23 213.17 6.96 50 202.90 9.57 51 205.67 10.53 23 224.52 11.06 50 209.62 12.21 51 215.61 10.40 414 220.21 10.57 273 211.99 8.49 379 225.80 10.67 414 230.23 14.97 273 225.08 11.76 379 239.86 12.42 Even though the traditional instruction groups had higher mean scores than the at-risk Bridges and FastForWord groups in each set of scores, it is noted that small achievement gaps existed between Bridges, FastForWord, and traditional learning groups. An MLR analysis was conducted to determine which of the computer assisted interventions by grade level accounted for a statistically significant amount of the variation in the dependent variable: WSAT language arts and math subscale test scores for each of the three testing periods. The analysis reported Rsquared values of the variation in student achievement in language arts and math on the basis of the predictor variables (see Tables 3 & 4). RIT Language Arts Subscale Analysis The ANOVA results at an alpha level of .05 were statistically significant for each of the testing periods. The researchers rejected the null hypothesis due to the fact that predictions on the RIT language arts subscale test scores can be made on a better than chance level when the predictor variables are simultaneously entered into the model. The Pearson’s R indicated moderate correlations between the dependent variable and the best linear combination of the predictor variables (see Table 3). The t-tests indicated if any of the Beta coefficients of the WSAT RIT language arts subscale were statistically significant. As shown in the Table 3 the predictors, gender, learning interventions, and economic status were statistically significant predictors of student achievement in the fall testing period. The predictor variables gender, grade level and interventions were statistically significant in the winter 2005 testing period. Gender, grade level, intervention, and economic status were statistically significant in the spring 2005 testing period (see Table 3). Table 3 Summary of Multiple Linear Regression Model Analysis for the Predictors of Student Achievement for RIT Language Arts Predictor β variable Constant .130 Gender .054 Grade level .302 Intervention -0.063 Ethnicity Economic status -0.210 R-squared value Adjusted R-square Fall 2004 t 27.37 3.17 1.31 7.31 -1.53 -5.06 p Winter 2005 β t p .000*** .002* .190 .000*** .127 .000*** 23.81 .119 .128 .200 -0.019 -0.106 Spring 2005 β t p .000*** 2.01 2.19 .045* .029* 3.38 -0.318 -1.81 .001** .750 .072 27.63 .122 .111 3.15 2.86 .434 -0.066 -0.170 11.12 -1.70 -4.36 .293 .084 .276 .082 .067 .269 000*** .002* .004* .000*** .091 .000*** *p<.05, **p<.01, ***p<.001 RIT Math Subscale Analysis The ANOVA results at an alpha level of .05 were statistically significant for each of the testing periods. The researchers rejected the null hypothesis due to the fact that predictions on the RIT math subscale test scores can be made on a better than chance level when the predictor variables are simultaneously entered into the model. The Pearson’s R indicated moderate correlations between the dependent variable and the best linear combination of the predictor variables (see Table 4). The t-tests indicated if any of the Beta coefficients of the WSAT RIT math subscale were statistically significant. As shown in the Table 4 the predictor variables gender, learning interventions, and economic status were statistically significant predictors of student achievement in the fall testing period. The predictor variables gender, grade level, and interventions were statistically significant in the winter 2005 testing period. Gender, grade level, intervention, and economic status were statistically significant in the spring 2005 testing period (see Table 4). Table 4 Summary of Multiple Linear Regression Model Analysis for the Predictors of Student Achievement for RIT Math Predictor variable Constant Gender Grade level Intervention Ethnicity Economic status R-squared value Adjusted R-square β -0.084 -0.084 .307 -0.080 -0.214 Fall 2004 t p 17.99 .000*** -2.11 5.32 7.62 -2.01 -5.80 .035* .000*** .000*** .045* .000*** .194 .186 Winter 2005 β t p -0.004 -0.183 .226 .021 -0.139 14.03 .000*** -0.070 .290 3.52 .233 -2.19 .970 .004* .001*** .739 .029* .103 .083 Spring 2005 β t p -0.128 .229 .439 -0.054 -0.164 14.03 .000*** -0.336 6.02 11.47 -0.142 -4.30 .001* .000*** .000*** .156 .000*** .305 .298 *p<.05, **p<.01, ***p<.001 Conclusions and Implications Students participating in the Fast ForWord and Bridges instructional programs were selected based on their identification as academically at-risk students. Historically, students identified as academically at-risk are more likely to achieve at a much lower level than regular education students who are not identified as academically at-risk. Three key influences, Bridges, FastForWord, and traditional instruction, were analyzed in this study using descriptive statistics and multiple linear regression. It is of noteworthy practical significance in the achievement data that the academically at-risk students involved in the Fast ForWord and Bridges interventions performed at a level that had only a small achievement gap (on average) lower than the achievement of the regular education students not identified as academically at-risk (who participated in traditional instructional interventions in this study). This study supports the conclusions reported by Mann et al. (1999) that it was the at-risk students who realized the greatest achievement gains. The conclusion for this study affirmed the key research question. Data indicated that there was an increase in the number of student’s meeting growth targets on the state mandated assessment. The overall school-wide student achievement, as measured by WSAT scores, increased by 22% following the first year of implementation of the Fast ForWord and Bridges interventions [reported by the middle school administration]. This The academically at-risk students who participated in the technology intervention programs did increase their average academic achievement above the Mean Growth Targets on WSAT language arts and math subscales achievement of the prior year. The overall increase in achievement for the middle school indicated a possible cause and effect of the technology intervention on language arts and math achievement. In addition, the teachers and administrators at the middle school attributed the success of the computer assisted learning interventions as a diagnostic tool, and used this information to improve student achievement for each of the testing periods. Implications for Future Study A planned future focus of the analysis of data collected in this project will include the effect of gender, socio- economic status, and ethnicity on middle school student language arts and math achievement. The same groups of at-risk students will be included as participants in this analysis. This future study will be conducted to link at-risk student participation in the Bridges and Fast ForWord intervention strategies with language arts and math achievement. References Attewell, P., & Battle, J. (1999). Home computers and school performance. Information Society. 15(1), 1-10. http://ehostvgw11.epnet.com/fulltext .asp?resultSetId=R00000000&hitNu m=72& booleanTerm=is%2001972243&fuzz yTerm= Bridges Transitions Company (2004). Learning styles inventory. Oroville, WA: Author. Fraenkel, J. & Wallen, N. (2006). How to design and evaluate research in education (6th ed.). McGraw-Hill, New York. Gall, M. D., Gall, J. P., & Borg, W. R. (2007). Educational research: An introduction (8thed.). Boston, MA: Allyn and Bacon. Hair, J. F., Black, B., Babin, B, Anderson, R. E., & Tatham. R. L. (2005). Multivariate data analysis (6th ed.). Upper Saddle River, NJ: PrenticeHall, Inc. Mann, D., Shakeshaft, C., Becker, J., & Kottkampk, R. (1999) West Virginia achievements gains from a state wide comprehensive instructional technology program. http://www.mff.org/pubterms.taf?fil e=http://www.mff.org/pubs/ME155. pdf Norusis, M. (2003). SPSS guide to data analysis. Upper Saddle River, NJ: Prentice-Hall, Inc. NWEA (2005). RIT scale norms for use with measures of academic progress and achievement level tests. Lake Oswego, OR: Author. Scientific Learning Corporation (2004). Fast ForWord literacy program. Oakland, CA: Author. Wenglinsky, H. (1998) Does it compute? The relationship between educational technology and student achievement in mathematics. Educational Testing Service. Princeton, New Jersey. ftp://etsis1.ets.org/pub/res/technolog. pdf