Twenty Years of MALL Project Implementation: A Meta-analysis of Learning Outcomes CALICO 2014 Jack Burston Honorary Research Fellow Cyprus University of Technology MALL Learning Outcomes: 1994-2012 • Few statistically reliable measures of learning outcomes • Over 40% of MALL publications unrelated to language learning applications • Pervasive methodological shortcomings MALL Research Problems • Sample size and treatment duration Figure 1: MALL Project Duration MALL Research Problems • Sample size and treatment duration Figure 2: MALL Student Numbers MALL Research Problems Minimal requirements for statistical analysis: • 4 weeks duration (20 class days) • 10 experimental subjects = 109 MALL studies MALL Research Problems • Lack of quantifiable data: 74 studies contain no quantitative learning outcome data = 35 MALL studies with objective learning outcomes from implementations that lasted at least a month and involved ten or more experimental subjects MALL Research Problems • Research design shortcomings – Student usage of mobile devices not tracked (3) – Inadequate description of the control group (3) – Uncontrolled variables (4) – Confounding variables (3) – Inadequate statistical analysis (3) = 19 valid research studies VALID MALL Studies: General Characteristics Figure 3: General Characteristics of MALL Studies VALID MALL Studies: Duration of Study Figure 4: Duration of Study VALID MALL Studies: Group Size Figure 5: Group Size VALID MALL Studies: Year of Study Figure 6: Year of Study VALID MALL Studies: Research Findings Vocabulary Acquisition: 11/19 L2 English 8/11 SMS/MMS on basic mobile phones 3/11 PDAs and smartphones VALID MALL Studies: Research Findings 5/11 = Positive MALL advantage 3 SMS (Iran) 2 MMS (Turkey, Taiwan) 3/11 SMS Mixed results 2 Delayed post-tests contradict initial results (China, Iran) 1 No significant difference (Iran) VALID MALL Studies: Research Findings 3/11 Non SMS/MMS projects 2 PDA-based projects positive results (Taiwan) 1 Smartphone negative results (Japan) VALID MALL Studies: Research Findings Reading Competency: 4/19 All PDA-based All positive MALL results 2 L1 Spanish (Chile) 2 L2 English (Taiwan) VALID MALL Studies: Research Findings Listening/Speaking Skills: 3/19 Mobile devices used as audio recording tools All positive MALL results L2 English (Taiwan) L2 Indonesian (Australia) L2 English (Cyprus) VALID MALL Studies: Research Findings Writing Skills: 1/19 Unspecified mobile device L2 English (Taiwan) Positive results VALID MALL Studies: Conclusion • Only 35 MALL studies meet minimal conditions of duration and sample size • Only 19 MALL studies provide statistically reliable outcome data • 15 Unequivocally positive results (80%): reading, listening, speaking and writing • 4 No significant difference: vocabulary MALL Studies: Fundamental Problems • Inadequate research design • Technocentricity MALL Learning Outcomes: 1994-2012 Thank you for your attention! Any questions? MALL Learning Outcomes: Bibliography Burston, J. (2014). 20 years of MALL project implementation: A meta-analysis of learning outcomes. ReCALL Journal, 27(1), xxx-xxx. (available online) Burston, J. (2014). The reality of MALL project implementations: Still on the fringes. CALICO Journal, 31(1), 43-65. Burston, J. (2013). Mobile-Assisted Language Learning: A selected annotated bibliography of implementation studies 1994-2012. Language Learning & Technology, Special volume on Mobile-Assisted Language Learning, 17(3), 157– 224.