Twenty Years of
MALL Project
Implementation:
A Meta-analysis of
Learning Outcomes
CALICO 2014
Jack Burston
Honorary Research Fellow
Cyprus University of Technology
MALL Learning Outcomes:
1994-2012
• Few statistically reliable measures of
learning outcomes
• Over 40% of MALL publications unrelated
to language learning applications
• Pervasive methodological shortcomings
MALL Research Problems
• Sample size and treatment duration
Figure 1: MALL Project Duration
MALL Research Problems
• Sample size and treatment duration
Figure 2: MALL Student Numbers
MALL Research Problems
Minimal requirements for statistical
analysis:
• 4 weeks duration (20 class days)
• 10 experimental subjects
= 109 MALL studies
MALL Research Problems
• Lack of quantifiable data:
74 studies contain no quantitative
learning outcome data
= 35 MALL studies with objective
learning outcomes from
implementations that lasted at least a
month and involved ten or more
experimental subjects
MALL Research Problems
• Research design shortcomings
– Student usage of mobile devices not tracked (3)
– Inadequate description of the control group (3)
– Uncontrolled variables (4)
– Confounding variables (3)
– Inadequate statistical analysis (3)
= 19 valid research studies
VALID MALL Studies:
General Characteristics
Figure 3: General Characteristics of MALL Studies
VALID MALL Studies:
Duration of Study
Figure 4: Duration of Study
VALID MALL Studies:
Group Size
Figure 5: Group Size
VALID MALL Studies:
Year of Study
Figure 6: Year of Study
VALID MALL Studies:
Research Findings
Vocabulary Acquisition: 11/19
L2 English
8/11 SMS/MMS on basic mobile phones
3/11 PDAs and smartphones
VALID MALL Studies:
Research Findings
5/11 = Positive MALL advantage
3 SMS (Iran)
2 MMS (Turkey, Taiwan)
3/11 SMS Mixed results
2 Delayed post-tests contradict
initial results (China, Iran)
1 No significant difference (Iran)
VALID MALL Studies:
Research Findings
3/11 Non SMS/MMS projects
2 PDA-based projects positive
results (Taiwan)
1 Smartphone negative
results (Japan)
VALID MALL Studies:
Research Findings
Reading Competency: 4/19
All PDA-based
All positive MALL results
2 L1 Spanish (Chile)
2 L2 English (Taiwan)
VALID MALL Studies:
Research Findings
Listening/Speaking Skills: 3/19
Mobile devices used as audio
recording tools
All positive MALL results
L2 English (Taiwan)
L2 Indonesian (Australia)
L2 English (Cyprus)
VALID MALL Studies:
Research Findings
Writing Skills: 1/19
Unspecified mobile device
L2 English (Taiwan)
Positive results
VALID MALL Studies:
Conclusion
• Only 35 MALL studies meet minimal
conditions of duration and sample size
• Only 19 MALL studies provide
statistically reliable outcome data
• 15 Unequivocally positive results (80%):
reading, listening, speaking and writing
• 4 No significant difference: vocabulary
MALL Studies:
Fundamental Problems
• Inadequate research design
• Technocentricity
MALL Learning Outcomes:
1994-2012
Thank you for your attention!
Any questions?
MALL Learning Outcomes:
Bibliography
Burston, J. (2014). 20 years of MALL project implementation: A meta-analysis of
learning outcomes. ReCALL Journal, 27(1), xxx-xxx. (available online)
Burston, J. (2014). The reality of MALL project implementations: Still on the
fringes. CALICO Journal, 31(1), 43-65.
Burston, J. (2013). Mobile-Assisted Language Learning: A selected annotated
bibliography of implementation studies 1994-2012. Language Learning &
Technology, Special volume on Mobile-Assisted Language Learning, 17(3), 157–
224.