Turner
Answer Key for Chapter Seven
Using statistics in small-scale language education research: Focus on non-parametric data
Chapter Seven
Practice Problems Answer Key
Study A: Re-analysis of Jen’s data
The data are presented in Table 6.2., but a downloadable dataset can be found on the Companion
Website (www.routledge.com/cw/turner). Go to the For Students tab and select Data Sets; you
can find the data for this problem from the Chapter Six menu (Box 6.5) or from the Chapter
Seven menu (Jen’s data). Click on the dataset and save it onto your computer using the csv
(DOS) format. Remember where you saved it so you can import it into R for analysis.
Remember from Chapter Six (pp. 141-154) that Jen wanted to determine whether learners’
perceived usefulness of authentic differed from their perceived usefulness of instructional
materials (see p. 142 for a detailed discussion of her dependent variable and data). Because the
assumption of normal distribution was not met (p. 144), we’ll now re-analyze the data using the
Wilcoxon Signed Rank statistic. The formal research hypotheses below are phrased for this nonparametric analysis.
Step 1. State hypotheses
H0: There is no statistically significant difference between the rankings of the learners’
perceptions of the usefulness of the authentic versus the instructional materials.
H1: There is a statistically significant positive difference in the rankings of learners’
perceptions of the usefulness of the authentic materials versus instructional materials.
H2: There is a statistically significant negative difference in the rankings of the learners’
perceptions of the usefulness of the authentic materials versus instructional materials.
Step 2. Set alpha
alpha = .01
Step 3. Identify the appropriate statistic for the analysis
I propose to analyze the data using the Wilcoxon Signed Rank statistic and converting the
observed value of the statistic to a z-score to interpret statistical significance because:
1) the independent variable is nominal and has two levels;
2) the comparison is of two sets of data for exactly the same set of participants;
3) there are 30 or more participants;
4) the dependent variable yields rankable data.
Step 4. Collect the data. (Retrieve from Companion Website at www.routledge.com/cw/turner.)
Step 5. Check the assumptions
1 & 2) the two levels of the independent variable are represented by the same people--there are two “perceived usefulness” scores for each person (one score is the perceived
usefulness of authentic materials and the other is the perceived usefulness score of
instructional materials);
3) there are 93 participants
1
Turner
Answer Key for Chapter Seven
Using statistics in small-scale language education research: Focus on non-parametric data
4) when I review the data, I see that they are rankable.
Step 6. Calculate the observed value of the statistic
I use R to calculate the observed value of the Wilcoxon Signed Rank statistic and the exact
probability of the observed value. (If you’d rather do the calculations with a calculator, see pages
203-206 for guidance.)
> jen.data =read.csv(file.choose (), header =T)
[import the Excel dataset in csv format; I
named the dataset “jen.data”]
> View (jen.data)
[view the dataset]
> summary(jen.data$totauth)
[calculate the mean and median]
Min. 1st Qu. Median Mean 3rd Qu. Max.
4.00 23.00 28.00 26.88 33.00 36.00
> summary(jen.data$totinst)
[calculate the mean and median]
Min. 1st Qu. Median Mean 3rd Qu. Max.
4.00 21.00 26.00 24.76 29.00 36.00
> sd(jen.data$totauth)
[1] 7.474376
[calculate standard deviation]
> sd(jen.data$totinst)
[1] 6.138612
[calculate standard deviation]
> par(mfrow = c(1,2))
[set up a space for side-by-side histograms]
> hist(jen.data$totauth, col = "tomato", breaks =10)
[make 1st histogram]
> hist(jen.data$totinst, col = "sea green 1", breaks =10)
[make 2nd histogram]
2
Turner
Answer Key for Chapter Seven
Using statistics in small-scale language education research: Focus on non-parametric data
> shapiro.test(jen.data$totauth)
[calculate Shapiro Wilk statistic—a
normal distribution isn’t required for this
analysis, but check it anyway!]
Shapiro-Wilk normality test
data: jen.data$totauth
W = 0.9195, p-value = 2.542e-05
> shapiro.test(jen.data$totinst)
[calculate Shapiro Wilk statistic—a
normal distribution isn’t required for this
analysis, but check it anyway!]
Shapiro-Wilk normality test
data: jen.data$totinst
W = 0.9519, p-value = 0.001797
> wilcox.test(jen.data$totauth, jen.data$totinst, paired = T) [calculate the observed value of the
Wilcoxon Signed Rank statistic;
perception for authentic materials
entered first]
Wilcoxon signed rank test with continuity correction
data: jen.data$totauth and jen.data$totinst
V = 2517.5, p-value = 0.005327
alternative hypothesis: true location shift is not equal to 0
3
Turner
Answer Key for Chapter Seven
Using statistics in small-scale language education research: Focus on non-parametric data
> wilcox.test(jen.data$totinst, jen.data$totauth, paired = T) [calculate the observed value of the
Wilcoxon Signed Rank statistic;
perception for instructional
materials entered first]
Wilcoxon signed rank test with continuity correction
data: jen.data$totinst and jen.data$totauth
V = 1223.5, p-value = 0.005327
alternative hypothesis: true location shift is not equal to 0
The observed value of the Wilcoxon Signed Rank statistic is the smaller of the two observed
values of V, V = 1223.5.
Step 7. Calculate the exact probability of the statistic
I simply retrieve the exact probability from the R output, so exact p = .005327.
Step 8. Compare the exact probability to alpha
The rules for interpreting exact probability are:
If exact probability ≥ alpha → accept the null hypothesis
If exact probability < alpha → reject the null hypothesis
The exact probability, p = .005227, is less than alpha, .01, so reject the null hypothesis. I accept
the first alternative hypothesis because I see that the mean perceived usefulness of the authentic
materials is higher than the mean perceived usefulness of the instructional materials.
H1: There is a statistically significant positive difference in the rankings of the learners’
perceptions of the usefulness of the authentic materials versus instructional materials.
Step 9. Make the probability statement
There’s 99% certainty the rankings of the learners’ perceptions of the usefulness of the authentic
materials are systematically higher than their perceptions of the usefulness of instructional
materials.
Step 10. Interpret the meaningfulness
There are two avenues for interpreting meaningfulness: 1) with reference to the research
question, and 2) by calculating effect size.
The research question that guided this part of Jen’s research was: What are language learners’
perceptions of the usefulness of authentic versus instructional materials?” (p. 142).
To calculate effect size, the observed value of the Wilcoxon Signed Rank statistic, V, is
converted to a z-score. The R commands for doing so are presented below—I use the command
with the values for “totinst” because it yields the lower value of V observed:
4
Turner
Answer Key for Chapter Seven
Using statistics in small-scale language education research: Focus on non-parametric data
> study.model = wilcox.test (jen.data$totinst, jen.data$totauth, paired =T)
> study.model
Wilcoxon signed rank test with continuity correction
data: jen.data$totinst and jen.data$totauth
V = 1223.5, p-value = 0.005327
alternative hypothesis: true location shift is not equal to 0
> z=qnorm(study.model$p.value/2)
[This command retrieves the value of z from
the calculation.]
>z
[1] -2.78657
[Type z to see the value of z.]
> effect.size = abs(z)/sqrt(93)
[Calculate the effect size using this formula;
93 is the number of participants.
> effect.size
[1] 0.2889538
[Type effect.size to see the effect size itself.]
Here is my interpretation of the meaningfulness of the findings:
For these 93 language learners, there is 99% certainty of a statistically significant difference in
their perception of the usefulness of authentic versus instructional materials, with the learners
reporting that authentic materials are more useful (Wilcoxon Signed rank value = 1223.5, p =
.005327; effect size = .29). The effect size is moderately strong. The ex post facto design does
not support causal interpretation of the findings, but indicates a need for further research on
learners’ perception of the usefulness of these two types of learning materials.
Study B:
1. The independent variable is +/- instruction in reading strategies.
2. The dependent variable is reading comprehension.
3. Student level of proficiency is a control variable—only high beginners are included in the
study. There is no explicit moderator variable.
4. The instruction on reading strategies is a new component added to the course; if the teacher
previously taught the course, she may have some difficulty fitting the new component into the
program. Perhaps the two teachers could meet regularly to discuss how their classes are
progressing and work together to find solutions to how to fit the new info into the existing
course.
5. External validity is threatened by the fact that only high beginners participated in the study.
This problem could be addressed by including intermediate level participants, too.
6. Descriptive statistics are presented in the chart below. The R commands are presented below
the chart.
5
Turner
Answer Key for Chapter Seven
Using statistics in small-scale language education research: Focus on non-parametric data
Experimental
Group
1
2
3
4
5
6
7
Nonexperimental
Group
47
42
39
37
37
36
32
mean
mode
median
sd
range
38.57
37.00
37.00
4.79
15 points
40.86
42.00
42.00
3.93
11 points
45
44
42
42
42
37
34
> non.exp = c(47, 42, 39, 37, 37, 36, 32)
[enter data for the non-experimental group]
> exp = c(45, 44, 42, 42, 42, 37, 34)
[enter data for the experimental group]
> summary (non.exp)
Min. 1st Qu. Median Mean 3rd Qu. Max. [calculate mean, median, and minimum &
maximum scores]
32.00 36.50 37.00 38.57 40.50 47.00
> summary (exp)
Min. 1st Qu. Median Mean 3rd Qu. Max. [calculate mean, median, and minimum &
maximum scores]
34.00 39.50 42.00 40.86 43.00 45.00
> subset(table(non.exp), (table(non.exp)==max(table(non.exp)))) [find mode]
37
2
> sd(non.exp)
[1] 4.790864
[calculate standard deviation]
> 47-32
[1] 15
[use minimum & maximum to find range]
6
Turner
Answer Key for Chapter Seven
Using statistics in small-scale language education research: Focus on non-parametric data
> subset(table(exp), (table(exp)==max(table(exp))))
42
3
[find mode]
> sd(exp)
[1] 3.933979
[calculate standard deviation]
> 45-34
[1] 11
[use minimum & maximum to find range]
> hist (non.exp, col = "dark sea green", breaks = 10)
> hist (exp, col = "medium spring green", breaks = 10)
7. Follow the steps in statistical logic....
Step 1. State hypotheses
H0: There is no statistically significant difference in the rankings of the reading
comprehension scores for the group that received reading strategies and the group that
did not.
H1: There is a statistically significant positive difference in the rankings of the reading
comprehension scores for the group that received reading strategies and the group that
did not.
7
Turner
Answer Key for Chapter Seven
Using statistics in small-scale language education research: Focus on non-parametric data
H2: There is a statistically significant negative difference in the rankings of the reading
comprehension scores for the group that received reading strategies and the group that
did not.
Step 2. Set alpha
I set alpha at .05.
Step 3. Identify the appropriate statistic for the analysis
I propose to the Wilcoxon Rank Sum statistic to analyze the data because:
1) the independent variable is nominal and has two levels;
2) the two groups include different participants;
3) the dependent variable yields rankable data.
Step 4. Collect the data
The data are presented in the table above.
Step 5. Check the assumptions
1 & 2) there are two levels of the independent variable and they are represented by
different people;
3) review of the data shows that the values are rankable.
Step 6. Calculate the observed value of the statistic.
The R commands are presented below:
> wilcox.test (non.exp, exp)
Wilcoxon rank sum test with continuity correction [The highlighted warning indicates that
“exact = FALSE” should be added so
the appropriate formula is used to
calculate the exact p-value.]
data: non.exp and exp
W = 16.5, p-value = 0.3304
alternative hypothesis: true location shift is not equal to 0
Warning message:
In wilcox.test.default(non.exp, exp) :
cannot compute exact p-value with ties
wilcox.test (non.exp, exp, exact =F)
[Calculate Wilcoxon Rank Sum value]
Wilcoxon rank sum test with continuity correction
data: non.exp and exp
W = 16.5, p-value = 0.3304
alternative hypothesis: true location shift is not equal to 0
8
Turner
Answer Key for Chapter Seven
Using statistics in small-scale language education research: Focus on non-parametric data
wilcox.test (exp, non.exp, exact =F)
[Reverse order of datasets to identify the
lower of the two observed values.
Wilcoxon rank sum test with continuity correction
data: exp and non.exp
W = 32.5, p-value = 0.3304
alternative hypothesis: true location shift is not equal to 0
Step 7. Retrieve the exact probability to consider the formal research hypotheses
Using the exact probability approach—
Retrieve the exact probability from the R output; p = .3304.
Step 8. Consider the formal research hypotheses
Using the exact probability approach—
Use the rules for comparing exact probability to alpha:
If exact probability ≥ alpha → accept the null hypothesis
If exact probability < alpha → reject the null hypothesis
The exact probability, p = .3304 is greater than alpha, .05, so accept the null hypothesis.
H0: There is no statistically significant difference in the rankings of the reading
comprehension scores for the group that received reading strategies and the group that
did not.
Step 9. Make the appropriate probability statement
There’s 95% certainty of no statistically significant difference in the rankings of the
reading comprehension scores for the group that received reading strategies and the group
that did not.
Step 10. Interpret meaningfulness
The researchers found that for this small group of participants there is no statistically significant
in the reading comprehension of learners who received instruction on reading strategies and
learners who did not receive instruction on these strategies (Wilcoxon Rank Sum W = 16.5, p =
.3304). Effect size cannot be calculated due to the small number of participants (Turner, 2014, p.
179).
Study C:
8. The independent variable is +/- study in the intermediate level Summer Intensive Arabic
Program
9. The dependent variable is Arabic oral abilities.
9
Turner
Answer Key for Chapter Seven
Using statistics in small-scale language education research: Focus on non-parametric data
10. The design is ex post facto, or if one considers participation in the Summer Intensive
Language Program to be a treatment, the study is pre-experimental (there is no legitimate
comparison group; there’s just the one group of students before and after participation in the
program).
11. Descriptive statistics & R commands
Posttest
80
60
50
48
44
26
30
R commands
1
2
3
4
5
6
7
Pretest
62
48
50
30
30
28
26
mean
39.14
48.29
> summary (pretest)
Min. 1st Qu. Median Mean 3rd Qu. Max.
26.00 29.00 30.00 39.14 49.00 62.00
> summary (posttest)
Min. 1st Qu. Median Mean 3rd Qu. Max.
26.00 37.00 48.00 48.29 55.00 80.00
> subset(table(pretest),
(table(pretest)==max(table(pretest))))
30
2
mode
> subset(table(posttest),
(table(posttest)==max(table(posttest))))
posttest
26 30 44 48 50 60 80
1 1 1 1 1 1 1
median
sd
range
30.00
14.04
36 points
48.00
18.24
54 points
See above (summary command for mean)
> sd (pretest)
[1] 14.04076
> sd(posttest)
[1] 18.23654
> 62-26
[1] 36
> 80-26
[1] 54
10
Turner
Answer Key for Chapter Seven
Using statistics in small-scale language education research: Focus on non-parametric data
12. Steps in statistical logic
Step 1. State hypotheses
H0: There is no statistically significant difference in the rankings of the Arabic oral skills
of the group before they received the SILP instruction and after they had completed the
SILP instruction.
H1: There is statistically significant positive difference in the rankings of the Arabic oral
skills of the group before they received the SILP instruction and after they had completed
the SILP instruction.
H2: There is statistically significant negative difference in the rankings of the Arabic oral
skills of the group before they received the SILP instruction and after they had completed
the SILP instruction.
Step 2. Set alpha
I set alpha at .05.
Step 3. Identify the appropriate statistic for the analysis
I propose to the Wilcoxon Signed Rank statistic to analyze the data because:
1) the independent variable is nominal and has two levels, represented by the same
people;
2) there are two sets of dependent variable data from the same group of people;
3) the dependent variable yields rankable data.
Step 4. Collect the data
The data are presented in the table above.
Step 5. Check the assumptions
1 & 2) there are two levels of the independent variable and they are represented by same
people;
3) review of the data shows that the values are rankable.
Step 6. Calculate the observed value of the statistic.
The R commands are presented below:
wilcox.test(pretest, posttest, paired=T, exact = F)
Wilcoxon signed rank test with continuity correction
data: pretest and posttest
V = 1, p-value = 0.05848
alternative hypothesis: true location shift is not equal to 0
> wilcox.test(posttest,pretest, paired =T, exact =F)
Wilcoxon signed rank test with continuity correction
11
Turner
Answer Key for Chapter Seven
Using statistics in small-scale language education research: Focus on non-parametric data
data: posttest and pretest
V = 20, p-value = 0.05848
alternative hypothesis: true location shift is not equal to 0
Step 7. Retrieve the exact probability to consider the formal research hypotheses
Using the exact probability approach—
Retrieve the exact probability from the R output; p = .05848
Step 8. Consider the formal research hypotheses
Using the exact probability approach—
Use the rules for comparing exact probability to alpha:
If exact probability ≥ alpha → accept the null hypothesis
If exact probability < alpha → reject the null hypothesis
The exact probability, p = .05848 is greater than alpha, .05, so accept the null hypothesis.
H0: There is no statistically significant difference in the rankings of the Arabic oral skills
of the group before they received the SILP instruction and after they had completed the
SILP instruction.
Step 9. Make the appropriate probability statement
There’s 95% certainty of no statistically significant difference in the rankings of the
group’s oral skills before and at the end of the SILP instruction.
Step 10. Interpret meaningfulness
The researchers found that for this small group of participants there is no statistically significant
in the oral skills of learners before and after they participated in the Summer Intensive Language
Program (fabricated data) (Wilcoxon Signed Rank V = 1, p = .05848). Effect size cannot be
calculated due to the small number of participants (Turner, 2014, p. 179).
12