WattsSEDU7006-8-3
0
Experimental Designs I
Stephen W. Watts
Northcentral University
WattsSEDU7006-8-3
1
Experimental Designs I
Jackson (2012) Chapter Exercises
#2. A randomized ANOVA indicates an analysis of variance conducted on data where
the subjects were randomly selected for a between-subjects design. A repeated-measures
ANOVA is an analysis of variance conducted on data where the subjects are correlated in either
a within-subjects or matched-subjects design. Thus, the difference between a randomized versus
repeated measures ANOVA is the underlying design of the research study being analyzed. A
one-way ANOVA indicates an analysis of data with a single independent variable.
#4. When analyzing multiple measures, the probability of a Type I error increases. This
probability can be determined by the formula 1 – ( 1 – α )c. According to the formula, the
current probability of a Type I error on at least one of the three conditions is 1 – (1 - .05)3 = 1 –
(.95)3 = 1 - .86 = .14. The Bonferroni adjustment used the formula α / k = .05 / 3 = .017.
#6. A post hoc comparison is performed when the ANOVA indicates that at least one of
the sample means is statistically different from the other means. The post hoc comparison allows
determining which means are statistically different from which other means.
#8. In a randomized ANOVA, error variance consists mostly in individual differences
between subjects. A repeated-measures ANOVA analyzes data that removes or minimizes
individual differences. Since error variance is the denominator in the ANOVA calculation, a
smaller value results in more sensitivity and more statistical power.
#10a.
SOURCE
df
SS
MS
F
Between Groups
2
22.167
11.084
6.763
Within Groups
9
14.750
1.639
11
36.917
Total
WattsSEDU7006-8-3
2
#10b. Fcv(.05) = 4.26, Fcv(.01) = 8.02; F(2,9) = 6.763, p < .05. Stress affects the number of
illnesses in at least one group and is statistically significant at the .05 level, but not at the .01
level.
#10c.
Tukey's Post Hoc Test
Q(3,9)
MSwithin
n
HSD
0.05
3.950
1.639
4
2.528
Minimal
Minimal
Moderate
High
0.010
5.430
1.639
4.000
3.476
Moderate
-
High
1
-
3
2
-
#10d. In the present study, only the differences between the minimal and high stress
conditions is significant at alpha = .05. All other comparison conditions in the study are not
statistically significant. Those who have high levels of stress are much more likely to get sick
than those with either moderate or low levels of stress.
#10e. The effect size eta-squared uses the formula η2 = SSbetween / SStotal = 22.167 / 36.915
= 0.60, meaning that approximately 60% of the variance among the illnesses can be attributed to
the level of stress.
Average Illnesses by Stress Level
7
6
5
4
3
2
1
0
#10f.
Minimal
Moderate
High
WattsSEDU7006-8-3
3
#12a.
SOURCE
df
SS
MS
F
2
1202.313
601.157
3.974
Within Groups
14
2118.000
151.286
Total
44
3320.313
Between Groups
#12b. Fcv(.05) = 3.74, Fcv(.01) = 6.51; F(2,14) = 3.974, p < .05. The groups affect the
measurement of depression in at least one group and is statistically significant at the .05 level,
but not at the .01 level.
#12c.
Tukey's Post Hoc Test
Q(3,14)
MSwithin
n
HSD
0.05
3.700
151.286
15
11.750
Control
Control
Placebo
Drug
-
0.010
4.890
151.286
15.000
15.530
Placebo
2.93
-
Drug
12.13
9.2
-
#12d. In the present study, only the differences between the control group and the drug
group conditions is significant at alpha = .05. All other comparison conditions in the study are
not statistically significant. The drug decreases the effect of depression over the use of a placebo
or no treatment at all.
#12e. The effect size (η2) = .362, meaning that approximately 36% of the variance in
depression score is attributable to application of the drug.
WattsSEDU7006-8-3
4
Depression Scores
40
30
20
10
0
#12f.
Control
Placebo
Drug
#14a.
SOURCE
df
SS
MS
F
Subject
Between Groups
Error
Total
9
2
18
29
2.75
180.050
21.650
204.450
0.306
90.025
1.203
74.848
#14b. Fcv(.05) = 3.55, Fcv(.01) = 6.01; F(2,18) = 74.848, p < .01. At least one group of
types of pizza slices is statistically significant at both the .05 and .01 level.
#14c.
Tukey's Post Hoc Test
Q(3,18)
MSerror
n
HSD
Hand-tossed
Thick
Thin
0.05
3.610
1.203
10
1.252
0.010
4.700
1.203
10
1.630
Hand-tossed
-
Thick
1.47
-
Thin
5.77
4.3
-
#14d. In the present study, all comparisons are significant at the .05 level, and two
comparisons are significant at the .01 level. Subjects preferred thin crust pizza over both handtossed and thick crust pizza at a .01 significance level. Subjects preferred thick crust over handtossed pizza at a .05 significance level.
WattsSEDU7006-8-3
5
#14e. The effect size (η2) = .881, meaning that approximately 88% of the variance of
pizza slice choice is attributable to the type of slice.
10
Average Slices of Pizza
8
6
4
2
0
#14f.
Hand-tossed
Thick
Thin
Part I Assignment Question Answers
What is an F-ratio? Define all the technical terms in your answer. Calculation of the
F-ratio is the third and last step in performing an analysis of variance (ANOVA). To get to the
third step of an ANOVA several (three or four depending on the type of ANOVA) sums of
squares must be calculated in the first step. For a one-way randomized ANOVA these consist of
(a) “the sum of the squared deviations of each score from the grand mean” (Jackson, 2012, p.
290) or total sum of the squares, (b) “the sum of the squared deviations of each score from its
group or condition mean” (Jackson, 2012, p. 290) or within-groups sum of the squares, and (c)
“the sum of the squared deviations of each group’s mean from the grand mean, multiplied by the
number of participants in each group” (Jackson, 2012, p. 292) or between-groups sum of the
squares. A one-way repeated measures ANOVA adds a between-subjects sum of squares which
is “the sum of the squared difference scores for the mean of each subject across conditions and
the grand mean, multiplied by the number of conditions” (Jackson, 2012, p. 302). In the second
step, the sums of squares are converted into mean squared deviation scores, which are an
WattsSEDU7006-8-3
6
estimate of the variance within and between groups. Variance is the capriciousness of data due
to error, confounds, differences in subjects, and the manipulation of the independent variable.
These variance estimates are calculated by dividing each sum of the squares by the appropriate
degrees of freedom. The F-ratio is calculated by dividing the mean square of the betweengroups estimate by the mean square of the within-groups estimate.
What is error variance and how is it calculated? In a randomized ANOVA the error
variance is reflected by the within-groups sum of squares, and indicates the changeability of
measures within each condition and is represented with the formula Σ (X – Mg)2. With human
beings, direct causation is rarely identifiable; if participants are treated exactly the same in any
given condition the results received will vary to some extent. The error variance is a measure of
how much subjects in a given condition, or within-groups, vary. In a repeated-measures
ANOVA, the within-groups sum of the squares is split into subject variance and error variance.
The error variance in this situation is calculated by subtracting the variance attributable to
between-subjects from the within-groups variance to determine the error variance.
Why would anyone ever want more than two (2) levels of an independent variable?
Few conditions in the real world are truly dichotomous. Independent variables with multiple
levels give researchers the chance “to address more complicated and interesting questions”
(Jackson, 2012, p. 281). A common experimental design with multiple levels uses control,
placebo, and experimental groups to counteract demand characteristics and provides an
opportunity to compare differences between no treatment, the expectation that something will
occur, and the experimental treatment. An independent variable with more than two levels gives
a more complete picture of the relationship between the independent variable and the dependent
variables.
WattsSEDU7006-8-3
7
If you were doing a study to see if a treatment causes a significant effect, what would
it mean if within-groups variance was higher than between-groups variance? If betweengroups variance was higher than within-groups variance? Explain your answer. The
within-groups variance reveals the amount of inconsistency of measures within a given condition
or treatment and is indicative of the differences between subjects plus error. The betweengroups variance reflects the difference between measures caused by the independent variable,
confounding variables, and error. In the situation where the within-groups variance is higher
than the between-groups variance, the F-ratio will be closer to one, representing that no or little
variation is attributable to the independent variable. In the situation where the between-groups
variance is higher than the within-groups variance, the F-ratio will be much greater than 1,
indicative that there are differences between the experiment conditions.
What is the purpose of a post-hoc test with analysis of variance? Only when the Fratio exceeds the value of Fcv will a post-hoc test be conducted. If the Fobt exceeds the critical
value, differences between at least one pair of conditions is significant. The post-hoc test
compares each condition with every other condition to establish “which ones differ significantly
from each other” (Jackson, 2012, p. 297).
What is probabilistic equivalence? Why is it important? In order to utilize
experimental results to determine cause and effect or predict behavior it is important to be able to
control the environment such that the manipulation of the independent variable is the only
difference between two groups. For many experiments it is either not plausible or possible to use
the same subjects in the manipulation groups. Without using the same subjects, a major
assumption of an experiment is broken; that the only difference in the environment is the way
that the independent variable is manipulated, because different subjects with different
WattsSEDU7006-8-3
8
characteristics are being compared. Probabilistic equivalence indicates that while two groups
may not have the same participants, the chance, as a group, they are different from the
population from which they are drawn is very small and predictable (Trochim & Donnelly,
2008). Probabilistic equivalence is obtained through random assignment of participants to
groups.
WattsSEDU7006-8-3
9
References
Jackson, S. L. (2012). Research methods and statistics: A critical thinking approach (4th ed.).
Belmont, CA: Wadsworth Cengage Learning.
Trochim, W. M. K., & Donnelly, J. P. (2008). The research methods knowledge base (3rd ed.).
Mason, OH: Cengage Learning.