Research Methods
SLAN0012
Week 8
Mixed
1
Summary of factorial ANOVA so far
Factorial BETWEEN subjects
design
Factorial WITHIN subjects
design
If you have two or more
independent variables
If you have two or more
independent variables
If you have DIFFERENT
subjects in all levels of the
independent factors
If you have the SAME
subjects in all levels of the
independent factors
If all of the other assumptions
have been met
(continuous normallydistributed outcome, etc)
If all of the other assumptions
have been met
(continuous normallydistributed outcome, etc)
Mixed Design
• Previously, we made a strict separation:
– Using different subjects in each group
(between subjects), v.s. …
– Using the same subjects in each group (within
subjects)
• Sometimes it makes sense to do both
– Mixed Design
• Especially suitable for treatment studies
– the performance of a treated group is
compared to the performance of an untreated
(or control) group
– before and after treatment
• so-called ‘pretest-posttest control group design’
3
Mixed Design – Example studies
Twenty people with agrammatic aphasia are
randomly allocated to two groups. The first group
receives a new treatment aimed at decreasing
grammatical errors but the second group is
untreated. Performance of both groups is tested
before and 3 months after treatment, using a
sentence elicitation test (assessing number of
grammatical errors).
Treatment type
Between Subjects
Treated
Within Subjects
Before
3 month post
Poll Everywhere
Untreated
Before
3 month post
4
Mixed Design – Example studies
A picture naming task is applied to 12 children with DLD and 12
age-matched children who are typically developing (TD). Two
picture-naming tests are applied to each child: one involving
nouns (e.g., tree), and the other verbs (e.g., to run).
Participants are asked to respond to stimuli as quickly as
possible while avoiding errors. Mean response times for each
child in each task were used as the outcome (dependent)
variable.
Diagnostic Group
Between Subjects
SLI
Within Subjects
Nouns
Control
Verbs
Nouns
Verbs
5
Mixed Design – Example studies
Tasks involving spatial rotation and verbal working memory
were applied to group of adults in order to investigate the
possibility of differences in performance between men and
women. Mean accuracy for each individual for each test was
used to assess performance.
Sex
Between Subjects
Female
Within Subjects
Spatial
rotation
Working
memory
Male
Spatial
rotation
Working
memory
When to use a Mixed design ANOVA
• Two independent factors (variables)
• One factor is between subjects
• Treatment type
• Diagnostic group
• Sex
• One factor is within subjects
• Time point
– pre-intervention vs 3 months post
• Word type
– Nouns vs verbs
• Task
– Spatial rotation vs verbal working memory
7
Assumptions of Mixed Design
ANOVA
• Scale outcome variable
• Data for all levels should approximate a normal
distribution
• Independence of observations
• Homogeneity of variance/Sphericity (>0.05)
– Use Greenhouse-Geisser if violated
8
SPSS data input for Mixed ANOVA
Everything put in one row
Shall from one participant flow
Things that are put on one line
Indicate a repeated measures design
9
SPSS guide: Mixed Design ANOVA
Twenty people with
agrammatic aphasia
are randomly allocated
to two groups. The
first group receives a
new treatment aimed
at decreasing
grammatical errors but
the second group is
untreated.
Performance of both
groups is tested before
and 3 months after
treatment, using a
sentence elicitation
test (assessing
number of
grammatical errors).
Treated Group
Before
After
16
17
23
12
9
6
10
4
18
12
21
12
12
5
14
2
20
11
15
6
Untreated Group
Before
After
14
12
15
17
18
15
14
11
22
12
16
19
21
15
9
6
12
9
14
12
Is this what the data
should look like in
SPSS?
10
SPSS guide: Mixed Design ANOVA
• One row = one
participant
• 20 participants =
20 rows
• Within-subjects
factor (time) =
separate columns
• Between-subjects
factor (treatment
type) = factor of 2
levels
11
Summary
SPSS data looks different for
different kinds of 2x2 ANOVA
12
2x2 between-subjects ANOVA
– One row = one participant
– One dependent
variable/outcome, two
independent variables (factors)
– 3 columns
• 1 outcome column
• 2 independent variables =
2 coding variables
2x2 within-subjects ANOVA
– One row = one participant
– 4 dependent variables
– 4 outcome columns
• 2 factors with 2 levels each
2x2 mixed design ANOVA
– One row = one participant
– 3 columns
• 2 outcome columns:
within subjects
• 1 independent variable:
between subjects factor
15
 better performance
Look at the data
poll everywhere
16
Check for normality in Explore
Tests of Normality
Kolmogorov-Smirnova
Treatment_type
Before
After
Statistic
df
Shapiro-Wilk
Sig.
Statistic
df
Sig.
Treated
.114
10
.200*
.968
10
.867
Untreated
.152
10
.200*
.951
10
.686
Treated
.216
10
.200*
.925
10
.404
Untreated
.183
10
.200*
.973
10
.917
*. This is a lower bound of the true significance.
a. Lilliefors Significance Correction
17
SPSS guide: Mixed Design ANOVA
Output
• Models
• Levene’s
(should be >0.05)
18
As in any ANOVA …
• the observed variance in the outcome is partitioned into
components attributable to different sources of variation
• and the statistics are based on the ratios of how much
variance is due to each factor compared to the overall
variability
• Here we have 3 aspects:
– the within groups effect
– the between groups effect
– the interaction
• e.g., is the effect of the intervention different in the groups?
19
SPSS guide: Mixed Design ANOVA
WITHIN SUBJECTS
MAIN EFFECT
• Time
–
–
–
–
df = 1,18
F = 33.896
p < 0.001
ƞp² = 0.653
• INTERACTION
Time*Treatment
–
–
–
–
df = 1,18
F = 6.83
p = 0.018
ƞp² = 0.275
20
SPSS guide: Mixed Design ANOVA
BETWEEN SUBJECTS
MAIN EFFECT: Treatment Type
• df = 1,18
• F = 1.189
• p = 0.29
• ƞ ² = 0.062
p
21
Mixed Design ANOVA
Summary
• Interaction = Time*Treatment type
– Significant
– F(1,18)=6.83, p=0.018, ƞ ²=0.275
• Main effect 1 – Time
– Significant
– F(1,18)=33.896, p<0.01, ƞ ²=0.653
• Main effect 2 – Treatment type
– Non-Significant
– F(1,18)=1.189, p=0.29, ƞ ²=0.062
p
p
p
22
Mixed Design ANOVA
Following up a significant interaction
with independent samples t-tests
Compare
1.Treated to
untreated group
before treatment
2.Treated to
untreated group
after treatment
23
SPSS guide: Mixed Design ANOVA
• No significant difference in scores before
treatment
• Significant difference between groups after
treatment
• barely!
24
Mixed Design ANOVA
Following up a significant interaction
with paired samples t-tests
Compare
1.Before and
after in the
untreated group
2.Before and
after in the
treated group
3.This seems
more sensible to
me!
25
SPSS guide: Mixed Design ANOVA
Paired Samples Test
Paired Differences
95% Confidence Interval of the
Difference
Treatment_type
Treated
Pair 1
Before - After
Mean
7.1000
Untreated
Before - After
2.7000
Pair 1
Std. Deviation Std. Error Mean
3.8715
1.2243
3.6530
1.1552
Lower
4.3305
Upper
9.8695
t
5.799
0.0868
5.3132
2.337
df
9
Sig. (2-tailed)
0.000
9
0.044
• Highly significant difference in scores before
and after treatment in the treated group
• A just significant difference in scores in the
treated group
• barely!
Poll Everywhere
26
Reporting results for a
Mixed Design ANOVA
The results of a two-way Analysis of Variance (ANOVA) with one within-subjects factor of
Time (before vs. after treatment), and one between-subjects factor of Group (treated vs.
untreated) showed a statistically significant main effect of Time (F(1,18)=33.896,
p<0.01, ƞp²=0.653) but no significant main effect of Group (F(1,18)=1.189, p=0.29,
ƞp²=0.062).
More importantly, there was a significant interaction (F(1,18)=6.83, p=0.018,
ƞp²=0.275), indicating a larger improvement in performance for the treated group as
opposed to the untreated group. The interaction was followed up by performing a pairedsamples t-test within the groups. Both groups improved significantly on the second
administration of the test (with errors decreasing from a mean of 15.8 (s.d.=4.7) to 8.7
(s.d.=4.7) , a decrease in the number of errors of 7.1 in the treated group, and from a
mean of 15.5 (s.d.=4.0) to 12.8 (s.d.=3.8), a decrease in the number of errors of 2.7 in
the untreated group. (Paired-samples t-tests: treated: t(9)= 5.8, p< 0.001, Cohen’s d =
1.83) untreated: t(9)= 2.3, p= .044, Cohen’s d = 0.74;
Although both groups improved to some extent, the greater improvement in the treated
group (as indicated by the significant interaction) suggest that the treatment method
was successful (although one should interpret the functional significance of the effect
cautiously).
[If you wanted, you could compare the two groups before and after with an independent
samples t-test, finding that the results showed a significant difference between the
groups after the treatment (t(18)= 2.192, p= .047, Cohen’s d = 0.98) but no difference
before the treatment (t(18)=0.154, p=0.879, Cohen’s d = 0.069).]
Research Methods
SLAN0012
Week 8
Workshop demonstration
28
SPSS guide: Mixed Design ANOVA
Twenty people with
agrammatic aphasia
are randomly allocated
to two groups. The
first group receives a
new treatment aimed
at decreasing
grammatical errors but
the second group is
untreated.
Performance of both
groups is tested before
and 3 months after
treatment, using a
sentence elicitation
test (assessing
number of
grammatical errors).
Treated Group
Before
After
16
17
23
12
9
6
10
4
18
12
21
12
12
5
14
2
20
11
15
6
Untreated Group
Before
After
14
12
15
17
18
15
14
11
22
12
16
19
21
15
9
6
12
9
14
12
Data should not look
like this in SPSS!
29
SPSS guide: Mixed Design ANOVA
• One row = one
participant
• 20 participants =
20 rows
• Within-subjects
factor (time) =
separate columns
• Between-subjects
factor (treatment
type) = factor of 2
levels
31
Analyze → Explore
works here too
32
Or claim your
legacy …
• Graphs→
Legacy Dialogs →
graph type
(e.g. boxplot)
within-subjects
repeated measures
mixed design
and separate groups
The graph that results …
Let’s make this prettier … and more legible!
34
SPSS guide: Mixed Design ANOVA
Analyze → Descriptives
→ Explore
• Move the levels of the
within subjects
outcomes to the
Dependent List
• Move the between
subjects factor to
Factor List
• Check for
–
–
–
–
Normality
Shapiro-wilk
Outliers (3s.d. rule)
You get homogeneity of
variance (Levene’s test)
later anyway
Check for normality in Explore
(you get homogeneity later
anyway)
Tests of Normality
Kolmogorov-Smirnova
Treatment_type
Before
After
Statistic
df
Shapiro-Wilk
Sig.
Statistic
df
Sig.
Treated
.114
10
.200*
.968
10
.867
Untreated
.152
10
.200*
.951
10
.686
Treated
.216
10
.200*
.925
10
.404
Untreated
.183
10
.200*
.973
10
.917
*. This is a lower bound of the true significance.
a. Lilliefors Significance Correction
36
SPSS guide: Mixed Design ANOVA
Perform the ANOVA
• Go to Analyze → General
Linear Model → Repeated
measures
• Define the within-subjects
factor
• We only have one:
• Time – 2 levels (Before vs After)
• Click Define
• Nothing is yet done with the
between subjects factor yet
37
SPSS guide: Mixed Design ANOVA
• Assign the withinsubjects levels to
the within subjects
variables box
(before and after
• Move the betweensubjects factor
(treatment type)
to the betweensubjects factor(s)
box
38
SPSS guide: Mixed Design ANOVA
39
Options: Mixed Design ANOVA
OBS! GUI different in different versions!
• Obtain marginal means
for all main and
interaction effects
• Select ‘Estimates of
Effect size’ and
‘homogeneity tests’
• Neither post-hocs nor
‘compare main effects’
are required as each
factor has only 2 levels
• Use the means to interpret
significant main effects
40
Plots in SPSS: Mixed Design ANOVA
Interactions
• Move one factor
to Horizontal Axis
and the other to
Separate Lines
– Will work without
regard to order
– but you may
prefer one
• Need to Add
– and choose Chart
Type & Error Bars
SPSS guide: Mixed Design ANOVA
WITHIN SUBJECTS
MAIN EFFECT
• Time
–
–
–
–
DF = 1,18
F = 33.896
p < 0.001
ƞ ² = 0.653
p
• INTERACTION
Time*Treatment
–
–
–
–
DF = 1,18
F = 6.83
p = 0.018
ƞp² = 0.275
42
SPSS guide: Mixed Design ANOVA
BETWEEN SUBJECTS
MAIN EFFECT: Treatment Type
• DF = 1,18
• F = 1.189
• p = 0.29
• ƞ ² = 0.062
p
43
Mixed Design ANOVA
Following up a significant interaction
with independent samples t-tests
Compare
1.Treated to
untreated group
before treatment
2.Treated to
untreated group
after treatment
44
SPSS guide: Mixed Design ANOVA
Following up on a
significant interaction:
• Run independent
samples t-Tests
• Move both levels of
the within subjects
factor to ‘Test
variables’ box
• Move the between
subjects variable to
the ‘Grouping variable’
box and define the
groups
45
SPSS guide: Mixed Design ANOVA
• No significant difference in scores before
treatment
• Significant difference between groups after
treatment
• barely!
46
Mixed Design ANOVA
Following up a significant interaction
with paired samples t-tests
Compare
1.Before and
after in the
untreated group
2.Before and
after in the
treated group
47
48
SPSS guide: Mixed Design ANOVA
Following up on a significant interaction II with a
paired samples t-Tests
Move both outcome measures to Paired Variables
box
SPSS guide: Mixed Design ANOVA
Paired Samples Test
Paired Differences
95% Confidence Interval of the
Difference
Treatment_type
Treated
Pair 1
Before - After
Mean
7.1000
Untreated
Before - After
2.7000
Pair 1
Std. Deviation Std. Error Mean
3.8715
1.2243
3.6530
1.1552
Lower
4.3305
Upper
9.8695
t
5.799
0.0868
5.3132
2.337
df
9
Sig. (2-tailed)
0.000
9
0.044
• Highly significant difference in scores before
and after treatment in the treated group
• A just significant difference in scores in the
tunreated group
• barely!
50
Reporting results for a
Mixed Design ANOVA
The results of a two-way Analysis of Variance (ANOVA) with one within-subjects factor of
Time (before vs. after treatment), and one between-subjects factor of Group (treated vs.
untreated) showed a statistically significant main effect of Time (F(1,18)=33.896,
p<0.01, ƞp²=0.653) but no significant main effect of Group (F(1,18)=1.189, p=0.29,
ƞp²=0.062).
More importantly, there was a significant interaction (F(1,18)=6.83, p=0.018,
ƞp²=0.275), indicating a larger improvement in performance for the treated group as
opposed to the untreated group. The interaction was followed up by performing a pairedsamples t-test within the groups. Both groups improved significantly on the second
administration of the test (with errors decreasing from a mean of 15.8 (s.d.=4.7) to 8.7
(s.d.=4.7) , a decrease in the number of errors of 7.1 in the treated group, and from a
mean of 15.5 (s.d.=4.0) to 12.8 (s.d.=3.8), a decrease in the number of errors of 2.7 in
the untreated group. (Paired-samples t-tests: treated: t(9)= 5.8, p< 0.001, Cohen’s d =
1.83) untreated: t(9)= 2.3, p= .044, Cohen’s d = 0.74;
Although both groups improved to some extent, the greater improvement in the treated
group (as indicated by the significant interaction) suggest that the treatment method
was successful (although one should interpret the functional significance of the effect
cautiously).
[If you wanted, you could compare the two groups before and after with an independent
samples t-test, finding that the results showed a significant difference between the
groups after the treatment (t(18)= 2.192, p= .047, Cohen’s d = 0.98) but no difference
before the treatment (t(18)=0.154, p=0.879, Cohen’s d = 0.069).]