EPSE 592
Experimental Designs and Analysis in
Educational Research
Instructor: Dr. Amery Wu
1

Where We Are Today
Mixed Design Analysis of Variance
Descriptive
Type of the
Inference
Inferential
Measurement of Data
Quantitative Categorical
A B
C D
Today, we will introduce the analysis and assumptions of a mixed design.
This design explains/predicts quantitative data using two independent variables- one (explicit) between groups and one
(implicit) within-subjects (repeated measures). Each IV has two or more conditions .
2

Example
2x3 Mixed Design ANOVA
8
9
10
5
6
7 case
1
2
3
4
11
12
13
14
15
16
17
18
1
1
1
1
1
1
1
1
0
0
1
0
0
0 diet before warm-up after
0 95 134 186
0 66 109 150
0
0
69
93
119
151
177
217
77
78
58
85
78
112
122
119
99
132
110
166
178
173
131
186
164
225
111
89
125
85
97
81
88
88
166
132
177
117
137
134
133
157
235
199
251
196
195
215
190
234
3

Example
2x3 Mixed Design ANOVA
Questions:
1. What is the DV?
2. What is the between-groups factor? How many levels are there?
3. What is the within-subjects factor? How many levels are there?
Lab Activity:
Using your own words, state as many (nonredundant) research questions as you can based on this research design.
4

Example
2x3 Mixed Design ANOVA
1. Does exercise has an effect on the pulse? (Dose the pulse change over time (before, during, and after exercise)?
2. Is there a difference in the pulse between the two dietary groups?
3. Dose the pattern of temporal change in pulse due to exercise differ (be moderated) between dietary groups?
Alternatively, is the relationship between diet and pulse differs (be moderated) by the timing of when the pulse is taken?
5

Research
Question
Inference
Descriptive vs. Inferential
Relational vs. Causal
Design
Experimental
Observational
Model
Descriptive/Summative
Explanatory/Predictive
Data
Quantitative
Categorical
6

There are six groups of samples separated by three columns (purple, blue, and yellow) and by the 2 categories of the diet column (in green and red texts) .
Questions: In what way, the observation of the six groups is supposed to be dependent and independent?
8
9
10
11
12
13
14
15 case
1
2
5
6
3
4
7
16
17
18
1
1
1
1
1
0
0
1 diet before warm-up after
0 95 134 186
0 66 109 150
0
0
0
0
0
69
93
77
78
58
119
151
122
119
99
177
217
178
173
131
1
1
1
85
78
112
111
89
125
85
97
81
88
88
132
110
166
166
132
177
117
137
134
133
157
215
190
234
186
164
225
235
199
251
196
195
7

An research design that incorporates both between- groups (i.e., independent) factor and within-subject (i.e., dependent, correlated, or repeated measures) factors.
Hence, for a mixed design, there is at least two
IVs with a possibility of inclusion of another interaction IV: “between* within” (i.e., two-way factorial design with interaction).
Mixed design is also known as split-plot design
8

For the between-groups factor, the group membership of the participants are independent (i.e., exclusive). Participants in one group differ from those in another groups.
Hence, the observation of the DV scores are unrelated across the groups.
Examples for between-group factors are gender groups, randomly assigned treatment groups, or teachers in different school districts.
9

For the within-subjects factor (e.g., repeated measures), the observation of the participants under one condition is dependent of that of another because the participants are selected to be related by design.
Examples for within-subject factors are repeated measures groups, matched groups, twin groups, group scores by the same raters, or dyads groups.
10

Mixed Design ANOVA is an inferential statistics that tests experimental treatment effects (or simply mean differences for observation data) with a mix of betweengroup and within-subject factors.
A full factorial mixed design with one between and one within factors is similar to a two-way factorial ANOVA. The only difference is that one of the IV now is a withinsubjects factor.
A full two-way factorial ANOVA may include testing of the following effects:
1. the main effect of the between-groups factor
2. the main effect of the within-subjects factor
3. interaction effect- “between x within” factor
11

See Excel file “Exercise_Diet_Pulse_Mixed
Design with Interaction.xlsx” for computing F statistics for the mixed design ANOVA.
12

Means – Cross Tabulation between by within
Within (Time)
Before Warm-up After
Between Vege 77.67
121.67
173.56
113.96
(Diet) Meat 97.33
146.56
215.56
110.51
94.88
143.44
210.31
138.72
Questions:
1. By comparing which values, is the main effect of the between group factor (diet) examined?
2. By comparing which values, is the main effect of the within-subject factor (time) examined?
3. By comparing which values, is the interaction effect of between*within (diet*time) examined?
13

The between-factor effect is investigated by comparing the between-factor marginal means the mean DV scores among the groups for the between-factor(e.g., treatment groups or vegetarians vs. meat eaters).
The F statistics is calculated using the error term for the between-factor (see Excel file).
Vege 77.67
121.67 173.56 113.96
Meat 97.33
146.56 215.56 110.51
14

The within-factor effect is investigated by comparing the within-factor marginal means - the mean DV scores among the groups for the withinfactor (e.g., repeated measures of same individuals; comparing their own repeated DV scores under different treatment conditions).
The F statistics is calculated using the error term for the within-factor (see Excel file).
Before Warm-up After
77.67
121.67
173.56
97.33
146.56
215.56
94.88
143.44
210.31
15

“Between x Within” Interaction Effect
The “between x within” interaction effect is investigated by conditional mean differences (i.e., simple main effects)comparing the mean differences in the DV score among the between-factor groups for each of the within-factor groups
(alternatively, among the within-factor groups for each of the between-factor groups).
Take the Pulse data for example, the interaction effect is investigated by comparing the mean differences between the diet groups (vege vs. meat) for each of the three within groups (before, warm-up, and after exercise).
The F statistics is calculated using the error term for the within-factor .
Within
Before Warm-up After
Between Vege 77.67
Meat 97.33
diff 19.67
121.67
146.56
24.89
173.56
215.56
42.00
16

The DV scores for each of the PxQ groups are all normally distributed (P= # of groups for the betweenfactor and Q= # of groups for the within-factor).
Independence and equal (homogeneity) variances assumption for the between-factor groups .
Normal distribution and sphericity assumption for the within-factor groups. The difference scores between any pairs of the repeatedly measured are normally distributed. Also, the variances of the difference scores are equal (sphericity). With Q groups, there are
Q(Q-1)/2 difference scores.
17

Interpretation of the between-factor effect makes sense only if the interaction effect is non-significant.
The between-factor effect is interpreted the same way as is the one-way independent ANOVA.
The omnibus F test for the between-factor examines whether there is any statistically significant mean difference in the DV between any pairs of betweenfactor groups.
If the omnibus F test is statistically significant, do posthoc pairwise t-tests to find out which pair(s) of mean difference is statistically significant.
If the equal variances assumption is not met, remember to choose a post-hoc pairwise t-test that does not assume equal variances.
18

Interpretation of the within-factor main effect makes sense only if the interaction effect is non-significant.
The within-factor main effect is interpreted the same way as is the repeated measures ANOVA (see Handout Four).
The omnibus F test for within-factor examines whether there is any statistically significant mean difference in the DV between any pairs of within-factor groups.
If the sphericity assumption is violated (Greenhouse-
Geisser’s ἓ less than 0.75), pick one of the alternative F-tests that corrects for violation from sphericity (i.e., Greenhouse-
Geisser, Huynh-Feldt, or Lower-bound in SPSS).
If the selected omnibus F test is statistically significant, do post-hoc pairwise t-tests to find which pair(s) of mean difference is statistically significant.
If sphericity assumption is violated, choose Bonferroni or
Sidak’s post hoc t-test for correction(see Handout Four).
19

Interpretation of the Mixed Design ANOVAInteraction
If the omnibus F test for the interaction effect is nonsignificant, interpret the each of the significant main effect separately.
Note that once the omnibus F test for the interaction effect is significant, it is inappropriate to interpret the between- or within-factor main effects even if these main effects are significant (always interpret the highest order of the effect that is significant).
SPSS drop-down menu does not provide a function to test the simple main effects. With SPSS, to test and interpret the interaction effect in terms of simple main effects (i.e., conditional mean differences). See later slides.
20

Mixed Design ANOVA by SPSS
Remember to hit “Add”
21

Mixed Design ANOVA SPSS Output
 First, examine whether the interaction effect is significant within the F table of within-subjects effects.
 To do so, we need to check whether the sphericity assumption (for with-factor) is met. Because the assumption is not met (using
Greenhouse-Geisser Epsilon <0.75 criterion), we should use an F test that does not assume sphericity.
22

Mixed Design ANOVA SPSS Output
Because the “Time x Diet” Interaction effect is significant, the main effects of time and diet should be interpreted in the context of the other by interpreting the simple mail effects.
23

Testing Simple Main Effects Using SPSS Syntax
One has to use the following syntax to produce the output:
GLM PulseBefore PulseWarmUp PulseAfter BY Diet
/WSFACTOR=Time 3
/MEASURE=Pulse
/METHOD=SSTYPE(3)
/EMMEANS=TABLES(Diet*Time) compare (diet) adj (Bonferroni)
/CRITERIA=ALPHA(.05).
24

Interpretation of the Mixed Design ANOVAInteraction
This is the extra table produced by the bolded line in the syntax for testing the interaction effect in terms of simple main effects.
For each of the within-factor groups (before, warm-up, and after), the mean difference in pulse between the two dietary groups is compared and tested.
25

If the Interaction Effect is Non-significant
When the interaction effect is non-significant, one can interpret the results separately for each of the between- and within-factor if they are significant.
One can test the planned contrasts or conduct post hoc t tests, correcting for type-one error rate and correcting for violation of equal variances assumptions (for both each between- and within- effects). See Power Points slides course handout #8 and #9.
26