Repeated measures ANOVA LabSyntax, 02/23/06 T. Florian Jaeger Intro Basis idea: We’re interested in populations The sample of subjects & items we see is (so we hope/assume) representative for the population, but not identical I.e. we assume that we observe the effect of our manipulation on a randomly selected sample of the population of interest (e.g. native speakers of English; question-embedding verbs) What this means: The statistical analysis should reflect our assumptions Subjects and Items are random effects/factors [2] What does it mean to be a random effect? Not all levels of the effect are observed in the experiment Sometimes we have such specific research questions that we DO observe all levels (in terms of items), consider e.g. irregular verbs with a specific stress pattern. [3] For corpus work: the # of levels for a random effect usually increases as the sample size increases The standard (psycholinguistic) analysis ANOVA (Analysis of variance) Designed for balanced design (unreliable for unbalanced designs) Check other assumptions (normally distributed error, sphericity/homogeneity of variance), e.g. Howell 1995 Separate subject (F1) and item (F2) analyses: F1: for each subject average/aggregate within each condition over all items F2: for each item average/aggregate within each condition over all subjects Advantage: even for missing cells, averaging gives you a mean value for each condition for each subject/item [4] SPSS Example courtesy of John Hale Repeated Measures in SPSS Open file, import data 1st subject, Subject ID=0 20 items Design: 2 (LIKE) x 2 (WAY) [6] UCP1.DIFF.CONJ He wants to go to the movies and mini-golfing. UCP1.SAME.CONJ He wants to go to the movies and to the mini-golf course. UCP1.DIFF.REPAIR He wants to go to the movies, I mean, mini-golfing. UCP1.SAME.REPAIR He wants to go to the movies, I mean, to the mini-golf course. Repeated Measures in SPSS, e.g. F1 Open file, import data Aggregate (e.g. over items) Data Aggregate [7] Repeated Measures in SPSS, e.g. F1 Open file, import data Aggregate (e.g. over items) Data Aggregate Open aggregate file Restructure Data Restructure [8] Repeated Measures in SPSS, e.g. F1 Open file, import data Aggregate (e.g. over items) Data Aggregate Open aggregate file Restructure Data Restructure [9] Then go “Next” “Next” “Next” “Finish” Repeated Measures in SPSS, e.g. F1 Open file, import data Aggregate (e.g. over items) Data Aggregate Open aggregate file Restructure Data Restructure Check! Look at output and data [10] SAVE!! 1 subject per line All 2 x 2 = 4 conditions in Columns in that line Repeated Measures in SPSS, ANALYSIS Start repeated measures Analysis GLM repeated measures [11] Define factors and measure Repeated Measures in SPSS, ANALYSIS Start repeated measures Analysis GLM repeated measures Define factors and measure Define levels Make sure order is right! [12] Repeated Measures in SPSS, ANALYSIS Start repeated measures Analysis GLM repeated measures H Define factors and measure Define levels LIST could go here, as could AGE, GENDER, e Frequency, Length could be entered as covariat Make sure order is right! [13] Define some plots Optionally post-hoc tests Add e.g. interaction plot Repeated Measures in SPSS, Results Sphericity is not an issue with binary factors But with more levels: check whether sphericity holds (Mauchly’s test) If not, use Greenhouse-Geisser [14] Repeated Measures in SPSS, Results (plots) Be aware that plots use marginal means (not the same as means) To read plots, look at Within-subject Factors Within-Subjects Factors Measure: rating like 1 2 way 1 2 1 2 Dependent Variable diff.conj diff.repair s ame.conj s ame.repair DIFF.REPAIR He wants to go to the movies, I mean, mini-golfing. [15] DIFF.CONJ He wants to go to the movies and minigolfing. SAME.CONJ He wants to go to the movies and to the minigolf course. SAME.REPAIR He wants to go to the movies, I mean, to the minigolf course. Repeated Measures in SPSS, do results hold? Item analysis Tests of Within-Subjects Effects Measure: rating Source like Error(like) way Error(way) like * way Spherici ty As sum ed Greenhouse-Gei ss er Huynh-Fel dt Lower-bound Spherici ty As sum ed Greenhouse-Gei ss er Huynh-Fel dt Lower-bound Spherici ty As sum ed Greenhouse-Gei ss er Huynh-Fel dt Lower-bound Spherici ty As sum ed Greenhouse-Gei ss er Huynh-Fel dt Lower-bound Spherici ty As sum ed Greenhouse-Gei ss er Huynh-Fel dt Lower-bound Spherici ty As sum ed Greenhouse-Gei ss er Huynh-Fel dt Lower-bound Type III Sum of Squares 1.712 1.712 1.712 1.712 1.726 1.726 1.726 1.726 7.552 7.552 7.552 7.552 4.210 4.210 4.210 4.210 .166 .166 .166 .166 4.847 4.847 4.847 4.847 df 1 1.000 1.000 1.000 19 19.000 19.000 19.000 1 1.000 1.000 1.000 19 19.000 19.000 19.000 1 1.000 1.000 1.000 19 19.000 19.000 19.000 Mean Square 1.712 1.712 1.712 1.712 .091 .091 .091 .091 7.552 7.552 7.552 7.552 .222 .222 .222 .222 .166 .166 .166 .166 .255 .255 .255 .255 F 18.847 18.847 18.847 18.847 Sig. .000 .000 .000 .000 34.086 .000 34.086 34.086 .000 .000 34.086 Interaction not confirmed by .000 F2 analysis But less power (look at DF of error, 19 instead of 86) Probably small effect size .651 .651 .651 .651 .430 .430 .430 .430 Exaggerated MORAL: Wasted subjects! Lots of power in terms of subjects does not help with F2 analysis! Error(like*way) [16] [17] The previous example was taken from a study by John Hale (in progress) not to be cited or circulated without his permission.