INFERENTIAL STATISTICS © LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON STRUCTURE OF THE CHAPTER • Measures of difference between groups • The t-test (a difference test for parametric data) • Analysis of variance (a difference test for parametric data) • The chi-square test (a difference test and a test of goodness of fit for non-parametric data) • Degrees of freedom (a statistic that is used in calculating statistical significance in considering difference tests) • The Mann-Whitney and Wilcoxon tests (difference tests for non-parametric data) STRUCTURE OF THE CHAPTER • The Kruskal-Wallis and Friedman tests (difference tests for non-parametric data) • Regression analysis (prediction tests for parametric data) • Simple linear regression (predicting the value of one variable from the known value of another variable) • Multiple regression (calculating the different weightings of independent variables on a dependent variable) • Standardized scores (used in calculating regressions and comparing sets of data with different means and standard deviations) MEASURES OF DIFFERENCE BETWEEN GROUPS • Are there differences between two or more groups of sub-samples, e.g.: – Is there a significant difference between the amount of homework done by boys and girls? – Is there a significant difference between test scores from four similarly mixed-ability classes studying the same syllabus? – Does school A differ significantly from school B in the stress level of its sixth form students? MEASURES OF DIFFERENCE BETWEEN GROUPS • The t-test (for two groups): parametric data • Analysis of Variance (ANOVA) (for three or more groups: parametric data • The chi-square test: for categorical data • The Mann-Whitney and Wilcoxon tests (for two groups): non-parametric data • The Kruskal-Wallis and the Friedman tests (for three or more groups): non-parametric data t-TEST • Devised by William Gossett in 1908; • Used when we have 2 conditions; the t-test assesses whether there is a statistically significant difference between the means of the two conditions; • The independent t-test is used when the participants perform in only one of two conditions; • The related or paired t-test is used when the participants perform in both conditions. t-TEST FOR PARAMETRIC DATA • t-tests (parametric, interval and ratio data) – To find if there are differences between two groups – Decide whether they are are independent or related samples Independent sample: two different groups on one occasion Related sample: one group on two occasions t-TEST FOR PARAMETRIC DATA Formula for computing the t-test Sample one mean – sample two mean t = Standard error of the difference in means Formula for calculating t t M1 M 2 d12 d 22 1 1 N1 N 2 2 N1 N 2 M = Mean d = difference between the means N = Number of cases t-TEST FOR INDEPENDENT SAMPLES The t-test computes a ratio between a measure of the between-groups variance and the within group variance. The larger the variance between the groups (columns), compared with the variance within the groups (rows), the larger the t-value. INDEPENDENT AND RELATED SAMPLES IN A t-TEST: EXAMPLES 1. Independent sample (two groups): • A group of scientists wants to study the effects of a new drug for insomnia. They have applied this drug to a random group of people (control group) and to a group of people suffering from insomnia (experimental group); 2. Related sample (same group in two conditions): • A group of therapists wants to study whether there is any difference in doing relaxation techniques on the beach or in an apartment. A group of people is asked to first do relaxation on the beach and later in an apartment; INDEPENDENT AND RELATED SAMPLES IN A t-Test: AN EXAMPLE 24 people were involved in an experiment to determine whether background noise affects short-term memory (recall of words); – If half of the sample were allocated to the NOISE condition and the other half to the NO NOISE condition (independent sample) – we use independent t-test; – If everyone in the sample has performed at both conditions (related sample) – we use paired or related t-test. AN EXAMPLE OF A t-TEST Participants were asked to memorize a list of 20 words in two minutes. Half of the sample performs in a noisy environment and the other half in a quiet environment; Independent variable - two types of environment: Quiet environment (NO NOISE condition) Noisy environment (NOISE condition) Dependent variable – the number of words each participant can recall. NOISE 5 10 6 6 7 3 6 9 5 10 11 9 = 87 X = 7.3 SD = 2.5 NO NOISE 15 9 16 15 16 18 17 13 11 12 13 11 = 166 X = 13.8 SD = 2.8 NOTE: participants vary within conditions: in the NOISE condition, the scores range from 3 to 11, and in the NO NOISE condition. They range from 9 to 18; The participants differ between the conditions too: the scores of the NO NOISE condition, in general, are higher than those in the NOISE condition – the means confirm it; Are the differences between the means of our groups large enough for us to conclude that the differences are due to our independent variable: NOISE/NO NOISE manipulation? t-TEST FOR INDEPENDENT SAMPLES Group statistics In which condition are you? N Mean How many words can you recall? Std. Deviation Std. Error Mean NOISE 12 7.2500 .71906 NO NOISE 12 13.8333 .79614 This shows: the name of 2 conditions; the number of cases in each condition; the mean of each condition; the standard deviation and standard error of the mean, of the two conditions. t-TEST FOR INDEPENDENT SAMPLES (SPSS) Independent Samples Test Levene’s Test for Equality of Variances F How many Equal variances words can assumed you recall? Equal variances Sig t-test for Equality of Means t .177 .676 -6.137 df Sig. (2-tailed) Mean Std. Error Differences Differences 95% Confidence Interval of the Difference Lower Upper 22 .000 -6.5833 1.07279 -8.808 -4.359 -6.137 21.78 .000 -6.5833 1.07279 -8.809 -4.357 not assumed The Levene test is for ‘homogeneity of variance’, and the t-test here indicates whether you should use the upper or lower row. Mean Difference means the difference between the means of the two groups. REPORTING FINDINGS FROM THE EXAMPLE Participants in the NOISE condition recalled fewer words (t (22) = 7.25, SD = 2.49) than in the NO NOISE condition (t (22) = 13.83, SD = 2.76). The mean difference between conditions was 6.58; the 95% confidence interval for the estimated population mean difference is between 4.36 and 8.81. An independent t-test revealed that, if the null hypothesis is true, such a result would be highly unlikely to have arisen (t (22) = 6.14; p<0.001). It is therefore concluded that listening to noise affects short-term memory, at least in respect of word recall. t-TEST FOR INDEPENDENT SAMPLES WITH SPSS Group Statistics Which group are you Mathematics post-test Control group score Experimental Group One N Mean Std. Std. Error Deviation Mean 166 8.69 1.220 .095 166 9.45 .891 .069 Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means 95% Confidence Interval of the Difference Mathematics post-test score Equal variances assumed Equal variances not assumed F 28.856 Sig. t .000 -6.523 Sig. (2Mean Std. Error tailed) Difference Difference Lower df Upper 330 .000 -.765 .117 -.996 -.534 -6.523 302.064 .000 -.765 .117 -.996 -.534 Read the line ‘Levene’s Test for Equality of Variances’. If the probability value is statistically significant then your variances are unequal; otherwise they are regarded as equal. If the Levene’s probability value is not statistically significant then you need the row ‘equal variances assumed’; if the Levene’s probability value is statistically significant then you need the row ‘equal variances not assumed’. Look to the column ‘Sig. (2-tailed)’ and the appropriate row, and see if the results are statistically significant. PAIRED SAMPLE t-TEST (SAME GROUP UNDER TWO CONDITIONS) WITH SPSS Paired Samples Statistics Std. Std. Error Mean N Deviation Mean Pair 1 Mathematics pre-test score Mathematics post-test score 6.95 252 1.066 .067 8.94 252 1.169 .074 This indicates: 1.The two conditions; 2.The mean of each condition; 3.The number of cases in each condition; 4.The standard deviation and standard error of the mean, for the two conditions. PAIRED SAMPLE t-TEST (SAME GROUP UNDER TWO CONDITIONS) WITH SPSS Paired Samples Correlations N Pair 1 Mathematics pre-test score & Mathematics post-test score Correlation 252 .020 Sig. .749 This shows that there is no association between the scores on the pre-test and the scores on the post test for the group in question (r = .02 and = .749). PAIRED SAMPLE t-TEST (SAME GROUP UNDER TWO CONDITIONS) WITH SPSS Paired Samples Test Paired Differences 95% Confidence Interval of the Difference Std. Error Std. Mean Deviation Mean Lower Upper Pair Mathematics 1 pre-test score - -1.992 Mathematics post-test score 1.567 t Sig. df (2-tailed) .099 -2.186 -1.798 -20.186 251 This shows that : 1.The difference between the mean of each condition (6.95 and 8.94) is 1.992. 2.The confidence intervals shows that we are 95% certain that the population difference lies somewhere between -2.186 and -1.798. 3.There is a statistically significant difference between the two sets of scores. .000 RESULT It can be seen from the paired t-test that the hypothesis is not supported (t (251) = 20.186; =.000). DEGREES OF FREEDOM The number of individual scores that can vary without changing the sample mean. The number of scores one needs to know before one can calculate the others. E.g.: If you are asked to choose 2 numbers that must add up to 100, and the first is 89, then the other has to be 11; there is 1 degree of freedom (89 + x = 100). If you are asked to choose 3 numbers that must add to 100, and the first of these is 20, then you have 2 degrees of freedom (20 + x + y = 100). DEGREES OF FREEDOM (WITH SPSS) Which group are you * Who are you Crosstabulation Chinese or non-Chinese Chinese Non-Chinese Total Which group Control group are you Experimental Group One Experimental Group Two Total 156 94.0% 166 100.0% 143 85.1% 465 93.0% 10 166 6.0% 100.0% 0 166 .0% 100.0% 25 168 14.9% 100.0% 35 500 7.0% 100.0% Degrees of freedom = 2 (1 degree of freedom in each of 2 rows, which fixes what must be in the third row) ANALYSIS OF VARIANCE (ANOVA) • Analysis of variance – Parametric, interval and ratio data – To see if there are any statistically significant differences between the means of two or more groups; – It calculates the grand mean (i.e. the mean of the means of each condition) and sees how different each of the individual means is from the grand mean. – Premised on the same assumptions as t-tests (random sampling, a normal distribution of scores, independent variable(s) is/are categorical (e.g. teachers, students,) and one is a continuous variable (e.g. marks on a test). ANOVA AND MANOVA • One way analysis of variance (one categorical independent variable and one continuous dependent variable) • Two-way analysis of variance (two categorical independent variables and one continuous dependent variable) • Multiple analysis of variance (MANOVA) (one categorical independent variable and two or more continuous variables) • Post-hoc tests (e.g. Tukey hsd test, Sheffe test) to locate where differences between means lie (in which group(s)) FORMULA FOR ANOVA between - groups variance F ratio within - groups variance d X N Between variance 2 mean df groups1 Within variance d 2 df N groups A1 9 9 9 9 9 X =9 A2 15 15 16 15 16 X = 15.4 A3 21 25 17 22 26 X = 22.2 Between-groups and within-groups variance: Variation between the groups (9 to 22.2); Variation within the first group (no variation since all participants scored the same); Variation within the second group (from 15 to 16); Variation within the third group (from 17 to 26). ANOVA 1. First, ANOVA calculates the mean for each of the three groups; 2. Then it calculates the grand mean (the three means added then divided by three); 3. For each group separately, the total deviation of each individual’ s score from the mean of the group is calculated (within-groups variation); 4. Then the deviation of each group mean from the grand mean is calculated (between-groups variation). F RATIO between - groups variance F ratio within - groups variance When we conduct our experiment, we hope that the between-groups variance is very much larger than the within-groups variance, in order to get a bigger F ratio; This shows us that one (or more) of the individual group means is significantly different from the grand mean; However, it does not tell us which means are statistically significantly different. De scri ptives Records of student s' progress 95% Confidenc e Interval for Mean 20-29 30-39 40-49 50+ Total N 7 5 4 1 17 Mean 3.29 3.80 3.25 4.00 3.47 St d. Deviation .76 1.30 .96 . .94 Between-groups variation St d. Error .29 .58 .48 . .23 Lower Bound 2.59 2.18 1.73 . 2.99 Within-groups variation Upper Bound 3.98 5.42 4.77 . 3.96 Minimum 2 2 2 4 2 Maximum 4 5 4 4 5 F (3,13) = .420, =.742 ANOVA Records of students ' progress Between Groups W ithin Groups Total Sum of Squares 1.257 12.979 14.235 df 3 13 16 Mean Square .419 .998 F .420 Sig. .742 RESULTS An F ratio of .420 has been given, with a probability of =.742. This tells us that there is no statistically significant difference between any of the groups. EFFECT SIZE: PARTIAL ETA SQUARED Partial eta squared ( 2 partial ) SS effect SS effect SS error SSeffect = The sums of the squares for whatever effect is of interest; SSerror = the sums of the squares for whatever error term is associated with that effect. EFFECT SIZE: PARTIAL ETA SQUARED FOR INDEPENDENT SAMPLES IN SPSS Analyze General Linear Model Univariate Options Estimates of effect size EFFECT SIZE: PARTIAL ETA SQUARED IN SPSS Between-Subjects Factors Value Label Which group are you 1 N Control group 166 2 Experimental Group One 166 3 Experimental Group Two 168 EFFECT SIZE: PARTIAL ETA SQUARED IN SPSS Tests of Between-Subjects Effects Dependent Variable:Mathematics post-test score Source Type III Sum of Squares Corrected Model Intercept 48.583a 41113.093 2 1 group 48.583 2 Error 521.105 497 Total 41684.000 500 569.688 499 Corrected Total df Mean Square F 24.291 23.168 41113.093 39211.30 1 24.291 23.168 1.049 a. R Squared = .085 (Adjusted R Squared = .082) Sig. Partial Eta Squared .000 .000 .085 .987 .000 .085 THE POST HOC TUKEY TEST • The null hypothesis for the F-test ANOVA is always that the samples come from populations with the same Mean (i.e., no statistically significant differences): H0 = μ1 = μ2 = μ3 = … • If the p-value is so low that we reject the null hypothesis, we have decided that, at least one of these populations has a mean that is not equal to the others; • The F-test itself only tells us that there are differences at least between one pair of means, not where these differences lie. POST HOC TESTS • To determine which samples are statistically significantly different; after having performed the F-test and rejected the null hypothesis, we turn to post hoc comparisons; • The purpose of a post hoc analysis is to find out exactly where those differences are; • Post hoc tests allow us to make multiple pair wise comparisons and determine which pairs are statistically significantly different from each other and which are not. THE POST HOC TUKEY TEST Tukey’s Honestly Significant Difference (HSD) Test is used to test the hypothesis that all possible pairs of means are equal; Tukey’s HSD test compares the mean differences between each pair of means to a critical value. If the mean difference from a pair of means exceeds the critical value, we conclude that there is a significant difference between these pairs. THE SCHEFFE TEST The Scheffe test is very similar to the Tukey hsd test, but it is more stringent that the Tukey test in respect of reducing the risk of a Type I error, though this comes with some loss of power – one may be less likely to find a difference between groups in the Sheffe test. FINDING PARTIAL ETA SQUARED IN SPSS Multivariate Testsb Effect scores Pillai's Trace Wilks' Lambda Hotelling's Trace Roy's Largest Root scores * Pillai's Trace group Wilks' Lambda Hotelling's Trace Roy's Largest Root a. Exact statistic b. Design: Intercept + group Within Subjects Design: scores Value .675 .325 2.079 2.079 .040 .960 .042 .042 Hypothesis F df 1033.477a 1.000 1033.477a 1.000 1033.477a 1.000 1033.477a 1.000 10.366a 2.000 10.366a 2.000 10.366a 2.000 10.366a 2.000 Error df 497.000 497.000 497.000 497.000 497.000 497.000 497.000 497.000 Sig. .000 .000 .000 .000 .000 .000 .000 .000 Partial Eta Squared .675 .675 .675 .675 .040 .040 .040 .040 USING TUKEY TO LOCATE DIFFERENCE IN SPSS Multiple Comparisons MEASURE_1 Tukey HSD 95% Confidence Interval Mean Difference Std. Lower Error Sig. Bound (I-J) * -.42 .084 .000 -.61 -.15 .084 .173 -.35 .42* .084 .000 .22 .27* .084 .005 .07 .15 .084 .173 -.05 -.27* .084 .005 -.46 (I) Which group are you (J) Which group are you Control group Experimental Group One Experimental Group Two Experimental Control group Group One Experimental Group Two Experimental Control group Group Two Experimental Group One Based on observed means. The error term is Mean Square(Error) = .588. *. The mean difference is significant at the .05 level. Upper Bound -.22 .05 .61 .46 .35 -.07 USING TUKEY TO LOCATE DIFFERENCE IN SPSS MEASURE_1 Tukey HSDa,,b,,c Which group are you N Subset 1 Control group 166 7.86 Experimental Group Two 168 8.01 Experimental Group One 166 Sig. 2 8.27 .174 Means for groups in homogeneous subsets are displayed. Based on observed means. The error term is Mean Square(Error) = .588. a. Uses Harmonic Mean Sample Size = 166.661. b. The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed. c. Alpha = .05. 1.000 CHI-SQUARE • A measure of a relationship or an association developed by Karl Pearson in 1900; • Measures the association between two categorical variables; • Compares the observed frequencies with the expected frequencies; • Determines whether two variables are independent; • Allows us to find out whether various subgroups are homogeneous. TYPES OF CHI-SQUARE • One-variable Chi-Square (goodness-of-fit test) – used when we have one variable; • Chi-Square test for independence: 2 x 2 – used when we are looking for an association between two variables, with two levels, e.g. the association between (drinking alcohol/does not drink alcohol) and (smoke/does not smoke); • Chi-Square test for independence: r x c – used when we are looking for an association between two variables, where one has more than two levels (heavy smoker, moderate smoker, does not smoke) and (heavy drinker, moderate drinker, does not drink). FORMULA FOR CHI-SQUARE 2 = (O E ) E 2 Where: O = observed frequencies E = expected frequencies = the sum of ONE-VARIABLE CHI-SQUARE OR GOODNESS-OF-FIT TEST • Enables us to discover whether a set of obtained frequencies differs from an expected set of frequencies; • One variable only; • The numbers that we find in the various categories are called the observed frequencies; • The numbers that we expect to find in the categories, if the null hypothesis is true, are the expected frequencies; • Chi-Square compares the observed and the expected frequencies. EXAMPLE: PREFERENCE FOR CHOCOLATE BARS A sample of 120 people were asked which of four chocolate bars they preferred; • We want to find out whether some brands (or one brand) are preferred over others – Research Hypothesis; • If some brands are not preferred over others, then all brands should be equally represented – Null Hypothesis; • If the Null Hypothesis is true, then we expect 30 (120/4) people in each category ONE-VARIABLE CHI-SQUARE OR GOODNESS-OF-FIT TEST Frequencies Chocolate A Chocolate B Chocolate C Chocolate D Observed 20 70 10 20 Expected 30 30 30 30 If all brands of chocolate are equally popular, the observed frequencies will not differ much from the expected frequencies; If, however, the observed frequencies differ a lot from the expected frequencies, then it is likely that all brands are not equally popular; ONE-VARIABLE CHI-SQUARE/GOODNESSOF-FIT TEST Observed N Expected N Residual (Difference between observed and expected frequencies) Brand A 20 30 -10.0 Brand B 70 30 40.0 Brand C 10 30 -20.0 Brand D 20 30 -10.0 120 120 Total Chocolate Chisquare df Asymp. Sig 73.333 3 .000 A chi-square value of 73.3, df = 3 was found to have an associated probability level of 0.000. A statistically significant difference was found between the observed and the expected frequencies, i.e. all brands of chocolate are not equally popular. More people prefer chocolate B (70) than the other bars of chocolate. CHI-SQUARE TEST FOR INDEPENDENCE (BIVARIATE): 2 X 2 Enables us to discover whether there is a relationship or association between two categorical variables of 2 levels; If there is no association between the two variables, then we conclude that the variables are independent of each other. A WORKED EXAMPLE Imagine that we have asked 110 students the following: A. Do you smoke and drink? B. Do you smoke but do not drink? C. Do you not smoke but drink? D. Do you abstain from both? Each student can only fall into one group, and thus we have 4 groups (they must be mutually exclusive); CHI-SQUARE TEST FOR INDEPENDENCE: 2 X 2 (WITH SPSS) Do you drink? * Do you smoke? Crosstabulation Do you smoke? Yes No Do you drink? Yes Count 50 15 Expected Count 41.4 23.6 No Count 20 25 Expected Count 28.6 16.4 Total Count 70 40 Expected Count 70.0 40.0 Total 65 65.0 45 45.0 110 110.0 row total x column total Expected value of a cell Overall total CHI-SQUARE TEST FOR INDEPENDENCE: 2 X 2 (WITH SPSS) Value df Asymp. Sig. Pearson Chi-Square 12.12 1 .000 Continuity Correction 10.759 1 .001 Likelihood Ratio 12.153 1 .001 Fisher’s Exact Test Exact Sig. (2-sided) .001 Linear-by-Linear Association 12.011 N of Valid Cases 110 1 Exact Sig. (1-sided) .001 .001 Chi-Square = 12.12 df (degrees of freedom) = (columns -1) x (rows -1) = (2-1) x (2-1) = 1 RESULTS A 2 x 2 Chi-square was carried out to discover whether there was a significant relationship between smoking and drinking. The Chisquare value of 12.12 has an associated probability value of p<0.001, df = 1, showing that such an association is extremely unlikely to have arisen as a result of sampling error. It can therefore be concluded that there is a significant association between smoking and drinking. MANN-WHITNEY U-TEST FOR INDEPENDENT SAMPLES • Mann-Whitney (non-parametric, nominal and ordinal data) for two groups under one condition – Difference between two independent groups (independent samples), based on ranks • This is the non-parametric equivalent of the t-test for independent samples. • Find the significant differences and then run a crosstabs to look at where the differences lie. • Note where there are NO statistically significant differences as well as where there are statistically significant differences MANN-WHITNEY U-TEST (SPSS) Ranks the contents are interesting form Primary 3 Primary 4 Total N 22 64 86 Mean Rank 43.52 43.49 Sum of Ranks 957.50 2783.50 MANN-WHITNEY U-TEST (SPSS) Test Statisticsa the contents are interesting 703.500 2783.500 -.006 Mann-Whitney U Wilcoxon W Z Asymp. Sig. .996 (2-tailed) a. Grouping Variable: form THE WILCOXON TEST FOR RELATED SAMPLES This is the non-parametric equivalent of the t-test for related samples. For paired (related) samples in a non-parametric test, e.g. the same group under two conditions. For example, here is the result for one group of females who have rated (a) their own ability in mathematics and (b) their enjoyment of mathematics, both variables using a 5-point scale (‘not at all’ to ‘a very great deal’). THE WILCOXON TEST FOR RELATED SAMPLES (SPSS) Ranks How good at mathematics do you think you are? How much do you enjoy mathematics? Negative Ranks Positive Ranks Ties N Mean Rank Sum of Ranks 11 94.08 11101.00 73 99.11 7235.00 57 Total 248 Test Statisticsb How good at mathematics do you think you are? - How much do you enjoy mathematics? Z Asymp. Sig. (2-tailed) a. Based on positive ranks. b. Wilcoxon Signed Ranks Test -2.631a .009 KRUSKAL-WALLIS TEST FOR INDEPENDENT SAMPLES • Kruskal-Wallis (non-parametric, nominal and ordinal data) for three or more independent groups under one condition – Difference between more than two independent groups (independent samples), based on ranks • This is the non-parametric equivalent of ANOVA for independent samples. • Find the statistically significant differences and then run a crosstabs to look at where the differences lie. • Note where there are NO statistically significant differences as well as where there are statistically significant differences. KRUSKAL-WALLIS TEST (SPSS) Ranks own made-up tests Age 20-29 30-39 40-49 Total N 7 5 5 17 Mean Rank 6.57 10.70 10.70 KRUSKAL-WALLIS TEST (SPSS) Test Statisticsa, b Chi-Square df Asymp. Sig. own made-up tests 4.319 2 .115 a. Kruskal Wallis Test b. Grouping Variable: Age THE FRIEDMAN TEST FOR 3 OR MORE RELATED GROUPS This is the non-parametric equivalent of ANOVA for related samples. For three or more related samples in a non-parametric test, e.g. the same groups under two conditions. For example, the result for 4 groups of students, grouped according to their IQ (Group 1= IQ up to 90; Group 2 = IQ from 91-110; Group 3 = IQ from 111-125; Group 4 = IQ over 125) who have rated (a) their own ability in mathematics and (b) their enjoyment of mathematics, both variables using a 5-point scale (‘not at all’ to ‘a very great deal’).