advertisement

The Two-Group Randomized Experiment The Basic Design R R X O O Note that a pretest is not necessary in this design. Why? Because random assignment assures that we have probabilistic equivalence between groups The Basic Design R R X O O Differences between groups on posttest indicate a treatment effect. Usually test this with a t-test or one-way ANOVA. Why no pretest? Internal Validity History Maturation Testing Instrumentation Mortality Regression to the mean Selection Selection-history Selection- maturation Selection- testing Selection- instrumentation Selection- mortality Selection- regression Diffusion or imitation Compensatory equalization Compensatory rivalry Resentful demoralization R X O R O Examples Experimental Design Variations The posttest-only two group design is the simplest; there are many variations. To better understand what the variations try to achieve, we can use the signal-to-noise metaphor. Signal to Noise What we observe can be divided into what we see Signal to Noise What we observe can be divided into Signal what we see Signal to Noise What we observe can be divided into Signal Noise what we see Signal to Noise Experimental designs can take two approaches: Signal to Noise Experimental designs can take two approaches: Signal Focus on (enhance) the signal Signal to Noise Experimental designs can take two approaches: Signal Focus on (enhance) the signal (what is this?) Signal to Noise Experimental designs can take two approaches: Signal to Noise Experimental designs can take two approaches: or reduce the noise Noise (what is this?) Signal to Noise Experimental designs can take two approaches: or reduce the noise Noise Signal to Noise Signal enhancers Noise reducers Factorial designs Covariance designs Blocking designs Noise and Signal in Significance Tests For interval and ratio dependent variables, you can conduct a difference in means test: ( X 2 X 1 ) 2 ( std .error1 ) ( std .error 2 ) 2 Signal std .error1 ˆ1 N1 Noise 2 Factorial Designs A Simple Example R R R R X11 X12 X21 X22 O O O O Factor 1: Time in Instruction Level 1: 1 hour per week Level 2: 4 hours per week Factor 2: Setting Level 1: In-class Level 2: Pull-out A Simple Example Setting Time in Instruction Factors: Major independent variables A Simple Example Time in Instruction In-class Pull-out Setting 1 hour/week 4 hours/week Levels: subdivisions of factors A Simple Example Time in Instruction factors 1 hour/week 4 hours/week In-class Group 1 average Group 3 average Pull-out Setting levels Group 2 average Group 4 average Usually, averages are in the cells. Multiplicative Notation A 3 x 4 factorial design How many factors? How many levels? How many cells with averages? Multiplicative Notation A 3 x 4 factorial design The number of numbers tells you how many factors there are. There are 2 factors because there are 2 numbers. Multiplicative Notation A 3 x 4 factorial design The number values tell you how many levels are in each factor. Multiplicative Notation A 3 x 4 factorial design The number values tell you how many levels are in each factor. Factor 1 has 3 levels. Factor 2 has 4 levels. The Null Case 4 hrs Out 5 5 5 In 5 5 5 5 5 8 7 6 5 4 3 2 in out 1hr 4hrs The lines in the graphs below overlap each other. 8 7 6 5 4 3 2 out in 4hrs 1 hr 1hr Setting Time A Main Effect A consistent difference between levels of a factor For instance, we would say there’s a main effect for time if we find a statistical difference between the averages for the hours of instruction between groups Main Effects Out 1 hr 4 hrs 5 7 6 Main Effect of Time 5 7 5 7 8 7 6 5 4 3 2 6 in out 1hr 4hrs 8 7 6 5 4 3 2 out in 4hrs In 1hr Setting Time Main Effects Out 1 hr 4 hrs 5 5 5 Main Effect of Setting 7 7 6 6 8 7 6 5 4 3 2 7 in out 1hr 4hrs 8 7 6 5 4 3 2 out in 4hrs In 1hr Setting Time Main Effects 1 hr 4 hrs Out 5 7 6 In 7 9 8 6 8 10 Main Effects of Time and Setting 10 8 1hr 4hrs 6 4 8 out in 6 4 in out 4hrs 2 2 1hr Setting Time An Interaction Effect When differences on one factor depend on the level you are on on another factor An interaction is between factors (not levels) You know there’s an interaction when can’t talk about effect on one factor without mentioning the other factor Interaction Effects 4 hrs Out 5 5 5 In 5 7 6 5 6 8 7 6 5 4 3 2 in out 1hr 4hrs The in-class, 4-hour per week group differs from all the others. 8 7 6 5 4 3 2 out in 4hrs 1 hr 1hr Setting Time Interaction Effects 4 hrs Out 7 5 6 In 5 7 6 6 6 8 7 6 5 4 3 2 in out 1hr 4hrs The 1-hour amount works well with pull-outs while the 4 hour works as well with in class. 8 7 6 5 4 3 2 out in 4hrs 1 hr 1hr Setting Time Advantages of Factorial Designs Offers great flexibility for exploring or enhancing the “signal” (treatment) Makes it possible to study interactions Combines multiple studies into one Randomized Block Designs The Basic Design R R X O O The Basic Design R R X O O R R X O O R R X O O R R X O O Homogeneous groups The Basic Design R R X O O R R X O O R R X O O R R X O O Homogeneity on the dependent variable (observations in group one tend to have higher levels of the dependent variable than observations in group two, etc.) Homogeneous groups Randomized Blocks Design R R X O O R R X O O R R X O O R R X O O Each replicate is a block. Randomized Blocks Design Can block before or after the study Can block on a measured variable or on unmeasured perceptions Is a noise-reducing strategy How Does Blocking Reduce Noise? Posttest 100 90 80 70 60 50 40 30 20 20 30 40 50 Pretest 60 70 80 How Does Blocking Reduce Noise? Range of x Posttest 100 90 80 70 60 50 40 30 20 20 30 40 50 Pretest 60 70 80 How Does Blocking Reduce Noise? Range of x Posttest 100 90 Variability of y 80 70 60 50 40 30 20 20 30 40 50 Pretest 60 70 80 How Does Blocking Reduce Noise? Range of x Posttest 100 80 Variability of y Mean difference 90 70 60 50 40 30 20 20 30 40 50 Pretest 60 70 80 How Does Blocking Reduce Noise? For block 3 Posttest 100 90 80 70 60 50 40 30 20 20 30 40 50 Pretest 60 70 80 How Does Blocking Reduce Noise? For block 3 Range of x in block Posttest 100 90 80 70 60 50 40 30 20 20 30 40 50 Pretest 60 70 80 How Does Blocking Reduce Noise? For block 3 Range of x in block Posttest 100 Variability of y in block 90 80 70 60 50 40 30 20 20 30 40 50 Pretest 60 70 80 How Does Blocking Reduce Noise? For block 3 Range of x in block Posttest Variability of y in block Mean difference 100 90 80 70 60 50 40 30 20 20 30 40 50 Pretest 60 70 80 How Does Blocking Reduce Noise? Posttest For block 3 Range of x in block Variability of y in block Mean difference 100 90 80 70 60 50 40 30 20 Same mean difference, 20 30 40 50 but lower variability on both x and y Pretest 60 70 80 Conclusion Instead of having one treatment effect based on the full variability of y, you have three treatment effects based on reduced variability (but with the same mean difference). The average of the three estimates gives a less noisy estimate than the nonblock one. Analysis of Covariance Experimental Designs Design Notation R R O O X O O Uses a pre-measure Can be a pretest, but doesn’t have to be Can have multiple covariates The Covariance Design Posttest 100 90 80 70 60 50 40 30 20 20 30 40 50 Pretest 60 70 80 The Covariance Design Posttest The range of y is about 70 points. 100 90 80 70 60 50 40 30 20 20 30 40 50 Pretest 60 70 80 How Does a Covariate Reduce Noise? Posttest 100 We fit regression lines to describe the pretest-posttest relationship. 90 80 70 60 50 40 30 20 20 30 40 50 Pretest 60 70 80 How Does a Covariate Reduce Noise? Posttest 100 90 80 70 60 50 40 30 20 20 30 40 50 We want to “adjust” the posttest scores for 60pretest 70 variability. 80 Pretest How Does a Covariate Reduce Noise? Posttest 100 We do this by “subtracting 90 out” the pretest -- by 80 “subtracting out” the line. 70 60 50 40 30 20 20 30 40 50 We want to “adjust” the posttest scores for 60pretest 70 variability. 80 Pretest How Does a Covariate Reduce Noise? Posttest 100 We do this by “subtracting 90 out” the pretest -- by 80 “subtracting out” the line. 70 Get the difference between the line and each point. 60 50 40 30 20 20 30 40 50 Pretest 60 70 80 How Does a Covariate Reduce Noise? Here is the plot with the effect of the pretest removed. Notice the range on y is now only about 50 points instead of 70 (although the difference between the means remains the same). 40 30 20 10 0 -10 -20 -30 -40 20 30 40 50 Pretest 60 70 80 Summary The Analysis of Covariance adjusts posttest scores for variability on the covariate (pretest). This is what we mean by adjusting for the effects of one variable on another. Summary You can use any continuous variable as the covariate, but the pretest is usually best. You can use multiple covariates, but if they are highly intercorrelated, you don’t improve the adjustment (and you pay a price for each covariate). Hybrid Experimental Designs Hybrid Designs Variations on randomized designs Help to address specific threats Incorporates different design features Two will be illustrated here. The Solomon Four-Group Design R R R R O O X X O O O O Explicitly addresses a testing threat Assesses the effect of taking a pretest Possible Outcomes Treatment Effect -- No Testing Effect 18 16 14 Tpre-post Cpre-post Tpost Cpost 12 10 8 6 Pre Post Possible Outcomes Treatment Effect and Testing Effect 18 16 14 Tpre-post Cpre-post Tpost Cpost 12 10 8 6 Pre Post Switching Replications Design R R O O X O O X O O Switching: the groups switch roles over the course of the study. Replications: the treatment is repeated or replicated. Possible Outcomes Short-term Persistent Treatment Effect 18 16 14 Group 1 Group 2 12 10 8 6 Pre Post1 Post2 Possible Outcomes Long-term Continuing Treatment Effect 18 16 14 Group 1 Group 2 12 10 8 6 Pre Post1 Post2 Switching Replications Very strong in internal validity Looks at short and longer term effects Strong ethically because all participants get the treatment Works well with some institutional structures (for example, semester system in schools)