Quasi-Experimental Design Jung Eun (Jessie) Hong Feb. 23, 2009 Outlines Experimental Design Definition Process A Key Point Types of Experimental Designs Quasi-Experimental Design Designs Strengths Weaknesses Examples in Geography Ongoing Debate Definition of Experimental Design A structured, organized method To determine whether some program or treatment causes some outcome or outcomes to occur. If X, then Y Because there may be lots of reasons, other than the program, for why you observed the outcome, If not X, then not Y needs to be addressed, too Process of Experimental Design To show that there is a casual relationship, Two “equivalent” groups The program or treatment group gets the program The comparison or control group does not The groups are treated the same in all other respects Differences in outcomes between two groups must be due to “the program” A Key Point of Experimental Design How do we create two groups that are “equivalent”? Assign people randomly from a common pool of people into the two groups The experiment relies on the idea of “random assignment” to obtain two similar groups. A key to the success of the experiment Assume that two groups are “probabilistically equivalent” Types of Designs Is random assignment used? Yes Randomized or True experiment No Is there a control group or multiple measures? Yes Quasi-experiment No Non-experiment Quasi-Experimental Design Similar to the experimental design, but lacks the key ingredient, “random assignment” Easily and more frequently implemented Extensively used in the social sciences A useful method for measuring social variables Two classic quasi-experimental designs The Nonequivalent Groups Design The Regression-Discontinuity Design The Nonequivalent Groups Design The most frequently used in social research Try to select groups that are as similar as possible to compare the treated one with the comparison one e.g. two comparable classrooms or schools Cannot be sure whether the groups are comparable The groups may be different prior to the study Any prior differences between the groups may affect the outcome of the study Require a pretest and posttest The Regression-Discontinuity Design A useful method for determining whether a program of treatment is effective Participants are assigned to program or comparison groups based on a cutoff score on a pretest e.g. Evaluating new learning method to children who obtained low scores at the previous test. Cutoff score = 50 The treatment group: children who obtained 0 to 50 The comparison group: children who obtained 51 to 100 The program (treatment) can be given to those most in need The Regression-Discontinuity Design With no treatment effect With Ten point treatment effect The Regression-Discontinuity Design Discontinuity Strengths of Quasi-Experimental Design Useful in generating results for general trends in social sciences Easily integrated with individual case studies Difficult pre-selection and randomization of groups Generated results can reinforce the findings in a case study Allow statistical analysis to take place Enable to reduce the time and resources required for experimentation Not required extensive pre-screening and randomization Weaknesses of Quasi-Experimental Design Without proper randomization, statistical tests can be meaningless Do not explain any pre-existing factors and influences outside of the experiment The researcher needs to control additional factors that may have affected the results Some form of pre-testing or random selection may be necessary to explain statistical results thoroughly Quasi-experiments vs. Non-experiments to address similar questions Both designs are applicable when the subjects are not able to be randomized Some variables cannot ethically be randomized e.g. Studying the effect of maternal alcohol use when the mother is pregnant Quasi-experiments Non-experiments Strengths Enable to compare with other groups Enable to focus on one variable Weaknesses Unexpected factors might affect the results Interpretations might be improper Example of Quasi-Experimental Design in Geography Baker and White (2003) The Effects of GIS on Students’ Attitudes, Self-efficacy, and Achievement in Middle School Science Classrooms Conducted the Nonequivalent quasi-experimental design Two eighth grade teachers, across ten classrooms Total 192 eighth grade students participated Treatment group: used a Web-based GIS application Control group: used paper maps Treatment Group Instructor 1 51 Instructor 2 42 Control Group 36 63 Example of Quasi-Experimental Design in Geography Impossible to randomly assign each student to a GIS or paper mapping conditions Randomly assigned whole classes to two conditions Different instructors affected the results differently Instructor effect played a substantial role in student attitudes and self-efficacy Ongoing Debate Whether true experiments or quasi-experiments represents the superior design Supporters of true experiments Difficult to isolate the program effects using quasiexperiments Quasi-experimental results are biased and sensitive to minor changes Not sure about whether quasi-experimental designs can adequately control selection bias Hard to determine better design True experiments are impossible and impractical in some cases Any questions???