Educational Research 250:205 Writing Chapter 3 – Part II Chapters 9 and 13: Procedures and Experimental Design I. A Systematic Process for Writing the Methods Chapter: Subjects. Instrumentation. Procedures. Experimental Design. Statistical Analysis. II. Procedures: A. Describes how the data was collected: 1. This section of the procedures is the most detailed section. a. In general, the following areas should be included: A. B. C. D. E. b. The importance of pilot testing: B. Procedures section should be used in descriptive, associational and intervention research. III. Experimental Design: A. What is it? B. Validity: 1. As you may already know, the term validity is used in a variety of contexts in research. You have already learned about instrument validity: a. Instrument validity: Does your instrument measure what it was intended to measure. 2. There are two other types that are based on the experiment itself as opposed to just the instrument: a. Internal validity: b. External validity: 3. There is a tension between internal and external validity: C. Internal Validity: Minimizing threats via experimental design. 1. There are many threats to the internal validity of your experiment and your weapons against these threats are “random sampling, sound procedures and sound experimental design.” a. The threats to internal validity include: i. Subject characteristics: Examples: How can you control for subject characteristics: ii. Mortality: Example: How can you control for mortality? iii. Location: iv. How can you control for location: Instrumentation Decay: Examples: How can you control for instrumentation? v. Data Collector Characteristics: Example: How can you control for data collector characteristics? vi. Researcher bias: vii. How to prevent researcher bias? Multiple testing: Example: How can you control for multiple testing? viii. History: Example: How can you control for history? ix. Maturation: Example: How can you control for maturation? x. Sampling methodology: Example: How can you control for sampling methodology? xi. Experimental treatment diffusion: Example: How do you control for experimental treatment diffusion? xii. Compensatory rivalry: Example: xiii. How do you control for compensatory rivalry? Resentful demoralization: Example: How do you control for resentful demoralization? Hawthorne effect: How to prevent this threat to internal validity? xiv. D. External Validity: Minimizing threats. 1. External validity is threatened as researcher attempts to increase the control of the experiment. This is not necessarily bad. E. Experimental Designs: 1. Experimental design describes how the researcher manipulates treatments and observes the effects. 2. Experimental design is the most powerful tool for minimizing threats to internal validity. 3. Terminology: a. Independent variable: b. Dependent variable: 4. In other words, the researcher is trying to determine the effect of the “independent variable/treatment on the dependent variable/measurement.” 5. There are many different types of experimental designs: a. Single-Group Designs: i. One-Shot Case Study: ii. One-Group PreTest Posttest Design: b. Quasi-Experimental Designs: Almost like true experiments (both contain control groups) however the subjects in the study have not been chosen at random. Groups are typically intact (convenience samples). These designs are typical in the classroom. i. Quasi version of the Single-Shot Case Study: Static-Group Comparison Design: ii. Quasi version of the One-Group Pretest Posttest Design: Static Group Pretest-Posttest Design: c. Experimental Designs: The next step up from the QuasiExperimental Design. Involves randomization of subjects which is harder than you think. i. Randomized Posttest-Only Control Group Design: ii. Randomized Pretest Posttest Control Group Design: d. Common Extensions to Research Designs: 6. Other experimental design solutions for educational research: a. Education researchers must often perform research on intact groups (classes) and are not able to use control groups (everyone wants the treatment). Here are some other designs that are used in educational research: i. Matching-Only Design: Used when randomization is impossible. ii. Counterbalanced Design: Used when everyone wants the treatment. iii. Time Series Design: Used when a control group is impossible.