Validity and Reliability Validity and Reliability Illustration of Types of Evidence of Validity (Figure 8.1) Reliability and Validity 8.2) (Figure Reliability of Measurement (Figure 8.3) Methods of Checking Validity and Reliability (Table 8.2) Validity (“Truthfulness”) Method Procedure Content-related evidence Criterion-related evidence Construct-related evidence Expert judgment Relate to another measure of the same variable Assess evidence on predictions made from theory Reliability (“Consistency”) Method Content Time Interval Procedure Test-retest Equivalent forms Equivalent forms/retest Identical Different Different Varies None Varies Internal consistency Different None Observer agreement Identical None Give identical instrument twice Give two forms of instrument Give two forms of instrument, with time interval between Divide instrument into halves and score each or use KR Compare scores obtained by two or more observers Reliability Worksheet (Figure 8.5) Educational Research Chapter 13 Experimental Research Gay and Airasian 13 Experimental Research Experimental Research Key Characteristics of Experimental Designs Procedures are designed that address potential threats to validity Internal External Construct Statistical Conclusion Statistical comparisons of different groups are conducted Selecting Participants and Assigning Them to Treatments Decide on the experimental unit of analysis to be treated individual group or groups organization Randomly assign individuals to groups control for extraneous characteristics that might influence the outcome Topics Discussed in this Chapter Defining characteristics The experimental process Manipulation and control Threats to validity Internal validity External validity Group designs Single subject designs Defining Characteristics Research designed to investigate cause and effect relationships through the direct manipulation of an independent variable and control of extraneous variables Independent variable – the variable being manipulated Dependent variable – the variable in which the effect of the manipulation of the independent variable are observed Researcher manipulation and control – choice of treatments, choice of a research design, use of specific procedures, etc. Experimental Process Selection and definition of problem Selection of participants & instruments Selection of research plan Comparison of two approaches Comparison of new versus existing approaches Comparison of different amounts of single approach Execution of research plan Selection and assignment of participants Sufficient exposure to treatment Operationally different treatments Analysis of data Formulation of conclusions Manipulation and Control Manipulation The researcher’s decisions related to what will make up the independent variable Active variables versus assigned variables Control The researcher’s efforts to remove the influence of any extraneous variables that might have an effect on the dependent variable The goal is to be assured the only differences between groups is that related to the independent variable 9 Internal Validity Experimental Validity Internal validity – the degree to which the results are attributable to the independent variable and not some other rival explanation External/ecological validity – the extent to which the results of a study are generalizable Relative importance of internal and external validity Internal Validity Threats to Internal Validity History Maturation Testing Instrumentation Statistical regression Differential selection of participants Mortality Selection-maturation interaction, etc. Mortality Threat to Internal Validity (Figure 9.1) Location Might Make a Difference (Figure 9.2) Instrument Decay (Figure 9.3) A Data Collector Characteristics Threat 9.4) (Figure A Testing Threat to Internal Validity (Figure 9.5) A History Threat to Internal Validity (Figure 9.6) Could Maturation be at Work Here? (Figure 9.7) The Attitude of Subjects Can Make a Difference (Figure 9.8) Regression Rears Its Head (Figure 9.9) Illustration of Threats to Internal Validity (Figure 9.10) Techniques for Controlling Threats to Internal Validity (Table 9.1) Threat Subject characteristics Mortality Location Instrumentation Testing History Maturation Subject attitude Regression Implementation Technique Obtain More Standardize Conditions Obtain More Information on Subjects Choose Information on Details X X X X X X X X X X X X X X Appropriate Design X X X X X X X Threats to External Validity Pre-test treatment interaction Multiple treatment interference Selection treatment interaction Specificity of variables Treatment diffusion Experimenter effects Reactive arrangements Artificial environment Hawthorne effect John Henry effect Placebo effect Novelty effect Controlling for Extraneous Variables Randomization Selection Assignment Matching Identifying pairs of subjects “matched” on specific characteristics of interest Randomly assigning subjects from each pair to different groups Difficulty with subjects for whom no match exists A Randomized Posttest-Only Control Group Design, Using Matched Subjects (Figure 13.7) Matching Process Based on Gender John Jim James Josh Jackson Jane Johanna Julie Jean Jeb Experimental Group Control Group Controlling for Extraneous Variables Comparing homogeneous groups Using subjects as their own controls Restricting subjects to those with similar characteristics Problems related to restriction of generalization Build the variable into the design (e.g., factorial design) Multiple treatments across time Problem with carry-over effect Analysis of covariance (ANCOVA) Group Designs Two major classes of designs Single-variable designs – one independent variable Factorial designs – two or more independent variables Three types of designs Pre-experimental designs Experimental designs Quasi-experimental designs Pre-Experimental Designs Types One-shot case study One-group pretest-posttest design Static group comparison Threats to internal validity – see Figure 13.1 Example of a One-Shot Case Study Design (Figure 13.1) Posttest-Only Control Group Design (Figure 13.4) Example of a Randomized Solomon Four-Group Design (Figure 13.6) Example of a One-Group Pretest-Posttest Design (Figure 13.2) Example of a Static-Group Comparison Design (Figure 13.3) Posttest-Only Control Group Design (Figure 13.4) True Experimental Designs Types Pretest-posttest control group design Posttest only control group design Solomon four-group comparison Threats to internal validity – see Figure 13.2 Pretest-Posttest Control Group Design (Figure 13.5) Quasi-Experimental Designs Types Non-equivalent control group design Time series design Counterbalanced design Threats to internal validity – see Figure 13.2 Using a Counterbalanced Design (Figure 13.8) Possible Outcome Patterns in a Time-Series Design (Figure 13.9) Using a Factorial Design to Study Effects of Method and Class Size on Achievement (Figure 13.10) Illustration of Interaction and No Interaction in a 2 by 2 Factorial Design (Figure 13.11) Example of a 4 by 2 Factorial Design (Figure 13.13) Effectiveness of Experimental Designs in Controlling Threats to Internal Validity (Table 13.1) Subject Characteristics Mortality Location Instrument Decay Data Collector Charac- Data Colteristics lector Bias – – – (NA) – One group preposttest – ? – – Static group comparison – – – Randomized posttest-only control group ++ + Randomized prepost-test control group ++ Solomon fourgroup Testing History Maturation – (NA) – – – – – – – – – – – – – + – – + ? + – – – – + – – ++ + ++ – ++ – + – + – – + + ++ – ++ – ++ ++ – + – – ++ + ++ – ++ – Randomized posttest only control group with matched subjects ++ + – + – – ++ + ++ – ++ – Matching-only pre-posttest control group + + – + – – + + + – + – Counterbalanced ++ ++ – + – – – ++ ++ ++ ++ – Time-series ++ – + _ – – – – + – ++ – Factorial with randomization ++ ++ – ++ – – + + ++ – ++ – Factorial without randomization ? ? – ++ – – + + + – ? – Design One-shot case study Atti- Regrestudinal sion Implementation KEY: (++) = strong control, threat unlikely to occur; (+) = some control, threat may possibly occur; (–) = weak control, threat likely to occur; (?) = can’t determine; (NA) = threat does not apply Single-Subject Research Designs that can be applied when the sample size is one Study behavior change in an individual as the result of some treatment Subject serves as his or her own control Rationale Sophistication of specific designs allows for the control of internal validity threats Research is focused on therapeutic impact in clinical settings, not contribution to a research base Group comparison designs are sometimes opposed or unethical Group comparison designs are not possible Single-Subject Research Concerns External validity Low generalizability due to the nature of the design The effect of the baseline condition on the subsequent effects of the treatment Threats can be lessened through replication Internal validity Possible to control for most threats Repeated and reliable measures Baseline stability Number of data points Single-variable rule Specification or the nature and conditions of the treatment Single-Subject Research Designs A B A Withdrawal ABAB Multiple Baseline Alternating Treatments Data analysis and interpretation Based on visual inspection and analysis of a graphic presentation of the results Criterion of effectiveness is clinical significance, not statistical significance Debate about the use of statistical procedures Single-Subject Research Replication Establishes the generalizability of findings Three types Direct – same researcher with the same or different participants in a specific setting Systematic – follows direct replication but with different researchers, behaviors, or settings Clinical – development of treatment packages composed of tow or more interventions which have been found to be effective individually