Experimental Design
1
http://thefifthlevel.blogspot.com/2011/05/prologue-genesis.html
Experimental Design
• Strongest design with respect to internal
validity
2
If X then Y
and
If not X, then not Y
or
If the program is given, then the outcome occurs
and
If the program is not given, then the outcome does
not occur
3
Dilemma
• 2 identical groups
• 2 identical contexts
• Same time
• ….
similarity
4
Course of Action
• Randomly assign people from a pool to the 2
groups
–  probabilistically equivalent
• One group gets the treatment and the other
does not
5
Random Selection and Assignment
• Random selection is how you draw the
sample of people for your study from a
population.
• Random assignment is how you assign the
sample that you draw to different groups or
treatments in your study.
Probabilistic Equivalence
• Means that we know perfectly the odds that
we will find a difference between two groups.
• When we randomly assign to groups, we can
calculate the chance that the two groups will
differ just because of the random assignment.
7
External validity
• Experiments are difficult to carry out
•  artificial situation  high internal validity
• Limited generalization to real contexts –>
limited external validity
?
8
Two-Group Experimental Design
• Simplest form: two-group posttest-only
randomized experiment
• No pretest required
• Test for differences: t-test or ANOVA
9
Advantages
• Strong against single-group threats and multigroup threats (except selection-mortality)
• Strong against selection testing and selectioninstrumentation
10
Classifying Experimental Designs
• Two components: signal and noise
• signal-enhancing experimental design (factorial
design)
• Noise-reducing experimental design (covariance
designs or blocking designs)
11
Factorial Designs
• A factor is a major independent variable
– Time and setting
• A level is a subdivision of a factor.
– Time (1h/4h), setting (pull-out/in-class)
• 2 x 2 factorial design
12
Factorial Design
•
•
•
•
X11 = 1h and in-class
X12 = 1h and pull-out
X21 = 4h and in-class
X22 = 4h and pull-out
13
The Null Outcome
• The null case is a situation where both
treatments have no effect.
14
The Main Effects
A main effect is an outcome that is a consistent
difference between levels of a factor.
15
Main Effects
16
Main Effects
17
Interaction Effects
• An interaction factor exists when differences
on one factor depending on the level of the
other factor.
18
How do you know if there is an
interaction in a factorial design?
• Statistical analysis
• When it can be talked about one factor
without mentioning the other factor
• In graphs of group means – the lines are not
parallel
19
Interaction Effects
20
Interaction Effects
21
Factorial Design Variations
• 2 x 3 Example
Factor 1: Treatment
– psychotherapy
– behavior modification
Factor 2: Setting
– inpatient
– day treatment
– outpatient
22
Main Effects
23
Main Effects
24
Interaction Effect
25
Interaction Effect
26
Factorial Design Variations
• A Three-Factor Example (2 x 2 x 3)
Factor 1: Dosage
– 100 mg
– 300 mg
Factor 2: Treatment
– Psychotherapy
– Behavior modification
Factor 3: Setting
– Inpatient
– Day treatment
– Outpatient
27
2 x 2 x 3 Design
28
Incomplete Factorial Design
• Common use is to allow for a control or
placebo group that receives no treatment
29
Randomized Block Design
• Stratified random sampling
• To reduce noise or variance in the data
• Division into homogeneous subgroups
• Treatment implemented to each subgroup
• Variability within each block is less than the
variability of the entire sample or each block is
more homogenous than the entire group
30
Randomized Block Design
• Stundents are a homogenous group with
exception of semester
freshman
sophomore
junior
senior
31
How blocking reduces noise?
32
Covariance Designs (ANCOVA)
• Pretest-posttest randomized design
• Pre-program measure = covariate
• Covary it with the outcome variable
• Covariates are the variables you adjust for
– Effect is going to be removed
33
How does a Covariate reduce Noise?
34
How does a Covariate reduce Noise?
35
How does a Covariate reduce Noise?
36
How does a Covariate reduce Noise?
37
How does a Covariate reduce Noise?
38
Hybrid Experimental Designs
• Are new strains that are formed by combining
features of more established designs.
39
The Solomon Four-Group Design
• Is designed to deal with a certain testing
threat
• 2 groups are pre-tested, 2 are not
• 2 groups get a treatment, 2 do not
40
The Solomon Four-Group Design
• T = Treatment Group, C = Control Group
41
The Solomon Four-Group Design
• T = Treatment Group, C = Control Group
42
Switching Replication Design
• The implementation of the treatment is
repeated or replicated.
• In the repetition, the two groups switch roles
• Finally, all participants have received the
treatment
• Reduces social threats
43
Switching Replication Design
• Period 1 – group 1 gets the treatment
• Period 2 – group 2 gets the treatment
44
Switching Replication Design
• Longterm treatment effect  group 1 improves
even though no further treatment was given
45