Tonight
Begin experimental design
? Krathwohl, Chapter 20
? Experimental Design
?
Experimental Design
– Increases Internal Validity
– Weakens External Validity
HD FS 503: Research Methods
Susan Hegland
March 11, 2002
1
2
If HD FS Foundation
gave you $100,000...
Why study experimental design?
Most of us enter HD FS to become
interventionists, or change experts
? Most of us eventually need to evaluate
programs designed to change something
? Finally, for the pure theorist/researcher:
? To understand something
...Try to change it!
To demonstrate that a given program
was effective in helping persons change...
? What would you implement?
? How would you show that this program
was effective?
?
?
3
4
Introduction
Basic Principles of Experiments
I. Basic Principles of Experiments
»tonight
? II. Complex Designs
»next week
Translating hypothesis to design
Controlling for unwanted explanations
? Common control problems
? Common control solutions
?
?
?
5
6
1
General Methods of Control
Remove and exclude
Exclude all variables that could contaminate
results
? Kostelnik (male) to control for additional male
(observer)
?
Remove and exclude
? Measurement and adjustment
? Spreading effect to control groups
? USE RANDOM ASSIGNMENT!!!
?
– observing father-infant attachment in the home
– Moved a TV box with red cellophane over hole
– After 2 weeks, sat inside and observed fatherinfant interaction
– Later: observed mother-infant interaction
– e.g., in Transition study
7
?
Providing treatment to waiting list
Spreading effect to
control (comparison) groups
Measurement and adjustment
?
?
E.g., control for pretest scores, age
Coleman (1982): controlled for SES; compared
public and private schools
Make sure that groups are similar on every
dimension except treatment!
? Give placebo to control (“comparison”)
group
?
– Is this possible?
?
Safest to covary for scores when
groups don’t differ on the variable!
?
More typically: analyze for the effect of any
other variable as Independent Variable
– pretest
– “special” program and label
– innocuous (and cheap) treatment
– control for individual differences
?
9
Give alternate treatment (known to be
effective) to comparison group
USE RANDOM
ASSIGNMENT!
Randomly assign participants to treatment
and control groups
Why?
? Uncontrolled variables randomly
distributed between treatment and
control
? Controls for anticipated influences
(e.g., income, education)
? Also controls for unanticipated influences
8
10
Independent Variable Designs
1: Case Study
? 2: One-group pretest-posttest
? 3: Nonequivalent control group
? 4: Pretest-posttest control group design
? 5: Posttest only control group design
?
11
12
2
1: Case study
2: One-group pretest-posttest
X O
O X O
13
14
4. Pretest-posttest
control group design
3. Nonequivalent control group
O X O
R
O X O
O C O
O C O
15
5. Posttest-only
control group design
R
16
Solomon four-group design
X O
C O
R
17
O X O
O C O
X O
C O
18
3
Solomon Four-group Design
Factorial Designs
Two or more categorical, independent
variables (factors)
? Each factor is studied at two or more
levels
e.g. Pretest (yes, no), Training (yes, no)
? Goal: to determine whether factors
combine to produce interaction effects
? If no interaction, effects can be studied
one at a time (main effects)
?
Treatment?
Yes
No
Yes
Group 1
Group 2
No
Group 3
Group 4
Pretest?
19
Describe significant
interaction and main effects
Example:
Solomon 4-group design
Study of elementary school children
learning child abuse prevention concepts
(Tutty, 1992)
? Four groups
?
Treatment?
Yes
No
Yes
Group 1
Group 2
No
Group 3
Group 4
– 2 Treatment groups (experiment &
no experiment)
– 2 Pretest groups (pretest & no pretest)
– Test for main effects (Experiment, Pretest)
– Test for interaction (Experiment X Pretest)
Pretest?
21
R
X
C
X
C
22
Figure 1: 2 x 2 C.K.A.Q. Posttest mean
scores (N = 398)
Explain each group
1. O
2. O
3.
4.
20
79
78
77
76
75
74
73
72
71
70
69
O
O
O
O
Experimental
Control
No Pretest
23
Pretest
24
4
Analyzed using
a two-way ANOVA
2 X 2 Factorial Design
(Tutty, 1992)
Interaction F-term tests pretest
sensitization
? If interaction is not significant,
?
Source
– pretest sensitization is a nonproblem
– F comparing main effect of experimental
group is examined
?
If interaction term is significant
– compare simple main effect means of
unpretested experimental and control
groups
df
Mean
Square
F-Ratio
Probability
Experiment
1
1771.10
6.23
0.013*
Pretest
1
1.29
0.01
0.9400
Exp x
Pretest
1
363.80
1.28
0.2600
394
284.50
Error
25
26
Interpretation ( Tutty, 1997)
Common sources of confounding
“There was no interaction between pretesting
and the experimental condition,
? which indicated that the pretest did not
sensitize participants so that they scored
better on the posttest,
? a desirable finding which allows for further
comparison of the four conditions,
? since those that were pretested did not have
an unfair advantage.”
?
When two or more variables co-occur, the
impact of these variables cannot be
separated:
? Motivation and ability
? Income and education
? Teacher and curriculum
? Maturation and learning
? Pretest and treatment
27
28
Common Control Problems:
Threats to Internal Validity
Obtrusiveness effects:
treatment groups
Retesting
Regression to the mean
? Local history
? Mortality (attrition)
? Maturation
? Instrument decay
(ceiling and floor effects)
? Selectivity (e.g., volunteer)
?
Hawthorne effects
Selective attrition: “guinea pig”
? Halo effect
? Novelty effect
?
?
?
29
30
5
Obtrusiveness effects:
control groups
Researcher expectancy effects
?
John Henry (Avis) effects
? “All the boats float higher”
? Horizontal diffusion
? Demoralization
?
Rosenthal effects
– Pygmalion (My Fair Lady) effects
?
Saved through double-blind procedures
– Neither participants nor researchers know
treatment group membership!
– Very difficult to implement in human
services
31
32
Field Research Challenges
?
?
For each threat to validity
Teaching to the test
Selective attrition
?
?
Give an example from HD FS Research
Suggest a strategy to eliminate this threat
– Retention of high ability students
– Lower attrition in Treatment Schools
?
All boats float higher
– More resources available in control
programs
33
34
6