Part II: Complex Designs
Nonequivalent Control Group
? Time-Series Designs
? Counterbalanced Designs
? Patched up Design
?
HD FS 503
Experimental Design II
March 25, 2002
1
Demographic questions: 4 points
?
?
2
Research Dilemma
Exhaustive
Mutually Exclusive
Fully informed consent
Confidentiality
? Trained and monitored interviewers
? Referrals or transfers to service providers
? Dissemination to practitioners and
researchers
?
?
3
Hoped for outcome of
nonequivalent control group!
Nonequivalent Control Group
Ideally, treatment group
at pretest is below control group
at posttest is above control group
? Eliminates explanations of:
?
70
60
50
40
? regression
to mean
? ceiling effects
?
4
Treatment
Control
30
Hard to plan for, because treatment
group score doesn’t reach control group
score!
20
Pretest
5
Posttest
6
1
Design
Time-series design
2 x 2 mixed design ANOVA
(time by treatment) design
? Time is a within subjects factor
(repeated measure)
?
? Each
Regression discontinuity design
O O O X O O O
? Impact of treatment should show up in
change in slope
?
subject experiences both levels
and Posttest
? Pretest
?
Treatment is a between subjects factor
? Each subject experience only
? Either
1 level
treatment or control
7
8
Interpretation of
regression discontinuity design
50
50
40
40
30
30
Score
Score
Hoped-for outcome of
regression discontinuity design
20
10
Treatment
20
Treatment
Control
10
0
0
1
2
3
4
5
6
1
Time of testing
2
3
4
5
6
Time of testing
9
10
MANOVA’s and Repeated
Measures Designs
Design
Within subject designs
Multiple dependent measures on the same
subjects
? Repeated Measures: Same measures repeated
at two or more times
?
?
Repeated measures ANOVA
6 measures of dependent variable
? Independent variable is Time
? Within subjects design: each subject
completes all 6 dependent measures
?
?
? Math
scores in first grade, second grade, third
grade
? Pre-test, post-test, and delayed post -test measures
?
MANOVA: Different measures
? sharing
11
a common construct: reading, math, and
science scores
12
2
An example of repeated
measures ANOVA
Repeated measures ANOVA
Creates new factors!!!
? Identify two (or more) dependent variables
?
?
? variables
represent two (or more) levels of a
factor (mathk and math1)
? Use the menu to tell SPSS the factor name
(time) (default is factor 1---this is confusing!)
?
?
2 (sex) by 2 (time) repeated measures
ANOVA
? factor
= childsex
= time
? two levels: fall kdg and spring grade 1
? factor
Test the impact of time and sex on math
Write out ANOVA:
X (aaa) by Y (bbb) mixed design ANOVA
? mathk
? math1
?
?
Use repmnva.sav and try this
Set up in GLM-Repeated Measures
13
Identifying factor levels
?
Try repeated measures ANOVA
Within subjects factor name
The effect of sex and time on math scores
MathK: fall kindergarten
? Math1: spring first grade
? Plot means
?
? Default
= factor 1
? Change to time
?
?
Number of levels?
? How
14
many times?
2
? Add
? Define
?
15
Within subjects Measures (time)
16
Repeated measures MANOVA
___?___(1)
? ___?___(2)
? Identify the two variables that represent the
dependent measures (math scores)
?
2 within subjects variables,
1 (or more) between subjects variables
? E.g., the effect of sex and grade on math
and reading scores (longitudinal)
? 2 (sex) x 2 (grade) x 2 (domain) mixed
design repeated measures ANOVA
? This gets tricky!
? Match the factors and levels correctly!
?
? Time 1?
? Time 2?
Mathk and Math1
Between subjects factor?
? Childsex
?
?
17
18
3
Longitudinal Study of
Reading & Math
?
Don’t change menu yet!!
Factors
Is that all the within subjects factors?
? No
? Domain
? Levels: 2
? Add
?
? 1:
time (kindergarten and first grade)
? 2: domain (reading and math)
? 3: sex (male and female)
?
Which are between subjects factor(s)?
?
Which are within subjects factor(s)?
19
20
Pitfalls of repeated measuresANOVA’s
?
Now Define: Be Careful!!!
No missing data permitted!
? Each
subject must complete all 6 measures
No way to estimate missing observations!
Within subjects variables (time, domain)
? __?__ (1,1)
? __?__ (1,2)
? __?__ (2,1)
? __?__ (2,2)
?
Assessments must be equally spaced
? If
measurement off by 1 month out of 12,
linear relationship looks cubic!
?
No two measures can be more highly
correlated than any others!
? But
you’d expect adjacent date points to be
more highly correlated!
? Violates assumptions of sphericity
21
22
Counterbalanced designs
(Latin squares)
Recent methodological advances
in developmental research:
?
Multiple treatments
? Each treatment appears first
Hierarchical linear modeling
? Latent growth curve analysis
?
? Each
treatment is preceded by all others
Observe after each treatment:
A O B O C O
B O C O A O
C O A O B O
? Each treatment occurs
?
? Once
23
in every row
? Once in every column
24
4
Example: (Latin square)
Counterbalanced design
Order of Treatment in Groups:
Group 1: A O B O C O
Treatments:
? A:
Reading a booklet about STD’s
? B: Role play about STD’s
? C: Hearing a health lecture about STD’s
Group 2: B O C O A O
Randomly assign each student to 1 of 3
groups: group 1, group 1, group 3
? Each group experiences all three
treatments,
?
Group 3: C O A O B O
? in one of 3 orders
25
Why use a
counterbalanced design?
Design
?
3 by 3 mixed design ANOVA
(Order by treatment)
? Order
26
?
is between subjects factor
within subjects factor
? MS
? Treatment is
?
(treatment) divided by df
design doubles group size
? Counterbalanced
Main effect of treatment?
?
? All
subjects do better following B than
following A or C
?
Group size affects statistical significance
testing
3 groups instead of 6 groups
? ABC (no ACB)
? BCA (no BAC)
Main effect of order?
? CAB (no CBA)
? BCA
does better than ABC or CAB
? Is BCA is different from BAC??!! Unknown
?
27
MS (treatment) divided by 2 instead of 5 -> more power in 3 groups than in 6
28
Sample Factorial Design
Review of Factorial Designs
?
?
Most outcomes have multiple causes...
Most studies have multiple independent
variables
? Factor = treatment = independent
variable (categorical = nominal)
? Typical factors:
Sex, Curriculum, Ability (high or low)
? Tested with ANOVA
?
Two factors: sex; type of sex education
Factor 1: sex
? Males
?
? Females
?
Factor 2: type of sex education program
? Booklet
? Role
play
? Health lecture
?
29
2 (sex) by 3 (program) ANOVA
30
5
Interactions:
Ordinal and Disordinal
Test for main effects and for
interactions
?
Main effects: one factor has significant
effect across all levels of the other factor
? Both
boys and girls perform better in one
treatment than in the other OR
? Both treatments work better for boys than for
girls
?
Interaction: Levels of one factor have
different effects at each level of second
factor
? Boys
? Girls
do better in one treatment; however,
do better in the other treatment
Examine significant interactions
before interpreting main effects:
? Ordinal interaction:
non parallel lines but
non intersecting lines
Therefore, main effects hold
? Disordinal
31
Appropriate graph is column (because
Treatment and Control are categorical, not
continuous)
Math Scores
30
10
20
10
Males
Females
Males
Females
0
0
Treatment
Treatment
Control
Control
33
Main effect of Treatment
Main effect of Sex
34
Main effect(s)?
Interactions?
30
Math Scores
30
Math Scores
32
Easier to interpret from lines
Main effect(s)?Interactions?
20
Math Scores
interactions:
intersecting lines
Therefore, main effects may not be
interpreted
20
10
20
10
Males
Males
Females
Females
0
0
Treatment
Control
Treatment
35
Control
36
6
Main effect of Sex
No main effect of Treatment
Main effect(s)?
Interactions?
30
Math Scores
Math Scores
30
20
10
20
10
Males
Males
Females
Females
0
0
Treatment
Control
Treatment
Control
37
Interaction between Treatment
and Sex: Ordinal
38
Main effect(s)?
Interactions?
30
Math Scores
Math Scores
30
20
10
20
10
Males
Males
Females
Females
0
0
Treatment
Control
Treatment
Control
39
Interaction between Treatment
and Sex: Disordinal
40
Main effect(s)?
Interactions?
Math Scores
Math Scores
30
20
10
15
Males
Males
Females
Females
0
5
Treatment
Control
Treatment
41
Control
42
7
Math Scores
Interaction between Treatment
and Sex: Disordinal
Now Define
Within subjects variables (time, domain)
? mathk (1,1)
? readk (1,2)
? math1 (2,1)
? read1 (2,2)
? Run this, plus means
? Write down 8 cell means
15
Males
Females
5
Treatment
Control
43
Choices in design: within or
between subjects design??
?
Design choice:
mixed design
Sex = between subjects
2 (sex) x 3 (program) mixed design ANOVA,
? Sex is between subjects factor
? Treatment is within subjects factor
? Each
? Two
?
?
44
subject is one and only one sex!!
levels: male and femals
Sex education
? Three levels:
? Use
? Booklet,
? Preferred method:
role play, lecture
counterbalanced design
more sensitive to treatment effects
Between subjects or within subjects?
? Should each subject experience only 1?
? Should each subject experience all 3?
45
Design choice:
between subjects design
46
Nested (hierarchical) designs
2 x 3 between subjects ANOVA
(sex by treatment)
? Less sensitive than within subjects design
? Each subject experiences only 1
treatment: why?
Reading curriculum is nested within the
classroom (teacher) effect
? Effects of home visit program nested
within home visitor effect
? Effects of therapy program nested within
therapist
? No interaction between effect of teacher
(home visitor) and program can be
tested!
?
? Each
treatment too costly to repeat
interact;
can’t counteract effects of one treatment
? Example: testing two different reading
curricula
? Treatments
47
48
8
Nested designs are forced by
situation
?
Nesting affects ANOVA
Unit of analysis is the classroom, not the
individual within the classroom
? First test the effects of treatment on
classrooms
?
Situations that require nested designs:
? Two
different curricula can’t be
implemented in same classroom
? Same client cannot experience two therapy
programs
? Same family can’t have two home visitors
?
? If
effect is significant (difference between
classrooms is greater than difference within
classrooms)....
? You cannot test the effect of the treatment
directly on the individual
Impossible to test interaction between
classroom and treatment:
these are confounded!
?
This reduces N! and reduces power!
49
Aids to hypothesis testing and
proposal writing:
Treatment Problems
?
Defining
HD FS 503 Handbook
Reading Research Articles: p. 5
(humorous, but true!)
? Nine steps to Hypothesis Testing
? Practice Exercise:
Testing Research Hypotheses, Handbook
? Writing a Research Proposal, Handbook
Krathwohl, p. 649
? what happened?
?
? to
whom?
? at what dosage?
?
Maintaining fidelity
?
Sensing Changes
? beware
50
of drift!
? Beware of teaching to the test!
? Swahili syndrome
51
52
Chi-square analyses
Current APA recommendations:
?
?
For nonparametric data
? Examples?
?
Include effect sizes
Translate effect sizes into meaningful
metrics
? Sex
by major
classification by sex
? Agree/Disagree by Political Party
? Attachment
? Statistical
? Address
?
significance is not enough!
psychological significance
?
Examples: IQ points, inches,
grade equivalents
53
Use the cross-tabs procedure here
(see summarize)
54
9
Chi Square Cautions
Next week:
Chi-square compares number observed and
expected (if no relationship existed)
? Not very sensitive to differences in small
samples
Share your hypothesis (2 - 3 minutes)
? Qualitative research
? Krathwohl, ch. 11, 12, 13, 14, 15
?
? At
?
least 5 cases must be expected per cell
?
Remember to collapse categories to obtain
minimum cases per cell
?
Responses must be independent cases
? What
SPSS command collapses cases?
55
56
Microsoft Graph
?
?
Find a spreadsheet with data already entered
Note that the top toolbar has changed
? The
? So
spreadsheet icon is “on”
you can change numbers and labels
Relabel rows and columns
? Rows = groups
? Columns = time
? Delete unneeded rows and columns
?
57
58
Consider graph format
More fine tuning of graphs
Do you want a line graph or a bar graph?
Format: Chart type
? 3d or 2d?
? Line or Bar?
? Line
? Look at line graph
To remove data from axis
? Return to spreadsheet
? Cut out columns A and B
? Paste at column B
? Enter 0’s in cells of columns A and D
? Delete 0’s in cells of columns A and D
? (Leave Column A and D active)
Ta-dah!!
?
?
59
60
10
Editing the graph
?
Click on any part of the graph to edit it
? Scale?
? Line
?
Change to possible range
width? color? pattern?
Insert
? Gridlines?
? Titles?
No
Yes
? Y-axis
? Type
? Click
to change alignment to vertical
61
62
11