Chapter 10 – Lecture 10
Internal Validity – Control through
Experimental Design
1) Test the effects of IV on DV
2) Protects against threats to internal validity
Causation
Experimental Design
 Highest Constraint
 Comparisons btw grps
 Random sampling
 Random assignment
Infer Causality
Experimental Design
(5 characteristics)
1) One or more hypothesis
2) Includes at least 2 “levels” of IV
3) Random assignment
4) Procedures for testing hypothesis
5) Controls for major threats to internal validity
Experimental Design









•
Develop the problem statement
Define IV & DV
Develop research hypothesis
Identify a population of interest
Random sampling & Random assignment
Specify procedures (methods)
Anticipate threats to validity
Create controls
Specify Statistical tests
Ethical considerations
Clear Experimental Design…
Experimental Design
2 sources of variance
1. between groups variance (systematic)
no drug
drug
2. Within groups variance (nonsystematic)
(error variance)
Remember…
Sampling error
Significant differences…variability btw means is larger than expected
on the basis of sampling error alone (due to chance alone)
Variance
Need it!
Without it…
No go
VARIANCE
“Partitioning of the variance”
Between Group
Experimental Variance
Within Group
(Due to your treatment)
+
Extraneous Variance
(confounds etc.)
CON
TX
Error Variance
(not due to treatment – chance)
Subs
Variance: Important for the statistical analysis
F=
F=
F=
between groups variance
Within groups variance
Systematic effects + error variance
error variance
1.00
No differences btw groups
Variance
Your experiment should be designed to
• Maximize experimental variance
•Control extraneous variance
•Minimize error variance
Maximize “Experimental” Variance
• At least 2 levels of IV (IVs really vary?)
•Manipulation check: make sure the
levels (exp. conditions) differ each other
Ex: anxiety levels (low anxiety/hi anxiety)
 performance on math task
anxiety scale
Control “Extraneous” Variance
1. Ex. & Con grps are similar to begin with
2. Within subjects design (carryover effects??)
3. If need be, limit population of interest (o vs o )
4. Make the extraneous variable an IV
(age, sex, socioeconomic) = factorial design
M
F
Lo Anxiety
M-low
F-low
Hi Anxiety
M-hi
F-hi
Factorial design
(2 IV’s)
YOUR Proposals
Control through Design – Don’ts
1.
2.
3.
4.
Ex Post Facto
Single-group, posttest only
Single-group pretest-posttest
Pretest-Posttest natural control group
1. Ex Post Facto – “after the fact”
Group A
Naturally Occurring Event
No manipulation
Measurement
Control through Design – Don’ts
Single group posttest only
Group A
TX
Posttest
Single group Pretest-posttest
Pretest Group A
TX
Compare
Posttest
Control through Design – Don’ts
Pretest-Posttest Naturalistic Control Group
Group A
Pretest
TX
Posttest
Compare
Group B
Natural
Occurring
Pretest
no TX
Posttest
Control through Design – Do’s – Experimental Design
•
•
•
Manipulate IV
Control Group
Randomization
4 Basic Designs
Testing One IV
1. Randomized Posttest only, Control Group
2. Randomized Pretest-Posttest, Control Group
3. Multilevel Completely Randomized Between Groups
4. Solomon’s Four- Group
Randomized Posttest Only – Control Group
(most basic experimental design)
R Group A
(Ex)
TX
R Group B
(Con)
no TX
Posttest
Compare
Posttest
Randomized, Pretest-Posttest, Control Group Design
R Group A
(Ex)
Pretest
R Group B
(Con)
Pretest
TX
Posttest
Compare
no TX
Posttest
Multilevel, Completely Randomized Between
Subjects Design (more than 2 levels of IV)
R Group A
Pretest
TX1
Posttest
R Group B
Pretest
TX 2
Posttest
R Group C
Pretest
TX3
Posttest
R Group D
Pretest
TX4
Posttest
Compare
Solomon’s Four Group Design
(extension Multilevel Btw Subs)
R Group A
Pretest
TX
R Group B
Pretest
----
Posttest
R Group C
--------
TX
Posttest
R Group D
--------
----
Posttest
Powerful Design!
Posttest
Compare
What stats do you use to analyze experimental designs?
Depends the level of measurement
Test difference between groups
Nominal data  chi square (frequency/categorical)
Ordered data  Mann-Whitney U test
Interval or ratio  t-test / ANOVA (F test)
t-Test
Independent
Samples (between Subs)
Evaluate differences bwt
2 independent groups
Compare 2 groups
One sample (Within)
Evaluate differences bwt two
conditions in a single groups
Assumptions to use t-Test
1. The test variable (DV) is normally distributed
in each of the 2 groups
2. The variances of the normally distributed test
variable are equal – Homogeniety of Variance
3. Random assignment to groups
t-distribution
Represents the distribution of t that would be obtained if a
value of t were calculated for each sample mean for all possible
random samples of a given size from some population
Degrees of freedom (df)
When we use samples we approximate means & SD
to represent the true population
Sample variability (SS = squared deviations) tends
to underestimate population variability
Restriction is placed = making up for this mathematically
by using n-1 in denominator
S2 = variance
ss (sum of squares)
df (degrees of freedom)
(x - )2
n-1
x
Degrees of freedom (df): n-1
The number of values (scores) that are free to
vary given mathematical restrictions on a
sample of observed values used to estimate
some unknown population = price we pay for
sampling
Degrees of freedom (df): n-1
Number of scores free to vary
Data Set  you know the mean
(use mean to compute
variance)
n=2 with a mean of 6
X
8
?
6
In order to get a mean of 6
with an n of 2…need a sum
of 12…second score must be
4… second score is restricted
by sample mean (this score is not
free to vary)
=x
Group Statistics
ENDURANC
DRUG
doped
no dope
N
10
10
Mean
7.9000
2.6000
Std. Error
Mean
.3786
.4000
Std. Deviation
1.1972
1.2649
Independent Samples Test
Levene's Test for
Equality of Variances
F
ENDURANC Equal variances
assumed
Equal variances
not assumed
.065
Sig.
.801
t-test for Equality of Means
t
Mean
Sig. (2-tailed) Difference
df
Std. Error
Difference
95% Confidence
Interval of the
Difference
Lower
Upper
9.623
18
.000
5.3000
.5508
4.1429
6.4571
9.623
17.946
.000
5.3000
.5508
4.1427
6.4573
ANOVA
ENDURANC
Between Groups
Within Groups
Total
Sum of
Squares
140.450
27.300
167.750
df
1
18
19
Mean Square
140.450
1.517
F
92.604
Sig.
.000
Analysis of Variance (ANOVA)
Two or more groups ….can use on two groups…
t2 = F
Variance is calculated more than once
because of varying levels (combo of differences)
Several Sources of Variance
SS – between
Partitioning
SS – Within
the variance
SS – Total
Sum of Squares: sum of squared deviations from the mean
Assumptions to use ANOVA
1. The test variable (DV) is normally distributed
2. The variances of the normally distributed test
variable is equal – Homogeniety of Variance
3. Random assignment to groups
F=
Systematic effects + error variance
error variance
F=
F=
between groups variance
Within groups variance
1.00
F = 21.50
No differences btw groups
22 times as much variance between
the groups than we would expect by chance
After Omnibus F…
Planned comparisons & Post Hoc tests
A Priori (spss: contrast)
A Posteriori
part of your hypothesis…before
Not quite sure where
data are collected…prediction is made differences will occur
Why not just do t-tests!
2 types of errors that you must consider when
doing Post Hoc Analysis
Alpha
1. Per-comparison error (PC)
2. Family wise error (FW)
Inflate Alpha!!!!
FW = c()
c = # of comparisons made
 = your PC
Ex: IV ( 5 conditions)
1 vs 2
1 vs 3
1 vs 4
1 vs 5
2 vs 3
2 vs 4
2 vs 5
3 vs 4
3 vs 5
4 vs 5
FW = c()
10 (0.05) = .50
HSD