Experimental Research
Experimental Research

What is experimental research?


Research investigation in which conditions are
controlled so that hypotheses can be tested and
alternative explanations can be ruled out
Research used to make “cause-and-effect”
statements (X causes Y)


X is the independent (or manipulated or causal) variable
Y is the dependent variable
Experimental Research

Evidence of causality (i.e., X causes Y)

Evidence of concomitant variation (X and Y covary)


Time order sequence of variables



The more of X, the better chance that we will get Y
If X causes Y, X must occur before Y
Elimination of other possible explanations of
why Y occurred
Key is to keep all experimental conditions
equal
Experimental Research


Must try to eliminate all possible extraneous
influences
Possible extraneous influences


History - event occurring during the course of an
experiment (but not really part of the experimental
manipulation) that influences the results
Maturation - changes which occur in the
experimental unit which occur during the course of
an experiment that are due to the passage of time
Experimental Research

Possible extraneous influences (continued)

Testing Effects - changes in experimental unit
due to the experiment itself (but not related to the
key experimental manipulation


Measurement effect -- prior measurements effect the
measure of the dependent variable
Interaction effects -- subject pays greater attention to
stimuli than they normally would because its and
experiment
Experimental Research

Possible extraneous influences (continued)


Instrument variation -- changes in the measuring
instrument cause changes in Y (i.e, interviewer
changes the way in which they ask questions)
Selection Bias -- if two groups are compared, this
notion suggests that the groups may not have
been equivalent to begin with; i.e., a manipulation
did not cause Y; group dissimilarities did
Experimental Research

Possible extraneous influences (continued)


Experimental mortality -- respondents in
experimental groups were lost during the duration
of the experiment
Statistical Regression -- Extreme responses
move closer to the midpoint during the course of
an experiment; subjects after questionning do not
want to appear to be extreme
Experimental Research

Experimental Symbols




X = Experimental Treatment (e.g., ad exposure)
O = Observation
R = Random Assignment of Subjects
Pre - Experimental Designs

One-Shot Case Study


X O
Problems -- Selection Bias, Control Group, Mortality,
History, etc.
Experimental Research

Pre-Experimental Designs (Cont’d)

One Group Pre-Test/Post-Test Design




O1 X O2
Analysis -- O2 - O1
Problems -- History, Maturation, interactive Testing Effect, etc.
Static Comparison



X
O1
O2
Analysis -- O2 - O1
Problems -- Selection Bias, Mortality, etc.
Experimental Research

True Experimental Designs

Before/After with Control Group

(R) O1 X O2
(R) O3
O4

Analysis (O4 – O3) – (O2 – O1)

Lessens History and Maturation Problems
However, interactive testing effect still an issue



O2 and O4 may a function of O1 and O3
Experimental Research

Experimental Designs (True Experimental
Designs)

Six Group - Four Study Design






(R) O1 X O2
(R) O3
O4
(R)
X O5
(R)
O6
O2 – O1 compared to O4 – O3 tells if X “worked” (manipulation
check) – If yes, go on
If O5 > O6 – we have true effects
Experimental Research



True experiments are very complex
Experimental designs that are used are often
flawed (or not “True”)
Random assignment helps
Experimental Research


Analyzing Data
Typically uses Analysis of Variance (ANOVA)
 Tests (determines if at least one treatment group mean is
different from the others)



Ho:
Ha:
Why not use t-tests?

Probability




Μ1 = M2 = M3 = … = Mk
M1 ≠ M 2 ≠ M3 ≠ … ≠ Mk
One comparison at 95% confidence – 95% of rejecting null when it should
be rejected
Two comparisons -- .95 x .95 = 90.25%
Three comparisons -- .95 x .95 x .95 = 86%
Like regression


Dependent Variable (interval or ratio scaled)
Independent Variable (nominal or ordinal scaled)

Often group membership
Experimental Research (Responses on 1 =
Bad; 7 = Outstanding)
Treat/Resp
Ad 1
Ad 2
Ad 3
1
1
2
3
2
5
2
6
3
5
5
6
4
6
2
6
5
3
2
7
6
6
2
5
Mean
4.33
2.50
5.50
Experimental Research


One-Way ANOVA
Attempts to partition variance

Total Variance (Within Group + Between Group)
 Within treatment groups (Observation – Group Mean)



Between treatment groups (Group Mean – Grand Mean)



SSb = Σ nj (Meanj – Meani) 2
MSb = SSb / dfb
F-statistic


SSw = Σ Σ (x ij – Meani)2
MSw = SSw / df w
MSb / MSw

With dfb,; df w
Follow-up test

Tukey (most common in MR)
Experimental Research

Additional Issues with ANOVA

Real World

Often used with survey data




Two-Way ANOVA

One dependent variable



Two or more independent variables
Simultaneous effects
MANOVA


Compare means of several groups
DV – is interval (or ratio) scaled
IV -- categorical
Multiple dependent variables
ANCOVA

Investigate effects after “controlling” for another variable (that is interval or ratio scaled)
Experimental Research

Compare means of
several groups

Cool
Warm
EXAMPLE (Survey Data)
Permissive
Neglecting
Restrictive
Authoritarian
Indulgent
Authoritative
Experimental Research

DV = Parent Responsibility to Restrict TV
Choices for Kids

F= 4.42; p = .005





Neglecting (M = 29.5)
Indulgent (M = 31.26)
Authoritarian (M = 30.26)
Authoritative (M = 32.27)
Follow-up Tests



Authoritative > Authoritarian & Neglecting
Indulgent > Neglecting
Authoritarian = Neglecting
Experimental Research

Two-Way ANOVA – Example

Main Effects


Similar to one-way ANOVA effect (for 2 variables)
Interactive Effects


Main Effects look null
Differences observed only when looking at both factors
simultaneously
Experimental Research

Two-Way ANOVA

One dependent variable – Main Effects
P
N
Named
Not-Named
Experimental Research

Two-Way ANOVA

One dependent variable – Interaction Effects
N
P
Named
Not-Named