COMM 301:
Empirical Research in
Communication
Lecture 10
Kwan M Lee
1
Experiments
• Things to know by the end of the lecture:
– Know advantages and disadvantages of experiments
– Know how to achieve experimental control to get
causal inference
• Factors influencing control
• Specific techniques to ensure control
– Know the various experimental designs, in terms of
•
•
•
•
Understand logic behind each design
structure, how they look like
advantages and disadvantages
How to apply
2
Experiments
• Experiment
– method that allows us to evaluate the
influence of some independent variables on
some dependent variables
– while controlling for other intervening
variables
3
Experiments
Advantages vs. Disadvantages
• Advantages
– Allow causal inference
– Replication possible by someone else.
• Disadvantages
– Lack ecological isomorphism
– Weak generalizability
4
A note on causality (remember lect2 ?)
• 3 requirements for causal relationships
– Temporal ordering: cause precedes effect in time
– Meaningful correlation: must have a theoretical
foundations for observed correlations
– No alternative causes (hypotheses): correlation cannot
be explained by other factors
• Control: the use of techniques for systematically
ruling out alternative causes
– Alternative hypotheses exercise
• E.g. no wash on an exam day 1) no wash is lucky vs. 2)
more time spend on reading
5
Factors influencing the amount of
control in experiment
• Manipulation of independent variable
• Ensuring group equivalence
• Control of intervening variables
6
Manipulation of independent variable
• Active manipulation
– Researcher determines participants’ level of
exposure to independent variable.
– Give some assurance that if dependent variable
changes, it is due to the independent variable
– E.g., default experiment setting
• Passive observation
– Cannot be under control of researcher
– Natural setting
• E.g.,Comparison of two existing classes which have different
teaching methods
7
Ensuring group equivalence
• Experiments use different groups of
participants
• treatment group (treatment)
• control group (no treatment)
• comparison groups (when two or more
treatment groups)
• Group equivalence assumption
– Must ensure that before imposing the treatment,
the groups are equivalent with regard to the
dependent variables.
– How to insure?  see next slides
8
How to ensure group equivalence
• Random assignment
– Participants assigned to treatment or control,
or comparison groups by randomized
method.
– No guarantee of getting equivalent groups
• But it usually works!
– Can use statistical testing to find out how
likely the group is non-equivalent.
• Comparisons of variables not related to
dependent variables
9
How to ensure group equivalence
(cont.)
• Pretesting
– Groups are measured on the dependent
variable(s) before any treatment.
– Expectation is that the groups will be similar
• If not, non-equivalent with regard to DVs
10
How to ensure group equivalence
(cont.)
• Matching
• Participants in the groups are matched on characteristics
important to dependent variable(s) (i.e. matched on
intervening variables)
– Constancy matching
• All participants in all groups kept uniform on the characteristic
thought to influence the dependent variable.
• e.g. holding gender constant: test only males
– Pairing
• Each participant in a group is matched with other participants
in other group on the variable(s) thought to affect the
dependent variable(s)
• e.g. for each male (female) in treatment group, there is a male
(female) in the control group – i.e., equal # of males and
females
11
Control of intervening variables
• Eliminate the influence of:
– participants (through ensuring group
equivalence)
– settings (through ensuring settings are
different only with regard to manipulations)
– individual researchers
• use of double-blind procedures
• use of scripts.
– other factors influencing internal validity (e.g.,
maturation, attrition.)
12
Experimental designs
• Experimental research designs are
categorized along four dimensions
–
–
–
–
Research settings
Amount of control
Number of independent variables
Different subjects vs. Same subjects in cells
13
Setting
• Experimental design setting
– Laboratory
• Research setting created by the researcher,
maintains strong control over the setting
• Sacrifice ecological isomorphism
– Field
• Naturally occurring research setting, with little
researcher control
• Enhance ecological isomorphism
• e.g. new training program at a company
14
Amount of Control – High Control
• Pretest posttest control group design
– Good control for group equivalence
• Randomized assignment to treatment and control
groups
• Pre-testing
– Handle most threats to validity well.
– Limitation: test sensitization
– See Graph in p. 86
15
Pretest-posttest control group design
•
•
•
•
•
•
•
T: treatment group
C: control group
R: Random Assignment
Pr: Pretest
X : Treatment (intervention)
- : No treatment (intervention)
Po: Posttest
16
Amount of Control – High Control
• Posttest only control group
– Randomized assignment to treatment and
control groups for group equivalence
– No pre-testing, sacrifice some group
equivalence
– Attempts to counter test sensitization in
pretest posttest control group design
17
Amount of Control – High Control
• Solomon four-group design
– Combines pretest posttest control group
and posttest only control group
– Highest level of control
• Ensures group equivalence through
randomized assignment and pretesting
• Allows assessment of test sensitization’s
impact (T1 vs. T2; C1 vs. C2; C1 [Pr vs. Po])
– Limitations:
• Costs, especially in subjects
• Difficulties in interpreting contradicting results
18
Solomon four-group design
19
Solomon four-group design
interpretation exercise
Group 1
Group 2
Group 3
Group 4
•
•
R
R
R
R
Pr1 (60)
X
X
Pr2 (62)
Po1 (82)
Po2 (74)
Po3 (70)
Po4 (62)
The numbers in the parentheses represent the average score of the group.
You can assume that a difference of 5% or more represents some sort of effect or
difference beyond random chance
• Were the four groups equivalent? Why?
– Random assignment
– Pr1 = Pr2 = Po4
• Does the treatment have an effect?
– Po1 vs Po3; Po2 vs. Po4; Po1 vs. Pr1
• Do you see prestest effects (test sensitization)?
– Po3 vs. Po4; Po1 vs. Po2; Pr2 vs. Po3
20
Amount of Control – Moderate Control
• High control designs use random
assignment.
• Random assignment not always possible.
• Shift to moderate control designs.
– Pretest posttest nonequivalent group design
– Interrupted and multiple interrupted time
series
21
Amount of Control – Moderate Control
• Pretest posttest nonequivalent group
– Identical to pretest posttest control group
but without random assignment into groups.
– Typically, groups already formed naturally
• E.g. classes
– Ability to make causal inferences weakened.
• Differences in the posttest may be due to initial
differences between the groups.
– many possible alternative Hs
T:Pr x
Po
C: Pr Po
22
Amount of Control – Moderate Control
• Interrupted time series design
– Only a single group available
•
•
•
•
The group must serve as both T and C groups
series of pretests, establish baseline measure
administer treatment
series of posttests
– Advantages
• can handle pretest sensitization (if all pretest scores are similar)
and maturation (if there is little change from one test to the next)
• can test the persistency of the treatment effect
Pr1
Pr2
Pr3
Pr4
x
Po1 Po2
• Multiple interrupted time series design
Pr1
Pr2
Pr3
Pr4
x
Po1 Po2
Pr1
Pr2
Pr3
Pr4
Po1 Po2
Po3
Po4
Po3
Po3
Po4
Po4
23
Amount of Control – Low Control
• Experimental designs, low control
– At times, circumstances do not allow random
assignment nor even moderate control by
pretest
– Not an experiment in a strict sense
– Usually for Exploratory studies
• Single group post test only design
• Single group pre-post test design
• Posttest only nonequivalent control group design
24