Unit 1d: Fundamentals
of Research Design and
Experimental Controls
PSYC 4310
COGS 6310
MGMT 6969
Michael J. Kalsher
Department of
Cognitive Science
© 2015, Michael Kalsher
1
1
The Role of Experimental Design
In most social and behavioral research studies, we attempt to
obtain at least one score from each participant (usually
more!). Any obtained score is comprised of a number of
components:
1. A ‘true score’ for the thing we hope we are measuring.
2. A ‘score for other things’ that we measure inadvertently.
3. Systematic (non-random) bias (usually ok as long as it affects all
participants equally).
4. Random (non-systematic) error (which should cancel out over large
numbers of observations).
We want our obtained score to consist of as much ‘true
score’, and as little of the other factors, as possible.
© 2015, Michael Kalsher
2
Research Study Control
Control removes sources of error in the inferences we
draw from the data. This:
– Reduces the chance of wrong conclusions
– Increases the power of statistics to find
relationships in the presence of random error
(“noise”)
Types of Control
– Direct Manipulation
– Randomization
– Statistical Control
© 2015, Michael Kalsher
3
Types of Control:
Direct Manipulation
Sources of error held constant by
research design or sampling decisions
– Example: a researcher investigating the effects of
seeing justified violence in video games on
children knows that young children cannot
interpret the motives of characters accurately.
She decides to limit her study to older children or
eliminate random responses or unresponsiveness
of younger children.
© 2015, Michael Kalsher
4
Types of Control:
Randomization
Unknown sources of error are equalized
across all research conditions by randomly
assigning subjects or by randomly choosing
experimental materials.
– Example: Many different factors are known to affect
the amount of use of Internet social networking sites. A
researcher wants to test two different site designs. He
randomly assigns subjects to work with each of the two
designs. This helps to equalize the amount of
confounding error from unknown factors in both
groups.
© 2015, Michael Kalsher
5
Types of Control:
Statistical Control
Known confounding variables are measured,
and mathematical procedures are used to
remove their effect.
– Example: A political communication researcher
interested in studying emotional appeals versus
rational appeals in political commercials suspects that
the effects vary with the age of the viewer. She
measures age, and uses it as an independent predictor
(with multivariate statistics) to isolate, describe, and
remove its effect.
© 2015, Michael Kalsher
6
Contrasting Methods of Control
Type of
Control
Strength
Weakness
Direct
Manipulation
• Removes effect completely
• Must know source of effect
• Reduces generalizability
Randomization
• Don’t have to know source of
effect
• Equalizes effect so there is no
systematic confound
• Reduces statistical power
by adding to unsystematic
error variance
Statistical
control
• Estimates effect of
confounding variables
• Expands theoretical model
• Must know source of effect
• Requires more complex
statistics
© 2015, Michael Kalsher
7
Basic Types of Research
• Observational Methods
• Quasi-Experimental Designs
• True Experimental Designs
© 2015, Michael Kalsher
8
Observational Methods
No direct manipulation of variables by the
researcher. Behavior is merely recorded--but
systematically and objectively so that the
observations are potentially replicable.
Advantages
•
•
Reveals how people normally behave.
Experimentation without prior careful observation can lead to a
distorted or incomplete picture.
Disadvantages
•
•
Generally more time-consuming.
Doesn’t allow identification of cause and effect.
© 2015, Michael Kalsher
9
Quasi-Experimental Design
In a quasi-experimental study, the experimenter
does not have complete control over manipulation
of the independent variable or how participants
are assigned to the different conditions of the
study.
Advantages
•
•
Natural setting
Higher face validity (from practitioner viewpoint)
Disadvantages
•
Not possible to isolate cause and effect as conclusively as with a
“true” experiment.
© 2015, Michael Kalsher
10
Quasi-Experimental Design: An Example
Kalsher, M. J., Geller, E. S., & Clarke, S. W. (1989). Safety belt promotion on a naval base: A
comparison of incentives vs. disincentives. Journal of Safety Research, 20, 103-113.
In one study, a team of researchers investigated how the safety belt use of drivers
on two Virginia naval bases would be affected by two different motivational
approaches: the use of incentives versus disincentives. The main questions were
whether safety belt use would increase more in response to incentives (a chance to
win prizes) or disincentives (loss of driving privileges if caught not buckled) and
whether any change effected would be maintained over time. Obviously, it was not
possible, or practicable, to assign people driving on the two bases randomly to the
two different conditions. Instead, the drivers on one base experienced the incentive
approach; drivers on the other base the disincentive approach. This is an example
of “incomplete” control over the manipulation of the independent variable and
constitutes a “quasi-experimental design.”
Other features of the design allowed the researchers to draw valid conclusions from
the study, including the use of two control groups (bases that did not implement any
intervention) and sequential implementation of the programs at the two “treatment”
bases. The results of the study showed that the disincentive approach worked best
initially, but returned to baseline levels shortly after the program (increased
enforcement) ended. The smaller gains in response to the incentive approach were
maintained long after the program ended so was judged more effective.
© 2015, Michael Kalsher
11
Types of
Quasi-Experimental Designs
© 2015, Michael Kalsher
12
One Group Post-Test Design
Measurement
Treatment
Time
Change in participants’ behavior may or may not be
due to the intervention.
Prone to time effects, and lacks a baseline against
which to measure the strength of the intervention.
© 2015, Michael Kalsher
13
One Group Pre-test Post-test Design
Measurement
Treatment
Measurement
Time
Comparison of pre- and post-intervention scores
allows assessment of the magnitude of the
treatment’s effects.
Prone to time effects, and it is not possible to
determine whether performance would have
changed without the intervention.
© 2015, Michael Kalsher
14
Interrupted Time-Series Design
Measurement
Measurement
Time
Measurement
Treatment
Measurement
Measurement
Don’t have full control over manipulations of
the IV. No way of ruling out other factors.
Potential changes in measurement.
© 2015, Michael Kalsher
Measurement
15
Static Group Comparison Design
Group A:
Treatment
Measurement
(experimental group)
Group B:
No Treatment
Measurement
(control group)
Time
Participants are not assigned to the conditions randomly.
Observed differences may be due to other factors.
Strength of conclusions depends on the extent to which
we can identify and eliminate alternative explanations.
© 2015, Michael Kalsher
16
Experimental Research:
Between-Groups and
Within-Groups Designs
© 2015, Michael Kalsher
17
Between-Groups Designs
Separate groups of participant are used for each
condition of the experiment.
Within-Groups (Repeated Measures) Designs
Each participant is exposed to each condition of
the experiment (requires less participants than
between groups design).
© 2015, Michael Kalsher
18
Between-Groups Designs
Advantages
•
•
•
Simplicity
Less chance of practice and fatigue effects
Useful when it is not possible for an individual to
participate in all of the experimental conditions
Disadvantages
•
•
Can be expensive in terms of time, effort, and number of
participants
Less sensitive to experimental manipulations
© 2015, Michael Kalsher
19
Examples of
Between-Groups Designs
© 2015, Michael Kalsher
20
Post-test Only / Control Group Design
Group A:
Measurement
Treatment
(experimental group)
Random
allocation:
Group B:
Measurement
No Treatment
(control group)
Time
If randomization fails to produce equivalence, there is no way
of knowing that it has failed. Experimenter cannot be certain
that the two groups were comparable before the treatment.
© 2015, Michael Kalsher
21
Pre-test / Post-test Control Group
Design
Group A:
Measurement
Treatment
Measurement
No Treatment
Measurement
Random
allocation:
Group B: Measurement
Time
Pre-testing allows experimenter to determine equivalence
of the groups prior to the intervention. However, pretesting may affect participants’ subsequent performance.
© 2015, Michael Kalsher
22
Solomon Four-Group Design
Random allocation:
This design guards against many of the threats to internal validity.
Can you determine which ones?
Group A: Measurement
Treatment
Measurement
Group B: Measurement
No Treatment
Measurement
Group C:
Treatment
Measurement
Group D:
No Treatment
Measurement
Time
© 2015, Michael Kalsher
23
Within-Groups Designs:
Repeated Measures
Advantages
• Economy
• Sensitivity
Disadvantages
• Carry-over effects from one condition to another
• The need for conditions to be reversible
© 2015, Michael Kalsher
24
Repeated-Measures Design
Treatment
Measurement
No Treatment
Measurement
Measurement
Treatment
Measurement
Random Allocation
No Treatment
Time
Potential for carryover effects can be avoided by randomizing the order
of presentation of the different conditions or counterbalancing the order
in which participants experience them.
© 2015, Michael Kalsher
25
Latin Squares Design
Three Conditions or Trials
order of conditions or trials:
One group of participants
A
B
C
Another group of participants
B
C
A
Yet another group of participants
C
A
B
Order of presentation of conditions in a within-subjects design can be
counterbalanced so that each possible order of conditions occurs just once.
Problem not completely eliminated because A precedes B twice, but B precedes
A only once. Same with C and A.
© 2015, Michael Kalsher
26
Balanced Latin Squares Design
Four Conditions or Trials
order of conditions or trials:
One group of participants
A
B
C
D
Another group of participants
B
D
A
C
Yet another group of participants
D
C
B
A
And yet another group of participants
C
A
D
B
Note: This approach works only for experiments with an even number of conditions. For
additional help with more complex multi-factorial designs, see: http://www.jic.bbsrc.ac.uk
© 2015, Michael Kalsher
27
Factorial Designs
• include multiple independent variables
• allow for analysis of interactions
between variables
• facilitate increased generalizability
© 2015, Michael Kalsher
28