III. Research Design
Part I: Experimental Designs
Hypotheses

A hypothesis is a testable statement of causal
relationship between two variables, derived
from theory

Must specify a relationship between an
independent and dependent variable

Clear, specific, amenable to empirical testing,
value-free (F-N & N)
Research Design

“The program that guides the investigator in
the process of collecting, analyzing, and
interpreting observations. It is a logical
model of proof that allows the researcher to
draw inferences concerning causal relations
among the variables under investigation” (F-N
and N).
Relationships (Covariation) between
Variables

Relationships between variables: Two
variables are related to one another (i.e. are
correlated) if one or more values of one
variable tend to be associated with one or
more values of the other variable.
Directional Relationships


Apply to cases where the values of the IV and
DV are orderable (directional) variables
Positive relationship:


As one’s education level increases, the frequency
of voting increases
There is a positive relationship between one’s
education level and voting frequency
Directional Hypotheses (cont’d)

Negative relationship:

As the number of hours of negative ads watched
increases, the frequency of voting decreases

There is a negative relationship between exposure
to negative advertising and one’s voting
frequency
.2
Manatee
.1
.15
Seminole
Okeechobee
Lee
Pasco
Polk
Liberty
HardeeLake
Duval
Highland
Pinellas
Washington
Calhoun
Escambia
Lafayette
Collier
Clay
Flagler
St. Johns
Volusia
Jackson
Hernando Bay
Charlotte
Sumter
Brevard
Monroe
Indian
RiverBaker Santa Rosa
Marion
Walton
Columbia
Palm Beach Hillsborough
DeSoto
Sarasota
Putnam
Hendry
Martin
Broward
Osceola
Orange
Citrus
Dixie
FranklinHolmes St. Lucie
Madison
Levy
Nassau Okaloosa
Jefferson
Union
Leon
Gilchrist
Wakulla
Alachua Gulf
Suwannee
Dade
Bradford
Gadsden
.05
Hamilton Taylor
-3
-2
-1
0
Local Conservatism (Initiative-Based)
Average Monthly Sanction Rate
1
Fitted values
2
Non-Directional Hypotheses



Appropriate for variables that are not orderable
Hypothesis describes comparison among categories
Examples


Men have greater levels of support for President Bush
than do women
Whites are most likely to be Republican, while AfricanAmericans are most likely to be Democrat
Research Design and Causality


Relationships between variables: Two
variables are related to one another (i.e. are
correlated) if one or more values of one
variable tend to be associated with one or
more values of the other variable.
Causal relationship: A relationship in which
one variable directly causes/explains the other
variable.
Establishing Causality (F-N&N)




3 Criteria (Evidence Needed) for Establishing
Causality
Covariation (X is correlated with Y)
Time Order (X precedes Y in time)
Nonspuriousness (The observed relationship
between X and Y is not spurious)
Spurious Relationship

A relationship between two variables that is
presumed to be causal, when in fact it is not

An observed relationship between X and Y is
said to be spurious (or partly spurious) if there
exists a third variable Z (a “control variable”)
which is both a cause of Y AND is correlated
with X.
Spurious Relationship
X
(Presumed Causal Relationship)
Y
(True Causal Relationship)
Z
Example of Spuriousness
Gender and Corruption – Causal or
Spurious?
http://www.iq.harvard.edu/blog/pb/2005/10/sex_and_corruption.html
The Democratic Peace: Causal or
Spurious?
War
Democracy
Z(???)
Experimental Designs
1. Select a sample
2. Randomly assign subjects into 2 or more groups. The
number of groups is equivalent to the number of values
of the independent variable(s).
3. Observe (measure) DV for all groups (if design includes
pretest)
4. Introduce the stimulus (IV)
5. Observe (measure) DV for each group
6. If the change in the value of the dependent variable varies
significantly across groups, then we conclude that X  Y

Key distinguishing features of an experimental design: Randomization and
Manipulation of IV by the researcher (when introduced and to whom)
Experimental Designs
Random
Assignment
A bunch of
people
Random
Assignment
Measure
the DV
Introduce
the IV
Measure
the DV
Treatment
Group
“Stimulus”
Treatment
Group
Measure
The DV
Control
Group
(PRE-TEST)
Measure
the DV
“Placebo”
Control
Group
(POST-TEST)
Simple Experimental Designs
2-Group Pretest - Posttest Design (Classical or
“Simple” Experiment)
 R O1 X O2
 R O3
O4
Simple Experimental Designs
2-Group Pretest - Posttest Design (Classical or
“Simple” Experiment)
 R O1 X O2
 R O3
O4
 OR
 R O1 X1 O2
 R O3 X2 O4
Experiments and Causality

Correlation?
Experiments and Causality


Covariation?
Comparison of two or more groups (on
dependent variable) experiencing different
levels of exposure to the causal (explanatory)
variable (X). This establishes covariation.
Experiments and Causality

Time Order?
Experiments and Causality


Time Order?
The introduction of the independent variable
(“stimulus”) is manipulated by the researcher
to insure that changes in IV precede changes
in DV.
Experiments and Causality

Spuriousness?
Experiments and Causality


Spuriousness?
Random assignment insures that rival
hypotheses are ruled out, thus eliminating the
threat of spuriousness. (How?)
Experiments and Causality



Spuriousness?
Random assignment insures that rival
hypotheses are ruled out, thus eliminating the
threat of spuriousness. (How?)
Use of “matching” as a strategy to control for
rival explanations
Simple Experimental Designs



2-Group Posttest Only Design
R X O1
R
O2
Simple Experimental Designs






2-Group Posttest Only Design
R X O1
R
O2
OR
R X1 O1
R X2 O2
Other Types of Experimental Designs

Multiple Group Pretest - Posttest Design
R O1 X1 O2
R O3 X2 O4
R O5 X3 O6
R O7 X4 O8

Multiple Group Posttest Only Design
R X1 O1
R X2 O2
R X3 O3
R X4 O4
Other Types of Experimental Designs

Solomon 4-Group Design
R O1 X O2
R O3
O4
R
X O5
R
O6
Extensions of the Classical Experiment

Multiple observations over time
R O1 X1 O2 O3……
R O4 X2 O5 O6……
Extensions of the Classical Experiment

Factorial designs - each group represents a unique combination of
values on two (or more) different variables.

For two independent variables X and Z (where X and Z each take on two
possible values):
R O1 X1, Z1 O2
R O3 X2, Z1 O4
R O5 X1, Z2 O6
R O7 X2, Z2 O8

Factorial designs allow the researcher to test for an interaction effect:
Two independent variables interact to affect a dependent variable if the
effect of one variable depends on the value of the second variable.
Zilber and Niven (SSQ)

Table 1: 2-Group Posttest Only
R X1(black) O1
R X2(A-A) O2
Zilber and Niven (SSQ)

Table 3: 2X2 Factorial Design

R
R
R
R




(black/liberal)
(A-A/liberal)
(black/conserv)
(A-A/conserv)
O1
O2
O3
O4
To see how the effect of racial label varies as a
function of ideology, we compare
O1-O2 to O3-O4
Zilber and Niven (SSQ)

Table 3: 2X2 Factorial Design

R
R
R
R




(black/liberal) O1
(A-A/liberal) O2
(black/conserv) O3
(A-A/conserv) O4
Conclusion: The choice of racial labels does affect white
attitudes toward blacks, but only among liberals.
1.What effect do A & B
have on O?
2. Is there an interaction
effect? (Explain)
Evaluating Research Designs:
Internal Validity

Internal Validity - the degree to which we can be sure that the
independent variable caused the dependent variable within the
current sample

“Extrinsic Factors”: Selection effects with respect to
recruitment/assignment of subjects (units) to treatment and control
groups

“Intrinsic Factors”: Factors threatening validity that…



occur outside the “laboratory” during the period of the study
result from changes in, reactions to (or general ineffectiveness of) the
measuring instrument, or
involve some type of reactive effect of observation
Evaluating Research Designs:
Internal Validity

Extrinsic factors:


Selection
Important intrinsic factors include:







History
Maturation
Experimental mortality
Instrumentation
Testing
Regression artifact
Interactions with selection - e.g. “selection history” and
“selection maturation”

For each of the following intrinsic threats to
internal validity, explain:


What the specific threat means
Whether or not (and why or why not) an
experimental design is protected from this threat
History?


R O1 X1 O2
R O3 X2 O4
Maturation?


R O1 X1 O2
R O3 X2 O4
Experimental Mortality?


R O1 X1 O2
R O3 X2 O4
Instrumentation?


R O1 X1 O2
R O3 X2 O4
Testing?


R O1 X1 O2
R O3 X2 O4
Regression Artifact?


R O1 X1 O2
R O3 X2 O4
“Selection History” and “Selection Maturation”


R O1 X1 O2
R O3 X2 O4
Evaluating Research Designs:
External Validity

External Validity - the degree to which the
results of the analysis can be generalized
beyond the current sample/study. Can be
maximized by:


Using subjects (units) that are representative of
the population to which one’s theory applies
Using a “laboratory” that is as close to “real life”
conditions as possible

Field experiments
Applications
1. Iyengar, Shanto, Mark D. Peters, and Donald R. Kinder. 1982. “Experimental Demonstrations
of the ‘Not-So-Minimal’ Consequences of Television News Programs.” American Political
Science Review 76: 848-58.
2. Schram, Sanford F., Joe Soss, Richard C. Fording and Linda Houser. 2009. “Deciding to
Discipline: A Multi-Method Study of Race, Choice, and Punishment at the Frontlines of
Welfare Reform .” American Sociological Review, 74(3), 398-422.
3. Gerber, Alan S., Donald P. Green, and Christopher W. Larimer. 2008. “Social pressure and
voter turnout: evidence from a large-scale field experiment.” American Political Science
Review 102:33–48.
Assignment #5
(Due October 5): In approximately 2 single-spaced pages, answer the following questions.
1. For each of the three application readings, identify and diagram (using the notation in
Frankfort-Nachmias and Nachmias) the specific type of experimental design employed by
the authors.
2. Choose one of the three studies to focus on for this question. Evaluate the internal validity and
external validity of the study you have chosen.