Research Design
Part II: Cross-sectional and
Quasi-Experimental Designs
•
•
•
•
Frankfort-Nachmias & Nachmias (Chapter 6 – Cross-Sectional and Quasi-Experimental Designs)
Gerring (Chapter 8)
Campbell and Stanley, “Experimental and Quasi-Experimental Designs for Research.” (remainder)
King, Keohane and Verba (Chapter 5, sections 5.2 - 5.6)
Applications
Stack, S. and Gunlach, J. (1992) “The Effect of Country Music on Suicide.” Social Forces 71: 211–18.
Lawrence S. Rothenberg; Mitchell S. Sanders, “Severing the Electoral Connection: Shirking in the Contemporary
Congress.” American Journal of Political Science, Vol. 44, No. 2. (Apr., 2000), pp. 316-325. (Difference-indifference)
Writing a Literature Review

Typical format for research article






Introduction
Literature Review
Theory
Research Design
Empirical Analysis and Results
Conclusion
Writing a Literature Review

Purpose of literature review


Inform reader of prior relevant work
Persuade reader that your work is important
(justify your research)
Possible Justifications for Your
Research







New question, new theory
New question, existing theory
Old question, new theory
Old question, conflicting theories
Old question, conflicting findings
Old question, new methods
Old question, new data
Literature Review Don’ts


Don’t (just) provide a chronological listing of
article summaries
Don’t provide every detail of every study
Literature Review Do’s




Organize your discussion of the literature in a way
that reflects and supports the justification for your
research
Provide more detail for seminal studies, less detail
(or simply a citation) for less cited studies
For questions that have been studied extensively, it
is not necessary to cite every study
End your literature review with a summary and
critique that justifies your research
Literature Review Assignment


Approx. 7-10 pages, double-spaced
Due November 2nd
How to Identify the Relevant Literature


Use electronic databases and keyword
searches (Google Scholar, JSTOR)
Prioritize:





Articles
Articles published in highly-ranked journals
Recently published articles
Seminal articles
Number of articles: 10-20?
Quasi-experimental and CrossSectional Designs

What are they?



Quasi-experimental – study of more than one sample
(often over period of time)
Cross-sectional – Analysis of a single sample (lacks
random assignment, temporal variation, and
manipulation); but includes comparison groups
Pre-experimental – Cross-sectional, with no comparison
group; causal inference impossible
Quasi-experimental and CrossSectional Designs

Why?



Property-disposition relationship
vs.
Stimulus-response relationship
Pre-Experimental Designs

One-Shot Case Study


One group
No variation in independent variable
X O
Pre-Experimental Designs

Example:



Dependent variable: Americans’ support
for campaign finance reform
Independent variable: Watergate scandal
Data: 1976 survey of American adults;
examine mean level of support
Cross-Sectional Designs

Static-Group Comparison Design


Two groups – observed at one time
Allows variation in the independent variable
X O1
O2
Elaboration of Static Group Comparison
Design

Correlational / Cross-Sectional Designs
X1 O1
X2 O2
X3 O3
X4 O4
Xi Oi

Problems with correlational/cross-sectional designs?
Example: Wine and Health
Hypothesis: Drinking wine causes individuals
to be healthier (esp. heart)
 Existing studies: compared the health of wine
drinkers to the health of those who do not
drink wine:
Research design
X1 (Wine drinkers) O1 (Health)
X2 (Non-drinkers) O2 (Health)

Spurious Results?
Controlling for Affluence
Research design:
X1 (Affluent Wine drinkers) O1 (Health)
X2 (Affluent Non-drinkers) O2 (Health)
X3 (Poor Wine drinkers)
X4 (Poor Non-drinkers)
O3 (Health)
O4 (Health)
Another Example (?)

http://www.washingtonpost.com/wpdyn/content/article/2006/05/25/AR200605250
1729.html

http://www.sciencedaily.com/releases/2007/0
4/070417193338.htm
Quasi-Experimental Designs

Contrasted Groups Design


Multiple groups, based on some categorical variable
Observed at one point in time (similar to cross-sectional)
O1
O2
O3
O4
Oi

Problems with contrasted groups designs?
Quasi-Experimental Designs

One-Group Pretest-Posttest Design


One group
Allows variation in the independent variable
O1 X O2
Elaborations of the One-Group PretestPosttest Design

Time Series Designs

simple vs. extended
O1…Ok X Ok+1…Om
k = # of pretest observations
m = total # observations
Or (“equivalent time samples design”)
X1 O1 X2 O2 X3 O3 … Xm Om
Example: Murray’s “Poverty-Spending
Paradox” (Schram 1991)
Example: Murray’s “Poverty-Spending
Paradox” (Schram 1991)
Nonequivalent Control Group Design

No random assignment
O1 X O2
O3
O4
or
O1 X1 O 2
O3 X2 O 4
Control Series Designs

Addition of second (control) group to time
series design (CS: “multiple time series
design”)
O1 O2 O3 X1 O4 O5 O6……
O7 O8 O9 X2 O10 O11 O12……
Panel Designs

Repeated observations of the same units over
time

Also goes by:


Pooled time-series design
Pooled cross-sectional time-series design
The Effect of Sanctioning in the TANF Program
Quarterly Income of Sanctioned Clients
1800
1600
1400
1200
1000
800
600
Sanctioned off TANF
400
3 Quarters 2 Quarters 1 Quarter
Prior to
Prior to
Prior to
Entry
Entry
Entry
TANF
Quarter
1 Quarter 2 Quarters 3 Quarters
After Exit After Exit After Exit
The Effect of Sanctioning in the TANF Program
Quarterly Income of Sanctioned and Non-sanctioned
Clients
1800
1600
1400
1200
1000
800
600
400
3 Quarters 2 Quarters 1 Quarter
Prior to
Prior to
Prior to
Entry
Entry
Entry
TANF
Quarter
Not Sanctioned
1 Quarter 2 Quarters 3 Quarters
After Exit After Exit After Exit
Sanctioned
Donahue and Levitt

Hypothesis: The (legal) availability of
abortion in a state is negatively related to the
crime rate (many years later).
O1 O2 O3 X1(early legalization) O4 O5 O6…
O7 O8 O9 X2 (legalize 1973) O10 O11 O12…
Gerring’s Criteria for Research Design










Plentitude (N)
Boundedness (relevant cases)
Comparability (descriptive/causal)
Independence
Representativeness
Variation (X, Y, X&Y)
Analytic Utility (of the sample)
Replicability
Mechanism
Causal Comparison
KKV – Overcoming Common
Problems





Omitted Variable Bias
Inclusion of Irrelevant Variables
Endogeneity
Assigning Values of the IV’s
Controlling the Research Situation
Does Country Music Cause Suicide?
Stack & Gundlach


Hypothesis: There is a positive relationship between
exposure to country music and suicide rates
Research design:
X1 (no country music) O1 (suicide rate)
X2 (1 station)
O2 (suicide rate)
X3 (2 stations)
O3 (suicide rate)
X4 (3 stations)
O4 (suicide rate)
Xi ( etc.)
Oi (suicide rate)
Stack & Gundlach

Findings: 51% of the variation in urban white
suicide rates can be explained by variation in
airtime devoted to country music

Internal Validity?
O1A
O2B
O3C
RetiredA
O4A
SW-OfficeB O5B
ReturnedC O6C
Regression Discontinuity Designs

Example: What is the effect of an award on
later achievement? (from CS)
Regression Discontinuity Designs

Example: What is the effect of an award on
later achievement? (from CS)

Inferential challenge: Award recipients are likely
to do well anyway, even without the award,
because criteria for receiving award also predict
future success
A regression discontinuity design is appropriate for any research design in which the assignment of the
treatment is determined by a continuous variable that is also related to the outcome of interest.
Ludwig and Miller, 2007
Head Start



Head Start is a federal program that provides
comprehensive education, health, nutrition,
and parent involvement services to lowincome children and their families.
Head Start began in 1965
Local nonprofits/education agencies must
apply to federal government to be a Head
Start provider
Head Start

Research Question: What is the effect of Head
Start on health/well-being of former
participants?

What problems might we encounter if we
were to conduct the following study:
X O (where X = Head Start participation)
O
Ludwig and Miller, 2007

OEO and application assistance to poor
counties in 1965


“Treatment” group – 300 poorest counties that
received head start application assistance
“Control” group – 228 poor counties that did not