INTRODUCTION TO RESEARCH DESIGN
Jonas Pontusson
04.11.2011
what is this about?
• common challenges that we all face in developing and executing a research project:
• research questions
• theory building (note: focus on causal arguments)
• testing/evaluating theories through empirical analysis
• practical approach: discussion of specific examples (optional readings), mostly in the
area of “comparative” but more broadly relevant
• key theme: negotiating trade-offs
• no prepared discussion of the following:
• “methodology” in a technical sense
• philosophy of science (epistemology)
• strictly “practical” matters (grant applications, field research, etc.)
• bottom line: what I think you ought to think about before embarking on your
dissertation research (cumulated wisdom)
about me
• US-trained “comparativist”
• started out as qualitative work, turning quantitative
• quantitative macro-comparative work: incorporating insights from “qualitative
tradition”
• recent efforts to engage “multi-level analyses:” combining individual-level
survey data and macro-level variables
readings for general discussion
• King, Keohane and Verba (KKV), Designing Social Inquiry (DSI) (1994)
• bridging qualitative/quantitative divide: different data and methods, but
common logic of inference
• telling qualitative researcher how to improve by adopting standards of
qualitative social science
• APSR symposium: Laitin, Caporaso, Collier, Rogowski, Tarrow
• Brady, Collier and Seawright, eds., Rethinking Social Inquiry (2004)
KKV
basic vision of how “science” work:
• elaborate a (casual) theory
• generate empirical implications (the more the merrier)
• test the hypotheses
advice to qualitative researchers:
•
•
•
•
•
specify clear what you are trying to explain (DV)!
make sure that your measures are valid!
maximize “leverage,” minimize “bias”!
maximize variation on X (N), never select on the DV!
estimate uncertainty!
today’s agenda
1. research questions: where do we get them from? why are some more
interesting than others?
2. link between theory and research
3. analytical leverage (research design)
4. causal mechanisms
5. combining different research designs (“mixed methods”)
warning
• 48 slides to follow
• I conceived this presentation before I saw your projects: much of it is not
directly relevant…
Topic #1:
types of research questions
• open-ended questions: why Y? what are effects of X?
not so interesting…
• testing (reasonably) well-specified theories: is the theory “true”?
potentially interesting…
• puzzles
usually interesting, at least a stepping stone…
theory-testing
•
interest depends on (a) status of theory and (b) empirical results
• interesting to confirm a controversial (or counter-intuitive) theory
• interesting to invalidate conventional wisdom
• not so interesting to confirm conventional wisdom
•
testing multiple (competing) theories is almost always interesting!
•
critical importance of engaging with alternative explanations: who are you arguing
against? why are they wrong?
• articulation of your own theory
• empirics: case selection, control variables
puzzles
• theory-based puzzles: theory X (“conventional wisdom”) would lead us to
expect Y (outcome), but this is not what we observe… why?
note: single observation can serve this purpose if the theory is sufficiently
well-specified (cf. Rogowski)
• “real-world” puzzles: why do observe Y in case A, but not case B?
note: such puzzles are only interesting if the cases are similar, i.e., share
(values on) many Xs (MSSD)
framing/marketing research questions
• broader (theoretical) implications of narrow research questions
• turning topics into research questions of broad appeal...
• key question: what is the intellectual community that you are seeking to
join? (political science?)
• in any case: reading broadly is crucial to effective framing!
example of puzzle-driven research project
Evan Lieberman, Race and Regionalism in the Politics of Taxation (2003)
Research question:
Narrow version: Why has South Africa relied more heavily on income taxation than Brazil
(to finance similar levels of public spending)?
Broad version: What determines reliance on income taxation (relative to other revenue
sources)?
Point of departure: level of economic development is not an adequate explanation…
Lieberman’s historical-institutionalist account
• definition of “national political community” at the moment of
founding as key => path dependency
• Brazil: federal race-inclusionary state undermines upper-class
willingness to pay taxes
• S.A.: centralized race-exclusionary state promotes upper-class
willingness to pay taxes
does the argument travel?
key principle (KKV): an inductive argument cannot be tested with cases that
were used to generate the argument
final chapter of Lieberman’s book: cross-section regression analysis of 71
countries (cases)
DV=income+property taxation in % of GDP (1990-94)
regression results
(A)
national govt
(C)
nat+subnat
.39**
(.08)
.83**
(.09)
federal
-3.54**
(1.17)
-3.54**
(1.36)
race-excl fragment
6.50**
(1.38)
4.29**
(1.61)
race-incl fragment
-2.89**
(1.03)
-3.10*
(1.19)
71
.51
71
.64
GDP/cap (000s$)
N
R-square
Lieberman’s process
1. scatterplot to identify exceptions to conventional explanation
2. generate new explanation by probing the experience of exceptional cases
3. test the new theory against data for many countries
Topic #2:
types of explanation
and their implications for research design
types of explanations:
•
frequentist vs. system-level explanations
•
partialist vs. holistic explanationist explanations
•
determinist vs. probabilistic explanations
2 types of research design (for now)
•
MSSD vs MDSD
(Przeworski and Teune, The Logic of Comparative Social Inquiry (1970); cf. John
Stuart Mill, A System of Logic (1843)
MSSD vs. MDSD
MSSD (Mill “method of difference”/matched-case comparisons)
• cases selected on DV (different values/outcomes)
• inductive approach: looking for variables (Xs) that can explain the
outcome
• the more similar the cases, the more confident we are that (a particular) X
is the cause of Y
MDSD (Mill “method of agreement”/akin to regression)
• cases selected on IV (different values)
• deductive approach: prior expectation that X influences Y
• the more different the cases, the more confident we are that X is the
cause of Y
Przeworski and Teune:
MDSD better than MSSD
multi-level questions: why are French workers less likely to vote Left than
Swedish workers?
• frequentist explanation: there is some individual-level variable (e.g.,
religiosity) that takes on different values in the two countries (more
religious workers in France)
• system-level explanation: the relationship between class and voting (also
religion and voting) is different in the two countries
MSSD immediately jumps to system-level; working in the MDSD mode, we
can and should add systemic factors as necessary
Przeworski and Teune:
substitute variables for proper names!
macro level:
• continuous variables (e.g., proportionality)
• typological variables (e.g., PR vs SMD)
partialist vs. holistic explanations
• “holistic” approach (MSSD): complete explanation of Y
e.g.: Immergut on health-care systems
• “partialist” explanation (MDSD): goal is to establish that a causal
relationship exists, not to explain (all of) Y
e.g., Rogowski, Commerce and Coalitions (1989)
deterministic vs. probabilistic explanations
• arguments for maximizing N and against selecting on the DV are premised
on probabilistic theories and/or theories about marginal effects
• deterministic theories (events as outcomes?):
(1) X is a necessary cause (condition) of Y => only cases where Y occurs
are relevant
(2) X is a sufficient cause (condition) of Y => only cases where X occurs are
relevant
Geddes debate
Geddes, “How the Cases You Choose Affect the Answers that You Get,
Political Analysis (1991)
critique of literature that argues (based on the experience of EA and LA
NICs) that political repression promotes econ
(1) “indeterminate research design:”
no inferences can be drawn
(2) truncated sample, East Asia
(3) full sample: no relationship
retorts
1). the theory is about necessary conditions: only upper-left space is relevant
(Jordan?)
2). scope restriction: the theory is only meant to apply to EA (and LA) =>
Przeworki and Teune: develop a theory that transforms “EA” into a
variable (or several variables)!
3). the theory is not about growth per se, but a particular kind of growth:
capital-intensive industrialization (Jordan isn’t relevant)
4). there is a relationship, but it is not linear! some but not too much
repression promotes industrialization
bottom line
• what is the scope of the theory you are testing?
• what kind of causal claims does it entail?
Topic #3:
analytical leverage
•
•
•
•
•
experimental and quasi-experimental designs
individual variation (political behavior)
variation across “macro-social units” (nations, regions/cantons)
variation across policy arenas
temporal variation (sequencing)
prominent hybrids (in comparative):
• pooled cross-section time-series analysis
• two-level pooling of micro and macro variables
an example of qualitative
case comparison
Immergut, “The Rules of the Game” in Steinmo, Thelen and Longstreth, eds.,
Structuring Politics (1992)
what is to be explained?
health systems ca. 1975:
• Sweden: de facto national health service (i.e., socialized medicine), doctors as
government employees, uniform fees
• Switzerland: private system, doctors free to set fees, role of government
limited to subsidies to low-income people
• France as “half-way house:” public insurance and private provision, regulation
of fees
=> outcomes of political struggles 1945-65
Sweden vs. Switzerland
• federal vs. centralized
• dominant party vs. permanent coalition
• overrepresentation of rural areas (CH)
• referenda as means to veto federal legislation (CH)
but the cases are different on many other dimensions… how similar is
similar enough?
key leverage:
temporal variation in the French case
Fourth Republic:
• weak executive
• party fragmentation
• lack of party discipline
 parliament as source of veto power for special interests
Fifth Republic:
• strong executive: directly elected president, decree powers
 completion of healthcare reform in early 1960s
pooled CSTS analysis
method of choice for quantitatively-inclined comparative political
economists working on OECD countries
• only 18-20 countries (due to data limitations, also scope of theories)
• ambition to test conditional arguments
• theorized interaction effects (contrast QCA)
debate:
Comparative Social Research, v. 24 (2007)
Shalev’s critique:
• we fundamentally care about cross-national variation: adding crosstemporal variation does buy us more leverage
• the determinants of cross-sectional and cross-temporal variation are
different
• the interaction effects involve multiple variables: something like QCA is
better suited for this
• don’t get rid of the proper names: they contain lots of useful information!
my principles
• we ought to theorize about specific interaction effects, not simply infer
interaction effects from the data
• we ought to aspire to an explanatory framework that can explain change
over time as well as cross-sectional variation
addressing causal complexity within an CSTS framework
•
are causal dynamics different within different clusters of countries?
• varieties of capitalism (LMEs vs. CMEs)
• welfare-state regimes (Esping-Andersen)
• PR vs. majoritarian systems
•
do causal dynamics change over time?
Kwon and Pontusson, Globalization, Labour Power and Partisan Politics
Revisited,” Socio-Economic Review 8(2010):251-281.
endogeneity problems
ways to address such problems:
• instrumental variables
• experiments
• “quasi-experiments:” discontinuous regression designs
Topic #4:
causal mechanisms
correlation ≠ causation
=> compelling story (theory)
=> empirical evidence
Kwon-Pontusson
theory (adaptation of Garrett):
•
partisan conflict increases with globalization when unions are strong
•
partisan conflict declines with globalization when unions are weak
 3-way interaction of government partisanship, globalization and union
strength: impossible to estimate (multicollinearity)
 second-best solutions…
more systematic approaches
• individual-level analysis of preferences
• process-tracing (qualitative analysis): what did actors think/say/do?
• sequencing (history) as key to causal inference
example of individual-level analysis
Iversen and Soskice, “An Asset Theory of Social Policy Preferences,” APSR
95(2001):874-893.
theory about skill specificity and demand for social insurance: specific skills
provide for well-paying jobs, but finding an equivalent job is difficult =>
demand for insurance
individual-level analysis: skill specificity associated with support for social
spending
the analysis doesn’t prove that insurance motives are at work, but do we
really need that? from a macro-comparative point of view, this analysis
speaks to causal mechanisms…
Topic #5:
combining different research designs
(or modes of argumentation)
• example 1: Lieberman (see above)
• example 2: Ansell, “University Challenges,” World Politics 60(2008):189230.
1. formal model as device to illustrate theory, generate specific
hypotheses
2. quantitative analysis (pooled CSTS)
3. “stylized” case studies to illustrate (demonstrate?) causal mechanisms
Ansell
higher education policy: enrollment levels and structure of funding
1.
2.
3.
formal model: preferences of income groups => partisan preferences
(tipping point)
quantitative analysis: pooled CSTS, OECD countries, 1980-97
case studies Britain, Germany and Sweden: causal mechanisms
note: new research question, but building on existing theoretical framework
critique
• untested assumptions about policy preferences by income
• untested assumptions about links between income groups and parties
• case studies are pretty “stylized:” don’t really add a lot of new information
summing up
• trade-offs in matters of research design: for general discussion
• research as cumulative process:
• you build on your previous work: new research questions
• you build on (and challenge) the work of others
(warning: previous work might be wrong, not only theoretically, but also
descriptively)
additional readings
• on “process tracing:”
Alexander George and Andrew Bennet, Case Studies and Theory Development
(2005)
• on ‘temporality:’
Paul Pierson, Politics in Time: History, Institutions, and Social Analysis (2004)