Epistemology and Methods
Small-N and Large-N Studies
May 18 2010
Conflict vs. co-existence
• Methods are used to test theories or assist in theory-building
• Quantitative or qualitative methods have different strengths
and weaknesses
• Different “group think” attitudes have led to sharp divisions
• Common quest, different routes…
Qualitative methods: what is this?
Other “label”: case study methods (single cases and comparison
of cases)
• Mostly used qualitative method is:
• Process-tracing
– Whether intervening variables between a hypothesized cause and
observed effect move as predicted by theories
• Also used, albeit less frequently, is:
• Counterfactual analysis
– Whether x in a specified case was necessary for y
Case study designs
Forms of single case study design
1)
Descriptive case study
Written by participants or historians
2)
Preliminary illustration of a theory
Keohane (1984) on the role of regimes
Case study designs
3)
Disciplined interpretative case study
• Interpretation/explanation of an event by applying a
known theory
• Could lead to improvement of theory
• Risk: underplaying evidence inconsistent with the
argument, eclectic approach (which factors are more
important)
• Remedy: Engage sincerely in alternative explanations,
add counterfactual arguments
Case study design
4)
Hypothesis-generating case study
•
Schattschneider (1935) Politics, Pressures, and the Tariff
Pressure group politics
•
Kindleberger (1973)“that for the world economy to be
stabilized, there has to be a stabilizer, one stabilizer“
Case study design
5)
Least-likely (theory-confirming) case study
• Extreme case that is highly unlikely to confirm
• Lends strong support if confirmed
• Example: The WTO treaties constrain actor’s national
policies – case-study on the US
Case study design
6)
Most-likely (theory-infirming) case study
• An important single case study that disconfirmed the
expected outcome even though conditions make the
case favorable for theory
• Example: The WTO dispute settlement system is biased
against emerging developing countries – case-study on
Brazil’s application and success rate…
Case study design
7)
Deviant case study (outlier cases)
•
•
•
Shedding light on the limits of a theory
Suggesting new hypotheses
Japan’s attack on Pearl Harbor and Deterrence Theory
(Russett 1967)
Case study design
Comparative methods (Mill’s Methods and Least-Similar and MostSimilar Case Comparisons)
• The method of agreement (least similar case design)
• Search for similar antecedent conditions / ideally necessary conditions
• E.g. negotiations in GATT vs. WTO (A: G2 power)
IV
DV
Case 1
ABCDE
Y
Case 2
AFGHI
Y
Case study design
Comparative methods
• The method of difference (most similar case design)
• Method of controlled comparison
• BCDE (constant)
• E.g. disputes on similar cases: GATT vs. WTO
IV
DV
Case 1
ABCDE
Y
Case 2
~ABCDE
~Y
Case study design
Advantages of case studies
• Generate valid theory
• Refining theory, generate new hypotheses
• Strong for documenting processes /making inference regarding
causal mechanisms
• Finding omitted variables
• Key events better explained than in large-n statistical tests…
Case study design
Limits of case studies
•
•
•
•
•
•
Less useful for systematic testing a theory
Case selection bias
Confirmation bias
Potential indeterminacy
Representativeness (generalizability vs. specificity)
Lesser precision of magnitude of causal effects
Quantitative methods
What is statistical method capable of doing?
• Short-cut: “it permits the researcher to draw inferences about
reality based on the data at hand and the laws of probability”
Quantitative methods
Advantages:
• Powerful tool to “aggregate information” from a large amount
of data
• Clear transparent coding process (high reliability, possibility
for replication)
• Visual display
• Test whether association between variables is a product of
chance
Quantitative methods
Advantages:
• Measure the effect of a change on the IV on the DV
• Assess the “contribution” (explanatory power) of an IV
(average explanatory effects)
• Mapping of “deviant cases”
• Generalizability
Quantitative methods
Limits:
•
•
•
•
Identifying new variables
Dealing with multiple conjunctural causality or equifinality
Validity of operationalization of variables
Role of important cases
Quantitative methods
Errors of Specification:
•
•
•
Too much effort calculating correlations with little attention
to theory (i.e. Democratic peace)
Theory itself often imprecise/shallow – does not lend itself
to be tested (i.e. Waltzian balancing vs. bandwagoning)
Imposing a statistical model on the theory (inattention to
causal processes...)
Quantitative methods
Errors of Inference:
• Focus on statistical significance (probability that relationship
between A and B occured by chance) vs. substantive
significance (magnitude of the relationship)
• Mining datasets /few non-results make it to publication
Summing Up (Mahoney and Goertz 2006)
Summing Up (Mahoney and Goertz 2006)