R ESEARCH M ETHODS
N ATIONAL R ESEARCH U NIVERSITY
HSE E CONOMICS , P H . D P ROGRAM
D R C S L EONARD
J UNE 2011
Lecture 1
What is a good question?
O UTLINE
2

Knowledge Claims




RESEARCH METHODS
Classical Approaches
Post-positivism
Strategies of inquiry


OF LECTURE
Data mining
Design-based research
Good questions emerge from
good research designs
8/06/2011
K NOWLEDGE C LAIMS
RESEARCH METHODS
3
8/06/2011
K NOWLEDGE C LAIMS
4

What warrants knowledge?

How is scientific method applied?
RESEARCH METHODS
8/06/2011
T WO P OSITIONS
5

Two scientific positions, inductivism and
deductivism

From which method emerges

Deductive (logic) argumentation


If premises are true and no fallacies in the
argument, then conclusion will be true

Not concerned with truth or falsity
Inductive arguments may have true premises,
but we cannot be certain that conclusions will
also be true (ampliative reasoning)
LEONARD GSOM PH.D. RESEARCH METHODS 2010
4/02/20010
I NDUCTIVE , D EDUCTIVE
6

Deductive, from general to particular, and inductive,
from particulars to general

Inductive: Frances Bacon vs the medieval Church:
purging ourselves of idols

Problem (Hume) Can we predict the future? No

Positivism is descended from Bacon

Research becomes historical, truth confined to a
systematic empirical study, that might obtain general
laws

Empirical findings worthless to some deductivists
LEONARD GSOM PH.D. RESEARCH METHODS 2010
4/02/20010
7
FALSIFICATION :
K ARL P OPPER (1902-1994)

Karl Popper’s critical rationalism has generated
much debate since the 1930s

Intellectual autobiography, Unended Quest

Background, early Marxist, training with Adler
and Freudian theories convinced them that the
theories were too broad


Later rejected psychologism
Favored theory of relativity, Einstein, could be
tested, verified, falsified
LEONARD GSOM PH.D. RESEARCH METHODS 2010
4/02/20010
8
L OGIC OF FALSIFICATIONISM

Scientific theories are abstract


can be tested only indirectly, by reference to their
implications.
Scientific theory, and human knowledge generally, is
irreducibly conjectural or hypothetical

to solve problems that have arisen in specific historiocultural settings

Logically, no number of positive outcomes at the level of
experimental testing can confirm a scientific theory

A single counterexample is logically decisive: it shows the
theory, from which the implication is derived, to be false.
RESEARCH METHODS
8/06/2011
VS
9
INDUCTIVISM

Direct antithesis of inductivism

Growth of knowledge requires overturning previous beliefs

Have a theory, test it, falsify it and move on

Many economists seek to prove theories correct; the job of
science is to disprove them

Don’t strive for certainty (verificationism)

Bans ad hoc adjustments to a theory to prevent it from being
falsified

A new theory will possess greater empirical content than its
predecessors
LEONARD GSOM PH.D. RESEARCH METHODS 2010
4/02/20010
C RITICS
10

This hinders not promotes science

Can’t reject theories so easily, some theories are
better at some things than others

Marxism, accommodates business cycles and
disequilibrium better then orthodox Keynsianism,
but the Monetarists better understand inflationary
processes than the Keynsians, who argue that their
theories and policies are more effective against
unemployment

Should not critique a new theory too rigorously,
because it may have something good in it
LEONARD GSOM PH.D. RESEARCH METHODS 2010
4/02/20010
DEFENDER :
11

B LAUG
Blaug (be taken seriously)

Have a prediction about the future

Require a formal model

Falsification is essential

Be scientific, or not

Falsificationism: much tougher

Lay down restrictions on what Popper calls
immunizing strategems
RESEARCH METHODS
8/06/2011
E NORMOUS
12

INFLUENCE
Mark Blaug

Even econometricians, however, econometric
results difficult to falsify

Plain fact

Most economists tend to verify..
LEONARD GSOM PH.D. RESEARCH METHODS 2010
4/02/20010
T HOMAS K UHN
13

Structure of Scientific Revolutions

Science is not good science unless it is working under the
framework of a theory, makes no progress

It adopts a particular view of the world, and all
subsequent research adds to that

Pre science—lots of theories hoping to explain the same
thing

The paradigm: an achievement so important that it
attracts an enduring group of adherents

Commitment and consensus are prerequisites for normal
science
LEONARD GSOM PH.D. RESEARCH METHODS 2010
4/02/20010
PARADIGMS
14

Older generations stay with their paradigms,
new ones acquire new paradigms

Releases scientists from the necessity of
debating fundamentals

They can then concentrate on subtle, esoteric
aspects of their subject
LEONARD GSOM PH.D. RESEARCH METHODS 2010
4/02/20010
K UHN : N ORMAL S CIENCE
15

Mopping up

Determination of facts

Setting the facts within theory


Articulating the theory
Then, anomalies, followed by crisis, followed by
fundamental change
LEONARD GSOM PH.D. RESEARCH METHODS 2010
4/02/20010
A PPLIED
16
ECONOMETRICS

Middle ground

Use theory, provide initial specification

Data exploration techniques to extend or refine it

Bridge theory and empirical data analysis

How do we know a theory is correct?

Different users have different tastes and beliefs

Complications with computation: large data sets
numerous models possible
LEONARD GSOM PH.D. RESEARCH METHODS 2010
4/02/20010
PARADIGMS
17

How researchers will learn/what they will learn,
assumptions

Philosophical assumptions, epistemologies (how
we know something), ontologies (what is
knowledge), axiology (what values go into
knowledge), methodology (process for studying)
LEONARD GSOM PH.D. RESEARCH METHODS 2010
4/02/20010
E XTENSIONS
18

These debates shaped much social science
theory about

Innovations

Science

Path dependence

Historical legacies

Nature of change

Nature of reform and timing
LEONARD GSOM PH.D. RESEARCH METHODS 2010
4/02/20010
P OSITIVISM
19

Can we be positive about our claims
of knowledge when studying
behavior and actions (Comte, Mill,
Durkheim, Newton and Locke)?

Causes probably determine effects?

Reductionism: reduce ideas into small
discrete sets to be tested
RESEARCH METHODS
8/06/2011
P OSITIVISM VS
20
I NTERPRETIVISM

Interpretivism: Weber (Verstehen)
RESEARCH METHODS
8/06/2011
P OSITIVISM AND ITS
O PPONENTS
21
RESEARCH METHODS

Quantitative, positivist, post-positivist research,
empirical science

Challenge to positivism: against the traditional
notion of the absolute truth of knowledge;
playing tennis with the net down
8/06/2011
M ETHODOLOGICAL
PLURALISM ?
22

Bruce Caldwell (let 100 flowers bloom)

Little economics will survive if we take this
seriously

Confirmationism Verification

Falsificationism is never practiced because it is
unpracticeable
RESEARCH METHODS
8/06/2011
B AYSIAN
23
METHODS
 Test, verify and refine
the laws and theories
governing behavior
 Baysian methods:
from theory, to test,
to revision
RESEARCH METHODS
8/06/2011
(1) P OST P OSITIVISM
24

Knowledge is conjectural

Research is to make claims and refine or
abandon them

Data, evidence and rational considerations
shape knowledge

Being objective is key
RESEARCH METHODS
8/06/2011
(2) S OCIAL CONSTRUCTION
25

Social construction

Mannheim, Burger, Luckmann, Neuman

Look at participants views

Judgments are subjective, meanings are varied and
multiple

Interviews: open ended questioning, the more the
interviewee talks spontaneously, the better

Participants allowed to construct meaning (rather
than responding to concrete situations)

Process of interaction, context of work
RESEARCH METHODS
8/06/2011
(3) P RAGMATISM
26

Pierce, James and Dewey

Knowledge claims arise out of actions, situations,
consequences, rather than ex ante conditions

Concern with what works

Pluralistic approach

Mixed methods, qualitative, quantitative

Research always occurs in social contexts

Stop asking questions about the laws of nature
RESEARCH METHODS
8/06/2011
L OGIC OF R ESEARCH D ESIGN
RESEARCH METHODS
27
8/06/2011
28
“A CROSS
MOST FIELDS … APPLIED ECONOMISTS ARE
NOW LESS LIKELY TO PIN A CAUSAL INTERPRETATION OF
THE RESULTS ON ECONOMETRIC METHODOLOGY ALONE .
D ESIGN - BASED STUDIES ARE DISTINGUISHED BY THEIR
PRIMA FACIE CREDIBILITY AND BY THE ATTENTION
INVESTIGATORS DEVOTE TO MAKING BOTH AN
INSTITUTIONAL AND A DATA - DRIVEN CASE FOR
CAUSALITY.”
J OSHUA D. A NGRIST
RESEARCH METHODS
AND
J ÖRN -S TEFFEN P ISCHKE
8/06/2011
G ENERAL TO S PECIFIC
M ETHODOLOGY
29
“LSE” tradition of time-series econometrics that began in the 1960s
at the London School of Economics
Mizon (1995) a brief history
The practitioners of LSE econometrics are now widely dispersed
among academic institutions throughout Britain and the world.
The LSE approach is described sympathetically in Gilbert (1986),
Hendry (1987,1995, esp. chs. 9-15), Pagan (1987), Phillips (1988),
Ericsson, Campos and Tran(1990), and Mizon (1995).
For more sceptical accounts, see Hansen (1996) and Faust and
Whiteman (1995, 1997)
RESEARCH METHODS
8/06/2011
G ENERAL
30
TO
S PECIFIC
Context: Linear (cross-country growth) Use: Time-Series
Step 1.
General regression will include every possible variable -- all the
information about the true determinants.
Step 2.
The information content is then sharpened by a more parsimonious
regression – the specific regression
it is statistically well specified (for example, it has white
noise errors);
(b) that it is a valid restriction of the general regression, and
(c) that it encompasses every other parsimonious regression
that is a valid restriction of the general regression
(a)
Criticism: data-mining,
RESEARCH METHODS
8/06/2011
31
E XTREME B OUNDS A NALYSIS :
THE NEW CRITIQUE

Edward Leamer’s “extreme-bounds analysis” (1983, 1985).

A coefficient of interest is robust only to the degree that it
displays a small variation to the presence or absence of other
regressors. Leamer and Leonard (1983) define the extremebounds for the coefficient of a particular variable within a search
universe as ranging between the lowest estimate of its value
minus two times its standard error to the highest estimate of its
value plus two times its standard error, where the extreme values
are drawn from the set of every possible subset of regressors that
include the variable of interest. A variable is said to be robust if
its extreme bounds lie strictly to one side or the other of zero.
RESEARCH METHODS
8/06/2011
J UDGMENT C ALLS
32
The main difference between structural and
experimental (or ``atheoretic'') approaches is not in
the number of assumptions but the extent to which
they are made explicit. (Michael Keane)
RESEARCH METHODS
8/06/2011
S UMMARY
33


Quantitative (numbers)

Experimental design--controls

Non experimental design, surveys
Qualitative (words)


Narratives, phenomenologies, ethnographies,
grounded theory, case studies
Mixed methods

RESEARCH METHODS
Sequential, concurrent, transformative
8/06/2011
Q UANTITATIVE
34

Random assignment of subjects to treatment

Quasi experiments: non random designs

Surveys cross sectional and longitudinal,
generalize from sample to population
RESEARCH METHODS
8/06/2011
Q UALITATIVE
35

Ethnographies: researcher studies an intact cultural
group in its setting over time (responses)

Grounded theory


Derive an abstract theory of a process, action or
interaction, grounded in views of participants
Case Studies

Exploring in depth a program, event, activity, process, or
individuals, bound in time, variety of procedures

Phenomenological: lived experiences

Narrative research: lives, stories, retellings
RESEARCH METHODS
8/06/2011
M IXED
36

Gets around biases in any one method used
exclusively

Progress from one method to another

Illustrate

Determine what the concept is

Test assumptions on one case
RESEARCH METHODS
8/06/2011
L OGIC
37


Qualitative:

Wider range of methods, non-numerical by definition

Small n, intensive interviews, depth analysis, discursive,
account of event or unit

Focus on event, decision, institution, location, issue or
legislation

Incident important in its own right (war, election, change
in leadership, marketing strategy, community decision,
etc)
Against bifurcation? mixed methods

Systematic, scientific research of all kinds

Most research does not neatly fit one or other category
RESEARCH METHODS
8/06/2011
G OAL OF SOCIAL SCIENCE
38
RESEARCH


Inference (descriptive, explanatory)

Attempting to infer beyond immediate facts to
something broader

Learning about causal effects from data
Public procedures (explicit, codified)

Replication

Conclusions are uncertain

Observes rules of inference
RESEARCH METHODS
8/06/2011
D ESCRIPTIVE
39
INFERENCE

Distinguish systematic from non-systematic
features

Systematic from stochastic

Counter-factuals (what would have happened,
had meters not struck the earth 65 million yrs
ago)
RESEARCH METHODS
8/06/2011
R ULES FOR R ESEARCH
D ESIGN
40

Intuition: Choice of better topics is idiosyncratic

Two ways to test if it is a good topic

Is it about something important in the real world

A research topic should make a specific contribution to
an identifiable scholarly literature by increasing our
ability to construct verified scientific explanations

ie: locating it within the framework of existing social
science literature

This is the subject of the second lecture today—what
makes a theory or theoretical contributions valuable to
the community of editors of journals
RESEARCH METHODS
8/06/2011
C AUTION
41

There may be reasons a theory is practicable,
even though its long term scientific value has
been questioned

Theoretically incoherent models used to
forecast the US economy—diversion of
macroeconomic theory and applied
macroeconomics (see Mankiw 1990)

New theories, however, remain speculative
RESEARCH METHODS
8/06/2011
42
D ESIGN
BASED
R ESEARCH

Leamer 1983 highlighted the benefits of
sensitivity analysis, a procedure in which
researchers show how their results change with
changes in specification or functional form.
Sensitivity analysis has had a salutary but not a
revolutionary effect on econometric practice.

As we see it, the credibility revolution in
empirical work can be traced to the rise of a
design-based approach that emphasizes the
identification of causal effects.
RESEARCH METHODS
8/06/2011
W OW -FACTOR
43

Design-based studies typically feature either
real or natural experiments and are
distinguished by their prima facie credibility and
by the attention investigators devote to making
the case for a causal interpretation of the
findings their designs generate.

Design-based studies are most often found in
the microeconomic fields of Development,
Education, Environment, Labor, Health, and
Public Finance, but are still rare in Industrial
Organization and Macroeconomics.
RESEARCH METHODS
8/06/2011
L ITERATURE R EVIEW
RESEARCH METHODS
44
8/06/2011
L ITERATURE R EVIEW
45

Classical

Systematic review

Meta-analysis

Narrative review

Search issues

Presentation
RESEARCH METHODS
8/06/2011
C LASSICAL L ITERATURE
R EVIEW
46

Large Disciplinary Differences

Sociology, Psychology, Business

Economics
RESEARCH METHODS
8/06/2011
S YSTEMATIC R EVIEW
47

A systematic review aims to provide an exhaustive
summary of literature relevant to a research
questions

The first step of a systematic review is a thorough
search of the literature for relevant papers.

The Methodology section of the review will list the
databases and citation indexes searched as well as
any individual journals. Next, the titles and the
abstracts of the identified articles are checked against
pre-determined criteria for eligibility and relevance.
RESEARCH METHODS
8/06/2011
M ORE
48

Ability to control for between-study variation

Including moderators to explain variation

Deal with information overload: the high number
of articles published each year.

It combines several studies and will therefore be
less influenced by local findings than single
studies will be.

Makes it possible to show if a bias for published
works exists.
RESEARCH METHODS
8/06/2011
S YSTEMATIC A NALYSIS
49

Meta-Analysis

Meta-analysis leads to a shift of emphasis from
single studies to multiple studies. It emphasizes
the practical importance of the effect size instead
of the statistical significance of individual studies.
This shift in thinking has been termed "metaanalytic thinking"
RESEARCH METHODS
8/06/2011
50
M ETA -A NALYSIS
( AVERAGE E FFECT S IZE , F OREST P LOT )
RESEARCH METHODS
8/06/2011
M ETA -A NALYSIS :
51

About science, but not science


CRIT
Statistical examination of scientific studies
Cannot propose ways to falsify a theory
RESEARCH METHODS
8/06/2011
52
M ETA -N ARRATIVE R EVIEW

Example: Connected Communities’,


Meta-narrative review shows the diversity in the meaning of
‘communities’ --various conceptualisations and meanings of community
across disciplines, over time, and within different cultures and contexts


a research programme of the Institute of Health and Human
Development (IHHD) to investigate the meanings of community within
and across research disciplines by adopting an innovative methodology
based on a meta-narrative systematic review approach. Policy and
academic interest in the concept of ‘community’ is longstanding and
such interest has become central to policy making in the last two
decades.
(Greenhalgh et al, 2005) is a type of ‘systematic’ review rather than a
traditional expert driven literature review
A focus on identifying the ‘storylines of research’ within and across
disciplinary boundaries. Identifies the meta-narratives of each discipline
and analyse the different ‘discourses’ and languages of ‘community’.
RESEARCH METHODS
8/06/2011
THE END