R ESEARCH M ETHODS
D OC TORAL P ROGRAM , N ATIONA L R ES EARC H U NIVERS IT Y
H IGHER S C HOOL OF E C ONOM IC S
D R C S L EONARD
JUNE 2 0 1 1
O UTLINE
2

What is a theoretical
contribution?

What is a concept?


A variable is a concept
Validity

RESEARCH METHODS
External, Internal
8/6/2011
T HEORY
RESEARCH METHODS
3
8/6/2011
P RACTICAL
4
GUIDELINES

Choose a hypothesis important in literature but for
which no systematic study exists—decide in favour
or against the hypothesis

Choose a theory you suspect is false and
investigate if it is, indeed, and what alternate might
replace it

Design research to illuminate unquestioned
assumptions in the literature

Argue that a topic of importance has been
overlooked

Show relation of theory from one literature to
separate problem from another
RESEARCH METHODS
8/6/2011
R EVISION OF FORMAL
THEORIES ?
5

More than just adding a
variable

Showing how it changes
perception of the problem

Producing data that are
inconsistent with a theory
RESEARCH METHODS
8/6/2011
B ORROW FROM OTHER
DISCIPLINES ?
6

Shed new light on old theories

Investigate qualitative boundaries
of a theory, not just quantitative
ones

Show why a theory will not work
in a new application, not just that
it does not

Focus on multiple elements of a
theory, not just one inconsistency
RESEARCH METHODS
8/6/2011
E VIDENCE REQUIRED FOR
7
REVISING A THEORY

Marshal compelling evidence.

This evidence can be logical (e.g., the
theory is not internally consistent),

empirical (its predictions are
inconsistent with the data
accumulated from several studies), or

epistemological (its assumptions are
invalid-given information from another
field).
RESEARCH METHODS
8/6/2011
P ROVIDE REMEDIES ,
8
ALTERNATIVES

Is the original actually
inferior,

Or simply the best we can
do?
RESEARCH METHODS
8/6/2011
IS
9
IT PUBLISHABLE ?

How radically new is the idea?

Is it linked to evidence, will it change
practice?

Why so? Are the underlying logic and supporting evidence compelling? Are the author's
assumptions explicit? Are the author's views
be-lievable?

Well rounded, broad, deep understanding?

At professional standards?

Who cares? Why now?
RESEARCH METHODS
8/6/2011
W HAT THEORY IS NOT:
(1) REFERENCES
10

For example:

"This pattern is consistent with findings that
industrialization produces growth (Marx 1880,
Solow 1956)
This sentence lists publications that contain
conceptual arguments (and some findings).
But there is no theory because no logic is
presented to explain why industrialization
produces growth
RESEARCH METHODS
8/6/2011
W HAT THEORY IS NOT:
(2) D ATA
11

Introduced to show what patterns are
there


Not why they are there
This approach relies on brute empiricism,

where hypotheses are motivated by
prior data

And not the relevance of theory
RESEARCH METHODS
8/6/2011
(3) L ISTS OF VARIABLES
12
AND DEFINITIONS

Lists of concepts, similarly,
are not theory

Need connections between
variables explained
RESEARCH METHODS
8/6/2011
(4) D IAGRAMS
13
ALONE

What is needed is causal
inference

Time horizon and change over
time

Diagrams as Stage props, not
the performance

Logic needs to be spelled out,
verbally
RESEARCH METHODS
8/6/2011
14
(5) H YPOTHESES : WHAT IS
EXPECTED TO OCCUR
Well-crafted conceptual argument includes
hypotheses
They serve as crucial bridges between theory
and data
Making explicit how the variables and
relationships that follow from a logical
argument will be operationalized.
These are statements about what is expected
to occur, not why it is expected to
RESEARCH METHODS
8/6/2011
I DEAS
15

AND CONCEPTS
A theory generally stems
from a small set of research
ideas, not a list of testable
hypotheses
RESEARCH METHODS
8/6/2011
16
Q UALITATIVE /Q UANTITATIVE
D ISTINCTIONS
Qualitative: Research
Questions
Quantitative: Set of
hypotheses (nul or
directional)
RESEARCH METHODS
8/6/2011
E VIDENCE FOR YOUR
HYPOTHESES / TESTING
17
THEORY
When theories are particularly
interesting or important,
empirical support can be partial:
a small set of interviews,
a demonstration,
experiment,
a pilot survey,
 archival data
RESEARCH METHODS
8/6/2011
E VIDENCE FOR YOUR
18
HYPOTHESES
May point to why a particular
process might be true.
Subsequent research indicated
May show whether the theoretical
statements hold up, or whether they
can repeatedly be falsified
RESEARCH METHODS
8/6/2011
E XTERNAL VALIDITY
RESEARCH METHODS
19
8/6/2011
20
T WO CONCEPTS OF EXTERNAL
VALIDITY,
AS APPLIED TO THEORY
Only one form of validity when the
objective of research is to test theory
RESEARCH METHODS
8/6/2011
E XTERNAL
21
VALIDITY
Such variables only become important in the
context of evaluating interventions based on
theory.
RESEARCH METHODS
8/6/2011
C ONSTRUCT
22
VALIDITY
One needs to make a distinction between
(1) the construct validity of a concept, as
reflected in the convergence (and
discrimination) of some particular set of
operationalizations of it, and
(2) the construct validity of a relation between
two concepts, as reflected in the "fit" of that
relation within some nomological network.
The fit is linked to considerations of external
validity.
RESEARCH METHODS
8/6/2011
23
S AMPLING IS CRITICAL FOR
VALIDITY
Four major sampling strategies that might be
adopted vis a vis any one aspect or facet of the
events under study
1.
Sampling homogeneously over the entire study
2.
Sampling several subsets, each homogeneous
within subset on the facet but differing on it
between subsets, so that all the subsets
together span the whole range
RESEARCH METHODS
8/6/2011
S AMPLING
24
3. Sampling heterogeneously. but in
a way that yields an overall
distribution of the facet among the
cases within the study that reflects
(is representative of) the distribution
of the facet among cases "in nature”
4. Sampling heterogeneously on the
facet but without regard to
representativeness.
RESEARCH METHODS
8/6/2011
S AMPLING THREATS TO
25
VALIDITY

These four strategies offer different
opportunities for—and pose different
threats to—the exploration of the external
validity of any given set of findings with
respect to the facet in question.

Define external validity:

the deliberate and systematic
search, on a number of facets, for
both the scope and the limits over
which that given set of findings does
and does not hold.
RESEARCH METHODS
8/6/2011
R EPORTING
26

Validity as robustnesss

Where are the boundaries?

Can findings be replicated?
RESEARCH METHODS
8/6/2011
TESTS FOR VALIDITY
27
STAGE 1


STAGE 2
Design:
Instrument validity
instrument use validity
Comparison validity
execution validity
Hypothesis:
Experiment:
RESEARCH METHODS
Construct validity
Operational
Nomalogical validity
Predictive
8/6/2011
C HECKING VALIDITY
28

Observation:
State validity
Attribute
Pattern validity
Process
RESEARCH METHODS
8/6/2011
T HEORY
RESEARCH METHODS
29
8/6/2011
A THEORY DOES NOT COME FIRST IN
30
RESEARCH

Theory:


A reasoned and precise
speculation about the answer to a
research question, including a
statement about why the
proposed answer is correct
Implies several specific descriptive
or causal hypotheses

RESEARCH METHODS
Is consistent with prior evidence
about a research question
8/6/2011
C HOOSING
31
A THEORY

Possibly, it will be wrong

It must be falsifiable-what evidence would
convince us?

Requires observable implications (many)

Observe the principle of parsimony (a
judgment about the nature of the world,
which is assumed to be simple)

These rules assume you have not yet
collected the data, which can be used
afterwards, to modify your theory
RESEARCH METHODS
8/6/2011
A FTER
32

YOUR RESEARCH
What to do with a theory

Expand it, dropping a condition or variable, to,
say, all countries, all regions

Make it embrace a larger range of phenomena

Do not do the opposite
RESEARCH METHODS

If the theory does not work for some of your
observations, do not squeeze new theory out of a
revised and qualified research base

Ie, make it less restrictive, but not more
restrictive—unless you go back and collect more
data and observations beginning with the new
hypothesis
8/6/2011
D ATA
33

Ensure reliability

Maximize leverage


RESEARCH METHODS
Explain as much as possible with as little
as possible

Increase numbers of observable
implications with confirmation

Improve the theory, improve the data
From the beginning, list all the possible
implications of your hypotheses—
outcomes, responses, interviews, at all
levels
8/6/2011
S CEPTICISM
34

Report negative findings

Report alternative
hypotheses
RESEARCH METHODS
8/6/2011
T HEORY
C ONCEPT
RESEARCH METHODS
35
8/6/2011
C ONCEPT F ORMATION
36

RESEARCH METHODS
Rigorous approach to
concept formation
(Osigweh)

Defining meaning
boundaries (what the
concepts do not include)

Minimizing concept
misuse, confusion
8/6/2011
C ONCEPT
37

RESEARCH METHODS
PRECISION
General, yet precise

Moving a concept from low
to high levels of abstraction

Economic oncepts must
span several contexts

Yet they must provide a
precise guide to what is not
included
8/6/2011
C ONCEPT I MPRECISION
38

Useless: new contexts too
easily fit

New topics can be suspect,
vulnerable to herding by
scholars (CR impact on
performance)
RESEARCH METHODS
8/6/2011
C ONCEPTS
39

Can be decomposed
taxonomically
RESEARCH METHODS
8/6/2011
VARIABLES
40
A variable is a special
kind of concept.
It is a classification into two or more
mutually exclusive and totally
inclusive
categories (e.g., Hage, 1972; Smith,
1975).
RESEARCH METHODS
8/6/2011
C ONCEPTS
RESEARCH METHODS
41
8/6/2011
42
C ONCEPT TRAVELLING :
P OSITIVE CONTRIBUTION
Concept travelling, in this sense,
means that the concept is
precise enough to allow
researchers to define it in the
same way, and so to test it in a
wide range of situations—that
is, that the concept is a
universal.
RESEARCH METHODS
8/6/2011
E XAMPLE
43

Puzzle -- Each use creates essentially the same
image anywhere:

(a) a maze of activities that cannot be
correctly solved outside settings or mental
frameworks that prescribe following certain
paths or specific combination of
paths;(jigsaw?Rubik’s cube)?

(b) an exercise of figures, numbers, or
behaviours that cannot be engaged in or
successfully disengaged in, except through a
specific order or steps, procedures, or
systems of thought processes;

(c) activities that when completed cannot fail
to yield exact answers, known solutions, or
ultimates that could be exactly predicted
beforehand.
RESEARCH METHODS
8/6/2011
44
C ONCEPT S TRETCHING ( TO
BE AVOIDED )
Multi level governance
when refracted through the lens of
lived political experiences-- notions
of policy transfer and of the complex
politics of scale of interventions in
time, place and space--needed
RESEARCH METHODS
8/6/2011
TOO VAGUE ( STRETCHING )
45

Decentralization

Would be misapplied in
different societies

Too broad
RESEARCH METHODS
8/6/2011
TOO NARROW, AS WELL AS
VAGUE : OBSERVATIONAL
46
CONCEPTS

Can be moved up or down
the abstract ladder

Communication puzzle,
group, decision, problem,
conflict, and participation.
RESEARCH METHODS
8/6/2011
HAL
High abstract
level
• Traveling concept
domain
— Broad coverage, of
distinct classes of
things
High
•Stretched
concept
domain
— Broad
coverage
— Too many
classes
— Few, but determinate
of things lumped
together (with
little
attributes of
attention to the
distinct classes
precision of their
— Precise at a
many attributes)
— Packed or too
allembracing
universal level
(
Connotation (Depth, Intension)
I
Low
Middle
47
E
X
T
E
N
S
I
O
N
,
D
E
N
O
T
A
T
I B
O R
N E
A
D
T
H
)
RESEARCH METHODS
8/6/2011
Connotation (Depth, Intension)
48
IIIIIIIIIIII
(
Low
Middle
MAL
Mid abstract
level
• Generalizable
nonuniversal concept
domain
— Breadth balanced
— Medium range
concepts
— Similarities
High
E
X
T
E
N
S
I
O
N
,
D
E
N
O
T
A
T
I B
O R
N E
A
D
T
H
)
RESEARCH METHODS
8/6/2011
49
HAL
MAL
Connotation (Depth, Intension)
IIIIIIIIIIII
Low
Middle
High
RESEARCH METHODS
• Taxonomic domain
— Narrow
coverage
— Many
attributes
for each class
— Distinctive typologies
or precise taxonomies
E
X
T
E
N
S
I
O
N
,
B
R
E
A
D
T
H
)
•
LAL
Configurative
situational
Low abstract concept
level
doman
— Narrow
coverage
— Too few
attributes
—
Configurative/
Imprecise
taxonomies
specific
generalities
precise at a specific level
taxonomies
— Precise at a
specific level
8/6/2011
M OVING TO THE UPPER
50
LEFT CORNER

Move up the ladder by
negating

Concepts with negation are
precise

Those without are
indeterminant
RESEARCH METHODS
8/6/2011
E XAMPLE : STUDIES OF
“P ROBLEM ”
51

The concept of problem is studies
without its meaning boundaries
being distinctively delineated.

Generated by concrete, intuitive,
ad hoc conceptualizations.

Studies of structured problems
fall widely apart from a focus on
conflict resolution to one on
negotiation.
RESEARCH METHODS
8/6/2011
T HEORETICAL
52
CONCEPTS

Theoretical or universal concepts are
defined by their systemic meaning in the
sense that each meaning derives from the
part that the concept plays in the theory.

Open concepts reflect the availability of
different operational criteria for
application to different contexts.

As a result, their meanings are not fully
defined by reference to observable things
and their characteristics.
RESEARCH METHODS
8/6/2011
E XAMPLES
53

Efficacy (Jones, 1986;
Osigweh, 1983)

Synergy, feedback,
adaptation (Osigweh 1985b,
pp. 153- 157),

Homeostasis and
isomorphism
RESEARCH METHODS
8/6/2011
E MPIRICAL
54
CONCEPTS

Observational

Can be moved up or down
RESEARCH METHODS
8/6/2011
E XAMPLES
55

communication,

puzzle,

group,

decision,

conflict
RESEARCH METHODS
8/6/2011
M OVING
56

UP THE LADDER
Requires that you keep it
clear and not just extend
meaning to other kinds of
observations
RESEARCH METHODS
8/6/2011
A DVICE
57

Define a concept by saying
what it is not;

Give it precise boundaries

Make it clear
RESEARCH METHODS
8/6/2011
T HEORETICAL C ONTRIBUTION
RESEARCH METHODS
58
8/6/2011
C OMPONENTS
59

Which factors (variables, constructs,
concepts) logically should be
considered as part of the explanation

Two criteria exist for judging the
extent to which we have included the
"right" factors:
RESEARCH METHODS

comprehensiveness (i. e., are all
relevant factors included?) and

parsimony (i.e., should some factors
be deleted because they add little
additional value to our understanding?).
8/6/2011
R ELATIONSHIPS
60

Operationally this involves using
"arrows" to connect the "boxes."

Such a step adds order to the
conceptualization by explicitly
delineating patterns.

It typically introduces causality.
RESEARCH METHODS
8/6/2011
W HY
61
IMPORTANT ?

What justifies the selection of factors and the
proposed causal relationships?

This rationale constitutes the theory's
assumptions

It welds the model together.

Why should colleagues give credence to this
particular representation of the phenomena?

The answer lies in the logic underlying the
model.

The soundness of fundamental views of
economics or processes
RESEARCH METHODS
8/6/2011
F ROM
62

THE EDITOR :
“The mission of a theorydevelopment journal is to
challenge and extend existing
knowledge, not simply to
rewrite it. Therefore, authors
should push back the
boundaries of our knowledge
by providing compelling and
logical justifications for
altered views.”
RESEARCH METHODS
8/6/2011
S ET
63
CONTEXTUAL LIMITS

These temporal and contextual
factors set the boundaries of
generalizability, and as such
constitute the range of the theory.

Do theoretical effects vary over
time, either because other timedependent variables are
theoretically important or because
the theoretical effect is unstable
for some reason.
RESEARCH METHODS
8/6/2011
S UMMARY
64

For purposes of publishing:

Clarity in main ideas

Currency in quality
RESEARCH METHODS
8/6/2011
P UBLISHING IN E CONOMICS
65

Papers are assumed to vary along
two quality dimensions

The former is interpreted as the
importance of a paper’s main
ideas and

the latter as other aspects of
quality. Observed trends are
thought of as reflecting increases
in this
RESEARCH METHODS
8/6/2011
T HE END