Naomi Jehlicka
Raw Wiki Notes for Week 6
Theme: Meta Issues in Research Design/Research Design
AGENDA Part 1: Meta Issues in Research Design (Prof Tedds)
o
o
o
o
o
o
o
o
I. Purpose of research design
II. Function of research design
III. Purpose of research
IV. Causality
 Very tricky and most poorly one in academic research
V. Unit of analysis
 Identify who are what you are studying
VI. Time
VII. Boundaries
VIII. Sampling
 Two types: non-probability and probability (more detail about nonprobability sampling)
 Probability sampling is much more important in positivist research
 You’ll get more info about probability sampling in 502 B
 Most 598s use non-probability samplings, same with scoping review, so
we will primarily focus on this method
AGENDA Part 2: Research Design (Prof Brady)
o I. Case Study Design
o II. Comparative Method
o III. Summary
1
Naomi Jehlicka
Raw Wiki Notes for Week 6
Theme: Meta Issues in Research Design/Research Design
Part One: Meta Issues in Research Design:
Meta Issues in Research Design are issues you need to consider in order to focus in on the
research question and adequately set out the tools/methodology you need to achieve your
research goals.
I.
Purpose of Research Design:
We were first introduced to research design in week 2 via Burnham et al et al. which discussed
the logical structure of inquiry. We need to consider which design will provide the best
evidence subject to our research constraints. The largest constraints are time and money.
Considerations:
What evidence is needed to answer your posed questions?
 Tax Evasion example: Different questions call for different methods.
o How many people are engaging in tax evasion?
o Why do people engage in tax evasion? (What is their motivation? You would go
out and explore, talk with people, gather subjective impressions. There is no
data set, the literature is rather unclear, so you’re looking at exploring why
people are doing this).
o Taxes cause tax evasion (according to literature) – does GST cause an increase in
tax evasion? (In here you are trying to explain a relationship, i.e. causality so you
need to use a methodology that will allow you to determine causality. Here you
would conduct a “natural experiment”)
Careful formation of research question and goals will allow you to design a project that will
allow you to answer the question you are trying to pose. Note that it is very easy to stray and
answer a completely different question. This is a common problem in 598.
Remedy: Do not be afraid to rewrite the research question to better reflect your research. “it
is easier to rewrite the question than redo the research” – Prof Tedds.
2
Naomi Jehlicka
Raw Wiki Notes for Week 6
Theme: Meta Issues in Research Design/Research Design
II.
Function of Research Design:
There are two functions of research design:
1. To develop or conceptualize an operational plan
2. To ensure the procedures adopted within the plan are adequate to provide valid,
objective, and accurate solutions.
Considerations:
Ask yourself: “Given this research question, what kind of evidence is needed to answer the
evidence in a convincing way?” This is also known as the evidence based policy mind frame.
 Building Analogy (Yin 1999)
o Before a builder or architect develops a work plan or blueprint, he or she needs
to first determine what type of structure are they building, what are the needs of
the occupant, which material is needed or accessible, etc. These are analogous
to the META ISSUES of a research project.
o Once the builder/architect has an idea of what is require, what the building will
be used for, and for what purpose, the actual work plan can then be constructed.
Meta issues/questions collectively form the GLUE that holds everything together.
III.
Purpose of Research:
Three purposes of research:
1. Exploration
2. Description
3. Explanation
Exploration Research:




Exploratory research projects generally take place during the preliminary stages of
policy and program development.
They are neither comprehensive nor exhaustive as their purpose is to familiarize
yourself and your client with the topic, issues, and problems of a given situation, and
perhaps identify preliminary solutions.
In short, exploratory research projects gather information that will further define any
further research.
Ex. Our scoping reviews
3
Naomi Jehlicka
Raw Wiki Notes for Week 6
Theme: Meta Issues in Research Design/Research Design
Exploratory research projects helps identify which research design is best suited, which data
collection methods you need to employ. In short, exploration provides the foundation for
future descriptive and explanatory research.
LIMITATION: while exploratory research provides insight into a subject matter, you cannot use
the results as basis for decision making. Exploratory research is not generalizable (i.e. when you
take something specific and expand it to include situations that are similar). As such,
recommendations are not allowed.
Descriptive Research:



Descriptive research project are sensory in that they attempt to describe (by saying or
by writing) how something looks, smells, sounds, feels, or tastes like.
Common Questions: Who, what, when, how, how often, how many?
Some look down on this form (“it’s easy”) but it is very important as it adds to our
knowledge in measurable ways.
o In order to explain something, you have to describe it first.
o Ex: Canadian Census an excellent example, absolutely valuable in policy
development
LIMITATION: You cannot determine causality from descriptive research
Explanatory Research:

Answers the “Why” question – the most important question clients generally want answers
too.
o You do not want to go down the “Why path” until the other types of research are
done.
o EX: Crime story: crime was rising in 1960s and there was social hysteria that civilization is
“moments away from bloodbaths in the streets” (1993). Hot research topic was what
causes crime.
o In the mid 1990s there was a sudden sharp reversal of crime trend.
o However, we still do not know what had caused the sharp, sudden reversal of the
trend. We still do not know the causality.
4
Naomi Jehlicka
Raw Wiki Notes for Week 6
Theme: Meta Issues in Research Design/Research Design
IV.
Causality:
Causality is tied to explanation research. X (dependent variable) causes Y (independent
variable)

Correlation vs. Causation:
o Statistics measure correlation
o Causation is dependent on correlation, but correlation does not necessarily mean
causation.
o Ex. Ice cream sales and shootings: ice cream sales are high, shootings are high  ice
cream sales cause shootings
 This is wrong. Though IC Sales and Shootings are correlated, they share a
common denominator: heat.

Intervening, moderating, antecedent variables:
o Intervening variables could produce an apparent relationship between two
variables such that if the intervening variable was absent, there would be no
relationship.
 Ex TV causes violence, obesity, tooth rot etc
 TV doesn’t cause anything; another variable is intervening.
o Antecedent variables could cause both of the other variables to change so if it
were absent, the two variables may not correlate at all.
o Moderating variables are those that explain a relationship between two
variables (“heat” in the case of ice cream/shootings)
 You might think x causes y, but z causes x which causes y, so it is really z
you are interested in.
Conditions that Conclude Causation
 Co-variation
 Temporal asymmetry
o Behaviour has to have a chance to be modified, so
they cannot happen at the same time.
 No plausible rival hypothesis
5
Naomi Jehlicka
Raw Wiki Notes for Week 6
Theme: Meta Issues in Research Design/Research Design
Group Activity: read handout and discuss: “Did privatization of liquor stores in BC cause
increased alcohol consumption” via co-variation, temporal asymmetry, rival hypothsis



Covariation present? yes
o Are they moving in the same direction? yes
Temporal asymmetry? questionable
o Consumption started increasing prior to legislation in 2000
o Did increase sales cause the privatization of liquor stores?
o Change in government (the Liberals came into power)
Rival hypothesis? Yes, several
o Legal age of drinking? Population increase (BC recovering from Asian flu so
people are returning to the province)
o Economic downturn, terrorism, Y2K?
o Alcohol is a normal good so income increased
 (as per income graph) we observe that there is temporal asymmetry with
income. Alberta privatized in 1993 and also shows that after incomes
increased, liquor stores were privatized. This could be an alternative
explanation.
You do not have to prove a definite rival hypothesis; you just need to explain if there is
another possible hypothesis.
Take away message: it’s very difficult to discern causality. It is by no means a simple task. Lots
of very intelligent people get it wrong all the time. The trick is to employ the tools gained in
this class to minimize the chances of incorrectly establishing causality.
As public sector researchers, you should be very concerned about why and how causality is
very difficult to discern. Not all research designs can be used to explain things. There is a group
of researchers who will tell you that randomized controlled trials (RCT) is the gold standard but
this is not always the case.
V.



Unit of Analysis:
What or whom you are studying
Ex. Individuals, groups, families, children, pregnant women, organizations, processes,
programs, policies, legislation, tools, literature, speeches, graphic images, etc.
Be sure to define correct unit of analysis
6
Naomi Jehlicka
Raw Wiki Notes for Week 6
Theme: Meta Issues in Research Design/Research Design


VI.
o Marriages or married couples
o Crimes or criminals
o Corporations or managers or shareholders
Make sure you do not make assertions based about one unit based on analysis of
another
Ecological fallacy
o When your inferences about individuals are based on information learned from
the group.
o Lawrence Summers  delivered a controversial diversifying engineering
workforce speech
 Women not in tenured positions in sciences/engineering (mostly men –
women hold maybe 6% of all positions)
 According to Summers, the pattern isn’t due to discrimination but
because of innate differences between the two sexes.
 However, Summers based his observation on research conducted in high
school students. Male students perform higher than women (wider
spread), so they’re the ones smart enough to get tenured positioned.
They lack that star quality.
 Ecological fallacy committed results taken from high school
students!!
Time:
There are two dominant time-related options in research design. You can either make
observations at one point in time or over a period of time. When deciding whether to collect
data at one period or several periods of time, much depends on a) your question, b) resources,
and c) reason for collecting data.
Cross-sectional studies: observation at one point in time
Longitudinal studies: observation at several point in time.
 Two types: time series vs. repeated measures.
 Cohort vs Panel
Although longitudinal studies tend to produce better data than cross sectional projects, there is
nothing wrong with cross sectional data as much depends on the question you are asking.
7
Naomi Jehlicka
Raw Wiki Notes for Week 6
Theme: Meta Issues in Research Design/Research Design
VII. Boundaries:
Several types of boundaries exist. Again, much depends on resources available and your
research questions.
Geographic boundaries
 Complex, street, neighbourhood
 Voting district, city, province
 Country, continent
 East of the Rockies; North of the Great Lakes
Economic boundaries
 Income, wealth, consumption,
 Poverty, lICO
 LDCs
Governance boundaries
 Parliamentary system, dictatorship
Other
 Individual wth phones, cars, drivers licenses, over 15, full time employees
 Again, much depends on your question and resources
VIII. Sampling:
A Sample is a specimen taken from testing or analysis (in our case, generally analysis).
Sampling refers to how you go about getting your unit of analysis. Sampling techniques
establish what or whom within your unit of analysis you will study and how you will obtain
observations. This can be a tedious and difficult process.
Why do we sample?
 Costs associated with obtaining population are often high
 Size of population can be difficult to work with
 Convenience and accessibility of elements make sampling a better choice than
population study
8
Naomi Jehlicka
Raw Wiki Notes for Week 6
Theme: Meta Issues in Research Design/Research Design
Two sampling strategies:
 Non probability sampling, where numbers are selected not according to the logic of
probability,
o Convenience (choosing units that are accessible)
o Purposive or reputation (choosing units based on your knowledge of the
population, finding programs with reputation of being successful).
o Snowball: (picking up units of analysis as you go, “roll down the hill” like a
snowball)
o Quota: (for ex, you want your sample to be representative of a population os
you set a quota at, say, 52% must be women. However, after establishing this
quota, you do not have any guidance on how to select the group members.
o Theoretical: iterative process – sample a few, learn a bit, remodel your theory,
sample more (sort of like the hermeneutics methods)
o Volunteer: your sample constitutes of volunteers (ex. First year psych students)

Probability sampling occurs when each member of the population has certain
probability of being selected. How likely is it that you’ll know your total population?
Tough. Here are various selection techniques.
o Simple random: where every possible unit of observation in the population has
an equal likelihood of being chosen. For example, Stats Canada data is often
used for random digit dialling.
o Systematic random: (refer to slide 21 for a step-by-step guide on how to apply
systematic sampling)
o Stratified – divide population into two or more subgroups according to common
characteristics. Then, a simple random sample is selected from each sub groups.
Samples from subgroups are combined into one.
o Cluster – divide population into several representative clusters and then select a
simple random sample of clusters (ex – 502A has been divided into two sections,
selecting one for a study would be cluster sampling)
Sampling considerations:
 What constraints
 Can you exploit a priori info?
 Do you know or have access to the population
 Does the sample need to be representative? Why?
 Do your results need to be generalizable?
9
Naomi Jehlicka
Raw Wiki Notes for Week 6
Theme: Meta Issues in Research Design/Research Design

Do you need to determine cause and effect?
Summary:
 Design is essential in providing an answer to your question
 Carefully consider the purpose of your research
 Identify your unit of analysis
 Establish any boundaries
 Decide on your sampling method.
Part 2: Research Design:
Because of time constraints, we will only examine Case Study and Comparative research
designs. Historical research design will be discussed next week.
I.
Case Study Design:
A case study design is based on the idea that there is a discrete case where boundaries can be
drawn. There is always something unique about the case you choose to study. Sampling is
based on the particular theory you would like to test. Theoretical sampling occurs in lieu of
random sample, as it is not the best practise to have a random sample in case studies because
you are examining a specific phenomenon.
Are case studies generalizable?
 Barzelay argues that they CAN produce empirical generalizations (non-statistical
generalizations)
o Scenarios : possible connections
o Problem-solution pairs: exemplary solutions to reoccurring problems
o Role frame : a way that you understand a particular problem by framing it
o Methodological: ways of doing public administration case studies
10
Naomi Jehlicka
Raw Wiki Notes for Week 6
Theme: Meta Issues in Research Design/Research Design
II.
Comparative method:
Introduction
The comparative method entails the selection of two or more jurisdictions in order to
understand the nature of, and reasons for, similarities and differences. There are two main
approaches:


Comparison used to approximate experimental condition (more positivist)
Used to facilitate the exploration of differences and extreme cases (hermeneutics)
Comparative research methods are especially important in the context of globalization.
Why compare?






To determine what makes our system distinct
To appreciate the similarities between jurisdiction
Policy Alternative awareness
Understand consequences of different political arrangements
Determine impact of global trends on specific jurisdictions and the challenges of global
governance.
“What do they know of England, those who only England know?”
Comparison for explanation:
This mode is similar to experimental designs based on the assumption that there are common
laws of causation across jurisdictions. This approach tends to be universalist, quantitative, and
interested in linear relationships.
An example is the “Logic of industrialization” thesis where Wilensky argues that you can line
States up on bases of industrialism and examine the size of their welfare state (positively
correlated).
Sampling techniques used in comparative for explanation
 “Most similar sampling” entails that jurisdictions as similar as possible on “intervening
variables but differ on independent variable
11
Naomi Jehlicka
Raw Wiki Notes for Week 6
Theme: Meta Issues in Research Design/Research Design
o Dependent Variables (y): phenomena that the research wants to explain
o Independent Variables (x) – the things we think may influence the dependent
variable
o “Intervening” Variables (x): everything else, but these are the ones that we might
not be interested in.
 Prof B would call them the ‘control variables”
Sampling problems:
 There is no clear method for sample selection – unsure of who should be included and how
many people should be sampled since we do not run significance tests
 For cross country studies, you can never find two countries that are the same.
 Over determination can occur in small N problems.
 Galton’s problem: when selected cases are not independent
o Ex. Canada, NZ, Aus  all former colonies and all have west ministerial system of
government. Problem because they are all formal colonies of England - what if they
are colonies of another empire, like Holland for example?
Comparison as a mode of exploration:
This method is interested in anomalous or critical cases, puzzles and paradoxes. It picks an
unusual case and attempts to explain why things are the way they are. Extreme policy cases
are favourites. You purposely pick cases that are NOT representative of the status quo.
o The focus is on qualitative differences
o Case selection guided by theoretical problem
o General specific hypotheses for testing
Challenges:
 Do attitudes/concepts/institutions travel?
o More of a problem for quantitative analysis that is trying to develop explanations
o Problem here is that concepts and attitudes may not travel
 Data inconsistencies
 Data driven analysis (this reflects the political priorities of official data production)
 Parsimony vs. richness in analysis.
 Case selection methods – no set guidelines, sample is not really all that random.
 Tendencies to exaggerate differences in between samples
12
Naomi Jehlicka
Raw Wiki Notes for Week 6
Theme: Meta Issues in Research Design/Research Design
Comparative summary:




Comparative research never replicates experimental conditions completely
Not all comparative research assumes universal causal patterns or seeks to discover linear
causation – indeed, this is not the only point of research.
Comparative research offers a rich source of testable hypotheses
Increasingly, comparative designs have grappled with impact of cross-national and global
pressures – governance systems like EU, NAFTA
III.



Lecture Summary:
Research design must be considered together with underlying epistemology, as
epistemology guides the specifics of the research design in each case
Positivist case studes, comparative studies, and historical studies seek to establish
generalization and see facts as neutral
Hermeneutic case studies, comparative studies attempt to paint an empathetic picture of
events.
Next week:


Experimental, mixed, historical methods
Laurie returns to discuss outlines.
13