12-Steps to a Research Design
Identify the resources available for the Study: time, money, expertise.
We have $______ for this study. We have ____ staff days. We have ____ partner days. We have
_____ participant’s days
Key Decision
CASE EXAMPLE1
STEP ONE: Determine the key research questions or hypotheses.
What do you need to know? What relationships are you interested
in investigating?
STEP TWO: Determine very clearly what your dependent (the
“effect” or impact you are interested in) and independent (the
proposed cause of the effect) variables (are). It always easier and
less costly to investigate a one-to-one relationship. However, it is
often the case that we want to know either how multiple causes lead
to a single effect, or to multiple effects. In this study, for example,
we are looking at two dependent variables and one independent
variable:
HISSA/MJT groups  economic security
HISSA/MJT groups  women’s empowerment
If you wish to investigate additional causal relationships, be
forewarned that each additional independent or dependent
variable will increase the load of “burden of rigorous proof”
exponentially. How does this translate in terms of your plans: if
you have the resources to actually assess the causal relationships
between HISSA/MJT groups on two different dependent variables
(effects, or impacts) among, say, 75,000 people…the addition of
one dependent or independent variable will mean that you probably
can only demonstrate such a relationship in around 20,000 people.
The addition of a second variable would take it down to around
5,000 people (for the same resources you have available to you).
STEP THREE: Identify crucial intervening or confounding
variables. These are variables that may intrude themselves between
your purported dependent and independent variables. Unless you
attempt to control for them – qualitatively or quantitatively – you
risk identifying false relationships.
STEP FOUR: Define and identify specific and measurable (can be
qualitative or quantitative) indicators for the dependent variable(s).
STEP FIVE: Determine data sources
1
“Women’s participation in village savings and
loans will have a positive impact on women’s
empowerment”
Dependent variables: women’s empowerment
Independent variable: Participation in a village
savings and loan program.
Urban vs. rural setting
Length of time a VSL has been in existence
Women’s marital status
Ethnicity
Empowerment is defined broadly as “the
expansion of assets and capabilities of poor
people to participate in, negotiate with,
influence, control, and hold accountable the
institutions that affect their lives”2 and “the
expansion in [women’s] ability to make
strategic life choices in a context where this
ability was previously denied to them.”3 (See
FY06 research framework for more specific
evidence categories and core indicators for
empowerment).

Participating Women

Husbands or male relatives of
participating women

Village headmen

CARE staff

Secondary studies
This column shows a hypothetical example. It is intended to give a very concrete picture of the products from each step. However,
it is important to keep in mind that the example is one in which mixed methods play an important research role, and in which
quantitative data from a survey are needed to answer the research question. Purely qualitative research designs may involve a number
of questions and design considerations that do not get covered in this example.
2
Deepa Narayan ed, Empowerment and Poverty Reduction: A Sourcebook. Washington, DC: World Bank, 2001.
3
Naila Kabeer, “The Conditions and Consequences of Choice: Reflections on the Measurement of Women’s Empowerment.”
UNRISD Discussion Paper 108. United Nations Research Institute for Social Development, Geneva, 1999.



STEP SIX: Determine the methods that you need in order to gather
the information and data required and meet levels of rigor that will
satisfy the intended audience(s) of the research. PRINCIPLE:
adopt methods that are as complex as needed but simple as possible.
1.
2.
3.
4.
5.
Government statistical records
(judiciary, participation in elections,
pro-women policies)
Local gender experts
Nonparticipating men/women
(comparison group)
Quantitative survey that allows us to
make statistically valid comparisons
between participants and nonparticipants on the dependent variable
(women’s empowerment).
Secondary data review (results of
similar studies in similar contexts)
Key informant interviews (on
sociocultural and gender context)
Semi-structured interviews with
women and men (qualitative and
participatory numbers4)
Focus group discussions (qualitative
and participatory numbers)
Think about your resources: Do you sense that they allow you to take on all four of these methods?
STEP SEVEN: Determine the overall research design strategy.
Longitudinal (data will be collected at least twice over some period
of time)? Cross-sectional (a single point in time)? To a very large
extent, this decision is determined by your actual research questions
from step one. It also can be influenced by the resources you have
available. It can also be influenced by a longer term evaluation
strategy that you have in mind.
STEP EIGHT: Determine the appropriate sampling population.
Who or what is the largest population that you wish to be able to
describe and/or account for in relation to your hypothesis?
Frequently, this decision is made not by the actual limits or
extent of a particular program but by the resources you have
available. The choice of the largest population that you wish to be
able to speak authoritatively about vis-à-vis impact is frequently
strategic and purposive. The key is to be very clear with ourselves
and our stakeholders about who or what we’re leaving out, and
why…and what population our research actually represents in terms
of its findings.
STEP NINE: Identify the critical sub-levels of analysis, from
largest to smallest.
Cross Sectional
9057 women and their households in 360
CARE village and savings and loans in one
project in North Region.
Largest: CARE project
Next: 360 VSLs
Next: 9057 Households
Smallest Analytical Unit: Individual women
Think about your resources: Do you sense that they allow you to take on all of this? Example:
imagine, right now, that you know you have a 2-hour long household questionnaire that you hoped
could be used in a random sample of the households. For it to be statistically valid, you would have
to randomly sample around 500 of those households. Is that even possible, given your resource
constraints? If not…
STEP TEN: Select a sampling strategy for every level
identified in step nine. There are basically two broad
kinds of sample: probability samples and non-probability
samples. Probability samples, also known as random
samples, allow every analytical unit to have an equal
chance of being selected. They allow you to generalize to
a larger population. They also are best for avoiding
researcher bias. Nonprobability samples, also known as
purposive samples, cannot, on their own, allow you to
generalize to a wider group. They are more subject to
researcher bias although this can be minimized through
4
1. LARGEST: One CARE Project
2.
Next Largest: 360 VSLs. Our resources do not
allow us to do rigorous impact research in all of them.
We’ll do a stratified random sample based on the
intervening variables of a) length of time in program
and b) urban/rural setting. We’ll classify all VSLs in
the total population then randomly choose:
-2 VSLs, urban, less than one year in program
-2 VSLs, urban, more than three years in program
-2 VSLs, rural, less than one year in program
-2 VSLs, rural, more than three years in program
See “Participation and Numbers,” pla notes 47 (August 2003): 6-12.
establishing strict, objective criteria for choosing data
sources.
Organize one female and one male (i.e., spouse or
HH head) focus group discussion in each of the
eight.
You can mix probability and nonprobability sampling
3.
Next largest: household. Among 8 VSLs – 240
strategies across methods, it needs to be emphasized.
women – randomly select the largest possible number
What remains very important, with nonprobability
that we can handle, given our budget and human
sampling (almost always used, for example, for
resources, for the quantitative survey. The random
identifying informants for in-depth interviews, focus
selection of the women results in random selection,
group discussions, key informant interviews, etc.) is that
too, of their spouses or other male household head
you are clear and transparent about your selection criteria
(if applicable). So…the resource question needs to be
and that you follow those criteria to the letter. Being clear
thought of in light of the pair of interviews. For the
about your sampling strategy for qualitative methods is
sake of this case example, imagine we’ve decided we
one very important form of impact research rigor
can handle 80 total quantitative surveys. We select,
frequently missing from impact assessments.
therefore, 5 women (and therefore their spouses or
other HH head, if applicable), at random, from each of
the eight VSLs.
4.
Next largest: individual. Among the 8 VSLs –
240 women – Randomly select two per group for indepth, individual interviews. Avoid selecting the
same women who participated in the quantitative
survey. This random selection of women results in
your random sample of men for in-depth interviews.
STEP ELEVEN: Select a comparison group. Identify at
least one comparison group – sometimes called a control
group – that can represent the “counterfactual”, in other
words, a group that has similar characteristics, contexts,
and cultures as the main analytical units from STEP TEN.
NOTE: formal comparison groups are frequently omitted
from development impact research and the
“counterfactual” is treated in a more qualitative manner,
triangulating between data sources, using secondary data,
etc. In the SII, we wish to include a comparison group if
financially possible.
In one of the urban and one of the rural villages, identify
all HHs that are similar to participating HHs with regard
to the intervening variables of marital status and ethnicity
(purposive sample). Then, within that, randomly select
10 nonparticipating households for the quantitative
questionnaire. Organize one FGD of nonparticipating
women in each of the 8 VSL sites. Organize one FGD of
nonparticipating spouses (why these numbers? Resource
constraints. It’s the most we can do in the time that we
have).
NOW: Calculate the actual costs of the research you’ve just described. Do you have the
resources? Do you need to change any decisions? Frequently, at this stage, you’ll find
you need to circle back as far as your research questions and make them more focused.
Or, you might find that while you hoped to do impact research in a given project across
35 villages, you need to focus on a much smaller subset of those villages. Or, you might
find that you need to reduce the scope of one or another particular method: for example,
you might still go ahead with a stratified random sample HH questionnaire in all 35
villages, but only choose two for in-depth, qualitative methods.
TOTAL EFFORT FOR ABOVE STUDY
A 1-hour long quantitative survey with 40 female participants = 40 hours (5 person days)
A 1 hour long quantitative survey with 40 husbands or elder males in women’s households = 40
hours (5 person days)
3. A 1 hour long quantitative survey with 20 comparison group women = 20 hours (2.5 person days)
4. A 1 hour long quantitative survey with 20 comparison group husbands/sr. males = 20 hours (2.5
person days)
5. In-depth interviews with 8 village headmen = three person days
6. Key informant interviews re., context = 3 person days
7. 8 focus group discussions with female participants = 3 person days
8. 8 focus group discussion with spouses of female participants = 3 person days
9. 2 focus group discussions with non-participating females = 1 person day
10. 2 focus group discussions with non-participating males = 1 person day
11. 16 in-depth interviews with female participations = 8 person days
12. 16 in-depth interviews with spouses of female participants = 8 person days
1.
2.
TOTAL PERSON DAYS NEEDED FOR DATA COLLECTION: 45
BUFFER DAYS (to account for travel, logistical snags, etc.): 9
TOTAL PLANNED PERSON DAYS FOR THIS RESEARCH: 54
Size of research team: 5
# of days of field research needed FOR DATA COLLECTION: 11
Cost/day for data collection: $1200
TOTAL COST FOR DATA COLLECTION: $13,200
STOP: DO YOU HAVE ADEQUATE RESOURCES? IF NOT: YOU’LL NEED TO RETURN TO
STEP ONE AND SCALE BACK YOUR AMBITIONS.
STEP TWELVE: Apply the appropriate methods at 1. Secondary research on gender, power, politicalthe appropriate levels from step ten
economic, and social context.
2. 8 VSLs and their villages: sociocultural context,
secondary data, interviews with 8 customary leaders
and eight key informants
3. In 8 VSLs, random sample of 40 women and
their spouses/male HH heads: quantitative
socioeconomic survey; related survey for husbands
(if appropriate) = 80 HH surveys
4. In 8 VSLs, one focus group of VSL
participants and one of their spouses = 16.
5. In 8 VSLs, individual interviews with two
VSL participants = 16 in-depth interviews
6. In two of the VSL villages: 10 female and
10 male survey questionnaires as a
comparison group = 40 surveys
7. In 8 VSL villages: one FGDs with
nonparticipating females = 8 FGDs
8. In 8 VSL villages: one FGD with
nonparticipating spouses = 8 FGDs