Introduction to Using Qualitative
Methods in Impact Evaluation
Ryan Sheely
Kennedy School of Government
Harvard University
DIME Workshop, Dubai
June 1, 2010
Multiple Reasons for Using
Participatory Methods


This morning’s session emphasized social
justice and accountability as reasons to
utilize participatory methods in impact
evaluation
There are are also methodological
justifications for integrating participatory
assessment and other qualitative measures
with randomized impact evaluation
What Are Qualitative Methods?
Qualitative methods are a diverse set of research tools
that are primarily concerned with the meaning of
social actions, statements, and documents
 Both alone and in combination with randomized
evaluation and sample surveys, these methods
provide leverage with respect to:

 Describing and categorizing social contexts
 Interpreting the social behavior, actions, and outcomes
 Tracing the specific processes and mechanisms
whereby programs lead to effects
The Diversity of Qualitative
Methods

Participatory and Action-Based Research

Interviewing and Discussion

Content Analysis of Documents and Records
The Diversity of Qualitative
Methods

Participatory and Action-Based Research
 Systematic Social Observation
 Drama and Role Playing
 Maps, Models, and Diagrams
 Ranking and Scoring

Interviewing and Discussion

Content Analysis of Documents and Records
The Diversity of Qualitative
Methods

Participatory and Action-Based Research

Interviewing and Discussion
 In-Depth, Semi-Structured Interviews
 Focus Group Discussions
 Oral Histories of Individuals, Communities, or Projects

Content Analysis of Documents and Records
The Diversity of Qualitative
Methods

Participatory and Action-Based Research

Interviewing and Discussion

Content Analysis of Documents and Records
 Diaries or Personal Time/Financial Records
 Archives and Government Records
 Newspapers and Civil Society Reports
Analyzing Qualitative Data:
The Role of Narrative


Despite the differences between these
methods of collecting qualitative data, the
common principle is using close observation to
understand the meaning of what people say
and do
As a result, the writing of field notes and
systematic the aggregation of notes into
narrative case studies are a central skill in
analyzing qualitative data
Combining Qualitative and
Quantitative Methods


Qualitative methods can be linked to randomized evaluation
Before Implementation:
 Use interviews, participation, and observation to design contextually
relevant surveys and behavioral activities
 Use observation or participation to design interventions that can test
the interaction of different program components

After Implementation:
 Aggregating and coding the content of interviews, behavioral
observation, and newspapers to cross-check survey data
 Analyzing process to understand the lack of statistically significant
differences
▪ Implementation failure vs. design failure?
Homework: Qualitative Methods
and Your Projects
When you meet with your clinic groups this afternoon,
consider this question:
What types of qualitative methods would add value to
the evaluation of your project?
Focus on three things:
1) Identify each question you want to answer
2) Choose a set of qualitative methods to integrate into
your evaluation design for each question and
how/when you would use each method
3) Why these methods add value to your research design
Qualitative Methods:
Methodological Limitations

Small, Nonrepresentative samples

Researcher/Facilitator Bias

Possibility of Contamination of Control Groups when
combining with Randomized Evaluation

Lack of standards for comparing qualitative and
quantitative findings
Qualitative Methods:
Practical Limitations

Time and Cost of Qualitative Research
 Understanding and Documenting Context
Requires Time to Develop Mutual Trust Between
Researcher and Community
 Increased Staffing and Training due to more
diverse skill sets

Coordination and Management Problems
Associated With Larger Teams
Conclusions: Mixing Qualitative
and Quantitative Methods
Qualitative Methods are a diverse set of tools and skills
that require training, practice, and careful research
design
 Qualitative methods can help to better understand
why a given program did or did not have an effect

 Helping to develop better treatments and outcome measures
for randomized evaluations
 Using local knowledge and cultural context to interpret the
meaning of social behavior and process of intervention

Ideal mix of qualitative and quantiative methods
depends on the questions you want to answer and the
local context
REFERENCES AND SUGGESTIONS FOR FURTHER READING
Bamberger, Michael, Vijayendra Rao and Michael Woolcock (2010) “Using Mixed Methods in Monitoring and Evaluation: Experiences
from International Development”, in Abbas Tashakkori and Charles Teddlie (eds.) Handbook of Mixed Methods (2nd revised edition)
Thousand Oaks, CA: Sage Publications
Barron, Patrick, Rachael Diprose and Michael Woolcock (2010) Contesting Development: Participatory Projects and Local Conflict
Dynamics in Indonesia New Haven: Yale University Press (in press)
Chambers, Robert. 1997. Whose Reality Counts: Putting the First Last, ITDG Publishing.
Cronk, Lee. 2007. “The influence of cultural framing on play in the trust game: a Maasai example” Evolution and Human Behavior.
Kanbur, Ravi, ed . 2001. Q-Squared? Qualitative and Quantitative Methods of Poverty Appraisal
Malan, N. 2000. “On the Relationship between participatory research and participatory development.” Africanus 2: 64-75.
Ravallion, Martin. 2001. “How Can Qualitative Methods Help in Measuring Poverty,” Qual-Quant: Qualitative and Quantitative Poverty
Appraisal- Complementarities, Tensions and the Way Forward, Permanent Black Publishers, 2002.
Rodrik, Dani. 2008. “The New Development Economics: We Shall Experiment, but How Shall we Learn?” Paper Prepared for the
Brookings Development Conference, May 29-30 2008.
Sheely, Ryan. 2010. “Community Governance, Collective Action, and the Maintenance of Local Public Goods: Qualitative and
Experimental Evidence from Rural Kenya.” Better Governance for Better Health Conference, Stanford University. April 26-27,
2010.
Van der Riet, Mary. 2008. “Participatory Research ant he Philosophy of Social Science.” Qualitative Inquiry 14 (4): 546-565.
White, Howard. 2002. “Combining Quantitative and Qualitative Approaches in Poverty Analysis,” World Development.
Woolcock, Michael (2009) ‘Toward a Plurality of Methods in Project Evaluation: A Contextualized Approach to Understanding Impact
Trajectories and Efficacy’ Journal of Development Effectiveness 1(1): 1-14