Theory Driven Evaluation:
tracing links between
assumptions and effects
Sixth European Conference on Evaluation of Cohesion Policy
Warsaw, 30 November – 1 December 2009
Karol Olejniczak, k.olejniczak@uw.edu.pl
EUROREG – University of Warsaw: www.euroreg.uw.edu.pl
Theory-Driven Evaluation in a
nutshell
• It relates evaluation research with the scholarly
socio-economic theories
• It treats programme as optimistic assumption about
causal relations: If we DO…. than we GET… and than…
• Programme is a set of theories…
– Underlying Theories (knowledge, experience, influences)
– Theory of Change (assumption about strategic change)
– Theory of Implementation (the way every-day work is organized)
• ...that works in a certain context & circumstances
• TDE is and approach, it is method-neutral
– Logic models
– 5 stages procedure
5 stages procedure
STAGE 1
Understanding
the context
STAGE 2
Defining
Theory of
Change and
tracing its
Underlying
Theories
STAGE 3
Reconstructing
detailed Theory
of Change and
its indicators
STAGE 4
Tracing the
real change outputs and
effects
STAGE 5
Explaining the
obtained
effects
Theory Driven Evaluation in
Practice
Case Study of Ex post Evaluation Neighbourhood Program
INTERREG/TACIS CBC Poland-Belarus-Ukraine 2004-06
Programme
• Border regions from Poland, Belarus, Ukraine
• Two programmes – two pots of money: INTERREG 37,8 mln EUR vs
TACIS 8 mln EUR
• Beneficiaries – local communities, local services, NGO
• 173 major projects + 300 micro-projects
Contracting Authority:
• Polish Ministry of
Regional Development,
Territorial Cooperation
Unit
Evaluator:
• EGO – Evaluation for
Government
Organizations s.c.
Contract:
• 5 monhts (July-Nov 09)
Stage 1: Understanding the context
Questions:
Answers:
• What theories tell us about
cross-border cooperation?
• CBC depends on permeability of the
border & partners complementarities
• What type of border it is?
• Highly peripheral, underdeveloped
area, border as barrier
• What was the scale of
intervention?
• Minor financial impulse, could be
visible only on a local scale
Methods:
• Literature review, review of earlier empirical research
• Analysis of general statistics
• Review of socio-political situation
Stage 2: Defining Theory of Change &
tracing its Underlying Theories
Questions:
Answers:
• What strategic objectives have
been set?
• Dual objective: IF we act on 5
thematic fields THAN quality of life
AND socio-economic integration will
be improved in a border area
• What were the assumptions and
inspirations for these decisions?
• Reasonable fields of intervention but
no measures for border permeability
• Stakeholders choices, intuition,
earlier experiences of PolandGermany CBC Programme
Methods:
• Desk research (programme documents)
• Interviews with key programme stakeholders involved in the
programme design
Stage 3: Reconstructing detailed Theory
of Change and its indicators
Questions:
Answers:
• How should we define the
change in terms of indicators &
assessment criteria?
• Effects both planned and sideeffects have to be measured on 2
scales: quality of life & integration
• How programme designers
defined these changes in terms
of programme & projects
indicators?
• They have to be traced in 3
dimensions: thematic (projects
topics), relational (partnerships) and
territorial (local communities)
Methods:
• Analysis & assessment criteria base
on logic models, they differ in details
but scale stays the same
• Logic models for each thematic group and partnerships
• Assessment system – two scales: quality of life vs integration
• Review of the programme indicators
Stage 4: Tracing real changes – outputs
& effects
Questions:
Answers:
• What was the funds spatial &
thematic distribution?
• What was the number &
structure of partnerships?
• What were the effects of
thematic group of projects?
• What were the effects of
partnerships?
• What were the impact on local
communities?
1. High disproportion between 3 sides
of the border
2. Focus on improving quality of life
3. Local and close-to-border effects
4. The main integration effect was
brought by soft-projects, it was
institutional, limited integration of
local societies
5. Balanced effects (quality +
integration) brought by tourism &
border security projects
Methods:
• Local visits of all project sites
(different observation tools applied, depending project’s topic)
• Survey of project beneficiaries, survey of partnerships
• Social survey of twin communities
• Interview with local stakeholders, review of local press
Stage 5: Explaining the obtained effects
Questions:
1. Why disproportion?
2. Why focus on quality of life?
3. Why close-to-border and
local effects?
4. Why institutional
integration?
5. Why tourism & border
security projects had best
effects?
Methods:
•
•
•
•
Answers:
1. Unbalanced money & procedures;
higher experience of Polish teams
2. High peripheriality & local needs;
unclear demarcation line
3. Border as a sealed barrier, small funds
spread spatially
4. Selection criteria, limited trust – focus
on smaller projects, expert-type
projects, micro-projects as top-down not
bottom up initiatives
5. Tourism easy to combine with next
initiatives (multiplier effects), security
required official cooperation agreements
Brainstorming with experts
Second review of qualitative & quantitative data again
Interviews with programme managers
Survey of unsuccessful and potential applicants
Summing up the case study
Main message:
• Close to border and local effects.
• In a given context every joint project was a success
How TDE helped us?
• Dealing with complexity - packing & unpacking
issues
• Focusing exploration on the right level
• Making the fair judgement – understanding
contextual limitations
• Writing the concise report - clear narrative
TDE for cohesion programmes
Advantages
• Articulates rationality of the
programme
• Provides clear conceptual
foundation for the study
• Focuses on effects and
treats implementation issues
only as one of the explaining
factors
• Relates to scholarly theories
and give bigger picture
• Allows to discuss causal
relations
Challenges
• Using too rigid model can
lead to tunnel vision and
omitting side-effects
• There is a trade-off
between level of details
and clarity of the models
• Too much sophisticated
theoretical considerations
can alienate stakeholders
and turn evaluation into
scholar research
Bibliography
•
Chen, H.T. (2004) Practical Program Evaluation: Assessing and Improving Planning,
Implementation, and Effectiveness. Thousands Oaks: Sage Publications, Inc.
•
Donaldson, S.I. (2007) Program Theory-Driven Evaluation Science: Strategies and
Applications. New York: Lawrence Erlbaum.
•
Knowlton, L.W. & Phillips, C.C. (2008) The Logic Model Guidebook: Better Strategies for
Great Results. Los Angeles: Sage Publications, Inc.
•
Leeuw, F.L. (2003) Reconstructing Program Theories: Methods Avaliable and Problems to
be Solved. American Journal of Evaluation, 24(1), pp.5-20.
•
Patton, M.Q. (2008) Utilization-focused evaluation. 4th edition. Los Angeles, London: Sage
Publications.
•
Pawson, R. (2009) "Introduction to Realist Evaluation and Realist Synthesis", lecture on The
Academy of Evaluation, EUROREG – University of Warsaw, Warsaw, 7.02.2009
•
Weiss, C.H. (1997) How can theory-based evaluation make greater headway? Evaluation
Review, 21(4), pp.501-524.
•
Weiss, C.H. (2004) On Theory-Based Evaluation: Winning Friends and Influencing People.
Evaluation Exchange, IX(4), pp.2-3.
Contact details
Karol Olejniczak, PhD
EUROREG – University of Warsaw
www.euroreg.uw.edu.pl
• E-mail: k.olejniczak@uw.edu.pl
• Phone: +48 22 826 16 54
• Mobile: +48 696 41 22 82