Developing Evaluation Capacity
Table of Contents
Key terms and definitions ..................................................................................... 2
Capacity ................................................................................... Ошибка! Закладка не определена.
Capacity development ....................................................... Ошибка! Закладка не определена.
Capacity building ................................................................. Ошибка! Закладка не определена.
What is Evaluation Capacity Building? ................................................................... 2
Key domains .......................................................................... Ошибка! Закладка не определена.
Evaluative capacity ............................................................. Ошибка! Закладка не определена.
Evaluative thinking .......................................................................................................................................... 3
Evaluative readiness .......................................................... Ошибка! Закладка не определена.
Challenges ............................................................................... Ошибка! Закладка не определена.
Types of ECB intervention ..................................................................................... 3
Levels targeted by ECB .......................................................................................... 5
Individuals ........................................................................................................................................................... 5
Teams .................................................................................................................................................................... 5
Organisations ..................................................................................................................................................... 5
Needs assessments and constraint analyses for ECB interventions......................... 5
Individuals .............................................................................. Ошибка! Закладка не определена.
Teams ....................................................................................... Ошибка! Закладка не определена.
Organisations ........................................................................ Ошибка! Закладка не определена.
How to design an ECB intervention ....................................................................... 5
Evaluating an ECB intervention ............................................................................. 7
Resources to follow up .......................................................................................... 7
Case studies......................................................................................................................................................... 7
Key terms and definitions




Great list of definitions here, Better Evaluation
An Overview of Current Status, Issues and Options
Understanding the distinction between development//building
A tale of value-adding
What is Evaluation Capacity Building?
HISTORY
 Many authors recognise that ECB has been a practice for some time
(Compton, Baizerman & Stockdill, 2002; Milstein, et. Al., 2002; Preskill, 2008)
 ECB is recognised as a process “long practiced but only recently named,
illuminated and explicated” (Compton, et al., 2002)
 The first decade of the 21st century could be considered as an important
evolutionary stage in the evaluation profession in that this was the time
when evaluators and organisation leaders became interested in ECB along
with the wide acceptance of participatory, collaborative and stakeholder
forms of evaluation (Preskill & Boyle, 2008).
DEFINITIONS
 Aims to enable individuals and organisations to adopt the concepts and
practices of evaluation
 ECB is the intentional work to continuously create and sustain overall
organisational processes that make quality evaluation and its uses routine
(Stockdill, Baiserman & Compton 2002, p.14)
o This definition views ECB as a systems approach, considers the whole of
the organisation as necessary to set up processes that facilitate routine
evaluation practice.
 Evaluation capacity-building within an organisation is typically understood
as an exercise in developing the evaluation skills and knowledge of some, or
all, of the organisation’s staff, with a view to increasing their ability to
undertake high-quality evaluations of the organisation’s projects and
programs (Beere, 2005, p.41)
o This definition views ECB as an approach through which influential
individuals can influence the larger organisational structure to improve
performance evaluation
 Evaluation capacity-building (ECB) is an intentional process to increase
individual motivation, knowledge, and skills, and to enhance a group or
organisation’s ability to conduct or use evaluation (Labin, et al., 2012, p308)
o This definition views ECB as an approach through which influential
individuals can influence the larger organisational structure to improve
performance evaluation
o ECB involves the design and implementation of teaching and learning
strategies to help individuals, groups, and organizations, learn about
what constitutes effective, useful, and professional evaluation practice.
The ultimate goal of ECB is sustainable evaluation practice—where
members continuously ask questions that matter, collect, analyze, and
interpret data, and use evaluation findings for decision-making and
action. For evaluation practice to be sustained, participants must be
provided with leadership support, incentives, resources, and
opportunities to transfer their learning about evaluation to their
everyday work. Sustainable evaluation practice also requires the
development of systems, processes, policies, and plans that help embed
evaluation work into the way the organization accomplishes its mission
and strategic goals (Preskill & Boyle, 2008, p. 444).
 This definition points to the importance of considering the full
spectrum of stakeholder levels within the organisation
o
o








A research synthesis of the ECB literature
ECB, The Knowledge Institute
https://tei.cgu.edu/evaluation-capacity-building-organizations/
Chasing a Ghost?: An Essay on Participatory Evaluation and Capacity
Development
Strengthen evaluation capacity, better evaluation
Evaluative thinking

NSW Govt guidance
Types of ECB intervention
Conceptual Model
(i) Framework of
Evaluation Capacity
Building (2002)
Stockdill, Baizerman,
Compton
Details
 The purpose of the framework is to provide a foundation for theory
development, research and practice, as well as to differentiate ECB from
evaluation.
 The framework describes ECB as an intentional, emergent and
continuous process, represented by three interactive structural elements:
o The overall process: a context-dependent, intentional ECB action
system
o The actual practices: the ongoing, guided processes and practices
for creating and sustaining an ECB action system
o The occupational orientation and practitioner role: keeping
evaluation as a necessary, routine administrative part of the
organization
Critique
(ii) Evaluation Capacity
Building Conceptual
Framework (2004)
Counsins, Goh, Clark, Lee


The purpose of the framework is to explore the interconnections and
linkages between evaluation utilization, ECB and organizational
learning.
The framework describes evaluation as an organisational learning system
where evaluative inquiry serves as a means for developing organisational

learning capacity.
There are key variables associated with the organization: organizational
support structures, organizational learning capacity and organizational
consequences. There are also key variables associated with evaluation:
evaluation inquiry, evaluation capacity and evaluation consequences.
Critique
(iii) Three-Component
Evaluation Capacity
Building Framework
(2008)


Taylor-Powell, Boyd
The purpose of this framework is to understand and evaluate ECB, as
well as distinguish between evaluator and ECB practitioner roles.
The framework is represented by three components, each with associated
elements. The components are professional development, resources and
supports, and organizational environment. These components interact to
achieve the following ECB goals:
o to develop and sustain an organization’s capacity to conduct and
manage effective evaluation projects;
o to build and use evaluation knowledge and skills;
o to foster organizational learning, improvement, and
accountability; and
o to create awareness and support for program evaluation.
Critique
(iv) Multidisciplinary
Model of Evaluation
Capacity Building
(2008)


Preskill and Boyle

The purpose of this model is to address the lack of comprehensive
conceptual models to guide ECB efforts and evaluate the effectiveness of
ECB processes and activities.
The model is depicted visually by two adjacent circles. The left circle
represents the initiation, planning, design, and implementation of an
ECB effort. It includes:
o the motivations, assumptions, and implicit or explicit
expectations regarding the ECB effort;
o the design, implementation, and evaluation of the ECB effort;
and
o the ECB strategies for developing individuals’ evaluation
knowledge, skills, and attitudes.
Motivations, assumptions and expectations affect the design and
implementation of ECB activities. The right circle represents the
processes, practices, policies and resources required for sustainable
evaluation practice.
Critique
(v) Integrated
Evaluation Capacity
Building Model (2012)
Labin, Duffy, Meyers,
Wandersma, Lesesne


The purpose of this model was to guide data extraction for a broad-based
research synthesis of the ECB empirical literature.
The model is represented as a basic linear logic model of need-activitiesoutcomes. Although it implies a causal direction from left to right, it
acknowledges the dynamic, iterative process of ECB. The need reflects
the motivations, goals, context, and resources and strengths for ECB.
The need affects the selection, design and implementation of ECB
activities. Activities are strategies to build evaluation capacity.
Critique
Catalyst-for-Change
Approach
ECB Methods
Mentoring
Evaluation Learning Circles
Participatory Training Evaluation Method




Great overview of frameworks here, Better Evaluation
How to strengthen evaluation capacity, Better Evaluation
ECB interventions and the development of sustainable evaluation practice
Narrative review of strategies by organisations for building evaluation
capacity
Levels targeted by ECB
Individuals
Teams
Organisations
Needs assessments and constraint analyses for ECB
interventions





Context analysis checklist for M&E capacity development
Example capacity assessment - organisational
top 10 tips for ECB in orgs
How to design an ECB intervention



Evaluation capacity building in complex organisations
Organisational Capacity Building Framework
A checklist for building organisational evaluation capacity

What is the evaluand? What are ECB learning/outcomes?

The concept of intervention suggests the idea of an intentional program
design. This means that ECB has implicit program theory (ToC). Thus, the
assumption of ECB as a “learning intervention” is to recognise the view
that ECB is both a teaching-learning process in the area of educational
theory as well as a programmatic intervention in the area of program
evaluation. Therefore, two ideas that appear useful to borrow for edu
measurement theories:
o (1) Developmental constructs of learning – in learning
intervention settings, developmental constructs are formulations
of the steps or stages of increasing competence. It is important that
practitioners use these stages to think developmentally about the
intervention to support learning. The importance of
developmental constructs is that they can provide a basis for
identifying the Zone of Proximal Developmental (ZPD) of the
learners, that is, the position in the learning progression a learner
is ready to learn. Once the SPD is identified, this information can
be used to plan and monitor the teaching-learning intervention.
 Developmental framework theories include: Bloom’s
Taxonomy – use this one, same as CH?
 Possible structuring of ECB content topics in developmental
progression as a reference for ECB implementation and
measurement e.g. ECB developmental proficiency.
o (2) Developmental approach to assessment – the
developmental approach to assessment followed from the ideas of
developmental constructs of learning and measurement theories
of Rasch (1960, 1980) and Glaser (1963) as cited by Griffin (2007).
This approach proposed that once a developmental progression to
learning is identified, it can be used for assessment that could
subsequently be used as a starting point for learning and the
beginning of change. E.g. Griffin’s (2007) Five-Step Approach to
Developmental Assessment. The key point re: this framework is
that ECB measurement can only find meaning when it can be
interpreted as a performance level on the development
progression.


The five core ECB conceptual models contain common elements that
provide a solid foundation for designing and implementing an ECB
initiative. These elements include:
o drivers of ECB, such as external mandates for accountability;
o organizational factors that mediate the effect of ECB strategies on
ECB outcomes, such as organizational learning capacity;
o organizational factors that support developing and sustaining
evaluation practice, such as access to technology;
o individual factors that contribute to effective evaluation practice, such
as evaluation knowledge and skills; and
o ECB strategies that develop the individual factors, such as evaluation
training.
*** Think about the ASRC model – mentoring, progressing learning and theory as
well as direct application in stages
Evaluating an ECB intervention
 Great resource here, Better Evaluation
 The Capacity Development Results Framework, Better Evaluation
 M&E of Capacity building programs!!!!!!!!!!!!!!
Quick and Dirty Evaluation of Capacity building

Whether the focus is on individual, team or organisational learning, the
teaching-learning processes need investigation. Theories of learning and
assessment may help reveal how ECBs could work, for example, by
examining the content material, the learning activities, and the way
learning is assessed.
-
Self evaluation rubric
Evaluation rubric
Summit workshop
Interviews / focus groups
Resources to follow up



AHHHHmazing resource
Some Under examined Aspects of Evaluation Capacity Building
Building Effective Evaluation Capacity: Lessons from Practice
Case studies



An Evaluation Capacity-Building Process for Sustainable Community IT
Initiatives: Empowering and Disempowering Impacts (Two Australian
communities)
It Is Only New Because It Has Been Missing for so Long: Indigenous
Evaluation Capacity Building (Indigenous: Canada)
A Case study of ECB in a private organisation, Canada

ECB – definitional and practical implications from an Australian case
study