Developing Evaluation Capacity Table of Contents Key terms and definitions ..................................................................................... 2 Capacity ................................................................................... Ошибка! Закладка не определена. Capacity development ....................................................... Ошибка! Закладка не определена. Capacity building ................................................................. Ошибка! Закладка не определена. What is Evaluation Capacity Building? ................................................................... 2 Key domains .......................................................................... Ошибка! Закладка не определена. Evaluative capacity ............................................................. Ошибка! Закладка не определена. Evaluative thinking .......................................................................................................................................... 3 Evaluative readiness .......................................................... Ошибка! Закладка не определена. Challenges ............................................................................... Ошибка! Закладка не определена. Types of ECB intervention ..................................................................................... 3 Levels targeted by ECB .......................................................................................... 5 Individuals ........................................................................................................................................................... 5 Teams .................................................................................................................................................................... 5 Organisations ..................................................................................................................................................... 5 Needs assessments and constraint analyses for ECB interventions......................... 5 Individuals .............................................................................. Ошибка! Закладка не определена. Teams ....................................................................................... Ошибка! Закладка не определена. Organisations ........................................................................ Ошибка! Закладка не определена. How to design an ECB intervention ....................................................................... 5 Evaluating an ECB intervention ............................................................................. 7 Resources to follow up .......................................................................................... 7 Case studies......................................................................................................................................................... 7 Key terms and definitions Great list of definitions here, Better Evaluation An Overview of Current Status, Issues and Options Understanding the distinction between development//building A tale of value-adding What is Evaluation Capacity Building? HISTORY Many authors recognise that ECB has been a practice for some time (Compton, Baizerman & Stockdill, 2002; Milstein, et. Al., 2002; Preskill, 2008) ECB is recognised as a process “long practiced but only recently named, illuminated and explicated” (Compton, et al., 2002) The first decade of the 21st century could be considered as an important evolutionary stage in the evaluation profession in that this was the time when evaluators and organisation leaders became interested in ECB along with the wide acceptance of participatory, collaborative and stakeholder forms of evaluation (Preskill & Boyle, 2008). DEFINITIONS Aims to enable individuals and organisations to adopt the concepts and practices of evaluation ECB is the intentional work to continuously create and sustain overall organisational processes that make quality evaluation and its uses routine (Stockdill, Baiserman & Compton 2002, p.14) o This definition views ECB as a systems approach, considers the whole of the organisation as necessary to set up processes that facilitate routine evaluation practice. Evaluation capacity-building within an organisation is typically understood as an exercise in developing the evaluation skills and knowledge of some, or all, of the organisation’s staff, with a view to increasing their ability to undertake high-quality evaluations of the organisation’s projects and programs (Beere, 2005, p.41) o This definition views ECB as an approach through which influential individuals can influence the larger organisational structure to improve performance evaluation Evaluation capacity-building (ECB) is an intentional process to increase individual motivation, knowledge, and skills, and to enhance a group or organisation’s ability to conduct or use evaluation (Labin, et al., 2012, p308) o This definition views ECB as an approach through which influential individuals can influence the larger organisational structure to improve performance evaluation o ECB involves the design and implementation of teaching and learning strategies to help individuals, groups, and organizations, learn about what constitutes effective, useful, and professional evaluation practice. The ultimate goal of ECB is sustainable evaluation practice—where members continuously ask questions that matter, collect, analyze, and interpret data, and use evaluation findings for decision-making and action. For evaluation practice to be sustained, participants must be provided with leadership support, incentives, resources, and opportunities to transfer their learning about evaluation to their everyday work. Sustainable evaluation practice also requires the development of systems, processes, policies, and plans that help embed evaluation work into the way the organization accomplishes its mission and strategic goals (Preskill & Boyle, 2008, p. 444). This definition points to the importance of considering the full spectrum of stakeholder levels within the organisation o o A research synthesis of the ECB literature ECB, The Knowledge Institute https://tei.cgu.edu/evaluation-capacity-building-organizations/ Chasing a Ghost?: An Essay on Participatory Evaluation and Capacity Development Strengthen evaluation capacity, better evaluation Evaluative thinking NSW Govt guidance Types of ECB intervention Conceptual Model (i) Framework of Evaluation Capacity Building (2002) Stockdill, Baizerman, Compton Details The purpose of the framework is to provide a foundation for theory development, research and practice, as well as to differentiate ECB from evaluation. The framework describes ECB as an intentional, emergent and continuous process, represented by three interactive structural elements: o The overall process: a context-dependent, intentional ECB action system o The actual practices: the ongoing, guided processes and practices for creating and sustaining an ECB action system o The occupational orientation and practitioner role: keeping evaluation as a necessary, routine administrative part of the organization Critique (ii) Evaluation Capacity Building Conceptual Framework (2004) Counsins, Goh, Clark, Lee The purpose of the framework is to explore the interconnections and linkages between evaluation utilization, ECB and organizational learning. The framework describes evaluation as an organisational learning system where evaluative inquiry serves as a means for developing organisational learning capacity. There are key variables associated with the organization: organizational support structures, organizational learning capacity and organizational consequences. There are also key variables associated with evaluation: evaluation inquiry, evaluation capacity and evaluation consequences. Critique (iii) Three-Component Evaluation Capacity Building Framework (2008) Taylor-Powell, Boyd The purpose of this framework is to understand and evaluate ECB, as well as distinguish between evaluator and ECB practitioner roles. The framework is represented by three components, each with associated elements. The components are professional development, resources and supports, and organizational environment. These components interact to achieve the following ECB goals: o to develop and sustain an organization’s capacity to conduct and manage effective evaluation projects; o to build and use evaluation knowledge and skills; o to foster organizational learning, improvement, and accountability; and o to create awareness and support for program evaluation. Critique (iv) Multidisciplinary Model of Evaluation Capacity Building (2008) Preskill and Boyle The purpose of this model is to address the lack of comprehensive conceptual models to guide ECB efforts and evaluate the effectiveness of ECB processes and activities. The model is depicted visually by two adjacent circles. The left circle represents the initiation, planning, design, and implementation of an ECB effort. It includes: o the motivations, assumptions, and implicit or explicit expectations regarding the ECB effort; o the design, implementation, and evaluation of the ECB effort; and o the ECB strategies for developing individuals’ evaluation knowledge, skills, and attitudes. Motivations, assumptions and expectations affect the design and implementation of ECB activities. The right circle represents the processes, practices, policies and resources required for sustainable evaluation practice. Critique (v) Integrated Evaluation Capacity Building Model (2012) Labin, Duffy, Meyers, Wandersma, Lesesne The purpose of this model was to guide data extraction for a broad-based research synthesis of the ECB empirical literature. The model is represented as a basic linear logic model of need-activitiesoutcomes. Although it implies a causal direction from left to right, it acknowledges the dynamic, iterative process of ECB. The need reflects the motivations, goals, context, and resources and strengths for ECB. The need affects the selection, design and implementation of ECB activities. Activities are strategies to build evaluation capacity. Critique Catalyst-for-Change Approach ECB Methods Mentoring Evaluation Learning Circles Participatory Training Evaluation Method Great overview of frameworks here, Better Evaluation How to strengthen evaluation capacity, Better Evaluation ECB interventions and the development of sustainable evaluation practice Narrative review of strategies by organisations for building evaluation capacity Levels targeted by ECB Individuals Teams Organisations Needs assessments and constraint analyses for ECB interventions Context analysis checklist for M&E capacity development Example capacity assessment - organisational top 10 tips for ECB in orgs How to design an ECB intervention Evaluation capacity building in complex organisations Organisational Capacity Building Framework A checklist for building organisational evaluation capacity What is the evaluand? What are ECB learning/outcomes? The concept of intervention suggests the idea of an intentional program design. This means that ECB has implicit program theory (ToC). Thus, the assumption of ECB as a “learning intervention” is to recognise the view that ECB is both a teaching-learning process in the area of educational theory as well as a programmatic intervention in the area of program evaluation. Therefore, two ideas that appear useful to borrow for edu measurement theories: o (1) Developmental constructs of learning – in learning intervention settings, developmental constructs are formulations of the steps or stages of increasing competence. It is important that practitioners use these stages to think developmentally about the intervention to support learning. The importance of developmental constructs is that they can provide a basis for identifying the Zone of Proximal Developmental (ZPD) of the learners, that is, the position in the learning progression a learner is ready to learn. Once the SPD is identified, this information can be used to plan and monitor the teaching-learning intervention. Developmental framework theories include: Bloom’s Taxonomy – use this one, same as CH? Possible structuring of ECB content topics in developmental progression as a reference for ECB implementation and measurement e.g. ECB developmental proficiency. o (2) Developmental approach to assessment – the developmental approach to assessment followed from the ideas of developmental constructs of learning and measurement theories of Rasch (1960, 1980) and Glaser (1963) as cited by Griffin (2007). This approach proposed that once a developmental progression to learning is identified, it can be used for assessment that could subsequently be used as a starting point for learning and the beginning of change. E.g. Griffin’s (2007) Five-Step Approach to Developmental Assessment. The key point re: this framework is that ECB measurement can only find meaning when it can be interpreted as a performance level on the development progression. The five core ECB conceptual models contain common elements that provide a solid foundation for designing and implementing an ECB initiative. These elements include: o drivers of ECB, such as external mandates for accountability; o organizational factors that mediate the effect of ECB strategies on ECB outcomes, such as organizational learning capacity; o organizational factors that support developing and sustaining evaluation practice, such as access to technology; o individual factors that contribute to effective evaluation practice, such as evaluation knowledge and skills; and o ECB strategies that develop the individual factors, such as evaluation training. *** Think about the ASRC model – mentoring, progressing learning and theory as well as direct application in stages Evaluating an ECB intervention Great resource here, Better Evaluation The Capacity Development Results Framework, Better Evaluation M&E of Capacity building programs!!!!!!!!!!!!!! Quick and Dirty Evaluation of Capacity building Whether the focus is on individual, team or organisational learning, the teaching-learning processes need investigation. Theories of learning and assessment may help reveal how ECBs could work, for example, by examining the content material, the learning activities, and the way learning is assessed. - Self evaluation rubric Evaluation rubric Summit workshop Interviews / focus groups Resources to follow up AHHHHmazing resource Some Under examined Aspects of Evaluation Capacity Building Building Effective Evaluation Capacity: Lessons from Practice Case studies An Evaluation Capacity-Building Process for Sustainable Community IT Initiatives: Empowering and Disempowering Impacts (Two Australian communities) It Is Only New Because It Has Been Missing for so Long: Indigenous Evaluation Capacity Building (Indigenous: Canada) A Case study of ECB in a private organisation, Canada ECB – definitional and practical implications from an Australian case study