PARTICIPANT-ORIENTED EVALUATION APPROACHES EDF5461 Summer 2011 Alison Moore Jenna Oster Aaron Paquette DEFINITION Participant-Oriented Evaluation Approaches "currently include many different models, but their commonality is that they all use stakeholders- people with an interest or "stake" in the program- to assist in conducting the evaluation." (Fitzpatrick 189) KEY CONCEPTS Involves participants in the program, managers, staff, and other key stakeholders (anyone involved with planning and implementation of the evaluation) as a key principle to good evaluation Involve stakeholders to teach evaluators about program Founded in Constructivism (humans gain knowledge and meaning from interaction and experience) DATA COLLECTION METHODS In Situ Observation Observing stakeholders in their interaction with the program Interviews Talking to staff, managers and stakeholders to learn more about the program Documents Using program documents to help with the evaluation TIMELINE Pre-1967 1967 Mid-1970s Present No desire to include the human elements and/or qualities within the evaluation process A push to bring human elements and/or qualities into the evaluation process: make people/ stakeholders a part of the evaluation process Observations and identifying concerns and issues of the stakeholders were examined and became enwound with the P-PE evaluation Jean King: “…overarching term that involves program staff actively in decision making” 1975: Responsive evaluation Cousins and Earl: “Applied social research… partnership between evaluator and stakeholders” New methods of participatory evaluation continue to emerge MAJOR DEVELOPERS Robert Stake: One of the first advocates for participantorientated evaluation The Countenance of Educational Evaluation (1967) Allows for a wide range of data to be collected A congruence between the intention and what is observed This model looks for continuity between what happens and what is intended Countenance Framework (1967) Countenance Matrix Responsive Evaluation Provides qualitative evidence of a programs effectiveness Includes stakeholders in the implementation of policies and goals MAJOR DEVELOPERS (CONTINUED) Egon Gruba and Yvonna Lincoln Naturalistic Inquir y (1985) Collects qualitative data to interpret and understand unique situations where there is a level of uncertainty Reliability and Validity: “Trustworthiness” Credibility Transferable Dependable Conformability Naturalistic Evaluation Responsive Evaluation + Naturalistic Methodologies = Naturalistic Evaluation Evaluators participate in the evaluation and try to uncover the needs of the all members Three phases 1. Familiarizing Phase 2. Action Phase 3. Synthesis Phase Four th-Generation Evaluation (1989) Uses constructionist paradigm to gather, analyze, and organize evaluative material based on the needs of stakeholders Never-ending process APPROACHES Participatory evaluation: “An overarching term for any evaluation approach that involves program staf f or participants actively in decision making and other activities related to planning and implementation of evaluation studies.” (p. 199) Two main types of participatory evaluation: P-PE Practical Participant Evaluation is used for practical reasons, limiting the evaluation to a particular organization. T-PE Transformative Participant Evaluation means just that, to transform and empower the stakeholders (social change). CATEGORIES OF PARTICIPATORY APPROACHES There are 3 dimensions in which participatory approaches differ, according to Cousins, Donohue, and Bloom (Fitzpatrick 200): 1) Control over the evaluation or technical decisionmaking process 2) Stakeholder selection 3) Depth of Participation APPROACHES (CONTINUED) M AT R I X ( P. 2 0 4 ) Approach Practical Participatory Evaluation (P-PE) Key Elements Context • Explicit focus on evaluation decisions • Evaluators work closely with primary stakeholders • Formative decisions • Stakeholder control • Many stakeholders • Emphasis on program participants • Much involvement • Participants are oppressed • Social justice is a concern • Participant empowerment is the goal Empowerment Evaluation • Stakeholder control • Much involvement • Need for empowerment to build staff • Need for building internal • Self-monitoring and program improvement Democratically Oriented Evaluation • Evaluator control • Many stakeholders • Limited involvement • Need for a dialogue among stakeholders Transformative Participatory Evaluation (T-PE) STRENGTHS OF PARTICIPANT-ORIENTED EVALUATION APPROACHES When stakeholders are involved in evaluations, they . . . . . . provide data sources, distribute surveys, and set up focus groups . . . are familiar with program context and environment . . . establish validity of the study because they possess/incorporate knowledge to inform decisions . . . can communicate program situation/details to facilitate evaluator’s understanding Including stakeholders in evaluations . . . . . . gains their trust in the evaluator and the evaluation process . . . leads to better chances of stakeholder use/adoption . . . improves stakeholders’ understanding of program/organization LIMITATIONS OF PARTICIPANT-ORIENTED EVALUATION APPROACHES Competence/feasibility Is the evaluator truly capable of acting as facilitator, coordinator, and consultant during the evaluation process? Are stakeholders able to perform tasks required of them ? Credibility Because stakeholders are deeply invested in the program, will they act to change the program based on the evaluation results? Political Due to involvement of people (primary stakeholders, managers, staff) Time-consuming and expensive REVIEW Evaluators include stakeholders in evaluation planning and implementation Stake and Gruba and Lincoln responded in the 1970s and 1980s to established methods of evaluation Wanted approaches offering consideration of program stakeholders Two main branches of participant -oriented approaches Practical participatory evaluation (P -PE): encourage involvement/ownership, thus increasing use of results by stakeholders Transformative participatory evaluation (T-PE): empower stakeholders by direct involvement in evaluation process Three primary dimensions of participatory approaches Evaluator or stakeholders possess control Individual stakeholder selection Extent of stakeholder involvement REVIEW (CONTINUED) Practical participatory (P -PE) approaches often rely on Stakeholder involvement Practical participatory evaluation Developmental evaluation Transformative approaches often rely on Empowerment evaluation Deliberative democratic evaluation Many definitions for participative evaluation exist All involve stakeholders, but no additional details are standard