Partnership What to Expect From This Series 1. Learn about steps required to implement outcome evaluation 2. Design a program logic model 3. Identify appropriate evaluation/measurement tools 4. Develop an implementation plan Understanding Outcome Evaluation: Core Concepts and Building the Foundations Session #1/3 Purpose of Today’s Session • To gain a basic understanding of the core concepts and basic steps in outcome measurement in particular, and program evaluation in general. • To share strategies for building buy-in to outcome measurement. • To begin work on a program logic model for a program of your choice. Today’s Agenda • Outcome evaluation language • Outcome evaluation steps • Break • Outcome evaluation steps • Lunch • Developing logic models Our Approach to Outcome Measurement Discussion Imagine… Our Approach to Outcome Measurement • Evaluation is already going on, informally, in any healthy organization and can be harnessed. • Evaluation is a natural and necessary part of organizational development, not an “add on.” • Evaluation strategies that help one organization may be confusing or even damaging to others: there is no “cookie cutter” approach. Our Approach to Outcome Measurement • Developing common language within and between organizations is key. • Organizations get better at measuring outcomes when they have practice, and when they identify measurement priorities themselves. • Evaluation is about organizing, interpreting and acting on information, not about gathering mountains of data. Improved Outcome Measurement Clear Evidence of Impact Useful Insights, New Questions Improved Practice A Few Key Definitions Community Based Research is a term that refers to a variety of efforts to apply research tools and strategies to the task of building communities. Community based research can take many forms, including needs assessment, environmental scans, etc. .. Program evaluation is one form of community based research. It focuses on the study of community interventions... Their processes, outcomes, and context. Outcome evaluation is one type of program evaluation, in which: 1. outcome objectives for activities are identified, 2. success in reaching objectives is measured, and 3. best practices are identified, and 4. recommendations for improvement result. Outcome measurement is a process in which: 1. outcome objectives are identified and 2. success in reaching objectives is measured The Dilemma of Terminology When Talking About… Evaluators and Funders Use Terms Like… The Process of Measuring and Learning from Outcomes • Stakeholder Consultation, Participatory Approach, Utilization Plan, Lessons Learned, Implementation, Process, Evaluability The Intent of an Intervention • Outcome Objectives, Implementation Objectives, Goals, Outcomes, Targets, Intended Results The Changes Caused by an Intervention • (Short and Long Term) Outcome Objectives, Goals, Impacts, Results The Information Used to Measure Change • Inputs, Outputs, (Performance) Indicators, Targets, Benchmarks, Data Sources Outcome Measurement The Basic Steps The Seven Practical Steps in Outcome Measurement 1. Get Buy-In 2. Clarify Theory 6. Communicate Results 7. Act on Results Laying the Foundations 3. Develop Evaluation Purpose & Evaluation Questions Evaluation Planning Acting on Findings Gathering & Analyzing Information 4. Design Methods and Measurement plans 5. Analysis Plan Data Doesn’t Change the World… People Do. Evaluation Requires an Ongoing Focus on Relationship Building The Role of Collaboration and Teamwork in Community Based Research Situating Yourself: Clarifying Values & Beliefs, Reviewing Resources Deciding & Acting Together Using Knowledge Clarifying The Issues and Identifying Research Questions Building & Maintaining Relationships For Action Gathering, Sharing, and Organizing Knowledge Step 1: Building Stakeholder Involvement from the Beginning Fostering Buy-in/Stakeholder Participation: How? • Have all those who have a stake in the project and its evaluation been identified? Is there support for the idea of evaluation? • Are there other ways that stakeholders could be included in the early planning phase and throughout the evaluation? • Has the political context of relationships among stakeholders been considered? • Have the interests of outside audiences been identified and considered? Creative Ways to Involve Stakeholders • Helping to build logic models • “Teaching” the evaluators • As surveyors or interviewers • As pilot testers for data collection tools • As note-takers at focus groups • As co-presenters of findings • As participants in a “data analysis forum” • OTHERS? Discussion • The first meeting of your evaluation committee • In the context of measuring outcomes, when do you run into stakeholder conflict? BREAK Step 2: Clarifying Your Theory of Change What is a Theory of Change? • A way of clearly explaining why you do what you do • An explanation that focuses on exactly how your work will lead to positive change. Discussion Which one of the following theory statements best describes program impact? We have implemented a new type of women’s shelter… – because a local report identified the need for more shelter beds locally. – so that our organization can provide shelter for 20% more women – that is at a different location more accessible to younger women – so that young women will be more likely to report abuse and seek support – in order to identify instances of abusive behaviour earlier in relationships and increase the chances of breaking the cycle of abuse Components of a Basic Program Logic Model • Activities explain the concrete things you do within your program • Short-Term Outcome Objectives identify the fairly immediate changes you expect to see as a direct result of your work • Long-Term Outcome Objectives identify the more distant benefits or changes you hope to contribute to by achieving your short-term outcomes • Goals are the longest term, broad vision of your program The Really Useful Idea (articulating your theory of change) Identify short-term outcome objectives that are within your control, measurable and achievable, and use a logic model to explain how these short-term achievements contribute to the achievements of long-term outcomes over time. Activity Refer clients to Job Leaders Short-term Outcome Improve clients’ knowledge of job opportunities Long-term Outcome Improve clients’ employment networks How Do Logic Models Help You to Measure Outcomes? • They help “unpack” complex, important long-term outcomes into short-term outcomes that can more easily be measured • They help you sort out exactly how and why you expect outcomes to occur, so you know where to look for evidence of success • They help you distinguish actual “measurement problems” from the much more common “theory problems” Components of a Basic Program Logic Model (After Rush & Ogborn (1991) CJPE) Activity Clusters – The concrete things you do within your program – The major categories or clusters which describe the activities that make up your work – Clusters should be composed of activities that are similar at a behavioural level, not activities with similar purposes (e.g., cooking in the kitchen and cooking at the barbecue are similar activities, but cooking dinner and buying the groceries for dinner are not). Components of a Basic Program Logic Model Outcome Objectives - Measurable in principle - Describes a meaningful change - Need not specify how much change will occur or what the target level is - Specifies a particular population and situation - Clearly explained and theoretically defensible - Focused on active rather than the neutral or passive change (i.e., “improved”, or “reduced” vs. “supported”, “promoted”, or “encouraged”) Example: Increased feelings of social support among participating parents Components of a Basic Program Logic Model Short-term Outcome Objectives - The immediate benefits or changes that the target groups are anticipated to experience or display as a result of the program activities. - It should be possible to identify a clear, direct intended causal link between at least one program activity and each identified short-term outcome objective. Components of a Basic Program Logic Model Long-Term Outcome Objectives - The more distant benefits or changes that the target groups are anticipated to experience or display as a result of the initiative. - Generally, long-term outcome objectives are the second-order changes that result from successful achievement of short-term outcomes over time. Refer clients to Job Leaders Improve clients’ knowledge of job opportunities Improve clients’ employment networks Example of a well-done logic model for a program with individualized outcomes Long-term outcome objectives Short-term outcome objectives Activities Develop and support clients’ individual rehab plans Clients become aware of +/or are connected to community resources Clients have improved informal supports Clients are able to find +/or maintain accommodation of their choice Clients have longer community tenure Provide clinical support and interventions Clients maintain or improve their level of function Clients have increased confidence in ability to live independently Client becomes a more productive, independent member within their community Assess and develop life skills Clients gain insight to healthier lifestyles/choices Reduce the # of hospitalizations or shorten lengths of stay Long-term outcome objectives Short-term outcome objectives Activities Example of a well-done logic model for a crisis oriented program Weekly self help group for women Women develop friendships in the group Women support each other outside of the group Women feel supported in the decisions they make Women are able to evaluate relationship with partner and make decisions about the relationship Partner is involved as part of family decision making Women access resources Family members reconstruct a support system and maintain family unity Women feel less fragile and are not dealing with daily crises Women articulate expectations of their partners Working on Logic Models 2-4 pm Exercise 1: Describing your Activities Think about what your staff and/or volunteers do on a day to day basis in as concrete terms as possible. Focus on activities that involve contact with clients or the community. Activities that should be in your model are those that you expect will impact individuals or communities in some positive way. Staff are most often “doing things”, like “providing”, “teaching”, “raising awareness”, “creating”, etc. so try to use these types of active verbs. List as many activities as you can and try to be as honest as you can about what you actually do. Try and “cluster” your activities if they seem to involve doing similar things. Exercise 1: Describing your Activities Avoid putting “outcome” language into your activities. For example, avoid terms like “prevent”, “increase”, or “improve” because these are changes that are a consequence of the activities. Identify the things your program does that leads to prevention, increases, or improvements. Avoid “double barreled” activities (e.g., “provide education and personal support”). This is especially important if they might lead to different outcomes. Separate them if possible. While administrative tasks are important and vital (e.g., team meetings, hiring functions, etc.), you can usually omit them in logic models because they do not tend to directly impact outcomes objectives. You can leave these out and add them later if it helps to clarify your program. Exercise 2: Identifying the Outcome Objectives for your Program Develop a list of the outcome objectives your program is intended to achieve. Include as many outcomes as you can. You might also think about your mission, your marketing materials, grant proposals, and your own reasons for doing your job. Focus on how the program makes a difference or change in your clients or the community. For example, if an outcome objective of the program is to give people someone to talk to, ask yourself how this may help an individual. Try to write your outcome objectives in way that refers to change that can be measured. Think about sequence. Logically, some outcome objectives will necessarily precede others. Try to arrange them temporally from top to bottom. Exercise 3: Putting it all Together • Using a pencil, begin to draw the connections between your activities and your short term outcome objectives and between your STOs and your long-term outcome objectives. Keep in mind the following: All STOs should be linked to at least one activity; all activities should be linked to at least one STO. All LTOs should be linked to at least one STO; all STOs should be linked to at least one LTO. Pick an arrow that you think might have questionable logic or weak assumptions and make it “dashed”. Pick an arrow that you think has strong logic and wellsupported assumptions and put a check mark beside it . • Think about logical causes. If a group support intervention is supposed to lead to greater feelings of social support, think critically about why and how that happens. Does it make sense? In some cases, this analysis might lead you to add, delete, or modify activities and outcome objectives. Using the On-Line Logic Model Design Tool What to Expect at Our Next Session • Respond to questions regarding finishing up logic models • Identify indicators • Start thinking about measurement tools Upcoming Workshops • Workshop 2 focuses on – Refining logic models and identifying measurement implications • Workshop 3 focuses on – Translating measurement plans into action Thank You! Step 3: Developing Your Evaluation Purpose & Evaluation Questions What do you really want to know? What kinds of questions can evaluation really answer for you? Breaking Down the Complexity What were we trying to change? (Long-Term Outcome Objectives) Did our program have an impact??? What particular contribution were we going to make to that change? (Short-Term Outcome Objectives) How were we planning to make that contribution? (Activities) What sorts of things would we see if the expected change was happening? (Indicators) How can we document those observations in a systematic way? (Methods) The Evaluation Purpose Statement • A purpose statement should specify exactly what the goals of the research project are, and what you intend to investigate in order to achieve these goals. A Good Purpose Statement Explains: What we are evaluating (and what are we NOT evaluating) Why we are evaluating What we will do with the results Defining Evaluation Questions • The empirical questions to which your evaluation will generate answers • Usually about 5 to 7 in number except in very complex evaluations • The things that will keep you focused as your evaluation project unfolds Clarifying the Levels of Questions To determine how effective our program is at helping people find meaningful jobs Evaluation Purpose Evaluation Questions Actual Interview or Survey Items How many of our clients find jobs? How satisfied are clients with their jobs? For clients: “How much do you enjoy your job?” What parts of our program seem to help the most? For staff: “How satisfied does your client seem?” Ways to Clarify Evaluation Questions • Focus on the action-oriented and solution focused (What can you do with the answers?) • Review what knowledge your group already has. • Whose knowledge is valued by decisionmakers? Whose language is being used? • What are the gaps in knowledge? • What are the empirical questions within this issue? What extra value can someone with research expertise bring to the table? WRAPPING UP STEPS 1-3 You’ve got stakeholder buy-in, you’ve articulated your theory of change, and you’ve figured out your evaluation purpose and the evaluation questions you want to answer… Are You Really Ready to Do Outcome Measurement? • This is the point at which you need to make a decision about whether you have stakeholder buy-in. Outcome measurement will not be successful without buy-in. You want to do more outcome measurement 5. Agree on purpose and questions, set roles, talk about use of findings 1. Develop Logic models & Evaluability assessment Needs assessment, program design, planning 6. Develop workplan 2. Develop Purpose and Questions 3. Identify Indicators NO YES Is outcome measurement the next step? Awareness raising & team building; recruitment of additional people 4. Bring together stakeholders NO YES Is there buy-in? Step 4: Developing Measurement Plans What are Indicators? • Bits of information that provide part of the answer to one of your questions • Things you can see or touch or hear – things that are observable in the world and don’t involve ‘interpretation’ • Are often numbers but can also be (e.g.) stories, quotations, examples, pictures Indicators are a Useful Idea Because... • They help to break down the complex task of “program evaluation” or “outcome measurement” into manageable chunks • They help others to understand what you mean in practical terms when you talk about a particular outcome • They help you build up a strong evaluation plan by combining different kinds of information from several sources The Notion of Triangulation Explaining the Analogy • When you ask an evaluation question, you’re like a navigator on a ship asking “where am I?” • Progress indicators are like the various compass bearings a navigator takes. The more “diverse” the readings, the better! • If all the bearings suggest you’re in the same spot, you know you’ve done a good job of measuring. Explaining the Analogy • No single indicator, regardless of how accurate, can tell you where you are! • Even if the indicators don’t converge or triangulate, you know more than you did and you know something about how much “error” is in your measurement technique. Discussion Indicator Exercise Something to Think About Over Lunch… • What makes a good lunch “good?” • How would you find out whether this lunch that we are about to have was successful? • How would you prove to someone who wasn’t at our lunch that it was (or was not) successful? LUNCH Choosing Information Gathering Methods Pros and Cons of Popular Measurement Techniques • Retrospective reflections or stories More Practical, Less Rigorous • Self report (interview or survey) • Peer/family/worker report • Direct observation • File review • Clinical assessments Less Practical, More Rigorous Step 5: Analyzing Your Data Data Analysis • Analysis is a process of bringing order to data, organizing what’s there in patterns and categories • Interpretation involves attaching meaning and significance, explaining descriptive patterns, looking for relationships among descriptive dimensions. Suggested Steps in Data Analysis 1. Organize Data 2. Review Original Questions 3. Summarize and Code Steps in Data Analysis 4. Generate Themes 5. Begin Writing 6. Provide and Receive Feedback WRAPPING UP STEPS 1-5 The Evaluation Plan Evaluation Methods Planning Chart Evaluation Questions Outcome Objectives May refer to Match up with success in Column #2 of meeting Worksheet #2 objectives (from your logic model) or other questions stakeholders feel the evaluation should address. Relevant Indicators Where will we get the information? What, specifically, will we ask? What data Data Analysis collection tools will we use? Things you can observe that will help you to answer this question. Who will you speak to? How would you access existing information? How will the questions be worded? Given what is written in columns to the left, what method is most efficient and effective? How are you going to make sense of the data you collect? The Seven Practical Steps in Outcome Measurement 1. Get Buy-In 2. Clarify Theory 6. Communicate Results 7. Act on Results Laying the Foundations 3. Develop Evaluation Purpose & Evaluation Questions Evaluation Planning Acting on Findings Gathering & Analyzing Information 4. Design Methods and Measurement plans 5. Analysis Plan Steps 6 & 7: Communicating and Using Results “Problems facing the poor and the powerless must be understood in the hearts and the guts, as well as in the heads. The people with the problems must talk to each other as whole persons with feelings and commitment as well as facts. As a tool of research, dialogue produces not just factual knowledge, but also interpersonal and critical knowledge, which defines humans as autonomous social beings” (Park, 1993 as cited in Nelson, Ochocka, Griffin, & Lord, 1998). Communication: Questions to Consider • How much interpretation do you want to do (how much do others do)? • Who should act after learning about the findings? • What do you want them to do with the findings? • How do you package information in order to facilitate this? Where to Start You want to do more outcome measurement 5. Agree on purpose and questions, set roles, talk about use of findings 1. Develop Logic models & Evaluability assessment Needs assessment, program design, planning 6. Develop workplan 2. Develop Purpose and Questions YES 3. Identify Indicators NO YES Is outcome measurement the next step? Awareness raising & team building; recruitment of additional people Is there a fit with resources? 4. Bring together stakeholders NO YES NO Is there buy-in? Revise plans, prioritize, discuss with funders Discussion: Navigating First Steps • From where you sit right now in your organization, how do you feel about these first steps? • Which step strikes you as being the most challenging? Action Checklist • Have you got key stakeholders involved? • Have you clarified program logic and evaluation purpose? • Have you identified indicators for key outcome questions? • Have you maximized the use of the data you already have or can easily get?