Logic Models for Evaluation: The Search for Clarity Gail Johnson Spring 2008 logic models class.ppt 1 Program Evaluation Common analytical technique in public administration and non-profit programs Purpose: to tell what happened during the implementation of a program, tell how well a program is working, and whether it is achieving the desired results. Two main types: Formative: focus on process Summative: focus on outcomes logic models class.ppt 2 Problem For too long, public agencies (and non-profits) have focused on reporting budgets and activities. Program managers: get enmeshed in fighting the alligators that they forget that the purpose of the organization is to drain the swamp. Captured by the demands of the job, lose the big picture logic models class.ppt 3 Emergence of Logic Modeling Logic modeling for evaluation grew out of two other techniques: Systems Thinking Performance Measurement logic models class.ppt 4 Systems Model Environment Inputs Black Box logic models class.ppt Outputs 5 Lessons Learned from Systems Thinking The importance of the environment—or context You can have a great program for empowering women through micro-lending, but if a civil war breaks out, it chnages everything. Complexity: Zoom-in and zoom-out: Logic models within logic models Challenge: to find the right focus Non-linear nature: importance of feedback loops logic models class.ppt 6 Performance Measures GPRA, GMAP, etc Move from focus on Process to focus on Outcomes: What changes because of the program? Need to focus on indicators to measure change Search for indicators logic models class.ppt 7 Thinking Differently: Working with Logic Models Models help us visualize a program in context Models help us identify the relationships between various components: Models help us think more analytically about cause and effect: Describes the theory of change guiding the program intervention Models help us identify key assumptions that can be tested logic models class.ppt 8 Models: Cause and Effect Did the program cause something to happen? Education Employment logic models class.ppt 9 SystemsTheory-based Evaluation Its focus is to understand the nature of the problem and the relationships between the problem, intervention and expected outcome. What is the theory? Why do we believe that training people will result in better evaluations? Why do we believe that funding microbusinesses will reduce poverty? If an assessment about the relevance is to be made, the theory has to be examined. logic models class.ppt 10 Theory and Causal Chains Training in Evaluation High Quality Evaluations Training in Evaluation High Quality Evaluations Useful Information Training in Evaluation High Quality Evaluations Useful Information Better Decisions logic models class.ppt 11 Logic Model Impact Outcomes Outputs Activities Inputs logic models class.ppt 12 Implementation Results Logic Model: • Long-term, widespread improvement in society Outcomes • Intermediate effects of outputs on clients Outputs • Products and services produced • Tasks personnel undertake to transform inputs to outputs • Financial, human, and material resources Goal Impacts) Activities Inputs logic models class.ppt 13 Logic Model: Adult Literacy Goal (Impacts) Outcomes • Higher income levels; increase access to higher skill jobs • Increased literacy skill; more employment opportunities Outputs • Number of adults completing literacy courses Activities • Literacy training courses Inputs • Facilities, trainers, materials logic models class.ppt 14 Logic Model Outcomes/Impacts: The changes to expect to see if your theory is correct and the program has been implemented well What would be different because of the program? Observable and Measurable Indicators of Change Outcomes: Initial, intermediate, and long-term change Impact: the big enchilada, long-term, related to program goal Reduce Poverty logic models class.ppt 15 Logic Models If we train the chronically unemployed, then their quality of life will be improved, and the overall poverty rate will be reduced. What is the underlying theory? What’s the model? Unemployed Training Employment Increased Income Reduced Poverty logic models class.ppt 16 Logic Model: Training Program Inputs Resources •money •staff •Supplies •mentors Activities •Training programs •Dress for success coaching •Interview coaching •Resume assistance Outputs Products •Number of graduates per training session •% graduate rate logic models class.ppt Outcomes Impacts Benefits changes Goals •Increased skills •% Obtain jobs •% Obtain high paying, quality jobs •Increased self-esteem •Increased income •Selfsufficiency •Family stability •Reduction in poverty 17 Exercise: Women’s Program Women-owned micro-enterprises Handout Put in logical order: Sequence of Inputs, Activities, Outputs and Outcomes logic models class.ppt 18 Logic Model: The Process It brings in all the key players: stakeholders, staff, clients/customers, political folks, and researchers It is a process that helps everyone get clear about what it is they are trying to accomplish It is a living thing: changes Handout: Compare Figure 6.4 and figure 6.8 logic models class.ppt 19 Complexity Challenge: level of detail Handling multiple theories of change Logic models within logic models Zoom-in or Zoom out: trying to get the right focus Accurate, clear but not overwhelming logic models class.ppt 20 Logic ModelEvaluation When done right, sets the focus for the evaluation (Figure 6.8) Focus on activities: Did they do what they said the were going to do? If not, what barriers were in the way? Focus on outcomes: 5 critical outcomes Do all, some, time-sequence? logic models class.ppt 21 Exercise Familiar Program Goals? Objectives? Activities? Underlying theory logic models class.ppt 22 Applying the Program Outcome Model Inputs Activities Outputs logic models class.ppt Outcomes Impact 23 Challenges in Developing Logic Model Outputs and activities are seen as outcomes Program managers focus on actions Visible and more within program managers’ control Easier to confirm task completion than measure an outcome Measures are not easy to come by Not all the can be counted really matters and all that matters can not necessarily be counted logic models class.ppt 24 Logic Model: No Panacea Can’t fix a situation where there is no real theory of change Still need a rigorous research design Other alternatives might work better Stuckness No intervention exists in isolation Larger events may be uncontrollable. logic models class.ppt 25 Summary of Program Outcome Model Identify components of a program Identify relationships between the program: expected and actual Identify what we want to measure Focuses our attention on outcomes and impacts logic models class.ppt 26 Resources Joy A. Frechtling, Logic Modeling Methods in Program Evaluation, Wiley and sons, 2007. Kellogg Foundation: Logic Model Guide http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf CCD: Logic Model Resources http://www.cdc.gov/eval/resources.htm#logic%20model logic models class.ppt 27