Evaluation Best Practices Results Based & Logic Model Programs The Extension Risk Management Education Grants Program Team Bob Craven, Director – Digital Center Brad Lubben, Director – North Central Region Jo Ann Warner – Assistant Director – Western Region Kevin Klair – Digital Center With help from: Kynda Curtis, Utah State University Gary Hachfeld, University of Minnesota Extension Our Mission “Helping America’s farmers and ranchers manage risk” Extension Risk Management Education Program University of Delaware – Northeast Region University of Arkansas – Southern Region University of Nebraska - North Central Region Washington State University Extension – Western Region University of Minnesota – Digital Center USDA, National Institute of Food and Agriculture Regional Centers North Central Center University of Nebraska Northeast Center University of Delaware Cooperative Extension Southern Center University of Arkansas Division of Agriculture Western Center Washington State University Extension Return on Investment • The return on investment of Extension RME awarded grant dollars are the risk management improvements that agricultural producers make as result of a project. • The risk management results/outcomes that agricultural producers achieve “are” the measure of success of any project we fund. Process Focused on Results/Outcomes • The results based process, including the grant selection, reporting, and management system, – increases the probability that funded programs will improve the risk management skills of the farmers & ranchers who participate, – is designed to improve, over time, the capacity and quality of the programs we fund, and – increases the opportunities to improve the economic viability of the farm and ranch family participants. Evaluation Best Practices Keep the focus on: Producer Results/Outcomes Project Team’s Accountability to Measuring/Verifying Producer Results Commitment to helping producers reach their risk management goals Ability to Measure Outcomes • Improve effectiveness by looking at your project from start to finish in terms of results for participants. • Demonstrate participant achievement of risk management tools and strategies by posting on the public reporting system (final reports on ExtensionRME.org & the Ag Risk Education Library) • Move the bar higher for the effectiveness of risk management education Beginning with the “end” in mind • Begin with the end in mind – Determine first what the result/outcome is that the participants will achieve. – Design the program to lead the participants to those results/outcomes. • Do not wait till the end of the project to determine what the outcomes might be. Outcomes - Will Depend Upon: - Knowledge of your target audience - Why ag producers will choose to participate - Action and learning processes (project steps) that help producers achieve long term outcomes - Program resources including the team, collaborators and expert presenters sufficient to lead participants to the desired outcomes. Outcomes – A Systems Approach The Proposal as a working document i. Application Stage ii. Program Delivery iii. Reporting Outcomes & the Logic Model Identifying and Articulating Program Outcomes Participant Outcomes “Begin with the end in mind” Or “plan backward, implement forward” Jon Newkirk, Director, WSU Western Center Beverly Anderson Parsons, WKKF Cluster Evaluator An Outcome Defined…. Change in: Knowledge Actions or Conditions Outcomes Defined…. Changes in knowledge… When project participants 1. Learn or become aware of new fundamental or applied knowledge which could include: a) Methods and techniques b) Policy Knowledge c) Improved skills such as decision making, life skills or quality of life choices Outcomes Defined…. Change in Actions… When project participants 1. Act upon what they have learned such as: a) Implementing methods and techniques b) Changing a practice c) Adoption of improved skills such as decision making, life skills or quality of life choices d) Direct application of knowledge gained Outcomes Defined…. Change in Conditions… A socio/economic condition is positively changed because of participant actions 1. Information resources that improve community and/or farming infrastructure 2. Technology transfer 3. Improved farm income for target audience 4. Decrease in farm foreclosures A Logic Model can help you if you begin with the end in mind…. A Logic Model can help to identify your outcomes (changes in knowledge, actions or conditions) and the activities and partners that will support the accomplishment of outcomes for project participants Logic Model as a Tool to Construct Realistic Outcomes First – Identify the situation(s) that your project will help to address (farm foreclosures, high price of inputs, lack of record keeping skills etc.) and consider if what you are proposing will be well received by the target audience you have identified (Producer Demand). 1. Outcomes – The identification of changes in knowledge, actions, or conditions for program participants 2. Outputs – (activities – curriculum development, workshops, trainings etc.) that will support and lead to the achievement of participant outcomes 3. Inputs – Resources that go into your program in the way of project team, partners, collaborators, who provide expertise, research, money etc. Logic Model Beginning With the End in Mind OUTCOMES OUTPUTS INPUTS (Identify Measurable Risk Management Results) (Lead producer participants to the desired risk management results) (Resources that will strengthen producer achievement of risk management results) Logic Model Implementing Forward INPUTS OUTCOMES OUTPUTS Producer Results Outcomes Impacts (Learning – Action – Condition ) A Hierarchy of Actions and Decisions that Range from Improved Understanding to Incorporating Beneficial Risk Management Practices Producer Results/Outcomes – an example of progressive steps… ↓ Understand or have improved understanding of the risk issue, tool or need to take action. ↓ Decide to take an action. Producer commitment to write a business or marketing plan, set up a record keeping system, conduct a risk analysis, or initiate other steps that will lead to improved risk management strategies ↓ Develop business tools to strengthen economic capacity, such as writing a marketing plan, business plan, farm transfer plan, or performing a risk assessment etc. Implement or take an action that improves or optimizes their profitability– implementing a marketing plan, using a record keeping system etc. In Summary… Results/Outcomes – Measure of success for the participant • Ability to change/incorporate/implement/adopt risk management practices as a result of participation in your project. – Achievable and measureable within a reasonable timeframe during and following training. – Sequential • Identifies changes in condition or practice you seek for the participant • Specifies the degree of change required for participant success • Stated in a measurable way knowing methods you will use to verify changes in condition, practice etc. Optimizing Producers’ Chances for Success First – Identify the situation(s) that your project will help to address (farm foreclosures, high price of inputs, lack of record keeping skills etc.) and consider if what you are proposing will be well received by the target audience you have identified (Producer Demand). Then…Identify the Results/Outcomes – changes in knowledge, actions, or conditions for program participants. Describe what outputs (activities – curriculum development, workshops, trainings etc.) will support and lead to the achievement of participant outcomes. Create a realistic timetable for your actions (Project Steps) Describe inputs – resources that go into your program in the way of project team, partners, collaborators, who provide expertise, research, money etc. Logic Model Beginning With the End in Mind OUTCOMES OUTPUTS INPUTS (Identify Measurable Risk Management Results) (Lead producer participants to the desired risk management results) (Resources that will strengthen producer achievement of risk management results) Logic Model Implementing Forward INPUTS OUTCOMES OUTPUTS Using the Logic Model to Identify Results/Outcomes & Create a Plan of Work (Project Steps) Planning Backwards – Beginning with the End in Mind Results/Outcomes Outcomes Activities/Tasks Outputs Responsible Entities & Roles Inputs Implementing Forward Responsible Entities & Roles Activities/Tasks Inputs Outputs Results/Outcomes Outcomes Post/Short Term Follow-up Long Term or 3-6 Month Follow-up Project Steps Production Risk Management Outputs or Outcomes? • A series of insurance workshops will be delivered to producers • A crop/livestock insurance spreadsheet will be developed • Producers will increase purchases of crop/livestock insurance • Producers will evaluate crop/livestock insurance decisions Production Risk Management Outputs or Outcomes? • A series of insurance workshops will be delivered to producers - output • A crop/livestock insurance spreadsheet will be developed - output • Producers will increase purchases of crop/livestock insurance – w. outcome • Producers will evaluate crop/livestock insurance decisions – s. outcome Marketing Risk Management Outputs or Outcomes? • Producers will attend a marketing workshop • A marketing curriculum will be written • Producers will implement a marketing plan • Producers will write a marketing plan Marketing Risk Management Outputs or Outcomes? • Producers will attend a marketing workshop – output • A marketing curriculum will be written - output • Producers will implement a marketing plan – s. outcome • Producers will write a marketing plan – w. outcome Financial Risk Management Outputs or Outcomes? • Producers will calculate financial ratios for case farms/ranches • Producers will understand and interpret financial ratios • Producers will evaluate the financial ratios for their farm/ranch Financial Risk Management Outputs or Outcomes? • Producers will calculate financial ratios for case farms/ranches - output • Producers will understand and interpret financial ratios – w. outcome • Producers will evaluate the financial ratios for their farm/ranch – s. outcome Legal Risk Management Outputs or Outcomes? • Producers will attend a farm program educational meeting • Producers will successfully analyze farm program signup decisions • Farm program fact sheets and calculators will be produced Legal Risk Management Outputs or Outcomes? • Producers will attend a farm program educational meeting - output • Producers will successfully analyze farm program signup decisions – s. outcome • Farm program fact sheets and calculators will be produced - output Human Resources Risk Management Outputs or Outcomes? • Succession planning materials will be published • A series of programs will discuss the components of a farm succession plan • Producers will develop a succession plan • Producers will implement a succession plan Human Resources Risk Management Outputs or Outcomes? • Succession planning materials will be published - output • A series of programs will discuss the components of a farm succession plan - output • Producers will develop a succession plan – w. outcome • Producers will implement a succession plan – s. outcome Exercise – Writing Effective Outcomes 1. Identify the risk situation(s) and the risk topic areas that you want your project to address 2. Build a set of measurable risk management results/outcomes Include both short and long term outcomes (results that will happen after the project – within 3 months to a year) 3. Determine if the outcomes are a good fit for the target audience Target Audience Are Outcomes suitable for audience being served? Realistic goals that can be achieved within specified timeframe? Linguistically and Culturally Appropriate? Learning Styles and Environment? Reliability of Information Science based content? Appropriate for outcomes being proposed? How well do you know your target audience? Relationship Capital? Evaluation Methodology Measuring Outcomes Evaluation Mechanics Protocols for Administering Evaluations – Preparing Workshop Facilitators • Evaluation Procedures Audience Preparation Methodologies – Pre/Post/Follow-Up – Ease of design and implementation • Methods suitable to audience • Framing questions appropriately to know if outcomes have been achieved. • Questions designed to measure outcomes should flow/follow from the hierarchy of results/outcomes. • Qualitative and Quantitative – Open and close ended questions Gathering informal data - Producer Stories and Testimonials Evaluation Mechanics Tabulating the results Who will collect the data? How will data be collected? When will data be collected? What will be done with the compiled data? These questions need to be addressed before administering your evaluations and are an integral part of program design! Evaluation Mechanics Suitable Question Formats – Multiple Choice – Yes/No – Rating scales – Open ended Simple for respondent to understand Easier to analyze for project teams with less evaluation experience Evaluation Mechanics Multiple Choice – Yes/No 1. Assign a numeric value to each question response For example code “yes” responses as “1” and “no” responses as “2”) 2. Use the “count” function in Excel to analyze the responses Pie Charts can be used to graphically depict responses Evaluation Mechanics Rating Scales – Allow respondents to indicate level of agreement with a statement or to rate level of importance. Straightforward and easy to understand Easier to analyze than other question formats (because responses are already in numeric form, it is not necessary to recode them) Shows how respondents feel on average about the question being asked. Evaluation Mechanics Open ended questions – Invite more detail from the respondent – Require a little more effort from respondent Can identify challenges/barriers to progress Provide a complete story or testimonial – Analysis may require identifying common themes Evaluation Mechanics – Add yes’s and no’s – Add up ranking #’s – Provide an average of answers, rankings, or range – Organize comments into similar patterns – Report results should be structured to show that participants goals/outcomes are being achieved. • Numbers first, percentages second Keep it simple! NIFA Metrics for Measuring Outcomes Testing your methods - Will your evaluation tool(s) clearly demonstrate progress participants are making towards achieving the outcomes? - Changes in Knowledge? Changes in Attitude? Changes in Behavior/Approach? Changes in Economic or Social Circumstances? Pre Course Evaluation Purpose 1. Benchmark info on knowledge levels 2. Information on participant background, past experience, attitudes and needs for program managers and instructors Combining Pre/Post into Post • “Please rate your knowledge PRIOR to attending today’s workshop” • “Please rate your knowledge AFTER attending this workshop” • “Please rate to what extent you have the knowledge, tools and resources needed to …” • “I better understand” Likert Scale or Similar Method Post Course Evaluation Purpose: Measures Knowledge Gain Attitude Change Actions participants expected to take as a result of attending course Post/Short Term Follow-up Follow-Up Evaluation Purpose: To see what actions participants have taken as a result of attending your course and how these actions have led to economic and other benefits. Long Term or 3-6 Month Follow-up 3 – 6 Month Follow-Up 3 – 6 Month Follow-Up Cont. Ease of Design Do evaluation questions flow logically from sequentially stated outcomes? Asked in a manner to obtain specific information on level of accomplishment or barriers encountered? Build on progress/commitments made from short term or post evaluations? Usability of information? Ease of Design Avoid “respondent fatigue”? Use definitive language to obtain accurate responses? Avoid words like “generally” and “usually” – i.e. rather than “Do you generally follow a business plan?”, “Do you follow a business plan?” Keep it short and simple! Exercise – Writing an Effective Evaluation Post – Short Term Follow-Up – Long Term Within 3 months to a year Evaluation Tools Variety of Methods: – Pre/Post Tests • Audience Response Systems (Clickers) – Follow-up Evaluations (3 Months – Year) • Mail or Internet Survey – – – – Telephone Interview Face-to-face interviews (in-person survey) Dot survey Informal Interview Evaluation Tools Considerations: Time Frame Budget Audience characteristics Results/Outcome Based Reporting • Documents: – Producer Progress towards achieving goals (short and long term) – Producer Stories and Testimonials – Unexpected results/outcomes • Provides Access To: – RME Curriculum and resource materials • Training methodologies • Evaluation Tools Integrating Results/Outcomes into Project Reporting Summarizing Outcomes Project Overview Actual Results/Outcomes Comments/Stories/Examples of Success Input into other reports Impactful Reporting • Builds capacity for educators who work directly with farmers and ranchers through… • Access to risk management curriculum and resources • Access to training methodologies • Access to successful evaluation methods and tools • Creates transparency • Improves accountability • Documents in a database format risk management outcomes/impacts for farmers and ranchers. • Available to federal and state agencies, government offices such as the GAO and OMB, institutions and other funding partners. • Keeps moving the bar higher for the effectiveness of risk management education. • Evaluation Best Practices Measuring results/outcomes successfully: • Choose methods that balance best practices with available resources – Combination of direct and indirect measures • Direct measures (assignments, action plans etc.) yield stronger evidence of change in knowledge and beneficial practices, but are more time intensive and may take more expertise to evaluate. • Indirect measures can give instant feedback but may not provide real evidence of knowledge gained. • Qualitative measures (like phone interviews or focus groups) that may be more resource intensive could be accomplished through a representative sample. Evaluation Best Practices Measuring results/outcomes successfully • A randomly sampled group, representing approximately 5 percent of participants submitting a post-meeting evaluation, will be surveyed by phone. Participants will be asked whether or not they have completed and implemented their farm/ranch transition and personal estate plan. They will be asked what challenges and obstacles they encountered during the process, how the program assisted them in the process and how useful the resource workbook was to them. Evaluation Best Practices Measuring results/outcomes successfully – Direct Interactions with Producers can increase the chance that participants will implement beneficial risk management practices. – Good follow-up can lead to collegial relationships that may bring results beyond the initial results/outcomes that were initially projected (Unexpected results). – Good program design allows for learning and application to take place in linked stages within a timeframe that encourages the implementation/adoption of longer term results/outcomes. Evaluation Best Practices • Interactive discussion on best practices – Successes – Pitfalls *Handout In Summary If the “Return on Investment” of grant dollars awarded are the risk management improvements that producers will make… Then measuring producer results (outcomes) becomes an integral component of program planning and design, not an afterthought! Thank You! Bob Craven, Director – Digital Center, University of Minnesota rcraven@umn.edu Brad Lubben, Director – North Central Region, University of Nebraska-Lincoln blubben2@unl.edu Jo Ann Warner , Assistant Director – Western Region, Washington State University warnerj@wsu.edu Kevin Klair – Digital Center, University of Minnesota kklair@umn.edu Kynda Curtis, Utah State University kynda.curtis@usu.edu Gary Hachfeld, University of Minnesota Extension hachf002@umn.edu Results/Outcome Resources The Rensselaerville Institute: http://whatworks.org/displaycommon.cfm?an=1&subarticlenbr=266 “Outcome Funding: A New Approach to Targeted Grantmaking” Based on the premise that grant makers are really investors in human gain, Outcome Funding addresses the key questions that the funder as investor should ask: What am I buying? What is the probability I will get it? Outcome Funding begins with an abrupt and timely challenge to the standard wisdom of the Request for Proposal process. The authors make the point that very little of the traditional proposal content addresses results, favoring explanations of process instead. This book is not only persuasive on the need for change; it is clear and practical on how to achieve it. Logic Model Templates: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodelworksheets.html Developing a Basic Logic Model For Your Program: http://www.publichealth.arizona.edu/chwtoolkit/pdfs/logicmod/chapter2.pdf I have become convinced that it makes a considerable difference if you do the outcomes before planning the activities. I definitely advocate doing the outcomes first! I find that people come up with much more effective activities when they do. Use the motto, “plan backward, implement forward.” Beverly Anderson Parsons, WKKF Cluster Evaluator