Project Evaluation Plan – Guidance and Templates This document has been developed to help you create an Evaluation Plan for your Nominet Trust project. It should take about an hour to go through the steps outlined below and create. Project evaluation. You’re working hard to get your project off the ground; surely worrying about evaluation is something that could wait for a bit? Well, unfortunately, no it isn’t. The best project evaluations are those that are planned at the very outset. If you have a clear vision of what success looks like to you, planning for evaluation will enable you to capture all the relevant information throughout the project lifecycle that you need to demonstrate the impact you have made. There are a variety of ways that you may choose to evaluate your project, ranging from self reporting to external evaluation. While the Nominet Trust welcomes its partner projects using existing evaluation approaches, we are equally committed to exploring new and alternative ways to understand social impact. Whatever the approach, we emphasize the importance of understanding what changes have resulted from your work, and what we can learn about the model/approach you have used to achieve these changes. Background to Monitoring & Evaluation Project monitoring and evaluation should have a clearly defined workstream in your plan, with resource allocated to it. There should also be a line item in your project budget specifically relating to monitoring and evaluation, with a minimum of 5% of your project budget allocated to it, so that finance is ring-fenced to allow you to deliver against these requirements in your Funding Agreement with us. If you intend to use external organisations to perform your end of project evaluation, you may need to allocate additional funding to this workstream. During your project, we expect you to complete our online Project Monitoring forms at an agreed frequency (usually quarterly). At the end of your project, you will be asked to complete our online Evaluation form. Time invested in developing your Evaluation Plan at the start of your project will allow you to deliver your monitoring and evaluation obligations more effectively. Your Evaluation Plan Before your project starts, we ask you to submit a two-part evaluation plan that tells us: How you will measure the changes that result from your work (the outcomes) How you will reflect on the benefits of your approach Practically this means: 1. Telling us what outcomes you are choosing to measure, and how you will measure them during the course of your project 2. Identifying assumptions about your approach, which you will reflect on at the end of the project This guide offers some step by step guidance to creating your evaluation plan: Step 1 looks at how to measure outcomes. Step 2 looks at identifying assumptions. At the end of each step is a template to help you create your plan. We highly recommend getting in touch with Nominet Trust’s Development Research team to help you create your evaluation plan. We are always happy to speak on the phone, online or in person. If Page 1 of 5 you would like to discuss any aspect of your project evaluation please contact us on developmentresearch@nominettrust.org.uk To view a presentation and read more about our evaluation approach, follow this link to the Nominet Trust website: http://www.nominettrust.org.uk/how-to-apply/guidance-notes/evaluating-your-project Step 1 – Measuring Outcomes As a social investor we’re interested in measuring ‘outcomes’, meaning the changes that happen in society as a result of your project. There’s more detail below about how to identify your outcomes. Deciding how to measure your outcomes can seem daunting at first, so we suggest breaking it down into 5 steps: 1. 2. 3. 4. 5. Identify your outcomes Decide what you will use to measure your outcomes Choose how to measure them Choose when to measure them Decide who will measure them 1. Identifying your outcomes We recommend that you identify around three to five outcomes for your work. To do this, think about what the main changes are that you want to see take place as a result of your project. This might include a change for project participants or beneficiaries (e.g. increased skills or confidence in an area), a change in the community, or even a change in the behaviour or an organisation or nation. To begin with, it is important that you distinguish between the things that your project will produce - for example a website, teaching resources or training sessions - and the changes (or outcomes) that these will effect in the world. The things your project produces tend to be called ‘outputs’ (or deliverables). It can be helpful to start by identifying your outputs, as these drive the changes that are the outcomes. For example, a project might support young people seeking employment by training them to find work online. The ‘output’ might be “running 10 training workshops for a total of 100 young people”. The hoped for ‘outcome’ might be that “young people who complete the course are more confident about searching for work online”. It is important to separate these out because evaluation is about measuring the changes that have happened as a result of a project (the outcomes) not just the things it has produced (the outputs). Someone may have run 1000 training workshops, but we only know it’s a successful project if they have achieved the outcomes they identified at the outset. We’ve provided a table below to help you identify outputs and outcomes. If you are finding it difficult to think about outcomes, it can be useful to list all the major ‘outputs’ of your project. And then think about the outcomes that these ‘outputs’ are trying to achieve. So, in our example project, there might be an entry “running 10 training workshops with 100 young people” in the output column and an entry “young people who complete the course are more confident about searching for work online” in the outcomes column. 2. Identifying what you will use to measure your outcomes Once you have identified your project outcomes, you need to think about what you will measure to know if you have achieved them. In our example “young people who complete the course are more confident about searching for work online” we will want to measure their confidence about searching for jobs online. Alternatively, if your work is about internet safety, you might want to measure if people are demonstrating Page 2 of 5 safer behaviours online, and so on. Look at each of your outcomes and decide what you will need to measure to know if there has been a change. 3. Choosing a way to measure your outcomes Now you know what you are measuring, you need to think about how you will measure it. This could be done through a range of different approaches depending on your project; for example interviews, questionnaires, or usage data from digital resources (such as the number of people using a website over a set time period). You may decide that giving a questionnaire to each participant and asking them to report on their progress is enough. Or you might decide to use other information to help, such as feedback from other participants in the process e.g. the trainer running the workshop. In our example of young people engaging in their job searching course, we might give them questionnaires to find out if they feel more confident about finding work online when they have completed the course. There are many scales and tools to choose from when measuring outcomes. Please contact the Development Research team if you are unsure about what would be the best way to measure outcomes. 4. Choosing when to measure it Now you know what you are measuring and how you are measuring it. The next decision is when you will measure it. Remember, if you want to know whether a change has happened, you have to measure the same thing at the start and at the end. In our example project, if we just measured young people’s confidence in searching for jobs online at the end of the project it might be very high. Great, it seems - but of course they may have been very confident at the start of the project and we’ve not made any difference! So, you may decide to just measure once at the start of the programme and once at the very end. Or if you are using a different method of evaluation it may be that you are getting feedback throughout. 5. Deciding who will measure it Lastly you need to decide who will be accountable for ensuring measurement takes place consistently. This breaks down into 2 areas: who is collecting the outcomes information, and who is analysing it. Of course this could be the same person if it is a small project; but may be different people if you have a bigger team. In our example project, the tutor ‘owns’ the job of giving out and collecting the questionnaires from the participants at the beginning and end of the project. The project manager ‘owns’ the job of analysing the results and reporting them. This is a really important step as it makes visible who has responsibility for each part of the evaluation process. Outcomes Template The table below provides a template to help you document your project outcomes. Copy it into a separate file so you can return the completed form to us as part of your project set-up process. You can list the project outputs if it helps you, but don’t worry about writing these down in detail as these will already be on your project plan. Project Name: List the project outputs / deliverables (optional) EXAMPLE: Running 10 training workshops with 100 young people List the intended project outcomes What do you intend to measure? How do you intend to measure these? Approximately when will this be measured? Who will measure it? EXAMPLE: Young people who complete the course are more confident about searching for jobs online EXAMPLE: Change in young people’s confidence in searching for jobs online EXAMPLE: Questionnaire before and after the programme asking them to rate their confidence on a EXAMPLE: At the start and end of every training programme EXAMPLE: Tutor hands out and collects forms; project manager collates them and reports Page 3 of 5 scale Step 2 – Identifying Assumptions We are always very interested to understand what models work to address social challenges. A core part of this involves understanding the assumptions that underpin a project. We would like you to identify three to five assumptions that underpin your project. These are core beliefs you have about why your project will work. In our example project, the programme creators believe that young people who are out of work will be able to find jobs online. Another assumption is that young people will be interested in searching for jobs online. A further assumption might be that young people will have access to the internet after they finish the course. At the end of the project we will ask you to reflect on whether these assumptions were borne out by your work. The table below provides a template to help you document your assumptions. Copy it into a separate file so you can return the completed form to us with your outcomes template as part of your project set-up process. Project Name: List of project assumptions EXAMPLE: Young people being taught how to access job sites will help them get into employment Final Thoughts We appreciate that coming up with an evaluation plan is not always straightforward, and sometimes it just really helps to talk through your ideas. That’s why we recommend getting in touch with us to help create your evaluation plan. We are always happy to speak on the phone, online or in person. Just drop us an email on developmentresearch@nominettrust.org.uk Page 4 of 5 Evaluation Plan: [project name] Summary Please summarise the approach EXAMPLE: Project Outcomes List the project outputs / deliverables (optional) List the intended project outcomes What do you intend to measure? How do you intend to measure these? Approximately when will this be measured? Who will measure it? EXAMPLE: Running 10 training workshops with 100 young people EXAMPLE: Young people who complete the course are more confident about searching for jobs online EXAMPLE: Change in young people’s confidence in searching for jobs online EXAMPLE: Questionnaire before and after the programme asking them to rate their confidence on a scale EXAMPLE: At the start and end of every training programme EXAMPLE: Tutor hands out and collects forms; project manager collates them and reports Project Assumptions / Research Questions List of project assumptions EXAMPLE: Young people being taught how to access job sites will help them get into employment Page 5 of 5