Measuring what Matters Workbook
{name} Big Local, December 2014
Draft workbook
Early Years Evaluation Team
2014
Contact the team on 020 7520 2548 for more information or by email at biglocal2014@ncvo.org.uk
Introduction
Evaluation is where you use information to make judgements on how your programme or project is doing; it’s about
assessing the value or worth of what you’re doing. It can help you in three main ways:
Evaluation involves four stages:
1.
2.
3.
4.
planning (what will success look like, what do you need to measure to understand how you’re doing?)
collecting information (gathering the evidence you need)
analysing it (figuring out what the evidence tells you, assessing how you’re doing)
using it (to learn, to improve, to share and celebrate your successes)
This “Measuring what Matters” workbook focuses on the first stage of evaluation – that is, planning. In it we share our
ten top tips for Big Local areas to plan how they could assess how they’re doing.
1

Plan and commit
Plan ahead and see evaluation as an ongoing process to help track your journey as well as your
destination. Commit to reviewing, reflecting and being honest about what works and what doesn’t so
that you don’t just prove but also improve as you go along!
Plan ahead so that you can learn as you go
Evaluation isn’t just about measuring success, it’s also about learning and helping you improve. Big Local is trying
something different. Sometimes it’s about taking risks and trying new ways to tackle old problems. This means we’re all
learning together about what works best, so it really helps to build into your plans time to review and reflect on how things
are going and what you’re learning.
Remember you can do a lot yourself
Sometimes people worry that evaluation is “best left to the experts”. This isn’t necessarily true. It’s important to value the
skills and knowledge you already have – listening skills, observation skills, some good story-tellers, and a lot of local
knowledge will get you far. Even better if your group has someone who likes to ask difficult questions - “but why?”, “but
how?”, and “so what?” are the basis of some of the most interesting evaluation questions!
Planning ahead can save headaches later
Planning ahead can save you headaches later because it’s really common when people aren’t
sure what information to collect, to then collect either too little information, or too much, or
simply not the right information you need to answer your important questions.
2

Define success clearly
Clearly define success. You can’t measure what you aren’t able to define.
Success in
what we’ve
done
Our activities –
what did we do,
was it what we
intended, have
we met our
targets?
Success in
making a
difference
What difference
have we made?
to whom? was
it what we’d
hoped?
Success in
how we’ve
worked
Other things
that matter to
us?
Did we stay true
to our values?
eg, have we
been inclusive?
What else
matters to us? –
eg, what people
feel about what
we do? making
our million go
further?
Is there anything else we want to consider when we think about what success will mean for us?
3

Define clear changes or results
Describe your intended changes clearly and simply. Try to get to the heart of them. Break complicated
changes down and use single, simple change statements to make things easier to measure.
Residents will report an increased confidence in accessing job
readiness and employment support services and a more
positive outlook on improving their financial situation through
work, having been assisted through a new local high-street
service that is friendly and community-led.
What would you measure in this example?

To make things easier it helps to express the differences you want to make as single, simple statements; to break big
changes down; and to try to get to the heart of the change so it’s clear what’s most important.
Activity
We will set up an informal,
friendly job club in our Big
Local Hub to support longterm unemployed
residents

Activity
Change
Members of the
community will help us run
the job club, sharing their
time and skills with others

Out-of-work residents who
use the club will feel more
confident about seeking
support to find
training/employment

Change
Out-of-work residents who
use the club will feel more
positive about their
chances of finding work.

4

Have a story that works
Have a story or map that explains the logic of your journey - a place where you’ve laid out clearly your
assumptions about how what you do will contribute to the results or changes you want to achieve.
It’s important to make sure there are no steps missing – sometimes the steps are the things you want to measure.
We will offer youth nights at the
community centre
We will offer group work
sessions during youth nights
at the community centre
Young people will be more
employable
Young people will improve
their social and
communication skills

Looking at your local story map, ask yourself:
(1) Does this tell our story? What would you add to tell a better story?
(2) Does the story ‘work’ – is there a logic to it - or are there steps (changes) missing?
Young people will be more
employable
5

Prioritise what matters most
Be realistic and prioritise what matters most. You may not be able to collect information about
everything you do as often as you’d like. Prioritising can help you collect less but collect smarter!
You can’t measure everything
Big Local is a long-term programme. A lot will be done and hopefully a lot will change. You can’t measure and report on
all of it. It helps to prioritise and focus on what are the most important elements of your Big Local story. To decide on
what matters most, it usually helps to focus on your goals.
Focus on what most helps you achieve your goals
Often when asked to measure their progress, people focus on how much they’ve done, what activities they’ve done.
That information is really important, and you may find it’s what a lot of people are interested in – because telling others
how many events you’ve held or how many people attended these gives a very tangible way of showing that you’re
getting things done. However, you will also (and more importantly) want to know if your activities are moving you closer
to your goals, to making the differences you want to make.
Be realistic and try to be proportionate
Practical considerations also shape what you measure.
 Some changes aren’t in your direct control or might happen after your work has ended. Focus first on the changes
that are in your control.
 Not all the changes you make may be equally important.
 Some things may simply be too difficult or time-consuming so you need to consider the next best thing that you can
realistically measure, remembering that what you do to measure your success shouldn’t get in the way of doing the
work you want to do.
6

Ask ‘how will we know’ questions
When you’re thinking about assessing whether you’ve made a difference, it helps to think in advance
about the signs of the change you could look for. Asking yourself ‘how will we know?’ for each
change really helps.
Asking “how will we know questions” can help you think about the signs you might see, things you could count, or things
people could tell you that might help you evidence the difference you’re making. For instance … say one of the big, longterm changes we want to make is that we want to improve our local environment so that the whole community can benefit
from it. We have a large park on our doorstep but it is under-used for a variety of reasons. We broke down our big
change (improved local environment) into three specific changes that felt most important to us. We said, “We will know
we’ve made progress with this change if ...
 Our park is cleaner
 Our park is safer for families to use
 Our park is better used by the whole community
Then we asked, “how will we know these changes are happening? What will we count, measure or observe?” We ended
up with the beginnings of a plan as below.
Big Change: Improved local environment for the whole community’s benefit
Specific changes
Our park will be cleaner
How will we know?
What will we count, measure or observe? What might others tell us?





Level of littering 
Level of graffiti 
Level of complaints 
Level of use of dog litter bins 
How clean people feel the space is 
7

Be lawyers not scientists – building a case
Think of yourselves as lawyers (collecting evidence to make the strongest case you can), rather than
scientists (collecting proof). To make the strongest case it helps to try and measure things from more
than one angle – using different types of evidence or perspectives to explore the same thing.
When we talked about signs of the change, we mentioned you might look for things you can see, things you can count or
measure, or things others might tell you. All three could be valid and if you combine one or more you could make a
stronger (more credible) case. Likewise you could think of assessing change from three different viewpoints to make a
stronger case.
eg, observation, film,
photos
what you see or
feel is
happening
eg, feedback forms,
testimonials, interviews,
case studies,
questionnaires
your change
what others
tell you
what you can
record/count
eg, monitoring, official
records, surveys
8

Recycle and re-use
When it comes to collecting information or evidence there are many toolkits, ideas, questionnaires and
forms you could use, and there may also be others collecting information you need which would save
you collecting it yourself.
Is there an easier way? Has someone else done this before?
Always check if there’s an easier way to collect the information you need. This way you can avoid reinventing wheels,
collecting information that’s already out there somewhere, or duplicating what someone else is doing. Check online or
with other organisations working in your area about how they collect information. There are lots of great questionnaires,
forms, ideas out there, many of which you can find by searching online. You can often adapt what someone else has
tried and tested to collect information you need.
Has someone else got this information or could they collect it for you?
Be a recycler! It’s also worth checking what information is already being collected about your community and seeing if
you can use it. Your local council, NHS and voluntary organisations and schools may have really useful information you
could use. Also, sometimes if you have a vital question you want to ask your community, one of your partners may be
willing to add that to their own survey or forms and to share information with you.
Note of caution – do a test run first!
If you are using or adapting someone else’s questionnaire or feedback form or using questions others have found helpful,
always test them first with a few people from your target audience. What works in one setting or with one group of
people, may not work in another.
9

Think about how you’ll use information before you collect it
Always think ahead to what you’ll do with information BEFORE you collect it, and try not to collect
information unless you know you will use it.
Think in advance about how you’ll store information you collect
Don’t collect personal information if you don’t need it. If you do, make sure you’re able to keep any personal information
in line with data protection guidelines.
Think in advance about how you’ll analyse information you collect
Remember that once you’ve collected information you have to be able to analyse it and use it. For instance, a survey
might seem a great idea until you realise you don’t have time to enter 300 questionnaire responses into a computer, and
you’re not sure how to make sense of all the questions people answered in words instead of numbers!
Think in advance about how you’ll use information you collect
It doesn’t make sense to collect information you won’t use “just in case it might come in handy one day”. Think carefully
about how you want to use information in advance or you could end up collecting information you can’t use because you
have too much of it and can’t make sense of it, because you collected it too late to be of use to you, or because it doesn’t
really answer your questions.
10

Make the most of being part of a national programme
Remember you’re part of a national programme. Local Trust and its national partners can support you
in measuring your progress in a number of ways.
Guidance, toolkits and case studies
 The Big Local Annual Review process includes tools to help you review how well you’re doing and to help you
assess yourself as a local partnership.
 Check Local Trust’s website or contact the team for a copy of the “Big Local: Checking Your Progress” guide or
the equally useful “Checking Your Progress” toolkit which contains lots of ideas and some useful sample
documents
 There are lots of Big Local case studies on the Local Trust website to inspire you and give you ideas for how you
too could use stories to share what you’re doing and the progress you’re making.
On the horizon
 Local Trust plans to set up evaluation workshops to help areas explore more of the “how to” side of things.
Training on measuring impact may also be made available in the near future.
 The School of Public Health Research is currently researching the links between community control and health and
wellbeing. In time they hope to share useful data and ideas on how areas could look at the impact of Big Local on
health in its broadest sense.
 The Early Years Evaluation Team will be producing a database of changes and indicators (signs of change) that
areas could visit to look at what others are doing and to get ideas for how to measure their changes over time.
11
Appendix 1. Reminder of our ten top tips for measuring what matters

Plan and commit to ongoing reflection and review

Define success clearly

Have clear changes and break them down where you can

Have a story that works – that links your activities and your results

Prioritise what matters most

Ask ‘how will we know’ questions

Be lawyers not scientists – build a strong case

Recycle and re-use

Think about how you’ll use information before you collect it

Make the most of being part of a national programme
12
Appendix 2. What a measurement plan might look like for a long-term goal