John Muir Trust (HGIOTSO case study)
The John Muir Trust was one of a number of organisations involved in piloting the
How good is our third sector organisation? (HGIOTSO) framework during its
development.
The John Muir Trust, John Muir Award
The John Muir Trust is a conservation charity
dedicated to protecting wild places. Its
engagement initiative, the John Muir Award, is
an environmental award scheme that
encourages people to connect with, enjoy, and
care for wild places.
Over 1000 organisations across the UK (600 in Scotland) use the John Muir Award
every year to encourage their participants to explore outdoors, take responsibility
for nature, and share what they do.
These organisations (Award Providers) range from schools, community learning
and development teams and health improvement programmes to outdoor centres,
youth clubs, universities and prisons. They support the involvement of over 30,000
people per year.
Why did we get involved in the HGIOTSO? pilot?
We had evaluation and impact assessment on our John Muir Award operational
plan and HGIOTSO was recognised to be a useful tool to help with this. The
framework created by Education Scotland is well recognised, has credibility, and
has a track record in schools and CLD (ref. other How Good Is Our.. documents).
It is also a resource that many of our Provider organisations may use in the future through direct experience of using it ourselves we will be better placed to support
our Providers. The opportunity to have extra support from Education Scotland
during a pilot phase was also appealing.
What did we do? And how?
After familiarising ourselves with the draft HGIOTSO document and the
Overarching Framework (OF) we chose to focus our pilot involvement on ‘How
well do we meet the needs of our stakeholders?’ - one of the six themes in the OF.
This tied in with our strategy to assess impact.
We looked specifically at Quality Indicator (QI) 2.1– Impact on the people we work
with.
Our staff team examined the Challenge Questions and Evidence sections of the
QI. This process was informal and encouraged reflection on our current practice,
highlighting areas for potential improvement. There was no expectation of formal
reporting from Education Scotland; we decided how to record our reflections in a
way that was useful/beneficial to us.
This process offered reassurance that we are already doing a lot of effective work
towards understanding how the John Muir Award impacts on the people we work
with. It also encouraged us to create ways to explore this further: we created a
Provider Survey, and invited all Award Providers in Scotland to respond via a
Survey Monkey questionnaire or a face to face/phone chat with an Award staff
member (the support and advice from Education Scotland in setting this up was
valuable). We had 66 responses, a rate of just over 10%.
What did we learn?
The Provider Survey has allowed us to answer some of the Challenge Questions
more fully. It takes our understanding (and recording) of the impact of our work
beyond anecdotal stories and quantitative data. It has helped us to further align
our work with Scottish Government National Outcomes with credible evidence.
The full analysis of information gathered is yet to be completed. This will be written
up in July ’15. It will include summaries of what we are doing well and what we
could do better to meet the needs of the people we work with. Top line information
at the moment (which we have interpreted in relation to SHANARRI indicators)
shows that the most common impact of the John Muir Award is on helping
participants be Included, Achieving and Active (appendix 1).
We also learned that a significant amount of respondents (44%) feel that using the
John Muir Award has a positive impact on their teaching/group work practice.
Our experience of piloting HGIOTSO? has been very positive. It is an easy-to-use
resource. It is wide ranging and thorough, yet not overwhelming to understand or
administer. It focuses on evaluation for improvement, which we value. We are
keen to continue to use HGIOTSO? in future years, and to self-evaluate other
themes in the resource (strategic leadership, capacity for improvement etc). We
definitely see this as an ongoing piece of work to be built into our practice.
Appendix 1
The pie chart is the summary answers for six questions in the Provider Survey, all
relating to SHANARRI indicators. We asked people ‘does the John Muir Award
help the people you work with be…Healthy? (etc). People answered on a scale of
1 – 5, from 1: not at all to 5: a great deal. The graphic shows the percentage of
people that answered 4 and 5. We didn’t ask about ‘safe’ and ‘nurtured’ (hence the
empty spaces), as we didn’t see a clear fit with our work. Respondents also had
the opportunity to comment on their rating and expand on the question (currently
being analysed).
We are aware of attribution and know that the majority of the answers do not relate
to the John Muir Award in isolation, but to activity that the John Muir Award was
delivered as part of.
The figures are from a response rate of 66.
Other outcomes to note so far include –
Forty-four per cent of respondents made specific reference that the John Muir Award made
an impact on their teaching/leadership practice by giving confidence/opportunity to
use/introduce new/different learning approaches/skills/activities.
Many respondent s commented that it is difficult to say whether the John Muir Award is
having a positive impact on their participants specifically. It is challenging to draw it out
beyond the impacts of their outdoor activity more widely. Attribution is always a challenge
for us, and this survey has highlighted this, and that it is recognised by Providers.
Some respondents highlighted that it is difficult to evaluate the impact of the John Muir
Award over a longer period of time (beyond the end of direct engagement with a programme
of activity). This is an area for potential development in evaluation in the future (following on
from impact work carried out in 2006-2008).