South West Spoke
Final Case Study Report Template
IMPORTANT: The SW final report guidance document must be consulted
before completing this report template. Please complete all sections.
Project Title: Demonstrating proven, effective outreach activity through
effective evaluation
Project Leader: Sophie Duncan
Department/School: N/A
Institution: National Co-ordinating Centre for Public Engagement
Other institutions/organisations involved in the project: Beacons for Public
Engagement (Manchester Beacon; CUE East; Beacon for Wales; Edinburgh
Beltane.)
Abstract:
This project created training resources to support HE STEM practitioners to develop their evaluation
work in particular, of STEM outreach activities. By developing their understanding and skills in
evaluation, practitioners would be able to improve the effectiveness of their projects, and demonstrate
this to others. This was prompted by a range of dravers: the access agreements universities provided
need to have evidence of effective outreach practice; a need to justify the funding spent on ourreach
in terms of effective outcomes from the work and a desire by many in the sector to evidence the
impact of their work.
Informed by a survey of current practice within the sector (with 161 respondents), three training
courses were developed. The Beginner’s Guide to Evaluation was aimed at those with little
experience of evaluation, and sought to build confidence and skills in developing an evaluation plan.
The Evaluation Masterclass was for those with more evaluation experience, who wanted to better
understand how to make an impact with their evaluation. The Training the Trainer course sought to
support trainers to run their own Beginner’s Guide to Evaluation course within their institution. Finally,
three one hour Plug and Play sessions were developed that could be integrated into conferences and
other meetings. These covered: In-house Evaluators; Use of Logic Models; and Measuring Quality.
The courses were supported through an online Ning site where participants could download all the
resources and ask questions of the course trainers. You can find out more at
www.publicengagement.ac.uk/evaluating-stem-outreach
List of Outputs:

Getting Started: A Beginner’s Guide to Evaluation (BG2E). A one day training course to
support participants to develop their evaluation work
(2 pilot short sessions with 30 participants; 3 one day sessions with 34 participants)
1
South West Spoke

Evaluation Masterclass: Making use of your evaluation: Informing practice and evidencing
impact (EM). A 3 hour masterclass to support participants to make use of their evaluation
(3 Masterclasses with 47 participants)

Training the trainer session (TtT): 3 hour sessions to support HE trainers to make use of the
resources to deliver the BG2E course
(3 TtT sessions with 19 participants; 1 training session still to run)

An online Ning site to support people who have participated in the courses and provide
resources to support their work (http://nccpetraining.ning.com/)

Three Plug and Play masterclasses (PP): One hour sessions to explore specific aspects of
evaluation (these will be run by 30 September 2012 – with around 30 participants in total
across the 3 sessions)

Resources to support each of the courses and course participants to either develop their
evaluation practice or to run the BG2E training in their own institution. Resources include:
o
o
o
o
o

Powerpoint of each of the three courses
Trainer handbook for BG2E
Participant handbook for BG2E
Activities for each course
Handouts and resource sheets
A report based on feedback from HE STEM practitioners on their current evaluative practice;
and the support they would like to have to improve their evaluation work
Project Highlights:
1. The Beginner’s Guide to Evaluation course aimed to build knowledge and confidence for HE STEM
practitioners to develop their evaluation work – the evaluation of the course clearly indicated that the
course delivered this, and much more. It was great to hear people reflecting on the increase in
confidence they now had to develop an evaluation plan for their work.
2. The Evaluation Master classes provided a great opportunity for people to discuss more detailed
evaluation practice. The evaluation indicated that people really valued the opportunity to come
together to discuss impact and reporting, with the interactive nature of the course was appreciated.
3. The impact agenda is encouraging more people to evidence the impact of their research – many of
the resources developed as part of this project are equally relevant to a wider audience of people
seeking to develop their evaluation practice. This is particularly true of the Evaluation Masterclass. In
addition we have had interest from HE staff in Europe who are also tackling these questions.
Background and Rationale:
Evaluation and impact are a key part of developing good practice. The project was developed to
support HE STEM practitioners to develop effective engagement with schools and colleges through
2
South West Spoke
science communication and outreach.
In developing effective and appropriate methods of capturing the benefits and impact of STEM
outreach, academics will be able to identify successful activities (and thus target expenditure more
effectively), and demonstrate to senior managers the value of the activities. They will also understand
the importance of reflecting on the purpose of their activities and on assessing how effectively the
activities deliver against that purpose, with a view to improving the activities that they undertake.
The project aimed to support the practical application of evaluation tools within STEM subjects across
a number of different HEIs and, through training and an online forum, to enable staff to develop and
utilise evaluation methodology to: improve their own practice; demonstrate impact; increase
awareness amongst senior managers and enhance recognition for outreach activities. The train the
trainer methodology proved an efficient way of building capacity and ensuring wide uptake of the
resources.
Implementation:
There were several strands to this work:
1. Survey to assess the need for evaluation support and training for HE STEM practitioners. This
survey was used to inform the development of the course materials.
2. BG2E course. This was piloted at two events – and then developed into a one day course. This
was run three times – with improvements being made each time, based on the feedback from the
participants. The most notable changes to the course included: providing all the handouts in a
participant workbook; providing larger evaluation plan templates for groups to fill in together; providing
opportunities to complete the draft evaluation plan before the end of the course; providing online and
telephone support for people wanting to develop their evaluation plan.
3. Evaluation Masterclass. A short version of the masterclass was piloted at an event in Newcastle.
The course was created with content that could be selected from dependant on the interests and
needs of participants. This was run three times – the second two were developed on the basis of
feedback from the first. The main feedback linked to delegates wanting a longer course, and to cover
more of the material – however they also recognized that they would not have prioritized coming to
the event if it was longer than 3 hours.
4. TtT. A three hour session was developed to introduce people to the BG2E materials and to walk
them through the course. Several people would have liked to have attended the BG2E course before
coming to the TtT, but very few had been able to afford the time to do so.
5. The PP modules were developed by the Beacons for Public Engagement, based on specific
expertise that they had. These are planned to be run before 30 September 2012. We are hosting a
further Training the Trainer Day on the 30th August and have emailed all the participants in the
Training the Trainer module to ask them to apply for support to run their first course.
6. The course resources including powerpoints; trainer notes; participant workbooks; activity sheets;
additional resources lists etc were developed and hosted on an online Ning site. This site provided an
opportunity for people to download the resources and to ask questions of the trainers. Very few of the
participants have chosen to use the online space to ask questions, although most of the participants
in the courses have joined the Ning to download materials. Several people have contacted us by
phone / email for specific support. This support has tended to be project specific, in terms of helping
to develop an evaluation plan relevant to their project. In addition people have requested information
about specific evaluation methodologies, including sampling and analyzing data,
7. BG2E courses run by others. The TtT course aimed to support trainers to run the course
themselves. Many of the participants are now planning to run the course, or adaptations of it, with
their own communities. We look forward to receiving feedback from trainers, to reflect on the
relevance of their course in their institutions. A couple of courses have already been run – and these
were well received, and met the learning outcomes.
3
South West Spoke
Evaluation:
The evaluation plan was quite simple, but reflected the methodology taught in the courses. Namely to
plan the evaluation from the very start; to only collect data that could usefully be used; to integrate the
evaluation into the actual project.
There were three elements to the overall evaluation of the project as a whole:
1. A survey was done before developing the content for the project to ascertain the nature of need
across the sector. This informed the development of the core courses.
2. Each of the three courses (BG2E; EMC and TtT) was evaluated using a mixed method approach
(baseline; questionnaire post event; online survey two weeks after the event; graffiti walls) to inform
the development of each of the courses and to ensure they were fit for purpose. This iterative
approach ensured that we met the key learning objectives of each course.
3. We also plan to evaluate the BG2E courses run by the trainers who attended the TtT course, using
the same mixed method approach.
Evaluation summary:
1. Survey
161 people responded to the survey with a good spread across subject disciplines and types and
locations of institution. In summary:
a) A wide range of outreach activities is currently provided, and the scope of information about
evaluation methodologies needs to accommodate this range
b) A significant proportion of practitioners already undertake evaluation
c) The two main purposes of the evaluation currently undertaken (for improvement and for
impact) are consistent with the planned twin foci of the training programme to be developed
d) Attitudes towards evaluation are generally very positive
e) There is significant interest in training in evaluation
f) On-line resources are the most popular way of accessing this support
g) Workshops are also a preferred method for two thirds of respondents
h) Current on-line resources for evaluation provided through the National HE STEM Programme
website appear to be under-used
i) There are no major differences in responses between the different subject disciplines
j) Those not interested in receiving further support show a less positive attitude towards
evaluation
2. Evaluation of courses
(a) BG2E course
Summary:
2 Short pilot sessions: 30 participants.
3 Courses: 34 participants
Evaluation headlines:

Delegates indicated increased confidence in evaluating their current activities having attended
the BG2E course, and that this confidence is maintained, even after the initial enthusiasm for
the course.

Only two participants indicated they were less confident to evaluate their activities following
the course, but they both indicated this was because they now had a deeper understanding of
4
South West Spoke
evaluation and that their confidence before the course was based on misunderstandings
about the nature of evaluation practice

Best things about the training course included the content (13 out of 33); discussion between
delegates (11 out of 33); and the interactive nature of the course (6 out of 33).

25 delegates (n=27 who completed online survey) were satisfied or very satisfied with the
format / design of the workshop; and 27 were satisfied or very satisfied with the content.
Related quotes:
What I learnt..
“A strong framework and resources for evaluation will really help me to feel I’m covering everything I
need to”
“Course made the logic feel very approachable”
The best thing...
”The opening sharing format – non threatening, interactive, lots of group exercises”
“Very friendly and accessible trainers and good resources”
“It was great that you took everything from bottom up and explained thoroughly. Really good training:
workshops, theoretical learning and learning from the group”
“Very warm and positive group facilitators (and funny too!)”
“I now know what the constituents of evaluation should be...”
“Requires much more thought than I had anticipated”
(b) Evaluation Masterclass
Summary:
1 pilot session: 15 participants
3 courses: 47 participants
Evaluation headlines:
Immediately after the course 42 respondents filled in feedback cards. Delegates were asked to write
three words to describe the course – and these have been translated into a wordle – where the size of
the word relates to the number of times it was quoted
5
South West Spoke




39 (of the sample of 42) agreed or strongly agreed that the content was relevant to them
40 people agreed or strongly agreed that they enjoyed the course.
34 people agreed or strongly agreed that the course was pitched at the right level for them (A
couple of respondents reflected that they would have either benefitted from attending the
BG2E course; others that they wanted to have more detailed content.)
38 people reported that they strongly disagreed or disagreed with the statement ‘I was
disappointed with the course’
The best thing about the course for the majority of respondents was sharing and listening to other
people’s experiences; the main area for improvement was the time allocated to the course – although
there was also recognition that a longer course might not have been prioritised by delegates.
The online survey over 2 weeks after the course had 18 respondents:




15 were satisfied or very satisfied with the format/ design
16 were satisfied or very satisfied with the content
12 said they were more confident to evaluate their activities; with 6 saying they were neither
more nor less confident. Of those who said they were more confident the majority referred to
new ideas; resources and perspectives as the reason they were more confident. Of those
who said they were neither more nor less confident 2 said they were already very confident;
and 2 referred to appreciating course content and discussions but that this did not affect their
confidence.
13 said that they had fresh ideas in how to evaluate their work; and a further 9 had new ideas
about how to share their evaluation with others.
Related quotes:
What did you learn?
“Useful thinking of the many / various aspects of evaluation. Using pre-evaluation to inform activity
6
South West Spoke
design”
“That you should use your evaluation data!”
“Evaluation from different points of view”
“Interesting 1st discussion in the small group about catering to audiences”
“Different learning styles (e.g. reflective, artistic) call for different evaluation techniques – one size
doesn’t fit all”
The best thing was...
“The course – very strong on clarity of purpose, with good real-life examples”
“Meeting others with similar challenges to face”
“Different exercises, networking and discussion”
“Various nuanced discussions about evaluation – approaches, challenges etc.”
“Highlights areas to think about more so cuts down time needed to gain a start”
“Opportunity to discuss practice with like minded people”
Any other comments...
“Many thanks. I really appreciated the effort that had been put into designing the format and materials
for the training and the way it was being piloted with us for possible improvement.”
(c) Training the trainer
Summary:
3 sessions: 19 participants
Evaluation headlines:
Immediately after the course 13 respondents filled in feedback cards. Respondents were asked for 3
words to describe the course – and these have been translated into a wordle.
7
South West Spoke




The 13 respondents to the questionnaire all agreed or strongly agreed that the content was
relevant for them
12 out of the 13 respondents agreed or strongly agreed that the content was pitched at the
right level for them, with 1 respondent strongly disagreeing
All 13 respondents stated they would make use of the Ning site
The majority of respondents (10) felt the three hour session was long enough but there were
2 people who neither agreed nor disagreed and 1 person who disagreed.
What one thing could be improved?
Time was the strongest theme in these answers; for example ‘It would need to be longer to more fully
introduce course material’, ‘A slightly longer time’.
Related quotes:
The best thing:
‘It’s a good structure which covers all the main chestnuts and helpfully emphasizes timing importance’
“Thank you very much for providing me the opportunity to reflect on and improve my practice in
evaluation and the wider public engagement movement. I am a great fan of the programme of
learning and will be recommending the NCCPE and using training materials on a regular basis both
within my employment role and role as a post-graduate student group member.”
The final evaluation is not complete as we are still getting feedback for the TtT course, and from
courses other trainers are running. The final evaluation report will be posted on the Ning site and the
SW Spoke regional website in September 2012.
Discussion, Learning and Impact:
The project has yet to realize its full impact due to the inevitable time delays between creating a great
course; training others to deliver it; and then the trainers setting up their own courses. Many of the
trainers we trained are planning events in the Autumn – so we have yet to determine how effective
these will be.
There was a strong demand for the actual courses – with several of them being over-subscribed.
There is a real appetite in the sector to develop effective evaluation work in order to demonstrate
impact and to improve practice.
The courses were appropriate to the needs of participants, and the resources proved useful in
developing more effective knowledge and understanding. The courses met the learning objectives,
and the iterative nature of the course development meant that we were able to constantly improve the
course over the training period, which was reflected in fewer suggestions for improvements as the
course developed.
A particular challenge has been reaching the right people to get involved in the training the trainer
sessions. A small number of people attended these sessions because they wanted to learn how to
evaluate rather than because they had an opportunity to train others.
A second challenge was the confidence of evaluative practice that the trainer had – with several
requesting partnerships with us to deliver their first training session, as they felt they needed more
support.
.
Finally in the original bid we made the assumption that offering small grants to institutions would help
8
South West Spoke
them to run a training session – it has become apparent that what is really needed is support from a
trainer, rather than money to help fund the event. Therefore we have offered this option to trainers
and are currently exploring whether this will enable them to get started. To this end we have arranged
another TtT day in August to support more institutions to get involved.
Further Development and Sustainability
Will the activity continue in the future?
(b) Yes (in a modified form)
The NCCPE plan to run additional training the trainer sessions – these will be offered to trainers for a
small charge to cover the costs of running the course. If there is demand, we also plan to run the
Evaluation Masterclass on a cost–recovery basis.
The plug and play modules will be run as and when required at conferences and events.
The resources will be available to people wanting to run courses for HE STEM practitioners on
evaluation and these will assessable via an online Ning site – and advertised on the NCCPE website:
www.publicengagement.ac.uk/evaluating-stem-outreach
In relation to the approaches to sustainability outlined below, we are very interested in
activities and commitments which have occurred within the timescale of the project.
However, we recognise that some approaches may still be in the development phase at the
official project end date and it would also be valuable to include these examples in the
template.
Approaches to
Sustainability
Continuance (finding
alternative sources of
funding)
Examples
In relation to your project

Commitment from institutions to
provide continuation funding

Network/ communities likely to
be sustained through inclusion in
future funding bids
There is interest in the training
courses, and the training the
trainer courses. We are keen
to build on this momentum to
ensure the resources are used
to support HE STEM
practitioners to develop their
evaluation work, to help
ensure that their activities are
developed in more effective
ways, and the impacts of the
activity are better understood.
The Ning site will continue to
be supported, as long as
9
South West Spoke
participants find it a useful
forum to develop their
evaluation practice

Embedding (within
institutional activity)
Mainstreaming
(changes in working
practices)
Legacy (passing on
important elements of
the project)
Identification of institutional
strategies that the project has
informed

Uptake which has taken place, or
is likely to take place, within
own/other HEIs

Influencing of organisations
external to HE Sector which has
occurred through partnership
working

Staff development which is
planned or has taken place as a
result of your project

Curriculum enhancement that
has occurred or is likely to take
place as a result of your project

Influence of senior managers
that has arisen as a result of the
project

Networks/communities likely to
be continued

Dissemination of project outputs

Evidence of impact of activities

Creative Learning Journey
material made available via the
SW Spoke Creative STEM
website where relevant.
Trainers trained through this
process can make use of the
resources in their own
institutions. There have
already been several courses
run by trainers who have
attended the TtT courses –
and these have been really
effective at delivering against
the learning objectives.
We are currently offering
trainers support to run their
first course, where necessary
The main development has
been of HE STEM practitioners
who have developed their
skills, knowledge and
confidence in evaluating their
activities – we hope to see the
impact of this through
engagement through the Ning
site – but it is early days to see
how effective the training is in
informing practice.
Ning site will continue – and
provides support for people
wanting to develop their
evaluation plans and/ or
deliver training courses. In
addition more resources can
be developed and added to the
site by trainers and
participants, as well as
NCCPE.
References:
All resources are on the Ning site, including lists of articles that are of relevance to HE STEM
practitioners developing their evaluation work.
10
South West Spoke
Quotes:
“I’ve learned lots of new techniques and being able to discuss their use in different situations
has made me quite look forward to evaluating activities, something I never thought I’d say”
Participant BG2E
“I’ve realised how much I already know, my knowledge has been refreshed and I’m also
excited about doing it instead of nervous”
Participant BG2E
“I feel with the frameworks/resources provided I can develop a really good evaluation plan”
Participant BG2E
“The best thing was the practical exercises, especially looking at the evaluation of real
projects”
Participant EM
Many thanks!
11