Career learning for the 21st
century
CPD series for Community Learning Champions
Module 5:
Introduction to evaluating and measuring impact in career development
1
Contents
Introduction to the group work workshop
3
Tutor notes
Workshop outline
Workshop aim and objectives
Checklist of resources required
LSIS workshop – trainer’s feedback
LSIS workshop – attendee’s feedback
3
3
4
4
21
23
[Table of contents will be created automatically by designer – you can ignore]
Download
This CPD module is available to download in PDF format from Excellence Gateway Career
Development section.
Publisher
Published by the Learning and Skills Improvement Service (LSIS).
© LSIS [month year]
Publication reference: LSIS[xxx]
2
Introduction to evaluating and measuring impact in
career development workshop
Tutor notes
Workshop outline
This workshop has been designed for Community Learning Champions )(CLCs) who support people’s
career development through their CLC role and are new to, or want to revisit, evaluating and measuring the
impact of the work they do. It raises issues of why we need to monitor and evaluate what we do and the
services we provide , the need for continual improvement, identifies the impact that high quality career
development activities can have on all our clients and on scheme/organisational performance.
This module can be delivered by anyone who is an experienced or qualified training professional if they
take the time to familiarise themselves with the materials including the web links, handouts and references.
It would, however, be an advantage if the trainer had experience and/or training in career development and
knowledge of the CLC programme, particularly in relation to those CLCs attending the workshop. This will
enable wider and deeper exploration of the topics covered in relation to the CLC roles of participants.
The materials can be delivered as they stand, contextualised or modified to suit the time available and the
needs of the participants, scheme or organisation. The PowerPoint slides and handouts are available as
separate files.
A short (two and half hour) follow on workshop is available. It can be run after this workshop to give CLCs
more in-depth knowledge of measuring impact and contribute to the development of an evaluation process
for their scheme.
This workshop is the fifth in a series of CPD workshops for CLCs produced by LSIS and available to
download from the LSIS Excellence Gateway. There are six others in the series:

An introduction to career development

Introduction to Interviewing

Introduction to values and ethics in career development

Introduction to delivering career development through group work

Introduction to reaching potential by raising aspirations

Introduction to the Blueprint for Careers
LSIS has also produced a series of 10 workshops for people within the FE and Skills sector who support
learners with their career development which are also available to download from the LSIS Excellence
Gateway. These may be relevant for CLC schemes specifically working in this area or for those CLCs who
wish to do more in depth work around a particular topic. These are available to download from the LSIS
Excellence Gateway.
LSIS have also developed an online resource, “Career learning for all,” available free of charge on the LSIS
virtual Learning Environment just go to www.leadershiplearning.org.uk and create your own login. This has
been designed for those involved in supporting learners with their career development, both specialists and
non-specialists.
3
Workshop aim and objectives
The aim of the workshop is to explore continuous quality improvement and why evaluation and
measuring impact are key to the effective delivery of career development activity in CLC schemes.
The objectives of the workshop are to enable participants to:



Describe the outcomes of effective career development within their CLC scheme
Understand what effective evaluation of career development includes
Identify areas of strength and weakness in their scheme’s/organisation’s review, monitoring,
evaluation and recording systems
Example workshop timetable
Time
Session/s
9:30
Welcome, domestics, aim & objectives
9.40
Introductions
9:50
Impact and reality
10.15
Monitoring, review and evaluation
10.55
Break
11:10
Drawing it together
11:15
Evaluation framework
11:35
Audit tool
12.00
Summary
12.10
Conclusion
Checklist of resources required
Fire evacuation procedures.
PowerPoint™ slides 1 to 13.
Flip chart and marker pens, or whiteboard.
Pre-prepared flip charts.
Laptop, LCD projector and screen.
Copies of handouts 1-3.
Evaluation form: LSIS workshop – trainer’s feedback (see page 21).
Evaluation form: LSIS workshop – attendee’s feedback (see page 23).
4
Introduction to evaluating and measuring impact in
career development
Presented by –
Date –
The power to question is the basis of all human progress.
Indira Gandhi
Questioning is the door of knowledge
Irish saying
Slide 1
Customise the visual to include your own details and date of the workshop
1.
Present the domestic arrangements and fire safety for the venue
2.
Introduce yourself and the session
3.
Ask the group to read the quotes and ask for their reactions. Normally groups say:



highlights the need to not be self satisfied,
prompts you to question and not just accept something is over and completed,
you do this when you are new or when you first do things, but complacency can set in etc.
Resources: Fire evacuation procedures and copies of PowerPoint slides
Timing: 5 minutes
Transition statement: Let’s look at the aim and objectives of this workshop.
Your notes:
5
Aim and objectives
Aim:
To explore continuous quality improvement and why evaluation
and measuring impact are key to the effective delivery of career
development activity within CLC schemes
Objectives
By the end of the session learners will be able to:
•Describe the outcomes of effective career development within
your CLC scheme
•Understand what effective evaluation of career development
includes
•Identify areas of strength and weakness in your
scheme’s/organisation’s review, monitoring, evaluation and
recording systems
Slide 2
Introduce the overall aim of the workshop.
Relate this aim back to the quote and group’s comments in the previous slide.
Go through the objectives and ask if there are any questions.
Note: It may be useful at this point to remind participants of the definition of career development and to
reinforce how their role as a CLC contributes to this – as discussed and identified in module 1.
Resources:
Timing: 5 minutes
Transition statement: In this session, we are going to discus what monitoring, review and evaluation are,
particularly in the context of the work you do as CLCs.
Your notes:
6
Introductions
Work in pairs, preferably with someone you do not know well.
Introduce yourselves: name, job role, experience in evaluating
and measuring impact of the work you do as a Community
Learning Champion.
Be prepared to introduce your partner to the group.
You have 5 minutes for this activity
Slide 3
Introduce the exercise by saying that it will be useful for us all to find out about each other’s experience in
evaluating and measuring the impact of the work you do as CLCs.
Go round the group asking them to introduce each other. Comment on the range of experience in the
group.
Resources:
Timing: 10 minutes: 5 minutes for the activity; 5 minutes for introductions
Transition statement:
Your notes:
7
Impact of career development
Work in pairs.
Agree between you the outcomes and impact that should result
from delivering a consistent and high quality service to the
people and communities you work with and support.
You have 5 minutes for this activity
Slide 4
Ask the group to split into pairs and try to ensure they work with people they do not know well if this is
possible. Talk through the slide ensuring everyone is aware of what is required. It may be helpful to
encourage participants to think out ‘outcomes and impact’ as the difference their work should/could make to
individuals and communities and/or the benefits of informal learning.
When the 5 minutes are up, give two pairs a piece of flip chart paper and ask them to write up their first two
ideas and pass it on to the next pair – continue this process until everyone’s ideas have been recorded.
Alternatively, to save paper, you could ask them to list ideas on a whiteboard.
Ask the whole group to read the list. Answers could include:













Clients are more self-aware and motivated
Clients access the opportunity (informal or formal learning) that is right for them
Increased confidence and self esteem
Increased employability skills
Improved learning culture in communities of need
People get the chance to try something new – clients and CLCs
Increased wellbeing – clients and CLCs
Reduced social isolation
Evidence to help the scheme access funding/support funding bids
Increased participation in learning within vulnerable/disadvantaged communities we work with
People gain skills and qualifications – clients and CLCS
Better networks and linked up services in the community
More people moving on to formal/accredited learning
Work through the list identifying which ones benefit the individuals, which benefit the CLCs as volunteers
and which benefit the organisation/scheme as a whole – use different coloured pens to highlight the areas.
Say that this shows the potential impact of the career development work we do and highlights some of the
things we might be able to measure.
Resources: flip chart paper and pens
8
Timing: 10 minutes: 5 minutes for the activity; 5 minutes feedback and discussion
Transition statement: We are now starting to see what things we could measure but let’s think about
whether we actually do it, and how we know we are achieving these outcomes for clients.
Your notes:
9
Reality?
As CLCs we may often be working with people who may be facing a
variety of barriers/challenges such as:
• Limited or no advice or support throughout their life
• No or few qualifications
• Low self esteem
• A lack of confidence
• Long term unemployment
• Social isolation
• Health problems
In pairs discuss:
How do we know we are not giving them bad information
and/or advice?
Slide 5
Before showing the slide, ask the group to discuss the sorts of barriers and challenges people they work
with could be facing. What examples have the group got (from own experience or feedback from clients) of
how easy/hard it is to get the right advice about taking a first step into learning or knowing what is the right
‘path’ for them. This could be from careers advice at school to present time and include both formal and
informal learning.
Show slide 5. Thinking about your role as a CLC working with people who may have had a lifetime of
limited or no advice/support or poor advice also may have no/few qualifications, low motivation/aspiration,
low self esteem/confidence ask, “How do we know we are not giving bad information and/or advice?”
In pairs, ask the group to consider what things happen (or could happen) that would help us to know this.
Answers include: feedback, complaints, monitoring activities, reviews, evaluations, past client’s comments,
client’s telling others about the CLCs, other organisations wanting to work with us, statistics, open culture,
repeat visits, CLC meetings, asking for feedback all the time etc.
Resources: Flip chart or whiteboard and pens.
Timing: 15 minutes: 5 minutes to introduce, 5 minutes to complete the exercise, 5 minutes for feedback
and discussion.
Transition statement: We are now going to look at how we can work towards ensuring we deliver a quality
service that meets the needs of people we work with.
Your Notes:
10
Definition - monitoring
Monitoring: is the regular observation and recording
of activities taking place in a programme. It is a
process of routinely gathering information on all
aspects of the activity. It is like watching where you
are going while riding a bicycle; you can adjust as you
go along and ensure that you are on the right track.
Slide 6
Tell the group that to find out what is happening in the career development work you do you need to
monitor activities. Say that formative monitoring and review is carried out during the year to assess if what
was planned and designed is what is actually taking place/did we meet our targets.
In small groups: Ask the groups to consider what (they know) is monitored/measured in their
scheme/project/organisation and how this is done.
Ask for feedback and record the answers on a white board or flip chart.
This list will vary for different CLC schemes but answers could include:









Evaluation sheets,
Volunteer hours
Numbers of clients engaged
Numbers signposted/supported to access learning
Number of new informal/self managed learning groups established
Client progression
Attendance lists
Learner retention
Learner success rates
Go through each one asking for the benefits and costs of each of these monitoring activities. Normally this
raises issues of:







Tick box exercise only
Time consuming exercise with no time given to the findings
Done half-heartedly by some and well by others
No collation across the organisation of the findings
Volunteer CLCs not really sure what/how they should be recording and when
Volunteers only give limited time to the scheme/project – want to be working with people not doing
paperwork
Paperwork ‘not my thing’
11

No one will to do anything with the results etc.
Have a pre-prepared flip chart or area of whiteboard or wall headed: ”Improvements” available. Ask the
group to write up how the existing monitoring could be improved, including anything currently not monitored
that we could monitor?
Say that true to the nature of the workshop you will keep a record of improvements that could be made and
circulate the findings after the end of the workshop.
Resources: Pre-prepared flip chart or whiteboard headed, “Improvements” and pens.
Timing: 10 minutes
Transition statement: Now we will look at review.
Your Notes:
12
Definition - review
Review: to look back upon; view retrospectively
Slide 7
Ask the group to compare the terms ‘review’ and ‘monitor’ – ask what are the differences?
Say that reviewing is looking back at what went well and not so well. “What went well,” implies a qualitative
measure. Say that monitoring could only be concerned if something took place, how many participated etc.
but not be concerned with quality. Ensure everyone understand the difference between monitor and review.
As before, ask participants to reflect for a minute on what their scheme/project/organisation does to review
their activities as CLCs:






Client feedback, evaluations and surveys
Observation reports
Case studies
Review/evaluation meetings
Partner/learning provider evaluation
Review of complaints, compliments etc.
Record these on a flip chart or whiteboard and ask for benefits and costs.
Many groups give the same responses as for slide 4:







Going through the motions
Little follow up to see if changes implemented
Meetings sometimes missed – volunteers have other commitments
Tendency to think another year over and focus on new year
Little communication
Volunteer CLCs not involved in the process
No feedback to new volunteers/employees etc.
Resources: Flip chart/ whiteboard, pens
Timing: 5 minutes
Transition statement: Next up is evaluation.
Your Notes:
13
Definition - evaluation
Evaluation: is the assessing and judging the
value of a piece of work, an organisation or a
service. Its main purpose is to help an
organisation reflect on what it is trying to achieve,
assessing how far it is succeeding, and identify
required changes.
Slide 8
Reinforce the fact that evaluation is about value, it answers the question, ‘Was it worth it?’ Ask if there are
any comments or questions. Say this is very different from monitoring if what was designed occurred – it is
the, “So what?” question?
Ask participants to form groups of four. Ask them to consider how a scheme/project could evaluate its CLC
career development activity (both with clients and staff/volunteers). Allow 5 minutes for this part of the
exercise.
Normally answers include: face to face or phone interviews, questionnaires, surveys, observations, case
studies, client follow up, CLC evaluation meetings, focus groups, personal logs or diaries, log books,
reports, workshops, creative methods (drawing, writing, photography), stakeholder meetings, reviewing
Facebook comments, blogs etc.
Allocate three ideas from the list, to each group of four and ask the groups to decide when each method
would be useful and to note any problems or issues with the method.
Ask each group to present their findings. Allow 10 minutes for this part of the exercise
Organise for this list to be word processed and circulated to all participants.
Summarise the findings and say that when doing evaluation you need to consider how you are going to
analyse, interpret and present the findings. It’s important that you also think about this in light of the
capacity and resource available to you – its needs to be ‘doable’!
Resources:
Timing: 25 minutes - 3 minutes to talk about the slide and form small groups, 10 minutes for the exercise
and 12 minutes for the presentations and summary.
Transition statement: We’ll have a break for 15 minutes now.
Your Notes:
14
Drawing this together
•
•
Monitoring and review activities -
•
•
•
Draw attention to successes
Highlight areas that need changing
Identify opportunities for development
Evaluation activities –
•
•
•
Generate information on impact and value
Provide evidence against which you can make judgments
relating to the quality of provision and practice
Inform the continuous improvement plan and the scheme’s
organisation’s capacity to improve
Slide 9
Remind participants that before the break we spent some time looking at monitoring, review and evaluation.
Let’s look at all of these together.
Monitoring and review gather the evidence – this is the formative process.
Evaluation is where you make the judgment relating to progress and the need for further development
(summative).
Draw attention to the following Ofsted quote from the September 2009 framework regarding an
organisation's capacity to improve’:
Just to identify strengths and areas to improve are no longer enough. An organisation needs to be
self knowing and have a robust continuous improvement plan - a plan that is reflected in all areas of
the organisation.
Discuss as a group and ensure all are clear on the differences. Have any further examples of monitoring,
review or evaluation activity (existing or possible) come to mind?
Resources:
Timing: 5 minutes
Transition statement: We’ll now go on to look at evaluation frameworks.
Your Notes:
15
Evaluation framework
•
•
•
•
•
•
Identifies the qualitative and quantitative data that is going to
be gathered
Outlines how the data is going to be used
Reach – numbers and types of individuals affected by, or in
contact with, your interventions
Reaction – Initial feedback from the clients
Learning – What has been learnt as a result of the activity or
information and can this be evidenced
Impact – What does the individual do differently and how
does this feed into the scheme’s/organisations’
effectiveness?
Based on Donald Kirkpatrick’s Learning Evaluation Model 1959
Slide 10
Ensure participants understand the difference between qualitative and quantitative data.
It is important that participants understand that as part of a rich and robust evaluation process we are
looking at:




Who we are working with? Say this has implications for equality, access and allocation of resources.
Ask if they feel all individuals and groups their scheme works with (or aims to work with) have equal
access and in particular those from disadvantaged groups or communities
Immediate feedback. Ask what they get feedback on, if formal or informal, where recorded, how
analysed, what is done with the information and who has access?
What clients have learnt/done after meeting with CLC, attending a course/workshop or other training
and how this is directly attributed to the career development activity (remind participants that this can
be a simple meeting with a CLC to get some information and/or advice about what’s available and
how to access)? Ask how they find this out from clients – do they ever find this out? Ask how they
know if clients have done anything positive from their interventions?
What clients are going to do differently as a direct result of the activity or information? How will this
also impact on their project/scheme/organisation? Ask how they know if their clients are changing
plans, taking alternative routes etc. as a result of their interventions? Ask how often the participants
themselves change their behaviour as a result of the schemes/organisation’s evaluation findings?
Give out handout 1 - Evaluation; further detail on evaluation specifically linked to CLC career development
activity. Ask participants to read it and discuss in pairs where they think their scheme/organisation is at
present in monitoring, reviewing and evaluating career development activity.
Ask the whole group for findings.
Resources: Copies of handout 2 – ‘Evaluation’. Can be adapted as relevant to specific CLC schemes.
Timing: 20 minutes: 5 minutes talking through slide, 10 minutes discussion in pairs and 5 minutes
feedback and summary.
Transition statement: Let’s take a closer look at our own schemes and see what judgments we can make.
Your Notes:
16
Completing the career
development audit
Activity
Individually go through the audit tool and decide what score you
would allocate to your scheme in relation to each of these
statements.
In fours, compare your assessments, noting similarities and
differences .
Slide 11
Before showing the slide
Give out handout 2 - Audit tool, to all participants.
Activity
Ask the group to work individually and then in fours. Tell group to also keep a note of any ideas for
evaluating and measuring impact in their scheme that might come out of discussions (these can be used
for the extension activity which looks at developing effective measures in CLC schemes).


Individually review each of the areas included in the audit tool for their organisation
In groups, compare and contrast their assessments 10 minutes
5 minutes
Take feedback from the group, drawing out on a flip chart the areas where schemes are effectively
evaluating career development provision and areas where there are gaps.
Resources: Copies of handout 2 - ‘Audit tool’, flipchart and markers
Timing: 25 minutes - 5 minutes set up and individual work, 10 minutes discussion in fours, 10 minutes for
feedback and discussion.
Transition statement: Let’s look at what we’ve covered today
Your Notes:
17
Summary
•
•
•
•
•
What does monitor mean?
Give some examples of reviewing
What does evaluation mean in terms of your scheme’s career
development activity?
What impact could effective career development activity have
on clients?
What aspects of the career development audit do you
remember?
Thank you for attending, please complete an evaluation form
Slide 12
Go through the questions and if necessary revisit slides if any areas of difficulty arise. Ask for any
outstanding questions.
If relevant/appropriate, tell participants there will be a follow up workshop where they can look more closely
at evaluation in their scheme and help to develop a plan for measuring impact and implementing
continuous improvement measures
Give out the evaluations and ask for these to be completed. Give out handout 1 if you have not already
done so.
Thank the group and close the session.
Resources: Evaluation forms
Timing: Ten minutes: five minutes for review and five minutes for evaluation and close.
Transition statement: I have included a list of the references used in this session so you can do further
reading if you wish.
Your Notes:
18
References
http://cec.vcn.bc.ca/cmp/modules/mon-wht.htm
http://www.definitions.net/definition/review
http://www.evaluationtrust.org/evaluation/definitions
Slide 13
Thank the group for their participation.
Resources:
Timings:
Transition statement:
Your notes:
19
Insert handouts here
CLC CPD Mod 5 handout 1 resources
CLC CPD Mod 5 handout 2 evaluation
CLC CPD Mod 5 handout 3 audit tool
20
LSIS workshop – trainer’s feedback
Please use this form to feed back how useful you found the training materials provided. This is not
to measure the success or otherwise of your workshop, but to help us improve the quality of the
materials provided.
Name of workshop: [insert name]
Date of workshop:
Using a rating of 1 = very good to 7 = very poor, please answer the following questions:
Rating
Overall how happy were you with the materials?
How well did the course meets its aim and objectives?
How would you rate the handouts?
How would you rate the PowerPoint™ presentation?
How would you rate the exercises?
How would you rate the trainer’s background information?
Which parts did you find most useful?
Which parts did you find the least useful?
What do you think could be done to improve the materials?
Please comment on the suitability of the timings provided.
Any other comments?
21
22
LSIS workshop – attendee’s feedback
Name of workshop: [insert name]
Date of workshop:
Using a rating of 1 = very good to 7 = very poor, please answer the following questions:
Overall how useful did you find the course?
How would you rate the materials provided?
How would you rate the trainer?
How useful were the exercises?
How would you rate the pace of the course?
How would you rate the length of the course?
Which parts did you find most useful?
Which parts did you find the least useful?
What do you think could be done to improve the workshop?
Any other comments?
23
24