This paper outlines the HEAT recommendations for implementing common evaluation questions to
use with outreach participants across the HEAT partnership. As such it suggests:
1. The reasoning behind attempting to use a limited number of common evaluation questions,
on a voluntary basis, across the partnership
2. The proposed wording of questions - arising out of the Common Evaluation Workshop held
in April with interested HEAT members and a comparison of HEAT aims with those of the
Department for Education’s ongoing Youth Cohort studies
3. Ways that the questions might be implemented, including thoughts about timing and
administration
1. Reasoning behind using a set of Common Evaluation Questions
The HEAT partnership concentrates primarily on the production and analysis of quantitative data
and reports. However, existing HEAT members have mooted finding a relatively simple way of
including some more qualitative data in our reports. Most members ask some or all of their outreach
participants to complete a questionnaire and so it is potentially a relatively easy process for most
HEIs to include a small number of common questions on their questionnaires. The inclusion of
questions would be on a voluntary basis but the HEAT membership hope to get a significant number
of participant responses, especially as the HEAT service expands. The service holds a lot of
quantitative data such as participation patterns and progress of individuals and any qualitative data
will add value to this. Equally, any qualitative data we collect will be enhanced by our quantitative
data – many other agencies asking attitudinal questions do not have the advantage of our richness
of data at an individual learner level.
2. Proposed questions
Non-specificity of questions - Questions have deliberately been formulated or chosen so as to be
generic - for instance they are non-Year Group specific. The influence of factors such as age will be
taken into account at the point of analysis.
The Youth Cohort Study + HEAT member input - the 4 proposed questions are a combination of
HEAT members input at the Common Evaluation Questions Workshop and two questions from the
Youth Cohort Study, a current longitudinal study run by the Department for Education and National
Statistics . The Youth Cohort Study questions are not dissimilar from ones suggested by the HEAT
membership but adopting the exact wording of the national study means that we are able to make a
direct comparison for those questions. We will also then be able to place our research within the
context of the wider findings of that study.
The questions which are suggested that are solely from wording suggested by the HEAT membership
workshop are statement based – many of the membership felt that statement questions, where a
respondent is asked to indicate a number of statements that they feel apply to them were a good
barometer to see changes in attitude over time but also to capture the breadth of issues
respondents might feel themselves faced with.
Note that a final question has been included which asks respondents about ‘barriers’ to Higher
Education. Some members of the workshop felt that focusing on the negative was not helpful.
Others felt that they were already sure of barriers facing their learners or that national research had
already covered this.
The question is included as optional and has therefore been placed at the end. It does not include
the word ‘barrier/s’. It is worth noting when deciding whether or not to use this question that what
our research proposes is not simply a ‘snapshot’ of barriers that any given student feels face them at
the time of asking. We will have a large sample across a number of year groups and we will be able
to look at how student concerns change over time. This may help with the delivery of activities in a
timely manner. We will also have access to information about what activities students with
particular concerns have/have not taken part in which sets this research apart from previous
national studies.
3. Implementation




Voluntary - The use of these questions is on a voluntary basis, although the greater the
sample size the more weight our qualitative research will have.
Timing - The YCS administers its questions in the Spring Term of each year. However, we
suggest that volunteering HEIs ask these questions in the Autumn Term, partly because of
the wording of the questions chosen. There is nothing to stop HEAT members including
using the questions all year if they wish to do so. However, any analysis done by the HEAT
Central Team will focus questionnaire responses from the Autumn Term in order to go some
way towards ameliorating any change in attitudes that happens over an academic year.
HEAT members will still be able to analyse all their questionnaires regardless of the timing of
administration.
Choosing Students – HEIs may take a view as to which students on which activities they
choose to administer these questionnaires to. Ideally they would be asked of students
whose full details (first name, last name, data of birth, postcode, gender, DOB) are known
and on the database in order that we make the most in analysis of the quantitative data we
hold about these students.
Collation of responses – data can be uploaded onto the Surveything facility of the database.
Alternatively it can be held by institutions until analysis takes place if HEIs have a separate
way of logging questionnaires. Where this is the case it is worth checking with the HEAT
Central Team in advance that you will collect all the data needed to enable analysis.
How likely do you think it is that you will ever apply to go to university to do a
degree? Would you say it’s….
1. Very likely
2. Fairly likely
3. Not very likely
4. Not at all likely



How likely do you think it is that if you do apply to go to university you will get in?
1. Very likely
2. Fairly likely
3. Not very likely
4. Not at all likely



Used in the Youth Cohort Study
Confidence
Scale forces them to choose a likelihood rather than
‘Don’t Know’
If you went to University it would be because you: (multiple choice/tick as many as
applicable)
a) need a degree for a particular career
b) want a well paid job
c) want to gain life experience
d) want independence
e) want to study something enjoyable
f) other.............

Knowledge/awareness
What would put you off applying to University? (Tick all that apply)
 Cost
 I might not get the grades
 I’m worried I won’t fit in
 I’ll miss my family/friends
 I don’t know anyone else who has been
 I can’t leave home because I care for someone
 I have a disability
 I don’t know what I want to do in the future

Attainment/awareness/confidence

Used in the Youth Cohort Study
Aspiration
Inclusion of ‘ever’ takes into account the non traditional
path that our students might take
Scale forces them to choose a likelihood rather than
‘Don’t Know’


I wouldn’t have the support of my family
Other:.........