DLHE advanced
2012
Objectives
Methodology
Recap and reflect on the methodology
Questionnaire and data
Review the questionnaire and data whilst learning how to overcome
common issues and challenges
Data submission and quality checking
Understand the validation kit and changes to data collection
Preparing for audit
Find out about the audit process and findings from HEFCE
DLHE and KIS
Learn how to use the downloadable files
Where to begin?
The review
• The review of DLHE took place ahead of 2011/12
• The sector was consulted through DLHE contacts
who were invited to respond to the proposals
• Changes were subsequently introduced for 2011/12
and communicated through seminars, circulars and
the coding manual
Today we recap and reflect on the changes and aim to
draw on institutional experience to improve on
procedures, practices and dealing with challenges…
Coverage
Student
completed
during HESA
reporting period
Reason for
ending not a
result of death
Obtained
applicable
qualification
Not a
visiting/exchange
student
Reported as
active (or as
dormant for PGR
students only)
Studied for 8
weeks+ in UK
Coverage
QUAL
The survey population has been extended to
include a wider range of qualifications
MODE
The survey population now includes all
qualifying postgraduate research students
including those from a dormant status
EXCHANGE
The survey population no longer includes
incoming exchange or visiting students
DOMICILE
The survey population now includes all
non-EU international leavers
DLHE Coverage
Flowchart
So much to do in so little time…
• HESA recognise that some institutions will now be
surveying a higher volume of leavers in comparison to
earlier years
– The contact period has not been increased
– Resource and budget will be impacted
It is important to note
Surveying a wider range of students generates a more diverse set of
results and will potentially provide more data…
DLHE 2011/12
Target response rates
90%
80%
70%
60%
50%
40%
80%
30%
70%
80%
50%
20%
10%
0%
UK-domiciled
leavers who
previously
studied full-time
All other EU
students
UK-domiciled Research Council
Non EU
leavers who funded students international
previously
leavers
studied parttime
How did we do last time?
Response rates 2010/11 UK-domiciled leavers who studied full-time
(First degree)
100.0%
95.0%
90.0%
85.0%
80.0%
Response rate
Target
75.0%
70.0%
65.0%
60.0%
Institutions
How did we do last time?
Response rates 2010/11 UK-domiciled leavers who studied part-time
(First degree)
100.0%
95.0%
90.0%
85.0%
80.0%
75.0%
Response rate
Target
70.0%
65.0%
60.0%
55.0%
50.0%
Institutions
The KIS agenda
• Key Information Set
– Course level information for prospective students
– Uses DLHE data
KIS
Course information
Financial information
Accreditation
Accommodation
Employment
Assessment
Promoting DLHE
Evidence shows there is a link between those institutions
who promote DLHE and those who exceed target response
rates
Why not consider graduation ceremonies, graduate contact,
internal publicity or offering prizes?
Surveying the correct students
What if it goes wrong?
• Missed entirely (cohort)
– Contact institutional liaison for further information
• Missed entirely (individual)
– Resurvey using correct questionnaire for intended census
These coverage errors can be corrected at a later date but
are best avoided
What if it goes wrong
• Dormant
– Students sometimes reported as dormant if in the final
stages of postgraduate study at the start of reporting period
2. This is a student reporting discrepancy and will result in
the institutions own record systems not identifying students
for DLHE purposes
• Wrong census
– Some institutions own record systems only allow for one end
date to be recorded. HESA defines the end date as at the
point the structured part of the course finishes whereas
some institutions consider this to be the date they are
awarded. If the end date is recorded as the award date then
the student could be potentially surveyed for the wrong
census
These coverage errors usually go unmissed and cannot be easily
corrected at a later date and therefore must be avoided
Methodology
Methodology
• 2011/12 saw the introduction of a new more relaxed
approach to DLHE methodology
• Institution’s now decide when and how to contact
leavers during the contact period (initial and followup no longer exist)
• Using predetermined methods to contact leavers was
viewed as restrictive…
It’s now up to you
Methods
Sequence
of
methods
Highest
response
rate is key
• The new methodology now
allows for the use of as many
methods as possible during the
contact period
• With the view that by
increasing the lines of
communication, the number of
responses will increase
Number
of
methods
• Institutions are advised to
review previous collection
response rates for an indication
of methods which attract the
highest response rates
Methods of contact
• Postal mailing
• Telephone
questionnaire
• PDF version (e-mail)
• Centrally-hosted online
questionnaire
Paper questionnaire
• HESA provide paper
questionnaires for
sending out by post
• Questionnaires are also
available in Welsh
• Institutions are contacted by
HESA with a set of questions
about the size of the DLHE
target population and the
planned surveying
arrangements in January (for
April) and June (for January)
Common issues
Pros
• One version of the
questionnaire simplifies the
DLHE process, is more cost
effective and easier to
manage
Cons
• Third party questions are
not identified
• Coding boxes have been
removed
Suggestions
• Laminated copies showing
third party questions
• Create coding reference
material
PDF questionnaire
• A PDF version of the questionnaire is available in both
English and Welsh
• Available to download from the HESA website
• PDF questionnaires are emailed to leavers who are then
required to print, complete, sign, date and return the
form back to the institution
• Institutions should consider using postage-paid returns
to increase likelihood of responses. Leavers will not
respond well to having to pay postage
What are the chances
Smartphone anyone?
Do you open your emails from your smartphone?
Will you go back to emails read from your
smartphone?
Is your smartphone remotely connected to a printer?
• Whilst PDF remains a fairly problem free approach to
surveying leavers, it is worth keeping in mind that
response rates to emailed PDFs may be lower than other
methods
• Therefore this method should only be considered in
combination with one or more other methods
DLHE online system
• Online version of the
questionnaire (Welsh version
also available)
• Institutions provide link
www.dlhe.ac.uk to leavers
• HESA provide free of charge to
institutions who register for
system
• Also know as ‘Centrally-hosted
online DLHE survey’
• Has in-built validation to ensure
questions and sections are
answered
• Leavers can complete it at their
own accord and at their own
pace
More than just…
An online DLHE survey tool
Why not use the (free, maintained and updated by
HESA) DLHE online survey to input responses?
Inbuilt validation will ensure all valid questions are both
answered and internally consistent
Remember to adjust the STATUS field to reflect how the
information was captured!
Telephone
• Often the most effective method – accounts for
majority of responses but requires the most
significant amount of resource
• Callers require perseverance, an excellent telephone
manner and a sound grasp of the requirements.
Some institutions recruit students as callers and
multilingual speakers are sometimes used
Telephone best practice
When to call
Preparation
• Experience shows calling
outside of standard working
hours yields better results
• Specifically between 5.30pm
and 8.30pm is often considered
the best time to call
• Also consider employing staff to
contact leavers at the weekend
• A cut off time should be agreed
and adhered to
• Log the times of calls to
establish peak or optimum
times for future surveys
• Time is of the essence - know
your leaver before calling
(create a profile)
• Note the qualification awarded
to the leaver vs. what they had
originally aimed for
• Do not block your telephone
number
• Ensure permissions are set to
allow calls to mobiles
• Have the correct questionnaire
for the survey period
Third parties
• If a leaver is unobtainable, a third party can be used as
an alternative source in providing the required
information on behalf of the leaver
• Most commonly this scenario will take place during a
telephone call (some third parties have been known to
also return paper questionnaires)
• Third party information is considered legitimate on the
basis they are fully confident the answers are correct
(and relate to the applicable census date)
• Third parties should only be asked a specific subset of
the questions
Other methods
• Direct contact
– During the field work period e.g. they visit the careers
office or during the graduation ceremony
• Academic departments Last resort
– Can also be used as a source of information as they may
have on-going interactions with the leaver(s)
• Employers Last resort
– If employer is known to the institution they too may be
contacted
Covering letter and email
It is mandatory for all leavers to receive specific information in connection
with DLHE. The covering letter, email or verbal confirmation provided during a
telephone call;
Tells the leaver
that they will
be contacted
again if they
do not reply
Informs the
leaver that
data may be
collected from
a third party if
they can’t be
contacted
themselves
Gives the
leaver an
opportunity to
refuse to give
data or be
contacted
again
Tells them that
they may be
contacted
again for the
longitudinal
survey
It is possible to include the covering letter or email for all methods with the
exception of using telephone. If the institution opts to use the telephone as
their first method it is important to remember to confirm this information
verbally as part of the introduction
Covering letter and email
• Printed on headed paper (covering letter)
• Do not amend HESA text
• Include institutions own data protection notice
• Include literature about the careers service,
forthcoming alumni events
• Branding helps
Interactive task
Discussion questions
• Map the DLHE process within your institution consider what additional tasks you might do as an
institution
• What barriers exist when undertaking DLHE?
• How is the DLHE data used internally within your
institution?
The questionnaire
Understanding the questionnaire
• Only one version of the questionnaire is available and this is
to be used for all methods
• Specific questions must be answered for a valid response!
Institution’s commonly follow up incomplete or illegible
responses
• Remember, third parties should only be asked set questions.
There are no markings on the form to indicate these
questions
• The need to ‘pad out’ (‘X’ or default codes) the data/answers
for questions which are not required or returned is no longer
necessary due to the format (XML)
Official use only
STATUS
• The top boxes are used to
record the dates of
attempted contact
• The boxes on the bottom
row are used to record the
STATUS of the response:
• These are not used to
indicate how many times an
institution should attempt
contact
• (2) Telephone survey: HE
leaver
• (3) Telephone survey: third
party
• (5) Own institution’s
student record
• (6) Institutional third party
• (7) Deceased
• (8) Explicit refusal NEW
• Many institutions attach
their own labels to include
more dates
Section A
Activities
Section A: Q1
ALLACT and MIMPACT
Required for a valid response
To be asked third parties
Ensures remainder of information returned is internally consistent
If the leaver does not complete any activities or completes more
than one activity but does not identify which is most important, in
the first instance try to contact the leaver. If this is not possible use
professional judgement to code ALLACT and MIMPACT based on
completed section(s)
A new concept…
• The leaver (or third party) must decide which one
activity is ‘most important’
• This doesn’t necessarily mean it has to be the job
which takes up the most time or pays the most
salary, or any further study, training or research
which takes up the most time. Some leavers may opt
to select the activity which will support their future
circumstances/aspirations
Section B
Employment
Section B: Q2
JOBSNO
Not required for a valid response
To be asked third parties
Will be used to understand the extent of portfolio careers
No requirement to return less than 2 jobs and only able to return
up to 9 jobs
A new concept…
• If JOBSNO is more than one, leavers are asked to provide
details about what they consider to their main job
• This doesn’t necessarily mean it has to be the job which
takes up the most time or pays the most salary! Some
leavers may opt to choose ‘on an internship’ or
‘Developing a professional portfolio/creative practice’
above full-time, paid work
Section B: Q3 and Q4
JOBTITLE and JOBDUTIES
Required for a valid response (at least one of the two)
To be asked third parties
Determines occupations that recent leavers from HE have entered
and identifies leavers who are employed on postdoctoral research
contracts
Must code up to highest level of detail. SOCDLHE2010 allows for up
to 5 digits of detail with 423 codes available in total!
SOCDLHE2010 and KIS
• SOCDLHE2000 replaced with SOCDLHE2010, a more
developed coding frame reflecting the changing
nature and scope of employment activities (423
codes)
• A new version of CASCOT has been released to
reflect the move to SOCDLHE2010 and is
downloaded from Aardvark
• KIS will include a proportion of those leavers who go
into occupations classified within SOC groups 1-3
‘Professional/Managerial’
SOCDLHE2010 and KIS
• Some previous SOC codes (SOC2000) have been
moved from groups <4 into >3 (e.g. paramedics)
• The table below shows examples of the relationships
between SOC2010 codes, SOC groups and KIS
Section B: Q5
EMPBASIS
Required for a valid response
To be asked third parties
Allows analysis of trends in the labour market and the demand for
HE leavers
If more than one box ticked, code the one nearest the top of the list
as ordered in the valid entries (coding manual not questionnaire)
Section B: Q6, Q7 and Q8
SALARY, PTHOURS and ESTEARN
Not required for a valid response
Not to be asked third parties
SALARY enables analysis and comparison of pay in different
employment sectors
PTHOURS will be used to calculate the full-time equivalent of
part-time salaries
If leaver has indicated unpaid work as their EMPBASIS then no value
should exist for SALARY
JOBNO must exist in order to ESTEARN to be returned
Exception warnings exist for SALARY <£10,000 and >£75,000
SALARY and KIS
• The information returned in SALARY will feed into the
KIS (providing threshold criteria met)
• Only salaries for full-time employment (ALLACT) will
be included in KIS
• Any salaries below £10,000 will not be included in
KIS
Section B: Q9 and Q10
EMPNAME and MAKEDO
Required for a valid response
To be asked third parties
Determines the type of industry that leavers enter
HESA now derive SIC2007 codes on behalf of institutions based on
information provided in Q9 and Q10
Informative answers (to both questions) allows for more accurate
coding (up to 60 characters can be returned for EMPTITLE and 200
for EMPDUTIES)
Section B: Q11
EMPPLACE, EMPPCODE and EMPCOUNTRY
Required for a valid response (at least one has to be provided)
To be asked third parties
Enables analysis of graduate mobility and supports the SIC2007
coding
Use online tools to support investigative work – Google maps, Royal
Mail, search engines etc. Postcodes are the most accurate form of
recording location(s) and are easily found online
Standard Industrial Classification (SIC) code
• Centralised SIC coding will be completed by Oblong on behalf
of HESA (on behalf of institutions)
• Data items
– EMPCODE
– EMPCOUNTRY
– EMPPLACE
– MAKEDO
– EMPNAME
• 4 days for completed coding following submission
Section B: Q12 and Q13
QUALREQ and EMPIMP
Not required for a valid response
Not to be asked third parties
Allows for better understanding of the qualification requirements
and variances for obtaining different jobs
If more than one box ticked, code the one nearest the top of the list
as ordered in the valid entries (coding manual not questionnaire)
Section B: Q14
JOBRSNALL
Not required for a valid response
Not to be asked third parties
Captures the different reasons why leavers decide to take different jobs
If the leaver does not complete any reasons or completes more than
one reason but does not identify which is the main reason, please try to
contact the leaver as it is not possible to use professional judgement to
code this particular question
Section B: Q15
JOBFOUND
Not required for a valid response
Not to be asked third parties
Provides information to current students on the different sources
used to find jobs
If more than one box ticked, code the one nearest the top of the list
as ordered in the valid entries (coding manual not questionnaire)
Section B: Q16 and Q17
PREVEMP and PREVWORK
Not required for a valid response
Not to be asked third parties
Used to find out how employment histories affect outcomes and to
provide current students with information
If more than one box ticked (PREVWORK), code the one nearest the
top of the list as ordered in the valid entries (coding manual not
questionnaire)
Section C
Newly qualified teacher status
Section C: Q18 and Q19
EMPLDTEACH and GTCSTIS
EMPLDTEACH required for a valid response
GTCSTIS not required for a valid response (and only applicable to
Scottish institutions)
EMPLDTEACH to be asked third parties
GTCSTIS not to be asked third parties
Used to identify those in teaching jobs and those in teaching jobs
through the GTC Scotland Induction Scheme
Section C: Q20 and Q21
TEACHFUND and TEACHPHS
Not required for a valid response
Not to be asked third parties
Identifies the split between the sectors and primary and secondary
levels
If more than one box ticked, code the one nearest the top of the list
as ordered in the valid entries (coding manual not questionnaire)
Section C: Q22
SEEKTEACH
Not required for a valid response
To be asked third parties
To identify leavers who are seeking a teaching post
Section D
Further study, training or research
Section D: Q23
COURSENO
Not required for a valid response
To be asked third parties
Used to capture the total number of courses leavers are
undertaking
No requirement to return less than 2 courses and only able to
return up to 9 courses
A new concept…
• If COURSENO is more than one, leavers are asked to
provide details about what they consider to their
main study, training or research
• This doesn’t necessarily mean it has to be the study,
training or research which takes up the most time or
awards the highest qualification. Some leavers may
base their decision on their future plans
Section D: Q24
TYPEQUAL
Not required for a valid response
To be asked third parties
Used to monitor the types of further qualifications leavers aim for
next
If more than one box ticked, code the one nearest the top of the list
as ordered in the valid entries (coding manual not questionnaire)
Section D: Q25 and Q26
COURSENAME and COURSESUBJECT
Required for a valid response (at least one of the two)
To be asked third parties
Allows analysis of subjects of further study, training and research
Third parties should now be asked both questions (change to
coding manual)
Joint Academic Coding system (JACS 3.0) code
• Owned, maintained and updated (Version 3.0) by both
HESA and UCAS and used to code HE provision subjects
• Should be coded to the highest level possible (1552
codes available), however no coding tool available
• Field only allows one JACS code to be returned. If
combined studies, leaver must specify one and if
returned by post, institution must use professional
judgement in returning one code (minor/major will
provide an indication)
Section D: Q27
UCNAME
Not required for a valid response
To be asked third parties
Used to derive UCPROV which monitors the movements of leavers
If the name of the institution cannot be derived to a UK university or
college, Non-UK university or college or UK FE college, institutions
should use their own judgement when deciding on coding as Other
public body in the UK or Other private body in the UK
Section D: Q28
STUDYFUND
Not required for a valid response
To be asked third parties
Provides information to current students about out how leavers
fund their study, training or research
If more than one box ticked, code the one nearest the top of the list
as ordered in the valid entries (coding manual not questionnaire)
Section E
Overall Higher Education experience
Section E: Q29, Q30 and Q31
HEWORKEXP, HESTUDYEXP and HEBUSNEXP
Required for a valid response
Not to be asked third parties
Will be used to understand how well the leaver feels that their HE
experience provided them with skills, competencies and knowledge
for the applicable activities
Leavers should complete (and be asked) all questions regardless of
current activities / activities on census date
Section F
Thank you
Section F Thank you for completing this
questionnaire
The end is near but the final part of the questionnaire will
ultimately determine whether the responses can be
returned to HESA…
The form must be signed and dated!
This can be by the leaver, caller or even a third party
Remember only leavers should be asked if they do not want
to be included in the follow-up survey. If the questionnaire
is completed by a third party the leaver must be excluded
from longitudinal DLHE
Data collection
The data model
XML
• XML standards for
• Survey responses will
eXtensible Mark-up
be inputted into a
Language
student record system
or database which will
• It is the international
be capable of producing
standard for data
an XML DLHE file
transfer
• The correct format
• Hierarchical data
structure must be
structures in a single file
followed to avoid errors
XML template in Excel can be used exceptionally
Validation kit
• Available to download from HESA website now
• Allows for local validation (download it to PC as
opposed to a network)…
• …ensuring you maximise submission time
• User friendly interface
• One kit framework for all HESA XML data collections
• Framework connects to web to download latest rule
version
• Demonstration of kit….
Submitting data
•
•
•
•
The process of submitting data remains unchanged
NEW TEST_COMMIT transaction
NEW! Data supply files available at COMMIT….
…downloadable, flattened data files which also
contain derived fields used for analysis
• Match reports B and C should be looked at, however
it is okay to send data for students not in POPDLHE –
we simply will disregard it
• Use the TQI downloadable files!
Check documentation….
Sign-off
• Data should be signed-off by the Head or Acting
Head of institution
• Some evidence to suggest this is not happening
• Escalate responsibility!
HEFCE
Preparing for audit
Welcome slide
DLHE Advanced Seminar
Preparing for audit
Ewa Wawrzynska
Leeds
25 September 2012
Pilot exercise of audits of the DLHE
survey
In spring 2012 HEFCE conducted a pilot exercise of audits of the DLHE
2009/10 survey:
• New area of HEFCE audit activity;
• 12 institutions selected;
• Reasons for audit activity:
• DLHE feeds into KIS;
• concerns over data reliability within the sector;
• need for assurance over the DLHE process in institutions;
• identifying good practice at institutions;
• identifying areas of difficulty, providing advice on these and
improving training and guidance on these where required.
Pilot exercise of audits of the DLHE
survey (continued)
• HESA involvement;
• Summary report on the findings to be published on the HEFCE
website.
Key messages
• Audit of the DLHE process and how institutions implement it,
rather than the primary source of the data;
• No evidence of data manipulation found;
• Our audit work in this area will continue.
Audit programme
Overview of the audit process:
• Pre-visit data analysis;
• One day on-site visit;
• Discussion of the DLHE process;
• Discussion of the data;
• Detailed testing – sample;
• Feedback meeting at the end of the visit;
• Reporting arrangements.
Issues encountered
• Insufficient Data Protection Notice;
• Lack of evidence of attempts made to contact the leavers, for
leavers for whom no response was obtained;
• Sign-off by individuals other than the Head of institution;
• Implications of using the centrally hosted online DLHE survey as a
data entry tool;
• Institutions returning data for unanswered multiple choice
questions;
• PROFSOCT field coding;
• Salary question;
Issues encountered (continued)
• Audit trail to the supplementary data;
• Lack of supporting evidence for data returned to HESA (individual
cases);
• Responses obtained from leavers not returned to HESA (individual
cases at two institutions, sample size was 50 in both cases);
• Sporadic data entry errors.
Other observations
• Timing of the visits;
• Variable practices around advertising the DLHE survey;
• Good practice points;
• Complaints about the timing of the survey;
• Match B and Match C reports;
• Validation of responses;
• Interpretation of responses;
• Date on the questionnaire (front and back).
Discussions with HESA.
Next steps
• Summary report to be published on the HEFCE website;
• Audit programme to be published on the HEFCE website;
• This activity will continue for the foreseeable future.
Thank you for listening
e.wawrzynska@hefce.ac.uk
DLHE and KIS
TQI tables
Key Information Set (KIS) and DLHE
•
•
-
DLHE data is used extensively in the KIS
Remember the publication thresholds:
23 respondents must have responded
For salary data to be published at least 23 students who
are employed full-time must have given a salary and
these must represent at least 50% of the students
employed full-time
• Where thresholds are not met the data will be
aggregated from a course level to a subject level
• Salaries under £10,000 will not be included
• Respondents who are working full-time and studying fulltime will not be included
Using the TQI tables to benchmark for KIS…
HESA-DLHE@JISCMAIL.AC.UK
HESA information and discussion forum for HESA DLHE
These lists are used to circulate email news alerts from
HESA regarding data requirements, coding manuals and
validation kit releases
Want to join?
email liaison@hesa.ac.uk
or visit www.jiscmail.ac.uk
Data collection and submission
The Institutional Liaison team are there to help with all
aspects of the return from local collection of data items
through to final submission and sign-off
Contact the team
liaison@hesa.ac.uk
01242 211144
Training support
If you require additional training help, including
bespoke visits to your institution, get in touch with
the training department…
w:
e:
t:
www.hesa.ac.uk/training
training@hesa.ac.uk
01242 211472
Follow us on Twitter: @HESATraining