Early Evaluation of KIS/Unistats
Institutional perspective
Report to HEFCE by the Higher Education Statistics Agency
Catherine Benfield
May 2013
© HEFCE 2013
Page 1 of 16
Executive Summary
1.
This report provides a summary of the process and outcomes from the Early Evaluation of the Key
Information Set (KIS), Strand B, which focused on the experiences and views of higher education
providers following the first year of the KIS collection and Unistats publication.
2.
The input to the evaluation was obtained from a survey of institutions, a set of workshops with, in
total, about 40 institutions, and further interaction with institutions at a number of events
organised by others. In addition, account was taken of the very early findings from the other
strands of the Early Evaluation which were looking at the user experience of the Unistats site,
making it possible to address these for year two. Detailed specification of changes was undertaken
by the Higher Education Funding Council for England (HEFCE) and the Higher Education
Statistics Agency (HESA) and documentation for year two issued to institutions in December 2012.
3.
Significant changes have been made to the specification of KIS to allow geographic locations to be
attached to each course and allow courses to be displayed against the teaching institution rather
than the institution that registers the students. In addition, a number of changes have been made
to the way data is captured to improve consistency of display and allow additional searching and
filtering functionality on the Unistats site.
Background
4.
HESA was commissioned by HEFCE to undertake the Early Evaluation of KIS from the
perspective of higher education (HE) providers. This was intended to address the practical issues
related to the compilation and submission of the KIS Record, together with any concerns about
aspects of the data specification and the consequent display of information on the Unistats website.
The intention was to identify changes and updates that might be made to the specification and
processing for the second year.
5.
From an institution’s perspective, the compilation of the information for publication on Unistats is
not straightforward as it involves linking a number of datasets across years. While the institutions
do not undertake this linking and compilation themselves, they do need to understand how it
works as much of the data they provide directly in the KIS Record provides the ‘glue’ for the
linking.
Page 2 of 16
6.
The KIS Record is a course-based record, while most of the linked datasets are person-based. The
institutions, in preparing their KIS Record submissions, need to ensure that the correct individuals
from the Student/Individual Learner Record (ILR) returns (and consequently Destinations of
Leavers from Higher education (DLHE) and National Student Survey (NSS) surveys) will be
linked to the correct KIS courses, as set out in Figure 1 below.
Figure 1 Data Linkages
DLHE
2010/11
UCAS fees for
2013/14
NSS
2012
HESA
Student/ILR
2010/11
KIS Record
2012
Unistats dataset
2012
HESA
Student/ILR
2009/10
NSS
2011
DLHE
2009/10
7.
In order to understand any practical difficulties institutions had experienced and any issues that
they had with the either the KIS Record data specification or the Unistats compilation, HESA
undertook a survey of submitting institutions. This was timed to follow institutions’ completion of
KIS Record submission for year one (August 2012). Of the 324 providers that submitted data to
HEFCE, 130 responded to the survey (85 Higher Education Institutions (HEIs), 42 Further
Education Colleges (FECs) and 3 alternative providers (APs)). It should be noted that this survey
took place before institutions had seen the resultant information published on Unistats, though
they would have seen a preview of their own information.
8.
The survey asked about the practicalities of how KIS work had been organised and managed in the
institution and any issues related to the technical specifications, and asked for suggestions for
change. The full questionnaire is at Annex A. At the end of the survey providers were asked to
indicate their willingness and availability to attend one of a series of three workshop sessions to be
held at HESA in late September/early October.
9.
Responses to the survey and hence discussions at the workshops focused on a desire from
institutions to provide prospective students with information that was consistent and not
Page 3 of 16
misleading while making the process of data compilation efficient and straightforward for
institutions.
10.
Each workshop followed a different agenda. The first focused on practicalities, the second on
issues related to the data specification, and the final one included interaction with the Universities
and Colleges Admissions Service (UCAS), regarding submission of fees data, and the Quality
Assurance Agency (QAA), in relation to the definitions of teaching and learning and assessment
types. Each workshop was attended by 12-15 representatives from institutions (HEIs, FECs and
APs) plus staff from HESA, HEFCE and, for the final one, UCAS and QAA.
11.
In addition, analysis of the data returned by institutions and review of the information on the
Unistats site by staff at HESA, HEFCE and others identified a number of issues relating
particularly to consistency.
12.
To assist in reaching the conclusions presented in this report, presentations were also made at a
number of sector events. This allowed sharing of ideas and work in progress, feedback from and
discussion with institutions before proposals were finalised. The report focuses first on changes to
the specifications that impact directly on what appears on Unistats. A later section considers
practical issues relating to the processes of preparation and submission of the data.
Data specification
13.
From the workshops and consequent feedback emerged a set of proposals for change to the data
specification which have subsequently been further discussed at sector events and developed in
detail by HESA and HEFCE. Documentation for institutions setting out the KIS-specific data to be
submitted for year two was published in December 2012 following agreement to these proposals
by the Higher Education Public Information Steering Group (HEPISG). The proposals relate both
to the specific KIS Record which institutions submit and also to the methodology for compiling the
Unistats dataset from the KIS Record and other datasets, as shown in the earlier diagram.
Location
14.
For year one the only geographic information available for Unistats was the location of each
institution’s main campus. No link was made between individual courses and the locations where
these were taught. Consequently it is not currently possible on the Unistats site for prospective
students to search for courses available in a particular locality or using distance-based criteria.
15.
Also in year one, institutions were asked to provide summary information about accommodation
across the institution: reporting costs averaged across all teaching locations. Where institutions
have geographically disparate teaching facilities this can result in misleading information for
students. Some locations may have institution-owned accommodation while others do not and
private rental costs could be very different in different towns.
16.
Institutions expressed concerns that this data structure did not allow them to provide appropriate
information and asked that location-specific information be captured about accommodation and
each course tagged with a location.
17.
Early evaluation work undertaken with users of Unistats also identified a clear requirement for
geographic search facilities.
Page 4 of 16
18.
In order to address the issues coming from both users and institutions, it is planned that for year
two institutions provide geographic and accommodation information for each teaching location
and then add a location identifier to each KIS course.
19.
Geographic information will take the form of longitude and latitude coordinates which will allow
for accurate mapping and distance calculations. It is intended that relevant distance learning
provision be included in all geographic search results.
Accreditation
20.
Information is included in the KIS to inform prospective students if a course is accredited by one
or more professional or regulatory bodies. In year one, each institution was asked to write its own
text to describe accreditation arrangements for courses and provide relevant links to the web sites
of the accrediting bodies. This information was then included on Unistats. Institutions indicated in
response to the survey that this information was generally not held centrally and had been collated
specifically for inclusion in the KIS. There was also a concern that the sentence structures that had
been suggested did not fit all circumstances.
21.
Analysis of the consequent information on Unistats revealed that institutions had found many
ways of saying what should have been the same thing and had submitted a wide variety of
Professional, Statutory and Regulatory Body (PSRB) website links. This is likely to be leading to
confusion for prospective students who cannot be clear which courses are providing the same
professional progression. For example, the only accreditation that can be attributed to a veterinary
nursing course is ‘eligibility to register as a veterinary nurse’ but the following represent a range of
ways that institutions have chosen to state this and include between them links to four different
webpages:

‘This course is recognised by the Royal College of Veterinary Science
(http://www.rcvs.org.uk/registration/how-to-register/registering-as-a-newly-qualifiednurse/) for the purpose of Registration.’

‘This course is recognised by the Royal College of Veterinary Science
(http://www.rcvs.org.uk/education/degrees-in-veterinary-nursing) for the purpose of
REGISTRATION (RCVS).’

‘This course is recognised by the Royal College of Veterinary Science
(http://www.duchy.ac.uk/online/course/100198) for the purpose of Graduates are able
register with the RCVS as Registered Veterinary Nurses.’

‘This course is recognised by the Royal College of Veterinary Science
(http://www.rcvs.org.uk/education/degrees-in-veterinary-nursing/) for the purpose of
registration as a Registered Veterinary Nurse.’

‘This course is accredited by the Royal College of Veterinary
(http://www.rcvs.org.uk/document-library/list-of-rcvs-approved-qualifications-inveterinary-nursing/).’

‘This course is recognised by the Royal College of Veterinary Science
(http://www.rcvs.org.uk/document-library/list-of-rcvs-approved-qualifications-inveterinary-nursing/) for the purpose of Entry onto the Register of Veterinary Nurses.’
Science
Page 5 of 16
22.
When these and similar observations were shared with institutions at the workshops, it also
became clear that compiling the text statements, identifying the appropriate (external) web links
and monitoring that the latter continue to be valid presents a significant challenge. Those
responding to the survey and attending the workshops encouraged HESA to consider a more
centralised system for managing this information.
23.
In response to this discussion, HESA and HEFCE have now developed standard statements to
describe each type of accreditation for each accrediting body and identified appropriate web links.
These have been shared with the accrediting bodies which have been given the opportunity to edit
where necessary. For year two, institutions will indicate as part of their KIS return which
statements from a coded list apply to which courses and the relevant statements will then be
included on the course pages on the Unistats website, along with standard web links.
Course titles
24.
Another scan of the Unistats website reveals that the provision by institutions of course titles as
free text has also resulted in a lack of consistency for users. Browsing the Unistats website shows
BSc (Hons) Chemistry written as ‘B.Sc HONS Chemistry’, ‘Chemistry BSC (HONS)’ etc.
25.
For year two HESA has prepared a coded list of all of the possible qualification types and it is
planned that this information be collected separately from the remainder of the course title
allowing consistent display and, additionally, searching by detailed qualification aim.
Additional information to improve Unistats searching
26.
For the first year institutions were instructed to provide a KIS for a part-time course only if there
was no full-time equivalent course offered. This was intended to limit the burden on institutions of
providing KIS. However, as a consequence those using Unistats to search for a part-time
programme were not able to obtain a complete list of courses available to them.
27.
Both in response to the survey and at the workshops, institutions expressed concern that some
part-time provision was not visible to students. For example, where an institution had longstanding part-time courses, with good NSS and DLHE results, but had recently started to offer the
same courses on a full-time basis (no NSS or DLHE yet) then the earlier part-time data was not
displayed on the site as no KIS was produced for the course.
28.
Following discussions, it is planned that for year two, separate KIS should be produced for fulltime and part-time versions of courses where both are advertised. Where the two courses are
identical other than the mode of study, the institution need only return one set of KIS data from
which two KIS will be produced for inclusion on Unistats. This will enable those searching for
part-time courses to see all those that are advertised.
29.
In addition institutions indicated that it would be useful to capture, in a more structured way than
just through course titles, elements of courses for which students might want to search. It has been
agreed that for each KIS course flags will indicate if the course contains compulsory or optional
foundation years, sandwich years or years abroad.
Page 6 of 16
Allocation of students to teaching institution
30.
There were concerns raised throughout the review process, and reported in The Times Higher on
15 November 2012, about the need to ensure the correct allocation of students to institutions for
the purposes of reporting NSS and DLHE results. This is complicated by the reporting
arrangements for some collaborative provision, most especially franchised provision, and added to
by the fact that collaborative arrangements may change over time meaning that different cohorts
on the same course are registered by, and hence reported by, different institutions.
31.
HESA and HEFCE have developed a methodology that will allow students to be reallocated from
the registering institution to the teaching institution and hence KIS prepared and reported for the
teaching institutions that include data for all of the students they teach. This will require those in
collaborative partnerships to work closely together, sharing course information and identifiers,
and will increase the number of institutions required to submit a KIS return. Full details of how
this works have already been published on the HESA website (http://www.hesa.ac.uk/includes/
C13061_resources/KIS_collaborative_provision.pdf).
Teaching and learning and assessment
32.
In response to the survey institutions indicated that the most difficult and resource-intensive part
of preparing the KIS return was capturing the data relating to assessment and teaching and
learning. Institutions reported a wide range of issues which can be summarised as follows:
a.
Difficulties in interpreting the definitions provided for different activities and applying these
to local provision – a desire for more examples.
b.
This data was not held centrally in many institutions and required significant effort to
collate.
c.
Specific difficulties in allocating activities for subjects with significant practical work – for
example creative and performance arts – which may be supervised but not scheduled.
d.
Difficulties in how to categorise clinical and other placement work which constitutes
significant parts of medical, nursing and related courses and also teacher training.
e.
It was difficult to understand how this data should be compiled for distance learning
courses.
f.
Putting placements and work-based learning in separate categories resulted in some
confusing information for students.
g.
Some courses require students to commit more than the 1200 hours for 120 credits that the
data is intended to represent.
h.
The requirement to discount ‘travel time’ from scheduled teaching time had been very
difficult to implement as it meant looking at detailed timetables for each course to see the
number and lengths of individual scheduled sessions rather than just knowing that a module
included say 20 scheduled hours.
i.
Academics were keen to include ‘office hours’, that is hours where they are available for
students to see them, in scheduled teaching and learning as these hours take up a lot of their
time.
Page 7 of 16
33.
34.
35.
Following discussion the following changes have been made for year two:
a.
Institutions are not required to deduct travel time from scheduled sessions (see h. above)
b.
Work-based learning is now to be included with placements (f. above and should also assist
with d.)
In respect of the other concerns identified, it was agreed that there should be no change from year
one in respect of the following:
a.
The focus should be on the requirements placed on the students and not on the effort of staff
so that office hours (i. above), should not be included as scheduled teaching and learning
unless there is a requirement on students to take advantage of these for a certain amount of
time and similarly with practical work (c. above) which should be included to the extent that
the student is expected to attend.
b.
That distance learning (e. above) should be regarded as consisting mainly of guided
independent study except where attendance (in person or virtually) was required at a
specific scheduled event.
HESA will work with QAA to bring together additional examples (a. above) for inclusion in the
documentation.
KIS Types
36.
The specification for KIS for year one identified three types of KIS. The first (type 1) included a full
set of information for a course and was to be used for all single subject courses and all multiple
subject courses with an intake of more than 20 students. Type 2 KIS were intended to be
‘placeholders’ for small multiple subject courses (intakes less than 20) and would provide links to
two or three type 3 KIS, which contained the remaining information at a subject level (more
detailed
descriptions
of
the
different
KIS
types
are
available
at
http://www.hesa.ac.uk/C13061/Coverage). This design was intended to limit the burden on
institutions with large numbers of multiple subject courses – created by a combined honours
programme for example. In the event, only 44 (out of 336) institutions returned any type 2 KIS.
However among those 44, 56% of the KIS returned were type 2. Overall 24% of the KIS returned
were type 2. Many institutions indicated that they had found the different KIS-type concepts
difficult to understand and implementing these actually added burden and complexity.
Additionally using type 2 meant that some relevant information could not be supplied
(accreditation for example).
37.
For year two institutions have been given more flexibility over the use of type 1 and type 2 KIS,
meaning that where it is less burdensome for institutions they can provide a type 1 KIS for all
courses regardless of entry cohort size. In addition, for those institutions who choose to continue to
supply type 2 KIS, the range of data that can be included has been extended to include
accreditation and fees data.
Page 8 of 16
Practicalities
38.
From all of the workshops it became obvious that responsibility for KIS in institutions had been
given to individuals with a very wide range of ‘day jobs’, and that the work had been organised in
different ways. This is summarised in Figure 2. In FECs there was most commonly a ‘one man
band’ consisting of the HE manager.
Figure 2 Institutional Working Arrangements
39.
Many institutions indicated that they would need to reconsider how to manage this work on an
on-going basis as the arrangements put in place for year one were not necessarily appropriate or
sustainable in the long term. A summary of the full range of comments received in the survey
relating to work loads and organisation/management issues is at Annex B.
40.
A consequence of these differing institutional arrangements was that many of those involved in
preparing the KIS Record had no previous experience of HESA and so found both the
documentation and the requirement to work with Extensible Mark-up Language (XML)
unfamiliar. The significant changes and updates between first issue of documentation (September
2011) and final submission made the process more challenging for institutions and may have
contributed to inconsistent data. Many institutions had not identified a specific KIS contact to
HEFCE or HESA until later in the process which meant that the correct people were not always
receiving notification of updates.
41.
This absence of identified contacts had also caused some institutions to miss out on training
opportunities. Those who attended training events reported that they had found these useful. In
the main, institutions also reported that they had found the helpdesks at HEFCE and HESA very
helpful in responding to queries. However, there was some confusion, particularly for FECs, as to
who was able to answer queries, HESA or HEFCE. While HESA was hosting all of the
documentation, HEFCE was responsible for data collection, meaning that HESA staff could not
answer specific queries relating to an institution’s data submission. Additionally, it had been
agreed that FECs should approach HEFCE with all of their queries, even those relating to the
HESA-hosted documentation. This situation should be improved for year two as HESA will be
undertaking the data collection for HEIs, meaning that HEIs should be directing all of their queries
to HESA while FECs direct all of theirs to HEFCE, which will continue be responsible for FEC data
submissions.
Page 9 of 16
42.
At the first workshop a number of detailed and specific points were raised by institutions, changes
to which would improve processes for year two. HESA has considered many of these in preparing
both the documentation and the system specifications for the collection system: for example,
changes to the output reports for institutions to make these easier to check, notification to
institutions of any updates to the FAQ documents and wide notification of training events.
43.
There was specific discussion at the workshops about the arrangements for institutions to sign off
their data for publication on the Unistats site. Institutions will need to sign off their initial
submission and then sign off again any updates they make during the year. HEFCE had used an
electronic sign-off process for year one which did not match with the paper process which HESA
currently uses for all of its data collections. Further consideration is still being given to future
arrangements.
44.
Additionally, institutions expressed the view that they were content that updated fees data, routed
into Unistats via UCAS, did not require sign-off from the institution as part of the KIS process.
However, following discussion with HEFCE the sign-off requirement will remain.
Conclusion
45.
The Early Evaluation process provided a wealth of useful and constructive feedback from
institutions relating to year one of the KIS and Unistats processing and enabled a range of
specification and practical changes and updates to be identified for year two. Many of these have
already been implemented and institutions notified through the publication of the documentation
on the HESA website. Systems for data collection and processing are currently in development and
training events in progress.
Page 10 of 16
Acronyms
AP
Alternative Provider
DLHE
Destinations of Leavers from Higher Education
FEC
Further Education College
FT
Full-time
HE
Higher Education
HEFCE
Higher Education Funding Council for England
HEI
Higher Education Institution
HEPISG
Higher Education Public Information Steering Group
HESA
Higher Education Statistics Agency
ILR
Individual Learner Record
KIS
Key Information Set
NSS
National Student Survey
PSRB
Professional, Statutory and Regulatory Body
PT
Part-time
QAA
Quality Assurance Agency
SROC
Student Records Officers Conference
UCAS
Universities and Colleges Admissions Service
XML
Extensible Mark-up Language
Page 11 of 16
Annex A
Early Evaluation of KIS Questionnaire
As part of the Early Evaluation of KIS, HESA is undertaking this survey to gather views from
participating providers on their experiences of the KIS process. The aim is to identify anything in the
data requirements or collection process which, from providers’ perspectives, is fundamentally ‘broken’
or disproportionately difficult or burdensome to do and any improvements that could be made in the
short term.
1.
Name
2.
Job Title
3.
Department
4.
Institution
5.
Provider

Higher Education Institution

Further Education College

Alternative Provider
Organisation and Implementation of KIS
6.
How did your HEI manage/organise the process of compiling the data for the KIS record? Who
was involved? How was the process governed?
7.
How well do the KISTYPE definitions and rules work for your HEI?
8.
Were there any issues in gathering data from other departments at your HEI? If so, how are these
being addressed?
9.
How much did your HEI have to make changes to its website to accommodate KIS and the
widget?

Very little, it slotted in fine

Minor changes but site layout remained broadly unchanged

Significant changes requiring overhaul of site

Any additional comments
Page 12 of 16
Data Specification
10.
Which concepts in the KIS were easy to understand? And which were more difficult?
11.
Which data were easier to collect (i.e. fit to the HEI's own data and readily available)?
12.
Which data were more difficult to collect?

Any suggestions as to how these could be made easier?
13.
Have you identified any specific changes you think are needed to the record specification?

Note - the following have already been identified:
i.
ii.
iii.
iv.
v.
vi.
Adding a location indicator to each course and collecting accommodation costs by location
Looking for a solution to the need for two KIS when students registered at an HEI and an FEC
are on the same course
Additional PSRBs to be included in valid entries
Capturing three years of HESA COURSEIDs to support continuation statistics
PT and FT KIS - should the practise of only providing PT specific KIS where no FT course exists
be retained
Type 2 KIS - are the data items right?
Compilation of the KIS dataset
14.
Were there any issues for your HEI in the way that the data was compiled from the various
datasets? (the algorithms, timing, linking etc.)
Submission and Checking
15.
How did your HEI find the process of data submission?

Very straightforward

Fairly straightforward

Difficult

Very difficult

Any additional comments
16.
Were the reports provided by HEFCE useful for checking the data?

Very useful

Fairly useful

Not useful
17.
Are there extra/alternative reports which would assist in this process?
Page 13 of 16
18.
Did the HESA documentation provide the information you needed to compile the return?

All of it

Most of it

Some of it

Any additional comments
19.
Did your HEI attend one of the HESA training events? If so did you find it useful?
20.
Has your HEI contacted HESA Liaison/HEFCE helpdesk? Did you get the information you
needed?
21.
What other methods of support has your institution used (e.g. SROC, mailing lists, software
houses, etc.)? Were they useful?
Page 14 of 16
Annex B
Summary of responses to survey questions about the process of compiling and submitting data
15.
How did your HEI find the process of data submission?
a)
Very straightforward – 31
b)
Fairly straightforward – 77
c)
Difficult – 26
d)
Very difficult – 4
Comments

HEFCE extranet issues – freezes/confusing and required technical help (failed weekend prior
to submission)

Difficulty around validation

Process for sign-off should be clearer
16.
Were the reports provided by HEFCE useful for checking the data?
a)
Very useful – 52
b)
Fairly useful – 74
c)
Not useful – 12
Comments

Could have done with more information around aggregation

Had to delete temporary internet files to use

More guidance over errors/general problems would be useful

Generally quite useful

Not easy to print
17.
Are there extra/alternative reports which would assist in this process?

Majority said no – other responses included
o
A single document with UCAS, HEFCE and HESA stamp on it
o
Would be useful to receive final KIS data in a tabbed excel spread sheet rather than
XML format to allow easy import into access
o
Further guidance on why errors are occurring
o
Less technical language in describing errors
o
A report providing a breakdown of how the NSS and DLHE data is aggregated would
be useful
Page 15 of 16
18.
Did the HESA documentation provide the information you needed to compile the return?
a)
All of it – 28
b)
Most of it – 94
c)
Some of it – 16
Comments

For those with less technical knowledge of coding and XML, quite difficult to understand

Documentation too word dense and detailed

Some areas required contact with HESA/HEFCE for clarification

Definitions can be perceived in a different way by different people/institutions

Email whenever significant updates take place otherwise they get lost
19.
Did your HEI attend one of the HESA training events? If so did you find it useful?

Yes – was useful

Yes – not so useful for FECs

No – was fully booked

No – didn’t know about them
20.
Has your HEI contacted HESA Liaison/HEFCE helpdesk? Did you get the information you
needed?

Yes – prompt responses, very useful
21.
What other methods of support has your institution used (e.g. SROC, mailing lists, software
houses, etc.)? Were they useful?

Other HEIs – Yes

HEFCE – Yes

FE Management Information Systems Managers mailing list – Yes

SROC LinkedIn – Yes, although could be monitored in order to be sure that information
shared is accurate

JISCMail forums – Yes

Software houses - Yes
Page 16 of 16