UCL Response to HEFCE Circular 23/2006 TQI/NSS
Questions from Annex C: proposals for the future TQI site
Question 1: Do you agree that data should be presented at level 3 in the subject
hierarchy, wherever possible?
Given that the present level of definition in the subject hierarchy is unhelpful, we have no
objection to this suggestion, subject to reasonable minimum participation thresholds being
set.
Question 2: Do you agree that the threshold for publishing NSS results should be
lowered to a 40% response rate, and 20 students responding?
We have no major objections to this proposal, although we do have some concerns that
lowering the threshold might affect the reliability of the data. We would suggest that, rather
than manipulating the methodology for analysing the results, further consideration should be
given to the operation (but not the principle) of the NSS. One way of improving response
rates might be to devolve the operation of the NSS to individual institutions, using a basic set
of questions and allowing institutions to add from the expanded list of questions to suit their
needs.
We also note the comment at Annex E (paragraph 16) of the documents that NSS results,
whilst appearing to be stable at the sector level between the 2005 and 2006 surveys, do
show “a fair degree of change when looking at particular institutions”. Our own analysis of
the results at UCL shows that, apart from one subject area, the variation is negligible. We
agree that further analysis would be required in order to determine the extent of and reasons
for change between years. UCL also has reservations regarding the scoring system for the
survey. For example, what in practice does a score of 4.1 really mean, or what is the
difference between a score of 4.2 and 4.3? There is also the unresolved issue of how much
weighting students give to each of the questions asked and whether they regard some
categories of questions as more important than others.
Question 3: Do you agree with the principle that data should be aggregated across
years and/or subjects, where it is unavailable at level three using one year’s data? Are
there any options for aggregating data that should not be offered to users?
We note that allowing custom aggregation of level 3 data would make more likely the
unintentional disclosure of information that the thresholds in Question 2 are intended to
prevent. We also note that amalgamating the views of students from different years might
compromise the data, as students in different cohorts might have very different experiences.
This is particularly relevant at subject level, where individual programmes can experience
substantial change throughout the years. There is also the matter of specific events (such
as industrial disputes or individual departments being subject to building disruption) which
can impact on the experience of particular cohorts of students.
Question 4: Do you have any comments about the suitability of the proposed items
for a summary of key data?
UCL notes the following:

metric C (“the proportion of honours degree students that achieved firsts and twoones”) is not viable for those programmes of study where the usual honours system
is not used (e.g. the M.B.,B.S.), and we would ask that this is made clear to users of
the data;

we strongly oppose metric D (“the proportion of leavers in graduate jobs”) for two
reasons:
1. Some undergraduate programmes of study are very likely to be followed by a
period of further study for professional reasons (e.g. architecture) and some
institutions will have a disproportionate number of their students going into
postgraduate level study (such as PhDs). Therefore it is misleading to consider
employment as the sole valid destinations for graduates.
2. We are concerned that the distinction between ‘graduate jobs’ and ‘nongraduate jobs’ is an artificial one and that the current definitions are inadequate;
users’ interests would be better served by not attempting to make this distinction;

we also question the use of metric E (“the distribution of responses to question 22 of
the NSS about ‘overall satisfaction’ with course quality” which could be the
distribution between ‘positive’, ‘neutral’, and ‘negative’ responses). This will already
be apparent in the published NSS results and it is not clear how positive or neutral
scores would be defined given the uncertainties of the scoring system.
Question 5: Do you have any comments about the proposed student profile data?
UCL has no comments regarding the proposed student profile data, although we think it is
worth considering whether additional profiles could be included, such as ethnicity and
disability.
22