UCL response to QAA consultation on the Operational Description
for institutional review of higher education institutions in England
and Northern Ireland
Consultation question 1
a. Do you agree that the changes noted take account of the requirements of the
sponsoring bodies and the QAS principles?
b. Will the changes help to strengthen management of quality and standards in
institutions?
c. Will the changes provide clearer information about quality and standards to a
variety of stakeholders?
1
UCL welcomes the opportunity to comment on the consultation. The consultation
process is taking place at a time of great change in the funding and regulation of the UK HE
sector and it is important to see it within the context of this broader debate. UCL welcomes
some elements of the proposed Operational Description as these will strengthen the review
process. The emphasis on the concept of threshold standards is particularly welcome as it
emphasises that the role of institutional review is to provide assurance to the public that there
is broad comparability of threshold standards across the sector but recognises that the
considerable diversity of different institutional missions and cultures means that the pursuit of
complete comparability is not a realistic goal. In particular we broadly endorse the following
aspects of the new institutional review process:

the proposal that the institutional review process will be more flexible with judgements
capable of being revised where appropriate, provided, of course, that the core
elements of the process, as set out in paragraph 1, are rigorously and consistently
applied;

the much stronger element of student engagement in the revised process which
rightly places the student learning experience at the heart of institutional review;

the stated aspiration to reduce the burden of the process on HEIs through the
reduction in the number of review visits to an HEI and an increased emphasis on
electronic submission of documentation etc. However, we feel that there is little
convincing evidence that the combined effects of the revised process will actually
lead to an overall reduction in burden; particularly when some of the additional
elements proposed, such as meetings between the review team and External
Examiners and alumni (usually working people with other commitments), are taken
into consideration. At best the revised process seems to offer no additional burden,
rather than any significant reduction;

the revised approach to the preparation of review reports seems likely to provide
clearer information about quality and standards to a variety of stakeholders and to
increase their value to these stakeholders;

the introduction of ‘affirmations’, which will acknowledge that there is remedial activity
already in train within an institution to address an identified issue and will not form
part of the overall judgement.
Consultation question 2
a. Do you agree that further consideration of the way in which collaborative provision
and overseas provision is reviewed is required?
b. Do you have any suggestions for improving the current processes?
2
The current way in which collaborative provision overseas is reviewed does require
further consideration, in view of the fact that this sort of provision has increased and
expanded considerably in recent years and that public assurance of the quality and
standards of collaborative provision is essential to the overseas reputation of UK HE.
However, any revised process would need to avoid increasing the burden on HEIs or
endangering this reputation by placing unnecessary obstacles in the way of institutions
wishing to enter into valid arrangements with overseas partners.
Consultation question 3
We want to express judgments in a useful, informative and credible way.
a. Does either Annex 2a or 2b do this, and is one more helpful than the other particular
groups of stakeholders?
b. Can you suggest alternatives for the words used to express grades of judgment?
c. If you feel that neither of the given formats is informative or credible, do you have
suggestions of other formats?
d. What will be the challenges for institutions and audit teams in the use of the
judgment formats given as examples?
e. Do you have any suggestions for further guidance on which to base judgments?
3
The main role of institutional review is to assure the public that HEIs are, within the
context of the external framework provided by the Academic Infrastructure, ensuring that the
standards of the institution’s awards meet or exceed threshold standards and that the
learning opportunities provided by HEIs for students enable them to achieve these
standards. Judgements should be limited to whether HEIs are meeting these
responsibilities. Not only are the gradations of judgement proposed in Annex 2b
inappropriate in this context but they would leave institutions vulnerable to inclusion in league
tables and would be very difficult to justify in the face of a legal challenge by an institution;
particularly as some of the guidance pointers in Annex 2b reveal the same lack of definition
which afflicts the QAA guidance on the audience for and use of programme specifications.
For example, the guidance pointers on applying the ‘good’ ‘satisfactory’ and ‘unsatisfactory’
grades of judgement on enhancement (page 37) all refer to ‘the management of this area’
(our italics). Given the difficulties experienced in the previous audit cycle around the
definition of ‘enhancement’ when it did not even form the basis of a judgement, more clarity
would have been expected.
4
A pass/fail approach as detailed in Annex 2a is much more appropriate, although this
could more helpfully be expressed as ‘satisfactory’ and ‘unsatisfactory’. The guidance
pointers for making a pass/fail judgement are helpful for HEIs undergoing reviews, and it will
be easy for the public to grasp the meaning of these judgements.
5
The refocusing of emphasis in the draft Operational Description from judgements
made on the management of quality and standards to judgements in relation to the outcomes
of institutional processes for managing quality and standards is not in itself problematic.
However, the rationale for this change of emphasis as explained in the Operational
Description document (page 6) is unconvincing. The impression gained is that the QAA has
refocused the judgements to look as though standards are being examined (as this is what
appears to be required) but that this has not been underpinned by any substantive change.
The explanation for this change of focus, however, whatever its flaws, will not feature in the
institutional review report and the public reading the report might well be confused about
what exactly is being reviewed. In light of this, UCL supports retaining the current focus on
management of quality and standards.
6
Notwithstanding the above, UCL considers that the emphasis on broad comparability
across the sector of whether institutions are meeting or exceeding threshold standards
remains the most appropriate way forward.
Annex 2a notes that
Consultation question 4
It is intended that all documentation provided for the review team (see Annex 5) will be
uploaded to a secure QAA electronic folder. Do you see any particular challenges for
institutions in providing documentation in this form, or for review teams in using the
documentation?
7
No. At its last institutional audit UCL made all documentation available to the audit
team in an electronic format and this worked well.
Consultation question 5
a. Do you agree that agreed reference points for review should be increased to include
more than the Academic Infrastructure?
b. Is agreement through the substantive/minor changes process (see paragraphs 7381) appropriate?
c. Do you see any problems with using the additional reference points currently being
suggested?
8
It might be more appropriate to consider the issues about reference points once the
revision of the Academic Infrastructure is complete. For example, the emphasis on
programme specifications in sections of the draft Operational Description (section 27 in
particular) seems to have ignored the issues which we and others have raised regarding the
usefulness of programme specifications during the consultation on the future of the
Academic Infrastructure.
9
While we acknowledge that it is important to assure the public that there are external
benchmarks for quality and standards, we think that agreed reference points for review
should NOT be increased to include more than the Academic Infrastructure. The key
elements of the Academic Infrastructure already provide an adequate framework within
which to consider whether HEIs are ensuring that their provision meets or exceeds threshold
standards. To increase this framework would impose an additional regulatory burden on
HEIs, and would not be consistent with the stated aim of a revised Quality Assurance System
to ‘avoid disproportionate use of institutional effort and resources which could otherwise be
directed to the delivery of front-line student teaching’.
10
The substantive/minor changes process is broadly acceptable where the changes are
being made by the QHEG, but the statement in paragraph 78 - ‘in addition the QAA will be
able to make changes to the operation of the review process without reference to the QHEG
or consultation’ needs clarification and we trust that the final version of the Operational
Description will set out in detail what this means. See also our response to question 10
below.
Consultation question 6
a. Do you agree that it would be useful to enhance the role of the institutional contact
as described in paragraphs 35-37?
b. Are there other tasks or responsibilities which the Institutional Facilitator should
carry out?
c. Do you agree that the role of ‘lead student representative’ as described in paragraph
38 should be included in the review process?
11
The role of Institutional Facilitator would be a valuable addition to the institutional
review process as this role can only enhance effective communication between the review
team and the HEI by providing information on institutional structures, policies and procedures
and also, crucially, on institutional mission and culture. However, the advice that ‘a member
of staff who has significant responsibilities relating to the design, maintenance and/or
operation of those arrangements may not feel comfortable with playing a key role in the
review process, or be best placed to provide objective advice and guidance to the review
team’ seems to undermine the effectiveness of this role almost entirely. It should be for
institutions to decide who should undertake this crucial role and it seems likely that in many
institutions it would be a member of staff with a significant role in the management of quality
and standards who is best placed to offer the sort of information envisaged above.
12
We also agree that the role of ‘lead student representative’ as described in paragraph
38 should be included in the review process, with the caveat that these representatives must
receive appropriate training and support from QAA.
Consultation question 7
a. What do you see as the challenges for institutions of online briefing?
b. Is it feasible to expect institutions to start preparing their SED after only an online
briefing and remote contact with the coordinating QAA officer?
13
We have no issues with this, provided that briefing is available sufficiently early to
allow the institution to prepare. It will also be important to ensure that any advice
subsequently offered to the institution by the co-ordinating QAA officer does not contradict
that contained in the briefing.
Consultation question 8
a. Do you agree that there should be provision for review judgments to be changed
after the follow-up process?
b. Should the visitors be peers or can QAA officers carry out this follow-up visit and
give a new judgment?
14
UCL supports the proposal that there should be provision for review judgements to be
changed but only on the understanding that any such revised judgements must be made by
the original review team as opposed to by the QAA. Failure to do this will severely
compromise the stated principle of peer review in relation to such judgements.
Consultation question 9
a. What do you see as the main challenges for institutions of the way in which the
process for thematic elements will operate?
b. Can you suggest more effective ways for the process to operate?
15
Thematic elements are not particularly problematic in themselves, but UCL would
need assurances that these would be based on substantive and generic issues and we
would support the identification of themes for a medium-term period. It is wholly unclear why
this cannot be done. If themes are not identified in this way, there will be a danger that the
setting of themes will be driven by the need to placate or refute media scare stories or in
response to other short-term concerns. The draft protocol published by QHEG on how such
themes would be identified appears to be broadly appropriate.
16
Care will need to be taken in designing the institutional proforma for each theme to
ensure that the additional work entailed in completing it is not disproportionate. Also, the
QAA’s review process has in the past presumed a highly centralised, managerial approach to
quality management and enhancement which may not be the case in all institutions. The
proforma should be designed so that it takes account of the significant variation in
institutional culture and organisational structure to the extent that a variety of approaches can
be accommodated and considered equally valid.
Consultation question 10
a. What do you see as the main challenges for institutions of the way in which the
process for introducing changes to the review process will operate?
b. Can you suggest more effective ways for the process to operate?
17
UCL would support the move to a rolling review of the institutional review process, to
ensure that the process is more flexible. The mechanism proposed for managing changes is
appropriate, but given that the QAA will be able to make changes to the operation of
institutional review without consultation with either the sector or the QHEG, there needs to be
greater clarity about the limits within which this will operate than is provided in the examples
offered by the draft protocol on changes to institutional review.
Consultation question 11
QAA is currently reviewing its complaints and appeals procedures.
a. Are there any comments that you would like to offer about the way that these
procedures currently operate?
b. Appeals are currently allowed against judgments of no confidence only; should the
scope of the appeals scheme be broadened?
18
UCL supports a satisfactory/unsatisfactory judgement (see response to question 3)
therefore appeals should be allowed for judgements of ‘unsatisfactory’.
Consultation question 12
What do you see as the particular challenges to institutions in working to the
proposed review timeline?
19
UCL supports the proposal that reviews be completed within an academic year and
the outcomes published in a more timely fashion. As previously noted, it will be important
that the online briefing materials are available to HEIs as soon as is possible.
Consultation question 13
Are there any other comments you wish to offer about any part of the proposed
method?
20
The lack of a feedback meeting at the end of the review, where the team outlines its
initial impressions to the institution, is baffling (page 23, paragraph 52) and a rationale for this
is not offered. UCL offers this sort of high-level feedback in its own internal quality review
process and would consider it unacceptable not to do so.
21
The proposed processes for following up on institutional review are on the whole
appropriate. The additional requirement of a formal action plan makes compulsory what is
already established internal practice at UCL. However, the proposal that the action plan
should be agreed and published within 10 weeks is not realistic and seems to assume a
compliance-led model which is inconsistent with the hope expressed elsewhere in the
document that pre- and post-review there should be a process of ‘continuous reflection’. It
might well be possible for key officers at institutional level to create an action plan in the
timescale proposed but there would be no time for consultation, engagement or reflection
within the institution.
22
The QAA proposes an annual desk-based analysis of an institution’s published
information on teaching quality (when we know what the information set will be) and we trust
that the final version of the Operational Description will make clear whether these annual
analyses will be made public.