3rd Report to the Higher Education Authority
on progress in the implementation of findings
arising from the
EUA IEP Review of Quality Assurance in Irish Universities
Period: September 2006 – October 2007
November 2007
1. Introduction and Background
The European University Association (EUA) Institutional Evaluation Programme (IEP)
was jointly commissioned in 2003 by the Irish Universities Quality Board (IUQB) and
the Higher Education Authority (HEA) to undertake a review of quality assurance
procedures and their effectiveness in the Irish university sector. Furthermore, a high
level reference panel representative of persons from outside the university sector
was established by the HEA to produce a ‘reflections document’ on quality assurance
in the Irish universities.
The EUA Sectoral report and the report of the high level reference panel were
considered by the HEA at a meeting in early 2005 and the HEA resolved at its
meeting that the IUQB be requested to provide the HEA with an action-oriented
implementation plan in respect of the findings of both reports. The Authority also
decided to maintain an on-going review of progress and development by considering
this issue every six months on the basis of an up-date report from the IUQB.
The EUA reports on the effectiveness of the quality assurance procedures in the Irish
Universities were received by the IUQB between February and March 2005 and each
university produced a response to its university report which was sent to IUQB before
April 2005.
On 25 April 2005, the Review of Quality Assurance in Irish Universities was launched
by the Minister for Education and Science and the individual university reports and
the two sectoral reports were subsequently published.
In its Strategic Plan approved in December 2004, IUQB had already anticipated
need for a task force to monitor the implementation of the recommendations in
review reports. The establishment of this task force in May 2005 complimented
request from the HEA to report on progress in the implementation of findings in
reports arising from the Review of Quality Assurance in Irish Universities.
the
the
the
the
In June 2005, IUQB, as requested, provided the HEA with an action-oriented
implementation plan in respect of the implementation of findings in both sectoral
reports.
In 2005, IUQB submitted the first report to the HEA on Progress in respect of the
implementation of findings in both sectoral reports.
In 2006, IUQB submitted the second report to the HEA on Progress in respect of the
implementation of findings in both sectoral reports.
This third report provides an overview on progress with the recommendations in the
Review of Quality Assurance in Irish Universities.
2
2. EUA University Reports
Recommendations in Reports
The EUA review teams made recommendations concerning quality assurance in the
reports on the individual universities. The universities have published reports in 2005
and 2006 detailing progress in the recommendations. All of these reports are
available on the IUQB website at the following link:
http://www.iuqb.ie/IUQB_EUA_Follow-up.html
3. EUA Sectoral Report
Recommendations in Report
The EUA review teams made 29 specific recommendations concerning quality
assurance in the following areas
•
•
•
•
•
Organisation and planning of QA process (4 recommendations)
Self-assessment phase (7 recommendations)
Peer review phase (3 recommendations)
Quality Improvement (6 recommendations)
Strategic governance and management (9 recommendations)
The universities have published reports in 2005 and 2006 detailing progress in the
recommendations. All of these reports are available on the IUQB website at the
following link:
http://www.iuqb.ie/IUQB_EUA_Follow-up.html
Many of the 29 recommendations related to quality assurance procedures, which
have been incorporated by the individual universities into their quality assurance
practices through changes in the procedures and processes for internal quality
assurance reviews.
A number of the other recommendations required consideration and agreement by
the universities collectively in relation to quality assurance procedures for the sector.
The universities (working under the auspices of the IUA Quality Committee, IUAQC)
agreed these changes. As a further commitment to collective progress on quality
assurance, the universities and IUQB embarked on an update of the Framework for
Quality document during 2006 and 2007.
3
4. Update of Framework Document (A Framework for quality in Irish
Universities)
The above document was published by CHIU in February 2003. There was a need to
update this document as a joint IUA/IUQB document setting out the separate
responsibilities of the universities and IUQB in relation to internal quality assurance
processes and external reviews of the effectiveness of quality assurance processes
respectively. The document established the consonance of the quality assurance
system with the Standards and Guidelines for Quality Assurance in the European
Higher Education Area as adopted in Bergen in May 2005 by the Ministers
responsible for higher education in the Bologna signatory countries.
The framework document was updated by a group jointly drawn from the IUQB
Executive and the IUA Quality Committee (IUAQC).
The 2nd edition of the document entitled A Framework for Quality in Irish Universities:
concerted action for institutional improvement was launched by Mr. Tom Boland,
Chief Executive, HEA at the 5th Annual IUQB Conference in Galway on 12 October
2007.
Appendix 8 of the framework document details the recommendations concerning
quality assurance arising from the 2004-05 review of quality assurance in Irish
universities. The responses of the universities and IUQB to the specific
recommendations are included in the document. The changes occasioned by the
recommendations are detailed in Appendix A. of this report.
The document is being widely distributed with the university community and
internationally and will form part of the documentation given to panel members
undertaking reviews of university academic and administrative units.
4
5. Specific Highlighted Actions taken by Universities in relation to
recommendations in the sectoral (or university reports)
The publication and updating of the Framework for Quality document has been very
helpful in showing the way forward for Irish universities and IUQB. However, quality
improvement can only truly take place within institutions. It is helpful to look at a
number of concrete examples of how the Irish universities have taken on board the
recommendations in the 2004 review. The actions are presented as examples of
practice and the list is not exhaustive. The information was supplied by the university
quality offices following a request by IUQB.
In many cases, it must be recognised that the developments outlined below have
also been influenced by initiatives of government and the HEA arising out of the
recommendations of other reviews such as the 2004 OECD Review of Higher
Education in Ireland. The establishment of the Strategic Innovation Fund is a
particular striking example in point.
Institutional Research
Paragraph 134 in the Sectoral Report. All universities need to strengthen their
capacities for institutional analysis and monitoring, in order to provide better
information for strategic governance and management.
In 2004, at the time of the EUA IEP review, no Irish university had an Institutional
Research officer/director in place. Since the review, DCU, NUIG, NUIM, UCC and
UCD (to date) have recruited Institutional Research Officers. These officers have
established a network and have been involved directly in the IUQB sectoral project
on Institutional Research, which provided the theme for the 5th Annual IUQB
Conference in NUIG on October 2007 (Institutional Research: benefiting the student
experience and university performance). The interim report for the project was
published at the conference and the editorial group is finalising the first draft of the
National Guidelines of Good Practice for Institutional Research.
Thematic Reviews
Paragraph 32 in the Sectoral Report. Universities should also consider reviewing
university-wide issues, not linked to any one unit, but essential for the ongoing
strategic development of the institution
In its published 2nd cycle of reviews (2008-2012),
http://www.iuqb.ie/DCU%20%202nd%20EUA%20Progress%20Report%20(Institutional%20and%20Sectoral)%2
019%20June%202006.pdf
DCU has agreed to undertake the following university-wide reviews, in addition to
school and faculty reviews:
-
Postgraduate experience (2008)
Community relations/community engagement (2009)
Internal communication (2010)
Physical environment (2011)
Decision-making processes (2012)
In consultation with the Quality Promotion Committee, Academic Council, and the
Executive in DCU and in light of university-wide discussions in relation to student
retention it was decided in 2007 to narrow the scope of this first thematic review to 1st
5
year and beginners students, to include transfer and international students. Postgraduate student issues will be dealt with in a subsequent thematic review, planned
for the year 2009.
Reviews of Cognate Units
Paragraph 131 in the Sectoral Report. Universities should consider reviewing groups
of cognate units (faculties, etc.) to achieve a better overview of how teaching,
learning and research can develop across these units and to break down current
boundaries to inter-disciplinary work.
Since the 2004 review, TCD and the constituent universities of the NUI (NUIG, NUIM,
UCC and UCD) have undergone substantial restructuring whereby large numbers of
small academic departments have coalesced into schools and colleges (DCU and
UL already had structures like this in place). This had necessarily meant that the next
cycle of reviews will involve reviewing a smaller number of cognate units.
Collaboration between Universities
Paragraph 136 in the Sectoral Report. There is a need to maximise collaboration in
research, infrastructure, human resources and strategic development across the Irish
universities in order to develop the greater critical mass necessary to be competitive
in certain areas.
The Strategic Innovation Fund rounds 1 and 2 have seen significant collaboration in
research, infrastructure, human resources and strategic development across not only
all the Irish universities but between the universities and the Institute of Technology
sector. The collaborative SIF 1 projects are detailed here:
http://www.education.ie/robots/view.jsp?pcategory=10861&language=EN&ecategory
=40280&link=link001&doc=33111
Synergies in QA Processes
Paragraph 118 in the Sectoral Report. Units undergoing review should make explicit
links between the formal quality review process and any other QA mechanism which
they may also operate. The potential synergies between these are vital.
From June 2006, the Quality Office in NUIG has worked closely with the Registrars
Office to ensure more effective use of External Examiners’ reports and ensure closer
integrations with the quality review process. All external examiners reports on taught
programmes are now also received by the Quality Office. Details are available in the
NUIG 2006 report on progress with implementation of the EUA IEP review at the
following link:
http://www.iuqb.ie/NUIG%20%202nd%20EUA%20Progress%20Report%20(Institutional%20and%20Sectoral)%2
021%20June%202006.pdf
6
Student Evaluation and Feedback
Paragraph 115 in the Sectoral Report. All units need to ensure that students, both
undergraduate and postgraduate are involved systematically in the self-assessment
and that their opinions and contributions are included in the reports.
Paragraph 119 in the Sectoral Report. The Irish universities need to ensure coherent
and regular student feedback on all courses and modules, and for this feedback to be
an explicit input to the QA process.
The latter is probably the most challenging recommendation arising from the Review
of Quality Assurance in Irish Universities. All universities operate systems of student
evaluation of courses and modules and in particular, there is always a systematic
approach taken to evaluation prior to the quality review of an academic department.
IUQB and the universities have recognised the challenge and are working on a HEAfunded project on Quality Assurance: Student Evaluation and Feedback. As part of
this project, the input of experts from Scotland and Finland indicated that it is difficult
to achieve the goal of universal student involvement in quality assurance without
significant training and funding and it is not reasonable that all of the burden should
fall upon the institutions. If effective, sustained, coherent and regular student
feedback on all courses and modules is to be achieved, it will required significant
energy from all the stakeholders including the funding authorities, quality assurance
agencies, the institutions and the student leadership.
Examples of university responses to the recommendation are given below:
Following the publication of the EUA IEP review reports, DCU decided to create an
annual Student Forum for Quality. The 1st Student Forum for Quality took place in
October 2006, the 2nd in October 2007. The report on the 1st Student Forum is
available at:
http://www.dcu.ie/qpu/pdf/1ST%20STUDENT%20FORUM%20ON%20QUALITY%20
Report%20final.pdf
The second report will be made available shortly, after authorisation by the DCU
Executive.
The Institutional Research function at NUIG, working through the quality office, has
introduced a monitoring process whereby students are requested if they have been
surveyed as to their opinion of the quality of teaching in respect of none, some or all
of their academic programmes. By December 2007, over 2,500 students had
responded to the survey.
7
6. 2nd cycle of Institutional Audits of the Effectiveness of Quality Assurance
Procedures
Many of the recommendations contained in the reports of the Review of Quality
Assurance in Irish Universities in 2004 required changes in quality assurance
procedures. The universities have agreed many of the changes in procedures
required and the updating and publication of the 2nd edition of the Framework for
Quality document has been a useful vehicle to express this accomplishment.
However, the effectiveness of these changes at unit and university level can best be
assessed by further review either at unit level by the universities (during the next
evaluation of an academic or administrative unit) or at university level by a further
external review of the effectiveness of quality assurance by IUQB.
In October 2006, the Board of IUQB, having consulted with the universities (and the
HEA) has agreed that a second round of institutional reviews will commence in 2009.
This review will use as its starting point the outcomes of the first review. The review
will take place 5 years after the first review, well within the 15 year cycle required by
the Universities Act.
The 2nd edition of the Framework for Quality Document ends with the following
statement:
“Finally, the second review of the effectiveness of quality assurance
procedures in the Irish universities planned for 2009-10 will assess
objectively how the system has progressed and how much has been
learned from the first national review.”
8
Appendix A
Incorporation of recommendations into 2nd edition of Framework for Quality
Document
Organisation and planning of QA processes
Recommendation in Sectoral
Report
Paragraph 109. The contribution of
QA to university strategy and
planning should be stressed to all
involved in the process.
Incorporation into Framework Document
“The self-assessment report: presents a succinct but
comprehensive statement of the unit’s strategic aims and
objectives and discusses how these are aligned with those of
the university.” (p47)
“The preparation of the self-assessment report acts as a
stimulus and provides opportunities for reflection and
consultation, enabling units to plan and manage strategically,
and to align their development plans with those of the whole
university. (p48)
Paragraph 110. It should be clear
from the start of the process that the
results of each evaluation will be
discussed between the senior
management, including the
President, and the unit evaluated.
“”Following consideration of the report and its recommendations
by the relevant university committee the unit is asked to
develop a quality improvement plan, usually according to
guidelines developed by the quality office. The unit is required
to agree this plan with the budget holder for the unit (VicePresident/Dean of Faculty/Head of College) This plan is
required as a basis for actions and for the follow-up review that
takes place subsequently. The plan will also include actions to
be taken by the university, where appropriate and agreed. (p63)
Paragraph 111. The university
President’s responsibility for the QA
process and role as one of its main
beneficiaries should be underlined.
Paragraph 112. This overall
responsibility of the President and
senior management for the process
should not lessen the fundamental
ownership of each evaluation by the
unit in question.
“Regular, formal self-assessment is the core component of the
Irish universities’ quality framework, where the emphasis is
placed on the immediate value to the unit of this analytical and
self-critical process.” (p48)
“All staff members of the unit are kept fully informed about the
self-assessment process and are given clearly signalled
opportunities to contribute their views.” (p48)
9
Self-Assessment Phase
Recommendation in Sectoral Report
Paragraph 113. The self-assessment reports
produced by any unit should not exceed 30
pages
Incorporation into Framework Document
“(in the preparation of the self-assessment report) units
are encouraged to adopt a self-critical and reflective
approach, in the spirit of quality improvement. The
preparation of a concise report (typically no longer than
30 pages) is recommended, with additional
appendices.” (p49)
Paragraph 114. The self-assessment phase
should not last longer than three months.
“(Typical time-scale for the quality review of an
academic or administrative unit) …
-4 months (to site visit): unit prepares self-assessment
Report
-1 month (to site visit): self-assessment report sent to
peer review group” (p88)
Paragraph 115. All units need to ensure that
students, both undergraduate and
postgraduate are involved systematically in
the self-assessment and that their opinions
and contributions are included in the reports.
“Examples of self-assessment methodologies … An
example of (the membership of) a coordinating
committee for an academic school of department …
(p49)
Postgraduate student(s)
Undergraduate students(s)”
“The (components of the) self-assessment report for an
academic unit …
Teaching and Learning, and feedback from students …
The report describes and analyses all taught
programmes, teaching and learning evaluations,
feedback from student questionnaires …” (p54)
Paragraph 116. All units need to take proper
account of the evolving institutional and
external environments when undertaking
their self-assessment, looking at the
opportunities and threats these may present
and situating themselves within these
contexts.
“The preparation of the self-assessment report acts as a
stimulus and provides opportunities for reflection and
consultation, enabling units to plan and manage
strategically, and to align their development plans with
those of the whole university.” (p48)
Paragraph 117. Units undergoing review
need systematically to consider their links to
the relevant university services and to make
sure these links are covered by the review
process.
“Support services
Paragraph 118. Units undergoing review
should make explicit links between the formal
quality review process and any other QA
mechanisms which they may also operate.
The synergies between these are vital.
“The (self-assessment) report describes and analyses
all taught programmes, teaching and learning
evaluations, external examiners and
employers/professional bodies.” (p54)
“Parallel consideration of university and college/faculty,
school and programme plans is undertaken to ensure
complementarity and, where possible, synergy between
plans." (p53)
This section (of the self-assessment report) details the
views of the unit on the effectiveness of the student and
staff support services in the university, including the
library and IT support. Units may include reference to
their involvement in collaborations with these services
with the objective of assisting their improvement.” (p55)
“The unit also describes (in the report) the process by
which the curricula of its programmes are developed
and reviewed o a periodic basis” (p54)
10
Recommendation in Sectoral Report
Paragraph 119. The Irish universities need to
ensure coherent and regular student
feedback on all courses and modules, and
for this feedback to be an explicit input to the
QA process.
Incorporation into Framework Document
“All relevant units preparing for review are required to
seek the views of all students availing of their services.
Student views are sought by means of questionnaires,
focus groups and other feedback mechanisms. In many
instances representative students are directly involved
as members of self-evaluation coordinating committees
and thus are continuous participants in the selfevaluation process.” (p42)
11
Peer Review Phase
Recommendation in Sectoral Report
Paragraph 120. Universities should ensure that
the guidelines and terms of reference supplied to
peer review teams encourage a broad view of
quality, including sufficient emphasis on research,
interdisciplinarity and internationalisation. These
guidelines should also ensure that any ensuing
recommendations are clear, realistic and
distinguish between those needing investment and
those where improvement can be made without
significant additional resources.
Incorporation into Framework Document
“The guidelines and terms of reference supplied to
peer review groups encourage a broad view of
quality, including appropriate emphases on
research, interdisciplinarity and
internationalisation. “ (p61)
Paragraph 121. The composition of peer review
teams needs to be more flexible, in order to
respond to the need for strategic benchmarking
with other universities worldwide and to respect
the diversity of profiles and structures among the
Irish universities.
“… allowance for diversity also enables
procedures to work effectively and efficiently by
facilitating adaptations to local circumstances and
the needs of the individual units under review.
There may be variations between units that
suggest a different balance to the membership of
the review group.” (p44)
“In the (peer group) report, any deficiencies should
be categorised as follows:
strategic, involving university policies, or
dependent on the college/faculty
due to limited resources
caused by poor management, policies or
operation and rectifiable with current
resources”
(p62)
“A review group always includes a majority of
external (national and international) members.
As an example, a typical peer review group for an
academic unit might include:
two external experts, one from abroad
and one from another Irish university
a representative of an employer group or
professional body relevant to the unit
under review”
(p60)
Paragraph 122. The choice of peers should be
independent of the unit under review.
“Normally the selection of members of the peer
review group is made by the university committee
responsible for quality assurance. The selection is
made independently of the unit being reviewed by
a means determined by the university committee
responsible for quality assurance.” (p60)
12
Quality Improvement
Recommendation in Sectoral Report
Paragraph 123. Following each review, the unit and
the senior university management should hold a short
seminar to discuss the evaluation, together with
proposals for improvement and action.
Incorporation into Framework Document
“Following consideration of the (peer review
group) report and its recommendations by the
relevant university committee the unit is asked
to develop a quality improvement plan …”
(p63)
“The unit is required to agree this plan with the
budget holder for the unit (Vice President/Dean
of Faculty/Head of College).” (p63)
“This plan will also include actions to be taken
by the university, where appropriate and
agreed.” (p63)
Paragraph 124. A maximum of six months should be
set for agreeing a quality improvement plan.
“+ 1 month (after visit)
Peer review group report received by
quality office
+ 5 months (after visit)
Unit’s quality improvement plan sent
to quality office and considered by
appropriate committee
+ 7 months (after visit)
University/faculty/school/college
management group consider quality
improvement plan and agree a
combined unit/faculty/university level
plan for implementation of the
recommendations in the peer review
group report “
(p88)
Paragraph 125. These quality improvement plans
should be taken into account in the strategic
management and other university-wide processes.
“The quality improvement plan must be a
realistic one with specific achievable actions,
timelines and where feasible, measurable
outcomes.” (p63)
“Implementation (of recommendations) is often
supportive of institutional strategic objectives.
Recurring recommendations have also led
directly to new institution-wide policies and
initiatives.” (p62)
“This plan will also include actions to be taken
by the university, where appropriate and
agreed.” (p63)
Paragraph 126. The university management must
respond to these plans, even in cases where
resources are scarce.
“In order to ensure that action is taken on the
recommendations in the peer review reports,
the quality office monitors the development,
completion and approval by the unit (and the
relevant university officers) of the
consequential quality improvement plan. In
addition, through the local quality committee,
this plan is monitored by means of subsequent
reports and/or internal reviews.” (p67)
“University/faculty/school/college management
group consider quality improvement plan and
agree a combined unit/faculty/university level
plan for implementation of the
recommendations in the peer review group
report “. (p88)
13
Recommendation in Sectoral Report
Paragraph 127. Information on the implementation of
agreed quality improvement plans should be included
in the university’s annual quality assurance reports.
Incorporation into Framework Document
“Each progress report may be considered by
the university committee responsible for quality
assurance (often a committee of governing
authority) which may comment on it,
sometimes requesting further information from
the unit and/or relevant services within the
university.”
(p67)
“In some cases progress reports (as well as
the original review report and quality
improvement plans) may be submitted directly
to the governing authority for consideration and
approval …” (p67)
“Consequential action plans and progress
reports are also published by some of the
universities.” (p63)
Paragraph 128. The university should consider their
entire budgets as quality improvement funds.
14
Strategic governance and management
Recommendation in Sectoral Report
Paragraph 129 The scheduling of evaluation should be
approached in a more strategic way.
Incorporation into Framework Document
Paragraph 130. Universities should explore the
possibilities for linking the quality review cycle to other
strategic cycles.
“The combined emphasis on institutional
planning and the primacy of selfassessment in all reviews promotes a
balanced approach to quality improvement
that incorporates ‘bottom-up’ initiatives that
favour change and enhancement” (p16)
Paragraph 131. Universities should consider reviewing
groups of cognate units (faculties etc) to achieve a better
overview of how teaching, learning and research can
develop across these units, and to break down current
boundaries to inter-disciplinary work.
Paragraph 132. Universities should also consider
reviewing university-wide issues, not linked to any one
unit, but essential for the ongoing strategic development
of the institution.
“Thematic reviews of university-wide issues
(e.g. research, examination and study
programmes) are also carried out.”
(p41)
“For university-wide thematic reviews, the
self-assessment process is headed by a
relevant high-level staff member and the
members of the self-assessment committee
are drawn from all relevant parts of the
institution.” (p48)
Paragraph 133. There is a need to ensure the regular
analysis and overview of the QA process and outcomes
across each university and to link these explicitly to
strategic management processes.
“Ownership of quality processes rests with
each university, and individual management
teams now view them as an essential tool to
enable continual improvement and the
achievement of planning and strategic
change.” (p35)
“In some cases progress reports (as well as
the original review report and quality
improvement plans) may be submitted
directly to the governing authority for
consideration and approval … “(p67)
15
Paragraph 134. All universities need to strengthen their
capacities for institutional analysis and monitoring, in
order to provide better information for strategic
governance and management. Better management
information systems are also needed in most
universities. The QA process must both contribute to and
benefit from these.
“(all universities) have recognised the
importance and usefulness of ‘enterprise
data’ as an important strategic objective for
the university sector. Both IUA and IUQB
have initiated important and co-ordinated
sector-wide projects to support these
initiatives. As apart of these moves,
‘institutional research officers’ have been
appointed to coordinate data and
information management in their own
institutions.” (p30)
“… as the internal and external environment
of quality assurance and quality
improvement developed, the remits of the
individual quality offices have expanded to
include additional function, such as :
Institutional research …”
(p37)
Paragraph 135. Leadership should be aware of the real
dangers of ‘paralysis by analysis’ The burden of
procedures must not obscure the purpose of establishing
a quality culture, and a standardised approach must not
obscure the primary focus on quality improvement.
Procedures and approaches need to be kept simple and
timely.
“The difference (in procedures) which are
evident on examination of the practice in
each institution, are a consequence of
institutional individuality and autonomy, and
of the broad specifications for evaluation
outlines in the Universities Act 1997. In
practice, this procedural variation facilitates
greatly a high degree of alignment with the
common framework.” (p43)
“Rigid, common regulations and processes
would undoubtedly be suboptimal for many
reviews, whereas responsive, locally owned
systems with shared principles offer more
effective outcomes.” (p44)
Paragraph 136. There is a need to maximise
collaboration in research, infrastructure, human
resources and strategic development across the Irish
universities in order to develop the critical mass
necessary to be competitive in certain areas. The Irish
QA framework can contribute greatly to this
collaboration.
Paragraph 137. Given the healthy university-led
approach so far, the lack of any governmental agency
and the relatively modest investments in QA, it is
suggested that the universities estimate how much time
and money have been used so far in setting up and
operating the quality assurance and quality improvement
process, and clarify the benefits obtained. Such an
analysis could then help the universities, individually and
collectively, to link back to the four basic methodological
questions and see to what extent these investments
have been effective and efficient in helping to clarify what
they are trying to do, how they are trying to do it, how
they know it works, and how they change in order to
improve.
16