Quality Reviews
Self Assessment Survey Results
David O’Sullivan
Tuesday 4th May 2010
Contents
Focus Groups Discussions.............................................................. 2
Meetings with Senior Management ................................................. 6
External Reviewers Comments ....................................................... 8
All Staff Survey ................................................................................ 10
Introduction
An institutional self assessment of the quality review process and associated quality
initiatives was undertaken in March and April 2010 under a directive from the Vice
President for Innovation and Performance. His directive was to find ways to improve
the system of review so that it could significantly increase quality, performance and
innovation within the university. A number of assessments were carried out in
addition to numerous ad hoc meetings and discussions. The purpose was to gather
ideas and requirements for the new review process and associated initiatives. This
report outlines the results of four assessments: (i) Focus Group Discussions; (ii)
Structured Meetings with Individual Senior Management; (iii) Requirements of
External Reviewers and (iv) Survey of All Staff. Concise results of all these activities
are presented in this report. These results will be used to inform the ongoing revision
and enhancement of the quality review process and development of associated
quality improvement and innovation initiatives.
1
Focus Groups Discussions
The responses below were gathered from staff and students during five identical
hour long focus group sessions held in April 2010 to ask participants to discuss their
views on the quality review process. Each year since 2001, approximately 10 units
have been reviewed. Units included academic units, administrative units, and
programmes. Each review consists of three phases: self assessment, the review visit
and the follow up. Each focus group was asked to discuss strengths and
recommendations for changes to the review process. An external facilitator facilitated
the focus groups.
Review Strengths
All participants agreed that the current review process provides a forum for many
opportunities including networking, collaboration, benchmarking and improved
efficiencies. In particular they noted the following strengths:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
The current review process is established, comprehensive, robust,
structured, thorough, supportive and is aligned with the guidelines from the
IUQB.
The self assessment phase provides a focus and forum for self reflection,
collaboration, gathering perspectives and the sharing of ideas. The self
assessment phase requires a self review of all the current activities and
capabilities of the unit, including the identification of any challenges and
opportunities facing the unit as well as providing a focus for future
planning/vision.
Self assessment phase produces a single document that maps out and
raises awareness of all the activities of the unit.
Review process is designed to involve everybody and is designed to
provide opportunities for all members of a unit to contribute.
Review process is designed with a view to effecting change and focusing
on improvements.
External peer reviewers provide valuable, open, positive, expert
constructive input with the intention and spirit of enhancing and supporting
the performance of the unit.
Review outcome is to establish a focused action plan with assigned
responsibilities and this is an important aspect of the review process.
Review process promotes transparency.
Review process provides opportunities to form links with the rest of the
university.
Benchmarking element is informative and useful.
Reviewers and reviewees are trained.
2
Recommended Changes
All participants would like the review process to be more efficient, streamlined,
meaningful, sharper, standardised, fit for purpose, less onerous, updated,
transparent, and user friendly. All participants agree that links into university strategy
will ensure more valuable outputs.
The main recommendations are as follows:
1.
The purpose/goal of the review process
The purpose of the review process needs clarification to ensure ‘fit for purpose’,
particularly with regard to its connection, linking and integration with university
strategic plans as well as internal and external stakeholders. This would ensure
actions are more binding and realistic and there would be greater “buy in” and
commitment to the process in general. More involvement, encouragement and
engagement by all staff including senior staff would ensure that quality is embedded
into the focus at all levels of each unit as a matter of course and not just ‘appear’ for
the review process.
2.
The selection, role and responsibility of the reviewers
The selection of reviewers must ensure impartiality, objectivity and independence.
Reviewer guidelines and the review criteria need clarification, including the type of
data/questions/templates used in the review process. This will ensure focus during
the review process on the relevant issues.
3.
The role of the reviewees
Clarity on the role of the reviewees to include guidelines on the review process, how
to prepare for reviews, and an overview of what is expected during all stages of the
review.
4.
Clarity on the volume, type and extent of data required
The current data gathering system is a very burdensome, overwhelming, and
fragmented and can lead to disengagement and frustration. The availability of
centralised, shared, relevant standardised data needs to be more easily accessible.
Ready access to this data will ensure awareness of current activities/data/trends
which is critical for the monitoring of progress and establishing future goals/vision,
benchmarking and planning. The nature and type of data under review should be
more aligned with university strategy and include performance indicators and
meaningful benchmarking. This data would feed into the quality review process as
well as other requirements including HEA, and university strategic documents.
5.
The review visit schedule needs clarity and focus
Clarity is required with regards to understanding what is expected to be achieved
during this review visit, which in turn will determine the number of meetings, identify
clear meeting agenda items and the nomination of who would attend the review
meetings. Removal of the preamble from the review report was suggested and a
short document with bullet point recommendations was recommended.
3
6.
The action plans and Follow up
The action plans must be agreed, realistic and achievable and in particular need to
be effectively followed up, transparent, accountable and linked to university strategy
to ensure quality valuable outputs. Clarification is needed on the implications and
consequences if actions are not carried out. Clarification is also needed on what
course of action can be taken when review recommendations are outside the remit
of the unit. There was discussion about the review outputs to be in the form of
simplified matrices of measures and comparable data, linked to strategic data (that is
easily accessible to the unit), however the point was made that the establishment of
action plans should also allow for an approach that facilitates qualitative unit
creativity and innovation, which may not always be quantifiable.
7.
The focus of the review
 Reviewing Schools: The impact of the restructuring on the review process
was discussed. The review of a school could select one of the flagship
programmmes as part of the review to look at the overall process of how
programmes are managed and structured within the schools. One
suggestion was to have a fluid and flexible assessment template that can
be adaptable to each unit/school including hybrid units that are neither
academic nor administrative.

Reviewing Processes: It was suggested that the review focus could be on
the effectiveness and efficiency of processes as opposed to reviewing
each unit individually. The process flow would review admissions,
registration, programme content and delivery, unit support, and exams
while research processes would include applications for funding etc.

Reviewing Themes: It was suggested that reviews might focus on one or
two main strategic themes/plans within the schools as opposed to trying to
addressing everything. One such example might be to focus on the
thematic approach of civic engagement or internationalisation and the
review and actions would focus on how the school/disciplines work and
engage in civic engagement or internationalisation.

Evidence Based Reviews: It was suggested that quality reviews should be
evidence based on best practices in the areas of strategy, structure,
performance and process. While benchmarking allows for external
international orientation, it is important to clearly identify the key
variables/performance improvement indicators and it is also critical that the
university provide the infrastructure and necessary supports to enable
monitoring of this success such as timely data gathering.
8.
The input from internal service provider’s
The role, representation and participation of these providers in the review process
needs to be clarified.
4
9.
The central gathering of common review findings
Themes across reviews could be collated with a view to addressing common trends
across reviews for example; all programme reviews might feed into establishing
good practice guidelines for programmes.
10.
Other comments/suggestions:

Clarity is needed regarding the confidentiality of the review documentation.
Is it public or private material and can be shared internally or externally?
The protection of intellectual property during the review process was
raised, particularly where you have innovation.

Should leadership and management issues be looked at under the review
process, including how the units are working together under the school
structure and how the units are engaging internally and externally?

A standardised formatting of feedback surveys could be used across the
university including the level and design of student feedback.

Is the review dinner the night before necessary and what is the role of the
reviewees at this dinner?

Why is the review schedule only every 5 years? It was suggested that
trends between reviews are presented at reviews.

How are all the reviews scheduled and how do they all connect?

It was suggested that each school has a quality representative whose brief
would be to manage quality processes. This representative would be the
main link and pulse point between colleges, centralise all quality activities
as well as maintaining links with other colleges and other institutions.

The timelines of the review process could be aligned with other external
reviews such as external examining.
5
Meetings with Senior Management
Below is a list of key points raised by senior University management with the Director
of Quality regarding changes they would like to see made to the Quality Review
process and associated activities. The responses were gathered during eight face to
face interviews in April 2010.

Need to focus on larger units only – schools, institutes, large support services

Need for additional reviews around cross functional processes

Student Feedback needs to be institutionalised

More sue of Blackboard by teachers

Continue student surveys (Institutional Research)

Redesign review process and documentation

More evidence based assessment and reviews

Get benefit of restructuring

Review reports to inform recruitment and resource allocations

Strictly independent reviewers with larger view of world

Quantitative self assessment

Bibliometric data to be used

Benchmarking against world class universities and organisations

Review large research institutes and centres as part of schools

Bring workload model into self assessment

Present evidence of workload and workload balancing

Cross functional and ‘cradle to grave’ process reviews

Annual reporting and service reviews

Incorporate health and safety into reviews

Manual for Processes and procedures

ISO9000 and other criteria for support service quality management systems

Continue to provide outside insights by reviewers

More time for reviewers to reflect on key findings

Critical self assessment by units

Create more continuous proactive culture – quality teams within units

Line management to plan and control quality and performance

Reviewers recommend too much
6

Disconnect resource allocation from reviews – resource allocation part of
normal line management

Keep the student in focus for schools and support services

Schools to use programme boards and accreditation for programme reviews

Continue to highlight strengths but perhaps exceptional strengths

Better balance between evidence and loose comments

More training in management best practice and use of systems

Focus on key 5-10 best indicators

Measure teaching loads – not FTEs but rather teaching credits or hours
taught

Better and more accurate data

Look at equality and gender issues occasionally

Support staff through mentoring

Maintain annual follow-up but get line management involved

Allow units to choose profile reviewers but not individuals

Senior staff as cognates, younger ambitious staff as coordinators

Understand boundary and overlaps between units under review

More strategic reviews rather than operational

Quality projects and heavyweight project managers

Keep focus around our own staff providing ideas and finding ways to improve

Self assessment more important that peer review

Seek feedback from students/staff/other stakeholders

Manage expectations between of need to increase value vs. need for more
resources

Explore service plans vs. change plans

More organisational development
7
External Reviewers Comments
Below is a list of key points raised by reviewers following the completion of reviews
in the 2009/2010 review season.













Time with students was somewhat limited and the reviewers felt that one hour would
have been more appropriate for this meeting, rather than the 30 minutes allocated
The review group would have welcomed more information on graduate career
pathways and on the current status of past graduates.
An opportunity to have met with an employer of graduates would also have been of
benefit to the reviewers.
The 30 minutes allocated for the initial discussion was insufficient to cover the
necessary topics for the review. These topics had already been highlighted through
the self assessment report, but required open and frank discussion with the Director
on a one-on-one basis.
Sensitive issues signalled in advance should have been reflected in the scheduling
arrangements. In general, the fact that the draft timetable for the review was provided
to the team 2 working days prior to the start of the review meant that it was not
possible for the team to make suggestions or requests regarding the content or
organisation of the schedule. The team would like to acknowledge, however, that
when such requests were made on site, the NUI Galway Quality Office was quick
and effective in its response to these.
The only meeting with NUI Galway student representatives was incorporated into a
meeting with a number of other NUI Galway stakeholder groups, including
representatives of senior management. It was therefore difficult for the SU
representative to speak openly about issues which may have needed to be
discussed. As a rule, a review team should always have a separately scheduled
meeting with students.
The highly dense nature of the schedule resulted in a lack of time for the team to
reflect and write. This was particularly the case in terms of preparing the exit report.
Given that most meetings were of 30 minutes duration, even with efficient time
management in place, it was a continuous challenge to ensure in-depth and focused
discussions.
Group sizes varied considerably. Meetings involving more than 10 interviewees
invariably mean that many of those present scarcely have the opportunity to speak,
particularly when the meeting lasts only 30 minutes. Interview groups should be kept
to 10 maximum, as a rule.
In situations where the review panel is expected to, or needs to, meet more groups
than can comfortably be scheduled, the panel could easily split into 2 groups and
conduct parallel meetings, thus covering a broader range of topics and meeting a
broader range of people.
The review team was based in the main office of the unit under review. This was felt
as generally inappropriate, for a number of reasons, although no specific issues in
this regard arose during the review under discussion. As a rule, a neutral space
would have been considered more appropriate.
The room itself was unsuitable for the review in question, given the size of the many
of the groups participating.
No computer or printout facilities were available to the review team. The team
considered it inappropriate to use the facilities of the unit under review, or of other
NUI Galway units beside the meeting room. Luckily, the office of the team rapporteur
was relatively nearby on the NUI Galway campus, so that drafts could be printed and
copies made for team members. This did however result in a loss of time at important
moments of the review.
8















Budgetary benchmarking could be a useful part of the quality review process,
particularly when the unit under review has a variety of income sources (projects, etc)
and has a broad range of responsibilities and activities. It allows the review team to
place the activities of the unit under review in the context of resources available to it
and its use of these resources.
A list of documents tabled during the review should be given to each member of the
review team. In this case, the additional documentation made available to the review
team in the meeting room, consisting of publications, reports, but also detailed survey
material and other project-related documentation, was extensive. A summary list
would have assisted the team in identifying documents of particular interest or
relevance to different parts of the review.
The self assessment report, while in compliance with the QA standard template,
could be better documented to reflect the best practice which exists in the actual
programme.
The self assessment report should contain standard templates to capture
information in a clear and consistent manner.
The schedule of meetings was extremely tight, and additional time might have been
allocated to the session with post-graduate students.
The opportunity to review and reflect on the Institute’s own self-assessment report
well in advance of the visit was essential
The only comment the Panel have to make about the methodology of the visit is that
we would have valued more time with the students.
A minor criticism is that the schedule of meetings was so full that inadequate time
was sometimes available to pursue discussions as fully as the review group might
have liked. This, however, did not prevent the review group from assembling all the
evidence it required to support its conclusions.
The Team would have liked to know (up front) the time commitment involved. Three
days was probably sufficient for such reviews in the future.
The timetable might have benefitted from additional time being given to the Team to
reflect on its findings. Some thought might be given to the level of documentation
(sent out prior to the review and available at the time of the review).
The Self-assessment document (and supporting appendices) was critical to the
process.
There was a good mix of expertise in the review team, subject based and local, on
the team. The absence of an Economist in the team was perhaps the only matter
noted by the team.
Unit presentation to be given in advance of meeting/discussion between review
group and Head of School would be most helpful.
Provision of most recent accreditation report and most recent review report (where
applicable) to be included in documentation sent to review group.
All documentation to be sent to the review group at least three weeks in advance of
Quality Review Visit.
9
All Staff Survey
Below is a list of responses from staff regarding changes they would like to see
made to the Quality Review process. The responses were gathered using an online
survey between March and April 2010.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
The review report asks for too much paperwork, much of which is background reading for the assessors,
and is probably available from the university website. The review process should not impose an
excessive paperwork burden on the unit, which may interfere with its day-to-day work. References to the
website for background information should be encouraged.
The review report should focus on how the office delivers its core mission, and how it satisfies its
customers. There is too great an emphasis on paperwork and record keeping in quality systems in
Ireland, to the exclusion of commonsense focus on performance and delivery. The approach of many
organisations in Ireland, and this is a national criticism, is to outsource quality and buy in the "quality kit"
paperwork, whether it is ISO9001, HACCP, or sign off sheets for chores such as checking the toilets.
Meanwhile, any sane person can see that there are huge variances in real quality, i.e. real customer
satisfaction delivered by real performance despite the paperwork being in the folder. NUI Galway must
be careful not to fall into this trap. The old way of doing things in Ireland was to have people in charge
checking on things and inspecting regularly to ensure that things were done properly.
To save on costs/time input of panel the dinners in the evening should be eliminated
Do away with the Exit meeting, as it is a very unusual atmosphere once the exit report is read out.
Perhaps, the delivery of a one-page report instead to the Head of Unit would be sufficient.
Incorporate more time for the drafting of the report (particularly on day 1)
The Head of Unit not be allowed to be the Chair or a member of the Self-Assessment panel which
should increase true particiaption from members of the unit.
Eliminate the surveys from central service providers such as library and buildings office etc, as they are
not very useful.
Self assessment process takes considerable time out of already busy schedules.
The size and boundary of the unit under review dictates whether the review is primarily operational or
strategic (due to need to engage ALL staff of the unit).
Specialist expertise of high calibre external assessors (chosen for their strategic credentials) may be
wasted on time consuming (3 days) review of mainly operational issues.
Action plans mainly reflect already planned projects and/or those already in train.
Benchmarking against "best in class" is not possible as so much time is spent on other aspects of the
process. Wed, Apr 14, 2010 10:28 AM Find...
Streamline reporting - narrative reports inefficient/unnecessary; who will read them? Should be a short
preamble with bulleted recommendations, listed in order of priority
Retain the social aspects of the review, dinner etc., important for external relations with outside
assessors
Condense the meetings into shorter timeframe; it is an excellent feature of the process that key
personnel across the University are involved but this needs to be balanced with what's efficient in terms
of participants' time; reducing the duration of the meetings could be a way of retaining involvement but
making it more efficient overall
More time for the External Review Group to visit the unit (two days very short, it is possible however,
that a further commitment from external participants is impractical).
More help to be provided in carrying out a Benchmarking Exercise, or that the Quality Office takes on
the burden of carrying out such an exercise for the Unit under review. Thu, Apr 8, 2010 3:18 PM Find...
It needs better PR! It is perceived as cumbersome and boring.
The review should be cross-functional/cross boundary rather than on a departmental basis as seems to
currently be the case
More visibility as to progress, actions....
Allow input from everyone at all levels of the organisation..and visibility to that input (but allow the input
to be anonymous also)
Encourage, reward and foster innovation in the system
Degree of self assessment is left up to the individual department, greater monitoring of this process by
the quality office to ensure a real self-assessment is carried out would be beneficial.
It might be more useful if stakeholders could input during the self-assessment phase rather than during
the Quality Review days - the unit never hear the detail of issues raised unless the reviewers choose to
provide that detail in their report. There is scope for frustration if stakeholder feels they weren't listened
to, but Department never actually heard the comment/issue.
Having been on a Review Group, an awful lot is packed into the three days - referring to point 2 above, if
some of the input could be gathered and documented in advance it might make the days more
manageable, as well as documenting feedback/input for the benefit of the department under review.
10
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
A greater level of follow up post review would be beneficial, particularly in relation to progress on the
Action Plan. There is a risk of a huge amount of effort going into the review and then nothing happening
post-review.
More reference during a Quality Review to previous reviews and actions taken arising from them.
"end-user" representation to provide input to the process although open to all is sparsely
taken-up. Suggest that consideration be given to assuring greater / more useful and
representative end-user input is secured.
recommendations could be restricted to say 10 priority actions, with assessed unit agreeing
to implement and subject to half-yearly/annual tracking. Tue, Mar 30, 2010 12:33 PM Find...
1. Longer interview sessions - we had only 20 minutes with 4 people in the room - very
difficult for everyone to get across their viewpoint
In these interview sessions - review should concentrate on a persons job description as
opposed to what people say they work at or what they think they should be working at.
Reduce the response time from once the quality review is completed to when report and
feedback is provided to the group. Sometimes momentum is lost and people have forgotten
what their issues were!
At the end of all of the individual interviews it might be a good idea to have a brain storming
session with an independent facilitator to outline what way the group would ideally like
their organisation to work? Would force groups to think of how to do things better. Work as
a team and have an almost solution before the review.
Before the quality review each person was asked to do a self review - I would include these
as a basis for the actual interviews - good information was captured here and yet was not
referred to in the interview/ review sessions. Tue, Mar 30, 2010 8:10 AM Find...
3. 1) In my opinion, the current system of reviewing interdisciplinary degrees independently,
but not "departmental" ones is not very good. The former deal exclusively with academic
issues (which is good, in a way) but in the latter the academic issues are often appear less
important than research.
2) The distinction between quality reviews, strategic plans, operational plans, etc, etc, is
blurred. Moreover, we are all getting rather tired of doing seemingly the same exercises
over and over again. I recommend that you work out the purpose of the quality review
system, particular with regard to planning, and make it part of the other exercises. Mon,
Mar 29, 2010 3:35 PM Find...
4. Seems now to be a 'One Off' or very occasional event rather than an ongoing process.
This should be changed.
A Master Plan should be made available for the entire University so that each unit would
know if/when they are to be reviewed well in advance. Such information now only seems to
be available to the 'high-up' people.
Seems to be inordinately complex, time consuming and stressful for both reviewers and
reviewed. Everyone is pressured, rushed, harassed and constrained. The process and
practice should be slowed down and simplified.
The reviews seem to obsess about process and paperwork rather than about people. I
believe that the review is therefore open to manipulation. I also believe that there is a deep
suspicion and unease among staff that a Quality Review seeks to criticise rather than to
compliment and to castigate rather than to create. Greater care should be taken to elicit
useful, relevant and real information from people working in the particular unit, from staff
working in the University and from students past and present. However the virtue of
absolute confidentiality in such a process must be emphasised and protected and the fears
of participants must be better allayed.
Sometimes the Units being reviewed seem too big - Student Services rather than say, for
instance, Counselling or Chaplaincy. A smaller, neater, more flexible, less expensive Quality
Review Process should be introduced for such small units, departments and services. Mon,
Mar 29, 2010 3:25 PM Find...
11
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
5. reviewers should, to the extent feasible, meet faculty individually rather than leaving it up
to the faculty to contact them via the organiser
when issues are raised regarding Centres by the wider school, the Centre should have time
to feedback (in a constructive way) to comments that may arise. It may be that the
observations are accurate or it may be that it is perception more than fact Mon, Mar 29,
2010 2:55 PM Find...
I think there needs to be more involvement at departmental level within all departments
More regular feedback to departments by the departmental representative Mon, Mar 29,
2010 2:25 PM Find...
Can sometimes be controlled, or be seen to be controlled, by people who have vested
interests in not making themselves look bad; Isn't always as transparent a process as it
should be, e.g. it can be faked; Doesn't work at all unless you have buy-in and commitment;
Can sometimes be seen as a pencil pushing exercise just so that the Quality Office is kept
happy; Can be very disappointing if the quality review process identifies a need for change
but that change doesn't subsequently happen or the reviewers recommendations are not
acted upon - this means that the next time you're up for quality review, the previous bad
experience means people are disenchanted Mon, Mar 29, 2010 11:56 AM Find...
1. "Quality costs money" - resources should be provided to the unit being reviewed to
enable staff to fully participate.
2. External reviewer's recommendations should be costed, in order that they can be properly
assessed for implementation.
3. Implementation of improvements following review should be properly recognised and
celebrated Mon, Mar 29, 2010 11:23 AM Find...
I don’t know why the system is important (I'm sure it is).
I don’t know why it is carried out.
I don’t know who it is for, or who compiles it.
Does it make any difference or is it carried out "for the sake of it"
Lot of work involved for small disciplines
Not much of what was recommended was acted upon Mon, Mar 29, 2010 10:44 AM Find..
Ability to meet members of staff, key researchers individually to tease out issues in the Unit
under review
For reviews of academic disciplines/units, more focus on streamlining the administrative
processes supporting the unit
More structured time for the review panel to reflect on the information
Emphasise to the Chair of the Review panel the importance of keeping to time Mon, Mar 29,
2010 10:43 AM Find...
1. Reduce the amount of paperwork involved in the system
2. Take the best ideas that have worked with other cognate departments and make them
available to other departments (so that we can all learn from each other and not keep
reinventing the wheel)
3. Give more time on the day for discussion, to find better ways to do things
4. Perhaps have more frequent (e.g. annual) reviews that focus on different aspects of a
department (e.g. research, teaching, support systems, etc). these could involve just local
reviewers. Mon, Mar 29, 2010 10:23 AM Find...
1. Vastly improve uniformity in quality of self-assessment reports and materials provided by
units to review group
2. Enforce accountability on units which ignore principal review findings
3. Find a more effective way of gathering inputs from linked academic stakeholders. There is
a good input mechanism for students and direct unit members, but not from indirectly
related units. The 'town hall' meeting format is not ideal, and a nominally anonymous survey
such as this one could be an alternative/supplementary route ...
12
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.
78.
79.
80.
81.
82.
83.
84.
85.
86.
87.
88.
89.
90.
91.
4. Create ways for units to understand recommendations of review groups - for example,
review groups could recommend study visits to best-practice comparators.
5. Make the review a point of "re-energising" in the lifecycle of all units. Management should
encourage and reward units which engage positively and genuinely attempt to build from
their review. Mon, Mar 29, 2010 9:42 AM Find...
Provide additional admin support to each unit/programme to prepare the self-assessment
report, especially as now benchmarking is meant to be included which takes considerable
time and effort.
Less paperwork and less time!
Timely follow-up to see that designated persons have fulfilled their commitments in the
report - at all levels of the University.
Increased engagement by University management in fulfilling their commitments in the
report - not just showing up for the meetings on the day of the review.
Panels to focus on quality and content of research and teaching, rather than on admin and
management. Mon, Mar 29, 2010 8:33 AM Find...
Ensure recommendations are realistic and deliverable. It is very important that having
carried out such a major exercise that realistic expectation are maintained.
Ensure the cost associated with recommendations is clear and the source of that funding.
A quality review is a major undertaking by any unit. It should be part of the normal
management process. I would like to see how it might be more integrated into the
management process, rather than a single item every 5 years.
As a follow on from the above, it might worth investigating the optimum size of unit to be
reviewed e.g. should it be the Buildings Office or should be the Planning & Development
Unit within the Office. Mon, Mar 29, 2010 8:12 AM Find...
--The Research Office review had an open forum where concerned stakeholders could
comment. However the meeting was extremely poorly advertised. In general the research
community didn't even know about the open forum. As a result, only 5 people attended! I
only found out from a colleague who happened to forward the email. Many of my research
colleagues were irritated to learn after the fact that the meeting had already occurred. For
quality review to really be effective, these meetings need to be adequately advertised to all
stakeholders, so that the review team can gather a representative sampling of comments.
Mon, Mar 29, 2010 8:03 AM Find...
The Self Assessment Report Template could include a section on CELT and Student Services.
Therefore encouraging units to comment on how they have engaged with the following in
order to enhance the experience of their students and staff:
--CELT: audio visuals, Community Knowledge Initiative, ALIVE, Blackboard, etc.
--Student Services: supporting the academic societies, student involvement, student support
i.e. disability, mature, counselling, etc. Sun, Mar 28, 2010 7:04 PM Find...
Preparation of the documentation can be very time-consuming, and may be over-elaborate.
Delay or failure to implement recommendations with up-front cost implications.
Tact can sometimes displace necessary candour.
Sigh of relief and return to inertia for the next 5 years when the process concludes!
A lighter, more continuous form of internal review may be appropriate Sun, Mar 28, 2010
1:13 PM Find...
1. More support in running the quality review days - used to be 100% run by quality office
2. Greater synchronisation of quality reviews with university, college and school strategies
and KPIs
3. Deliberate effort by university to derive general lessons from quality reviews rather than
regarding each review as separate and distinct
4. Each review should be followed by 6 month and 1 year reviews to ensure
recommendations have been met
13
92.
93.
94.
95.
96.
97.
98.
99.
100.
101.
102.
103.
104.
105.
106.
107.
108.
109.
110.
111.
112.
113.
114.
115.
116.
5. University to look at reviews as opportunities to invest in centres, schools, courses etc
rather than taking a defensive stance as it always does hoping that it does not have to spend
more money Sat, Mar 27, 2010 4:13 PM Find...
1. Shorter and simpler reviews. The whole process is overly burdensome and takes too long
2. Lack of unit leadership support. There has been a tendency to rely on a few individual
‘quality champions’ to get reviews done.
3. Simpler and shorter matrix measures would be more valuable (e.g. quality measures of
research output or student feedback evaluations, rather than bland descriptive and quantity
emphasis).
4. Expectations of any real or meaningful change as a result are very low. Perceptions it's a
pseudo-exercise which reinforces university’s system of centralised control over units and
schools.
5. Very little feedback to staff post-review visits and excessive delays in disseminating
reports back to staff involved. This needs to be quicker and of higher profile / priority. Once
done much if forgotten. Sat, Mar 27, 2010 2:01 PM Find...
1. Prepare and promote the organisational TQM strategy and disseminate
2. Provide basic level tutoring on TQM / Six sigma / lean / relevant quality processes to the
candidate group, so they are aware of the best practice and can strategically align the
function/unit approach to the University approach
3. Provide more time for comments from customers in other departments (e.g., the Bursar's
office review allowed only short interview slots for customers)
4. Allow further scope for functions/units to propose (non-resource based) TQM-based
solutions
5. QR recommendations frequently require top level support/interaction; follow up must be
quicker Fri, Mar 26, 2010 6:33 PM Find...
Follow up for area
Follow up where invited to attend reviews for other areas of points raised and
how they are dealt with
Reviewers not receive report in time prior to first meeting - time given to review far too
short (how many don’t overrun on day one?)
Needs to be done every 5 years
Where all aims can not be achieved needs official report added and amended
and communicated to all NUIG not simply HOS etc Fri, Mar 26, 2010 4:31 PM Find...
1) Quality has to be internally driven not imposed by people who know nothing of the
culture and mission of the college
2) We need to move away from tokenism and make quality a core cultural value of the
college
3) Quality is not something that should be reviewed every 5 years but part of our systems
4) Once improved systems are identified they should become the minimum standard
5) People need to see a connection between the review process and improvements in a
timely manner. The last review took up too much time Fri, Mar 26, 2010 4:06 PM Find...
1. Where individual taught programmes are subject to external review/accreditation I think
the results of these should be submitted in the place of specific QR requirements. There
should be no need to review internally as well as externally.
2. Some of the data required regarding programmes (exit rates, retention, outcomes) could
be collated centrally. Asking each discipline or School to do so is duplication. Monitoring of
certain outcomes (eg numbers registering on courses, numbers withdrawing, passing etc)
could be managed centrally and reflection of specific issues taken at the level of the unit.
These stats could be made available yearly for disciplines to access and discuss if they wish
as part of ongoing monitoring, which would feed into QRs.
14
117.
118.
119.
120.
121.
122.
123.
124.
125.
126.
127.
128.
129.
130.
131.
132.
133.
3. Again, with regard to disciplines, the QR office could provide a service whereby they
provide a workshop/facilitated session once per year engaging teaching teams to explore
their outcomes (hard and soft, e.g. eval. data) for the past year and identify changes,
improvements for forthcoming year. Participation in this session would form part of the
evidence assembled for QR.
4. While the QR prompts important reflection on quality of service and product, often the
ways in which quality could be improved are beyond the control of the individual unit or
discipline.
5. Improvement in quality is not always resource dependent - but much of the time it is. If
resources are not available then it an empty and inefficient exercise to review quality. Some
budget should be put aside for implementation of quality improvements that arise out of a
QR. This would incentivise people to participate. Fri, Mar 26, 2010 4:01 PM Find...
Very time consuming it must be said
Often frustrating because so much time is put into it and the review itself is over in a hurry:
in particular, there is little opportunity to comment individually to the review panel.
The provision of making an individual appointment with the review committee, while
positive in itself, is problematic in practice because it means that if you have a point to raise
which would be uncomfortable to bring up in an open session you declare to everyone that
you have such an issue by arranging a slot with the review committee.
In a recent quality review I felt the process turned into a strategic review and that an
opportunity was missed to evaluate working practices
The ways in which things are reported in the written documentation by the unit under
review often obscures the truth. There is an investment at times in making it a PR exercise
rather than an chance to self-evaluate honestly or to represent the reality of the situation
Fri, Mar 26, 2010 3:54 PM Find...
1. The external review group should be given a written brief, which should be published as a
preamble to the written report. This would make it clear whether the emphasis on particular
issues arose out of (i) requests from the School and/or University (ii) independent initiative
by the review group.
2. The burden on School staff (especially, but not exclusively, those on the self-assessment
group) in preparing the self-assessment report is enormous. The costs of this burden (e.g., in
reduced time available for teaching preparation) should be measured, and compared with
the benefit from what are often rather lightweight external review group reports. It seems
to me that the exercise (in its present form) is not worthwhile in cost-benefit terms and is
therefore detrimental to quality.
3. Duplication should be reduced. The assessment should be either at program level or at
school level, but not both.
4. The Quality Office should be provided by the University with the resources to enable it
(and not the School staff) to lead self-assessment. School staff should be required to
contribute by providing whatever data is required by the Quality Office but should not be
expected to take ownership of the process. If the University is unwilling or unable to provide
the necessary resources to the Quality Office for this purpose then the University should
admit that it is unwilling or unable to conduct proper quality assessment. Fri, Mar 26, 2010
3:54 PM Find...
A sense of "box-ticking" in the whole approach - move away from the current somewhat
mechanical procedure
Quality assessment should be continuous, not an occasional event every few years
Ensure better follow-up of recommendations
Put more emphasis on metrics and external benchmarking
Review the validity of putting so much so much on self-assessment which can make for a
very inward-looking approach Fri, Mar 26, 2010 3:39 PM Find...
15
134.
135.
136.
137.
138.
139.
140.
141.
142.
143.
144.
145.
146.
147.
148.
149.
150.
151.
152.
153.
154.
155.
156.
157.
158.
159.
Change to continuous improvement system.
Streamline process to make it less time consuming & more efficient.
Reduce costs to university.
Make all reports available on intranet. Fri, Mar 26, 2010 3:27 PM Find...
1. Carefully consider the level at which the Quality Review is needed. Is it appropriate for
Colleges, Schools, programmes?
2. Carefully consider the level of detail that is required. The current system is entirely too
labour intensive, in part because the enormous amount of data that was requested.
3. Determine a mechanism to acknowledge the contribution of the non-leaders of the selfassessment group.
4. Determine the link between the quality review process and the strategy process. Both
processes are laborious and there is considerable overlap between them. Fri, Mar 26, 2010
3:21 PM Find...
Make it slimline and fast.
Remember €200 an hour is what an academic's time should be costed at, so don't get us
wasting inordinate amounts of time writing waffle. Fri, Mar 26, 2010 3:16 PM Find...
- More emphasis on admin units than teaching units
a.
No need for units that already have accreditation system in place Fri, Mar 26, 2010
3:12 PM Find...
1 Make it more evidence based, by for example use more data on teaching and research
performance.
2 Link more closely with strategic plans, goals and targets. Measure performance against
targets.
3.Make the process lighter. The self evaluation is too long and 'wordy'. It needs to be shorter
and evidence based rather than apparitional. Template based?
4 Review larger academic units, Schools rather than disciplines.
5 Encourage review teams to be more specific in their comments. Develop publishable final
reports which include self assessment data and outcomes and could be used to inform
potential students, parents and potential research partners of the status of the unit
reviewed. Fri, Mar 26, 2010 3:10 PM Find...
More involvement from all types of esp. internal stakeholders ... more of a bottom-up
approach.
Can staff input suggestions for areas to be reviewed ...
Could Quality Report writing skills be developed via workshop. Fri, Mar 26, 2010 3:08 PM
Find...
1. review course content, not just the availability of topics but their scope, proper delivery
and effectiveness in achieving international standards in learning.
2. Make sure Bologna structures are properly and not nominally adhered to. Fri, Mar 26,
2010 3:02 PM Find...
1. Since its a box ticking exercise for the UMT then it should be replaced with something that
has value for the owners of the programmes or units.
2. Since we also do accreditation of programmes then its a waste of time doing this for
programmes also.
3. If management were serious about the programme they would support its
recommendations wrt staffing and resources. - I have not seen this happen.
4. The self assessment report is too big and one can make it up without any real challenge as
its a case of we won't hurt each other as your turn will come around and if you hurt a
dept/programme then yours will "get it in the ear" next time.
5. If the assessment were truly independent then it would be run along the lines of ISO
certification where accredited(By INAB) 3rd parties would carry out the review. Fri, Mar 26,
2010 2:58 PM Find...
16
160.
161.
162.
163.
164.
165.
less "templating";
less waste of paper;
fewer headings;
real action to follow from all relevant stakeholders;
electronic documentation throughout Fri, Mar 26, 2010 2:56 PM Find...
Need to tie review process to concrete deliverables in terms of quality, such as outputs
(students, publications) and rewards (hiring of personnel, additional income, equipment
funding) to achieve improved engagement with the process (by both the university and the
units).
166. What is the point doing a review if there are no tangible repercussions?
167. Need to provide institutional support for the preparation of a standard benchmark across
units (i.e. research metrics) and a threshold/comparison in international rankings for
example (i.e. select units of comparable size/quality from international counterparts). Fri,
Mar 26, 2010 2:39 PM Find...
168. 1. Quality delivered within the existing resources needs to be a focus rather than 'we would
do better if the lectureship was filled' etc.
169. 2. More /better communication from the outset with the line manager of the head of unit
being reviewed
170. 3. Review to be set in the context of the units' strategic /action plans, and that of their home
entity [College, School, Directorate etc], including benchmarking, the KPI's linked to those
plans, workload models etc.
171. 4. review to address alignment of academic and support services where appropriate
172. 5. Process to be 'developmentally' focussed by reference to models of best practice,
excellence etc.
173. 6. Transparent mechanism for 'triggering' a review. Fri, Mar 26, 2010 2:39 PM Find...
174. ALL STAFF SHOULD BE GIVEN THE OPPORTUNITY TO CONTRIBUTE FREELY TO THE PROCESS.
MY EXPERIENCE HAS BEEN THAT MANY SEE THE QUALITY REVEIW PROCESS (LIKE HEALTH
AND SAFETY RISK ASSESSMENTS) AS PAIN IN THE NECK, AND OF REAL BENEFIT.
175. ORGANISE PRESENTATIONS TO ALL STAFF ON THE NEED FOR QUALITY. PRESENTLY MOST
STAFF, EVEN SOME O THE MOST SENIOR STAFF HERE, COULD NOT IDENTIFY OR DEFINE
WHAT QUALITY IN THE 3RD LEVEL SCTOR ACTUALLY MEANS, EITHER TO THEM PERSONALLY
OF TO THE UNIVERSITY. TO THEM, I SUSPECT, IT'S JUST LIP SERVICE TO THE 97 UNIVERSITIES
ACT.
176. DETERMINE WHERE THE SERVICE QUALITY GAPS EXIST IN NUIG. REIN-IN ACADEMIC STAFF
WHO SAY THAT QUALITY ASSESSMENT (LIKE RISK ASSESSMENT) IS NOTHING TO DO WITH
THEM. THYEY BLEAT ON ABOUT NOT BEING QUALIFIED TO ENGAGE IN THE PROCESS.
177. SENIOR NUIG STAFF RECENTLY HAVE ROLLED OUT THE NOTION OF 'THE CUSTOMER'.
ASSUMING THEY HAVE IDENTIFIED THE CUSTOMER(S) (FOR THERE CAN BE MORE THAN I
TYPE) THEY SHOULD ADOPT THE KANO MODEL FOR ASSESSING CUSTOMER SATISFACTION.
178. THERE SHOULD BE SANCTIONS IMPOSED ON UNITS THAT DO NOT MEET THEIR TARGETS,
POST-REVIEW. Fri, Mar 26, 2010 2:38 PM Find...
179. Involve more time for students
180. Involve more time for individual staff reports...either in interview of written
181. Allow for individual units to be more involved in criteria setting Fri, Mar 26, 2010 2:34 PM
Find...
182. - Reviews should be proactive rather than the existing reactive or "gotcha" system!
183. That the review be completely external to the Irish University system.
184. Quality Office and CELT should be amalgamated.
185. Areas should be re-reviewed after three years.
186. Quality success is not all about league table positions, other considerations should be built
into quality audit system. Fri, Mar 26, 2010 2:29 PM Find...
17
187.
188.
189.
190.
191.
192.
193.
194.
195.
196.
197.
198.
First and absolutely foremost, the university should act on recommendations made.
Following a very positive review we were given a long list of things to change, and on the
university's side it said 'we will continue to strive to increase funding in third level'.
Complete and utter dereliction of duty. We were recommended an extra academic staffing
post. Nothing. We were recommended replacement of technicians through succession
planning. Nothing.
It is a long laborious tedious administrative exercise that pulls us away from our teaching
and research duties. Many of these duties seemed solely to justify the existence of the
quality review office, and made a potentially efficient exercise a long drawn out pen pushing
exercise.
The panel are expected to talk to so many different groups that the time available to each
was worthless. They also ran way over time, meaning more time was wasted simply waiting
to be called to the review.
But overall, unless the university acts they are a total waste of time. Fri, Mar 26, 2010 2:13
PM Find...
It needs to be made less time-consuming by an order of magnitude, for both the reviewers
and reviewees.
If it is to mean anything it needs to assess the outcomes of what we do, not the process.
It needs to ask clear questions that can be answered concisely, not with 100-page formulaic
document. Fri, Mar 26, 2010 2:10 PM Find...
The closing report needs to highlight ownership of soluble deficiencies, with a precise
timeline in place for address, and review of address.
The process should, at its core, strive to explicitly align recommendations with the
University's strategy.
Access to a central administrative resource for the duration of the process, particularly in
the initial stages, to assist with data collation.
A scheduled revisit(s) by the Director of Quality, to oversee implementation and/or
achievement of agreed changes specified upon closure of the quality review.
A reward mechanism for the School/College associated with the unit under review, when
agreed deliverables and targets are made.
18