National University of Ireland, Galway An Coiste Feabhais Acadúil The Committee on Academic Quality Improvement The Academic Quality Assurance Programme 2008 ­ 2009 REVIEW OF Digital Enterprise Research Institute (DERI) NATIONAL UNIVERSITY OF IRELAND, GALWAY Final REPORT (5th March 2009)
This report arises from a visit by a review team to the Digital Enterprise Research Institute (DERI) from 3 rd ­5 th February, 2009. The staff had already prepared and submitted a 'Self Assessment Report' that, with other documentation, was made available to the review team well in advance of the visit. The review team consisted of:
· Professor Grigorios Antoniou, Institute of Computer Science, Foundation for Research and Technology Hellas (FORTH), PO Box 1385, Heraklion, Crete, GR­ 71110, Greece (Chair)
· Professor Orla Feely, School of Electrical, Electronic and Mechanical Engineering, University College Dublin (UCD)
· Dr Brian Hughes, Director of the Centre for Research on Occupational and Life Stress (CROLS), NUI Galway acting as Cognate
· Dr Martina Prendergast, Development Manager of the Environmental Change Institute, NUI Galway acting as Rapporteur.
2 1. Aims and Objectives The mission of the Digital Enterprise Research Institute (DERI) is “to exploit semantics for people, organisations and systems to collaborate and interoperate on a global scale” Its vision is to be “recognised as the leading international web science research institute interlinking technologies, information and people to advance business and benefit society.” DERI was established in mid­2003, and its research has focused on two strands: The Semantic Web and Semantic Web Services. The Institute decided from its inception to focus on these very specific areas of research and have established the largest research Institute in the world in this area. The challenge of establishing a research centre of excellence was especially onerous, as a critical mass of researchers with expertise in semantic web technologies did not exist at NUI Galway. Since its establishment, DERI has experienced a very rapid rate of growth and currently is home to 56 PhD students and 16 Postdoctoral Research Fellows. The Institute has grown to over 120 members over the past five years and has been highly successful in recruiting and producing high­calibre researchers. A testament to the international success of DERI is the fact that the research community is drawn from 27 nations. The Institute has been very successful in drawing down significant amounts of funding from national and European sources, and DERI is currently hosting 18 EU projects and six Enterprise Ireland (EI) projects. This success has been recognised by Science Foundation Ireland (SFI) who in 2008 awarded continuation of CSET core funding to DERI. DERI has clearly formulated for itself a set of eight strategic objectives and priorities and has considered how these may best be targeted. The strategy has set out an ambitious direction for DERI. The management of DERI should reflect on and consider ‘personalising’ some of the strategic objectives as some of the objectives tend to be quite generic and could relate to any Institute. The objectives reflect the NUI Galway Research Strategy and contribute to the ‘Informatics, Physical and Computational Sciences’ Thematic Priority Area of the University. A very detailed set of actions has been identified for each strategic priority and DERI has already made significant progress in achieving these objectives, and has identified the need for collaborative, inter­institutional and interdisciplinary research in order to successfully implement the strategic priorities. In order to understand how a modern research Institute in the ICT sector can stay competitive and thrive in the more challenging conditions concerning today’s research environments, a benchmarking exercise has been carried out. The Inno Group compared DERI’s activities with three institutes in Europe: DFKI in Germany, SICS in Sweden and KMi in the UK under the headings Organisation, Governance, Financing and Strategic Highlights. Based on the summary of the exercise reported on page 10 of the Self­Assessment Report, the Institute compares well with the three international Institutes established in similar research areas.
3 DERI has drafted a ‘Corporate Strategy 2008­2013’ which outlines the strategic priorities for organisational growth. The Institute has identified leadership in research, research collaboration, industrial partnerships, university partnership, best staff and students, sustainable funding, enhanced profile and expanded outreach as the most important factors to be addressed in order to implement their vision. The document is the main strategic plan for the Institute, and outlines clearly the mission, vision, strategic objectives, key challenges and opportunities, and actions for delivering the vision. The authors conducted an extensive consultation exercise to obtain views of relevant stakeholders and to test their response to DERI’s strategic intent. The document is very well written and presented and is timely given the University’s current strategic planning exercise. The document highlights the long­term sustainability of the Institute as a key goal, and outlines plans for increased engagement with industry as playing a vital role in achieving this goal. University Management felt that DERI was well supported and has purchased the building in which DERI currently resides. A complete refurbishment of the building was undertaken in consultation with DERI Management at a cost to the University of €2 million. University Management has made clear a willingness to engage in dialogue with DERI about its current needs and future development. Due to its success in securing a large number of national and international grants, DERI is largely a self­ sustaining entity in terms of its personnel and administrative functions. The panel was of the opinion that DERI, in consultation with the wider University community, should explore specific strategies that would enable the Institute to be supported and embedded permanently within macro­level University structures. That is to say, it should be a long­term objective of DERI to establish a standing academic and administrative infrastructure under the auspices of the University’s central finances, with grants pursued at the Institution level primarily in direct support of research activities. The Review Group also noted that one consequence of the current geographic location of DERI was that its members felt somewhat isolated from the main NUI Galway campus and, as a result, from University affairs and identity generally. Occasionally, this manifested itself in the form of expressions of pride in being attached to DERI but of ambivalence about the attachment to NUI Galway. This sense of separateness appeared at times also to be present in the attitudes of University personnel who work outside DERI. So long as DERI is located at a physical distance from the main campus, both the Institute and the University at large should explore ways of reducing the psychological distance as much as possible (e.g., by encouraging all parties to place greater emphasis on the positive relationship between DERI and the wider University in internal and external communications). This is in the interests of morale, staff/student welfare, and institutional cohesion. 2. Organization and Management
4 a. Governance Structure The management of DERI is separated into core research activities and supporting operations. In addition, we commend the establishment of an Advisory Board and an Oversight Board that are able to provide valuable support and advice in operational and strategic issues. At the core of the management structure is the DERI Senior Executive Committee where the leaders of the research (Director and Deputy Director) and operations side (CEO, Operations Manager) work together closely. Our impression formed both during the review and when examining the achievements of the Institute in fulfilling its objectives indicates that DERI is managed very efficiently and effectively. This is also due to the deep commitment of all senior staff to the long­term success of the Institute. However, the management structure has also some weaknesses, which may show negative effects if strong differences between the research and operations sides should arise. The current structure suggests that such differences would be resolved outside DERI, i.e. by the Oversight Board. This aspect seems problematic as ultimately the Director is responsible and accountable for DERI’s success. Recommendation 1: The management structure should be revisited and modified to include a clear conflict resolution policy within DERI. b. Permanent Academic Staff DERI is in the rare situation that it was established with little reference to pre­ existing local expertise. As a consequence, its personnel are generally not rooted within the permanent structures of NUI Galway as a whole. At present, only the Director of DERI is employed as a tenured Professor by the University, although it is acknowledged that a second Professorship in the area of the Deputy Director has been advertised. In addition, a Stokes Lectureship which will eventually translate into a permanent academic position, has been funded. We commend these recent actions but believe that more needs to be done in this direction, as with current arrangements the long­term existence of DERI depends too heavily on attracting external funding. One possible direction to be explored is stronger engagement of other academic staff of the University in DERI operations, while keeping the distinct research profile of DERI. Recommendation 2: A strategy for increasing engagement with academic staff (from within and perhaps beyond NUI Galway) should be developed. This strategy might incorporate incentives (such as teaching relief or credit toward promotion) for participating in major DERI operations, subject to relevant University career­development policies. c. Fixed­term Contract Research Staff i. The Review Panel were particularly struck by the commitment of staff to DERI. Equally impressive was the close rapport that exists between students and staff, a feature identified by students as a fundamentally positive part of their experience in DERI. Resources and facilities are state­
5 of­the­art, and there is no doubt that this contributes to a positive working environment and good atmosphere among those working in DERI. ii. Management is very supportive of the career progression of staff, e.g., most training requirements are met by the Institute. Continuing professional development is encouraged and resourced. iii. Strategies should be developed to facilitate the integration of administrative functions and procedures between DERI and the central support services provided by the university, e.g., Human Resources, Finance, Computer Services. There appears to be some duplication across administrative functions organised in­house in DERI and those offered by central University. In some cases, the in­house functions have been established in order to meet needs specific to DERI and it is clear that DERI’s own activities represent best­practice in these cases. Nonetheless, given the clear efficiencies and economies­of­scale that would pertain, the University should seek in the long term to have maximum integration of such services across all its Institutes and other units. iv. Information flow between management and research staff to PhD students could be improved, especially with regard to informing students about the activities and ambitions of the Institute as a whole. d. PhD Students i. Students are very pleased with standard of the facilities and equipment. ii. There is an excellent collegiate environment between PhD students at different stages of training. Strong mentorship was evident among PhD students and between research staff and PhD students. The fact that PhD students often shared office space with more senior colleagues contributed to strong working relationships. iii. The panel suggest that the PhD community nominate a representative to deliver formal feedback to DERI committees on recommendations, suggestions or issues requested by students. iv. The students who we spoke to appear to be mature and professional and focused. e. Administrative Assistants i. All the researchers stressed that they are very happy with the support provided by the administrative staff. ii. It would be desirable if closer links with administrative service providers elsewhere in the University could be established.
6 iii. The Institute has employed its own accountant. This appointment is funded from external sources. iv. As alluded to above, there is some duplication between administrative functions borne in­house in DERI and those offered centrally by the University, which contributes significantly to the workloads of DERI’s administrative staff. v. Improved resources should be provided by central university to off­set the provision of administrative staff currently funded by external research grants. f. Technical Staff and Outreach Officer i. Technical staff are highly skilled and knowledgeable about the area in which they work. ii. There is a good working relationship between the technical staff and Computer Services technical staff. One of the technical staff had worked for a number of years in the Computer Services Department. iii. The issue of having increased front­line technical support was raised. A suggestion of having a DERI technical help desk was made. iv. The Outreach Officer was very experienced and effective at engaging the wider community. The outreach programme seems well resourced and serves to increase the profile of DERI both locally and at a national level. 3. Programmes and Instruction The topic of PhD supervision arose repeatedly throughout the review. Although DERI has a large number of PhD students, the number of PhDs supervised to completion by DERI staff members to date (within or outside NUI Galway) is very low. In addition, most of the supervision is undertaken by senior researchers who are not members of the permanent academic staff of the university. As the University’s own guidelines state that direct supervision should “normally” be carried out by permanent staff (University Guidelines for Research Degree Programmes, Sect. 5), the heavy reliance on adjunct staff supervisors—although pragmatic in DERI’s case—may be said to be suboptimal. The University’s Guidelines both enshrine the institution’s duty of care to its students and serve to influence the students’ own expectations of their entitlements: as such, the current supervision model within DERI is potentially an area of concern for the University. Our discussions with PhD students suggested that the students are generally happy with their experience in DERI. Several PhD students praised the Institute as a whole and expressed pride in its attainments and international standing. Students also highlighted a number of concerns, most of which were flagged by the DERI postgraduate survey of June 2008. The Review Group was very impressed by the recent work conducted within DERI on establishing PhD Best Practice Guidelines for the Institute, based on the guidelines
7 recently adopted at University and College level. If implemented successfully, these guidelines should go a considerable way to resolving the concerns mentioned above. Recommendation 3: The operation of the new PhD regulations for DERI should be reviewed after 18­24 months by a small group that includes a nominee of the Dean of Engineering and Informatics. The review should include a survey similar to that conducted in June 2008. We noted two aspects of the DERI regulations that should be amended to comply with University guidelines. Recommendation 4: (a) The Postgraduate Research Committees (PRCs) within DERI should each contain at least one professor (cf., University Guidelines, Sect. 4.1); (b) All first­time PhD supervisors must receive training on student supervision and project management (cf., University Guidelines, Sect. 5). Further to Recommendation 4, we encourage DERI to cultivate an ethos of continuing professional development as regards their supervision duties, to ensure currency in the Institute’s pedagogical activities. The Review Group discussed the question of whether PRCs should be required to contain at least one experienced PhD supervisor from outside DERI. This would not only bring additional experience in supervision, but would also assist in the integration of DERI within the University. We decided not to make this a formal recommendation, but we urge that DERI give serious consideration to such a practice. Some of the concerns highlighted by PhD students arise from the tension between PhD studies and work on funded projects. The experience of working on such projects is potentially of significant benefit to the PhD students in DERI, but it does pose challenges for the students and the Institute. Recommendation 5: DERI should work with other units in the University, under the auspices of the Office of the Dean of Graduate Studies, to assist in the development of modules in areas such as project management and entrepreneurship. These will then form part of structured PhD programmes. Since DERI students have a particular need for such modules, and there is particular expertise in these areas within DERI, it is vital that DERI be to the forefront as such modules are rolled out within the wider University. A number of PhD students also expressed the desire to gain teaching/tutoring experience. While reference was made to the low current intake levels on the main undergraduate programme in the Department of IT, it is the case that the school offers a range of ancillary programmes at undergraduate and postgraduate levels, some in collaboration with other Schools and Colleges. As such, it should be possible to find ample opportunities for tutoring experience within the University, and students should be assisted in this.
8 At present, DERI does not have any formal teaching degrees at the Masters level. This situation is consistent with the fact that DERI did not grow out of pre­existing local expertise, and, thus, is not tightly linked to other academic activities of the University. But DERI has significantly grown over the past few years, and needs to address strategic issues regarding its sustainability in the long run, so this issue may warrant serious consideration. The establishment of a Masters course would meet some internal needs stated before (desire of PhD students and research staff to engage in teaching) and, more importantly, might assist in attracting and developing more talent to support its future operations. Due to its strong international reputation, DERI would also be perfectly placed to participate in teaching activities at the European level related to semantic web education. Clearly, DERI cannot support such a serious expansion of its core activities with its current personnel. Such a step could only be undertaken in close cooperation with the College of Engineering and Informatics, and with the commitment of the University to fund additional permanent academic staff within DERI. Recommendation 6: DERI should seriously consider the establishment of a Masters programme, in close cooperation with the College of Engineering and Informatics and other University units. Such a step should be planned with the view of reducing the dependency on external competitive research funding. The University should consider how such a move could be supported in terms of the allocation of appropriate financial and human resources. 4. Scholarship and Research 4.1 The Research Problems The activities of DERI focus around two fundamental problems: (a) social semantic information spaces, bridging semantic web and Web 2.0 technologies; (b) semantic reality, using semantic technologies to interpret, combine and use low­ level sensor input coupled with rich semantic descriptions. Both problems are important for the future of the information society and knowledge economy. Problem (a) is concerned with providing the right tools so that people can work together on creating and using new knowledge. Problem (b) is concerned with how input from heterogeneous sources can be interpreted and combined to produce new knowledge and services. Semantic technologies appear promising to address these issues and associated security and privacy requirements. Advances on these problems will have implication to a number of areas of human activities. Thus, a third direction of the DERI activities is to study various application domains, in order to influence these domains, study rich test­beds for the new technologies and to demonstrate the utility of these technologies. Overall, the problems identified are challenging and strategically important for the creation of the information society and knowledge economy.
9 4.2 Publications, Activities and Reputation DERI has grown dramatically over the past few years, and in doing so, has strengthened its research base. Together with this development came a strong research output, strong involvement in the activities of the international research community, and a high international reputation. DERI is considered to be a key player in the area of the semantic web, thus being among those groups that shape the future of this field. In terms of activities, DERI staff are actively involved in organising international conferences and workshops, presenting tutorials and courses in international fora, and participating in key international standardisation activities. Also, a number of high profile alliances with key international players have been established. DERI staff have been very successful in publishing papers in the best international conference on the semantic web, such as ISWC and WWW conferences. It seems to have developed an ethos of encouraging its staff to publish in these key events. The journal output of DERI can be considered to be a relative weakness. One needs to take into account the pressures created by the necessity to be self­sustained financially, as well as the greater importance of conferences in the ICT domain, compared to other scientific disciplines. Still we believe that more effort should be placed at publishing in top international journals of the area. Recommendation 7: DERI should improve its publication record in leading international journals. 5. Community Service DERI is exceptionally strong in the area of Outreach, and because of its dedication to issues such as integration of immigrant communities, it is skilled at engaging the community with these ideas. The panel were very impressed with the amount and scope of outreach activities offered by DERI and the professionalism of the outreach programme. In fact, DERI has won many awards over the past few years due to its innovative outreach programme. The Outreach Officer described an initiative called ‘BEO’, which combines new technology with Cultural, Social and Natural Heritage. This is just one of many on­going outreach projects, which have very clear value in contributing to learning across generations. There is very strong participation in internally organised events, such as graduate recruitment fairs and in local events such as the Galway Science and Technology Fair. The provision of educational outreach activities to primary and secondary schools is very worthwhile and the engagement with community activities will continue to be important in establishing DERI as being nationally recognised. The role of Outreach Officer is a crucial one and is important in influencing the perceptions of DERI and its role, both inside and outside the University. The Outreach Officer was very experienced and expert at engaging the wider community and it is clear that DERI contributes in a significant way to the community of the University and to the wider community.
10 6. Summary and Concluding Remarks The panel were convinced that DERI is working at the cutting edge of research in semantics and web science. The Institute has come through a period of uncertainty in the early stages of its evolution, but is now recognised as one of the world leaders in the area of Semantic Web and Semantic Web Services. The Institute has clear management strengths, and is effectively run and organized. The main issues which arose during the course of the review relate to integration, supervision, and sustainability. A summary of the main issues include: Ø Impressive growth despite the difficulties in the initial stages of establishment Ø Excellent research, international leader in the area of semantic web and semantic web services Ø Well managed, professional operation Ø All staff pulling together in the same direction, high respect for managers Ø Good working environment and collegiate atmosphere Ø Excellent track record in attracting research funding allowing DERI to be self­ sustaining Ø State­of­the­art equipment and facilities Ø Strong industry collaboration and relevance Ø Research staff have access to excellent international networks The following recommendations are meant to improve some aspects BUT we would warn against changes, particularly those of an administrative nature that might endanger the success so far. Specific Recommendations: Recommendation 1: The management structure should be revisited and modified to include a clear conflict resolution policy within DERI. Recommendation 2: A strategy for increasing engagement (from within and perhaps beyond NUI Galway) with academic staff should be developed. This strategy might incorporate incentives (such as teaching relief or credit toward promotion) for participating in major DERI operations, subject to relevant University career­ development policies. As a follow­on from DERI's strategic planning exercise for the period 2008­2013, it should prepare an outline sustainability plan for the 10­15 year horizon, considering best­case, expected and worst­case scenarios for support within NUI Galway and from external sources. Recommendation 3: The operation of the new PhD regulations for DERI should be reviewed after 18­24 months by a small group that includes a nominee of the Dean of Engineering and Informatics. The review should include a survey similar to that conducted in June 2008.
11 Recommendation 4(a): The Postgraduate Research Committees (PRCs) within DERI should each contain at least one professor (cf., University Guidelines, Sect. 4.1). Recommendation 4(b): All first­time PhD supervisors must receive training on student supervision and project management (cf., University Guidelines, Sect. 5). Recommendation 5: DERI should work with other units in the University, under the auspices of the Office of the Dean of Graduate Studies, to assist in the development of modules in areas such as project management and entrepreneurship. These will then form part of structured PhD programmes. Recommendation 6: DERI should seriously consider the establishment of a Masters programme, in close cooperation with the College of Engineering and Informatics and other University units. Such a step should be planned with the view of reducing the dependency on external competitive research funding. The University should consider how such a move could be supported in terms of the allocation of appropriate financial and human resources. Recommendation 7: DERI should improve its publication record in leading international journals Overall, DERI is an international research leader in the area of semantic web technologies, well placed to shape the future of this field, both scientifically and technologically. In this report, we have tried to analyse its operation and performance, and outlined a number of recommendations for further improvements. However, any significant changes of strategic importance need to be carefully studied and planned so that they do not jeopardise the success story DERI has achieved so far. Professor Grigorios Antoniou (Chair) Professor Orla Feely Dr. Brian Hughes Dr. Martina Prendergast (Rapporteur) (5 th March 2009).
12 Comments on the Methodology of the Review Process The Review Panel wish to express their sincere thanks and gratitude to the Director of Quality and his team for their professionalism and efficiency in the co­ordination of this quality review.
13