UK e-Science Technical Report Series ISSN 1751-5971 Research School of Systems Engineering A Visualization Service for the National Grid Service: A Workshop to derive User Needs Systems Engineering Innovation Centre, Loughborough University R.S. Kalawsky, Loughborough University 18-05-2006 Abstract: This report gives an account of the visualization requirements workshop held at the Systems Engineering Innovation Centre, Loughborough University on 18 May 2006. The workshop brought together users, visualization practitioners and potential service providers to discuss a proposal to augment the National Grid Service (NGS) with a National Visualization Service (NVS). The workshop discussed the demand from the user community for an NVS, and the form that any such service should take. It was agreed that there is consensus support for an NVS, as a distributed and diverse service with a far greater level of provision than PC desktop hardware can offer. The workshop highlighted the need to consider visualization requirements from a broad spectrum of users, in the social sciences, arts and humanities as well as science and engineering. Furthermore, there are diverse computational needs, from data-centric operations, to visualization-intensive tasks, to interactive computational steering. The benefits to the community of collaborative visualisation working were recognised, as was a need to gather the views of more users to strengthen the case for an NVS. Potential users will need training and support, which VizNET can provide. Users will also require a facility to reserve visualization facilities. Software licensing is an issue that may need vendor participation to resolve. With regard to funding, the workshop recommends that JISC consider including an NVS within their budgetary planning for funding as soon as possible. UK e-Science Technical Report Series Report UKeS-2006-05 Available from http://www.nesc.ac.uk/technical_papers/UKeS-2006-05.pdf Copyright © 2006 The University of Edinburgh. All rights reserved Research School of Systems Engineering A Visualization Service for the National Grid Service: A Workshop to derive User Needs Date Held: 18 May 2006 Venue: Systems Engineering Innovation Centre, Loughborough University Summary There is growing evidence that advanced scientific users (and other developing communities within the social sciences and, arts and humanities) require access to high end visualization resources that are too expensive or generally unavailable within their respective organisations. Currently, the National Grid Service (NGS) does not provide a visualization service thus leaving the user to process any visualization locally. The addition of a national visualization service with the capability to process large and diverse datasets and deliver complex visualizations as a complete end to end service is an attractive and compelling proposition. The national visualization service could form an important adjunct to the NGS. In order to support a wide user base any high-end visualization resource will to need to satisfy a diverse set of user needs through a distributed or remote visualization capability. This report identifies recommendations from a visualization requirements workshop held at the Systems Engineering Innovation Centre, Loughborough University on 18 May 2006. The workshop brought together users, visualization practitioners and potential service providers. Compiled by: R.S. Kalawsky Loughborough University VizNET-LOUGH-0001-060811 1 Issue 1 Research School of Systems Engineering Contents 1. 2. 3. 4. 5. Introduction ................................................................................................................................3 Workshop Process ......................................................................................................................4 Emerging Factors/issues to be considered arising from the Keynote address............................6 Emerging Factors/issues to be considered arising from the NGS presentation..........................8 Establishing the need – Users perspective/requirements..........................................................10 5.1. Users Needs Session #1 .......................................................................................................10 5.2. Users Needs Session #2 .......................................................................................................12 6. Service Provision......................................................................................................................14 6.1. Service Provision Session #1 ...............................................................................................14 6.2. Service Provision Session #2 ...............................................................................................16 7. Plenary Session.........................................................................................................................19 8. Recommendations ....................................................................................................................20 9. Concluding Remarks ................................................................................................................21 10. Acknowledgements ..................................................................................................................21 11. Appendix A: Attendees ............................................................................................................22 VizNET-LOUGH-0001-060811 2 Issue 1 Research School of Systems Engineering 1. Introduction Visualization is acknowledged as an important area within the UK academic research community requiring additional support. Consequently, a number of visualization centres around the UK have been funded by JISC to harness their strengths and create a virtual entity that provides both a regional and a national perspective. The resulting UK visualization network (VizNET) is reaching out to visualization users to offer advice, training and examples of best practice. VizNET has been motivated by the availability of cost effective visualization hardware that makes it feasible for research groups (who would not normally be regarded as serious visualization users) to consider improving presentation and understanding of their data sets by employing advanced visualization techniques. There is growing evidence that advanced scientific users (and other developing communities within the social sciences and, arts and humanities) require access to high end visualization resources that are too expensive for or generally unavailable within their respective organisations. Currently, the National Grid Service (NGS) does not provide a visualization service thus leaving the user to process any visualization locally. The addition of a national visualization service with the capability to process large and diverse datasets and deliver complex visualizations as a complete end to end service is an attractive and compelling proposition. The national visualization service could form an important adjunct to the NGS. In order to support a wide user base any high-end visualization resource will need to satisfy a diverse set of user needs through a distributed or remote visualization capability. This report identifies recommendations from a visualization requirements workshop held at the Systems Engineering Innovation Centre, Loughborough University on 18 May 2006. The workshop brought together users, visualization practitioners and potential service providers. VizNET-LOUGH-0001-060811 3 Issue 1 Research School of Systems Engineering 2. Workshop Process The format for the workshop is shown in Figure 1 and consisted of an initial plenary session followed by two sessions and then a final plenary session to discuss and formulate an agreed set of requirements. Figure : Workshop Format The workshop commenced with an invited keynote address by Professor P.V. Coveney, who presented examples of where visualization had been successfully used in a number of important eScience projects. This was followed by a short introduction of the National Grid Service (NGS) by Dr S. Pickles on how interested parties could become NGS partners or affiliates. The workshop was arranged in two parts: Part 1: User Needs - The objective was to try and establish if there is strong support from the user community for a national visualization service and if so, what form this should take. Part 2: Service Provision - Assuming that there is a case for support then what form should the service provision take. Due to the number of people who attended this workshop it was necessary to split the attendees into two parallel sessions, each chaired by a facilitator and supported by a scribe. This was also necessary to reduce the potential for biasing or possible dominating effects caused by some institutions being represented by a larger number of people than others. Networking opportunities VizNET-LOUGH-0001-060811 4 Issue 1 Research School of Systems Engineering were provided over lunch and afternoon tea breaks. This also allowed the session facilitators to ensure each session was progressing as required. Each facilitator promoted discussion by means of a brainstorming session to capture requirements and issues. A scribe was provided to record important discussion points on colour coded post-it notes. At appropriate times in each session the facilitator organised the post-it notes into a logical structure. Output from the workshop will be used to inform the JISC Committee for Supporting Research (JCSR) of the strength of support for a national visualization resource as part of an extension to the NGS. It is hoped that if sufficient support exists in the visualization community then JISC may consider funding an open call to provide a visualization capability for the NGS. VizNET-LOUGH-0001-060811 5 Issue 1 Research School of Systems Engineering 3. Emerging Factors/issues to be considered arising from the Keynote address Visualization on the grid – Peter Coveney The presentation given by Peter Coveney raised a number of important issues worth noting. These are summarised below: • The current e-Science infrastructure is incomplete and does not yet fully provide the end to end capability that people will expect from a future national resource. • It is probably more important to use the term ‘High end computing’ rather than High performance Computing (HPC) because scientists will want to be able to make use of any resources that are off the desktop. HPC tends to be associated with large computing systems. • Visualization is indispensable in computational science and plays a major role in revealing important information in large complicated data sets. In order to ensure simulations are free from finite size effects, the trend will be for larger data sets which in turn will lead to more demanding visualization challenges. • Real time interaction and computational steering interaction between simulations and visualizations with distributed infrastructure is likely to be a key requirement. • What is the best combination of software and hardware? • Activity across administrative domains will become an increasingly important issue for grid based resources. This will have an international dimension because of international collaborations. It is helpful to remind ourselves of the following important points, as identified in Peter Coveney’s slides and extracted from: Ref – ‘International Review of Research Using HPC in the UK’ - December 2005 – EPSRC (ISBN 1-904425-54-2), http://www.epsrc.ac.uk/CMSWeb/Downloads/Other/HPCInternationalReviewReport.pdf Recommendations: 2. Strengthen the computational infrastructure by: a. Systematically deploying leading-edge capability systems, large-scale capacity computing, and resources deployed widely at universities b. Supporting and developing a state-of-the-art applications software infrastructure encompassing algorithms, data management and analysis, visualization, and best practices software engineering. (p16) VizNET-LOUGH-0001-060811 6 Issue 1 Research School of Systems Engineering 4.4.Visualization Resources While the reviewed research groups well understand that visualization is indispensible in computational science, the Panel observed that visualization in the UK lags behind international standards. ..... The Panel are concerned that without an improvement in visualization sophistication (both hardware and software) hidden scientific treasures will increasingly lurk undiscovered in the massive data to be produced by the enchancement of HPC capability and capacity. ...... Therefore, the Panel recommends that the UK HPC community prepare immediately to establish a balance among leading-edge computing facilities, visualization technologies, and welleducated computational scientists. (p15) VizNET-LOUGH-0001-060811 7 Issue 1 Research School of Systems Engineering 4. Emerging Factors/issues to be considered arising from the NGS presentation What it means to join the NGS – Stephen Pickles Stephen Pickles gave a useful introduction to what it means to join the National Grid Service (NGS). The UK's National Grid Service (NGS) (http://www.ngs.ac.uk) provides a core eInfrastructure that underpins UK e-Research, providing standardised access to data management and compute resources. The aim is to support and facilitate collaborative computing across the UK. The NGS also provides a national "gateway" to international collaborations through collaboration with related e-infrastructures internationally. The National Grid Service entered full production in September 2004 at which point the Grid Operations Support Centre was created with support from the EPSRC core e-Science programme. The National Grid Service resources comprise the 4 founding members of the CCLRC, University of Oxford, White Rose Grid - Leeds and University of Manchester, plus the national HPC facilities of CSAR and HPCx and a growing number of partner and affiliate sites. There are two distinct ways for sites to join the NGS, i.e. as an NGS partner, or as an NGS affiliate. Both partners and affiliates run NGS compatible software, and integrate monitoring and support arrangements with the NGS. Partners also contribute significant resources or services to NGS users. Full details can be downloaded from http://www.ngs.ac.uk/man/documents/NGSBaseline-2.4.pdf. Current NGS partner sites NGS partners are sites that meet the requirements for affiliation with the NGS and in addition: • • • Define level of support commitments though a Service Level Description (SLD), as agreed with the GOSC Board May include in their SLD commitments to provide additional services to NGS approved users or projects Allow additional monitoring and accounting for verification of the services provided. Services offered may include access to hardware resources, data archives or appropriately licensed software in addition to that required by the NGS The NGS currently has four partner sites: • Bristol Grid Node • Cardiff Grid Node • Lancaster Grid Node • Westminster Grid Node VizNET-LOUGH-0001-060811 8 Issue 1 Research School of Systems Engineering Current NGS Affiliate sites To affiliate with the NGS an institution or resource provider must: • • • • Deploy and support the minimum required set of NGS software to enable interoperability with the NGS central services and other NGS sites. Provide access to allow NGS monitoring Agree to the NGS conditions of use and security practices. Sites should accept certificates issued by the UK e-Science Certificate Authority and those Certification Authorities with which the UK e-Science Programme has reciprocal agreements. The Certificate Revocation Lists are updated on a regular basis. Current sites conforming to the NGS Affiliate status are: • National e-Science Centre, Edinburgh VizNET-LOUGH-0001-060811 9 Issue 1 Research School of Systems Engineering 5. Establishing the need – Users perspective/requirements Key Questions to be addressed during the ‘Users needs session’ include: Is there a case for a Visualization Service to augment the NGS? Who are the potential users? What applications would it support? Aims and objectives of service? 5.1. Users Needs Session #1 The results from the User Needs Session are shown in Figure 2. Figure 2: User Needs – Session #1 VizNET-LOUGH-0001-060811 10 Issue 1 Research School of Systems Engineering The main points that emerged include: • NGS polices would need to change to incorporate a visualization component. • There is a requirement to support a wide range of users • Users would need to be targeted toward appropriate visualization resources (not all visualization applications need the same resource). This is especially true for the visualisation display hardware. For example, a CAVE is not appropriate for all applications. • Many visualization users will require access to familiar tools on the grid. • This may present licensing issues because licenses are typically specific to particular machines and in some cases particular users. • There are a wide range of applications involving visualization – this is likely to grow because of the need to increase the scale of jobs. As new users (in non-scientific communities) take up greater use of visualization resources then initial/existing resources will become resource limiting. • There may be a case for a single ‘monster’ visualization resource as well as a wider network of ‘mini monster’ visualization resources. This could save institutions from having to spend vast sums of money on a resource that could not be fully justified by a single institution. • What form would a ‘monster’ visualization resource take? • Improved tools to support remote visualization will be required (One commercial supplier who provides such tools may soon go out of business) o Which service e.g. VizServer – availability question? • Visualization covers interactive and batch techniques. o Interactive visualization presents a number of difficult challenges including: Highly interactive computational steering extremely demanding – most difficult problem Resource scheduling to support interactivity Improved networking to facilitate interaction o Batch visualization could support high quality animations and render farm visualization requirements • Visualization should include all modalities (visual, auditory and haptic) • Some users will require access to visualization on the move – lightweight visualization clients will need developing • Collaborative tools should support a shared visualization environment VizNET-LOUGH-0001-060811 11 Issue 1 Research School of Systems Engineering o This raises interesting challenges such as scalability of collaborating environments o How to incorporate/facilitate Access Grid style collaborations • Where is the data located with respect to the location of the visualization rendering? o Data remote from visualization – data transfer issues o Data local to visualization – local storage issue o Where is the data computed with respect to computational resource location o This will be a trade off decision – novice users may need help with this 5.2. Users Needs Session #2 The results from the User Needs Session #2 are shown in Figure 3. Figure 3: User Needs – Session #2 Additional comments not covered by User Needs Session #1 • There is a strong case for training resources to aid users in using visualisation tools and visualising their data sets. • Visualization is a demanding subject with a much wider diversity than general computational resources VizNET-LOUGH-0001-060811 12 Issue 1 Research School of Systems Engineering • Visualization is more multi-disciplinary than almost any other area of computing • Users from disciplines out side of computational science may require demonstrations or tutorials to realize the potential of visualisation. • There is a chicken and egg situation – it is difficult to determine real user demand until users have been shown the possibilities that powerful visualization resources can offer • There are different classes of user – their demands are likely to be different o Power user – use every machine available o Occasional user – investment in visualization facility may not be possible o There may be visualization tool developers • Different user classes will have varying levels of expertise and require different levels of training and support • There is a widespread concern that full economic cost recovery (FEC) will force institutions to charge for use. It is unclear how this can be accommodated within the current NGS partnership model. • A national visualization service might be characterised as being a "wide-area distributed collaborative visualization service". • We need to have the views of more users in order to strengthen the case – the attendees of the workshop do not represent the full user community. Need to consider visualization in social sciences, arts and humanities – these are less well understood. VizNET-LOUGH-0001-060811 13 Issue 1 Research School of Systems Engineering 6. Service Provision Key Questions to be addressed during the ‘Service Provision session’ include: Issues of Service Provision? How to manage access to remote visualization tools and environments? One size fits all versus support to all? Requirement for common software stacks? Options for providing Visualization Service Single site versus multiple sites? Funding Options/Mechanisms? How to facilitate access to display hardware? 6.1. Service Provision Session #1 The results from the Service Provision Session #1 are shown in Figure 4. Figure 4: Service Provision Session #1 Issues: • How to manage interactive services (for real-time activities)? It will be necessary to reserve graphics machines for interactive elements with separate visualization machines for rendering. This resource reservation requirement is different from typical compute services. VizNET-LOUGH-0001-060811 14 Issue 1 Research School of Systems Engineering • The issue of suppliers was discussed – one of the major suppliers of high-end graphics may not be around for very much longer. This means people will have to rely more on PC class graphics hardware. This may not be an issue for most users because PC graphics hardware is improving all the time. Additionally greater functionality and innovation is being stimulated by the gaming industry. For instance, XBox live is one example of how the on-line multiplayer gaming industry is driving developments in collaborative environments, and scene rendering. • Academic institutions are adopting FEC and this means that institutions will typically levy a charge against all facilities and space. This may affect how institutions participate in the current NGS model. It will be necessary to show a real cost benefit such as increased access to resources and greater redundancy. Managing Access: • Job submission mechanisms for NGS will need changing to accommodate visualization resource access o For example, interactive operation (i.e. real-time interaction with remote visualization and computing resources for computational steering) needs to be added to NGS to deal with interactive use. • Visualization system configuration may be better controlled through the use of a wizard to assist the user. • Policies will need to be developed to cater for the diversity of visualization tasks. • Co-reservation schemes will need to be developed? One ‘size’ fits all versus support all: • Diversity of visualization users and tasks will require different visualization resources (and software) – one ‘size’ will not fit all • It will be necessary to support visualization service for batch jobs and interactive sessions • User support needs will vary according to their expertise (some users will be expert and require specialised support whilst others will be novice users and require more fundamental visualization support (VizNET can play a role here)). • Visualization tasks require very different visualization hardware (some tasks require polygon or triangular mesh processing whilst others require much greater sophistication. Provision Options: • Whatever service is provided, it must be reliable and robust. • Unlike general computing resources where multiple jobs may run concurrently a visualization machine will only be able to manage a single interactive session per pipe. This will need careful resource management and user prioritisation. VizNET-LOUGH-0001-060811 15 Issue 1 Research School of Systems Engineering • Users will need additional support to manage their visualization jobs Hardware: • How will institutional nodes be linked in (a common software stack may not be appropriate)? • Interactive visualization will require good quality of service from the network. • A test-bed system can be built now and be based on current software stacks, and range of software options for applications. This could explored to help understand/test user needs. Miscellaneous: • We will need to achieve the same level of capability as our international collaborators in order to stay in the game 6.2. Service Provision Session #2 The results from the Service Provision Session #2 are shown in Figure 5. This session tried to imagine what a National Visualization Service would look like in order to derive requirements. Figure 5: Service Provision Session #2 VizNET-LOUGH-0001-060811 16 Issue 1 Research School of Systems Engineering Access to remote visualization: • ‘One stop’ sign in is needed for users to quickly adopt a remote visualisation system • Screen/room booking tool 1. Sharing display resources such as a CAVE or stereo display wall will require tools to allocate time and generate billing • Sharing of physical resources issue: 1. Visualization machine will only be able to manage a single interactive session per pipe 2. User prioritisation – resource owner or first come first served • Opportunistic resource use verses allocated resources: 1. Some users will want to use any resource to visualise data on demand 2. Other users will require allocated times for visualisation (e.g. demos) • Detailed requirements descriptions are needed to correctly allocate visualisation resources • A video service might be useful – ideally a batch service to give high quality rendering Local versus remote visualization: • Close coupling with NGS will be required for some tasks – Co-allocation with simulations will be required by some users • Data caching or staging services may be necessary for interactive visualisations remote from the data source. • Bi-directional issue for computational steering (remote visualization issue) One ‘size’ fits all versus support all: • There is a requirement to support a range of different visualization services (heterogeneous visualization services). • There is a perceived need for different security requirements (e.g. medical data will require much tighter security than some other scientific tasks) o Requirement for secure authentication and authorisation to visualisation resources o Secure data integrity (encrypted data and image streams) may also be necessary • Need to acknowledge different user communities o User who has their own visualization code o User who wants set application software to be provided o User (this is probably a significant group) who need the NGS for some part of their work VizNET-LOUGH-0001-060811 17 Issue 1 Research School of Systems Engineering Common software stacks: • Application software will be user demand driven – wide diversity of applications. • Probably sensible to include some core software provision • Support for popular visualisation workflow builders Single versus multiple sites: • One size does not fit all – a visualization service must cater for different people • A good idea to have a pool of resources • Diversity important (specialist sites) • Some sites could offer specialised display resources (CAVES, Domes, Reality Centres, etc.). • Facilities based across multiple sites will provide a degree of redundancy and resilience. • Data locality – how to move large data around • Broker facility to support usage of different hardware facilities at different sites • Need to keep things simple • Licenses will be an endemic problem for the NGS (who pays for licenses and how do users who own licenses execute code on a remote site?) Funding Options: • Funding is required urgently to establish a National Visualization Service • Recommendation that JISC funds the National Visualization Service through its Infrastructure or other budget VizNET-LOUGH-0001-060811 18 Issue 1 Research School of Systems Engineering 7. Plenary Session The final plenary session provided an opportunity for the two sessions to meet to review the outcomes from the workshop. In particular, the following were considered. Harmonising viewpoints Making the case Recommendations & Conclusions Is there a clear justification for a national visualisation service for the NGS? Both sessions fully endorsed what the other session group had discussed and formulated as key issues and recommendations in next section. VizNET-LOUGH-0001-060811 19 Issue 1 Research School of Systems Engineering 8. Recommendations Recommendations arising from the workshop (not in any order of priority) include: National Visualization Service • There was consensus support for a National Visualization Service, to run alongside the NGS • This should be a distributed, rather than centralized, service in which different sites would offer different facilities and access to different visualization resources (to best meet national needs in terms of capability and geography) • The level of provision will be orders of magnitude greater than can be provided by PC desktop hardware User Requirements • We need to consider visualization requirements from a broad spectrum of users – for example, users in social sciences, arts and humanities have requirements that are currently less well understood than those from science and engineering community (and are likely to be very different) • Needs span data-centric operations to highly specialized visualization intensive tasks and support for interactive computational steering • Collaborative visualization working could be of major benefit to the community – especially at an international level • We need to gather views of more users in order to strengthen the case for the service Accessibility • Barriers may exist until potential users are up to speed – VizNET has a role here in providing training and support. • Reservation of visualization facilities is important – for example, for interactive computational steering, or high-profile demonstrations • Software licensing is an issue that may need vendor participation to resolve. Funding • JISC should consider including a National Visualization Service within their budgetary planning for funding as soon as possible VizNET-LOUGH-0001-060811 20 Issue 1 Research School of Systems Engineering 9. Concluding Remarks The overall recommendation from the workshop attendees supported the view that there is a need for a distributed National Visualization Service. A test-bed system could be built on current software stacks, and range of software options for applications now if required. However, the exact form such a service will take needs further definition work by considering the wider spectrum of users. Whilst the case could be made for scientific/engineering use it would be prudent to consider other disciplines. It should be stressed that whilst the workshop was extremely well attended there is a need to provide a more complete picture by including other potential visualization users. A ‘Birds of a Feather’ (BOF) session is currently scheduled for the All Hands Meeting 2006 in September to look at Computation Steering and Visualization. In addition an additional workshop based on the one reported in this report is being organised (date to be confirmed in October 2006 at NeSC) and will include people from other disciplines. Output from this and the other workshops will be consolidated into a report that reflects the view of the wider visualization community. This report will also address the funding implications and service level provision. 10. Acknowledgements Support from Ken Brodlie, Stephen Pickles, Nick Avis, Martin Turner and Alex Hardisty in reviewing this workshop report and for making helpful suggestions are gratefully acknowledged. VizNET-LOUGH-0001-060811 21 Issue 1 Research School of Systems Engineering 11. Appendix A: Attendees Group 1: Roy S. Kalawsky – Loughborough - (Group 1 Facilitator) Ian Holmes – Loughborough - (Group 1 Scribe) Chris Greenwell – Bangor Joanna Leng - Manchester Stephen Pickles – Manchester Rob Haines – Manchester Tobias Schiebeck – Manchester Lakshmi Sastry – CCLRC Ronald Fowler – CCLRC Nick Avis – Cardiff Steven Young – Oxford Rob Aspin – Salford James F Annett – Bristol Peter Coveney – UCL Shantenu Jha – UCL Stefan Zasada – UCL Mary-Ann Thyveetil – UCL Marco D. Mazzeo - UCL Group 2: Ken Brodlie – Leeds - (Group 2 Facilitator) John O’Brien – Loughborough - (Group 2 scribe) Helen Wright – Hull Paul Hatton – Birmingham Ian Grimstead – Cardiff Alex Hardisty – Cardiff Srikanth Nagella – CCLRC Martin Turner – Manchester Alastair Knowles – Edinburgh Colin C. Venters – Manchester Yuwei Lin – Manchester Mark Riding - Manchester Robert Frank - Manchester Jeremy Walton - NAG Clare Gryce – UCL James Suter – UCL Simon Clifford – UCL Radhika Saksena – UCL Stefan Zasada - UCL VizNET-LOUGH-0001-060811 22 Issue 1