The University of Bristol Grid, a production campus grid
David Wallom, Ian Stewart & Jon Wakelin
University of Bristol
Abstract
With the increasing cost of providing resource for research computing it is becoming
increasingly important that Universities make more efficient use of existing resources. One
approach adopted by the University of Bristol to address this issue has involved the design and
construction of a campus grid; the UoBGrid. This has enabled better utilization of systems that
have been connected to the grid. As the grid is more suited for processing serial work, the
UoBGrid has the additional advantage of reducing the throughput load on clusters, thereby
allowing these systems to process more parallel work Within the University there are many
departments that run their own computational clusters. In addition there are several centrally
and departmentally managed student PC terminal rooms whose resources lie idle for significant
periods. Stage one of the Bristol project involved coupling several Departmental Beowulf
clusters to form the nucleus of the UoBGrid. After consideration of available solutions, we
designed our own Resource Broker, Resource Usage Service and Virtual Organisation
Management Servers which have proved easier to manage and upgrade as new or improved
functionality is required. The current system peak usage is ~15000 individually submitted jobs
per week with ~3500 in a single day. The largest single job successfully run was a 2000 child
process protein-ligand docking experiment.
1. Introduction
With the increasing cost of providing resource for
research computing it is becoming increasingly
important that Universities make the most efficient
use of existing resources. One approach adopted by
the University of Bristol to address this issue has
involved the design and construction of a campus
grid; the UoBGrid. This has enabled the utilization
of otherwise spare capacity on those systems that
have so far been connected to the grid. The
implementation of this strategy was driven by the
following major considerations. Firstly, the
contributing clusters are very heavily used and
connecting them onto the grid must be done in a
non-invasive way to ensure currently executing
applications, or any other cluster operations, are not
compromised. Secondly the grid has been
considered by many scientific users to be over
hyped. This latter issue led us to believe that we
needed to get a system up and running quickly
however ‘simple’ it appeared in order to prove the
concept was viable. It was hoped that involving real
users at an early stage of the UoBGrid’s construction
would encourage more users or resource owners to
join.
2. Systems Installations
A key driver for the software choice to run UoBGrid
was that the University of Bristol had agreed to
become one of the first volunteer nodes joining the
National Grid Service (NGS). It was decided that
each local Bristol cluster joining UoBGrid would
have the VDT [1] middleware installed. This
provides the Globus, GSI-SSH, myProxy and grid-
mapfile tools components. Each system would also
have the SRB client and Big Brother monitoring tool
client installed to provide added functionality not
available with VDT. Installing middleware is in
general a complicated procedure so for simplicity an
application specific installation script was designed.
On start-up, the script takes all of the variables
required to configure VDT. The specified version of
the base and added components such as the
jobmanagers, GLUE schema and grid mapfile
management tools are automatically downloaded,
installed and configured. To allow for easy updating,
the installation directory is then renamed to account
for version number and a soft link to the operation
directory created. This way files that have specific
hardcoded locations may be preserved during
upgrades.
This installation script, although initially only for
clusters, has been extended to the UoBGrid central
service stack to make installation and configuration
as easy as possible. The central service stack was
considered vital in order to fulfil the functions of
resource brokering, information and monitoring data
publishing, virtual organisation management and
large storage. The hardware systems for this service
stack were kindly donated by Viglen Computers [2]
and arrived in June 2004. The key functionality on
these servers is provided by the following products,
Condor-G [3], Globus MDS [4], Big Brother [5] and
Storage Resource Broker [6].
Several of the components in the UoBGrid software
stack have been configured as out of the box. An
example of this is the MDS system which has a
standard VDT installation, with all services except
the Monitoring and Discovery service stopped. This
runs a standard Grid Index Information Server
(GIIS) which has all the UoBGrid systems reporting
to it, i.e. not just the Beowulf clusters but also the
central UoBGrid servers. Extra information loaded
into the GIIS is provided by the Glue Schema [7]
which allows significantly more information to be
obtained from the installed scheduling systems on
clusters so that improved reference can be made for
job distribution and broking.
The key services underpinning a campus grid are the
Resource Broker, Resource Usage Service and
Virtual Organisation Management systems. This
give the ability to do fair share of submitted tasks,
accounting of resources used, charging thereof and
user administration. All of these services currently
have no ‘off the shelf’ solution, so after extensive
testing of other possible solutions it was decided that
our own custom solution would be developed. The
main reason for this was that all currently solutions
required significant extra dependant software to be
installed on client systems. This though easy to do
on a new, cleanly installed system would present
significant barriers to inclusion to some of the
operational systems.
The operating model devised is shown in Figure 1.
The important part of the design is that it is only
loosely dependant on the grid information servers.
The total size of the Resource Broker installation is
~150Kb and has therefore fulfilled one of the key
requirements. It is also able to remotely query the
NGS information server and so include their
resources in our operational systems. This particular
point allowed for very rapid testing and user
engagement.
2.2 Virtual Organisation Management
2.1 Resource Broker
Figure 2 Front page for the Virtual Organisation
Management system
Figure 1 Functional model of the Resource Broker
The Resource Broker has been designed with several
key ideas in mind;
1) Other current RB systems are incredibly
large and their full installation and
configuration requires a huge number of
different components to be installed. The
Bristol RB should be small and loosely
coupled to other servers.
2) Only a small ‘beige box’ was available as
the hardware resource during development
so the RB had to be lightweight and small.
3) Perl has significant advantages, allowing
rapid development cycles and easy
debugging etc. This was the language of
choice as there are also several key API
available for grid tools.
In order to control user access to systems in the
UoBGrid it was necessary to use a Virtual
Organisation server. After investigation of other
possible products it was again decided that it would
be better to devise our own system. The alternatives
were decided to be too reliant on third party software
or had again many dependant components without
which they would be inoperable. This has lead to a
web-based system in which the following
functionality has been included;
a) Currently registered user/system listing.
b) New user/system addition including system ↔
user mapping.
c) User/System removal.
Finally the vom system distributes grid mapfiles
onto registered systems. This has been implemented
by adding the VOM server certificate as a registered
user onto each registered system and mapping this
onto a local limited privilege user account. The file
is then copied from the VOM server using GSI-SCP.
To increase the overall security of the system the
administrator interface is secured using certificates.
The system also presents within the details available
for each user/system their current usage statistics as
collated by the RUS.
2.3 Resource Usage Service
Here again, we preferred to develop our own
lightweight solution. The way we chose to achieve
this was to alter the installed Globus jobmanagers
and use standard installed information transfer
mechanisms. The process for keeping a Resource
Usage Record is as follows:
1) The user job is submitted as normal, the
created jobmanager creates a task ID
specific file containing the start-time for the
job.
2) The job runs as standard to completion (if it
is cancelled the start-file is removed as part
of the jobmanager shutdown procedure).
3) When the jobmanager detects a successful
completion signal from the underlying
scheduling system
the following
information is logged straight from the
jobmanager;
a. Task ID
b. Start time,
c. End time
The jobmanager then queries the
underlying system history reporting to
retrieve the following information;
d. CPU time,
e. Wall time,
f. Physical memory,
g. Total memory.
4) The jobmanager then reformats this data
and communicates using CGI to connect
with the RUS/VOM server.
5) On the server the sent information is then
saved into a relational database from which
further analysis can be done.
The summary page for jobs run since the design of
the RUS is shown in Figure 3.
repository for the software and documentation for
building the above services. This allows both CeRB
and Bristol system managers to easily obtain the
software.
Each of the installations is also fully documented
using Centre formatted individual instructional
documents. These are also available to the general eScience community through the Centre for eResearch Bristol website. The software stack is also
available by request.
3. Using the UoBGrid
The user interaction with the system is currently
very simple. The checking of the submitted job
queue and the results of latest system tests are all
accessible through a simple portal on the main eScience centre website. Examples of these are
shown in Figure 4 and Figure 5.
Figure 4 Resource broker currently running job
display
Figure 5 GITS test results
Figure 3 Resource Usage Service summary page
showing jobs per user and total per month.
2.4 Software for further dissemination
One of the roles of the Storage Resource Broker,
part of the UoBGrid service stack, is to act as a
Job submission to the grid is currently done from the
command line on the resource broker. In future we
hope to develop to produce a job-submission client
that can be automatically installed onto users
systems from a central software update system. In
addition, we plan to have a web-based submission
system for those who employ non-Linux operating
systems.
To increase system reliability we found it necessary
to make some changes in usage of software; for
example to the MDS component of the Globus
toolkit 2.4. Operationally there are now only two
systems that may directly access the GIIS server.
These are the Resource Broker (once every five
minutes) and the web interface which uses a cached
copy of the downloaded information which is
updated every ten minutes. This has allowed the
GIIS to operate at near 100% reliability for the past
3 months, excluding planned outages.
A key part of producing and maintaining a
successful production system is appropriate user
support. We are very grateful to the staff members
of the University of Bristol Information Services
Help Desk, who have been working closely with us
to enable front-line user support mechanisms.
Specific email addresses have been set up to which
all queries about the grid are passed. These are then
processed and recorded by the helpdesk staff and
subsequently passed onto the expert staff from
CeRB for resolution.
We currently have five different groups of users
from the Departments of Astro-Physics, Polymer
Physics, Geographic Sciences, Computer Science
and BioChemistry. The number of users has been
growing at approximately 1 group per two weeks as
word of mouth between active researchers finds
more and more suitable groups.
The current system layout as of 31st March 2005 is
shown Figure 6. The systems within departments in
green are currently connected and accepting jobs,
systems in red are currently being rebuilt with
appropriately upgraded operating systems and
schedulers, systems in blue are currently under
negotiation to join.
The current system records for usage are ~15000
individually submitted jobs per week with ~3500 in
a single day. The largest single job successfully run
was a 2000 child process protein-ligand docking
experiment. The latest science result produced by
the
UOBGrid
is
described
at:
http://escience.bristol.ac.uk/Science_results.htm.
[1] Virtual Data Toolkit, http://www.cs.wisc.edu/vdt
[2] Viglen Computers, http://www.viglen.co.uk
[3] The Condor Project,
http://www.cs.wisc.edu/condor
[4] The Globus Alliance, http://www.globus.org
[5] Big Brother Systems Monitoring, http://bb4.org
[6] Storage resource Broker,
http://www.sdsc.edu/srb/
[7] Glue Schema,
http://www.hicb.org/mailman/listinfo/glue-schema
Figure 6 Current layout of the UoBGrid showing systems registered