A General-Purpose Heterogeneous
Distributed Computing System
Thomas Keane B.Sc. (Hons)
Masters Thesis
M.Sc. Computer Science
Department of Computer Science
National University of Ireland Maynooth
Submitted: July 2004
Head of Department: Prof. Ronan Reilly
Supervisor: Thomas J. Naughton
Table of Contents
Chapter 1
Introduction
1.1
Aims and Motivations
11
1.2
Existing Systems Review
14
1.3
Requirements
19
1.4
Summary of Remaining Chapters
24
Chapter 2
Design Considerations
2.1
Parallel Programming Models
26
2.2
Communication Models
31
2.3
Parallel Architectures
35
Chapter 3
Design
3.1
Overall System Design
40
3.2
Server Design
41
3.2
Client Design
47
3.3
Remote Interface Design
50
3.5
Scheduling
51
Chapter 4
Implementation
4.1
Development Language
56
4.2
Communications Model Implementation
57
4.3
System Safety and Security
58
4.4
Installation and Deployment
58
4.5
Sample Distributed Computations
61
4.6
Testing and Performance
65
Chapter 5
Programming Model
5.1
Problem Suitability
72
5.2
Overview of Programming API
73
5.3
Compiling, Testing, and submitting a Problem
80
2
Chapter 6
DSEARCH
6.1
Introduction
83
6.2
Alignment Algorithms
87
6.3
DSEARCH Algorithm
94
6.4
Performance Analysis
97
Chapter 7
Conclusions
7.1
Summary of Work
102
7.2
Future Research and Development
105
References
108
Appendix A
121
Appendix B
123
3
List of Figures
Figure 2.1
Illustration of client-server model of communication
32
Figure 2.2
Illustration of peer-to-peer model of communication
33
Figure 2.3
Distributed memory MIMD parallel architecture
36
Figure 2.4
Pipeline processor with 4 stages
37
Figure 3.1
Diagram of the complete system
40
Figure 3.2
Server design split into three subsections
42
Figure 3.3
Overview of client design
49
Figure 3.4
Remote interface design
50
Figure 4.1
Illustration of the two stage distributed mergesort algorithm
64
Figure 4.2
Illustration of the stages in the character counting application
65
Figure 4.3
Imbalance in servicing values caused by addition of jobs
66
Figure 4.4
Proportion of processing time allocated to jobs
67
Figure 4.5
Processing time decrease for biomedical application
69
Figure 4.6
Processing time decrease for cryptography application
70
Figure 4.7
Processing time decrease for substring application
71
Figure 4.8
Speedup for gained three applications
71
Figure 5.1
General format of an extended DataManager constructor
73
Figure 5.2
General format of generateWorkUnit() method
73
Figure 5.3
General format of processResults() method
74
Figure 5.4
General format of the adjustGranularity () method
75
Figure 5.5
General format of getStatus() method
75
Figure 5.6
General format of closeResources() method
76
Figure 5.7
General format of the processUnit() method
77
Figure 5.8
Screenshot of job add screen on the remote interface
80
Figure 5.9
Sample screenshot of real-time feedback of a problem’s status
80
Figure 6.1
A dotplot produced by the author using Dotter
85
Figure 6.2
PAM250 scoring matrix
86
Figure 6.3
Graphical representation of 3 possible alignment options
89
Figure 6.4
First stage of Needleman-Wunsch algorithm
89
Figure 6.5
Traceback algorithm of second stage of Needlman-Wunsch
90
Figure 6.6
First stage of Smith-Waterman algorithm
91
Figure 6.7
Traceback algorithm of the second stage of Smith-Waterman
92
Figure 6.8
Illustration of the two stage alignment process
95
Figure 6.9
Illustration of overall DSEARCH alignment computation
4
96
Figure 6.10
Flowchart showing the decision process followed by DSEARCH
96
Figure 6.11
Part of a database search output
98
Figure 6.12
Decrease in processing time
99
Figure 6.13
Speedup achieved by DSEARCH
100
Figure 6.14
DSEARCH runtime increase
101
5
List of Tables
Table 1.1
Well-known existing Distributed Computing systems
14
Table 3.1
List of functionality available via the remote interface
51
Table 3.2
The methods that must be implemented by a scheduling algorithm 54
Table 4.1
List of the platforms that we have installed our software on
60
Table 4.2
Table showing the difference in completion times
68
6
Abstract
Due to the high cost of dedicated parallel and cluster machines, one is forced to look
for alternatives to fulfil the requirement for high performance computing in modern
scientific research. Distributed computing has emerged as a viable alternative to
dedicated parallel computing. By harnessing the spare clock cycles of idle machines, it
is possible to emulate the computing power offered by a specialised parallel machine at
a fraction of the cost. However there are still many limitations with existing distributed
computing platforms. To address these, we have developed a general-purpose
programmable distributed system suitable for deployment in a typical university or
corporate environment where semi-idle desktop PC’s are connected via a network. The
system is fully cross-platform compatible as it is written entirely in Java. Some of the
features of our system include a multi-problem scheduler, a remote server interface,
remote updating of client software, dynamic updating of problem granularity, guarantee
of security of the server and donor machines from subversive distributed applications,
and inbuilt compression of data. To program a distributed computation, the user is only
required to extend two Java classes. Our system has already been deployed across a
number of academic departments at NUI Maynooth on approximately 250 processors
and we have demonstrated the usefulness of our system with a bioinformatics
application.
Database searching for similar sequences is one of the fundamental tasks in
bioinformatics. The massive accumulation of genomic data in public databases has led
to increased interest in performing sensitive database searches. The two most rigorous
search algorithms are the Needleman-Wunsch and Smith-Waterman algorithms. Many
approaches to parallelising database searching have been proposed. However the
overriding problem with many of these programs is that specialised or proprietary
parallel hardware and software is often required. For most researchers, this can make
these programs either prohibitively expensive or simply too complicated to set up. In
developing DSEARCH, we have addressed this and other shortcomings of existing
parallel database search programs. For those in an academic or corporate environment
with hundreds of idle desktop machines, we have shown how DSEARCH can deliver a
‘free’ database search supercomputer.
7
Preface
When writing up my thesis I decided, as it is custom in scientific presentations to use
the scientific ‘we’, which I prefer over the personal pronoun ‘I’.
The distributed computing system and applications developed as part of this project are
available from the system webpage located at http://www.cs.may.ie/distributed
All of the software produced during this project is freely available under the GNU
general public licence.
8
Declaration
I declare that this thesis is my own work and has not been submitted in any form for
another degree or diploma at any university or other institution of tertiary education.
Information derived from the published or unpublished work of others has been
acknowledged in the text and a list of references is given.
________________________
_________________
Signature
Date
9
Acknowledgements
Firstly I would like to thank my supervisor, Tom Naughton, for all of his dedication,
help, advice, and support over the past number of years. I would especially like to thank
him for helping me greatly improve the standard of my writing and presentation skills
since I have started working with him.
I gratefully acknowledge the assistance of the technicians at the Department of
Computer Science for assisting me in this project by allowing me to run my software in
the computing laboratories and department cluster. I would like to thank various
members of the Department of Biology at NUI Maynooth for helping me acquire the
knowledge required to apply my primary area of research to the exciting field of
bioinformatics.
I would like to thank all of my family for providing me with constant support
throughout my education. Thanks to my friends, especially Richie and Andy, for their
sometimes non-sensical conversations over beers that helped me take my mind off my
work when it seemed like things weren’t going too good and thanks to Jacqui for her
support.
Last but not least, I thank the Irish Research Council for Science, Engineering and
Technology (IRCSET) for funding me throughout my research under the Embark
Initiative.
10
Chapter 1
Introduction
In recent years, the area of distributed computing has emerged as a means of harnessing
the spare computing capacity of semi-idle machines in order to simulate the computing
power of dedicated supercomputers [Buy99, CDK01]. The advent of large scale
distributed computing can be mainly attributed to the spiralling costs of specialised
parallel hardware which has meant that it has become infeasible for many organisations
that require high performance computing to purchase expensive parallel hardware. The
task of this project is to examine the current set of distributed systems and identify the
areas in which each system fails with the overall goal being to design and implement,
according to our requirements, one of the most generally usable distributed computing
system ever built. In the rest of this chapter we set out and explain the main motivations
for the project and review the current set of popular distributed computing systems.
1.1 Aims and Motivations
We can clearly identify several motivating factors that lead to the commencement of
this project. The major motivation is that modern scientific research often involves very
long computations that can often take days, weeks, or even years to complete on a
single desktop machine. Yet an examination of any modern organisation reveals that
there are thousands of desktop computers sitting idle for much of the day. A recent
study by IBM showed that Windows and UNIX servers are idle for approximately 95%
and 85% of the day respectively [Hea03] while another study showed that on average
desktop machines are idle for between 60-80% of the day [AES97]. We proposed to
develop a general-purpose ‘virtual supercomputer’ that could utilise the latent
computing power of an organisation’s machines. By giving access to previously
unattainable amounts of computing power, researchers will have the ability to perform
much more detailed experiments and simulations than previously possible. There are a
number of clearly identifiable areas of science that require large amounts of processing
power, such as meteorology, geophysics, space science, mathematical physics,
computational chemistry, computational biology, and computer science. Publications
such as [HPC02, VAP00] provide information on how parallel computing is currently
being used to solve many previously unanswered scientific questions. At the outset of
this project, we identified bioinformatics as one of the many areas of science that has a
11
pressing need for large amounts of computing power. As part of this project, we
proposed to develop at least one useful distributed bioinformatics application.
One of the main reasons for the rise in popularity of distributed computing is
the prohibitive cost of parallel hardware. In the early days of parallel computing, the
only users of parallel computing technology consisted of cutting edge researchers with
multi-million dollar budgets. In the mid-90’s, machines such as the Cray T3E [ABG97]
were the most powerful parallel machines available, with expensive custom built
interconnections between hundreds or even thousands of tightly coupled processors in a
single shared box [GKG03]. For the ordinary researcher1, these specialised parallel
machines have always been too expensive to even be considered as a research tool.
Distributed computing is based on taking existing semi-idle2 computing resources and
using these machines to their full potential. This has been exemplified by the success of
the SETI@home project [KWA01]. The SETI@home webpage points out that one of
world’s most powerful computers, IBM's ASCI White, is rated at 12 TeraFLOPs3 and
costs $110 million, while SETI@home currently gets about 15 TeraFLOPs and has cost
approximately $500k so far. Therefore one of the major advantages of distributed
computing to a researcher on a modest budget is that they are not required to purchase
any new hardware or software. Our main aim was to develop a freely available generalpurpose distributed system that could operate on any computer platform or network,
thus allowing a user to gain maximal use of their organisations computing resources.
One factor that is often overlooked when it comes to evaluating
supercomputers is the technical skills required to setup and administrate a parallel
computer. Often a parallel computer consists of a specialised piece of computer
hardware where a number of processors are connected via a dedicated internal bus to
some shared piece of memory [GKG03]. These are quite complicated machines that
often require quite advanced technical skills to setup, install, and use. It is not feasible
for a researcher to spend months or years of their time reading up on how to configure
and use such a machine. One idea that has become popular in recent years is the
concept of taking a number of old (or new) processors and connecting them together
using a high speed network, such as Myrinet [BCF95], to form a dedicated processing
cluster. This is the “pile of PC’s” or Beowulf cluster approach [GLS03]. A number of
open source operating systems [Ope04a] and management systems [HMR03, Jcl04]
have been developed to schedule and manage the processing resources of such parallel
1
A researcher that occasionally requires parallel computing facilities
Machines that are only in use for a small part of the day and whose primary purpose is not parallel computing
3
1 TeraFLOP = 1012 floating point operations per second
2
12
systems. These systems have been extremely effective in bringing supercomputing
capabilities to ordinary researchers. However these systems still require a dedicated
pool of processors that are physically close together. Furthermore, these systems often
require a full-time system administrator to maintain, upgrade, and update the system in
the long term. One of the systems mentioned above [Ope04a] even requires a user of
the system to make modifications to an existing operating system kernel. We wanted to
develop a distributed system that would hide all of these technical issues and remove
the need for a dedicated administrator of the system, thus allowing ordinary scientists to
become the users and administrators of their own distributed computing system.
Recently grid computing has started to emerge as the dominant paradigm in
wide area distributed computing [FoK03, JoF04]. The overall aim of grid computing is
to develop standardised protocols to allow users to share computing resources, storage
resources, and access to specialised devices [FoK03, JoF04]. A common analogy to
grid computing is the electrical power grid where it is hoped that accessing computing
resources will be as easy as the way in which we currently access electrical power via
electrical outlets. Grid computing technologies are currently being developed and
deployed in industry, government, and academic environments [FoK03]. Although the
possible benefits of grid computing are massive, grid computing is still in its infancy
and there are still a number of serious problems limiting the usability of current grid
computing technology. At the moment, grid computing is used to share computing
resources between supercomputers and dedicated computing clusters at different sites
such as in the TeraGrid [Ree03]. This means that current grid technology still suffers
from all of the limitations outlined in previous paragraphs such as huge expense,
technical knowledge required to setup or become part of a grid, and requirement for
dedicated administrators to maintain and upgrade the systems. As grid computing
develops and matures in the future, it will become more suitable for the non-technical
user to make full use of the great potential of grid computing.
Anyone that has attempted to implement a parallel program on a parallel
machine will be fully aware of the extreme amount of effort that can be required to
program these machines [Len00, GKS97]. Often a programmer is required to learn a
specialised parallel programming language that has been designed specifically for the
particular parallel machine. We believe that it should not be necessary for the ordinary
researcher to spend months trying to learn a complicated parallel programming
language to implement their parallel or distributed computation [SkT98, KaB88]. In
fact it should be possible for a potential parallel programmer to use a mainstream (non13
parallel) programming language and methodology to implement their distributed
computation.
In the following section, we perform a review of the most popular distributed
computing systems currently available. By completing this review, we illustrate many
of the failings of the current set of distributed computing systems and subsequently
propose a set of requirements for the most general-purpose distributed computing
system.
1.2 Existing Systems Review
The main ideas behind distributed computing are not new and several successful
distributed systems have already been developed on the basis of harnessing the idle
capacity of machines [KWA01, Wol96, LSS03, TTL03, KrV02, Ale99, Fig04, Ber04].
These systems are being used to tackle computationally intensive problems in diverse
areas of science such as searching for extraterrestrial intelligence [KWA01], finding
large prime numbers [Wol96], testing the strength of modern encryption techniques
[Dis04], running complex weather simulations [Ale99], and performing various types
of analysis and simulations on genomic data [Fig04, LSS03]. Table 1.1 shows that there
are limitations with all of the current popular systems.
Table 1.1: Summary of a number of the well-known existing Distributed Computing systems (all
accessed July 2004), OS=open source, P=programmable and general-purpose, CP=fully cross platform
(without re-compilation of source), A=currently available for download, S=Inbuilt security mechanisms
(X indicates not applicable), D=Supports loosely-coupled distributed computing
Project Name
Project Internet Address
OS
P CP A S D
BOINC
Climateprediction.net
Condor
Distributed.net
FightAIDS@Home
Folding@Home
GIMPS
Javelin/CX
JavaParty
JCluster
Models@Home
SETI@Home
http://boinc.berkeley.edu/
http://www.climateprediction.net/
http://www.cs.wisc.edu/condor/
http://www.distributed.net
http://fightaidsathome.scripps.edu/
http://www.stanford.edu/group/pandegroup/folding/
http://www.mersenne.org
http://javelin.cs.ucsb.edu/
http://www.ipd.uka.de/JavaParty/
http://vip.6to23.com/jcluster/
http://www.yasara.com/models.htm
http://setiathome.ssl.berkeley.edu/
Y
N
N
N
N
N
N
N
Y
N
N
N
Y
N
Y
N
N
N
N
Y
Y
Y
Y
N
N
N
N
N
N
N
N
Y
Y
Y
N
N
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
N
X
N
X
X
X
X
N
N
N
N
X
Y
Y
Y
Y
Y
Y
Y
Y
N
N
Y
Y
1.2.1 Single Application
Several of the existing systems are only designed with one application in mind
[KWA01, Wol96, LSS03, Ale99, Fig04]. That is, there are few distributed systems out
there that can be programmed by a user to perform arbitrary distributed computations.
14
For any distributed system to be generally usable by scientists, it must be possible to
run arbitrary distributed computations. One approach taken by two systems [TTL03,
KrV02] is to allow the user to submit arbitrary executable programs and datasets to be
run by the distributed system. Although this strategy has been successful for these
systems [TTL03, KrV02], it leads to several restrictions which limit the scalability of
the distributed applications. The main problem with this approach is that each binary
executable that is submitted will only function on certain operating system(s) or
platform(s). Most organisations have a heterogeneous set of computing resources
ranging from file servers, management servers, dedicated computing clusters, and
hundreds of standard desktop PC’s, consisting of multiple different architectures and
operating systems. In the systems mentioned previously [TTL03, KrV02], the user is
immediately limited to only being able to utilise a fraction of the overall available
computing resources of a heterogeneous distributed system.
1.2.2 Security
Security of donor (someone that offers their machine’s spare clock cycles to the
distributed system) machines becomes a major issue in a programmable distributed
system, as these systems have the ability to download and execute arbitrary code. This
was one area that several systems [TTL03, KrV02, NPR00, CaM02, HMR03, Jcl04]
did not consider a priority. The security concerns of a potential donor to the distributed
system are plainly obvious. For example, someone could write and submit a job that
would only consist of the line ‘rm –rf *’ (or ‘del *.*’) and irreparably damage
the data contained on every donor’s hard drive. It is not just the local donor machines
that may be under threat from a subversive application. One strategy that has been used
with great success by virus creators has been to use arbitrary machines to launch denialof-service attacks on networks or individual web sites [TuV03, Bru02]. Therefore it is
essential that there are security measures built into distributed computing software to
prevent subversive distributed applications from using donor machines as a starting
point of a virus attack. In one system, security of the donor machines is completely
ignored [KrV02] and in two other systems [TTL03, HMR03] it is left up to the
administrator or donor to correctly configure and monitor the distributed client software
to ensure that their machine cannot be subverted. We believe that the distributors of the
distributed computing software should have protection mechanisms explicitly built into
their software so that they are in a position to offer a guarantee of security to the donors
of machines.
15
1.2.3 Cross-Platform
From Table 1.1, it can be seen that most of the distributed systems that we reviewed
were highly operating system specific [KWA01, Wol96, LSS03, TTL03, KrV02,
Ale99, Fig04, Ber04], requiring a different version of the software for each operating
system and/or platform. In most organisations, there is a vast array of different types of
machines that are available to participate in any distributed computing system. In a
truly dynamic and heterogeneous computing environment, it is essential for a
distributed application to be able to run on any type of machine regardless of the
architecture or operating system of the machine. This important factor seems to have
been either overlooked or only partially addressed by the current set of distributed
computing systems by offering precompiled versions of their software for just a few
different platforms [KWA01, Wol96, LSS03, TTL03, Ale99, Fig04].
1.2.4 Existing Java Systems
It should be noted that we did review four fully cross platform distributed computing
systems [NPR00, CaM02, HMR03, Jcl04]. The common characteristic between all of
these cross platform systems are that they are all implemented in some platform
independent language, namely Java [Jav04a]. The Javelin project [NPR00] and its
successor, the CX project [CaM02], seek to provide a Java based infrastructure for
parallel Internet computing. Javelin provides much of the functionality required of a
general-purpose distributed computing environment such as the potential to use the idle
computing power of a heterogeneous set of machines and a relatively easy to
understand programming API. While reviewing Javelin, we noticed that the authors did
not place much emphasis on security of donor machines and the network. Indeed one of
the sample computations available from the project webpage can discover sensitive
network information about donor machines and return this to a user of the system. The
authors also note that for a user to setup any security mechanisms in Javelin, a user is
required to have an in-depth knowledge of the Java security API [Jav04b] so that they
can manually setup security in their installation of Javelin. Unlike the authors of these
other systems, we place heavy emphasis on the security threat that is posed to donor
machines by a distributed system.
We also reviewed two Java based parallel computing systems such as
JavaParty [HMR03] and JCluster [Jcl04]. We noted that both of these systems are
designed to run on dedicated tightly coupled clusters of machines. The JavaParty
16
website notes that JavaParty is “targeted towards cluster computing where network
outages and failure of single computing nodes are rare” and that there are no explicit
mechanisms built into JavaParty to handle such an event. For any truly cross platform
distributed computing to fully harness the idle clock cycles of semi-idle machines, it
must be assumed that the environment is extremely dynamic with availability to
resources changing constantly. It should also be noted that JCluster [Jcl04] is not an
open source project. JavaSpaces [FAH99] is another Java based programmable
distributed system. JavaSpaces is based on the Linda coordination language [Gel85]
and seeks to provide a user with ‘spaces’ to share information among a set of
processors, similar to a shared-memory parallel system. The major advantage of
JavaSpaces is that is has a minimal programming API that allows potential users to
quickly implement distributed programs. However one of the major obstacles to using
JavaSpaces is the complexity in getting the system up and running, as was explained in
an article in a Java technical article [Hup00], meaning that someone hoping to setup the
system is required to understand many technicalities of network communications in
Java including setting up a http server, creating an RMI4 server, initialising an RMI
lookup service, and starting a transaction manager. As was noted in section 1.1, we
wanted to develop an easy-to-use system where the user is not required to have
advanced technical skills to setup the system.
1.2.5 Commercial Systems
It should be noted that there are a number of commercially available distributed
systems [Ent04, Uni04, Par04, Gre04]. We did not review these systems because there
is very little published information publicly available thus limiting any detailed analysis
of these distributed systems. Furthermore, we are only interested in software systems
that are free for all researchers to use.
1.2.6 Condor
Condor [TTL03] is one of the longest running distributed computing projects. It has
been operational since 1986. Condor converts collections of workstations and dedicated
clusters into a high-throughput distributed computing facility. The software is primarily
aimed at the Windows and UNIX operating systems. Some of the main features of
condor include being able to handle arbitrary client failure, support for parallel
programming environments (MPI [SOH96] and PVM [Pvm04]), ability to save state
4
Remote Method Invocation is a technology that allows an object running in one Java virtual machine to invoke
methods on an object running in a different Java virtual machine [Jav04a]
17
information (referred to as ‘checkpointing’) of the system in case of power failure or
client failure, ability to run unmodified applications and the ability to allow separate
Condor pools to collaborate (referred to as ‘flocking’). Recently the developers of
Condor have released a modified version of the system (called Condor-G) that can
operate in a grid environment [FTL02]. Condor is one of a very few distributed systems
that have successfully made the transition to grid computing.
Despite the significant success of Condor, it still has a number of limitations.
Firstly, the software is not fully cross platform compatible. A different version of the
software is required for different platforms and there is no version of the software
available for some architectures and platforms because the source code for Condor is
not freely available for recompilation. Security of donor machines is not considered the
responsibility of the Condor software and it is up to the user or administrator of the
donor machine to ensure that a Condor computation cannot subvert the donor machine.
If a user incorrectly installs Condor, then the donor machine can be compromised by an
application downloaded via Condor. Every problem that is entered into Condor must
specify the operating environment that it requires [RLS98, RLS03]. If the operating
environment is not available somewhere in the Condor pool then the problem will
never get run on the system and will be forced to wait indefinitely without the user
knowing (called ‘starvation’ in scheduling terminology).
1.2.7 JDCL and Extensions
The origins of this project can be traced back to the development of a system called the
Java Distributed Computation Library (JDCL). The original JDCL system, developed
by Fritsche et al. [FPW01], was designed to provide an easy-to-use platform for
developers who want to quickly implement a distributed computation in the context of
a Single Program, Multiple Data (SPMD) architecture. The system was in an early
stage of development and did solve some of the major issues with current distributed
computing systems outlined in section 1.2.3 such as network and platform
independence and an easy-to-use programming interface. However there were a
number of severe limitations with the system such as inadequate exception handling,
lack of a user interface, no security mechanisms, limitation of only running a single
distributed application at a time, and the user being required to restart the entire system
to run a new computation.
A number of extensions were made to the original JDCL by Keane et al.
[KAN03, KAN02] that brought the system up to a level where it was possible to
18
demonstrate the completion of a relatively large scale distributed computation. Some of
the new features that were added to the system included full logging on the client and
server, coping with arbitrary client failure, ease of distribution by bundling the software
into single executable files, addition of security mechanisms on the client, and
expanding the range of suitable applications by the emulation of a MIMD parallel
architecture. However there were still a number of serious limitations to the system
which severely limited the usability and practicality of the system. The system lacked a
scheduler which meant that it was not possible to run several distributed computations
simultaneously. The server had to be completely restarted to change the distributed
computation being run on the system and there were no security mechanisms built into
the server. The granularity of the parallelisation for each computation was static and
could not be dynamically adjusted in response to changes in the set of available donor
machines throughout a computation. There was no interface to the system once the
server was started and it was not possible to update the client software remotely. It
should be noted that for the project of this thesis we did use some of the ideas and
experience gained while working on the extended JDCL. However, we did not use
either the design or any of the code from the original or extended JDCL.
1.3 Requirements
In the previous section, we completed an extensive review and evaluation of the current
most popular distributed systems and noted many of the limitations of these systems. In
this section we produce a set of requirements that combines the functionality offered by
current systems while addressing all of the limitations outlined previously. We believe
that the following set of requirements specifies a system that, if implemented, would be
the most general and usable distributed computing system currently freely available.
The requirements for this distributed system can be summarised as follows.
1. The system must ensure that any application uploaded by users cannot pose
any threat to any machine in the system
2. All communications should be initiated by the donor machines and the
distributed system should not maintain any knowledge of individual donor
machines, other than information required for network communication and
selecting suitable computations for the machine to perform
19
3. The entire system must be completely platform and network independent,
without the need to recompile source code
4. The system must be fully fault tolerant and capable of operating in an
unpredictable environment where available computing resources vary arbitrarily
over time
5. The system should not adversely impact the performance or usability of any
machine that is running any part of the system
6. It should be possible to remotely update the software running on donor
machines when new versions become available
7. It should be possible to dynamically alter the granularity of the parallelism to
suit the current set of donor machines at any point in a distributed computation
8. The system must have the ability to dynamically change the algorithm being
used on donor machines to perform the processing, without the need to restart the
software
9. It must be possible to simultaneously distribute several different individual
computations of varying computational and parallel complexity. Furthermore, it
should be possible to dynamically prioritise computations within the system so
that different problems can be allocated greater or lesser proportions of the
overall system processing power
10. It should be possible to completely control all functionality of the system
remotely
11. Users that enter problems into the system should be able to remotely view the
progress of their problem in real-time as it executes within the system, and
remotely download the results of their computations
20
12. A potential user of the system should not be required to possess any special
technical skills or knowledge of parallel computing to fully install and setup the
system
13. There should be a simple process to be followed by developers that wish to
design and implement their own distributed computations to run on the system
A quick analysis of the requirements reveals that they can be grouped into three
categories. Requirements 1 and 2 are concerned with the privacy and security of the
donors and their machines. Requirements 2 through 7 contribute to the scalability and
longevity of the distributed system. Requirements 7 through 13 are concerned with the
functionality required if this system is to be useful and widely usable. In the following
sections, we discuss the requirements listed above.
1.3.1 Privacy and Security
A quick analysis reveals that requirements 1 and 2 are designed to protect the privacy
and security of the every machine in the system. We felt that it is important for donors
to be able to offer their idle clock cycles without having to worry about any of their
personal details or personal data stored on their machine being examined and possibly
transmitted to a remote server. The only pieces of information that it should be
necessary to glean from a donor machine are the machine’s IP address (for the purposes
of network communication with the server) and some basic statistics about the
processing capacity of the machine.
Also for our system to be acceptable to users of donor machines, we felt that it
was extremely important that we could offer a guarantee that our software could not
subvert their machine in any way. We live in a time where computer viruses routinely
cause havoc on personal computers, therefore the first question on a donor’s mind is –
“what damage could this application cause to my machine?” The owners of donor
machines should have the confidence to install distributed computing software without
having to worry about manually setting any security permissions or restrictions on the
software in order to protect their machine. Therefore it is essential that any developer of
a distributed system design their system with donor machine security as one of the most
basic requirements.
21
1.3.2 Scalability and Longevity
Requirement 3 is essential for any large scale distributed system that hopes to operate
in a heterogeneous environment. The basis for distributed computing is to harness the
idle capacity of arbitrary machines regardless of their architecture and operating
system. It was noted in chapter 1 that this has not yet been achieved by most of the
current set of distributed computing systems, as most systems are written and compiled
in a platform specific language. In many large organisations, it is usual to have a
heterogeneous set of computing resources including various types of servers, dedicated
computing clusters, parallel machines, and ordinary desktop machines. Typically all of
these machines run different operating systems and this is often one of the main reasons
that a distributed computing system cannot exploit the computing resources of an
organisation to the absolute maximum. Some systems do offer their source code for
download and compilation by users so that they can run the distributed program on
their particular platform. However we believe that this approach is not feasible because
it is unreasonable to expect a potential, possibly non-technical, user to have the
necessary knowledge and tools to recompile a distributed system.
Requirement 4 states that the system should be capable of operating in an
unpredictable environment where resources can arbitrarily fail. Therefore the system
must have fault tolerance mechanisms to handle arbitrary donor machine failure,
network failure, any errors or exceptions in running distributed computations, and
exceptions in the system software. It must be assumed that any machine can fail or
completely disconnect from the system at any time, therefore the system must take the
necessary measures to ensure that this variability does not impact on any of the
computations running in the distributed system.
The basis of all of the systems outlined in chapter 1 is that they operate
seamlessly on donor machines using only the spare clock cycles. Requirement 5 seeks
to take this to its logical conclusion, meaning that our system should not affect any user
working at any machine running any part of our system. Requirement 6 is really a
feature of the system that we felt would be very useful while the software is still in
development. Once the software has begun to be deployed to users by electronic means,
and given requirement 2, we decided that the ability to remotely update the software is
the only effective way of keeping the software on the donor machines up to date as
newer versions of the software become available.
22
1.3.3 Essential Functionality
Requirement 7 is a unique feature to our system. We believed that in a truly
heterogeneous operating environment where the overall set of clients is constantly
changing, it becomes necessary for the system to have the ability to dynamically adapt
the size of work units (parallel granularity) that are being sent out to clients.
Requirement 8 is one of the most basic requirements for any general-purpose
distributed system. As mentioned in chapter 1, many of the current systems are limited
by the fact that they are only designed for one application and are not generalisable or
programmable. Often the only way to change the algorithm being executed on each
dataset is to change or upgrade the version of the software. We believe that a generalpurpose distributed system must allow one to dynamically change the algorithm that the
donor machines are using to process work units that are sent from the server. By
‘dynamically’ we mean change the algorithm without the need to restart the software
on donor machines. Requirement 9 is largely concerned with the scalability of the
system. If the system is to be always available to the users, it must have the ability to
run several distributed computations simultaneously. This leads to the need to prioritise
the amount of processing time being allocated to jobs running in the system. Some
computations may take long periods of time to complete while others may take a much
shorter time to complete. Priority could be based on importance of the particular user
that submitted the computation, the size of the computation, and so on.
1.3.4 Desirable Functionality
Requirements 10 and 11 are desirable features of any distributed system that relate to
the functionality and practicality of the system. It may not always be possible for the all
users of the system to be physically at the location where the server machine is located.
Therefore the administrator and users of the system must have the ability to control and
operate the server remotely (via the network). Also users of the system should be able
to view the status and get the results of their computation from their own desktop
machine without the need to contact the administrator of the system.
Requirement 12 is something that is missing from most of the current set of
distributed computing systems. We wanted to develop a system that could be installed
and setup by anyone that possesses the most basic computer skills. For many of the
distributed systems mentioned in chapter 1, a user is required to possess certain
specialised technical skills in order to be able to successfully setup the distributed
system. Any distributed system that meets both requirement 3 and 12 should allow any
23
user to easily install and setup the system on whatever operating system and platform
they are already familiar with.
Requirement 13 refers to the process that is followed by developers that wish
to implement their own distributed computations to run on the system. A number of
different approaches are taken to allow potential users of a distributed system to design
and implement their own distributed computations [SkT98, Leo00]. The common
characteristic in all of these schemes is that there is some well-defined process that
potential users should follow so that they can develop their own distributed
computations (requirement 13).
1.4 Summary of Remaining Chapters
The rest of this thesis is organised as follows. In chapter 2, we describe the background
to the high level design decisions that we made once our requirements were complete.
We state each high level design decision taken and explain the reasons behind each
decision.
Chapter 3 contains a detailed description of the lower level system design. The
system consists of three independent software applications. The design of each
application and the system’s scheduling algorithm interface are described in detail. We
also outline the scheduling algorithm that we designed for the system.
Chapter 4 gives details on how the design was implemented. We dedicate a
section to explaining the reasons behind our choice of programming language. We
explain how we implemented the communications model outlined in chapters 2 and 3.
We reaffirm the security principles behind the entire system. Finally, we give details of
how we deployed and tested the system in our university environment.
Chapter 5 outlines the programming model that should be followed by
potential users of the system. This chapter also includes details on how to identify
problems that would gain the greatest benefit from the system and how to compile, test,
and submit problems to our system.
Chapter 6 describes the design and implementation of our distributed
bioinformatics application. We implemented a program that distributes the task of
searching large genomic databases using the most rigorous search algorithms. This
application showcases all of the functionality available in our system.
Chapter 7 concludes the thesis with a summary of the achievements of this
project and outlines a few of the areas of the project that, we believe, warrant further
24
research and development. We also identify a number of specific areas which we see as
most suitable for a distributed implementation on our system.
25
Chapter 2
Design Considerations
In this chapter, we describe the high-level models that we investigated when
commencing the design of our system. At the highest level of a distributed system is the
parallel programming model that a developer uses to design and implement their
distributed applications. We describe and evaluate the main parallel programming
models. Next we evaluate the suitability of the two main communication models in
distributed computing systems. Finally we state the parallel architecture that our system
is based on and give a proof showing how our system is equivalent to this architecture.
2.1 Parallel Programming Models
Parallel programming models exist at the highest level of a distributed computing
system as an abstraction above the parallel hardware and operating systems. All of the
current models can be subdivided into four categories: the shared memory model, the
threads model, the message passing model, and the data parallel model. It should be
noted that a parallel programming model is independent of the underlying parallel
architecture of a system. We completed a review of these models as follows and then
took a decision on what model would be the most suitable programming model to
implement in our system. For more extensive information on each of these models see
[GKG03, DFF02].
2.1.1 Shared Memory Model
The shared memory model is based on the concept of single piece of memory that
allows shared access between multiple processors or processes5. In a distributed
computing environment, the shared memory model extends the concept of a single
piece of memory among a set of loosely coupled processors, their cache, and secondary
storage. In the shared memory model every processor has direct access to the memory
of every other processor in the system. This means a processor can directly load or
store any shared memory address. The programmer can also declare certain pieces of
memory to be private to a particular processor with various types of locks and
semaphores used for synchronising access to variables. This provides a very simple yet
5
A processor is a physical computing unit. A process is an individual program being executed by the computer
[PeS02].
26
powerful model to the programmer for expressing and managing parallelism in an
application.
Advantages
• Conceptually easy to understand and hence design programs for
• Easy to identify opportunities for parallelism
Disadvantages
• Lack of portability as this model is often implemented in an architecture
specific programming language
• May not be suitable for loosely coupled distributed processors due to the high
communication cost
Most of the parallel systems that implement the shared memory programming model
consist of a physical shared memory architecture also. This is due to the high
communication cost that is traditionally associated with shared memory programs.
However there are exceptions to this such as the KSR ALLCACHE [RSW93] which
provided a shared memory view even though the physical memory of the machines was
distributed among a set of processors.
2.1.2 Threads Model
In the threads model, a single process can have multiple separate paths of execution
running on one or more processors. In a serial program, threads are commonly used to
perform operations that may take a long time to complete such as certain I/O or
synchronisation operations. In a parallel program, threads can be created at locations in
the program that involve repetitive operations, e.g. a loop. Each thread can be sent to a
different processor to be run to completion and the results are returned to the
controlling program.
Advantages
• The overhead associated with creating a thread is much less than creating an
entire process
• Switching between threads requires much less work by the operating system
• Most programmers are familiar with writing multi-threaded programs because
threads are a basic construct in many modern programming languages
27
Disadvantages
• Writing a multi-threaded program requires much more thought than for other
programming models
• Synchronisation mechanisms are required to control access to shared variables
Java [Jav04a] is the most well known programming language that allows programmers
to implement multi-threaded programs. A number of parallel systems based on the Java
thread model have been developed [HMR03, Jcl04]. Other parallel programming
languages that implement the thread model are POSIX [Wal95] and openMP [Ope04b].
2.1.3 Message Passing Model
Message passing is the most popular parallel programming model. This is largely due
to the fact that message passing programs can run on any type of parallel system from
distributed or shared-memory multi-processors, networks of workstations, or even
single processor systems [Kra03, Sap97, TGL99, MiG97, ABD95]. Message passing
provides the programmer with mechanisms to exchange information between separate
co-operating processes running on either the same or different machines. A
programmer usually implements message passing through a special set of libraries or
method calls, provided by the programming language, that are inserted into their code.
Message passing is a powerful technique as it gives the programmer a level of
parallelism control that is not possible with other models. In recent years a number of
standards have emerged for message passing programs [SOH96, GBD95].
Advantages
• This model is applicable to both tightly coupled supercomputers and
geographically distributed systems
• Message passing libraries provide a set of functionality and level of control
that is not found in any of the other models
• All other parallel programming models can be implemented by the message
passing model
Disadvantages
• All of the responsibility for an effective parallelism scheme is placed on the
programmer. The programmer must explicitly implement a data distribution
28
scheme and all interprocess communication and synchronization, while
avoiding deadlock and race conditions
• Some parallel programmers prefer to have this level of control however it can
be difficult for novice programmers to implement effective parallel programs
2.1.4 Data Parallel Model
The idea behind the data parallel programming model is the provision, by the
programming language, of parallel constructs and built-in procedures that operate on
large volumes of data seamlessly. Typically the programmer writes a single sequential
program, inserting special language constructs such as parallel loops, which indicate to
the compiler where the program can be parallelised. Many of the data parallel
languages are array or vector based so by writing a command such as a=cos(b), where
a and b are both arrays, a whole array of cosines can be computed in one line of code.
The aim of this model is to process large volumes of data by distributing the data over a
number of processors. Usually all message passing is performed implicitly without any
intervention by the programmer and this model is commonly built on top of another
message passing scheme.
Advantages
• Gives the user the ability to process large volumes of data very fast
• Only one piece of code needs to be produced to implement all of the
parallelism
Disadvantages
• Due to the large volumes of data involved in a typical data parallel
computation, this model may not be suitable for geographically distributed
processors
• Requires high bandwidth communications to transfer and share data
A number of popular programming languages have been extended to include a number
constructs and declarations specific to this model, e.g. High Performance Fortran (HPF)
[Fox91] and Fortran F90.
29
2.1.5 Programming Models Analysis
After completing the above review of the most popular programming models currently
in use in parallel programming, we decided to base our programming model on the
message passing model. We can identify several reasons for this decision as follows.
Firstly we wanted our system to implement a general programming model that would
not impose any restrictions on developers, thus allowing for the development a wide
range of distributed computations to run on our system. It has been widely
acknowledged that the success of the message passing model is due to its generality,
and not simplicity. This is proven by the fact that the message passing model has been
implemented on every type of parallel computer from distributed and shared memory
systems, tightly coupled processor clusters, and geographically distributed computing
systems [Kra03, Sap97, TGL99, MG97, ABD95].
It was noted in chapter 1 that one of our original aims was to develop one of
the most general-purpose distributed computing systems ever built. If we are to
implement such a system, then we should allow the submission of any type of parallel
computation. The message passing model gives the developer a level of control of both
algorithmic and data parallelism that is not found in the other models, thus allowing the
programmer to express whatever level of data or algorithmic parallelism is required for
their particular distributed application.
Requirement 4 states that the system should not adversely impact the
performance or usability of any machine that is running any part of the system. This
requirement also encompasses the donor machines access to the communication
network. We decided that we had to select the programming model that had the
potential to result in the least amount of network traffic. It has been shown in studies
that the message passing model can result in less network communication for certain
applications than the other models [LDC95, KJA93], thus making it the most suitable
model to implement in our system.
2.2 Communication Models
Each of the successful distributed computing systems that we reviewed has been
designed around one of two communication models of computing namely the peer-topeer model [Ora01] and the client-server model [Ren96]. Early in the project, we
performed an evaluation of each model and used this information when deciding on
which model to base our system design on.
30
2.2.1 Client-Server Model
The client-server model is a concept for describing communications between
computing processes that are classified as service consumers (clients) and service
providers (servers) [Ren96]. Clients and servers are functional modules with welldefined interfaces. The functions performed by a client and a server can be
implemented by a set of software modules, hardware components, or a combination
thereof. Each client-server relationship is established between two functional modules
when one module (client) initiates a service request and the other (server) processes the
request, and chooses to respond to the service request (see Fig. 2.1). Information
exchange between clients and servers is strictly through messages meaning that no
information is exchanged through global variables or shared memory. The service
request and additional information is placed into a message that is sent to the server.
The server's response is similarly another message that is sent back to the client.
Typically clients and servers reside on separate machines connected through a network.
Advantages
• Security - Since the clients can only contact the server it is difficult for anyone to
compromise the network or system
• Network Maintenance and Support - In the client-server model it is not too
important if a number of the client machines fail. The only machine that must be
maintained is the server
• Bandwidth - The client-server model uses very little bandwidth. The clients
connect to the server to get some information and then disconnect while they
process it
Server
Request
Response
Request
Client
Response
Client
Fig. 2.1: Illustration of the client-server model of communication
Disadvantages
31
• Robustness - If any individual client machine on the network becomes
unavailable there is little or no consequence for the overall system. However, loss
of the server can be catastrophic
• Scalability - In a client-server environment the addition of thousands of clients
reduces the ability of servers to the point where the system will collapse unless
more server bandwidth or power is added
We prove the generality of the client-server model in section 2.3 by showing that it is
possible to emulate the most complex parallel architectures using the client-server
model. However it should also be noted that the client-server is more suited to certain
types of problems. There are two main criteria for identifying the suitability of a
problem to the client-server model, namely coarse-grained parallelism and compute-todata ratio. Coarse grained parallelism refers to the way in which a single large problem
can be easily split up into discrete independent sub-blocks that can be processed
individually. This is an important characteristic for the client-server as the server is the
only channel of communication between the two client machines. If a problem requires
a large amount of communication between sub-blocks of the problem, then this can
overload the server and decrease system efficiency. A high compute-to-data ratio is
also important for the same reason, as large amounts of communication between clients
and the server will either overload the single server machine or increase communication
times by overloading the communication channel to the server. Although these problem
characteristics may place some restriction on the number of problems suited to clientserver distributed computing, these characteristics are evident in problems from many
different areas of science [KWA01, Wol96, LSS03, Ale99, Fig04, Ber04].
2.2.2 Peer-to-Peer Model
The peer-to-peer model was first implemented when two computers were connected
together in such a way that each could gain access to the other's resources [Ora01]. It
differentiates itself from other distributed computing models by the principle that
participating computers have equal status and resources are distributed rather than
centralised (see Fig. 2.2). Each node in the system can act as a client and/or server. The
major difference between the peer-to-peer model and the client-server model is that the
system does not rely on an individual central resource for its operation.
32
Advantages
• Robustness - If any individual machine on a peer-to-peer network becomes
unavailable there is little or no consequence for the operation of the overall
network
• Scalability - As machines are added, the processing power and disk space
available to the system increases
• Self-Optimising - Resources get distributed around the network, as demand
requires
Client/Server
R
Q
Q
Q
R
Client/Server
Client/Server
R
Fig. 2.2: Illustration of the peer-to-peer model of communication showing that requests can originate
from any node; Q: Request, R: Response
• Data Redundancy - Resources are unlikely to be lost or damaged as they can be
replicated all over the network, and the system remains available even when
individual components become unavailable
Disadvantages
• Security - With no central point of access or management it is difficult to prevent
penetration of the network
• Version control - Proliferation of files and applications, which can be updated
without central management, makes it very difficult to maintain version
consistency
• Network Maintenance and Support - The peer-to-peer model has certain selfmanaging properties but these are not perfect or sufficient and when problems
occur in a chaotic model, it is very difficult to track down causes and resolve
them
• Bandwidth - The 'conversational' nature of peer-to-peer networks mean that
machines must talk to each other regularly in order to establish where resources
can be located and who is active. This means that each machine may have to
handle massive volumes of data passing on its way to other machines on the
network. Traditionally communication is implemented by a broadcast message
33
system. This can lead to serious reductions in network performance at times when
a large number of machines are communicating simultaneously
The peer-to-peer model of communication emulates the traditional communication
models of parallel or cluster computers where each node of the system can
communicate directly with any other node of the system [ScB90]. The types of
problems that benefit from this communication system are problems where it may not
be possible to break the overall problem into independent sub-blocks or where the subblocks require regular communication with each other to produce the overall result. A
good example of this is the distributed neural network training environment - Joone
[Mar04]. Joone is built on the Jini distributed computing technology [Wal99] and
implements the peer-to-peer model of communication. As there is no single controlling
machine which must process all communication between nodes, a peer-to-peer system
can often scale to much larger numbers of processors than a strict client-server system.
This peer-to-peer model has recently been used to great success by file-sharing
networks [Jus04].
2.2.3 Communications Model Analysis
We decided to design our distributed system around the client-server model. We can
identify a number of reasons for this choice. Requirement 4 states that the entire system
should not adversely impact the performance or usability of any machine in the system.
If we implemented the peer-to-peer model, our non-dedicated standard network would
be flooded with messages and performance would be reduced [Mar02, SGD02]. The
client-server model typically produces less network traffic as all communications are
directed towards one single machine [Ren96]. Of course, adopting the client-server
architecture does mean that our system is not ideally suited to peer-to-peer parallel
algorithms. This is discussed in more detail in chapter 5.
The client-server model is one of the oldest and most established models of
network computing. Requirements 12 and 13 state that it should be relatively easy for
potential users of the system to set-up and install the system. We believed that it would
be much easier for potential users to install our system and understand how it works if
our system implemented the most established model of network computing. Also, since
the client-server model is such a well-known communications model, it would be easier
for potential developers to quickly design and implement their own distributed
34
computations as they wouldn’t be required to learn a new communications paradigm
first.
The client-server model is the most popular communication model among the
current set of successful distributed computing projects [KrV02, Ale99, Fig04, Ber04]
and is implemented by two popular general-purpose distributed systems [TTL03,
Ber04]. At the start of this project, we had no predefined algorithms that we wished to
run on our system and our main criteria for choosing a model was to select the model
that would allow us to get the greatest performance from our architecture. As the clientserver model has proven to be the most successful communications model in wide area
distributed computing, we also decided to follow this model.
2.3 Parallel Architectures
The most well-known and commonly referenced parallel architecture classification is
Flynn’s taxonomy [Fly66, Fly72, SkT98]. Flynn classifies all computers according to
two dimensions: instruction and data. An instruction stream is defined a set of
instructions to be executed on a particular data stream. Single instruction, single data
(SISD) is the simplest form of computing where one instruction is being executed by
one data stream on a single CPU at any time. Single instruction, multiple data executes
(SIMD) a single instruction stream on multiple datasets. The earliest parallel
computers, such as array or vector processors [OeW93, Hil86], were SIMD machines.
Multiple instruction, single data (MISD) executes several different algorithms in
parallel on a single dataset. This form of parallelism is exploited by most modern
microprocessors in the form of a pipeline processor where a single data stream is
processed in stages in a pipeline [KGO97]. Multiple instruction, multiple data stream is
the most general and prevalent forms of parallel computing in recent times. A typical
MIMD parallel architecture consists of an asynchronous set of processors where every
processor in the system can execute a different instruction stream on different data
streams.
CPU Mem
network
CPU Mem
CPU Mem
Fig. 2.3: Distributed memory MIMD parallel architecture
35
As there is a huge variety of machines that can be classes as MIMD parallel
architectures, it has been suggested by some authors [Joh88, Hwa87] that the MIMD
classification is too general and should be further subdivided according to memory
organisation. In a shared memory system, the processors share a common memory and
are connected via a common bus. Often there are two separate buses, one for accessing
the shared memory and one for communication between processors. Various types of
bus strategies have been investigated but the main drawback of these systems is that
contention on the bus limits their scalability beyond a few processors. In a distributed
memory system (see Fig. 2.3), the processors all have their own local memory and
communicate via a dedicated or non-dedicated network. A distributed memory system
does not suffer the memory contention issues of a shared memory system, however the
increased physical distance between processors means that communication latency
between processors is increased, especially where the processors are connected via a
non-dedicated network, such as in geographically distributed computing systems.
2.3.1 MIMD Emulation
As MIMD is the most general and prevalent form of parallel computing, it must be
possible for developers to implement parallel problems to run on the system that
require MIMD capabilities. Although the client-server model is a SIMD architecture, it
is possible to emulate a MIMD parallel architecture over a client-server based system.
In conjunction with other authors [KAN02, KAN03], we justify our choice of the
client-server communications model by showing formally how the client-server model
can be augmented with a pipeline processor in order to emulate a MIMD parallel
architecture. Consider an input X and a computation, C(X), on that input that returns
some result r. We could say that
r = C( X ) .
(2.1)
In client-server computing, the server partitions the input data into n segments
n−1
X = ∑ xi
i =0
(2.2)
such that the following transform
xi → C ( xi ) = ri
(2.3)
can be applied to each data segment xi. Each segment is sent to one of a set of clients,
which performs the transformation, and returns the corresponding ri. The server will
have the ability to reconstruct the original result by combining these partial results
36
n −1
r = C ( X ) = U C ( xi ),
i =0
(2.4)
where U denotes an appropriate combination operation. This is the starting assumption
of work related to SPMD (single program, multiple data) computation through
functional programming [LHF00], where single program refers to an asynchronous
version of SIMD that executes the same program on n processors independantly.
In a pipeline processor, a problem is divided into a series of tasks that have to
be completed sequentially one after another. Each of these tasks is called a pipeline
stage and is executed by a separate process or processor. The output of the previous
stage forms the input to the subsequent stage with multiple stages operating in parallel
(see Fig. 2.4).
Input
Data
Stage 0
Stage 1
Stage 2
Stage 3
Output
Data
Fig. 2.4: Pipeline processor with 4 stages. Output of stage n forms the input to stage n+1
In pipeline processing, the ability exists to decompose a computation into m smaller
transformations that each act on the result of the previous transformation,
r = C ( X ) = cm −1 (cm − 2 (Lc1 (c0 ( X ))L)) ,
(2.5)
where X is the input. A recursive definition of this concept could be written as follows,
 c0 ( X ) if j = 0 ;
rj = 
c j (rj −1 ) if j > 0 ,
(2.6)
where r = rm −1 can be regarded as the seed to the recursion and defines the final result.
The first clause in Eq. 2.6 is the terminating condition (passing the input to the first
transformation) and the second clause describes how the result of any one
transformation depends on the preceding transformation. We use the following compact
notation to represent the recursive definition of r,
m −1
r = C( X ) = ∏ c j ( X ) ,
j =0
(2.7)
where Π denotes the operation to appropriately deal with intermediate results. Equation
2.7 describes passing the complete input X to transformation c0(), and passing the result
37
to transformation c1(), and so on. In pipeline processing, the ability also exists to
partition the input into n segments as described in Eq. 2.2. In this case, each of the n
segments could be passed in turn through all m transformations and the partial results
combined as in Eq. 2.4. Equation 2.7 could then be written as
n −1  m −1

r = C ( X ) = U  ∏ c j ( xi )  .
i =0  j =0

(2.8)
The advantages of this include the ability to arbitrarily change the granularity of the
data throughput (some transformations may have restrictions on the size or
representation of their arguments) and also to permit parallelisation of the computation.
The computation depicted in Eq. 2.9 could be described by rewriting Eq. 2.8 as
m+n−2
i
i =0
j =0
r = C ( X ) = U ∏ c j (xi − j ) ,
(2.9)
where processor pj is responsible for transformation cj().
Finally it is possible to combine both the client-server and pipeline models to
allow our clients to effect arbitrary transforms rather that all performing the same cj().
In this case, the server divides the computation as well as the data. It distributes to the
clients a description of a transformation cj() as well as a data segment xi. Since the
partitioning shown in Eq. 2.2 is possible, there will not be any interdependencies
between different parts of the data stream. Equation 2.7 could be rewritten as
m −1 n −1
r = C ( X ) = ∏ U c j ( xi ) ,
j =0 i =0
(2.10)
which describes transforming all of the data segments with cj() before applying cj+1(),
and so on. Since Eq. 2.9 and 2.10 describe the same computation, this shows that the
order in which each cj(xi) is effected is unimportant. This computational theory
provides no indication of how to find the two appropriate combination operations, Π
and U , for arbitrary problems. However it does provide us with a proof that shows by
augmenting the server with a pipeline processor, it is possible to fully emulate the
operations of a MIMD parallel architecture.
38
Chapter 3
Design
In this chapter, we describe the design of the distributed system. We start by giving an
overall description of the system design. The design is then broken down into three
main parts, namely the server, client, and remote interface. We devote a section to
describing how different scheduling algorithms can be easily integrated into our design.
Finally, we describe our scheduling algorithm that was designed with another member
of our research group, Andrew Page, and has been recently published in [PKN04].
3.1 Overall System Design
Our system, based on the client-server model, is divided into three separate pieces of
software: server, client, and remote interface. An overview of the system is illustrated
in Fig. 3.1. The server software stores the problem (problem data and an algorithm to
process it) and breaks it down into smaller sub-problems, called work units. The client
software is installed on each donor machine and connects to the server over the
Internet. The client requests a work unit, performs the processing, returns the result to
the server, and requests another work unit. Multiple clients can make such requests to
the server. The server collates the results of the smaller problems from the clients and
constructs the result to the larger, original, problem. The remote interface is used to
access all functionality on the server such as adding problems, removing problems,
getting results of past problems, monitoring the progress of running problems,
monitoring and configuring the server.
The following subsections give detailed descriptions of the design of the
server, client, and remote interface. We took a modular approach when designing each
application and completed full UML style class diagrams illustrating the relationships
between each module. Each module has well-defined functionality and interfaces with
other modules in the system. There are several clear advantages to modular software
[Pre04]. Each module can be isolated from the overall system, which means it can be
independently developed and tested. As the number of users of a system grows, a
modular design allows for easy addition of new features or functionality to the system.
A modular approach to designing a system means that individual modules can easily be
39
Server
Data
Computation 1
Computation 2
Computation 3
Computation 4
Computationn 5
Client
Internet/Intranet
Client
Remote
Interface
Client
Client
Client
Fig. 3.1: Diagram of the complete system. Although all communication is bi-directional, the arrows
indicate the direction of initiation of communication.
upgraded or changed. Once a module has been fully tested and proven to be reliable,
there is also the possibility for its use in other software development projects.
3.2 Server Design
When we began designing the server, we identified three distinct aspects to our server
application (see Fig. 3.2). Firstly there are the server initialisation routines contained in
the core server module. The job of the core server modules (labelled 1 in Fig. 3.2) are
to start the server, serve requests for work from donor machines in the system, manage
the current set of work units being processed, and implement a logging scheme to track
system events. The second aspect to the server is the scheduler (labelled 2 in Fig. 3.2).
The main responsibility of the scheduler is to implement a scheduling algorithm to
manage and allocate the overall processing power available in the distributed system
between the set of problems running in the system. Other responsibilities of the
scheduler would include all interaction with the remote interface which includes the
management of all problem files (including compression of problem result files before
transmission to the remote interface and deletion by an administrator). The final section
40
of the server corresponds to the set of distributed computations that are running in the
server at any time (labelled 3 in Fig. 3.2). Each problem must be individually managed
and separated from the rest of the server (due to security issues mentioned in chapter 2).
The sections that follow describe in some detail the functionality that is provided by
each individual module in the server.
FastDiskDataStore
1
ClientCommunications
GUICommunications
2
PendingStore
SchedulingAlgorithm
ExponentialSmoother
ExpiredStore
Server
Scheduler
Zipper
TimerThread
Deleter
FileServer
Compressor
Logger
Problem
SocketCommunicator
SystemLog
3
ErrorLog
Logger
ProblemLog
Association (‘has a’)
ProblemLoader
DataManager
ProblemObjectInputStream
Generalisation (‘is a’)
Fig. 3.2: Server design split into three subsections where the arrow connector indicates inheritance and
the rectangle connector indicated aggregation. Overlap indicates shared modules.
3.2.1 Server Module
One of the main responsibilities of the Server module is to read in and parse the
server start-up parameters (received via the server.ini file) and initialise all of the
other modules of the server. There are a number of parameters that must be passed to
the server upon start-up. These parameters include the server machine’s IP address and
ports that the server will use for receiving incoming client connections. It is possible in
most programming languages to dynamically discover a machines IP address, however
if a user is running the server on a multi-homed machine (a machine with multiple
network cards) the user may wish to specify the most public or private network card
(by specifying its IP address) that the server will listen on. The other parameters
contained in the parameter file are the server timeout, the administrator password for
41
the system, and the maximum size that the log files can grow to before they are
overwritten. Typically these parameters can all be left at the default values.
Requirement 4 (see chapter 1) of our system is that the system’s operating
environment is unpredictable and donor machines can fail arbitrarily. This variability
was managed in the original and extended JDCL by a fault tolerance scheme based on
time-outs and replication of data [FPW01, KAN03, KAN02]. Since this scheme proved
successful in the JDCL, we choose to implement a similar fault tolerance strategy in
this system. Each time a work unit is generated by a problem in the system, a copy of
this unit is placed in the PendingStore. If a client does not either return the results
of the unit or request a time extension within this time limit, then the server assumes
that the client has been terminated and places the work unit on the expired list
(ExpiredStore). Expired units are given priority in the system and are immediately
resent out to clients to be processed.
The ClientCommunications module defines the communications
interface with the client software. The Server module receives all client
communication requests. Typically these requests include requests for work units,
receiving results of processed work units, requests for time extensions to complete
work units, requests for newer versions of the client software, requests for particular
algorithms to process work units and error reports from a fault in the client software or
in a downloaded algorithm. Many of these requests relate to individual problems
running in the system and are communicated to the individual problem via the
Scheduler. If a newer version of the software is uploaded to the server, then this
must be sent out to every client in the system. Every time a client requests a work unit,
the server includes the version ID of the current client software in its response. If this
ID is different to the version of the client software running on the donor machine, then
the client will request a copy of the newer client software and restart. If any error or
exception occurs in any part of the client software, then details of the exception are
reported back to the Server. This feature was very useful while the client was still in
the development and testing stage as we could get real-time error reports from
executing clients. As the system loads external user code into both the server and client
(see chapter 5), it is necessary to catch all errors or exceptions that can occur while this
code is running. Therefore all external code is monitored while it is running on the
server and client. If an exception occurs while a user’s code is running on a donor
machine, this exception is fed back to the server and the problem is removed from the
42
system. Details of the error are recorded in the problem log files that are returned to the
user with the problem’s result set.
The Server also maintains certain statistics on the system’s status such as
how many units are currently being processed, how many individual donor machines
have connected since the last timeout, the total number of donor machines that have
connected since the server was started, the number of pending units, the number of
expired units, and the number of problems running in the system. This information is
fed back, as requested, to the remote interface (via the Scheduler).
3.2.2 FileServer and SocketCommunicator Modules
Certain types of data-intensive problems may require large data files to be made
available to every client (e.g. geocomputation [OpA00], weather simulation [Jou04] or
particle physics [Reg00] applications) in the system. A more specific example would be
in bioinformatics where a biologist wishes to search a large genomic database (such as
Genbank [BKL00] or Swiss-Prot [BaA00]) for similar sequences (see chapter 6). We
decided that the most efficient way to transfer these large data files would be to stream
them directly from the server machine’s disk via TCP sockets [TTL03], thus avoiding
ever reading any of these files entirely into the server’s memory. The FileServer,
as its name suggests, acts as a separate server whose only purpose is to receive
connections for uploading and downloading large data files from the remote interface
and to the clients. The FileServer runs as a separate low priority thread on the
server so that the transfer of these files does not adversely impact on the essential
functions of the server and creates a SocketCommunicator per connection to
perform the communication.
3.2.3 PendingStore and ExpiredStore
As mentioned in section 3.2.1, the server must maintain a copy of all work units that
are currently being processed by donor machines. Pending units are work units that
have been issued to a donor machine to be processed but have not returned a result yet.
Each pending unit is given a timeout and it must report back with a status message or
set of results within the time limit. If no status message or results set is received within
the time limit, then it is assumed that the client has been terminated and the unit is
moved to the ExpiredStore, and is subsequently re-issued to the next available
client to be processed.
43
As there may be thousands of clients connecting to the server, it would not be
feasible to store every data unit in the server machine’s memory. Therefore we decided
that copies of the pending and expired units must be stored on disk. Each problem
running in the system is assigned a unique ID when it is entered into the system. In
order to maximise disk space and speed up lookups and searches of the work units, the
units are compressed and arranged in a directory structure according to their problem
ID. As there is a lot of shared functionality between the PendingStore and
ExpiredStore, therefore both of these modules extend from a common module
called FastDiskDataStore. The Server module controls the movement of work
units between the PendingStore
and ExpiredStore and checks the
PendingStore for expired units periodically via the TimerThread module.
3.2.4 Timer Module
As mentioned in previous subsection, the Server must periodically check for expired
work units. The TimerThread is a low priority thread that notifies the server every n
minutes (where n is the timeout period) to check the pending list for expired data units.
The time period can be updated dynamically via the remote interface.
3.2.5 Log Files
Full log files are maintained by the server. There are two distinct types of log file. The
SystemLog records general system events as they happen in real-time, while the
ErrorLog records any errors that occur in the entire system (including any errors that
occur remotely on clients). Each problem that is running in the system is assigned a
working directory on the server. Included in this directory is a set of problem log files
(ProblemLog) which are used by the system to record certain events (errors,
granularity updates, and average unit processing time) while the problem is running in
the system. A copy of these log files is returned with every set of results.
As it is envisaged that the server application may run for a number of years
without being restarted, there is a limit on the maximum size of all log files (set via the
server initialisation file). When this size is reached, the server overwrites the oldest log
file entries.
3.2.6 Scheduler Module
The main task of the Scheduler is to implement the scheduling algorithm to allocate
processing time appropriately among the set of problems running in the system. All
44
requests for work units from clients are passed to the Scheduler and it decides
which problem to request a work unit from, according to the scheduling algorithm.
When designing the scheduler, we found that it would be useful to provide a
generalised interface (SchedulingAlgorithm interface) that would allow other
developers to easily develop and test other scheduling algorithms. We implemented a
scheduling algorithm based partly on an exponential smoothing function (see section
3.5 for a detailed description).
All communication with the remote interface is performed by the
Scheduler and is specified by the GUICommunications interface. It was decided
that it would be more efficient for the remote interface to communicate directly with
the Scheduler rather than the Server because the majority of communications
from the remote interface relate to operations on the current set of problems running in
the system, such as adding new problems, removing problems, getting the status of
problems, and downloading results sets (see Table 3.1 for an exhaustive list). The
Deleter and Zipper are low priority threads that are created by the Scheduler as
needed to perform deletion and compression of problem result files when a problem is
finished.
3.2.6 Compressor Module
As our distributed system is designed to run in a heterogeneous environment such as
the Internet where network communication speeds can vary greatly, we decided that
one of the most essential features of the system is the compression of all data before
transmission. The Compressor takes any data type as input, compresses the data, and
returns the compressed data as a sequence of bytes (and vice versa). The Compressor
module is used by both the Scheduler and the Problem modules. The
Scheduler uses the Compressor to compress and decompress all communications
between the remote interface and the server, while the Problem module uses the
Compressor to compress all work units as they are generated by a problem and
decompress all results received from clients. The choice of compression algorithm used
by the entire system can easily be changed by modifying the Compressor module
(default is GZIP [Deu96]).
3.2.7 Problem Module
Each problem that is running in the system is represented by an instance of the
Problem module. A Problem module instance keeps all of the state information of a
45
particular problem that is running in the system such as the problem ID, problem
priority, number of units issued, number of results received, number of exceptions that
have occurred, and the computational requirements of the problem. The computational
requirements are defined as the minimum memory required by a donor machine to
process the problem’s work units and are specified by the user when entering their
problem into the system. A Problem is also made up of several components: a
ProblemLoader,
a
DataManager,
a
Compressor,
and
a
ProblemObjectInputStream. Each problem in the system has its own individual
ProblemLoader used for dynamically loading problem specific code and libraries
necessary to execute the problem. This clear separation between the problems running
in the server means that different problems cannot gain access to the server code or to
other problems’ code. The DataManager module is one of the two user defined
classes that are extended in order to implement a distributed computation (see chapter
5). The ProblemObjectInputStream is used to read in data types that are
specific to a problem and require the server to use problem specific libraries to interpret
this data. All problem specific libraries must be provided by the user when entering
their problem into the system.
3.3 Client Design
The main purpose of the client software is to continually request and process work units
downloaded from the server. As our system is based on the client-server model, all
communications are initiated by the client software. This was a design decision on our
part and is not default as a result of adopting the client-server paradigm. The client is
made up of a number of modules as follows in sections 3.3.1 through 3.3.4. An
overview of the design of the client is shown in Fig. 3.3.
3.3.1 Client Stub
The ClientStub module is responsible for initially starting the client, which
includes checking the input parameters (e.g. the server IP and port information), setting
up the client security mechanisms, and starting the main client software. Every time a
new version of the software is received, the ClientStub is responsible for restarting
the software using the new version. If any unhandled exception occurs, the
ClientStub is responsible for recording the exception and resetting the client
application.
46
3.3.2 ClientController
After the client is started by the ClientStub, the ClientController module
takes over control of the client software. The ClientController essentially
consists of an unbounded loop that continually requests, processes, and returns results
of work units. If at any time the client cannot contact the server or the server has no
more work units to be processed, the client will go into a ‘sleep mode’ (consuming
minimal donor machine resources) and attempt to connect to the server periodically.
The sleep time for the client is set via one of the command line parameters. The
ClientController starts by connecting to the server and requesting a work unit.
The ClientController uses the Compressor to uncompress a downloaded
work unit and then determines what Algorithm is required to process the work unit.
The
client
maintains
a
cache
of
all
downloaded
algorithms
(via
the
AlgorithmLoader). If the client does not already have a copy of the required
algorithm, then it sends a message to the server requesting a copy of the algorithm. The
downloaded algorithm is loaded into the client using the AlgorithmLoader
module. The client starts a TimerThread and proceeds to process the work unit using
a separate AlgorithmMonitorThread module. If the client is not finished
processing the work unit within the time limit specified by the server, the
TimerThread will signal the ClientController to contact the server and
request a time extension. If the server denies the extension request, then the
AlgorithmMontiorThread
terminates
the
work
unit
and
the
ClientController requests another work unit to process. There are safeguard
mechanisms built into the client for a situation where an Algorithm with an infinite
loop is inadvertently added to the system and subsequently is distributed to clients. If a
any unit does not finish running after a specific period of time (set system-wide to 8
hours by default), the client terminates the unit and reports an exception to the server
causing the problem to be completely removed from the system.
47
ClientStub
Association (‘has a’)
AlgorithmLoader
ClientController
Compressor
AlgorithmMonitorThread
Deleter
TimerThread
Algorithm
ProblemObjectInputStream
ClientCommunications
Fig. 3.3: Overview of the client design
3.3.3 Algorithm Loader
As noted in the previous subsection, the client maintains a cache of all previously
downloaded Algorithms. If the client does not have a copy of a particular algorithm
that is required to process a work unit, then it will request a copy from the server. Every
downloaded Algorithm is dynamically loaded into the client using the
AlgorithmLoader. The main tasks of the AlgorithmLoader are to check that
the downloaded Algorithm is valid, dynamically load the Algorithm, and place a
copy of the Algorithm into a cache of downloaded algorithms. It should also be
noted that some problems may have third party libraries or packages that are needed by
the Algorithm. These packages must also be loaded using the AlgorithmLoader
module.
3.3.4 AlgorithmMonitorThread
The job of this module is to create, start and monitor the Algorithm thread. This
module starts the Algorithm and passes the downloaded work unit to the
Algorithm. If an exception is generated by a running Algorithm, then this module
collects information about the exception and passes it to the ClientController.
When the Algorithm is finished processing a work unit, the results are collected by
this module and passed to the ClientController before being sent to the server.
48
3.4 Remote Interface Design
The remote interface is the only means of accessing functionality on the server. It
provides users and administrators of the system with an easy-to-use graphical user
interface (GUI). The remote interface is also treated as a client in our system so all
communications with the server are initiated by the remote interface. An overview of
the remote interface design is shown in Fig. 3.4.
GUICommunications
Association (‘has a’)
JobConfigure
ServerControl
MultiFilter
ServerStatistics
RemoteInterface
JobAdd
Filter
JobRemove
ConnectToServer
JobView
Compressor
AboutPanel
JobsView
JobResults
Fig. 3.4: Remote Interface Design
3.4.1 Remote Interface Module
The remote interface is started by the RemoteInterface module. This module sets
up the initial GUI frame and server connection screen. Every time a user requests a
certain piece of functionality, the module corresponding to the piece of functionality is
created (see Fig. 3.4). Each module creates its own specific screen within the remote
interface GUI.
The user connects to the server via the ConnectToServer module. This
uses the GUICommunications module to communicate with the server and connect
in either administrator mode or user mode. The Compressor module is used to
decompress all information received from the server. All of the other modules whose
names begin with Server or Job correspond to different functionality that is
available using the remote interface. In administrator mode, all of the functionality of
the tool is available to the user. Administrator mode is accessed through a password
49
that is encrypted and checked against the encrypted administrator password stored on
the server. A complete list of the functionality available is included in Table 3.1.
Menu
File
File
Job
Job
Job
Job
Job
Job
Server
Server
Server
Table 3.1: List of functionality available via remote interface
Function
Mode+ Explanation
Connect
U
Connect to server
Exit
U
Exit the application
View
U
View the progress of a particular job in the system
View All
A
View progress of all jobs in the system
Add
A
Add a job to the system
Remove
A
Remove a job from system
Configure
A
Change priority of job
Get Results
U
Download results of completed job
Statistics
A
Display some statistics on current state of server
Control
Change server timeout, pause/restart server,
A
delete past problem files
Update Client A
Upload newer version of client software
+
A: Administrator mode only, U: User and administrator mode
Limited functionality is available in user mode. No password is required and in order to
access any functionality in this mode a specific problem identification number (issued
when a problem is entered into the system) must be known. A problem identification
number is typically a pseudorandom eight digit number thus making it difficult for a
user to access someone else’s problem on the system.
3.5 Scheduling
One of the main features of our system is that it has the ability to run several distributed
computations simultaneously. Therefore one of the most important tasks of the server is
to fairly allocate the available computing power among the current set of distributed
computations. The system is designed to operate in an unpredictable environment that
consists of heterogeneous computing resources. Therefore, the performance that can be
delivered varies dynamically as users with competing goals share resources, new
resources become available, resources fail, and are upgraded. One of the most
important aspects of a successful scheduling algorithm is to take the constantly
changing environment into account. Our scheduling algorithm can be broken down into
two distinct parts. The first aspect of scheduling in our system is the dynamic
adaptation of each problem’s granularity (amount of processing per work unit) to
changes in the set of donor machines. This idea was inspired by the congestion control
algorithm that is employed by TCP [SMM98, Pos81, Tan03] where the sender
dynamically decides how many bytes may be sent to the receiver during a
communication. The second part is the efficient allocation and management of the
50
available system processing power offered by the set of donor machines. In section
3.5.2, we describe our general scheduling interface that provides an easy-to-use
mechanism for other developers of the system to design, implement, and test alternative
scheduling algorithms. Section 3.5.3 gives details of the scheduling algorithm that we
have implemented in the system.
3.5.1 Dynamic Unit Size
Even though users specify the minimum requirements for their problem when they
enter their problem into the system via the remote interface, it is not possible to have a
priori knowledge of the capacity of the network and the exact processing power of the
available donor machines. Therefore, the appropriate granularity (how much work to
give each donor machine) of the parallel computation will have to be ascertained
dynamically. With too fine a parallelism, clients will return results after a very short
processing time and might overload the network. Too coarse a parallelism might result
in some processors being left idle and could cause large amounts of processing time to
be wasted if a donor machine is unexpectedly switched off. Therefore, we implemented
the exponentially weighted moving average (EWMA) function [Rob59] to monitor
each problem’s average unit processing time. The EWMA value is updated with the
processing time of every result set that is received. The EWMA function (defined in
Eq. 3.1) is a very popular scheme used to produce a smoothed time series by assigning
exponentially decreasing weights to observations as they get older. The EWMA
function we implemented to monitor the average processing time is
E = αp + (1 − α ) E
t
t − 1
t
(3.1)
where Et − 1 is the previous EWMA average processing time, Et is the updated EWMA
average processing time, pt ≥ 0 is the current unit processing time, and α ∈ [0,1] is the
smoothing coefficient. The choice of α parameter determines the influence that new
values have on the EWMA. A value of α = 1 means that only the most recent
measurement influences the EWMA, while a smaller non-zero value gives more weight
to older values. As our system is expected to operate in a dynamic environment, we
wanted the EWMA value to only respond to gradual and sustained trends in the
operating environment so we choose our default value for the α parameter to be 0.2.
If the EWMA average unit processing time, Et , is not sufficiently close to
some target time, t, then an attempt is made to alter the granularity of the problem for
subsequent units. The percentage of Et (represented by d) by which the granularity
needs to be altered is calculated from
51
 t − E 
t − E 

*100 : if | 

 *100 | > v
d =  E 
 E 

0
: otherwise
t
t
t
t
(3.2)
where t is the target time and v is the percentage variance threshold below which the
processing times are allowed to fluctuate. The default values for v and t are 15% and
one hour, respectively, and can be changed dynamically using the remote interface. A
positive value for d indicates that the problem’s granularity should be increased and a
negative value means that the problem’s granularity should be decreased. The value of
d is sent to the problem code, submitted by the user, as only the problem code can take
the appropriate action to alter the granularity of subsequent units. Choosing a higher
(respectively, lower) value for α will allow the system to adapt more (respectively,
less) quickly to changes in the network, but will cause it to be more (respectively, less)
likely to overreact in the presence of transient fluctuations in network capacity and
client processor power. To allow a change in granularity to reflect in the value of Et ,
the system waits a period of 2t hours before making a subsequent request.
3.5.2 Scheduling Algorithm Interface
One of the most important tasks for a distributed system capable of running several
problems simultaneously is the efficient allocation and management of the available
processing power. Many different scheduling algorithms for multiprogrammed systems
have been proposed [SiG99, Gal99, CiJ01, PKN04, HSL03]. To aid further research
and development into alternative scheduling strategies, we have provided a generalised
interface that allows for the design, implementation, and testing of alternative
scheduling algorithms. This interface is defined in the SchedulingAlgorithm
module in Fig. 3.2 and consists of five methods (see Table 3.2) that a developer of a
scheduling algorithm must implement.
Table 3.2: The methods that must be implemented by a scheduling algorithm
Method
Purpose
nextUnitProblemID()
Which problem to select the next work unit from, returning its ID
update()
Pass the runtime (seconds) and problem ID of last result set received
addProblem()
A new problem has been added to the system, passes its ID
removeProblem()
A problem has been removed from the system, passes its ID
updatePriority()
The priority of a particular problem has been changed, passes its ID
52
3.5.3 Servicing Scheduling Algorithm
In our scheduling algorithm (ExponentialSmoother module in Fig. 3.2), we
compute a servicing value for each problem every time a result set is received by the
server [PKN04]. The servicing value (defined in Eq. 3.3) is a measure of how much
processing time each problem has received relative the problem’s priority in the system
and is equal computed from
si = ( pi / P) − (ti / T )
(3.3)
where si is the servicing value for problem i, pi ≥ 0 is the priority of problem i,
n
P = ∑ pi is the summation of the priorities of all the problems in the system, ti ≥ 0 is
i =1
n
total processing time received by problem i, and T = ∑ ti is the total system processing
i =1
time of all the problems currently in the system where n is the total number of problems
in the system. As the distributed system is expected to operate in a dynamic
environment which may result in transient fluctuations in the servicing values of
problems, we use the exponentially weighted moving average (EWMA) function
[Rob59] to update a smoothed average of the servicing value so that changes in the
servicing values are gradual. The EWMA servicing value is calculated from
E = αs + (1 − α ) E
t
t − 1
t
(3.4)
where Et − 1 is the previous EWMA servicing value, Et is the updated EWMA
servicing value, st is the current computed servicing value (see Eq. 3.3), and α ∈ [0,1]
is the smoothing coefficient. Our default value for the α parameter is 0.2.
53
Chapter 4
Implementation
This chapter gives details of how we implemented our design described in chapter 3.
We state our programming language of choice and give reasons for choosing this
particular language. We give details on how we implemented all communications that
take place within the system. We also devote an entire section to discussing issues
related to the safety and security of our system. Finally there are some details of the
testing that we performed after the system was completed. In this section, we also
describe several sample distributed applications (of varying complexity) that were
developed to run on the system.
4.1 Development Language
The entire system is implemented in Java [Jav04a]. The main reason for choosing Java
is that it is platform and network independent. Java programs are compiled to an
architecture-neutral byte-code format. Therefore a Java application can run on any
system, as long as that system implements the Java Virtual Machine (JVM). This
portability is extremely important for any distributed system because it is expected that
the clients will be running simultaneously at multiple sites on different platforms. Only
one version of our software has to be produced rather than a different executable for
each platform. Our system can run on any platform that supports a JVM with absolutely
no modifications to the source code required. Java also comes with an extensive set of
classes, arranged in packages, which are all readily available to the programmer. Java is
a dynamic language as any Java class can be loaded into a running JVM at any time.
In the early days of Java (mid-late 90’s), much of its criticism stemmed from
the fact that Java performed poorly compared to native languages such as C and
Fortran. However much has changed since then. In recent years, there have been huge
improvements in the performance of the JVM, mainly attributed to the introduction of
the just-in-time compiler and hotspot technology [Arm98, Sun01]. These improvements
have led to the performance of modern JVM’s being comparable to that of other native
languages [BSP01]. One of the most important aspects of Java is the security model
[Oak01, Jav04b]. This comprehensive security model greatly simplifies the task of
enforcing a strict security policy in a Java application. This assurance of security means
that Java applications can download and dynamically load code without having to
54
worry about the possible security implications. All of the features mentioned above
made Java the ideal language to implement our distributed system.
4.2 Communications Model Implementation
As our system is based on the client-server model, all communication with the server is
initiated by clients (where both the client and remote interface software are considered
to be clients). When implementing our communications model, we noted that in certain
problems it may be necessary to have certain problem specific data available at both the
server and clients. For example, in bioinformatics computations it may be necessary to
have a copy of a particular database or genome available to all of the clients. Therefore
we implemented our communications model using a combination of Java Remote
Method Invocation (RMI) [Sun02] and ordinary Java sockets.
RMI is a built-in facility in Java that allows one to interact with objects that
are actually running in Java virtual machines on remote hosts over a network. RMI
reduces much of the complexity involved in designing and implementing a specific
communications protocol. The user is only required to define one interface class, which
fully specifies the entire communications protocol. RMI is fully supported in the
standard distribution of the JVM so no external software installation, other than the
standard JVM, is required. In our system, all control communications are short and
consist of only a few bytes, therefore RMI provided the simplest solution for
transferring these messages. However the one major drawback of RMI (typical of all
“remote procedure call”-like systems) that we identified is that all of the data being
transmitted must be entirely stored in the memory of the two machines during the
transfer.
For certain types of problems, it may be necessary to make some relatively
large data files available to some or all of the clients. In most situations, it is not
feasible to store very large data files in memory, so we decided to use ordinary Java
sockets to stream large problem data files directly to and from the server’s disk, thus
removing the necessitation of bringing the large files completely into memory. The
remote interface and client upload and request large files via the FileServer and
SocketCommunicator server modules (see Fig. 3.2).
4.3 System Safety and Security
The main principle behind large scale distributed computing is that the owners of
machines offer the use of the spare clock cycles of their machine. The general
55
mechanism for doing this is to ask the owner of the machine to run an agent or client
application on their machine. In our experience, one of the first questions asked by
owners of potential donor machines often relates to the security of the distributed
application. The owner of the machine must be confident that the application cannot do
any harm to their machine before they will even consider running the application. We
believe that the developers of any distributed computing system must be in a position to
offer a cast-iron guarantee that their distributed application cannot pose any threat to a
donor machine.
From the start of this project, machine security has been one of our major
priorities. We have developed our server and client software in such a way that any
distributed problem that is entered into the distributed system cannot subvert any part
of the system software or any machine that is running any part of our software. There is
a comprehensive security manager [Oak01, Jav04b] in operation throughout the server
and client software. Each problem loaded into the server has its own separate class
loader from which to work within, thus preventing problems from interfering with each
other’s code and the server and client code. If a running problem attempts to perform an
operation on the client or server that could pose a threat to the security of the system or
of a machine, the operation is logged and the problem is removed from the system.
Furthermore, all of our software is digitally signed meaning that if there is an attempt to
alter or replace any part of our software, the application will fail to run.
4.4 Installation and Deployment
Requirement 12 states that users of the system should not be required to possess any
special technical skills or knowledge to set-up and install the distributed system. We
believe that we have achieved this aim by developing this system. The entire system
consists of three executable Java JAR files corresponding to the server, client, and
remote interface. The user is only required to enter the server machine’s IP address into
a parameter file before running the server with the standard ‘java’ command. The client
application can be run directly from the command line with all necessary start-up
parameters passed in as arguments (server IP, ports, etc.). The remote interface is a
stand-alone GUI application that is started without any command line parameters; the
user connects to the server using a simple ‘point-and-click’ interface. There is a full
instructions document outlining how to setup and run the software available from the
system webpage [Gen04].
56
4.4.1 Client Deployment Options
There are a few different ways that the client software can be deployed. We identified
three deployment options for anyone seeking to deploy our software:
1. The client software can be distributed to donors via electronic means (ftp, http,
email, network share, etc.) or physically distributed to donors (DVD, CD,
floppy) with some form of script or batch file to automate starting the software.
The major problem with this approach is that users may not be willing (or may
simply forget) to run the software, which could lead to a situation where the
distributed system may not receive enough computing cycles to perform any
large scale distributed computations.
2. The second option for deploying the client software is to install the software as
a low priority background service on donor machines. For most modern
operating systems, it is possible to have programs set up to run as background
services. This means that even if there is nobody logged on at the machine the
client software can be processing 24 hours a day in the background with no
interaction with the screen. There are obvious issues such as needing
administrator rights to set this up but in the long run this is an effective means
for gaining maximal usage of existing computing resources. The most practical
way to deploy the software in this way is via a ghost image (a clone disk image
used by system administrators to setup the operating system and applications on
large numbers of PC’s simultaneously).
3. The final way we identified to deploy the software is to schedule the software to
run at off-peak times on donor machines. This type of deployment is more
suitable for a situation where the owners of donor machines are sensitive to
having their machines being used for distributed computing while they are
themselves using the machines. All modern operating systems provide facilities
to schedule tasks to run periodically and in most organisations, desktop PC’s are
only in use for approximately 8 hours per day. The client software could be
scheduled to run on every desktop machine for the other two thirds of the day.
57
4.4.2 Our Deployment
To maximise the usage of the semi-idle desktop PC’s in the deployment at NUI
Maynooth, we chose to run the client as a low priority background service. This means
that even if there is nobody logged on at a donor machine, the client software can run in
the background 24 hours a day using only the spare clock cycles. At our peak, we have
our client software installed across a number of academic departments running on
approximately 250 desktop PC’s (various hardware specifications from Pentium II’s up
to Pentium IV’s) running multiple operating systems (Windows 98/NT/2000/XP, Sun
Solaris, Mac OS9, Linux – Gentoo, Debian, Redhat). Due to reasons out of our control
such as powering down of machines during low usage periods, this number is typically
reduced to approximately one hundred. To illustrate the portability of our system, we
have also installed our client on every node of an IBM Linux cluster (32 Dual Pentium
IV 1 GHz nodes with between 256 and 768 MB of memory per node) with the desktops
and cluster nodes connecting to a single server. With our current deployment of the
system, we have the ability to perform approximately 20 GigaFLOPS of processing
(obtained by running Linpack [Don88] on each machine and assuming 100% parallel
efficiency). Table 4.1 provides a complete summary of the operating systems and
hardware’s that we have installed our system on at different points during the project.
We have provided full details of how to install the client software as a background
service on the system webpage [Gen04].
Table 4.1: Summary of the operating systems and hardwares that we have installed our distributed
system software on during the project
Operating System
Hardware
Windows 95/98/NT/2000/XP Intel Pentium III/4
Windows 95/98/NT/2000/XP AMD Athlon
Sun Solaris
Sparc
Redhat Linux 7
Intel Pentium II/III/4
Redhat Fedora Core 1/2
Intel Pentium II/III/4/Centrino
Debian Linux
OS9
Intel Pentium II/III/4
HP PA-RISC
Apple
4.5 Sample Distributed Computations
While designing and developing the distributed system, we also designed and
implemented a number of sample distributed problems that we used to test various
58
aspects of the system. Our main criterion for selecting sample problems was that the
solution to each problem had to be easily verifiable so we could verify the results
produced by the system. In the following sub sections, we give a description of each
application and in section 4.6, we illustrate some performance data gathered while
running these applications. It should be noted that each of these applications was
written according to our programming model outlined in chapter 5. We have divided
the applications into two categories – SIMD and MIMD applications. The full source
code for each of these applications is available for download from the system webpage
[Gen04].
4.5.1 SIMD Applications
The first applications that we developed to run on the distributed system were SIMD
applications where the server generates multiple data sets to be processed, and where
each client uses the same algorithm to process the data and returns the results to the
server. SIMD applications typically can be executed with the least amount of parallel
overhead as there is no communication between clients required and the sole task of the
server is to break up the problem, issue the work units to be processed by the donor
machines, and collate the results. Using the notation introduced in chapter 2, these
applications can be described by
r = C( X )
(4.1)
where a single transformation C is applied to an entire dataset X to produce a result r. In
a SIMD computation, the server breaks the data up into discrete segments xi and
executes C on each segment on a donor machine to produce ri. The overall result to the
computations is produced by combining the results using a combine operator ( U ) on
the server,
n −1
r = C ( X ) = U C ( xi ),
i =0
.
(4.2)
Distributed Sorting
Sorting is one of the most fundamental tasks performed by a computer. Although
sorting groups of numbers can take anywhere between O(n log n) or n2 time to sort
(depending on the choice of sorting algorithm) [Knu98], the results can always be
verified in n time. The first sorting application we implemented was a simple
distributed sorting program (similar to those in [BDH84, FPW01]) that generates
groups of random numbers (xi), sends these to donor machines to be sorted using
59
bubblesort [Knu98] (C), and returns the sorted numbers to the server where the
concatenated partially sorted list is printed to a file ( U ). As this distributed application
uses bubblesort to perform all sorting, its complexity is O(n2) time. This is a relatively
simple application and the full source code of this application is also available as an
example in [Kea03].
Finding Repeated Substrings
The second application that we implemented is an extended version of a distributed
bioinformatics application that we have previously published while working on the
extensions to the JDCL [KAN02, KAN03]. The original application took a large string
of characters and searched for repeated substrings (above a given length) within the
input string in the forward direction. We extended this application so that it could
search for repeated substrings in both forward and reverse directions and treat the
sequence as being circular. In this computation, the main task of the server is to issue
groups of substrings (xi) of the input string (in the form of an index and length) and
send these to clients to be compared to the entire string for repeats. The client is sent
the algorithm (C), a copy of the string, and a group of substrings (in the form of an
index and length of the original string). Details of the repetitions are returned to the
server where the server prints them to a file ( U ). The time complexity of this
application is O(n4). Each time we ran this application on our system, we used the same
DNA sequence [FAE02] as was used in the previous publications [KAN02, KAN03].
This meant that we could verify the results of each test by comparing the output file
with the output that was produced by a different distributed system [KAN02, KAN03].
Cryptographic Application
In conjunction with another member of our research group, Andrew Page, and a
member of the Crypto research group at the Department of Computer Science, NUI
Maynooth (http://crypto.cs.may.ie/), we have developed a distributed application that
attempts to break the widely used RSA encryption algorithm [MVV96] using PollardRho factorisation methods [Pol75]. It has been noted that while the theoretical
complexity of integer factorisation is unknown, we do know that it takes a long time to
factorise large numbers [Bre99]. The complexity of the Pollard-Rho factorisation
algorithm is O ( f 1 / 2 .(log n ) 2 ) for finding a non-trivial factor f of a composite integer N
[Bre99]. Our parallel implementation is a SIMD application (similar to those outlined
in [Bre99, ElB95]) where the server sends out the public key (X) and the number of
60
attempts the client should make to find the private key using Pollard-Rho factorisation
(C). Each client reports back whether or not they find the private key. The program can
test arbitrary key sizes and we have used the program to test and break keys ranging
from 20-bits to 64-bits.
4.5.2 MIMD Applications
To test the MIMD capabilities of the system, we also wrote a number of applications
that assume a MIMD parallel architecture. Each problem takes a number of datasets (xi)
and consists of a number of discrete stages where each stage executes a different
algorithm (cj), thus emulating a MIMD parallel architecture. Results from intermediate
stages are passed to subsequent stages with the final result of the overall computation
being produced when all of the data has passed through all stages of the computation.
Using the notation introduced in chapter 2, a MIMD application can be described by
Eq. 2.10, where m different algorithms are executed on n different datasets, with
intermediate results being passed from one stage to the next by combining the results
from each stage using the appropriate combination operation.
Further to the MIMD applications described below, we have also implemented
a bioinformatics database search application that utilises the full MIMD capabilities of
the system (see chapter 6).
Distributed Mergesort
Along with the simple sorting application outlined above, we also implemented a
distributed version of merge sort [Bro72, Akl85]. The aim of developing this
application was to illustrate how it is possible to implement a computation that consists
of a number of distinct stages where the donor machines execute a different algorithm
at each stage of the overall computation. Our implementation of mergesort [Bro72] is a
two stage algorithm where the first stage sends out blocks of numbers to be sorted by
the donor machines and the second stage sends out the sorted blocks of numbers to be
merged into a single sorted group of numbers (see Fig. 4.1). The time complexity of
mergesort is O(n log n) [Knu98]. The server is responsible for initially breaking the
unsorted data up into discrete blocks, collating the sorted blocks, issuing the sorted
groups of numbers to be merged together by a donor machine.
61
Server
Client
Read-in and
split unsorted
file
Unsorted numbers
Sort numbers
Stage 1
Merge groups
Stage 2
Sorted numbers
Issue sorted
groups
Sorted groups
Sorted group
Output sorted
numbers
Fig. 4.1: Illustration of the two stage distributed mergesort application
Character Occurrence Counting
The second MIMD application that we implemented is an application that takes any
text file and recursively counts the number of occurrences of each character, prints the
output, removes every occurrence of the highest occurring character from the file, and
repeats the process until the file is empty. To our knowledge, this parallel application
has not been published anywhere else and the purpose for implementing this
application was simply to demonstrate that it is possible to implement a computation
where the number of computational stages varies for different datasets and is not
predetermined by the programmer (see Fig. 4.2). The time complexity of this
application is O(n3).
Server
Client
File
Read-in File
Compute
alphabet
Stage 1
Count
characters
Stage 2
Alphabet
Issue characters
from alphabet
File + character
Number of occurrences of character
Record number
of occurrences
and remove
highest occurring
character
No
Is the file
empty?
Yes
Terminate
Fig. 4.2: Illustration of the distributed character counting application
62
4.6 Testing and Performance
During the development phase of this project, we performed two different forms of
testing on our software. As illustrated in chapter 3, each of the three applications in our
distributed system was designed in a modular way (see Figs. 3.2, 3.3, and 3.4). This
design approach meant that it was relatively easy to individually test each module
during the development phase of this project. One of the most difficult aspects of a
modular system to test is the interaction between the individual components of the
system. Our approach to testing this aspect of the system, although not ideal, was to
execute the distributed applications described in the sections 4.5.1 and 4.5.2 and verify
the correctness of the results produced by the system.
We used the applications outlined in the previous section to perform tests to
verify the effectiveness of our scheduling algorithm described in section 3.5.3. We also
used the applications to test the parallel performance of the system by completing full
processing time and speedup graphs based on running these applications on our
distributed system.
4.6.2 Scheduling Algorithm Performance
In chapter 3, we outlined the scheduling algorithm that we implemented in the
distributed system. To verify the effectiveness of this algorithm, we performed three
tests that were designed to illustrate different properties of the algorithm. The first test
that we performed was designed to show how the scheduler adapts the servicing values
by allocating more (or less) processing time to problems. To demonstrate this
functionality, we modified the scheduler so that every time a result set was received it
recorded the sum, S, of the absolute values of every problem’s servicing value from
n −1
S = ∑ | si |
(4.6)
0
where n is the number of problems running in the system, and si is the servicing value
for problem i. A value of zero for S is the optimal value and indicates that the scheduler
is balanced (all problems have received an amount of processing proportional to their
priority). Using a set of 30 donor machines (Pentium III 600Mhz with 128Mbytes of
memory per machine) and the substring matching program outlined previously, we
completed a graph showing how the scheduler adapts to new problems being
introduced into the system (see Fig. 4.3).
63
0.45
New job added
0.4
Imbalance in servicing values
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0
500
1000
1500
2000 2500 3000
Units Processed
3500
4000
4500
Fig. 4.3: A graph illustrating the imbalance in servicing values caused by the addition of three jobs
Initially we added two equal priority jobs simultaneously to the server and over a
number of hours we separately added three more equal priority computations to the
system. The two leftmost peaks in Fig. 4.3 indicate the initial imbalance that is
introduced into the system by the addition of the initial two jobs. The three arrows in
Fig. 4.3 indicate the resulting imbalance in the servicing values caused by the addition
of the three new jobs. After the last job is added, it can be seen that the system proceeds
to become balanced by allocating the appropriate amount of processing time to the
job(s) that caused the scheduling imbalance.
The next test we performed was designed to illustrate how the scheduler
responds to dynamic updates of the priority of jobs in the system by allocating to jobs
more or less processing time according to the new job priorities. For this test, we used
the same set of 30 donor machines and initially started the server with 4 equal priority
substring matching jobs (labelled A through D in Fig. 4.4). After approximately 1000
results had been received, we altered the priority of the four jobs to the ratio 8:4:2:1
(for jobs A:B:C:D). It can be clearly seen that the scheduler immediately proceeds to
allocate relative amounts of processing time according to the new priorities of the jobs.
The rate at which this transition takes place can be controlled by varying the α
parameter of the EMWA function outlined in section 3.5.3. A higher α parameter will
result in shorter transition time while a lower α value will result in a more gradual
transition. For this test, we set the α parameter to 0.2.
64
Percent of total Processing Time per problem
100
90
80
~8
70
Job A
60
50
40
Job B
~4
Job C
~2
30
20
10
Job D
0
0
1000
2000
3000
4000
Units Processed (Timesteps)
~1
5000
Fig. 4.4: A graph showing the proportion of total processing allocated to jobs A through D. At each
timestep, we normalised the vertical axis values according to the amount of total processing available.
The area of each section in the graph denotes the amount of processing allocated to each job
Using the same set of 30 machines, we performed a test designed to investigate how
close the scheduler allocates processing time to computations according to the relative
priorities of the computations. By adding a loop to the substring matching application
outlined in section 4.5.1, we created three different computations where the total
amount of processing time required by each computation was exactly in the ratio 4:2:1.
We added each of these computations simultaneously to an already running distributed
system that had another separate job being executed also. We added the three jobs to
the system with priorities 4, 2, and 1, respectively, and we noted the finish time of each
computation. If the scheduler is effective, then we would expect all of computations to
complete at approximately the same time, as each job was allocated a priority
equivalent to their computational requirements. We also wanted to investigate what
effect different α parameters in our EWMA scheduling algorithm (see section 3.5.3)
would have on the relative completion times. We ran this experiment three times with
different α parameters and the results of these experiments are shown in Table 4.2.
By examining Table 4.2, we can conclude that our scheduling algorithm is
effective because there is less than 4% difference between the completion times of the
computations. It is also interesting to note that as the value of α is increased, the
deviation between the completion times of the jobs decreases. This can be explained by
the fact that a higher α parameter will cause the servicing value for each problem (see
chapter 3) to adapt more quickly when results are received from clients and hence the
65
distributed system will move quicker to address the resulting imbalance in the
problem’s servicing values. It should be noted that this particular test is only valid in a
homogeneous environment where all processors can provide the same amount of
processing power.
Table 4.2: Table showing the total runtime (RT) in minutes for each job for different values of α . We
also give the percentage difference of the running times compared to the total running time for each
value of α
α
RT Job 1
RT Job 2
RT Job 4
% Difference
0.01
1862
1905
2055
3.3%
0.2
1844
1899
1949
1.8%
0.6
1878
1916
1925
0.8%
4.6.3 Parallel Performance
A number of standard measures have emerged in parallel computing for measuring the
performance of parallel programs [KaF90]. Running time measures the amount of time
from when a parallel program is started to when the program produces the final result
of the computation. Speedup, s(n), is the ratio between the running times using one
processor and n processors. It measures the performance improvement gained through
parallelisation and is calculated from
s (n) = t (1) / t (n) ,
(4.5)
where t(1) is the running time of the program using a single processor, and t(n) is the
running time of the program using n processors. The maximum theoretical speedup
(called linear speedup) occurs when there is a speedup of n achieved using n
processors. The theoretical maximum is rarely achieved because parallel programs
incur a penalty due to certain amount of communication and management overhead. It
should also be noted that the maximum speedup achievable depends greatly on the
degree of parallelism in a particular algorithm [Amd67] and not all algorithms can be
parallelised completely. Scalability is the ability to maintain performance levels as the
workload increases by adding more system capacity (adding more processors and/or
computations running simultaneously).
66
4
14
x 10
12
Time (sec)
10
8
6
4
2
0
5
10
15
20
25
30
35
40
Number of processors
45
50
55
60
Fig. 4.5: Processing time decrease for the distributed biomedical application
We took two of the distributed applications outlined in section 4.5.1 and a biomedical
application currently in the early stages of development and completed processing time
and speedup graphs based on measurements gathered by running these applications
with different numbers of donor machines (ranging from 5 to 60 processors). The
reason for choosing only the cryptography and substring applications (see section 4.5.1)
was that they illustrate the main difference between all of these problems: the large
variation in the amount of data involved. The substring application requires that a
complete copy of a genome (consisting of several Mbytes of data) be sent to each client
while the cryptography application only sends a few bytes of data to each client. The
distributed biomedical application used in these experiments models the path of
photons through human tissue and was developed using the algorithm outlined in
[PKJ89]. This program implements the Monte Carlo method of tracing the path of
photons of light as they pass through a medium by reflection (see [PKJ89] for a full
description). This application is currently being jointly developed between our research
group and the electronic engineering department at NUI Maynooth. For the substring
matching computation, the set of clients consisted of a software laboratory of 60
Pentium III 600Mhz PC’s with 128 Mbytes of memory each running either Windows
NT or RedHat Linux 7.2. For the other two problems, the set of donor machines
consisted of 60 PIV 2.4GHz PC’s with 512 Mbytes of memory each running Windows
2000 or RedHat Linux 7.2. Our server was running on a Pentium III 600MHz with 256
Mbytes of memory running Debian Linux with a 10Mbit/s network connection to the
donor machines. Figures 4.5, 4.6, and 4.7 show the processing time graphs and Fig. 4.8
67
shows the combined speedup graphs completed based on the results of these
experiments. For all of these experiments the laboratories were also in use by
undergraduate students so we were relying on utilising the ‘spare clock cycles’ of the
PC’s.
4
5
x 10
4.5
Time (sec)
4
3.5
3
2.5
2
1.5
5
10
15
20
25
30
35
40
Number of processors
45
50
55
60
Fig. 4.6: Processing time decrease for the distributed cryptography application
5
6
x 10
5
Time (sec)
4
3
2
1
0
5
10
15
20
25
30
35
40
Number of processors
45
50
55
60
Fig. 4.7: Processing time decrease for the substring application
It can be seen from Fig. 4.8 that the general-purpose nature of the system means that it
is possible to achieve significant speedup gains across a wide range of problems from
different areas of science. The deviations between the three speedup graphs can be
68
attributed to the different characteristics and parallel overheads of each problem and
differences in the dynamic non-dedicated computing environment that these tests were
run in.
60
Linear Speedup
Cryptography Application
Biomedical Application
Substring Application
55
50
45
Speedup
40
35
30
25
20
15
10
5
5
10
15
20
25
30
35
Processors
40
45
50
55
60
Fig. 4.8: Speedup gained for all three distributed applications. Linear speedup corresponds to the
theoretical maximum speedup achievable
69
Chapter 5
Programming Model
As part of our general-purpose distributed system, we designed a general programming
model that allows a user to submit arbitrary problems to our system and distribute the
computation among a fully cross-platform set of donor machines. As noted in chapter
2, our programming model has at its basis the message-passing paradigm. Information
is exchanged between the server and client by messages explicitly programmed by the
developer of the distributed application. In the following sections, we will outline the
general characteristics that a problem should exhibit to be suitable to run on our system.
When we refer to the word suitable in this chapter we are referring to applications that
will gain the greatest benefit from parallelisation on our system. We also give a detailed
description of the programming API that should be implemented by anyone wishing to
develop a distributed application to run on our system.
5.1 Problem Suitability
Firstly it should be stated that not all applications are suitable for distributed
computing. A potential developer of a distributed computation should think of tasks
that take whole days, weeks, months, or even years to complete on a single machine. It
has been widely acknowledged that there are a number of clearly identifiable
characteristics that a problem should exhibit in order to be suitable for a distributed
computing implementation [DAF95, Cod93, SKT97]. One of the two most important
characteristics is the ability to split a problem into independent units that can be
processed individually with little or no communication required between the subblocks. Furthermore, if a distributed system is to operate successfully in an
environment as diverse as the Internet where network communication links vary
greatly, the other important characteristic is that the volumes of data being transferred
across the network be kept to a minimum.
5.1.1 Coarse Grained Parallelism
With the advent of many large scale Internet based supercomputing projects [KWA01,
Wol96, LSS03, Ale99], a class of algorithmic parallelism referred to as ‘coarse-grained
parallelism’ has emerged as a means of describing the suitability of problems to large
scale distributed computing. Coarse grained parallelism refers to the way in which a
70
single large problem can be easily split up into discrete independent sub-blocks that can
be processed individually. There a number of areas where this characteristic is clearly
evident such space science [KWA01], finding large prime numbers [Wol96], testing the
strength of modern encryption techniques [Dis04], running complex weather
simulations [Ale99], and performing various types of analysis and simulations on
genomic data [Fig04, LSS03].
5.1.2 High Compute-to-Data Ratio
In any large scale distributed computing system, it is expected that donor machines may
be located anywhere on the Internet with communication links ranging from the fastest
gigabit networks right down to home users with standard 56Kb phone dial-up
connections. This network heterogeneity makes it infeasible for a distributed system to
routinely transfer large volumes of data between nodes of the system. Therefore it has
been identified that the most suitable problems for distributed computing should exhibit
a high ‘compute-to-data’ ratio. This means that a very small amount of data generated
from a problem should take a long time to process on a donor machine. A good
example of this is in cryptographic or mathematical applications where just two
numbers can define an entire search range for a donor machine to search within.
Similarly, the processing of work units should not generate large volumes of data to be
transferred back to a controlling server.
5.2 Overview of Programming API
Requirement 13 states that there should be a simple mechanism to be followed by
developers of potential distributed applications for our system. We decided that the
only way that we could provide such a mechanism would be to implement a simple and
intuitive programming model in our system. Our main goal was to reduce the
programming complexity that can be associated with other parallel programming
languages, by allowing the developer to program their distributed application at a level
that hides all of the lower level details involved in physically distributing out a
computation among a heterogeneous set of processors. Therefore, to fully define a
distributed computation on our system, the developer is only required to extend two
Java classes: the DataManager and Algorithm classes. Each of these parent
classes are part of the distributed system (see Figs. 3.2 and 3.3). The DataManager
class runs on the server while the Algorithm class runs on the client. The developer
must overwrite and implement several methods (listed in sections 5.2.1 and 5.2.2) in
71
order to set up their distributed computation. Extra Java libraries or user-defined classes
can also be included with a problem and must be made available to the problem
through the standard Java library mechanisms (see import keyword in Java [Fla02]).
Using a subset of the material presented in this chapter, we have completed a
developer manual that gives full details on how to program the system (including full
working examples) [Kea03]. The source code for a number of sample applications is
also available from the system webpage [Gen04].
5.2.1 DataManager Class
The DataManager class is a module of the server (see Fig. 3.2). The purpose of the
DataManager is to generate all work units to be sent to the clients, process all results
returned by clients, monitor and adjust the granularity of work units, generate status
information on the problem to be returned to the remote interface, and terminate the
distributed computation. The DataManager parent class consists of a number of
methods that a developer must implement (corresponding to the functionality outlined
in the previous sentence) in order for a computation to be accepted by the system. Each
problem that is entered into the system is given its own working directory on the server
to use during a computation. Typically this directory is used to create log files, store
temporary data, record results, and store data relating to the problem. This directory is
preset by the server when a problem is entered into the system and is available to a
DataManager through a final variable called PROBLEMDIRECTORY (type
java.io.File – see Java API). When the computation is finished, the contents of
this directory are zipped and returned to the user as the results set. It should be noted
that each of the DataManager methods outlined below should complete almost
instantaneously. Any tasks in the DataManager that will take any longer than a few
seconds to complete should be implemented using a Java thread.
The developer must implement a default constructor for their extended
DataManager class. Any uncaught exceptions or errors will be fed back to the server
via the throws clause. The server will automatically report these uncaught exceptions
back to the user via the remote interface. The sample DataManager constructor
shown in Fig. 5.1 also shows how to access the problem’s working directory on the
server.
72
public extendedDataManager() throws Throwable
{
//initialise DataManager here
//open problem directory
File problemDir = new File( PROBLEMDIRECTORY );
}
Fig. 5.1: General format of an extended DataManager constructor
The next method that has to be implemented is the generateWorkUnit() method.
This method is called by the system every time a client requests a work unit to process.
This method should perform whatever calculation is necessary to generate a work unit.
The return type of this method is java.util.Vector. The Algorithm running at
the client receives this Vector as its work unit to process (see section 5.2.2).
Therefore, it is usual for elements of this Vector to be explicitly type-casted back to
their original types in the Algorithm (via Java’s safe type casting mechanisms). If at
any time in the problem there are no work units currently available, then the
DataManager should return null. This indicates to the server that the problem has
no work units to issue at this time. As mentioned in section 3.3.2, the client would then
sleep for a preset amount of time. Fig. 5.2 shows the general format of this method.
public Vector generateWorkUnit() throws Throwable
{
Vector workUnit = new Vector();
//fill workUnit here or set to null
return workUnit;
}
Fig. 5.2: General format of generateWorkUnit() method
The third method that must be implemented is the processResults() method.
This method is called every time a client returns a set of results. There are two
parameters to this method. The first is the unique work unit ID (type is
java.lang.Long) generated outside this method by the system and given to every
work unit as it is generated by the generateWorkUnit() method. The second
73
parameter (type java.util.Vector) is the set of results that were returned by the
Algorithm running on the client. If the distributed computation is finished then this
method should return true, otherwise it returns false. When this method returns
true, the server removes the computation from the system and compresses the
problem’s working directory before it is downloaded by a user. The general format of
this method is shown in Fig. 5.3.
public boolean processResults( Long unitID, Vector
results ) throws Throwable
{
//process results set here
if( <problem is finished> )
{
return true;
}
else
{
return false;
}
}
Fig. 5.3: General format of processResults() method
It should be noted that there are no specific data structures provided by the system for
communicating
between
the
generateWorkUnit()
method
and
the
processResults() method. It is suggested that when a developer is implementing
their distributed computation, they should provide their own communication scheme to
identify which result set corresponds to which work unit. Typically this communication
is achieved by adding some identification information (or flags) to each work unit
generated by the generateWorkUnit() method and returning this information with
the corresponding results set passed to the processResults() method.
There are three other methods that must also be implemented by the developer.
These
are
the
adjustGranularity(),
getStatus(),
and
closeResources() methods. As noted in chapter 2, our system is expected to
operate in an unpredictable environment where network and processing resources are
extremely dynamic. Therefore our scheduling algorithm outlined in chapter 3 includes a
facility where the parallel granularity of problems running in the system be updated
continually throughout the lifetime of a computation. The dynamic updates of a
problem’s granularity is implemented via the adjustGranularity() method. This
74
method is called periodically by the server with one (positive or negative) parameter
corresponding to the percentage of how much the parallel granularity should be
adjusted so that the average processing time will match the optimal processing time
(see section 3.5.1).
public void adjustGranularity ( int percent )
throws Throwable
{
//assuming granularity is controlled by a
//variable called ‘granularity’
granularity = granularity + ( granularity * (
percent / 100 ) );
}
Fig. 5.4: General format of the adjustGranularity() method
This percentage is based on the average processing time of previous result sets
computed using the exponentially weighted moving average function (described in
section 3.5.1). Hence the problem is constantly adjusted to respond to changes in the
dynamic heterogeneous network of clients available to the system. The general format
of this method is outlined in Fig. 5.4.
The getStatus() method is called every time the remote interface makes a
status request to the server about this particular problem. The information returned by
this method is of type java.lang.String and is expected to contain some useful
information about the current state of the computation. The information returned is
printed to the screen of the remote interface (see Fig. 5.9 for a sample screenshot). Fig.
5.5 shows the general format of this method.
public String getStatus() throws Throwable
{
String s = //something meaningful about
problem progress
return s;
}
Fig. 5.5: General format of getStatus() method
The closeResources() method is called just before the problem is removed from
the system. The purpose of this method is to close any resources (e.g. files, directories,
input/output streams) that may be open. The user must implement the necessary code to
close any resources so that the problem files can be zipped and subsequently deleted
75
from the server’s disk. There are no parameters or return values from this method. The
general format of this method is shown in Fig. 5.6.
public void closeResources() throws Throwable
{
//close all resources
}
Fig. 5.6: General format of closeResources() method
Each of these methods is an abstract method in the parent DataManager class so
each of these methods must be implemented before a computation will be accepted by
the system.
5.2.2 Algorithm Class
The Algorithm is the piece of code that runs on the client and processes the work
units generated by the generateWorkUnit() method of the DataManager. Only
one method must be implemented by the Algorithm – the processUnit()
method. Typically an Algorithm can be made up of a number of sub-methods to
process different types of work unit generated by the DataManager. The work unit
Vector generated by the DataManager is passed as a parameter to the
processUnit() method and all elements of this Vector can be type-casted back
to their original types via Java’s safe type-casting mechanisms. The results of the work
unit are returned from this method as a Vector and are subsequently returned to the
processResults() method of the DataManager. The general format of the
processUnit() method is illustrated in Fig. 5.7. Any uncaught exceptions or errors
will be caught by the throws clause of this method and fed back to the server where
they are logged in the problem log files that are included with the results set of every
computation. If an exception occurs in the same work unit on more than one donor
machine, then the entire problem is removed from the system.
76
public Vector processUnit( Vector workUnit ) throws
Throwable
{
//process the work unit here
Vector results = new Vector();
//fill the vector with results of computation
return results;
}
Fig. 5.7: General format of the processUnit() method of an Algorithm
5.2.3 Extra User defined Classes and Libraries
When designing our programming API, we recognised that developers may wish to
utilise extra third party classes or Java libraries in their distributed application. We have
included a facility whereby a user can specify extra libraries (Java JAR files) or class
files when they are entering their problem into the system using the remote interface
(see section 5.3.3). To utilise these extra libraries in either the Algorithm or
DataManager, the developer should follow the standard Java mechanisms (using the
import keyword at the top of each source file [Fla02]).
5.2.4 Problem Data
We recognised that for certain problems it may be necessary to transfer relatively large
amounts of data between the server and client. Therefore we have included a feature in
our programming model that makes it possible for large amounts of data to be
transferred to the Algorithm running on the client. It should be noted that this feature
should only be utilised where there is a high bandwidth network connection between
the server and every client. These data files are specified by the user when they are
submitting their computation to the system (via the remote interface). These files are
written to the problem’s working directory on the server (see section 5.2.1 PROBLEMDIRECTORY variable). Whenever a problem data file is required by the
Algorithm, the Algorithm should simple attempt to open the file using the
PROBLEMDIRECTORY variable set as the parent. The client software will
automatically download this file from the server using TCP sockets if it has not already
been downloaded to the client. In chapter 6, we describe a full working distributed
application that utilises this functionality throughout a computation. A simpler full
working example showing how to utilise this functionality is available from the system
webpage [Gen04].
77
5.3 Compiling, Testing, and Submitting a Problem
In this section we outline the steps that a developer must follow to actually compile and
submit a problem to the system.
5.3.1 Compiling Problem Files
Once a developer has completed the design and implementation of their distributed
application, the Algorithm and DataManager files can be compiled using the
standard javac command with one minor addition. The file compile.jar should
be included in the classpath of the javac command (using the –classpath option)
as follows:
javac –classpath compile.jar *.java
A copy of this file is included in the standard distribution of the system.
5.3.2 Testing a Problem
We found that it may be useful for a developer to have the ability to test their
distributed application on a single machine while it is still in development. Although it
is possible to set the distributed system up on a single machine, we developed a simple
Java application that can be used to quickly test a distributed computation on a single
machine. This application simulates the calls that are made to the DataManager and
Algorithm by the distributed system.
As noted previously, this distributed system is ideally suited to problems that
fit into the class of coarse grained parallelism with a high ‘compute-to-data’ ratio.
Hence it is envisaged that each work unit that is sent out should take substantially
longer to process than it took to transfer the data over the network. This application can
be used by a developer to ascertain an initial estimate of a suitable parallel granularity
(amount of processing to be completed by each donor machine per work unit) for their
problem. This application is included in the standard distribution of the distributed
system and is run from the command line with the following command:
java -jar problemTester.jar <Algorithm> <DataManager>
Any extra Java libraries that are required by the problem should be included in the
classpath of the previous command (via the standard –cp option). Any problem data
78
files for the particular problem being tested should be included in the same directory
where the problemTester.jar is located.
5.3.3 Submitting a Problem
When a distributed computation has been compiled and tested appropriately, the
problem can be submitted to the server via the remote interface. To submit a problem to
the system, the user is required to be connected to the server in administrator mode.
New jobs are added to the system via the jobs add screen (shown in Fig. 5.8). The user
selects the files for their distributed computation using the various ‘Browse’ buttons.
The user should give each computation a unique name using the ‘Problem Description’
text box. For a situation where a user has a particularly processor- or memory-intensive
computation, we have provided a facility where the user can specify the minimum
processor or memory requirements that any potential donor machine should meet to
process work units for the computation. The priority field is used to prioritise
computations in the system. If a user requires the results of their computation very
soon, then the priority of the job should be increased from the default value. The final
two fields relate to the tolerance of the distributed system to exceptions in the
Algorithm and how many times an individual work unit can expire before the
system will remove the problem. Most users do not need to concern themselves with
these fields and these fields can normally be left at the default values. The computation
is submitted to the system when the user presses the ‘Add’ button. If there is an
exception generated while the remote interface adds the computation to the server, then
an error message will be displayed. Once a computation has been successfully
submitted to the distributed system, the progress of the computation can be monitored
in real-time using the remote interface (see Fig. 5.9 for a sample screenshot).
79
Fig. 5.8: Screenshot of job add screen on the remote interface
Fig. 5.9: Sample screenshot of real-time feedback of a problem’s status
80
Chapter 6
DSEARCH: Sensitive database searching using distributed
computing
In this chapter we introduce the bioinformatics application, DSEARCH, which we
developed as a useful tool for bioinformatics researchers to speed up sensitive database
search computations. The development of this application took place after the
distributed system was completed. A paper based on the contents of this chapter (see
Appendix A) has recently been accepted for publication [KeN04]. A user manual, full
source code and binaries for DSEARCH are available for download from the system
webpage [Gen04]. Appendix B contains a glossary for many of the bioinformatics
terms used in this chapter.
6.1 Introduction
Database searching is one of the most fundamental tasks in bioinformatics [BCH98]. In
this chapter we focus on the problem of searching large sequence databases for similar
(or related) sequences using the most sensitive search algorithms. In biological terms a
sequence can be made up of DNA, amino acid, or proteins sequences, and when
digitised, these sequences can be considered as strings of characters [Les02, Mou04].
The evolution of a biological sequence is commonly explained as a series of
evolutionary changes or events (such as mutations, insertions, and deletions) that have
transformed an ancestral sequence into the current sequence. Two sequences are said to
be homologous if they have evolved from a common ancestral sequence. One of the
biggest problems in reconstructing the evolutionary history of two sequences is that no
information about the common ancestor or the occurred events is available. Therefore
we must use an evolutionary model to simulate the evolution of sequences in such a
way that the evolutionary history can be inferred computationally [Hig04]. Using an
evolutionary model, we can measure the evolutionary relatedness by identifying the
number of mutations, insertions, and deletions of bases necessary to transform one
DNA sequence into another one. Searching for a related sequence in a database requires
one to adopt such an evolutionary model. Database searching is also a useful tool for
identifying regions of interest in newly discovered sequences by comparing the new
sequence with previously studied sequences. “Each cell in an organism contains a
complete copy of the genome (the ‘program’ for making the organism) encoded in
81
Deoxyribonucleic Acid (DNA). And just like a programming language, DNA has
‘coding bits’ (called exons) and ‘non-coding bits’ (called introns), the latter merely
providing structural integrity and regulatory functions. Alternatively, exons are data,
whereas introns provide the logic. A gene is a segment of DNA that specifies how to
make a protein and thus plays a structural or functional role in all parts of an
organism’s/species’ body. The DNA also determines the types of different cells by using
a mechanism called differential gene expression. Each cell contains a complete copy of
the organism’s genome, however, not all genes are expressed in all types of cells all the
time. Differential gene expression determines where, when, and in what quantity each
gene is expressed. Only about 40% of the estimated 24,000 human genes are expressed
at any time” [Cra04]. One of the most basic techniques used to identify a coding region
of a genome is to compare the genome (via sequence alignment) with parts of other
closely related species whose coding regions have already been identified. In this way,
a researcher can hypothesize the function of a coding region by finding a similar region
(which may or may not be from another organism) whose function is known.
6.1.1 The Dotplot
A dotplot is a graphical representation showing regions of similarity between two
sequences. It consists of a 2D grid with the sequences placed on the horizontal and
vertical axes, respectively. A window length is fixed, together with a criterion defining
when two sequence windows are deemed to be similar. Whenever one window in one
sequence resembles another window in the other sequence, a dot or short diagonal line
is drawn at the intersection of the indices of the positions of the windows in their
respective sequences (see Fig. 6.1). If two sequences only share patches of similarity,
then this will be revealed by isolated diagonal stretches. The dotplot captures in a single
picture the complete set of all possible alignments between two sequences. Any path
from the top-left of the dotplot moving right or down corresponds to a possible
alignment. For sequences that display high degrees of similarity, it should be possible
to directly read off alignments from the dotplot. A dotplot is a powerful tool for
displaying global or local regions of similarity and identifying repeat sequences.
However it may be difficult to visually detect distantly related sequences using only a
dotplot. An alignment program tries to find the best alignment between two sequences,
which can be seen as trying to find the optimal path through the dotplot including all
(or the most visible) diagonals.
82
6.1.2 Measures of Alignment Quality
For large numbers of sequences, it becomes necessary for a researcher to use a less
time-consuming approach to detecting sequence similarity than computing every
possible dotplot. In the past number of years, computational techniques for database
searching have become an essential tool for biologists [BCH98]. The basis for all
alignment algorithms is defining some quantitative measure (or score) of sequence
similarity. The basis of these measures is that certain evolutionary changes are more
likely to occur in molecular sequences than others. Scoring schemes define the value of
aligning sequences and are derived from models of evolution [AtP99].
Fig. 6.1: A dotplot produced by the author using Dotter [SoD95] of two highly-similar cDNA sequences
obtained from dbEST [BLT93]
The earliest and most widely used scoring scheme is that of Dayhoff [Day78] and is
called the PAM (point accepted mutation) matrices [Cen04]. This scoring scheme
consists of a number of different matrices which contain probabilities of particular
mutations (see [Cen04] for further details). Figure 6.2 shows the PAM250 matrix with
the horizontal and vertical axes representing the residues. The intersection of two
residues in the matrix defines the score obtained by aligning the two residues. The
reward for conserving a particular residue varies per residue and is represented by the
83
diagonal. To evaluate a particular alignment of two sequences, the sum of the
corresponding residue entries is computed and compared with the score obtained from
other possible alignments. The alignment with the highest score corresponds to the
optimal alignment (according to the evolutionary model defined in the scoring matrix).
Fig. 6.2: PAM250 scoring matrix [Day78]
One final aspect of a scoring scheme is the scoring of insertions and deletions (often
referred to as ‘indels’ or gaps) in the alignment of two sequences. The introduction of a
gap allows an alignment to continue where one sequence has either been newly inserted
into it, or a subsequence that has been lost in the other sequence. A penalty for gaps is
subtracted from a sequence alignment’s score because a gap introduces uncertainty into
the alignment. The choice of gap penalty is very important because a low penalty may
allow unrelated sequences to appear similar. Biologically, it is easier for a protein to
accept a different residue in a position (mutation), rather than having parts of the
sequence chopped away or inserted. Gaps/insertions should therefore be rarer than
point mutations so gap penalty values are designed to reduce the score when an
alignment has been broken by an insertion in one of the sequences. On the other hand,
the value should also be small enough to allow a previously accumulated alignment to
continue with an insertion in one of the sequences and should not be so large that this
previous alignment score is removed completely. The most common way to penalise a
84
gap is to use a combined penalty, consisting of a gap opening penalty and a penalty
based on the length of the gap. Typically the combined gap penalty is computed from
P = o p + (l ∗ l p )
(6.1)
where P is the total penalty, o p is the gap opening penalty, l is the length of the overall
gap, and l p is the penalty per gap entry.
6.2 Alignment Algorithms
All alignment algorithms can be placed into one of two categories: global or local. A
global alignment considers similarity across the entirety of the two sequences by
attempting to produce a maximal match between the two sequences (defined as the
optimal matching sequence that can be identified between two complete sequences,
allowing for indels). One of the first alignment algorithms proposed by Needleman and
Wunsch [NeW70] automated the task of producing a global alignment between two
sequences. The Needleman-Wunsch algorithm works well for sequences displaying a
high global similarity but for distantly related sequences that exhibit only small regions
of local similarity, a suitable global alignment is often not found. A local alignment
algorithm, such as that proposed by Smith and Waterman [SmW81], focuses on finding
regions of similarity between the two sequences and forms the basis of most modern
database search programs. Both of these algorithms are based on dynamic
programming where the solution to the problem of finding the optimal alignment is
solved by sequentially finding the solution to a set of smaller sub-problems. For both of
these algorithms, we assume that we have a scoring scheme (similar to one of those
described in section 6.1.2), f ( x, y ) = z where x and y are characters and z is an integer,
and we have a linear function, g (u ) = v where u and v are integers, that defines the
cost, v, of introducing a gap of u characters into the alignment.
6.2.1 Needleman and Wunsch Algorithm
The Needleman-Wunsch algorithm was first proposed in 1970 [NeW70]. Both the time
3
and space complexity of the original algorithm was O ( n ) , where n is the length of the
2
sequences, however this was reduced to O ( n ) due to enhancements to the original
algorithm made by Sellers [Sel74]. The algorithm can be broken down into two main
stages where the first stage computes the cost of all possible alignments of the two
sequences by building a cost matrix (similar to a dotplot). The second stage retraces the
85
cost matrix to find the optimal global alignment (one with the least cost) of the two
sequences.
The first stage of the algorithm, shown in Fig. 6.4, starts by constructing an
empty 2D matrix, C, of size a × b , where a and b are the lengths of the two sequences.
The leftmost vertical column and topmost horizontal row are initialised with the initial
gap penalties for starting the alignment at each offset starting position (as NeedlemanWunsch is a global alignment algorithm). The matrix is then populated iteratively
according to the score and gap functions. The value of each element is calculated as the
maximum of one of three possible options:
1. Continue an existing alignment (diagonal arrow in Fig. 6.3) and add the cost of
aligning the two characters (defined by the function f) to the accumulated score
so far in C(i-1,j-1)
2. A gap is inserted into the alignment by aligning s1(i) with a gap character
(horizontal arrow in Fig. 6.3). The cost of inserting a gap is calculated using the
function g which is a linear function based on the length of a pre-existing gap
(if any) and added to the score so far in C(i-1,j)
3. A gap is inserted into the alignment by aligning s2(j) with a gap character
(vertical arrow in Fig. 6.3). The cost of inserting a gap is again calculated using
g and added to the score so far in C(i,j-1)
i-1
i
C(i-1,
C(i,
j-1
j-1)
j-1)
j
C(i-1,
j)
+f(s1(i),s2(j))
+g(u)
+g(v)
Fig. 6.3: Graphical representation of 3 possible alignment options at each step when computing
matrix C
86
Inputs:
Sequence s1 (length = a)
Sequence s2 (length = b)
Score function f(x,y)
Gap Cost function g(z)
Algorithm:
Create matrix C
C(0,0) = 0
for( i = 1 to a )
C (i,0) = g(i)
end for
for( j = 1 to b )
C (0,j) = g(j)
end for
for( i = 1 to a )
for( j = 1 to b )
C(i − 1,j − 1)+ f(s1(i),s2(j))

C (i,j) = maxC(i − 1,j)+ g(u)
C(i,j − 1)+ g(v)
end for

end for
Output:
Matrix C
Fig. 6.4: First stage of Needleman-Wunsch algorithm that populates the matrix C where u and v are the
corresponding lengths of a pre-existing gap in the alignment [NeW70, Sel74]
The second stage of the algorithm is to find the optimal alignment from the matrix, C,
computed in the first stage. Figure 6.5 illustrates the traceback algorithm that is used to
find the optimal alignment. The algorithm attempts to traceback the optimal alignment
by starting at the bottom rightmost entry of the matrix, C(a,b) in Fig. 6.5, and works
its way back through the matrix following the path of an optimal alignment. It should
be noted that there may be several possible paths that have the same cost (defined by
places where the score after a gap insertion or mutation is the same). When this occurs,
the algorithm described in Fig. 6.5 defaults to choosing the path that corresponds to a
mutation. The optimal alignment is contained in sequences s3 and s4 which
correspond to the original two sequences with gap characters inserted (represented by a
dash) to make up the optimal alignment.
87
Inputs:
Sequence s1 (length = a)
Sequence s2 (length = b)
Score function f(x,y)
Gap Cost function g(z)
Matrix C
Algorithm:
Sequence s3, s4
i = a, j = b, k = 0
while( i > 0 or j > 0 )
if( i > 0 & j > 0 & C(i-1,j-1) + f(s1(i),s2(j)) = C(i,j) )
s3(k) = s1(i)
s4(k) = s2(j)
i = i – 1
j = j - 1
else if( i > 0 & C(i-1,j) - g(u) = C(i,j) )
s3(k) = s1(i)
s4(k) = i = i – 1
else if( j > 0 & C(i,j-1) - g(v) = C(i,j) )
s3(k) = s4(k) = s2(j)
j = j – 1
end if
k++
end while
Output:
Aligned sequences s3 and s4
Fig. 6.5: Traceback algorithm of second stage of Needlman-Wunsch algorithm where u and v are the
corresponding lengths of an existing gap in the alignment [New70, Sel74]
6.2.2 Smith and Waterman Algorithm
In 1981, Smith and Waterman [SmW81] proposed a variation of the NeedlemanWunsch algorithm that could identify local regions of similarity by locating regions of
both sequences that exhibit a high similarity. The best local alignment is the highest
scoring alignment of a substring (arbitrary length) of each sequence. The SmithWaterman algorithm takes a matrix based approach like Needleman-Wunsch and uses a
similar traceback algorithm to produce the optimal local alignment of the two
2
sequences. Both the time and space complexity of Smith-Waterman is also O ( n ) ,
where n is the length of either sequence. The first stage of the algorithm computes a
cost matrix similar to Needleman-Wunsch with two important differences. All elements
of the cost matrix, C, are first initialised to zero (see Fig 6.6) and there is a fourth
option which allows the score of the alignment to be reset to zero at any point in the
algorithm. This gives the algorithm the functionality to restart the alignment at arbitrary
positions within the matrix.
88
Inputs:
Sequence s1 (length = a)
Sequence s2 (length = b)
Score function f(x,y)
Gap Cost function g(z)
Algorithm:
Create matrix C
for( i = 0 to a )
for( j = 0 to b )
C(i,j) = 0
end for
end for
for( i = 1 to a )
for( j = 1 to b )
C(i − 1,j − 1)+ f(s1(i),s2(j))
C(i − 1,j)+ g(u)

C(i,j) = max
C(i,j − 1)+ g(v)
end for
C(i,j) = 0
end for
Output:
Matrix C
Fig. 6.6: First stage of Smith-Waterman algorithm where matrix C is populated where u and v are the
corresponding lengths of a pre-existing gap in the alignment [SmW81]
The second stage of the algorithm (see Fig. 6.7) must first locate the end position of the
optimal alignment by locating the element of C that contains the highest value. Once
this is found, the algorithm must traceback the path of the optimal alignment by moving
back through C until the alignment score diminishes to zero. It should be noted that as
this is a local alignment algorithm, the length of the alignment produced is usually less
than the length of the input sequences.
89
Inputs:
Sequence s1 (length = a)
Sequence s2 (length = b)
Score function f(x,y)
Gap Cost function g(z)
Matrix C
Algorithm:
Sequence s3, s4
max = 0, mi = 0, mj = 0
for( i = 0 to a )
for( j = 0 to b )
if( C(i,j) > max )
max = C(i,j)
mi = i
mj = j
end if
end for
end for
i = mi, j = mj, k = 0, score = max
while( score > 0 )
if( i > 0 & j > 0 & C(i-1,j-1) + f(s1(i),s2(j)) = C(i,j) )
s3(k) = s1(i)
s4(k) = s2(j)
i = i – 1
j = j – 1
score = C(i-1,j-1)
else if( i > 0 & C(i-1,j) + g(u) = C(i,j) )
s3(k) = s1(i)
s4(k) = i = i – 1
score = C(i-1,j)
else if( j > 0 & C(i,j-1) + g(v) = C(i,j) )
s3(k) = s4(k) = s2(j)
j = j – 1
score = C(i,j-1)
end if
k++
end while
Output:
Aligned sequences s3 and s4
Fig. 6.7: Traceback algorithm of the second stage of Smith-Waterman algorithm where u and v are the
corresponding lengths of an existing gap in the alignment [SmW81]
6.2.3 Distributed Computing and Database Searching
The algorithms described in the previous section are the most rigorous alignment
algorithms as they consider every possible alignment of two sequences. Consequently
2
both the space and time complexity of these algorithms is quite high, O ( n ) where n is
the length of either sequence, making the process of searching large databases of
sequences using these algorithms infeasible using only a single processor. Several
authors have attempted to address this problem by proposing heuristic algorithms to
90
reduce the search time [AGM90, LiP85]. These programs have made the process of
searching large numbers databases feasible using only a single processor. However
these programs are based on algorithms that reduce the sensitivity of a search and can
fail to detect certain types of matches. The heuristic algorithm implemented by BLAST
[AGM90] first identifies short exact matching sequences of a certain word size and
then expands these matches into local alignments. The obvious problem with this
approach is that if an exact match of sufficient length does not exist within an
alignment, BLAST will fail to detect the entire alignment. One approach to counter this
loss in sensitivity is to reduce the word size in BLAST to a small value in order to
detect short sequences, however this defeats the purpose of the BLAST heuristic and
erodes any performance benefit that might have been gained. When given the choice, a
biologist would prefer to use the more rigorous Needleman-Wunsch or SmithWaterman algorithms for their database searches.
Several authors have attempted to parallelise the process of searching large
sequence databases [Hua89, Lop87, ASP95, BSW02, DCF03, BPC01, GlC97, PCB99].
Most of the attempts to parallelise the alignment process can be placed into one of three
categories. The first approach has been to split the set of query sequences up into
groups and distribute these among a set of processors to align against the entire
database [BSW02, DCF03]. The second approach has been to split the database and
distribute the database pieces and query sequences among a set of processors [BSW02,
DCF03, Hua89]. Large databases of short genomic sequences such as ESTs (Express
Sequence Tags), genes, coding DNA, protein sequences, and so on, can easily be split
up into variable size independent sub-blocks meaning that individual alignments can be
performed completely independently of the rest of the database. The final approach is
to exploit some hardware specific property to accelerate the sequence comparisons
[Rog01, Lop87, Hua89, ASP95]. In chapter 5, we outlined the general characteristics
that a problem should exhibit to be most suitable for a distributed computing
implementation such as fitting into the class of coarse grained parallelism and
displaying a high ‘compute-to-data’ ratio. So when designing DSEARCH, we decided
to take a hybrid approach where we split both the database and set of query sequences
(similar to [BSW02, DCF03]) up to gain optimal parallelisation and speedup of the
computation.
91
6.3 DSEARCH Algorithm
DSEARCH implements a distributed version of both the Needleman-Wunsch and
Smith-Waterman search algorithms where the database of sequences is split into a
number of smaller groups of sequences and sent to donor machines with subsets of
query sequences to be searched using an alignment algorithm and scoring scheme
chosen by the user. For every query sequence, the program computes the top n matches
found in the database (where n is an input parameter) and returns details of these
matches via a results file.
6.3.1 Query Window
Before we describe the distributed algorithm implemented in DSEARCH, we must
describe one of the basic data structures in DSEARCH called a query window. A query
window consists of a number of query sequences grouped together that must be aligned
with every sequence in the database. Initially DSEARCH splits the database into fixed
size chunks so the query window was designed as a mechanism to allow the system to
dynamically adapt the parallel granularity of DSEARCH computations (see section
6.3.3). This was inspired by the congestion control algorithm of the TCP protocol
[SMM98, Pos81, Tan03].
We have divided the process of aligning a set of query sequences in a query
window into two stages (see Fig. 6.8). In stage 1, each query sequence is first aligned
with each database sequence using a low memory equivalent [SmW81, NeW70] ( O (n)
in memory, where n is the length of the shortest sequence) of the full alignment
algorithm that only returns the score of the optimal alignment of two sequences (not the
optimal alignment itself). A donor machine is issued the query window and a database
chunk to search for similar sequences. For every query sequence, the output of stage 1
is a list of the top n highest alignment scores and what database sequence these scores
occurred in. This information is passed to stage 2, where the top n matches are aligned
using the full alignment algorithm. For each query sequence, the set of final n
alignments are written to a results file in the standard ‘sequence above sequence’
format (see Fig. 6.11) with the alignment score to allow further examination and
analysis by the user.
92
Inputs
Stage 1
Query 1
Low Memory
Equivalent
Query 2
Low Memory
Equivalent
Query 3
Low Memory
Equivalent
.
.
.
.
Query m
Top n scores
Top n scores
Top n scores
.
.
.
.
Low Memory
Equivalent
Top n scores
Stage 2
Outputs
Full
Alignment
Alignments
Details
Full
Alignment
Alignments
Details
Full
Alignment
Alignments
Details
.
.
.
.
.
.
.
.
Full
Alignment
Alignments
Details
Fig. 6.8: Illustration of the two stage alignment process implemented by a query window
6.3.2 DSEARCH Distributed Algorithm
A DSEARCH computation creates a number of query windows by partitioning the set
of query sequences into variable sized groups. Each query window aligns its set of
query sequences with each database sequence as outlined in the previous section. To
maximise donor machine usage and obtain the lowest overall running time for a
computation, a DSEARCH computation will typically have multiple query windows in
operation simultaneously (see Fig. 6.9).
Figure 6.10 shows a flowchart of the steps that are followed every time a
donor machine requests a work unit from DSEARCH. DSEARCH first checks all of the
existing query windows to see if any window has more database chunks to be searched.
If any query window had an alignment to be completed, then DSEARCH issues the
alignment to be performed by the donor machine. If none of the existing query
windows has an alignment to issue, then the program attempts to create a new query
window and issue a set of alignments. If all of the query sequences have already been
aligned and the current list of query windows are not finished their alignments, then the
program reports to the distributed system that it has no work to be completed at this
time. If all of the query windows are finished searching the database and there are no
more query sequences left, then the program terminates.
93
Inputs
Query
Sequence
12
Query
Sequence
Query
Sequence
.
Query
Sequence
Query Sequence. m
Sequence
Database
Scoring
Scheme
Query Window 1
Database
Query Window 2
QS 1
QS 2
QS 3
Database
QS 4
QS 5
QS 6
Query Window w
....
Database
QS m-2
QSm-1
QS m
Outputs
QS Alignments 1
QS Alignments 2
QS Alignments 3
QS Alignments 4
QS Alignments 5
QS Alignments 6
....
QS Alignments m-2
QS Alignments m-1
QS Alignments m
Fig. 6.9: Illustration of overall DSEARCH alignment computation, where m is the number of query
sequences and w is the number of query windows created during the computation (QS: query sequence).
The database is initially split up into fixed size chunks that are issued with the query windows
Server
Client
Is the list of QWs >
0 && does any QW
have alignments?
Yes
Perform
Alignments
No
Are there any
query sequences
left?
No
No work
Yes
Create new
QW
Perform
Alignments
Fig. 6.10: Flowchart showing the decision process followed by DSEARCH when a client requests a work
unit (QW: query window)
6.3.3 Dynamic Granularity Updates
It was noted in chapter 5 that the adjustGranularity() method must be
implemented by every distributed computation to dynamically control the granularity of
a problem. This method is called periodically by the server with one parameter
corresponding to how much the parallel granularity of the problem should be adjusted
by so that the average processing time will match the user-defined optimal processing
time (set to 1 hour by default). In DSEARCH, we have implemented this method to
dynamically alter the size of subsequently created query. Therefore the total number of
94
query windows that are created during a computation (corresponding to w in Fig. 6.8)
cannot be predetermined and depends on the processing capacity of the donor machines
available when a computation is executing.
6.3.4 Inputs and Outputs
Early in the design of this application, we decided that an important feature is that
DSEARCH should conform to commonly used formats in bioinformatics. Therefore,
we followed a modular approach when designing DSEARCH, which meant that we
could subdivide the program’s functionality into well-defined modules, such as those
for reading databases, interpreting scoring matrices, and performing alignments. In the
initial release of DSEARCH, there is support for FASTA format [LiP85] query
sequences and databases. Due to the modular design of DSEARCH, this could be easily
extended to other formats (Genbank [BKL00], SwissProt [BaA00], EMBL [KAA04],
PIR [GBM96]) with little modification required to the rest of the program. The user
must also supply either a basic scoring scheme (consisting of match reward, and
mismatch and gap penalties) or the scoring matrix that they want to use to align the
sequences. A scoring matrix is submitted in the standard table format used to represent
the PAM [Day78] and BLOSUM6 [HeH92] matrices where there is one row and one
column for each defined character (see Fig. 6.2 for an example). We have included
examples of the commonly used scoring matrices with the DSEARCH distribution. The
other parameter that must be supplied by a user is the number of alignments to return
for each query sequence (corresponding to n in Fig. 6.8). The program outputs a results
file that contains one alignment file per query sequence containing details of the top n
alignments over the entire database in the standard ‘sequence above sequence’ format
(see Fig. 6.11).
6.4 Performance Analysis
As mentioned in chapter 4, there are a number of standard metrics used to measure the
performance of parallel programs such as speedup and scalability. We have completed
preliminary speedup and scalability tests for DSEARCH by running the application
with different numbers of donor machines (ranging from 5 to 40 processors) and
varying database sized. The set of donor machines for these tests consisted of a
software laboratory of 40 Pentium 4 2.4GHz PC’s with 512 Mbytes of memory each
running either Windows 2000 or RedHat Linux 7.2. Our server was running on a
6
BLOSUM is from BLOcks Substitution Matrix
95
Pentium III 600MHz with 256 Mbytes of memory running Debian Linux with a
10Mbit/s network connection to the donor machines. For all of these experiments we
did not have sole use of the laboratories and were relying on utilising the ‘spare clock
cycles’ of the PC’s. Furthermore, we compared the performance of DSEARCH with the
current most popular single processor native program for performing Smith-Waterman
searches.
Query File: gi|40787518|gb|CK445276.1|CK445276 LP14156.5prime LP Drosoph
Database Sequence: gi|41344752|gb|CK623866.1|CK623866 mi14e09.y1 Mouse R
CTGTGA-GCGTT--TTTCAGA-GACACGTGAC-CATCAATAAGCATGAAGAACCCA-TTTGC-ACACCTTGC
C GT A G GTT
T AGA G CA G G C C
CAAT
C T
G
CCCA TT G
A A CTT C
CAGTAAAGAGTTCCAATAAGATGTCA-G-G-CGC--CAAT-C-CCT-TCG—CCCAGTTCGAGAAAACTTTCT
Score: 2152
Database Sequence: gi|41338864|gb|CK617978.1|CK617978 ml04b02.y1 Mouse r
GAGGAC-TC-A-AG-A--TACGGACG-CTTACCATTTTCTATCAGAGTTCTCCTGGAGGCAGCCGTTCGGAA
G
C TC A AG A
TA G
C
C T CC T
TC ATC GAGT CTC TGGAG C GC GTTCG AA
GCCTCCATCGATAGTAAATATGATCAAC-TGCCCTACTCGATCCGAGTGCTCTTGGAGTCGGCGGTTCGCAA
Score: 1853
Fig. 6.11: Part of a database search output in ‘sequence above sequence’ format (produced by
DSEARCH). The dashes are gaps in the alignment and the spaces are mismatches. The scores were
computed using the Blosum62 matrix [Day78]
6.4.1 Single Processor Comparison
We wanted to compare the performance of DSEARCH with the current most popular
single processor Smith-Waterman search program, SSEARCH. SSEARCH [LiP85] is
part of the FASTA family of programs and is used by biologists to perform SmithWaterman searches. We tested the performance of DSEARCH against SSEARCH by
running DSEARCH on a single processor and searched a 13 Mbyte EST database with
a 173 Kbyte set of 219 query sequences. The comparative runtimes were 1207 minutes
and 493 minutes, respectively. Although DSEARCH is approximately two and half
times slower than SSEARCH, DSEARCH can makes up for this performance loss by
its ability to distribute a large database search across a cross-platform set of processors.
6.4.2 Runtime Time and Speedup
For our runtime and speedup tests, we obtained a database of 13.5 Mbytes in size and a
set of 219 query sequences consisting of 173 Kbytes in total. Both of these databases
consist of thousands of EST sequences that were obtained from dbEST [BLT93] (a
96
division of Genbank [BKL00]). Figures 6.12 and 6.13 show the processing time and
speedup graphs that were obtained by running this alignment with varying numbers of
clients.
It can be clearly seen that DSEARCH performs quite impressively as the
number of processors is steadily increased up to 40. For this dataset and set of
processors, we did not reach the limits of the performance benefits from that can be
gained by using DSEARCH. By examining Fig. 6.7, we suspect that for this dataset and
set of processors the performance gains may plateau at 80 to 100 processors due to the
small database size used in this test.
4
12
x 10
10
Time (sec)
8
6
4
2
0
5
10
15
20
25
Number of processors
30
35
Fig. 6.12: Decrease in processing time as the number of processors is increased
97
40
40
DCPal
Theoretical maximum
35
30
Speedup
25
20
15
10
5
5
10
15
20
25
Number of processors
30
35
40
Fig. 6.13: Speedup achieved by DSEARCH over a varying numbers of processors
6.4.3 Scalability
One of the major challenges for a distributed system in this type of computation is the
ability to manage the large volumes of data of large sequence databases. Genomic
databases can range from only a few hundred kilobytes to hundreds of gigabytes. We
wanted to test the scalability of DSEARCH by searching successively larger databases
and measuring what effect the increase in database size has on the overall running time
of the program. For these tests, we used the same set of query sequences (173Kbytes)
and fixed the number of clients at 40 processors. We obtained a set of EST databases
from dbEST [BLT93] of varying sizes, ranging from 25 Mbytes to 400 Mbytes. Figure
6.14 shows the corresponding runtimes that were achieved for each database size.
These tests were conducted using a standard non-dedicated 10 Mbit/s network and
Fig. 6.14 shows that the network overhead remains negligible because the runtimes
increase approximately linearly with increasing database size. From the evidence
presented in Fig. 6.14, we expect DSEARCH to scale to much larger databases before
the network overhead would become a significant limitation. We give a critical
appraisal of DSEARCH in section 7.1.4.
98
9000
8000
7000
Time (min)
6000
5000
4000
3000
2000
1000
0
0
50
100
150
200
250
Database size (MB)
300
350
Fig. 6.14: DSEARCH runtime increase for an increasing database size
99
400
Chapter 7
Conclusions
This chapter provides an overall summary of the achievements of this project. This
includes a brief summary of our review of the current state of distributed computing.
We also highlight some of the distinguishing features of our system with the emphasis
on illustrating the capabilities of our general-purpose distributed system. As we
concluded the work on this project, we also identified several areas of the system that
warrant further research and development.
7.1 Summary of Work
The main work of this project can be broken into three main sections. Initially we
outlined a summary of our review of distributed computing and how we drew up our
list of requirements for the system. Our design and implementation chapters explain our
system design and how we implemented this design in Java. As part of this project, we
completed one distributed application that has the potential to be used by
bioinformatics researchers to significantly speedup rigorous database searches. We
review each of these aspects of the project in the following sections.
7.1.1 Distributed Computing Review
At the outset of the project, we completed an extensive review of the current set of
distributed computing systems currently in widespread use. In chapters 1 and 2, we
outlined the major limitations that are imposed by existing systems. As part of this
review, we produced a set of requirements that specify the most general-purpose
distributed computing system ever built. Our goal was to identify the best features of
the current set of distributed computing systems, combine these features into a single
set of requirements, and produce a distributed system according to these requirements.
Before this project, there were no free distributed systems that could claim to fully
implement all of these requirements.
7.1.2 General-Purpose Distributed System
In chapter 2, we outlined the high level models that our system is based on. Our system
can be viewed on a number of different levels. At the highest level, our system can be
classified as a MIMD parallel computing architecture that is based on the distributed
100
memory model. We also examined a number of parallel programming models. Our task
was to design a general programming model that would allow developers to quickly
and easily implement MIMD distributed programs. We decided to use the message
passing model as the basis of our user programming model. This choice was mainly
based on the generality of the message passing model. As demonstrated in chapter 4,
the generality of our programming model means that the system can run multiple
computations of varying parallel complexity. Our communications model is based on
the client-server model of communication and we proved in chapter 2 how it is possible
to emulate a MIMD parallel machine with a client-server based system that emulates a
pipeline processor. All communications in our system are initiated by the client
software and remote interface. The server maintains no prior knowledge of the set of
clients and it is assumed that clients can fail arbitrarily.
The overall design of our distributed system was covered in detail in chapter 3.
We followed a modular approach to designing our system because the advantage of a
modular design is that individual modules can be easily tested, changed, swapped or
upgraded at a later stage to offer greater or more efficient functionality with minimum
impact to the rest of the system. Some of the main features of our system include full
platform independence, full protection for machines from subversive distributed
applications, automatic updating of client software, dynamic adjustment of parallel
granularity within running distributed programs, full remote control of the entire
system, ability to run and prioritise several distributed computations simultaneously,
and an easy-to-use programming model. Our system is implemented entirely in Java.
The main reason for choosing Java is the fact that it is platform and network
independent. It should also be noted that by choosing Java, we could easily implement
a strict security policy that guarantees the security of both the server and donor
machines from subversive applications. This was shown in chapter 4 by the
heterogeneity of our deployment of the system. We also provided details of a number
of sample distributed applications and some performance evaluations on various
aspects of the system in chapter 4.
One of the achievements of our programming model (described in chapter 5) is
the simplification of the task of implementing a distributed computation. Our
programming model allows a developer to write their distributed computation at a level
that does not require them to worry about any of the low level details of how to
physically distribute a computation among a heterogeneous set of unreliable semi-idle
processors. This means that a developer can focus on the most important issues relating
101
to determining the best parallelisation scheme for their problem. We believe that the
completion of our distributed bioinformatics application, DSEARCH, demonstrates the
effectiveness of our programming model (see chapter 6).
7.1.3 DSEARCH
Chapter 6 describes our main distributed application developed to run on our system.
DSEARCH is a distributed bioinformatics program that dramatically speeds up the
process of searching large databases using the most sensitive and accurate search
algorithms. We have shown the power of DSEARCH by performing a full performance
analysis including a set of tests that show how that the program scales impressively
with increasing database size over a standard 10 Mbit/s network.
DSEARCH also serves as a good example of a distributed program that takes
advantage of many of the features of our system. A DSEARCH computation can be
classed as a MIMD parallel computation as there are typically several different
computational stages, executing different types of alignment algorithms, operating on
different datasets simultaneously within a single DSEARCH computation. DSEARCH
also takes full advantage of our hybrid communications model as all short control
communications are performed by the system using Java RMI, with all transfer of large
data files (e.g. database sections and groups of query files) performed using regular
Java sockets (to avoid overloading the server’s memory).
7.1.4 Critical Appraisal
One of the main criticisms of our system is that as the system is purely Java based,
applications developed to run on the system may not perform as well as natively
compiled programs. Recently much work has gone into improving the performance of
Java and this has begun to show in certain applications [BSP01]. Although Java
programs may never perform equally to native code, due to the platform independence
of Java bytecode, we believe that a novice user benefits greatly from using a system
like ours as they don’t need to concern themselves with recompiling the system for
different platforms. The other big criticism of our system is that all problems must be
programmed in Java. We acknowledge that this does add to the effort of a developer
that wishes to implement a distributed computation to run on our system. However, the
main reason for limiting our system to Java is that we could easily add strict security
mechanisms to our system, which greatly adds to the appeal and trust of our system
among donors.
102
Although we have shown how DSEARCH can be very effective for speeding
up sensitive database searches, it must be acknowledged that there are many
alternatives to DSEARCH [Hua89, Lop87, ASP95, BSW02, DCF03, BPC01, GlC97,
PCB99]. As mentioned in the previous paragraph, the main drawback of using
DSEARCH is that it does not perform as well as similar, natively compiled, programs.
We have acknowledged this in our performance tests in chapter 6 and noted that
DSEARCH addresses this by providing a user with the ability to run DSEARCH over a
network of cross-platform machines, without the need to recompile the program for
each target platform. The other drawback of DSEARCH is that it does not offer some
of the search algorithms offered by other parallel programs [BSW02, DCF03]. It will
require a substantial amount of work to add these algorithms to DSEARCH as we are
unaware of any current Java implementation of these search algorithms.
7.2 Future Research and Development
In chapter 5 we described the general characteristics that a suitable problem should
exhibit to be most suitable for a distributed computing implementation. We hope that
other researchers will be interested in installing our system and subsequently
developing more distributed applications to run with the system. We also hope that
other developers will be interested in taking our design and source code and expanding
the capabilities of the system. We have made the source code and Java compiled files
available from the system webpage [Gen04]. Upon completion of this project, we
identified a number of specific areas of the system that warrant further research and
development.
7.2.1 Distributed System
The main area of the system that we would like to see further research and development
is in scheduling algorithms. The computational power offered by a large distributed
system with thousands of processors is comparable to any supercomputer, therefore it is
extremely important that this computational power is allocated and managed
effectively. It is the task of the scheduling algorithm to manage the allocation of the
computational resources among a set of running distributed applications. A poor
scheduling algorithm can lead to poor performance for the applications running in the
system and erode the computational advantages offered by a distributed system.
Therefore we would like to see more research into the development and integration of
adaptive and predictive scheduling algorithms.
103
The scalability of the system is another area that we have identified as a future
area of research and development. The current version of the distributed system is
limited to operating in a strict client-server environment where there is only one server
and multiple clients. As the number of clients increases, there will be a certain point at
which the server will become overloaded. One solution is to add more bandwidth or
computational power to the server but this is not a sustainable solution. It is hoped that
future work will focus on investigating various possibilities to overcome this limitation
in our current system. One possibility our group at NUI Maynooth is currently
investigating is to incorporate multiple peer servers into the system [PKN04]. These
servers could be given the ability to communicate in a peer-to-peer manner in order to
share the overall management workload evenly.
7.2.2 Bioinformatics
We have identified the area of bioinformatics as an area of science that provides a rich
set of problems that are suitable for distributed computing. In chapter 6, we described
how we have already implemented a distributed bioinformatics application for
performing rigorous searches of large sequence databases. We can identify two areas
where DSEARCH could be further expanded, namely support for the many other
sequence database formats and the addition of other search algorithms (such as the
heuristics in [AGM90, LiP85]).
Now that we have completed the development of the distributed system and an
initial set of distributed applications, it is hoped that future work will focus on the
development and implementation of an entire library of distributed bioinformatics
applications that will allow bioinformatics researchers to easily take advantage of the
idle computational power in their institutions. There are several areas of bioinformatics
which may also be suited to distributed computing such as microarray data clustering7
and principal component analysis8. One area of bioinformatics that we have already
identified as very suitable for distributed computing is phylogenomics9 (as noted in
[SHB01, SSV02, StL03]). This is an area that we are currently investigating and our
initial analysis of the suitability of this problem is positive.
7.2.3 Grid Computing
7
Statistical analysis of microarray data in order to find expressed genes and cluster genes [Ste03]
A common technique for analysis of multivariate data such as gene expression data [BDG02]
9
Phylogenomics is the study of the evolutionary relationships of organisms [Eis98]
8
104
It is undeniable that grid computing is the way forward in large scale distributed
computing. We believe that this should be embraced by distributed computing
developers and users alike. Indeed the leap from distributed computing to grid
computing has already been made by another similar distributed computing project
[TTL03]. As grid computing matures in the future, we see our system fitting in as an
application that could run on top of a grid, allowing users to easily harness the
computing resources offered by the grid. However, we still see a unique place for our
system, possibly on the smaller or more local scale, to take advantage of processor
resources that for reasons of unreliability or not being dedicated (thus being too
difficult to schedule), will be passed over by the grid community.
105
References
[AES97] Acharya, A., Edjlali, G., and Saltz, J. (1997) The utility of exploiting idle
workstations for parallel computing, Proceedings of the 1997 ACM
SIGMETRICS International Conference on Measurement & Modeling of
Computer Systems, 225-236, Seattle, Washington, United States, ISBN0897-9190-92
[Akl85]
Akl, S.G. (1985) Parallel Sorting Algorithms, Academic Press, ISBN- 01204768-00
[Ale99]
Allen, M. (1999) Do-it-yourself climate prediction, Nature, 401, 642
[AGM90] Altschul, S. F., Gish, W., Miller, W., Myers, E. W., and Lipman, D. J.
(1990) Basic local alignment search tool, Journal of Molecular Biology,
215, 403-410
[Amd67] Amdahl, G.M. (1967) Validity of the Single Processor Approach to
Achieving Large-Scale Computing Capabilities, In AFIPS Conference
Proceedings, 30, 483-485, AFIPS Press, Reston, Va
[ABG97] Anderson, E., Brooks, J., Grassl, C., and Scott, S. (1997) Performance of the
CRAY T3E multiprocessor, Proceedings of the 1997 ACM/IEEE conference
on Supercomputing, 1-17, San Jose, CA, ISBN-0897-9198-58
[ASP95] Archambaud, D., Saraiva, I., and Penne, J. (1995) Systolic implementation
of Smith and Waterman algorithm on a SIMD co-processor, Algorithms and
Parallel VLSI Architectures III, 155-166
[Arm98]
Armstrong, E., (1998) HotSpot: A new breed of virtual machine, Javaworld,
accessed July 2004
<http://www.javaworld.com/jw-03-1998/jw-03-hotspot.html>
[ABD95] Attiya, H., Bar-Noy, A., and Dolev, D. (1995) Sharing memory robustly in
message-passing systems, Journal of ACM, 42(1), 124-142
[AtP99]
Attwood, T.K and Parry- Smith, D.J. (1999) Introduction to bioinformatics,
Prentice Hall, ISBN-0582-3278-81
[BaA00]
Bairoch, A., and Apweiler, R. (2000). The SWISS-PROT protein sequence
database and its supplement TrEMBL in 2000, Nucleic Acids Research, 28,
45-48
[BKL00] Benson, D.A., Karsch-Mizrachi, I., Lipman, D.J., Ostell, J., Rapp, B.A., and
Wheeler, D.L. (2000) GenBank, Nucleic Acids Research, 28, 15-18
[Ber04]
Berkeley Open Infrastructure for Internet Computing, accessed July 2004
< http://boinc.berkeley.edu/>
[BDG02] Berrar, D.P., Dubitzky, W., Granzow, M. (2002) A Practical Approach to
Microarray Data Analysis, Kluwer Academic Publishers, ISBN-1402-072600
106
[BSW02] Bjornson, R.D., Sherman, A.H., Weston, S.B., Willard, N., and Wing, J.
(2002) TurboBLAST: A Parallel Implementation of BLAST Built on the
TurboHub, International Parallel and Distributed Processing Symposium:
IPDPS Workshops, 183
[BCF95] Boden, N.J., Cohen, D., Felderman, R.E., Kulawik, A.E., Seitz, C.L.,
Seizovic, J.N., and Su, W.K. (1995) Myrinet: A Gigabit-per-Second Local
Area Network, IEEE Micro, 15(1), 29-36
[BLT93] Boguski, M.S., Lowe, T.M., and Tolstoshev, C.M. (1993) dbEST—
Database for 'Expressed Sequence Tags', Nature Genetics, 4, 332-333
[BPC01] Braun, R., Pedretti, K., Casavant, T., Scheetz, T., Birkett, C., and Roberts,
C. (2001) Parallelization of local BLAST service on workstation clusters,
Future Generation Computer Systems, 17 (6), 745-754
[BCH98] Brenner, S.E., Chothia, C., and Hubbard, T.J.P (1998) Assessing sequence
comparison methods with reliably structurally identified distant
evolutionary relationships, Proceedings of the National Academy of
Sciences USA, 95, 6073-6078
[BDH84] Bitton, D., DeWitt, D.J., Hsaio, D.K., and Menon, J. (1984) A taxonomy of
parallel sorting, ACM Computing Surveys, 16(3), 287-318
[Bre99]
Brent, R.P. (1990) Parallel Algorithms for Integer Factorisation, in Number
Theory and Cryptography (Ed. J. H. Loxton), Cambridge University Press,
26-37, ASIN-0521-3987-70
[Bro72]
Bron, C. (1972) Merge sort algorithm [M1], Communications of the ACM,
15(5), 357-358
[Bru02]
Brustoloni, J. (2002) Protecting electronic commerce from distributed
denial-of-service attacks, Proceedings of the eleventh international
conference on World Wide Web, 553-561, Honolulu, Hawaii, USA, ISBN-158113-449-5
[BSP01]
Bull, J.M, Smith, L.A., Pottage, L., and Freeman, R. (2001) Benchmarking
Java against C and Fortran for Scientific Applications, Proceedings of ACM
Java Grande/ISCOPE Conference, 97-105, ACM Press
[Buy99]
Buyya, R., editor (1999) High Performance Cluster computing:
Architectures and Systems, Prentice Hall, ISBN-0-1301-3784-7
[CaM02] Cappello, P.R. and Mourloukos, D. (2002) CX: A scalable, robust network
for parallel computing, Scientific Programming, 10(2), 159-171
[Cen04]
Centre of Bioinformatics at Peking University, Advanced Sequence
Analysis Documents, accessed October 2004
< http://www.cbi.pku.edu.cn/Doc/SeqAnalysis/ajb2/ajb2.htm>
[CiJ01]
Cirou, B., Jeannot, E. (2001) Triplet: A Clustering Scheduling Algorithm
for Heterogeneous Systems, IEEE ICPP International Workshop on
107
Metacomputing Systems and Applications (MSA'2001), 231-236, Valencia,
Spain
[Cod93]
Coddington, P.D.D. (1993) An Analysis of Distributed Computing Software
and Hardware for Applications in Computational Physics, Proceedings of
the Second International Symposium on High Performance Distributed
Computing (HPDC-2), 179-186, Spokane, WA
[CDK01] Coulouris, G., Dollimore, J., and Kindberg, T. (2001) Distributed Systems Concepts and Design, Addison Wesley, ISBN-0-2016-1918-0
[Cra04]
Crane, M., Private Communication, September 2004
[DCF03] Darling, A.E., Carey, L., Feng, W. (2003) The Design, Implementation, and
Evaluation of mpiBLAST, ClusterWorld Conference & Expo and the 4th
International Conference on Linux Clusters, San Jose, CA, June 2003
[Day78]
Dayhoff, M.O. (1978) Survey of new data and computer methods of
analysis, in M. O. Dayhoff, ed., Atlas of Protein Sequence and Structure,
5(3), 29, National Biomedical Research Foundation, Silver Springs,
Maryland
[Deu96]
Deutsch, P. (1996) DEFLATE Compressed Data Format Specification
version 1.3, RFC-1951, Internet Requests for Comments
[Dis04]
Distributed.net, accessed July 2004
<http://www.distributed.net>
[Don88]
Dongarra, J. (1988) The LINPACK Benchmark: An Explanation,
Proceedings of the 1st International Conference on Supercomputing, 456474, Athens, Greece, ISBN-0387-1899-12
[DFF02] Dongarra, J., Foster, I., Fox, G., Kennedy, K., White, A., Torczon, L., and
Gropp, W. (2002) The Sourcebook of Parallel Computing, Morgan
Kaufmann, ISBN-1558-6087-10
[DAF95] Dubey, P., Adams III, G.B., and Flynn, M. (1995) Evaluating Performance
Tradeoffs Between Fine-Grained and Coarse-Grained Alternatives, IEEE
Transactions on Parallel and Distributed Systems, 6(1), 17-27
[Eis98]
Eisen, J.A. (1998) Phylogenomics: improving functional predictions for
uncharacterized genes by evolutionary analysis, Genome Research, 8(3),
163
[ElB95]
Eldershaw, C., and Brent, R.P. (1995) Integer factorisation on the AP1000,
Proceedings of the Fourth International Parallel Computing Workshop
(PCW'95), 233-242, Imperial College, London
[Ent04]
Entropia™ (2004) PC Grid Computing
<http://www.entropia.com>
[Fig04]
FightAIDS@Home, accessed July 2004
< http://fightaidsathome.scripps.edu/index.html>
108
[Fla02]
Flanagan, D. (2002) Java in a Nutshell (4th Edition), O'Reilly & Associates,
UK, ISBN-0596-0028-31
[Fox91]
Fox, G. (1991) Draft High Performance Fortran Language Specification:
High Performance Fortran Forum, Computer Information Technical, ISBN9992-2463-67
[FAE02] Fleischmann, R.D., Alland, D., Eisen, J.A., Carpenter, L., White, O.,
Peterson, J., DeBoy, R., Dodson, R., Gwinn, M., Haft, D., Hickey, E.,
Kolonay, J. F., Nelson, W.C., Umaymam L.A., Ermolaeva, M., Salzberg,
S.L., Delcher, A., Utterback, T., Weidman, J., Khouri, H., Gill, J., Mikula,
A., Bishai, W., Jacobs, W.R., Venter, J.C. and Fraser, C.M. (2002) WholeGenome Comparison of Mycobacterium tuberculosis Clinical and
Laboratory Strains, Journal of Bacteriology, 184(19), 5479-5490
[Fly66]
Flynn, M.J. (1966) Very high-speed computing systems, Proceedings of the
IEEE, 54, 1901–1909
[Fly72]
Flynn, M.J. (1972) Some Computer Organizations and Their Effectiveness,
IEEE Transactions on Computing, C-21(9), 948-960
[FoK03]
Foster, I. and Kesselman, C. (2003) The Grid: Blueprint for a New
Computing Infrastructure, Morgan Kaufmann, ISBN-1-5586-0933-4
[FAH99] Freeman, E., Arnold, K., and Hupfer, S. (1999) JavaSpaces Principles,
Patterns, and Practice, 1st edition, Addison-Wesley, ISBN-0201-3095-56
[FTL02]
Frey, J., Tannenbaum, T., Livny, M., Foster, I., and Tuecke, S. (2002)
Condor-G: A Computation Management Agent for Multi-Institutional Grids,
Cluster Computing, 5(3), 237-246
[FPW01] Fritsche, K., Power, J. and Waldron, J. (2001) A Java distributed
computation library, Proceedings of the 2nd International Conference on
Parallel and Distributed Computing, Applications and Technologies
(PDCAT2001), 236-243, Taipei, Taiwan
[Gal99]
Galli, D. (1999) Distributed Operating Systems: Concepts and Practice,
Prentice Hall, ISBN-0130-7984-36
[GBD95] Geist, A., Beguelin, A., Dongarra, J., Jiang, W., Manchek, R., Sunderam,
V.S. (1995) PVM: Parallel Virtual Machine - A Users' Guide and Tutorial
for Networked Parallel Computing, MIT Press, ISBN-0262-5710-80
[Gel85]
Gelernter, D. (1985) Generative communication in Linda, ACM
Transactions on Programming Languages and Systems, 7(1), 80-112, ISSN0164-0925
[GBM96] George, D.G., Barker, W.C., Mewes, H.W., Pfeiffer, F., and Tsugita, A.
(1996) The PIR-International Protein Sequence Database, Nucleic Acids
Research, 24, 17-20
109
[Gen04]
General purpose distributed computing homepage, Department of Computer
Science, National University of Ireland Maynooth, accessed July 2004
<http://www.cs.may.ie/distributed>
[Gib03]
Gibbs, W.W. (2003) The unseen genome: gems among the junk, Scientific
American, 289(5), 46-53.
[GKS97] Giloi, W.K., Kessler, M., and Schramm A. (1997) A High-Level, Massively
Parallel Programming Environment and its Realization, Proceedings of
1997 Real World Computing Symposium (RWC'97), RWC-TR-96-001,
Tokyo, Japan
[GlC97]
Glemet, E. and Codani, J. (1997) LASSAP, a LArge Scale Sequence
compArison Package, Computer Applications in the Biosciences, 13(2),
137-143
[GKG03] Grama, A., Kumar, V., Gupta, A., and Karypis, G. (2003) An Introduction
to Parallel Computing: Design and Analysis of Algorithms, Second Edition,
Pearson Addison Wesley, ISBN-0-2016-4865-2
[GLS03] Gropp, W., Lusk, E., and Sterling, T. (2003) Beowulf Cluster Computing
with Linux, Second Edition, MIT Press, ISBN-0262-6927-40
[Gre04]
Green Tea Technologies Inc. (2004) Pure Java P2P Distributed Computing
Software
<http://www.greenteatech.com>
[HMR03] Haumacher, B., Moschny, T., Reuter, J., and Tichy, W.F. (2003)
Transparent Distributed Threads for Java, Proceedings of the 5th
International Workshop on Java for Parallel and Distributed Computing in
conjunction with the International Parallel and Distributed Processing
Symposium (IPDPS 2003), 136, Nice, France, IEEE Computer Society,
ISBN-0769-5192-61
[Hea03]
Heap, D.G. (2003) Taurus – A Taxonomy of Actual Utilization of Real
UNIX and Windows Servers, Technical White Paper GM12-0191, IBM
eServer Library on ibm.com
[HeH92] Henikoff, S. and Henikoff, J. G. (1992) Amino acid substitution matrices
from protein blocks, Proceedings of the National Academy of Sciences, 89,
10915-10919
[HSL03] He, X., Sun, X.-H., and Laszewski, G. (2003) A QoS Guided Scheduling
Algorithm for Grid Computing, Journal of Computer Science and
Technology, Special Issue on Grid Computing, 18(4)
[Hig04]
Higgs, P.G. (2004) Bioinformatics and Molecular Evolution, Blackwell
Publishing, ISBN-1405-1068-32
[HPC02] High Performance Computing for Computational Science (VECPAR 2002):
5th International Conference, Porto, Portugal, June 26-28, 2002, Selected
Papers and Invited Talks, Lecture Notes in Computer Science, SpringerVerlag Heidelberg, Volume 2565, ISSN-0302-9743
110
[Hil86]
Hillis, D. (1986) The Connection Machine, MIT Press, ASIN-0-2620-81571
[Hua89]
Huang, X. (1989) A space-efficient parallel sequence comparison algorithm
for a message passing multi-processor, International Journal of Parallel
Programming, 18 (3), 223-239
[HGP04] Human Genome Project Homepage, U.S. Department of Energy Office of
Science, accessed October 2004
<http://www.ornl.gov/sci/techresources/Human_Genome/home.shtml>
[Hup00]
Hupfer, S. (2000) The Nuts and Bolts of Compiling and Running
JavaSpaces Programs, Java Technical Article, accessed July 2004
<http://java.sun.com/developer/technicalArticles/jini/javaspaces/>
[Hwa87] Hwang, K. (1987) Advanced Parallel Processing with Supercomputer
Architectures, Proceedings of the IEEE, 75, 1348-1379
[Jav04a]
Java Technology (2004) Sun Microsystems, accessed July 2004
<http://java.sun.com>
[Jav04b]
Java Security Homepage, Sun Microsystems, accessed July 2004
<http://java.sun.com/security>
[Jcl04]
JCluster: A Java Parallel Environment, accessed July 2004
< http://vip.6to23.com/jcluster/>
[Joh88]
Johnson, E.E. (1988) Completing an MIMD multiprocessor taxonomy, ACM
SIGARCH Computer Architecture News, 16(3), 44-47
[JoF04]
Joseph, J. and Fellenstein, C. (2004) Grid Computing, Prentice Hall, ISBN0-1314-5660-1
[Jou04]
Journal of the Earth Simulator, The Earth Simulator Center, Japan Agency
for Marine-Earth Science and Technology, Yokohama, Japan, ISSN-13492675, accessed October 2004
<http://www.es.jamstec.go.jp/esc/eng/publications/index.html>
[Jus04]
Justiniano, C. (2004) Tapping the Matrix, O’Reilly OpenP2P.com, accessed
October 2004
<http://www.openp2p.com/pub/a/p2p/2004/04/16/matrix.html>
[KGO97] Kagan, M., Gochman, S., Orenstien, D., and Lin D. (1997) MMX™
Microarchitecture of Pentium® Processors with MMX Technology and
Pentium® II Microprocessors, Intel Technology Journal, Q3’97, Intel
Corporation
[KaB88]
Karp, A.H. and Babb, R.G. (1988) A comparison of 12 parallel Fortran
dialects, IEEE Software, 5(9), 52-67
[KaF90]
Karp, A.H. and Flatt, H.P. (1990) Measuring parallel processor
performance, Communications of the ACM, 33(5), 539-543
111
[KeN04] Keane, T.M. and Naughton, T.J. (2004) DSEARCH: Sensitive database
searching using distributed computing, Bioinformatics, in press
[KAN03] Keane, T., Allen, R., Naughton, T., McInerney, J., and Waldron, J. (2003)
Distributed Java platform with programmable MIMD capabilities, in Guelfi,
N., Astesiano, E., Reggio, G. (Eds.), Scientific Engineering for Distributed
Java Applications, Lecture Notes in Computer Science, 2604, 122-132,
Springer-Verlag Heidelberg, ISBN-3540-0067-96
[KAN02] Keane, T., Allen, R., Naughton, T.J., McInerney, J., and Waldron, J. (2002)
Distributed computing for DNA analysis, in James F. Power and John T.
Waldron, Eds., Recent Advances in Java Technology: Theory, Application,
Implementation, 99-108, Computer Science Press, Trinity College Dublin,
Ireland, ISBN-0954-4145-00
[Kea03]
Keane, T. (2003) Java Distributed System: Developer Manual, Technical
Report NUIM-CS-TR2003-03, Department of Computer Science, National
University of Ireland, Maynooth, Ireland
[Knu98]
Knuth, D. (1998) The Art of Computer Programming: Vol 1-3, Addison
Wesley, ISBN-0201-4854-19
[KWA01] Korpela, E., Werthimer, D., Anderson, D., Cobb, J., and Lebofsky, M.
(2001) SETI@home-Massively Distributed Computing for SETI, IEEE:
Computer Science and Engineering, 3(1), 77-83
[KJA93] Kranz, D., Johnson, K., Agarwal, A., Kubiatowicz, J., and Lim, B.H. (1993)
Integrating message-passing and shared-memory: early experience,
Proceedings of the fourth ACM SIGPLAN symposium on Principles and
practice of parallel programming, 54-63, California, USA, ISSN-0362-1340
[Kra03]
Krawezik, G. (2003) Performance comparison of MPI and three openMP
programming styles on shared memory multiprocessors, Proceedings of the
fifteenth annual ACM symposium on Parallel algorithms and architectures,
118-127, San Diego, California, USA, ISBN-1581-1366-17
[KrV02]
Krieger, E. and Vriend, G. (2002) Models@Home: distributed computing in
bioinformatics using a screensaver based approach, Bioinformatics, 18(2),
315-318
[KAA04] Kulikova, T., Aldebert, P., Althorpe, N., Baker, W., Bates, K., Browne, P.,
Van Den Broek, A., Cochrane, G., Duggan, K., Eberhardt, R., Faruque, N.,
Garcia-Pastor, M., Harte, N., Kanz, C., Leinonen, R., Lin, Q., Lombard, V.,
Lopez, R., Mancuso, R., McHale, M., Nardon, F., Silventoinen, V., Stoehr,
P., Stoesser, G., Tuli, M.A., Tzouvara, K., Vaughan, R., Wu, D., Zhu, W.
and Apweiler, R. (2004) The EMBL Nucleotide Sequence Database,
Nucleic Acids Research, 32, 27-30
[LSS03]
Larson, S.M., Snow, C.D., Shirts, M.R., and Pande. V.S. (2003)
Folding@Home and Genome@Home: Using distributed computing to
tackle previously intractable problems in computational biology, to appear
in Computational Genomics, Richard Grant editor, Horizon Press
112
[Leo00]
Leopold, C. (2000) Parallel and Distributed Computing: A Survey of
Models, Paradigms and Approaches, John Wiley & Sons Inc, ISBN-04713583-12
[Len00]
Lengauer, C. (2000) A Personal, Historical Perspective of Parallel
Programming for High Performance, Communication-Based Systems,
Kluwer Academic Publishers, ISBN-0792-3617-33
[Les02]
Lesk, A.M. (2002) Introduction to Bioinformatics, Oxford University Press,
ISBN-0199-2519-67
[LiP85]
Lipman, D. J. and Pearson, W. R. (1985) Rapid and sensitive protein
similarity searches, Science, 227, 1435-1441
[Lop87]
Lopresti, D.P. (1987) P-NAC: A systolic array for comparing nucleic acid
sequences, Computer, 20(7), 98-99
[LHF00] Loulergue, F., Hains, G., and Foisy, C. (2000) A calculus of functional BSP
programs, Science of Computer Programming, 37, 253-277
[LDC95] Lu, H., Dwarkadas, S., Cox, A.L., and Zwaenepoel, W. (1995) Message
passing versus distributed shared memory on networks of workstations,
Proceedings of the 1995 ACM/IEEE conference on Supercomputing, 37-es,
San Diego, California, United States
[Mar02]
Markatos, E.P. (2002) Tracing a Large-Scale Peer to Peer System: An Hour
in the Life of Gnutella, 2nd IEEE/ACM International Symposium on Cluster
Computing and the Grid (CCGRID'02), 65, Berlin, Germany
[Mar04]
Marrone, P., Java Object Orientated Neural Engine (Joone), accessed
October 2004
<http://www.jooneworld.com/>
[MVV96] Menezes, A.J., Van Oorschot, P., Vanston, S.A. (1996) Handbook of
Applied Cryptography, CRC Press, ISBN-0-8493-8523-7
[MiG97] Mintchev, S. and Getov, V. (1997) Towards Portable Message Passing in
Java: Binding MPI, Proceedings of the 4th European PVM/MPI Users'
Group Meeting on Recent Advances in Parallel Virtual Machine and
Message Passing Interface, 135-142, ISBN-3540-6369-78
[Mou04] Mount, D.W. (2004) Bioinformatics: Sequence and Genome Analysis, Cold
Spring Harbor Laboratory Press, ISBN-0879-6971-21
[NPR00] Neary, M.O., Phipps, A., Richman, S., and Cappello, P.R. (2000) Javelin
2.0: Java-Based Parallel Computing on the Internet, Proceedings of 6th
International Euro-Par Conference, 1231-1238, Munich, Germany
[NeW70] Needleman, S.B and Wunsch, C.D. (1970) A general method applicable to
the search for similarities in the amino acid sequences of two proteins,
Journal of Molecular Biology, 48, 443-453
113
[Oak01]
Oaks, S. (2001) Java Security, O’Reilly UK, ISBN-0-5960-0157-6
[OeW93] Oed, W. and Walker, M. (1993) An overview of Cray research computers
including the Y-MP/C90 and the new MPP T3D, Proceedings of the fifth
annual ACM symposium on Parallel algorithms and architectures, 271–272,
ISBN-0-8979-1599-2
[Ope04a] OpenMosix: A Linux kernel extension for single-system image clustering
which turns a network of ordinary computers into a supercomputer,
accessed July 2004
<http://openmosix.sourceforge.net>
[Ope04b] OpenMP Architecture Review Board, OpenMP Specifications, accessed
July 2004
<http://www.openmp.org>
[OpA00] OpenShaw, S. and Abrahart, R.J. (2000) Geocomputation, Taylor & Francis,
ISBN-0748-4090-09
[Ora01]
Oram, A. (2001) Peer-to-Peer: Harnessing the Power of Disruptive
Technologies, O’Reilly and Associates, ISBN-0596-0011-0
[Par04]
Parabon Computation Inc. (2004) Computing outside the box™, accessed
July 2004
<http://www.parabon.com>
[PKN04] Page, A., Keane, T., and Naughton, T.J. (2004) Adaptive Scheduling Across
a Distributed Computation Platform, Proceeding of the 3rd International
Symposium on Parallel and Distributed Computing, 141-149, University
College Cork, Cork, Ireland, ISBN-0-7695-2210-6
[PCB99] Pedretti, K., Casavant, T., Braun, R., Scheetz, T., Birkett, C., and Roberts,
C. (1999) Three complementary approaches to parallelization of local
BLAST service on workstation clusters, Lecture Notes In Computer
Science, Springer-Verlag, 1662, 271-282
[PeS02]
Peterson, J.L. and Silberschatz A. (2002) Operating System Concepts, 6th
edition: XP Version, John Wiley & Sons Inc, ISBN-0471-2627-22
[Pol75]
Pollard, J.M. (1975) A Monte Carlo method for factorisation, BIT, 15, 331334
[Pos81]
Postel, J. (1981) Transmission Control Protocol, RFC-793, Internet
Requests for Comments
[Pos80]
Postel, J.B. (1980) User datagram protocol, RFC-768, Internet Requests for
Comments
[PKJ89]
Prahl, S.A., Keijzer, M., Jacques, S.L., and Welch, A.J. (1989) A Monte
Carlo model of light propagation in tissue, In G. J. Müller and D. H. Sliney,
editors, SPIE Proceedings of Dosimetry of Laser Radiation in Medicine and
Biology, volume IS 5, 102-111
114
[Pre04]
Pressman, R. (2004) Software Engineering: A Practitioner's Approach,
Higher Education, ISBN-0-0712-3840-9
[Pvm04]
PVM: Parallel Virtual Machine, accessed July 2004
<http://www.csm.ornl.gov/pvm/pvm_home.html>
[RLS98]
Raman, R., Livny, M., and Solomon, M. (1998) Matchmaking: Distributed
Resource Management for High Throughput Computing, Proceedings of the
Seventh IEEE International Symposium on High Performance Distributed
Computing, 140-147, Chicago, IL
[RLS03]
Raman, R, Livny, M., and Solomon, M. (2003) Policy Driven
Heterogeneous Resource Co-Allocation with Gangmatching, 12th IEEE
International Symposium on High Performance Distributed Computing
(HPDC'03), 80, Seattle, Washington
[Ree03]
Reed, D.A (2003) Grids, the TeraGrid, and Beyond, Computer, 36(1), 6268, ISSN-0018-9162
[Reg00]
Regler, M. (2000) Data Analysis Techniques for High Energy Physics
Experiments, Cambridge University Press, ISBN-0521-6321-96
[Ren96]
Renaud, P. (1996) Introduction to Client/Server Systems: A Practical Guide
for Systems Professionals, John Wiley & Sons, ISBN-0471-1333-37
[Rob59]
Roberts, S. W. (1959), Control Chart Tests Based on Geometric Moving
Averages, Technometrics, 1, 239 -250
[Rog01]
Rognes, T. (2001) ParAlign: a parallel sequence alignment algorithm for
rapid and sensitive database searches, Nucleic Acids Research, 29, 16471652
[RSW93] Rosti, E., Smirni, E., Wagner, T.D., Apon, A.W., and Dowdy, L.W. (1993)
The KSR1: experimentation and modeling of poststore, Proceedings of the
1993 ACM SIGMETRICS conference on Measurement and modeling of
computer systems, Santa Clara, California, USA, 74-85, ISSN-0-1635-999
[Sap97]
Saphir, W. (1997) A Survey of MPI Implementations, National HPCC
Software Exchange (NHSE) review, 1
[SGD02] Saroiu, S., Gummadi, K.P., Dunn, R.J., Gribble, S.D., and Levy, H.M.
(2002) An Analysis of Internet Content Delivery Systems, Proceedings of
the 5th Symposium on Operating System Design and Implementation, 315327, Boston, Massachusetts
[SSV02] Schmidt, H.A., Strimmer, K., Vingron, M., and von Haeseler, A. (2002)
TREE-PUZZLE: maximum likelihood phylogenetic analysis using quartets
and parallel computing, Bioinformatics, 18, 502-504
[ScB90]
Schwan, K and Bo, W. (1990) “Topologies”—distributed objects on
multicomputers, ACM Transactions on Computer Systems, 8(2), 111-157,
ISSN-0734-2071
115
[Sel74]
Sellers, P.H. (1974) On the theory and computation of evolutionary
distances, SIAM Journal of Applied Mathematics, 26, 787-793
[SMM98] Semke, J., Mahdavi, J., and Mathis M. (1998) Automatic TCP buffer tuning,
ACM SIGCOMM Computer Communication Review, 28(4), 315-323, ACM
Press
[SiG99]
Silberschatz, A. and Galvin, P. (1999) Operating System Concepts, John
Wiley and Sons, ISBN-0471-3641-42
[SkT98]
Skillicorn, D.B. and Talia, D. (1998) Models and languages for parallel
computation, ACM Computing Surveys, 30(2), 123-169, ISSN-0-3600-300
[SmW81] Smith, T.F. and Waterman, M.S. (1981) Identification of common
molecular subsequences, Journal of Molecular Biology, 147, 195-197
[SOH96] Snir, M., Otto, S., Huss-Lederman, S., Walker, D., and Dongarra, J. (1996)
MPI: The Complete Reference, Massachusetts Institute of Technology, MIT
Press, ISBN-0262-6921-63
[SoD95]
Sonnhammer, E.L.L and Durbin, R. (1995) A dot-matrix program with
dynamic threshold control suited for genomic DNA and protein sequence
analysis, Gene, 167, 1-10
[SKT97] Sperber, M., Klaeren, H., and Thiemann, P. (1997) Distributed Partial
Evaluation, Proceedings of the Second International Symposium on Parallel
Symbolic Computation, 80-87, Hawaii
[Ste03]
Stekel, D. (2003) Microarray Bioinformatics, Cambridge University Press,
ISBN-0521-5258-7X
[StL03]
Stamatakis, A.P., and Ludwig, T. (2003) Phylogenetic Tree Inference on PC
Architectures with AxML/PAxML, Proceedings of IPDPS2003 (High
Performance Computational Biology workshop), 157, Nice, France
[SHB01] Stewart, C.A., Hart, D., Berry, D.K., Olsen, G.J., Wernert, E.A., and
Fischer, W. (2001) Parallel implementation and performance of fastDNAml
– a program for maximum likelihood phylogenetic inference, Proceedings
of SC2001, Denver, CO, USA
[Sun01]
Sun Microsystems (2001) The Java HotSpot Virtual Machine, Technical
White Paper, Sun Microsystems, Inc., accessed July 2004
<http://java.sun.com/products/hotspot/docs/whitepaper/Java_HotSpot_WP_
Final_4_30_01.html>
[Sun02]
Sun Microsystems (2002), Java™ Remote Method Invocation Specification,
accessed July 2004
<ftp://ftp.java.sun.com/docs/j2se1.4/rmi-spec-1.4.pdf>
[Tan03]
Tanenbaum, A.E. (2003) Computer Networks, Prentice Hall, ISBN-01303848-87
116
[TTL03]
Thain, D., Tannenbaum, T., and Livny, M. (2003) Condor and the Grid, in
Berman, F., Hey, A., Fox, G. editors, Grid Computing: Making The Global
Infrastructure a Reality, John Wiley
[TGL99] Thakur, R., Gropp, W., and Lusk, E. (1999) On implementing MPI-IO
portably and with high performance, Proceedings of the sixth workshop on
I/O in parallel and distributed systems, 23-32, Atlanta, Georgia, United
States, ISBN-1581-1312-32
[TuV03]
Tupakula, U.K. and Varadharajan, V. (2003) A practical method to
counteract denial of service attacks, Proceedings of the twenty-sixth
Australasian computer science conference on Conference in research and
practice in information technology, 275-284, Adelaide, Australia, ISBN-0909-92594-1
[Uni04]
United Devices Inc. (2004) Grid Computing Solutions, accessed July 2004
<http://www.ud.com>
[VAP00] Vector and Parallel Processing (VECPAR 2000): 4th International
Conference, Porto, Portugal, Selected Papers and Invited Talks, Lecture
Notes in Computer Science, Springer-Verlag Heidelberg, ISBN-3-54041999-3
[Wal99]
Walldo, J. (1999) The Jini architecture for network-centric computing,
Communications of the ACM, 42(7), 76-82
[Wal95]
Walli, S.R. (1995) The POSIX family of standards, StandardView, ACM
Press, 3(1), 11-17, ISSN-1-0679-936
[Wol96]
Woltman, G. (1996) Great Internet Mersenne Prime Search, accessed July
2004
<http://www.mersenne.org>
117
Appendix A
DSEARCH: sensitive database searching using distributed
computing
Keane T.M. and Naughton T.J.∗
Department of Computer Science, National University of Ireland, Maynooth, Ireland
Email: tom.naughton@may.ie
Keywords: database search, sequence analysis, sensitive search, distributed computing,
Java
∗
To whom correspondence should be addressed
118
Abstract
Summary: We present a distributed and fully cross-platform database search program
that allows the user to utilise the idle clock cycles of machines to perform large
searches using the most sensitive algorithms. For those in an academic or corporate
environment with hundreds of idle desktop machines, DSEARCH can deliver a ‘free’
database search supercomputer.
Availability: The software is publicly available under the GNU general public licence
from http://www.cs.may.ie/distributed
Contact: tom.naughton@may.ie
Supplementary Information: Full documentation and a user manual is available from
http://www.cs.may.ie/distributed
1 Introduction
Database searching for similar sequences is one of the fundamental tasks in
bioinformatics. The two most rigorous search algorithms are the Needleman-Wunsch
(Needleman and Wunsch, 1970) and Smith-Waterman (Smith and Waterman, 1981)
algorithms. However for large databases it is not feasible to perform searches using
these algorithms. Many heuristic search algorithms have been developed but these
reduce the sensitivity of a search and can fail to detect certain matches. When given the
choice, most biologists would prefer to use the more rigorous algorithms for their
searches.
One way to significantly reduce the runtime of long searches is to parallelise the
search process across multiple processors. Many approaches to parallelising database
searching have been investigated because database searching is both computationally
intensive and easily parallelised. However the overriding problem with many of these
programs is that specialised parallel hardware and software is often required, making
these programs either prohibitively expensive or simply too complicated to set up.
DSEARCH is a fully cross-platform parallel database search program that does not
require any specialised parallel hardware or software.
2 Description
DSEARCH operates in a master-slave environment and the search is parallelised by
splitting the database into fixed sized units that are subsequently searched on the donor
machines. DSEARCH is one of a number of distributed applications that runs on our
general purpose distributed computing platform which provides the user with a remote
interface to monitor the progress of the application as their search executes (Keane,
2004). The parallel granularity is dynamically controlled during each search to match
the processing abilities of the current set of donor machines (Keane, 2004). The user
edits a straightforward configuration file to tailor their computation and chooses one of
the built-in search algorithms (Smith and Waterman, 1981; Needleman and Wunsch,
1970; Crochemore et al., 2003). The inputs to the program are a FASTA database file,
a FASTA query sequences file, a scoring scheme, and a configuration file.
The generality of DSEARCH is demonstrated by the fact that DSEARCH, written in
Java, can run on virtually any architecture and operating system simultaneously while
only using the spare clock cycles of donor machines. No specialised computer
hardware or software is required, and no expense is incurred if idle computing
resources are harnessed. This would not be as straightforward for a parallel application
written in a native language because the application would have to be compiled for
each particular architecture and operating system. We have demonstrated the ease of
119
use and platform heterogeneity of DSEARCH with experiments that utilise the spare
computing resources of several architectures and operating systems simultaneously.
DSEARCH’s main features are:
• Full cross-platform compatibility
• Ability to run several database searches simultaneously
• Remote real-time progress updates via the remote interface
• Full support for arbitrary donor machine failure
3 Deployment and Performance
We have deployed DSEARCH in our university by running it as a low priority
background service in a number of computing laboratories, consisting of approximately
200 desktop PC’s of various modest specifications (Pentium II’s up to Pentium IV’s
running assorted versions of Windows and Linux OSs) and on every node of an IBM
Linux cluster (32 Dual PIII 1 GHz nodes) with all machines connecting via a 10 Mbit/s
network to a single server (Pentium III 500 MHz). We tested the performance of
DSEARCH with SSEARCH (Pearson and Lipman, 1988) by running DSEARCH on a
single processor and searching a 13 Mbyte EST database with a 173 Kbyte set of query
sequences. The comparative runtimes were 1207 minutes and 493 minutes,
respectively. DSEARCH’s performance reduction is overcome by its greater crossplatform compatibility. Figure 1 shows how DSEARCH scales with increasing
numbers of processors. In our tests, DSEARCH reached a speed of 511 million cells
per second on a network of 83 semi-idle workstations (Pentium III 900 MHz).
Acknowledgements
This research has been funded by the Embark Initiative from the Irish Research Council
for Science, Engineering and Technology: funded by the National Development Plan.
We thank Chris Creevey and Andrew Page for their assistance. DSEARCH uses the
NeoBio Java library (http://neobio.sourceforge.net) to perform all comparisons.
References
Crochemore, M., Landau, G., and Ziv-Ukelson, M. (2003) A Subquadratic Sequence
Alignment Algorithm for Unrestricted Scoring Matrices, SIAM Journal of Computing
32 (6), 1654-1673
Keane, T.M. (2004) A General-Purpose Heterogeneous Distributed Computing System,
M.Sc. Thesis, Department of Computer Science, National University of Ireland,
Maynooth
Needleman, S.B and Wunsch, C.D. (1970) A general method applicable to the search
for similarities in the amino acid sequences of two proteins, Journal of Molecular
Biology, 48, 443-453
Pearson, W.R. and D.J. Lipman (1988) Improved Tools for Biological Sequence
Analysis, Proceedings of the National Academy of Sciences, 85, 2444-2448
Smith, T.F. and Waterman, M.S. (1981) Identification of common molecular
subsequences, Journal of Molecular Biology, 147, 195-197
120
Figure Captions
Figure 1. Speedup achieved by DSEARCH over a network of 83 semi-idle machines
121
80
Linear Speedup
DSEARCH
70
Speedup
60
50
40
30
20
10
10
20
30
40
50
Number of Processors
Fig. 1
122
60
70
80
Appendix B: Bioinformatics Glossary
The following is a glossary of many of the bioinformatics terms that are used in chapter
6. This information was collected from the “Human Genome Project” homepage of the
U.S. Department of Energy Office of Science [HGP04].
Amino acid
Any of a class of 20 molecules that are combined to form proteins in living things. The
sequence of amino acids in a protein and hence protein function are determined by the
genetic code.
Base
One of the molecules that form DNA and RNA molecules.
Base pair (bp)
Two nitrogenous bases (adenine and thymine or guanine and cytosine) held together by
weak bonds. Two strands of DNA are held together in the shape of a double helix by
the bonds between base pairs.
Bioinformatics
The science of managing and analyzing biological data using advanced computing
techniques. Especially important in analyzing genomic research data.
BLAST
A computer program that identifies homologous (similar) genes in different organisms.
Deletion
A loss of part of the DNA from a chromosome; can lead to a disease or abnormality.
DNA
The molecule that encodes genetic information. DNA is a double-stranded molecule
held together by weak bonds between base pairs of nucleotides. The four nucleotides in
DNA contain the bases adenine (A), guanine (G), cytosine (C), and thymine (T). In
nature, base pairs form only between A and T and between G and C; thus the base
sequence of each single strand can be deduced from that of its partner.
123
Expressed sequence tag (EST)
A short strand of DNA that is a part of a cDNA molecule and can act as identifier of a
gene. Used in locating and mapping genes.
Genome
All the genetic material in the chromosomes of a particular organism; its size is
generally given as its total number of base pairs.
Insertion
A chromosome abnormality in which a piece of DNA is incorporated into a gene and
thereby disrupts the gene's normal function.
Mutation
Any heritable change in DNA sequence.
Protein
A large molecule composed of one or more chains of amino acids in a specific order;
the order is determined by the base sequence of nucleotides in the gene that codes for
the protein. Proteins are required for the structure, function, and regulation of the
body's cells, tissues, and organs; and each protein has unique functions. Examples are
hormones, enzymes, and antibodies.
124