Globus and the eMinerals
project
Mark Calleja (CeSC)
Background
• Model the atomistic processes involved in environmental
issues (radioactive waste disposal, pollution,
weathering).
• Use many different codes/methods: large scale
molecular dynamics, lattice dynamics, ab initio DFT,
quantum Monte Carlo.
• Jobs can last minutes to weeks, require a few Mb to a
few Gb of memory.
• Compute, rather than data intensive.
• Project has 12 postdocs (4 scientists, 4 code developers,
4 grid engineers).
• Spread over a number of sites: Bath, Cambridge,
Daresbury, Reading, RI and UCL.
Minigrid Resources
• Two Condor pools: large one at UCL (~930 Windows
boxes) and a small one at Cambridge (~25
heterogeneous nodes).
• Four Linux clusters running PBS queues.
• An IBM pSeries platform with 24 processors under
LoadLeveller.
• A 4-node Mac X-server cluster, under SGE.
• A number of Storage Resource Broker (SRB) instances,
providing ~3TB of distributed, transparent, storage.
• Application server, including SRB Metadata Catalogue
(MCAT), and database cluster at Daresbury.
• All accessed via Globus (pre-WS).
eMinerals minigrid
Condor
• Department of CS, University of Wisconsin.
• Allows the formation of pools from heterogeneous mix of
architectures and operating systems.
• Excellent for utilising idle CPU cycles.
• Highly configurable, allowing very flexible pool policies
(when to run Condor jobs, with what priority, for how
long, etc.)
• Provides Condor-G, a client tool for submitting jobs to
Globus gatekeepers.
• Also provides DAGMan, a meta-scheduler for Condor
which allows workflows to be built.
Storage Resource Broker (SRB)
• San Diego Supercomputer Center
• SRB is client-server middleware that provides a uniform
interface for connecting to heterogeneous data
resources over a network and accessing replicated data
sets.
• In conjunction with the Metadata Catalog (MCAT),
provides a way to access data sets and resources based
on their attributes and/or logical names rather than their
names or physical locations.
• Provides a number of user interfaces: Scommands
(useful for scripting), Jargon (java toolkit), inQ (Windows
GUI) and MySRB (web browser).
Typical work process
• Start by uploading input data into the SRB using one of the three
client tools:
a) S-commands (command line tools)
b) InQ (Windows)
c) MySRB (web browser)
Data in the SRB can be annotated using the Metadata Editor, and then
searched using the CCLRC DataPortal. This is especially useful for the
output data.
• Construct relevant Condor/DAGman submit script/workflow.
• Launch onto minigrid using Condor-G client tools.
Job workflow
1)
2)
Gatekeeper
SRB
Condor
pool
Condor PBS
JMgr
JMgr
3)
On remote gatekeeper, run a
jobmanager-fork job to create
a temporary directory and
extract input files from SRB.
Submit next node in workflow
to relevant jobmanager, e.g.
PBS or Condor, to actually
perform the required
computational job.
On completion of the job, run
another jobmanager-fork job
on the relevant gatekeeper to
ingest the output data into the
SRB and clean up the
temporary working area.
my_condor_submit
• Needed to hide integration details from average user.
• Hence, provided our own wrapper to these workflows,
called my_condor_submit.
• This takes as its argument an ordinary Condor, or
Condor-G, submit script, but also recognises some SRBspecific extensions.
• Limitations: the SRB extensions currently can’t make use
of Condor macros, e.g. job.$$(OpSys).
• Currently also developing a job submission portal (due
May ‘05).
my_condor_submit
# Example submit script for a remote Condor pool
Universe
Globusscheduler
Executable
Notification
= globus
= lake.esc.cam.ac.uk/jobmanager-condor-INTEL-LINUX
= add.pl
= NEVER
GlobusRSL = (condorsubmit=(transfer_files ALWAYS)(universe vanilla)(transfer_input_files A,
B))(arguments=A B res)
Sget
Sput
Sdir
Sforce
Output
Log
Error
Queue
= A, B
= res
= test
= true
= job.out
= job.log
= job.error
# Or just “Sget = *”
# To turn into a PBS job replace with:
#
# Globusscheduler
= lake.esc.cam.ac.uk/jobmanager-pbs
# GlobusRSL
= (arguments=A B res)(job_type=single)
Experience of setting up a minigrid
• Generally, users seem happy.
• Installing GT hasn’t always been painless, but good for the
soul.
• Job monitoring a bit of an issue: “what’s the state of health
of my week-old simulation?”, especially for Condor. Some
tools provided.
• Load balancing across the minigrid still an issue, though we
have provided some simple tools to monitor the state of the
queues.
• gsissh access only provided for porting code.
• Jobmanager modules have needed tweaking, both for
Condor and for PBS.
Jobmanager tweaks
• Users need different versions of MPI (MPICH, LAM),
built with different compilers.
• Added new attribute mpi_type to
$GLOBUS_LOCATION/share/globus_gram_job_manager/pbs.rvf
• Modified:
$GLOBUS_LOCATION/lib/perl/Globus/GRAM/JobManager/pbs.pm
• E.g. use mpi_type = lam_intel in the Globus_RSL.
• Prior to SRB usage, we had to modify condor.pm in
order for output files to be copied back to Condor submit
node.
Summary and Outlook
• eMinerals minigrid now in full service.
• It can meet vast majority of our requirements (except
maybe for very low latency MPI jobs; e.g. Myrinet).
• Use of my_condor_submit with NGS clusters has proven
to be straightforward.
• The integration of the SRB with the job-execution
components of the minigrid have provided most obvious
added value to the project.
• Next major step will be when the ComputePortal goes
online (May ’05).
• Keeping an eye on WSRF developments and GT4.