http://www.grid-support.ac.uk
http://www.ngs.ac.uk
The Storage Resource Broker
and the NGS
Slides from Wayne Schroeder, SDSC and
Peter Berrisford, RAL
http://www.nesc.ac.uk/
http://www.pparc.ac.uk/
http://www.eu-egee.org/
Acknowledgements
• This tutorial selects slides from several sources,
specifically from talks given by Wayne Schroeder
(SDSC) and Peter Berrisford (RAL)
Induction to Grid Computing and the NGS
2
Goal
• Introduce use of the SRB for distributed file
management on the NGS
– This is the focus of the practical that follows
• NOTE:
– Wayne Schroeder: “SRB does so much, people tend to
learn subsets and are often unaware of useful features”
– So explore further!
• http://www.sdsc.edu/srb/
• For a full SRB tutorial, see: http://www.niees.ac.uk/events/srb
Induction to Grid Computing and the NGS
3
SRB Projects
•
•
•
•
•
•
•
•
Digital Libraries
–
–
UCB, Umich, UCSB, Stanford,CDL
NSF NSDL - UCAR / DLESE
NASA Information Power Grid
Astronomy
–
–
National Virtual Observatory
2MASS Project (2 Micron All Sky Survey)
Particle Physics
–
–
–
Particle Physics Data Grid (DOE)
GriPhyN
SLAC Synchrotron Data Repository
Medicine
–
Digital Embryo (NLM)
Earth Systems Sciences
–
–
ESIPS
LTER
Persistent Archives
–
–
NARA
LOC
Neuro Science & Molecular Science
–
–
TeleScience/NCMIR, BIRN
SLAC, AfCS, …
Induction
Grid Computing
the NGS
Over
90 toTera
Bytes and
in 16
million files
7
What is SRB?
• Storage Resource Broker (SRB) is a software product
developed by the San Diego Supercomputing Centre
(SDSC).
• Allows users to access files and database objects
across a distributed environment.
• Actual physical location and way the data is stored is
abstracted from the user
• Allows the user to add user defined metadata
describing the scientific content of the information
Induction to Grid Computing and the NGS
9
How SRB Works
• 4 major components:
MCAT
Database
c
d
MCAT
Server
b
e
f
SRB A
Server
SRB B
Server
– The Metadata Catalogue
(MCAT)
– The MCAT-Enabled
SRB Server
– The SRB Storage Server
– The SRB Client
g
a
SRB
Client
Induction to Grid Computing and the NGS
10
SRB Client Tools
• Provide a user interface to send
requests to the SRB server.
• 4 main interfaces:
–
–
–
–
Command line (S-Commands)
MS Windows (InQ)
Web based (MySRB).
Java (JARGON)
• Web Services (MATRIX)
Induction to Grid Computing and the NGS
11
Planned Deployment on
NGS
Disk
Farm
Database Servers @ Manchester
MCAT
DB1
SRB
Server
DB n
MCAT Server @ Manchester
Online Replication
Failover link
User
MCAT Server @ RAL
MCAT
DB1
SRB
Server
DB n
Database Servers @ RAL
SRB server @ Leeds
Resource Driver
Disk
Farm
SRB server @ RAL
Resource Driver
Disk
Farm
SRB server @ Oxford
Resource Driver
SRB server @ HPCX
Resource Driver
Disk Farm
Induction to Grid Computing and the NGS
Disk
Farm
12
Summary
• SRB provides NGS users with
–
–
–
–
–
a virtual filesystem
Accessible from all core nodes and from the “UI” / desktop
(will provide) redundancy – mirrored catalogue server
Replica files
Support for application metadata associated with files
Induction to Grid Computing and the NGS
13
Tutorial Overview
• Use of the Scommands
– Commands for unix based access to srb
– Strong analogy to unix file commands
• Accessing files from multiple (two) sites
Induction to Grid Computing and the NGS
14
Getting Started
• Launch two “putty” connections to pub-234
– One for commands run on pub-234
– One for connecting to grid-data.rl.ac.uk and running
commands from there
• Open browser and follow link from
http://homepages.nesc.ac.uk/~gcw/NGS/
• Follow the instructions from there.
• Your SRB name is the same as your account on
pub-234
Induction to Grid Computing and the NGS
15