A Status Report on the Prototype
NYS Grid: September 21, 2006
Russ Miller
Comp Sci & Eng, SUNY-Buffalo
Hauptman-Woodward Medical Res Inst
Cyberinfrastructure
(e-Science)
n CI, HPC, & CSE are Critical to 21st Century
q Discovery
q Economic Development
q EOT
n Digital Data-Driven Society
n Knowledge-Based Economy
n Mission
q Seamless, Ubiquitous, Secure, Interwoven, Dynamic
mCompute Systems, Storage, Instruments, Sensors
mComputational Methodologies (Algorithms)
mNetworking
mHCI
n Immediate Goals Include
q Develop Software, Algorithms, Portals, Interfaces
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Grid Computing Overview
Data Acquisition
Imaging Instruments
Advanced Visualization
LHC
Analysis
Computational
Resources
Large-Scale Databases
n Coordinate Computing Resources, People, Instruments in Dynamic
Geographically-Distributed Multi-Institutional Environment
n Treat Computing Resources like Commodities
q Compute cycles, data storage, instruments
q Human communication environments
n No Central Control; No Trust
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Grid Services and Applications
Applications
Shake-and-Bake
ACDC-Grid
ACDC-Grid
Computational
Computational
Resources
Resources
High-level Services
and Tools
Globus
Toolkit
MPI
Apache
MPI-IO
C, C++, Fortran, PHP
MySQL
Oracle
NWS
globusrun
Core Services
Metacomputing
Directory
Service
Condor
LSF
Globus
Security
Interface
Local Services
Stork
MPI
PBS
Maui Scheduler
TCP
UDP
GRAM
GASS
RedHat Linux
WINNT
Irix
ACDC-Grid
ACDC-Grid
Data
Data
Resources
Resources
Solaris
Adapted from Ian Foster and Carl Kesselman
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
“Middleware”
n Intermediate Software Layer between Application
Codes and Grid Resources
n Required for applications, users, and resource
providers to operate effectively in a manner
transparent to the user
n Security; Resource Management; Data Access;
Policies; Accounting;
n Globus; Condor
n Checks availability of Resources
q CPUs; Storage; Networking; Render Farms; etc.
n Scheduling / Workload Management System
n Resource Broker
q Evaluates Job and Breaks Up/Submits
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
NSF Middleware Initiative (NMI)
n Develop, improve, and deploy a suite of
reusable software components for use in
national-scale “cyberinfrastructure”.
n APST, Condor, CPM, DataCutter,
DataCutter STORM, Globus Toolkit, GPT,
Gridconfig, GridPort, GridSolve, GSI
OpenSSH, Inca, KX.509/KCA, Look,
MPICH-G2, MyProxy, Network Weather
Service, OpenSAML, PERMIS, PyGlobus,
Shibboleth, SRB Client, UberFTP, and
WebISO (Web Initial Sign-on).
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Grid Issues
n High-Throughput Computing
n Transparent Integration of Data, Computing,
Sensors/Devices, Networking
n Heterogeneous Resources
n Standards (Grid, Data)
n Major User Communities
q High-Energy Physics and Astrophysics
q Medicine and Biological Sciences
q Earth Sciences
n Public Funding Still Critical
n Grids are in their Infancy
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Major Grid Initiatives
n EGEE: Enabling Grids for E-SciencE (European Commision)
q Initial Focus on CERN (5PB of Data/Year)
mHigh-Energy Physics and Life Sciences
q Expanded Focus Includes Virtually All Scientific Domains
q 200 Institutions; 40 Countries
q 20K+ CPUs; 5PB; 25,000 jobs per day!
n OSG (DOE, NSF)
q High-Throughput Distributed Facility
q Open & Heterogeneous
q Biology, Computer Science, Astrophysics, LHC
q 57 Compute Sites; 11 Storage Sites;
q 10K CPUS; 6PB
n TeraGrid (NSF)
q Integrates High-End Resources
q High-Performance (Dedicated) Networks
q 9 Sites; 100TF & 15PB
q 100+ Databases Available
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Open Science Grid
Courtesy of Paul Avery
n EGEE
qFocus on CERN (5EB of Data/Year)
n OSG
n TeraGrid
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Organization of
Cyberinstitute at SUNY-Buffalo
CSNY (Miller)
HPC
CSE
CI
Enabling
(Furlani: CCR)
•Computing
•Data
•Visualization
•Networking
•MultiScale
•Sciences
•Engineering
•Life Sciences
•Media
•Scheduling
•Monitoring
•Virtual Reality
•Programmers
•GUI Design
•Integration
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Proposed Organization of
CyberInstitute of New York State
CINYS
HPC
CSE
CI
Enabling
•Computing
•Data
•Visualization
•Networking
•MultiScale
•Sciences
•Engineering
•Life Sciences
•Media
•Scheduling
•Monitoring
•Virtual Reality
•Programmers
•GUI Design
•Integration
Russ Miller (www.cse.buffalo.edu/faculty/miller)
Don’t Panic – 1 of 4
NYS Cyberinfrastructure @ RPI 9/21/2006
Proposed Organization of
CyberInstitute of New York State
CINYS
NYS Resources
•Computing
•Data Storage
•Networking
•Rendering
•Sensors
•Instruments
CSE
CI
Enabling
•MultiScale
•Sciences
•Engineering
•Life Sciences
•Media
•Scheduling
•Monitoring
•Virtual Reality
•Programmers
•GUI Design
•Integration
Russ Miller (www.cse.buffalo.edu/faculty/miller)
Don’t Panic – 2 of 4
NYS Cyberinfrastructure @ RPI 9/21/2006
Proposed Organization of
CyberInstitute of New York State
CINYS
NYS Resources
•Computing
•Data Storage
•Networking
•Rendering
•Sensors
•Instruments
NYS CSE
•MultiScale
•Sciences
•Engineering
•Life Sciences
•Media
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS CI
•Scheduling
•Monitoring
•Networking
•Security
CINYS Enablers
•Programmers
•GUI Design
•Integration
Don’t Panic – 3 of 4
NYS Cyberinfrastructure @ RPI 9/21/2006
legend
OC-12
GigE
10 GigE
ESnet
DWDM
CA*net
NYSERNet PoP
R&E Network
Abilene
HEAnet
Rochester
Syracuse
Buffalo
Albany
MAN LAN
Abilene
NLR
32 AoA
Courtesy of NYSERNet
Proposed Organization of
CyberInstitute of New York State
ExCom
NYS Resources
•Computing
•Data Storage
•Networking
•Rendering
•Sensors
•Instruments
NYSERNet
CINYS Director
NYS CSE
•MultiScale
•Sciences
•Engineering
•Life Sciences
•Media
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS CI
•Scheduling
•Monitoring
•Networking
•Security
CINYS Enablers
•Programmers
•GUI Design
•Integration
Yell & Scream – 4 of 4
NYS Cyberinfrastructure @ RPI 9/21/2006
Proposed CINYS Budget
n Participants
q Standard Resources
mCompute Systems
mData Storage
mVisualization Devices
mSensors
mInternet-Ready Devices
q Percent FTE Sysadmin
q Faculty Research Groups
q Intellectual Capital
Russ Miller (www.cse.buffalo.edu/faculty/miller)
n New York State
q Seed Funds
q Special Resources
mNetworking
mLarge Data Storage
q Operating Budget
mEnablers/Programmers
mAccess Grid Nodes
mGeneral Operating
¢ Travel
¢ Training
NYS Cyberinfrastructure @ RPI 9/21/2006
Federal Funding Opportunity
n NSF “High-Performance Computing for Science
and Engineering Research and Education:
Operations (User Support, System Administration
and Maintenance)”
q NSF 06-599
q Due Nov 28
q $2-10M/year
q Integrate with TeraGrid
q Min of 5TF Sustained
q Min 50% of Machine
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
NYSGrid Status: Implementation Details
(Jon Bednasz & Steve Gallo)
Getting Started (Build Cluster, etc.)
Resource Manager
Install OSG software stack
Request/install Host certificate
Configure OSG
Local Site-verify.sh
Grid-cat monitoring
ACDC monitoring
Remote Site-verify.sh
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Getting Started
n Physically build a cluster
q 1 head node
q 4+ compute nodes
n Install Cluster Software
q Operating System (Red Hat)
q Drivers for Interconnect (Myrinet, Infiniband, etc.)
q Resource Manager (PBS, LSF, Condor, SGE)
n Identify Gatekeeper Node for OSG Software
q Either stand alone machine or co-resident on Head Node
q 5GB of space in /opt/grid
q 5GB of space in /grid-tmp
n Need to have ability to adjust firewalls
n Need to have ability to add users
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Installing OSG Stack on Gatekeeper
n Installs are done via PACMAN
q wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-3.16.1.tar.gz
n Install OSG software
qpacman -get OSG:ce
n Install (1) Package for your Resource Manager
qpacman -get OSG:Globus-Condor-Setup
qpacman -get OSG:Globus-PBS-Setup
qpacman -get OSG:Globus-LSF-Setup
qpacman -get OSG:Globus-SGE-Setup
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Obtain OSG Secure Certificate
n Request integration of new resource into OSG
n Request host certificate to identify resource
n Approved host certificate is imported into web
browser
n Export host certificate from web browser
n Install host certificate on resource
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Configure OSG
n Configure OSG
qcd $VDT_LOCATION/monitoring
q./configure-osg.sh
n Determine and configure OSG to use range
of firewall ports.
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Local Site-Verify.sh
$ cd $VDT_LOCATION
$ source ./setup.sh
$ grid-proxy-init
Enter "Your Passphrase"
$ cd verify
$ perl site_verify.pl -host=hostname.domain.tld
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Grid-cat Monitor
http://osg-cat.grid.iu.edu/
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
ACDC Monitor
http://osg.ccr.buffalo.edu/operations-dashboard.php?grids=3&vos=10
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Remote Site-Verify.sh
n From another OSG site:
$ cd $VDT_LOCATION
$ source ./setup.sh
$ grid-proxy-init
Enter "Your Passphrase"
$ cd verify
$ perl site_verify.pl -host=hostname.domain.tld
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Status I
n On-Line
q RIT (later today)
q SUNY-Albany
q SUNY-Binghamton
q SUNY-Buffalo
q Syracuse University
n Waiting on Host Cert
q Niagara University
q NYU
n Installing
q University of Rochester
q Cornell University
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Status II
n Building Gatekeeper Machine
qHauptman-Woodward Medical Research
Institute (HWI)
qSUNY-Geneseo
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
Acknowledgments I
n Jon Bednasz, CCR
n Steve Gallo, CCR
n Mark Green, ITR/CCR
n Cathy Ruby, ITR
n Amin Ghadersohi, ITR
n Naimesh Shah, ITR
n Jason Rappleye, CCR
Russ Miller (www.cse.buffalo.edu/faculty/miller)
n Sam Guercio, CCR
n Martins Innus, CCR
n Cynthia Cornelius, CCR
n Tom Furlani, CCR
n NSF, NIH, NYS, NIMA,
NTA, Oishei, Wendt,
DOE
NYS Cyberinfrastructure @ RPI 9/21/2006
Acknowledgments II
n UAlbany: Eric Warnke
n RIT: Rick Bohn
n SU: Jorge González
Outeiriño
n NYU: Chris Grim
n U of R: Bill Webster
n Cornell: Resa Alvord
n Binghamton: Steaphan
Greene
Russ Miller (www.cse.buffalo.edu/faculty/miller)
n Niagara U: Ann Rensel
n HWI: Steve Potter
n Geneseo: Kirk Anne
NYS Cyberinfrastructure @ RPI 9/21/2006
Acknowledgments III
n Cornell (CTC & Administration): Vision, Leadership,
& Guts to bring us all together in the hope that by
working together we, and many others in NYS, will all
prosper.
n Cornell: Hosting Workshop I
n RPI: Hosting Workshop II
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006
miller@buffalo.edu
www.cse.buffalo.edu/faculty/miller
Russ Miller (www.cse.buffalo.edu/faculty/miller)
NYS Cyberinfrastructure @ RPI 9/21/2006