Grid Technology:
The Rough Guide
Grid Building Hands-On Session
Ashok Adiga, Texas Advanced Computing Center
Victor Bolet, Georgia State University
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Grid Lab Session
• Overview:
– Laptop Environment
– Pacman Install
– Globus Install and Config
– Configure GSI
– Certificates
– Command Line Job Submission
– GridFTP
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Grid Lab Session
• Overview: (cont)
– Ganglia Install
– CondorG
– MyProxy
– Using a Grid Portal: SURAGrid
– Portal Job Submission
– Wrap Up
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Grid Lab Session
• Components
– GridPort
– SimpleCA
– MyProxy
– CondorG
– Globus
– Ganglia
• Hardware
– Portal Server
– CA Server
– Ganglia Server
– MPI Cluster
– Laptops (Grid Nodes)
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Laptop Environment
• Red Hat Enterprise 2.6.11
• Bash shell
– Root Account: login: root password: Unx4me.
– User Account: login: griduser password: griduser
• Log in as griduser
– Right click on desktop and select “Open Terminal” (you
will need 2 terminals)
• Login as root in one of the terminal windows
– In one of the terminals type: su –
– Enter the root password
• Files for workshop are located in /root/workshop
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Network Setup
• Grids are picky about networks
• Host names
– For a host cert, the fully qualified host name must
be in there
– When using services, host names are provided to
call back to send status information
• Firewalls
– GridFTP, status messages can be blocked
• On your laptop:
– Applications menu -> System Settings -> Network
– In the DNS tab, set the Hostname to be
• labXX.tacc.utexas.edu (replace XX with your number)
– File -> Save and then File -> Quit
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Time Setup
• Grid security requires that the times on the
hosts match up well
– Grid-proxy-init
• On a host that thinks it is 11am
• You get a proxy valid from 11am today
– Globusrun to a host
• To a host that thinks that it is 10am
• Does not allow you to run - your proxy isn’t valid
• Check the time on your laptop
– If it isn’t correct:
• Use the date command to correct it
– date [MMDDhhmm.ss]
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Pacman Install
• In the root window, change into the vdt directory:
– cd /usr/local/vdt
• Fetch the pacman tarball:
– wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacmanlatest.tar.gz
• Expand the tarball:
– tar zxvf pacman-latest.tar.gz
• Install Pacman:
– cd pacman-3.15/
– source setup.sh
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Globus Install and Config
• In the root window, change to Globus install directory
– cd /usr/local/vdt
• Fetch Globus package using Pacman: (for the classroom)
– pacman -get http://129.114.2.195/pacman/vdt_136_cache:Globus
• To fetch the Globus package outside the classroom use: (note: this is
usually a 200-300mb download)
– pacman -get http://www.cs.wisc.edu/vdt/vdt_136_cache:Globus
• Installation Prompts:
– Answer ‘n’ to the following:
Where would you like to install CA files?
Choices:
r (root) - install into /etc/grid-security/certificates
(existing CA files will be preserved)
l (local) - install into $VDT_LOCATION/globus/share/certificates
n (no) - do not install
– Answer ‘y’ to all other prompts
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Configure GSI
•
In the root window, change to the install directory:
– cd /usr/local/vdt
•
Configure Globus GSI:
–
–
–
–
–
•
source setup.sh
mkdir /etc/grid-security
gpt-build /root/workshop/globus_simple_ca_bffae137_setup-0.17.tar.gz gcc32dbg
gpt-postinstall (NOTE: Ignore Warning Messages)
/usr/local/vdt/globus/setup/globus_simple_ca_bffae137_setup/setup-gsi -default
Copy Certificates:
– cd /etc/grid-security/certificates
– tar xvf /root/workshop/certs.tar
•
Link the Globus Files
– /root/workshop/gsi_certlinks.sh
– This script does the following:
ln -s /etc/grid-security/certificates /usr/local/vdt/globus/TRUSTED_CA
ln -s /etc/grid-security/certificates /usr/local/vdt/globus/etc/certificates
ln -s /etc/grid-security/globus-host-ssl.conf /usr/local/vdt/globus/etc/globus-host-ssl.conf
ln -s /etc/grid-security/globus-user-ssl.conf /usr/local/vdt/globus/etc/globus-user-ssl.conf
ln -s /etc/grid-security/grid-security.conf /usr/local/vdt/globus/etc/grid-security.conf
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Obtaining Host & User certificates
• simpleCA used as Certificate Authority in
this hands-on session
– Part of Globus Toolkit distribution
– Server side code pre-installed for hands-on
– Client utilities were created and packaged
during simpleCA server install
• globus_simple_ca_bffae137_setup-0.17.tar.gz
– For detailed documentation see:
http://www.globus.org/toolkit/docs/4.0/admin/docbook/ch07.html
• installing and configuring simpleCA server
• signing user & host certificates
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Certificates
• Globus Certificates are stored in 2
places:
– Host / Server Certificates:
• /etc/grid-security/certificates
– User Certificates (private key):
• ~/.globus
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Requesting Certificates
• Generate Host Certificate Request (in root window):
–
–
–
–
source /usr/local/vdt/setup.sh
grid-cert-request –host `hostname`
(answer yes to the warning about fully qualified names)
cp /usr/local/vdt/globus/etc/hostkey.pem /etc/grid-security/
• Generate User Certificate Request (in griduser window):
– source /usr/local/vdt/setup.sh
– grid-cert-request
• Enter your name
• set a passphrase (Need to remember this!!!)
• Certificates are ready to be Signed:
– Inform the Instructor your certificates are ready
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Grid Security Files (local certs)
• Verify policy files and “.0” files (in griduser
window):
– ls /etc/grid-security/certificates
• Generate a proxy certificate:
– grid-proxy-init
– enter pass phrase used to generate certificate
• Verify the certificate details:
– grid-cert-info
– grid-cert-info –subject
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Command Line Job Submission
• Verify your DN is in the current grid-mapfile on
bandera.tacc.utexas.edu
• Test the certificate (GRAM Authentication):
– globusrun –a -r bandera.tacc.utexas.edu
• Try a simple job:
– globus-job-run bandera.tacc.utexas.edu /bin/hostname
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
GridFTP
Create a test file to copy
– echo “1 2 3 4 5” > /tmp/<hostname>numbers.txt
• Copy the file to bandera using GridFTP
– globus-url-copy file:///tmp/<hostname>numbers.txt \
gsiftp://bandera.tacc.utexas.edu/tmp/<hostname>numbers.txt
• See your file on Bandera (instructor to display)
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Ganglia Monitoring Tool
•
•
•
•
Collector Machine: lab01 (preinstalled)
URL: http://129.114.2.195/ganglia
Collector machine runs gmetad service
Grid nodes report to lab01 through local
gmond service
• Server install documentation/download:
– http://ganglia.sourceforge.net
– Version 3.0.1
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Ganglia Install
• Install gmond Service: (in root window)
– rpm –i /root/workshop/ganglia-gmond-3.0.1-1.i386.rpm
• edit /etc/gmond.conf
– Under cluster:
• name=<hostname>
• owner=“Your Name”
– Under udp_send_channel:
• comment out (#) the mcast_join statement
• add a "host = labXX.staff.utexas.edu" statement
– Under udp_recv channel:
• comment out (#) the mcast_join statement
• comment out (#) the bind statement
• Restart gmond
– service gmond restart
• Go to URL and see your node:
– http://129.114.2.195/ganglia
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Condor Install, Config, Startup
•
Installs & configures all of Condor
– We’ll just be using the Condor-G and DAGMan parts
•
In the root window, change to Globus install directory
– cd /usr/local/vdt
•
Fetch Condor package using Pacman: (for the classroom)
– pacman -get http://129.114.2.195/pacman/vdt_136_cache:Condor
•
To fetch the Globus package outside the classroom use: (note: this is a 100mb
download)
– pacman -get http://www.cs.wisc.edu/vdt/vdt_136_cache:Condor
•
Installation Prompts:
– Would you like Condor to run automatically?
• Yes
•
Update root environment so that you can use Condor
– source /usr/local/vdt/setup.sh
•
Start the Condor Master (which starts all other Condor daemons)
– cd /usr/local/vdt/condor/sbin/
– ./condor_master
•
You should see the condor daemons running as the condor user
–
ps -eaf | grep condor
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Using Condor-G
• Working in the griduser window
• Update the environment
– source /usr/local/vdt/setup.sh
• Look at the condor commands in
/usr/local/vdt/condor/bin
– condor_q to look at queued jobs
– condor_submit to submit a condor job
– condor_rm to remove a submitted job
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Simple Condor Submission
• Create a simple condor submit script called,
say, globus-ls.sub containing:
Universe = globus
globusscheduler = bandera.tacc.utexas.edu
Executable = /bin/ls
Arguments = /usr/bin
Transfer_Executable = false
Output = ls.out
Error = ls.err
queue
• Then do a condor_submit globus-ls.sub
• You can do a condor_q to watch your job run
• Can also try to submit jobs to other labXX
systems, if you are in the grid-mapfile there
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
MyProxy
• Install MyProxy client (In root window):
– cd /usr/local/vdt
– pacman -get http://www.cs.wisc.edu/vdt/vdt_136_cache:MyProxy
• MyProxy host name:
– pecos.tacc.utexas.edu
– for MyProxy server installation instructions, see:
http://www.globus.org/toolkit/docs/4.0/security/myproxy/admin-index.html
• Load MyProxy Certificate:
– myproxy-init -s pecos.tacc.utexas.edu -l <portal_username>
* portal_username is the same as your wireless account
– enter User Certificate passphrase
– enter MyProxy password
– Need to Remember this Password!!
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Using a Grid Portal: SURAGrid
• Based on GridPort
– Quick install instructions:
• http://gridport.net/main/quickstart-download.html
• Log into the User Portal:
– https://gridportal.sura.org
– Username: (same as wireless account name – ie lab01)
– Password: (same as username)
• Retrieve MyProxy Certificate:
–
–
–
–
Click on “Proxy Management” tab
Click “Get New Proxy”
Enter username and password used in myproxy-init
Verify subject of MyProxy certificate
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
SURAGrid Portal (cont.)
• File Management tab:
– Select Host: bandera.tacc.utexas.edu
– Manage local files on bandera
• Job Submission tab:
– Host: bandera.tacc.utexas.edu
– Port: 2119 (default)
– Executable: /bin/hostname
– Standard Output: /home/griduser/hostname.out
– Use File Management tab to see if file is there
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX
Wrap Up
Questions?
SURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
December 8 & 9, 2005, Austin, TX