IBM India
Compiling and Executing Applications on
BlueGene
Subba R Bodda and Ravi K Komanduri
IBM India Systems and Technology Laboratory
© 2007 IBM Corporation
IBM India
Agenda
2
BlueGene
setup
Compiling
applications on BlueGene
Executing
applications on BlueGene
© 2007 IBM Corporation
IBM India
Redbooks
Link: http://www.redbooks.ibm.com/cgi-bin/searchsite.cgi?query=blue+AND+gene













3
Blue Gene System Administration, SG24-7417-00
IBM System Blue Gene Solution: System Administration, SG24-7178-04
IBM System Blue Gene Solution: Configuring and Maintaining Your Environment, SG24-7352-00
Blue Gene/L: Hardware Overview and Planning, SG24-6796-02
IBM System Blue Gene Solution: Hardware Installation and Serviceability, SG24-6743-00
Blue Gene Safety Considerations, REDP-4257-00
Blue Gene/L: Safety Considerations, REDP-3983-01
Blue Gene/L: Performance Analysis Tools, SG24-7278-00
IBM System Blue Gene Solution: Application Development, SG24-7179-04
Unfolding the IBM eServer Blue Gene Solution, SG24-6686-00
GPFS Multicluster with the IBM System Blue Gene Solution and eHPS Clusters, REDP-4168-00
IBM System Blue Gene Solution Problem Determination Guide, SG24-7211-00
Evolution of the IBM System Blue Gene Solution, REDP-4247-00
© 2007 IBM Corporation
IBM India
BlueGene setup





4
bglfen01(10.16.28.30),bglfen02 is the front
end node
bglsn is the service node
Login to bglfen and compile your code
Use mpirun or LL to submit your jobs
bglsn has the service daemons which
would spawn your jobs on actual BG nodes
© 2007 IBM Corporation
IBM India
IBM Compilers for Blue Gene
IBM compilers (Fortran, C, C++) for Blue Gene are located on the frontend
node in /opt/ibmcmp
 Fortran:
/opt/ibmcmp/xlf/bg/10.1/bin/blrts_xlf
/opt/ibmcmp/xlf/bg/10.1/bin/blrts_xlf90
/opt/ibmcmp/xlf/bg/10.1/bin/blrts_xlf95
 C:
/opt/ibmcmp/vac/bg/8.0/bin/blrts_xlc
 C++:
/opt/ibmcmp/vacpp/bg/8.0/bin/blrts_xlC
Note : Users can add the respective directories to their PATH.

5
© 2007 IBM Corporation
IBM India
GNU Compilers for Blue Gene
6

GNU compilers (Fortran, C, C++) for Blue Gene are located in /opt/blrts-gnu/

Fortran:
/opt/blrts-gnu/bin/ powerpc-bgl-blrts-gnu-g77

C:
/opt/blrts-gnu/bin/ powerpc-bgl-blrts-gnu-gcc

C++:
/opt/blrts-gnu/bin/ powerpc-bgl-blrts-gnu-g++
© 2007 IBM Corporation
IBM India
MPI on Blue Gene
7

MPI implementation on Blue Gene is based on MPICH-2 from Argonne
National Laboratory.

Include files mpi.h and mpif.h are at the location:
-I/bgl/BlueLight/ppcfloor/bglsys/include

Libraries to link for MPI are:
-L/bgl/BlueLight/ppcfloorbglsys/lib -lmpich.rts -lmsglayer.rts -lrts.rts ldevices.rts
Note : The linking order for the MPI libraries should be as mentioned above.
© 2007 IBM Corporation
IBM India
Compiler Optimizations
8
© 2007 IBM Corporation
IBM India
Compiler Optimizations
9
© 2007 IBM Corporation
IBM India
IBM ESSL libraries

10
The IBM ESSL (Engineering Scientific Subroutine Library) is located in
/opt/ibmmath/essl/4.2/
© 2007 IBM Corporation
IBM India
Executing jobs using mpirun
cd /gpfs/fcHome/scratch/userid
mpirun –np 32 –partition R000 -cwd /gpfs/fcHome/scratch/ -exe
/gpfs/fcHome/scratch/hello_c
 Common options:
-args : List of arguments to the executables in double quotes
-env : List of environment variables in double quotes. “VARIABLE=value”
-mode : CO or VN (coprocessor/virtual-node)
-only_test_protocol : This options is used only to test communication protocol
 coprocessor mode : one MPI process per node, 1 GB limit per process.
Default mode
 virtual-node mode : two MPI processes per node, 512 MB limit per process; L3
cache, memory, networks, are shared


11
For more details type the following at a command prompt : mpirun -h
Note : A user can have only one job per Blue Gene partition
© 2007 IBM Corporation
IBM India
Executing jobs using submit_job












12
Login to bglsn
cd /bgl/BlueLight/ppcfloor/bglsys/bin
./mmcs_db_console
mmcs$ allocate M03B_J214_64
mmcs$ submitjob M03B_J214_64 /bgl/hello.rts /bgl/
OK
jobId=25
mmcs$ waitjob M03B_J214_64 25
OK
mmcs$ free M03B_J214_64
OK
mmcs$ quit
© 2007 IBM Corporation
IBM India
Environment Variables

Environment variables can be specified using –env “XYZ=123 ABC=456”

Some of the environment variables are





13
BGLMPI_EAGER
BGL_APP_L1_WRITE_THROUGH
BGL_APP_L1_SWOA
BGLMPI_COPRO_SENDS
BGLMPI_INTERRUPT
© 2007 IBM Corporation
IBM India
Redbooks
Link: http://www.redbooks.ibm.com/cgi-bin/searchsite.cgi?query=blue+AND+gene













14
Blue Gene System Administration, SG24-7417-00
IBM System Blue Gene Solution: System Administration, SG24-7178-04
IBM System Blue Gene Solution: Configuring and Maintaining Your Environment, SG24-7352-00
Blue Gene/L: Hardware Overview and Planning, SG24-6796-02
IBM System Blue Gene Solution: Hardware Installation and Serviceability, SG24-6743-00
Blue Gene Safety Considerations, REDP-4257-00
Blue Gene/L: Safety Considerations, REDP-3983-01
Blue Gene/L: Performance Analysis Tools, SG24-7278-00
IBM System Blue Gene Solution: Application Development, SG24-7179-04
Unfolding the IBM eServer Blue Gene Solution, SG24-6686-00
GPFS Multicluster with the IBM System Blue Gene Solution and eHPS Clusters, REDP-4168-00
IBM System Blue Gene Solution Problem Determination Guide, SG24-7211-00
Evolution of the IBM System Blue Gene Solution, REDP-4247-00
© 2007 IBM Corporation
IBM India
Summary







15
Compilation
IBM XLF/XLC and GNU Compilers
ESSL Libraries
submit_job
mpirun
LL
Environment Variables
© 2007 IBM Corporation