Executing
Message-Passing
Programs
Mitesh Meswani
Presentation Outline
 Introduction to Top Gun (eServer pSeries
690)
 MPI on



Top Gun (AIX/Linux)
Itanium2 (Linux) Cluster
Sun (Solaris) Workstation Cluster



Environment Set-up
Compilation
Execution
 Contacts and Web Sites
Introduction to Top Gun
Top Gun is an IBM eServer pSeries 690 (p690)
multiprocessor
System Summary
Architecture
IBM POWER4
OS
AIX/LINUX
CPUs
16
Peak GFLOPS
83.2
Memory
32GB
Disk Storage
2TB
Some features of the p690*
 Logical Partitioning
 Processor – IBM POWER4 dual-core, 1GHz
processor with a fast shared L2 cache
 Self Management – Autonomic computing
tools for self correction and detection of
errors
 AIX Operating System – supports large files,
system partitioning and large main memory
*Source : http://www-1.ibm.com/servers/eserver/pseries/hardware/tour/690_text.html
POWER4 Processor Layout*
D
D
>1GHz CPU
>1GHz CPU
I
I
Shared L2 Cache
L3 Controller
L3 Directory
L3 Cache
FABRIC CONTROLLER
DISTRIBUTED SWITCH
Processor local bus
I/O Bus
Description:
•Two 64-bit processors per chip
•64KB Icache and 32KB Dcache per
processor
• 1.44MB L2 unified cache with a
125GB/s transfer rate per chip
Some POWER4 Features:
• Supports speculative execution
• Supports out-of-order execution
• Each processor is superscalar with 8
functional units: two floating-point units,
two integer units, two load/store units,
one branch prediction unit and one
logical unit
Memory
*Source : http://www-1.ibm.com/servers/eserver/pseries/hardware/whitepapers/power4.html
Multi-Chip Module Layout
P
P
L2
P
P
L2
L3
P
P
L2
P
P
L2
• Each MCM has
• four POWER4 chips, a total of 8 processors.
• a shared 128MB L3 unified cache with max transfer rate of
13.8GB/s
• Max of four MCMs per p690 server/cabinet
• Multiple p690 servers can be connected to form a cluster
32-Processor p690 Configuration
P
P
MCM 0
MCM 1
P
P
P
L2
L2
P
P
L2
P
L2
L3
L3
P
P
P
L2
P
P
P
L2
P
P
L2
MCM 2
MCM 3
P
P
P
L2
P
L2
P
L2
P
P
L2
P
L2
L3
L3
P
P
L2
P
P
L2
P
P
L2
P
P
L2
Projects using Top Gun - 1
 Dr. Stephen Aley, Department of Biological Sciences:
programs for the analysis and data mining of DNA
sequences, analysis of micro array data, and
proteomic analysis by LC/MS-MS.
 Robert Jeffrey Bruntz, Department of Physics:
Analyzing the output of magneto hydrodynamics
(MHD) simulations via visualization enabled by
OpenDX.
 Dr. Michael Huerta, Department of Mechanical and
Industrial Engineering: Modeling shaped charges and
other shock physics problems via hydro code
calculations.
Projects using Top Gun - 2
 Dr. Ramon Ravelo, Department of Physics:
Performing multi-million atom simulations of material
response employing parallel molecular dynamics
algorithms.
 Dr. Ming-Ying Leung, Department of Mathematical
Sciences : Developing efficient algorithms for pseudo
knot prediction in long RNA sequences.
 Dr. Patricia Teller, Diana Villa, and Research
Assistants, Department of Computer Science:
Developing performance-counter based metrics and
techniques to tune application performance,
mathematical models to steer HPC applications,
memory access models of large complex application
to explore sources of performance degradation, and
techniques to dynamically adapt the Linux operating
system.
Software on Top Gun
 System Software: OS - AIX 5.3



Compilers: XL, Visual Age, GCC
Parallel Programming Libraries: MPI, OpenMP, SHMEM
Parallel Run Time Environment: Parallel Operating
Environment, POE
 Other Software






OpenGL 5.2
Loadleveler for AIX 2.0
PAPI
XL Fortran Compiler
Perl 5
Java 1.4.1
Complete list available at:
http://research.utep.edu/Default.aspx?tabid=20686
MPI on Top Gun
Sample hello world MPI program:
#include <stdio.h>
#include <mpi.h>
main(argc, argv)
int argc;
char *argv[];
{
char name[BUFSIZ];
int length;
MPI_Init(&argc, &argv);
MPI_Get_processor_name(name, &length);
printf("%s: hello world\n", name);
MPI_Finalize();
}
MPI on Top Gun
Environment Set-up
1. Create a file .rhosts and type topgun.utep.edu and
save the file in your home directory.
2. Create a host file .hf containing eight lines with the
string topgun.utep.edu. Save this file in your home
directory. Then set the environment variable
MP_HOSTFILE to the absolute path of your .hf file.
Example:
%setenv MP_HOSTFILE /home/mitesh/.hf
3. Define an environment variable called MP_PROCS to
have a default value for number of processors.
Example:
%setenv MP_PROCS 4
MPI on Top Gun
Program Compilation
 Use mpcc
 Example:
%mpcc HelloWorldMPI.c -o hello
This creates an executable called hello.
 Other MPI compilers:


Fortran77: mpxlf
C++: mpCC
 Complete list available at:
http://research.utep.edu/Default.aspx?tabid=2
0687
MPI on Top Gun
Program Execution - 1
 Use poe utility to execute parallel programs.
The poe command invokes the Parallel
Operating Environment (POE) for loading and
executing programs on remote nodes.
 The flags associated with poe:


-hostfile: a file with a list of hosts can be
specified – This will override the .hf file.
-procs: number of MPI tasks to create – This
flag will ignore the MP_PROCS environment
variable.
MPI on Top Gun
Program Execution - 2
 Example with two tasks:
%poe hello -procs 2
%topgun.utep.edu: hello world
%topgun.utep.edu: hello world
 Example with a user specified host file:
%poe hello -hostfile ./host.txt -procs 3
%topgun.utep.edu: hello world
%topgun.utep.edu: hello world
%topgun.utep.edu: hello world
Link for information on IBM’s POE ( Parallel
Operating Environment):
http://www.llnl.gov/computing/tutorials/ibm_sp/#POE
MPI on Itanium2 Cluster
 Vampyre Cluster: 8-processor Intel Itanium2
Cluster, with dual (SMP) 900MHz Itanium2
processors per node
 Network Features: Externally accessible by
100 Mbps Ethernet; internal network runs at
1Gbps
 OS: Linux kernel 2.4.18-e.25
 MPICH
MPI on Itanium2 Cluster
Environment Set-up
 In your home directory create an .rhosts file
that contains the following four lines:
it01.vampyre.cs.utep.edu <account-name>
it04.vampyre.cs.utep.edu <account-name>
it03.vampyre.cs.utep.edu <account-name>
it02.vampyre.cs.utep.edu <account-name>*
•
•
it02 is down and cannot be used for MPI at the moment.
Additionally only two of the three remaining nodes are active
because of routing problems caused by it02 being down.
MPI on Itanium2 Cluster
Program Compilation
 Use mpicc <progname> <prog related flags>
-o <object filename> to compile program
Example:
%mpicc HelloWorldMPI.c -o hello
MPI on Itanium2 Cluster
Program Execution - 1
 Use mpirun <options> to execute your program
 Some mpirun options:
 -machinefile <machine-file name> : Create the list of
possible machines on which to execute from the file
<machine-file name>
 -np <np>: specify the number of processors on which
to execute
 -nolocal: avoids executing MPI tasks on local host
 Example:
%mpirun -np 2 ./hello
%sabina.vampyre.cs.utep.edu: hello world
%clarimonde.cs.utep.edu: hello world
MPI on Itanium2 Cluster
Program Execution - 2
 Example: Create one task while logged into host
sabina and use host clarimonde to execute the
MPI task.
%mpirun –nolocal –np 1 ./hello
%clarimonde.cs.utep.edu: hello world
MPI on Sun Workstation Cluster
Environment Set-up
 Create an .rhosts file with hostname
<username>

Example:
station10 mitesh
station11 mitesh
.
.
station20 mitesh
 Copy the latest .cshrc file for mpi
%cp /usr/local/cshrc/.cshrc.mpi .cshrc
MPI on Sun Workstation Cluster
Program Compilation
 Use mpicc <progname> <prog related flags>
-o <object filename> to compile program
Example:
%mpicc HelloWorldMPI.c -o hello
MPI on Sun Workstation Cluster
Program Execution - 1
 Use mpirun <options> to execute a program
 Some mpirun options:

-machinefile <machine-file name>: Create the list of
possible machines on which to execute from the file
<machine-file name>. The format of machinefile list is
nodename



Example:
station10
station11
-np <np>: specify the number of processors to run on
-nolocal: avoids executing MPI tasks on local host
 Example:
%mpirun -np 3 ./hello
%station11.: hello world
%station13.: hello world
%station12.: hello world
MPI on Sun Workstation Cluster
Program Execution - 2
 Example: Create two tasks while logged into station11 and use
only station12 and station13 to execute the MPI tasks.
%mpirun –nolocal –np 2 ./hello
%station12.: hello world
%station13.: hello world
 Refer to this MPICH Web Site for complete list of mpirun options:
http://www-unix.mcs.anl.gov/mpi/www/www1/mpirun.html
Contacts and Websites
 System Administrators:
 Jose Hernandez (jose@cs.utep.edu) for Top Gun and
Sun Workstation Cluster
 Leopoldo Hernandez (leo@cs.utep.edu) for Itanium2
Cluster
 System Web Sites:
 Top Gun: http://research.utep.edu/topgun
 Itanium2 Cluster:
http://www.cs.utep.edu/~bdauriol/courses/ParallelAndConcur
rentProgramming/vampyre.html
 MPI Links:


http://www.llnl.gov/computing/tutorials/workshops/work
shop/mpi/MAIN.html#References
http://www-unix.mcs.anl.gov/mpi/
Questions?