Redbooks Paper
Xinghong He
Installing and using MPICH, MPICH-GM,
and MPICH-MX on Linux systems
This paper discusses simple scripts to build and install message passing interface (MPI)
packages (MPICH, MPICH-GM, MPICH-MX) using PATHSCALE compilers on EM64T
systems running RHEL3 and on Opteron systems running SLES9. It also shows how to build
MPI applications using other compilers (from GNU GCC, PGI, and Intel®) without rebuilding
the MPICH, MPICH-GM, and MPICH-MX libraries.
About the author
Xinghong He is an HPC specialist at the IBM xSeries and HPC Benchmark Center in
Poughkeepsie, New York. He has over 10 years of experience in parallel and high
performance computing. He holds a Ph.D. degree in Theoretical and Computational Physics
from Queens University of Belfast. His areas of expertise include applications porting and
performance tuning for AIX and Linux. He has written extensively on performance
comparisons and analyses.
© Copyright IBM Corp. 2006. All rights reserved.
ibm.com/redbooks
1
Introduction
MPICH was developed at the Argonne National Lab. MPI stands for message passing
interface, and the CH represents the chameleon layer that was originally written for portability
between systems.1 Refer to this Web site for more background information:
http://www-unix.mcs.anl.gov/mpi/mpich/
It is one of the most frequently used library packages for High Performance Computing (HPC)
on Linux® clusters. Installation and configuration of MPICH or its variants (such as
MPICH-GM and MPICH-MX from Myricom) are often among the first tasks in building a Linux
HPC cluster, which usually has more than one compiler installed. Due to compatibility issues
among compilers, especially the I/O part of the Fortran compilers, one common practice is to
install an MPICH for each compiler and store them in different places on the system. This
makes it difficult to maintain these libraries on systems with multiple compilers.
Therefore, in this paper, we present simple scripts to compile and install MPICH, MPICH-GM,
and MPICH-MX using one compiler—the PATHSCALE compiler. The resulting libraries can
still be used by other compilers to build user applications. Since both Opteron and EM64T
systems support 32-bit and 64-bit address space for applications, we cover both 32-bit and
64-bit builds. The building matrix is listed in Table 1.
Table 1 Packages to be built and installed using PATHSCALE compilers
Package
EM64T/RHEL3
Opteron/SLES9
32-bit
64-bit
32-bit
64-bit
MPICH-1.2.7
x
x
x
x
MPICH2-1.0.2p1
x
x
x
x
MPICH-GM- 1.2.6..14b
x
x
x
x
MPICH-MX- 1.2.6..0.94
*N/A
*N/A
x
x
* N/A: No systems available for test.
We also show how to compile MPI applications using different compilers through functions
provided by MPICH scripts (mpicc, mpiCC, mpif77, and mpif90) that are created
automatically during the installation process. The following compilers will be used for this
purpose:
򐂰
򐂰
򐂰
򐂰
PATHSCALE compilers Version 2.1
GNU GCC compilers Versions 3.2.3 and 3.3.3
Intel compilers Version 8.1
PGI compilers Version 6.0-1
In addition, we discuss performance considerations, customizations to tackle bigger problem
sizes, and the use of shared libraries.
1
2
http://www-unix.mcs.anl.gov/mpi/mpich/faq.htm#mpichname
Installing and using MPICH, MPICH-GM, and MPICH-MX on Linux systems
Compatibility of compilers
Compilers from vendors differ from each other in many aspects. Here our focal point is
compatibility with each other. We want to make sure that object code created by one compiler
can be used by another compiler. We need to examine the interface level of subroutines and
functions:
򐂰 Data types
򐂰 Global names of object symbols
򐂰 Fortran90 modules
Sizes of basic C data types
MPICH is written in C. The sizes of basic C data types play a critical role in compatibility
concerns. In fact, checking data types is one of the basic things done by the configure script
of most software packages (including MPICH). If sizes of some data types, especially the
frequently used data types from two compilers, do not match, then the object code created by
one compiler cannot be used by the other. This issue can usually be solved by carefully
choosing either compiler options or macro definitions. Table 2 lists the sizes of basic C data
types of some compilers.
Table 2 Lengths in bytes of basic C data types of compilers under 32-bit and 64-bit addressing spaces
32-bit
64-bit
GNU
PATHSCALE
Intel
PGI
GNU
PATHSCALE
Intel
PGI
char
1
1
1
1
1
1
1
1
short
2
2
2
2
2
2
2
2
int
4
4
4
4
4
4
4
4
long
4
4
4
4
8
8
8
8
long int
4
4
4
4
8
8
8
8
long long
8
8
8
8
8
8
8
8
float
4
4
4
4
4
4
4
4
double
8
8
8
8
8
8
8
8
long double
12
12
12
8
16
16
16
8
pointers
4
4
4
4
8
8
8
8
From Table 2, we see that:
򐂰 GNU, PATHSCALE, and Intel compilers are the same in all cases.
򐂰 The PGI compiler differs from the rest only in long double, where PGI is always 8-bytes
long (the same as double), while the other compilers are 12-bytes in 32-bit mode and
16-bytes in 64-bit mode.
Fortunately, long double is not used by MPICH. Therefore, default compiler options from each
one of the compilers listed in Table 2 are compatible with each other when it comes to data
type sizes.
Installing and using MPICH, MPICH-GM, and MPICH-MX on Linux systems
3
Table 2 on page 3 holds true for both systems under discussion:
򐂰 The EM64T system running RHEL3 (Linux kernel 2.4.21-15.EL)
򐂰 The Opteron system running SLES9 (Linux kernel 2.6.5-7.139-smp)
An exception is that the Intel compiler is only available on an EM64T system.
Options for control addressing spaces
Compiler options control whether a 32-bit or a 64-bit address space is used. They are given in
Table 3.
Table 3 Compiler options to control address space size
Compiler
32-bit
64-bit
GNU
gcc -m32
gcc -m64
PATHSCALE
pathcc -m32
pathcc -m64
Intel
<32-bit path>/icc
<64-bit path>/icc
PGI
pgcc -tp k8-32
pgcc -tp k8-64
On both the EM64T and the Opteron systems, the default for GNU and PATHSCALE
compilers is 64-bit. Intel compilers are only available for the EM64T system and separate
installations have to be done for 32-bit and 64-bit versions (as of Version 8.1). The default for
PGI depends on the PATH setting. Nevertheless, the command line option -tp k8-32 or -tp
k8-64 overrides the default.
Fortran name mangling
To prevent name conflicts between Fortran objects and C objects, all four compilers
discussed here do name mangling to Fortran objects. This is done by adding one or two suffix
underscores to Fortran global symbols such as subroutine and function names and common
block names. However, the mangling is done differently from compiler to compiler.
򐂰 The Intel compiler ifort will add one underscore by default. Specifying option -nus turns
this off so that no underscore is added.
򐂰 GNU g77 will, by default, add one underscore to global symbols whose name does not
end with an underscore. For global symbols ending with an underscore, it will add two
underscores. The option -fno-second-underscore will turn off the adding of the second
underscore, so there is always only one underscore added. Another option,
-fno-underscoring, will completely disable name mangling (this is equivalent to -nus in
Intel ifort).
򐂰 PATHSCALE pathf90 acts exactly the same as GNU g77. Even the option names and
default behaviors are the same.
򐂰 PGI pgf77 and pgf90 will, by default, add one underscore to all global symbols,
disregarding the name of the symbols themselves. This behavior is the same as Intel ifort.
However, pgf77 and pgf90 can add a second underscore to global symbols whose name
ends with an underscore. This is achieved by using option -Msecond_underscore. As with
GNU g77 and Intel ifort, name mangling can be disabled completely. For PGI this is done
by specifying the option -Mnounderscoring.
4
Installing and using MPICH, MPICH-GM, and MPICH-MX on Linux systems
To summarize, suppose we have two Fortran subroutines whose names are suba and subb_
(note the ending character is an underscore). We have summarized the name mangling in
Table 4.
Table 4 Name mangling of Fortran global symbols suba and subb_ by different compilers and options
Compiler
suba
subb_
Compiler options
Max
underscores
addeda
g77
suba_
subb___
Default
2
pathf90
suba_
subb___
Default
2
pgf77b
suba_
subb__
Default
1
ifort
suba_
subb__
Default
1
g77
suba
subb_
-fno-underscoring
0
g77
suba_
subb__
-fno-second-underscore
1
pathf90
suba
subb_
-fno-underscoring
0
pathf90
suba_
subb__
-fno-second-underscore
1
pgf77
suba
subb_
-Mnounderscoring
0
pgf77
suba_
subb___
-Msecond_underscore
2
ifort
suba
subb_
-nus
0
a. The last column is the maximum number of underscores added.
b. pgf90 (not listed) does the same as pgf77.
We can see that complexities and confusions are mainly caused by differences in how to treat
global symbols ending with an underscore (subb_ in the table). Intel ifort does not add two
underscores. In order to be compatible with Intel ifort, one can choose to add an underscore
unanimously or not to add an underscore at all. We chose to add one in our building scripts,
as you will see later.
Fortran90 modules
Fortran90 modules created by different compilers are incompatible with each other. This
means that MPICH built by one compiler cannot be used by another compiler to build
Fortran90 applications if the Fortran application code makes use of MPI Fortran90 modules
(use mpi statements in Fortran90 source code). However, for Fortran90 applications that do
not use MPI Fortran90 modules, the MPICH libraries can be used by different compilers.
Fortunately, the vast majority of Fortran90 applications use MPI libraries in the same way as
Fortran77 applications do; that is, they use the statement include mpif.h rather than use mpi.
Fortran77 and C MPI applications do not have this issue.
Installing and using MPICH
MPICH-1.2.7 is the last release of MPI1 from ANL. Users have been advised to migrate to
MPI2 for the additional features, such as MPI I/O and one-sided communications. However,
we expect that MPI1 (MPICH-1.2.7) will continue to be used for years.
In this section we discuss MPICH-1.2.7 limitations. Some of the limitations affect performance
and others restrict the problem sizes to be solved. We also provide simple scripts to install
Installing and using MPICH, MPICH-GM, and MPICH-MX on Linux systems
5
MPICH-1.2.7 and MPICH2-1.0.2p1 on EM64T and Opteron systems. Then we show how to
build user MPI applications using compilers that are not used in building MPICH libraries.
Customizing for performance and large messages
MPICH-1.2.7 has some limits that do not fully utilize today's computing power. When running
MPI applications using MPICH, two environment variables (P4_GLOBMEMSIZE and
P4_SOCKBUFSIZE) often need to be set to large values. P4_SOCKBUFSIZE has a default
value of 16 KB. Normally one needs to increase its value to at least 64 KB (65535) for optimal
performance. The default value of this parameter can be changed through the configuration
script (see the installation sections for details) and source modifications.
The default value for P4_GLOBMEMSIZE is 4 MB, which is very low for many real-world
applications. When this value is too small, the application program stops, giving an error
message and suggesting a larger value. However, one cannot set an arbitrary high value
since there is an upper limit of 256 MB that is hard coded in the MPICH-1.2.7 source code.
To increase these values, one can modify the mpid/ch_p4/p4/lib/p4_MD.h file under the
MPICH-1.2.7 source tree. The default value is defined by GLOBMEMSIZE in the header file
within SYSV_IPC macro (line 460). The maximum allowed shared memory is determined by
two parameters, P4_SYSV_SHM_SEGSIZE on line 453 and P4_MAX_SYSV_SHMIDS on
line 468. They are defined to be 1*1024*1024 and 256, respectively, resulting in a total of 256
MB. Either of these two macros can be increased to allow a bigger max value. We chose to
increase P4_SYSV_SHM_SEGSIZE to 16*1024*1024.
Shared memory segments are not only resources for the MPICH implementation, but also
resources for the system. Linux kernel 2.4 has a default maximum segment size of 32 MB, a
default maximum number of segments of 4096 (system wide), and a default maximum shared
memory of 8 GB. These can be altered, either by recompiling the kernel or modifying system
resource configuration files.
The maximum size of a single message in the MPICH-1.2.7 implementation is 256 MB. It is
defined on line 24 of file mpid/ch_p4/p4/lib/p4_sr.h by macro P4_MAX_MSGLEN. This can
also be increased to accommodate larger messages. We recommend a value of 1 GB.
Installing MPICH
After the necessary changes (patch files are available upon request) to MPICH, we are ready
to do the installation.
MPICH-1.2.7 on EM64T
The script listed in Example 1 builds and installs MPICH-1.2.7 on an EM64T system running
RHEL3 Linux. The PATHSCALE compiler is used to compile the source. This script builds
both 32-bit and 64-bit libraries and installs them in different places specified by $PREFIX.
Example 1 mpich.em64t script to build MPICH-1.2.7 using PATHSCALE compilers on EM64T
#!/bin/sh
pathscale_dir=/vol/linproducts/PSCALE/intel/2.1_rhel3
export PATH=${pathscale_dir}/bin:${PATH}
for mymode in 64 32; do
if [ "$mymode" = "32" ]; then
libsp=${pathscale_dir}/lib/2.1/32
else
libsp=${pathscale_dir}/lib/2.1
fi
6
Installing and using MPICH, MPICH-GM, and MPICH-MX on Linux systems
export LD_LIBRARY_PATH=${libsp}
myprefix=/vol/stcfs/pkgs/mpich/1.2.7/emt${mymode}
mkdir -p $myprefix
mysuffix="-fno-second-underscore"
myopt="-O2"
CC="pathcc -march=auto -m$mymode"
CXX="pathCC -march=auto -m$mymode"
FC="pathf90 -march=auto -m$mymode $mysuffix"
F90="pathf90 -march=auto -m$mymode $mysuffix"
export CC FC CXX F90
export RSHCOMMAND=ssh
extra_libs="-Wl,-rpath,${libsp} -L${libsp} -lpscrt"
./configure --enable-sharedlib --with-device="ch_p4:-socksize=524288" \
--with-comm=shared --disable-devdebug -prefix=$myprefix \
-opt="$myopt" -lib="$extra_libs" > lp1 2>&1
make > lp2 2>&1
make install > lp3 2>&1
make distclean
mv lp1 lp2 lp3 $myprefix/
cp $0 $myprefix/
done
A few notes about the script
Take note of the following:
򐂰 We used option -fno-second-underscore so that PGI (pgf77 and pgf90) and Intel (ifort)
compilers can use the libraries with their default name mangling options. Without this
option, the libraries created cannot be used by Intel ifort. See “Fortran name mangling” on
page 4 for details.
򐂰 We used option -march=auto so the correct value (em64t) is used instead of the
hard-coded value of opteron. PATHSCALE2.1 will always use opteron as the -march
values, no matter what the actual system is.
򐂰 The PATHSCALE compiler library path is added to the -rpath, so at runtime it is not
necessary to include it in the environment variable LD_LIBRARY_PATH.
򐂰 Options --with-device="ch_p4:-socksize=524288 and --with-comm=shared are for
performance purposes. Without these options, the configure script will use ch_p4 as the
default device, which does not do shared memory communication for MPI tasks on the
same node. It will set a small (16 KB) socket buffer size for communication.
򐂰 For a stand-alone large SMP machine, we recommend the option
-with-device=ch_shmem so shared memory will be used for all communications.
MPICH-1.2.7 on Opteron
With very few modifications (compiler paths and build directories), the script used to build
MPICH-1.2.7 on an EM64T system can be used on an Opteron system. For completeness,
the script is listed in Example 2.
Example 2 mpich.opteron script to build MPICH-1.2.7 using PATHSCALE compilers on Opteron
#!/bin/sh
pathscale_dir=/vol/linproducts/PSCALE/opteron/2.1_sles9_sp2
export PATH=${pathscale_dir}/bin:${PATH}
for mymode in 64 32; do
if [ "$mymode" = "32" ]; then
Installing and using MPICH, MPICH-GM, and MPICH-MX on Linux systems
7
libsp=${pathscale_dir}/lib/2.1/32
else
libsp=${pathscale_dir}/lib/2.1
fi
export LD_LIBRARY_PATH=${libsp}
myprefix=/vol/stcfs/pkgs/mpich/1.2.7/amd${mymode}
mkdir -p $myprefix
mysuffix="-fno-second-underscore"
myopt="-O2"
CC="pathcc -march=auto -m$mymode"
CXX="pathCC -march=auto -m$mymode"
FC="pathf90 -march=auto -m$mymode $mysuffix"
F90="pathf90 -march=auto -m$mymode $mysuffix"
export CC FC CXX F90
export RSHCOMMAND=ssh
extra_libs="-Wl,-rpath,${libsp} -L${libsp} -lpscrt"
./configure --enable-sharedlib --with-device="ch_p4:-socksize=524288" \
--with-comm=shared --disable-devdebug -prefix=$myprefix \
-opt="$myopt" -lib="$extra_libs" > lp1 2>&1
make > lp2 2>&1
make install > lp3 2>&1
make distclean
mv lp1 lp2 lp3 $myprefix/
cp $0 $myprefix/
done
MPICH2-1.0.2p1 on EM64T and Opteron
MPICH2 is a complete rewrite of MPICH1. The overall procedure to compile and install
MPICH2 is the same as MPICH1, but the capabilities and available options to the configure
script are very different from those of MPICH1. For more details and a more thorough
description, see the MPICH2 documentation at:
http://www-unix.mcs.anl.gov/mpi/mpich/downloads/mpich2-doc-user.pdf
Example 3 provides a script to build the package on an Opteron system. For EM64T systems,
only the pathscale_dir needs to be modified.
Example 3 mpich2.opteron script to build MPICH-1.2.7 using PATHSCALE compilers on Opteron
#!/bin/sh
pathscale_dir=/vol/linproducts/PSCALE/opteron/2.1_sles9_sp2
export PATH=${pathscale_dir}/bin:${PATH}
for mymode in 64 32; do
if [ "$mymode" = "32" ]; then
libsp=${pathscale_dir}/lib/2.1/32
else
libsp=${pathscale_dir}/lib/2.1
fi
export LD_LIBRARY_PATH=${libsp}
myprefix=/vol/stcfs/pkgs/mpich2/1.0.2p1/amd${mymode}
mkdir -p $myprefix
mysuffix="-fno-second-underscore"
myopt="-O2"
CC="pathcc -march=auto -m$mymode $myopt"
CXX="pathCC -march=auto -m$mymode $myopt"
8
Installing and using MPICH, MPICH-GM, and MPICH-MX on Linux systems
FC="pathf90 -march=auto -m$mymode $myopt $mysuffix"
F90="pathf90 -march=auto -m$mymode $myopt $mysuffix"
LDFLAGS="-Wl,-rpath,${libsp} -L${libsp} -lpscrt"
export CC FC CXX F90 LDFLAGS
export RSHCOMMAND=ssh
./configure -prefix=$myprefix --with-device="ch3:ssm" \
--enable-fast --enable-sharedlibs=gcc > lp1 2>&1
make > lp2 2>&1
make install > lp3 2>&1
mv lp1 lp2 lp3 $myprefix/
cp $0 $myprefix/
done
MPICH2 has removed many options from the configure script compared with MPICH1. Some
seemingly trivial changes we have made from MPICH1-2.7 to MPICH2-1.0.2p1 are not trivial
at all. For example, options -lib= and -opt= are no longer there. The option --enable-sharedlib
has become --enable-sharedlibs=gcc.
Using MPICH with multiple compilers
MPICH provides wrapper scripts (mpicc, mpiCC, mpif77, mpif90, and so on) to simplify the
task of compiling applications. These wrappers take care of paths for finding MPI header files
and library files. They also provide convenient ways to select different compilers other than
the one used in compiling MPICH libraries. There are two easy ways to do this:
򐂰 Using command-line options
򐂰 Using environment variables
Choosing compilers with command-line options
To use MPICH with another compiler, it is necessary to add the corresponding MPICH PATH
and compiler PATH.
64-bit
The following examples are for 64-bit MPICH, and it is assumed that the proper PATH has
been set.
Example 4 A typical setting
export PATH=/vol/stcfs/pkgs/mpich/1.2.7/emt64/bin:\
/vol/linproducts/PSCALE/intel/2.1_rhel3/bin:\
/vol/linproducts/PGI/6.0-1.em64t/linux86-64/6.0/bin:\
/vol/linproducts/intel/8.1_em64t/bin:${PATH}
Example 5 To compile applications with the native (here PATHSCALE) compiler
mpif77 app.f
mpif90 app.f90
mpicc app.c
mpiCC app.C
Example 6 To compile using GNU (no f90)
mpif77 -fc="g77 -fno-second-underscore" app.f
mpicc -cc=gcc app.c
mpiCC -CC=g++ app.C
Installing and using MPICH, MPICH-GM, and MPICH-MX on Linux systems
9
Example 7 To compile using PGI
mpif77 -fc=pgf77 app.f
mpif90 -f90=pgf90 app.f90
mpicc -cc=pgcc app.c
mpiCC -CC=pgCC app.C
Example 8 To compile using Intel compilers (only available on EM64T)
mpif77 -fc=ifort app.f
mpif90 -f90=ifort app.f90
mpicc -cc=icc app.c
mpiCC -CC=icpc app.C
In the case of the Intel compiler, we recommend that this option be added so that the system
will not complain when it cannot find the Intel shared libraries:
-Wl,-rpath,/vol/linproducts/intel/8.1_em64t/lib
32-bit
In the case of 32-bit, one needs to use a 32-bit MPICH PATH and a 32-bit Intel compiler
PATH explicitly since the address space (being 32-bit or 64-bit) cannot be selected from a
simple option such as -m32 or -m64 for GNU and PATHSCALE compilers. So the path setting
is as shown in Example 9.
Example 9 Path setting
export PATH=/vol/stcfs/pkgs/mpich/1.2.7/emt32/bin:\
/vol/linproducts/PSCALE/intel/2.1_rhel3/bin:\
/vol/linproducts/PGI/6.0-1.em64t/linux86-64/6.0/bin:\
/vol/linproducts/intel/8.1/bin:${PATH}
Example 10 To compile applications with the native (here PATHSCALE) compiler
mpif77 app.f
mpif90 app.f90
mpicc app.c
mpiCC app.C
Example 11 To compile using GNU (no f90)
mpif77 -fc="g77 -m32 -fno-second-underscore" app.f
mpicc -cc="gcc -m32" app.c
mpiCC -CC="g++ -m32" app.C
Example 12 To compile using PGI
mpif77 -fc="pgf77 -tp k8-32" app.f
mpif90 -f90="pgf90 -tp k8-32" app.f90
mpicc -cc="pgcc -tp k8-32" app.c
mpiCC -CC="pgCC -tp k8-32" app.C
10
Installing and using MPICH, MPICH-GM, and MPICH-MX on Linux systems
Example 13 To compile using Intel compilers (only available on EM64T)
mpif77 -fc=ifort app.f
mpif90 -f90=ifort app.f90
mpicc -cc=icc app.c
mpiCC -CC=icpc app.C
More options
Choosing compilers from the command-line option causes both compiling and linking to use
the specified compiler. One useful option to watch the process is -show. For example:
mpif77 -fc=ifort -show app.f
This shows what is to be done. It does not do any compiling and linking.
Another option, -shlib (the default is -noshlib), tries to link the user application with shared
MPICH libraries if they are available. This option is honored only by MPICH wrapper scripts
and only affects the linking of MPICH libraries, not the compiler or other system libraries.
Choosing compilers through environment variables
The following environment variables can also be used to choose compilers and linkers:
򐂰
򐂰
򐂰
򐂰
MPICH_CC MPICH_CLINKER for compiling and linking C code.
MPICH_CCC MPICH_CCLINKER for compiling and linking C++ code.
MPICH_F77 MPICH_FLINKER for compiling and linking Fortran77 code.
MPICH_F90 MPICH_F90LINKER for compiling and linking Fortran90 code.
Specifying both compiler and linker to be the same compiler command is equivalent to using
command-line options. For example:
export MPICH_F77=pgf77
export MPICH_FLINKER=pgf77
mpif77 app.f
That is equivalent to:
mpif77 -fc=pgf77 app.f
If only a compiler (for example, MPICH_F77) is specified, then mpif77 will compile using the
specified compiler, but it will link using the built-in linker, which is pathf90 in our case.
The environment variable that controls the use of shared MPICH libraries is
MPICH_USE_SHLIB. Setting it to the value of yes will cause the wrapper scripts (mpicc and
others) to link a user application against the shared MPICH libraries.
Installing and using MPICH, MPICH-GM, and MPICH-MX on Linux systems
11
Installing and using MPICH-GM
The MPICH-GM installation is similar to that of MPICH. The major difference is the choice of
device (ch_gm instead of ch_p4) and the use of Myricom's GM driver. A script to install
MPICH-GM on an EM64T system is given in Example 14.
As in the case of MPICH, this script works on Opteron system. One only needs to modify
paths relating to the compiler and the GM driver (pathscale_dir and gm_dir in the script).
Example 14 mpichgm.em64t script to build MPICH-GM using PATHSCALE compilers on EM64T
#!/bin/sh
gm_dir=/opt/gm/2.0.18-2.4.21-15.EL
pathscale_dir=/vol/linproducts/PSCALE/intel/2.1_rhel3
export PATH=${pathscale_dir}/bin:${PATH}
for mymode in 64 32; do
if [ "$mymode" = "32" ]; then
libp1=${gm_dir}/lib
libsp=${pathscale_dir}/lib/2.1/32
else
libp1=${gm_dir}/lib64
libsp=${pathscale_dir}/lib/2.1
fi
export LD_LIBRARY_PATH=${libsp}:${libp1}
myprefix=/vol/stcfs/pkgs/gmpi/1.2.6..14b/emt${mymode}
mkdir -p $myprefix
mysuffix="-fno-second-underscore"
myopt="-O2"
CC="pathcc -march=auto -m$mymode"
CXX="pathCC -march=auto -m$mymode"
FC="pathf90 -march=auto -m$mymode $mysuffix"
F90="pathf90 -march=auto -m$mymode $mysuffix"
export CC FC CXX F90
export RSHCOMMAND=ssh
export CFLAGS="-I${gm_dir}/include"
extra_libs="-Wl,-rpath,${libp1}:${libsp} -L${libp1} \
-L${libsp} -lpscrt -lgm -lpthread"
./configure --enable-sharedlib --with-device=ch_gm \
--disable-devdebug -prefix=$myprefix -opt="$myopt" \
-lib="$extra_libs" > lp1 2>&1
make > lp2 2>&1
make install > lp3 2>&1
make distclean
mv lp1 lp2 lp3 $myprefix/
cp $0 $myprefix/
done
Using MPICH-GM to compile and link applications is the same as using MPICH. Both
command-line options and environment variables can be used to choose a different compiler.
One basic difference between MPICH-GM and MPICH (Version 1.2.7) is the default setting for
the use of shared MPI libraries. MPICH-GM defaults to using shared libraries, while
MPICH-1.2.7 defaults to using static libraries even though shared libraries are built.
12
Installing and using MPICH, MPICH-GM, and MPICH-MX on Linux systems
Installing and using MPICH-MX
MX (Myrinet Express) is Myricom's low-level message passing software for Myrinet networks.
It is the successor to GM and has a lower latency. Like MPICH-GM, MPICH-MX is Myricom's
MPICH that has MX driver support. In configuring MPICH-MX, a device should be chosen as
ch_mx (as opposed to ch_gm for MPICH-GM and ch_p4 for MPICH). A script that makes use
of the configure script to compile and install MPICH-MX on an Opteron system is shown in
Example 15. Here the PATHSCALE compilers are again used. This script should work on
EMT64T systems as long as PATHSCALE compilers are installed.
Example 15 mpichmx.opteron script to build MPICH-MX using PATHSCALE compilers on Opteron
#!/bin/sh
pathscale_dir=/vol/linproducts/PSCALE/opteron/2.1_sles9_sp2
mx_dir=/opt/mx/mx
export PATH=${pathscale_dir}/bin:${PATH}
for mymode in 64 32; do
libp1=${mx_dir}/lib${mymode}
if [ "$mymode" = "32" ]; then
libsp=${pathscale_dir}/lib/2.1/32
else
libsp=${pathscale_dir}/lib/2.1
fi
export LD_LIBRARY_PATH=${libsp}:${libp1}
myprefix=/vol/stcfs/pkgs/mxmpi/1.2.6..0.94/amd${mymode}
mkdir -p $myprefix
mysuffix="-fno-second-underscore"
myopt="-O2"
CC="pathcc -march=auto -m$mymode"
CXX="pathCC -march=auto -m$mymode"
FC="pathf90 -march=auto -m$mymode $mysuffix"
F90="pathf90 -march=auto -m$mymode $mysuffix"
export CC FC CXX F90
export RSHCOMMAND=ssh
export CFLAGS="-I${mx_dir}/include"
extra_libs="-Wl,-rpath,${libp1}:${libsp} -L${libp1} -L${libsp}\
-lpscrt -lmyriexpress -lpthread"
./configure --enable-sharedlib --with-device=ch_mx --disable-devdebug\
-prefix=$myprefix -opt="$myopt" -lib="$extra_libs" > lp1 2>&1
make > lp2 2>&1
make install > lp3 2>&1
make distclean
mv lp1 lp2 lp3 $myprefix/
cp $0 $myprefix/
done
Using MPICH-MX to compile and link applications is the same as with MPICH. Both
command-line options and environment variables can be used to choose a different compiler.
Like MPICH-GM, the MPICH-MX default is to use shared MPI libraries.
Summary
We have shown how to compile and install MPICH-1.2.7, MPICH2-1.0.2p1,
MPICH-GM-1.2.6..14b, and MPICH-MX-1.2.6..0.94 using a PATHSCALE compiler (Version
Installing and using MPICH, MPICH-GM, and MPICH-MX on Linux systems
13
2.1) and how to use the these libraries with other compilers (GNU, PGI, Intel). The
PATHSCALE compiler was chosen for a few reasons, including its growing popularity. It is
claimed to be 100% compatible with GNU compilers and, unlike GNU, it has Fortran90
support. This makes it easier to maintain MPI libraries and thus other scientific libraries that
have dependencies on MPI libraries (for example, FFTW, BLACS, and SCALAPACK).
14
Installing and using MPICH, MPICH-GM, and MPICH-MX on Linux systems
Notices
This information was developed for products and services offered in the U.S.A.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area.
Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM
product, program, or service may be used. Any functionally equivalent product, program, or service that does
not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not give you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.
The following paragraph does not apply to the United Kingdom or any other country where such
provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION
PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR
IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of
express or implied warranties in certain transactions, therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in any
manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the
materials for this IBM product and use of those Web sites is at your own risk.
IBM may use or distribute any of the information you supply in any way it believes appropriate without
incurring any obligation to you.
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to the names and addresses used by an actual business
enterprise is entirely coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrates programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and
distribute these sample programs in any form without payment to IBM for the purposes of developing, using,
marketing, or distributing application programs conforming to IBM's application programming interfaces.
© Copyright International Business Machines Corporation 2006. All rights reserved.
Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by
GSA ADP Schedule Contract with IBM Corp.
15
This document created or updated on June 1, 2006.
®
Send us your comments in one of the following ways:
򐂰 Use the online Contact us review redbook form found at:
ibm.com/redbooks
򐂰 Send your comments in an email to:
redbook@us.ibm.com
򐂰 Mail your comments to:
IBM Corporation, International Technical Support Organization
Dept. HYTD Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400 U.S.A.
Trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States,
other countries, or both:
Eserver®
Eserver®
Redbooks (logo)
xSeries®
™
IBM®
Redbooks™
The following terms are trademarks of other companies:
Intel, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel
Corporation or its subsidiaries in the United States, other countries, or both.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
16
Installing and Using MPICH, MPICH-GM and MPICH-MX on Linux Systems