Using Maali to Efficiently Recompile Software
Post-CLE Upgrades on a CRAY XC System
Chris Bording, Chris Harris and David Schibeci
Pawsey Supercomputing Centre,
Perth, Western Australia
Maali Origins
The Pawsey Supercomputing Centre, nee iVEC
has been one of the fasting growing Petascale
HPC centres measured in FLOPS in the last five
years. Has gone from < 10 Tflops on a single
system to having six systems with 3 Cray XC
systems, including Magnus which is ranked #41
on the Top 500.
In the beginning there was two
EPIC
Fornax
Not-a-Cray
• 800 nodes(9600-cores)
• Intel X5660 Westmere
• 24 GB memory
• QDR infiniband
• Centos 6.3
• Lustre 1.8.8
• PBS Pro 11.3
Not-a-Cray
• 96 nodes (1152 -cores)
• Intel X5650 Westmere
• 72 GB memory
• 7TB local disk / node
• Nvidia Tesla C2075/node
• Centos 6.3
• Lustre 1.8.8
• PBS Pro 11.3
Maali Origins
• Multiple systems with different hardware
configurations.
• More projects to support.
• More researchers with more complex software
applications and libraries to support.
• The Pawsey Supercomputing Centre is going to be
a Petascale centre.
Initial Design Criteria
• Needs to be simple!
• Establish Conventions and Policies that
support those Conventions.
• Reduce staff effort to maintain the the
software stack.
• Greater use of automation.
A brief primer on compiling and
installing software
Download
source
Compile
code
Create an
environment
Module
Configure
code
Install
code
Document
process
Maali Core design
• Maali is a set of BASH scripts designed initially
to allow for the automation of the
Autoconf/Automake process of “configure”,
“make” and “make install”.
• Uses a system-level configuration file that
defines the Maali environment.
Maali Automatically
• Downloads applications with wget and
maintains a repo of source packages.
• Runs the configure/cmake
• make
• make install
• Writes a new environment module file.
• Generates a separate build log file for each
application
Maali System Configure File
Maali System Configuration
Keypair Values
• MAALI_OS=cle52
• MAALI_ROOT=”/ivec/$MAALI_OS/$MAALI_SY
STEM”
• MAALI_SRC=”$MAALI_ROOT/src”
• MAALI_BUILD_DIR=”$MAALI_ROOT/build"
• MAALI_MODULE_DIR="$MAALI_ROOT/modul
efiles"
Maali Build Directory
The MAALI_BUILD_DIR is the keypair value that
defines the absolute path to the directory where
here the package sources are copied to from the
MAALI_SRC directory then uncompressed and
extracted.
• Ideally this should be somewhere in on the
scratch working directory as builds take a lot
of space.
• Not necessary to maintain.
Maali Source Directory
MAALI_SRC is the keypair value that defines the
absolute path to the directory that is the
repository where all the original source
packages maintained as tar-balls and
compressed formats.
• Needs to be on a partition of sufficient size.
• Backed up if possible.
Maali Modulefiles Directory
The MAALI_MODULE_DIR is the absolute path
where Maali will install all new modules.
• The MAALI_MODULE_DIR will need to be
added to the default MODULEPATH.
Maali build – core functions
• maali_build
Function maali_build {
# this is the core function for creating software
# allows late evaluation
MAALI_TOOL_CONFIGURE_EVAL=`eval echo ”$MAALI_TOOL_CONFIGURE"`
cd ”$MAALI_TOOL_BUILD_DIR"
maali_run "./configure --prefix=$MAALI_INSTALL_DIR
$MAALI_TOOL_CONFIGURE_EVAL"
maali_run "make"
maali_run "make install"
}
Maali cmake build- core function
maali cmake build
function maali_cmake_build {
# for tools that use cmake
# allows late evaluation
MAALI_TOOL_CONFIGURE_EVAL=`eval echo "$MAALI_TOOL_CONFIGURE"`
cd "$MAALI_TOOL_BUILD_DIR"
# cmake likes to build in a director of it's own
mkdir "$MAALI_TOOL_NAME-build"
cd "$MAALI_TOOL_NAME-build"
maali_run "cmake -DCMAKE_INSTALL_PREFIX=$MAALI_INSTALL_DIR $MAALI_TOOL_CONFIGURE
$MAALI_CMAKE_PATH"
if [ $DEBUG ]; then
maali_run "make VERBOSE=TRUE"
else
maali_run "make"
fi
maali_run "make install"
}
Maali Build Scripts
Maali uses a build script for each application or
library that defines:
• The environment (PrgEnv-*)
• Dependent environment modules (cray-hdf,
etc)
• The configuration and optimization flags.
• Defines a set “Maali” variables to generate a
new environment module.
Szip build file
##############################################################################
# $Id: szip.maali,v 1.5 2013/10/24 02:48:41 stapops Exp $
##############################################################################
# maali config file for szip
##############################################################################
# specify which compilers we want to build the tool with
MAALI_TOOL_COMPILERS="$MAALI_DEFAULT_COMPILERS"
# URL to download the source code from
MAALI_URL="http://www.hdfgroup.org/ftp/lib-external/$MAALI_TOOL_NAME/$MAALI_TOOL_VERSION/src/$MAALI_TOOL_NAME$MAALI_TOOL_VERSION.tar.gz"
# location we are downloading the source code to
MAALI_DST="$MAALI_SRC/$MAALI_TOOL_NAME-$MAALI_TOOL_VERSION.tar.gz"
# where the unpacked source code is located
MAALI_TOOL_BUILD_DIR="$MAALI_BUILD_DIR/$MAALI_TOOL_NAME-$MAALI_TOOL_VERSION"
# type of tool (eg. apps, devel, python, etc.)
MAALI_TOOL_TYPE="devel"
# for auto-building module files
MAALI_MODULE_SET_LD_LIBRARY_PATH=1
MAALI_MODULE_SET_CPATH=1
MAALI_MODULE_SET_FPATH=1
MAALI_MODULE_SET_FCPATH=1
MAALI_MODULE_WHATIS="SZIP compression software, providing lossless compression of scientific data."
Silo build script key elements
# specify which compilers we want to build the tool with
MAALI_TOOL_COMPILERS="$MAALI_DEFAULT_COMPILERS"
# URL to download the source code from
MAALI_URL="https://wci.llnl.gov/codes/$MAALI_TOOL_NAME/$MAALI_TOOL_NAME-$MAALI_TOOL_VERSION/$MAALI_TOOL_NAME-$MAALI_TOOL_VERSION.tar.gz"
# location we are downloading the source code to
MAALI_DST="$MAALI_SRC/$MAALI_TOOL_NAME-$MAALI_TOOL_VERSION.tar.gz"
# where the unpacked source code is located
MAALI_TOOL_BUILD_DIR="$MAALI_BUILD_DIR/$MAALI_TOOL_NAME-$MAALI_TOOL_VERSION"
# type of tool (eg. apps, devel, python, etc.)
MAALI_TOOL_TYPE="devel"
# tool pre-requisites
MAALI_TOOL_PREREQ="hdf5/1.8.12"
MAALI_TOOL_CONFIGURE='--enable-silex --enable-fortran --enable-shared --withhdf5=$HDF5_DIR/include,$HDF5_DIR/lib'
MAALI_TOOL_CONFIGURE_NOHDF5='--enable-silex --enable-fortran --enable-shared'
# for auto-building module files
MAALI_MODULE_SET_PATH=1
MAALI_MODULE_SET_LD_LIBRARY_PATH=1
MAALI_MODULE_WHATIS="Silo is a library which implements and application programing interface(API) designed for reading and writing a wide variety of scientific data to binary, disk files. The
files that Silo produces and the data within them can be easily shared and exchanged between wholly independently developed applications running on disparate computing platforms”
Silo build file build function
##############################################################################
function maali_build {
cd "$MAALI_TOOL_BUILD_DIR"
MAALI_TOOL_CONFIGURE_EVAL=`eval echo "$MAALI_TOOL_CONFIGURE_NOHDF5"`
maali_run "./configure --prefix=$MAALI_INSTALL_DIR $MAALI_TOOL_CONFIGURE_EVAL"
maali_run "make"
maali_run "make install"
maali_run "make clean"
MAALI_TOOL_CONFIGURE_EVAL=`eval echo "$MAALI_TOOL_CONFIGURE"`
maali_run "./configure --prefix=$MAALI_INSTALL_DIR $MAALI_TOOL_CONFIGURE_EVAL"
maali_run "make"
maali_run "make install"
}
##############################################################################
GMT build file using cmake
# maali config file for GMT
# specify which compilers we want to build the tool with
MAALI_TOOL_COMPILERS="$MAALI_DEFAULT_GCC_COMPILERS"
# URL to download the source code from
MAALI_URL="http://gmt.soest.hawaii.edu/files/download?name=${MAALI_TOOL_NAME}-${MAALI_TOOL_VERSION}-src.tar.bz2"
# location we are downloading the source code to
MAALI_DST="$MAALI_SRC/${MAALI_TOOL_NAME}-${MAALI_TOOL_VERSION}-src.tar.bz2"
# where the unpacked source code is located
MAALI_TOOL_BUILD_DIR="$MAALI_BUILD_DIR/${MAALI_TOOL_NAME}-${MAALI_TOOL_VERSION}"
# type of tool (eg. apps, devel, python, etc.)
MAALI_TOOL_TYPE="devel”
# tool pre-requisites
MAALI_TOOL_PREREQ="cray-netcdf/4.3.2 fftw/3.3.4.0 gdal/1.11.1"
# tool build pre-requisites - not added to the module, only needed for building (loaded after MAALI_TOOL_PREREQ)
MAALI_TOOL_BUILD_PREREQ="cmake/2.8.12.2"
# add additional configure options
MAALI_TOOL_CONFIGURE='-DFFTW3_INCLUDE_DIR=$FFTW_INC -DFFTW3F_LIBRARY=$FFTW_DIR/libfftw3f.a'
# for auto-building module files
MAALI_MODULE_SET_PATH=1
MAALI_MODULE_SET_MANPATH=1
MAALI_MODULE_SET_LD_LIBRARY_PATH='lib64’
MAALI_MODULE_SET_WHATIS="GMT is an open source collection of about 80 command-line tools for the manipulating geographic and Cartesiandata sets (including filtering, trend setting,
gridding,projecting,etc.) and producing PostScript illustrations ranging form simple x0y plots via contour maps to artificially illuminated surfaces adn 3D perspective views; the GMT supplements
add another 40 more specialized and discipline-specific tools."
export CRAYPE_LINK_TYPE='dynamic'
Maali build files with cmake
# tool pre-requisites
MAALI_TOOL_PREREQ="cray-netcdf/4.3.2 fftw/3.3.4.0 gdal/1.11.1"
# tool build pre-requisites - not added to the module, only needed for
building (loaded after MAALI_TOOL_PREREQ)
MAALI_TOOL_BUILD_PREREQ="cmake/2.8.12.2"
# add additional configure options
MAALI_TOOL_CONFIGURE='-DFFTW3_INCLUDE_DIR=$FFTW_INC DFFTW3F_LIBRARY=$FFTW_DIR/libfftw3f.a'
# for auto-building module files
MAALI_MODULE_SET_PATH=1
MAALI_MODULE_SET_MANPATH=1
MAALI_MODULE_SET_LD_LIBRARY_PATH='lib64’
Haswell Sprint/CLE52 upgrade
• Magnus all the compute nodes where
upgraded from Intel Sandybridge to Intel
Haswell.
• Cray Linux Environment upgrade 50 to 52
• Magnus went into production January 2015!
Application Support at the Pawsey
A set of 31 applications/libraries were identified
from the early adopter’s projects on Magnus.
These included:
•
•
•
•
•
•
•
•
•
Siesta
exabayes
mrbayes
Beagle-lib
Cmake
gsl
lapack
xeres-c
ephem
•
•
•
•
•
•
•
•
•
•
gamess
qbox
ncview
Hypre
Gromacs
Amber
Libffi
Glib
Udunits
gts
•
•
•
•
•
•
•
•
•
•
zlib
szip
scons
numpy
scipy
mercurial
Distribute
astropy
d2to1
python
Maali Command
•
•
•
•
•
•
maali –t python –v 2.6.8 –c magnus
maali –t python –v 2.7.9 –c magnus
maali –t scons –v 2.2.0 –c magnus
maali –t numpy –v 1.9.1 –c magnus
maali –t scipy –v 0.11.0 –c magnus
maali –t gsl –v 1.16 –c magnus
Haswell Sprint Outcomes
• Effort was spread out over 2 days with 7
members of the Supercomputing team.
• Majority of packages installed in a couple of
hours.
• The Sprint was a success!
Haswell Sprint review
• Be smarter about using Maali, use current
source repo, use the compute nodes.
• Improve the dependency documentation
• Create a verification and validation tests using
the “must” build packages.
• Validating build outcomes automatically. Did it
install stuff in /bin,/lib,/man correctly?
Maali System Configuration
Keypair Values
• MAALI_OS=cle52
• MAALI_ROOT=”/ivec/$MAALI_OS/$MAALI_SY
STEM”
• MAALI_SRC=”$MAALI_ROOT/src”
• MAALI_BUILD_DIR=”$MAALI_ROOT/build"
• MAALI_MODULE_DIR="$MAALI_ROOT/modul
efiles"
Simple Maali bash
#!/bin/bash
module load maali/1.0b1
maali –t python –v 2.6.8 –c magnus
maali –t scons –v 2.2.0 –c magnus
maali –t numpy –v 1.9.1 –c magnus
maali –t scipy –v 0.11.0 –c magnus
maali –t gsl –v 1.16 –c magnus
Maali Future
• OPEN SOURCE in May 2015
• Improve the module files prereqs and
dependencies and conflicts.
• Implemented on NeCTAR research cloud to
use for bioinformatics pipeline construction.
• Regression testing of build scripts
• Improve wiki page creation.
Acknowledgements
David Schibeci - the lead developer for Maali.
Chris Harris- leading and organizing the Haswell sprint.
Ashley Chew – System admin on Fornax.
The Supercomputing team Rebecca Hartman-Baker, Paul Ryan,
Daniel Grimwood, Charlene Yang, Brian Skjerven,Kevin Stratford,
Moshin Shaikh at the Pawsey Supercomputing Centre.
George Beckett who leads the Supercomputing Team and Neil
Stringfellow the Executive Director at the Pawsey
Supercomputing Centre for their support.
2nd International HPC User Support Tools Workshop
HUST15 at SC15 in Austin
http://hust15.github.io