MILC Code Basics
Carleton DeTar
First presented at Edinburgh EPCC HackLatt 2008
Updated 2013
HackLatt 2008
1
MILC Code Capabilities
•
Molecular dynamics evolution
–
–
–
–
•
Hadron spectroscopy
–
–
–
–
–
•
Staggered mesons and baryons
Clover mesons and baryons
Mixed staggered/clover mesons
Static/light spectroscopy
Quarkonium spectroscopy (S and P-wave)
Current matrix elements
–
–
•
Staggered fermion actions (Asqtad, Fat7, HISQ, etc)
Clover fermion action
Pure gauge
Schroedinger functional
Leptonic decay (fpi, fB, fD)
Semileptonic decay (heavy-light)
Miscellaneous
–
–
–
Topological charge
Dirac matrix eigenvectors and eigenvalues
Nonperturbative renormalization of currents
HackLatt 2008
2
Two views of the code. Powerful and capable.
HackLatt 2008
3
Or a path to confusion and misery.
Mathias Gruenewald: Temptation of St Anthony (1515)
HackLatt 2008
4
Supported File Formats
• Gauge configuration file formats
– MILC, SciDAC (ILDG), NERSC, Fermilab
• Dirac propagator file formats
– USQCD, Fermilab
• Staggered propagator file formats
– USQCD, MILC, Fermilab
HackLatt 2008
5
Supported SciDAC C-Coded
Packages
•
•
•
•
•
•
QIO (I/O)
QMP (Message passing)
QLA (linear algebra – single processor)
QDP/C (linear algebra – data parallel)
QOPQDP (“Level 3” optimized)
(More in the next session)
http://usqcd.jlab.org/usqcd-software/
HackLatt 2008
6
Precision
• Global single or double precision
• Mixed precision in some applications
Portability
• Any scalar machine
• Any MPP machine with MPI
• GPU, Intel Xeon Phi (somewhat)
HackLatt 2008
7
MILC Code Organization
• Application directories
– With compilation targets
• Library directory
– Linear algebra routines
• Shared procedures (“generic”) directories
– Shared across applications
HackLatt 2008
8
MILC Code Organization
• Application directories: examples
– cd ks_imp_dyn (application)
• make su3_rmd (target) (Asqtad R algorithm)
• make su3_spectrum (another target) (staggered
spectroscopy)
– cd ks_imp_rhmc (application)
• make su3_rhmc (Asqtad RHMC algorithm)
• make su3_rhmc_hisq (target) (HISQ algorithm)
– cd clover_invert2 (application)
• make su3_clov (clover spectroscopy, etc.)
HackLatt 2008
9
MILC Code Organization
• Shared procedures directories: examples
– generic
(common to all applications)
– generic_ks
(common to staggered fermion applications)
– generic_wilson
(common to clover and Wilson fermion apps)
HackLatt 2008
10
Building the MILC Code
• Download source
http://www.physics.utah.edu/~detar/milc_qcd.html
•
•
•
•
Unpack
Configure
Build
Check
HackLatt 2008
11
Building the MILC Code
• Unpack
– tar –xvzf milc_qcd-7.7.10-a7.tar.gz
HackLatt 2008
12
Building the MILC Code
• Configure (crude old fashioned!)
– Copy default Makefile to application directory
• cd ks_imp_rhmc
• cp ../Makefile .
– Edit (example in next slide)
• Makefile
• ../libraries/Make_vanilla
• ../include/config.h
HackLatt 2008
13
Editing the Makefile -- examples
#---------------------------------------------------------------------# 2. Architecture
# Compiling for a parallel machine?
#MPP = true
Uncomment this line.
#---------------------------------------------------------------------# 3. Precision
# 1 = single precision; 2 = double
PRECISION = 1
#---------------------------------------------------------------------# 4. Compiler
# Choices include mpicc cc gcc pgcc g++
ifeq ($(strip ${MPP}),true)
CC = /usr/local/mvapich/bin/mpicc
else
CC = gcc
endif
HackLatt 2008
14
Optimization Possibilities
• Build plain MILC version
• Build with SciDAC optimized QOP support
– Requires installing the SciDAC packages. (See
second tutorial and code home page).
• Build with GPU QUDA support
– Requires installing QUDA and QUDA-MILC
packages. (See the code home page)
• Build with OpenMP directives on loops
– Very rudimentary and spotty. May need to edit
some code by hand to insert OMP macros.
HackLatt 2008
15
Otimization choices controlled in the Makefile -- examples
#---------------------------------------------------------------------# 5. Compiler optimization level
OPT = -O3
#---------------------------------------------------------------------# 6. Other compiler optimizations (depending on compiler)
OCFLAGS =
# Compiling with OpenMP?
OMP = true
#---------------------------------------------------------------------# 10. SciDAC package options
WANTQOP = true # turns on all optimized QOPQDP modules
#---------------------------------------------------------------------# 14. GPU/QUDA Options
WANTQUDA
= true # turns on possible QUDA selections
WANT_FN_CG_GPU = true # turns QUDA CG
WANT_FL_GPU = true # QUDA link fattening
WANT_FF_GPU = true # QUDA fermion force term
WANT_GF_GPU = true # QUDA gauge force
ifeq ($(strip ${MPP}),true)
CC = /usr/local/mvapich/bin/mpicc
else
CC = gcc
endif
HackLatt 2008
16
Building and Checking the Code
• Build (for example)
– make su3_rmd
• Check single precision su3_rmd
– make check “PROJS=su3_rmd” “PRECLIST=1”
• Check all targets in this directory
– make check
HackLatt 2008
17
Running the code
• su3_rhmc < inputfile > outputfile
• su3_rhmc inputfile > outputfile
• su3_rhmc inputfile outputfile
HackLatt 2008
18
Sample parameter input (su3_rmd)
prompt 0
nflavors1 2
nflavors2 1
nx 16
ny 16
nz 16
nt 64
iseed 5682304
lattice dimensions
warms 0
trajecs 2
traj_between_meas 1
beta 6.76
mass1 0.05
mass2 0.5
u0
0.8441
microcanonical_time_step 0.02
steps_per_trajectory 4
max_cg_iterations 300
max_cg_restarts 5
molecular dynamics
error_per_site .000005
error_for_propagator .000002
npbp_reps 1
measurements
prec_pbp 1
reload_serial ../../binary_samples/lat.sample.l4444
save_serial_scidac lat.test.scidac
HackLatt 2008
19
Summary
• The MILC code is versatile and portable
• I have given a brief overview of the code
structure
• I have touched on the procedures for
building and running the code
HackLatt 2008
20
Tutorial 1 Goals
•
•
•
•
•
Run precompiled code
Modify the input parameters
Build a different target
Modify the Makefile and build
Download and unpack the code (if time)
HackLatt 2008
21