Ratko Vasic
NCWCP, April 1 st 2015

• https://svnemc.ncep.noaa.gov/projects/nems/trunk
• Usage:
#>svn co https://svnemc.ncep.noaa.gov/projects/nems/trunk
• Directory structure: trunk: exe/ job_notused/
README
README.GFS
README.NMM
README.TESTS
src/ tests/

Directory structure contd.
src: atmos/ chem/ conf/ configure*
ENS_Cpl/
ESMFVersionDefine_ESMF_3.h
ESMFVersionDefine_ESMF_630rAPI.h
ESMFVersionDefine_ESMF_NUOPC.h
ESMFVersionLogic.h
MAIN_NEMS.F90
makefile module_EARTH_GENERIC_COMP.F90
module_EARTH_GRID_COMP.F90
module_EARTH_INTERNAL_STATE.F90
module_MEDIATOR.F90
module_NEMS_GRID_COMP.F90
module_NEMS_INTERNAL_STATE.F90
atmos: fim/ gen/ gsm/ io/ makefile module_ATM_GRID_COMP.F90
module_ATM_INTERNAL_STATE.F90
nmm/ phys/ post/ share/

Compilation on known machines:
#> cd ~/trunk/src
#> ./configure
Run ./configure with one argument:
'configure 3_wcoss' : ESMF 3.1.0rp2 library on wcoss
'configure 3_zeus' : ESMF 3.1.0r series library
(i.e. ESMF 3.1.0rp2, 3.1.0rp5) on zeus
'configure 3_gaea' : ESMF 3.1.0rp2 library on gaea
'configure 3_yellowstone' : ESMF 3.1.0rp2 library on yellowstone
....................
....................
#> ./configure 6_nmm_zeus
#> make nmm
#> ls –g ../exe total 30548
-rwxr-xr-x 1 ptmp 11801 Mar 28 00:05 mkDepends.pl
-rwxr-xr-x 1 ptmp 31248845 Mar 29 07:02 NEMS.x

• Configure directory
#> cd ~/trunk/src/conf
#> ls configure.nems.Gaea.intel
configure.nems.Gaea.pgi
configure.nems.Jet.ifort
configure.nems.Linux.g95
.......
modules.nems.Gaea_ESMF_NUOPC* modules.nems.wcoss_ESMF_3* modules.nems.wcoss_ESMF_630rAPI_gsm* modules.nems.wcoss_ESMF_630rAPI_nmm*
.......
#> cat modules.nems.Zeus_ESMF_630rAPI_nmm
# This script is responsible for loading modules that are
# compatible with the ESMF 6.3.0r API.
module purge module load mpt intel esmf/6.3.0rp1 netcdf

• Configure file (Snippet form configure.nems.Zeus.intel_12)
SHELL = /bin/sh include $(ESMFMKFILE)
ESMF_INC = $(ESMF_F90COMPILEPATHS)
ESMF_LIB = $(ESMF_F90LINKPATHS) $(ESMF_F90LINKRPATHS) $(ESMF_F90ESMFLINKLIBS)
NETCDF_INC = -I$(NETCDF)/include
NETCDF_LIB = -L$(NETCDF)/lib -lnetcdf
LIBDIR=/contrib/nceplibs/nwprod/lib
NEMSIO_INC = -I${LIBDIR}/incmod/nemsio
BACIO_LIB = -L${LIBDIR} -lbacio_4
EXTLIBS = $(NEMSIO_LIB) \
$(BACIO_LIB) \
FC = ifort -lmpi -openmp -convert big_endian -assume byterecl -mkl=sequential
FPP = -fpp
F77 = ifort -lmpi -openmp -convert big_endian -assume byterecl -mkl=sequential
FREE = -free
FIXED = -fixed
R8 = -r8
FINCS = $(ESMF_INC) $(NEMSIO_INC) $(NETCDF_INC)
TRAPS =
#TRAPS = -g -fno-inline -no-ip -traceback -ftrapuv -fpe0 -ftz -check all -check noarg_temp_created -fp-stack-check
FFLAGS = $(TRAPS) $(FINCS) -fp-model strict
OPTS_NMM = -O3

• checkout model: svn co https://svnemc.ncep.noaa.gov/projects/nems/trunk
• configure: cd ~/trunk/src ; ./configure 6_nmm_zeus
• make: make nmm
• executable: cd ~trunk/exe ; ls NEMS.x

• More groups work on the same project
• Possibility of breaking other’s code
#> cd ~trunk/tests
#> ./rt.sh
Usage: ./rt.sh -c <model> | -f | -s | -l <file> | -m | -h
-c create new baseline results for <model>
-f run full suite of regression tests
-s run standard suite of regression tests
-l runs test specified in <file>
-m compare against new baseline results
-h display this help

Use of regression tests to run the model
#> cd ~trunk/tests
#> more rt.conf
COMPILE | nmm_post | standard | wcoss | 6.3r_nmm |
COMPILE | nmm | standard | zeus | 6_nmm |
RUN | nmm_cntrl | standard | | nmm |
RUN | nmm_nemsio | | | |
RUN | nmm_rest | | | |

Use of regression tests to run the model
#> cd ~trunk/tests
#> more rt.conf
COMPILE | nmm_post | standard | wcoss | 6.3r_nmm |
COMPILE | nmm | standard | zeus | 6_nmm |
RUN | nmm_cntrl | standard | | nmm |
RUN | nmm_nemsio | | | |
RUN | nmm_rest | | | |

Use of regression tests to run the model
#> cd ~trunk/tests
#> more rt.conf
COMPILE | nmm | standard | zeus | 6_nmm |
RUN | nmm_cntrl | standard | | nmm |
#> ./rt.sh –s
Three files: nmm_run nmm_qsub configure_file_01

RUNDIR=_RUND_
SRCDIR=_SRCD_ mkdir -p $RUNDIR cd $RUNDIR cp _RTPWD_/NMMB_reg/input_nmmb_regional.d01_nemsio input_domain_01_nemsio cp _RTPWD_/NMMB_reg/nmmb_rst_01_nio_0024h_00m_00.00s restart_file_01_nemsio cp _RTPWD_/NMMB_reg/boco.* .
cp _RTPWD_/data/* .
cp $SRCDIR/tests/solver_state.txt .
elif [ $SCHEDULER = 'pbs' ]; then cp $SRCDIR/tests/nmm_qsub .
qsub nmm_qsub

#!/bin/sh
#PBS -o out
#PBS -e err
#PBS -N rt_31806_1
#PBS -A nems
#PBS -l procs=48
#PBS -l walltime=00:15:00 export MPI_GROUP_MAX=96 export ESMF_RUNTIME_COMPLIANCECHECK=ON:depth=4 source .../modules.nems
module list cd $PBS_O_WORKDIR echo "Model started: " `date` mpiexec_mpt -prefix " %g:" -np $PBS_NP ./NEMS.x
echo "Model ended: " `date`

##### Grid Decomposition ##### inpes: 05 # Number of compute tasks in the I direction jnpes: 09 # Number of compute tasks in the J direction
##### Fundamental Timestep (seconds) ##### dt_int: 180 # Integer seconds dt_num: 0 # Numerator of fractional second dt_den: 1 # Denominator of fractional second nhours_fcst: 48 # Length of the forecast (hours) minutes_history: 180 # Frequency of history output (minutes) restart: false # True--> A restarted run minutes_restart: 720 # Frequency of restart output (minutes)

(contd.) global: true # True--> Global ; False--> Regional hydro: false # True--> Hydrostatic ; False--> Nonhydrostatic adiabatic: false # True--> Adiabatic ; False--> Diabatic shortwave: rrtm # Radiation schemes longwave: rrtm convection: bmj # Convection scheme microphysics: fer_hires # Microphysics scheme turbulence: myj sfc_layer: myj land_surface: noah
# Turbulence schemes
# Surface layer schemes
# Land surface schemes nemsio_input: false write_hst_bin: true # True--> Write history files in binary format write_hst_nemsio: true # True--> Write history files in NEMSIO format write_rst_bin: true # True--> Write restart files in binary format write_rst_nemsio: true # True--> Write restart files in NEMSIO format

• Adding local variable to internal state:
Add it to a file:
~trunk/src/atmos/nmm/module_SOLVER_INTERNAL_STATE.F90
follow existing example i.e. “MAVAIL”
REAL(kind=KFPT),DIMENSION(:,:),ALLOCATABLE :: MAVAIL
ALLOCATE(int_state%MAVAIL(IMS:IME,JMS:JME))
REAL(kind=KFPT),DIMENSION(:,:),ALLOCATABLE :: XXX
ALLOCATE(int_state%XXX(IMS:IME,JMS:JME))

• Adding variable for history/restart:
1) Add it to a file:
~trunk/src/atmos/nmm/module_SOLVER_INTERNAL_STATE.F90
follow existing example i.e. “ACPREC”
REAL(kind=KFPT),DIMENSION(:,:),ALLOCATABLE :: ACPREC
CALL SET_VAR_PTR(int_state%VARS,NV,'ACPREC'
,int_state%ACPREC ,(/ IMS,JMS /),(/ IME,JME /) )
2) Add in output namelist:
~trunk/tests/solver_state.txt
follow same example (“ACPREC”)
'ACPREC' H R O T 'Accumulated precip (m)'

• While running original run, turn on restart file writing and choose interval in configure_file: minutes_restart: 120 # Frequency of restart output (minutes) write_rst_nemsio: true # True--> Write restart files in NEMSIO format
• Running restarted run: mv nmmb_rst_01_nio_0016h_00m_00.00s restart_file_01_nemsio
In configure file: restart: true # True--> A restarted run
• Submit model as usual, it will find restart file, read in data and continue from that point

• copy file ~trunk/tests/ts_locations.nml to run directory
• choose variables in solver_state.txt :
'PBLH' H R O T 'PBL height (m)'
• ts_locations.nml example:
&TS_LOCATIONS
NPOINTS=2,
POINTS_LON=-106.0, -110.0,
/
POINTS_LAT=54.0, 50.0