Adding a Microphysics Variable to wrfout
John D McMillen
This document describes the programming required to extract a local microphysics
variable from within a microphysics scheme in the WRF_ARW and add the local variable to
the standard wrfout variable set. In addition, I will outline the process and coding I used to
extract a local variable. It will also highlight difficulties I encountered while programming
the WRF to perform this task.
Resources
To start, here are some resources I found very useful in understanding the WRF and
my problem in particular.
http://www.mmm.ucar.edu/wrf/users/tutorial/201207/WRF_Registry_and_Examples.pdf
The link above points to a pdf file that describes the Registry and in slides 54-66 provides
step-by-step instructions for creating a new array from existing microphysics data. I
recommend programming this array by following the instructions to help you understand
the WRF data-handling process. (Note I used version 3.4 of the WRF_ARW and the guide
was created for a different version. At one point it adds the new array to the call for the
microphysics driver within a Fortran file called first_rk_step_part1.F. In my
version of WRF the call for the microphysics driver was in solve_em.F)
http://www.mmm.ucar.edu/wrf/users/tutorial/tutorial_presentation_summer_2012.htm
The link above has several presentations describing the WRF model. I recommend reading
the presentation entitled architecture. The architecture presentation describes the
domain, patch, and tile system which allows for parallel processing and which is used to
define the dimensions of the various arrays passed through the WRF processing system. I
will try to clear that up below when I talk about the programming I did.
http://forum.wrfforum.com/index.php
The link above is for the WRF Forum. This was useful to search and find similar problems,
and their solution. This particular thread was very similar to what I was trying to do:
http://forum.wrfforum.com/viewtopic.php?f=9&t=3514
There were still differences, but I referred to this particular thread many times.
General Process
After researching the resources above, the first step is problem identification. In
this case, identify the variable to include in the wrfout and the characteristics of the
variable such as dimension, intent and type. Next, identify all of the code that must be
changed to pass the variable out of the subroutine it exists in all the way to a routine or
subroutine that operates on the domain/grid level. To pass the variable out of its
subroutine, the relationship between the subroutine and other routines and subroutines
must be known. Another significant consideration as the variable is passed up to the
domain/grid level is any kind of dimension, intent, and/or type transformations that may
be required or that occur to the variable along the way. Finally, the registry must be edited
to add the new variable to the wrfout file.
After identifying the problem and the aspects of the problem listed above, the
various routines, subroutines, and the registry can be edited. Once editing is complete the
WRF must be recompiled. In cases where the registry and/or the WRF code itself has been
changed a full recompile must be done including the clean, configure and compile steps.
Always write out the results of the recompile and search for severe or fatal errors.
Also look in the main folder for the WRF executables. If there are no executables, another
round of coding and a recompile will be required. Check the compile output to determine
what compiling errors caused the compilation failure. Even if executables were created
check the compile output for errors. If errors are present attempt to find and correct them.
After recompiling, run the WRF to test the output. Based on my experience a run
with the mpi command and a run without should be tested. For several of my iterations I
successfully wrote my variable to the wrfout while running without the mpi command, but
not when I ran the WRF with the mpi command. I am not sure why this occurred,
furthermore I am not sure if I would trust the output from the run without the mpi
command if the mpi run wouldn’t compute correctly.
My Case
I programmed the WRF to write the value of a microphysics tendency source term
to the wrfout file. I first found the code for the microphysics scheme. All of the
programming was done on select files within the WRFV3 folders (directories, sorry I
started in windows gui land) listed below:
drwxr-xr-x
drwxr-xr-x
drwxr-xr-x
2 u0758091 steenbur
2 u0758091 steenbur
2 u0758091 steenbur
8192 Feb
20480 Feb
4096 Feb
1 10:34 dyn_em
1 10:32 phys
1 10:23 Registry
…
The microphysics scheme is inside the phys folder and saved in a Fortran file called
module_mp_thompson.F. Within the microphysics scheme are equations for the
tendency of every microphysics species in the scheme. These equations are a sum of
several source and sink terms. For example:
qsten(k) = qsten(k) + (prs_iau(k) + prs_sde(k) &
+ prs_sci(k) + prs_scw(k) + prs_rcs(k) &
+ prs_ide(k) - prs_ihm(k) - prr_sml(k)) &
* orho
Here qsten(k) is the tendency of snow and the various terms on the RHS are the
sources and sinks of snow due to various processes. I chose to write out prs_iau(k),
which represents the auto-conversion of cloud ice to snow. Note that these variables are
column arrays which are indicated by the (k), I will return to that point later.
Now that I found the variable I wanted to write out, I needed to figure out how to
pass the variable out from inside the microphysics scheme to the grid/domain. prs_iau
is in module_mp_thompson.F but there are multiple subroutines in the file. I brute
force traced the variable assignment and subroutine calls throughout the WRFV3 file
structure.
(A better method is the find command:
find –iname “*.F” | xargs grep –iR what you want to find
> find.txt
After the command finishes running, open up the text file and there will be a list of
everywhere the “what you want to find” is written in a .F file. You can use this to search
for a variable or the scheme name or the name of a subroutine to find what other routines
and subroutines reference it. Not all places that “what you want to find” is written will be
relevant to this process, but if you don’t check them all out and make sure editing code and
debugging will be painful. Remember you are looking for the most direct path from the
variable of interest to the grid/domain level. Another good use of this tool is to find your
variables if you need to eliminate them.)
I found within module_mp_thompson.F there were two relevant subroutines,
mp_thompson and mp_gt_driver. I also found that subroutine mp_gt_driver was
called by subroutine microphysics_driver within
module_microphysics_driver.F. I further found that subroutine
microphysics_driver was called by subroutine solve_em within solve_em.F.
solve_em.F works on variables at the grid level so I needed to pass prs_iau or its
equivalent from subroutine mp_thompson through to subroutine solve_em to allow
prs_iau or its equivalent to appear in the wrfout file. Passing prs_iau through each of
these routines required me to edit variable declarations, subroutine assignments, and
subroutine calls in each of these subroutines. In addition I needed to add prs_iau or its
equivalent to the registry. Displayed roughly hierarchically the paragraph above looks like:
solve_em.F {grid/domain operations}
solve_em calls microphysics_driver
module_microphysics_driver.F
microphysics_driver calls mp_gt_driver
module_mp_thompson.F
mp_gt_driver calls mp_thompson {prs_iau locally defined}
The last bit of preparation I needed to do before adjusting the code was to figure out
the characteristics of the variable. This is the point I referenced earlier. Within subroutine
mp_thompson, prs_iau(k) is declared as a local variable by the following statement:
DOUBLE PRECISION, DIMENSION(kts:kte):: prs_iau
The variable is a one-dimensional array of double precision values. The dimension
is a vertical column indexed by tile indices. kts indicates the axis, k, the type of index, t
for tile, and s indicates start while e indicates end. Here are the dimensions of the various
data subdivisions as they appear in the variable call of subroutine mp_gt_driver
ids,ide, jds,jde, kds,kde, &
ims,ime, jms,jme, kms,kme, &
its,ite, jts,jte, kts,kte, &
! domain dims
! memory dims
! tile dims
(I think data stored and processed in the tile subdivision is usually created at the
beginning of each time step and then deleted at the end of each time step. Data in the
memory and domain subdivisions is usually carried from time step to time step. I know
these subdivisions exist to ensure that the code can be run in parallel, but I do not quite
understand how this is done. For example several times during my coding and debugging
process I could compile correctly and run the WRF but the output from running WRF with
mpi would be incorrect in strange ways {see the Problems I Encountered section for more
details}. The output from runs without the mpi command seemed to be fine. The main
point here is the need to establish what the variable characteristics are in order to compare
them to similar variables, variables in the registry, and to facilitate any transformations if
required.)
In the registry folder, in a file called Registry.EM_COMMON, variables are
defined and several columns of metadata exist describing the behavior of the variables. I
noted that no double precision variables exist in the registry. Based on other variables in
the Thompson scheme and the example from the resources section, I thought about how to
establish my variable in the registry. Here are the metadata columns in the registry:
<Table><Type><Sym><Dims><Use><NumTLev><Stagger><IO><DNAME><DESCRIP><UNITS>
<Table> should be state. All grid/domain level variables are state.
<Type> should be real because there are no double precision variables in the
registry. This was my first transformation issue. I needed to convert from a double
precision number to a real number at some point in the code as prs_iau passed out of the
microphysics scheme.
<Sym> is the variable name. My plan was to examine multiple microphysics
schemes and extract the same source and sink terms from each. I decided to create a new
variable name that could be tied to multiple microphysics schemes. I believe it also helps in
dealing with the Type transformation. When my code is done I will not have one variable
name associated with two different Types within the code.
<Dims> are the dimensions of the variable. I needed to know the microphysics
sources and sinks at every grid point. Therefore I needed a three dimensional array. This
is the second transformation I had to make. prs_iau is one dimensional, but other
variables output from the microphysics scheme are three dimensional so I needed to pay
close attention to how the microphysics scheme changes the dimensions of its variables
and either copy their approach or ensure my code is inside the schemes’ dimension
transforming loop.
<Use> associates the variable with groups of other variables. I found that I was
unable to compile if it was not set to dyn_em. I do not know why.
<NumTLev> I am not sure what this does. I used a dash.
<Stagger> sets the variable on the momentum grid if X,Y, and/or Z are specified. I
set a dash because moisture variables are not staggered.
<IO> describes the input and output handling of the variable. I set the column to h
for history. This will put the variable into the output stream. There are other options here,
I know the i stands for input and r stands for restart. The resources listed above document
other options for this column.
<DNAME> sets the variable name as it will appear in the wrfout file.
<DESCRIP> text description of the variable.
<UNITS> text description of the variables’ units.
After I chose a variable; traced its path through the WRF code; determined the need
to add a variable in order to transform from the original variables dimension and type; and
determined the characteristics of the final variable should be; I was ready to code. Here is
the registry definition I added for my microphysics variable, iausn (sorry for the wrap
around):
<Table><Type><Sym><Dims><Use><NumTLev><Stagger><IO> <DNAME><DESCRIP><UNITS>
state real iausn ikj dyn_em h
"Ice Auto Snow" "Ice autoconverted
to snow" "kg kg-1"
Next I edited the subroutine solve_em in solve_em.F. solve_em.F is found in
the dyn_em folder. I found and edited the call for subroutine microphysics_driver
within the subroutine solve_em as below:
CALL microphysics_driver(
…
&
, iausn=grid%iausn
&
)
This call lists all of the variables used by microphysics_driver. As noted
above iausn will need to come through this subroutine, so I added iausn to the call for
microphysics_driver. The way variables are assigned at this level is by referencing
them to the grid that is why the variable is assigned by iausn=grid%iausn. I added
iausn to the end of the call. I don’t think position matters. This is all I had to do in the
solve_em.F file.
The next two files I edited are in the phys folder;
module_microphysics_driver.F and module_mp_thompson.F. After opening
module_microphysics_driver.F, I found and edited the subroutine
microphysics_driver. First I added my variable to the subroutine
microphysics_driver subroutine variable list:
SUBROUTINE microphysics_driver(
…
,iausn)
&
In the microphysics_driver subroutine I also declared iausn with the
following statement:
REAL,DIMENSION(ims:ime,kms:kme,jms:jme),INTENT(INOUT):: iausn
Note: Type matches the registry, as do Dims, and Sym. The dimension type is
memory. I chose memory because other variables that pass in and out of subroutines have
memory dimensions. The INTENT(INOUT) portion of the declaration allows the variable
to pass between subroutines. If no INTENT(XXX) statement is declared for a variable it is
considered a local variable for the subroutine. I wanted iausn to pass through this
subroutine so I declared INTENT(INOUT). The microphysics_driver subroutine
includes a call to the mp_gt_driver subroutine. I needed to add iausn to the
mp_gt_driver subroutine variable call list as well. I didn’t add iausn to the end of the
call list in this subroutine, it’s in the middle, but I don’t think it makes a difference. The call
list is below:
CALL mp_gt_driver(
…
iausn=iausn,
&
&
Next I opened module_mp_thompson.F to edit subroutine mp_gt_driver
and subroutine mp_thompson. I added iausn to the subroutine mp_gt_driver
variable list:
SUBROUTINE mp_gt_driver(qv, qc, qr, qi, qs, qg, ni, nr, &
…
iausn)
In addition I declared iausn in the subroutine mp_gt_driver variable
declaration list. I also declared prs_iau, as a local variable in this subroutine so I could
code the transformation of variable type and variable dimension. No INTENT(XXX)
statement is declared for prs_iau so it is a local variable for the subroutine.
REAL,DIMENSION(ims:ime, kms:kme, jms:jme),INTENT(INOUT):: iausn
DOUBLE PRECISION, DIMENSION(kts:kte):: prs_iau
I chose this subroutine as the step to make the transformation between iausn and
prs_iau because the subroutine included code to perform variable dimension
transformations for several other variables. I then examined the subroutine to determine
where my code changes needed to be inserted and what they should look like. Here is an
excerpt of the code without modification:
…
kmax_ni = 0
kmax_nr = 0
do i = 1, 256
mp_debug(i:i) = char(0)
enddo
j_loop:
i_loop:
do j = j_start, j_end
do i = i_start, i_end
pptrain = 0.
pptsnow = 0.
…
do k = kts, kte
t1d(k) = th(i,k,j)*pii(i,k,j)
p1d(k) = p(i,k,j)
…
enddo
call mp_thompson(qv1d, qc1d, qi1d, qr1d, qs1d, qg1d, ni1d, &
nr1d, t1d, p1d, dz1d, &
pptrain, pptsnow, pptgraul, pptice, &
kts, kte, dt, i)
pcp_ra(i,j) = pptrain
pcp_sn(i,j) = pptsnow
…
do k = kts, kte
qv(i,k,j) = qv1d(k)
qc(i,k,j) = qc1d(k)
…
write(mp_debug,*) 'WARNING, negative qc ', qc1d(k),
' at i,j,k=', i,j,k
CALL wrf_debug(150, mp_debug)
…
enddo
enddo i_loop
enddo j_loop
…
END SUBROUTINE mp_gt_driver
&
I have added ellipsis to shorten the code into blocks, where the code within the
blocks seem to perform similar functions. The functions of these code blocks are important
because they reveal where my code edits should be placed. I believe my early attempts
failed because my edits were not in the correct location in the code.
The first block of code initializes several variables of various type and dimension to
zero. The second block of code loops through the tile index (k) and equates a local onedimensional variable with a tile index of (k) to a non-local three-dimensional variable
with a memory index of (i,k,j). This step is required because the subroutine
mp_thompson operates in one dimension with a tile index of (k) and these variables are
passed to the subroutine mp_thompson to compute the microphysics scheme. The next
block of code is the call to the subroutine mp_thompson. All code after this call will have
access to all of the variables produced by the subroutine mp_thompson. Any extraction of
variables must occur after this block of code. In fact the two lines immediately following
the close parentheses of the call and the next block of code do just that. The two lines
immediately following the close parentheses of the call to the subroutine mp_thompson
equate a local variable to a non-local variable that appears in the subroutine
mp_thompson call statement. The next block of code is a loop through the tile index (k)
and equates a non-local three-dimensional variable with a memory index of (i,k,j)to a
local one-dimensional variable with a tile index of (k) that has been populated within the
subroutine mp_thompson. This code is actually writing out the results of the scheme.
Therefore, this block of code is the likely spot to transform prs_iau to iausn. The next
three blocks of code are debugging of some sort; closing off the do loops; and the
completion of the subroutine.
With a rough understanding of how this subroutine worked I added my edited code
to the subroutine. First I initialized my variable:
iausn(i,k,j) = -999.9
I placed this bit of code in the first block at the very beginning of the subroutine
outside of any loop structures. At first I attempted to initialize with a loop copied from the
example in the resources section, but I did not get good results in wrfout if I ran the WRF
with the mpi command. I believe my attempt at a loop approach to initialization failed
because I didn’t use the correct index variables in the loop. Note that in block one above
the i_loop and j_loop index through variables called i_start, i_end and j_start,
j_end . Not shown in the excerpt are statements that set i_start, i_end and j_start,
j_end equal to the general tile dimension variables its, ite and jts, jte. My point is to
be aware of index assignments and try to use what is in the subroutine already or forgo
them as I did. An example I found online was similar to my code above so I used it. With
such an odd value I could easily tell if iausn had been written over by looking for values of
-999.9 in my wrfout file.
I did not edit anything in block two despite my need to transform my threedimensional variable to one dimension. The reason I didn’t edit at this step is because I
would not be passing iausn into the subroutine mp_thompson.
My next edit was in the block of code with the call to the subroutine
mp_thompson:
call mp_thompson(qv1d, qc1d, qi1d, qr1d, qs1d, qg1d, ni1d, &
nr1d, t1d, p1d, dz1d, &
pptrain, pptsnow, pptgraul, pptice, &
#ifdef WRF_CHEM
rainprod1d, evapprod1d, &
#endif
kts, kte, dt, i, j,prs_iau)
Note that the last variable in the call variable list is prs_iau. This variable must
appear in the call variable list so that it can be passed out of the subroutine mp_thompson
and manipulated in this subroutine mp_gt_driver.
My final edit in subroutine mp_gt_driver is to actually assign the value of
prs_iau to iausn. In the fourth block of code a loop through the tile index (k)equates
a non-local three dimensional variable with a memory index of (i,k,j)to a local one
dimensional variable with a tile index of (k) that has been populated within the
subroutine mp_thompson. I took advantage of the existing loop and added a line within
the loop to equate iausn to prs_iau as below:
do k = kts, kte {EXISTING CODE}
iausn(i,k,j) = prs_iau(k)
…
{EXISTING CODE}
enddo
{EXISTING CODE}
This code completed the transformation of iausn in both dimension and type.
Apparently within WRF if you set a real number equal to a double precision number no
additional code is necessary. I believe setting a double precision number equal to a real
number requires the addition of code double = DBLE(real). Your mileage may vary.
The last editing I did was in subroutine mp_thompson, where I added prs_iau to
the end of the subroutine variable list:
subroutine mp_thompson (qv1d, qc1d, qi1d, qr1d, qs1d, qg1d, ni1d, &
nr1d, t1d, p1d, dzq, &
pptrain, pptsnow, pptgraul, pptice, &
#ifdef WRF_CHEM
rainprod, evapprod, &
#endif
kts, kte, dt, ii, jj,prs_iau)
I also changed prs_iau from a local variable within subroutine mp_thompson
to a nonlocal variable by editing the variable declaration. Again this was done by declaring
INTENT(INOUT).
DOUBLE PRECISION, DIMENSION(kts:kte),INTENT(INOUT):: prs_iau
prs_iau is manipulated extensively within subroutine mp_thompson. By only
changing the INTENT(INOUT) I believe I have preserved calculation of the variable and
all variables that depend on it in the scheme.
After performing the edits noted above I recompiled. I navigated to the WRFV3
folder. Once in this folder I executed the ./clean –a command. This command removed
executables and various files created by the compiling process. I then reinstated my
configure.wrf file with the command mv configure.wrf.backup
configure.wrf. This is necessary because the ./clean – a command converts
configure.wrf to configure.wrf.backup. The next step was to configure the WRF
with the ./configure command. Working on ATMOS01 options 4 and 1 are required at
the prompts. Then I compiled using the ./compile em_real > & compile.txt
command. This generated a text file, compile.txt that recorded any compilation errors.
After the compile was complete I looked into the main folder to see if the WRF executables
were present and I checked compile.txt for errors. Finding none I ran the WRF.
After the first history output after the initial hour was produced I checked the
wrfout using ncview. I saw iausn was in the list of three-dimensional variables by
the <DNAME> “Ice Auto Snow” that I had assigned in Registry.EM_COMMON. I
opened it up and was able to view the data. The values and placement appeared to
correspond with other microphysics variables so I knew I had successfully written the
variable out.
Problems I Encountered
I worked on this issue for a few weeks. I would edit the code, go through the
compiling process, fix compiling errors, run the WRF, find out it hadn’t produced correct
values of the variable and repeat. This is not a simple task. Take your time and be
methodical. I actually knew the relationship between all of the routines and subroutines
early in the first week. The devil was in the details of deciding how far to pass the actual
microphysics variable out through the subroutines and how to edit the code to accomplish
that task.
Compiling errors are easily seen in compile.txt. When the severe or fatal
error is found in the text file it is then easy to search for the error in the file and line of code
specified in the error. Finally it is easy to edit the code and recompile. However, a
successful compile does not mean success. This took most of my time and was the most
frustrating. By all means fix the errors pointed out by the compiler, but also think about
the logic of your variable passing. Think about if your edits are in the appropriate place in
the code or not. Think about if the variable types and dimensions match, etc.
Late in the process I found out that at some point I had changed the type of
prs_iau to real. I was also trying to equate iausn to prs_iau with the DBLE
command. After I reverted prs_iau to double precision and did away with the awkward
DBLE conversion I successfully got iausn from wrfout. This problem reveals a lack of a
methodical approach. I had at some point made multiple edits and actually generated a
conflict. Try to only change one aspect of your edits at a time.
Finally and most significantly, I frequently successfully compiled but when I ran the
WRF using the mpi command, the output was incorrect. Specifically for me, at model level
5, three tiles had the same incorrect value of iausn over the entire tile. The incorrect tiles
were in the southeast corner of the domain. Running the WRF without the mpi command
resulted in correct output. I have no good explanation for why this would occur at this
time. It might have been related to the mismatch in the type of my variable and the actual
microphysics variable. It might have been related to the initialization approach.