Example Application:
3D Scalar Wave Equation
The Cactus Team
Albert Einstein Institute
cactus@cactuscode.org
Scalar Wave Model Problem

Scalar waves in 3D are solutions of the hyperbolic
wave equation
-f,tt + f,xx + f,yy + f,zz = 0

This is an initial value problem, if we give data for f and
its first time derivative at some initial time, the wave
equation tells us how it evolves in the future
time
r
Numerical Method

Numerical solve by discretising on a grid, using explicit
finite differencing (centered, second order)
f n+1i,j,k = 2f ni,j,k - f n-1i,j,k
+ Dt2/Dx2(f ni+1,j,k -2 f ni,j,k + f ni-1,j,k )
+ Dt2/Dy2(f ni,j+1,k -2 f ni,j,k + f ni,j-1,k )
+ Dt2/Dz2(f ni,j,k+1 -2 f ni,j,k + f ni,j,k-1 )
time
r
Numerical Method

Finite grid, so need to apply outer boundary conditions

Main parameters:


grid spacings: Dt, Dx, Dy, Dz, which coords?, which initial data?
Simple problem, analytic solutions, but contains many
features needed for modelling more complex problems
Stand Alone Code: Main.f
c
c
c
c
c
====================================
program WaveToy
====================================
Fortran 77 program for 3D wave equation.
Explicit finite difference method.
====================================
x(i,j,k) = dx*(i-1) + x_origin
y(i,j,k) = dy*(j-1) + y_origin
z(i,j,k) = dz*(k-1) + z_origin
r(i,j,k) = sqrt(x(i,j,k)**2+y(i,j,k)**2+z(i,j,k)**2)
end do
end do
end do
c
Global variables in include file
include "WaveToy.h"
integer i,j,k
c
OPEN OUTPUT FILES
open(unit=11,file=“out.xl”)
open(unit=12,file=“out.yl”)
open(unit=13,file=“out.zl”)
c
SET UP PARAMETERS
nx = 30
[MORE PARAMETERS]
c
SET UP INITIAL DATA
call InitialData
call Output
c
EVOLVING
do iteration = 1, nt
call Evolve
if (mod(iteration,10).eq.0) call Output
end do
c
SET UP COORDINATE SYSTEM AND GRID
x_origin = (0.5 - nx/2)*dx
y_origin = (0.5 - ny/2)*dy
z_origin = (0.5 - nz/2)*dz
do i=1,nx
do j=1,ny
do k=1,nz
stop
end
Standalone Program
Setting up parameters
Setting up grid and coordinate system
Opening output files
Setting up initial data
Performing iteration 10
Performing iteration 20
Performing iteration 30
Performing iteration 40
Performing iteration 50
Performing iteration 60
Performing iteration 70
Performing iteration 80
Performing iteration 90
Performing iteration 100
Done
Wave Equation Program

Developed as single processor 3D program (~1hr)

Should you use Cactus? Not needed for such a simple
program but if you need:


parallelism for more resolution?

to share development with colleagues?

to run on a variety of platforms?

2D/3D output and checkpointing?

to solve an elliptic equation for initial data?

adaptive mesh refinement?, remote monitoring? ….
All can be provided by Cactus
Cactus Flesh Provides

Parameter parser with types, ranges, checking

Scheduling of routines

Make system on many architectures

Dynamic argument lists

Utilities such as program checking test suites
Cactus Thorns Provide

Coordinates


Boundary conditions


(Fixed, flat, radiation,...)
IO


(Cartesian, various domains with symmetries)
(screen, 0D-3D, IsoSurfaces, Jpeg, streaming, configurable)
Driver

(create grid variables, handle storage, communications)
Stand Alone Code: Main.f
c
c
c
c
c
====================================
program WaveToy
====================================
Fortran 77 program for 3D wave equation.
Explicit finite difference method.
====================================
c
Global variables in include file
include "WaveToy.h"
integer i,j,k
Configuation
c
SET UP PARAMETERS
nx = 30
[MORE PARAMETERS]
x(i,j,k) = dx*(i-1) + x_origin
y(i,j,k) = dy*(j-1) + y_origin
z(i,j,k) = dz*(k-1) + z_origin
r(i,j,k) = sqrt(x(i,j,k)**2+y(i,j,k)**2+z(i,j,k)**2)
end do
end do
end do
c
OPEN OUTPUT FILES
open(unit=11,file=“out.xl”)
open(unit=12,file=“out.yl”)
open(unit=13,file=“out.zl”)
c
SET UP INITIAL DATA
call InitialData
call Output
c
EVOLVING
do iteration = 1, nt
call Evolve
if (mod(iteration,10).eq.0) call Output
end do
files
c
SET UP COORDINATE SYSTEM AND GRID
x_origin = (0.5 - nx/2)*dx
y_origin = (0.5 - ny/2)*dy
z_origin = (0.5 - nz/2)*dz
do i=1,nx
do j=1,ny
do k=1,nz
stop
end
CartGrid3D
Output from
parameters
You write
these!!
Converting Your Code ...


Decide on arrangement/thorn structure
Write thorn configuration files








param.ccl, interface.ccl, schedule.ccl
Add Cactus “stuff” to source code
Add source code filenames to make.code.defn
Compile and debug ...
Write a parameter file
Run and debug …
Create a test suite to be sure it keeps working
If you want, make your thorns available to
friends/colleagues/the world
Splitting Into Thorns

Could just write one thorn containing all the routines

Better to split the code in different thorns

IDScalarWave
(Initial data)

WaveToyF77
(Evolving routines)

Make use of Toolkit thorns for Coordinates,
Boundaries, IO.

Why separate thorns?

Easier to develop and share

Better code planning - modularity
Create New Thorns

Checkout Cactus, and in the Cactus home directory type
gmake newthorn

Follow instructions to create a new thorns “WaveToyF77”
and “IDScalarWave” in an arrangement e.g. “Tutorial”

This gives you the thorn structure and the basic files
needed
Thorn Structure
WaveToyF77
schedule.ccl
param.ccl
interface.ccl
README
src
doc
par
make.code.defn
WaveToy.F77
InitSymBound.F77
test
Thorn Configuration Files

Need to decide:

What are the grid variables:
– interface.ccl

What are the parameters:
– param.ccl

In which order should the routines be scheduled:
– schedule.ccl


Do I use grid variables, parameters or scheduling from other thorns?
These configuration files are parsed (Perl) during
compilation, generating code for argument lists, parameters,
program flow, etc.
Thorn Configuration: param.ccl

Parameters: set in a parameter file read and verified at run
time.

Example: How many grid points? What initial data? What to output?

Specify:

data type (real, integer, keyword, boolean, string)

range (nx>0, initial_data is “gaussian wave” or “plane wave”)

description

visability to other thorns (private, restricted, global)

default value

parameters used from other thorns
WaveToyF77: param.ccl
# Parameter definitions for thorn WaveToyF77
private:
Only the evolver needs
to know about the
boundary condition
KEYWORD bound "Type of boundary condition to use"
{
"none”
:: "No boundary condition"
"flat"
:: "Flat boundary condition"
"radiation"
:: "Radiation boundary condition"
"zero"
:: "Zero boundary condition"
} "none"
IDScalarWave: param.ccl
# Parameter definitions for thorn IDScalarWave
shares: grid
USES KEYWORD type
restricted:
KEYWORD initial_data "Type of initial data"
{
"plane"
:: "Plane wave"
"gaussian"
:: "Gaussian wave"
"box"
:: "Box wave"
} "gaussian"
private:
REAL radius "The radius of the gaussian wave"
{
0:* :: “Radius must be positive”
} 0.0
IDScalarWave uses the
parameter type from
CartGrid3D
Other thorns can
use or extend
initial_data
Only for
IDScalarWave
to use
Thorn configuration: interface.ccl

Object orientated concepts:


Implementation, Inherits, Friends
Grid variables

group name (many flesh functions act on groups)

group type (grid array, grid function, grid scalar)

variable type (real, int, complex)

dimension

description

visability to other thorns (private, protected, public)
WaveToyF77: interface.ccl
# Interface definition for WaveToyF77

Implements: describes what this
thorn “does”, WaveToyF77 can
be replaced by any other thorn
which “does” the same thing and
has the same public interface.

Timelevels: finite difference
method is a 3 time level scheme,
phi_n, phi, phi_p. Time levels are
rotated at each iteration.

Scope: grid variables can be
public, protected or private.
implements: wavetoy
public:
cctk_real scalarevolve type = GF
timelevels=3
{
phi
} "The evolved scalar field"
IDScalarWave: interface.ccl
# Interface definition for IDScalarWave

Inherits: what IDScalar wave
takes from other thorns
(implementations)

Needs phi from WaveToyF77
(wavetoy) to fill in initial data

Needs coordinate grid functions
and parameters from CartGrid3D
(grid)
implements: idscalarwave
inherits: wavetoy grid
Thorn Configuration: schedule.ccl

Schedules when the thorn routines are run, and when
storage (and communications) for variables are activated

WaveToyF77 has one base routine, WaveToyF77_Evol

when should it be run?

should it be BEFORE or AFTER any other routine?

what language (Fortran or C) is it written in?

is variable storage or communication needed?

is the variable storage needed just for the routine, or for the whole
run?
Example Scheduling Tree
Startup routines
Cactus: Register banner for Cactus
CartGrid3D: Register GH Extension for GridSymmetry
CartGrid3D: Register coordinates for the Cartesian grid
IOASCII: Startup routine
IOBasic: Startup routine
IOUtil: IOUtil startup routine
PUGH: Startup routine.
WaveToyC: Register banner
Parameter checking routines
CartGrid3D: Check coordinates for CartGrid3D
IDScalarWave: Check parameters
Initialisation
CartGrid3D: Set up spatial 3D Cartesian coordinates on the GH
PUGH: Report on PUGH set up
Time: Set timestep based on speed one Courant condition
WaveToyC: Schedule symmetries
IDScalarWave: Initial data for 3D wave equation
do loop over timesteps
WaveToyC: Evolution of 3D wave equation
t = t+dt
if (analysis)
endif
enddo
CCTK_STARTUP
CCTK_PARAMCHECK
CCTK_INITIAL
CCTK_EVOL
WaveToyF77: schedule.ccl
# Schedule definitions for thorn WaveToy77
STORAGE: scalarevolve
Storage always
assigned
schedule WaveToyF77_Evolution as WaveToy_Evolution at EVOL
{
LANG: Fortran
Synchronize
SYNC: scalarevolve
group on exit
} "Evolution of 3D wave equation”
schedule WaveToyF77_Boundaries as WaveToy_Boundaries at EVOL
AFTER WaveToy_Evolution
{
LANG: Fortran
Function alias
} "Boundaries of 3D wave equation"
IDScalarWave: schedule.ccl
# Schedule definitions for thorn IDScalarWave
schedule IDScalarWave_CheckParameters at CCTK_PARAMCHECK
{
LANG: Fortran
} "Check parameters"
schedule IDScalarWave_InitialData at CCTK_INITIAL
{
Should already be
STORAGE: wavetoy::scalarevolve
on, but make sure
LANG:
Fortran
} "Initial data for 3D wave equation"
WaveToy.F77
dx2i = 1.0/dx2
dy2i = 1.0/dy2
dz2i = 1.0/dz2
#include "cctk.h"
#include "cctk_Parameters.h”
#include "cctk_Arguments.h"
factor = 2*(1-(dt2)*(dx2i+dy2i+dz2i))
subroutine WaveToyF77_Evolution(CCTK_ARGUMENTS)
implicit none
c
c
c Declare variables in argument list
DECLARE_CCTK_ARGUMENTS
INTEGER i,j,k,ierr
CCTK_REAL dx,dy,dz,dt
CCTK_REAL dx2,dy2,dz2,dt2
CCTK_REAL dx2i,dy2i,dz2i
CCTK_REAL factor
c Set up shorthands
c ----------------dx = CCTK_DELTA_SPACE(1)
dy = CCTK_DELTA_SPACE(2)
dz = CCTK_DELTA_SPACE(3)
dt = CCTK_DELTA_TIME
dx2
dy2
dz2
dt2
=
=
=
=
dx*dx
dy*dy
dz*dz
dt*dt
&
&
&
&
Do the evolution
---------------do k = 2, cctk_lsh(3)-1
do j = 2, cctk_lsh(2)-1
do i = 2, cctk_lsh(1)-1
phi_n(i,j,k) = factor*phi(i,j,k) phi_p(i,j,k) + (dt2) *
((phi(i+1,j,k)+phi(i-1,j,k))*dx2i
+(phi(i,j+1,k)+phi(i,j-1,k))*dy2i
+(phi(i,j,k+1)+phi(i,j,k-1))*dz2i)
end do
end do
end do
return
end
WaveToy.F77 (2)
c Apply the outer boundary conditions
subroutine
if (CCTK_EQUALS(bound,"flat")) then
&
WaveToyF77_Boundaries(CCTK_ARGUMENTS)
call BndFlatVN(ir,cctkGH,sw,"wavetoy::phi")
else
if (CCTK_EQUALS(bound,"zero")) then
implicit none
call BndScalarVN(ir,cctkGH,zero,sw,
&
"wavetoy::phi")
DECLARE_CCTK_ARGUMENTS
else
if
(CCTK_Equals(bound,"radiation").eq.1)
then
DECLARE_CCTK_PARAMETERS
call BndRadiativeVN(ir,cctkGH,sw,zero,one,
DECLARE_CCTK_FUNCTIONS
&
"wavetoy::phi”,"wavetoy::phi")
end if
c Local declarations
CCTK_REAL zero,one
if (ierr < 0) then
integer ir
call CCTK_WARN(0,”BCs not applied!!");
integer sw(3)
end if
ierr = -1
return
zero = 0.0
end
one = 1.0
c Set the stencil width
sw(1)=1
sw(2)=1
sw(3)=1
c Apply the symmetry boundary conditions
call
& CartSymGN(ir,cctkGH,"wavetoy::scalarevolve")
Compiling WaveToy Application

Get hold of the additional Cactus thorns needed









CactusBase/CartGrid3D
(sets up the Cartesian grid)
CactusBase/Boundary
(provides standard boundary conditions)
CactusBase/Time
(set the timestep)
CactusPUGH/PUGHSlab
(extract hyperslabs of data from grid arrays)
CactusPUGH/PUGHReduce (reduction operations)
CactusPUGH/PUGH
(the driver, always needed)
CactusBase/IOASCII
(output in 1D ASCII format)
CactusBase/IOBasic
(output scalars (inc. reductions) and to screen)
CactusBase/IOUtil
(all the Cactus IO thorns need this thorn)
Compiling WaveToy Application

Set up a configuration by typing
gmake wave (or gmake wave-config)

Compile the code using
gmake wave

If successful you’ll create exe/cactus_wave
Running Executable

Cactus runs from a parameter
file
> exe/cactus_wave wavetoy.par
ActiveThorns = "idscalarwave time
wavetoyf77 pugh pughreduce cartgrid3d
pughslab ioutil ioascii"
time::dtfac = 0.5
idscalarwave::initial_data = "gaussian"
idscalarwave::sigma = 2.8
idscalarwave::radius = 0
wavetoyf77::bound = "zero"


All thorns you need must be
activated using the ActiveThorns
parameter
Parameters must be qualified
with the implementation or thorn
name, depending on their type
grid::type = "BySpacing"
grid::dxyz = 0.6
driver::global_nx = 30
driver::global_ny = 30
driver::global_nz = 30
cctk_itlast = 100
IOASCII::out1D_every = 10
IOASCII::out1D_vars = "wavetoy::phi "
IOASCII::outinfo_every = 10
IOASCII::outinfo_vars = "wavetoy::phi"
IO::outdir = "StandAlone"
Run Time Output
> mpirun -np 2 exe/cactus_wave wavetoy.par
----------------------------------------------------------------------------10
1
0101
************************
01 1010 10
The Cactus Code V4.0
1010 1101 011
www.cactuscode.org
1001 100101
************************
00010101
100011
(c) Copyright The Authors
0100
GNU Licensed. No Warranty
0101
----------------------------------------------------------------------------Activating thorn Cactus...Success -> active implementation Cactus
Activating thorn iobasic...Success -> active implementation IOBasic
Activating thorn idscalarwave...Success -> active implementation idscalarwave
Activating thorn time...Success -> active implementation time
Activating thorn wavetoyf77...Success -> active implementation wavetoy
Activating thorn pugh...Success -> active implementation driver
Activating thorn pughreduce …Success -> active implementation pughreduce
Activating thorn cartgrid3d...Success -> active implementation grid
Activating thorn ioutil...Success -> active implementation IO
Activating thorn ioascii...Success -> active implementation IOASCII
-----------------------------------------------------------------------------
Run Time Output (2)
Startup routines
CartGrid3D: Register GH Extension for GridSymmetry
CartGrid3D: Register coordinates for the Cartesian grid
PUGH: Startup routine
PUGHReduce: Startup routine
IOUtil: Startup routine
IOASCII: Startup routine
IOBasic: Startup routine
WaveToyF77: Register banner
Parameter checking routines
CartGrid3D: Check coordinates for CartGrid3D
IDScalarWave: Check parameters
Initialisation
CartGrid3D: Set up spatial 3D Cartesian coordinates on the GH
IOASCII: Choose 1D output lines
IOASCII: Choose 2D output planes
PUGH: Report on PUGH set up
Time: Set timestep based on Courant condition
WaveToyF77: Schedule symmetries
IDScalarWave: Initial data for 3D wave equation
Run Time Output (3)
do loop over timesteps
WaveToyF77: Evolution of 3D wave equation
WaveToyF77: Boundaries of 3D wave equation
t = t+dt
if (analysis)
endif
enddo
Termination routines
PUGH: Termination routine
Shutdown routines
--------------------------------------------------------------------------------------------------------------------------------------------------------Driver provided by PUGH
----------------------------------------------------------------------------WaveToyF77: Evolutions of a Scalar Field
----------------------------------------------------------------------------INFO (CartGrid3D): dx=>3.0000000e-01 dy=>3.0000000e-01 dz=>3.0000000e-01
INFO (CartGrid3D): x=>[-0.150, 8.550] y=>[-0.150, 8.550] z=>[-0.150, 8.550]
INFO (PUGH): MPI Evolution on 4 processors
INFO (PUGH): 3-dimensional grid functions
INFO (PUGH):
Size: 30 30 30
INFO (PUGH):
Processor topology: 1 x 2 x 2
INFO (PUGH):
Local load: 7680
[30 x 16 x 16]
INFO (PUGH):
Maximum load skew: 0.000000
Run Time Output (4)
-------------------------------------------it |
| phi
|
|
t
| Min
Max
|
-------------------------------------------0 |
0.000| 0.00000000 | 0.99142726 |
10 |
1.500| -0.14944313 | 0.00005206 |
20 |
3.000| -1.16992203 | 0.00000000 |
30 |
4.500| -0.72347868 | -0.00000002 |
40 |
6.000| -0.41127922 | -0.00000094 |
50 |
7.500| -0.29834817 | -0.00002427 |
60 |
9.000| -0.23577373 | -0.00035044 |
70 |
10.500| -0.19532474 | 0.00000311 |
80 |
12.000| -0.16688911 | 0.00010812 |
90 |
13.500| -0.14582895 | 0.00022987 |
100 |
15.000| -0.12957000 | 0.00033802 |
110 |
16.500| -0.11631732 | 0.00025752 |
120 |
18.000| -0.06945640 | 0.00019346 |
----------------------------------------------------------------------------Done.
If it Doesn’t Work ...







Read the Thorn Writers chapter in the documentation
Remember that the Fortran code is preprocessed, to see the
postprocessed code that is actually compiled, look in
<config>/build/<thorn> (wave/build/WaveToyF77)
Run with a high warning level and read all the warning
messages
Review the parameters for the thorns you are using, they
often have some debugging features (PUGH, enable_all_storage)
Submit bug reports via www.CactusCode.org
Join the relevant mailing lists via www.CactusCode.org
Contact cactusmaint@cactuscode.org