GEOS-Chem structure, documentation, platforms, and parallelization

advertisement

GEOS-CHEM: Structure,

Documentation, Platforms, and Parallelization

Bob Yantosca

Software Engineer

Atmospheric Chemistry Modeling Group

Harvard University

GEOSCHEM Users’ Meeting

02 June 2003

Topics

1. GEOS-CHEM history

2. GMAO met fields used to drive GEOS-CHEM

3. GEOS-CHEM model description

4. GEOS-CHEM standard versions and benchmarks

5. GEOS-CHEM user support and documentation

6. GEOS-CHEM platforms

7. Parallelization: OpenMP and MPI

8. Effective coding practices

9. Chemical forecasting

2 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

In the beginning …

 GEOS-CHEM was originally created from:

 The old Harvard/GISS-II 9-layer model (Larry, Yuhang, et al)

 GEOS-CTM (Dale, Mian) – ancestor of GOCART

 Pieces that came from the Harvard/GISS-II model:

 Emissions (EMISSDR, BIOBURN, etc)

 Dry Deposition (DRYDEP, DEPVEL, etc.)

 Chemistry (SMVGEAR etc.)

 Diagnostics (NDxx) and various input files

 Pieces that came from the GEOS-CTM model:

 Transport (TPCORE, by Lin & Rood)

 BL mixing (TURBDAY) and Cloud Convection (NFCLDMX)

3 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

In the beginning …

GEOS-CHEM evolution:

 We did not foresee all of the potential uses of GEOS-CHEM at the time we first created GEOS-CHEM.

At first, newer code was “onion-skinned” on top of older code.

This resulted in functional, albeit, convoluted code.

 However, a number of recent developments (new met fields, mechanisms, etc.) has forced us to make some widespread changes to the GEOS-CHEM source code.

 These problems are not unique to GEOS-CHEM, but also apply to any software product.

4 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

In the beginning …

 Philosophical Digression:

 “If builders built buildings the way programmers write programs, then the first woodpecker that came along would destroy civilization.” – Weinberg’s Law

 “Inside every large program there is a small program struggling to get out.” – Hoare’s Law of Large Programs

 Interpretation:

 If you want to build a new building, you tear down the old one first and start from scratch.

 With software, usually you start with an existing program and add new stuff on top of that. Your foundations may be shaky…

5 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

In the beginning …

 Much of the existing GEOS-CHEM code has been recently updated, or brand-new code has been added

 Biomass and biofuel emissions

 Dry and wet deposition

 Date/time functions

 Error handling and file I/O

 Sulfate chemistry

 Main program

 Some sections of GEOS-CHEM still need improvement

 Diagnostics (especially timeseries)

 Input file reading

 Emissions for full chemistry (incl. Isoprene, Soil NOx)

6 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GMAO met fields

 GEOS-CHEM is driven by assimilated met data from

NASA/GMAO (formerly DAO), located at GSFC.

 GMAO has produced 4 major met data sets:

 GEOS-1 (1985 – 1995, 20 layers to 10 hPa)

 GEOS-STRAT (1995 – 1997, 46 layers to 0.1 hPa)

 GEOS-3 (2000 – 2002, 48 layers to 0.01 hPa

 GEOS-4/fvDAS (2003 –, 55 layers to 0.01 hPa)

 At present, GEOS-CHEM can run with GEOS-1, GEOS-

STRAT, and GEOS-3 met fields.

 GEOS-4 support is currently being added to GEOS-CHEM.

7 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GMAO met fields

 Native horizontal resolution of GMAO met fields:

 GEOS-1 2 o

 GEOS-STRAT 2 o lat x 2.5

o lon lat x 2.5

o lon

 GEOS-3 1 o

 GEOS-4/fvDAS 1 o lat x 1 o lon lat x 1.25

o lon

 We also can regrid met fields to coarser resolution in order to gain computational advantage for long simulations

 GEOS-1 4

 GEOS-STRAT 4 o o lat x 5 o lat x 5 o lon lon

 GEOS-3 2

 GEOS-4/fvDAS 2 o o lat x 2.5

o lat x 2.5

o lon AND 4 o lon AND 4 o lat x 5 o lat x 5 o lon lon

 Regridding of met fields is done offline, and can be automated

8 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GMAO met fields

 I-6 fields (6hr instantaneous “snapshot” quantities)

 U, V, P, T, Specific humidity, albedo

 A-6 fields (6-hr averaged quantities)

 Cld. Frac. – random & max overlap (GEOS-1, GEOS-S)

 In-cloud optical depth and total cld frac (GEOS-3)

 Tendency in specific humidity (i.e. precip or evap)

 Cloud mass flux and detrainment

 A-3 fields (3-hr averaged quantities)

 10m winds, roughness height, friction velocity

 PBL height, column cloud fraction, surface temperature

 Convective & total precipitation at ground

9 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GMAO met fields

A-6 fields

A-3 fields

Times of day when you need to read met fields

Z = “Zulu” = abbreviation for GMT

GEOS-1

GEOS-S

GEOS-3

A-6 fields

A-3 fields

From the fvDAS file spec on GMAO’s website

10

GEOS-4

Start of day

GEOS-CHEM: Structure, Platforms, ...

Start of next day

Monday, June 2, 2003

GMAO met fields

 GEOS-1, GEOS-STRAT, GEOS-3

 A-6 fields are stamped w/ center times: 00, 06, 12, 18 GMT (read them at 21, 03, 09, 15 GMT)

 A3 fields are stamped w/ end times: 00, 03, 06, … 21 GMT

 GEOS-4 / fvDAS

 A-6 fields are stamped w/ center times: 03, 09, 15, 21 GMT (read them at 00, 06, 12, 18 GMT )

 A-

3 fields are stamped w/ center times: 1:30, 4:30, 7:30, … 22:30

 GEOS-4 timestamps and averaging periods differ from

GEOS-3, GEOS-1, and GEOS-STRAT!

11 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GMAO met fields

 Preparing GEOS-CHEM for GEOS-4/fvDAS met fields:

1. Download, regrid, and format fvDAS met fields. (in progress)

2. Adapt code to read in fvDAS met fields w/ different timestamps and averaging intervals. (in progress)

3. Implement new TPCORE transport code for fvDAS. This contains new semi-lagrangian vertical advection. (done)

4. Fit Philip Cameron-Smith's pressure fixer to the fvDAS TPCORE transport scheme. Add additional mixing ratio adjustment for residual mass difference to ensure mass conservation. (done)

5. Implement new convection schemes, J. Hack's scheme for moist convection and Phil Rasch's scheme for convective transport. (in progress)

6. Run GEOS-CHEM with GEOS-4/fvDAS meteorology. Examine results, compare to observations and previous model simulations (yet to be done)

12 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM description

 GEOSCHEM is currently 8 models in one…

 4 x 5 GEOS-1

 4 x 5 GEOS-STRAT

 4 x 5 GEOS-3

 2 x 2.5 GEOS-1

 2 x 2.5 GEOS-STRAT

 2 x 2.5 GEOS-3

 1 x 1 GEOS-3 (nested grid over China)

 1 x 1 GEOS-3 (nested grid over N. America)

… soon to be 12 models in one!

 1 x 1 GEOS-3 (nested grid over Europe)

 4 x 5 GEOS-4

 2 x 2.5 GEOS-4

 and (maybe not too soon) 1 x 1.25 GEOS-4

13 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM description

 GEOS-CHEM can perform several different kinds of chemistry simulations, which are selectable at run time.

The main simulation is “full chemistry” (i.e. Ozone, etc.):

 Chemistry solver: SMVGEAR package by Mark Jacobson

 Photolysis code: FAST-J by O. Wild & M. Prather

 300+ chemical reactions

 50+ photolysis reactions

 80+ chemical species

 31 tracers (i.e. species which are transported)

• NOx, Ox, HC’s, DMS, SO2, SO4, MSA, NH3, NH4, Sulf. Nitrates

 Heterogeneous chemistry on aerosol surfaces

 Aerosol thermodynamic equlibrium (RPMARES)

14 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM description

In addition to the “full chemistry” simulation, GEOS-CHEM also can perform other chemistry simulations:

 222 Rn – 210 Pb – 7 Be

 Tagged CO (loss w/ archived OH)

 Tagged Ox (with pre-saved P-L rates)

 Methane

 Ethane

 Methyl Iodide

 CO with parameterized OH (B. Duncan)

 Offline sulfate aerosol

All of the “extra” chemistry simulations listed above come standard with GEOS-CHEM

 However, some simulations may be in need of updating

 Check w/ author of simulation before using it

15 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM description

GEOS-CHEM source code structure

 Input and initialization

Dynamic loop

 Read daily and monthly and various monthly quantities

 Interpolate instantaneous (I-6) fields

 Strat fluxes and Transport

 PBL mixing and deep cloud convection

 Diagnostics

 Dry deposition

 Emissions breakpoint

 Chemistry breakpoint

 Wet deposition

Save output to files and quit

16 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM description

 GEOS-CHEM diagnostic output quantities include:

 Tracer concentrations

 OH, NO, NO2, HO2, fields

 Photolysis rates (J-values)

 Emissions

 Cloud and aerosol optical depths

 Chemical production and loss

 Dry deposition fluxes and velocities

 Methyl chloroform lifetime

 Wet scavenging and wet deposition losses

 Surface pressure, air mass, air density, etc.

 GEOS-CHEM can produce average and timeseries output

 You can use timeseries output to create 2-D or 3-D animations

17 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM description

GEOS-CHEM directory structure

 Source code directory (e.g. Code.v5-04 )

 Contains Makefiles and Fortran source code

 Run directory (e.g. run.v5-04 )

 Each user keeps run directories in his/her own disk space

 Contains input files which must be modified by the user in order to select the parameters for a particular GEOS-CHEM run

 GEOS-CHEM output files will be created in the run directory

 Users can keep separate run directories for different simulations

 Data directory (e.g. /data/ctm/GEOS_2x2.5

)

 Located in a common disk space available to all users

 Contains emissions, aerosol files, LAI, tropopause, etc, etc.

18 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

Std versions and benchmarks

 Bob Yantosca currently maintains the standard GEOS-CHEM code at Harvard, using archive software (RCS, soon CVS?)

 Procedure for adding new code into the standard model:

1.

User downloads a standard GEOS-CHEM version from Harvard

2.

User adds new code into their own local copy of GEOS-CHEM

3.

User debugs and thoroughly tests his/her local GEOS-CHEM

4.

User submits a list of his/her modifications to Bob Y.

5.

Bob Y. incorporates modifications into standard GEOS-CHEM

6.

Bob Y. benchmarks the new standard GEOS-CHEM version

7.

Daniel examines the new standard GEOS-CHEM version

8.

Upon Daniel’s blessing, Bob Y. releases GEOS-CHEM version

 Single point of code modification

19 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

Std versions and benchmarks

 Pros of single point of code modification :

 One person is responsible for entire standard model. This prevents divergent versions from propagating among users.

 Consistency is achieved in both coding style and naming files.

 Documentation and revision history are consistently maintained.

 Cons of single point of code modification

 If point person is otherwise occupied, turnaround time for incorporating new revisions can be very long.

 As the number of potential code developers grows, it is hard for one person to keep up with everyone’s submissions.

 The challenge: to incorporate many revisions from multiple developers into GEOS-CHEM in a timely fashion while preventing the model from diverging.

20 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

Std versions and benchmarks

222 Rn — 210 Pb — 7 Be Benchmark

 GEOS-3 meteorology w/ 48 layers @ 4x5

 1 year: 1 Jan 2001 – 1 Jan 2002

 Diagnostic quantities

 Tracer ratios w/ previous version (surface, 500hPa)

 Frequency Distribution of tracer ratios

 Tracer concentrations (surface, 500hPa)

 Tracer zonal means

 Budgets for Rn, Pb, Be

 Benchmark results are posted at Harvard and on the web site.

21 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

Std versions and benchmarks

 Full chemistry benchmark

 GEOS-3 meteorology w/ 30 layers @ 4x5

 1 month: 1 Jul 2001 – 1 Aug 2001

 Diagnostic quantities

 Tracer ratios w/ previous version (surface, 500hPa)

 Frequency distribution of tracer ratios

 J-Value ratios w/ previous version (surface, 500hPa)

 Tracer concentrations (surface, 500hPa) and zonal means

 Budgets for Ox and CO; CH3CCl3 lifetime

 Emissions table (comparison to previous version)

 Benchmark results are posted at Harvard and on the web site.

22 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

Std versions and benchmarks

 A TAR file w/ GEOS-CHEM source code and benchmark results is created for each new GEOS-CHEM version and is posted on the GEOS-CHEM website.

 Benchmark results are compared to that of the previous version. We especially focus on the global budgets, emissions, tracer ratios, and the frequency distribution of tracer ratios.

 Any major problems with a new GEOS-CHEM version should be evident from the benchmark simulation output.

 A new GEOS-CHEM version is only released after Daniel approves the benchmark results for that version.

23 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

User support & documentation

 GEOS-CHEM source code + benchmark output is available via website

 GEOS-CHEM user packages

 TESTRUN – script for compiling & running GEOS-CHEM jobs

(on website)

 GEOS-CHEM documentation

 GEOSCHEM Users’ Guide and FAQ (on website)

 GEOS-CHEM Style Guide (on website)

 Visualization packages

 GAMAP (needs IDL; available from GAMAP website @ Harvard)

 GRADS (free; see http://grads.iges.org/grads/grads.html)

 Matlab (student versions available; check w/ your IT dept)

24 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

User support & documentation

 How to download GEOS-CHEM source code and other related files from the web:

1.

Visit www-as.harvard.edu/chemistry/trop/geos/

2.

Click on Source Code & Data Files at the left of the page

3.

Enter the proper password (contact Bob Y.)

4.

Select the files that you wish to download

 The following TAR files are available for download:

 GEOS-CHEM source code w/ benchmark simulation output

 2 x 2.5 GEOS-CHEM run directories

 4 x 5 GEOS-CHEM run directories

 2 x 2.5 GEOS-CHEM data directories

 4 x 5 GEOS-CHEM data directories

25 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

User support & documentation

 TESTRUN

 Perl script for compiling & running GEOS-CHEM, located at: www-as.harvard.edu/chemistry/trop/geos/documentation/

 What TESTRUN does

1.

Creates two scripts: one for compiling GEOS-CHEM and another for running GEOS-CHEM

2.

Submits compilation script to local batch queue system

3.

After compilation has completed, submits run script to local batch queue system. This starts the GEOS-CHEM run.

4.

Pipes GEOS-CHEM output to a timestamped log file

5.

Appends a timestamp to GEOS-CHEM output files

Use TESTRUN to make sure you don’t accidentally overwrite files that you wanted to save!

26 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

User support & documentation

 GEOSCHEM Users’ Guide

 Is the official GEOS-CHEM manual page, located at www-as.harvard.edu/chemistry/trop/geos/documentation/

 The GEOSCHEM Users’ Guide covers the following topics:

1.

Introduction and brief overview of GEOS-CHEM

2.

Installing and compiling GEOS-CHEM

3.

Coding: practice and style

4.

Files contained in the GEOS-CHEM data directories

5.

Files contained in the GEOS-CHEM run directories

6.

Running and debugging GEOS-CHEM

7.

Vertical & horizontal resolution of GEOS-CHEM

8.

Met fields used in GEOS-CHEM

9.

GEOS-CHEM tracers, chemical species, and diagnostics

27 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

User support & documentation

 GEOS-CHEM Frequently Asked Questions (FAQ)

 Shortcuts to the proper section in the User’s Guide, located at www-as.harvard.edu/chemistry/trop/geos/documentation/

 The GEOS-CHEM FAQ covers the following topics:

1.

What’s new in GEOS-CHEM?

2.

What kind of simulations can we run with GEOS-CHEM?

3.

What is the horizontal resolution of GEOS-CHEM?

4.

What is the vertical resolution of GEOS-CHEM?

5.

What are the met fields used by GEOS-CHEM?

6.

What diagnostic quantities are archived by GEOS-CHEM?

7.

Which chemical species are used in the GEOSCHEM “full chemistry” mechanism?

8.

Who is using GEOS-CHEM?

28 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

User support & documentation

 GEOS-CHEM Style Guide

 Describes GEOS-CHEM coding & documentation practices

 See www-as.harvard.edu/chemistry/trop/documentation/

 The GEOS-CHEM Style Guide covers the following topics:

1.

Background and brief introduction to Fortran 90

2.

Adding documentation headers, indentation, white space

3.

Numeric and character data types

4.

New language features of Fortran 90

5.

DO loops

6.

Fortran 90 modules

 GEOS-CHEM Style Guide is continually being updated!

29 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

User support & documentation

 GAMAP

 Visualization package for GEOS-CHEM (and other models too!)

 See www-as.harvard.edu/chemistry/trop/gamap/

 Advantages of using GAMAP:

1.

Exploits IDL’s powerful mapping and image processing features

2.

Can read output from both GEOS-CHEM and GISS models

3.

Contains a user-friendly menu-driven interface

4.

GAMAP’s low-level routines can be called separately

5.

You can combine GAMAP’s mapping routines with IDL’s HDF,

HDF-EOS, and netCDF file I/O functions

6.

With the IDL ION package (sold separately!) you can develop an interactive web-page interface for GAMAP (cf. ITCT 2k2)

30 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

User support & documentation

 We expect GEOS-CHEM developers and users to be resourceful in troubleshooting and debugging

 Check input files and make sure code is debugged

 Inform Bob Y. of any major bugs that you discover

 Bob Y. can provide support for:

 GEOS-CHEM: coding issues and problems

 GAMAP: problems, bugs, and extensions

 Bob Y. cannot provide support for:

 General IDL or Fortran help (read the manual)

Bob Y. will “farm out” questions to other GEOS-CHEM users if he cannot answer them himself

31 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM platforms

 As of June 2003, GEOS-CHEM has been ported to the following shared-memory architectures

 SGI Power Challenge / Origin

 Compaq Alpha

 Linux (w/ Portland Group compiler)

 Sun / Sparc

 The standard code ships with Makefiles for each of the four supported platforms.

 At this time, GEOS-CHEM is being recoded to run on distributed-memory architectures w/ help from NASA/NCCS

 Hybrid “distributed-shared” memory Compaq Alpha at GSFC

32 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM platforms

Shared Memory: one “box” w/ many processors

 Each processor can see all of the memory within the “box”

Array in memory System Memory

Proc 1 Proc 2 Proc 3 Proc 4

33 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM platforms

Distributed Memory: standalone “nodes” w/ network cnx

 A processor can only see the memory on its own node

Mem 1 Mem 2 Mem 3

Proc 1 Proc 2

Proc 3a

Proc 3b

34 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM platforms

Hybrid “distributed-shared” memory (e.g. “Halem” at GSFC)

 300+ nodes, with 4 processors and 2 GB memory per node

 Halem is the 18 th fastest machine in the world (see top500.org)

2GB Memory per Node

Cnx. to other nodes

1 Node

Proc 1 Proc 2 Proc 3 Proc 4 Cnx. to other nodes

35 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM parallelization

 GEOS-CHEM achieves parallelization on shared-memory machines using the OpenMP parallelization commands

 OpenMP is an open standard for parallelization which is supported by many compiler vendors (see openmp.org)

 SGI

 Compaq / HP

 Sun

 IBM

 Portland Group (Linux compiler)

 NAG, etc.

 OpenMP commands parallelize individual DO loops

36 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM parallelization

!$OMP PARALLEL DO

!$OMP+DEFAULT( SHARED )

!$OMP+PRIVATE( I, J )

DO J = 1, 1000 ! We have to do this

DO I = 1, 1000 ! a million times!

A(I,J) = A(I,J) ** 1.35 ! Exponentiation is

ENDDO ! very computationally

ENDDO ! intensive!

!$OMP END PARALLEL DO

 The blue statements are OpenMP sentinels which tell the compiler that we want this DO loop to be executed in parallel

 (I,J) pairs are split up among the available processors

37 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM parallelization

 PRIVATE vs. SHARED

 Each processor needs to keep a local copy of some variables

 DO-loop indices (e.g. I, J) should be declared PRIVATE

 Arrays themselves can be declared SHARED, as long as array indices (I,J) are declared PRIVATE

 To optimize a parallel loop, make sure that it contains as much work as possible

 Parallelize the outermost loop

 Parallelize loops which require many iterations (such as looping over tracers, altitude, latitude, longitude)

 Parallelize loops containing computationally expensive operations (e.g. divisions, logarithms, exponentiations, etc).

38 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM parallelization

 Advantages of OpenMP

 Commands are easy to use

 Good scalability with increasing # of processors

 Cross-platform support is guaranteed

 Disadvantages of OpenMP

 Assumes machine has a shared-memory architecture

 Cannot use for distributed-memory machines, including:

• Beowulfs (i.e. PC’s hooked together running Linux)

Blade servers (IBM)

 On hybrid “distributed-shared” machines (e.g. Halem), OpenMP can only be used on individual nodes

39 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM parallelization

GEOS-CHEM v5-04 fullchem, 1 day

30 levels, 4x5 grid

1 Processor

Compaq Alpha

@ GSFC

4 Processors

23.3 min / 1 day

(97% utilization)

8.5 min / 1 day

(330% utilization)

SunFire 3800

@ Dalhousie

47.2 min / 1 day

14.6 min / 1 day

(396% utilization)

Speedup 2.7 times 3.2 times

Going from 1  4 processors results in a speedup of approximately 3 times on Alpha and Sun

40 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM parallelization

 OpenMP is not optimal for distributed memory machines, these require MPI (message passing interface) parallelization

 GEOS-CHEM can run as-is on Halem, but only on 1 node

 To use more than 1 node (4 processors) of Halem, we need to use MPI coding commands

For hybrid “distributed-shared” memory machines (e.g.

Halem), you can combine MPI and OpenMP:

 Use MPI to distribute memory between nodes

 Use OpenMP to operate among the processors of a single node

 Tom Clune & Jack Yatteau will update us on the status of recoding GEOS-CHEM for MPI.

41 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

Effective coding strategies

 Summary of good coding style

 Include lots of comments (Bob Y. doesn’t know much chemistry!)

 Add standard documentation headers to each routine

 Use lots of white space – readability prevents bugs!!

 Indent code – makes it easier for eye to follow!

 Place new code into F90 modules

 Use allocatable arrays (in modules) instead of COMMON blocks

 Use new F90 language features whenever possible

 See the GEOS-CHEM Style Guide for more details

42 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

Chemical forecasting

 Past aircraft campaigns for which GEOS-CHEM was used to generate real-time forecasts

 NASA TRACE-P (2001)

 NOAA ITCT 2k2 (2002)

 Future aircraft campaigns in which GEOS-CHEM will be used to generate real-time forecasts

 NASA INTEX-NA (2004)

 NOAA ITCT 2k4 (2004)

 For TRACE-P and ITCT 2k2, GEOS-CHEM forecasts were generated on computers at Harvard

 In the future we may run the GEOS-CHEM forecasts offsite

(maybe at GMAO???) We’re not sure yet.

43 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

Marketing strategies (?)

44

GEOS-CHEM license plates ????

(photo courtesy P. Palmer)

GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

Extra slides …

45 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GMAO met fields

Comparison of GEOS met fields up to 3 km

46 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GMAO met fields

Comparison of GEOS vertical levels to ~32 km

47 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GMAO met fields

1 x 1 nested grid: China

48 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GMAO met fields

1 x 1 nested grid: North America

49 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GMAO met fields

1 x 1 nested grid: Europe

50 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM description

 New GEOS-CHEM code is written using F90 modules

 Advantages of F90 modules:

 Arrays and routines can be grouped together in a single “librarylike” unit. This allows for more convenient packaging.

 Arrays and routines in modules can be referenced by other modules elsewhere in the code

 Module arrays can be dynamically allocatable – better memory management

 Internal variables and routines can be “hidden” from view to calling routines outside of the module – black box!

 Modules can be used to centralize operations which may have previously been computed in several different places

 Existing routines can be packaged into F90 modules as well

51 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM description

 More about F90 modules

 You can use PRIVATE declaration to hide module variables or routines from all other external subroutines or modules

 You can select which variables or routines to import into another program via: USE MY_MODULE, ONLY : MY_SUB

 If Module B references Module A, then Module A must be compiled before Module B

 You can “overload” module routines with INTERFACE s so that they can accept different argument types

 More references

 A Programmer’s Guide to Fortran 90

 GEOS-CHEM Style Guide (on website)

52 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM description

GEOS-CHEM timesteps

 Dynamic timestep (transport, wetdep, archive diags)

 10 min (1 x 1)

 15 min (2 x 2.5)

 30 min (4 x 5)

 Convective timestep (PBL mixing, cloud convection)

 Same as dynamic timestep

 Emissions and dry deposition timestep

 Once per hour

 Photolysis and chemistry timestep

 Once per hour

53 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

Std versions and benchmarks

 GEOS-CHEM v5-04 is the last released standard version

 Added nested-grid capability and cleaned up lots of old code

 GEOS-CHEM v5-05-03 is the current development version

 SMVGEAR II chemistry solver (cf. M. Jacobson)

Written by Mark Jacobson (then at UCLA)

• More recent version of our sparse matrix GEAR solver

 New transport module (TPCORE) for GEOS-4/fvDAS winds

Written by S-J Lin (GSFC)

• Now with Mass Conservation! (from LLNL)

 GEOS-CHEM v6-01 will be the first version to contain full support for fvDAS met fields (release date ????)

 New cloud convection scheme (cf. P. Rasch) for fvDAS

54 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM description

MODULE MY_MODULE

IMPLICIT NONE ! Force declarations

REAL*8, ALLOCATABLE :: ARRAY(:,:) ! Allocatable array

CONTAINS

SUBROUTINE MY_SUB( X ) ! Subroutine example.

REAL*8, INTENT(IN) :: X ! Of course, GEOS-CHEM

ARRAY(:,:) = ARRAY(:,:) * X ! code is much more

END SUBROUTINE MY_SUB ! complex

SUBROUTINE INIT_MY_MODULE ! Subroutine to allocate

ALLOCATE( ARRAY( 100, 100 ) ) ! And initialize the array

ARRAY(:,:) = 0

END SUBROUTINE INIT_MY_MODULE

SUBROUTINE CLEANUP_MY_MODULE ! Subroutine to

IF ( ALLOCATED( ARRAY ) ) DEALLOCATE( ARRAY ) ! Deallocate the

END SUBROUTINE CLEANUP_MY_MODULE ! array

END MY_MODULE

55 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

GEOS-CHEM description

PROGRAM MODULE_TEST

USE MY_MODULE ! References all arrays and

! routines in MY_MODULE

IMPLICIT NONE ! Force explicit declarations

CALL INIT_MY_MODULE ! Initializes and allocates

! module arrays

ARRAY(:,:) = 100.0d0 ! Assignment using array mask

CALL MY_SUB( 2.0d0 ) ! Multiplies ARRAY by 2.0

PRINT*, MINVAL( ARRAY ) ! and prints min & max

PRINT*, MAXVAL( ARRAY )

CALL CLEANUP_MY_MODULE ! Deallocates module arrays

END PROGRAM MODULE_TEST

56 GEOS-CHEM: Structure, Platforms, ...

Monday, June 2, 2003

Download