An ESG Walkthrough
-ESG Federation website
-- DCC File system for ESG
Muhammad Atif
ESG-NCI Gateway
Search
We highly recommend
subscribing to our
tweets
Latest news
Search and Access Data



You can search without login; but not download
Quick Links → Create Account

Follow the on screen instructions

You will receive an email to confirm your
registration; at the same time the administrators
are also notified.

Admin has to validate you before you can download
the data.
After confirmation email from admin; login to the NCI
Node

Account → Apply for Group Membership
 Mk-3.6
 CMIP-5 research (not in our control – request
goes to PCMDI)

Requests to NCI are usually quick.

Same ID can be used on all the gateways.
Your OpenID: https://esg.nci.org.au/esgcet/myopenid/<username>
Searching for data
•Recommend that you browse the website and get familiarized.
•If asked about authentication, you may use a temporary openid
https://esg.nci.org.au/esgcet/myopenid/dcc000
Password: abc123
Please note that this openid will be removed after the workshop, it is highly recommended
that you create your openid
Download data from the Gateway

Three Download Methods



Using the web browser.
A set of wget-scripts.
Via GridFTP (Data mover lite)
Download Method-1 (Web browser based)

Intuitive but slow


Works like normal downloads from the browser
of your choice i.e. click to download.


Follow on-screen instructions and nothing can go wrong.
IE, FF, Chrome and Safari are supported.
Works well if you are after couple of files
Download -2 (Wget Scripts)

Ability to download multiple files




Select the files (variables) you are interested in.
Presents a wget-download.sh script, that you need to save and run.
Command line based – No GUI
Two methods of Authentication

Authorization token (Depreciated)



My Proxy Login (Official)

Needs a separate step for authentication.

Need to run Java applet or MyProxyClient software
If authentication expires, just run the MyProxyClient software


PCMDI gateway only.
No login required, However the token expires in 24 hours and
the script is of no use after that.
Note: If you are interested in doing lots of downloads, we can provide a custom
script to speedup the process on DCC………
Example to follow later
Process for MyProxyLogon
Download on the DCC
•ssh abc123@dcc.nci.org.au -Y
•Download MyProxyLogon-ESG.jar file
•wget http://esg.nci.org.au/esgcet/webstart/myProxyLogon/MyProxyLogonESG.jar
•Run MyProxyLogon.jar file;
•module load java
•Java –jar MyProxyLogon-ESG.jar
•It writes the certificates to your $HOME/.esg folder
•Run the wget-download.sh command
Download-3 (DML)

Parallel Downloads


Faster than wget


DML Preferences Concurrency
Uses GridFTP
Caveat: Not available on all ESG nodes

NCI one of the few that has the facility
ESG Data on the DCC

IPCC AR5/CMIP5




CSIRO-QCCCE Mk-3.6
CAWCR ACCESS
Replicated data from the other ESG nodes.
Other data



CMIP3
Observational data
Processed data
DCC File system organization

All ESG data in /projects/ESG:
•
•
•

/projects/ESG/Authoritative




Authoritative
Unofficial-ESG-replica
CAWCR_CVC_processed
Serves data using the policies of the ESG Federation
This is the directory that our ESG software serves data from
All data is the current official copy
User example: login to the DCC and have a look.
Unofficial Replica

/projects/ESG/Unofficial_Replica


IPCC
 The IPCC directory is where you can reference data that we have
downloaded from other nodes (though not an official replica).
The subdirectories could be partial datasets or complete ones.
IPCC_tmp_flat



Direct symlinks to files, flat directory structure
tmp
 You can download your data here in $USER folder.
 We can provide you with scripts to help download data here
GlobalObs_and_Reanalysis
 data sourced from various places that Lawrie Rikus/Ben Hu have
been maintaining.
 Also served through a THREDDS service - for remote access
Unofficial Replica

/projects/ESG/Unofficial-ESG-replica/IPCC

User downloads using wget scripts/DML.

Partial data; Not all of the data is downloaded.

Does not necessarily contain the most up to date version



Data may be changed by the remote node since last
download.
ESG (and official replica directory) always has the latest
version.
Organised as Data Reference Syntax (DRS)
Data Reference Syntax (How files are organized @ DCC)
• This is how the tree looks like compared to DRS
• DRS
cmip5.<product>.<institute>.<model>.<experiment>.
<time_frequency>.<realm>.<cmor_table>.<ensemble>
• File System
/projects/ESG/unofficial-ESG-replica/IPCC/CMIP5/output1/NCC/NorESM1M/historical/mon/seaIce/OImon/r1i1p1/v20110901/sic/<FILE>
Downloading data to DCC File System
•If you would like a significant amount of data that
we don’t have, then … please contact us.
•Reasons:
• It may already be downloaded but not linked
• Downloading data is still tricky
• Space management
•That said – we would like to facilitate downloads
of priority data.
•How? …. Lets do it
Demo
•
•
•
•
•
•
•
•
•
Download the wget file from esg-gateway
ssh –Y user@dcc.nci.org.au
java –jar MyProxyLogon-ESG.jar
Copy wget-file to dcc (scp, copy n paste) in a new folder
./esg-download.py wget-download.sh
View the directory, it should have a number of wget-split-*
./esg-qsub-download.py –i wget-splitPress “y”
Check the files after some time
We will be streamlining it further
Help
• DCC and ESG both are evolving continuously
• Comments and suggestions are always welcome
• Help Desks
– Anything related to ESG federation website/ other
models are not native to NCI
• Cmip5-helpdesk@stfc.ac.uk
– Related to DCC compute cluster
• help@nf.nci.org.au
•Downloads are managed by Us
•GridFTP
•Fast
•Downloads managed by you as a user

Controlled Vocab:

http://esg-pcmdi.llnl.gov/internal/esg-data-nodedocumentation/cmip5_controlled_vocab.txt
ESG-NCI Gateway
Search by
categories
We highly recommend
subscribing to our tweets
Search and Access Data



You can search without login; but not download
Quick Links → Create Account

Follow the on screen instructions

You will receive an email to confirm your
registration; at the same time the
administrator(s) are also notified.

Admin has to validate you before you can
download the data.
After confirmation email; login to the NCI Node

Account → Apply for Group Membership
 Mk-3.6
 CMIP-5 research (not in our control –
request goes to PCMDI)

Requests to NCI are usually quick.

For others, this may take time (one – two days)
Searching for data
Download data from the Gateway

Three Methods



Using the web browser.
A set of wget-scripts.
Via GridFTP (Data mover lite)
Download – 1 (Web based)

Intuitive but slow


Works like normal downloads from the browser
of your choice


Follow on-screen instructions and nothing can go
wrong.
IE, FF, Chrome and Safari are supported.
Works well if you are after couple of files
Download -2 (Wget Scripts)

Ability to download multiple files



Presents you with a wget-download.sh script.
Command line based – No GUI
Two methods

Authorization token (Depreciated)



PCMDI gateway only.
My Proxy Login (Official)

Needs a separate step for authentication

Need to run Java applet or MyProxyClient software
Note: If you are interested in doing lots of downloads, we
can provide a custom script to speedup the downloads on
DCC.
Process for MyProxyLogon
Download on the DCC
•ssh abc123@dcc.nci.org.au -Y
•Download MyProxyLogon-ESG.jar file
•wget http://esg.nci.org.au/esgcet/webstart/myProxyLogon/MyProxyLogonESG.jar
•Run MyProxyLogon.jar file; instructions are provided in the wget download
script that you have already downloaded via web-browser.
•module load java
•Java –jar MyProxyLogon-ESG.jar
•It writes the certificates to your $HOME/.esg folder
Download-3 (DML)

Parallel Downloads


Faster than wget


DML Preferences Concurrency
Uses GridFTP
Caveat: Not available on all ESG nodes

NCI one of the few that has the facility
Interacting with ESG data on DCC
ESG Data on the DCC

IPCC AR5/CMIP5




CSIRO-QCCCE Mk-3.6
CAWCR ACCESS
Replicated data from the other ESG nodes.
Other data



CMIP3
Observational data
Processed data
DCC File system organization

All ESG data in /projects/ESG:
•
•
•

/projects/ESG/Authoritative
 Serves data using the policies of the ESG Federation



Authoritative
Unofficial-ESG-replica
CAWCR_CVC_processed
This is the directory that our ESG software serves data from
All data is the current official copy.
User example: login to the DCC and have a look.
Unofficial Replica

/projects/ESG/Unofficial_Replica


IPCC
 The IPCC directory is where you can reference data that we have
downloaded from other nodes (though not an official replica).
The subdirectories could be partial datasets or complete ones.
IPCC_tmp_flat



Direct symlinks to files, flat directory structure
tmp
 You can download your data here in $USER folder.
 We can provide you with scripts to help download data here
GlobalObs_and_Reanalysis
 data sourced from various places that Lawrie Rikus/Ben Hu have
been maintaining.
 Also served through a THREDDS service - for remote access
Unofficial Replica

/projects/ESG/Unofficial-ESG-replica/IPCC

User downloads using wget scripts/DML.

Partial data; Not all of the data is downloaded.

Does not necessarily contain the most up to date version



Data may be changed by the remote node since last
download.
ESG (and official replica directory) always has the latest
version.
<MOVE TO NEW SLIDE>Organised as Data Reference Syntax
(DRS)
cmip5.<product>.<institute>.<model>.<experiment>.<time_frequency>.<realm>.
Downloading data to our ESG
•If you would like a significant amount of data that
we don’t have, then … please contact us.
•Reasons:
• It may already be downloaded but not linked
• Downloading data is still tricky
• Space management
•That said – we would like to facilitate downloads
of priority data.
•How? …. (new slide)
•Downloads are managed by Us
•GridFTP
•Fast
•Downloads managed by you as a user

Controlled Vocab:

http://esg-pcmdi.llnl.gov/internal/esg-data-nodedocumentation/cmip5_controlled_vocab.txt