Tutorial on the Manipulation of Netcdf Climate Data:
Application to Statistically Downscaled Climate Projections
Michael Notaro
Nelson Institute Center for Climatic Research
University of Wisconsin-Madison
mnotaro@wisc.edu
608-261-1503
20th Annual Conference of The Wildlife Society
STATISTICAL DOWNSCALING
Observational Data
The source of observed data was roughly 4100 weather stations measuring daily maximum/minimum temperature and
precipitation during 1950-2006 in ten of the central-eastern North American Landscape Conservation Cooperatives
(LCCs), from the NWS Cooperative Observer Program.
Model Projection Data
The climate output that was analyzed was produced by 9 GCMs from the Coupled Model Intercomparison Project
Phase 3, according to 3 emission scenarios (A2, A1B, B1). The coarse climate projections were downscaled to a 0.1° lat
x 0.1° lon grid over 10 LCCs and debiased against observed temperature/precipitation from station observations within
the NWS Cooperative Observer Program. By interpolating and debiasing probability distribution functions and their
attributes, a realistic representation of the variance and extremes of temperature/precipitation was achieved, in
addition to a realistic representation of the means of both variables. In computing the projected changes in a variable,
the difference in the mean of that variable was computed from 1961-2000 to either 2046-2065 or 2081-2100.
NCO
CMIP3 Models
CCCMA_CGCM3_1 = Canadian Centre for Climate Modelling and Analysis
CNRM_CM3 = Meteo-France / Centre National de Recherches Meteorologiques
CSIRO_MK3_0 = CSIRO Atmospheric Research, Australia
CSIRO_MK3_5 = CSIRO Atmospheric Research, Australia
GFDL_CM2_0 = US Department of Commerce / NOAA / Geophysical Fluid Dynamics Laboratory
GISS_MODEL_E_R = NASA / Goddard Institute for Space Studies, US
MIUB_ECHO_G = Meteorological Institute University of Bonn, Germany
MPI ECHAM5 = Max Planck Institute for Meteorology, Germany
MRI_CGCM2_3_2a = Meteorological Research Institute, Japan
Project Funding
The statistical downscaling project was funded by: (1) Michigan DNR through an EPA grant with the Great Lakes
Restoration Initiative, (2) Upper Midwest and Great Lakes LCC, and (3) Wisconsin Focus on Energy. The downscaling
was performed by Dr. David Lorenz.
Interactive mapping website for examining the downscaled data
http://ccr.aos.wisc.edu/resources/data_scripts/LCC
NCO
NCDUMP
NCVIEW
NCO
TOOLS
NCL
Additional tools:
Text editor: Emacs or vi
Ghostview
BASIC UNIX COMMANDS
cd = change directory
Example: cd /data/Share/workshop/20c3m
ls = list files in a directory
NCO
pwd = list the name of directory you are in
mkdir = create a directory
Example: mkdir dirname
rm = delete file
Example: rm filename
rmdir = delete directory
Example: rmdir dirname
emacs = text editor
mv = move (or rename) file
Example: emacs = opens emacs
Example: mv oldfile newfile
Example: mv oldfile /data/Share/workshop/ Example: emacs & = opens emacs in background
cp = copy file
Example: cp oldfile newfile
vi = text editor
Example: vi = opens vi
Example: vi filename = opens old or new file
UNIX Tutorials:
http://kb.iu.edu/data/afsk.html
http://www.math.utah.edu/lab/unix/unix-commands.html
http://people.ischool.berkeley.edu/~kevin/unix-tutorial/toc.html
Change directory
cd /data/Share/workshop/20c3m
Open ncview and look at a
NCO file for 1990.
temperature
Ampersand puts ncview in
background, so you can still
type in terminal.
ncview temp_01_1990.nc &
Click on “tmax” button
NCO
Variable: Maximum
daily temperature
365 Days, beginning
Jan 1, 1990
Range of values 18.06°C to 40.825°C
Move your pointer
across map & notice
the x,y coordinates
and associated
temperature values
change
Click on Opts to
set options
Choose USA
states
NCO
Click OK
State boundaries
should appear
Quit = Exit ncview
3gauss = Toggle among color schemes
Inv P = flip the map vertically
NCO
Inv C = Invert color map
Range = Modify the max/min
displayed values
Bi-Lin = Either show raster boxes or
interpolated values
Print = Save the plot as a postscript
file or send to a printer
Click on ?
The
variable tmax is
NCO
“maximum daily
temperature” of type short
with units of °C and a missing
value of -32768
Click on <<, <, ||, >, and >> to
moveNCO
backwards/forwards in
time, just like on a DVD
player.
Click on any grid cell on map.
A time series will appear of
daily maximum temperatures
during 1990.
MoveNCO
your pointer over time
series to see data values.
Click on Print button to save
the plot as a postscript file.
Click on Dump button to
write actual data values to a
text file.
Click NCO
on 4 different grid cells
on the map to compare their
time series.
NCDUMP
ncdump –h prcp_01_1982.nc
NCO
NCDUMP
ncdump –c prcp_01_1982.nc
List the actual latitudes, longitudes, times
lon = -115.3, -115.2, -115.1, ……, -67
lat = 24.6, 24.7, 24.8, ……, 54.7
timeNCO
= 0, 1, 2, ……, 364
The resolution is 0.1 degree.
ncdump –k prcp_01_1982.nc
Tells you “netCDF-4”
Possible outcomes: Classic, 64-bit offset, netCDF-4, netCDF-4 classic model
NCO
The temperature files, such as temp_01_1982.nc, contain multiple variables
(tmax, tmin). Let’s say we want to create a new file, only containing tmin.
ncks: “kitchen sink”
NCO
ncks –v tmin temp_01_1982.nc newfile.nc
(-v stands for variable)
Now, to verify that it worked, use ncdump.
ncdump –h newfile.nc
The file only contains tmin, not tmax. So it worked.
NCO
The downscaling data covers a large region, 24.6-54.7N, 115.3-67W.
What if we wanted to extract precipitation data in Wisconsin only?
ncks –d lat,42.,47.5 –d lon,-93.5,-86.5 prcp_01_1996.nc small.nc
(remember
NCO to use decimal points)
To check the data,
ncview small.nc
Our dataset is now much smaller, only
containing data around Wisconsin.
NCO
How can we compute the mean precipitation for July 1985?
On the Julian calendar, starting with 0, July runs from day 181 to 211.
First, extract only the July days from the file.
NCO
ncks –d time,181,211 prcp_01_1985.nc july.nc
Now, use the record averager, ncra.
ncra july.nc julymean.nc
We can view the results with ncview.
ncview julymean.nc
NCO
How can we compute the difference between the mean annual daily
maximum temperature in 1990 and 1989?
First, compute the annual mean for 1989.
ncraNCO
temp_01_1989.nc 89.nc
Second, compute the annual mean for 1990.
ncra temp_01_1990.nc 90.nc
Third, compute the difference between the
two files using the operator, ncdiff.
ncdiff 90.nc 89.nc diff.nc
Finally, view the file with ncview.
ncview diff.nc
NCO
How can we extract daily precipitation (mm/d) from 1995 for Madison, Wisconsin and
dump the values to a text file?
Use ncdump to check the content of file prcp_01_1995.nc.
ncdump –h prcp_01_1995.nc
NCO
Notice that variable prcp is type short, with scale factor of 0.025 & offset of 819.175.
If we print out values, they will not be easily used. They need to be converted to float.
ncap2 –s ‘prcp=float(prcp)’ prcp_01_1995.nc float95.nc
This created file float95.nc, containing precipitation in
easily readable values.
Now, let’s extract data values for Madison only.
ncks –d lat,43.07 –d lon,-89.40 float95.nc madison.nc
Finally, let’s pipe those values to a text file.
ncdump madison.nc > madison.txt
Open madison.txt with a text editor (emacs, vi) to see the result.
NCL: NCAR Command Language
Every script should begin with load commands, to access key libraries.
I’ll only show them once here, but always include them at the top.
In a text editor, create a file called hello.ncl. Insert the following text and save.
NCO
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl”
print(“hello”)
* To save time, cut and paste these load commands out of file loadcommands.
Now, run the script by:
ncl script.ncl
The word hello should be printed to the screen. You have run your first
NCL script.
NCL: NCAR Command Language
Given that the downscaled data is so large, it was written in a compact format,
specifically netcdf4. We need to specify that we’re working with this unique
type of data.
setfileoption
NCO = specifies the type of data we’ll be working with
setfileoption(“nc”,”Format”,”NetCDF4Classic”)
Then, we can open the netcdf4 file.
addfile = opens a data file
“r” means readable, “w” is writable
This command allows us to open these types of data files:
netcdf, grib, hdf4, hdfeos, hdf5, hdfeoss, ccm, shapefile
a=addfile(“temp_01_1991.nc”,”r”)
NCL: NCAR Command Language
We can retrieve the latitude and longitude values from the file.
lat=a->lat
lon=a->lon
NCO
We can print the latitude values.
print(lat)
NCL: NCAR Command Language
Our current script: script.ncl
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
loadNCO
"$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl”
setfileoption(“nc”,”Format”,”NetCDF4Classic”)
a=addfile(“temp_01_1991.nc”,”r”)
lat=a->lat
lon=a->lon
print(lat)
NCL: NCAR Command Language
Let’s also retrieve the variable tmax from the file. Remember that tmax
was in a short format and so we want to convert it to float.
short2flt = Converts short format into float, using the offset and scale factor
NCO
We can check the attributes/dimensions of our new variable using
the command printVarSummary.
setfileoption(“nc”,”Format”,”NetCDF4Classic”)
a=addfile(“temp_01_1991.nc”,”r”)
lat=a->lat
lon=a->lon
tmax=short2flt(a->tmax)
printVarSummary(tmax)
Now run the script.
Notice the dimensions (time,lat,lon)
and type float.
NCL: NCAR Command Language
Let’s try to print out the daily maximum temperature values for Madison
from 1991.
Recall that tmax had dimensions of time,lat,lon.
NCO
When specifying the closest grid cell to Madison,
provide the latitude and longitude inside { }.
To acquire all time values, specify : for the
time dimension.
setfileoption(“nc”,”Format”,”NetCDF4Classic”)
a=addfile(“temp_01_1991.nc”,”r”)
tmax=short2flt(a->tmax)
print(tmax(:,{43.07},{-89.40}))
NCL: NCAR Command Language
Let’s collect some basic statistics on a data file, specifically 1985’s precipitation.
We’ll use the functions: avg, min, and max.
setfileoption("nc","Format","NetCDF4Classic")
a=addfile("prcp_01_1985.nc","r")
NCO
prcp=short2flt(a->prcp)
print(avg(prcp))
print(min(prcp))
print(max(prcp))
NCL: NCAR Command Language
Let’s say we want to plot a time series of
Madison’s precipitation during 1985.
First, let’s retrieve Madison’s values and
printNCO
them to the screen as a check.
setfileoption("nc","Format","NetCDF4Classic")
a=addfile("prcp_01_1985.nc","r")
prcp=short2flt(a->prcp)
madison=prcp(:,{43.07},{-89.40})
print(madison)
NCL: NCAR Command Language
We want to plot time on x-axis and precipitation on y-axis.
First, create a “workstation”, wks.
We will be creating a postscript (ps) plot, named script.ps.
NCO
Temperature
setfileoption("nc","Format","NetCDF4Classic")
a=addfile("prcp_01_1985.nc","r")
prcp=short2flt(a->prcp)
madison=prcp(:,{43.07},{-89.40})
wks=gsn_open_wks(“ps”,”script”)
Time
NCL: NCAR Command Language
We will use the function gsn_csm_xy to make an XY plot.
gsn_csm_xy(workstation, x-axis variable, y-axis variable, resource)
For the x-axis, we will use “ispan(1,365,1)” which means a range of
NCO
integers
spanning from days 1 to 365 at a 1-day interval.
For simplicity, assign “False” to resource options in gsn_csm_xy.
setfileoption("nc","Format","NetCDF4Classic")
a=addfile("prcp_01_1985.nc","r")
prcp=short2flt(a->prcp)
madison=prcp(:,{43.07},{-89.40})
wks=gsn_open_wks(“ps”,”script”)
plot=gsn_csm_xy(wks,ispan(1,365,1),madison,False)
Run the script.
To view the plot, use ghostview in command line: gv script.ps
NCL: NCAR Command Language
Let’s now try plotting a spatial map of the mean climatological maximum
temperatures for March (averaged across 1961-2000).
This information is contained in file temp_mean_1961_2000.nc.
NCO
Let’s use ncdump to check the file’s content.
ncdump –h temp_mean_1961_2000.nc
Notice the dimensions of tmax:
time 12 (one value per month)
latitude 302
longitude 484
NCL: NCAR Command Language
Dimensions of tmax are time, lat, lon.
To retrieve March only and all latitudes/longitudes, specify we want
tmax(2,:,:). Remember the counting starts at zero, so March=2.
NCO
setfileoption("nc","Format","NetCDF4Classic")
a=addfile("temp_mean_1961_2000.nc","r")
lat=a->lat
lon=a->lon
tmax=short2flt(a->tmax)
march=tmax(2,:,:)
J
0
F
1
M
2
A
3
M
4
J
5
J
6
A
7
S
8
O
9
N
10
D
11
NCL: NCAR Command Language
setfileoption("nc","Format","NetCDF4Classic")
a=addfile("temp_mean_1961_2000.nc","r")
lat=a->lat
lon=a->lon
tmax=short2flt(a->tmax)
NCO
march=tmax(2,:,:)
Opens a new workstation
wks = gsn_open_wks ("ps", "script")
Plot will be called script.ps
res=True
res@cnFillOn=True
res@gsnAddCyclic=False
res@mpMinLatF=min(lat)
res@mpMaxLatF=max(lat)
res@mpMinLonF=min(lon)
res@mpMaxLonF=max(lon)
res@mpOutlineBoundarySets = "AllBoundaries”
plot = gsn_csm_contour_map_ce(wks,march, res)
NCL: NCAR Command Language
setfileoption("nc","Format","NetCDF4Classic")
a=addfile("temp_mean_1961_2000.nc","r")
lat=a->lat
lon=a->lon
tmax=short2flt(a->tmax)
NCO
march=tmax(2,:,:)
wks = gsn_open_wks ("ps", "script")
Creates a new set of
res=True
resources named res
res@cnFillOn=True
res@gsnAddCyclic=False
res@mpMinLatF=min(lat)
res@mpMaxLatF=max(lat)
res@mpMinLonF=min(lon)
res@mpMaxLonF=max(lon)
res@mpOutlineBoundarySets = "AllBoundaries”
plot = gsn_csm_contour_map_ce(wks,march, res)
NCL: NCAR Command Language
setfileoption("nc","Format","NetCDF4Classic")
a=addfile("temp_mean_1961_2000.nc","r")
lat=a->lat
lon=a->lon
tmax=short2flt(a->tmax)
NCO
march=tmax(2,:,:)
wks = gsn_open_wks ("ps", "script")
res=True
Specify to turn color fill on
res@cnFillOn=True
res@gsnAddCyclic=False
res@mpMinLatF=min(lat)
res@mpMaxLatF=max(lat)
res@mpMinLonF=min(lon)
res@mpMaxLonF=max(lon)
res@mpOutlineBoundarySets = "AllBoundaries”
plot = gsn_csm_contour_map_ce(wks,march, res)
NCL: NCAR Command Language
setfileoption("nc","Format","NetCDF4Classic")
a=addfile("temp_mean_1961_2000.nc","r")
lat=a->lat
lon=a->lon
tmax=short2flt(a->tmax)
NCO
march=tmax(2,:,:)
wks = gsn_open_wks ("ps", "script")
res=True
res@cnFillOn=True
Our data is not global, or
res@gsnAddCyclic=False
“cyclic”
res@mpMinLatF=min(lat)
res@mpMaxLatF=max(lat)
res@mpMinLonF=min(lon)
res@mpMaxLonF=max(lon)
res@mpOutlineBoundarySets = "AllBoundaries”
plot = gsn_csm_contour_map_ce(wks,march, res)
NCL: NCAR Command Language
setfileoption("nc","Format","NetCDF4Classic")
a=addfile("temp_mean_1961_2000.nc","r")
lat=a->lat
lon=a->lon
tmax=short2flt(a->tmax)
NCO
march=tmax(2,:,:)
wks = gsn_open_wks ("ps", "script")
res=True
res@cnFillOn=True
res@gsnAddCyclic=False
res@mpMinLatF=min(lat)
res@mpMaxLatF=max(lat)
res@mpMinLonF=min(lon)
res@mpMaxLonF=max(lon)
res@mpOutlineBoundarySets = "AllBoundaries”
plot = gsn_csm_contour_map_ce(wks,march, res)
Specify the area to plot
NCL: NCAR Command Language
setfileoption("nc","Format","NetCDF4Classic")
a=addfile("temp_mean_1961_2000.nc","r")
lat=a->lat
lon=a->lon
tmax=short2flt(a->tmax)
NCO
march=tmax(2,:,:)
wks = gsn_open_wks ("ps", "script")
res=True
res@cnFillOn=True
res@gsnAddCyclic=False
res@mpMinLatF=min(lat)
res@mpMaxLatF=max(lat)
res@mpMinLonF=min(lon)
res@mpMaxLonF=max(lon)
res@mpOutlineBoundarySets = "AllBoundaries”
plot = gsn_csm_contour_map_ce(wks,march, res)
Show state and country
boundaries
NCL: NCAR Command Language
setfileoption("nc","Format","NetCDF4Classic")
a=addfile("temp_mean_1961_2000.nc","r")
lat=a->lat
lon=a->lon
tmax=short2flt(a->tmax)
NCO
march=tmax(2,:,:)
wks = gsn_open_wks ("ps", "script")
res=True
res@cnFillOn=True
res@gsnAddCyclic=False
res@mpMinLatF=min(lat)
res@mpMaxLatF=max(lat)
res@mpMinLonF=min(lon)
res@mpMaxLonF=max(lon)
res@mpOutlineBoundarySets = "AllBoundaries”
plot = gsn_csm_contour_map_ce(wks,march, res)
Plot a contour map
of march temperatures,
using resources “res”
NCL: NCAR Command Language
setfileoption("nc","Format","NetCDF4Classic")
a=addfile("temp_mean_1961_2000.nc","r")
lat=a->lat
lon=a->lon
tmax=short2flt(a->tmax)
march=tmax(2,:,:)
NCO
wks = gsn_open_wks ("ps", "script")
res=True
res@cnFillOn=True
res@gsnAddCyclic=False
res@mpMinLatF=min(lat)
res@mpMaxLatF=max(lat)
res@mpMinLonF=min(lon)
res@mpMaxLonF=max(lon)
res@mpOutlineBoundarySets = "AllBoundaries”
plot = gsn_csm_contour_map_ce(wks,march, res)
Now, run the script and use ghostview to display the plot.
ncl script.ncl
gv script.ps
NCL: NCAR Command Language
NCO