Group 3: Data Intensive Cloud Computing Services
LIDAR and Beyond
This document will detail the steps needed to go from a selected OpenTopography LiDAR area, download the DAYMET data for
the specified site, and as the end result, have a file saved on the HPC you’re computing on with all resulting solar insulation
model outputs in it.
Assume everything is done in the command line unless otherwise specified with a URL/other.
1.
2.
Have access to an HPC system with OpenStack Havana available for use on it.
a. This iteration and all commands in this document are run on Future Grid’s India HPC.
Navigate to Open Topography and select your preferred LIDAR CZO data zone
(http://opentopo.sdsc.edu/gridsphere/gridsphere?cid=datasets)
a. For this guide’s example, we used the Southern Sierra Nevada CZO Snow Off
(http://opentopo.sdsc.edu/gridsphere/gridsphere?cid=geonlidarframeportlet&gs_action=lidarDataset&open
topoID=OTLAS.042013.26911.2)
b. Below copied from Tyson Swetnam: (https://pods.iplantcollaborative.org/wiki/display/ACIC/Data+Sources)
1.
Select subheading 'LiDAR Point Cloud & Processing'
a.
Select any one of the CZO LiDAR flights:
i.
Boulder Creek (snow-off)
ii.
Christina River Basin (April)
iii.
Eel River
iv.
Jemez River Basin (snow-off)
v.
Southern Sierra (snow-off)
vi.
Susquehanna Shale Hills (leaf-off)
Create the Bare Earth Digital Elevation Model (DEM, units in elevation), Zmean vegetation height model, and the point density
models

1a. Select area of data to download or process

Select a sub-set area within one of the Critical Zone Observatory sites

OpenTopography only allows you to select <200,000,000 points at a time - if the area is too large you will
get an Warning message and be asked to re-select a new area.

Check to make sure only one type of Return Classification is turned on (e.g. Ground or Unclasssified) at a
time - this will reduce the total point load.

1b. Choose Return Classification

Select only the 'Ground' classified points for the Bare Earth model


First we are going to generate the Bare Earth DEM (using the TIN option in #3b)
At the same time we can calculate the point count density of the point butt (using the Local Gridding Option
in #3a)


Both of these outputs can be generated in the same submission request.
For the second step, select only 'Unclassified' points for the Zmean (biomass height) model - you will do this AFTER
you've submitted a request for the Bare Earth DEM

There is an option after you've submitted your request to 'Modify and Resubmit Job'

Clicking on that hyperlink takes you back to your same job with the previous parameters
selected

You can reset the parameters to use the unclassified points rather than the Ground points for the
second request.

2. Point Butt Data Download

We don't need any of the point butt, so you can unselect all files


It appears that OpenTopography still generates the LAS file for you regardless of whether you selected it
3a.DEM Generation (Local Gridding)

We will only use the Local Grid option for the Bare Earth model to calculate the point count grid density. In the
second job request we will also select the Zmean option.

Gridding Method

For both the Bare Earth model and the Zmean models we want to calculate the point butt density
- the density layers will be used to calculate the 'Leaf Area Index' later on.

Select 'Calculate point count grid'

In the second job request, Select 'Calculate Zmean grid'


This will produce a point density layer for the resolution of the pixel
We don't want the Zmean of the Bare Earth Layer, only calculate this for the
Unclassified points

Zmean - average elevation of non-ground unclassified points


where Z = z-axis or elevation, mean = average elevation
Gridding Parameters

Grid Resolution: 10m

For right now let's try 10 meters - but, we may need to back out to a larger resolution,
e.g. 30 meters, for computational limitations

Remember the default is 1 meter! These layers are really cool to look at - but may be
computationally too expensive even for our HPC!


Grid Format

Null Filling



Radius value: leave set as the default value
Set to 'GeoTiff'
Set to '3'
We also want to calculate the density of the points (so we can come up with Leaf Area Index estimates later on)

Gridding Method

Select 'Calculate Point count grid'

We are going to run this twice - first with the Ground Classified points and then with
the Unclassified points

Gridding Parametres

Grid Resolution: 10m





Same option as above - we will start with 10m resolution
Radius value: default
Grid Format
Set to 'GeoTiff'





3b. DEM Generation (TIN) Option

Grid Method

Gridding Parameters





Null Filling
Set to '3'
Check on 'Calculate TIN'
Grid Resolution: 10 meters
Max Triangle Size: leave as Default (50)
Grid Format: GeoTiff
4. Derivative Products


You can unselect these for right now - unless you want to look at the hillshade in a GUI
The derivatives that are created are (1) a Hillshade and (2) a Slope model


The Hillshade is used for visualziation purposes
The Slope Model is used for calculating the Hydrological Terrain products (#6)

Because OpenTopo breaks up this slope model separately from the DEM we will not use this
option and instead create our own slope in GRASS later on.

5. Visualization


Same as above -unselect this option unless you plan on looking at your outputs in a GUI
FYI this produces a 'hillshade' or 'relief' model that is easy on the eyes to look at topography, e.g. the Google Maps
Terrain Surface in the viewer is a type of relief model.

6. Hydrologic Terrain Analysis Products (TauDEM)

Select 'Hydrologically Correct DEM'

We want to generate a hydrologically correct DEM - this will be the actual layer we use in the later
modelling exercises



Select 'D-Infinity Flow Direction'
Select 'D-Infinity Specific Catchment Area'
Do not select 'D-8 Flow Direction'

This uses a simpler 8 directional calculation - whereas the D-Infinity option gives proportional flow path
weights


Do not select 'D-8 Contributing Area'
Select 'Topographic Wetness Index'

3.
a.
4.
This is a model derived from the Catchment area divided by the slope.
Download the resulting Local Gridding tar.gz and extract the output.dem.tif file, and download the openTopo metadata file as well.
Upload these files into your chosen HPC by your method of choice
Follow the steps 5.2.1 through 5.2.9 at (http://futuregrid.github.io/manual/openstackhavana.html) to setup a running Ubuntu OpenStack image.
a.
In step 5.2.8 change the line that says “ —image “futuregrid/ubuntu-12.04” \ “ to “ “ —image “futuregrid/ubuntu14.04” \ “
b.
Depending on the resources you’re allocated, the tutorial suggested m1.small flavor might be sufficient, but this can also be scaled up to
the large or xlarge for more computational power
5.
Once you have ssh’ed into your image,:
a.
**go out and download our tar file**
b.
Pick your favorite method to copy the output.dem.tif file and the metadata file from OpenTopo into your currently running OpenStack
image. Place the output.dem.tif into the DEM’s folder and the metadata file just in the top level folder.
c.
$ sudo ./install.sh
d.
the install.sh script will run for a while and install all the necessary tools and dependencies needed to make all the grass commands and
everything else work on this image.
e.
f.
$ ./sol.sh <OpenTopo DEM> <OpenTopo Metadata> <start year> <end year>
i.
arg1: the absolute path to the OpenTopo DEM data file
ii.
arg2: the absolute path to the OpenTopo Metadata file
iii.
arg3,4: the start and end years to get the DAYMET data of the region for
$ ./getTiles.sh <absolute path to metadata file> <first year> <last year>
i.
arg1 ex: /home/username/metadata-18592775924791045765
ii.
arg2 ex: 2012
iii.
arg3 ex: 2013
iv.
this command will download all the DAYMET data for the specified OpenTopo dem region