United States Department of the Interior
NATIONAL PARK SERVICE
INTERMOUNTAIN REGION
12795 West Alameda Parkway
PO Box 25287
Denver, Colorado 80225-0287
No hard copy to follow
April 12, 2012
Memorandum
To: Dustin Perkins
From: Bruce Bingham, Inventory and Monitoring Program, IMR
Subject: Approval of Land Cover and Land Use Protocol for the Northern
Colorado Plateau Network
Dear Dusty:
I have reviewed your request for approval of the land cover/land use standard operating
procedures and submission of supporting documentation. The methods NCPN has adopted
have received sufficient peer review through their publication in reputable journals and I see
no need for further peer review. I am pleased to hear of your plans to submit the first report
applying the methods to NCP parks for publication in a peer reviewed journal. I look
forward to seeing the report once completed. Congratulations to you and your staff on the
excellent progress NCPN continues to make in completing NCPN monitoring protocols.
Sincerely,
/s/ Bruce Bingham
Deputy Chief
Inventory and Monitoring Division
1
Landsat Imagery for Landscape Scale Change Detection and Trend Analysis
– NCPN Application
Landsat-based detection of Trends in Disturbance and Recovery (LandTrendr) and TimeSync are a set of
tools that extract spectral trajectories of land surface change from yearly Landsat time-series image stacks
(LTS). Developed by Oregon State University and the Pacific Northwest Research Station LandTrendr
identifies important features in time-series signatures by minimizing the influence of noise caused by
changes in illumination, phenology, atmosphere and geometric registration (Kennedy et al., 2010).
Temporal trends in reflectance are evaluated on a pixel by pixel basis by regression-based and point-topoint fitting. LandTrendr is accompanied by TimeSync to help interpret temporal patterns in reflectance
by allowing analysts to visually compare segmentation and fitted functions with yearly spectral values
(Cohen et al., 2010). The combination of algorithms embodied in LandTrendr and TimeSync enable
quantitative description of change at pixel and landscape scales. Specifically, these tools allow NCPN to
determine the start, end, duration and rate of disturbance as well as the magnitude of disturbance and
recovery. The evolution of project work leading to implementation of LandTrendr and TimeSync in
NCPN applications informs the appropriate application of these tools and the kind of interpretation that
can be made.
Pilot Studies
Over the course of several years, collaborators at OSU worked with staff from the N&SCPN on a
multi-stage project to evaluate the potential utility of Landsat imagery for monitoring vegetation
on the Colorado Plateau. The initial stage of the project focused on baseline and single-timeframe mapping. Because of the difficulties encountered at this stage, later work focused on
detection and interpretation of change, using increasingly sophisticated methods that
incorporated longer time-series of imagery.
Baseline mapping
Identification of change requires both information about the starting condition and the creation of
models to identify vegetation types from the Landsat imagery. Thus, substantial effort was
initially aimed at baseline mapping. Baseline maps from the NPS Vegetation Mapping project
are both extremely rich in information and generally high in spatial detail, but are limited to the
parks proper and their immediate vicinity and to a single occasion. Expanding the temporal and
spatial scope with Landsat imagery would provide context for monitoring within the parks. Work
focused on Mesa Verde (MEVE) and Canyonlands (CANY), which had no NPS Vegetation
Mapping products at the time of the studies, and at Zion, which did have a completed NPSVM
map.
Key conclusions from this stage of the project included

As expected from the initial literature review, the soil signal dominates the spectral
response in most of the areas of N&CPN, making vegetation discrimination difficult
under any circumstance when Landsat data are used. This is a limitation inherent to the
data, regardless of the spectral indices being used or the data analysis method employed.

Mapping vegetation components with Landsat is done most robustly for forest and
woodland vegetation cover. When thematic maps such as those from the NPSVM
program are available, aggregation to simpler, spectrally-separable classes is feasible and
relatively accurate.

Available reference data are generally inadequate for the purposes of training and testing
maps with Landsat data. New airphoto interpretation can fill some gaps, but is costly,
and even airphoto interpretation lacks the separability needed to go beyond aggregated
class labels with confidence.
Because of these challenges, the N&SCPN and OSU groups determined that the potential utility
of Landsat data for baseline mapping alone was lower than that desired by the parks, but that
some of the methods explored for baseline mapping may be relevant when used in the context of
change detection.
Two-date change detection
Detection of landscape change was thus the next major focus. Change detection studies focused
on MEVE, ZION, and Wupatki National Monument (WUPA). At the onset of the project, the
N&SCPN were interested in the potential to apply the method known as “POM change
detection” (for “probability of membership” change) that OSU had developed for the parks of the
North Coast and Cascades Network (NCCN). However, unlike the situation in the NCCN,
existing land cover maps were available, and thus new methods to aggregate and merge the
existing landcover maps with the spectral strategies of the POM were developed. Also, because
of the different ecosystem properties of the two networks and the expected difficulties of
working in soil-dominated systems, OSU also examined other change detection approaches
based on statistical models between the imagery and airphoto-interpreted estimates of component
cover characteristics.
Key conclusions from this stage of the project included

Reference data are an even greater bottleneck for many change-detection approaches than
they are for single-date mapping, as reference data must be both appropriate for remotelysensed models and consistent across time. Rarely are such data available. Validation of
change results is thus particularly challenging.

Detection of landscape change is possible using either continuous-variable maps or the
POM (probability of membership) approach originally envisioned for these projects. The
former approach is primarily limited to forest and woodland cover types. Landcover
classes in the POM-change approach were limited to fairly broad labels that may not be
useful for some the N&SCPN’s goals.

Any map of change using only two dates of imagery is suspect in these ecosystems
because vegetation phenology and year-to-year variance in precipitation cause many
potential false-positive spectral changes.
As was the case with baseline mapping, these results showed that the methods tested were
appropriate for some uses, but the level of detail needed by the N&SCPN were greater than was
generally feasible. Thus, on both of the primary efforts of the initial agreement between OSU
and the parks, the potential utility of monitoring with Landsat imagery was inconclusive at best.
Many-date change detection
Rather than map landscape change by comparison of two dates of imagery, OSU developed
methods to detect trends across time-series of many years of imagery. A primary benefit of this
approach is that year-to-year changes caused by phenology, sun angle, or ephemeral weather
events can fade to noise around longer-term trends. Also, change is defined for each pixel
relative to its own spectral trajectory, not relative to other pixels. In theory, the influence of
cross-pixel variation in soil brightness would become less of an impediment than for more
traditional methods. Additionally, the availability of stacks of imagery allows for more robust
interpretation of the satellite imagery itself for validation using tools developed in-house at OSU
(known as “TimeSync”). If these interpretation tools were applicable in the CP, many of the
potential problems with reference data could potentially be avoided. Because of these potential
advantages, a small pilot project was funded under a new agreement between OSU and the
N&SCPN to investigate the potential utility of the new method (termed “LandTrendr”) in the
parks.
Key conclusions from this stage of the project included:

LandTrendr segmentation captured many of the key disturbance processes of interest,
including fire, insect mortality, and apparent drought disturbance, as well as
growth/recovery processes

Change mapping in herbaceous areas would likely require new rules to link spectral
change to increase or decrease in cover, but these new rules would likely be generally
applicable once found

Linking POM labeling to LandTrendr was technically successful, resulting in yearly
landcover maps with potentially large information content.

However, POM labeling required aggregation of classes that altered the detailed class
labels

Additionally, the spectral conditions present directly after a change result in labels that
often do make ecological sense; true labeling will require significant effort to develop
post-disturbance spectral class labels that are different from those applicable to the rest of
the landscape

TimeSync interpretation was much more challenging in the non-forest and non-woodland
areas of the CP than expected, suggesting that this validation approach is still not
sufficient to replace inadequate reference data from existing sources.
At a teleconference in August 2009, OSU and the CP networks made progress toward evaluating
which of the methods may be useful for actual implementation in the parks of the CP. The
LandTrendr change maps appear to be the best balance of cost and utility, and the POM
approaches (with two-date and LandTrendr + POM) are likely too costly to justify the further
research needed to fully understand and implement them.
NCPN Applications are Focused Studies
LandTrendr and TimeSync are “one size fits all-wall to wall” processing algorithms in the sense that they
work everywhere at the pixel scale. Thus in one sense they are an implementable protocol that can be
applied in all NCPN parks to detect long-term change that cannot be detected at broad scales any other
way. However, these are sophisticated resource intensive tools that require “tuning” for optimum change
detection depending on the change of interest (slow, abrupt, disturbance, recovery). Furthermore, longterm change assessments at broad scales are unlikely to be necessary in every park every year. For these
reasons NCPN plans to implement LandTrendr on a project basis to address management relevant
questions appropriate to the tools such as the following.
Based on the potential demonstrated in the many-date change detection pilot study NCPN contracted with
OSU to investigate management-driven questions at DINO and CARE. Relevant questions at DINO
center on understanding dynamics of pinion/juniper, sage and the river corridor vegetation condition. At
CARE change in vegetation condition in grazing allotments and along the Fremont River are of interest.
In all of these cases either Parks or NCPN have ancillary data to help interpret the cause of observed
change and its magnitude as measured on the ground. As learning and analytical approaches for using
both LandTrendr output and ancillary data together mature we expect the techniques to be transferrable to
other park units.
Kennedy, R. E., Yang, Z., & Cohen, W. B. (2010). Detecting trends in forest disturbance and
recovery using yearly Landsat time series: 1. LandTrendr — Temporal segmentation
algorithms. Remote Sensing of Environment, 114(12), 2897-2910. Elsevier Inc.
doi:10.1016/j.rse.2010.07.008
Cohen, W. B., Yang, Z., & Kennedy, R. (2010). Detecting trends in forest disturbance and
recovery using yearly Landsat time series: 2. TimeSync — Tools for calibration and
validation. Remote Sensing of Environment, 114(12), 2911-2924. Elsevier B.V.
doi:10.1016/j.rse.2010.07.010
LandTrendr Quickstart Guide
Author: Robert Kennedy (robert.kennedy@oregonstate.edu)
Overview
LandTrendr (Landsat Detection of Trends in Disturbance and Recovery) describes a set
of algorithms that extract useful landscape dynamics information from a dense temporal
stack of Landsat Thematic Mapper images. The overall workflow involves three broad
steps: preprocessing, temporal segmentation, and mapping. The details of the
processing steps are described in "Standard Operating Procedures" (or SOPs).
NOTE for Beta Users: The SOPs contain significant detail and are the product of
collaborative efforts of many individuals. By giving Beta users the authority to edit our
SOPs, we expect you to help us improve consistency in these SOPs by making
corrections to any small errors they find. If there are issues that you cannot resolve,
please note them in the "Items to do/resolve for LandTrendr SOPs" worksheet. It is
easiest to view this document as a form: Open the worksheet in Google Docs, then
under the "Form" menu item, select "Go to live form", where you can enter any notes
with a simple entry format.
Processing:
To handle stacks efficiently, it is important to maintain consistent file naming and
directory structures, for the algorithm and for your own sanity. Please see SOP #8
(Data management), Section 4 to set up your directory structure BEFORE
you do anything else. It is best to handle processing on a scene basis (i.e. one "scene"
is the Landsat path/row of interest; it is composed of many years of "images").
All processing steps are conducted in the IDL language. You must have either IDL or
ENVI+IDL to run LandTrendr.
The core library of LandTrendr code should be stored in a central location on your
computer (for example, C:\landtrendr_code\). This central library will be called from
other locations as you process different image stacks. This code will be updated
periodically to fix bugs, so make sure you know where you put the code, and make sure
there is only copy of the code on your machine.
Processing each image stack is handled through IDL batchfiles. Take the template
batchfiles provided, copy them to the "idl_batchfiles" subdirectory in a given scene's
directory structure (you set that up using SOP #8, right?), and then update them for the
paths, filenames, and parameter sets on your own system.
To run the batchfiles, make sure the IDL preferences in your IDL session have the
startup directory pointing to the idl_batchfiles subdirectory, and then also make sure to
add the folder with the core library of LandTrendr code in the "path" section of the
preferences.
Batchfiles are run from the command line of IDL, e.g. IDL>
@set_up_files_landtrendr_4629
As you progress through the processing steps, the algorithms build and populate an IDL
structure variable that retains all of the relevant information about each stack. This
variable is called "image_info", and is stored in the "image_info_savefile" file whose
name you first assign in the "set_up_files...." batchfile. You must make sure that
everything in the image_info structure is the way you want it -- that all images get found
in the set_up_files step, that the cloud masks show up for all years, etc. It is best to
check on the image_info structure frequently as you progress through processing.
All output images are in flat binary, primarily signed-16 bit type. The file names are
given the ".bsq" extension to signify band-sequential formatting.
LandTrendr Steps
Use the S
These are the most common steps needed to run LandTrendr:
1.
2.
3.
4.
Identify and download images (SOP 1)
Set up a pre-processing study area (SOP 2)
Atmospherically correct a single reference image (SOP 2)
Use "@set_up_files_......" batchfile to identify all relevant 6-band Landsat images
in a given scene (SOP 2)
5. Check the image_info structure to make sure they're all there (See section VIII in
SOP 2 for tips)
6. Use "@mad_landtrendr_..." to normalize the rest of the images in the stack to
the single reference image (SOP 2)
7. Use "@make_landtrendr_tcimages..." to create tasseled-cap images from the
normalized images for the entire stack (SOP 2)
8. Use "@make_cloudscore_images..." to create cloud and shadow score images
from the TC images (SOP 2)
9. Use ENVI to identify thresholds in the cloud-score and shadow-score images that
separate clouds and shadows (SOP 2)
10. Enter those values into the "@mpcm_...." batchfile to create cloud masks.
11. Check the masks of a few images to make sure it worked as expected.
12. Use "@lt045_...." to run the LandTrendr segmentation algorithms on the stack
(SOP 3)
13. Use "@lt045_..._changemaps" to create various map outputs (SOP 4).
14. Check on the veracity of your results using TimeSync (SOP 5)
Some useful tips
To check on the status of the image info, make sure that you have done a restore,
image_info_savefile, and type something like this:
print, image_info.image_file
print, image_info.year
print, tag_names(image_info)
print, image_info[0]
pst, image_info[0]
If the image info gets screwed up somewhere along the way, it's probably easiest to just
rebuild it.
1. Re-run "@set_up_files...." batchfile, and confirm that the image info has all of the
image years and dates.
2. type "repopulate_image_info, image_info_savefile", where "image_info_savefile"
has been set previously to point to the filename of the save file.
The repopulate image info routine will scour the directories that were found in the
@set_up_files... step, and find any files from madcal, cloudscoring, or cloud masking
steps.
When running madcal, it is tempting to only check the band correlations in the .csv file
to determine which images worked right.
DON'T limit yourself to that, however. For any images that do not look stellar, take a
look at the scatterplot .tif files within each image's directory. Sometimes an image will
have a decent overall correlation, but one band may be completely screwed up. If that is
the case, then re-run madcal on just that image, using different parameter values (more
constrictive) or more targeted subsets.
One of the most common ways to break things is to have coordinates somewhere that
are either outside of bounds, or not an integer multiple of pixels away from each other.
The simplest way to avoid this is to use a single usearea file (mask image, which defines
the zone where processing should occur) throughout the whole process (from the
@set_up_files... step through the segmentation phase).
Protocol for LandTrendr
Standard Operating Procedure (SOP) #1
Acquiring imagery
Version 0.01 (October 1, 2009)
Revision history log:
Previous Version Revision Date Author Changes Made Reason for Change
New Version #
New
April 12, 2009 REK
New
0.01
Aug 1, 2009
PN
Populated
0.01
Sep 2, 2010
PN
revised
clarification and consistency
1.
a. Overview
i. I. Landsat Thematic Mapper: Basic information
ii. II. File structure and naming
1. A. Data storage consideration
2. B. Creation of file structure
3. C. Monitoring imagery processing
iii. III. Strategies for selecting imagery for Landtrendr
1. A.. Dates (seasonal and phenological considerations)
2. B. Clouds
3. C. Scene drift
4. D. Missing years
5. E. Trends in imagery dates
iv. IV. Acquiring imagery
1. A. Identifying imagery
2. B. Record your image selections
3. C. Downloading and Ordering imagery
0.01
OVERVIEW
An overview of the LandTrendr processing work flow involves many steps. First, Landsat images for
each year between 1984 to present, primarily during July or August time period, is obtained from the
Landsat data archive (SOP 1). Second, COST atmospheric correction, MADCAL radiometric
normalization, and cloud masking are performed on the images in the yearly stack (SOP 2). Third, the
image stack is analyzed using the LandTrendr process (SOP 3) which summarizes and labels the
Landsat observed long-term trends from year to year. Next, the LandTrendr output results are evaluated
for their accuracy using independent data visualization using TimeSync (SOP 5). Finally, the analysis
and final products are documented and archived (SOP 8).
This SOP describes how to find, evaluate, and download Landsat Thematic Mapper data for use in
LandTrendr-based change detection projects. Because image downloading represents the first step in
data management, this SOP also describes the directory structure for file folders to most efficiently handle
the large volumes of data that will be processed in LandTrendr work.
Figure 1. SOP 1 within the LandTrendr process.
I. LANDSAT THEMATIC MAPPER: BASIC
INFORMATION
The Thematic Mapper is a sensor mounted on the Landsat family of satellites that acquires digital
pictures of the planet's surface by measuring reflected sunlight. The Thematic Mapper sensor has been
included on Landsat satellites since 1984. Currently, two Thematic Mapper sensors are actively recording
imagery: one on Landsat 5 and one on Landsat 7 (also known as ETM+, for Enhanced Thematic Mapper
Plus). For this protocol, the relevant digital information is recorded in footprints on the ground
represented by square picture elements (pixels) with a size of approximately 30m on a side. Imagery is
acquired at the satellite as it orbits the Earth, and then is downloaded to various ground-receiving
stations. For imagery over the continental U.S., images are stored at a central repository in South
Dakota, the EROS (Earth Resources Observation Systems) Data Center (EDC). Information on the center
can be obtained at that center’s web site: http://edc.usgs.gov/about/background.html. Information about
the Landsat family of sensors is found at http://landsat.usgs.gov/.
Currently, the Landsat Archive imagery is being processed and made available for use through a system
which creates a standardized Terrain-corrected image product. This process begins when each pixel in
the image can be linked directly with a map coordinate on the ground, in coordinates determined by a
map projection and an associated spheroid and datum. The spheroid represents a model of the Earth and
the datum a set of coordinates developed through actual measurements of points on the ground.
A key step in linking the geometric properties on the ground to a given image is accounting for the
potential distortion effects of topography and the curvature of the Earth. When Landsat TM measures
reflectance in a particular pixel at the center of the image, it is viewing that pixel straight down, but when it
measures reflectance at the edge of the image, it is viewing that pixel at an angle. In the latter case, the
elevation of the land can cause distortion in the apparent position of the pixel from the perspective of the
sensor: pixels at the tops of mountains will appear to be farther away than they really are, and pixels in
valleys will appear closer. The curvature of the Earth also interacts with the view angle to affect apparent
position of the pixel. Therefore, a geometric model that includes both curvature and the effects of surface
topography is needed to link the observed pixels with their actual locations on the ground. Once the
effects of topography and view angle have been accounted for, the image is said have been terrain
corrected.
As of February 2009, all Landsat Thematic Mapper imagery (as well as imagery from its predecessor, the
MultiSpectral Scanner [MSS]) were made available for free acquisition to the public.
II. FILE STRUCTURE AND NAMING
A. Data storage consideration
1. Amount of data storage needed The files associated with LandTrendr for a single Landsat scene can be of considerable size. It is
recommended that data storage of 120 to 140 GB be set aside for a single Landsat scene for a 25 year
period. If multiple Landsat scenes are being processed, such as for a region or park, the use of an
external hard drive with a large amount of data storage can be used (500 GB to 1 TB). The use of
external hard drives allows easier transportation of the drive, such as between collaborators, if necessary.
2. Back up data Due to the large amount of processing time to get a final LandTrendr product it is wise to backup your
data. The use of a backup or slave external drive and a data backup software program such as SyncBack
may be useful. Note: Pay close attention to backup preferences as files can be lost if not chosen
correctly.
B. Creation of file structure
Creating file folders for Landsat scenes - A logical and simple file structure is necessary to keep the large
amounts of files used and created by LandTrendr well organized. Figure 2 displays a typical file structure
used to organize LandTrendr files. See SOP 8: Data Management for further details.
Figure 2. Typical file structure for LandTrendr
processing of a Landsat scene.
C. Monitoring imagery processing
Use of a processing workbook, such as Figure 3, to track the Landtrendr progress, changes, and results
allows collaborators to quickly assess the processing of the imagery stack. The 'Journal’ spreadsheet
should be updated whenever a task is completed so that all parties are aware of the progress of the
stack, which can avoid misunderstandings and repetition of finished tasks.
1. Worksheets within the workbook
Each major processing task has a tab in the spreadsheet (Figure 3). In addition there is also a Journal tab
which should be updated whenever there are changes so as to offer at a quick glance the progress of the
processing of the stack. The following are the tabs of the processing spreadsheet.
a. Journal
A simple summary of the processing progress of the scene. The who, what, and when of theprocessing.
b. Image status
This is where the specifics of eachimage are tracked. Here you make mention of the reference images.
This is also good place to make mention of specific issues with images. It also helps you evaluate the
date range of your imagery stack.
c. Archive Image Order
This is the location to store information on the images that needed to be ordered from the archive. Fields
include image date, Glovis ID,order number, progress of order, and comments.
d. COST reference
This is where the dark objects values and other COST related values are housed that are later inputted
into the COST model.
e. Madcal
The madcal tab is where copies of the .csv files from the resultant madcal runs are stored. This should
include all runs used to get better results for difficult images (fixes).
f. Cloudscreen
The cloudscreen tab is where the cloudscreening thresholds are recorded prior to creating the cloud
mask batchfile (mpcm).
2.
Updating the processing workbook during imagery acquisition
a. Journal worksheet
The journal tab should be updated whenever one of the main tasks for processing has been completed. It
is also a good idea to mention any notable problems or anything else that would help a collaborator
assess the progress of the preprocessing of the stack.
b. Imagery status worksheet
Most of the work with this tab occurs during or shortly after the imagery acquisition step. In regards to
listing the raw data files the user may want to wait until the archive files have been renamed after the
layer stacking and reprojecting steps.
i. Dates For each image the year, Julian day, and calendar date should be filled in. In addition it is wise to set up
an equation or graph to show the median, mean, and perhaps range of dates to better understand the
dates of the images in regards to the whole stack. For the archive imagery the Julian day is the 3
numbers following the year in the unzipped file name (e.g. LT50460291986222xxx07.tar in this case the
three blue numbers 222 for the 1986 image from path 46 and row 29 from the landsat 5 sensor). This will
later help determine which image is to be used as the reference image. An image from a date near the
Julian day mean or median is preferred. Once unzipped the calendar month and day are the last four
numbers in the file name prior to the band extension and following the year.
ii. Raw data on the drive name of image after layer stacking and reprojecting.
iii. Reference image Two images could be selected here a COST reference and possibly a separate Cloud reference image.
iv. COST on reference Has the COST model been run on the COST reference image? Yes or no.
v. Madcal'd Has the image been madcal'd? Here you may want to put the final overall correlation values for the image
and if it is an image needing more work a comment on its status.
vi. Cloudscore calc'd Has the image had cloudscorescreated for it? Yes or no.
vii. Cloud thresholds determined Has the image had a cloud threshold determined yet? Yes or no.
viii. Comments Any comments regarding the image in general.
Figure 3. 'Image status' worksheet within the LandTrendr processing log.
III. STRATEGIES FOR SELECTING
IMAGERY FOR LANDTRENDR
A.
Dates (seasonal and phenological considerations)
1. July-August imagery is preferred; however, late June and early September may be acceptable.
a.
Before July, there can be problems due to snow or early summer phenological differences.
b.
After August, a reduced sun angle and late summer phenological differences can be problematic.
i. Image date is among the most important criteria to consider in image selection. Remember, clouds
are easy to screen, phenological problems are much harder to correct.
B.
1.
Clouds
Images should be as cloud-free as possible, but some clouds are fine.
2. Try to find an image free of clouds over land, early year-wise in the stack, as this could be used for a
cloud reference image. Date of image is not as important for this image.
3. If images within a year are cloudy try to get more than one image within that year preferrably images
that do not have their clouds in the same places. LandTrendr can handle more than one image per year it
simply chooses the clear pixel with the best Julian date for the stack.
4. If images within a stack have clouds, check to make sure that the clouds are not over the same
location in many of theimages as the cloudy areas will be masked out.
5.
Avoid cirrus clouds and haze. Cumulus clouds are much easier to mask out.
C.
Scene drift
1. Avoid images that drift far from the borders of the other images in your stack. These images will limit
your study area extent later.
D.
Missing years
1. Having every year for LandTrendr is best, but it is not critical. If good quality free images are not
available, it is okay to have images for only every 2 years.
2.
Anchor images
a. Images flanking a missing image year should be high quality and have a tight date range. For
example, if there is no image available for 1995, images for 1994 and 1996 should be relatively cloud free
and have dates of no more than about one month apart.
i.
Good anchor images help LandTrendr avoid detecting false trends due to missing images.
E.
1.
Trends in imagery dates
It is best if the image dates within a stack are either consistent, or consistently inconsistent.
a. Eg. In a stack with mostly images from late August, it is best to avoid using a single image from early
July. Outliers in date range could potentially produce false positives during LandTrendr.
b. When images in a stack generally span the date range
from early July to late August, any date within July or August is okay.
IV. ACQUIRING IMAGERY
There are several steps needed to obtain Landsat imagery from the online archive. First, identify the
location of the study area within the Worldwide Reference System. Second, navigate to the online
Landsat archive. Next, select and order the images. Finally, download the ordered Landsat imagery to
your local LandTrendr workspace. Note, with the rate that technology is changing, the following specific
instructions about websites, downloading procedures, and browsers may become outdated and
inaccurate. Therefore, analysts should use this as a guide and verify the websites and programs
mentioned are accurate.
A. Identifying imagery
1. The Worldwide Reference System (WRS) is a global notation system for Landsat data. It enables a
user to inquire about satellite imagery over any portion of the world by specifying a nominal scene center
designated by PATH and ROW numbers. The WRS has proven valuable for the cataloging, referencing,
and day-to-day use of imagery transmitted from the Landsat sensors
(http://landsat.gsfc.nasa.gov/about/wrs.html). Landsat MSS imagery is referenced the WRS-1 system
while Landsat TM and ETM+ imagery is referenced by the WRS-2 system because of the different image
footprints.
Figure 4. Landsat imagery is located by the Worldwide Reference System 2 with a path and row address
(path/row). Yosemite is located within 42/34. Kings Canyon is located within the overlap of two scenes
42/34 and 42/35. Sequoia is located within the overlap of three scenes 42/34, 42/35, 41/35.
2. Go to the USGS Global Visualization Viewer (GLOVIS) internet site where the imagery of the Landsat
Archive is available for browsing (http://glovis.usgs.gov/). Figure 5 is a screenshot of this webpage.
From the 'Select Collection' drop down choose: 'Landsat Archive'.
From the next drop down choose 'Landsat 4-7 Combined'.
a. Turn off the pop-up blocker function for the internet browser you are using. Pop-up blockers will
interfere with both viewing and downloading images.
Figure 5. Front web page for the Glovis imagery website where the Landsat image archive is browseable.
3. Click on the map somewhere near your study area. Once the USGS Global Visualization Viewer
opens you can choose your specific WRS-2 path and row if known (Figure 6) or navigate using the arrow
buttons. Alternatively, if you have a shapefile of your study area, use GLOVIS/Map Layers/Read
Shapefile.
Figure 6. Choosing a WRS-2 scene.
4. Navigate to the year and date you would like to download
(Figure 7). While viewing the available images take into consideration the suggestions and criteria found
above in section III. Strategies for selecting imagery for LandTrendr.
Figure 7. Choosing an image date.
B. Record your image selections
Record your image selections in the appropriate fields in the 'Image Status' worksheet within the
'LandTrendr_Processing_pprr_template.xls as described in SOP 8 and shown previously in Figure 2.
.
C. Downloading and Ordering imagery
Once a suitable image is found in the Glovis archive directly download the image or order it for
processing. Both the downloading and ordering of images are free of charge.
Before you begin...
>Create the appropriate directory structure, see Figure 2
>Register and/or login with the EarthExplorer website (Figure 8)
Figure 8. EarthExplorer initial registration page.
1. Add image to Scene List
With the desired image active in the USGS Global Visualization Viewer click the 'Add' (Figure 9) radio
button in the bottom left of the viewer. At this point only the 'Submit' button may become active or the
'Submit' and 'Download' radio buttons will both become active. If 'Download' is active you can mouse-click
to immediately download the image . Otherwise select 'Submit' to order the image.
Figure 9. Adding an image for download to the scene list. If the image is shown with a 'downloadable '
note in the upper left corner of the image viewer, the selected image is available for immediate
downloading.
2.
Downloading an image (Figure 10).
a. Confirm Download
After clicking the Download radio button simply click “Yes” in the Confirm download for no charge dialog
box. You will be prompted to log into the EarthExplorer website (figure 6), if you have not previously.
b. Submit request
Once you have submitted your request the download dialog box will come up. Soon after a smaller dialog
box with series of red blinking dots (status bar) will also come up. note: if this smaller download retrieval
dialog box does not come up and the larger download dialog box seems idle (Internet Explorer flag not
waving) it is not retrieving the request. Popup blockers are likely interfering; this caninclude not only the
Internet Explorer or Firefox pop up blockers, but also others such as those on the Google toolbar. Allow
pop ups and this should work.
c. Select location to save image
Following this, a download dialog box should come up asking where you would like to save the image.
Create a destination folder based on the year of the image (e.g. 4527\Images\1984 ) navigate to that
directory and the actual downloading of the image will begin. Note: If using Mozilla Firefox it may be
necessary to go to Tools/Options/Downloads and select Always ask me where to save files, so that you
can navigate to the folder you want to save it in.
d. Download Time
Downloading time will vary depending upon your internet
speed. The files are compressed, usually hundreds of megabytes, and will need to be uncompressed
after download.
Figure 10. Downloading immediately available Landsat Archive imagery is accomplished with a series of
mouse-clicks.
e.
i.
Ordering an image
If an image is not available directly for download then click the 'Submit' radio button (Figure 11).
Figure 11. Submit imagery for processing if it is not immediately available for download.
(1). If the image has clouds it may ask you if you want to include low quality scenes. These images are
often fine, so check the image box and click continue.
(2). If the image is a Landsat 7 SLC-off image you will be asked if you wish to continue with your
request even though there are data gaps . Select ‘Yes’.
ii.
Then in your EarthExplorer browser window will be step1 of 5 of a shopping basket (figure 12).
Figure 12. Shopping basket for ordering Landsat Archive imagery.
(1) Make sure the image is correct then select 'Checkout'
(2) Then click 'Continue' & Enter your address
(3) Make sure all of your address information is correctly filled out especially your e-mail. Click 'Submit
Address Information.'
(4) You should then see that your order has been received. It is a good idea to enter your Imagery order
number and other information into your preprocessing workbook under a worksheet named 'Landsat
Archive orders'.
iii. Confirmation Email.
You should receive an e-mail in approximately 2 to 4 days. You may be alerted that your image could not
be processed. If your image was processed you will be given a link in the e-mail. Click the link and from
here you can follow the instructions above under section IV.C.2. 'downloading an image'.
Protocol for LandTrendr
Standard Operating Procedure (SOP) #2
Preprocessing image stacks
Version 0.01 (March 9, 2009)
Revision history log:
Previous Version Revision Date Author Changes Made
New
April 12, 2009 REK
New
July 30, 2009
populated
PN
Reason for Change New Version #
0.01
0.01
March, 2009
REK
Clarifications
0.01
Sep, 2010
PN
revisions & clarifications
0.01
Table of Contents
1.
a.
i. Overview
ii. Processing Datasheet
iii. I. Conventions for Describing Software Methods
1. A. IDL
2. B. ENVI
3. C. Erdas Imagine
iv. II. Setting up file structure (SOP 8)
1. A. Naming of files
2. B. Determine map projection
v. III. Converting Landsat Archive files
1. A. Landsat Archive files
2. B. Batch conversion and reprojection using 'envi_glovis2landtrendr.pro'
vi. V. Establish Reference Images
vii. VI. Reprojecting Images
viii. VII. Compensating for Sensor and Atmospheric Influences in Reference Image
(Atmospheric Correction - COST Method)
1. A. Selecting dark-object values for each band:
ix. VIII. Setting up image information structure
x. IX. Defining Study Areas
1. A. Overlap area count method
2. B. Vector-based method
3. C. Thiessen scene area method
xi. X. Compensating for Sensor and Atmospheric Influences in Input Images
(Madcal)
1. A. Identify the radiometric-reference image.
2. B. Identify subsets
3. C. Run MADCAL procedure
4. D. Analyzing MADCAL results
5. E. Verify Image_Info variable
6. F. Create header files
xii. XI. Generating LandTrendr Tasseled-Cap images
1. A. Load Image_Info variable into Memory
2. B. Alter make_landtrendr_tcimages_pprr.pro
3. C. Run batchfile
4. D. Verify Results
xiii. XII. Cloudscreening
1. A. Producing cloudscore images
2. B. Determine thresholds for clouds and shadows
xiv. XIII. Creating cloud masks
1. A. Alter batchfile
2. B.
Run batchfile
3. C. Create header files
4. D. Verify Results
xv. XV. Works Cited
OVERVIEW
An overview of the Trajectory Based Change Detection (TBCD) processing work flow involves
many steps. First, Landsat images for each year between 1984 to present, primarily during July or
August time period, is obtained from the Landsat data archive (SOP 1). Second, COST atmospheric
correction, MADCal radiometric normalization, and cloud masking is performed on the images in the
yearly stack (SOP 2). Third, the image stack is analyzed using the LandTrendr process (SOP 3) which
summarizes and labels the Landsat observed long-term trends from year to year. Next, the LandTrendr
output results are evaluated for their accuracy using independent data visualization using TimeSync
(SOP 6). Finally, the analysis and final products are documented and archived (SOP 8).
Figure 1. SOP 2 within the LandTrendr process.
This SOP explains the procedures for processing data acquired in SOP 1 into a geometrically robust,
radiometrically standardized-reflectance product that can then be used for change detection and
evaluating disturbance and recovery trends. Preprocessing is a critical step in any change-detection
study, but especially for this method.
PROCESSING DATASHEET
Many detailed and sometimes complicated steps are in this SOP. Because the analysts may need to
split work between this work and other responsibilities, we provide table 1 that can be printed out
separately for the analyst to track the steps that have been completed. This should facilitate easier
recovery from interruptions. Figure 2 shows a screenshot of the LandTrendr process log composed of
multiple tables for documenting
specifics results of each LandTrendr step, as described in SOP 8: Data Management.
Figure 2. LandTrendr process log template. An excel workbook comprised of multiple worksheets to
track the results of the LandTrendr process including the preprocessing steps of image dates, COST
correction, MADCAL, and cloud screening.
I. CONVENTIONS FOR DESCRIBING
SOFTWARE METHODS
All core methodologies for implementing this protocol are written in a proprietary software language
known as IDL (Interactive Data Language; ITTVIS). This software underlies ENVI (Environment for
Visualizing Imagery; ITTVIS), which is one of two major image processing software packages in the
research and commercial field (the other being ERDAS Imagine). The components of this protocol that
require standard image processing methods are therefore preferentially described in terms of
implementation in ENVI, allowing most users the ability to entirely carry out the work herein using a single
package (ENVI+IDL).
A. IDL
The main console of the IDL Development Environment (IDLDE) is presented in Figure 2. Commands to
be entered at the command line prompt will be documented as:
IDL> @landtrendr_4234_run1.pro
where IDL> is the command line prompt and "@landtrendr_4234_run1.pro" is the command for IDL to
initiate.
Figure 3. IDL Development Environment (IDLDE) main screen. The IDLDE is a graphical interface
providing editing and programming for IDL programs such as LandTrendr.
IDL Basics
Understanding some basic IDL operations is necessary to operate LandTrendr. The relevant concepts
are presented below. For a more thorough understanding of IDL in general, the program has a robust
documentation system, IDL Online Help accessible through the menu or by typing (IDL> ?).
a. Search Path
The search path is a list of directories that IDL searches to find a program, procedure, batch file, or
save file. Creating a link to the working directories is necessary for IDL to operate on the downloaded
Landsat imagery. In the IDLDE following File->Preferences->Path allows a user to insert a new path to a
specific directory such as the location of your LandTrendr working directory, as demonstrated in figure 4.
Insure that your path is 'checked' for IDL to search.
Figure 4. Setting IDLDE Search Path Preferences.
b. Variable
A variable is a symbolic name associated with a value and whose associated value may be changed
(wikipedia, 2009). In IDL, variable names must start with a letter, are case-insensitive, and may be up to
128 characters in length. An IDL variable are described with two attributes: type and organization. The
type of variable determines the type of data allowed to be represented and the range of values possible
(Table 1). IDL variables can be organized into scalars, arrays, or structures. Scalars are single values
and are recognized by the format variable name = variable value. Arrays are multiple values of the same
data type arranged in x columns and y rows and recognized by the values in square brackets ([]).
Structures are a combination of data types and organizations allowing different data types to be stored in
a single variable. Below is an example of creating variables:
Scalar Text Variable
IDL> scalar1 = 'string value1'
IDL> scalar2 = 'string value2'
Array Text Variable
IDL> array = [scalar1, scalar2]
IDL> print, array
string value1 string value2
Table 1. Variable data types.
Type
Size (bytes)
Range
byte
1
0-255
integer
2
-32768 - 32768
unsigned integer
2
0 - 32768
long integer
4
±2^31-1
unsigned long integer
4
0 - 4294967295
64-bit integer
8
±2^63-1
unsigned 64-bit integer
8
0 - 2^64-1
float
4
±1E±38
double
8
±1E±308
complex
8
±1E±38
double complex
16
±1E±308
string
pointer
4
object
4
c. Statements
A statement is an instruction that tells a computer what to do. A program is a sequence of statements
that act as a unit. The LandTrendr program is comprised of many individual statements divided into
procedures and functions.
d. Procedures
A procedure is a self-contained IDL program; i.e. any variable defined in a procedure is deleted when
the program ends, unless it is passed out through a parameter. A procedure starts with the procedure
declaration statement (keyword PRO), the name of the procedure, and any parameters for the
procedure. The body of the procedure follows. A procedure is terminated with an END statement
(ITTVIS, 2007).
e. Functions
A function is another self-contained IDL program, like a procedure; however, a function always returns
information to its caller. A function begins with the function declaration statement (keyword FUNCTION),
the name of the function, and any parameters for the function. The body of the function, which usually
includes at least one RETURN statement, follows next. A function is terminated with an END statement.
The syntax for calling a function includes positional and keyword parameters (ITTVIS, 2007).
example:
overlap_file = build_image_overlap_area_image(image_info_savefile)
f. Parameter
Parameters are used to pass information between IDL programs. There are two types: positional and
keyword. Positional parameters are typically used for required information while a keyword parameter is
used for optional information. The order of positional parameters in a call is important while keyword
parameters (denoted with a "/") may be listed in any order (ITTVIS, 2007).
IDL> function_above, 5, /noedit
g. Batch File
A file containing IDL statements, with no declaration or terminating END statement. Batch files are
simple text files that can be created in the IDLDE interface, or in any standard text editor. Each line in the
file is evaluated by IDL as equivalent to a single entry on the command line.
IDL > @lt042_4529_nbr
The command above instructs IDL to find and run all of the lines in the file named "lt042_4529_nbr".
The file must reside in the IDL search path.
2. User interaction with IDL in LandTrendr processing flow
a. The LandTrendr workflow relies heavily on the use of batchfiles. For each major step in the
process, the user will run a batchfile based on templates provided along with this protocol. The user must
first adjust key file names, paths, and parameters to suit their own needs. The specifics of the template
files and the changes the users must make are provided in these SOPs.
3. General structure of landtrendr -- the idea of the image info structure
Working with large amounts of data requires a systematic structure of storing and accessing the proper
file at the proper time. LandTrendr uses an IDL structure variable to store the necessary information that
the program uses at various stages of processing. Table 3 shows the attributes and values which create
the 'image_info_file' variable. Certain procedures will alter and populate the variable values so it is
important to verify the quality of this variable at specific points throughout the process.
Table 3. Variables and their definitions of the 'image_info' structure variable.
Variable
definition
IMAGE_FILE
Path and Name of the 6-band Landsat
Archive image
IMAGE_PATH
Path to image file
TYPE
Tracks reference images
example values
3
(1) MTBS
(2) NLAPS
(3) COST reference image
(4) other
NBR_FILE
Path and name of the Normalized
Burn Ratio raster file
TC_FILE
Path and Name of the Tasseled Cap
raster file
B6_FILE
Path and Name of the thermal band
raster file
YEAR
Year (1984-present)
1985
JULDAY
Julian Date (1-365)
226
UNIQUE_YEAR
Is this a unique year
(0= no, 1 = yes)
N_IN_YEAR
Number of images in the year
1
IMAGE_PRIORITY
Based on the median julian date of
1
the scene stack, how does this image
rank compared to any other image in
the year. This is used for determining
which image to use for a masked
pixel.
CLOUDYEAR
Cloud reference file (0= no, 1 = yes)
CLOUD_DIFF_FILE
Path and Name of the cloud
difference file created in step XII.
SHADOW_DIFF_FILE
Path and Name of the shadow
difference file created in step XII.
1
TC_CLOUD_DIFF_FILE Path and Name of the Tasseled Cap
cloud difference file created in step
XII.
CLOUD_FILE
Path and Name of the cloud mask file
created in step XII.
SUBSET
The coordinates of the upper left and
lower right coordinates of the image.
USEAREAFILE
mask raster for intersection of
images. determines spatial extent of
madcal, cloud scores.
J:\4334\gis_data\usearea.img
PATH
Path for the Landsat scene being
processed.
J:\4334\
B. ENVI
The Environment for Visualization (ENVI) is a menu-driven software built upon IDL with a "Main
Console" shown in figure 5. The buttons arranged horizontally refer to the main modules of the program.
Actions are taken by selecting a button, which then either starts an action or brings up pull down menu
with several options, some of which have pull down options of their own. Actions will be referred to by
the box in the main console, followed by a forward slash ("/") and the name of the option from the
pulldown menu, such as File/Open External File/IP Software/Erdas Imagine. A second slash and
pulldown option will be listed as necessary. The IDLDE is also opened and available for use during an
ENVI session.
Figure 8. ENVI main menu console.
i. Viewing images
File/Open External File/IP Software/Erdas Imagine/filename.img
1. File/Open Image File/filename (ENVI format)
2. Image Display Windows
Viewing images in ENVI involves choosing the desired bands for display in the red, green, blue display
channels from within the 'Available Bands List' dialogue as shown in figure 9.
Figure 9. ENVI image display windows is comprised of a scroll, image, and zoom windows. A pixel
locator can be toggled on and off in the zoom windows.
C. Erdas Imagine
Imagine is a menu-driven software package that begins with a "Main Console," shown in figure 11. The
buttons arranged horizontally refer to the main modules of the program. Actions are taken by selecting a
button, which then either starts an action or brings up pull down menu with several options, some of
which have pull down options of their own. Actions will be referred to by the box in the main console,
followed by a forward slash ("/") and the name of the option from the pulldown menu. A second slash and
pulldown option will be listed as necessary. For example, image subsetting is in a pulldown menu from a
pulldown menu from the Interpreter button on the main console, referred to as
"Interpreter/Utilities/Subset."
Figure 11. Main console of Imagine
Once an action is selected, an action window typically appears that provides a suite of options for a
given action. The "Subset" action window, for example, is shown in figure 12.
Figure 12. Subset window.
Instructions for filling in boxes and checking options on these action windows are referenced by the
name on the window followed by a colon. For example, to switch the default Output type in the subset
window above to Signed 16 bit, the instruction in this SOP would read "Output: select Signed 16-bit."
II. SETTING UP FILE STRUCTURE (SOP 8)
Creating and following the directory structure and naming conventions described in SOP 8 is critical for
the algorithms to find files and write outputs.
A. Naming of files
File names are concatenated sequences of codes that uniquely describe a file. To be used in the
LandTrendr processing stream, filenames must adhere to a strict naming convention. This naming
convention allows the processing algorithms to identify which type of image is being used, and the
julian date of the image, which is used in several key steps.
We advocate descriptive code names that allow a user to immediately identify each files purpose.
Table 3 provides a list of all of the component code names for one example of images for one
scene (path 42 row 34). An example file name strings together these codes in this order:
(IMAGETYPE)(PATH)(ROW)_(YEAR)_(JULIAN DATE)_(IMAGESOURCE).(FILE
FORMAT)
LT5046031_1985_219_archv.img.
Table 3. Preprocessing Filenames. Key distinctions highlighted in Bold.
Preprocessing
Distinction
Example
Description
step
LT4
LT41320261989169AAA02.tar.gz Landsat 4
LT5
LT5046031_1985_219_archv.img Landsat 5
pp
rr
World
Reference
System-2
Path
LT5046031_1985_219_archv.img
number (1?). NorthSouth
Direction.
World
Reference
System-2
Row
number. (1LT5046031_1985_219_archv.img ?). EastWest
Direction.
LandTrendr_processing***.xls
1. Stack
Image
_.tar.zip
Landsat
Archive tar
zipped
compressed
file
containing
all Landsat
bands.
Landsat
Archive
Band #.
Each #
corresponds
to the
Landsat
band
number.
_b#.tif
2. Image Info Image_Info_pprr
Image_Info_4234.sav
B. Determine map projection
1. Header Template
A template header file, defining the projection of the associated file, is used in several steps of the
LandTrendr process. This file is created through ENVI tools and stored in the /images/ directory as
described in SOP 8.
a.
b. Example header template file
ENVI
description = {
File Imported into ENVI.}
samples = 8064
lines = 7872
bands = 1
header offset = 0
file type = ENVI Standard
data type = 2
interleave = bsq
sensor type = Unknown
byte order = 0
map info = {MRLC_albers_con_eq_area, 1.5000, 1.5000, -2169570.0000, 3131490.0000,
3.0000000000e+001, 3.0000000000e+001, North America 1983, units=Meters}
projection info = {9, 6378137.0, 6356752.3, 23.000000, -96.000000, 0.0, 0.0, 29.500000, 45.500000,
North America 1983, MRLC_albers_con_eq_area, units=Meters}
wavelength units = Unknown
III. CONVERTING LANDSAT ARCHIVE
FILES
Creating the required 6-band Landsat image is a necessary first step in the LandTrendr process.
Imagery downloaded from the Landsat Archive needs to be stacked into one file. There are at least two
approaches within ENVI: Manual stacking of individual files (part A below) or batch processing using
script-based approach (part B below). The former is more time-consuming but requires less start up
learning (mostly about how to arrange files and folders).
A.
Manual extraction of Landsat Archive files in ENVI
Landsat Archive images acquired from GLOVIS come in a standard format: the UTM projection, GeoTiff
format, where each band of the image is represented by a separate file. These must be stacked,
reprojected, and the file placed in a filename format that LandTrendr can recognize.
Before you start:
-Confirm that you have downloaded the .zip file with the image into an appropriate folder
-Confirm that the unzipping process worked. There should be a series of files with names such as
"L71047027_02720080805_B10.TIF." See Figure 1 below. Note that Landsat 5 images will have only
one B6 file (not two, as shown here) and no Band 8 (the Panchromatic band only on ETM+). Some
images also come with a similar naming convention to above, but with a *_b1.tif file suffix.
-Confirm that the geographic projection to which you will be projecting exists in ENVI's list.
Figure 13. An example of GeoTiff files that come from the Landsat archive.
a. Open ENVI if it is not opened already
a. Stack and reproject images in ENVI
b. The basic approach is to stack the visible (1-3) and non-thermal infrared (4, 5, 7) bands
and reproject them at the same time.
c. Identify image
i. Select "Basic Tools\Layer Stacking." You will see this dialog box (Figure 14):
Figure 14. The Layer Stacking dialog in ENVI.
i. Click "Import File...", and on the subsequent Layer Stacking Input File dialog,
click the "Open/New File" button, and identify Band 1 of the GeoTiffs. It will show
up in the Select Input File box of the Layer Stacking Input file dialog. Click "OK".
The file will appear in the upper left box of the Layer Stacking Parameters dialog
box in Figure 14.
Figure 15.
ii. Repeat step b for the five files with the following filename components, in this
order: ....B20.tif, ....B30.tif, ....B40.tif, ...B50.tif, ...B70.tif
1. Tip: Be sure to put these in the correct order, as this will determine the
order of the layers in the resultant file.
2. Tip: If you are importing more than one date of imagery in a folder, it will
be easy to inadvertently grab the wrong date of imagery. Be careful!
Additionally, pay particular attention that you have correct source image
iii. Confirm that all six files are in the box in the correct ascending order.
Figure 16.
a. Change the projection of the output image
i. On the right of the Layer Stacking Parameters dialog box, click on the projection
you are using (e.g. the continental albers conical equal area projection for
projects based on MRLC data should be set up ahead of time and in this list).
Figure 17.
a.
Pixel Size
i. X and Y pixel size should remain at 30.0000 m
1. Resampling: Nearest Neighbor
b. Enter the output filename
i. For images that come from the Landsat Archive, use this naming convention,
placed in the same folder as the geotiffs:
1. lt5027026_1999_205_archv
i. Where "lt5" corresponds to Landsat Thematic Mapper 5, "027026" to the path
row of the image (retain leading zeros), "1999" is the four digit year, "205" is the
Julian day of the year, and "archv" is the unique identifier for images obtained
from the archive. It is critical to retain exactly this structure, since the programs
count characters in the file names to identify the julian day of each image.
1. Julian days: If you only have information on the actual date/day of an
image (e.g. July 24th, 1999 instead of 205), calculate the julian day in a
standard spreadsheet with a column in date format for the date/days,
and a second column with sequential numbers. For simplicity, do not
include leap days -- the difference of 1 day for the calculated julian day
will not affect the algorithms.
Figure 18
f.
Click "OK" to begin calculation.
i. The File Map Projection Conversion dialog will open as ENVI processes the
scene.
a. Upon completion of the process, the new 6-band image will show in the 'Available Bands
List' dialog box. Confirm that the image has appropriately converted (check band names,
projection) by viewing it in ENVI.
Reminder: naming conventions
COST reference image> juldayoffset = 11
Archive images> juldayoffset = 15
;format: 5047027008621310_refl_cost.img
;format: lt5027026_1999_205_archv.bsq
B. Batch conversion and reprojection using
'envi_glovis2landtrendr.pro'
This approach allows for quick importing and reprojecting of archive files, once you have the file
structure set up appropriately. The key issue here is to have all of the information about the file (path,
row, year, day of year, etc.) in the correct character-location in the name of the folder that holds the
individual *.tif files. The batch conversion process begins with unzipping the Landsat Archive imagery to a
directory within '\\pathrow\workspace\' folder.
1. Getting files into the correct location
a. Assumes: You have a folder named "workspace", e.g. 4532\workspace.
b. During the imagery download process, save your *.tar.gz files to the
workspace folder (Figure 19).
Figure 19.
a. Using WinZip or similar program extract the file to the same workspace folder.
i. NOTE: You may be prompted to extract to a temporary folder. CHOOSE "NO" -you do not want a temporary folder (see next step).
Figure 20.
i. The result should be a .tar file in the workspace directory.
a. Extract the .tar file, and ensure that the folder name has this format:
i. LX50460322008235
1. Where: X could be "T" or "E", and is not really critical except for being a
character spaceholder.
2. 5 is the designation of the sensor -- could also be 7 for Landsat 7
3. 046 is the path
4. 032 is the row
5. 2008 is the year
6. 235 is the julian day of year
7. Anything after that is ignored, so you can add characters as needed.
Figure 21.
a. Run the conversion program
Start IDL and type (Figure 24):
IDL> envi_glovis2landtrendr
*note the default is to reproject the Archive imagery from UTM to MRLC Albers Conical Equal Area. If the
imagery is required in a different projection than MRLC Albers, passing a 'proj' variable to the function
allows this reprojection to be user controlled.
IDL>Envi_glovis2landtrendr, proj=#
# Projection Type
0 : albers
(MRLC Albers Conical Equal Area)
1 : map_info.proj (native UTM projection from Archive)
2 : orlam (Oregon Lambert )
3 : Southwest Alaska Network NPS (SWAN - Albers Conical Equal Area)
Figure 24.
1. A dialog box (figure 25) will open asking you to browse for an input
folder.
a. Choose the "workspace" folder -- i.e. not the one you just
created, but the one "above" it.
b. The program will look within the "workspace" folder for
subfolders that have the file naming conventions described
above for path, row, year, etc.
Figure 25.
1. Another dialog box (figure 26) will open asking you to browse for an
output folder.
a. Again choose workspace. Conversion should begin. It may take
several hours.
Figure 26.
i. Batch stacking results (figure 27):
1. You should have in the workspace folder
a. A subfolder named after the scene path/row
e.g. \\workspace\4532
a. Within that folder, subfolders with the years of images converted.
e.g. \\workspace\4532\1985, <years>, etc.
a. Within each year folder, you should find:
i. Converted *.bsq files for the bands 1-5 and 7 stack and
b6 in the correct naming convention and projection.
i. Copy the new stacked imagery from workspace directory to \images\
directory(figure 28)
i.e. G:\4532\workspace\4532\<years> would move to G:\4532\images\<years>
V. ESTABLISH REFERENCE IMAGES
Two categories of preprocessing are to be carried out in this SOP: radiometric and cloud
identification. In each category, the user must define an image acquired on a specific date to be a
reference towards which all other images will be matched. Thus a reference radiometric image and a
reference cloud-free image are needed.
The radiometric-reference image is one whose radiometric properties will anchor the radiometric
properties of later images. The radiometric properties simply refer to how the spectral reflectance of
objects on the ground surface are represented as numbers in the image, and takes into both the sensor
engineering design and the state of the atmosphere that interferes with the reflected light from the ground
as it travels to the satellite. The radiometric-reference image is one whose radiometric properties are well
defined.
In practice, choice of a good radiometric-reference image comes down to a clear atmosphere during
image acquisition and a date of acquisition similar to that of most of the other images for which the image
is to be used as a reference. This latter condition is useful because a key step in preprocessing is
normalizing later images to the spectral properties of the radiometric-reference image. Differences
caused by phenological state of the vegetation could confuse the normalization if the reference image
were obtained during a point in the season completely unusual relative to the other images desired for
use.
Therefore a radiometric reference image should be chosen from the same late-summer date windows
that are recommended for all images (SOP 1). The clarity of the atmosphere for a given image typically
cannot be known, but can be inferred. First, minimal cloud cover is preferred, since cloud cover indicates
presence of increased levels of moisture in the atmosphere. A dry atmosphere is preferred. Cloud-free
images can still have significant levels of atmospheric haze or forest fire smoke, which also is to be
avoided. Haze levels can be inferred from the spectral response of pixels across the six visible bands of
the Landsat imagery. Haze affects the bands at shorter wavelengths more than longer wavelengths
because of the relation between the size of the particle causing the haze and the wavelength of light
passing through it. Therefore, images with the least inferred scattering in Band 1 (the blue band of
Landsat, with the shortest wavelength) are the images least affected by atmospheric haze. Thus, if there
is a choice of several images for potential radiometric reference image, then it is recommended that the
images be processed through to the COST processing, detailed below, and then the image with the
lowest overall dark-object value in Band 1 be used as the radiometric-reference image.
A second type of reference image needed is one that is free of clouds. This image may be the same
as the radiometric reference. Two requirements for the cloud-free reference include 1) free of clouds 2)
early in the scene stack (1984-1988 preferably). Since the cloud-reference image will be compared to
other images in the stack, the second requirement minimizes false-positives to the cloud screening
procedure. If there is not a cloud-free image early in the stack, additional screening steps will be
necessary.
Screening the images used in the LandTrendr process for clouds is important because these data
anomolies create false positives of change and recovery. A cloud-free pseudo image is created on the fly
by LandTrendr by using multiple images for a given year to fill in missing data for clouds present in a
primary scene (closest to mean julian date of stack).
<<Figure 27. Example of appropriate reference images>>
VI. REPROJECTING IMAGES
It is critical that all images and vector coverages be in the same projection, spheroid, and
datum before any radiometric or change-detection processing is begun! Although Imagine and
ArcGIS will tolerate different projections and spheroids when processing, it is best not to allow Imagine or
ArcGIS to reproject "on-the-fly" without knowing explicitly that images are aligned as is necessary. Only
by viewing spatial data and ensuring that spatial relations are as expected can robust results be
achieved.
Although many existing Digital Elevation Models and some vector coverages are in the Clarke
1866/NAD27 Spheroid/Datum combination, those are based on older model of the Earth, a poorer datummeasurement network, and thus should be avoided as baselines for future monitoring. The conversion
between NAD27 and NAD83, although not perfect, can be achieved with accuracy more than sufficient for
Landsat-based investigations.
Reprojection to the NPS standard projection can be incorporated into the envi_glovis2landtrandr.pro
procedure discussed previously in SOP 2:IIIB. .
VII. COMPENSATING FOR SENSOR AND
ATMOSPHERIC INFLUENCES IN
REFERENCE IMAGE (ATMOSPHERIC
CORRECTION - COST METHOD)
The atmosphere affects the reflectance signal that impinges on the sensor from the surface of the
Earth. Differences in atmospheric conditions between dates of imagery will cause differential artifacts in
the images that will confuse later change detection. Therefore, all reasonable efforts must be made to
compensate for these effects. The first step is to bring both images into a common system of
measurement, reflectance.
The units of a Landsat image are simply digital numbers (DNs), corresponding to the magnitude of energy
being measured by each of the sensor elements in the satellite. Because the engineering properties of
the sensor are known, the DNs can be converted into physical units of radiance. Knowing the emission
spectrum of the sun entering the atmosphere, these units of radiance can further be quantified as a
proportion of the incoming radiation ranging from 0 to 1.0. This is known as "top-of-atmosphere
reflectance." It does not take into account the effects of the atmosphere.
Without direct measurements of atmospheric absorption at the moment of image acquisition, it is
impossible to know this atmospheric effect directly, but it can be approximated. To approximate the
effects of the atmosphere, it must be assumed that there are some objects on the surface with little to no
reflectance. These so-called "dark objects" should indicate a reflectance of zero or near-zero. Because
the atmosphere introduces scattering between the objects and the sensor, the apparent reflectance of
these objects from the sensor's perspective is non-zero and positive. The additive effect of this scattering
can be removed by simply subtracting the offset above these dark objects from all pixels in the image.
The multiplicative effects of the atmosphere can be approximated simply by a correction factor that scales
with the path length through the atmosphere, which requires only that the sun angle and elevation be
known. All of these steps can be incorporated in a single-transformation process and placed in a graphic
model in Imagine. Because these were best described in (Chavez, 1996) as COS-theta or COST
methods, these steps are referred to here as COST processing.
A. Selecting dark-object values for each band:
The first part of the COST processing is selection of dark-object values for each of the six visible
bands in Landsat. An Excel worksheet is included in the
"Processing_LandTrendr_pprr_template.xls" and serves as a template for which the relevant
values for COST processing can be stored.
Figure 28. COST processing worksheet.
a. This worksheet is named 'Cost Reference'.
b. Save this Excel file under a new name that corresponds to the name of the image, so that it can
be connected easily with that image in the future.
c. Make a note of this filename in the metadata associated with this image.
a. Follow the conventions in SOP 8 Data Management for naming convention.
d. It is helpful to use the entire image area, not just the area within the study area of the park, for
locating potential dark objects because the population of potential targets is much higher.
Therefore, for a given image, go back to the first version of the image that includes the entire
footprint of the original Landsat image directly after importing into Imagine format.
a. i. This image should be the one that was imported in section III above. It need not be
geometrically referenced for selection of dark-object values.
1. Open the Landsat reference image in an ArcGIS session (figure 29):
i. Use the open-folder icon to bring up the " Add Data" window.
ii. Navigate to the location of the 6-band Landsat image.
iii. Select the 6-band Landsat image and 'Add' to the display (note this will display
an RGB composite of the first 3 bands in the list. Use the 'layer properties' to
change the display of the image).
Figure 29. Add 6-band Landsat image to ArcMap session.
2. Add individual bands of the Landsat image (figure 30)
a. Use the open-folder icon to bring up the " Add Data" window.
b. Navigate to the location of the 6-band Landsat image.
c. Double mouse-click on the 6-band raster image to reveal the individual bands, then select each
individual band, add to display.
Figure 30. Add individual bands of the 6-band Landsat image to an ArcMap session.
Identify dark object value for each band (figures 31-34)
Identifying the dark object value for each Landsat band begins with examining the digital number
(DN) histogram and DN minimum value. Symbolizing the raster helps the analyst identify the darkobject value for this band based on acceptable surface features.
a. Right-mouse-click on the '*_archv.img - Layer 1' raster file to open the context menu, select
'properties'
b. Select 'Symbology' Tab
c. Select 'Classified' for symbology type. Change number of classes='2'
d. Select 'Classify' button to view histogram.
Figure 31. Open layer properties of the band through a right-mouse-click and the context menu.
e. Note the shape of the histogram, the minimum raster value of the histogram.
i. The Histogram value (y axis) indicates the count of pixels that have a cell raster value indicated in the xaxis. For Landsat images, these range from 0 to 255.
ii. If there is a high histogram count in the row with value 0. These are the pixels outside the edge of the
image area, and should be ignored.
iii.For bands 1-3 and often 4, the lowest DN value is not zero, but rather another higher value. This
indicates that even the darkest objects in the scene have non-zero reflectance, and indicates the level of
atmospheric scattering. It is not wise to simply take the lowest non-zero DN as the dark-object value,
however, because these low values could be artifacts of processing upstream of image acquisition or
even of steps conducted in the computer software.
Figure 32. Setting the breakpoint for the dark object value.
2. Start with the lowest non-zero histogram value and examine the location of the pixels corresponding to
that value using step 2.a.i.-vii. below, and step sequentially to higher values until a pixel value is identified
whose pixels reside in the landscape, and which correspond to features that are expected to be darkobjects step 2.b.i.-ii. below.
a. To examine where pixels at a given pixel value reside on the image, do the following:
i. Look in the viewer at the image and see where the black pixels reside.
ii. Evaluate these pixels according to the criteria in step b. below.
iii. If these pixels are not considered valid dark objects. Then move to a pixel value one step higher and
repeat the evaluation process.
iv. If these pixels are considered legitimate dark objects, then take this value as the approximate darkobject value.
(1) Because few objects are truly nonreflective, it is typical to assume that the observed dark objects have
an actual reflectance on the ground of roughly 1 percent. Therefore, the true value for zero reflectance
will be somewhat lower than the observed value in the image. The correction factor can be approximate,
since these dark objects are an approximation of the actual atmospheric effects. Therefore, take the
tentative dark object value and adjust it as follows to calculate the actual dark-object value for the band:
!!SeventhLevel!!(a) For bands 1-3 and band 7, subtract 1 from the observed value.
(b) For bands 4 and 5, subtract 2 from the observed value.
!!EighthLevel!!(i) In both cases, however, the minimum allowed dark-object value is zero.
v. Enter this value in the dark-object value column for the row corresponding to the band in the Excel
spreadsheet saved in step 1.b. above.
Figure 33.
b. Evaluating dark-object pixels.
i. Characteristics of valid dark-object-type pixels.
(1) The pixels reside on the landscape, not on the margins of the scene area.
(2) The pixels are in areas where dark reflectance is expected:
(a) Water bodies.
(b) Topographic shadow.
(c) Deep shadow on shaded aspects of forests.
Figure 34. Example of valid dark-object type pixels in this Landsat Band 2 is a deep body of water.
ii. Characteristics of dark objects that are artifacts:
(1) The pixels reside on the very margin of the active area of the image.
(a) These typically are formed because of bilinear or cubic convolution of zero values outside the image
with non-zero values inside the image.
(2) The pixels reside directly next to a very bright pixel and the image has been subjected to cubic
convolution resampling beforehand.
(a) Cubic convolution can depress values of pixels neighboring bright objects, and thus can force
otherwise dark objects to even darker values that are an exaggeration of the actual dark-object value.
c. Repeat this process for all six of the visible bands in the Landsat image. Record each dark-object value
in the Excel file from step 1.b. above.
Figure 35. Example histograms of the digital numbers (DN) of the pre-COST Landsat image values.
i. When this has been done for all bands, evaluate the relation between dark-object values across bands.
(1) The effect of haze is greatest at shorter wavelengths in the blue bands of the image, and tapers off as
band number increases as shown in figure **. Therefore, the dark-object values should be highest in band
1, lower in band 2, etc. The dark-object values of each band should be no higher than the band number
lower than it. If this is not the case, then it indicates that the dark-object value for the lower-numbered
band may have been too low. Review the dark-object pixels for that band.
(2) Bands 5 and 7 may have almost no scattering, and therefore may have dark-object values of 1 or 0.
Figure 36. Example of decreasing atmospheric influence on dark objects in each Landsat band.
3. Identifying sun elevation and angle:
a. When an image is ordered from the Landsat Archive, the associated *_MTL.txt file contains information
on the sun elevation and sun azimuth.
Figure 37. The sun elevation is found in the metadata file (*_MTL.txt) associated with the Landsat
Archive image.
b. These values should be entered into the excel spreadsheet noted above. Note that the formula for the
COST processing requires the sun zenith, which is 90 degrees minus the sun elevation.
4. Identifying gains and offsets:
a. The DN values reported in a band from Landsat do not correspond to actual physical quantities. To
convert the DNs to physical values of radiance, the gains and offsets of the sensors in the satellite must
be taken into account. The gain of a sensor is akin to the amplification factor needed to convert it from
one set of units into another the slope of the line relating the DNs to radiance values, in this case. The
offset or bias is simply the equivalent of the intercept of the same line. As a sensor ages, these translation
factors will drift, and thus it is necessary to know them for each specific image being processed.
i. For Landsat, minimum and maximum are listed for each band in the header file. These should be
entered into the Excel spreadsheet, from which the gain and bias are calculated using the formula:
Figure 38. The minimum and maximum radiance values for each Landsat band is entered into the
processing spreadsheet from which the gain and bias of each band is calculated.
APPLYING THE COST MODEL
5. Entering data into an Imagine spatial modeler
a. An Imagine graphic model appears in the appendix that will apply the values collected in steps 1-3 to
an image to create a COST-processed version of the image. This model is named
applying_cost_template.gmd
i. From Imagine's main icon panel, select the icon labeled "Modeler."
ii. Select "Model Maker."
iii. In the Model Maker, use the open folder to navigate to the template model and open it (fig. **).
iv. Save this file under a name that links it to the image being processed. Do
this using File/Save As. Use the naming conventions of SOP 8 Data Management.
Figure 39. Graphic modeler in Imagine, with the template for COST processing.
b. Conventions of the spatial modeler in Imagine.
i. See figure **.
ii. All spatial data and actions applied to it are represented as shapes linked with arrows that indicate the
flow of processing.
iii. Double-click on any shape to see information about the spatial data, numerical data, or function
contained in the shape.
iv. Click on the hammer symbol to show a window of tools that can be used to update or build a graphic
model.
v. Circles correspond to functions. For a function to operate, it must have an arrow leading to it from some
data object, such as an image or a table.
vi. Stacked oblique squares represent spatial data. These are used as inputs to functions.
vii. Squares correspond to data that must be entered manually or that is not spatial, such as tables taken
from external text files.
c. Enter data accumulated in the Excel spreadsheet into the modeler.
i. In the upper-left of the model, double-click on the image-stack symbol labeled 'Image to be converted.'
The Raster information layer will pop up (see fig. **).
ii. In the filename field, use the open folder icon to navigate to the appropriate image and select it.
Figure 40. Locate and identify 6-band input image.
(a) NOTE: Even though the dark objects were chosen using the entire original Landsat image footprint,
the actual correction model should be applied to the image that has been clipped to the study area! The
use of dark objects outside the study area is justified on the simplifying assumption that atmospheric
conditions are stable across the entire image, even though this assumption is technically false. However,
dark objects are critical to this correction and also are rare, so the use of the entire image allows inclusion
of more dark objects, which outweighs the potential problems in variable atmospheric conditions. If
enough dark objects are viewed across the image, then these conditions should approximately average
out.
iii. All of the other values in the window can be left in their default conditions.
d. In the box near the top center labeled "Sun Zenith Angle" enter the sun zenith angle from the Excel
spreadsheet. Recall that this is 90 degrees minus the sun elevation, which typically is what is listed in the
header for the file.
Figure 41. Enter the sun zenith angle (90-'sun elevation') into the COST graphic model.
e. In the box on the right-hand side of the model labeled "Gains and Offsets" double-click and enter the
gains and offsets recorded in the Excel file. Note that the matrix window that appears will have six rows.
Rows 1-5 correspond to bands 1 through 5, while row 6 corresponds to band 7 of Landsat.
Figure 42. Enter the appropriate gains and bias values into the COST graphic model
f. In the box labeled "Min Vals for Subtraction" enter the dark-object values from the excel spreadsheet
that were determined in step 1 in this section above.
Figure 43. Enter the dark-object values into the minimum values for subtraction table.
h. At the bottom of the model, click on the image stack shape labeled "Output COST image." Output
image should be signed 16-bit type.
Figure 44. Enter an output name based on the naming convention with a *refl_cost.img ending.
i. Save the model, ensuring that the name can be linked easily to the image on which the processing is to
be conducted.
Figure 45. Save graphic model file.
j. Run the model by clicking on the red lightning bolt icon in the icon row at the top of the graphic model
window.
Figure 46.
6. The result of this process is an image in units of apparent reflectance on the surface of the Earth. It
likely will not correspond to true reflectance on the ground because the approximation for atmospheric
effects is not perfect, and because the illumination angle of slope facets on the landscape is not
considered. Nevertheless, the gross effects of
atmospheric scattering and atmospheric path effects
have been addressed. This image will be referred to as the COST version of an image.
Figure 47. The results of the COST process minimize the effects of the atmosphere on the Landsat
imagery.
a. Note that the units of this image are in scaled reflectance. True reflectance is considered a proportion
from 0 to 1.0. Representing these values requires the use of the floating-point data type, which requires
twice as much space on the disk as the integer type. Therefore, all of the reflectances have been scaled
to run from 0 to 1000, with 1000 equal to 1.0 reflectance. It is best to have a range of values for
reflectance (scaled) that approximately match the range of the other values in the non-corrected images.
so if the original images range from about 0 to 255, then scale the reflectance to about 0 to 500 or maybe
0 to 1000. Note also that some pixels will have reflectances less than zero, these correspond to the
pixels that were noted as artifacts in the dark-object-identification step above.
b. If the resulting COST image has a file size greater than 1 GB then a mask needs to be applied to the
edge pixels which is inflating the size of the file. This tends to happen after reprojecting an *.img file
which creates large areas of edge pixels.
IX. DEFINING STUDY AREAS
A study area is a geographic region within which image processing will occur. For LandTrendr, two types
of study area are used. For this SOP (Preprocessing), a "preprocessing study area" must be defined,
which will constrain the geographic footprint of radiometric normalization and cloud masking. Beginning
with SOP #3, a "analysis study area" must be defined, which will constrain the geographic footprint of
image segmentation (SOP #3), change detection mapping (SOP #4, #5), temporal fitting (SOP #7), and
landcover mapping (SOP #8). The two study areas can be the same, if desired, or the analysis study
area can be smaller, but at a minimum, the preprocessing study area must fully contain the analysis study
area.
The study area file itself is a raster mask in either Imagine or ENVI format. For preprocessing study
areas, it should cover most of the image footprint of a Landsat scene, as this allows maximal spatial
coverage to develop robust radiometric normalization statistics across images. The mask file can also be
used to create a vector file that allows clipping of other datasets to the study area. Both can be created in
Imagine, or, if the study area has been defined as a vector elsewhere, Imagine can be used to make a
raster file of the vector coverage.
The delineation of the analysis study area is an important step in the long-term success of this protocol,
as the analysis study area defined here will be used for reference when creating summary metrics of
change over time. Therefore, this step should involve the input of as many interested parties as possible.
There are few rules by which such a designation could be made generic, but there are several
components to consider. First, of course, the study area should include the entire area of interest.
Second, the study area may extend beyond that area of interest by either an arbitrary distance or by
some distance defined by ecological or economic factors. It may be difficult to predict these factors at the
current time, which argues for a conservatively large study area that could allow future contraction. It will
be significantly easier to make the study area smaller in the future than it would be to make it larger.
For projects that require analysis across multiple adjacent scenes, the use of Thiessen-polygons based
on the Landsat WRS-II reference system is advised. Thiessen polygons partition a space so that all
points within each polygon are closer to its center point than to the center point of any other polygon. We
provide a Thiessen-polygon coverage for Landsat scenes within the continental United States for use in
study area delineation. Ideally, the preprocessing and analysis study areas should include both the
Thiessen polygon proper and a small buffer around it, to later allow evaluation in the overlap area of
potential incongruities between the two scenes.
Absent a Thiessen polygon or area of interest, the study area should be defined as an area with
maximal overlap among the Landsat images in the stack to be processed. An IDL batchfile is provided
that can guide this process.
Therefore, the following steps are recommended for different situations:
If the area of interest is entirely contained in one Landsat scene, and if no adjacent areas of interest are
likely to be processed:
- Use the overlap area count method to create a preprocessing study area
- Use study area vector files, with a buffer, to create an analysis study area
If the area of interest is an entire scene, and if no adjacent areas of interest are likely to be processed:
-
Use the overlap area count method to create a single preprocessing and analysis study area
If the area of interest is an entire scene, and if adjacent scenes are also of interest:
- Use the buffered Thiessen Scene Area approach to create a single preprocessing and analysis
study area
A. Overlap area count method
Before starting:
- Ensure that the Image Info structure has been populated appropriately, with every desired image
included, and that the image_info_savefile has been created. Make sure that "image_info_savefile"
points to the full pathname of that savefile, then type:
IDL> overlap_file = build_image_overlap_area_image(image_info_savefile)
This program will run for a while, calculating the total intersection of all images in your stack. The
resultant image will be called "IMAGE_overlap_count.bsq" and will reside in the images directory (usually
\<pathrow>\images\). The filename will be contained in the "overlap_file" variable in the IDL session, in
case there is any doubt.
The resultant image will have a count for each pixel of the number of images that contain real data in that
pixel's location. The higher the count in a pixel, the more that pixel is overlapped by all of the images. As
you move to the edge of the stack, fewer and fewer images will overlap and the count image will
decrease in value as pixels as traversed toward the edge.
To identify the area of interest, identify where the number of scenes decreases fairly rapidly, or
alternatively where the number of scenes passes a pre-determined threshold. Then, the area of interest
mask can be created in an image-math module, such as ENVI's band math, to identify pixels where the
overlap count exceeds the threshold.
B. Vector-based method
If the desired study area already exists as a vector layer, the following steps are required:
1. Start Imagine and load the radiometric reference image in a viewer (see section IV if this has not been
done already).
2. In the viewer, choose “File/Open/Vector Layer” and navigate to the directory where the new study area
vector layer is located.
a. Open the vector layer, and ensure that it matches with the reference image as expected.
b. Use the ImageInfo (section IV, step 3 above) to ensure that the vector coverage is in the appropriate
projection and datum for the geometric-reference image.
i. Assuming that the radiometric reference will be UTM and NAD83, respectively, then the vector
coverage must be in UTM, NAD83.
(1) NOTE: If a vector coverage is in the Clarke 1866/NAD 27 projection/datum space, it must be
reprojected. It is recommended that this projection process occur in ESRI Arc software, using the
NADCON transformation approach. Arc/Info documentation on “Projections” provides more information.
However, if this approach is not possible, the steps for reprojecting a vector coverage in Imagine are
given below.
(2) If the coverage is not in the appropriate datum, it will be necessary to reproject it into the correct
datum. This can be done in various ESRI Arc products and in Imagine. The steps for doing this in Imagine
are detailed below. GIS professionals also may prefer to do this in Arc workstation or ArcTools.
ii. Reprojecting a vector coverage in Imagine.
(1) In a blank viewer, open the vector coverage.
(2) Click on the ImageInfo icon (section IV, step 3 above).
(a) The window associated with the “General” tab will pop up.
(3) Select “Edit/Reproject the coverage.”
(a) A “reproject coverage coordinates” window will pop up.
(4) Name the output vector in such a manner that the distinction in datums can be inferred. Follow datamanagement considerations in SOP 8 Data Management.
(5) Click on “Define new projection.”
(a) The “Projection chooser” window will pop up.
(6) Use the pulldown menu for the spheroid name field to select NAD83.
(7) Use the pulldown menu for “Datum name” to pick the NAD83 datum from the list.
(8) Ensure that zone is 10 and “North or South” is set to north.
(9) Click OK, then click OK again in the “reproject coverage coordinates” window.
(a) Save the coverage.
3. With the vector coverage and the geometric-reference image open in the same viewer, select
“Vector/Viewing properties.”
a. The “Properties for test” window for the vector coverage will pop up (fig. 9).
b. When vector layers are first shown in Imagine, the default is to display them as arcs, not polygons. To
manipulate polygons, as will be done below, the polygons must be displayed.
c. First, uncheck the box for “Arcs” by clicking it.
d. Then check the box to the left of “Polygon.”
i. To change the border and fill properties of the polygon, select the box to the right of the colored field to
the right of the polygon (in cyan in fig. 9).
ii. Keep this window open for now.
4. Then from the viewer with the vector coverage in it, select “Vector/Attributes.”
a. The “Attributes” window will pop up.
i. Each polygon will have a row in the table of attributes. Select all the rows that include polygons that are
desired to defined the study area. Typically, this should be a small number of polygons – often only one –
because the study area should be a large, internally connected space.
(1) To select a row, click on the row number in the left-hand side of the window (the “Record” column in
fig. 10).
ii. To select multiple rows, click and drag down all rows.
iii. Because polygons are displayed, selecting these rows will change all of the polygons on the viewer to
the selection color, which defaults to yellow.
5. With these polygons selected, select “AOI/Copy selection to AOI” from the viewer.
a. Note: If the coverage and the image are not in the same projection, this step will fail. It is not sufficient
to simply remove the image from the viewer to avoid this failure, because the AOI must be associated
with the correct datum of the image for all later steps. Therefore, if the coverage is not in the correct
datum, it must be either reprojected or the datum must be overruled, as noted in step 2 above.
b. This step creates a new AOI in the viewer, anchored to the geometric reference image.
6. Save the AOI.
a. In the viewer, select “File/Save/AOI Layer as.”
b. Navigate to the appropriate directory and save the AOI layer. Follow the SOP 5 on naming and file
location conventions.
7. This AOI is now the base AOI for all subsequent steps that involve clipping of imagery to the studyarea boundary, including clipping the DEM.
If the study area does not e
xist in vector form and must be created, follow steps 8 through 11.
C. Thiessen scene area method
a. Create a study area polygon based on the Thiessen/voronoi polgon
()
i.
In ArcMap open a new empty map
ii.
Select add data and navigate to and select the nwfp_Thiessen_polys.shp shapefile
iii. Select the Thiessen polygon for the scene you are working on.
1. In the table of contents, box to the far left that shows the layers in the project, right click on the
nwfp_Thiessen_polys shapefile
2.
Choose open attribute table
3.
Within the WRS_ALB_1 attribute find the path and row that you need.
4. Select that polygon by left clicking the grey box at the far left of the respective row. The row
should now be bright aquamarine
iv. Create a shapefile of just your study area (scene).
1.
In the table of contents right click on the nwfp_Thiessen_polys shapefile.
2.
Choose Data\export data
3. A dialog box will come up. There should be no need to change any of the options as the defaults
should be export selected features and use this layers source data. Change the name and path then click
OK.
4. After the processing you will be prompted as to whether or not you want to add the layer to the
project, choose yes.
v.
Buffer the Thiessen polygon to create the study area polygon
1. In the same ArcMap project make sure that the ArcToolbox window is open, if not click the red
toolbox icon to open the ArcToolbox window.
2. In ArcToolbox within the Favorites tab open the Buffer tool which is under Analysis
Tools\Proximity\Buffer or simply search for Buffer within the search tab.
3.
Populate the fields in the Buffer dialog box.
a. In the input features dropdown select the newly created polygon of the scene you are working on
(e.g. 4532_Thiessen_poly).
b. In the output feature class navigate to where you would like to save the buffer output and rename to
with a name reflecting a 2 kilometer buffer (e.g. 4532_thiessen_poly_2km_bufr).
c.
Make sure the linear units of the polygon you are about to buffer are in meters. If so make sure
linear unit is checked under Distance [value or field] in the Buffer dialog. Enter 2000 in the box directly
under linear unit . Select Meters from the drop down to the right.
d.
Hit the radio button for OK
e. A progress window will show that the buffer function is processing. This should eventually read
“Completed”. Hit the radio button for close. The new buffered polygon should appear in the table of
contents box.
4.
Double check that the buffer worked
a. In the table of contents make sure that the Thiessen polygon of your scene is checked and at the top
of the list of files. Make sure the second shapefile in the list of files in the table of contents is the newly
created buffered polygon. Also make sure it is checked.
b. Both of these shapefiles should appear in the display window to the right. Use the zoom tool to zoom
into an area, so that the perimeters of the 2 polygons are easily discernable from each other.
c.
Use the measure tool (measuring stick icon) to measure the distance between the two perimeters.
With the measuring tool active left click once on one perimeter move your mouse perpendicular to the
perimeter and left click on the other polygon’s perimeter.
d. At the bottom left of your screen just above “Start” there should be a segment length. This number
should be fairly close to 2000. If so you have successfully created a 2 kilometer buffer. Double click to end
the measure tool and get out of the tool by clicking on the arrow icon.
b. Converting thiessen polygon to mask
The study area polygon that was created in the previous step now needs to be converted to a mask.
i. Convert buffered polygon to raster
1. In ArcMap have your newly created study area with a 2 km buffer active in a project.
2. Open ArcToolbox (red tool box GUI).
3. Within the Favorites tab in ArcToolbox go to Conversion Tools\ To Raster\Feature to
Raster and double click to open this function.
4. Fill out the Feature to Raster dialog box:
a. Input features- Select the study area buffered by 2km polygon that you just
created in the previous section.
b. Field- Select the field that contains the scene name such as WRS etc..,
probably does not matter much though as there should only be one record.
c. Output raster- navigate to the folder you want to save it to (e.g.
4532\Study_area) and name it similar to the following: 4532_studyarea_compressed.img this will make
the file a compressed Imagine image.
d. Output cell size- input 30
e. Click o.k.
ii. Convert raster file from compressed to uncompressed format
1. Create a model to uncompress the .img file and make a study area mask.
a. Open ERDAS Imagine
b. Click the GUI for Modeler
c. In the Spatial Modeler window select Model Maker, this will open a new model.
d. From the toolbar on the right click the raster object (upper left icon) which looks like
a stack of squares (Figure 1). Place the raster object in the upper left corner of your blank model. This will
be the input raster.
e. Repeat step 4 placing another raster object in the lower right corner of the
model. This will be the output raster.
f. Place a function (circle icon) in the model between the two raster objects.
g. Select the connecting arrow (arrow icon above the question mark) from the
toolbar and connect the input raster to the function (circle). Do the same from the function to the output
raster. This will connect your model.
Figure 50. Study area mask model
2. Change your session preferences so that you are not using compressed Imagine
format for images.
a. From the main ERDAS Imagine toolbar select session.
b. Select Preferences from the Session drop down menu.
c. In the category window choose Spatial Modeler.
d. In the data compression drop down menu make sure none is chosen.
e. Click User save and then close.
3. Complete and run the study area mask model.
a. Double click on the input raster in the model. A dialog box will come up inside
under file name navigate to your *_studyarea_compressed.img file. Leave everything else as the defaults.
b. Double click on the function icon (circle) in the model (Figure 1).
c. Click on the only available input then add ne 0 (i.e. not equal to zero) in the
formual box at the bottom. This will change all non-zero values to 1 and leave zero values that are
outside the mask as zeros. Click OK.
d. Double click the output raster in the lower right of the model. Under file name
navigate to the Study_area folder and name the file similar to 4532_2km_buffer_studyarea_mask.img.
Under Output: File type: choose Thematic. click OK. choose Signed 16 bit
e. Save the model (File\Save as) to the scene's Study_area folder with a similar
name such as 4532_2km_buffer_studyarea_mask.gmd.
f. To run the model click the red lightning bolt gui to execute the model.
iii. Make sure masks have a value of 1 .
1. In Erdas Imagine open a viewer.
2. Click the folder GUI to navigate to and open your newly created study area mask.
3. Rt click in the viewer and choose Fit image to window to better view the image.
4. Click on the cross hairs GUI in viewer #1 (Start/Update Inquire Cursor). Cross hairs
should come up in your viewer as well as another dialog box showing the pixel value at the location of
the cross hairs.
5. Either with the arrows on your keyboard or with your cursor move the cross hairs
around in the viewer to make sure that the study area foot print has values of 1 and outside the footprint
values of 0 or further out no values.
VIII. SETTING UP IMAGE INFORMATION
STRUCTURE
Working with stacks of imagery means handling many sets of files, often with the same process being
applied to all images separately. To allow for repetition and efficiency, and to ensure consistency of file
naming, all LandTrendr processing is documented and updated in a single IDL "structure variable" that is
passed along through all steps. The structure is saved to a separate "image info" file in the proprietary
IDL format (with the file name ending in "*.sav").
The image info structure is critical to success. It must be set up correctly, and it must be
maintained and checked for verity at every step in the preprocessing and analysis steps!
Remember it this way: "As the image info structure goes, so goes your analysis."
There are essentially two operations that involve the image info structure: Setting up a new structure,
and repopulating an existing one. During set-up, an algorithm scours a directory for files that conform to
a baseline formatting rule and builds the image info structure variable around those files. If you are
setting up for the first time, or adding a new image into the stack, you need to run the set up routine.
Repopulating the image info structure occurs after you have set up the basic structure, but need to
update it to capture either a modification or a new file, or if the original structure has been lost but all of
the files resultant from later processing exist and need to be replaced into the structure.
1. Setting up the image info structure
a. Start an IDL session
Open the file "set_up_landtrendr_files_template.pro"
Save file as "set_up_landtrendr_files_pprr.pro" where pp= path number and rr = row number (example:
"set_up_landtrendr_files_4234.pro" for landsat scene path: 42 row: 34.)
Figure 48.
iv. Alter the following variables in the file:
(a) path = 'image path'
example: path = 'g:\4234\glovis\'
(b) image_info_savefile = path + 'landtrendr_image_info_pprr.sav'
example: image_info_savefile = path + 'landtrendr_image_info_4234.sav'
(c) useareafile = 'path to mask image'
example: useareafile = 'g:\4234\gis_data\4234_final_usearea.img'
(d) Save the file
Figure 49. Alter the path, image_info_savefile, and useareafile variables in the
'set_up_files_landtrendr_pprr.pro' procedure file.
v. Run the procedure
(a) IDL> @set_up_landtrendr_files_4234.pro
One thing that happens in this step is that the COST image is identified as different from the other images
-- it is labeled in the save-structure as the reference image for the next step
vi. Verify results of procedure
(a)IDL> print, image_info
This will print in the unformatted data stored in the image_info variable in the output log window.
(b) IDL> print, image_info.year
This will print a list of the image years stored in the image_info variable in the output log window.
(c)IDL> print, image_info.image_file
This will print a list of the image names stored in the image_info variable in the output log window
(d) IDL> print, image_info.image_info_variable
This will print a list of an other variable of interest stored in the image_info variable in the output log
window. A full list of variable names is listed in table .
(e) Rerun procedure if there are any files missing files or errors in the expected values.
X. COMPENSATING FOR SENSOR AND
ATMOSPHERIC INFLUENCES IN INPUT
IMAGES (MADCAL)
In most cases, COST processing will not take care of all of the atmospheric effects that could cause
confusion when change detection is conducted later. Two COST-processed images likely will differ
somewhat in their claimed reflectances for a given pixel, even if no real change has occurred, because
the true nature of the atmosphere above every pixel is not known. Because some of the parks’ monitoring
goals require distinction of subtle spectral difference, even these slight artifacts could be problematic.
Therefore, it is necessary to further normalize images before change detection can take place.
Many approaches exist in the remote sensing literature to normalize the radiometric and spectral
qualities of one image to another. The authors tested a variety of these approaches and determined that
a recently published approach known as Multiple Alteration Detection Calibration (MADCAL) works
extremely well and has the great advantage of being automated.
This step is carried out with the IDL>@mad_landtrendr_<pprr> batchfile.
Populating this batchfile requires that you follow the steps below.
A. Identify the radiometric-reference image.
1. Radiometric-reference image is the COST corrected image from Section VI.
B. Identify subsets
Madcal requires the user to define the upperleft coordinate of a subset box. This subset is used
during the MADCAL process to identify no-change pixels and then are pooled together with nochange pixels from other subsets in order to derive the relationship between the reference image
and the input image. A large number of subsets are created to improve the derivation of nochange pixels. For error checking purposes, a shapefile of each subset is created.
1. Open the COST corrected image and usearea mask in a new ArcMap session. Enable the
drawing toolbar.
2. Create Subsets as drawing objects
a.
Draw subset 1
i. Using the rectangle draw tool, draw a rectangle of a random size.
ii. Edit the properties of the rectangle drawing object
1. Symbol: no fill color, red outline color
2. Size and Position:
1000 pixel subset
width: 30,000 m
length: 30,000 m
*note the width and length field units are based on the Layer properties and the coordinate system
i. Select and move rectangle drawing object to the location for subset
a. Create subset 2 to 100
i. Select subset 1, copy and paste, to duplicate subset 1. This is now subset 2.
ii. Select and move subset 2 to a location (possibly overlapping subset 1) within the
bounds of the COST corrected image and the usearea mask.
iii. Repeat the previous steps until there are enough subsets to cover the entire
image (and without any subset exceeding the bounds of the reference and input
images).
Figure 51. Madcal subsets located using ArcMap drawing objects.
3. Create subsets as a polygon shapefile
a. Select all the graphic drawing objects representing the MADCAL subsets created in step
2. Use a tool to convert the graphics to a polygon shapefile. (For example, the extension
XTOOLS Pro/Feature Conversions/Convert graphics to shapefiles). Save file according
to the instructions in SOP 8: Data Management.
4. Identify Upper Left coordinate of subsets
a. Convert the polygons into points using ArcToolbox/Data Management Tools/Feature
Vertices To Points/
i. Input Features: Subset polygons created in step X.B.3
ii. Output Feature Class: pprr_madcal_upperleft_coordinates_pts.shp
iii. Point Type (optional): mid
a. Calculate the x, y coordinate of the pprr_madcal_upperleft_coordinates_pts.shp using
ArcToolbox/Data Management Tools/Add XY Coordinates/.
This operation will add two fields (Point_X, Point_Y) to the attribute table of the point shapefile.
i. Input Features: Subset polygons created in step X.B.4
C. Modify mad_landtrendr_<pprr> batchfile
In this step, the analyst modifies the madcal batchfile to include the subsets created in step X.B.4 and
initiates the MADCAL batchfile.
1 . Format the Upper Left coordinate point shapefile attributes using Excel.
The MADCAL batchfile requires the upper left coordinate for each subset to be entered in a specific
manner. This step uses the x and y attribute from the point shapefile and then formats it as required.
Finally, these values are pasted into the MADCAL batchfile.
a. Open the pprr_madcal_upperleft_coordinates_pts.dbf in Excel
b. In an empty cell to the right of the attribute values (usually cell G3 or so), enter the following
formula:
CONCATENATE("[",A2,",",B2,"], $")
where A2 is the X coordinate field for subset 1, B2 is the Y coordinate field for subset 1.
i. This should create a cell value which looks like:
[-2299354.68659,2029246.4872], $
c. Populate the remaining subset rows with the formula in the step above.
d. Copy the results of the formula into a new row using paste special/values.
2. Populate the MADCAL batchfile with subset coordinates
a. Copy the formatted coordinates from the previous step (X.C.1.d).
b. Paste the new subset values into the batchfile in the appropriate location, starting after the
'subset_coordinates = [' line.
subset_coordinates = [
[-2299354.68659,2029246.4872],$
[-2279910.84659,2052429.5272],$
[-2293745.88659,2062899.2872],$
[-2307954.84659,2047568.5672],$
[-2335624.92659,2192649.5272] ]
c. Verify correct syntax of the subset values.
(1) note the lack of comma and dollar sign at the end of the last subset.
3. Alter the scene specific variables in the MADCAL batchfile.
a. path = 'path to images directory'
example: path = 'K:\4533\glovis\'
b. image_info_savefile = path+'landtrendr_image_info_pprr.sav'
example: image_info_savefile = path+'landtrendr_image_info_4533.sav'
c. madcal_summary_csv_file = path+ 'pprr_madcal_summary.csv'
example: madcal_summary_csv_file = path+ '4533_madcal_summary.csv'
d. min_correlation= 0.75
possible values range between 0-1.
e. fix_only_these: -1
possible values include:
-1 (operate on all images), - default option
[year] (to operate on a specific year,
[year1, year2, year3] (to operate on a list of specific years)
f. save file
4. Load Image_Info variable into memory
IDL>path = 'K:\pprr\glovis\'
IDL>image_info_savefile = path+'landtrendr_image_info_pprr.sav'
IDL>restore, image_info_savefile
IDL>print, image_info.image_file
5. Run batchfile
After loading the image_info variable into the computer memory, the batchfile is initiated using the
command below. Progress can be tracked through the variety of windows which appear as the process
proceeds.
IDL>@madcal_pprr.pro
6. Errors while running batchfile
The MADCAL batchfile may stop or produce an error message. At which point the analyst can refer to
the list of common errors and the fix. In addition, the analyst would want to consult the following files:
i. Diagnosis File located in C:\temp\
ii. IDL Processing Log WIndow
D. Analyzing MADCAL results
After the MADCAL batchfile has successfully completed, the analyst must examine the resulting log files
and scatterplots to insure adequate results of the radiometric calibration between the COST reference
image and the input image. Inadequate subset coverage, differing phenology or significant land cover
change, or cloudy images may influence the correlation between images.
1. Summary File
a. Open the file *pprr_madcal_summary.csv
b. Copy and paste the summary rows into the 'MADCAL' worksheet within the
'Processing_LandTrendr_pprr.xls' workbook.
c. Examine the Summary table
Note any images which you can answer 'yes'
i. Are there any correlations below 90%?
ii. Are there any images with a 'Success' value of '0'
iii. Are there any images which used a very small number of subsets?
If there are any images an analyst notes with an answer of 'yes' to the above questions, examine the
scatterplots for further clarification about the relationship between the images.
2. Methods for improving madcal results on images with poor initial results
*Evaluating scatterplots showing band to band relationships*
i. If the summary spreadsheet indicates that a given band to band relationship
appears to have questionable correlations, it is necessary to evaluate the source
of that poor correlation. Two broad classes of poor correlation are possible:
1. Case 1: There is significant scatter about the fitting line, but that scatter
is relatively evenly distributed about the line
a. The fitting line is still appropriate, meaning that no alteration
need be done, and the madcal normalization for this band can be
accepted.
i. Case 2: The fitting line is being drawn away from the dominant axis by pixels on
one side of the fit
Figure 52. Poor calibration. The pixels represented by the points to
the right and below the red fitting line are pulling the fit away from the
line that would be captured by the tight cluster of pixels (in reds,
oranges and blues) toward the left of the scatterplot. This is most likely
due to contamination in one or more subsets.
a. In this case, the image must be re-run with different settings.
i. Step 1: If the number of subsets used for the run was
high enough, the best option is to simply re-run this
image with a higher standard for minimum correlation.
The higher correlation threshold will eliminate some of
the questionable subsets.
ii. Step 2: If Step 1 does not help, then additional subsets
will need to be identified. Identify more subsets (See
above), and then re-run the MADCAL for only the years
where problems exist (See below for re-running
madcal).
iii. Step 3: If Steps 1 and 2 do not help think about not
using the image.
3. Re-running MADCAL batchfile
a. Specific year
i. Alter the madcal_pprr.pro variable
fix_only_these: -1
to fix_only_these: [year]
ii. IDL>@set_up_landtrendr_files_pprr.pro
iii. IDL>@madcal_pprr_year.pro
E. Verify Image_Info variable
Once you've completed MADCAL, the next step is to verify that the image_info file is updated and that the
image_info is recognizing that the MADCAL images now exist. This is particularly useful if you're simply
adding a few new images to an existing stack to make sure that the new ones got updated appropriately.
1. Bring the most up-to-date version of the image_info into memory
a. From the command line of IDL, type:
IDL> path = 'K:\pprr\glovis\' ;this defines the directory
IDL> image_info_savefile = path + "landtrendr_image_info_pprr.sav"
;this defines the file name.
IDL> restore, image_info_savefile
; this brings the information in the file into memory.
2. Now that the image_info is in memory, you can simply print the names of the image files to see if
they're right.
a. From the command line of IDL, type:
IDL> image_info_savefile.imagefile
2. Check the file names.
a. After madcal, all of the file names should have "_to_" in the filename. If you still see any
that do not have "_to_" in the filename, other than the single image that you used as the
MADCAL reference image, then do the following.
i. In a normal file explorer, look in the directory of the offending image to see if the
madcal'd image exists -- i.e. if the filename with "_to_" is actually in existence.
1. If not, it means that MADCAL didn't run on it. Check to make sure that
you did not inadvertently leave this year out of the run by, for example,
setting the "fix_only_these" parameter to years that didn't include this
one.
2. If so, then it means that somehow the new file did not make it into the
image info structure. First, check to make sure that the time stamp on
the filename is appropriate -- it may be an old version that should have
been overwritten, in which case you should re-run the madcal batchfile
with fix_only_these set to this particular file. If it's got an appropriate
timestamp, then you need to update the image info. From the command
line type:
IDL> repopulate_image_info, image_info_savefile.
F. Create header files
The files created during the MADCAL process must have a header file (*.hdr.flat) defined before ENVI or
ArcGIS will correctly recognize the raster file. The header file type created by the MADCAL algorithms
(and all others associated with this protocol using IDL routines) are a generic type that is only
recognizable to the internal image reading/writing procedures associated with LandTrendr. Therefore,
they must be converted into the ENVI structure.
Example Header file template created in step IV.A.1
ENVI
description = {
File imported into ENVI.}
samples = 8689
lines = 8214
bands = 6
header offset = 0
file type = ENVI Standard
data type = 2
interleave = bsq
sensor type = Unknown
byte order = 0
map info = {MRLC_albers_con_eq_area, 1.5000, 1.5000,-2393850.0,2041530.0,30.0000,30.0000, North
America 1983, units=Meters}
projection info = {9, 6378137.0, 6356752.3, 23.000000, -96.000000, 0.0, 0.0, 29.500000, 45.500000,
North America 1983, MRLC_albers_con_eq_area, units=Meters}
wavelength units = Unknown
The routine to convert headers requires both the header that was created by the madcal routine, and a
template header. The template header is an ENVI format file from which only one key piece of
information will be taken: the map projection information. Thus, you must have at least one image in
ENVI format for the study area in which you are working, in the projection used for all of the images,
before the conversion of headers can occur.
1. IDL>convert_bsq_headers_to_envi, 'input directory', 'header file template'
example: convert_bsq_headers_to_envi,
'J:\SoCal_chng\socal_images\4335\',’J:\SoCal_chng\socal_images\mrlc_template_headerfile.hdr'
example: convert_bsq_headers_to_envi,
'J:\SoCal_chng\socal_images\4335\',’J:\SoCal_chng\socal_images\mrlc_template_headerfile.hdr',
/overwrite
XI. GENERATING LANDTRENDR
TASSELED-CAP IMAGES
A. Load Image_Info variable into Memory
IDL>path = 'K:\pprr\glovis\'
IDL>image_info_savefile = path+'landtrendr_image_info_pprr.sav'
IDL>restore, image_info_savefile
IDL>print, image_info.image_file
B. Alter make_landtrendr_tcimages_pprr.pro
Change the values in the batchfile to relfect the scene and appropriate path.
path = 'K:\pprr\glovis\'
savefile = path+'landtrendr_image_info_pprr.sav'
C. Run batchfile
IDL>@make_landtrendr_tcimages_pprr.pro
D. Verify Results
1. Verify Image_Info variable
Verify that the variable 'tc_file' was populated with a file name after the operation.
IDL>path = 'K:\pprr\glovis\'
IDL>image_info_savefile = path+'landtrendr_image_info_pprr.sav'
IDL>restore, image_info_savefile
IDL>print, image_info.tc_file
2. View results
a. Create header files
The files created during the Tasseled Cap process must have a header file (*.hdr.flat) defined before
ENVI or ArcGIS will correctly recognize the raster file.
i. IDL>convert_bsq_headers_to_envi, 'input directory', 'header file template'
example: convert_bsq_headers_to_envi,
'J:\SoCal_chng\socal_images\4335\',’J:\SoCal_chng\socal_images\mrlc_template_head
erfile.hdr'
example: convert_bsq_headers_to_envi,
'J:\SoCal_chng\socal_images\4335\',’J:\SoCal_chng\socal_images\mrlc_template_head
erfile.hdr', /overwrite
b. Open the Tasseled Cap files using ENVI.
XII. CLOUDSCREENING
A. Producing cloudscore images
1. Determine reference cloud image
The cloud reference image should be the first cloud-free image from the available images. Document this
image in the appropriate column in the 'Processing_LandTrendr_pprr.xls'/Image Status worksheet.
Recommendations for selecting the cloud reference image include:
a. Early image in stack
b. Cloud free
c.
2. Alter the batchfile 'make_cloudscore_images_pprr.pro'
a. Save file, replacing pprr with the appropriate path, row number.
example: 'make_cloudscore_images_4533.pro'
b. path = 'path to images'
example: path = 'G:\4533\glovis\'
c. image_info_savefile = path+'landtrendr_image_info_pprr.sav'
example: image_info_savefile = path+'landtrendr_image_info_4533.sav'
d. clear_year = year
example: clear_year = 1987
'
e. Save file
3. Load Image_Info variable into Memory
IDL>path = 'K:\pprr\glovis\'
IDL>image_info_savefile = path+'landtrendr_image_info_pprr.sav'
IDL>restore, image_info_savefile
IDL>print, image_info.image_file
*verify the image_file values are the MADCAL images.
4. Run Batchfile
IDL>@make_cloudscore_images_4630.pro
5. Create header files
The files created during the cloud scoring process must have a header file (*.hdr.flat) defined before ENVI
or ArcGIS will correctly recognize the raster file.
IDL>convert_bsq_headers_to_envi, 'input directory', 'header file template'
example: convert_bsq_headers_to_envi,
'J:\SoCal_chng\socal_images\4335\',’J:\SoCal_chng\socal_images\mrlc_template_headerfile.hdr'
example: convert_bsq_headers_to_envi,
'J:\SoCal_chng\socal_images\4335\',’J:\SoCal_chng\socal_images\mrlc_template_headerfile.hdr',
/overwrite
B. Determine thresholds for clouds and shadows
1.
Using ENVI open cldiff and refl image in separate windows
Figure 52.
2.
Use geographic link to connect the two windows.(Right Click on image and choose
geographic link)
a.
Check alignment by making sure a landmark point is in the same point
in both images.
3.
Go to tools/color mapping/density slice on the clddiff window
a. Select the clddiff image band 1, and delete all but lowest value range.
b.
Edit range to have a max value of -100. Apply the range.
c.
Check visible clouds in the refl image to ensure that they are masked
over in red on the clddiff image.
d.
Clouds do not need to be entirely covered, but the range should be
adjusted so that at least part of all major clouds are masked.
e.
If there are no clouds in the image choose a very low value, but not
lower than the minimum.
Figure 53.
4.
Input the determined value into the excel processing sheet (Link), making sure to note
the year being viewed and that the value is for clddiff.
5.
Repeat steps 1-4 using shddiff instead of clddiff, but instead of using -100 for a
starting value, use -400.
Additional notes on cloud/shadow screening
The shadow images are tough to get thresholds, because it's easy to start picking up
parts of the "real" landscape outside of shadows. All of these thresholds are a
balancing act.
Insure that shadows from the clouds are adequately covered. As long as the
reference cloud image year is earlier than the one you're doing, a more aggressive
shadow threshold (i.e. one that gets more shadow) should only pick up landscape
features that are slowly changing, primarily regrowing clearcuts in the PNW, at least.
And for those areas of the landscape, it's not a huge penalty to mask them out
occasionally because it's the longer-term trend in those areas that we want.
In other words, when considering the balance between picking up more of the target
features (cloud shadows) and non-target features (non-shadowed landscape
features), recognize that some non-target features are more "expendable" than
others. We have many other years of imagery to pick up the changes, as long as we
don't mask them out in every single year, which should be relatively unlikely.
Two notes:
a. Cloud shadows tend to be much harder to pick out of the trajectories than
clouds. Clouds produce spikes that are extreme; shadows are usually less
dynamic. As each pixel is processed in the landtrendr algorithm, big spikes
that remain after masking are relatively easily taken out, so the penalty for
leaving some clouds in is generally less than leaving cloud shadows in.
b. The "increased aggressivness" directive doesn't really apply to the cloud
masking (vs shadow masking), because the landscape features that tend to
get picked up with aggressive cloud masking are new clearcuts, and for those,
it's important to retain them as much as possible so we get the year right.
In other words, be more aggressive in shadow masking and retain some conservatism with the cloud
masking.
Two more notes:
a. It's best to be the most aggressive with shadow masking when working with the first image in the
stack
b. It's best to be the most aggressive with cloud masking when working with the last image in the
stack
It is also a good idea to keep notes in the cloud and shadow threshold spreadsheet regarding any
anomalies that you see (snow, missing data), potentially alternate values for the thresholds (maybe a
more conservative value for clouds or shadows - this helps if the thresholding has to be rerun because it
did not catch enough or caught too much), the types of clouds (fog, smoke, haze, cirrus, cumulus, etc)
XIII. CREATING CLOUD MASKS
A. Alter batchfile
1. Open the template file 'mpcm_pprr.pro'
a. Save file with a new name, replacing the pprr with the appropriate path, row
b. path = 'path to images'
example: path = 'G:\4533\glovis\'
c. image_info_savefile = path+'landtrendr_image_info_pprr.sav'
example: image_info_savefile = path+'landtrendr_image_info_4533.sav'
d. cloudthresholds = [[year1, clddiff_value, shddiff_value], $
[year2, clddiff_value, shddiff_value]]
where the years are in sequential order (by julian date if more than one image in one year), clddiff_value
was determine in step XII.B.4., and shddiff_value was determined in step XII.B.5.
example:
cloudthresholds = $
[ [1985, -400, -3200], $
[1986, -150, -23800], $
[1987, -100, -500], $
[1989, -100, -31500], $ ;This is julian date 187
[1989, -100, -32000], $ ;This is julian date 230
[1990, -215, -26800], $
[1991, -110, -450], $
[1993, -150, -1000], $
[1994, -120, -25700], $
[1995, -120,-400], $
[1996, -130, -31700], $
[1997, -120, -32600], $
[1998, -100, -31500], $
[1999, -120, -150], $
[2000, -120, -25200], $
[2001, -120, -100], $
[2002, -260, -29900], $
[2003, -215, -31800], $
[2004, -100, -250], $
[2005, -120, -400], $
[2006, -120, -200], $
[2007, -110, -200], $
[2008, -120, -32500] $
]
'
e. Save file
B.
Run batchfile
1. Load Image_Info variable into Memory
IDL>path = 'K:\pprr\glovis\'
IDL>image_info_savefile = path+'landtrendr_image_info_pprr.sav'
IDL>restore, image_info_savefile
IDL>print, image_info.image_file
*verify the image_file values are the MADCAL images.
2. Initiate batchfile
IDL>@mpcm_pprr.pro
C. Create header files
The files created during the cloud scoring process must have a header file (*.hdr.flat) defined before ENVI
or ArcGIS will correctly recognize the raster file.
a. IDL>convert_bsq_headers_to_envi, 'input directory', 'header file template'
example: convert_bsq_headers_to_envi,
'J:\SoCal_chng\socal_images\4335\',’J:\SoCal_chng\socal_images\mrlc_template_head
erfile.hdr'
example: convert_bsq_headers_to_envi,
'J:\SoCal_chng\socal_images\4335\',’J:\SoCal_chng\socal_images\mrlc_template_head
erfile.hdr', /overwrite
D. Verify Results
1. Verify Image_Info variable
Verify that the variable 'tc_file' was populated with a file name after the operation.
IDL>path = 'K:\pprr\glovis\'
IDL>image_info_savefile = path+'landtrendr_image_info_pprr.sav'
IDL>restore, image_info_savefile
IDL>print, image_info.tc_file
2. View results
a. Create header files
The files created during the Tasseled Cap process must have a header file (*.hdr.flat) defined before
ENVI or ArcGIS will correctly recognize the raster file.
i. IDL>convert_bsq_headers_to_envi, 'input directory', 'header file template'
example: convert_bsq_headers_to_envi,
'J:\SoCal_chng\socal_images\4335\',’J:\SoCal_chng\socal_images\mrlc_template_headerfile.hdr'
example: convert_bsq_headers_to_envi,
'J:\SoCal_chng\socal_images\4335\',’J:\SoCal_chng\socal_images\mrlc_template_headerfile.hdr',
/overwrite
b. Open each images Cloud mask file using ENVI.
i. Repeat process if image is not as expected.
Figure 54. Example of cloud and shadow mask result.
XIV. WORKS CITED
ITTVIS:
Protocol for LandTrendr
Standard Operating Procedure (SOP) #3
LandTrendr Segmentation
Version 0.01 (October 1, 2009)
Revision history log:
Previous Version
Revision Date
Author Changes Made
Reason for Change
New Version #
New
April 12, 2009
REK
New
0.01
Aug 12, 2009
PN
populated
0.01
September 2009
REK
Updated
0.01
Table of Contents
1. Overview
2. I. Temporal segmentation
a. A. Prepare batchfile
b. B. Run the batchfile
3. II. Fit-to-vertices (optional)
a. A. Prepare the FTV component of the LT batch file.
b. B. Outputs from FTV
c. III. Evaluate results
032
,,,,
Overview
After completing the steps described in SOP 2, there exists a radiometrically normalized stack of
images with cloud-masks. In SOP 3, the core LandTrendr algorithms are applied to this stack of images
in a process called temporal segmentation (Figure 1). The temporal trajectory of each pixel in the image
stack is extracted and a series of fitting algorithms are applied to determine which straight-line segments
can be used to efficiently describe the overall shape of that trajectory. The product of SOP 3 is a set of
files that capture the information about those fitted segments, and which are used for all subsequent
mapping (SOPs 4 & 6).
Figure 1. SOP 3 within the LandTrendr process.
As with the pre-processing steps in SOP 2, the LandTrendr algorithms are implemented using an IDL
batchfile. In the case of the LandTrendr algorithms, all of the relevant steps can be incorporated into a
single batchfile described in this SOP. However, each section of that batchfile involves separate
considerations and parameter sets, and thus each subsection is described in separate sections below.
I. TEMPORAL SEGMENTATION
The core step in the LandTrendr process is implementing a set of algorithms that simplify spectral
trajectories using sequential straightline segments (as in Figure 1 above). The user passes a set of "run
parameters" described below to the central LandTrendr processing algorithms.
This portion of the batchfile is mandatory.
Prepare batchfile
a. Create new batch file:
i. Open template file 'lt043_templatefile_nbr.pro' using the IDLDE. The following
text shows the portion of this file where parameters controlling the run are
defined.
ii. Note that any text that follows a semi-colon (;) is considered a comment by IDL,
not a command, allowing notes to be inserted into the batchfile for reference.
iii. Also note that the formatting of the document displayed here is narrower than the
original text files, causing some text to wrap around to the next line when viewed
here. IDL will not see the wrap of text.
iv. Save the file as 'lt043_pprr_nbr.pro' where pprr is replaced with the appropriate
path, row and nbr is replaced with the appropriate index value for this
LandTrendr run.
-----------------------------------Start of file below here:
pathrow_name = '4526' ;no slashes
top_path = 'y:\' ;make sure to put slash (\) at end
base_index = 'nbr'
run_name = '' ;a unique identifier, if you're trying different runs on same index and stack
mask_image = 'Y:\4526\Study_Area\4526_study_area_mask_ultimate.img'
;1's for all pixels to be processed, 0 otherwise
;the mask image defines the subset. subset can be manually set further down, if desired
images_path = top_path + pathrow_name+ '\glovis\'
;change if glovis is not the path
outputs_path = top_path + pathrow_name+ '\outputs\'+base_index+'\'
;change if outputs is not the path
template_file_for_map_projection = 'f:\mrlc_template_headerfile.hdr'
;this is an envi headerfile
;which contains the datum/spheroid information for the images
; this image does not contain the actual coordinates of the images
; in this run; it only needs to contain the generic projection
; information that applies to the coordinates that will be
; read from the study area and raw image files.
;these parameters will likely stay the same for most run situations
;segmentation parameters
background_val=0 ;value to ignore in input image bands
divisor=1 ;-1 = calculate it for me
minneeded=6 ;minimum number of years needed in a pixel for it to calculate trajectory
mask_image=mask_image ;mask image has 1's everywhere LT should be run, 0's otherwise
output_base=output_base ;the basic name for all output files, other strings concat'd to end
kernelsize=3 ;the size of box over which to average before run LT on a pixel -- defines one side of the box
(i.e. 3 = 3 by 3 box)
pval=0.05 ;do not accept trajectories whose p_of_f is greater than this. 0.05 is typical.
fix_doy_effect=1 ;set to 1 if mean effect of doy should be taken out using 2nd order curve
max_segments=6 ;the maximum number of segments to search for, cannot be > 6. Set to -1 to have it
calc'd on the fly
recovery_threshold=1.0
;=1/min_number_years_for recovery segments cannot result in rate faster than 1/min number of years
skipfactor=3 ;If >1, then not every pixel is run, but ever "skipfactor"th pixel is run in both x and y and the
;remainder interpolated. 3 is a good choice
desawtooth_val=0.9 ;0.0-1.0 if set to <1.0, spikes that start and end at near same value are removed. the
value sets the
;threshold on how different they can be to be removed -- 1.0 means no desawtoothing, .9 means that
;spikes with differences > 0.90 in start and end point are fixed.
distweightfactor= 2
vertexcountovershoot= 3
bestmodelproportion= 0.75
a. Alter scene specific variable values
i. pathrow_name = '4526 '
The path and row refer to the path and row of the Landsat sensor. This string will be concatenated to the
top_path parameter below to find the image info savefile that was produced in SOPs 1 and 2. no slashes
i. top_path = 'y:\'
drive letter of location of LandTrendr project. Put slash (\) at end
i. base_index = 'nbr'
Different Landsat indices enable understanding of different phenomena on the surface of the earth. Table
1, below, is a list of valid indices which LandTrendr will evaluate for trends. Enter the 'Index' value of the
desired spectral index into this variable.
Table 1. Spectral index values available for LandTrendr processing.
Index
Name
Formula
band1
Landsat Band 1
--
band2
Landsat Band 2
--
band3
Landsat Band 3
--
band4
Landsat Band 4
--
band5
Landsat Band 5
--
band7
Landsat Band 7
--
ndvi
Normalized Difference Vegetation (Band 4 - Band 3) / (Band 4 +
Index
Band 3)
nbr
Normalized Burn Ratio
(Band 4 - Band 7) /
(Band 4 + Band 7)
brightness
Tasseled Cap Brightness
--
greenness
Tasseled Cap Greenness
--
wetness
Tasseled Cap Wetness
--
i. run_name = 'name'
This is a unique identifier for the benefit of the user, to help distinguish among different runs, if different
parameter sets are going to be used runs on same index and image stack. It will be inserted into the file
name for all output files
i. mask_image =
'Y:\4526\Study_Area\4526_study_area_mask_ultimate.img'
This image will define the footprint of the LandTrendr analysis. The image is a raster (either Envi or
Imagine format) with the value 1 for all pixels where LandTrendr is to be run, and the value 0 (zero) for
pixels where it is not to be run.
i. year_range = [1985,2008]
This parameter describes the range of years over which the user wishes to run the LandTrendr algorithm.
Currently, this parameter cannot be used to constrain the analysis to fewer years than are available in
the entire stack. Therefore, please ensure that the first and last years noted in this parameter correspond
to the actual first and last years of the tack.
i. template_file_for_map_projection =
'f:\mrlc_template_headerfile.hdr'
This is an envi headerfilewhich contains the datum/spheroid information for the images this image does
not contain the actual coordinates of the images in this run; it only needs to contain the generic projection
information that applies to the coordinates that will be read from the study area and raw image files.
a. Alter segmentation parameters (usually not neccessary)
i. background_val
LandTrendr will ignore pixels with this value.
In most cases, this should be set to zero, as the cloud-masking algorithms in SOP #2 will convert masked
areas to 0.
i. divisor
default: 1
LandTrendr can optionally scale the input spectral index to a different value before calculating the fitting.
This may be necessary to maintain the outputs in an appropriate range for the signed-16-bit output
format (which only ranges from -32767 to + 32766).
Set to a value of 1 to use the index without scaling.
Set to a value > 1 to make the index smaller before running the segmentation -- for example, a value of
10 will convert spectral index values of 4250 to 425
Set to a value < 1 to inflate the values -- for example, a value of 0.1 will convert a spectral index of 0.425
to 4.25
i. minneeded
Default: 6
The minimum number of usable years in a pixel for it to be considered valid for calculating trajectories.
Each year a pixel is masked out for clouds or gaps, it will reduce the number of usable years for the
trajectory.
Note that the number of desired segments (a different parameter) should be no greater than this value.
default: The full path name of the image that defines the area where LandTrendr segmentation is to
occur.
The footprint (the upper left and lower right corners) of this image defines the bounds of the output
images.
The footprint cannot exceed the bounds of the input image stack.
default: none
Example: "run1"
A string code for the user's benefit that will identify this particular run. Useful for distinguishing among
different runs with slightly different run parameter sets.
The string will be concatenated to all of the output filenames.
i. kernelsize
default: 1
The size of box over which to average before run LT on a pixel -- defines one side of the box (i.e. 3 = 3 by
3 box)
i. pval
default: 0.05
The value above which trajectories for a given pixel will be considered "no-change."
The pval is calculated from a standard f-statistic probability table, based on the variance explained by the
fitted segmentation vs the residual variance in the original spectral trajectory not explained by the fitted
trajectory.
i. fix_doy_effect
default: 1
Set to 1 for LandTrendr to remove the mean effect of day-of-year on the spectral index.
A second-order curve is fit to the spectral data as a function of day of year.
Note: pixels that experience large disturbances will have poor fits, and thus will not show any obvious
day-of-year effect, and thus will not be corrected. Only pixels that are essentially stable, but showing a
spectral response solely as a function of sun angle or phenology, will be corrected. Thus, this setting has
the potential to remove false positive signals.
i.
max_segments=6
default: 6
The maximum number of segments into which any given pixel's trajectory can be divided.
Note that this number should be significantly (factor 3, roughly) less than the number of available years of
imagery. Thus, if fewer than approximately 15 or 20 years of imagery are available, the maximum
number of segments should be less than 6.
j.
recovery_threshold=1.0
default: 1.0
range: 0.0 to 1.0
= 1/min_number_years_for recovery segments cannot result in rate faster than 1/min number of years
This value defines a filter value used by the segmentation algorithm to eliminate segments that recovery
too fast for realistic ecological conditions of the site.
The value itself is calculated as: 1/minimum_number_of_years_for_recovery segments
k.
skipfactor=1
If >1, then not every pixel is run, but ever "skipfactor"th pixel is run in both x and y and the remainder
interpolated. In general, a skip factor of 3 is appropriate for exploratory runs, and a skip factor of 1 is
good once parameters have been settled upon. A skip factor of 3 runs nine times faster than a skip factor
of 1.
l.
desawtooth_val=0.9
0.0-1.0 if set to <1.0, spikes that start and end at near same value are removed. the value sets the
threshold on how different they can be to be removed -- 1.0 means no desawtoothing, .9 means that
spikes with differences > 0.90 in start and end point are fixed.
m. distweightfactor= 2
For forested and woody systems, set to 2. For grassland or very open systems, set to 1. This parameter
affects the algorithms that determine whether a given segment should be assigned to the trajectory; if set
to 2, this increases the chance that abrupt disturbance segments will be captured. The trick, however, is
making sure that disturbance always looks like disturbance: in woody systems, it can be defined fairly
well, but in herbaceous dominated systems, disturbance (especially from fire) can cause the spectral
response to move in the opposite direction from that expected for disturbance.
i. vertexcountovershoot= 3
This is another fairly esoteric parameter (like distweightfactor). In the fitting process, there are two
sequential algorithms that search for potential vertices between segments in the spectral trajectory. The
first algorithm is more conservative than the second one, and less sensitive to small "bumps" in the
trajectory. To ensure that all reasonable vertices are picked, we force the first algorithm to identify more
segments (more vertices) than we will eventually want, and then use the second algorithm to cull those
back to the desired count. The "vertexcountovershoot" value determine how many more vertices will be
picked in the first step (before vertex culling).
o. bestmodelproportion= 0.75
Once the vertices have been picked to with a number to find the "max_segments" count, different
algorithms sequentially remove the least informative vertex from the group and calculate overall fitting
statistics. Each segmented fit is considered a model to describe the spectral trajectory -- one model for a
single-segment description of the trajectory, one model for two-segment description of the trajectory, etc.
up to the maximum number of segments. Each model has an associated set of fitting statistics, with the
fitting statistics being penalized for higher-complexity models (i.e. those with more segments) through
degrees-of-freedom adjustments. At the end of this process, there exist (max_segments) number of fitting
models for the single trajectory. At this point, we can choose either the model with the best fitting score,
but experience has shown that the degrees-of-freedom penalty can sometimes force a legitimate,
slightly-more-complex model to be passed over in favor of a generalized, less-complex model. If the goal
of the segmentation is to capture abrupt phenomena against a backdrop of fairly noisy year to year
variation, we would instead want to allow a good model -- but not quite as good as the BEST model -- to
be chosen as the descriptor for this pixel. Setting the best model proportion to 1.0 forces the use of the
best model, but setting this value to a proportion less than 1 allows models with slightly lower fitting
scores to picked if they are more complex than the best-fitting model.
This parameter is quite important in determining the overall character of a run. It is often useful to do one
run with this parameter set to 1.0, and one run with it set to 0.75, and compare the final maps. It may
even be useful to attempt a run at an intermediate value, say 0.90. In all of these cases, it's often useful
to only conduct such a run on a small area where output maps can be judged for utility.
Run the batchfile
Once the parameters in section A have been set, you may run the batchfile from the IDL command
line.
However, do not run the batchfile until you determine whether you need to run the Fit-to-Vertices
(FTV) routines. (see section II below).
If you do NOT need to run the FTV routines, then you need to comment the sections of the
batchfile that correspond to the fitting. In the batchfile you will see indicators for where to start and
commenting. To comment out pieces of the batchfile, you will need use the semicolon ( ie. ;). You
can select large blocks and click the semicolon button the IDLDE interface to do this quickly.
Once you've determined whether you need to run the FTV, then run the batchfile.
IDL>@landtrendr_v0.43_pprr.pro
Be aware: If you are running this batchfile on a large area, with a skip-factor of 1, this batchfile
may run for a *long* time. Depending on the machine used, a segmentation run of an entire
Landsat image at a single-pixel skip-factor could take several days of constant operation.
As always, ensure that the batchfile has been saved in a directory/folder that IDL knows to look for
code.
When you run the batchfile, the programs first establish output files in the directories you have
indicated in the "outputs path" parameters.
TIP:
A common immediate error is that IDL cannot write to the outputs directory. Check to make sure
that you have a directory called "outputs" in the appropriate folder:
outputs_path = top_path + pathrow_name+ '\outputs\'+base_index+'\'
In other words, if you have a folder called "C:\4529\glovis\" with the images from SOP 2, and you
are doing a LandTrendr run using the normalized burn ratio (NBR), you will need to have a
separate folder set up called "C:\4529\outputs\nbr\" that IDL can place the output files in.
Once you start the batchfile, it will initially appear to lock up the IDL interface (in most cases) as the
first routines set up the large output files that will be populated by the algorithms. After a short
period, you should see a small popup status meter with the label "Processing chunks" and a
second one with "Performing fitting".
The way this program works is to divide a large image into a series of sequential smaller chunks
whose multiyear images can be read into the memory all at once. The large size of the image
stacks prohibits reading them into memory all at once. Thus, all processing is conducted on
chunks. You can monitor the overall status of the process by looking at the "processing chunks"
dialog, which indicates how many chunks the programs have decided to break the image into, and
which one it is working on.
What happens if it gets interrupted?
The programs save all of the necessary information at the end of each processed chunk, and keep
track of which chunk is being processed in a diagnosis file. If the program is interrupted for some
reason during a long run, you can simply start the batchfile over again (exactly the same way) and
it will pick up on the chunk it was processing, minimizing the wasted time.
How do I know when it's done?
When the entire run is done, the outputs window will show text indicating how long the whole
process ran. The two dialog boxes will be gone.
LandTrendr Segmentation Outputs
The segmentation run produces quite a few output files. They are described below.
All of these images are "flat binary" files with header text files in a format that really only has
meaning to the image routines used in the LandTrendr algorithms. Therefore, if you would like to
view these images in the ENVI software package, you will need to run the
"convert_bsq_headers_to_envi" procedure, described in SOP 2, section X above (but reproduced
here):
convert_bsq_headers_to_envi, outputs_path, <template_file.hdr>
In this example, outputs_path is the variable that was assigned in the batchfile for the
segmentation, and thus can be typed exactly as above. In the place of the <template_file.hdr>
field, use the full path name of a file with the map projection information of the images in the image
stack, thus:
convert_bsq_headers_to_envi, outputs_path,
"F:\4529\glovis\1988\lt05_045029_2002_234_to_1988_bsq.hdr"
The output files from the LandTrendr segmentation are:
*_diag.sav =
*_distrec.bsq = disturbance – recovery ratio
Band 1: largest single disturbance
Band 2: largest single recovery
Band 3: Scaled ratio between disturbance and recovery
-1000 is all recovery, 0 is a balance, 1000 is all disturbance
(-1500 = no change during time series)
*_durs.bsq = duration of each segment (in image years between vertex points)
Each band refers to the vertex point
Example: Band1: 24 === an undisturbed trend with no
intermediate points between start and end of image
series
Band1: 10, Band2: 10, Band 3: 4 ======a trend with 3
vertex points where there is a change in trend at image 10,
image 20, and image 24.
Band 1: Vertex 1
Band 2: Vertex 2
Band 3: Vertex 3
Band 4: Vertex 4
Band 5: Vertex 5
Band 6: Vertex 6
*_mags.bsq = magnitude of trend change
(ie. wetness is negative if disturbed or positive if growth)
Band 1: Difference between trend vertex 1 and 2
Band 2: Difference between trend vertex 2 and 3
Band 3: Difference between trend vertex 3 and 4
Band 4: Difference between trend vertex 4 and 5
Band 5: Difference between trend vertex 5 and 6
Band 6: Difference between trend vertex 6 and 7
*_stats.bsq = statistics for the entire fit
Band 1:P of f
Band 2:f_stat
Band 3:ms_regr
Band 4:ms_resid
Band 5:1=direct 2=interpolated
Band 6:n_segments
Band 7: first non-masked year for the pixel
Band 8: number of non-masked years
Band 9: The x offset for the pixel from which this pixel's values were taken if this pixel was interpolated.
Band10: The y offset for the pixel from which this pixel's values were taken if this pixel was interpolated.
Layer 7 is now filled with the first non-masked year for the pixel. If there are clouds in the first few years,
for example, the first year will be different from the earliest year of imagery.
Layer 8 is now filled with the number of non-masked years. If some areas of the landscape have
consistent cloud cover, this number may be significantly lower than the number of years in the whole
series, so this may help in evaluating the outputs
*_vertvals.bsq = trend vertex values
Band 1: Index value of first point in trend
Band 2: Index value of second point
Band 3: Index value of third point
Band 4: Index value of fourth point
Band 5: Index value of fifth point
Band 6: Index value of sixth point
Band 7: Index value of seventh point
*_vertyrs.bsq = important years describing the trend
(usually this is the year of disturbance or recovery);
There are 7 bands because of the parameters set to find 6 vertices plus the first year
(change this parameter to describe the trend with more or less vertices).
Band 1: First year of stack/trend line
Band 2: year of second point
Band 3: year of third point
Band 4: year of fourth point
Band 5: year of fifth point
Band 6: year of sixth point
Band 7: year of seventh point
*_segmean.bsq = the mean spectral value for the segment
Band 1: Mean spectral value for the years in the first segment
Band 2: Mean spectral value for the years in the second segment, if it exists.
etc.
*_segmse.bsq = segment mean-square error image
The mean-square error is simply the sum of each year's spectral value in a segment minus the mean
spectral value of the segment, squared. Segments with more variability will show much higher values in
this image.
Band 1: MSE of the first segment
Band 2: MSE of the second segment, if it exists.
Etc. up to Band 6.
*_source.bsq = source spectral data sent to the segmentation algorithm
*_fitted.bsq = the spectral data that comes out of the segmentation fitting
Band # = image year of stack
Band 1: the spectral value (source or fitted) for the first year in the stack
Band 2: the spectral value for the second year, etc.
There are as many layers to each of these images as there are years in your stack of input imagery.
II. FIT-TO-VERTICES (OPTIONAL)
The core LandTrendr segmentation used in section I above is applied to a single spectral index, which
can be sufficient to create maps of change on the landscape described in terms of that index alone (See
SOP #4). For some purposes, this may be inadequate, as the information content of any single spectral
index is rarely enough to fully describe changes on the landscape. If yearly landcover maps are desired
(and SOP 6 will be used), we must do more in preparation.
Therefore, another possible step in the LandTrendr processing flow is to apply the information gleaned
from the segmentation of the original spectral index to other spectral indices. In this step, the years
associated with all vertices identified by the segmentation algorithm in section I are used to constrain the
segmentation applied to another spectral index. The result is a file of fitted values for that other index that
agrees temporally with the fitted values of the original spectral index, but with fitting applied to the new
spectral index values.
This part of the batch file is optional if only change maps (SOP #4) are desired, but required if yearly
landcover maps based are desired (SOP #6)
Prepare the Fit-To-Vertices (FTV) component of the LT batch file.
The template batchfile above already includes all of the pieces needed to run the FTV, assuming that you
are running the core segmentation "upstream" of this. Therefore, no further preparation of the batchfile is
necessary.
Thus, the FTV runs will be a part of the batchfile run for section I above, assuming that you do not
comment them out.
Outputs from FTV
FTV also produces many files. These are analogous to those from the core segmentation run, and are
placed in the same directory, but include the index and the word "ftv" in the filename, e.g.
LT042_nbr_4529_greenness_ftv_source.bsq is the image with the stacked values of the tasseled-cap
greenness data used as the source for the fitting algorithms.
Fit to Vertices (FTV) outputs:
FTV_*_diag.sav
FTV_*_fitted.bsq = fitted values of this index based on the original Landtrendr.
Band # = image year of stack
FTV_*_stats.bsq = statistics for the entire fit
Band 1:P of f
Band 2:f_stat
Band 3:ms_regr
Band 4:ms_resid
Band 5:1=direct 2=interpolated
Band 6:n_segments
FTV_*_vertvals.bsq = the fitted index values of the input band at those years
Band 1:
Band 2:
Band 3:
Band 4:
Band 5:
Band 6:
Band 7:
Band 8:
*_darkseg.bsq
This is a six-layer image that captures for each pixel the mean (bands 1-3) and standard
deviation (bands 4-6) value of the tasseled-cap bands (Brightness, Greenness, and
Wetness) of the segment that appears to be most forested. Displaying bands 1,2 and 3
in RGB should produce a pretty standard tasseled-cap image, but one that captures each
pixel in its least disturbed state. This can be used to create a forest mask, for example,
that will be minimally affected by disturbances that happen sometime during the period of
the stack, since the forest condition before the disturbance will be captured in the
darkseg image. Bands 4-6 are used to identify pixels that are noisy during their most
forest-like period, which can be used to separate agricultural lands (with a lot of noise,
even if they happen to look forested spectrally in one period) from stable forest lands.
III. EVALUATE RESULTS
It is useful to check on the outputs from any LandTrendr run to make sure that things worked
appropriately.
The best set of images to evaluate the are the "fitted" and "source" images, as these capture all of the
important phenomena.
Compare source and fitted files
1. Open source file and fitted file in different viewers in ENVI.
a. Each layer in the image corresponds to a different year in the image stack.
b. Display each as a 3-band RGB composite, with an early, middle, and end year image for
R, G, and B respectively.
*hint: edit the 'wavelength' attribute in the ENVI header file to reflect the year and julian date of the image
processed through LandTrendr, such as the '*_source' and '*_fitted' files where each band represents an
image year. This will improve the graphing results so the x-axis is actually the date value in the form of
year.julian_day instead of the defaut 'Band Number'.
2. Link the viewers together
3. Compare trajectories for all years
a. When three layers from different years are displayed together, areas that have not
change will be approximately the same value across years, and thus should appear as
some shade of black, grey, or white, with white areas being those that are consistently
high in the spectral index of interest and black areas those that are consistently low.
b. Any time you see patches of a non-grey-scale color, something is happening there.
Center the viewer on those locations and examine the profiles.
i. Right-click in each viewer and select "Z-profile" from the pulldown menu.
ii. Overlay the 'source' and the LandTrendr 'fitted' Z-profile by dragging the Z-profile
symbol of the 'fitted' file to the Z-profile graph of the 'source' file.
a. Evaluate how well the LandTrendr algrorithm captured the overall trend of the source
Landsat data.
i. Some things to look for:
1. Do clouds or shadows cause a false positive, in terms of a vertex?
2. Is a known disturbance missed?
3. Are there any data anomolies possibly due to poor radiometric
calibration?
Protocol for LandTrendr
Standard Operating Procedure (SOP) #4
Developing LandTrendr Maps of Change
Version 0.01 (October 1, 2009)
Revision history log:
Previous
Version
Revision Date Author Changes Made
New
April 12, 2009 REK
New
0.01
September
25
REK
populated
0.01
2 Sep, 2010
PN
revised & clarified
0.01
1 Mar, 2011
PN
added error matrix
description
0.01
Table of contents
Overview
Reason for
Change
New Version
#
I. Setting up batchfiles
II. Running the batchfile
III. Outputs
OVERVIEW
Figure 1. SOP 4 within the LandTrendr process.
Although the algorithms applied in SOP #3 are the core of the LandTrendr process, they result in images
that are too rich in change information to be viewed succinctly. Therefore, we use other algorithms to slice
through the information content of those outputs to create a variety of simpler (and more readily
interpretable) maps of change. The algorithms described in this SOP (#4) create those maps.
As with the prior SOPs, the maps are produced using a batchfile. There are three basic types of change
map output, each with a different section in the batchfile. All of the types can be run by running the
batchfile, or some subset of the three types can be commented out and therefore not run.
The three types of change map are disturbance patches, segment-based change labels, and
disturbance/recovery snapshots.
The disturbance maps are highly filtered maps that focus on disturbance alone. The algorithms used to
make these maps require that the user create a function that converts the spectral index of interest into a
physical measure of percent cover that is used to evaluate whether a given disturbance event passes a
minimum change threshold. After identifying all pixels that meet this threshold, groups of pixels
experiencing disturbance in the same year are grouped together, and groups smaller than a minimum
mapping unit are eliminated. Finally, the groups of pixels in a single patch are grouped and labeled with a
patch ID and written to several files that have a separate layer for each year in the stack, with pixels in
each layer for each patch assigned the magnitude or duration of the disturbance event beginning in that
patch in that year. Because each layer is separate, each layer can be easily converted into a polygonbased shapefile format that allows for quick mapping of the full disturbance record.
Segment-based change label maps allow capture of a full range of disturbance and recovery patterns. A
set of algorithms searches through the LT segmentation outputs to identify interesting segment types or
sequences of segment types, and labels each interesting situation with a class-label. The definition of
what is "interesting" and the change labels are set by the user using a string-based change labeling
syntax defined in the batchfile. The result is a single-layer classified image, where each pixel is labeled
according to the type of change trajectory occurring in that pixel. Additionally, each class has an
associated multi-layer image that captures the year of onset, the duration, and the magnitude of the
segments that define that type. Taken together, these maps can be used to both quickly visualize the
patterns of change across a landscape, and to isolate the behavior of particular change types of interest.
Disturbance/recovery snapshot maps provide the user with a view of the direction of change (either
gaining or losing vegetative cover) for each pixel on the landscape for each year in the stack. This allows
quick assessment or reporting of overall landscape condition for a particular year of interest, or a means
of "watching the evolution" of the landscape over time by stepping through the snapshots sequentially.
I. SETTING UP BATCHFILES
We use a single batchfile to produce all three types of change map. The parameters needed for each
change algorithm are defined below. First, load the template batchfile named "lt042_changemaps.pro"
and save it to your active batchfile directory, including the path/row of the image being mapped in the
filename.
Set up batchfile parameters as follows:
1. Generic file information:
a. The following parameters should be copied directly from the lt042_ segmentation
batchfile that created the segmentation layers that you'll be using for the maps here.
Making these parameters the same ensures that the mapping algorithms are
referenceing the correct files:
i. pathrow name
ii. top_path
iii. Base_index
iv. run_name
v. mask_image
vi. images_path
vii. outputs_path
viii. ltversion
ix. base_lt_output_names
x. image_info_savefile
xi. template_file_for_map_projections
2. Disturbance patch parameters
a. Static_model_function :
i. This is a string that points to an IDL function that calculated percent cover from
the spectral index used to make the segmentation runs from SOP 3.
1. Example: 'static_nbr_model_pilot_scenes'
ii. The function that this points to is a file, with filename the same as the string
parameter + the '.pro' extension, e.g. 'static_nbr_model_pilot_scenes.pro'
1. The 'static_nbr_model_pilot_scenes.pro' file is provided for guidance.
2. The file takes as input a single value of the spectral index, e.g. nbr or
greenness, and returns a single value of the output cover value.
b. change_model_function
i. As with the static model, except that the function captured in this model should
accept the change value as input, and reports the change in cover as output.
ii. Note that both models must ingest the fitted value from the LandTrendr
segmentation directly, and scale the outputs accordingly. If the "divisor"
parameter in the original segmentation is set to a value other than 1 (see SOP 3,
Section IA), then the fitted outputs will be scaled relative to the original spectral
index.
iii. IMPORTANT: If you specify "none" for the change_model_function, then the
mapping algorithm will infer the change in cover by applying the static percent
cover model to the fitted values of your spectral index before and after the
disturbance and subtracting.
c. year_range
i. A bracketed two-element vector, with the first element corresponding to the first
year in the image stack and the second element the last year in the stack. This
is used to assign the correct year to the disturbance patches, which can then be
imported as an attribute in polygons derived from the disturbance stack outputs.
ii. Example: year_range = [1985, 2008]
d. collapse_dist_angle (collapse disturbance angle)
i. Default: 15
ii. This number determines when the mapping algorithm will combine sequential
segments of disturbance. It is defined in degrees, but these are based on a
relative data space defined as follows: the full spectral trajectory is scaled in both
the X (years) and Y (spectral range) dimensions, and the angle between each
succeeding segment in that space is calculated.
iii. If the angle between two adjacent segments of the disturbance type is less than
the "collapse_dist_angle" threshold parameter, then the two segments are
combined for further analysis.
iv. Practically speaking, this will primarily affect the duration of a disturbance event,
since it will string together sequential disturbance segments if they are roughly
tracking the same direction, rather than considering each one separately, making
one long disturbance event. In some cases, it can also falsely move the onset of
a real disturbance, if a chance dip before the actual disturbance is captured
under this threshold, which would cause the actual disturbance to be merged
with the pre-disturbance "blip."
e. collapse_rec_angle (collapse recovery angle)
i. default: 15
ii. Defined the same as the collapse_dist_angle.
f. pct_loss_threshold_at_one_year, pct_loss_threshold_at_20_years
i. These two parameters are considered together. They define the percent cover
value below which a given disturbance segment is considered no-change. We
use two values: one defines the threshold to be used for abrupt disturbances,
and one for long-duration disturbances. We typically use a lower threshold for
the long-duration disturbances, since segments that persist for many years
typically have a much greater signal-to-noise ratio than those that are simply one
year. For years in between 1 year and 20 years, the threshold is determined by
simple interpolation. thus, if the values were 10 and 5 for 1 and 20 years,
respectively, then the threshold at 10 years would be approximately halfway
between the two, or roughly 7.5.
ii. Default values: 10 and 5 (always defined as positive values)
iii. Note: The percent cover values used in this thresholding step are determined
using the percent cover value calculated by the user-defined models described
above.
g. pre_disturbance_cover_thresh
i. If the spectral value before a disturbance is translated through the percent cover model
into a value that is lower than this parameter threshold, then the disturbance segment will
be considered "no-change." This allows for removal of disturbance "noise" that occurs in
non-vegetated land cover types, and thus is only appropriate if the disturbance being
mapped is focused on fairly highly vegetated systems.
h. pct_gain_threshold_for_recovery
i. This parameter is the analog to the pct_loss_thresholds for disturbance, but
applied to the gain in cover under recovery. Recovery segments that are less
than this value (always defined as a positive number) are considered no-change.
i. mmu_pixels
i. This defines the minimum number of pixels to be retained in a patch of
disturbance.
ii. Default: 11
iii.
j. dur_threshold
i. This parameter helps the algorithm separate long- from short-duration
disturbance events that happen adjacent to each other in the same year.
Although statistically an unusual event, it does occur, and when it does, this
parameter is used to separate pixels that are likely in the short-duration category
from those that are in the long-duration category.
ii. Default: 10 (years)
1. Segment-based change label parameters
a. diag_file
i. This is simply the pathname of the diagnosis file from the segmentation run in
SOP 3. Typically, it can be left at the default value of:
1. base_LT_output_name + '_diag.sav'
b. disturbance_proportion, recovery_proportion
i. These values range from 0 to 1.0, and define a threshold that is used to eliminate
small segments that precede a larger segment of the same type. For each
trajectory, the largest disturbance or recovery segment is identified and the
magnitude noted; a segment before that high value that is less than a proportion
of that magnitude will be considered no change. That proportion is set here.
ii. For example, if a small disturbance segment with value 35 precedes (either
directly before, or any time before, even if separated by a recovery segment) a
large disturbance segment with value 700, that small segment is 5% of the large
disturbance value. If the disturbance_proportion parameter is set to 0.10, then
that smaller segment would be eliminated from consideration in the labeling
exercise.
c. Modifier
i. As in other cases, this value points in the direction of disturbance, and is either 1
or -1. For indices that become more positive when vegetative cover increases,
such as the NDVI, wetness, or the NBR, the modifier should be se to -1. For
indices such as band 5 or brightness, that become more positive when
vegetative cover decreases (at least in most cases!), this number should be set
to 1.
d. Label_run_name
i. A string that will uniquely define the outputs from this run. This string will be
used to create a directory in the outputs directory that is parallel to the spectral
index directory of the original segmentation outputs, but this one will have the
label_run_name as part of the folder name.
ii. You can test different labeling rules based on the same core segmentation run by
simply changing the label_run_name, since each time you change the name, an
entirely new subdirectory with its own outputs will be created.
e. Class_codes
i. This is the most important setting in the change labeling module. It defines the
labels of the change classes. Each change class is defined with a number, a
name, and change-code-syntax, separated by the "#" sign. There can be as
many change classes as desired, each one being defined with a single string in
an string array with as many elements as there are classes. The following is the
generic change labeling class structure:
class_codes = ['3#moderate_fast_disturbance#GD0075XX0000L04', $
'4#big_fast_dist#GD0300XX0000L03', $
'5#slow_disturbance#FD0050XX0000G06', $
'6#long_slow_disturbance#FD0025XX0000G18', $
'7#growth#FR0050XX0000G06', $
'8#long_growth#FR0050XX0000G18', $
'9#recent_slow_dist#FD0050GE1996G06', $
'10#rec_then_fast_dist#FR0050XX0000G06#FD0150XX0000L03', $
'11#rec_then_mod_dist#FR0050XX0000G06#FD0100XX0000G03', $
'12#rec_then_slow_dist#FR0050XX0000G06#FD0050XX0000G06', $
'13#fast_dist_then_mod_dist#FD0150XX0000L03#FD0075XX0000G02', $
'14#fast_dist_then_slow_dist#FD0150XX0000L03#FD0050XX0000G06',
$ '15#fast_dist_then_rec#FD0150XX0000L03#FR0050XX0000G04', $
'16#slow_then_fast_dist#FD0050XX0000G06#FD0150XX0000L03', $
'17#slow_then_fast_dist_then_rec#FD0050XX0000G06#FD0150XX0000L03#FR0050XX0000G04', $
'18#slow_then_recovery#FD0050XX0000G06#FR0050XX0000G02']
i. The change-code syntax is as follows:
1. "<G/F><D/R><0000><EQ/GE/LE/XX><0000><G/L><00>'
2. The first field is either G or F, signifying either "greatest" or "first". If set to
G, then the greatest magnitude segment of the type specified in the
second field (D for disturbance or R for recovery) will be identified. If set
to F, then the first segment of that type will be identified. Once that
segment is identified, it is evaluated relative to three rules corresponding
to the segment's magnitude, year of onset, and duration.
3. The first rule is a four-digit field specifying the magnitude of the segment;
if the segment is less than this threshold (always defined as a positive
number for either recovery or disturbance), it will be passed by.
4. The second rule is two-character plus four digit rule that specifies
whether the segment must begin on, before, or after a particular year
(EQ/LE/GE). If set to XX, then segments beginning any year are
allowed.
5. The third rule is a 1-character and two-digit rule that specifies whether
the segment must have a duration greater than (G) or less-then-or-equalto (L) the two-digit year threshold.
ii. Each segment in the trajectory is evaluated according the rules defined by the
syntax.
1. If a segment meets only some of the rules, it is passed by and the next
segment in the trajectory is evaluated for the rules.
2. If a change class is defined using two change-code rulesets strung
together (separated by the "#" sign), then the rules are evaluated in
sequence, with the rules for the second portion of the ruleset only being
evaluated once a segment that meets the first ruleset is found.
3. If a pixel's trajectory has a segment or sequence of segments that meets
the rule(s) in the ruleset(s), that pixel is assigned to the change class
whose rules it meets.
4. At the end of the evaluation process, there will be a single file with the
change labels for each pixel. Additionally, each change class will
produce a file that will store the details of the information about the
segments that met the criterion of that class: the segment start,
magnitude, and duration.
a. E.g. If there are 16 change classes defined in the change code
array, then there will be 16 output files with filenames that
include the class name string in the change code array.
b. If a change class has a ruleset that involves two successive
segments, then output image will have six layers, not three,
since each segment in the ruleset has the start, magnitude, and
duration values. If three segments make up the ruleset, then the
output image will have 9 layers, etc.
5. Redundant classes: Classes are evaluated in sequence from beginning
to end in the change-label array. If a given pixel meets the rules of a
change class early in the array of change classes, and then meets the
rules of change class further down in the array, it will be assigned the
change label of the latter class. Thus, it is imperative that the change
labels be arranged in order of increasing complexity from beginning to
end, so that classes are only overwritten for a given pixel if the
overwriting class is more informative than the simpler class.
a. Note: the 3-layer image (or 6 or 9, etc. depending on the
number of segments in the ruleset) associated with the simpler
class will not be affected if the pixel also meets the ruleset of a
later class. In this way, output images for both simple and
complex phenomena are created separately.
4. Disturbance/recovery snapshot parameters
a. The user-defined parameters for this mapping algorithm are minimal:
i. modifier:
1. As above, this points in the direction of disturbance, with -1 for spectral
indices that increase when vegetative cover increases, and positive 1 for
those that become more positive when vegetative cover decreases.
ii. fitted_image_file:
1. The full path and filename of the fitted image output file from the
segmentation run.
b. This mapping routine simply takes the fitted image, and subtracts each layer
(corresponding to a single year) from the year ahead of it
II. RUNNING THE BATCHFILE
As with the prior batchfiles, this batchfile is run with a simple command at the IDL command line, such as:
IDL>@lt04_changemaps
Note that the actual command will depend on the filename you saved the batchfile under.
When the batchfile runs, there will be little feedback. The disturbance patch analysis routine takes less
than an hour, usually, even for large images. The segment-based change labeling may take a while,
since each change class is evaluated individually. Thus, the more change classes defined, the longer the
run will take. A landsat scene may take the better part of a day if there are roughly 15 to 20 change
classes. The disturbance/recovery snapshot image will move quickly relative to the other two.
III. OUTPUTS
The three different types of map output are described separately.
1. Disturbance patch maps
a. There are two phases to this mapping process, each with several output files.
i. The first phase is the creation of a yearly stack of magnitude and duration values
for disturbance and recovery, filtered by the cover thresholds defined by the user
1. *_drstack_distrec_duration.bsq
a. An image with as many layers as there are in the original fitted
stack, one for each year. The value in each pixel is the duration
of the event that started in the year of that layer. If the value is 0
for a given pixel in a given layer, then there was no disturbance
event in that pixel for that year.
2. *_drstack_distrec_mag.bsq
a. As with duration, but for the magnitude of the event. Note that
both disturbance and recovery events are captured here, with
disturbance events having a negative value and recovery events
having a positive value.
3. *_drstack_distrec_preval.bsq
a. As with the prior two, but for the percent cover of the spectral
index at the onset of the event.
ii. The second phase is the creation of the patch-images, where the disturbance
events are isolated, then filtered by the minimum mapping unit (MMU) and patchaverages computed for those pixels.
1. *drstack_patch_duration.bsq
a. an image with as many layers as there are in the original fitted
stack, one for each year. Each pixel is assigned the mean value
of the duration for all of the pixels in the patch to which it
belongs.
2. *drstack_patch_mag.bsq
a. As with the duration image, but for the magnitude of the pixels in
the patch
3. *drstack_patch_patchid.bsq
a. A yearly stack image like the prior two, but here the value
recorded in each pixel is a unique id for the patch to which that
pixel belongs.
4. *drstack_patch_patch_summary.txt
a. This is a comma-delimited file that reports the year, number of
pixels, number of perimeter pixels, the mean cover value before
the disturbance, the mean magnitude of the disturbance within
the patch, the coefficicent of variation of magnitude within the
patch, the mean duration of pixels within the patch, and area
multiplied by the magnitude for the patch. The patch ID is also
reported, and is linked with the patchid in the _patchid.bsq file
2. Change_label maps
a. As noted above, there are two map types produced by the change label algorithms.
i. The change label map
1. *_LTlabel.bsq
a. This is classified image with numeric values that correspond to
the numbers assigned by the user in the batchfile.
2. *_<change class name>bsq
a. This image has three layers for each segment component in the
ruleset defined for this class. If the class is defined by meeting
just one segment rule, then there are three layers:
i. layer 1: onset of the segment that meets the rule
ii. layer 2: magnitude of the segment that meets the rule
iii. layer 3: duration of the segment that meets the rule.
b. if there are two or more segment rules, then the output image will
have that multiple of three layers, each grouping of three layers
corresponding to the layers described for the single-segment
type above.
ii. Also, there will be a change label spreadsheet in text form, comma-delimited
format, that describes the change labels associated with each class number, in
case the batchfile that originally described those classes is separated from the
output images.
1. *_LTlabel_classnames.csv
3. Disturbance/recovery snapshots
a. Two files are created for each run of the snapshot, one *_disturbance and one
*_recovery.bsq.
i. Each file has as many layers as there are years in the stack. The first layer (the
first year) is always empty, since the layers are defined as the difference
between the current year and the prior year, so it is not defined for the first year
in the stack. The blank layer is just retained to maintain consistency of layer
count among all outputs.
ii. The value in each pixel in each layer is the magnitude of change in the fitted
outputs from SOP 3 for the year in question. Both recovery and disturbance
values in the two respective files are defined as positive values for easier display
in most mapping packages.
IV. ERROR MATRIX
Step 3: Extract Landtrendr Disturbance map for validation plots
In this step, values from landtrendr map will be extracted to compare with TimeSync interpretation. You
will use the R language to link the plot locations in the .csv file to the disturbance map.
Modify data extraction script.
o Run R
This should be in C:\Program Files\R\R-2.12.0\bin\i386\Rgui.exe
o Once R is open, go to File\Open Script and navigate to the file for LT_READ (see
table above for full path/file name)
o Modify Parameter section in the script to point to your file locations. You will need to
change the “Work dir” so it points to you group directory. Otherwise, the rest should be
left the same.
Run script
o On Script Editor, EditRun All
o The script extracts from the greatest_fast_disturbance.bsq the year of disturbance,
using a 3 by 3 window to extract pixel values. Thus, there are 9 separate values for each
TimeSync plot location.
The script also extracts the magnitude of disturbance and the duration for those 9 pixels.
Examine output
o Output is a csv file: ts_lt_disturbance.csv. It ends up in the “Work dir” you can find in
the script file you just ran.
o Check ou the Landtrendr disturbance Year of Disturbance (YOD), Magnitude (MAG),
and duration (DUR) are extracted for each plot in the PLOT_FILE.
o In addition to the 9 pixels values (named as yod1,…yod9, mag1…mag9, dur1…dur9),
Majority for YOD (names as yod) and DUR (named as dur), mean value for MAG (named
as mag) are also extracted.
Step 4: Generate map accuracy table
Open ACCURACY_DB
o Double click on the ACCURACY_DB file (see Data Input and Output table above).
This will open the main MS Access interface. Next, you’ll open a separate MS Access
window type called the “Visual Basica for Application” window. Through this step, you’ll
need to alternate back and forth between these two windows to process things.
o We are going to use VBA in the database, which is diabled by default. Click on Enable
Content.
Import TimeSync segments from TimeSync Database
o Double click CREATE_SUMMARY under Modules (on the left, see Figure 4). This will
open a new window called the Microsoft Visual Basic for Application window.
Figure 4. Double click on the Create Summary link.
o In Microsoft Visual Basic for Applications window (Figure 5), you’ll see a window with a
script called “tsa_4627_accuracy – CREATE SUMMARY (Code)”.
o Within that text script, find “Sub ImportTsSummary()”. You’ll need to scroll down to the
bottom of the file. Change the string ts_db to match your TimeSync database location –
this is listed in the first line of the table at the beginning of this exercise. You will need to
change the path.
Figure 5. The Visual Basic for Applications window.
o With cursor inside ImportTsSummary(), click run icon on the toolbar, this will create a
new table “ts_summary” under Tables.
o Note: this step could also have been done using External DataAccess import option.
Import image list file: The TimeSync summary in step 2 reports the beginning vertex of a
segment, s, while Landtrendr disturbance maps are based on when disturbance is first detectable. By
definition, the beginning of a disturbance segment is before the disturbance occurred, so we cannot use
that. A disturbance is detectable only on the next valid year after a starting vertex. Thus, we need to
adjust the TimeSync segment summary based on available images. We’ll run a script to do that, and that
script needs to know about the image dates that are available.
o In Access main window, click External Data Text File
o Select TC_IMG_LIST for import
This file is in the Data Input and Output table at the beginning of the
exercise.
In the import window queries, accept all of the default options.
When you get to the last window, it asks for the name of the table to import
to. Type in “img_list” (please note the name has to match for later script to
recognize it).
o Run AdjustTsSegment in vba window.
Import landtrendr disturbance file from step 3.
o External Data Text File
o Select TC_IMG_LIST (see the table at the beginning of the exercise) for import
o Again, in the file import wizard accept the defaults. At the end, when asked for the
name of the table, name it “lt_disturbance” (please note the name has to match for later
script to recognize it).
Run summary script to create a confusion matrix
o Double click CREATE_SUMMARY module.
o If ts_lt_majority_fn, ts_lt_majority_fn, ts_lt_majority_fp, ts_lt_majority_match, and
ts_lt_majority_error have not been created, run CreateSummaryView.
Query
lt_summary
ts_lt_majority_match
ts_lt_majority_fn
ts_lt_majority_fp
ts_lt_majority_error
ts_lt_majority_error2
Description
Landtrendr disturbance using majority rule with magnitude
assigned. High > 66, medium 33-66, low <33
Matched disturbance
List of false negative disturbance
List of false positive disturbance
Basic error matrix
Error matrix with magnitude group
Step 5. Examine error matrix
The result from ts_lt_majority_error can be reformatted as
TimeSync
Disturbance
No Disturbance
Landtrendr
Disturbance
27
3
No Disturbance
57*
**
* disturbance from TimeSync is based on a 3x3 plot window interpretation, while the disturbance from the
lantrendr map has minimum mapping unit of 11 pixels.
** The number in this cell should be interpreted with caution. It does not correspond to the number of no
disturbance segment agreement between TimeSync and landtrendr.
Step 6. Examine error matrix by disturbance magnitude
Query ts_lt_majority_error2 shows the disturbance matrix with magnitude information.
Landtrendr
High
Medium
Low
ND
High
5
9
0
9
Medium
1
7
2
13
TimeSync
Low
0
0
3
35
ND
0
2
1
Protocol for LandTrendr
Standard Operating Procedure (SOP) #5
Utilizing TimeSync for validation
Version 0.01 (October 1, 2009)
Revision history log:
Previous Version Revision Date
Author Changes Made
New
April 12, 2009
REK
New
April 29, 2009
PN
Populated
Reason for Change New Version #
0.01
September 11, 2009 YANG
Modification
0.01
2 Sep, 2010
update & revision
0.01
PN
1. Overview
2. Method 1
3. I. Software Used
a. A. TimeSync
b. B. ArcGIS 9.3
c. C. Microsoft Access
4. II. Required Input Imagery From Prior SOPs
5. III. TimeSync Workflow
a. A. Devise Plot List
b. B. Create imagery list
c. C. Set-up TimeSync Database
d. D. Using the Validation Database/TimeSync
e. E. Summarizing TimeSync Plot Results
f. F. Determining Map Accuracy using TimeSync
Overview
An overview of the LandTrendr processing work flow involves many steps. First, Landsat images for each
year between 1984 to present, primarily during July or August time period, is obtained from the Landsat
data archive (SOP 1). Second, COST atmospheric correction, MADCal radiometric normalization, and
cloud masking is performed on the images in the yearly stack (SOP 2). Third, the image stack is analyzed
using the LandTrendr process (SOP 3) which summarizes and labels the Landsat observed long-term
trends from year to year. Next, the LandTrendr output results are evaluated for their accuracy using
independent data visualization using TimeSync (SOP 5). Finally, the analysis and final products are
documented and archived (SOP 8).
Figure 1. SOP 5 within the LandTrendr process.
The tool (TimeSync) uses geographic coordinates for an area of interest to extract image chips from a
time series stack for that area and its neighborhood. TimeSync displays a times series of chips for easy
viewing, along with a spectral plot of raw bands and indices over the time series for the area of interest.
Using these two data visualization windows, an analyst identifies changes that have occurred, if any,
within the area of interest and uses a series of pick-lists associated with a relational database to label
segments in the spectral profiles that are associated with cover changes in the area of interest. The
process involves selection of time-series vertices that identify dates associated with start and end points
of change segments, and the segments are labeled according to the cause of the observed change. The
tool is linked to Google Earth which displays a recent high resolution image for detailed spatial reference
(commonly a georeferenced aerial photograph).
TimeSync can also be used to validate disturbance polygons generated by LandTrendr. Colleagues at
the Great Lakes I&M Network (GLKN) have developed an alternative technique to validation. This
technique also incorporates TimeSync, however, it is utilized differently. This will be described as an
alternative and can be used depending on the users' goals. The main difference between this technique
and the previous technique is that GLKN validates every polygon produced by LandTrendr, not a random
sample of the landscape of interest.
Method 1
Maybe we should describe method 1 and it's goals and applications here. Basically tell people why they
would want to use one method over the other.
I. Software Used
TimeSync uses a combination of Access database and Python scripting code. You must have Microsoft
Access installed on your computer.
A. TimeSync
a. Setting up TimeSync Program
i. Extract TimeSync.zip into C:\TimeSync\
ii. Copy TimeSync shortcut from within TimeSync directory to desktop
B. ArcGIS 9.3
a. 3d analyst extension
i. To convert plot locations into Google Earth
b. Hawth’s Tools add-on
i. Used to generate points, either within LandTrendr polygons or randomly on the
landscape
ii. Freely available at http://www.spatialecology.com/index.php
C. Microsoft Access
1. Access 2003 or 2007
II. Required Input Imagery From Prior SOPs
TimeSync requires the user create a text file containing a list of radiometrically calibrated Landsat
reflectance imagery and Tasseled Cap imagery created in SOP 3.
III. TimeSync Workflow
Figure 1: TimeSync work flow. 1) Devise plot list. 2) Create reflectance and tasseled cap imagery
list. 3) Set-up TimeSync database referencing the plot list and imagery lists. 4) Within TimeSync
Program, analyst determines segments of land cover changes and recovery for each plot using a
combination of Landsat image chips, spectral trajectory, and high resolution imagery in Google Earth. 5)
Summarize plot segments table. 6) Compare TimeSync plot results to the Trajectory-Based Change
Detection map results using a confusion matrix.
A. Devise Plot List
TimeSync requires a list of plots which is loaded into the Access database table: Plots.
choose between random plot points (a) or plots based on change-maps from LandTrendr (b)
a. Random plot points
i. describe why
a. Plots based on change-maps from LandTrendr
i. If the user would like to validate only change polygons produced by LandTrendr
then this is the methodology to follow
1. Create plot point file
a. Create shapefiles with python script
i. use "landtrendr_to_shapefile_pprr".py
b. Reformat output txt file to csv with python script
i. use "reformat_patch_summary_pprr".py
a. Join shapefile to patch attributes
i. base join on patchid
b. Export joined shapefile
i. delete some fields
1. ID, GRIDCODE, yrs, objectid, patchid_1, index,
F12
ii. output name: lt0423_nbr_pprr_joined.shp
c. Clip the joined, modified shapefile to the largest analysis extent
i. output name: lt0423_nbr_pprr_joined_clipped.shp
d. Dissolve the clipped polygon by patchid
i. after clipping, polygons on the edge can become
multiple polygons, which is not what we want to happen.
To fix this, you can use the dissolve tool in Arc to fix
these errors.
1. open dissolve tool
a. input feature class:
lt0423_nbr_pprr_joined_clipped.shp
b. output feature class:
lt0423_nbr_pprr_joined_clipped_dissolv
ed.shp
c. dissolve field: patchid
d. create multipart features: yes
e. Create excel file with column identifying the number of points per
polygon
i. in ArcMap view attribute table of joined, clipped polygon
shapefile, export data to csv file
1. output name: step01_pts_per_polygon.csv
ii. open csv file, delete all fields except patchid, year and
num_pixels
iii. save csv file to excel file
1. output name: step01_pts_per_polygon.xls
iv. delete previous csv file
f.
g.
h.
i.
a.
a.
b.
v. open excel file and create new column (pts) that will
inform Hawth's Tools how many points per polygon to
create
1. create 1 point per 6 pixels and use the
"roundup" function in excel to roundup to the
nearest integer
a. because this column cannot have a
formula, copy and "paste special" >
values
b. delete old column with formulas
2. save and close excel file
Join excel file with shapefile, base the join upon patchid
Export the newly joined shapefile
i. output name: stepXX_shapefile_for_hawths.shp
Bring stepXX_shapefile_for_hawths.shp into ArcMap
Create pre-determined number of points in each polygon using
Hawths tools
i. open hawths tools > sampling tools > generate random
points
1. select layer: stepXX_shapefile_for_hawths.shp
2. minimum distance between points: 40 meters
3. sample size: stratified sampling design
a. polygon unique id field: patchid
b. generate number of points in field
i. choose "pts" field
4. output name: step03_points_for_timesync.shp
Add XY coordinates to shapefile
i. start "add XY coordinates" tool in Arc
ii. input features: step03_points_for_timesync.shp
export attribute table to csv
i. output name: step04_points_for_timesync.csv
open and save this csv file to an excel file
i. output name: step04_points_for_timesync.xls
ii. this excel file should have four columns (PLOTID, X, Y,
isprocessed)
Figure XX. Excel file with points ready to import into the Access database for TimeSync
1. Create Interpretation plots: Viewing each plot using high resolution imagery (usually aerial
photography), it is necessary to first create a 810 m2 (90m x 90m: 3 pixels x 3 pixels) plot based
on the plot center coordinate and then export these plots into the Google Earth format (*.kml).
a. Using ArcGIS, open the plot point shapefile.
b. Using the ‘Hawth’s Tools’ ArcGIS extension to create square 90m x 90m plot around the
plot point. ArcMap->Hawth’s Tools->Sampling Tools->Create Sample Plots
Figure 2. Shows a screen shot of creating a square buffer. *note this will create a shapefile with the
projection of the point layer. Also, the dimension is length from plot center to edge.
c.
TimeSync 3x3 window for GoogleEarth (kml). Use the ArcGIS 3D Analyst tool> Layer to KML to
convert the 90m x90m square polygon file to a GoogleEarth kml file.
Figure 3. Convert plot shapefile to *.kml format for viewing in Google Earth.
B. Create imagery list
1. In IDL use the script ts_image_list_maker.pro
a. alter 'path'
b.
c.
d.
e.
f.
alter 'image_info_savefile'
alter 'outpath'
alter 'refl_out_name'
alter 'tc_out_name'
save as 'ts_image_list_maker_<pprr>.pro' where <pprr> equals the path row you are
working on.
g. IDL> @ts_image_list_maker_<pprr>.pro
In the format of : imgtype,imgyear,imgday,id,file
where imgtype = image type (TM or ETM)
imgyear = image year (yyyy)
imgday = image day (yyyy-ddd) ddd= julian day
id = directory path to images
file = name of madcal reflectance or Tasseled Cap file
2. Tasseled Cap List
Figure 4. Tasseled Cap imagery list example.
3. Reflectance List
Figure 5. Reflectance imagery list example.
C. Set-up TimeSync Database
1. Create database
a. The TimeSync Database (TimeSync_template_pprr.mdb) is an Access database which
references the plot list and imagery lists while recording the analysts decision regarding
any land cover changes noted in the TimeSync visualization (step 4).
b. Open ‘TimeSync_template_pprr.mdb’, save a copy with a new descriptive name, such as
‘TimeSync_SEKI_change1984to2008.mdb’.
Figure 6. Save template database as new file.
2. Populate Plot list
a. Populate the Table 'Plots' with known plot locations and a unique identification number
(plotid).
Figure 7. 'Table: plots' contains the plot center coordinate.
3.
Define Image Settings
a. You will need to modify imageSettings table to reference your imagery list.
b. Table: imageSettings: this is where the image lists reside
i. field: bgw_image_list = enter path to Tasseled Cap imagery list created in
Section II.
ii. field: refl_image_list = enter path to reflectance imagery list created in Section
II.
iii. field: scalar = take default value (1000). This value is the same as the value
used to scale raw reflectance values (0-1) and Tasseled Cap values.
Figure 8. ImageSettings Table provides the location of the Tasseled Cap (field: bgw_image_list) and
reflectance (refl_image_list) image lists created in step 2. A scaling value for the imagery is set to insure
proper viewing of the images in TimeSync.
Tip: if your TimeSync chips in the viewer are very dark or not providing the correct Tasseled Cap
colors, the you might need to adjust the scaling value.
4. Set Spectral Properties (if necessary)
a. The spectral properties table holds the global statistics for proper viewing of a Tasseled
Cap image. Altering these values should not be necessary unless the analysts would like
to use the local values.
Figure 9. spectralProperties Table: global statistical properties for brightness, greenness, and wetness
which provides consistant viewing across many ecosystems and Landsat scenes..
5. Other tables
Figure 10. Tables 'plotspectral' and 'plotattribute' store analyst created TimeSync interpretation result.
These tables are summarized in step 5.
6. Close TimeSync database.
D. Using the Validation Database/TimeSync
1. Open TimeSync
a. Open an empty TimeSync session using the TimeSync shortcut to Timesync.exe
Figure 11. Empty TimeSync session.
2. Open TimeSync database
a. File->Open Database or use icon in upper left
b. Navigate to TimeSync Access database created in Step 3. This will load the Tasseled Cap
image chips to the Image Viewer, load the plot list in the left-side frame, and display the
spectral trajectory of the first plot.
Figure 12. Navigate to database created in Step 3.
Figure 13. TimeSync Visualization of image chips and spectral trajectory.
a. Check for any errors or anomalies in plot location, image stretch (See Appendix A for
examples), etc.
3. Open Google Earth
a. NOTE: Google Earth is not integral to TimeSync, but is a convenient approach to find
high resolution imagery. Other software platforms may be used if your site has
standardized imagery or formats that must be used. We provide examples in Google
Earth for reference.
b. Double Click on the Google Earth file (*.kml or *.kmz ) created in step 4c.
c. Check for location errors of Google Earth plots compared to TimeSync image chips.
Figure 14. Showing TimeSync plot locations in Google Earth.
Tip: Enabling the ‘Historical imagery’ option allows the opportunity to verify change using aerial
photography obtained by Google Earth. Beware that there is a combination of black/white and color, and
different spatial resolutions.
Figure 15. Showing TimeSync plot locations in Google Earth with the historical imagery
feature activated (View>Historical Imagery).
4. Describe the Plot
Figure 16. Tasseled Cap Spectral Space Image showing land cover
separability using R, G, B color space
a. See for examples of properly stretched Tasseled Cap imagery and examples of different
types of changes as a way of calibrating an analyst to the Landsat imagery.
b. Examine image chips in Image Viewer for changes. If change is observed use <ctrl>
button plus left-mouse-click on the image chip to create a new vertex point in the spectral
trajectory. This action will also add a row to the TimeSync database which allows the
analyst to describe the change.
c. Examine spectral plots located in TimeSync database. More options can be added
using the button next to the save button.
d. Examine plot using Google Earth. Analyst questions to ask:
i. Does the high resolution plot look like the Landsat image chip?
ii. Does the high resolution plot show land cover change corresponding to image
chips?
e. Describe trajectory segments in TimeSync database.
i. Cover Type
Cover Type (REQUIRED)
Agriculture
Barren
Forest
Other
Urban
Water
Wetlands
2. Cover Confidence
Cover Confidence (REQUIRED)
&nbsp;
Low
Medium
High
i. Segment Description
Description (REQUIRED:
Describes Segment)
*note first vertex does not have a description
Blowdown – low
Blowdown – medium
Blowdown – high
Decreasing Vigor
False Segment
Only for beginning or end points
Fire - low
Fire - medium
Fire - high
Harvest - low
Harvest - medium
Harvest - high
Insect - low
Insect - medium
Other disturbance - low
Other disturbance - medium
Other disturbance - high
Recovery towards conifer
Recovery towards hardwoods
Recover towards other
Stable
i. Affected Pixels
Affected Pixels (REQUIRED)
1–9
i.
Segment Confidence
Segment Confidence (REQUIRED)
Low
Medium
High
i.
Interpreter
Interpreter (REQUIRED)
i. Others
Comments
Brightness (automatically updated from imagery)
Greenness (automatically updated from imagery)
Wetness (automatically updated from imagery)
5. Save database
E. Summarizing TimeSync Plot Results
Validation requires comparing the results of independent data sets to create a confusion matrix. This is
achieved by first summarizing the analyst populated (Section V) segment table in the TimeSync
database and secondly cross-referencing with the results of the final LandTrendr map (SOP 4) to create a
confusion matrix.
The TimeSync interpretation data are stored in Microsoft Access, a desktop relational database. The
interpretation can be summarized using SQL. There are two built-in SQL queries with TimeSync
database, which can be used to summarize the data. Additional summary can be built using customized
SQL.
1. Summarize TimeSync Segments
a. Open timesync database with Access.
b. Open Queries "summary". The summary query summarize the number of segments by
segment type.
c.
Open Queries "segmentDates". This summary query list all the segments for each plot
interpreted.
2. The above summary table can be linked with Landtrendr data in next step.
F. Determining Map Accuracy using TimeSync
1. Create plot table with TBCD and TimeSync label
a. Open ArcGIS.
b. Add plot points created in step 4a or 4b.
i. WHICH STEP 4A OR 4B?
c. Add raster: Landtrendr final map. This could be any of the output maps from step SOP
#4.
d. Add ‘**’ table from ‘TimeSync_*.mdb’
i. IS THERE REALLY A "**" TABLE? IF NOT, PERHAPS GIVE AN EXAMPLE OF
THE ACTUAL TABLE.
e. Join summarized table to plot_point.shpbased on plot number
i. HOW? IS THIS AN ARCMAP FUNCTION?
‘Identity’ Landtrendr Map results with Plot points shapefile to add the
Landtrendr
map value to the Plot points attribute table. Now you have
both the TimeSync
classification and the Landtrendr classification in the
same table.
i. HOW? IS THIS DONE IN ARCMAP OR IN ACCESS? SOME DETAILS OF THE
STEPS WOULD BE HELPFUL.
g. Export plot_point.shp table, save as ‘plot_points_landtrendr<file>_TimeSync.dbf’
i. AGAIN, IN ARCMAP OR ACCESS?
f.
2. Create confusion matrix
a. Open ‘plot_points_landtrendr<file>_TimeSync.dbf’ in Excel, save as
‘TimeSync_confusion_matrix_landtrendr<file>.xls’
b. Data>PivotTable and PivotChart Report…>
c. Drop fields onto pivot table
i. Row Field = TimeSync
ii. Column Field = LandTrendr<file>
iii. Data Items = Count of ‘Plot Id’
Protocol for LandTrendr
Standard Operating Procedure (SOP) #6
Linking LandTrendr with landcover maps
Version 0.01 (October 1, 2009)
Revision history log:
Previous Version Revision Date
Author Changes Made Reason for Change New Version #
New
REK
April 12, 2009
New
August 12, 2009 YANG Modification
0.01
0.01
Overview
I. Important background on software
II. Inputs and Outputs
III. Defining POM Classification Rule
a. A. Unsupervised Classification of Base Image
b. B. Extraction of Classification Rule
c. C. Linking Spectral Classes with Existing Vegetation Map
5. IV. Creating POM Image Series
a. A. Create spectral classes
b. B. Convert spectral class to vegetation class
1.
2.
3.
4.
Overview
A common goal in monitoring landscapes is to label the kind of change that is occurring over time. The
maps produced in SOP 4 (Change maps) of this protocol identify where change is occurring, and give
some sense of whether the change is of a disturbance, recovery, or combined type, but do not describe
the condition of the landscape before and after the change in the detail needed for some natural resource
applications.
Therefore, this SOP describes a method by which a basic landcover map from a single point in time can
be used as the basis for building analogous landcover maps for every year in the image stack produced
in SOPs 1 and 2. Once the landcover maps are produced for each year, the change maps from SOP 4
can be considered in relation to the cover type changes that they engendered. Note that this SOP
assumes that there is a landcover map already in existence, produced external to this project, that needs
to be extended forward and backward in time.
The starting information in this SOP is the fitted tasseled-cap imagery that was optionally included in the
original LandTrendr segmentation batchfile (SOP 3) using the fit-to-vertices (FTV) commands. The fitted
tasseled-cap images are very stable over time, which allows us to build a classification in the spectral
space of an image close in time to the date of an existing landcover map, and then apply it to the spectral
space of the other images in the stack.
I. IMPORTANT BACKGROUND ON SOFTWARE
In addition to the ENVI+IDL environment, this protocol also uses ArcGIS 9.x and Erdas
Imagine 9.x for additional processing.
II. INPUTS AND OUTPUTS
Based on the landtrendr segmentation, produce yearly image for selected spectral images.
Input: land cover map; fitted images;
Output: yearly land cover map
III. DEFINING POM CLASSIFICATION RULE
A. Unsupervised Classification of Base Image
1. Select the Landtrendr produced yearly tasseled cap image with date close to existing Vegetation
Map as the base image.
2. Normalize the base tasseled cap image to allow equal spectral contribution in spectral space
partition. The normalization is done by Di,j = (Di,j - Di)/std(Di), where Di,j is the pixel value for
band i pixel j; Di is the mean value for band i; and std(Di) is the standard deviation of all pixels in
band i. This normalization step is introduced for the purpose of partitioning the spectral space
with equal weight to brightness, greenness, and wetness. This step is done in ERDAS Imagine
as follows:
a. In Erdas Imagine, Click on Modeler
(
)
b. Click Model Maker (
a. Open the normalization spatial model
) to open Spatial Modeler
a. Double click Input icon in the modeler to select the based image using file browser.
a. Double click Output icon to set normalized image.
3. Click on Classifier (
a. Click on
) on Erdas Imagine
on Classification Toolbox
a. Set Input Raster File, Output Cluster Layer, Number of Classes, Maximum Iterations on
the Unsupervised Classification dialog box.
i. set Input Raster File to the normalized Tassel Cap image from 2.5 above.
ii. set Output Cluster Layer to an nonexist image file.
iii. set Number of Classes to 50 (the number of classes can be changed to user
preference).
iv. set Maximum Iterations to 60.
v. uncheck Output Signature Set.
B. Extraction of Classification Rule
Unsupervised classification from above is used to define the spectral properties, which will be applied
across all the LandTrendr fitted images.
a. Derive spectral statistics for unsupervised classes in ENVI. Click on NPS Functions on
ENVI main menu, and select POM Utilities/Create Class Likelihood File
a. On Image with classes window, select unsupervised classification image from Select
Input File list, and click OK.
i. If unsupervised classification image has not been open, use the Open button to
open the file to open the image
a. On Tasseled-cap image window, select the non-normalized tassel cap image
corresponding to the unsupervised classification, and click OK.
a. If the tasseled cap image is not in the opened image list, follow step 2.1 above to open
the file.
a. On Select classes to use window, uncheck [1] Unclassified to exclude deriving statistical
properties for background pixels. Click OK.
a. On Choose output probability space save file window.
C. Linking Spectral Classes with Existing Vegetation Map
1. Characterize spectral clusters from unsupervised classification with exising vegetation map. This
step requires two input files: the unsupervised spectral cluster image from A.3 above, and
existing vegetation map in raster format. This step is done in ENVI.
a. Select POM Utilities-->Summarize Class by Class from ENVI menu.
b. On Locate the physiognomic class maximum likelihood image window, select
unsupervised classification image.
c. if unsupervised image has not open,click Open->New File to open the image.
d. On Locate the original park-specific classified image window, select existing vegetation
image to be linked with spectral classes.
e. On Choose output summary matrix file name window, type in an output file name or use
Choose button to browse to a directory to provide a file name.
2. The previous step creates a cross table between spectral class and the existing vegetation map.
Open the file file in Excel, the columns correspond to existing vegetation classes, and the rows
correspond to the spectral classes.
3. Normalize spectral class summary of vegetation classes.
a. In the Excel file that has the opened cross table, Click the new worksheet icon to add a
new worksheet in Excel, and optionally name it "normalized".
b. Copy the existing worksheet with the cross table summary and paste it to the new
worksheet.
c. In "normalized" worksheet, divide each cell by the sum of the corresponding column.
d. Delete "background" and "total" column.
e. Delete "Unclassified" and "total" row.
f. Click new worksheet icon to add a new worksheet, and optionally name it "transition".
g. Select all the cells exclude the first row and first column in worksheet "normalized".
h. In worksheet "transition", right click mouse on the first cell, and select "Paste Special".
i.
On Paste Special window, select "Values" and check "Transpose", click OK.
j.
Save the "transition" worksheet as a tab-delimited text file. This text file will be used to
translate spectral classes from landtrendr fitted images into vegetation classes.
IV. CREATING POM IMAGE SERIES
With the classification rules and spectral-to-vegetation transition file defined in the "Defining POM
Classification Rule" step, they can be now applied to the time series of spectral images derived from
Landtrendr (see SOP # for details on Landtrendr fitted images). This step is done with IDL batch files.
A. Create spectral classes
1. Preparing batch file
a. Open template file "run_apply_pom_template.pro" using the IDL Workbench. The
following text shows the portion of this file where parameters controlling the run are
defined.
runinfo = {
outdir: "J:\NAFD\3735\POM2\Clasification By Rules\", $ ; output directory
base_file: "ftv_tc_3735_wetness_", $ ;base output filename
spec_savefile: "J:\NAFD\3735\POM2\Classification Rules\pom_3735_spec.sav", $ ;spectral classification
rule
tc_dir: "J:\NAFD\3735\outputs\wupa\fitted_tc\", $ ;input tc file directory, assuming all the fitted image in
the same directory
image_info_file: "J:\NAFD\3735\glovis\landtrendr_image_info_3735.sav" ;image_info structure used to
run landtrendr
}
apply_pom, runinfo
b. Note that any text that follows a semi-colon (;) is considered a comment by IDL, not a
command, allowing notes to be inserted into the batchfile for reference.
c. Also note that the formatting of the document displayed here is narrower than the original
text files, causing some text to wrap around to the next line when viewed here. IDL will
not see the wrap of text.
d. Save the file as "run_apply_pom_pprr.pro, where pp is path, and rr is row of the images
to be evaluated.
2. Modify the runinfo variables:
a. outdir: "J:\NAFD\3735\POM2\Clasification By Rules\"
i. change this to where the output spectral classification images are saved.
b. base_file: "ftv_tc_3735_wetness_"
i. change this to the name to be used as part of output image file.
c. spec_savefile: "J:\NAFD\3735\POM2\Classification Rules\pom_3735_spec.sav"
i. change this to the file location of classification rule created in III.B.
d. tc_dir: "J:\NAFD\3735\outputs\wupa\fitted_tc\"
i. change this to the directory name where the tasseled cap image from Landtrendr
are located.
e. image_info_file: "J:\NAFD\3735\glovis\landtrendr_image_info_3735.sav"
i. change this to the image info file used to run Landtrendr.
3. Run batch file
a. run the batch as "@run_apply_pom_pprr.pro" on IDL Workbench command line
b. For each year there is a Landtrendr tasseled cap image, the above step will create an
image "*_probs.bsq", where * is the base_file specified in the batch file.
B. Convert spectral class to vegetation class
1. Preparing batch file
a. Open template file "run_convert_spec_to_veg_template.pro" using the IDL Workbench.
The following text shows the portion of this file where parameters controlling the run are
defined.
runinfo = {
matfile: "F:\3834\POM\ftv_tc_spec_to_veg.dat", $ ;spec to veg conversion matrix
probdir: "F:\3834\POM\", $ ;spectral probability image
outdir: "F:\3834\POM\", $ ; output directory
base_file_name: "ftv_tc_3834_wetness_", $ ;base output filename
image_info_file: "F:\3834\glovis\landtrendr_image_info_3834.sav" ;image_info structure used to
run landtrendr
}
convert_spec_to_veg, runinfo
b. Save the file as "run_convert_spec_to_veg_pprr.pro, where pp is path, and rr is row of
the images to be evaluated.
2. Modify the runinfo variables:
a. matfile: "F:\3834\POM\ftv_tc_spec_to_veg.dat"
i. change this to the conversion tab-delimited file created in III.C.
b. probdir: "F:\3834\POM\"
i. change this to where the output from step IV.A are saved.
c. outdir: "F:\3834\POM\"
i. change this to where output will be saved
d. base_file: "ftv_tc_3834_wetness_"
i. change this to the name to be used as part of output image file.
e. image_info_file: "F:\3834\glovis\landtrendr_image_info_3834.sav"
i. change this to the image info file used to run Landtrendr.
3. Run batch file
a. run the batch as "@run_convert_spec_to_veg_pprr.pro" on IDL Workbench command
line.
b. for each image year, the batch file will create "*_veg_prob.bsq" and "*_veg_class.bsq",
where * is the base_file_name specified in the batch file.
i. "*_veg_prob.bsq" is a multi-bands image with each band represent the
probability of the corresponding class. For each predicted vegetation class,
derived from the input vegetation map, there is a corresponding band in this file,
therefore the size of the file will be determined by the number of vegetation
classes.
ii. "*_veg_class.bsq" is a classified image using the same classes as the existing
vegetation map.
4. View classification image
a. The generated vegetation class image can be viewed directly in ENVI.
Protocol for LandTrendr
Standard Operating Procedure (SOP) #7
Field Evaluation of LandTrendr outputs
Version 0.01 (October 1, 2009)
Revision history log:
Previous Version
Revision Date
Author Changes Made
Reason for Change
New Version #
New
April 12, 2009
REK
New
0.01
October 1, 2009
BG
Updated
0.02
1. Overview
2. I. Software used
a. A. ArcGIS 9.X
b. B. Powerpoint
c. C. Excel
d. D. MapSource
e. E. DNRGarmin
f. F. Pathfinder Office
3. II. Hardware used
a. A. GPS
b. B. Digital camera
c. C. Radios
4. III. Required data from prior SOPs
a. A. Disturbance polygons
5. IV. Before the field
a. A. Datasheet
b. B. Maps and polygon attributes
c. C. Preparing GPS units
d. D. Field equipment list (see Table XX)
6. V. In the field
a. A. Direct visit
7. VI. After the field
a. A. Download GPS
b. B. Download pictures from camera
c. C. Enter datasheets into feature class
Overview
After validating disturbances in the lab using available airphotos and ancillary data sources, there will be
some disturbances in which you were unable to determine if a change occurred.
Figure 1. SOP 7 within the LandTrendr process.
I. SOFTWARE USED
The list below provides the software that the authors have used to perform field reconnaissance and
measurements. It is recommended to use the list as only a guideline. There are many software
packages available, many for free, which could be utilized to produce the products necessary for
validation. Before purchasing expensive software or hardware, please consult the authors to determine if
you have an alternate piece of software that would suffice. The authors have made efforts to minimize
the amount of expensive software used for this protocol, however, the authors also have used the best
tools or software available to carry out a long term monitoring protocol.
A. ArcGIS 9.X
1. used to develop maps and attributes for the field
B. Powerpoint
1. contains all information (maps and miscellaneous image) to eventually be printed out for the field
C. Excel
1. used to produce datasheet, if preferred, Microsoft Access may be used
D. MapSource
1. used to load topo maps of park onto Garmin handheld GPS
2. the software comes along with the GPS when you purchase the unit
3. topo maps are available by purchasing through Garmin
E. DNRGarmin
1. used to transfer waypoints to and from handheld Garmin GPS
2. freely available online at
http://www.dnr.state.mn.us/mis/gis/tools/arcview/extensions/DNRGarmin/DNRGarmin.html
F. Pathfinder Office
1. used to transfer polygon shapefile to the Trimble unit
2. only necessary if using a Trimble GPS
II. HARDWARE USED
The list below provides the hardware used for field validation. As was the case with the software, this
hardware list is an ideal scenario. Field validation can be carried out in many different ways, with many
different pieces of hardware. There are two key pieces of hardware critical to field validation: 1) GPS
capable of recording waypoints and 2) digital camera.
A. GPS
1. Trimble GeoXT or comparable GPS device
a. load compressed high resolution airphotos, topo maps, a recent NVCS classification, and
disturbance polygons will be loaded onto this unit, depending on available memory
b. A Trimble unit is helpful because it allows the user to load the polygon shapefile and
while in the field, you can view where you are (the blinking dot) in relation to the polygon.
2. Garmin or comparable handheld GPS device
a. load centroids of polygons to be field-validated
b. load topo maps of areas to be visited in field
B. Digital camera
1. used to document field visits
2. any digital camera with 5 megapixel resolution is fine
3. a shockproof/waterproof camera such as the Olympus Stylus 1030 SW is recommended but not
necessary
C. Radios
1. used at parks to communicate with dispatch and personnel
III. REQUIRED DATA FROM PRIOR SOPS
A. Disturbance polygons
IV. BEFORE THE FIELD
Preparation for the field is an important part of field validation and care and time spent on pre-field
preparation will pay dividends while in the field. The steps below provide guidance in preparing maps in
addition to ancillary data (polygon attributes) for use in the field. In addition to maps, preparing the GPS
unit(s) and walking through the checklist will also prepare the user for the fieldwork ahead. The fieldwork
checklist provides useful items necessary to conduct work in the field, as well as some items easy to
forget such as sunscreen and a first aid kit.
A. Datasheet
1. a template datasheet (.xlsx) has been included for guidance
2. users can modify this datasheet as they deem necessary
B. Maps and polygon attributes
1. maps will be created in ArcMap 9.X, exported (pdf), then copied into the blank powerpoint project
the user has created
2. limit polygons to only those where 'field_validation_candidate' = 'yes'
a. if a recent vegetation map exists, overlay this over most recent high resolution airphoto
with the vegetation classes labeled
b. export these limited polygons to be loaded on the Trimble unit for field validation
3. create a 2 maps for each polygon
a. the first map should have the following information
i. labeled roads (if present)
ii. labeled trails (if present)
iii. labeled (formation level) NVCS polygons
iv. scale (for distance)
v. labeled polygon with patchid
vi. scale of map can be anywhere from 1:4000 to 1:12000, depending on the size of
the disturbance polygon
1. for example, for small disturbances, the scale could be larger because
for this map we're only interested in the polygon and the immediate
surrounding area
vii. for the background, use the most current high resolution airphoto
viii. when satisfied with map, export as pdf, name map with patchid
b. the second map should set up and have the following information
i. labeled (formation level) NVCS polygons
ii. scale (for distance)
iii. labeled polygon with patchid
iv. scale of map should be approximately twice of that of the first map
v. for the background, use the best available topo map
vi. when satisfied with map, export as pdf, name map with patchid, followed by
_topo
c. import these maps into the powerpoint project, which each map being it's own slide
4. in addition to exporting the map, click on the information button (figure XX) for each patchid
a. press Alt + PrtScn to copy this window (Figure XX) to the clipboard
b. paste this screen capture into the same powerpoint presentation
i. you can place 4 of the attribute screen captures on one slide
®
1. when finished, print out entire powerpoint project onto "Rite in the Rain " paper
C. Preparing GPS units
1. Turn on unit(s), make sure they're functioning
2. If a Trimble unit is being used, check for updates of Terrasync, firmware, and operating system
for your particular make and model
a. following the Pathfinder office user guide, load the appropriate data onto the unit
i. compressed high resolution airphotos
ii. topo map
iii. polygons to be field validated
1. use definition query to select only polygons where field validation is
required
2. export these selected polygons for upload into the Trimble unit
3. Following the instructions in the MapSource user guide, load the appropriate topos onto the
Garmin unit
D. Field equipment list (see Table XX)
1. The hardware listed in Section II should be inspected at least a week before the field work is to
begin.
2. Before driving to the field site, review Table XX
Table XX. Field equipment for field validation
Number Description
Req.
1
Radio programmed for park (with additional battery and charger)
1
Digital camera (with 1 additional set of batteries or battery charger)
1
Compass (with declination set for particular park
1
Binoculars
1
Handheld GPS unit for navigating to points (with 4 additional AA batteries)
1
Trimble GPS unit for displaying location in relation to polygon boundary (with battery charger)
Variable Data sheets for the number of plots to sample
3
Pencils for marking datasheets
3
Sharpie marker for making notes on airphoto
1
Clipboard for holding amd marking datasheets
1
First aid kit
1
Sunscreen
1
Insect repellent
1
Watch
®
V. IN THE FIELD
A. Direct visit
**At the beginning of each day turn on the Trimble, Garmin, and camera. Sync the time of all them, using
the Trimble unit as reference.
1. A direct visit in the field will be used to ascertain whether a disturbance has occurred. Using
whatever means available, hike to, boat to, or fly to the polygon centroid of interest using the
handheld GPS and maps.
2. As you are getting close to the polygon, turn on the Trimble GPS, load the disturbance polygon
layer to the map, and wait to see where you are in relation to the disturbance polygon. While you
are walking through the polygon to the centroid, look around for any signs of disturbance.
Disturbance signs could include, but are not limited to, recently fallen trees, dead standing trees,
recently flooded area, dying trees, and leaf diseases.
3. Once at polygon centroid (center of polygon), record a waypoint, fill out field-validation datasheet
(fig. XX) and take multiple photos of area. Digital photos will aid in long-term storage of
disturbance examples and provide useful information in the instance the imagery does not explain
data collected during the field visit. Important characteristics of photos include a photo directly
overhead, oblique photo of understory, closeups of disturbance evidence such as down woody
debris or stumps and any closeups of disease or infection. Record this photo number(s) on
datasheet for future reference. If there are any other notable features of this disturbance polygon
make sure to note these in the 'comments' space of datasheet. Once finished filling in the
datasheet, take a photo of the datasheet. This should be the last photo of the area, to aid in
identifying which group of pictures goes with a polygon.
a. Before finishing the datasheet, walk to the other end of the polygon, just to make sure
you have seen the extent of the polygon
4. Determination of change call accuracy:
a. Using the airphoto and attribute table for this patchid, reconstruct what was determined in
the lab. By using this information we are able to construct what we think we should see
in the field. For example, if the lab notes state "possible slow decline in jack pine", then
this could guide you in the field. Of course, this doesn't mean that you should only look
for a slow decline jack pine, rather, look for signs of slow decline or another subtle longlasting disturbance.
5. Filling out the datasheet:
a. Park
i. filled in before the field, or enter the four-letter code for the current park
b. PatchID
i. the unique id of the patch (polygon)
c.
d.
e.
f.
g.
h.
i.
j.
k.
l.
ii. fill out a datasheet for each polygon
Date
i. date of field visit
1. example: 10/1/09
Personnel
i. initials of people in the field
GPS unit
i. make and model of GPS unit taking the waypoints
Camera
i. make and model of camera
GPS and pictures
i. Waypoint ID
1. record waypoint ID
ii. Time
1. record the time (24 hr format) of when the waypoint was created
iii. # pictures @ wpt
1. record the number of pictures taken at a waypoint
2. do not take pictures without first saving a waypoint
iv. filling in these fields carefully and fully each time will save time in the lab after the
field work is finished
Evidence of disturbance
i. circle the evidence of disturbance, if evidence not listed, make a note and add it
at the end under the "comments" section
False reason
i. in the case the polygon is a 'false' change, circle reason why this polygon was
detected
ii. if unknown, do not circle any
iii. do not spend too much time on this field, it is for qualitative purposes and
tracking development of LandTrendr
Starting/ending classes, disturbance agent
i. this is the same as was done in the lab, although in the field you will have to walk
around the polygon to determine which areas were affected and what their
starting class was
ii. choose classes from list provided, under "starting/ending classes"
Starting/ending classes
i. list of possible classes to choose from
Comments
1. write down any comments in this section, the more verbose the better
a. generally describe the area: cover types, visible bedrock,
standing water, overall vegetation composition, slope, elevation
VI. AFTER THE FIELD
A. Download GPS
1. if your handheld unit is a Garmin, use DNRGarmin to download the points collected in the field
and save them as a point shapefile
B. Download pictures from camera
1. download the camera's pictures to the appropriate folder
2. use a filename renaming program, such as Total Commander (http://www.ghisler.com/), to
automatically rename files using the following naming protocol
a. four-letter park code_YMD_hms.jpg
i. four letter park code
ii. YMD = year, month, date of picture
iii. hms = hours, minutes, seconds of picture
iv. patchid
1. this will be added after you rename all files, then go back and select only
pictures associated with a single patch and rename these with the
patchid at the end of the filename
C. Enter datasheets into feature class
1. Open the LandTrendr_validation mxd
2. Start editing the polygon layer and edit each patchid (polygon) that was visited in the field
a. use the data collected in the field to update and make changes to the polygons
b. when finished, save edits, and stop editing.
Protocol for LandTrendr
Standard Operating Procedure (SOP) #8
Data management
Version 0.01 (October 1, 2009)
Revision history log:
Previous
Version
Revision
Date
Author
Changes
Made
Reason for
Change
New Version
#
New
April 12,
2009
REK
New
0.01
May 20,
2009
PN
populated
0.01
Table of Contents
1.
a.
i. Overview
ii. I. Journaling
iii. II. Metadata
1. A. FGDC - compliant metadata
iv. III. File naming
v. IV. Directory and File structure
1. A. Scene Path and Row\
vi. V. Works Cited
Overview
An overview of the LandTrendr processing work flow involves many steps. First,
Landsat images for each year between 1984 to present, primarily during July or August
time period, is obtained from the Landsat data archive (SOP 1). Second, COST
atmospheric correction, MADCal radiometric normalization, and cloud masking is
performed on the images in the yearly stack (SOP 2). Third, the image stack is analyzed
using the LandTrendr process (SOP 3) which summarizes and labels the Landsat
observed long-term trends from year to year. Next, the LandTrendr output results are
evaluated for their accuracy using independent data visualization using TimeSync (SOP
6). Finally, the analysis and final products are documented and archived (SOP 8).
This Standard Operating Procedure describes the steps needed to ensure that images,
image processing steps, and interpreted data are maintained and documented correctly.
I. Journaling
It is recommended that an electronic text file journal be maintained to keep track of all
filenames and decisions made during the steps of this protocol. This journal should be
maintained in chronological order, with the date and associated filenames for each
entry. Whenever a processing step is conducted, all of the relevant information about
that step is entered into the journal. Enough detail should be given to fully replicate the
step, including the names of all input and output files, the software module and tool
used for the analysis, and all processing options (buttons selected, data types, etc.). The
use of hyperlinks to other files can also be a useful feature because it provides full
pathnames of files that can be used later for interpretation, even if the hyperlinks
themselves become out of date because files are moved.
Additionally, the journal should be used to record rationale for making decisions and for
noting observations about imagery or datasets that may be useful later for error tracking
or evaluation of unexplained results. Although the key steps leading up to any
geospatial product will also be stored in the metadata, this journaling process allows
much more flexibility in keeping track of details. Most of the time, these details are not
needed later, but occasionally they are the difference between a quick fix and starting
from scratch.
It is useful to keep track of the file processing flow in a spreadsheet. We have provided a
file name worksheet in the larger LandTrendr processing workbook that will
automatically generate file names for all important steps based on user updates of scene,
LandTrendr process, and image date fields. This file is extremely helpful for keeping
track of processing steps. It will be covered in more depth below.
II. Metadata
Good metadata will allow complete replication of a product. The processing of imagery
requires many steps, each of which requires documentation to be replicable. In Table 1,
the major steps in image processing through this protocol are listed, along with the
resultant image and the information that must be stored in the metadata.
The validation process also requires metadata.
Table 1. Image processing steps and their metadata requirements.
Step
Resultant
image
Image
download
7-band Landsat
Date of image acquisition; date image was downloaded; all
TM or ETM+
header information included with image
image
COST
Atmoshpericcorrected 6band image
MADCAL
Name of parent image and of radiometric reference image;
radiometrically calculated gains and offsets relating each band in parent
normalized
image to reference image; location and size of image subsets
image
used to identify no-change pixels; name of batchfile used to
run MADCAL algorithm
Application of
tasseled cap
tasseled-cap
image
coefficients
cloud score,
shadow score,
Cloud Mask
cloud-shadow
mask images
LandTrendr
Required metadata
Name of parent image. Dark object values, gains, bias, solar
elevation.
Name of parent image; coefficients used for tasseled-cap
calculations
Name of parent image and of cloud reference image; cloud
score and shadow score thresholds; name of batchfile used
to run algorithm
Parameters of Landtrendr run; name of batchfile used to
run algorithm;
A. FGDC - compliant metadata
For National Park Service projects, refer to the appropriate metadata tools and
requirements as described at http://science.nature.nps.gov/nrdata/tools/
Information about general metadata standards please refer to: www.fgdc.gov/metadata
The Content Standard for Digital Geospatial Metadata (CSDGM), Vers. 2 (FGDC-STD001-1998) is the US Federal Metadata standard. The Federal Geographic Data
Committee originally adopted the CSDGM in 1994 and revised it in 1998. According to
Executive Order 12096 all Federal agencies are ordered to use this standard to
document geospatial data created as of January, 1995. The standard is often referred to
as the FGDC Metadata Standard and has been implemented beyond the federal level
with State and local governments adopting the metadata standard as well. (FGDC,
2009)
1. Required elements (ISO 19115 Core Metadata Elements)
Dataset title
Dataset reference date
Dataset language
Dataset topic category
Abstract
Metadata point of contact
Metadata date stamp
2. Topic Categories (ISO 19115 Topic Categories)
farming
biota
boundaries
climatologyMeteorologyAtmosphere
economy
elevation
environment
geoscientificInformation
health
imageryBaseMapsEarthCover
intelligenceMilitary
inlandWaters
location
oceans
planningCadastre
society
structure
transportation
utilitiesCommunication
III. File naming
File names are concatenated sequences of codes that uniquely describe a file. We
advocate descriptive code names that allow a user to immediately identify each files
purpose. A filename usually consists of a root name with an additional component
describing the file. Table 2 provides a list of all of the component code names for one
example of images for one scene (path 42 row 34). An example file name strings
together these codes in this order:
(IMAGETYPE)(PATH)(ROW)_(YEAR)_(JULIAN DATE)_(IMAGESOURCE).(FILE FORMAT)
LT5046031_1985_219_archv.img
Table 2. Filename component templates.
[To be corrected: REK 9/1/11]
IV. Directory and File structure
We recommend a directory structure where the primary node of the directory is defined
by path and row number, with subsequent directories and files defined by the
processing outputs. While processing a Landsat scene, LandTrendr creates new files in
specific directories. Text in bold with a trailing slash (\) represent a directory name
while text in italics represent a file name. pp = path, rr = row, year = specific year,
juldate = julian date.
A. Scene Path and Row\
pprr_year_COSTmxd.mxd
prrr_year_madcal.mxd
Processing_LandTrendr_pprr.xls
1. IDL_Batchfiles\
set_up_files_landtrendr_pprr.pro
COST_pprr_year.gmd
mad_landtrendr_pprr.pro
make_landtrendr_tcimages_pprr.pro
make_cloudscore_images_pprr.pro
mpcm_pprr.pro
lt043_pprr_nbr.pro
2. GIS_data\
studyarea_pprr_msk.img
madcal_pprr_subsets_polygons.shp
madcal_pprr_subsets_pts.shp
landcover_raster
3. Images\
1984\
LT5045033_1986_231_archv.bsq
LT5045033_1986_231_archv.hdr
LT5045033_1986_231_archv.hdr.flat
LT5045033_1986_231_b6.bsq
LT5045033_1986_231_b6.hdr
LT5045033_1986_231_b6.hdr.flat
LT5045033_1986_231_archv1986231_to_1998.bsq
LT5045033_1986_231_archv1986231_to_1998.hdr
LT5045033_1986_231_archv1986231_to_1998.hdr.flat
LT5045033_1986_231_archv1986231_to_1998.sav
LT5045033_1986_231_archv1986231_to_1998_ch0.tif
LT5045033_1986_231_archv1986231_to_1998_ch1.tif
LT5045033_1986_231_archv1986231_to_1998_ch2.tif
LT5045033_1986_231_archv1986231_to_1998_ch3.tif
LT5045033_1986_231_archv1986231_to_1998_ch4.tif
LT5045033_1986_231_archv1986231_to_1998_ch5.tif
LT5045033_1986_231_archv1986231_to_1998_ch6.tif
LT5045033_1986_231_archv1986231_to_1998_ltc.bsq
LT5045033_1986_231_archv1986231_to_1998_ltc.hdr
LT5045033_1986_231_archv1986231_to_1998_ltc.hdr.flat
LT5045033_1986_231_archv1986231_to_1998_madcaloutputs
.txt
LT5045033_1986_231_archv1986231_to_1998clddiff.bsq
LT5045033_1986_231_archv1986231_to_1998clddiff.hdr
LT5045033_1986_231_archv1986231_to_1998clddiff.hdr.flat
LT5045033_1986_231_archv1986231_to_1998cloudmask.bsq
LT5045033_1986_231_archv1986231_to_1998cloudmask.hdr
LT5045033_1986_231_archv1986231_to_1998cloudmask.hdr.f
lat
LT5045033_1986_231_archv1986231_to_1998shddiff.bsq
LT5045033_1986_231_archv1986231_to_1998shddiff.hdr
LT5045033_1986_231_archv1986231_to_1998shddiff.hdr.flat
radcal_1986231_to_1998.sav
radcal_1986231_to_1998_nochangepixels.bsq
radcal_1986231_to_1998_nochangepixels.hdr
radcal_1986231_to_1998_nochangepixels.hdr.flat
radcal_1986231_to_1998_subset1.sav
radcal_1986231_to_1998_subset1.sav
radcal_1986231_to_1998_subset1.sav
radcal_1986231_to_1998_subset2.sav
radcal_1986231_to_1998_subset3.sav
radcal_1986231_to_1998_subset4.sav
radcal_1986231_to_1998_subset5.sav
radcal_1986231_to_1998nochangepixels.sav
1985\
similar to *1984\* directory
1986\
similar to *1984\* directory
1987\
similar to *1984\* directory
1988\
similar to *1984\* directory
1989\
similar to *1984\* directory
1990\
similar to *1984\* directory
1991\
similar to *1984\* directory
1992\
similar to *1984\* directory
1993\
similar to *1984\* directory
1994\
similar to *1984\* directory
1995\
similar to *1984\* directory
1996\
similar to *1984\* directory
1997\
similar to *1984\* directory
1998\
LT5045033_1998_200_archv.bsq
LT5045033_1998_200_archv.hdr
LT5045033_1998_200_b6.hdr.flat
LT5045033_1998_200_b6.bsq
LT5045033_1998_200_b6.hdr
LT5045033_1998_200_b6.hdr.flat
LT5045033_1998_200_archv_cost.img
LT5045033_1998_200_archv_cost_ltc.bsq
LT5045033_1998_200_archv_cost_ltc.hdr
LT5045033_1998_200_archv_cost_ltc.hdr.flat
LT5045033_1998_200_archv_costclddiff.bsq
LT5045033_1998_200_archv_costclddiff.hdr
LT5045033_1998_200_archv_costclddiff.hdr.flat
LT5045033_1998_200_archv_costcloudmask.bsq
LT5045033_1998_200_archv_costcloudmask.hdr
LT5045033_1998_200_archv_costcloudmask.hdr.flat
LT5045033_1998_200_archv_costshddiff.bsq
LT5045033_1998_200_archv_costshddiff.hdr
LT5045033_1998_200_archv_costshddiff.hdr.flat
1999\
similar to *1984\* directory
2000\
similar to *1984\* directory
2001\
similar to *1984\* directory
2002\
similar to *1984\* directory
2003\
similar to *1984\* directory
2004\
similar to *1984\* directory
2005\
similar to *1984\* directory
2006\
similar to *1984\* directory
2007\
similar to *1984\* directory
2008\
similar to *1984\* directory
4. Outputs\
nbr\
LT043_nbr_4533_nwfp_durs.bsq
LT043_nbr_4533_nwfp_durs.hdr
LT043_nbr_4533_nwfp_durs.hdr.flat
LT043_nbr_4533_nwfp_mags.bsq
LT043_nbr_4533_nwfp_mags.hdr
LT043_nbr_4533_nwfp_mags.hdr.flat
LT043_nbr_4533_nwfp_vertvals.bsq
LT043_nbr_4533_nwfp_vertvals.hdr
LT043_nbr_4533_nwfp_vertvals.hdr.flat
LT043_nbr_4533_nwfp_vertyrs.bsq
LT043_nbr_4533_nwfp_vertyrs.hdr
LT043_nbr_4533_nwfp_vertyrs.hdr.flat
LT043_nbr_4533_nwfp_distrec.bsq
LT043_nbr_4533_nwfp_distrec.hdr
LT043_nbr_4533_nwfp_distrec.hdr.flat
LT043_nbr_4533_nwfp_fitted.bsq
LT043_nbr_4533_nwfp_fitted.hdr
LT043_nbr_4533_nwfp_fitted.hdr.flat
LT043_nbr_4533_nwfp_stats.bsq
LT043_nbr_4533_nwfp_stats.hdr
LT043_nbr_4533_nwfp_stats.hdr.flat
LT043_nbr_4533_nwfp_segmean.bsq
LT043_nbr_4533_nwfp_segmean.hdr
LT043_nbr_4533_nwfp_segmean.hdr.flat
LT043_nbr_4533_nwfp_segmse.bsq
LT043_nbr_4533_nwfp_segmse.hdr
LT043_nbr_4533_nwfp_segmse.hdr.flat
LT043_nbr_4533_nwfp_source.bsq
LT043_nbr_4533_nwfp_source.hdr
LT043_nbr_4533_nwfp_source.hdr.flat
LT043_nbr_4533_nwfp_diag.sav
LT043_nbr_4533_nwfp_drstack__distrec_duration.bsq
LT043_nbr_4533_nwfp_drstack__distrec_duration.hdr
LT043_nbr_4533_nwfp_drstack__distrec_duration.hdr.flat
LT043_nbr_4533_nwfp_drstack__distrec_mag.bsq
LT043_nbr_4533_nwfp_drstack__distrec_mag.hdr
LT043_nbr_4533_nwfp_drstack__distrec_mag.hdr.flat
LT043_nbr_4533_nwfp_drstack__distrec_preval.bsq
LT043_nbr_4533_nwfp_drstack__distrec_preval.hdr
LT043_nbr_4533_nwfp_drstack__distrec_preval.hdr.flat
LT043_nbr_4533_nwfp_drstack__patch_duration.bsq
LT043_nbr_4533_nwfp_drstack__patch_duration.hdr
LT043_nbr_4533_nwfp_drstack__patch_duration.hdr.flat
LT043_nbr_4533_nwfp_drstack__patch_mag.bsq
LT043_nbr_4533_nwfp_drstack__patch_mag.hdr
LT043_nbr_4533_nwfp_drstack__patch_mag.hdr.flat
LT043_nbr_4533_nwfp_drstack__patch_patchid.bsq
LT043_nbr_4533_nwfp_drstack__patch_patchid.hdr
LT043_nbr_4533_nwfp_drstack__patch_patchid.hdr.flat
LT043_nbr_4533_nwfp_drstack__patch_preval.bsq
LT043_nbr_4533_nwfp_drstack__patch_preval.hdr
LT043_nbr_4533_nwfp_drstack__patch_preval.hdr.flat
LT043_nbr_4533_nwfp_drstack__patch_patch_summary.txt
LT043_nbr_4533_nwfp_diag.sav
LT043_nbr_4533_nwfp_darkseg.bsq
LT043_nbr_4533_nwfp_darkseg.hdr
LT043_nbr_4533_nwfp_darkseg.hdr.flat
LT043_nbr_4533_nwfp_distrec.bsq
LT043_nbr_4533_nwfp_distrec.hdr
LT043_nbr_4533_nwfp_distrec.hdr.flat
LT043_nbr_4533_nwfpbrightness_ftv_vertvals.bsq
LT043_nbr_4533_nwfpbrightness_ftv_vertvals.hdr
LT043_nbr_4533_nwfpbrightness_ftv_vertvals.hdr.flat
LT043_nbr_4533_nwfpbrightness_ftv_fitted.bsq
LT043_nbr_4533_nwfpbrightness_ftv_fitted.hdr
LT043_nbr_4533_nwfpbrightness_ftv_fitted.hdr.flat
LT043_nbr_4533_nwfpbrightness_ftv_segmse.bsq
LT043_nbr_4533_nwfpbrightness_ftv_segmse.hdr
LT043_nbr_4533_nwfpbrightness_ftv_segmse.hdr.flat
LT043_nbr_4533_nwfpbrightness_ftv_stats.bsq
LT043_nbr_4533_nwfpbrightness_ftv_stats.hdr
LT043_nbr_4533_nwfpbrightness_ftv_stats.hdr.flat
LT043_nbr_4533_nwfpbrightness_ftv_segmean.bsq
LT043_nbr_4533_nwfpbrightness_ftv_segmean.hdr
LT043_nbr_4533_nwfpbrightness_ftv_segmean.hdr.flat
LT043_nbr_4533_nwfpbrightness_ftv_source.bsq
LT043_nbr_4533_nwfpbrightness_ftv_source.hdr
LT043_nbr_4533_nwfpbrightness_ftv_source.hdr.flat
LT043_nbr_4533_nwfpbrightness_diag.sav
LT043_nbr_4533_nwfpgreenness_ftv_fitted.bsq
LT043_nbr_4533_nwfpgreenness_ftv_fitted.hdr
LT043_nbr_4533_nwfpgreenness_ftv_fitted.hdr.flat
LT043_nbr_4533_nwfpgreenness_ftv_vertvals.bsq
LT043_nbr_4533_nwfpgreenness_ftv_vertvals.hdr
LT043_nbr_4533_nwfpgreenness_ftv_vertvals.hdr.flat
LT043_nbr_4533_nwfpgreenness_ftv_segmse.bsq
LT043_nbr_4533_nwfpgreenness_ftv_segmse.hdr
LT043_nbr_4533_nwfpgreenness_ftv_segmse.hdr.flat
LT043_nbr_4533_nwfpgreenness_ftv_source.bsq
LT043_nbr_4533_nwfpgreenness_ftv_source.hdr
LT043_nbr_4533_nwfpgreenness_ftv_source.hdr.flat
LT043_nbr_4533_nwfpgreenness_ftv_stats.bsq
LT043_nbr_4533_nwfpgreenness_ftv_stats.hdr
LT043_nbr_4533_nwfpgreenness_ftv_stats.hdr.flat
LT043_nbr_4533_nwfpgreenness_ftv_segmean.bsq
LT043_nbr_4533_nwfpgreenness_ftv_segmean.hdr
LT043_nbr_4533_nwfpgreenness_ftv_segmean.hdr.flat
LT043_nbr_4533_nwfpgreenness_diag.sav
LT043_nbr_4533_nwfpwetness_ftv_fitted.bsq
LT043_nbr_4533_nwfpwetness_ftv_fitted.hdr
LT043_nbr_4533_nwfpwetness_ftv_fitted.hdr.flat
LT043_nbr_4533_nwfpwetness_ftv_stats.bsq
LT043_nbr_4533_nwfpwetness_ftv_stats.hdr
LT043_nbr_4533_nwfpwetness_ftv_stats.hdr.flat
LT043_nbr_4533_nwfpwetness_ftv_vertvals.bsq
LT043_nbr_4533_nwfpwetness_ftv_vertvals.hdr
LT043_nbr_4533_nwfpwetness_ftv_vertvals.hdr.flat
LT043_nbr_4533_nwfpwetness_ftv_segmean.bsq
LT043_nbr_4533_nwfpwetness_ftv_segmean.hdr
LT043_nbr_4533_nwfpwetness_ftv_segmean.hdr.flat
LT043_nbr_4533_nwfpwetness_ftv_segmse.bsq
LT043_nbr_4533_nwfpwetness_ftv_segmse.hdr
LT043_nbr_4533_nwfpwetness_ftv_segmse.hdr.flat
LT043_nbr_4533_nwfpwetness_ftv_source.bsq
LT043_nbr_4533_nwfpwetness_ftv_source.hdr
LT043_nbr_4533_nwfpwetness_ftv_source.hdr.flat
LT043_nbr_4533_nwfpwetness_diag.sav
5. Workspace\
LE70450332000230EDC02.tar.gz
LE70450331999227EDC00.tar.gz
LE70450332002219EDC00.tar.gz
LE70450332004241EDC02.tar.gz
LE70450332005243EDC00.tar.gz
LT50450331986231XXX03.tar.gz
LT50450331987186XXX01.tar.gz
LT50450331988237XXX04.tar.gz
LT50450331989191XXX02.tar.gz
LT50450331991181AAA03.tar.gz
LT50450331992184XXX02.tar.gz
LT50450331993186XXX02.tar.gz
LT50450331994221AAA02.tar.gz
LT50450331995208XXX01.tar.gz
LT50450331996211XXX01.tar.gz
LT50450331997213XXX02.tar.gz
LT50450331998200XXX02.tar.gz
LT50450332001240LGS01.tar.gz
LT50450332003198EDC02.tar.gz
LT50450332006206EDC00.tar.gz
LT50450332007209EDC00.tar.gz
LT50450332008244EDC00.tar.gz
LE70450332000230EDC02.tar\
L71045033_03320000817_B10.TIF
L71045033_03320000817_B20.TIF
L71045033_03320000817_B30.TIF
L71045033_03320000817_B40.TIF
L71045033_03320000817_B50.TIF
L71045033_03320000817_B61.TIF
L72045033_03320000817_B62.TIF
L72045033_03320000817_B70.TIF
L72045033_03320000817_B80.TIF
L71045033_03320000817_GCP.txt
L71045033_03320000817_MTL.txt
README.GTF
LE70450331999227EDC00.tar\
Similar to *LE70450332000230EDC02.tar* directory
LE70450332002219EDC00.tar\
Similar to *LE70450332000230EDC02.tar* directory
LE70450332004241EDC02.tar\
L71045033_03320040828_B10.TIF
L71045033_03320040828_B20.TIF
L71045033_03320040828_B30.TIF
L71045033_03320040828_B40.TIF
L71045033_03320040828_B50.TIF
L71045033_03320040828_B61.TIF
L71045033_03320040828_B62.TIF
L71045033_03320040828_B70.TIF
L71045033_03320040828_B80.TIF
L71045033_03320040828_GCP.txt
L71045033_03320040828_MTL.txt
README.GTF
gap_mask\
L7G045033_03320040828_B10.TIF.gz
L7G045033_03320040828_B20.TIF.gz
L7G045033_03320040828_B30.TIF.gz
L7G045033_03320040828_B40.TIF.gz
L7G045033_03320040828_B50.TIF.gz
L7G045033_03320040828_B61.TIF.gz
L7G045033_03320040828_B62.TIF.gz
L7G045033_03320040828_B70.TIF.gz
L7G045033_03320040828_B80.TIF.gz
L7G045033_03320040828_B10\
L7G045033_03320040828_B10.TIF
L7G045033_03320040828_B20\
L7G045033_03320040828_B20.TIF
L7G045033_03320040828_B30\
L7G045033_03320040828_B30.TIF
L7G045033_03320040828_B40\
L7G045033_03320040828_B40.TIF
L7G045033_03320040828_B50\
L7G045033_03320040828_B50.TIF
L7G045033_03320040828_B61\
L7G045033_03320040828_B61.TIF
L7G045033_03320040828_B62\
L7G045033_03320040828_B62.TIF
L7G045033_03320040828_B70\
L7G045033_03320040828_B70.TIF
L7G045033_03320040828_B80\
L7G045033_03320040828_B80.TIF
LE70450332005243EDC00.tar\
similar to *LE70450332004241EDC02.tar* directory
LT50450331986231XXX03.tar\
L5045033_03319860819_B10.TIF
L5045033_03319860819_B20.TIF
L5045033_03319860819_B30.TIF
L5045033_03319860819_B40.TIF
L5045033_03319860819_B50.TIF
L5045033_03319860819_B60.TIF
L5045033_03319860819_B70.TIF
L5045033_03319860819_GCP.txt
L5045033_03319860819_MTL.txt
LT50450331987186XXX01.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450331988237XXX04.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450331989191XXX02.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450331991181AAA03.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450331992184XXX02.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450331993186XXX02.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450331994221AAA02.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450331995208XXX01.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450331996211XXX01.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450331997213XXX02.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450331998200XXX02.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450332001240LGS01.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450332003198EDC02.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450332006206EDC00.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450332007209EDC00.tar\
Similar to *LT50450331986231XXX03.tar* directory
LT50450332008244EDC00.tar\
Similar to *LT50450331986231XXX03.tar* directory
V. Works Cited
FGDC (2009). Geospatial Metadata Standards - Federal Geographic Data Committee.
http://www.fgdc.gov/metadata/geospatial-metadata-standards.