Agricultural Stress Index System (ASIS)
based on remote sensing data
Interim meeting 2 (18-22 March 2013 at FAO):
13/07/2016
Software and others
Roel Van Hoolst, Herman Eerens, Antoine Royer
Dominique Haesen, Lieven Bydekerke
13/07/2016
© 2011, VITO NV
1
OVERVIEW (see Report)
1. Intro: Definitions & Overview of the work
2. Basic data: RS-images & Ancillary info
3. Basic tools: GLIMPSE, SPIRITS and others
4. Data processing
5. Results: Display & evaluation
6. Conclusions: Done’s & todo’s
Others: METOP simulation, cropmask,...
DEFINITION OF THE ASI = Agricultural Stress Index
ASI = Percentage of “cropped areas” per “administrative region”, which
is “affected by drought” according to “EO-observations” and defined
over the course of the “growing season”:
•
Admin. Regions
 GAUL1 (ca. 3400 units)
•
Cropland
 JRC crop mask (adapted version)
•
EO-observations
 from AVHRR-sensors
•
•
•
NOAA (16km, since 1984) & METOP (1km, since 2007)
Two variables: NDVI (vegetation state) & BT4 (temperature)
In advance: all data smoothed & converted to ten-daily frequencies
DEFINITION OF THE ASI = Agricultural Stress Index
First: Make long-time series of such EO-images.
Then: Derive the following:
•
LT-statistics  Per dekad: min, mean, max of NDVI and BT4
(computed over all the available years).
•
Anomalies  For all dekads: VCI, TCI and VHI
(low VHI’s are indicative for drought).
•
Phenology  Definition of “growing season” (SOS/EOS )for 1/2
seasons per year from LT-mean NDVI
•
Mean VHI  Repeated for both seasons (s=1/2) and in two versions:
• µVHI,s,t
from SOS to current data, recomputed every dekad
• µVHI,s,y
from SOS to EOS, recomputed yearly
•
Final ASI
•
•
ASIs,t
ASIs,y
 % of cropland pixels per region with µVHI,s < 35%
This definition of “drought” on a regional scale.
Again for both seasons (s=1/2) and in 2 versions:
from SOS to current data, recomputed every dekad
from SOS to EOS, recomputed yearly
DEFINITION OF THE ASI = Agricultural Stress Index
Two processing modes:
•
Back-processing: Compute ASIs,y for all previous years
• Check in how far it reveals known drought periods in the past
• Useful for validation (+ suggestions for improvements)
•
NRT-processing: Compute ASIs,t per dekad in on-going year
• Detect dry zones (delineation and intensity)
• Useful in the daily practices of FAO.
DEFINITION OF THE ASI = Agricultural Stress Index
Everything is based on the paper by Rojas, Rembold & Vrieling.
But there are many differences:
• Africa  Global
• NOAA (16 km)  METOP (1km resolution).
• Now also automated processing, both for back- and NRT-modes
• Always (per dekad/year) with IMGs, QLKs and RUM-files
New issues:
• Conversion to 1km increases data amounts and processing times
• METOP only starts in 2007, and the ASIS requires longer time series
 For the past, 1km METOP data must be simulated from 16km NOAA
• Are the assumptions, made for Africa, valid for the entire globe?
• Etc.
OVERVIEW of the WORK in the ASIS PROJECT
Of first priority: Run the ASIS with METOP on a global scale
• Develop all necessary software tools (GLIMPSE)
• Apply in back- and NRT processing modes
• Install at FAO
• Etc.
Other tasks:
• Include all ASIS-GLIMPSE modules in SPIRITS
• Adapt the global crop mask
• Set up a web service, at least a demo version, through which FAO can
rapidly view all data.
• Set up another service to deal with specific regioons of interest (ROI)
OVERVIEW (see Report)
1. Intro: Definitions & Overview of the work
2. Basic data: RS-images & Ancillary info
3. Basic tools: GLIMPSE, SPIRITS and others
4. Data processing
5. Results: Display & evaluation
6. Conclusions: Done’s & todo’s
Others: METOP simulation, cropmask,...
CONCLUSIONS: SUMMARY of MAJOR ACTIVITIES
1.
•
•
•
•
Software Development
Many new GLIMPSE modules (esp. related to phenology)
Inclusion in SPIRITS (V1.1.1)
BAT scripts for automated ASIS processing
Scenario files, esp. QnQ-files for the definition of the quicklooks.
2. Collection & adaptation of ancillary information
• GAUL maps: rastering & extraction of thinned versions for the QLKs
• JRC crop mask: adaptation & conversion to Area Fraction IMG (AFI)
3.
•
•
•
Acquisition & preparation of RS data
NOAA from STAR, METOP and VGT from VITO/MARS.
Simulation of METOP-1km from NOAA-16km for 1984-2006
Setup of a thinned version of the entire ASIS: ROI GLt at 51 km.
CONCLUSIONS: SUMMARY of MAJOR ACTIVITIES
4.
•
•
•
•
•
Processing
Phenological parameters from SPOT-VGT.
Fill the ASIS at 1km in back-processing mode (1984-2011)
Idem, now in all details and with NRT, but only for thinned GLt data set.
Analysis of results & comparison with other sources
First implementation at FAO
5.
•
•
•
Downstream applications
SPIRITS database with all RUM values
Demo version of web services to display all data
FAO: Prototype of derived service to deal with specific ROIs
6. Reports & Participation/contribution to conferences
7. Most important: Iterated improvement/repetition of all procedures
OVERVIEW (see Report)
1. Intro: Definitions & Overview of the work
2. Basic data: RS-images & Ancillary info
3. Basic tools: GLIMPSE, SPIRITS and others
4. Data processing
5. Results: Display & evaluation
6. Conclusions: Done’s & todo’s
Others:
METOP simulation,
cropmask,...
See presentation
by Roel Van
Hoolst
OVERVIEW (see Report)
1. Intro: Definitions & Overview of the work
2. Basic data: RS-images & Ancillary info
3. Basic tools: GLIMPSE, SPIRITS and others
4. Data processing
5. Results: Display & evaluation
6. Conclusions: Done’s & todo’s
Others: METOP simulation, cropmask,...
BASIC TOOLS: Glimpse, Spirits & others
Spirits
Glimpse
• DOS C-programs
• Command line
• Complex chains via scripts
(BAT, TCL, Python,…)
•
•
•
•
Windows based GUI
Glimpse programs & others
Database!
Simple chains via “User Tools”
GLIMPSE general principles
• IMAGE DATA
• Binary files with fixed extension *.IMG (X.IMG)
• Only 2D (one IMG-file = one image layer)  3D via “Metafiles”
• No leader/trailer bytes
• Only 4 datatypes allowed: Byte, Integer, Long, Float
• Image size = Ncol x Nrec x BPP (bytes-per-pixel)
• ANNOTATION
• Mainly via ENVI HDR-files (X.HDR)
• Also support for IDRISI DOC-files (X.DOC)
ENVI HDR-file
OTHER
CAT
SPATIAL
SPECTRAL
3D
TMP
(Grey=Added by GLIMPSE)
KEYWORD
Description = {…}
Comment = {…}
Program = {…}
Sensor type
Bands
Interleave
File type
Header offset
Data type
Byte order
Values = { Vname,
Vunit,
Vlo,
Vhi,
Vmin,
Vmax,
Vint,
Vslo }
DESCRIPTION
Textual info, general title
More textual info
Name of program, which generated this IMG (+ version between brackets)
E.g. SPOT-VGT, NOAA-AVHRR,… (only textual information)
Nr. of image layers (for GLIMPSE: normally Bands=1)
BSQ, BIL or BIP – Only for 3D-IMGs with Bands > 1
“ENVI Standard” for ordinal IMGs, “ENVI classification” for categorical IMGs
Number of leader bytes before the real image data
1=BYTE, 2=INTEGER, 3=LONG, 4=FLOAT
0=High-Endian, 1=Low-Endian (only if datatype > 1)
Name of physical variable Y
(e.g. reflectance, temperature, class,…)
Dimension of physical variable Y (e.g. %, °C, -, …)
Lowest digital value of significant range (values beyond Vlo/Vhi are flags)
Highest …
Lowest significant value which really occurs in this IMG
Highest …
NB: Vlo  Vmin  Vmax  Vhi
Intercept of linear scaling:
physical Y = Vint + Vslo x V
Slope …
NB: The scaling only applies to significant range
Classes
Class names = {…}
Class lookup = {…}
Flags = {…}
Samples
Lines
Map info = { Name,
Colm, Recm,
Xm, Ym,
X, Y }
Master = {…}
Date
Days
Nr. of classes, incl. unavoidable class 0. More correct: highest class_ID + 1
For each class, starting with 0: class name (avoid commas!)
For each class, starting with 0: R, G, B-values in range 0-255
For each flag: “V=meaning” with V=digital value (only textual info)
Number of IMG columns
Number of IMG records
Projection_Name
(= entry in file Map_proj.txt)
IMG Col/Rec co-ordinates of “Magic Point” (see figure 3.3)
Map X/Y co-ordinates of same “Magic Point”
X/Y pixel size in MAP-units
Same as Map Info; only for Pseudo-IMGs; original geo-referencing
YYYYMMDD: IMG registration date, or startdate for composite IMGs
Periodicity in days: 1, 10, 30,…; 0=unknown/irrelevant; -1=actual registration 15
Keyword VALUES = {Yname, Yunit, Vlo, Vhi, Vmin, Vmax, Vint, Vslo}
Scale: Meaning of the digital image values (V)
- Ordinal images: linear relation: Physisical variabele Y = Vint + Vslo.V
E.g. VGT-S1/10:
NDVI [-]
[-0.1 … +1.0] = -0.1 + 0.0040.V
Reflectance [-]
[ 0.0 … +1.0] = 0.0 + 0.0005.V
- Categorical images (classifications): V = ID-nr of class (a=0, b=1)
E.g. “legend" or “key"
1=forest, 2=cropland, 3=water, …
Range of Digital Values V
- Potential range: Defined by Datatype
DataType Bytes/Pixel
Min.Value
Max.Value
1. Byte
1
0
+255
2. Integer
2
-32 768
+32 768
3. Long
4
-2 147 483 648 +2 147 483 647
4. Float
4
-3.4E+38
+3.4E+38
NB: 99% of images are (or should be) Byte (most compact)
- Significant range: Only these V are meaningful, others are excluded  “flags”
(Vlo, Vhi)
The scaling only holds for potential range, not for flags!
- Observed range: Truly occurring (Vlo  Vmin  Vmax  Vhi)
(Vmin, Vmax)
No essential information, only used for visualisation.
Flags  Keyword FLAGS={V1=meaning1, V2=meaning2, …}
- Special values/codes, selected beyond potential range
- Used to indicate special cases: no data, clouds, water, …
16
METAFILES (*.mta, *.var)
• ASCII-files with names of IMGs belonging
together for particular analysis
• Alternative for 3D-images
• Two versions:
• ENVI:
*.MTA
• GLIMPSE *.VAR
ENVI META FILE
File :
D:\TESTS\clas\f1r.img
Bands: 1
Dims : 1-500,1-500
File :
D:\TESTS\clas\f1n.img
Bands: 1
Dims : 1-500,1-500
• Two programs to generate them:
• VARmakeP: for time series
• VARmakeI: for non-periodic IMGs
File :
D:\TESTS\clas\f1s.img
Bands: 1
Dims : 1-500,1-500
• Automatically generated by many modules
---Vu
---11
12
13
---CODE
---R1
N1
S1
----------------COMPLETE FILENAME
----------------d:\tests\clas\f1R
d:\tests\clas\f1N
d:\tests\clas\f1S
17
PROCESSING of TIME SERIES (like ASIS)
General image names (without extensions): P[date]S
P
= Prefix (may be blank, may include drive/path)
[date] = Date, according to one of twelve date formats
S
= Suffix (may be blank)
1
2
3
4
5
6
7
8
9
10
11
DATE
FORMAT
YYYYMMDD
YYMMDD
YYYYmDD
YYmDD
YYYYTT
YYTT
YYYYMM
YYMM
YYYYm
YYm
YYYY
12
YY
N
MINIMAL
PERIOD
Day
Dekad
Month
EXPLANATION of TERMS
 2049]
YYYY
= Year
[1950
YY
= Year
[50=1950  49=2049]
MM
= Month in year
[01=Jan.  12=Dec.]
m
= Month in year
[A=Jan.  L=Dec.]
TT
= Dekad in year [01
 36]
DD
= Day in month
 31]
[01
Year
18
UNIflags SYSTEM
General flagging system followed whenever possible
BYTE
INTEGER
MEANING / INTERPRETATION
0 – 250
0 – 32767
Significant range - scaling: Y = Vint + Vslo.V
251
-5
Missing value over land (data error, missing inputs)
252
-4
Cloud over land
253
-3
Snow/Ice over land
254
-2
Sea/Water
255
-1
Background (no information at all, mostly: no data)
SPx-FILES (ASCII)
Additional (and rather fixed) inputs for some programs:
• IMGscale

SPs
• SMOOTH

SPp
• PHENOdat

SPp
• IMG2RUM

Spu
 Generated manually (from example files) of via SPIRITS
19
GLIMPSE basic modules
Image resampling
INTERPOL
Smoothing of NDVI & BT4
SMOOTH & SMOOTHwk
Long-Term statistics per dekad
LTA
Differences (VCI, TCI, VHI)
DIFFER & VHI
Definition of SOS/EOS (Phenology)
PHENOdat
Temporal aggregation
PHENOcum
Progress in season
PHENOnow
Spatial aggregation  RUM-files
IMG2RUM
RUM-file  Image (flags are lost)
RUM2IMG
Restore flags
ASIflag
Program INTERPOL: Image resampling
Create resampled OUT-IMG, without changing the map projection.
Thinning (High to Low Resolution)
•
•
•
•
Densification (Low to High Resolution)
Usage in ASIS:
Conversion METOP-1km  STAR-16km.
Extraction of 1 or 16 km AFI (Area Fraction IMG) from 250m global cropmask.
Extraction of specific ROIs (Regions of Interest).
Many others…
Program SMOOTH: Smoothing of NDVI & BT4 profiles
FLAGGED & CLEANED NDVI profiles from METOP 2008-2010 (108 dekads).
The peaks (values > 250) are “flags”, i.e. detected clouds/snow/errors.
Program SMOOTH: Smoothing of NDVI & BT4 profiles
FLAGGED & CLEANED BT4 profiles from METOP 2008-2010 (108 dekads).
The peaks (values > 250) are “flags”, i.e. detected clouds/snow/errors.
Program SMOOTH: Smoothing of NDVI & BT4 profiles
METOP-NDVI
2009, May, Dekad 3
RAW
SMOOTHED
Red pixels are missing values, found by the snow/cloud detection algorithms (SM).
SMOOTH: Removes missing values and noise. It works for NDVI and for BT4!
Indispensible step, but it complicates the NRT processing!
NB: Smoothing algorithm of STAR is unknown  Raw NDVI/BT4 used.
Same VITO smoothing applied on STAR and METOP (consistency!)
But program SMOOTH works on 10-daily inputs ( weekly STAR)
Program SMOOTHwk: Smoothing of weekly profiles
STAR-NDVI, 1998
Raw, weekly  Smoothed, 10-daily
•
•
•
•
SMOOTHwk:
Weekly inputs (NOAA-STAR)
Same SWETS smoothing
Interpolation of S10 values
Centre of each dekad
First tested on simulated S1-profiles
 Correct S10 values known
 Best “interpolation” method
Methods to define S10-value
1. Interpol to dekad centre
2. Mean of closest week(s)
3. Maxval in closest week(s)
Programs LTA & DIFFER: Historical statistics & Differences
LTA:
Historical statistics (Min/Max/Mean…) per dekad.
DIFFER: VCI=(NDVI-NDVImin)/(NDVImax-NDVImin), TCI=(BT4-BT4min)/(BT4max-BT4min)
All applied on smoothed dekadal profiles of METOP and STAR.
Program VHI: Vegetation Health Index
Basic equation:
VHI = w.VCI + (1-w).TCI
Input parameters:
1. Name of VCI and TCI input images (computed with DIFFER)
2. Name of VHI output image.
3. How is TCI computed?
Kogan:
TCI1=(BT4max - BT4)/(BT4max - BT4min)
DIFFER:
TCI0=(BT4 - BT4min)/(BT4max - BT4min)
NB:
TCI1=1-TCI0
4. Definition of the “weights” w:
Constant for all pixels, e.g. w=0.5  VHI = (VCI + TCI )/2
Variable per pixel  Specify image with w-values
Notes:
• W=1  VHI=VCI, W=0  VHI=TCI
• Variable weights so far not used but can be useful later.
Program PHENOdat: Detect SOS/EOS for 1 or 2 cycles per year
GENERAL:
•
•
Input=3 civil years of smoothed dekadal VI-data (NDVI, fAPAR, LAI,…)  108 IMGs.
Target= central year  Outputs relate to this civil year
Composition of the three requested input years:
– BACK-proc.: All data available (e.g. for 2005, use all images of 2004-2006).
– LTA:
Input 36 dekads of LTA. Program duplicates them before/after.
– NRT:
Lacking/future dekads are replaced with corresponding LTA-images.
But outputs are restricted to current date.
Repeat procedure  As season progresses, SOS/EOS get more reliable.
•
•
•
•
Number of seasons allowed per year: 0 (no phenology), 1 or 2.
SOS/EOS defined for each season in target year
 Four basic OUT-IMGs per year (SOS1/EOS1, SOS2/EOS2)
SOS/EOS in dekads since start of series: 1-36=prev. year, 37-72=current, 73-108=next year.
All OUT-IMGs BYTE (8 bit):
–
–
•
Values 1-108:
Real dekads
Values 251-255: Flags
Program is based on a set of
modifiable thresholds (T1, T2, …).
Program PHENOdat: Detect SOS/EOS for 1 or 2 cycles per year
ANALYSIS PER PIXEL:
•
First flagging of exceptions in OUT-IMGs:
–
–
•
•
Smooth 3-years profile with running mean filter of length 5.
Second flagging of exceptions in OUT-IMGs (land pixels without phenology)
–
–
•
•
 OUT-flag = DESERT
 OUT-flag = EQUATORIAL FOREST
Slope of MIN-MAX (or MAX-MIN) < T3
If MAX < T4
If distance MAX1-MAX2 < T5
Etc.
 Remove both
 Remove MAX + highest MIN
 Remove MIN + lowest MAX
NB:
A season belongs to year n, if its MAX falls in that year.
Corresponding SOS may be in previous year, SOS in following year.
If two seasons  SOS/EOS1 refer to FIRST maximum, SOS/EOS2 to the LAST one.
Define SOS/EOS for the 1 or 2 seasons in central/target year:
–
–
•
All IN-values < T1
All IN-values > T2
Find all local maxima (seasons) & Prune them iteratively until at most 2 remain per year:
–
–
–
–
•
All IN-values flagged
 OUT-flag = SEA
Too much missing IN-values OUT-flag = MISSING
NB:
SOS = Last dekad, between MIN and MAX, when rising
profile cuts value: MIN + T6(MAX-MIN).
EOS = First dekad, between MAX and MIN, when descending profile cuts value: MIN + T7(MAX-MIN).
For ASIS, so far: T6=0.25 and T7=0.25.
T7=0.25 might compensate for the fact that we do not use EGF = EOS – 42 days.
Program PHENOdat: Detect SOS/EOS for 1 or 2 cycles per year
PHENOdat examples for NDVI of SPOT-VGT of 2003-2005 (108 dekads).
Red=original profile, Green=further smoothed with running mean of length 5.
White circles=original extremes, Red circles=retained extremes.
The white squares and purple blocks indicate the periods from SOS to EOS.
These are only searched for the target year 2004.
Program PHENOdat: Detect SOS/EOS for 1 or 2 cycles per year
SOS and EOS of the first season, from the long term NDVI averages of SPOT-VGT.
Program PHENOdat: Detect SOS/EOS for 1 or 2 cycles per year
SOS and EOS of the second season, from the long term NDVI averages of SPOT-VGT.
Program PHENOdat: Additional OUT-IMGs derived from SOS/EOS
Length in dekads (0-36) of the green seasons (0/1/2).
Seasonality Index SI = Ratio of mean NDVI’s in green seasons over total year.
Black=Low variation  Green  White=High variation
Program PHENOdat: Conclusions
• Results seem OK:
Comparison with other sources,
Tests on spatial & multi-annual consistency.
• The African ASIS (Rojas et al.):
–
–
–
–
First derive SOS/EOS for all years, per pixel.
Then define multi-annual averages, per pixel.
Finally define mean values of SOS/EOS per administrative region.
Use these (regionally and multi-annualy) fixed SOS/EOS for all further computations (µVHI,…)
• But SOS/EOS are “circular variables”  Computational problems
– Taking the simple (spatial/multi-annual) mean is wrong (mean of Jan/Dec=June-July).
– Spatial aggregation: mode > mean.
– Many standard programs, developed for normal cases, should be adapted!
• Problems avoided in new global version of ASIS:
– NB: Accounting for multi-annual variations would complicate matters and hamper the interannual comparisons, the heart of the system.
– SOS/EOS defined on LTA NDVI  more stable results  No multi-annual averaging needed.
– Compute essential ASIS info (µVHI, etc.) using pixel-based values of SOS/EOS.
– These ordinal variables can be spatially aggregated with standard programs (IMG2RUM).
– Hence, direct spatial aggregation of SOS/EOS is not needed.
• Remaining questions:
– Make 3 output sets: SOS/EOS11 (1 season), SOS/EOS21 (1st of 2 seasons) , SOS/EOS22 (2nd).
– Treat these groups separately, also in the database.
– For ASIS, so far: T6=0.25 and T7=0.25. Need to apply: EGF = EOS – 42 days ?
Program PHENOcum: Mean/sum of VI between SOS and EOS
Input parameters:
1. PHENOdat results for a given year (Can also be derived from LTA)
2. Series of dekadal images to process:
- Start: must be in year before target, best from January onwards
- End: BACK: December of year after target
NRT: Actual date (no replacement of future dekads with LTA)
- Missing values in centre of series are replaced by linear interpolation
3. OUT: Two IMGs (one per season) with mean/sum of values …
- Between SOS and EOS or actual date (whichever comes first)
- OUT-values can be flagged …
. If series contains too much missing values
. If progress of season falls below a certain percentage (“too early”).
- Two types of accumulation:
. MEAN: OUT-image has same datatype as IN-IMGs
. SUM: OUT-image always “floating point” (4 bytes/pixel)
Uses in ASIS: Pixel-specific computation of means of VHI, VCI,… µVHI, µVCI,…
Program PHENOcum: Mean/sum of VI between SOS and EOS
IN1:
IN2:
OUT:
SOS/EOS from NDVI-LTA of SPOT-VGT (computed by PHENOdat)
NDVI-images from January 2003 to December 2005 (target year=2004)
Mean value between SOS and EOS or June 2004.
Example of
BACK-processing
Mean NDVI
Season 1
No season
Too early
Missing NDVIs
Season 2
Program PHENOcum: Mean/sum of VI between SOS and EOS
IN1:
IN2:
OUT:
SOS/EOS from NDVI-LTA of SPOT-VGT (computed by PHENOdat)
NDVI-images from January 2003 to June 2004 (target year=2004)
Mean value between SOS and EOS or June 2004.
Example of
NRT-processing
Mean NDVI
Season 1
No season
Too early
Missing NDVIs
Season 2
Program PHENOnow: Progress of season(s) at current date
IN1:
IN2:
OUT:
SOS/EOS1 & SOS/EOS2 (PHENOdat). Here from NDVI-LTA of SPOT-VGT
Current date in target year. Here: end of June 2004
IMGs with relative progress (in %) of seasons 1 and 2, per pixel.
Example of
NRT-processing
Season 1
0-100%
No season
Season 2
Program IMG2RUM: Spatial aggregation 1 (ordinal variables)
Images  RUM-files (Regional Unmixed Means) [ Database]
Any IN-IMG
(e.g. µVHI)
Regions Map
(GAUL1)
Global Crop
Mask
RUM-FILE:
• ASCII-TXT
• Comma-separated values
• Each line contains MEAN value
of IN-IMG-values of crop pixels
in a given GAUL1-region
• Additional items: date, sensorID, variable-ID,…
Works for IN-IMGs with:
• ordinal variables (NDVI,…)
• categorival variables (“events”)
 Regional frequencies (ASI)
Program IMG2RUM: Spatial aggregation 2 (categorical var’s)
Images  RUM-files (Regional Unmixed Means) [ Database]
•
Any IN-IMG
•
•
Regions Map
(GAUL1)
Global Crop
Mask
•
•
OUTPUT:
Same type of RUM-files
as for IMG2RUM
Instead of regional MEANS,
the file contains regional
FREQUENCIES of a given
“event”.
If IN-IMG = classification 
Event = frequency of a
given class
If IN-IMG = ordinal 
Event = frequency of pixels with
values in a given range.
E.g. for ASI: µVHI < 35%
“Event” can be specified in SPUfile.
Program RUM2IMG: Reconvert RUM-values to IMG format
Regions Map
(GAUL1)
Any RUM-file
+ Selection of “case”: class,
variable, date, …
• Needed to visualize the ASI RUM-values via QL-generator.
• But in the conversion RUMIMG, all flags are lost!  program ASIflag
Program ASIflag: Flag an ASI image generated by RUM2IMG
OUTPUT IMG = Copy of INPUT IMG but with flags (251-255) added:
FLAG
255 = water
253 = no season
251=too early
254=masked
SOURCE
from land/sea mask (GLC2000)
SOS image
Mean VHI image
Cropland AFI (if area fraction = 0)
Latter masking is optional  Always two versions: masked or not.
GLIMPSE basic modules: review
Image resampling
INTERPOL
Smoothing of NDVI & BT4
SMOOTH & SMOOTHwk
Long-Term statistics per dekad
LTA
Differences (VCI, TCI, VHI)
DIFFER & VHI
Definition of SOS/EOS (Phenology)
PHENOdat
Temporal aggregation
PHENOcum
Progress in season
PHENOnow
Spatial aggregation  RUM-files
IMG2RUM
RUM-file  Image (flags are lost)
RUM2IMG
Restore flags
ASIflag
SPIRITS
•
All GLIMPSE modules were incorporated in SPIRITS
 V1.1.1 of February 2013.
•
•
Especially useful for data inspection and analysis:
•
Quicklook generator
•
RUM database & viewer
But de facto, ASIS is run as a “stand-alone tool”:
•
SPIRITS not used in the chains (except QLK-generator)
•
FAO can develop its own analysis tools, starting from the
RUM-files
 More on this in Chapter 5
SPIRITS QLK-GENERATOR: QuickLook Maps
Two steps:
1. SPIRITS interface to setup a “template” (QnQ-file) for the QLKgeneration of a specific map  region, variable, accessoires,...
2. Script SPIRITS_QL.BAT can be used (outside SPIRITS) to create
Quicklooks for a complete time series of such images.
SPIRITS RUM-Database
CLASSESSETS
PK
CLASSES
SENSORS
CLASSESSET_KEY
PK
CLASS_KEY
CLASSESSET_ID
NAME
SHORT
FK1
CLASSESSET_ID
CLASS_ID
NAME
SHORT
PK
SENSOR_KEY
SENSOR_ID
NAME
SHORT
VARS
PK
RUM_HEADERS
REGIONSSETS
PK
REG0
PK
RUM_KEY
FK1
FK5
FK6
REG0_KEY
CLASS_KEY
METHOD_ID
THRESHOLD
SENSOR_KEY
VAR_KEY
TYPE_ID
PERIODICITY_DAYS
VAR_ID
NAME
SHORT
METHODS
REGIONSSET_KEY
PK
REG0_KEY
NAME
SHORT
REGIONSSET_ID
FK1
REGIONSSET_ID
REG0_ID
NAME
SHORT
FK4
FK3
FK7
FK2
PK
METHOD_ID
NAME
SHORT
PERIODICITY
PK
RUM_KEY
DATE
MEAN
SD
RA1
RA2
PERIODICITY_DAYS
NAME
SHORT
TYPES
RUM_VALUES
PK,FK1
PK
VAR_KEY
PK
TYPE_ID
NAME
SHORT
Architecture of the RUM database in SPIRITS. In advance a number of ancillary tables must
be filled with the specifications of the allowed regions, classes, sensors, variables, unmixing
methods, periodicities and types (ACTUAL or HISTORICAL, for the latter there is only one
“year”). A “series” is defined as the combination of all these ancillary elements, for instance the
actual, dekadal METOP-NDVI of cropland in the Belgian province of Antwerp, unmixed with
UMmet=2. The core of the database is formed by the following two tables. RUM_HEADERS
contains the ID (PK=RUM_KEY) of all such “series” ever encountered. RUM_VALUES
contains their actual values (mean, standard deviation, relative areas RA1/RA2) for each
subsequent date.
SPIRITS:
RUM-Database & Viewer
APPROACH:
• All RUM-values stored in DB
• ca. 3400 GAUL-regions
• All variables: NDVI, BT4,
VCI, TCI, VHI, µVHI, progress,
ASI,…
• Separate for pixels with one
or two seasons
• Etc.
SPIRITS:
• DB can be > 10 GB
• Still very fast
• Wide range
analyses
of
graphical
OTHER SOFTWARE TOOLS
1. Commercial software
•
•
PyroBatchFTP Import ZIP-files from FAO-site (new METOP data)
WZUNZIP
Unzip these files
2. Auxiliary GLIMPSE executables
•
•
•
FILEsize
HDRadapt
RUMadapt
Give the size of a file (+ check HDR if it’s an IMG)
Adapt keyword(s) in a HDR
Adapts columns in a RUM-file (date, period, IDs)
3. Auxiliary BAT-scripts
•
•
•
FINDdat
Spirits_QL
F
All kinds of date functions
Make a PNG quicklook of a given image
Permits to jump directly between all ASIS-folders
Software installation at FAO: See report (§3.5)
OVERVIEW (see Report)
1. Intro: Definitions & Overview of the work
2. Basic data: RS-images & Ancillary info
3. Basic tools: GLIMPSE, SPIRITS and others
4. Data processing
5. Results: Display & evaluation
6. Conclusions: Done’s & todo’s
Others: METOP simulation, cropmask,...
DATA PROCESSING (Chapter 4)
Discussed so far:
1. General definition of the ASIS & Tasks
2. Basic inputs: EO-data & Ancillary info
3. Basic tools
Here: Combine all these to realise the ASIS
1. Data organisation, file names, scaling and flags, etc.
2. Phenology from SPOT-VGT
3. ASIS in back-processing mode for 1985-2011
4. ASIS in NRT-mode
5. Current status & history
PROCESSING: 1. Organisation, file names, etc.  See report §4.1
PROCESSING: 2. Phenologies from SPOT-VGT
Operations:
1. Acquire all global NDVI data from VITO/JRC-MARS
2. Smooth series from 1999 until 2011 (13 years)
3. Compute long-term averages of NDVI per dekad (see fig. below)
4. Apply PHENOdat on this LTA-NDVI series (see next slides)
5. Apply PHENOnow on results, for all dekads (see later)
PROCESSING: 2. Phenologies from SPOT-VGT
PROCESSING: 2. Phenologies from SPOT-VGT
PROCESSING: 2. Phenologies from SPOT-VGT
PHENOnow: Progress of the seasons
1. Fixed for all years, thus valid for any “target year”
2. Computed for dekads 1-72 (current + next year)
3. Analysis: Both seasons finished at dekad 18 of next year
PROCESSING:
3. Back-Processing 1985-2011
STEPS: always IMGs, QLKs & RUM-files
1. Data acquisition of EO-images (NDVI, BT4)
• METOP from VITO:
since 2007, 1km
• NOAA from STAR:
since 1984, 16 km
 Simulation of 1km METOP data for the past
2. Smoothing:
• Since 2007:
• 1984-2006:
Swets (back-processing!)
Done during simulation
3. Long-Term Min and Max of NDVI/BT4 per dekad
4. Differences:
• VCI
• TCI
• VHI
NB: Other definition (TCI1=1-TCI0)
PROCESSING:
3. Back-Processing 1985-2011
Examples of smoothed k-NDVI and t-BT4 for dekad 2 of July 2010
PROCESSING:
3. Back-Processing 1985-2011
Long-term minimum (top) and maximum (bottom) of smoothed
k-NDVI in dekad 18 (computed over the years 1984-2011)
Anomalies: VCI, TCI, VHI
PROCESSING:
3. Back-Processing 1985-2011
PROCESSING:
3. Back-Processing 1985-2011
STEPS:
5. Mean VHI µVHI and ASI per season (1/2) and per year/dekad
Definitions:
• µVHI:
Mean VHI from SOS till EOS or today
Program: PHENOcum
• ASI:
Fraction of crop pixels per region with µVHI<35%
Programs: IMG2RUM, RUM2IMG, ASIflag
Two modalities:
• µVHI,s,t and ASIs,t
• µVHI,s,y and ASIs,y
per dekad
per year
 NRT-mode
 Back-Processing
Always for two season (s=1/2).
NOTES:
• Analysis for both seasons stopped in dekad 18 of next year.
• Here: only back-processing
PROCESSING:
3. Back-Processing 1985-2011
Mean VHI for season 1 (top) and 2 (bottom) in the year 2010
(both seasons finished).
PROCESSING:
3. Back-Processing 1985-2011
ASIs,y for season s=1 in y=2010 (both seasons finished).
Bottom: non-cropland pixels have been masked.
PROCESSING:
4. NRT-Processing
Preliminary Remarks
1. Objectives:
• Development of set of scripts for automated processing at FAO
• Minimal human interaction
2. NRT actions at VITO:
• Every dekad, automated & quality checked as well
• Adapt METOP data (as available via JRC-MARS) to needs of ASIS
• Send all data in ZIP-form to FAO FTP site
3. Complications of NRT-mode (compared to back-processing):
• Every new dekad: Smooth & Anomalies must be recomputed for
last 6 dekades
• Now also dekadal µVHI,s,t and ASIs,t
 Much more work, computing time and disk space.
4. An additional “thinned” data set GLt was created:
• Systematic thinning: every 51th pixel from GLO in both directions
• Used for tests: much faster & still representative.
PROCESSING:
4. NRT-Processing
BAT scripts
1. AsiSET:
General settings (read by all following)
2. Individual Scripts:
• AsiIMP:
• AsiSMO:
• AsiDIF:
• AsiMUt:
• AsiASt:
• AsiMUy:
• AsiASy:
Import & Unzip of new METOP-dekad from VITO
Smoothing & backprocess last 6 dekads
VCI, TCI, VHI for last 6 dekads
dekadal µVHI,s,t
dekadal ASIs,t
annual µVHI,s,y
annual ASIs,y
3. AsiNRT:
All at once for one dekad
• Repeat via Window Scheduler  Entire process automated!
• But errors always possible
 Some monitoring needed
PROCESSING:
5. STATUS & HISTORY
2012 at VITO:
• Back-processing of archive (1984-2011)
• Sent to FAO on Hard Disks
Jan-Feb 2013: Installations at FAO:
• Server PC with 10 TB disk and connections: FTP and VPN
• All data copied Hard Disks to FAO 10 TB disk
Feb-Mar 2013 at VITO:
• Development of NRT scripts
• Modification of folder organisation, file names, additional IMGs,…
• Succesful testing of all scripts at VITO
• Partly applied on GLO (1km), partly on GLt (51 km)
Meeting at FAO (18-22 March 2013):
• Adapt all software and existing data at FAO & copy new ones
• Implement & test entire chain.
PROCESSING:
5. STATUS & HISTORY
This is a highly iterative process:
1. Develop/adapt program or chain
2. Run it on real or test data (GLO/GLt)
3. Inspect results & detect errors or define improvements
4. Back to 1.
OVERVIEW (see Report)
1. Intro: Definitions & Overview of the work
2. Basic data: RS-images & Ancillary info
3. Basic tools: GLIMPSE, SPIRITS and others
4. Data processing
5. Results: Display & evaluation
6. Conclusions: Done’s & todo’s
Others: METOP simulation, cropmask,...
Agricultural Stress Index System (ASIS)
based on remote sensing data
Interim meeting 2 (18-22 March 2013 at FAO):
13/07/2016
Software and others
Roel Van Hoolst, Herman Eerens, Antoine Royer
Dominique Haesen, Lieven Bydekerke
13/07/2016
© 2011, VITO NV
69