PUMA/AMESD Receiving Station
Thembani Moitlhobogi, AMESD SADC-THEMA
Pretoria, South Africa
26th February 2012
African Monitoring of the Environment for Sustainable Development
Training of trainers, Toulouse June 2010
Regional Training #2,Pretoria, February 2012
Satellite Data Reception Chain
EUMETCast dissemination system
•Files are transferred via a dedicated communications line from EUMETSAT to the
uplink facility. These files are encoded and transmitted to a geostationary
communications satellite for broadcast to user receiving stations.
Regional Training #2,Pretoria,
February 2012
2
Satellite Data Reception Chain
EUMETCast dissemination system
•Each service delivered via EUMETCast is assigned to a specific channel at the uplink provider
site. A typical channel allocation resembles the following:
– Announcement Channel - includes the user keys required for decryption of the
DVB
– EUMETSAT Data Channel 1 - a set of data files provided by EUMETSAT
– EUMETSAT Data Channel 12 – AMESD products e.t.c.
– External Data Provider Channel - a dedicated channel allocated to those data
provided by an external data provider
•Each channel, which has its own multicast address, is itself assigned to a Packet Identifier (PID)
Regional Training #2,Pretoria,
February 2012
3
Satellite Data Reception Chain
EUMETCast dissemination system
•The satellite used by the uplink providers to relay the EUMETCast
multicast over Africa is Eutelsat ATLANTIC BIRD™ 3 carries a
Ku/C-band payload of 45 transponders (35 Ku-band, 10 C-band)
with European African and western Asia coverage. Under the
current arrangements provided by the Africa turnaround provider,
EUMETCast is available via one of the C-band transponders.
Regional Training #2,Pretoria,
February 2012
4
Satellite Data Reception Chain
EUMETCast dissemination system
•EUMETCast - EUMETSAT’s Broadcast System for Environmental Data, is a
multi-service dissemination system based on standard Digital Video Broadcast
(DVB) technology. It uses commercial telecommunication geostationary satellites
to multicast files (data and products) to a wide user community
Regional Training #2,Pretoria,
February 2012
5
AMESD Satellite Data
Reception Chain
• Each full AMESD station is composed of an
outdoor and an indoor sub-systems as shown
:
Regional Training #2,Pretoria,
February 2012
6
PUMA Satellite Data
Reception Chain
LAN
8 ports switch
Dual screen
video card
EumetCast Key Unit
Existing C-band
outdoor
subsystem
Synergie license key
Spare
DVB card
DVB card
UPS
Puma2 :
Processing
Puma1 :
Acquisition
Puma3 :
Visualization
Linux CentOS
Linux CentOS
Linux CentOS
Driver DVB
Driver DVB
Driver DVB
TelliCast
TelliCast
TelliCast
FRAC
FRAC
FRAC
Archipel PUMA
chain
Archipel PUMA
chain
Archipel PUMA
chain
Synergie
server
Synergie
server
Synergie
server
Synergie client
Synergie client
Synergie client
SYNERGIE-PUMA nominal architecture
Legend
Synergie
server
Linux
CentOS
FRAC
Active modules
Regional Training #2,Pretoria,
February 2012
FRAC
Installed but not
active module
7
Acquisition System
Architecture
Regional Training #2,Pretoria,
February 2012
8
Acquisition System
Architecture
1. Tellicast acquires raw data and puts them down in dedicated directories
2. The FRAC pre-processes data and distributes satellite data to Archipel by FTP
(archipel User)
3. The FRAC disseminates meteorological data to Synergie server by FTP
(pumadata User)
4. Archipel processes satellite data and disseminates satellite products to Synergie
server by FTP (pumadata user)
5. The FRAC duplicates data and make them available for an external system by
FTP or SAMBA
6. Archive data requested on UMARF can be manually put down in the FRAC
Regional Training #2,Pretoria,
February 2012
9
Acquisition System
Architecture
1. Raw
data
TelliCast
Local file
Raw data
acquisition
UMARF
External
system
7. Archive
8. RGB definitions
Local file
SAMBA
FRAC
Raw data
pre-processing
and dispatching
2.
Satellite
data
Archipel
4.
Satellite
product
FTP
FTP
Satellite image
production
Synergie
server
5.
Internal
data
SSH
Synergie
client
Visualization
Database
FTP
6. Duplicated data
8. User’s data (e.g.: screenshot)
3. Meteorological data
FTP or SAMBA
SAMBA
External
system
External
system
SYNERGIE-PUMA general dataflow
Legend
Synergie
server
FRAC
Regional Training #2,Pretoria,
February 2012
Puma
modules
UMARF
External
system
FTP
4.
Satellite
data
Data flow
protocol
10
Data type
Acquisition Software
Structure
Tellicast
FRAC
Directory
/space/tellicast/
/usr/local/bin/
/home/ebs/recv_database/
Directory
/home/ebs/
/space/efts/
/data/data_share/
/space/decomp/
/space/trs/FMSG/
Directory
Archipel
Synergie
Regional Training #2,Pretoria,
February 2012
Description
Root directory
Contains the executable file
Contains the file database: temporary store for the
received file fragments until a file is completely
received
Description
Distribution and FTP push
Workspace buffer
Data duplication
Decompression
Cycles assembling
Description
/archipel/
/soft/
Contains the Archipel chain directory structure
Contains generic software components
Directory
/data/data_syn
/home/synergie
Description
Synergie data
Synergie server programs and configuration files
Directory
/home/synergie
/home/syndocs
Description
Synergie server programs and configuration files
Synergie end-user’s documents (screenshot, docs, etc)
11
Services status
• As root user, type the command:
puma-services-status.sh
The active and inactive services will be displayed.
• On PC1: only TelliCast and FRAC services must be active
• On PC2: only Archipel service must be active
• On PC3: only Synergie service must be active
Regional Training #2,Pretoria,
February 2012
12
EUMETCAST DATA
FLOW
•
When data is received from EUMETCast, it is dispatched to the appropriate
directories according to the Frac configuration
• The data reception and decoding is performed by two main services: dvbeumetcast and tellicast-client. These services are started automatically when the
Acquisition PC is powered On.
• Data received from EUMETCast are decoded and re-assembled by Tellicast
• The Tellicast incoming directory is the following:
/space/tellicast/data/received/default/
• After 2 hours, data that were not dispatched are deleted from this directory.
Regional Training #2,Pretoria,
February 2012
13
Data dispatch
•
•
The data dispatch is performed on the files received in Tellicast incoming directory.
It is based on EFTS software developed by EUMETCast, using the EFTS-PreProcessor
agent, started automatically when Acquisition is powered On.
•
The configuration of this agent is based on configuration file, which can be found in the
following directory:
/home/ebs/EFTS/DataUnits/
•
This file specifies the Housekeeping durations, for both Tellicast Incoming directory and
target directories where the files are dispatched.
•
As explained earlier, the default Housekeeping value for Tellicast Incoming directory is set to
2 hours. It is advised not to change this value, because increasing it will induce unnecessary
CPU Load on Acquisition PC.
•
The default Housekeeping value for target directory is set to 24 hours which is, as it will be
described later on in Archive section, enough to ensure all dispatched data are archived.
Regional Training #2,Pretoria,
February 2012
14
EFTS PreProcessor
•
•
•
•
•
The EFTS PreProcessor configuration file is called pumaEFTS_PreProcessor_EUMETCast.props, which is actually a symbolic link to the
file used for either AMESD or PUMA stations.
The rational of this is to provide homogeneous architecture between both AMESD
and PUMA Upgrade stations, so that e-station can be plugged into either one.
However, on AMESD Acquisition PC, EFTS PreProcessor is optimized to keep
only the data that will be processed and exploited by the e-stations.
Default list of these data include:
LAND-SAF products;
– VITO products;
– MSG products;
– MPE products.
Regional Training #2,Pretoria,
February 2012
15
Shared data
• As explained earlier, there are two shared
directories that provide received data to the estation:
• For all data products, the /data/data_share/
directory is accessible from e-stations, under
the directory name rawdatabase;
• For e-station software packages, the
/data/AMESD_updates is accessible from estations, under the directory name AMESDupdates;
Regional Training #2,Pretoria,
February 2012
16
Archives
• The archiving of EUMETCast data for AMESD includes two
steps:
• Move all files from /data/data_share/ to a dedicated directory
tree, with a main directory for each year, and a sub-directory
for each day of the year; for example, the archive for the 25th
of May 2010 would be in /data/AMESD_archives/2010/145/;
• Verification of hard disk usage: when it is higher than 60%
(configurable) of hard disk capacity, the oldest archive
directories are automatically deleted.
• The configuration of the percentage of use of hard disk is
configurable in the following file:
• /data/AMESD_archive/config_backup.cfg
• The parameter to change is: percentage_disk; it is
recommended to set a value lower than 80.
Regional Training #2,Pretoria,
February 2012
17
Setting AMESD/PUMA receiving station to
process AMESD SADC products
Regional Training #2,Pretoria,
February 2012
18
Receiving AMESD products
To set the receiving station to keep AMESD products the
following changes have to be made:
• Request permission to receive AMESD SADC products from
EUMETSAT
• Activate data archiving
• Add trigger to keep AMESD products on the FRAC EFTS
service configuration
Regional Training #2,Pretoria,
February 2012
19
Accessing AMESD products
The received products are archived for 24 hours in the directory
/data/data_share
To set the receiving station to transfer the archived data to
another computer the following changes have to be made:
• Connect PUMA station to the Local Area Network (or connect
the computer to the PUMA station private network)
• Share a directory on the computer and allow read/write
permissions to everyone
• Configure DataTransfer.sh script and add a schedule to run
the script every 5 minutes
Regional Training #2,Pretoria,
February 2012
20
Activating data archiving
To set the receiving station to keep AMESD products the
following changes have to be made:
• Open the terminal and change user to ebs by typing the
following command: su ebs
• Check the archive mode status by command:
/home/ebs/EFTS/bin/efts_mode status
• Change archive mode to ON by command:
/home/ebs/EFTS/bin/efts_mode SHARE
• Exit ebs user and return to root by typing command: exit
• Restart the EFTS agent service by typing command:
service efts_EUMETCast-agents restart
• You may close the terminal
Regional Training #2,Pretoria,
February 2012
21
Add AMESD trigger
Open the configuration file /home/ebs/EFTS/DataUnits/pumaEFTS_PreProcessor_EUMETCast.props by following the steps below:
• Open Places > Computer > FileSystem > home > ebs >
EFTS > DataUnits
• Create a backup of the pumaEFTS_PreProcessor_EUMETCast.props file by copying and
pasting it in the same directory
• Double-click on the file to open it and scroll to the bottom and
add a trigger as shown below, ensure the trigger number
follows the existing sequence
Regional Training #2,Pretoria,
February 2012
22
Add AMESD Trigger
#------------------------------------------------------------------------------------------# P31
#
Affected files : AMESD products
#
Input directory: tellicast receiving directory
#
Output directory: shared directory
#
#------------------------------------------------------------------------------------------EFTS.PreProcessor.Incoming31.Dir= /space/tellicast/data/received/default/
EFTS.PreProcessor.Incoming31.FileMatchRegexp=^AMESD_SADC.*$
EFTS.PreProcessor.Incoming31.FileExcludeRegexp=^IGNORE.*$
EFTS.PreProcessor.Incoming31.CopyTo1=/data/data_share/
EFTS.PreProcessor.Incoming31.IgnoreSuffix=.TEMP
Regional Training #2,Pretoria,
February 2012
23
Add AMESD Trigger
• Save the puma-EFTS_PreProcessor_EUMETCast.props file
• Open the terminal and restart the EFTS processor service by
typing command:
service efts_EUMETCast-PreProcessor restart
Regional Training #2,Pretoria,
February 2012
24
Accessing AMESD products
Create a directory where the shared directory of the client
computer will be mounted by following steps below:
•
•
•
•
Go to Places > Home
Copy the DataTransfer.sh script to the Home directory
Create the mount directory, e.g. you may call it “AMESD”
Open the DataTransfer.sh script and edit it to the
configuration of your client computer
• Put the DataTransfer.sh script on a schedule to run every 5
minutes
Regional Training #2,Pretoria,
February 2012
25
Edit DataTransfer.sh
# define input and output parameters
share_ip=10.226.1.131
# ip address of client
share_user=amesd
# user name of client
share_pass=amesd
# password of client
shareddir=//${share_ip}/AMESD
# shared directory
localdir=/root/AMESD
# local mount directory
client_data_dir=${localdir}/data
# directory in share where
# data will be dumped
Regional Training #2,Pretoria,
February 2012
26
Edit DataTransfer.sh
# if client computer has been successfully mounted move the data there
if [ -d ${client_data_dir} ]; then
for data in `find /data/data_share/ -iname AMESD_SADC*`; do
mv -v ${data} ${client_data_dir}/
done
for data in `find /data/data_share/ -iname VGT* -or -iname *VGT*`; do
mv -v ${data} ${client_data_dir}/
done
for data in `find /data/data_share/ -iname *rfe* -or -iname rfe*`; do
mv -v ${data} ${client_data_dir}/
done
for data in `find /data/data_share/ -iname *NDVI*`; do
mv -v ${data} ${client_data_dir}/
done
fi
}
Regional Training #2,Pretoria,
February 2012
27
Scheduling DataTransfer.sh
Open terminal
• Open terminal
• Open scheduler by typing command: crontab –e
• Add the following line at the bottom of the crontab window:
*/5 * * * * /root/DataTransfer.sh >/dev/null 2>&1
• Save and exit crontab
Regional Training #2,Pretoria,
February 2012
28
Regional Training #2,Pretoria,
February 2012
29