1
ERA Monitoring (and Control) System using Yamcs
ASTRA 2013
Nicolae Mihalache
2
ERA Support Center
•
The ERA Support Center (ERA SC) is hosted in the Erasmus building in ESTEC and is
responsible for the following functions
– Training of Russians instructors and engineers
– Monitoring of ERA operations in real-time with video, voice and data
– Analysis of flight data
– Support troubleshooting for issues during flight
– Support to mission planning, operations development, simulation and rehearsal of
procedures
– Provide maintenance of the ERA flight software
– Provide additional sustaining engineering support as needed
ERA MCS using Yamcs
ASTRA-2013
3
ERA TM Transmission
ERA SC
Internet
Dutch Space
IGS
M CC-M
Col-CC
IGS
DaSS server
ERA MCS using Yamcs
ASTRA-2013
4
Software needed for ERA TM Processing
MCC-M
emulation
displays
ERA TM Server
Col-CC DaSS
•CCSDS packet
processing
•Archiving
OLMS
MS-IMMI
Enhanced
displays
Col-CC: Columbus Control Center (the main European node for ISS data)
DaSS: Data Services System (protocol used to send data through Col-CC)
OLMS: On-line Monitoring System (legacy ERA monitoring software)
MS-IMMI: another legacy ERA 3D monitoring software
The green boxes have to be populated with actual software.
ERA MCS using Yamcs
ASTRA-2013
5
Software needed for ERA TM Processing
•
•
The required software needs to have the following functionality:
– connect via DaSS to receive the processed and path TM data from Col-CC.
– extract the ERA TM frames from the CCSDS packets and relay them to OLMS
clients executing inside VMWare virtual machines on user workstations.
– allow displays to be implemented emulating the alphanumeric displays on the
universal workplaces used in MCC-M
– archive all TM parameters and packets.
– support realtime and replay data at the same time.
– relay parameters/packets to external clients at Dutch Space via Internet/VPN.
– satisfy ESA security architecture separating the ERA SC network into two
distinctive LANs
The chosen software:
– Yamcs for the TM reception and processing
– USS for the TM display
ERA MCS using Yamcs
ASTRA-2013
6
Why Yet Another Mission Control System(1)
•
•
•
•
•
The Columbus Mission Control System used in Col-CC is built around the
Astrium CGS software which was the Columbus EGSE
ESA has decided to build a “stripped down” version of the Columbus MCS
and distribute it to all the USOCs, calling it CD-MCS (Columbus Decentralized
MCS).
None of the payloads which the USOCs are operating have been developed
using the CGS software, thus the CD-MCS is not particularly suited (nor is it
particularly unsuited) for the operations of these payloads.
Yamcs has been gradually developed as a set of tools needed to complement
CD-MCS functionality, in order to support operations in the B.USOC and the
Erasmus USOC.
There was by no means a grand plan to build the ultimate MCS, rather the
system has evolved incrementally (but we are not there yet).
ERA MCS using Yamcs
DASIA-2011
7
Why Yet Another Mission Control System(2)
The main needs that drove the development of Yamcs are:
• Multiple parallel commanding/monitoring chains
• Complicated data structures and encodings used by some instruments.
• Need for special telemetry processing that goes beyond what can be
achieved with a template based definition like XTCE.
• Need to comply to strict security rules when offering access to
telemetry/telecommand to remote users.
• Although the CD-MCS does contain an archiving system (called HRDP), it
does not offer the possibility to easily inspect/retrieve data from the archive
and to assess the archive completeness.
ERA MCS using Yamcs
DASIA-2011
8
Yamcs features
•
•
•
•
•
•
•
•
Acquisition of telemetry and dispatching of commands using different
protocols: CD-MCS multicast, raw packets over TCP, DaSS (Columbus
Ground Segment specific protocol) and others.
Comfortable packet structure definition, partially implementing the XTCE
standard: www.xtce.org
Archive completeness.
Compatible with the Unified Synoptic System.
Detailed packet and parameter inspection.
Remote and cascaded connectivity from external science users or other
centers.
Username/password or SSL (certificate) based authentication.
Authorization at parameter, packet, and command level.
ERA MCS using Yamcs
DASIA-2011
9
Technologies used inside Yamcs
•
•
•
•
•
•
The HornetQ messaging bus is used for all the inter process communications. (To
maintain compatibility with the old CD-MCS based applications, there is also a module
offering a CORBA interface).
The messages are serialized using (Google) Protocol Buffers.
A data archive for stream based data (TM/TC packets, events) is implemented on top of
an embedded key-value datastore called Tokyo Cabinets. Same datastore is used to
implement a real-time archive index (used to asses the archive completeness in realtime).
Previously this was implemented in a PostgreSQL database but it was limiting the
performance.
A parameter archive is being implemented on top of a column oriented embedded
database called FastBit.
An experimental Complex Event Processing (or Stream Based Processing) is being
studied to validate the use of such technology inside Control Centers.
Web-based data access is provided by using VAADIN Java framework.
ERA MCS using Yamcs
27-July-2012
10
Middleware selection: RPC versus Messaging
•
•
The External (across Internet) connections to Yamcs (for compatibility with CD-MCS)
were performed using CORBA, running through a VPN tunnel and proxied by a
CORBA proxy.
We found this configuration difficult to work with, for several reasons:
– Firewall unfriendliness of CORBA: by default it requires opening ports on the client
side
– Bi-Directional IIOP (Internet Inter-Orb Protocol) somehow helps but then the
resulting multiplexing of connections on one TCP socket, causes problems with the
CORBA proxy in between.
– CORBA specifications related to security is CSIv2 (Common Secure
Interoperability Protocol Version 2). Unfortunately the specification is very complex
(400+ pages long adding to 1100+ pages of CORBA specification).
– Even worse, the support of CSIv2 in the CORBA implementation used by Yamcs,
is relatively poor, requiring a few additions as custom developments
ERA MCS using Yamcs
27-July-2012
11
Middleware selection: RPC versus Messaging(2)
•
•
•
Why dropping CORBA even for internal communication? In the current MCSs we are
most familiar with (SCOS2K, CD-MCS, OCTAVE from CNES), CORBA is very much
the standard.
We have identified three communication patterns often used inside a MCS:
– Request-Reply: typical pattern used when sending a command (telecomand,
control command for some service, etc).
– Publish-Subscribe: typical pattern used when retrieving realtime data
– Large Data Transfer: used when retrieving data in dump (bulk) mode.
Only the first pattern is implemented properly by the original CORBA, and even then
there are problems with handling network failures.
ERA MCS using Yamcs
27-July-2012
12
Traditional SQL Databases versus
Key-Value Datastores
•
•
•
•
•
SCOS-2000 has moved from an embedded database (C-tree) based storage to a
standard network Database based storage (MySQL or Oracle). OCTAVE seems also to
use a traditional SQL database to store the data.
One cannot miss noticing the recent movement in the open-source community called
”NoSQL” that promotes different kinds of databases optimized especially for high
volume of data and/or low latency.
In an MCS there are two types of somehow conflicting utilization scenarios for the
archive retrieval:
– retrieval from a time interval - this is used for data replay to displays or for data
extraction for post processing;
– Searching for individual parameters values (usually the engineering values).
The first one calls for storing the entire data packets or parameters in time ordered
manner
The second one calls for storing a reversed view which maps parameter values to time
when they have been generated (in traditional database terms indexing on all
parameters).
ERA MCS using Yamcs
27-July-2012
13
Key-Value Datastore for “Packet” archiving
•
•
•
•
•
For the stream based data(TM, TC, Events) a key-value datastore that sorts based on
the key is the obvious choice. The key can be chosen such to achieve proper ordering
and uniqueness.
For Yamcs we have selected the product called Tokyo Cabinet
(http://fallabs.com/tokyocabinet/) for the stream based data.
A “table” layer is built on top of it that allows one to store columns like in a traditional
database.
Each table requires a fixed number of columns that is used as a key (these have to be
present in each row) and a variable number of columns that is used as value.
Alternatives to TokyoCabinets:
– Kyoto Cabinets (immediate successor to TC, more portable)
– LevelDb (from Google)
– Oracle NoSQL Database (TBC)
– Probably many others
ERA MCS using Yamcs
27-July-2012
14
Column Oriented Database for “Parameter” archive
•
•
•
•
•
Common wisdom (e.g. SCOS2k) to reduce archiving space and improve retrieval speed
is to store change-only TM data in triplets like (parameter id, time, value).
This breaks the relationship inherent in parameters coming out of the same packet,
duplicating the timestamp for these parameters.
We believe that better efficiency can be obtained by using a so called column oriented
database (as opposed to row oriented databases):
– Data is logically viewed as “tables” (time, parameter1, parameter2, …)
– Each column (including the time) is stored physically as a single block
– Because all data of one column is of the same type, it can compress much better,
and can use bitmap and run length encoding algorithms which do not even require
a decompression to read the data.
The implementation of parameter database in Yamcs is not finished but is based
partially on Fastbit (http://crd.lbl.gov/~kewu/fastbit/).
Alternatives to Fastbit: unfortunately not many (LucidDB is one of them but is relatively
heavy).
ERA MCS using Yamcs
27-July-2012
15
(Google) Protocol Buffers
•
Data encoding on wire is performed using Protocol Buffers:
– Supported by most popular programming languages.
– Binary encoded – low overhead (compared to XML).
– Extensible (unlike CORBA IDL).
message ParameterValue {
optional NamedObjectId id=1;
optional RawValue rawValue=2;
optional EngineeringValue engValue=3;
optional int64 acquisitionTime=4;
optional int64 generationTime=5;
optional AcquisitionStatus acquisitionStatus=6;
optional bool processingStatus=7;
optional MonitoringResult monitoringResult=8;
}
ERA MCS using Yamcs
27-July-2012
16
YAML Ain't Markup Language
•
•
•
Human readable version of XML.
Supports data types common to most high-level languages: list, associative array
(maps), and scalar.
Used in Yamcs to encode all the configuration data (replacing the java properties files).
ERA MCS using Yamcs
27-July-2012
17
Yamcs archive browser and packet viewer
Two of the yamcs client tools are presented below:
• Archive Browser
• Packet Viewer
ERA MCS using Yamcs
DASIA-2011
18
Architecture Overview
ERA MCS using Yamcs
27-July-2012
19
Architecture overview
•
•
•
•
•
•
Everything is organized around a message bus. The message bus contains multiple
independent data streams which are part of a more generic “product” called Yarch.
Multiple concurrent instances of the message bus are supported.
Currently instances defined at B.USOC: yops, ysim, estm.
The old yamcs server has been split into components connected to the bus:
– TM receivers – injects telemetry packets and processed parameters into the bus
– TC uplinkers – receives commands from the bus and sends them to various
destinations
– TM/TC processor (this was the old channel) encodes/decodes TM/TC
– CORBA adapter – provides connection to USS and MCS Tools.
Additional components have been added for TM, PP, Cmd History and Event recording
Future parameter archive may also be implemented.
ERA MCS using Yamcs
27-July-2012
20
Event Stream Processing engine - Yarch
Producer 1
Producer 2
t1, t2
Stream
t1 t2 t3 …
t 1 t3 …
Consumer
t3
t3 t2 t1
•
•
•
Filter stream (where x=3)
table
Stream: a (theoretical infinite) ordered sequence of tuples
Tuple: a sequence of (name,value) pairs.
Table: an ordered sequence of tuples.
ERA MCS using Yamcs
27-July-2012
21
Yarch tables
•
•
A table in yarch is mapped into a Tokyo Cabinets B-Tree (sorted) table.
The definition resembles tables in relational database:
create table X (col1 type, col2 type…
primary key(col1,…))
<options>
•
•
•
•
•
The mapping between the Yarch table and the Tokyo Cabinets B-Tree table:
– Yarch primary key columns becomes the Tokyo Cabinets key
– All the other columns are encoded into the Tokyo Cabinet value
Tables schemas are dynamic: one can add more columns than initially defined.
The primary key columns have always to be present but the other can miss.
Each column which is not part of primary key is given an id.
The id is encoded together with the column -> the table can be sparse.
ERA MCS using Yamcs
27-July-2012
22
Yarch tables
•
•
The following types are supported: short, int, float, double, string, enum, ProtoBuf
– enum is like string just that instead encoding the value, it encodes an index. We do
this to have the possibility to change the string value without touching the table
data. In this case pname is the packet name.
– ProtoBuf is a Protocol Buffer message encoded in binary (note that due to the
compressed encoding of Protocol Buffers, these messages are not suitable to be
used as keys).
Options:
– Histogram(colname): means that Yarch will keep statistics for each value of the
colname.
– Partitioning by time, value or both time and value: means that Yarch will create
one file for each day for each value.
– table_format=compressed means that Tokyo Cabinets will compress all the values
on disk.
ERA MCS using Yamcs
27-July-2012
23
Yarch streams
•
•
•
•
•
The idea behind event stream processing (called also CEP: complex event processing)
is to enable rapid development of stream analyzers.
This could be done in a declarative way using a sql like language, or using a graphical
design language.
Concepts in CEP are:
– continous queries: they are executed as soon as a new tuple comes into the
stream
– moving windows: for example to be able to run continuous queries on the last 10
minutes of data
– event-pattern detection: correlate between events in different streams.
Think for example of automatic checking of a switch on instrument sequence.
Yarch implements some rudimentary event stream processing features: simple filters,
merging ordered streams, simple window aggregation.
ERA MCS using Yamcs
27-July-2012
24
XTCE Packet Processing
•
•
•
•
The Yamcs Server implements a subset of the XTCE (XML Telemetric and Command
Exchange) for telemetry processing.
Sequence Containers: Sequence containers are the equivalent of packets in the usual
terminology, or ADU in the MDB terminology. The sequence containers employ two
mechanism to avoid the limitation of traditional "packet with parameters" approach:
Container aggregation. A sequence container contains sequence entries which can
be of two types:
– Parameter Entries - these point to normal parameters.
– Container Entries - these point to other containers which are then included in the
big container
Container inheritance. Sequence containers can point to another sequence container
through the baseContainer property, meaning that the baseContainer is extended with
additional sequence entries. The inheritance is based on a condition put on the
parameters from the baseContainer (e.g. a SOLAR_HK packet is a CCSDS packet
which has the apid=991 and the packetid=0x….).
ERA MCS using Yamcs
27-July-2012
25
XTCE Packet Processor
CCSDS
CCSDS_PACKETID==318813001
SOLAR_Tlm_Pkt_HK
CCSDS_SH_PACKETTYPE==12
SOLAR_Tlm_Pkt_Event
CCSDS_RESPONSE_PACKET
PacketType==1
MeasurementsTable
ERA MCS using Yamcs
ParametersTable1
27-July-2012
26
TM Recorder
•
The telemetry table definition:
create table tm (gentime timestamp,
seqNum int,
rectime timestamp,
packet binary,
pname enum,
primary key(gentime, seqNum)
)
histogram(pname)
partition by time_and_value(gentime, pname)
table_format=compressed
•
•
•
•
gentime is the CCSDS generation time converted to Yamcs internal representation.
seqNum is the CCSDS sequence count
packet is the complete binary packet
pname is the packet name
ERA MCS using Yamcs
27-July-2012
27
TM Recorder
•
•
•
•
•
histogram(pname): means that Yarch will keep statistics per packet name: this is the
archive index seen in the Archive Browser.
partition by time_and_value(gentime,pname) means that Yarch will create
one file for each day for each packet name.
table_format=compressed means that Tokyo Cabinets will compress all the values
on disk.
The partitioning on pname allows to trade-off performance between retrieving the
complete stream or retrieving individual packets.
pname is the packet name as extracted from MDB
–
–
–
–
currently the name used is at the level 1 of the XTCE hierarchy.
If it doesn’t match any container at level 1, then it is saved as the root container
only the first match is used.
in the future this should be configurable: the XTCE data structure will contain some
markers to configure the level of recording.
ERA MCS using Yamcs
27-July-2012
28
PP recorder
Structure of the pp table:
create table pp (gentime timestamp,
ppgroup enum,
seqNum int,
rectime timestamp,
umi_0xABCDEF PROTOBUF,
primary key(gentime, ppgroup)
)
histogram(ppgroup)
partition by time_and_value(gentime, ppgroup)
table_format=compressed
•
•
•
The actual values are not defined in the table but are dynamically added.
One PP tuple consists of all the parameters having the same timestamp and the same
group name.
The group name is assigned based on the OpsName of the parameter in the MDB.
• E.g. PDU2_ISPR_F3_Main_Pwr_Bus_SSPC1_On_Off_Stat_DMC_PPgoes into
the group “PDU2”.
ERA MCS using Yamcs
27-July-2012
29
Event recorder
Structure of the events table:
create table events (gentime timestamp,
source enum,
seqNum int,
body PROTOBUF('org.yamcs.gpb.Yamcs$Event’),
primary key(gentime, source, seqNum))
partition by time(gentime) table_format=compressed
The body represents a ProtoBuf message Event :
message Event {
required string source
= 1;
required int64 generationTime = 2;
required int64 receptionTime
= 3;
required int32 seqNumber
= 4;
optional string type
= 5;
required string message
= 6;
enum EventSeverity {INFO=0; WARNING=1; ERROR=2;};
optional EventSeverity severity
= 7[default=INFO];
}
ERA MCS using Yamcs
27-July-2012
30
Telemetry Providers
•
•
•
•
FilePollingTmProvider
– Polls regularly a directory and reads all the telemetry packets it finds.
– Supports PaCTS files, HRDP dat files or files containing raw CCSDS data.
TcpTmProvider
– Receives raw CCSDS packets over TCP.
– Can be used for connection directly to Col-Emu or the ESTM.
MulticastTmProvider
– Listens to a multicast socket for telemetry CCSDS packets.
– This was the most non intrusive way to receive data from CD-MCS.
DassPacketProvider
– Connects to DaSS for retrieving CCSDS packets.
– The request (APID, private header source) can either be specified in the
configuration file or retrieved from the MDB.
ERA MCS using Yamcs
27-July-2012
31
Histograms and Completeness Index
•
•
•
•
•
The initial idea behind the archive browser was to have an overview of the available
data in the archive. We implemented an algorithm to build records of the shape:
(packet name, start, stop, count)
Some heuristics were used to detect when the data was “interrupted” and, so create a
new record.
The index was also used to quickly find data in the HRDP archive which otherwise
contains no index and requires scanning raw data files.
After Columbus launch it turned out that we have lots of problems with packet loss and
the archive index was transformed into an completeness index. The heuristics were
eliminated and instead the (APID, seqCount) was used to compute the index records.
The records were then subdivided per packet name.
Trying to generalize the completeness index concept turned out be very difficult:
– generic histograms have been introduced and will provide the necessary overview
of the data available in the archive (with no formal indication of data missing)
– the completeness index is still there but simplified (only APID, no packet name).
ERA MCS using Yamcs
27-July-2012
32
Backup slides
ERA MCS using Yamcs
xx-mmm-yyyy
33
Cmd History Recorder
Command History table definition:
create table cmdhist (gentime timestamp,
origin enum,
seqNum int,
primary key(gentime, origin, seqNum)
)
partition by time(gentime)
table_format=compressed
•
•
•
•
gentime: command generation time
origin: workstation or IP address of the sender
seqNum: some internally generated sequence count
All the other columns are created on the fly from the actual values in the command
history.
ERA MCS using Yamcs
27-July-2012
34
Telecommand Uplinkers
•
•
•
org.yamcs.tctm.TcpTcUplinker
– Sends raw CCSDS command packets via TCP
– Can be used for connecting directly to Col-EMU.
org.yamcs.tctm.TcapTcUplinker
– Sends telecommands to PaCTS/TCAP.
org.yamcs.tctm.DassTcUplinker
– Sends commands via DaSS
ERA MCS using Yamcs
27-July-2012