MITLL Presentation #63P-08-26
A Real-Time Publish-Subscribe
Control Plane for a COTM Node
HPEC
24 September 2008
J. Darby Mitchell
Software Architect
Wideband Tactical Networking
MIT Lincoln Laboratory
MIT Lincoln Laboratory
Slide 1
JDM 7/12/2016
This research was sponsored by the Department of the Army under Air Force Contract FA8721-05-C-0002.
Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the United States Government.
Outline
•
•
•
•
•
Introduction
–
–
Assumptions
Requests
Problem Statement
–
–
–
Project Vision and System Context
System Architecture
Software Architecture Problem
Software Architecture
–
–
–
–
Quality Attributes and Architectural Styles
Architectural Reasoning
Quality Attribute Tradeoffs
Runtime View
Design and Implementation
–
–
–
Designing Topics
Topic Mapping
Handling Exceptions
Conclusion
–
–
Slide 2
JDM 7/12/2016
Lessons Learned
Acknowledgements
MIT Lincoln Laboratory
Outline
•
•
•
•
•
Introduction
–
–
Assumptions
Requests
Problem Statement
–
–
–
Project Vision and System Context
System Architecture
Software Architecture Problem
Software Architecture
–
–
–
–
Quality Attributes and Architectural Styles
Architectural Reasoning
Quality Attribute Tradeoffs
Runtime View
Design and Implementation
–
–
–
Designing Topics
Topic Mapping
Handling Exceptions
Conclusion
–
–
Slide 3
JDM 7/12/2016
Lessons Learned
Acknowledgements
MIT Lincoln Laboratory
Assumptions and Requests
• Assumptions:
–
–
–
–
–
You know what MIT Lincoln Laboratory does
You recognize the value of buying vs. building software
You know that there’s no such thing as a “silver bullet”
You are familiar with the concepts of call-return middleware
Many of you are familiar with real-time publish-subscribe
• Requests
– If you’d like to discuss any of these assumptions, please talk
with me offline
– Please hold your questions until the end of the talk
Slide 4
JDM 7/12/2016
MIT Lincoln Laboratory
Outline
•
•
•
•
•
Introduction
–
–
Assumptions
Requests
Problem Statement
–
–
–
Vision and System Context
System Architecture
Software Architecture Problem
Software Architecture
–
–
–
–
Quality Attributes and Architectural Styles
Architectural Reasoning
Quality Attribute Tradeoffs
Runtime View
Design and Implementation
–
–
–
Designing Topics
Topic Mapping
Handling Exceptions
Conclusion
–
–
Slide 5
JDM 7/12/2016
Lessons Learned
Acknowledgements
MIT Lincoln Laboratory
Vision: Evolution of Terminal to Node
•
Milstar On-The-Move (MOTM) Terminal
–
–
–
–
–
•
3-axis positioner (MITLL)
Multi-band antenna and feed (44/30/20 GHz)
Blockage mitigation technology for COTM
IP over Milstar capability
Single link capability
COTM Node
–
–
–
–
–
Manage multiple links
Compose links from
modular HW/SW
components
Facilitate integration of
“stovepipe” COTS radios
Dynamic routing for
cooperative networking
Commercial (Ku)
WGS (Ka)
DVB
LDR/
MDR
CDL
Support for insertion of
additional radios
WLAN
HMMWV
Slide 6
JDM 7/12/2016
Milstar
GBS
WLAN
Dismount
MOTM
COTMTerminal
Node
MIT Lincoln Laboratory
COTM Node: Phase 0 Data Plane
Ethernet
GBS
Receiver
RF
Subsystem
Ethernet
Ethernet
Milstar
Modem
Ethernet
802.11
Radio
COTM Node
KEY
Milstar
Milstar
DVB
IP
LDR/
MDR
WLAN
WLAN
HMMWV
Slide 7
JDM 7/12/2016
COTM Node
Dismount
MIT Lincoln Laboratory
COTM Node: Phase 0 Data Plane
Ethernet
GBS
Receiver
RF
Subsystem
Ethernet
Ethernet
Milstar
Modem
Ethernet
802.11
Radio
COTM Node
GBS
KEY
Milstar
DVB
IP
DVB
WLAN
WLAN
HMMWV
Slide 8
JDM 7/12/2016
COTM Node
Dismount
MIT Lincoln Laboratory
System Architecture Concept
Network
Agent
Physics
Package
Black
Network
Modem
Red
Network
RF
HAIPE
Modem
Reconfigurable Links
user data (IP/RF)
node control
aperture pointing
Non-reconfigurable
link
Static Links
Slide 9
JDM 7/12/2016
MIT Lincoln Laboratory
Software Architecture Problem
Milstar Terminal antenna subassembly
Positioner
Antenna Control
Processor
Physics
Package
Unit
Milstar OTM
Terminal
Inertial
Navigation
Unit
RFI
Beacon
DataRx
Interface
Processor
API
Data
GBS Beacon Receiver
Modem Control
Processor (MDR)
Interface
Processor
MDR
Modem
Controller
RFI
Terminal Control Processor
RFI
??
Terminal
Controller
I
RF
RFI
MDR
Downlink
Processor
RFI
I
RF
Downlink Processor
RFI
RFI
Network
Terminal
Locator
RF
I
RFI
MDR User
Interface
RFI
RFI
RFI
Key
RF
I
Satellite
Locator
I
RF
Agent
Key Enabling
Hardware Decisions:
RF
I
–
–
–
–
Separate control and data
planes
802.11
Router
GBS DVB
Physics
RF Interface
Control
Control LDR Modem backplane
Package
SwitchedAPIGigabit
Ethernet
CompactPCI
API
Assembly
API
Unit
API
System boards are Intel x86 SBCs running Linux
Modem boards
shallWireless
be PPC
VxWorks
RF Interface
Radio running
Router
GBS DVB
Invocation
Notification
RFI
Antenna Control
Processor
MUX
Slide 10
JDM 7/12/2016
Component
RFI
Receiver
MIT Lincoln Laboratory
Outline
•
•
•
•
•
Introduction
–
–
Assumptions
Requests
Problem Statement
–
–
–
–
Project Vision
System Context
System Architecture
Software Architecture Problem
Software Architecture
–
–
–
–
Quality Attributes and Architectural Styles
Architectural Reasoning
Quality Attribute Tradeoffs
Runtime View
Design and Implementation
–
–
–
Designing Topics
Topic Mapping
Handling Exceptions
Conclusion
Slide 11
JDM 7/12/2016
–
–
Lessons Learned
Acknowledgements
MIT Lincoln Laboratory
Quality Attributes and Architectural Styles
• Essential Qualities
– Predictability: Ability to anticipate task scheduling requirements
– Timeliness: Ability to meet real-time constraints
– Reliability: Ensures delivery of critical control data
• Desirable Qualities
– Modularity: Facilitates decomposition and encapsulation
– Extensibility: Facilitates addition of components (i.e. functionality)
– Simplicity: Component development should be straightforward
• Architectural Styles
Style
Call-return
Example Design Patterns
Client-server, forwarder-receiver
Implicit Invocation
GoF Observer, Publish-subscribe
Slide 12
JDM 7/12/2016
MIT Lincoln Laboratory
Publish-Subscribe
•
•
•
Subscribers register to collect issues to a particular Topic
Publishers register to distribute issues to a particular Topic
A Topic acts as a GoF Mediator to decouple Publishers and
Subscribers
Publisher
S1
Topic
“VehiclePosition”
struct Position {
double latitude;
double longitude;
double altitude;
}
Slide 13
JDM 7/12/2016
Subscriber
Issue 1
longitude = -71.225
latitude = 42.447
altitude = 44.8
MIT Lincoln Laboratory
Publish-Subscribe
•
•
May be zero or more publishers per topic
May be zero or more subscribers per topic
Subscriber
Publisher
Subscriber
Publisher
Topic
Subscriber
Publisher
Slide 14
JDM 7/12/2016
Subscriber
MIT Lincoln Laboratory
Architectural Reasoning
Publish-Subscribe
•
•
•
•
•
Ideal for one-to-many or
many-to-many relationships
Promotes predictability
Data-centric (data identifier)
No assumption of existence
Data source always initiates
communication
Call-return
•
•
•
•
•
Ideal for one-to-one and manyto-one relationships
Client reliability
Forwarder
Promotes
Object-centric (object identifier)
Assumption of existence
Data source may or may not
initiate communication
Data
Flow
Data Flow
– forwarder-receiver
–
Result: decoupled interaction
client-server
Result: highly coupled interaction
Server
Slide 15
JDM 7/12/2016
Receiver
MIT Lincoln Laboratory
Quality Attribute Tradeoffs
Publish-Subscribe
•
•
•
•
Slide 16
JDM 7/12/2016
Timeliness
Predictability
Modularity
Extensibility
Call-return
•
•
Reliability
Simplicity
MIT Lincoln Laboratory
Real-time Publish-Subscribe with NDDS
•
•
•
•
•
•
•
Quality of Service Parameters
The Network Data Distribution Service is a real-time publishDeadline
Presentation
subscribe middleware developed by Real-Time Innovations, Inc.
Destination
Reliability
NDDS was
designedOrder
for distributed
real-time systems
Resource
Limits
Provides Durability
a number of QoS settings
to customize
the collection
and distribution
of issues
Entity Factory
Time-Based Filter
At the time
we selected the product,
the OMG DDS specification
History
Transport Priority
was still being finalized
Budget
Group
Data
RTI was aLatency
significant
contributor
to the
OMG DDS specification
effort
Lifespan
Topic Data
RTI had plans
to refactor NDDSUser
to conform
to the DDS spec
Liveliness
Data
RTI had already
published theirReader
RTPS Data
wireLifecycle
protocol
Ownership
Ownership Strength
Writer Data Lifecycle
Partition
Slide 17
JDM 7/12/2016
MIT Lincoln Laboratory
Software Architecture: Runtime View
Milstar Terminal antenna subassembly
Antenna Control
Processor
Milstar OTM
Terminal
Positioner
Physics
Package
Unit
RFI
RFI
Inertial
Navigation
Unit
RFI
Space Tracking Processor
Legacy
LDR API
Tracker
AHRS
Adapter
Antenna
Adapter
DTR
Adapter
Beacon Rx
API
GBS Beacon
Receiver
NDDS
Key
LDR
Adapter
Logger
Network
Agent
Router
Manager
RIM
Adapter
Wireless
Adapter
Node
Controller
Position
Service
Pub-sub Domain
Component
Node Control Processor
Router
API
Router
Slide 18
JDM 7/12/2016
RF
Interface
API
RF Interface
MUX
802.11
Control
API
Wireless Link
SSH
UDP
Invocation
Notification
Vehicle Position & Velocity
Black
LAN
Signal Strength
Pointing Angle
MIT Lincoln Laboratory
Outline
•
•
•
•
•
Introduction
–
–
Assumptions
Requests
Problem Statement
–
–
–
–
Project Vision
System Context
System Architecture
Software Architecture Problem
Software Architecture
–
–
–
–
Quality Attributes and Architectural Styles
Architectural Reasoning
Quality Attribute Tradeoffs
Runtime View
Design and Implementation
–
–
–
Designing Topics
Topic Mapping
Handling Exceptions
Conclusion
Slide 19
JDM 7/12/2016
–
–
Lessons Learned
Acknowledgements
MIT Lincoln Laboratory
Designing Topics
•
Samples – periodic, independent measurements of the environment
Examples:
– Vehicle position and velocity
– Link state
– Modem signal strength
– Satellite location and velocity
– UTC Time
•
RELIABILITY: BEST EFFORT
HISTORY: KEEP LAST
Events – sporadic, relative changes in system state
Examples:
– Link formation and teardown
– Status messages
– Error messages
– Parameter changes
– Routing changes
Slide 20
JDM 7/12/2016
RELIABILITY: RELIABLE
HISTORY: KEEP ALL
MIT Lincoln Laboratory
Topic Mapping
Position
Service
LDR
Adapter
Node
Controller
S9
E7
E6
RF
Interface
Module
E5
S4
E2
E1
DTR
Adapter
S2
Key
S3
S1
E4
E3
Component
S7
S5
Sample Topic
S8
Event Topic
S6
Publication
Subscription
Slide 21
JDM 7/12/2016
AHRS
Adapter
Tracker
Antenna
Adapter
MIT Lincoln Laboratory
Topic Mapping: Samples
Samples
S1 – UTCTime
S2 – AHRSLocation
S3 – AHRSDisplacement
S4 – AHRSVelocity
S5 – AntennaReferenceAngle
S6 – AcquisitionMetric
S7 – AntennaAngles
S8 – DTRSamples
S9 – LDREnergyMetric
Key
Position
Service
LDR
Adapter
Node
Controller
S9
E7
E6
RF
Interface
Module
E5
S4
E2
E1
DTR
Adapter
S2
S3
S1
E4
E3
Component
S7
S5
Sample Topic
S8
Event Topic
S6
Publication
Subscription
Slide 22
JDM 7/12/2016
AHRS
Adapter
Tracker
Antenna
Adapter
MIT Lincoln Laboratory
Topic Mapping: Events
Events
E1 – DeviceStatus
E2 – TrackCommand
E3 – AntennaCommand
E4 – DTRParams
E5 – DeviceCommand
E6 – LDRCommand
E7 – RIMCommand
Position
Service
struct DeviceStatus {
S9
string
deviceId;
int statusType;
struct DeviceCommand E7
{
E6
int code; string deviceID;
E5
string msg;int command;
};
};
S4
E2
S2
Key
Node
Controller
LDR
Adapter
S3
S1
Component
Sample Topic
RF
Interface
Module
E1
struct AntennaCommand {
string deviceID;
int command;
E4
E3
double az;
double el; S7
}; S5
S8
Event Topic
DTR
Adapter
S6
Publication
Subscription
Slide 23
JDM 7/12/2016
AHRS
Adapter
Tracker
Antenna
Adapter
MIT Lincoln Laboratory
Outline
•
•
•
•
•
Introduction
–
–
Assumptions
Requests
Problem Statement
–
–
–
–
Project Vision
System Context
System Architecture
Software Architecture Problem
Software Architecture
–
–
–
–
Quality Attributes and Architectural Styles
Architectural Reasoning
Quality Attribute Tradeoffs
Runtime View
Design and Implementation
–
–
–
Designing Topics
Topic Mapping
Handling Exceptions
Conclusion
Slide 24
JDM 7/12/2016
–
–
Lessons Learned
Acknowledgements
MIT Lincoln Laboratory
Lessons Learned
• Using publish-subscribe:
– Made component development slightly more complicated
– Greatly facilitated software integration
– Enabled us to successfully defer some components, while
still making progress on the project
– Is not as straightforward when you are marshalling
parameters with commands
• Respect the invariants of the architectural style:
– NodeController could be killed and later restarted with no
detrimental impact to system in steady state
– Debug topics could be published for later use with negligible
impact on system performance
• Actively managing consistency of QoS settings was
•
essential
Having a commercial vendor to delegate middleware
support concerns to was very helpful
Slide 25
JDM 7/12/2016
MIT Lincoln Laboratory
Acknowledgements
• Sponsor: PM WIN-T, Ft. Monmouth
• Group Leaders: Dr. Marc Zissman and Scott Sharp
• Systems Engineer: Dr. Andrew Worthen
• RF team: Dr. Jim Vian, John Murphy, Ted O’Connell
• Hardware team: Steve Pisuk, John Delisle, Jason Hillger
• Software team: Darby Mitchell, Curran (Nachbar)
Schiefelbein, Marc Siegel, Marie Heath
• Testing team: Dr. Mark Smith, Ted O’Connell
Slide 26
JDM 7/12/2016
MIT Lincoln Laboratory
Current Work
•
TSAT Reference Terminal (TRT)
–
–
–
•
A joint project with Group 64 based on TRUST-T
A COTM Node that is based on the Software Communications
Architecture (SCA) for software defined radios.
The SCA mandates the use of CORBA middleware, so DDS will not be
used.
Network and Link Emulation Testbed (NLET)
–
–
Slide 27
JDM 7/12/2016
A distributed network emulation testbed
Uses DDS for a distributed real-time context simulation and real-time
dynamic control of link emulation.
MIT Lincoln Laboratory
References
• Mitchell et. al. “Applying Publish-Subscribe to COTM Node
Control”, MIT Lincoln Laboratory Journal, Volume 16, No. 2,
December 2006.
http://www.ll.mit.edu/news/journal/journal.html
• L. Bass, P. Clements and R. Kazman, Software Architecture
•
in Practice, Addison Wesley, 1998.
Garlan, D. and M. Shaw, Software Architecture:
Perspectives on an Emerging Discipline, Prentice Hall,
1996.
Questions?
mitchelljd@ll.mit.edu
Slide 28
JDM 7/12/2016
MIT Lincoln Laboratory
Backup Slides
MIT Lincoln Laboratory
Slide 29
JDM 7/12/2016
Reasoning About Connectors
•
•
Invocation
Reasoning about connectors vs. components
Consider several dimensions:
–
–
•
synchronous vs. asynch
cardinality (1 : 1 vs. 1 : n)
(1:1)
Delegation
(synchronous)
1:n ?
Ignore implementation concerns
n:1 ?
Activation
Evocation
(1:1)
Notification
Distribution
(asynchronous)
(1:n)
Publish-subscribe =
Collection
Distribution + Implicit Invocation + Collection
Slide 30
JDM 7/12/2016
(n:1)
MIT Lincoln Laboratory
System Architecture: Connection View
Milstar OTM Terminal
Assembly
Space
Tracking
Processor
Serial
Antenna
Serial
Control
Processor
Antenna Assembly
Antenna / Positioner
Serial
GBS Beacon Receiver
Ethernet
GBS DVB Receiver
GBS Modem Assembly
Ethernet
Key
Ethernet
802.x Wireless Link
Assembly
Processor
Router
Notional
Assembly
Serial
Control only
Ethernet
Ethernet
Node
Control
Processor
Slide 31
JDM 7/12/2016
Serial
Multiband RF
Assembly
Control & Data
User Laptop
MIT Lincoln Laboratory
Driver and Adapters
•
•
•
•
There is a one to one relationship between Drivers and Adapters
Node Controller only interacts with an Adapter through its Driver
A Driver caches Status and Error updates from its Adapter
Adapters may interact with other Adapters
Test Harness
Commands
Parameters
Node
Controller
Test
Case
Driver
Status
Adapter
Device
Errors
Logger
Slide 32
JDM 7/12/2016
MIT Lincoln Laboratory
Exception Handling
•
Based on concepts from online article:
Agarwal, Sachin, “C++ Exception-Handling Tricks for Linux”, IBM Software Labs, Feb 2005.
http://www-128.ibm.com/developerworks/linux/library/l-cppexcep.html?ca=dgr-lnxw1fExceptionTricks
•
•
Added
mechanism
to throw exceptions
up to C++ from
existing C code
=================================================================
------------------------Caught LL::Exceptions::RTE
------------------------WTN
Node
Controller
------------ at [ll/devices/obj/x86-linux/RimNDDSAdapter.cpp:99] ------------Added mechanism
to translate
POSIX signals (e.g. SEGV) to C++ exceptions
node0 : spiral1
Type: CriticalError
----------------------------------------------------------------Text: /dev/ttyS2 is not readable
--------------------------Caught
---------------------------Where: virtual
void std::exception
LL::SYS::SerialHWInterface::open()
at
node0> at ll/sys/obj/x86-linux/SerialHWInterface.cpp:195
-----------[ll/common/obj/x86-linux/TerminateHandler.cpp:44] -----------rim0: Type:
Type: ->
SIGSEGV
Trace:CriticalError
Text:
/dev/ttyS2
is not readable
Text: Received signal
(0) LL::Exceptions::RTE::RTE(LL::Exceptions::RTE::RTETypeEnum,
Where:
virtual
void
at at
Where: LL::Exceptions::SignalTranslator::SignalTranslator(int)
std::stringLL::SYS::SerialHWInterface::open()
const&, std::string const&) [RTE.cpp:32]
ll/sys/obj/x86-linux/SerialHWInterface.cpp:195
ll/common/obj/x86-linux/SignalHandler.cpp:122
(1) LL::SYS::SerialHWInterface::open() [SerialHWInterface.cpp:195]
Trace:
(2) LL::SYS::SerialHWInterface::binaryInvoke(std::vector<unsigned
(0) LL::Exceptions::SignalTranslator::SignalTranslator(int)
[SignalHandler.cpp:125]
char, std::allocator<unsigned char> > const&, int)
[SerialHWInterface.cpp:128]
(1) LL::Exceptions::SignalHandler::handler(int)
[SignalHandler.cpp:102]
(3) LL::Devices::Rim::isOscillatorLocked()
[Rim.cpp:86]
rim0 DS_ERROR EC_CRITICALERROR
(2) [DeviceStatus/rim0]
[libpthread.so.0]
(4) LL::Devices::RimNDDSAdapter::sendDeviceStatus()
[RimNDDSAdapter.cpp:99]
(exceptionMsg=Type: CriticalError
Text:
(3) RtiThreadSleep
(5) [libutilsip.so]
LL::NDDS::Adapter::toggleDummy(bool)
[NDDSAdapter.cpp:77]
/dev/ttyS2 is not readable Where: virtual void
(4) NddsUtilitySleep
[libndds.so]
(6) main
[RimNDDSAdapter_main.cpp:95]
LL::SYS::SerialHWInterface::open()
at
(5) LL::NDDS::Adapter::enterMainLoop(double)
[NDDSAdapter.cpp:92]
(7) __libc_start_main
[libc.so.6]
ll/sys/obj/x86-linux/SerialHWInterface.cpp:195
(6) main [RimNDDSAdapter_main.cpp:96]
(8) _start [??:0]
guid=rim0)
(7) __libc_start_main
[libc.so.6]
---------------------------------rethrown ---------------------------------(8) _start [??:0]
Node
Controller
Logger
Adapter
E
NDDS
Slide 33
JDM 7/12/2016
E
MIT Lincoln Laboratory