Large-Scale System Integration with
DDS for SCADA, C2, and Finance
Rick Warren, Principal Engineer rick.warren@rti.com
What do I mean by “large system”?
l Systems of systems
–
–
–
–
Modular, hierarchical design
Legacy components, subsystems
Multiple technologies
Global scale
l Decoupled subsystem lifecycles
–
–
–
–
Independent development
Independent deployment and use
Independent management
Independent revision and retirement
l Multiple communities of interest
– Different data interest, entitlements
– Non-uniform levels of trust
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
2
What matters to integrators of large systems?
l Governance
–
–
–
–
What information will be exchanged?
Under what conditions will it be exchanged?
Who is allowed to exchange this information?
If these SLAs are violated, can the exchange be prevented?
Can I be notified?
– In the past, what has occurred wrt these SLAs?
l Isolation
– When I connect A and B, ensure they don’t break (each other)
– When I disconnect A, ensure B doesn’t break
– When I connect C, don’t change A or B
l Scalability
(like in any system,
l Fault Tolerance but stakes are higher)
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
3
Part 1: Architecture
(we’ll get to technology later)
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
4
A Modest Proposal
l Fundamental design principles scale
– Abstraction: Provide interface based on relevant concepts
– Encapsulation: Hide internal implementation, communication
– Composition: Combine existing capabilities into new ones
Component
Class
State
Behavior
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
Component
Class
System
System
Component
5
Schematic of a Composed System
Isolation
P
P
P
Router/
Gateway
Router/
Gateway
LAN
P
P
WAN
P
Subsystem 1
P
Additional
Governance
LAN
P
P
Subsystem 2
l Subsystems may have different network environments
l Integration may have different network environment
than subsystems themselves
l Data may need to be transformed/cleansed as it
moves among subsystems
l Routing/gateway services will adapt data types /
formats / protocols
P
Schematic of a Composed System
P
P
P
Router/G
ateway
P
P
P
Subsystem 1
l Wash, rinse, repeat
P
Router/G
ateway
P
P
P
Subsystem 2
Apples and Oranges
P
P
Router/
Gateway
P
P
Router/
Gatewa
y
P
Subsystem 1
P
P
Subsystem 2
P
P
P
P
P
P
P
P
P
P
Subsystem 1 + Subsystem 2
P
P
P
Nothing New Under the Sun
Subsystem
Subsystem
Data
Data Space
Space
Subsystem
Subsystem
Data
Data Space
Space
Router/
Gateway
Router/
Gateway
Integration
Integration
Data
Data Space
Space
Router/
Gateway
Subsystem
Subsystem
Data
Data Space
Space
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
9
Nothing New Under the Sun
Radar
Radar
Data
Data Space
Space
Weapons
Weapons
Data
Data Space
Space
Router/
Gateway
Router/
Gateway
Integration
Integration
Data
Data Space
Space
Router/
Gateway
Readiness
Readiness
Data
Data Space
Space
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
Defense
Industry
10
Nothing New Under the Sun
Order
Order
Execution
Execution
Data
Data Space
Space
Market
Market Data
Data
Data
Data Space
Space
Router/
Gateway
Router/
Gateway
Integration
Integration
Data
Data Space
Space
Router/
Gateway
Compliance
Compliance
Data
Data Space
Space
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
Financial
Services
Industry
11
Nothing New Under the Sun
Phasor
Phasor
Data
Data Space
Space
PDC
PDC
Data
Data Space
Space
Router/
Gateway
Router/
Gateway
Integration
Integration
Data
Data Space
Space
Router/
Gateway
Administration
Administration
Data
Data Space
Space
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
Power
Industry
12
Architecture Recap
l Solve big problems
like you solve small
ones
– Break them into
pieces
– Define interfaces
between pieces
– Implement each piece
– Integrate along the
interfaces
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
l Translation
– Define subsystems
– Implement them how
you like
• May have different
requirements, therefore
technology
– Router/gateway
defines/enforces
interfaces
13
Remaining Problems to Address
l Fault tolerance: Router can’t be single point of failure
3 answers:
– Persistent data survives system failures
– Segmented/load-balanced configuration limits scope of failures
– Redundancy allows continued service during failure
l Scalability: How big can a subsystem be?
1. How big a subsystem can you understand, test, and debug?
2. How well does infrastructure scale?
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
14
Part 2: DDS
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
15
How Does DDS Stack Up?
l
Governance
–
–
–
–
–
l
–
–
Built in
Built in
Products available; future standards
Built in
–
Products available
Prevent data “leaks” and side
effects
Allow dynamic (dis-|re-)connection
Support independent integration
and evolution
–
Built in: domains, partitions
–
–
Built in
Built in, esp. w/ DDS-Xtypes
–
–
–
Built in
Built in if router uses DDS
Built in if router uses DDS
–
–
–
Products available; future standards
Highly scalable
Protocol support if necessary
Fault tolerance
–
–
–
l
–
–
–
–
Isolation
–
l
Structural SLA: data type
Behavioral SLA: delivery QoS
Security
Problem
detection/prevention/notification
System monitoring and recording
Data persistence
Segmented/load-balanced routing
Redundant routing
Scalability
–
–
–
WAN connectivity
Peer-to-peer communication
Brokered/managed communication
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
16
Redundant Data Routing
Multiple
readers:
no problem
Transform
and Forward
Multiple
writers:
prevent
duplicates!
Router 1
Write
MyData
Read
MyData'
Router 2
Fortunately,
built into
RTPS
(DDSI)
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
17
Redundant Data Routing
RTPS
Message
Inline QoS
Original
Writer Info
Inline
QoS
Orig.
Writer
Info
Orig.
Writer
GUID
Data
Other
Metadata
Orig.
Writer
Seq. #
Identity of original writer
Identity of original sample
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
Read
MyData'
Reader discards
duplicates
based on these
18
Redundant Data Routing
Transform
and Forward
Writer 5AC2
Seq. # 42
Writer 5AC2
Router 1
Data
Seq. # 42
Data
Write
MyData
Read
MyData'
Writer 5AC2
Seq. # 42
Data
Writer 5AC2
Router 2
Seq. # 42
Data
If your impl.
doesn’t support,
fall back to
ownership.
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
19
DDS Scalability: Two Aspects
l Application data
– How does performance fall off as # participants, writers,
readers increase?
– Any single writer/reader pair can saturate gig-E:
aggregate throughput is not main issue
– Interesting issue: Fan-out – number of readers per writer
l Discovery
– How many DDS applications can discover one another?
– Interesting issue: How many applications need to discover
one another?
l Hard limits, if any, very dependent on DDS
implementation, machine configuration, network, etc.
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
20
DDS Scalability: Application Data
l RTPS reliable multicast scales at least:
– …to hundreds of readers per writer
– …with very little degradation in throughput.
– Larger testing facilities welcomed. J
< 15% decrease
1~900 readers
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
21
DDS Scalability: P2P Discovery Scenarios
l Single domain, symmetric discovery
– Everyone discovers everyone else
– Easiest to configure; most challenging wrt scalability
– RTI test results: 1,800 participants; 3.2M endpoint matches
l Single domain, asymmetric discovery
– Each participant only knows of certain others in its domain
– Good for partitioning domains in which not everyone talks
1
…
l Multiple domains
n
– Greatest separation for subsystems that rarely exchange data
• RTPS maps to different IP ports
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
22
DDS Scalability: Stronger Measures
l P2P RTPS allows participant to talk to at least 1-2K
others (based on current experimental data)
– My system is properly decomposed. Discovery is optimized.
But I still need to scale bigger.
1. Brokered application data
…
…
…
n<m
n
m
1. Brokered discovery
n
m
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
23
DDS Scalability: Brokered Discovery
l Can be built with RTPS-standard building blocks
RTPS Metadata
Orig. Metadata
RTPS Metadata
Data
Data
Built-In Topic
Application Topic
RTPS Metadata
Data
RTPS InfoSource submessage provides data forwarding capability:
“Treat the following as if it came from that guy instead of from me.”
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
24
Future Direction: Enhanced Built-in Security
l System of systems may have non-uniform trust
– Multiple communities of interest w/ different
entitlements/authorization
– Infrastructure components may (not) be trusted
(e.g. brokers, daemons, persistence, recording/playback, etc.)
– There is no “system high”
l Desirable new specifications:
– Topic-level access control
– Sample-level tagging and labeling
– Sample-level signing and/or encryption
l More on security concerns in other talks this week
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
25
Future Direction: Standardized WAN Support
l Site-to-site comms need DDS across WAN(s)
– Including untrusted networks
– Including firewall, NAT traversal
l RTPS (DDSI) protocol designed for layering atop diverse
lower-level protocols
– Proven: UDP, TCP, serial, switched fabric, Infiniband, VME,
shared memory…
– Blessed for interop by OMG: UDP
– Challenge: UDP may not be routable, especially if multicast
l Additional protocol mappings (PSMs) could provide
improved site-to-site interop
– TCP for WAN routing
– (D)TLS for transport-level security (a clarification, not a new PSM)
– Could be standardized quickly
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
26
Recap
l Large systems frequently composed of smaller
subsystems
– Developed independently
– …With different network environments
– …And different data interest/entitlements
l Best expressed in hierarchical system design
– Subsystems “gated” by DDS router/gateway
• Broker internal ßà external protocols (may be different)
• Transform/cleanse internal ßà external data types
• Impose governance: enforce policy, attach monitoring
– Inter-router integration space is itself a “subsystem” for
purposes of hierarchical composition
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
27
Recap
l DDS already provides much of what’s needed
– Strong SLAs for governance
– Multiple topologies for fault-tolerant subsystem isolation
– Flexible, scalable interoperability protocol
l Promising future directions:
– RTPS/TCP protocol layering for WAN traversal
– Enhanced security model
• Topic-level access control
• Sample-level tagging, signing, and encrypting
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
28