Wojciech Sliwinski
Wojciech.Sliwinski@cern.ch
Beams Department, Controls Group
CERN
Outline

Defining Middleware

Types of Middleware

CERN Acc. Controls Messaging Infrastructure

Middleware Review for CERN Acc. Controls

Conclusions
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
2
Outline

Defining Middleware

Types of Middleware

CERN Acc. Controls Messaging Infrastructure

Middleware Review for CERN Acc. Controls

Conclusions
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
3
Why Middleware?

How do we build/connect
applications over a
network?

How do we facilitate
Distributed Computing
(2-tier, …, n-tier)?

How to support the
heterogeneous
environments?
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
4
What is Middleware?

Software, which allows an application to interoperate with other
software, without requiring the user to understand and to code
the low-level operations required to achieve interoperability
 Software layer between OS and the applications
 Hides complexity & heterogenity of distributed system
 Handles issues related to OS, Network Protocols & hardware platforms
Application
Standard API
Application
Standard API
MW
Specific API
Operating
System1
Application
Standard API
MW
Specific API
Operating
System2
MW
Specific API
Operating
SystemN
Communication system
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
5
Introduction to Middleware

Middleware provides support for:
 Naming, Location, Service discovery, Replication
 Protocol handling, Communication faults, QoS
 Synchronisation, Concurrency, Failover, Scalability
 Access control,Middleware
Authentication
is everywhere.
You might think you can leave it but it just finds you again :-)

Middleware dimensions:
 Request-Reply
 Language-specific
 Proprietary
 Small-scale
 Tightly-coupled
26th October 2011
vs.
vs.
vs.
vs.
vs.
Asynchronous Messaging
Language-independent
Standards-based
Large-scale
Loosely-coupled components
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
6
Outline

Defining Middleware

Types of Middleware

CERN Acc. Controls Messaging Infrastructure

Middleware Review for CERN Acc. Controls

Conclusions
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
7
Middleware – large domain … which to choose?
Network
Embedded
LAN
req/rep
direct
p2p
26th October 2011
WAN/
Internet
pub/sub
broker
rpc/orb
message
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
8
Types of Middleware

Remote Procedure Call (RPC)

Object-Oriented Middleware (OOM)

Message-Oriented Middleware (MOM)

Event-Based Middleware a.k.a Publish-Subscribe
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
9
Remote Procedure Call - RPC






Masks remote function calls as being local
Client-Server model with Request-Reply paradigm
Synchronous communication
Marshalling of function parameters and return value
Provides Location & Platform transparency
Technologies:
 DCE-RPC
 XML-RPC
Client App.
Server App.
IDL Stub
IDL Skeleton
Req/Reply
Network
26th October 2011
Network
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
10
RPC - Disadvantages

Synchronous Request-Reply interaction
 Tight coupling between client and server
 Client may block for a long time if server loaded
○ Multi-threaded programming at client
 Slow/failed clients may delay servers when replying
○ Multi-threaded programming at servers

RPC paradigm is not object-oriented
 Invoke functions on servers as opposed to methods on objects
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
11
Object-Oriented Middleware - OOM




Client-Server model with Request-Reply paradigm
Synchronous communication
Marshalling of function parameters and return value
Provides Location & Platform transparency
 Objects can be local or remote
 Remote objects accesible via proxy objects

Technologies:
 OMG CORBA
Client App.
IDL Skeleton
 MS DCOM
 Java RMI
 SOAP
Server App.
IDL Stub
POA Adapter
Client ORB
Server ORB
Req/Reply
IIOP
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
IIOP
12
Object-Oriented MW - Disadvantages

Synchronous Request-Reply interaction
 Tight coupling between client and server
 Client may block for a long time if server loaded
○ Multi-threaded programming at client
 Slow/failed clients may delay servers when replying
○ Multi-threaded programming at servers

Distributed garbage collection
 Releasing memory for unused remote objects
 Error-prone and difficult to debug

Static mappings and heavy-weight development
 Difficult to extend an operational system
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
13
Message-Oriented Middleware - MOM





Data exchange via Messages & Message Queues
Asynchronous communication (Point-to-point  One-to-One)
Client and Server are loosely coupled
Message Servers may do filtering, transforming, logging…
Technologies: IBM WebSphereMQ, TIBCO Rendez-vous, JMS, SOAP
Client App.
Message
Servers
Server App.
local message
queues
message
queues
local message
queues
Network
Network
Network
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
14
Message-Oriented MW - Disadvantages

Poor programming abstraction (but has evolved)
• Rather low-level
• Request-Reply more difficult to achieve, but can be done

Message formats originally unknown to middleware
• No type checking (solved in JMS)

Queue abstraction only gives One-to-One communication
• Limits scalability (solved in JMS with Publish-Subscribe)
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
15
What we lack, so far

General interaction patterns
• We have Request-Reply and One-to-One
• Publish-Subscribe? One-to-Many? Many-to-Many?
• Dynamic joining and leaving?

Location transparency
• Anonymity of communicating entities
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
16
Event-Based Middleware a.k.a Publish-Subscribe





Publishers (advertise and) publish Events (Messages)
Subscribers express interest in Events with Subscriptions
Event Service notifies interested Subscribers of published Events
Events can have arbitrary content (typed)
Technologies (Brokers): JMS (ActiveMQ, RabbitMQ), AMQP (…)
Publisher1
Event Service
(Broker)
publish
Publisher2
Subscriber1
subscribe
notify
Subscriber2
…
…
topics
PublisherN
26th October 2011
SubscriberN
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
17
Properties of Publish-Subscribe

Asynchronous communication
• Publishers and Subscribers are loosely coupled

Many-to-Many interaction between Pubs and Subs
• Scalable scheme for large-scale systems
• Publishers do not need to know Subscribers, and vice-versa
• Dynamic join and leave of Pubs & Subs (thanks to Brokers)

Topic based Publish-Subscribe very expressive
• Filtered information delivered only to interested parties
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
18
Outline

Defining Middleware

Types of Middleware

CERN Acc. Controls Messaging Infrastructure

CERN Acc. Controls Middleware Review

Conclusions
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
19
CERN Controls Middleware – in all SW tiers…
GENERAL
PURPOSE
NETWORK
OPERATOR
CONSOLES
CMW client (C++/Java)
GUIs, LabView, RADE
FILE SERVERS
TCP/IP
communication services
JMS client
(Java)
GUIs
APPLICATION SERVERS
SCADA SERVERS
TCP/IP communication services
CMW server (C++/Java)
JMS client (Java)
TIMING GENERATION
Proxy, DIP, AlarmMon, AQ
Servers: Logging, InCA, SIS
CMW client (Java servers)
Logging, LSA, InCA, SIS
RT Lynx/OS
VME FRONT ENDS
WORLDFIP
Front Ends
T
T
T
T
M IDDLE TIER
CERN GIGABIT ETHERNET TECHNICAL NETWORK
FIXED
DISPLAYS
PRESENTATION TIER
OPERATOR
CONSOLES
PLCs
BEAM POSITION MONITORS,
BEAM LOSS MONITORS,
BEAM INTERLOCKS,
RF SYSTEMS, ETC…
T
T
QUENCH PROTECTION AGENTS,
POWER CONVERTERS FUNCTIONS
GENERATORS, CRYO TEMPERATURE
SENSORS…
DIRECT I/O
T
T
FIP/IO
T
PROFIBUS
OPTICAL
FIBERS
T
CMW server (C++)
PVSS (Cryo, Vacuum)
RESOURCE TIER
CMW server (C++)
FESA, FGC, GM
WorldFIP SEGMENT
(1, 2.5 MBits/sec)
TCP/IP communication services
ACTUATORS AND SENSORS
CRYOGENICS, VACUUM, ETC…
LHC MACHINE
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
20
JMS for Accelerator Controls
Java Messaging Service
 Purpose

 Reliable transport of data between Java Processes
 Controls
○ Alarm System
○ Data Rendering Services
○ (Control) GUIs
○ Logging Services
○ Beam Security System(s)
26th October 2011
No JMS No Beam !
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
21
JMS for Accelerator Controls
~50 Processes
~46K Subscriptions
280 Connections
JAVA Process
JAPC
JMS
CMW
~100 Processes
GUI Process
JMS
JMS
CMW
JAPC
C/C++ Process
CMW
CMW
Front End
4000 servers @
1700 machines
Devices / Hardware
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
22
Examples of JMS ”aware” systems
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
23
History of JMS in Accelerator Controls

Usage of SonicMQ JMS for LHC Alarm System since 2001

ActiveMQ introduced in 2005

Other project started to use JMS service

Today vital part of Beam Operations
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
24
Technical Aspects

Broker Configuration
 Non-persistent messages on topics
 No subscription recovery policy
 No durable subscribers
 No exclusive consumers
 No replyTo mechanism
 Broker is persistent

Clients
 Pure JAVA - Wireformat Protocol
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
25
General JMS Deployment Situation

SonicMQ : 12 Brokers
 TIM, LASER, DIAMON
 Always as two node cluster

ActiveMQ : 16 Brokers
 One main default Broker Service
 Otherwise project dedicated Brokers

All Brokers on Technical Network
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
26
Overall Broker Service Jan 2011
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
27
JMS Service Data Handling

Central JMS Service (276 days in 2010)
 58 Million msg / day
 120 TB IN
 650 TB OUT

=> 440 GB /day
=> 2.5 TB /day
YES, ActiveMQ
can!
Additional handling of
 Heavy Connection and
Subscription handling
 Bursts loads

Connected subscribers grew
from 2K to 12K in 6 months
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
28
Data Handling during LHC startup 2010
Total Data Out
2.57 TB/day
Increase in data demand
1TB/day
LHC startup
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
29
JMS Service Availability 2010*

Total hours : 6624

ActiveMQ:
 = 8 minutes downtime (due to kernel update)

SoniqMQ :
 = 8 hours downtime
* November 2010
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
30
Monitoring and Troubleshooting Tools

DIAMON & Lemon

JConsole / JVisualVM

Dump of broker information to SQLite DB

Traffic Monitoring Tool

Various Message Listeners
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
31
Easy Identification of Bad Producers
Problematic Producer
Fixed
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
32
JMS Service – Lessons Learnt

ActiveMQ was a good choice
 Running reliably
 Performs well
 Provides many metrics for diagnostic and monitoring
 Good integration in BE-CO monitoring tools via JMX

Projects are not always aware of sending
so much data
 Service Level Agreements with projects required
 More analyzing tools for end users required
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
33
JMS Service – Lessons Learnt

Monitoring is a vital part of operations
 Message reception speed test
 Memory Percent usage
 JMX attributes (#topics, #consumers, #subscriptions)

Broker Configuration is not always obvious
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
34
JMS Service – Lessons Learnt

Clustering
 Good for failover and load balancing
 Additional complexity
 Updating clients for new cluster members
○ Automatic rebalancing of clients

Memory is important
 Tuning of Java GC options
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
35
Outline

Defining Middleware

Types of Middleware

CERN Acc. Controls Messaging Infrastructure

CERN Acc. Controls Middleware Review

Conclusions
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
36
CERN Controls Middleware – CMW
GUI operator consoles
Controls Middleware
Front ends
“virtual devices”
Embedded Middleware
Hardware Devices
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
37
CMW Mandate & Scope
Standard set of MW solutions & libraries
 Centrally managed services
 Track & optimize runtime parameters

Control System
GUI Applications

Scope: CERN Accelerator Complex
 Operational 24*7*365
Control Logic
Middleware
 Must be Reliable & High Quality
 ~80’000 HW devices, ~4’000 servers
 ~1700 machines, ~2’000’000 IO points
 Huge infrastructure!
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
38
CMW Data Flow
JMS
Broker
JAVA Process
JAPC
JMS
CMW
JAVA Process
MONITOR
C/C++ Process
CMW
FEC
JMS
CMW
JAPC
GET / SET
CMW
publisher
direct
Devices / Hardware
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
39
CERN Controls Middleware
Linux/Windows
Problems:
- C++ and Java implementations differ
- Heavy in memory usage
- Complex error prone API
- No direct support for pub/sub
Controls
Middleware
- Blocking
issues
(JacORB)
- Shrinking community
- Lack of new releases and bug fixes
C++/Java process
Controls Middleware API

CORBA


TCP/IP


CORBA

Controls Middleware API
Long LHC shutdown at the end of 2012 C++ process
A unique possibility!
RT Linux/LynxOS
Instead of patching let’s provide a new one!
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
40
CERN Middleware Requirements
Lightweight
Active community
Desirable
Friendly API, documentation
Request/reply & pub/sub patterns
QoS
Asynchronous
Mandatory
Performance & Scalability
Stability, Maturity & Longevity
Open source, redistributable license
C++/Java
Fundamental
Linux/Windows
Over TCP/IP LAN
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
41
How did we evaluate –> our criteria
Appearance
Simple usage
• Creators
• specification
• documentation
• Users
• forums
• bug reports
• Internet
Testing
• Download
• licensing
• Compile
• Linux
• LynxOS
• Run examples
• Communication
patterns
• Performance
• QoS
• Exceptional
situations
CRITERIA
API, look & feel,
documentation
26th October 2011
resources,
binary size,
memory
Community,
Communications
maturity
patterns
QoS
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
performance
42
Surprising how many great new products!
2011
1991
CORBA
???
RPC
1983
BSD
sockets
TODAY
1969 1974 1980 1981
26th October 2011
1998
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
2011
43
Evaluated middleware products
All opinions are based only on our knowledge and evaluation. Each of the
products, depending on the requirements, may constitute a good solution.
CoreDX
OpenAMQ
RTI DDS
QPid
ZeroMQ
OpenSpliceDDS
RabbitMQ
YAMI
Ice
omniORB
JacORB
26th October 2011
MQtt RSMB
Thrift
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
Mosquito
44
CORBA (omniORB, JacORB)
Object-oriented communication platform standardized by OMG. It is
over 20 years old, and the standard is well established but...





Complex, hard to learn and use
Differences between implementations
Big memory footprint
Shrinking community
Seldom updates and bug fixes
26th October 2011





Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
45
Ice
Object-oriented middleware by ZeroC, people who left CORBA to fix
its problems. Conceptually similar to CORBA but...





Better design and implementation
Modern easier to use API and IDL mapping
Concise C++ and Java implementation
Support for versioning
Active community
26th October 2011





Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
46
Thrift
Middleware by Facebook (moved to Apache)





Lightweight, free of unnecessary dependencies
Modern, clean API
Active community
Ongoing development, still incomplete
No documentation, empty tutorial
26th October 2011





Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
47
YAMI4
Message-oriented middleware developed by one of our former
colleagues. Used by some CERN Controls services.





Lightweight, free of unnecessary dependencies
Well designed, modern API
Asynchronous req/rep and pub/sub patterns
Dynamic type specification (paid in performance)
Small community
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...





48
AMQP – OpenAMQ, Qpid, RabbitMQ
Middleware where messages are distributed by a broker. A few
independent implementations of the broker and clients.






Big community
Designed to support pub/sub
Protocol standard issues
Broker – single point of failure
Broker – additional hop, slower communication
Broker – additional, non compliant monitoring tools
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...






49
DDS – RTI, CoreDX, OpenSplice
Data-oriented middleware with p2p communication. Standard
defined by OMG, with a few compatible implementations.






Big industrial and military community
Possible compatibility with CORBA
Designed to support pub/sub
Req/rep possible but needs 2 channels
Steep learning curve
Complex API
26th October 2011






Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
50
ZeroMQ
Message-oriented middleware by iMatix (they dropped OpenAMQ.)
API resembles BSD sockets but it is so much more.






Simple yet powerful API
Support for in-proc, inter-proc, TCP, PGM
Lightweight, free of unnecessary dependencies
Many communication patterns
May be used as a concurrency framework
Active community
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...






51
Performance requirements reviewed
Instead of estimating
 We asked our users what they will need
 Gathered statistics from the current system
Most demanding users identified
 Definition of a few performance tests

26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
52
Performance tests, reliable Request-Reply
4000 msg/sec
Payload = 4B
Many small
messages
26th October 2011
5 msg/sec
Payload = 10MB
C++ server
with client in:
A few big
messages
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
53
Performance/Scalability tests, reliable Pub-Sub
400 msg x 8 B
10 clients, <50ms
Throughput
26th October 2011
30 msg x 8 B
10 clients, <20ms
Latency
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
54
Sync, async &
msg patterns
QoS
Dependencies
& memory f-p
Performance
Look & feel,
API, docs
Community &
maturity
Score
Products comparison (according to the criteria)
ZeroMQ






6
Ice






5
YAMI4






4
RTI






3
Qpid






3
CORBA






2
Thrift






2
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
55
Middleware Review Summary
Several good middleware solutions available.
 The choice is dictated by the most critical requirements
for any given application. Not easy  QoS policies and
performance matter, but also ease of use, community, …


Concerning CERN Controls Middleware…
 Prototype with the most
promising candidates:
Ice

ZeroMQ
YAMI
Deploy the new middleware before the long
accelerator shutdown at the end of 2012
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
56
Conclusions

Middleware is an important abstraction for building
the distributed systems

Synchronous vs. asynchronous communication

One-to-one vs. many-to-many communication

Better performance with direct communication

Better scalability and less coupling with Brokers

System requirements  Messaging patterns to apply
26th October 2011
Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...
57