Felix Ehm
CERN BE-CO
Content

Introduction

JMS in the Controls System

Deployment and Operation

Conclusion
Introduction to JMS – The API

Java Messaging Service
 Messaging API defined in 2002
 Decouples Source and Destination
 Topic / Queue Concepts for named Destinations
Introduction to JMS – A Broker

Technology originated from the financial sector
 “Few” sources and large set of readers

A Broker is the central entity
 Accepts messages from 1..* publisher and
Broker
re-distributes to 0..* subscribers
 Provides Quality of Service for Messages
○ Persistency, Transactions, Buffering, Expiration Policies,
Slow Client Handling
 Design to distribute data reliably and scalable
 Load balancing / fail-over / embedded
deployment scenarios possible
Introduction to JMS - A Broker

Many broker implementations available
 Proprietary
IBM WebsphereMQ, SonicMQ, TIBCO, RedHat MRG, …
 OpenSource
Apache ActiveMQ, Apache Apollo, RabbitMQ, HornetMQ, …

Broker may be implemented in another language
than Java (e.g. Qpid & RabbitMQ)

Communication protocol is non-standard
 Brokers of different vendors are not interoperable
 AMQP & STOMP address this issue
Introduction to JMS - Request/Reply

“Direct communication” via Broker using
Request – Reply mechanism
Requestor
sends
creates
Broker
Request
MSG
Request Queue
Temporary Reply
Queue

Replier
Reply
MSG
sends
Due to intermediate Broker two hops are required
Introduction to JMS - Selectors

 Each message can be enriched
CYCLE=A
TIME=9845
with Attributes
 Subscriber receives filtered
messages via defined selector
Broker
BUILDING=A
BUILDING=B
BUILDING=C
Topic/
Queue
BODY
selector:
BUILDING ≠ A
Subscriber
JMS Message
Filter messages from one Destination using
Messages Selectors
HEADERS
Introduction to JMS - Persistency
JMS defines persistent and non-persistent
messages
1. send
 Broker stores message to
Sender
disk before acknowledging
2. store
3. ack
to sender

Avoids loss of data when broker crashes
 Drawbacks

 Higher disk load
 Slower message processing
CERN’s Accelerator Complex
CERN’s Accelerator Complex
JMS in the Controls System

Purpose
 Reliable transport of data between (Java) Processes
 High level controls applications + middle tier servers
○ Alarm System
○ Data Rendering Services
○ (Control) GUIs
○ Logging Services
○ Beam Security System(s)
○ Access Systems, ….
No JMS No Beam !
Today vital part of Beam Operations
 Product choice:

JMS in the Controls System - History

History
 SonicMQ since 2001 for CERN’s Alarm
and CO Monitoring system
 OpenSource ActiveMQ since 2005
○ Middle tier servers update GUIs
○ Middle tier to Middle tier
Phased out
in 2013
JMS for Accelerator Controls

Early days only Java clients via JAPC
 japc-ext-remote contains JMS communication

Today multi-language clients
 JAPC clients remain
 Java using ActiveMQ client libraries directly
 C++/Python using STOMP
Deployment and Operation
29 production brokers
 Single or broker cluster


Large spectrum of usage patterns
 from few topics to 40K topics
 from few subscribers to >23 K subscribers
 from couple msg / minute to 10K msg / sec
 from few bytes/msg to 3 Mbyte/msg
Deployment and Operation
Some numbers :
 300 Applications
 4’400 Connections
 40’000 Subscriptions
 85’000 Topics
 8 Million msg/h IN,
3.5M msg/h OUT
Archived Uptime
in 2012: 99.998%
Not all data which
is produced is also
consumed
Deployment and Operation

Distribute Load
 Interconnect two Brokers
 Consumers/Producer can
choose any Broker
 Sharing of work

Smooth Upgrades possible
 Client library version != Broker version OK

Drawbacks
 Consumer might not get all messages in case of
network error
 Routing overhead
The STOMProtocol
The STOMProtocol
Simple Text Oriented Message Protocol
 Defines messaging API + wire format
 Many open source clients libraries available

 Python, C/C++, Java, Perl, Ruby, PHP, Erlang, …
Targets the messaging interoperability among
language, platforms and brokers
 Supported by many brokers
 Very easy to use

Example to send STOMP in Python
mysocket = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
mysocket.connect( ("jms-co-dev", 61680) )
text = '''SEND\n\
destination:/queue/CERN.DEPLOYMENT\n\
HOST:%s\n\
INSTALLPATH:%s\n\
PRODUCT:%s\n\
TIME:%s\n\
persistent:false\n\n\x00''‘
mysocket.sendall(text)
mysocket.sendall("DISCONNECT\x00")
mysocket.close()
The AMQProtocol
The AMQProtocol
Advanced Message Queue Protocol
 Defines messaging API + wire format
 Next evolution after JMS

 Consortium of RedHat, Credit Suisse, Microsoft,
JPMorgen, Barclays, VMWare, Cisco, …

Released as Version 1.0
 More and more brokers implement it

Direction for the future ?
Conclusion

JMS is vital part of Accelerator Controls System

Performs very well for publish - subscribe scenarios

ActiveMQ was a good choice
 Running reliably
 Low maintenance and flexible deployments
 Provides many metrics for diagnostic and monitoring

Multi-language applications can use service

Central (redundant) service
Result Example
Messages received/sec over time
AVG Messages / sec

time
Result Example
Histogram of delayed message distribution
# Messages

Delay time [msec]
Result Example
Number of “late” (>300ms) messages for test
duration
Delayed Messages

time
Message Delay Summary

Measuring delayed Message
 ~ 99.32% of messages delivered < 300ms
 Rest < 1 sec

Doubling numbers of messages
(~9500 msg/sec)
 100% between 5 and 10 seconds
 Further studies for improvement needed
Result Example

CPU load
Consumer
disconnects
% CPU load
Consumer
connects
time
40% less CPU load thanks to optimization
Broker Memory Consumption

Initial Memory is proportional to
 Connections
 Subscriptions
1000 Subscription / Connection
500 Subscriptions / Connection
400
800
350
700
300
600
250
500
200
400
150
300
100
200
50
100
0
0
0
5000
10000
Heap Memory Usage
20000
30000
Live Threads
40000
0
10000
20000
Heap Memory Usage
40000
60000
Live Threads
80000