Reflective Memory vs. Ethernet: Evaluating Data Network Solutions for LCLS Fast
Feedback Controls
Marya D. Pearson
Office of Science, Science Undergraduate Laboratory Internship Program
Norfolk State University
Stanford Linear Accelerator Center
Menlo Park, California
August 15, 2008
Prepared in partial fulfillment of the requirement of the Office of Science, Department of
Energy’s Science Undergraduate Laboratory Internship under the direction of Ernest L.
Williams Jr. and Sheng Peng in the Controls department at Stanford Linear Accelerator
Center.
Participant:
___________________________________
Signature
Research Advisor:
___________________________________
Signature
TABLE OF CONTENTS
ABSTRACT…………………………………………………………….……..3
INTRODUCTION……………………………………………………….…….4
CONTENT
Reflective Memory...…………………………………………….…….5
Ethernet………………………………………………….……………..6
DISCUSSION and CONCLUSION…………………….……………………..8
FUTURE WORK…….…...…………………………………………………...8
ACKNOWLEDGEMENTS…………………………………………………...9
LITERATURE CITED………………………………………………………..10
FIGURES and TABLES………………………………………………………11
2
ABSTRACT
Reflective Memory vs. Ethernet: Evaluating Data Network Solutions for LCLS Fast
Feedback Controls. MARYA PEARSON (Norfolk State University, Norfolk, VA 23504)
ERNEST WILLIAMS JR. and SHENG PENG (Stanford Linear Accelerator Center,
Menlo Park, CA 94025).
For reliable beam performance and X-ray Free Electron Laser delivery, the Linear
Coherent Light Source (LCLS) requires a feedback system. LCLS has software in place
for temporary use, but currently, no dedicated data network exists for feedback. While
software is an essential factor in the feedback system, the focus of this study is an
appropriate data network system that can support 120 Hz beam operation, provide
reliable data transfer, and remain scalable for future modifications. Reflective memory
and Ethernet are particularly interesting solutions for this task as they may provide
deterministic, scalable, and unique networking options. Reflective memory handles data
by simultaneously replicating and storing data to multiple memories in the network
architecture. Ethernet, a common local area network technology, transports data
according to MAC address and other higher-level protocol. A review of the advantages
and disadvantages of each data network solution was conducted based on cost,
performance, topology, and compatibility. Although no measurements were collected in
favor of either solution, this assessment suggests that Ethernet with multicast capability
will fulfill the performance requirements.
3
INTRODUCTION
High energy electrons travel the two-mile linear accelerator (linac) for physics
experiments. For high-quality research, the electron beam must remain stable in terms of
transverse and longitudinal position, bunch length, and beam energy to supply the X-ray
Free Electron Laser. Feedback is a technique for controlling these parameters through a
process of collecting measurements, evaluating algorithms, and producing instructions
for corrective action [3]. Furthermore, a physical connection via data network is needed
to facilitate the proposed feedback system.
The data network implemented in the linac must satisfy some constraints outlined
by LCLS physicists in the Physics Requirement Document (PRD). From this document,
the fast feedback network should aim to support 120 Hz beam operations, and provide an
architecture that is scalable, flexible, deterministic and reliable [3]. The PRD also
describes various feedback loops that will combine into one dedicated data network. For
simplicity, the injector launch feedback loop is the primary example. According to the
PRD, the injector launch consists of seven beam position monitors (BPM), four magnet
correctors, and a loop rate of 10 Hz. A 10 Hz loop rate translates to 100 ms for feedback
in the injector launch, however to support 120 Hz beam operation the system has an 8.3
ms time budget. With respect to the injector launch, the dedicated network for feedback
will be a connection from the BPM input/output controllers (IOC), a master feedback
IOC, and an actuator IOC (Refer to figures). The BPM IOC receives BPM data in ≤ 1ms,
and then passes the information to a feedback IOC where algorithms produce
instructions. The actuator IOCs, in this case magnet correctors, receive this data in ≤ 1ms
and respond to the within 6ms. Thereby, in approximately 8ms, the entire injector launch
4
feedback loop is complete. According to the 8.3 ms time budget, this time estimate
satisfies the outlined constraint, however, it hardly provides a cushion. By choosing an
appropriate network technology and architecture, the 1ms between BPM IOC, master
feedback IOC, and actuator IOC can be optimized and deliver the fast feedback for
LCLS.
Reflective Memory
Reflective memory (RM) is a memory bus technology in which data is replicated
and stored in local memory until it reaches its destination [6]. For example, if the BPM
IOC, master feedback IOC, and actuator IOC are connected using RM, the data from the
BPM IOC would be copied and stored at each node. Though this function can be done
speedily, approximately 40 ns latency each node, the disadvantage is that every node
connected with RM will receive the same data whether it needs the data or not; in this
case some of the actuator IOCs would receive unnecessary BPM data. Aside from this
inconvenience, RM offers a deterministic system with little software overhead. Since the
processor recognizes it as local memory, there is no need for scrupulous software to be
written to transfer the data. If this data runs into an error due to hardware failure, RM
offers built-in redundancy and error correction to avoid losing information. Another
consideration for RM is the system topology. Reflective memory can be configured in
either a star topology or a ring topology. The two topologies differ in the hardware
required for a connection. A star topology is connected using RM cabling with transmit,
receive ends, a RM hub that allows up to 256 nodes to connect, and a VMIPMC card that
houses the actual RM technology. A feedback system configured in the star topology is
very useful for scalability because the hub can support expansion; however, more
5
hardware is involved and therefore more cost. Ring topology connects each device
directly to the next using RM cabling. For example, a ring configuration between the
BPM IOC, master feedback IOC, and actuator IOC would attach the BPM IOC to master,
master to actuator IOC, and actuator IOC to BPM IOC. Scalability is limited here
because if a device should be added, the connection would have to be broken, but the ring
can also support up to 256 nodes. Each node, or device, in both configurations, must
have a RM card and RM cables to participate in the feedback loop. Considering the
fifteen feedback loops listed in the PRD, RM can be very costly. In fact, the cost for GE
Fanuc RM products for a star topology totals over $17k; a ring topology totals over $13k
[6]. With low latency, built-in redundancy, and high-volume data transport capabilities
RM is a viable solution for the LCLS feedback network. However, the low cost and
readily compatible Ethernet is a worthy competitor.
Ethernet
Over coaxial cables, Ethernet allows devices to communicate through a series of
protocol layers. Essentially, data is encapsulated with headers that specify the intrinsic
network routing [7]. Recently, Gigabit Ethernet has become available for transferring
speed up to 1000 megabits per second [1]. For LCLS feedback controls, Gigabit Ethernet
offers the speed that the system requires. Generally, Gigabit Ethernet is sufficient for this
project, but transporting data with Ethernet requires extensive programming that in turn
slows down the message being sent. Delayed messages affect determinism and latency,
and are therefore a major concern. An Ethernet based network has one major advantage
for the LCLS application. Each LCLS device has Ethernet capability ensuring a smooth
installation. All of the IOCs have two Ethernet network ports. One port is standard
6
EPICS for channel access which has unrestricted availability; the other port is presently
empty and available for use in a dedicated, private network [3]. BPM IOCs need an
additional network port to support a private Ethernet data network but this will only
minimally affect cost. Gigabit Ethernet could definitely be the hardware solution for
feedback but the architectural design issue still remains.
Point-to-point Ethernet works well as a local area network technology, but as for
the rigid needs of LCLS, point-to-point may not be able to deliver. The name is a quick
description of the technology involved in this network architecture. To set up point-topoint Ethernet, simply connect each device to the next using Ethernet cables. For
instance, a point-to-point connection between the BPM IOC, master feedback IOC, and
actuator IOC would connect the BPM IOC to master and master to actuator IOC. The
concern here is that the latency would increase exponentially with the number of nodes,
according to the equation for latency, 15n2+85n+3 [5]. Though it is probably the least
expensive option, determinism and flexibility are not guaranteed [1]. In this environment
a point-to-point architecture would not be appropriate.
Using Ethernet as the network backbone, and multicast technology to satisfy
broad, fast communication requirements is a favorable option. Multicast is a form of
selectively communicating data to many consumer devices [4]. On the physical layer,
Ethernet cabling connects all devices, then a switch, programmed to support multicast,
directs the data traffic. Further, the switch directs the traffic to a multicast group, or a set
of users authorized to receive the data, and creates a star topology that increases
scalability. For instance, if multiple feedback IOCs need information from a BPM IOC,
multicast arranges for all to receive the information, given they subscribe to the multicast
7
group and are connected to the multiple ports switch. Several other advantages apparent
in this technology are the flexibility of the architecture, bandwidth conservation, and
affordable cost. A scalable system design is possible at a rate of about $10.5k for a Cisco
Catalyst 4948 switch, plus the cost of additional Ethernet ports [2]. Performance
measures, such as total latency, collision detection, and failure recovery, are difficult to
estimate. The total latency is heavily dependent on the network driver that sends and
receives messages; however, Cisco switches average a latency of 6 microseconds per 64
byte package [2]. Between reflective memory and Ethernet, the decision for an
appropriate data network solution is intense but Ethernet with multicast capability could
satisfy the physics requirements at a more affordable cost.
DISCUSSION AND CONCLUSION
Evaluating reflective memory and Ethernet as data network solutions involves
comparing the technology, available products, and projected costs. With respect to the
system requirements, Ethernet with multicast is most compatible. Ethernet/multicast
delivers data with low-latency, and the star topology provides scalability. Unfortunately,
the limited determinism is a large disadvantage to this technology. The multicast
technology, however, improves Ethernet determinism by decreasing data package loss
and other data transfer errors. The ease of use and affordable hardware costs makes
Ethernet/multicast a practical solution. After evaluating each data network technology on
its cost, performance, topology, and compatibility, it was determined that Ethernet with
multicast is capable of delivering the fast feedback system LCLS requires.
FUTURE WORK
8
In order to concretely prove Ethernet with multicast as the best choice,
simulations must be done to measure performance. Before Ethernet/multicast can be
implemented as the data network for LCLS feedback, we will use Cisco network products
and collaborate with Cisco engineers to begin collecting performance measurements.
Theoretically, Ethernet/multicast is capable but with hard data we can assess the latency,
data transfer errors, and other functions realistically. Also, the comparisons were solely
based on the physics requirements and the injector launch feedback loop, so a
comprehensive design for the entire linac will follow.
ACKNOWLEDGEMENTS
This research was conducted at the U.S. Department of Energy Stanford Linear
Accelerator Center. I would like to express my gratitude to my mentors, Ernest Williams
Jr. and Sheng Peng. I am also thankful for the assistance of Farah Rahbar, Susan Schultz,
and Stephen Rock.
9
LITERATURE CITED
[1] Internetworking Technologies Handbook. 10 July 2008
<http://www.cisco.com/en/us/docs/internetworking/technology/handbook/ito_doc
.html>.
[2] "Introduction: Cisco 4900 Series Switches." Cisco Systems, Inc. 11 July 2008
<http://www.cisco.com/en/us/products/ps6021/index.html>.
[3] Krejcik, Patrick. Controls Requirements for LCLS Feedback Systems. Tech.No.
Stanford Linear Accelerator Center.
[4] "Overview of IP Multicast." Cisco Systems. 23 July 2008
<http://www.cisco.com/en/us/tech/tk828/technologies_white_paper09186a00800
92942.shtml>.
[5] Peng, Sheng. "Possible Fast Feedback solution." Powerpoint. Menlo Park. 25 June
2008.
[6] Real Time Networking with Reflective Memory. Tech.No. GE Fanuc Embedded
Systems. 2007.
[7] "The Link Layer and Local Area Networks." Computer Networking : A Top-down
Approach Featuring the Internet. By James F. Kurose and Keith W. Ross. New
York: Addison-Wesley Longman, Incorporated, 2004.
10
FIGURES
Injector Launch . Fast Feedback . Reflective Memory D1 . Star
Transmit Cable
Receive Cable
Channel Access port
Control/Data
Master Feedback IOC/Processor/Algorithm
BPM Private Ethernet
Timing/Event
VMIxxx5565
RM PMC Card
VMIxxx5565
RM PMC Card
Reflective Memory Hub 8 ports
VMIxxx5565
RM PMC Card
VMIxxx5565
RM PMC Card
VME
IOC:IN20:BP02
BPM Private Switch
IOC:IN20:MG01
MCOR
EIOC
EIOC
XC04
XC07
EIOC
EIOC
EIOC
EIOC
EIOC
YC04
YC07
BPM 9
BPM 10
BPM 11
BPM 12
BPM 13
BPM 14
BPM 15
Figures A: Reflective memory depicted in a star configuration based on the injector
launch.
11
Injector Launch . Fast Feedback . Reflective Memory D2 . Ring
Transmit Cable
Receive Cable
Channel Access port
Fast Feedback Ethernet
Control/Data
BPM Private Ethernet
Timing/Event
Master Feedback IOC/Processor/Algorithm
VMIxxx5565
RM PMC Card
VMIxxx5565
RM PMC Card
VMIxxx5565
RM PMC Card
IOC:IN20:MG01
MCOR
VMIxxx5565
RM PMC Card
VME
IOC:IN20:BP02
XC04
BPM Private Switch
XC07
YC04
EIOC
EIOC
EIOC
BPM 9
BPM 10
EIOC
BPM 11
EIOC
EIOC
EIOC
BPM 12
BPM 13
BPM 14
YC07
BPM 15
Figure B: Reflective memory diagram in a ring topology, based on the injector launch.
12
Injector Launch . Fast Feedback . Point-to-Point
Fast Feedback Ethernet
Channel Access
Control/Data
BPM Private Ethernet
Timing/Event
Master Feedback IOC/Processor/
Algorithm
Channel Access
IOC:IN20:MG01
MCOR
XC04
XC07
YC04
YC07
Channel Access
VME IOC:IN20:BP02
BPM Private Switch
EIOC
EIOC
EIOC
BPM 9
BPM 10
EIOC
BPM 11
EIOC
EIOC
EIOC
BPM 12
BPM 13
BPM 14
Figure C: Ethernet in a point-to-point configuration.
13
BPM 15
Injector Launch . Fast Feedback . Ethernet/Multicast
Master Feedback
IOC/Processor/
Algorithm
Fast Feedback Ethernet
Master Feedback
IOC/Processor/
Algorithm
Master Feedback
IOC/Processor/
Algorithm
Channel Access
Control/Data
BPM Private Ethernet
Timing/Event
BPM Multicast Group
Actuator Multicast Group
Cisco Switch 24-ports
IOC:IN20:MG01
Channel Access
Channel Access
VME IOC:IN20:BP02
BPM Private Switch
MCOR
EIOC
EIOC
EIOC
EIOC
BPM 10
BPM 11
BPM 13
BPM 14
BPM 15
Figure D: Ethernet/Multicast configuration, based on the injector launch.
14
YC04
BPM 9
YC07
BPM 12
EIOC
XC07
EIOC
XC04
EIOC
Figure E: Image of the linear accelerator showing how feedback will fit in [3].
15