s
17 January 2012 Dietrich Beck, EE(BEL)

• GSI Timing Team: Marcus Zweig, Stefan Rauch, Mathias Kreider,
Cesar Prados, Wesley Terpstra, Ralph B är, Dietrich Beck former members: Tibor Fleck, Sergio Mauro
• CERN Timing Team: Tomasz Włostowski, Javier Serrano, Maciej
Lipinski, Evangelia Gousiou, Erik van der Bij, Jean-Claude Bau,
Pablo Alvarez
• GSI/EE: Jan Hoffmann, Nikolaus Kurz, Holger Brand, …
17 January 2012 Dietrich Beck, EE(BEL)

17 January 2012 Dietrich Beck, EE(BEL)

17 January 2012 Dietrich Beck, EE(BEL)
23 November 2011

17 January 2012 Dietrich Beck, EE(BEL)
8 December 2011

17 January 2012 Dietrich Beck, EE(BEL)
12 January 2012

17 January 2012 Dietrich Beck, EE(BEL)
12 January 2012

• parallel execution of beam production chains
• cycles: 20ms to hours
• trigger and sync. equipment actions
• 1 µs precision in 99% of all cases
• few ns precision for kickers
• (ps for rf-systems: BuTiS)
• many rings
• > 2000 devices connected to timing system
• large distances
• robustness: lose at most one message per year
17 January 2012 Dietrich Beck, EE(BEL)

Not:
Timing Master do this
Equipment 1 do this
1km distance:

5
 s propagation time
1
 s precision: requires compensation for cable length 
Equipment 2
17 January 2012 Dietrich Beck, EE(BEL)

"Oh dear! Oh dear! I shall be too late!"
Instead: ???
do this
@ 15:03
Timing Master Equipment 1 do this
@ 15:03 do that
@ 15:01
Equipment 2
• equipment pre-programmed for autonomous action at a given time
• pre-programming via timing-events
• required: clock synchronization
 ns required
 distribution of information and execution of action are decoupled !!!
• timing-events must be sent “early enough”
• upper bound latency for transmission (e.g. 100  s): real-time!
• lossless transmission: robustness!
17 January 2012 Dietrich Beck, EE(BEL)

• network: Gigabit-Ethernet
• PTP (Precision Time Protocol) IEEE1588-2008 with value t
1
.. t
4 one can
• calculate one-way link delay
• syntonize clock rate by tracking the value
• calculate clock offset
17 January 2012 Dietrich Beck, EE(BEL)

• network: Gigabit-Ethernet
• PTP (Precision Time Protocol) IEEE1588-2008
– free-running oscillators on network nodes
– need to re-sync often: lot’s of traffic, bad for determinism
– only 1-100  s synchronization of clocks
• SyncE (Synchronous Ethernet)
– receiver’s clock recovered from 125MHz carrier
– 8ns precision
• precise phase measurement and adjustment
• clock synchronization with sub-ns precision and low-ps jitter
17 January 2012 Dietrich Beck, EE(BEL)

multi-layered
WR Switches
• Clock propagation:
• Determinism
GPS
System
Timing
Master
– “timing events shall arrive early enough”
– upper bound latency from timing master to nodes
– no handshake: use UDP or raw Ethernet (no TCP)
– switches implement cut-through for high-priority messages
• Robustness
– bit errors: Hamming code
– packet loss: Reed-Solomon code
• Redundancy
– redundant links between switches
– link aggregation (in combination with Reed-Solomon)
WR
Nodes
17 January 2012 Dietrich Beck, EE(BEL)

• path asymmetry (two cables) impacts phase compensation
 White Rabbit PTP
– one bidirectional single-mode optical fiber
– different wave lengths for TX/RX: speed differs slightly (dispersion)
– link-delay-model
• relative delay coefficient α depends only on the type of fiber (off-line)
• fixed delays are measured on-line
• White Rabbit node
VCO 125MHz
VCO 20MHz
17 January 2012 transceiver
WR clock!!!
FPGA host bus interface
Dietrich Beck, EE(BEL)

FPGA within FPGA: Wishbone “System-on-Chip” bus for interconnection of IP cores
17 January 2012 Dietrich Beck, EE(BEL)

• idea: Connect distant Wishbone buses via a network protocol layer. Ethernet + Wishbone = Etherbone
• direct memory access to devices via White Rabbit network
• used for timing-events
17 January 2012 Dietrich Beck, EE(BEL)

• development in the frame of CERN’s renovation and the FAIR projects
• collaboration: CERN, GSI, …, industry partners
• Open HardWare Repository: www.ohwr.org
– Open Source: Software and HDL
– “CERN Open Hardware License”: Hardware
• commercial production and support
“The White Rabbit (WR) project is a multi-laboratory and multi-company effort to bring together the best of the data transfer and timing worlds …” (Proc. ICALEPCS 2009)
17 January 2012 Dietrich Beck, EE(BEL)

General Machine Timing System
• based on White Rabbit
• fairly cheap: € 600,- SPEC board
• sub-ns synchronization
• distribution of
• (clock)
• time-stamps
• timing-events
• BEL group
• one global timing master
• clock derived from BuTiS
Talk T. Fleck, FAIR Technikforum
Bunch phase Timing System (BuTiS)
• invest € 15k per receiver station
• ps precision (100ps/km accuracy)
• distribution of
• clocks!!!
• NO time-stamps at FAIR
• NO timing-events
• HF group
• one global BuTiS center
Talk P. Moritz, FAIR Technikforum
Remark: If you are interested time-stamps, you need White Rabbit.
If you are interested in the most accurate clocks, you need BuTiS.
If you are interested in both, you need both.
17 January 2012 Dietrich Beck, EE(BEL)


Settings
Management
1a: Data supply: set-values
1b: Data supply: schedules, alternatives
Front-end
Controller
Front-end
Controller
Timing Master
2: broadcast timing-events
3. timing receiver
Timing
1.
settings management (via normal network) a.
preloads set-values to the FECs (setvalue can also be a ramp…) b.
provides schedule to timing-master
2.
according to schedule, timing-master broadcasts timing-events to FECs
(timing-events: time of execution and reference to a set-value)
3.
timing-receiver of FEC: timely triggering of FEC, FEC applies set-value
17 January 2012 Dietrich Beck, EE(BEL)


Settings
Management
Data supply: set values
Data supply: schedules, alternatives
Timing Master
Front-end
Controller
Front-end
Controller timing receiver
Timing
Analogy to performing a classical piece of music:
• composer: settings management
• conductor: timing master
• musicians: FECs
• move of conductor's baton: timing-events - one timing-event may trigger multiple FECs
17 January 2012 Dietrich Beck, EE(BEL)

Control System (application layer)
Post Mortem System
LSA (settings management)
Timing Master
BuTiS
Beam Interlock System
Common
Systems
Alarm, Archive,
Logging,
Oscilloscope
Equipment 1 Equipment 2 Equipment 3 Equipment 4 Equipment ..
Equipment N
Disclaimer: This is my personal view and may be incorrect and not up-to-date
17 January 2012 Dietrich Beck, EE(BEL)

17 January 2012 Dietrich Beck, EE(BEL)

• campus wide distribution system for
– timing-events
– time-stamps
– other machine or beam related telegrams
• not limited to accelerator controls
• available to “everyone”
• typically, data is propagated top-down
• other traffic only in exceptional cases
– kicker control
– bunch-to-bucket transfer
– beam interlock notification
17 January 2012 Dietrich Beck, EE(BEL)

• general purpose communication
• transmission of bulk data
• personnel protection system
• things not related to machine timing
Exceptions only in well justified cases
• FECs not connected to FESA
• Info-Service
• part of machine protection system
17 January 2012 Dietrich Beck, EE(BEL)

@GSI: Set-up a first network 2012
• usage in a field environment
• investigation of stability and long-term effects
• connection between new and existing timing system
• first tests with existing machines at GSI
17 January 2012 Dietrich Beck, EE(BEL)

@GSI/CERN: WR starter kit (timing system demo) spring 2012
@Saclay: Low energy part of p-linac: first WR based timing system provided by GSI (early 2013)
@CERN:
• Installation of WR based timing system at AD ring (2013)
• Installation of WR link between CERN and Gran Sasso (2012)
17 January 2012 Dietrich Beck, EE(BEL)

• Slovenia: In-Kind contribution timing receivers. Receivers and
FPGA building blocks for FAIR timing system specified.
• Germany/GSI: In-Kind contribution General Machine Timing
System: specification until March 2012.
• White Rabbit PTP link between two nodes established
• sub-ns synchronization with a jitter of  hundred ps
• propagation of WR PTP over three layers of switches
• FPGA building blocks for simple timing system exists
• integration work in progress
• planning of WR network cabling in progress
• First WR nodes developed by EE
17 January 2012 Dietrich Beck, EE(BEL)

• www.ohwr.org
• https://wwwacc.gsi.de/wiki/Timing/TimingSystemDocumentation
(search for “FAIR timing system”)
17 January 2012 Dietrich Beck, EE(BEL)

17 January 2012