The LCLS Timing & Event System
- An Introduction –
John Dusatko / Instrumentation & Controls - EIE
December 7, 2011
Introduction to the LCLS Timing System
1
John Dusatko
jedu@slac.stanford.edu
Outline
1.
2.
3.
4.
5.
Introduction to LCLS
Some background on SLAC Timing
The SLAC Linac Timing System
The LCLS Timing System
Comments / Future Directions
December 7, 2011
Introduction to the LCLS Timing System
2
John Dusatko
jedu@slac.stanford.edu
Introduction
December 7, 2011
Introduction to the LCLS Timing System
3
John Dusatko
jedu@slac.stanford.edu
LCLS Introduction
The Linac Coherent Light Source is a tunable Xray FEL based on the SLAC Linac:
1.0nC, 14-4.3 GeV e- are passed thru an undulator, a
Self Amplifying Stimulated Emission process produces
1.5-15 Angstrom X-Rays.
LCLS is an addition to the existing SLAC Linac: it
uses the last 1/3 of the machine
►This is important to note because we have to integrate
the New LCLS Timing System with the Existing Linac
(SLC) Timing System.
December 7, 2011
Introduction to the LCLS Timing System
4
John Dusatko
jedu@slac.stanford.edu
The LCLS – Schematic View
(ignoring photon beamline)
Single bunch, 1-nC charge, 1.2-mm slice emittance, 120-Hz repetition rate…
6 MeV
z  0.83 mm
  0.05 %
250 MeV
z  0.19 mm
  1.6 %
Linac-X
L =0.6 m
rf= -160
4.54 GeV
z  0.022 mm
  0.71 %
135 MeV
z  0.83 mm
  0.10 %
rf
gun
Linac-1
L 9 m
rf  -25°
Linac-0
L =6 m
...existing linac
21-1b
21-1d
DL-1
L 12 m
R56 0
X
Linac-2
L 330 m
rf  -41°
Linac-3
L 550 m
rf  -10°
21-3b
24-6d
25-1a
30-8c
BC-1
L 6 m
R56 -39 mm
SLAC linac tunnel
BC-2
L 22 m
R56 -25 mm
14.1 GeV
z  0.022 mm
  0.01 %
undulator
L =130 m
LTU
L =275 m
R56  0
research yard
(RF phase: frf = 0 is at accelerating crest)
December 7, 2011
Introduction to the LCLS Timing System
5
John Dusatko
jedu@slac.stanford.edu
December 7, 2011
Introduction to the LCLS Timing System
6
John Dusatko
jedu@slac.stanford.edu
LCLS Timing – Some Definitions
The LCLS Timing System can be viewed as
consisting of three parts:
Part 1: ‘Standard’ Accelerator Timing
10ps Triggers for Acceleration and Diagnostics
Part 2: S-Band Timing
2856MHz LCLS RF Phase Reference Distribution
Part 3: Ultra-Precise Timing
10fs Synchronization for Experiments (LBL System)
December 7, 2011
Introduction to the LCLS Timing System
7
John Dusatko
jedu@slac.stanford.edu
LCLS Timing – Some Definitions
The LCLS Timing System can be viewed as
consisting of three levels:
Part 1: ‘Standard’ Accelerator Timing
10ps Triggers for Acceleration and Diagnostics
‘Triggers’ are signals from the timing system used by
HW to accelerate & measure the beam
Part 2: S-Band Timing
2865MHz RF Phase Reference Distribution
Part 3: Ultra-Precise Timing
10fs Synchronization for Experiments (LBL System)
December 7, 2011
Introduction to the LCLS Timing System
8
John Dusatko
jedu@slac.stanford.edu
LCLS Timing – Performance Requirements
LCLS Timing System Requirements
Maximum trigger rate:
360 Hz (120Hz)
Clock frequency:
119 MHz
Clock precision:
20 ps
Delay Coarse step size:
8.4 ns ± 20 ps
Delay range;
>1 sec
Fine step size:
20 ps
Max timing jitter w.r.t. clock;
10ps rms
Differential error (skew), location
to location:
8 ns
Long term stability:
20 ps
Signal Level:
December 7, 2011
Introduction to the LCLS Timing System
9
TTL
John Dusatko
jedu@slac.stanford.edu
Requirements Comparison
Timing Reqmts for earlier SLAC systems:
PEP – II:
Original Linac:
(~1968)
(~1998)
- Resolution: 50 nSec
- Resolution: 2.1 nSec
- Jitter: 15 nSec
- Jitter: 20 pSec
- Main Trigger Line
- NIM Level
Waveform:
Waveform:
+ / - 400 Volts
0 to –0.7 V into 50 Ohms
December 7, 2011
Introduction to the LCLS Timing System
10
John Dusatko
jedu@slac.stanford.edu
Background
December 7, 2011
Introduction to the LCLS Timing System
11
John Dusatko
jedu@slac.stanford.edu
SLAC’s Timing Systems
In order to explain the New LCLS timing system,
we first need to understand how the old SLAC
timing system works – i.e. how we got from there
to here
The SLAC Accelerator complex consists of several
machines: Linac, Damping Rings, Stanford Linear
Collider, PEP-II, FFTB, NLCTA / each with its own
timing sub-system
►The overall timing system consists of incremental addons to the original system
►Design Challenge for LCLS Timing System was that it
had to know about and work with the existing system
December 7, 2011
Introduction to the LCLS Timing System
12
John Dusatko
jedu@slac.stanford.edu
(pre-LCLS) SLAC Accelerator Complex
(Lots of Pieces)
December 7, 2011
Introduction to the LCLS Timing System
13
John Dusatko
jedu@slac.stanford.edu
SLAC Linac Timing System
December 7, 2011
Introduction to the LCLS Timing System
14
John Dusatko
jedu@slac.stanford.edu
Old SLAC Timing System
We’ll talk a little about the existing SLAC Linac
Timing System (PEP-II Timing was very similar):
The Linac is a Pulsed Machine (get a packet of
beam per pulse) runs at a max of 360Hz
Three Main Timing Signals:
476MHz Master Accelerator Clock (runs down 2mile
Heliax Main Drive Line cable)
360Hz Fiducial Trigger (used to ‘tell’ devices when the
beam bunch is present) / encoded onto the 476MHz
master clock
128-Bit PNET (Pattern Network) Digital Broadcast
(contains trigger setup, beam type & rate information)
December 7, 2011
Introduction to the LCLS Timing System
15
John Dusatko
jedu@slac.stanford.edu
Some More Details
Why 360Hz?
Original design rate of the Linac / derived from the 3phase, 60Hz AC power line frequency: want to trigger
devices (Klystrons, etc.) consistently so as to not create
huge transients on the Power Line Sync’d to 476MHz
PNET Broadcast
A special computer called the Master Pattern Generator
(triggered by the 360Hz fiducial) broadcasts a 128-bit
digital message containing information (conditions, rate,
charge, etc.) about the beam
This is used by the Trigger Generator computers to set
up triggers and their delays
Sent over SLAC coax cable TV network
December 7, 2011
Introduction to the LCLS Timing System
16
John Dusatko
jedu@slac.stanford.edu
How The 360Hz is Generated
The Sequence Generator
creates a 360Hz signal as well
as 6 Timeslot pulses (used for
further synchronization)
Source: SLAC
Blue Book
c.1962
December 7, 2011
Introduction to the LCLS Timing System
17
John Dusatko
jedu@slac.stanford.edu
Timing HW at HeadEnd of Linac
For Phase
Stabilization
Source: D. Thompson
December 7, 2011
Introduction to the LCLS Timing System
18
John Dusatko
jedu@slac.stanford.edu
Generation of The Linac Timing Signal
(AM Modulator)
The 360Hz Signal is Amplitude Modulated
onto the 476MHz Accelerator Clock and
propagated down the Main Drive Line.
The AM process is not ideal and some FM
occurs; in addition, the signal gets more
dispersed as it heads down the 2 mile MDL
December 7, 2011
Introduction to the LCLS Timing System
19
John Dusatko
jedu@slac.stanford.edu
Linac RF Phase Reference Distribution
This slide is to give you an idea of how the Linac Phase Reference is used
Each sector (30 total) taps off the MDL, to extract the RF clock
This is just the Phase Ref, trigger generation is accomplished by
a different set of HW
December 7, 2011
Introduction to the LCLS Timing System
20
John Dusatko
jedu@slac.stanford.edu
CAMAC Trigger Generation
Timing CAMAC Crate
Old Timing System (CAMAC
based) generates triggers by
combining the RF Clock,
360Hz Fiducial and PNET
Data (processed by local
micro)
 476MHz is divided/4 to get
119Mhz + fiducial by another
system / This is because the
older technology HW could
not run at 476MHz
PNET Data on
Serial Link
 The Programmable Delay
Unit (PDU) Module generates
the triggers. It contains digital
counters that get set with
delay values from PNET and
started when the Fiducial
Pulse comes along
December 7, 2011
Introduction to the LCLS Timing System
21
John Dusatko
jedu@slac.stanford.edu
The LCLS Timing System
December 7, 2011
Introduction to the LCLS Timing System
22
John Dusatko
jedu@slac.stanford.edu
Finally – The LCLS Timing System
Old CAMAC System is no longer viable for new
Systems (performance limited, obsolete)
Seek to implement a new Timing System that has
similar functionality, better performance, and can
be laid atop the old system, working alongside it
In addition, LCLS has have its own master
oscillator (PLL sync’d with Linac MO) and local
phase reference distribution system at S20
►LCLS System is VME based, using High-Speed
digital serial links to send Clock, Trigger and Data
all on one optical Fiber to timing clients
December 7, 2011
Introduction to the LCLS Timing System
23
John Dusatko
jedu@slac.stanford.edu
The LCLS Timing System
New Timing HW:
Event Generator (EVG): Similar to the MPG
Event Receiver (EVR): Similar to the PDU
Timing is more closely integrated into each subsystem:
Each Subsystem has its own EVR (living in VME crate
with an IOC)
Compared to SLC timing which had multiple types of
devices served by one PDU typically
LCLS Timing is architected as a Star network with
one master broadcasting to many clients
December 7, 2011
Introduction to the LCLS Timing System
24
John Dusatko
jedu@slac.stanford.edu
Timing Compare/Contrast
LCLS Timing:
SLC Timing:
VME Based
Fiber data & clk cable @
2.38Gb/s
EVR Triggers:
CAMAC Based
Copper data cable @ 5Mb/s
PDU Triggers:
NIM-Level
Width: Fixed @ 67ns
Polarity: Fixed NIM level
Delay Range: 0…2.7ms (5.4ms)
Step size: 8.4ns (100ps w/VDU)
16 outputs per PDU
TTL-Level (can have others)
Width: Adjustable: 8.4ns…550us
Polarity: Selectable Pos/Neg
Delay Range: 0…36 seconds
Step size: 8.4ns (420ps w/EVRRF)
14 outputs per EVR
One PDU per CAMAC crate
SCP-Based
December 7, 2011
Introduction to the LCLS Timing System
Can have multiple EVRs per
VME Crate
EPICS-Based
25
John Dusatko
jedu@slac.stanford.edu
LCLS RF Front End
December 7, 2011
Introduction to the LCLS Timing System
26
LCLS Master Osc –
slaved to Linac MO /
Lower Phase noise
req’d by LCLS
John Dusatko
jedu@slac.stanford.edu
LCLS Timing/Event System Architecture
Linac main drive line
Low Level RF
~
TRD
FIDO
PDU
Fiber Cable
Linac
Master Osc
Raw 360 Hz
LCLS Master 476 Sync/Div
MHz
Oscillator
119Mhz + FID
Sq Wave on Coax
119Mhz + FID
Sq Wave on Coax
TO:
- Cav BPM
- MPS BLM
- MPS PIC
- BCS
TRD
Rx
P
E
I
N SLC
LCLS
O
V
events
E events
C
G
T
*
F
A
N
EPICS Network
*MicroResearch
TRD
Tx
360 Hz
119 MHz
SLC
MPG
Rx
LCLS Timeslot Trigger
P P
m N D
P E U
T
December 7, 2011
Introduction to the LCLS Timing System
System is based
around the EVent
Generator and EVent
Receiver
27
fiber
distribution
I E
O V
C R*
Precision<10 ps
D
E
V
TTL
TTL-NIM
convert.
SLC Trigs
John Dusatko
jedu@slac.stanford.edu
Digitizer
LLRF
BPMs
Toroids
Cameras
Wire Scanner
SLC klystrons
LCLS Timing/Event System Architecture
Linac main drive line
~
TRD
FIDO
PDU
Fiber Cable
Linac
Master Osc
Rx
Raw 360 Hz
LCLS Timeslot Trigger
LCLS Master 476 Sync/Div
MHz
Oscillator
TRD
Tx
119Mhz + FID
Sq Wave on Coax
119Mhz + FID
Sq Wave on Coax
360 Hz
119 MHz
TO:
- Cav BPM
- MPS BLM
- MPS PIC
- BCS
TRD
Rx
360Hz IRQ (Fiducial)
60Hz Timeslot 1 Marker
V
C
E
M IRQ
LCLS
P
V
T Timeslot
events
U
G
G
*
F
A
N
Fiber
distribution
EPICS Network
C E
P V
U R
System Upgraded:
 Removed Dependency on old
MPS using new HW (VMTG) and SW
December 7, 2011
Introduction to the LCLS Timing System
28
*
D
E
V
TTL
TTL-NIM
convert.
John Dusatko
jedu@slac.stanford.edu
Digitizer
LLRF
BPMs
Toroids
Cameras
Wire Scanner
Klystrons
The Event System
Based on Commercial Hardware (MicroResearch Finland)
Which was based on a design from ANL-APS Timing System
Designed Around Xilinx Virtex-II FPGAs
Uses FPGA’s internal High-Speed 2.38 Gb/s Serial xcvr,
which connects to a fiber optical transceiver
FPGA logic implements all of the timing functions
Uses 119MHz clock from Linac which get multiplied up to
by FPGA internal PLL to 2.38GHz
EVG sends out 8-bit event code to EVRs along with clock
and trigger information over one fiber
EVR Receives event code & with its associative memory,
generates a trigger with a delay set by digital counters
December 7, 2011
Introduction to the LCLS Timing System
29
John Dusatko
jedu@slac.stanford.edu
The Event System
119MHz RF Clock
HW
Triggers
From SLC Timing
System
360Hz Fiducial
Fiber
Fanout(s)
EVG
128-bit PNET Pattern
EVR
PNET
EPICS TS
MPS Data
MPS Signals
One EVG Per System
Data Frame:
2 Bytes @ 119MHz
Timing Event
Code Byte
EVR
EVR
8-bit
Distributed
Databus /
Data Buffer
Bus
PNET
EPICS Timestamp
►Upon RX’ing a 360Hz Fid, the EVG
sends out a stream of serial Data to the
EVRs over a fiber link. The serial
stream consists of 16-bit words sent
every 8.4ns.
8-bit
Event Code
[Note: there can be multiple
EVGs in one system,
arranged in a daisy-chain
fashion with priority
encoded data tranmission]
EVR
►Each word contains an Event Code
byte and some trigger setup data
Event System Architecture
December 7, 2011
Introduction to the LCLS Timing System
30
John Dusatko
jedu@slac.stanford.edu
Event Generator
December 7, 2011
Introduction to the LCLS Timing System
31
►EVG contains a RAM
that gets loaded with
event codes, based on
PNET data. 360Hz
Fiducial causes the
RAM to get sent out
over the Serial fiber link
to the EVRs.
John Dusatko
jedu@slac.stanford.edu
Event Receiver
►EVR contains
another RAM that looks
for matches of event
codes to its contents. If
match, it starts a
counter running that
generates a trigger
December 7, 2011
Introduction to the LCLS Timing System
32
John Dusatko
jedu@slac.stanford.edu
EVG Hardware
Event Generator
VME-64x Module: Sits in Master Timing Crate with
VME PNET receiver and Master Timing CPU.
 Receives 119MHz reference and 360Hz master
timing fiducial from SLC timing system.
 Receives PNET pattern from SLC system.
 Broadcasts timing system data in the form of a
high-speed 2.38Gb/s serial data stream to the
EVRs over an optical fiber.
December 7, 2011
Introduction to the LCLS Timing System
33
John Dusatko
jedu@slac.stanford.edu
EVR Hardware
Event Receiver
Comes in two flavors: VME and PMC.
Receives 2.5Gb/s serial datastream
from EVR and generates triggers based
on values of the event codes.
Also receives and stores PNET timing
pattern, EPICS timestamp and other data
and stores them in an internal data
buffer.
Can output 14 total pulsed-output
triggers and several more level-type
Triggers are output via a rear transition
module (not shown)
Trigger delay, width, duration and
polarity are fully programmable
Trigger signal level format is TTL
VME Version has 10ps jitter
PMC Version has 25ps jitter
December 7, 2011
Introduction to the LCLS Timing System
34
John Dusatko
jedu@slac.stanford.edu
Fanout Hardware
Fiber Fanout Module
Basic function is to receive one optical
signal and generate copies of the same
signal for transmission.
1:12 way fanout (1 In  12 Out)
Contains Clock-Data Recovery (CDR)
circuitry to re-generate & clean-up the
signal
Fits into a VME crate, but ONLY uses
power
Uses commercial SFP (Small Formfactor Pulggable) optical xcvrs
SFP units are hot-swappable
Front Panel LEDs indicate link health
No other diagnostics than this
There is one of these modules at each
critical distribution point in the system / if
one module fails, it takes timing out for a
large group of clients
December 7, 2011
Introduction to the LCLS Timing System
35
John Dusatko
jedu@slac.stanford.edu
System SW
December 7, 2011
Introduction to the LCLS Timing System
36
John Dusatko
jedu@slac.stanford.edu
What Happens during one 2.8mS machine interval:
Record processing (event, interrupt)
Hardware Triggers
Triggering
Event Codes
Event Timestamp, Start
pattern records,
and BSA ready
Receive pattern for
3 pulses ahead
Kly Standby Beam
Acq
Kly Accel Trigger
Fiducial Event
Received
Fiducial
B0
F3
0
18
0.3 100
December 7, 2011
Introduction to the LCLS Timing System
500
37
1023
John Dusatko
jedu@slac.stanford.edu
LCLS Systems – Master Timing Rack
 Located in LI20 RF Hut (Rack LKG-21)
Master FODU
Connects fibers to Long-Haul
Trunks for entire machine
Master Timing Crate
Contains:
• VME CPU
• VME PNET Rx
• EVG
• Master Fanouts
119MHz Synchronizer Chassis
December 7, 2011
Introduction to the LCLS Timing System
38
John Dusatko
jedu@slac.stanford.edu
LCLS Timing System – BPM Client
 What you’d see in a typical service building (e.g. B105) …
BPM Crate w/VME-EVR
Rx FODU & Fanout Crate
Rear of BPM Crate /
Showing Trigger Rear
Transition Module
December 7, 2011
Introduction to the LCLS Timing System
39
John Dusatko
jedu@slac.stanford.edu
LCLS Timing System – Other Clients
Toroid Crate w/PMC-EVR
Profile Monitor
Crate w/ (4)
CPUs & PMCEVRs
MCOR Magnet
Crate
Rear of Toroid Crate /
Showing Trigger Rear
Transition Module
December 7, 2011
Introduction to the LCLS Timing System
40
John Dusatko
jedu@slac.stanford.edu
Performance
 The Event System Trigger Jitter
was measured using an Agilent
Infinium 54845A Digital
Oscilloscope in Jitter Histogram
Mode / data was collected for 30
minutes
EVR jitter
w.r.t. fiducial
The EVR output (shown) was
measured against the system input
trigger (360Hz fiducial)
The Actual jitter performance is
much better after subtracting off the
intrinsic jitter of the scope:
9 ps rms jitter
Actual Jitter:
JITTERsystem = [ (JITTERsys_meas)2 – (JITTERscope)2 ]1/2
Event System Jitter:
JITTEREVR =[ (9.7472ps)2 – (6.3717ps)2 ]1/2
= 7.3763 ps
December 7, 2011
Introduction to the LCLS Timing System
41
John Dusatko
jedu@slac.stanford.edu
Comments
Current LCLS Timing System has growing
pains (integration w/old system, SW, HW)
Use of commercial HW doesn’t quite fit our
needs / having to modify EVG & EVR
Problems Seen Thus Far:
“Trigger Storms”: Due to LCLS Master Osc
unlocking / Fix: New MO / De-Couple LCLS
Timing Sys from it (connect direct to MDL)
Dead Links: Fibers / Xcvrs / Still
Troubleshooting
December 7, 2011
Introduction to the LCLS Timing System
42
John Dusatko
jedu@slac.stanford.edu
LCLS Timing – Future Directions
MPG Replacement:
Planning on Moving the EVG Master Crate to MCC and Replacing
the MPG Micro With it  COMPLETED Summer 2011
Additional Diagnostics:
Designed & Built New Syncer Chassis with added diagnostics:
Fiduciual Loss, Rate Monitors, RF Input Power, Clock Lock Detect,
Clock Rate Monitors, etc.
Status: Chassis Installed / Need to wire up diagnostic readout HW and
finish the SW
Would like to re-design the Fanout modules to take advantage to
the built-in diagnostics (Tx & Rx optical power, temp, voltage, etc.)
of the Fiber SFP modules
Additional Timing System Upgrades:
FIDO Replacement / New Fid Generation Scheme (future…)
Linac Sector 0 Timing Upgrade (future…)
December 7, 2011
Introduction to the LCLS Timing System
43
John Dusatko
jedu@slac.stanford.edu
LCLS Timing – Future Directions
Eventual Goal:
Completely replace SLC Timing System in the
Linac while retaining all of the legacy
functionality and providing new features
Develop a timing system that is expandable &
adaptable to new machines (e.g. PEP-X)
Challenges:
Temperature-induced phase drifts
Handling PEP Injection
Damping Ring Timing
Coming Soon….LCLS-II (First Light 2019)
December 7, 2011
Introduction to the LCLS Timing System
44
John Dusatko
jedu@slac.stanford.edu
LCLS-II:
Uses middle 1/3rd of Linac
-New Injector
- Re-purpose PEP-II Inj Line
- 2 Undulators:
- Hard X-Ray (2-13KeV)
-Soft X-Ray (0.25-2KeV)
First Light: Early 2018
December 7, 2011
Introduction to the LCLS Timing System
45
John Dusatko
jedu@slac.stanford.edu
End of Talk
December 7, 2011
Introduction to the LCLS Timing System
46
John Dusatko
jedu@slac.stanford.edu