Status of MicroTCA LLRF
Development
Zheqiao Geng
On behalf of the LLRF AIP team
6/4/2012
Outline







History of LLRF at SLAC
MicroTCA Based LLRF System Overview
FPGA Firmware Design
EPICS Software Design
Applications for Operation
System Tests
Summary
2015/4/9
Zheqiao Geng, MicroTCA LLRF
2
History of LLRF at SLAC
2015/4/9
Zheqiao Geng, MicroTCA LLRF
3
SLAC Linac
2015/4/9
Zheqiao Geng, MicroTCA LLRF
4
Existing RF System of Linac



30 Sectors (LCLS uses the last 10 of them)
Each sector contains of 8 klystrons and 1 sub-booster
Every RF station has 2 racks for controls
2015/4/9
Zheqiao Geng, MicroTCA LLRF
5
WG & Cable
Penetration
to Tunnel
Existing Linac Klystron Station RF Control, Monitoring, and Interlocking System
IPA Chassis
Controls RF Phase and
Amplitude
PIOP CAMAC
Module
Controls IPA, PAD,
and MKSU. Interface
to control system
PAD Chassis
Measures RF Phase and
Amplitude
MKSU Chassis
Interlock and Control for
Klystron SLED Support
Systems
Existing
Controls
Racks
7
PIOPs (4)
Controls Upgrade Implementation
PDU Timing
12-01-10
8
Controls Upgrade Implementation
12-01-10
9
LLRF for LCLS


2015/4/9
Several critical RF stations were upgraded for LCLS with newly designed Phase
and Amplitude Detector (PAD), Phase and Amplitude Controller (PAC) and VME
running EPICS
The old IPA, MKSU and CAMAC system is kept (not shown in the diagram below)
Zheqiao Geng, MicroTCA LLRF
10
LLRF for LCLS (cont.)


2015/4/9
PAD : Custom chassis with 4 channels of down mixers and ADCs
PAC : Custom chassis with a DAC board and an I/Q modulator
Zheqiao Geng, MicroTCA LLRF
11
Some Limitations of PAD/PAC




A feedback control loop has to follow the chain of PAD-VME-PAC
connected with Ethernet, the real-time performance is limited. It is
not possible to do intra-pulse control (pulse width ~ 3 µs)
Computation power of the Coldfire MCU used in PAD/PAC
chassis is quite limited. One more Channel Access client
connected to the EPICS software in the Coldfire MCU can
significantly degrade its real-time performance
One PAD chassis (2U or 3U) only contains 4 ADC channels. The
density is too low to efficiently use the rack space
Custom designed chassis is difficult to maintain
The MicroTCA based LLRF system presented in this talk tends to
upgrade the PAD/PAC system to be more compact, flexible,
maintainable and reliable…
2015/4/9
Zheqiao Geng, MicroTCA LLRF
12
Overview of the MicroTCA Based
LLRF System
2015/4/9
Zheqiao Geng, MicroTCA LLRF
13
PAD/PAC LLRF VS MicroTCA LLRF
2015/4/9
Zheqiao Geng, MicroTCA LLRF
14
RF Support Chassis
2015/4/9
Zheqiao Geng, MicroTCA LLRF
15
MicroTCA Crate
2015/4/9
Zheqiao Geng, MicroTCA LLRF
16
AMC Carrier + PMC EVR
2015/4/9
Zheqiao Geng, MicroTCA LLRF
17
AMC ADC Board – SIS8300

Struck SIS8300 Board

4 lane PCI Express

10 Channels 125 MS/s 16-bit
ADC

Two 16-bit DACs for Fast
Feedback Implementation

Twin SFP Card Cage for High
Speed System Interconnects

Virtex 5 FPGA
The board is equivalent to the digital
parts of 2.5 PADs + 1 PAC!
2015/4/9
Zheqiao Geng, MicroTCA LLRF
18
Summary of Hardware Architecture

Compared to the PAD/PAC chassis, the MicroTCA based LLRF system uses
commercial digital boards to reduce the R&D time

MicroTCA system has much higher density of ADC/DAC channels. The
system is more compact

Compared to the PAD/PAC chassis, the digital hardware of the new design is
installed in the MicroTCA crate. The boards are hot-swappable and easy to
maintain

To be improved (nice to have):
2015/4/9

An AMC EVR board will be introduced to route trigger signals via the backplane.
ADC boards will take triggers from backplane to remove the trigger cables

The RTM with S-band down mixers from DESY will be evaluated. The system can
be more compact by moving the down mixers and up converter to the RTM board
from the RF support chassis

The klystron beam voltage conditioner board will be improved to directly measure
the flattop of the voltage pulse by adding offset to the klystron beam voltage signal
Zheqiao Geng, MicroTCA LLRF
19
FPGA Firmware (for SIS8300)
Design
2015/4/9
Zheqiao Geng, MicroTCA LLRF
20
Major Requirements to the Firmware




2015/4/9
Intra-pulse phase feedback control
Provide 64K sampling buffer for each ADC channel (10 channels)
DAC can be used as an arbitrary waveform generator
Exception detection and handling
Zheqiao Geng, MicroTCA LLRF
21
Overview of the Firmware
2015/4/9
Zheqiao Geng, MicroTCA LLRF
22
Algorithm for Intra-pulse Phase Control
Q
Vector
Rotation
Demodulation
Correction
Q
I
I


2015/4/9
Q
I
Align the vector along the I axis so that Q components will be
proportional to the phase jitter
Phase error is estimated at the first part of the RF pulse and the
correction is applied at the second part of the RF pulse (latency < 1 µs)
Zheqiao Geng, MicroTCA LLRF
23
ADC Data Acquisition
Spectrum by FFT
64 K samples from ADC2
(SNR = 77 dBFS)

2015/4/9
Allow up to 64K point data acquisition for ADC SNR calculation
Zheqiao Geng, MicroTCA LLRF
24
DAC Waveform Generation
Waveforms Generated by 2048point Buffers (triggered output)



Waveforms Generated by 2048point Buffers (CW output)
Specify arbitrary I/Q waveforms in two 2048-point buffers for two DACs
Allow CW output regardless of trigger
Example: In-phase and Quadrature waveforms for single side band
modulation
2015/4/9
Zheqiao Geng, MicroTCA LLRF
25
RF System Simulator
Simulated Klystron Output (25.5 MHz IF)
Input to the RF System Simulator


Simulate the klystron output and
SLED output
The simulator can be used to test
most of the functions in the lab
Simulated SLED Output (25.5 MHz IF)
2015/4/9
Zheqiao Geng, MicroTCA LLRF
26
EDM Panel for Firmware Control
2015/4/9
Zheqiao Geng, MicroTCA LLRF
27
Summary of Firmware Design

MicroTCA based LLRF system connects ADCs and DACs to the same FPGA
to enable the intra-pulse control

The powerful FPGA is used to implement most of the complex real-time
functions to relax the CPU load. 360 Hz operation or multi-bunches operation
can be well supported

PCI Express links the FPGA and CPU to enable faster data transfer which
improves the data acquisition capability

To be improved (basic):

The firmware will be extended to be general and configurable for all RF stations
(CW control, Laser control, RF Gun control, X-band and S-band klystron control)

Intra-pulse phase feedback will be upgraded for both amplitude and phase control.
I/Q control scheme will be used
To be Improved (nice to have):

2015/4/9

Improve the intra-pulse control refer to the klystron high voltage jitters if they have
stronger correlations

Improve the up conversion algorithm to reduce the non-linearity of the phase and
amplitude actuation
Zheqiao Geng, MicroTCA LLRF
28
EPICS Software Design
2015/4/9
Zheqiao Geng, MicroTCA LLRF
29
Software Architecture
2015/4/9
Zheqiao Geng, MicroTCA LLRF
30
EDM Panels – RF Station Top
2015/4/9
Zheqiao Geng, MicroTCA LLRF
31
EDM Panels – Pulse-pulse Phase Control
2015/4/9
Zheqiao Geng, MicroTCA LLRF
32
EDM Panels – LLRF Timing Settings
2015/4/9
Zheqiao Geng, MicroTCA LLRF
33
EDM Panels – RF Synchronous DAQ


2015/4/9
Save all phase and amplitude values of the RF signals for
the same RF pulse synchronously up to 65536 pulses
Save all waveforms for the same RF pulse synchronously
up to 2048 pulses
Zheqiao Geng, MicroTCA LLRF
34
EDM Panels – RF Waveforms
2015/4/9
Zheqiao Geng, MicroTCA LLRF
35
Summary of Software Design

PAD/PAC based LLRF system has software pieces in PAD CPU, PAC CPU
and VME CPU, they communicate with each other via UDP. The architecture
is complex and difficult to maintain. The computation power of these CPUs
are quite limited

MicroTCA based LLRF system has one much more powerful CPU. Real-time
Linux OS will be used and it is much more flexible to be adapted to the
newest multi-core CPUs

MicroTCA software is compiled to a single IOC process so the number of
maintenance points is reduced

Data and waveforms can be saved at 120 Hz for diagnostics. Later this
function can be synchronized by Timing System for all RF stations so that the
behavior of the entire machine can be analyzed within one RF pulse

Software architecture is more modular and understandable

To be improved (basic):
2015/4/9

Software should be improved to fit the extended firmware

MicroTCA infrastructure (such as software development tools, EPICS base, boot
up tools) needs to be improved. This topic will be covered by Charlie Xu’s talk
Zheqiao Geng, MicroTCA LLRF
36
LLRF Applications
2015/4/9
Zheqiao Geng, MicroTCA LLRF
37
LLRF Applications









Examples of Applications for A RF Station
Measure the klystron energy no-load
Measure klystron saturation curve
Measure the beam phase with beam induced signal
Calibration of the imbalance of the I/Q modulator and DAC offset
Loop phase correction for intra-pulse feedback control
Intra-pulse feedback gain optimization
DAC waveform generation for single side band up-conversion
Set klystron mode to ACC or standby
Most of the applications already exist for the old RF stations. The new
LLRF software will inherit the existing algorithms and codes
2015/4/9
Zheqiao Geng, MicroTCA LLRF
38
System Test
2015/4/9
Zheqiao Geng, MicroTCA LLRF
39
ADC Noise Measurement
510-04516D
100MHz
25MHz
1/4 Freq
Divider
4
I
SIS8300
MicroRTM
100MHz
CH0
CH1
CH2
CH3
CH4
CH5
CH6
CH7
CH8
CH9
CLK2
CLK1
CLK0
6570
-1.906
6560
-1.907
6550
-1.908
6540
-1.909
6530
0
2
4
6
8
-1.91
Q
x 10
0
2
4
6
4
Signal to Noise Ratio (dBFS)
4
2.019
x 10
Amp
x 10
Pha/deg
-70.9
ADC0
ADC1
ADC2
ADC3
ADC4
ADC5
ADC6
ADC7
ADC8
ADC9
2.018
-70.95
At Lab
77.4
78.6
78.2
79.4
78.2
78.8
77.0
78.8
79.1
79.2
2.017
-71
At 28-2
76.1
76.7
76.7
77.2
76.5
77.4
76.7
77.6
77.4
77.4
2.016
-71.05
2.015
Crosstalk Matrix (dB)
0
2
4
6
8
-71.1
0
2
4
6
4
ADC1
ADC2
ADC3
ADC4
ADC5
ADC6
ADC7
ADC8
ADC9
ADC0
0
88.9
87
97.3
95.2
111.6
106.2
112.2
103.5
112.3
ADC1
82.7
0
87.1
98.5
97
105
108.5
112.9
104.6
108.9
x 10
0
Data
Noise Floor
Signal Level
-20
82.3
84.1
0
86.1
87
98.6
96.8
113
104.4
112.2
ADC3
95.8
94.7
81.2
0
97.1
98.5
98.9
109.8
103.9
111.5
ADC4
91.5
93.4
83.8
97.7
0
85.5
85.1
98.6
98.6
109.9
ADC5
99.2
101.1
95.1
94.9
81.3
0
92.5
101.1
100.4
111.8
ADC6
100.2
101.4
94.3
93.6
81
88.5
0
87.2
88.9
107.3
ADC7
98.9
102.4
103.5
102.4
94.9
97.8
81.7
0
84.7
109.7
ADC8
100.5
102.1
105.4
104.6
96
94.7
84.3
81.1
0
95.2
ADC9
99.3
102.3
104.8
104.8
100.9
106.4
106.5
101.5
92.4
0
-40
Amplitude / dBFS
ADC2
8
4
x 10
ADC0
8
4
x 10
-60
-80
-100
2015/4/9
Zheqiao Geng, MicroTCA LLRF
-120
-140
0
5
10
15
20
25
30
Frequency / MHz
35
40
45
40
50
Phase and Amplitude Measurement Noise
With a -1dBFS 25.5 MHz IF input and
257 fs clock jitter, the measurement
noises (in full bandwidth of ADC) are
expected to be:

119 MHz clock jitter: 257 fs
integrated from 10 Hz to 40 MHz



2015/4/9
Zheqiao Geng, MicroTCA LLRF
Phase
: 0.01 deg RMS
Amplitude: 0.02 % RMS
Considering the bandwidth of the RF
system in the view of the beam (~1.2
MHz), the measurement noises will
meet the LCLS-II requirements (0.07
deg RMS phase jitter and 0.06 %
RMS amplitude jitter for the most
critical RF station of L1S)
41
Installation at LI28-2
SSSB
RF Support Chassis
MicroTCA Crate
MKSUII
2015/4/9
Zheqiao Geng, MicroTCA LLRF
42
System Test at LCLS Linac (LI28-2)
- RF Signal Measurement
2015/4/9
Zheqiao Geng, MicroTCA LLRF
43
RF Reference Signal
2015/4/9
Zheqiao Geng, MicroTCA LLRF
44
I/Q Modulator Output Signal
2015/4/9
Zheqiao Geng, MicroTCA LLRF
45
Klystron Drive Signal
2015/4/9
Zheqiao Geng, MicroTCA LLRF
46
Klystron Output Signal
2015/4/9
Zheqiao Geng, MicroTCA LLRF
47
SLED Output Signal
2015/4/9
Zheqiao Geng, MicroTCA LLRF
48
System Test at LCLS Linac (LI28-2)
- Reference Tracking
2015/4/9
Zheqiao Geng, MicroTCA LLRF
49
Reference Tracking



Subtract the phase of the reference signal from other RF signal phases
Remove the common mode error caused by the RF detectors
experiencing the same temperature fluctuations
Remove the phase jump caused by the clock re-synchronization
Reference signal
59.5
Reference Phase
Phase / deg
59.4
59.3
59.2
59.1
59
0
5000
10000
15000
10000
15000
10000
15000
Klystron output signal
Phase / deg
57
Klystron Output Phase
56
55
54
53
0
5000
SLED output signal
104.5
SLED Output Phase
Phase / deg
104
103.5
103
102.5
102
0
5000
Time / s
2015/4/9
Zheqiao Geng, MicroTCA LLRF
50
System Test at LCLS Linac (LI28-2)
- Pulse-pulse Feedback
2015/4/9
Zheqiao Geng, MicroTCA LLRF
51
Long Term Stability with Feedback
SLED phase without feedback
Phase / deg
31
30.5
30
29.5
29
0
1000
2000
0
1000
2000
3000
4000
5000
Time / s
SLED phase with feedback
6000
7000
8000
6000
7000
8000
Phase / deg
31
30.5
30
29.5
29
3000
4000
Time / s
5000
2-hour phase measurement
2015/4/9
Zheqiao Geng, MicroTCA LLRF
52
System Test at LCLS Linac (LI28-2)
- Intra-pulse Feedback
2015/4/9
Zheqiao Geng, MicroTCA LLRF
53
Intra-pulse Feedback
Phase error is estimated at the first part of the RF pulse and the
correction is applied at the second part of the RF pulse



Loop delay should < 1 µs
Phase jitter of different parts of the RF pulse should be correlated
The entire loop delay is
~ 600 ns, quite
promising for intrapulse feedback.
2015/4/9
Zheqiao Geng, MicroTCA LLRF
54
Phase Jitter Correlation
Correlation coefficient of the phase jitter
between different parts of the RF pulse ~
0.3-0.4
2015/4/9
Zheqiao Geng, MicroTCA LLRF
55
Intra-pulse Feedback Gain Sweeping
Phase Jitter / deg
Feedback Gain



Effects of the intra-pulse feedback on the beam can only be tested at the sensitive
RF stations (like L1S)
Correlation of the phase jitters of different parts of the RF pulse is not so strong
How about the correlation between phase jitter and the klystron high voltage?
2015/4/9
Zheqiao Geng, MicroTCA LLRF
56
System Test at LCLS Linac (LI28-2)
- Intra-pulse phase slope compensation
2015/4/9
Zheqiao Geng, MicroTCA LLRF
57
Motivations



2015/4/9
At rising edge of the klystron signal, the phase changes > 100 deg, which
will lower the efficiency to fill the SLED cavities
Klystron signal after PSK has a phase change > 60 degree, which will
lower the integrated E field seen by the beam
Idea: Remove the phase slope with feed forward to possibly increase the
energy gain from the klystron
Zheqiao Geng, MicroTCA LLRF
58
Phase Slope Compensation with Feed Forward
2015/4/9
Zheqiao Geng, MicroTCA LLRF
59
SLED Amplitude with Feed Forward Iterations
Amplitude of SLED output can be increased by ~ 3 %

Note: There is still 10 degree phase change at the SLED output pulse
during the part that interacts with the beam!
The MicroTCA based LLRF system provides a very powerful platform to
implement the new ideas from physicists

2015/4/9
Zheqiao Geng, MicroTCA LLRF
60
System Test at LCLS Linac (LI28-2)
- Phasing the klystron
2015/4/9
Zheqiao Geng, MicroTCA LLRF
61
Phasing the Klystron with Matlab
Without phase feed forward control

2015/4/9
With 50 iterations of phase feed forward control
Need further measurement to clarify if we can really increase the energy
gain or not!
Zheqiao Geng, MicroTCA LLRF
62
More Tests Need to be Done




2015/4/9
Intra-pulse I/Q control need to be tested at critical RF stations like
L1S before the first bunch compressor. The effects of the intra-pulse
control can be examined by monitoring the beam stability
Measurement of beam induced signal need to be done at LI28-2. To
do this test, the klystron need to be in standby state
Correlation between RF amplitude/phase jitters and klystron high
voltage jitter need to be tested. This will tell us if we can use the
klystron high voltage jitter for intra-pulse feedback. As mentioned
before, a new klystron beam voltage conditioner board need to be
designed
More tests need to be done to check if the intra-pulse phase slope
compensation can increase the energy gain or not
Zheqiao Geng, MicroTCA LLRF
63
Summary
2015/4/9
Zheqiao Geng, MicroTCA LLRF
64
Summary of MicroTCA System





2015/4/9
MicroTCA introduces intra-pulse control to reduce fast jitters
MicroTCA system uses PCI express for faster data acquisition which enables
to save the RF waveforms at 120 Hz or even at 360 Hz
MicroTCA system uses more powerful FPGA and CPU which enables to
upgrade the system for 360 Hz or multi-bunch operation without changing
the hardware
MicroTCA system contains more ADC channels in a single board which
enables to implement the reference tracking to remove the phase jump
problem in digital I/Q demodulation algorithm
The PAD/PAC system uses Coldfire MCU which is a bottleneck for real-time
performance. The PAD-VME-PAC chain is connected with Ethernet and it is
not possible to perform intra-pulse control. 120 Hz waveform saving is poorly
supported as well
Zheqiao Geng, MicroTCA LLRF
65
Summary of MicroTCA System (cont’d)






2015/4/9
MicroTCA has better upgradability: overhead in data transfer speed (PCI
Express) and computation power (FPGA + multi-core CPU)
MicroTCA has better maintainability: hot-swappable – reducing repair time –
more availability
MicroTCA has better reliability – simpler and more compact system structure;
redundant MCH and power supply
MicroTCA has better platform management – IPMI
MicroTCA is a new platform which needs more development efforts
Cost is relatively high due to small market
Zheqiao Geng, MicroTCA LLRF
66
Thank you!
2015/4/9
Zheqiao Geng, MicroTCA LLRF
67
Backup Slides
2015/4/9
Zheqiao Geng, MicroTCA LLRF
68
Phase and Amplitude Actuation Noise



I/Q modulator Input Jitters: Phase 0.083 deg RMS, Amplitude 0.01%
I/Q modulator Output Jitters: Phase 0.074 deg RMS, Amplitude 0.03%
The phase and amplitude actuation does not introduce extra noise (or at least
neglectable)
2015/4/9
Zheqiao Geng, MicroTCA LLRF
69