Upgrade possibilities for the Hardware
of the LEIR type Beam Control
Information from:
M.E. Angoletta, P. Baudrenghien, E, Bracke, A. Butterworth,
J. Ferreira, J. Molendijk, R. Olsen, F. Pedersen, T. Rohlev, J. Sanchez
1. LHC – LEIR approach
2. Present LEIR Hardware
3. Possible upgrades
4. Planning
A.Blas
AB/RF/FB
Team meeting 21/1/2008
1
Beam Control
Typical architecture
Both the LHC and LEIR system start with the same initial structure
Typical constraint for the signal processing delay:
Loop computation time + Hdw delay < 1/(8.Fmod) [phase lag @ max Fmod < π/4]
< 21 us for PSB & LEIR phase loop (3.Fs=6kHz)
A.Blas
AB/RF/FB
Team meeting 21/1/2008
2
Beam Control
Signal processing
In LEIR the FPGA-DSP-FPGA triptych is always used
In LHC it depends.
Note that using the DSP in a loop implies major timing problems when a
precise loop delay is required (TFB, 1TFB) (see example in Elettra)
A.Blas
AB/RF/FB
Team meeting 21/1/2008
3
Beam Control
Board layout
Interconnect architecture shared by the LHC
and LEIR hardware
The Central FPGA (fpga array in Leir) routs the
data from and to the different locations
•The special function board can be in the form of
an on-board specific circuit, a daughter cards or
a dedicated VME board
A.Blas
AB/RF/FB
Team meeting 21/1/2008
4
Beam Control
Clocking
LHC:
A central VCO, controlled by a central processor, outputs the rf (400MHz
+/- e) feeding the cavity. This rf signal is used to create the 380, 80, 40,
20, Frev which are dispatched within the hardware.
LEIR:
A central DDS, controlled by a central processor, outputs the tagged rf
clock distributed to all rf sources or demod inputs.
(+) Absolute phase control over distant inputs/outputs without (except
practical) limitations on their number.
(+) Synchronous change of parameters on all boards.
(+) The Clock harmonic can be changed on the fly (factor 1, 2, 4 and 8)
for use on a wide frequency span (factor 16)
(+) synchronous signal acquisition
A.Blas
AB/RF/FB
Team meeting 21/1/2008
5
Beam Control
LHC: VME 64 crate, h: 9U
Hardware standards
4 front slots, 160 mm depth with
standard P1/P2 bus
15 front slots, 220 mm depth with
dedicated P2 bus
Spec. power supplies setup
6 kCHF + 10 kCHF CPU
LEIR: VME 64X crate, h: 9U 23 front slots, 160 mm depth
23 back slots, 160 mm depth
All standard P0, P1, P2 connectors
Standard Power supply
8 kCHF + 10 kCHF CPU
A.Blas
AB/RF/FB
Team meeting 21/1/2008
6
Beam Control
Samtec QSE, 80 (4 small
ribons of 20 coax cables on a single
connector
Gigabit serial link with
8b/10b coding
16bit// 50 MHz -> 1Gb/s serial
DSP bus + mother
board to daughter card
bus at 40 MHz and 32 bit word:
Communication links
160 Gb/s (at 2 GHz)
3.2 Gb/s (at 40 MHz)
Used in LHC hardware
800Mb/s
Full duplex at the
same rate
Used in LHC hardware
320 Mb/s
Used in LEIR
Very common standard
DSP <-> DC
1.28 Gb/s (to be divided by the
number of wait states +2 = 4)
Link port clocked at 40 MHz
320 Mb/s
(can be used at 80 MHz)
A.Blas
AB/RF/FB
Used in LEIR
DSP <-> DSP
Team meeting 21/1/2008
7
FPGA comparison
Xilinx
Virtex 2
XC2V20006FG676C
(Tuner loop)
Altera Stratix
EP1S20F
484C5
(Leir
daughter
cards)
Xilinx
Virtex 4
XC4VLX40
Beam Ph
+ rf Mod.
Altera
Stratix II
EP2S90F
1020 C4
LHC 1TFB
Logic cells
24,192
18,460
41,472
90,960
RAM [kb]
1,000
1,600
1728
4414
18 x 18 multipliers
56
80
64
192
I/O
456
361
640
758
Price [USD]
278
350
267
2890
A.Blas
AB/RF/FB
Team meeting 21/1/2008
8
DSPs comparison
Model
Clock
[MHz]
MMACS
Link
ports
Serial
Links
(not used)
Unit price
for 1000
(USD)
ADSP-21160M
(Leir)
80
160
6
80 MHz max
2
40 MHz max
159
ADSP-21160N
(upgrade Leir)
100
200
6
100 MHz max
2
50 MHz max
170
ADSP-TS101S
(LHC Tuner loop)
300
2100
4
125 MHz max
Incompatible with Sharc DSP!?
Dual edge = 250 MHz eq
0
186
ADSP-TS201S
(available novelty)
600
4800
4
500 MHz max
0
242
A.Blas
AB/RF/FB
Team meeting 21/1/2008
9
DSP chip
ADSP 21160M
(Leir at present)
80 MHz, 4Mbit on chip SRAM, 2.5V core
A.Blas
AB/RF/FB
Team meeting 21/1/2008
10
DSP chip
Possible LEIR upgrade ADSP 21160N
Same as M version except 20% faster (100 MHz) and 1.8V core
A.Blas
AB/RF/FB
Team meeting 21/1/2008
11
DSP chip
A.Blas
AB/RF/FB
Tiger Sharc ADSP-TS101 (LHC)
Team meeting 21/1/2008
12
Leir BC <-> LHC BC
A.Blas
LHC
LEIR
Modified VME 64 crate
160 mm (CO)+ 220 mm (rf)
Private Backplane bus with:
Power, ECL Clocks, Timing, Function
Gen, Interlocks, Alarms, LVDS data bus,
JTAG, Auto slot addressing, Module serial
number bus
Standard VME 64X crate
160mm with rear access to bus
Cadence blocks:
Tiger sharc + environment
Virtex 4 + environment
1 Gb serial link + environment
Tagged clock + distribution
(connectors are fragile; why??)
RTM for timing, data, clock, link ports
High speed parallel connection between
VME neighbors
Samtec QSE 80 // coax lines
Mother-daughter card structure
interconnected via the DSP bus
AB/RF/FB
Team meeting 21/1/2008
13
Leir BC <-> LHC BC
LEIR
Pros:
•Standard VME 64X crate with front and rear access
•Very modular approach; with 3 DC (DDC, SDDS, MDDS), 1 MB and 1 RTM,
you have the base for all type of applications (except yet for 1TFB and TFB)
•Synchronous control over all the hardware
LHC
Pros:
•Power supplies adapted to low noise requirements
•More up-to-date circuits and connectors
A.Blas
AB/RF/FB
Team meeting 21/1/2008
14
Leir Beam Control
Clock core
Output rf frequency <12 MHz (for a swept frequency - without up-conversion)
<25 MHz at fixed frequency (without up-conversion)
Limitations due to the basic structure (max ADC DAC sampling + tag creation circuit)
A.Blas
AB/RF/FB
Team meeting 21/1/2008
15
Leir Beam Control
A.Blas
AB/RF/FB
Acquisition
Team meeting 21/1/2008
16
Leir Beam Control
A.Blas
AB/RF/FB
Outputs
Team meeting 21/1/2008
17
Beam Control
A.Blas
AB/RF/FB
General architecture (LEIR)
Team meeting 21/1/2008
18
Beam Control Constraints
Data flow
Rule of thumb:
The loop will be sufficiently stable if its delay leads to a phase lag < p/4 at the unity loop
gain frequency
 Loop computation time + Hdw delay < [1/(8.Fmod)]
< 21 us for PSB & LEIR (3.Fs=6kHz)
< 26 us for PS (3.Fs=4.8 kHz)
In LEIR, the in-loop DSP is sampling the data every TS-DSP = 12.5 us (80 kHz).
The loop delay within the DSP = import data from DDC (<50ns) + compute error (<7us) + send error to MDDS
or SDDS (via another DSP or not < 150 ns) + equivalent 1st order S/H delay (6.25 us) => ≈14us delay within
the DSP
With a 80 kHz DSP sampling clock, an averaging (CIC) of 1000 80-MHz-samples in the DDC would be
adequate. We actually use 256, which means <6.4 us extra delay. This means that we are approaching the
reasonable limits required for the LEIR and PSB phase loop.
The DSP process time in LEIR is the most time consuming and multiplying by a factor 2
this process speed would almost double the possible bandwidth.
A.Blas
AB/RF/FB
Team meeting 21/1/2008
19
Leir Beam Control
Data Flow
Here, the central DSP B has all its links ports used (limiting factor)
In case of more DSP boards, we would need to add links to the leading DSP
or implement a pipe lined communication protocol through DSP boards or
make all DSPs share a single bus with a communication protocol to be
defined.
In terms of bandwidth, A gigabit link could replace the link ports and the DSPto-daughter card link
A.Blas
AB/RF/FB
Team meeting 21/1/2008
20
Leir Beam Control
Daughter cards
The present DDC (ADC) and SDDS (DAC) daughter cards having 4 channels
are limited in terms of FPGA logic cells.
The signal monitoring circuit could not be implemented.
The 4 channels of the DDC need to share 2 LOs
The DDC FIR could not be implemented
The present chip: Altera Stratix EP1S20F 484C5 should be replaced with a version with 6 times
more logic elements to have a 50% loading at most. 18k LE -> 110 k LE and twice as much
RAM (1.6 Mb -> 3.2 Mb) and same I/O: 361.
The Stratix 3 EP3SL150 can do the job: 142 kLE, 5.5 Mb, 480 or 736 I/O, 1400 euros (Spoerle)
A.Blas
AB/RF/FB
Team meeting 21/1/2008
21
DSP board
Background
04/2002
Original design by Joe Delong (BNL)
03/2005
Original version imported in the CERN database (EDA 00990) and called version 1
13 units constructed and tested. Followed-up by J. Ferreira-Bento.
08/2006
Upgrade to version 3 by Tony Rohlev, after a faulty version 2
which had a bad set of connections on a FPGA
Main improvements: latched registers interfacing the VME bus + adequate power-up
sequence.
(see AB-Note-2007-031-RF for a detailed description of the modifications)
09/2007
Version 4. Cleaned-up version : removed 4 daughter card connectors (only 2 DC slots
now), 1 FPGAs, the Event link circuit, the link-port connections to the DC connectors
(reflections), Added DSP address bus lines 15-19 to the DC to avoid paging of the SRAM
address + allow use of DMA (DMA lines added) + use of delayed SELN bits for the register
address decoding, all 16 RTM signal connected (6 before).
A.Blas
AB/RF/FB
Team meeting 21/1/2008
22
DSP board
DSP Data[39,32]
DSP Addr[31,0]
DSP Addr[19,0]
DSP Control
VME
interface
FPGA
VME Data [7,0]
Version 1 Block Diagram
DSP
Flash
1M x 8b
DSP Addr[14,0]
2 link ports F.P.
DSP Addr[31,0]
DSP Data[63,0]
VME Addr [31,1]
DSP Data[32]
4 link ports
VME P2
DSP
DSP Control
OE
VME Data [31,0]
OE
Site SEL
4 link ports
Site connectors
DSP Data [63,32]
DIR
Bi-Dir
Transceiver
Mem Data [63,32]
Mem
Addr [17,0]
DSP Data
[63,32]
MEAS SRAM
256k x 32b
X4
Tri-state
Buffer
Mem
Addr [31,18]
OE
Tri-state
Buffer
Mem Addr [31,0]
Bit 31 always at 0
DSP control
MS[3,0]
Mem
select
EPLD
RAM
Data
[15,0]
SEL[3,0]
Selects 1 of
the 4 banks
SEL[3,0]
Selects 1 of
the 4 banks
Mem Addr [31,0]
Bit 31,30 always at 0
Daughter
card SRAM
1M x 16b
DSP SRAM
256k x 32b
X4
DSP Addr
[18,1]
OE
L,DSP Addr [31,1]
DSP Control
Site
connector
1+2
or 3+4
DIR
Bi-Dir
Transceiver
L,L,VME Addr [31,2]
DSP Data[63,32]
H,Mem Addr [31,19]
DSP control
MS[3,0]
Mem
select
EPLD
DSP Addr[14,0]
LSB always H
DSP Data[63,32]
RAM
Addr
[19,0]
Daughter
card FPGA
DSPWRH
OE
DSP Data [31,0]
DIR
Bi-Dir
Transceiver
Mem Data [31,0]
Bit 31 always at 0
DSP Data[39,32]
DSP Addr[31,0]
DSP Data[63,32]
Timing int.
FPGA
Trig[5,0] from VME P2
Problem: When the DSP SRAM
is controlled by the VME bus
The DSP is blocked
A.Blas
AB/RF/FB
Site SEL[7,0]
DSP Addr[31,0]
DSP Data[63,32]
DSP2RCV
FPGA
DSP Control
Team meeting 21/1/2008
23
DSP board
DSP Data[39,32]
DSP Addr[31,0]
DSP Control
DSP Addr[19,0]
VME
interface
FPGA
VME Data [7,0]
Version 3 Block Diagram
DSP
Flash
1M x 8b
DSP Addr[14,0]
VME Addr [31,1]
DSP Data[63,0]
DSP Control
Site
connector
1+2
or 3+4
Site SEL
4 link ports
Site connectors
OE, DIR, LE
Bi-Dir
latched
Transceiver
DSP Control
4 link ports
VME P2
DSP
DSP Data[32]
VME Data [31,0]
DSP Data[63,32]
2 link ports F.P.
DSP Addr[31,0]
DSP Data [63,32]
OE, DIR, LE
Bi-Dir
latched
Transceiver
Mem Data [63,32]
Mem
Addr [17,0]
OE,LE
Tri-state
latched
Buffer
L,L,VME Addr [31,2]
Mem
Addr [31,18]
Tri-state
Buffer
L,DSP Addr [31,1]
Mem Addr [31,0]
Bit 31 always at 0
DSP control
MS[3,0]
Daughter
card SRAM
1M x 16b
DSP SRAM
256k x 32b
X4
DSP Addr
[18,1]
RAM
Data
[15,0]
SEL[3,0]
Selects 1 of
the 4 banks
SEL[3,0]
Selects 1 of
the 4 banks
Mem Addr [31,0]
Bit 31,30 always at 0
OE
DSP Data
[63,32]
MEAS SRAM
256k x 32b
X4
H,Mem Addr [31,19]
Mem
select
EPLD
DSP control
MS[3,0]
Mem
select
EPLD
DSP Addr[14,0]
LSB always H
DSP Data[63,32]
RAM
Addr
[19,0]
Daughter
card FPGA
DSPWRH
DIR
DSP Data [31,0]
OE
Bi-Dir
Transceiver
Mem Data [31,0]
Bit 31 always at 0
DSP Data[39,32]
DSP Addr[31,0]
DSP Data[63,32]
Timing int.
FPGA
Trig[5,0] from VME P2
Here the DSP SRAM data can be stored
on the VME bus to allow the “long” ~1us
VME Read process while letting the DSP run
A.Blas
AB/RF/FB
Site SEL[7,0]
DSP Addr[31,0]
DSP Data[63,32]
Team meeting 21/1/2008
DSP2RCV
FPGA
DSP Control
24
DSP board
Some limitations
1.
DSP blocked when the VME (which has priority) accesses the DSP SRAM (improved in version
3), Problem solved in Leir with pre-defined time slots for each access. Not convenient for long
cycling machine as AD where operational data should be accessed during the cycle.
Bus lines arbitration by 5 (4 in V4) distinct FPGAs => not flexible and uneasy to maintain
2.
DSP limited computing power. Impairs the rf loops pipe-line delay and signal quality (aliases),
requires assembly coding (instead of a more universal C coding)
3.
DSP configuration device not accessible via VME for remote programming,
nor daughter card JTAG programming.
4.
Only 6 timing (or digital) inputs from J2/P2 connector (RTM allows 16) (solved in V4)
5.
4 MB DSP memory + 4 MB for measurement memory foreseen to be too little (-> 32 MB total)
6.
VME base address selected by FPGA coding (-> put switches also)
A.Blas
AB/RF/FB
Team meeting 21/1/2008
25
LHC tuner loop
DSP board
The Tiger Sharc (tested) circuit can
be re-used.
For the FPGA there might be better
choices in other circuit using Virtex 4.
A.Blas
AB/RF/FB
Team meeting 21/1/2008
26
Proposed upgrade for the LEIR type DSP mother board
Maria Elena’s
note
1.
2.
3.
4.
5.
6.
7.
8.
Other possible
improvements
1.
2.
3.
4.
5.
6.
7.
8.
9.
A.Blas
AB/RF/FB
Single FPGA interconnecting all buses, timing, resets and clocks.
Fast VME/DSP arbitration for memory access.
Additional FPGA as link port multiplexer.
Upgrade / doubling of the DSP chip
Increase memory size (4x -> 32 MB)
Additional High speed inter-processor links
Increase the number of lines between RTM and J2/P2
Misc….
Add a couple of gigabit serial links from central FPGA to front panel. This would allow
using future specific boards as we do presently with daughter cards
A DAC could be connected to the Central FPGA for monitoring/test purposes
Provide a connection port to a stand alone VME daughter card (Samteq QSE) same
as for LHC boards
VME offset address selected by jumpers instead of hardcoding
Suppress link ports lines to daughter card sockets (electric reflections) (done V4)
Increase the address bus width connecting the daughter cards to avoid paging (DMA)
and avoid the delayed SELN address decoding bit (done V4)
Use better connectors for the tagged clock distribution
Implement a communication protocol to chain DSP boards to ease Beam control
upgrades and avoid piling-up of cables (critical in the CPS)
JTAG remote programming as in the CO with a dedicated FPGA-EPROM pair used as
a VME-to-JTAG interface. All the FPGAs should be in a single JTAG loop accessible
on the front panel for local programming.
Team meeting 21/1/2008
27
Leir Type DSP BC upgrade.
Summary
Some CadenceTM blocks can be used from LHC designs with improved specs (tiger shark) + Gigabit
link + general interconnect philosophy
The Virtex 4 or Stratix 2 circuits could also be imported but we should move to Virtex 5 or Stratix 3
(lower power dissipation 65nm, serial link interface not bugged + higher performances). To be used
on DSP board and daughter cards.
The standard VME 64X remains a good choice. Noise issues have still to be checked for AD (linear
Power Supplies).
The tagged clock and its distribution are a real plus for synchronous operation but the actual
connectors are too fragile and could be replaced the RJ45 model used in LHC rf.
Add all the hardware details mentioned on the previous slide.
A.Blas
AB/RF/FB
Team meeting 21/1/2008
28
Leir Type DSP BC upgrade.
Possible scenario
Install a test beam control in the PSB for the 2008 run, using our present stock of Hardware. A few
units are still to be tested. Can be ready in mid 2008.
Analysis of the software requirements if using a new DSP (MEA)
Definition of an adequate inter-DSP link (h/w + protocol) (1man.month)
Update the mother board with all the new features keeping the compatibility with the present
daughter cards. 8 man-month for the hardware, 6 man-month for the software + 40 kCHF. Ideally
the software should be finished before we start the hardware.
Update of the daughter boards (DDC + SDDS): 3 man-month x 2 for the hardware + 1 man-month x
2 for the software + 40 CHF -> 1 working prototype for each function.
Total: 2 man-year + 80 kCHF for a “final” DSP beam control prototype
hopefully fulfilling the requirements of all the Injectors on the Meyrin site.
A.Blas
AB/RF/FB
Team meeting 21/1/2008
29
DSP BC upgrade
A.Blas
AB/RF/FB
Generic view
Team meeting 21/1/2008
30
A.Blas
AB/RF/FB
Team meeting 21/1/2008
31
DSP board A
Binj, Bej
B TRAIN
Critical
Process
time
Freq Prog (500 kHz)
FREV 
Master DDS
on DSP
board B
DSP board B
Frequency
offset
F  B
h
B 2  B02
Monitoring
PID
Control
Clock
1 GHz
Monitoring
Radial
PID
Limiter
PID
Control
Steering
Phase
PID
Sync
PID
PID
Control
Stable
Phase
Tagged
Clock
4 CH receiver
on DSP board A
DDS
R L
R L
R L
R L
Synchro
Phase
I/Q
to
Angle
RF
to
I/Q
RF
to
I/Q
RF
to
I/Q
RF
to
I/Q
PU 1
Left
PU 1
Right
PU 2
Left
PU 2
Right
Tagging
at FREV
Tagged Clock
at h*FREV
I/Q
to
Angle
Leir
Beam
Control
Freq.
discri
RF
to
I/Q
Beam
Pick-up
Tagged
Clock
RF
to
I/Q
CAV
Sum
4 CH receiver
on DSP board B
REF
RF
Two other
channels
for cavity 2
1st harmonic
RF
to
I/Q
2nd harmonic
2*RF
to
I/Q
4 CH receiver
on DSP board C
Tagged
Clock
DSP board C
Tagged
Clock
h2
Cavity Voltage
/
to
I/Q
Cavity
Azimuth
H2
Cavity
PID
I/Q
to
RF
RF
Cavity
2
Critical
Process
time
Voltage
ratio
h4/h2
Phase
h4/h2
/
to
I/Q
H4
Cavity
PID
I/Q
to
2*RF
2*RF
4 ch modulator
on DSP board C
Two other
channels
for cavity 2
PID
Control
A.Blas
AB/RF/FB
Team meeting 21/1/2008
32
Misc
Arctan computation:
FPGA Cordic John: 17 clock periods for a 16 bit resolution
DSP Pawel: 33 clock period for a 18 bit resolution
A.Blas
AB/RF/FB
Team meeting 21/1/2008
33