Measuring propagation delay
over a coded serial
communication channel using
FPGAs
P.P.M. Jansweijer, H.Z. Peek
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
October 15, 2009
VLVnT-09 Athens
1
Introduction
It is feasible to measure propagation delay
over an 8B/10B coded link over 100 Km
of fibre
y A 3.125 Gbps serial link implemented in
FPGA provides a resolution of 320 ps.
y The method presented originates from
thinking about KM3NeT timing…
y …but applies more general to
“Measurement and control applications”
y
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
October 15, 2009
VLVnT-09 Athens
2
Measurement and control
application example - I
KM3NeT
OM
OM
GPS
OM
OM
Shore
Station
OM
OM
Distributed: 1 cubic Kilometer
Synchronize system timing
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
1 Km
y
OM
OM
1
Km
1 Km
High precision: ~ 1 ns
October 15, 2009
VLVnT-09 Athens
3
Measurement and control
application example - II
y
(Super) LHC
P
5
P4
Distributed: LHC diameter 8,6 Km
P2
CCR
Synchronize system timing
High precision: aim < 100 ps
P8
TTC backbone
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
From a presentation given at the
ATLAS Upgrade "ROD" Workshop
Sophie Baron – CERN
June 18, 2009
TTC off-detector
TTC on-detector
October 15, 2009
VLVnT-09 Athens
4
Measurement and control
application common
y
Distributed
◦ Large systems which (often) use (serial)
communication channels
y
Synchronize system timing
◦ Know the time offsets between clocks in the system
◦ Measure offsets = measure propagation delay
y
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
Could we use existing serial communication
channels to measure propagation delay?
October 15, 2009
VLVnT-09 Athens
5
Serial Communication Coding
Properties
Clock & Data coded into one stream
y2 DC-Balance
y3 Special code-groups / Word Alignment
y1
Run Length
DC Balance
Bit Synchronization
Clock Recovery
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
8B/10B
64B/66B and 64B/67B
5
Relies on Scrambler
Excellent
64B/66B Not guaranteed, 64B/67B Over one frame
Demanding for receiver
1
Excellent
Word Alignment
"Comma" KCharacters
Special code-groups
K-Characters
3
October 15, 2009
Relies on Scrambler
Sync-Header
Control-Codes
VLVnT-09 Athens
6
Measure propagation delay with
high precision
y
Lock the Receiver to Transmitter Clock =>
1
Clocks are Isochronous:
◦ Use the same time reference
◦ But have an offset
y
Use SerDes Word Alignment information
3
◦ Resolution one Unit-Interval (bit time)
y
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
Using property 1 and 3 is just a hardware
implementation on the OSI-model Data Link
layer that is transparent to higher levels of
hierarchy:
◦ IEEE 802.3 “CSMA/CD” (Ethernet)
◦ IEEE 1588 “Precision Clock Synchronization Protocol”
October 15, 2009
VLVnT-09 Athens
7
Test setup block diagram
y
y
Measure propagation delay using FPGA SerDes
Word Alignment information
@ 3.125 Gbps
Xilinx
ML507 Board
156.25
MHz
Lattice
Xilinx
LFSCM25
Virtex-5
SerDes
Lattice SC
PCI Expressx1
Evaluation Board
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
SerDes
Test-Bed
Transmitter
100 Km
fiber
Start
Test-Bed
Receiver
156.25
MHz
LEDs
Stop
October 15, 2009
VLVnT-09 Athens
8
Test setup transmission scheme
Tx
8B/10B
Encoded
K28.5 D16.2 K23.7 Dx.y K28.5 D16.2 K28.5 D16.2 K28.5 D16.2 K28.5 D16.2
IDLE
CharExt
IDLE
IDLE
IDLE
IDLE
Start
2 x 50 Km fiber ~ 490 us
Stop
Rx
8B/10B
Encoded
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
K28.5 D16.2 K28.5 D16.2 K28.5 D16.2 K28.5 D16.2 K23.7 Dx.y K28.5 D16.2
IDLE
IDLE
IDLE
IDLE
CharExt
IDLE
3.125 Gbps / 20 bits
= 156.25 MHz system speed (6.4 ns)
Without additional information the Stop
Time is known with a 6.4 ns resolution
October 15, 2009
VLVnT-09 Athens
9
Real Test setup
50 Km
fiber
Start
Stop
Test-bed
Receiver
RX (Stop)
Xilinx Virtex-5
Board (ML507)
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
50 Km
fiber
Test-bed
Transmitter
TX (Start)
Lattice LFSCM25
Board
October 15, 2009
VLVnT-09 Athens
10
Measuring varying propagation delay
Constant Impedance
Trombone Line
+0 psps
+700
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
Reset
(= resynchronize)
Resynchronize…
What is happening?
October 15, 2009
VLVnT-09 Athens
11
What happens during
resynchronization
1.
2.
TX is transmitting a serial bit stream based on the
reference clock
RX using the reference clock to try to lock its PLL in
the CDR onto the incoming bit stream (note: usually the
TX and RX reference clock do not have the same source...)
3.
4.
5.
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
Once the PLL in the CDR is in phase, RX switches
over from its reference clock to the RX recovered
clock “RxRecClk” (this happens on a random bit)
Next the Word Aligner is searching for a “Comma”
Once a Comma is found the word aligner knows
how to set its multiplexer and feed properly aligned
sets of 20 bits to the FPGA fabric
October 15, 2009
VLVnT-09 Athens
12
Test setup block diagram
Xilinx
ML507 Board
156.25
MHz
Lattice
Xilinx
LFSCM25
Virtex-5
SerDes
Lattice SC
PCI Expressx1
Evaluation Board
SerDes
Test-Bed
Transmitter
100 Km
fiber
Start
Test-Bed
Receiver
156.25
MHz
LEDs
Stop
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
October 15, 2009
VLVnT-09 Athens
13
Resynchronization in action
011101011000001010110111010110000010101101110101
3 10
RxRecClk
BitSlide(4:0)
0
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
0001 = 3
0000
0011
1
0
Start/Stop
delay
Algorithm:
Propagation Delay = “Start-Stop” Delay + “LED Value” * 320 ps
October 15, 2009
VLVnT-09 Athens
14
Algorithm Propagation Delay:
0 ps
320 ps
960 ps
“Start-Stop” Delay + “LED Value” * 320 ps
Leds: “00000” * 320 ps
delay => “add 0 ps”
Leds: “00001” * 320 ps
delay => “add 320 ps”
Leds: “00011” * 320 ps
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
delay => “add 960 ps”
Start/Stop
delay
October 15, 2009
VLVnT-09 Athens
15
Add delay and Resynchronize
111010110000010101101110101100000101011011101011
111010110000010101101110101100000101011011101011
+ 320 ps
19
19
0
RxRecClk
BitSlide(4:0)
1
0
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
6.4 ns
0011 = 0
0000
19
320 ps
Algorithm:
Propagation Delay = “Start-Stop” Delay + “LED Value” * 320 ps
October 15, 2009
VLVnT-09 Athens
16
FPGA SerDes remarks
y
y
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
y
The Receiver Deserializer should provide
a means to (manually control) “Bit Slip”.
Tested in:
Xilinx
Altera
Lattice
Family
Virtex-5
Stratix-IV-GX
SC/M
SerDes Name
GTX
GXB
FlexiPCS
Bit Slip
RxSlide
Rx_BitSlip
x
Test
Okay
Okay
Fail
Implementation verified at CEA-SACLAY
October 15, 2009
VLVnT-09 Athens
17
Conclusion
It is feasible to measure propagation delay
over an 8B/10B coded link over 100 Km
of fibre.
y A 3.125 Gbps serial link provides a
resolution of 320 ps.
y This can be implemented in an FPGA
y
P. Jansweijer
Nikhef
Amsterdam
ElectronicsTechnology
Thank you
October 15, 2009
VLVnT-09 Athens
18