Application Note
TestPoint
10G PCS Capture: Getting the details
Introduction
This application note describes the 64B/66B coding mechanism within the PCS, and shows
how PCS capture provides details at both the physical and data link layers. PCS capture
enables users to ensure compliance to the standards, and perform troubleshooting. This
tool can be used during the design, verification, and system evaluation phases.
64B/66B Coding Mechanism
The 64B/66B PCS provides robust error detection based on the nature of its coding
mechanism. In the receive direction, the PCS feeds 64 bits from each 66-bit block to the
MAC layer via the XGMII and reconciliation sublayers. A pictorial view of this process is
shown in Figure 1. An inverse procedure applies in the transmit direction.
Figure 1: PCS Receive Process
Since the XGMII has a bus width of 32 bits divided into 4 lanes, two consecutive
transactions are needed to transfer information from the PCS to the XGMII. The first two
bits in each 66-bit block form the synchronization header (sync header). A sync header of
‘10’ indicates a control block while a sync header of ‘01’ indicates a data block. Other sync
header bit combinations are not valid. When the receive interface does not achieve PCS
sync, the link is effectively down. The High BER condition is declared when bit error rate,
based on sync header monitoring, is greater than or equal to 10-4.
The sync header bits bypass the descrambler in the receive direction, and the scrambler
in the transmit direction. Data is scrambled prior to transmission in order to ensure bit
transitions for clock recovery. The TestPoint PCS capture feature displays 64B/66B
information, including the sync header, after descrambling. The Block Sync function of
the PCS receives 16-bit wide data from the PMA or WIS sublayer that interfaces with the
Physical Medium. The WIS is effectively the SONET/SDH layer in 10GBASE-W.
A data block contains eight data characters of eight bits each. A control block starts with
an 8-bit block type field, which indicates the format of the remainder of the block. Table
1 shows the data and control block formats including valid Block Type Field values. The
blank bits in Table 1 are reserved, and should be coded as binary “0”.
WEBSITE: www.jdsu.com/test
Application Note: 10G PCS Capture – Getting the Details
2
Table 1: 64B/66B Block Formats
In control blocks, control codes are noted in Table 1 under the control block format column as Cn, Sn, Tn,
and On. The respective meanings are:
•
•
•
•
Cn: 7-bit control codes, such as idle characters
Sn: implicit frame start character
Tn: implicit frame terminate character
On: 4-bit ordered set code, which is associated with three Dn 8-bit codes
Application Note: 10G PCS Capture – Getting the Details
3
Table 2 contains the list of control codes as defined in IEEE 802.3ae. Ordered sets, whether sequence or
signal, extend the ability to send control and status information over a link. In 10GbE, remote fault and local
fault are the defined ordered sets. An ordered set always begins on lane 0 at the XGMII. Signal ordered sets
are only used in 10G Fibre Channel.
Control Character
idle /I/
start
terminate
error
sequence ordered_set
reserved0
reserved1
reserved2
reserved3
reserved4
reserved5
signal ordered_set
Notation
0x00
/S/
/T/
/E/
/Q/
/R/
/A/
/K/
/Fsig/
10GBASE-R/W Control Code
Implicitly encoded by Block Type Field
Implicitly encoded by Block Type Field
0x1e
Encoded by Block Type Field plus On code
0x2d
0x33
0x4b
0x55
0x66
0x78
Encoded by Block Type Field plus On code
10GBASE-R/W On Code
0x0
0xf
Table 2: Control Codes
10 Gigabit Ethernet
The IEEE 802.3ae-2002 standard defines the Physical Coding Sublayer (PCS) for 10 Gigabit Ethernet
(10GbE). The 64B/66B PCS, where each block is 66 bits in size, applies to both 10GbE serial interfaces, which
are 10GBASE-R (10GbE LAN) and 10GBASE-W (10GbE WAN). The same 64B/66B PCS applies to the
serial interfaces used in 10G Fibre Channel as defined in ANSI INCITS 364-2002 (10G FC). Figure 2 shows
the relationship between the 10GbE sublayers, including MAC and PCS, for 10GBASE-R and 10GBASE-W
PHY.
Typically, Ethernet captures are performed at the MAC layer where the user can analyze frame content. A
PCS capture provides the user with more detailed information at the lower layers. More specifically, a PCS
capture preserves Inter-Frame Gaps (IFG), ordered sets, preambles, and the full MAC frame content.
Figure 2: 10GbE Reference Model
Application Note: 10G PCS Capture – Getting the Details
4
10 Gigabit Fibre Channel
The 10G FC serial interfaces, referred to as 1200-MX-SN-I and 1200-SM-LL-L, use the same 64B/66B PCS
as 10GbE. With 10G FC, the layers above the PCS provide fibre channel primitive qualification and mapping
capabilities. This subsection provides an introduction to the 10G FC primitives and how they are mapped at
the PCS. This information is required in order to effectively use PCS capture with 10G FC.
Figure 3 provides a pictorial view of the 10G FC reference model. The 10G FC Primitive Sequence
Qualification function validates primitive sequences and establishes low level link connectivity. The 10G
FC Translation/Mapping function adapts the fibre channel FC-1 primitives as defined in the INCITS ANSI
FC-FS standard to 10G FC.
Figure 3: 10G FC Reference Model
As in 10GbE, 10G FC uses the Sn and Tn control codes to indicate the start and end of a fibre channel frame.
10G FC makes use of primitive signals and primitive sequences, which are respectively called signal ordered
sets and sequence ordered sets in Table 2 on page 3. An example of primitive signal is the Receiver Ready
(R_RDY) primitive used for flow control in buffer-to-buffer credit management. A Receiver Ready primitive
signal is composed of an On value of 0xf (as in Table 2) followed by three Dn 8-bit codes (see Table 3 on
page 5). Primitive sequences are used to establish low level link connectivity in the fibre channel Port State
Machine.
The full list of 10G FC primitives is provided in Table 3. The table provides the following information:
• The 10G FC primitive name
• The primitive function
• The ordered set Dxx.y/Kxx.y representation
– This representation is the notation used at the 10G FC Translation/Mapping level
– Dxx.y are data characters and Kxx.y are control characters
• The ordered set representation at the 64B/66B PCS level
5
Application Note: 10G PCS Capture – Getting the Details
Primitive Name
SOFc1_X
SOFi1_X
SOFn1_X
SOFi2_X
SOFn2_X
SOFi3_X
SOFn3_X
SOFc4_X
SOFi4_X
SOFn4_X
SOFf_X
EOFt_X
EOFdt_X
EOFa_X
EOFn_X
EOFni_X
EOFdti_X
EOFrt_X
EOFrti_X
IDLE
R_RDY_X
VC_RDY_X
BB_SCs_X
BB_SCr_X
SYNx_X
SYNy_X
SYNz_X
OLS_X
LR_X
LRR_X
LF
RF
Function
Ordered Set
10G FC Translation/Mapping
SOF Connect Class 1
K27.7
D21.5
D23.0
SOF Initiate Class 1
K27.7
D21.5
D23.2
SOF Normal Class 1
K27.7
D21.5
D23.1
SOF Initiate Class 2
K27.7
D21.5
D21.2
SOF Normal Class 2
K27.7
D21.5
D21.1
SOF Initiate Class 3
K27.7
D21.5
D22.2
SOF Normal Class 3
K27.7
D21.5
D22.1
SOF Activate Class 4
K27.7
D21.5
D25.0
SOF Initiate Class 4
K27.7
D21.5
D25.2
SOF Normal Class 4
K27.7
D21.5
D25.1
SOF Fabric
K27.7
D21.5
D24.2
EOF Terminate
D21.4
D21.3
D21.3
EOF Disconnect Terminate Class 1 EOF Deactivate D21.4
D21.4
D21.4
Terminate Class 4
EOF Abort
D21.4
D21.7
D21.7
EOF Normal
D21.4
D21.6
D21.6
EOF Normal Invalid
D10.4
D21.6
D21.6
EOF Disconnect Terminate Invalid Class 1
D10.4
D21.4
D21.4
EOF Disconnect Deactivate Invalid Class 4
EOF Remove Terminate Class 4
D21.4
D25.4
D25.4
EOF Remove Terminate Invalid Class 4
D10.4
D25.4
D25.4
Idle
k07
k07
k07
Receiver Ready
K28.2
D21.4
D10.2
Virtual Circuit Ready
K28.2
D21.7
VC_ID
Buffer to Buffer State Change (SOF)
K28.2
D21.4
D22.4
Buffer to Buffer State Change (R_RDY)
K28.2
D21.4
D22.6
Clock Synchronization Word X
K28.2
D31.3
CS_X
Clock Synchronization Word Y
K28.2
D31.3
CS_Y
Clock Synchronization Word Z
K28.2
D31.3
CS_Z
Port State Machine Primitives
Offline
K28.4
D21.1
D10.4
Link Reset
K28.4
D9.2
D31.5
Link Reset Response
K28.4
D21.1
D31.5
Local Fault
K28.4
D00.0
D00.0
Remote Fault (NOS equivalent)
K28.4
D00.0
D00.0
D23.0
D23.2
D23.1
D21.2
D21.1
D22.2
D22.1
D25.0
D25.2
D25.1
D24.2
K29.7
K29.7
Ordered Set
64B/66B PCS
Sn
0xb5
Sn
0xb5
Sn
0xb5
Sn
0xb5
Sn
0xb5
Sn
0xb5
Sn
0xb5
Sn
0xb5
Sn
0xb5
Sn
0xb5
Sn
0xb5
0x95
0x75
0x95
0x95
0x17
0x57
0x37
0x55
0x35
0x56
0x36
0x19
0x59
0x39
0x58
0x75
0x95
0x17
0x57
0x37
0x55
0x35
0x56
0x36
0x19
0x59
0x39
0x58
Tn
Tn
K29.7
K29.7
K29.7
K29.7
0x95
0x95
0x8a
0x8a
0xf5
0xd5
0xd5
0x95
0xf5
0xd5
0xd5
0x95
Tn
Tn
Tn
Tn
K29.7
K29.7
k07
D10.2
VC_ID
D22.4
D22.6
CS_X
CS_Y
CS_Z
0x95
0x8a
Note
On:0xf
On:0xf
On:0xf
On:0xf
On:0xf
On:0xf
On:0xf
0x99
0x99
0x00
0x95
0xf5
0x95
0x95
0x7f
0x7f
0x7f
0x99
0x99
0x00
0x4a
VC_ID
0x96
0xd6
CS_X
CS_Y
CS_Z
Tn
Tn
0x00
0x4a
VC_ID
0x96
0xd6
CS_X
CS_Y
CS_Z
D21.2
D9.2
D9.2
D01.0
D02.0
On:0x0
On:0x0
On:0x0
On:0x0
On:0x0
0x35
0x49
0x35
0x00
0x00
0x8a
0xbf
0xbf
0x00
0x00
0x55
0x49
0x49
0x01
0x02
Table 3: 10G Fibre Channel primitives
Note: The first idle character in an Inter-Frame Gap is implicitly encoded by Tn; all others are 0x00 (7-bit
control code)
Application Note: 10G PCS Capture – Getting the Details
6
The Dxx.y/Kxx.y representation can be converted to bit values as follows. In Dxx.y/Kxx.y, ‘xx’ is a 5-bit
decimal number represented as ‘EDCBA’, and ‘y’ is a 3-bit number represented as ‘HGF’. Character Dxx.y/
Kxx.y corresponds in hexadecimal to ‘HGFE DCBA’. For example, primitive code D28.5 is equivalent to
0xBC at the PCS. The Kxx.y control characters at the 10G FC Translation/Mapping level map to the following
at the PCS sublayer:
• Start of frame character K27.7 corresponds to Sn
• End of frame character K29.7 corresponds to Tn
• Primitive signal character K28.2 corresponds to On value 0xf
• Primitive sequence character K28.4 corresponds to On value 0x0
• Error character K30.7 corresponds to Cn value 0x1e (7 bits)
• Idle character k07 corresponds to Cn value 0x00 (7 bits)
Note that k07 is represented using a lowercase ‘k’ to indicate a hexadecimal unencoded character at the
XGMII level. Uppercase K and D are used to indicate character encoding at the FC-FS level.
TestPoint 64B/66B PCS Capture
With PCS capture, it is possible to visualize 64B/66B blocks. This subsection provides examples on how to
use this tool and capture information using the trigger capabilities.
Interpreting Captures
On TestPoint, the PCS capture display is formatted into three headings that are: Block Format, Sync Header,
and Block Payload. The Block Format shows the character types in the form of Cn, Sn, Tn, On, and Dn.
The ‘n’ index associated with each character type indicates the logical character position within the block.
On the very left, the line number indicates the order in which consecutive blocks were captured. The two
lines in white characters indicate the trigger point. The actual trigger point is found within one of these two
blocks. The trigger is distributed over two lines because the Data (character) Pattern Match trigger, of up to 8
octets, matches any consecutive data character sequence that may span across either one or two blocks. The
data character trigger looks for a set of data characters found between Sn and Tn characters; hence within a
frame.
Figure 4: Pre-amble and IFG Capture
Application Note: 10G PCS Capture – Getting the Details
7
Ethernet Preambles
The IEEE 802.3ae standard describes data at the XGMII layer as in Figure 5. The associated text provides
additional details on how data is presented at the PCS sublayer. The value of the SFD character, when viewed
on TestPoint in a PCS capture, is 0xd5. The standard 802.3ae preamble bytes, which precede the SFD, are
displayed as 0x55. In a PCS capture, the most significant bit is the first bit received in a sequence and the
order of the received bits on the display is from left to right.
Figure 5: Preamble and Data at the XGMIIl
The standard preamble size is 8 bytes, which includes the SFD (Start Frame Delimiter). At the PCS level, the
first character in the preamble is Sn; the EFD (End Frame Delimiter) is encoded as Tn.
Figure 4 on page 6 shows a 10GbE PCS capture. A MAC preamble is found on line 16. This standard
preamble starts with the implicit S0 followed by six 0x55 data character and ends with an SFD of 0xd5 (0x78
is the Block Type Field). This is an 8-character preamble including SFD. With a PCS capture, the user can for
example: • Count the number of characters in a preamble
• Determine the content of the preamble
• Verify the SFD value
The user can also examine the contents of proprietary or non-standard preambles. TestPoint supports
long and short preamble counters to identify preambles of size other than 8 (including SFD). On the
receive interface, TestPoint searches for the SFD to find the beginning of the MAC frame, which starts
with the destination MAC address. Following the preamble, the capture shows the destination address of
44:44:44:44:44:44 and the source address of 22:22:22:22:22:22 on lines 17 and 18 of Figure 4. In order to
perform a PCS capture and examine preambles, the user can use a Data Pattern Match trigger based on
the preamble byte values, where the standard byte value is 0x55 and the SFD is 0xd5, or the six bytes of the
destination MAC address. The preamble is found immediately before the destination MAC address. Another
possibility is to look for the Sn character in the block format. Since start of frame can only be found in lane
0 or 4 at the XGMII, the user can trigger on Block Type Field values 0x33, 0x66, or 0x78 (reference Table 1
on page 2). Once a PCS capture has been performed, another possibility is to visually identify preambles by
looking for idle control characters, which are found in groups of 0x00 7-bit codes within control blocks. Idle
control characters always precede a preamble.
Inter-Frame Gaps
PCS capture allows the user to view and analyze Inter-Frame Gaps (IFG). IFGs are composed of a series of
idle control characters. More specifically, it is possible to count and record the exact number of idle characters
separating MAC frames. The Tn character, which corresponds to the end of frame delimiter, marks the end
of a MAC frame. Each Tn implicitly encodes an idle character from the XGMII. Following Tn, is a sequence
of 0x00 7-bit control characters, each representing an idle character. The number of 0x00 7-bit control
characters plus the Tn character represents the number of idle control characters. When directly converted
at the XGMII, this corresponds to a number of idle bytes. In Figure 4, there are two sets of 8 idle characters
on lines 15 and 25. Since the 10GbE maximum MAC data rate averages to 12 bytes of idle characters between
frames, it can be concluded that the captured traffic exceeded this MAC line rate. The absolute minimum
number of idle characters on a 10GbE receive interface is 5 according to IEEE 802.3ae. TestPoint supports
a short IFG counter based on a user specified threshold that defaults to 5 to highlight potential violations.
In support of this feature, the PCS capture lets users capture and display the information. Please note that
on the transmit interface, TestPoint supports a deficit idle count of +/- 3. In practice, this means that the
provisioned number of IFG characters when generating traffic on the transmit interface is guaranteed +/- 3.
The provisioned IFG value is an average and the variations on the IFG size depend on the MAC frame size.
For certain frames sizes, the IFG will be exactly as provisioned and for others, it will vary within three. For
example, the provisioned IFG size is the exact value for a fixed frame size of 64 bytes as long as the regular
Application Note: 10G PCS Capture – Getting the Details
8
preamble of 8 bytes (including SFD) is used. This is because 64 bytes is a multiple of 8 and there are 8 data
characters per block. With a frame size of 65 bytes, the IFG size will vary during transmission to ensure that
the average matches the provisioned IFG value.
Errored blocks
Another application of PCS capture is to analyze errored blocks. An errored block can occur as a result of
many conditions. They include:
• Invalid Sync Header (‘00’ or ‘11’)
• Invalid Block Type Field (i.e. value not found in Table 1)
• Invalid control code (i.e. value not found in Table 2)
• Error control character (as in Table 2)
PCS capture triggers include Sync Header Error and Invalid PCS Code, the latter corresponding to an invalid
Block Type Field. These triggers can be used with a middle or end trigger position to capture the preceding
blocks. When using the PCS Sync Loss trigger, an end trigger position must be used as the blocks that follow
the loss of sync cannot be delineated. Figure 6 shows an example of invalid code where a ‘00’ Sync Header
was received on line 2049.
Figure 6: Invalid Sync Header
Application Note: 10G PCS Capture – Getting the Details
9
Figure 7 shows a PCS capture with an invalid Block Type Field. On line 1, the Sync Header is ‘10’ indicating a
control block, and the first octet is 0x00, which is a Block Type Field value not found in Table 1 on page 2.
Figure 7: Invalid Block Type Field
Another example in Figure 8 shows the data captured when an error control character is received. This likely
indicates that a low-level error has occurred at the far-end transmitter. The 0x1e 7-bit control codes on line
2049 represent the error control character.
Figure 8: Error Control Character
Application Note: 10G PCS Capture – Getting the Details
10
XGMII Lane Troubleshooting
An interesting application of PCS capture is the troubleshooting of XGMII lane skew issues. In the transmit
direction, data from the reconciliation sublayer is placed onto four separate lanes at the XGMII, where each
lane is 8-bit wide. Should a time skew occur between the lanes, data within the 64B/66B blocks will misalign.
A PCS capture can highlight this issue by using a repetitive, predictable test pattern. For a data character
stream, the XGMII character on lane 0 will be reflected at a D0 or D4 position. Similarly, lane 1 characters
will appear at a D1 or D5 position. The same concept applies to lanes 2 and 3, which will appear at D2/D6
and D3/D7 respectively.
Ordered Sets
Figure 9shows a PCS capture with a sequence ordered set. On line 2049 and on the following lines, each O0/4
code of value 0x0 is followed by three Dn characters of values 0x00, 0x00, and 0x02. This capture shows a
remote fault ordered set. A local fault ordered set would be displayed as an O0/4 value of 0x0 associated with
three Dn characters of values 0x00, 0x00, and 0x01.
Figure 9: Remote Fault Ordered Set
Application Note: 10G PCS Capture – Getting the Details
11
10G FC Capture
Figure 10 shows a 10G FC PCS capture. Here is a description of a number of elements that can be observed.
On line 1, series of 0x00 idle characters are found. They are followed by S4 and three data characters which
are 0xb5, 0x56, and 0x56. As in Table 3, this corresponds to an SOF Initiate Class 3 10G FC primitive. Hence,
this is the start of the fibre channel frame. On line 5, a T7 character is preceded by 0x95, 0x75 and 0x75.
This corresponds to an EOF Terminate as in Table 3. Another primitive that can be observed is a Receiver
Ready primitive signal (signal ordered set). Receiver Ready is used in flow control for buffer-to-buffer credit
management. It is found on line 17. It starts with an O0 control character of value 0xf followed by 0x95,
0x4a, and 0x4a (the Block Type Field is 0x4b). 10G FC PCS capture shows all the details on the information
received.
This includes all primitives, data characters, and framing information.
Figure 10: 10G FC PCS Capture
Summary
PCS capture is an innovative and powerful tool to gain insight into the physical coding used in 10GbE and
10G FC. The power of this feature also lies in the troubleshooting capabilities it provides. 10G PCS capture
is essential to assist hardware and firmware designers in validating their designs. Further, it is a valuable tool
for engineers who seek an in-depth understanding of the systems they test or evaluate.
Test & Measurement Regional Sales
NORTH AMERICA
LATIN AMERICA
ASIA PACIFIC
TEL: +1 866 228 3762
FAX: +1 301 353 9216
TEL: +1 954 688-5660
FAX: +1 954 3454668
TEL: +852 2892 0990
FAX: +852 2892 0770
EMEA
TEL: +49 7121 86 2222
FAX: +49 7121 86 1222
Product specifications and descriptions in this document subject to change without notice. © 2010 JDS Uniphase Corporation
WEBSITE: www.jdsu.com/test
30149160 500 0710
10GPCSCAPTURE.AN.OPT.TM.AE