Need for Speed: Beyond 100GbE
Moderator: Scott Kipp, President of Ethernet Alliance, Principle Engineer, Brocade
Panelist #1: Alan Weckel, Vice President, Dell’Oro group
Panelist #2: Dr. Jeffery J. Maki, Distinguished Engineer, Juniper
Panelist #3: Dr. Gordon Brebner, Distinguished Engineer, Xilinx
© 2013 Ethernet Alliance
1
Agenda


Introductions: Scott Kipp, Moderator
Panelist #1: Alan Weckel,


Panelist #2: Dr. Jeffery J. Maki,



Stepping Stones to Terabit-Class Ethernet
Panelist #3: Dr. Gordon Brebner,


10, 40 and 100GbE Deployments in the Data Center
Technology Advances in 400GbE Components
Q&A
2:40 – Live Broadcast from IEEE 802.3 Meeting in
Orlando from John D’Ambrosia

Update on 400GbE Call For Interest
© 2013 Ethernet Alliance
© 2012 Ethernet Alliance
2
Disclaimer

The views WE ARE expressing in this
presentation are our own personal views
and should not be considered the views or
positions of the Ethernet Alliance.
© 2013 Ethernet Alliance
3
Bandwidth Growth
Broadband
2010- 7Mbps
2015 – 28 Mbps
15B Devices
In 2015
Increased #
of
More
Users
Increased
+ Access
Devices
Rates and
Methods
More Internet
Users
+
Increased
Services
Key
Growth
Factors
Speed
= Bandwidth
Explosion
Increasing
Everywhere
More Rich Media
Content
2010- 1 Minute video
2015 – 2 hour HDTV Movie
3B Users
In 2015
Source: nowell_01_0911.pdf citing Cisco Visual Networking Index (VNI) Global IP Traffic Forecast, 2010–2015,
http://www.ieee802.org/3/ad_hoc/bwa/public/sep11/nowell_01_0911.pdf
© 2013 Ethernet Alliance
4
Bandwidth Growth Vs
Ethernet Speeds


IP Traffic is growing ~ 30%/year
If 400GbE is released in 2016, Ethernet speeds will
grow at about 26%/year
Ethernet Speed (Gb/s)
Internet traffic normalized
to 100 in 2010
1600
1400
1200
1000
800
600
Internet traffic
would grow ~10X by
2019 at 30%/year
Ethernet speeds
to grow 4X by
2016 at 26%/year
Ethernet Speed
Internet Traffic
400
200
0
© 2013 Ethernet Alliance
5
Data Rate and Line Rate (b/s)
Ethernet Optical Modules
CFP
CFP2
100G
300 Pin
MSA
XENPAK
XPAK
X2
CFP4
100GbE
XFP
CXP
40GbE
40G
QSFP28
QSFP+
10GbE
10G
SFP+
GBIC
SFP
Key:
Ethernet
Standard
Released
Module
Form Factor
Released
GbE
1G
1995
2000
2005
2010
2015
Standard Completed
© 2013 Ethernet Alliance
6
Data Rate and Line Rate (b/s)
Ethernet Speeds 2010-2025
If Ethernet line rates doubles the line rate every
3 years at 26% CAGR, then 400GbE would come
out in 2016 and TbE would come out in 2020.
Something will have to change.
1.6TbE
16X100G
TbE
10X100G
1T
400G
100G
40G
10G
400GbE
16X25G
100GbE
10X10G
400GbE
8X50G
100GbE
4X25G
100GbE
1X100G
2010
8X50G
4x25G
2015
16x25G
2020
Ethernet
Electrical
Interfaces
Hollow
Symbols =
predictions
Stretched
Symbols =
Time
Tolerance
nX100G
40GbE
4X10G
4x10G
10X10G
400GbE
4X100G
Key:
Ethernet
Speeds
2025
Standard Completed
© 2013 Ethernet Alliance
7
Ethernet Success



Ethernet has been extremely successful at
lowering the price/bit of bandwidth
If the cost of a new speed/technology is too
high, then it is not widely deployed
Technology needs to be ripe for picking



400GbE is ripe with 100GbE technology
TbE isn’t ripe and a revolutionary breakthrough
would be needed to get it before 2020
This panel will look at how high speeds of
Ethernet are being deployed and the
technology that is leading to the next
generation of Ethernet
© 2013 Ethernet Alliance
8
10, 40 and 100GbE Deployments in the
Data Center
Alan Weckel
Vice President, Data Center Research
Dell’Oro Group
© 2013 Ethernet Alliance
9
Introduction

Progress on server migration from 1 GbE to
10 GbE

10G Base-T update

Data center networking market update

40 GbE and 100 GbE market forecasts
© 2013 Ethernet Alliance
10
Overview

Dell’Oro Group is a market research firm that
has been tracking the Ethernet Switch and
Routing markets on a quarterly basis since
1996

We also track the SAN market, Optical
market, and most Telecom equipment
markets

We produce quarterly market share reports
that include port shipments as well as market
forecasts
© 2013 Ethernet Alliance
11
700
350
© 2013 Ethernet Alliance
20
16
20
15
20
14
20
13
20
12
20
11
20
10
0
20
09
Petabytes per Second Shipped
per Year
Data Center Bandwidth Shipping –
Ethernet Switching
12
1 GbE
40 GbE
10 GbE
100%
75%
50%
25%
© 2013 Ethernet Alliance
20
16
20
15
20
14
20
13
20
12
20
11
20
10
0%
20
09
Percent of Server Shipments
Switch Attach Rate on Servers
13
180
10G Base-T controller
and adapter ports
90
10G Base-T switch ports
© 2013 Ethernet Alliance
12
4Q
12
3Q
12
2Q
12
1Q
11
4Q
11
3Q
11
2Q
11
0
1Q
Port Shipments in Thousands
Data Center Port Shipments –
10 G Base-T Port Shipments
14
50
1 GbE
10 GbE
40 GbE
100 GbE
25
© 2013 Ethernet Alliance
20
16
20
15
20
14
20
13
20
12
20
11
20
10
0
20
09
Port Shipments in Millions
Data Center Port Shipments –
Ethernet Switching
15
6
40 GbE
100 GbE
3
© 2013 Ethernet Alliance
20
16
20
15
20
14
20
13
20
12
20
11
20
10
0
20
09
Port Shipments in Millions
Data Center Port Shipments –
Ethernet Switching
16
Summary

Ethernet Switches will be responsible for the
majority of 40 GbE and 100 GbE port
shipments over the next five years

Form-factor and cost driving 40 GbE over
100 GbE

10 GbE server access transition is key to
higher speed adoption
© 2013 Ethernet Alliance
17
Stepping Stones to Terabit-Class Ethernet:
Electrical Interface Rates and
Optics Technology Reuse
Jeffery J. Maki
Distinguished Engineer, Optical
Juniper Networks, Inc.
© 2013 Ethernet Alliance
18
100G
© 2013 Ethernet Alliance
19
CFP, CFP2 and CFP4 for
SMF or MMF Applications
CFP MSA Form Factors:
http://www.cfp-msa.org/
CFP4
CFP2
CFP
Optical Connector
• LC Duplex (depicted)
Courtesy of
TE Connectivity
• MPO
© 2013 Ethernet Alliance
20
Module Electrical Lane
Capability
CFP4
CFP2
CFP
12x10G
electrical
lanes
10x10G or 8x25G
electrical
lanes
4x25G
electrical
lanes
CAUI for 10x10G
CPPI & CAUI for 10x10G
CAUI-4 for 4x25G
CAUI-4 for 4x25G
© 2013 Ethernet Alliance
21
CFP, CFP2, and CFP4 for
100G Ethernet SMF PMD
Gear Box
Transmit side only depicted.
LAN WDM
1295.56 nm
1300.05 nm
1304.58 nm
1309.14 nm
Gear Box
LAN WDM
1295.56 nm
1300.05 nm
1304.58 nm
1309.14 nm
4 λ on LAN WDM
CFP
Current Options
• Up to 10 km:
100GBASE-LR4
• Up to 40 km:
100GBASE-ER4
CFP2
CFP4
© 2013 Ethernet Alliance
22
400G
© 2013 Ethernet Alliance
23
Projection of Form Factor
Evolution to 400G
speculation
CD-CFP4
16x25G
electrical
lanes
CD-CFP2
CD-CFP
CFP4
CFP2
CFP
4x100G
8x50G
electrical electrical
lanes
lanes
CFP4
CFP4
CFP4
CFP4
Roman Numerals
XL = 40
C = 100
CD = 400
400G
defensible
100G
© 2013 Ethernet Alliance
24
Likely MSA Activity

CFP MSA http://www.cfp-msa.org/




CD-CFP: Current CFP needs revamping to support 16 x 25G
CD-CFP2: Current CFP2 is ready for 8 x 50G
CD-CFP4: Unclear
New CDFP MSA http://www.cdfp-msa.org/

High-density form factor supporting 16 x 25G

From slide 26 of
http://www.ieee802.org/3/cfi/0313_1/CFI_01_0313.pdf
© 2013 Ethernet Alliance
25
400G Optics Requirements

First-generation transceivers have to be
implementable that meet and eventually
do better than these requirements




Size (Width):  82 mm (CFP width, ~4 x CFP4)
Cost:  4 x CFP4
Power:  24 W (4 x 6 W power profile of CFP4)
Improved bandwidth density transceivers
will need higher rate electrical-lane
technology


50G
100G
© 2013 Ethernet Alliance
26
How 400G Ethernet Can
Leverage 100G Ethernet
100G Ethernet up to 10 km
CFP4-LR4
Duplex Single-Mode
Fiber Infrastructure
CFP4-LR4
400G Ethernet up to 10 km
Parallel Single-Mode Fiber Infrastructure
CFP4-LR4
CFP4-LR4
Only 8
Fibers
Used
CFP4-LR4
CFP4-LR4
CFP4-LR4
CFP4-LR4
CFP4-LR4
CFP4-LR4
© 2013 Ethernet Alliance
27
Possible SMF Ethernet Road
Map: 100G, 400G, 1.6T
Early Adopter 400G
Mature 400G
4 x 100GBASE-LR4
or
“400GBASE-PSM4”
400GBASE-???
CFP4(LC)
CD-CFP2(LC)
CD-CFP4(LC)
CFP4(LC)
CD-CFP(MPO)
CD-CFP2(MPO)
4 x 400GBASE-???
or
“1600GBASE-PSM4”
CD-CFP4(LC)
CD-CFP4(LC)
CFP4(LC)
CFP4(LC)
Early Adopter 1.6T
CD-CFP4(LC)
CD-CFP4(LC)
Parallel Single Mode, 4 Lanes (PSM4)
4, Tx Fibers and 4, Rx Fibers
1x12 MPO Connector
(High-Density 100GE)
© 2013 Ethernet Alliance
28
Early Adopter 400G using
SMF Structured Cabling
Parallel SMF:
“400GBASE-PSM4”
Technology Reuse:
4 x 100GBASE-LR4
Courtesy of
Commscope
© 2013 Ethernet Alliance
29
Early Adopter 400G using
MMF Structured Cabling
Courtesy of
Commscope
Technology Reuse:
4 x 100GBASE-SR4
Parallel MMF:
“400GBASE-SR16”
Parallel Multi-Mode
• 100GBASE-SR4, 4 x 25G optical lanes:
4, Tx Fibers and 4, Rx Fibers using 1x12 MPO
• “400GBASE-SR16”, 16 x 25G optical lanes:
16, TX Fibers and 16, Rx Fibers using 2x16 MPO
© 2013 Ethernet Alliance
30
MMF Breakout Cables—
Enabling 400G Adoption
2 x 16 MPO
1 x 12 (8 used) MPO
2 x 16 MMF MT ferrule
1 x 12 (8 used) MPO
1 x 12 (8 used) MPO
1 x 12 (8 used) MPO
© 2013 Ethernet Alliance
Courtesy of
USConec
31
100G Can Build 400G at
the Cost of 4 x 100G
Technology Reuse:
4 x 100GBASE-LR4
Parallel SMF:
“400GBASE-PSM4”
Technology Reuse:
4 x 100GBASE-SR4
© 2013 Ethernet Alliance
Parallel MMF:
“400GBASE-SR16”
32
Ethernet PMD Maturity &
Possible Obsolescence

Early Adopter PMD





Parallel Fiber, SMF or MMF
Leverage of mature PMD from previous speed of
Ethernet
Planned obsolescence
Implementation (with MPO connector) persists as
high-density support of previous speed of Ethernet
(e.g., 4 x 100G)
Mature PMD


SMF: Duplex SMF cabling (e.g., with LC duplex
connector)
MMF: Lower fiber count MMF cabling
© 2013 Ethernet Alliance
33
SMF Density Road Map
16
CD-CFP4(LC)
(mature)
Front-Panel
Bandwidth
Density
(Relative)
8
(early
adopter)
CD-CFP2(MPO)
CD-CFP2(MPO)
CD-CFP2(LC)
(mature)
(early adopter)
(mature)
4
CFP4(LC)
2
1
CFP2(LC)
CFP(LC)
100G
4x
CD-CFP4(LC)
4 x CFP4(LC) or CD-CFP(MPO)
(early adopter)
Port Bandwidth
400G
© 2013 Ethernet Alliance
1.6T
34
Summary





Form-factor road map for bandwidth
evolution
Early adopter 400G Ethernet by reusing 100G
module and parallel cabling, SMF or MMF
Need for a new, 2 x 16 MMF MT ferrule
Possible common module for 400G Ethernet
and high-density (4-port) 100G Ethernet
Need for new electrical interface definitions
supporting lane rates at


50G
100G
© 2013 Ethernet Alliance
35
Technology Advances in 400GbE
Components
Gordon Brebner
Distinguished Engineer
Xilinx, Inc.
© 2013 Ethernet Alliance
36
400GbE PCS/MAC

Expect first: 16 PCS lanes, each at 25.78125 Gbps




Glueless interface to optics
Possible re-use of the 802.3ba PCS
Other options possible for PCS, maybe native FEC
Later: 8 lanes, each at 51.56Gbps

Or 4 lanes with 2 bits/symbol at 56Gbaud (e.g. PAM4)

Packet size 64 bytes to 9600 bytes

Use 100GbE building blocks where possible
© 2013 Ethernet Alliance
37
Silicon technology

Technology nodes (silicon feature size)


Application-Specific Integrated Circuit (ASIC)




130nm, 65nm, 40nm, 28/32nm, 20/22nm, 14/16nm
Fixed chip
Increasingly expensive: need high volumes
Best suited to post-standardization Ethernet
Field Programmable Gate Array (FPGA)



Programmable logic chip
Suitable for prototyping and medium volumes
Best choice for pre-standardization Ethernet
© 2013 Ethernet Alliance
38
400GbE line/system bridge
Wide parallel data path between blocks
CDFP
or
4xCFP4
16 x 25G
SERDES
400GbE
400GbE
PMA/PCS
MAC
Bridge
logic
40 x 12.5G
or
48 x 10G
SERDES
500G
Interlaken
Optical
ASIC or FPGA chip
Line side
System side
© 2013 Ethernet Alliance
39
MAC rate = Width x Clock
400 Gbps and 1 Tbps Ethernet MAC options
MAC rate
Silicon node
Technology
Data path width
Clock frequency
100 Gbps
45, 40nm
ASIC
160 bits
644 MHz
100 Gbps
45, 40nm
FPGA
512 bits
195 MHz
400 Gbps
28, 20nm
ASIC
400 bits
1 GHz
400 Gbps
28, 20nm
FPGA
1024 bits
1536 bits
400 MHz
267 MHz
1 Tbps
20, 14nm
ASIC
1024 bits
1 GHz
1 Tbps
20, 14nm
FPGA
2048 bits
2560 bits
488 MHz
400 MHz
© 2013 Ethernet Alliance
40
Multiple Packets/Word

Bus width
Max packets
Max EOPs
512
2
1
1024
3
2
1536
4
3
512 * n
n+1
n
Up to 512-bit, only one packet completed


Just need to deal with EOP then SOP in word
Beyond 512-bit, multiple packets completed


Need to add parallel packet processing
Must deal with varying EOP and SOP positions
© 2013 Ethernet Alliance
41
400GbE CRC Example

All Ethernet packets carry Cyclic Redundancy
Code (CRC) for error detection



Computed using CRC-32 polynomial
Critical function within Ethernet MAC
Requirements



Computed at line rate
Deal with multiple packets in wide data path
Economical with silicon resources
© 2013 Ethernet Alliance
42
400GbE CRC Prototype

Xilinx Labs research project



Modular: built out of 512-bit 100G units
Computes multiple CRCs per data path word
Targeting 28nm FPGA (Xilinx Virtex-7 FPGAs)
512-bit unit
CRC results
combined
to get final
CRC results
N-bit data
path
partitioned
into 512-bit
sections
© 2013 Ethernet Alliance
43
400GbE CRC Prototype



Results:
Data bus word size
1024-bit
1536-bit
2048-bit
Max clock frequency (MHz)
400
381
326
Maximum line rate (Gbps)
409
585
668
Latency (ns)
17.5
18.4
21.5
FPGA resources (slices)
2,888
4,410
5,719
1024-bit width is feasible for 400GbE
Other widths:


Less challenging clock frequencies
Demonstrate scalability beyond 400GbE
© 2013 Ethernet Alliance
44
Conclusions

Can anticipate 400GbE PCS/MAC standard

Ever-increasing rates mean ever-wider
internal data path width in electronics

Leading to multiple packets per data word

Possible to prototype pre-standard
PCS/MAC using today’s FPGA technology

Demonstrated modular Ethernet CRC block
based on 100GbE units

Silicon resource scales linearly with line rate
© 2013 Ethernet Alliance
45