UCS Networking Deep Dive
BRKCOM-2003
Michael Ciesla
Customer Support Engineer
#clmel
Cisco Unified Computing System (UCS)
Single Point of Management
Logical Building Blocks
BRKCOM-2003
Hardware/Software
(Service Profiles)
3
Cisco Public Abstraction
© 2015 Cisco and/or its affiliates. All rights reserved.
UCS Components
LAN
MGMT
SAN
Fabric Interconnect
UCS Chassis
Heartbeat link (No Data)
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
4
UCS Components
LAN
MGMT
SAN
Fabric Interconnect
UCS Chassis
Heartbeat link (No Data)
IO Module
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
5
UCS Components
LAN
MGMT
SAN
Fabric Interconnect
4 x 10G KR lanes to each
half width blade slot.
UCS Chassis
Heartbeat link (No Data)
IO Module
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
6
UCS Components
LAN
Fabric Interconnect
MGMT
SAN
4 x 10G KR lanes to each
half width blade slot
UCS Chassis
Heartbeat link (No Data)
IO Module
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
7
UCS Components
Fabric Interconnect
Cisco
VIC
IO Module
Heartbeat link (No Data)
UCS Blade
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
8
UCS 6248 Hardware Diagram
Carmel 1
Carmel 2
Carmel 3
South
Bridge
PCIe x8
Flash
Serial
12 Gig
Intel
Jasper Forest
Memory
NVRAM
PEX 8525
4 port PCIE
Switch
12 Gig
PCIe x4
Carmel
CPU
Unified Crossbar Fabric
Sunnyvale
12 Gig
12 Gig
Carmel 4
Carmel 5
Carmel 6
10 Gig
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
9
0
1
PCIE
Dual Gig
DDR3 x2
Carmel 1
Carmel 2
Carmel 6
Carmel cpu
Sunnyvale
10 Gig
PCIe x4
PCIe x4
PCIE
Dual Gig
0
Xcon1
1
Mgmt
Xcon2 Console
PCIE
Dual Gig
0
1
N/C
TAC Tip: Carmel ASIC Port Mapping
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Cisco UCS 6200 Series Fabric Interconnects
Flexibility
Product Features and Specs
UCS 6248UP
UCS 6296UP
960 Gbps
1.92 Tbps
1RU
2RU
1 Gigabit Ethernet Port Density
48
96
10 Gigabit Ethernet Port Density
48
96
8G Native FC Port Density
48
96
Port-to-Port Latency
2.0us
2.0us
Active # of VLANs
2000
2000
Switch Fabric Throughput
Scalability
Multipurpose
BRKCOM-2003
Switch Footprint
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
UCS Mini: 6324 Fabric Interconnect
+
UCS B
IO Modules
+
6248 or 6296 Fabric
Fabric Interconnects
UCS 5108 Chassis
Supports existing and future blades
+
UCS Mini
6324 Fabric Interconnect
UCS 5108 Chassis
Supports existing and future blades
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Chassis Connectivity Options
UCS Fabric Topologies
Chassis Bandwidth Options
2208XP
only
2x 1 Link
2x 2 Link
2x 4 Link
2x 8 Links
20 Gbps per Chassis
40 Gbps per Chassis
80 Gbps per Chassis
160 Gbps per Chassis
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
14
UCS 2200 IO Module (FEX)
• UCS-IOM-2204XP
• UCS-IOM-2208XP
• 40G to the Network
• 80G to the Network
• 160G to the Hosts
• 320G to the Hosts
– 2x10G Half width slot
– 4x10G Full width slot
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
– 4x10G Half width slot
– 8x10G Full width slot
Cisco Public
UCS IOM 220x Architecture
Fabric Ports to FI
Network Interfaces (NIFs)
2208
2204
FLASH
DRAM
Feature
2204-XP
2208-XP
ASIC
Woodside
Woodside
Fabric Ports
(NIF)
4
8
Host Ports
(HIF)
16
32
Latency
~ 500ns
~ 500ns
EEPROM
Chassis
Management
Controller
Control
Woodside ASIC
IO
CIMC Switch
2204
Chassis
Signals
2208
Internal backplane ports to blades
Host Interfaces (HIFs)
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
16
UCS Internal Block Diagram
UCS 6248
UCS 6248
Fabric
Interconnects
16x SFP+
16x SFP+
Expansion Module
16x SFP+
16x SFP+
Expansion Module
Fabric Uplinks (NIFs)
IO Modules
2208XP
2208XP
Backplane Ports (HIFs)
Midplane
Adapter
mLOM
Mezz
CPU 0
Server Blade
CPU 1
QPI Link
UCS Blade Chassis
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
IOM Fabric & Backplane Interfaces
UCSB-2-A# connect nxos !
UCSB-2-A(nxos)# show fex detail!
FEX: 1 Description: FEX0001
state: Online!
…!
Extender Model: UCS-IOM-2204XP, Part No: 73-14488-01!
pinning-mode: static
Max-links: 1!
Fabric interface state:!
Eth1/3 - Interface Up. State: Active!
Eth1/4 - Interface Up. State: Active!
Fex Port
State Fabric Port!
Eth1/1/1 Down
Eth1/3!
Eth1/1/2 Down
None!
Eth1/1/3
Up
Eth1/4!
Eth1/1/4 Down
None!
Eth1/1/5
Up
Eth1/3!
Eth1/1/6
Up
Eth1/3!
Eth1/1/7
Up
Eth1/4!
Eth1/1/8 Down
None!
Eth1/1/9
Up
Eth1/3!
Eth1/1/10 Down
None!
Eth1/1/11
Up
Eth1/4!
Eth1/1/12
Up
Eth1/4!
Eth1/1/13
Up
Eth1/3!
Eth1/1/14 Down
None!
Eth1/1/15
Up
Eth1/4!
Eth1/1/16 Down
None!
Eth1/1/17
Up
Eth1/4!
FI ports IOM connects to
Backplane port to blade 1/3
Backplane to FI pinning
1G link to CIMC switch.
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
UCSB-2-A# connect iom 1!
fex-1# show platform software woodside sts!
Uplink #:
1 2 3 4 5 6 7 8 !
Link status:
| | : :
!
+-+--+--+--+--+--+--+--+-+!
SFP:
[$][$][ ][ ][ ][ ][ ][ ]!
+-+--+--+--+--+--+--+--+-+!
| N N N N N N N N |!
| I I I I I I I I |!
| 0 1 2 3 4 5 6 7 |!
|
|!
|
NI (0-7)
|!
+------------+-----------+
!
|
!
+-------------------------+-------------+-------------+---------------------------+
!
|
|
|
|
!
+------------+-----------+ +-----------+------------+ +------------+-----------+ +-------------+----------+!
|
HI (0-7)
| |
HI (8-15)
| |
HI (16-23)
| |
HI (24-31)
|!
|
| |
| |
| |
|!
| H H H H H H H H | | H H H H H H H H | | H H H H H H H H | | H H H H H H H H |!
| I I I I I I I I | | I I I I I I I I | | I I I I I I I I | | I I I I I I I I |!
| 0 1 2 3 4 5 6 7 | | 8 9 1 1 1 1 1 1 | | 1 1 1 1 2 2 2 2 | | 2 2 2 2 2 2 3 3 |!
|
| |
0 1 2 3 4 5 | | 6 7 8 9 0 1 2 3 | | 4 5 6 7 8 9 0 1 |!
+-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+!
[ ][ ][ ][ ][ ][ ][ ][ ]
[ ][ ][ ][ ][ ][ ][ ][ ]
[ ][ ][ ][ ][ ][ ][ ][ ]
[ ][ ][ ][ ][ ][ ][ ][ ]!
+-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+!
- |
- |
| |
- |
- |
| |
- |
- : !
1 1
1 1
1 1
1 9
8 7
6 5
4 3
2 1 !
6 5
4 3
2 1
0
!
\__\__/__/ \__\__/__/
\__\__/__/ \__\__/__/
\__\__/__/ \__\__/__/
\__\__/__/ \__\__/__/ !
blade8
blade7
blade6
blade5
blade4
blade3
blade2
blade1
!
fex-1# !
To Eth1/3-4 on FI
IOM backplane interfaces Eth1/1/5-6
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
19
IOM Traffic Rate Monitoring
UCSB-2-A# connect iom 1!
fex-1# show platform software woodside rate !
+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+!
| Port
|| Tx Packets | Tx Rate |
Tx Bit
|| Rx Packets | Rx Rate |
Rx Bit
|Avg Pkt|Avg Pkt|
|!
|
||
| (pkts/s) |
Rate
||
| (pkts/s) |
Rate
| (Tx) | (Rx) |Err|!
+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+!
| 0-BI
||
15 |
3 |
3.05Kbps ||
11 |
2 |
5.76Kbps |
107 |
307 |
|!
| 0-CI
||
45 |
9 | 16.84Kbps ||
34 |
6 | 18.46Kbps |
214 |
319 |
|!
| 0-NI1 ||
76 |
15 | 58.50Kbps ||
154 |
30 | 62.91Kbps |
461 |
235 |
|!
| 0-NI0 ||
2 |
0 |
3.48Kbps ||
131 |
26 | 44.30Kbps | 1072 |
191 |
|!
| 0-HI27 ||
83 |
16 | 33.44Kbps ||
2 |
0 |
1.00Kbps |
231 |
293 |
|!
| 0-HI23 ||
97 |
19 | 35.65Kbps ||
1 |
0 | 768.00 bps |
209 |
464 |
|!
| 0-HI22 ||
97 |
19 | 35.64Kbps ||
1 |
0 | 352.00 bps |
209 |
200 |
|!
| 0-HI19 ||
138 |
27 | 57.78Kbps ||
41 |
8 | 40.82Kbps |
241 |
602 |
|!
| 0-HI15 ||
97 |
19 | 35.65Kbps ||
1 |
0 | 768.00 bps |
209 |
464 |
|!
| 0-HI11 ||
1 |
0 | 152.00 bps ||
1 |
0 | 352.00 bps |
78 |
200 |
|!
| 0-HI7 ||
83 |
16 | 33.44Kbps ||
1 |
0 | 768.00 bps |
231 |
464 |
|!
| 0-HI3 ||
1 |
0 | 152.00 bps ||
1 |
0 | 352.00 bps |
78 |
200 |
|!
+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+!
fex-1# !
Statistics from perspective of IOM!.
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
20
Fabric Link Connectivity
21
Chassis Connectivity Policy
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
22
IO Module HIF to NIF Pinning
2208XP – 1 Link
Slot 1
HIF1-4
NIF1
HIF1-4
Slot 2
HIF5-8
HIF5-8
Slot 3
HIF9-12
HIF9-12
Slot 4
HIF13-16
HIF13-16
Slot 5
HIF17-20
HIF17-20
Slot 6
HIF21-24
HIF21-24
Slot 7
HIF25-28
HIF25-28
Slot 8
HIF29-32
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
HIF29-32
Cisco Public
23
NIF1
IO Module HIF to NIF Pinning
2208XP – 2 Links
Slot 1
HIF1-4
HIF1-4
Slot 2
NIF1
HIF5-8
HIF5-8
Slot 3
NIF2
HIF9-12
NIF2
HIF9-12
Slot 4
HIF13-16
HIF13-16
Slot 5
HIF17-20
HIF17-20
Slot 6
HIF21-24
HIF21-24
Slot 7
HIF25-28
HIF25-28
Slot 8
HIF29-32
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
HIF29-32
Cisco Public
24
NIF1
IO Module HIF to NIF Pinning
2208XP – 4 Links
Slot 1
HIF1-4
HIF1-4
Slot 2
NIF1
HIF5-8
Slot 3
NIF2
NIF3
NIF4
HIF5-8
HIF9-12
NIF2
HIF9-12
Slot 4
HIF13-16
Slot 5
HIF17-20
Slot 6
HIF21-24
HIF21-24
Slot 7
HIF25-28
HIF25-28
Slot 8
HIF29-32
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
HIF29-32
Cisco Public
25
NIF3
NIF4
HIF13-16
HIF17-20
NIF1
IO Module HIF to NIF Pinning
2208XP – 8 Links
Slot 1
HIF1-4
HIF1-4
Slot 2
NIF1
HIF5-8
Slot 3
NIF2
NIF3
NIF4
HIF9-12
HIF13-16
NIF7
NIF8
HIF13-16
Slot 5
HIF17-20
NIF6
HIF21-24
NIF7
NIF8
HIF25-28
Slot 8
HIF29-32
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
HIF29-32
Cisco Public
26
NIF4
NIF5
Slot 7
HIF25-28
NIF3
HIF17-20
Slot 6
HIF21-24
NIF1
NIF2
HIF9-12
Slot 4
NIF5
NIF6
HIF5-8
IOM Link Failure Scenario
Slot 1
HIF1-4
HIF1-4
Slot 2
NIF1
HIF5-8
Slot 3
NIF2
NIF3
Link Failure
NIF4
HIF5-8
HIF9-12
NIF2
HIF9-12
Slot 4
HIF13-16
Slot 5
HIF17-20
Slot 6
HIF21-24
HIF21-24
Slot 7
HIF25-28
HIF25-28
Slot 8
HIF29-32
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
HIF29-32
27
NIF3
NIF4
HIF13-16
HIF17-20
NIF1
IOM Link Failure Scenario
Slot 1
HIF1-4
HIF1-4
Slot 2
NIF1
HIF5-8
Slot 3
NIF2
NIF3
NIF4
HIF5-8
HIF9-12
NIF2
HIF9-12
Slot 4
HIF13-16
Slot 5
HIF17-20
Slot 6
HIF21-24
HIF21-24
Slot 7
HIF25-28
HIF25-28
Slot 8
HIF29-32
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
NIF3
NIF4
HIF13-16
HIF17-20
NIF1
HIF29-32
IOM Link Failure Scenario
Slot 1
HIF1-4
HIF1-4
Slot 2
NIF1
HIF5-8
HIF5-8
Slot 3
NIF2
NIF3
HIF9-12
NIF2
HIF9-12
Slot 4
HIF13-16
HIF13-16
Slot 5
HIF17-20
HIF17-20
Slot 6
HIF21-24
HIF21-24
Slot 7
HIF25-28
HIF25-28
Slot 8
HIF29-32
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
NIF1
HIF29-32
NIF3
NIF4
Port-channel Pinning
HIFs
VIC 1200/1300 adaptor with
DCE links in Port-Channel
2200-IOM
Pinned
to Po
Gen-1 adaptor with
single 10G link
HIF
UCSB-2-A(nxos)# show port-channel load-balance !
!
Port Channel Load-Balancing Configuration:!
System: source-dest-ip!
!
Port Channel Load-Balancing Addresses Used Per-Protocol:!
Non-IP: source-dest-mac!
IP: source-dest-ip source-dest-mac!
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
NIF
Cisco Public
30
Increased Bandwidth Access to Blades
4 links, Discrete -­‐ Today slot 1
slot 2
slot 3
slot 4
slot 5
slot 6
slot 7
slot 8
F
E
X
Fabric
Interconnect
8 links, Discrete slot 1
slot 2
slot 3
slot 4
slot 5
slot 6
slot 7
slot 8
F
E
X
Fabric
Interconnect
Up to 8 links, Port-­‐channel F
E
X
Fabric
Interconnect
• Available bandwidth per
blade – 10Gb
• Available bandwidth per
blade – 20Gb
• Available bandwidth per
blade – up to 160Gb
• Statically pinned to
individual fabric links
• Statically pinned to
individual fabric links
• Statically pinned to Portchannel
• Deterministic Path
• Deterministic Path
• Increased and shared
bandwidth
• Guaranteed 10Gb to
each blade
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
31
• Higher Availability
Server Connectivity
Cisco Virtual Interface Cards (VIC)
1st Gen (Palo)
2nd Gen (Sereno)
3rd Gen (Cruz)
• M81KR, P81E
• 1240, 1280, 12xx
• 1340, 1380
• 128 PCIe devices
• 256 PCIe Device
• Dual 8x PCIe Gen 3
• Dual 10Gb
• Dual 40Gb (4 x 10Gb)
• VXLAN & NVGRE
• 16x PCIe Gen1
• 16x PCIe Gen 2
• Native 40Gb Support
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
• RoCE
VIC 1280/1380
UCS 2208 IOM
UCS 2208 IOM
1280
Sereno
4 x 10G Ports
Side B
Side A
UCS 1200/1300 VIC
1380
Cruz
256 PCIe devices
Mezzanine Cards.
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
34
VIC 1240/1340 + Port Expander Card
Base option supports dual 2x10Gb
• Port Expander =
passive connector
device
1240
Sereno
• Port Expander fits in
Mezzanine slot
• mLOM vs Mezzanine.
Modular LOM (mLOM)
1340
Cruz
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
35
VIC 1240/1340 to IOM Connectivity
MLOM only
UCS 6248
UCS 6248
Fabric
Interconnects
16x SFP+
16x SFP+
Expansion Module
16x SFP+
16x SFP+
Expansion Module
IO Modules
2208XP
2208XP
Midplane
Adapter
1340 VIC
x8 Gen 3
Server Blade
CPU 0
§ Dual 2x10 Gb port-channel from VIC 1240/1340
to 2208 IO Modules
Empty
x8 Gen 3
CPU 1
QPI Link
B200 M3/M4
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
UCS Blade Chassis
Cisco Public
36
VIC 1240/1340 to IOM Connectivity
MLOM plus Port Expander
UCS 6248
UCS 6248
Fabric
Interconnects
16x SFP+
16x SFP+
IO Modules
Midplane
Adapter
16x SFP+
2208XP
16x SFP+
2208XP
Port Channel 2
1340 VIC
CPU 0
§ Port Expander Passive
§ Increase BW to 80Gbps
§ Dual 4x10Gbps Port-channel
Port Exp
x8 Gen 3
CPU 1
QPI Link
B200 M3/M4
BRKCOM-2003
Expansion Module
Port Channel 1
x8 Gen 3
Server Blade
Expansion Module
© 2015 Cisco and/or its affiliates. All rights reserved.
UCS Blade Chassis
Cisco Public
37
What Does The OS See?
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
38
VIC 1x40 & 1x80 to IOM Connectivity
UCS 6248
UCS 6248
Fabric
Interconnects
16x SFP+
16x SFP+
IO Modules
Expansion Module
16x SFP+
2208XP
16x SFP+
Expansion Module
2208XP
Midplane
Adapter
1340 VIC
x8 Gen 3
Server Blade
CPU 0
VIC1380
§ Adapter Redundancy
§ Split vNIC across adapters
§ 4 2x10 Gb Port-channels
x8 Gen 3
CPU 1
QPI Link
B200 M3/M4
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
UCS Blade Chassis
Cisco Public
39
No mixing of
12xx/13xx.
Full Width Blade to IOM Connectivity
MLOM, Port Expander, VIC1x80
UCS 6248
UCS 6248
Fabric
Interconnects
16x SFP+
16x SFP+
IO Modules
Midplane
2208XP
Port Channel 1
1340 VIC
Port Exp
x8 Gen 3
CPU
4x10
§ Total BW is 160G
§ Four 40G port-channels
VIC1380
x8 Gen 3
CPU
QPI Link
B260 M4
BRKCOM-2003
Expansion Module
Port Channel 2
x8 Gen 3
Server Blade
16x SFP+
2208XP
4x10
Adapter
16x SFP+
Expansion Module
© 2015 Cisco and/or its affiliates. All rights reserved.
UCS Blade Chassis
Cisco Public
40
Blade Module on the bottom is the “Master” and the Blade Module on the top is the “Slave.”
The server is controlled with UCS Manager (UCSM) version 2.2(2).
UCS
M4,
320Gbps
Aggregated BW
Figure 1 B460
Cisco UCS
B460 M4
Blade Server
Front View
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
UCS Mini: Fabric to Server Connectivity
§
Same server-side connectivity
as the 2204XP IOM
§
40G per half width blade
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Connectivity IOM to Adapter
2208 IOM • Implicit Port-channel
2208 IOM • Flows hashed across port-channel
UCSB-2-A# connect nxos!
UCSB-2-A(nxos)# show port-channel summary!
-------------------------------------------------------------------------------!
Side B
Side A
Group PortType
Protocol Member Ports!
Channel!
-------------------------------------------------------------------------------!
UCS 1200/1300 VIC
11
Po11(SD)
88
Po88(SD)
...!
1314 Po1314(SU)
1315 Po1315(SU)
UCSB-2-A(nxos)#!
vNIC1
VM
BRKCOM-2003
VM Flows 1.
10 Gb FTP traffic
2.
10 Gb UDP traffic
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
43
Eth
Eth
LACP
LACP
Eth1/11(D)!
Eth1/20(D)!
Eth
Eth
NONE
NONE
Eth1/1/5(P)!
Eth1/1/6(P)!
Virtual Interfaces
Abstracting the Logical Architecture
Physical
6200-A
Logical
6200-A
Switch
6200-A
vEth
1
vFC
1
vFC
1
vEth
1
Eth 1/1
IOM A
IOM A
10GE
A
10GE
A
ü
Dynamic, Rapid
Provisioning
ü
Location
Independence.
Cable
Adapter
vHBA
1
Blade
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
vNIC
1
Service Profile
(Server)
Cisco Public
45
vNIC
1
vHBA
1
(Server)
Physical Cable
Virtual Cable
(VN-Tag)
VN-TAG: Pre-Standard IEEE 802.1BR
FEX architecture
LAN
Switch
Application
Payload
TCP
IP
VN-TAG
Ethernet
Frame
VNTAG
Frame
FEX
VN-TAG Ethertype"
d" p"
l" r" ver"
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
destination virtual interface"
source virtual interface"
Fabric Extender Evolution – Virtual Interfaces
• VN-TAG/IEEE 802.1BR allows
cascading FEXs
LIF
• Cisco VIC is an extension of FEX
FEX
VIF
Adapter FEX
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
• VN-TAG associates the Logical
Interface (LIF) to a Virtual Interface
(VIF)
UCS Life of a Packet
1
Blade x/y
2
3
4
5
7
UCS IOM
UCS Fabric Interconnect
UCS IOM
Blade x/y
vNIC
vNIC
vNIC
6
VIC
Cruz
ASIC
1. Frame received
on VIC from OS
Woodside
ASIC
4. Ingress Carmel
performs forwarding
and queuing
© 2015 Cisco and/or its affiliates. All rights reserved.
Egress
Carmel
5. If required egress
queuing and flow
control
3. IOM forwards frame to
fabric uplink (NIF port)
2. VIC appends VNTag
and forwards frame to
IOM HIF port
BRKCOM-2003
Sunnyvale
Cross Bar
Ingress
Carmel
Cisco Public
6. Egress Carmel
appends destination
VNTag and forwards
frame on fabric link
Woodside
ASIC
VIC
Cruz
ASIC
vNIC
7. VNTag stripped by
VIC and frame
forwarded out on vNIC
VN-Tag at the VIC Adapter
UCSB-2-A# show service-profile circuit server 1/3!
Service Profile: miciesla/ciesla-cisco-live1!
Server: 1/3!
Ether!
Fabric ID: B!
Path ID: 2!
VIF
vNIC
Link State Oper State
---------- --------------- ----------- ---------1447 eth3
Up
Active
1449 eth5
Up
Active
1451 fc1
Up
Active
9643
Up
Active
Prot State
------------No Protection
No Protection
No Protection
No Protection
Prot Role
----------Unprotected
Unprotected
Unprotected
Unprotected
Admin Pin
---------0/0
0/0
0/0
0/0
Oper Pin
---------1/1
1/1
0/1
0/0
Transport!
---------!
Ether!
Ether!
Fc!
Ether!
UCSB-2-A# connect adapter 1/3/2!
adapter 1/3/2 # connect !
adapter 1/3/2 (top):1# attach-mcp !
adapter 1/3/2 (mcp):1# vnic!
------------------------------------------ --------- --------------------------!
v n i c
l i f
v i f
!
id name
type
h:bb:dd.f state lif state uif ucsm
idx vlan state !
--- -------------- ------- --------- ----- --- ----- --- ----- ----- ---- -----!
13 vnic_4
enet
1:8c:00.0 UP
2 UP
=>1 1449
144
1 UP
!
14 vnic_1
enet
0:85:00.0 UP
3 UP
=>1 1447
143
1 UP
!
15 vnic_2
enet
0:86:00.0 UP
4 UP
=>0 1448
64 511 UP
!
16 vnic_3
fc
0:87:00.0 UP
5 UP
=>1 1451
145 2000 UP
!
!
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
49
UCSB-2-B(nxos)# show vifs interface port-channel 1313!
!
Interface
MAX-VIFS VIFS!
-------------- -------- ---------------------------------------------------------!
Po1313
63
Veth1447, Veth1449, Veth9643,!
UCSB-2-B(nxos)# !
!
UCSB-2-B(nxos)# show port-channel summary interface po 1313!
--------------------------------------------------------------------------------!
Group PortType
Protocol Member Ports!
Channel!
--------------------------------------------------------------------------------!
1313 Po1313(SU) Eth
NONE
Eth1/1/6(P) !
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
50
Fabric Extender Evolution: VM-FEX
VN-TAG/IEEE 802.1BR*
• VMs assigned dedicated NIC on host PCIE bus
• No local vSwitch
• VMs get dedicated switch port on the FI
FEX
IEEE 802.1BR* VN-TAG/IEEE 802.1BR*
Hypervisor
Legacy
BRKCOM-2003
Adapter FEX
UCSB-2-A(nxos)# connect nxos !
UCSB-2-A(nxos)# show interface virtual summary vm-fex !
Veth
Bound
Port
Mac
VM
!
Interface Interface
Profile
Address
Name
!
--------------------------------------------------------------!
Veth32769 Po1281
VM-Data-2-511 00:25:b5:02:37:2e vmk0
!
Veth32770 Po1281
VM-Data-2-511 00:50:56:a1:d6:6d matao-vcenter5.1.eth0!
Veth32771 Po1281
VM-Data-2-511 00:50:56:a1:2d:3e matao-vcenter5.5.eth0!
Veth32772 Po1281
VM-Data-2-511 00:50:56:a1:1c:a6 matao-ubuntu.eth0!
Veth32773 Po1281
VM-Data-2-511 00:50:56:a1:f9:60 matao-centos6.eth0!
Veth32775 Po1281
VM-Data-2-511 00:50:56:a1:ca:a3 matao-win7.eth0!
VM-FEX
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Fabric Failover
End Host Mode (only)
SAN A
LAN
SAN B
UCS Fabric
Interconnects
Chassis
Fabric Extender
Adapter
vNIC
vNIC
vNIC
Adapter
vNIC
Fabric Extender
CiMC
Half Width Blade
BRKCOM-2003
CiMC
UCS-6200-A /chassis/server/adapter/host-eth-if # show vif!
!
VIF:!
ID
Fabric ID Transport Tag
Status
Oper State!
---------- --------- --------- ----- ----------- ----------!
1201 A
Ether
0 Allocated
Active!
1202 B
Ether
0 Allocated
Passive!
Half Width Blade
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
52
Fabric Forwarding - Ethernet
Ethernet Fabric Forwarding Mode of Operations
LAN
• Switch mode:
• End-host mode (EHM):
– FI acts like regular ethernet switch
– VLAN/Mac based forwarding
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
– No spanning-tree protocol (STP)
• Active/Active for all links & VLANs
– Policy based forwarding
Cisco Public
54
End Host Mode
LAN
Spanning Tree • Server to Server uses VLAN/MAC
forwarding
• vNICs are pinned to uplink interfaces
FI A vEth 3
vEth 1
VLAN 10
Fabric A MAC
Learning
MAC
Learning
• UCS presents as a bunch of hosts to
the upstream network.
• Simplified network – no STP.
L2
Switching
VNIC 0
VNIC 0
Server 2
Server 1
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
55
End Host Mode: Unicast Forwarding
• Server to Server is locally switched
LAN
Server 2
Uplink Ports
Deja-Vu
• Policies to prevent packet looping
1. No uplink to uplink forwarding
2. Déjà Vu check
3. RPF
RPF
FI
• No unknown unicast.
vEth 1
VLAN 10
VNIC 0
VNIC 0
Server 2
Server 1
BRKCOM-2003
– Silent VM?
vEth 3
© 2015 Cisco and/or its affiliates. All rights reserved.
• FI Mac Aging vs. Router ARP Timeout
Cisco Public
56
End Host Mode: Broadcast Forwarding
LAN
• Broadcast traffic for a VLAN is pinned to
one uplink port only
B
B
Broadcast
Listener
per VLAN
Uplink
Ports
• Broadcast Listener prevents duplicate
packets
FI
• Server to server broadcast traffic is
locally switched
vEth 1
vEth 3
• RPF and Déjà Vu check also applies for
broadcast traffic.
B
BRKCOM-2003
VNIC 0
VNIC 0
Server 2
Server 1
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
57
End Host Mode: Disjointed L2 Domains
UCSM by default assumes all uplinks are part of all VLANs DMZ 1 (vlans 10,20,30) DMZ 2 (vlans 40,50,60) Broadcast Link EHM 6200 A Cannot see DMZ 2 Broadcasts BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
EHM DMZ 1 Server DMZ 2 Server Cisco Public
6200 B BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Switch Mode
Root
LAN
• Fabric Interconnect behaves like a
normal L2 switch
• Rapid-STP+ to prevent loops
• Server vNIC traffic follows STP
forwarding states
MAC
Learning
vEth 3
vEth 1
• MAC address learning on both
uplinks and server links
VLAN 10
L2
Switching
• Mainly used for workarounds in
UCS release prior to 2.0.
VNIC 0
VNIC 0
Server 2
Server 1
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
60
Fabric Forwarding - Multicast
End Host Mode: Multicast Forwarding
LAN
• Multicast traffic flow is similar to
broadcast traffic
M
M
Broadcast
Listener
per VLAN
Uplink
Ports
• Egress multicast traffic sent out pinned
uplink interface.
FI
vEth 1
vEth 3
M
BRKCOM-2003
VNIC 0
VNIC 0
Server 2
Server 1
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
62
IGMP Querier?
• Three Options:
1. Upstream IGMP Querier / PIM Router
2. Fabric Interconnect IGMP Querier
3. IGMP Snooping disabled
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
UCS Multicast
IGMP Querier / PIM Router
IGMP Querier Upstream
1. IGMP Querier
LAN
Uplink
Ports
3. IGMP Report
Broadcast
Listener
per VLAN
FI
vEth 1
vEth 3
2. IGMP Report
VNIC 0
VNIC 0
Server 2
Server 1
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
UCSB-2-B(nxos)# show ip igmp snooping vlan 511!
IGMP Snooping information for vlan 511!
IGMP snooping enabled!
Optimised Multicast Flood (OMF) disabled!
IGMP querier present, address: 10.67.87.2, version: 3, i/f Eth1/1!
Querier interval: 125 secs!
Querier last member query interval: 1 secs!
Querier robustness: 2!
Switch-querier disabled!
IGMPv3 Explicit tracking enabled!
IGMPv2 Fast leave disabled!
IGMPv1/v2 Report suppression enabled!
IGMPv3 Report suppression disabled!
Link Local Groups suppression enabled!
Router port detection using PIM Hellos, IGMP Queries!
Number of router-ports: 1!
Number of groups: 0!
VLAN vPC function disabled!
Active ports:!
Eth1/1
Eth1/3 Veth1447
Veth32769!
Veth32770
Veth1449
Veth32771
Veth32772!
UCSB-2-B(nxos)# show ip igmp internal info global | grep "Vlan 511"!
Vlan 511: G-pinned if - curr: Eth1/1; prev: !
Cisco Public
UCS Multicast
Internal Querier
LAN
Uplink
Ports
Broadcast
Listener
per VLAN
FI
IGMP Querier
vEth 1
vEth 3
1. IGMP Querier
2. IGMP Report
BRKCOM-2003
VNIC 0
VNIC 0
Server 2
Server 1
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
UCSB-2-B(nxos)# show ip igmp snooping vlan 511!
IGMP Snooping information for vlan 511!
IGMP snooping enabled!
Optimised Multicast Flood (OMF) disabled!
IGMP querier present, address: 1.1.1.1, version: 3!
Querier interval: 125 secs!
Querier last member query interval: 1 secs!
Querier robustness: 2!
Switch-querier enabled, address 1.1.1.1, currently running!
IGMPv3 Explicit tracking enabled!
IGMPv2 Fast leave disabled!
IGMPv1/v2 Report suppression enabled!
IGMPv3 Report suppression disabled!
Link Local Groups suppression enabled!
Router port detection using PIM Hellos, IGMP Queries!
Number of router-ports: 1!
Number of groups: 0!
VLAN vPC function disabled!
Active ports:!
Eth1/1
Eth1/3 Veth1447
Veth32769!
Veth32770
Veth1449
Veth32771
Veth32772!
UCS Multicast Configuration (2.1+)
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
UCS & Microsoft Network Load Balancing (NLB)
• Unicast Mode
– Ethernet Switching mode only
– Nexus 1000v (no mac auto-static-learn)
• Multicast Mode
• IGMP Multicast
– Requires igmp querier
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Uplink Pinning
End Host Mode - Dynamic Pinning
• UCSM manages the vEth pinning to
the uplink
LAN
vlan10
• Pinned uplink must pass VLAN used
by vNIC
vlan20,30
FI A vEth 2
vlan10
vEth 3
vlan20
Pinning
vEth 1
Switching
vlan30
VNIC 0
VNIC 0
VNIC 0
Server 2
Server 3
Server 1
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
69
• UCSM periodically redistributes the
vEths
End Host Mode – Individual Uplinks
Dynamic Re-­‐pinning of failed uplinks FI-­‐A Sub-second re-pinning
vEth 3 Pinning
vEth 1 Switching
Fabric A L2
Switching
üGARP aided upstream convergence
üSub-second re-pinning
VNIC stays up
VNIC 0 MAC A
VNIC 0 BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
70
Server 2 vSwitch / N1K ESX HOST 1 VM 1
MAC B
VM 2
MAC C
End Host Mode – Port Channel Uplinks
No disruption
No GARPs
needed
FI-­‐A Sub-second convergence
vEth 3 vEth 1 Switching
Fabric A üMore Bandwidth per Uplink
üNo Server NIC disruption
üFewer GARPs needed
üFewer moving parts
RECOMMENDED BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Pinning
NIC stays up
VNIC 0 MAC A
VNIC 0 Cisco Public
71
Server 2 vSwitch / N1K ESX HOST 1 VM 1
MAC B
VM 2
MAC C
End Host Mode – Static Pinning (LAN Pin Group)
LAN
Administrator Pinning Definition
FI A vEth 2
vEth 3
vEth Interfaces
Uplink
vEth 1
Blue
vEth 2
Blue
vEth 3
Purple
Pinning
vEth 1
Switching
• Administer controls the vEth pinning
• Deterministic traffic flow
• No re-pinning with in the same FI
VNIC 0
VNIC 0
VNIC 0
Server 2
Server 3
Server 1
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
• Static and dynamic pinning can
co-exist
Cisco Public
72
Which uplink is the servers’ vEth pinned to?
UCSB-2-B(nxos)# show pinning border-interfaces !
--------------------+---------+----------------------------------------!
Border Interface
Status
SIFs
!
--------------------+---------+----------------------------------------!
Po122
Down
!
Eth1/1
Active
sup-eth2 Veth921 Veth926 Veth941 !
Veth952 Veth1188 Veth1190 Veth1416 !
Veth1417 Veth1420 Veth1421 Veth1445 !
Veth1447 Veth1449 Veth32769 Veth32770 !
Veth32771 Veth32772 Veth32773 Veth32775 !
Veth32776 Veth32777 Veth32778 Veth32779 !
Veth32780 Veth32781
!
Eth1/2
Down !
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
73
Fabric Forwarding - QoS
UCS Congestion Management
8 Classes: 1 FCoE, 1 best effort, 4 user-definable, 2 reserved for control
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
UCS QoS – Marking / Classification
UCSB-2-B(nxos)# show run interface vethernet 1453!
interface Vethernet1453!
description server 1/6, VNIC eth1!
untagged cos 5!
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Cisco Virtual Interface Card: QoS
FEX FEX Fabric A Cisco VIC 7 Strict Priority VIC Mgmt 10% 10GE 10GE TX 40% 2 1 RR 3 No Drop 10% 4 BRKCOM-2003
40% 10% 6 0 1 RR Limit TX 1GE vNIC 1 7 Bandwidth Scheduler 40% 5 Strict Priority TX Bandwidth Scheduler 0 Fabric B 2 3 10% 4 No Drop VIC Mgmt 40% 5 6 (8) COS Queues Limit TX 1GE vNIC 2 COS 1 vHBA 1 vNIC 3 COS 3 COS 4 © 2015 Cisco and/or its affiliates. All rights reserved.
vNIC 4 COS 5 Cisco Public
vNIC 5 vNIC 6 COS 1 vHBA 2 vNIC 7 vNIC 8 COS 3 COS 4 COS 5 58 Pause!
LAN
PFC or 802.3x Pause
UCS 6200 PFC Pause
Woodside
ASIC
UCS IOM
PFC
Side A
Adapter
BRKCOM-2003
Side B
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
ossless rubric will continue leveraging their QoS/CoS semantics to ensure reliability.
Priority Flow Control
below displays the differences in the format of the legacy PAUSE frame with that defined in
Qbb. Note how the PFC frame now has fields targeting different traffic classes.
Classical Ethernet Pause vs. Data Centre Ethernet PFC Pause
Priority Flow Control
Transmit Queues
One
Two
Three
Four
Five
Six
Seven
Eight
STOP
Ethernet Link
PAUSE
Receive Buffers
One
Two
Three
Four
Five
Six
Seven
Eight
• Enables lossless Fabrics for each class of service
•PAUSE sent per virtual lane when buffers limit exceeded
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
79
Eight
Virtual
Lanes
PFC Pause: What does it look like?
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
UCS QoS Identifying Congestion
UCSB-2-B(nxos)# show interface priority-flow-control
============================================================
Port
Mode Oper(VL bmap) RxPPP
TxPPP
============================================================
Ethernet1/1
Auto Off
0
0
Ethernet1/2
Auto Off
0
0
Ethernet1/3
Auto Off
39648
3597850
Ethernet1/4
Auto Off
46784
4069738
Ethernet1/5
Auto Off
0
0 !
fex-1# show platform software woodside loss !
+-------+-------------------------------------+------------+-+-----------------------------------+---------------------------------------+!
|
|
|
| |
|
|!
|
|
|
| |
| frm_to
|!
|
|
|Port Extra | |
+---------------------------------------|!
|
|
RMON
|
Drop
|S| SS Loss Counters
| COS
| XOFF |!
|
+------------+-----------+------------+------------|S|-----------+-----------+-----------+---------------------------------------|!
| Port | Tx Pause
| Rx Pause | Errors
| Counters
|x| RX SS
| Tx SS
| SS Total |0 |1 |2 |3 |4 |5 |6 |7 |0 |1 |!
+-------+------------+-----------+------------+------------+-+-----------+-----------+-----------+---+---+---+---+---+---+---+---+---+---+!
| 0- NI1|
0|
32|
0|
0|0|
0|
0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| !
+-------+-------------------------------------+------------+-+-----------+-----------+-----------+---+---+---+---+---+---+---+---+---+---+!
| 0-HI19|
26|
0|
0|
0|0|
0|
0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| !
+-------+-------------------------------------+------------+-+-----------+-----------+-----------+---+---+---+---+---+---+---+---+---+---+!
fex-1# !
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Fabric Forwarding - Storage
SAN “End Host” NPV Mode
N-Port Virtualisation Forwarding
FLOGI
FDISC
SAN A
SAN B
NPIV
NPIV
VSAN 20
N_Proxy
6200-A
vFC 1
vFC 3
• FI in NPV mode means:
N_Proxy
6200-B
vFC 2
F_Proxy
vFC 4
F_Proxy
N_Port
vHBA
vHBA
0
1
N_Port
vHBA
vHBA
0
1
Server 1
Server 2
VSAN 1
BRKCOM-2003
• FI proxies FC messages to
NPIV switch
F_Port
F_Port
VSAN 10
NPV
• vHBAs are pinned to SAN
uplinks
© 2015 Cisco and/or its affiliates. All rights reserved.
VSAN 1
Cisco Public
NPV
–
–
–
–
Uplinks connect to F port
No domain ID consumption
Multi-vendor interoperability
Zoning performed upstream
SAN “End Host” NPV Mode
N-Port Virtualisation Forwarding with MDS, Nexus 5000
SAN A
NPIV
F_ Port
Channel &
F_Port
VSAN
Trunk
1,2
SAN B
• Port channel support
– Increased Bandwidth
– Redundancy
NPIV
VSAN
1,2
• VSAN Trunking support
N_Proxy
6200-A
6200-B
vFC 1
vFC 3
vFC 2
vFC 4
F_Proxy
N_Port
vHBA
vHBA
0
1
vHBA
0
vHBA
1
Server 1
Server 2
VSAN 1
VSAN 2
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
UCSB-2-B(nxos)# show vsan!
vsan 1 information!
name:VSAN0001 state:active !
interoperability mode:default !
loadbalancing:src-id/dst-id/oxid
operational state:up !
!
SAN FC Switch Mode
Direct Attach FC & FCoE Storage to UCS
FC
FCoE
• UCS acts like an FC SAN switch
Optional
MDS
N_Port
• Local or Remote Zoning
SAN
VSAN 1
VSAN 2
MDS
TE_Port
F_Port
• Direct attached storage
6200-A FC Switch
vFC 1
vFC 3
6200-B FC Switch
vFC 2
vFC 4
F_Port
N_Port
vHBA
vHBA
0
1
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
vHBA
0
vHBA
1
Server 1
Server 2
VSAN 1
VSAN 2
Multi-Hop FCoE
FCoE STORAGE
• Supports MDS, N5K & N7K
• “Unified Uplink” port type
FCoE
• FI in FC ENM Mode
FCoE
MDS/N5K/N7K
MDS/N5K/N7K
– VNP port type
FCoE/Ethernet
Unified Uplinks
NPV/EHM
• FI in FC Switching Mode
UCS FI
– VE port type
UCS FI
UCS B-Series
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
FCoE/Ethernet
FCoE with Adapter FEX
VFC interface bound to 802.1BR / VN-Tag virtual Ethernet interface
vfcX
vfcX
Fabric Interconnects
SAN A
Can be FC or FCoE
vethX
vethX
Can be FC or FCoE
IOM
802.1BR / VN-Tag
IOM
vHBA A
Adapter
802.1BR / VN-Tag
vHBA
B
Binding
FC1
FC0
OS
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Used for QoS, MTU, FIP
SAN B
Cisco Public
UCSB-2-B(nxos)# show interface brief !
..!
---------------------------------------------------------!
Interface Vsan
Admin Admin
Status
Bind
!
Mode
Trunk
Info
!
Mode
!
---------------------------------------------------------!
vfc925
2000
F
on
trunking
Vethernet9117 !
vfc943
2000
F
on
trunking
Vethernet9135 !
vfc1192
2000
F
on
trunking
Vethernet9384 !
vfc1306
2000
F
on
trunking
Vethernet9498 !
vfc1322
2000
F
on
errDisabled Vethernet9514 !
vfc1424
2000
F
on
trunking
Vethernet9616 !
vfc1451
2000
F
on
trunking
Vethernet9643 !
Appliance Ports
Direct attach SCSI, NFS & CIFS storage
• Supported in Ethernet EHM Mode
Storage
• Works similar to Server interfaces
iSCSI
NFS
CIFS
Appliance Port
UCS FI
UCS FI
UCS B-Series
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Topologies
89
Recommended Topology for Upstream Connectivity
Access/Aggregation Layer
vPC/VSS
FI-B
FI-A
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
90
UCS VM Traffic Flow
• All VMs in same VLAN
• VM1 to VM2
L2 Switching
• VM1 to VM3
• VM1 to VM4
EHM
FI-A
VNIC 0
VM1
BRKCOM-2003
EHM
VNIC 1
FI-B
VNIC 0
VNIC 1
ESX HOST 1
ESX HOST 2
vSwitch / N1K
Mac Pinning
vSwitch / N1K
Mac Pinning
VM2
© 2015 Cisco and/or its affiliates. All rights reserved.
VM3
Cisco Public
91
VM4
UCS & ACI Fabric
APIC
vCenter
1.
FI sends CDP/LLDP to leaf, CDP
to ESXi blade.
2.
ESXi blade & leaf send discovery
data to vCenter & APIC
3.
APIC receives discovery data
from vCenter
4.
APIC downloads policy on all
leafs providing path to the ESXi
blade
CDP / LLDP
CDP
BRKCOM-2003
IOM
© 2015 Cisco and/or its affiliates. All rights reserved.
IOM
Cisco Public
New Features
93
VIC 1300 New Features
94
Native 40 Gigabit Ethernet
10 Gb Encoding
40 Gb Encoding = 4 x 10 Gb
40G Port
n
… 4
3
64 bits
2
1
64/66B Encoding
1 packet
Tx 1
5
1
Tx 2
6
2
Tx 3
…
3
Tx 4
4
66 bits
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
VIC 1300 VXLAN & NVGRE Offload
Hardware based TCP segmentation & checksum verification
• Overlay encap/decap performed by hypervisors:
– Increases CPU utilisation
– Decrease network throughput
• Inner & outer packet hardware processing:
– TCP segmentation
– TCP/UDP checksum
– IP checksum
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
VXLAN Performance Test
• 2 x ESXi 5.5 GA, VMware DVS
• UCS B200 M3, VIC 1380
• FI 6248, IOM 2208
• VM guest OS: RHEL6.4 64 bit, 1GB RAM, 1vCPU
• Unicast iPerf test between two VMs
– iperf –s, iperf –c 17.1.1.1, TCP window size 23.2 KB
• Locally switch path on FI
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
VXLAN Offload Performance - CPU
VXLAN stateless offloads disabled
Host 1
Host 2
VXLAN stateless offloads enabled
Host 1
Host 2
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
VXLAN Offload Performance - Throughput
VXLAN stateless offloads disabled
VXLAN stateless offloads enabled
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
VXLAN Offload Configuration
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
VIC 1300 RoCE Support
Remote Direct Memory Access over Converged Ethernet (RoCE)
• Access remote node’s memory w/o CPU interruption
• Lower latency, better CPU use
• "RoCE does for InfiniBand what FCoE did for Fibre Channel”*
* Scott Lowe - http://blog.scottlowe.org/2010/04/20/am-i-understanding-roce-correctly
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Protocol
Stack
RoCE vs
InfiniBand
Protocol Stack
RoCE Based
Applications written
over IB Transport Layer
Socket applications
RDMA applications
BRKCOM-2003
IB transport
TCP
L3
IB GRH
IPv4
IB
L1
IB
(S/D/Q)
© 2015 Cisco and/or its affiliates. All rights reserved.
© 2011 MELLANOX TECHNOLOGIES
ULP
L4
L2
Standard Ethernet
applications written over
Sockets API
Ethernet
XAUI
Cisco Public
XFI
SGMII
8
Packet Format
InfiniBand
RoCE vs InfiniBand Packet Format
LRH
(L2 Hdr)
GRH
L3 Hdr
BTH+
(L4 Hdr)
InfiniBand Payload
ICRC
VCRC
ICRC
FCS
RoCE
No
Changes
BRKCOM-2003
Eth L2
Header
GRH
L3 Hdr
BTH+
(L4 Hdr)
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
InfiniBand Payload
Microsoft SMB Direct (SMB over RMDA)
• Initial use case for RoCE is Microsoft
Windows 2012 SMB Direct 3.0
User
Kernel
Network w/
RDMA support
Network w/
RDMA support
NTFS
SCSI
Jose Barreto, Tech Ed, 2013, http://channel9.msdn.com/Events/TechEd/
NorthAmerica/2013/MDC-B335#fbid=
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
New Features in 2.2
105
IPv6 Management Support
• UCS FI management can be configured with IPv6 address
– IPv6 capable external services (i.e., NTP, SSH, TACACs, HTTP/HTTPs, etc)
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
106
Inband Management for CIMC
• Separate server management (CIMC) traffic from UCSM
– Designating servers’ CIMC into different groups
• Higher bandwidth
– 10G vs 1G
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
107
Faster and Better: Link Layer Enhancements
• Uplink ports only
• Faster link failure detection with UDLD
• LACP
– Fast timer fast (1 sec) support
– Enable suspend-individual link.
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
108
PVLAN Enhancements
• Community Support
• Promiscuous on Appliance Port
• PVLAN trunking on the vNIC (extend PVLAN to the virtual switches)
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
109
Netflow+
• Improve workload visibility
– Capacity planning
– Security
– Troubleshooting
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
110
Other Sessions
Thursday:
• BRKCOM-2601 - Hyper-Converged Computing
• BRKCOM-2640 - UCS C-Series Deployment Options, Best Practice and UCSM Integration
• BRKVIR-2044 - Multi-Hypervisor Networking - Compare and Contrast
• BRKCOM-3002 - UCS Performance Troubleshooting
Friday:
• BRKCOM-2017 - UCS Systems Management Deep Dive with UCS Foundational Software
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Q&A
Complete Your Online Session Evaluation
Give us your feedback and receive a
Cisco Live 2015 T-Shirt!
Complete your Overall Event Survey and 5 Session
Evaluations.
• Directly from your mobile device on the Cisco Live
Mobile App
• By visiting the Cisco Live Mobile Site
http://showcase.genie-connect.com/clmelbourne2015
• Visit any Cisco Live Internet Station located
throughout the venue
T-Shirts can be collected in the World of Solutions
on Friday 20 March 12:00pm - 2:00pm
BRKCOM-2003
© 2015 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Learn online with Cisco Live!
Visit us online after the conference for full
access to session videos and
presentations. www.CiscoLiveAPAC.com
Thank you.