Ethernet connectivity and options
5.3
Unit objectives
After completing this unit, you should be able to:
• List the BladeCenter components required for Ethernet connectivity
• Name the management components supporting Ethernet switch modules
• Select the Ethernet I/O switch module to meet a specific requirement
• List the components required to provide additional Ethernet connectivity to a blade
server
• Identify the method used to provide blade server Ethernet connection redundancy
2
BladeCenter chassis I/O overview
• BladeCenter chassis I/O overview
• Internal chassis traffic flow
• Switch I/O module management connectivity
• BNT (Nortel) Ethernet switch I/O modules
• Server connectivity module
• Pass-thru module
• High speed I/O modules overview
• Ethernet expansion cards
3
BladeCenter chassis overview: Switching path
BladeCenter Chassis Physical Components
4
Chassis overview: Gigabit Ethernet routing
SERDES
Ethernet
SW Module 1
SW Module 3
1………..14
1………..14
Line 1
Midplane (Upper Section)
Line 3
Network
Interface
Expansion
Card
SERDES
Ethernet
Processor blade
#1
Line 4
Line 2
Midplane (Lower Section)
1………..14
1………..14
SW Module 2
SW Module 4
5
Chassis overview: Ethernet port assignments (1 of 2)
Top ESM (Bay 1 )
1
2
3
4
5
6
7
8
9
10 11 12 13 14
Mgt 1
ESM
External
uplinks
Mgt 2
Eth 1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
B
S
B
S
B
S
B
S
B
S
B
S
B
S
B
S
B
S
1
2
3
4
5
6
7
8
9
B
S
1
0
B
S
1
1
B
S
1
2
B
S
1
3
B
S
1
4
2
2
2
2
2
2
2
2
2
2
2
2
2
2
MM 1
Uplink
Eth 0
MM 1
MM 2
Eth 0
MM 2
Uplink
Eth 1
1
2
3
4
5
Bottom ESM (Bay 2 )
6
7
8
9
10 11 12 13 14
Mgt 1
Mgt 2
ESM
External
uplinks
BladeCenter Chassis
6
Chassis overview: Ethernet port assignments (2 of 2)
Top ESM (Bay 1 )
1
2
3
4
5
6
8 9 10 11 12 13
ESM
External
uplinks
Mgt 1
Eth 1
1
1
1
1
1
1
B
S
B
S
B
S
B
S
B
S
B
S
1
2
3
4
5
6
2
2
2
2
2
2
Eth 0
MM 1
Uplink
MM 1
BladeCenter S Chassis
7
Internal chassis traffic flow
• BladeCenter chassis Ethernet overview
• Internal chassis traffic flow
• Switch I/O module management connectivity
• BNT (Nortel) Ethernet switch I/O modules
• Server connectivity module
• Pass-Thru module
• High speed I/O modules overview
• Ethernet expansion cards
8
Ethernet connectivity: Layer 2 traffic flow
Hidden internal filter that blocks traffic flow between the External ports and Mgmt Mod ports.
Prevents STP loops between ESM uplinks and MM uplinks (as well as between ESMs)
Upstream connections
Upstream connections
Red = Ethernet
External ports
MM1
Management
Module
Ports
CPU
MM2
14 Internal Blade ports
Ethernet
Switch
Module
(ESM)
BladeCenter Chassis
9
Traffic flow: NIC teaming and trunk failover
If the switch fails in such a way that
the link toward the Blade server goes
down, or NIC fails, Blade server can
sense this and redirect traffic out the
other NIC toward the bottom switch.
NIC Teaming can take care of this
without the need for trunk failover.
123456
123456
x
If failure anywhere on the link toward
the upstream switch, the NIC on
blade server does not know about the
failure and may
continue to send traffic toward the top
switch, which will discard the traffic.
The Trunk Failover feature
addresses this issue.
ESM
X
ESM
VLAN X
X
Teamed
Active/Standby
NIC
1
NIC
2
Logical NIC
Interface
Blade Server
10
Traffic flow: Trunk failover example 1
• If Po1 goes down, trunk failover
takes over and shuts down internal
downstream defined port(s)
– This alerts NIC Teaming to an upstream
failure at which point Teaming switches
to the other path out of the server
– This example shows a single VLAN to
Teamed NICs, It is possible to also carry
multiple VLANs to the Teamed NICs
123456
VLAN X Must
•Po2
be carried on
all uplinks
from ESM
123456
•Po1
VLAN X
•Po2
Must be carried
between ESMs
via an external path
•Link state group1
Upstream
•Po1
•Po1
ESM
•Link state group1
Upstream
ESM
VLAN X
•Link state group
Downstream
Teamed
Active/Standby
•Link state group
Downstream
•NIC
Teaming
NIC
1
NIC
2
Logical NIC
Interface
Blade Server
11
Traffic flow: Trunk failover example 2
• If only one port channel goes down,
spanning tree allows backup path to
other port channel
• If both port channels go down, trunk
failover takes over and shuts down
internal downstream defined port(s)
– This alerts NIC Teaming to an
upstream failure at which point it
switches to the other CIGESM
– This example shows a single VLAN to
Teamed NICs, It is possible to also
carry multiple VLANs to the Teamed
NICs
123456
VLAN X Must
•Po2
be carried on
all uplinks
from ESM
•Po3
123456
•Po1
•Po2
•Link state group1
Upstream
•Po3
•Link state group1
Upstream
•Po1
•Po2
•Po1
ESM
•Po2
ESM
VLAN X
•Link state
group1
Downstream
Teamed
Active/Standby
•Link state
group1
Downstream
•NIC
Teaming
NIC
1
NIC
2
Logical NIC
Interface
Blade Server
12
Switch I/O module management connectivity
• BladeCenter chassis Ethernet overview
• Internal chassis traffic flow
• Switch I/O module management connectivity
• BNT (Nortel) Ethernet switch I/O modules
• Server connectivity module
• Pass-Thru module
• High speed I/O modules overview
• Ethernet expansion cards
13
Management connectivity for the ESM
Management
Workstation
Legend
Ethernet
Ethernet path
2
I2C interface
1
I2C path
Serial
Serial path
3
Routed Production
Network
Management
Network
3
4
5
2
Web Interface
1A
Management Module
Ethernet
Switch
Module
1B
4
Blade
Server
14
BNT (Nortel) Ethernet switch I/O modules
• BladeCenter chassis Ethernet overview
• Internal chassis traffic flow
• Switch I/O module management connectivity
• BNT (Nortel) Ethernet switch I/O modules
• Server connectivity module
• Pass-Thru module
• High speed I/O modules overview
• Ethernet expansion cards
15
BNT GbESM switch I/O modules: Layer 2-3
• Layer 2/3 copper GbESM
– 14 internal 1 Gb FDX
connections
– 6 external 1 Gb FDX
connections
– 2 10/100 Mb FDX
management ports
BNT Copper and Fibre Layer 2-3 Switch I/O Modules
• Layer 2/3 Fibre GbESM
– 14 internal 1 Gb FDX
connections
– 6 external 1 Gb FDX fibre (SX)
connections
– 2 10/100 FDX management
ports
16
Control
Gigabit
Ethernet
Accelerator
Data
Application Switch
connector
BNT L2/3 GbESM: Simplified block diagram
Management
Processor
Memory
6 external 10/100/1000 Mbps Copper Ports
(Auto-negotiation enabled)
Or
6 external 10/100/1000 Mbps Copper Ports
(Auto-negotiation enabled)
And
14 internal 1000 Mbps server links
2 internal 100 Mbps management links
17
Blade 14
Blade 13
Blade 12
Blade 3
Blade 2
Blade 1
BNT L2/3 GbESM: Network interfaces
Default VLAN
Management VLAN
VLAN 1
VLAN 4095
IP Interface 128
Int1
(1)
Int2
(2)
Int3
(3)
Int12
(12)
Int13
(13)
Int14
(14)
Customer
Network
Ext1
(17)
Ext2
(18)
Ext5
(21)
Mgt1
(15)
Management
Module 1
Mgt2
(16)
Management
Module 2
Ext6
(22)
18
BNT Nortel layer 2-7 GbESM
• The Layer 2-7 GbESM features:
• Support for up to 300,000
simultaneous Layer 2-7 sessions
with full wire-speed packet
forwarding for all connections
• Link aggregation on the four
external ports
• Layer 4 load balancing for
computing optimization
• Virtual router redundancy to
facilitate high application
availability in the event of
application, operating system or
hardware failures
Layer 5-7
Layer 4
Layer 2/3
19
BNT (Nortel) Layer 2-7 GbESM: Additional function
20
BNT 1/10 Gb uplink Ethernet switch module
•
Combined 1 Gb and 10 Gb external ports
•
Three slots for 10 Gb Ethernet SFP+ modules
•
Six 1 Gb RJ45 ports
• Fourteen internal 1 Gb ports
21
Server connectivity module
• BladeCenter chassis Ethernet overview
• Internal chassis traffic flow
• Switch I/O module management connectivity
• BNT (Nortel) Ethernet switch I/O modules
• Server connectivity module
• Pass-Thru module
• High speed I/O modules overview
• Expansion cards
22
Server connectivity module: Product highlights
• Low-cost Layer 2 device
• Simplified user interface to enable
server administrator control of
device
• Ideal for SMB customers and
applications
• Function summary
– Basic Layer 2 connectivity within
chassis
– “Non-networking”, server admin setup
and control
– Simple, GUI/CLI interface
• Create blade groups (simple VLANs)
• Create external port groups (auto
Etherchannel)
• Enable trunk failover (redundancy option)
• Set external port speed
• Assign tagged VLANs to blade groups
23
Pass-Thru module
•
BladeCenter chassis Ethernet overview
•
Internal chassis traffic flow
•
Switch I/O module management connectivity
•
BNT (Nortel) Ethernet switch I/O modules
•
Server connectivity module
•
Pass-Thru module
•
High speed I/O modules overview
•
Expansion cards
24
Intelligent copper pass-thru module: Product highlights
• 14 external RJ-45 copper ports for making connections to the
external network.
– The external ports can operate at 100 or 1000 Mbps.
• 14 internal bi-directional ports, connected to each of the blade
servers
• One-to-one mapping of internal Ethernet ports on the blade to
external copper RJ-45
• GUI based operator interface
25
High speed I/O modules overview
• BladeCenter chassis Ethernet overview
• Internal chassis traffic flow
• Switch I/O module management connectivity
• BNT (Nortel) Ethernet switch I/O modules
• Server connectivity module
• Pass-Thru module
•
High speed I/O modules overview
• Expansion cards
26
High speed I/O modules: Overview
• BNT 10Gb Ethernet Switch I/O Module
– BladeCenter H and HT High Speed Module
– 10 External Ports
– 14 Internal Ports
• BNT 10Gb Ethernet Pass-Thru I/O Module
– BladeCenter H and HT High Speed Module
– 14 External Ports
– 14 Internal Ports
BNT 10Gb Ethernet Switch I/O Module
BNT 10Gb Ethernet Pass-Thru I/O Module
27
Expansion cards
• BladeCenter chassis Ethernet overview
• Internal chassis traffic flow
• Switch I/O module management connectivity
• BNT (Nortel) Ethernet switch I/O modules
• Server connectivity module
• Pass-Thru module
• High speed I/O modules overview
• Expansion cards
28
Ethernet expansion cards: Combination I/O vertical
(CIOv)
•
•
•
•
PCI Express host interface
Broadcom BCM5709S communication module
BladeCenter Open Fabric Manager (BOFM) support
Connection to 1000BASE-X environments using BladeCenter Ethernet switches
IBM HS22 Blade Server with Ethernet CIOv card installed
CIOv Ethernet Expansion Card
29
Ethernet expansion cards: Combination form factor
vertical (CFFv)
• Based on Broadcom 5704 chipset
• Two 1 Gb ports per blade server
IBM HS22 Blade Server with Ethernet CFFv card installed
Ethernet CFFv Expansion Card
30
Ethernet expansion cards: QLogic iSCSI
• Dual ports
• TCP/IP Offload Engine
– TC/IP stack implemented in firmware
• Provides iSCSI diskless boot capability
QLogic iSCSI Management Utility
QLogic iSCSI StFF Expansion Card
31
Ethernet expansion cards: High speed overview
• High Speed Form horizontal (CFFh) expansion cards
– Interface to Horizontally mounted I/O modules
– BladeCenter H and HT chassis
• PCI-E bus connections
QLogic 2-Port 10Gb Converged Network
Adapter CFFh
Broadcom 10Gb 4-Port CFFh card
IBM BladeCenter 2/4-Port CFFh card
NetXen 10Gb 2-Port CFFh card
32
Key words
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Device Manager
Failover
Fibre Ethernet
Graphical user interface (GUI)
Local Area Network (LAN)
Layer 2 – 7
Media Access Control (MAC) Address
Power On Self Test (POST)
Spanning Tree
Serializer/deserializer (SERDES)
Server Connectivity Module (SCM)
Open Shortest Path First (OSPF)
Uplink
Virtual Local Area Network (VLAN) tagging
Virtual Router Redundancy Protocol (VRRP)
Trunking
33
Checkpoint (1 of 2)
1. True/False: In order to provide Ethernet connectivity, each blade server
requires an Ethernet expansion card.
2. A customer requires policy management capability in the application through
Transport layers of the OSI model. Select the switch module appropriate for
these requirements.
a. BNT 10Gb Pass-thru Module
b. BNT L2/3 Copper GbESM
c. IBM Server Connectivity Module
d. BNT L2-7 GbESM
3. True/False: The Ethernet switch I/O modules can be managed through the
AMM management network, or through the switch I/O module external ports
34
Checkpoint solutions (1 of 2)
1. True/False: In order to provide Ethernet connectivity, each blade server requires
an Ethernet expansion card.
Answer: False
2. A customer requires policy management capability in the application through
Transport layers of the OSI model. Select the switch module appropriate for
these requirements.
a. BNT 10Gb Pass-thru Module
b. BNT L2/3 Copper GbESM
c. IBM Server Connectivity Module
d. BNT L2-7 GbESM
Answer: d
3. True/False: The Ethernet switch I/O modules can be managed through the AMM
management network, or through the switch I/O module external ports
Answer: True
35
Checkpoint (2 of 2)
4. In order to support 4 Ethernet connections from a Blade server, which of the
following is correct?
a. An expansion blade is required
b. Add an Ethernet expansion card and switch module in Bay 2
c. Add an Ethernet expansion card and switch modules in Bays 3 and 4
d. Add an IBM Server connectivity module
5. True/False: NIC Teaming can be used to provide redundancy, utilizing the paths
from the blade server onboard NICs through the midplane to Ethernet switch I/O
modules in Bays 1 and 2 of the chassis.
36
Checkpoint solutions (2 of 2)
4. In order to support 4 Ethernet connections from a blade server, which of the
following is correct?
a. An expansion blade is required
b. Add an Ethernet expansion card and switch module in Bay 2
c. Add an Ethernet expansion card and switch module in Bays 3 and 4
d. Add an IBM Server connectivity module
Answer: c
5. True/False: NIC Teaming can be used to provide redundancy, utilizing the paths
from the blade server onboard NICs through the midplane to Ethernet switch I/O
modules in Bays 1 and 2 of the chassis.
Answer: True
37
Unit summary
Having completing this unit, you should be able to:
• List the BladeCenter components required for Ethernet connectivity
• Name the management components supporting Ethernet switch modules
• Select the Ethernet I/O switch module to meet a specific requirement
• List the components required to provide additional Ethernet connectivity to a blade
server
• Identify the method used to provide blade server Ethernet connection redundancy
38