Exploring Brocade ISL
Trunking
SOLUTIONware ----Rev1
Tech Note
Last modified: January 2002
This document provides an in-depth examination
of trunking and how trunking impacts the design,
implementation, and maintenance of a SAN.
Category:
SOLUTIONware Technical Note
Sub-category:
SAN Design
Key words:
ISL trunking, SAN Design
1
53-0000263-01
Table of Contents
Table of Contents..........................................................................................................................................................2
Introduction ..................................................................................................................................................................3
Trunking Behavior .......................................................................................................................................................3
Virtual Channels......................................................................................................................................................6
High Availability and Dynamic Bandwidth Allocation ...............................................................................................7
Adding An ISL .........................................................................................................................................................7
An ISL Failure or Removal ....................................................................................................................................8
Performance................................................................................................................................................................12
Monitoring Performance.......................................................................................................................................12
ISL Performance....................................................................................................................................................12
Trunking Maximizes Performance and Minimizes The SAN Management Effort .........................................13
How Does Trunking Impact SAN Design..................................................................................................................15
Tip: Place Trunking-Capable Switches Adjacent To Each other .....................................................................15
Tip: Place Trunking-Capable Switches In The Core .........................................................................................15
Tip: Leave Open Ports On The Quad..................................................................................................................16
Tip: If Two Trunkable Switches Are Connected By More than One ISL, Use Trunking...............................16
Trunking Requirements..............................................................................................................................................17
ISL Trunking Administration ....................................................................................................................................19
Summary .....................................................................................................................................................................21
Appendix A: Terms and Definitions..........................................................................................................................22
Appendix B: Helpful Trunking Commands..............................................................................................................24
Copyright.....................................................................................................................................................................25
Evaluation Form.........................................................................................................................................................26
Page 2 of 26
53-0000263-01
Introduction
This document provides an in-depth examination of trunking and how trunking impacts the
design, implementation, and maintenance of a SAN. Currently the SilkWorm 3800 and 12000
switches support trunking.
Trunking is a feature that enables traffic to be optimally shared across available inter-switch
links (ISLs) while preserving in-order delivery. A trunk group logically joins two, three, or four
ISLs into one logical ISL. Use of trunking can minimize or eliminate congestion in the SAN
because trunking optimizes ISL utilization. Additionally, the use of trunking minimizes the
effort of managing a SAN since ISLs are now managed as a group instead of individually.
Trunking can also increase availability. As long as at least one ISL link remains, I/O continues
if an ISL failure occurs -- albeit at a lower bandwidth. It is also possible to dynamically increase
bandwidth by adding ISLs to a trunk -- without impacting I/O -- to enable up to 8Gb/s of
bandwidth over a single logical link. For reference, a convenient list of trunking definitions and
terms is provided at the end of this document in Appendix A.
Trunking Behavior
Due to the implementation of some Fibre Channel protocol devices, frame traffic between a
source device and a destination device must be delivered in-order within an exchange. Out of
order and in order delivery is shown in Figure 1. The host is sending a SCSI sequence in the
order 0, 1, 2, 3. The in order delivery for the storage to receive this sequence is 0, 1, 2, 3 and any
other order, such as 2, 3, 0, 1 is considered out of order.
Figure 1 In order and Out of Order Delivery
In Order
Delivery
3210
SAN
3210
2301
Out Of
Order
Delivery
Page 3 of 26
53-0000263-01
This restriction forces the current Brocade frame routing method to fix a routing path within a
fabric. As a consequence, certain traffic patterns may become unevenly distributed, leaving some
paths congested and other paths underutilized. This situation is depicted in Figure 2. Hosts A
and B require an aggregate of 300 MB/s of bandwidth while host C requires 10 MB/s of
bandwidth (Note: The SilkWorm 3800 switch supports 2 Gb/s technology). Fabric Shortest
Path First (FSPF) establishes routes in a round robin fashion while preserving in-order delivery.
In Figure 2, hosts A & B, which require 150 MB/s each of bandwidth will share the left-hand
ISL, while host C will utilize the right-hand ISL. Note that the aggregate bandwidth required is
310 MB/s. Hosts A & B will experience congestion, as only 200 MB/s is available from the
utilized ISL and the bandwidth requirement is 300 MB/s. Host C will experience no congestion,
as there is sufficient bandwidth to support the 10 MB/s requirement.
The two ISLs provide an aggregate bandwidth of 400 MB/s. If the routing initialization
happened in a different order, there would not be any congestion. However, even if the route
assignments were different, say host A and host C on one ISL and host B on the other ISL, it is
still possible to encounter congestion due to load variations and fluctuations. For example, a
tape backup might create enough of a spike in demand to cause congestion on an ISL that would
otherwise be avoided if trunking were utilized.
It is possible to deterministically setup the routes of a fabric with static routes by using the
uRouteConfig command and prevent the scenario depicted in Figure 2 from happening;
however, doing so require some effort and doing so for a large SAN with numerous ISLs can
become a very complex task. With trunking it is not necessary to setup static routes or manage
individual ISLs to achieve optimal performance.
Page 4 of 26
53-0000263-01
Figure 2 Uneven distribution of traffic without trunking
host A
1 5 0 M B /s
host B
1 5 0 M B /s
host C
1 0 M B /s
A g g r e g a te r e q u ir e d b a n d w id th o f
h o s t A a n d h o s t B is 3 0 0 M B /s .
A v a ila b le b a n d w id th ( le ft- h a n d
IS L ) is 2 0 0 M B /s .
H o s t C o n ly r e q u ir e s 1 0 M B /s o f
b a n d w id th . A v a ila b le b a n d w id th
( r ig h t- h a n d IS L ) is 2 0 0 M B /s
B a n d w id th S u r p lu s = 1 9 0 M B /s
B a n d w id th D e fe c it = 1 0 0 M B /s
host A
host B
host C
Figure 3 depicts a similar scenario as shown in the Figure 2 configuration. Notice that all
variables are the same, except that trunking is utilized. The trunk is a logical aggregation of the
two 200 MB/s ISLs into one logical 400 MB/s trunk. This means that we have gone from an
unbalanced load situation with a deficit of 100 MB/s to a surplus bandwidth of 90MB/s.
Note that trunking allows the grouping of up to four 2 Gb/s ISLs into a maximum of one logical
8 Gb/s trunk. It is also possible to divide the four ISLs in a quad between two trunk groups, with
two 2-ISL trunks (4 Gb/s each) – each trunk connected to a different switch. As shown in
Figure 3, the trunk provides sufficient bandwidth for all traffic streams. Trunking is dynamic
and responds in real-time to performance issues.
Page 5 of 26
53-0000263-01
Figure 3 Performance and management benefits of trunking
A g g re g a te re q u ire d b a n d w id th o f h o s t A , h o s t B , & h o s t C
is 3 1 0 M B /s . A v a ila b le b a n d w id th is 4 0 0 M B /s .
B a n d w id th S u r p lu s = 9 0 M B /s
host A
1 5 0 M B /s
host B
1 5 0 M B /s
host C
1 0 M B /s
host A
host B
host C
host A
host B
host C
2 IS L T ru n k = 4 0 0 M B /s
Virtual Channels
Brocade Virtual Channels is a unique technology that logically partitions physical bandwidth and
prioritizes traffic within inter-switch links (ISLs). It is essential for enabling large fabrics by
insuring the reliable operation of the network, improving the efficiency and performance of
switch-to-switch communication, simplifying capacity planning and fabric design, and reducing
the total cost of SAN ownership.
Brocade Virtual Channels has individual circuits for eight different channels with four different
priority levels. Critical traffic for management communication and link control receive
dedicated channels and the highest priorities. Data traffic can be sent across multiple channels
with different priority levels.
Page 6 of 26
53-0000263-01
Brocade Virtual Channels is integral to Brocade ISL Trunking and is built into the switch ASIC,
which is a part of every Brocade switch product. The underlying technology uses an antistarvation algorithm that insures that all Brocade switches are non-blocking and it is used as the
foundation for Brocade Extended Fabrics to communicate over long distances. The virtual
channel technology logically partitions bandwidth within a single ISL and ISL Trunking
logically aggregates bandwidth across up to four different ISLs. Together, they provide
maximum network efficiency and they enable Brocade switches to scale cost effectively into
large high performance reliable storage area networks.
High Availability and Dynamic Bandwidth Allocation
When adding ISLs to an existing trunk, there is no disruption of I/O and I/O is immediately
spread across the existing ISLs and the new ISL. With trunking, the deletion or failure of an ISL
to an existing trunk can impact the I/O traversing that trunk.
Adding An ISL
Figure 4 depicts the configuration utilized to quantify the impact of an ISL addition to a trunk
Essentially, the hosts and storage communicate via a 4 Gb/s trunk, which consists of two 2 Gb/s
ISLs.
Figure 4 Configuration Utilized To Quantify Dynamic Bandwidth Allocation
CLARiiON
4500
Sun Hosts
I/O
3800
3800
3800
3800
JMR JBODs
y 4 x SilkWorm 3800 w/ v3.0.1 FabricOS
y 6 x Sun Hosts w/ Solaris 5.8
y Emulex PCI LP8000 (1 Gb/s) & LP9000 (2
Gb/s) w/ 4.2 drivers
y Clariion 4500 w/ 2 controllers
y 4 x JMR JBODs w/ (8) Seagate ST336605
FC disks
Page 7 of 26
53-0000263-01
In Figure 5, the portPerfShow command is executed while an I/O load is issued between hosts
and storage. Notice how the aggregate bandwidth utilized, which is approximately 400 MB/s,
now distributes over the additional ISL. Furthermore, the I/O consumption expands from
400 MB/s to 420 MB/s to utilize the additional bandwidth enabled by the ISL addition to the
trunk. No interruption to I/O was noticeable.
The test involved the use of the UNIX dd utility issuing a maximal I/O load from the six Sun
hosts to the Clariion and JMR storage. Ports 4 and 5 are the existing 4 Gb/s trunk. Port 6 was
then enabled. Once port 6 was enabled, it immediately joined the trunk and I/O was immediately
distributed.
Figure 5 I/O Immediately Spreads To The New ISL When An ISL Is Added To A trunk
2 ISL Trunk
0
1
2
3
4
5
6
3rd ISL
7
8
9
10
11
12
13
14
15
-------------------------------------------------------------------------------0
0
2.2k 2.0k
0
0
201m 201m
0
0
55m
91m
93m
55m
22m
22m
30m
30m
0
0
198m 199m
0
0
55m
91m
92m
55m
22m
21m
28m
29m
0
0
0
0
201m 201m
0
0
55m
91m
92m
55m
22m
22m
31m
31m
112
0
112
0
200m 201m
0
0
55m
91m
91m
55m
21m
22m
31m
31m
0
0
0
0
154m 154m 109m
0
55m
91m
94m
55m
30m
28m
30m
31m
0
0
0
0
140m 140m 140m
0
55m
91m
93m
55m
31m
31m
30m
30m
0
0
0
0
139m 139m 139m
0
55m
91m
93m
55m
30m
30m
30m
30m
0
0
0
0
140m 139m 139m
0
55m
92m
92m
55m
31m
30m
30m
31m
Point where 3rd ISL is added
An ISL Failure or Removal
When an ISL that is not a trunk master is removed or fails, there is minimal or no disruption of
I/O and the I/O will be distributed over the remaining ISLs that form the trunk. When an ISL
that is a trunk master is removed or fails, there will be a pause in I/O associated with that trunk
as the I/O is re-routed and distributed over the remaining ISLs that form the trunk.
It is possible that upon failure or removal of the ISL that some frames were in flight and that
these frames will be lost. When this happens, the behavior and impact to I/O is dependent upon
the host, application, storage, and HBA utilized. The recovery period for lost frames needs to be
accounted for before I/O can be expected to fully recover. If the master link happens to be an
upstream or downstream port (see Appendix A: Terms and Definitions for more detail), then the
time for a fabric reconfiguration also needs to be included.
Page 8 of 26
53-0000263-01
Figure 6 depicts the configuration utilized to quantify the impact of an ISL failure within a trunk.
The test involves performing I/O between the host and the storage with the dd UNIX utility.
Once the I/O begins, a trunk port is disabled (portDisable command) and the period until I/O
resumes is measured. Specifically the portPerfShow command is monitored for a pause or
decrease in I/O, at which point, the timer starts. Once I/O resumes, the timer stops.
When a trunking slave is disabled, there was no pause in I/O. The I/O rate decreases slightly and
after approximately three seconds, the I/O rate returns to steady state. The same test was
executed and instead of disabling a trunking slave, a trunking master was disabled. For this test,
the average pause in I/O was twenty seconds.
Figure 6 Configuration Utilized To Quantify The Impact Of An ISL Failure Within A
Trunk Upon I/O
CLARiiON
4500
Sun Hosts
I/O
3800
3800
3800
3800
JMR JBODs
y 4 x SilkWorm 3800 w/ v3.0.1 FabricOS
y 6 x Sun Hosts w/ Solaris 5.8
y Emulex PCI LP8000 (1 Gb/s) & LP9000
(2 Gb/s) w/ 4.2 drivers
y Clariion 4500 w/ 2 controllers
y 4 x JMR JBODs w/ (8) Seagate ST336605
FC disks
Page 9 of 26
53-0000263-01
Figure 7 displays a capture of portPerfShow data from a switch while I/O was running and a
trunk slave ISL was disabled (portDisable). Note that I/O stops for port 6, which was the port
that was disabled. The flow of data essentially continues with minimal impact; however, there is
a pause while ports 12 and 14 get rerouted across the remaining ISLs that form the trunk.
Figure 7 PortPerfShow Data During A Trunk Slave Failure
Trunk
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
-------------------------------------------------------------------------------0
0
0
0
0
145m 144m 145m
30m
91m
93m
93m
27m
27m
35m
35m
0
0
0
0
0
143m 142m 143m
30m
91m
93m
93m
25m
25m
34m
35m
0
0
0
0
0
144m 143m 144m
30m
91m
91m
93m
27m
27m
35m
35m
0
0
0
0
0
162m
0
162m
29m
89m
91m
91m
15m
27m
19m
40m
0
0
0
0
0
185m
0
185m
26m
88m
90m
90m
0
26m
0
48m
0
0
0
0
0
186m
0
186m
29m
87m
90m
89m
0
27m
0
48m
0
0
0
0
0
193m
0
192m
27m
87m
89m
89m 8.0m
25m
14m
42m
0
0
0
0
0
202m
0
202m
28m
87m
87m
89m
21m
19m
34m
36m
0
0
0
0
0
207m
0
206m
29m
89m
91m
90m
19m
20m
35m
35m
0
0
36
76
0
209m
0
209m
28m
91m
92m
91m
21m
21m
35m
35m
0
0
0
0
0
207m
0
207m
25m
91m
93m
92m
21m
20m
34m
35m
Slave failure
Page 10 of 26
53-0000263-01
The portPerfShow data shown in Figure 8 is taken during the failure of a trunk master ISL. Once
the trunk master (port 5) fails by disabling the trunk master, I/O pauses for approximately twenty
seconds before it resumes.
Figure 8 PortPerfShow Data During A Trunking Master Failure
Trunk
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
-------------------------------------------------------------------------------0
0
0
0
0
142m 141m 141m
30m
88m
90m
91m
27m
27m
35m
35m
0
0
0
0
0
143m 143m 143m
31m
89m
92m
90m
27m
27m
35m
35m
0
0
0
0
0
142m 141m 141m
30m
89m
92m
91m
25m
25m
34m
35m
0
0
0
0
0
0
20m
61m
63m
61m
18m
18m
21m
21m
680
40
936
220
0
0
824
0
0
0
0
0
0
0
0
0
0
0
0
0
133k 133k 275k 259k
0
0
0
0
94m
95m
1.1k
0
267k 267k 267k
0
0
0
0
0
0
0
0
0
84
0
0
0
0
0
data deleted to preserve space – essentially a 20 second pause when no I/O is active
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Master failure
1.4m 1.4m 2.8m 6.6k 6.6k 6.6k 1.1k 832
832
25m
25m
51m
13k
35m
35m
51m
18m
0
0
0
0
0
0
191m 190m
50m
91m
62m
85m
19m
17m
26m
28m
206m 206m
29m
90m
91m
91m
19m
19m
34m
36m
206m 206m
30m
88m
91m
91m
20m
20m
35m
35m
205m 205m
30m
88m
90m
90m
20m
20m
35m
35m
I/O Resumes
Page 11 of 26
13k
832
13k 1.6k 1.6k 1.6k 1.6k
53-0000263-01
Performance
The combination of 2 Gb/s technology and the capability of ISL trunking enable SilkWorm
switches to deliver impressive performance. With duplex 2 Gb/s ISLs, the potential bandwidth is
really 400 MB/s – one stream achieving 200 MB/s in one direction and another stream achieving
200 MB/s in the opposite direction. To realize the value of trunking, it is only necessary to have
an I/O load that is capable of saturating a single ISL. Once this saturation occurs, it is possible to
starve other devices – even if additional ISLs are available between the communicating devices.
As discussed earlier in this document and as is detailed later in this section, trunking eliminates
this problem.
Monitoring Performance
Port statistics may not be the same across all participating ISLs within a trunk. The level of
uniformity depends on the payload variations at the frame level. While optimal bandwidth
utilization is the goal of the Brocade ISL Trunking feature, traffic might not be distributed
exactly the same across all ISL links within a trunk.
ISL Performance
In Figure 9, if Host A writes to Storage 1 at 200 MB/s and Host B reads from Storage 2 at 200
MB/s it is possible to generate 400 MB/s of I/O on a single ISL. This performance capability
equates to 1600 MB/s (16Gb/s) of potential bandwidth for a four ISL trunk. Achieving such
performance in a real life situation is unlikely. To do so requires perfectly balanced I/O and
devices capable of generating sufficient levels of I/O.
Figure 9 2 Gb/s Duplex Performance
Storage 1
Host A
200 MB/s (write)
3800
3800
200 MB/s (read)
Storage 2
Host B
all device connections and ISL are 2 Gb/s
Page 12 of 26
53-0000263-01
In a lab setting it is possible to approach maximal ISL bandwidth. Given the configuration
shown in Figure 10 and using the dd UNIX utility, it was possible to generate over 350 MB/s on
a single ISL. The limiting variable in this test was that there was not enough storage to generate
a 400 MB/s I/O load.
Figure 10 Configuration Used To Achieve Maximal ISL Bandwidth
ISL
portPerfShow 1
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
-------------------------------------------------------------------------------0
0
0
0 355m
0
0
0
31m 43m 93m 94m 22m 22m 23m 23m
0
0
0
0 354m
0
0
0
31m 43m 94m 93m 22m 22m 23m 22m
0
0
0
0 349m
0
0
0
31m 43m 93m 93m 22m 22m 20m 20m
CLRiiON
4500
Sun Hosts
I/O
3800
3800
JMR JBODs
y 2 x SilkWorm 3800 w/ v3.0.1 FabricOS
y 6 x Sun Hosts w/ Solaris 5.8
y Emulex PCI LP8000 (1 Gb/s) & LP9000
(2 Gb/s) w/ 4.2 drivers
y Clariion 4500 w/ 2 controllers
y 4 x JMR JBODs w/ (8) Seagate ST336605
FC disks
Trunking Maximizes Performance and Minimizes The SAN Management Effort
In Figure 11, an identical read oriented I/O load is generated in the configuration with the dd
UNIX utility. The load is primarily read oriented, with a small percentage of writes. The load is
run with and without trunking. Ports 4, 5, and 6 are utilized for ISLs, while ports 8 through 15
are used for storage.
Due to the way in which FSPF established routes, the ISL on port 5 is operating at maximum
bandwidth (208 MB/s) in the test where no trunking is utilized. Should another device attach to
the switch and be assigned a route over port 5, that device would encounter congestion and it is
likely that intervention would be necessary to manage the performance conflict. It is also
possible that the devices using port 5 were experiencing congestion. In addition, notice how the
loads in the test run without trunking enabled are unbalanced.
Page 13 of 26
53-0000263-01
In the test run with trunking enabled, the load is evenly distributed and there exists surplus
bandwidth within the trunk to accommodate additional devices. It is not necessary to intervene
to manage the addition of devices since the resources are optimally utilized.
Figure 11 Fabric Performance With and Without Trunking
Without Trunking
3 ISLs
portPerfShow 1
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
-------------------------------------------------------------------------------0
0
0
0 101m 208m 89m
0
55m 76m 76m 90m 28m 27m 23m 23m
112
0 112
0 103m 208m 94m
0
55m 76m 76m 94m 28m 28m 23m 23m
0
0
0
0 100m 208m 93m
0
55m 76m 76m 93m 26m 28m 22m 22m
With Trunking
6 Gb/s trunk
portPerfShow 1
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
-------------------------------------------------------------------------------0
0
0
0 132m 132m 132m
0
55m 90m 93m 55m 27m 27m 23m 23m
0
0
0
0 132m 132m 131m
0
55m 90m 92m 55m 27m 27m 23m 23m
0
0
0
0 131m 132m 131m
0
55m 91m 93m 55m 26m 27m 22m 22m
CLARiiON
4500
Sun Hosts
I/O
3800
3800
JMR JBODs
y 2 x SilkWorm 3800 w/ v3.0.1 FabricOS
y 6 x Sun Hosts w/ Solaris 5.8
y Emulex PCI LP8000 (1 Gb/s) & LP9000
(2 Gb/s) w/ 4.2 drivers
y Clariion 4500 w/ 2 controllers
y 4 x JMR JBODs w/ (8) Seagate ST336605
FC disks
Page 14 of 26
53-0000263-01
How Does Trunking Impact SAN Design
Trunking optimizes the utilization of ISLs and reduces the SAN administration effort. These
benefits enhance the utility of ISLs and so enhance the utility of designs that make use of ISLs.
Given two SANs of equivalent functionality, the SAN that is easier to manage has the advantage.
Instead of monitoring multiple ISLs, a single trunk is now monitored.
The high performance capabilities of a trunking-capable switch also makes these types of
switches ideal for placement in the Core of a Core/Edge fabric (refer to the Brocade SAN Design
Guide, publication number 53-0000231-xx for a detailed discussion regarding SAN Design).
Because the trunk is efficient, it is less likely that congestion will be experienced and it is
possible that fewer ISL are required in the fabric topology, yielding additional ports for attaching
SAN devices.
Tip: Place Trunking-Capable Switches Adjacent To Each other
When designing a SAN with trunking-capable switches or introducing trunkable switches to an
existing SAN, it is important to place these switches adjacent to each other when working with
any topology, including Core/Edge, Mesh, Cascade, and Ring topologies. Doing so enables the
ability to trunk.
Tip: Place Trunking-Capable Switches In The Core
When working with a Core/Edge topology, it is recommended to place the trunkable switches in
the Core. As new trunking-capable switches are added to a Core/Edge topology, it will allow the
added switches to connect with trunking-capable switches. Migrating existing SilkWorm
switches not capable of trunking from the core to the edge and inserting trunking-capable
switches into the core is an effective strategy, as shown in Figure 12. Then as new trunking
switches are added, these switches can harness the power of trunking. This strategy also
preserves the investment in switches that are not capable of trunking.
Page 15 of 26
53-0000263-01
Figure 12 Place trunking-capable switches in the core and redeploy the former core to the edge
Tip: Leave Open Ports On The Quad
When connecting two switches in a fabric with one, two or three ISLs, consider leaving open the
other ports on the quad for future trunk growth and fill in the remaining open ports with SAN
devices first.
Tip: If Two Trunkable Switches Are Connected By More than One ISL, Use Trunking
When connecting two switches with two or more ISLs consider utilizing trunking to achieve
optimal performance and minimize the management effort.
Page 16 of 26
53-0000263-01
Trunking Requirements
For trunking to be enabled several requirements must first be met. If a switch is licensed and
meets these requirements, it is only necessary to connect up the ISLs to form a trunk. The
following requirements are necessary to trunk ISLs:
1. The switch platform must support trunking. Current switches that support trunking are
the SilkWorm 3800, and 12000. Trunking is not supported on the SilkWorm 2000 series
or SilkWorm 1000 series (Fabric OS 1.x or 2.x) switches. It is possible, supported, and
encouraged to connect switches that are not capable of trunking to switches that are
capable of trunking.
2. The two switches that are being trunked together must be licensed for trunking. To check
whether or not a switch is licensed for trunking, use the licenseShow command.
3. The ports that form a trunk must reside in the same contiguous four-port groups, as
shown in Figure 14. For example: 0-3, 4-7, 8-11, 12-15. Note, that the SilkWorm 3800
and 12000 quads are color coded to delineate a quad. It is supported to have two trunk
groups per quad, meaning that two 2-ISL trunks (4 Gb/s each) are resident on the same
quad and each trunk then connects to a different switch. The ports on both switches
being trunked must be configured as E_Ports and these ports must run at 2Gb/s. Note
that trunking is enabled by default in v3.x and v4.x Fabric OS if there exists a trunking
license. The default state of 2 Gb/s switch ports is to auto-negotiate, so nothing is
required to enable 2 Gb/s connectivity as the default is to connect at 2 Gb/s if the other
device is 2 Gb/s capable. Also the default state for SilkWorm 2000, 3000, and 12000
series switch ports is to automatically configure as an E-port when attached to another
Brocade switch. If there is some question as to a port’s suitability for trunking, use the
command portCfgShow to verify the port configuration.
4. The difference in cable length between all ports in a trunking group must be less be less
than 400 meters, and should be kept to 30 meters or less within a trunk to ensure optimal
performance and bandwidth utilization.
Page 17 of 26
53-0000263-01
Figure 13 Port groupings that form a Quad For SilkWorm 12000
Quads
Each SilkWorm 12000 switch assembly blade consists of 16-ports,
grouped into four quads
Figure 14 Port groupings that form a Quad For SilkWorm 3800
Quads
0 1 2 3
4 5 6 7
Page 18 of 26
8 9 10 11
12 13 14 15
53-0000263-01
ISL Trunking Administration
Once an ISL Trunking license has been purchased and installed ISL Trunking can be
administered either through telnet commands or through Brocade Web Tools. Combined
together, both methods provide excellent ease of use and an appropriate level of control. For
convenience, a list of helpful trunking commands has been included as a reference in Appendix
B of this document.
Brocade Web Tools provides an intuitive graphical interface to enable and review ISL Trunking
settings. The ISL Trunking screens are accessed through a web browser using the Admin View
of Web Tools. Figure 15 shows the Trunk Information Tab that lists the trunk groups in the
switch, the Master Ports for the trunk groups, and the Member Ports for the trunk groups.
Figure 16 shows the Port Setting Tab that displays which ports in the switch have ISL Trunking
enabled. All ports have ISL Trunking enabled by default. ISL Trunking can be enabled or
disabled by simply clicking the check box.
Figure 15 Web Tools Trunk Information Tab
Page 19 of 26
53-0000263-01
Figure 16 Web Tools Port Settings Tab Shows Which Ports Have Trunking Enabled
Page 20 of 26
53-0000263-01
Summary
A trunk logically joins two, three, or four ISLs into one logical ISL with a capacity of up to
8 Gb/s while preserving in-order delivery. In supported products, Brocade 2 Gb/s switches also
have the ability to create up to two trunk groups per quad.
Use of trunking can minimize or eliminate congestion in the SAN because trunking optimizes
ISL utilization. Since ISLs are now managed as a group instead of individually, the use of
trunking minimizes the effort of managing a SAN. Trunking also increases high availability.
Virtual Channels are integral to Brocade ISL Trunking and are essential for enabling large
fabrics by insuring the reliable operation of the network, and improving the efficiency and
performance of switch-to-switch communication.
To trunk, it is necessary to use switches that are licensed for trunking and support trunking, such
as the SilkWorm 3800 and 12000. Ports that form a trunk must reside in a quad, run at 2 Gb/s,
and be cabled together with cables that ideally differ in length by less than 30 meters. It is
possible to dynamically increase bandwidth in a trunk and not impact ongoing I/O by simply
adding an ISL to the trunk. The failure of an ISL within a trunk will result in some impact to
I/O; however, as long as one ISL remains communication between the switches will continue.
An ISL is capable of supporting up to 400 MB/s when operating full duplex. A four ISL trunk is
potentially capable of 1600 MB/s when operating full duplex.
When designing and implementing SANs, place trunking capable switches adjacent to each other
and in the core of a core/edge fabric. Fill up the open ports of a quad last so as to leave open
these ports for trunk expansion. If two switches are connected by more than one ISL, consider
the use of trunking to achieve optimal performance and minimize than SAN management effort.
Page 21 of 26
53-0000263-01
Appendix A: Terms and Definitions
Several terms, software functionalities, and definitions relative to trunking and ISLs are defined
and explained below.
•
Deskew Counter: This value is a metric used to identify the differences in cabling
distances between trunking links within a trunking group. This metric is used to ensure
in-order delivery across trunking links within the same trunking group. The higher the
deskew counter value, the longer the cable lengths. If using the same cable lengths to
connect two switches, expect to see the deskew counters for each ISL in the trunk to have
very similar values.
•
FSPF: Fabric Shortest Path First protocol. The FSPF protocol was developed by
Brocade and subsequently adopted by the Fibre Channel standards community for
allowing switches to discover the fabric topology and route frames correctly. It is the
industry standard routing protocol for Fibre Channel networks.
•
Trunking Group: Trunking uses a simple algorithm to optimally distribute frames
across a set of available paths that link two adjacent switches. Such a set of links is called
trunking group (see Figure 17). It is possible to have two trunk groups per quad. This
means having two 2-isl trunks, with each trunk connected to a different switch.
•
Trunking Ports: Ports that belong to a trunking group are called trunking ports (see
Figure 17).
•
Trunking Links: The components that comprise a trunk are termed trunking links. A
link consists of the two trunking ports and the connecting components: SFP and cable. A
trunking link is analogous to an ISL.
•
Trunking Master: Only one of the trunking ports is used to set up all routing paths for
the entire trunking group. This port is called trunking master port (see Figure 17). Note
that if a master trunking link goes offline or fails, the remaining slave links will also go
offline and regroup with a new master if there are at least two remaining active trunking
ports remaining. The failure of any master port impacts the traffic that traverses this
trunk since a regrouping of the trunks slaves must occur and because some in-flight data
may be lost.
•
Trunking Slave: Non-master trunking ports from a trunking group are termed trunking
slaves. The failure of any slave port has minimal impact on the trunk or the traffic the
traverses this trunk.
•
Upstream/Downstream Designator: The terms upstream and downstream designate
that the particular position of a switch in reference to the principal switch in the fabric.
These paths are used in the process for assigning switch domain IDs. In Figure 17, notice
that switch “edge1” is the principal switch and all “stream” designators are downstream
of switch edge1. For switch core1, the path to the principal switch is upstream through
port 0. The principal switch has no upstream ports.
Page 22 of 26
53-0000263-01
Figure 17 Trunking and ISL Elements
Principal
edge1
edge2
edge3
edge4
core1
edge5
edge6
edge7
1
2
3
4
core2
edge1:admin> switchshow
switchName: edge1
switchType: 2.4
switchState: Online
switchRole: Principal
switchDomain: 2
switchId:
fffc02
switchW wn:
10:00:00:60:69:10:8d:fd
switchBeacon: OFF
port 0: sw Online
E-Port 10:00:00:60:69:50:03:de "core1" (downstream)
port 1: sw Online
E-Port 10:00:00:60:69:50:03:a2 "core2" (downstream)
port 2: sw No_Light
port 3: cu No_Light
port 4: sw No_Light
port 5: sw No_Light
port 6: -- No_Module
port 7: -- No_Module
port 8: id No_Light
port 9: -- No_Module
port 10: id No_Light
port 11: -- No_Module
port 12: id No_Light
port 13: sw No_Light
port 14: -- No_Module
port 15: sw No_Light
core1:admin> switchshow
switchName: core1
switchType: 9.1
switchState: Online
switchRole: Subordinate
switchDomain: 8
switchId:
fffc08
switchW wn:
10:00:00:60:69:50:03:de
switchBeacon: OFF
port 0: id 1G Online
E-Port 10:00:00:60:69:10:8d:fd "edge1" (upstream)
port 1: id 1G Online
E-Port 10:00:00:60:69:11:f9:f7 "edge2"
port 2: id 1G Online
E-Port 10:00:00:60:69:10:9b:52 "edge3"
port 3: id 1G Online
E-Port 10:00:00:60:69:12:f9:8c "edge4"
port 4: id 1G Online
E-Port 10:00:00:60:69:12:f9:8c "edge4"
port 5: id 1G Online
E-Port 10:00:00:60:69:10:9b:5b "edge5"
port 6: id 2G No_Light
port 7: id 2G No_Light
port 8: id 2G Online
E-Port 10:00:00:60:69:50:03:b3 "edge7" (Trunk master)
port 9: id 2G Online
E-Port (Trunk port, master is port #8)
port 10: id 2G Online
E-Port (Trunk port, master is port #8)
port 11: id 2G No_Light
port 12: id 2G Online
E-Port 10:00:00:60:69:50:02:d0 "edge6" (Trunk master)
port 13: id 2G Online
E-Port (Trunk port, master is port #12)
port 14: id 2G Online
E-Port (Trunk port, master is port #12)
port 15: id 2G Online
E-Port (Trunk port, master is port #12)
Page 23 of 26
3800
2800
Trunk Master
}
}
Trunking Group
Trunking Group
ISL
ISL Trunk
ISL Trunk
ISL Trunk
53-0000263-01
Appendix B: Helpful Trunking Commands
An annotated list of trunking related commands follow to provide a quick reference. For detail
on a particular command, please reference the manuals or “help” page.
trunkShow: Use this to display detailed trunking information. Helpful for identifying deskew
constant. This command is new in V3.0 Fabric OS.
switchShow: Use this command to check status of switch ports and trunking. This command is
available in all versions of Fabric OS.
portCfgTrunkport: Use this command to individually configure a port for trunking. This
command is new in V3.0 Fabric OS.
switchCfgTrunk: Use this command to enable/disable an entire switch for trunking. This
command is new in V3.0 Fabric OS. Trunking is enabled by default in the V3.x and V4.x Fabric
OS releases.
islShow: Use this command to display ISL status. Identifies aggregate trunk bandwidth. This
command is new in V3.0 Fabric OS.
trunkDebug: Use this command to assist with troubleshooting a trunk. Supply two local port
numbers. This command is new in V3.0 Fabric OS.
portCfgShow: Use this command to view the configuration status of all ports. An improperly
configured port cannot join a trunk group. For example, a port could be configured as a nonE_Port, which would prevent that port from joining into the trunk group. This command is new
in V3.0 Fabric OS.
Page 24 of 26
53-0000263-01
Copyright
IMPORTANT NOTICE
This document is the property of Brocade. It is intended solely as an aid for installing and
configuring Storage Area Networks constructed with Brocade switches. This document does not
provide a warranty to any Brocade software, equipment, or service, nor does it imply product
availability. Brocade is not responsible for the use of this document and does not guarantee the
results of its use. Brocade does not warrant or guarantee that anyone will be able to recreate or
achieve the results described in this document. The installation and configuration described in
this document made use of third party software and hardware. Brocade does not make any
warranties or guarantees concerning such third party software and hardware.
2002, Brocade Communications Systems, Incorporated.
ALL RIGHTS RESERVED. Part number: 53-0000263-01
Brocade, SilkWorm, SilkWorm Express, and the Brocade logo are trademarks or registered
trademarks of Brocade Communications Systems, Inc., in the United States and/or in other
countries.
All other brands, products, or service names are or may be trademarks or service marks of, and
are used to identify, products or services of their respective owners.
NOTICE: THIS DOCUMENT IS FOR INFORMATIONAL PURPOSES ONLY AND DOES NOT SET
FORTH ANY WARRANTY, EXPRESS OR IMPLIED, CONCERNING ANY EQUIPMENT, EQUIPMENT
FEATURE, OR SERVICE OFFERED OR TO BE OFFERED BY BROCADE. BROCADE RESERVES THE
RIGHT TO MAKE CHANGES TO THIS DOCUMENT AT ANY TIME, WITHOUT NOTICE, AND
ASSUMES NO RESPONSIBILITY FOR ITS USE. THIS INFORMATIONAL DOCUMENT DESCRIBES
FEATURES THAT MAY NOT BE CURRENTLY AVAILABLE. CONTACT A BROCADE SALES OFFICE
FOR INFORMATION ON FEATURE AND PRODUCT AVAILABILITY.
Export of technical data contained in this document may require an export license from the
United States Government.
Brocade Communications Systems, Incorporated
1745 Technology Drive
San Jose, CA 95110
Page 25 of 26
53-0000263-01
Evaluation Form
Your feedback is valued by Brocade. In particular, we are interested in understanding how the
SOLUTIONware report helped make a difference in a SAN task, problem, or decision. Please
FAX this form back to: FAX (408) 392-5200.
SOLUTIONware Title: Exploring Brocade ISL Trunking
Document number: 53-0000263-01
Review:
What other subjects would you like to see SOLUTIONware reports on?
Please rate your overall satisfaction: ( )very satisfied ( )satisfied ( )not satisfied
Please identify yourself as belonging to one of these groups:
( ) End-User Customer
( ) Fabric Integrator
( ) Master Reseller
( ) Partner
( ) Brocade Employee
( ) Brocade OEM
( )Other
Page 26 of 26