Alex Osuna
Piet de Jonge
Redpaper
IBM System Storage N series
MetroCluster Planning Guide
Introduction
The purpose of this IBM® Redpaper is to present a guide for planning the
upgrade of an Active/Active configuration to a MetroCluster. This document is
intended to be a supplement to the standard Active/Active Configuration Guide,
IBM System Storage N series Data ONTAP 7.2 Active/Active Configuration
Guide, GC26-7964. See “References” on page 13. This IBM Redpaper is not a
step-by-step guide to actually performing the upgrade.
Figure 1 on page 2 shows a MetroCluster.
© Copyright IBM Corp. 2006. All rights reserved.
ibm.com/redbooks
1
Primary Site
Remote Site
Servers
Servers
LAN
IBM N5500
IBM N5500
Dark
Fibre
FC Switches
FC Switches
DWDM
Switch
Vol X
DWDM
Switch
Vol Y Minor
Vol X Minor
Vol Y
Cluster Interconnect
Figure 1 MetroCluster
You must consider a number of factors in planning the upgrade of an
active/active configuration to a MetroCluster. We discuss these factors in the
following sections.
Intended audience
This IBM Redpaper is intended for storage administrators or architects who are
already familiar with Data ONTAP Administration, active/active configurations,
and MetroCluster and are considering deployments for production environments.
Distance considerations
In order to determine the upgrade components necessary, you need to
determine the estimated distance between nodes. If the distance between the
nodes is 500 meters or less, then you can use a stretch MetroCluster
configuration with OM3 cabling. If the distance is greater than 500 meters, then
2
IBM System Storage N series MetroCluster Planning Guide
you must choose a fabric-attached MetroCluster, which makes Fibre Channel
switches necessary.
Storage requirements
In analyzing the additional storage requirements necessary, it is important that
you remember that Advanced Technology Attachment (ATA) expansion units are
not currently supported on a fabric-attached MetroCluster. Typically in a
MetroCluster configuration, each expansion unit on a storage controller has a
mirror on its partner. This includes the expansion unit containing the root volume.
Not all expansion units on the local node need to be mirrored (other than the
expansion unit containing the root volume). However, any data contained on an
expansion unit that is not mirrored on its partner will obviously not be available in
a failover situation. Figure 2 shows a standard active/active configuration on a
N5000 series with each node containing two expansion units.
Figure 2 Standard active/active configuration
In this configuration, both loops are hooked up to the same Fibre Channel (FC)
Port on each filer: local loop to Port 0a and partner loop to Port 0b. This is not
strictly necessary but has the advantage of more easy administration, because
the shelves and disks now have the same ID on each node. If you convert this
configuration to a stretch MetroCluster configuration, then you need four
additional shelves, two for each node. Figure 3 on page 4 shows the converted
stretch MetroCluster. Notice each node now has two additional shelves on the
IBM System Storage N series MetroCluster Planning Guide
3
opposite side. These shelves contain the mirrored aggregates of the node's
partner.
As we stated before, ATA shelves are supported in a stretch MetroCluster. Mixed
ATA and FC configurations are allowed, provided the following requirements are
met:
򐂰 No intermixing of FC and ATA shelves on the same loop.
򐂰 Mirror shelves must be of the same type as their parents.
Figure 3 Stretch MetroCluster
4
IBM System Storage N series MetroCluster Planning Guide
In a fabric-attached MetroCluster configuration, the additional storage
requirements remain the same other than, as stated before, ATA drive shelves
are not supported. See Figure 4.
Figure 4 Fabric-attached MetroCluster
Cluster interconnect
Connectivity of the nonvolatile random access memory (NVRAM) NVRAM5 or
NVRAM6 cards is:
򐂰 For a stretch MetroCluster configuration, the cluster interconnect embedded
on the NVRAM card will continue to be used. A 4xIB Multi-Fiber Push-On
(MPO) optical cluster interconnect cable is required in the stretch
configuration.
򐂰 For a fabric-attached MetroCluster configuration, you can no longer use the
cluster interconnect embedded on the NVRAM card. You must install a
separate Fibre Channel/virtual interface (FC/VI) card. Consult the appropriate
hardware and service guide for the correct card and slot placement.
IBM System Storage N series MetroCluster Planning Guide
5
Important: As we have stated, in a stretch MetroCluster, as well as in a
standard active/active configuration, the connection between the two nodes is
established using connectors on the NVRAM cards. For the N5000-series,
these cards must be inserted in expansion slot 3 in order to be successful. It is
impossible to create a working active/active configuration or stretch
MetroCluster without your removing the NVRAM card from expansion slot 1
and reinserting it into slot 3. In fact, the most important difference between a
single node of the N5x00-A20 and a (standalone) N5x00-A10 is the placement
of the NVRAM card.
Alternatively, in a fabric-attached MetroCluster, the NVRAM card of the N5x00
units should be repositioned to slot 1.
For the IBM N7000 series, the NVRAM card remains in slot 1 in all
configurations.
Licensing
For either MetroCluster configuration, the following licenses are required on both
nodes of the MetroCluster:
򐂰 cluster
򐂰 cluster_remote:
– Provides “cf –d.”
– Provides the ability for an administrator to declare a site-specific disaster
and have one node takeover its partner’s identity without a quorum of
disks.
– Root volumes of both filers must be synchronously mirrored.
– Only synchronously mirrored aggregates are available during a site
disaster.
– Requires administrator intervention as a safety precaution against a “split
brain” scenario (cf forcetakeover –d).
Note: Site-specific disasters are not the same as a normal Cluster
Failover.
򐂰 syncmirror_local
6
IBM System Storage N series MetroCluster Planning Guide
Cable considerations
In upgrading an active/active configuration to a MetroCluster, additional and
different cables will be necessary. The guidelines are discussed next.
Overview
A stretched MetroCluster (see Figure 3 on page 4 for a diagram of a stretched
MetroCluster) can support a maximum 500 meter distance with OM3 cable. A
fabric-attached MetroCluster through the use of Fibre Channel switches further
extends this distance to 100 Km. This extended distance capability gives you
much greater flexibility in the physical location of the clustered nodes while
maintaining the high availability benefits of Clustered Failover.
The maximum permissible distance is a function of a combination of factors
including speed, number of connections, cable type, and interconnection type.
To deploy Clustered Failover with extended distance, you must adhere to the
following requirements:
1. For cable lengths greater than 30 meters, customers must purchase the
cables directly from cable vendors.
2. In order to achieve the maximum 500 meter distance between clustered
nodes, the 4xIB Multi-Fiber Push-On (MPO) optical cluster interconnect cable
must be a direct point-to-point connection with no intermediate device
between the clustered nodes (such as a patch panel).
3. The maximum distance between MetroCluster nodes will be reduced when
cluster interconnect is routed through an intermediate passive device (such
as a patch panel) due to degradation in the optical signal quality. The actual
distance reduction will vary. The only practical way to establish the maximum
supported distance is to test the desired configuration in an actual client
environment.
4. Do not mix cable types. The signal loss due to splice will probably result in a
total attenuation greater than specifications allow.
5. The minimum number of extended distance cables required is three: one for
cluster interconnect and one for each remote loop from each clustered filer.
Cluster interconnect
N5000 and N7000 systems have a copper interface for the cluster interconnect.
Customers have the option of ordering a 2 m, 5 m, or 10 m copper cable (the 5 m
copper cable is the default). The integrated NVRAM/cluster adapter supports the
IBM System Storage N series MetroCluster Planning Guide
7
4xInfiniBand standard. For MetroCluster at distances exceeding 10 m, fiber
interconnect is necessary.
Clients, who want to run either stretch MetroCluster or fabric-attached
MetroCluster, need to run fiber optic cable between the clustered systems.
Additional Cu-to-fiber converters are necessary when you use the fiber cables.
Each clustered pair requires four Cu-to-fiber converters. The feature code is
FC 1042 (description: Cu-to-fiber converter).
Figure 5 shows you an actual Cu-to-fiber converter.
Figure 5 Cu-Fiber converter
IBM provides the choice of either 5 m or 30 m fiber optic 4xInfiniBand cable
between the cluster systems. Each clustered pair requires two fiber optic cables.
The feature code for the 5 m optical cable is FC 1040. The feature code for the
30 m optical cable is FC 1041. IBM does not offer optical cables greater than 30
m (up to 500 m). You must order them directly from an IBM approved vendor.
Note: Cluster nodes that connect through a patch panel might need different
cables depending on the connector type required by the patch panel.
Customers need to buy four optical cables (of the same type) directly from
another vendor.
Disk interconnect
The same distance considerations apply to the disk cables. The length of B-loop
cables connecting filers to storage should match the length of the cluster
interconnect.
The minimum number of disk cables required is two (one for each remote loop on
each clustered filer). Each additional loop requires at least one additional long
cable. The number of disk cables doubles for a stretch MetroCluster connected
through patch panels or in a fabric-attached MetroCluster configuration. Some of
the long cables connect to the “A” side of the shelf. Also, multipathing is
supported on the stretch cluster, so you need long cables for both A and B
connections.
8
IBM System Storage N series MetroCluster Planning Guide
Cabling distance calculations
Use Table 1 to determine if the desired cable length is within the supported
maximum specification. This table is provided for guidance to determine the
distances but the best way to determine the best cable type is by testing this in
the environment (because there are many variables such as types of cables,
panels, and so forth).
For example, a client needs to run a FC cable over approximately 260 m (850 ft.)
and wants to run at 2 Gbps. How do we determine the maximum distance that
can be allowed for cabling between nodes?
Table 1 Cable selection chart
Cable
type
Fiber
core
type
Mode
Wave
length
Maximum
distance
(m)
Attenuation
(Db/Km)
Maximum
channel
attenuation
Splice
lossa
Connector
pair loss
OM1
62.5/
125
um
Multi
850
275
3.20
3.01
0.3
0.75
OM2
50/
125
um
Multi
850
550
3.00
3.25
0.3
0.75
OM3
50/
125
um
Multi
850
550
3.00
3.50
0.3
0.75
OS1a
9/125
um
Single
1310
2000
0.40
7.80
0.3
0.75
OM1
62.5/
125
um
Multi
850
275
3.20
2.10
0.3
0.75
OM2
50/
125
um
Multi
850
550a b
3.00
2.62
0.3
0.75
OM3
50/
125
um
Multi
850
550
3.00
3.25
0.3
0.75
OS1a
9/125
um
Single
1310
2000
0.40
7.80
0.3
0.75
1Gbps
2Gbps
IBM System Storage N series MetroCluster Planning Guide
9
Cable
type
Fiber
core
type
Mode
Wave
length
Maximum
distance
(m)
Attenuation
(Db/Km)
Maximum
channel
attenuation
Splice
lossa
Connector
pair loss
OM1
62.5/
125
um
Multi
850
70
3.20
1.78
0.3
0.75
OM2
50/
125
um
Multi
850
150
3.00
2.06
0.3
0.75
OM3
50/
125
um
Multi
850
150
3.00
3.00
0.3
0.75
OS1a
9/125
um
Single
1310
500
0.40
7.80
0.3
0.75
4Gbps
IB 1X 250MB/sec
OM1
62.5/
125
um
Multi
850
125
3.50
1.94
0.3
0.75
OM2
50/
125
um
Multi
850
250
3.50
2.38
0.3
0.75
OM3
50/
125
um
Multi
850
500
3.50
3.25
0.3
0.75
a. OS1 Cabling is not currently supported.
b. According to the maximum channel attenuation (2.62 dB) and an attenuation of 3.00 dB/km, the
maximum distance for this cable type is 406 m, so be careful with longer distances (up to 550 m)
with OM2 cable at 2 Gbps.
Table 1 on page 9 summarizes data related to optical cabling for data
communications that is available in documents published by various standards
organizations. We focus on data that is relevant to fiber deployments supported
on IBM N series systems.
To determine whether the desired cable run length is within the supported
maximum specification:
1. Determine the needed transfer rate based on the type of shelf you use.
2. Find out what fiber type is installed for the system.
3. Determine the number of connectors in the path between nodes.
10
IBM System Storage N series MetroCluster Planning Guide
Refer to Table 1 on page 9 and locate the maximum supported operating
distance and compare this value to the distance for the actual intended
application and verify that the actual does not exceed the supported maximum.
Consider the 260 m distance for the above client. Let us assume:
򐂰 Desired transfer rate: 2 Gbps
򐂰 Fiber core type: 50/125 OM2 Multimode cabling
򐂰 Number of connector pairs: two
Table 1 on page 9 shows that 260 m would be within the operating maximum
distance for all Fibre Channel transfer rates. This also assumes that the fiber
connection is point-to-point with only source and destination connections and no
patch panels or splices.
If the parameters remain the same, except that you include patch panels, the
installation might look like Figure 6.
Figure 6 Patch panel cabling
The table shows the distance of 260 m is within the distance limits. But you must
also take into account the potential optical signal loss added by the patch panels.
Referring to Table 1 on page 9, the total channel attenuation must be no more
than 2.60 dB.
Calculate the potential channel attenuation based on the per component
specification allowance.
From Table 2 on page 12, we find component allowances of:
򐂰 0.75 dB loss per mated connector pair
򐂰 3.0 dB loss per kilometer for the cable
򐂰 0.3 dB loss per splice
Note: Splices are unlikely to be a normal occurrence for most installations.
IBM System Storage N series MetroCluster Planning Guide
11
Table 2 Total channel attenuation
Total channel attenuation (TCA)
= 3.0 dB x (channel distance in
meters)/1000 m + .75 B x (# connector
pairs) + .3 dB x (# splices)
= 3.0 dB x 260 m/1000 m + .75 dB x 4
(source, destination, two patch panels) +
.3 dB x 0 (no splices)
TCA
= .78 dB + 3 dB + 0 dB
TCA
= 3.78 dB
Margin
= 2.60 dB - 3.78 dB = -1.18 dB
This indicates that by introducing two patch panels, the amount of potential
signal loss can exceed the limit shown in the specification by 1.18 dB. One of the
following must then occur:
򐂰 Remove the patch panels.
򐂰 Change the cabling type.
Another possible solution is to have the channel attenuation measured by a
qualified fiber installer and verified that it is within the supported specification
limits.
This naturally implies that by physically measuring a channel's attenuation and
finding it is within the limits of the specification for the fiber in use, you can
achieve a longer distance than the specification's maximum.
This decision is left up to the implementer, because the various standards take
no official stance in support for distances beyond the stated maximums, other
than to note the possibility of longer operating distances through the use of better
than worst-case performing components.
Fibre Channel small form-factor pluggables (SFPs)
There are four types of small form-factor pluggables (SFPs) associated with
fabric-attached MetroCluster configuration. They are:
Short wavelength (SWL) laser - Short wavelength laser transceivers based on
850 nm lasers are designed to transmit short distances. This is the most
common type of media and is the default on the Brocade 200E.
Long wavelength (LWL) laser - Long wavelength laser transceivers can be
based on 1310 nm lasers. Use them for long distance native FC links. Generally,
we used these media types with single-mode fiber cable.
12
IBM System Storage N series MetroCluster Planning Guide
Extended long wavelength (ELWM) laser - Extended long wavelength laser
transceivers can be based on 1550 nm lasers. Use them to run native Fibre
Channel connections over even greater distance than LWL media can support.
Generally, these media types use single mode fiber cable.
Dense wavelength division multiplexing (DWDM) - Dense wavelength division
multiplexing works by combining and transmitting multiple signals simultaneously
at different wavelengths on the same fiber. In effect, one fiber is transformed into
multiple virtual fibers.
The SFP type required is a function of the distance and the interconnect
technology that you use. Table 3 summarizes the supported types and
specifications.
Table 3 Supported interconnect technology
SFP type
Maximum
distance
Speed
IBM feature
code
Outside vendor and part
number
SWL
500 ma
4 Gbpsb
FC 2010
N/A
LWL
10 Kmc
2 Gbpsd
N/A
Finisar/FTLF1319P1xTL
Kme
f
N/A
Finisar/FTLF1519P1xCL
2 Gbpsg
N/A
Finisar/FWLF-1631-xx
ELWL
80
DWDM
500 Km
2 Gbps
a. Using 50/125 multimode cable. 300 m when using 62.5/125 cable.
b. 4 Gbps speed is not supported yet. New 4 Gbps LWL and ELWL SFPs are
being qualified.
c. Using 9/125 single mode cable.
d. 4 Gbps speed is not supported yet. New 4 Gbps LWL and ELWL SFPs are
being qualified.
e. Using 9/125 single mode cable.
f. 4 Gbps speed is not supported yet. New 4 Gbps LWL and ELWL SFPs are
being qualified.
g. 4 Gbps speed is not supported yet. New 4 Gbps LWL and ELWL SFPs are
being qualified.
References
Use these publications for reference:
򐂰 IBM System Storage N series Data ONTAP 7.2 Active/Active Configuration
Guide, GC26-7964
򐂰 IBM System Storage N series Data ONTAP 7.1.1 Cluster Installation and
Administration Guide, GC26-7790
IBM System Storage N series MetroCluster Planning Guide
13
Appendixes
Installation worksheets
Table 4 Installation worksheet
Local node (primary)
Name
IP Address
Fibre Channel
Switches:
Name
IP Address
Domain ID
Slot # ____
Switch 1 port #
Switch 2 port #
Slot # ____
Switch 1 port #
Switch 2 port #
Expansion unit #
Switch 1 port #
Switch 2 port #
Port #
Switch 1 port #
Switch 2 port #
1
2
Fibre Channel HBA 1
Port A (0a for N5000)
Port B (0b for N5000)
Fibre Channel HBA 2
Port A (0c for N5000)
Port B (0d for N5000)
Expansion unit
Pool 0 (Local) A IN
Pool 0 (Local) B IN
Pool 1 (Remote) A IN
Pool 1 (Remote) B IN
Cluster adapter
A
B
Inter-switch link
14
IBM System Storage N series MetroCluster Planning Guide
Table 5 Installation worksheet
Remote node (secondary)
Name
IP Address
Fibre Channel
Switches:
Name
IP Address
Domain ID
Slot # ____
Switch 3 port #
Switch 4 port #
Slot # ____
Switch 3 port #
Switch 4 port #
Disk shelf #
Switch 3 port #
Switch 4 port #
Port #
Switch 3 port #
Switch 4 port #
3
4
Fibre Channel HBA 1
Port A
Port B
Fibre Channel HBA 2
Port A
Port B
Disk shelves
Pool 0 (Local) A IN
Pool 0 (Local) B IN
Pool 1 (Remote) A IN
Pool 1 (Remote) B IN
Cluster adapter
A
B
Inter-switch link
References
Use these publications for reference:
򐂰 IBM System Storage N series MultiStore Management Guide, GA32-0524-01
򐂰 IBM System Storage N series Cluster Installation and Administration Guide,
GC26-7790
IBM System Storage N series MetroCluster Planning Guide
15
The team that wrote this IBM Redpaper
This IBM Redpaper was produced by a team of specialists from around the world
for the International Technical Support Organization, Tucson Center.
Alex Osuna is a Project Leader with the IBM International Technical Support
Organization, Tucson Center. Alex writes extensively and teaches IBM classes
about all areas of storage. Before joining the ITSO two years ago, Alex was a
Systems Engineer for Tivoli® Sales. Alex has over ten certifications from IBM,
Microsoft®, and Red Hat. Alex has 28 years in the I/T industry with a focus on
storage.
Piet de Jonge is an IBM Training Instructor teaching zSeries® and IBM storage.
Piet has 20 years in the I/T industry. Prior to working for IBM, Piet was a System
Programmer with Dutch Telecom. Before becoming a training instructor, Piet was
an I/T Specialist and Sales Specialist. Piet holds a Graduate degree in
Biochemistry.
Thanks to the following people for their contributions to this project:
Jim Larson, Network Appliance™ Corporation
Roland Tretau, IBM ATS Europe
16
IBM System Storage N series MetroCluster Planning Guide
Notices
This information was developed for products and services offered in the U.S.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area.
Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM
product, program, or service may be used. Any functionally equivalent product, program, or service that
does not infringe any IBM intellectual property right may be used instead. However, it is the user's
responsibility to evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document.
The furnishing of this document does not give you any license to these patents. You can send license
inquiries, in writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.
The following paragraph does not apply to the United Kingdom or any other country where such
provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION
PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR
IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer
of express or implied warranties in certain transactions, therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may
make improvements and/or changes in the product(s) and/or the program(s) described in this publication at
any time without notice.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in any
manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the
materials for this IBM product and use of those Web sites is at your own risk.
IBM may use or distribute any of the information you supply in any way it believes appropriate without
incurring any obligation to you.
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm
the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on
the capabilities of non-IBM products should be addressed to the suppliers of those products.
This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to the names and addresses used by an actual business
enterprise is entirely coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrates programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the
sample programs are written. These examples have not been thoroughly tested under all conditions. IBM,
therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy,
modify, and distribute these sample programs in any form without payment to IBM for the purposes of
developing, using, marketing, or distributing application programs conforming to IBM application
programming interfaces.
© Copyright International Business Machines Corporation 2006. All rights reserved.
Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by
GSA ADP Schedule Contract with IBM Corp.
17
Send us your comments in one of the following ways:
򐂰 Use the online Contact us review IBM Redbook form found at:
ibm.com/redbooks
򐂰 Send your comments in an e-mail to:
redbook@us.ibm.com
򐂰 Mail your comments to:
IBM Corporation, International Technical Support Organization
Dept. HYTD Mail Station P099, 2455 South Road
Poughkeepsie, NY 12601-5400 U.S.A.
®
Redpaper
Trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States,
other countries, or both:
Redbooks (logo)
zSeries®
™
IBM®
System Storage™
Tivoli®
The following terms are trademarks of other companies:
Network Appliance, MultiStore, Data ONTAP, and the Network Appliance logo are trademarks or registered
trademarks of Network Appliance, Inc. in the U.S. and other countries.
Microsoft, and the Windows logo are trademarks of Microsoft Corporation in the United States, other
countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
18
IBM System Storage N series MetroCluster Planning Guide