Redpaper
Alex Osuna
Roland Tretau
IBM System Storage N series
MetroCluster
Introduction
The IBM® System Storage™ N series MetroCluster extends Clustered Failover capabilities
from a primary to a remote site. It replicates data from the primary site to the remote site to
ensure that the data there is completely up-to-date and available. If Site A goes down,
MetroCluster allows you to rapidly resume operations at a remote site minutes after a
disaster:
򐂰 Stretch MetroCluster provides a disaster recovery option at distances up to 500 meters
between each N series system (Figure 1 on page 2).
– Available on N5000 & N7000 series.
򐂰 Fabric MetroCluster provides a disaster recovery option at distances up to 100 km using a
Fibre Channel switched network.
– Available on N5000 series
In this IBM Redpaper publication, we discuss the following:
򐂰 MetroCluster Features & Benefits
򐂰 MetroCluster Functionality
򐂰 Failure Scenarios
© Copyright IBM Corp. 2006. All rights reserved.
ibm.com/redbooks
1
Figure 1 shows a MetroCluster configuration.
Site A
Site B
Stretch
Fabric
X
Vol-X
Ym
Mirrored Vol-Y
Xm
Mirrored Vol-X
Y
Vol-Y
Figure 1 MetroCluster configurations
MetroCluster Benefits
The benefits of using MetroCluster are:
򐂰 MetroCluster is designed to be a simple-to-administer solution that extends fail-over
capability from within a data center to a remote site.
򐂰 MetroCluster is also designed to provide replication of data from the primary site to a
remote site, which helps to keep the data at the remote site current.
򐂰 The combination of failover and data replication aids in the recovery from disaster —
helping prevent loss of data — in less time than otherwise possible.
򐂰 Extends Clustered Failover capabilities from a primary to a remote site.
򐂰 Replicates data from the primary site to the remote site to ensure that data there is
completely up-to-date and available.
򐂰 If Site A goes down, MetroCluster allows you to rapidly resume operations at a remote site
minutes after a disaster.
N series Business Continuity Solutions
The environment or business need determines the necessary configuration for you.
2
IBM System Storage N series MetroCluster
Protection levels
MetroCluster provides three protection levels:
򐂰 Within the data center
򐂰 Campus distances
򐂰 WAN distances
In the following section, we further discuss the protection levels.
Within the Data Center
For protection of power faults, node failure, and network connector failure, a Cluster Failover
may be the right protection for you, as shown in CFO Figure 2.
Campus distances
For protection from building, power, node, volume, switch, and network connector failures,
the Stretch MetroCluster might be the right configuration for you. By creating a scenario
where failover occurs to another building or installation on the campus, availability is
increased even one step further.
WAN distances
For those of you whose business requires the highest level of availability, the MetroCluster
Fabric configuration delivers the needed options. By making available different synchronous
versions of SnapMirror®, the Stretch MetroCluster configuration protects against location,
building, power, node, volume, switch, and network connector failures.
Figure 2 shows the protection levels.
IBM System Storage
WAN Distances
Campus distances
Async SnapMirror
Primary
Data Center
Within DataCenter
Clustered
Failover (CFO)
• High system
protection
MetroCluster (Stretch)
• Cost effective zero RPO
protection
• Most cost effective with RPO
from 10 min. to 1 day
MetroCluster (Fabric)
• Cost effective zero RPO
protection
Sync SnapMirror
• Most robust zero RPO
protection
5
1
N Series Metro Cluster
© 2006 IBM Corporation
Figure 2 Protection levels
IBM System Storage N series MetroCluster
3
What is MetroCluster
MetroCluster software provides an enterprise solution for high availability over wide area
networks (WAN). MetroCluster deployments of IBM N series storage systems are used for:
򐂰 Business continuance
򐂰 Disaster recovery
򐂰 Achieving recovery point and recovery time objectives
MetroCluster technology is an important component of enterprise data protection strategies. If
a disaster occurs at a source site, businesses can continue to run and access data from a
clustered node in a remote site. The primary goal of MetroCluster is to provide mission-critical
applications with redundant storage services in case of site-specific disasters. It is designed
to tolerate site-specific disasters with minimal interruption to mission-critical applications and
zero data loss by synchronously mirroring data between two sites. A MetroCluster system is
made up of the following components and requires the following licenses:
򐂰 Multiple storage controllers and high availability (HA) configuration
Provides automatic failover capability between sites in case of hardware failures.
򐂰 SyncMirror®
Provides an up-to-date copy of data at the remote site. The data is ready for access after
failover without administrator intervention.
򐂰 Cluster remote
Provides a mechanism for the administrator to declare site disaster and initiate a site
failover using a single command, for ease of use.
򐂰 FC switches
Provide storage system connectivity between sites that are greater than 500* meters
apart.
MetroCluster allows the active/active configuration to be spread across data centers up to
100 kilometers apart (see Figure 3 on page 5). In the event of an outage at one data center,
the second data center can assume all affected storage operations, that are lost, with the
original data center. SyncMirror is required as part of MetroCluster to ensure that an identical
copy of the data exists in the second data center, should the original data center be lost:
򐂰 MetroCluster and SyncMirror extend active/active clustering across data centers that are
up to 100 kilometers apart. Either dark-fiber or Dense Wave Division Multiplexing (DWDM)
between the switches is required.
򐂰 MetroCluster and SyncMirror provide the highest level of storage resiliency across a local
region.
򐂰 Highest levels of regional storage resiliency ensure continuous data availability in a
particular geography.
4
IBM System Storage N series MetroCluster
Figure 3 shows how MetroCluster allows the active/active configuration to be spread across
data centers that are up to 100 kilometers apart.
IBM System Storage
MetroCluster is a unique, cost-effective synchronous replication solution for
combined high availability and disaster recovery within a campus or metro
area
Major Data Center
Nearby Office
LAN/SAN
Stretch MetroCluster
provides Campus DR
protection
- Can stretch up to 500m
N Series
Fabric MetroCluster
provides Metropolitan DR
protection
Disks
- Can stretch up to 100km
with FC switches
5
1
Configurations
N Series Metro Cluster
© 2006 IBM Corporation
Figure 3 MetroCluster
The benefits of MetroCluster: Synchronous Mirroring with
SyncMirror
Synchronous Mirroring for MetroCluster mirrors the Write Anywhere File Layout (WAFL®)
volumes (aggregates). Both copies or plexes are updated synchronously on writes, which
insures consistency. The design of IBM N series and MetroCluster provides data availability
even in the event of multiple disk failures. Read performance is optimized by performing
application reads from both plexes (Figure 4 on page 6). Performance is directly affected by
the distance between plexes. One of the additional performance benefits of Synchronous
mirroring is the low write overhead and latencies. Mirroring is done closer to the hardware,
and there is Zero impact on the secondary controller.
IBM System Storage N series MetroCluster
5
Figure 4 illustrates a MetroCluster with mirrored plexes.
IBM System Storage
FC
FC
x
Ple0
Plex1
6
1
RAID Groups
N Series Metro Cluster
RAID Groups
© 2006 IBM Corporation
Figure 4 MetroCluster with mirrored plexes
MetroCluster: SyncMirror configuration
Use the sysconfig –r command (Figure 5) to see the volume/plex/raidgroup relationship in
Data ONTAP®.
Î M irrored volum e
Volume spiel (online, normal, mirrored) (zoned checksums)
Plex /spiel/plex0 (online, normal, active)
RAID group /spiel/plex0/rg0
RAID Disk
--------parity
data
Device
-----------3.2
3.3
HA
SHELF BAY
--------------3
0
2
3
0
3
CHAN
---FC:A
FC:A
Used (MB/blks)
-------------8579/17570816
8579/17570816
Phys (MB/blks)
-------------8683/17783112
8683/17783112
CHAN
---FC:A
FC:A
Used (MB/blks)
-------------16979/34774016
8579/17570816
Phys (MB/blks)
-------------17560/35964296
17560/35964296
Plex /spiel/plex2 (online, normal, active)
RAID group /spiel/plex2/rg0
RAID Disk
--------parity
data
Device
-----------6.8
6.9
HA
SHELF BAY
--------------6
1
0
6
1
1
7
1
N Series Metro Cluster
Figure 5 Volume relationships
6
IBM System Storage N series MetroCluster
© 2006 IBM Corporation
Use aggr mirror to start mirroring the plexes, as we show in Example 1.
Example 1 aggr mirror syntax
aggr mirror <atv-name> [-n] [-f] [-v <victim-aggr>] [-d <disk_name1> ... <disk_n
ameN>] - add a mirror to aggregate or traditional volume <atv-name>
MetroCluster: Cluster_Remote License
The Cluster_Remote License provides features that enable the administrator to declare a site
disaster and initiate a site failover using a single command:
򐂰 Enables the cf forcetakeover -d.
– Initiates a takeover of the local partner even in the absence of a quorum of partner
mailbox disks.
򐂰 Provides the ability for an administrator to declare a site-specific disaster and have one
node takeover its partner’s identity without quorum of disks.
򐂰 Root volumes of both filers MUST be synchronously mirrored.
򐂰 Only synchronously mirrored aggregates are available during a site disaster.
򐂰 Requires administrator intervention as a safety precaution against a ‘split brain’ scenario
(cf forcetakeover –d).
Important: Site-specific disasters are not the same as a Normal Cluster Failover!
MetroCluster components
MetroCluster is an integrated solution made up of the following components:
򐂰 Clustered Failover
Provides the high-availability failover capability between sites. The administrator controls
the failover decision depending on the impact of the disaster.
򐂰 SyncMirror
Provides an up-to-date copy of data at the remote site. You can access the data after
failover, without administrator intervention.
򐂰 Cluster _Remote
Provides a mechanism for an administrator to declare a site disaster and initiate a site
failover using a single command, for ease of use.
򐂰 FC Switches
Provides filer connectivity between sites that are greater than 500 meters apart, which
enables sites to be located at a safe distance away from each other.
IBM System Storage N series MetroCluster
7
MetroCluster configurations
Table 1 lists the MetroCluster configurations.
Table 1 MetroCluster configurations
Configuration
Supported
platform
Switch
Mode
operation
Number of
dedicated
switches
Drives
supported
capacity
Shelves
per
loop
Maximum
number of
shelves
Short Distance
<500 Meters
N5000
N7000
series
N/A
N/A
SATA or FC
6
Same as
N5XXX A20
and N7XXX
A20
Long Distance
500 Meters to
100KM
N5000
series
Fabric
2 per site (4
per
MetroClust
er pair)
FC
2
24 (12 each
side)
Table 2 Switch distances
Switch
Up to
500m
<=10km
<=10km
<=50km
<=80km
IBM 2005-B16
Brocade 200E
Standard
SFPs
Vendor: Finisar
Vendor #:
FTLF-1319P1xCL
Vendor: Finisar
Vendor #:
FTLF-1419P1BCL
Vendor: Finisar
Vendor #:
FTLF-1419P1BCL
Vendor: Finisar
Vendor #:
FTLF-1519P1BCL
IBM 2005-H16
Brocade 3850
IBM 2005-H08
Brocade 3250
IBM 2109-F16
Brocade 3800
IBM 3534-F08
Brocade 3200
Table 3 shows the supported N series Gateway configurations.
8
IBM System Storage N series MetroCluster
Table 3 Supported Gateway Configurations
Gateway Model
Data ONTAP
Versions
Switches
Storage
N5500
N5200
7.2.3
Brocade 48000,
7500, 5000,
4900, 4100, 200E
with switch
FW 5.2.1b
HDS TagmaStore FW 50-09-06,
50-08-06, 50-07-64
HDS Lightning 9980V/9970V FW
21-14-33
HDS Lightning 9960V/9910 FW
01-19-99
IBM DS4700, DS4800 FW
6.23.05.00
N5500
N5200
N/A (Stretch
MetroCluster)
N5500
N5200
Cisco MDS 9506,
9509 with
switch FW 3.0(2)
HDS TagmaStore FW 50-09-06,
50-08-06, 50-07-64
HDS Lightning 9980V/9970V FW
21-14-33
HDS Lightning 9960V/9910 FW
01-19-99
N7800
N7600
N5600
N5300
McData 6140 FW
v9.02.01
N7800
N7600
N5600
N5300
Brocade 48000,
7500, 5000,
4900, 4100, 200E
with switch
FW 5.2.1b
HDS TagmaStore FW 50-09-06,
50-08-06, 50-07-64
HDS Lightning 9980V/9970V FW
21-14-33
HDS Lightning 9960V/9910 FW
01-19-99
IBM DS4700, DS4800 FW
6.23.05.00
N7800
N7600
N5600
N5300
N/A (Stretch
MetroCluster)
Stretch MetroCluster and Fabric MetroCluster
The IBM N series supports two configurations: Stretch and Fabric. We cover both
configurations in this section.
Stretch
Stretch MetroCluster provides a disaster recovery option at distances up to 500 meters
between each N series system. It only requires the MetroCluster license and the SyncMirror
license.
To connect the N series nodes over a distance greater than 30m, you need to convert the
IB-copper to fiber using feature 1042 - Copper-Fiber Converter. This adaptor has multi fiber
Push-On (MPO) connectors and needs a MTP plug (NVRAM5).
What is MTP/MPO? Optical links specify the LC duplex connector for 1X links, the MTP/MPO
for 4X links, and dual MTP/MPO for 12X links. All of these optical connectors are accepted as
ad hoc industry standards for other optical data, communication protocols as well.
IBM System Storage N series MetroCluster
9
Stretch is available on N5000 & N7000 series.
Fabric
Fabric MetroCluster provides a disaster recovery option at distances up to 100km using a
Fibre Channel switched network.
Available on N5000 and N7000 series
Fabric MetroCluster configuration details
First of all you will need to make sure you have the right Storage Initiator HBA’s see Table 4.
Platform
N5200,N5500
N5300,N5600
N7000
Min Data ONTAP
Version
Feature Code
1006
Yes
No
7.1h2
Feature Code
1014
No
Yes
7.2.3
Feature Code
1029
No
Yes
7.2.3
Table 4 Fabric MetroCluster Storage Initiator HBA’s
Onboard ports 0a,0b,0c and 0d can be used
The heartbeat and NVRAM uses a separate Dual-port MetroCluster HBA. This HBA is
dedicated and not used for the IO traffic of SyncMirror see Table 5.
Table 5 Fabric MetroCluster Interconnect HBA’s
Platform
N5200,N5500
N5300,N5600
N7600,N7800
Min Data ONTAP
Feature Code
1018
Yes
No
7.1.H12
Feature Code
1032
No
Yes
7.2.3
Additional Requirements for Fabric MetroCluster support
򐂰 Must use Clustered models
򐂰 No-Charge SyncMirror License on each storage controller
򐂰 EXN2000 or EXN4000 storage only (EXN1000
Fabric MetroCluster HBAs
In this section we discuss 1042 - Copper-Fiber Converter (1042).
To connect the filer heads over a distance greater than 30 m, for a stretched cluster you need
to convert the IB-copper to fiber using feature 1042 - Copper-Fiber Converter. This adaptor
has MPO connectors and needs a MTP plug (NVRAM5).
10
IBM System Storage N series MetroCluster
Table 6 lists the Fabric MetroCluster Storage Initiator HBAs.
Table 6 Fabric MetroCluster Storage Initiator HBAs
Platform
N5200, N5500
N5300,N5600
N7600,N7800
Min Data ONTAP
Feature code 1006a
Yesb
No
No
7.1h2
a. Fabric MetroCluster only 1006 supported, Stretch MetroCluster all Storage Initiator HBAs
supported.
b. Can use onboard ports 0a,0b,0c,0d.
Table 7 shows the Fabric MetroCluster Interconnect HBAs.
Table 7 Fabric MetroCluster Interconnect HBAs
Platform
N5200,N5500
N5300,N5600
N7600,N7800
Min Data ONTAP
Feature Code 1018
Yes
No
No
7.1.H12
Stretch MetroCluster: Campus Distance (<500m)
In a stretch MetroCluster, the controllers and expansion shelves are attached to Fibre
Channel switches, and the switches have GBICs to communicate across the WAN to one
another. SyncMirror is built into MetroCluster in that every write is written to two separate
expansion units to two separate aggregate groups.
The advantage of MetroCluster is that it can take a high availability solution and stretch it
outside of the frame. Think about taking a disk subsystem, such as the IBM DS4800,
separating the controllers miles apart, and maintaining two separate disk groups to ensure
failover, instead of building two completely separate DS4800 systems and synchronously
mirroring between the two. The benefits of Stretch MetroCluster, Campus Distance are:
򐂰 Stretch MetroCluster provides a disaster recovery option at distances of up to 500 meters
between each IBM N series system. See Figure 6 on page 12.
򐂰 Stretch MetroCluster only requires the MetroCluster license and the SyncMirror license.
򐂰 Stretch MetroCluster is available on N5000 and N7000 series.
IBM System Storage N series MetroCluster
11
Figure 6 shows the MetroCluster at less than 500 meters.
Figure 6 MetroCluster <500M
You can use SATA disks either in mirrored configuration or non-mirrored configuration (with
one of the nodes only) on a Stretch MetroCluster.
Fabric MetroCluster: Metro Distance (>500m)
The benefits of MetroCluster, Metro Distance are:
򐂰 Fabric MetroCluster provides a disaster recovery option at distances up to 100km using a
Fibre Channel switched network (Figure 7)
򐂰 Fabric MetroCluster uses switches for longer distance disaster recovery solutions.
12
IBM System Storage N series MetroCluster
Figure 7 shows an example of the Fabric MetroCluster.
Figure 7 Fabric MetroCluster
Using DWDM switches
Dense Wave Division Multiplexing is a method of multiplexing multiple channels of fiber
optic-based protocols, such as ESCON®, Fibre Channel, FICON®, and Gbit Ethernet, onto
physical cable by assigning different wavelengths of light (that is colors) to each channel and
then fanning it back out at the receiving end. The major players in the enterprise class DWDM
marketplace are: Nortel Networks, Cisco (ONS 15540), and Lucent.
Dense Wave Division Multiplexors are data link Layer 2 tools. Thus, the typical DWDM
machine does not perform any switching, routing, or protocol conversion.
IBM System Storage N series MetroCluster
13
Figure 8 depicts a Fabric MetroCluster installation using DWDM.
Figure 8 DWDM environment
Fabric attach MetroCluster Switch Matrix
Table 8 is shows the required Firmware and Data ONTAP levels for San switches.
Table 8 MetroCluster Switch Matrix
Data ONTAP level
Switch Model
IBM 2105-H08/Brocade 3250
7.1h2
7.2 or above
Minimum Firmware level
5.2.0a
IBM 2105-H16/Brocade 3850
IBM 3534-F08/Brocade 3200
3.3.1b
IBM 2109-F16/Brocade 3800
IBM 2005-B16/Brocade 200E
5.1.0
Cable selection
Use Table 9 on page 15 to determine if the desired cable length is within the supported
maximum specification. We provide this table for guidance to determine the distances;
however, the best way to determine the best cable type is by testing the cable length in the
environment (because there are many variables such as types of cables, panels, and so
forth), for example, a client needs to run an FC cable over approximately 260 m (850 ft.) and
wants to run it at 2 Gbps. How do we determine the maximum distance that can be allowed
14
IBM System Storage N series MetroCluster
for cabling between nodes? Table 9 is a cable selection chart than can help you determine if
the cable length you want is within the supported maximum specification.
Table 9 Cable selection chart
Cable
type
Fiber core
type
Mode
Wave
length
Maximum
distance
(m)
Attenuation
(Db/Km)
Maximum
channel
attenuation
Splice
loss
Connector
pair loss
OM2
50/
125 um
Multi
850
550
3.00
3.25
0.3
0.75
OM3
50/
125 um
Multi
850
550
3.00
3.50
0.3
0.75
OS1
9/125 um
Single
1310
2000
0.40
7.80
0.3
0.75
OM2
50/
125 um
Multi
850
550a
3.00
2.62
0.3
0.75
OM3
50/
125 um
Multi
850
550
3.00
3.25
0.3
0.75
OS1
9/125 um
Single
1310
2000
0.40
7.80
0.3
0.75
OM2
50/
125 um
Multi
850
150
3.00
2.06
0.3
0.75
OM3
50/
125 um
Multi
850
150
3.00
3.00
0.3
0.75
OS1
9/125 um
Single
1310
500
0.40
7.80
0.3
0.75
1Gbps
2Gbps
4Gbps
IB 1X 250MB/sec
OM2
50/
125 um
Multi
850
250
3.50
2.38
0.3
0.75
OM3
50/
125 um
Multi
850
500
3.50
3.25
0.3
0.75
a. According to the maximum channel attenuation (2.62 dB) and an attenuation of 3.00 dB/km, the maximum distance for this cable type is 406 m, so be careful with longer distances (up to 550 m) with OM2
cable at 2 Gbps.
Table 9 summarizes data that is related to optical cabling for data communications, which is
available in documents that are published by various standards organizations. We focus on
data that is relevant to fiber deployments, which are supported on IBM N series systems.
To determine whether the desired cable run length is within the supported maximum
specification:
1. Determine the needed transfer rate based on the type of shelf you use.
2. Find out what fiber type is installed for the system.
3. Determine the number of connectors in the path between the nodes.
Refer to Table 9 and locate the maximum supported operating distance, compare this value to
the distance for the actual intended application, and verify that the actual does not exceed the
supported maximum.
IBM System Storage N series MetroCluster
15
Consider the 260 m distance for the above client. Let us assume:
򐂰 Desired transfer rate: 2 Gbps
򐂰 Fiber core type: 50/125 OM2 Multimode cabling
򐂰 Number of connector pairs: two
Table 9 on page 15 shows that 260 m is within the operating maximum distance for all Fibre
Channel transfer rates. This also assumes that the fiber connection is point-to-point with only
source and destination connections and no patch panels or splices.
MetroCluster best practices
The following list contains MetroCluster best practices:
򐂰 Read the Data ONTAP 7.2 Active/Active Configuration Guide.
򐂰 Read Chapter 3: “Installing a MetroCluster” in the Data ONTAP 7.2 Active/Active
Configuration Guide.
򐂰 Ensure correct cabling!!!
򐂰 Ensure correct switch configuration, and apply 16port and fabric licenses.
򐂰 Test the installation and fail-over scenarios.
򐂰 Train the staff.
N series resiliency
Figure 9 gives you an idea of what events the N series can protect against with MetroCluster.
Event
Does th e Event T rigger a Failover?
Single Disk F ailure
No
Yes
Double Disk Failure (sam e RAID group)
No
Yes, with no CFO necessary
Yes, with no CFO necessary
Triple Disk Failure (sam e RAID group)
No
Single HBA (initiator) failure, LO OP A
No
Yes, with no CFO necessary
Single HBA (initiator) failure,
No
Yes, with no CFO necessary
Single HBA initiator failure, (Loop A + B)
No
Yes, with no CFO necessary
LRC F ailure (Loop A)
No
Yes, with no CFO necessary
LRC F ailure (Loop B)
No
Yes
ESH Failure (Loop A)
No
Yes, with no CFO necessary
LO OP B
ESH Failure (Loop B)
No
Yes
Shelf (backplane) Failure
No
Yes, with no CFO necessary
Shelf, Single Power F ailure
No
Yes
Shelf, Dual Power Failure
No
Yes, with no CFO necessary
Head, Single Power Failure
No
Yes
Head, Dual Power F ailure
Yes
Yes, if CFO is successful
Cluster Interconnect Failure (1 port)
No
Yes
Cluster Interconnect Failure (both ports)
No
Yes
Ethernet Interface Failure (prim ary, no VIF )
No
No
Ethernet Interface Failure (prim ary, VIF )
No
Yes
Ethernet Interface Failure (secondary, VIF)
No
Yes
Ethernet Interface Failure (VIF, all ports)
No
No
Tape Interface Failure
No
Yes
Heat +/- operating tem peratures
Yes
Yes, if partner is within operating tem p...
Fan Failures (disk shelves or filer head)
No
Yes
Reboot/Panic
No*
M aybe (depends on cause of panic)
22
Figure 9 N series resiliency
16
Is data still available o n the affected volum e
after th e event?
IBM System Storage N series MetroCluster
Failure scenarios
In this section, we give examples of some possible failures and the associated recovery that
is available when you use MetroCluster.
Host failure
In this scenario, Host 1 is lost; however, access to data continues uninterrupted. Host 2
continues accessing data, as shown in Figure 10.
Figure 10 Host failure
Node failure
In this scenario, environmental factors destroyed IBM N series N1. IBM N series takes over
access to its disks, as shown in Figure 11 on page 18. The fabric switches provide the
connectivity for the IBM N series and the hosts, to continue data access uninterrupted.
IBM System Storage N series MetroCluster
17
Figure 11 is an example of Node failure.
Host 1
Data
Center
#1
Host 2
Data
Center
#2
Recovery - Automatic
N #2 can access the
disks in both DCs
N1
DC1
Primary
DC 2
Mirror
N2
DC 2
Primary
DC1
Mirror
Figure 11 Node failure
Node and expansion unit failure
This scenario is likely happen when a catastrophic action occurs to a rack with a IBM N series
and its expansion units. Both the IBM N series and the expansion units become unavailable,
as shown in Figure 12 on page 19. In order to continue access, you must perform the cluster
failure with the cfo -d command. Data access is restored because DC1 Mirror was
synchronized with DC1 Primary. All hosts through connectivity provided by the fabric
switches can once again have access to required data.
18
IBM System Storage N series MetroCluster
Figure 12 depicts a node and expansion unit failure.
Host 1
Data
Center
#1
Data
Center
#2
Host 2
Dual Failure
Recovery
– One Step failover
with cfo –d
command
N1
DC1
Primary
DC 2
Mirror
N2
DC 2
Primary
DC1
Mirror
Figure 12 Node and expansion unit failure
MetroCluster interconnect failure
In this scenario, the fabric switch interconnects failed (Figure 13 on page 20). Although this is
not a critical failure, resolution must occur promptly in case of a more critical failure. During
this period, data access is uninterrupted to all hosts. Mirroring and failover are disabled,
which reduces data protection. Upon resolution of interconnect failure, re-syncing of mirrors
occurs.
IBM System Storage N series MetroCluster
19
Figure 13 illustrates the MetroCluster interconnect failure, where the fabric switch
interconnects failed.
Host 1
Data
Center
#1
Data
Center
#2
Host 2
Recovery
- No failover; Mirroring
disabled
- Both filer heads will
continue to run serving
it’s LUNs/volumes
N1
DC1
Primary
DC 2
Mirror
N2
- Re-syncing happens
automatically after
interconnect is
reestablished
DC 2
Primary
DC1
Mirror
Figure 13 Interconnect failure
MetroCluster site failure
In this scenario, a site disaster occurred, and all switches, storage systems, and hosts are
lost (Figure 14 on page 21). To continue data access a cluster failover must be initiated with
the cfo -d command. Both primaries now exist at Data Center 2, and hosting of Host1 is also
done at Data Center 2.
Note: If the site failure is staggered in nature and the interconnect fails before the rest of
the site is destroyed there is a chance of data loss. This occurs because processing has
continued after the interconnect has failed. Typically site failures occur pervasively and at
the same time.
20
IBM System Storage N series MetroCluster
Figure 14 depicts the MetroCluster site failure, where a site disaster occurs, and all switches,
storage systems, and hosts are lost.
Figure 14 MetroCluster site failure
MetroCluster site recovery
After the hosts, switches, and storage systems are recovered at Data Center 1, a recovery
can be performed. A cf giveback command is performed to resume normal operations, as
shown in Figure 15 on page 22. Mirrors are re-synchronized and primaries and mirrors are
reversed to previous status.
IBM System Storage N series MetroCluster
21
Figure 15 illustrates a MetroCluster site recovery.
IBM System Storage
MetroCluster – Site Recovery
Data
Center
Center
#1
Host 1
Host 2
Host 1
Emulation
Data
Center
#2
Site Recovery
One Step return to
normal
CF giveback
N2
N1
DC1 Primary
reverts back
to DC1 after
rebuild
DC 2
Primary
DC1
Primary
DC1
Break
DC1
Mirror
Mirror
DC 2
Mirror
N Series Metro Cluster
© 2006 IBM Corporation
Figure 15 MetroCluster site recovery
The team that wrote this Redpaper
This Redpaper was produced by a team of specialists from around the world working at the
International Technical Support Organization (ITSO), Poughkeepsie Center.
Alex Osuna is a Project Leader at the International Technical Support Organization, San
Jose Center. He has worked in the IT industry for 26 years, including 25 years with IBM. He
has extensive experience in software and hardware storage in service, support, planning,
early ship programs, performance analysis, education, and published flashes. Alex has also
provided presales and postsales support.
Roland Tretau is an IBM Certified IT Specialist and Consulting IT Specialist who provides
technical leadership for IBM System Storage N series NAS solutions throughout Europe. He
has a solid background in project management, consulting, operating systems, storage
solutions, enterprise search technologies, and data management. Roland led software
development, large-scale IT solution and architectures designs in customers’ mission critical
environments. He holds a Master's degree in Electrical Engineering with an emphasis in
telecommunications. He is a Red Hat Certified Engineer (RHCE) and a Microsoft® Certified
Systems Engineer (MCSE). He also holds a Masters Certificate in Project Management from
The George Washington University School of Business and Public Management.
22
IBM System Storage N series MetroCluster
Notices
This information was developed for products and services offered in the U.S.A.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area.
Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM
product, program, or service may be used. Any functionally equivalent product, program, or service that does
not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not give you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A.
The following paragraph does not apply to the United Kingdom or any other country where such
provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION
PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR
IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of
express or implied warranties in certain transactions, therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in any
manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the
materials for this IBM product and use of those Web sites is at your own risk.
IBM may use or distribute any of the information you supply in any way it believes appropriate without
incurring any obligation to you.
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to the names and addresses used by an actual business
enterprise is entirely coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs.
Send us your comments in one of the following ways:
򐂰 Use the online Contact us review redbooks form found at:
ibm.com/redbooks
򐂰 Send your comments in an email to:
redbook@us.ibm.com
򐂰 Mail your comments to:
© Copyright International Business Machines Corporation 2006. All rights reserved.
Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by
GSA ADP Schedule Contract with IBM Corp.
23
IBM Corporation, International Technical Support Organization
Dept. HYTD Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400 U.S.A.
®
Trademarks
The following terms are trademarks of the International Business Machines Corporation in
the United States, other countries, or both:
Redbooks (logo)
ESCON®
®
FICON®
IBM®
Redpaper
System Storage™
The following terms are trademarks of other companies:
WAFL, SyncMirror, SnapMirror, Data ONTAP, and the Network Appliance logo are trademarks or registered
trademarks of Network Appliance, Inc. in the U.S. and other countries.
Microsoft, and the Windows logo are trademarks of Microsoft Corporation in the United States, other
countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
24
IBM System Storage N series MetroCluster