IBM Systems & Technology Group
Global Copy, Metro Mirror, Global Mirror, and
Metro/Global Mirror Overview
Charlie Burger
Storage Systems Advanced Technical Support
© 2004 IBM Corporation
Advanced Technical Support, Americas
Table of Contents
 Background
– Consistency and dependent writes
 Peer-to-Peer Remote Copy - PPRC
 PPRC Considerations
 Establish Path Considerations
 FCP vs ESCON Links
 Metro Mirror
 Global Copy
 Global Mirror
 Metro/Global Mirror
 Failover
 Failback
 Addendum
2
© 2005 IBM Corporation
Advanced Technical Support, Americas
Background
 Dependent writes
– The start of one write operation is dependent upon the completion of a
previous write to a disk in either the same subsystem frame or a different
subsystem frame
– Basis for providing consistent data for copy operations
 Consistency
– Preserves the order of dependent writes
• For databases, consistent data provides the capability to perform a data
base restart rather than a data base recovery
– Restart can be measured in minutes while recovery could be hours or
even days
 Asynchronous processing
– The separation of data transmission from the signaling of I/O complete
• Distance between primary and secondary has little impact upon the
response time of the primary volume
• Helps minimize impact to application performance
3
© 2005 IBM Corporation
Advanced Technical Support, Americas
Database RESTART or RECOVER?
Operating
System
Applications
Other
UNIX/NT
AS/400's
RS/6000
S/390
Operations Staff
Other
UNIX/NT
AS/400's
Network Staff
RS/6000
Data
Applications Staff
Management Control
Telecom Network
Physical Facilities
Consistency provides:
"DB Restart" - To start a DataBase application following an outage without having to
restore the database. This is a process measured in minutes.
and avoid: "DB Recover" - Restore last set of DataBase Image Copy tapes and apply
log changes to bring database up to point of failure. This is a process measured in
hours or even days
4
© 2005 IBM Corporation
Advanced Technical Support, Americas
Dependent Writes
1. Log Update
OK
B
C
OK
L
M
X
Y
B
C
L
M
DataBase
Application
OK
3. Mark Log Complete
2. Update database
X
Y
Many examples where start of one write operation is time
dependent on the completion of a previous write on a different
disk group or even different disk frame
Data base & log for example
Synchronous copy insures data integrity
5
© 2005 IBM Corporation
Advanced Technical Support, Americas
Dependent Write Consistency (1)
1. Log Update
OK
DataBase
Application
B
C
L
M
X
Y
B
C
L
X
X
M
Y
Scenario: 'Update Database' doesn't get propagated to 2nd site
Synchronous copy and 'dependant writes' means 'Mark Log Complete' will
never be issued by application
Result:
Database is consistent
6
© 2005 IBM Corporation
Advanced Technical Support, Americas
Dependent Write Consistency (2)
1. Log Update
OK
L
DataBase
Application
OK
C
B
X
M
X
Y
B
C
L
M
X
Y
2. Update database
Scenario: 'Mark Log Complete' doesn't get propagated to 2nd site
Result:
Secondary site logs say update was not completed
Backout of valid data will be done upon restart at secondary site
But database is consistent
7
© 2005 IBM Corporation
Advanced Technical Support, Americas
Consistency
 To achieve consistency at a remote mirror location you must
maintain the order of dependent writes
– You cannot have a write to one volume mirrored and then have a
dependent write to another volume not mirrored
 The remote mirror functions each maintain consistency in
their own way
– Metro Mirror uses ELB (Extended Long Busy) for CKD volumes and
I/O Queue Full for FB LUNs
– Global Mirror holds write I/Os while building an alternate bit map prior
before draining the OOS (out of sync) bit map when creating a
consistency group
– z/OS Global Mirror (XRC) uses timestamps to create consistency
groups
– Global Copy requires procedures to create consistency
8
© 2005 IBM Corporation
Advanced Technical Support, Americas
Peer-to-Peer Remote Copy - PPRC
 Metro Mirror – Synchronous PPRC
– Synchronous mirroring with consistency at the remote site
• RPO of 0
 Global Copy – PPRC Extended Distance (XD)
– Asynchronous mirroring without consistency at the remote site
• Consistency manually created by user
– RPO determined by how often user is willing to create consistent data at the remote
 Global Mirror
– Asynchronous mirroring with consistency at the remote site
• RPO can be somewhere between 3-5 seconds
 Metro/Global Mirror
– Three site mirroring solution using Metro Mirror between site 1 and site 2 and Global
Mirror between site 2 and site 3
• Consistency maintained at sites 2 and 3
– RPO at site 2 near 0
– RPO at site 3 near 0 if site 1 is lost
– RPO at site 3 somewhere between 3-5 seconds if site 2 is lost
9
© 2005 IBM Corporation
Advanced Technical Support, Americas
PPRC Considerations
PPRC secondary volume must be off-line to all attached systems
One-to-one PPRC volume relationship
Only IBM (DS8000/DS6000/ESS) to IBM is supported
•Logical paths have to be established between Logical Subsystems
•FCP links are bidirectional
•FCP links can also be used for server data transfer
•Up to 8 links per LSS
More than 8 links per physical subsystem is allowed
•Up to 4 LSS secondaries can be connected to a primary LSS
A secondary LSS can be connected to as many primary LSS systems
as links are available
Distance
ESCON links - 103 KM
FCP links - 300 KM
10
© 2005 IBM Corporation
Advanced Technical Support, Americas
Establish Path Considerations
 If paths have been established, issuing another path establish will
overlay the existing established path
 For example:
– 2 Paths are established using this command
mkpprcpath -dev IBM.2107-75FA120 -remotedev IBM.2107-75FA150 -srclss 01 -tgtlss 01
–remotewwnn 12341234000A000F I1A10:I2A20 I1A11:I2A21
– I wish to add another path so this command is issued
mkpprcpath -dev IBM.2107-75FA120 -remotedev IBM.2107-75FA150 -srclss 01 -tgtlss 01
–remotewwnn 12341234000A000F I0100:I0100
– The result will be the loss of the 2 previously established paths and only having
the new path established
– To add the path, the following should be issued
mkpprcpath -dev IBM.2107-75FA120 -remotedev IBM.2107-75FA150 -srclss 01 -tgtlss 01
–remotewwnn 12341234000A000F I0100:I0100 I1A10:I2A20 I1A11:I2A21
11
© 2005 IBM Corporation
Advanced Technical Support, Americas
FCP vs ESCON
•ESCON links run at 17. MB/sec with a sustained rate of approximately 12-14 MB/sec
•Fibre links can run 100-400 MB/sec depending upon the adapter with a sustained rate of approximately 80-320 MB/sec
•DS8000 and DS6000 only support PPRC fcp links
•ESS 800 supports both PPRC ESCON and fcp links
12
© 2005 IBM Corporation
Advanced Technical Support, Americas
What is Metro Mirror?
Disaster protection for all IBM supported platforms
Other potential uses:
–Data migration/movement between devices
–Data workload migration to alternate site
Hardware and LIC solution
Synchronous copy, mirroring (RAID 1) to another
DS8000/DS6000/ESS
Application independent
Some performance impact on application I/Os
Established at a disk level
A 2 site solution
13
© 2005 IBM Corporation
Advanced Technical Support, Americas
Profile of a PPRC Synchronous Write
Synchronous write
1 Write (channel end)
2 Write to secondary
3 Write acknowledged by secondary
4 Acknowledgement (Device end -- I/O
complete
1.
4.
3.
2.
14
© 2005 IBM Corporation
Advanced Technical Support, Americas
Maintaining Consistency – Metro Mirror
 Consistency group on the establish path commands
– Loss of communication between primary and secondary sites will
cause an extended long busy (ELB) for zSeries and I/O queue full for
open systems to be returned to any write issued to a volume in the
LSS that lost communication
 Automation can issue FREEZE commands to all of the LSSs
that have dependent data with the LSS that lost
communication
– Returning ELB or I/O queue full causes the next dependent write to
NOT be issued maintaining the order of the dependent writes
– After all of the FREEZE commands have been issued, RUN
commands can be issued to the LSSs to resume, otherwise the ELB
or I/O queue full will be returned for a default of 2 minutes
 Automation is required when dependent data spans across
multiple physical disk subsystems and is HIGHLY
recommended when all primaries are within a single
subsystem
15
© 2005 IBM Corporation
Advanced Technical Support, Americas
Consistency – Metro Mirror One Physical Subsystem
No automation
1
2
3
4
16
x
All paths lost means that
no updates are xmitted to
secondaries and consistency
is maintained
© 2005 IBM Corporation
Advanced Technical Support, Americas
Consistency – Metro Mirror One Physical Subsystem
No automation
1
x
2
3
4
One pair suspends,
others still mirror,
lose consistency
17
© 2005 IBM Corporation
Advanced Technical Support, Americas
Consistency – Metro Mirror Multiple Physical Subsystems
No Automation
1
2
x
3
4
Without automation,
order of dependent writes not
maintained
and consistency is lost
18
© 2005 IBM Corporation
Advanced Technical Support, Americas
Consistency – Metro Mirror Multiple Physical Subsystems
1
Automation
2
3
x
4
Automation insures
that the order of dependent
writes is maintained
19
© 2005 IBM Corporation
Advanced Technical Support, Americas
When to use Metro Mirror
Recovery system required to be current with the primary application
system
Can accept some performance impact to application write I/O
operations at the primary location
Recovery is on a disk-by-disk basis
Distance within maximum limits
103 KM for ESCON links and 300 KM for fcp links
RPQ for greater disances
20
© 2005 IBM Corporation
Advanced Technical Support, Americas
Metro Mirror Sequential Write Data Rate – Turbo R2
21
© 2005 IBM Corporation
Advanced Technical Support, Americas
Pre-Turbo DS8000 Metro Mirror 4 KB Write Service
Time Comparisons
22
© 2005 IBM Corporation
Advanced Technical Support, Americas
Pre-Turbo DS8100 Metro Mirror Sequential Write
Throughput
23
© 2005 IBM Corporation
Advanced Technical Support, Americas
Pre-Turbo DS8100 Metro Mirror Sequential Write
Throughput
24
© 2005 IBM Corporation
Advanced Technical Support, Americas
What is Global Copy?
 Global Copy uses an additional PPRC mode designed for high
performance data copy at long distances
– TSP OPTION(XD) Extended Distance or ds cli –type gcp
– Disk level option
 Asynchronous transfer of application primary writes to secondary
allows mirroring over long distances with minimal impact to host
performance
– Writes to primary disk receive immediate completion status while in XD mode
 Writes can be out of sequence on secondary disk
– Develop procedures to create a point in time consistency
 A 2 site solution
25
© 2005 IBM Corporation
Advanced Technical Support, Americas
Profile of an Asynchronous Write
Asynchronous write
Synchronous write
1. Write
Write (channel end)
Write to secondary
2. Write acknowledgement
(channel end / device end)
Write acknowledged by secondary
Acknowledgement (Device end -- I/O
complete
1.
1.
2.
4.
3. Write to secondary
3.
4. Write acknowledged by
secondary
2.
4.
3.
26
© 2005 IBM Corporation
Advanced Technical Support, Americas
Global Copy – How it works (1)
 Synchronous PPRC establish (initial copy) is done in two phases:
– Phase I - Copy all tracks in the volume starting at zero and going to the end of
the volume. Use a bitmap to keep track of which tracks need to be copied. Do
not transfer any host updates - just set the bit in the bitmap for new host
updates.
– Phase II - Go back through the bitmap to copy any host updates received
while in Phase I. Any host updates received during this phase, and for the
remainder of the PPRC pair life, will be sent synchronously to the remote
volume.
 Extended Distance PPRC
– Stay in Phase I forever
– No impact to host write response time
– Copy at remote site is "fuzzy" - updates are not sent in order or in time
consistent groups
27
© 2005 IBM Corporation
Advanced Technical Support, Americas
Global Copy – How it works (2)
 Establish PPRC pairs with Extended Distance option
– Writes to primary receive immediate completion status
 Primary records updated tracks in a bitmap
 Incremental copy of changed tracks or records periodically
sent to secondary
 To create a point in time consistency:
– Transition to PPRC synchronous until full duplex state is reached
• Usually a matter of seconds
– Alternatively, quiesce of I/O and flushing of buffers on primary host will
result in consistent secondary disk
28
© 2005 IBM Corporation
Advanced Technical Support, Americas
Global Copy – How it works (4)
 Agents process a
volume using the
Out-of-Sync (OOS)
bit map to determine
which tracks to xmit
Primary
Secondary
Primary
Secondary
Primary
Secondary
Primary
Secondary
 Not all volumes are
processed at the
same time
 As the volume is
processed, tracks
updated behind the
active track being
xmitted is recorded
in the OOS and will
be processed the
next time
29
© 2005 IBM Corporation
Advanced Technical Support, Americas
PPRC State Changes
 Transition to simplex means PPRC is withdrawn
 XD is established at the volume/LUN level
Simplex
Start
Syncronous
Extended
Distance
Pending
Duplex
Fuzzy
Suspended
30
© 2005 IBM Corporation
Advanced Technical Support, Americas
Volume State Transitioning
To transition from…..
To…..
Use the following command…..
 SYNC
SIMPLEX
CDELPAIR
 SIMPLEX
SYNC
CESTPAIR OPTION(SYNC)
 SYNC
SUSP
CSUSPEND
 SUSP
SYNC
CESTPAIR MODE(RESYNC)
 XD
SUSP
CSUSPEND
 SUSP
XD
CESTPAIR MODE(RESYNC) OPTION(XD)
 XD
SIMPLEX
CDELPAIR
 SIMPLEX
XD
CESTPAIR OPTION(XD)
 SUSP
SIMPLEX
CDELPAIR
 XD
SYNC
CESTPAIR OPTION(SYNC)
31
© 2005 IBM Corporation
Advanced Technical Support, Americas
Maintaining Consistency – Global Copy
 Consistency group is NOT specified on the establish path
command
– Data on Global Copy secondaries is not consistent so there is no need
to maintain the order of dependent writes
 Consistent data is created by the user
– Quiesce I/O
– Suspend the pairs
• FREEZE can be used and ELB will not be returned to the server since
consistency group was NOT specified on the establish path
– FlashCopy secondary to tertiary
• Tertiary will have consistent data
– Reestablish paths (if necessary)
– RESYNC (resumepprc) Global Copy
32
© 2005 IBM Corporation
Advanced Technical Support, Americas
When to use Global Copy
 Recovery system does not need to be current with the
primary application system
– RPO in the range of hours or days
– User creates consistent copy of recovery data
 Minor impact to application write I/O operations at the
primary location
 Recovery uses copies of data created by the user on tertiary
volumes
 Distance beyond ESCON or fcp limits
– 103 KM for ESCON links and 300 KM for fcp links
• RPQ for greater disances
 A great tool for migrating data
33
© 2005 IBM Corporation
Advanced Technical Support, Americas
What is Global Mirror?
 A Disaster Recovery (DR) data replication solution
– Reduced (less than peak bandwidth) network bandwidth requirements (duplicate
writes not sent)
 A 2 site solution
 Asynchronous data transfer
– No impact to the production write I/Os
 Peer-to-peer (no, outside the box, server MIPS)
– Microcode controlled
– Peer-to-peer data copy mechanism is Global Copy
– Consistency Group formation mechanism is FlashCopy
 3 copies (A  B  C)
– Or 4 copies (if test/practice copy (D copy) & DR is to be continued)
 Unlimited distance
 Very little data loss (Recovery Point Objective (RPO))
– Single digit seconds (goal was/is 3-5 seconds)
 Scalable
– Up to 8 primary and secondary physical subsystems
• More with an RPQ
34
© 2005 IBM Corporation
Advanced Technical Support, Americas
Global Mirror: Basic concept
Concept
Asynchronous long distance copy (Global Copy), i.e., little to no impact to application writes
Momentarily pause application writes (fraction of millisecond to few milliseconds)
Create point in time consistency group across all primary subsystems (in OOS bitmap)
New updates saved in Change Recording bitmap
Restart application writes and complete write (drain) of point in time consistent data to remote site
Stop drain of data from primary (after all consistent data has been copied to secondary)
Logically FlashCopy all data (i.e., 2ndary is consistent, now make tertiary look like 2 ndary)
Restart Global Copy writes from primary
Automatic repeat of sequence every few seconds to minutes to hours (selectable and can be
immediate)
Intended benefit
Long distance, no application impact (adjusts to peak workloads automatically), small RPO,
remote copy solution for zSeries and Open Systems data, and consistency across multiple
Global Copy (PPRC-XD)
subsystems
over long distance
Host
I/O
Primar
y
Local
Site
35
Could require channel
extenders
FlashCopy
FCP links only
(record, nocopy, persistent, inhibit target
write)
Secondary
Tertiary
(Asynchronous)
Global Copy
Remot
e Site
© 2005 IBM Corporation
Advanced Technical Support, Americas
Consistency Group Formation
Coordinate
local units
Drain Time
FlashCopy Relationships
being established
CG Interval Time
...
...
...
...
Coordination
Time
Let CG data drain to remote
Record new writes in bitmaps
but do not copy to remote
All FlashCopy Relationships
established
Global Copy continually cycles
through volume bitmaps
copying changed data to remote
mirror volumes
36
© 2005 IBM Corporation
Advanced Technical Support, Americas
Tuneables (input parameters)
 Maximum Coordination Time
– Maximum allowed pause of production write updates for the Consistency Group coordination
action
•
I.e., when the Master coordinates the formation of the Consistency Group with all Subordinates
– When coordination is completed, writes are allowed to continue
– Default = 50 milliseconds {Range: 0 to 65535 ms (65+ seconds)}
– If the ‘coordination time” is exceeded, coordination is stopped and all writes are allowed to
continue
– Design point is 2-3 ms
 Maximum Drain Time
– Maximum CG drain time in seconds before failing (terminating) current drain activity
– Default = 30 seconds {Range: 0 to 65535 (just over 18 hours)}
– After 5 failures, drain time is infinite, i.e., until a consistency group is form, i.e., completely drained
 Consistency Group Interval Time
– Time to wait before again starting the next consistency group formation process
– Default = 0 seconds {Range = 0 to 65535 seconds (just over 18 hours)}
37
© 2005 IBM Corporation
Advanced Technical Support, Americas
Typical Global Mirror Configuration
Multiple primary to multiple secondary subsystems
Consistency across all primary subsystems
LS
S
LS
S
Host I/O
When forming a Consistency Group,
PPRC-XD continues transmitting /
draining the consistent data to the
secondary site. Once the consistency
group is formed, the new update data
will be transmitted as in a PPRC-XD
environment without Asynchronous
PPRC.
LS
S
Once the A volumes have
been drained to the B
volumes, the B volumes
will be FlashCopied to the
C volumes.
LS
S
Local Site
LS
S
One Master,
multiple
Subordinates
Subordinate
Subordinate
LS
S
Remote Site
LS
S
LS
S
LS
S
LS
S
Master
Master communicates with
Subordinates to form consistency
groups
Note: The Master performs the same operations on volumes in the consistency group in its box when it directs the
Subordinates to perform operations.
38
© 2005 IBM Corporation
Advanced Technical Support, Americas
Global Mirror Initialization Process
4. Define Global Mirror session and
add volumes to the session
1. Establish Global Copy paths
Subordinate
2. Establish Global Copy pairs
FlashCop
y
Subordinate
SAN
.
.
FlashCop
y
FlashCop
y
Master
Wait until Global
Copy pairs have
completed 1st pass
copy then establish
FlashCopy pairs
3. Establish FlashCopy pairs
6. Start Global Mirror with Start
command sent to Mast
5. Establish control paths between
Master and Subordinates
Note: These paths could be created earlier
39
© 2005 IBM Corporation
Advanced Technical Support, Americas
Maintaining Consistency – Global Mirror
 Consistency group is NOT specified on the establish path
command
– Loss of communication will NOT cause ELB to be returned for writes
– FREEZE command can be used to suspend pairs after Global Mirror
session is paused but ELB will NOT be returned for writes to LSSs
 Consistency is maintained by not returning CE/DE or I/O
complete during the coordination phase when forming a
Consistency Group
– Not returning CE/DE or I/O complete causes the next dependent write
to NOT be issued maintaining the order of the dependent writes
40
© 2005 IBM Corporation
Advanced Technical Support, Americas
When to use Global Mirror?
 RPO can be greater than 0 but still needs to be very current
– In the single digit second range
 Limited impact to application write I/O operations at the
primary location
– Asynchronous data transfer
 Recovery is on a disk-by-disk basis
 Distance exceeds maximum limits for synchronous data
transfer
– 300 KM for fcp links
• Global Mirror only supports fcp links
41
© 2005 IBM Corporation
Advanced Technical Support, Americas
Global Mirror at 1000 mi DS8300 vs ESS 800 (both w/
128 x 10k RPM disk)
42
© 2005 IBM Corporation
Advanced Technical Support, Americas
What is Metro/Global Mirror
 A 3 site Disaster Recovery (DR) data replication solution
– Metro Mirror from local (A) to intermediate (B) and Global Mirror from
intermediate to remote (C)
– The Metro Mirror secondary is cascaded to the remote site
 4 copies of data (A  B  C  D)
– C and D are Global Mirror secondary and FlashCopy volumes
– Or 5 copies (if test/practice copy (D copy) & DR is to be continued)
 Unlimited distance between intermediate site and remote
 RPO of 0 for “A” site failure
– Zero RPO implies automation to ensure no production updates if
mirroring stops
 Potential RPO of 3-5 seconds for “A” and “B” twin site
failure
– Depends on workload and bandwidth between B and C
43
© 2005 IBM Corporation
Advanced Technical Support, Americas
When to use Metro/Global Mirror
 When two recovery sites are required
44
© 2005 IBM Corporation
Advanced Technical Support, Americas
Remote Mirror Comparisons
RPO
Distance
45
Global Mirror
for zSeries
(XRC)
Metro Mirror
(PPRC)
Global Copy
(PPRC-XD)
Global Mirror
> 0 (3-5 sec)
0
> 0 (hours)
> 0 (3-5 sec)
Unlimited
300 KM (fcp)
Unlimited
Unlimited
CKD
Yes
Yes
Yes
Yes
FB
Yes
Yes
Yes
Yes
CKD & FB
No
Yes
Yes
Yes
© 2005 IBM Corporation
Advanced Technical Support, Americas
Failover Processing (1)
 The secondary volume to which the command was issued
becomes a suspended PPRC primary
 The targeted volume gets a Change Recording bitmap
– Used to track changes that make it different from its partner
 Establishes a new relationship between the volume the
command was issued to and its PPRC primary volume
 Valid for both Metro Mirror and Global Copy
Failover
Secondary
Primary
GC or MM
Before
46
Primary
SUS
?
CR
SUS
or
active
SUS
or
active
Primary
After
0
1
0
1
0
0
0
1
0
0
0
0
1
0
© 2005 IBM Corporation
Advanced Technical Support, Americas
Failover Processing (2)
 No communication occurs between the two volumes
 Typically, failover is used when the relationship between the
volumes is suspended
– Consider a path failure – Primary goes suspended, Secondary does
not know anything is wrong, is not suspended
 If the relationship is NOT suspended when the command is
issued:
– The secondary volume WILL become a suspended primary
– The primary volume will BECOME a suspended primary when host I/O
is targeted to the volume – or a suspend command is issued to the
primary volume
– If neither I/O nor suspend occur, problems may arise during failback
47
© 2005 IBM Corporation
Advanced Technical Support, Americas
Failback Processing (1)
 The primary volume to which this command is issued has
it’s PPRC partner converted, if necessary, to a PPRC
secondary
 A path(s) must exist between the pairs
 The volume to which the command was issued
– Combines the partner bitmaps to get total “difference”
– begins to resync to its partner which becomes a PPRC secondary
volume – data begins to transfer
Example 1: Failback original Primary
Primary
48
0
1
0
1
0
0
0
1
0
0
0
0
1
0
Before
0
1
0
1
0
0
0
1
0
0
0
0
1
0
SUS
0
1
0
1
0
0
0
1
0
0
0
0
1
0
OOS
CR
SUS
Secondary
Primary
CR
Primary
CR
Failback
0
1
0
1
0
0
0
1
0
0
0
0
1
0
Secondary
Primary
GC
GC or MM
During
After
© 2005 IBM Corporation
Advanced Technical Support, Americas
Failback Processing (2)
 Similar to all PPRC establish operations, the resync begins
processing in “Global Copy” mode (First Pass) keeping
track of updates received during the resync (CR bitmap)
 The pairs will return to their original mode (Global Copy or
Metro Mirror) at the conclusion of the resync operation
 Failover and Failback are applicable to all PPRC
relationships, not just Global Mirror, as we will see in later
lectures and labs
Example 2: Failback original Target
Failback
49
0
1
0
1
0
0
0
1
0
0
0
0
1
0
Before
CR
CR
SUS
0
1
0
1
0
0
0
1
0
0
0
0
1
0
SUS
Primary
Secondary
GC
During
0
1
0
1
0
0
0
1
0
0
0
0
1
0
0
1
0
1
0
0
0
1
0
0
0
0
1
0
Secondary
Primary
GC or MM
After
OOS
CR
Primary
Primary
© 2005 IBM Corporation
Advanced Technical Support, Americas
Failover and Failback Command Parameters
 For both failover and failback, the Primary and Secondary
parameters must reflect the “new direction” of the copy
operation
Serial
85551
A
Serial
ABC2A
Serial
85551
Serial
ABC2A
B
A
B
Secondary
Primary
Primary
Primary
To issue FAILOVER to B:
To issue FAILBACK to A:
CESTPAIR FO DEVN(B) PRI(ABC2A) SEC(85551)
failoverpprc –dev abc2a –remotedev 85551 b:a
CESTPAIR FB DEVN(A) PRI(85551) SEC(ABC2A)
failbackpprc –dev 85551 –remotedev abc2a a:b
50
© 2005 IBM Corporation
Advanced Technical Support, Americas
Addendum
 Management Tools
 Establish Paths
 Establish Pairs
 FREEZE/RUN
 Resynchronizing Pairs
 Path and Pair Status
 Line speeds
 References
51
© 2005 IBM Corporation
Advanced Technical Support, Americas
Management Tools
Runs on
z/OS
Runs on
Open
Server
Manages CKD
Manages FB
TSO
Yes
No
Yes
Yes (1)
API
Yes
No
Yes
Yes (1)
ICKDSF
Yes
No
Yes
No
DS CLI
No
Yes
Yes
Yes
TPC for R
No
Yes
Yes
Yes
GDPS
Yes
No
Yes
Yes (1)
Note:
1. A CKD unit address (and host UCB) must be defined in the same DS8000/DS6000
server against which host I/O may be issued to manage FB LUNs.
52
© 2005 IBM Corporation
Advanced Technical Support, Americas
Establish Paths
 DS CLI
mkpprcpath –dev storage_image_id –remotedev storage_image_id –remotewwnn wwnn
–srclss source_LSS_ID –tgtlss target_LSS_ID source_port_ID:target_port_ID
 ICKDSF
PPRCOPY ESTPATH UNIT(ccuu) FCPPATHS(X’aaaabbbb’) PRIMARY(ssid,ser#)
SECONDARY(ssid,ser#) LSS(X’pp’,X’ss’) WWNN(pwwnn,swwnn)
 TSO
CESTPATH DEVN(device_number) PRIM(ssid wwnn lss) SEC(ssid wwnn lss)
LINK(aaaabbbb)
53
© 2005 IBM Corporation
Advanced Technical Support, Americas
Establish Pairs
 DS CLI
mkpprc –dev storage_image_ID –remotedev storage_image_ID –type gcp –mode
full SourceVolumeId:TargetVolumeId
 ICKDSF
PPRCOPY ESTPAIR DDNAME(dname) PRIMARY(ssid,ser#,cca)
SECONDARY(ssid,ser#,cca) LSS(X’pp’,X’ss’) MODE(COPY) OPTION(XD)
 TSO
CESTPAIR DEVN(device_number) PRIM(ssid serialno cca lss) SEC(ssid serialno
cca lss) MODE(COPY) OPTION(XD)
54
© 2005 IBM Corporation
Advanced Technical Support, Americas
FREEZE
 DS CLI
freezepprc –dev storage_image_ID –remotedev storage_image_ID
Source_LSS_ID:Target_LSS_ID
 ICKDSF
PPRCOPY FREEZE DDNAME(dname) PRIMARY(ssid,ser#) SECONDARY(ssid,ser#)
LSS(X’pp’,X’ss’)
 TSO
CGROUP DEVN(device_number) PRIM(ssid serialno lss)
SEC(ssid serialno lss) FREEZE
55
© 2005 IBM Corporation
Advanced Technical Support, Americas
RUN
 DS CLI
unfreezepprc –dev storage_image_ID –remotedev storage_image_ID
Source_LSS_ID:Target_LSS_ID
 ICKDSF
PPRCOPY RUN DDNAME(dname) PRIMARY(ssid,ser#) SECONDARY(ssid,ser#)
LSS(X’pp’,X’ss’)
 TSO
CGROUP DEVN(device_number) PRIM(ssid serialno lss)
SEC(ssid serialno lss) RUN
56
© 2005 IBM Corporation
Advanced Technical Support, Americas
Resynchronizing Pairs
 DS CLI
resumepprc –dev storage_image_ID –remotedev storage_image_ID –type gcp
SourceVolumeId:TargetVolumeId
 ICKDSF
PPRCOPY ESTPAIR DDNAME(dname) PRIMARY(ssid,ser#,cca)
SECONDARY(ssid,ser#,cca) LSS(X’pp’,X’ss’) MODE(RESYNC) OPTION(XD)
 TSO
CESTPAIR DEVN(device_number) PRIM(ssid serialno cca lss) SEC(ssid
serialno cca lss) MODE(RESYNC) OPTION(XD)
57
© 2005 IBM Corporation
Advanced Technical Support, Americas
Path Status DS CLI
lspprcpath –dev storage_image_ID Source_LSS_ID
58
© 2005 IBM Corporation
Advanced Technical Support, Americas
Pair Status DS CLI
lspprc –dev storage_image_ID –remotedev storage_image_ID –l
SourceVolumeId:TargetVolumeId
59
© 2005 IBM Corporation
Advanced Technical Support, Americas
Path Status ICKDSF
PPRCOPY QUERY DDNAME(dname) PATHS
ICK00700I DEVICE INFORMATION FOR 5C11 IS CURRENTLY AS FOLLOWS:
PHYSICAL DEVICE = 3390
STORAGE CONTROLLER = 3990
STORAGE CONTROL DESCRIPTOR = E9 DEVICE DESCRIPTOR = 0A
ADDITIONAL DEVICE INFORMATION = 4A001B35
TRKS/CYL = 15, # PRIMARY CYLS = 150
ICK04030I DEVICE IS A PEER TO PEER REMOTE COPY VOLUME
QUERY REMOTE COPY – PATHS
PRIMARY CONTROL UNIT INFORMATION
SERIAL
WORLD WIDE
NUMBER SSID LSS NODE NAME
------- ---- --- ---------------14940
9400 00
5005076300C01F4C
SECONDARY CONTROL UNIT INFORMATION
SERIAL
WORLD WIDE
NUMBER SSID LSS NODE NAME
------- ---- --- ---------------14940
9401 01
PATHS:
SERIAL
WORLD WIDE
NUMBER SSID LSS NODE NAME
PATH SAID DEST S*
------- ---- --- ---------------- ---- ---- ---- -1ST:
60
14940
9401
01 5005076300C01F4C
1
002C 00AC 13
2
00AC 002C 13
2ND: ....... .... ... ................
.... ....
00
3RD: ....... .... ... ................
.... ....
00
4TH: ....... .... ... ................
.... ....
00
© 2005 IBM Corporation
Advanced Technical Support, Americas
Pair Status ICKDSF
PPRCOPY QUERY UNIT(ccuu)
ICK00700I DEVICE INFORMATION FOR 5C11 IS CURRENTLY AS FOLLOWS:
PHYSICAL DEVICE = 3390
STORAGE CONTROLLER = 3990
STORAGE CONTROL DESCRIPTOR = E9
DEVICE DESCRIPTOR = 0A
ADDITIONAL DEVICE INFORMATION = 4A001B35 TRKS/CYL = 15, # PRIMARY CYLS = 150
ICK04030I DEVICE IS A PEER TO PEER REMOTE COPY VOLUME
QUERY REMOTE COPY - VOLUME
(PRIMARY) (SECONDARY)
SSID CCA
DEVICE LEVEL
STATE
PATH STATUS
------
---------
-------------- ----------- -
5C11
PRIMARY DUPLEX
ACTIVE
SSID CCA
SER # LSS SER # LSS
----------
-----------
9400 11
9401 11
14940 00
14940 01
PATHS SAID/DEST STATUS DESCRIPTION
----- --------- ------ ---------------2 002C 00AC 13 ESTABLISHED FIBRE CHANNEL PATH
00AC 002C 13 ESTABLISHED FIBRE CHANNEL PATH
ICK02206I PPRCOPY QUERY FUNCTION COMPLETED SUCCESSFULLY
ICK00001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0
61
© 2005 IBM Corporation
Advanced Technical Support, Americas
Path Status TSO
CQUERY DEVN(device_number) PATHS
62
© 2005 IBM Corporation
Advanced Technical Support, Americas
Pair Status TSO
CQUERY DEVN(device_number)
63
© 2005 IBM Corporation
Advanced Technical Support, Americas
Pair Status ISMF
To Further Limit the Generated List, Specify a Single Value or List
of Values
in any of the following:
Rel Op
Value
Value
Value
Value
------ --------- --------- ---------
--------Cache Fast Write Status
. .
CF Volume Status . . . . . .
Dasd Fast Write Status . . .
Duplex Status
. . . . . . .
Index Status . . . . . . . .
Physical Status
Read Cache Status
. . . . . .
Shared Dasd
. . . . .
. . . . . . . .
Use Attributes . . . . . . .
64
© 2005 IBM Corporation
Advanced Technical Support, Americas
Pair Status ISMF
Use ENTER to continue, END to exit Help.
HELP--------------------
DUPLEX STATUS (Page 2 of 2)
-------------------HELP
COMMAND ===>
65
PPRIMARY
The volume is primary of a PPRC pair.
PSECNDRY
The volume is secondary of a PPRC pair.
PPRI-PEN
The volume is primary of a PPRC pair in the process of being established.
PSEC-PEN
The volume is secondary of a PPRC pair in the process of being established.
PPRI-SUS
The volume is primary of a PPRC pair that is suspended.
PSEC-SUS
The volume is secondary of a PPRC pair that is suspended.
PPRI-FAI
The volume is primary of a PPRC pair in fail status.
PSEC-FAI
The volume is secondary of a PPRC pair in fail status.
© 2005 IBM Corporation
Advanced Technical Support, Americas
Line Speeds
66
Mbps
ApproximateMB/Sec
Equivalent T1
lines
T1
1.544
.1544
1
T3
44.746
4.4746
28
OC3
155
15.5
100
OC12
622
62.2
400
OC48
2488
248.8
1600
© 2005 IBM Corporation
Advanced Technical Support, Americas
IBM Copy Services Technologies – DS6K/DS8K
FlashCopy

Point in time copy


Available on:





Synchronous mirroring

Asynchronous mirroring
Available on:

Available on:




DS8000, DS6000, ESS
SAN Volume Controller
DS4000
N Series
Within
Storage
System
Global Mirror
Metro Mirror
DS8000, DS6000, ESS
SAN Volume Controller
DS4000
N Series




DS8000, DS6000, ESS
SAN Volume Controller
DS4000
N Series
Metro / Global Mirror

Three site synchronous and
asynchronous mirroring

Available on:


DS8000, ESS
N Series
Primary
Metro distance
Primary
Primary
Metro
Site A
<300km
Site A
Site A
Site B
Site B
Out of
67
Region
Out of
Site
Region
B
Site C
© 2005 IBM Corporation
Advanced Technical Support, Americas
Copy Services Matrix
Device
Is
GMz10
GMz10
(XRC)
Primary
(XRC)
Secondary
No
Metro Mirror
or Global
Copy Primary
Metro Mirror
or Global
Copy
Secondary
Global
Mirror
Primary
Global
Mirror
Secondary
FlashCopy
Source
FlashCopy
Target
Incremental
FLC Source
Incremental
FLC Target
Concurrent
Copy
Source
Yes11
Yes
No
Yes
No
Yes
Yes
No
No
Yes
Yes11
No
Yes
No5
Yes
No5
Yes
No5
Yes
No5
Yes
Metro Mirror
or Global
Copy
Primary
Yes
Yes
No
Yes 1
No6
Yes1
Yes
Yes
Yes
No6
Yes
Metro Mirror
or Global
Copy
Secondary
No
No5
Yes 1
No
Yes1
No
Yes
Yes8
Yes
Yes
No
Global
Mirror
Primary
Yes
Yes
No6
Yes1
No
No
Yes
Yes
Yes
Yes
Yes
Global
Mirror
Secondary
No
No5
Yes1
No
No
No
Yes
Yes8
Yes9
No
No
FlashCopy
Source
Yes
Yes
Yes
Yes
Yes
Yes
Yes 3,4
Yes 4
Yes3
No
Yes
FlashCopy
Target
No
No5
Yes 2
No
No
No
Yes 4
Yes4
No
No
No
Incremental
FLC Source
No7
Yes
Yes
Yes
Yes
Yes9
Yes
No
No
No
Yes
Incremental
FLC Target
No7
No5
Yes 2
No
No
No
No
No
No
No
Yes
Concurrent
Copy
Source
Yes
Yes
Yes
No
Yes
No
Yes
Yes
Yes
Yes
Yes
May
Become
GMz10
(XRC)
Primary
GMz10
(XRC)
Secondary
68
© 2005 IBM Corporation
Advanced Technical Support, Americas
Notes:
1.
Only in a Metro/Global Copy (supported on ESS) or a Metro/Global Mirror Environment (supported
on ESS and DS8000).
2.
FlashCopy V2 at LIC 2.4.0 and higher on ESS800 (DS6000 and DS8000 utilize FlashCopy V2 by
default).
–
–
–
You must specify the proper parameter to perform this
Metro Mirror primary will go from full duplex to copy pending until all of the flashed data is transmitted to remote
Global Mirror primary cannot be a FlashCopy target
3.
FlashCopy V2 Multiple Relationship.
4.
FlashCopy V2 Data Set FlashCopy (only available for z/OS volumes).
5.
The Storage Controller will not enforce this restriction, but it is not recommended.
6.
A volume may be converted between the states Global Mirror primary, Metro Mirror primary and
Global Copy primary via commands, but it two relations cannot exist at the same time (i.e. multitarget).
7.
GMz (XRC) Primary, Global Mirror Secondary, Incremental FlashCopy Source and Incremental
FlashCopy Target all use the Change Recording Function. For a particular volume only one of these
relationships may exist.
8.
Updates to the affected extents will result in the implicit removal of the FlashCopy relationship, if
the relationship is not persistent.
9.
This relationship must be the FlashCopy relationship associated with Global Mirror – i.e. there may
not be a separate Incremental FlashCopy relationship.
10. Global Mirror for zOS (GMz) is supported on ESS and DS8000
11. In order to ensure Data Consistency, the XRC Journal volumes must also be copied.
69
© 2005 IBM Corporation
Advanced Technical Support, Americas
References
 SC26-7916 DS8000 Command-Line Interface User’s Guide
 GC26-7922 DS6000 Command-Line Interface User’s Guide
 SC35-0428 DFSMS Advanced Copy Services
 SG24-6788 IBM System Storage DS8000 Series: Copy Services in Open Environments
 SG24-6787 IBM System Storage DS8000 Series: Copy Services with IBM System z
 SG24-6783 IBM System Storage DS6000 Series: Copy Services in Open Environments
 SG24-6782 DS6000 Series: Copy Services with IBM System z Servers
 Performance White Paper
– http://w31.ibm.com/sales/systems/portal/_s.155/254?navID=f320s260&geoID=All&prodID=System%20Storage&do
cID=ditlDS8000PerfWPPower5
 DS8000/DS6000 Copy Services: Getting Started
– WP100905
– http://www03.ibm.com/support/techdocs/atsmastr.nsf/WebDocs/?Search&Query=[HTMLDocumentName=WM*]+AN
D+(burger)&Start=1&Count=50&SearchOrder=1&SearchMax=10000
70
© 2005 IBM Corporation
Advanced Technical Support, Americas
Trademarks
The following terms are trademarks or registered trademarks of the IBM Corporation in either the United States, other countries or both.
AIX
AIX 5L
BladeCenter
Chipkill
DB2
DB2 Universal Database
DFSMSdss
DFSMShsm
DFSMSrmm
Domino
e-business logo
Enterprise Storage Server
ESCON
eServer
FICON
FlashCopy
GDPS
Geographically Dispersed
Parallel Sysplex
HiperSockets
i5/OS
IBM
IBM eServer
IBM logo
iSeries
Lotus
ON (button device)
On demand business
OnForever
OpenPower
OS/390
OS/400
Parallel Sysplex
POWER
POWER5
Predictive Failure Analysis
pSeries
S/390
Seascape
ServerProven
System z9
System p5
System Storage
Tivoli
TotalStorage
TotalStorage Proven
TPF
Virtualization Engine
X-Architecture
xSeries
z/OS
z/VM
zSeries
Linear Tape-Open, LTO, LTO Logo, Ultrium logo, Ultrium 2 Logo and Ultrium 3 logo are trademarks in the United States and other countries of
Certance, Hewlett-Packard, and IBM.
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States and/or other countries.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States and/or other countries.
Intel, Intel Inside (logos), MMX and Pentium are trademarks of Intel Corporation in the United States and/or other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Linux is a trademark of Linus Torvalds in the United States and other countries.
Other company, product, or service names may be trademarks or service marks of others.
71
© 2005 IBM Corporation