DS8000/DS6000 Copy Services: Getting Started This document can be found on the web, www.ibm.com/support/techdocs Search for document number WP100905 under the category of “White Papers. Version 1.1 Version Date: November 30, 2006 San Jose Advanced Technical Support Center Charlie Burger Consulting I/T Storage Specialist Special Notices This document reflects the IBM San Jose Advanced Technical Support center’s understanding on many of the questions asked about Copy Services on IBM DS8000 and DS6000 storage. It was produced and reviewed by the members of the IBM San Jose ATS. This document is presented “As-Is” and IBM does not assume responsibility for the statements expressed herein. It reflects the opinions of the IBM San Jose ATS. These opinions are based on several years of joint work with the IBM Storage group. If you have questions about the contents of this document, please direct them to the IBM San Jose ATS. Trademarks The following terms are registered trademarks of International Business Machines Corporation in the United States and/or other countries: AIX, AS/400, DB2, IBM, z/OS, OS/390, OS/400, Parallel Sysplex, RS/6000, S/390, System/390. Microsoft, Windows, Windows NT and the Windows logo are registered trademarks of Microsoft Corporation in the United States and/or other countries. Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States and/or other countries. Other company, product and service names may be trademarks or service marks of others. Table of Contents Introduction and Acknowledgments ……………………………………. 5 Chapter 1: Remote Mirror and Copy……………………………………... 6 Recovery Point Objective (RPO) and Recovery Time Objective (RTO) …6 Dependent Writes and Consistent Data ………………………………….. 6 Asynchronous vs Synchronous Processing………………………. ……… 7 What are the Solutions? …………………………………….…………….. 8 Metro Mirror …………………………………………………….... 8 Global Copy ………………………………….…………………… 9 Global Mirror ………………………………………….………….. 9 How does each solution maintain consistency …………………………… 10 Metro Mirror ……………………………………………………… 10 Global Copy ………………………………………………………. 11 Global Mirror ……………………………………………………... 12 Choosing the correct solution …………………………………………….. 13 Sizing the solution ……………………………………………………….... 15 Metro Mirror ………………………………………………………. 15 Global Copy ……………………………………………………..… 16 Global Mirror ……………………………………………………… 16 Selecting the management tool ……………………………………………. 18 Setting up PPRC pairs ………………………………………….………….. 20 Metro Mirror …………………………….…………………………. 20 Global Copy …………………………………………………….….. 22 Global Mirror …………………………………………….………… 23 Failover/Failback explanation ……………………………..……………….. 23 Planned outage ………………………………………………….………….. 25 Metro Mirror ……………………………………………….………. 25 Global Copy ………………………………….…………………….. 28 Global Mirror …………………………………….………………… 30 Unplanned outage …………………………………………….……………. 34 Metro Mirror ………………………………….……………………. 34 Global Copy …………………………………………………….….. 36 Global Mirror …………………………………….………………… 39 Primary secondary & tertiary placement ..………………………………….. 42 DS8000/DS6000 and ESS 800 ………………………………….………….. 43 Chapter 2: FlashCopy …..………………………………………..…………. 44 What is FlashCopy? ………………………..……………………….. 44 How does it work?...................................... ………………………… 44 What types of FlashCopy are there? …………………………….…. 45 Full volume FlashCopy ……………………….……………. 45 Background nocopy to cop ………………………………… 46 Persistent FlashCopy ………………………………………. 46 Data set FlashCopy ………………………………………… 46 Multiple relationships ……………………………………… 46 Incremental FlashCopy …………………………………….. 46 Consistency group FlashCopy ……………………………… 47 © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 -3- Inband FlashCopy …………………………..…………….. 47 FlashCopy, PPRC, and z/OS Global Mirror....…………… 48 Invoking FlashCopy Functions …………………………… 48 Source and Target Placement ……………………..……… 49 FlashCopy and z/OS ………………………………..…….. 49 FlashCopy and AIX …………………………………..…... 49 FlashCopy and Windows ……………………………..…... 50 Chapter 3: DS CLI ………………..……………………………………….. 54 DS CLI Intro ………………………………..…………….. 54 DS CLI Command Overview ………….………………..… 54 DS CLI Modes …………………………………………….. 56 DS CLI Profile …………………………………………….. 57 DS CLI Password File ……………………………….…….. 59 Collecting DS CLI Output ………………………….……… 61 Chapter 4: DS CLI Scripts and Batch Files …………………………..……. 63 Scripts ……………………………………………..……….. 63 Batch File ……….……………………………………..…… 65 Copy Services Matrix ……………..…………..…………… 69 © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 -4- Introduction, Notes about this document and/or Acknowledgments This document addresses the various Copy Services available with a DS6000 or DS8000, how to choose the one that best fits the user’s needs and some tips on how to implement the function. This document is not meant to be a comprehensive implementation guide, but rather a starting point for those looking to implement Copy Services. The Copy Services discussed in this paper are: Remote Mirror and Copy (RMC) Metro Mirror Global Copy Global Mirror Point in Time Copy (PTC) FlashCopy Copy Services not discussed in this paper are: Remote Mirror and Copy (RMC) Metro/Global Mirror Metro/Global Copy Remote Mirror for z/Series (RMZ) Global Mirror for zSeries (XRC) For detailed information on how to implement Copy Services, the following references can be used: SC35-0428 DFSMS Advanced Copy Services SC26-7616 IBM System Storage DS8000 Command-Line Interface User's Guide GC26-7922 IBM System Storage DS6000 Command-Line Interface User's Guide SG24-6787 The IBM TotalStorage DS8000 Series: Copy Services with IBM eSeries zSeries SG24-6788 The IBM TotalStorage DS8000 Series: Copy Services in Open Environments SG24-6782 The IBM TotalStorage DS6000 Series: Copy Services with IBM eSeries zSeries SG24-6783 The IBM TotalStorage DS6000 Series: Copy Services in Open Environments Thank you to the following reviewers and contributors: Bob Kern, Senior IT Specialist and IBM Master Inventor Henry Sautter, Consulting IT Specialist Steve West, IBM Consulting IT Specialist John Sing, Senior Consultant - Business Continuity Strategy and Planning Nick Clayton, Certified Consulting IT Specialist Brian Sherman, IBM Senior Consulting Storage Specialist Jim Sedgwick, Senior IT Specialist David Sacks, W-W Sales Support - Enterprise-Class Disk System Competition © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 -5- Chapter 1: Remote Mirror and Copy Recovery Point Objective (RPO) and Recovery Time Objective (RTO) When designing a disaster recovery solution, two key objectives must be considered during the planning process, the Recovery Point Objective (RPO) and the Recovery Time Objective (RTO). The RPO is the amount of data that has been lost due to a disaster and can be in the range from zero to hours. RPO will be discussed in more detail when each Remote Mirror and Copy solution is discussed. RTO is the amount of time from the point of the disaster to the applications being back up and running. This has multiple components including the recovery of data, server, network and workload. RTO is determined by the customer unless they implement end to end automation such as GDPS that deals with the recovery of each of the components allowing the applications to once again run. If you have to perform a database recovery at the secondary site, the amount of time it would take to get to the point where you could begin running your recovery systems could take hours, possibly even days. In contrast, if you can perform a database restart at the secondary site, it’s possible that the recovery system could be up and running in a matter of a couple of hours…. or less! Therefore, having data at the secondary site that allows a database restart is highly desirable and is made possible if the data is consistent! Dependent Writes and Consistent Data What is consistent data? Consistency means that the order of dependent writes is maintained. Dependent writes means that the start of one write operation is dependent upon the completion of a previous write to a volume in either the same disk system frame or a different disk system. Dependent writes are the basis for providing consistent data for copy operations. How each Remote Mirror and Copy implementation provides consistency at the secondary site will be covered when that implementation is discussed. Here are some examples of dependent writes and how consistency is maintained in a remote mirroring environment. The figures were taken from a presentation developed by John Sing. The first figure shows a series of dependent writes. In this example, the database log is updated first (possibly to show that a transaction has started), after that write completes, the database is updated, and finally, after the successful update of the database, the log is once again updated to indicate that the transaction has successfully completed. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 -6- Figure 1 The second figure shows an example where the local database volume has lost communication with the secondary site meaning that if the database is updated, the changes will not be transmitted to the secondary site. Figure 2 If the dependent write to update the log after the completion of the database update is held until all volumes with dependent data stop transmitting data to the secondary site, consistency will be maintained. Once transmission to the secondary site of all dependent updates is stopped, updates to the primary volumes can continue. The log and the database at the secondary site are in sync, therefore, consistent. When the database is restarted, the in flight transaction will be backed out. The key here is that once one volume in a set of dependent volumes is incapable of having its updates transmitted to the secondary site by the primary disk system, all of the other dependent volumes must stop having their updates transmitted to the secondary site. Also note that no timestamps are required to maintain the consistency at the secondary site. Asynchronous vs Synchronous Processing There are two methods for transmitting data from the primary site to the secondary site, asynchronous processing and synchronous processing. Asynchronous processing • Signaling of I/O complete (FB volumes) or CE/DE (Channel End/Device End for CKD volumes) is done immediately and the data transmitted shortly thereafter meaning that the © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 -7- distance between primary and secondary has little impact upon the response time of the primary volume Examples are Global Copy, Global Mirror, and Global Mirror for zSeries RPO > 0 Synchronous processing I/O complete or DE is not returned until the update is transmitted to the secondary site and acknowledged by the secondary Maximum supported distance between a Metro Mirror primary and secondary site using fibre links is 300 KM o An RPQ can be requested for longer distances Since the I/O complete or DE is not returned until the secondary site acknowledges successful receipt of the data, I/O response time will be elongated o How much depends upon the distance between the two sites. An example is Metro Mirror RPO = 0 What are the solutions? Metro Mirror Description Metro Mirror uses synchronous processing and is designed to provide real-time mirroring of logical volumes within a physical disk system or between two physical disk systems. Since Metro Mirror is synchronous, the RPO for a Metro Mirror environment is 0. The sites using Metro Mirror must conform to the following conditions: Can accept some performance impact to application write I/O operations at the primary location. If two physical disk systems are involved, they cannot be more than 300 KM apart with Fibre connectivity without an RPQ. How it works A Metro Mirror data copy to the recovery storage disk system is synchronous with the primary volume’s I/O operation. This means that: The primary system server writes data to a primary volume’s disk system The data is transferred to cache and nonvolatile storage (NVS) The disk system sends Channel End complete status to the host. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 -8- The primary site storage control returns Device End status to the primary system when the transfer to the recovery site storage control cache and NVS is complete. The primary system notifies the application program that the operation is complete. Note: The Metro Mirror copy function does not consider the primary system disk write operation complete until the data that is sent to the recovery storage disk system has received acknowledgement that the data was successfully received by the secondary disk system. Each primary disk system write to the recovery disk system causes an increase to the primary system response time. Global Copy Description Global Copy uses asynchronous processing which allows unlimited distance between the primary and secondary sites as well as having minimal impact on application I/O operations. Global Copy is designed for those sites that conform to the following conditions: Need a disaster recovery solution with a recovery point object (RPO) of many hours, or even several days. Need the flexibility to use both synchronous and asynchronous data transfers, especially when bandwidth restrictions are a consideration Sufficient bandwidth is still needed for Global Copy and a bandwidth sizing should still be performed How it works When Global Copy is active, the disk system captures information about updates to the primary volumes and periodically sends those updates to the secondary. As a result, there is no guarantee that application dependent writes are transferred in the same sequence as they were applied to the primary volume and therefore, the data at the secondary site is inconsistent. Manual intervention is required to create consistent data. Global Copy is a wonderful tool for data migration. Global Mirror Description A Global Mirror environment addresses the issue of manual intervention to create consistent data at the remote site when transferring data asynchronously. Global Mirror gives you the ability to establish Global Copy pairs over a long distance and maintain consistent versions of the © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 -9- data at the disaster recovery site automatically without suffering any significant local application performance degradation. How it works To accomplish the above, the addition of a Global Mirror Master Session in the hardware configuration has been made. The master session manages the creation of consistent copies of the primary local volumes at the secondary remote site at user specified intervals. A Global Mirror session consists of a single master LSS, one or more optional, subordinate subsystems (up to the maximum allowed by the latest microcode release), and one or more Global Copy primary volumes. At the time a session is started, the master and parameters of the session are specified. The session should have been previously identified to LSSs that are going to participate in the session. These LSSs may reside in the master or in subordinate subsystems. The session will be inactive until volumes are added to the session. As of 11/30/2006, a total of 8 physical subsystems in any combination of primary and secondary subsystems are supported. An RPQ can be requested to add more subsystems to the session. How does each solution maintain consistency? As noted at the beginning of the chapter, maintaining the order of dependent writes is what provides consistency at the secondary site which in turn allows for a database restart rather than requiring a database recovery. Metro Mirror To maintain the order of dependent writes in a Metro Mirror environment, you begin by specifying consistency group on the establish path command. Normally, when an error occurs with a member of a remote mirror and copy volume pair, the storage unit places the volume in a suspended state. However, if the volume participates in a consistency group, it will also return extended long busy to the server issuing the I/O. The host I/O will be queued and then reissued when the extended long busy is released. When the volume is suspended and returning long busy to the hosts, dependent writes that are scheduled to follow the write on the volume experiencing the error are not issued. This status will last for a default of two minutes or until the RUN command is issued to the LSS. Automation should be used to detect the error condition, and while the writes to the volume experiencing the error are being held, automation can then issue the ‘FREEZE’ command to the other LSSs that have volumes with data that is dependent with the data on the volume experiencing the error. The FREEZE command will suspend the volumes in the LSS, delete the paths from that LSS, and the volumes in that LSS will return long busy until two minutes elapses or the RUN command is issued. Once all of the dependent LSSs have had the FREEZE command issued to them, the RUN command can be issued to the LSSs to resume updates. With automation, the process of issuing the © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 10 - FREEZE/RUN to all of the required LSSs should take a matter of seconds. At this point, the order of the dependent writes has been maintained, the pairs have been suspended, and the data at the secondary site is consistent. If this process is performed during a rolling disaster, it is possible to have data written at the production site that is not mirrored. The data at the secondary site is consistent, but any data written at the primary site written after the FREEZE will be lost. GDPS has the capability of stopping all writes during a rolling disaster meaning that the all the data at the primary site will be at the secondary site. This capability is known as FREEZE/STOP. The difference between the GDPS GO and STOP options are as follows: GO - Says that after the Data Freeze is completed, application I/O is release and permitted to continue against the suspended primary PPRC (MM) volumes. A subsequent site switch will then find lost data at the secondary site as the secondary site data was frozen at the time of the data freeze. STOP - Says that after the data freeze is completed that the GDPS automation code will place all processors at the primary site into a restart able Wait State. Therefore, all applications will be stopped and no data will be updated at the primary or secondary site until further action is taken. This option insures that NO Data Los will occur if a subsequent disaster happens. This solution requires some sort of automation to detect the error conditions and automatically issue the FREEZE/RUN commands to the appropriate LSSs. GDPS provides automation that will issue FREEZE/GO or FREEZE/STOP. Global Copy If Global Copy is being used to transmit data to the secondary site, it is up to the user to intervene and create a consistent copy. Global Copy transmits the data to the secondary site asynchronously. Within a device adapter, there are several tasks, each processing a volume and transmitting data, but there aren’t enough tasks to process all of the volumes at the same time. The number of tasks varies upon how much application work is being done on the device adapter. What this means is that the order of the updates is not maintained at the secondary site and therefore, the data is inconsistent. Since the order of the updates cannot be maintained, you should not code consistency group on the establish path command as it will provide no benefit. To create consistent data at the secondary site when using Global Copy, the user would: Quiesce the production applications Allow Global Copy to drain the primary volumes or possibly use the go-to-sync command Once the volumes are drained or in ‘duplex’ mode, suspend the pairs (FREEZE command can do this but remember it also deletes paths) Once the pairs are suspended: Resume application updates FlashCopy secondary volumes to tertiary volumes at the secondary site © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 11 - If necessary, reestablish the paths between primary and secondary sites Resume (resync) the primaries to secondaries Since a user wouldn’t be able to go through this process very often, possibly once a day, the RPO would have to be high for Global Copy to be considered as a disaster recovery solution. But, again, Global Copy is a great tool for data migration. Global Mirror Global Mirror provides the capability of creating consistent data at a secondary site that can be continental distances from the primary site and maintain a best case RPO of 3 to 5 seconds. Global Mirror uses Global Copy as its data transmission mechanism and since Global Copy uses asynchronous processing, it can go long distances with minimal impact on the application. But, if Global Mirror uses Global Copy to asynchronously transmit the data between the primary and secondary sites, how is consistent data maintained at the secondary site? The Global Mirror session will coordinate the formation of consistent data at the remote site at an interval defined by the user. This interval is called the ‘Consistency Group Interval Time’. If CGI time is specified as 0, Global Mirror will continually attempt to form consistency groups at the secondary site. The formation of a consistency group is a 3 step process: Coordination time Drain time FlashCopy secondary volumes to tertiary volumes at the secondary site Coordinate local units FlashCopy Relationships being established Drain Time CG Interval Time ... ... ... ... Coordination Time Let CG data drain to remote Record new writes in bitmaps but do not copy to remote All FlashCopy Relationships established Global Copy continually cycles through volume bitmaps copying changed data to remote mirror volumes Figure 3 © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 12 - The key to maintaining consistency in the Global Mirror environment is the Coordination step. In this step, a new bit map is created in addition to the Out-of-Sync (OOS) bit map. The OOS is used to record tracks that Global Copy needs to transfer and the additional bit map is only used during the formation of consistency groups. It records changes as tracks are drained from the OOS during Drain Time. As this additional bit map is being created, writes to that volume do not have I/O complete or CE/DE returned to the host. Reads proceed normally. This means that if there is a dependent write scheduled after the write that’s being held, that dependent write will not be issued. Once all of the volumes in the Global Mirror session have this additional bit map built, writes proceed normally and are recorded in the new bit map. The OOS now records consistent data since the order of dependent writes was maintained while the new bit map was being built and when the OOS is drained to the secondary site, the data on the secondary volumes is consistent. The secondaries are then FlashCopied to tertiary volumes. Once the FlashCopy completes, the new bit map is merged into the OOS and Global Copy continues transferring data. The paths in a Global Mirror environment do not, and should not have consistency group specified. Metro Mirror requires it because once a volume experiences an error, the resulting long busy allows for automation to detect the error and then FREEZE the other related LSSs. Global Mirror doesn’t depend upon detecting an error and then using automation to create consistency. Global Mirror creates consistency on its own at an interval determined by the user and during this process, Global Mirror maintains the order of dependent writes during the coordination step. Choosing the correct solution When attempting to select the appropriate Remote Mirror and Copy solution, several factors should be considered: RPO Distance CKD, FB or both I don’t list RTO since all of the solutions provide or can provide consistent data at the secondary site meaning that a database restart can be performed. RTO may be a consideration if using automation to start systems and subsystems at the secondary site is a possibility. RPO Distance CKD FB CKD & FB Global Mirror for zSeries (XRC) > 0 (3-5 sec) Unlimited Yes Yes No Metro Mirror (PPRC) 0 300 KM (fcp) Yes Yes Yes © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started Global Copy (PPRC-XD) > 0 (hours) Unlimited Yes Yes Yes Global Mirror > 0 (3-5 sec) Unlimited Yes Yes Yes 11/30/2006 - 13 - Here is a table taken from the DFSMS Advanced Copy Services manual that provides additional considerations when choosing a remote mirror solution: Notes: 1. When PPRC extended distance is enabled, data at the secondary site is not consistent with the primary site. 2. Adding channel extenders can extend the distance by sending the data across telecommunication lines. 3. For PPRC extended distance, the distance between storage controls can be greater than that supported with ESCON 4. The initial volume copy to the secondary device is asynchronous. Primary updates are transmitted asynchronously to secondary volumes when PPRC extended distance is enabled. 5. The initial volume copy to the local PPRC secondary is synchronous. Updates are transmitted asynchronously to the PPRC-XD secondary volumes. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 14 - 6. The global Mirror master will form FlashCopy Consistency Groups at a user-specified interval. The data at the recovery site is consistent to the last successful consistency group formed by the master. Sizing the solution Metro Mirror IBM Disk Magic for Windows is a modeling tool that helps estimate IBM disk subsystem performance. Some examples of when Disk Magic might be used: • Move the current I/O load to a different Disk Subsystem. • Merge the I/O load of multiple Disk Subsystems into a single one. • Insert a SAN Volume Controller in an existing disk configuration. • Increase the current I/O load. • Storage consolidation. • Increase the Disk Subsystem cache size. • Change to larger capacity disk modules. • Upgrade from SCSI to 1 or 2 Gbit fibre. • Use fewer or more Logical Unit Numbers (LUN). • Activate Peer-to-Peer Remote Copy Disk Magic uses iostat data for Open systems and RMF data for zSeries. One of the parameters defined for the model is the number of PPRC links: Figure 4 Once the data is processed, you will receive predictions for I/O response time. If the times are too high, you may have to add more PPRC links. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 15 - Network bandwidth is also a consideration. If the network bandwidth is too small to handle the traffic Metro Mirror passes on to it, application I/O response times will elongate since Metro Mirror is a synchronous solution and the transfer to the secondary site will take longer. Disk magic model will model performance vs number of FCP ports but, there is still the potential of limited bandwidth that could affect performance. You need to be very careful to size the network bandwidth so that the network does NOT introduce additional elongation of the response time. Mbps T1 T3 OC3 OC12 OC48 1.544 44.746 155 622 2488 Approximate MB/Sec .1544 4.4746 15.5 62.2 248.8 Equivalent T1 lines 1 28 100 400 1600 Dark fibre can support either ESCON or fcp protocols. ESCON links run at 17. MB/sec with a sustained rate of approximately 12-14 MB/sec. Fibre links can run 100-400 MB/sec depending upon the adapter with a sustained rate of approximately 80-320 MB/sec. Global Copy The sizing methodology used for Global Mirror could be used for Global Copy. It is important to remember that you need enough bandwidth to allow Global Copy to catch up during low activity times. If there is not enough bandwidth, the catch up time may be too long to meet service level agreements or might not be able to catch up at all. In addition, if the update rate is significantly higher than the available bandwidth, it is possible that the application could be effected. Global Mirror The following excerpt was taken from Nick Clayton’s Global Mirror Technical Whitepaper which can be found on Techdocs as entry WP100642. 9.1 Managing peak activity With asynchronous replication solutions that use a cache sidefile on the primary disk system and/or the secondary disk subsystem, only a finite amount of data can be held in the cache. If the mirror falls more than a certain amount of time behind, the replication solution must either pace the production applications or suspend the mirror. As Global Mirror does not use a cache sidefile, it is possible to deliberately under configure the bandwidth provided in order to reduce the total cost of the solution. If there are significant peaks then this cost saving could be considerable as the network costs are often a very large portion of the ongoing costs. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 16 - Figure 5 shows a typical profile for a production workload with a relatively low write rate during the online day and significant peaks at various points overnight. A bandwidth of around 15MB/s might be provided if a low RPO was required at all points during a 24 hour period. However if an increased RPO was acceptable at times if high write activity during the overnight period then we might potentially configure as little as 8MB/s of bandwidth which would reduce the network requirements by around 47%. The minimum bandwidth that can be provided must allow for the formation of consistency groups during at least some periods of the day and must allow for the environment to catch up after any significant periods of delay. This minimum bandwidth will be at least the average write bandwidth after taking account any savings due to duplicate write activity. Figure 5 Production workload profile and Global Mirror bandwidth options With Global Mirror, the production workloads will continue without significant impact during the peak period and the Global Mirror solution will transition automatically to Global Copy mode when a consistency group is not transmitted to the secondary site within the maximum drain time. When the peak activity has passed and Global Copy is once more able to drain the changed data in a timely manner the disk subsystems will transition automatically back to Global Mirror and resume the formation of consistency groups. At all points, a consistent restartable image of the production data will be available in the recovery location. 9.2 Bandwidth reduction As Global Mirror removes duplicate updates within a consistency group before sending them to the secondary location less data will tend to be transmitted than is written to the primary devices. The © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 17 - amount of savings here will be workload dependant as well as depending on the interval between consistency group formation. Figure 6 The graph in figure 6 shows the MB/s sent by Global Mirror compared to the production write activity for a particular customer environment over a period of a day. It also shows the percentage of the production writes that were sent by Global Mirror. The data is sorted by host activity from high to low. In this case, the environment is not bandwidth constrained and the RPO is generally of the order of a few seconds. Even in this case we can see that a reasonable percentage of the production writes do not need to be transmitted to the secondary location. Another factor we see is that for this workload when the activity is higher the percentage savings are actually lower as for this workload we have a higher proportion of sequential activity at these times. Sequential updates will not generally be duplicated and so the data that is written to the disk subsystem will have to be sent by Global Mirror. Selecting the Management Tool There are several tools and services available to manage the Remote Mirror and Copy environment. They all support Metro Mirror, Global Copy and Global Mirror. They are: TSO API (z/OS) ANTRQST macro documented in the DFSMS Advanced Copy Services manual ANTTREXX © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 18 - Samples can be found in SYS1.SAMPLIB(ANTXTSO) and SYS1.SAMPLIB(ANTIMAIM) ICKDSF ICKDSF Toolkit Shipped on a CD with each new microcode level DS CLI TPC for Replication GDPS TSO API ICKDSF DS CLI TPC for R GDPS Runs on z/OS Yes Yes Yes No No Yes Runs on Open Server No No No Yes Yes No Manages CKD Manages FB Yes Yes Yes Yes Yes Yes Yes (1) Yes (1) No Yes Yes Yes (1) Note: 1. A CKD unit address (and host UCB) must be defined in the same DS8000/DS6000 server against which host I/O may be issued to manage FB LUNs. The tool you select will probably be based upon what the user is most comfortable with. z/OS users tend to use TSO or the API (although ICKDSF is also used) and Open users will tend to prefer DS CLI or TPC for Replication rather than have TSO manage their FB LUNs. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 19 - Setting up PPRC pairs DS8000 and DS6000 ONLY support fcp PPRC links. But, to give you an idea of what benefits fcp links provide, here is a comparison of the ESCON and FCP technologies along with some rules when establishing paths and pairs: Figure 7 An establish path command overlays the existing path definitions rather than adding to them For example, if I had paths 1 & 2 established, and I wanted to add paths 3 & 4, I would issue the establish path command with all 4 paths. If I issued the establish path command with only 3 & 4, it would overlay the existing 1 & 2 and the only paths I would have would be 3 & 4. Also, remember that a PPRC primary volume can only have 1 secondary volume As mentioned previously, ESCON links run at 17. MB/sec with a sustained rate of approximately 1214 MB/sec. Fibre links can run 100-400 MB/sec depending upon the adapter with a sustained rate of approximately 80-320 MB/sec. Metro Mirror When setting up Metro Mirror pairs, the paths between the LSSs should have consistency group specified to allow automation to detect errors and issue the appropriate FREEZE/RUN commands. The actual establish pair command should specify the Metro Mirror option. Here are some examples: TSO © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 20 - The DEVN (device number) is any device in the LSS. The path is between LSSs, not devices, but the command itself is issued to a device. The PRIM and SEC specify SSID, WWNN, and LSS for the path. The link is the primary fibre port and secondary fibre port. CGROUP(YES) says that the path has the consistency group attribute. CESTPATH DEVN(X'7000') PRIM(X'2040' 500507630EFFFC01 X'00') SEC(X'2070' 500507630EFFFCA0 X'00') LINK(X'00000000',X'01000100') CGROUP(YES) This establish pair command is performing the initial establish of the pairs. The DEVN is the actual PPRC primary device. PRIM specifies SSID, serial number, cca, and LSS of the primary. SEC specifies SSID, serial number, cca and LSS of the secondary. OPTION(SYNC) is the default. It has been recommended that OPTION(XD) be used when initially establishing pairs and then after establish has gone through its first pass, change it to OPTION(SYNC) by simply issuing another CESTPAIR command. This has less effect on application I/O during the establish process. CESTPAIR DEVN(X'7000') PRIM(X'2040' ABC2A X'00' X'00') SEC(X'2070' AAVCA X'00' X'00') MODE(COPY) OPTION(SYNC) ICKDSF PPRCOPY ESTPATH UNIT(7000) – PRI(X’2040’,ABC2A) SEC(X’2070’,AAVCA) – LSS(X’00’,X’00’) – FCPPATHS(x’00000000’, x’01000100’) – WWNN(500507630EFFFC01,500507630EFFFCA0) CGROUP(YES) PPRCOPY ESTPAIR UNIT(7000) LSS(X’00’,X’00’) – PRI(X’2040’,ABC2A,X’00’) SEC(X’2070’,AAVCA,X’00’) – MODE(COPY) OPTION(SYNC) DS CLI © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 21 - dscli> mkpprcpath –dev ibm.1750-13abc2a -remotedev ibm.1750-13aavca -remotewwnn 500507630efffca0 -srclss 00 -tgtlss 00 -consistgrp I0000:I0000 I0100:I0100 dscli> mkpprc -remotedev ibm.1750-13aavca -type mmir -mode full 7000-707F:7000-707F Global Copy TSO Note that this establish path command does not specify CGROUP(YES) and the option for the establish pair is XD. CESTPATH DEVN(X'7000') PRIM(X'2040' 500507630EFFFC01 X'00') SEC(X'2070' 500507630EFFFCA0 X'00') CESTPAIR DEVN(X'7000') PRIM(X'2040' ABC2A X'00' X'00') SEC(X'2070' AAVCA X'00' X'00') MODE(COPY) OPTION(XD) ICKDSF PPRCOPY ESTPATH UNIT(7000) – PRI(X’2040’,ABC2A) SEC(X’2070’,AAVCA) – LSS(X’00’,X’00’) – FCPPATHS(x’00000000’, x’01000100’) – WWNN(500507630EFFFC01,500507630EFFFCA0) PPRCOPY ESTPAIR UNIT(7000) LSS(X’00’,X’00’) – PRI(X’2040’,ABC2A,X’00’) SEC(X’2070’,AAVCA,X’00’) – MODE(COPY) OPTION(XD) DS CLI dscli> mkpprcpath –dev ibm.1750-13abc2a -remotedev ibm.1750-13aavca -remotewwnn 500507630efffca0 -srclss 00 -tgtlss 00 I0000:I0000 I0100:I0100 dscli> mkpprc -remotedev ibm.1750-13aavca -type gcp -mode full 7000-707F:7000-707F © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 22 - Global Mirror Global Mirror uses Global Copy as its data transfer mechanism so the establish paths and pairs follow the same format as for Global Copy examples shown above. But Global Mirror requires additional setup above and beyond establishing the Global Copy paths and pairs. The steps for creating the Global Mirror session are: 1. Establish the paths without consistency group 2. Establish Global Copy pairs 3. After the establish pair first pass, establish FlashCopy pairs at secondary site between the B volumes and C volumes 4. Create a Global Mirror session 5. If there are multiple physical subsystems, establish Global Mirror control paths (they are PPRC links) between the Master and subordinates 6. Add the Global Copy primary volumes to the session 7. Start the Global Mirror session 4. Create Global Mirror Session 6. Add the Primary volumes to the session 2. Establish Global Copy Pairs Subordinate 3. Establish FlashCopy Pairs 1.Establish PPRC Paths FlashCop y Subordinate SAN . Master . FlashCop y FlashCop y 5. Establish control paths between Master and Subordinate 7. Start Global Mirror Session with start command sent to Master Failover/Failback explanation Failover Typically, failover is used when the relationship between the volumes is suspended and you expect changes to be made on the secondary. Consider a path failure, the primary becomes suspended and the secondary does not know anything is wrong and is not suspended. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 23 - If the relationship is NOT suspended when the command is issued, the secondary volume WILL become a suspended primary. It will have a bit map associated with it and changes to the volume will be recorded. There is NO communication between the two volumes. The primary volume will BECOME a suspended primary when host I/O is targeted to the volume but will remain as a non-suspended primary until either an I/O is issued to the volume or a suspend command is issued to the primary volume or a FREEZE command is issued to the primary LSS. If neither I/O nor suspend occur, problems may arise during failback expects suspended volumes. I suggest that FREEZE be issued before doing a failover. The failover command is valid for both Metro Mirror and Global Copy. Since Global Copy is used by Global Mirror to transmit data it is also valid in a Global Mirror environment. You would want to use it in a Global Mirror environment when you plan to do a fast reverse restore FlashCopy as that will make changes to the Global Mirror secondary. Failback The volume to which this command is issued has its PPRC pair at the other site converted, if necessary, to a PPRC secondary. The volume to which the command was issued kept track of changes in its bit map and then merges its partner’s bit map into its own bit map to determine which tracks to resync to its PPRC partner. Unlike the failover command, there is actual communication between the pairs and transmits data. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 24 - Planned outage procedure (for testing) Metro Mirror Prior to executing the planned outage procedure, initial setup must occur. Initial state of volumes Primary Secondary Simplex Simplex Establish paths Primary Secondary Simplex Simplex Establish pairs Primary Secondary Copy pending Copy pending Primary Secondary Duplex Duplex © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 25 - The planned outage procedure can now be executed. Issue FREEZE to the LSSs to SUSPEND the pairs (this will also delete the paths between the primary and secondary) and once all of the FREEZE commands have been executed, issue the RUN command to the all of the LSSs. Primary Suspended Suspended Secondary Suspended Issue FAILOVER to the secondary Primary Suspended Suspended Reestablish the paths between the primary and secondary Primary Secondary Primary Suspended FlashCopy the secondary (now in suspended primary mode) to a tertiary Primary Suspended Primary Suspended Tertiary Simplex You could simply FlashCopy the entire volume, use Incremental FlashCopy, or if you are using DS CLI, perform an Incremental FlashCopy with a sequence number. This will allow the verification that all copies are consistent and helps debug in the case where some of the FlashCopies fail. For example: mkflash –record –persist –seqnum nnnnnnnn © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 26 - On the next flash: mkflash –record –persist –seqnum nnnnnnnn+1 If some of the pairs are not the correct sequence num, query with -seqnum to get a list of the incorrect pairs. lsflash -seqnum 22222222 -s -hdr off > fc.list Then use fc.list to flash only those pairs. resyncflash -record -persist -seqnum 33333333 - <fc.list Issue FAILBACK to the production primary volumes Primary Copy pending Primary Copy pending Tertiary Simplex Primary Secondary Duplex Duplex Tertiary Simplex Test on the tertiary copies © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 27 - Global Copy Prior to executing the planned outage procedure, initial setup must occur Initial state of volumes Primary Secondary Simplex Simplex Establish paths Primary Secondary Simplex Simplex Establish pairs Primary Secondary Copy pending Copy pending The planned outage procedure can now be executed. Quiesce application I/O Primary Secondary Copy pending Copy pending Go-to-SYNC Primary Secondary Duplex Duplex © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 28 - FREEZE/RUN the LSSs Primary Suspended Secondary Suspended FlashCopy the secondary to tertiary Primary Suspended Secondary Suspended Tertiary Simplex You could use DS CLI to FlashCopy with sequence numbers just you could with the Metro Mirror scenario. Reestablish paths from primary to secondary Primary Suspended Secondary Suspended Tertiary Simplex Test on the tertiary copies Resync the primary to secondary using the XD option © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 29 - Primary Secondary Copy pending Copy pending Tertiary Simplex Global Mirror Prior to executing the planned outage procedure, initial setup must occur Initial state of volumes Primary Secondary Simplex Simplex Establish paths Primary Secondary Simplex Simplex Establish pairs Primary Secondary Copy pending A Copy pending B Add FlashCopy volume at secondary site Primary Copy pending A © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started Secondary Copy pending B FC C 11/30/2006 - 30 - Add primary volumes to Global Mirror session Primary Copy pending A Secondary FC Copy pending B C Global Mirror Session Start the Global Mirror session Primary Copy pending A Secondary FC Copy pending B C Global Mirror Session The planned outage procedure can now be executed. Pause the Global Mirror session Primary Copy pending A Secondary FC Copy pending B C Global Mirror Session FREEZE/RUN the Global Copy pairs Primary Copy pending A Secondary FC Suspended B C Global Mirror Session FAILOVER to B volumes Primary Suspended A Primary Suspended B FC C Global Mirror Session Fast Reverse Restore FlashCopy C > B © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 31 - Primary Suspended A Primary FC Suspended B C Global Mirror Session After background copy completes, reestablish FlashCopy B > C Primary Suspended A Primary FC Suspended B C Global Mirror Session If you have a D volume and intend to test on the D volume FlashCopy B > D so that testing can be done on the D volumes Primary Suspended A Primary FC Suspended B C Global Mirror Session D Note: While testing on the D volume, Global Mirror can be running concurrently maintaining the RPO of the data. If you do NOT have a D volume and plan to test on the B volume Begin testing on the B volume Primary Suspended A Primary Suspended B FC C Global Mirror Session Note: While testing on the B volume, the C volume would be the gold copy (a copy of consistent data). Be aware that while testing is being performed, the gold copy is aging and the RPO is growing. After testing is complete on the B volume or once the D volume has been created, reestablish paths between A and B © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 32 - Primary Suspended A Primary FC Suspended B C Global Mirror Session D FAILBACK A to B Primary Copy pending A Secondary FC Copy pending B C Global Mirror Session D Resume the Global Mirror session Primary Copy pending A Secondary Copy pending B FC C Global Mirror Session D © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 33 - Unplanned outage procedure Metro mirror Initial setup same as for planned outage Primary Secondary Duplex Duplex An error occurs Primary Suspended Secondary X Automation detects the error and FREEZE/RUNs all the related LSSs Primary Suspended Secondary Suspended FAILOVER issued to secondary volumes Primary Suspended Duplex Primary Suspended Recovery is started at remote site and changes are recorded in a bit map Primary Suspended © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started Primary Suspended 11/30/2006 - 34 - Primary site returns, establish paths from recovery site to production site and Failback to recovery site Secondary Copy pending Secondary Duplex Suspended Primary Duplex Primary Suspended Failover to the production site Primary Suspended Copy pending Shutdown recovery site and FREEZE the LSSs (suspending the pairs and deleting the paths) Secondary Primary Primary Suspended Establish paths from production site to recovery site and Failback production to recovery Primary Secondary Copy pending Copy pending Primary Secondary Duplex Duplex © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 35 - Restart workload at production site Primary Secondary Duplex Duplex Global Copy • Initial setup is same as for planned outage Primary Secondary Copy pending Copy pending Tertiary Simplex Error occurs Primary Suspended Secondary X Copy pending Tertiary Simplex © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 36 - FAILOVER to secondary Primary Suspended Primary Suspended Tertiary Simplex FlashCopy tertiary to secondary Primary Suspended Primary Suspended Tertiary Simplex Run recovery on the secondary volume (tertiary is gold copy) Primary Suspended Primary Suspended Tertiary Simplex © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 37 - Production site returns, establish path from recovery site to production site and Failback recovery to production Secondary Copy pending Primary Copy pending Tertiary Simplex Shutdown recovery system allowing the recovery system to drain to production, then issue FREEZE/RUNs to the recovery volumes Secondary Suspended Primary Suspended Tertiary Simplex Failover to production site Primary Suspended Primary Suspended Tertiary Simplex © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 38 - Establish paths from production to recovery system and FAILBACK production site to recovery site Primary Secondary Copy pending Copy pending Tertiary Simplex At some point, create another point in time copy of consistent data at the recovery site Global Mirror Initial setup same as for planned outage Primary Secondary Copy pending A FC Copy pending B C Global Mirror Session Error occurs Primary Suspended A Secondary X Copy pending B FC C Global Mirror Session © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 39 - Check status of FlashCopy pair to determine whether or not a commit or revert action is required. If Global Mirror session is active, terminate the session and issue Query relations to each PPRC secondary Primary Secondary Suspended A Revertible Sequence # All Equal At least one Equal None Mixed Equal Mixed Revertible equal and non-revertible equal but different from revertible Primary Specification on FlashCopy Withdraw Revert to all FlashCopy source volumes Commit to all FlashCopy source volumes Now withdraw required Revert to all FlashCopy source volumes Primary FC Suspended B C Perform Fast Reverse Restore FlashCopy C > B Primary Suspended A C FAILOVER to B volume Suspended A FC Copy pending B Primary FC Suspended B C Reestablish FlashCopy B > C after Fast Reverse Restore FlashCopy background copy completes Primary Suspended A © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started Primary Suspended B FC C 11/30/2006 - 40 - Start Recovery on B volumes Primary Suspended A Primary FC Suspended B C When production site returns, establish paths from recovery site to production site and Failback B>A Secondary Copy pending A Primary FC Copy pending B C Shutdown recovery system and allow recovery volumes (B) drain to the production volumes (A) Once the volumes have drained, suspend the pairs using FREEZE and issue Failover to the production volumes (A) Primary Suspended A FC C Establish paths between production (A) and recovery (B) sites and issue Failback to production (A) volumes Primary Copy pending A Primary Suspended B Secondary FC Copy pending B C Start the Global Mirror session (assumption is that session definition with volumes added still exists) Primary Copy pending A Secondary Copy pending B FC C Global Mirror Session Start production © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 41 - Primary secondary and tertiary placement The following was taken from Nick Clayton’s Global Mirror White Paper which can be found on Tech Docs as entry WP100642. . 9.4.1 Placement of devices on secondary disk subsystem In order to spread the load of any hotspots on the secondary RAID ranks it is recommended to split the FlashCopy source and target volumes on separate RAID ranks and device adapters within the same server. In this way if we have particular busy volumes the B volume RAID rank will be writing and reading and the C volume RAID rank will be writing rather than a single RAID rank performing all the activity. Figure 8 Spreading B and C volumes over different ranks If we have an additional D copy for testing purposes then we might either choose to place the D volumes on the same ranks as the B and C volumes. If we intended to perform a lot of heavy activity such as backup processing or stress testing then we might dedicate a rank to the D volumes in order to protect the Global Mirror environment from this workload. Figure 9 Options for placement of D volumes A customer might also choose to implement smaller faster drives for the B volumes but larger drives for the C and D volumes in order to take advantage of the cost efficiencies in doing so. This does © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 42 - result in the ranks containing the B volumes having a higher workload that the larger ranks and so might not be recommended for optimal performance. A better configuration might be to spread the B and C volumes over both sets of drives for optimal performance of both Global Mirror and the production workloads. The D volumes would then utilize the additional space on the larger drives. Figure 10 Volume layouts with two drive sizes on secondary disk subsystem There have also been further optimizations to FlashCopy that enhance the Copy on Write processing when the FlashCopy source and target are managed by the same storage server processor complex (cluster) on the disk subsystem. This means that the recommendation is to have a particular FlashCopy source and target both on either odd or even LSS. DS8000/DS6000 and ESS 800 A DS8000/DS6000 can participate in a PPRC relationship with an ESS 800. Other ESS models cannot participate in a PPRC relationship with the DS8000/DS6000. That is because the DS8000/DS6000 only supports fcp PPRC links and only the 800 model of the ESS has fcp PPRC link capability. The ESS 800 must be at microcode level 2.4.3.15 or higher. The PPRC commands must be invoked using DS CLI, ESS 800 Copy Services cannot create tasks that include DS8000/DS6000 boxes. DS CLI can be used for ESS 800 copy services but not ESS 800 configuration. This removes the need to define tasks on the ESS. Use one of the Copy Services Servers (A or B) IP addresses as the management console address. The PPRC source has to be in the same Copy Services Domain as the server but the target does not need to be within the same domain. If you wish to use DS CLI to invoke FlashCopy on an ESS, the ESS where you wish to perform FlashCopy must be a client of the server or the server itself. When specifying the serial number in a DS CLI command, the serial number for a DS8000/DS6000 is IBM.2107-ccsssss or IBM.1750-ccsssss whereas the ESS is IBM.2105-sssss. There is no country code used in the ESS specification. When configuring a DS8000/DS6000 that will be participating in a PPRC relationship with an ESS 800, specify –type ess on the mkfbvol DS CLI command. This is not an issue with CKD volumes. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 43 - Chapter 2: FlashCopy What is FlashCopy? FlashCopy is an “instant” T0 (Time 0) copy where the source and target volumes are immediately available for processing with full read/write access. It is a hardware solution that is invoked by software. FlashCopy can be invoked by DFSMSdss, TSO, ICKDSF, z/OS API, DS CLI, and the DS Storage Manager (GUI). When you request a copy from a source to a target, a relationship is created between the volumes. Once the relationship has been created, the target volume is available for processing. The FlashCopy establish process is extremely fast on the DS8000. For example, 256 3390-3 volumes in under a second. The FlashCopy is a set of tracks that can consist of an entire volume, a data set, or just a selected set of tracks. How does it work? When a FlashCopy is requested, you have the option of specifying background copy or nocopy. Background copy will result in all of the tracks being physically copied from the source volume to the target volume. Once all of the tracks have been copied, the relationship is withdrawn. You don’t have to wait for the background copy to complete before using the target volume. It can be used as soon as the relationship is established. Background nocopy will result in only those tracks that are updated on the source volume being physically copied to the target volume. This background copy by demand maintains the T0 copy on the target. If all of the source tracks are updated, the relationship will be withdrawn. Attempts to read/write data on the target that have already been copied from the source volume proceed as normal. That is, read from the target. Attempts to read a target track not already copied are intercepted and the data obtained from the source volume. Even though the data is obtained from the source, it appears to the host to have been read from the target. Reads Source Target Figure 5 © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 44 - Reads from source processed normally Reads from target to tracks that have been copied from source are read from the target Reads from target to tracks that have NOT been copied from source are actually read from source Attempts to write a source track that has not already been copied are intercepted and the source track is copied to the target volume before the update occurs. This maintains the T0 copy on the target. Source Write s Target Figure 12 Attempts to write data already copied proceed as normal Attempts to write a source track not already copied intercepted and source track copied to target before update occurs Writes to the target volume would proceed and the source volume bit map is updated to prevent the source track from being copied to the target volume What types of FlashCopy are there? DS8000 and DS6000 provide these FlashCopy functions: Data set FlashCopy (z/OS and z/VSE only) Multiple relationships Incremental FlashCopy Consistency group FlashCopy Inband FlashCopy Fast Reverse Restore enabled FlashCopy Fast Reverse Restore Full volume FlashCopy This is the most basic type of FlashCopy. It copies the entire source volume to a target and you can specify either background copy or nocopy. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 45 - Background nocopy to copy If you have a FlashCopy relationship with background nocopy, the target volume will become unusable if the relationship is withdrawn without all of the source tracks being copied to the target. If you wish to have the target volume be useable after a withdraw, you can convert the background nocopy to background copy. Once the background copy completes, the relationship will be withdrawn and the target volume is useable. You may want to do this, for example, if you wanted to FlashCopy the target volume. Persistent FlashCopy If you have a background copy and the background copy completes, the relationship is withdrawn. There is no way to query the source volume to see what the last FlashCopy target was if the relationship is withdrawn. If you specify that the FlashCopy is persistent, the relationship is maintained even after the background copy completes. The relationship would then be manually withdrawn. You may want to consider using persistent if you have procedures where you FlashCopy a source volume to a target one night and then FlashCopy it to a different target the following night. With persistent FlashCopy, you can query the source to see where it was last flashed, withdraw the relationship, and then flash to a different target. This lessens the chance of accidentally flashing to the wrong target. Data set FlashCopy Data set FlashCopy is only supported on z/OS and z/VSE. Multiple relationships A single source volume can have up to 12 targets of which only one can be incremental. Incremental FlashCopy Incremental FlashCopy provides the capability to only background copy those tracks that have been changed since the last increment was taken. This reduces the amount of data background copy has to process and reduces the amount of time it takes for the background copy to complete. When the FlashCopy relationship is established, the persistent, change data recording, and background copy parameters are used. If TSO is used, inhibit target write is forced. An incremental flash can be performed in either direction. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 46 - A bit map is used for to track changes to both the source and target volumes. Once the initial incremental FlashCopy is established, which does a background copy on the entire source on the first FlashCopy, the next increment will only background copy those tracks that have been updated since the last increment. It will also background copy those tracks that were updated on the target and need to be overlaid. To maintain the incremental relationship, specify change recording with each increment. If change recording is not specified, the relationship will be withdrawn after the incremental background copy completes. If you have done an incremental FlashCopy A > B, you don’t have to wait for the background copy to complete before issuing another increment A > B. However, if you have done an increment A > B and now wish to do an increment B > A, you must wait until the A > B background copy completes. Data set FlashCopy is NOT supported by incremental FlashCopy. Consistency group FlashCopy FlashCopy is often used to make copies of data that crosses the volume boundary. In cases where the data is dependent upon each other, the data that crosses boundaries needs to be consistent. That is, it must be copied in the proper order. FlashCopy Consistency Groups provides a mechanism for achieving a consistent data copy across multiple volumes without requiring that application I/O be quiesced. In the case of production data, application impact must be minimized. Prior to Consistency Group FlashCopy, customers would have to first quiesce the application, or in the case of DB2, use the SET LOG commands, establish their FlashCopy relationships, and then restart the application. This process can be very disruptive, causing application outages or data unavailability for an unacceptable period of time. Now, the user can specify consistency group for FlashCopy by using the ‘freeze’ option. This will hold off initiation/completion of write I/O to the source volumes until a thaw command is issued or 2 minutes elapses. The FlashCopy command is on a volume basis but the thaw command is at the LSS level, meaning that if there were more than one set of volumes using consistency, the thaw command would affect all of the consistency groups. The target of each source volume is within one physical disk subsystem but source volumes within a consistency group can span physical disk subsystems. Inband FlashCopy Inband FlashCopy allows FlashCopy requests to be issued remotely through an existing PPRC link. Once a FlashCopy establish is issued, the direct host connection from local to remote ESS is not required for a background copy to complete. The host connection would be needed, however, before © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 47 - any new FlashCopy tasks could be initiated. Inband FlashCopy can be useful if the host at the recovery site is not online. The Inband option eliminates the need for a host connection from local to remote exclusively for FlashCopy backup. The FlashCopy request must be issued at a host processor connected to the PPRC primary volume, with the PPRC secondary volume specified as the FlashCopy source. All supported full-volume FlashCopy commands can be issued with the Inband option, however, the THAW portion of consistency group processing is not supported with the Inband option. If it is acceptable to have long busy reported on a PPRC secondary device or an XRC secondary, the FREEZE portion of consistency group processing can be issued Inband and it will automatically thaw after 2 minutes. FlashCopy, PPRC, and z/OS Global Mirror A FlashCopy target can be a PPRC source volume if you specify that the target is a PPRC primary. If you are using Metro Mirror, the pair will fall out of duplex into copy pending mode until all of the flash data is transferred to the secondary volume. This could mean a loss of consistency during this time. A Global Copy source that is in a Global Mirror session and a z/OS Global Mirror source cannot be a FlashCopy target. Other copy services combinations can be found in the Copy Services Matrix later in this document. Invoking FlashCopy Functions Here is a table that summarizes the different FlashCopy functions and which tool can invoke them: Full volume (copy or nocopy) NOCOPY to COPY Persistent Data Set Multiple relationships Incremental Consistency Group Inband Fast Reverse Restore enabled Fast Reverse Restore Open LUN support DFSMSdss Yes TSO Yes API Yes ICKDSF Yes DS CLI Yes Yes No Yes Yes Yes Yes No No Yes No No Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No No Yes Yes No Yes Yes Yes Yes No Yes Yes Yes Yes Yes No No Yes Yes Yes Yes Yes No Yes Yes © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 48 - FlashCopy Source and Target placement In general: Spread evenly across disk subsystems Within each disk subsystem, spread evenly across clusters Within each cluster, spread evenly across device adapters Within each device adapter, spread evenly across ranks Place FlashCopy target in same cluster as source If using BACKGROUND COPY, target on a different device adapter Cluster Device Adapter Rank FlashCopy Establish Performance Same cluster Doesn’t matter Different ranks BACKGROUND COPY Performance Same cluster Different Different ranks device adapter FlashCopy Impact to Same cluster Doesn’t matter Different ranks Applications See the Copy Services RedBooks for more performance related information. FlashCopy and z/OS FlashCopy can be invoked by DFSMSdss, ICKDSF, API, Storage Manager GUI, TPC for Replication and DS CLI. The only tool that performs serialization is DFSMSdss. DFSMSdss requires that the source and target volumes be online to the system. The other tools can perform FlashCopy on offline volumes but do not do any serialization. FlashCopy and AIX Recommended environment: AIX V5.2 or V5.3 JFS2 file systems Use of consistency groups where multiple volumes must be copied © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 49 - The usual scenario would then follow this script: Set the application to on-line backup mode, if possible. Issue a sync command. Issue the file system freeze command: o chfs –a freeze=<timeout in seconds> /fsname Issue the point-in-time copy command to the required LUNs of the underlying storage subsystem. Wait for the point-in-time copies to complete. Issue the file system thaw command: chfs –a freeze=off /fsname Set the application back to normal mode. Note if you FlashCopy on same AIX host: chdev –l vpath –a pv=clear Recreatevg –y tgt_flash –L /backup –Y backup vpath4 vpath5 fsch <filesystem> If the recommended environment is not available: The only supported procedure is to unmount the file system If it is not possible to do that then the following actions should be taken: Use JFS2 with inline log so that all data is in a single volume If not using JFS2 or inline logs, don’t log multiple file systems to a single log Take action to quiesce applications in order to prevent I/O from continuing during the copy process (Use FlashCopy consistency groups or equivalent, use application function to quiesce, use sync/fsync to schedule/force writes to disk and delay some time prior to initiating the FlashCopy operation) Do not attempt to mount the copied file system until the following steps are done: Run “logredo /dev/<logname>”. If the “log wrap” error is displayed then the log for the source volume needs to be extended and the copy recreated Run a full fsck on the copied file system after it is created. Any error produced from this indicates that there was a problem with the copy and the copied file system should not be used FlashCopy and Windows 2000 and 2003 If you do not use VSS/VDS to invoke the FlashCopy, you should ensure that the target is NOT mounted and that data is flushed from the server. The mountvol command will perform this function. From the IBM TotalStorage DS8000 Series: Copy Services in an Open Environment (SG24-6788-01) RedBook: Copy Services limitations with Windows 2000 and Windows 2003 Having the drive information stored on the disk itself imposes some limitations when using Copy Services functionality on a Windows system: © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 50 - The source and target volumes must be of the same physical size. Normally the target volume can be bigger than the source volume; with Windows, this is not the case, for two reasons: The LDM database holds information relating to the size of the volume. As this is copied from the source to the target, if the target volume is a different size from the source, then the database information will be incorrect, and the host system will return an exception. The LDM database is stored at the end of the volume. The copy process is a track-by-track copy; unless the target is an identical size to the source, the database will not be at the end of the target volume. It is not possible to have the source and target FlashCopy volumes on the same Windows system when they were created as Windows dynamic volumes. The reason is that each dynamic volume has to have its own 128-bit GUID. As its name implies, the GUID must be unique on one system. When you perform FlashCopy, the GUID gets copied as well, so this means that if you tried to mount the source and target volume on the same host system, you would have two volumes with exactly the same GUID. This is not allowed, and you will not be able to mount the target volume. Copy services with Windows volumes In order to see target volumes on a second Windows host, you have to do the following: 1. Perform the Remote Mirror and Copy/FlashCopy function onto the target volume. Ensure that when using Remote Mirror and Copy that the primary and secondary volumes were in duplex mode, and write I/O was ceased prior to terminating the copy pair relationship. 2. Reboot the host machine on which you wish to mount the Copy Services target volume. 3. Right-click Open Computer Management, and then click Disk Management. 4. Find the disk that is associated with your volume. There are two panes for each disk; the left one should read Dynamic and Foreign. It is likely that no drive letter will be associated with that volume. 5. Right-click that pane and select Import Foreign Disks. Select OK, then OK again. The volume now has a drive letter assigned to it, and is of Simple Layout and Dynamic Type. You can read/write to that volume. Note: When using Windows dynamic disks, remember that to read FlashCopy targets, if the FlashCopy pair has been rescanned or if the server reading targets has been rebooted, FlashCopy targets will appear as foreign disks to that server. Manual intervention will be required to re-import those disks and restore operation. Tip: Disable the Fast-indexing option on the source disk; otherwise, operations to that volume get cached to speed up disk access. However, this means that data is not flushed from memory and the target disk may have copies of files/folders that were deleted from the source system. When performing subsequent Remote Mirror and Copy/FlashCopies to the target volume, it is not necessary to perform a reboot, because the target volume is still known to the target system. However, in order to detect any changes to the contents of the target volume, you © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 51 - should remove the drive letter from the target volume before doing the FlashCopy. Then, after carrying out the FlashCopy, you restore the drive letter in order for the host it is mounted on to be able to read/write to it. There is a Windows utility, DiskPart, that enables you to script these operations so that FlashCopy can be carried out as part of an automated backup procedure. DiskPart can be found at the Microsoft download site http://www.microsoft.com/downloads with a search on the key word DiskPart. A description of DiskPart commands can be found at the Web site: http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/diskpart.msp x More detail can be found in IBM TotalStorage DS8000 Series: Copy Services in an Open Environment - SG24-6788-01. Again…. The following excerpts have been taken from the IBM TotalStorage DS8000 Series: Copy Services in an Open Environment – SG24-6788-01 The Microsoft Volume Shadow Copy Services (VSS) is a storage management interface for Microsoft Windows Server 2003. VSS enables your storage array to interact with third-party applications that use the VSS Application Programming Interface (API). Microsoft VSS is included in the Windows Server 2003 installation. Prior to Microsoft VSS, if you did not have an online backup solution implemented, you either had to stop activities on your server during the Backup Process, or live with the side effects of an online backup, such as inconsistent data and open files that could not be backed up. With Windows Server 2003 and VSS enabled applications, online backup results in consistent data and files that are open during the backup are never a problem. Microsoft Volume Shadow Copy Services (VSS) enables you to perform online backup of applications, which traditionally is not possible. VSS is supported on the DS8000 storage server subsystem with FlashCopy capabilities. Microsoft VSS accomplishes the fast Backup Process when a backup application initiates a shadow copy backup. Microsoft VSS coordinates with the VSS-aware writers to briefly hold writes on the databases, applications, or both. Microsoft VSS flushes the file system buffers and asks a provider to initiate a FlashCopy of the data. Once the FlashCopy is logically completed, Microsoft VSS allows writes to resume and notifies the requestor that the backup has completed successfully. The volumes are mounted, hidden, and for read-only purposes, to be used when rapid restore is necessary. Alternatively, the volumes can be mounted on a different host and used for application testing or backup to tape. The following steps are performed by the Microsoft VSS in conjunction with FlashCopy when a backup application initiates a request for backup on a DS8000 Storage Unit. 1. VSS retrieves a list of volumes from the DS8000 and selects appropriate target volumes from the free pool (VSS_FREE). 2. VSS moves the target volumes to the reserved pool (VSS_RESERVED) and database suspends on writes. 3. VSS issues a FlashCopy from the source volumes to the target volumes and database resumes on writes after completion of FlashCopy. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 52 - 4. VSS assigns the target volumes to the backup server’s Host Bus Adaptors (HBAs) where Windows mounts the volumes on the backup server. 5. Requestor reads the data of the target volumes and copies it to tape. 6. Once the tape copy completes, Windows un-mounts the volumes and VSS unassigns target volumes from the backup server’s HBAs. 7. VSS assigns target volumes back to the free pool (VSS_FREE). The DS8000 is supported for VSS and has been fully tested by MicroSoft with Tivoli Storage Manager and Symantec. More information is available on the Microsoft web site. Please refer to the RedBook for the entire text. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 53 - Chapter 3: DS CLI DS CLI Intro Copy services can be managed by TSO, z/OS API, ICKDSF, the DS GUI, DS CLI, and GDPS. Most open users will choose DS CLI. This chapter will attempt to provide some hints and tips for using the DS CLI. DS CLI is the command line interface for doing logical configuration of the DS6000 and DS8000, managing copy services functions for DS6000, DS8000 and ESS 800. The ESS 800 has an additional CLI for Copy Services (CS CLI) which executes Copy Services tasks defined through the ESS Copy Services Web GUI. But, if you wish to have an ESS 800 participate with a DS8000 or DS6000 in a PPRC environment, the ESS CLI cannot be used and the DS CLI must be used instead. (TSO, ICKDSF, z/OS API can also be used to manage DS8000/DS6000 and ESS 800 environments.) DS CLI runs on Open systems platforms (including laptops) and the DS6000 SMC, but it is not supported on the DS8000 HMC. The DS command-line interface (CLI) can be installed on these operating systems. AIX 5.1, 5.2, 5.3 HP-UX 11i v1, v2 HP Tru64 version 5.1, 5.1A v Linux (RedHat 3.0 Advanced Server (AS) and Enterprise Server (ES) SUSE Linux SLES 8, SLES 9, SUSE 8, SUSE 9 Novell Netware 6.5 v OpenVMS 7.3-1 (or newer) Sun Solaris 7, 8, 9 Windows 2000, Windows Datacenter, Windows 2003, and Windows XP The DS CLI cannot be installed on a Windows 64-bit operating system. DS CLI Command Overview A command-line interface command consists of the following types of components, arranged in the following order: The command name. The command flags and flag parameters. One or more command parameters, each followed by any sub parameters it might require. “DSCLI Flag Parameters” are preceded by minus sign (-), are not positional EXCEPT they must precede a command parameter if there is one. Some are used for multiple commands and may be specified in the DSCLI profile file “DSCLI Command Parameters” are not preceded by a minus sign(-). There is only one on a DSCLI command and it is positional, it must be last parameter for the command. For example: © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 54 - mksession -dev ibm.1750-13aavca -lss 06 -volume 0600-0603 a2 flag parameter command parameter Here is a list of the different DS CLI command types: ls (list) One or more resources Formatting applies Short (-s) and long (-l) forms show (show) Additional detail on a single resource mk (create) ch (change) rm (delete) DS CLI requires resource IDs as command inputs such as: Storage disk system(-dev, -remotedev, -conduit) IBM.1750-13AAVCA IBM.2107-7585551 IBM.2105-24663 LSS x’xx’ Volume x’xxyy’ xx=LSS number; yy=volume number within LSS DS volume ID ESS volume ID prefixed by 0 or 1 NOT the z/OS unit address Note: that the serial number for the 1750 and 2107 include a country code whereas the 2105 does not. DSCLI input is not case sensitive (except for userids and passwords). It is recommended that a profile file be used with DS CLI as it simplifies the command syntax you will have to use. It contains common flag parameters such as hmc1, devid, or remotedevid. NOTE: This does not eliminate the need to enter Storage Image ID as a command parameter for some commands © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 55 - DS CLI Modes DS CLI commands can be invoked three ways. Single shot You would use the ds cli single-shot command mode if you wanted to issue an occasional command but did not want to keep a history of the previous commands that you have issued. You must supply the login information and issue the command that you want to process at the same time. Here is an example of a single shot ds cli command: dscli -hmc1 n.n.n.n -user userid -passwd password mkflash –dev IBM.2107-AZ12341 0000:0100 When using single shot mode, you exit ds cli after each command. Interactive Mode You would use the ds cli interactive mode when you have multiple transactions to process that cannot be incorporated into a script or hasn’t yet been scripted. The interactive command mode provides a history function that makes repeating or checking prior command usage easy to do. In addition to being able to enter ds cli commands at the ds cli command prompt, a history function provides a view of the last ds cli commands that you have used. It also allows you to repeat any of the last commands more quickly than having to type out the entire command. The example at the end of this process shows how the history function works. To enter interactive mode, issue: dscli –cfg profile\dscli.profile –user userid –passwd password You would then have the following prompt: dscli> From the prompt, you can now issue commands and not exit ds cli until you issue ‘quit’. To use the ds cli history function that is associated with the interactive command mode, perform the following steps: 1. Issue an exclamation mark (!) to display CLI commands that you have used in the current session. For example: dscli>! results in a list of commands such as the following: [4] mkflash -dev IBM.2107-1300771 0000:0100 [3] lspprc -dev IBM.2107-1300771 0000:0100 [2] lspprcpath -dev IBM.2107-1300771 –remotedev IBM.2107-7512341 00 [1] lsflash -dev IBM.2107-1300771 0000:0100 2. Issue dscli>!1 to retry the last command. Or, issue dscli>!3 to retry command [3]. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 56 - – – – – dsscli Enter commands at shell prompt Exit code on exit (use stdout output for status) For commands that act on multiple arguments, cli will complete all it can, and report on successes and failures Script mode Use the DS CLI script command mode if you want to issue a sequence of DS CLI commands. Administrators can use this mode to create automated processes; for example, establishing remote mirror and copy relationships for volume pairs. Consider the following when using the DS CLI script command mode: The DS CLI script can contain only DS CLI commands. Use of shell commands results in a process failure. You can add comments to the scripts. Comments must be prefixed by the number sign (#); for example, # This script contains PPRC Path establish procedures. The following is an example of a script that could be used to establish remote mirror and copy relationships for volume pairs. The commands are in a text file. For this example, the name of the file is mkpprc.txt mkpprc -dev IBM.2107-1303561 -remotedev IBM.2107-7504491 -type mmir 1000-103F:2300-233F mkpprc -dev IBM.2107-1303561 -remotedev IBM.2107-7504491 -type gcp 1100-113F:2340-237F mkpprc -dev IBM.2107-1303561 -remotedev IBM.2107-7504491 -type mmir 1800-187F:2800-287F mkpprc -dev IBM.2107-1303561 -remotedev IBM.2107-7504491 -type gcp 1200-127F:2500-257F mkpprc -dev IBM.2107-1303561 -remotedev IBM.2107-7504491 -type mmir 1040-1054:2700-2714 mkpprc -dev IBM.2107-1303561 -remotedev IBM.2107-7504491 -type gcp 1055-107F:2400-242A mkpprc -dev IBM.2107-1303561 -remotedev IBM.2107-7504491 -type mmir 1140-117F:2600-263F To invoke the script in single shot mode, you would enter: dscli –hmc1 n.n.n.n –user userid -passwd password –script c:\filename ds cli scripts are executed one line (one command) at a time. The script will exit on the first failure. You will receive an exit code upon exit (use stdout output for status). For commands that act on multiple arguments, if one operation fails, CLI will issue a failure exit code and exit the script after the failed command Tip: When pointing to a script file, avoid blanks in either the directory or file names. DS CLI Profile © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 57 - A default profile file is created at installation time: C:\Program Files\IBM\dscli\profile\dscli.profile This file can be updated for your environment. The profile file is invoked at DS CLI startup and the -cfg flag can be used on dscli command to invoke a profile other than default profile. The profile must be in dscli\profile directory and profile settings are overridden by flag parameters specified on command. Here is the sample profile that is shipped with DS CLI: # Management Console/Node IP Address(es) # hmc1 and hmc2 are equivalent to -hmc1 and -hmc2 command options. #hmc1:n.n.n.n #hmc2:n.n.n.n # # Default target Storage Image ID # "devid" and "remotedevid" are equivalent to # "-dev storage_image_ID" and "-remotedev storage_image_ID" command options, #respectively. #devid:IBM.2107-AZ12341 #remotedevid: IBM.2107-AZ12341 # # Default locale is based on user environment. #locale: en # #timeout: 900 # # Socket connection timeout value in seconds. #timeout.connection: 20 # # Output settings fullid: off paging:off #rows: 24 #format: default #delim: | banner:on header: on verbose: off Here is a profile that has been updated with –hmc, -dev, -remotedev and verbose on: # # DS CLI Profile # # © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 58 - # Management Console/Node IP Address(es) # hmc1 and hmc2 are equivalent to -hmc1 and -hmc2 command options. hmc1: n.n.n.n #hmc2:n.n.n.n # # Default target Storage Image ID # "devid" and "remotedevid" are equivalent to # "-dev storage_image_ID" and "-remotedev storeage_image_ID" command options, # respectively. devid: IBM. .2107-75FA120 remotedevid: IBM. IBM.2107-75FA150 # off : No verbose information. verbose: on If you customize the default profile and save it for your use, MAKE SURE that there are NO BLANKS at the end of a variable. If there is a blank, you will receive an error message. It is very difficult editing the file to see that there is a blank at the end of the variable. DS CLI Password File The password file is an encrypted file where DS CLI is installed. There is only a single password file per DS CLI. The default location is <User Home>\dscli\security.dat. Windows: %USERPROFILE%\dscli\security.dat o Documents and Settings UNIX or Linux: $HOME/dscli/security.dat The DS CLI managepwfile command creates/updates the password file. For example: dscli>managepwfile -action add -name testuser –pw AB9cdefg If the password file is not security.dat and was created using managepwfile –pw filename then the password file is specified as an option on the DS CLI command. – dscli –pwfile pwfilename Assume that a profile has been created with –hmc1, –remotedev and –dev and that password file dscli/security.dat has been created. DS CLI can be run in several different modes. They are: Interactive C:\Program Files\ibm\dscli> dscli –cfg profile/profilefilename –user user Date/Time: February 16, 2005 1:55:35 PM EST IBM DSCLI Version: 5.0.1.33 DS: IBM.2107-75FA120 dscli> lspprc –l 0100-0101:0100-0101 Date/Time: February 16, 2005 1:55:35 PM EST IBM DSCLI Version: 5.0.1.33 DS: IBM.2107-75FA120 © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 59 - dscli> Single shot C:\Program Files\ibm\dscli> dscli –cfg profile/profilefilename –user user lspprc –l 0100-0101:0100-0101 Date/Time: February 16, 2005 1:59:40 PM EST IBM DSCLI Version: 5.0.1.33 DS: IBM.210775FA120 exit status of dscli = 0 © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 60 - C:\Program Files\ibm\dscli> Script C:\Program Files\ibm\dscli> dscli –cfg profile/profilefilename –user user -script C:\lspprcscript.txt Date/Time: February 16, 2005 1:59:40 PM EST IBM DSCLI Version: 5.0.1.33 DS: IBM. 2107-75FA120 exit status of dscli = 0 C:\Program Files\ibm\dscli> For this example, C:\lspprccript.txt file contains only lspprc –l 0100-0101:0100-0101 Collecting DS CLI Output Singleshot mode – Pipe and append to text file • dscli command >> c:\dscli_output\75FA120.txt – Pipe and append to .csv file • dscli command –fmt delim –delim , >> C:\dscli_output\75FA120.csv • Open with spreadsheet tool – Command is not piped • Can use DOS echo command and redirect echo output C:\>echo hello hello DSCLI -script mode – Pipe and append to text or .csv file (same considerations as singleshot mode) © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 61 - To collect the output in the above lspprc script: lspprc –l 0100-0101:0100-0101 >> c:\dscli_output\75FA120.txt Tip: When pointing to an output file, avoid blanks in either the directory or file names. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 62 - Chapter 4: DS CLI Scripts and Batch Files Scripts A ds cli script can be a series of ds cli commands in a text (.txt) file. The following script in c:\dscli_scripts\MM_Set_Up_abc2a.txt will create a Metro Mirror environment. It would be invoked by issuing: C:\Program Files\IBM\dscli>dscli -cfg profile\d6abc2a.profile -user workshop -script c:\dscli_scripts\MM_Set_Up_abc2a.txt >c:\dscli_output\abc2a.txt The profile contains the hmc1 address. The userid password is in security.dat. #The following script will set up a Metro Mirror environment. # A dscli -hmc1 n.n.n.n -user userid -dev ibm.1750-13abc2a # B dscli -hmc1 n2.n2.n2.n2 -user userid -dev ibm.1750-13aavca # 1) Establish the Metro Mirror paths and pairs mkpprcpath -dev IBM.1750-13abc2a -remotedev IBM.1750-13aavca -remotewwnn 500507630efffca0 -srclss 00 -tgtlss 00 I0000:I0000 I0100:I0100 mkpprcpath -dev IBM.1750-13abc2a -remotedev IBM.1750-13aavca -remotewwnn 500507630efffca0 -srclss 10 -tgtlss 10 I0000:I0000 I0100:I0100 lspprcpath -dev IBM.1750-13abc2a 00 10 mkpprc -dev IBM.1750-13abc2a -remotedev IBM.1750-13aavca -type gcp -mode full 0000-0003:0000-0003 1000-1003:1000-1003 lspprc -dev IBM.1750-13abc2a 0000-0003:0000-0003 1000-1003:1000-1003 The following script in c:\dscli_scripts\MM_Set_Up_abc2a.txt will clean up the Metro Mirror environment. It would be invoked by issuing: C:\Program Files\IBM\dscli>dscli -cfg profile\d6abc2a.profile -user workshop -script c:\dscli_scripts\MM_Clean_Up_abc2a.txt >c:\dscli_output\abc2a.txt The profile contains the hmc1 address. The userid password is in security.dat. #The following script will set up a Global Mirror environment. # A dscli -hmc1 n.n.n.n -user userid -dev ibm.1750-13abc2a # B dscli -hmc1 n2.n2.n2.n2 -user userid -dev ibm.1750-13aavca # 1) Remove the Metro Mirror paths and pairs rmpprc -dev IBM.1750-13abc2a -remotedev IBM.1750-13aavca -quiet 0000-0003:00000003 1000-1003:1000-1003 lspprc -dev IBM.1750-13abc2a 0000-0003:0000-0003 1000-1003:1000-1003 rmpprcpath -dev IBM.1750-13abc2a -remotedev IBM.1750-13aavca -remotewwnn 500507630efffca0 -quiet 00:00 © IBM Copyright, 2006 11/30/2006 Web location of document (www.ibm.com/support/techdocs) - 63 DS8000/DS6000 Copy Services: Getting Started rmpprcpath -dev IBM.1750-13abc2a -remotedev IBM.1750-13aavca -remotewwnn 500507630efffca0 -quiet 10:10 lspprcpath -dev IBM.1750-13abc2a 00 10 Note that the text script commands are being directed to a single HMC. If your DS8000s and DS6000s are not cross configured to an HMC that also manages a different DS8000 or DS6000, you would not be able to use a script to set up a Global Mirror environment. The reason for this is that Global Mirror requires FlashCopy pairs be defined in the secondary box which is normally a different physical disk system than that of the Global Copy primaries physical subsystem. Batch File The following batch file in c:\dscli_bat\GM_Set_Up_abc2a.bat will create a Global Mirror environment. It would be invoked by issuing: C:\dscli_bat\GM_Set_Up_abc2a.bat cd c:\Program Files\IBM\dscli\ @echo mkpprcpath >c:\dscli_output\abc2a.txt >c:\dscli_output\abc2a.txt dscli -hmc1 n.n.n.n -user userid -passwd password mkpprcpath -dev IBM.175013abc2a -remotedev IBM.1750-13aavca -remotewwnn 500507630efffca0 -srclss 00 tgtlss 00 I0000:I0000 I0100:I0100 >>c:\dscli_output\abc2a.txt dscli -hmc1 n.n.n.n -user userid -passwd password mkpprcpath -dev IBM.175013abc2a -remotedev IBM.1750-13aavca -remotewwnn 500507630efffca0 -srclss 10 tgtlss 10 I0000:I0000 I0100:I0100 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo lspprcpath 00 10 >>c:\dscli_output\abc2a.txt >>c:\dscli_output\abc2a.txt dscli -hmc1 n.n.n.n -user userid -passwd password lspprcpath -dev IBM.175013abc2a 00 10 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo mkpprc >>c:\dscli_output\abc2a.txt >>c:\dscli_output\abc2a.txt dscli -hmc1 n.n.n.n -user userid -passwd password mkpprc -dev IBM.1750-13abc2a -remotedev IBM.1750-13aavca -type gcp -mode full 0000-0003:0000-0003 10001003:1000-1003 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo lspprc >c:\dscli_output\abc2a.txt >>c:\dscli_output\abc2a.txt © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 64 - dscli -hmc1 n.n.n.n -user userid -passwd password lspprc -dev IBM.1750-13abc2a -remotedev IBM.1750-13aavca 0000-0003:0000-0003 1000-1003:1000-1003 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo mkflash >>c:\dscli_output\abc2a.txt >>c:\dscli_output\abc2a.txt dscli -hmc1 n2.n2.n2.n2 -user userid -passwd password mkflash -dev IBM.175013aavca -nocp -record -tgtinhibit -persist 0000-0003:0100-0103 1000-1003:11001103 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo lsflash >>c:\dscli_output\abc2a.txt >>c:\dscli_output\abc2a.txt dscli -hmc1 n2.n2.n2.n2 -user userid -passwd password lsflash -dev IBM.175013aavca 0000-0003 1000-1003 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo mksession >>c:\dscli_output\abc2a.txt >>c:\dscli_output\abc2a.txt dscli -hmc1 n.n.n.n -user userid -passwd password mksession -dev IBM.175013abc2a -lss 00 -volume 0000-0003 a1 >>c:\dscli_output\abc2a.txt dscli -hmc1 n.n.n.n -user userid -passwd password mksession -dev IBM.175013abc2a -lss 10 -volume 1000-1003 a1 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo lssession >>c:\dscli_output\abc2a.txt >>c:\dscli_output\abc2a.txt dscli -hmc1 n.n.n.n -user userid -passwd password lssession -dev IBM.175013abc2a 00 >>c:\dscli_output\abc2a.txt dscli -hmc1 n.n.n.n -user userid -passwd password lssession -dev IBM.175013abc2a 10 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo mkgmir >>c:\dscli_output\abc2a.txt >>c:\dscli_output\abc2a.txt dscli -hmc1 n.n.n.n -user userid -passwd password mkgmir -dev IBM.1750-13abc2a -lss 00 -session a1 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo showgmir >>c:\dscli_output\abc2a.txt >>c:\dscli_output\abc2a.txt © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 65 - dscli -hmc1 n.n.n.n -user userid -passwd password showgmir -dev IBM.175013abc2a 00 >>c:\dscli_output\abc2a.txt The following batch file in c:\dscli_bat\GM_Set_Up_abc2a.bat will take down a Global Mirror environment. It would be invoked by issuing: C:\dscli_bat\GM_Clean_Up_abc2a.bat cd c:\Program Files\IBM\dscli\ @echo chsession >c:\dscli_output\abc2a.txt dscli -cfg profile\d6abc2a.profile -hmc1 n.n.n.n -user userid -passwd password chsession -lss 00 -action remove -volume 0000-0003 a1 >>c:\dscli_output\abc2a.txt dscli -cfg profile\d6abc2a.profile -hmc1 n.n.n.n -user userid -passwd password chsession -lss 10 -action remove -volume 1000-1003 a1 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo rmsession >>c:\dscli_output\abc2a.txt dscli -cfg profile\d6abc2a.profile -hmc1 n.n.n.n -user userid -passwd password rmsession -lss 00 -quiet a1 >>c:\dscli_output\abc2a.txt dscli -cfg profile\d6abc2a.profile -hmc1 n.n.n.n -user userid -passwd password rmsession -lss 10 -quiet a1 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo lssession 00 10 >>c:\dscli_output\abc2a.txt dscli -cfg profile\d6abc2a.profile -hmc1 n.n.n.n -user userid -passwd password lssession 00 10 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo rmgmir -quiet -lss 00 -session a1 >>c:\dscli_output\abc2a.txt dscli -cfg profile\d6abc2a.profile -hmc1 n.n.n.n -user userid -passwd password rmgmir -quiet -lss 00 -session a1 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo showgmir 00 >>c:\dscli_output\abc2a.txt © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 66 - dscli -cfg profile\d6abc2a.profile -hmc1 n.n.n.n -user userid -passwd password showgmir 00 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo rmflash -quiet 0000-0003:0100-0103 1000-1003:1100-1103 >>c:\dscli_output\abc2a.txt dscli -cfg profile\d6aavca.profile -hmc1 n2.n2.n2.n2 -user userid -passwd password rmflash -quiet 0000-0003:0100-0103 1000-1003:1100-1103 @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo lsflash –hmc1 n2.n2.n2.n2 -user userid -passwd password 0000-0003:01000103 1000-1003:1100-1103 >>c:\dscli_output\abc2a.txt dscli -cfg profile\d6aavca.profile -hmc1 n2.n2.n2.n2 -user userid -passwd password lsflash 0000-0003:0100-0103 1000-1003:1100-1103 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo rmpprc >>c:\dscli_output\abc2a.txt dscli -cfg profile\d6abc2a.profile -hmc1 n.n.n.n -user userid -passwd password rmpprc -quiet 0000-0003:0000-0003 1000-1003:1000-1003 >>c:\dscli_output\abc2a.txt @echo ------------------------------------------------------------->>c:\dscli_output\abc2a.txt @echo lspprc >>c:\dscli_output\abc2a.txt dscli -cfg profile\d6abc2a.profile -hmc1 n.n.n.n -user userid -passwd password lspprc 0000-0003:0000-0003 1000-1003:1000-1003 >>c:\dscli_output\abc2a.txt @echo rmpprcpath >>c:\dscli_output\abc2a.txt dscli -cfg profile\d6abc2a.profile -hmc1 n.n.n.n -user userid -passwd password rmpprcpath -quiet 00:00 10:10 >>c:\dscli_output\abc2a.txt dscli -cfg profile\d6abc2a.profile -hmc1 n.n.n.n -user userid -passwd password lspprcpath 00 10 >>c:\dscli_output\abc2a.txt Note that the batch file issues a series of single shot ds cli commands that can be directed to any HMC with network connectivity. This allows a batch file to define a Global Mirror environment. With batch files, you can use batch parameters anywhere within a batch file to extract information about your environment settings. This provides flexibility in that one batch file can be used for multiple environments. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 67 - For example, assume that selected srclss, tgtlss, wwnn, and paths have been listed in file c:\dscli_lists\abc2a.path.lst 00 00 5005076303FFFCA0 I0000:I0000 I0100:I0100 10 10 5005076303FFFCA0 I0000:I0000 I0100:I0100 Establishing PPRC paths can be done using the batch file shown below. The path information will be read from a file for each profile defined, c:\dscli_lists\abc2a.paths.lst. The batch file is in c:\dscli_bat\ and the profile is in c:\Documents and Settings\Administrator\dscli\profile for %%f in (profile/%1*.ini) do @call :make %%f %2 @goto :EOF :make @Echo %~n1 Establish paths %2 @for /F "eol=# tokens=1,2,3*" %%S in (%~n1.path.lst) do ( dscli -cfg %1 mkpprcpath %2 -srclss %%S -tgtlss %%T -remotewwnn %%U %%V ) The batch file is invoked using :\Documents and Settings\Administrator\dscli>c:\dscli_bat\MM_Set_Up_abc2a.bat abc2a -consistgrp Est_path abc2a Est_path * -consistgrp establish paths for profiles named profile/abc2a*.ini add the consistency group option with all profiles Other PPRC operations could be performed using the above technique. © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 68 - Enterprise Disk Copy Services Matrix This matrix can be found in TechDocs: IBMers: http://w3-3.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD103375 BPs: Device Is http://partners.boulder.ibm.com/src/atsmastr.nsf/WebIndex/TD103375 GMz10 (XRC) Primary GMz10 (XRC) Secondary Metro Mirror or Global Copy Primary Metro Mirror or Global Copy Secondary Global Mirror Primary Global Mirror Secondary FlashCopy Source FlashCopy Target Incremental FLC Source Incremental FLC Target Concurrent Copy Source GMz10 (XRC) Primary No Yes11 Yes No Yes No Yes Yes No No Yes GMz10 (XRC) Secondary Yes11 No Yes No5 Yes No5 Yes No5 Yes No5 Yes Metro Mirror or Global Copy Primary Yes Yes No Yes 1 No6 Yes1 Yes Yes Yes No6 Yes Metro Mirror or Global Copy Secondary No No5 Yes 1 No Yes1 No Yes Yes8 Yes Yes No Global Mirror Primary Yes Yes No6 Yes1 No No Yes Yes Yes Yes Yes Global Mirror Secondary No No5 Yes1 No No No Yes Yes8 Yes9 No No FlashCopy Source Yes Yes Yes Yes Yes Yes Yes 3,4 Yes 4 Yes3 No Yes FlashCopy Target No No5 Yes 2 No No No Yes 4 Yes4 No No No Incremental FLC Source No7 Yes Yes Yes Yes Yes9 Yes No No No Yes Incremental FLC Target No7 No5 Yes 2 No No No No No No No Yes Concurrent Copy Source Yes Yes Yes No Yes No Yes Yes Yes Yes Yes May Become Notes: 1. Only in a Metro/Global Copy (supported on ESS) or a Metro/Global Mirror Environment (supported on ESS and DS8000). 2. FlashCopy V2 at LIC 2.4.0 and higher on ESS800 (DS6000 and DS8000 utilize FlashCopy V2 by default). a. You must specify the proper parameter to perform this b. Metro Mirror primary will go from full duplex to copy pending until all of the flashed data is transmitted to remote c. Global Mirror primary cannot be a FlashCopy target 3. FlashCopy V2 Multiple Relationship. 4. FlashCopy V2 Data Set FlashCopy (only available for z/OS volumes). 5. The Storage Controller will not enforce this restriction, but it is not recommended. 6. A volume may be converted between the states Global Mirror primary, Metro Mirror primary and Global Copy primary via commands, but it two relations cannot exist at the same time (i.e. multi-target). © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 69 - 7. GMz (XRC) Primary, Global Mirror Secondary, Incremental FlashCopy Source and Incremental FlashCopy Target all use the Change Recording Function. For a particular volume only one of these relationships may exist. 8. Updates to the affected extents will result in the implicit removal of the FlashCopy relationship, if the relationship is not persistent. 9. This relationship must be the FlashCopy relationship associated with Global Mirror – i.e. there may not be a separate Incremental FlashCopy relationship. 10. Global Mirror for z/OS (GMz) is supported on ESS and DS8000 11. In order to ensure Data Consistency, the XRC Journal volumes must also be copied. References SC35-0428 DFSMS Advanced Copy Services © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 70 - SC26-7616 IBM System Storage DS8000 Command-Line Interface User's Guide GC26-7922 IBM System Storage DS6000 Command-Line Interface User's Guide SG24-6787 The IBM TotalStorage DS8000 Series: Copy Services with IBM eSeries zSeries SG24-6788 The IBM TotalStorage DS8000 Series: Copy Services in Open Environments SG24-6782 The IBM TotalStorage DS6000 Series: Copy Services with IBM eSeries zSeries SG24-6783 The IBM TotalStorage DS6000 Series: Copy Services in Open Environments © IBM Copyright, 2006 Web location of document (www.ibm.com/support/techdocs) DS8000/DS6000 Copy Services: Getting Started 11/30/2006 - 71 -