The Value of the Mainframe

IBM Advanced Technical Support - Americas

IBM DS8000 Data Replication Best Practices

Bob Kern – bobkern@us.ibm.com

Jim Sedgwick – jsedgwic@us.ibm.com

Hank Sautter – sautter@us.ibm.com

© 2009 IBM Corporation

2


Agenda

 DS8000 Copy Services - Best Practices

 DS8000 Data Replication Technology

– Key Concepts

– Selecting the Right Solution to Match your Clients Business

Needs

 Planning for Data Replication

– Configuration

– Data Collection

– BandWidth Studies

– Automation

 Case Studies – What you do not want to do…..



3

DS8000 Copy Services - Best Practices

 Map solution to Customer’s Business Requirements (RTO/RPO)

–

PiT Solution vs Continuous Mirror, Sync vs Async

– For several small customers doing PTAM -> a PiT Inc FC + GC can be attractive.

– High Availability

– z/OS Basic HyperSwap or GDPS HyperSwap Mgr,

– Dist. System – Software Dual Write across 2 Luns

– Metro Mirror on Same Data Center Floor -> outage, re-ipl minimizes outage time.

 Configuration Guidelines for Primary & Secondary

– Balance Primary & Secondary Performance (subsystems/cache/drives/etc.)

– 1:1 or 2:1 Configurations tend to be simpilier to configure & Manage.

– Be careful reusing “old” technology boxes as targets.

– Ability to Test D/R while Maintaining D/R Protection.

• Standard for most customer.

• Emerging Standard -> Test the Way the Recover & Recover the Way they Test.

– Failover/Failback Functionality

• Site Toggle can perhaps reduce D/R Test costs. Run 6 months in each site.

 Bandwidth Analysis -> Data Collection

– Use Tools: Disk Magic, RMF Magic etc.

– Analysis is VERY Important to understand BW for MB/Sec Update rates.

– MB/SEC Update Rates can also help customer Manager his business. Understand activity 24X7…

– Planning for Capacity Growth

• Initial Deployment should have some capacity Growth planned into Solution.

• Customer needs to understand how to do this as the years go by, workloads change etc.

 Review - D/R Lessons Learned

– White Paper: IBM Storage Infrastructure for Business Continuity – Updated This year

– http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS2511

 Management Software -> TPC-R or GDPS

– DSCLI can be made to work, but can be very complicated.

–

Best Implementations involve TPC-R or GDPS Software.


4


Map the Right Solution to the

Clients Business Requirements



Aspects of Availability

5

High Availability

Fault-tolerant, failureresistant infrastructure supporting continuous application processing

Protection of critical business data

Continuous Operations

Non-disruptive backups and system maintenance coupled with continuous availability of applications

Disaster Recovery

Protection against unplanned outages such as disasters through reliable, predictable recovery

Operations continue after a disaster

Recovery is predictable and reliable Costs are predictable and manageable



Business Continuity Tiers

Recovery from a disk image Recovery from tape copy

6

Tier 7

– Site Mirroring with automated recovery

Tier 6 - Disk mirroring (with/without automation)

Tier 5 –Software replication

15 Min.

1-4 Hr..

Tier 4 - Point in Time disk copy

4 -8 Hr..

8-12 Hr..

Tier 3 - Electronic Vaulting

Tier 2 - Hot Site,

Restore from Tape

12-16 Hr..

24 Hr..

Recovery Time Objective

Days Tier 1

–

Restore from Tape

Best Business Continuity practice is to blend solutions in order to maximize application coverage at optimum cost


7


Data Replication Enabling Core Technologies -

FlashCopy

Internal Copy

Available on:

DS6000, DS8000, ESS

SAN Volume Controller, XIV,

DS4000, DS5000,

PiT Incremental FlashCopy

+ Metro Mirror

Available on:

DS6000, DS8000, ESS

Copy data command issued

Copy is immediately available

Time

Source

Write

Target

Read Read and write to both source and copy possible

Primary

Prod

PiTCopies

PiT

Incrementa l

FlashCopy

Global Copy

(Asynchronous

)

WAN

Optional background copy

REMOTE

When copy is complete, relationship between source and target ends

PPRC-XD

Secondar y

FlashCop y

PiT

Incremental

Copy



IBM Copy Services Technologies

FlashCopy

 Point in time copy



DS8K FCSE

 Available on:

 DS8000, DS6000, SAN

Volume Controller



DS4000/DS5000

 N Series

 XIV

Metro Mirror

 Synchronous mirroring

 DS8K HyperSwap



Available on:



DS8000, DS6000, ESS

 SAN Volume Controller

 DS4000/DS5000

 N Series



XIV

Global Mirror

 Asynchronous mirroring



Available on:



DS8000, DS6000

 SAN Volume Controller

 DS4000/DS5000

 N Series

Metro / Global Mirror

 Three site synchronous and asynchronous mirroring

 Available on:

 DS8000

 N Series

Within

Storage

System

Primary

Site A

Metro distance

<300km

Site B

Primary

Site A

Primary

Site A

Metro

Site B

8

Out of

Region

Site

B

Out of

Region

Site C


9


Configuration Guidelines



10

Configuration Guidelines for Global Mirror

 Whitepapers:

–

Global Mirror http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100642

–

GM Secondary http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101105

 TPC for Replication or GDPS

– Required

–

Managing GM with dscli scripts is not a realistic customer option

 Do not undersize the GM Secondary

 Solution Planning Considerations

– Asymmecrical .vs symmetrical

– Volume size and layout

• LH, RH, RJ (A, B, C) are exactly the same size

– Failover & failback considerations

• More on layout later

– LSS Considerations – balance resources

• Dedicate two LSS(s) per DS8000 to an application, one even, one odd

– Tends to group volume numbers in ranges – easily recognizable ranges of volumes

– Management is easier with fewer LSSs

– Performance evaluation & Bandwidth Sizing between sites

– Use dedicated PPRC link ports, 2 minimum, on separate host adapters



Testing !!


11


Global Mirror Prerequisite –> Performance at the Primary Site

 Global Mirror Primary Performance must be good on volumes to be replicated

– Rank Performance

•

Backend response times

• Frontend Write response times ≤ 2ms

•

No cache issues

– Host Port Performance

– Good I/O balance across Ranks, Host Adapters

– Performance Tools:

• TPC for Disk

•

RMF Magic

 Plan and Monitor for Growth

– Workloads increase

– Additional replication requirements

•

New applications

Local Site

Global

Mirror

Remote Site

Global Mirror Primary Global Mirror Secondary



Global Mirror Secondary - Performance Capacity

Ok.

 Equal to or Greater than the performance capacity of the primary

 Global Mirror places more write performance stress on the secondary

– More storage capacity

– Flash Copy processing

 Cache required per volume count

 Do not undersize the Secondary

Global Mirror

Primary

ESS 800

ESS 800

DS8100

DS8100

Global Mirror Secondary

ESS 800

ESS 800 with Arrays Across

Loops

DS8100

DS8300

ESS 800

DS6800

DS8000 /

16GB

DS8000 /

32GB

DS8000 /

64GB

DS8000 /

128GB

DS8000 /

256GB

Recommende d max number of

GMir volumes

1000

350

1500

Recommended max number of

GMir volumes

DS8000 LIC

Release 3.0 or later

N/A

N/A

4500

1500

3000

6000

12000

Global

Mirror

4500

9000

18000

36000

DS8100 model 921

DS8100 model 931

DS8100 model 931

DS8300 any model

DS8300 model 932

Local Site

Global Mirror Primary

Remote Site

Global Mirror Secondary


13


Placement of B and C volumes

Primary

Secondary

A



B C

 Same RPM

 Same RAID type

B C

B C B C

 On Secondary

– All ranks contain equal numbers of B and C volumes

– B and C copies for particular volumes kept on separate ranks

– Activity for busy volumes spread over two ranks

– DDM size equal to or one size greater than primary

 On Secondary

– B and C copies on same rank results in hotspot being concentrated on single rank


14


Placement of B, C and D volumes

B C D B C

B C D

B C D



All ranks contain equal numbers of B, C and

D volumes

 B and C copies for particular volumes kept on separate ranks



Placement of D volumes as shown above less critical but easier to manage/track/implement

B C

D D

 D volumes placed on separate ranks to keep testing and backup activity separate



B and C copies for particular volumes on different ranks

 This does reduce available ranks for B & C volumes



FlashCopy Source & Target Placement

 In general:

–

Spread evenly across disk subsystems

– Within each disk subsystem, spread evenly across clusters

–

Within each cluster, spread evenly across device adapters

–

Within each device adapter, spread evenly across ranks

 Place FlashCopy target in same cluster as source

– If using BACKGROUND COPY, target on a different device adapter

 FlashCopy Space Efficient

– Use FlashCopy Space Efficient when economy is more important than performance and for short-lived relationships with low update rate on source volumes

• Short Term FlashCopy relationships

• Good for read only applications

– Tape Backup, 24 hour online backup, etc

15

Cluster Device Adapter Rank

Same cluster Doesn’t matter Different ranks FlashCopy Establish

Performance

Background Copy

Performance

FlashCopy Impact to

Applications

Same cluster Different device adapter

Different ranks

Same cluster

Doesn’t matter

Different ranks



FlashCopy SE Relationships

 Full volume only

 NOCOPY only in first release

– Background copy cannot be initiated to a SE volume by any means

 Must specify “SE target ok” at establish

 Recommended Usage

– Use FlashCopy Space Efficient when economy is more important than performance and for short-lived relationships with low update rate on source volumes

•

Short Term FlashCopy relationships

• Good for read only applications

– Tape Backup, 24 hour online backup, etc

16


17


Data Collection

Performance and Bandwidth

Studies


18


Data Collection Essentials

 Historical Data

– Data selection is critical for obtaining valid results

• Single data points or averages are not very useful

• Size and duration of peaks are important

– Need to identify daily peaks and the workload profile over time

• End of month, end of quarter, end of year, etc.

– Identify active volumes for workload balancing

– Quantify expected growth

 Configuration details

– Production vs. test data

– Temporary data may not be part of Global Mirror

– Volume layout by array and storage pool

– Network configuration including bandwidth available

 Monitoring Performance and Status

– TPC Standard Edition

– RMF data (enable ESS data collection)

– TPC for Replication

– Global Mirror Monitor


19


Data Collection Essentials (2)

 Data needed for evaluation

– Performance data for 1 week

– I/O rates, Data Rates, Response times

– Configuration details and event timelines (when are peaks expected)

 Data Sources

– RMF for zSeries

– iSeries PT reports

– Total Storage Productivity Center (TPC) reports

– iostat reports, windows perfmon reports

 Evaluation tools

– Disk Magic Capacity planning

– RMF Magic Performance evaluation

– DS8Qtool DS8000 physical and logical configuration details

 Contact IBM ATS for Performance and Bandwidth Studies

– add a link to partner world.


20


Lessons & Automation


21


Lessons Learned About IT Survival



Repeated Testing before a disaster is crucial to successful recovery after a disaster

– TTWYR – Test The Way You Recover

– RTWYT – Recover The Way You Test

 After a disaster, everything is different

– Staff well-being will be 1st priority

–

– Company will benefit greatly from well-documented, tested, available and

Repeated Testing before a disaster is crucial to successful recovery after a disaster

–  May be necessary to implement in-house D/R solution to meet RTO/RPO

–  Plan geographically dispersed IT facilities

–

– IT equipment, control center, offices, workstations, phones, staff, . . .

After a disaster, everything is different

–  Installed server capacity at second data center can be utilized to meet normal



Failover capacity can be obtained by

– Prioritizing workloads

– Exploit new technology: Capacity Back Up (CBU)

 Data backup planning and execution must be flawless

– Disk mirroring required for <12hr RTO (need 2x capacity)

– Machine-readable data can be backed up; not so for paper files

 Check D/R readiness of critical suppliers, vendors



Automation: Critical for successful rapid recovery & continuity

22

 The benefits of automation:

– Allows business continuity processes to be built on a reliable, consistent recovery time

– Recovery times can remain consistent as the system scales to provide a flexible solution designed to meet changing business needs

– Reduces infrastructure management cost and staffing skills

– Reduces or eliminates human error during the recovery process at time of disaster

– Facilitates regular testing to help ensure repeatable, reliable, scalable business continuity

– Helps maintain recovery readiness by managing and monitoring the server, data replication, workload and the network along with the notification of events that occur within the environment

Automate - Automate - Automate



Tivoli Storage Productivity Center for Replication (TPC-R)

Overview



Replication management solution

– Simplified replication management

& monitoring

– Powerful commands and logic

 Multiple Storage subsystems

– DS8000, DS6000, ESS800, SVC

 Multiple logical volume types

– Open systems (FB) LUNs

– z/OS (CKD) volumes

 Multiple replication types

– FlashCopy

–

Metro Mirror

– Global Mirror

–

Metro/Global Mirror



High performance and scalability

 TPC for Disk, TPC for Data and TPC for Fabric are not required but can coexist on the same server

– Shared instance of DB2

–

Shared SNMP port

23



TPC Replication Manager

• Setup Copy Sessions

• Execute Copy Operations

• Monitor Copy Status

• Manage/Monitor Consistent

Groups

• Alert Operations on Exceptions /

Failures

Primary/Source Site

TPC For

Replication

• DS6000, DS8000 support

• Global Mirror Support

• Replication Progression

Monitoring

• High Availability

• Disaster Recovery

Automation (failover, failback)

Second/Target Site

DS8000

SAN Volume

Controller

ESS

DS6000

DS8000

SAN Volume

Controller

ESS

DS6000

• Automated copy services configuration

• Central operations for copy services

• Operational status on copy services operations

• Assistance with recovery on failures

24



TPC for Replication GUI



My Work hyperlinks on left

 Display area for panels on right

 Select session, select action

(from dropdown list) and GO

 Tables with hyperlinks and sortable columns

 Health Overview on every panel

 Session view

– Triangle Indicates application access (active host)

– Arrows between roles indicate direction of active replication

25



TPC-R Video Series (new on Techdocs)

 Series of live demonstrations captured on video managing various environments

 Link to Summary of all the videos:

– http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS3687

 Series includes the following demonstrations:

– CLI vs. TPC-R – (2:13) – adding copysets using CLI and TPC-R

– TPC-R 3.3 GM – (17:36) – GM setup with FO/FB using TPC-R 3.3

– TPC-R 3.4 GM with Practice – (6:39) – GM with Practice Volumes setup with FO/FB

– TPC-R 4.1 overview – (14:24) – Overview of TPC-R with MM setup demonstration

– TPC-R 4.1 adding HW – (10:01) – how to add DS8000 and SVC to TPC-R

– TPC-R 4.1 MM setup – (19:20) – using TPC-R 4.1 to manage MM with FO/FB

– TPC-R 4.1 GM setup – (17:36) – Using TPC-R 4.1 to manage GM with FO/FB

– TPC-R 4.1 MM with Practice – (8:15) Using TPC-R 4.1 to create practice volumes

26


27


TPC-R Simplification: Starting a Global Mirror Copy

Using the DS8000 Hardware

Commands

1.

Determine where to place Master GM session given the

PPRC paths.

2.

Establish PPRC links between Master and Subordinate

DS8000’s.

3.

Establish PPRC paths between A and B volumes

4.

Establish Subordinate sessions on the A volumes of the DS8000’s

5.

Establish a GC relationship between A and B

6.

Query A to determine first pass complete

7.

Establish Flash copy between B and C with incremental

8.

Add A to the subordinate Global Mirror session

9.

If first A volume on this DS8000, then start the Global

Mirror Master with new configuration

 Monitor the Global Mirror Master with 051 queries and calculate RPO.

 Monitor for failures and fatal conditions

Using TPC-R

1.

Commands

START


28


TPC-R Simplification: Recover a Global Mirror Copy

Using the DS8000 Hardware

Commands

1.

Establish PPRC B to A Failover

2.

Query all B to C Flash Copy relationships and determine if they are revertible and have the same sequence number

3.

If the sequence numbers are all the same AND at least one relationship is not revertible, issue a “withdraw

Flash Copy with commit ” to all of the revertible relationships

4.

If all of the Flash Copy relationships are Revertible, issue a “withdraw

Flashcopy with revert” to all

Flashcopy relationships.

5.

Issue “establish Flashcopy C to B” with Fast Reverse Restore

1.

Using TPC-R

Commands

RECOVER



The right level of business continuity protection for your business….GDPS family of offerings

Continuous

Availability of Data within a Data Center

Continuous

Availability /

Disaster Recovery

Metropolitan Region

Disaster Recovery at

Extended Distance

Continuous

Availability Regionally and Disaster Recovery

Extended Distance

Two Data Centers Three Data Centers Single Data Center

Applications remain active

Near-continuous availability to data

Two Data Centers

Systems remain active

Automated D/R across site or storage failure

No data loss

Automated

Disaster Recovery

“seconds” of Data Loss

Data availability

No data loss

Extended distances

SDM A B

GDPS/PPRC

HyperSwap Manager

29

GDPS/PPRC


GDPS/PPRC

GDPS/GM

GDPS/XRC

C

GDPS/MGM

GDPS/MzGM



The right level of protection for your business – Distributed

Platforms

DR at extended distance

CA / DR within a metropolitan region

Two data centers - systems remain active; designed to provide no data loss

GDPS/PPRC

K-Sys

Rapid systems recovery with only ‘seconds” of data loss

GDPS/XRC

SDM

K-sys

Tivoli SA AppMan Platforms:

 IBM System p AIX 5.2, 5.3, 6.1, Linux: SUSE

SLES 9,10 RedHat RHEL 4,5

 IBM System x Linux: Suse SLES 9,10, RedHat

RHEL 4,5; Windows 2003,2008

 IBM System I Linux: Suse SLES 9,10, RedHat

RHEL 4,5

 IBM System z z/OS V1.7+, Linux: Suse SLES

9,10, RedHat 4,5

 VMWARE ESX Win ServerLinux: Suse SLES

9,10, RedHat RHEL 4,5; Windows 2003, 2008

K-Sys

Ref IBM Tivoli System Automation 3.1 Installation &

Customization Guide in the Release notes for a more detailed reference on GDPS DCM Supported configurations.

Site-1

VCS or

SA AppMan

GCO

Site-2

VCS or SA AppMan &

GDPS DCM

Agent

Site-1

VCS GCO

Site-2

VCS and

GDPS DCM

Agent Symantec VCS Platforms:

 IBM System P & pHype - AIX 5.3

 IBM System x (Intel / AMD x86_64) - Suse SLES

9 & RH 4

 HP (Itanium / PA RISC) – HP-UX 11.23.

 SUN (SPARC) – Solaris 9 & 10.

 VMWare ESX 3.0 (Intel / AMD x86_64) - Suse

SLES 9 & RH 4 & Windows AS & Windows 2300.

30


31


Case Study #1

Bandwidth Requirements



Data Replication Case #1 - Summary

32

 Global Mirror consistency group formation fails

– Drain time exceeded

– Suspended volumes

– Link incidents (Frame transmission retries <1%)

 Configuration Details

–

Two 9 Mbit links between sites

– 80 volumes mirrored

– 3:1 compression on links

– PPRC ports on the same HBA

 Operational Details

– Script written to monitor Out-of-Sync Tracks

– TPC for Disk used to provide performance data

– TPC-R used to automate Global Mirror

– Started with a few mirrored volumes – no issues with Global Mirror

– Increased number of volumes and workload

• Noticed host impact due to slow links

•

Switched to Global Copy

• Large number of OOS Tracks


33


Data Replication Case #1 - Bandwidth Requirements

 The following chart shows the mirrored write MB/s profile with indications of the Link Bandwidth required

– 3:1 Compression and 80% Link efficiency

– Exceeding the bandwidth will result in higher RPO

 Workload values above the “9 Mb” links indicate that the capacity of the link is exceeded

– 1-3 Distance Links do not handle the workload

–

4- Distance Links are sufficient for the current workload

• With some longer RPO times during peaks

• Should suspend GM during large peaks

– 8-12- Distance Links would be needed for the 25 MB/s peak

 Available bandwidth is 2 links

– Not dedicated

– Link timeouts occur when over-driven


34


4 Links = min required bandwidth

2 Links = available bandwidth


35


Data Replication Case #1 – Out-of-Sync Tracks

 The 1st chart shows the PPRC Link activity

– Available Link bandwidth was insufficient

– Over driving the links resulted in redriving frames (timeout)

– Delays resulted in full track transfers instead of sending data from cache

–

Full track transfers increased the bandwidth requirement

 The 2nd chart shows the OOS Tracks and suspended volumes

– Peak workload occurred at 2:00-3:00 am on most days

– Large number of OOS Tracks could not be copied before the next peak

– OOS Tracks must be low to for Consistency Groups to form

– OOS Tracks fully copied about 2 times per week


36

CGs possible only when links are not overdriven


Overdriving links causes delays

Full Track Transfers increase required bandwidth


37


Overdriving links causes OOS Tracks and suspended volumes

Cannot “catch-up” before the next peak

CGs possible only when

OOS

Tracks are low


38


Data Replication Case #1 – Conclusion

 Bandwidth equivalent to 4 – “9 Mbit” links to handle the workload

– Sufficient for the “normal” workload

– May need to suspend Global Mirror during large peaks (measured 25

Mb/sec)

– Should consider future growth

 Follow Best Practices for Data Replication

– Links should be dedicated to guarantee required bandwidth and reduce link timeouts.

– PPRC Ports should be on dedicated cards and

• Do not share cards with host activity

• Do not put both links on the same HBA (single point of failure)

• 4 links on 2 HBAs would be preferred

–

Monitor Status and Performance

• Use TPC for Disk to monitor link activity and workload growth

• Do not ignore TPC-R messages

• Test DR procedures


39


Case Study #2

Performance Requirements


40


Data Replication Case #2 - Summary

 Global Mirror consistency group formation fails

– Drain time exceeded

– Host performance impacted while GM is active

 Configuration Details

–

Distance between sites

– Link information

– Primary DS8000

– Secondary DS8000

37 miles / 60 km

GigE = 100 MB/sec

300G 15K Drives Raid 5

300G 15K Drives Raid 5

 Operational Details

– TPC for Disk used to provide performance data

– TPC-R used to automate Global Mirror

– Increased workload since initial GM Design

• Noticed host impact when GM was implemented

•

Switched to Global Copy


41


Data Replication Case #2 – Performance & Bandwidth

 Bandwidth analysis

– Available bandwidth is not the cause of Global Mirror issues

– Usually 20-30 MB/sec short peaks 50, 90 MB/sec

– GigE Link can handle 100 MB/sec without compression

 Current Global Mirror Configuration Follows Best Practices

– Volume placement is not the cause of performance issues

–

PPRC ports are separate from Host ports (good)

•

Should use every other port on HBA card for best performance

– Volumes spread over all Arrays (good)

•

PPRC volumes share arrays with Flash Copy targets

• Do Not have the FlashCopy source and target in the same array

•

Target should be in the came cluster

 Performance analysis

– Primary disk utilization is too high without any Mirror activity

– Secondary disk utilization is at the maximum when Global Mirror is active

– Flash Copy activity adds to HDD utilization

– When Global Copy is active HDD utilization decreases

– Raid 10 will reduce the HDD Utilization (double number of arms)


42


Global

Mirror

Global

Copy


43


Global Mirror

Warning level

Caution level

Part 1

06-21 14:20 to 06-22 13:40


44


Global Copy

Part 2

06-22 15:35 to 06-23 14:45


45


Global Mirror

GM Flash Copy adds to

HDD activity

Utilization is too high!

Part 1

06-21 14:20 to 06-22 13:40


46


Global Copy

Part 2

06-22 15:35 to 06-23 14:45


47


Data Replication Case #2 – Conclusion



Previous Disk Magic Study Showed Good Results

– Original workload

• 4,800 I/O per second HDD Utilization 35%

– However the workload has grown

• 7,300 I/O per sec HDD Utilization >90%

• Other systems were added

 HDD Utilization is too high

– This condition will cause Host performance issues even without Global Mirror active

– Overloaded arrays at remote site causes Host performance issues when

Global Mirror is active

– Added copy activity at the primary site causes Host performance issues when Global Copy is active

 Current workload has reached the limit of the current configuration

– Need to spread the workload over more arrays and use Raid 10

– Need to plan for future growth

– Continue to use TPC Standard Edition to monitor performance



Further Assistance

 DS8000 Architecture & Best Practices - Replay



TIME Given: August 13; 11:00 a.m. New York, 4:00 p.m. London, 5:00 p.m. Paris, 15:00:00 GMT

 IBMers: http://w3.ibm.com/sales/support/ShowDoc.wss?docid=Q824765C05775Z31&infotype=SK&infosubtype=N0&node=doctype,N0|doctype,

TLC|brands,B5000|brands,B8S00&appname=CC_CFSS

 BPs: http://www.ibm.com/partnerworld/wps/servlet/ContentHandler/tsee081309km

 Contact ATS:

 Business Partners:

PARTNERWORLD CONTACT SERVICES (US & Canada)

1-800-426-9990 or fill out a request online: http://ibm.com/partnerworld/techline

 IBMer’s: Open Techline Request

48


49


Additional References


50


Data Replication Enabling Core Technologies -

FlashCopy

Internal Copy

Available on:

DS6000, DS8000, ESS

SAN Volume Controller, XIV,

DS4000, DS5000,

PiT Incremental FlashCopy

+ Metro Mirror

Available on:

DS6000, DS8000, ESS

Copy data command issued

Copy is immediately available

Time

Source

Write

Target

Read Read and write to both source and copy possible

Primary

Prod

PiTCopies

PiT

Incrementa l

FlashCopy

Global Copy

(Asynchronous

)

WAN


REMOTE

When copy is complete, relationship between source and target ends

PPRC-XD

Secondar y

FlashCop y

PiT

Incremental

Copy



DS8000 FlashCopy Options

 Once Source/Target “Logical” relationship established – Both volumes are available for Read/Write.

 Multiple Relationships - Single Source may have up to 12 Targets.

 Background Copy Optional

– NoCopy to Copy – Background Copy

 Persistent & Incremental

 Consistent FlashCopy (across volumes in single DS8K or across multiple

DS8Ks)

 Target device may be in any LSS

 Space Efficient FlashCopy (single FC repository for all target volumes)

– For Backup typically requires 10-20% of actual space.

 Remote Pair FlashCopy

51



52

IBM HyperSwap Technology -> Higher Availability for Parallel

SYSPLEX !

 Ability to swap enterprise class System z Disk Subsystems in seconds.

 HyperSwap substitutes Metro Mirror secondary for primary device

 No operator interaction,

 Designed to scale to multi-thousands of z/OS volumes

 Includes volumes with SYSRES, page data sets, catalogs



Non-disruptive - applications keep using same device addresses



HyperSwap integration with z/OS yielding Higher Availability for z/OS.

application

P

UCB

Metro

Mirror

UCB

S

 Basic HyperSwap (GA 2008)

• Single site continuous availability function

• Unplanned failures

• Planned fail over (testing)

•

Aimed at masking disk failures

• IBM Disk Subsystems ONLY (ESS 800, DS6000, DS8000)

 GDPS/PPRC HyperSwap Manager (GA 2006)

• Single site or multiple sites

• Continuous availability and/or Entry level DR solution

• Any Vendors disk subsystem that supports the IBM PPRC Architecture (ex. IBM,

EMC, Hitachi (HP & SUN)

 GDPS/PPRC w/HyperSwap (GA 2002)

• Full function HyperSwap across multiple Sites for D/R and High Availability.

•

Includes Server, Workload & Network Management across sites in addition to

Storage

• Supports GDPS/MzGM and GDPS/MGM environments.

•

Any vendors disk subsystem that supports the IBM PPRC Architecture. (ex. IBM,

EMC, Hitachi (HP & SUN)



DS8000 Global Mirror:



Concept



Asynchronous long distance copy (Global Copy), i.e., little to no impact to application writes



Verify End to End Data Integrity



CRC sent & verified w/each changed Block/record.



GM detects dropped FCP Frames



For ECKD devices track format also sent & verified in Metadata.

 CG “Marked” at primary, but CG is formed at Target Site, yields continuous BW utilization.



Momentarily pause application writes (fraction of millisecond to few milliseconds)



Create point in time consistency group across all primary subsystems (in OOS bitmap)



New updates saved in Change Recording bitmap



Restart application writes and complete write (drain) of point in time consistent data to remote site



Stop drain of data from primary (after all consistent data has been copied to secondary)



Logically FlashCopy all data (i.e., 2 nd ary is consistent, now make tertiary look like 2 nd ary)



Restart Global Copy writes from primary



Automatic repeat of sequence every few seconds to minutes to hours (selectable and can be immediate)



Intended benefit

 Long distance, no application impact (adjusts to peak workloads automatically), small RPO, remote copy solution for zSeries and Open Systems data, and consistency across multiple subsystems

Global Copy (PPRC-XD) over long distance

Could require channel extenders

FCP links only

(Asynchronous)

Global Copy

FlashCopy

(record, nocopy, persistent, inhibit target write)

Secondar y

Tertiary

Remote

Sit e


53

Host

I/O

Primar y

Loca lSit e


Metro/Global Mirror architecture

54

Server or Servers

4 normal application I/Os

***

Global Mirror network asynchronous large distance

Global Mirror FlashCopy

NOCOPY

A

1

Metro Mirror

2

3

B a b

C c

D

Metro Mirror network synchronous small distance

Metro Mirror write

1.

application to VolA

2.

VolA to VolB

3.

write complete to A

4.

write complete to application

Global Mirror

Global Mirror consistency group formation (CG) a.

write updates to B volumes paused (< 3ms) to create CG b.

CG updates to B volumes drained to C volumes c.

after all updates drained, FlashCopy changed data from C to D

Local Site (Site A) Remote Site (SiteC)



Integrated solutions

GDPS/MGM

GDPS/GM

GDPS/PPRC

GDPS/XRC

GDPS/PPRC


RCMF/PPRC RCMF/XRC

GDOC Veritas

Clusters + IBM MM, GM or EMC SRDF, SRDF/A or VVR

AIX/HACMP 5.1 +

Metro Mirror

Windows GeoDistance

MSCS + Metro-Mirror

Scripts

Delivered by IBM Global Services Delivered by IBM Global Services

FlashCopy -

Point in time copy available on DS8000™, DS6000™, and ESS

Features:



Multiple relationships - Single Source may have up to 12 Targets.



Background Copy optional



NoCopy to Copy

– Background Copy



Persistent and Incremental



Consistent FlashCopy



Target device may be in any LSS

 PiT Inc FlashCopy to MM, GC, or GM Primary

Tim e

Source

Write

Target

Read

Read and write to both source and copy possible

Metro Mirror

- Synchronous mirroring available on DS8000, DS6000, and ESS

Designed to provide:



No data loss



Industry leading replication performance

 High Availability with GDPS™ HyperSwap™:

 System z™ and open systems data



Ease of use, lower cost

Req (1)

Ack (4)

IB

M Host


When copy is complete, relationship between source and target ends

VOLUME

(2)

A

Confirm I/O write (3)

VOLUME

B

PRIMARY

SECONDARY

55

TECHNICAL SALES SUPPORT AMERICAS

DS6000/DS8000 -

Data Replication

Data Replication Options:

• FlashCopy® (FC) (within Box)

• Metro Mirror (MM/PPRC) (Sync Copy)

• Global Copy (GC) Async Copy, No

Consistency

• PiT Incremental FC to MM, GC, or GM

Primary

• Global Mirror (GM) Async Copy

• Global Mirror for zSeries® (zGM)

• Metro Global Mirror (MGM)

• zGM + MM/PPRC Multi-Target (MzGM)

Managed by:

• TotalStorage® Productivity Center –

Replication Manager (TPC-R)

• Geographically Dispersed Parallel

Sysplex (GDPS)

(See integrated solutions on inside flap.)

For more information, please contact your local IBM Representative or IBM

Business Partner.

May 2008



Global Mirror

- Asynchronous mirroring available on

DS8000™, DS6000™, and ESS

DS6K/DS8K/ESS key design to points:

 Capability to achieve an RPO of 3-5 seconds with sufficient bandwidth and resources

 Do not impact production applications when insufficient bandwidth and/or resources are available



Scalable; providing consistency across multiple primary and secondary disk subsystems

 Allow for removal of duplicate writes within a consistency group before sending data to remote site

 Allow for less than peak bandwidth to be configured by allowing RPO to increase without restriction at peak times

 Provide consistency between System z and open systems data and between different platforms on open systems.

Metro / Global Mirror – Three-site synchronous and asynchronous mirroring

Available on DS8000


 Performance, scalability

 Metro Mirror, Global Mirror

 Satisfy all 3-site requirements:

 Fast failover / failback to any site

 Fast re-establishment of threesite recovery, without production outages

 Resynchronize any site with incremental changes only*

 Ease of use, autonomic, self-monitoring

Metro

I/O Write

A

Global

I/O Write

B

'A‘

Primary PRIMARY

HOSTS

REMOTE

HOSTS

Native performance

FlashCopy

SA

N

DS8000, ESS

Global Mirror

–

SA

N

‘ B’

Global Copy

Secondary

Consistent

Data

C

© International Business Machines Corporation, 2007.

IBM, the IBM logo, GDPS, Global Dispersed Parallel

Sysplex, HyperSwap, z/VM, z/OS, z.VSE, System z, are trademarks of International Business Machines Corporation in the United States, other countries or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United

States, other countries, or both.

Other company, product and service names may be trademarks or service marks of others.

56 z/OS Global Mirror -- Asynchronous mirroring for System z available on DS8000 and ESS 800


 Premium performance and scalability

 Data moved by DFSMS™ System Data

Mover (SDM) address space(s) running on z/OS.



Supports heterogeneous disk subsystems



Unlimited distances



Time consistent data at the recovery site



RPO within seconds

 Supports System z™ and System z Linux® data



Over 200 installations worldwide



3-

Site GDPS®/PPRC HyperSwap and

GDPS/XRC “Multi-Target” Supported

Metro / zGlobal Mirror Multi-Target

(MzGM) -

Three-site synchronous and asynchronous mirroring Available on

DS8000


•

Performance, scalability

•

Metro Mirror, zGlobal Mirror

• Satisfy all 3-site requirements:

• HA and CA capability for Metro distance

•

Out of region DR with fast recovery

•

Resynchronize any site with incremental changes only*

•

Ease of use, autonomic, self-monitoring

Primary

Host

System

Data

Mover

System

Data

Mover

Secondary

Host

2

Primary

DS8000

1

4



DS6K/DS8K Copy Services Matrix

May

Device

Is

Become

GMz 10

(XRC)

Primary

GMz 10

(XRC)

Secondary

Metro Mirror or

Global Copy

Primary

Metro Mirror or Global

Copy

Secondary

Global

Mirror

Primary

Global Mirror

Secondary

GMz 10

(XRC)

Primary

No Yes 11 Yes No Yes No

GMz 10

(XRC)

Secondary

Yes 11 No Yes No 5 Yes No 5

Metro Mirror or Global

Copy Primary

Yes Yes No Yes 1 No 6 Yes 1

FlashCopy

Source

Yes

Yes

Yes

FlashCopy

Target

Incremental

FLC Source

Yes

No 5

Yes

No

Yes

Yes

Incremental

FLC Target

Concurrent

Copy

Source

No

No 5

No 6

Yes

Yes

Yes

No 5 Yes 1 Yes 1 Yes 8 Metro Mirror or Global

Copy

Secondary

Global Mirror

Primary

Global Mirror

Secondary

FlashCopy

Source

FlashCopy

Target

Incremental

FLC Source

Incremental

FLC Target

Concurrent

Copy Source

No

Yes

No

Yes

No

No 7

No 7

Yes

Yes

No 5

Yes

No 5

Yes

No 5

Yes

No 6

Yes 1

Yes

Yes 2

Yes

Yes 2

Yes

No

Yes 1

No

Yes

No

Yes

No

No

No

No

Yes

No

Yes

No

Yes

No

No

No

Yes

No

Yes 9

No

No

Yes

Yes

Yes

Yes 3,4

Yes 4

Yes

No

Yes

Yes

Yes 8

Yes 4

Yes 4

No

No

Yes

Yes

Yes

Yes 9

Yes 3

No

No

No

Yes

Yes

Yes

No

No

No

No

No

Yes

No

Yes

No

Yes

No

Yes

Yes

Yes

57



DS6K/DS8K Copy Services Matrix Notes

1.

Only in a Metro/Global Copy (supported on ESS) or a Metro/Global Mirror Environment

(supported on ESS and DS8000).

2.

FlashCopy V2 at LIC 2.4.0 and higher on ESS800 (DS6000 and DS8000 utilize FlashCopy V2 by default).

– You must specify the proper parameter to perform this

– Metro Mirror primary will go from full duplex to copy pending until all of the flashed data is transmitted to remote

– Global Mirror primary cannot be a FlashCopy target

3.

FlashCopy V2 Multiple Relationship.

4.

FlashCopy V2 Data Set FlashCopy (only available for z/OS volumes).

5.

The Storage Controller will not enforce this restriction, but it is not recommended.

6.

A volume may be converted between the states Global Mirror primary, Metro Mirror primary and Global Copy primary via commands, but it two relations cannot exist at the same time (i.e. multi-target).

7.

GMz (XRC) Primary, Global Mirror Secondary, Incremental FlashCopy Source and Incremental

FlashCopy Target all use the Change Recording Function. For a particular volume only one of these relationships may exist.

8.

Updates to the affected extents will result in the implicit removal of the FlashCopy relationship, if the relationship is not persistent.

9.

This relationship must be the FlashCopy relationship associated with Global Mirror – i.e. there may not be a separate Incremental FlashCopy relationship.

10.

Global Mirror for zOS (GMz) is supported on ESS and DS8000

11.

In order to ensure Data Consistency, the XRC Journal volumes must also be copied.

58


59


Reference Resources


60


References

 “Global Mirror Whitepaper”, V1-3

– By Nick Clayton, 13/09/2005

– http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100642

 z/OS DFSMS Advanced Copy Services

– SC35-0428, January 2006

 DSx000 Commandline Interface User’s Guide

– DS6000  GC26-7922, September 2006

– DS8000  SC26-7916, November 2006

 Device Support Facilities User’s Guide and Reference, Release 17

– GC35-0033, March 2005

 Redbooks

– www.redbooks.ibm.com

or w3.itso.ibm.com

– Search on “DS6000” & then “DS8000”

• Many choices available

– Also search on “copy services”

• Again, many choices available



To COPY or to NOCOPY?…. That is the question!

 BACKGROUND NOCOPY is typically the best choice to minimize rank and DA activity within the physical box

 But…. You must ask why are you making a copy? And…. What type of application workload do I have?

– For example:

•

Is the copy only going to be used for creating a tape backup?

–

BACKGROUND NOCOPY should be used and the relationship withdrawn after the tape backup is complete

• Is the copy going to be used for testing or development?

–

NOCOPY again is typically the best choice

•

Will you need a copy of the copy?

– BACKGROUND COPY must be used so that the target will be withdrawn from its relationship after all of the tracks are copied thereby allowing it to be a source in a new relationship

> Possibly use NOCOPY to COPY option

• Is the workload OLTP (NOCOPY typically is the choice) or are there a large number of random writes and are not cache friendly (COPY may be the better choice)

61



References

 SC26-7916 DS8000 CommandLine Interface User’s Guide

 GC26-7922 DS6000 CommandLine Interface User’s Guide

 SC35-0428 DFSMS Advanced Copy Services

 SG24-6788 IBM System Storage DS8000 Series: Copy Services in Open

Environments

 SG24-6787 IBM System Storage DS8000 Series: Copy Services with IBM

System z

 SG24-6783 IBM System Storage DS6000 Series: Copy Services in Open

Environments

 SG24-6782 DS6000 Series: Copy Services with IBM System z Servers

 GC350033 Device Support Facilities User’s Guide and Reference, Release 17

62



References

63

 “Global Mirror Whitepaper”, V1-3 By Nick Clayton, 13/09/2005 WP100642


 Performance White Paper

– http://w3-

1.ibm.com/sales/systems/portal/_s.155/254?navID=f320s260&geoID=All&prodID=Syst em%20Storage&docID=ditlDS8000PerfWPPower5

 DS8000/DS6000 Copy Services: Getting Started WP100905


 DS8000 Disk Mirroring Licensing - Frequently Asked Questions

– http://w3-

1.ibm.com/sales/systems/portal/_s.155/?navID=f220&geoID=AM&prodID=System%20

Storage&docID=ditlDS8000MirroringLicFAQ

 Redbooks

– www.redbooks.ibm.com

or w3.itso.ibm.com

• Search on “DS6000” & then “DS8000” Also search on “copy services”

– IBM System Storage DS8000 Series: Copy Services in Open

Environments , SG24-6788-02

Redbook, published 29 November 2006

– IBM System Storage DS8000 Series: Copy Services with IBM System z ,

SG24-6787-02

Redbook, published 14 December 2006



References

 Technical information on IBM TotalStorage Business Continuity Solutions:

– IBM ITSO Redbook: TotalStorage Business Continuity Solutions Guide SG24-6547-02 :

• http://www.redbooks.ibm.com/abstracts/sg246547.html?Open

– IBM ITSO Redbook: TotalStorage Business Continuity Solutions Overview SG24-6684

:

• http://www.redbooks.ibm.com/redbooks/pdfs/sg246684.pdf

 Technical ITSO Redpapers on Business Continuity Solutions:

– REDP4062 TotalStorage Business Continuity Solution Selection

Methodology

• This ITSO Redpaper describes an IT Business Continuity Solution Selection

Methodology and how to apply it to your computing environment. There are several scenarios which demonstrate the application of this methodology.

– REDP4063 Planning for Heterogeneous Platform BC and DR

•

This ITSO Redpaper discusses how to plan for IT Business Continuity in a highly heterogeneous platform server and storage installation. Discussion is included of of IBM storage-based tools that can be useful to provide Business Continuity and

Disaster Recovery in this diverse environment.

• It is a 2005 version of this presentation

64



References

 Technical ITSO Redpapers on Business Continuity Solutions:

– REDP4064 Small and Medium Business Considerations for IT

BUsiness Continuity

• Small and Medium Business (SMB) enterprises have IT Business Continuity needs and concerns similar to large enterprises; yet in other ways, SMB companies have key differences. This redpaper discusses those differences, and describes an overview of IT Business Continuity Solution selection in the SMB business environment.

– REDP4066 Networking Tutorial for IT Business Continuity Planners

• Confused by terms such switches, routers, bridges, OC3, DWDM, dark fibre?

• This Redpaper is intended to enable the IT Business Continuity planner to better understand many commonly used networking concepts, in order to be able to better evaluate and select the appropriate networking components for your IT Business

Continuity solution.

65



References

 IBM Implementation Services for Geographically Dispersed Open Clusters

(GDOC)

– http://www-

03.ibm.com/servers/storage/solutions/business_continuity/continuous_availability/technical_details.h

tml

 IBM Geographically Dispersed Parallel Sysplex:

– http://www-

03.ibm.com/servers/storage/solutions/business_continuity/continuous_availability/technical_details.h

tml

 IBM System Storage Business Continuity Solutions website:

 http://www-03.ibm.com/servers/storage/solutions/business_continuity/

 IBM Global Services Business Continuity and Recovery Services:

 http://ibm.com/services/continuity/

 IBM Business Continuity and Recovery Services

– Your local IBM Global Services ITS representative

66


67


System Storage Enterprise Disk Practices Resources

System Storage Business Continuity Solutions website

 http://www-03.ibm.com/servers/storage/solutions/business_continuity/index.html

System Storage Technology Center

 http://www-03.ibm.com/system/storage/

 Storage Education http://www-03.ibm.com/systems/education/cust/crossprod/custcp.html

System Storage Interoperation Center

 http://www-01.ibm.com/systems/support/storage/config/ssic/index.jsp

System Storage Services

 http://www-03.ibm.com/systems/storage/services/index.html

Redbooks/Redpapers



 http://www.redbooks.ibm.com/redbooks.nsf/portals/Storage

The IBM TotalStorage DS8000 Series: Concepts and Architecture (SG24-6452-00)









IBM System Storage Business Continuity Solutions Overview (SG24-6684-01)

IBM System Storage DS8000 Series: Copy Services with IBM System z (SG24-6787-02)

IBM System Storage DS8000 Series: Copy Services in Open Environments (SG24-6788-02)

IBM System Storage Solutions Handbook (SG24-5250-06)

White papers





IBM Storage Infrastructure for Business Continuity Solution

Global Mirror Technical Whitepaper



References

 IBM System Storage Business Continuity Solutions website:

 http://www-03.ibm.com/servers/storage/solutions/business_continuity/

 IBM Global Services Business Continuity and Recovery Services:

 http://ibm.com/services/continuity/

 IBM Business Continuity and Recovery Services

– Your local IBM Global Services ITS representative

68



Following are the IBM links to presentations on the

Enterprise Disk Mirroring Sales Kit on System Sales Web

Site

 IBM Three Site Enterprise Disk Mirroring Executive Summary

 IBM Three Site Mirroring Competitive Marketing Summary for IBM*

 IBM Two and Three Site Enterprise Disk Mirroring Overview

 IBM Three Site Disk Mirroring for Open Systems Presentation Guide

 IBM Three Site Disk Mirroring for zSeries Presentation Guide

 IBM z/OS Global Mirror and Global Mirror Positioning Guide

 IBM Two and Three Site Disk Mirroring Technical Presentation Guide

 Deep Dive on IBM Global Mirror – from US Storage Symposium 2005

 IBM Disk Mirroring Update – from US Storage Symposium 2005

 IBM DS8000 DS6000 ESS Disk Mirroring Link Efficiency TCO Studies*

 IBM Two and Three Site Mirroring Competitive Marketing Guide*

69

* IBM Confidential documents



Trademarks

The following are trademarks of the International Business Machines Corporation in the United States and/or other countries.

CICS*

ClearCase

DB2* e-business logo

FICON*

GDPS*

HyperSwap

IBM*

IBM eServer

IBM logo*

IMS*

MQSeries*

On Demand Business logo

* Registered trademarks of IBM Corporation

Parallel Sysplex*

Rational*

System z9

Tivoli*

WebSphere* z/OS* z/VM* zSeries*

The following are trademarks or registered trademarks of other companies.

Intel is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States, other countries or both.

Microsoft is a trademark of Microsoft Corporation in the United States, other countries, or both.

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

* All other products may be trademarks or registered trademarks of their respective companies.

Notes :

Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here.

IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.

All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.

This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.

All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

70



Disclaimers

Copyright © 2009 by International Business Machines Corporation.

No part of this document may be reproduced or transmitted in any form without written permission from IBM Corporation.

Product data has been reviewed for accuracy as of the date of initial publication. Product data is subject to change without notice. This information could include technical inaccuracies or typographical errors. IBM may make improvements and/or changes in the product(s) and/or programs(s) at any time without notice.

Any statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such such products, programs or services available in all countries in which IBM operates or does business. Any reference to an IBM Program Product in this document is not intended to state or imply that only that program product may be used. Any functionally equivalent program, that does not infringe IBM’s intellectually property rights, may be used instead. It is the user’s responsibility to evaluate and verify the operation of any on-IBM product, program or service.

THE INFORMATION PROVIDED IN THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR

IMPLIED. IBM EXPRESSLY DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR

NONINFRINGEMENT. IBM shall have no responsibility to update this information. IBM products are warranted according to the terms and conditions of the agreements (e.g., IBM Customer Agreement, Statement of Limited Warranty, International Program License

Agreement, etc.) under which they are provided. IBM is not responsible for the performance or interoperability of any non-IBM products discussed herein.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents or copyrights. Inquiries regarding patent or copyright licenses should be made, in writing, to:

IBM Director of Licensing

IBM Corporation

North Castle Drive

Armonk, NY 10504-1785

U.S.A.

71
