New approach to the design and deployment of storage Piotr Biskupski – IBM STG Poland IBM Forum 2012 – Estonia Tallinn, October 9, 2012 To realize the promise of a Smarter Planet, enterprises must address the IT challenges that have emerged. 31.9 million servers worldwide Internet connected devices— 22B in 2020 (24.9% CAGR) Last 10 years servers up 2X and virtual machines up 42% CAGR Since 2000 security vulnerabilities grew eightfold The volume of digital content, up 48% from 2011, is expected to reach 2.7 ZB (1ZB = 1 billion terabytes) in 2012 and 8ZB by 2015. Data centers have doubled energy use in past 5 years And respond to demands for new services… Real-time analytics workloads growing 13% CAGR with related server spend growing 3x average 2 Collaborative software workloads growing 10% CAGR …while dealing with constraints like a flat budget. IBM Storage is key to Smarter Computing Storage Virtualization enables a robust cloud deployment Designed for Storage Efficiency is datato outpace essential data growth Smarter Computing Storage must adapt to changing information patterns Introducing IBM Smarter Storage for Smarter Computing Efficient by Design Manage cost and capacity growth Self-optimizing Improve performance and productivity Cloud Agile Increase information access and improve ROI Efficient by Design Manage Cost and Capacity Growth Decreases Space Requirements • Store more information in less space • Supports non-IBM storage, savings throughout the data center Reduce Capital Expenses Simplifies Administration • Reduce complexity Reduce Operating Expenses • Automate tasks Efficiency for Active Workloads • Efficiency for transaction processing, big data, clouds • Not just for inactive data Benefits • Reduce space requirements for active data up to 80% • Save up to 47% in administrator time • Reduce complexity by up to 30% Self-optimizing Improve Performance and Productivity Analyzes and adapts • Analyzes access patterns and adapts automatically • Flash optimization makes high performance affordable Automates performance tuning • Saves time, reduces errors and eliminates user disruption • Easy to match critical workloads with performance features Analyze Big Data Fast • Tuning and tiering are faster and more accurate • Optimization is a continuous automated process Benefits • Increase performance 3X with only 3% SSDs • Reduce storage costs by 50% Optimize performance and cost Increase administrator productivity Cloud Agile Increase Information Access and Improve ROI Adaptive • Storage adapts easily to unpredictable workloads • Data is automatically synchronized between facilities Increase Agility Automated • Self-service provisioning • Simpler and faster administration Increase productivity Deploy Clouds Faster • Administrators manage far more capacity • Leverage existing storage systems Benefits • Reduce disk space needs by up to 50% • Speed storage deployment up to 26% • Improve application availability by 29% Tying IBM Smarter Storage to New Capabilities and Vision: June Cross-IBM Announcements and SoDs Efficient by Design Real-time Compression integrated into Storwize V7000, SVC TPC improved GUI, enhanced reporting, tiering optimization, cloud support SONAS TPC support TPC now supports XIV/Storwize V7000 GUI ProtecTIER file protocol support Storwize V7000U real-time compression for file workloads (SoD) Cloud Agile Self-optimizing DS8000 Easy Tier extended to server based storage (SoD) DS8000 new high-density flash storage option (SoD) DS8000 Easy Tier support for application API (SoD) IBM SmartCloud Virtual Storage Center will introduce analytics-based storage tier recommendations and automation to migrate data to the right tier (SoD) SONAS and Storwize V7000 file storage virtualization (SoD) SONAS and Storwize V7000 ACE enables global file collaboration with multi-writer Informationsupport (SoD) SONAS and Storwize V7000 Unified, cloud Governance storage self-service service portal (SoD) Storwize V7000 support for Active Cloud Engine (SoD) XIV clustering (SoD) XIV Gen3 GTS Implementation Services IBM Smarter Storage Virtualization Thin Provisioning Compression RACE Easy Tier • option: TPC for Disk Midrange Edition FlashCopy • option: FlashCopy Manager Advanced copy services - Metro and/or Global Mirror • option: TPC for Replication Active Cloud Engine Storage virtualization 25% capacity SAN 95% capacity 50% 55% capacity capacity Storage virtualization with HA …no walls Virtual DC… Virtualized strorage VMmachine Thin-provisioning Traditional (“fully allocated”) virtual disks use physical disk capacity for the entire capacity of a virtual disk even if it is not used With thin-provisioning, Storwize V70000 allocates and uses physical disk capacity when data is written Dynamic growth Without thin provisioning, pre-allocated space is reserved whether the application uses it or not With thin provisioning, applications can grow dynamically, but only consume space they are actually using Available at no additional charge with base virtualization license Support all hosts supported with traditional volumes and all advanced features (EasyTier, FlashCopy, etc.) Reclaiming Unused Disk Space – When using Volume Mirroring to copy from a fully-allocated volume to a thin-provisioned volume, Storwize V7000 will not copy blocks that are all zeroes – When processing a write request, Storwize V7000 detects if all zeroes are being written and does not allocate disk space for such requests in the thin-provisioned volumes ● Helps avoid space utilization concerns when formatting Volumes Done at Grain Level (32/64/128/256KiB) If grain contains all zeros don’t write Real-Time Compression • A compressed Volume is a third type of volume – Regular (fully allocated) – Thin Provisioned (Space Efficient) – Compressed • A compressed volume is “a kind of” Thin Provisioning – Compression supports all Thin Provisioning features (e.g. autoexpand) – Only uses physical storage to store the compressed version of the volume – Can be on back-end virtualized storage Storwize V7000 R W W – Supported with CLI and GUI • GUI also has a preset for compressed volumes • Performance monitoring tab support for Compression CPU resource usage • Compression operates at a volume copy level – Maximum of 200 compressed Volume copies per IO group Copy 0 • Specified at creation of the volume copy – No conversion in place, but a volume can be converted via Volume Mirroring1 • Compressed volumes can be source and target of Remote Copy relationship or FlashCopy mapping – Data between copies is uncompressed 13 Copy 1 Real-Time Compression Compression Ratio 100% 80% 2.0x Backup lic. 60% 2.5x data 3.3x replication 40% 3.5x RTOs 6.6x backup 20% 20% do 60% 50.0% 70.0% 72.0% 85.0% 0% 14 Dane użytkowników Bazy danych CAD/CAM Engineering Serwery wirtualne Oil & Gas Orion (Oracle I/O Calibration Tool) is a standalone tool for calibrating the I/O performance for storage systems that are intended to be used for Oracle databases. The calibration results are useful for understanding the performance capabilities of a storage system, either to uncover issues that would impact the performance of an Oracle database or to size a new database installation IBM EasyTier Hot-spots Optimized performance and throughput Transparent reorganization What is Easy Tier? – A function that dynamically re- distributes active data across multiple tiers of storage class based on workload characteristics – The goal being to reduce response time – Users have automatic and semi-automatic extent based placement and migration management SSDs HDDs SSDs HDDs Automatic Relocation Hot-spots – – – – Optimized performance and throughput Why it matters? Solid State Storage has orders of magnitude better throughput and response time with random reads Full volume allocation to SSD only benefits a small number of volumes or portions of volumes, and use cases Allowing dynamic movement of the hottest extents to be transferred to the highest performance storage enables a small number of SSD to benefit the entire infrastructure Works with Thin-provisioned Volumes IBM EasyTier PRELIMINARY 240% from Original Brokerage transaction Application Transaction Throughput Easy Tier In Action Easy Tier Learning 12 hours (Elapsed Time) 18 Snapshots, backup etc. Backup, Recovery, Tests, FlashCopy • Full • Snapshot Target Backup Cascaded copy – copy from copy Reverse FlashCopy Vol1 FlashCopy target of Vol0 Vol0 Source Target Map 1 Vol2 FlashCopy target Source of Vol1 OR Map 2 2. Reverse FlashCopy 256 copies Source Map 4 Vol3 FlashCopy target of Vol1 Vol4 FlashCopy target of Vol3 Advanced Copy Services Volume Mirroring FlashCopy Metro Mirror Global Mirror Volume Mirroring Point-in-Time Copy Synchronous Mirror Consistent Asynchronous Mirror “outside the box” “outside the box” 2 close sites (<10Km) 2 close sites (<10Km) Warning, there is no consistency group Warning, this is not real time replication – Write IO response time doubled + distance latency – No data loss 2 close sites (<300 Km) Warning, production performance impact if inter-site links are unavailable, during microcode upgrades, etc. – Limited impact on write IO response time – Data loss – All write IOs are sent to the remote site in the same order they were received on source volumes – 1 source and 1 target volumes + change volumes on each site 2 remote sites (>300 Km) Vol0 R W Vol0’ Vol0’ Storwize V7000 Storwize V7000 Internal Storage 20 External Storage Internal Storage Source and target can have different characteristics and be from different vendors Source and target can be in the same cluster Remote Copy between SVC & Storwize V7000 is supported Non-disruptive Data Migration Traditional SAN 1. Stop applications 2. Move data 3. Re-establish host connections 4. Restart applications Storwize V7000 1. Move data Host systems and applications are not affected In both cases, if it is with a new back-end external storage, you need to… – – – – Cabling, Change the SAN zoning, Create arrays & create LUNs in the arrays, Assign those LUNs to the server or Storwize V7000 Once Managed Disk Groups are created using the assigned LUNs, Volumes will be migrated between MDG with just one command/click SAN Migration scenarios: – – – – Image mode to image mode Image mode to managed mode Managed mode to managed mode Managed mode to image mode SAN Storwize V7000 Migrate on-line to decommission, up-Tier, down-Tier, re-Tier… Image mode = Native mode Managed mode = Virtual mode HARDWARE COMPONENTS IBM Storwize v7000 unified innovative GUI Snapshot SAN Easy Tier virtualization RAID Unified IBM Storwize V7000 IBM Active Cloud Engine™ Cluster system* NAS Storwize V7000 Overview Modular Hardware Building Blocks Software inherited from prior Offerings plus Enhancements Software inherited from SVC and DS8000 RAID New and enhanced Software functions •FCP, iSCSI & FCoE •RAID 0, 1, 5, 6, 10 •Storage Virtualization (Internal & external disks) •Non-disruptive Data Migration •Global & Metro Mirror (Multi-site) New! •FlashCopy (Point-In-Time Copy) •Thin Provisioning •Real Time Compression •New GUI (Easy-to-use, web based, XIV like) •Easy TierTM SSD exploitation •RAID & enclosure RAS services and diagnostics •Additional scalability improvements •Enhanced Host, Controller and ISV interoperability •Integration with IBM Systems Director •Enhancements to TPC, FCM and TSM support Enclosure Front View 2076-112/212/312 – 12 x 3.5” LFF Drive Bays – Four columns of three horizontally mounted drive carriers – Drive slots are numbered 1 to 12 starting top left and going left to right and top to bottom 2076-124/224/324 – 24 x 2.5” SFF Drive Bays – One row of 24 vertically mounted drive carriers – The drive slots are numbered 1 to 24 from left to right 25 Drive Options Drive Type Model 124/224 2.5” Small Form Factor (SFF) Model 112/212 3.5” Large Form Factor (LFF) Vendor Drive Name Drive Sizes SSD (eMLC) STEC Hitachi Hikari Ralston Peak 200, 300 & 400GB SAS 15K RPM Seagate Hornet 146 & 300GB Firefly 300GB Firestorm 450, 600 & 900GB SAS 10K RPM Seagate NL-SAS 7.2K RPM Seagate NL-SAS 7.2K RPM Seagate 1TB Muskie 2 & 3TB Control Enclosure Rear View 10Gbps dual port Converged Network Adapter (CNA) can now carry not only iSCSI but also FCoE frames Top node canister Two 6Gbps SAS ports (connections to expansions) Four 2/4/8Gbps Fibre Two USB2 ports Channel ports 2 4 1 1 2 3 4 1 2 3 CAUTION 4 2 2 4 1 3 Disconnect all supply power for complete isolation 1 Disconnect all supply power for complete isolation Tx/Rx 2 10Gb/s LNK 2 1 1 2 LNK 1 2 10Gb/s Tx/Rx Disconnect all supply power for complete isolation Disconnect all supply power for complete isolation 2 4 1 1 3 2 1 2 3 4 1 2 3 2 4 4 CAUTION CAUTION Power Supply Unit (PSU) Power Cooling Module (PCM) & Battery Bottom node canister Note: identical but installed upside down 27 1 3 Power Supply Unit (PSU) Power Cooling Module (PCM) & Battery Note: identical but installed upside down 2 4 2 4 1 3 1 3 Two 10Gbps Ethernet ports 1 3 CAUTION Two 1Gbps Ethernet ports Modular Scalability • Start with a single control enclosure (a pair of controllers) – And add up to 9 expansion enclosures to scale-in • Mix 2.5” & 3.5” drive enclosures • Mix drive type and drive size in an enclosure • Add control enclosures independently of expansion enclosures – Build a cluster (a Grid) of controller pairs to scale-out – Add expansion enclosures as required for capacity and performance Control Enclosures System Storage System Storage 1 4 1 5 8 5 8 9 12 9 12 1 4 1 5 8 5 8 9 12 9 12 24 1 System Storage System Storage 1 System Storage System Storage 1 24 System Storage System Storage 4 24 System Storage System Storage System Storage 1 4 1 5 8 5 8 9 12 9 12 24 System Storage Expansion Enclosures 1 24 System Storage 24 1 1 8 5 8 9 12 9 12 1 4 1 24 1 5 8 5 8 9 12 9 12 System Storage 1 4 1 8 5 8 5 8 9 12 9 12 9 12 24 1 24 1 24 24 1 24 1 24 1 24 1 24 System Storage 1 24 System Storage 1 12 4 System Storage System Storage 8 9 System Storage 4 1 4 5 4 5 System Storage 1 System Storage 1 System Storage 12 24 1 System Storage 8 9 System Storage 4 System Storage 5 24 24 System Storage 4 System Storage System Storage 1 1 4 5 System Storage System Storage System Storage 24 1 System Storage 1 24 1 System Storage 24 System Storage 1 24 System Storage System Storage 1 24 4 System Storage 1 1 System Storage 1 System Storage 4 System Storage 1 24 Overview of the I/O Stack SCSI target • SCSI Target – looks like a disk system and presents Volumes to Hosts (HBA driver layer) • Remote Copy – replicates data to second site for disaster recovery • Cache – mirrors writes between nodes in I/O Group • FlashCopy creates point-in-time copies • Volume Mirroring – provides pointers to two copies of data for high availability • Thin-provisioned Volume – consumes disk space only when data written –Random Access Compression EngineTM sits parallel to Thin-Provisioning acts in coordination • RACE is running in a controlled binary executable Comperssion • Virtualization – Volume MDisk mappings, non-disruptive data migration • RAID – Internal storage Array 1 array = 1 Mdisk migration • SCSI Initiator – talks to disk backend just like a host accessing LUNs (MDisks) – HBA driver layer Remote Copy Cache FlashCopy Volume Mirroring Thin-provisioning Virtualization RAID SCSI Initiator Virtualisation Concepts & Limits Enhanced! Hosts Volumes •Up to 2,048/8,192 Volumes per IO group/cluster •Max. Volume size of 256TiB • Image Mode = Native Mode • Managed Mode = Virtualized Mode LAN Single Multipath driver (SDD, MPIO) iSCSI, FCP or FCoE Host Mapping •Up to 256/1024 Hosts per IO group/cluster •Up to 512/2048 FC ports per IO group/cluster •Up to 256/1024 iSCSI ports per IO group/cluster SAN Storwize V7000 SSD SAS NL-SAS Storage Pools Pool Vol11 Vol2 Pool Vol32 Internal Storage Internal Storage 1 to 8/16 drives (HDD) to form an Array 1 Array Managed Disk (MD) Pool Vol7 Vol43 Pool 4 Vol5 Pool Vol65 •Up to 128 MDisks per Storage Pool •Up to 128 storage pools per cluster External Storage Up to 4,096 MDisks Max. External MDisk Size of 2TiB Or greater(1) External Storage Multiple Storage Subsystems 1 LUN Managed Disk (MDisk) Round-robin port selection for back-end storage subsystems • • • • • • • • • • Storwize V7000u Features New! FCP, iSCSI & FCoE Block Access Protocols, and file (NAS) • Cache partitioning Easy to use GUI – Real Time Performance Statistics Embedded SMI-S agent E-mail, SNMP trap & Syslog error event logging Authentication service for Single Sign-On & LDAP Virtualise data without data-loss – External storage virtualization (optional) Thin-provisioned & Compressed Volumes • – Reclaim Zero-write space • – Thick to thin, thin to thick & thin to thin migration Volume Expand or shrink Volumes on-line Storwize V7000 On-line Volume Migration between MDisk MDisk Storage Pools & IO groups Source Target Volume Snapshot-FlashCopy (Point-In-Time copy) – Up to 256 target per source – Full (with background copy = clone) – Partial (no background copy) – Space Efficient – Incremental Up to 256 Vol0 Source Map 1 Vol1 FlashCopy target of Vol0 Vol2 FlashCopy target of Vol1 Map 2 – Cascaded Map 4 – Consistency Groups Vol3 FlashCopy target of Vol1 – Reverse Vol4 FlashCopy target of Vol3 Microsoft Virtual Disk Service & Volume Shadow Copy Services hardware provider Remote Copy (optional) – Synchronous & asynchronous remote replication with Storwize V7000 Storwize V7000 Consistency groups • Optional Cycling Mode using snapshots Consolidated MM or GM MM or GM Relationship DR Site Relationship Storwize V7000 Storwize V7000 • Volume Mirroring Volume copy 0 Volume copy 1 • EasyTier: Automatic relocation of hot andSSDs cold extents SSDs HDDs HDDs Automatic Relocation Hot-spots Optimized performance and throughput MM or GM Relationship Storwize V7000 • VMware – Storage Replication Adaptor for Site Recovery Manager – VAAI support & vCenter Server management plug-in Data Movement - V7000 unified Information Lifecycle Management /home /home/appl/data/web/ /appl important_big_spreadsheet.xls No change to the logical directory big_architecture_drawing.ppt /data unstructured_big_video.mpg /web All File Modules participate in parallel data movement 5. Perform results of scan 0 1 2 3 4 5 6 Management role Active* 500 GB 500 GBS NLS AS N -LSA Storwize V7000 Unified 7 8 9 10 11 12 13 14 15 0 Interface role Storage role File Module 1 3 4 5 6 7 8 9 10 11 12 13 14 15 Interface role Management role Standby* Storage role File Module Sys t emx3650 M 3 1 3 2 Sys t emx3650 M 3 2 4 1 3 2 4 500 GB 500 GBS NLS AS N -LSA ACE System Storage Storage Module 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 1 24 Data Movement Hierarchical Storage Management /home /home/appl/data/web/ /appl important_big_spreadsheet.xls big_architecture_drawing.ppt /data unstructured_big_video.mpg /web External TSM server 5. Perform results of scan 0 1 2 3 4 5 6 Management role Active* 500 GB 500 GBS NLS AS N -LSA Storwize V7000 Unified 7 8 9 10 11 12 13 14 15 0 Interface role Storage role File Module 1 3 4 5 6 7 8 9 10 11 12 13 14 Tier 3 15 Interface role Management role Standby* Storage role File Module Sys t emx3650 M 3 1 3 2 Sys t emx3650 M 3 2 4 1 3 2 4 500 GB 500 GBS NLS AS N -LSA ACE System Storage Storage Module 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 600 GB 10K 6Gb SAS FRU 85Y5864 1 24 Stub files left on disk Auto-recall if accessed ... User Interface Based on the well-received XIV interface System recommendations Visual task flow guidance Integrated video instruction System status always available 34 Link to more information if needed VMware Site Recovery Manager Site Recovery Manager leverages VMware Infrastructure to transform disaster recovery • What it is: – Site Recovery Manager is a VMware product for disaster recovery • What it does: – Simplifies and automates disaster recovery processes • Discovery of replicated storage • Non-disruptive failover test using a writable copy of replicated data • Emergency and planned failover • Reverse replication after failover as part of failback • Restore replication after failover as part of a production test • Site Recovery Manager works with VMware Infrastructure to enable faster, more reliable, affordable disaster recovery VMware vStorage API for Array Integration VAAI support • Integration with vStorage API’s to improve performance – Full copy (and Block zeroing which was delivered in 6.1.0) offload work from production virtual servers to storage which improves application performance – Hardware-assisted locking enables a finer grained level of locking on VMFS metadata which is more efficient and also scales better in larger VMware clusters • Example tasks that can benefit from improved performance : – VM creation/cloning/snapshots/deletion – vMotion and storage vMotion – Extending a VMFS Volume – Extending the size of a VMDK file VMware Storage Stack Provisioning / Cloning VMFS NFS VMware LVM NFS Client Data Mover vStorage APIs NFS vStorage API for MultiPathing HBA Drivers Network Stack NIC VMware Storage Management Plugin • Gives a detailed look at current storage use • Supports Storwize V7000, Storwize V7000 Unified, Scale Out NAS, SAN Volume Controller & XiV • Easy to use GUI • Allows the VMware operator to provision storage to: – Create datastores – Create RDMs – Extend existing datastores and RDMs – Move datastores and RDMs between storage pools Flexible Storage Infrastructure with Storwize V7000 Make changes to the storage without disrupting host applications Volume Volume Volume Volume SAN Manage the storage pool from a central point Storwize V7000 / SVC Apply common copy services across the storage pool Advanced Copy Services Storage Pool HP DS8000 HDS DS4000 EMC Combine the capacity from multiple arrays into a single pool of storage IBM Smarter Storage for Smarter Computing Efficient by design • With Smarter Storage, essential storage technologies are built in and tightly integrated. Efficiency is effortless – requiring no manual intervention. Management is intuitive and easy – improving the efficiency of your most important asset, your people. Self-optimizing • Smarter Storage analyzes data access patterns, adapts and improves performance. Sophisticated internal analytics automatically place data on the appropriate storage tier. Cloud agile • Smarter Storage fully participates in highly virtualized cloud environments. Both IBM and non-IBM storage are managed as one virtual storage pool. Existing storage systems inherit IBM's latest, most advanced capabilities. Data is automatically synchronized between facilities. Files flow to users wherever they are. piotr.biskupski@pl.ibm.com +48 723703549 innovation that matters