Esra Ufacık, Mainframe Client Technical Specialist, IBM Turk – esrau@tr.ibm.com Introduction to Mainframe Hardware © 2012 IBM Corporation Agenda WHO uses Mainframes? And Why? WHY the Mainframe delivers ARCHITECTURE & Hardware Platform HIGHLIGHTS of z196 z196 Book Concept z196 Processor Unit Design z196 Memory z196 Connectivity z196 RAS BACKUP Slides 2 1 Definition of Mainframe The Mainframe is a computer system designed to continuously run very large, mixed workloads at high levels of utilization, meeting user-defined service level objectives. A Mainframe is what businesses use to host their commercial databases, transaction servers, and applications that require a greater degree of security and availability than is commonly found on smaller-scale machines. 3 What is a Mainframe? A style of operation, applications, and operating system facilities. Computers that can support thousands of applications and input/output devices to simultaneously serve thousands of users. A mainframe is the central data repository, or hub, in a corporation‟s data processing center, linked to users through less powerful devices such as workstations, terminals or media devices. Can run multiple, but isolated operating systems concurrently. Centralized control of resources. Optimized for I/O in all business-related data processing applications supporting high speed networking and terabytes of disk storage. 4 2 WHO uses Mainframes? And Why? 5 Sample of Customers Large and Small from Around the World BANKS OTHER FINANCIAL INST. RETAILERS HEALTHCARE UTILITIES / ENERGY Banco Itau Bank of NZ Bank of Montreal [Slovenia] [Peru] Bank of China Central Bank of Russia GOVERNMENTS TRANSPORTATION [Brazil] Philippines Savings Bank OTHERS [Gaming] [Brazil] [India] Industrial Bank of Korea HOSTING PROVIDERS [Kazakhstan] [USA] [Belarusian Railways] [Russian Hydrometeorological Research Institute] [Germany] [Vietnam] [Manufacturer/USA] 6 3 Mainframe is the platform of choice for the top 25 global banks Based on asset size, 96 out of the top 100 worldwide banks use System z.1 50 out of the top 50 worldwide banks use System z. 72 out of the top 75 worldwide banks use System z. For the latest generation of Mainframe; $1.5 billion in R&D investment for z196 5,000+ IBMers worked more than three years, 31 million hours, 24hour development process across 18 IBM labs worldwide. More than 40 IBM Fellows and Distinguished Engineers led this effort. The zEnterprise 196 is the world‟s fastest and most scalable enterprise system. 2 1 Based on „The Banker‟, System z install base and financial records 2 Based on 5.2GHz core processor speed 7 Workloads That Customers Run on Mainframe Banking Insurance Retail Healthcare Public Sector Core Banking Internet Rate Quotes On-line Catalog Patient Care Systems Electronic Tax Processing Wholesale Banking – Payments Policy Sales & Management (e.g. Life, Annuity, Auto) Supply Chain Management On–line Claims Submission & Payments Web based Social Security Customer Care & Insight Claims Processing Customer Analysis Fee payments What is a workload? The relationship between a group of applications and/or systems that are related across several business functions to satisfy one or more business processes. 8 4 Academic Initiative 1,067 Schools enrolled in the IBM System z Academic Initiative program, reaching students in 67 countries (more than half outside the US). Mainframe contests continue to draw high school, college, and university student participation around the world with 43,825 students from 32 countries. 3,246 students from 40 countries have taken the The IBM System z Mastery Test. This test is offered to students globally in 2011 at no cost. http://www-05.ibm.com/tr/industries/education/mainframe-index.html IBM Academic Initiative System z resources (system access, course materials, education) are available globally for universities and colleges. Systemzjobs.com (Job board) is a no-fee service that connects students and experienced professionals seeking System z job opportunities with IBM System z clients, partners, and businesses. 9 WHY the Mainframe delivers 10 5 The Differentiation of Mainframe: Why the Mainframe Is Just Another Computer, but a Different Kind of Computer? No other platform offers reliability, availability, scalability, resiliency, security, and management features comparable to Mainframe, including: High levels of application and data availability Solid security and privacy Massive horizontal and vertical scalability Optimized operating environment Advanced virtualization capabilities Intelligent workload management Utilization of open and industry standards Unmatched platform for heritage (legacy) applications and data Specialty engines that integrate with existing assets World-class support Centralized control Continuing compatibility 11 Mainframe Operating System Heritage OS/360 -> OS/VS -> MVS -> MVS/SP -> MVS/XA -> MVS/ESA -> OS/390 -> z/OS z/OS, IBM‟s premier zSeries operating system, is a highly secure, scalable, high-performance enterprise operating system on which to build and deploy traditional and Java-enabled applications, providing a comprehensive and diverse application execution environment. DOS/360 -> DOS/VS -> VSE/SP -> VSE/ESA -> z/VSE z/VSE enables proven, robust, and cost-effective solutions. z/VSE provides sturdy batch and industrial strength on-line transaction processing (CICS) capabilities. z/VSE can fit comfortably into a legacy of thrifty, dependable z/VSE solutions. ACP -> TPF-> z/TPF TPF is the platform driving the business-critical systems for many of IBM's largest and most sophisticated users of online transaction processing - airlines, railroads, hotels, financial services, government, credit card and banking industries. CP/67 -> VM/370 -> VM/SP -> VM/XA -> VM/ESA -> z/VM z/VM provides a highly flexible test and production environment for enterprises deploying the latest e-business solutions. z/VM helps enterprises meet their growing demands for multi-user server solutions with support for a broad range of operating systems. 12 6 Mainframe Software Customer Information Control System (CICS®) - a transaction processing environment that allows applications to share information and to satisfy customer requirements for information. Information Management System Transaction Manager (IMS TM) – a transaction processing environment and an application server Information Management System Database Server (IMS DB) - a database system that is capable of extremely high performance. DB2 - a full, functional relational database that leverages all of the features of Mainframe and z/OS to provide high availability and high performance. WebSphere Application Server (WAS) – an application server that runs on z/OS, and provides true application portability between Mainframe and distributed systems 13 Maximizing Utilization of Resources High business growth Up to 100% server utilization compared to 10-20% distributed server utilization1 Shared everything infrastructure allows for maximum utilization of resources z HIGH SCALABILITY Green strategy Running out of energy & space – CPU, Memory, Network, Adapters, Cryptography, Devices Lowers power consumption for each work unit z GREEN SAVINGS Customer savings Underutilized distributed servers Up to 100% utilized Mainframe server 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Typically single application per server 14 1 Source: App App App App App App App App App App App App App App App App App App App App App App App App Multiple applications on one Mainframe server goMainframe.com Joe Clabby 7 Mainframe‟s Extreme Virtualization Flexibility to respond – Built into the architecture not an “add on” feature VIRTUALIZATION Mainframe Hardware support: 10% of circuits are used for virtualization Deploy virtual servers in seconds Highly granular resource sharing (<1%) Each LPAR is one Virtual Mainframe PR/SM™ Hypervisor in firmware Add physical resources without taking system down, scale out to 1000s of virtual servers Do more with less: More virtual servers per core, Share more physical resources across servers LPAR – Up to 60 Logical partitions Extensive virtual server life-cycle management Hardware-enforced isolation z/VM® – 100s of virtual servers – Shared Memory Distributed Platforms Limited per-core virtual server scalability Physical server sprawl is needed to scale The potential performance impact of the Linux server farm is isolated from the other Logical partitions (LPAR) Operational complexity increases as virtual server images grow VMWare, Xen, Virtual Iron focus only on x86, not cross platform hardware management. 15 Chip Design Affects Virtualization Capabilities Replicated Server Chip Design Consolidated Server Chip Design Cache Cache Workload Chip Real Estate Processor Speed Workload Workload Workload Workload Mixed workloads stress cache usage, requiring more context switches Working sets may be too large to fit in cache “Fast” processor speed is not fully realized due to cache misses Workload RAS RAS Workload Chip Real Estate Processor Speed System z cache is able to contain more working sets Processor speed is optimized by increased cache usage Additional RAS function is beneficial for mixed workloads Note: System representations are not to scale, proportions may vary based on generation of chip and model 16 8 A Fully virtualized infrastructure enables optimization to maximize efficiency Green strategy Running out of energy & space Mainframe EFFICIENCY Consolidate the workloads of thousands of distributed servers in a single system Up to Up to Up to Up to 80% 90% 90% 70% Flexibility to respond EXTREME VIRTUALIZATION Reduction in energy consumption Reduction in Floor Space Reduction in SW License costs Reduction in total cost of ownership 17 Mainframe Security Reduce business risk Reducing risk – Protecting businesses Mainframe SECURITY Helping businesses: Protect from INTRUSION – z/OS and z/VM Integrity Statement Protect DATA – Built in encryption accelerators in every server – FIPS-140-20 Level 4 certified encryption co-processors for highly secure encryption Ensure PRIVACY – Access to all resources is controlled by an integrated central security manager Protect VIRTUAL SERVERS – The only servers with EAL5 Common Criteria Certification for partitioning Respond to COMPLIANCE REGULATIONS – Up to 70% in security audit savings Up to 52% lower security administrative costs. The Gold Standard for Security 18 9 Manage Risk with Mainframe Resiliency Continuous business operations Availability built in across the system z HIGH Mainframe AVAILABILITY AVAILABILITY RELIABILITY RELIABILITY Designed to deliver continuous availability at the application level Single Mainframe Multiple Mainframe Multiple Data Centers Site 1 Where mean time between failure is measured in decades Designed for application availability of 99.999% Site 2 Industry-leading solution for disaster recovery Avoiding the cost of downtime Maintaining productivity of users Ensuring access to critical applications Open to clients 24/7 19 Summary of the Benefits of Mainframe 20 10 ARCHITECTURE & Hardware Platform 21 History 7th April 1964, Poughkeepsie NY: Announcement of the first general-purpose computers: A new generation of electronic computing equipment was introduced today by International Business Machines Corporation. IBM Board Chairman Thomas J. Watson Jr. called the event the most important product announcement in the company's history. The new equipment is known as the IBM System/360. "System/360 represents a sharp departure from concepts of the past in designing and building computers. It is the product of an international effort in IBM's laboratories and plants and is the first time IBM has redesigned the basic internal architecture of its computers in a decade. The result will be more computer productivity at lower cost than ever before. This is the beginning of a new generation - - not only of computers - - but of their application in business, science and government ." * * from the April 1964 announcement press release 22 11 Evolution of the Mainframe IBM IBM System/360 System/370 1964 1970 IBM IBM System System/390 z900 1990 2000 IBM System z990 IBM System z9 EC IBM System z9 BC IBM System z10 EC IBM System z10 BC IBM System zEnterprise 196 IBM System zEnterprise 114 2003 2005 2006 2008 2009 2010 2011 23 Conceptual diagram OLD S/360 The central processor box contains the processors, memory,1 control circuits and interfaces for channels RECENT Current CPC designs are considerably more complex than the early S/360 design. This complexity includes many areas: I/O connectivity and configuration I/O operation Partitioning of the system 24 12 Mainframe basic components 25 Mainframe is a balanced system design System I/O Bandwidth 288 GB/Sec* Balanced System CPU, nWay, Memory, I/O Bandwidth* 172.8 GB/sec* 96 GB/sec Memory 3 TB** 24 GB/sec 1.5 TB** 512 GB 256 GB 300 64 GB 450 600 920 PCI for 1-way 1202 16-way 32-way z196 54-way 64-way * Servers exploit a subset of its designed I/O capability ** Up to 1 TB per LPAR PCI - Processor Capacity Index z10 EC z9 EC zSeries 990 80-way Processors zSeries 900 26 13 The virtualization concept The ability to share everything is based on one of the major concepts of the Mainframe. As it is commonly used in computing systems, virtualization refers to the technique of hiding the physical characteristics of the computing resources from users of those resources. For example, each operating system in a logical partition of the Mainframe platform thinks it has exclusive access to a real hardware image. The actual resources are shared with other logical partitions. The virtualization concept is not an add-on to the platform. z/Architecture contains many facilities in both the hardware and software that facilitate resource virtualization. All elements of Mainframe enable the virtualization concept. Virtualizing the Mainframe environment involves creating virtual systems (logical partitions and virtual machines), and assigning virtual resources, such as memory and I/O channels, to them. Resources can be dynamically added or removed from these logical partitions though operator commands. Dynamic logical partitioning and virtual machines increase flexibility, enabling selected system resources such as processors, memory, and I/O components to be added and deleted from partitions while they are actively in use. The ability to reconfigure logical partitions dynamically enables system administrators to dynamically move resources from one partition to another. Processor Resource/Systems Manager™ (PR/SM™) is a hypervisor integrated with all Mainframe elements that maps physical resources into virtual resources so that many logical partitions can share the physical resources. 27 PR/SM and logical partitions Processor Resource/Systems Manager is a feature that enables logical partitioning of the central processor complex (CEC). PR/SM provides isolation between partitions, which enables installations to separate users into distinct processing images, or to restrict access to certain workloads where different security clearances are required. Each logical partition operates as an independent server running its own operating environment. On the latest Mainframe models, you can define up to 60 logical partitions running z/VM®, z/OS, Linux on IBM Mainframe, z/TPF, and more operating systems. PR/SM enables each logical partition to have dedicated or shared processors and I/O channels, and dedicated memory (which you can dynamically reconfigure as needed). Either can be use to configure the IOCDS Support Elements (SEs) 28 14 LPAR concept 29 Logical Partitions (LPARs) or Servers A system programmer can assign different operating environments to each partition with isolation An LPAR can be assigned a number of dedicated or shared processors. Each LPAR can have different storage (CSTOR) assigned depending on workload requirements. The I/O channels (CHPIDs) are assigned either statically or dynamically as needed by server workload. Provides an opportunity to consolidate distributed environments to a centralized location 30 15 LPAR Logical Dispatching (Hypervisor) The next logical CP to be dispatched is chosen from the logical CP ready queue based on the logical CP weight. LPAR LIC dispatches the selected logical CP (LCP5 of MVS2 LP) on a physical CP in the CPC (CP0, in the visual). The z/OS dispatchable unit running on that logical processor (MVS2 logical CP5) begins to execute on physical CP0. It executes until its time slice (generally between 12.5 and 25 milliseconds) expires, or it enters a wait, or it is intercepted for some reason. In the visual, the logical CP keeps running until it uses all its time slice. At this point the logical CP5 environment is saved and control is passed back to LPAR LIC, which starts executing on physical CP0 again. LPAR LIC determines why the logical CP ended execution and requeues the logical CP accordingly. If it is ready with work, it is requeued on the logical CP ready queue and step 1 begins again. 31 HiperDispatch PR/SM and z/OS work in tandem to more efficiently use processor resources. HiperDispatch is a function that combines the dispatcher actions and the knowledge that PR/SM has about the topology of the server. Performance can be optimized by redispatching units of work to same processor group, keeping processes running near their cached instructions and data, and minimizing transfers of data ownership among processors/books. The nested topology is returned to z/OS by the Store System Information (STSI) 15.1.3 instruction, and HiperDispatch utilizes the information to concentrate logical processors around shared caches (L3 at PU chip level, and L4 at book level), and dynamically optimizes assignment of logical processors and units of work. z/OS dispatcher manages multiple queues, called affinity queues, with a target number of four processors per queue, which fits nicely into a single PU chip. These queues are used to assign work to as few logical processors as are needed for a given logical partition workload. So, even if the logical partition is defined with a large number of logical processors, HiperDispatch optimizes this number of processors nearest to the required capacity. The optimal number of processors to be used are kept within a book boundary where possible. 32 16 LPAR Characteristics Summary LPARs are the equivalent of a separate mainframe for most practical purposes System administrators assign: – Memory – Processors – CHPIDs either dedicated or shared Each LPAR runs its own operating system Devices can be shared across several LPARs Processors can be dedicated or shared When shared each LPAR is assigned a number of logical processors (up to the maximum number of physical processors) Each LPAR is independent This is done partly in the IOCDS and partly in a system profile on the Support Element (SE) in the CEC. This is normally updated through the HMC. Changing the system profile and IOCDS will usually require a power-on reset (POR) but some changes are dynamic A maximum of 60 LPARs can be defined in a z196 CPC 33 Terminology Overlap 34 17 The channel subsystem One of the main strengths of the Mainframe computers is the ability to deal with a large number of simultaneous I/O operations. The channel subsystem (CSS) has a major role in providing this strength. The CSS manages the flow of information between I/O devices and central memory. By doing so it relieves CPUs of the task of communicating directly with I/O devices and permits data processing to proceed concurrently with I/O processing. The channel subsystem is built on the following components: SAP (system assist processor) One of the Mainframe processor types, the SAP connects the channel subsystem to the I/O devices that are attached to the channels. The SAP uses the I/O configuration loaded in the Hardware System Area (HSA), and knows which device is connected to each channel, and what is its protocol. The SAP manages the queue of I/O operations passed to the channel subsystem by z/OS. Channels These are small processors that communicate with I/O control units (CU). They manage the transfer of data between central storage and the external device. Channel path The channel subsystem communicates with I/O devices through channel paths. If a channel is shared between multiple logical partitions, each logical partition establishes a unique channel path to the each device using this channel. Subchannels A subchannel provides the logical appearance of a device to the program and contains the information required for performing a single I/O operation. One subchannel is provided for each I/O device addressable by the channel subsystem. 35 Channel Subsystem Relationship to Channels, Control Units and I/O Devices z196 Channel Subsystem Controls queuing, de-queuing, priority management and I/O identification of all I/O operations performed by LPARs Partitions Supports the running of an OS and allows CPs, memory and Subchannels access to channels Subchannels This represents an I/O device to the hardware and is used by the OS to pass an I/O request to the channel subsystem Channels CU CU The communication path from the channel subsystem to the I/O network and connected Control Units/devices CU Control Units Devices (disk, tape, printers) @ @ = = @ @ = = 36 18 The Mainframe I/O Logical Channel Subsystem Schematic H I P E R V I S O R H S A Cache MBA SAP Logical-channel Subsystem 0 Cache Cache MBA MBA SAP Logical-channel Subsystem 1 Cache MBA SAP Logical-channel Subsystem 2 SAP Logical-channel Subsystem 3 Physical-Channel Subsystem FICON Switch, Control Unit, Devices, etc. -One FICON channel shared by all LCSs and all partitions -A MCSS-Spanned Channel Path - One ESCON channel shared by all partitions configured to LCS15 - A MIF-shared channel path ESCON Switch, Control Unit, Devices, etc. 37 Multiple channel subsystems and multiple subchannel sets 38 19 I/O Device addressing 39 System z I/O Connectivity ESCON and FICON channels Switches to connect peripheral devices to more than one CEC CHPID addresses are two hex digits (FF / 256) Multiple partitions can share CHPIDs (MIF) I/O subsystem layer exists between the operating system and the CHPIDs I/O control layer uses a control file IOCDS that translates physical I/O addresses into devices numbers that are used by z/OS Device numbers are assigned by the system programmer when creating the IODF and IOCDS and are arbitrary (but not random!) On modern machines they are three or four hex digits example - FFFF = 64K devices can be defined The ability to have dynamic addressing theoretically 7,848,900 maximum number of devices can be attached. 40 20 MIF Channel Consolidation - example statically assigned dynamically assigned 41 HCD & Sample IOCP ID MSG2='ESRAU.IODF99 - 2011-02-15 22:05',SYSTEM=(2097,1), * LSYSTEM=CECDCEA, * TOK=('CECDCEA',00800007C74E2094220520410111046F00000000,* 00000000,'11-02-15','22:05:20','ESRAU','IODF99') RESOURCE PARTITION=((CSS(0),(PCF2GAR2,5),(PRDA,1),(PRDC,2),(PR* DE,3),(PRDF,4),(*,6),(*,7),(*,8),(*,9),(*,A),(*,B),(*,C)* ,(*,D),(*,E),(*,F)),(CSS(1),(*,1),(*,2),(*,3),(*,4),(*,5* ),(*,6),(*,7),(*,8),(*,9),(*,A),(*,B),(*,C),(*,D),(*,E),* (*,F)),(CSS(2),(*,1),(*,2),(*,3),(*,4),(*,5),(*,6),(*,7)* ,(*,8),(*,9),(*,A),(*,B),(*,C),(*,D),(*,E),(*,F)),(CSS(3* ),(*,1),(*,2),(*,3),(*,4),(*,5),(*,6),(*,7),(*,8),(*,9),* (*,A),(*,B),(*,C),(*,D),(*,E),(*,F))) CHPID PATH=(CSS(0),00),SHARED,PARTITION=((PRDA,PRDC),(=)), * CPATH=(CSS(0),00),CSYSTEM=CECDD6A,AID=0C,PORT=1,TYPE=CIB ... CNTLUNIT CUNUMBR=FFFD,PATH=((CSS(0),08,09,0A,0B,0C,0D,0E,0F)),* UNIT=CFP IODEVICE ADDRESS=(FF90,007),CUNUMBR=(FFFD),UNIT=CFP IODEVICE ADDRESS=(FF97,007),CUNUMBR=(FFFD),UNIT=CFP 42 21 PAV (Parallel Access Volume) & MA (Multiple Allegiance) IBM PAV enables System z to process multiple I/Os to same logical volume at once IBM Multiple Allegiance (concurrent execution of multiple requests from multiple hosts) expands capability across multiple System z servers I/O Priority enables use of info from z/OS Workload Manager to manage I/O processing order I/O without PAV/MA Appl. A Appl. B UCB 100 Appl. C UCB 100 Appl. A UCB 100 UCB Busy One I/O to one volume at one time I/O with PAV/MA Appl. B UCB 1FF Alias to UCB 100 Device Busy Parallel Access Volumes 100 Appl. C UCB 100 UCB 100 100 Multiple Allegiance 43 IBM PAV – Static, Dynamic & Hyper Static PAV: Aliases are always associated with the same Base Address. Dynamic PAV: Aliases are assigned up front but can be reassigned to any base address as need dictates: WLM function call Dynamic Alias Management - reactive alias assignment HyperPAV: On demand/ Proactive alias assignment I/O flow of PAV & MA 9000 8 8000 7 7000 6 6000 5 5000 4 4000 3 3000 2 2000 1 1000 0 I/O per second Response (MS) PAV Performance 9 <Queue <Pend <Connect <Disc I/O Rate> 0 NoPAV PAV 44 22 Static PAV & Dynamic PAV Aliases assigned to specific addresses 45 Hyper PAV Alias is taken from and returned to a pool instead of LCU. Define max. number of aliases per LCU in HCD. Every I/O Request is queued when all aliases allocated in the pool in the same LCU are exhausted. Aliases kept in pool for use as needed 46 23 ESCON Connectivity ESCON (Enterprise Systems Connection) is a data connection created by IBM commonly used to connect their mainframe computers to peripheral devices. ESCON replaced the older, slower parallel Bus&Tag channel technology The ESCON channels use a director to support dynamic switching. 47 ESCON Director ESCD ESCD 48 24 Fiber Connectivity (FICON) FICON (for Fiber Connectivity) was the next generation high-speed input/output (I/O) interface used by for mainframe computer connections to storage devices. FICON channels increase I/O capacity through the combination of a new architecture and faster physical link rates to make them up to eight times as efficient as ESCON. 49 FICON Connectivity 50 25 ESCON vs FICON ESCON - 20 Mbytes / Second - Lots of “dead time”. One active request at a time. - One target control unit FICON - 800 Mbytes / Second - Uses FCP standard - Fiber Optic cable (less space under floor) - Currently, up to 64 simultaneous “I/O packets” at a time with up to 64 different control units - Supports Cascading switches 51 Hardware Management Console Mainframe hardware can be managed by using either the Support Elements directly attached to the server or using the Hardware Management Console (HMC), which is a desktop application providing the end user interface to control and monitor the status of the system. Working from an HMC, an operator, a system programmer, or IBM technical personnel can perform basic operations on the Mainframe servers. The HMC is a powerful console that can load and shut down system images as well as the entire machine. Some of the common capabilities of the HMC are: Load the Mainframe hardware configuration. Load or reset a system image. Add and change the hardware configuration (most of them dynamically). Access the hardware logs. All these functions can be executed using a browser interface in a totally secure environment. 52 26 Reserved processors & storage Reserved processors are defined by the Processor Resource/Systems Manager (PR/SM) to allow for a nondisruptive capacity upgrade. Reserved processors are like spare logical processors, and can be shared or dedicated. Reserved CPs should be defined to a logical partition to allow for nondisruptive image upgrades. Reserved storage can optionally be defined to a logical partition, allowing a nondisruptive image memory upgrade for this partition. Reserved storage can be defined to both central and expanded storage, and to any image mode, except the coupling facility mode. LPAR dynamic storage reconfiguration Dynamic storage reconfiguration on z196 servers allows an operating system running in a logical partition to add (nondisruptively) its reserved storage amount to its configuration, if any unused storage exists. This unused storage can be obtained when another logical partition releases some storage or when a concurrent memory upgrade takes place. With dynamic storage reconfiguration, the unused storage does not have to be continuous. 53 HIGHLIGHTS of z196 54 27 z196 Overview Machine Type – 2817 5 Models – M15, M32, M49, M66 and M80 Processor Units (PUs) – 20 (24 for M80) PU cores per book – Up to 14 SAPs per system, standard – 2 spares designated per system – Dependant on the H/W model - up to 15,32,49,66 or 80 PU cores available for characterization • Central Processors (CPs), Integrated Facility for Linux (IFLs), Internal Coupling Facility (ICFs), Mainframe Application Assist Processors (zAAPs), Mainframe Integrated Information Processor (zIIP), optional - additional System Assist Processors (SAPs) – Sub-capacity available for up to 15 CPs • 3 sub-capacity points Memory – System Minimum of 32 GB – Up to 768 GB per book – Up to 3 TB for System and up to 1 TB per LPAR • Fixed HSA, standard • 32/64/96/112/128/256 GB increments I/O – Up to 48 I/O Interconnects per System @ 6 GBps each – Up to 4 Logical Channel Subsystems (LCSSs) STP - optional (No ETR) 55 zEnterprise 196 Functions and Features (GA Driver 86 – August, 2010) Five hardware models Three subchannel sets per LCSS Quad core PU chip Up to 80 processors configurable as CPs, zAAPs, zIIPs, IFLs, ICFs, or optional SAPs Increased capacity processors Out of order instruction execution Over 100 new and enhanced instructions Improved processor cache design Up to 15 subcapacity CPs at capacity settings 4, 5, or 6 Up to 3 TB of Redundant Array of Independent Memory (RAIM) Unified Resource Manager suites Cryptographic enhancements On Demand enhancements Energy efficiencies 2 New OSA CHPIDs – OSX and OSM 8 slot, 2 domain I/O drawer z196 Concurrent I/O drawer add, remove, replace FICON discovery and autoconfiguration Doubled HiperSockets to 32 Physical Coupling Links increased to 80 Doubled Coupling CHPIDs to 128 CFCC Level 17 Optional water cooling Optional High Voltage DC power Static Power Save Mode Optional Top Exist I/O cable exit STP enhancements zBX-002 with IBM Smart Analytics Optimizer, IBM Blades 56 28 zEnterprise 196 Functions and Features (GA Driver 93 – September, 2011) Additional Cryptographic enhancements 8 GBps PCIe interconnects Additional STP enhancements Doubled Coupling CHPIDs to 128 Improved PSIFB Coupling Link Physical Coupling Links increased to 104 New 32 slot PCIe Based I/O Drawer Increased granularity of I/O adapters New form factor I/O adapters i.e FICON Express8S and OSA-Expres4S 57 zEnterprise 114, 196 and zBX Enhancements - October 12, 2011) zHPF enhancements EAL5+ Certification for z196 GDPS® Support for zEnterprise GDPS xDR support for z/VSE GDPS reduced HyperSwap® impact and improved scalability New IBM Implementation Services for System z New and revised zEnterprise Statements of Direction zBX IEDN TOR 1 GbE optics for the customer network Additional zBX 8 Gbps FCS optics for POWER7 and System x blade connections to the customer SAN z196 z114 zBX support for Windows on select System x blades New DataPower XI50z firmware support zBX support for AIX 7.1 on select POWER7 blades Unified Resource Manager APIs Unified Resource Manager dynamic discovery of storage resources z/VM 6.2 enhanced virtualization capabilities IBM DB2 Analytics Accelerator zBX 58 29 Design highlights Offer a flexible infrastructure Offer state-of-the-art integration capability Offer high performance Offer the high capacity and scalability Offer the capability of concurrent upgrades for processors, memory, and I/O connectivity Implement a system with high availability and reliability Have broad internal and external connectivity offerings Provide the highest level of security in which every two PUs share a CP Assist for Cryptographic Function (CPACF). Be self-managing and self-optimizing – IRD, Hiperdispatch Have a balanced system design 59 z196 Processor Design Basics Logical View of Single Book Processor Design – CPU (core) • cycle time • pipeline, execution order • branch prediction • hardware vs. millicode* – Memory subsystem (nest) • high speed buffers (caches) on chip, on book private, shared • buses number, bandwidth • latency distance speed of light Memory Shared Cache Private Cache Private Cache CPU CPU Private Cache … CPU *executes complex instructions 60 30 Performance Drivers with z196 Hardware –Memory subsystem • focus on keeping data “closer” to the processor unit • new chip-level shared cache • much larger book-level shared cache –Processor • Out-of-Order execution • new instructions to allow for reduced processor quiesce effects reduced cache misses reduced pipeline disruption • up to 80 configurable processor units • Total of 4 different uni speeds, 3 sub-capacity HiperDispatch –exploits new cache topology –reduced cross-book “help” –better locality for multi-task address spaces State benefits if you re-compile (Run-time modules or to make use of out-of-order execution) 61 Hypervisor and OS Basics Hypervisor (PR/SM) – Virtualization layer at OS level – Distributes physical resources • Memory • Processors Logicals dispatched on physicals Dedicated Shared Affinities OS – Virtualization layer at address space level – Distributes logical resources • Memory • Processors Tasks dispatched on logicals Enhanced cooperation – HiperDispatch with z10 EC • z/OS + PR/SM Logical View of 2 books Memory Memory L2 Cache L2 Cache L1 L1 CPU CPU … L1 L1 L1 CPU CPU CPU … L1 CPU PR/SM LP1 LP2 TCBa TCBb … z/OSi LPn TCBx LP1 LP2 TCBa TCBb … z/OSj LPm TCBx 62 31 z196 – Under the covers (Model M66 or M80) Internal Batteries (optional) Processor Books, Memory, MBA and HCA cards Power Supplies Ethernet cables for internal System LAN connecting Flexible 2 x Support Elements (FSP) cage controller cards Service Processor InfiniBand I/O Interconnects I/O cage 2 x Cooling Units I/O drawers Fiber Quick Connect (FQC) Feature (optional) FICON & ESCON FQC 63 z196 Water cooled – Under the covers (Model M66 or M80) front view Internal Batteries (optional) Power Supplies Support Elements I/O cage I/O drawers Ethernet cables for internal System LAN connecting Flexible Service Processor (FSP) cage controller cards Processor Books, Memory, MBA and HCA cards InfiniBand I/O Interconnects 2 x Water Cooling Units z196TLLB64 64 32 z196 System Upgrades: Protecting your investments z9 EC M80 M49 z10 EC M32 M15 Concurrent Upgrade M66 z196 to higher hardware z196 model – Upgrade of z196 Models M15, M32, M49 and M66 to M80 is disruptive – When upgrading to z196 all the Books are replaced – Upgrade from Air to Water cooled not available Any z9 EC to any z196 Any z10 EC to any z196 65 z196 Architecture Continues line of upward-compatible Mainframe processors Rich CISC (Complex Instruction Set Computer) Instruction Set Architecture (ISA) – 984 instructions (762 implemented entirely in hardware) – 24, 31, and 64-bit addressing modes – Multiple address spaces robust inter-process security – Multiple arithmetic formats – Industry-leading virtualization support • High-performance logical partitioning via PR/SM • Fine-grained virtualization via z/VM scales to 1000‟s of images – Precise, model-independent definition of hardware/software interface Architectural extensions for IBM z196 – 110+ new instructions added to improve compiled code efficiency – Decimal floating point quantum exceptions – New crypto functions and modes – Virtual architectural level 66 *RISC = Reduced Instruction Set Computer 33 z196 Book Concept 67 Book design A z196 system has up to four books on a fully connected topology, up to 80 processor units can be characterized, and up to 3 TB of memory capacity. Memory has up to 12 memory controllers, using 5-channel redundant array of independent memory (RAIM) protection, with DIMM bus cyclic redundancy check (CRC) error retry. The 4-level cache hierarchy is implemented with eDRAM (embedded) caches. Up until recently eDRAM was considered to be too slow for this kind of use, however, a break-through in IBM technology has negated that. In addition eDRAM offers higher density, less power utilization, fewer soft-errors, and better performance. Concurrent maintenance allows dynamic book add and repair. The z196 server uses 45nm chip technology, with advancing low latency pipeline design, leveraging high speed yet power efficient circuit designs. The multichip module (MCM) has a dense packaging, allowing modular refrigeration units (MRUs) cooling, or as an option, water cooling. The water cooling option is recommended as it can lower the total power consumption of the server. 68 34 Book Layout Backup Air Plenum 8 I/O FAN OUT 2 FSP MCM @ 1800W Refrigeration Cooled or Water Cooled 16X DIMMs 100mm High Fanout Memory Rear DCA Power Supplies 3x DCA 11 VTM Card Assemblies 8 Vertical 3 Horizontal Front Cards Memory 14X DIMMs 100mm High Cooling from/to MRU 69 z196 PU chip, SC chip and MCM z196 Quad Core PU CHIP 96 MB SC CHIP BOOK Front View Front View Fanouts MCM 70 35 z196 Cache / Book Topology Fully connected 4 Book system: Mem2 Mem1 Mem0 Book: CP2 FBCs CP1 SC1 CP3 CP0 FBCs SC0 CP4 CP5 The z196 Books are fully interconnected in a point to point topology as shown in the diagram 96 total cores, 80way Total system cache 768 MB shared L4 (eDRAM) 576 MB L3 (eDRAM) 144 MB L2 private (SRAM) 19.5 MB L1 private (SRAM) Data transfers are direct between Books via the Level 4 Cache chip in each MCM. Level 4 Cache is shared by all PU chips on the MCM 71 z196 Book Level Cache Hierarchy PU L1 L1 L1 PU L1 L1 L1 L1 PU L1 L1 L1 L1 PU L1 L1 L1 L1 PU L1 L1 L1 L1 PU L1 L1 L1 L1 L1 1.5MB 1.5MB 1.5MB 1.5MB L2 L2 L2 L2 1.5MB 1.5MB 1.5MB 1.5MB L2 L2 L2 L2 1.5MB 1.5MB 1.5MB 1.5MB L2 L2 L2 L2 1.5MB 1.5MB 1.5MB 1.5MB L2 L2 L2 L2 1.5MB 1.5MB 1.5MB 1.5MB L2 L2 L2 L2 1.5MB 1.5MB 1.5MB 1.5MB L2 L2 L2 L2 24MB eDRAM Inclusive L3 24MB eDRAM Inclusive L3 24MB eDRAM Inclusive L3 24MB eDRAM Inclusive L3 24MB eDRAM Inclusive L3 24MB eDRAM Inclusive L3 LRU Cast-Out CP Stores Data Fetch Return 72 192MB eDRAM Inclusive L4 2 SC Chips 3/2/2012 36 Cache levels and memory structure 73 Enhanced book availability Book Add – Model Upgrade by the addition of a single new book adding physical processors, memory, and I/O Connections. Continued Capacity with Fenced Book – Make use of the LICCC defined resources of the fenced book to allocate physical resources on the operational books as possible. Book Repair – Replacement of a defective book when that book had been previously fenced from the system during the last IML. Book Replacement – Removal and replacement of a book for either repair or upgrade 74 37 z196 Processor Unit Design 75 Mainframe Servers Continue to Scale with z196 96 engines 80-way z196 77 engines 64-way z10 EC Maximum PCI Each new range continues to deliver: – New function – Unprecedented capacity to meet consolidation needs – Improved efficiency to further reduce energy consumption – Continues to delivering flexible and simplified on demand capacity – A Mainframe that goes beyond the traditional paradigm 64 engines 54-way z9 EC 48 engines 32-way z990 z900 20 engines 16-way Minimum PCI z/OS release used for LSPR measurements z196 measurements are for a xx-way PCI - (Processor Capacity Index z900 z/OS 1.6 z990 z/OS 1.6 z9 EC z/OS 1.6 z10 EC z/OS 1.8 z196 z/OS 1.11 76 38 z196 Continues the CMOS Mainframe Heritage 5.2 GHz 4.4 GHz 5000 Fastest Microprocessor In The Industry 4000 MHz 3000 1.7 GHz 2000 1000 300 MHz 420 MHz 550 MHz 1998 G5 1999 G6 770 MHz 1.2 GHz 0 1997 G4 G4 – 1st full-custom CMOS S/390® G5 – IEEE-standard BFP; branch target prediction G6 – Copper Technology (Cu BEOL) 2000 z900 2003 z990 2005 z9 EC z900 – Full 64-bit z/Architecture z990 – Superscalar CISC pipeline z9 EC – System level scaling 2008 z10 EC 2010 z196 z10 EC – Architectural extensions z196 – Additional Architectural extensions and new cache structure 77 z196 GHz Leadership Mainframe continues to grow frequency by – Leveraging 45nm technology – Advancing low latency pipeline design – Leveraging high speed yet power-efficient circuit designs z196 achieves industry leadership 5.2 GHz operation High-frequency processor design is enabled by – Innovative power-efficient circuit designs – IBM‟s SOI silicon technology – Leadership packaging technology including glass-ceramic MCMs – Leadership cooling technology, with choice of MRU or chilled-water cooling GHz is especially important for CPU-intensive applications System performance is not linear with frequency – Need to use LSPR + Mainframe capacity planning tools for real client / workload sizing GHz is not the only dimension that matters – Must balance frequency, pipeline efficiency, energy efficiency, cache/memory design, I/O design 78 39 z196 Out of Order (OOO) Value OOO yields significant performance benefit for compute intensive apps through –Re-ordering instruction execution • Later (younger) instructions can execute ahead of an older stalled instruction –Re-ordering storage accesses and parallel storage accesses OOO maintains good performance growth for traditional apps Instrs In-order core execution Out-of-order core execution 1 2 L1 miss L1 miss 3 4 5 Time Execution Time Storage access 79 Superscalar processor A scalar processor is a processor that is based on a single-issue architecture, which means that only a single instruction is executed at a time. A superscalar processor allows concurrent execution of instructions by adding additional resources onto the microprocessor to achieve more parallelism by creating multiple pipelines, each working on its own set of instructions. A superscalar processor is based on a multi-issue architecture. In such a processor, where multiple instructions can be executed at each cycle, a higher level of complexity is reached, because an operation in one pipeline stage might depend on data in another pipeline stage. Therefore, a superscalar design demands careful consideration of which instruction sequences can successfully operate in a long pipeline environment. On the z196, up to three instructions can be decoded per cycle and up to five instructions/operations can be executed per cycle. Execution can occur out of (program) order. If the branch prediction logic of the microprocessor makes the wrong prediction, removing all instructions in the parallel pipelines also might be necessary. Obviously, the cost of the wrong branch prediction is more costly in a high-frequency processor design. For this reason, a variety of history-based branch prediction mechanisms are used. The branch target buffer (BTB) runs ahead of instruction cache pre-fetches to prevent branch misses in an early stage. Furthermore, a branch history table (BHT) in combination with a pattern history table (PHT) and the use of tagged multi-target prediction technology branch prediction offer an extremely high branch prediction success rate. 80 40 Branch prediction Because of the ultra high frequency of the PUs, the penalty for a wrongly predicted branch is high. For that reason a multi-pronged strategy for branch prediction, based on gathered branch history combined with several other prediction mechanisms, is implemented on each microprocessor. The branch history table (BHT) implementation on processors has a large performance improvement effect. Originally introduced on the IBM ES/9000 9021 in 1990, the BHT has been continuously improved. The BHT offers significant branch performance benefits. The BHT allows each PU to take instruction branches based on a stored BHT, which improves processing times for calculation routines. Besides the BHT, the z196 uses a variety of techniques to improve the prediction of the correct branch to be executed. The techniques include: Branch history table (BHT) Branch target buffer (BTB) Pattern history table (PHT) BTB data compression The success rate of branch prediction contributes significantly to the superscalar aspects of the z196. This is because the architecture rules prescribe that, for successful parallel execution of an instruction stream, the correctly predicted result of the branch is essential. 81 Wild branch When a bad pointer is used or when code overlays a data area containing a pointer to code, a random branch is the result, causing a 0C1 or 0C4 abend. Random branches are very hard to diagnose because clues about how the system got there are not evident. With the wild branch hardware facility, the last address from which a successful branch instruction was executed is kept. z/OS uses this information in conjunction with debugging aids, such as the SLIP command, to determine where a wild branch came from and might collect data from that storage location. This approach decreases the many debugging steps necessary when looking for where the branch came from. IEEE floating point Over 130 binary and hexadecimal floating-point instructions are present in z196. They incorporate IEEE Standards into the platform. Translation look-aside buffer The translation look-aside buffer (TLB) in the instruction and data L1 caches use a secondary TLB to enhance performance. In addition, a translator unit is added to translate misses in the secondary TLB. Instruction fetching, decode, and grouping The superscalar design of the microprocessor allows for the decoding of up to three instructions per cycle and the execution of up to five instructions per cycle. Both execution and storage accesses for instruction and operand fetching can occur out of sequence. Extended translation facility Used in data conversion operations for data encoded in Unicode, causing applications that are enabled for Unicode or globalization to be more efficient. These data-encoding formats are used in Web services, grid, and on-demand environments where XML and SOAP technologies are used. Instruction set extensions Hexadecimal floating point, Immediate, Load, Cryptographic, Extended Translate Facility-3 and Assist instructions, 82 41 z196 Compression and Cryptography Accelerator Data compression engine – Static dictionary compression and expansion – Dictionary size up to 64KB (8K entries) • Local 16KB cache per core for dictionary data CP Assist for Cryptographic Function (CPACF) – Enhancements for new NIST standard – Complemented prior ECB and CBC symmetric cipher modes with XTS, OFB, CTR, CFB, CMAC and CCM – New primitives (128b Galois Field multiply) for GCM Core 1 Core 0 2nd Level Cache IB OB TLB Cmpr Exp TLB 16K OB IB Cmpr Exp 16K Accelerator unit shared by 2 cores – Independent compression engines – Shared cryptography engines Crypto Cipher Crypto Hash 83 Decimal floating point accelerator The decimal floating point (DFP) accelerator function is present on each of the microprocessors (cores) on the quad-core chip. Its implementation meets business application requirements for better performance, precision, and function. Base 10 arithmetic is used for most business and financial computation. Floating point computation that is used for work typically done in decimal arithmetic has involved frequent necessary data conversions and approximation to represent decimal numbers. This has made floating point arithmetic complex and error-prone for programmers using it for applications in which the data is typically decimal data. Benefits Avoids rounding issues such as those happening with binary-to-decimal conversions. Has better functionality over existing binary coded decimal (BCD) operations. Follows the standardization of the dominant decimal data and decimal operations in commercial computing supporting industry standardization (IEEE 745R) of decimal floating point operations. Instructions are added in support of the Draft Standard for Floating-Point Arithmetic, which is intended to supersede the ANSI/IEEE Std 7541985. 84 42 Processor error detection and recovery The PU uses something called transient recovery as an error recovery mechanism. When an error is detected, the instruction unit retries the instruction and attempts to recover the error. If the retry is not successful (that is, a permanent fault exists), a relocation process is started that restores the full capacity by moving work to another PU. Relocation under hardware control is possible because the R-unit has the full architected state in its buffer 85 z196 Multi-Chip Module (MCM) Packaging 96mm x 96mm MCM – 103 Glass Ceramic layers – 8 chip sites – 7356 LGA connections – 20 and 24 way MCMs – Maximum power used by MCM is 1800W PU 2 PU 1 PU 0 CMOS 12s chip Technology – PU, SC, S chips, 45 nm – 6 PU chips/MCM – Each up to 4 cores • • • • One memory control (MC) per PU chip 23.498 mm x 21.797 mm 1.4 billion transistors/PU chip L1 cache/PU core 64 KB I-cache 128 KB D-cache • L2 cache/PU core 1.5 MB • L3 cache shared by 4 PUs per chip 24 MB • 5.2 GHz – 2 Storage Control (SC) chip S10 S00 SC 1 SC 0 S11 S01 • • • • 24.427 mm x 19.604 mm 1.5 billion transistors/SC chip L4 Cache 96 MB per SC chip (192 MB/Book) L4 access to/from other MCMs – 4 SEEPROM (S) chips PU 3 PU 4 PU 5 • 2 x active and 2 x redundant • Product data for MCM, chips and other engineering information – Clock Functions – distributed across PU and SC chips • Master Time-of-Day (TOD) function is on the SC 86 43 z196 Quad Core PU Chip Detail Up to Four active cores per chip – 5.2 GHz – L1 cache/ core • 64 KB I-cache • 128 KB D-cache 12S0 45nm SOI Technology – 13 layers of metal – 3.5 km wire 1.4 Billion Transistors Chip Area – 512.3mm2 – 23.5mm x 21.8mm – 8093 Power C4‟s – 1134 signal C4‟s – 1.5 MB private L2 cache/ core Two Co-processors (COP) – Crypto & compression accelerators – Includes 16KB cache – Shared by two cores 24MB eDRAM L3 Cache – Shared by all four cores Interface to SC chip / L4 cache – 41.6 GB/sec to each of 2 SCs I/O Bus Controller (GX) – Interface to Host Channel Adapter (HCA) Memory Controller (MC) – Interface to controller on memory DIMMs – Supports RAIM design 87 z196 SC Chip Detail 12S0 45nm SOI Technology L4 Cache (24MB) – 13 layers of metal Chip Area – 478.8mm^2 L4 Cache (24MB) – 24.4mm x 19.6mm – 7100 Power C4‟s – 1819 signal C4‟s 1.5 Billion Transistors – 1 Billion cells for eDRAM eDRAM Shared L4 Cache Fabric IOs PLL Data Perv ETR/ TOD Data Clk BitStack L4 Controller BitPerv Stack Fabric IOs Perv Repower – 96 MB per SC chip – 192 MB per Book 6 CP chip interfaces 3 Fabric interfaces 2 clock domains L4 Cache (24MB) L4 Cache (24MB) 5 unique chip voltage supplies 88 44 z196 Microprocessor Summary Major advance in Mainframe processor design – Deep, high-frequency (5.2 GHz) pipeline – Aggressive out of order execution core – 4-level cache hierarchy with eDRAM L3 and L4 Synergy between hardware and software design – z/Architecture (ISA - Instruction Set Architecture) extensions – Compiler and micro-architecture co-optimization – Robust performance gain on existing binaries (code) Major step up in processor performance – Up to 40% performance gain on existing compute-intensive code – Additional gains achievable with recompilation 89 z196 Processor Features Model Books/ PUs CPs IFLs uIFLs zAAPs zIIPs ICFs SAPs Std Optional SAPs Std. Spares M15 1/20 0-15 0-15 0-14 0-7 0-7 0-15 3 0-4 2 M32 2/40 0-32 0-32 0-31 0-16 0-16 0-16 6 0-10 2 M49 3/60 0-49 0-49 0-48 0-24 0-24 0-16 9 0-15 2 M66 4/80 0-66 0-66 0-65 0-33 0-33 0-16 12 0-20 2 M80 4/96 0-80 0-80 0-79 0-40 0-40 0-16 14 0-18 2 z196 Models M15 to M66 use books each with a 20 core MCM (two 4-core and four 3-core PU chips) Concurrent Book Add is available to upgrade from model to model (except to the M80) z196 Model M80 has four books each with a 24 core MCM (six 4-core PU chips) Disruptive upgrade to z196 Model M80 is done by book replacement Notes: 1. At least one CP, IFL, or ICF must be purchased in every machine 2. One zAAP and one zIIP may be purchased for each CP purchased even if CP capacity is “banked”. 3. “uIFL” stands for Unassigned IFL 90 45 z196 Memory 91 Memory design The z196 memory design also provides flexibility and high availability Memory assignment or allocation is done at power-on reset (POR) when the system is initialized. PR/SM is responsible for the memory assignments. PR/SM has knowledge of the amount of purchased memory and how it relates to the available physical memory in each of the installed books. PR/SM has control over all physical memory and therefore is able to make physical memory available to the configuration when a book is nondisruptively added. PR/SM also controls the reassignment of the content of a specific physical memory array in one book to a memory array in another book. 92 46 Large page support By default, page frames are allocated with a 4 KB size. The z196 supports a large page size of 1 MB. The first z/OS release that supports large pages is z/OS V1R9. The translation look-aside buffer (TLB) exists to reduce the amount of time required to translate a virtual address to a real address by dynamic address translation (DAT) when it needs to find the correct page for the correct address space. It is very desirable to have one's addresses in the TLB. With 4 K pages, holding all the addresses for 1 MB of storage takes 256 TLB lines. When using 1 MB pages, it takes only 1 TLB line. This means that large page size exploiters have a much smaller TLB footprint. Hardware system area The hardware system area (HSA) is a non-addressable storage area that contains server Licensed Internal Code and configuration-dependent control blocks. The HSA has a fixed size of 16 GB and is not part of the purchased memory that is ordered and installed. 93 z196 Redundant Array of Independent Memory (RAIM) z196 introduces the redundant array of independent memory (RAIM). The RAIM design detects and recovers from DRAM, socket, memory channel or DIMM failures. The RAIM design requires the addition of one memory channel that is dedicated for RAS The parity of the four “data” DIMMs are stored in the DIMMs attached to the fifth memory channel. Any failure in a memory component can be detect and corrected dynamically. This design takes the RAS of the memory subsystem to another level, making it essentially a fully fault tolerant “N+1” design. 94 47 z196 Memory Offerings Purchase Memory – Memory available for assignment to LPARs Model Standard Memory GB Flexible Memory GB M15 32 - 704 NA M32 32 - 1520 32 - 704 M49 32 - 2288 32 - 1520 M66 32 - 3056 32 - 2288 M80 32 - 3056 32 - 2288 Hardware System Area – Standard 16 GB outside customer memory for system use Standard Memory – Provides minimum physical memory required to hold base purchase memory plus 16 GB HSA Flexible Memory – Provides additional physical memory needed to support activation base customer memory and HSA on a multiple book z196 with one book out of service. Plan Ahead Memory – Provides additional physical memory needed for a concurrent upgrade (LIC CC change only) to a preplanned target customer memory 95 z196 Connectivity 96 48 z196 I/O Infrastructure Book 0 Book 1 Book 2 Book 3 Memory Memory Memory Memory PU PU PU PU PU PU PU PU FBC/L4 Cache PU FBC/L4 Cache FBC/L4 Cache PU PU HCA PU PU PU PU PU PU HCA PU PU PU FBC/L4 Cache PU PU PU PU HCA2-C fanout HCA HCA 1st level CopperCables 6 GBps IFB-MP RII IFB-MP IFB-MP RII 2 GBps mSTI IFB-MP IFB-MP RII 500 MBps mSTI IFB-MP IFB-MP IFB-MP RII 2nd level Embedded 333 MBps mSTI 2 GBps mSTI 1 GBps mSTI 97 . ESCON ESCON ESCON ESCON . 2GBps mSTI FPGA ISC ISC ISC ISC FPGA . Channels Coupling Links Channels Ports FICON Express8 2/4/8 Gbps ISC-3 ESCON OSA-Express3 10 GbE z196 system structure for I/O Acronym Full name AID Adapter identification CIB Coupling using InfiniBand HCA Host Channel Adapter MBA Memory Bus Adapter PSIFB Parallel Sysplex using InfiniBand IFB-MP InfiniBand Multiplexer STI-MP Self-Timed Interconnect Multiplexer 98 49 z196 Channel Type and Crypto Overview Supported Channel Types I/O Channels FICON Express8S FICON Express8 FICON Express4 (Carry forward on upgrade) ESCON – Migrate Away (240 or fewer) Networking OSA-Express4S OSA-Express3 • 10 Gigabit Ethernet LR and SR • Gigabit Ethernet LX and SX • 1000BASE-T Ethernet OSA-Express2 (Carry forward on upgrade) • 1000BASE-T Ethernet • Gigabit Ethernet LX and SX HiperSockets (Define only) Coupling Links InfiniBand Coupling Links • 12x InfiniBand • 1x InfiniBand ISC-3 – Migrate Away (Peer mode only) IC (Define only) 99 Crypto Crypto Express3 • Configurable Coprocessor / Accelerator OSA Express 3 Configuration Options Summary CHPID type Purpose / Traffic Operating Systems OSD All OSA features z196, z10, z9, zSeries Supports Queue Direct Input/Output (QDIO) architecture TCP/IP traffic when Layer 3 (uses IP address) Protocol-independent when Layer 2 (uses MAC address) z/OS, z/VM z/VSE, z/TPF Linux on Mainframe Non-QDIO; for SNA/APPN/HPR traffic and TCP/IP “passthru” traffic z/OS, z/VM z/VSE OSA-Integrated Console Controller (OSA-ICC) Supports TN3270E, non-SNA DFT to IPL CPCs & LPs z/OS, z/VM z/VSE OSM 1000BASE-T z196 exclusive OSA-Express for Unified Resource Manager Connectivity to intranode management network (INMN) from z196 to Unified Resource Manager functions z/OS, z/VM Linux on Mainframe* OSN GbE, 1000BASE-T z196, z10, z9 exclusive OSA-Express for NCP Appears to OS as a device supporting CDLC protocol Enables Network Control Program (NCP) channel-related functions Provides LP-to-LP connectivity OS to IBM Communication Controller for Linux (CCL) z/OS, z/VM z/VSE, z/TPF Linux on Mainframe OSX 10 GbE z196 exclusive OSA-Express for zBX Connectivity and access control to intraensemble data network (IEDN) from z196 to zBX z/OS, z/VM Linux on Mainframe* OSE 1000BASE-T z196, z10, z9, zSeries OSC 1000BASE-T z196, z10, z9, z990, z890 *IBM is working with its Linux distribution partners to include support in future Linux on Mainframe distribution releases. 100 50 FICON Express8S – SX and 10KM LX in the PCIe I/O drawer For FICON, zHPF, and FCP environments – CHPID types: FC and FCP • 2 PCHIDs/CHPIDs Auto-negotiates to 2, 4, or 8 Gbps Increased performance compared to FICON Express8 FLASH 2, 4, 8 Gbps SFP+ HBA ASIC IBM ASIC 2, 4, 8 Gbps SFP+ HBA ASIC IBM ASIC 10KM LX - 9 micron single mode fiber – Unrepeated distance - 10 kilometers (6.2 miles) – Receiving device must also be LX FLASH PCIe Switch FC 0409 – 10KM LX, FC 0410 – SX SX - 50 or 62.5 micron multimode fiber – Distance variable with link data rate and fiber type – Receiving device must also be SX 2 channels of LX or SX (no mix) Small form factor pluggable (SFP) optics – Concurrent repair/replace action for each SFP LX OR SX 101 z196 HiperSockets – doubled the number High-speed “intraserver” network Linux LP2 z/OS LP1 Independent, integrated, virtual LANs Communication path – system memory HiperSockets Communication across LPARs – Single LPAR - connect up to 32 HiperSockets Support for multiple LCSS's & spanned channels Virtual LAN (IEEE 802.1q) support z/VSE LP3 L2 z/VM LP5 HiperSockets Network Concentrator Broadcast support for IPv4 packets L3 z/VM LP4 IPv6 HiperSockets Network Traffic Analyzer (HS NTA) No physical cabling or external connections required 102 51 z196 Coupling Link Options Type Description Use Link data rate Distance IC (ICP) Internal Coupling Channel Internal communication Internal speeds NA 32 12x InfiniBand z196 & z10 z196 & z10 to z9 6 GBps 3 GBps* 150 meters (492 feet) 32 1x InfiniBand z196 & z10 5 Gbps or 2.5 Gbps** 10 km unrepeated (6.2 miles) 100 km repeated 32 32 InterSystem Channel-3 z196, z10, z9 2 Gbps 10 km unrepeated (6.2 miles) 100 km repeated 48 48 InfiniBan d (CIB) InfiniBan d (CIB) ISC-3 (CFP) z196 z196 Maximum Max links z196 Max CHPIDs NA 128 CHPIDs Maximum of 16 InfiniBand fanouts are allowed, 2 links per fanout Maximum of 128 coupling CHPIDs (ICP, CIB, CFP) per server Each InfiniBand link supports definition of multiple CIB CHPIDs, up to 16 per fanout * z196 & z10 negotiate to 3 GBps when connected to a z9 ** May negotiate to 2.5 Gbps when connected to a DWDM Note: ICB-4 is not supported on z196 103 InfiniBand coupling links on z196 and z114 Summary Type Speed Distance Fanout Cabling 50µ MM (OM3) fiber 9µ SM fiber 12x InfiniBand 6 or 3 GBps 150 meters HCA2-O HCA3-O 1x InfiniBand 5 or 2.5 Gbps 10 km HCA2-O LR* HCA3-O LR IFB IFB HCA2-O Up to 16 CHPIDs – across 2 ports IFB IFB HCA2-O LR* Up to 16 CHPIDs – across 2 ports IFB & IFB3 HCA3-O IFB & IFB3 Up to 16 CHPIDs – across 2 ports** IFB IFB IFB HCA3-O LR IFB Up to 16 CHPIDs – across 4 ports** Ports exit from the front of a book/processor drawer with HCAs. Unlike ISC-3 links, does not use I/O card slots 12x InfiniBand – z196, z114, z10, z9 – 6 GBps • z196/z114 and z10 – 3 GBps • z196/z114 & z10 to z9 • z9 to z9 connection not supported 1x InfiniBand – z196/z114 and z10 (not available for z9) – 5 Gbps (Server to Server and with DWDM) – 2.5 Gbps (with DWDM) * For z114 carry forward only ** Performance considerations may reduce the number of CHPIDs per port 104 52 Mainframe Crypto History 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Cryptographic Coprocessor Facility (CCF) Cryptographic Coprocessor Single Chip Module G3, G4, G5, G6, z900, z800 OS/390 ICSF PCI Cryptographic Coprocessor (PCICC) G5, G6, z900, z800 PCI Cryptographic Accelerator (PCICA) z800/z900 PCIX Cryptographic Coprocessor (PCIXCC) z990 z890 z990 z890 CP Assist for Cryptographic Functions z990 Crypto Express2 z890 z990/z890 z9 EC z9 BC z10 EC/BC z9 EC z9 BC z10 EC/BC Crypto Express3 z10 EC/BC z196 Cryptographic Coprocessor Facility – Supports “Secure key” cryptographic processing PCICC Feature – Supports “Secure key” cryptographic processing PCICA Feature – Supports “Clear key” SSL acceleration PCIXCC Feature – Supports “Secure key” cryptographic processing CP Assist for Cryptographic Function allows limited “Clear key” crypto functions from any CP/IFL – NOT equivalent to CCF on older machines in function or Crypto Express2 capability Crypto Express2 – Combines function and performance of PCICA and PCICC Crypto Express3 – PCI-e Interface, additional processing capacity with improved RAS 105 Crypto Express3 2-P Physical View Coprocessor #1 PCIe interfaces Coprocessor #2 106 53 Crypto Express3 features highlights Virtualization: all logical partitions (LPARs) in all Logical Channel Subsystems (LCSSs) have access to the Crypto Express3 feature, up to 32 LPARs per feature Designed for improved reliability, availability and serviceability (RAS) Executes its cryptographic functions asynchronously to a Central Processor (CP) operation Dynamic addition / configuration of cryptographic features to logical partitions without an outage Dynamic power management to maximize RSA performance while keeping within temperature limits of the tamper-responding package Secure code loading that enables the updating of functionality while installed in application systems Lock-step-checking of dual CPUs for enhanced error detection and fault isolation of cryptographic operations performed by coprocessor when a PCIe adapter is defined as a coprocessor Updated cryptographic algorithms used in loading the Licensed Internal Code (LIC) with the TKE workstation to keep in step with current recommendations for cryptographic strength Support for smart card applications using Europay, MasterCard, Visa specifications Health Monitoring of mesh, temperature, voltage, soft tamper and low battery. Monitoring is internal to the card. Cannot be accessed or exploited by the customer 107 z196 New and exclusive cryptographic capabilities Elliptic Curve Cryptography Digital Signature Algorithm (ECC), an emerging public key algorithm expected eventually to replace RSA cryptography in many applications. ECC is capable of providing digital signature functions and key agreement functions. The new CCA functions provide ECC key generation and key management and provide digital signature generation and verification functions compliance with the ECDSA method described in ANSI X9.62 "Public Key Cryptography for the Financial Services Industry: The Elliptic Curve Digital Signature Algorithm (ECDSA)". ECC uses keys that are shorter than RSA keys for equivalent strength-per-key-bit; RSA is impractical at key lengths with strength-per-key-bit equivalent to AES-192 and AES-256. So the strength-per-key-bit is substantially greater in an algorithm that uses elliptic curves. ANSI X9.8 PIN security which facilitates compliance with the processing requirements defined in the new version of the ANSI X9.8 and ISO 9564 PIN Security Standards and provides added security for transactions that require Personal Identification Numbers (PIN). Enhanced Common Cryptographic Architecture (CCA), a Common Cryptographic Architecture (CCA) key token wrapping method using Cipher Block Chaining (CBC) mode in combination with other techniques to satisfy the key bundle compliance requirements in standards including ANSI X9.24-1 and the recently published Payment Card Industry Hardware Security Module (PCI HSM) standard. Secure Keyed-Hash Message Authentication Code (HMAC), a method for computing a message authentication code using a secret key and a secure hash function. It is defined in the standard FIPS 198, "The Keyed-Hash Message Authentication Code". The new CCA functions support HMAC using SHA-1, SHA-224, SHA-256, SHA-384, and SHA-512 hash algorithms. The HMAC keys are variablelength and are securely encrypted so that their values are protected. Modulus Exponent (ME) and Chinese Remainder Theorem (CRT), RSA encryption and decryption with key lengths greater than 2048-bits and up to 4096-bits. 108 54 Certifications for Mainframe z/VM The Common Criteria program developed by NIST and NSA establishes an organizational and technical framework to evaluate the trustworthiness of IT Products and protection profiles Common Criteria EAL4+ ‒ z/VM 5.4 + RACF with CAPP and LSPP – designed for compliance ‒ Plan to have z/VM 6.1 evaluated during 2011 z/OS z/VM Linux Linux Linux on Mainframe Linux z/OS Common Criteria EAL4+ Virtualization with partitions ‒ with CAPP and LSPP Cryptography ‒ z/OS 1.10 z9 EC, z9 BC, z10 EC, z10 BC – EAL5 ‒ z/OS 1.11 June 2010 ‒ z/OS 1.12 under evaluation IdenTrust™ certification for z/OS as a Digital Certificate Authority (PKI Services) Common Criteria ‒ Novell SUSE SLES9 and 10 certified at EAL4+ with CAPP ‒ RHEL5 EAL4+ with CAPP and LSPP z196 - EAL5+ Common Criteria EAL5 for Logical partitions FIPS 140-2 level 4 for Crypto Express2 See: www.ibm.com/security/standards/st_evaluations.shtml 109 z196 RAS 110 55 Mainframe overall RAS Strategy …..Continuing our RAS focus helps avoid outages Sources of Outages Pre z9 -Hrs/Year/Syst- Prior Servers Impact of Outage Unscheduled Outages Scheduled Outages Planned Outages Scheduled (CIE+Disrupt ive Pat ches + ECs) Preplanning requirements Planned - (MES + Driver Upgrades) Unscheduled (UIRA) Power & Thermal Management z9 EC Z10 EC z196 Temperature = Silicon Reliability Worst Enemy Wearout = Mechanical Components Reliability Worst Enemy. 111 Redundancy in Mainframe Hardware Power 2x Power Supply 2x Power feed Internal Battery Feature Optional internal battery in cause of loss of external power) Cooling, air & water-cooling options Dynamic oscillator switchover Processors Multiprocessors Spare PUs Memory RAIM Chip sparing Error Correction and Checking Enhanced book availability The zEnterprise System continues to reduce customer down time by focusing on all sources of outages: unscheduled outages, scheduled outages and planned outages. Power and cooling requirements were reduced while still managing reliability. – Major new Memory design for preventing outages – Introducing new IO drawers technologies designed for concurrent service – Introducing Mainframe management to the mixed computing environment – Delivering Green functions and RAS together 112 56 Preventing All Outages Unscheduled Outages Advanced Memory RAIM (Redundant Array of Independent Memory) design Enhanced Reed-Solomon code (ECC) – 90B/64B Protection against Channel/DIMM failures Chip marking for fast DRAM replacements Mirrored Key cache Improved chip packaging Improved condensation management Integrated TCP/IP checksum generation/checking Integrated EPO switch cover (protecting the switch during repair actions) Continued focus on Firmware Scheduled Outages Double memory data bus lane sparing (reducing repair actions) Single memory clock bus sparing Field Repair of interface between processor chip and cache chip and between cache chips (fabric bus) Fast bitline delete on L3/L4 cache (largest caches) Power distribution using N+2 Voltage Transformation Modules (VTM) Redundant (N+2) Humidity Sensors Redundant (N+2) Altitude Sensors Unified Resource Manager for zBX 113 Improving Reliability Power & Thermal Optimization and Management Static Power Save Mode under z/OS control Smart blower management by sensing altitude and humidity Enhanced evaporator design to minimize temperature variations MCM cooling with N+1 design feature MCM backup cooling with Heat Exchanger Available air to water heat exchanger for customer water cooling Concurrent Maintenance and Upgrades – Duplex Units • Power Supplies, – Concurrent Microcode (Firmware) updates – Hot Pluggable I/O – PU Conversion – Permanent and Temporary Capacity Upgrades • Capacity Upgrade on Demand (CUoD) • Customer Initiated Upgrade (CIU) • On/Off Capacity on Demand (On/Off CoD) – Capacity BackUp (CBU) – Flexible & Plan Ahead Memory Options 114 57 z196 RAS Design of ……..Fully Redundant I/O Subsystem – of existing IO cage and drawers processor Fully Redundant I/O Design Failover I/O Hub I/O Hub Failover I/O Mux I/O Mux I/O Mux I/O Mux I/O Switch I/O Switch Network Switch Network Switch Control Unit Control Unit End User End User ISC Links ISC Links Crypto Accelerator Crypro Accelerator Network adapter. Network adapter. I/O adapter. I/O adapter. Alternate Path SAP / CP sparing SAP Reassignment I/O Reset & Failover I/O Mux Reset / Failover Redundant I/O Adapter Redundant I/O interconnect Redundant Network Adapters Redundant ISC links Redundant Crypto processors I/O Switched Fabric Network Switched/Router Fabric High Availability Plugging Rules I/O and coupling fanout rebalancing on CBA Channel Initiated Retry High Data Integrity Infrastructure I/O Alternate Path Network Alternate Path Virtualization Technology 115 Create a redundant I/O configuration LPARn LPAR2 LPAR1 LPARn LPAR2 LPAR1 CSS / CHPID Director (Switch) DASD CU DASD CU .... 116 58 BACKUP Slides 117 Did You Know... 118 59 Did You Know... 119 Did You Know... 120 60 Did You Know... 121 Quick Quiz: Which is the Better Vehicle? VS. Maserati MC12 Peterbilt Semi The Race - ¼ Mile Dragstrip 122 61 Quick Quiz: Which is the Better Vehicle? VS. 100 Maserati MC12 Peterbilt Semi The Race - ¼ Mile Dragstrip Carrying 10 tons of crates 123 Is this Better? VS. Peterbilt Semi 10 Ford F-450 1 ton Pickups The Race - ¼ Mile Dragstrip Carrying 10 tons of crates 124 62 Mainframe History - Myth vs. Truth http://www-01.ibm.com/software/info/getztruth/index.jsp ftp://public.dhe.ibm.com/s390/misc/bookoffer/download/360revolution_040704.pdf http://www.vm.ibm.com/devpages/jelliott/pdfs/zhistory.pdf ftp://ftp.software.ibm.com/s390/audio/pdfs/zSeriesMainframeInnovations.pdf http://www.ca.com/us/bigiron.aspx 125 Academic Initiative: Success Story IBM Training Center for Mainframe opens at VIT, Vellore, India – Students in Indian universities and colleges are generally taught about personal computers and small server technology. But mainframes run businesses, and students need exposure to this critical technology before they enter the workforce to maximize their effectiveness. The need was recognized by Chancellor G. Viswanathan of the Vellore Institute of Technology (VIT) in India – as well as by an IBM team looking for engineering institutes. • The opportunity opened in September 2010 for IBM to establish a partnership with VIT, a top-tier academic institute, which was ready to commit to nurturing and growing a System z ecosystem. Developed as part of the IBM Academic Initiative System z Program, this unique initiative builds partnerships with colleges and universities worldwide to add IBM enterprise systems education to curricula and put the mainframe back in computer science programs. – The program kicked off with several training sessions that the India System z ISV team conducted at the VIT campus to educate professors and students on z/OS concepts. IBM helped the university explore enterprise system solutions as they built a long-term partnership with professors and the student community. In August, the “IBM Training Center for Mainframes” was officially launched at VIT. This is the second Training Center for Mainframe of its type in the world, and the first one in Asia. It has 60 dedicated high-end computers with 24/7 connectivity to the centralized System z in Dallas, Texas. – VIT has added a course on “Introduction to Mainframe Computing” and offered it as an elective in its Master of Computer Applications curriculum. – Chancellor Viswanathan has referred to IBM as a big role model and ensured that VIT will make use of IBM‟s presence to foster innovation. “India has the big advantage of producing the largest number of IT professionals in the world. The VIT will try to emerge as one of IBM‟s best partners,” he said. 126 63 Master the Mainframe Contest No experience with mainframes is necessary. In fact, the contest is designed for students with little or no mainframe experience, increasing with difficulty as the contest progresses. Students just need to bring drive and competitive spirit and be ready to compete! Experience! Today's mainframes are growing in popularity and require a new generation of mainframe experts. This contest is designed to equip you with basic skills to make you more competitive for jobs in the enterprise computing industry. Participation in the Master the Mainframe contest could give you the edge you need. 127 References Introduction to the New Mainframe: Large-Scale Commercial Computing, SG24-7175-00 IBM Mainframe Strengths and Values, SG24-7333-01 Introduction to the New Mainframe: z/OS Basics, SG24-6366-01 ABCs of z/OS System Programming Volume 1, SG24-6981-01 ABCs of z/OS System Programming Volume 2, SG24-6982-02 ABCs of z/OS System Programming Volume 3, SG24-6983-03 ABCs of z/OS System Programming Volume 4, SG24-6984-00 ABCs of z/OS System Programming Volume 5, SG24-6985-01 ABCs of z/OS System Programming Volume 6, SG24-6986-00 ABCs of z/OS System Programming Volume 7, SG24-6987-01 ABCs of z/OS System Programming Volume 8, SG24-6988-00 ABCs of z/OS System Programming Volume 9, SG24-6989-04 ABCs of z/OS System Programming Volume 9, SG24-6989-05 ABCs of z/OS System Programming Volume 10, SG24-6990-03 ABCs of z/OS System Programming Volume 11, SG24-6327-01 ABCs of z/OS System Programming Volume 12, SG24-7621-00 ABCs of z/OS System Programming Volume 13, SG24-7717-00 IBM zEnterprise 196 Technical Guide, SG24-7833-00 IBM zEnterprise System Technical Introduction, SG24-7832-00 IBM zEnterprise 114 Technical Guide, SG24-7954-00 IBM System z Connectivity Handbook, SG24-5444-12 z/Architecture Principles of Operation, SA22-7832-08 z/Architecture Reference Summary, SA22-7871-06 128 64