AIX Performance Monitoring OCEAN Tech Conference Irvine - July 2009 Mike Rede Field Technical Sales Specialist Power Systems mrede@us.ibm.com © 2009 IBM Corporation Agenda • Topas/NMON • LPARSTAT • LSLPARUTIL • LPAR2RRD • PM for Power Systems 2 © 2009 IBM Corporation CPU Monitoring tools • lparstat (new in 5.3, interval count) • mpstat (new in 5.3, show CPU utilization) • • procmon (new in 5.3) topas ( display process info) • sar • iostat • vmstat • • ps trace • • • • • curt splat truss gprof, pprof, prof, tprof time, timex – > lparstat 1 5 > lparstat -i – > mpstat 1 5 – > topas -P – > sar 1 5 > sar -P ALL 1 3 – > iostat -t 1 5 (show CPU utilization) – > vmstat 1 5 – > trace -a; ls; trcstop (then) > ls -l /var/adm/ras/trcfile 3 © 2009 IBM Corporation Agenda • Topas/NMON • LPARSTAT • LSLPARUTIL • LPAR2RRD • PM for Power Systems 4 © 2009 IBM Corporation AIX Enhancements • nmon integration into topas – nmon function has been integrated into topas infrastructure including the ability to use nmon analyzer to process topas output – Installed by default with AIX & fully supported by AIX Support $ nmon script /usr/bin/nmon /usr/bin/topas_nmon or $ topas and hit ~ – Identical to nmon “classic” V12 – Including saving nmon data to file •nmon –f –s300 –c 288 just works as normal – Additional smitty support to make saving a daily nmon file easy – For older AIX releases, continue to use nmon “classic” •Which is supported via the AIX Wiki/Performance Tools Forum 5 © 2009 IBM Corporation Topas/nmon SMIT Menus 6 © 2009 IBM Corporation topas - CEC monitoring screen Split screen accessible with -C or the "C" command Upper section shows CEC-level metrics Lower sections shows sorted list of shared and dedicated partitions Configuration info retrieved from HMC or specified from command line c means capped, C - capped with SMT u means shared, U - uncapped with SMT S means SMT Use topas –R to record to /etc/perf/topas_cec.YYMMDD and topasout to display recording Sample Full-Screen Cross-Partition Output Topas CEC Monitor Interval: 10 Wed Mar 6 14:30:10 2005 Partitions Memory (GB) Processors Shr: 4 Mon: 24 InUse: 14 Mon: 8 PSz: 4 Shr_PhysB: 1.7 Ded: 4 Avl: 24 Avl: 8 APP: 4 Ded_PhysB: 4.1 Host OS M Mem InU Lp Us Sy Wa Id PhysB Ent %EntC Vcsw PhI --------------------------shared-----------------------------------------ptools1 A53 u 1.1 0.4 4 15 3 0 82 1.30 0.50 22.0 200 5 ptools5 A53 U 12 10 1 12 3 0 85 0.20 0.25 0.3 121 3 ptools3 A53 C 5.0 2.6 1 10 1 0 89 0.15 0.25 0.3 52 2 ptools7 A53 c 2.0 0.4 1 0 1 0 99 0.05 0.10 0.3 112 2 -------------------------dedicated---------------------------------------ptools4 A53 S 0.6 0.3 2 12 3 0 85 0.60 ptools6 A52 1.1 0.1 1 11 7 0 82 0.50 ptools8 A52 1.1 0.1 1 11 7 0 82 0.50 ptools2 A52 1.1 0.1 1 11 7 0 82 0.50 http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.doc/doc/base/aixinformation.htm 7 © 2008 IBM Corporation Agenda • Topas/NMON • LPARSTAT • LSLPARUTIL • LPAR2RRD • PM for Power Systems 8 © 2009 IBM Corporation LPARSTAT command • LPARSTAT command usage: • The "lparstat" command shows the available physical processors in the shared pool on a micropartitioned server. • The lparstat command can be run from any logical partition that has enabled "Allow shared processor pool utilization authority". • Use the values of “app” and “psize” that are reported in the output to calculate the overall CPU utilization percent. 9 © 2009 IBM Corporation LPARSTAT command P595-1 ---------# lparstat System configuration: type=Sharedmode=Uncapped smt=Onlcpu=4mem=12288psize=64ent=0.50 %user%sys%wait %idlephysc %entclbusyappvcsw phint ----------------------------- -------------- ----0.7 0.9 0.6 97.8 0.01 1.8 1.1 51.503188322977 74323912 • “psize” is the size of the shared processor pool. • “app” is the number of available physical processors. 10 © 2009 IBM Corporation LPARSTAT command P595-2 ---------# lparstat System configuration: type=Sharedmode=Uncapped smt=Onlcpu=24mem=15360psize=56ent=1.20 %user%sys%wait%idlephysc %entclbusyappvcsw phint -------------------------------- ---------------- ----5.41.4 0.7 92.5 0.09 7.4 3.3 37.6820060323490603098954 • “psize” is the size of the shared processor pool. • “app” is the number of available physical processors. 11 © 2009 IBM Corporation LPARSTAT command P595-3 ---------# lparstat System configuration: type=Sharedmode=Uncapped smt=Onlcpu=24mem=16384psize=64ent=2.00 %user%sys%wait%idle physc %entclbusyappvcsw phint ----- ----------------- ----- ---------------------- ----5.02.5 0.8 91.7 0.16 8.2 2.4 41.90378202018391134448816 • “psize” is the size of the shared processor pool. • “app” is the number of available physical processors. 12 © 2009 IBM Corporation LPARSTAT command • The "lparstat" command output shows: – the size of the shared pool is 64 CPU's (psize) on P595-3 and P595-1. – the size of the shared pool is 56 CPU's (psize) on P595-2. • The overall CPU utilization percentage for each machine can be calculated this way: – Overall CPU utilization = (psize - app) / psize – P595-1 has a CPU utilization percentage of 19.5 % = (64 - 51.5) / 64) – P595-2 has a CPU utilization percentage of 32.7 % = (56 - 37.68) / 56) – P595-3 has a CPU utilization percentage of 34.5 % = (64 - 41.9) / 64) 13 © 2009 IBM Corporation LPARSTAT command • These results are representative of the AVERAGE CPU Utilization for the last couple seconds. • To compare the DAILY AVERAGE using the “lparstat” command – and to compare to the results of the “lpar2rrd” graph – you would have to collect data for a 24 hour period like this: • # lparstat 86400 1 14 © 2009 IBM Corporation Agenda • Topas/NMON • LPARSTAT • LSLPARUTIL • LPAR2RRD • PM for Power Systems 15 © 2009 IBM Corporation LSLPARUTIL command • Calculate the CPU utilization also as a percent. • Use the "lslparutil" command to look at the "total_pool_cycles" and "utilized_pool_cycles" for each server. • Secure login (SSH) to the HMC through a PuTTY session. • > lslparutil –r pool –m (servername) –F total_pool_cycles utilized_pool_cycles – NOTE: You can also get the same information - reported by “lslparutil” – by using the HMC GUI screen and “View”ing the Utilization Data. 16 © 2009 IBM Corporation LSLPARUTIL command - example > lslparutil –r pool –m (server name) –F total_pool_cycles utilized_pool_cycles 8967037181413078158257737255071 • Notice that the “utilized_pool_cycles” value is 15 digits. – 158257737255071 • Notice that the “total_pool_cycles” value is 16 digits. – 8967037181413078 • If you divide “utilized” by “total” (1.58 by 89) you can get a rough estimate of the CPU utilization. In this case it is equal to 1.77 %. 17 © 2009 IBM Corporation LSLPARUTIL command - examples • P595-1 - 64 active cpu, 453GB – utilized_pool_cycles = 174287276922931140 – total_pool_cycles =913273813303689687 • P595-2 - 56 active cpu out of 64 installed, 1TB – utilized_pool_cycles = 157461400067418451 – total_pool_cycles =487460714682764159 • P595-3 - 64 active cpu, 1TB – utilized_pool_cycles = 192316829352525158 – total_pool_cycles =557721130434850807 18 © 2009 IBM Corporation LSLPARUTIL command • Using the first two digits (and rounded) from utilized_pool_cycles and total_pool_cycles and doing the math shows that: • P595-1 has a Utilization percentage of 17/91 = 19% • P595-2 has a Utilization percentage of 16/49 = 32% • P595-3 has a Utilization percentage of 19/56 = 34% 19 © 2009 IBM Corporation LPARSTAT comparison • P595-1 has a CPU utilization percentage of 19.5 % = (64 - 51.5) / 64 • P595-2 has a CPU utilization percentage of 32.7 % = (56 - 37.68) / 56 • P595-3 has a CPU utilization percentage of 34.5 % = (64 - 41.9) / 64 • So, "lparstat" and "lslparutil" CPU utilization percentages match up. 20 © 2009 IBM Corporation Agenda • Topas/NMON • LPARSTAT • LSLPARUTIL • LPAR2RRD • PM for Power Systems 21 © 2009 IBM Corporation LPAR2RRD Tool • LPAR2RRD graphical tool: • http://www.ibm.com/developerworks/wikis/display/virtualizatio n/lpar2rrd+tool • "lpar2rrd" tool gets its data directly from the "lslparutil" command. 22 © 2009 IBM Corporation LPAR2RRD Tool The CPU Utilization graph for the P595-1 shows the average utilization taken over time on Monday and into Tuesday. 23 © 2009 IBM Corporation LPAR2RRD Tool There are 64 available processors = 49 Configured + 15 Borrowed. So the CPU utilization was calculated as 10.39 / 64 = 16.2 % . But that was for the last day. 24 © 2009 IBM Corporation LPAR2RRD – last month CPU utilization 25 © 2009 IBM Corporation LPAR2RRD Tool – last year 26 © 2009 IBM Corporation Agenda • Topas/NMON • LPARSTAT • LSLPARUTIL • LPAR2RRD • PM for Power Systems 27 © 2009 IBM Corporation PM for Power Systems • An automated capacity planning and performance analysis report and graphs offering. • Designed to help you plan for and manage the growth and performance of your system. • Summary level • Detailed level 28 © 2009 IBM Corporation PM for Power Sytems • Summary level: – The summary level report offering is available for no additional charge. It provides you with summary level measurements on key performance metrics about your system or specific partition being monitored. – Also included is a projection, based on your actual data, of your utilization and growth against the capacities of the system or partition. – To qualify for the no additional charge option, the system has to be under processor warranty or on an IBM Hardware Maintenance Agreement. 29 © 2009 IBM Corporation PM for Power Sytems • Detailed level: – For a nominal additional charge, an IBM Global Services fee service provides all of the features of the no charge offering plus additional information. – To substantiate the data on the summary graphs, up to 26 additional more detailed reports and graphs are available. These graphs are designed to help you begin to understand the detail behind some of the summary averages. – As a client, you also have access to the interactive graphing function of the offering. This allows you to drill down into selected graphs to see the data at an hourly or daily level; and you can scroll backwards in time to see the same graphs for previous months to help create a visual comparison. 30 © 2009 IBM Corporation IBM Performance Management for Power Systems - - What is it and what are the options? • PM for Power Systems (AIX support): an offering to simplify your systems management by helping you plan for and manage the growth and performance of the IBM Power System family • Previously known as PM for System p • Supports AIX (Version 5.2 or later) • Note: PM for Power Systems also supports the IBM i operating system • Two offerings for customers to choose from: (both available worldwide) http://www.ibm.com/systems/power/support/pm/ or http://www.ibm.com/systems/p/support/pm/ • No Charge service….. A no additional charge option that provides web delivered interactive access to key performance and growth data plus a projection of remaining growth of the system or partition - System must be under IBM warranty or on an IBM hardware maintenance agreement to be eligible • Full function service….. An IBM Global Technology Services (billable) report set that provides multiple detail reports on an ongoing basis depicting the growth and performance of the system • Available either as a stand alone fee offering or as part of an IBM premium service offering like the IBM Enhanced Technical Support offering 31 © 2009 IBM Corporation PM for Power Systems data and workflow A rolling history of 24 months of client performance and growth data is securely stored at IBM Workload Estimator integration…for sizing future options 32 © 2009 IBM Corporation All clients receive a record for each of their partitions reporting data on the Server Information page A tab is available showing IBM i systems; another shows IBM p systems Select the report icon on the left nav for the respective system or partition 33 © 2009 IBM Corporation Interactive Management Summary Graphs The No Additional Charge graphs This data is available for no additional charge if the system is under warranty or on an IBM Hardware Maintenance Agreement. The individual graphs are accessed interactively by the user and are then printable as Individual graphs. The user does have the ability to go ‘backwards in time’ to select the month of interest. (up to 24 mos) 34 © 2009 IBM Corporation AIX CPU daily graph 35 © 2009 IBM Corporation Selecting other views of the data (interactive graphing example) Please select a graph from the list Average Processor Utilization in % per day Usage by the Hour/Average during the period Peak Processor Utilization RUN Q > Limit in % per day RUN Q > Limit by Hour during the period 36 © 2009 IBM Corporation AIX Usage by Hour – monthly view (interactive graphing example continued) 37 © 2009 IBM Corporation Peak processor utilization 38 © 2009 IBM Corporation Memory utilization per hour 39 © 2009 IBM Corporation Disk Space usage – daily (hard disk and file system) 40 © 2009 IBM Corporation PM for Power Systems • Remember most everyone can take advantage of these offerings and tools for no additional charge. Step one is to get started with PM for Power Systems. Simply visit the following URL, select your operating system interest and then follow the ‘getting started’ instructions. • http://www.ibm.com/systems/power/support/pm/ 41 © 2009 IBM Corporation Thank you! © 2009 IBM Corporation Useful Links – 1 of 3 •IBM POWER Systems – http://www-03.ibm.com/systems/power/ •Toolbox for IT – http://unix.ittoolbox.com/groups/technical-functional/ibm-aixl/ •AIX and UNIX Forums – http://www.ibm.com/developerworks/forums/dw_auforums.jsp a •AIX and POWER Portal – http://rootvg.net/ •Technotes & Tips – http://www03.ibm.com/support/techdocs/atsmastr.nsf/Web/Technotes •IBM Systems Information Center – http://publib.boulder.ibm.com/eserver/ 43 © 2009 IBM Corporation Useful Links – 2 of 3 •Fix Central – http://www912.ibm.com/eserver/support/fixes/fixcentral/main/pseries/ai x •Contacting IBM (Support) – http://techsupport.services.ibm.com/guides/pseries/contacting. shtml •IBM - Software Support Handbook – http://www14.software.ibm.com/webapp/set2/sas/f/handbook /home.html •IBM developerWorks: Wiki – http://www-941.haw.ibm.com/collaboration/wiki/ – http://www.ibm.com/developerworks/wikis/dashboard.action •Training – http://www.mux-ed.com/uglypink.pdf 44 © 2009 IBM Corporation Useful Links – 3 of 3 • AIX User Group Community – http://www.ibm.com/community/usergroups •IBM Redbooks – http://www.redbooks.ibm.com/ Power Virtualization: http://www.redbooks.ibm.com/abstracts/sg247940.html?Open Workload Partition Mgmt: http://www.redbooks.ibm.com/abstracts/sg247431.html?Open LPAR Simplification Tools Handbook: http://www.redbooks.ibm.com/abstracts/sg247231.html?Open Power VM - Managing and Monitoring: http://www.redbooks.ibm.com/abstracts/sg247590.html?Open AIX Enterprise Edition SA Guide: http://www.redbooks.ibm.com/redpieces/abstracts/sg247738.html?Open AIX 6.1 Differences Guide: http://www.redbooks.ibm.com/abstracts/sg247559.html?Open WLAR Mgmt in AIX: http://www.redbooks.ibm.com/abstracts/sg247656.html?Open Multi-Tenant Utility Computing on Power Systems and AIX: http://www.redbooks.ibm.com/abstracts/sg247681.html?Open •AIX Performance articles (first in a series (with links to others)): http://www.ibm.com/developerworks/aix/library/au-aixoptimization/index.html •Official IBM Visio Stencils: – http://www.visiocafe.com/ibm.htm 45 © 2009 IBM Corporation Special notices This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in other countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM offerings available in your area. Information in this document concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY 10504-1785 USA. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or guarantees either expressed or implied. All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations and conditions. IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions worldwide to qualified commercial and government clients. Rates are based on a client's credit rating, financing terms, offering type, equipment type and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension or withdrawal without notice. IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies. All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. Many of the features described in this document are operating system dependent and may not be available on Linux. For more information, please check: http://www.ibm.com/systems/p/software/whitepapers/linux_overview.html Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this document may have been made on development-level systems. There is no guarantee these measurements will be the same on generallyavailable systems. Some measurements quoted in this document may have been estimated through extrapolation. Users of this document should verify the applicable data for their specific environment. Revised January 19, 2006 46 © 2009 IBM Corporation Special notices (cont.) The following terms are registered trademarks of International Business Machines Corporation in the United States and/or other countries: AIX, AIX/L, AIX/L(logo), alphaWorks, AS/400, BladeCenter, Blue Gene, Blue Lightning, C Set++, CICS, CICS/6000, ClusterProven, CT/2, DataHub, DataJoiner, DB2, DEEP BLUE, developerWorks, DirectTalk, Domino, DYNIX, DYNIX/ptx, e business(logo), e(logo)business, e(logo)server, Enterprise Storage Server, ESCON, FlashCopy, GDDM, i5/OS, IBM, IBM(logo), ibm.com, IBM Business Partner (logo), Informix, IntelliStation, IQ-Link, LANStreamer, LoadLeveler, Lotus, Lotus Notes, Lotusphere, Magstar, MediaStreamer, Micro Channel, MQSeries, Net.Data, Netfinity, NetView, Network Station, Notes, NUMA-Q, Operating System/2, Operating System/400, OS/2, OS/390, OS/400, Parallel Sysplex, PartnerLink, PartnerWorld, Passport Advantage, POWERparallel, Power PC 603, Power PC 604, PowerPC, PowerPC(logo), PowerPC 601, Predictive Failure Analysis, pSeries, PTX, ptx/ADMIN, RETAIN, RISC System/6000, RS/6000, RT Personal Computer, S/390, Scalable POWERparallel Systems, SecureWay, Sequent, ServerProven, SpaceBall, System/390, The Engines of e-business, THINK, Tivoli, Tivoli(logo), Tivoli Management Environment, Tivoli Ready(logo), TME, TotalStorage, TURBOWAYS, VisualAge, WebSphere, xSeries, z/OS, zSeries. The following terms are trademarks of International Business Machines Corporation in the United States and/or other countries: Advanced MicroPartitioning, AIX 5L, AIX PVMe, AS/400e, Chipkill, Chiphopper, Cloudscape, DB2 OLAP Server, DB2 Universal Database, DFDSM, DFSORT, ebusiness(logo), e-business on demand, eServer, Express Middleware, Express Portfolio, Express Servers, Express Servers and Storage, GigaProcessor, HACMP, HACMP/6000, I5/OS (logo), IBMLink, IBM TotalStorage Proven, IMS, Intelligent Miner, iSeries, Micro-Partitioning, NUMACenter, ON DEMAND BUSINESS logo, OpenPower, POWER, Power Architecture, Power Everywhere, Power Family, Power PC, PowerPC Architecture, PowerPC 603, PowerPC 603e, PowerPC 604, PowerPC 750, POWER2, POWER2 Architecture, POWER3, POWER4, POWER4+, POWER5, POWER5+, POWER6, POWER6+, Redbooks, Sequent (logo), SequentLINK, Server Advantage, ServeRAID, Service Director, SmoothStart, SP, System i, System i5, System p, System p5, System Storage, System z, System z9, S/390 Parallel Enterprise Server, Tivoli Enterprise, TME 10, TotalStorage Proven, Ultramedia, VideoCharger, Virtualization Engine, Visualization Data Explorer, X-Architecture, z/Architecture, z/9. A full list of U.S. trademarks owned by IBM may be found at: http://www.ibm.com/legal/copytrade.shtml. UNIX is a registered trademark in the United States, other countries or both. Linux is a trademark of Linus Torvalds in the United States, other countries or both. Microsoft, Windows, Windows NT and the Windows logo are registered trademarks of Microsoft Corporation in the United States and/or other countries. Intel, Intel Xeon, Itanium and Pentium are registered trademarks or trademarks of Intel Corporation in the United States and/or other countries. AMD Opteron is a trademark of Advanced Micro Devices, Inc. Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States and/or other countries. TPC-C and TPC-H are trademarks of the Transaction Performance Processing Council (TPPC). SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and SPECsfs are trademarks of the Standard Performance Evaluation Corp (SPEC). NetBench is a registered trademark of Ziff Davis Media in the United States, other countries or both. Revised January 19, 2006 AltiVec is a trademark of Freescale Semiconductor, Inc. Other company, product and service names may be trademarks or service marks of others. 47 © 2009 IBM Corporation Thank you! © 2009 IBM Corporation