Front cover
Enterprise Workload
Manager z/OS Support
Installing the EWLM managed server
on z/OS
Customizing the EWLM
management domain
Evaluating performance
data
Paola Bari
G. Michael Connolly
Franco Meroni
ibm.com/redbooks
Redpaper
International Technical Support Organization
Enterprise Workload Manager z/OS Support
July 2005
Note: Before using this information and the product it supports, read the information in “Notices” on
page vii.
First Edition (July 2005)
This edition applies to the EWLM product within Version 1, Release 1 of the IBM Virtualization Engine
Management Servers (product number 5724-i71).
© Copyright International Business Machines Corporation 2005. All rights reserved.
Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.
iii
iv
Enterprise Workload Manager z/OS Support
Contents
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
The team that wrote this Redpaper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x
Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x
Chapter 1. Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.1 EWLM for z/OS description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.1.1 Preventive service planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.1.2 Installation requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.1.3 General considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 SMP/E installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2.1 Installing EWLM for z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 Middleware instrumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3.1 DB2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3.2 WebSphere Application Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1
2
2
3
5
6
6
8
8
8
Chapter 2. EWLM configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1 ITSO environment for the EWLM management domain . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2 Setting up z/OS as managed server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.1 Configuring IBM SDK for z/OS, Java 2 Technology Edition, Version 1.4 . . . . . . . 11
2.2.2 Updating the BPXPRMxx member in PARMLIB . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.3 Configuring a managed server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.4 Authorizing z/OS users to start the EWLM managed server. . . . . . . . . . . . . . . . . 14
2.3 Start EWLM on the z/OS managed server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.1 Enable instrumentation for middleware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.2 Operational commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4 Running the workload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.4.1 Create z/OS DB2 database for Trade3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.4.2 Distribute the DB2 Universal JDBC Driver for z/OS . . . . . . . . . . . . . . . . . . . . . . . 23
2.4.3 Installing the Trade3 distributed application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.4.4 Configure Trade3 JDBC driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.4.5 Verifying the workload through the Control Center . . . . . . . . . . . . . . . . . . . . . . . . 33
Chapter 3. Monitoring and reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.1 EWLM and zWLM policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 EWLM, WLM, and DB2 transaction concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 ARM enablement on z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Monitoring the workload with EWLM and z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 A comparison between monitoring reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.1 EWLM Control Center reports. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.2 RMF postprocessor workload activity reports . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.3 DB2 PE accounting and statistics reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.4 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5 Performance considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
36
40
41
43
58
58
61
63
67
68
Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
© Copyright IBM Corp. 2005. All rights reserved.
v
IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
vi
Enterprise Workload Manager z/OS Support
71
71
71
71
72
Notices
This information was developed for products and services offered in the U.S.A.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not give you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.
The following paragraph does not apply to the United Kingdom or any other country where such provisions are
inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS
PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of
express or implied warranties in certain transactions, therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in any
manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the
materials for this IBM product and use of those Web sites is at your own risk.
IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring
any obligation to you.
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to the names and addresses used by an actual business
enterprise is entirely coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrates programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and
distribute these sample programs in any form without payment to IBM for the purposes of developing, using,
marketing, or distributing application programs conforming to IBM's application programming interfaces.
© Copyright IBM Corp. 2005. All rights reserved.
vii
Trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States,
other countries, or both:
AIX 5L™
AIX®
CICS®
DB2 Universal Database™
DB2®
DRDA®
Eserver®
eServer™
i5/OS™
IBM®
IMS™
OS/390®
Parallel Sysplex®
RACF®
Redbooks (logo)
™
RETAIN®
RMF™
Virtualization Engine™
WebSphere®
z/OS®
zSeries®
The following terms are trademarks of other companies:
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems,
Inc. in the United States, other countries, or both.
Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States,
other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Other company, product, and service names may be trademarks or service marks of others.
viii
Enterprise Workload Manager z/OS Support
Preface
This IBM® Redpaper will help you install, tailor, and configure the EWLM managed server on
the z/OS® image. It also provides an understanding of how the transaction concept differs
between the distributed and z/OS worlds and how you can interpret the EWLM performance
data compared with traditional z/OS monitors such as RMF™ and DB2® Performance
Expert.
The team that wrote this Redpaper
This Redpaper was produced by a team of specialists from around the world working at the
International Technical Support Organization, Poughkeepsie Center.
Paola Bari is an Advisory Programmer at the International Technical Support Organization,
Poughkeepsie Center. She has 23 years of experience as a systems programmer in
OS/390®, z/OS. and Parallel Sysplex®, including several years of experience in
WebSphere® MQ and WebSphere Application Server.
G Michael Connolly is an IT consultant at the International Technical Support Organization,
Poughkeepsie Center. He has more than 30 years of IBM software development experience
in both distributed systems and the mainframe zSeries®. He holds a BA in Humanities from
Villanova University. His areas of expertise include TCP/IP Communications, UNIX® System
Services, WebSphere for z/OS, and most recently the IBM Virtualization Engine™ EWLM
product.
Franco Meroni is a consultant IT architect for IBM Italy Global Services, Integrated
Technology Services organization. He has more than 35 years of experience in mainframe
hardware and software including OS/390, z/OS, and Parallel Sysplex. He has worked with
PSSC Montpellier for z/Series benchmarks and customer environment assessments.
Thanks to the following people for their contributions to this project:
Rich Conway
International Technical Support Organization, Poughkeepsie Center
Georg Bosch
IBM Germany
Yuk Chan
IBM Poughkeepsie
Yuksel Gunal
IBM Poughkeepsie
Hugh Smith
IBM Santa Teresa
Dino Tonelli
IBM Poughkeepsie
Peter Yocom
IBM Poughkeepsie
© Copyright IBM Corp. 2005. All rights reserved.
ix
David Zhang
IBM San Jose
Become a published author
Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with
specific products or solutions while getting hands-on experience with leading-edge
technologies. You’ll team with IBM technical professionals, Business Partners, and/or
customers.
Your efforts will help increase product acceptance and customer satisfaction. As a bonus,
you’ll develop a network of contacts in IBM development labs, and increase your productivity
and marketability.
Find out more about the residency program, browse the residency index, and apply online at:
ibm.com/redbooks/residencies.html
Comments welcome
Your comments are important to us!
We want our papers to be as helpful as possible. Send us your comments about this
Redpaper or other Redbooks in one of the following ways:
򐂰 Use the online Contact us review redbook form found at:
ibm.com/redbooks
򐂰 Send your comments in an e-mail to:
redbook@us.ibm.com
򐂰 Mail your comments to:
IBM Corporation, International Technical Support Organization
Dept. HYJ Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400
x
Enterprise Workload Manager z/OS Support
1
Chapter 1.
Installation
This chapter provides information about the tasks that are required to install and customize
the EWLM code on the z/OS platform.
© Copyright IBM Corp. 2005. All rights reserved.
1
1.1 EWLM for z/OS description
The IBM Virtualization Engine is a set of technologies and systems services that enable
system administrators to access and manage resources across multiple platforms (z/OS,
AIX®, i5/OS™, Windows® and so forth). The Enterprise Workload Manager (EWLM)
systems service enables you to define business-oriented performance goals for an entire
domain of servers across multiple platforms, then provides an end-to-end view of actual
performance relative to those goals. EWLM for z/OS includes the EWLM managed server but
not the EWLM domain manager. EWLM managed servers run on each server that hosts
workloads to be monitored by EWLM. They collect and aggregate data and send the data to
the domain manager. For more information, visit the IBM eServer™ Information Center at:
http://publib.boulder.ibm.com/eserver/
The program number for EWLM for z/OS is 5655-M76. EWLM for z/OS is distributed as z/OS
priced feature at no additional cost. EWLM for z/OS consists of the following FMIDs:
򐂰 HVE1110, IBM Virtualization Engine Enterprise Workload Manager for z/OS V1.1
򐂰 HJVA140, IBM SDK for z/OS, Java™ 2 Technology Edition, V1.4
EWLM product can be ordered as CBPDO or ServerPac.
1.1.1 Preventive service planning
Before installing EWLM for z/OS, you should review the current Preventive Service Planning
(PSP) information. If you obtained EWLM for z/OS as part of a CBPDO, there is HOLDDATA
and PSP information included on the CBPDO.
Table 1-1 lists the UPGRADE and SUBSET values to be used to locate the EWLM for z/OS
PSP.
Table 1-1 PSP UPGRADE and SUBSET ID
UPGRADE
SUBSET
Description
VEV1R1
EWLM
EWLM for z/OS
JAVAOS390
HJVA140
IBM SDK for z/OS V1.4
Table 1-2 lists the Components IDs that belong to the EWLM code.
Table 1-2 FMID COMPID component name
FMID
COMPID
Component name
RETAIN®
release
HVE1110
5655M7601
EWLM, z/OS-specific
110
HVE1110
5655M7602
EWLM, common code
110
HJVA140
5648-C9800
IBM SDK for z/OS, Java 2 Technology Edition, V1.4
140
HJVA140
5648-C9801
IBM SDK for z/OS, Java 2 Technology Edition, V1.4
140
Verify that the following PTFs are installed on your z/OS 1.6 system:
򐂰 UA15188 (WLM)
򐂰 UA15189 (WLM Kanji)
򐂰 UA15456 (USS)
2
Enterprise Workload Manager z/OS Support
1.1.2 Installation requirements
The following sections identify the system requirements for installing and activating EWLM for
z/OS. In these sections, the system used to install the EWLM code is identified as the driving
system and the system where the EWLM will run is identified as the target system. Depending
on your installation policy, these can be represented by either the same system or two
different systems.
Driving system
The driving system can run in any hardware environment that supports the required software.
Table 1-3 describes the software requirements that are necessary to install the EWLM code.
Table 1-3 Driving system software requirements
Program number
Any one of the following:
Product name and minimum service level
5694-A01
z/OS V1.4 with IBM SMP/E V3R2 (5655-G44) or higher and Java 2
Technology Edition, SDK 1.3.1 (5655-D35) or higher
5655-G52
z/OS.e V1.4 with IBM SMP/E V3R2 (5655-G44) or higher and Java 2
Technology Edition, SDK 1.3.1 (5655-D35) or higher
Keep in mind that:
򐂰 z/OS UNIX System Services must be up in full function mode on your driving system.
򐂰 Before installing EWLM for z/OS, you must ensure that the target system’s HFS is
available (OMVS active and the target file systems mounted) for processing.
򐂰 An SMP/E utility entry for the binder is required. You can specify any of these program
names in the UTILITY entry:
– IEWLINK
– HEWL
– LINKEDIT
– HEWLH096
(The linkage editor that uses the names HEWLKED, HEWLF064, HEWLF440, and
IEWLF128 cannot be used.)
򐂰 The following installation jobs, as well as the SMP/E APPLY job, should be run under a
user ID with UID(0), or with a user ID with READ access to the BPX.SUPERUSER
resource in the RACF® FACILITY class:
–
AJVISMKD
–
HVEISMKD
This installation user ID must also have READ access to these FACILITY class resources:
– BPX.FILEATTR.PROGCTL
– BPX.FILEATTR.APF
– BPX.FILEATTR.SHARELIB
Chapter 1. Installation
3
Target system
The target environment can run on any hardware that supports the software listed in
Table 1-4. This table describes the software requirements for running the EWLM code.
Table 1-4 Software prerequisites
Program number
Product name and minimum service level
5694-A01
z/OS V1.6 or higher
5655-G52
z/OS.e V1.6 or higher
5655-I56
IBM SDK for z/OS, Java 2 Technology Edition, V1.4 (included in this product)
at PTF level UQ90449 or later
5694-A01
z/OS V1.6 PTF UA15188 (Workload Manager) and UA15189 (Workload
Manager Kanji)
z/OS V1.6 with PTFS UA12453, UA13606, and UQ92626 or higher
5655-G52
z/OS.e V1.6 PTF UA15188 (Workload Manager) and UA15189 (Workload
Manager Kanji)
z/OS.e V1.6 with PTFS UA12453, UA13606, and UQ92626 or higher
DASD requirements
Table 1-5 lists the total space required for each type of library.
Table 1-5 DASD space
Library type
Total space required
Target
4 tracks of 3390 for EWLM for z/OS
3 tracks of 3390 for IBM SDK for z/OS, V1.4
Distribution
475 tracks of 3390 for EWLM for z/OS
1500 tracks of 3390 for IBM SDK for z/OS, V1.4
HFS
270 tracks of 3390 for EWLM for z/OS
3600 tracks of 3390 for IBM SDK for z/OS, V1.4
The following tables contain the list of the libraries that are either updated or created during
the installation process.
Table 1-6 Target libraries requirement
Library DDNAME
Type
Number 3390 trks
Number Dir. blocks
PROCLIB
Ea
1
1
SAJVSSMP1
Ub
3
2
SAMPLIB
E
3
1
a. E represents an existing shared data set, used by this product and others. This data set is
not allocated during the installation process of this product. To determine the correct storage
needed for this data set, the storage size given here must be added to the current dimension
of the data set.
b. U represents an unique data set, allocated by this product and used only by this product.
4
Enterprise Workload Manager z/OS Support
Table 1-7 HFS paths
DDNAME
Type
Path name
SAJVJ14
Na
/usr/lpp/java/J1.4/IBM
SHVECLAS
N
/usr/lpp/VE_R1/EWLM/IBM
a. N represents a new path, created by this product.
Table 1-8 Distribution libraries requirements
Library DDNAME
Type
Number of 3390 tracks
Number of dir. blocks
AAJVHFS
U
1813
2
AAJVSMP1
U
3
2
AHVEDATA
U
5
2
AHVEHFS
U
466
3
APROCLIB
E
1
1
ASAMPLIB
E
3
1
1.1.3 General considerations
The IBM Software Development Kit for z/OS Java 2 Technology Edition, Version 1.4 (program
number 5655-I56), which is included as part of EWLM for z/OS, is a functional successor to
various Java products from IBM. If you have previously installed this product, you should take
the following considerations into account when installing EWLM for z/OS.
If you already have a copy of IBM SDK for z/OS, Java 2 Technology Edition, V1.4 at the
required maintenance level for EWLM for z/OS, you can continue to use the existing version
or install the IBM SDK for z/OS, Java 2 Technology Edition, V1.4 shipped with EWLM for z/OS
and replace your existing IBM SDK for z/OS, Java 2 Technology Edition, V1.4 with it.
We recommend that you install EWLM for z/OS into the same SMP/E zones with your z/OS or
z/OS.e (V1.6 or later) system, including SMPCSI, target, distribution data sets, except for the
HFS data sets.
During installation of EWLM for z/OS on the driving system, the target HFS file systems must
be mounted at the following mountpoints:
򐂰
򐂰
<PathPrefix>/usr/lpp/VE_R1/EWLM
<PathPrefix>/usr/lpp/java/J1.4
<PathPrefix> is an HFS path prefix such as /SERVICE. Use of a path prefix allows
maintenance to be applied to a copy of the EWLM for z/OS HFS file systems without
disturbing the production HFS file systems.
During customization and operation of EWLM for z/OS on the target system, the target HFS
file systems must be mounted at the following mountpoints:
򐂰
򐂰
A Java SDK mountpoint, normally /usr/lpp/java/J1.4
An EWLM for z/OS mountpoint, normally /usr/lpp/VE_R1/EWLM
Chapter 1. Installation
5
1.2 SMP/E installation
From an SMP/E point of view, it is recommended that you install the EWLM code into the
z/OS CSI. If you want to install EWLM for z/OS into its own SMP/E environment, you must
create and initialize its own SMPCSI and the SMP/E control data sets.
1.2.1 Installing EWLM for z/OS
In order to install the EWLM code, you have to execute all of the installation steps from a user
ID that is defined to z/OS UNIX System Services, and which has the following attributes:
򐂰 UID(0) or READ access or higher to BPX.SUPERUSER in the FACILITY class
򐂰 READ access or higher to BPX.FILEATTR.PROGCTL, BPX.FILEATTR.APF, and
BPX.FILEATTR.SHARELIB in the FACILITY class
The installation of the EWLM code follows the normal installation path through the SMP/E
RECEIVE, APPLY, and ACCEPT steps.
Table 1-9 contains the list of the sample installation jobs provided as part of the product. You
can access the sample installation jobs by performing an SMP/E RECEIVE and then copying
the jobs from the relfiles to a work data set for editing and submission.
Table 1-9 Installation jobs
Job name
Job type
Description
RELFILE
AJVALLOC
ALLOCATE
Sample job to allocate target and
distribution libraries for Java Software
Development Kit
IBM.HJVA140.F2
AJVISHFS
ALLOCATE
Sample job to allocate a new HFS data set
for Java Software Development Kit
IBM.HJVA140.F2
AJVISMKD
MKDIR
Sample job to invoke the supplied
AJVMKDIR EXEC to allocate HFS paths for
Java Software Development Kit
IBM.HJVA140.F2
AJVDDDEF
DDDEF
Sample job to define SMP/E DDDEFs for
Java Software Development Kit
IBM.HJVA140.F2
HVEALLOC
ALLOCATE
Sample job to allocate target and
distribution libraries for EWLM for z/OS
IBM.HVE1110.F2
HVEISHFS
ALLOCATE
Sample job to allocate a new HFS data set
for EWLM for z/OS
IBM.HVE1110.F2
HVEISMKD
MKDIR
Sample job to invoke the supplied
HVEMKDIR EXEC to allocate HFS paths
IBM.HVE1110.F2
HVEDDDEF
DDDEF
Sample job to define SMP/E DDDEFs for
EWLM for z/OS
IBM.HVE1110.F2
These tasks must be executed to accomplish the EWLM for z/OS code installation:
1. If you are installing the EWLM for z/OS as part of a CBPDO, run the RCVPDO job found in
the CBPDO RIMLIB data set to RECEIVE the EWLM for z/OS FMIDs as well as any
service, HOLDDATA, or preventive service planning (PSP) information included on the
CBPDO tape. This step is not required if you are installing EWLM for z/OS as part of a
service pack.
6
Enterprise Workload Manager z/OS Support
2. Edit and submit the sample jobs HVEALLOC and AJVALLOC to allocate the SMP/E target
and distribution libraries for EWLM for z/OS and the Java Software Development Kit
component.
3. If you are installing EWLM for z/OS and the Java Software Development Kit component
into new HFS data sets, customize and run sample jobs HVEISHFS and AJVISHFS to
allocate the HFS data sets.
4. Mount the newly created HFS with the MKDIR command. The MKDIR commands are
described in sample jobs HVEISHFS and AJVISHFS. Mount the new HFS data sets at:
–
–
<PathPrefix>/usr/lpp/VE_R1/EWLM
<PathPrefix>/usr/lpp/java/J1.4
<PathPrefix> is an HFS path prefix.
5. Edit and submit sample jobs HVEISMKD and AJVISMKD to create the HFS target
directories needed to install EWLM for z/OS and the Java Software Development Kit.
6. If you plan to create a new HFS for this product, you should consider updating the
BPXPRMxx PARMLIB member to mount the new HFS at IPL time.
7. Edit and submit the sample jobs HVEDDDEF and AJVDDDEF to create the DDDEF
entries for the SMP/E target and distribution libraries.
8. Edit, customize with your installation values, and submit the sample job shown in
Example 1-1 to perform an SMP/E APPLY CHECK for EWLM for z/OS.
After you have successfully run the APPLY CHECK job and solved any potential actions
indicated by the APPLY CHECK, run the job again with the CHECK operand removed.
Example 1-1 Sample APPLY job
//APPLY JOB
//STEP1 EXEC PGM=GIMSMP,REGION=M,TIME=NOLIMIT
//SMPCSI DD DSN=csiname,DISP=SHR
//SMPCNTL DD
SET BOUNDARY(targetzone) .
APPLY CHECK
FORFMID(HVE111,HJVA14) → remove FMID HJVA140 if java is already installed
SELECT(HVE111,HJVA14) → remove FMID HJVA140 if java is already installed
GROUPEXTEND(NOAPARS,NOUSERMODS)
BYPASS(HOLDSYSTEM,
HOLDUSER,HOLDCLASS(UCLREL,ERREL,HIPER)) .
/*
9. Edit, customize with your installation values, and submit the sample job shown in
Example 1-2 to perform an SMP/E ACCEPT CHECK. Before using SMP/E to load new
distribution libraries, it is recommended that you set the ACCJCLIN indicator in the
distribution zone. This will cause entries produced from JCLIN to be saved in the
distribution zone whenever a SYSMOD containing inline JCLIN is ACCEPTed.
After you have successfully run and taken any actions indicated by the ACCEPT CHECK,
remove the CHECK operand and run the job again to perform the ACCEPT.
Example 1-2 Sample ACCEPT job
//ACCEPT JOB
//STEP1 EXEC PGM=GIMSMP,REGION=M,TIME=NOLIMIT
//SMPCSI DD DSN=csiname,DISP=SHR
//SMPCNTL DD
SET BOUNDARY(dlibzone) .
ACCEPT CHECK
FORFMID(HVE111,HJVA14) → remove FMID HJVA140 if java is already installed
Chapter 1. Installation
7
SELECT(HVE111,HJVA14) → remove FMID HJVA140 if java is already installed
GROUPEXTEND(NOAPARS,NOUSERMODS)
BYPASS(HOLDSYSTEM,
HOLDUSER,HOLDCLASS(UCLREL,ERREL,HIPER)) .
/*
After you complete this step, the EWLM for z/OS is installed and ready to be customized.
Now you can customize the managed node on z/OS. Instructions are provided in “Setting up
z/OS as managed server” on page 11.
1.3 Middleware instrumentation
The following middleware running on z/OS is ARM instrumented:
򐂰 DB2 UDB for z/OS V8
򐂰 WebSphere Application Server for z/OS V6
1.3.1 DB2
The following maintenance is required to provide ARM enablement on DB2 Universal
Database™ Version 8.2. After you apply the following maintenance, DB2 is ARM-enabled by
default and no additional steps are required to enable ARM on DB2 for z/OS managed
servers.
򐂰 APAR PQ91509 EWLM correlator support to allow performance measurement using ARM
instrumentation.
򐂰 APAR PQ99707 with the connection reset optimization code for JCC clients.
򐂰 The JCC driver should be at level 2.6.16 or later.
Transactions initiated through DDF are accounted as separate transactions. DB2 transactions
initiated through DDF are accounted as separate transactions, but DB2 transactions initiated
by a local WebSphere Application Server executed in binding mode are accounted to the
WebSphere thread.
In this release, DDF cannot be hop 0 for a transaction to be measured by EWLM. It will not
create a correlator for the transactions but will only handle the correlator coming from a
distributed environment.
1.3.2 WebSphere Application Server
z/OS requires WebSphere Application Server V6.0.1. No further maintenance is required.
Refer to “Enable instrumentation for middleware” on page 17 for the required tasks to enable
the ARM instrumentation.
8
Enterprise Workload Manager z/OS Support
2
Chapter 2.
EWLM configuration
This chapter discusses:
򐂰 The ITSO environment for the EWLM management domain
򐂰 Setting up z/OS as managed sever
򐂰 Starting the managed server on z/OS
򐂰 Discussion about the workload
© Copyright IBM Corp. 2005. All rights reserved.
9
2.1 ITSO environment for the EWLM management domain
Figure 2-1 represents the environment we used at ITSO: a Web Server tier, a two-server
WebSphere cluster as the application tier, and a database tier. These four servers were
configured to participate in one EWLM management domain as managed servers and
connected to a domain manager.
EWLM1
(Windows)
Web Server
HTTP Plugin 5.1.1
EWLM2
(Windows)
WTSC69
(z/OS)
EWLM4
(AIX)
Application Servers
WAS 5.1.1
Database Server
DB2 8.2
Figure 2-1 Three-tier architecture
Our entry application was the HTTP plugin running on ewlm1, which runs on Windows Server
2003. The WebSphere cluster consisted of the ewlm2 and ewlm4 servers running
WebSphere Application Server 5.1.1, with ewlm2 running on Windows Server 2003 and
ewlm4 running maintenance level three of AIX 5L™ Version 5.2. DB2 V8.2 ran on WTSC60
z/OS 1.6. The domain manager EWLMDM1 in our environment ran on Red Hat Enterprise
Linux® AS 2.1; this was where we ran the EWLM Control Center.
Figure 2-2 on page 11 shows our management domain configuration.
10
Enterprise Workload Manager z/OS Support
Managed server
-m p 3333
Laptop
9.x.x.x
W orkload sim ulator
wtsc43
Dom ain
m anager
-m p 3333
-jp 9092
-wasPorts 20000
HTTP Plugin
Server
EW LM1
9.12.4.141
Managed server
-m p 3333
EW LM2
9.12.4.140
EW LMDM1
9.12.4.142
Managed server
-m p 3333
EW LM3
9.12.4.139
Managed server
-m p 3333
wtsc69
9.12.6.22
Figure 2-2 ITSO management domain
Note: We assume that the domain manager has already been configured and that the
Control Center application is active. Same for the managed nodes running on the Web
server and the WebSphere Application cluster.
In this chapter we focus only on the managed node running on the z/OS platform as we
describe the setup of the z/OS managed server. In this configuration we use the default
security setting, and there is no firewall present in the described configuration.
2.2 Setting up z/OS as managed server
The tasks that are required to customize the z/OS managed server are:
򐂰
򐂰
򐂰
򐂰
Configuring IBM SDK for z/OS, Java 2 Technology Edition, Version 1.4
Updating the BPXPRMxx member in PARMLIB
Configuring a managed server
Authorizing z/OS users to start the EWLM managed server
2.2.1 Configuring IBM SDK for z/OS, Java 2 Technology Edition, Version 1.4
After installing a z/OS managed server and prior to configuring and starting the managed
server, ensure that the SDK (IBM SDK for z/OS, Java 2 Technology Edition, Version 1.4) is
installed on the managed server and that the SDK path variable used by EWLM points to the
path where the SDK resides:
1. Log on to the managed server system with a user ID that has superuser authority.
Chapter 2. EWLM configuration
11
2. Go to the /usr/lpp/VE_R1/EWLM/samples directory and issue the following command to
copy the EWLM managed server environment variable into the /etc directory:
cp ewlmms_environment.conf
/etc
3. In the /etc directory, update the SDK path variable that EWLM will use (JRE_ROOT) in the
ewlmms_environment.conf file. For example:
JRE_ROOT=/usr/lpp/java/J1.4/bin
This is the default path if you installed the SDK as part of the EWLM product. If your SDK
resides on a different path, specify that directory in the JRE_ROOT definition.
Example 2-1 shows the definitions that we used on SC69.
Example 2-1 ewlmms_environment.conf on SC69
EWLM_ROOT=/usr/lpp/VE_R1/EWLM
JRE_ROOT=/usr/lpp/java/J1.4/bin
LOGFILE_ROOT=/var/adm/ewlm/ewlmms
2.2.2 Updating the BPXPRMxx member in PARMLIB
EWLM requires z/OS UNIX domain (AF_UNIX) sockets. Verify that this type of file system is
defined in the BPXPRMxx member of PARMLIB in use in your installation. You can use the
D OMVS,O command to discover which PARMLIB member is in use to define the HFS
configuration. Example 2-2 shows the definition used on SC69.
Example 2-2 Extract of BPXPRM member used on SC69
FILESYSTYPE TYPE(UDS) ENTRYPOINT(BPXTUINT)
NETWORK DOMAINNAME(AF_UNIX)
DOMAINNUMBER(1)
MAXSOCKETS(1000)
TYPE(UDS)
2.2.3 Configuring a managed server
You can configure the managed server on z/OS by using the createMS script provided in the
the <install>/VE_R1/EWLM/bin directory (where <install> maps to the directory in which the
EWLM product has been installed). The <install>/VE_R1/EWLM/bin directory should contain
the following scripts:
򐂰
򐂰
򐂰
򐂰
򐂰
changeMS.sh
createMS.sh
deleteMS.sh
displayMS.sh
startMS.sh
The following command syntax creates the managed server on z/OS:
createMS workingDir -ma address -mp port -auth [None | ServerSSL | ClientServerSSL]
-sslks path -sslpw password
In this syntax:
workingDir
12
Defines the directory in which the script creates the managed server
instance. This directory should not already exist; createMS creates it for
you. You must provide the working directory value as an absolute path.
Enterprise Workload Manager z/OS Support
Note: If you use a working directory name with lowercase characters, remember when
starting the managed server with the START command that you must preserve the case
(for example, by using quotations around the directory name).
-ma
The IP address or host name of the domain manager. This value must
match the -ma value that you specify when you configure the domain
manager. This enables the managed server to properly identify and
communicate with its domain manager.
-mp
The port that the managed server uses to communicate with the domain
manager. This value must match the -mp value that you specify when you
configure the domain manager.
-auth
The authority level used to secure communications between the domain
manager and the managed servers. Possible values are None, ServerSSL,
and ClientServerSSL.
-sslks
The path to the certificate keystore that contains the SSL keys that are
used for securing the communications between the managed servers and
the domain manager or firewall broker, if applicable, when -auth is
ServerSSL or ClientServerSSL.
-sslpw
Password used to access the keystore specified on the -sslks parameter.
For a detailed description of these fields, refer to the eServer Software Information Center:
http://publib.boulder.ibm.com/infocenter/eserver/v1r1/en_US/index.htm?info/icmain.htm
Example 2-3 shows the command we used to create the managed server on SC69. You
might create a separate HFS for the EWLM data and mount it before issuing createMS.
Example 2-3 createMS script used on SC69
cd /usr/lpp/VE_R1/EWLM/bin
./createMS.sh /ewlm/ewlmms -ma 9.12.4.142 -mp 3333 -auth None
You can check the result of the script in the var/adm/ewlm/ewlmms directory. You should find
a createMS.log file with the output of the command as shown in Example 2-4.
Example 2-4 createMS log output
*****************************************************************************
BEGIN
createMS command started at: Wed Mar 16 16:32:39 EST 2005 Platform = OS/390
STEP 1: Create EWLM working directory
EWLM working directory: /ewlm/ewlmms
STEP 2: Validate user input
STEP 3: Configure the Managed Server via WLMConfig
WLMCONFIG_PARMS: -mode ManagedServer -auth None -ma 9.12.4.142 -mp 3333
/usr/lpp/java/J1.4/bin/java -classpath
/usr/lpp/VE_R1/EWLM/classes/ewlm_updates.jar:/usr/lpp/VE_R1/EWLM/classes/ms/Managed_Server.jar:/usr/lpp/VE_
R1/EWLM/classes/db2j.jar:/usr/lpp/VE_R1/EWLM/classes/samplebroker.jar:/usr/lpp/VE_R1/EWLM/classes/dhbbroker
.jar:/usr/lpp/VE_R1/EWLM/classes/dhbcore.jar:/usr/lpp/VE_R1/EWLM/classes/jms.jar:/usr/lpp/VE_R1/EWLM/classe
s/CL3Export.jar:/usr/lpp/VE_R1/EWLM/classes/CL3Nonexport.jar: com.ibm.wlm.rte.WLMConfig /ewlm/ewlmms -mode
ManagedServer -auth None -ma 9.12.4.142 -mp 3333
0
WLMConfig command succeeded
STEP 4: Process the working directory
PROCESSING COMPLETE
Chapter 2. EWLM configuration
13
As a result of executing createMS, the working directory /ewlm/ewlmms is created with the
following subdirectories:
Diagnostics
Designed to contain dump produced by the EWLM managed node if
problems occur.
Interfaces
Contains API code.
eWLMData
Contains database data such as policy, reporting data, and properties.
This directory also includes the System.log file that contains the log data for the startup of the
managed server.
2.2.4 Authorizing z/OS users to start the EWLM managed server
In your installation, you might want to start the z/OS managed with a user ID that does not
have superuser authority but can access the necessary files and directories.
The following tasks are required to define a user that will start the managed server and be
associated with the managed server process:
1. Define a group and user that will be used to start the managed server. The managed
server process will run using the user ID defined here. Use the user ID and group name
that you specify in this step throughout rest of these instructions. You can use the following
RACF commands to create a group and add a user to the group; you must provide similar
definitions if your installation is using a different security product:
ADDGROUP groupname SUPGROUP(SYS1) OWNER(SYS1) OMVS(GID(group-identifier))
ADDUSER userid DFLTGRP(groupname) NOPASSWORD OMVS(UID(user-identifier)
PROGRAM(/bin/sh) HOME(/usr/lpp/VE_R1/EWLM))
In this syntax:
groupname
The name of the group that contains EWLM user IDs
userid
The user ID that the managed server process will run as
group-identifier
The numeric group identifier
user-identifier
The numeric user identifier
Example 2-5 is the set of commands we used on SC69 to define our EWLM user as
EWLMMS belonging to the EWLMGRP group.
Example 2-5 Defining EWLM user ID and group to the security product
ADDGROUP EWLMGRP SUPGROUP(SYS1) OWNER(SYS1) OMVS(GID(6899))
ADDUSER EWLMMS DFLTGRP(EWLMGRP) NOPASSWORD OMVS(UID(6824) PROGRAM(/BIN/SH)
LU EWLMMS
USER=EWLMMS NAME=UNKNOWN OWNER=BARI
CREATED=05.070
DEFAULT-GROUP=EWLMGRP PASSDATE=N/A
PASS-INTERVAL=N/A
ATTRIBUTES=PROTECTED
REVOKE DATE=NONE
RESUME DATE=NONE
LAST-ACCESS=05.073/09:42:06
CLASS AUTHORIZATIONS=NONE
NO-INSTALLATION-DATA
NO-MODEL-NAME
LOGON ALLOWED
(DAYS)
(TIME)
--------------------------------------------ANYDAY
ANYTIME
GROUP=EWLMGRP
AUTH=USE
CONNECT-OWNER=BARI
CONNECT-DATE=05.070
CONNECTS=
04 UACC=NONE
LAST-CONNECT=05.073/09:42:06
14
Enterprise Workload Manager z/OS Support
CONNECT ATTRIBUTES=NONE
REVOKE DATE=NONE RESUME DATE=NONE
SECURITY-LEVEL=NONE SPECIFIED
CATEGORY-AUTHORIZATION
NONE SPECIFIED
SECURITY-LABEL=NONE SPECIFIED
***
2. Grant the user ID access to call z/OS platform services. To do this, issue the following
RACF commands:
RDEFINE FACILITY MVSADMIN.EWLM.AGENT UACC(NONE)
PERMIT MVSADMIN.EWLM.AGENT CLASS(FACILITY) ID(userid) ACCESS(READ)
SETROPTS RACLIST(FACILITY) REFRESH
In this syntax, userid is the user ID that the managed server process will use, as defined
in the previous step.
Example 2-6 is the set of RACF commands we used on SC69 to enable our user ID,
EWLMMS, to use ARM services.
Example 2-6 Granting the usage of ARM services to EWLMMS
RDEFINE FACILITY MVSADMIN.EWLM.AGENT UACC(NONE)
PERMIT MVSADMIN.EWLM.AGENT CLASS(FACILITY) ID(EWLMMS) ACCESS(READ)
SETROPTS RACLIST(FACILITY) REFRESH
3. Provide the user the ability to start the EWLM managed server by issuing the following
RACF commands:
RDEFINE STARTED EWLMMS.** STDATA(USER(userid) GROUP(groupname) TRUSTED)
SETROPTS RACLIST(STARTED) REFRESH
In this syntax:
userid
The user ID, created earlier, that is associated to the EWLM
managed server process.
groupname
The group name defined in the previous step.
Example 2-7 is the set of commands we used on SC69 to associate the EWLMMS user ID
to the managed server process.
Example 2-7 Providing an RACF user ID to the EWLM managed server process
RDEFINE STARTED EWLMMS.** STDATA (USER(EWLMMS) GROUP(EWLMGRP) TRUSTED)
SETROPTS RACLIST(STARTED) REFRESH
4. Switch to a user with superuser authority to perform the following tasks.
5. Grant the group that you created in the previous step access to the EWLM directories by
issuing the following commands:
chgrp -R groupname workingDir
chgrp groupname /etc/ewlmms_environment.conf
chgrp -R groupname /var/adm/ewlm
In this syntax:
groupname
The group name representing the EWLM user ID, created in the
earlier step.
Chapter 2. EWLM configuration
15
workingDir
The EWLM working directory that was created at the time of the
createMS command.
/var/adm/ewlm
The directory where the scripts are allocating their output files.
Example 2-8 is the set of commands we use don SC69 to associate the EWLMMS user ID
to the managed server process.
Example 2-8 Commands to grant access to directories and file to EWLMMS user ID
chgrp -R EWLMGRP workingDir
chgrp groupname /etc/ewlmms_environment.conf
chgrp -R groupname /var/adm/ewlm
6. Grant the group access to the configuration file and remove access for all other users by
issuing the following commands:
chmod g+w /etc/ewlmms_environment.conf
chmod o-rx /etc/ewlmms_environment.conf
2.3 Start EWLM on the z/OS managed server
In the <install>/VE_R1/EWLM/bin directory, you can find the createMS.sh script that can be
used to start the EWLM managed server.
The recommended way to start the EWLM is to use the procedure provided by the installation
in your PROCLIB. At start time, the EWLM process will be associated to the newly created
user ID and defined in the RACF STARTED class.
Issue the following command:
START EWLMMS,MSDIR='workingDir'
In this syntax, workingDir is the working directory that is created by the createMS
configuration script.
Remember to use quotation marks if the working directory name includes lowercase
characters to preserve the case of parameters.
This is the start of the EWLM managed server on SC69 using the procedure:
START EWLMMS,MSDIR='/ewlm/ewlmms'
Example 2-9 shows checking the output log to verify that the process started successfully.
Example 2-9 Verifying a successful start by checking the output log
cd /var/adm/ewlm/ewlmms/
obrowse startMS.log
****************************************************************************
* BEGIN
*
startMS command started at: Mon Mar 14 09:42:06 EST 2005 Platform = OS/390
STEP 1: Access EWLM working directory
EWLM working directory: /ewlm/ewlmms
There is no way to bring down EWLMMS gracefully. If you need to stop the managed server,
you must cancel the started task.
16
Enterprise Workload Manager z/OS Support
Figure 2-3 shows the Managed Servers view on the Control Center after the z/OS managed
node was started on SC69. You can see that SC69 has joined the management domain.
Figure 2-3 Managed Servers view
Selecting the managed server on SC69 and requesting the properties opened Figure 2-4.
Figure 2-4 SC69 managed server properties
2.3.1 Enable instrumentation for middleware
Currently DB2 V8.2 and WebSphere V6.0.1 are the available middleware for z/OS that have
ARM enablement capability.
DB2
No further action is required to enable ARM instrumentation on DB2 through DDF.
Chapter 2. EWLM configuration
17
WebSphere Application Server for z/OS
To enable ARM instrumentation on WebSphere Application Server, enable ARM and the
Request Metrics. After you complete the following customization steps, restart the
WebSphere Application Server.
Enable PMI Request Metrics
The following steps are required to enable PMI metrics:
1. Log on to the administrative console of WebSphere Application Server V6 at
http://<hostname>:9060/admin, where 9060 is the default HTTP port for a Network
Deployment Cell.
2. Select Monitoring and Tuning → Request metrics in the administrative console
navigation tree.
a. On the Configuration tab of the Request Metrics panel, check the Enable Request
metrics box. Figure 2-5 on page 19 shows a sample of this panel.
b. Under Components to be instrumented, select ALL.
c. Choose the trace level you want to use from the Trace level list: We use Hops.
d. Under Request Metrics Destination:
i. Standard Logs is optional.
ii. Check Application Response Measurement (ARM) agent.
iii. Specify ARM4 from the Agent Type pull-down list.
iv. Type the class name in the ARM transaction factory implementation class name text
field: com.ibm.wlm.arm40SDK.transaction.Arm40TransactionFactory
3. Click Apply and OK.
4. Click Save, and Save to Master Configuration.
18
Enterprise Workload Manager z/OS Support
Figure 2-5 PMI metrics Configuration panel
Add custom properties to the JVM
The following steps are required to update the JVM custom properties:
1. Copy the arm4.jar from the default location /usr/lpp/VE_R1/EWLM/classes into a directory
of your choice. The .jar file must reside in the /ARM subdirectory of the directory you
choose. In our example, we choose /u/ewlmadm/ARM. Remember to give permission so
that the WebSphere Application Server for z/OS has read access to the directory. After
you copy the directory, log on to the WebSphere Administrative Console to complete the
steps.
2. Go to Servers → Application Servers → <server name>. In the Java and Process
Management section, select Process Definition
3. Select Servant.
4. In the Additional Properties section, expand Java Virtual Machine → Custom
Properties.
5. Add these custom properties as shown in Figure 2-6 on page 20:
– com.ibm.websphere.pmi.reqmetrics.PassCorrelatorToDB = true
– ws.ext.dirs = <user_directory>/ARM where the <user_directory> is the directory
specified in step 1 on page 19.
Chapter 2. EWLM configuration
19
Figure 2-6 JVM Custom Properties panel
6. Set the WAS SERVER ONLY server region libpath variable.
a. Expand Environments → WebSphere variables.
b. Append the WAS SERVER ONLY server region libpath variable to include
:/usr/lpp/VE_R1/EWLM/lib:/usr/lib as shown in Figure 2-7.
Figure 2-7 WebSphere variables panel
Considerations for Java2 Security
If Java2 Security is enabled: In order to use WebSphere Application Server V6 with EWLM,
you must update the WebSphere Application Server server.policy files in the
WAS6_Profile/properties directory, adding the following lines in that file (change
EWLMMS_HOME according to your EWLMMS installation path):
grant codeBase "file:/EWLMMS_HOME/classes/ARM/arm4.jar" {
permission java.security.AllPermission;
};
This avoids a Java 2 security exception for violating the Java 2 security permission.
20
Enterprise Workload Manager z/OS Support
User applications
The following task is required if you want to authorize any applications or middleware running
in your installation to use ARM services. This task is not required if your application is using
enclave support.
For an application that is instrumented using native C or Java services, the user ID the
application is running under either:
򐂰 Must have READ access to the System Authorization Facility (SAF) resource
BPX.WLMSERVER in the FACILITY class.
or
򐂰 Must be defined as a superuser. You can define the user ID as a superuser either with
UID(0) or with READ access to the SAF resource BPX.SUPERUSER in the FACILITY
class using the following commands:
RDEFINE FACILITY BPX.WLMSERVER UACC(NONE)
PERMIT BPX.WLMSERVER CLASS(FACILITY) ID(userid) ACCESS(READ)
2.3.2 Operational commands
The following commands are available to manage the ARM service on z/OS.
Display the status
You can use the following command to verify the status of ARM services on z/OS:
DISPLAY WLM,AM
Example 2-10 is the way the command was used on SC69.
Example 2-10 Display WLM,AM command sample
D WLM,AM
RESPONSE=SC69
IWM075I 15.07.05 WLM DISPLAY 500
EWLM ARM SERVICES ARE ENABLED
EWLM MANAGED SERVER JOBNAME=EWLMMS2 ASID=0049
EWLM POLICY NAME=ITSO TRADE3 AND PLANTS SERVICE POLICY
NUMBER OF REGISTERED PROCESSES=1, APPLICATIONS=1
Enabling ARM services
If EWLM ARM processing becomes disabled, you can enable ARM services by issuing this
command:
MODIFY WLM,AM=ENABLE
The only way to disable the ARM agent is though the operator command. In this case, you
can then use ENABLE to re-enable the services.
Disabling ARM services
To disable EWLM ARM services, issue this command:
MODIFY WLM,AM=DISABLE
Chapter 2. EWLM configuration
21
2.4 Running the workload
To run our scenario, we used the WebSphere workload provided by Trade3.
Trade3 workload was used on the ITSO EWLM management domain to verify and
demonstrate EWLM capabilities. Trade3 is the third generation of WebSphere end-to-end
benchmark and performance sample applications. The Trade3 benchmark models an online
stock brokerage application and has been redesigned and developed to cover the significantly
expanding WebSphere programming model. This simulates a real-world workload driving the
WebSphere implementation of J2EE 1.3, Web Services including key WebSphere
performance components and features.
You can download the distributed Trade3 application from:
http://www.ibm.com/software/webservers/appserv/benchmark3.html
This distributed version of the Trade3 application package assumes that you will install and
run Trade3 entirely on distributed platforms. However, in our environment the Trade3
application was installed on the distributed WebSphere Application Servers but was required
to connect to a backend Trade3 database installed on a DB2 running on our z/OS system.
This required downloading the Trade3 package for the z/OS WebSphere environment and
installing Trade3 within our ITSO environment, utilizing only those elements from each
package as needed.
You can download the z/OS Trade3 application from:
http://w3quickplace.lotus.com/QuickPlace/wasperf/PageLibrary85256DD600586369.nsf/h_C2A5E65F
D719B3DC85256DD600588CFC/523DE1DBA14E2F3185256E69007787C3/
The z/OS Trade 3 package documentation contains the following warning:
Note: the trade package contained in this procedure is a z/OS specific version
intended to be installed in a 'non-clustered' environment (either base on ND) on
WebSphere Application Server for z/OS V5.0x / 5.1.
Of course, our ITSO environment was a clustered configuration consisting of WebSphere
Application Servers on non-z/OS platforms. However, the only portion of the Trade3 for z/OS
package that we needed was setup procedures for a Trade 3.1 database on a z/OS platform.
2.4.1 Create z/OS DB2 database for Trade3
For our purposes the only file we needed to download from the Trade 3.1 for WebSphere
Application Server for z/OS V5.0x / 5.1 Web site was the trade3db_package.jcl.txt file. After
downloading the file to a Windows platform, we only had to complete Step 1 (installing the
trade 3.1 database) from the procedure for installing Trade 3.1 using scripts on z/OS. This
step installs the Trade 3.1 database by customizing and running the Trade 3.1 database job:
1. Upload to a dataset in the TSO environment and customize by following instructions
detailed in the prolog.
2. Submit job to create the TRADE3DB database and its associated tables and table spaces.
We customized the Sample JCL for our environment and chose to edit it and submit it as a
SPUFI job. The SPUFI version of the Sample JCL was executed using the ID mconnol, which
is the ID we will assign to the Data Source on the ewlm4 AIX platform. An additional ID, paola,
was granted access authority to the database and will be assigned to the Data Source on the
ewlm2 Windows Server 2003 platform.
22
Enterprise Workload Manager z/OS Support
2.4.2 Distribute the DB2 Universal JDBC Driver for z/OS
We distributed the db2jcc.jar(at level 2.7.33) and db2jcc_license_cisuz.jar files required to
support the DB2 Universal JDBC Driver Provider (XA) type 4 connections to both the elwm2
and ewlm4 nodes. (You must use a db2jcc.jar at level 2.6.16 or above.)
Both files were sent by FTP into the \db2 directory on the Windows Server 2003 ewlm2 node
and the /db2 directory on the AIX ewlm4 node. The locations of these .jar files will be needed
later when configuring the DB2 Universal JDBC Driver Provider on ewlm2 and ewlm4.
2.4.3 Installing the Trade3 distributed application
You must install the following features of WebSphere Application Server to configure and
deploy the Trade3 application:
򐂰 Administration
– Scripted Administration
򐂰 Embedded Messaging
򐂰 Ant and Deployment Tools
– Deploy Tool
– Ant Utilities
The distributed trade3Install.zip file was downloaded to our ewlm2 system, which also served
as our Deployment Manager. The downloaded file was unzipped and the ReadmeCluster.html
Trade3 installation/configuration instructions were followed, although with modifications.
Step 1 - Prepare for installation includes these steps:
1. Download and unzip the Trade3 install package on the ND Cell Manager host.
2. Open a shell/command window to perform the installation, and change to the t3install
directory.
3. Run the setupCmdLine script to set up your shell environment.
4. Start the WebSphere 5.0 ND Cell Manager.
Because our ITSO WebSphere Network Deployment environment was already
established there was no need to federate, so we skipped this step.
5. On each Node machine, add the Node to the Cell Manager.
We skipped this step, as our Network Deployment environment was already established.
The installation of the Trade 3.1 application into a cluster environment is performed by the
Trade3.jacl script. However, the trade3.ear file provided for use in a distributed environment
was assembled with a backend ID DB2UDBNT_V8_1. This ID will not match the z/OS DB2
that we will connect to so it will be necessary to modify the Trade3.jacl script.
You must replace the default vendor ID used for the DB2 by making the following change to
the Trade3.jacl script:
1. The following statement is located around line 162:
proc installAppOut {cluster dbtype} {
set deployvendor ""
2. Edit it to read:
proc installAppOut {cluster dbtype} {
set deployvendor "-deployejb.dbtype DB2UDB0S390_v7"
Chapter 2. EWLM configuration
23
Under Step 2 - (DB2) Create the DB2 database for Trade3 and Modify DB2 for Read
Stability, we skipped all of the steps because they apply only to a distributed database and
we utilized a DB2 installed on z/OS.
Under Step 3 - Install, configure and start Trade3 application and resources, we
performed these steps:
1. On the ND Cell Manager host, install Trade3 JDBC/JMS resources and application
interactively.
By default (pressing the Enter key), the Trade3.jacl script performs the following actions in
support of the Trade3 application in our Network Deployment configuration:
a. Create a cluster named TradeCluster consisting of two WebSphere Application Server
nodes: ewlm2 and ewlm4.
b. Create an Application Server called TradeClusterServer1 on node ewlm2 and
TradeClusterServer2 on node ewlm4. This is the sequence of steps:
•
Cell → TradeCluster
•
Nodes → ewlm2 → Cluster members → TradeClusterServer1
•
Nodes → ewlm4 → Cluster members → TradeClusterServer2
The %WAS_HOME%\bin\wsadmin -f Trade3.jacl config_install command was
issued from the directory where the setupCmdLine.bat was executed (step 2 on
page 23).
Note: The Trade3.jacl script assumes that the WebSphere cluster is
homogenous and will define a DB2 JDBC Provider (XA) at the cell level.
2. The Trad3.jacl script prompts for the name of the node containing our JMS Server as well
as the Node names of the two nodes we previously federated into our ND configuration.
No default names apply, so here are the prompts along with our replies:
Running Trade in a cluster topology requires one of the nodes to act as the JMS
Server. Please enter one of the nodes where the WebSphere JMS provider is
installed.
Enter the name of the node that contains a JMS Server:
ewlm2
Before this step, you should have run the addNode step at each installed node.
Please enter the node names you used when installing each node.
Enter a node name to add to the cluster or <enter> to end:
ewlm2
Enter a node name to add to the cluster or <enter> to end:
ewlm4
3. A prompt asks for information regarding our database configuration. The answers given to
these prompts are used to configure a DB2 JDBC Provider (XA) at the cell level as well as
a TradeDataSource definition and J2C Authentication entries. Our cluster environment
contains both an AIX and a Windows Server 2003 platform, so the DB2 JDBC Provider
must be defined at the Node level where the platform-specific format of the resource
provider classes can be uniquely entered.
Additionally, in our ITSO environment the ewlm2 node and the ewlm4 node each use a
unique user ID and password for connecting to the Trade3 database on z/OS. The “one
size fits all” approach of the Trade3.jacl script does not satisfy our database requirements.
24
Enterprise Workload Manager z/OS Support
Therefore, we just responded to all database configuration prompts with the Enter key and
accepted all of the defaults. Our manually constructed node level database configuration
(2.4.4, “Configure Trade3 JDBC driver” on page 25) will superseded the resulting cell-level
database configuration.
Note: We observed that although the Trade3.jacl script can be run multiple times, it will not
process any configuration changes if the targeted resource already exits. If you need to
execute the Trade3.jacl script to change a previously defined resource, you must first use
the WebSphere Administration console to delete the resource.
We also accepted the default of yes to the last prompt Install trade application? This will
install the trade3 application onto both the Trade3ClusterServer1 and Trade3ClusterServer2
application servers.
2.4.4 Configure Trade3 JDBC driver
When a Trade3 transaction flows from either the ewlm2 or ewlm4 node to the backend
database on z/OS, the correlator associated with this transaction must also be included as
part of the transaction flow. The support for propagating the ARM correlator from the
WebSphere Application Server hop to the z/OS DB2 hop is provided by the db2jcc.jar JCC
driver. This .jar file provides the DB2 Universal JDBC Driver (also known as the JCC driver),
which supports both JDBC type 2 and type 4 mode and is shipped as part of the z/OS DB2
Universal Database V8.2.
You have to install the db2jcc.jar file at level 2.6.16 or higher. See 2.4.2, “Distribute the DB2
Universal JDBC Driver for z/OS” on page 23 and configure a data source to use this driver.
For reference, we document the procedure we followed to create a JDBC provider:
1. Configure a JDBC provider for node ewlm2 on Windows Server 2003 platform.
a. Log on to the WebSphere Administrative Console.
b. Click Resources → JDBC Providers.
c. Select a Scope of Node, choose ewlm2, and click Apply.
d. Click New to create a new JDBC provider.
e. Select DB2 Universal JDBC Driver Provider (XA) from the pull-down list. Click OK.
f. The settings page for the new JDBC provider appears. Update the Name by prefixing
Trade3 to the default JDBC provider name. Next update the Classpath properties with
the actual path to the previously installed JCC driver .jar files (2.4.2, “Distribute the
DB2 Universal JDBC Driver for z/OS” on page 23). See property values as shown in
Table 2-1 and Figure 2-8. Leave all other property values as they are.
Table 2-1 JDBC provider properties for Trade3
Property name
Value
Name
Trade3 DB2 Universal JDBC Driver Provider (XA)
Classpath
\db2\db2jcc.jar
\db2\db2jcc_license_cisuz.jar
Chapter 2. EWLM configuration
25
Figure 2-8 Create JDBC provider
g. Click Apply.
h. Select Data Sources.
i. Click New to create a new data source within the Trade3 DB2 Universal JDBC (XA)
provider.
j. Enter properties as shown in Table 2-2 and Figure 2-9 on page 27, and click Apply.
Leave other properties as is.
Table 2-2 Data source properties for Trade3
26
Property name
Value
Name
TradeDataSource
JNDI name
jdbc/TradeDataSource
Description
Trade3 Datasource
Component-managed Authentication Alias
TradeDataSourceAuthData
Container-managed Authentication Alias
TradeDataSourceAuthData
Enterprise Workload Manager z/OS Support
Figure 2-9 Create data source
k. Select Custom Properties.
l. Change custom properties as shown in Table 2-3.
Table 2-3 TradeDataSource custom properties
Property name
Value
databaseName
DB8E
driverType
4
serverName
wtsc69.itso.ibm.com
portNumber
38042
m. Click Apply.
n. Click Save.
2. Test the data source connection.
a. Click Resources → JDBC Providers.
b. Select Scope and click Apply.
c. Select Trade3 DB2 Universal JDBC Driver Provider (XA) → Data Sources.
a. Check TradeDataSource and click Test Connection to ensure that the connection for
the datasource is successful.
Figure 2-10 Connection test message
b. Click Save.
c. Check Synchronize changes with nodes and click Save to save all your changes.
3. Restart the application server.
Chapter 2. EWLM configuration
27
Execute the following tasks on the Windows server:
1. Configure a JDBC provider for node ewlm2 on Windows Server 2003 platform.
a. Log on to the WebSphere Administrative Console.
b. Click Resources → JDBC Providers.
c. Select a Scope of Node, choose ewlm2, and click Apply.
d. Click New to create a new JDBC provider.
e. Select DB2 Universal JDBC Driver Provider (XA) from the pull-down list. Click OK.
f. The settings page for the new JDBC provider appears. Update the Name by
prepending Trade3 to the default JDBC provider name. Update the Classpath
properties with the actual path to the previously installed JCC driver .jar files. See
property values as shown in Table 2-4 and Figure 2-11. Leave all other property values
as they are and click OK.
Table 2-4 JDBC provider properties for Trade3
Property name
Value
Name
Trade3 DB2 Universal JDBC Driver Provider (XA)
Classpath
\db2\db2jcc.jar
\db2\db2jcc_license_cisuz.jar
Figure 2-11 Create JDBC provider
g. Click Apply.
h. Select Data Sources.
i. Click New to create a new data source within the JDBC provider.
28
Enterprise Workload Manager z/OS Support
j. Enter properties as shown in Table 2-5 on page 29 and Figure 2-12 on page 29. Leave
other properties as is and click Apply.
Table 2-5 Data source properties for Trade3
Property name
Value
Name
TradeDataSource
JNDI name
jdbc/TradeDataSource
Description
Trade3 Datasource
Component-managed Authentication Alias
TradeDataSourceAuthData
Container-managed Authentication Alias
TradeDataSourceAuthData
Figure 2-12 Create data source
k. Select Custom Properties.
l. Change custom properties as shown in Table 2-6.
Table 2-6 TradeDataSource custom properties
Property name
Value
databaseName
DB8E
driverType
4
serverName
wtsc69.itso.ibm.com
portNumber
38042
m. Click Apply.
n. Select J2C Authentication Data Entries.
o. Select TradeDataSourceAuthData. This TradeDataSourceAuthData was installed as
part of the default.
Chapter 2. EWLM configuration
29
2. Test the data source connection.
a. Click Resources → JDBC Providers.
b. Select Scope and click Apply.
c. Select Trade3 DB2 Universal JDBC Driver Provider (XA) → Data Sources.
d. Check TradeDataSource and click Test Connection to ensure connection for the
datasource is successful.
Figure 2-13 Connection test message
e. Click Save.
f. Check Synchronize changes with nodes and click Save to save all your changes.
3. Restart the application server.
The same tasks are now executed on the AIX server:
1. Configure a JDBC provider for node ewlm4 on AIX platform.
a. Log on to the WebSphere Administrative Console.
b. Click Resources → JDBC Providers.
c. Select a Scope of Node, choose ewlm4, and click Apply.
d. Click New to create a new JDBC provider.
e. Select DB2 Universal JDBC Driver Provider (XA) from the pull-down list. Click OK.
f. The settings page for the new JDBC provider appears. Update the Name by
prepending Trade3 to the default JDBC provider name. Next update the Classpath
properties with the actual path to the previously installed JCC driver .jar files. See
property values as shown in Table 2-7 and Figure 2-14 on page 31. Leave all other
property values as they are and click OK.
Table 2-7 JDBC provider properties for Trade3
30
Property name
Value
Name
Trade3 DB2 Universal JDBC Driver Provider (XA)
Classpath
/db2/db2jcc.jar
/db2/db2jcc_license_cisuz.jar
Enterprise Workload Manager z/OS Support
Figure 2-14 Create JDBC provider
g. Click Apply.
h. Select Data Sources.
i. Click New to create a new data source within the JDBC provider.
j. Enter properties as shown in Table 2-8 and Figure 2-15 on page 32. Leave other
properties as is and click Apply.
Table 2-8 Data source properties for Trade3
Property name
Value
Name
TradeDataSource
JNDI name
jdbc/TradeDataSource
Description
Trade3 Datasource
Component-managed Authentication Alias
TradeDataSourceAuthDataAix
Container-managed Authentication Alias
TradeDataSourceAuthDataAix
Chapter 2. EWLM configuration
31
Figure 2-15 Create data source
k. Select Custom Properties.
l. Change custom properties as shown in Table 2-9.
Table 2-9 TradeDataSource custom properties
Property name
Value
databaseName
DB8E
driverType
4
serverName
wtsc69.itso.ibm.com
portNumber
38042
m. Click Apply.
2. Test the data source connection.
a. Click Resources → JDBC Providers.
b. Select Scope and click Apply.
c. Select Trade3 DB2 Universal JDBC Driver Provider (XA) → Data Sources.
d. Check TradeDataSource and click Test Connection to ensure that the connection for
the data source is successful.
Figure 2-16 Connection test message
e. Click Save.
f. Check Synchronize changes with nodes and click Save to save all your changes.
3. Restart the application server.
32
Enterprise Workload Manager z/OS Support
2.4.5 Verifying the workload through the Control Center
At this point, the ARM instrumentation has been enabled on all of the middleware elements
and the WebSphere workload is running. If we log on to the Control Center, we can verify that
the performance data is monitored and the application topology.
In our scenario, we modify the domain policy described in the redbook IBM Enterprise
Workload Manager, SG24-6350. We use the same classification rules against the same
transaction classes in order to obtain very granular reporting but we associate the service
class to only three service classes to make the comparison with the z/OS easier.
Figure 2-17 shows the workload in this transaction class.
Figure 2-17 Transaction Class report
We select the transaction class TC_Trade3_general and the Application Topology task.
Figure 2-18 shows the application middleware layers that are part of this transactions group.
You can see that because the transactions are getting to DB2 on z/OS through DDF, they are
accounted as separate transactions running on the DB2 hop.
Figure 2-18 Application topology
Chapter 2. EWLM configuration
33
We can also verify the servers that are hosting the middleware elements by selecting the
transaction class TC_Trade3_quotes and the Server Topology task. Figure 2-19 shows the
servers that are part of this transactions group. You can see the z/OS managed node icon as
the last hop in the transaction flow.
Figure 2-19 Server topology
The next chapter explores the performance information that EWLM provides and how you can
relate them to the data provided by zWLM.
34
Enterprise Workload Manager z/OS Support
3
Chapter 3.
Monitoring and reporting
This chapter presents some background information for the coexistence of EWLM and zWLM
policies, and the concept of transaction in EWLM, zWLM, and DB2. It also discusses the
monitoring and reporting data provided by EWLM Control Center and by Resource Monitoring
Facility (RMF) and DB2 Performance Expert (DB2PE) for z/OS, highlighting and explaining
the correlations and the differences.
This chapter contains the following topics:
򐂰 EWLM and zWLM policies
򐂰 EWLM, zWLM, and DB2 transaction concept
򐂰 Monitoring the workload with EWLM
򐂰 A comparison between monitoring reports
© Copyright IBM Corp. 2005. All rights reserved.
35
3.1 EWLM and zWLM policies
EWLM and zWLM are driven by different service policies, operate independently, and have
different scopes. EWLM manages multi-tiered application environments that may run on
different hardware platforms and operating systems including z/OS and has an end-to-end
scope of control. The zWLM scope is limited to z/OS platforms. The future might see
functionality that converges these policies.
As shown in Figure 3-1, EWLM and zWLM policy structures are conceptually the same. The
main difference is related to the Work Managers and to the logical components that receive
and submit the work requests. As Work Managers, EWLM supports applications and
platforms, and zWLM supports several subsystems, including CB, CICS®, DDF, IMS™.
To associate a work request to a Service Class, a Work Manager passes some attribute
values that EWLM matches against filters and zWLM matches against subsystem work
qualifiers. This process is called classification.
EWLM and zWLM Policy
EWLM
Domain Policy
Service Policy
Workloads
Service classes
Applications
Transaction classes
Filters
Platforms
Process classes
Filters
zWLM
Service Definition
Service Policy
Workloads
Service classes
Subsystems Classification
rules
- Work qualifiers
Report classes
Application environments
Scheduling environments
Options
Figure 3-1 WLM and zWLM policies
The attribute values for work requests received by the entry application of an EWLM
management domain might be different from those passed to zWLM by JDBC type 4 if the
business transaction has to be sent to a z/OS hop to get data, especially if the classification
on the distributed platform happened in an application environment other than DB2. For this
reason it may not always be possible to have a one-to-one mapping for EWLM service
classes and zWLM service classes.
As an example refer to Figure 3-2 on page 37 where the EWLM and zWLM classification logic
is shown for application transactions whose topology uses a z/OS hop as a database server.
36
Enterprise Workload Manager z/OS Support
Distributed Application Topology - EWLM, zWLM Classification Logic
DDF work qualifiers
- accounting info
- collection name
- correlation info
- package name
- plan name
- store proc name
- userid
- .........................
Application Filters
- URI
- query string
- transaction name
- application instance
- user name
- managed domain name
- ...............................
Figure 3-2 Classification flow
The policies set up to run Trade3 workload in the ITSO environment have three service
classes specified for the distributed application in the EWLM domain policy and one service
class specified for subsystem DDF in the zWLM policy. In our configuration, the Trade3
workload uses the HTTP plugin middleware as entry point, so we specified the classification
rules only for the application IBM Webserving Plugin in the EWLM service policy.
Table 3-1 is a summary of the EWLM policy definitions:
Table 3-1 Definition for the Trade3 workload service classes
Service
class
Transaction class
Application
Filter type
Filter
Filter value
SC_Trade3_s
hopping
TC_Trade3_buy
IBM Webserving
Plugin
EWLM:URI
Equal
/trade(\*)
+ QueryString
Equal
action=buy(\*)
SC_Trade3_s
hopping
TC_Trade3_sell
IBM Webserving
Plugin
EWLM:URI
Equal
/trade(\*)
+ QueryString
Equal
action=sell(\*)
SC_Trade3_
general
TC_Trade3_account
IBM Webserving
Plugin
EWLM:URI
Equal
/trade(\*)
+ QueryString
Equal
action=account(\*)
SC_Trade3_
general
TC_Trade3_register
IBM Webserving
Plugin
EWLM:URI
Equal
/trade(\*)
+ QueryString
Equal
action=register(\*)
Chapter 3. Monitoring and reporting
37
Service
class
Transaction class
Application
Filter type
Filter
Filter value
SC_Trade3_
general
TC_Trade3_update_profile
IBM Webserving
Plugin
EWLM:URI
Equal
/trade(\*)
+ QueryString
Equal
action=update_profile(\*)
SC_Trade3_
default
TC_Trade3_home
IBM Webserving
Plugin
EWLM:URI
Equal
/trade(\*)
+ QueryString
Equal
action=home(\*)
SC_Trade3_
default
TC_Trade3_logout
IBM Webserving
Plugin
EWLM:URI
Equal
/trade(\*)
+ QueryString
Equal
action=logout(\*)
SC_Trade3_s
hopping
TC_Trade3_portfolio
IBM Webserving
Plugin
EWLM:URI
Equal
/trade(\*)
+ QueryString
Equal
action=portfolio(\*)
SC_Trade3_
general
TC_Trade3_quotes
IBM Webserving
Plugin
EWLM:URI
Equal
/trade(\*)
+ QueryString
Equal
action=quotes(\*)
SC_Trade3_s
hopping
TC_Trade3_general
IBM Webserving
Plugin
EWLM:URI
Equal
/trade(\*)
Figure 3-3 is a summary of the filters and classification rules used in the EWLM and zWLM
policies.
EWLM and zWLM Classification
Application - IBM Webserving
Plugin
Service Classes
SC_Trade3_shopping
SC_Trade3_general
SC_Trade3_defauly
Filters
EWLM:URI Equal /trade(\*)
QueryString Equal action=...
Subsystem DDF
Service Class
SCDB2STP
Plan Name DISTSERV
Figure 3-3 Classification rules
The reason why we were not able to put together a one-to-one relationship among EWLM
and zWLM service classes is that, for a JDBC type 4 application, attribute values passed to
zWLM, such as PLAN, PACKAGE, CONNECTION, and CORRELATION, are the same for
each message sent to DDF. In our scenario these are the attribute values passed to DDF:
򐂰
򐂰
򐂰
򐂰
38
PLAN = DISTSERV
PACKAGE = SYSLN300
CONNECTION = SERVER
CORRELATION = db2jccSe
Enterprise Workload Manager z/OS Support
The USERID attribute passed to zWLM changes, the reason being the association to each of
the two WebSphere Application Servers in the configuration (ewlm2, ewlm4). Because
different transaction types are executed in the WebSphere Application Server, the USERID
attribute could not be used to link transactions to different zWLM service classes.
Example 3-1 is a DB2 PE accounting report (short type) where you can see the work qualifier
for these transactions.
Example 3-1 DDF work qualifiers
LOCATION: DB8E
GROUP: N/P
MEMBER: N/P
SUBSYSTEM: DB8E
DB2 VERSION: V8
DB2 PERFORMANCE EXPERT (V2)
ACCOUNTING REPORT - SHORT
PAGE: 1-1
REQUESTED FROM:
TO:
INTERVAL FROM:
TO:
ORDER: CONNECT-PLANNAME
SCOPE: MEMBER
ALL
DATES
04/19/05
04/19/05
15:00:00.00
15:20:00.00
15:00:00.62
15:19:59.98
CONNECT
£OCCURS £ROLLBK SELECTS INSERTS UPDATES DELETES CLASS1 EL.TIME CLASS2 EL.TIME GETPAGES SYN.READ LOCK SUS
PLANNAME
£DISTRS £COMMIT FETCHES
OPENS CLOSES PREPARE CLASS1 CPUTIME CLASS2 CPUTIME BUF.UPDT TOT.PREF £LOCKOUT
--------------------------- ------- ------- ------- ------- ------- ------- -------------- -------------- -------- -------- -------SERVER
DISTSERV
17168
17168
0
17168
0.00
3.56
0.16
2.23
0.27
0.00
0.00
2.28
0.019523
0.002156
0.004103
0.001528
10.91
0.87
0.00
0.26
0.00
0
------------------------------------------------------------------------------------------------------!PROGRAM NAME
TYPE
£OCCURS SQLSTMT CL7 ELAP.TIME
CL7 CPU TIME CL8 SUSP.TIME CL8 SUSP!
!SYSLN300
PACKAGE
17168
13.07
0.002394
0.001428
0.000742
0.35!
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------!REQUESTER
METH £DDFS
TRANS £ROLLBK £COMMIT SQLRECV ROWSENT CONVI!
!9.12.4.138
DRDA
8536
0.00
0
0
10.85
1.32
0.00!
!9.12.4.140
DRDA
8632
0.00
0
0
10.91
1.33
0.00!
----------------------------------------------------------------------------------1
LOCATION:
GROUP:
MEMBER:
SUBSYSTEM:
DB2 VERSION:
DB8E
N/P
N/P
DB8E
V8
DB2 PERFORMANCE EXPERT (V2)
ACCOUNTING REPORT - SHORT
ORDER: CONNECT-PLANNAME
SCOPE: MEMBER
TOP FIELD: PROCESSING (CL.2) TIME SPENT IN DB2
---1
PAGE:
REQUESTED FROM:
TO:
INTERVAL FROM:
TO:
CONNECT
PLANNAME
-------------------------------------------------------SERVER
DISTSERV
TOP NUMBER REQUESTED:
VALUE
--------------26.238675
1-2
ALL
DATES
04/19/05
04/19/05
15:00:00.00
15:20:00.00
15:00:00.62
15:19:59.98
10
PAGE
---------1-1
ACCOUNTING REPORT COMPLETE
It is also possible to classify based on the first stored procedure called in a transaction, but
this was not our case.
Tip: If you need to differentiate DDF transactions coming from a distributed environment,
you can use one of the following programming approaches:
򐂰 Use a different datasource within WebSphere Application Server for each classification
and use a specific datasource within the application. Each datasource could have either
a different AUTHID associated with it, or a JDBC driver collection. It is possible to
classify based on either of those.
򐂰 Use DB2 client strings, which are text attributes associated with the connection that can
be used for classification of the workload. The client strings can be set as a part of the
datasource definition, or they can be set within the application so that every transaction
can have a different value. Client strings can be used for workload classification and for
end-to-end auditing. The fields are written in DB2 accounting reports. The
recommended client strings are application name and end user id. Additional
information is available in the white paper Managing Enterprise Java Applications for
DB2, G507-1457.
Chapter 3. Monitoring and reporting
39
3.2 EWLM, WLM, and DB2 transaction concept
A domain manager and managed servers are components of an EWLM management
domain. The domain manager runs on one server in the management domain, and the
managed server runs in every server that is hosting workload that should be monitored. The
managed server EWLM component collects, correlates, and aggregates three types of data:
򐂰 Transaction data flowing through ARM instrumented middleware
򐂰 Process data for non-instrumented applications (not supported in z/OS)
򐂰 System resources data
To monitor work requests in a multi-tiered heterogeneous environment, you should be able to
identify and track the performance of those requests across server boundaries. It is possible
to collect this data using versions of middleware that have been instrumented with ARM V4.0
standard. These are a set of interfaces that an application calls and are then used by EWLM
to calculate response time and status of work processed by the application.
Figure 3-4 shows a general flow of core ARM function calls used by an application that EWLM
supports for capturing transaction data.
A p p lic a tio n C o d e
I n itia liz a tio n
a r m _ r e g is te r _ a p p lic a tio n
a r m _ s ta r t _ a p p lic a t io n
a r m _ r e g is te r _ t r a n s a c tio n
T r a n s a c tio n p r o c e s s in g
a r m _ s t a r t_ tr a n s a c tio n
d o w o rk
a r m _ b lo c k _ tr a n s a c tio n
p a s s r e q u e s t t o a n o th e r s e r v e r
a r m _ u n b lo c k _ t r a n s a c t io n
c o m p le t e w o r k
a r m _ s t o p _ t r a n s a c t io n
Te r m in a tio n
a r m _ s to p _ a p p lic a t io n
a r m _ d e s tr o y _ a p p lic a t io n
Figure 3-4 API flow
This is a quick look at the function calls:
򐂰 An application issues arm_register_application to become registered with EWLM by its
application name and arm_start_application to indicate that the application is active.
Transaction registration via arm_register_transaction is similar to application registration,
and each transaction is associated to an application.
򐂰 arm_start_transaction indicates that a transaction is starting on this application or
middleware and EWLM starts response time timing.
򐂰 arm_block_transaction indicates that the transaction instance is blocked because it is
waiting for some other service to complete (for example, WebSphere waits for a DB2 call
to complete).
40
Enterprise Workload Manager z/OS Support
򐂰 arm_unblock_transaction indicates that the transaction is no longer blocked. Therefore,
the time between arm_block_transaction and arm_unblock_transaction can be
distinguished from the elapsed time between arm_start_transaction and
arm_stop_transaction.
򐂰 arm_stop_transaction signals the end of a transaction; EWLM stops timing the response
time.
򐂰 arm_stop_application indicates that the application instance is ending. Therefore, no more
transactions can be started or stopped. arm_destroy_application removes the registration.
Note: A transaction in an EWLM environment is encapsulated between an
arm_start_transaction and an arm_stop_transaction function calls issued by an
application or middleware.
3.2.1 ARM enablement on z/OS
Workload Manager on z/OS has provided workload management services to middleware
such as DB2, WebSphere, CICS, and IMS for a long time. For example, DB2 DIST uses WLM
enclave services to schedule and process received work requests. In the current
implementation, EWLM on z/OS leverages the existing WLM services and maps them to
ARM calls, so that EWLM managed server can simply pick up the performance data and send
it to the domain manager. Figure 3-5 depicts a general flow.
ARM and Enclave
DB2 DDF - DB2DIST address space
Connect to W LM (IW N CO NN )
arm _register_application
EW LM M S address space
EW LM _connect
arm _start_application
arm _register_transaction
Policy m anagem ent
Collect data
Send data to Control
Center
EW LM _disconnect
Transactions
E
N
C
L
A
V
E
T
R
A
N
A
R
M
T
R
A
N
S
Create enclave (IW MECREA)
Enclave workrequest start and
stop calls ( IW M ESTR T, IW M ESTO P )
arm _start_transaction
arm _stop_transaction
arm _start_transaction
arm _stop_transaction
arm _start_transaction
arm _stop_transaction
Enclave delete (IW M EDELE)
D isconnect from W LM
arm _stop_application
arm _destry_application
Figure 3-5 ARM and enclave
When EWLM address space is started it connects to WLM. Its main functions are:
򐂰 Manage the EWLM policy received from the domain manager.
򐂰 Collect the performance data and send it to the domain manager.
When DB2 is started it connects to WLM and specifies that it will participate in EWLM. DB2
DIST address space is then registered to ARM on z/OS as an EWLM application participant.
Chapter 3. Monitoring and reporting
41
ARM data is captured only if a correlator is sent in from an ARM enabled distributed
application such as WebSphere; otherwise, DB2 continues processing distributed requests
as it usually does.
When DB2 receives the correlator from a distributed application, DB2 DDF creates an
enclave and starts a work request. Attribute values such as PLAN, PACKAGE, and
CONNECTION are passed with the correlator to zWLM to classify the work request and
associate it to a service class.
When an enclave is created by DDF, an arm_start_transaction is implicitly invoked to mark
the beginning of a transaction, then an arm_stop_transaction gets called at the end of
processing a database request message. A pair of start and stop transaction is associated
with each database request message from a distributed application or DRDA® requester until
a commit is received and the enclave is then deleted.
In other words, in an EWLM environment, a DB2 transaction no longer has the same meaning
as one enclave transaction, or a unit-of-work between two DB2 commits. In our case, a
transaction becomes an ARM work request, and its elapse time is simply the duration of
receiving and processing a database request message to DB2 from the DRDA requestor.
Figure 3-6 illustrates where in monitor reports you can find the number of ARM and enclave
transactions.
O c c u rre n c e s (D B 2 P E )
D B A T s (D B 2 P E )
C o m m its (D B 2 P E )
T ra n s a c tio n E n d e d (R M F )
R e p o rte d a s
E n c la v e
c re a te
(IW M E C R E A )
E n c la v e
d e le te
(IW M E D E L E )
S R M tra n s a c tio n
a rm w o rk
re q u e s t
(a rm tra n )
R e p o rte d a s
a rm tra n
a rm tra n
a rm tra n
a rm tra n
Legenda:
a rm _ s ta rt_ tra n s a c tio n
a rm _ s to p _ tra n s a c tio n
A p p lic a tio n & S e rv e r To p o lo g y Ta b le v ie w (E W L M )
D is trib u te d A c tiv ity M e s s a g e s (D B 2 P E )
D R D A M e s s a g e s (D B 2 P E )
Figure 3-6 ARM and enclave transaction relationships
EWLM Control Center reports the number of transactions (ARM work requests) in application
and server topology tableview. See Figure 3-23 on page 57.
The number of ARM transactions (ARM work requests) are reported as Messages in the
Distributed Activity and DRDA REMOTE LOCSDB2 sections of the Performance Expert
accounting and statistics reports.
An enclave transaction corresponds to an SRM transaction and its number is reported as
occurrences, DBATS, and commits by DB2 Performance Expert, and Transaction Ended by
RMF in the Workload Activity report in the Service class and Report class.
42
Enterprise Workload Manager z/OS Support
3.3 Monitoring the workload with EWLM and z/OS
The database server configuration consists of a z/OS 1.6 LPAR with two logical CPs on an
IBM zSeries 990 2084-318. The LPAR is a sysplex image in which a DB2 data sharing group
is active. Only the DB2 member DB8E, active in our partition, is used as database server by
the distributed application.
Because the DB2 workload during our test runs was light, we decided to load the image with
some batch jobs with a CPU-intensive profile to generate some contention. (See Figure 3-15
on page 50.) During our tests there was no contention with the other logical partitions defined
in the processor.
Figure 3-7 shows the service class goals for the z/OS workloads that were run during our
tests.
zWLM - Workloads Service Classes
DB2 server A.S. (DBM1, MSTR,
IRLM), DB2DIST
SYSSTC
Subsystem DDF
SCDB2STP
1. Avg r/t 20 msec, importance 2,
duration 100 (around 2 mips)
2. Vel 50%, importance 3
Subsystem JES
DISC
1. Discretionary
Figure 3-7 DDF service class on z/OS
We classify the DB2, MSTR, DBM1, DIST, and IRLM address space in the SYSSTC service
class, although in a production environment we suggest assigning the DB2, MSTR, DBM1,
and DIST address spaces to a managed service class with an IMP of 1, velocity of 50 or 60,
and IRLM to the SYSSTC service class.
The RMF recording interval is set to one minute. To collect DB2 monitor data, accounting and
statistic traces have been started as shown in Example 3-2.
Example 3-2 DB2 trace activation
-DB8E DISPLAY TRACE
RESPONSE=SC69
DSNW127I -DB8E CURRENT
TNO TYPE
CLASS
01 STAT
01,03,04,05,
01
06
02 ACCTG 01
03 ACCTG 01,02,03,07,
03
08
TRACE ACTIVITY IS DEST QUAL
SMF NO
SMF
SMF
NO
NO
Chapter 3. Monitoring and reporting
43
Note: To be able to relate the EWLM Control Center and z/OS RMF and DB2 monitors
information, we confirmed that the platform clocks were synchronized at one minute
boundary.
To analyze the workload behavior we use a top-down approach; that is, we start from the
EWLM Control Center information and then we drill down to the z/OS monitors data. Because
z/OS does not support EWLM process classes, the scope of our analysis is limited to the
service and transaction classes, and managed servers.
You should already be familiar with the EWLM Control Center and you should know that, to
exploit the top-down approach logic, it provides high-level views to address a problem and
detail views to obtain more in-depth statistics.
In summary the selectable high-level views from the Control Center Monitor panel are:
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
Exceptions
Workloads
Service classes
Transaction classes
Process classes (not supported by z/OS)
Managed servers
From each of the high-level views, you can get selectable reports by choosing a pull-down
option. These are the low-level views:
򐂰 Detailed monitor reports about service, transaction, process classes, and managed servers
򐂰 Topology reports for the service and transaction classes with additional detailed statistics
򐂰 Real-time performance monitors for:
– Goal achievement
– Transaction rate and count
– Processor utilization
EWLM domain manager maintains statistics data for one-hour intervals with a wrap-around
logic. The EWLM Control Center displays the aggregated statistics for the interval starting
from the last minute sampled with the duration length selected through the preference interval
duration pull-down option.
The first step in controlling the behavior of our EWLM managed domain is to go through the
high-level views. Some fields in the views are described in this Redpaper, and additional
information is available in the help panels of the EWLM Control Center, in the IBM Redbook
IBM Enterprise Workload Manager, SG24-6350, and in the Redpaper Enterprise Workload
Manager - Interpreting Control Center Performance Reports, REDP-3963.
We start from the Exceptions view that reports the service classes that are not meeting their
performance objectives. A service class is listed in the exception report if its performance
index (PI) is greater than 1. If no service classes are listed in this report, all of the service
classes are meeting or exceeding their goals, as happened during our test.
As an example of the Exceptions view, see Figure 3-8 on page 45. Service class
SC_Trade3_default, importance medium, average response time goal of 100 milliseconds,
had a PI of 1.24 because the average response time in the last 30 minutes has been
124 milliseconds.
44
Enterprise Workload Manager z/OS Support
Figure 3-8 Exception view
We now go through high-level views to summarize the views objectives and depict our
management domain environment.
򐂰 Workloads: Identifies the workloads that are defined in the active domain policy. If a
service class appears in the Exceptions report, you can use the Workloads view to
determine which workload is affected by the service class that is not meeting its goal.
Figure 3-9 shows the Workloads view of the domain policy deployed in our environment.
Figure 3-9 Workloads view
Chapter 3. Monitoring and reporting
45
򐂰 Service Classes: Specifies all of the service classes that are defined in the active domain
policy. In Figure 3-10 you can view performance statistics for each service class defined in
our domain policy. You can see that the workload Trade3 uses only the first three service
classes because we have the PI value along with the performance and achieved goal. The
service class SC_internetExplorer reports a PI because an Internet Explorer
administrative session is opened for the server ewlm2, which runs on a Windows platform.
Figure 3-10 Service Classes view
򐂰 Transaction Classes: Specifies all of the transaction classes for each application defined in
the active domain policy. In Figure 3-11 you can view the performance statistics for each
transaction class along with the entry application and the associated service class. You
can see the 10 service classes used by workload Trade3.
Figure 3-11 Transaction Classes view
46
Enterprise Workload Manager z/OS Support
򐂰 Managed Servers: Identifies all of the managed servers in the EWLM domain. In
Figure 3-12 you can view the status of the four servers of our configuration, the processor
utilization percentage over the specified interval, that the operating system is running, and
the release level.
Figure 3-12 Managed Servers view
When you access a high-level view, you can select a low-level view in the Select Action field.
The low-level views provide more detailed statistics as:
򐂰 Details reports: Specifies transaction and performance details for a specific report.
򐂰 Topologies: Another way of drilling down to see detailed data for a monitored service or
transaction class in the EWLM management domain.
򐂰 Real-time performance monitors: Provides graphical views that EWLM updates
automatically. The specific available monitor views are:
– Goal achievement monitor
– Processor utilization monitor
– Transaction count and rate monitor
In Managed Servers view, Figure 3-12, you can see the managed server
wtsc69.itso.ibm.com, which runs the operating system z/OS Release 1.6 and has 89.0%
average processor utilization over the specified one-hour interval.
To see more detailed information, we select the low-level view Managed server details in the
Select Action field. Figure 3-13 on page 48 shows the report.
Chapter 3. Monitoring and reporting
47
Figure 3-13 Managed Server details view
The view has two sections: Statistics and Service classes processed in the server.
Statistics: This section specifies performance data that is specific to this managed server.
򐂰 Managed server name: Unique name by which a server is known on a network. The name
consists of the host name (wtcs69) and domain name (itso.ibm.com).
򐂰 Average processor utilization %: Average processor utilization (88.1%) of the managed
server that includes all work requests that the server is processing, not just the EWLM
monitored work.
Important: The reported average processor utilization percentage includes only the
captured time that corresponds to the APPL% of workload activity RMF report for
policy.
򐂰 Real memory: Total amount of memory of the LPAR hosting the server.
򐂰 Number of logical processors: Number of logical CPs of the LPAR.
򐂰 Page fault rate: Number of page faults per second managed by the operating system. A
high number means that the memory is not sufficient to sustain all of the workloads the
server is running, not just the EWLM monitored work.
48
Enterprise Workload Manager z/OS Support
򐂰 Application (Group): Application instances that support EWLM monitored work. In our
domain the application environment name is DDF associated to DB2 member DB8E.
Service classes processed in server: This section reports all service classes associated with
the managed server.
򐂰 Service class name: Name of the EWLM domain policy service class associated to a
transaction by the Entry application. This is not the zWLM service class used to manage
the DDF workload in z/OS.
Important: The following using and delay percentages are derived from zWLM sampling
status data for the zWLM service classes managing the EWLM monitored workload.
Status samples categories are: (processor using), (processor, i/o, storage, others delays),
and idle. The percentages are derived from using and delay status samples with the
exclusion of the idle status samples.
Because there may not be a one-to-one relationship between an EWLM service or
transaction class and a zWLM service class, the using and delay sample data provided by
zWLM are apportioned to EWLM service or transaction classes with a weighted average
based on EWLM service classes transaction rate and active time. Transaction active time
is explained within the fields description on page 55.
򐂰 Processor utilization %: This value indicates the percentage of the number of samples
when the EWLM monitored workload was found using the processor. It should be read as
Processor using %.
Important: This information should not be interpreted as the percentage of the average
server processor utilization used by the service class.
򐂰 Processor delay %: This value indicates the percentage of the number of samples when
the EWLM monitored workload was found delayed because the processor was busy with
other workloads.
򐂰 Storage delay %: This value indicates the percentage of the number of samples when the
EWLM monitored workload was found delayed because of page faults.
򐂰 I/O delay %: This value indicates the percentage of the number of samples when the
EWLM monitored workload was found delayed waiting for an I/O operation to complete.
A processor utilization real-time performance monitor view is selectable from the pull-down
list. The view shows the processor utilization with one-minute granularity, and it covers a
one-hour interval with a scrollable cursor. Figure 3-14 on page 50 shows the graph.
Chapter 3. Monitoring and reporting
49
Figure 3-14 Processor utilization monitor
For comparison, Figure 3-15 shows CPU utilization reported by RMF Monitor I for a time
frame that includes the EWLM Control Center Processor utilization view. There is a little
difference because the EWLM Control Center does not report the uncaptured time (the top
layer in Figure 3-15). You may also notice that the main LPAR load is batch.
Figure 3-15 RMF Mon I processor utilization
50
Enterprise Workload Manager z/OS Support
Server wtsc69.itso.ibm.com hosts three service classes monitored by EWLM. To drill farther
down the behavior of EWLM monitor workload, from the Service Class high-level view we
select SC_Trade3_general for a Service class detail view as shown in Figure 3-16.
Figure 3-16 Service class details view
The view has three sections: Attributes, Statistics, and a Response Time Distribution Chart.
򐂰 Attribute: This section includes the following items:
–
–
–
–
A description of the service class
The workload to which the service class belongs
The goal type, the goal, and the importance
The transaction classes that are associated with this service class
򐂰 Statistics: This section specifies performance data that is specific to this service class.
– Performance: A value that specifies the actual performance, response time, of the
transaction processed by the service class in the interval. A transaction starts and
terminates in the outer hop 0 at the middleware level, and the response time covers the
end-to-end elapsed time.
– Performance index: A value that indicates how well the service class is meeting the
goal. If the PI is less than or equal to 1, the goal is being met; if it is greater than 1, the
goal is not met.
– Transactions: Completed successfully: The number of transactions associated with this
service class that successfully completed processing at hop 0 level. A transaction rate
real-time monitor view is selectable from the pull-down list. The view shows the
transaction rate with one-minute granularity, and it covers a one-hour interval with a
Chapter 3. Monitoring and reporting
51
scrollable cursor. Figure 3-18 on page 53 shows an example for the
SC_Trade3_general service class.
– The number of transactions that Failed, Stopped, Ended in unknown state, and the
Total. Failed or stopped transactions can be caused by middleware or operating system
problems or user errors. The hop where the error may have been occurred is not
indicated.
򐂰 Response Time Distribution Chart: For service classes specifying Average or Percentile
response time, a graph is presented for the selected interval length. Figure 3-17 shows
service class SC_Trade3_general Response Time Distribution Chart covering a one-hour
interval.
SC_Trade3_general service class
Figure 3-17 Response Time real-time monitor
52
Enterprise Workload Manager z/OS Support
Figure 3-18 Transaction rate real-time monitor
Because the transactions that are associated with a service class are considered end-to-end
transactions, we now select from the Service Classes high-level view pull-down option the
topology views to see the applications and servers that are processing the service class.
Topology provides a graphical view of the service and transaction classes in the EWLM
management domain. From the topology view, you can determine which servers are
processing work for each application.
Application topology is a starting point for understanding the transaction flow as well as for
problem determination. The view is a visual of the transaction as it traverses through the
middleware applications and the servers where the middleware resides. Figure 3-19 shows
the Application topology for service class SC_Trade3_general.
DDF
application
Figure 3-19 Application topology view
Chapter 3. Monitoring and reporting
53
Server topology is similar to the Application topology view and shows all servers that the
transaction traversed. Clicking on the server boxes, you can see the middleware applications
resident on the server that the transaction traversed. Figure 3-20 on page 54 shows the
server topology for service class SC_Trade3_general.
z/OS server
Figure 3-20 Server topology view
From either topology view, we select the tableview icon for detailed statistics about the
amount of resources that each application or server is absorbing along with more granular
data such as the average response times, delays percentages, number transactions that
completed successfully or unsuccessfully, and many other granular statistics.
Depending on which topology view you display, you can see some differences in the tableview
data.
򐂰 Application topology: The data related to service or transaction classes is for the
middleware application environments. If an application environment spans more than one
server, the tableview will have more than one instance row and the application
environment row aggregates the data for all instances (scope application).
򐂰 Server topology: The data related to service or transaction classes is for the middleware
applications active on the specific server (server scope).
Figure 3-21 shows the rightmost part of the tableview window from the application topology
view for service class SC_Trade3_general. We selected the application topology because we
are interested in data for the z/OS platform.
54
Enterprise Workload Manager z/OS Support
Figure 3-21 Tableview data for SC_Trade3_general service class
The table fields for the z/OS platform have the following meanings:
򐂰 Hop: Indicates the application or server where a transaction is processed. The database
server on z/OS is hop 2.
򐂰 Name: Name of the server, application, and instance to which the transaction moved.
򐂰 Type: Describes whether the Name field is server (instrumented operating system),
application environment (instrumented middleware), or instance (DB2 instance). The
application environment DDF has one instance (DB8E).
򐂰 Platform: Specifies the operating system on which the server is running (z/OS).
򐂰 Average response time: The field indicates the average amount of time in seconds that the
transactions classified to the service class spent at this middleware hop plus any time
spent in downstream hops. The average response time is the elapsed time from the
arm_start_transaction and the arm_stop_transaction API calls. In our sample, the average
response time is the same as the active time because this is the last hop. It includes the
queue time within DDF and it is 1 millisecond.
򐂰 Average active time: The field indicates in seconds how long the server took to process
the request. Active time does not include the time the application indicates that the request
is blocked because it is waiting for a subtransaction sent to another server (hop) to
complete.
In our example the active and response time are the same because DDF middleware is
the last hop. See Figure 3-22 on page 56 for response and active times concepts. Our
example shows a one-to-one relationship among hops, though this is not always the case.
Attention: In Figure 3-21, hop 1 average response time is 9 - 11 milliseconds and
average active time is 6 - 7 milliseconds; consequently the average block time is 3 - 4
milliseconds or the waiting time for hop 2 to complete the request.
Hop 2 average response and active time is 1 millisecond with no queue time, so the
2 - 3 milliseconds not accounted for might be network time.
Chapter 3. Monitoring and reporting
55
arm _start_transaction
arm _stop_transaction
arm _blockt_transaction
arm _unblockt_transaction
response tim e
hop 0
active tim e
active tim e
arm _start_transaction
arm _stop_transaction
hop 1
arm _blockt_transaction
arm _unblockt_transaction
response tim e
active tim e
hop 2
active tim e
arm _start_transaction
arm _stop_transaction
response tim e
active tim e
Figure 3-22 Transaction response and active time flow
Important: The following processor time, using, and delay percentages are derived from
zWLM data for the zWLM service classes managing the EWLM monitored workload.
Status samples categories are: (processor using), (processor, i/o, storage, others delays),
and idle. The percentages are derived from using and delay status samples with the
exclusion of the idle status samples.
Because there may not be a one-to-one relation between an EWLM service or transaction
class and a zWLM service class, the processor time, using, and delay sample data
provided by zWLM are apportioned to EWLM service or transaction classes with a
weighted average based on EWLM service classes transaction rate and active time.
Processor time (seconds): This field shows the actual CPU time used over the interval (in our
example the interval is one hour). In Figure 3-21 on page 55, 01:20.764565 should be read as
1 minute, 20 seconds, 764 milliseconds.
򐂰 Processor delay%: This field shows the percentage of time the application was not able to
obtain processor resource when requested over the interval.
򐂰 Processor using%: This field shows the percentage of time the application was able to
obtain processor resource when requested over the interval.
The Figure 3-23 on page 57 shows the leftmost part of the Tableview window from the server
topology view for service class SC_Trade3_general.
56
Enterprise Workload Manager z/OS Support
Figure 3-23 Tableview window for SC_Trade3_general service class
The table fields for the z/OS platform have the following meanings:
򐂰 Successful transactions: This field specifies the number of transactions associated to the
service class that successfully terminate its processing in hop 2, the database server on
z/OS. A transaction is delimited by issuing arm_start_transaction, arm_stop_transaction at
the middleware level, in our case DDF.
Note: This transaction count is for hop 2 middleware; it should not expect a one-to-one
relationship with the hop 0 transaction count in the first row of the tableview (137582) or
in the transaction count of the Service Class detail view. See Figure 3-16 on page 51.
򐂰 I/O%. This field shows the percentage of time the application was waiting for an I/O to
complete.
򐂰 Page delay%: This field shows the percentage of time the application was delayed
because of a page fault: central storage not available.
򐂰 Idle%: This field shows the percentage of time the application threads were in idle state.
This status is not used to evaluate the processor using and delay percentages.
򐂰 Other%: This field shows the percentage of time the application was found in a non-idle
state that could not be accounted to page, processor, or I/O delay.
򐂰 Hard page faults%: This field shows the number of page faults solved by doing a physical
I/O.
򐂰 Average queue time (seconds): This is the time spent in the middleware preceding the
actual processing of the transaction arm_start_transaction. Our example shows
13 milliseconds; this is a known problem where the value is currently reported incorrectly
and it is addressed by the next release.
򐂰 Standard deviation (seconds): This field specifies a numerical value that indicates how
much the transactions deviate from average response time.
There are other fields such as Stopped, Failed transactions, Topology uncertainty count, and
Reporting gap count, which are not addresses in this Redpaper.
Chapter 3. Monitoring and reporting
57
3.4 A comparison between monitoring reports
Now we correlate EWLM Control Center monitoring data with data provided by RMF and DB2
PE monitors for the application environment DDF.
Example 3-3 contains the control statements used to obtain the RMF and DB2 PE report.
Example 3-3 RMF and DB2 PE control statements
--- RMF-------------------------------------------------------------------SUMMARY(INT,TOT)
RTOD(1500,1520)
DINTV(0020)
REPORTS(CPU)
SYSRPTS(WLMGL(POLICY,WGROUP,SCPER,RCLASS,SYSNAM))
------------------------------------------------------------------------------ DB2 PE Accounting long ------------------------------------------------GLOBAL
TIMEZONE(+4)
INCLUDE(SUBSYSTEMID(DB8E)
CONNECT(SERVER)
)
FROM(,15:00) TO(,15:20)
ACCOUNTING
REPORT
LAYOUT(LONG)
EXEC
------------------------------------------------------------------------------ DB2 PE Statistics long ------------------------------------------------GLOBAL
TIMEZONE(+4)
INCLUDE(SUBSYSTEMID(DB8E)
CONNECT(SERVER)
)
FROM(,15:00) TO(,15:20)
STATISTICS
REPORT
LAYOUT(LONG)
EXEC
We have already seen the overall processor utilization percentage correlates as shown in
Figure 3-14 on page 50 and Figure 3-15 on page 50. We look now at processor utilization
data and transaction count and response time for the DDF workload.
3.4.1 EWLM Control Center reports
The Trade3 workload uses three EWLM service classes: the processor utilization and
transaction data, transaction response, and queue time can be found in the tableview either
for Application or Server topology views.
򐂰 Service Class SC_Trade3_default Application topology tableview (Figure 3-24 on page 59)
58
Enterprise Workload Manager z/OS Support
Figure 3-24 Application topology for SC_Trade3_default service class - right
Figure 3-25 Application topology for SC_Trade3_default service class - left
򐂰 Service Class SC_Trade3_general Application topology tableview
Data is shown in Figure 3-21 on page 55 and in Figure 3-23 on page 57.
򐂰 Service Class SC_Trade3_shopping Application topology tableview (Figure 3-26 on
page 60)
Chapter 3. Monitoring and reporting
59
Figure 3-26 Application topology for SC_Trade3_shopping service class - right
Figure 3-27 Application topology for SC_Trade3_shopping service class - left
The Control Center aggregation interval for the tableviews is one hour starting at 2:45 pm
April 19th. We normalize for DDF application environment (hop 2) the processor time seconds
and the successful transactions count to a second to be able to relate to RMF monitors data.
We did not experience stopped or failed transactions during our test run.
򐂰 Processor time seconds:
– SC_Trade3_default 5.32 (5.7% of total)
– SC_Trade3_general 80.77 (87% of total)
– SC_Trade3_shopping 6.74 (7.3% of total)
– Total 92.83 seconds that divided by 3600 seconds equals 2.58% utilization of one CP.
򐂰 Successful transactions:
– SC_Trade3_default 8735 (6% of total)
– SC_Trade3_general 130984 (86.7% of total)
– SC_Trade3_shopping 11167 (7.3% of total)
– Total 150886 transactions that divided by 3600 seconds equals 41.91 transactions per
second.
60
Enterprise Workload Manager z/OS Support
򐂰 Transaction response time and queue time (the queue time is currently reported
incorrectly and will be addressed in the next release. The value should be read as queue
time / transaction count; in our case we obtain a negligible value):
– SC_Trade3_default r/t = 1 millisecond, queue time = 1 millisecond
– SC_Trade3_general r/t = 1 millisecond, queue time = 13 millisecond
– SC_Trade3_shopping r/t = 1 millisecond, queue time = 0
Note: Those are hop 2 transactions delimited by issuing an arm_start_transaction,
arm_stop_transaction at the middleware level (DDF).
3.4.2 RMF postprocessor workload activity reports
From the RMF postprocessor, we extrapolate reports for zWLM workload DB2RES and for
service class SCDB2STP that have been set up for DDF workload. In Figure 3-7 on page 43
you can see that the report duration interval is 20 minutes, which matches the EWLM Control
Center interval.
Figure 3-28 depicts the CPU utilization behavior for DDF workload during our run. You can
see that the utilization ranges between 2.5 and 3% of one CP (APPL %).
Figure 3-28 DDF workload CPU utilization
Chapter 3. Monitoring and reporting
61
Example 3-4, Example 3-5, and Example 3-6 on page 63 show excerpts from the RMF
reports.
Example 3-4 RMF workload report
z/OS V1R6
SYSPLEX WTSCPLX1
START 04/19/2005-15.00.00 INTERVAL 000.19.59
RPT VERSION V1R5 RMF
END
04/19/2005-15.20.00
MODE = GOAL
============================================================================================================ WORKLOAD
REPORT BY: POLICY=SPSTPC
TRANSACTIONS
AVG
0.26
MPL
0.26
ENDED
17170
END/S
14.31
£SWAPS
0
EXCTD
0
AVG ENC 0.26
REM ENC 0.00
MS ENC
0.00
WORKLOAD=DB2RES
DB2 stored procedures
TRANS.-TIME HHH.MM.SS.TTT
ACTUAL
17
EXECUTION
17
QUEUED
0
R/S AFFINITY
0
INELIGIBLE
0
CONVERSION
0
STD DEV
28
--DASD I/O-SSCHRT
0.0
RESP
0.5
CONN
0.2
DISC
0.2
Q+PEND
0.1
IOSQ
0.0
---SERVICE---IOC
0
CPU
668803
MSO
0
SRB
0
TOT
668803
/SEC
557
ABSRPTN 2183
TRX SERV 2183
--SERVICE TIMES-TCB
32.2
SRB
0.0
RCT
0.0
IIT
0.0
HST
0.0
IFA
N/A
APPL% CP
2.7
APPL% IFACP
0.0
APPL% IFA
N/A
PAGE-IN RATES
SINGLE
0.0
BLOCK
0.0
SHARED
0.0
HSP
0.0
HSP MISS
0.0
EXP SNGL
0.0
EXP BLK
0.0
EXP SHR
0.0
----STORAGE---AVG
0.00
TOTAL
0.00
CENTRAL
0.00
EXPAND
0.00
SHARED
0.00
You can see that the workload and service class reports are for enclaves because the IOC
and MSO service values are zero: DDF exploits enclaves.
The APPL% reported is 2.7% of one CP that becomes 2.81% after normalizing with the
capture ratio of 96% that we measured during our run. The RMF value is close to the
aggregated processor utilization of the analyzed EWLM service classes (2.58%). See
Figure 3-27 on page 60.
The number of SRM (enclave) transactions executed per second is 17170/1200 = 14.3, with
an average execution time of 17 milliseconds, no queue time, and a standard deviation of
28 milliseconds. On average, within one enclave transaction, we have 41.9/14.3 = 2.93 hop 2
(arm_start / arm_stop) transactions.
For comparison we use the data for SC_Trade3_general service class because it accounts for
87% of total workload. The transaction queue time—that is, the time spent in the middleware
(hop 1) preceding the first arm_start_transaction in hop 2—is probably a negligible value.
(From the Application topology tableview, the value of 13 milliseconds average per transaction
is incorrect.) Furthermore we have on average in hop 2 three ARM transactions per enclave
that last 1 millisecond each.
Using these numbers, the enclave response time using EWLM Control Center number is
(0 + 1) * 3 = 3 milliseconds; RMF reports 17 milliseconds.
Example 3-5 RMF DDF transaction service class workload report - period 1
z/OS V1R6
SYSPLEX WTSCPLX1
START 04/19/2005-15.00.00 INTERVAL 000.19.59
RPT VERSION V1R5 RMF
END
04/19/2005-15.20.00
REPORT BY: POLICY=SPSTPC
TRANSACTIONS
AVG
0.25
MPL
0.25
ENDED
17103
END/S
14.25
£SWAPS
0
EXCTD
0
AVG ENC 0.25
REM ENC 0.00
MS ENC
0.00
WORKLOAD=DB2RES
TRANS.-TIME HHH.MM.SS.TTT
ACTUAL
17
EXECUTION
17
QUEUED
0
R/S AFFINITY
0
INELIGIBLE
0
CONVERSION
0
STD DEV
28
SERVICE CLASS=SCDB2STP
CRITICAL
=NONE
--DASD I/O-SSCHRT
0.0
RESP
0.5
CONN
0.2
DISC
0.2
Q+PEND
0.1
IOSQ
0.0
GOAL: RESPONSE TIME 000.00.00.020 AVG
62
Enterprise Workload Manager z/OS Support
---SERVICE---IOC
0
CPU
667285
MSO
0
SRB
0
TOT
667285
/SEC
556
ABSRPTN 2183
TRX SERV 2183
RESOURCE GROUP=*NONE
--SERVICE TIMES-TCB
32.2
SRB
0.0
RCT
0.0
IIT
0.0
HST
0.0
IFA
N/A
APPL% CP
2.7
APPL% IFACP
0.0
APPL% IFA
N/A
MODE = GOAL
PERIOD=1 IMPORTANCE=2
PAGE-IN RATES
SINGLE
0.0
BLOCK
0.0
SHARED
0.0
HSP
0.0
HSP MISS
0.0
EXP SNGL
0.0
EXP BLK
0.0
EXP SHR
0.0
----STORAGE---AVG
0.00
TOTAL
0.00
CENTRAL
0.00
EXPAND
0.00
SHARED
0.00
SYSTEM
RESPONSE TIME EX
PERF AVG
HHH.MM.SS.TTT VEL% INDX ADRSP
SC69
000.00.00.017 87.2 0.9
<
<=
<=
<=
<=
<=
<=
<=
<=
<=
<=
<=
<=
>
--- USING% --- ---------- EXECUTION DELAYS % --------- ---DLY%-- -CRYPTO%- ---CNT%-%
CPU IFA I/O TOT CPU
UNKN IDLE USG DLY USG DLY QUIE
0.2
7.4
N/A
0.0
1.1
1.1
91.1
0.0
0.0
0.0
0.0
0.0
0.0
----------RESPONSE TIME DISTRIBUTION-----------NUMBER OF TRANSACTIONS--------PERCENT------- 0
10
20
30
40
50
60
70
80
90
100
CUM TOTAL
IN BUCKET
CUM TOTAL
IN BUCKET !....!....!....!....!....!....!....!....!....!....!
5285
5285
30.9
30.9 >>>>>>>>>>>>>>>>
6954
1669
40.7
9.8 >>>>>>
8770
1816
51.3
10.6 >>>>>>
10441
1671
61.0
9.8 >>>>>>
11951
1510
69.9
8.8 >>>>>
13198
1247
77.2
7.3 >>>>
13681
483
80.0
2.8 >>
14446
765
84.5
4.5 >>>
14992
546
87.7
3.2 >>
15329
337
89.6
2.0 >>
15609
280
91.3
1.6 >>
16317
708
95.4
4.1 >>>
16878
561
98.7
3.3 >>
17103
225
100
1.3 >
----TIME---HH.MM.SS.TTT
00.00.00.010
00.00.00.012
00.00.00.014
00.00.00.016
00.00.00.018
00.00.00.020
00.00.00.022
00.00.00.024
00.00.00.026
00.00.00.028
00.00.00.030
00.00.00.040
00.00.00.080
00.00.00.080
Example 3-6 RMF DDF workload report - period 2
REPORT BY: POLICY=SPSTPC
TRANSACTIONS
AVG
0.00
MPL
0.00
ENDED
67
END/S
0.06
£SWAPS
0
EXCTD
0
AVG ENC 0.00
REM ENC 0.00
MS ENC
0.00
WORKLOAD=DB2RES
TRANS.-TIME HHH.MM.SS.TTT
ACTUAL
65
EXECUTION
65
QUEUED
0
R/S AFFINITY
0
INELIGIBLE
0
CONVERSION
0
STD DEV
60
SC69
--DASD I/O-SSCHRT
0.0
RESP
0.3
CONN
0.3
DISC
0.0
Q+PEND
0.0
IOSQ
0.0
RESOURCE GROUP=*NONE
---SERVICE---IOC
0
CPU
1518
MSO
0
SRB
0
TOT
1518
/SEC
1
ABSRPTN 2150
TRX SERV 2150
GOAL: EXECUTION VELOCITY 50.0%
SYSTEM
SERVICE CLASS=SCDB2STP
CRITICAL
=NONE
VELOCITY MIGRATION:
RESPONSE TIME EX
PERF AVG
VEL% INDX ADRSP
--N/A--
0.0 0.0
0.0
I/O MGMT
0.0%
--SERVICE TIMES-TCB
0.1
SRB
0.0
RCT
0.0
IIT
0.0
HST
0.0
IFA
N/A
APPL% CP
0.0
APPL% IFACP
0.0
APPL% IFA
N/A
INIT MGMT
PERIOD=2 IMPORTANCE=3
PAGE-IN RATES
SINGLE
0.0
BLOCK
0.0
SHARED
0.0
HSP
0.0
HSP MISS
0.0
EXP SNGL
0.0
EXP BLK
0.0
EXP SHR
0.0
----STORAGE---AVG
0.00
TOTAL
0.00
CENTRAL
0.00
EXPAND
0.00
SHARED
0.00
0.0%
--- USING% --- ---------- EXECUTION DELAYS % --------- ---DLY%-- -CRYPTO%- ---CNT%-%
CPU IFA I/O TOT
UNKN IDLE USG DLY USG DLY QUIE
0.0
N/A
0.0
0.0
100
0.0
0.0
0.0
0.0
0.0
0.0
The reports for the two periods of zWLM service class SCDB2STP confirm the CPU
utilization and the number of enclaves. In addition they show the response time distribution
and the low number of transactions (0.06 per second) completed in period 2. Period 1
duration is 100 SU, roughly equivalent to 2 MIPS.
zWLM sample state percentages for CPU (there are no storage or I/O delays) are:
򐂰 CPU using = 7.4 %
򐂰 CPU delay = 1.1 %
If we relate those two values, it turns out that CPU using samples are 87% and CPU delay
states are 13%. Those percentage values correlate with what you can see in the Managed
server details view for the z/OS platform in Figure 3-13 on page 48.
This information can be used to adjust the zWLM service class goals if necessary.
3.4.3 DB2 PE accounting and statistics reports
Now we look at the DB2 PE accounting and statistics reports and cross-check with the EWLM
Control Center and RMF reports. The interval duration is 15 minutes.
We refer to Example 3-7, which shows a DB2 PE accounting report excerpt. The interval and
the duration correlates with EWLM and RMF data.
Chapter 3. Monitoring and reporting
63
Example 3-7 DB2 PE accounting report extract
1
LOCATION: DB8E
GROUP: N/P
MEMBER: N/P
SUBSYSTEM: DB8E
DB2 VERSION: V8
PRIMAUTH: MCONNOL PLANNAME: DISTSERV
DB2 PERFORMANCE EXPERT (V2)
ACCOUNTING REPORT - LONG
ORDER: PRIMAUTH-PLANNAME
SCOPE: MEMBER
ELAPSED TIME DISTRIBUTION
---------------------------------------------------------------APPL
|==========================================> 84%
DB2
|====> 8%
SUSP
|====> 9%
AVERAGE
-----------ELAPSED TIME
NONNESTED
STORED PROC
UDF
TRIGGER
APPL(CL.1) DB2 (CL.2)
---------- ---------0.023883
0.003889
0.023883
0.003889
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
PAGE:
REQUESTED FROM:
TO:
INTERVAL FROM:
TO:
IFI (CL.5)
---------N/P
N/A
N/A
N/A
N/A
CPU TIME
AGENT
NONNESTED
STORED PRC
UDF
TRIGGER
PAR.TASKS
0.002302
0.002302
0.002302
0.000000
0.000000
0.000000
0.000000
0.001720
0.001720
0.001720
0.000000
0.000000
0.000000
0.000000
N/P
N/A
N/P
N/A
N/A
N/A
N/A
SUSPEND TIME
AGENT
PAR.TASKS
STORED PROC
UDF
0.000000
N/A
N/A
0.000000
0.000000
0.002033
0.002033
0.000000
N/A
N/A
N/A
N/A
N/A
N/A
N/A
NOT ACCOUNT.
DB2 ENT/EXIT
EN/EX-STPROC
EN/EX-UDF
DCAPT.DESCR.
LOG EXTRACT.
N/A
N/A
N/A
N/A
N/A
N/A
0.000135
27.15
0.00
0.00
N/A
N/A
N/A
N/A
N/A
N/A
N/P
N/P
1-1
ALL
DATES
06/23/05
06/23/05
10:24:00.00
10:40:00.00
10:24:02.99
10:39:58.32
CLASS 2 TIME DISTRIBUTION
---------------------------------------------------------------CPU
|======================> 44%
NOTACC |=> 3%
SUSP
|==========================> 52%
CLASS 3 SUSPENSIONS
-------------------LOCK/LATCH(DB2+IRLM)
SYNCHRON. I/O
DATABASE I/O
LOG WRITE I/O
OTHER READ I/O
OTHER WRTE I/O
SER.TASK SWTCH
UPDATE COMMIT
OPEN/CLOSE
SYSLGRNG REC
EXT/DEL/DEF
OTHER SERVICE
ARC.LOG(QUIES)
ARC.LOG READ
DRAIN LOCK
CLAIM RELEASE
PAGE LATCH
NOTIFY MSGS
GLOBAL CONTENTION
COMMIT PH1 WRITE I/O
ASYNCH CF REQUESTS
TOTAL CLASS 3
AVERAGE TIME
-----------0.000001
0.000455
0.000000
0.000455
0.000000
0.000000
0.001578
0.000000
0.000000
0.000000
0.000000
0.001578
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0.002033
AV.EVENT
-------0.00
0.22
0.00
0.22
0.00
0.00
1.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.23
HIGHLIGHTS
-------------------------#OCCURRENCES
:
1964
#ALLIEDS
:
0
#ALLIEDS DISTRIB:
0
#DBATS
:
1964
#DBATS DISTRIB. :
0
#NO PROGRAM DATA:
0
#NORMAL TERMINAT:
1964
#ABNORMAL TERMIN:
0
#CP/X PARALLEL. :
0
#IO PARALLELISM :
0
#INCREMENT. BIND:
0
#COMMITS
:
1964
#ROLLBACKS
:
0
#SVPT REQUESTS :
0
#SVPT RELEASE
:
0
#SVPT ROLLBACK :
0
MAX SQL CASC LVL:
0
UPDATE/COMMIT
:
0.30
SYNCH I/O AVG. : 0.002044
The number of occurrences and commits per second are 1964/900 = 2.18 and it should
match the RMF number of enclaves per second (report not shown).
򐂰 Class 2 CPU utilization is 2.18 * 0.002302 = 0.0050, or 0.05% of one CP.
򐂰 Class 1 average elapsed time per commit is 23.8 milliseconds and it should correspond
with RMF average transaction (enclave) response time.
Example 3-8 shows the distributed activity from the DB2 PE accounting report.
Example 3-8 DB1 PE Distributed Activity report
1
LOCATION:
GROUP:
MEMBER:
SUBSYSTEM:
DB2 VERSION:
DB8E
N/P
N/P
DB8E
V8
DB2 PERFORMANCE EXPERT (V2)
ACCOUNTING REPORT - LONG
ORDER: PRIMAUTH-PLANNAME
SCOPE: MEMBER
PAGE:
REQUESTED FROM:
TO:
INTERVAL FROM:
TO:
1-4
ALL
DATES
06/23/05
06/23/05
10:24:00.00
10:40:00.00
10:24:02.99
10:39:58.32
---- DISTRIBUTED ACTIVITY ----------------------------------------------------------------------------------------------------REQUESTER
: 9.12.4.138
TRANSACTIONS RECV. :
0.00
MESSAGES SENT
:
5.89
MSG.IN BUFFER:
7.88
PRODUCT ID
: JCC
#COMMIT(1) RECEIVED:
0
MESSAGES RECEIVED:
5.89
ROWS SENT
:
1.07
METHOD
: DRDA PROTOCOL
#ROLLBK(1) RECEIVED:
0
BYTES SENT
:
2147.51
BLOCKS SENT :
5.89
CONV.INITIATED
:
0.00
SQL RECEIVED
:
7.37
BYTES RECEIVED
:
2086.46
#DDF ACCESSES:
966
#COMMIT(2) RECEIVED:
0
#COMMIT(2) RES.SENT:
0
#PREPARE RECEIVED:
0
#FORGET SENT :
0
#BCKOUT(2) RECEIVED:
0
#BACKOUT(2)RES.SENT:
0
#LAST AGENT RECV.:
966
#COMMIT(2) PERFORM.:
966
#BACKOUT(2)PERFORM.:
0
#THREADS INDOUBT :
0
REQUESTER
: 9.12.4.140
PRODUCT ID
: JCC
METHOD
: DRDA PROTOCOL
CONV.INITIATED
:
0.00
#COMMIT(2) RECEIVED:
0
#BCKOUT(2) RECEIVED:
0
#COMMIT(2) PERFORM.:
998
64
TRANSACTIONS RECV. :
#COMMIT(1) RECEIVED:
#ROLLBK(1) RECEIVED:
SQL RECEIVED
:
#COMMIT(2) RES.SENT:
#BACKOUT(2)RES.SENT:
#BACKOUT(2)PERFORM.:
Enterprise Workload Manager z/OS Support
0.00
0
0
7.77
0
0
0
MESSAGES SENT
:
MESSAGES RECEIVED:
BYTES SENT
:
BYTES RECEIVED
:
#PREPARE RECEIVED:
#LAST AGENT RECV.:
#THREADS INDOUBT :
6.36
6.36
2132.75
2151.28
0
998
0
MSG.IN BUFFER:
ROWS SENT
:
BLOCKS SENT :
#DDF ACCESSES:
#FORGET SENT :
7.76
1.22
6.54
998
0
The number of commits are split between the two application servers and they match the
number of DDF accesses.
DDF middleware is enabled for ARM instrumentation, and a pair of arm_start_transaction,
arm_stop_transaction is associated with each database request message from a distributed
application—in our environment the WebSphere Application Servers (hop1). You can see that
the number of Messages Received per commit is 5.89 from one server and 6.36 from the
other.
Total number of messages received per second is ((966 * 5.89)+(998 * 6.36)) / 900 = 13.37,
which is very close to the 12.66 successful transactions per second for DDF middleware
(hop 2) given by EWLM Control Center as shown in Figure 3-29 on page 66, Figure 3-30 on
page 66 and Figure 3-31 on page 66.
DB2 PE statics report also has a section related to DRDA. Example 3-9 shows the report
excerpt.
Example 3-9 DB2 PE statistics report
1
LOCATION:
GROUP:
MEMBER:
SUBSYSTEM:
DB2 VERSION:
DB8E
N/P
N/P
DB8E
V8
DB2 PERFORMANCE EXPERT (V2)
STATISTICS REPORT - LONG
SCOPE: MEMBER
PAGE:
REQUESTED FROM:
TO:
INTERVAL FROM:
TO:
1-15
ALL
DATES
06/23/05
06/23/05
10:24:00.00
10:40:00.00
10:24:31.33
10:39:43.61
---- HIGHLIGHTS ---------------------------------------------------------------------------------------------------INTERVAL START : 06/23/05 10:24:31.33
SAMPLING START: 06/23/05 10:24:31.33
TOTAL THREADS
:
0.00
INTERVAL END
: 06/23/05 10:39:43.61
SAMPLING END : 06/23/05 10:39:43.61
TOTAL COMMITS
: 1890.00
INTERVAL ELAPSED:
15:12.283772
OUTAGE ELAPSED:
0.000000
DATA SHARING MEMBER:
N/A
DRDA REMOTE LOCS
--------------------------TRANSACTIONS
CONVERSATIONS
CONVERSATIONS QUEUED
SQL STATEMENTS
SINGLE PHASE COMMITS
SINGLE PHASE ROLLBACKS
ROWS
MESSAGES
BYTES
BLOCKS
MESSAGES IN BUFFER
CONT->LIM.BLOCK FETCH SWTCH
STATEMENTS BOUND AT SERVER
SENT
-------0.00
0.00
0.00
0.00
0.00
0.00
2160.00
11544.00
4020.8K
11682.00
14705.00
0.00
0.00
RECEIVED
-------0.00
0.00
PREPARE REQUEST
LAST AGENT REQUEST
TWO PHASE COMMIT REQUEST
TWO PHASE BACKOUT REQUEST
FORGET RESPONSES
COMMIT RESPONSES
BACKOUT RESPONSES
THREAD INDOUBT-REM.L.COORD.
COMMITS DONE-REM.LOC.COORD.
BACKOUTS DONE-REM.L.COORD.
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1890.00
0.00
0.00
1890.00
0.00
0.00
0.00
0.00
0.00
14249.00
0.00
0.00
0.00
11544.00
3995.6K
0.00
The number of commits and messages received per second match the accounting report:
򐂰 Commits: 1890 / 900 = 2.1
򐂰 Messages received = 12.82
Figure 3-29 on page 66, Figure 3-30 on page 66, and Figure 3-31 on page 66 show the
EWLM Application topology tableview for the measured interval. The transaction count field
represents the number of ARM transactions executed in the DB2 middleware and it
corresponds to the distributed messages reported in the DB2 PE reports.
If you sum the transactions spread across the three service classes serving the trade
application, you get 11397 (12.66 transaction/second), which is very close to the DB2 report.
Chapter 3. Monitoring and reporting
65
Figure 3-29 DB2 transaction count for SC SC_Trade3_default
Figure 3-30 DB2 transaction count for SC SC_Trade3_general
Figure 3-31 DB2 transaction count for SC SC_Trade3_shopping
66
Enterprise Workload Manager z/OS Support
3.4.4 Summary
From our studies on our EWLM management domain, we offer some fundamental tips for
implementing EWLM in your installation:
򐂰 Carefully plan your EWLM policy versus the zWLM if you want to relate performance data
between the z/OS and the distributed environment.
򐂰 When you evaluate data either from EWLM or from the z/OS monitors, keep in mind the
different concept of “transaction” as illustrated in Figure 3-32 on page 67.
EWLM
messages
DDF
commit/enclave
z/OS
Figure 3-32 Transaction concept
Chapter 3. Monitoring and reporting
67
3.5 Performance considerations
Figure 3-33 depicts the key new required functional components in an example of a simple
two-system EWLM management domain. One OS image houses the EWLM domain
manager, and two other OS images each house an EWLM managed server. This Redpaper
focuses on the z/OS image on the right side of the diagram. Within an EWLM managed OS
image, there are essentially three functional areas that consume system resources (CPU,
Memory, I/O) to accommodate EWLM:
򐂰 The EWLM managed server process (started via S EWLMMS command). This is a Java
application that is common across all platforms. Its primary responsibility is to collect and
aggregate transaction (ARM instrumented), process, and system resource data.
Additionally, it is responsible for communicating this aggregated data to the EWLM domain
manager on an interval basis, asynchronous to transactions themselves.
򐂰 The EWLM platform and ARM services. This is the operating system extension support
that provides the ARM implementation interface for ARM instrumented middleware and a
platform interface for process and system resource statistics. Data is passed to the EWLM
managed server process asynchronous to the transactions themselves.
򐂰 The DB2 DDF ARM calls and associated processing. This is essentially the
instrumentation hooks placed into DB2 DDF to provide transaction statistics (for example
Response Time, Queue time, an so forth) on a DRDA TCP/IP message-level basis.
z/OS
Management Server Host
EWLM managed server for
z/OS
FMID HVE1110
EWLM domain manager
Domain Policy
and Topology
Asynchronous to
Transaction
JNI Layer
EWLM Platform & ARM services
EWLM Control Center
Other platform
Control
Server Operating
System
style sheets
Websphere Express
as servlet engine
EWLM managed
server
EWLM Plat/ARM services
Server Operating
System
Application
Instrumentation
Interfaces
GUI
Application
Interfaces
ARM
(Enclaves)
Instrumentation
ARM (C, Java)
Call ARM…..
DB2
User log on via
WEB Browser
Client
Transaction
Flow
WebSphere
Application
Server
Synchronous to
Transaction
JCC Driver
Figure 3-33 EWLM components flow
Instrumenting transactions in a distributed environment typically requires capturing statistics
on a greater number of the smaller subtransactions that make up the total end-user
transaction. For example, as mentioned earlier, to accurately account for time spent in DB2
DDF versus time in the calling application, the DB2 DDF instrumentation is implemented on a
DRDA TCP/IP message-level boundary, as opposed to a DB2 Commit boundary, the current
instrumentation scope for z/OS zWLM.
68
Enterprise Workload Manager z/OS Support
While setting up and running the Trade3 workload, we made the following general resource
observations:
Monitoring transaction response time from the WebSphere Studio Workload Simulator, we
noticed a negligible increase when running with EWLM. The Average Transaction response
time went from 12 milliseconds to 13 milliseconds. This data is from a simpler 2-hop
configuration than the 3-hop ITSO scenarios shown earlier.
We noticed an increase in the average system CPU utilization when running TRADE3 with
EWLM, versus without EWLM (EWLMMS not started). Based on a discussion with the EWLM
development team, here are some thoughts to consider with regard to processor costs.
The relative processor performance cost of a z/OS DB2 DDF workload exploiting EWLM is
primarily driven by the following factors:
򐂰 The number of DRDA TCP/IP messages per DB2 Commit.
DDF instrumentation for EWLM is provided at a TCP/IP message level, hence
instrumentation overhead for a Commit is driven by the number of messages.
򐂰 The base processing cost (size) of a DRDA TCP/IP message in the Commit scope.
The per-message processor consumption overhead of the DDF instrumentation for EWLM
is essentially the same for all messages, hence a message with a greater DB2 cost (for
example, path length) will, proportionally, experience a lower processor overhead than a
message with lesser DB2 cost.
Therefore, processor consumption overhead due to DDF instrumentation for EWLM is
minimized, perhaps negligible, for workloads that have small numbers of DRDA messages
with a greater base DB2 processing. Conversely, and as expected, DDF workloads with
comparatively smaller base DB2 processing per message and with a larger number of
messages can potentially result in some notice of additional processor cost.
Chapter 3. Monitoring and reporting
69
70
Enterprise Workload Manager z/OS Support
Related publications
The publications listed in this section are considered particularly suitable for a more detailed
discussion of the topics covered in this Redpaper.
IBM Redbooks
For information on ordering these publications, see “How to get IBM Redbooks” on page 71.
Note that some of the documents referenced here may be available in softcopy only.
򐂰 IBM Enterprise Workload Manager, SG24-6350
򐂰 Enterprise Workload Manager - Interpreting Control Center Performance Reports,
REDP-3963
Other publications
These publications are also relevant as further information sources:
򐂰 IBM Virtualization Engine Enterprise Workload Manager for z/OS V1.1 License
Information, GC53-1120-00
򐂰 IBM SMP/E for z/OS and OS/390 User's Guide, SA22-7773
򐂰 IBM SMP/E for z/OS and OS/390 Commands, SA22-7771
򐂰 IBM SMP/E for z/OS and OS/390 Reference, SA22-7772
򐂰 IBM SMP/E for z/OS and OS/390 Messages, Codes, and Diagnosis, GA22-7770
򐂰 Managing Enterprise Java Applications for DB2, G507-1457-00
Online resources
These Web sites and URLs are also relevant as further information sources:
򐂰 IBM Eserver Software Information Center
http://publib.boulder.ibm.com/infocenter/eserver/v1r1/en_US/index.htm?info/icmain.htm
򐂰 Washington System Center Technical note: "Implementing an EWLM managed node on
z/OS"
http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD102355
How to get IBM Redbooks
You can search for, view, or download Redbooks, Redpapers, Hints and Tips, draft
publications and Additional materials, as well as order hardcopy Redbooks or CD-ROMs, at
this Web site:
ibm.com/redbooks
© Copyright IBM Corp. 2005. All rights reserved.
71
Help from IBM
IBM Support and downloads
ibm.com/support
IBM Global Services
ibm.com/services
72
Enterprise Workload Manager z/OS Support
Back cover
®
Enterprise Workload
Manager z/OS Support
Redpaper
Installing the EWLM
managed server on
z/OS
Customizing the
EWLM management
domain
Evaluating
performance data
This IBM Redpaper will help you install, tailor, and configure the
EWLM managed server on the z/OS image. It also provides an
understanding of how the transaction concept differs between
the distributed and z/OS worlds and how you can interpret the
EWLM performance data compared with traditional z/OS
monitors such as RMF and DB2 Performance Expert.
INTERNATIONAL
TECHNICAL
SUPPORT
ORGANIZATION
BUILDING TECHNICAL
INFORMATION BASED ON
PRACTICAL EXPERIENCE
IBM Redbooks are developed
by the IBM International
Technical Support
Organization. Experts from
IBM, Customers and Partners
from around the world create
timely technical information
based on realistic scenarios.
Specific recommendations
are provided to help you
implement IT solutions more
effectively in your
environment.
For more information:
ibm.com/redbooks