Front cover
Problem Determination
Methodology for
WebSphere on z/OS
Analyzing a problem, finding a cause
and solution
Exchanging information with IBM
Support
Resources available for
problem solving
Rica Weller
Robyn Nostalgi
ibm.com/redbooks
Redpaper
International Technical Support Organization
Problem Determination Methodology for WebSphere
on z/OS
March 2006
Note: Before using this information and the product it supports, read the information in “Notices” on page v.
First Edition (March 2006)
This edition applies to Version 6, Release 0, Modification 1 of WebSphere Application Server for z/OS
(program number 5655-N01).
© Copyright International Business Machines Corporation 2006. All rights reserved.
Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.
Contents
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .v
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
The team that wrote this Redpaper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
Chapter 1. Problem determination methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.1 What problem determination is . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 What PD/PSI is not . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 Problem determination approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3.1 Problem determination flowchart. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3.2 Problem determination process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3.3 WebSphere for z/OS specific consideration for PD . . . . . . . . . . . . . . . . . . . . . . . .
1.3.4 The importance of a test environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4 The skills needed for WebSphere for z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4.1 System skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4.2 Skills for deploying and running an application . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1
2
2
2
2
3
6
7
7
8
8
Chapter 2. Contacting IBM: Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1 Communicating with IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 The IBM WebSphere support structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Before contacting IBM Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.1 Define the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.2 Determine if this situation has already been reported. . . . . . . . . . . . . . . . . . . . . .
2.3.3 Mustgather: Gather background information. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.4 Determine the business impact. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4 How your call is handled by IBM Software Support . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.1 Opening a Problem Management Record (PMR) . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.2 Investigating a PMR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.3 How technical questions are handled by IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5 Exchanging data with IBM via FTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5.1 Copy the Job log into a z/OS data set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5.2 Compress data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5.3 FTP instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5.4 Naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.6 IBM contacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
12
12
13
13
13
14
16
16
16
17
18
18
19
19
20
21
21
Chapter 3. Information sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.1 WebSphere for z/OS and its support pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Techdocs: Whitepapers, Hints and Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Redbooks and draft publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Sources of information for developers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5 Other helpful Web sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.6 Educational information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23
24
29
29
30
31
34
Appendix A. Messages and codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
A.1 WebSphere for z/OS message codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
© Copyright IBM Corp. 2006. All rights reserved.
iii
A.1.1 Specific Java component messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.1.2 Minor codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.1.3 Abends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 System and component message table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
40
41
41
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
iv
Problem Determination Methodology for WebSphere on z/OS
Notices
This information was developed for products and services offered in the U.S.A.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not give you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.
The following paragraph does not apply to the United Kingdom or any other country where such provisions are
inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS
PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of
express or implied warranties in certain transactions, therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in any
manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the
materials for this IBM product and use of those Web sites is at your own risk.
IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring
any obligation to you.
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to the names and addresses used by an actual business
enterprise is entirely coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrates programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and
distribute these sample programs in any form without payment to IBM for the purposes of developing, using,
marketing, or distributing application programs conforming to IBM's application programming interfaces.
© Copyright IBM Corp. 2006. All rights reserved.
v
Trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States,
other countries, or both:
Eserver®
alphaWorks®
eServer™
z/OS®
zSeries®
DB2®
IBM®
IBMLink™
Language Environment®
MVS™
Parallel Sysplex®
Redbooks™
Redbooks (logo)
RACF®
RETAIN®
S/390®
Tivoli®
WebSphere®
™
The following terms are trademarks of other companies:
EJB, Java, JDK, JVM, J2EE, Sun, Sun Java, SNM, and all Java-based trademarks are trademarks of Sun Microsystems,
Inc. in the United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States,
other countries, or both.
Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep,
Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States
and other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
vi
Problem Determination Methodology for WebSphere on z/OS
Preface
This IBM® Redpaper describes the general problem determination methodology and how it
applies to a WebSphere Application Server for z/OS environment.
We review the skills necessary to perform problem diagnosis for WebSphere® for z/OS® and
provide information to assist you when communicating with IBM support.
We also provide you with additional information and resources all aimed at helping you find
solutions to your IT systems’ problems.
The team that wrote this Redpaper
This Redpaper was produced by a team of specialists from around the world working for a
limited period, a residency, at the International Technical Support Organization,
Poughkeepsie Center.
Rica Weller is a Project Manager at the International Technical
Support Organization (ITSO), working out of New Zealand and the
U.S.A. She was a Systems Engineer for S/390® for two years and as a
Senior Consultant for IBM WebSphere Business Integration on z/OS in
the Competence Center with IBM Germany for three years. She also
taught classes, presented at several conferences, and co-authored
several Redbooks™ about WebSphere on z/OS and zScholar Basics.
Rica holds a degree in Business Administration from TU Dresden,
Germany, and a master’s degree in Business from Massey University,
New Zealand.
Robyn Nostalgi is a IT Software Support Specialist working in the
IBM Support Center in Sydney, Australia, and she has been in this role
for over 10 years. She has specialized in support customers running
WebSphere Application Server on z/OS, and also worked in the
zSeries® Software Support team doing defect and non-defect support
for all z/OS operating system related software components.
Thanks to all authors of the Redbooks WebSphere for z/OS V5 Problem Determination,
SG24-6880, and WebSphere Application Server V6 Problem Determination for Distributed
Platforms, SG24-6798, and the support team of the IBM International Technical Support
Organization, Poughkeepsie Center, for their contributions to this project.
© Copyright IBM Corp. 2006. All rights reserved.
vii
Become a published author
Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with
specific products or solutions, while getting hands-on experience with leading-edge
technologies. You'll team with IBM technical professionals, Business Partners and/or
customers.
Your efforts will help increase product acceptance and customer satisfaction. As a bonus,
you'll develop a network of contacts in IBM development labs, and increase your productivity
and marketability.
Find out more about the residency program, browse the residency index, and apply online at:
ibm.com/redbooks/residencies.html
Comments welcome
Your comments are important to us!
We want our papers to be as helpful as possible. Send us your comments about this
Redpaper or other Redbooks in one of the following ways:
򐂰 Use the online Contact us review redbook form found at:
ibm.com/redbooks
򐂰 Send your comments in an email to:
redbook@us.ibm.com
򐂰 Mail your comments to:
IBM Corporation, International Technical Support Organization
Dept. HYJ Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400
viii
Problem Determination Methodology for WebSphere on z/OS
1
Chapter 1.
Problem determination
methodology
In this chapter we discuss a general approach to problem determination methodology and
how it applies to a WebSphere Application Server for z/OS environment. We explain how to
analyze a problem and what steps can be taken to find its cause and solution. We discuss the
skills needed for WebSphere for z/OS problem determination.
Problem determination (PD) is not a process unique to the software industry. A doctor for
example uses a problem determination process when presented with a sick patient. They
identify initial symptoms and ask their patient questions to gain a better understanding of the
problem. They order tests, analyze results and sometimes consult with a specialist. The steps
we take during the PD process likewise help us define and identify the problem and find a
solution.
© Copyright IBM Corp. 2006. All rights reserved.
1
1.1 What problem determination is
The term Problem Determination/Problem Source Identification (PD/PSI) is used to describe
the broad topic of finding out “what went wrong?” and “why did it go wrong?” when there is a
problem with a system. This means identifying which component of the system is responsible
for causing the problem.
The goal of PD/PSI, in its most basic sense, is to get to the root of a problem. It is similar to
what a programmer might call debugging, but on a much larger scale. PD/PSI includes
debugging applications, but also diagnosing the system at large by investigating product
configurations and verifying the means by which all of the system's components interact.
When a problem or system is complex then you need to adopt a more structured and
systematic approach in order to determine what caused the problem. You may follow the
process shown in the general problem determination (PD) flow chart we describe in 1.3.1,
“Problem determination flowchart” on page 2.
1.2 What PD/PSI is not
There are many symptoms that an enterprise system can show that are often classified as
“problems”.
Poor system performance, for instance, can definitely be a problem. The process of checking
and solving performance problems is often referred to as Tuning. Tuning a system involves its
own separate set of tools and processes.
Similarly, poor page flow on a Web site can cause confusion and problems for users, but this
is an interface design issue, and outside the scope of PD/PSI.
Understanding the difference between PD/PSI and Tuning is very important and knowing
when to use what will save you a lot of time.
PD/PSI fixes functional problems while Tuning means adjusting the system and application; it
handles problems associated with slow processes.
1.3 Problem determination approach
In this section we describe the steps involved in the problem determination process in general
terms. We introduce the common problem symptoms found in a WebSphere Application
Server for z/OS V6 environment.
First we use a flowchart to graphically show the steps, decisions and flow in our problem
determination process. Then we describe each step in the process and what is involved and
what should be considered at each step.
1.3.1 Problem determination flowchart
We use a flowchart to graphically represent the steps, decisions and flow of the problem
determination process.
2
Problem Determination Methodology for WebSphere on z/OS
1
2
Identify problem
symptom
Ask questions
3
Gather problem
documentation
4
Analyze
documentation
10
8
5
Documented
and
conclusive
?
No
7
Identified
problem/
solution
?
Consult reference
information sources
No
9
Prepare and send
problem
documentation
Yes
Yes
6
6
Contact IBM support
Take corrective
action
Take corrective
action
Figure 1-1 General problem determination flowchart
1.3.2 Problem determination process
The tasks outlined in the flowchart are expanded into detail in the following steps:
1. Identify the problem symptoms
2. Ask questions
3. Gather the documentation
4. Analyze the documentation
5. Documented and conclusive
6. Take corrective action
7. Consult reference information sources
8. Identified problem and solution
9. Prepare and send problem documentation
10.Contact IBM support
1) Identify the problem symptoms
Before you can begin to solve a problem you need to know what type of problem you have.
We start the problem determination process by identifying the symptoms of the problem.
Every time a software problem occurs, some kind of indication is given about it. It may be an
error message, a wrong output, an abend, no response or bad response times, or a message
returned by the browser. These are called symptoms.
Chapter 1. Problem determination methodology
3
It is very common for more than one condition to exist per problem. The more complicated the
scenarios are the more likely it is, a combination of problems lead to the symptoms you
experience. Therefore, always keep an open mind when performing Problem Determination.
Describe the symptoms of the problem. If there are multiple problems, try to separate them
and deal with them independently. Be careful about assumptions.
2) Ask questions
During this step we want to identify the background or supporting information to the problem
and we do this by asking questions. A good place to start is to identify if any recent changes
have taken place on the system.
The questions to consider:
򐂰 Has there been a change to the software operating system or the application server
software?
򐂰 Had there been an upgrade, new maintenance level applied or IPL?
򐂰 Have there been any changes to the environment like network topology, hardware
configuration or increase in the number of users?
򐂰 Have we made changes to the back end systems that we are connecting to?
򐂰 Have there been any new applications deployed, changed, upgraded?
򐂰 Have we run this application, server or system successfully before?
򐂰 When did the symptoms first appear? Always when system under peak load? Always after
backup jobs?
򐂰 Can you reproduce the error?
Asking these types of questions can help us eliminate potential causes early in the process.
The answers to the questions form part of our symptom data.
3) Gather the documentation
The type of information we need to gather depends on the type of symptoms we are
experiencing but essentially what we are doing at this stage is collecting evidence of the
problem. So if the symptom is an error message then you will need to obtain and examine the
log or trace which shows the message.
򐂰 Document your PD steps. Keep a log of symptoms, messages, files, tests, results of tests,
and conclusions.
򐂰 Retrace the steps described to recreate the problem and see the results yourself.
򐂰 Understand the meaning behind the request that has created or induced the problem. This
will help you isolate the problem.
򐂰 Use a controlled test environment when possible.
Knowing what data to collect and how can be difficult. There will be more documentation
about the analysis of symptoms in detail in future. Until then use the Redbooks WebSphere
for z/OS V5 Problem Determination, SG24-6880, and WebSphere Application Server V6
Problem Determination for Distributed Platforms, SG24-6798.
We also use the term mustgather doc. This is a term used to describe the essential or
minimum problem documentation required to analyze a problem. We continue to explore the
mustgather documentation in 2.3.3, “Mustgather: Gather background information” on
page 14.
4
Problem Determination Methodology for WebSphere on z/OS
Regardless of the type of problem IBM will need to know the software levels of your system.
4) Analyze the documentation
The documentation that is obtained will depend on the type of problem symptom. It is also
dependent on what you have enabled on your system. Some output is available by default
while others, like traces, may need to be enabled. How to enable the different output received
from an error is covered in Chapter 2, “Tools for Problem Determination” in the Redbook
WebSphere for z/OS V5 Problem Determination, SG24-6880.
Symptoms like Abend, Loop and Incorrect Output are often accompanied by messages, or
you can find indications in traces and logs. Check the data you have collected for this.
Analyze the messages, logs, and traces.
5) Documented and conclusive
In this step you should check the product documentation. These are the product reference
manuals or product Web sites to determine if your problem is documented. In the case of an
error message the product documentation describes the reason for the error and also
possible corrective action.
6) Take corrective action
The action you take depends on what is the cause and the recommended solution to your
problem. The typical outcome or action you take falls into the following categories:
򐂰 The product works as designed.
In this case you can accept the design and adjust your system accordingly or you can
request a design change. This is an official request to change the product design that is
assessed by technical staff, usually the product developer.
򐂰 You find a work-around for your problem.
This means that changes have to be made to your WebSphere for z/OS system to
circumvent the problem. In some cases this work-around is the solution. In other cases it is
a temporary solution in pace till a permanent fix is found.
򐂰 You find a problem scenario or symptom described in an APAR.
In this case you apply the fix (PTF) associated with the APAR to correct the problem.
򐂰 Your problem scenario or symptom is not found in the WebSphere for z/OS IBM support
page.
In this case you need to consider the problem is either:
– A new WebSphere Application Server for z/OS problem. This need to be reported to
IBM so they can produce a fix.
– It is a user error. This includes configuration, setup or procedural error. This needs to
be corrected by the user.
– It is an application problem. This need to be presented to the application owner or
developer to correct.
7) Consult reference information sources
Information sources available come in many forms. It can be a product manual, a Web site
pointing to product fixes, a colleague with specialist skill, an online technical forum and IBM
software support.
If your symptom is an error message then you should check the meaning of the message in
the product manuals. As this may point you to the exact cause of the error and tell you what is
Chapter 1. Problem determination methodology
5
required to fix it. If not then you can access IBM support data and search on your symptoms.
When you search on your symptom you may find other like problems reported by customers.
These problem records can tell you what was done or recommended in order to fix the
problem.
Refer to Chapter 3, “Information sources” on page 23, in which we outline what information
sources are available to you and how you can get access to them.
Sometimes your problem is very serious and your expertise in the product area is limited.
Then you may choose to go directly to the step of calling IBM support rather than try to
analyze your own trace or dump.
8) Identified problem and solution
Using the information sources you may have identified the problem and found a solution and
can now take corrective action. If you have not been able to identify the problem or find a
solution you need to prepare and gather the problem determination documentation.
9) Prepare and send problem documentation
If after consulting your information sources you have been unable to determine exactly what
the problem is or find the cause then you will need to prepare all the problem documentation
you have collected and get this ready to forward to IBM support.
10) Contact IBM support
Please refer to Chapter 2, “Contacting IBM: Information” on page 11 where we provide, in
detail, the options available when you need to contact IBM. We also explain the WebSphere
support teams and structure.
1.3.3 WebSphere for z/OS specific consideration for PD
Keep these points in mind when working with WebSphere for z/OS problems:
򐂰 WebSphere for z/OS is a complex software product involving many z/OS components, and
therefore requires intensive system administration.
򐂰 WebSphere for z/OS components require many environment parameters and variables to
be set to a specific value.
򐂰 User-set WebSphere for z/OS components require consistency throughout your
environment.
򐂰 Not all problems are WebSphere for z/OS related. Consider all your variables in your
specific z/OS environment in order to eliminate those that are definitely not relevant.
We suggest that you always keep detailed, up-to-date records of the following items when
working on a problem:
1. Your network topology
2. A high-level application description
3. A detailed model of your application
4. A detailed model of how your WebSphere for z/OS application interacts with other IBM
products, tools, or third-party software
5. A log of your WebSphere for z/OS setup
6. e-fixes installed for WebSphere for z/OS and other components interacting with your
WebSphere for z/OS system, such as WebSphere MQ and others
7. A log of your hardware specifications
6
Problem Determination Methodology for WebSphere on z/OS
Keep in mind that a history log of changes works much better than a simple log of the current
environment conditions. It is always best to retrace your steps up to the point of failure.
1.3.4 The importance of a test environment
You also need to consider the environment your problem occurs in. There are two major types
of environments in which problems have to be identified:
1. Test Environments and
2. Production Environments
Test Environments are usually easier to debug because they can easily be changed without
any business impact. On the other hand, debugging production environments impact your
business, and is therefore much more difficult. You might not be able to introduce change to
this environment easily as it could lead to interruptions or disturbance of you business.
However, it is not recommended to change your environment drastically rather apply
thoughtful PD/PSI methodology to your components and configurations based on symptoms,
causes, and analysis in order to find logical solutions to your problems.
1.4 The skills needed for WebSphere for z/OS
It is very important to consider that managing and administering a WebSphere system
properly requires expertise in many different areas.
This is also illustrated in Figure 1-2.
Share
information
Communicate
z/OS System Programmer
Type
text
Type
text
Type
text
Type
text
Type
text
Type text
Networking / TCP/IP
Security / RACF
WebSphere
on z/OS
Application deployment
Application Development
Work
together!
Cooperate
WebSphere Administrator
Figure 1-2 Working together
Working closely together will avoid many problems and save time and money.
Ensure that there are sufficient systems programming and application deployment skills and
experience, since WebSphere for z/OS utilizes most advanced features and functions of the
Chapter 1. Problem determination methodology
7
operating system. A list of these functions is available in WebSphere Application Server for
z/OS Version, 6.0.1: Migrating, coexisting, and interoperating, SA23-2207. You will need
systems programming skills in all these areas. If you try to set up the WebSphere runtime
without good skills or assistance in these areas, you are bound to have many frustrating
problems and delays.
See 3.6, “Educational information” on page 34 for resources to improve your skills for
Problem Determination for WebSphere for z/OS.
1.4.1 System skills
The following skill areas are critical to successfully installing and establishing a runtime of
WebSphere for z/OS. These skills are more or less “traditional”:
򐂰 z/OS - to install software products and related pre-requisites; and to set up required OS
resource definitions and settings
򐂰 UNIX® System Services - to set up a functional HFS and UNIX environment
򐂰 TCP/IP - to configure connectivity for WebSphere clients and servers
򐂰 RACF® (or equivalent) - to authenticate WebSphere clients and servers and authorize
access to resources
򐂰 Logger - to set up log streams for RRS and the WebSphere error log
򐂰 Parallel Sysplex® - to implement multi-system configurations
򐂰 RRS - to implement resource recovery services and support two-phase commit
transactions
򐂰 ARM - to set up automation process for stopping and starting up WebSphere runtime
environment. Although deemed to be optional, it is crucial to have all operational
processes automated in a multiple LPARs multiple application servers environment.
򐂰 WebSphere - to customize and set up WebSphere administrative servers and application
servers. Configure WebSphere resources as required by application.
1.4.2 Skills for deploying and running an application
Beyond the traditional skills, you need people with skills in developing, deploying and running
the applications, while the development is usually done by third-party vendors.
The people who are deploying and running the application should:
򐂰 Understand the WebSphere for z/OS structure
򐂰 Understand the J2EE™ architecture (e.g., understand how the deployment descriptor fits
into their WebSphere for z/OS configuration)
򐂰 Be able to use ASTK or WebSphere Studio to modify deployment descriptors, view layout
of JARs, WARs, etc. or to create EAR files.
򐂰 Be familiar with and capable of setting trace settings for WebSphere to trace various
components (Web container, EJB™ container, class loader etc.)
򐂰 Be able to use the Administrative Console, or WSAdmin, to create J2EE server instances
and install J2EE applications.
򐂰 Be able to understand output of job log, WebSphere for z/OS error log, and trace output of
runtime and J2EE servers as they relate to troubleshooting deployment problems
򐂰 Work closely with system programmers for WLM configuration, and creating procedures
for J2EE server instances
8
Problem Determination Methodology for WebSphere on z/OS
򐂰 Work closely with the Security Administrator to define userids and groups for J2EE Control
Regions and Server Regions as well as the EJB security environment
򐂰 Work closely with application developers for defining data sources, resolving external EJB
references, understanding the logic of application, etc.
򐂰 Be familiar with USS and comfortable using the shell
򐂰 Be familiar with their TCP/IP setup and able to use commands such as tracert, netstat
And, optionally they should:
򐂰 Work closely with the DB2® Administrator, WebSphere MQ Administrator, etc. in case of
problems with data sources
򐂰 Be familiar with LDAP and able to understand LDAP traces, and know how to use the
LDAP browser, etc.
Figure 1-3 illustrates what Application Development and Deployment mean. You should
always be aware of all these dependencies.
The Theory
Design
Build
Assemble
Deploy
Build
Assemble
Deploy
Reality
Design
Rules and constraints down the line influence earlier actions
Figure 1-3 Deploying applications
It is unlikely that any one person can possess all these skills. It takes a team of specialists to
set up the WebSphere runtime and run the server. For specific courses and the “roadmaps”
on the same site for an organized view of the curricula, see the class catalogs at:
http://www.ibm.com/services/learning/
Chapter 1. Problem determination methodology
9
10
Problem Determination Methodology for WebSphere on z/OS
2
Chapter 2.
Contacting IBM: Information
This chapter provides information to assist you when communicating with IBM Support. We
describe the WebSphere Application Server for z/OS support team structure and cover how
to place a Problem Management Record (PMR) with IBM Support and also how to exchange
information and send documentation to IBM Support Teams.
© Copyright IBM Corp. 2006. All rights reserved.
11
2.1 Communicating with IBM
This section deals with the communication between customers and IBM. Customer
experiences with IBM Support departments show that some critical situations in handling and
solving a software problem are results of a lack of communication and information. We want
to contribute to filling this gap.
How is IBM Support structured? What to do first if a software problem is encountered? What
information and data is needed? How to report this problem to IBM? These are the questions
we answer here.
Many of these items are an excerpt of what is also pointed out in the IBM Software Support
Guide on the Internet, where much more information is available, such as:
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
Software Support Handbook Overview
Enhanced Support
Contacting IBM
No support contract
Preventing problems
Support contacts (worldwide)
Additional offerings
Acronyms and other terms (really helpful for finding IBM terms)
You can find the Guide at:
http://techsupport.services.ibm.com/guides/handbook.html
2.2 The IBM WebSphere support structure
Figure 2-1 shows a bird’s eye view of how IBM Support is structured.
IBM Software Support - Worldwide Structure
customer
entitlement
Frontend
Frontoffice
(domestic)
Teams
Backoffice
experts
Backend
Change Teams and
Development Teams
usually in labs
Figure 2-1 IBM support structure
If a problem is reported via a Problem Management Record (PMR), this usually happens via
the Problem Entry helpdesk or Front office Teams. The next level is the Front-end Support
Personnel, who usually have the broader skills about IBM software products. These people,
especially in the Front Office Teams, also have a national language approach.
If more in-depth skills are needed, the back end becomes involved (for example, IBM software
laboratories), where the software is developed and necessary code changes are made.
Communication between front-end and back-end support works very well due to the
worldwide IBM network and the fact that the teams usually know each other well and use all
communication vehicles.
12
Problem Determination Methodology for WebSphere on z/OS
2.3 Before contacting IBM Support
There are various means to getting support for your specific problem. You can benefit
immediately from the IBM extensive Web-based self help support Web site, where you can
download fixes, search on keywords, find how-to information and possibly solve your problem,
all before contacting IBM Support directly.
The latest information about getting support for WebSphere z/OS can be found at:
http://www.ibm.com/software/webservers/appserv/zos_os390/support/
Tip: Search using the component id 5655I3500 as one of the keywords. This will reduce
the search results to only WebSphere Application Server for z/OS problems and APARs.
The following sections describe actions you should take before contacting IBM Support
Center. Most of these information are also described in the TechDoc Steps to getting support
for WebSphere Application Server, at:
http://techsupport.services.ibm.com/guides/handbook.html
2.3.1 Define the problem
Being able to articulate the problem and symptoms before contacting software support will
expedite the problem solving process. It is very important that you are as specific as possible
in explaining a problem or question to IBM software specialists. The specialists want to be
sure that they provide you with exactly the right solution, so the better they understand your
specific problem scenario, the better they are able to resolve it.
Try to recreate the problem. Document the following:
򐂰 Steps you took to recreate the problem, and any symptoms or error messages you
observe. Some typical details are:
– Date and Time
– User name or user ID involved
– LPAR name, server name, job name or STC name, etc.
򐂰 Recent changes that have been made to your processing environment, such as hardware
or software that has been added or removed
򐂰 System configuration updates
Please be aware that you should report only one problem or question per PMR. This avoids
confusion and misunderstandings about the case that’s reported in the PMR.
2.3.2 Determine if this situation has already been reported
The problem might already be documented and resolved, so check these product support
resources to see if the answer you are looking for is available:
1. Information Centers and Release Notes
Information Centers provide fast, centralized access to WebSphere Application Server
product information, available in multiple languages and updated regularly. The problem
might also be documented in the release notes and in the readme file packaged with the
product.
2. Software and hardware prerequisites
Chapter 2. Contacting IBM: Information
13
Verify the product release and major update requirements for the software you are
running, i.e., WebSphere Application Server for z/OS V6 and its requirements.
3. WebSphere Application Server related
Product support Access APARs, Technotes, and PTFs, register to receive e-mail
notifications about technical alerts or new downloads, and use an advanced search
feature that searches all IBM knowledge bases, such as Redbooks and Information
Centers.
4. MySupport
Register to receive e-mail notification about critical issues, IBM product updates, and
items of interest.
5. Link2000
For IBM Eserver® zSeries users with installations that have access to Link2000
(previous IBMLink™), an interactive online database program, you can:
– Search for an existing authorized program analysis report (APAR) that is similar.
– Search for an available program temporary fix (PTF) for the existing APAR.
– Order the PTF if it is available.
6. DeveloperWorks WebSphere
The gateway to WebSphere technical information for developers and administrators,
featuring:
– Zones and roadmaps for specific products, in depth technical articles, tutorials, white
papers, and links to downloads, technical previews, (2 of 8) steps to getting support for
WebSphere Application Server and plug-ins.
– Latest news on WebSphere products and offerings.
2.3.3 Mustgather: Gather background information
Mustgather documents are documents that aid in problem determination and save time
resolving Problem Management Records (PMRs). These documents are located on the
eSupport site and contain specific instructions about what documentation to gather for
specific problems. You can find Mustgather documents by searching on the word Mustgather
on the eSupport Web site:
http://www.ibm.com/software/webservers/appserv/zos_os390/support
These are some of the MustGather documents for WebSphere for z/OS which might help you:
򐂰 Mustgather: Read first for WebSphere Application Server for z/OS
򐂰 Mustgather: High CPU causing Hang or Loop running V5 for z/OS
򐂰 MustGather: Plug-in Regeneration Problems for V5.0 and V5.1
򐂰 MustGather: Plug-in problems in V5.0 and V5.1 on z/OS
򐂰 MustGather: System management for synchronization failures
򐂰 MustGather: System management discovery problems
򐂰 MustGather: ABENDEC3 RC=413000x for 4.0, 5.0 and 5.1 for WebSphere Application
Server for z/OS
򐂰 MustGather: wsadmin problems in V5
򐂰 MustGather: Administrative console problems
򐂰 MustGather: A hang occurs when running WebSphere Application Server for z/OS
14
Problem Determination Methodology for WebSphere on z/OS
򐂰 Mustgather: Security problems with WebSphere Application Server z/OS V5
To effectively and efficiently solve a problem, the software specialist needs to have all of the
relevant information. Being able to answer the following questions will help IBM Support
personnel in their efforts:
򐂰 What levels of software were you running when the problem occurred? Please include all
relevant products, that is, the operating system as well as related products. Typically the
following information should be included:
– Operating System version
– WebSphere Application Server (Base or ND) version
– Java™ run time (JDK™ or JRE) version
– Host Security software (RACF, ACF2 or TopSecret) version
– Optionally: DB2, MQ, HTTPD, etc.
򐂰 Major configuration such as the following should also be mentioned, if relevant:
– Monoplex or Sysplex
– Global security
– Clustered or non-clustered application
򐂰 Has the problem happened before, or is this an isolated instance?
򐂰 What steps led to the failure?
򐂰 Can the problem be recreated? If so, what steps are required?
򐂰 Have any changes been made to the system (hardware, network or software)?
򐂰 Were any messages or other diagnostic information produced? If yes, what were they (for
example, trace record, dump output)?
It is often helpful to have a printout of the message numbers of any messages you received
before calling IBM.
The most common data requested by IBM is the following:
򐂰 System log (SYSLOG): The z/OS system log, holding assorted system error messages
and a few WebSphere error messages
򐂰 WebSphere Server job logs: Application server job logs contain most of the configuration
settings, stderr and stdout messages as well as CEEDUMP and snap dumps. Job logs for
deployment manager, node agent and daemon servers may also be required depending
on the problem nature.
򐂰 WebSphere Error Log: Target for WebSphere error messages
򐂰 Dump data sets: If system abend occurred a JVM™ transaction dump and SVC dump
may be produced. In most cases, you will be asked to compress (terse) and FTP them to
designated FTP site for IBM Support center. In some situations where you have to cancel
server region in order to overcome a problem, be sure to request a SVCdump when
cancelling.
򐂰 Component trace (CTRACE) message log: If more details information is required, IBM
Support will request you to have the component trace writer and debug be turned on to
display more detailed trace information in the message log.
How to get this data is also described in the WebSphere for z/OS Information Center at:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp
Chapter 2. Contacting IBM: Information
15
Define your technical question in specific terms and provide the version and release level of
the products in question.
2.3.4 Determine the business impact
You need to assign a severity level to the problem when you report it, so you need to
understand its business impact.
Table 2-1 shows descriptions of the severity levels.
Table 2-1 Problem severity levels
Severity
Level
Definitions
Examples
Severity 1
Critical business impact. This indicates you
are unable to use the program, resulting in a
critical impact on operations in a production
environment. This condition requires an
immediate solution.
a) All users of Tivoli® Problem
Management are unable to register a
call.
b) An Application Server running
business-critical applications crashes
and does not come up again.
Severity 2
Significant business impact. This indicates
the program is usable but is severely limited.
All users of Tivoli Problem
Management receive a database
manager error while attempting to view
open problems.
Severity 3
Some business impact. This indicates the
program is usable with less significant
features (not critical to operations)
unavailable.
A client cannot connect to a server.
Severity 4
Minimal business impact. This indicates the
problem causes little impact on operations or
that a reasonable circumvention has been
implemented.
Documentation is incorrect.
2.4 How your call is handled by IBM Software Support
In order to submit a problem to the WebSphere Application Server for z/OS support team you
need to open a Problem Management Record (PMR).
2.4.1 Opening a Problem Management Record (PMR)
The following are necessary when opening a PMR:
򐂰
򐂰
򐂰
򐂰
IBM Customer number
Contact name, phone number and e-mail address
Operating System Name and Version
The product name and the Component ID of the product.
The component ID for WebSphere Application Server for z/OS V6 is 5655-N01.
16
Problem Determination Methodology for WebSphere on z/OS
There are several options available to submit your problem to IBM support some you are
familiar with or prefer already. These options are:
1. Telephone your local IBM support directly and the person who takes you call creates a
Problem Management Record (PMR) with the details you provide. The software support
handbook provides a list of IBM service numbers.
http://techsupport.services.ibm.com/guides/contacts.html
2. Using the Web problem submission tool to submit an Electronic Service Request (ESR) at
the following site.
http://www.ibm.com/software/support/probsub.html
3. Use IBMLink 2000 to place your own electronic version of the Problem Management
Record which is sometime referred to as an ETR. You need to be a registered IBMLink
2000 user to use this option. IBMLink 2000 is also referred to as servicelink.
https://www.ibm.com/ibmlink/link2/logon/logonPage.jsp
Regardless of which option you choose all software support calls for z/OS software products
are records in the IBM RETAIN® (REmote Technical Assistance Information Network)
system. This system is used world wide by all the support teams and is a very effective tool
for IBM support teams to communicate.
The advantage of placing an electronic call using ESR or IBMLink 2000 is that you are
available to view the updates in the record and monitor the status of your request.
Once logged, a unique problem management record (PMR) or Incident/Support Case, is
created.This record number is allocated regardless of how the problem is submitted.
Make note of this PMR number, Incident number, or Support Case number and use it in any
future communication on this issue with the support center.
Your PMR, Incident, or Support Case is routed to a resolution team for handling. You may be
transferred directly to the resolution team or your issue will be placed in a queue for callback.
2.4.2 Investigating a PMR
At the resolution team level your call is researched, resolved, or escalated as appropriate.
Due to the level of specialization required to maintain superior technical expertise at the team
level, it is sometimes necessary to involve more than one support team in resolving a
particular software problem. This is easily handled, as IBM support teams are all networked
together and work as one to resolve whatever problems or issues arise.
In order to investigate the issue, IBM may need to access information on your system relative
to the failure, or may need to recreate the failure to get additional information. It can also
happen that any problem data/material is requested by IBM Support during a problem’s
lifetime. In case of a very difficult problem scenario, some possible areas have to be excluded
where it is sometimes necessary to gather different data, such as traces, to isolate the
problem.
Should the problem be configuration-related, it is possible you may need to recreate the
problem to get that required information.
Chapter 2. Contacting IBM: Information
17
During this investigation process, the Resolution Team determines whether your defect issue
falls into one of three categories:
1. It is the result of a software defect that has previously been reported.
A fix or work-around is provided to circumvent or correct the issue. If no work-around is
available and it is determined that one is required, the Resolution Team will work with you
to find the best feasible work-around. The Resolution Team advises you when the defect
(APAR) is closed, assists in fix implementation and updates your problem record.
2. It is the result of an IBM software defect that has not been reported before.
The Resolution Team will work with you to create an Authorized Program Analysis Report
(APAR) or Software Problem Report (SPR) to track the resolution of the defect. These
APARs and SPRs are routed to the appropriate development teams.
Because of the complexities of the environments supported and the development,
verification and testing resources required, defect fixes may require an extended period of
time for resolution.
For high impact problems, the resolution teams will make every effort to develop a
work-around that you can use until your APAR or SPR has been resolved.
3. It is a problem that is not defect-related.
If it is not a software defect in supported IBM code, then the Resolution Team might
continue to work on the problem for a solution only at the request and agreement of the
customer under a separate service agreement.
2.4.3 How technical questions are handled by IBM
Technical question support enables you to obtain assistance from IBM for product-specific,
task-oriented questions regarding the installation and operation of currently supported IBM
software. In the course of providing answers to your technical questions, Support people may
refer you to product documentation or publications, or they may be able to provide a direct
answer to assist you with the following:
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
Installation
Usage (how-to)
Specific usage/installation questions for documented functions
Product compatibility and interoperability questions
Technical references to publications (Redbooks, manuals, etc.)
Assistance with interpretation of publications
Configuration samples
Planning information for software fixes
IBM database searches
Software Maintenance or Support Line are not structured to address questions on
performance, consulting, or extensive configuration questions. Additional telephone and
on-site support services are available to meet these needs.
For further information about these services, contact your IBM Representative who can help
direct you to the persons who can discuss your needs. This could be handled, for example, by
consultants and architects in the ITS Offerings area.
2.5 Exchanging data with IBM via FTP
The most common and usually preferred way to exchange data with IBM is to send it to an
FTP server. Follow the steps described in this section to ensure smooth data transfer.
18
Problem Determination Methodology for WebSphere on z/OS
2.5.1 Copy the Job log into a z/OS data set
If you got a dump or trace, this is written to a z/OS data set or to the job log. There are two
ways to copy a job log into a data set. You can use a normal TSO command, such as:
TSO OUTPUT jobname(jobnumber) PRINT(TEST.DATASET) KEEP
Unfortunately, this function is often limited to your own user id and you can only print jobs that
are in the HELD OUTPUT queue.
So it might be better to use the SDSF-based function.
You can either use the XDC action line command on any SDSF panel (x - output, d dataset, c - close after print), or you can go into the actual system output (SYSOUT)
file and use the print command on the command line.
You first have to open the print with the print odsn datasetname command. Afterwards you
can use the print command without any options to print the complete file, or use print 200
400 to print a specific range of lines. You can enter help print and press Enter to get more
details and options.
Then you either have to enter print close or leave SDSF to close the print file. Otherwise
you can amend the file with other print commands.
2.5.2 Compress data
Make sure that large files or data sets are compressed before sending them to any FTP
server. Compression decreases the size of the file and the amount of time needed to send the
file to an FTP server. Be conscious of the binary/ASCII settings and send your file/data set in
the proper format. Compressed data should always be sent in binary instead of ASCII format.
There are three possible ways to compress data:
1. TRSMAIN/Packlib
2. tar command
3. zip
TRSMAIN/Packlib
Using TRSMAIN (also known as Packlib) for compressing data is the most common way in
the z/OS environment. A big advantage of data sets in “tersed” (packed) format is that the
data set attributes are stored and the file can easily be uploaded and “untersed” (unpacked) in
another z/OS system without guessing on the DCB parameters.
If you send a data set without using TRSMAIN, make sure to provide IBM Support with values
such as LRECL, RECFM, BLKSZ and space requirements.
TRSMAIN can be downloaded from:
ftp://ftp.software.ibm.com/s390/mvs/tools/packlib/
Once the TRSMAIN is installed, use the sample JCL in Example 2-1 to create your own job
with proper modification to &PACKLIB_PDS, &input_dataset and &tersed_output in order to
compress &input_dataset into its compressed format.
Example 2-1 Job to invoke TRSMAIN
//PACKIT JOB 'ACCOUNTING INFORMATION',NOTIFY=&SYSUID.
//****************************************************
Chapter 2. Contacting IBM: Information
19
//*
*
//*
TRSMAIN with PACK option
*
//*
*
//****************************************************
//JOBLIB DD DISP=SHR,DSN=&PACKLIB_PDS
//STEP
EXEC PGM=TRSMAIN,PARM=PACK
//SYSPRINT DD SYSOUT=H
//INFILE
DD DISP=SHR,DSN=&input_dataset
//OUTFILE DD DISP=(NEW,CATLG),UNIT=SYSDAL,
//
DSN=&packed_output,
//
SPACE=(CYL,(ppp,sss),RLSE)
The JOBLIB DD can be eliminated if the &PACKLIB is included in the LNKLST concatenation.
The &input_dataset in the INFILE DD must be modified with the proper data set name that
needs to be compacted, and the &packed_output in the OUTFILE DD must be modified with
the data set name accordingly. The ppp and sss are the primary and secondary space for the
output data set. You can also do that via an ISPF dialog by entering the program name on the
command line and filling in the fields on the related panels.
The tar command
Files in any HFS directory in Unix System Services can be compressed using the tar
command. Here are two examples of how to use it:
1. This command takes a directory and places it in an archive in compressed format:
tar -cvzf archive directory
2. To identify all files that have been changed in the last week (7 days), and to archive them
to the /tmp/posix/testpgm file, enter:
find /tmp/posix/testpgm -type f -mtime -7 | tar -cvf testpgm.tar -
-type -f tells find to select only files. This avoids duplicate input to tar.
More information about this command can be found in z/OS V1R3.0 UNIX System Services
Command Reference, SA22-7802-03.
ZIP file
This format is especially relevant for files on the PC. You can also put multiple files in a
so-called ZIP archive. The most common programs used for this approach are Winzip and
PKZip. These can be found on:
http://www.winzip.com
http://www.pkzip.com
After packing a file into that ZIP archive, it should be an EXE (executable) file. This enables
the recipient of the file to extract it properly without having Winzip or PKZip installed.
2.5.3 FTP instructions
The different IBM geographies and/or regions provide you with the information about
specifically where and how to send in your data. Most likely the IBM support personnel will
request the data be sent to the erucrep ftp site:
ftp.emea.ibm.com
Some additional information about file upload and download procedures is provided on the
following site:
http://www.ibm.com/de/support/ecurep/mvs.html
20
Problem Determination Methodology for WebSphere on z/OS
2.5.4 Naming conventions
When sending data to an FTP server in IBM, it always makes sense to use some naming
conventions that give an indication on the type of a file or data set. Further, an indication on
the PMR number should be given, so it is much easier to identify the file/data set and where it
is coming from. Examples are shown in Table 2-2.
Table 2-2 PMR numbers and what they indicate
File/data set name
Comments
PMR12345.CEEDUMP
This data set belongs to PMR #12345 and contains a CEEDUMP.
PMR12345.CEEDUMP.TERSED
This data set belongs to PMR #12345, contains a CEEDUMP and is
compressed using TRSMAIN.
PMR12345.CONFIGFILES.ZIP
This data set belongs to PMR #12345, contains configuration
files and is compressed on the PC using a ZIP program.
PMR12345.CONFIGFILES.EXE
This data set belongs to PMR #12345, contains configuration
files, is compressed on the PC using a ZIP program and then
converted to an EXE file.
PMR12345.CONFIGFILES.TAR
This data set belongs to PMR #12345, contains configuration
files and is compressed using the USS tar command.
These conventions may be slightly different in various IBM geographies and/or regions, but
IBM Support personnel will advise you how and with which naming conventions data should
be sent to the FTP server.
2.6 IBM contacts
There are various possibilities for contacting IBM Support personnel.
Usually the person working on the problem you reported provides you with an e-mail address
and phone number. This information exchange allows for direct contact to discuss the
problem or to give additional information.
It is very helpful for IBM Support to know who your backup is (with phone number and e-mail
address) in case of illness or vacation. Additional information may be needed by IBM during
your absence.
Furthermore, you can update the PMR by yourself to provide additional information.
Providing problem documentation to IBM can be done via e-mail (small files or configuration
data), FTP, or other tools. IBM Support will give you detailed information about how and
where to send the data. The most common and preferred way is FTP, as described in 2.5,
“Exchanging data with IBM via FTP” on page 18.
Many Support teams have their own team-related e-mail address that allows all team
members access.
Looking for any IBM contact for the first time, refer to the IBM Directory of worldwide
contacts:
http://www.ibm.com/planetwide
Chapter 2. Contacting IBM: Information
21
22
Problem Determination Methodology for WebSphere on z/OS
3
Chapter 3.
Information sources
In addition to this Redpaper, there is plenty of documentation—books and Internet Web
sites—available concerning WebSphere for z/OS and supporting components.
This chapter describes some of the resources available. We found very helpful material and
sites for solving problems in the WebSphere for z/OS environment.
© Copyright IBM Corp. 2006. All rights reserved.
23
3.1 WebSphere for z/OS and its support pages
Traditionally the most referenced source of information was the product manuals. While these
are still important the main source for WebSphere Application Server for z/OS is the
homepage. From there you can access various other pages with important information about
WebSphere on z/OS, which we describe in this section.
The WebSphere for z/OS home page
This page is a central entry point for a wide variety of information about WebSphere for z/OS,
see Figure 3-1. You can find the page at:
http://www.ibm.com/software/webservers/appserv/zos_os390/
Figure 3-1 IBM WebSphere Application Server for z/OS Home Page
Click the links of the gray box on the left side for specific information categories like system
requirements, library (manuals and InfoCenter), and services.
WebSphere Support page
The support link, see Figure 3-2 on page 25, provides you with all sorts of online help in case
of Problem Determination for WebSphere for z/OS:
24
Problem Determination Methodology for WebSphere on z/OS
Figure 3-2 IBM WebSphere for z/OS support Web site
Chapter 3. Information sources
25
WebSphere for z/OS V6 Product Manuals
Figure 3-3 IBM WebSphere Application Server Library Page
This page lists the WebSphere product manuals. When you select show on the WebSphere
Application Server - z/OS section it provides a link to the Information Center and also displays
a list of the product manuals for z/OS that are available in PDF format. At time of writing this
redbook the following manuals were available:
򐂰 Program Directory, GI11-2825
򐂰 Migrating, Coexisting, and Interoperating, SA23-2207
򐂰 Installing Your Application Serving Environment, GA22-7957
򐂰 Administering Applications and Their Environment, GA22-7962
򐂰 Setting Up the Application Serving Environment, GA22-7958
򐂰 Using the Administrative Clients, SA23-2208
򐂰 Securing Applications and Their Environment, SA22-7961
򐂰 Developing and Deploying Applications, SA22-7959
򐂰 Troubleshooting and Support, GA22-7964
26
Problem Determination Methodology for WebSphere on z/OS
򐂰 Tuning Guide, SA22-7963
Attention: There is no longer a Messages and Codes manual. For messages and codes
see the InfoCenter or Appendix A, “Messages and codes” on page 37.
WebSphere for z/OS V6 InfoCenter
The Information Center displays documentation for several WebSphere Application Server
products.
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp
Click the product name WebSphere Application Server for z/OS V6 listed in the information
center navigation to display the welcome page containing information that is specific to this
product.
Figure 3-4 IBM WebSphere Application Server for z/OS InfoCenter Page
You can limit the search scope by clicking the link Search scope on the top of the page, click
New in the window popping up, define a list name and tick the box for WebSphere for z/OS.
To look up messages and codes either search for the particular message or code in the
InfoCenter or go to the Contents panel in the InfoCenter and click Reference →
Troubleshooter → Messages, like in Figure 3-5 on page 28. Then choose the tab according
to the first few letters in your message code.
Chapter 3. Information sources
27
Figure 3-5 Messages and codes in the InfoCenter
Also see the Appendix A, “Messages and codes” on page 37 for WebSphere for z/OS
messages and their code explanations.
WebSphere Application Server for z/OS IBM services
Consider using WebSphere Application Server for z/OS IBM services, for example, having a
“WebSphere Proof of Concept for z/OS,” in which IBM consultants design and implement a
working solution to your e-business problem identified during the Architecture and Design
Workshop by using preconfigured IBM hardware and software, allowing you to implement a
WebSphere production environment without interfering with your day-to-day business
functions. For more information, see:
http://www.ibm.com/software/webservers/appserv/zos_os390/services/
Recommended reading list: WebSphere Application Server
Learn about using IBM WebSphere Application Server with this reading list, compiled for
customers, consultants, and other technical specialists, by IBM Software Services for
WebSphere.
This list of recommended reading material on the IBM WebSphere Application Server is
compiled from a variety of sources by IBM Software Services for WebSphere. Many of these
documents focus on critical areas that should be understood before diving into Web
28
Problem Determination Methodology for WebSphere on z/OS
application design and implementation. Others illustrate different stages of the project life
cycle and should be reviewed before proceeding with each progressive phase.
http://www-128.ibm.com/developerworks/websphere/library/techarticles/0305_issw/recommend
edreading.html
3.2 Techdocs: Whitepapers, Hints and Tips
This Web site provides technical content about recommendations and actual experiences
written by IBM specialists worldwide.
http://www.ibm.com/support/techdocs/
You will find guides on configuration, installation tips, operational recommendations, and
several other documents regarding tuning and debugging. There are also good search
capabilities for any search term. It links to:
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
PTFs and APARs
Flashes
Presentations and Tools
Hints, Tips and Technotes
FAQs
Whitepapers
3.3 Redbooks and draft publications
The number of IBM Redbooks in this technology area is growing. It is a hot topic and there is
a continuous stream of new titles on the Redbook Web site. To get brand-new information
about latest (not yet fully published) books, check the “Drafts” section.
Figure 3-6 on page 30 provides a small subset of examples
Chapter 3. Information sources
29
Figure 3-6 Recent Redbooks and Redpaper samples
3.4 Sources of information for developers
There are various sources of information for developers which also help administration and
support staff when trying to solve problems in WebSphere for z/OS.
WebSphere Developer’s Domain
At this Web site you can find platform-independent information about best practices, hints and
tips, documentation, tools, and links to other technical information.
http://www.ibm.com/developerworks/websphere/newto/
Unlike the other documents, these are not z/OS-specific and some of the tools may not be
available for all platforms. It links to:
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
New to WebSphere
Products
How to buy
Downloads
Technical Library
Training
Support
Services
Forums and community
News
alphaWorks community
alphaWorks® provides a unique opportunity for developers around the world to experience
the latest innovations from IBM. These emerging “alpha code” technologies are available for
30
Problem Determination Methodology for WebSphere on z/OS
download at the earliest stages of development — before they are licensed or integrated into
products — allowing users to evaluate and influence IBM research and development. In
addition, early adopters have access to a virtual collaborative community to learn more about
the uses of a particular technology, and opportunities for commercial use of alphaWorks’
technologies.
http://www.alphaworks.ibm.com/
Java
On the Java community process Web site:
http://jcp.org/
You will find many hints and tips of how to code J2EE applications. It also contains a
reference section with the specifications, whitepapers, and other Java-related information.
On the SUN Java technology home page:
http://java.sun.com
You can find first-hand information directly from the founder of Java technology. Sun™ Java™
Technology still is a major contributor to the Java community. The Web site contains technical
information, specifications, examples, and references for J2EE.
See the JAVA documentation Web site:
http://java.sun.com/j2se/download.html
For information and downloads about J2EE, see
򐂰 J2EE Software Development Kits (SDK)
򐂰 J2EE API Documentation
򐂰 J2EE Platform Specification
See the DeveloperWorks and alphaWorks domains for more Java related information.
3.5 Other helpful Web sites
When solving problems in WebSphere for z/OS you need to consider that the Application
Server is implemented into z/OS, a complex and function rich operating system of a platform
which provides a lot of special features. Problems therefore can appear in an application or in
the application server but could stem from the underlying operating system or platform, or
from not exploiting both of them efficiently. The Web sites in this section can help to find the
problems related to the underlying technology of WebSphere on z/OS.
IBM eServer zSeries support
Figure 3-7 on page 32 shows the support-related links on the Web site:
http://www.ibm.com/servers/eserver/support/zseries
Chapter 3. Information sources
31
Figure 3-7 IBM eServer™ zSeries Support
z/OS home page
http://www.ibm.com/servers/eserver/zseries/zos/
The main z/OS home page covers the complete range of software products and technologies
for this operating system.
LookAt messages
http://www-03.ibm.com/servers/eserver/zseries/zos/bkserv/lookat/
This site provides a great search mechanism for z/OS messages, see Figure 3-8 on page 33.
32
Problem Determination Methodology for WebSphere on z/OS
Figure 3-8 LookAt messages
Also refer to the message tables provided in Appendix A, “Messages and codes” on page 37.
All software products
http://www.ibm.com/software/sw-atoz/
On this page you can search for all IBM software products via an index, sometimes very good
to seek help and information.
IBM Software support guide
The purpose of this document is to provide guidelines and reference materials that customers
may need when they require IBM service and support.
http://techsupport.services.ibm.com/guides/handbook.html
It links to:
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
Overview
Enhanced Support
Contacting IBM
No support contract
Preventing problems
Support contacts
Additional offerings
Acronyms and other terms
Chapter 3. Information sources
33
z/OS Internet library
On this Web site you can view, and print or order all available IBM manuals.
http://www.ibm.com/servers/eserver/zseries/zos/bkserv/
The books can be accessed in HTML format or you can download them in PDF format for
easy reference and printing.
For zSeries softcopy information, see:
http://www.ibm.com/servers/eserver/zseries/softcopy/
3.6 Educational information
IBM offers a comprehensive portfolio of technical training and education services designed
for individuals, companies, and public organizations to acquire, maintain, and optimize their IT
skills.
IBM Global Service
Go to the IBM Global Service Web site:
http://www-1.ibm.com/services/us/index.wss/home
Click the “training” panel and browse through:
򐂰
򐂰
򐂰
򐂰
򐂰
Course catalog
e-Learning
Blended learning
Save money
On-site training
WebSphere for z/OS training and certification
Click the “Training and Certification” tab in the left panel on the WebSphere for z/OS Web site:
http://www.ibm.com/software/webservers/appserv/zos_os390/
This Web site offers the latest information about upcoming training events, e-learning
courses, and online tutorials, IBM offers for WebSphere and Java.
These are simply a few examples:
WBSR6
WebSphere for z/OS V6 Implementation Workshop
WSW06
Security Workshop: WebSphere Application Server for z/OS
Education Assistant
This site integrates narrated presentations, Show Me Demonstrations, tutorials, and resource
links to help you successfully use the IBM software products.
http://www-306.ibm.com/software/info/education/assistant/
Go directly to the education modules and choose a product.
If you choose WebSphere Application Server for z/OS you get a whole list of education
material, amongst them: Problem Determination. Click this link, and you can download
several presentations and audio material specifically and work with them in your own pace.
34
Problem Determination Methodology for WebSphere on z/OS
Figure 3-9 IBM Education Assistant WebSphere for z/OS, Problem determination
Chapter 3. Information sources
35
36
Problem Determination Methodology for WebSphere on z/OS
A
Appendix A.
Messages and codes
This appendix provides messages and codes for WebSphere Application Server components
and subsystems of z/OS to support you in analyzing errors and problems.
We explain the format of WebSphere for z/OS message codes, list specific Java component
messages, mention minor codes, provide WebSphere for z/OS related abend codes, and the
most common non-WebSphere for z/OS-related message prefixes with details about where
they come from.
The following tables are summaries from the WebSphere Application Server for z/OS V5.1:
Messages and Codes, GA22-7915, and the WebSphere for z/OS Information Center
available at:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp
© Copyright IBM Corp. 2006. All rights reserved.
37
A.1 WebSphere for z/OS message codes
The prefix for WebSphere for z/OS messages is BBO. The format is BBOcnnnnt. Table A-1
provides the WebSphere for z/OS message formats.
Table A-1 WebSphere for z/OS message formats
Message format
Description
BBO
Identifies it as a WebSphere for z/OS message.
DYNA
Identifies it as a WebSphere for z/OS Dynamic Fragment Cache message.
c
Indicates the component.
nnnn
A unique identifier.
t
Severity (Information, Warning, or Error).
Table A-2 gives an overview of where BBO messages come from and where they appear.
Table A-2 WebSphere for z/OS messages overview
Prefix
Come from
Appear on or in
BBOJnnnnt
Java Virtual Machine (JVM).
Operator’s console error log, job log
BBOMnnnnt
Runtime environment.
Operator’s console, error log, job log
BBOOnnnnt
Control process, servant process, daemon,
CORBA. (These are general messages.)
Operator’s console, error log, job log
BBOSnnnnt
Security system.
Operator’s console, error log, job log
BBOTnnnnt
Transaction service.
Operator’s console, error log, job log
DYNAnnnnt
Dynamic Fragment Cache.
Job log
To look up specific message codes follow these steps:
1. In the InfoCenter navigation panel click the product name WebSphere Application Server
for z/OS V6 to see the table of contents.
2. Click Reference → Troubleshooter → Messages.
3. Then choose the tab according to the first few letters in your message code.
You can also search for the specific message or code with the search function at the top of
the window.
A.1.1 Specific Java component messages
We include the Java component messages that are prefixed by the BBOO0222I message in
Table A-3 on page 39 to provide a quick reference for your convenience. (Mgmt stands for
Management.)
38
Problem Determination Methodology for WebSphere on z/OS
Table A-3 BBOO0222I message components
Msg
Component
Msg
Component
Msg
Component
ACIN
ACWA
ADFS
Access Intent
Work Area
Mgmt File Service
Subsystem
Application Deployment
Mgmt Config Archive
Subsystem
Mgmt Connector Subsystem
Mgmt Process Discovery
CWSIY
CWSIZ
Y SIBus Mediation Handlers
Z SIBus Mediation
Framework
A Admin
B inter-bus messaging
engine
C SIBus Core SPI
D Admin
PMON
PMI, Tivoli Performance
Viewer
Performance Monitoring
Request Metrics
PME Edition Support
Process Mgmt and Spawning
Facility
Proxy
Scheduler
CWSJO
O SDO Repository
Component
Q MFP MQ interoperability
component
R SIBus
U Jetstream Message
Tracing
W WLM Classifier
S SIBus Web Services
SECG
SECJ
SESN
SIEG
SOAP
SRMC
ADMN
ADMR
ADMS
Mgmt Event Subsystem
Mgmt Command Framework
Mgmt Connector Subsystem
Mgmt Utilities
Mgmt Process Launching
Tool
Activity Service
Mgmt Repository
Mgmt Subsystem
SRVE
SSLC
WEBUI SecurityCenter
Security
Session and User Profiles
Example
SOAP Support
Service Reference
ManagerTransactions
Transactions
SSL Channel
ADMU
ADNT
APPR
ASYN
BBOJ
BBOM
BBOO
BBOS
BBOT
Mgmt Utilities
Adaptive Entity
Application Profile
Asynchronous Beans
EJB Container
Naming
Runtime, Web
Security
OTS and RRS
CWUDD
Web Services UDDI
Deployment & Removal
UDDI User Console
UDDI Mgmt Interface
UDDI Node Manager
UDDI Migration
UDDI Logging and Tracing
UDDI SOAP Interface
STFF
STUP
TCPC
TRAS
TUNE
UDAI
UDCF
UDDA
UDDM
Staff Support Service
Startup Beans
TCP Channel
Trace Facility
Perform Auto-Tuning Support
UDDI API
UDDI Configuration
UDDI Data Types
UDDI DOM
BBZW
BCDS
CWUDT
CHKP
WBI SF Install
Business Context Data
Service for Event
Infrastructure
Binding EJB References
Channel Framework
Event Infrastructure
Validation
PME Validation
UDDI Registry Transaction
Manager
CWUDU UDDI Utility Tools
CWUDV UDDI Value Set Tools
CWUDX Web Services JAXR
CWWCW W Validation
CWWDR R Data Replication Service
CWWSG G Web Service Gateway
DCSV
DCS
UDEJ
UDEX
UDIN
UDLC
UDPR
UDRS
UDSC
UDSP
UDUC
UDDI EJB Interface
UDDI Exceptions
UDDI Installation
UDDI Local API
UDDI Persistence
UDDI Logging
UDDI Security
UDDI SOAP Interface
UDDI User Console
CHKS
CHKW
CHKX
CMPN
CNTR
CONM
CSCP
CWRCB
CWSIA
SIB Validation
Validation
XD Validation
Compensation
EJB Container
Connection Manager
CScope Service
B Core Group Bridge
A Service Integration Bus
DSRA
DWCT
UDUT
UDUU
UTLS
WACS
WACT
WASX
WBIA
UDDI Utility Tools
UDDI UUID
Utilities
Activity Session Service
Activity Service
Non WSCP Scripting
Support for Business
Integration Adapters
Handler Framework
ADMA
ADMB
ADMC
ADMD
ADME
ADMF
ADMG
ADMK
ADML
BNDE
CHFW
CHKC
CWSJA
CWSJB
CWSJC
CWSJD
CWSJQ
CWSJR
CWSJU
CWSJW
CWSWS
CWUDG
CWUDM
CWUDN
CWUDQ
CWUDR
CWUDS
DYNA
EAAT
ECNS
ESOP
Resource Adapters
Dynamic Workload Mgmt
Client
Dynacache
Placeholder
Entity Change Notification
Service
State Observer Plug-in for
Event Infrastructure
PMRM
PMWC
PROC
PROX
SCHD
WHFW
Appendix A. Messages and codes
39
Msg
Component
Msg
Component
Msg
Component
CWSIB
CWSIC
CWSID
CWSIE
CWSIF
CWSIH
CWSII
CWSIJ
CWSIK
B SIBus Common
C Communications
D Admin
E SIBus Externals
F SIBus MFP
H Jetstream MatchSpace
I Security
J COmmunications
K SIBus Return Codes
GWIN
HMGR
HTPC
I18N
ILMC
INST
IVTL
J2CA
JSAS
Web Services Gateway
HA Manager
HTTP Channel
Internationalization Service
Instance Location Manager
Install
Installation Verification Tool
J2EE Connector
Security Association
WKSP
WKSQ
WLTC
WMSG
WSBB
WSCL
WSCP
WSEC
WSGW
Work Space
Workspace Query Utilities
Transaction Monitor
Messaging Service
WsByteBuffer
WebSphere Client
Non WSCP Scripting
Web Services Security
Web Services Gateway
CWSIL L PSB
CWSIM M SIBus Mediations
SIMediationSession Interface
N SIBus Mediations
CWSIN Framework
O SIBus Migration
CWSIO P Jetstream Message
CWSIP Processor
JSFG
JSPG
JSSL
LTXT
MIGR
MSGS
NMSV
OBPL
ODCF
jsf (bean class type)
Java Server Pages
ORB SSL Extensions
Localizable Text
Release-to-Release
Migration Tooling
JMS Server
Naming Service
ObjectPool
WSIF
Web Services Invocation
Framework
SOAP Channels
Web Services Security
Kerberos
Validation Manager
Implementation
Server Runtime
Web Services
CWSIQ
CWSIR
CWSIS
CWSIT
CWSIU
CWSIV
CWSIW
CWSIX
ORBX
PLGC
PLGN
On Demand ConFiguration
ORB Extensions
Plug-in Configuration
Generator
Transactions Plug-in
Processor
Persistence Manager
PMI
WTRN
WUDU
Q MQFap Channel
R SIBus Core
S MessageStore
T TRM
U Utilities
V SIBus Resource Adapter
W SIBus Mediations
X SIBus Mediations
PLPR
PMGR
PMI
WSSC
WSSK
WSVM
WSVR
WSWS
Transaction recovery
WebUI Deployment
Descriptor Utilities
WUPD Update Installer
WVER Product History Information
WWLM WLM Client
XMEM XMem Channel
A.1.2 Minor codes
A minor code is shown in the WebSphere for z/OS error log and SYSPRINT. It is a
hexadecimal value with the format C9C2nnnn, where:
C9C2
Identifies the code as WebSphere for z/OS
nnnn
Uniquely identifies the code
A minor code is often associated with an exception, as shown in Example A-1.
Example: A-1 Exception with minor code
Trace: 2004/10/10 13:37:59.801 01 t=8BD0F0 c=UNK key=S2 (00000004)
Description: Throw CORBA system exception
exception id: CORBA::INTERNAL
minor code: c9c2110f
from filename: ./bboosyse.cpp
at line: 719
Some of the minor code meanings are described in the WebSphere for z/OS Information
Center at:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp
Important: Error (minor) codes not listed in the WebSphere for z/OS Information Center
should be reported directly to the IBM Support Center.
40
Problem Determination Methodology for WebSphere on z/OS
A.1.3 Abends
Table A-4 shows the WebSphere for z/OS related abend codes.
Table A-4 WebSphere related abend codes
Abend code
Issuer
CC3
Daemon processing failure
DC3
Controller region processing failure
EC3
Servant region processing failure
Some reason codes are also passed along with these abend codes. They are described in
detail in InfoCenter. Search for “Abend (reason) codes”.
Table A-5 shows an example.
Table A-5 Example abend code and related reason code
Abend
code
Abend
reason
Explanation
Suggested action
CC3
000C0009
An exception occurred on the main thread
of execution, probably during initialization.
The address space is abended with this
code to cause the space to terminate.
Further information about the
exception should be found in
the joblog for the space and
also possibly in the error log.
If no explanation is given in the reason code, and no indication is found in any information
source, the problem should be reported to IBM.
A.2 System and component message table
Table A-6 shows the most common non-WebSphere for z/OS-related message prefixes and
where they come from.
Table A-6 System and component messages
Prefix
Product
Message structure
DSN
IBM DB2 UDB for
z/OS
DSNcnnnt
c: Subcomponent identifier
nnn: Unique numeric identifier
t: Type with I - information, A - immediate action, D - immediate decision, E - eventual
action
Example: DSNB209I
Source: DB2 UDB for z/OS Version 8 Messages and Codes, GC18-7422
DB2 Information Center topic “Messages and Codes,” available at:
http://publib.boulder.ibm.com/infocenter/dzichelp/index.jsp
EZA
EZB
EZD
EZY
EZZ
SNM
™
Communications
Server (TCP/IP)
pppnnnnt
ppp: Prefix
nnnn: Unique identifier
t: Type with A - immediate action, E - eventual action, D - immediate decision, I information
Example: EZZ0902I
Source: z/OS V1.6 Communications Server: IP Messages: Volume 1-4,
GC31-8783/4/5/6
Appendix A. Messages and codes
41
Prefix
Product
Message structure
ICH
Security Server
(RACF)
c: Identifies the RACF function, where:
ICHcnnt
0: SAF initialization
3: RACROUTE REQUEST=VERIFY macro
4: RACF processing
5: RACF initialization
7: RACF status
8: RACROUTE REQUEST=AUTH macro
9: RACROUTE REQUEST=DEFINE macro
nn: Message serial number
t: Type, where:
A - action; operator must perform a specific action.
D - decision; operator must choose an alternative.
E - eventual action required.
I - information.
W - Wait; processing stops.
Example: ICH500I
Source: z/OS V1R6.0 Security Server RACF Messages and Codes, SA22-7686
ISP
ISR
FLM
ISPF,
PDF
SCLM
pppannna
ppp: Prefix
a: Alphabetic character
nnn: Unique identifier
Example: ISPA001
Source: z/OS V1R6.0 ISPF Messages and Codes, SC34-4815
IKJ
TSO/E
pppccnnnt
ppp: Prefix
cc: System module prefix (in decimal)
nnn: Message serial number identifying the program that issued the message
t:- Type, where:
A - action; the terminal user must perform the action specified in the message text.
E - error; processing terminates.
I - information; no action is required.
Example: IKJ55112E
Source: z/OS V1R6.0 TSO/E Messages, SA22-7786
IRX
TSO REXX
processing
pppccnnt
ppp: Prefix
cc: System module prefix (in decimal)
nnn: Message serial number identifying the program that issued the message
t: Type, where:
E - error; processing terminates.
I - information; no action is required.
Example: IRX0042I
Source: z/OS V1R6.0 TSO/E Messages, SA22-7786
42
Problem Determination Methodology for WebSphere on z/OS
Prefix
Product
Message structure
IWM
Workload Manager
(WLM)
pppnnnt
ppp: Prefix
nnn: Message serial number
t: Type, where:
A - action by operator, D - decision by operator,
E - eventual action by operator, I - information for operator/programmer, S - severe
error, W - wait for operator action
Example: IWM003I
Source: z/OS V1R6.0 MVS System Messages, Vol 9 (IGF-IWM), SA22-7639
CEE
EDC
z/OS Language
Environment®
Runtime
C/C++ Runtime
pppnnnnt
ppp: Prefix
nnnn: Message serial number
t: Type, where:
I - informational message, W - warning message,
E - error message, S - severe error message,
C - critical error message
Example: CEE0252W
Source: z/OS V1R1 Language Environment Run-Time Messages, SA22-7566
GIM
SMP/E
pppnnnnnt
ppp: Prefix
nnnnn: Message serial number
t: Type, where:
I - informational, W - warning, E - error, S - severe, T - terminating
Example: GIM20101S
Source: z/OS V1R1 SMP/E Messages, Codes, and Diagnosis, GA22-7770
FDB
FOM
FSUM
IXG
UNIX System
Services (USS)
Debugger
USS Shell &
Utilities
System Logger
pppcnnnn
ppp: Prefix
c: Component identifier
nnnn: Unique identifier
Example: FOMC1013
Source: z/OS V1R6.0 UNIX System Services Messages and Codes, SA22-7807
pppnnnt
ppp: Prefix
nnnnn: Message serial number
t: Type, where:
I - informational message, E - recoverable error,
W - warning, S - serious error, T - terminating
Example: IXG004I
Source: z/OS V1R6.0 MVS System Messages, Vol 10 (IXC-IZP), SA22-7640
ATR
Resource
Recovery Services
(RRS)
pppnnnt
ppp: Prefix
nnnn: Message serial number
t: Type, where:
I - informational message, E - recoverable error,
W - warning, S - serious error, T - terminating
Example: ATR120I
Source: z/OS V1R6.0 MVS System Messages, Vol 3 (ASB-BPX), SA22-7633
Appendix A. Messages and codes
43
Prefix
Product
Message structure
IMW
HTTP Server
pppnnnnt
ppp: Prefix
nnnnn: Message serial number
t: Type, where:
I - informational message, E - recoverable error,
W - warning, S - serious error
Message ID ranges: Components:
IMW0001-IMW2000 - IMWHTTPD IMW2000-IMW2500 - Proxy Server
IMW3501-IMW3700 - CONSOLE IMW3701-IMW3999 - HTCounter
IMW4000-IMW5000 - HTIMAGE IMW5001-IMW6000 - HTADM
IMW6100-IMW6900 - SSL Security
Example: IMW0442E
Source: IBM HTTP Server Planning, Installing, and Using, SC34-4826
z/OS Internet Library, available at:
http://www.ibm.com/servers/eserver/zseries/zos/bkserv/
44
Problem Determination Methodology for WebSphere on z/OS
Index
A
abend 3, 41
code 41
system abend 15
alphaWorks 30–31
APAR 14, 18
Authorized Program Analysis Report 14, 18
API documentation 31
ARM 8
ASCII 19
B
BBO 38
binary 19
C
CC3 41
CEEDUMP 15
communication
with IBM 12
Communications Server (TCP/IP) 41
component
ID 16
component ID 16
compress large files 19
compressing data 19
control region
failure code 41
CPU
problem 14
CTRACE 15
D
daemon
failure code 41
job log 15
data set
compress 19
format 19
tersed 19
data source 9
DB2 41
Administrator 9
messages 41
DC3 41
debug
hints and tips 29
production environment 7
deployment manager 15
developer information 30
DeveloperWorks 14, 30
diagnose
© Copyright IBM Corp. 2006. All rights reserved.
system 2
Draft publications 29
dump
data set 15
transfer 19
E
EC3 41
education 34
Web site 34
e-fixes 6
EJB
references 9
error
log 8, 15, 40
message 3, 15, 43
state 3
eServer zSeries support 31–32
F
FAQs 29
flashes 29
FTP
data transfer 21
naming conventions 21
to IBM 18
G
gather background information 14
H
hardware specifications 6
HFS
directory 20
environment 8
hints and tips 29–31
HTTP
Server
message 44
I
IBM
contact 21
Link 14
Support 12–13, 16
guide 33
Incident/Support Case 17
Information Center 13
Internet
helpful pages 31
ISPF
message 42
45
J
J2EE
server
create 8
Java
messages 38
JCL 19
job log 15, 19
JOBLIB 20
L
Language Environment 43
LDAP 9
LNKLST 20
logger 8, 43
LookAt messages 32
loop 14
M
message 32
prefix 37, 41
returned 3
minor code 40
MQ Administrator 9
Mustgather 14
N
naming
conventions 21
network
topology 6
node agent
job log 15
P
Packlib 19
Parallel Sysplex 8
PD/PSI 2, 6–7
What PD/PSI is 2
PDF 34, 42
performance
problems 2, 18
PKZip 20
PMR 13, 16–17, 21
Investigating a PMR 17
problem
PMR 12–13, 17
Problem Management Record 12, 16
product
documentation 18
information 13
support resources 13
production environment 28
PTF 14
R
RACF 8, 15, 42
46
Redbooks 29
Redbooks Web site
Contact us viii
Release Notes 13
Resolution Team 17–18
resource
Recovery Services 43
response time 3
RETAIN
REmote Technical Assistance Information Network
17
RRS 8, 43
S
scenario 13, 17
SCLM 42
SDSF 19
security
administrator 9
server
region 15, 41
severity level 16
skills 7–9, 12, 34
SMP/E
messages 43
Software
defect 18
Maintenance 18
Problem Report 18
support
before contacting IBM 13
case 17
line 18
pages 24
symptoms 2, 4, 7, 13
SYSLOG 15
SYSPRINT 20, 40
system
administration 6
log 15
programming skills 8
T
tar command 20
TCP/IP
commands 9
messages 41
skills 8–9
Techdocs 29
Technotes 14, 29
test
environment 4
trace
component 15
exchange 19
skills 8
TRSMAIN 19, 21
TSO REXX processing 42
TSO/E 42
Problem Determination Methodology for WebSphere on z/OS
tuning 2, 29, 39
U
UNIX
System Services 8, 20, 43
USS
messages 43
skills 9
W
Wait 42–43
warning message 43
Web
helpful pages 31
WebSphere
Proof of Concept for z/OS 28
support structure 12
whitepapers 14, 29, 31
Winzip 20
WLM 8
messages 43
wrong output 3
WSAdmin 8
Z
z/OS Internet library 34
ZIP file 20
Index
47
48
Problem Determination Methodology for WebSphere on z/OS
Back cover
Problem Determination
Methodology for
WebSphere on z/OS
Analyzing a problem,
finding a cause and
solution
Exchanging
information with IBM
Support
Resources available
for problem solving
When a business experiences a problem with their IT systems the
impact can be devastating. It therefore becomes critical to have
the information, tools, and support available to help identify the
type, source, cause, and solution.
®
Redpaper
INTERNATIONAL
TECHNICAL
SUPPORT
ORGANIZATION
This IBM Redpaper describes the general problem determination
methodology and how it applies to a WebSphere Application
Server for z/OS environment.
We review the skills necessary for WebSphere for z/OS problem
determination and provide information to assist you when
communicating with IBM support.
We also provide you with additional information and resources all
aimed at helping you find solutions to your IT systems’ problems.
BUILDING TECHNICAL
INFORMATION BASED ON
PRACTICAL EXPERIENCE
IBM Redbooks are developed
by the IBM International
Technical Support
Organization. Experts from
IBM, Customers and Partners
from around the world create
timely technical information
based on realistic scenarios.
Specific recommendations
are provided to help you
implement IT solutions more
effectively in your
environment.
For more information:
ibm.com/redbooks