Debugging UNIX System Services,
Lotus Domino, Novell Network Services,
and other Applications on OS/390
Paul Rogers, Manfred Hauff, Juergen Hoerner, Wilhelm Michel
International Technical Support Organization
www.redbooks.ibm.com
SG24-5613-00
International Technical Support Organization
Debugging UNIX System Services,
Lotus Domino, Novell Network Services,
and other Applications on OS/390
November 1999
SG24-5613-00
Take Note!
Before using this information and the product it supports, be sure to read the general information in Appendix B,
“Special notices” on page 293.
First Edition (November 1999)
This edition applies to all versions of OS/390 (5647-A01) and to all subsequent releases and modifications until
otherwise indicated in new editions.
Comments may be addressed to:
IBM Corporation, International Technical Support Organization
Dept. HYJ Mail Station P099
522 South Road
Poughkeepsie, NY 12601-5400
When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any
way it believes appropriate without incurring any obligation to you.
© Copyright International Business Machines Corporation 1999. All rights reserved.
Note to U.S Government Users - Documentation related to restricted rights - Use, duplication or disclosure is subject to restrictions
set forth in GSA ADP Schedule Contract with IBM Corp.
Contents
Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
The team that wrote this redbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
Comments welcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvi
Chapter 1. OS/390 UNIX System Services . . . . . . . . . . . . . . . . . . . . . . . . . . .1
1.1 Introduction to OS/390 UNIX System Services . . . . . . . . . . . . . . . . . . . . . .1
1.1.1 OS/390 UNIX support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1
1.1.2 OS/390 UNIX software interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . .2
1.1.3 Hardware considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4
1.1.4 Workstation connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5
1.1.5 Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5
1.1.6 Application programmers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6
1.1.7 Administrative tasks using the ISPF shell . . . . . . . . . . . . . . . . . . . . . .6
1.2 Installation concepts for OS/390 UNIX System Services . . . . . . . . . . . . . . .7
1.2.1 Startup definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
1.2.2 Levels of operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
1.2.3 Dynamic activation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
1.2.4 Job Wait Time (JWT). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
1.2.5 Started procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
1.3 Installation highlights of OS/390 UNIX . . . . . . . . . . . . . . . . . . . . . . . . . . .11
1.3.1 How UNIX System Services has changed . . . . . . . . . . . . . . . . . . . . .12
1.3.2 Implementing SMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
1.3.3 Implementing a security system for UNIX System Services . . . . . . . .15
1.3.4 Allocating and restoring the HFS files . . . . . . . . . . . . . . . . . . . . . . . .16
1.3.5 Performing HFS post-restore activities . . . . . . . . . . . . . . . . . . . . . . .16
1.4 Debugging OS/390 UNIX System Services . . . . . . . . . . . . . . . . . . . . . . . .17
1.4.1 Hints and tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17
1.4.2 Common problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28
1.5 UNIX System Services dump processing . . . . . . . . . . . . . . . . . . . . . . . . .30
1.5.1 How to provide a dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30
Chapter 2. Lotus Domino for S/390—troubleshooting/hints/tips . . . . .
2.1 OS/390 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.1 Domcon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.2 IEFUSI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.3 OS/390 commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.4 SYSLOG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.5 Dump options/IPCS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.6 CEEDumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 UNIX System Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.1 nsd.sh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.2 Return codes and how to interpret a USS message . . . . . . . . . .
2.2.3 Useful UNIX commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Lotus Domino . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.1 Startup script and Language Environment (LE) run-time options .
2.3.2 How to activate the Java environment . . . . . . . . . . . . . . . . . . . . .
2.3.3 Common parameters in the notes.ini . . . . . . . . . . . . . . . . . . . . . .
© Copyright IBM Corp. 1999
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.33
.34
.34
.36
.38
.40
.41
.45
.45
.45
.49
.51
.53
.53
.54
.56
iii
iv
2.3.4 Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.5 The Domino console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.6 Programs (compact, update, updall, and fixup) . . . . . . . . . . . . . . .
2.4 Hints and tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.1 How to edit the notes.ini file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.2 Cleaning up server resources . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.3 The HTTP task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.4 The POP3 task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.5 The SMTP task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.6 How mail routes in a Domino system . . . . . . . . . . . . . . . . . . . . . . .
2.4.7 Transaction logging in Domino R5 . . . . . . . . . . . . . . . . . . . . . . . . .
2.5 Troubleshooting documents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5.1 HTTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5.2 SMTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5.3 POP3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5.4 Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5.5 Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5.6 Calendaring/Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5.7 Common info/data needed by support personnel in case of errors .
2.5.8 More troubleshooting information . . . . . . . . . . . . . . . . . . . . . . . . . .
2.6 Information sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.6.1 Required APARs and PTFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.6.2 Web sites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.6.3 Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.6.4 Other information sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. 58
. 61
. 75
. 83
. 83
. 85
. 86
. 90
. 96
. 98
100
103
103
105
105
106
107
111
114
115
118
118
119
122
123
Chapter 3. Novell Network Services . . . . . .
3.1 Benefits . . . . . . . . . . . . . . . . . . . . . . . . . .
3.1.1 NDS and NDS practices. . . . . . . . . .
3.1.2 Uses of NNS for OS/390 . . . . . . . . .
3.2 Introduction . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 The structure of NNS . . . . . . . . . . . .
3.2.2 Features of NNS . . . . . . . . . . . . . . .
3.2.3 NNS functions . . . . . . . . . . . . . . . . .
3.2.4 Licensing system . . . . . . . . . . . . . . .
3.2.5 NNS networking . . . . . . . . . . . . . . . .
3.3 Installation and customization . . . . . . . . .
3.3.1 Software requirements . . . . . . . . . . .
3.3.2 Hardware requirements . . . . . . . . . .
3.4 Customization . . . . . . . . . . . . . . . . . . . . .
3.4.1 UNIX System Services environment
3.4.2 Environment variables . . . . . . . . . . .
3.4.3 Network configuration . . . . . . . . . . .
3.5 Installing NNS for OS/390 . . . . . . . . . . . .
3.5.1 Before you get started . . . . . . . . . . .
3.5.2 Server installation . . . . . . . . . . . . . .
3.5.3 Installing NetWare client software . .
3.6 Serviceability and debugging . . . . . . . . . .
3.6.1 Hints and tips. . . . . . . . . . . . . . . . . .
3.6.2 Tuning the NNS server . . . . . . . . . .
3.6.3 Debugging . . . . . . . . . . . . . . . . . . . .
3.6.4 Known problems . . . . . . . . . . . . . . .
3.6.5 Helpful commands . . . . . . . . . . . . . .
125
125
125
126
127
127
128
129
129
129
130
131
131
131
132
134
135
146
147
148
155
162
162
162
164
170
174
Debugging UNIX Applications on OS/390
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
3.7 Getting NetWare client software
3.8 Emergency recovery . . . . . . . . .
3.9 Installation Worksheets . . . . . . .
3.10 Documentation . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
.
.
.
.
.
.
.
.
..
..
..
..
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
.
.
.
.
.
.
.
.
.180
.181
.183
.184
Chapter 4. DCE/DFS . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.1 Distributed Computing Environment (DCE) overview.
4.1.1 DCE base service . . . . . . . . . . . . . . . . . . . . . . .
4.1.2 Configuring a cell . . . . . . . . . . . . . . . . . . . . . . .
4.1.3 DCE start, stop, and debug hints . . . . . . . . . . . .
4.1.4 How to produce a dump . . . . . . . . . . . . . . . . . .
4.1.5 How to get a trace output from DCEKERN . . . .
4.1.6 DCE components . . . . . . . . . . . . . . . . . . . . . . .
4.2 Distributed File Service (DFS) . . . . . . . . . . . . . . . . . .
4.2.1 DFS installation . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.2 Starting and stopping DFS . . . . . . . . . . . . . . . .
4.2.3 Daemon configuration file . . . . . . . . . . . . . . . . .
4.2.4 DFS daemon environment variable files . . . . . .
4.3 Information sources . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.1 OS/390 Distributed File Service publications . . .
4.3.2 OS/390 DCE publications . . . . . . . . . . . . . . . . .
4.3.3 Security Server publications . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.185
.185
.185
.190
.192
.195
.195
.196
.199
.200
.202
.206
.208
.208
.208
.209
.210
Chapter 5. Web server—troubleshooting/hints/tips . . . . . . . . . . . . . . .
5.1 Controlling your server using the console . . . . . . . . . . . . . . . . . . . . . .
5.2 Tracing and logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2.1 Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2.2 Language Environment tracing . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2.3 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3 Common error messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.1 02AF (address space dirty) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.2 401 errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.3 Failed to load security module: . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.4 "Forbidden" or "Unauthorized" error message . . . . . . . . . . . . . . .
5.3.5 IMW0161E with error 500 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.6 Error using message catalog . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.7 Setup.sh error messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.8 ID and password prompt when accessing at a nonstandard port .
5.3.9 IMW messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.10 How to read a message. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4 Tuning hints and tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4.1 Performance hints when running Web server under USS shell . .
5.4.2 Tuning your Web server for better performance . . . . . . . . . . . . .
5.5 Tips for enabling WebSphere Application Server under OS/390 . . . . .
5.6 Information sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.6.1 Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.7 Useful Web sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.7.1 Informational APARs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.211
.211
.212
.212
.213
.213
.216
.216
.218
.219
.219
.219
.220
.220
.220
.221
.222
.223
.223
.225
.233
.236
.236
.237
.238
Chapter 6. Network File System (NFS). . . . . . . . .
6.1 Setting up the OS/390 NFS server environment
6.1.1 Environment examples . . . . . . . . . . . . . . .
6.1.2 Enable TCP/IP and the portmapper . . . . . .
6.1.3 RACF profiles . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.239
.239
.239
.241
.242
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
v
6.1.4 Customize the OS/390 NFS server startup procedure
6.1.5 Security using the NFS server . . . . . . . . . . . . . . . . . .
6.1.6 BPXPRMxx parmlib member . . . . . . . . . . . . . . . . . . .
6.1.7 Starting the NFS server . . . . . . . . . . . . . . . . . . . . . . .
6.1.8 Mount . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.1.9 HFS data set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.1.10 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.1.11 Supported clients for the OS/390 NFS server. . . . . .
6.1.12 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2 Setting up the OS/390 NFS client environment . . . . . . . . .
6.2.1 UNIX System Services. . . . . . . . . . . . . . . . . . . . . . . .
6.2.2 RACF profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2.3 Starting the NFS client . . . . . . . . . . . . . . . . . . . . . . . .
6.2.4 MOUNT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2.5 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3 Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
244
244
245
245
246
247
248
249
249
257
257
258
258
258
259
263
Chapter 7. Infoprint Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.1 What the Infoprint Server does for you . . . . . . . . . . . . . . . . . . .
7.2 Printing OS/390 UNIX System Services data on AFP printers . .
7.3 Components of the OS/390 Infoprint Server . . . . . . . . . . . . . . .
7.3.1 OS/390 Print Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.3.2 NetSpool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.3.3 IP PrintWay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.3.4 UNIX System Services commands . . . . . . . . . . . . . . . . . . .
7.4 Relevant APARs for OS/390 Infoprint Server or Print Interface .
7.4.1 II11165 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.4.2 II11659 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.4.3 OW39059 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.5 Information Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.5.1 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.5.2 Web sites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
265
265
266
266
266
267
267
268
270
270
271
272
272
272
273
Chapter 8. Language Environment for OS/390 . . . .
8.1 LE run-time options . . . . . . . . . . . . . . . . . . . . . . .
8.1.1 LE run-time options . . . . . . . . . . . . . . . . . . .
8.1.2 Setting LE run-time options . . . . . . . . . . . . .
8.1.3 Some LE run-time options of interest . . . . . .
8.2 When are which libraries used . . . . . . . . . . . . . . .
8.3 Common Execution Library (CEL) . . . . . . . . . . . .
8.3.1 Set of common functions and routines of LE
8.3.2 Important modules to remember . . . . . . . . .
8.3.3 Messages and abends . . . . . . . . . . . . . . . . .
8.3.4 Run-time options . . . . . . . . . . . . . . . . . . . . .
8.4 LE hands on debug . . . . . . . . . . . . . . . . . . . . . . .
8.4.1 CEEDUMP debugging . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
..
..
..
..
..
..
..
..
..
275
275
275
276
276
277
277
277
277
278
280
284
284
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
..
..
..
..
..
..
..
..
..
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Appendix A. Tracing within TCP/IP. . . . . . . . . . . . . . . . . . . . . .
A.1 The Component Trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.1.1 Using the Component Trace with internal buffer . . . . . . .
A.1.2 Using the Component Trace with external writer . . . . . . .
A.2 The Packet Trace. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
vi
Debugging UNIX Applications on OS/390
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..
..
..
..
..
..
..
..
..
..
..
..
......
......
......
......
......
. . . . . 289
. . . . . 289
. . . . . 289
. . . . . 291
. . . . . 292
Appendix B. Special notices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
Appendix C. Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.1 International Technical Support Organization publications . . . . . . . . . . . . . .
C.2 Redbooks on CD-ROMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.3 Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
297
297
298
298
How to get ITSO Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .301
IBM Redbook fax order form. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .303
ITSO Redbook evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .305
vii
viii
Debugging UNIX Applications on OS/390
Figures
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
© Copyright IBM Corp. 1999
USS overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
BPXPRMxx definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Downloading domcon.nsf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Display of active exits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Display of IEFUSI exit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Display all UNIX System Services address spaces . . . . . . . . . . . . . . . . . . . . . 38
Display currently mounted file systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Display the current settings of BPXPRMxx parameters . . . . . . . . . . . . . . . . . . 39
Display thread information for a process ID . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Sample IEASLP01 member . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Display of all active slips on a system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Partial output of nsd.sh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
List of all available parameters for the nsd.sh script . . . . . . . . . . . . . . . . . . . . 47
Terminating the server with option nsd.sh -kill . . . . . . . . . . . . . . . . . . . . . . . . . 48
Error message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Sample output of the UNIX command df -P . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Sample output of UNIX command env . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Sample output of the UNIX command ps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Java error messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Enable http logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
domlog.nsf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
cconsole . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Set secure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Sample notes.ini - Part 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Sample notes.ini - Part 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Display of semaphores and shared memory resources . . . . . . . . . . . . . . . . . . 86
Activating HTTP - Internet protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Activating HTTP - server configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Activating HTTP - Internet password . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Activating HTTP - password prompt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Activating HTTP - accessing Domino databases using an Internet browser . . 89
TSO NETSTAT command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Activating POP3 - server configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Defining a POP3 user - basic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Defining a POP3 user - mail. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Creating mail database for a POP3 user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
ACL of the POP3 user’s mail database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Netscape mail server properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Netscape Communicator 4.6 - Messenger view . . . . . . . . . . . . . . . . . . . . . . . 94
Available server tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Integration of POP3 in a Notes mail environment . . . . . . . . . . . . . . . . . . . . . . 96
Enable the SMTP listener task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Enable SMTP inbound and outbound ports . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Mail routing in a Domino environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Configuring transaction logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
Load router with variable DebugRouter set to 3. . . . . . . . . . . . . . . . . . . . . . . 107
Agent log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
readme.nsf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
help5_admin.nsf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
APARs and PTFs needed for Domino S390 . . . . . . . . . . . . . . . . . . . . . . . . . 119
ix
51. UNIX System Services (USS) site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
52. Domino R5 home page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
53. Domino tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
54. Lotus Domino - Tools for Developers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
55. UNIX System Services library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
56. Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
57. Lotus Domino R5 help documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
58. Lotus Domino R5 documentation using an Internet/intranet browser . . . . . . . 124
59. Structure of NNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
60. NNS networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .130
61. IFAPRDxx parmlib member . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
62. NetWare/IP domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .138
63. IOCDS definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
64. VTAM definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
65. OSA2 configuration overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
66. OSA-2 CHPID selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .142
67. OSA CHPID selection continued. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
68. OSA configuration list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
69. OSA-2 configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
70. High Performance Data Transfer Multipath Channel settings . . . . . . . . . . . . 143
71. Syntax of the nwdiscover command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .144
72. Using nwdiscover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
73. drouter information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
74. Successful nwserver command to start the server . . . . . . . . . . . . . . . . . . . . . 152
75. Continuation of the syslog in Figure 74 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
76. Messages because of the dsinstall command . . . . . . . . . . . . . . . . . . . . . . . . 153
77. Directory service installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .154
78. Right mouse click the Network Neighbor icon. . . . . . . . . . . . . . . . . . . . . . . . .156
79. Novell NetWare login panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
80. Novell IntranetWare Client and Connection panels . . . . . . . . . . . . . . . . . . . . 157
81. Novell IntranetWare Client Server list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
82. Select the server from the list and click OK to log in. . . . . . . . . . . . . . . . . . . . 158
83. Login result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
84. Check the IntranetWare Connections using Network Neighbor . . . . . . . . . . . 159
85. Installation environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .159
86. Server identification showing the connection type IPX . . . . . . . . . . . . . . . . . . 160
87. Server error log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
88. Environment of the ADMIN user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
89. LANalyzer dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
90. LANalyzer filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
91. LANalyzer viewing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
92. Capture an SVC dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
93. GTF parms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
94. Messages in the syslog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
95. NWSERVER example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
96. Example of the nwserver command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
97. Example issue of a nwserver command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
98. Result of the failing SET OMVS command . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
99. Unsuccessful LOGIN Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
100.OMVS, A=ALL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
101.D NET,TRL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .175
102.D NET,E,ID=INRT1100 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .176
103.Display unit command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
x
Debugging UNIX Applications on OS/390
104.The Novell server is not up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
105.The Novell server is coming up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
106.The Novell server is up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
107.Ipxinfo example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
108.Download the client code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
109.OS/390 DCE architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
110.DCE cell. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
111.DCE daemons in the cell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
112.MODIFY command to display server configuration . . . . . . . . . . . . . . . . . . . .
113.MODIFY command to display server statistics . . . . . . . . . . . . . . . . . . . . . . .
114.Turn on and off very verbose tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
115.Restarting the server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
116.MODIFY command to display all parameters . . . . . . . . . . . . . . . . . . . . . . . .
117.Controlling the trace level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
118.WebSphere Application manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
119.Command output: D TCPIP,TCPV34,NET,CACH . . . . . . . . . . . . . . . . . . . . .
120.IBM WebSphere troubleshooter for OS/390 . . . . . . . . . . . . . . . . . . . . . . . . .
121.NFS network configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
122.NFS 2-tier versus 3-tier overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
123.Modifying the hlq.ETC.RPC file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
124.Port definitions for the portmapper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
125.RACF primary option menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
126.RACF user profile services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
127.RACF add an NFS user. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
128.Set operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
129.Select to set up the OMVS segment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
130.Add OMVS parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
131.Add OMVS parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
132.Example of a filesystype statement in the BPXPRMxx member . . . . . . . . . .
133.Server messages in the log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
134.Example to mount a file system within OS/390 UNIX . . . . . . . . . . . . . . . . . .
135.Select option M . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
136.Define the data set type HFS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
137.NFS server debug9 trace. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
138.Output of a ping command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
139.Example of a showattr command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
140.Showmount -e <host> example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
141.RPCINFO -p <host> example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
142.NFS Internet page http://www.s390.ibm.com/nfs/faq/faqindex.html . . . . . . .
143.Environmental Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
144.Problem Determination, Q&A’s Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
145.Technical Support Site. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
146.NFS client BPXPRMxx parmlib member entry . . . . . . . . . . . . . . . . . . . . . . .
147.Client messages in the log. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
148.Mount syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
149.Example of a mount. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
150.NFS client trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
151.OS/390 Infoprint Server capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
152.USS data to AFP printers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
153.Partial traceback information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
154.Example dump output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
177
178
178
178
181
186
191
191
211
211
211
211
212
213
215
228
238
240
241
241
242
242
243
243
243
243
243
244
245
246
246
247
248
250
251
251
251
252
254
255
256
257
257
258
259
259
260
265
266
284
288
xi
xii
Debugging UNIX Applications on OS/390
Tables
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
© Copyright IBM Corp. 1999
Sample jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Parmlib members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Jobs for allocating and restoring HFS files. . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Shell initialization files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Files associated with /etc/init . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
OMVSDATA keywords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Report type levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Filtering keywords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
|USS errno . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
List of Return Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
USS errno2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Reason Code example 006D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
List of Reason Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Useful UNIX commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Common notes.ini parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
cconsole commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Command line switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Comparison of the three styles of compaction . . . . . . . . . . . . . . . . . . . . . . . . . 77
Short summary of options for the Compact utility . . . . . . . . . . . . . . . . . . . . . . 78
Compare Update and Updall characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Updall options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Fixup options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
notes.ini parameters for transaction logging . . . . . . . . . . . . . . . . . . . . . . . . . 101
Useful commands to control the router task . . . . . . . . . . . . . . . . . . . . . . . . . 107
Doppelganger options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Server prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
DNS source information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
How to configure three networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
nwdiscover parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
nwdiscover examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
DCE Dump commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
DFS procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
The MODIFY DFS command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Daemon’s status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Message ranges for the server components . . . . . . . . . . . . . . . . . . . . . . . . . 221
USS errno . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
Return codes listed by value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
The Reason code is made up of 4 bytes in the following format . . . . . . . . . . 261
NFS client Reason codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Reason codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
Common IPCS commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
LE-related IPCS commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
xiii
xiv
Debugging UNIX Applications on OS/390
Preface
In this redbook we discuss troubleshooting and debugging terms in the UNIX
System Services (USS) area. We provide a broad understanding of what to do in
case of problems with USS and a number of applications.
Chapter 2, which discusses troubleshooting Lotus Domino for S/390, is of
particular value because of the rapidly increasing interest in this product in a
UNIX System Services environment. Information about what to do with server
problems is described here.
Chapter 3, which describes Novell Network Services (NNS), also contains much
useful information. In it we discuss the most common problems you are likely to
encounter and provide helpful information about the NetWare File System,
WebSphere Application Server, DCE/DFS, and more.
This redbook will be useful for IT specialists, system programmers, application
developers, and administrators in the product areas that are covered here. It will
help to pinpoint problems with logs, traces, dumps, and other troubleshooting
issues.
The team that wrote this redbook
This redbook was produced by a team of specialists from around the world
working at the International Technical Support Organization, Poughkeepsie
Center.
Paul Rogers is a senior member of the International Technical Support
Organization, Poughkeepsie Center. He writes extensively and teaches IBM
classes worldwide on all areas of OS/390.
Manfred Hauff is an I/T specialist working for IBM PSS Mainz, Germany. He has
10 years of experience in supporting IBM OS/390 Software and has worked for
one year in the Lotus Domino S/390 EMEA second-level support. His areas of
expertise include OS/390 SCP, TSO and ISPF, UNIX System Services,
e-business, and Web applications. He has written extensively on Lotus Domino
S/390 and Webserver.
Juergen Hoerner is an I/T specialist working for IBM PSS Mainz, Germany. He
has seven years of experience in supporting IBM OS/390 Software. His areas of
expertise include OS/390 DFP, DFSMS, NFS, UNIX System Services, and the
LAN Server Consolidation products LFS and LANRES. He has written extensively
on Network File Service and Novell Network Services.
Wilhelm Michel is an I/T specialist working for IBM PSS Mainz, Germany. He has
10 years of experience in supporting IBM OS/390 Software and has worked for
more than one year in the Lotus Domino S/390 EMEA 2nd level Support. His
areas of expertise include OS/390 Publishing Software, UNIX System Services,
e-business, Internet applications, and Windows NT. He has written extensively on
Lotus Domino S/390, Webserver, and Print Server.
© Copyright IBM Corp. 1999
xv
Thanks especially to the following people for helping to write this redbook:
Peter Hilger, IBM PSS Mainz, Germany
Matthias Korn, IBM PSS Mainz, Germany
Stefan Bretz, IBM PSS Mainz, Germany
And, in addition, to the following people for their contributions on this project:
Bob Haimowitz, IBM Raleigh
Rich Conway, IBM Poughkeepsie
Anthony Grech, IBM Poughkeepsie
Brian Sullivan, IBM Poughkeepsie
Bill White, IBM Poughkeepsie
Jon Entwistle, IBM Poughkeepsie
Johannes Schneider, IBM Mainz, Germany
Andreas Horn, IBM Mainz, Germany
Comments welcome
Your comments are important to us!
We want our redbooks to be as helpful as possible. Please send us your
comments about this or other redbooks in one of the following ways:
• Fax the evaluation form found in “ITSO Redbook evaluation” on page 305 to
the fax number shown on the form.
• Use the on-line evaluation form found at http://www.redbooks.ibm.com/
xvi
Debugging UNIX Applications on OS/390
Chapter 1. OS/390 UNIX System Services
The following sections contain information about common items, troubleshooting,
debugging issues, and hints and tips about UNIX System Services (OS/390
UNIX).
1.1 Introduction to OS/390 UNIX System Services
The term OS/390 UNIX System Services and its abbreviation OS/390 UNIX are
new names for what was previously known as OpenEdition in earlier levels of
OS/390 and MVS/ESA.
OS/390 UNIX System Services provides support for two open systems interfaces
on the OS/390 operating system:
• An application program interface (API)
• An interactive shell interface
With the APIs, programs can run in any environment, including in batch jobs, in
jobs submitted by TSO/E interactive users and in most other started tasks—or in
any other MVS application task environment. The programs can request:
• Only MVS services
• Only OS/390 UNIX System Services
• Both MVS and OS/390 UNIX System Services
The shell interface is an execution environment analogous to TSO/E, with a
programming language of shell commands analogous to the Restructured
Extended Executor (REXX) language. The shell work consists of:
• Programs run interactively by shell users
• Shell commands and scripts run interactively by shell users
• Shell commands and scripts run as batch jobs
1.1.1 OS/390 UNIX support
OS/390 UNIX responds to requests from programs and the OS/390 shell and is
made up of system and application services.
UNIX System Services provides:
• XPG4 UNIX 1995 conformance
• Assembler callable services
• TSO/E commands to manage the file system
• ISPF shell environment
UNIX System Services Application Services interprets commands from users or
from programs, called shell scripts, and requests MVS services in response to the
commands. It also provides the dbx debugger to help an application programmer
debug source programs written in C. UNIX System Services Application Services
provides:
• A TSO/E command to enter the shell environment
© Copyright IBM Corp. 1999
1
• A shell environment for developing and running applications
• Utilities to administer and develop in a UNIX environment
• The dbx debugger
• Support for socket applications
• rlogin (remote login) and inetd functions
• Direct telnet based on TCP/IP
• Support for full-screen applications (curses support)
• Communications Server login monitor support
UNIX System Services Application Services also contains the code that was
provided in the optional Shell and Utilities and the Debugger features prior to
OS/390.
1.1.2 OS/390 UNIX software interaction
OS/390 UNIX interacts with other products and operating system services,
including:
• Workload Manager (WLM), to create address spaces
• System Management Facilities (SMF), for accounting
• C/C++ Compiler, to compile programs
• Language Environment, to execute the shell and utilities or any other
XPG4-compliant shell application
• Data Facility Storage Management Subsystem/MVS (DFSMS/MVS), to
support Hierarchical File System (HFS) data sets
• Security Server (RACF), to control access
• Resource Measurement Facility (RMF), to monitor and report on the use of
resources
• System Display and Search Facility (SDSF), to monitor/control address
spaces and printing on JES2 systems
• Time Sharing Option Extensions (TSO/E), to support the ISPF shell and
OMVS environments
• Transmission Control Program/Internet Protocol (TCP/IP), to use telnet and
rlogin.
• Interactive System Productivity Facility (ISPF), to use OBROWSE, OEDIT and
the ISPF shell
• BookManager READ/MVS, to use the OHELP on-line help facility
Figure 1 on page 3 shows how OS/390 UNIX, the shell interface, and the API
relate to the rest of the OS/390 operating system:
2
Debugging UNIX Applications on OS/390
Figure 1. USS overview
1.1.2.1 Workload Manager (WLM)
The WLM transaction initiators provide address spaces when programs issue the
fork() or spawn() C function or OS/390 callable services.
1.1.2.2 System Management Facility (SMF)
SMF collects data for accounting. SMF job and job step accounting records
identify processes by user, process, group, and session identifiers. Fields in
these records also provide information on resources used by the process. SMF
File system records describe file system events such as file open, file close, and
the file system mount, unmount, quiesce and unquiesce.
1.1.2.3 C/C++ Compiler
The C/C++ Compiler provided with OS/390 is required to compile C code using
the c89 command or to compile C++ code using the cxx command.
1.1.2.4 Language Environment (LE)
Language Environment (LE) provided with OS/390 is required to run a shell
command or utility. User-provided application programs written in C/C++ require
the C/C++ run-time library provided with LE.
1.1.2.5 Data Facility Storage Management Subsystem/MVS - DFSMS
DFSMS/MVS is required to support the HFS data set type. HFS data sets must
reside on DFSMS/MVS managed DASD.
1.1.2.6 Security Server
RACF or an equivalent security product manages system and data security by
verifying that a user is allowed access to a resource.
A user is identified by a user ID, which is kept in the RACF user profile and a GID,
which is kept in the RACF group profile.
OS/390 UNIX System Services
3
1.1.2.7 Resource Measurement Facility (RMF)
RMF collects data used to describe OS/390 UNIX performance. RMF reports
support an address space type of OMVS for address spaces created by fork or
spawn callable services and support two swap reason codes.
When an installation specifies an OMVS subsystem type in the IEAICSxx parmlib
member or in the WLM service policy, RMF shows the activity of forked address
spaces separately in the RMF Workload Activity report.
RMF monitors the use of resources in an OMVS Kernel Activity report.
1.1.2.8 System Display and Search Facility (SDSF)
Shell users can enter TSO/E sessions and use SDSF to:
• Monitor printing
• Monitor and control a batch job
• Monitor and control forked address spaces
• Find out which users are logged on to TSO/E sessions
1.1.2.9 Time Sharing Options/Extended (TSO/E)
One way to enter the shell environment is by using TSO/E. A user logs on to a
TSO/E session and enters the TSO/E OMVS command.
The OS/390 environment has other TSO/E commands, for example, to logically
mount and unmount file systems, create directories in a file system, and copy
files to and from MVS data sets. Users can switch from the shell to their TSO/E
session, enter commands or do editing, and switch back to the shell.
1.1.2.10 Transmission Control Protocol/Internet Protocol (TCP/IP)
Another way to enter the shell environment is by using rlogin or telnet from a
workstation in the TCP/IP network.
User-written socket applications can use TCP/IP as a communication vehicle.
Both client and server socket applications can use the socket interface to
communicate over the Internet (AF_INET) and between other socket applications
by using local sockets (AF_UNIX). An assembler interface is also provided for
those applications that do not use the C/C++ run-time library.
1.1.2.11 Interactive System Productivity Facility (ISPF)
Users of ISPF can use the ISPF shell environment to create, edit, browse and
perform other functions for files and directories in the HFS.
1.1.2.12 BookManager READ/MVS
An on-line help facility is provided. You can invoke this facility with the TSO/E
OHELP command and view on-line publications in BookManager format.
1.1.3 Hardware considerations
You can use the same hardware as the other components of the OS/390 system.
Use the same network connections that TSO/E uses and the processor and
network connections that JES uses.
4
Debugging UNIX Applications on OS/390
Additional hardware considerations are:
• If you want to use rlogin the connections are different from those for TSO/E
users.
• The optional Suppression on Protection feature, if not present, negates certain
functions such as mmap() and fork() copy-on-write.
• For Communications Server support, a LAN or channel-attached RISC
System/6000 is required.
• For improved TCP/IP performance, install the CHECKSUM hardware
improvement.
• To take advantage of improved performance in semaphore processing, you
must be running on hardware that supports the PLO (Perform Lock Operation)
instruction.
1.1.4 Workstation connections
To access kernel services using TSO/E, you need the same hardware as other
OS/390 components. You also need the workstation connections that TSO/E uses
and the processor and network connections that JES2 or JES3 use. Network
connections can be made through:
• Systems Network Architecture (SNA) network: Configure the workstation
hardware and software to access TSO/E through the Virtual
Telecommunications Access Method (VTAM). The system requires no
additional network definitions for access to OS/390 UNIX through TSO/E.
• Transmission Control Protocol/Internet Protocol (TCP/IP): Configure the
workstation hardware and software to communicate with TCP/IP for OS/390.
For the Telnet (TN3270) server, define the Telnet VTAM parameters.
• TCP/IP rlogin or telnet: For rlogin or telnet, configure the workstation
hardware and software to communicate with TCP/IP for OS/390. If you use
rlogin, you may need additional network capacity to support additional rlogin
users.
• Communications Server: Communications Server is a terminal attachment
capability. It consists of an RS/6000 that is LAN- or channel-attached to the
OS/390 host system. Using this connection, terminals on asynchronous ports
of the RS/6000 can operate as if they were directly attached to the OS/390
system. Additionally, Communications Server support enables a user to telnet
or rlogin to the OS/390 system.
1.1.5 Users
With the UNIX System Services Application Services users can:
• Request services from the system through shell commands. Shell commands
are like TSO/E commands.
• Write shell scripts to run tasks. Shell scripts are analogous to REXX execs.
• Run programs interactively (in the foreground) or in the background.
Many users use similar interfaces on other systems, such as AIX for the RS/6000
or UNIX, and use terminology different from OS/390 terminology. For example,
they refer to virtual storage as memory. The work done by their system
administrators is handled by system programmers in an OS/390 system.
OS/390 UNIX System Services
5
1.1.6 Application programmers
Application programmers are likely to do the following when creating
UNIX-compliant application programs:
1. Design, code, and test the programs on their workstations using XPG4
UNIX-conforming systems.
2. Send the source modules from the workstation to OS/390.
3. Copy the source modules from the OS/390 data sets to HFS files.
4. Compile the source modules and link-edit them into executable programs.
5. Test the application programs.
6. Use the application programs.
An OS/390 UNIX program can be run interactively from a shell in the foreground
or background, run as an OS/390 batch job, or called from another program.
1.1.6.1 Types of applications
The following types of applications exist in an OS/390 system with OS/390 UNIX:
• Strictly conforming XPG4 UNIX-conforming applications
• Applications using only kernel services
• Applications using both kernel and OS/390 services
• Applications using only OS/390 services
An OS/390 program submitted through the job stream or as a job from a TSO/E
session can request kernel services through the following:
• C/C++ functions
• Shell commands after invoking the shell
• Callable services
At the first request, the system dubs the program as an OS/390 UNIX process,
unless it is a POSIX(OFF) program (which is not dubbed).
1.1.7 Administrative tasks using the ISPF shell
The ISPF shell is a panel interface that you can use instead of TSO/E commands
or shell commands to perform certain tasks. The appropriate sections of this book
mention for which tasks you can use this interface.
You can use the ISPF shell to work with the file system to do the following tasks:
• Display all mounted file systems
• Display the attributes of a mounted file system such as total blocks, blocks in
use or ddname
• Create a file system (allocate an HFS data set)
You can use the ISPF shell to perform the following tasks, which require you to:
• Set up the root file system
• Create character special files
• Mount a file system
6
Debugging UNIX Applications on OS/390
• Unmount a file system
• Reset a pending unmount
• Reset a quiesce status
• Change attributes for an OS/390 UNIX user
• Display a list of users and sort by name, UID and GID
• Print a list of users
• Set up OS/390 UNIX users
• Set up OS/390 UNIX groups
• Permit users to alter their own home directory and initial program
1.2 Installation concepts for OS/390 UNIX System Services
Beginning with OS/390 V1R3, installation of service was simplified as follows:
• OpenEdition System Services (FMID HOMxxxx) was merged with the BCP
and is now part of the BCP FMID. In addition, the OMVS address space is
started automatically like traditional OS/390 address spaces such as Dump
Services and Global Resource Serialization (GRS) address space.
• The OMVS address space is always started and gets its input from the
OMVS= system parameter in the IEASYSxx parmlib member or as a
parameter entered during IPL.
• If OMVS= is not specified, the OMVS address space is still started. To
dynamically reconfigure kernel services use the SET OMVS or SETOMVS
command. If a full restart of kernel services is required, the system must be
re-IPLed.
• The START OMVS and STOP OMVS command is no longer supported.
In OS/390 V1R2, the Shell and Utilities and Debugger (FMIDs HSUxxxx and
HDXxxxx) were merged with OpenEdition Application Services and made part of
FMID HOTxxxx.
1.2.1 Startup definitions
Using the OMVS parameter in the IEASYSxx parmlib member you are able to
specify one or more BPXPRMxx parmlib members to be used to specify the initial
parmlib settings for the kernel. If you omit the OMVS parameter or if you specify
OMVS=DEFAULT in the IEASYSxx parmlib member, the kernel is started in a
minimum configuration mode. See 1.4, “Debugging OS/390 UNIX System
Services” on page 17 for more details according to the BPXPRMxx member and
its statements.
1.2.2 Levels of operation
It is possible to activate the OS/390 UNIX kernel services by three ways:
• Minimum mode
• Sockets-only mode
• Full-function mode
OS/390 UNIX System Services
7
1.2.2.1 Minimum mode
If you do not specify OMVS= in the IEASYSxx parmlib member or if you specify
OMVS=DEFAULT, then kernel services start up in minimum mode when the
system is IPLed. This mode is intended for installations that do not plan to use the
kernel services.
In minimum mode:
• Many services are available, but some functions such as TCP/IP sockets
require additional system customization, may not work.
• TCP/IP sockets (AF_INET) are not available.
• A Temporary File System (TFS) is used. A TFS is an in-storage file system,
hence no physical DASD data set needs to exist or to be mounted. The TFS is
initialized and primed with a minimum set of files and directories. Any data
written to this file system is not written to DASD.
The TFS is initialized with these directories and files:
/ (root directory)
/bin directory
/etc directory
/tmp directory
/dev directory
/dev/null file
There are no executables in the TFS (that is, you will not find the Shell and
Utilities). Do not attempt to install UNIX System Services Application Services in
the TFS, since no data will be written to DASD.
This is the minimum requirement for applications to be able to use kernel
services.
In this mode, you do not need to install or customize SMS, or customize the
security product, for working with kernel services.
If the default is set up, you do not need to define users who want to run an
application that uses APIs. For example, you would not need to assign a UID to
an application that wants to use C pthread functions.
1.2.2.2 Sockets-only mode
For a sockets-only level of kernel services that is more than the minimum but
does not need SMS or the shell and other UNIX utilities (such as rlogin), you can
create a BPXPRMxx parmlib member containing definitions as shown in the
following figure:
8
Debugging UNIX Applications on OS/390
FILESYSTYPE TYPE(UDS)
ENTRYPOINT(BPXTUINT)
NETWORK DOMAINNAME(AF_UNIX)
DOMAINNUMBER(1)
MAXSOCKETS(2000)
TYPE(UDS)
FILESYSTYPE TYPE(INET)
ENTRYPOINT(EZBPFINI)
NETWORK DOMAINNAME(AF_INET)
DOMAINNUMBER(2)
MAXSOCKETS(10000)
TYPE(INET)
Figure 2. BPXPRMxx definitions
The BPXPRMxx parmlib member:
• Takes the default minimum mode and uses the TFS.
• Sets up the DOMAIN name specifications for the Communication Server
socket connections.
Whether you can run in minimum mode when using sockets depends on a
number of factors. If you are using only AF_UNIX sockets, you should be able to
run in minimum mode. However, if you are using AF_INET sockets, it will depend
on which transport provider you are using:
• Because TCP/IP requires the use of a file system, it cannot be run in minimum
mode.
• Other AF_INET providers, such as OESTACK, do not require a file system.
Those can be run in minimum mode.
For each transport provider you specify in the FILESYSTYPE or
SUBFILESYSTYPE statements in your BPXPRMxx PARMLIB member, check
whether a file system is required.
1.2.2.3 Full-function mode
If an active BPXPRMxx parmlib member specifies "FILESYSTEM TYPE(HFS)",
then the kernel services start up in full-function mode when the system is IPLed.
To use the full function, you need to:
• Set up BPXPRMxx parmlib definitions
• Set up DFSMS/MVS
• Set up the Hierarchical File System (HFS)
• Install UNIX System Services Application Services
• Set up the security product definitions for OS/390 UNIX
• Set up the users’ access and their individual file systems
OS/390 UNIX System Services
9
1.2.3 Dynamic activation
You can modify configuration aspects by using the SETOMVS operator command,
or you can change the BPXPRMxx PARMLIB members that are in effect by using
the SET OMVS operator command.
1.2.3.1 SETOMVS command
The SETOMVS command allows you to dynamically reconfigure one or more of
the system characteristics. A sample SETOMVS invocation is:
SETOMVS MAXTHREADTASKS=100,MAXPROCUSER=8
1.2.3.2 SET OMVS command
The SET OMVS (or its abbreviated form T OMVS) command lets you specify one
or more BPXPRMxx parmlib members to dynamically reconfigure the settings of
the kernel services. With the SET OMVS command, you can have multiple
BPXPRMxx definitions so you can easily reconfigure a large set of the system
characteristics. You can keep the reconfiguration settings in a permanent
location for subsequent reference or reuse. A sample SET OMVS invocation is:
SET OMVS=(AA,BB)
or
T OMVS=(AA,BB)
If a parameter is specified more than once, with a different value, in the parmlib
members, the first value specified is the value used. For example, if you specify
SET OMVS=(AA,BB) where AA has a MAXPROCUSER=10 value and BB has a
MAXPROCUSER=5 value, MAXPROCUSER=10 is used.
1.2.4 Job Wait Time (JWT)
When an address space is dubbed as a process, or when a forked or spawned
process is created, the process may go into signal-enabled waits. In a
signal-enabled wait, the address space is made exempt from long-wait time-outs
as specified by the JWT value in parmlib member SMFPRMxx. This enables
parent processes to wait forever while child processes are running. Otherwise, if
a parent process is terminated due to a JWT time-out, a SIGHUP is sent to the
running child process and work is lost.
However, shell users, whether logged on through TSO/E and the OMVS
command, or through rlogin or telnet, are exempt from JWT time-out because the
shell is in a signal-enabled wait while waiting for a command from the user. If you
want the shell users to time out and be logged off for long detected waits, then
you need to specify the TMOUT environment variable in /etc/profile. The TMOUT
environment variable contains the number of seconds before user input times out.
If user input is not received within this length of time, the shell ends.
1.2.5 Started procedures
OS/390 UNIX System Services requires started procedures to be set up. There
are three procedures, OMVS, BPXOINIT, and BPXAS.
1.2.5.1 OMVS
The OMVS cataloged procedure runs a program that initializes the kernel. The
STARTUP_PROC statement in the BPXPRMxx parmlib member specifies the
10
Debugging UNIX Applications on OS/390
OMVS cataloged procedure. In our examples we have used the default, which is
OMVS.
1.2.5.2 BPXOINIT
In OS/390 Release 3, BPXOINIT is the started procedure that runs the
initialization process. BPXOINIT is also the jobname of the initialization process
(prior to OS/390 Release 3, the initialization process was created using an APPC
allocate and the jobname was OMVSINIT).
BPXOINIT is shipped in SYS1.PROCLIB.
The BPXOINIT address space has two categories of functions:
1. It behaves as PID(1) of a typical UNIX system. This is the parent of /etc/rc,
and it inherits orphaned children so that their processes get cleaned up using
normal code in the kernel. This task is also the parent of any MVS address
space that is dubbed and not created by fork or spawn. Therefore, TSO/E
commands and batch jobs have a parent PID of 1.
2. Certain functions that the kernel performs need to be able to make normal
kernel calls. This address space is used for these activities. For example,
mmap() and user ID alias processing.
The STEPLIB DD statement is propagated from OMVS to BPXOINIT. Therefore, if
there is a STEPLIB DD statement in the BPXOINIT procedure, it will not be used
if a STEPLIB DD statement was specified in the OMVS procedure.
1.2.5.3 BPXAS
Prior to OS/390 V2R4, APPC/MVS transaction initiators provided address spaces
when programs used the fork() or spawn() C function or callable services. Now
Workload Manager (WLM) creates the address spaces. The BPXAS PROC found
in SYS1.PROCLIB will now be used to create the WLM-managed address
spaces. When using WLM, you do not need to do any tuning or issue any
commands.
1.3 Installation highlights of OS/390 UNIX
Here, the most important highlights of the OS/390 UNIX installation are
discussed. For a detailed description of the installation part of UNIX System
Services, examine the OS/390 Version 2 Release 6 UNIX System Services
Implementation and Customization, SG24-5178. In the third chapter of this
redbook you will find a step-by-step guide to implement and customize UNIX
System Services on OS/390.
OS/390 UNIX System Services files are organized in a hierarchy, as in a UNIX
system. MVS views an entire file hierarchy as a collection of hierarchical file
system data sets called HFS data sets. These HFS data sets are unloaded when
you restore your ServerPac tapes to your previously initialized SMS DASD.
The following steps identify the tasks that establish an HFS for the first time. SMS
must be active and managing a volume prior to restoring the file system:
• Implement SMS for UNIX System Services
• Implement a security system for UNIX System Services
OS/390 UNIX System Services
11
• Allocate and restore the HFS files
• Perform post-restore activities
Once you have completed all these tasks, your system will have:
• A root file system that contains the products and elements that install into the
root file system
• Separate file systems for those products or elements that do not install into the
root file system
• An HFS file system mounted to the /etc directory
• A BPXPRMFS member that reflects the restored file system
A Web site that provides setup hints is available at URL:
http://www.s390.ibm.com/os390/support/os390tst/hfs.htm
1.3.1 How UNIX System Services has changed
As of OS/390 Release 3, the OMVS address space (UNIX System Services)
starts automatically at IPL. Following the IPL sequence identified in this book,
the first IPL starts UNIX System Services with limited function and a temporary
root file system (see 1.2.2.1, “Minimum mode” on page 8). This level is usually
selected by the BPXPRM00 parmlib member of a delivered parmlib (for instance,
CPAC.PARMLIB):
FILESYSTYPE TYPE(HFS)
ENTRYPOINT(GFUAINIT)
PARM(’ ’)
Limited function gives you the ability to UNMOUNT the temporary ROOT file
system and MOUNT your permanent ROOT file system during restore
processing. Limited function does not give you all the capability full function UNIX
System Services has. UNIX System Services is being started in this mode only
for the purposes of the installation.
The next IPL described in this section starts UNIX System Services in
full-function mode with all the file systems mounted. This level is selected by the
BPXPRMFS parmlib member shipped in one of the parmlibs of the installation
pac (for example, CPAC.PARMLIB).
1.3.2 Implementing SMS
HFS data sets must reside on System Managed Storage (SMS) DASD. To use
UNIX System Services in full-function mode, SMS must be active.
In this section, the goal is to define a minimal SMS environment that allows the
UNIX System Services to be run. The following are the SMS implementation
steps:
1. Define a volume to be SMS managed.
2. Allocate the SMS control data sets.
12
Debugging UNIX Applications on OS/390
3. Set up your user ID as storage administrator.
4. Define a base configuration.
5. Define a storage class.
6. Define a storage group.
7. Define volumes in a storage group.
8. Create ACS routines.
9. Translate ACS routines.
10. Update parmlib.
11. Activate SMS.
12. Activate the configuration.
The following sections describe the above SMS implementation steps in more
detail. The instructions listed are used to activate DFSMS with minimal
configuration. You should also refer to DFSMS/MVS Implementing
System-Managed Storage, SC26-3123, for more information about activating
SMS.
1.3.2.1 Defining a volume to be SMS managed
The first step to activate System Managed Storage (SMS) is the preparation of
DASD volume(s) for SMS storage groups.
Use the Device Support Facility (ICKDSF) product to initialize DASD volumes to
be SMS managed. The STORAGEGROUP keyword of the INIT command is used
to make a DASD volume available for the allocation of the new system-managed
data sets. Here is an example:
INITUNITADDRESS(01C6)PURGENOVERIFYDEVTYPE(3390)VOLID(VS17SM)VTOC(9,1,38)INDEX(0,3,76)STGR
Note
If your OS includes DFP 3.3, use the SG keyword instead of the STGR
keyword.
1.3.2.2 Allocating the SMS control data sets
Before you can activate an SMS configuration, there are three types of SMS
control data sets that are required:
• CDS—source control data set
• ACDS—active control data set
• COMMDS—communications data set
Usually these data sets are allocated when you restore your target and
distribution data sets. In the corresponding SAMPLIB of your pac (for instance,
OS/390 UNIX System Services
13
CPAC.SAMPLIB) there are examples provided on allocating the SMS control data
sets.
Table 1. Sample jobs
Job name
Description
SMSSCDS
Define an SCDS data set
SMSACDS
Define an ACDS data set
SMSCOMS
Define a COMMDS data set
1.3.2.3 Setting up your user ID as storage administrator
To perform certain definitions, the user must be set up as a storage administrator
for ISMF.
1.3.2.4 Defining the SMS base configuration
The base configuration contains installation defaults and identifies the systems to
which the SMS configuration applies.
1.3.2.5 Defining a storage class
A storage class is a list of performance objectives and availability requirements.
Each storage class represents a list of services that are available to data sets and
objects having similar access requirements. For a detailed description see
Chapter 3 of OS/390 Version 2 Release 6 UNIX System Services Implementation
and Customization, SG24-5178.
1.3.2.6 Defining a storage group
Storage groups represent the physical storage managed by SMS. Each storage
group is a collection of physical devices, virtual input/output, (VIO) or dummy
volumes defined by the storage administrator. For a detailed description see
Chapter 3 of OS/390 Version 2 Release 6 UNIX System Services Implementation
and Customization, SG24-5178.
1.3.2.7 Defining volumes in a storage group
The next step is to define a list of volumes in the previously defined storage
group. For a detailed description see Chapter 3 of OS/390 Version 2 Release 6
UNIX System Services Implementation and Customization, SG24-5178.
1.3.2.8 Creating ACS routines
The following example shows storage class and storage group routines that will
be used to restore the HFS data sets to your SMS volumes:
Example: Storage Class ACS Routine
PROC STORCLAS
SELECT
WHEN (&DSNTYPE = ’HFS’) /*select HFS data sets */
SET &STORCLAS = ’OMVS’
OTHERWISE
SET = ’ ’
END
END /* end of storage class routine proc */
14
Debugging UNIX Applications on OS/390
In the preceding routine, SMS manages any data set that has a a data set type of
HFS, puts these data sets in the Storage Class OMVS and on the SMS-managed
volume that was defined. All other data sets in this routine are not SMS managed.
In the following routine, all SMS-managed data sets reside in the storage group
OMVS which was previously created.
Example: Storage Group ACS Routine
PROC STORGRP
SET &STORGRP = ’OMVS’
END
1.3.2.9 Translating ACS routines
The ACS routines need to be translated into object form and placed in the SCDS.
For a detailed description see Chapter 3 of OS/390 Version 2 Release 6 UNIX
System Services Implementation and Customization, SG24-5178.
1.3.2.10 Updating PARMLIB
The following parmlib members need to be updated with the corresponding
information:
Table 2. Parmlib members
Member
Description
IGDSMSxx
This member should contain the names of the ACDS and COMMDS
data sets you previously defined.
IEFSSNxx
This member identifies SMS as a subsystem to MVS (OS/390). To
ensure that SMS starts before the primary subsystem, the SMS record
is placed before the primary subsystem’s (JES2 or JES3) record.
1.3.2.11 Activating SMS
The SMS subsystem is activated during the initial IPL sequence because of the
subsystem identification in member IEFSSNxx.
1.3.2.12 Activating the configuration
The SMS configuration is activated during the initial IPL sequence.
1.3.3 Implementing a security system for UNIX System Services
The following security actions should have been established in the RACF
database. For more information, refer to UNIX System Services Planning,
SC28-1890.
This is an example of tasks that should have been performed:
1. Defined user with OMVS segment and superuser authority for OMVS tasks
2. Defined terminal group name tty for OMVS access
3. Changed SYS1 group to enable UNIX System Services access
4. Defined the OMVS Cataloged Procedure to RACF by using both of the
following methods:
a) RACF started procedure table (module ICHRIN03)
b) RACF started facility class
OS/390 UNIX System Services
15
1.3.4 Allocating and restoring the HFS files
With ServerPac/SystemPac installs, you should run the jobs that are supplied.
These jobs are new since OS/390 V2R6. Two jobs are supplied to allocate and
restore the HFS files as follows:
Table 3. Jobs for allocating and restoring HFS files
Job name
Description
ESTABFS
This job allocates HFS data sets that are defined in your installation
variables option of the installation dialog. The job also mounts the new
root file system, creates mount points, and mounts any additional file
system.
Note: When running this job, you may encounter the IRA200E
AUXILLIARY STORAGE SHORTAGE message. It is recommended to
allocate a second local page data set prior to submitting this job.
RESTFS
This job mounts the transient file system, which contains the files
needed for the restore operation. It then restores those files into the
root file system.
Note: This job may end with an RC=12 if the transient file system is
already mounted.
1.3.5 Performing HFS post-restore activities
To complete the installation process, you need to perform post-restore activities.
The activities that you need to perform will vary depending on the elements that
you are installing.
There are some tasks that appear only when certain products or elements are
ordered, for example, the need to set HFS symbolic links may be required.
The following shell initialization files have to be copied from the /samples directory
to your /etc directory if you want to customize the shell initialization or the user
wide shell environment:
Table 4. Shell initialization files
Files
Description
/etc/init.options
UNIX System Services initialization options file
/etc/profile
System-wide profile for shell users
/etc/rc
Default initialization shell script file
For more details on the files in Table 4 refer to 1.4.1.1, “The Shell initialization
files and user profiles” on page 17.
16
Debugging UNIX Applications on OS/390
1.4 Debugging OS/390 UNIX System Services
This section is the main part of the OS/390 UNIX System Services chapter. The
most frequently asked questions and problem situations are discussed; the first
part provides useful hints and tips to work with and to debug OS/390 UNIX. The
second part handles known problems and possible solutions.
1.4.1 Hints and tips
Some hints and tips in the USS area follow.
1.4.1.1 The Shell initialization files and user profiles
When OS/390 UNIX is started, initialization includes running the /etc/init program.
If this program is not found, OS/390 UNIX attempts to run /usr/sbin/init. This file
contains the default initialization program shipped with OS/390 UNIX Shell and
Utilities.
Table 5. Files associated with /etc/init
Files
Description
/bin/sh
Default shell that /etc/init invokes to execute /etc/rc or another
shell script specified in /etc/init.options
/etc/init.options
Initialization options file, which is read by /etc/init
/etc/rc
Default initialization shell script
There are two types of profiles where an authorized administrator, respectively
the OS/390 UNIX user, is able to define specific options:
Table 6. Profiles
Profiles
Description
/etc/profile
System-wide user profile. Any time a user invokes the OS/390 UNIX
shell, the options specified there become active for this user.
/$HOME/.profile
User-specific profile. Any user who has a .profile in his or her HOME
directory may specify individual options here. These options may be
additional to the specifications in /etc/profile or may overwrite
parameters defined there.
The read/load sequence from starting OS/390 UNIX (IPL) up to the ready-to-work
state of a user who invoked the shell follows:
• Start OS/390 UNIX automatically during IPL.
• Does /etc/init exist?
OK: Run /etc/init.
NO: Run /usr/sbin/init.
• /etc/init invokes /bin/sh, the default shell.
• /etc/init reads /etc/init.options.
• /etc/init runs the initialization shell script specified in /etc/init.options.
This is the sequence for every OS/390 UNIX IPL. When a user invokes the
OS/390 UNIX shell by running the TSO OMVS command the following happens:
OS/390 UNIX System Services
17
• Is there a .profile file in the user’s home directory?
NO: Use the options specified in /etc/profile for this user.
YES: Use the options specified in this .profile for this user plus the options out
of /etc/profile if not overwritten by the corresponding parameter in the .profile.
1.4.1.2 Setting the time zone
The time zone is an environment variable used by the time service to set the
correct time for OS/390 UNIX and applications running on OS/390 UNIX. Since it
is possible to specify this variable in four different files, we will clarify here which
application is affected by which setting.
At first, it is recommended to have a time zone (TZ) specification active in
/etc/init.options and /etc/profile. If there are daemons to start using the /etc/rc
script, export the TZ variable there before starting the daemons.
A TZ setting in /etc/rc is only valid for jobs started in this script file at system
initialization time—as the INETD, for example.
A TZ setting in /etc/init.options is only valid for all processes that are active at
OS/390 UNIX kernel initialization time. Settings here have no effect on what the
user will get when asking for the time (for instance, by issuing the date shell
command) after entering the shell.
The /etc/profile is the system-wide profile for shell users. The TZ variable may be
set here simply by specifying TZ=<value>. This value becomes active every time
a user invokes the OS/390 UNIX shell. Since this is a system-wide profile you
need to have the permission to update /etc/profile if you want to change the value
of TZ here.
Every user has the chance to create a .profile in his HOME directory and to
overwrite variables of /etc/profile here. Therefore, it is possible to specify another
TZ value here than in the /etc/profile. If the corresponding user invokes the
OS/390 UNIX shell again for the next time, this TZ value is the valid one for this
user.
Note
For many users it may not be necessary to have an individual HOME directory
for their own use. Many installations specify /usr as the HOME directory for
every TSO user who has an OMVS segment and is allowed to enter the
OS/390 UNIX shell. If there is a .profile in /usr, every user will inherit the
settings of this profile.
If there was no TZ setting in either of these files, time conversions behave as if TZ
was set to TZ=UTC0 (Universal Time Coordinated).
18
Debugging UNIX Applications on OS/390
The format of the TZ variable recognized by the tzset() kernel function is as
follows:
<std><offset><dst><offset>,<rule>
Where
std and dst
indicate neither less than three, nor more than TZNAME_MAX bytes that are the
designation for the standard (std) and summer (dst - daylight saving time) time
zones. If more than TZNAME_MAX bytes are specified for std or dst, tzset()
truncates to TZNAME_MAX bytes. Only std is a required parameter. If its setting
is less than three bytes, the whole TZ value is invalid and UTC0, respectively
GMT0, becomes active.
offset
indicates the value to add to the local time to arrive at Universal Time
Coordinated. The offset has the following format:
<hh<mm ss>>
where
minutes and seconds are optional. The hour (hh) is required.
rule
indicates when to change to and back from summer time. The rule has the form:
<date/time,date/time>
where
the first date describes when the change from standard to summer time
occurs and the second date describes when the change back happens. The
format of the date must be one of the following:
Jn
The Julian day n (1-n-365). Leap days are not counted. In all years,
including leap years, February the 28th is day 59 and March the 1st
is day 60. It is impossible to explicitly refer to the 29th of February.
n
The zero-based Julian day n (0-n-365). Leap days are counted and
it is possible to refer to the 29th of February.
Mm.n.d
The dth day of the week n of month m of the year (1-n-5 and
1-m-12 and 0-d-6, where week 5 may occur for the last days of a
month). Week 1 is the first week in which the dth day occurs. Day
zero is Sunday.
The time has the same format as offset except that no leading sign, minus (-)
or plus (+) is allowed. The default, if no time was specified, is 02:00:00.
If dst is specified and the rule is not specified by TZ, the default for the summer
time start date is M4.1.0 and for the summer time end date is M10.5.0.
Examples:
• For users located in Boston or New York, the value of the TZ variable should
be EST5EDT. EST means Eastern Standard Time, five hours to add (west) to
arrive GMT. EDT means Eastern Daylight Time which is four (5 minus 1).
OS/390 UNIX System Services
19
• For users located in Germany, the value of the TZ variable is
MEZ-1MESZ,086,286. MEZ is the abbreviation for Mitteleuropaeische Zeit and
is one hour east of GMT. MESZ is the name of the zone during daylight saving
time. The value 086 is the zero-based Julian date that describes when the
daylight saving time starts and the 286 value when it ends.
1.4.1.3 OS/390 UNIX Component Trace
OS/390 UNIX uses the OS/390 (MVS) Component Trace (CTRACE) facility.
OS/390 UNIX registers itself to OS/390 as a traceable component of OS/390. It is
up to the component, in this case OS/390 UNIX, to determine when and what
kind of data is written to the component trace.
The OS/390 UNIX parmlib members are used not only to turn on component
tracing for OS/390 UNIX, but specify which OS/390 UNIX functions are to be
traced. OS/390 UNIX initialization, when finding out that component tracing is
requested, sets up a data space, referred to as SYSZBPX1, which is associated
with the OS/390 UNIX kernel address space.
What kind of information is in a USS Component Trace?
Generally, OS/390 UNIX provides CTRACE records for the following OS/390
UNIX functions:
• chars
• devpty
• file
• ipc
• lock
• pipe
• process
• ptrace
• signal
• storage
• syscall
For a more detailed description of each function traced for OS/390 UNIX, refer to
OS/390 MVS Diagnosis: Tools and Service Aids, SY28-1085.
How to request an OS/390 UNIX Component Trace
When OS/390 UNIX starts, the trace automatically starts. The trace cannot be
completely turned off while OS/390 UNIX is running. The OS/390 UNIX parmlib
member CTnBPXxx specifies what functions specifically are to be traced. The
particular CTnBPXxx parmlib member read is pointed to from the
’CTRACE(CTnBPXxx)’ statement in the BPXPRMxx parmlib member.
The CTnBPXxx parmlib member also specifies the size of the trace buffers. You
cannot change this size while OS/390 UNIX is running.
Your BPXPRMxx parmlib member, which is the main parmlib member for OS/390
UNIX initialization, checks the CTRACE statement to see what CTnBPXxx
member is to be checked for CTRACE specifications. There are two areas that
20
Debugging UNIX Applications on OS/390
you would typically need to alter in CTnBPXxx to request that OS/390 UNIX
writes component trace records:
• size - enter a buffer size here
/*--------------------- */
BUFSIZE(4M)
/*--------------------- */
• trace options
/*----------------------------------------- */
/* OPTIONS: NAMES OF FUNCTIONS TO BE TRACED */
/*----------------------------------------- */
/* OPTIONS(
’ */
/* ’ALL
’ */
/*,’CHARS
’ */
/*,’DEVPTY
’ */
/*,’FILE
’ */
/*,’IPC
’ */
/*,’LOCK
’ */
/*,’PIPE
’ */
/*,’PROCESS
’ */
/*,’PTRACE
’ */
/*,’SIGNAL
’ */
/*,’STORAGE
’ */
/*,’SYSCALL
’ */
/* )
*/
To allow component tracing, you need to remove the comment (/* */) on the
following lines:
• OPTIONS line
• The ’)’ line
• Whatever function line(s) you wish to affect
Once OS/390 UNIX starts with the appropriate OS/390 UNIX functions writing
component trace entries to the OS/390 UNIX data space, you can stop the tracing
using the OS/390 trace command:
TRACE CT,OFF,COMP=SYSOMVS
You can also resume tracing with any CTnBPXxx parmlib member using:
TRACE CT,ON,COMP=SYSOMVS,PARM=CTnBPXxx
Finally, you can use the display command to find information about tracing:
DISPLAY TRACE,COMP=SYSOMVS
Obtaining OS/390 UNIX Component Trace data
The OS/390 UNIX CTRACE data is written to OS/390 UNIX data space
SYSZBPX1. Usually it is OS/390 UNIX support that asks for this data to interpret.
Once this data is collected, you should dump both the OS/390 UNIX address
space and the OS/390 UNIX data space together:
DUMP COMM=(OMVS - or your own dump description)
R#,SDATA=(CSA,SQA,LPA,RGN,TRT,GRSQ)
R#,JOBNAME=OMVS,DSPNAME=(’OMVS’.*),END
OS/390 UNIX System Services
21
You may replace the JOBNAME parameter with the ASID parameter. If you issue
the following command you will find out the hexadecimal address space identifier
of the OMVS address space in the ’A=’ field:
DA,OMVS
Formatting the OS/390 UNIX Component Trace Data
You can use the IPCS subcommand ’CTRACE’ to format out the OS/390 UNIX
trace data from the dump:
CTRACE COMP(SYSOMVS) FULL
or
CTRACE COMP(SYSOMVS) SUB((KERNINFO)) FULL
You will find more information in OS/390 MVS Interactive Problem Control System
(IPCS) Commands, GC28-1754.
Other documentation for OS/390 UNIX Component Trace
Other references for OS/390 UNIX component tracing are:
• II08038 - Informational APAR
• OS/390 MVS Diagnosis: Tools and Service Aids, SY28-1754
1.4.1.4 Useful OS/390 UNIX commands
To monitor the actual status of the OMVS address space there are some display
commands that provide you with lots of information. The OS/390 operator can
use the DISPLAY command to obtain:
• OS/390 UNIX System Services status information (for example, active or
terminating)
• Hierarchical File System (HFS) information
• OS/390 UNIX System Services processes information for address spaces
• The current setting for all OS/390 UNIX System Services parmlib statements
• Information about multiple parmlib members
You can use this command to display address space information for a user who
has a process that is hung. You can also use the information returned from this
command to determine how many address spaces a TSO/E user ID is using,
whether an address space is using too many resources, and whether a user’s
process is waiting for an OS/390 UNIX kernel function to complete.
Displaying Summary
D OMVS,S
The command, D OMVS,S, displays status of OS/390 UNIX processes, file
systems, and servers (for example, active or terminating) and the BPXPRMxx
parmlib member specified during initialization or specified by the SET OMVS=
OS/390 UNIX System Services command.
The output of this command looks like the following:
RESPONSE=******* (the corresponding system image symbol)
BPXO042I 10.31.10 DISPLAY OMVS 939
OMVS
000E ACTIVE
OMVS=(00)
22
Debugging UNIX Applications on OS/390
The OMVS= field describes which BPXPRMxx member was taken during
initialization; in this case BPXPRM00. Also you can see the address space
identifier x’E’.
Displaying ASID
D OMVS,A=ALL
or
D OMVS,A=xx
This command displays process information for all OS/390 UNIX System
Services address spaces. The output of this command should look like:
RESPONSE=******* (the corresponding system image symbol)
BPXO040I 10.40.12 DISPLAY OMVS 113
OMVS
000E ACTIVE
OMVS=(00)
USER
JOBNAME ASID
PID
PPID STATE
START
CT_SECS
BPXOINIT BPXOINIT 0077
1
0 MKI
11.36.57
.460
LATCHWAITPID=
0 CMD=BPXPINPR
SERVER=Init Process
AF=
0 MF=65535 TYPE=FILE
In this case the specific information for address space x’77’ was requested, the
BPXOINIT address space. Using ASID=ALL in the DISPLAY command you will
get information about all OMVS-related address spaces. If the ASID specified is
not OMVS related you will get an error message.
Displaying user ID
D OMVS,U=<UID>
This command displays process information for all processes associated with the
specified TSO/E user ID. Use this operand when a user requests that a hung
process be canceled. You can display all processes owned by the user and find
the address space ID of the process that needs to be canceled. Then use the
CANCEL command to cancel the address space.
The output of this command looks like the following:
RESPONSE=******* (the corresponding system image symbol)
BPXO040I 10.50.45 DISPLAY OMVS 346
OMVS
000E ACTIVE
OMVS=(00)
USER
JOBNAME ASID
PID
PPID STATE
START
TEST
TEST
008F 117440522
1 1RI
10.19.48
LATCHWAITPID=
0 CMD=EXEC
TEST
TEST
008F
67108876 117440522 1CI
10.22.39
LATCHWAITPID=
0 CMD=-sh
CT_SECS
3.745
3.745
In this case you can see information about the user TEST and about all
processes that are owned by this user. There are two processes identified by a
different process ID but the same address space identifier x’8F’. This means all
processes started by user TEST are running in the same address space. In our
example the first process is the parent process and the second one is its child.
OS/390 UNIX System Services
23
Displaying a specific process
D OMVS,PID=<processID>
This command displays thread information for the processID that is specified in
decimal numbers. The output of the command looks like:
RESPONSE=****** (the corresponding system image symbol)
BPXO040I 10.59.58 DISPLAY OMVS 563
OMVS
000E ACTIVE
OMVS=(00)
USER
JOBNAME ASID
PID
PPID STATE
START
CT_SECS
TEST
TEST
008F
67108876 117440522 1CI
10.22.39
3.748
LATCHWAITPID=
0 CMD=-sh
THREAD_ID
TCB§
PRI_JOB USERNAME
ACC_TIME SC STATE
0DCBCF9000000001 0079C498 OMVS
.219 RED C
In this case the thread information for process 67108876 was requested.
Displaying options
D OMVS,O
This command displays all options that can be set in the parmlib member
BPXPRMxx. The output of this command should look like:
RESPONSE=******* (the corresponding system image symbol)
BPXO043I 11.05.04 DISPLAY OMVS 822
OMVS
000E ACTIVE
OMVS=(00)
OPENEDITION MVS CURRENT CONFIGURATION SETTINGS:
MAXPROCSYS=200
MAXPROCUSER=200
MAXFILEPROC = 2000
MAXFILESIZE= NOLIMIT
MAXCPUTIME = 2147483647
MAXUIDS= 50
MAXRTYS = 256
MAXPTYS= 256
MAXMMAPAREA = 4096
MAXASSIZE = 2147483647
MAXTHREADS =500
MAXTHREADTASKS = 500
MAXCORESIZE = 4194304
MAXSHAREPAGES
= 2048000
IPCMSGQBYTES= 262144
IPCMSGQMNUM = 10000
IPCMSGNIDS= 500
IPCSEMNIDS = 500
IPCSEMNOPS= 25
IPCSEMNSEMS= 25
IPCSHMMPAGES= 256
IPCSHMNIDS= 500
IPCSHMNSEGS= 200
IPCSHMSPAGES= 786432
SUPERUSER= BPXROOT
FORKCOPY= COPY
STEPLIBLIST= /system/steplib
USERIDALIASTABLE=
PRIORITYPG VALUES: NONE
PRIORITYGOAL VALUES: NONE
24
Debugging UNIX Applications on OS/390
In this case all active options specified in the BPXPRM00 member were
displayed, yet there is one exception. If the system administrator used the
SETOMVS or SET OMVS command to change any of the options, then the new
value is displayed here and not the value out of the member. So this command
always displays the currently active options.
Using the Modify command to cancel the BPXAS OMVS address spaces
Prior to OS/390 V1R3, it was known that OMVS should be stopped before
attempting to stop JES2. When OMVS was stopped, it took down the colony
address spaces that could prevent bringing down JES2 properly.
With "PERMKERN" in OS/390 V1R3 and forward, OMVS is brought up at IPL and
cannot be stopped, canceled, or forced. The colony address spaces that it
started must be stopped or canceled in order for JES2 to come down properly,
since they are not coded SUB=MSTR when started and use JES2 for some
functions. The operator instructions for cleaning up before stopping JES2 must
now be changed to include stopping or canceling the colony address spaces
started using the BPXPRMxx parmlib member. Those are the FILESYSTYPEs
which include ASNAME(procname). The colony address spaces are those
specified by ASNAME(procname).
The JES2 command $DA,X displays all jobs (JOBs/STCs/TSUs) that are active in
execution on the JES2 member on which the command is issued. The colony
address spaces that need to be stopped or canceled to bring down JES2 properly
will be shown in this display. The batch initiators running under JES2 all have the
name of INIT. These are stopped automatically by JES2. SYSLOG will have the
name of SYSLOG and is automatically stopped by JES2.
For the OMVS colony address spaces, whether the STOP command is supported
or the address space must be canceled is defined by the component owner of
that procedure and is documented in the owner’s publications.
The command D OMVS,A=ALL will show all the OMVS processes that are running.
Any of those using JES2 resources must be stopped or canceled before stopping
JES2. In OS/390 V2R4 and following releases, the Modify command must be
used to cancel the BPXAS OMVS address spaces.
After issuing the command $P JES2, the message HASP0607, MSGHASP607
JES2 NOT DORMANT, ACTIVE PROCESSES OR ADDRESS SPACES, RC=03
comes up. In this case use the following command to cancel the address spaces
that caused this message:
F BPXOINIT,SHUTDOWN=FORKINIT
This will cancel all remaining BPXAS address spaces and will allow you to purge
JES2.
Displaying OMVS data out of a dump
If there is any problem with OS/390 UNIX or any corresponding application and
the system automatically dumps the OMVS-related address spaces or if you
request a dump of the OMVS-related address spaces using the DUMP console
command as previously described, you have the chance to format the most
important OMVS-related information using the following IPCS command:
OMVSDATA
OS/390 UNIX System Services
25
This will format diagnostic reports about OS/390 UNIX users and resources.
OMVSDATA divides the information about OS/390 UNIX into six reports. Each
report corresponds to the following OMVSDATA keywords:
Table 7. OMVSDATA keywords
Keywords
Description
COMMUNICATIONS
Information about pseudoterminal user connections and
OCS remote terminal connections.
FILE
Information about each OS/390 UNIX file system type and its
mounted file systems.
IPC
Information about interprocess communication activity for
shared memory, message queues, and semaphores.
NETSTAT
Information about eNetwork Communications Server High
Speed Access Services (HSAS). The NETSTAT report type
has six further subtypes: SOCKETS (default), ROUTE,
INTERFACE, PERFORMANCE, STATISTICS and MEMORY.
PROCESS
Information about kernel processes.
STORAGE
Information about the storage manager cell pools.
PROCESS is the default here.
For each report type, you can select one or more of the following levels:
Table 8. Report type levels
Levels
Description
SUMMARY
Summary information for each requested report type. SUMMARY
is the default if no level is specified.
EXCEPTION
Diagnostic information for error or exceptional conditions for each
requested report type.
DETAIL
Detailed information for each requested report type.
For each report, you can select one or more of the following filtering keywords to
limit the amount of data in the report:
Table 9. Filtering keywords
26
Keywords
Description
ASIDLIST(asidlist)
Requests that information be provided for the asids
specified in asidlist. ASIDLIST(asidlist) can be specified
either as a single ASID or as a range of ASIDs. When a
range is specified, the two ASIDs (first and last in the
range) must be separated by a colon. The ASID can range
from 1 through 65 535. ASID can be expressed using the
notation X’nnn’, F ’nnn’, or B’nnn’. An unqualified number
is assumed to be fixed. The alias is ASID.
USERLIST(userlist)
Requests that information displayed be restricted to that
associated with the user IDs specified in userlist. The
contents of userlist may contain one or more user IDs,
separated by commas. USERLIST (userlist) can be
specified as a 1-to-8-character name. The alias is USER.
Debugging UNIX Applications on OS/390
Keywords
Description
PROCESSID
For the NETSTAT Sockets and NETSTAT Detail report
types only. Requests that information be provided for a
single PID. PROCESSID may contain up to eight
hexadecimal characters.
The OMVSDATA report header prefixes all the reports provided by the
OMVSDATA command. It appears regardless of the OMVSDATA options that are
selected. The selected OMVSDATA options are displayed followed by system
information pertinent to all reports.
For a more detailed description of the OMVSDATA IPCS command see OS/390
MVS Interactive Problem Control System (IPCS) Commands, GC28-1754.
1.4.1.5 Interpreting OS/390 UNIX return and reason codes
In some cases OS/390 UNIX System Services Application Services may issue
OS/390 UNIX-related error messages which contain a two-byte return code and a
four-byte reason code. Using the OS/390 UNIX System Services Messages and
Codes manual and having a look at the return codes description may show
enough information to find the reason for the error message. If the return code
description is not sufficient to find the error, you have the chance to use the
reason code to come closer to the error.
The reason code itself is a four-byte character and usually delivered by register
15 at the time of error. This reason code is divided into the left most two bytes
and the right most two bytes. The value of the left most two bytes is responsible
for the type of the reason code. See also OS/390 UNIX System Services
Messages and Codes, SC28-1908, and Chapter 2, “Lotus Domino for
S/390—troubleshooting/hints/tips” on page 33 in this redbook.
Example:
An OS/390 UNIX-related application abnormally ends with an ABENDEC6,
reason code 0BFC0012 and the return code is 9C. The return code describes a
Process initialization error and does not point out what the problem is. The
content of the two left most bytes is 0BFC. This means we will have to look at the
OS/390 UNIX reason codes. The content of the right most two bytes is 0012.
Having a look at the OS/390 UNIX reason codes, you will find the following
description for 0012:
JRMaxChild - The maximum number of processes for this user ID has been
exceeded.
In this example, there was an OS/390 UNIX user or application that tried to
initiate a child process using the fork() or spawn() kernel function, but the number
of processes for this user ID has been exceeded. So you either have to increase
the MAXPROCUSER parameter in the BPXPRMxx member or, if this value is
already high enough, you have to look for this user ID and to find out what kind of
processes this user invokes. You may use the following command:
D OMVS,U=<userID>
This command finds all the processes related to this ID.
OS/390 UNIX System Services
27
1.4.2 Common problems
In the following sections, some known problems and possible pitfalls are
discussed.
1.4.2.1 The "Root Permission" problem
In some cases a TSO/E user who tries to invoke the OS/390 UNIX shell may
suffer the following problem after entering ’TSO OMVS’ :
The OMVS command cannot use the message catalog.+
No session was started. No pseudo-TTYs are available.+
Function = catopen(), catalog name = ’/usr/lib/nls/msg/C/fsumucat.cat’,
return value = -1, errno = 111 (X’0000006F’), reason code = 5B150043,
description = ’EDC5111I Permission denied.’
Function = stat(), ending name = ’/dev/ptyp0000’, return value = -1, errno =
11
1 (X’0000006F’), reason code = 5B1D0043, description = ’EDC5111I Permission
denied.’
***
The reason for this problem is pretty simple. If you are installing OS/390 UNIX,
there is one step to specify and allocate the new root file system. This new root
file system will be allocated with the permission bits of 700 by default. This means
only the owner of this file system is allowed to access it. All other users cannot
access it after invoking the OS/390 UNIX shell. You can change the permission
bits of the root file system to 755 by using the OS/390 UNIX ISPF shell if you are
authorized to do this.
1.4.2.2 Problems caused by the MAXFILESIZE parameter
Some OS/390 UNIX users that tried to install maintenance into the HFS suffered
the following problem. During the apply of the corresponding PTFs (maybe for
another component like TCP/IP) the following message occurred:
BPXF106E RETURN CODE 0077, REASON CODE 7FFCC000. AN ERROR OCCURED DURING THE
WRITING TO HFS FILE <pathname>
Since this reason code is not described anywhere, you will have to look at the
return code. The description states, The file is too large. What will solve this
problem? In your BPXPRMxx member there is a parameter called MAXFILESIZE.
If the file to install is greater than the value specified there the message above
will occur. The solution is either to specify a value that is high enough (for
example the maximum of 524228) or to specify MAXFILESIZE=NOLIMIT.
Another error that is solved by the same parameter is the following. OS/390 UNIX
users suffered an ABENDEC6 with a reason code of 0000FD1E. This reason
code is described nowhere, but the solution is the MAXFILESIZE parameter
again. There is no recommendation about a specific value of the MAXFILESIZE
parameter. Specifying MAXFILESIZE=NOLIMIT will protect you from problems as
described but will not protect your HFS from being increased.
1.4.2.3 Problems during OS/390 UNIX installation
Very often OS/390 UNIX users report problems during OS/390 UNIX installation.
Prior to OS/390 V2R6 there were two jobs called HOT11xxB and HOT11xxC
depending on the version of OS/390 UNIX. The first job allocates the new root file
system and the second one fills in all the functions and directories into the file
system. Most of the problems occurred with the second job because it is only
possible to copy all the files to the file system if OS/390 UNIX was started in full
28
Debugging UNIX Applications on OS/390
function mode previously. So there has to be an OMVS=xx statement in the
IEASYSxx parmlib member to specify the BPXPRMxx member that contains the
initialization parameters of OS/390 UNIX. The BPXPRMxx member should
contain the corresponding FILESYSTYPE statement to start the HFS and, if the
corresponding root file system was allocated successfully, it should contain the
ROOT specification.
You may check the status of OS/390 UNIX by entering the D OMVS,F operator
command. This will show you in the OMVS= field whether OS/390 UNIX was
started in DEFAULT mode or whether a BPXPRMxx member was used during
initialization. Furthermore, you will see whether a temporary file system was
mounted or the new root file system is specified.
In OS/390 V2R6 and above, the two new jobs are ESTABFS and RESTFS and
they may end unsuccessfully for the same reason.
1.4.2.4 Insufficient memory
One of the most frequent problems that might occur is because of insufficient
memory, so many OS/390 UNIX users ask:
I am getting an insufficient memory message, how can I determine the value
specified for my region size?
The answer is:
If you are using the TSO OMVS command, your region size when coming in
through TSO is the REGION size you specify on your logon panel.
For batch, your region size is the REGION parameter in the JOB or STEP.
If you are running through rlogin or telnet, ask your system programmer about
increasing MAXASSIZE in the BPXPRMxx parmlib member.
A Web CGI process also gets its region size from the MAXASSIZE in the
BPXPRMxx parmlib member.
However, if your installation has an IEFUSI exit that changes region size, then the
kernel will honor whatever the exit requested. For example, you could have
MAXASSIZE set to 100M, but if your IEFUSI exit changes that to 40M, then you
get 40M.
Because of the way fork() works with IEFUSI exits, IEFUSI sees the region size
for a new job before the kernel can propagate the region size from the parent. To
prevent IEFUSI from seeing a region size of 0, see APAR OW32459 where a
solution is shown. This change was made in Release 6.
Users are encouraged to bypass IEFUSI processing for address spaces in the
OMVS subsystem. This can be done in either of the following ways:
• Exclude IEFUSI from processing address spaces in the OMVS SUBSYS. This
is done by modifying the SMFPRMxx parmlib member.
• The IEFUSI exit can recognize when it is running in an OMVS SUBSYS
address space and just leave the region size alone.
There is also a page on the Internet that contains the most frequently asked
questions. If you go to http://www.s390.ibm.com/products/oe/ and if
OS/390 UNIX System Services
29
you select FAQ on the left bar, you will find lots of questions asked by OS/390
UNIX end users and solutions provided by IBM.
1.5 UNIX System Services dump processing
For some problems in a UNIX System Services environment it may be helpful to
have a dump including the data spaces. These kinds of problems may be a hang
or enqueue problem.
1.5.1 How to provide a dump
Before you enter the dump command, check the address space ID or the job
name of the UNIX System Services address space using SDSF option DA display active users. Now issue the console command DISPLAY DUMP (/D D) to
get the dump data set status information.
You receive the following output:
ISFPCU41 ---------------- SDSF PRIMARY OPTION MENU -- COMMAND ISSUED
COMMAND INPUT ===> /D D
SCROLL ===> CSR
RESPONSE=SC63
IEE852I 09.11.44 SYS1.DUMP STATUS 859
SYS1.DUMP DATA SETS AVAILABLE=002 AND FULL=001
CAPTURED DUMPS=0000, SPACE USED=00000000M, SPACE FREE=00001536M
AUTOMATIC ALLOCATION IS: ACTIVE
NO SMS CLASSES DEFINED
AVAILABLE DASD VOLUMES: SBOX03
NAME=DUMP.D&MON.&DAY..H&HR..&SYSNAME..&JOBNAME..S&SEQ.
EXAMPLE=DUMP.D0729.H13.SC63.#MASTER#.S00000
SYS1.DUMP AVAILABLE DASD DATA SETS : 01,99
SYS1.DUMP FULL DASD DATA SETS : 00
If there are no empty dump data sets, you can issue the following commands to
clear one or all dump data sets:
•DD CLEAR,DSN=00
Clear one (00) data set
• DD CLEAR,DSN=ALL
Clear all data sets
It is very important to display the dump options. If you do not request useful dump
options, for example, CSA, RGN, etc., the dump may not be useful. Display the
dump options using the DISPLAY DUMP,OPTIONS command which shows the
following:
30
Debugging UNIX Applications on OS/390
ISFPCU41 ---------------- SDSF PRIMARY OPTION MENU -- COMMAND ISSUED
COMMAND INPUT ===> /D D,O
SCROLL ===> CSR
RESPONSE=SC63
IEE857I 09.22.54 DUMP OPTION 862
SYSABEND- ADD PARMLIB OPTIONS SDATA=(LSQA,TRT,CB,ENQ,DM,IO,ERR,SUM),
PDATA=(SA,REGS,LPA,JPA,PSW,SPLS)
SYSUDUMP- ADD PARMLIB OPTIONS SDATA=(SUM), NO PDATA OPTIONS
SYSMDUMP- ADD OPTIONS (NUC,SQA,LSQA,SWA,TRT,RGN,LPA,CSA,SUM,ALLNUC,
GRSQ)
SDUMP- ADD OPTIONS (ALLPSA,SQA,LSQA,RGN,LPA,TRT,CSA,SWA,SUMDUMP,
ALLNUC,Q=YES,GRSQ,COUPLE,XESDATA),TYPE=(XMEME),
BUFFERS=00000000K,MAXSPACE=00001536M,
MSGTIME=99999 MINUTES
SYSFAIL NO STRLIST OPTIONS
To request a dump, enter:
• DUMP COMM=(dump title)
for example (OMVS)
• reply id,JOBNAME=OMVS,DSPNAME=('OMVS'.*),END
The reply ID is shown after issuing the DUMP command
('OMVS'.*) means: dump all address spaces belonging to UNIX System
Services.
Note
After the dump processing ends, you should get a Dump complete message.
Here is the complete SYSLOG output:
COMMAND INPUT ===> /DUMP COMM=(OMVS)
*14 IEE094D SPECIFY OPERAND(S) FOR DUMP COMMAND
R 14,JOBNAME=(OMVS),DSPNAME=('OMVS'.*),END
IEE600I REPLY TO 14 IS;JOBNAME=(OMVS),DSPNAME=('OMVS'.*),END
IEA794I SVC DUMP HAS CAPTURED: 630
DUMPID=001 REQUESTED BY JOB (*MASTER*)
DUMP TITLE=OMVS
IEF196I IGD100I 01D2 ALLOCATED TO DDNAME SYS00006 DATACLAS (
IEF196I IEF285I SYS1.MCEVSM.DMP00001
IEF196I IEF285I VOL SER NOS= VSMCAT.
IEA611I COMPLETE DUMP ON SYS1.MCEVSM.DMP00001 634
DUMPID=001 REQUESTED BY JOB (*MASTER*)
FOR ASID (000E)
)
CATALOGED
For useful hints about IPCS commands to debug the dump or for a quick check,
refer to MVS Diagnosis: Reference, LY28-1084 (available to IBM licensed
customers only).
OS/390 UNIX System Services
31
32
Debugging UNIX Applications on OS/390
Chapter 2. Lotus Domino for S/390—troubleshooting/hints/tips
This chapter is written especially for those people who spend their time working
with implementing, customizing, and programming the Lotus Domino for S/390
server.
In this chapter you will learn what to do in case of errors shown in the MVS- or
Notes-Log, how to analyze this information, and use methods to determine the
source of the problem.
Also some common hints and tips are covered here. We divided this chapter into
logical environmental parts (OS/390, UNIX System Services, and Lotus Domino
for S/390) and some common parts (Hints and Tips, Troubleshooting documents,
and Information Sources).
Note
The following sections are mainly based on Lotus Domino Server S390 R5, but
most of the troubleshooting and hints and tips parts also apply to the 4.6.x
servers.
Here is an overview about the logical parts discussed in this chapter:
Table 10. Overview
OS/390
UNIX System
Services
Lotus
Domino
Hints and
Tips
Troubleshooting
Info
Sources
Domcon
nsd.sh
LE run-time
options
Editing
notes.ini
HTTP
APARs
IEFUSI
Interpreting
messages &
codes
Activating
JAVA
environment
Cleaning up
server
resources
SMTP
Web sites
Useful
commands
Useful
commands
notes.ini
parameters
Activating
HTTP
POP3
Redbooks
Syslog
Logs
Activating
POP3
Router
Other...
Dump options
Domino
console
Activating
SMTP
Agents
IPCS
Programs
How mail
routes
Calendaring &
scheduling
Transaction
logging
What info in
case of error
CEEDUMP
More...
© Copyright IBM Corp. 1999
33
2.1 OS/390
This section describes some main debugging issues for the Lotus Domino S/390
server in the OS/390 environment.
2.1.1 Domcon
This section is an excerpt from the Domcon Internet site:
http://www.s390.ibm.com/products/domino/domcon/dmcnmain.html
To get more information about Domcon, download the related database and have
a look at it. There is also a good description about Domcon in Chapter 8 of Lotus
Domino for S/390 R5 Installation,Customization and Administration, SG24-2083.
2.1.1.1 Overview
This package allows you to manage the Domino for S/390 server from the
operator's console. Also, the Domino for S/390 console is routed and displayed
on the operator's console. The OS/390 console already provides functions to
switch to alternate consoles in the event it becomes disconnected; typically,
OS/390 consoles are secured from unauthorized access. In addition, OS/390
functions such as SDSF and console can allow access to the OS/390 operator's
console from any authorized location/user ID.
The DOMCON package provides:
• An automated and synchronized way to start the Domino for S/390 server with
other resources at system IPL time.
• Management of all Domino partitioned servers on the OS/390 image from the
same console.
• Remote access with SDSF/console functions with appropriate controls.
• The ability for operations to send server commands to the Domino for S/390
server from the console.
• Operations with the capability of performing an orderly shutdown of the
Domino for S/390 server during OS/390 shutdown.
• An interface to automation systems that can start, stop, or send commands to
the server in response to error situations.
• RACF protected data set to store Domino passwords (server.id) and the ability
to automatically feed them into the server's input stream at startup. Automated
startups with secure server.id files.
• Synchronized S/390 resource utilization. Signal the Domino for S/390 server
to start replications at the completion of the batch processing (may be variable
on end-of-month or week).
• An interface from UNIX Services to send commands to your Domino Servers
and see the responses. RACF Surrogate authority is used to control which
users can access which servers.
The DOMCON package was built to be a nonintrusive monitor of your Domino for
S/390 server. This means that at any point in time you can start your Domino for
S/390 server without DOMCON. There are no changes to make and you will have
removed this package from the startup sequence of this server. For additional
34
Debugging UNIX Applications on OS/390
information about this feature refer to the Problem Determination section in the
database domcon.nsf.
2.1.1.2 Support
This package is provided on an as-is basis. IBM does not offer warranty, service,
or formal technical support.
2.1.1.3 Functions
DOMCON provides you with the following three command processes from the
S/390 operator's console:
• DOMINS—used to start a Domino for S/390 server and the monitor process to
route messages to the operator's console.
• DOMINK—used to shut down a Domino for S/390 server. It will also stop the
monitor process and clean any resources left over after a server has stopped.
• DOMINC—used to send a command from the console to a Domino for S/390
server.
DOMCON provides you with the following process from UNIX Services:
• DOMOE—used to send (if authorized) console commands to individual
Domino Servers and display its results from the server's log.
2.1.1.4 Download domcon.nsf
The DOMCON package comes as part of a Notes discussion database
(domcon.nsf). This discussion database will be updated on a weekly basis.
Please obtain a fresh copy to review any changes, fixes, or updates that are
made.
To download domcon.nsf, simply press the download button on page
www.s390.ibm.com/products/domino/domcon/dmcnmain.htm as shown in Figure 3 on
page 36.
Lotus Domino for S/390—troubleshooting/hints/tips
35
Figure 3. Downloading domcon.nsf
2.1.2 IEFUSI
We have encountered many different storage problems coming up due to the use
of SMF exit IEFUSI. These problems can be a simple message like insufficient
memory as well as a Domino server abend with user abend code U034 RC
070E0303. This happens if IEFUSI limits or changes the region size for OMVS
work.
Prior to OS/390 R6, IEFUSI saw a value of 0, which implied a 2 GB region. Many
customers’ IEFUSI exits saw that value and would then set the region size lower.
OMVS code would then use the new region value instead of propagating the
value from the parent. This caused OMVS address spaces to not run properly.
In OS/390 R6, code was added to show a value of 54 MB for the region size for
OMVS processes. As long as the exit did not touch the region size, OMVS would
then propagate the parent's region size to the FORKed/SPAWNed process.
To check if this exit is active in your environment, you can use the OS/390 system
DISPLAY command as shown in Figure 4 on page 37 and Figure 5 on page 37.
36
Debugging UNIX Applications on OS/390
This is screen. COMMAND INPUT ===> /D PROG,EXIT
RESPONSE=SC63
CSV460I 14.56.25 PROG,EXIT DISPLAY 131
EXIT
DEF EXIT
DEF EXIT
SYS.IEFACTRT
E SYSSTC.IEFACTRT E SYS.IEFUJI
SYSSTC.IEFUJI
E SYS.IEFU83
E SYSSTC.IEFU83
CSVDYLPA
E IEASDUMP.QUERY
E IEASDUMP.GLOBAL
IEASDUMP.LOCAL
E IEASDUMP.SERVER E IXC_ELEM_RESTART
IXC_WORK_RESTART E IEF_ALLC_OFFLN
E IEF_SPEC_WAIT
IEF_VOLUME_ENQ
E IEF_VOLUME_MNT
E IEFDB401
IEFJFRQ
E SYSSTC.IEFUSO
E SYSSTC.IEFUJP
SYSSTC.IEFU85
E SYSSTC.IEFU84
E SYSSTC.IEFU29
SYS.IEFU29
E SYS.IEFUTL
E SYS.IEFUSO
SYS.IEFUJP
E SYS.IEFUSI
E SYS.IEFUJV
SYS.IEFU85
E SYS.IEFU84
E IRREVX01
DEF
E
E
E
E
E
E
E
E
E
E
E
Figure 4. Display of active exits
COMMAND INPUT ===> /D PROG,EXIT,EXITNAME=SYS.IEFUSI,DIAG
RESPONSE=SC63
CSV464I 16.03.04 PROG,EXIT DISPLAY 354
EXIT SYS.IEFUSI
MODULE
STATE EPADDR
LOADPT
LENGTH
JOBNAME
IEFUSI
A 84197888 00000000 00000000 *
Figure 5. Display of IEFUSI exit
Currently, there is a recommendation in the UNIX System Services Planning
book that the IEFUSI exit should not change the region size for OMVS work:
Chapter - Customizing OS/390 UNIX
Section - Customizing the BPXPRMxx Parmlib Member
SubSection - MAXASSIZE
The IEFUSI user exit can modify the region size of an
address space. Users are strongly discouraged from altering
the region size of address spaces in the OMVS subsystem
category.
2.1.2.1 How to find your region size
• If you are using the TSO OMVS command, your region size when coming in
through TSO is the REGION size you specify on your logon panel.
• For batch, your region size is the REGION parameter in the JOB or STEP.
• If you are running through rlogin or telnet, ask your system programmer about
increasing MAXASSIZE in the BPXPRMxx parmlib member.
• A Web CGI process also gets its region size from the MAXASSIZE in the
BPXPRMxx parmlib member.
However, if your installation has an IEFUSI exit that changes region size, then
the kernel will honor whatever the exit requested. For example, you could have
MAXASSIZE set to 100M, but if your IEFUSI exit changes that to 40 MB, then you
get 40 MB.
Lotus Domino for S/390—troubleshooting/hints/tips
37
Users are encouraged to bypass IEFUSI processing for address spaces in the
OMVS subsystem. This can be done in either of these two ways:
1. The IEFUSI exit can recognize when it is running in an OMVS SUBSYS
address space and just leave the region size alone. The documentation for
IEFUSI states that word 8 contains the subsystem name. The exit should
check whether the name is OMVS, if so do nothing.
2. Exclude IEFUSI from processing address spaces in the OMVS SUBSYS. This
is done by modifying the SMFPRMxx parmlib member.
For more information on this refer to OS/390 MVS Installation Exits, SC28-1753,
APAR OW38477, and APAR OW32459.
2.1.3 OS/390 commands
There are several OS/390 system commands which can be very useful to retrieve
information about system parameters being set during initialization, to check the
status of processes and threads, or to change parameters dynamically. In this
section, we describe some of the commands we thought were the most
interesting. For a complete list of all parameters of the commands, refer to
OS/390 MVS System Commands, GC28-1781.
2.1.3.1 The OS/390 DISPLAY system command
You can use the OS/390 DISPLAY command to obtain:
• OS/390 UNIX System Services status information (for example, active or
terminating)
• Hierarchical File System (HFS) information
• OS/390 UNIX System Services process information for address spaces
• The current setting for all UNIX System Services parmlib statements
• Information about multiple parmlib members
To display process information for all UNIX System Services address spaces:
COMMAND INPUT ===> /D OMVS,A=ALL
RESPONSE=SC63
BPXO040I 11.15.22 DISPLAY OMVS 705
OMVS
000F ACTIVE
OMVS=(7C)
USER
JOBNAME ASID
PID
PPID STATE START
CT_SECS
OMVSKERN BPXOINIT 00FB
1
0 MR
09.42.31
8.239
LATCHWAITPID=
0 CMD=BPXPINPR
SERVER=Init Process
AF= 13 MF=00000 TYPE=FILE
TCPIPMVS TCPIPMVS 0039 83886082
1 1R
11.29.21
117.949
LATCHWAITPID=
0 CMD=EZBTMCTL
TCPIPMVS TCPIPMVS 0039 117440515
1 1F
11.29.21
117.949
LATCHWAITPID=
0 CMD=EZASASUB
Figure 6. Display all UNIX System Services address spaces
38
Debugging UNIX Applications on OS/390
To display detailed file system information on currently mounted files:
COMMAND INPUT ===> /D OMVS,FILE
RESPONSE=SC63
BPXO044I 11.20.57 DISPLAY OMVS 768
OMVS
000F ACTIVE
OMVS=(7C)
TYPENAME DEVICE ----------STATUS----------TFS
6 ACTIVE
NAME=/TMP
PATH=/tmp
MOUNT PARM=-s 500
HFS
19 ACTIVE
NAME=OMVS.STYRES1.HFS1
PATH=/u/guts
HFS
18 ACTIVE
NAME=OMVS.DOMINO5.DOMINO4.MAIL.HFS
PATH=/domino4/notesdata/mail .
MODE QJOBNAME QPID
RDWR
READ
RDWR
Figure 7. Display currently mounted file systems
To display all options set during initialization by the parmlib member BPXPRMXX
or with the OS/390 SETOMVS command:
COMMAND INPUT ===> /D OMVS,O
RESPONSE=SC63
BPXO043I 11.25.33 DISPLAY OMVS 772
OMVS
000F ACTIVE
OMVS=(7C)
OPENEDITION MVS CURRENT CONFIGURATION SETTINGS:
MAXPROCSYS
=
300
MAXPROCUSER
MAXFILEPROC
=
65535
MAXFILESIZE
MAXCPUTIME
= 2147483647
MAXUIDS
MAXRTYS
=
256
MAXPTYS
MAXMMAPAREA
=
4096
MAXASSIZE
MAXTHREADS
=
1000
MAXTHREADTASKS
MAXCORESIZE
=
4194304
MAXSHAREPAGES
IPCMSGQBYTES
=
262144
IPCMSGQMNUM
IPCMSGNIDS
=
20000
IPCSEMNIDS
IPCSEMNOPS
=
32767
IPCSEMNSEMS
IPCSHMMPAGES
=
25600
IPCSHMNIDS
IPCSHMNSEGS
=
1000
IPCSHMSPAGES
SUPERUSER
= BPXROOT
FORKCOPY
STEPLIBLIST
=
USERIDALIASTABLE=
PRIORITYPG VALUES: NONE
MAXQUEUEDSIGS =
1000
=
10125
= NOLIMIT
=
50
=
256
= 2147483647
=
1000
= 32768000
=
10000
=
20000
=
25
=
20000
=
2621440
= COPY
Figure 8. Display the current settings of BPXPRMxx parameters
Lotus Domino for S/390—troubleshooting/hints/tips
39
To display the thread information for the process ID 1:
COMMAND INPUT ===> /D OMVS,PID=1
RESPONSE=SC63
BPXO040I 11.45.27 DISPLAY OMVS 784
OMVS
000F ACTIVE
OMVS=(7C)
USER
JOBNAME ASID
PID
PPID STATE START
OMVSKERN BPXOINIT 00FB
1
0 MR
09.42.31
LATCHWAITPID=
0 CMD=BPXPINPR
SERVER=Init Process
AF= 13 MF=00000
THREAD_ID
TCB@
PRI_JOB USERNAME ACC_TIME SC
08B1AF0000000000 008E2A70 OMVS
.316 WAT
08B1BAE000000001 008E28D8
.258 VRT
08B1C6C000000002 008E4E88 OMVS
.001 KIN
08B1D2A000000003 008E4CF0
7.343 BND
08B5CFE000000004 008E26B8
.125 VRW
SCROLL ===
CT_SECS
8.279
TYPE=FILE
STATE
W
Y
K
Y
Y
Figure 9. Display thread information for a process ID
2.1.3.2 The OS/390 SETOMVS system command
Use the SETOMVS command to change dynamically the options that OS/390
UNIX System Services currently is using. These options are originally set in the
BPXPRMxx parmlib member at the time of initially program loading the system.
Figure 8 shows the parameters that can be changed dynamically.
2.1.3.3 The OS/390 SET command
In a Lotus Domino environment you may use the SET command to:
• Dynamically add modules to, or remove modules from the LPA
• SET PROG=xx: The two alphanumeric characters indicating the PROGxx
member of the logical parmlib containing definitions that control the
addition of modules to, and the removal of modules from the LPA after IPL.
• Dynamically change the configuration of OS/390 UNIX System Services
system characteristics
• SET OMVS=(xx[,xx...,nn]): The two alphanumeric characters that specify
one or more parmlib members in BPXPRMxx. If you specify only one
member, putting parentheses around the member is optional. If you specify
more than one parmlib member, you must put parentheses around the
members.
2.1.4 SYSLOG
In cases when the server cannot be started or an error occurs during start of the
server, it is very often helpful to check the OS/390 system log for error messages
or any other hints such as permission problems. These can be authority problems
with the access to any file or directory that is shown in detail by a RACF message
like the example here:
ICH408I USER(DOMINO4 ) GROUP(OEDOMINO) NAME(####################) 576
/u/domino4/domino4.id
CL(FSOBJ
) FID(01E2C2D6E7F0F6412804000000130000)
INSUFFICIENT AUTHORITY TO OPEN
ACCESS INTENT(RW-) ACCESS ALLOWED(OTHER ---)
or any kind of authorization problem shown in the next example:
40
Debugging UNIX Applications on OS/390
+WSC/DOMINO1 2A88F4B8: HTPasswd...user"dsynb".pwlen 6, newpwlen 0,
failed SAF.Errno:157 Errno2:090c02af,MSG:
+WSC/DOMINO1 EDC5157I An internal error has occurred
There are many other situations where the log may be a good point to start
problem determination.
2.1.5 Dump options/IPCS
2.1.5.1 U034
In case of an abendu034, it is very often necessary to get a slip dump. This topic
describes how to set a valid slip trap to catch a dump for the abend. We
recommend that you place your slip command into a parmlib member IEASLPxx
and enter the command SET SLIP=xx to activate the slip. A dump is written to
your system dump data sets. The following slip should be used to capture a
dump:
Figure 10. Sample IEASLP01 member
This slip can then be activated using the command SLIP SET=01 on the system
console. After you captured the dump, you can disable this slip using command
SLIP MOD,ID=U034,DISABLE. To delete the slip from your system use command
SLIP DEL,ID=U034. To check which slips are active the display system command
may be used as shown in Figure 11 on page 42.
Lotus Domino for S/390—troubleshooting/hints/tips
41
COMMAND INPUT ===> /D SLIP
RESPONSE=SC63
IEE735I 12.26.25 SLIP DISPLAY 021
ID STATE
ID STATE
ID STATE
X013 ENABLED X028 ENABLED X052 ENABLED
X070 ENABLED X073 ENABLED X0DX ENABLED
X13E ENABLED X1C5 ENABLED X222 ENABLED
S3C4 ENABLED X422 ENABLED X47B ENABLED
X804 ENABLED X806 ENABLED X80A ENABLED
X9FB ENABLED XB37 ENABLED XC1A ENABLED
XE37 ENABLED XEC6 ENABLED XXC6 ENABLED
SCROLL ==
ID
X058
X0E7
X322
X622
X81A
XD1A
U034
STATE
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ID
X066
X0F3
X33E
X71A
X91A
XD37
STATE
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
Figure 11. Display of all active slips on a system
2.1.5.2 Interactive Problem Control System (IPCS)
Once you retrieved a dump you may use IPCS to format the dump to get the
necessary information out of it. Some information about IPCS commands and
what information can be gathered here are described on the following pages.
IPCS example
Here is an example of how to come closer to a problem source using IPCS:
Following error (shown in the Notes Log database log.nsf):
Thread=Ý1157627992:1476460551¨
PANIC: Unable to perform control function on semaphore
PANIC ENTRY VALUES: decimal errno=111 / hexadecimal errno2=70E0303
CEE3250C The system or user abend U 034 R=070E0303 was issued.
From compile unit ./ospanic.c at entry point Panic at compile unit
offset +00041C4E at address 1AA1B696.
First the messages have to be analyzed where errno=111 and errno2=070E0303
point to an ACCESS/PERMISSION DENIED problem. To find out how to read
these messages, see 2.2.2, “Return codes and how to interpret a USS message”
on page 49.
If a slip dump is available, load it in Interactive Problem Control System (IPCS).
To determine how a slip dump is set, see 2.1.5.1, “U034” on page 41.
Type IPCS command IP SUMMARY FORMAT.
The output of the IP SUMMARY FORMAT command shows several control blocks such
as ASCBs, ASXBs, or TCBs. To find further information, type the command such
as FIND ’T C B’ to show the TCB SUMMARY. Here you can see TCB (the one
related to the Abend Code - what is decimal 34 and hex 22 in this example),
Jobname, and ASID. This is shown in the following figure (where the TCB has a
value of x’006D78B0’, Jobname is CDOMINS5, and the ASID is x’01EB’):
42
Debugging UNIX Applications on OS/390
IPCS OUTPUT STREAM -----------------------------------------------------------Command ===>
* * * *
T C B
S U M M A R Y
* * * *
JOB CDOMINS5 ASID 01EB ASCB 00E93700 FWDP 00E52880 BWDP 00F96A00 PAGE 00000006
TCB AT
CMP
NTC
OTC
LTC
TCB
BACK
PAGE
006FE240 00000000 00000000 00000000 006FDE88 006FF150 00000000 00000032
006FF150 00000000 00000000 006FE240 00000000 006FDE88 006FE240 00000036
006FDE88 00000000 006FF150 006FE240 006DE9B0 006DE9B0 006FF150 00000039
006DE9B0 00000000 00000000 006FDE88 006DE110 006DE110 006FDE88 00000042
006DE110 00000000 00000000 006DE9B0 006A95E0 006D8B68 006DE9B0 00000046
006D8B68 00000000 00000000 006DE110 00000000 006D88D8 006DE110 00000055
006D88D8 00000000 006D8B68 006DE110 00000000 006D8740 006D8B68 00000058
006D8740 00000000 006D88D8 006DE110 00000000 006D8330 006D88D8 00000061
006D8330 00000000 006D8740 006DE110 00000000 006D8070 006D8740 00000064
006D8070 00000000 006D8330 006DE110 00000000 006D7C68 006D8330 00000066
006D7C68 00000000 006D8070 006DE110 00000000 006D7AD0 006D8070 00000068
006D7AD0 00000000 006D7C68 006DE110 00000000 006D78B0 006D7C68 00000071
006D78B0 84000022 006D7AD0 006DE110 00000000 006D7718 006D7AD0 00000074
006D7718 00000000 006D78B0 006DE110 00000000 006D7580 006D78B0 00000080
006D7580 00000000 006D7718 006DE110 00000000 006D7360 006D7718 00000082
006D7360 00000000 006D7580 006DE110 00000000 006D71C8 006D7580 00000084
006D71C8 00000000 006D7360 006DE110 00000000 006D6E88 006D7360 00000086
006D6E88 00000000 006D71C8 006DE110 00000000 006D6838 006D71C8 00000088
006D6838 00000000 006D6E88 006DE110 00000000 006D65A8 006D6E88 00000090
Now note the TCB and the ASID and use the IPCS command ip omvsdata to get
more information. After this command type find ’* PROCESS’ to get the PROCESS
SUMMARY REPORT. Here you can find out more information related to the ASID
noted before like process ID, parent PID, user ID, and so on. See the following
figure (where the process ID is 45000058 - belonging to the ASID found out
before):
***** PROCESS SUMMARY REPORT *****
Process
Userid
Asid
Parent
Process
Session
Status
ID
PID
Group ID
ID
--------------------------------------------------------------------------00000001
ESSTC
0019
00000000
00000001
00000001 01000002
EOSTC
008B
00000001
01000002
01000002 23000003
CDOMT01
0093
68000044
69000057
69000057 53000004
CDOMT01
0056
68000044
69000057
69000057 00000005
ESSTC
002D
00000001
00000005
00000005 .
.
.
.
.
.
69000057
EDOMCON
02D5
00000001
69000057
69000057 .......
45000058
CDOMT01
01EB
68000044
69000057
69000057 56000059
CDOMT01
0092
68000044
69000057
69000057 1600005A
ESSTC
019F
00000001
1600005A
1600005A -
In the PROCESS SUMMARY REPORT you also have the possibility to step the
process chain back by checking the PPIDs to come to the first PID in chain
finally.
The Process ID determined previously is needed for the next step. This is
command ip omvsdata detail. After this command type find Process ID (what is
Lotus Domino for S/390—troubleshooting/hints/tips
43
"45000058" in our example). Now a lot of details about the failing process are
shown. In our example we had an Access denied problem and the Memory Map
of the Process shows the files that could have caused this problem (for example,
wrong permissions bits set). See the figure below:
Process ID: 45000058
Status: Active
Last exec() Program Name:
/usr/lpp/lotus/notes/latest/os390/http
ID Data:
Userid: CDOMT01
Parent PID: 68000044
Process Group ID: 69000057
Memory Map Files:
Asid: 01EB
Ptrace Parent PID: 00000000
Session ID: 69000057
GYMM: 1D372AB0
Origin: 1CAE1000
Length:
0047F000
Offset: 00000000
Type:
SHARED
Subranges:
1
/usr/lpp/lotus/notes/latest/os390/ltscsc13.tlb
GYMM: 1D3729A0
Origin: 1CF60000
Length:
00061000
Offset: 00000000
Type:
SHARED
Subranges:
1
/usr/lpp/lotus/notes/latest/os390/res/en_US/strings.res
GYMM: 1D372890
Origin: 1CFC1000
Length:
00004000
Offset: 00000000
Type:
SHARED
Subranges:
1
/usr/lpp/lotus/notes/latest/os390/res/en_US/httprs.res
Further information using IPCS
With the IPCS command ip omvsdata some helpful information about startup
options can be found. Typing find Startup after the command ip omvsdata
shows the following:
IPCS OUTPUT STREAM ---------------------------------------------------Command ===> ip omvsdata
Startup options
Parmlib member:
BPXPRM00
CTRACE parmlib member:
CTIBPX00
Maximum processes on system:
44
200
Maximum users on system:
50
Maximum processes per user id:
25
Maximum thread tasks per process:
1,000
Maximum threads per process:
1,000
Maximum allocated files per process:
1,012
Maximum growth for pseudo-terminal sessions:
512
Maximum growth for remote-terminal sessions:
512
Debugging UNIX Applications on OS/390
Another helpful command is ip verbx ledata to show the definite HEAP
parameter values in the Language Environment (LE) as shown in the following
figure:
IPCS OUTPUT STREAM ---------------------------------------------------Command ===> ip ledata
PROGRAM INVOCATION
OVR
ALL31(ON)
PROGRAM INVOCATION
OVR
ANYHEAP(00020000,00016384,ANY ,FREE)
INSTALLATION DEFAULT
OVR
NOAUTOTASK
PROGRAM INVOCATION
OVR
BELOWHEAP(00490000,00004096,FREE)
INSTALLATION DEFAULT
OVR
CBLOPTS(ON)
INSTALLATION DEFAULT
OVR
CBLPSHPOP(ON)
INSTALLATION DEFAULT
OVR
CBLQDA(OFF)
INSTALLATION DEFAULT
OVR
CHECK(OFF)
INSTALLATION DEFAULT
OVR
COUNTRY(US)
INSTALLATION DEFAULT
OVR
NODEBUG
INSTALLATION DEFAULT
OVR
DEPTHCONDLMT(00000000)
INSTALLATION DEFAULT
OVR
ENVAR("")
INSTALLATION DEFAULT
OVR
ERRCOUNT(00000000)
INSTALLATION DEFAULT
OVR
ERRUNIT(00000006)
INSTALLATION DEFAULT
OVR
FILEHIST
DEFAULT SETTING
OVR
NOFLOW
PROGRAM INVOCATION
OVR
HEAP(01984384,00032768,ANY ,
KEEP,00016384,00004096)
INSTALLATION DEFAULT
OVR
HEAPCHK(OFF,00000001,00000000)
INSTALLATION DEFAULT
OVR
HEAPPOOLS(OFF,
00000008,00000010,
00000032,00000010,
00000128,00000010,
2.1.6 CEEDumps
In case of abending situations in the Lotus Domino server (for example, panics or
ABENDU034) often CEEDumps are written. These can be found in the
.../notesdata directory and are required for problem determination by support
personnel. How to debug CEEDumps is described in Chapter 8, “Language
Environment for OS/390” on page 275.
2.2 UNIX System Services
This section describes some main debugging issues for the Lotus Domino S390
server in the UNIX System Services environment.
2.2.1 nsd.sh
This section describes the nsd.sh shell script, how to use it and what can be
found in the output.
2.2.1.1 What is the nsd.sh shell script
The nsd.sh shell script is used to gather information about your Domino Server. It
was originally designed to be run with various UNIX systems and has been
enhanced for use with UNIX System Services on the OS/390 platform.
Today NSD collects the following data:
• Environment variables
Lotus Domino for S/390—troubleshooting/hints/tips
45
• Job call stack
• Job locks
• Display open files
• IPC statistics
• Shared memory
• Contents of the notes.ini file
• Process list
• Listing of the Domino directory and its subdirectories
• System information like host name and number of file descriptors
• General information about the script itself (version)
• TCP/IP information based on /etc/resolv.conf
2.2.1.2 Where can you find NSD on your system
NSD is located in the /usr/lpp/lotus/bin/tools/diag directory. The output of the
script can be found by default in your notes data directory. You can set the
environment variable NSD_LOGDIR to redirect the output to any other directory
where you want it to be automatically saved. Make sure the user issuing this
command has write access to this directory.
The output is saved in a log file with the following structure:
<nsd_option>_<OS name>_<hostname>_<timestamp>.log
Here are examples for nsd.sh -kill and nsd.sh -info files:
kill_OS390_SC63_07_14@14_46.log
sysinfo_OS390_SC63_07_14@15_19.log
2.2.1.3 How to run nsd.sh
The command nsd.sh should be run through UNIX System Services from the
notes data directory. You may add the complete path to the PATH environment
variable in the file .profile located in your home directory or issue the command
as shown in Figure 12.
HAUFF @ SC63:/domino4/notesdata>/usr/lpp/lotus/bin/tools/diag/nsd.sh
INFO: Using cache file /tmp/nsd.AK/nsd_3.2.7_cache.ins
Script Version : /usr/lpp/lotus/bin/tools/diag/nsd.sh 3.2.7
Notes Version : Release 5.0a (Intl) 14 June 1999
Notes Base
: 5.0
Data Dir
: /domino4/notesdata
Notes Exec Dir : /usr/lpp/lotus/notes/latest/os390
Search Path
: /usr/lpp/lotus/notes/latest/os390 /usr/lpp/lotus/notesapi
Debugger
: /bin/dbx
Debugger Version: Compiled: Feb 23 1999 11:09:56 GMT as BFP
Script Dir
: /usr/lpp/lotus/bin/tools/diag
Host Info
: OS/390 SC63 07.00 02 9672
User
: AK (AK)
Date
: Wed Jul 14 13:55:13 EDT 1999
Clearcase View : ins
Input arguments :
Figure 12. Partial output of nsd.sh
46
Debugging UNIX Applications on OS/390
2.2.1.4 Parameters of the nsd.sh script
There are several parameters available for nsd.sh. To get a complete and actual
list of them run /usr/lpp/lotus/bin/tools/diag/nsd.sh -help for all available
options as shown in Figure 13.
The /usr/lpp/lotus/bin/tools/diag/nsd.sh -info command gives you most of the
information needed to get a quick overview of your environment:
DOMINO4 @ SC63:/domino4/notesdata>/usr/lpp/lotus/bin/tools/diag/nsd.sh -help
Usage: /usr/lpp/lotus/bin/tools/diag/nsd.sh [options] [ core_file | pid ]
Options:
-batch
-info
-noinfo
-nolog
-ver*sion
-ps
-kill
-memcheck
-nomemcheck
-lsof
-nolsof
-user <user_id>
-exec_path <dir[:dir]*>
-filter <log_file>
-help
-help <option>
-help gen*eral
-help lim*itations
-help update
(run in batch mode -- don't write to tty)
(just report system info)
(don't report system info)
(don't log output to log file)
(just show version header)
(show process tree)
(kill all/user notes processes and cleanup IPCs)
(run the Notes memory checker only)
(don't run the Notes memory checker by default)
(run lsof only -- list Notes open files)
(don't run lsof by default)
(operate only on notes process run by 'user_id')
(add additional directories to the search path)
(filter stack output of log_file)
(show this help list)
(where option is any one of the above)
(general info about the script and how it works)
(general info on script limitations)
(list script version update info)
Figure 13. List of all available parameters for the nsd.sh script
Note
There have been cases where nsd.sh did not work due to a DBX issue, which
requires a fix. DBX fixes can be downloaded from the Internet home page:
http://www.s390.ibm.com/oe
2.2.1.5 Terminating the server using option nsd -kill
Domino R5 no longer provides the killnotes script to terminate the server. It has
been replaced by the -kill option of the nsd.sh. To run this you must be logged on
with the user ID of the domino server you want to terminate. Invoke the command
as shown in Figure 14. Ensure that you are in the notes data directory.
Lotus Domino for S/390—troubleshooting/hints/tips
47
DOMINO4 @ SC63:/u/domino4>cd /domino4/notesdata
DOMINO4 @ SC63:/domino4/notesdata>/usr/lpp/lotus/bin/tools/diag/nsd.sh -kill
INFO: Using cache file /tmp/nsd.DOMINO4/nsd_3.2.7_cache.ins
Script Version : /usr/lpp/lotus/bin/tools/diag/nsd.sh 3.2.7
Notes Version :
Notes Base
: UNKNOWN
Data Dir
: /domino4/notesdata
Notes Exec Dir : /usr/lpp/lotus/notes/latest/os390
Search Path
: /usr/lpp/lotus/notes/latest/os390 /usr/lpp/lotus/notesapi
Debugger
: /bin/dbx
Debugger Version: Compiled: Feb 23 1999 11:09:56 GMT as BFP
Script Dir
: /usr/lpp/lotus/bin/tools/diag
Host Info
: OS/390 SC63 07.00 02 9672
User
: DOMINO4 (DOMINO4)
Date
: Wed Jul 14 14:08:11 EDT 1999
Clearcase View : ins
Input arguments :
INFO: Killing /usr/lpp/lotus/notes/5001/os390/http 285212688
INFO: Killing /usr/lpp/lotus/notes/5001/os390/replica 872415250
INFO: Killing /usr/lpp/lotus/notes/5001/os390/stats 1207959572
INFO: Killing /usr/lpp/lotus/notes/5001/os390/update 251658262
INFO: Killing /usr/lpp/lotus/notes/5001/os390/adminp 771751961
INFO: Killing /usr/lpp/lotus/notes/5001/os390/sched 268435484
INFO: Killing /usr/lpp/lotus/notes/5001/os390/amgr 553648157
INFO: Killing /usr/lpp/lotus/notes/5001/os390/amgr 234881063
INFO: Killing /usr/lpp/lotus/notes/5001/os390/server 1744830494
INFO: Killing /usr/lpp/lotus/notes/5001/os390/event 318767142
INFO: Killing /usr/lpp/lotus/notes/5001/os390/calconn 838860840
INFO: Killing /usr/lpp/lotus/notes/5001/os390/smtp 134217775
INFO: Removing Share Mem IPC keys 0xf83a8000 0xf83a8002 0xf83a8003 0xf83a8001
INFO: Removing Semaphore IPC keys 0xf83a8005 0xf83a8000 0xf83a8006 0xf83a8001
Remaining IPC Keys (Notes IPC keys start with 0xf8)
Message Queues:
Shared Memory:
T
ID
KEY
m
957508 0xf83a8000
m
760901 0xf83a8002
m
957510 0xf83a8003
m
564295 0x010d0144
m
957512 0xf83a8001
MODE
--rw-rw-----rw-rw-----rw-rw-----rw-rw-rw--rw-rw----
OWNER
DOMINO4
DOMINO4
DOMINO4
DOMINO4
DOMINO4
GROUP
OEDOMINO
OEDOMINO
OEDOMINO
OEDOMINO
OEDOMINO
Semaphores:
T
ID
KEY
MODE
OWNER
GROUP
s
1068581 0x020d0145 --ra-ra-ra- DOMINO4 OEDOMINO
s
1068582 0x030d0145 --ra-ra-ra- DOMINO4 OEDOMINO
s
1003051 0x010d0145 --ra-ra-ra- DOMINO4 OEDOMINO
Note:
You can set the environment variable NSD_LOGDIR to point
to the directory where you want your logs/cores to be
automatically saved
Run /usr/lpp/lotus/bin/tools/diag/nsd.sh -help for more info on new options/
features
Figure 14. Terminating the server with option nsd.sh -kill
48
Debugging UNIX Applications on OS/390
2.2.2 Return codes and how to interpret a USS message
How can a Lotus Domino message be interpreted? The following example shows
such a message coming up in the notes log (log.nsf):
PANIC: fatal_error signal handler
PANIC ENTRY VALUES: decimal errno=116 / hexadecimal errno2=57C006C
Figure 15. Error message
2.2.2.1 Explanations of errno
• errno=116: This errno is a decimal USS return code and can be found in
OS/390 UNIX System Services Messages and Codes, SC28-1908. The codes
described there are represented in decimal and hex values. So look for "116"
decimal) or "0074" (hex) in the manual and you will find:
Table 11. |USS errno
Decimal value
Hex value
Return Code
Description
116
0074
EDEADLK
A resource deadlock is avoided
The USS message manual also contains the list shown in Table 12.
Table 12. List of Return Codes
Decimal value
Hex value
Return Code
Description
1
0001
EDOM
Error in the domain
2
0002
ERANGE
Result is too large
111
006F
EACCES
Permission is denied
112
0070
EAGAIN
The resource is temporarily
unavailable
113
0071
EBADF
The file descriptor is incorrect
114
0072
EBUSY
The resource is busy
115
0073
ECHILD
No child process exists
116
0074
EDEADLK
A resource deadlock is avoided
......
.......
.......
...................................
1155
0483
ETcpBadObj
Streams push object error
1156
0484
ETcpClosed
Streams closed error
1157
0485
ETcpLinked
Streams link error
1158
0486
ETcpErr
TCP error
1159
0487
EINTRNODATA
Accept_and_receive is
interrupted after the connection
arrived but before the first data
arrived
1160
0488
ENOREUSE
Socket descriptor reuse is not
supported
Lotus Domino for S/390—troubleshooting/hints/tips
49
2.2.2.2 Explanations of errno2
• errno2() =57C006C: This errno2 is divided into two parts:
Table 13. USS errno2
cccc
rrrr
057C
006C
where
• cccc is a halfword reason code qualifier
• rrrr is the halfword reason code described in the message manual
The two high-order bytes of the reason codes returned by OS/390 UNIX
contain a value that is used to qualify the contents of the two low-order bytes.
If the contents of the two high-order bytes are within the range of X’0000’ to
X'20FF', the error represented by the reason code is defined by OS/390 UNIX.
If the contents of the two high-order bytes is outside the range, the error
represented by the reason code is not an OS/390 UNIX reason code.
Because the hexadecimal value "057C" (halfword reason code qualifier) is
within the range of X’0000’ to X'20FF' in the preceding example, code "006C"
is an OS/390 UNIX reason code. Looking for "006C" in OS/390 UNIX System
Services Messages and Codes, SC28-1908, results in:
Table 14. Reason Code example 006D
006D
JRFileNotThere
The requested file does not exist.
Action: The service cannot be performed unless the
named file exists.
Table 15 shows a partial list of reason codes. For a complete list, refer to the USS
messages manual.
Table 15. List of Reason Codes
Reason Code
Decimal
Hex
JRAccept
531
0213
JRAccess
331
014B
...........
......
.......
JRWriteUserStorageFailed
448
01C0
29188
7204
317
013D
JRWrongKey
1125
0465
JRWrongPID
557
022D
JRZeroOrNegative
846
034E
JRWrongBand
JRWrongInstance
50
Debugging UNIX Applications on OS/390
2.2.3 Useful UNIX commands
To navigate in the UNIX System Services environment, it is very helpful to have a
short reference for the most common UNIX commands. The following list covers
most of them in an alphabetical order:
Table 16. Useful UNIX commands
Command
Short description
Example
cd
Changes the current directory.
cd /usr/lpp/lotus
chmod
Changes the access permissions or modes of the
specified file or directory. Modes are read(r),
write(w), and execute(x).
chmod -w /usr/lpp/test
chown
Changes the owner or group of a file or directory.
chown 755 /etc/profile
cp
Copies files to a target named by the last
argument.
cp file1 file2
df
Displays the amount of free space in a file system
df -Pk
and the total amount available. The -Pk options
list complete information on the space used, in
the following order:
File system name
Total space
Space used
Space free
Percentage of space used
File system root
echo
Writes argument to standard output.
echo $PS1
env
Displays or sets the environment variables.
env
export
Format: export name=value
export
PATH=/usr/lpp:/notesda
ta:$PATH
Marks each variable name so that the current
shell makes it automatically available to the environment of all commands run from that shell.
find
Find searches a given file hierarchy specified by
path, finding files that match the criteria given by
expression.
find . -name test.c
kill
Ends a process by sending a signal.
kill 715 (715 is a
processid )
ls
Lists files and directories. The -l option displays
ls -l
permissions, links, owner, group, size, time, and
name.
mkdir
Creates a new directory.
mkdir /usr/lpp/lotus/test
mv
Rename or move a file or directory.
mv test1 test2
ps
Displays a list of currently running processes.
ps -ef
pwd
Displays the fully qualified pathname of the current
working directory.
pwd
rm
Removes a file or directory.
rm -r /usr/lpp/lotus/test
Lotus Domino for S/390—troubleshooting/hints/tips
51
Command
Short description
Example
su
Switches to superuser or another ID.
su uid / use exit to
switch back
For a complete list of all available commands and parameters, please refer to
OS/390 UNIX System Services User's Guide, SC28-1891.
Here are output examples of the commands:
df command
HAUFF @ SC63:/u/hauff>df -P /usr/lpp/lotus
Filesystem
512-blocks
Used Available Capacity Mounted on
OMVS.DOMINO5.PROD.HFS 784800
695968
88832
89% /usr/lpp/lotus
Figure 16. Sample output of the UNIX command df -P
env command
DOMINO4 @ SC63:/u/domino4>env
Mail=/usr/mail/DOMINO4
HOSTNAME=SC63
PATH=/bin:.:/usr/lpp/java/J1.1/lib/mvs/native_threads:/usr/lpp/lotus/bin:/usr/lp
p/lotus/bin/tools:
SHELL=/bin/sh
_WORK_UNIT=SYSDA
_INCDIRS=/usr/include /usr/lpp/ioclib/include
Notes_Directory=/domino4/notesdata
_PLIB_PREFIX=CEE
PS1=$LOGNAME @ $HOSTNAME:$PWD>
_CEE_RUNOPTS=AL(ON) ENVAR(_BPXK_SETIBMOPT_TRANSPORT=TCPIPMVS) POS(ON)
_=/bin/env
CLASSPATH=:/usr/jpp/Java/J1.1/lib/classes.zip
LOGNAME=DOMINO4
STEPLIB=none
LANG=C
LIBPATH=/lib:/usr/lib:.:/usr/lpp/Java/J1.1/lib/mvs/native_threads
_SLIB_PREFIX=SYS1
TERM=vt100
_CLIB_PREFIX=CBC
_BPX_SHAREAS=YES
HOME=/u/domino4
TZ=EST5EDT
MANPATH=/usr/man/%L
NLSPATH=/usr/lib/nls/msg/%L/%N
Notes_SHARED_DPOOLSIZE=8388608
Figure 17. Sample output of UNIX command env
52
Debugging UNIX Applications on OS/390
ps command
HAUFF @ SC63:/u/hauff>ps -a
PID TTY
TIME CMD
268435467 ttyp0002 0:43 /bin/find
117440529 ttyp0001 0:01 /bin/sh
100663327 ttyp0001 3h05 /usr/lpp/lotus/notes/latest/os390/cconsole
234881059 ttyp0002 0:43 /bin/sh
38 ttyp0003 0:02 /bin/sh
50331687 ttyp0003 2h14 /usr/lpp/lotus/notes/latest/os390/cconsole
671088680 ttyp0002 0:43 /bin/ps
Figure 18. Sample output of the UNIX command ps
2.3 Lotus Domino
This section describes main debugging issues for the Lotus Domino S/390 server
in its own environment.
2.3.1 Startup script and Language Environment (LE) run-time options
The incorrect use of LE run-time variable values can cause some serious
problems for Lotus Domino in an S/390 environment.
This issue is described in Information APAR II11603 as follows:
ERROR DESCRIPTION:
Various Symptoms, server abends with 0C1, (ABEND0C1) during
server setup/upgrade. Domino running with the SMTP MTA hangs
during shutdown, (Server in a loop) causing a hang type symptom.
Other symptoms include running out of storage below the line,
or server Panic during start up. It's possible that the customer
has had Domino running on another of this S390 systems but is
failing on "this" one.
Solution:
The Language Environment, LE, provides the runtime environment
for the Domino executables. There is parmlib member CEEDOPT
which is used to specify system wide defaults for the many
runtime parameters. Some customer sites have altered these
defaults as required by other work for their systems, COBOL
programs for instance. These parameters can be reset to IBM
defaults or these parameters may be set locally for Domino
executables. This is accomplished through the use of a UNIX
style environment string variable, called _CEE_RUNOPTS. In
order to set this variable properly it is necessary to edit
either the startup script or the ose90setup script, depending
on the failure that is occurring.
If the failure is during setup then
/usr/lpp/lotus/bin/server/tools/os390setup is the script that
should be modified. Remember to save a copy before editing!
Add the line: export _CEE_RUNOPTS="STORAGE(,,NONE,1K)"
If the failure is occurs during startup or shutdown the script
to edit is /usr/lpp/lotus/bin/server/tools/startup. Remember
to save a copy before editing! The server executables are hidden
behind links through this script which sets up (most of) the
Lotus Domino for S/390—troubleshooting/hints/tips
53
environment. Find the section for os390 and edit the LE
runoption specification to the following:
_CEE_RUNOPTS="RPTSTG(OFF) \
ALL31(ON) \
STACK(140000,140000,ANY,KEEP) \
LIBS(9K,9K,FREE) \
STORAGE(NONE,NONE,NONE,1K) \
HEAP(19843484,32K,ANYWHERE,KEEP,8K,4K) \
BELOWHEAP(490000,4096,FREE) \
ANYHEAP(20000,8K,ANYWHERE,FREE)
NONONIPTSTACK(4K,4K,BELOW,KEEP)
THREADHEAP(4K,4K,ANY,KEEP) \
TRAP(ON)"
export _CEE_RUNOPTS
For 'readability, backslash characters have been added after
a space and before the newline (carriage return). Other than
spaces (or commas) to separate the suboptions, this is the only
character allowed within the environment string _CEE_RUNOPTS.
The purpose of these scripts is
environment is available to the
This section of the script sets
runtime options for the process
to insure that the proper
process which is being invoked.
the LE (Language Environment)
correct.
For QMR's 4.63 and 4.56, this portion of the startup script has already
been updated. Previous releases have seen failures as a result
of unusual default system wide runtime parameters adversely
affecting the Domino server, the options specified here have
given good results. Please also refer to Lotus Domino TechNote
number #168426 (this technote contains nearly identical informations).
2.3.2 How to activate the Java environment
Especially with Lotus Domino 4.6.x there were error messages during the startup
of the server when the Java environment should be activated.
2.3.2.1 Lotus Domino Server 4.6.x
JVM: The JVM runtime library could not be found.
JVM: Java Virtual Machine failed to start.
Figure 19. Java error messages
Solution:
To avoid these error messages, two environment variable settings are required:
• PATH must include the location of libjava.a.
• CLASSPATH must include the location of the classes.zip file.
Note
libjava.a and classes.zip file are part of Java Development kit
54
Debugging UNIX Applications on OS/390
The following are examples of setting PATH and CLASSPATH:
Example
export Path=$PATH:/usr/lpp/java/J1.1.1/lib/mvs/native_threads
export CLASSPATH=$CLASSPATH:/usr/lpp/java/J1.1.1/lib/classes.zip
2.3.2.2 Lotus Domino Server 5.0
The following is mentioned in the Domino S/390 5.0 Release Notes (database
readmes.nsf on the installation CD) for enabling Java:
This topic describes setting up the agent manager configuration for running Java
agents. It includes:
• Java Virtual Machine initialization
• Setting the environmental variables
• Defining the platform property
Java Virtual Machine initialization
Domino for S/390 Release 5.0 supports the creation of agents using the Java
programming language. See Java Programmer's Guide for more information.
Install the Java Development Kit (JDK) 1.1.6 or higher separately.
If the agent manager does not initialize properly, the following message is
displayed:
JVM:
The Java runtime library could not be found
If you run a Java agent without successful JVM initialization, the following error
message is displayed:
JVM:
Java Virtual Machine failed to start
This message cannot be ignored and you must perform the Java initialization
again.
Setting the environmental variables
Set the three environmental variables as follows:
• PATH must include the location of libjava.a and the Java executable.
• LIBPATH must include the location of libjitc.so (usually in the same directory
as libjava.a).
• CLASSPATH must include the location of the classes.zip file.
Notes
• The libjava.a and classes.zip files are part of the JDK.
• The LIBPATH environment variable points to a version of the JDK and must
be kept in agreement with the PATH and CLASSPATH settings for the
agent manager setup. Each environment variable must point to the same
level of the JDK.
Lotus Domino for S/390—troubleshooting/hints/tips
55
Settings: The following are examples of setting the required environmental
variables. Replace the directory names shown here with the names you are
currently using.
Examples
export PATH=$PATH:/usr/lpp/java/J1.1.6/lib/mvs/native_threads:/usr/lpp/java/
J1.1.6/bin
export LIBPATH=$LIBPATH:/usr/lpp/java/J1.1.6/lib/mvs/native_threads
export CLASSPATH=$CLASSPATH:/usr/lpp/java/J1.1.6/lib/classes.zip
Defining the platform property
Use UNIX as the platform property to support the creation of agents using the
Java programming language on the S/390 platform.
Important
Be careful, Java will crash the server if it is not run from a specific directory.
2.3.3 Common parameters in the notes.ini
This section describes some of the main notes.ini parameters:
Table 17. Common notes.ini parameters
Variable
Short description of the variable
Example
Admin
The user name of the server administrator. Enter each part of the
name in canonical format, separated by a slash(/), where:
CN is the common name, OU is the organization unit,
O is the organization, C is the country code.
Admin=CN=Manfred Hauff/O=ITSO2
CompanyName
Your company name as entered during server setup.
CompanyName=IBM
Directory
The notes data directory.
Directory=/domino4/notesdata
Domain
The server’s domain name entered during setup.
Domain=ITSO2
DST
Specifies that a server observes daylight saving time.
DST=1 Observe daylight saving time
KeyFilename
Specifies the location of the server ID file.
KeyFilename=/u/domino4/domino4.id
KitType
Indication if this notes.ini belongs to a client or server.
KitType=1 (client) , KitType=2 (server)
56
Debugging UNIX Applications on OS/390
Variable
Short description of the variable
Example
Log
Specifies the contents of the log file and controls other logging
actions.
Syntax: log=logfilename,log_option,not_used,days,size
Log=log.nsf,1,0,7,40000 (default)
MailServer
Specifies the server where the user's mail file resides.
MailServer=CN=Domino4/O=ITSO2
Names
Specifies the names of the secondary Domino Directories the server
searches to verify recipient names. By default only the primary
Domino Directory NAMES.NSF is searched.
Names=names,westnames,eastnames
<TCPPortname>_
TCPIPAddress
Defines the IP address and port number for a Domino server.
<TCPportname> is the name of the TCP port specified on the Ports
statement in the notes.ini.
TCPIP_TcpIpAddress=0,9.12.2.35:1352
Ports
Specifies the port, which is enabled for the server.
Ports=TCPIP
ServerTasks
The servertasks being loaded during server startup.
Servertasks=Router, Replica,Update
ServerTasksAt1
This line schedules when this task will be run. Number represents
the time (1:00 A.M.).
ServertasksAt1=Catalog,Design
Timezone
Specifies the time zone for a server. Time zones begin at
Greenwich, England (0 = Greenwich Mean Time) and move
westward around the world.
Timezone=5
Translog_Status
Enable/disable transaction logging for all R5.0 databases on the
server. "0" is disabled, "1" is enabled.
Important: You must upgrade databases to Domino 5.0 format
before they can use transaction logging.
Translog_Status=1
Lotus Domino for S/390—troubleshooting/hints/tips
57
2.3.4 Logs
There are multiple logs that are written by Lotus Domino S/390, either
permanently or especially in case of errors. The most important logs are
described in the following sections.
2.3.4.1 NSD logs
The NSD output is saved in a log file with the following structure:
<nsd_option>_<OS name>_<hostname>_<timestamp>.log
Here are some examples for nsd.sh -kill and nsd.sh -info files:
kill_OS390_SC63_07_14@14_46.log
sysinfo_OS390_SC63_07_14@15_19.log
For more information, see 2.2.1, “nsd.sh” on page 45.
2.3.4.2 HTTP logs
These logs are written for the HTTP task.
Enabling HTTP logs
To get HTTP logs, this has to be enabled in the Server Configuration first as
shown in Figure 20.
Figure 20. Enable http logs
58
Debugging UNIX Applications on OS/390
DOMLOG
One of the Domino logs is the database domlog.nsf that contains the following
information:
Figure 21. domlog.nsf
Access log
The access log is usually written to the .../notesdata directory and readable using
the Browse command in the ISHell.
Format: accessdate.log
Example: access07151999.log
Contents (small excerpt):
BROWSE -- /domino4/notesdata/access07151999.log ---- Line 00000000 Col 001 080
Command ===>
Scroll ===> PAGE
9.12.2.163 9.12.2.35 CN=Wilhelm Michel/O=Itso2 Ý15/Jul/1999:16:20:35 +0500¨ "GETT
/catalog.nsf/9801faacd5f39e9b852566640055b0aa?OpenView HTTP/1.0" 200 10630
9.12.2.163 9.12.2.35 - Ý15/Jul/1999:16:20:55 +0500¨ "GET /?Open HTTP/1.0" 401
2601 263
Agent log
The agent log is usually written to the .../notesdata directory and readable using
the Browse command in the ISHell.
Format: agentdate.log
Example: agent07151999.log
Contents (small excerpt):
Lotus Domino for S/390—troubleshooting/hints/tips
59
BROWSE -- /domino4/notesdata/agent07151999.log ----- Line 00000000 Col 001 058
Command ===>
Scroll ===> PAGE
********************************* Top of Data **********************************
Ý15/Jul/1999:16:20:35 +0500¨ "Mozilla/4.6 Ýen¨ (WinNT; I)"
Ý15/Jul/1999:16:20:35 +0500¨ "Mozilla/4.6 Ýen¨ (WinNT; I)"
Ý15/Jul/1999:16:20:55 +0500¨ "Mozilla/4.6 Ýen¨ (WinNT; I)"
Cgi-error log
The cgi-error log is usually written to the .../notesdata directory and readable
using the Browse command in the ISHell.
Format: cgi-errordate.log
Example: cgi-error07151999.log
Error log
The error log is usually written to the .../notesdata directory and readable using
the Browse command in the ISHell.
Format: errordate.log
Example: error07151999.log
Contents (small excerpt):
BROWSE -- /domino4/notesdata/error07151999.log ----- Line 00000000 Col 001 076
Command ===>
Scroll ===> PAGE
********************************* Top of Data **********************************
Ý15/Jul/1999:16:20:24 +0500¨ Ý?¨ Ýhost: ¨ Can't init groups for user DOMINO4
******************************** Bottom of Data ********************************
Referer log
The error log is usually written to the .../notesdata directory and readable using
the Browse command in the IShell.
Format: refererdate.log
Example: referer07151999.log
Contents (small excerpt):
BROWSE -- /domino4/notesdata/referer07151999.log --- Line 00000000 Col 001 080
Command ===>
Scroll ===> PAGE
********************************* Top of Data **********************************
Ý15/Jul/1999:16:21:05 +0500¨ "http://9.12.2.35:8080/?Open"
Ý15/Jul/1999:16:21:05 +0500¨ "http://9.12.2.35:8080/catalog.nsf?OpenDatabase"
Ý15/Jul/1999:16:21:15 +0500¨ "http://9.12.2.35:8080/catalog.nsf/9801faacd5f39e9b
0¨ "http://9.12.2.35:8080/catalog.nsf/9801faacd5f39e9b852566640055b0aa?OpenView"
60
Debugging UNIX Applications on OS/390
2.3.4.3 Noteslog
Every Domino server has a log file (LOG.NSF) that reports all server activity and
provides detailed information about databases and users on the server. The log
file is created automatically when you start a server for the first time. You can do
the following:
• Record additional information in the log file.
• View the log.
• Search the log.
• Control the size of the log.
For more information about the log.nsf, see database Domino 5 Administration
Help (help5_admin.nsf).
2.3.4.4 Console log
Every line prompted on the console can be rerouted to a so-named debug file
using the Debug_Outfile parameter in the notes.ini. For information on how to
activate this file, see 2.5.8.1, “Implementing the DEBUG_OUTFILE parameter” on
page 115.
2.3.5 The Domino console
In this section possibilities with the Lotus Domino console are covered.
2.3.5.1 Invoking the console multiple times - cconsole
Using the Domino Character Console to access the server console
The Domino Character Console (the cconsole program) provides a way to access
the server console from the command line. This feature is supported only for
UNIX platforms.
You can invoke the cconsole program multiple times. You can also run the
cconsole program when there is already an operational Domino server console;
however, the cconsole input and output may also reflect commands launched
from other console processes.
Note
The cconsole program is installed into your Notes bin directory.
To start the cconsole program
1. To use cconsole, you must be listed as an administrator in the name and
address book server document.
2. Change the active directory to your data directory. For example, enter:
c d ~/notesdata
3. Enter the cconsole command. For example, enter:
/opt/lotus/bin/cconsole
4. Enter the path and file name of your Notes user ID.
5. Enter the password for your Notes user ID.
6. To exit cconsole, type:
done
Lotus Domino for S/390—troubleshooting/hints/tips
61
Remote cconsole
The cconsole program does not start if the Domino server is not running on the
same machine as the cconsole program. If the server fails while cconsole is
running, cconsole may not automatically shut down. In this case, enter the done
command to exit the cconsole program.
To run cconsole from a remote machine, first telnet to the machine running the
Domino server.
Note
There is a security risk when running the cconsole program from a remote
machine or from a remote X display. The cconsole program warns you of this
security risk before proceeding. Deploy a secure remote protocol such as
encrypted telnet. To address this security risk, if you do not deploy a secure
remote protocol, run the cconsole program only from the local Domino server
machine.
Additional cconsole commands
In addition to the current set of Domino server console commands, cconsole also
supports these commands:
Table 18. cconsole commands
Command
Result
done
Exits cconsole while the Domino server continues to run.
live on
Enables cconsole as a live console so that you see messages sent to
the server console from other sources.
live off
Disables the live console so that you see only the commands entered
and the responses to these commands.
Command line switches
There are several command line switches that streamline using cconsole. You
type the switches when you start cconsole:
Table 19. Command line switches
Switch
Result
-f
Lets you enter the path and file name for the Notes user ID when you start
cconsole so that you are not required to respond to the prompts.
-i
Lets you ignore warnings; warnings continue to appear on the console, but
you will not be required to respond to them.
-l
Lets you automatically start that console live when you start cconsole.
For example, if you do not want to wait for the prompt to enter the path and file
name for the Notes user ID, enter this command:
/opt/lotus/bin/cconsole -f domino4/notesdata/UserID.id
62
Debugging UNIX Applications on OS/390
Figure 22. cconsole
2.3.5.2 Domino server/console commands
List of all available server commands
This list briefly describes the server commands. These commands are typed in at
the server console:
BROADCAST "msg" ["user"] Broadcast a message to user(s) of this server
DBCACHE
Database Cache management commands
DISABLE
Disable use of database cache
FLUSH
Clear out database cache
SHOW
Show contents of database cache
DROP ["username"] [ALL] Drop one or more sessions
EXIT [password]
Exit server
HELP
Help (Displays this help information)
LOAD pgmname
Load program
PULL servername
Replicate one-way (pull)
PUSH servername
Replicate one-way (push)
QUIT [password]
Quit (exit server)
REPLICATE servername
Replicate two-way request
RESTART
Restart information:
SERVER [password]
Restart Server
PORT portname
Disable/Enable transactions on port
ROUTE servername
Route mail to server
SET
Set server option:
CONFIGURATION "variable=value" Configuration variable
SECURE [current-password] [new-password] Secure Console Password
STAT [Facility] [Statname] Reset statistics
SHOW
Show server information:
ALLPORTS
Show configuration for all ports
CLUSTER
Cluster information
CONFIGURATION variable
Configuration variable
Lotus Domino for S/390—troubleshooting/hints/tips
63
DATABASE filename
Show Database Information
DBS
Show Open Database Information
DIRECTORY
Directory Information
DISKSPACE filesystem
Available disk space
MEMORY
Memory information
PORT portname
Port specific information
SCHEDULE
Next Schedule [Server/Program/Location] [Appl]
SERVER
Server information
STATISTIC variable
Statistic variable
TASKS
Server tasks
USERS
Users with open sessions
START
Starts the specified service
PORT portname
Enable transactions on port
STOP
Stops the specified service
PORT portname
Disable transactions on port
TELL taskname command-string Send command-string to a task
Note
The following pages show the syntax, description, and examples of most of the
Domino server commands when typed on the server console.
Be aware that all these issues can also be handled using the Domino
Administration function on a Workstation Client.
How to handle all these commands is also described in detail in the Domino 5
Administration Help Guide (help5_admin.nsf), Chapter "Domino server
commands".
Broadcast
Syntax: Broadcast message usernames
Description: Sends a message to specified users or to all users of this server.
Use this command to warn users when a server is brought down for maintenance.
The message you enter appears in the user's status bar.
Example:
> broadcast "this is a test note" "Wilhelm Michel"
Dbcache
Syntax: dbcache show/flush/disable
Description:
64
• DISABLE
Disable use of database cache
• FLUSH
Clear out database cache
• SHOW
Show contents of database cache
Debugging UNIX Applications on OS/390
Example:
> dbcache show
Database
/domino4/notesdata/admin4.nsf
> dbcache flush
> dbcache show
Database
>
Command
Result
Command
Result
Drop
Syntax: Drop username
Description: Closes one or more server sessions. To visually confirm which
sessions are dropped, you must enter the Log_Sessions=1 setting in the server's
NOTES.INI file.
Examples:
> Drop "Sandy"
> Drop "Lee" "Fran"
> Drop All --
Closes the current session running under the user name
Sandy
Closes the sessions running under the user names Lee and
Fran
Closes all server sessions
Exit
Syntax: Exit
Description: Stops the server. This command is identical to Quit .
Help
Syntax: Help
Description: Displays a list of server commands with a brief description,
arguments (if any), and the proper syntax for each (see “List of all available
server commands” on page 63).
Load
Syntax: Load programname
Description: Loads and starts a specified server task or program on the server.
You can start a server add-in program or one that takes a command line for
additional data, such as a backup program. The program you run must be on the
server's search path. Use the Load command to run a program until it completes
or, if the program runs continually, until you stop the server. Where applicable,
you can include arguments that determine how the program runs.
Example:
> load Fixup
Loads and runs the Fixup server task
Lotus Domino for S/390—troubleshooting/hints/tips
65
Pull
Syntax: Pull servername [databasename]
Description: Forces a one-way replication from the specified server to your
server. You can also replicate a single database from the specified server to your
server by including the database name on the command line. The initiating server
receives data from the named server but does not request that the other server
pull data from it. This forces a server to replicate immediately with the initiating
server, overriding any replication scheduled in the Domino Directory. Enter the
server's full hierarchical name, if applicable.
You can pull changes immediately if an important database, such as the Domino
Directory, has changed or if a database on your server is corrupted or has been
deleted.
For replication to succeed, make sure that:
• The Domino Directory contains a Server document for each server in the
domain.
• The Domino Directory contains a Connection document to connect to a remote
server.
• Each server's ID file contains a certificate that the other server recognizes and
trusts.
• Database ACLs allow replication, and the source server has sufficient access
in the ACLs to replicate changes. If you are using server access lists, servers
must have prop er access in the Server document.
If the server is currently replicating, Domino queues the Pull server command
until the current task completes. To check the status of the Replicator before
using Pull, enter this command at the console:
Show Tasks
The server displays one of the following messages:
• If the server is not replicating, the word Idle appears next to the Replicator
task.
• If the server is replicating, a message such as Replicating CONTRACT.NSF
from MARKETING\CONTRACT.NSF appears.
Example:
> pull domino5\ITSO2
> pull domino5\ITSO2 NAMES.NSF
66
Debugging UNIX Applications on OS/390
Forces one-way replication with the
server domino5.
Forces one-way replication of the
NAMES.NSF file from server domino5.
Push
Syntax: push servername [databasename]
Description: Forces a one-way replication from your server to the specified
server. You can also replicate a single database from your server to the specified
server by including the database name on the command line. The initiating server
sends data to the named server but does not request data in return. This forces a
server to replicate immediately with the initiating server, overriding any replication
scheduled in the Domino Directory. Specify the server's full hierarchical name, if
applicable.
In effect, the Push server command is the functional opposite of the Pull server
command.
Example:
> push domino5\ITSO2
> push domino5\ITSO2 NAMES.NSF
Forces one-way replication with the
server domino5.
Forces one-way replication of the
NAMES.NSF file to server domino5.
Quit
Syntax: Quit
Description: Stops the server. This command is identical to the Exit server
command. However, the Quit server command differs from the Tell server
command, which you use to stop a particular server task without stopping the
server.
If you stop a server while it is replicating databases or routing mail, these tasks
resume at the next scheduled interval after you restart the server. Replication or
mail routing continues until the databases are fully replicated and until the
complete mail message is transferred or returned to the sender.
Before you use the Quit server command to stop the server, use the Broadcast
server command to warn users to finish their current tasks before you stop the
server.
Replicate
Syntax: Replicate servername [databasename]
Description: Forces replication between two servers (the server where you enter
this command and the server you specify). Use the server's full hierarchical
name. If the server name is more than one word, enclose the entire name in
quotes. To force replication of a particular database that the servers have in
common, specify the database name after the server name. The initiating server
(where you are currently working) first pulls changes from the other server, and
then gives the other server the opportunity to pull changes from it. You can use
this command to distribute changes quickly or to troubleshoot a replication or
communication problem.
Lotus Domino for S/390—troubleshooting/hints/tips
67
If the server is already replicating when you issue the command, Domino queues
the command until the current replication ends. To check the status of the
Replicator, enter this command at the console:
Show Tasks
The server displays one of the following messages:
• If the server is not replicating, the word Idle appears next to the Replicator
program.
• If the server is replicating, a status line, such as Replicating CONTRACT.NSF
from MARKETING\CONTRACT.NSF, appears.
To optimize resources Domino only replicates what is necessary. For example, if
the servers have recently replicated and no changes have been made to any
databases on either server since, the servers do not replicate when you enter a
Replicate command. Also, the replication is two-way only if databases on both
servers have changed since the last replication. If databases on only one of the
servers have changed, the replication is one-way.
To force replication in only one direction, use the Pull or Push server commands.
Examples:
> replicate domino5\ITSO2
Initiates replication between your server
and the domino5\ITSO2 server.
> replicate domino5\ITSO2 NAMES.NSF
Initiates replication of NAMES.NSF between
your server and the domino5\ITSO2
Restart
Syntax: Restart
Description: Stops the server and then restarts the server after a brief delay.
If you stop a server while it is replicating databases or routing mail, these tasks
resume at the next scheduled interval after you restart the server. Replication or
mail routing continues until the databases are fully replicated and until the
complete mail message is transferred or returned to the sender.
Before you use the Restart server command to stop the server, use the
Broadcast server command to warn users to finish their current tasks before you
stop the server.
Route
Syntax: Route servername
Description: Initiates mail routing with a specific server. The Route command
overrides any mail routing schedules that you create in the Connection
documents in the Domino Directory. Use the Route command for servers that are
configured for Pull, Pull Push, Push, or Push Wait routing in the Connection
document. Use the server's full hierarchical name, if applicable. If the server
name is more than one word, enclose the entire name in quotes.
Use the Route command to troubleshoot mail problems and to send mail to or
request mail from a server immediately.
68
Debugging UNIX Applications on OS/390
If no mail is queued for routing, Domino ignores the Route command. To check
which servers have mail queued, enter this command at the console:
Tell Router Show
Example:
> route domino5\ITSO2
Sends mail to the Marketing server in the
ITSO2 domain. The server console displays
messages indicating when routing begins.
Set configuration
Syntax: Set Configuration setting
Description: Adds or changes a setting in the NOTES.INI file.
Example:
> set config SERVERTASKS=Router,Replica,Update,Amgr,AdminP,CalConn,Sched,Event,S
tats,HTTP,DECS
Set secure
Syntax: Set Secure currentpassword
Description: Password-protects the console.
After you password-protect the console, you can not use the Load, Tell, Exit,
Quit, and Set Configuration server commands or other programs that are not run
automatically through Program documents in the Domino Directory or through the
NOTES.INI file until you enter the password again. Console security remains in
effect until you clear the password by entering a second Set Secure command
with the same password.
Even if the console is password-protected, keep the server physically secure to
prevent breaches of security at the operating system level.
Examples:
> set secure dominor5
> load http
This command is not permitted when console security is in effect
> set secure dominor5
> load http
If you do not want to type the password on the console (because it is shown
there), type it using Domino administration:
1. From the Domino Administrator, click the Server - Status tab.
2. If necessary, click Tools to display the tool bar, and click Server - Secure
console.
Lotus Domino for S/390—troubleshooting/hints/tips
69
Figure 23. Set secure
Set statistics
Syntax: Set Statistics statisticname
Description: Resets a statistic that is cumulative. Statisticname is a required
parameter that names the statistic to be reset. You cannot use wildcards (*) with
this argument.
Example:
> set stat Server.Trans.Total
Resets the Server.Trans.Total statist to 0
Show allports
Syntax: Show Allports
Description: Displays the configuration for all enabled and disabled ports on the
server.
Example:
> show allports
Enabled Ports:
TCPIP=TCP, 0, 15, 0
Disabled Ports:
SPX=SPX, 0, 15, 0
Serial1=XPC,1,15,0,
Serial2=XPC,2,15,0,
Show cluster
Syntax: Show Cluster
Description: Displays the local server's cluster name cache, which includes a list
of all cluster members and their status, based on information received during the
server's cluster probes.
70
Debugging UNIX Applications on OS/390
This example displays the cluster name cache of the Mars server, which is in the
Planets cluster, which is in the Solarsys domain:
> show cluster
Cluster Information
Cluster name: planets/solarsys, Server name: mars/solarsys
Server cluster probe timeout: 1 minute(s)
Server cluster probe count: 2604
Server cluster probe port: NetBEUI
Server availability threshold: 10
Server availability index: 98 (state: AVAILABLE)
Cluster members (2)...
server: mars/solarsys, availability index: 98
server: saturn/solarsys, availability index: BUSY
>
Show configuration
Syntax: Show Configuration setting
Description: Displays the current value for a NOTES.INI setting. Use the Show
Configuration and Set Configuration server commands together to ensure that
you correctly set the NOTES.INI settings.
Examples:
> sh config domain
DOMAIN=ITSO2
> sh config servertasks
SERVERTASKS=Router,Replica,Update,Amgr,AdminP,CalConn,Sched,Event,Stats,HTTP,DEC
S
>
Show directory
Syntax: Show Directory
Description: Lists all database files (for example, .NSF and .NTF) in the data
directory and specifies whether the data directory contains multiple replicas of a
database. This command works only for the data directory; you cannot specify
another directory.
Example:
> show directory
.
.
.
/notesdata/admin4.ntf
/notesdata/admin4.nsf
/notesdata/log.nsf
/notesdata/names.nsf
lica's)
>
V4
V5
V5
V5
N/A
N/A
N/A
N/A
05/06/99
07/01/99
07/01/99
07/01/99
05:00:51
12:02:07
06:29:07
02:00:21
AM
AM
PM
AM (3 Rep
Lotus Domino for S/390—troubleshooting/hints/tips
71
Show diskspace
Syntax: Show Diskspace location
Description: Displays the amount of space, in bytes, available on the HFS file
system. An HFS is selected by adding the HFS’s mount point using the location
parameter.
Example: The directories /u, /etc, and /var that are mentioned in the following
figure are all located on the same physical disk but are mounted using different
HFS data sets:
> show diskspace /u
Available disk space 12,591,104 bytes
> show diskspace /etc
Available disk space 37,535,744 bytes
> show diskspace /var
Available disk space 6,557,696 bytes
>
Show memory
Syntax: show memory
Description: Displays the memory statistics for the server.
Example:
> show memory
Memory Available: 512 Mbytes
>
Show port
Syntax: show port portname
Description: Displays traffic and error statistics
Example:
> show port tcpip
TCP/IP Port Driver
Transport Provider: TCP
Notes Session
175A0004
175A0007
175C0003
17600002
Local Address
9.12.2.35:1352
9.12.2.35:1352
9.12.2.35:1352
9.12.2.35:1352
Foreign Address
9.12.2.167:1049
9.12.2.163:1136
9.12.2.163:1150
*:*
Show schedule
Syntax: Show Schedule servername/taskname/location
Description: Shows the next time that a server task runs. Output includes the type
of task and the time it runs next. If you enter a location as an argument, the
workstation replication schedule for that location appears.
72
Debugging UNIX Applications on OS/390
Examples:
> show schedule
> show schedule fixup
Displays a list of all scheduled tasks
Shows when the Fixup task is scheduled
to run next
Show server
Syntax: Show Server
Description: Shows server status information including the server name, data
directory on the server, time elapsed since server startup, transaction statistics,
and the status of shared, pending, and dead mail.
Example:
> show server
Lotus Domino (r) Server (Release 5.0a (Intl) for UNIX) 07/01/99 04:13:37 PM
Server name:
Server directory:
Partition:
Elapsed time:
Transactions/minute:
Peak # of sessions:
Transactions:
Availability Index:
Message Tracking:
Shared mail:
Number of Mailboxes:
Pending mail: 0
Waiting Tasks:
Domino4/Itso2
/domino4/notesdata
.domino4.notesdata
06:05:34
Last minute: 4; Last hour: 1; Peak: 126
6 at 07/01/99 11:29:05 AM
588
100 (state: AVAILABLE)
Not Enabled
Not Enabled
1
Dead mail: 0
0
Show stat
Syntax: Show Stat statisticname
Description: Used without the optional statisticname argument, displays a list of
server statistics for disk space, memory, mail, replication, and network activity.
To display a single statistic, enter the name of the statistic as the optional
argument. To display only a subset of statistics, add a group of statistics as an
optional argument by using an asterisk (*) as a wildcard.
Example:
> show stat server.version.notes
Server.Version.Notes = Release 5.0a (Intl)
> show stat server.version.os
Server.Version.OS = OS/390
> show stat server.users.peak.time
Server.Users.Peak.Time = 07/01/1999 11:29:05 EDT
> show stat database.n*
Database.NAMELookupCache.Peak = 1,048,576
Database.NAMELookupCache.Used = 468,224
Database.NIFPool.Peak = 1,048,576
Database.NIFPool.Used = 249,248
Database.NSFPool.Peak = 1,048,576
Database.NSFPool.Used = 221,824
>
Lotus Domino for S/390—troubleshooting/hints/tips
73
Show tasks
Syntax: Show Tasks
Description: Displays the name of the server, the path of the Domino program
directory, and the status of the active server tasks.
Example:
Lotus Domino (r) Server (Release 5.0a (Intl) for UNIX) 07/01/99 04:37:07 PM
Server name:
Domino4/Itso2
Server directory:
/domino4/notesdata
Partition:
.domino4.notesdata
Elapsed time:
06:29:04
Transactions/minute:
Last minute: 2; Last hour: 2; Peak: 126
Peak # of sessions:
6 at 07/01/99 11:29:05 AM
Transactions:
678
Availability Index:
100 (state: AVAILABLE)
Message Tracking:
Not Enabled
Shared mail:
Not Enabled
Number of Mailboxes:
1
Pending mail: 0
Dead mail: 0
Waiting Tasks:
0
Transactional Logging: Not Enabled
Task
Description
Database Server
Perform console commands
Database Server
Listen for connect requests on TCPIP
Database Server
Load Monitor is idle
Database Server
Perform Database Cache maintenance
Database Server
Idle task
Database Server
Server for Manfred Hauff/Itso2 on TCPIP
Database Server
Server for Wilhelm Michel/Itso2 on TCPIP
Database Server
Server for Wilhelm Michel/Itso2 on TCPIP
HTTP Web Server
Listening on port(s) 80
Stats
Idle
Event Monitor
Idle
Schedule Manager
Idle
Calendar Connector Idle
Admin Process
Idle
Agent Manager
Executive '1': Idle
Agent Manager
Idle
Indexer
Idle
Replicator
Idle
Router
Idle
Show users
Syntax: Show Users
Description: Displays a list of all users who have established sessions with the
server, whether the users are actively working in databases or not, the names of
databases that each user has open, and the elapsed time, in minutes, since the
databases were last used.
Tell
Syntax: Tell serverprogram
Description: Issues a command to a server program or task. The command is
especially useful for stopping a server task without stopping the server. This is
the most powerful server command with a lot of possibilities.
74
Debugging UNIX Applications on OS/390
Example:
> tell http quit
Stops only the http task. All other
tasks on the server continue to run.
Specialized Tell commands
Some Tell commands are common to all server tasks, for example, Tell task Quit.
Other Tell commands are unique to a particular task. These tasks have unique
Tell commands and are available for the following services:
• Administration Process
• Agent Manager
• Cluster Replicator
• NNTP
• LDAP
• Router
• Schedule Manager
• Statistic Collector
• Web Navigator
• Web Server
Note
For detailed information about all the different tell commands refer to the
Domino5 Administration Help Guide (help5_admin.nsf) Chapter "Domino
server commands".
2.3.6 Programs (compact, update, updall, and fixup)
In this section we describe the four major utilities, compact, update, updall, and
fixup, how to use each and some of the available parameters. It is a summary of
the Domino 5 Administration Help.
2.3.6.1 Compact
When documents and attachments are deleted from a database, Domino tries to
reuse the unused space, rather than immediately reducing the file size.
Sometimes Domino will not be able to reuse the space or, due to fragmentation,
cannot reuse the space effectively, until you compact the database. Note that
Release 5 is much better at reusing space in Release 5 format databases, so
compacting is needed less often than in past releases.
There are three styles of compacting used in Release 5:
• In-place compacting with space recovery only
• In-place compacting with space recovery and reduction in file size
• Copy-style compacting
Lotus Domino for S/390—troubleshooting/hints/tips
75
In-place compacting with space recovery only
This style of compacting recovers unused space in a database but does not
reduce the size of the database on disk. Databases retain the same database
instance IDs (DBIIDs), so the relationship between the compacted databases and
the transaction log remains intact. Users and servers can continue to access and
edit databases during compacting. This style of compacting is useful for Release
5 databases that you expect to either stay the same size or grow in size.
When you run Compact without specifying options, Domino uses this style of
compacting on all Release 5 format databases enabled for transaction logging.
Domino also uses this style of compacting when you use the -B option
(case-sensitive) when compacting any Release 5 format database.
Use this compacting method the most frequently, it is the fastest method and
causes the least system impact.
In-place compacting with space recovery and reduction in file size
This style of compacting reduces the file size of Release 5 databases as well as
recovering unused space in databases. This style of compacting is somewhat
slower than in-place compacting with space recovery only. This style of
compacting assigns new DBIIDs to databases, so if you use it on logged
databases and you use a Release 5-certified backup utility, do full backups of the
databases shortly after compacting is complete. This style of compacting allows
users and servers to continue to access and edit databases during compacting.
When you run Compact without specifying options, Domino uses this style of
compacting on Release 5 databases that are not enabled for transaction logging.
Domino also uses this style of compacting when you use the -B option
(case-sensitive) when compacting Release 5 format databases. To optimize disk
space, it is recommended that you run Compact using the -B option on all
Release 5 format databases once a week or once a month.
Copy-style compacting
Copy-style compacting creates copies of databases and then deletes the original
databases after compacting completes, so extra disk space is required to make
the database copies. This style of compacting essentially creates a new database
with a new database ID. If you use copy-style compacting on Release 5 logged
databases (using the -C option), compacting assigns new DBIIDs, so if you use a
Release 5-certified backup utility, you should do full backups of databases shortly
after compacting completes. When you use copy-style compacting, users and
servers cannot edit databases during compacting and they can only read
databases if the -L option is used.
Domino uses copy-style compacting by default when you use an option with
Compact to enable a database property that requires a structural change to a
database or when you run Compact on a database that has a structural change
pending that was initiated from the Database Properties box. Enabling or
disabling the database properties, Document table bitmap optimization, and Do
not support specialized response hierarchy, requires structural database
changes. Compacting a Release 4 format database also uses copy-style
76
Debugging UNIX Applications on OS/390
compacting. Note that if you compact a Release 4 format database without using
the -R option, compacting converts the database to the Release 5 format.
Table 20. Comparison of the three styles of compaction
Characteristics
In place, space
recovery
In place, space
recovery with file
size reduction
Copy-style
Databases that use
it when compact
runs without
options
R5 logged
databases with no
pending structural
changes
R5 unlogged
databases with no
pending structural
changes
R4 databases; R5
databases with
pending structural
changes
Databases you can
use it on
R5
R5
All (Need -C for R5
databases)
Relative speed
Fastest
Medium
Slowest
Users can read
databases during
compacting
Yes
Yes
No (unless -L option
used)
Users can edit
databases during
compacting
Yes
Yes
No
Reduction in file
size
No
Yes
Yes
Extra disk space
required
No
No
Yes
There are four ways to compact databases:
1. Run Compact using the Compact tool in the Files tab of the Domino
Administrator. Use this method to compact a few databases; you can easily
select the databases to compact, but you cannot use the Domino
Administrator until compacting finishes.
2. Run Compact using the Task-Start tool in the Domino Administrator. Use this
method to easily compact all databases on a server; you can continue to use
the Domino Administrator during compacting and you do not have to
remember specific command-line options.
3. Run Compact using the Domino console. Use this method if you are
comfortable using command-line options or to compact databases directly at
the server when there is not a Domino Administrator client running on the
server.
•Load compact databasepath options
4. Run Compact using a Program document. Use this method to schedule
compact to run at particular times.
• From the Domino Administrator, click the Configuration tab.
• Next to "Use Directory on," select the server with the replica of the Domino
Directory that you want to modify.
• Expand Server - Programs and then click Add Program.
• Complete the Basics tab.
• Complete the Schedule tab.
Lotus Domino for S/390—troubleshooting/hints/tips
77
• Click Save and Close.
The following table describes the options you can use with the Compact server
task. The first column lists the options as they appear when you run Compact
using the Task - Start tool or the Files tab in the Domino Administrator. The
second column lists the equivalent command-line options that you use when you
run Compact using a console command or using a Program document:
Table 21. Short summary of options for the Compact utility
78
Option
Command-line
equivalent
Description
Compact only this database
or folder
database path
Specify any
additional options
after the database
path.
To compact a database in the Domino
data folder, enter the file name, for
example SALES.NSF. To compact
databases in a folder within the data
folder, specify the database path
relative to the data folder.
Keep or revert database
back to R4 format
-R
Compacts databases without
converting to the current release file
format of the server that stores the
databases or reverts databases in the
current release file format to the
previous release file format.
Compact database only if
unused space is greater
than x percent
-S percent
Compacts all databases with a
specified percent of unused space.
Discard any built view
indexes
-D
Use this option to compact databases
just before you store them on tape, for
example. Does copy-style
compacting.
Compaction style: In-place
(recommended)
-b
Uses in-place compaction and
recovers unused space without
reducing the file size, unless there is a
pending structural change to a
database, in which case copy-style
compaction occurs. This is the
recommended method of compacting.
Compaction style: In-place
with file size reduction
-B
Uses in-place compaction, recovers
unused space and reduces file size,
unless there is a pending structural
change in which case copy-style
compacting occurs.
Compaction style:
Copy-style
-C
Uses copy-style compaction. Use this
option, for example, to solve database
corruption problems with Release 5
format databases.
Compaction style:
Copy-style: Allow access
while compacting
-L
Enables users to continue to access
databases during compacting. If a
user edits a database during
compacting, compacting is canceled.
This is useful only when copy-style
compacting is done.
Debugging UNIX Applications on OS/390
As mentioned before, there are a couple of more options available. For a
complete list of them please refer to the Domino 5 Administration Help.
2.3.6.2 Update and Updall
The Update and Updall tasks keep view indexes and full-text indexes up-to-date.
Update is loaded at server startup by default and runs continually, checking its
work queue for views and folders that require updating. When a view or folder
change is recorded in the queue, Update waits approximately 15 minutes before
updating all view indexes in the database so that the update can include any
other database changes made during the 15-minute period. After updating view
indexes in a database, it then updates all databases that have full-text search
indexes set for immediate or hourly updates.
When Update encounters a corrupted view index or full-text index, it rebuilds the
view index or full-text index in an attempt to correct the problem. This means it
deletes the view index or full-text index and rebuilds it.
Updall is similar to Update, but it does not run continually or work from a queue;
instead you run Updall as needed. You can specify options when you run Updall,
but without them Updall updates any view indexes or full-text search indexes on
the server that need updating. To save disk space, Updall also purges deletion
stubs from databases and discards view indexes for views that have been unused
for 45 days, unless the database designer has specified different criteria for
discarding view indexes. Use the NOTES.INI setting
Default_Index_Lifetime_Days to change when Updall discards unused view
indexes.
Like Update, Updall rebuilds all corrupted view indexes and full-text search
indexes that it encounters.
By default Updall is included in the NOTES.INI setting ServerTasksAt2, so it runs
daily at 2:00 A.M. Running Updall daily helps save disk space by purging deletion
stubs and discarding unused view indexes. It also ensures that all full-text search
indexes that are set for daily updates are updated.
The following table compares the characteristics of Update and Updall. For
Updall, the table describes default characteristics, some of which you can modify
using options:
Table 22. Compare Update and Updall characteristics
Characteristic
Update
Updall
When it runs.
Continually after server startup.
2:00 A.M. and when you
run it.
Runs on all databases?
No. Runs only on databases
that have changed.
Yes
Refreshes views indexes?
Yes
Yes
Updates full-text indexes?
Yes. Updates full-text indexes
set for immediate and hourly
updates.
Yes. Updates all full-text
indexes.
Lotus Domino for S/390—troubleshooting/hints/tips
79
Characteristic
Update
Updall
Detects and attempts to
rebuild corrupted view
indexes?
Yes
Yes
Detects and attempts to
rebuild corrupted full-text
indexes?
Yes
Yes
Can customize with
options?
No
Yes
You can use three different methods to run Updall:
1. Task - Start tool in the Domino Administrator. Use this method if you do not
want to remember command-line options.
2. Load updall console command. Use this method if you are comfortable using
command-line options or if you want to run Updall directly at the server
console when there is no Domino Administrator running on the server
machine.
•Load updall databasepath options
3. Program document that runs Updall. Use this method to schedule Updall to
run at particular times.
• From the Domino Administrator, click the Configuration tab.
• Next to "Use Directory on," select the server with the replica of the Domino
Directory that you want to modify.
• Expand Server - Programs and then click Add Program.
• Complete the Basics tab.
• Complete the Schedule tab.
• Click Save and Close.
Table 23. Updall options
Option in Task - Start tool
Command-line option
Description
- Index all databases
- Index only this database
or folder
database path
"Only this database" updates
only the specified database.
"Index all databases" (or no
database path) updates all
databases on the server.
Update this view only
database -T viewtitle
Updates a specific view in a
database. Use, for example,
with -R to solve corruption
problems.
Update: All built views
-V
Updates built views and does
not update full-text indexes.
Update: Full text indexes
-F
Updates full-text indexes and
does not update views.
Please see the Domino 5 Administration Help for additional information.
80
Debugging UNIX Applications on OS/390
2.3.6.3 Fixup
When you restart a server, the server quickly searches for any Release 4 format
databases and any unlogged Release 5 databases that were modified but
improperly closed because of a server failure, power failure, hardware failure,
and so on. A few minutes after server startup is complete, the Fixup task then
runs on these databases to attempt to fix any inconsistencies that resulted from
partially written operations caused by a failure. When users attempt to access
one of these databases and Fixup has not yet run on the database, the users see
the message, "This database cannot be opened because a consistency check of
it is in progress." A similar Fixup process occurs when you restart a Notes client.
Multiple Fixup tasks run simultaneously at server startup to reduce the time
required to fix databases. The number of Fixup tasks that Domino runs by default
at startup is equal to two times the number of processors available on the server.
Although this default behavior should be adequate in most circumstances, you
can edit the NOTES.INI file to include the Fixup_Tasks setting. The actual
number of tasks run is the smaller of the configured number of tasks that can run
and the number of databases that require fixing. For example, if you set
Fixup_Tasks to 4, but only one database requires fixing, then only one Fixup task
runs.
Note: Keep in mind that after you convert Release 4 databases to Release 5 and
set up transaction logging, Fixup is not needed or used to bring databases back
to a consistent state.
Four ways to run Fixup manually
1. Run Fixup using Fixup tool in the Files tab. Use this method to run Fixup on
one or a few databases; you can easily select the databases and you do not
have to remember command-line options, but you cannot use the Domino
Administrator until Fixup finishes.
2. From the Domino Administrator, select the server that stores the databases on
which you want to run Fixup. If the Domino Administrator does not run on a
server, you can select local to run Fixup on databases stored on the client.
• Click the Files tab.
• Select the databases on which to run Fixup.
• On the Tools panel at the right, select Database - Fixup.
• (Optional) Select options to control how Fixup runs.
• Click OK.
3. Run Fixup using the Task - Start tool. Use this method to run Fixup on all
databases; you can continue to use the Domino Administrator while Fixup
runs and you do not have to remember command-line options.
4. Run Fixup using a console command. Use this method if you are comfortable
using command-line options or to run Fixup directly at the server console
when there is not a Domino Administrator client available.
•Load fixup options
5. Run Fixup using a program document. Use this method to schedule Fixup to
run at particular times.
• From the Domino Administrator, click the Configuration tab.
Lotus Domino for S/390—troubleshooting/hints/tips
81
• Next to "Use Directory on" select the server with the replica of the Domino
Directory that you want to modify.
• Expand Server - Programs.
• Select Server - Programs and then click Add Program.
• On the Basics tab, complete the fields.
• On the Schedule tab, complete the fields.
• Click Save and Close.
Table 24. Fixup options
Fixup options in Fixup
tool and Task - Start tool
Command-line
equivalent
Description
- Fixup all databases
- Fixup only this database
or folder
database path
"Fixup only this database or folder" runs
Fixup only on a specified database or all
databases in a specified folder. "Fixup
all databases" or no command-line
database path runs Fixup on all
databases on the server.
Report all processed
databases to log file
-L
Reports to the log file every database
Fixup opens and checks for corruption.
Without this argument, Fixup logs only
actual problems encountered.
Scan only since last fixup
-i
Running Fixup on a specific database
causes Fixup to check only documents
modified since Fixup last ran. Without
this option, Fixup checks all documents.
Scan all documents
-F
Runnning Fixup on all databases causes
Fixup to check all documents in the
databases. Without this option, Fixup
checks only documents modified since it
last ran.
Perform quick fixup
-Q
Causes Fixup to check documents more
quickly but less thoroughly. Without this
option, Fixup checks documents
thoroughly.
Exclude views (faster)
-V
Prevents Fixup from running on views.
This option reduces the time it takes
Fixup to run. Use if view corruption is not
a problem.
Fixup transaction-logged
databases
-J
Causes Fixup to run on Release 5
databases that are enabled for
transaction logging. Without this option,
Fixup generally does not run on logged
databases.
If you are using a backup utility certified
for Domino Release 5, it is important that
you schedule a full back of the database
as soon as possible after Fixup finishes.
See Domino 5 Administration Help for more details.
82
Debugging UNIX Applications on OS/390
2.4 Hints and tips
This section contains common hints and tips that cover possible pitfalls and
important information about Lotus Domino for S/390.
2.4.1 How to edit the notes.ini file
The notes.ini file is an important configuration file that contains initialization
settings for the Lotus Domino server. For example, it tells the server which server
tasks to run during startup. The notes.ini file is located in the Notes data
directory, which is by default /notesdata on S/390. On OS/390 this file is stored in
ASCII format and not in EBCDIC.
The Domino server uses the first notes.ini file it finds in the path. If you have
multiple copies of the notes.ini file on your system, the server may start with
incorrect settings, or it may not even start. It is therefore recommended to have
only one notes.ini file accessible to the server.
The following two figures show a sample notes.ini file:
[Notes]
Directory=/domino4/notesdata
KitType=2
SetupDB=setupweb.nsf
UserName=
CompanyName=
SERVER_POOL_TASKS=100
SERVER_CONSOLE_CODEPAGE=1047
SERVER_ENABLE_THREADPOOL=1
NSF_BUFFER_POOL_SIZE_MB=5
PhoneLog=2
Log=log.nsf, 1, 0, 7, 40000
Passthru_LogLevel=0
Figure 24. Sample notes.ini - Part 1
Lotus Domino for S/390—troubleshooting/hints/tips
83
NSF_BUFFER_POOL_SIZE_MB=50000000001010100000A0000000000000100A0050A0000006
400A0050A00000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000940400000000
NAMEDSTYLE1=030042756C6C65740000000000000000000000000000000000000000000000
00000000000000000001010100000A000000000000000008070A000000640008070A000000
0000000000000000
NAMEDSTYLE1_FACE=Default Sans Serif
NAMEDSTYLE2=0300486561646C696E65000000000000000000000000000000000000000000
000000
NAMEDSTYLE2=0300486561646C696E65000000000000000000000000000000000000000000
00000000000000010101010B0C0000000000000100A0050A0000006400A0050A0000000000
00000000000000000000000000000000000000000000000000000000000000000000000000
0000009404000000000000
NAMEDSTYLE2_FACE=Default Sans Serif
$$$NotesNIC=CN=Home/OU=Notes/O=NET, welcome.nsf, Notes NIC Welcome, Notes
Network Information Center on the Internet
DefaultMailTemplate=mail50.ntf
Preferences=32
ServerTasks=Router,Replica,Update,Amgr,AdminP,CalConn,Sched,Event,Stats,HT
TP,DECS
ServerTasksAt1=Catalog,Design
ServerTasksAt2=UpdAll,Object Collect mailobj.nsf
ServerTasksAt3=Object Info -Full
ServerTasksAt5=Statlog
TCPIP=TCP, 0, 15, 0
TCPIP_TcpIpAddress=0,9.12.2.35:1352
SPX=SPX, 0, 15, 0
Serial1=XPC,1,15,0,
Serial2=XPC,2,15,0,
$$HasLANPort=1
Ports=TCPIP
DisabledPorts=SPX,Serial1,Serial2
LOG_REPLICATION=0
LOG_SESSIONS=0
ExistingServerName=CN=tot105/O=itso2
SETUP_ERRORPATH=File already exists: /u/domino4/domino4.id
KeyFilename=/u/domino4/domino4.id
SETUP_PERCENTDONE=100
SETUP_STATUS=Updating network settings
ServerKeyFileName=/u/domino4/domino4.id
Domain=ITSO2
Admin=CN=Manfred Hauff/O=ITSO2
EXTMGR_ADDINS=decsext
TemplateSetup=54
Setup=58
ServerSetup=50
Timezone=5
DST=1
CleanSetup=1
TRANSLOG_AutoFixup=1
TRANSLOG_UseAll=0
TRANSLOG_Style=0
TRANSLOG_Performance=2
Figure 25. Sample notes.ini - Part 2
84
Debugging UNIX Applications on OS/390
There are several different ways to edit this file:
• Edit the file from an OMVS session using the OEDITASCII editor.
• Edit the file from an rlogin or telnet session using the viascii editor.
• Transfer the file to a workstation, edit there, and then transfer it back.
Note
Make sure you have a valid copy of notes.ini before editing this file.
2.4.1.1 Edit notes.ini from OMVS using OEDITASCII
1. Log on to TSO with a user ID having an OMVS segment.
2. Issue the TSO OMVS command to enter the OS/390 UNIX shell.
3. In the OS/390 UNIX shell, change to the /notesdata directory, using the UNIX
command cd /notesdata.
4. Copy the notes.ini file to a backup, for example, cp notes.ini notes.old.
5. Edit the notes.ini file: /usr/lpp/lotus/bin/tools/oeditascii notes.ini.
Now you can use all availabe ISPF edit line commands to move, copy, delete, or
repeat lines.
2.4.1.2 Edit from a telnet or rlogin session using viascii
1. Log on to the S/390 server using telnet or rlogin from a workstation. You
will enter the OS/390 UNIX shell.
2. Change to the notesdata directory, for example, cd /notesdata.
3. Copy the notes.ini file to a backup, for example, cp notes.ini notes.old.
4. Edit the notes.ini file: /usr/lpp/lotus/bin/tools/viascii notes.ini.
You should be aware of the vi editor commands when using this method.
2.4.1.3 Transfer notes.ini file to a workstation and edit it there
You can transfer the notes.ini file to a workstation using FTP or any other
program to download files from the host to your workstation.
1. Download the file to the workstation in binary mode.
2. Edit the file using a workstation editor.
3. Upload the file to the host in binary mode.
2.4.2 Cleaning up server resources
It may happen that after a server shutdown using the quit or exit command, a
server crash, or some other unknown circumstances there are still some
resources allocated. Therefore, we recommend to check the availability of these
resources and remove them before the server is started again. To get a list of the
IPC resources (interprocess communication) use the ipcs -bo UNIX command as
shown in Figure 26.
Lotus Domino for S/390—troubleshooting/hints/tips
85
DOMINO4 @ SC63:/u/domino4>ipcs -bo
Message Queues:
Shared Memory:
T
ID
KEY
MODE
m
40004 0xf83a8000 --rw-rw---m
40005 0xf83a8001 --rw-rw---m
40006 0x010d0144 --rw-rw-rwm
40007 0xf83a8002 --rw-rw---Semaphores:
T
ID
KEY
MODE
s
85540 0xf83a8000 --ra-ra---s
20005 0xf83a8270 --ra-ra---s
20006 0x010d0145 --ra-ra-ras
20007 0x020d0145 --ra-ra-ras
20008 0x030d0145 --ra-ra-ras
20009 0xf83a8001 --ra-ra---s
20010 0xf83a8002 --ra-ra---s
20011 0xf83a8003 --ra-ra---s
20012 0xf83a8004 --ra-ra---s
20013 0xf83a8005 --ra-ra---s
20014 0xf83a8006 --ra-ra---DOMINO4 @ SC63:/u/domino4>
OWNER
DOMINO4
DOMINO4
DOMINO4
DOMINO4
GROUP
OEDOMINO
OEDOMINO
OEDOMINO
OEDOMINO
NATTCH SEGSZPG
14
1682
14
2048
14
7
14
2048
OWNER
DOMINO4
DOMINO4
DOMINO4
DOMINO4
DOMINO4
DOMINO4
DOMINO4
DOMINO4
DOMINO4
DOMINO4
DOMINO4
GROUP NSEMS
OEDOMINO
16
OEDOMINO
16
OEDOMINO
3
OEDOMINO
3
OEDOMINO
3
OEDOMINO
16
OEDOMINO
16
OEDOMINO
16
OEDOMINO
16
OEDOMINO
16
OEDOMINO
16
Figure 26. Display of semaphores and shared memory resources
If it happens that these Lotus Domino related resources are still available, you
have to clean them up manually using the UNIX command ipcrm:
• ipcrm -m <id> (for shared memory)
• ipcrm -s <id> (for semaphores)
Note
Check if the resources are Lotus Domino related and do not belong to any
other application like the Websphere Application Server.
In some rare cases we have not been successful in getting rid of these resources
as previously described. You then have to use one of the following methods:
• From the OS/390 UNIX shell using the command kill <process ID>
• From the MVS console or SDSF using:
• STOP command (p)
• CANCEL command (c)
• FORCE command (consider the FORCE command as a last resort when
the CANCEL command still fails)
• Modify command: F BPXOINIT,TERM=<process IDd>
• Modify command: F BPXOINIT,FORCE=<process IDd>
2.4.3 The HTTP task
The following sections give hints and tips concerning the HTTP task.
86
Debugging UNIX Applications on OS/390
2.4.3.1 Starting/stopping the HTTP task
The HTTP task is started in two ways:
1. Using the SERVERTASKS parameter in the notes.ini:
.
.
ServerTasks=Router,Replica,Update,HTTP,AdminP,CalConn,Sched,Event
.
.
2. Using the server console command load:
> load http
07/05/99 09:00:49 PM HTTP Web Server started
The HTTP task is stopped with the tell command:
> tell http quit
> 07/05/99 08:59:02 PM HTTP Web Server shutdown
>
2.4.3.2 Activating HTTP in the server’s administration
In the administration function of the Lotus Domino S/390 server, select Server -->
Current Server Document --> Internet Protocols where you can allow HTTP
clients to browse databases:
Figure 27. Activating HTTP - Internet protocols
Lotus Domino for S/390—troubleshooting/hints/tips
87
Configure the server by clicking Server --> Current Server Document --> Ports
to define a port number to http; in Figure 28 this is port 8080. Also Anonymous or
Name and password settings can be defined here:
Figure 28. Activating HTTP - server configuration
The user name and password are set in the Person Document in the server’s
Names and Address Book (NAB). This can be done only by the Administrator:
Figure 29. Activating HTTP - Internet password
88
Debugging UNIX Applications on OS/390
Now the server can be accessed using an Internet browser after verification of
the user name and password as shown in the following figures:
Figure 30. Activating HTTP - password prompt
Figure 31. Activating HTTP - accessing Domino databases using an Internet browser
2.4.3.3 Problems with the HTTP task
If there are problems with the HTTP task, use the TSO NETSTAT command for
checking what ports are defined and listening, as shown in the following example:
Lotus Domino for S/390—troubleshooting/hints/tips
89
------BPXOINIT
DOMINO1
DOMINO1
DOMINO2
DOMINO4
DOMINO4
DOMINO6
FTPD61
INETD7
INETD7
INETD7
INETD7
INETD7
TCPIP6
TCPIP6
TCPIP6
---0000E
00140
00141
0013E
001C0
00123
00216
00005
00121
0000B
0000A
0000D
0000C
00006
00007
00002
-----------0.0.0.0..10007
0.0.0.0..389
0.0.0.0..143
0.0.0.0..25
9.39.3.213..1352
0.0.0.0..1352
0.0.0.0..8080
0.0.0.0..21
9.39.3.213..623
0.0.0.0..514
0.0.0.0..623
0.0.0.0..512
0.0.0.0..513
127.0.0.1..1026
127.0.0.1..1025
0.0.0.0..1025
-------------0.0.0.0..0
0.0.0.0..0
0.0.0.0..0
0.0.0.0..0
9.12.2.163..1472
0.0.0.0..0
0.0.0.0..0
0.0.0.0..0
9.12.2.163..1066
0.0.0.0..0
0.0.0.0..0
0.0.0.0..0
0.0.0.0..0
127.0.0.1..1025
127.0.0.1..1026
0.0.0.0..0
----Listen
Listen
Listen
Listen
Establsh
Listen
Listen
Listen
Establsh
Listen
Listen
Listen
Listen
Establsh
Establsh
Listen
Figure 32. TSO NETSTAT command
With the command TSO NETSTAT ALL, more detailed information is shown.
There are also ways that you can check for the availability of a task. See
“Available server tasks” on page 95 for more information.
Note
Be aware of conflicting port numbers, means- check if any other task is just
mapped on the port you want to use for the http task, for example in the
/etc/services or TCPIP PROFILE.
2.4.4 The POP3 task
POP3 stands for Post Office Protocol (Version) 3. POP3 is an Internet protocol
that allows any client running POP3 (such as Netscape Navigator or Microsoft
Outlook Express) to retrieve mail from a Notes server that has the POP3 server
task loaded. You can set up a Domino server to run the POP3 service. Then,
POP3 users can periodically connect to the Domino server to retrieve their mail.
To allow POP3 users to send mail, the SMTP protocol must be used. SMTP must
be installed, configured, and the SMTP task must be loaded on the server.
The following sections give hints and tips concerning the POP3 task.
2.4.4.1 Starting/stopping the POP3 task
The POP3 task is started in two ways:
1. Using the SERVERTASKS parameter in the notes.ini:
.
.
ServerTasks=Router,Replica,Update,POP3,AdminP,CalConn,Sched,SMTP
.
.
90
Debugging UNIX Applications on OS/390
2. Using the server console command load:
load pop3
> 07/05/99 10:55:25 PM POP3 Server: Started
The POP3 task is stopped with the tell command:
> tell pop3 quit
07/05/99 10:53:55 PM POP3 Server: Waiting for all tasks to complete
07/05/99 10:53:55 PM POP3 Server: All tasks have completed
07/05/99 10:53:55 PM POP3 Server: Shutdown
>
2.4.4.2 Activating POP3 in the server’s administration function
In the administration function of the Lotus Domino S/390 server, select Server -->
Current Server Document --> Ports --> Internet Ports to enable TCP/IP port
110 (POP). If password checking should be done, set Name and password
checking to YES as shown in Figure 33:
Figure 33. Activating POP3 - server configuration
Defining a POP3 user
In the Domino administration select People & Groups --> Add person and fill
out the forms. An example is shown in Figure 34 and Figure 35:
Lotus Domino for S/390—troubleshooting/hints/tips
91
Figure 34. Defining a POP3 user - basic
Figure 35. Defining a POP3 user - mail
92
Debugging UNIX Applications on OS/390
Create a mail database for the POP3 user
Choose File --> Database --> New. Figure 36 shows an example:
Figure 36. Creating mail database for a POP3 user
ACL setting
Now the ACL for the POP3 user’s mail database has to be set up by opening the
mail database and then selecting File --> Database --> Access Control. Edit the
ACL to give the user Manager with Delete documents access. An example is
shown in Figure 37:
Figure 37. ACL of the POP3 user’s mail database
Lotus Domino for S/390—troubleshooting/hints/tips
93
Configure the browser to receive mail
The following example is based on Netscape Communicator 4.6.
Select Edit --> Preferences --> Mail Server and make the necessary entries for
the POP3 server as shown in Figure 38:
Figure 38. Netscape mail server properties
Now you can get mail using the mail server into your browser by choosing
Communicator --> Messenger. The browser’s mail tool comes up where the
messages can be read as shown in Figure 39:
Figure 39. Netscape Communicator 4.6 - Messenger view
94
Debugging UNIX Applications on OS/390
2.4.4.3 Problems with the POP3 task?
This section discusses some of the possible pitfalls of the POP3 task.
TSO NETSTAT command
If there are problems with the POP3 task, it is a good choice to use the TSO
NETSTAT command for checking what ports are defined and listening as shown
in Figure 32 on page 90.
Available server tasks
By using the show tasks command the availability of a task can be determined.
This can also be done in the Domino R5 administration, therefore select Server
--> Status:
Figure 40. Available server tasks
POP3Domain error
If the POP3 task is started using the command load pop3, the following error
message could come up:
> load pop3
Initialization Failure: POP3Domain Environment Variable is Not Defined
In this case you should add this environment variable to the notes.ini as follows:
.
.
POP3Domain=ITSO2
.
.
Lotus Domino for S/390—troubleshooting/hints/tips
95
2.4.4.4 How POP3 is integrated in a Notes mail environment
The Lotus Domino POP3 server task allows non-Notes client hosts to participate
in the Notes Mail environment without using the Notes Workstation software. In
this way, a POP3 server is integrated into the Notes Mail environment.
Figure 41. Integration of POP3 in a Notes mail environment
2.4.5 The SMTP task
Simple Mail Transfer Protocol (SMTP) runs on a Domino Server and is used for
the transfer of mail. It is not a mail client. The end user will need to have a mail
client such as Notes, IMAP, or POP3.
The following sections give some hints and tips concerning the SMTP task:
2.4.5.1 Starting and stopping the SMTP task
As described in the preceding section, the SMTP task can be started and stopped
like the POP3 task. Please refer to 2.4.4.1, “Starting/stopping the POP3 task” on
page 90 for the description and usage of the commands.
2.4.5.2 Enabling SMTP in the server’s administration function
To enable the SMTP listener task, choose Server --> Current Server Document
--> Basics.
96
Debugging UNIX Applications on OS/390
Figure 42. Enable the SMTP listener task
The next step is to set up and enable the ports for SMTP. The default TCP/IP port
for SMTP is port 25. Use the administration function Server --> Current server
document --> Ports --> Internet Ports --> Mail to enable the SMTP inbound and
outbound ports.
Lotus Domino for S/390—troubleshooting/hints/tips
97
Figure 43. Enable SMTP inbound and outbound ports
Loading SMTP, you should now see two SMTP tasks running as shown in Figure
40 on page 95: the SMTP listener task and the SMTP control task.
2.4.5.3 Problems sending mail to the Internet
We encountered a problem trying to send or receive mail to or from the Internet.
In the noteslog there was the following entry:
0090
0090
0090
0090
0090
+WSC/DOMINO 28/05/99 11:56:27
SMTPMTA: oseshlr0 Attempting SMTP
session with - 79
+WSC/DOMINO DE.IBM.COM - 23
+WSC/DOMINO 28/05/99 11:56:29 SMTPMTA: osesctl Outbound oseshlr0
SMTP session failed - 89
+WSC/DOMINO one or more messages.- 32
+WSC/DOMINO 28/05/99 11:56:31
SMTPMTA: osesctl Received message
notification - 79
It was because of this error that we missed the NSInteraddr statement in the
configuration file /etc/resolv.conf . This statement is used to define the IP address
of a name server in dotted decimal format.
2.4.6 How mail routes in a Domino system
These steps describe how mail routes in a Domino mail system:
1. A user creates and addresses a mail message to a recipient.
2. The user sends the message.
3. The user's client software:
• Uses Notes protocols to deposit the message in the MAIL.BOX database on
the user's Domino mail server.
98
Debugging UNIX Applications on OS/390
• Uses SMTP to send the message to the user's Domino mail server, which
must be running the SMTP listener task. The SMTP listener task deposits the
message in MAIL.BOX (Lotus Notes, IMAP clients, POP3 clients).
• Uses HTTP to send the message to the user's Domino mail server, which
must be running the HTTP task. The HTTP task deposits the message in
MAIL.BOX (Web clients).
4. The Router finds the message in MAIL.BOX and determines where to send
the message for each recipient. The Router calculates the next "hop" for the
message on the path to its recipients and uses the best protocol, either SMTP
or Notes routing, to transfer the message.
• Using SMTP routing, the Router connects to the destination server—the
recipient's mail server, a relay host, a smart host, or one of the servers in the
recipient's Internet domain—and transfers a copy of the message.
• Using Notes routing, the Router moves the message to the MAIL.BOX
database on the server that is the next hop in the path to the recipient's mail
server. The Router on that server moves the message to the next hop, until
the message is deposited in the MAIL.BOX database on the recipient's home
server.
5. The Router on the recipient's server finds the message (in MAIL.BOX on a
Domino server) and delivers it to the recipient's mail file.
6. The recipient uses a client that supports Notes protocols, POP3, IMAP, or
Web access (HTTP) to read the message, which is stored in the recipient's
mail file.
Note
Starting with Domino Version 5 there is no longer a need to have an
SMTP.BOX around. All mailing is handled using the MAIL.BOX.
.
Figure 44. Mail routing in a Domino environment
Lotus Domino for S/390—troubleshooting/hints/tips
99
2.4.7 Transaction logging in Domino R5
Domino Release 5 supports transaction logging and recovery. With this feature
enabled, the system captures database changes and writes them to the
transaction log. Then if a system or media failure occurs, you can use the
transaction log and a third-party backup utility to recover your databases.
Note
Transaction logging works with databases in Domino Release 5 format but not
with databases that use formats from earlier releases. After you enable
transaction logging, all databases in R5 format are automatically logged. To
check database formats, use the Files tab of the Administrator.
2.4.7.1 What is transaction logging
Transaction logging keeps a sequential record of every operation that occurs to
data. If a database becomes corrupted, you can "roll back" the database to a
point before it was corrupted and replay the changes from the transaction log.
A single transaction is a series of changes made to a database on a server.
Transaction logging provides three major benefits:
• In most situations, you no longer need to run the Fixup task to recover
databases following a system failure. Excluding Fixup results in quicker server
restarts since Fixup must check every document in each database, while
transaction log recovery applies or undoes only those transactions not written
to disk at the time of the system failure.
• Transaction logging saves processing time because it allows Domino to defer
database updates to disk during periods of high server activity. Transactions
are recorded sequentially in the log files, which is much quicker than database
updates to random, nonsequential parts of a disk. Because the transactions
are already recorded, Domino can safely defer database updates until a
period of low server activity.
• Using transaction logging simplifies your daily backup procedure. You can use
a third-party backup utility to perform daily incremental backups of the
transaction logs, rather than perform full database backups.
Note
Enabling transaction logging can improve server performance in most cases.
Transaction logging saves processing time because it allows Domino to defer
database updates to disk during high server activity. Transactions are recorded
sequentially in the log files, which is much quicker than database updates to
random, nonsequential parts of a disk. Because the transactions are already
recorded, Domino can safely defer database updates until a period of low
server activity.
100
Debugging UNIX Applications on OS/390
2.4.7.2 How to set up transaction logging
1. Ensure that all databases to be logged reside in the Domino data directory,
either at the root or in subdirectories.
2. From the Administrator, click the Configuration tab.
3. In the "Use Directory on" field, choose the server's Domino Directory.
4. Click Server Configuration, and then click Current Server Document.
5. Click the Transactional Logging tab.
6. Complete these fields, and then save the document.
Figure 45. Configuring transaction logging
The following parameters will be added to your notes.ini file:
Table 25. notes.ini parameters for transaction logging
Variable name
Description of the variable
Example
TRANSLOG_AutoFixup
If set to on, Domino automatically runs Fixup if a database is
corrupted and Domino cannot use the transaction log to
recover.
TRANSLOG_AutoFixup=1
TRANSLOG_MaxSize
The maximum size, in MB, for the transaction log. A value of
at least 192 MB is recommended.
TRANSLOG_MaxSize=1000
Lotus Domino for S/390—troubleshooting/hints/tips
101
Variable name
Description of the variable
Example
TRANSLOG_Path
Path name location of the transaction log. The default path
name is /logdir in the server's data directory. However, it is
strongly recommended to store the transaction log on a
separate device. If you change this field and have an existing
transaction log, you must use the operating system to move
all the log files to the new log path.
TRANSLOG-Path=/u/hauff/translog
TRANSLOG_Status
Enables or disables transactional logging.
TRANSLOG_Status=1
TRANSLOG_Style
Describes if the log files will be reused and overwritten or not
to be reused until they have been archived using a backup
utility.
TRANSLOG_Style=0
TRANSLOG_Performance
This field controls how often Domino records a recovery
checkpoint in the transaction log, which affects the server
performance.
TRANSLOG_Performance=2
TRANSLOG_UseAll
Set to 1 = use all available space on the device for the
transaction log.
TRANSLOG_UseAll=0
2.4.7.3 How to fix corrupted databases
Corrupted databases do not occur frequently when you use Release 5 databases
and transaction logging. The server uses the transaction log to restore and
recover databases after a system failure.
Note
You cannot use transaction logging on Release 4 databases. For these and
other reasons, it is recommeded that you convert Release 4 databases to
Release 5 and use transaction logging.
2.4.7.4 To disable transaction logging
1. Perform a full backup of all databases.
2. Edit the Transactional Logging field, and then save the Server document.
3. Restart the server so that the change takes effect. Domino runs restart
recovery to ensure all databases are consistent, and then disables transaction
logging.
2.4.7.5 FIXUP -J
This parameter causes to run FIXUP on Release 5 databases that are enabled
for transaction logging. Without this option, FIXUP generally does not run on
logged databases.
102
Debugging UNIX Applications on OS/390
2.5 Troubleshooting documents
In this section we describe how and what data to collect in case of a problem with
several parts of the Lotus Domino Server for S/390.
Lotus Domino S390 documentation especially Release Notes: Domino/Notes 5.0
(readme.nsf) and Domino Administration Help (help5_admin.nsf) in the
.../notesdata/help directory also provide very helpful troubleshooting information.
2.5.1 HTTP
This document provides a basic overview of the steps that Lotus Domino Support
personnel uses to troubleshoot Domino 4.6x and 5.0 Web server (HTTP) crashes
and hangs. It includes information for customers about the files required to
troubleshoot such issues.
2.5.1.1 HTTP crash
When troubleshooting crashes, several files are necessary to determine the cause.
The HTTP task can crash for a number of different reasons. The first step is to
ensure that HTTP is the crashing process. To accomplish this, Support personnel
must review the nsd output (before and after the crash if the error is reproducible)
and the notes log (log.nsf).
Possible reasons for crashes
• Agents or CGIs
• Bad icons or images
• Corrupted data
• Unusual form properties, for example, controlled-access sections, hide-when
formulas, etc.
Troubleshooting crashes may include disabling any suspected agents or CGIs,
performing database maintenance, rebuilding views, and other general measures.
Support must gather diagnostics for every crash to determine the following:
1. What the server was doing when it crashed
2. What URL the server was processing at the time of the crash
It is important to establish a pattern to the crashing URL, if indeed there is a pattern.
Customers will be asked to provide (preferably in separate files or attachments) the
following files for each and every crash. It will take a minimum of two to three
occurrences to detect a pattern to the crash.
Files to collect for every crash
• NSD: Support will request the NSD output produced from the crash.
• Domino logs: To help determine the flow of Web activity on the server,
customers will be asked to enable Domino logging (Access/Referer/Error logs
or DOMLOG.NSF) in the Server document to track URL hits during the crash
period. For details on these logs refer to 2.3.4.2, “HTTP logs” on page 58.
• Console output and/or LOG.NSF: Console output is preferred since this
data is typically not buffered. In some cases, errors are shown only on the
console. Customers will be asked to redirect console output using the
DEBUG_OUTFILE parameter in the NOTES.INI. How to enable this parameter
Lotus Domino for S/390—troubleshooting/hints/tips
103
is described in 2.5.8.1, “Implementing the DEBUG_OUTFILE parameter” on
page 115.
• NOTES.INI and Server document: This data is needed to determine if any
abnormal HTTP settings are affecting the crash.
2.5.1.2 HTTP Hang
HTTP hangs are more difficult to troubleshoot. More investigative work is required
with HTTP hangs, and several iterations of data must take place to narrow down the
problem.
Possible reasons for hangs
• CPU spin (due to excessive view rebuilds, corrupted views, or documents)
• Excessive agent executions or agent hangs
• Semaphore timeout issues
• Network or port binding problems
• Too much traffic on the server (as determined from Domino statistics)
To troubleshoot the hang, Support must know the exact circumstances of the
problem. First, Support determines if the server is truly in a hang state or is simply
encountering performance problems that cause it to become unresponsive for a
certain period of time. One way to determine whether the behavior is a performance
slowdown or a true hang is to remove the load from the server and see if HTTP
begins responding eventually, for example, after 5 to 10 minutes. This
troubleshooting step may not always be feasible, but it will confirm whether the
server has encountered a true hang state. If the server begins responding again,
then the server has not actually hung but has experienced a performance slowdown.
In such a case, Support troubleshoots this as a performance problem, not as a hang.
Files to collect for every hang
• Access log and/or DOMLOG.NSF
• LOG.NSF and/or console output
• NSD output
Additional files that may be requested:
• NOTES.INI and Server document
• STATREP.NSF
• Other operating-system level diagnostics
HTTP Hangs due to agent hangs
A majority of HTTP hangs are caused by a hang of a Web-triggered agent. There
are three phases in troubleshooting an HTTP hang due to an agent:
Phase 1: Determining if the hang is due to an agent
Phase 2: Finding the originating thread/URL hang
Phase 3: Determining the cause of the agent hang
Phase 3 requires several iterations of debug for the agent in question. For
troubleshooting agents, see 2.5.5, “Agents” on page 107.
104
Debugging UNIX Applications on OS/390
2.5.2 SMTP
To get information about the SMTP server and client, the Notes log can be
reviewed by searching for SMTP. To get more information about SMTP, add the
debug parameter SMTPDEBUG to the notes.ini.
Setting smtpdebug=1 forces minimal logging of the SMTP listener and
smtpdebug=2 shows some additional debugging information as you can see in
the following figure where a normal note transmission using Netscape
Communicator to a Domino S390 server is shown:
07/13/99 04:16:47
07/13/99 04:16:47
07/13/99 04:16:47
07/13/99 04:16:47
07/13/99 04:16:47
07/13/99 04:16:47
07/13/99 04:16:47
07/13/99 04:16:47
PM
PM
PM
PM
PM
PM
PM
PM
SMTP Server [805306408:00004-671154179] State change from Greeting to Greeting
SMTP Server [805306408:00004-671154179] Processing in Greeting state
SMTP Server [805306408:00004-671154179] State change from Greeting to Connected
SMTP Server [805306408:00004-671154179] Processing in Connected state
SMTP Server [805306408:00004-671154179] EHLO command received
SMTP Server [805306408:00004-671154179] Processing in Connected state
SMTP Server [805306408:00004-671154179] MAIL command received
SMTP Server [805306408:00004-671154179] Processing in Connected state
07/13/99 04:16:47 PM SMTP Server [805306408:00004-671154179] RCPT command received
07/13/99 04:16:47
07/13/99 04:16:47
07/13/99 04:16:47
07/13/99 04:16:47
07/13/99 04:16:47
07/13/99 04:16:47
07/13/99 04:16:47
PM
PM
PM
PM
PM
PM
PM
SMTP Server [805306408:00004-671154179] Processing in Connected state
SMTP Server [805306408:00004-671154179] DATA command received
SMTP Server [805306408:00004-671154179] Processing in Connected state
SMTP Server [805306408:00004-671154179] DATA command (cont.)
SMTP Server [805306408:00004-671154179] Processing in Connected state
SMTP Server [805306408:00004-671154179] QUIT command received
SMTP Server [805306408:00004-671154179] State change from Connected to Terminal
Note
Before trying the preceding debug parameter, it is imperative that you discuss
it with a Support Analyst beforehand. There may be issues surrounding its
use, or special precautions that must be considered prior to debug parameter
usage. Debug parameters may require a large amount of disk space,
dependent upon when the server crashes, the number of users, and the
number of databases that are opened on the server; the longer the server stays
up, the larger the debug output will be. These files can grow large enough to
cause disk space shortages.
In case of configuration problems refer to 2.4.5, “The SMTP task” on page 96 for
more information.
2.5.3 POP3
To get information about the POP3 server and client, the Notes log can be
reviewed by searching for POP3. To get more information about POP3, add the
debug parameter POP3DEBUG=1 to the notes.ini. This generates an output that
can be seen either in the Notes log or on the administrator’s console. The
following example shows the logon of a POP3 user named wpoppy—first with an
invalid (wrong password), then with a valid attempt to log on to the server using
Netscape Communicator:
Lotus Domino for S/390—troubleshooting/hints/tips
105
07/13/99 02:01:05 PM POP3 Server: [855638046:00004-503382019] Greeting state
07/13/99 02:01:05 PM POP3 Server: [855638046:00004-503382019] AuthenticateUser state
07/13/99 02:01:05 PM POP3 Server: [855638046:00004-503382019] AUTH command
07/13/99 02:01:05 PM POP3 Server: [855638046:00004-503382019] NO AUTH PARAMETER!
07/13/99 02:01:05 PM POP3 Server: [855638046:00004-503382019] AuthenticateUser state
07/13/99 02:01:05 PM POP3 Server: [855638046:00004-503382019] USER command
07/13/99 02:01:05 PM POP3 Server: [855638046:00004-503382019] AuthenticatePass state
07/13/99 02:01:05 PM POP3 Server: [855638046:00004-503382019] PASS command
07/13/99 02:01:05 PM POP3 Server: Authentication failure for wpoppy: Password not found in the Name and Address
Book entry or password did not verify
07/13/99 02:01:11 PM POP3 Server: [855638046:00004-503382019] Terminal state
07/13/99 02:01:16 PM POP3 Server: [855638046:00004-503382019] Greeting state
07/13/99 02:01:16 PM POP3 Server: [855638046:00004-503382019] AuthenticateUser state
07/13/99 02:01:16 PM POP3 Server: [855638046:00004-503382019] USER command
07/13/99 02:01:16 PM POP3 Server: [855638046:00004-503382019] AuthenticatePass state
07/13/99 02:01:16 PM POP3 Server: [855638046:00004-503382019] PASS command
07/13/99 02:01:16 PM POP3 Server: [855638046:00004-503382019] mail/wpoppy.nsf is now open for exclusive access
07/13/99 02:01:16 PM POP3 Server: [855638046:00004-503382019] Transaction state
07/13/99 02:01:16 PM POP3 Server: [855638046:00004-503382019] STAT command
07/13/99 02:01:16 PM POP3 Server: [855638046:00004-503382019] Transaction state
07/13/99 02:01:16 PM POP3 Server: [855638046:00004-503382019] LIST command
07/13/99 02:01:18 PM POP3 Server: [855638046:00004-503382019] Transaction state
07/13/99 02:01:18 PM POP3 Server: [855638046:00004-503382019] UIDL command
07/13/99 02:01:18 PM POP3 Server: [855638046:00004-503382019] Transaction state
07/13/99 02:01:18 PM POP3 Server: [855638046:00004-503382019] Transaction state
07/13/99 02:01:18 PM POP3 Server: [855638046:00004-503382019] RETR command
07/13/99 02:01:18 PM POP3 Server: [855638046:00004-503382019] Transaction state
07/13/99 02:01:18 PM POP3 Server: [855638046:00004-503382019] QUIT command
07/13/99 02:01:18 PM POP3 Server: [855638046:00004-503382019] Terminal state
07/13/99 02:01:18 PM POP3 Server: [855638046:00004-503382019] mail/wpoppy.nsf is closed for exclusive access
Note
Before trying the preceding debug parameter, it is imperative that you discuss
it with a Support Analyst beforehand. There may be issues surrounding its
use, or special precautions that must be considered prior to debug parameter
usage. Debug parameters may require a large amount of disk space,
dependent upon when the server crashes, the number of users, and the
number of databases that are opened on the server; the longer the server stays
up, the larger the debug output will be. These files can grow large enough to
cause disk space shortages.
In case of configuration problems refer to 2.4.4, “The POP3 task” on page 90 for
more information.
2.5.4 Router
To debug router problems, there is a parameter to be set in the notes.ini file,
which may give you some help or ideas on the problem.
debugrouter=0 (off,default), 1, 2 ,3
The most useful is debugrouter=3. This will display (in the log.nsf and on the
console) how the router is building the routing tables, and from this you can
determine the actual path taken. If you use the "set conf debugrouter=3"
statement at the console, this variable will take effect immediately. To get
106
Debugging UNIX Applications on OS/390
information on how the table is being built, quit and reload the router. Figure 46
shows an example of this:
> load router
07/16/99 01:02:25 PM Mail Router started for domain ITSO2
07/16/99 01:02:25 PM Router: Internet SMTP host domino4 in domain itso.ibm.com
Server CN=DOMINO3/O=ITSO2 added to server table, tasks = 0001.
Server CN=DOMINO4/O=ITSO2 added to server table, tasks = 0001.
Server CN=TOT105/O=ITSO2 added to server table, tasks = 0001.
Connection from CN=DOMINO3/O=ITSO2 to CN=TOT105/O=ITSO2 is Disabled
Connection from CN=DOMINO3/O=ITSO2 to CN=TOT105/O=ITSO2 has a Cost = 1 and Tasks
= 0001
Connection from CN=DOMINO4/O=ITSO2 to CN=TOT105/O=ITSO2 is Disabled
Connection from CN=TOT105/O=ITSO2 to CN=DOMINO1/O=ITSO has a Cost = 1 and Tasks
= 0001
Connection from CN=TOT105/O=ITSO2 to CN=DOMINO1/O=ITSO does not include Mail Rou
ting
Domain table is 0 segments (0 bytes)
Rule table is 0 segments (0 bytes)
Sorted Rules follow:
SymList table is 0 segments (0 bytes)
Server table is 1 segments (2048 bytes)
Connection table is 1 segments (2048 bytes)
Pull Connection table is 0 segments (0 bytes)
Symbol table has 11 buckets (22660 bytes)
Figure 46. Load router with variable DebugRouter set to 3
Here are some helpful commands to use in conjunction with the router:
Table 26. Useful commands to control the router task
Command
Result
Tell Router Delivery Stats
Shows Router delivery statistics.
Tell Router Compact
Compacts MAIL.BOX and cleans up open Router queues.
You can use this command to compact MAIL.BOX at any
time. If more than one MAIL.BOX is configured for the
server, each MAIL.BOX database will be compacted in
sequence.
Tell Router Show Queues
Shows mail held in transfer queues to specific servers.
Tell Router Exit / Quit
Stops the Router task on a server.
2.5.5 Agents
In this section we cover some troubleshooting items and possible pitfalls for
Agents/Agent Manager. For more information on Agents and Agent Manager
troubleshooting, see Domino Administration Help, Troubleshooting Agent
manager and agents (help5_admin.nsf) in the .../notesdata/help directory.
2.5.5.1 Parameter Debug_AMgr
Here are some parameter options that can be set in the notes.ini or dynamically
using the console to ease Problem Determination.
Lotus Domino for S/390—troubleshooting/hints/tips
107
Note
Before trying the debug parameters listed below, it is imperative that you
discuss them with a Support Analyst beforehand. There may be issues
surrounding their use, or special precautions that must be considered prior to
debug parameter usage. Debug parameters may require a large amount of
disk space, dependent upon when the server crashes, the number of users and
the number of databases that are opened on the server; the longer the server
stays up, the larger the debug outfile will be. These files can grow large
enough to cause disk space shortages.
To debug agents in a database, you can add the following to your notes.ini:
Log_Sessions=1
Log_AgentManager=1
Doppelganger=*
For the Doppelganger parameter the following options are valid:
Table 27. Doppelganger options
Options
Debug output
*
Turns on all following
c
Agent control parameters display
l
Agent loading event
m
Agent memory warning
p
Agent performance starts
r
Agent execution event
s
Agent scheduling event
v
Miscellaneous
Take care with these parameters and keep in mind that they are not needed in
most cases.
The following output can be seen in the notes log if using option “*”:
108
Debugging UNIX Applications on OS/390
g
g
07/12/99 03:51:17 PM AMgr: Daytime period is from '08:00:00 AM' to '08:00:00 PM'
07/12/99 03:51:17 PM AMgr: Nighttime period is from '08:00:00 PM' to '08:00:00 AM'
07/12/99 03:51:17 PM AMgr: Current control parameters in effect:
07/12/99 03:51:17 PM
AMgr: Daily agent cache refresh is performed at '12:00:00 AM'
07/12/99 03:51:17 PM
AMgr: Currently in Daytime period
07/12/99 03:51:17 PM
AMgr: The maximum number of concurrently executing agents is '1'
07/12/99 03:51:17 PM
AMgr: The maximum number of minutes a LotusScript/Java agent is allowed to run is '10'
07/12/99 03:51:17 PM
AMgr: Private Agents ACL - all users may run private agents
07/12/99 03:51:17 PM
AMgr: Restricted LS Agents ACL - select users may run LS agents with restricted access
07/12/99 03:51:17 PM
AMgr: Unrestricted LS Agents ACL - select users may run LS agents with unrestricted access
07/12/99 03:51:17 PM AMgr: Searching 'discuss.nsf' for agent documents
07/12/99 03:51:17 PM AMgr: Agent 'Send Newsletters
Send Newsletters' in 'discuss.nsf' is disabled
07/12/99 03:51:18 PM AMgr: Searching 'statmail.nsf' for agent documents
07/12/99 03:51:21 PM AMgr: Searching 'mail/wmichel.nsf' for agent documents
07/12/99 03:51:21 PM AMgr: Agent 'OutOfOffice
OutOfOffice' in 'mail/wmichel.nsf' added as a scheduled task
07/12/99 03:51:21 PM AMgr: Agent 'OutOfOffice
OutOfOffice' is scheduled to run next at: 07/12/99 03:45:00 PM
07/13/99 12:01:49 AM AMgr: Logging daily statistics for Manfred Hauff/Itso2
07/13/99 12:01:49 AM
Total scheduled runs: 1
07/13/99 12:01:49 AM
Total event triggered runs: 0
07/13/99 12:01:49 AM
Total errors: 0
07/13/99 12:01:49 AM
Total access denials: 0
07/13/99 12:01:49 AM
Total agent elapsed run time (seconds) 0
07/13/99 12:01:49 AM AMgr: Logging daily statistics for Wilhelm Michel/Itso2
07/13/99 12:01:49 AM
Total scheduled runs: 3
07/13/99 12:01:49 AM
Total event triggered runs: 0
07/13/99 12:01:49 AM
Total errors: 0
07/13/99 12:01:49 AM
Total access denials: 0
07/13/99 12:01:49 AM
Total agent elapsed run time (seconds) 1
07/13/99 12:01:49 AM AMgr: Total daily statistics
07/13/99 12:01:49 AM
Total scheduled runs performed: 5
07/13/99 12:01:49 AM
Total event triggered runs: 0
07/13/99 12:01:49 AM
Total unsuccessful runs: 0
07/13/99 12:01:49 AM
Total access denied runs: 2
07/13/99 12:01:49 AM
Total agent elapsed run time (seconds) 1
The output in the the previous figure can be seen on the administrator’s console
additionally.
2.5.5.2 Agent log
To get a quick idea what happens at an agent’s execution time, you can also
have a look at the agent log by selecting Agent --> Log on the action bar of a
database’s Agents view. See Figure 47 on page 110 (example log for the Out Of
Office agent):
Lotus Domino for S/390—troubleshooting/hints/tips
109
Figure 47. Agent log
2.5.5.3 Debugging at the server console - tell commands
This section describes the tell commands you can use with Agent Manager.
Tell Meagre Pause
Pauses scheduling of agents.
Tell Meagre Resume
Resumes scheduling of agents.
Tell Meagre Schedule
Shows the schedule for all agents scheduled to run for the current day. In
addition, the command shows the agent trigger type, the time the agent is
scheduled to run, the name of the agent, and the name of the database on which
the database runs. Checking the Agent Manager schedule lets you see if an
agent is waiting in one of the Agent Manager queues.
Agent Manager queues:
• E—Agents eligible to run
• S—Agents scheduled to run
• V—Event-triggered agents waiting for their events to occur
Trigger types:
• S—Agent is scheduled to run
• M—Agent is a new mail-triggered agent
• U—Agent is a new/updated document-triggered agent
Tell Meagre Status
This command shows a snapshot of the Agent Manager queues and displays the
Agent Manager settings in the Server document.
110
Debugging UNIX Applications on OS/390
> tell amgr status
> 07/14/99 04:43:07 PM AMgr: Status report at '07/14/99 04:43:07 PM'
07/14/99 04:43:07 PM
Agent Manager has been running since '07/14/99
02:51:20 PM'
07/14/99 04:43:07 PM
There are currently '1' Agent Executives running
07/14/99 04:43:07 PM
There are currently '0' agents in the Scheduled Task
Queue
07/14/99 04:43:07 PM
There are currently '0' agents in the Eligible Queue
07/14/99 04:43:07 PM
There are currently '0' databases containing agents
triggered by new mail
07/14/99 04:43:07 PM
There are currently '0' agents in the New Mail Event
Queue
07/14/99 04:43:07 PM
There are currently '0' databases containing agents
triggered by document updates
07/14/99 04:43:07 PM
There are currently '0' agents in the Document Update
Event Queue
07/14/99 04:43:07 PM AMgr: Current control parameters in effect:
07/14/99 04:43:07 PM
AMgr: Daily agent cache refresh is performed at
'12:00:00 AM'
07/14/99 04:43:07 PM
AMgr: Currently in Daytime period
07/14/99 04:43:07 PM
AMgr: The maximum number of concurrently executing
agents is '1'
07/14/99 04:43:07 PM
AMgr: The maximum number of minutes a
LotusScript/Java agent is allowed to run is '10'
07/14/99 04:43:07 PM AMgr: Executive '1', total agent runs: 2
07/14/99 04:43:07 PM AMgr: Executive '1', total elapsed run time: 2
2.5.6 Calendaring/Scheduling
In this section we cover some troubleshooting items and possible pitfalls for
Calendaring and Scheduling. More information about this item (in Domino R5)
can be found in the Release Notes: Domino/Notes 5.0 Troubleshooting Client/Server - Calendar and Scheduling Issues (readme.nsf) and Domino
Administration Help, Troubleshooting Agent manager and agents
(help5_admin.nsf) in the.../intestate/help directory.
2.5.6.1 Schedule Manager errors
The Schedule Manager errors in the Notes log file report information about certain
databases that may have a mismatch of a certain field in a calendar entry, with
respect to the name that is listed in the Busytime.nsf file and the Public Name &
Address Book (NAB). This does not necessarily mean that the Calendaring &
Scheduling system is not working.
Here are the most common error messages and possible solutions:
•“SchedMgr: Error processing calendar profile document (NoteID: NT00xxxxxx)
in database xx.nsf: Appointment record not found"
The appointment record or resource/room in the Resource Reservation
database has been deleted. This issue has been reported to Lotus Quality
Engineering.
•"Can't Find User in Name and Address Book"
The $BusyName in the Note ID does not exist in the NAB for some reason.
Usually the user actually left the company and physically does not exist in the
Public NAB any longer.
Lotus Domino for S/390—troubleshooting/hints/tips
111
This can happen if an administrator deleted a user from the Public Address
Book and did not choose to delete the mail file. AdminP will remove the
person document but not the mail file. When the server is started it finds the mail
file, but there is no person document associated with it. Removing the mail file
fixes the problem.
•"SchedMgr: Error processing appointment document (NoteID: NTxxxxxx) in
database mail\xx.nsf: Can't find appointment end time"
A reminder was created; reminders do not contain an end time.
•"SchedMgr: Error processing appointment document: Can't find appointment
start time"
First, make sure that the user name in the Calendar Profile matches the user's
name in the public NAB. If not, check the All Documents view in the mail file
for the Calendar and Delegation Profile documents. These two documents
should show in the view as draft documents with no subject and no author.
Once these are found and identified they can be deleted. Once these
documents are deleted, restart the server and the error should not occur
again.
2.5.6.2 Daylight Saving Time
Some geographic regions observe Daylight Saving Time (DST). This means that
twice a year (normally in March and October) the clocks have to be changed,
gaining or losing an hour.
Due to the current observance of differing DST settings within users'
configurations, some users are experiencing issues where Notes appointments
and events are appearing off by an hour. This document describes the settings
that should be implemented to address these issues.
If you work in a region that uses Daylight Saving Time, you should enable both
Notes and your operating system's Daylight Saving Time settings. Certain
regions do not observe DST, and if you work in one of these regions you should
not have Daylight Saving Time observed in either Notes or your operating
system.
Note: The settings are relevant to both Notes clients and servers.
Notes needs to know when the Daylight Saving Time should be handled. Prior to
Notes Releases 4.5.4 and 4.6.1 (including all localized versions), Notes used the
North American rule to enable the change to Daylight Saving Time, which is:
- the first Sunday of April and the last Sunday of October
Starting with 4.5.4 and 4.6.1, the international English version will by default use
the most common European rule, which is:
- the last Sunday of March and the last Sunday of October
You still have the opportunity to change the hardcoded rule and define your own
settings by specifying the DSTLAW variable in the NOTES.INI. For North
American versions, DSTLAW=4 1 1 10 -1 1, whereas the international versions
use DSTLAW=3 -1 1 10 -1 1 (see the following explanation).
3 = 3rd month
-1 = last week of the month
112
Debugging UNIX Applications on OS/390
1 = first day of the weekfirst
10 = 10th month
-1 = last week of the month
1 = first day of the weekfirst
2.5.6.3 Debugging at the server console - tell commands
Here are tell commands you can use with Schedule Manager.
Tell Sched Stats
Displays totals of reservations and appointments in the Free Time database.
> tell sched stats
> 07/14/99 03:30:07 PM SchedMgr: wilhelm michel/itso2 has 2 appointment(s)
07/14/99 03:30:07 PM SchedMgr: manfred hauff/itso2 has 3 appointment(s)
07/14/99 03:30:07 PM SchedMgr: Total of 2 user(s) with a total of 5
appointment(s)
07/14/99 03:30:07 PM SchedMgr: Total of 0 resource(s) with a total of 0
reservation(s)
07/14/99 03:30:07 PM SchedMgr: Total of 2 users/resources with a total of 5
appts/reservations
Tell Sched Show username
Displays the specified user's schedule on the server console. Use this command
to investigate problems in the Free Time database.
Tell Sched Validate
Immediately validates a Free Time database on a server. Validation occurs by
default at 2:00 A.M.; however, you can use this command to force it to occur
sooner. Another way to force validation is to stop and restart the Schedule
Manager.
Validation can take some time. You must issue this command at all servers where
mail files have been removed and/or added to ensure that old free time
information is removed and new free time information is added to the free time
database on the server.
Do not use this command when you add new users. The administration process
creates person documents for users in the Domino directory before creating their
mail file on their mail server. Schedule Manager watches for database creations
and automatically picks up new users' mail files.
Tell Sched Validate username
Validates the information for the specified user.
This command is faster than using the Tell Sched Validate command because it
allows you to validate individual users, rather than validating all of the data on a
server.
Lotus Domino for S/390—troubleshooting/hints/tips
113
2.5.7 Common info/data needed by support personnel in case of errors
Following is data and information that is important in case of problems with the
Domino Server on S390.
2.5.7.1 Common questions
• What version of OS/390 is used?
• What version of TCP/IP is installed?
• What patches are installed?
• Have the system files been modified?
2.5.7.2 Files and information to obtain prior to opening a problem
Further information:
Detailed description of the problem
• Problem symptoms
• What changed?
• Can it be recreated?
• Severity
Were all PREREQs met?
• IEASYSxx
• BPXPRMxx
• Required PTF list
• Required Software/Hardware (OS/390 version, TCP/IP, C++ class library,
Security product, etc.)
How was the server started?
• Telnet/rlogin session using non-superuser ID
• Foreground
• Background (rc.notes)
• DOMCOM (not officially supported)
• See DOMCOM discussion database
• TSO OMVS shell (not supported)
Log files
• log.nsf, notes.log, stats database, http logs (access-log.*, error-log.*,
agent_log.*, referer_log.*)
• MVS Operator's console
• SYS1.LOGREC
• Any CEEDUMPs created in /notesdata
• Are the sticky bits turned on for the executables (for example, running out of
linklist, LPA, or steplib)?
114
Debugging UNIX Applications on OS/390
Troubleshooting aids
1. Run nsd.sh as Notes Server Account in the Notes data directory. See 2.2.1,
“nsd.sh” on page 45 for information on how to handle this script.
2. Collect show stat, show tasks, show users, show tasks debug, show user
debug.
3. Collect statrep.nsf to show load over a week's time.
4. In the NOTES.INI file, implement setting log_sessions=1 and
server_show_performance=1 (see 2.5.8.2, “Performance events using
console” on page 116).
5. Do the stats show semaphore timeouts?
6. Is the server a mail server, database server, hub, or general purpose server?
7. Is a CEEDdump file created at the time of the crash? If so, obtain the file.
8. Does the log report an error, a panic, or fatal error? If not, is the Notes server
running in the foreground?
9. If the Notes server is not running in the foreground, is it already logging output
to a file? If so, panic or fatal error may be recorded at the end of the output
file. If not, ask to run in the foreground to capture the next crash or start with
an output file.
10. If there are no errors, panics, or fatal errors in the Notes log or output log, this
suggests a protocol problem or a performance issue.
11. If "Server is Not Responding" is generated on the clients, can commands be
issued at the server console?
12. Is it a specific database, a database view, or a user that causes the crash?
2.5.8 More troubleshooting information
Here are other troubleshooting items.
2.5.8.1 Implementing the DEBUG_OUTFILE parameter
You are experiencing a server crash and would like to implement the
DEBUG_OUTFILE=<filename> NOTES.INI parameter and have the following
questions:
• How should it be implemented?
• Where should the debug file location point on the hard drive?
• What happens when the location of the log file runs out of disk space? Does
this cause a server or operating system crash?
The first step in implementing this parameter is to add an entry to the NOTES.INI
file, as in the following example:
DEBUG_OUTFILE=/debuglog/debug.txt
Note
Before implementing the parameter, read this document completely.
Once the DEBUG_OUTFILE parameter has been added to the NOTES.INI file,
the Domino Server must be shut down and restarted, for the logging to begin.
Lotus Domino for S/390—troubleshooting/hints/tips
115
It does not matter what location the log file is placed; however, it is strongly
recommended that the DEBUG_OUTFILE parameter be configured to save the
LOG file in a location that is isolated from a directory where the operating system,
Domino Server, and other applications may potentially create files that are
necessary for those programs to function correctly. The reason for this is that the
LOG file is created on a partition where vital applications are running, and the
LOG file fills the hard drive/file system to its capacity, which may result in a
system crash. This crash may occur in any running application(s) that use that
file system. Placing it on an isolated HFS file avoids any of these potential
problems.
When the log file is placed in an isolated HFS not being used by any currently
running applications and then space problems take place, the Domino Server will
stop logging to the file specified in the DEBUG_OUTFILE parameter. With no
disk space to write to the log file, the Domino Server's overall performance is not
affected and continues to function.
Be aware that this log file may grow significantly in size over a period of time on a
Domino Server that has a high rate of activity. One example of this was on a
busy system that was running over a five day period with the DEBUG_OUTFILE
parameter enabled. The log grew to a size of 125 MB. Again, the size that the
log file will grow to depends on the activity rate of that server.
The log file cannot be deleted while the Domino Server is running. If an attempt
is made to delete the file, a sharing violation will occur.
2.5.8.2 Performance events using console
How can performance events be routed to the server console?
To display server performance events to the server console, type the following
command:
SHOW PERFORMANCE
This will set the NOTES.INI Server_Show_Performance parameter to a value of
1. If the NOTES.INI Server_Show_Performance parameter value is set to 0,
server performance events will be displayed in Notes Log.
To stop server performance events from being displayed on the console type SHOW
PERFORMANCE again and performance events will no longer be displayed on the
console.
On the Server Console the concurrent transactions will display as shown below:
> show performance
07/15/99 04:05:23 PM SERVER_SHOW_PERFORMANCE changed to 1.
Server Performance Monitoring is now enabled.
>
07/15/99 04:05:41 PM 12 Transactions/Minute, 5 Users
> show performance
07/15/99 04:05:50 PM SERVER_SHOW_PERFORMANCE changed to 0.
Server Performance Monitoring is now disabled.
>
116
Debugging UNIX Applications on OS/390
2.5.8.3 Lotus Domino’s helpful troubleshooting documentation
To get more information on troubleshooting for all areas in the Lotus Domino
S390 server, there are two very helpful databases that are delivered with the
product and can usually be found in the .../notesdata directory.
Release Notes
This is database readme.nsf.
Figure 48. readme.nsf
Lotus Domino for S/390—troubleshooting/hints/tips
117
Domino 5 Administration Help
This is database help5_admin.nsf.
Figure 49. help5_admin.nsf
2.6 Information sources
This section shows very complete lists about APARs, books, and sites with
additional information about Lotus Domino for S/390.
2.6.1 Required APARs and PTFs
Information about APARs and PTFs needed for Lotus Domino S/390 are listed at
http://www.s390.ibm.com/products/domino/domptf.html
118
Debugging UNIX Applications on OS/390
Figure 50. APARs and PTFs needed for Domino S390
2.6.2 Web sites
More information about related IBM and Lotus products can be found at the
following Web sites:
• The IBM Homepage: http://www.ibm.com
• The Lotus Homepage: http://www.lotus.com
• Domino and Notes document library: http://www.notes.net/notesua.nsf
• Domcon download page:
http://www.s390.ibm.com/products/domino/domcon/dmcndl.html
• The UNIX System Services Page: http://www.s390.ibm.com/unix
Lotus Domino for S/390—troubleshooting/hints/tips
119
Figure 51. UNIX System Services (USS) site
• The Domino R5 home page: http://www.lotus.com/home.nsf/welcome/r5home
Figure 52. Domino R5 home page
120
Debugging UNIX Applications on OS/390
• The Domino tools zone: http://www.lotus.com/home.nsf/welcome/dominotools
Figure 53. Domino tools
• Lotus Domino for S/390 - Tools for Developers:
http://www.s390.ibm.com/products/domino/domdown.html
Figure 54. Lotus Domino - Tools for Developers
Lotus Domino for S/390—troubleshooting/hints/tips
121
• UNIX System Services documentation:
http://www.s390.ibm.com/oe/bpxa1pub.html
Figure 55. UNIX System Services library
2.6.3 Redbooks
To see all available redbooks for Lotus Domino for S/390, simply open the
Internet site http://www.redbooks.ibm.com and search for "domino":
122
Debugging UNIX Applications on OS/390
Figure 56. Redbooks
2.6.4 Other information sources
Here is additional information.
2.6.4.1 Lotus Domino documentation
The Domino Server documentation delivered on the installation CD contains
some very useful help databases such as:
Figure 57. Lotus Domino R5 help documentation
These databases are usually installed to the .../notesdata/help directory.
If HTTP is enabled on your server, the help documentation can also easily be
accessed using any Internet browser by simply selecting the server name or
IP-Adress (default) as shown in Figure 58 on page 124:
Lotus Domino for S/390—troubleshooting/hints/tips
123
Figure 58. Lotus Domino R5 documentation using an Internet/intranet browser
2.6.4.2 How to debug in OS/390
• OS/390 MVS Interactive Problem Control System (IPCS) Commands,
GC28-1754
This book presents, in alphabetical order, the IPCS commands,
subcommands, and dialog commands. It provides reference information about
using the Interactive Problem Control System and gives examples and output
generated by the commands. This book is intended for system programmers
who diagnose problems and debug dumps on OS/390.
• OS/390 MVS Diagnosis: Tools and Service Aids, SY28-1085
The tools and service aids described in this book are designed to help in
identifying the symptoms of the problem, gathering relevant data from system
data areas to isolate the problem to the component level, and analyzing the
component to determine the cause of the problem. This publication explains
how, why, and when to use IBM tools and service aids programs.
124
Debugging UNIX Applications on OS/390
Chapter 3. Novell Network Services
This chapter has information about the product Novell Network Services (NNS)
for OS/390, which features NetWare Directory Services (NDS). It is also called
NNS for OS/390. This is the first release of the product.
By hosting NDS and other applications on S/390, you get all the classic strengths
that have made S/390 the defining standard in enterprise computing. The high
availability of S/390 is the result of many years of refinement and this kind of
availability is not easily achieved. Specific S/390 hardware has been designed
over the years to assist the operating system and application subsystems to
create a highly reliable and serviceable system.
3.1 Benefits
Directory Services is one of the most critical components of your network, and
you want to make it as reliable as possible, so it should be deployed on the most
reliable enterprise server.
By running NDS on an S/390 rather than multiple instances on smaller platforms,
you can enhance productivity and help lower network administration costs. NDS
for OS/390 provides a compatible implementation of NDS Version 4.10b across
your enterprise in a way that is transparent to users and administrators.
For existing NDS environments, Novell Network Services for OS/390 provides a
seamless migration toward a consolidated directory infrastructure with minimal
disruption, while leveraging your current NetWare TM administration skills and
your S/390 infrastructure.
Other Novell Network Services for OS/390 benefits can include:
• Reduction/consolidation of NDS servers
• Support for NetWare-IP in addition to IPX/SPX
• No client change required
• Support for NDS read/write replication and synchronization across multiple
servers or partitions
• Reduction in total cost of ownership
• NetWare Administrator (NWAdmin) providing a single point of administration
through icon-oriented management, which can reduce administration costs.
3.1.1 NDS and NDS practices
This section gives a brief tutorial on NDS and NDS practices.
In case you are not familiar with NDS practices, here is a quick tutorial on
Novell's and customers' recommendations, based on actual experiences. There
probably is one NDS tree for the entire enterprise. This tree then is subdivided
into partitions, for example, maybe one partition per geographical site or one per
product division or any other meaningful administrative subdivision. Neither the
tree nor the partitions physically exist in a file or a data set. They are logical
descriptions, kind of like high-level qualifiers. Contained in each of these
partitions are NDS objects. Objects are used to describe every entity that you
© Copyright IBM Corp. 1999
125
administer with NDS. For example, a user is an object, a server is an object, as is
a printer or a program or a data volume, etc. Objects that belong together are
associated with each other, common objects can be managed as groups to ease
administration, permissions are defined to control access, etc. These are the
tasks of the LAN administrators, and their results are stored in what is called NDS
replicas. These are replicas of the partitions. They are physical files on NetWare
servers, and for availability and disaster-recovery purposes, it is recommended
that there always be three replicas per partition. And, to prevent clogging up the
network with NDS traffic it is recommended that there be no more than three
replicas per partition. It also turns out that based on actual experience more than
three replicas is unnecessarily redundant. One last bit of information is the
administrator updates what is called the master replica. The NDS program then
automatically propagates every change to the master replica to both of the
read/write replicas, and what makes NDS so efficient is NDS propagates only the
changes, not the entire replica. The collection of all the replicas is called the NDS
Database. In practice, the NDS databases do not need to dedicate NetWare
servers to function solely as "NDS Servers", but rather share one of the
production NetWare file/print/application servers. This is in sharp contrast to
Windows NT Domains where Windows NT Servers (in practice, never fewer than
three, and almost always many more) are dedicated to doing only Domain data
storage and processing. This distinction is more apparent in the following
paragraphs, because it will open up an opportunity to eliminate Windows NT
Domain Servers without requiring you to install NetWare on any LAN servers.
3.1.2 Uses of NNS for OS/390
This section mentions some suggested uses of this first release of NNS on
OS/390 and its advantages.
• Store one of the read/write replicas of each of your NDS partitions on OS/390,
rather than on another NetWare server, thus, making OS/390 the master
repository of your entire NDS structure.
• This then provides a highly reliable storage and a full-function data
management system for this most important portion of your LAN
environment. It also brings to this LAN data the full power of OS/390 data
recovery and disaster recovery tools and disciplines, and probably more
importantly total confidence that your NDS data can always be recovered
should that ever be necessary. Full confidence in recovering backed up
LAN data is a serious shortcoming of today's LAN data management tools.
Also, your company's NDS data will now get the full protection of your
long-proven mainframe disaster-recovery discipline and practices,
something which normally is not possible with most LAN disaster-recovery
plans.
• Eliminate some of your Windows NT Domain Controllers without having to
install NetWare on any LAN server.
• Prior to NDS for OS/390, to replace Windows NT Domains with NDS a
customer would have to install a NetWare LAN Server. This is
objectionable to some customers even though they recognize the
superiority of NDS over Windows NT Domains. This migration from
Domains to NDS now can be accomplished with NNS for OS/390 without
having to install NetWare on any LAN servers.
126
Debugging UNIX Applications on OS/390
3.2 Introduction
Novell Network Services for OS/390 is a Novell Directory Services (NDS) server
based on Novell Cross-Platform Services 4.10b (NCPS). This product provides
support for:
• Client for DOS/WIN (VLMs) Version 1.21 or above
• Client 32 for DOS and Windows 3.1x Version 2.11
• Client 32 for Windows 95 Version 2.11
• Client 32 for Windows NT Version 1.0
• WIN-95 Client, distributed as msnds.exe
• WIN-NT 4.0 NetWare Client and Gateway
Only the Ethernet network is supported.
3.2.1 The structure of NNS
This section illustrates the overall architecture of Novell Network Services. On a
host operating system, it runs as a series of drivers and application programs.
These modules divide into two packages: transports and services.
Figure 59 illustrates the major modules of these two packages; the NetWare
services are shown in the upper right-hand corner.
Figure 59. Structure of NNS
Novell Network Services
127
The ovals represent user-space processes. The rectangles represent kernel
drivers. The lines represent the significant paths of communication and are not
meant to be inclusive. The resources under the bottom dashed line indicate some
of the host system modules that the Novell Network Services server depends on.
Novell Network Services has three controlling daemons:
• NetWare/IP (NWIP) daemon—controlling daemon for the NetWare/IP stack
• NetWare Protocol STACK (NPS) daemon—the controlling daemon for the
NetWare transports
• NetWare daemon—the controlling daemon for the NetWare services
The NetWare transport modules can run independently of NetWare services.
3.2.2 Features of NNS
3.2.2.1 Novell Directory Services (NDS)
Novell Directory Services provides a global naming service that is distributed
across the entire NetWare network with a single point of administration. NetWare
users log into the Directory tree and, with appropriate rights, have access to any
resource on the network, regardless of physical location.
NDS provides:
• Distributed information service that stores many types of data
• Scalability and reliability provided by its ability to be partitioned and replicated
• X.500 modeling
• Access services through the use of Access Control List (ACL)
• An extensible schema, enabling an application developer to customize the
information contained in the Directory
3.2.2.2 Client support and utilities
Allows workstations running DOS, Windows, Windows 95, or Windows NT to log
in to the NetWare server and share network resources. Novell Network Services
provides each client type with a set of NetWare utilities that allow users to
configure their NetWare environment and to manage some aspects of the
NetWare server.
3.2.2.3 Account security
Provides secure NDS authentication with private key/public key encryption login
restrictions.
3.2.2.4 SANDS and SCALE server operating modes
Novell Network Services servers can operate in either of two modes: Stand-alone
NDS (SANDS) or SCALE. In the SANDS mode, a single NDS server provides
most of the required NDS functions, such as ACLs, Authentication Time
Synchronization, and Bindery Emulation and Schemes. It lacks the function that
allows NDS to be a global, distributed, and replicated directory. A SANDS server
is the only server in a tree but still visible to all other NDS servers. A SCALE
server includes the functions of SANDS server and also Partition Management,
Replication, and Synchronization with other servers or trees outside its tree.
128
Debugging UNIX Applications on OS/390
3.2.3 NNS functions
• Customer support for central and hierarchical management of users, groups,
networks, security, and applications
• NetWare/IP support in addition to IPX support
• No Front End Processor (FEP) required
• Acts as NDS server for NetWare PC clients/browsers
• Acts as router for other servers
• Transparent to administrators and users
• Clients supported: Windows 95, Windows 98, Windows NT, DOS32
• Frame types supported: Ethernet 802.2, Ethernet II, Ethernet Snap
- Need router to support other LAN/frame types
The IPX protocol is the communication method between clients and servers in a
Novell network. IPX can travel directly on the wire or it can be encapsulated with
TCP/IP. The latter is known as NetWare/IP. Novell Network Services supports
both communication methods.
• NetWare/IP requires Domain Name Service (DNS). It also requires Domain
SAP/RIP service (DSS). DNS and DSS are not included in Novell Network
Services. NetWare/IP requires its own domain.
• IPX requires at least one Fast Ethernet OSA-2 (FENET OSA-2) installed on
the system. Fast Ethernet OSA-2 supports Ethernet II, Ethernet 802.2 and
Ethernet SNAP frame types. Multiple OS/390 images can share the same
card.
3.2.4 Licensing system
The license for NDS is included with NNS for OS/390. It includes full SCALE
functionality and limited client login capability.
3.2.5 NNS networking
Figure 60 on page 130 shows an example of NNS networking using a single OSA
card that can support multiple OS/390 systems in an LPAR environment:
Novell Network Services
129
Figure 60. NNS networking
3.3 Installation and customization
This section has information about how NNS on OS/390 is installed and
customized. The NNS product was installed on an OS/390 V2R7 system for this
installation.
After NNS for OS/390 has been installed using SMP/E, update parmlib on the
target system with the following changes:
• Add library SINRLMOD to the appropriate LNKLSTxx member.
• Add library SINRLMOD to the appropriate PROGxx member to APF-authorize
the library.
• Update the BPXPRMxx parmlib member to add a FILESYSTYPE statement
for STREAMS.
For more important information see the PROGRAM DIRECTORY of NNS for
OS/390 regarding the FMIDs HNWS110 and JBB6616, for Program Number
5655-B12 and with Document Number GI10-0632.
For more information see Novell Network Services for OS/390 Installation,
GA22-7312.
130
Debugging UNIX Applications on OS/390
3.3.1 Software requirements
This section describes the software requirements of NNS.
• OS/390 V2R6 (FMID HBB6606) or a later release /Component ID 564701NWS
- OS/390 2.6.0 BCP APAR OW34660
- USS UNIX System Services (FMID HOT1180) with APAR OW34480 or
higher
- If OSA-2 feature used, require OSA/SF V1R2 (FMID H0G1200) with APAR
OW33393
- RACF Security Server feature enabled
Figure 61 shows the entry in the IFAPRDxx parmlib member to enable the
RACF Security Server feature:
PRODUCT OWNER('IBM CORP')
NAME(OS/390)
ID(5647-A01)
VERSION(*) RELEASE(*) MOD(*)
FEATURENAME('SECURITY SERVER')
STATE(ENABLED)
Figure 61. IFAPRDxx parmlib member
• HFS requirements
- Minimum size 4.492 tracks on 3390 to install
Note
If you are installing NNS for OS/390 on OS/390 V2R7 system (or higher
release) target system, remove FMIDJBB6616 from the APPLY job. Do not
install FMID JBB6616 since FMIDJBB16 is superseded and deleted by OS/390
V2R7 BCP. If you are installing NNS for OS/390 on an OS/390 V2R6 target
system, you must install both FMIDs HNWS110 and JBB6616.
3.3.2 Hardware requirements
In this section we describe the hardware requirements of NNS.
• S/390 G2 or newer class processor if running NWIP; S/390 G3 or newer
processor if running IPX
• OSA2 Fast Ethernet card (using IPX)
3.4 Customization
In this section we describe about how to customize NNS for OS/390. Ensure that
the sample job INRISMKD has created the HFS directories for NNS. Set up the
UNIX System Services (USS) environment for NNS, see 3.4.1, “UNIX System
Services environment” on page 132 and IPL the system.
Novell Network Services
131
3.4.1 UNIX System Services environment
3.4.1.1 NNS volumes
This section informs you of the volume definitions that must be done related to
HFS.
For Novell Network Services to function, the SYS volume must be installed and
mounted in the HFS data set as /etc/netware/vol.
Its control directory is also kept in that same directory. The size of the SYS
volume should be 100 MB as a minimum and is limited to 120 MB. The
/etc/netware/vol directory must be mounted as a separate HFS. No additional
volumes are allowed.
The PUBLIC and LOGIN directories can be populated with utilities to allow clients
to access the server. These utilities and associated files are installed in the
/usr/lpp/netware/SYS directory and can be copied to the /etc/netware/vol/SYS
directory if it is desired to make them available to clients.
OS/390 UNIX System Services Planning describes the process for allocating and
mounting an HFS data set. A brief example is shown here to illustrate one way to
allocate and mount an HFS data set to use as a SYS volume:
ALLOCATE DATASET('OMVS.NWSYS.HFS') DSNTYPE(HFS) SPACE(150,0)
DIR(1) CYL
FREE DATASET('OMVS.NWSYS.HFS')
MOUNT FILESYSTEM('OMVS.NWSYS.HFS') TYPE(HFS)
MOUNTPOINT('/etc/netware/vol')
To store data objects, configuration information, etc., an additional HFS data set
must be allocated. The calculation for space required depends on the size of the
customer tree. We recommend a minimum size of 200 CYL. This volume must be
installed and mounted in the HFS data set as /etc/netware.
A brief example is shown here to illustrate one way to allocate and mount an HFS
data set to be used as a volume to store data objects, configuration information,
etc. ALLOCATE DATASET('OMVS.NW.HFS') DSNTYPE(HFS) SPACE(200,0)
DIR(1) CYL
FREE DATASET('OMVS.NW.HFS')
MOUNT FILESYSTEM('OMVS.NW.HFS') TYPE(HFS)
MOUNTPOINT('/etc/netware')
Recommendation:
Issue the MOUNT commands in the USS ISHELL to get a better explanation in
case of a failure. But this mount is only valid till the next IPL. Thus, also define the
MOUNTs to the BPXPRMxx parmlib member to mount both volumes as HFS data
sets automatically.
3.4.1.2 STREAMS considerations
Novell Network Services requires STREAMS support in the OS/390 UNIX kernel.
STREAMS runs as a physical file system (PFS) inside the kernel. It is started with
132
Debugging UNIX Applications on OS/390
a FILESYSTYPE statement in the BPXPRMxx member. Note that the change will
not take effect until the next system IPL.
Note
It is in progress to change the FILESYSTYPE related values dynamically also.
See OS/390 Release 8 or 9.
An example of this statement is:
FILESYSTYPE TYPE(STREAMS) ENTRYPOINT(BPXTNSTR) PARM('-m
INRKMAP -h 1000')
The values for the operands are as follows:
• TYPE must be STREAMS
• ENTRYPOINT must be BPXTNSTR
PARM is a quoted string of options. Unless otherwise specified, if
multiple options are specified, only the last is used. The options are:
—m (mapname)
Is the name of a load module that contains the device and module information for
a set of device drivers to be loaded. To load the Novell Network Services required
STREAMS modules, use INRKMAP as the map name.
Note
Ensure that the following message is shown in the SYSLOG to indicate that the
STREAMS support is available:
BPXTN005I STREAMS module map INRKMAP loaded
—h (high_storage)
Specifies the maximum amount of storage allocated for STREAMS message
blocks. This number is specified in units of kilobytes.
Default: —h 280
—t (trace_opt)
Specifies the type of trace to enable. Multiple —t options can be specified. In
addition to this specification, SYSOMVS CTRACE must be enabled for character
special for the tracing to occur. Default: —t normal.
off
Disable tracing.
all
Enable all trace types.
code or nocode
Enable/disable tracing for internally coded traces.
proc or noproc
Enable/disable tracing for procedure entry and exit.
normal or nonormal
Enable/disable tracing for code and procedure.
Novell Network Services
133
diag or nodiag
Enable/disable tracing for internal debugging.
data or nodata
Enable/disable tracing for message block data content
nw or nonw
Enable/disable tracing for NetWare device tracing.
3.4.1.3 User IDs required
Novell Network Services requires special OS/390 user IDs and groups to store
files and NDS objects in the file system. Novell Network Services uses NDS to
control access to the NetWare data. It uses these user IDs to store the NetWare
data on OS/390. The required user IDs and groups are NWROOT, NWUSER,
NOBODY, NWGROUP, and NOGROUP. Use the following sample statements to
create these definitions from a TSO session:
• ADDUSER NWROOT OMVS(UID(0))
• ADDUSER NWUSER OMVS(UID(0))
• ADDUSER NOBODY OMVS(UID(0))
• ADDGROUP NWGROUP OMVS(GID(0))
• ADDGROUP NOGROUP OMVS(GID(0))
Recommendation:
Ensure through the RACF LU and LG commands that the users and groups are
defined. An example is:
LU (NWROOT) OMVS
LG (NWGROUP) OMVS
3.4.2 Environment variables
Novell Network Services uses the NLSPATH shell environment variable to
access the message catalog directory. The directory that holds the message
catalogs must be added to the NLSPATH shell variable. For example, the
following command would add the Novell Network Services path to the NLSPATH
environment variable:
• export NLSPATH=$NLSPATH:/usr/lpp/netware/nls/msg/%L/%N
Use the PATH environment variable to define the default command path. The
command directory, /usr/lpp/netware/bin, can be added to the PATH environment
variable. For example:
• export PATH=$PATH:/usr/lpp/netware/bin
Recommendation:
Add both PATH commands to your profile because they are only valid for the
actual ISHELL session.
For more information see, “Server installation” on page 148.
See OS/390 UNIX System Services Planning, SC28-1890, for more information
about SHELL environment variables.
134
Debugging UNIX Applications on OS/390
3.4.3 Network configuration
This section describes how to set up the network using the OSA2 configuration
for IPX and to set up the network using NetWare/IP.
Novell Network Services can be configured with as many FENET OSA-2
interfaces as can be installed on a system. If more than one is used, then in
addition to routing packets to the server, Novell Network Services will also route
IPX packets between the networks attached to each FENET OSA-2.
The configuration information required to make this work is:
• OSA Name—This is an 8-character name that must be in the format
INRT11nn where nn is a two-digit decimal number within the range 00 to 99.
• Device Numbers—Use these hardware numbers associated with the FENET
OSA-2 to identify the communication path between VTAM and the hardware.
Only one pair of device numbers is required for each OS/390 image using the
FENET OSA-2.
• Character Special Files—Use character special files that must exist in the /dev
directory for each FENET OSA-2. These files are created dynamically by
Novell Network Services using defaults. These defaults can be overridden if
desired by creating a file called /etc/netware/devices. A sample is installed
with the product in the /usr/lpp/netware/samples directory. This file contains
an entry for each character special file. Each entry consists of the following:
- Character Special File Name
- OSA Name
The default files created are:
• inrdo0 INRT1100
• inrdo1 INRT1101
• inrdo2 INRT1102
• inrdo3 INRT1103
• inrdo4 INRT1104
• inrdo5 INRT1105
• inrdo6 INRT1106
• inrdo7 INRT1107
Each FENET OSA-2 can be shared between NNS servers running on different
OS/390 images. Each image is configured with its own device number pair. A
device on the LAN will see two servers, each with the same Network Node
Number (Ethernet MAC address); however, each is configured with a unique
Internal Network Number. Therefore, the complete IPX Address is unique.
Recommendation: Do not use the defaults to prevent the following messages
shown in the SYSLOG as:
INRM0004I IPX Activate Failed. TRLE=INRT1101 Major=1101
INRM0004I IPX Activate Failed. TRLE=INRT1102 Major=1102
INRM0004I IPX Activate Failed. TRLE=INRT1103 Major=1103
Novell Network Services
135
We only use the first file inrdo0 INRT1100.
3.4.3.1 Frame types on the Ethernet LANs (logical LANs)
The FENET OSA-2 supports the following frame types:
• Ethernet 802.2
• Ethernet II
• Ethernet SNAP
You can use more than one frame type simultaneously on a LAN. Each one is
considered its own logical LAN since a device using one frame type will not be
able to see devices using other frame types even if they are on the same LAN
segment. FENET OSA-2 does support multiple frame types on the same LAN. It
is recommended that multiple frame types on a single LAN segment be avoided
as it results in extra traffic on the LAN for the RIPs and SAPs. Also, when devices
from one logical LAN try to communicate with another logical LAN, the packets
must go into a router and back out onto the LAN, thereby doubling the LAN traffic.
3.4.3.2 IPX network numbers
IPX addresses consist of three parts:
• Network number—Use an 8-digit hexadecimal number to identify each logical
LAN and each server in the network. Each LAN and server must be configured
with a unique network number. The network number for the server is known as
the internal network number. The network number for the LAN is known as the
external network number.
• Node address—This is the address associated with an Ethernet card. It is
usually burned in by the manufacturer. The server typically has a node
address of 1.
• IPX socket number—Use this number to identify the destination of the packet
within a server. Some are hardcoded for specific purposes (for example, 451
is for NCPs) and the rest are allocated dynamically. The network numbers
must be configured in Novell Network Services. An internal network number
must be configured using nwcm and must be unique in the network. An
external network number must be configured for each frame type on each
Ethernet LAN. The external network number must be the same as is
configured for all routers using the same frame type on the same network
segment. Clients do not need to configure their IPX addresses as they will
automatically discover their addresses when they get onto the network.
3.4.3.3 NetWare/IP configuration
In this section we mention basic information about how to configure an IP
connection. For more information about how to set up NNS using NetWare/IP as
connectivity see Novell Network Services for OS/390 Supervising the Network,
SA22-7317.
136
Debugging UNIX Applications on OS/390
Note
• During our installation under OS/390 V2R7 the NNS product was
successfully tested with IPX as connection type only.
• Under OS/390 V2R6 the NNS product was successfully tested with
NetWare/IP as a connection type within the development environment also.
Before configuring NetWare/IP on the NCPS server, make sure the following are
functioning:
• DNS server with a NetWare/IP domain
• DSS server
• TCP/IP protocol on the host system for NCPS server
Use the ping command to make sure the servers can see and communicate with
each other. The NetWare/IP software consists of the following modules:
• NWIP daemon—initializes the NWIP driver, sets up communication, and then
maintains the SAP and RIP databases by synchronizing them with a DSS
server.
• NWIP driver—performs the communication tasks through its links to the UDP
port and the IPX driver. After the NWNET package is installed, NetWare/IP
needs to be configured for it to initialize. This involves:
- Configuring the NetWare/IP LAN
- Customizing the NetWare/IP process
NetWare/IP consists of both server and client software, which enables servers
and clients to use TCP/IP instead of, or in addition to, IPX. Nodes on an existing
IP network have transparent access to NetWare services and applications. Both
servers and clients must be configured to use IP as the transport protocol. The
following are not included in Novell Network Services and are required for a
NetWare/IP server configuration as prerequisites:
• Domain Name System (DNS)—Provides a centralized database of
host-to-address mapping
• Domain SAP/RIP Service (DSS)—Maintains and distributes SAP and RIP
information for the NetWare/IP network.
• Forwarding Gateway—A NetWare/IP server that forwards the SAP and RIP
information to the NetWare/IP network.
NetWare/IP for NCPS servers is based on, and dependent upon, the NetWare/IP
2.2 product currently available for native NetWare 4.11 servers. Table 28 lists
sources for the support services not included in the NetWare/IP for NCPS
software but which are required by NetWare/IP.
Table 28. Server prerequisites
Name
Description
Source
Domain Name System
(DNS)
Provides a centralized
database of host-to-address
mapping
UNIX server
Novell Network Services
137
Name
Description
Source
Domain SAP/RIP
Service (DSS)
Maintains and distributes SAP
and RIP information for the
NetWare/IP network
Native NetWare/IP
server
Forwarding Gateway
A NetWare/IP server which
forwards the SAP and RIP
information into the
NetWare/IP network
Native NetWare/IP
server
Use the required character special file (/dev/nwip), which is created automatically
by NNS when configuring the nwcm lan_x_adapter parameter for NetWare/IP.
A DNS server maintains a centralized database of host names and addresses,
and TCP/IP nodes use the database to locate nodes on the network. These
addresses are organized into domains. Figure 62 illustrates this concept. The
domain is divided into three subdomains and those subdomains into other
subdomains. The white squares represent workstations and servers in the
network.
Figure 62. NetWare/IP domain
NetWare/IP uses only a few of the features of DNS. It requires that you create
and set up a special NetWare/IP domain. The domain is created like any other
domain but is different because it cannot have any subdomains or hosts. It
becomes a logical domain for all the NetWare/IP servers on the network,
regardless of their physical location. The NetWare/IP domain unites these
servers into a single domain. NetWare/IP also requires that a unique IPX external
network number be assigned. All NetWare/IP servers in the same domain must
be configured with the same IPX external network number. This is used by the
IPX networks and the NetWare/IP network.
138
Debugging UNIX Applications on OS/390
The native NetWare/IP modules allow the native NetWare server to become a
DNS server. However, NetWare/IP can use the standard DNS servers found on
TCP/IP networks. The NetWare/IP modules for NCPS do not include DNS
functionality.
NetWare/IP servers and clients use the DNS server to find the nearest DSS
server in the NetWare/IP domain. The DSS server must be a native NetWare
server.
Table 3 shows the sources for information about configuring a DNS server:
Table 29. DNS source information
Topic
Source
Required DNS features
NetWare/IP Administration Guide for
NetWare 4.11
Configuring a native NetWare server as a
DNS server
NetWare/IP Administration Guide for
NetWare 4.11
The DSS maintains the database of SAP and RIP information. NetWare/IP
servers use User Datagram Protocol (UDP) packets to exchange SAP and RIP
information with the DSS server. NetWare clients use the DSS server to obtain
the IP address of their preferred NetWare server or Directory tree so they can log
in to the network. On an IPX network, NetWare servers broadcast SAP and RIP
information every 60 seconds. On a NetWare/IP network, NetWare/IP servers
download their SAP and RIP information from the DSS database. At configurable
intervals, the NetWare/IP server synchronizes its SAP and RIP information with
the DSS server, and DSS servers synchronize SAP and RIP information with
other DSS servers. The DSS server maintains an accurate list of IPX networks
and services. NetWare/IP servers inform the DSS server when they initialize,
when they are going down, and periodically that their services and networks are
still available (the periodic interval is configurable). If the NetWare/IP server fails
to send the periodic alive message, the DSS server removes that server's
services from its database. For information on setting up a DSS server, see the
NetWare/IP Administration Guide for NetWare 4.11.
3.4.3.4 LAN numbers
To configure multiple connections to Novell Network Services, each connection
on that server must be assigned a number 1–n. Use this number to group the
parameters by connection. They have no correlation to LAN numbers assigned
on other servers. For example, to configure three networks on a server, use the
numbers 1, 2, and 3 as shown in Table 30.
Table 30. How to configure three networks
Network 1
nwcm -s lan_1_adapter="/dev/inrdo0"
nwcm -s lan_1_network=0x50
nwcm -s lan_1_frame_type="ETHERNET_802.2"
Network 2
nwcm -s lan_2_adapter="/dev/inrdo2"
nwcm -s lan_2_network=0x51
nwcm -s lan_2_frame_type="ETHERNET_802.2"
Novell Network Services
139
Network 3
nwcm -s lan_3_adapter="/dev/inrdo3"
nwcm -s lan_3_network=0x52
nwcm -s lan_3_frame_type="ETHERNET_802.2"
3.4.3.5 Preparing the FENET OSA2 for IPX protocol
Use IP or IPX as a protocol. In this section we show you how to set up an IPX
configuration. Before Novell Network Services can be configured to use a FENET
OSA-2, the FENET OSA-2 must be configured to the system. This is described in
detail for the System/390 Open Systems Adapter Feature, but a summary is
provided here. For more information on how to set up an IP configuration see
Novell Network Services for OS/390 Installation, GA22-7312.
For each FENET OSA-2 connection desired, the following system configuration is
required:
• IOCDS configuration
In the system hardware I/O configuration (IOCDS), associate one even/odd,
read/write pair of device numbers with the channel path of each FENET
OSA-2 that is to be run used with IPX. Note that this device pair cannot be
shared with other communication methods such as TCP/IP or SNA. However,
it can share the FENET OSA-2 itself.
We used the IOCDS definition shown in Figure 63 for our installation:
RESOURCE PARTITION=((A1,1),(A10,A),(A11,B),(A12,C),(A2,2),(A3,
3),(A4,4),(A5,5),(A6,6),(A7,7),(A8,8),(A9,9),(C1,D),(C2,
E),(C3,F))
CHPID PATH=(30),SHARED,
PARTITION=((A1,A10,A11,A12,A2,A3,A4,A5,A6,A7,A8,A9),(A1,
A10,A11,A12,A2,A3,A4,A5,A6,A7,A8,A9)),TYPE=OSA
CNTLUNIT CUNUMBR=2320,PATH=(30),UNIT=OSA
IODEVICE ADDRESS=(2320,015),UNITADD=00,CUNUMBR=(2320),
STADET=Y,UNIT=OSA
IODEVICE ADDRESS=232F,UNITADD=FE,CUNUMBR=(2320),STADET=Y, UNIT=OSAD
Figure 63. IOCDS definitions
• MPC OAT configuration
Using OSA/SF, create an MPC OAT entry for each FENET OSA-2 for the IPX
protocol. The OSA name must be in the form INRT11nn where nn is a
two-digit decimal number in the range 00–99. The name must match the entry
in the /etc/netware/devices file or one of the defaults if the file is not present
(see the defaults in 3.4.3, “Network configuration” on page 135). Each OSA
name must be unique on the system. The unit address of the even device
number configured in the IOCDS is also defined here.
• VTAM or CS for OS/390 SNA configuration
Define the OSA name and a pair of OSA device numbers in the TRLE
statement in the TRLE macro of VTAM or CS for OS/390 SNA, whichever is
running on the OS/390 system image. The OSA name must match the one
defined in the MPC OAT configuration. The device numbers must match the
ones defined in the IOCDS as well as correspond to the unit address defined
in the MPC OAT configuration.
140
Debugging UNIX Applications on OS/390
Figure 64 on page 141 shows the VTAM TRLE definitions used for our
installation in the SYS1.VTAMLST(NETW2320):
IPX2320 VBUILD TYPE=TRL
INRT1100 TRLE LNCTL=MPC,READ=(2320),WRITE=(2321),MAXBFRU=9
Figure 64. VTAM definitions
Figure 65 shows how the OSA2 card should be configured on OS/390:
Figure 65. OSA2 configuration overview
Note
1. The even unit address in the MPC OAT (for example, 04) must correspond to
the device address in the IOCDS (for example, 304).
2. The OSA name (for example, INRT1100) must match in the MPC OAT, TRLE
statement, and the /etc/netware/devices file (or the default values).
3. The character special file name (for example, /dev/inrdo0) must correspond
to the desired entry in the /etc/netware/devices file or the default values (for
example, inrdo0).
Novell Network Services
141
Figure 65 through Figure 70 on page 143 describe how to define the OSA-2 card
in more detail.
Figure 66. OSA-2 CHPID selection
Figure 67. OSA CHPID selection continued
Figure 68. OSA configuration list
142
Debugging UNIX Applications on OS/390
Figure 69. OSA-2 configuration
Figure 70. High Performance Data Transfer Multipath Channel settings
For more information about the network configuration see Novell Network
Services for OS/390 Installation, GA22-7312, and OS/390 Open Systems
Adapter Support Facility User's Guide, SC28-1855.
3.4.3.6 Network discovery
In this section we describe how to analyze the network environment.
Novell Network Services includes a utility, nwdiscover, which queries the network
to set up the IPX router configuration. Use this utility to verify that the FENET
OSA-2 configuration is correct. This is highly recommended for initial setup and
any time the FENET OSA-2 configuration is changed.
The nwdiscover parameters allow several uses:
1. Query the network for network numbers
2. Query the network and update the configuration with the discovered frame
types and external network numbers. The parameters that get set for each
network are:
• lan_x_adapter
• lan_x_frame_type
• lan_x_network
Novell Network Services
143
If this is the first NetWare Server in the network, it is recommended that the
external network numbers be assigned rather than randomly generated by
nwdiscover. The network numbers are important for troubleshooting network
problems and having a numbering convention is very helpful. In any case, the
internal network number must be manually configured.
Figure 71, Figure 72 on page 144, Table 31 on page 144, and Table 32 on page
144 show some information about the nwdiscover command.
nwdiscover [-a] [-f frame_type] [-r retry_count]
[-t timeout] [-d pathname] [-v] [-u] [-e frame_type]
Figure 71. Syntax of the nwdiscover command
You must be the root user to use nwdiscover.
The nwdiscover command discovers the network number, frame type, and
device of connected IPX networks. It does this by generating two Service
Access Protocol (SAP) Get Nearest Server (GNS) requests to the network, and
evaluating the results.
The first message is a service request message for NetWare servers; the
second is a service request message for UNIX servers.
If there is no response to the server request messages, an IPX Router
Information Protocol (RIP) message is sent requesting information on all
networks.
If a response is received to any of the messages, nwdiscover extracts the
network and frame type from the reply. If the -u option is set, the NetWare
configuration file information is updated.
The network information is sent to stdout.
If an IPX network is already configured via nwcm, it is discovered first by
nwdiscover.
If no reply is received to any of the request messages, nwdiscover uses the
information already configured, or if nothing is configured, it invents a network
number, configures the specified frame type, and configures the specified
device.
Figure 72. Using nwdiscover
Table 31. nwdiscover parameter
Parameter
Usage
-a
Check all frame types and device types, even if there is a
response from a NetWare server.
If set with the -u option, any existing configuration is erased, all
networks discovered are configured, and the ipx_auto_discovery
flag is turned off.
-u
Update the NetWare configuration file (/etc/netware/nwconfig).
If this option is on, the configuration file is updated with the frame
type, device name, and network address.
-v
View in verbose mode. Information is sent to stdout.
Table 32. nwdiscover examples
144
Usage
Command
Check all frame types, update
configuration files, and turn off boot-up
auto-discovery.
nwdiscover -au
Debugging UNIX Applications on OS/390
Usage
Command
Try discovery using the specified
device
/dev/NE2000_0.
nwdiscover -d /dev/NE2000_0
Configure an ETHERNET II network,
even if it is not used on your network.
nwdiscover -f ETHERNET_II -u
Determine what networks are
connected to your platform.
nwdiscover -av
Novell Network Services includes another utility, drouter, which queries and
displays the network help to set up the NetWare/IP configuration. This is
recommended for initial setup and any time the NetWare/IP configuration is
changed. All the available servers should be displayed to connect them.
Figure 73 shows an example of the drouter command:
HOERNER @ SC64:/etc/netware>drouter
NETWORK HOPS TIME NODE
-------- ---- ---- -----------3784C340 0000 0001 000000000001
END OF TABLE
2 known networks
NETWORK HOPS TIME NODE
-------- ---- ---- -----------DEADBABE 0000 0001 00203512DF78
Figure 73. drouter information
For more information see Novell Network Services for OS/390 Utilities Reference,
SA22-7318.
3.4.3.7 System parameters
The current recommendations for BPXPRMxx are documented in OS/390 UNIX
System Services Planning, SC28-1890. For running the NetWare Server, if you
run with all the defaults in the NWCONFIG file, you cannot use all the defaults for
BPXPRMxx (see APAR OW39749). If you want to change the max_connections
in the NWCONFIG file, you need to customize the following:
• NWCONFIG
shm_sizeNetWare Server requires 9 MB plus 1 MB for every 25 connections.
For example, if you want max_connections to be 900, then specify 9M +
(900/25) = 45 MB
Issue the following command from the /usr/lpp/netware/bin directory set using
the PATH command in “Environment variables” on page 134.
nwcm -v shm_size(to view the actual setting)
nwcm -s shm_size=45000000(to set the shm_size to 45 MB)
Possible problem:
/bin:.:/usr/lpp/netware/bin
HOERNER @ SC64:/>nwcm -v
-¦--: NWCM-2.1-255.
or
Cannot bind message domain. NWCM error = 255. Exiting.
Novell Network Services
145
Solution:
Ensure access to the message catalog. This is done by issuing the correct
export command for the NLSPATH. See, 3.4.2, “Environment variables” on
page 134.
Note
The nwcm command results in an entry in the /etc/netware/nwconfig file. Use
the obrowse command to check your final configuration in the nwconfig file.
• BPXPRMxx
Use the following sizes for 900 max_connections with defaults and minimum
values which can be set up based on the guidelines defined in the preceding
mentioned document. The key is to increase the size of the following
parameters in the same proportion as the shm_size increase.
ipcshmspages= 12500 - default is 262144 - max is 524032
Note
The IPCSHMSPAGES value needs to be changed. On OS/390 the R7 default
value is 262144. The IPCSHMSPAGES value for 900 max_connections should
be IPCSHMSPAGES(2621440). See APAR OW39749.
ipcshmmpages= 12800 - default is 256 - max is 524287
Note
Increase the IPCSHMMPAGES in the BPXPRMxx parmlib member also if you
get the following message during the start of the server using NWSERVER:
BPXF024I (HOERNER) SERVER-4.10-67: The virtual process identifier 642
table was not correctly set up.
ipcshmnsegs= 500 - default is 10 - max is 1000
3.5 Installing NNS for OS/390
In this section we describe how to install Novell Network Services after the
SMP/E installation has been completed. Refer to Novell Network Services
Program Directory for the SMP/E installation instructions and the system
requirements. To perform the following steps, the installer must log in to OS/390
UNIX System Services as root or another user ID with equivalent (superuser)
authority. The NDS configuration process (dsinstall) will create a user object of
ADMIN. The ADMIN user has supervisory authority within Novell Network
Services and can access all of the objects in the directory tree.
Note
For our installation we use IPX as a connection type only.
146
Debugging UNIX Applications on OS/390
3.5.1 Before you get started
3.5.1.1 Check the environment
Before you start the server installation, verify the following:
• Ensure that you have activated your VNET.
• Ensure that your OSA card is online.
3.5.1.2 Worksheet
The following must be configured when installing Novell Network Services. Use
the worksheet provided in Appendix 3.7, “Getting NetWare client software” on
page 180 as an organizational aid when installing Novell Network Services. See
the values in brackets () used for our installation.
For more information see, Figure 65 on page 141 and 3.5.2, “Server installation”
on page 148.
server_name
(NWREDPROJECT)
Provide the name under which all NetWare Services (NCP, AFP, and NVT) are
advertised. It must be unique from other NetWare servers and cannot contain
spaces or punctuation marks. Alphabetic characters are converted to uppercase.
Length: 2 to 47 characters
ipx_internal_network
( 0x3784C340)
Provide a network address for access from multiple LANs. This parameter
specifies the address of the internal network (LAN 0). This address must be
unique on the IPX internetwork. Enter values in hexadecimal (0xNNNNN),
decimal (NNNNN), or octal (0NNNN).
Values are displayed in hexadecimal format. Supported values: 0x1 to 0xfffffffe
ts_type
(Single)
Determine how the server synchronizes time and is initially set during
NDS installation. By default, time synchronization occurs within an NDS
tree. Supported types: SINGLE (typically used for smaller LANs and serve as
the sole time source); REFERENCE (typically used for larger LANs and serve as
a time source for Primary and Secondary servers); PRIMARY (typically used
with a Reference server or other Primary servers); SECONDARY (obtain time
from a Single Reference, Reference, or Primary time server).
lan_x_adapter
(/dev/inrdo0)
Specify the name of the device driver for the network board. Values are the path
and file name of a valid LAN driver. The x indicates by number which LAN you are
configuring. To activate NetWare/IP, set the value to /dev/nwip. The
lan_x_network parameter must also be configured.
Maximum length: 127
Novell Network Services
147
lan_x_network
(0x1)
Specify the IPX external network number for the cabling system to which the
network board is attached. All IPX drivers linked to this cabling system must use
the same network number for the cabling system and this frame type. (The x
indicates by number which network you are configuring.) Each frame type
requires its own unique IPX external network number. Supported values: Any
hexadecimal integer from 0x1 to 0xFFFFFFFE that does not conflict with other
assigned IPX network numbers (internal and external).
Recommendation:
Use the nwdiscover command to check which external network number already
exists.
lan_x_frame_type
(ETHERNET_802.2)
Specify the frame type. (The x indicates by number which LAN you are
configuring.) Supported values: ETHERNET_802.2, ETHERNET_II,
ETHERNET_SNAP. The default is ETHERNET_802.2.
The worksheet section titled Optional Configuration (see 3.9, “Installation
Worksheets” on page 183) can be filled in with other optional parameters. These
are described in the Novell Network Services for OS/390 Utility Reference and
Novell Network Services for OS/390 Supervising the Network.
Note
At this point, you can proceed to the next chapter, which describes the steps
involved in bringing up the server for the first time. If you have not read the
publication Novell Network Services for OS/390 Introduction to NDS, it is highly
recommended that you do so at this time.
3.5.2 Server installation
The following is a step-by-step procedure for server installation. See the values in
brackets () used for our configuration.
1. Log in to OS/390 UNIX System Services.
• Log in to TSO/E and enter the OMVS command to invoke the shell, or, if
you prefer the Asynchronous Terminal Interface, rlogin or telnet in to the
shell. Remember you must log in with superuser authority, so use the root
or an equivalent ID.
• Check the environment variables for the logged on ID. Both the PATH and
NLSPATH variables must be set. To display the current values of these
variables, enter at the shell prompt:
-$ echo $PATH
-$ echo $NLSPATH
• To add these values, enter at the shell prompt:
-$ export PATH=$PATH":/usr/lpp/netware/bin"
-$ export NLSPATH=$NLSPATH":/usr/lpp/netware/nls/msg/%L/%N"
148
Debugging UNIX Applications on OS/390
If you do not want to export these variables each time you log in, add the
export statements to the profile or login script of the ID you are using for
the installation.
Note
The common profile is available in the directory /etc and the user profile is
available in the directory, which is defined in the OMVS segment of the user.
Recommendation: Add both export commands to the existing PATH and
NLSPATH statements in the common profile using the oedit command that all
users are able to issue the NetWare nwcm commands.
2. Set up the SYS volume.
Be certain that the HFS for the SYS volume already exists, and that it is
mounted. If you are not sure, see 3.4.1.1, “NNS volumes” on page 132 and
review the sample procedure. The SYS volume is created in the
/etc/netware/vol directory after you enter the following command:
$ nwvm -R SYS
Additionally, the client utilities can be copied to the SYS volume. This is
necessary if it is desired to access the administration executables and other
client functions (such as ncopy, nwadmin etc.) when the client logs on to this
server. These utilities are installed into /usr/lpp/netware/SYS. The following
command will copy them all into the SYS volume:
cp -R /usr/lpp/netware/SYS/* /etc/netware/vol/SYS
3. If you are using NetWare/IP, set the nwip configuration parameters:
$ nwcm -s lan_<x>_adapter= "/dev/nwip"
Replace the <x> with the number of LAN you are configuring. If you are using
both IPX and IP, the number must be different from the number used for the
IPX LAN.
To set the IPX number for the LAN, type:
$ nwcm -s lan_<x>_network=hexnum
(nwcm -s lan_<1>_network=50)
Replace the <x> with the number you used before. These two parameters
must be set up for the same LAN for NetWare/IP to initialize. Replace hexnum
with the same external network number as is configured for all other NetWare
servers in the same domain.
The following message is shown in case of a conflict:
NPSD-2.1-90: LAN configuration is invalid, network number for lan 0 and lan
1 are duplicates
4. Configure the LAN. There are two methods for configuring the LAN.
If the server is the first NetWare Server installed in the network, follow the
installation worksheet. Use the values you have recorded there for the board
device file name, frame type, and external network number that describe LAN
1.
• Set lan_1_adapter to the device file name. Values are the path and file name
of a valid LAN driver (maximum length: 127 characters).
$ nwcm -s lan_1_adapter="device_file_name"
(/dev/inrdo0)
Novell Network Services
149
For more information see Figure 65 on page 141. Ensure that the entry inrdo0
INRT1100 is available in the etc/netware/devices file.
• Specify the lan_1_frame_type as ETHERNET_802.2, ETHERNET_II,
or ETHERNET_SNAP.
$ nwcm -s lan_1_frame_type="frame_type"
(ETHERNET_802.2)
• Set the lan_1_network value to the external network number you have
defined. The value can be any hexadecimal integer from 0x1 to 0xFFFFFFFE
that does not conflict with other assigned IPX network numbers, internal or
external.
$nwcm -s lan_1_network="External_Network_Number"
(0x1)
If this is NOT the first NetWare server defined in the network, use Network
Discovery to query the network and to update the Novell Network Services
configuration automatically. At the shell prompt, enter:
$ nwdiscover -auv
The nwdiscover command discovers the network number, frame type, and
device of connected IPX networks. This command updates the
/usr/lpp/netware/nwconfig file.
For more information see 3.4.3.6, “Network discovery” on page 143.
5. Assign an internal network number.
Regardless of the method used (manual or automated) to configure external
network numbers, the address of the internal network can only be set from
nwcm. The internal network number must be unique in the network. For
example:
$ nwcm -s ipx_internal_network=0xfffffff2 (0X3784C340)
6. Name the server.
Assign a name to the server that is unique from other NetWare servers on the
network. Do not use spaces or special characters in the name. The name is
not case-sensitive (Novell Network Services folds it to uppercase). For
example:
$ nwcm -s server_name=NNS001
(NWREDPROJECT)
7. Establish the time synchronization type.
Specify the type of time synchronization carried out by this server. For
example:
$ nwcm -s ts_type="Single"
8. Start the network protocol stack.
If you are using NetWare/IP, TCP/IP, and INETD must be started on OS/390.
Please refer to documentation for OS/390 TCP/IP OpenEdition if you need
assistance with this step. Once TCP/IP is initialized, at the shell prompt issue:
$ nwipd
Then for either NetWare/IP or IPX protocol, issue:
$ startnps
INRM0001I IPX Open Successful. TRLE=INRT1100 Major=1100 Minor=00
150
Debugging UNIX Applications on OS/390
The following message is shown if the network protocol stack is already
started: NPSD-2.1-121: Open of /dev/ipx0 failed: EDC5114I Resource busy.
9. Start the license manager.
$ lsman
10. Start the server.
$ nwserver
Continue to the next step when the following messages are displayed:
BPXF024I (xxxxxx) Directory Services: Could not open local database,
error: -723
BPXF024I (xxxxxx) NetWare server N40NOV2 is waiting for Directory
Services to be installed. To finish bringing the server up, you must run
DSInstall.
Figure 74 on page 152 and Figure 75 on page 153 show the messages in the
syslog, if the server is successfully started using the nwserver command after
setting the shared memory within NetWare to 150 MB and setting the
IPCSHMMPAGES in the BPXPRMxx Parmlib member to 524200.
For more information about known problems starting the Novell server using
the nwserver command see 3.6.4, “Known problems” on page 170.
Novell Network Services
151
PXF024I (HOERNER) Initializing inform.
BPXF024I (HOERNER) NetWare Services 4.10b for OS/390
BPXF024I (HOERNER) Copyright 1997 Novell. All Rights Reserved.
BPXF024I (HOERNER) SERVER-4.10-3: The NetWare server is using 806
/etc/netware/nwconfig as the configuration file.
BPXF024I (HOERNER) Initializing LSManager.
BPXF024I (HOERNER) SERVER-4.10-8: The NetWare server is configured 808
for 990 client connections.
BPXF024I (HOERNER) SERVER-4.10-11: Shared memory size is 150000000 809
bytes.
BPXF024I (HOERNER) SERVER-4.10-608: License file processing complete.
810
Total number of licensed connections is 0.
BPXF024I (HOERNER) SERVER-4.10-13: No valid license files were found.
811
Authorizing 990 licensed connections.
BPXF024I (HOERNER) Getting shared memory of 150994944
BPXF024I (HOERNER) shmAttachPoint = 62300000
BPXF024I (HOERNER) Initializing server information lib.
BPXF024I (HOERNER) Initializing statistics.
BPXF024I (HOERNER) Initializing shared file descriptors.
BPXF024I (HOERNER) Initializing communications.
cmn_err NOTICE: NCPIPX: Hash table allocated at 0x9070060 32 entries
BPXF024I (HOERNER) The server address is 3784c340.
BPXF024I (HOERNER) Initializing file system.
BPXF024I (HOERNER) nwserver (NWS Daemon): Mounting NetWare volume 0
821 as SYS.
BPXF024I (HOERNER)
BPXF024I (HOERNER) Mounted volume 0: 827
Volume name: SYS
File System Type: Standard
Host mount point: /etc/netware/vol/SYS
Control path: /etc/netware/vol/control/SYS
Name spaces: UNIX DOS OS2
Status: READ and WRITE
BPXF024I (HOERNER) Maximum Inode File Size: 6291456 828
Minimum Directory Sync Interval: 10
Mandatory Directory Sync Interval: 900
Directory Purge Threshold: 32
BPXF024I (HOERNER) Initializing connection table.
BPXF024I (HOERNER) Initializing lock manager.
BPXF024I (HOERNER) Initializing events.
BPXF024I (HOERNER) Initializing time synchronization.
Figure 74. Successful nwserver command to start the server
152
Debugging UNIX Applications on OS/390
BPXF024I (HOERNER) Initializing NetWare Directory Services.
BPXF024I (HOERNER) Initializing Netware Directory Services.
BPXF024I (HOERNER) Initializing qms.
BPXF024I (HOERNER) Initializing accounting.
BPXF024I (HOERNER) Initializing spooler emulator.
BPXF024I (HOERNER) Initializing NetWare Trustee Database.
BPXF024I (HOERNER) Setup trustee database.
BPXF024I (HOERNER) SERVER-4.10-156: A new trustee file is being 840
created for volume 0.
BPXF024I (HOERNER) Setup NCPX information.
cmn_err CONTINUING: NEMUX Initialized
BPXF024I (HOERNER) Spawning Timer Daemon.
BPXF024I (HOERNER) Spawning NTSD.
BPXF024I (HOERNER) Spawning SAP.
BPXF024I (HOERNER) Spawning Logical Lock Daemon.
BPXF024I (HOERNER) Spawning File Lock Daemon.
BPXF024I (HOERNER) Spawning Physical Lock Daemon.
BPXF024I (HOERNER) Spawning Semaphore Daemon.
BPXF024I (HOERNER) Spawning Background Daemon.
BPXF024I (HOERNER) Spawning Janitor Daemon.
BPXF024I (HOERNER) Spawning Skulker Daemon.
BPXF024I (HOERNER) Initializing async events.
BPXF024I (HOERNER) sfdopen: EDC5129I No such file or directory. err
877
on /etc/netware/_netware/0.DSB, errno = 129, errnoJr = 05620062
BPXF024I (HOERNER) Directory Services: Could not open local 878
database, error: -723
BPXF024I (HOERNER) NetWare server NWREDPROJECT is waiting for 879
Directory Services to be installed. To finish bringing the server up,
you must run DSInstall.
Figure 75. Continuation of the syslog in Figure 74
11.Configure NDS using dsinstall.
$ dsinstall
Figure 76 shows the messages from the syslog issuing the dsinstall command
in the ISHELL:
BPXF024I (HOERNER) Bindery open requested by the SERVER
BPXF024I (HOERNER) Established communication with server NWREDPROJECT.
883 IBM
BPXF024I (HOERNER) Directory Services: Local database is open
BPXF024I (HOERNER) NWS NDS SAP Daemon: The server internal address is
885 3784c340.
BPXF024I(HOERNER) NWS NDS SAP Daemon:Theserver nodeaddressis886 000001.
BPXF024I (HOERNER) NWS NDS SAP Daemon: The server socket is 451.
Figure 76. Messages because of the dsinstall command
Novell Network Services
153
The following menu appears:
1) Install a new SANDS tree
2) Install a new SCALE tree
3) Add a new server into an existing SCALE tree
4) Upgrade this SANDS server/tree to SCALE
5) Remove Directory Services from this server
6) Upgrade mounted volumes into the Directory
7) Exit
Enter option number:
Figure 77. Directory service installation
If you are installing a new NDS tree, enter 2 to install a new SCALE tree and
proceed with step 13. If you are adding the server to an existing tree, enter 3
to add the server to an existing tree and proceed with step 16.
For more information see 3.2.2, “Features of NNS” on page 128.
12.The following prompt appears:
Enter the NDS Tree Name:
Type the name of the NDS tree name and press Enter. (NW_RED_IBM)
13.The following prompt appears:
Enter the Server Context (for example, OU=aa.OU=bb.O=cc): Type the
context in which you wish to place the server in NDS and press
Enter.
(O=IBM)
14.The following prompt appears:
Enter the ADMIN password:
(admin)
Reenter the ADMIN password: (admin)
Type the password for the ADMIN user ID. The password needs to be entered
again to verify that it was entered correctly.
NDS installation is now complete. Proceed to step 18.
15.The following prompt appears:
Enter the NDS Tree Name:
Type the name of the NDS tree name and press Enter.
16.The following prompt appears:
Enter the ADMIN user context (for example, OU=aa.OU=bb.O=cc):
Type the context of the ADMIN user ID and press Enter.
17.The following prompt appears:
Enter the ADMIN password:
Type the password for the ADMIN user ID and press Enter.
NDS installation is now complete.
18.Verify the state of the Novell Network Services server.
At the shell prompt, enter the nwserverstatus command. You should see
the following:
$ nwserverstatus
154
Debugging UNIX Applications on OS/390
NetWare server <the server name> is up.
HOERNER @ SC64:/>nwserverstatus
NetWare server NWREDPROJECT is up.
If you do not get this message, review the installation and make sure that all
steps completed successfully before contacting support.
3.5.3 Installing NetWare client software
Before you can continue setting up your network, you must install a single client.
From the client, you can run either NetWare Administrator (NWADMIN) or
NETADMIN to start creating objects on your network. You can install and
customize other clients later. See the Novell publications for details. Before you
can install client software, you must have access to installation diskettes for
NetWare client software. For instructions on how to obtain client diskettes, see
3.7, “Getting NetWare client software” on page 180.
Once you have client installation diskettes available, use the installation
procedures for your client operating system.
Note
At this point, your server is up and running. You can now go to the publication
Novell Network Services for OS/390 Supervising the Network, SA22-7317,
which describes how to administer and maintain your NetWare server.
3.5.3.1 Login from a NetWare client
In this section we describe the login from a Windows NT Novell client. Figure 78
on page 156 through Figure 82 on page 158 show the login process.
Choose the IntranetWare Login from the Network Neighbor icon using the right
mouse button, as shown in the following figure:
Novell Network Services
155
Figure 78. Right mouse click the Network Neighbor icon
Selecting the IntranetWare Login provides the following Novell NetWare Login
panel:
Figure 79. Novell NetWare login panel
156
Debugging UNIX Applications on OS/390
Type your user ID and password on the login panel, click OK and the panels
appear as shown in Figure 80.
Figure 80. Novell IntranetWare Client and Connection panels
Get a list of the available servers by clicking the server button from the
Connection panel:
Figure 81. Novell IntranetWare Client Server list
The expected server in the list verifies that the connection is possible.
Novell Network Services
157
Note
If no server is found in the list shown in Figure 82, the connection is not
possible.
Figure 82. Select the server from the list and click OK to log in
Figure 83. Login result
158
Debugging UNIX Applications on OS/390
Figure 84. Check the IntranetWare Connections using Network Neighbor
Figure 85 through Figure 88 on page 161 show the small Novell environment
used in our installation based on the definitions in 3.5.2, “Server installation” on
page 148.
Figure 85. Installation environment
Novell Network Services
159
Figure 86. Server identification showing the connection type IPX
Figure 87. Server error log
160
Debugging UNIX Applications on OS/390
Figure 88. Environment of the ADMIN user
Novell Network Services
161
3.6 Serviceability and debugging
In this section we provide information about how to service and debug the NNS
product.
3.6.1 Hints and tips
This section contains information on how to prevent and service an NNS problem.
• Ensure that all the available maintenance for FMID HNNWS110/COMPID
564701NWS is installed.
• Ensure that all the available UNIX Sytem Services maintenance is installed.
• If running OS/390 V2R7 or higher, ensure that all available DFSMS 1.5
maintenance is installed.
• Ensure that USS is successfully initialized shown by the message in the
syslog: BPXI004I OMVS INITIALIZATION COMPLETE
• Ensure that a message in the syslog for all the NNS-related HFS data sets
defined indicates that they are mounted (for example, BPXF013I FILE
SYSTEM OMVS.NWSYS.HFS 781 WAS SUCCESSFULLY MOUNTED).
• Ensure that the following message is shown in the syslog to indicate the
STREAMS support is given: BPXTN005I STREAMS module map INRKMAP loaded
If the following message is shown instead, BPXTN004I Module map load failed
for module INRKMAP, the STREAMS support is not given. Thus, the installation
of the STREAMS has to be checked. Any change requires an IPL.
• Ensure that the client is able to connect to Ethernet instead of token-ring
because the OSA-2 card on the OS/390 host does not support token-ring.
• Ensure that the Ethernet cable is 100 m as max length.
• To be sure that the correct TCP/IP stack is used issue the following command
from the ISHELL: export_bpxk_setibmopt_transport=tcpipoe. See the section
titled "Using Specific Transports under Common INET" in OS/390 UNIX
System Services Planning, SC28-1890.
3.6.1.1 Interpret a USS message
How to interpret a USS message (for example, BPXF002I FILE SYSTEM
HOERNER.HFS.TEST WAS 539 NOT MOUNTED. RETURN CODE = 00000081,
REASON CODE = 0594003D) see, Chapter 5, NFS section "5.2.6.2".
For more information on how to interpret a USS message see 6.2.2, “RACF
profiles” on page 258.
3.6.2 Tuning the NNS server
In this section we discuss how to limit or increase the UnixWare resources
available for NetWare Services. We also describe how to manage engines,
configure shared memory size, limit error log file size, and set packet burst limits.
3.6.2.1 Managing engines
NetWare Services servers refer to certain processes as NCP engines. NCP
engines are often considered the workhorses of NetWare Services servers and
are responsible for the initial processing of all client requests. NetWare Services
162
Debugging UNIX Applications on OS/390
requires a minimum of two engines running at all times. The more work you
require of your system, the more engines you will need to start.
Adding too many engines, however, may decrease performance. So generally
you need two engines per CPU. Running more engines requires more RAM. After
the NetWare server has booted, engines can be increased or decreased using
the nwengine utility. The number should be increased if the nxinfo utility reports
an excessive number of packets dropped because of server activity. To prevent
NetWare from using too many resources, you can limit the number of engines
that can be started after boot time. You can specify the number of NCP engines
to start by using the nwcm command line utility to set "Number of NCP Engines to
Start" and "Maximum Number of NCP Engines". To specify how many engines to
start automatically when the NetWare server process is started, see 3.6.2.4,
“Tuning your server” on page 163.
3.6.2.2 Configuring shared memory
Shared memory is divided into pools and is controlled by internal locking
mechanisms. Connections tables, data structures, trustee databases, NetWare
server information, and volume table information are all data that shared memory
uses. If you use Packet Burst to add more connections, volumes, trustee
assignments, and so on, you should increase the shared memory size.
See the description of shm_size in 3.6.2.4, “Tuning your server” on page 163 to
configure shared memory.
3.6.2.3 Adjusting the size of your error log
Any errors your server experiences are saved to the error log file,
SYS$LOG.ERR. You can change the size of this log depending upon how many
messages you want to store in this file. Make sure you monitor this file because if
messages come in and the file is full, messages are saved to a backup
(SYS$LOG.OLD) and a new file, SYS$LOG.ERR, is created. Only one
SYS$LOG.ERR.OLD is maintained. See the description of err_log_file_size in
3.6.2.4, “Tuning your server” on page 163 to change the size of your error log file.
3.6.2.4 Tuning your server
Use the following information to tune your system.
Parameters: Using the nwcm command line utility, set the following parameters
as appropriate. For further information on how to use the options within nwcm,
refer to "nwcm" in Novell Network Services for OS/390 Utilities Reference,
SA22-7318. For further discussion of the parameters, please see "General Server
Parameters" in Novell Network Services for OS/390 Utilities Reference,
SA22-7318.
The Basic category shows the most commonly changed NetWare
Services tunable variables.
• ncp_engines_to_start. This variable specifies the number of NCP engines you
want running as the default every time the NetWare server process starts. The
default is 2. This is the minimum number that allows your system to run
efficiently under normal conditions. We suggest that one NCP engine be set
for every four clients that are active and not exceed four engines per CPU.
Novell Network Services
163
• max_ncp_engines. This variable specifies the maximum engines that can run
simultaneously. The maximum default is 10. The maximum allowed is 25. This
variable should always be greater than the "Number of NCP Engines to Start"
variable.
• shm_size. This variable specifies shared-memory segment size. This should
be increased as the number of connections, trustee assignments, volumes,
open files, and record locks increases. The OS/390 system parameters may
need to change if you increase this parameter. See the Novell Network
Services for OS/390 Installation for more information.
• err_log_file_size. This is the maximum size, in bytes, of the error log file,
SYS$LOG.ERR. When the maximum size is reached, error log messages are
copied to a backup and a new error log file is created; "0" means logging is not
performed.
For more information see Novell Network Services for OS/390 Supervising the
Network, SA22-7317.
3.6.3 Debugging
In this section we give information about the tools that are available to analyze a
problem and common information to debug the NNS product.
3.6.3.1 LANalyzer—packet flows on the wire
LANalyzer usage
• Check server health
- Server still SAPing/RIPing
- Packets sent back to client
• Determine packet flow for typical transaction
- SAP/RIP
- Connection/login requests
- Other NetWare Core Protocol requests
• Debug transaction problems
- Error return codes to clients
+ Use to trace back to code issuing RC
- Packet fields sent to/from client/server
Figure 89 on page 165 through Figure 91 on page 166 show information about
the LANalyzer trace. In our environment we installed the LANalyzer software
package on a Windows 95 client. We were not able to install it on a Windows NT
client.
Figure 89 on page 165 displays the main LANalyzer screen. From here you select
Capture from the action bar to get the filter shown in Figure 90 on page 165. The
trace should be started using the Start button.
164
Debugging UNIX Applications on OS/390
Figure 89. LANalyzer dashboard
Figure 90 shows the following filter criteria:
• Between This Workstation and any other server (you can select a specific
one)
• ALL packets of the NetWare protocol
Figure 90. LANalyzer filter
Figure 91 on page 166 displays the result of a Begin LOGIN request after clicking
the Stop button from the main screen shown in Figure 89 on page 165. If the IVN
is selected from the NDS screen, the packet (in hex), which goes through the
wire, is shown as highlighted on the lower screen. This packet should be found in
the GTF Trace on the host to confirm that the request was successful.
Novell Network Services
165
Figure 91. LANalyzer viewing
3.6.3.2 CTRACE—trace information for kernel level
In this section we describe the use of the CTRACE in more detail.
For OMVS, the STREAMS FILESYSTYPE statement must have character special
enabled (-t (trace_opt)).
• See 3.4.1.2, “STREAMS considerations” on page 132.
Parmlib member CTIBPXxx has OS/390 UNIX tracing options pointed to from
BPXPRMxx using CTRACE(CTIBPXxx)—default CTIBPX00.
In CTIBPXxx, indicate TRACEOPTS ON, and OPTIONS('CHAR') or
OPTIONS('ALL').
Figure 92 shows an example of how to force an SVC dump, which incorporates
the CTRACE.
DUMP COMM=(PROCESS CTRACE)
*041 IEE094D SPECIFY OPERAND(S) FOR DUMP COMMAND
R 41,JOBNAME=(OMVS,INR*),END
*IEA911E COMPLETE DUMP ON SYS1.DUMP99 411
Figure 92. Capture an SVC dump
166
Debugging UNIX Applications on OS/390
Use the STREAMS configuration utility configstrm to dynamically alter trace
options (requires USS APAR OW34480):
/usr/sbin/configstrm
Syntax: configstrm [-bimv] [-h high_mem | ?] [-l loadmod]... [-t trace_opt | ?]...
Description: This utility will set and query the STREAMS physical file system
configuration.
Options:
t
trace_opt
Set and query trace options.
• Parameters to set in BPXPRMxx parmlib member.
format: FILESYSTYPE TYPE(STREAMS) ENTRYPOINT(BPXTNSTR)
PARM(-t options)
• off - disable
• all - enable all trace types
• code|nocode - internally coded traces
• proc|noproc - procedure entry/exit traces
• normal|nonormal - code and proc traces
• diag|nodiag - internal debugging traces
• data|nodata - msg block data content traces
• nw|nonw - NetWare device traces
PROC—shows details about STREAMS flows, OSA device flows, NetWare UDP
flows
NW—all NetWare traces for kernel (came with base product)
DATA—packet contents at various points in kernel (mostly at puts)
Recommendation: Run with normal as the default.
For more information see 3.4.1.2, “STREAMS considerations” on page 132.
For more information on how to capture the CTRACE see Chapter 6, “Network
File System (NFS)” on page 239, "6.2.5, “Debugging” on page 259, OS/390 UNIX
System Services Planning, SC28-1890, and:
http://www.s390.ibm.com/oe/samples/oectrace.html
Novell Network Services
167
3.6.3.3 GTF traces—trace information for user level
In this section we inform you on settting up and capturing a GTF trace.
• Create a data set to hold traces (vb,lrecl 27994, blk 27998, 300/20 cyls)
• Parmlib member (for example, GTFCL1) to put GTF parms in:
TRACE=USRP
USR=(F70,F71,F72,F73)
END
Figure 93 decribes the GTF parms:
F70
F70 (GTRACE_STREAMS
(GTRACE_STREAMS in
in gtrace.h)
gtrace.h)
getmsg,putmsg,ioctl
getmsg,putmsg,ioctl (GTRACE
(GTRACE macro)
macro)
Streams
Streams interface
interface data
data trace
trace
F71
F71 (GTRACE_ENTR
(GTRACE_ENTR in
in gtrace.h)
gtrace.h)
entry/exit
entry/exit macros
macros (INRENTn
(INRENTn Macro)
Macro)
C++
C++ Entry
Entry and
and parameter
parameter entry
entry tracing
tracing
F72
F72 (GTRACE_EXIT
(GTRACE_EXIT in
in gtrace.h)
gtrace.h)
exit
exit macros
macros (INREXIT
(INREXIT macro)
macro)
C++
C++ exit
exit tracing
tracing
F73
F73 (GTRACE_DSTRACE
(GTRACE_DSTRACE in
in gtrace.h)
gtrace.h)
DStrace
DStrace (GTRACE
(GTRACE macro)
macro)
DStrace
DStrace facility
facility
Event
Event scheduling
scheduling tracing
tracing
Figure 93. GTF parms
• Proclib member to start GTF:
//GTFCL PROC MEMBER=GTFCL1
//IEFPROC EXEC PGM=AHLGTF,PARM='MODE=EXT,DEBUG=NO,
TIME=YES,BLOK=20M,TIME=1440,REGION=2880K'
//IEFRDER DD DSNAME=dataset,DISP=SHR,UNIT=3390
//SYSLIB DD DSNAME=SYS1.PARMLIB(&MEMBER),DISP=SHR
• Before starting the server, enter USS shell command:
export _CEE_RUNOPTS="test"
• Start GTF trace procedure: S GTFCL.GTF and respond ’U’ to prompt for
additional parms
• Start up server
• Perform tests
• Stop traces: ’P GTF’
168
Debugging UNIX Applications on OS/390
3.6.3.4 CEEDUMP and SVCDUMP
If NetWare code generates a dump (using nwabort), both CEEDUMP and
SVCDUMP are created. The CEEDUMP is useful for initial investigation—it
contains information around registers and trace back information. The SVCDUMP
will always have return code EC6 with reason code 7FF. System hang conditions
or unusual errors may require forcing an SVCDUMP.
• Forcing an SVCDUMP
- Determine address spaces to capture
+ nwserver, engines, daemons, and so forth based on problem
- Identify address space IDs
+ MVS command: D OMVS,A=ALL
+ Find asid for corresponding name
- DUMP COMM=("dump title - up to 100 chars")
R xx,ASID=(up to 15 asids), DSPNAME=(ka.SYSZBPX1,ka.SYSZBPX2),
SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,GRSQ) (ka=omvs asid)
Note
For more information how to capture a USS dump see, 3.6.3.2,
“CTRACE—trace information for kernel level” on page 166, OS/390 UNIX
System Services Planning, SC28-1890-06 , and
http://www.s390.ibm.com/oe/samples/oectrace.html
• For hangs add LSQA and NUC to SDATA
3.6.3.5 Common address space names
In this section we give information about the related address space names. The
complete list is in /u/netware/nwu_top/os390inc/abbrev.h.
• inrsapdn - SAP daemon (n is numeric value)
• inrtime - timer daemon
• inrjani - janitor daemon
• inrskul - skulker daemon
• inrengn - NCP engine
• inrservn - nwserver
• inrflck - file lock daemon
• inrslck - semaphore lock daemon
• inrphlk - physical lock daemon
• inrntsd - timer synchronization daemon
• inrback - background daemon
Novell Network Services
169
3.6.3.6 Server recovery—hangs
Symptom: nwshut results in error message
• Cancel/kill processes
- In omvs, issue: ps -A (note process IDs)
+ 33554457 ?
0:00 /usr/lpp/netware/bin/nwserver
- Then kill processid
+ Check the processid using ps -A
+ If this does not work, try kill -9 processid
• As a last resort, cancel/force address space(s)
- Issue: D OMVS,A=ALL (or SDSF option DA)
- CANCEL jobname
- If this fails, issue FORCE jobname ARM
+ If that fails, issue FORCE jobname
Recommendations:
•
•
•
•
Use the command nwshut only to down the OS/390 Novell server.
Do not interrupt the shutdown process as mentioned by nwshut.
Do not issue stopnps
Check the status of the server with nwserverserver and use nwserver to bring
the server up again.
• Make sure all of your users are logged out of the server if you shut it down.
3.6.4 Known problems
This section has information about known problems from the installation of the
NNS for OS/390 product.
• Message CEE3204S System detected a protection exception during startnps
Additional symptoms:
- From the CEE dump:
Condition Information for /u/netware4/nwu_top/os390dir/rtl/time.c
CIB Address: 000E4630
Current Condition:
CEE3204S The system detected a protection exception.
Location:
Program Unit: /u/netware4/nwu_top/os390dir/rtl/time.c
- RC 79 Reason 11B3005A
- AbendU4039
With APAR OW38371 installed a message will come up inform to define a
STREAMS Filesystype.
Solution: Define the correct STREAMS Filesystype in the BPXPRMxx parmlib
member.
170
Debugging UNIX Applications on OS/390
• Message BPXF024I shmget returned 79, errno2=07200316
Additional symptoms:
- SERVER-4.10-14: The NetWare server is unable to create a shared
memory segment.
- Server initialization stops
Solution: Increase the IPCSHMSPAGES in the BPXPRMxx parmlib member.
See Apar OW39726 and 3.4.3.7, “System parameters” on page 145.
• Message BPXF024I SERVER-4.10-31:The server is unable to access host user
or group "nwgroup".
Additional symptoms:
- Message BPXF024I SERVER-4.10-24: The NetWare file system could not be
initialized.
The problem was the groups were defined as users.
Solution: Ensure the user IDs and groups are defined correctly. Use the RACF
display and check that the OMVS portion is also defined. See 3.4.1.3, “User
IDs required” on page 134.
• Message IEW2648E during SMP/E installation.
Problem: The Base code and the PTFs are installed together (the supporting
documentation may state that this is possible).
This is a known problem with service and will be fixed in a future PTF.
Circumvention: Install the Base code first and then the PTFs.
See Apar OW38371 for more information.
• Message BPXF024I SERVER-4.10-63 the NetWare server is unable to complete
initialization and is shutting down.
Problem: After a hang occurs the recovery of the server was done; see
3.6.3.6, “Server recovery—hangs” on page 170. But the server was neither
unable to come down nor initialize correctly to come up.
Figure 94 show the related messages in the syslog.
BPXF024I (HOERNER) Getting shared memory of 150994944 135
BPXF024I (HOERNER) shmget returned 75, errno2=07200304 136
BPXF024I (HOERNER) SERVER-4.10-14: The NetWare server is unable to 137
create a shared memory segment.
BPXF024I (HOERNER) SERVER-4.10-15: Shared memory cannot be 138
initialized.
BPXF024I (HOERNER) SERVER-4.10-63: The NetWare server is unable to 139
complete initialization and is shutting down.
Figure 94. Messages in the syslog
Solution: Ensure using SDSF option DA that all INR* job names are canceled.
• Message Problem with max_connection and shm_size
In this section we describe a problem during the installation of NNS regarding
the shared memory and the connections.
Figure 95 shows an example of where the nwserver command fails:
Novell Network Services
171
BPXF024I (HOERNER) Initializing inform.
BPXF024I (HOERNER) NetWare Services 4.10b for OS/390
BPXF024I (HOERNER) Copyright 1997 Novell. All Rights Reserved.
BPXF024I (HOERNER) SERVER-4.10-3: The NetWare server is using 592
/etc/netware/nwconfig as the configuration file.
BPXF024I (HOERNER) Initializing LSManager.
BPXF024I (HOERNER) SERVER-4.10-8: The NetWare server is configured 594
for 50 client connections.
BPXF024I (HOERNER) SERVER-4.10-11: Shared memory size is 45000000 595
bytes.
BPXF024I (HOERNER) SERVER-4.10-608: License file processing complete.
596
Total number of licensed connections is 0.
BPXF024I (HOERNER) SERVER-4.10-13: No valid license files were found.
597
Authorizing 990 licensed connections.
BPXF024I (HOERNER) SERVER-4.10-120: The max_connections parameter is
598
set too low. You have 990 authorized licensed connections but only
50 max connections. The max_connections parameter should be set to
the number of licensed connections PLUS the number of expected
server to server connections needed to support NetWare Directory
Services in your directory tree.
BPXF024I (HOERNER) Getting shared memory of 45088768
BPXF024I (HOERNER) shmAttachPoint = 62300000
BPXF024I (HOERNER) Initializing server information lib.
BPXF024I (HOERNER) SERVER-4.10-67: The virtual process identifier 602
table was not correctly setup.
BPXF024I (HOERNER) SERVER-4.10-63: The NetWare server is unable to 603
complete initialization and is shutting down.
BPXF024I (HOERNER) Services are down till TimeSync
BPXF024I (HOERNER) Returning from the IOCTL NMXP_SHUTDOWN
BPXF024I (HOERNER) Services are down till LockManager
Figure 95. NWSERVER example
The example in Figure 95 shows two problems based on the
max_connections:
- The shared memory of 45 MB is too low using the default of 990
- The max_connection set to 598 is too low while the default of 990 is used
Use the nwcm command max_connections=990 to increase the possible
connections.
See the formula in 3.4.2, “Environment variables” on page 134.
Note
The max_connections value is set to 990 as a default. Thus, the check for the
related values is based on 990. See 3.4.3.7, “System parameters” on page
145.
Figure 96 still shows a problem if we increased the shared memory to 85 MB
and the max connections to 990 within NetWare.
172
Debugging UNIX Applications on OS/390
BPXF024I (HOERNER) SERVER-4.10-13: No valid license files were found.
638
Authorizing 990 licensed connections.
BPXF024I (HOERNER) Getting shared memory of 85983232
BPXF024I (HOERNER) shmAttachPoint = 62300000
BPXF024I (HOERNER) Initializing server information lib.
BPXF024I (HOERNER) SERVER-4.10-67: The virtual process identifier 642
table was not correctly setup.
BPXF024I (HOERNER) SERVER-4.10-63: The NetWare server is unable to 643
complete initialization and is shutting down.
BPXF024I (HOERNER) Services are down till TimeSync
BPXF024I (HOERNER) Returning from the IOCTL NMXP_SHUTDOWN
BPXF024I (HOERNER) Services are down till LockManager
BPXF024I (HOERNER) Services are down till File System
BPXF024I (HOERNER)
BPXF024I (HOERNER) CleanUp successful
Figure 96. Example of the nwserver command
Figure 97 still shows a problem if we increased the shared memory to 150 MB
and increased the IPCSHMMPAGES to 524288 (see 3.4.3.7, “System
parameters” on page 145).
::::::::::::::::::::::::::::::::::::::::::::::
BPXF024I (HOERNER) SERVER-4.10-11: Shared memory size is 150000000 763
bytes.
BPXF024I (HOERNER) SERVER-4.10-608: License file processing complete.
764
Total number of licensed connections is 0.
BPXF024I (HOERNER) SERVER-4.10-13: No valid license files were found.
765
Authorizing 990 licensed connections.
BPXF024I (HOERNER) Getting shared memory of 150994944
BPXF024I (HOERNER) shmget returned 79, errno2=07200316
BPXF024I (HOERNER) SERVER-4.10-14: The NetWare server is unable to 768
create a shared memory segment.
Figure 97. Example issue of a nwserver command
SET OMVS=GU
IEF196I IEF237I 0CC1 ALLOCATED TO IEFPARM
IEF196I IEF237I 25E6 ALLOCATED TO SYS00183
IEF196I IEF237I 25E6 ALLOCATED TO SYS00184
IEF196I IEF237I 25E6 ALLOCATED TO SYS00185
IEE252I MEMBER BPXPRMGU FOUND IN SYS1.PARMLIB
BPXI006I ERROR IN PARMLIB MEMBER BPXPRMGU ON LINE 727, 778
POSITION 25. INPUT PARAMETER VALUE IS OUT OF
THE ALLOWED RANGE OF 1 TO 524287.
A SYSTEM VALUE OF 35600 IS USED.
DETECTING MODULE IS BPXIPMZ1. INPUT LINE:
IPCSHMMPAGES (524288)
BPXO031I ERRORS IN PARMLIB MEMBER=BPXPRMGU 779
REFER TO THE HARD COPY LOG. SET OMVS COMMAND FAILED.
IEF196I IEF285I SYS1.SYSPROG.PARMLIB
Figure 98. Result of the failing SET OMVS command
Novell Network Services
173
Solution: Decrease the IPCSHMMPAGES in the BPXPRMxx Parmlib member
to 524200 for example. See 3.4.3.7, “System parameters” on page 145. For
more information see APAR OW39726.
• Problems during login from a client
Figure 99 shows a problem during login from a Novell client.
Figure 99. Unsuccessful LOGIN Problem
Reason: The OSA-2 card is plugged into a FENET ring and the PC with the
Novell client used a token ring card and the OSA card does NOT support
token ring connection.
Solution: Use a FENET card in the PC to connect to the host instead of the TR
card.
This symptoms are shown also if the FENET cable is longer than 100 m as a
max.
Solution: put a bridge, a router or a repeater in between to return/transfer the
signal or use a shorter cable between the host and the client PC.
3.6.5 Helpful commands
In this section we describe some helpful commands to analyze the environment
by displaying the status of the parts involved.
3.6.5.1 OS/390 console commands
D OMVS, A=ALL
Display all the USS-related information.
174
Debugging UNIX Applications on OS/390
Figure 100 shows a example of a /D OMVS, A=ALL command.
D OMVS,A=ALL
BPXO040I 12.36.57 DISPLAY OMVS 474
OMVS
000F ACTIVE
OMVS=(GU)
USER
JOBNAME ASID
PID
PPID STATE START
CT_SECS
OMVSKERN BPXOINIT 00FB
1
0 MR
15.55.04
1.282
LATCHWAITPID=
0 CMD=BPXPINPR
SERVER=Init Process
AF=
1 MF=00000 TYPE=FILE
TCPIPMVS TCPIPMVS 0039 16777218
1 MR
15.56.42
94.787
LATCHWAITPID=
0 CMD=EZBTCPIP
TCPIPMVS TCPIPMVS 0039 16777219
1 1R
15.56.46
94.787
LATCHWAITPID=
0 CMD=EZBTMCTL
HOERNER INRTIME 0047 301989917 721420314 1F
11.44.46
4.181
LATCHWAITPID=
0 CMD=timerd (NWS Timer Daemon) ......... ./..
SERVER=NETWARE
AF=
0 MF=00000 TYPE=SFDS
HOERNER INRSKUL 004E 234881054 268435479 1DI 11.44.56
.148
LATCHWAITPID=
0 CMD=dsskulker (NWS DS Skulker Daemon) ......
HOERNER INRSERV6 004F 654311448 251658278 1FI 11.45.22
.347
LATCHWAITPID=
0 CMD=nwserver (NWS Daemon)
SERVER=NETWARE
AF=
0 MF=00000 TYPE=SFDS
HOERNER
0000 100663321
1 1L
11.44.28
.001
HOERNER INRSERV2 004B 721420314
1 1WI 11.44.46
.053
LATCHWAITPID=
0 CMD=nwserver (NWS Daemon)
SERVER=NETWARE
AF=
8 MF=00000 TYPE=SFDS
Figure 100. OMVS, A=ALL
D NET,TRL
Display the status of the Transport Resource List Elements (TRLEs).
Figure 101 shows a example of a /D NET,TRL command.
D NET,TRL
IST097I DISPLAY ACCEPTED
IST350I DISPLAY TYPE = TRL 194
IST1314I TRLE = IUTL2100 STATUS
IST1314I TRLE = ISTT6463 STATUS
IST1314I TRLE = INRT1100 STATUS
IST1314I TRLE = MPC0210 STATUS
IST1454I 6 TRLE(S) DISPLAYED
IST314I END
=
=
=
=
ACTIV
ACTIV
ACTIV
ACTIV
CONTROL
CONTROL
CONTROL
CONTROL
=
=
=
=
TCP
XCF
MPC
MPC
Figure 101. D NET,TRL
D NET,E,ID=INRT1100
Figure 102 show the status of the defined TRL id INRT1100. See 3.4.3.5,
“Preparing the FENET OSA2 for IPX protocol” on page 140.
Novell Network Services
175
D NET,E,ID=INRT1100
IST097I DISPLAY ACCEPTED
IST075I NAME = INRT1100, TYPE = TRLE 327
IST486I STATUS= ACTIV, DESIRED STATE= ACTIV
IST087I TYPE = LEASED
, CONTROL = MPC , HPDT = YES
IST1715I MPCLEVEL = HPDT
MPCUSAGE = SHARE
IST1577I HEADER SIZE = 4092 DATA SIZE = 32 STORAGE = ***NA***
IST1221I WRITE DEV = 2321 STATUS = ACTIVE
STATE = ONLINE
IST1577I HEADER SIZE = 4092 DATA SIZE = 32 STORAGE = DATASPACE
IST1221I READ DEV = 2320 STATUS = ACTIVE
STATE = ONLINE
IST1500I STATE TRACE = OFF
Figure 102. D NET,E,ID=INRT1100
D U,,,2320,2
Figure 103 shows the correct status of the defined units. See 3.4.3.5, “Preparing
the FENET OSA2 for IPX protocol” on page 140.
IEE457I 14.04.43 UNIT STATUS 331
UNIT TYPE STATUS
VOLSER
2320 OSA A-BSY
2321 OSA A-SPD
VOLSTATE
Figure 103. Display unit command
3.6.5.2 NNS server utilities
In this section we describe some of the server utilities and their meanings and
show outputs of some key administration utilities issued as a command from the
/usr/lpp/netware/bin directory in the ISHELL. For a complete list of utilities, refer
to Novell Network Services for OS/390 Utilities Reference, SA22-7318.
• dsinstall—install and remove Novell Directory Services (NDS) from a server
• drouter—list network information known to router, for example, networks,
number of hops to a network, best path
• dsrepair—repair records, schema, bindery objects and external objects in
NDS database
• dsadmin—nwcm, display the NDS tree, set synch intervals
• ipxinfo—display IPX socket and LAN statistics kept by IPX driver
• ndsbackup—backup master replicas
Note: Disaster-recovery utility, should not be primary backup mechanism
used.
• ndsrestore—restore master replicas (should not be primary restore
mechanism)
• npsd—used by startnps to start IPX protocol stack, initialize drivers, and
daemons
• nwcm—used to view and configure server values, view device, adapter,
volume information
176
Debugging UNIX Applications on OS/390
• nwdiscover—optional, to discover network configuration information and
update server values
• nwdump—view NEMUX device status information
• nwengine—view total number of currently running NetWare engines
• nwipadmin—view and configure NCPS parameters for NWIP
• nwlicense—add or delete user licenses on the server (file/print serving)
• nwsapinfo—view information maintained in the server information tables
• nwsaputil—start or stop advertising a local server, or query contents of
"sapouts" file (servers advertised with permanent option)
• nwserver—boots NWS on UNIX system, reads NWS configuration from the
configuration file, mounts all volumes listed in the voltab file, starts all other
NWS processes, and processes asynchronous events
• nwserverstatus—check the status of the server
• nwshut—bring down the NetWare server
• nwvm—view/modify information in the voltab file
• ripinfo—view router driver statistics from the protocol stack
• rrouter—reset/rebuild IPX router table by requesting route information from
other servers
• spxinfo—view statistics for the SPX driver
• startnps—starts npsd, which initializes the IPX protocol stack
• startsapd—start the Service Advertise Protocol (SAP) daemon
• statnps—use in a script to check the status of the NetWare protocol stack
• stopnps—stop the NetWare protocol stack
• stopsapd—stop the SAP daemon
• track—start/stop display of incoming and outgoing SAP packets
• tsadmin—display time synchronization information, restart time
synchronization
$ nwserverstatus
Figure 104 through Figure 106 on page 178 show different statuses of the Novell
server on OS/390 via the nwserverstatus command.
HOERNER @ SC64:/>nwserverstatus
shmget returned 81, errno2=07200306
NetWare server NWREDPROJECT is down.
Figure 104. The Novell server is not up
Novell Network Services
177
HOERNER @ SC64:/>nwserverstatus
NetWare server NWREDPROJECT is coming up.
Figure 105. The Novell server is coming up
HOERNER @ SC64:/>nwserverstatus
NetWare server NWREDPROJECT is up.
Figure 106. The Novell server is up
ipxinfo
Use at the UNIX prompt to display the IPX(tm) socket and LAN statistics kept
by the IPX driver. Use this command in adition to the nserverstatus command to
verify that the server is up correctly sending and receiving packets.
Figure 107 shows an example of ipxinfo information.
HOERNER @ SC64:/>ipxinfo
IPX Socket Multiplexor (ISM) Version: 4.10
IPX LAN Router Version: 4.10
IPX LAN Router Statistics:
Information about packets received from the LAN(s)
0 Packets with DLPI header too small, dropped
0 Packets not DLPI data type, dropped
0 Data IPX packets coalesced
50996 Data size trimmed to match IPX data size
0 IPX/NetBIOS packets routed to other LAN(s)
0 IPX/NetBIOS packets that have reached route limit, not routed
89839 Total IPX data packets sent to a LAN or ISM
85052 Total IPX data packets received from applications
Figure 107. Ipxinfo example
startnps—initializes NetWare protocol stack (IP/IPX)
For more information see 3.5.2, “Server installation” on page 148.
nwdiscover—optional, determines IPX network characteristics
For more information see 3.4.3.6, “Network discovery” on page 143.
nwcm—configures server variables
nmcm -v ’nwconfig parm’ —view the actual settings
For more information about the nwcm parameters see Novell Network Services
for OS/390 Utilities Reference, SA22-7318.
For more information see 3.4.3.4, “LAN numbers” on page 139, 3.4.3.7, “System
parameters” on page 145, and 3.5.2, “Server installation” on page 148.
178
Debugging UNIX Applications on OS/390
nwserver—brings up the NetWare Services engine
For more information see 3.5.2, “Server installation” on page 148, Figure 74 on
page 152, Figure 75 on page 153, Figure 95 on page 172, Figure 96 on page 173,
and Figure 97 on page 173.
dsinstall—required for first bringup, creates NDS tree
For more information see, Figure 76 on page 153.
nwshut—brings down NetWare Services engine
For more information see 3.6.3.6, “Server recovery—hangs” on page 170.
stopnps—brings down NetWare protocol stack
For more information see 3.6.3.6, “Server recovery—hangs” on page 170
For more information about the listed commands and others see Novell Network
Services for OS/390 Utilities Reference, SA22-7318.
3.6.5.3 Novell client workstation utilities
In this section we describe some of the workstation utilities and their meanings.
For a complete list of utilities, refer to Novell Network Services for OS/390
Utilities Reference, SA22-7318.
• ATOTAL—totals accounting charges on a network
• COLORPAL—change color of menu elements
• CX—view/change context, view tree structure objects/containers
• FILER—manage files/directories
• FLAG—view/modify directory attributes, modify file/directory owner
• LOGIN—log in to server to access network, run login script
• LOGOUT—exit the network or log out from a server
• MAP—view/change drive mappings
• NCOPY—copy files/directories to different network locations
• NCUPDATE—update net.cfg files with new name context
• NDIR—view file/directory/volume information
• NETADMIN—view/manage NDS objects/properties
• NETUSER—manage various network tasks
• NetWare Administrator—perform administration tasks such as FILER,
NETADMIN, PARTMGR, PCONSOLE (has GUI interface)
• NLIST—view/search on objects and object properties
• NMENU—access customized menus
• NPATH—view NetWare search sequence for file
• NVER—view version information for workstation and servers
• NWXTRACT—extract files from diskette or CD-ROM to network
• PARTMGR—manage partitions and replicas
Novell Network Services
179
• PURGE—permanently remove deleted files from workstation
• RENDIR—rename a directory
• RIGHTS—view/modify user/group rights for a file/directory/volume
• SEND—send/receive network messages
• SETPASS—change user password
• SETTTS—view/modify locks for Transaction Tracking System (TTS)
• SYSTIME—synchronize workstation date/time with server
• UIMPORT—add objects to directory tree from ASCII import file
• WHOAMI—view connection information
• WSUPDATE—update a file on multiple drives and subdirectories
• WSUPGRD—upgrade IPX driver to corresponding ODI driver
For more information about the listed commands and others see Novell Network
Services for OS/390 Utilities Reference, SA22-7318.
3.7 Getting NetWare client software
For a PC client to communicate with the Novell Network Services server code,
the NetWare client software must be installed on the client. You can use the client
software provided with your operating system, or you can use the software
provided by Novell. IBM suggests that you download the client software from the
Novell Web site:
http://www.s390.ibm.com/oe/samples/oectrace.html
This will ensure that you are using the latest Novell client code. Look under the
topic NetWare Clients and obtain the client software for your operating system.
Create diskettes or use the network facilities to install the client software.
Figure 108 on page 181 shows the list of the NetWare page to download the
client code. We downloaded the newest WIN/NT client code.
180
Debugging UNIX Applications on OS/390
Figure 108. Download the client code
3.8 Emergency recovery
In the event of a system crash, you must have a written record of your network
organization. Use the worksheets in 3.9, “Installation Worksheets” on page 183 to
help you reinstall your system. The steps needed to recover Novell Directory
Services on your system depend on server factors that must be considered to
avoid creating problems in other servers in your network.
For information on using dsrepair, refer to "Managing the Directory Services
Tree" in Novell Network Services for OS/390 Supervising the Network,
SA22-7317.
Backup and recovery of the NDS database
If there should be a failure of the Novell Network Services for OS/390 server,
there are two recovery scenarios for which a customer should be prepared:
1. There are other servers in the network, holding Master or Read/Write replicas
also being kept on the Novell Network Services for OS/390 server.
2. The Novell Network Services for OS/390 server is the only one holding the
NDS tree, or there is a network-wide failure where all replicas are corrupted or
lost.
Note
The second scenario, while less likely in a multiserver environment, should still
be planned for, so appropriate backup images of the NDS database are taken,
and later made available as necessary.
Novell Network Services
181
• Following are the recovery scenarios for each case.
- Other replicas are available
The first thing to determine is whether the Novell Network Services for
OS/390 server can be restarted without requiring a rebuild (dsinstall). Once
the Novell Network Services for OS/390 server is restarted, the
administrator should run a dsrepair Full Unattended Repair. After this is
done, check that users can log in and access resources for replicas held on
the server. Additionally, the operator should check for other servers in the
network using the dsrepair utility (Advanced Operations => Replica and
Partition Operations => View Replica Ring). In the case where users can
log in, the other servers are visible, and there are no error messages, no
further actions need be taken. If the replica data has been corrupted on the
Novell Network Services for OS/390 server, as indicated by error
messages or the inability to log in or access the data, then the server will
need to go through dsinstall and be reconnected to the tree. The steps to
take are as follows:
+ If the Novell Network Services for OS/390 server was holding any
Master replicas, then for each of those partitions, another server holding
a Read/Write replica will need to be promoted to Master. This can be
done on an IntraNetWare server using NDSMGR, or on a nonnative
server using NetWare Administrator or PARTMGR. Refer to Novell
Network Services for OS/390 Utilities Reference, SA22-7318, for
details.
+ Bring up the Novell Network Services for OS/390 server, using the
dsinstall utility, and choosing option 3, to be inserted into an existing
NDS tree.
+ When dsinstall completes, add the replicas formerly kept on the Novell
Network Services for OS/390 server back as Read/Write replicas, using
NetWare Administrator or PARTMGR (see Novell Network Services for
OS/390 Utilities Reference, SA22-7318, for details).
+ Run a dsrepair Full Unattended Repair.
+ If the Novell Network Services for OS/390 server was holding any
Master replicas, the Read/Write replicas just created can be promoted
back to Master using either NetWare Administrator or PARTMGR.
• Other replicas are not available
In the case where no replicas exist for the NDS database, a previously backed-up
image of the database must be restored. Assuming the OS/390 NDS database
information was backed up, the Novell Network Services for OS/390 server can
simply be restarted without need to run dsinstall. If the Novell Network Services
for OS/390 server is the only server holding the tree, no further action is
necessary. If other servers had replicas of partition information, then they will
have to be rebuilt, and the replicas recreated, along the same lines as rebuilding
the Novell Network Services for OS/390 server described under "Other replicas
are available". All NDS data is stored in the /etc/netware/_netware directory. This
could be backed up at a minimum. For a more complete backup, the
customer-modified data—configuration files, device files, log files—are stored in
the /etc/netware directory. In either case, these files and directories can be
backed up using your current backup mechanism. Note that to avoid backing up
182
Debugging UNIX Applications on OS/390
incomplete data, the Novell Network Services for OS/390 server should be shut
down before backing up the data.
As an example of a recovery scenario, use the cpio utility to back up NDS data to
another HFS. Within OMVS, change to the /usr/netware directory. Then issue:
ls . | cpio -o >arch
This command will concatenate all the appropriate files into the output file "arch".
This file can then be copied to another HFS or other storage medium. To restore
the NDS database files, the concatenated file ("arch" in the example) needs to be
copied back into the /usr/netware directory. The Novell Network Services for
OS/390 server must not yet be started, and there should not be any other
directories or files in the /usr/netware directory. After copying, issue the
command:
cpio -i -I arch
This will recreate the NDS files. After deleting the concatenated file, the Novell
Network Services for OS/390 server can be started. The frequency of running
installation backup procedures should be determined based on the critical nature
of the data, and the number of servers holding that data.
3.9 Installation Worksheets
This section contains the NetWare Services Installation Worksheet and the
Novell Directory Services Replication Worksheet. Photocopy as needed and
complete them. They are valuable resources should you need to reinstall your
system. Make sure you keep them up to date.
Required Configuration
server_ name
ipx_internal_network
ts_type
Network Configuration
lan_1_adapter
lan_4_adapter
lan_1_network
lan_4_network
lan_1_frame_type
lan_4_frame_type
lan_2_adapter
lan_5_adapter
lan_2_network
lan_5_network
lan_2_frame_type
lan_5_frame_type
lan_3_adapter
lan_6_adapter
lan_3_network
lan_6_network
lan_3_frame_type
lan_6_frame_type
Novell Network Services
183
Optimal Configuration
3.10 Documentation
You can find documentation about Novell Network Services (NNS) on OS/390 at
the following sites:
On the Internet at:
http://www.s390.ibm.com/marketing/g2219072.html
• NNS books on the Internet:
http://www1.s390.ibm.com/os390/bkserv/novell.html
184
Debugging UNIX Applications on OS/390
Chapter 4. DCE/DFS
In this chapter we share some hints and tips for the DCE/DFS environment.
4.1 Distributed Computing Environment (DCE) overview
What is OS/390 UNIX System DCE?
The OSF Distributed Computing Environment (DCE) is a set of services that
make up a high-level, coherent environment for developing and running
distributed applications.
It is based on a client/server model, where a client requests a service from a
server. DCE provides a set of services that make this interaction possible. The
four core services in DCE are Remote Procedure Call, Directory Service,
Security Service, and Distributed Time Service. The elements of these services
run as long-running processes of daemons.
OS/390 UNIX System Services DCE offers the DCE services that enable an
OS/390 host to operate as a member of a DCE cell. OS/390 UNIX System
Services DCE includes the following daemons:
• DCE daemon
• Security server daemon
• CDS Advertiser daemon
• CDS Clerk daemon
• CDS daemon
• DTS Null Time Provider daemon
• DTS daemon
• Audit daemon
• Password Management daemon
• GDA daemon
The following sections will provide a brief description about DCE installation and
a complete description about error handling and getting cell informations.
4.1.1 DCE base service
The following figure gives a brief overview of the services available with OS/390
DCE:
© Copyright IBM Corp. 1999
185
Figure 109. OS/390 DCE architecture
The following information is provided in SYSTEMPAC Installation Guide.
4.1.1.1 Update BPXPRMxx parmlib member
Make the appropriate changes in parmlib member BPXPRMxx for the UNIX
System Services customization.
Detailed steps to customize parmilb member BPXPRMxx for OS/390 UNIX
System Services Application Services are defined in OS/390 UNIX System
Services Planning, SC28-1890.
BPXPRMxx considerations
• DCE Base Services MAXFILEPROC(256)
• MAXTHREADTASKS(500)
• MAXTHREADS(500)
Verify that both the AF_UNIX and AF_INET socket file systems are specified to
be initialized.
• For AF_UNIX, MAXSOCKETS(2000)
• For AF_INET, MAXSOCKETS(10000)
For specific information about AF_UNIX and AF_INET, refer to OS/390 MVS
Initialization and Tuning Guide, SC28-1751, and OS/390 UNIX System Services
Planning, SC28-1890.
186
Debugging UNIX Applications on OS/390
4.1.1.2 PROCLIB member considerations
The following procedure libraries were copied to the CPAC.PROCLIB data set,
since they must be added to a PROCLIB concatenation accessible to JES.
PROCLIB concatenation:
•DDNAME=SEUVPRC
• Element= DCE Base Services
Note
This data set contains ALIASes, which must also be accessible.
4.1.1.3 DCE base services IDL PROC customization considerations
If you intend to use the IDL Compiler under TSO, you may optionally make the
following changes to the IDL PROC found in the SEUVPRC library:
• Change DCEPFX to match the high-level qualifier under which you have
installed the DCE libraries.
• Change LNGPFX to match the high-level qualifier that you have used for the
SCEEH.H and SCEEMSGP libraries.
4.1.1.4 Update ISPF Command Table ISPTCM
If you add any of the following to the LPA list, or to the modified link pack areas,
you must add these command names to the ISPF Command Table (ISPTCM):
EUVBCONF
EUVCCP
EUVDCECP
EUVRCP
EUVRUGEN
EUVSADMIN
EUVSAED
EUVSCDB
EUVSEDB
EUVSLGN
EUVSKDST
EUVSKINI
EUVSKLST
EUVRIDL
EUVSRED
EUVTCP
For further information and instructions, see OS/390 Interactive System
Productivity Facility (ISPF) Planning and Customizing, SC28-1298.
4.1.1.5 Define DCEKERN to RACF
To define DCEKERN to RACF you must create the following definitions with these
exact names:
• Define DCEGRP as a group.
• Define DCEKERN as a user.
• Define DCEKERN as a started task.
SystemPac has already defined DCEKERN to RACF for you. Please refer to
CPAC.SAMPLIB(RACFOE) for the detailed definitions that were made.
For RACF, the following commands were used to make the definitions to the
STARTED class:
ADDGROUP DCEGRP SUPGROUP(SYS1) OMVS(GID(2))
ADDUSER DCEKERN OMVS(HOME(/opt/dcelocal/home/dcekern/) UID(0))
DFLTGRP(DCEGRP) AUTHORITY(CREATE) UACC(ALTER)
RDEFINE STARTED DCEKERN.** STDATA(USER(DCEKERN) GROUP(DCEGRP))
RDEFINE STARTED DCESECD.** STDATA(USER(DCEKERN) GROUP(DCEGRP))
RDEFINE STARTED DCECDSD.** STDATA(USER(DCEKERN) GROUP(DCEGRP))
RDEFINE STARTED DCEDTSTP.** STDATA(USER(DCEKERN) GROUP(DCEGRP))
DCE/DFS
187
RDEFINE STARTED DCEAUDD.** STDATA(USER(DCEKERN) GROUP(DCEGRP))
RDEFINE STARTED DCEPWDD.** STDATA(USER(DCEKERN) GROUP(DCEGRP))
RDEFINE STARTED DCEGDAD.** STDATA(USER(DCEKERN) GROUP(DCEGRP))
SETROPTS RACLIST(STARTED) REFRESH
In addition to the STARTED class definitions, the RACF Started task table was
also updated with the following definitions:
DCEKERN DC CL8’DCEKERN’ Started Task Name
DC CL8’DCEKERN’ Userid
DC CL8’DCEGRP’ Group
DC X’00’,XL7’00’ Neither Privileged nor Trusted
DCESECD DC CL8’DCESECD’ Started Task Name
DC CL8’DCEKERN’ Userid
DC CL8’DCEGRP’ Group
DC X’00’,XL7’00’ Neither Privileged nor Trusted
DCECDSD DC CL8’DCECDSD’ Started Task Name
DC CL8’DCEKERN’ Userid
DC CL8’DCEGRP’ Group
DC X’00’,XL7’00’ Neither Privileged nor Trusted
DCEDTSTP DC CL8’DCEDTSTP’ Started Task Name
DC CL8’DCEKERN’ Userid
DC CL8’DCEGRP’ Group
DC X’00’,XL7’00’ Neither Privileged nor Trusted
DCEAUDD DC CL8’DCEAUDD’ Started Task Name
DC CL8’DCEKERN’ Userid
DC CL8’DCEGRP’ Group
DC X’00’,XL7’00’ Neither Privileged nor Trusted
DCEPWDD DC CL8’DCEPWDD’ Started Task Name
DC CL8’DCEKERN’ Userid
DC CL8’DCEGRP’ Group
DC X’00’,XL7’00’ Neither Privileged nor Trusted
DCEGDAD DC CL8’DCEGDAD’ Started Task Name
DC CL8’DCEKERN’ Userid
DC CL8’DCEGRP’ Group
DC X’00’,XL7’00’ Neither Privileged nor Trusted
For details, see the following:
• Data Set CPAC.SAMPLIB(RACFOE)
• OS/390 UNIX System Services Planning, SC28-1890
• OS/390 Security Server (RACF) Security Administrator’s Guide, SC28-1915
• OS/390 Security Server (RACF) Command Language Reference, SC28-1919
4.1.1.6 Define the DCE administrator user ID to RACF
To define the DCE administrator user ID to RACF, you must define DCEADM as a
user.
For RACF, use the following commands. Replace parameters with any
system-specific TSO information required on your system.
MKDIR ’/u/dceadm/’ MODE(7,5,5)
ADDUSER DCEADM OMVS(HOME(/u/dceadm) PROGRAM(/bin/sh) UID(0))
DFLTGRP(DCEGRP) AUTHORITY(CREATE) UACC(NONE)
TSO(-parameters-)
SystemPac has already created the DCEADM user ID in the RACF database.
188
Debugging UNIX Applications on OS/390
For details, see the following:
• Data Set CPAC.SAMPLIB(RACFOE)
• OS/390 UNIX System Services Planning, SC28-1890
• OS/390 Security Server (RACF) Security Administrator’s Guide, SC28-1915
• OS/390 Security Server (RACF) Command Language Reference, SC28-1919
4.1.1.7 Create the DCEKERN.START.REQUEST resource profile
To successfully run the DCE configuration, SystemPac made the following
changes to your delivered RACF database.
• Define the following FACILITY class resource profiles to your security
program.
• Activate the FACILITY class.
• Permit DCEADM and DCEKERN update access to the resources defined
below.
For RACF, use the following commands:
RDEFINE FACILITY DCEKERN.START.REQUEST UACC(NONE)
RDEFINE FACILITY DCESECD.START.REQUEST UACC(NONE)
RDEFINE FACILITY DCECDSD.START.REQUEST UACC(NONE)
RDEFINE FACILITY DCEDTSTP.START.REQUEST UACC(NONE)
RDEFINE FACILITY DCEAUDD.START.REQUEST UACC(NONE)
RDEFINE FACILITY DCEPWDD.START.REQUEST UACC(NONE)
RDEFINE FACILITY DCEGDAD.START.REQUEST UACC(NONE)
PERMIT DCEKERN.START.REQUEST CLASS(FACILITY) ID(DCEKERN DCEADM)
ACCESS(UPDATE)
PERMIT DCESECD.START.REQUEST CLASS(FACILITY) ID(DCEKERN DCEADM)
ACCESS(UPDATE)
PERMIT DCECDSD.START.REQUEST CLASS(FACILITY) ID(DCEKERN DCEADM)
ACCESS(UPDATE)
PERMIT DCEDTSTP.START.REQUEST CLASS(FACILITY) ID(DCEKERN DCEADM)
ACCESS(UPDATE)
PERMIT DCEAUDD.START.REQUEST CLASS(FACILITY) ID(DCEKERN DCEADM)
ACCESS(UPDATE)
PERMIT DCEPWDD.START.REQUEST CLASS(FACILITY) ID(DCEKERN DCEADM)
ACCESS(UPDATE)
PERMIT DCEGDAD.START.REQUEST CLASS(FACILITY) ID(DCEKERN DCEADM)
ACCESS(UPDATE)
For details, see the following:
• Data Set CPAC.SAMPLIB(RACFOE)
• OS/390 Security Server (RACF) Security Administrator’s Guide, SC28-1915
• OS/390 Security Server (RACF) Command Language Reference, SC28-1919
DCE/DFS
189
4.1.1.8 Update the eNetwork Communications Server IP profile
DCE requires use of the eNetwork Communications Server IP port 135. Ensure
that data set hlq.PROFILE.TCPIP does not reserve port 135 for another
application. Refer to TCP/IP for MVS: Customization and Administration Guide,
SC31-7134, for information about the PORT statement and data set
hlq.PROFILE.TCPIP.
4.1.1.9 OS/390 2.7 DCE Application Support
To complete the installation process relink load module ASUVSC for CICS.
Relink Load Module ASUVSC for CICS
DCE Application Support load module ASUVSC needs to be rebuilt with the code
from your existing SDFHEXCI (CICS) library. The SDFHEXCI DDDEF entry was
redefined based on the variable entry you entered for installed CICS (if you have
it) and run report CALLLIBS to generate the job that will relink the module.
The following is a sample JCL that you may use to generate the CALLLIBS
report. For more information about the SMP/E REPORT CALLLIBS command,
refer to OS/390 SMP/E Reference, SC28-1806.
//CICSLINK JOB <job card parameters>
//STEP1 EXEC PGM=GIMSMP
//SMPCSI DD DISP=SHR,DSN=your.SMP/E.zone.csi
//JOBCDD DD DISP=SHR,DSN=your.dataset.with.the.job.card.member
//SMPPUNCH DD DISP=SHR,DSN=the.generated.link.jcl.dataset
//SMPCNTL DD *
SET BDY(GLOBAL).
REPORT CALLLIBS (SDFHEXCI) /* REPORT CALLLIBS */
ZONES(your_target_zone)
JOBCARD(JOBCDD,job_card_member)
4.1.2 Configuring a cell
Here is some helpful information about a DCE cell.
4.1.2.1 What is a DCE cell
The basic unit of DCE operation is the cell, which is a logical grouping of users,
computers, data, and other resources that share either a common purpose or a
common level of trust. The cell provides security, administration, and boundaries
naming for users and resources within that cell. The DCE services are managed
within the context of a cell. The following figure illustrates a DCE cell:
190
Debugging UNIX Applications on OS/390
Figure 110. DCE cell
Cells can vary in size from a cell with a single machine to a cell with thousands of
machines. The DCE services were designed to scale up to meet this demand. A
cell must have at minimum of:
• One Security Server
• One Cell Directory Server (CDS)
• One Distributed Time Server
The typical daemon configuration in a cell is shown in Figure 111:
Figure 111. DCE daemons in the cell
DCE/DFS
191
4.1.2.2 Configure the cell
Before you start to configure the DCE cell provide the following information:
• Cell Admin ID
• Cell Admin Password
• Cell Name
• TCP/IP Name
• Internet Addr
• Security Host Name
• Cell Directory Host Name
• Directory Time Server Names
OS/390 systems provide panels to configure the cell. Please refer to OS/390 DCE
Configuring and Getting Started, SC28-1583, for more information.
4.1.3 DCE start, stop, and debug hints
More detailed information about DCE is documented in the following sections:
4.1.3.1 Start and stop of DCEKERN and ist daemons
To start the DCEKERN address space and all ist daemons, enter the S DCEKERN
command, either from a system console or from any authorized console.
If, for test purposes, or other reasons, you want to start DCE without starting its
daemons (for example, if you want to disable DCE after a problem), start
DCEKERN as follows:
S DCEKERN, PARM='-NODCE'
Be sure to enter the parameter exactly as shown. This is equivalent to a local
start of DCE.
To stop the DCEKERN address space, simply enter:
P DCEKERN
To ask about the status of DCEKERN and all its daemons, enter the following
command:
F DCEKERN,QUERY ALL
For more detailed information refer to OS/390 DCE Administration Guide,
SC28-1584.
4.1.3.2 Ways of starting the OS/390 USS DCE daemons
OS/390 DCE daemons are started automatically or manually, depending on the
situation and on the system setup. You can start the daemons:
• When running the DCE configuration program DCECONF
Using the MODIFY DCEKERN operator command, you can start the daemons:
• During a System IPL, if DCEKERN is among the tasks that are started
automatically
• By an Automated Operations product
192
Debugging UNIX Applications on OS/390
To start the DCE daemons manually, follow this sequence:
F
F
F
F
F
F
F
DCEKERN,
DCEKERN,
DCEKERN,
DCEKERN,
DCEKERN,
DCEKERN,
DCEKERN,
START
START
START
START
START
START
START
DCED
SECD
CDSADV
CDSCLERK
DTSTP
DTSD
AUDITD
For example, to start the CDSADV daemon successfully, the DCED daemon must
already be up and running.
To start all daemons collectively (and in the right sequence) enter:
F DCEKERN, START ALL
To stop all daemons, enter the following:
F DCEKERN, STOP ALL
If you want to stop only a specific daemon, enter:
F DCEKERN, STOP daemonname
The path name to the daemon configuration file is:
/opt/dcelocal/etc/euvpdcf
This file reflects the configuration status as well as the startup options of the
daemons that run in the DCEKERN address space.
Detailed information about how to start and stop daemons can be found in:
OS/390 DCE Administration Guide, SC28-1584
4.1.3.3 MVS/DCE debug tracing
MVS/DCE debug tracing is part of the overall RAS Message Logging Services.
The traces provide DCE-specific information about events that occur within the
DCE components. For each DCE component you can specify the level and type of
output you wish to see.
Valid values for the debug levels range from 1 to 9. The general guideline is that
the higher the level the more detailed the information produced by the trace.
Unless you are familiar with how a particular component has implemented debug
levels, you should probably indicate a level of 9 to get as much information as
possible. Specify a level of 0 to indicate that you do not want tracing for that
component.
How to stop and start debug tracing
In the envvar file for DCEKERN as well as for each of the daemons there is a
variable _EUV_SVC_DBG_MSG_LOGGING=n, where n=0 or 1. A value of 1
indicates that debug tracing is to be done. To change this variable you can use
the panel driven UNIX System Services Ishell. Simply changing the value is not
enough, you also need to stop and start the particular daemon for which you wish
to produce trace output. There is a set of associated environment variables that
DCE/DFS
193
provide component specific settings for producing debug trace output. The default
settings, as provided in the envar files are the following:
_EUV_SVC_MSG_LOGGING=CONSOLE_LOGGING
_EUV_SVC_DBG_MSG_LOGGING=0
_EUV_AUTOLOG=NO
SVC_AUD_DBG=*.9:STDOUT:-SVC_CDS_DBG=*.9:STDOUT:-SVC_DHD_DBG=*.9:STDOUT:-SVC_DTS_DBG=*.9:STDOUT:-SVC_PLT_DBG=*.9:STDOUT:-SVC_RPC_DBG=*.9:STDOUT:-SVC_SEC_DBG=*.9:STDOUT:-SVC_SED_DBG=*.9:STDOUT:-_EUV_FTRACE=0
TZ=MEZ-1MSZ-2,M3.5.0,M10.5.0/03:00:00
NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/En_US.IBM-1047/%N
LANG=En_US.IBM-1047
You can find the envvar file name in the startup file for daemons under:
/opt/dcelocal/etc/euvpdcf copied from /usr/lpp/dce/etc/euvpdcf.
You can direct the debug trace output to STDOUT, STDERR, or a file name. Each
debug entry has two parts:
The default data and the data unique to the component. Each entry will start with
the default data which includes:
• TCB address
• Thread ID
• Process ID
• Component
• Subcomponent
• Name of the routine making the debug entry
For more detailed information please call your support center.
4.1.3.4 MVS/DCE function trace
The MVS/DCE function trace is simply a trace of all the DCE C subroutine calls.
Note that function traces are also done for any C Program compiled with TEST
(NONE) option.
When Language Environment (LE) detects an entry or an exit from a C language
subroutine call, LE will first determine if the called routine is a DCE C function.
For such functions LE will call the corresponding DCE code. The following
information is available in a function trace entry:
• TCB address
• Thread ID
• Process ID
• Programming level for the function
• Direction of the call
194
Debugging UNIX Applications on OS/390
• The name of the function
• Return code for the function
How to start and stop function tracing
Function tracing is controlled by the environment variable _EUV_FTRACE.
Setting this variable to 1 will enable function tracing.
The directory names for the daemons are:
cdsadv
cdsclerk
dtsd
sec_clientd
rpcd
Function trace output is directed to STDOUT. You may, of course, redirect your
output to a file as well. For more detailed information please call your support
center.
4.1.4 How to produce a dump
DCE users may find themselves in situations where they find that DCE as such
(DCEKERN) and/or DCE applications has stopped working or is looping. To find
out whether the application hangs or is looping start by going into the SDSF
Display Active panel. A task that is in a loop should show a very high percentage
of CPU utilization under the heading CPU% on the screen. A hang or wait
situation would normally be indicated by a swapped out indication in the pos field.
However, DCEKERN itself is mostly defined as non-swallowable which means
you would have to watch the address space for a while to determine if it uses
CPU or performance I/O operations. In any case we would normally suggest to
dump the DCEKERN ASID, the UNIX System Services ASID and the TSO/E user
address space.
You can accomplish this with the following commands:
Table 33. DCE Dump commands
Command
Description
/d d
To check whether dump data sets are available
/dd clear,dsn=nn
If necessary, clear dump data set (dsn=nn or all)
/d d,o
To check dump options
/dump comm=(dump title)
Issue your dump command
/yy,ASID=(aa,bb,cc),END
Where yy is the reply ID and aa,bb,cc are the address
space IDs in hex
How to dump the UNIX System Service address space and its data spaces is
described in Chapter 1, “OS/390 UNIX System Services” on page 1.
4.1.5 How to get a trace output from DCEKERN
What about extracting the output from the JES spool to read it at your leisure on
your workstation or using ISPF browse? What you really need is a way to copy
the log to a data set even if the job is not completed.
DCE/DFS
195
Sure enough, you can do this from SDSF. What you will end up getting is a
complete job log, including everything for all daemons. This is okay since you can
then go back and trim it down to the portion you need to worry about. Here is how
to do it:
1. In SDSF display the active users (choose da, or in ISPF panel sd.da)
2. Allocate a data set using the xd prefix command in SDSF. Put the cursor next
to any active job and type xd and press Enter. Fill in the data set name, NEW
for disposition, and the size of the data set on the panel that comes up.
3. Go to the job log from SDSF panel da (ISPF panel sd.da). From the panel, put
an s next to the job for which you want to save the log. This will bring up the
log.
4. On the command line, type print and press Enter.
5. A message in the upper right corner of the screen should indicate how many
lines were printed.
6. Leave SDSF and use ISPF browse to look at the data.
You can then FTP the data set to the target platform for further investigation.
4.1.6 DCE components
DCE components include:
• The DCE Security Service
• The DCE Directory Service
• The DCE Distributed Time Service
• The DCE Remote Procedure Call
• The Audit Service
4.1.6.1 The DCE Security Service
The DCE Security Service is composed of three primary services—the
Authentication Service, the Registry Service, and the Privilege Service.
• The DCE Authentication Service component performs security functions that
interact, for example, with DFS. It ensures that only certified users can log in
and use the system, and it ensures that only authorized machines can
communicate with other machines in the network.
• The DCE Registry Service maintains a registry database. This database
contains information similar to that stored in UNIX password files, such as
users, groups, and account information. An account defines who can log in to
the system and includes information about passwords and home directories.
• The DCE Privilege Service component ensures that those who are using the
system have the necessary permissions to perform the operations they
request.
These three services rely on the DCE Security Server, the secd process. The
secd process runs all DFS server machines to provide access to the security
services in the previous list.
196
Debugging UNIX Applications on OS/390
The DCE Security Service also includes the following two facilities:
• The DCE Access Control List Facility provides an interface that allows users to
set different levels of protection on file system objects, such as directories and
files. Users can grant permissions to individuals, or they can define groups of
users and grant permissions to the groups. They can then add individuals to a
group to grant them the same permissions as the group or they can remove
individuals from a group to restrict their permissions. An object’s ACL interacts
with the protections provided by the object’s UNIX mode bits.
• The DCE Login Facility initializes a user’s security environment in the DCE. It
employs a user’s password to authenticate the user to the DCE Security
Service, returning authentication information associated with the user. This
information is used to authenticate the user to other distributed services such
as those in DFS.
4.1.6.2 The DCE Directory Service
The DCE Directory Service provides a consistent way to identify and locate
people and resources, including files and directories, anywhere in a network
computing environment.
The Directory Service has three main components:
• The Cell Directory Service (CDS) manages names within a cell. Each
resource has a CDS entry that is unique within its local cell.
• The Global Directory Service (GDS) supports the global naming environment
between cells and outside of cells. GDS is an implementation of a directory
service standard known as X.500.
• The Global Directory Agent (GDA) is a "gateway" between the local and global
naming environments. It supports cell interoperability by allowing CDS to
access a name in another cell through either the GDS or the Domain Name
System (DNS), a widely used global naming environment.
Examples of CDS entries:
Examples of CDS entries in both GDS and DNS global naming formats follow.
The first example shows a CDS entry for a server machine in DNS format:
/.../abc.com/hosts/fs1/self
The second example shows a similar entry in GDS format:
/.../C=US/O=abc/OU=Writers/hosts/fs1/self
In addition to their global names, all CDS entries have a cell-relative name, or
local name, that is usable only within the cell where the entry exists. The
cell-relative name begins with the /.: prefix, which replaces the global cell name.
An example of a CDS entry that uses the cell-relative prefix follows:
/.:/hosts/fs1/self
4.1.6.3 The DCE Distributed Time Service
The DCE Distributed Time Service (DTS) provides precise synchronization for
system clocks in a network. For example, in DFS, clock synchronization is
important for communication between client machines using the Cache Manager
and server machines running the File Exporter and other server processes.
DCE/DFS
197
Clients and servers must refer to a common time standard for communications to
remain constant and for data to remain available.
For example, each client that obtains tokens from a File Exporter has a lifetime
with respect to that File Exporter. The client must renew its lifetime before it
expires to ensure that its tokens are not revoked without its knowledge. If the
client and the File Exporter disagree on the current time, the File Exporter may
believe the client’s lifetime has expired before the client does. In this case, the
File Exporter may revoke the client’s tokens without its knowledge.
Clock synchronization is also important for replicated location databases and
backup databases, which must be coordinated on different server machines.
Machines that house replicated databases must remain in constant contact to
ensure that each server has the current copy of the database. If the machines
disagree on the time, they may believe they are no longer in touch with each
other, in which case they can refuse all requests for information. Synchronization
problems of this nature can result in unnecessary disruption of database access.
4.1.6.4 The DCE Remote Procedure Call
The DCE Remote Procedure Call (RPC) facility provides communication between
client and server machines in a network. For the Cache Manager on a client
machine to send a request for data or other resources to a server machine, it
must know how to locate the File Server machine in the network and how to
communicate with it. The RPC requires the use of a binding handle for the File
Server machine on which a file set resides.
The binding handle includes the server machine’s network address, an identifier
for the protocol used to communicate with the machine, and an endpoint (often a
port number) for communication with the machine. It also contains user
authentication information about a user who requests data. The Cache Manager
uses this binding handle to communicate with the server machine.
The same process is used to effect communication between different server
machines. For example, DFS employs Ubik to synchronize copies of the Fileset
Location Database and Backup Database. Instances of Ubik that coordinate the
databaes on different server machines rely on RPCs to communicate with each
other. Communication failures resulting from RPC problems can cause
unnecessary disruption of database access.
When a server process first starts, it registers its process endpoint with the rpcd
process. The rpcd process running on a server machine provides the location
information required by clients to communicate with sever processes running on
the server machine. Note that in DCE RPC, rpccp is a program that, among other
things, allows system administrators to administer and control rpcd on a machine.
Most RPC administrative tasks, however, are performed automatically when the
server first starts.
The UUID facilities are another component of the DCE RPC employed by DFS
and other DCE components. The commands and routines in the facility are used
to generate Universal Unique Identifiers (UUIDs). The UUIDs are used to
uniquely identify resources such as machines and processes. For example, the
backup system uses UUIDs to identify the tape coordinator processes on
machines that are used to back up data to tape.
198
Debugging UNIX Applications on OS/390
4.1.6.5 The DCE Audit Services
Audit plays a critical role in distributed systems. Adequate audit facilities are
necessary for detecting and recording critical events in distributed applications.
Audit, a key component of DCE, is provided by DCE Audit Services.
The DCE Audit Services have the following features:
• An Audit daemon writes audit records based on specified criteria.
• Application Programming Interfaces (APIs) can be used as a part of
application server programs to record audit events. These APIs can also be
used to create tools that can analyze the audit records.
• An administrative command interface for the Audit daemon directs the daemon
to select the events to be recorded according to specified criteria. This
interface is accessed through the DCE control program (dcecp).
• An event classification mechanism allows the logical grouping of events into
sets to ease administration.
• Audit records can be directed to logs or to the system console.
For detailed information please refer to OS/390 DCE Administration Guide,
SC28-1584.
4.2 Distributed File Service (DFS)
What is DFS?
A distributed computing environment involves many different systems working
together to perform the same types of functions usually performed on one
machine in a traditional computing environment. For example, in a distributed
computing environment, one system (or component) can be responsible for login
procedures and file protections; another can be responsible for providing a
uniform way to name and view all objects in the environment.
These components communicate through a computer network, providing system
access to machines and users at different geographical locations.
A distributed file system joins the file systems of individual machines into a single,
global file system (filespecs) accessible to many machines. OSF (Open System
Foundation) DCE employs the Distributed File System (DFS). Working with DFS
is similar to working with the file system of your local machine. However, in DFS,
you can access files stored on different machines in the computer network as
easily as you can access files stored on your machine’s local disk. You do not
need to know the physical location of a file, even though a given file can be stored
on any machine in the network. Simply request a file by its DCE pathname, and
DFS finds the correct file automatically.
Groups of DFS users from different locations can share information and work
together on the same files. By using DCE Access Control Lists (ACLs) you can
prohibit certain users from accessing specific files or directories. This is one
aspect of DFS that provides a secure computing environment.
DFS is based on a distributed client/server model. In a client/server model, server
machines store data that client machines can access. The DFS model is
considered distributed because data stored on many different DFS server
DCE/DFS
199
machines is potentially available to users on every DFS client machine in the
DCE environment.
4.2.1 DFS installation
4.2.1.1 Update BPXPRMxx PARMLIB Member
Add the following parameters to BPXPRMxx:
FILESYSTYPE TYPE(DFSC)
ENTRYPOINT(IOECMINI)
PARM(’ENVAR("_EUV_HOME=/opt/dfslocal/home/dfscm")/
>DD:IOEDFSD 2>&1’)
ASNAME(DFSCM)
4.2.1.2 PROCLIB member considerations
The procedures needed to start the DFS started task are located in the target
data set SIOEPROC.
After updates to these procedures are made, the members listed below must be
made available to the system by copying them to a procedure library which is in
your JES procedure library concatenation.
Table 34. DFS procedures
Name
Alias
Description
IOEP0001
DFS
JCL that starts the DFS server
IOEP0002
DFSCM
JCL that starts the DFS client
IOEP0003
DFSKERN
JCL to run DFSKERN process in a
separate address space
See also OS/390 Distributed File Service Configuring and Getting Started,
SC28-1722 for detailed reference.
4.2.1.3 Define DFS Server (DFS) and DFS Client (DFSCM) to RACF
To define DFS to RACF, you must create the following definitions with these exact
names:
• Define DFSGRP as a group.
• Define DFS as a user.
• Define DFS as a started task.
• Define DFSCM as a started task.
SystemPac has already defined DFS to the RACF database using these
definitions. Definitions were made to both the RACF Started Task Table
(ICHRIN03) and the RACF STARTED Class. For details see
CPAC.SAMPLIB(RACFOE).
For RACF, the following commands were used:
ADDGROUP DFSGRP SUPGROUP(SYS1) OMVS(GID(2))
ADDUSER DFS OMVS(HOME(/opt/dfslocal/home/dfscntl) UID(0))
DFLTGRP(DFSGRP) AUTHORITY(CREATE) UACC(ALTER)
RDEFINE STARTED DFS.** STDATA(USER(DFS)) 1
200
Debugging UNIX Applications on OS/390
Note
1 The user ID DFS should not be enabled for DCE single sign-on.
RDEFINE STARTED DFSCM.** STDATA(USER(DFS))
SETROPTS RACLIST(STARTED) 2
Note
2 The RACF Started Task table (ICHRIN03) was updated with the following:
DFS
DFSCM
DC
DC
DC
DC
DC
DC
DC
DC
CL8’DFS’
Started Task Name
CL8’DFSGRP’
X’00’,XL7’00’
CL8’DFSCM’
CL8’DFS’
CL8’DFSGRP’
X’00’,XL7’00’
Group
Neither Priv nor Trusted
Started Task Name
Userid
Group
Neither Priv nor Trusted
SETROPTS RACLIST(STARTED) REFRESH
For details, see the following:
• Data Set CPAC.SAMPLIB(RACFOE)
• OS/390 UNIX System Services Planning, SC28-1890
• OS/390 Security Server (RACF) Security Administrator’s Guide, SC28-1915
• OS/390 Security Server (RACF) Command Language Reference, SC28-1919
4.2.1.4 Create the DFSKERN.START.REQUEST Resource Profile
A RACF facility for DFSKERN must be added to the security program and be
permitted to use ID(s) that run DFS:
• Define the FACILITY class resource profile ’DFSKERN.START.REQUEST’ to
your security Program.
• Ensure the FACILITY class is active.
• Permit DFSKERN update access to the ’DFSKERN.START.REQUEST’
resource.
SystemPac has already defined DFSKERN to the RACF database and permitted
it to the user ID(s) that run DFS.
For RACF, the following commands were used:
RDEFINE FACILITY DFSKERN.START.REQUEST UACC(NONE)
PERMIT DFSKERN.START.REQUEST CLASS(FACILITY) ID(user ID(s))
ACCESS(UPDATE)
SETROPTS CLASSACT(FACILITY)
For details, see the following:
• Data Set CPAC.SAMPLIB(RACFOE)
• OS/390 Security Server (RACF) Security Administrator’s Guide, SC28-1915
• OS/390 Security Server (RACF) Command Language Reference, SC28-1919
DCE/DFS
201
Note
The user ID(s) refer to the user IDs under which the DFS started task runs. In
the previous section DFS was used as the example user ID.
4.2.1.5 New customer considerations
If this is the first time you have installed OS/390 Distributed File Service,
installation customized parameter files must be created by running the
dfs_cpfiles program as described below. The files created by the dfs_cpfiles
program are listed in OS/390 Distributed File Service Configuring and Getting
Started, SC28-1722.
4.2.1.6 Previous customer considerations
If you are migrating from a previous release of OS/390 Distributed File Service
and installing this DFS release on a host system where DFS is already installed
and configured, you can ensure that all the required installation customized
parameter files are created by running the dfs_cpfiles program. Note that the
dfs_cpfiles program only creates files that do not exist and does not replace any
existing file so that any previous installation data is not overlaid. Also, if you want
to simplify the installation verification processing, you may want to update the
/opt/dfslocal/etc/CacheInfo file to specify memory caching before you IPL the
system and restart the new version of the DFS Client (DFSCM).
If you have DFS installed on a host system but are installing this release of DFS
on a host system where DFS is not already installed and configured, you can use
the default configuration files that are created by the dfs_cpfiles program. You
cannot copy all the DFS customized files and configuration files from one host
system to another because the files contain DCE cell and host system specific
data. For a description of the HFS files containing DFS customization data that
can be selectively copied to another host system, refer to the OS/390 Distributed
File Service Configuring and Getting Started, SC28-1722.
4.2.1.7 Running the dfs_cpfiles program
To invoke the dfs_cpfiles program:
• Log in as ROOT on the local machine. In OS/390 DFS, this means as a user
with UID = 0.
• While in the OS/390 UNIX shell environment, invoke the OS/390 Default
Configuration Files Creation program by entering the following:
$ /usr/lpp/dfs/global/scripts/dfs_cpfiles
Refer to OS/390 Distributed File Service Configuring and Getting Started,
SC28-1722, for additional information about the dfs_cpfiles Program.
4.2.2 Starting and stopping DFS
In these sections we describe how to start and stop DFS.
4.2.2.1 Who can start and stop DFS daemons?
There are two types of users who can start or stop DFS daemons in DFS:
1. A user with MVS/ESA operator privileges.
202
Debugging UNIX Applications on OS/390
2. A user who has updates to the DFSKERN.START.REQUEST RACF facility.
This facility is created during the installation of DFS. For more information on
this RACF facility, refer to the Program Directory for OpenEdition DCE DFS.
4.2.2.2 Starting DFS daemons
DFS daemons are started in any of the following ways:
• During the system IPL
• Using the MODIFY DFS operator command
• Using the START operator command
Based upon the control files for the DFS Control Task and the BOS Server, set up
by the DFS Administrator, all of the DFS daemons are automatically started when
the DFS started task is begun.
The DFS/NVS daemons all run in one address space, which is a started task
(default name DFS). A user with MVS operator privileges can start and stop the
DFS started task. The DFS address space may be started automatically during
IPL (after DCEKERN is started). To start or stop the DFS address space, use the
MVS commands. For example:
START DFS
STOP DFS
You can start the DFS address space without starting the configured DFS
daemons using the following command:
START DFS,PARMS=’-NODFS’
Ideally, the daemons run continuously in the background and do not need to be
started or stopped again. However, the DFS daemons may have to be started
manually in certain situations, for example, if a daemon ends abnormally.
You can use the MODIFY operator command to manually start or stop the DFS
daemons.
For more information refer to OS/390 Distributed File Service Configuring and
Getting Started, SC28-1722.
4.2.2.3 The MODIFY DFS operator command
The DFS daemons can be started or stopped using the MODIFY DFS operator
command. Using MODIFY DFS, you can also view the status of the DFS
daemons.
Following is the syntax of the MODIFY DFS command:
MODIFY DFS, command daemon [start_options]
DCE/DFS
203
This can be read as follows:
Table 35. The MODIFY DFS command
Command part
Description
DFS
The name of the DFS address space
command
The action that is to be performed on the DFS daemon
or daemons. It can have any of the following values:
start - Starts the DFS daemon(s)
stop - Stops the DFS daemon(s)
query - Displays the state of the DFS daemon(s)
daemon
The name of the DFS daemon for which the action is
requested. It can have any of the following values:
dfskern - DFS kernel program
export - Export program to make aggregates available for
exporting
unexport - Unexport program to unexport aggregates
boserver - Basic OverSeer Server to monitor all server
processes on the local machine and restart any failed
processes automatically
butcnn - Backup Tape Coordinator that controls the
behavior of the backup tape drive and accepts requests
from the bak program (nn is a numeric value, 01 through
08).
all - All the DFS daemons
start_options
Values that are passed when the daemons are started.
Using MODIFY DFS to start DFS daemons: With the Modify DFS operator
command, you have the option of starting an individual daemon or starting all the
daemons using a single command.
For example, to start the BUTC01 daemon enter the following:
MODIFY DFS, START BUTC01
To start all the daemons, enter the following:
MODIFY DFS, START ALL
Note
Do not use the MODIFY command to start the DFS daemons while the DFS
address space is still initializing. During initialization, DFS will attempt to start
all the DFS daemons that have been configured on the MVS/ESA host. If you
issued the MODIFY command while DFS is initializing, the DFS daemons may
be started out of order or stopped erroneously. This may lead to unexpected
errors during initialization and cause DFS to end abnormally.
You must wait until DFS has issued a log message indicating that DFS
initialization has completed before using the MODIFY commands.
204
Debugging UNIX Applications on OS/390
Order of starting DFS daemons
When DFS daemons are started manually, the successful startup of some
daemons depends on the availability of the services provided by other daemons.
This implies that the DFS daemons must be started in a particular order.
Before you can start up any of the DFS daemons, the DCE Security daemon and
the DCE CDS daemon must already be running in the cell.
The DFS daemon should be started in the following sequence:
1. dfskern
2. export
3. boserver
Note
This is only applicable if you need to start any of the DFS individually. If the
DFS daemons are started collectively (for example, using the start all option of
the MODIFY DFS command), DFS ensures that the correct starting sequence
is followed.
For example, to successfully start the boserver daemon, the dfskern daemon
must already be up and running.
Note
Make sure that the passwords of the DFS daemons are valid before starting
them up. If the passwords have expired, the daemons cannot be started.
Viewing the status of DFS daemons
You can query the status of the DFS daemons using the query option of the
MODIFY system command. You do not need the special privileges of a DCE or
DFS administrator or an operator to use the query option.
For example, to query the status of the dfskern daemon, enter the following:
MODIFY DFS, QUERY DFSKERN
A message about the status of the daemon will be written on the system log. This
message will also contain the process ID of the daemon.
The status of the daemon can be any of the following:
Table 36. Daemon’s status
Status
Description
READY
Indicates that the daemon is running, has been initialized, and is
ready to receive and process incoming requests.
ACTIVE
Indicates that a manual process is running. When an active
process stops, it is never considered an error and it is never
restarted automatically.
DCE/DFS
205
Status
Description
INITIALIZING
Indicates that the daemon has been started, but is not yet ready
to receive and process incoming requests.
STOPPING
Indicates that a request to stop the daemon has been received
and that the daemon is in the process of stopping.
DOWN
Indicates that the daemon is not active.
UNKNOWN
Indicates that the status of the daemon cannot be determined.
This can occur if the daemon was started, but no response was
received by the system indicating a change in its status. Note: You
can issue a command to stop a daemon if it is in the UNKNOWN
state.
The status of the DFS/MVS daemons controlled by the Control Task can be
queried by MVS operator commands. For example:
MODIFY DFS,QUERY ALL
MODIFY DFS,QUERY DFS
MODIFY DFS,QUERY BOSERVER
MODIFY DFS,QUERY BUTC01
The following is an example of a query command to the Control Task. The output
is sent to the MVS Operator’s console:
MODIFY DFS,QUERY DFSKERN
Output:
IOEP000221 DFS daemon DFSKERN status is >READY and process ID is 781
4.2.3 Daemon configuration file
The daemon configuration file is used by the Control Task to obtain necessary
information when starting the DFS daemons. The daemon configuration file
contains the following information:
• Parameters that are passed to the load module when a daemon is started,
called the argument list (including LE/370 run-time options).
• The Minimum Restart Interval. The Control Task attempts to restart a daemon
that ends abnormally only if the daemon was running for at least this time
interval. If a daemon ends during this time interval, it will not be restarted.
• The Time-out Period, which is the maximum time interval that the Control Task
waits for the daemon to complete its initialization after it has been started.
When this time interval elapses, and the Control Task has not received
confirmation from the daemon that initialization has completed, the status of
the daemon is set to unknown.
The pathname to the daemon configuration file is /opt/dfslocal/etc/ioepdcf.
206
Debugging UNIX Applications on OS/390
4.2.3.1 DFS administrative lists and groups
The administration of DFS in a cell or domain is regulated by DFS administrative
lists. An administrative list is a file that determines which system administrators
are allowed to issue commands that affect DFS server processes on server
machines. Each process is associated with an administrative list. Individual users
can be placed on an administrative list to grant them the privileges associated
with the list.
Groups of users can also be created and placed on an administrative list. A group
is a defined collection of users placed on an administrative list to grant the
administrative privileges associated with the list to all of the members of the
group simultaneously. The same group can be included on multiple administrative
lists; for a user to be extended the privileges associated with each of those lists,
the user needs only to be added to the proper group. This feature greatly
simplifies the responsibilities of the DFS system administrator.
For example, the Fileset Server on each file server machine has an administrative
list name admin.ft associated with it. Each user and each member of a group is
listed in the admin.ft file on that machine. A user can acquire the
administrative-level-commands that affect the Fileset Server on that machine. A
user can acquire the administrative privileges associated with the admin.ft file
either directly, by being added to the file or indirectly, by being added to a group
listed in the file.
For instance, you can create a group called domain1.admin and include it in the
administrative lists necessary to allow its members to administer data on the file
server machine in a single domain. You can then assign users to the
domain1.admin group to grant them administrative privileges on the file server
machine in the domain; you do not need to include each individual user in all of
the necessary administrative lists in the domain.
Similarly, you can create additional groups for other administrative tasks, such as
managing processes or installing new system binaries, and include the same or
different users in these groups. Users have only the privileges associated with the
administrative lists in which they are included. Unless users are also members of
other administrative lists in a domain or in a cell to which a domain belongs, their
membership is an administrative list on a machine that grants no additional
privileges beyond the scope of that administrative list in a domain.
The user performing a task must be included in the appropriate administrative
lists. Users can be included directly, by having their user names included in the
list, or they can be included indirectly, by being assigned to a group that is
included in the list; either method is sufficient.
Administrative lists are only one form of security used in DFS; DCE ACLs are also
used to limit access to files and directories. Many DFS operations require that
the issuer be included on the proper administrative lists and have the proper ACL
permissions.
4.2.3.2 DFS security
Several security measures are used to ensure that only valid users can access
files stored in the DFS filespace. When you correctly authenticate to a DCE cell,
you receive authentication information in the form of a ticket.
DCE/DFS
207
When you attempt to access data using a client machine, the Cache Manager
presents your ticket to the file server machine that houses the data. The Cache
Manager receives the data you requested when mutual authentication is
complete between the Cache Manager and the file server machine. Mutual
authentication is achieved when the two machines prove their identities to each
other. DFS requires this mutual authentication whenever a server and a client
communicate.
With DCE ACLs, you designate who can access the information in your files. An
ACL can exist for each directory and each file in DFS, specifying the actions that
different users can perform on the directory or file (ACLs exist only for DCE local
file system directories and files).
4.2.4 DFS daemon environment variable files
DFSCNTL and each DFS daemon has its own home directory. Each home
directory contains the environment variable file (ENVAR), where the environment
variables are set for each of the daemons.
The ENVAR files of the DFS daemons are in:
/opt/dfslocal/home/dfscntl/envar
/opt/dfslocal/home/dfskern/envar
/opt/dfslocal/home/dfsxport/envar
/opt/dfslocal/home/boserver/envar
/opt/dfslocal/home/butc01/envar
/opt/dfslocal/home/butc02/envar
/opt/dfslocal/home/butc03/envar
/opt/dfslocal/home/butc04/envar
/opt/dfslocal/home/butc05/envar
/opt/dfslocal/home/butc06/envar
/opt/dfslocal/home/butc07/envar
/opt/dfslocal/home/butc08/envar
/opt/dfslocal/home/ftserver/envar
/opt/dfslocal/home/rpserver/envar
/opt/dfslocal/home/upserver/envar
/opt/dfslocal/home/upclient/envar
You can customize the values of the environment variables in these files to suit
your operational needs.
4.3 Information sources
This section lists and provides a brief description of each publication in the
OS/390 Distributed File Service library. Also listed are publications from the
OS/390 DCE library that may be useful.
4.3.1 OS/390 Distributed File Service publications
This section lists and provides a brief description of each publication in the
OS/390 Distributed File Service (DFS) library.
4.3.1.1 Administration
OS/390 Distributed File Service Configuring and Getting Started, SC28-1722
208
Debugging UNIX Applications on OS/390
This book helps system and network administrators configure the OS/390
Distributed File Service.
OS/390 Distributed File Service Administration Guide and Reference, SC28-1720
This book introduces the DFS concepts to system and network administrators
and provides an in-depth understanding of DFS, its uses and benefits. This book
also provides reference information for the commands and files used by system
and network administrators to work with DFS.
4.3.1.2 Reference
OS/390 Distributed File Service Messages and Codes, SC28-1724
This book provides detailed explanations and recovery actions for the messages,
status codes, and exception codes issued by the OS/390 DFS.
4.3.2 OS/390 DCE publications
In this section we list and provide a brief description of each publication in the
OS/390 DCE library.
4.3.2.1 Overview
Distributed Computing Environment: Understanding the Concepts, GC09-1478
This book introduces Open Software Foundation (OSF) DCE. It describes the
technology components of DCE, from a high-level overview to a discussion of the
interdependencies among the components.
OS/390 DCE Introduction, GC28-1581
This book introduces OS/390 DCE. Whether you are a system manager, technical
planner, OS/390 system programmer, or application programmer, it will help you
understand DCE, and evaluate the uses and benefits of including OS/390 DCE as
part of your information processing environment.
4.3.2.2 Planning
OS/390 DCE Planning, SC28-1582
This book helps you plan for the organization and installation of OS/390 DCE. It
discusses the benefits of distributed computing in general, and describes how to
develop plans for a distributed system in an OS/390 DCE environment.
4.3.2.3 Administration
OS/390 DCE Configuring and Getting Started, SC28-1583
This book helps system and network administrators configure OS/390 DCE.
OS/390 DCE Administration Guide, SC28-1584
This book helps system and network administrators understand OS/390 DCE,
and tells how to administer it from the batch, TSO, and shell environments.
OS/390 DCE Command Reference, SC28-1585
This book provides reference information for the commands that system and
network administrators use to work with OS/390 DCE.
DCE/DFS
209
OS/390 OpenEdition DCE User’s Guide, SC28-1586
This book describes how to use OS/390 DCE to work with your user account, use
the directory service, work with namespaces, and change access to objects that
you own.
4.3.2.4 Application development
OS/390 DCE Application Development Guide: Introduction and Style, SC28-1587
This book assists you in designing, writing, compiling, linking, and running
distributed applications in OS/390 DCE.
OS/390 DCE Application Development Guide: Core Components, SC28-1588
This book assists programmers in developing applications using application
facilities, threads, remote procedure calls, distributed time service, and security
service.
OS/390 DCE Application Development Guide: Directory Services, SC28-1589
This book describes the OS/390 DCE directory services and assists
programmers in developing applications for the cell directory service and the
global directory service.
OS/390 DCE Application Development Reference Volumes 1 and 2, SC28-1590
This book explains the DCE Application Program Interfaces (APIs) that you can
use to write distributed applications on OS/390 DCE.
4.3.2.5 Reference
OS/390 DCE Messages and Codes, SC28-1591
This book provides detailed explanations and recovery actions for the messages,
status codes, and exception codes issued by OS/390 DCE.
4.3.3 Security Server publications
In this section we list and provide a brief description of books in the Security
Server library that may be needed for the DCE Security Server and for RACF
interoperability.
OS/390 OpenEdition DCE Security Server Overview, GC28-1938
This book describes the DCE security server and provides a road map for DCE
security server information in the OS/390 DCE library.
OS/390 Security Server (RACF) Security Administrator’s Guide, SC28-1915
This book explains RACF concepts and describes how to plan for and implement
RACF.
210
Debugging UNIX Applications on OS/390
Chapter 5. Web server—troubleshooting/hints/tips
In this chapter we cover explanations about error messages, possible pitfalls, and
some performance issues that are usually not found in any other books.
These pieces are results of worldwide experiences with the handling and
debugging either of the Domino Go Webserver or WebSphere Application Server.
5.1 Controlling your server using the console
The OS/390 MODIFY console command can be used to display statistics about
your server, turn on and off the very verbose trace, restart your server and turn
on and off SMF recording. The following figures are showing the use of the
command and all available parameters:
COMMAND INPUT ===> /F server_jobname,APPL=-D CONFIG
RESPONSE=SC63
IMW3501I Config: Hostname: wtsc63oe.itso.ibm.com, Port: 80, SSL Port:
443, Server root: /web/apple.
Trace is currently disabled.
SMF recording is currently disabled.
Figure 112. MODIFY command to display server configuration
COMMAND INPUT ===> /F server_jobname,APPL=-D STATS
RESPONSE=SC63
IMW3502I Stats: Threads running: 40, Threads idle: 40, Requests: 5,
Bytes rcvd: 3368, Bytes sent: 1947, Actv In Conns: 0, Actv Out Conns: 0
Figure 113. MODIFY command to display server statistics
COMMAND INPUT ===> /F server_jobname,APPL=-VV
RESPONSE=SC63
IMW3518I More trace enabled. (Very Verbose)
COMMAND INPUT ===> /F WEBAPPLE,APPL=-NODEBUG
RESPONSE=SC63
IMW3508I Debug has been disabled for all modules.
Figure 114. Turn on and off very verbose tracing
COMMAND INPUT ===> /F server_jobname,APPL=-RESTART
IMW3537I SA 234881038 0.0.0.0:80 * * RESTARTING
IMW3538I SA 234881038 0.0.0.0:80 * * RESTART SUCCESSFUL
Figure 115. Restarting the server
© Copyright IBM Corp. 1999
211
COMMAND INPUT ===> /F server_jobname,APPL=-??
RESPONSE=SC63
IMW3528I -d config : Server configuration
-d stats
: Server statistics
-debug <mod_name> : Enables trace
-? debug
: Lists mod_names for "-debug"
-nodebug <mod_name>: Disables trace
-? nodebug : Module names for "-nodebug"
-restart
: Restarts the Server
-smf <opt> : Enables SMF recording
-? smf
: Lists options for "-smf"
-nosmf <opt>: Disables SMF recording
-? nosmf
: Lists options for "-nosmf"
-v
: Verbose (trace all modules)
-vv
: VeryVerbose (more tracing)
-mtv
: MuchToVerbose (w/ trace stacks)
-vc
: VerboseCache (trace cache)
-version : Version of Server
Figure 116. MODIFY command to display all parameters
5.2 Tracing and logging
Trace and log issues are discussed in the following sections.
5.2.1 Tracing
5.2.1.1 Controlling the tracing level
For debugging any server or configuration problems the trace log is most useful.
It is written to the SYSOUT DD definition in the Web server procedure. If you
want to have it written into the HFS, you should code a TraceLog directive in your
httpd.conf file.
There is a function supplied with the Web server which allows you to control the
level of tracing. To activate it code the following in your httpd.conf file:
service /admin-bin/trace*
INTERNAL:TraceFn
To access the dialog, enter the following at your Web browser:
http://your.web.server:yourportnumber/admin-bin/trace
212
Debugging UNIX Applications on OS/390
Figure 117. Controlling the trace level
5.2.1.2 Starting the trace using the start procedure
There is another possibility to enable the very verbose trace using the -vv option
in the procedure to start the server. This looks like:
//WEBAPPLE PROC P1='-p 80',
// P2='-r /web/apple/httpd.conf',
// P3=' -vv',
// LEPARM='ENVAR("_CEE_ENVFILE=/web/apple/httpd.envvars")'
//WEBSRV
EXEC PGM=IMWHTTPD,REGION=0K,TIME=NOLIMIT,
//
PARM=('&LEPARM/&P1 &P2 &P3')
The output of the trace can be found in the OS/390 system log.
5.2.2 Language Environment tracing
Language Environment (LE) provides various tracing options for programmers to
use in optimizing parameters such as the sizes of heaps and stacks. In the Web
server proc, RPTSTG(ON) or RPTOPTS(ON) parameters cause LE tracing.
5.2.3 Logging
5.2.3.1 Using logs for problem determination
If you have a problem with your Internet application, it is always best to first check
your log and trace files to identify the cause of the problem.
The Domino Go Webserver can be configured to generate different logs in
varying formats. Each day at midnight, the server closes the logs for that day and
Web server—troubleshooting/hints/tips
213
creates new logs. Logs hold all the raw data that is used to generate all the
reports. See Domino Go Webserver Webmaster's Guide, SC31-8691, for more
information.
5.2.3.2 The following logs are generated
• The server logs activity in the access log files and stores them each night. At
midnight each night, the server closes the current access log and creates a
new access log file for the coming day. The access log contains entries for
page request mode to the server.
• The agent log indicates which Web browser was used to access a Web page.
• The referer log identifies the Web page that referred or linked to the requested
Web page. By default the server writes an entry to the agent and referer logs
each time a client sends the server a request.
• Cache access log: Logs all access requests for files served out of the proxy
server's cache.
• CGI error log: Logs standard error output (stderr) from Common Gateway
Interface (CGI) programs.
• Error log: Logs errors encountered by your server's clients, such as timing
out, authentication errors, and access problems to Web pages.
• Proxy access log: Logs access requests for files that come from a proxy
server.
• Trace log: Logs all trace data according to trace levels that are set when you
start your server. These trace settings should only be turned on to trace a
server problem since they will affect Web server performance. The trace log
path is defined using the TraceLog directive in the httpd.conf file. The trace
levels include:
• verbose (-v)
• very verbose (-vv)
• much too verbose (-mtv)
Check your httpd.conf file and the webmaster’s guide for a detailed explanation of
all the logs, how they can be enabled, and to where they will be written.
5.2.3.3 Additional system logs and traces
• The OS/390 System Log (SYSLOG) will contain all messages related to your
security product, such as RACF.
• The UNIX System Services syslogd daemon captures information on selected
TCP/IP servers.
• TCP/IP traces. For more information on this, see the OS/390 eNetwork
Communications Server IP Diagnosis Guide, SC31-8521.
• Component, Master, and System Trace are for your service representative.
• RACF GTF Trace if you suspect a RACF dirty environment. See 5.3, “Common
error messages” on page 216.
• SMF Data is useful if you are looking into performance and security access
issues. If you think somebody has tampered with your environment, you will
likely find the cause here if you are capturing all the relevant SMF records.
214
Debugging UNIX Applications on OS/390
5.2.3.4 More logs for the WebSphere Application Server
The WebSphere Application Server manager enables you to generate and view
two types of log files for servlet activity. To access the WebSphere Application
Server manager dialog enter the following URL: (port 9090 is the default port)
http://your.web.server:9090
The two types of log files are:
• Event log: contains a log of all servlets events
• Error log: lists errors related to servlets
Use the log files task in the manager dialog to define the settings for the servlet
error and events log files as shown in Figure 118.
Figure 118. WebSphere Application manager
The log file task IBM default names of the provided servlet log files are:
IMWebAS_rootlogs/servlet/servletservice/event_log
IMWebAS_rootlogs/servlet/servletservice/error_log
The application server supports two other types of logging:
• Native DLL logging: log messages produced by the Webserver C code before
entering Java.
• Java standard out logging. Any system.out and system.err prints go to this log.
Web server—troubleshooting/hints/tips
215
To control these types of logging, you must manually edit the jvm.properties file
and restart the server.
• Enable logging:
1. Set the property ncf.native.logison=true.
2. Set the property ncf.native.logfile=Websphere Application
server_root/logs/native.log.
3. Restart your server.
• Disable logging:
1. Set the property ncf.native.logison=false.
2. Restart your server.
To control Java standard out logging, edit the jvm.properties file.
• Enable logging to a file:
1. Set the property ncf.jvm.stdoutlog.enabled=true.
2. Set the property ncf.jvm.stdoutlog.file=true.
3. Set the property ncf.jvm.stdout.filename=Websphere Application
server_root/logs/ncf.log.
4. Restart your server.
• To enable logging to the Java console window:
1. Set the property ncf.jvm.stdoutlog.enabled=true.
2. Set the property ncf.jvm.stdoutlog.file=false.
3. Restart your server.
• To disable logging to the Java console window:
1. Set the property ncf.jvm.stdlogout.enabled=false.
2. Set the property ncf.jvm.stdoutlog.file=false.
3. Restart your server.
5.3 Common error messages
In this section we show the most common error messages in the Web server
environments and how to handle them. This information is mainly extracted from
http://www.software.ibm.com/webservers/httpservers/doc where other
troubleshooting information can be found.
5.3.1 02AF (address space dirty)
Some possible reasons for 02AF.
5.3.1.1 All users...
If a file system (HFS) is mounted with NOSETUID or Ignore SETUID, the program
control extended attribute will be ignored for any executable loaded from the file
system.
If you program control using ****** to point to the IPL volume, be sure that the
data sets exist on the one volume that is IPLed and not on any other volume that
216
Debugging UNIX Applications on OS/390
is used to holding program product data sets. For the ****** to work, the data sets
must exist on the one volume that was IPLed.
5.3.1.2 If using WebSphere Application Server...
Check to make sure the program control bit is on for the following files:
• /usr/lpp/WebSphere/AppServer/lib/libadpter.so
• /usr/lpp/WebSphere/AppServer/lib/libicsnativ.so
Check to make sure you have turned on program control for the HFS resident
JDK DLLs.
To turn on program control:
1. Give a superuser ID Read access to the BPX.FILEATTR.PROGCTL Facility to
enable this superuser ID to update the HFS file attributes using the extattr
command:
RDEFINE FACILITY BPX.FILEATTR.PROGCTL UACC(NONE)
PERMIT BPX.FILEATTR.PROGCTL CLASS(FACILITY) ID(superuserID) ACCESS(READ)
SETROPTS RACLIST(FACILITY) REFRESH
2. Go into the OMVS shell using the superuser ID (UID=0) that you just permitted
to the BPX.FILEATTR.PROGCTL class above and enter:
cd <JAVA_HOME>/lib/mvs/native_threads
extattr +p *.*
ls -E
For <JAVA_HOME> enter the path or the root directory for your JDK. Check to
make sure you have turned on program control for the C++ load library.
To turn on program control, issue the following RACF commands:
RALTER PROGRAM * ADDMEM('CBC.SCLBDLL'//NOPADCHK) UACC(READ)
SETROPTS WHEN(PROGRAM) REFRESH
5.3.1.3 Using ServletExpress support
When running V5.0 Lotus Domino Go Webserver with ServletExpress support,
and the Web server is configured with a user ID of %%CLIENT%% or a surrogate
ID in the httpd.conf configuration file, clients may receive an error 500 message.
Review the Web server trace log to determine if the following messages were
issued for the particular client request:
Failed access as Surrogate: , Errno: 139,
Errno2: 0be802af, Error: EDC5139I Operation not permitted.
IMW0241E Access denied - surrogateuser setup error.
When configured with %%CLIENT%%, the Errno2 value=090c02af.
Web server—troubleshooting/hints/tips
217
If these messages are generated for the failing client request, you must turn on
program control for the Java DLLs residing in the HFS. Instructions are
documented in APAR PQ18310 and are as follows:
1. Give a superuser ID Read access to the BPX.FILEATTR.PROGCTL Facility to
enable this superuser ID to update the HFS file attributes using the extattr
command:
RDEFINE FACILITY BPX.FILEATTR.PROGCTL UACC(NONE)
PERMIT BPX.FILEATTR.PROGCTL CLASS(FACILITY) ID(superuserID) ACCESS(READ)
SETROPTS RACLIST(FACILITY) REFRESH
2. Go into the OMVS shell using the superuser ID (UID=0) that you just permitted
to the BPX.FILEATTR.PROGCTL class above and enter:
cd JAVA_HOME/lib/mvs/native_threads
extattr +p *.*
ls -E
where JAVA_HOME is the path or the root directory for your JDK.
5.3.1.4 RACF considerations
Ensure all relevant data sets have been marked under RACF program control. All
of the following must be loaded from a program-controlled PDS:
SYS1.LINKLIB
pwapi.so
hlq.SIMWMOD1
hlq.SCEERUN
hlq.SCLBDLL
If running DB2, hlq.DSNLOAD and hlq.DSNEXIT
If using JDBC, hlq.CSSLIB
Any DLL loaded by a service class verb
The volume serial must be correct for these data sets.
The sequence of data sets in the LNKLST is important. If something is loaded
from an uncontrolled PDS that appears before a controlled PDS, the address
space will be dirty.
If you cannot identify the failing module, follow the instructions in RACF
Information APAR II08176. If you still cannot determine the cause of the problem,
follow the instructions in OS/390 UNIX Information APAR II10548.
5.3.1.5 Using another security product
Follow the instructions in OS/390 UNIX Information APAR II10548.
5.3.2 401 errors
When a password for an MVS user ID expires and that user ID attempts to
access the server, the server responds with a 401 pwexpired error. To change
the password using the Netscape browser:
1. Click Reload.
2. Enter the user ID, oldpassword/newpassword/newpassword, for the
password.
3. Select OK.
218
Debugging UNIX Applications on OS/390
4. Select Cancel. You will receive the 401 pwchanged Error Page because the
password has been changed successfully.
5. Click Reload.
6. Enter the user ID and new password. The Front Page should be displayed.
If you now receive the 401 pwnewneq error or the 401 pwnewinv error, you have
entered the verifying password (/newpassword/verifyingpassword) incorrectly, or
the password you entered is not a valid password.
5.3.3 Failed to load security module:
EDC5207S Load request for shared load module unsuccessful.
At initialization, the server attempts to load the North American Security DLL,
followed by the Export Security DLL. This message is issued if you do not have
one or both of these DLLs installed. If neither DLL is found, the server is started
as a base server without any security feature loaded.
5.3.4 "Forbidden" or "Unauthorized" error message
These error messages can indicate the following problems:
There could be an OS/390 UNIX setup problem. Ensure the access flags are
rwxr-xr-x for directories and rw-r--r-- for files. This includes the root ('/') directory.
The UID or GID values may not be correct. Try logging onto OS/390 UNIX using
the PUBLIC access control user ID and access files to the UID or GID values.
Check the console log or security product audit logs for protection problems.
5.3.5 IMW0161E with error 500
If the following message is accompanied by error 500, this indicates that the
helpout file cannot be executed because of a permission bit problem.
IMW0161E The script request is not valid. No variation of
/usr/lpp/internet/server_root/admin-bin/helpout is executable.
Check permission bits along the directory structure using the ls -dl command to
make sure that the execute bit is turned on.
The following example shows a valid set of permission bits:
MICHELW @ SC63:/>ls -dl /
drwxr-xr-x 19 OMVSKERN TTY
8192 Jun 23 14:59 /
MICHELW @ SC63:/>ls -dl /usr
drwxr-xr-x 10 AK
OMVSGRP
8192 Jan 26 11:48 /usr
MICHELW @ SC63:/>ls -dl /usr/lpp
drwxr-xr-x 37 OMVSKERN OMVSGRP
8192 Jun 21 10:29 /usr/lpp
MICHELW @ SC63:/>ls -dl /usr/lpp/internet
drwxr-xr-x
9 OMVSKERN OMVSGRP
8192 Jun 18 14:42 /usr/lpp/internet
MICHELW @ SC63:/>ls -dl /usr/lpp/internet/server_root
drwxr-xr-x 15 WEBADM IMWEB 8192 Jan 26 11:51 /usr/lpp/internet/server_root
Web server—troubleshooting/hints/tips
219
MICHELW @ SC63:/>ls -dl /usr/lpp/internet/server_root/admin-bin
drwxr-xr-x 4 WEBADM IMWEB 8192 Apr 2 13:07
/usr/lpp/internet/server_root/admin-bin
MICHELW @ SC63:/>ls -dl /usr/lpp/internet/server_root/admin-bin/helpout
-rwx-----2 WEBADM
IMWEB
258048 Jan 27 21:13
/usr/lpp/internet/server_root/admin-bin/helpout
5.3.6 Error using message catalog
If you run the Web usage mining command and see the error message, Error
using message catalog, this usually indicates that the NLSPATH setting in your
.profile does not match the setting in the server's httpd.envvars file.
To change environment variable settings in your .profile, use the export
command, for example:
export NLSPATH=setting
5.3.7 Setup.sh error messages
If you have run setup.sh more than once, you will receive the following messages
that can be ignored:
ln: FSUM6245 link to target "/usr/lpp/internet/bin/mvsds.so" failed: EDC5117I
file exists.
This message is received because you are trying to create an external link that
has been previously created.
chown: FSUM6180 file "/usr/lpp/internet/bin/wwwss.so": EDC5129I No such file or
directory.
This message is received because you are trying to change ownership in a file
that does not exist. There is no file associated with wwwss.so, only an external
link to an MVS data set.
5.3.8 ID and password prompt when accessing at a nonstandard port
If you access the HTTP Server at a nonstandard port number (such as port
8400), you are prompted for a user ID and password to access the configuration
and administration forms. If you keep the same browser up and access the same
server at default port 80, you do not get prompted for a user ID and password.
This is a browser problem.
If you started with a new browser, then access the server on port 80; you will be
prompted for the user ID and password.
220
Debugging UNIX Applications on OS/390
5.3.9 IMW messages
This section contains information about how to read IMW messages.
5.3.9.1 Message ranges
This table shows the relationship between the server's components and their
corresponding message ID ranges:
Table 37. Message ranges for the server components
Message ID Range
Component
IMW0001–IMW2000
IMWHTTPD messages
IMW2001–IMW2500
Proxy Server messages
IMW3501–IMW3700
CONSOLE messages
IMW3701–IMW3999
HTCounter Program messages
IMW4000–IMW5000
HTIMAGE messages
IMW5001–IMW6000
HTADM messages
IMW6001–IMW6100
CERTREQ Security messages
IMW6101–IMW6300
CERTUTIL Security messages
IMW6401–IMW6600
HTSHTTPD Security messages
IMW6801–IMW6900
SSL Security messages
5.3.9.2 Process descriptor
Messages IMW3536I through IMW3542E contain a process_descriptor, which
includes mode processid ServerIPA:ServerPort serverSN serverAE
Where mode is one of the following:
• SA—the server started as a stand-alone process (not managed by WLM)
• QM—the server started as a WLM Queue Manager
• QS—the server started as a WLM Queue Server
The server's process identifier as assigned by UNIX System Services is
processid.
ServerIPA is the server's IP address, in the form n.n.n.n, where n is from 0 to
255. If ServerIPA is 0.0.0.0, then the server is not bound to any specific IP
address (if neither the BindSpecific directive nor the -HN start parameter was
used).
ServerPort is the primary port number that the server is listening on.
serverSN is the WLM subsystem name. If mode indicates SA, then serverSN will
be '*'. Otherwise, it reflects the value passed as the -SN start parameter.
serverAE is the WLM Application Environment name. If mode indicates SA or
QM, then serverAE will be '*'. Otherwise, it reflects the value passed as the -AE
start parameter.
Web server—troubleshooting/hints/tips
221
5.3.10 How to read a message
How can an IMW message be interpreted? This is shown in the following
example:
+0600.: Failed on attach of shared memory, errno=111
+0600.: IMW0264E __errno2() =7170303
Explanations:
• errno=111: This errno is a USS return code and can be found in the OS/390
UNIX System Services Messages and Codes, SC28-1908. The codes
described there are represented in hex and decimal values. So look for "111"
decimal) or "006F" (hex) in the book and you will find:
111
006F
EACCES
Permission is denied.
• IMW0264E: For this message look at the Web server message manual, for
example Domino Go Webserver Messages, SC31-8692, and you will find:
IMW0264E errno2=hex number
Explanation: This is additional, OpenEdition MVS specific error information
associated with a previous message.
User Response: Contact the IBM Software Support Center.
• errno2() =7170303: This errno is a USS reason code and can be found in
OS/390 UNIX System Services Messages and Codes, SC28-1908. It is
divided into two parts:
Table 38. USS errno
cccc
rrrr
0717
0303
• cccc is a halfword reason code qualifier
• rrrr is a halfword reason code described in the message manual appendix
The two high-order bytes of the reason codes returned by OS/390 UNIX
contains a value that is used to qualify the contents of the two low-order bytes.
If the contents of the two high-order bytes are within the range of 0000 to
X'20FF', the error represented by the reason code is defined by OS/390 UNIX.
If the contents of the two high-order bytes are outside the range, the error
represented by the reason code is not an OS/390 UNIX reason code.
Cause 0717 is within the range of 0000 to X'20FF', 0303 is an OS/390 UNIX
reason code. Looking for 0303 in the message manual results in:
0303 JRIpcDenied
Access was denied because the caller does not have the correct permission.
Action: The caller lacked ownership, read or alter permission.
Note
Look at the MVS system log in case of errors and problems in the Web server
environment because in a lot of cases very useful information can be found
there.
222
Debugging UNIX Applications on OS/390
5.4 Tuning hints and tips
In this section we cover actual performance and tuning hints and tips that can be
found mainly on Web sites.
5.4.1 Performance hints when running Web server under USS shell
Production Web servers should be configured as OS/390 Started Tasks. It is
difficult to achieve the desired performance characteristics when running the Web
server from the OS/390 UNIX shell.
If you choose to execute the Web server from a UNIX shell, the Web server will
probably run in an S/390 UNIX forked address space. To guarantee that the Web
server runs in its own address space, run it as a background shell job.
To run as a background job, enter the name of the program followed by the
parameters and then the character "&", for example, httpd parms &. The Web
server will execute at the dispatching priority assigned to forked address spaces.
These are designated as SUBSYS=OMVS in your IEAICSxx member of
SYS1.PARMLIB or WLM policy.
Generally, dispatch priority assigned to this type of work is much too low for
production Web servers, although it may be adequate for testing.
Normally TSO, TELNET and forked address spaces have three performance
periods defined, with the third period having low dispatching priority. After a few
seconds of execution, the Web server will fall into the third performance period
and remain at low priority for the duration of its life.
Make sure that you assign a high enough priority to the address space that
initiates the Web server and to the resulting forked address space. This can be
accomplished by adding additional qualifiers such as the USERID or ACCTINFO
keywords to the IEAICSxx or WLM policy and assigning a performance group or
service class with the desired priority.
For example, if you are running in compatibility mode, your IEACICSxx might
have an entry similar to the following:
SUBSYS=OMVS,PGN=99
USERID=WEBSERV
And your IEAIPSxx might have a corresponding entry similar to the following.
Note that when selecting a dispatching priority it is important to consider the
relationship to other kinds of work. The relative priority is more important than the
absolute value you have selected. The example below is for illustration and is not
meant to suggest that you must run at priority F82:
PGN=99,(DMN=99,DP=F82)
If you are executing your Web server in Scalable Server mode (WLM support
enabled), you will probably want to modify the classification rules for the OMVS
service class.
In addition to SRM and WLM parameters, if the user entered the shell using rlogin
or telnet, the performance of the Web server running under the shell will be
subject to certain limitations specified in the BPXPRMxx member of
Web server—troubleshooting/hints/tips
223
SYS1.PARMLIB. These are global values. Increasing them will allow all OS/390
UNIX processes to increase their use of these resources.
• MAXASSIZE(nnnnn)
Specifies the RLIMIT_AS resource values that a process receives when it is
identified as a process.
RLIMIT_AS indicates the address space region size.
• MAXCPUTIME(nnnnn)
Specifies the RLIMIT_CPU resource values that a process receives when it is
identified as a process. RLIMIT_CPU indicates the CPU time, in seconds, that
a process can use.
• MAXTHREADS(nnnnnn)
Specifies the maximum number of pthread_created threads, including running,
queued, and exited but undetached, that a single process can have
concurrently active. A value of 0 prevents applications from using
pthread_create.
A Web server started by a TSO user who entered the UNIX shell by running the
OMVS command, will inherit the TSO address space REGION and TIME limits
specified in the TSO LOGON PROC.
To provide optimal support for running the Web server from the shell, the region
and time limits need to be set to unlimited and the task and thread limits may
need to be raised. This can be accomplished in two ways:
1. Write a C or C++ program that calls the following services:
• setrlimit to change the region size to unlimited
• setrlimit to change the time to unlimited
• set_thread_limit_np to set thread and task limits
These services are described in OS/390 C/C++ Run-Time Library Reference,
SC28-1663.
Execute this program before invoking the Web server. Note that you need to
be a superuser to change these limits to higher values. This would imply that
this program would need to be a setuid program that runs with UID=0. This
requirement is also present for solution 2.
2. Set the system thread and task limits as appropriate for the Web server and
write a program to invoke __spawn2(). The call to __spawn2() should set the
following:
• Region size unlimited
• Time limit unlimited
• Jobname
These settings are described in OS/390 C/C++ Run-Time Library Reference,
SC28-1663.
224
Debugging UNIX Applications on OS/390
5.4.2 Tuning your Web server for better performance
In this section we describe tuning hints and tips that should improve your Web
server's performance. Information has been gathered under testing conditions or
from Web server users. Results in your environment may vary.
5.4.2.1 Performance settings
Each time your server receives a request from a client, it uses one or two threads
to perform the requested action. One thread is used if the server is not performing
DNS lookup; two threads, if the server is performing DNS lookup. If no threads
are available, the server holds requests until more threads are available. The
MaxActiveThreads setting in your configuration file specifies the maximum
number of active threads.
If your server is running at maximum capacity on a sustained, non-stop basis, the
amount of virtual memory used increases. This increase is temporary and is
alleviated as the number of requests decline and the server catches up on
serving requests.
Hints and tips
• You can lower the amount of virtual memory used by lowering the maximum
number of active threads setting. A good starting point is half of your current
MaxActiveThreads setting. Keep in mind when you are lowering this value
that, when no threads are available, the server holds requests until more
threads are available.
• You can use the ServerPriority setting to override the OS/390 UNIX default
priority that your operating system gives to the Web server.
• If your server is running too slowly, it could be related to any of the following
factors:
- Your network speed
- The traffic on your LAN
- The number of clients that are sending requests to your server
- The number of threads set on your server
- You specified -v or -vv tracing
• V5.1 Web server: Use the Fast Response Cache Accelerator for dynamic
caching. The EnableFRCA setting is used to turn this caching function on. If
you are not caching files with the Cache Accelerator, use the CacheLocalFile
setting to load your most popular files into the server's memory at startup time.
You can specify the maximum amount of memory and the maximum number
of files for caching with the CacheLocalMaxBytes and CacheLocalMaxFiles
settings.
• Use the PersistTimeout and the MaxPersistRequest settings to specify the
characteristics of a persistent connection. A persistent connection allows the
server to accept multiple requests and to send responses over the same
TCP/IP connection. Overall throughput is increased because the server does
not have to establish a separate TCP/IP connection for each request and
response. Also, the TCP/IP connection is used more efficiently because a
client can make multiple requests without waiting for the response to each
request.
Web server—troubleshooting/hints/tips
225
• The following defaults have been set for better performance. You should
modify these directives only if you want the additional function they provide:
- DNS-Lookup off
- LiveLocalCache off
- UseMetaFiles off
- DirAccess off
- imbeds off
5.4.2.2 Changing performance settings on the Web server
You can change your server's performance settings by using the configuration
and administration forms or by editing your server configuration file directly:
• To use the configuration and administration forms, click Configuration and
Administration Forms from the Front Page of your server and use the
System Management forms.
V5.1 Web server: If using the Fast Response Cache Accelerator, use those
forms to change Fast Response Cache Accelerator settings.
• To edit the configuration file, open the httpd.conf file and update the
appropriate directives.
You must restart your server to use the new settings.
5.4.2.3 Cache management guidelines
In this section we describe caching options that can help you optimize your
server's performance.
Use the Fast Response Cache Accelerator (V5.1 Web server)
• Overview
The Fast Response Cache Accelerator can improve the performance of the
Web server when serving text and image files; dynamic content and protected
pages are not cached.
Because the Cache Accelerator cache is automatically loaded during server
operation, you are not required to list the files to be cached in your server
configuration file. In addition, the server will automatically recache changed
pages and remove outdated pages from the cache.
The Cache Accelerator provides support for caching on multiple Web servers
and on servers with multiple IP addresses. Currently, support is not available
for running the Cache Accelerator on a proxy server.
• Planning considerations
• WLM access
Before using the Cache Accelerator, ensure that the Web server has
access to Workload Management.
• Unique subsystem name for WLM
If you are not using the -SN parameter when starting the server, you must
specify a unique subsystem name for the Cache Accelerator when you
configure dynamic caching.
226
Debugging UNIX Applications on OS/390
• TCP/IP stack name
If you are using common INET and multiple TCP/IP stacks, identify the
name of the TCP/IP stack which supports the Cache Accelerator. This
name must match the name on the SubFileSysType statement in the
OS/390 UNIX BPXPRMxx parmlib member.
• Enabling and configuring the Cache Accelerator
By default, the Cache Accelerator is not enabled. You can enable and
configure the Cache Accelerator by using the configuration and administration
forms or by editing your server configuration file directly:
• To use the configuration and administration forms:
1. From the Front Page of your server, click Configuration and
Administration Forms.
2. Click Fast Response Cache Accelerator.
3. Click Enable Fast Response Cache Accelerator to turn dynamic
caching on.
4. Configure the Cache Accelerator using the appropriate options. Refer to
the online help for more information on each option.
5. Restart the server to use the new settings.
• To edit the configuration file:
1. Open the httpd.conf configuration file.
2. Set the EnableFRCA directive on.
3. To configure the Cache Accelerator, use the FRCA directives.
• Monitoring and managing the Cache Accelerator
Two resources for monitoring and managing the Cache Accelerator are Resource
Measurement Facility (RMF) reports and output from the DISPLAY TCPIP command.
• RMF reports
The WLM enclave created for the Cache Accelerator is a scheduled SRB.
Therefore, to manage the Cache Accelerator workload, you need one
service class period that has an execution velocity goal. The RMF report
will show no transactions, and CPU information will be included under TCB
CPU consumption.
For more information on RMF, see the RMF User's Guide.
• DISPLAY TCP/IP command
To display information on the Cache Accelerator enter:
D TCPIP,FRCA_stack_name,NET,CACH
For FRCA_stack_name, specify the name of the OS/390 UNIX physical file
system that supports the TCP/IP stack used by the Cache Accelerator, for
example, TCPV34. This is the name that appears on the FRCAStackName
directive in the server configuration file.
Web server—troubleshooting/hints/tips
227
Sample output:
D TCPIP,TCPV34,NET,CACH
EZZ2500I NETSTAT CS V2R7 TCPV34 324
CLIENT: WEBSRV3 LISTENING SOCKET: 0.0.0.0..8137
MAXCACHESIZE:
0000002500 CURRCACHESIZE:
MAXNUMOBJECTS:
0000000200 CURRNUMOBJECTS:
NUMCONNS:
0000002116 CONNSPROCESSED:
CONNSDEFERRED:
0000000540 CONNSTIMEDOUT:
REQUESTSPROCESSED: 0000002116 INCOMPLETEREQUESTS:
NUMCACHEHITS:
0000001576 NUMCACHEMISSES:
NUMUNPRODCACHEHITS: 0000001576
1 OF 1 RECORDS DISPLAYED
0000000120
0000000045
0000001576
0000000000
0000000000
0000000540
Figure 119. Command output: D TCPIP,TCPV34,NET,CACH
For more information on DISPLAY TCPIP output fields, see the TCP/IP OE
User's Guide.
Place frequently referenced files in memory cache
Note
If you are using the Fast Response Cache Accelerator for dynamic caching,
you do not need to use the tips described in this section.
When the Web server starts, it places specified files in a memory cache. Then,
when a file is requested it can be retrieved from the cache, rather than from the
disk
Hints and tips
• When specifying the files you want cached, you might also need to increase
the default maximum bytes and maximum files limits. To increase the default
maximum limits, use the CacheLocalMaxBytes and CacheLocalMaxFiles
settings.
• To specify which files are cached, edit the configuration file by adding more
CacheLocalFile statements.
Example:
CacheLocalFile /usr/lpp/internet/server_root/pub/file1k.html
You can specify an asterisk (*) for some parts of the data set name. For
example, to apply CacheLocalFile to all the html files in the ../cachedir/
directory, specify:
CacheLocalFile /usr/lpp/internet/server_root/cachedir/*.html
If you have a lot of different directories containing files that you want to cache,
the following technique might be useful. An example is the SPECweb
workload. You might want to cache all class 0, 1, and 2 files because they are
smaller and are referenced more frequently, while you might not be able to
cache the larger class 3 files due to memory limits. To cache class 0, 1, and 2
files for SPECweb, you would want something similar to:
# cache all class 0 files
CacheLocalFile /specdata/file_set/dir0/class0*
CacheLocalFile /specdata/file_set/dir1/class0*
228
Debugging UNIX Applications on OS/390
# in "dir0"
# in "dir1"
...
CacheLocalFile /specdata/file_set/dir117/class0*
# cache all class 1 files
CacheLocalFile /specdata/file_set/dir0/class1*
CacheLocalFile /specdata/file_set/dir1/class1*
...
CacheLocalFile /specdata/file_set/dir117/class1*
#cache all class 2 files
CacheLocalFile /specdata/file_set/dir0/class2*
CacheLocalFile /specdata/file_set/dir1/class2*
...
CacheLocalFile /specdata/file_set/dir117/class2*
# in "dir117"
# in "dir0"
# in "dir1"
# in "dir117"
# in "dir0"
# in "dir1"
# in "dir117"
Suppress time stamp check for memory cached files
With each reference to a file in the memory cache, the Web server can check the
time stamp of the file on the disk to make sure the cached copy is current.
Suppress this time stamp check for files in the memory cache.
5.4.2.4 RACF and other SAF-based security products
For controlling access to Web resources on MVS, the Web server provides
extensive access control support, including system validation of user IDs and
passwords and access control through surrogate user ID support.
Resource Access Control Facility (RACF), or an equivalent SAF-based security
product, manages system and data security by verifying a user's identity and
access to a resource.
In general, SAF security checking with RACF, or another vendor's SAF-based
security product, is recommended because it can protect the installation from
unauthorized access to MVS as well as Web server resources.
Users are identified by an OS/390 UNIX user ID (alphanumeric) kept in the RACF
user profile, and an OS/390 UNIX group ID (GID) kept in the RACF group profile.
5.4.2.5 RACF performance suggestions
If using RACF or another SAF-based security product, consider the following
hints and tips:
• Use surrogate user IDs, if that is acceptable from a security point of view.
• If you use %%SAF%% with the PUBLIC surrogate user ID, this provides
protection of your server resources without a high CPU cost. OS/390 Release
3 or greater automatically enhances performance when you use surrogate
user IDs and RACF.
• If you require Web clients to have a unique SAF-based user ID and password
on the Web server system, specifying %%SAF%% with %%CLIENT%% will
give you more robust security but at a higher CPU cost.
• Put your RACF data set on a control unit with caching and the DASD fast write
feature. Instructions for enabling caching are in the documentation for
DFSMS. Contact your DASD support programmer.
• Cache highly used pages.
• If running Workload Management (WLM), make sure you permit WLM to
RACF.
Web server—troubleshooting/hints/tips
229
5.4.2.6 SSL performance hints and tips
Secure Web server operation takes more CPU and elapsed time; use only if
required.
A fix for V5.0 of the Web server APAR PQ19981 corrected serialization problems
when the Web server was executing large numbers of concurrent SSL
handshakes. Tests indicated that when this fix was installed, the CPU time for an
SSL handshake was reduced up to 30% and the throughput of SSL requests was
increased up to 500% compared to V5.0 without the fix.
5.4.2.7 SSL and hardware encryption
Beginning with V5.0 of the Web server, you can designate a preference order for
cipher specifications using the SSLCipherSpec directive. This enables you to
configure the Web server to perform DES and Triple-DES encryption using
hardware encryption. For information on enabling this support on the Web server,
see the Web server documentation.
Tests indicated that some workloads did not exhibit performance benefits when
hardware encryption was used for securing the data in an SSL session. This may
be true of workloads that do not have persistent SSL sessions and contain many
small files. Installing V5.0 (with fixes for APARs PQ19981 and PQ22108) or later
releases of the Web server will greatly enhance the benefit of cryptographic
hardware exploitation.
It is the handshake phase of SSL that makes use of digital certificates and
performs asymmetric (private key) decryption.
The V5.0 fix for APAR PQ22108 enables hardware encryption for these SSL
private key operations. The PTFs also include additional SSL performance items.
The PTF numbers for this APAR are UQ25474 (base Web server) and UQ25475,
UQ25476, UQ25477 for the U.S., export, and France Web servers. In conjunction
with these Web server PTFs, you will also require the fix for OS/390 APAR
OW36159.
Tests showed up to a 100% throughput increase with the above V5.0 PTF fixes
(compared with APAR PQ19981) and up to 80% CPU time reduction for SSL
handshakes with hardware cryptography enabled.
More information on SSL performance can be found on the OS/390 UNIX System
Services Web site (http://www1.s390.ibm.com/unix/).
5.4.2.8 Logging guidelines
In the following section we show how performance can be improved by avoiding
log events that are not needed.
Turn access logging off
By default the Web server records that a Web browser has accessed a file in a
server disk log. If you do not need this access logging, the Web server performs
better if the logging is off. To set logging off, edit the httpd.conf configuration file
and insert a comment character (#) at the beginning of the AccessLog line.
If you are using the Fast Response Cache Accelerator for dynamic caching, you
can turn access logging off but use the FRCA Access log to log only requests
served from the Cache Accelerator cache.
230
Debugging UNIX Applications on OS/390
Suppress access logging from selected browsers
If you must log accesses for some cases, the Web server performs better if you
suppress logging for those cases where you can. There are several methods for
configuring which accesses should not be logged. For example, to suppress
logging from all browsers with IP addresses 9.*.*.* using the configuration and
administration forms:
1. Click Logging and Reporting.
2. Click Access Log File Configuration.
3. For Exclusions from the Access log, enter 9.*.*.* in the Host name or IP
address field.
Turn agent logging off
To turn agent logging off, edit the configuration file and insert a comment
character (#) at the beginning of the AgentLog line.
Turn referer logging off
To turn referer logging off, edit the configuration file and insert a comment
character (#) at the beginning of the RefererLog line.
5.4.2.9 Resource mapping templates
When the Web server begins processing a request, it searches the directives
specified in the configuration file. This search time can be reduced for specific
URL subdirectories by placing corresponding Pass directives closer to the top of
the configuration file.
Using the workload SPECweb for example, you could place the following
directive just before user authentication and document protection directives:
Pass
/file_set/*
/specdata/file_set/*
Then when a request comes in for http://server_name/file_set/.. (from the first
parameter of the above Pass directive), the Web server will get to the operating
system files, /specdata/file_set/* (from the second parameter of the above Pass
directive), more quickly. This occurs because the Pass statement is placed above
the authentication directives so the server saves some authentication processing.
SPECweb does not require authentication of its files.
Be cautious with this approach. For example, if a general Pass directive, such as
Pass /*, is moved to the top, then subsequent, more specific directives will be
ignored.
To reduce the search time, edit the configuration file and locate the appropriate
Pass statements.
5.4.2.10 Additional guidelines
Here are additional performance issues.
Increase listen backlog
In the Web server proc, -lb n sets the Listen Backlog to n. Also, SOMAXCONN in
the TCP/IP profile member must be greater than n because it sets an upper limit
for the -lb Web server parameter. We used -lb 50, and SOMAXCONN 1024.
Web server—troubleshooting/hints/tips
231
Increase MAXFILEPROCS
Change MAXFILEPROCS from 64 to 65535 by editing
SYS1.PARMLIB(BPXPRMxx).
Put recommended LE modules in LPA
For LE 1.6, see "Appendix F Modules Eligible for the Link Pack Area" in OS/390
Language Environment for OS/390 and VM Run-Time Migration Guide,
SC28-1944.
Adjust TCP/IP dispatching priority
Set the dispatch priority of TCP/IP equal to one less than VTAM with nothing else
(like JES) in between TCP/IP and VTAM.
If you are not using WLM (that is, your system executes in WLM compatibility
mode), you can specify dispatching priority in the IEAIPSxx member of
SYS1.PARMLIB. You should create a performance group for the Web server.
Then use the IEAICSxx member of PARMLIB to assign the Web server to this
performance group. If your system runs in WLM GOAL mode, you should specify
a velocity goal for the Web server that is lower than the VTAM velocity goal.
TCP/IP packet sizes >= 4 KB. The Web server uses a block size of 4 KB for TCP
transmission. Make sure your TCP/IP parms allow >= 4 KB packet sizes.
Store HTML files in ASCII format
If possible store your HTML files in ASCII format and name the files with a suffix
of .ascii. Your users should specify a URL with the fully qualified name, including
the .ascii suffix. If users do not include the suffix, expensive multiformat
processing will be done by the Web server.
During the caching process, the Web server does not convert data to ASCII.
Therefore, there is a performance benefit to storing them in ASCII. You need to
decide if the performance benefit outweighs the convenience of having the files in
EBCDIC so you can more easily edit the files on OS/390.
You can use the utility EDCICONV to convert your text files to ASCII format.
Minimize directory depth
The Web server uses the file system when accessing a file. If the Pass directive
being used for a particular file has a deep operating system directory structure,
the file system opens and checks security at each level each time the file is
accessed. Reduce this processing by placing the files in directories with a
shallow directory structure. This will not help files that are already covered by
LiveLocalCache off because the file system is not consulted.
To reduce the file system subdirectory searching, place files in directories that
are closer to the root directory.
Optimize directory listings
If your Web server is returning directory listings, avoid some of the slower
options. Edit the configuration file and set the following directives off:
• DirReadme
• DirShowDate
• DirShowIcons
232
Debugging UNIX Applications on OS/390
• DirShowDescription
Shorten the persistent timeout
Edit the configuration file and reduce the value of the PersistTimeout directive
from 1 minute to 10 seconds.
5.5 Tips for enabling WebSphere Application Server under OS/390
The following information is intended to help you implement servlet support. The
information in this section applies only to WebSphere Application Server.
First, some background information. WebSphere Application Server is not a Web
server in itself. To use WebSphere Application Server, a Web server must also
be running. Under OS/390 R5 and R6, the only Web server that supports
WebSphere Application Server is Domino Go Webserver (DGW) R5.
WebSphere Application Server is intended to replace ServletExpress. Under
OS/390 R5 and R6, WebSphere Application Server is nearly identical to
ServletExpress, except that it has corrections integrated into it that are provided
as PTFs for ServletExpress, along with other coding improvements.
If WebSphere Application Server is to be installed on a system that already has
ServletExpress installed, then ServletExpress can be removed from the system
by using the jobs that are supplied in the WebSphere Application Server
package. WebSphere Application Server also includes a job that migrates the
ServletExpress configuration files to WebSphere Application Server files.
This information is intended to supplement the basic installation and verification
instructions provided by:
• WebSphere Application Server Program Directory
http://www.software.ibm.com/webservers/appserv/hejsw10.htm
• WebSphere Application Server for OS/390 Getting Started
http://www.software.ibm.com/webservers/appserv/doc/os390/os1gsldw.htm
• WebSphere Application Server Checklist
http://www.software.ibm.com/webservers/httpservers/doc/v51/twasck.htm
1. Use JDK 1.1.6 in conjunction with WebSphere Application Server.
Do not use earlier releases of the JDK.
You can find out your JDK level by entering the following command at an
OMVS shell command prompt:
java -fullversion
2. Ensure that all required JDK, DGW, WebSphere Application Server, and
Language Environment (LE) PTFs are installed. The required PTFs are
subject to change at any time. The following search words can be used to
obtain lists of APARs and PTFs for the Web server products:
DGW 5.0 base
(for OS/390 R5, R6)
"5697D4300 500"
WebSphere Application Server
for DGW 5.0 under OS/390 R5, R6
"5697D4300 W10"
Web server—troubleshooting/hints/tips
233
(must be downloaded from web)
JDK 1.1.6 under OS/390 R4, R5
"5752MJAVA 11C"
JDK 1.1.6 under OS/390 R6
"5752MJAVA 11D"
In addition, ensure that the PTFs (or the appropriate PTFs that supersede
them) listed in the DGW Program Directory and listed in the PSP bucket
(orderable from IBM software support) are installed. Do not overlook the LE
PTFs that are listed in the Program Directory.
The WebSphere Application Server Program Directory specifies that PTFs
UQ23043 and UW50654 are required. Do not disregard this requirement. It is
not possible to even perform the SMP/E installation of the WebSphere
Application Server without these PTFs. In addition, we recommend that PTF
UQ27624 be applied.
PTF UQ27624 is mandatory for environments in which the OS/390 FTP client
is not available (such as when using a non-IBM TCP/IP suite). In addition to
correcting allocation problems, PTF UQ27624 allows the WebSphere
Application Server distribution file to be unpacked directly on to the OS/390
system by replacing the "ftp" command in the IMWJWEB job with a "file"
command. For example, if the ejsw10.bin file is copied to a sequential data
set called SMITH.EJSW10.BIN, then the IMWJWEB job can be used to
unpack the data set directly on the OS/390 system by using the following
command format in the IMWJWEB job:
rfget file://SMITH.EJWS10.BIN +
WEBAS +
volume(MVSCA2) unit(3390)
The above example results in the rel files being placed under the HLQ of
WEBAS on volume MVSCA2 on unit type 3390.
3. Ensure that the host name of the OS/390 system is either registered on the
site's domain name server, or listed in the appropriate hosts file (such as
/etc/hosts). If a domain name server is being used, ensure that both an "A"
record and a "PTR" record are defined on the domain name server for the
OS/390 system host name. Servlet support performs a reverse DNS (or host
table) lookup during initialization, and if the IP address of the OS/390 system
cannot be resolved back to a valid host name, servlet initialization will fail.
Also ensure that the /etc/resolv.conf file is properly configured with a "domain"
directive and at least one "nameserver" directive.
4. The WebSphere Manager applet provides a graphical interface that allows
many servlet-related parameters to be modified. The actual parameters that
control operation of WebSphere Application Server are located in the files that
are in the directory:
/usr/lpp/WebSphere/AppServer/properties /server/servlet/servletservice
Generally, you can use the WebSphere Manager applet (which is accessed by
connecting to port 9090 of the Web server host) to update servlet-related
parameters. Be aware, however, that it is possible to make changes to the
WebSphere Application Server control files that will then prevent WebSphere
Application Server from operating. If this happens, then you will be unable to
correct the problem by running the WebSphere Manager applet, because it
will not be accessible (a "catch-22" situation). Therefore, we recommend that
you make a backup copy of the WebSphere Application Server parameter files
234
Debugging UNIX Applications on OS/390
before running the WebSphere Manager. This can be done by entering the
following commands at your OMVS shell prompt:
cd /usr/lpp/WebSphere/AppServer/properties/server/servlet
cp -R servletservice servletservice.backup
5. Modify the jvm.properties file to increase the number of threads available to
the JVM. The jvm.properties file is located in the directory:
/usr/lpp/WebSphere/AppServer/properties/server/servlet/servletservice
6. In the httpd.conf file, change:
MaxActiveThreads=100 (at least)
7. In the SYS1.PARMLIB(BPXPRMxx) member, change:
MAXTHREADTASKS(5000)
MAXTHREADS(10000)
(at least greater than the combination of
ncf.jvm.threads.max + MaxActiveThreads)
(at least = to MAXTHREADTASKS x 2)
8. If starting the Web server using JCL, specify on the PARM= statement:
PARM='STAC(200k),STOR(,,,8k),LIBS(1k,1k),BE(400k,50k,keep)/'
Note
DGW parms can be added after the '/'. LE parms are specified prior to the
'/'.
If starting httpd from an OMVS shell session, add the following to the
/etc/profile or your own .profile file:
_CEE_RUNOPTS="STACK(200k),STORAGE(,,,8k),LIBSTACK(1k,1k),
BELOWHEAP(400k,50k,keep)"
9. When accessing the WebSphere Manager applet, use Netscape V4.04 or
above. At present, no release of Internet Explorer can be used to reliably
access the WebSphere Manager applet.
Even IE 4.01-SP1 cannot be used to reliably access the WebSphere Manager
applet.
10.The WebSphere Manager is normally accessed using port 9090. The port
number can be changed using the WebSphere Manager applet, but if port
9090 is not initially available for some reason, then you will not be able to
change the port (catch-22) using the WebSphere Manager applet. In such a
case, the port can be changed by modifying the following parameter:
In directory:
/usr/lpp/WebSphere/AppServer/properties/server/servlet/adminservice
Edit the following file:
admin_port.properties
Change the parameter "endpoint.main.port" to refer to an available port, for
example:
endpoint.main.port=4040
11.If you are unable to connect to the WebSphere Manager, check to be sure that
it is really listening by running the "netstat" command. You should see the
Web server running on port 80 (typically), and the WebSphere Manager
running on port 9090 (typically). If you do not see programs listening on those
Web server—troubleshooting/hints/tips
235
two ports, then check your Web server joblog (produced by setting the "-vv"
option in the PARM in the Web server startup proc) for errors.
12.If you enter invalid data into WebSphere Manager, or if a defective file (such
as a bad .jar file) is loaded into WebSphere Application Server, then it is
possible that you will be unable to restart the Web server or the WebSphere
Manager applet. To resolve this problem, you will need to manually edit the
WebSphere Manager data files and remove the offending entry. The files are
located in the directory:
/usr/lpp/WebSphere/AppServer/properties/server/servlet/servletservice
The critical parameters are in the jvm.properties file. You can refer to the
backup files that you made in step 4 to help resolve the problem.
5.6 Information sources
In the following sections we show where to find information about Web servers in
the OS/390 environment.
5.6.1 Redbooks
The following redbooks are available:
• Global Server Certificate Usage with OS/390 Webservers, SG24-5623
• OS/390 Version 2 Release 4 Performance Figures for CICS Web-Enabled
Applications, SG24-5612
• IMS e-Business Connect Using the IMS Connectors, SG24-5427
• Developing an e-Business Application for the IBM WebSphere Application
Server, SG24-5423
• e-business Application Solutions on OS/390 Using Java: Samples, SG24-5365
• e-business Application Solutions Using Java: Volume 1, SG24-5342
• OS/390 Workload Manager Implementation and Exploitation, SG24-5326
• Accessing DB2 for OS/390 Data from the World Wide Web, SG24-5273
• CICS Transaction Server for OS/390: Web Interface and 3270 Bridge,
SG24-5243
• Internet Security in the Network Computing Framework, SG24-5220
• Capacity Planning for CICS Web-Enabled Applications on OS/390,
SG24-5168
• Ready for e-business: OS/390 Security Server Enhancements, SG24-5158
• Net.Commerce for OS/390, SG24-5154
• IP Network Design Guide, SG24-2580
• Software Configuration Management, A Websphere Development Example
Using Visual Age TeamConnection Enterprise Server, SG24-5473
• Connecting Domino to the Enterprise Using Java, SG24-5425
• Using VisualAge for Java Enterprise Version 2 to Develop CORBA and EJB
Applications, SG24-5276
236
Debugging UNIX Applications on OS/390
5.7 Useful Web sites
Within these Internet pages you should find all the latest and very helpful
information on the WebSpere Application Server and the Domino Go Webserver.
http://www.software.ibm.com/webservers
http://www.software.ibm.com/webservers/httpservers
http://www.software.ibm.com/webservers/httpservers/doc
http://www.software.ibm.com/webservers/dgw
http://www.fastcgi.com
Web server—troubleshooting/hints/tips
237
The following Web page is the most current and up-to-date checklist regarding
maintenance, troubleshooting, preventive service plan as well as product and
other related documentation:
http://www.software.ibm.com/webservers/dgw/doc50.htm#WSTG
Figure 120. IBM WebSphere troubleshooter for OS/390
5.7.1 Informational APARs
These two APARs contain a lot of information regarding the common error
message 02af and setting up the servletexpress manager.
• II11345—additional information on installing Lotus Domino Go Webserver
including servletexpress manager
• II10548—environment dirty bit caused by load of uncontrolled module from the
hfs tcbnctl set by UNIX System Services ( error 02af )
238
Debugging UNIX Applications on OS/390
Chapter 6. Network File System (NFS)
In this chapter we inform you about the OS/390 NFS server and the OS/390
client. In addition we give information on how to install and use this product.
Helpful debugging hints and tips are included from experience in supporting both
products.
Note
See the debugging hints in each chapter and see 6.1.12, “Debugging” on page
249 for information on how to debug and solve problems.
Client systems in a TCP/IP network that support the NFS client protocol can use
traditional MVS data sets and UNIX System Services HFS files as part of their file
system.
Since OS/390 Release 2.6, the SUN NFS Version 3 Protocol is supported, in
addition to the current support of NFS Version 2. Consequently, NFS can be
regarded as a good tool for transferring data across many different platforms,
including OS/390, UNIX systems, and Windows NT.
The major difference between the Windows NFS clients and UNIX clients is that
on the PC, the NFS code is installed as a separate piece of software, and the
mounts are accessed from a GUI administration window. On UNIX, the NFS client
code is standard, and administration is typically done from the command line
using the mount and other commands.
6.1 Setting up the OS/390 NFS server environment
In the following sections we inform you how to set up and customize the NFS
server. The server runs stateless as a single started task. The NFS Server since
OS/390 Release 6 has FMID HDZ11TS. It uses OE-Sockets and supports both
HFS and traditional MVS data sets.
6.1.1 Environment examples
Figure 121 on page 240 shows the work configuration that we set up to test these
various solutions. The host S/390 system is shown at the top right, and the middle
tier servers, used for the three-tier solutions, are shown at the middle left. Eight
clients on the 16 Mbps token-ring and on the 100 Mbps fast Ethernet LANs are
shown at the bottom. The Windows NT server also acted as a router between the
token-ring, fast Ethernet LANs, and FDDI LANs.
© Copyright IBM Corp. 1999
239
NFS Network Configuration
9672 model R75
OS/390 V2 R5
10.1.1.201 (TCP HSAS) or
100Mb FDDI
10.1.1.202 (regular TCP)
OSA-2
FDDI
10.1.1.175
DASD
DASD
3390-3
3390-3
OSA-2
T/R or Ethernet
DASD
DASD
3390-3
3390-3
skNet
FDDI
AIX or
NT
Servers:
PC 330
Windows NT 4.0 Server
with Maestro NFS
Gateway
10.1.3.3
10.1.3.5
10.1.3.100
16Mb T/R
10.1.3.6
10.1.3.4
10.1.3.1
10.1.3.2
Clients:
PC 750
Windows NT 4.0 WS
PC 350
Windows 95
PC 350
Windows NT 4.0 WS
Aptiva P200
Windows NT 4.0 WS
Figure 121. NFS network configuration
A network of computers was set up as shown in Figure 122 on page 241 to test
the NFS solution. It implements two kinds of TCP/IP stacks and interfaces: FDDI
and token-ring. The clients included Windows NT 4.0 and Windows 98
workstations.
The two-tier solution requires that the NFS client code can be installed on all
clients. The three-tier solution requires that the NFS “gateway” code be installed
on the middle tier Windows NT server.
240
Debugging UNIX Applications on OS/390
NFS
SM B
AIX or
NT
OS/390
NT Server
w/ NFS
Gateway
NFS
W indows w/
N FS C lient
W indows
(SMB) Client
NFS 3 tier
NFS 2 tier
Figure 122. NFS 2-tier versus 3-tier overview
6.1.2 Enable TCP/IP and the portmapper
Since the OS/390 NFS server uses the UNIX System Services Sockets ensure
that the connection between TCP/IP and UNIX System Services is established
before you start the NFS server.
NFS uses the TCP/IP RCP protocol for client-server communication. This is why
the portmapper function has to be enabled. If not already done, copy the sample
PORTMAP procedure from hlq.SEZAINST(PORTPROC) to your procedure
library. PORTMAP needs read access to the hlq.ETC.RPC file. Copy it from the
hlq.SEZAINST(ETCRPC) file.
Add the entries to the hlq.ETC.RPC file for the service provided by the NFS
server. See Figure 123.
Service
------NFS
mountd
mvsmount
showattrd
pcnfsd
Number
-----100003
100005
100044
100059
150001
Description
-----------NFS daemon
Mount daemon
daemon for mvslogin and mvslogout
showattr daemon
pcnfs daemon
Figure 123. Modifying the hlq.ETC.RPC file
Network File System (NFS)
241
6.1.2.1 Possible problems
• The Portmap started task does not come up as indicated by the message:
ICH408I OMVS segment (user) failed
Solution:
Issue the following commands to authorize the procedure:
SETROPTS GENERIC(STARTED)
RDEFINE STARTED PORTPPROC.* STDATA(USER(TCPIP) GROUP(SYS1)
TRUSTED(YES)
SETROPTS CLASSACT(STARTED) RACLIST(STARTED)
SETROPTS RACLIST(STARTED) REFRESH
For more information see, "Preparing for Security" in OS/390 UNIX System
Services Planning, SC28-1890.
• Portmap cannot bind indicated by message
EZY4063E PERMISSION DENIED (EACCES)
Solution:
Some hints point to TCP/IP’s Obey file, but using the Communication Server
2.7 to redefine the Obey file does not solve the problem. After changing the
settings for the portmapper in the TCP/IP profile see Figure 124; the problem
was solved.
PORT
111 TCP OMVS
111 UDP OMVS
; Portmap Server
; Portmap Server
Figure 124. Port definitions for the portmapper
For more information see OS/390 eNetwork Communications Server IP
Configuration, SC31-8513.
6.1.3 RACF profiles
The NFS procedure must be known and authorized as a user and a started task
to the system as the portmapper. See 6.1.2, “Enable TCP/IP and the portmapper”
on page 241.
6.1.3.1 Setting up the NFS started task as an authorized user
The authorization can be done twice using RACF through panel or
command using a batch job. See Figure 125 on page 242 through
Figure 131 on page 244 as examples of using the panels.
RACF - SERVICES OPTION MENU
4 USER PROFILES AND YOUR OWN PASSWORD
Figure 125. RACF primary option menu
242
Debugging UNIX Applications on OS/390
1
ADD
Add a user profile
ENTER THE FOLLOWING INFORMATION:
USER
===> NFSUSER
User ID
Figure 126. RACF user profile services
OWNER
===> HOERNER
User ID or group name
USER NAME
===> NFS
DEFAULT GROUP
===> SYS1
Group name
PASSWORD
===>
===>
User's initial password
Re-enter password to verify
Figure 127. RACF add an NFS user
GROUP ACCESS
ADSP
OIDCARD
NO-PASSWORD
===>
===>
===>
===>
NO
NO
NO
NO
SPECIAL
OPERATIONS
AUDITOR
===> NO
===> YES
===> NO
IDENTIFY THE MODEL PROFILE FOR USER DATA SETS (OPTIONAL):
MODEL PROFILE
===>
TO CREATE THE FOLLOWING, ENTER YES (OPTIONAL):
A GENERIC DATA SET PROFILE
A MINIDISK PROFILE
TO ADD OPTIONAL INFORMATION, ENTER YES
===> NO
===> NO
===> YES
Figure 128. Set operations
_ CICS PARAMETERS
_ WORK ATTRIBUTES
s OMVS PARAMETERS
Figure 129. Select to set up the OMVS segment
Enter User Identifier (UID) below, then press ENTER:
USER IDENTIFIER
===> 0
0 - 2147483647
Figure 130. Add OMVS parameters
Network File System (NFS)
243
UID= 0000000000
HOME= /
PROGRAM= /bin/sh
Figure 131. Add OMVS parameters
Note
The need for this definition is in process by the NFS development. The
authority of TRUSTED instead of OPERATIONS for the NFS started task and
the UID other than 0 for the NFS server is actually in testing error.
For more information see:
• "Preparing for Security" in OS/390 UNIX System Services Planning,
SC28-1890.
• " Protecting Your Data" in OS/390 NFS Customization and Operation,
SC26-7253.
6.1.4 Customize the OS/390 NFS server startup procedure
Copy the sample MVSNFS procedure from hlq.NFSSAMP(GFSAPROC) to your
procedure library and customize it. Assign a user ID to the NFS procedure with an
OMVS segment. See the sample of the procedure for more information on how to
customize it.
6.1.5 Security using the NFS server
6.1.5.1 OS/390 NFS security levels
Note
Since OS/390 2.6 the DEFAULT security level has changed from NONE to
SAFEXP. The four levels are the same, but their usage has changed. See the
NFS documentation in 6.3, “Documentation” on page 263 for more details.
Four different security levels can be selected in the MVS NFS attributes data set:
• NONE—Neither SAF checking nor exports list checking. For HFS, checks
UNIX permission bits; obtains the UID from the client RPC request.
• SAF—SAF checking. No exports checking. For HFS, checks UNIX permission,
the UID from the OS/390 UNIX segment using mvslogin.
• EXPORTS—Exports list checking. For HFS, checks UNIX permission bits for
HFS data; obtains the UID from the client RPC request. No SAF checking.
• SAFEXP—SAF checking and exports list checking. For HFS, checks UNIX
permission bits; obtains the UID from the OS/390 UNIX segment using
mvslogin.
244
Debugging UNIX Applications on OS/390
Recommendation: When using NFS for the first time set the security to NONE to
see whether you can access the data as expected.
There will be no mvslogin/mvslogout commands necessary for all of the client
platforms. The customer will only need to turn on pcnfsd (from nopcnfsd) in the
attribute file when initializing the OS/390 NFS server. This pcnfsd is an existing
support for the most client platforms.
For additional information see OS/390 Version 2 Release 6 UNIX System
Services Implementation and Customization, SG24-5178.
For the most common security problems see 6.1.12.4, “Known problems” on page
252.
6.1.5.2 HFS security exit
Today the security exit GFSAUXF is not called for HFS files. It will be called for
HFS files next.
6.1.6 BPXPRMxx parmlib member
In this section we inform you about the settings in the BPXPRMxx parmlib
member.
During UNIX System Services file system initialization, the FILESYSTYPE
statement for the network in the SYS1.PARMLIB(BPXPRMxx) member may have
incorrect port information. This causes a DFSMS/MVS NFS Server and/or Client
communication problem.
Figure 132 shows an example of a filesystype statement:
FILESYSTYPE TYPE(INET) ENTRYPOINT(BPXTCINT)
NETWORK DOMAINNAME(AF_INET)
DOMAINNUMBER(2)
MAXSOCKETS(64)
TYPE(INET)
INADDRANYPORT(2000)
INADDRANYCOUNT(325)
Figure 132. Example of a filesystype statement in the BPXPRMxx member
The statement in Figure 132 causes OS/390 UNIX to reserve a range of ports
starting at 2000 until 2324. DFSMS/MVS NFS port 2049 happens to be within this
reserved range. This causes port 2049 to be unavailable to DFSMS/MVS NFS.
This causes the communication problem. Choose instead another unused port
range (INADDRANYPORT,INADDRANYCOUNT).
For more information see the NFS APARs mentioned in 6.3, “Documentation” on
page 263.
6.1.7 Starting the NFS server
Ensuring that the NFS server comes up correctly is indicated by the messages
shown in Figure 133 on page 246. If the server does not start see 6.1.12.4,
“Known problems” on page 252.
Network File System (NFS)
245
BPXI004I OMVS INITIALIZATION COMPLETE
EZZ4202I OPENEDITION-TCP/IP CONNECTION ESTABLISHED FOR TCPIP6
:::::::::::::::::::::::
$HASP373 PORTPROC STARTED
IEF403I PORTPROC - STARTED - TIME=22.44.34
GFSA348I OS/390 NETWORK FILE SYSTEM SERVER (HDZ11TS) STARTED.
Figure 133. Server messages in the log
6.1.8 Mount
In this section we mention some common information about how to mount a data
set on the OS/390 system.
6.1.8.1 Mounting a file system within OS/390 UNIX
It is possible to mount an existing directory or a mounted file system (see the
examples in 6.1.8.2, “Mount from an NFS client” on page 247).
From the mount itself there is no difference, but a file system must be mounted
within OS/390 UNIX before.
• The HFS data set used as the file system must be allocated in prior.
- Allocate an additional HFS data set, using either:
+ A TSO/E ALLOCATE command
ALLOCATE DATASET('OMVS.USER.JOE') DSNTYPE(HFS)
SPACE(5,5) DIR(1) CYL
+ A JCL DD statement
Free the data sets and use the TSO/E MOUNT command under a superuser ID to
logically mount the new file system in the directory of an existing file system. For
example, the directory /u/joe is a mount point for OMVS.USER.JOE and /u/jane is
a mount point for OMVS.USER.JANE. The mount point must be a directory. If it is
not an empty directory, files in that directory are not accessible while the file
system is mounted.
• The FREE commands for the mounted file systems are:
FREE DATASET('OMVS.USER.JOE')
FREE DATASET('OMVS.USER.JANE')
Figure 134 shows a example how to mount a file system:
MOUNT FILESYSTEM('OMVS.USER.JOE') TYPE(HFS) MOUNTPOINT('/u/joe')
MOUNT FILESYSTEM('OMVS.USER.JANE') TYPE(HFS) MOUNTPOINT('/u/jane')
Figure 134. Example to mount a file system within OS/390 UNIX
For more information see "Mounting File Systems in the File Hierarchy"
in OS/390 UNIX System Services Planning, SC28-1890.
246
Debugging UNIX Applications on OS/390
6.1.8.2 Mount from an NFS client
To issue a mount from an NFS client see " Mounting MVS Data Sets onto a Client
Mount Point" in OS/390 Network File System Customization and Operation,
SC26-7253.
For examples about the syntax to issue a mount from an NFS client see, OS/390
Network File System User's Guide, SC26-7254.
To issue a mount from the OS/390 NFS client see 6.2.4, “MOUNT” on page 258.
Note
It is in process that a mount can be issued for a data set on the OS/390 system
using both binary and text as mixed conversion (see the forum on the Internet
at
http://itsont2.itso.ibm.com ).
6.1.9 HFS data set
In this section we give brief information about the HFS data set itself.
6.1.9.1 DFSMS/MVS
Since DFSMS/MVS 1.5, support is available for a multivolume HFS data set. Use
DFSMS/MVS 1.5 to get the latest HFS enhancements.
For more information see the DFSMS/MVS books in 6.3, “Documentation” on
page 263.
6.1.9.2 Allocation
An HFS data set can be allocated using TSO ALLOCATE command (see 6.1.8.1,
“Mounting a file system within OS/390 UNIX” on page 246) or using ISPF 3.2.
Figure 135 and Figure 136 on page 248 show examples of using ISPF 3.2 to
allocate an HFS data set:
Data Set Utility
A Allocate new data set
blank Data set information
C Catalog data set
M Allocate new data set
Other Partitioned, Sequential or VSAM Data Set:
Data Set Name . . . HFS.TEST
Option ===> M
Figure 135. Select option M
Network File System (NFS)
247
Allocate New Data Set
More:
Data Set Name . . . : HOERNER.HFS.TEST
Data set name type : HFS
(LIBRARY, HFS, PDS, or blank) *
Figure 136. Define the data set type HFS
Note
An HFS guide as a redbook will be available to describe in more detail the
usage of an HFS data set. See http:/www.redbooks.ibm.com.
For more information about HFS data sets, see the DFSMS/MVS documentation
in 6.3, “Documentation” on page 263.
6.1.10 Performance
This section has some hints and tips to improve performance.
• Using the NFS server FMID HDZ11TS ensure that the APAR OW38745 is
installed.
• The overall performance belongs to the network also. Check the settings
within the network based on the information given in the environmental
checklist in OS/390 Network File System Customization and Operation,
SC26-7263, and in OS/390 Network File System Performance Tuning Guide,
SC26-7255.
• The Maximum Transmission Unit (MTU) on the whole network is an
important factor to influence the performance at all.
Note
See the OS/390 Network File System Performance Tuning Guide, SC26-7255,
for some recommendations regarding the network environment.
• Since OS/390 2.5 the TCP/IP is known as the Communication Server 2.5.
Ensure that all the CS PTFs are installed.
• Accessing HFS data sets changes the default for the USS option FORKCOPY
in the BPXPRMxx parmlib member dynamically from COW to COPY by using
the SETOMVS command.
• Ensure that the DEBUG9 trace is switched off using INFO again.
• The following attributes from the NFS attribute file should be set
cachewindow(256), logicalcache(128M), bufhigh(128M), nfstasks(16,8),
percentsteal(20)
• For additional performance hints tuning, especially the UNIX System Services
environment, see:
• OS/390 Version 2 Release 6 UNIX System Services Implementation and
Customization, SG24-5178, using:
248
Debugging UNIX Applications on OS/390
•http://www.s390.ibm.com/products/oe/bpxpftgt.html
For more information see:
http://SSDDOM01.storage.ibm.com/TechSup/techdocframes.nsf?OpenDatabase
OS/390 Network File System Performance Tuning Guide, SC26-7255
OS/390 Version 2 Release 6 UNIX System Services Implementation and
Customization, SG24-5178
6.1.11 Supported clients for the OS/390 NFS server
Network File System client software for other IBM platforms is available from other
vendors. You can also access the DFSMS/MVS NFS server from non-IBM clients
that use/support the SUN NFS Version 2 protocol (with OS/390 V2R6 the NFS
server supports the NFS Version 3 protocol).
Windows will be added to the list of supported platforms in upcoming releases along
with a list of suggested or tested third-party vendors who provide NFS Client such as
FTP Software's Interdrive Windows 95 and Windows NT, HummingBird’s Maestro
for Windows 95 and Windows NT, etc.. As there are hundreds of NFS vendors in the
market, it is impossible to test them all; hence, we only provide a list of suggested or
tested vendor products.
There is no additional work to IBM other than for us to verify that these NFS Client
products installed on our Windows 95 and Windows NT machines as said in the
CD-ROM/manual, follow instructions as stated in their README/manual so that their
NFS client can communicate and work with our OS/390 NFS Server.
Any questions regarding usage of NFS on Windows platform should be asked
directly to third-party vendors such as HummingBird or FTP software. Those
products provide the GUI front-end interface to make the necessary connection
(mount) to NFS servers and display the remote files. They also provide another
front-end to set up users' uid/gid, permission mode bits.
Questions such as:
• How do I mount an NFS file system? What's the correct syntax? ", or
• "Nothing is displayed on My Computer? What's wrong?"
should be directed to the third-party vendors, not IBM.
6.1.12 Debugging
In this section we describe the documented commands to debug server-related
problems and give additional helpful information to analyze problems.
6.1.12.1 Hints and tips
• Ensure that all available maintenance for the FMID HDZ11TS is installed.
• To diagnose an NFS problem see "Diagnosing and Reporting Problems" in
OS/390 Network File System Customization and Operation, SC26-7253. In the
most cases the NFS DEBUG9 trace is very helpful to analyze a server-related
problem. Sometimes a simple find, for example, "error" or the debug messages
with a suffix of (E) points to a problem.
Network File System (NFS)
249
Figure 137 on page 250 shows an example of an NFS server trace.
GFSA301I (D) NFS_REQ WRITE(69153E38):FH(B22124E8 0E85311E B221
GFSA301I (D) NFS_REQ WRITE(69153E38):C2800078 005A001E)
PATTR(BL MAPLOW LF TXT MAPDOT )
GFSA301I (D) NFS_REQ WRITE : (69153E38) : OFF(0) CNT(3310) TOT(3310)
14:26:15
GFSA304I (D) REQUEST 1B4CA260 GENERATING NFS ERROR (5) NFSERR_IO
5695DF121_____ R1SM <-CMP-REL CS*ESS Mr.Stollenwerk_______ <-CON *IBM IN
DUE TO ERRNO (5) I/O ERROR
GFSA899I (E) : BLANKSTRIP MODE: TRAILING BLANK(S) IN RECORD 46 IS NOT
ALLOWED.
GFSA899I (E) : DSN = SPFIT.RZ4.P0.PROD.DEVNN.SRC(JJO265)
GFSA304I (D) : LUB 1B4752E0: ERRNO = 5
::::::::::::::::::::::::
GFSA301I (D) : NFS_REPL WRITE :(69153E38): FATTR(TYPE:0 MODE:0 LNKS:0 U:
G:0 SZ:0
Figure 137. NFS server debug9 trace
• Capture the debug9 trace:
• Use the parm DEBUG9 instead of the default INFO in the NFS server
procedure or use the Modify command /f nfs_stc,log=debug9 to activate
it dynamically.
• Ensure that the related log files, named as DD statements NFSLOG1
and NFSLOG2, are preallocated. See 6.1.4, “Customize the OS/390
NFS server startup procedure” on page 244.
• Because using the communications server/TCP/IP as the protocol layer a
packet or socket trace must be captured. For more information see Appendix
A, “Tracing within TCP/IP” on page 289 and the TCP/IP documentation in 6.3,
“Documentation” on page 263. Also see OS/390 eNetwork Communications
Server IP Diagnosis Guide, SC31-8521, and APAR II11160.
• To minimize the trace output specify only the involved IP addresses.
Note
For OS/390 TCP/IP OpenEdition, CTRACE replaces LESSTRACE,
MORETRACE, TRACE, and NOTRACE commands. These commands still
work with a V3R2 stack.
6.1.12.2 Messages and Return Codes
For more information about the Messages see, Appendix A in OS/390 Network
File System Customization and Operation, SC26-7253.
For more information about the Return codes see Appendix B in OS/390 Network
File System Customization and Operation, SC26-7253.
6.1.12.3 Helpful commands
The following commands are helpful to analyze a network problem. For some
commands see the outputs as an example.
• PING , OPING, and OEPING
250
Debugging UNIX Applications on OS/390
The PING command shows the established connections using the IP address.
The PING is a TCP/IP command. It tests connectivity of the network or routing
to a remote host by sending an echo request.
Figure 138 shows an example of a PING command:
# ping 205.148.2.127
CS/390 V2R5: Pinging host 205.148.2.127
Ping #1 response took 0.023 seconds.This is screen.
Figure 138. Output of a ping command
• SHOWATTR
This command displays the active NFS server attributes from the attribute file.
Figure 139 shows an example of a SHOWATTR command.
EH07:/usr/lpp/NFS: >showattr -t mvsa /u
lrecl(8196),recfm(vb),blksize(0),space(100,10),blks,dsorg(ps),dir(27),
unit(sysda),volume(),recordsize(512,4K),keys(64,0),nonspanned,
shareoptions(1,3),mgmtclas(),dsntype(pds),norlse,dataclas(),storclas()
Figure 139. Example of a showattr command
For more information see the OS/390 Network File System User's Guide,
SC26-7254.
• SHOWEXPORT (known as showexp)
- This command is only possible from some clients, for example, OS/2.
- To display the exported directories and file systems use the command
showmount.
•SHOWMOUNT
This command displays the server mount information or the exported
directories and file systems and as with "showexport" is based on the entries
in the export file.
Figure 140 shows an example of a SHOWMOUNT command.
>showmount -e os390s1
export list for os390s1:
/ETC (everyone)
/USR (everyone)
Figure 140. Showmount -e <host> example
For more information see the OS/390 Network File System User's Guide,
SC26-7254.
Network File System (NFS)
251
• RPCINFO and ORPCINFO
Use the RPCINFO command to display the servers that are registered and
operational with any portmapper on your network. The RPCINFO command
makes a remote procedure call (RPC) to an RPC server and displays the
results.
Figure 141 shows an example of an RPCINFO command.
tso rpcinfo -p if3
program vers proto
port
100000
2
udp
111
portmapper
100000
2
tcp
111
portmapper
100059
2
udp
7000
showattrd
100044
1
udp
7001
mvsmount
100005
1
udp
7002
mountd
100003
2
udp
2049
nfsd
Figure 141. RPCINFO -p <host> example
For more information see OS/390 eNetwork Communications Server IP User's
Guide, GC31-8514.
•NETSTAT
For the OS/390 NFS, the MVS TCP/IP netstat command provides information
that may be useful for tuning as well. See the OS/390 Network File System
Performance Tuning Guide, SC26-7255, and OS/390 Version 2 Release 6
UNIX System Services Implementation and Customization, SG24-5178, for
more information.
6.1.12.4 Known problems
In this section we mention information about known NFS server problems and
their solutions.
The OS/390 NFS server does not start (Message GFSA348I is missing)
If you receive the message IEE132I "Cmd device allocation error" Abend047,
ensure that all the defined libraries are APF authorized.
AbendU4093 during startup
Check your region size and ensure that the SMF exit IEFUSI does not limit your
available space.
Access denied
• For GID and UID checking using UNIX clients see 6.1.5, “Security using the
NFS server” on page 244. The GID/UID must be consistent (see APAR
II09734).
• Using the pcnfsd log in and authentication (see OS/390 Network File System
Customization and Operation, SC26-7253, and OS/390 Network File System
Users Guide, SC26-7254).
• Ensure that GID/UID 0 is set in the OMVS segment and the server uses
OPERATIONS security (see 6.1.5, “Security using the NFS server” on page
244).
252
Debugging UNIX Applications on OS/390
• Ensure that site attribute security is set correctly (see the Note in 6.1.5.1,
“OS/390 NFS security levels” on page 244).
No access to the HFS data sets
If you receive the message GFSA833I:
• Ensure that your HFS site attribute hfs(/prefix) is set.
• Analyze your NFS Export file whether the entry is missing or incorrect.
• Ensure that you have mounted the HFS file system directly because an
access across a mount point is not possible (see APAR II09734).
No access from the WIN NT MAESTRO client using NFS V3
If you receive the message GFSA786I (E):
• Define the usage of V2 (NOV3) from the Windows NT registry file.
• See APAR II11447.
• Ask the OEM vendor HUMMINGBIRD for the final fix.
Unpredictable results
• Ensure that NFS port 2049 is not reserved.
- See 6.1.6, “BPXPRMxx parmlib member” on page 245.
- See the information APARs II08210, II09734, and OW22624.
• Analyze the environment regarding the TCP/IP stack you are using and the
related definitions.
Note
For addition information visit the NFS Internet page shown in Figure 142 on
page 254.
6.1.12.5 Most frequently asked questions
In this section we inform you where and how to find more common helpful
information on the Internet.
Figure 142 on page 254 shows the NFS Internet page informing you about the
most frequently asked questions:
Network File System (NFS)
253
Figure 142. NFS Internet page http://www.s390.ibm.com/nfs/faq/faqindex.html
Click Environmental Checklist to get more information as shown on Figure 143
on page 255.
254
Debugging UNIX Applications on OS/390
Figure 143. Environmental Checklist
Click one of the mentioned problems from the Problem Determination, Q&A’s
Guide list shown in Figure 144 on page 256 to see the solution.
Network File System (NFS)
255
Figure 144. Problem Determination, Q&A’s Guide
Click Technical Support Site at the bottom of the page shown in Figure 145 on
page 257 to get the page shown in figure 17 and select the NFS product.
256
Debugging UNIX Applications on OS/390
Figure 145. Technical Support Site
6.2 Setting up the OS/390 NFS client environment
In this section we describe how to set up and customize the NFS client.
6.2.1 UNIX System Services
The client runs as a procedure started during the UNIX System Services (USS)
known as OMVS or OpenEdition.
6.2.1.1 BPXPRMxx parmlib member
Since the NFS UNIX client runs under control of USS it must be defined in the
BPXPRMxx parmlib member.
The following entry shown in Figure 146 must be added to the BPXPRM parmlib
member to start the NFS client automatically.
FILESYSTYPE
TYPE(NFS)
ENTRYPOINT(GFSCINIT)
PARM('installation parms')
ASNAME(proc_name)
Figure 146. NFS client BPXPRMxx parmlib member entry
The procedure name shown in Figure 146 is the name of the started task in the
proclib.
Network File System (NFS)
257
Customize the OS/390 NFS client startup procedure.
Copy the sample MVSNFS procedure from hlq.NFSSAMP(GFSCPROC) to your
procedure library and customize it. Assign a user ID to the NFS procedure with an
OMVS segment. See the sample of the procedure for more information on how to
customize it.
For more information see OS/390 Network File System Customization and
Operation, SC26-7253.
6.2.2 RACF profiles
The NFS client must be authorized as a user and a started task within RACF. See
6.1.3, “RACF profiles” on page 242.
6.2.2.1 Possible problems
• Message ICH408I OMVS segment not defined
• Message IEF450I NFSC ended ABENDSEC6 U0000 RSN11BE0026
- Solution: Define an OMVS segment and see 6.1.2.1, “Possible
problems” on page 242.
6.2.3 Starting the NFS client
Ensure that the NFS client comes up correctly indicated by the Messages shown
in Figure 147. IF the client does not start see 6.2.5.4, “Known problems and most
frequently asked questions” on page 262.
BPXI004I OMVS INITIALIZATION COMPLETE
EZZ4202I OPENEDITION-TCP/IP CONNECTION ESTABLISHED FOR TCPIP6
:::::::::::::::::::::::
$HASP373 NFSCR
STARTED
BPXI004I OMVS INITIALIZATION COMPLETE
GFSC700I OS/390 NETWORK FILE SYSTEM CLIENT
(HDZ11TC) STARTED
Figure 147. Client messages in the log
6.2.4 MOUNT
In this section we inform you about the mount using the NFS client.
Use the TSO MOUNT command to make a connection between a mount point on your
local HFS file system and one or more files on a remote MVS, AIX, UNIX, OS/2,
OS/390, or other file system. The MOUNT command can only be used by a MVS
superuser.
Figure 148 on page 259 shows the complete syntax of the mount.
258
Debugging UNIX Applications on OS/390
MOUNT FILESYSTEM(file_system_name)
TYPE(NFS)
MOUNTPOINT(local_mountpoint)
MODE(RDWR|READ)
PARM('hostname:"path_name,server_attributes", options')
SETUID|NOSETUID
WAIT|NOWAIT
Figure 148. Mount syntax
Figure 149 shows an example of a mount using the NFS client.
mount filesystem(’omvs.usr1.hoer’) type(nfs) mountpoint('/usr')
parm('mcevsd:soft,timeo(100)')
Figure 149. Example of a mount
For more information about the mount and its operands see the OS/390 Network
File System User's Guide, SC26-7254.
6.2.5 Debugging
In this section we describe the documented commands to debug client-related
problems and give additional helpful information to analyze problems.
6.2.5.1 Hints and tips
• Ensure that all available maintenance for the FMID HDZ11TC is installed
• Capture the NFS client trace:
To store the client trace both trace data sets of the NFS client procedure,
named as the DD statements NFSCMSG1 and NFSCMSG1 must be
pre-allocated (see the sample procedure hlq.NFSSAMP(GFSCPROC)).
• To initialize the client trace during startup the procedure will need to be
updated. In the BPXPRMxx Parmlib member the client entry will need to have
PARM('INITD,xxx,yyy,) where xxx,yyy, are your current parameters and the
INITD must be in uppercase, for example: PARM('INITD,biod(6)')) INITD must
be removed after the startup problem is resolved.
Note
The Filesystype related parms (see figure 14) cannot be changed dynamically.
It can be changed dynamically starting with OS/390 R8.
• To initialize the client trace after successful startup:
- Start NFS Client address space
- Go to ISHELL /usr/lpp/NFS/ directory, and run the program
/usr/lpp/NFS/nfsstat with the parameter " -d 1 " ( to turn on debug tracing)
- Perform the problem at all
- Go to ISHELL /usr/lpp/NFS, and run the program "/usr/lpp/NFS/nfsstat"
with the parameter " -d f " (to flush the messages to file)
Network File System (NFS)
259
- Go to ISHELL /usr/lpp/NFS, and run the program /usr/lpp/NFS/nfsstat with
the parameter " -d 0 " (to turn off debug tracing)
Figure 150 shows an example of an NFS client trace:
GFSC099I (D) CMAIN.0D #MQctl.1: qnum=0,qbytes=262144
GFSC099I (D) CMAIN.0D #MQctl.4: qlim=1024,qbytes=262144
GFSC099I (D) CMAIN.0E #Start : ErrD
GFSC099I (D) CMAIN.0E #Start : MntD
GFSC099I (D) CMAIN.0E #Start : BioD
GFSC099I (D) CMAIN.0F #Process: Mount
GFSC700I OS/390 NETWORK FILE SYSTEM CLIENT (HDZ11TC) STARTED
:::::::::::::::::::::::::::::::
15:11:30 GFSC098I (D) CXDR0 07 VN.REMOV: RQF(003CC3E2:xREMOVE ) RC=5
RQ(-1,45E,0)
15:11:30 GFSC098I (D) CXDR0 07 VN.REMOV: RQF(003CC3E2:xREMOVE )
CLNT_CALL FAILED EZA4339E RPC: Timed out
15:11:30 GFSC098I (D) CXDR0 07 VN.REMOV: RQF(003CC3E2) ERRP=05000000
00000000
15:11:30 GFSC100E (E) CXDR0 12 VN.REMOV: RPC REQUEST (003CC3E2) FAILED,
RETURN VALUE -1 RETURN CODE 00000467h REASON CODE 6E020403h
15:11:30 GFSC100E (E) CXDR0 12 VN.REMOV: (TIMED OUT)
15:11:30 GFSC110E (E) CVNRM 2F VN_REMOV: NFS_REMOVE FUNCTION FAILED,
RETURN VALUE -1 RETURN CODE 00000467h REASON CODE 6E020403h
15:11:30 GFSC098I (D)
$REPL VN_REMOV: RQF(003CC3E2:7F665C30)
RV(-1,467h,6E020403h)
Figure 150. NFS client trace
• Because the NFS client runs under USS, a component trace must be
captured in some cases:
- Capture the USS component trace.
The following example shows how to set up the ctrace and a dump:
1. CTRACE for SYSOMVS:
- Update CTIBPX00 member with BUFSIZE(2M)
- Enter MVS command: TRACE CT,ON,COMP=SYSOMVS
r xx, OPTIONS=(ALL),JOBNAME=(nfsid),END
2. MVS dump after hang occurs:
- DUMP COMM=("NFS DUMP"),
- r xx,JOBNAME(nfsid,OMVS),DSPNAME=(’OMVS’.*),
SDATA=(SQA,CSA,LPA,TRT,RGN),END
- See OS/390 UNIX System Services Planning, SC28-1890
- See http://www.s390.ibm.com/oe/samples/oectrace.html
• In some cases the TCP/IP CS eNetwork trace must be captured. See 6.1.12.1,
“Hints and tips” on page 249 and Appendix A, “Tracing within TCP/IP” on page
289.
260
Debugging UNIX Applications on OS/390
6.2.5.2 How to interpret a USS message
In this section we inform you how to interpret a USS message.
Example:
BPXF002I FILE SYSTEM HOERNER.HFS.TEST WAS 539 NOT MOUNTED. RETURN CODE =
00000081, REASON CODE = 0594003D
• BPXF002I FILE SYSTEM name WAS NOT MOUNTED. RETURN CODE =
retcode, REASON CODE = reason
Explanation: The system could not mount the specified file system.
• RETURN CODE 81
Table 39. Return codes listed by value
Decimal value
Hex value
Return code
Description
129
0081
ENOENT
No such file,
directory, or IPC
member exists.
For more information see, "Return Codes listed by value " in OS/390 UNIX
System Services Messages and Codes, SC28-1908.
• REASON CODE 0594003D
Table 40. The Reason code is made up of 4 bytes in the following format
cccc
rrrr
halfword reason code qualifier
halfword reason code
The two high-order bytes of the reason codes returned by OS/390 UNIX contain a
value that is used to qualify the contents of the two low-order bytes. If the
contents of the two high-order bytes are within the range of 0000 to X'20FF', the
error represented by the reason code is defined by OS/390 UNIX. If the contents
of the two high-order bytes is outside the range, the error represented by the
reason code is not an OS/390 UNIX reason code.
6E00–6EFF OS/390 NFS Client File System. See OS/390 Network File System
Customization and Operation, SC26-7253, for an explanation of the code.
0594 This value fits in the x'0000'-x'20FF' USS category.
003D JRDirNotFound
A directory in the pathname was not found.
Action: One of the directories specified was not found.
Verify that the name specified is spelled correctly.
• REASON CODE 6E22046F
6E22 This value fits in the x’6E00’–x’6EFF’ category.
Network File System (NFS)
261
046F
Table 41. NFS client Reason codes
NFS Version 3
Protocol value
OS/390 UNIX
value
Hex value
Description
NFS3ERR_REMOT
EREMOTE
046F
Too many levels of
remote in path
For more information see "Reason Codes listed by value" in OS/390 UNIX
System Services Messages and Codes, SC28-1908, and OS/390 MVS System
Messages, Volume 2 (ASB-EZM) , GC28-1785, and Appendix B in OS/390
Network File System Customization and Operation, SC26-7253.
For more information about the NFS client messages see Appendix A in OS/390
Network File System Customization and Operation, SC26-7253.
6.2.5.3 Helpful commands
See 6.1.12.3, “Helpful commands” on page 250 for the commands to analyze a
NFS client problem also. Within the ISHELL the commands could be different. For
example, use the command find . -name *rpcinfo to find the directory of the
orpcinfo command.
6.2.5.4 Known problems and most frequently asked questions
In this section we mention information from known NFS client problems and the
solutions.
The NFS client does not start (message GFSC700I is missing)
• DFSMS NFS MVS Client ABEND0C4 RC 4 at OMVS Startup
- Ensure that the PPT entries PGMNAME(BPXINIT,BPXVCLNY) are
removed from the SCHEDxx parmlib member.
Message BPXF028I NOT MOUNTED
RETURN CODE = 0000046A
REASON CODE = 6E2204E7
• Ensure that the SYSTCPD DD statement of the NFS client procedure points to
the active TCP/IP.
The INCLUDES of a PL1 program cannot be accessed and compiled
• The problem is that the compiler is not able to compile HFS files.
- With DFSMS it should be possible.
- INCLUDE books on SYSLIB must exist as a native PDS on the host
system. This is because there is apparently no BPAM support for HFS.
- Use the command OCOPY to copy the source to the OS/390 system.
Message GFSC711E NETWORK FILE SYSTEM CLIENT INITIALIZATION
FAILED
• GFSC098I (D) CMAIN.0D #MQctl.0: -1,0000006F,07020303
- Ensure that the OMVS segment with GID/UID 0 and the FILESYSTYPE
statement in the BPXPRMxx member are defined (see 6.2.1.1,
“BPXPRMxx parmlib member” on page 257 and 6.2.2, “RACF profiles” on
page 258).
262
Debugging UNIX Applications on OS/390
ABENDEC6
• Do not stop USS or the NFS client with the REXX procedure BPXSTOP.
Message GFSC110E during startup
• Check your region size and ensure that the SMF exit IEFUSI does not limit
your available space.
Note
For more information see the NFS Internet page as described in Figure 142
through Figure 145 in 6.1.12.5, “Most frequently asked questions” on page 253.
6.3 Documentation
In this section we inform you where helpful information about NFS, UNIX System
Services (USS), DFSMS, and TCP/IP can be found. Also, a list of related
documents is included in Appendix C, “Related publications” on page 297.
• On the Internet:
- Common OS/390 information
http://www.s390.ibm.com
• NFS
- Books
Note
See the OS/390 2.6+ documentation because it is updated to include additional
common information and the new functions.
+ NFS books on the Internet:
http://www.s390.ibm.com/nfs/pubs.html
+ NFS home page
http://www.s390.ibm.com/nfs/faq/Nfsfaq30.html
+ NFS APARs: II11447, II09734, II11572, II06805, OW34846, OW38745,
OW38862
+ NFS Q&A and nondefect on the Internet:
http://SSDDOM01.storage.ibm.com/techsup/swtechsup.nsf/support/dfsmste
chd
• Common forum about the International Support Organization Archives
- See MVS-OE@LISTSERV.GEORGETOWN.EDU or on the Internet:
http://itsont2.itso.ibm.com
• USS
- Internet:
http://www.s390.ibm.com/oe/samples/oectrace.html
Network File System (NFS)
263
• DFSMS/MVS
- Internet:
http://ssddom01.storage.ibm.com/techsup/swtechsup.nsf
http://SSDDOM01.storage.ibm.com/techsup/swtechsup.nsf/support/dfsmstechd
ata
http://SSDDOM01.storage.ibm.com/techsup/swtechsup.nsf/support/dfsmsmain
264
Debugging UNIX Applications on OS/390
Chapter 7. Infoprint Server
In the following sections we provide some hints and tips about the OS/390
Infoprint Server.
7.1 What the Infoprint Server does for you
The OS/390 Infoprint Server lets you consolidate your print workload from many
servers to a central OS/390 Infoprint Server.
Consolidating onto a central print server lets you use the right printer for specific
print jobs, balance print workload across all available printers, and more easily
manage the inventory of printers.
No matter what type of data you want to print (payroll, invoices, Web documents,
sales reports), you can use OS/390 batch applications, VTAM applications,
applications running on remote workstations, and OS/390 UNIX System Services
applications to send print requests to the OS/390 Infoprint Server. The OS/390
Infoprint Server then sends the print requests to OS/390 printers, including local
printers and remote printers in a TCP/IP network. (See Figure 151 for the printing
solutions the OS/390 Infoprint Server provides.)
Printing solutions the OS/390 Infoprint Server provides:
Figure 151. OS/390 Infoprint Server capabilities
© Copyright IBM Corp. 1999
265
7.2 Printing OS/390 UNIX System Services data on AFP printers
A midsize retailer runs UNIX applications that have been ported to OS/390 UNIX
System Services. Until now, the applications have printed business statements to
ASCII printers controlled by a print server running on a UNIX system. Now, the
retailer wants to use a higher-speed AFP printer attached to an S/390, such as
the IBM 3130 printer, for more efficient printing.
Here is how this retailer can use the OS/390 Infoprint Server components (in
gray) to meet its requirements:
Figure 152. USS data to AFP printers
This works as follows:
1. The UNIX user logs on to OS/390 UNIX System Services using the rlogin
function.
2. Using the OS/390 UNIX System Services printing commands provided with
the OS/390 Infoprint Server, the UNIX applications print directly to the OS/390
Print Interface component.
3. The OS/390 Print Interface component creates output data sets on the JES
spool.
4. PSF/MVS selects the output data sets from the JES spool and prints them on
an AFP printer. If the retailer has more than one AFP printer, the first available
printer can print the data sets.
7.3 Components of the OS/390 Infoprint Server
The OS/390 Infoprint Server provides these components that run on the OS/390
host system:
• OS/390 Print Interface
• NetSpool
• IP PrintWay
7.3.1 OS/390 Print Interface
The OS/390 Print Interface provides a line printer daemon (lpd) that extends
JES2 and JES3 print capabilities to users and application programs in a TCP/IP
LAN environment. It also provides printing support for users and application
programs in the OS/390 UNIX System Services environment. Users and
applications can print to OS/390-controlled printers, including Advanced Function
Presentation (AFP) printers attached to OS/390 using PSF/MVS or LAN-attached
ASCII printers attached to OS/390 using IP PrintWay.
266
Debugging UNIX Applications on OS/390
The OS/390 Print Interface provides these benefits:
• Prints any type of data format supported by the printer
• Validates print requests
• Notifies users when processing is complete
• Provides a Printer Inventory
• Identifies users of printed output
• Supports double-byte character set (DBCS)
7.3.2 NetSpool
NetSpool directs VTAM application data to the JES spool without requiring
application program changes. JES or PSF for OS/390 can print the output data
sets or transmit them to another location for printing. Or IP PrintWay can transmit
the data sets to a remote printer in your TCP/IP network.
NetSpool provides these benefits:
• Data integrity
• Central routing
• Printer sharing
• Print broadcasting
• AFP formatting
• IP PrintWay and NetSpool integration
• Double-byte character set (DBCS) support
• Transparent-data support
For more information about NetSpool, see the NetSpool Web page at:
http://www.printers.ibm.com/R5PSC.NSF/Web/index
7.3.3 IP PrintWay
IP PrintWay routes JES2 or JES3 print data from OS/390 to remote printers or to
host systems in your TCP/IP network. A print server can be running on the host
system. The remote printer or host system must support either lpr/lpd protocol or
direct socket printing.
IP PrintWay provides these benefits:
• Data integrity
• JCL parameters to route data sets
• Automatic routing of data sets
• Distribution of data to a workstation
• NetSpool and IP PrintWay integration
• Accounting
• Installation exits
• Migration from Network Print Facility (NPF)
• Double-byte character set (DBCS) support
Infoprint Server
267
7.3.4 UNIX System Services commands
The OS/390 Infoprint Server also provides USS commands.
OS/390 UNIX System Services commands (lp, lpstat, cancel) let OS/390 UNIX
System Services environment applications submit print, query, and cancel
requests to the OS/390 Print Interface.
OS/390 UNIX System Services commands provide these benefits:
• Notifying users when processing is complete
• Specifying AFP printing options
• Specifying printing options in a file
• Printing on remote printers that are not defined to the OS/390 Print Interface
7.3.4.1 Examples
Here are a few examples of the USS commands.
• Command lp - send a Job to a printer
Format:
lp [-cmsw] [-d destination] [-n copies] [-o option ...] [-t title]
[filename ... | -]
Description:
The lp command sends a job containing one or more files to a printer. If you do
not specify any files on the command line, or if you specify a filename of -, lp
prints from standard input.
The files can be:
- OS/390 data sets, such as partitioned data sets or sequential data sets
- Hierarchical File System (HFS) files
- Lists of printable files
Example:
For example, to print three copies of myfile1 and myfile2 on
Printer2, enter:
lp -d Printer2 -n 3 myfile1 myfile2
• Command lpstat - Show Printer Names and Locations and Status of Print
Jobs:
Format:
lpstat [-dt] [-a [printername ...]] [-o [printername ...]]
[-p [printername ...]] [-u[ userid ...]] [jobid ...]
Description:
lpstat writes the names and locations of printers or the status of print jobs to
standard output.
For printers, the lpstat command returns
- The name of the printer
- The number of jobs submitted to the printer using the OS/390 Print Server
268
Debugging UNIX Applications on OS/390
- The location of the printer
- A description of the printer
For jobs, the lpstat command returns:
- The job ID
- The user ID of the person who submitted the job
- The state of the job like:
pending
The job is waiting to print.
Note: JES3 cannot distinguish job states and returns pending for all
jobs on the JES spool.
processing
The job may be:
+ Being transmitted to a LAN printer for printing
+ Printing
+ Retained on the JES spool after printing
held
The job is held on the JES spool and cannot print for one of these
reasons:
+ The user specified hold=true when submitting the job.
Note: JES3 cannot recognize a job held for this reason and returns
pending.
+ The operator held the job.
+ An error occurred.
- The data format of the job, as specified by the document-format job
attribute or as determined by the OS/390 Print Server
- The number of bytes in the job
- The name of each file or file-reference document in the job
When lpstat returns information about multiple jobs, the order is not
significant. The first job listed may not be the next job to print.
Examples:
Use the lpstat command to query printer names and locations. For example, to
see the names and locations of all printers known to the OS/390 Print Server,
enter:
lpstat -a
You can also use lpstat to query printer location and job status at the same
time. For example, you sent a job to Printer3 and want to pick it up if it has
printed instead of waiting to have it delivered to your output bin. To find out
where Printer3 is and whether any job that you submitted to it has printed,
enter:
lpstat -o Printer3
• Command cancel - cancel a print Job
Infoprint Server
269
Format:
cancel jobid ...
Description:
The cancel command cancels one or more print jobs that you submitted, with
these restrictions:
- You can only cancel your own jobs.
- You cannot cancel a job after it has started processing.
- In a JES3 environment, you may not be able to cancel a job held on the
JES spool.
Example:
Use the cancel command to cancel a job. For example, you realize that you
need to make some changes in the file that you just sent to print on Printer3.
If you do not remember the job ID that the lp command returned, use the
lpstat command to query all the jobs that you submitted to Printer3:
lpstat -o Printer3
Suppose that your job has an ID of 17. To cancel it, enter:
cancel 17
7.4 Relevant APARs for OS/390 Infoprint Server or Print Interface
These are APARs that describe the most common problems with the OS/390
Infoprint Server or Print Interface.
For the APARs and PTFs needed for Infoprint Server or Print Interface, refer to
site http://www.s390.ibm.com/os390/bkserv/new_tech_info.html.
7.4.1 II11165
APAR Identifier ...... II11165
Last Changed ........ 99/06/14
INFO APAR FOR OS/390 PRINT SERVER OR PRINT INTERFACE
COMPID 5647A01OP R100, FMID HOPI100, PSP U:OS390R5 S:PRINTSRV
This APAR describes the following problems in the OS/390 Infoprint Server or
Print Interface environment:
• Section A—SMP/E apply for Infoprint Server yields return code 4
• Section B—MSGAOP027E is issued from the aopstart command
• Section C—MSGAOP027E is issued from the cancel or lpstat command
• Section D—lp command, expected output not generated
• Section E—MSGCEE5207E signal SIGABRT daemon not start
• Section F—msgaop004e aopd: bind() failed EDC5111I
• Section G—MSGAOP997 cannot open message catalog
• Section H—ICH408I /var/printsrv/aopd.515
• Section I—Japanese and Spanish support
• Section J—Additional Information on the Security setup for OS/390 Infoprint
Interface
270
Debugging UNIX Applications on OS/390
• Section K—Automatically start AOPSTART from the /etc/rc file.
Note
Refer to the APAR itself to see details about the sections mentioned previously.
7.4.2 II11659
APAR Identifier ...... II11659
Last Changed ........ 99/07/14
INFO APAR FOR OS/390 PRINT SERVER OR PRINT INTERFACE
5647A01OP R100 HOPI100 PSP U:OS390RX S:PRINTSRV II11165
This APAR is the continuation of II11165 (see the previous section) and describes
the following problems in the OS/390 Infoprint Server or Print Interface
environment:
• Section L— Running aopd with another LPD process such as LPSERV
• Section M—BPXF024I AOP004E EDC5111I Permission denied message
dealing with port 515
• Section N—AOPSTART and AOPSTOP batch information
• Section O—Web sites for additional information
• Section P—BPXF024I EDC5112I TCP/IP not up and active
• Section Q—AOPIM019 Print Interface has not been registered
Note
Refer to the APAR itself to see details about the sections mentioned previously.
Infoprint Server
271
7.4.3 OW39059
APAR Identifier ...... OW39059
Last Changed ........ 99/06/17
PRINT SERVER FAILS WITH MSGBPXF024I ERRNO2 = 0X5620064
Release 100
: UW59599 available 99/05/24 (F905 )
PROBLEM SUMMARY:
****************************************************************
* USERS AFFECTED: Customers who use the Open Source version of *
*
the lpr command called "rlpr".
*
****************************************************************
* PROBLEM DESCRIPTION: The rlpr program uses several TCP send *
*
operations to communicate the parts of *
*
an LPD protocol command. The OS/390
*
*
Print Interface LPD (aopd) does not
*
*
correctly identify printer name
*
*
because it does not wait for all of
*
*
the parts of the command to be
*
*
received.
*
*
*
*
The messages issued are:
*
*
BPXF024I (user) open
*
*
(/var/Printsrv/printers/,ORDWR)
*
*
failed in FileHandle::restore()
*
*
const at ./src/filehandle.cpp
*
*
108: EDC5123I Is a directory.
*
*
errno2=055620064
*
****************************************************************
* RECOMMENDATION: Apply the applicable PTF.
*
****************************************************************
7.5 Information Sources
Here are some helpful information sources, including books and Web sites.
7.5.1 Books
These IBM publications provide additional details about the OS/390 Infoprint
Server:
• IBM IP PrintWay Guide, S544-5379
This book provides information about IP PrintWay, including how:
- The MVS system administrator creates and maintains installation data sets
and handles failed transmissions
- An MVS job submitter submits a job for processing by IP PrintWay
- The MVS console operator starts, stops, and monitors IP PrintWay
- An MVS diagnostician diagnoses problems
• IBM NetSpool Guide, G544-5301
This book provides information about NetSpool, including how:
- An MVS system programmer configures NetSpool
- The MVS console operator starts, stops, and monitors NetSpool
- An MVS diagnostician diagnoses problems
272
Debugging UNIX Applications on OS/390
• Print Interface Configuration Guide, G544-5544
This book provides information about the OS/390 Print Interface, including
how:
- An OS/390 system programmer installs and configures the OS/390 Print
Interface
- The OS/390 system administrator creates and maintains an inventory of
printers and assists users in submitting print requests
- The OS/390 console operator starts, stops, and monitors the OS/390 Print
Interface
- An OS/390 diagnostician diagnoses problems
• OS/390 Infoprint Server Overview, G544-5545
This book provides an overview of the OS/390 Infoprint Server for
management and technical personnel and includes the benefits of the OS/390
Infoprint Server, how it can be used, and how it works.
• OS/390 Infoprint Server User's Guide for OS/390 UNIX System Services,
S544-5543
This book explains how OS/390 UNIX System Services users use the OS/390
Infoprint Server to:
- Submit jobs to print on OS/390
- Query printer names and locations or print job status
- Cancel print jobs
- Specify your requirements for a print job
• OS/390 Infoprint Server User's Guide for Windows, S544-5511
This book explains how to install and use the following OS/390 Infoprint Server
WindowsTM programs to print documents on OS/390 printers or view AFP
files:
- AFP Printer Driver
- AFP Viewer plug-in
- OS/390 Printer Port Monitor
7.5.2 Web sites
Here are some interesting Web sites:
• The IBM OS/390 Web site provides additional information about OS/390:
http://www.s390.ibm.com/os390/
• The DOC APARs and ++HOLD DOC Web page provides information about the
latest documentation updates for the OS/390 base elements and optional
features: http://www.s390.ibm.com/os390/bkserv/new_tech_info.html
• IBM S/390 Printing solutions: http://www.printers.ibm.com/s390.html
• The Print Server home page:
http://www.printers.ibm.com/R5PSC.NSF/web/printservhome
Infoprint Server
273
274
Debugging UNIX Applications on OS/390
Chapter 8. Language Environment for OS/390
IBM OS/390 Language Environment for OS/390 and VM (also called Language
Environment) provides common services and language-specific routines in a
single run-time environment for C, C++, COBOL, Fortran (OS/390 only; no VM,
UNIX System Services, or CICS support), PL/I, and assembler applications. It
offers consistent and predictable results for language applications, independent
of the language in which they are written.
Language Environment is the prerequisite run-time environment for applications
generated with the following IBM compiler products:
OS/390 C/C++
IBM C for VM/ESA C/C++
Compiler for MVS/ESA
AD/Cycle C/370 Compiler
COBOL for OS/390 & VM
COBOL for MVS & VM
SAA AD/Cycle for COBOL/370
PL/I for MVS & VM
AD/Cycle PL/I MVS & VM
VS FORTRAN and FORTRAN IV (in compatibility mode)
Language Environment supports, but is not required for, an interactive debug tool
for debugging applications in your native OS/390 environment. The IBM
interactive Debug Tool is available with OS/390 or with the latest releases of the
C/C++, COBOL, and PL/I compiler products.
Language Environment supports, but is not required for, VS Fortran Version 2
compiled code (OS/390 only).
Language Environment consists of the common execution library (CEL) and the
run-time libraries for C/C++, COBOL, Fortran, and PL/I.
For detailed information, refer to OS/390 Language Environment for OS/390
Customization, SC28-1941.
8.1 LE run-time options
In this section we describe what the LE run-time options are, how they can be set,
and some options of interest.
8.1.1 LE run-time options
The LE run-time options essentially control various aspects of LE's management
of our application environment. For example, LE run-time options specify how LE
performs its storage management, what national language to use for the run-time
environment, how LE is to handle abends and exceptions on behalf of an
application, and what level of information is to be produced for errors that are not
handled.
High level languages encapsulate complicated functions into simple functions:
• Obtaining storage
© Copyright IBM Corp. 1999
275
• Obtaining the system date and time
• Output messages
• Math functions
• etc.
The run-time environment is called by the user program to do these pieces of
work. LE combines several run-times; binding occurs at run-time, not at link time.
8.1.2 Setting LE run-time options
You may wish to set the LE run-time environment for a process that runs from a
shell command, from the TSO environment, and from submitted batch jobs.
1. From the shell:
You can simply export environment variable _CEE_RUNOPTS to set LE
run-time options. For example, if you want to set run-time TRAP to a value of
OFF and set run-time option TERMTHDACT to a value of dump, you can enter
the following from the shell:
export _CEE_RUNOPTS='TRAP(OFF),TERMTHDACT(DUMP)'
You could also set the environment variable in your envar file. Just add the
same export statement to your envar file.
2. From TSO:
You can pass the LE run-time options when you invoke the module name. For
example, you can pass the same run-time options mentioned above for
example to the dcelogin process by issuing:
euvslgn 'TRAP(OFF),TERMTHDACT(DUMP)'/myprincipal
3. From batch:
If your program, which is invoked from a batch JCL stream, wants to affect the
LE run-time options for that program (process), you can pass these run-time
options in the JCL. Here is an example of the descrying prod and how it can
pass LE run-time options to kern program/process EUVPCT(where 'Lorgnette'
represents an LE run-time option):
//GO
EXEC PGM=EUVPCT,REGION=someSIZE,TIME=1440,
// PARM=('LErtopt1(value),LErtopt2(value)' /&PARMS >DD:CTOUT')
8.1.3 Some LE run-time options of interest
• ABPERC: Specifies on abound that is exempt from the LE condition handler,
thus percolating the abound code to the operating system, which will create a
system dump.
• ENVAR: Sets the initial values for the environment variables specified.
• LIBSTACK: Controls the allocation of the thread's library stack storage.
• POSIX: Specifies whether the enclave can run with the POSIX semantics.
• STACk: Controls the allocation and management of thread-level heap storage.
• TERmthdact: Sets the level of information produced due to an error that is not
handled.
• TRACE: Determines whether LE run-time library tracing is active.
276
Debugging UNIX Applications on OS/390
• TRap: Specifies how LE routines handle abends and program interrupts.
8.2 When are which libraries used
1. C/C++ Compile steps
> prefix SCEEH... in SYSLIB concatenation
2. Prelink steps
> SCEEOBJ and/or SCEECPP in SYSLIB
Note: Never SCEELKED or SCEELKEX
3. Link Edit/Binder step
> SCEELKED in SYSLIB
> SCEELKEX then SCEELKED in SYSLIB (Binder) (LE 1.8 and up only)
Note: Never SCEERUN
4. Run/Go step
> SCEERUN in STEPLIB/LPA/LINKLST
Note: Never SCEELKED
8.3 Common Execution Library (CEL)
Language Environment consists of the common execution library (CEL) and the
run-time libraries for C/C++, COBOL, Fortran, and PL/I.
The prefix of this environment ( modules, messages, etc. ) is CEE.
8.3.1 Set of common functions and routines of LE
• Initialization
• Storage Management
• Condition Handling (error processing)
> Includes CEEDUMPs
• Messaging
• Date/Time services (year 2000)
• Termination
8.3.2 Important modules to remember
• CEEHDSP: Always the top module in CEEDUMPs. This module schedules the
CEEDUMP to be taken, IGNORE CEEHDSP.
> LE Condition handling modules start with CEEHxxxx
• CEEPLPKA: LE's main load module. Contains most of the parts for CEL. Most
LE reported ABENDs occur in CEELPKA. This does not indicate an LE
problem. Use the data from U4039 and/or CEEDUMP to determine what the
problem is.
• CEEBINIT: LE's main initialization load module.
> LE Initialization modules will start with CEEBxxxx.
Language Environment for OS/390
277
• CEECCICS: LE's main interface module with CICS.
> LE CICS interface modules will start with CEECxxxx.
• CEEEVxxx: LE event handlers; xxx represents the member number of the
language.
003 C/C++Runtime
005 COBOL
007 FORTRAN
008 DCE
010 PL/I
012 Cebug Tool
8.3.3 Messages and abends
8.3.3.1 Prefixes for messages and modules
• CEE—represents a CEL environment but may be reporting a problem
elsewhere
• IGZ—represents a COBOL environment
• IBM—represents a PL/I environment
• AFH—represents a FORTRAN environment
• EDC—represents a C/C++ environment
See IBM Language Environment Debugging Guide and Run-Time Messages,
SC26-4829, for exact details on LE messages and/or ABENDs.
8.3.3.2 Common CEL messages
• CEE3201S—Indicates ABEND0C1
• CEE3204S—Indicates ABEND0C4
• CEE32xxS—Indicates ABEND0Cy
Where y is the equivalent for decimal xx
• CEE3250C—Indicates some non-0Cx Abend occurred
• CEE1000S—CICS only. Report LE ABEND to operator console
• CEE0802C and CEE0813S—Error with HEAP storage (normally a user
problem)
• CEE0374C—Indicates that some error (may or may not be within LE) occurred
- examine token.
8.3.3.3 LE condition token (feedback code)
Here is an example of a CEE3204S token and how to interpret it:
00030C84 59C3C5C5 xxxxxxxx
0003 | 0C84 | 59 | C3C5C5 | xxxxxxxx
0003 Severity
0000 Informational (I)
0001 Warning (W)
0002 Error (E)
0003 Severe (S)
0004 Critical (C)
278
Debugging UNIX Applications on OS/390
0C84 Hex message number (3204)
59 Flags (ignore)
C3C5C5 Prefix (Facility ID)
xxxxxxxx Used internally
8.3.3.4 Common CEL abends
• U4038—Some severe error occurred but no dump was requested (useless)
• U4039—Some severe error occurred and a dump was requested (see
CEEDUMP)
• U4083*—Save area backchain in error
• U4087*—Error during error processing
U4083 and U4087 may mask original error
• U4093*—Error during initialization
• U4094*—Error during termination
* You need to obtain the corresponding reason code for a more meaningful
error description
Note
Abend codes mentioned above show decimal value. Reg 15 in SVC Dump
shows hex value.
8.3.3.5 Getting useful information
1. A CEEDUMP and/or U4039 Dump
Remember U4038 does not generate a dump.
2. Use the following LE run-time options to ask LE to take a dump:
TERMTHDACT
TRAP
3. U4039 Dump
Allocate SYSMDUMP DD card
• Must point to a data set (not SYSOUT)
• LRECL 4160, RECFM FBS
APAR II10573 describes details on how to request a dump in various LE/370
runtime environments. Here is an excerpt of the APAR regarding UNIX System
Services:
Unix System Services Shell
Prior to OS/390 2.7, go to the following URL:
http://www.s390.ibm.com/le/assist/support/ii10573.html
At OS/390 2.7 or later:
1.Specify where to write the system dump
To write the system dump to a data set, issue the command:
export _BPXK_MDUMP=filename
Language Environment for OS/390
279
where filename is a fully qualified data set name with LRECL=4160 and
RECFM=FB
2.Specify Language Environment run-time options:
export _CEE_RUNOPTS="termthdact(suboption)"
where suboption is UAONLY, UADUMP, UATRACE, or UAIMM. If UAIMM is set,
TRAP(ON,NOSPIE) must also be set.
3. Rerun the program and the dump will be written to the specified data set.
Different ways to get a CEEDUMP
• In batch
CEEDUMP DD card - directed to SYSOUT or a data set. If not specified
dynamically allocated to SYSOUT=*
_CEE_DUMPTARG to SYSOUT=x (LE 2.7)
• CICS - CESE Queue - (for more information please see APAR PQ20892)
• UNIX System Services - Current working directory
_CEE_DUMPTARG to redirect to another directory (LE 1.9)
8.3.3.6 Non-useful information
• Remember, when ABTERMENC(ABEND) is set the original ABEND (such as
0C4) is reissued.
DO NOT SLIP ON THIS ABEND!
• LE reissues this ABEND at the end of LE termination.
• The LE environment has already been cleaned up and therefore a dump at
this point is useless.
• Work with dump of U4039 instead.
8.3.4 Run-time options
Changing run-time options:
• Installation Defaults—Samples in SCEESAMP
CEEWCOPT—CICS only
CEEWDOPT—all non-CICS environments
• Region Level (LE 2.7 and back to LE 1.8 with PQ22683)
CEEWROPT—CICS and IMS (with LRR) only
- Unlike Installation defaults, loaded at run-time
280
Debugging UNIX Applications on OS/390
• Application level
CEEWUOPT
- CEEUOPT must be linked with the application
• Program level
Compiled into program
- #progma runopts() in C
- PLIXOPT for PL/I
• Program invocation
In PARM= statement on EXEC card
- COBOL
+ PARM='program opts/LE runtime opts'
- C,PLI, FORTRAN, LE enabled assembler
+ PARM='LE runtime opts/program opts'
+ The '/' delimits the two sets of arguments and is required when LE opts
are being set
• UNIX System Services shell
Set the _CEE_RUNOPTS environment variable (use export as needed)
- no slash is needed!
• Some important run-time options (See LE Customization for a complete list)
ABTERMENC(option)
RETCODE-Step ends with return code (job continues)
ABEND-Step will be ABENDed (job terminates)
Recommendations:
+ CICS - ABTERMENC(ABEND)
+ OTHERS - ABTERMENC(ABEND)
• ALL31()
OFF - for environments with AMODE 24 programs
- May be determined dynamically (see APARs PQ17931 and PQ20527)
ON - for environments with all AMODE 31 programs
Recommendations:
+ CICS - ALL31(ON) whenever possible
+ OTHERS - ALL31(ON) whenever possible
• ERRCOUNT()
Terminates environment if number of conditions is exceeded.
0 - Allows infinite number of conditions to occur
Required by PL/I for 'on unit' processing—each invocation of an on unit
is an LE condition
- Required by C/C++ for signal processing—signals are processed as
LE conditions
Language Environment for OS/390
281
Greater 0—Indicates the number of conditions allowed before LE
terminates the process
- Recommendations:
+ CICS—0 (required)
+ Others—0
• HEAP(int,inc,where,type)
-
int - Minimum size of initial heap segment
inc - Minimum size of any additional heap segments
where - BELOW, ANYWHERE
type
KEEP - do not free when empty (better performance)
FREE - free when empty (better memory management)
Notes:
Used for Cobol working storage
Dynamic storage (C malloc/PLI allocate)
Recommendations:
-
CICS
- HEAP(4K,4080.ANYWHERE,KEEP,4K,4080)
OTHER - HEAP(32K,32K,ANYWHERE,KEEP.8K,4K)
When BELOW uses FREE
When ANYWHERE uses KEEP
• STACK(int,inc,where,type)
-
int—Size of initial stack segment
inc—Minimum size of any additional stack segments
where—BELOW,ANYWHERE - Must be BELOW if ALL31(OFF)
type
KEEP - do not free when empty (better performance)
FREE - free when empty (better memory management)
Notes:
Used for Dynamic Save Area
Used for C/C++ and PL/I local variables
Recommendations:
- CICS - STACK(4K,4080,ANYWHERE,FREE)
- OTHER - STACK(128K,128K,ANYWHERE,KEEP)
Use KEEP when ANYWHERE is specified
Use FREE when BELOW is specified
• POSIX(option)
- ON - Indicates POSIX semantics should be used
- OFF - Indicates ANSI semantics should be used
Notes:
Where ANSI and POSIX conflict this setting is used to clear up any
ambiguities.
282
Debugging UNIX Applications on OS/390
ANSI programs may access the UNIX System Services HFS without
POSIX(ON)
POSIX(ON) must be set to use POSIX only functions like
thread_create()
POSIX(ON) is the default in the USS Shell
Recommendations:
- CICS
- POSIX(OFF)
- OTHER - POSIX(OFF)
• STORAGE(getheap,freeheap,stack,reserve allocation)
- getheap—One byte value used to initialize every heap allocation; 00 is
equivalent to WSCLEAR (COBOL).
- freeheap—One byte value used to initialize every heap free.
+ Debug purposes
+ Security purposes
- stack—One byte value used to initialize every stack (DSA) allocation.
- reserve allocation—Reserve stack storage (does not really belong on this
option)
Recommendations:
- CICS—STORAGE(00,NONE,00,0K)
- OTHER—STORAGE(NONE,NONE,NONE,8K)
• TERMTHDACT (option)—The action a thread task takes when terminating
-
QUIET—Messages off
MESSAGE—Just message no dump
TRACE—Traceback only (no dump)
DUMP—CEEDUMP only
UADUMP—CEEDUMP/system dump*
UAONLY—system dump no CEEDUMP (LE 2.7)
UATRACE—system dump and traceback (LE 2.7)
UAIMM—system dump of the original error (LE 2.7)
+ Requires TRAP(ON,NOSPIE)
+ Takes dump in LE ESTAE using SETRP prior to LE condition for
Program checks only.
Recommendations:
- CICS—TERMTHDACT(MSG)
- OTHER—TERMTHDACT(TRACE)
• TRAP
Used to turn LE condition handling on or off.
- TRAP(ON,SPIE) recommended. Simply TRAP(ON) prior to LE 1.9
- TRAP(ON,NOSPIE) (LE 1.8)
Allows user applications to have their own ESPIE routine.
LE condition handling will still take place using an ESTAE.
Required if using TERMTHDACT(UAIMM) (LE 2.7)
- TRAP(OFF)
Language Environment for OS/390
283
• LE condition handling is disabled (almost)
• SPIE/NOSPIE suboption is ignored
Recommendations:
- CICS
- TRAP(ON,SPIE)
- OTHER - TRAP(ON,SPIE)
8.4 LE hands on debug
As already mentioned, to debug an LE problem you can set the run-time options
to get either a CEEDUMP or to force an SVCDUMP. To debug a real application
program problem the CEEDUMP will provide useful information to fix the
problem. If a system ABEND occurs, request an SVCDUMP
8.4.1 CEEDUMP debugging
The following CEEDUMP provides information for:
• Domino server ABENDs with U034 RC 070E0303
8.4.1.1 Traceback
The traceback section includes a sequential list for all active routines and the
routine name, statement number, and offset where the exception occurred.
I
Figure 153. Partial traceback information
The traceback shows the last accessed module at the top of the trace output. In
this trace the exception status is shown for module Panic. This module will
always show an exception status whenever a problem occurs. Because we have
a security problem and not a coding problem we only find the exception status in
the Panic module. If a coding problem hits we will first find the exception status
(examine the trace starting from bottom to top) in this module followed by the
Panic module.
284
Debugging UNIX Applications on OS/390
The following condition information section displays a message describing the
condition and the address of the condition information block (CIB) which has
pointers to other useful control blocks and areas:
Condition Information for Active Routines
Condition Information for ./ospanic.c (DSA address 1E5E6A60)
CIB Address: 0002E618
Current Condition:
CEE0198S The termination of a thread was signaled due to an unhandled
condition.
Original Condition:
CEE3250C The system or user ABEND U 034 R=070E0303 was issued.
Location:
Program Unit: ./ospanic.c
Program Unit:Entry:
Panic Statement: Offset: +00041C4E
Machine State:
ILC..... 0002
Interruption Code..... 000D
PSW..... 078D1400 9ADE2696
GPR0..... 84000000 GPR1..... 84000022 GPR2..... 0000006F GPR3.....070E0303
GPR4..... 1E5E6DA8 GPR5..... 1C75EBD8 GPR6..... 1C6CDF10 GPR7..... 1C75EBD8
GPR8..... 1D8DCB20 GPR9..... 0000000A GPR10.... 000FE3B8 GPR11.... 1F940C58
GPR12....1E5DA5A8 GPR13....1E5E6A60 GPR14.... 9ADA1068 GPR15.... 070E0303
ABEND code: 00000022 Reason code: 070E0303
It is important to know how to find information about the reason code, which in
this case can be found in OS/390 UNIX System Services Messages and Codes,
SC28-1908, in the chapter "Reason Codes Listed by Value."
The reason code is made up of 4 bytes in the following format:
Table 42. Reason codes
cccc
rrrr
070E
0303
• cccc is a halfword reason code qualifier
• rrrr is the halfword reason code
In the previous mentioned Message and Codes manual you can find that 070E
points to a security problem. To get more information we have to look for the rrrr
0303 value.
This leads to: JRlpcDENIED
Access was denied because the caller does not have the correct permission.
Even though it would be better to debug this problem using an SVC Dump, let us
still have a look at the CEEDUMP to become more familiar.
To get information where the problem occurred we have to look for a control block
which is called ZMCH. This control block is pointed to by our Condition
Information Block (CIB). The storage address where we can find our CIB is
provided in the Condition Information for Active Routines section (mentioned
above). Now we have to look for this storage area to find the CIB eye catcher. On
hex offset '24' in this control block you will find the storage address for our ZMCH.
ZMCH control block starts with his eye catcher and shows on hex offset '8' the
General Purpose Registers (GPRs) followed by the Program Status Word (PSW)
which leads us to the failing module.
Language Environment for OS/390
285
If it is difficult to find the ZMCH like described issue only a 'find ZMCH' command.
Another possibility to find CIB and ZMCH is to start with register 12 value at the
time of the error. Register 12 will point to a control block called CAA (Anchor
Vector). On offset 2D8 you will get the address to find the CIB. To verify whether
register 12 points to a CAA check the eye catcher which should be seen on the
register 12 address minus hex 10.
If you would like to go deeper into CEEDUMP debugging refer to IBM Language
Environment for MVS and VM Debugging Guide and Run-Time Messages,
SC26-4829.
8.4.1.2 System Dump debugging
Now let us have a look at the System Dump. For this special problem we had to
set two SLIP Traps. The first one was set to catch the U034 ABEND and another
one was set to get a dump at the time where RACF returned the bad reason code
which ends with a security violation.
SLIP SET,C=U0034,ID=U034,SDATA=(ALLNUC,PSA,CSA,LPA,SQA,RGN,
SUM,TRT,GRSQ),AL=(0),JL=(OMVS),DSPNAME=('OMVS'.SYSZBPX1,
'OMVS'.SYSZBPX2),END
For more information on how to set this slip refer to 2.1.5 Dump options / IPCS in
OS/390 MVS System Commands, GC28-1781.
Note
If the problem seems to be UNIX System Services related provide a dump
including the OMVS address space and all its data spaces.
Interactive Problem Control System (IPCS) is the tool to debug System Dumps.
For detailed information please refer to OS/390 MVS Interactive Problem Control
System (IPCS) Commands, GC28-1754.
The following command will provide you a quick opportunity to look for a search
argument which can be used to ask the support center for an already known
solution or to search in different databases provided by IBM.
Table 43. Common IPCS commands
286
IPCS command
Output
IP ST
Status during initial part of problem
determination process
IP CBF RTCT+9C? STR(SDUMP) VIEW
FLAGS
Shows specified dump options
IP SELECT ALL
Shows all active address spaces at time of
dump
IP ST CPU REGI
Shows registers at time of the slip matched
IP SYSTRACE TIME(GMT or LOCAL)
Shows system trace including time values
VERBX LOGDATA
Shows EREP and ABEND information
IP VERBX MTRACE
Provides part of the SYSLOG
Debugging UNIX Applications on OS/390
To get LE-related information you can use the following commands:
Table 44. LE-related IPCS commands
IPCS command
Output
IP VERBX LEDATA
General LE data
IP VERBX CEEERRIP ’CEEDUMP’
CEEDUMP information
IP VERBX CEEERRIP ’CM’
LE control block information
IP VERBX CEEERRIP ’HEAP’
All storage management control blocks
IP VERBX CEEERRIP ’ALL’
All control block including CTRL and COBOL
IP VERBX CEEERRIP ’CAA(xxxxxxxx)’
Combined with other options
- allows specific CAA to be used as anchor (required for CICS)
- VERBX CEEERRIP ’ALL CAA(xxxxxxxx)
IP VERBX CEEERRIP 'DSA(xxxxxxxx)'
Combined with other options
- allows specific DAS (Dynamic Save Area) to be used for trace back
IP VERBX CEEEDDI 'TCB(xxxxxxxx)'
Combined with other options
- allows specific TCB to be used as starting point
Getting the name of the failing csect (LE module/function/):
From option1 (Browse) in IPCS:
1. Get address (second word) from PSW.
2. Issue L xxxxxxxx (where xxxxxxxx is the address you got from PSW).
3. Issue F CEE prev.
4. Back up 5 bytes to the 47F0xxxx instruction.
• This is the beginning of the CSECT function.
• Repeat steps 3 and 4 as needed (to step further back for more module
checking).
5. Add the value at offset x'0C' to the module address.
6. Go to the location.
• You can do this by issuing the L value you calculated.
7. First byte is offset (or half offset) to name.
8. This is a 2-byte prefixed string with the function name.
Language Environment for OS/390
287
0B14D100
0B14D110
0B14D120
0B14D130
0B14D140
0B14D150
0B14D160
0B14D170
0B14D180
0B14D190
0B14D1A0
0B14D1B0
0B14D1C0
0B14D1D0
0B14D1E0
0B14D1F0
0B14D200
0B14D210
0B14D220
47F0F014 00C3C5C5 00000098 000000D8 ! .00..CEE...q...Q
Start address B14d100 + offset value
47F0F001 90ECD00C 18BF41F0 00005800 ! .00...ü....0....
B1345810 D04C1E01 5500C00C 47D0B038 ! ....ü<....ä..ü..
58F0C2BC 05EF181F 5000104C D7011000 ! .0B.....&..<P...
100018FD 18D150F0 D00450D0 F00898F1 ! .....J&0ü.&ü0.q1
F010D203 D0801000 5850D080 58A05000 ! 0.K.ü....&ü...&.
1F44BF2F A39C4780 B0745830 20081223 ! ....t...........
4770B06A 58D0D004 18F458E0 D00C980C ! .....üü..4.Öü.q.
D01407FE D7C1E3C3 C840C1D9 C5C14060 ! ü...PATCH AREA 40C3C5C5 D6E7D2C5 C440F9F8 4BF0F6F8 ! CEEOXKED 98.068
B0A0B0A2 B0A4B0A6 B0A8B0AA B0ACB0AE ! ...s.u.w.y......
B0B0B0B2 B0B4B0B6 B0B8B0BA B0BCB0BE ! ................
B0C0B0C2 B0C4B0C6 B0C8B0CA B0CCB0CE ! .ä.B.D.F.H......
B0D00000 00000000 20CEB000 0B14D204 ! .ü............K.
go 20 bytes ahead
TO 0B14D1EF (X'00000010' bytes)--All bytes contain X'00'
00000000 00200000 0008C3C5 C5D6E7D2 ! ..........CEEOXK
C5C40000 01000001 00000000 00000000 ! ED..............
00000014 0B14D100 F1F9F9F8 F0F3F0F9 ! ......J.19980309
F1F2F1F2 F0F0F0F1 F0F9F0F0 0001F000 ! 121200010900..0.
Figure 154. Example dump output
Remember step 7 above (offset or half offset).
For more information about debugging refer to the previous mentioned
publications.
288
Debugging UNIX Applications on OS/390
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
Appendix A. Tracing within TCP/IP
This appendix describes how to use and set up the TCP/IP component trace and
the TCP/IP packet trace.
A.1 The Component Trace
The most important facility to debug problems in OS/390 eNetwork
Communications Server is the Component Trace. It can be used to investigate
the cause of problems in the TCP/IP area by tracing internal processes and data
flows. The MVS Component Trace support provides the following functions:
• Captures trace requests.
• Adds trace records to an internal buffer.
• If you request, it will also write this internal buffer to an external writer.
• You can use the IPCS subcommand CTRACE to format the trace records.
• Each data area in the trace record is dumped separately. A descriptor at the
beginning records the address and length of each data area, so you can
determine from where it came.
The Component Trace traces individual components (such as STORAGE,
Internet, PFS, and so on) and writes the information to individual data sets. You
will typically use Component Trace while recreating a problem.
A.1.1 Using the Component Trace with internal buffer
The TCP/IP Component Trace options can be specified at TCP/IP initialization, in
the SYS1.PARMLIB member CTIEZBxx, or after TCP/IP was initialized. A default
minimal component trace is always started during TCP/IP initialization. If you
want to start TCP/IP with a specific trace member you must use the command:
/S tcpip_procedure_name,PARMS=CTRACE(CTIEZBxx)
The CTIEZBxx member contains a BUFSIZE parameter. The default for this
parameter is 256 K, but you should use a BUFSIZE of 16M to ensure that traces
written to the internal buffer do not wrap. Please note that you need to restart
TCP/IP to change the BUFSIZE parameter. For more information about the
CTRACE options refer to OS/390 eNetwork Communications Server IP Diagnosis
Guide, SC31-8521.
If you want to change the Component Trace options after TCP/IP initialization,
you must use the MVS TRACE CT command. All options except for the size of
the TCP/IP Component Trace buffer can be changed using this command. To
specify the trace options you can either use the previously mentioned CTIEZBxx
parmlib member, or you can enter the trace options as prompt reply.
© Copyright IBM Corp. 1999
289
• To use the trace options specified in the parmlib member, you must start the
Component Trace with the command:
TRACE CT,ON,COMP=SYSTCPIP,SUB=(tcpip_procedure_name),
PARM=CTIEZBxx
• If you want to enter the trace options at the prompt, you must start the
CTRACE with the command:
TRACE,CT,ON,COMP=SYSTCPIP,SUB=(tcpip_procedure_name)
You will be prompted to specify the trace options to be in effect and you must
respond using the following syntax:
R xx,OPTIONS=(option1,option2,option3,etc.),END
The OPTIONS parameter list contains the trace options that are used for the
Component Trace to trace specific TCP/IP components, such as TCP, UDP,
ENGINE or SOCKETS. A complete list of all CTRACE options is available in
OS/390 eNetwork Communications Server IP Diagnosis Guide, SC31-8521.
After starting the Component Trace, you should verify that the CTRACE options
are in effect by issuing:
DISPLAY TRACE,COMP=SYSTCPIP,SUB=(tcpip_procedure_name)
Watch out that CTRACE buffer size is displayed as set in your CTIEZBxx
member. CTRACE buffer size not being at the configured size indicates a
possible syntax error in the CTIEZBxx member. MVS System log may contain
messages pointing to the syntax error.
If you are sure that the CTRACE is running with all the options you have
specified, then recreate the event you want to trace and obtain the CTRACE data
with a dump:
• DUMP COMM=(your dump title here)
• R n,JOBNAME=tcpipprocname,DSPNAME=’TCPIPPROCNAME’.TCPIPDS1,
CONT
• R n,SDATA=(nuc,rgn,csa,sqa),END
TCPIPDS1 is the name of the data space for each TCP/IP in an MVS image. For
the dump to be meaningful, CSA and SQA must be specified (minimally) for the
SDATA parameter.
The input can be either a dump of the TCP/IP address space and the data space,
TCPIPDS1, containing the trace table or the data set(s) produced by the external
writer (see next chapter).
290
Debugging UNIX Applications on OS/390
• To deactivate the CTRACE use the command:
TRACE CT,ON,COMP=SYSTCPIP,SUB=(tcpip_procedure_name)
and reply
R xx,OPTIONS=(MINIMUM),end
This way your CTRACE buffer size will stay as configured, but all options are
reset.
• Using the command:
TRACE CT,OFF,COMP=SYSTCPIP,SUB=(tcpip_procedure_name)
will reset the CTRACE buffer size to 128k and you will need to restart TCPIP
to get it back to 16M.
A.1.2 Using the Component Trace with external writer
If you do not want the CTRACE writing to the internal buffer, you can define an
external writer data set. This data set is defined in the CTTCP procedure in
SYS1.PROCLIB.
• To use the CTRACE with external writer you need to start the writer with the
command:
TRACE CT,WTRSTART=CTTCP
• Then turn on the trace and connect the external writer. The following
command connects TCP to the external writer.
TRACE CT,ON,COMP=SYSTCPIP,SUB=(tcpip_procedure_name)
- Once the system responds, issue:
R nnnn,OPTIONS=(....), WTR=CTTCP,END
• Verify that all the CTRACE options are in effect by issuing:
DISPLAY TRACE,COMP=SYSTCPIP,SUB=(tcpip_procedure_name)
• Then recreate the event you want to trace and disconnect the external writer
immediately:
TRACE CT,ON,COMP=SYSTCPIP,SUB=(tcpip_procedure_name)
• Once the system responds, reply:
R nn, OPTIONS=(MINIMUM), WTR=DISCONNECT,END
The CTRACE is set to minimum mode after this command.
• Then you can stop the external writer with the command:
TRACE CT,WTRSTOP=CTTCP
To view the trace data you must use the IPCS Component Trace formatting.
Specify the external writer data set as the default in IPCS and use the CTRACE
formatting with COMP=SYSTCPIP and SUB=tcpip_procedure_name.
Tracing within TCP/IP
291
A.2 The Packet Trace
The TCP/IP Packet Trace contains all outgoing and incoming packets for a
specified packet option, broken down into headers and data. It is very useful to
investigate problems with FTP or other applications that transfer data using TCP
or UDP.
• To start the Packet Trace with the packet option that should be monitored, use
the following command:
VARY TCPIP,tcpip_procedure_name,PKT,ON,packet_option
- The packet option can be one of the following:
• Packet Length (ABBREV=abbrev_length)
• Protocol Type (PROT=TCP,UDP,ICMP)
• Packet Destination Address (IP=xxx.xxx.xxx.xxx)
• Packet Source Port (SRCPORT=source_port)
• Packet Destination Port (DESTPORT=destination_port)
• If you want the Packet Trace written to an external data set, you must activate
the external writer with the command:
TRACE CT,WTRSTART=PTTCP,WRAP
• The Status of the Trace should be verified with the command:
DISPLAY TRACE,COMP=SYSTCPDA,SUB=(tcpip_procedure_name)
• Then connect the Packet Trace to the external writer:
TRACE CT,ON,COMP=SYSTCPDA,SUB=(tcpip_procedure_name)
and reply to the prompt with:
R nnnn,WTR=PTTCP,END
• Then recreate the problem situation you want to trace
• Disconnect the external writer immediately using the command:
TRACE CT,ON,COMP=SYSTCPDA,SUB=(tcpip_procedure_name)
reply to the prompt with:
R nnnn,WTR=DISCONNECT,END
• Stop the external writer with the command:
TRACE CT,WTRSTOP=PTTCP
• And finally stop the packet trace
VARY TCPIP,tcpip_procedure_name,PKT,OFF
To view the trace data you must use the IPCS Component Trace formatting.
Specify the external writer data set as the default in IPCS and use the CTRACE
formatting with COMP=SYSTCPDA and SUB=tcpip_procedure_name.
292
Debugging UNIX Applications on OS/390
Appendix B. Special notices
This publication is intended to help systems programmers and IBM customer
support personnel to solve problems related to applications executing under
UNIX System Services. The information in this publication is not intended as the
specification of any programming interfaces that are provided by OS/390 and
UNIX System Services. See the PUBLICATIONS section of the IBM
Programming Announcement for OS/390 for more information about what
publications are considered to be product documentation.
References in this publication to IBM products, programs or services do not imply
that IBM intends to make these available in all countries in which IBM operates.
Any reference to an IBM product, program, or service is not intended to state or
imply that only IBM's product, program, or service may be used. Any functionally
equivalent program that does not infringe any of IBM's intellectual property rights
may be used instead of the IBM product, program or service.
Information in this book was developed in conjunction with use of the equipment
specified, and is limited in application to those specific hardware and software
products and levels.
IBM may have patents or pending patent applications covering subject matter in
this document. The furnishing of this document does not give you any license to
these patents. You can send license inquiries, in writing, to the IBM Director of
Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785.
Licensees of this program who wish to have information about it for the purpose
of enabling: (i) the exchange of information between independently created
programs and other programs (including this one) and (ii) the mutual use of the
information which has been exchanged, should contact IBM Corporation, Dept.
600A, Mail Drop 1329, Somers, NY 10589 USA.
Such information may be available, subject to appropriate terms and conditions,
including in some cases, payment of a fee.
The information contained in this document has not been submitted to any formal
IBM test and is distributed AS IS. The information about non-IBM ("vendor")
products in this manual has been supplied by the vendor and IBM assumes no
responsibility for its accuracy or completeness. The use of this information or the
implementation of any of these techniques is a customer responsibility and
depends on the customer's ability to evaluate and integrate them into the
customer's operational environment. While each item may have been reviewed
by IBM for accuracy in a specific situation, there is no guarantee that the same or
similar results will be obtained elsewhere. Customers attempting to adapt these
techniques to their own environments do so at their own risk.
Any pointers in this publication to external Web sites are provided for
convenience only and do not in any manner serve as an endorsement of these
Web sites.
Any performance data contained in this document was determined in a controlled
environment, and therefore, the results that may be obtained in other operating
environments may vary significantly. Users of this document should verify the
applicable data for their specific environment.
© Copyright IBM Corp. 1999
293
This document contains examples of data and reports used in daily business
operations. To illustrate them as completely as possible, the examples contain
the names of individuals, companies, brands, and products. All of these names
are fictitious and any similarity to the names and addresses used by an actual
business enterprise is entirely coincidental.
Reference to PTF numbers that have not been released through the normal
distribution process does not imply general availability. The purpose of including
these reference numbers is to alert IBM customers to specific information relative
to the implementation of the PTF when it becomes available to each customer
according to the normal IBM PTF distribution process.
The following terms are trademarks of the International Business Machines
Corporation in the United States and/or other countries:
IBM 
AD/Cycle
AFP
AIX
AS/400
AT
BookManager
C/MVS
CICS
COBOL/370
C/370
CT
DB2
DFSMS
DFSMS/MVS
DFSMSdfp
eNetwork
Hummingbird
IMS
IMS/ESA
Infoprint
IPDS
IP PrintWay
Langauge Environment
MQ
MQ
MQSeries
MVS/ESA
Net.Data
Netfinity
NetSpool
OpenEdition
OS/2
OS/390
PrintWay
RACF
RMF
RS/6000
S/390
SP
SP1
System/390
TeamConnection
VisualAge
VM/ESA
VTAM
WebSphere
XT
400
The following terms are trademarks of other companies:
C-bus is a trademark of Corollary, Inc. in the United States and/or other
countries.
Java and all Java-based trademarks and logos are trademarks or registered
trademarks of Sun Microsystems, Inc. in the United States and/or other countries.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of
Microsoft Corporation in the United States and/or other countries.
PC Direct is a trademark of Ziff Communications Company in the United States
and/or other countries and is used by IBM Corporation under license.
ActionMedia, LANDesk, MMX, Pentium and ProShare are trademarks of Intel
Corporation in the United States and/or other countries.
294
Debugging UNIX Applications on OS/390
UNIX is a registered trademark in the United States and/or other countries
licensed exclusively through X/Open Company Limited.
SET and the SET logo are trademarks owned by SET Secure Electronic
Transaction LLC.
Other company, product, and service names may be trademarks or service marks
of others.
Special notices
295
296
Debugging UNIX Applications on OS/390
Appendix C. Related publications
The publications listed in this section are considered particularly suitable for a
more detailed discussion of the topics covered in this redbook.
C.1 International Technical Support Organization publications
For information on ordering these ITSO publications see “How to get ITSO
Redbooks” on page 301.
• OS/390 Version 2 Release 6 UNIX System Services Implementation and
Customization, SG24-5178
• File Server Consolidation on S/390, SG24-5330
• Global Server Certificate Usage with OS/390 Webserver, SG24-5623
• OS/390 Version 2 Release 4 Performance for CICS Web-Enabled
Applications, SG24-5612
• IMS e-business Connect Using the IMS Conenctors, SG24-5427
• Developing an e-business Application for the IBM WebSphere Application
Server, SG24-5423
• e-business Application Solutions on OS/390 USing Java: Samples,
SG24-5365
• e-business Application Solutions on OS/390 Using Java: Volume 1,
SG24-5342
• OS/390 Workload Manager Implementation and Exploitation, SG24-5326
• Accessing DB2 for OS/390 Data from the World Wide Web, SG24-5273
• CICS Transaction Server for OS/390: Web Interface and 3270 Bridge,
SG24-5243
• Internet Security in the Network Computing Framework, SG24-5220
• Ready for e-business: OS/390 Security Server Enhancements, SG24-5158
• Net.Commerce for OS/390, SG24-5154
• IP Network Design Guide, SG24-2580
• Connecting Domino to the Enterprise Using Java, SG24-5425
• OS/390 eNetwork Communications Server V2R7 TCP/IP Implementation
Guide Volume 1: Configuration and Routing, SG24-5227
• OS/390 eNetwork Communications Server V2R7 TCP/IP Implementation
Guide Volume 2: UNIX Applications, SG24-5228
• OS/390 eNetwork Communications Server V2R7 TCP/IP Implementation
Guide Volume 3: MVS Applications, SG24-5229
© Copyright IBM Corp. 1999
297
C.2 Redbooks on CD-ROMs
Redbooks are also available on the following CD-ROMs. Click the CD-ROMs
button at http://www.redbooks.ibm.com/ for information about all the CD-ROMs
offered, updates and formats.
CD-ROM Title
System/390 Redbooks Collection
Networking and Systems Management Redbooks Collection
Transaction Processing and Data Management Redbooks Collection
Lotus Redbooks Collection
Tivoli Redbooks Collection
AS/400 Redbooks Collection
Netfinity Hardware and Software Redbooks Collection
RS/6000 Redbooks Collection (BkMgr Format)
RS/6000 Redbooks Collection (PDF Format)
Application Development Redbooks Collection
IBM Enterprise Storage and System Management Solutions
Collection Kit
Number
SK2T-2177
SK2T-6022
SK2T-8038
SK2T-8039
SK2T-8044
SK2T-2849
SK2T-8046
SK2T-8040
SK2T-8043
SK2T-8037
SK3T-3694
C.3 Other publications
These publications are also relevant as further information sources:
• OS/390 MVS Installation Exits, SC28-1753
• OS/390 MVS System Commands, GC28-1781
• OS/390 MVS Diagnosis: Tools and Service Aids, SY28-1085
• OS/390 MVS Interactive Problem Control SYstem (IPCS) Commands,
GC28-1754
• OS/390 MVS Initialization and Tuning Guide, SC28-1751
• OS/390 MVS System Messages, Volume 2 Part 1 (ASB–CBR), GC28-1785
• OS/390 SMP/E Reference, SC28-1806
• Understanding Distributed Computing Environment Concepts, GC09-1478
• OS/390 DCE Introduction, GC28-1581
• OS/390 DCE Planning, SC28-1582
• OS/390 DCE Configuring and Getting Started, SC28-1583
• OS/390 DCE Command Reference, SC28-1585
• OS/390 DCE Administration Guide, SC28-1584
• OS/390 DCE Messages and Codes, SC28-1591
• OS/390 DCE Application Development Guide: Core Components, SC28-1588
• OS/390 DCE Application Development Guide: Directory Services, SC28-1589
• OS/390 DCE Application Development Reference Volumes 1 and 2,
SC28-1590
• OS/390 C/C++ Run-Time Library Reference, SC28-1663
• OS/390 Language Environment for OS/390 and VM Run-Time Migration
Guide, SC28-1944
• OS/390 OpenEdition DCE User’s Guide, SC28-1586
298
Debugging UNIX Applications on OS/390
• OS/390 Language Environment for OS/390 Customization, SC28-1941
• OS/390 Language Environment for OS/390 & VM Programming Guide,
SC28-1939
• OS/390 Language Environment for OS/390 & VM Programming Reference,
SC28-1940
• OS/390 Language Environment for OS/390 & VM Debugging Guide and
Run-Time Messages, SC28-1942
• IBM Language Environment for MVS & VM Debugging Guide and Run-Time
Messages, SC26-4829
• OS/390 OpenEdition DCE Security Server Overview, GC28-1938
• Interactive System Productivity Facility (ISPF) Planning and Customizing,
SC28-1298
• MVS Diagnosis: Reference, LY28-1084 (available to IBM license customers
only)
• OS/390 UNIX System Services Messages and Codes, SC28-1908
• OS/390 UNIX System Services User's Guide, SC28-1891
• OS/390 UNIX System Services Planning, SC28-1890
• OS/390 UNIX System Services Command Reference, SC28-1892
• OS/390 Version 2 Release 6 UNIX System Services Implementation and
Customization, SG24-5178
• OS/390 V2R6 Print Server User’s Guide for OS/390 UNIX System Services,
S544-5543
• IBM IP PrintWay Guide, S544-5379
• IBM NetSpool Guide, G544-5301
• OS/390 Print Interface Configuration Guide, G544-5544
• OS/390 Print Server Overview, G544-5545
• OS/390 Print Server User’s Guide for Windows, S544-5511
• OS/390 Distributed File Service Configuring and Getting Started, SC28-1722
• OS/390 Distributed File Service Administration Guide and Reference,
SC28-1720
• OS/390 Distributed File Service Messages and Codes, SC28-1724
• OS/390 Network File System Performance Tuning Guide, SC26-7255
• OS/390 NFS Customization and Operation, SC26-7253
• OS/390 Network File System User's Guide, SC26-7254
• IBM TCP/IP for MVS Customization and Administration Guide, SC31-7134
• OS/390 Open Systems Adapter/Support Facility User’s Guide for OSA-1 and
OSA-2, SC28-1855
• Planning for the OS/390 Open Systems Adapter (OSA-1, OSA-2) Feature,
GC23-3870
• DFSMS/MVS Version 1 Release 5 Planning for Installation, SC26-4919
• DFSMS/MVS Version 1 Release 5 Using Data Sets, SC26-4922
Related publications
299
• DFSMS/MVS Version 1 Release 5 General Information, GC26-4900
• DFSMS/MVS Version 1 Release 5 DFSMSdfp Advanced Services, SC26-4921
• DFSMS/MVS Version 1 Release 5 Implementing System-Managed Storage,
SC26-3123
• DFSMS/MVS Version 1 Release 5 Access Method Services for the Integrated
Coupling Facility, SC26-4906
• OS/390 SecureWay Comuunications Server IP Configuration, SC31-8513
• OS/390 SecureWay Communications Server IP User's Guide, GC31-8514
• OS/390 SecureWay Communications Server IP Programmer's Reference,
SC31-8515
• OS/390 SecureWay Communications Server IP Diagnosis Guide, SC31-8521
• OS/390 SecureWay Communications Server SNA Operation, SC31-8567
• OS/390 Security Server (RACF) Security Administrator’s Guide, SC28-1915
• OS/390 Security Server (RACF) Command Language Reference, SC28-1919
• Lotus Domino for S/390 R4.6 Installation, Customization and Administration,
SG24-2083
• Domino Go Webserver Webmaster’s Guide, SC31-8691
• Domino Go Webserver Messages, SC31-8692
• Novell Network Services for OS/390 Installation, GA22-7312
• Novell Network Services for OS/390 Supervising the Network, SA22-7317
• Novell Network Services for OS/390 Utilities Reference, SA22-7318
• Novell Network Services for OS/390 Introduction to Novell Directory Services,
SA22-7314
• Novell Network Services for OS/390 Concepts, SA22-7313
• Novell Network Services for OS/390 Messages, SA22-7316
• Novell Network Services for OS/390 Schema, SA22-7342
300
Debugging UNIX Applications on OS/390
How to get ITSO Redbooks
This section explains how both customers and IBM employees can find out about ITSO redbooks, redpieces, and
CD-ROMs. A form for ordering books and CD-ROMs by fax or e-mail is also provided.
• Redbooks Web Site http://www.redbooks.ibm.com/
Search for, view, download, or order hardcopy/CD-ROM redbooks from the redbooks Web site. Also read
redpieces and download additional materials (code samples or diskette/CD-ROM images) from this redbooks
site.
Redpieces are redbooks in progress; not all redbooks become redpieces and sometimes just a few chapters will
be published this way. The intent is to get the information out much quicker than the formal publishing process
allows.
• E-mail Orders
Send orders by e-mail including information from the redbooks fax order form to:
In United States
Outside North America
e-mail address
usib6fpl@ibmmail.com
Contact information is in the “How to Order” section at this site:
http://www.elink.ibmlink.ibm.com/pbl/pbl/
• Telephone Orders
United States (toll free)
Canada (toll free)
Outside North America
1-800-879-2755
1-800-IBM-4YOU
Country coordinator phone number is in the “How to Order” section at
this site:
http://www.elink.ibmlink.ibm.com/pbl/pbl/
• Fax Orders
United States (toll free)
Canada
Outside North America
1-800-445-9269
1-403-267-4455
Fax phone number is in the “How to Order” section at this site:
http://www.elink.ibmlink.ibm.com/pbl/pbl/
This information was current at the time of publication, but is continually subject to change. The latest information
may be found at the redbooks Web site.
IBM intranet for employees
IBM employees may register for information on workshops, residencies, and redbooks by accessing the IBM
Intranet Web site at http://w3.itso.ibm.com/ and clicking the ITSO Mailing List button. Look in the Materials
repository for workshops, presentations, papers, and Web pages developed and written by the ITSO technical
professionals; click the Additional Materials button. Employees may access MyNews at http://w3.ibm.com/ for
redbook, residency, and workshop announcements.
© Copyright IBM Corp. 1999
301
IBM Redbook fax order form
Please send me the following:
Title
Order Number
First name
Last name
Company
Address
City
Postal code
Country
Telephone number
Telefax number
VAT number
Card issued to
Signature
Invoice to customer number
Credit card number
Credit card expiration date
We accept American Express, Diners, Eurocard, Master Card, and Visa. Payment by credit card not
available in all countries. Signature mandatory for credit card payment.
302
Debugging UNIX Applications on OS/390
Quantity
Index
E
A
env command
environmental
ETC 241
Access log 59
Agent log 59, 109
APF 252
F
B
BPAM 262
BPXPRMxx 186 , 200, 248
BPXPRMxx parmlib member
BPXSTOP 263
52
248
FDDI 239, 240
FILESYSTYPE 130, 245, 262
Filesystype 259
Fixup 81
FORKCOPY 248
130
C
G
Cache Accelerator 227
Calendaring and Scheduling 111
cconsole program 61
CEEDUMP debugging 284
Cgi-error log 60
compact databases 77
compacting 75
Copy-style compacting 76
ctrace 260
GFSAUXF
245
H
HFS 239, 244
http 249
HTTP task 86
HummingBird 249
I
IEFUSI 252, 263
Infoprint Server 265
In-place compacting 76
IP PrintWay 267
IPX protocol 129
D
DCE Audit Services 199
DCE cell 190
DCE components 196
DCE daemons 192, 193
DCE debug tracing 193
DCE Directory Service 197
DCE Distributed Time Service 197
DCE function trace 194
DCE Remote Procedure Call 198
DCE Security Service 196
DCEKERN address space 192
debug router problems 106
DEBUG9 249
debug9 trace 250
define DCEKERN to RACF 187
df command 52
DFS 199
DFS daemons 205, 206
DFS installation 200
DFS security 207
DFSMS 247, 249
Distributed Computing Environment (DCE)
Distributed File Service (DFS) 199
DOMCON 35
Domcon 34
Domino console 61
Domino Directory 66
Domino Go Webserver 211
DOMLOG 59
dsrepair 181
© Copyright IBM Corp. 1999
L
Language Environment 275
LE run-time options 275
logs for the WebSphere Application Server
Lotus Domino for S/390 33
215
M
Maximum 248
multivolume 247
MVSNFS procedure
258
N
185
NDS 125
NetSpool 267
NetWare Directory Services (NDS)
NFS procedure 242
NFS server 249
NLSPATH 134
notes.ini file 83
Noteslog 61
Novell Directory Services 128
Novell Network Services 125
nsd.sh shell script 45
125
303
O
Obey 242
OMVS 242
OS/390 UNIX System Services
definition 1
P
PATH 134
pcnfsd 241, 245, 252
PING 251
POP3 server 105
PORTMAP 241
Print Interface 266
ps command 53
R
RACF performance suggestions
recovery 181
Referer log 60
replicas 126
229
S
server logs 214
ServletExpress 233
ServletExpress support 217
SETOMVS command 40
Simple Mail Transfer Protocol 96
SMF exit IEFUSI 36
SMTP 96
SMTP server 105
Sockets 239, 241
SSL 230
SSL and hardware encryption 230
STARTED class definitions 188
Starting DFS daemons 203
starting the NFS server 245
T
TCP 239, 240, 241, 262
Tell commands 75
trace output from DCEKERN 195
transaction logging 100
Troubleshooting aids 115
TSO/E MOUNT command 246
U
UNIX commands 51
Update and Updall tasks
79
W
Web server (HTTP) crashes 103
Web server's performance 225
WebSphere Application Server 211, 233
WebSphere Manager applet 234
Windows 239, 249
304
Debugging UNIX Applications on OS/390
ITSO Redbook evaluation
Debugging UNIX Applications on OS/390
SG24-5613-00
Your feedback is very important to help us maintain the quality of ITSO redbooks. Please complete this
questionnaire and return it using one of the following methods:
• Use the online evaluation form found at http://www.redbooks.ibm.com/
• Fax this form to: USA International Access Code + 1 914 432 8264
• Send your comments in an Internet note to redbook@us.ibm.com
Which of the following best describes you?
_ Customer _ Business Partner
_ Solution Developer
_ None of the above
_ IBM employee
Please rate your overall satisfaction with this book using the scale:
(1 = very good, 2 = good, 3 = average, 4 = poor, 5 = very poor)
Overall Satisfaction
__________
Please answer the following questions:
Was this redbook published in time for your needs?
Yes___ No___
If no, please explain:
What other redbooks would you like to see published?
Comments/Suggestions:
© Copyright IBM Corp. 1999
(THANK YOU FOR YOUR FEEDBACK!)
305
Printed in the U.S.A.
Debugging UNIX Applications on OS/390
SG24-5613-00
SG24-5613-00