EView/400i Insight for iSeries (AS/400)
Splunk Integration
Installation and Administration Guide
Software Version:
7.0
July 2015
Copyright 2015 EView Technology, Inc.
Legal Notices
Warranty
EView Technology makes no warranty of any kind with regard to this manual, including, but not limited
to, the implied warranties of merchantability and fitness for a particular purpose. EView Technology
shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential
damages in connection with the furnishing, performance, or use of this material.
Restricted Rights Legend
All rights are reserved. No part of this document may be copied, reproduced, or translated to another
language without the prior written consent of EView Technology, Inc. The information contained in this
material is subject to change without notice.
Use, duplication or disclosure by the U.S. Government is subject to restrictions as set forth in
subparagraph (c) (1) (ii) of the Rights in Technical Data and Computer Software clause at DFARS
252.227-7013 for DOD agencies, and subparagraphs (c) (1) and (c) (2) of the Commercial Computer
Software Restricted Rights clause at FAR 52.227-19 for other agencies.
EView Technology, Inc.
4909 Green Road
Raleigh, North Carolina 27616
United States of America
Copyright Notices
Copyright 2015 EView Technology, Inc.
No part of this document may be copied, reproduced, or translated into another language without the
prior written consent of EView Technology, Inc. The information contained in this material is subject to
change without notice.
Trademark Notices
EView/400® is a registered trademark of EView Technology, Inc.
iSeries, AS/400 are trademarks of International Business Machines Corporation.
Microsoft®, Windows® is a U.S. registered trademarks of Microsoft Corporation.
UNIX® is a registered trademark of the Open Group.
All other product names are the property of their respective trademark or service mark holders and are
hereby acknowledged.
2
Contents
Table of Contents
Concepts ................................................................................................ 6 About EView/400i Architecture and Data Flow ..................................................... 7 Increasing Productivity...................................................................................... 7 What the EView/400i Agent Does ......................................................................... 8 Forwarding iSeries Messages .......................................................................... 8 Event and Message Buffering........................................................................... 8 Splunk Dashboards ............................................................................................... 9 EView Dashboard ............................................................................................. 9 Installing and De-installing EView/400i ............................................. 11 Installation Requirements ................................................................................... 12 Hardware Requirements ................................................................................. 12 Software Requirements .................................................................................. 12 Obtaining License Keys ...................................................................................... 13 Installing EView/400i on a Windows or Linux Server .......................................... 14 Installation Steps for Windows........................................................................ 14 Installation Steps for Linux.............................................................................. 14 Installing EView/400i on the iSeries Agent ......................................................... 15 Installing the Library ..................................................................................... 15 Running the Installation Program ................................................................. 16 Start the EVSBS Subsystem ........................................................................ 17 Cleanup of Temporary Files ......................................................................... 17 Installing the EView/400i Splunk Application ...................................................... 19 Configuring the Splunk Forwarder ...................................................................... 20 Stopping the EVSBS Subsystem ........................................................................ 21 3
Contents
De-installing EView/400i ..................................................................................... 21 To Remove EView/400i Components from the Splunk Forwarding Server .... 21 To Remove EView/400i from the iSeries systems .......................................... 21 Configuring EView/400i....................................................................... 22 Phase 1: Add iSeries Node Configuration........................................................... 23 Phase 2: Add, Modify, and Distribute Message Queues and Message IDs ....... 29 Configure Message Queues ........................................................................... 29 Configure Message ID Filters ......................................................................... 31 Message Queue Filters ................................................................................ 31 QHST Filters ................................................................................................. 34 Phase 3: Identify Command Audit Filters ............................................................ 34 Using EView/400i ................................................................................. 36 Collecting iSeries Messages on the Splunk Forwarding Server ......................... 37 Collecting Performance Data .............................................................................. 37 Troubleshooting EView/400i .............................................................. 38 General Troubleshooting..................................................................................... 39 Use EVSTATUS Command to Verify Status of iSeries Agent ........................ 39 Specific Troubleshooting ..................................................................................... 40 Verifying Connectivity and Agent Operation ................................................... 40 EView/400i Agent Jobs ....................................................................... 42 EView/400i Subsystem (EVSBS) ........................................................................ 43 Message Text of Audit Journal Entries ............................................. 45 Audit Journal Type AD (Auditing changes) ..................................................... 46 Audit Journal Type AF (Authority failure) ........................................................ 46 Audit Journal Type AU (Attribute changes) .................................................... 48 Audit Journal Type CA (Authority changes).................................................... 48 Audit Journal Type CD (Command string) ...................................................... 49 Audit Journal Type CO (Create Object) .......................................................... 49 Audit Journal Type CP (User profile changed, created, or restored) .............. 50 Audit Journal Type DO (Delete Operation) ..................................................... 50 4
Audit Journal Type DS (DST security password reset)................................... 51 Audit Journal Type NA (Network Attribute Change) ....................................... 51 Audit Journal Type OW (Object ownership changed)..................................... 52 Audit Journal Type PA (Program changed to adopt authority) ....................... 52 Audit Journal Type PG (Change of an object's primary group) ...................... 53 Audit Journal Type PW (Invalid password) ..................................................... 53 Audit Journal Type ST (Use of service tools) ................................................. 54 Audit Journal Type SV (System value changed) ............................................ 55 Audit Journal Type VA (Changing an access control list) ............................... 55 Audit Journal Type VP (Network password error) .......................................... 55 Audit Journal Type VU (Changing a network profile) ...................................... 56 Audit Journal Type ZC (Object accessed (changed)) ..................................... 56 Audit Journal Type ZR (Object accessed (read)) ........................................... 57 Performance Collection Metrics Classes .......................................... 59 Selecting Performance Metrics ........................................................................... 60 PERFDATA1................................................................................................... 60 PERFDATA2................................................................................................... 64 5
1
Concepts
This chapter describes EView/400i Insight (EView/400) and provides a brief overview of
its benefits, architecture, and data flow.
6
About EView/400i Architecture and Data Flow
EView/400i consists of two main components: the agent component that runs on the
iSeries (AS/400) server, and the server component that runs on the EView Splunk
Collector server. Events and performance data are forwarded from the agent to the
EView Splunk Collector and written to a file that is monitored by a standard Splunk
forwarder. The EView Splunk Collector sends data to the Splunk server where the
EView/400i Splunk app maps data from common event fields. The EView/400i Splunk
app contains dashboards to help get you started in viewing iSeries event and
performance data.
Figure 1-1 shows the data flow between the iSeries, the EView/400 Splunk Collector and
the Splunk server.
Figure 1-1: EView/400i Data Flow
Increasing Productivity
Consolidating the events of mainframes and other systems with Splunk enables you to
act proactively and quickly analyze data from all of your enterprise systems. Using this
intuitive and cost-effective solution as the central end-user interface provides the basis
for enterprise problem analysis.
7
What the EView/400i Agent Does
The EView/400i agent operates as a subsystem with multiple jobs. iSeries messages are
collected by the agent from several sources, outlined below. Pre-defined messages filters
identify important messages that are then packaged into a common data structure and
forwarded via TCP/IP to the Splunk server for processing.
Forwarding iSeries Messages
Messages can include information from the following:
§
System Operator Message Queue (QSYSOPR)
§
Application Message Queues
§
History Log (QHST)
§
System Audit Journal
§
System Performance Data
Event and Message Buffering
If event, message, or performance data cannot be sent to the EView Splunk Collector for
any reason, the EView/400i agent will save or buffer the data until the connection from
the EView Splunk Collector is available. This ensures that important data will not be
lost.
8
Concepts
Splunk Dashboards
The EView/400i Splunk app contains default dashboards to provide examples of different
ways iSeries data can be viewed as the information is seamlessly integrated into Splunk.
EView Dashboard
The EView Performance and Event Dashboard shows performance data and system
audit events.
Figure 1-2: EView Performance and Event Dashboard
9
2
Installing and De-installing EView/400i
This chapter describes how to install and de-install EView/400i Insight for Splunk
(EView/400).
EView/400i Insight for Splunk consists of two components. The “Client” component is
installed on a Windows or Linux server where a Splunk forwarder is installed. The
“Agent” component is installed on each iSeries (AS/400) operating system partition that
will be sending event and performance data.
The EView/400i Insight for Splunk is installed first on a Windows or Linux server and
includes the Agent software installation file which is transferred to the iSeries partitions
for installation.
11
Installation Requirements
This section describes the operating system, hardware, and software requirements for
installing EView/400i software. To avoid problems during installation, read this section
before you start the installation process.
Hardware Requirements
•
•
EView Splunk Collector
-
Intel 64-bit architecture
-
Appropriate Ethernet hardware on the client to communicate via TCP/IP
iSeries (AS/400) Agent
-
Appropriate Ethernet hardware on the iSeries to allow for TCP/IP
communication with the EView Splunk Collector
In addition, make sure that the EView Splunk Collector and iSeries partitions meet the
disk space requirements described in Table 2-1.
Table 2-1: Additional Disk-Space Requirements
Platform
Disk Space
EView Splunk Collector
5MB
iSeries
50MB
Software Requirements
•
On the EView Splunk Client:
−
Windows Client:
- Microsoft Windows 2008 R2 or later
−
Linux Client:
- Linux 64-bit kernel Version 2.6.24 or later
- Perl Version 5.8 or later
- glibc Version 2.7 or later
−
The TCP/IP network protocol stack must be active.
All other software requirements are the same as the requirements for a Splunk
forwarding server.
•
On the iSeries agent:
−
System i OS V5.1 or later
−
The TCP/IP network protocol stack must be active.
12
Obtaining License Keys
EView/400i requires a license key to be applied to the configuration of each iSeries
system that will be configured on the EView Splunk Collector. One license is required for
each physical iSeries system. The same license key may be used for multiple LPARs on
the same physical system. Contact EView Technology at +1-919-878-5199 or e-mail
support@eview-tech.com to get the necessary license keys. Be prepared to give the
serial number and processor group of the iSeries system. The serial number can be found
by issuing the DSPSYSVAL QSRLNBR command on the iSeries system. The processor
group can be found by issuing the WRKLICINF OUTPUT(*) command on the iSeries.
13
Installing EView/400i on a Windows or Linux Server
The EView/400i installation program is run as an executable on a Windows server or
installed using the Linux RPM install process on a Linux server.
Installation Steps for Windows
1. Copy the EView/400i Insight installation executable to the server where it is to be
installed.
2. Double-click Eview400InsightInstall.exe.
3. The installation process copies the necessary files to the Splunk forwarding server in
the directory path you specify. The default path for EView/400i files is:
\Program Files\EView Technology\EView 400\
Installation Steps for Linux
1. Copy the EView/400i Insight rpm file to the Splunk forwarding server where it will
be installed.
2. Run the Linux rpm command:
rpm --install /tmp/EView400INSIGHT-7-0.x86_64.rpm
where “/tmp” is the directory where the rpm file was saved.
After the rpm command is run, the vp400conf service will start which will allow access
to the web browser configuration application. (See “Phase 1: Add iSeries Node
Configuration” on page 23.)
14
Installing and De-installing EView/400i
Installing EView/400i on the iSeries Agent
This section explains how to start the EView/400i installation process on the iSeries
agent using the following steps:
1.
Library installation
2.
Running the Install Program
3.
Start the EVSBS Subsystem
4.
Cleanup of Temporary Files
Installing the Library
Use the EDTLIBL command to verify that the EVIEW library is not in your
library list on the iSeries agent.
Follow these steps to load the agent components of EView/400i:
1. Sign on to the iSeries system as QSECOFR or other user with *ALLOBJ authority.
2. Create a temporary save file named EVREL70 in any available library (e.g., QGPL)
to receive the installation save file:
CRTSAVF FILE(libname/EVREL70)
3. On the Splunk forwarding server, change directory to the as400 directory:
- On Windows: cd \Program Files\EView Technology\EView 400\as400
- On Linux:
cd /opt/OV/vp400/as400
then start an ftp session to the iSeries system. Set the file type to binary, then
change directory to the library name of the save file created in Step 2. Use the put
command to place the library on the iSeries agent.
# cd /opt/OV/vp400/as400
# ftp iSeriesName
User: qsecofr
Password: ****
ftp> bin
ftp> cd libname
ftp> put EVREL70.SAVF
ftp> quit
4. Restore the EView/400i library on the iSeries (a temporary library named EVREL70
will be created):
RSTLIB SAVLIB(EVREL70) DEV(*SAVF) SAVF(libname/EVREL70) RSTLIB(EVREL70)
15
Running the Installation Program
From an iSeries command line, enter the following command to create the EView/400i
runtime library, EVIEW:
EVREL70/EVINSTALL
Press F4 to see the installation options, or use the defaults described below:
Table 3-2 EVINSTALL Options
16
Parameter Keyword
Default
Description
MMS Port
MMSPORT
9000
The TCP/IP port number which will be opened
and
listened on for connections from the MMS
process
on the forwarding server. Enter any unused
port
number between 1024 and 49151. This number
will be ignored if the installation is upgrading a
previous EView/400i version.
CS Port
CSPORT
9001
The TCP/IP port number which will be opened
and listened on for connections from the CSS
process
on the forwarding server. Enter any unused
port
number between 1024 and 49151. This number
will be ignored if the installation is upgrading a
previous EView/400i version.
HCI Port
HCIPORT
9002
The TCP/IP port number which will be used for
inter-process communications by jobs in the
EVIEW subsystem. Enter any unused port
number
between 1024 and 49151. This number will be
ignored if the installation is upgrading a
previous
EView/400i version.
Backup
Library
BACKUPLIB
QGPL
The name of the library where a backup of the
current EVIEW library will be saved with a
savefile name EVIEW7SAVE. This parameter
will only be used if the installation is upgrading
a
previous EView/400i version. Use “*NONE” to
skip the creation of a backup savefile.
Installing and De-installing EView/400i
Start the EVSBS Subsystem
Start the EVSBS subsystem using one of the following commands:
CALL EVIEW/EVINIT
or:
STRSBS EVIEW/EVSBS
The EVSBS subsystem will start using either the TCP/IP port numbers specified in the
EVINSTALL, or the existing defined port numbers if the installation is an upgrade from
a previous EView/400i version.
Optional PARM values are available for the EVINIT command:
ALL
Start all jobs in the subsystem that have been configured.
This is the default option.
CLEARQ
Clear any buffered messages from the EView message queues
before starting the subsystem jobs.
TEST
Instead of starting the jobs, EVINIT will display the
SBMJOB commands that would be used to start the jobs.
This may be useful to verify that the jobs are being started
with the desired options.
VERSION
Display the version of the installed agent software.
jobname
Start specific job(s) in the subsystem. If a job has fallen into
a Message Wait status, use ENDJOB OPTION(*IMMED) to
stop the individual job, then restart it by specifying the
specific process name in the PARM when calling EVINIT.
Job names are listed in Appendix A, or use the
EVIEW/EVSTATUS PARM('JOBS') command to view which
jobs are not running that should be.
Examples:
To clear the agent’s internal data queues before starting the EVSBS agent subsystem:
CALL EVIEW/EVINIT PARM(CLEARQ)
To start only the EVSHSTPROC, EVSTCPPROC, and EVTCTLPROC jobs:
CALL EVIEW/EVINIT PARM('EVSHSTPROC EVSTCPPROC EVTCTLPROC')
(Specify individual job names only when the EVSBS subsystem is already running.)
Cleanup of Temporary Files
Delete the temporary installation library and save file:
DLTLIB LIB(EVREL70)
17
DLTF FILE(libname/EVREL70)
18
Installing and De-installing EView/400i
Installing the EView/400i Splunk Application
Download the .spl file from the Splunk apps web page at
https://apps.splunk.com/app/2726/
Alternatively, use the eview400i_insight.spl file from the Splunk forwarding
server, found in the following directory:
- On Windows: \Program Files\EView Technology\EView 400\splunkapp
- On Linux:
/opt/OV/vp400/splunkapp
On the Splunk server GUI, go to the Apps page and click the “Install app from file”
button. On the “Upload app” page, identify the location of the EView application file:
Figure 1-1: Upload App
19
Configuring the Splunk Forwarder
Refer to the Splunk documentation to install the forwarder software on the Splunk
forwarding server and connect it to the server/indexer. After the forwarder is installed,
follow these steps to add EView/400i to the list of sources for the forwarder:
1. Edit the inputs.conf file in the following directory:
- On Windows: \Program Files\Splunk\etc\system\local\
- On Linux:
/opt/splunk/etc/system/local/
(Create a new inputs.conf file in this directory if it does not already exist.)
2. Add the following four lines to the end of the inputs.conf and save the file:
On Linux:
[monitor:///var/opt/OV/log/vp400/ev400.insight.*.log]
host_regex = ev400\.splunk\.(.+)\.log
sourcetype = eview-iSeries
disabled = false
On Windows:
[monitor://C:\Program Files\EView Technology\EView 400\log\ev400.insight.*.log]
host_regex = ev400\.splunk\.(.+)\.log
sourcetype = eview-iSeries
disabled = false
(If the EView path was changed during the installation, modify the first line to the
new path.)
3. Restart the Splunk forwarder.
On Linux:
/opt/splunk/bin/splunk restart
On Windows: Restart the Splunk forwarder service “splunkd”.
20
Stopping the EVSBS Subsystem
To terminate a running EView/400i subsystem on the iSeries agent, use the command:
ENDSBS EVSBS *IMMED
The EVSBS subsystem must be ended prior to executing any save commands that would
allocate an EView/400i object, such as when performing a backup.
De-installing EView/400i
This section describes how to remove EView/400i software from the following:
§
Splunk forwarding server
§
iSeries managed nodes
To Remove EView/400i Components from the Splunk Forwarding
Server
On Windows: use the “Add/Remove Programs” utility from Windows Control Panel to
remove EView/390z files and registry entries.
On Linux: use the rpm command:
rpm --erase EView400SPLUNK-7-0.x86_64
To Remove EView/400i from the iSeries systems
To remove EView/400i from the managed nodes, follow these steps:
1. Stop the EView/400i subsystem using the OS/400 command:
ENDSBS EVSBS *IMMED
2. Enter the following commands to delete the EVIEW library from the iSeries system:
CLROUTQ EVIEW/EVCMD
CLROUTQ EVIEW/EVTRACE
CLROUTQ EVIEW/EVHSTOQ
DLTLIB LIB(EVIEW)
3. Enter the following command to delete the EVUSER user profile:
DLTUSRPRF USRPRF(EVUSER)
21
3
Configuring EView/400i
This chapter describes how to configure and start the EView/400i component on the
Splunk forwarding server.
22
Phase 1: Add iSeries Node Configuration
New iSeries nodes to be monitored by Splunk must first be configured using the
EView/400i web configuration interface. The configurator is launched using a web
browser and URL constructed as follows:
http://proxyserver:9850
where “proxyserver” is the hostname or IP address of the Splunk forwarding server
where the EView/400i software was installed. 9850 is the default port number used by
the web configuration interface. If port 9850 is not available the default port number can
be changed by editing the vp400info file in the EView/400i configuration directory and
changing parameter EV400_CONFIG_PORT:
On Windows:
On Linux:
\Program Files\EView Technology\EView 400\parm\ev400info
/etc/opt/OV/share/conf/vp400/vp400info
then restart the configurator service:
On Windows:
On Linux:
Stop and restart the “EView/400i Configurator” service
service vp400conf stop
service vp400conf start
The EView/400i Node Configuration screen is used to add a new iSeries LPAR to be
monitored by Splunk. (Splunk views LPARs as separate nodes, even if they exist on the
same physical box.)
Figure 3-1 EView/400i Configurator
To add iSeries nodes follow these steps:
1.
Start the EView/400i configuration interface from a web browser.
2.
Click on the “Add Node” button and enter the iSeries LPAR’s fully qualified
hostname as defined in your DNS.
23
3.
Highlight the new node and click “Edit Node” to configure the EView/400i server
parameters.
Update the EV400_LICKEY parameter with the license key supplied by EView
Technology for this system.
You can accept the default values created (recommended) or provide custom values
for the configuration parameters. The node parameters are:
Table 3-1 EView/400i Node Parameters
Parameter
Description
EV400_ADDMSG_FIELDS
Indicates whether EView/400i will YES – The EView/400i message YES
send the Program Name and
server will send the “Program
Message
Name” and “Message Type”
Type fields in the messages that are fields in its messages to the
sent to the server.
server.
These fields were added
in the EView/400i Version 2.0 and
will need to be accommodated
in any existing template
conditions that were written
for Version 1.0 of the product.
NO – Use this option if you are
using message template
conditions
from EView/400i Version 1.0 and
do not wish to modify those
existing templates to utilize the
new fields.
EV400_AS400_ADDR
Fully qualified network name of the
iSeries system where the EView/400i
agent component is installed.
EV400_AS400_BIND_ADDR
Address on the agent that the EVSBS IPV4 dotted decimal address in the
subsystem should bind to when opening format nnn.nnn.nnn.nnn. The value
its TCP/IP listening ports (useful when must be a defined address on the
iSeries system
the
iSeries has multiple IP addresses
defined).
0.0.0.0
TCP/IP port number assigned to the
EView/400i Command Server process.
Any unused port number on the
iSeries agent between 1024 and
49151.
9001
TCP/IP port number assigned for
communication between the
EView/400i Message Server process
and Command Server process
8004*
Any unused port number on the
forwarding server between 1025 and
65535.
TCP/IP port number assigned to the
EView/400i Master Message Server
process.
Any unused port number on the
iSeries agent between 1024 and
49151.
EV400_AS400_CMD_PORT
EV400_AS400_CMD_RSP_PORT
EV400_AS400_MSG_PORT
24
Valid Values
Name of iSeries managed node.
Default Value
None
(the
INADDR_ANY
default)
9000
Parameter
EV400_AS400_SERV_ADDR
EV400_AS400_SERVER_PORT
EV400_CMD_CLIENT_PORT
EV400_CMD_SERVER_ADDR
EV400_CMD_TIMEOUT
EV400_LICKEY
EV400_MON_AUDJRNL
EV400_MONITOR_QHST
EV400_MON_RESOURCES
Description
Valid Values
Default Value
Address (or address range) of the
IPV4 dotted decimal address in the 0.0.0.0 – Any
forwarding server(s) that are allowed to format nnn.nnn.nnn.nnn, optionally address may
connect to the iSeries agent. Use a “/” followed
connect to the
followed by a CIDR prefix or subnet
by a slash and either a dotted decimallistening
mask
address representation of a subnet EView/400i
to specify a range of allowed addresses. mask
ports
or a number (0-32) representing the
number of bit positions to use for the
mask.
A TCP/IP port number reserved for
inter-process communications on the
iSeries
agent.
Any unused port number on the
iSeries agent between 1025 and
49151.
9002
A TCP/IP port number used by the
Any unused port number on the
8003*
Command Server process to
forwarding server between 1025 and
communicate
65535.
with the Master Message Server
process.
This port number must be unique on the
forwarding server where the Command
Server and Master Message Server
processes are running.
The name of the forwarding server
where the Command Server
process is to run.
A DNS-recognized server name.
The forwarding
server name
The amount of time to wait for an
An integer greater than or
iSeries command response (in seconds). equal to 1 (second).
30
License key for the managed node.
To obtain a license key, contact
EView Technology support at
support@eview-tech.com. (See
page 13.)
Valid license key
None
A list of two-character entry types from
the QAUDJRN that should be
forwarded
from the iSeries agent. Entry types are
separated by commas. See iSeries
documentation (such as the iSeries
Security Reference) or Appendix B
for descriptions of journal entry types.
AD,AF,AU,CA,CD,CO,CP,DO,DS,
NA,OW,PA,PG,PW,ST,SV,VA,VP,
VU,ZC,ZR
ALL – All of the above
NONE – None of the above
NONE
Indicates whether the EView/400i
YES or NO
agent should monitor for messages that
are sent to the system QHST history
log.
If set to “YES”, then verify that the
EV400_QHST_MON_FREQ field
is greater than 0.
YES
Indicates whether the EView/400i
YES or NO
agent should monitor the status
of iSeries resources (lines, controllers,
and devices). This function is not used
in EView/400i for Windows and should
NO
25
Parameter
Description
Valid Values
Default Value
always be set to "NO".
EV400_MSG_DISTRIB
EV400_MSG_SERVER_ADDR
EV400_PATH
EV400_PERF1
EV400_PERF1_INT
EV400_PERF2
EV400_PERF2_INT
EV400_PRIMARY_SERVER
26
YES
Should the iSeries agent send its
YES – Send unsolicited iSeries
collected messages to all servers that
messages to all EView/400i servers
are in contact
that are in contact with this agent.
with it? (If “NO”, then specify in the
EV400_PRIMARY_SERVER field whichNO – Send unsolicited messages
server is the primary recipient of
only to the primary server.
messages.)
The name of the forwarding server
A DNS-recognized server name.
where
the Master Message Server process is to
run.
The forwarding
server name.
The EView/400i installation directory
on the forwarding server.
EView/400i home directory
Windows:
\Program
Files\EView
Technology\
EView 400\
Linux:
/opt/OV/vp400
Specifies whether the performance
gathering function will be
activated on the iSeries agent to gather
the data for performance group 1. See
Appendix C for the list of metrics
collected
in group 1.
YES – Activate the performance
gathering function on the iSeries
agent.
NO
NO – Do not activate performance
data gathering for group 1.
The interval, in minutes, at which groupAn integer greater than or equal
1 performance data is collected on the to 1 (minute).
iSeries agent and sent to the forwarding
server. This field is only needed if
EV400_PERF1 is set to "YES".
5
Specifies whether the performance
gathering function will be
activated on the iSeries agent to gather
the data for performance group 2. See
Appendix C for the list of metrics
collected
in group 2.
NO
YES – Activate the performance
gathering function on the iSeries
agent.
NO – Do not activate performance
data gathering for group 2.
The interval, in minutes, at which groupAn integer greater than or equal
2 performance data is collected on the to 1 (minutes).
iSeries agent and sent to the forwarding
server. This field is only needed if
EV400_PERF2 is set to "YES".
30
The fully qualified name of the primary An EView/400i forwarding server
forwarding server to receive messages name
from this agent. Although multiple
servers
may be connected to the iSeries agent at
one time, only the server named here
will receive unsolicited iSeries messages
This field is only necessary when the
null
Parameter
Description
Valid Values
Default Value
EV400_MSG_DISTRIB parameter
is “NO”.
EV400_QHST_MON_FREQ
EV400_WORK_AREA
EV400_VP400CS_TRACE
Frequency (in seconds) that the
An integer greater than or
EView/400i agent collects new messagesequal to 1 (seconds)
from the system QHST history log. This
field is only necessary when the
EV400_MONITOR_QHST
parameter is “YES”.
30
Specifies where EView/400i places
Any existing directory on the
temporary work files on the forwarding forwarding server
server.
/var/opt/OV/
share/tmp/vp400
Set tracing level for the command server 0 - No tracing output enabled
0001 - general program trace enabled
(ev400cs on Windows, vp400cs on
0002 - internal tracing enabled
Linux).
0004 - program detail tracing enabled
Multiple values can be added together
0008 - warning messages enabled
in hexadecimal.
0010 - error tracing enabled
0.
0020 - dump output enabled
0040 - loop tracing enabled
0080 - verify tracing enabled
0 - No tracing output enabled
Set tracing level for the master message
0001 - general program trace enabled
server (ev400mms on Windows,
vp400mms on Linux). Multiple values 0002 - internal tracing enabled
can be added together in hexadecimal. 0004 - program detail tracing enabled
0
0 - No tracing output enabled
Set tracing level for the host command
0001 - general program trace enabled
client. Multiple values can be added
0002 - internal tracing enabled
together in hexadecimal.
0
0 - No tracing output enabled
Set tracing level for the agent message
0001 - general program trace enabled
TCP/IP task (EVCMSG). Multiple
0002 - internal tracing enabled
values
can be added together in hexadecimal. 0004 - program detail tracing enabled
0
EV400_EVCHCI_TRACE
Set tracing level for the agent message 0 - disables tracing
transfer process (EVCHCI)
1 - enables tracing
0
EV400_EVC050_TRACE
Set tracing level for the agent command 0 - disables tracing
processor (EVC050)
1 - enables tracing
0
EV400_EVCQSCAN_TRACE
Set tracing level for the agent message 0 - disables tracing
queue monitor (EVCQSCAN)
1 - enables tracing
0
EV400_VP400MMS_TRACE
0008 - warning messages enabled
0010 - error tracing enabled
0020 - dump output enabled
0040 - loop tracing enabled
0080 - verify tracing enabled
0100 - log messages sent to Splunk
0200 - log performance records
EV400_VP400HOSTCMD_TRACE
0004 - program detail tracing enabled
0008 - warning messages enabled
0010 - error tracing enabled
0020 - dump output enabled
0040 - loop tracing enabled
0080 - verify tracing enabled
EV400_EVCMSG_TRACE
0008 - warning messages enabled
0010 - error tracing enabled
0020 - dump output enabled
0040 - loop tracing enabled
0080 - verify tracing enabled
27
Parameter
Description
Valid Values
Default Value
Set tracing level for the agent
performance monitor process
(EVPERFM)
0 - disables tracing
1 - enables tracing
0
EV400_EVCCTL_TRACE
Set tracing level for the API
interface process (EVCCTL)
0 - disables tracing
1 - enables tracing
0
EV400_EVC070_TRACE
Set tracing level for the agent resource 0 - disables tracing
monitor (EVC070)
1 - enables tracing
0
EV400_EVCCMD_TRACE
Set tracing level for the agent command 0 - disables tracing
TCP/IP process (EVCCMD)
1 - enables tracing
0
EV400_EVHSTPGM_TRACE
Set tracing level for the agent
history log (QHST) monitor
0 - disables tracing
1 - enables tracing
0
Set the maximum log size in 1K
increments for the master message
server (ev400mms)
1-99999 (kilobytes)
3000
Set the maximum log size in 1K
increments for the command server
(ev400cs)
1-99999 (kilobytes)
3000
Set the code page to be used for
converting command responses.
Any codepage supported by the
forwarding server such as
1252 – Latin I
932 – Japanese Shift-JIS
936 – Simplified Chinese
949 – Korean
A value of “UTF-8” indicates that
command output is not converted
using any codepage.
UTF-8
EV400_EVPERFM_TRACE
EV400_VP400MMS_LOGSIZE
EV400_VP400CS_LOGSIZE
EV400_CMDRSP_CODEPAGE
EV400_CMDRSP_ALT_CODEPAGE
EV400_NLS_CCSID
* This
Set an alternate code page to be used forAny valid code page, but
converting command responses when in most cases the default value of
EV400_CMDRSP_CODEPAGE is
437 would be used.
set to UTF-8. This parameter is only
used
with the ev400hostcmd option 81. If
EV400_CMDRSP_CODEPAGE is
not set to UTF-8, this parameter is
ignored.
Set the CCSID for the language library
that is being used as the subsystem
library for the
EView/400i agent subsystem.
437
Any CCSID supported on i5OS (OS/400). 37
Some typical values are:
QSYS2924 English – 37
QSYS2928 French – 297
QSYS2929 German – 273
QSYS2931 Spanish – 284
QSYS2932 Italian – 280
QSYS2962 Japanese – 5026
QSYS2986 Korean – 933
QSYS2989 Simplified Chinese – 935
port number will be incremented automatically for new nodes that are added so that the port numbers remain
unique on the server.
28
4.
Save the parameters for this agent. The Node Configuration program will save the
parameters locally on the forwarding server.
5.
Select any nodes in the list of defined nodes that have the “Distributed?” field
marked as “No” and click the [Distribute...] button to send the configuration
parameters to the iSeries agents. The EVSBS subsystem on the iSeries system
must be running to accept the parameters. If the edited parameters result in a
change to the operation of the agent, restart the EVSBS subsystem.
6.
Click the [Start] button to start the EView/400i server processes for the iSeries
node.
Phase 2: Add, Modify, and Distribute Message Queues
and Message IDs
iSeries messages can be captured from any message queue or the QHST message log.
This section explains how to identify which queues are to be monitored and which
messages should be captured and passed from the EView/400i agent to the Splunk
forwarding server.
Configure Message Queues
1.
Start the Message Queue Configuration utility from the EView/400i Configurator
(Figure 3-1) by clicking on the “Message Queue Configuration” link.
Figure 3-2: Message Queue Configurator
2.
To change the message queues being monitored, add a new configuration group
using the [New] button or edit an existing group using the [Edit] button.
29
Figure 3-3 Editing a Message Queue Group
3.
The QSYSOPR/QSYS queue is listed by default in a group. Use the [New Row]
button to add another line for additional queues to be added to this group. To
delete a listed queue, check the trash can icon to the right of the line. The options
for each queue are:
§
In the Message Queue field, enter the name of the message queue to be
monitored.
§
In the Library field, enter the name of the library where the message queue
resides.
§
Set the Filter option to one of the following:
§
−
YES: if the message ID filters should be applied to messages coming from this
queue, restricting which messages will be forwarded to the server.
−
NO: allow messages to be passed on to the server regardless of their message ID.
−
SEV: allow any non-inquiry messages with a severity equal to or greater than the
Min Sev. field to be forwarded to the server regardless of the message ID.
Messages with a severity less than the Min Sev. value will be forwarded only if
the message ID is in the message ID filter table.
Set the Mode option to one of the following:
−
30
BREAK: to allow EView/400i to set the queue in *BREAK mode. EView/400i
provides a break message-handling program that will be called each time a
new message is written to the queue. Break Mode advantage: instant
processing of incoming messages.
−
SCAN: to have EView/400i scan the queue on the interval (by default, every 5
seconds) to check for new messages. Scan Mode advantage: does not require a
lock on the message queue and can co-exist with other message queue monitoring
programs.
§
Set the Min Sev. field to a numeric value 0-99 indicating the necessary
minimum severity of an incoming message. Messages with a lower severity will
not be passed on to the server, even if matched to a message ID filter. Enter “0”
to allow all messages to be processed, regardless of severity.
§
In the Age Limit field, enter a time limit (in seconds) of how old a message can
be and still be passed on to the forwarding server. This field is only used for
queues that are monitored with the "Scan" mode option (see above). This is
useful during startup of the subsystem on the iSeries agent. When the subsystem
is started for the first time (or if it has been brought down for any length of time),
the Age Limit prevents the agent from sending a flood of old unnecessary
messages to the server.
§
If the Inquiry field is set to "Yes" then all messages in that queue with a
Message Type of Inquiry (messages that ask for a reply) will be forwarded to the
server, regardless of the message ID if the Filter option is set to "Yes".
4.
Click the [Confirm] button when all message queues are added to the
configuration group.
5.
Click the [Assign] button to assign queue configuration groups to iSeries agents.
The same configuration group may be assigned to multiple agents.
6.
Select a queue configuration and click the [Distribute] button to send the list of
monitored queues to the iSeries agent. The EView/400i EVSBS subsystem must
be running on the agent at the time of the distribution.
When the EView/400i agent subsystem is running, it will begin monitoring message
queues defined with Scan mode immediately after the distribution is completed. Queues
defined with Break mode monitoring will begin monitoring after the next time the
EVSBS subsystem is restarted.
Configure Message ID Filters
Message ID filters restrict the number of messages that are sent from the iSeries
agent to the forwarding server and save the server from receiving a flood of
unnecessary messages. Each iSeries agent has two message filters, one for message
queues and one for the QHST message log.
Message Queue Filters
Start the Message Queue Filters application from the EView/400i Configurator
(Figure 3-1) by clicking on the [Message Queue Filters] link.
Figure 3-4: Message Queue Filters
31
1.
32
To change the list of message IDs that are sent to the forwarding server, add a
new filter group using the [New] button or edit an existing one using the [Edit]
button. New filters may also be created by copying an existing filter or the
supplied default filter (default.msg.filter) by selecting an existing filter and using
the [Copy] button.
Figure 3-5: Editing a Message Queue Filter
2.
Enter new message IDs to the list in the open field and click the [Add Msg ID]
button. To delete from the list, click the message ID(s) to remove and click the
[Delete Msg ID] button.
Message IDs must be no more than seven alphanumeric characters, but any
message ID entered can contain the special period character (.) to indicate that
any character in that position should match. If the message ID is terminated with
an asterisk (*), matching will only occur on characters preceding the asterisk. See
the following examples:
Table 0-2: Message Filter Examples
To forward the following
messages:
Enter the following in the list of
Message IDs:
All messages
*
ABC1234
ABC1234
All messages beginning with
“ABC”
ABC*
Any 7-character message
beginning with “ABC” and
ending with “9”
ABC...9
3.
Click [Confirm] button when all message IDs are added to the filter group.
4.
Click the [Assign] button to assign filter groups to iSeries agents. The same filter
group may be assigned to multiple agents.
5.
Select a filter group name and click the [Distribute] button to send the list of
message IDs to the iSeries agent. The EView/400i EVSBS subsystem must be
running on the agent at the time of the distribution.
The EView/400i agent subsystem will begin monitoring with the new message ID filters
immediately after the distribution is completed.
33
QHST Filters
Start the QHST Filters utility from the EView/400i Configurator by clicking on the
[QHST Filters] button.
1.
To change the list of message IDs that are sent to the server, add a new filter
group using the [New] button or edit an existing one using the [Edit] button.
2.
Enter new message IDs to the list in the open field and click the [Add Msg ID]
button. To delete from the list, click the message ID(s) to remove and click the
[Delete Msg ID] button.
3.
Click the [Confirm] button when all message IDs are added to the filter group.
4.
Click the [Assign] button to assign filter groups to iSeries agents. The same filter
group may be assigned to multiple agents.
5.
Select a filter group name and click the [Distribute] button to send the list of
message IDs to the iSeries agent. The EView/400i EVSBS subsystem must be
running on the agent at the time of the distribution.
The EView/400i agent subsystem will begin monitoring with the new message ID filters
immediately after the distribution is completed.
Phase 3: Identify Command Audit Filters
The Command Audit Filters work with the iSeries’ QAUDJRN audit journal to
determine which audit entries of type CD (Command) will be forwarded to the server. If
an iSeries user’s profile is set up (using CHGUSRAUD) to journalize the user’s issued
commands, the Command Audit Filters can be used to reduce the number of journal
entries that are forwarded to the server.
Note that this section is only necessary if the “CD” value is specified in the
EV400_MON_AUDJRNL parameter for this node.
1. To change the list of commands that are sent to the forwarding server, add a new
filter group using the [New] button or edit an existing one using the [Edit] button.
2. Enter new commands to the list in the open field and click the [Add Command]
button. To delete from the list, click the command(s) to remove and click the
[Delete Command] button.
3. Click the [Save and Close] button when all commands are added to the filter
group.
34
Figure 3-6: Editing the Command Audit Filters
4. Click the [Assign] button to assign filter groups to iSeries agents. The same filter
group may be assigned to multiple agents.
5. Select a filter group name and click the [Distribute] button to send the list of
commands to the iSeries agent. The EView/400i EVSBS subsystem must be
running on the agent at the time of the distribution.
See Appendix B for the displayed format of the CD and other audit journal
command types.
35
Phase 3: Identify Command Audit Filters
4
Using EView/400i
This chapter describes how to use EView/400i Insight to capture mainframe messages
and forward them to Splunk.
36
Using EView/400i
Collecting iSeries Messages on the Splunk
Forwarding Server
The EView/400i component on the Splunk forwarding server writes mainframe messages
to a log file in one of the following directories:
on Windows:
\Program Files\EView Technology\EView 400\log
on Linux:
/var/opt/OV/log/vp400
The log’s file name will include the name of the iSeries system that is being monitored.
Each line of the log file will contain a timestamp, source prefix, and message text. The
source prefix indicates where the message was generated on the iSeries:
*OS400 MSG Message from an iSeries message queue or the QHST history log
or System Audit Journal
*PERFDATA1 Performance data from Group 1
*PERFDATA2 Performance data from group 2
These source prefixes will be interpreted by the Splunk server when displaying the
messages.
Collecting Performance Data
If the optional performance job EVPERFPROC is running on the iSeries agent,
*PERFDATA1 and/or *PERFDATA2 lines will be sent to the Splunk server for analysis
at the desired interval. See Appendix C for the description of metrics collected.
37
Collecting Performance Data
5
Troubleshooting EView/400i
This chapter describes how to troubleshoot problems with EView/400i.
38
Troubleshooting EView/400i
General Troubleshooting
Before you troubleshoot a particular problem you run into when using EView/400i,
you should verify that your EView/400i environment is correctly installed and
configured.
Correct installation and configuration of EView/400i ensures, among other things,
that messages are processed correctly:
§
Message Capture
Messages are collected by the EView/400i agent from the several sources on the
iSeries system.
§
Message Transmission
Messages are sent to the EView/400i server component on the Splunk forwarding
server.
Use EVSTATUS Command to Verify Status of iSeries Agent
On the iSeries agent, use the command EVIEW/EVSTATUS to collect the status of the
several components of the EView/400i agent and their interaction with the iSeries
system. The command is called from an iSeries (5250) terminal. The format is:
EVIEW/EVSTATUS PARM('options') OUTPUT(outoption)
where:
options
One or more of the following, separated by spaces:
VER
CONF
JOBS
TCP
DQS
AUD
options
USP
SYS
ALL
? or HELP
outoption
EView/400i version information
Current distributed configuration files
Status of EVSBS jobs
Defined TCP/IP ports and current status
Data queues status
System QAUDLVL vs. EView/400i audit
Defined user spaces
iSeries system information
All of the above (Default)
Display help options
One of: * For output to a terminal *PRINT For output to the user's print queue
(Default) Example call:
EVIEW/EVSTATUS PARM('JOBS TCP SYS') OUTPUT(*)
39
Specific Troubleshooting
Browse the output text of this command and look for NOTE or WARNING messages
that may indicate how to resolve outstanding problems. Retain a copy of the output
for possible transmission to support personnel.
Specific Troubleshooting
This section explains how to solve specific problems you may encounter when using
EView/400i.
Verifying Connectivity and Agent Operation
Symptom
No apparent communication between the Splunk forwarding server and the iSeries
agent.
Solution
To verify the correct operation of the server and agent components, use the following
steps:
On the forwarding server:
1. Start the EView/400i web configurator interface. Verify that all processes are
running for that agent. If a node's Command Server is running but the
Master Message Server is not, this is usually due to an incorrect license key.
Check the ev400mms log file for this error (step 3 below).
2. Check the status of the TCP/IP ports used to connect to the agent. For
example, if the default ports (9000 and 9001) are used, issue the command
netstat –a and look for ports 9000 and 9001 to have a status of
"Established".
3. Check for errors in the ev400mms.as400name.log and
ev400cs.as400name.log files
On Windows: in the \Program Files\EView Technology\EView 400\log
directory
On Linux: in the /var/opt/OV/log/vp400 directory.
On the iSeries managed node:
1. Enter the EVIEW/EVSTATUS command as described on page 39. Look for any
“Note” or “Warning” messages in the output which may indicate a problem.
40
Troubleshooting EView/400i
2. Issue the command WRKACTJOB SBS(EVSBS)
The following six jobs should be listed in an active (not "Message Wait") status:
EVACMDPROC PGM-EVCCMD
EVCCTLPROC PGM-EVCCTL
EVSCMDPROC PGM-EVC050
EVSMSGPROC PGM-EVC010
EVSTCPPROC PGM-EVCHCI
EVTCTLPROC PGM-EVCMSG
If the QHST monitoring option was selected in the iSeries node's
configuration (EV400_MONITOR_QHST parameter is "YES"), then verify the
additional job is active:
EVSHSTPROC PGM-EVHSTCL
If the performance monitoring option was selected in the iSeries node's
configuration (EV400_PERF1 and/or EV400_PERF2 parameter is "YES"),
then verify the additional job is active:
EVPERFPROC PGM-EVPERFM
Check the agent message queue for any errors that may have been issued:
DSPMSG EVIEW/EVLOGQ
3. Check the agent trace files for any error output. The trace files are in the
EVTRACE output queue of the EVIEW library:
WRKOUTQ EVIEW/EVTRACE
4. Check the status of the TCP/IP ports used by the agent using the command
NETSTAT *CNN If the forwarding server processes are connected, the ports configured in
parameters EV400_AS400_MSG_PORT and EV400_AS400_CMD_PORT (9000 and
9001 by default) should show as "Established". It is normal for these two ports to also be
in a "Listen" state. The port configured in parameter EV400_AS400_SERVER_PORT
must be "Established" before any messages can be sent to the forwarding server.
5. Check the condition of the agent data queues. The agent uses several data queues to
store requests and messages. Data queue objects may become damaged due to
unexpected interruption or system errors, which can cause agent jobs to fail. Issue the
following commands to check the data queues:
ADDLIBLE EVIEW
EVIEW/DDQ EVIEW/EVSENDQ
EVIEW/DDQ EVIEW/EVAPIQ
EVIEW/DDQ EVIEW/EVCMDQ
EVIEW/DDQ EVIEW/EVMRSPQ
If a data queue has been damaged, an exception message will be generated when issuing
the DDQ command for that queue. If the data queue properties are displayed, verify that
the maximum entry length is not zero, which is another indication of a damaged data
queue.
41
Specific Troubleshooting
A
EView/400i Agent Jobs
This appendix describes the various jobs that run under the EVSBS subsystem on the
iSeries.
42
EView/400i Agent Jobs
EView/400i Subsystem (EVSBS)
The jobs that execute in the EVSBS Subsystem:
Job Name
Program
Description
EVACMDPROC
EVCCMD
Establishes the TCP/IP socket for bidirectional command and response link.
EVCCTLPROC
EVCCTL
Controls the processing of pre-defined API's
used in command processing.
EVMSGQMON
EVCQSCAN
Monitors message queues configured for
SCAN mode monitoring.
EVPERFPROC
EVPERFM
Gathers performance data.
EVSCMDPROC
EVC050
Executes the command processor.
EVSMSGPROC
EVC010
Message queue allocation and message
processing.
EVSHSTPROC
EVHSTCL
Extracts messages at a configured time
sequence from the QHST message queue
depending on the message ID's added to the
filter file.
EVSRSCPROC
EVC070
Monitors status changes on discovered
resources
at a configured time sequence.
EVSTCPPROC
EVCHCI
Receives and forwards all processed
messages,
commands, and API output. Manages a
central
data queue that allows for message buffering
in
case the TCP/IP connection to the forwarding
server is lost.
EVTCTLPROC
EVCMSG
Controls multiple connectivity between the
forwarding server(s) and the EView/400i
agent.
EVAUDJRNL
RCVJRNE
The RCVJRNE exit which collects audit
records from the QAUDJRN journal.
43
44
Message Text of Audit Journal Entries
B
Message Text of Audit Journal Entries
This appendix describes how iSeries audit records received from the QAUDJRN will be
presented to the Splunk forwarding server. All journal messages begin with an
“AUD0000” message ID header.
45
Audit Journal Type AD (Auditing changes)
(AD) {cmdname|Undefined} command, Object: objname/libname Type:
objtype Value: audval Level: {actlvl[,actlvl...]|NONE} [DLO Object:
dloobj]
where:
cmdname – The command which triggered this audit entry, one of:
CHGDLOAUD
CHGAUD
CHGATTR
CHGUSRAUD
objname – The name of the object for which auditing was changed.
libname – The name of the library of the object.
objtype – The type of object.
audval – The audit value specified in the command. If the scan attribute was changed using
the CHGATR command, audval contains the scan attribute value.
actlvl – The level of activity that is audited for objname.
dloobj – The DLO object, if one exists.
Sample Message:
AUD0000 (AD) CHGUSRAUD command, Object: USER1/QSYS Type: *USRPRF
Value: *ALL Level: *CMD,*CREATE,*DELETE
Audit Journal Type AF (Authority failure)
(AF) failuretext [Validation Error Action: actiontext]
[(violationcode) violationtext] Object: objname[/libname] [Type:
objtype] Job Name: jobname User Profile: usrprf
where:
failuretext – Description of the authority failure, one of:
Not authorized to object
Restricted instruction
Validation failure:
Use of unsupported interface
Storage protection error
ICAPI authorization error
ICAPI authentication error
Scan exit program action:
System Java inheritence not allowed
Submit job profile error
Profile token not regenerable
Optical object authority failure
Profile swap error
Hardware protection error
Default sign-on attempt
46
Message Text of Audit Journal Entries
Not authorized to TCP/IP port
User permission request not valid
Profile token not valid for generating new token
Profile token not valid for swap
System violation:
Not authorized for a clear JUID operation
Not authorized for a set JUID operation
Undefined violation
actiontext – If failuretext is either "Validation failure: " or "Scan exit
program action: " then this action is taken, one of:
Object translation not attempted or failed
Object translation was successful
System install time error detected
Restore failed, signature not in OS/400 format
Unsigned system or inherit state object found
Unsigned user state object found
Mismatch between object and its signature
IBM certificate not found
Invalid signature format found
Scan exit program modified the object
Scan exit program wanted object marked as failure
Unrecognized action
violationcode, violationtext – If failuretext is "System violation: " then
this describes the type of violation that occurred, one of:
(HCA) Service tool user not authorized for hardware
config
(LIC) PTF not applied due to signature violation
(SFA) Not authorized for system file access
(CMD) Command disabled by sysadmin
objname – The name of the object. If failuretext is "Not authorized to TCP/IP
port", then this field will contain the port number.
libname – The name of the library of the object. This is not displayed if failuretext is
"Not authorized to TCP/IP port".
objtype – The type of object. This is not displayed if failuretext is "Not authorized
to TCP/IP port".
jobname – The name of the job.
usrprf – The name of the user that caused the authority failure.
Sample Message:
AUD0000 (AF) Not authorized to object Object: MYOBJ/MYLIB Type:
*FILE Job Name: QPADEV0001 User Profile: USER1
47
Audit Journal Type AU (Attribute changes)
(AU) [New CSSID: newcssid Old CSSID: oldcssid][, ][New Country ID:
newcountry Old Country ID: oldcountry][, ][New Language ID: newlang
Old Language ID: oldlang][, ][Attribute: attrname New Value: newattr
Old Value: oldattr]
where:
newcssid,oldcssid – The new and old CSSID values, if there was a change.
newcountry,oldcountry – The new and old Country ID values, if there was a change.
newlang,oldlang – The new and old Language ID values, if there was a change.
attrname – The name of the attribute, if there was a change.
newattr,oldattr – The new and old attribute values, if there was a change.
Sample Message:
AUD0000 (AU) New Country ID: DE
Old Country ID: US
Audit Journal Type CA (Authority changes)
(CA) Object: objname/libname User: usrprf Command type: cmdtype
Authorities altered: {auth[,auth...]|NONE}
where:
objname – The name of the object.
libname – The library of the object.
usrprf – The user profile whose authority is being modified.
cmdtype – The type of command used, one of:
Grant
Grant/Replace
Revoke
GRTUSRAUT
auth – The authorities granted or removed, one or more of:
*OBJEXIST
*OBJMGT
*OBJOPR
*AUTLMGT
*AUTL
*READ
*ADD
*UPD
*DLT
*EXCLUDE
48
Message Text of Audit Journal Entries
*EXECUTE
*OBJALTER
*OBJREF
Sample Message:
AUD0000 (CA) Object: OBJ1/MYLIB User: USER1 Command type: Grant
Authorities altered: *ADD,*UPD,*DLT
Audit Journal Type CD (Command string)
(CD) Command: cmdstring issued from job: job/user/jnum CL Program
Call: {Yes|No}
where:
cmdstring – The name of the command executed.
job – The name of the job that caused this entry to be created.
user – The user profile associated with job.
jnum – The job number.
NOTE: To generate a message to the server, the cmdstring must be in the list of commands defined in
“Phase 3: Identify Command Audit Filters” (see page 34).
Sample Message:
AUD0000 (CD) Command: DLTUSRPRF issued from job: USER1/USER1/123456
CL Program Call: No
Audit Journal Type CO (Create Object)
(CO) Object: objname/objlib {created|replaced}, Type: objtype from
job: job/user/jnum
where:
objname – The name of the object.
objlib – The library of the object.
objtype – The type of the object.
job – The name of the job that caused this entry to be created.
user – The user profile associated with job.
jnum – The job number.
Sample Message:
49
AUD0000 (CO) Object: MYOBJ/MYLIB created, Type: *MODULE from job:
QPADEV0003/USER1/123456
Audit Journal Type CP (User profile changed, created, or restored)
(CP) User profile: usrprf changed via method [ (password changed) ]
[Profile status: status] [User class: class] from job: job/user/jnum
where:
usrprf – The user profile that was changed.
method – The type of command used, one of:
CRTUSRPRF command
CHGUSRPRF command
RSTUSRPRF command
QSECOFR password reset using DST
QSYSRESPA API
Undefined method
status – The user profile status, if changed.
class – The user class of the user, if one exists.
job – The name of the job that caused this entry to be created.
user – The user profile associated with job.
jnum – The job number.
Sample Message:
AUD0000 (CP) User profile: USER1 changed via CHGUSRPRF command
Profile status: *ENABLED from job: QPADEV0003/USER1/123456
Audit Journal Type DO (Delete Operation)
(DO) Object: objname/objlib action, Type: objtype from job:
job/user/jnum
where:
objname – The name of the object.
objlib – The library of the object.
action – The type of action taken, one of:
deleted
pending delete committed
pending create rolled back
50
Message Text of Audit Journal Entries
delete pending
pending delete rolled back
objtype – The type of the object.
job – The name of the job that caused this entry to be created.
user – The user profile associated with job.
jnum – The job number.
Sample Message:
AUD0000 (DO) Object: MYOBJ/MYLIB created, Type: *FILE from job:
QPADEV0003/USER1/123456
Audit Journal Type DS (DST security password reset)
(DS) Service Tools User: userid action as requested by requestor
where:
userid – The service tools user ID.
action – The type of action taken, one of:
ID was changed
password reset
password changed
requestor – The service tools user ID that requested the change.
Sample Message:
AUD0000 (DS) Service Tools User USER1 password changed as requested
by QSECOFR
Audit Journal Type NA (Network Attribute Change)
(NA) {Network|TCP/IP} attribute: val changed from: oldval to: newval
from job: job/user/jnum
where:
val – The name of the attribute that was modified.
oldval – The value before it was changed.
newval – The new value.
job – The name of the job that caused this entry to be created.
user – The user profile associated with job.
jnum – The job number.
51
Sample Message:
AUD0000 (NA) TCP/IP attribute: TCPKEEPALV changed from: 120 to: 140
from job: QPADEV0003/USER1/123456
Audit Journal Type OW (Object ownership changed)
(OW) Object: objname/libname ownership changed from: old to: new
from job: job/user/jnum
where:
objname – The name of the object.
libname – The name of the library of the object.
old – The old owner of the object.
new – The new owner of the object.
job – The name of the job that caused this entry to be created.
user – The user profile associated with job.
jnum – The job number.
Sample Message:
AUD0000 (OW) Object: MYOBJ/MYLIB ownership changed from: USER1 to:
USER2 from job: QPADEV0003/USER1/123456
Audit Journal Type PA (Program changed to adopt authority)
(PA) {Program pgmname/libname adopted authority of owner: ownername
| Object: {objname|NONE} [SETUID mode: {Y|N}] [SETGID mode: {Y|N}]}
where:
pgmname – The name of the program that was modified.
libname – The name of the library of the pgmname.
ownername – The name of the owner.
objname – The name of the object, if it exists and if the SETUID or SETGID has been
modified.
Sample Message:
AUD0000 (PA) Program MYPROG/MYLIB adopted authority of owner: USER1
52
Message Text of Audit Journal Entries
Audit Journal Type PG (Change of an object's primary group)
(PG) Object: objname/objlib changed group from: oldgrp to: newgrp
where:
objname – The name of the object for which the group was changed.
libname – The name of the library of the objname.
oldgrp – The previous primary group, or "*N" if the old group was not available.
newgrp – The new primary group for the object.
Sample Message:
AUD0000 (PG) Object MYOBJ/MYLIB changed group from GRP1 to GRP2
Audit Journal Type PW (Invalid password)
(PW) User: username failed: violation on: device [remote name:
remote] [local name: local]
where:
username – The job user name or service tools user ID.
violation – The type of violation, one of:
APPC bind failure
Service Tools ID name not valid
Service Tools ID password not valid
Password invalid
SQL Decryption password not valid
User name not valid
Service Tools user ID disabled
Service Tools ID not valid
Service Tools ID password not valid
Undefined violation
device – The name of the device where the user ID or password was entered. If
violation is one of: "Service Tools user ID disabled ", "Service Tools ID
not valid ", or "Service Tools ID password not valid ", then the device field
will contain the name of the service tool being accessed.
remote – The name of the remote location for the APPC bind, if one exists.
local – The name of the local location for the APPC bind, if one exists.
Sample Message:
AUD0000 (PW) User: USER1 failed: Password invalid on: QPADEV0007
53
Audit Journal Type ST (Use of service tools)
(ST) Service tool type accessed[ object objname/libname][ for job
jobname/username/jobnum]
where:
type – The type of service tool, one of:
ANZJVM
STRCPYSCN
QTACTLDV
QWTCTLTR
DMPCLUTRC
DLTCMNTRC
DMPDLO
DMPJVM
DMPOBJ
DMPSYSOBJ,QTADMPTS
ENDCMNTRC
ENDRMTSPT
QYHCHCOP(DASD)
QYHCHCOP(LPAR)
QPYRTJWA
PRTCMNTRC
PRTERRLOG
PRTINTDTA
QP0FPTOS
QWTSETTR
STRCMNTRC
STRSRVJOB
STRRMTSPT
STRSST
TRCTCPAPP
TRCCNN(*FORMAT)
ENDTRC,ENDPEX
TRCINT,TRCCNN(*ON/*OFF/*END)
STRTRC,STRPEX
UNKNOWN
objname – The object accessed, if given.
libname – The name of the library of the objname.
jobname – Part 1 of the qualified job name, if given.
username – Part 2 of the qualified job name.
jobnum – Part 3 of the qualified job name.
Sample Message:
54
Message Text of Audit Journal Entries
AUD0000 (ST) Service Tool QPOFPTOS accessed object MYOBJ/MYLIB for
job TEST/USER1/123456
Audit Journal Type SV (System value changed)
(SV) System value change: sysval changed from: oldval to: newval
where:
sysval – The system value that was modified.
oldval – The value before it was changed.
newval – The new value.
Sample Message:
AUD0000 (SV) System value change: QAUDLVL changed from: *AUTFAIL
*SYSMGT to: *AUTFAIL *SYSMGT *SECURITY
Audit Journal Type VA (Changing an access control list)
(VA) Access control list {addition|modification|deletion}
{successful|failed} from user username at location for resource
rscname
where:
username – The name of the user issuing the request to change the access control list.
location – The name of the computer issuing the request.
rscname – The name of the resource to be changed.
Sample Message:
AUD0000 (VA) Access control list modification successful from user
USER1 at QPADEV0005 for resource n
Audit Journal Type VP (Network password error)
(VP) User: username network password error on: device
where:
username – The name of the user attempting to log on.
device – The computer initiating the logon request.
55
Sample Message:
AUD0000 (VP) User: USER1 network password error on: DEV1
Audit Journal Type VU (Changing a network profile)
(VU) User: username on device: device requested network profile
action: action for record: rectype resource: rscname
where:
username – The name of the user requesting the profile change.
device – The name of the computer requesting the profile change.
action – The requested action, one of:
addition
change
deletion
incorrect password
undefined
rectype – The type of record changed, one of:
group
user
user profile global information
undefined
rscname – The name of the resource.
Sample Message:
AUD0000 (VU) User: USER1 on device: DEV1 requested network profile
action: change for record: user resource: n
Audit Journal Type ZC (Object accessed (changed))
(ZC) Object: objname/libname type: objtype {changed|upgraded} by
job: job/user/jnum access type: acctype
where:
objname – The object accessed.
libname – The name of the library of the objname.
objtype – The object type of objname.
job – The name of the job that caused this entry to be created.
56
Message Text of Audit Journal Entries
user – The user profile associated with job.
jnum – The job number.
acctype – The type of access, one of:
Add
Activate program
Analyze
Apply
Call or TFRCTL
Configure
Change
Check
Close
Clear
Compare
Cancel
Copy
Create
Convert
Debug
Delete
Dump
Display
Edit
End
File
Grant
Hold
Initialize
List
Send
Move
Start
Merge
Transfer
Open
Trace
Print
Verify
Query
Vary
Reclaim
Work
Receive
Read/change DLO attribute
Read
Read/change DLO security
Reorganize Read/change DLO content
Release
Read/change DLO all parts
Release
Add constraint
Remove
Change constraint
Rename
Remove constraint
Replace
Start procedure
Resume
Get access on *OOPOOL
Restore
Sign object
Retrieve
Remove all signatures
Run
Clear a signed object
Revoke
Mount
Save
Unload
Save with storage free
Save and delete
Submit
End rollback
Set
Undefined: n
Sample Message:
AUD0000 (ZC) Object: MYOBJ/MYLIB type: *FILE changed by job:
QPADEV0003/USER1/123456 access type: Change
Audit Journal Type ZR (Object accessed (read))
(ZR) Object: objname/libname type: objtype read by job:
job/user/jnum access type: acctype
where:
objname – The object accessed.
libname – The name of the library of the objname.
objtype – The object type of objname.
57
job – The name of the job that caused this entry to be created.
user – The user profile associated with job.
jnum – The job number.
acctype – The type of access, one of:
Add
List
Send
Activate program Move
Start
Analyze
Merge
Transfer
Apply
Open
Trace
Call or TFRCTL
Print
Verify
Configure
Query
Vary
Change
Reclaim
Work
Check
Receive
Read/change DLO attribute
Close
Read
Read/change DLO security
Clear
Reorganize Read/change DLO content
Compare
Release
Read/change DLO all parts
Cancel
Release
Add constraint
Copy
Remove
Change constraint
Create
Rename
Remove constraint
Convert
Replace
Start procedure
Debug
Resume
Get access on *OOPOOL
Delete
Restore
Sign object
Dump
Retrieve
Remove all signatures
Display
Run
Clear a signed object
Edit
Revoke
Mount
End
Save
Unload
File
Save with storage free
Grant
Save and delete
Hold
Submit
End rollback
Initialize
Set
Undefined: n
Sample Message:
AUD0000 (ZR) Object: MYOBJ/MYLIB type: *FILE read by job:
QPADEV0003/USER1/123456 access type: Read
58
Performance Collection Metrics Classes
C
Performance Collection Metrics Classes
This appendix lists the performance metrics that can be collected by EView/400i.
59
Selecting Performance Metrics
Use the EView/400i Node Configurator web interface to change the EV400_PERF1 and/or
the EV400_PERF2 parameter to "YES" (see "Error! Reference source not found."
beginning on page 23.) based on the desired metrics classes listed below. Save and
redistribute the modified configuration to the iSeries agent and restart the agent subsystem.
Performance data lines will be labelled *PERFDATA1 and *PERFDATA2 when presented to
the Splunk forwarding server.
PERFDATA1
Performance Group 1 (*PERFDATA1) data metrics in the following order:
Short Name
Description
Unit
Time Stamp
Time stamp in seconds since epoch 00:00 1/1/1970
Integer
Avg Users Signed In
Average number of users signed in over the polling
interval
Integer
Min Users Signed In
Minimum number of users signed on to the system
during the polling interval
Integer
Max Users Signed In
Maximum number of users signed on to the system
during the polling interval
Integer
Avg Global CPU Util
Average percent of the polling interval time during
which the CPUs were in use
Integer, in tenths
Min Global CPU Util
Minimum percent of the polling interval time during
which the CPUs were in use
Integer, in tenths
Max Global CPU Util
Maximum percent of the polling interval time during
which the CPUs were in use
Integer, in tenths
Average total number of user and system jobs that
are currently in the system, including jobs waiting
on queues
Integer
Minimum total number of user and system jobs that
are currently in the system, including jobs waiting
on queues
Integer
Maximum total number of user and system jobs that
are currently in the system, including jobs waiting on
Integer
Avg Jobs in System
Min Jobs In System
Max Jobs in System
60
Performance Collection Metrics Classes
queues
Avg Pct DB Cap
Average percentage of processor database capability
that was used during the polling interval
Integer, in tenths
Min Pct DB Cap
Minimum percentage of processor database capability
that was used during the polling interval
Integer, in tenths
Max Pct DB Cap
Maximum percentage of processor database
capability that was used during the polling interval
Integer, in tenths
Average number of faults over all pools during the
polling interval for pages containing either database
data or access paths
Integer, in tenths
representing
faults per second
Maximum number of faults over all pools during the
polling interval for pages containing either database
data or access paths
Integer, in tenths
representing
faults per second
Average cumulative rate over all pools during the
polling interval at which database pages are brought
into the storage pool
Integer, in tenths
representing
pages per second
Average number of faults over all pools during the
polling interval for pages other than those designated
as database pages
Integer, in tenths
representing
faults per second
Maximum number of faults over all pools during the
polling interval for pages other than those designated
as database pages
Integer, in tenths
representing
faults per second
Average cumulative rate over all pools during the
polling interval at which pages other than those
designated as database pages are brought into the
storage pool
Integer, in tenths
representing
pages per second
Avg Job CPU Util
Average percentage of CPU time used by all batch
jobs during the polling interval
Integer
Min Job CPU Util
Minimum percentage of CPU time used by all batch
jobs during the polling interval
Integer
Max Job CPU Util
Maximum percentage of CPU time used by all batch
jobs during the polling interval
Integer
Avg Int CPU Util
Average percentage of CPU time used by all
interactive jobs during the polling interval
Integer
Avg Database Faults
Max Database Faults
Database Pages
Avg Non DB Faults
Max Non DB Faults
Avg Non DB Pages
61
Min Int CPU Util
Minimum percentage of CPU time used by all
interactive jobs during the polling interval
Integer
Max Int CPU Util
Maximum percentage of CPU time used by all
interactive jobs during the polling interval
Integer
Average number of user interactions, such as
pressing the Enter key or a function key, for all
interactive jobs during the polling interval
Integer
Average interactive response time for the initial
thread of all interactive jobs during the polling
interval
Integer, in
hundredths of
seconds
Maximum interactive response time for the initial
thread of all interactive jobs during the polling
interval
Integer, in
hundredths of
seconds
Avg I/O Per Second
Average number of blocks transferred to and from the
disk units during the polling interval
Integer
Max I/O Per Second
Maximum number of blocks transferred to and from
the disk units per second during the polling interval
Integer
Avg Read Per Second
Average number of blocks transferred from the disk
units per second during the polling interval
Integer
Maximum number of blocks transferred from the disk
units per second during the polling interval
Integer
Average number of blocks transferred to the disk
units per second during the polling interval
Integer
Maximum number of blocks transferred to the disk
units per second during the polling interval
Integer
Average percentage of time that the disk queues of all
disks contained data to read or write during the
polling interval
Integer,
expressing
percentage in
thousandths
Maximum percentage of time that the disk queues of
all disks contained data to read or write during the
polling interval
Integer,
expressing
percentage in
thousandths
Number Int Trans
Avg Response Time
Max Avg Resp Time
Max Read Per
Second
Avg Write Per
Second
Max Write Per
Second
Avg Disk Busy
Max Disk Busy
62
Performance Collection Metrics Classes
63
PERFDATA2
Performance Group 2 (*PERFDATA2) data metrics in the following order:
Short Name
Description
Unit
Time Stamp
Time stamp in seconds since epoch 00:00 1/1/1970
Integer
Percent Perm Addr
Percentage of maximum possible addresses for
permanent objects that have been used
Integer,
expressing
percentage in
thousandths
Percent Temp Addr
Percentage of maximum possible addresses for
temporary objects that have been used
Integer,
expressing
percentage in
thousandths
System ASP
Storage capacity of the system auxiliary storage pool
(ASP1)
Integer,
expressed in
Megabytes
Percentage of the system storage pool currently in use
Pct System ASP Used
Decimal,
expressed in ten
thousandths
Total auxiliary storage on the system
Integer, in
Megabytes
Current amount of storage in use for temporary
objects
Integer, in
Megabytes
Max Unprot Stor
Used
Largest amount of storage for temporary objects used
at any one time since the last IPL
Integer, in
Megabytes
Main Storage Size
Amount of main storage in the system. On a
partitioned system, the main storage size can change
while the system is active
Integer, in
Kilobytes
Num of Memory
Pools
The number of memory pools allocated
Integer
Total Aux Storage
Cur Unprot Stor
Used
64