Before start
The status of the each step from this guide should be described in incident description. Attached screenshots
for each step are welcome. Specify which step from this guide was failed.
It’s highly recommended to use internal JVM for each WebSphere monitor. SiteScope creates new java process
for each Monitors with external JVM. It takes 64-254 Mb of memory for each monitor. Also it takes some time
to create external process and connect to it. Monitor with internal JVM use SiteScope’s JVM and doesn’t
create new java process for each monitor and doesn’t take additional memory for each monitor. It was
specially implemented to reduce system loading and increase SiteScope performance.
If customer is not able to find some special counters in counters tree look at “Find missed counters” section to
troubleshoot this.
If customer has already few worked monitors and can’t create new one it could be the issue with caching of jks
files by WebSphere client. As workaround they can merge certificate from different servers into one jks and
use it for all monitors. You can find step-by-step guide for this action in “WAS 7 prevent caching workaround”
doc.
Check SiteScope WebSphere Monitor configuration for Websphere 7 with internal JVM
%WAS_ENV% – the folder with WebSphere client, this folder should be specified in “Websphere directory”
option of WebSphere monitor.
Step 1: Check that WAS “version” in monitor properties combo box specified as 7.0x
Step 2: Check that %WAS_ENV% folder contains two jars:
com.ibm.ws.admin.client_7.0.0.jar and com.ibm.ws.runtime.jar
The name of jars should be exactly as above. If these jars have different names rename them.
If these jars are missed copy them from <WAS_SERVER>\runtimes\com.ibm.ws.admin.client_7.0.0.jar
and <WAS_SERVER>\plugins\com.ibm.ws.runtime.jar
Step 3: Check that SiteScope\templates.application\websphere.props file contains next line:
was70x_classpath=com.ibm.ws.admin.client_7.0.0.jar;com.ibm.ws.runtime.jar;
Replace any other line which is started from was70x_classpath= with this one.
Step 4: Check that SiteScope\java\lib\ext folder contains com.ibm.ws.security.crypto.jar.
Copy this jar from <WAS_SERVER>\WebSphere\AppServer\plugins\ com.ibm.ws.security.crypto.jar in
other case.
Step 5: Check that %WAS_ENV% folder contains two jks files:
DummyClientTrustFile.jks and DummyClientKeyFile.jks.
Be sure that these files were copied from <WAS_SERVER>\WebSphere\AppServer\
profiles\<ServerName>\etc\ folder. Where <ServerName> is the name of monitored WAS server and
not folder with name “default”. Copy these two files from this folder if they are missed.
Step 6: (SSL only) Check that SSL certificate was imported into SiteScope java keystore:
Open SiteScope UI -> Preferences -> Certificate Management -> Import Certificates ->
Insert the host name and port of WAS server. Load certificates and import them.
The port should be WAS SOAP port. This port should be specified in WAS monitor. (8880, 8881 or
something like this, it could be checked in WAS admin console, look at “Check WebSphere
configuration” section).
If SiteScope didn’t tell that such certificates exist already you need to restart SiteScope.
Step 7: (SSL only) Check that SiteScope\java\lib\security\java.security file contains next lines in
“Default JSSE socket factories” section:
# Default JSSE socket factories
ssl.SocketFactory.provider=com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl
ssl.ServerSocketFactory.provider=com.sun.net.ssl.internal.ssl.SSLServerSocketFactoryImpl
Modify this section as above in other case.
Step 8: Restart SiteScope and create new WebSphere monitor. Check that all properties have the valid values
and try to get the counters. Look at “Check WAS side” and “Check logs” if you still can’t get them.
Testing SiteScope WebSphere Monitor configuration for Websphere 7 with external JVM
%WAS_ENV% – the folder with WebSphere client, this folder should be specified in “Websphere directory”
option of WebSphere monitor.
Step 1: Check that WAS “version” in monitor properties combo box specified as 7.0x.
Step 2: Check that %WAS_ENV% folder contains java folder from WebSphere installation folder.
Copy all files from <WAS_SERVER>\java folder to the %WAS_ENV%\java folder in other case.
Step 3: Check that %WAS_ENV% folder contains two jar:
com.ibm.ws.admin.client_7.0.0.jar and com.ibm.ws.runtime.jar
The name of jars should be exactly as above. If these jars have different names rename them.
If these jars are missed copy them from <WAS_SERVER>\runtimes\com.ibm.ws.admin.client_7.0.0.jar
and
<WAS_SERVER>\plugins\com.ibm.ws.runtime.jar
Step 4: Check that SiteScope\templates.application\websphere.props file contain next line:
was70x_classpath=com.ibm.ws.admin.client_7.0.0.jar;com.ibm.ws.runtime.jar;
Replace any other line which is started from was70x_classpath= with this one.
Step 5: Check that %WAS_ENV% folder contains two jks files:
DummyClientTrustFile.jks and DummyClientKeyFile.jks.
Be sure that these files were copied from <WAS_SERVER>\WebSphere\AppServer\
profiles\<ServerName>\etc\ folder. Where <ServerName> is the name of monitored WAS server and
not folder with name “default”. Copy these two files from this folder if they are missed.
Step 6: (SSL only) Check that SSL certificate was imported into WAS java keystore:
Get SSL certificate from WAS server – open URL to the WAS server with SOAP port in Internet Explorer
with https protocol, for example: https://<WAS_HOST>:8880/
Download the server certificate by double-clicking the key lock icon in Internet Explorer when there is
an SSL connection. The icon is located in the status bar for Internet Explorer 6, and to the right of the
URL field for Internet Explorer 7 (the field is marked red when self-signed certified is used by the
server).
In the Certificate dialog box, select the Details tab, and click Copy to File.
In the Certificate Export Wizard, export the server certificate as Base-64 encoded X.509 (.CER)
certificate to %WAS_ENV%\was_certificate.cert (in base-64 format).
Go to the %WAS_ENV%\java\jre\lib\security folder and run in cmd next command to get the certificate
MD5 fingerprints:
keytool -printcert -file “%WAS_ENV%\was_certificate.cert”
Copy the MD5 line from output. It should look like:
MD5: 0D:B8:52:36:C7:AB:F3:EC:BF:BA:CF:8F:3D:7F:A4:D8
Run the next command to see all certificates in WAS java keystore:
keytool -list -keystore cacerts
The password should be – changeit. This is default keystore password.
Copy the output somewhere and try to find certificate with MD5 equals to was_certificate.cert
If you are not able to find such certificate you need to import it into keystore. Run this command to do
this:
%WAS_ENV%\java\bin\keytool -import -v -file %WAS_ENV%\was_certificate.cert -alias was_cert
%WAS_ENV%\java\jre\lib\security\cacerts
-keystore
When prompted for the password, type changeit (default password for JRE).
When asked if you trust the imported certificate, type yes.
Step 7: (SSL only) Check that %WAS_ENV%\java\lib\security\java.security file contains next lines in
“Default JSSE socket factories” section:
# Default JSSE socket factories
ssl.SocketFactory.provider=com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl
ssl.ServerSocketFactory.provider=com.sun.net.ssl.internal.ssl.SSLServerSocketFactoryImpl
Modify this section as above in other case.
Step 8: Restart SiteScope and create new WebSphere monitor. Check that all properties have the valid values
and try to get the counters. Look at “Check WebSphere configuration” and “Check logs” if you still can’t get
them.
Check WebSphere configuration
Step 1: Check that PMI is enabled from WebSphere side:
Open WebSphere admin console in browser. The URL should look like: http://host:9043/ibm/console
or https://host:9044/ibm/console for secure connection. The port could be different. You need to get
this information from WAS administrator.
Open “Monitoring and Tuning” - > “Performance Monitoring Infrastructure (PMI)”
And choose monitored server.
Check that “Enable Performance Monitoring Infrastructure (PMI)” checkbox is enabled. Also check that
“Custom” radio button is enabled.
Enable it and apply the changes if it was disabled. You need to restart WebSphere after this change.
Find missed counters
Step 1: Check “Additional Metrics” node in counters tree. Some of the counters could be there. Also some
counters in this node could have not localized names.
Step 2: Check if these counters are enabled from WAS side:
Go to “Performance Monitoring Infrastructure (PMI)” section in WAS admin console (Look at “Check
WebSphere configuration”).
Make sure that “Custom” check box are enabled:
Open “Custom” link and try to find required counters:
Make sure that these counters are enabled. Check the same counters in “Runtime” tab, they should be
also enabled there.
Save any changes and restart WebSphere. Try to reload counters for WebSphere Monitor. All enabled
counters should be in counters tree.
Step 3: Check WebSphere cache.
SiteScope creates new xml file with full counters tree for each WebSphere monitor in
SiteScope\templates.application folder. This file have name like:
WebSphere_localhost_8880_admin___7.0x_CWebSphere7AppServer_soap.client.props_.xml
The name contains WebSphere host, port, user name, version and path to the %WAS_ENV% folder.
This xml contains full counters tree in xml representation. Try to find missed counters in this file. Attach
this file to the QC case if you still have some problems with counters.
Check SiteScope logs
Step 1: Enable debug for WebSphere monitor:
Open SiteScope\conf\core\Tools\log4j\PlainJava\log4j.properties file and add next lines to the bottom
of this file:
###################################################################################
# WebSphere monitor
###################################################################################
log4j.category.com.mercury.sitescope.monitors.j2ee.websphere=DEBUG, websphere.appender
log4j.additivity.com.mercury.sitescope.monitors.j2ee.websphere=false
log4j.category.com.mercury.sitescope.monitors.j2ee.rmiprocess=DEBUG, websphere.appender
log4j.additivity.com.mercury.sitescope.monitors.j2ee.rmiprocess=false
log4j.appender.websphere.appender=org.apache.log4j.RollingFileAppender
log4j.appender.websphere.appender.File=../${log.file.path}/websphere_monitor.log
log4j.appender.websphere.appender.MaxFileSize=${def.file.max.size}
log4j.appender.websphere.appender.MaxBackupIndex=${def.files.backup.count}
log4j.appender.websphere.appender.layout=org.apache.log4j.PatternLayout
log4j.appender.websphere.appender.layout.ConversionPattern=%d [%t] (%F:%L) %-5p - %m%n
Increase the size of logs by changing lines at the top of the file:
From
def.file.max.size=1000KB
def.files.backup.count=5
To
def.file.max.size=5000KB
def.files.backup.count=10
Restart SiteScope after this change.
Step 2: Check SiteScope\logs\RunMonitor.log file if it has any follow exceptions:
1. The DummyClientTrustFile.jks and DummyClientKeyFile.jks files are not valid. Probably they were copied
from wrong profile. You need to check step 5 from “Testing SiteScope WebSphere Monitor
configuration”.
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: javax.net.ssl.SSLHandshakeException:
com.ibm.jsse2.util.g: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a
valid CertPath.; internal cause is…
2. The com.ibm.ws.admin.client_7.0.0.jar and com.ibm.ws.runtime.jar are missed or have different names.
You need to check step 2 from “Testing SiteScope WebSphere Monitor configuration”.
Caused by: com.mercury.sitescope.monitors.j2ee.rmiprocess.ConnectionException: Inner exception is:
java.lang.ClassNotFoundException: com.ibm.websphere.management.AdminClientFactory
3. The com.ibm.ws.security.crypto.jar is missed. You need to check step 4 from “Testing SiteScope
WebSphere Monitor configuration”.
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.net.SocketException:
java.lang.ClassNotFoundException: com.ibm.websphere.ssl.protocol.SSLSocketFactory;
targetException=java.lang.IllegalArgumentException: Error opening socket: java.net.SocketException:
java.lang.ClassNotFoundException: com.ibm.websphere.ssl.protocol.SSLSocketFactory
4. The server host or port is not valid. You need to specify valid host and port.
Caused by: [SOAPException: faultCode=SOAP-ENV:Protocol; msg=; targetException=java.net.MalformedURLException]
5. Jks files are missed from %WAS_ENF% folder. Check step 5.
Caused by: com.ibm.websphere.management.exception.ConnectorNotAvailableException: [SOAPException: faultCode=SOAP-ENV:Client;
msg=Error opening socket: java.net.SocketException: Default SSL context init failed: IBMKeyManager: Problem accessing key store
java.lang.Exception: Keystore file does not exist: D:\AppClient D:\AppClient\RSP4\DummyClientKeyFile.jks;
targetException=java.lang.IllegalArgumentException: Error opening socket: java.net.SocketException: Default SSL context init failed:
IBMKeyManager: Problem accessing key store java.lang.Exception: Keystore file does not exist: D:\AppClient
D:\AppClient\RSP4\DummyClientKeyFile.jks]
6. Certificate wasn’t imported into java keystore. Check step 6.
Caused by: com.ibm.websphere.management.exception.ConnectorNotAvailableException: [SOAPException: faultCode=SOAP-ENV:Client;
msg=Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: No trusted certificate found;
targetException=java.lang.IllegalArgumentException: Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h:
No trusted certificate found]
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:344)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.<init>(SOAPConnectorClient.java:177)
... 29 more
7. SSL certificate on WebSphere side was created using wild card in CN, example: cn=*.mydomain.com.
To resolve it you need to change CN in the SSL certificate without wild card, example cn=mydomain.
This action could be done with using of ikeyman utility – on WebSphere machine
WebSphere\AppServer\bin.ikeyman.bat. This action should be done by WebSphere administrator.
Caused by: com.mercury.sitescope.monitors.j2ee.rmiprocess.ConnectionException: ADMC0053E: The system cannot create a SOAP
connector to connect to host waspapp06 at port 8881 with SOAP connector security enabled.; nested exception is:
com.ibm.websphere.management.exception.ConnectorException: ADMC0053E: The system cannot create a SOAP connector to connect
to host waspapp06 at port 8881 with SOAP connector security enabled.
at com.mercury.sitescope.monitors.j2ee.websphere.WebSphereMonitorJMX.connect(WebSphereMonitorJMX.java:171)
at com.mercury.sitescope.monitors.j2ee.websphere.WebSphereMonitorJMX.<init>(WebSphereMonitorJMX.java:102)
... 10 more
8. This issue was reproduced only on Windows 2008 R2 64bit with external JVM. We currently investigate
it. As solution customer should use internal JVM.
WebSphereServiceImpl.main() encountered an exception: java.rmi.server.ExportException: Port already in use: 0; nested exception is:
java.net.BindException: Address already in use: NET_Bind
java.rmi.server.ExportException: Port already in use: 0; nested exception is:
java.net.BindException: Address already in use: NET_Bind
9. The description about any other exception which contains exception code like ADMC0053E could be found
by following link (choose your code by first letters in it from left panel)http://publib.boulder.ibm.com/infocenter/wxdinfo/v6r0/index.jsp?topic=/com.ibm.websphere.messag
es.doc/com.ibm.ws.management.resources.connector.html