HPE Serviceguard Toolkit for Oracle
on Linux User Guide
Abstract
This guide describes instructions for configuring an Oracle single instance in a Serviceguard cluster on Linux and to integrate
an Oracle database server with HPE Serviceguard for Linux.
Part Number: 766173-004
Published: June 2016
Edition: 4
© Copyright 2015 Hewlett Packard Enterprise Development LP
The information contained herein is subject to change without notice. The only warranties for Hewlett Packard Enterprise products and services
are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting
an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.
Confidential computer software. Valid license from Hewlett Packard Enterprise required for possession, use, or copying. Consistent with FAR
12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed
to the U.S. Government under vendor's standard commercial license.
Links to third-party websites take you outside the Hewlett Packard Enterprise website. Hewlett Packard Enterprise has no control over and is not
responsible for information outside the Hewlett Packard Enterprise website.
Acknowledgments
Oracle ® is a registered trademark of Oracle Corporation.
Linux is a registered trademark of Linus Torvalds.
Contents
1 Overview..............................................................................................................5
Advantages...........................................................................................................................................5
Prerequisites.........................................................................................................................................5
Hardware support.................................................................................................................................5
2 Installing and uninstalling HPE Serviceguard toolkit for Oracle..........................6
Installing Oracle toolkit..........................................................................................................................6
Upgrading Oracle toolkit.......................................................................................................................6
Uninstalling Oracle toolkit.....................................................................................................................7
Files and directories created for HPE Serviceguard toolkit for Oracle after installation.......................7
Rolling Upgrade..................................................................................................................................10
3 Supported configuration....................................................................................11
Without Automatic Storage Management...........................................................................................11
Shared configuration.....................................................................................................................11
Multiple instance configuration......................................................................................................11
With Automatic Storage Management................................................................................................12
4 Using HPE Serviceguard toolkit for Oracle.......................................................13
Configuring modular Oracle ASM instance and Database instance toolkit packages........................13
Configuring modular Oracle database package without ASM.......................................................13
Configuring modular ASM instance package................................................................................19
Managing a package..........................................................................................................................30
Adding the package to the cluster.................................................................................................30
Running a package........................................................................................................................30
Maintaining a package...................................................................................................................31
Online Modification of the Oracle Toolkit package........................................................................32
Handling failures during online package reconfiguration.........................................................34
Halting a package..........................................................................................................................35
Deleting a package........................................................................................................................35
5 Oracle toolkit features........................................................................................37
Alert mail notification...........................................................................................................................37
Cluster verification feature for Oracle toolkit.......................................................................................37
Support for sourcing user defined environment variables for Oracle instance..............................39
6 Supporting Oracle features................................................................................40
Support for Oracle 12c Multitenant architecture.................................................................................40
Support for multiple listeners..............................................................................................................41
Single service to monitor all the listeners together........................................................................41
A separate service to monitor each listener..................................................................................42
Support for user defined database services.......................................................................................43
Support for database hang detection..................................................................................................43
Support for Oracle database backup mode........................................................................................44
7 Troubleshooting.................................................................................................45
8 Support and other resources.............................................................................46
Accessing Hewlett Packard Enterprise Support.................................................................................46
Accessing updates..............................................................................................................................46
Websites.............................................................................................................................................46
Customer self repair...........................................................................................................................47
Remote support..................................................................................................................................47
Documentation feedback....................................................................................................................47
Contents
3
A Sample package configuration file for Oracle database modular package without
ASM......................................................................................................................48
B Sample package configuration file for Oracle ASM instance package and
database modular package..................................................................................49
C Sample package configuration file for Oracle multitenant database with
pluggable databases configured for monitoring-..................................................51
Glossary...............................................................................................................52
Index.....................................................................................................................53
4
Contents
1 Overview
HPE Serviceguard toolkit for Oracle on Linux enables you to configure an Oracle single instance
in a Serviceguard cluster on Linux and integrate an Oracle database server with Serviceguard
for Linux (SGLX). This toolkit consists of a set of scripts that are used to start, stop, and monitor
an Oracle database instance, Automatic Storage Management (ASM) instance, and the configured
listeners.
Advantages
Following are the advantages of Serviceguard toolkit for Oracle:
•
Allows easy integration of Oracle application in a Serviceguard environment.
•
Provides high availability to Oracle database and ASM instances.
•
Provides high availability to Oracle 12c Container Database (CDB).
•
Serviceguard has monitoring capabilities to check the system resources such as network,
volume groups and files systems. It also initiates a failover, in the instance of the failure of
these system resources.
•
You can configure the packages using Serviceguard Manager. For more information, see
Managing HPE Serviceguard for Linux at http://www.hpe.com/info/
linux-serviceguard-docs.
Prerequisites
Following are the prerequisites for Serviceguard toolkit for Oracle:
•
Serviceguard for Linux must be installed on all the nodes in the cluster.
•
Oracle application must be installed on all the package nodes.
•
Shared storage must be connected to all the package nodes.
Hardware support
Unless otherwise stated, this toolkit runs on all hardware supported by SGLX.
Advantages
5
2 Installing and uninstalling HPE Serviceguard toolkit for
Oracle
This chapter covers the details related to installing, uninstalling, and upgrading the Oracle toolkit.
Installing Oracle toolkit
Oracle toolkit RPM must be installed on all the nodes of the cluster that is configured to run the
Oracle toolkit package. For a fresh installation of the Oracle toolkit, run the following command:
rpm -ivh serviceguard-oracle-toolkit-<version
number>.<OS_destro>.noarch.rpm
For example:
rpm -ivh serviceguard-oracle-toolkit-A.12.00.x-00.sles.noarch.rpm
Upgrading Oracle toolkit
IMPORTANT: The cmupgrade command is a new tool introduced in Serviceguard for Linux
A.12.00.00 which helps you upgrade Serviceguard and its components such as Serviceguard
Manager, Toolkits, Extended Distance Cluster, etc. This command can be used to perform online
or offline upgrade of Serviceguard across major versions starting from A.11.20.00 and later. It
is recommended to use this command to upgrade the Oracle toolkit. For more information on
this command see HPE Serviceguard for Linux Release Notes at http://www.hpe.com/info/
linux-serviceguard-docs.
To follow the usual method to upgrade an already installed version of the Oracle toolkit, run the
following command:
rpm -Uvh serviceguard-oracle-toolkit-<version
number>.<OS_destro>.noarch.rpm
For example:
rpm -Uvh serviceguard-oracle-toolkit-A.12.00.x-00. redhat.noarch.rpm
The files are installed in the following directories:
•
$SGROOT/oracletoolkit
•
$SGCONF/scripts/tkit/oracle
•
$SGCONF/modules/tkit/oracle
The packages that are created before the installation of RPM will continue to run and you can
halt these packages, and failover them without any issues. However, they cannot be modified
and reapplied.
To use the new features bundled with this RPM, delete the old packages and recreate them
using the new scripts.
NOTE: If you have upgraded from any previous version to A.12.00.00, then ensure you include
the new module version in to the existing packages.
6
Installing and uninstalling HPE Serviceguard toolkit for Oracle
Uninstalling Oracle toolkit
To uninstall the Oracle toolkit:
•
To search the version of Oracle toolkit, run the following command:
# rpm –qa | grep oracle-toolkit
•
To remove the earlier version of Oracle toolkit, run the following command:
# rpm –e <oracle toolkit name retrieved after executing the # rpm
–qa | grep oracle-toolkit>
Files and directories created for HPE Serviceguard toolkit for Oracle after
installation
The following table describes the scripts and files that are installed.
Uninstalling Oracle toolkit
7
Table 1 Scripts for Serviceguard toolkit for Oracle
File name
Description
haoracle.sh
This script contains a list of internally $SGROOT/oracletoolkit
used variables and functions that
start, stop, and monitor Oracle CDB,
non CDB instance, and ASM
instances.
The toolkit.sh script invokes the
haoracle.sh script to perform the
following:
• On package startup, it starts the
database or ASM instance. It also
starts the monitor processes.
• On package halt, it stops the
database or ASM instance, and
monitor processes. In case of
database instance, it also stops
listener process(es).
haoracle_sql.sh
This script contains functions for
starting, stopping, and monitoring
Oracle CDB, non CDB instances, and
configured listeners. This script also
supports starting Pluggable
Databases (PDBs) in the context of
CDB in READ WRITE mode, when
CDB is started in open mode. The
haoracle.sh script invokes
haoracle_sql.sh script.
haoracle_utils.sh
This script is a utility script which
contains the common functions used
by other oracle toolkit scripts.
haoracle.mon
This script contains a list of internally
used variables and functions for
monitoring an Oracle server instance.
The haoracle.sh script invokes this
script.
By default, the following processes
are monitored:
• ora_pmon_$ORACLE_SID_NAME
• ora_smon_$ORACLE_SID_NAME
• ora_lgwr_$ORACLE_SID_NAME
• ora_dbwr_$ORACLE_SID_NAME
• ora_ckpt_$ORACLE_SID_NAME
• ora_reco_$ORACLE_SID_NAME
NOTE: $ORACLE_SID_NAME is
the session id name of the Oracle
instance.
These process names are contained
in the variable
MONITOR_PROCESSES. To include
other Oracle processes that need to
be monitored, update the
MONITOR_PROCESSES package
parameter.
8
Installing and uninstalling HPE Serviceguard toolkit for Oracle
Available in directory
Table 1 Scripts for Serviceguard toolkit for Oracle (continued)
File name
Description
Available in directory
NOTE: If you have upgraded from
any previous version to A.12.00.00,
then ensure you include the new
module version to see the above
mentioned changes.
halistener.mon
This is the listener monitor script. The
haoracle.sh script invokes the
halistener.mon script to monitor the
configured listeners.
hadbhang.mon
hadbhang.mon script monitors the
Oracle CDB and non CDB instances
for possible “hung” state. Also, the
script monitors the status of PDB in
the context of CDB and sends email
alerts to the email id’s configured in
the package if the status of PDB is in
UNUSABLE state.
haoraclepdb.mon
haoraclepdb.mon script monitors
the accessibility to the Oracle 12c
pluggable database. If the PDB is not
accessible, a failover of the
package(CDB) is initiated.
hagetdbstatus.sh
This script is used to check the status
of CDB and non CDB instances. This
script is a utility script which contains
the common functions used by other
Oracle toolkit scripts.
hatimeoutdbhang.sh
This script checks whether a process
with the specified PID exists. If the
process exists, then it kills the
process and exits.
toolkit.sh
This script is the interface between
the toolkit module script and the
package startup script.
oracle.1
This is an attribute definition file, used $SGCONF/modules/tkit/oracle/
to generate a package ASCII
oracle.1
template in modular style of
packaging.
NOTE: This module file is
applicable only for the packages
created by using the previous Oracle
toolkit versions.
oracle.2
This is an attribute definition file, used $SGCONF/modules/tkit/oracle/
to generate a package ASCII
oracle.2
template in modular style of
packaging.
oracle.3
This is an attribute definition file which $SGCONF/modules/tkit/oracle/
is used to generate a package ASCII oracle.3
template in modular style of
packaging.
Files and directories created for HPE Serviceguard toolkit for Oracle after installation
9
Table 1 Scripts for Serviceguard toolkit for Oracle (continued)
File name
Description
Available in directory
oracle.4
This is an attribute definition file,
which is used to generate package
ASCII template from oracle toolkit
version 12.00.50.
$SGCONF/scripts/tkit/oracle
tkit_module.sh
The Serviceguard master control
script invokes this script. This script
acts as an interface between the
Serviceguard master control script
and the toolkit interface script,
toolkit.sh.
$SGCONF/scripts/tkit/oracle
Rolling Upgrade
The toolkit package nodes in a cluster can be upgraded with the new toolkit, on one node at a
time, by following the steps below:
1. Backup the package directory and any package control scripts.
2. Move the running Oracle toolkit packages from the node that is being upgraded to another
package node.
3. Upgrade the Oracle toolkit using the following command:
rpm -Uvh
serviceguard-oracle-toolkit<version_number>.<OS_distro>.noarch.rpm
4.
Verify the installation using the following command:
# rpm -q serviceguard-oracle-toolkit
The system displays the following output:
serviceguard-oracle-toolkit-<version_number>
At this point, packages can be migrated back to the current node.
5.
10
Repeat all the above steps on all the other toolkit package nodes in the cluster.
Installing and uninstalling HPE Serviceguard toolkit for Oracle
3 Supported configuration
This section explains the following supported configurations for Oracle CDB and non CDB toolkit
packages:
•
Without Automatic Storage Management
•
With Automatic Storage Management
Without Automatic Storage Management
This section explains how to configure an Oracle CDB and non CDB instance package without
ASM in a Serviceguard environment. You can configure Oracle CDB and non CDB instance
package using one of the following ways:
•
Shared configuration
•
Multiple instance configuration
Shared configuration
In this configuration method, the configuration and database files are kept on shared disks but
you can place the Oracle binaries on the local storage. Since the configuration files and data
files reside on a shared location, you need not ensure that all the nodes have the same
configuration. Alternatively, you can place the Oracle binaries too on the shared location along
with the configuration and database files.
A password file that is, orapw${SID_NAME} must be created for each Oracle instance in the
configuration directory. The password file is created via the orapwd utility.
To use Serviceguard toolkit for Oracle with Oracle 10g, 11g, and 12c, ensure that all the Oracle
binaries reside locally on each cluster node and do not reside on a shared volume group. You
can use one of the following steps:
•
Install Oracle 10g, 11g, and 12c software on all the cluster nodes. Ensure that the database
related files (that is, all control files, data files, and log files) reside on a shared volume group.
The Oracle binaries however reside locally on each node. For 12c CDB databases, ensure
that all the database related files of each PDB in the context of CDB resides on a shared
volume group.
•
Install Oracle 10g, 11g, and 12c software on all the cluster nodes. Place the Oracle binaries
as well as the database related files on a shared volume group.
NOTE:
•
The Oracle binaries can reside on a shared disk for Oracle 11g. For Oracle 11g, if the binaries
are located on shared disk, then do not configure Oracle ASM on that node.
•
If multiple Oracle database instances are configured, you cannot use shared storage to keep
the Oracle binaries. If you are configuring multiple instances of Oracle db, Hewlett Packard
Enterprise recommends that you keep the Oracle binaries only on a local storage.
•
Hewlett Packard Enterprise recommends that you use local storage for Oracle binary
installation. During a Oracle patch update, the toolkit allows users to run the oracle instance
on any other package nodes during a failure.
Multiple instance configuration
Oracle supports multiple DB instances running on a node, simultaneously. This allows you to
move one instance from its own node while leaving another instance up. Each node runs several
instances. Each Oracle package corresponds to a separate Oracle database server instance
with its own SID_NAME, unique volume group, logical volume, and file system for storing control
Without Automatic Storage Management
11
files, data files, and redo-log files. The configuration details for each instance must be identical
for each node configured to run that instance.
With Automatic Storage Management
ASM instance can be managed with the Serviceguard toolkit for Oracle. This toolkit makes use
of the multi-node package and package dependency feature to integrate ASM with Serviceguard.
MNP (Multi-node Package) is used to encapsulate the per node ASM instances, with one or
more Oracle database instance failover packages dependent on this MNP. This configuration
enables the database instance to start up in the right order in relation to the ASM instance, and
in the event of a failover, to relocate only to a node where an ASM instance is available.
To configure Oracle database using ASM in a Serviceguard environment:
1. Configuring ASM in a Serviceguard environment and creating the ASM instance.
a. Configure logical volumes or devices to be used in the ASM disk groups.
b. Install the Oracle standalone server GRID (needed in 11gR2) and ASM binaries in local
storage on all the nodes. You can skip the step c, in case ASM instance is created
through OUI during install.
c. Create the ASM instance on one node by creating a ASM disk group (DG) with raw
logical volumes or raw devices. Ensure that logical volume or raw device used here are
accessible from all the cluster nodes.
d. Create the ASM instance on the other cluster nodes by specifying different ASM DG
name and using different raw logical volumes or raw devices.
e. Copy the parameter file or server parameter file of the ASM instance to all the nodes
with appropriate file permissions.
2.
12
Creating the Oracle database in a Serviceguard environment.
a. Install the Oracle binaries on local storage in ORACLE_HOME on all the nodes configured
in the cluster.
b. Create the database instance on one node by choosing the storage option as Automatic
storage management in "dbca" GUI. Mention the ASM DG name which has been created
on that node as the Storage name.
c. Optional: Configure a listener.
d. Copy the parameter file or server parameter file of the database instance to all the nodes
with appropriate file permissions.
e. Distribute the listener configuration file if listener is configured, listener.ora from
${ORACLE_HOME}/network/admin/ to all the package nodes.
f. Copy the directories from ${ORACLE_HOME}/admin to all the nodes.
Supported configuration
4 Using HPE Serviceguard toolkit for Oracle
After installing the Serviceguard toolkit for Oracle, you can use this toolkit to:
•
Deploy Oracle ASM and Oracle CDB and non CDB instances in Serviceguard cluster
environment.
•
Manage the packages.
Configuring modular Oracle ASM instance and Database instance toolkit
packages
This section describes the steps to configure Oracle ASM instance and Database instance toolkit
packages.
•
“Configuring modular Oracle database package without ASM” (page 13)
•
“Configuring modular ASM instance package” (page 19)
Configuring modular Oracle database package without ASM
To configure a modular Oracle database toolkit packages without ASM:
1. Create a directory in $SGCONF/<package dir>. For example:
#mkdir $SGCONF/oracle_pkg
2.
3.
cd $SGCONF/<oracle_pkg>.
Create the modular package configuration file, pkg2.conf using the command:
# cmmakepkg -m tkit/oracle/oracle pkg2.conf
4.
Edit the modular package configuration file pkg2.conf. Table 2 (page 13) describes the
list of attributes that you can edit.
Table 2 Lists the attributes that you can edit
Attribute
Description
Example
package_name
The package name must be unique oracle_pkg
in the cluster.
INSTANCE_TYPE
This parameter determines whether database
the instance is an ASM instance or
a database instance. Set this
parameter to database. The
default value is database.
ORACLE_HOME
This is the Oracle home directory /ORACLE_TEST1
where Oracle binaries are installed.
ORACLE_ADMIN
This is the user name of the Oracle oracle
database administrator who
manages the database. The default
value is oracle.
SID_NAME
This is the Oracle session name,
ORACLE_TEST0
also called the session ID (SID).
For Oracle 12c CDB databases,
you must mention CDB session ID.
Configuring modular Oracle ASM instance and Database instance toolkit packages
13
Table 2 Lists the attributes that you can edit (continued)
Attribute
Description
Example
NOTE: If a change is made to
this attribute after the #
cmmakepkg -i -m -t command
is run, you must reconfigure the
MONITOR_PROCESSES attribute
defined in this file to reflect the
same. Also, reconfigure PFILE to
reflect this change if PFILE is uses
this attribute.
START_MODE
Oracle CDB and non CDB startup open
mode. You can set this parameter
to mount, nomount, or open. The
default value is open.
NOTE: For Oracle 12c CDB
databases, if the start mode for
CDB is set to open, CDB and PDB
databases will be opened only in
READ WRITE mode. Seed PDB
will still remain in READ-ONLY
state.
LISTENER
If this attribute is set to no, you
must comment the
LISTENER_NAME and
oracle_listener_service
attributes. For example:
#LISTENER_NAME
yes
The default value is yes.
LISTENER_NAME
This parameter holds the names of LISTENER
the Oracle listener processes. You
can add more than one listener
name by adding to the array
variable LISTENER_NAME. The
default value is LISTENER.
LISTENER_PASS
This holds the passwords of the
ora123
Oracle listener processes. You can
add more than one listener
password by adding to the array
variable LISTENER_PASS.
LISTERNER_RESTART
This parameter defines the number 2
of attempts to be made to restart
the listeners when a listener failure
is detected.
If the listener does not restart after
LISTENER_RESTART number of
consecutive attempts, the package
fails over. If the listener restart is
successful, then the next time
listener fails, the toolkit again tries
LISTENER_RESTART number of
times to restart the listener.
You can add more than one listener
restart value by adding to the array
variable LISTENER_RESTART.
14
Using HPE Serviceguard toolkit for Oracle
Table 2 Lists the attributes that you can edit (continued)
Attribute
Description
Example
NOTE: If the package is in
maintenance mode, then the
listener will not be restarted even if
LISTENER_RESTART is set to a
definite value.
The Oracle CDB and non CDB
instance parameter file. If this
attribute is not specified, Oracle
picks this up from the Oracle
configuration directory
$ORACLE_ORACLE_HOME.
PFILE
${ORACLE_ORACLE_HOME}/dbs/
init${ORACLE_SID_NAME}.ora
When this parameter is configured,
it overrides the default file location.
If both PFILE and SPFILE are
present and this parameter is not
specified, then the Oracle toolkit
makes use of PFILE as the
parameter file.
To make use of SPFILE in the
toolkit do not configure this
parameter, and remove the default
PFILE from its default location. The
default value is
${ORACLE_ORACLE_HOME}/dbs/
init${ORACLE_SID_NAME}.ora.
This is the list of all the critical
Oracle processes for an instance
that must be executing to assume
that the instance is up and running.
You can add more instance
processes. The processes
mentioned will be monitored by
default.
• ora_pmon_${ORACLE_SID_NAME}
MONITOR_INTERVAL
The time interval, in seconds, this
script waits between checks to
ensure that Oracle instance is
running. The default value is 30
seconds.
30
TIME_OUT
The time period, in seconds, this
toolkit waits for a completion of a
normal shutdown before initiating
forceful halt of the application.
30
MONITOR_PROCESSES
• ora_dbw0_${ORACLE_SID_NAME}
• ora_ckpt_${ORACLE_SID_NAME}
• ora_smon_${ORACLE_SID_NAME}
• ora_lgwr_${ORACLE_SID_NAME}
• ora_reco_${ORACLE_SID_NAME}
The TIME_OUT variable is used to
protect against a worst case
scenario where a hung database or
ASM instance prevents the halt
script from completing, therefore
preventing the standby node from
starting the instance.
The TIME_OUT variable has no
effect on package failover times.
The default value is 30 seconds.
PARENT_ENVIRONMENT
This parameter is used to mention no
if the Oracle or ASM user's shell
must be invoked as a new shell or
Configuring modular Oracle ASM instance and Database instance toolkit packages
15
Table 2 Lists the attributes that you can edit (continued)
Attribute
Description
Example
as a subshell that inherits the
variables set in the parent shell.
You can set this parameter to yes
or no. The default value is no. Set
this parameter to yes if the Oracle
or ASM user's shell must be
invoked as a subshell.
Set to no if the Oracle or ASM
user's shell must be invoked as a
new shell. If set to no, the Oracle
or ASM user's .profile file is
executed and the variables set in
this file are available to the new
shell.
CLEANUP_BEFORE_STARTUP
This parameter indicates whether
shutdown abort must be executed
before the startup of the Oracle or
ASM instance.
no
You can set this parameter either
to yes or no. The default value is
no.
USER_SHUTDOWN_MODE
This parameter is used to specify abort
the instance shutdown mode only
when a shutdown is initiated by the
user and not because of the failure
of a service.
You can set this parameter to
abort or immediate.
If abort is specified, the instance
is shutdown using the abort
option. If immediate is specified,
the instance is shutdown using the
immediate option. The default
value is abort.
KILL_ASM_FOREGROUNDS
If ASM is being used, this
no
parameter specifies whether any
ASM foreground processes having
file descriptors open on the
dismounted disk group volumes
need to be killed or not.
You can set this parameter either
to yes or no. The default value is
yes.
16
DB_SERVICE
This parameter is used to start and all
stop the DB service through the DB
package. It is commented by
default. You can set the attribute to
all or specify the DB service
names that must be started and
stopped by the DB package.
DISABLE_BACKUP_MODE
This parameter is used to disable no
the Oracle database from the
backup mode. The valid values for
this parameter are yes and no. By
default, the value is set to no. This
Using HPE Serviceguard toolkit for Oracle
Table 2 Lists the attributes that you can edit (continued)
Attribute
Description
Example
means, if the database is in the
backup mode after package failover
or restart, the Oracle toolkit
package fails to start. You have to
manually get the database out of
the backup mode and then start the
package.
When the value of
DISABLE_BACKUP_MODE is set to
yes and the database is in the
backup mode after failover or
restart, the Oracle toolkit package
will alter the database to remove
the backup mode and start the
database. The database is altered
to remove the backup mode only if
START_MODE is set to open.
RETRY_INTERVAL_ORA_ERROR This parameter defines the time
interval to wait in case of ORA
errors during database hang
monitor.
email_id
0
This parameter is used to provide abc@domain-name.com
email addresses for the toolkit alert
notification recipients. Each
email_id parameter can have one
of the following values:
• A complete email address
• An alias
• A distribution list
You can list multiple recipients by
repeating the email_id field and
address. Toolkit sends an alert
email on listener and package
failure. If configured, the alert mail
is sent for Oracle database hang
detection. For example: $SGCONF/
scripts/tkit/oracle/
tkit_module.sh
oracle_hang_monitor 30
alert. By default, email_id is
commented.
CRITICAL_PDB
This parameter is used to provide
the pluggable database names,
whose failure triggers the failover
of the package and thereby causes
the restart of the CDB on the
adoptive node. This attribute can
have one of the following values:
• PDB names
CRITICAL_PDB any
or
CRITICAL_PDB none
CRITICAL_PDB pdb1
CRITICAL_PDB pdb2
CRITICAL_PDB pdb3
• value “any”
• value“ none”
To monitor multiple PDB names,
repeatCRITICAL_PDB field and
PDB name. The package failover
Configuring modular Oracle ASM instance and Database instance toolkit packages
17
Table 2 Lists the attributes that you can edit (continued)
Attribute
Description
Example
is decided based on the value
specified for this attribute.
• If the value specified is “any”,
the package fails over if any of
the plugged PDB fails.
• If the value is "none", the
package will not failover in case
of PDB failures. The package
fails over only if the CDB fails.
5.
Edit the following service parameters, if required:
•
•
•
18
Monitor service for Oracle processes:
Attribute Name
Example
service_name
oracle_service
service_cmd
"$SGCONF/scripts/tkit/oracle/
tkit_module.sh oracle_monitor"
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
Monitor service for Oracle listener processes:
Attribute Name
Example
service_name
oracle_listener_service
service_cmd
"$SGCONF/scripts/tkit/oracle/
tkit_module.sh oracle_monitor_listener"
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
Monitor service for Oracle hang detection:
Attribute Name
Example
service_name
oracle_hang_service
service_cmd
"$SGCONF/scripts/tkit/oracle/
tkit_module.sh oracle_hang_monitor 30
failure"
service_restart
none
service_fail_fast_enabled
no
Using HPE Serviceguard toolkit for Oracle
•
6.
Attribute Name
Example
service_halt_timeout
300
service_halt_on_maintenance
no
Monitor service for oracle pluggable database health monitoring:
Attribute Name
Example
service_name
oracle_pdb_monitor_service
service_cmd
"$SGCONF/scripts/tkit/oracle/
tkit_module.sh oracle_pdb_monitor"
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
Apply the package configuration using the command:
# cmapplyconf -P pkg2.conf
7.
Start the "oracle_pkg" package using the command:
#cmrunpkg oracle_pkg
Configuring modular ASM instance package
NOTE: The following conditions are applicable for the Oracle Toolkit to support the MetroCluster
and Continentalcluster:
•
The Oracle ASM MNP package must not be configured as a Metrocluster or
Continentalclusters package.
•
The appropriate Metrocluster or Continentalclusters modules must be included when creating
database package.
To configure a modular ASM instance Multi-node package (MNP) and Database instance failover
package:
1. Setting up and configuring ASM multi-node package.
a. Create a directory in $SGCONF/<package dir>.
b. cd $SGCONF/<package dir>.
c. Create the modular package configuration file, pkg1.conf using the command:
# cmmakepkg -m sg/multi_node -m tkit/oracle/oracle pkg1.conf
d.
Edit the following attributes manually in the package configuration file:
Attribute
Description
Example
package_name
The package name must be
unique in the cluster.
oracle_asm_pkg
INSTANCE_TYPE
This parameter determines
ASM
whether the instance is an ASM
instance or a database instance.
Configuring modular Oracle ASM instance and Database instance toolkit packages
19
Attribute
Description
Example
Set this parameter to ASM. The
default value is ASM.
ORACLE_ADMIN
This is the user name of the
oracle
Oracle database administrator
who manages the database. The
default value is oracle.
ORACLE_HOME
This is the base directory where
Oracle is installed.
/ORACLE_TEST0
SID_NAME
The Oracle session name. It is
also called the session ID (SID).
It uniquely identifies an Oracle
database instance. For Oracle
12c CDB databases, you must
mention CDB session ID.
ORACLE_TEST0
NOTE: If you modify this
attribute after the # cmmakepkg
-i -m -t command is run, you
must reconfigure the
MONITOR_PROCESSES
attribute defined in this file to
reflect the same. Also,
reconfigure PFILE attribute
defined in this file to reflect this
change, if PFILE is uses this
attribute.
ASM
This parameter determines
whether ASM is being used to
manage the storage for the
database instance.
no
The default value is no. You can
either set the value as yes or no.
Set this parameter to yes if ASM
is used to store DB files, else, set
it to no.
ASM_HOME
The home directory where ASM
is installed.
ASM_USER
The user name of the Oracle
oracle
ASM administrator. It is used to
start or stop the ASM instance
and mounting or dismounting the
ASM disk groups. The default
value is oracle.
ASM_SID
The ASM session name that
uniquely identifies an ASM
instance. The default value is
+ASM.
PFILE
It is the Oracle database
${ORACLE_ASM_HOME}/dbs/
parameter file. If this attribute is init${ORACLE_ASM_SID}.ora
not specified, Oracle picks this up
from the Oracle configuration
directory $ASM_HOME/dbs.
When this parameter is
configured, it overrides the default
file location. If both PFILE and
SPFILE are present and this
parameter left unspecified, then
20
Using HPE Serviceguard toolkit for Oracle
/asm_home
+ASM
Attribute
Description
Example
Serviceguard toolkit for Oracle
makes use of PFILE as the
parameter file.
To make use of SPFILE in the
toolkit do not configure this
parameter and remove the default
PFILE from its default location.
The default value is
${ORACLE_ORACLE_HOME}/dbs/
init${ORACLE_SID_NAME}.ora.
MONITOR_PROCESSES
This is the list of all the critical
Oracle ASM processes for an
instance that must be executing
to assume that the instance is up
and running. You can add more
instance processes. The
processes mentioned will be
monitored by default.
• asm_pmon_${ORACLE_ASM_SID}
• asm_dbw0_${ORACLE_ASM_SID}
• asm_ckpt_${ORACLE_ASM_SID}
• asm_smon_${ORACLE_ASM_SID}
• asm_lgwr_${ORACLE_ASM_SID}
• asm_reco_${ORACLE_ASM_SID}
MONITOR_INTERVAL
The time interval, in seconds, this 30
script waits between checks to
ensure that ASM instance is
running. The default value is 30
seconds.
TIME_OUT
The time period, in seconds, this 30
toolkit waits for a completion of a
normal shutdown before initiating
forceful halt of the application.
The TIME_OUT variable is used
to protect against a worst case
scenario where a hung database
or ASM instance prevents the halt
script from completing, therefore
preventing the standby node from
starting the instance.
The TIME_OUT variable has no
effect on package failover times.
The default value is 30 seconds.
PARENT_ENVIRONMENT
This parameter is used to mention no
if the Oracle or ASM user's shell
must be invoked as a new shell
or as a subshell that inherits the
variables set in the parent shell.
You can set this parameter to yes
or no. The default value is no. Set
this parameter to yes if the
Oracle or ASM user's shell must
be invoked as a subshell.
Set to no if the Oracle or ASM
user's shell must be invoked as a
new shell. If set to no, the Oracle
or ASM user's .profile file is
executed and the variables set in
this file are available to the new
shell.
CLEANUP_BEFORE_STARTUP
This parameter indicates whether no
shutdown abort must be executed
Configuring modular Oracle ASM instance and Database instance toolkit packages
21
Attribute
Description
Example
before the startup of the Oracle
or ASM instance.
You can set this parameter to yes
or no. The default value is no.
USER_SHUTDOWN_MODE
This parameter is used to specify abort
the instance shutdown mode only
when a shutdown is initiated by
the user and not because of the
failure of a service.
You can set this parameter to
abort or immediate.
If abort is specified, the instance
is shutdown using the abort
option. If immediate is specified,
the instance is shutdown using
the immediate option. The
default value is abort.
KILL_ASM_FOREGROUNDS
If ASM is being used, this
yes
parameter specifies whether any
ASM foreground processes
having file descriptors open on
the dismounted disk group
volumes must be killed or not.
You can set this parameter to yes
or no. The default value is yes.
Listener
You must set this parameter to
no.
email_id
This parameter is used to provide abc@domain-name.com
email addresses for the toolkit
alert notification recipients. Each
email_id parameter can have
one of the following values:
no
• A complete email address
• An alias
• A distribution list
You can list multiple recipients by
repeating the email_id field and
address. Toolkit sends an alert
email on listener and package
failure. If configured, the alert mail
is sent for Oracle database hang
detection. For example:
$SGCONF/scripts/tkit/
oracle/tkit_module.sh
oracle_hang_monitor 30
alert. By default, email_id is
commented.
e.
22
Monitor service for Oracle ASM instance processes:
Attribute Name
Example
service_name
oracle_asm_service
service_cmd
"$SGCONF/scripts/tkit/oracle/
tkit_module.sh oracle_monitor"
Using HPE Serviceguard toolkit for Oracle
Attribute Name
Example
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
NOTE: Remove the Oracle listener, Oracle db hang monitor services, and Oracle
PDB monitoring service for the ASM instance package, shown in the tables below.
f.
Apply the package configuration using the command:
# cmapplyconf -P pkg1.conf
2.
Setting up and configuring database failover package which uses ASM:
a. Create a directory in $SGCONF/<package dir>.
b. cd $SGCONF/<package dir>.
c. Create the modular package configuration file, pkg2.conf using the following command:
#cmmakepkg -m tkit/oracle/oracle pkg2.conf
d.
Edit the following attributes manually in the package configuration file:
Attribute
Description
Example
package_name
The package name must be
unique in the cluster.
oracle_asmdb_pkg
INSTANCE_TYPE
This parameter determines
database
whether the instance is an ASM
instance or a database instance.
Set this parameter to database.
ORACLE_HOME
This is the base directory where
Oracle is installed.
ORACLE_ADMIN
This is the user name of the
oracle
Oracle database administrator
who manages the database. The
default value is oracle.
SID_NAME
The Oracle session name. It is
also called the session ID (SID).
It uniquely identifies an Oracle
database instance.
/ORACLE_TEST0
ORACLE_TEST0
NOTE: If you modify this
attribute after the # cmmakepkg
-i -m -t command is run, you
must reconfigure the
MONITOR_PROCESSES attribute
defined in this file to reflect the
same. Also, reconfigure PFILE
attribute defined in this file to
reflect this change, if PFILE is
using this attribute.
START_MODE
This parameter defines the Oracle open
database startup mode. Valid
options are no mount, mount,
and open. The default mode is
open.
Configuring modular Oracle ASM instance and Database instance toolkit packages
23
Attribute
Description
Example
NOTE: For Oracle 12c CDB
databases, if the start mode for
CDB is set to open, CDB and
PDB databases will be opened
only in READ WRITE mode. Seed
PDB will remain in READ-ONLY
state.
ASM
This parameter determines
whether ASM is being used to
manage the storage for the
database instance.
yes
You can set the value to yes or
no. Set this parameter to yes if
ASM is used, else set to no. The
default value is no.
ASM_DISKGROUP
If ASM is being used to manage diskgroup1
the storage for the database
instance, configure this parameter
to indicate the ASM diskgroups
used by the database instance.
You can add more than one disk
group by adding to the array
variable ASM_DISKGROUP.
NOTE: For Oracle 12c, if
different diskgroups are used for
storing data files of different
PDBs, then add all the diskgroups
used by PDBs in the package
configuration file created for
packaging CDB database.
ASM_VOLUME_GROUP
Defines the volume groups used vgora_asm1
in the ASM disk groups for the
database instance. You can add
more than one volume group by
adding to this array variable.
NOTE: For Oracle 12c, if
different volume groups are used
for storing data files of different
PDBs, then add all the volume
groups used by PDBs in the
package configuration file created
for packaging CDB database.
ASM_PV
This parameter represents the
/dev/raw/raw1
physical volume which is used to
create ASM disk groups using
raw devices for the database
instance. You can add more than
one physical volume to this array
variable. This parameter can be
used only in modular packages.
NOTE: For Oracle 12c, if
different raw devices are used for
creating diskgroup for different
PDBs, then add all the raw
devices used by PDBs in the
package configuration file created
for packaging CDB database.
24
Using HPE Serviceguard toolkit for Oracle
Attribute
Description
Example
ASM_HOME
The home directory where ASM
is installed.
/ASM_TEST0
ASM_USER
User name of the Oracle ASM
oracle
administrator. It is used to start or
stop the ASM instance and
mounting or dismounting the ASM
disk groups.
ASM_SID
The ASM session name that
uniquely identifies an ASM
instance.
+ASM
NOTE: If you modify this
attribute after the # cmmakepkg
-i -m -t command is run, you
must reconfigure the
MONITOR_PROCESSES attribute
to reflect the same. Also,
reconfigure PFILE attribute to
reflect the change if PFILE is
using this attribute.
If this attribute is set to no, you
must comment the
LISTENER_NAME and
oracle_listener_service
attributes. For example:
#LISTENER_NAME
LISTENER
yes
The default value is yes.
LISTENER_NAME
This parameter holds the names LISTENER
of the Oracle listener processes.
You can add more than one
listener name by adding to the
array variable LISTENER_NAME.
The default value is LISTENER.
LISTENER_PASS
The passwords of the listener
processes.
ora123
This parameter is commented by
default.
LISTENER_RESTART
This parameter defines the
number of attempts to be made
to restart the listener. If the
listener does not restart after
LISTENER_RESTART number
of consecutive attempts, the
package fails over.
2
If the listener restart is successful,
then the next time listener fails,
the toolkit again tries
LISTENER_RESTART number of
times to restart the listener.
This parameter is commented by
default.
NOTE: If the package is in
maintenance mode, then the
listener will not be restarted even
if LISTENER_RESTART is set to
a definite value.
Configuring modular Oracle ASM instance and Database instance toolkit packages
25
Attribute
Description
Example
PFILE
The Oracle database parameter
file. If this attribute is not
specified, Oracle picks this up
from the Oracle configuration
directory
$ORACLE_ORACLE_HOME/dbs.
${ORACLE_ORACLE_HOME}/dbs/
init${ORACLE_SID_NAME}.ora
When this parameter is
configured, it overrides the default
file location.
If both PFILE and SPFILE are
present and this parameter is not
specified, then Serviceguard
toolkit for Oracle makes use of
PFILE as the parameter file.
To make use of SPFILE in the
toolkit, do not configure this
parameter, and remove the
default PFILE from its default
location.
MONITOR_PROCESSES
This is the list of all the critical
Oracle processes for an instance
that must be executing to assume
that the instance is up and
running. You can add more
instance processes.
• ora_pmon_${ORACLE_SID_NAME}
• ora_dbw0_${ORACLE_SID_NAME}
• ora_ckpt_${ORACLE_SID_NAME}
• ora_smon_${ORACLE_SID_NAME}
• ora_lgwr_${ORACLE_SID_NAME}
• ora_reco_${ORACLE_SID_NAME}
MONITOR_INTERVAL
The time interval, in seconds, this 30
script waits between checks to
ensure that Oracle instance is
running. The default value is 30
seconds.
TIME_OUT
The time period, in seconds, this 30
toolkit waits for a completion of a
normal shutdown before initiating
forceful halt of the application.
The TIME_OUT variable is used
to protect against a worst case
scenario where a hung database
or ASM instance prevents the halt
script from completing, therefore
preventing the standby node from
starting the instance.
The TIME_OUT variable has no
effect on package failover times.
The default value is 30 seconds.
PARENT_ENVIRONMENT
This parameter is used to mention no
if the Oracle or ASM user's shell
must be invoked as a new shell
or as a subshell that inherits the
variables set in the parent shell.
You can set this parameter to yes
or no.
Set this parameter to yes if the
Oracle or ASM user's shell must
be invoked as a subshell.
26
Using HPE Serviceguard toolkit for Oracle
Attribute
Description
Example
Set to no if the Oracle or ASM
user's shell must be invoked as a
new shell. If set to no, the Oracle
or ASM user's .profile file is
executed and the variables set in
this file are available to the new
shell.
CLEANUP_BEFORE_STARTUP
This parameter indicates whether no
shutdown abort must be executed
before the startup of the Oracle
or ASM instance.
You can set this parameter to yes
or no. The default value is no.
USER_SHUTDOWN_MODE
This parameter is used to specify abort
the instance shutdown mode only
when a shutdown is initiated by
the user and not because of the
failure of a service.
You can set this parameter to
abort or immediate.
If abort is specified, the instance
is shutdown using the abort
option. If immediate is specified,
the instance is shutdown using
the immediate option. The
default value is abort.
KILL_ASM_FOREGROUNDS
If ASM is being used, this
yes
parameter specifies whether any
ASM foreground processes
having file descriptors open on
the dismounted disk group
volumes need to be killed or not.
You can set this parameter to yes
or no. The default value is yes.
If this parameter is set to yes,
ASM foreground processes of the
form oracle<ASM_SID> having
file descriptors open will be killed.
If this parameter is set to no, and
if the ASM foreground processes
have file descriptors open, it leads
to package halt and hence, fails
the package failover.
DB_SERVICE
This parameter is used to start
all
and stop the DB service through
the DB package. It is commented
by default. You can set the
attribute to all or specify the DB
service names that must be
started and stopped by the DB
package.
DISABLE_BACKUP_MODE
This parameter is used to disable no
the Oracle database from the
backup mode. The valid values
for this parameter are yes and
no. By default, the value is set to
no. This means, if the database
Configuring modular Oracle ASM instance and Database instance toolkit packages
27
Attribute
Description
Example
is in the backup mode after
package failover or restart, the
Oracle toolkit package fails to
start. You have to manually get
the database out of the backup
mode and then start the package.
When the value of
DISABLE_BACKUP_MODE is set
to yes and the database is in the
backup mode after failover or
restart, the Oracle toolkit package
will alter the database to remove
the backup mode and start the
database. The database is altered
to remove the backup mode only
if START_MODE is set to open.
RETRY_INTERVAL_ORA_ERROR This parameter defines the time
interval to wait in case of ORA
errors during database hang
monitor.
email_id
0
This parameter is used to provide abc@domain-name.com
email addresses for the toolkit
alert notification recipients. Each
email_id parameter can have
one of the following values:
• A complete email address
• An alias
• A distribution list
You can list multiple recipients by
repeating the email_id field and
address. Toolkit sends an alert
email on listener and package
failure. If configured, the alert mail
is sent for Oracle database hang
detection. For example:
$SGCONF/scripts/tkit/
oracle/tkit_module.sh
oracle_hang_monitor 30
alert. By default, email_id
is commented.
CRITICAL_PDB
This parameter is used to provide
the pluggable database names,
whose failure triggers the failover
of the package and thereby
causes the restart of the CDB on
the adoptive node. This attribute
can have one of the following
values:
• PDB names
• value “any”
• value“ none”
To monitor multiple PDB names,
repeatCRITICAL_PDB field and
PDB name. haoraclepdb.mon.
The package failover is decided
28
Using HPE Serviceguard toolkit for Oracle
CRITICAL_PDB any
CRITICAL_PDB none
CRITICAL_PDB pdb1
CRITICAL_PDB pdb2
CRITICAL_PDB pdb3
Attribute
Description
Example
based on the value specified for
this attribute.
• If the value specified is “any”,
the package fails over if any of
the plugged PDB fails.
• If the value is "none", the
package will not failover in
case of PDB failures. The
package fails over only if the
CDB fails.
e.
Edit the following service parameters:
•
•
•
Monitor service for Oracle DB instance processes:
Attribute Name
Example
service_name
oracle_db_service
service_cmd
"$SGCONF/scripts/tkit/oracle/
tkit_module.sh oracle_monitor"
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
Monitor service for Oracle listener processes:
Attribute Name
Example
service_name
oracle_listener_service
service_cmd
"$SGCONF/scripts/tkit/oracle/
tkit_module.sh
oracle_monitor_listener"
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
Monitor service for Oracle hang detection:
Attribute Name
Example
service_name
oracle_hang_service
service_cmd
"$SGCONF/scripts/tkit/oracle/
tkit_module.sh oracle_hang_monitor
30 failure"
service_restart
none
service_fail_fast_enabled
No
Configuring modular Oracle ASM instance and Database instance toolkit packages
29
•
f.
g.
Attribute Name
Example
service_halt_timeout
300
service_halt_on_maintenance
no
Monitor service for Oracle pluggable database:
Attribute Name
Example
service_name
oracle_pdb_monitor_service
service_cmd
"$SGCONF/scripts/tkit/oracle/
tkit_module.sh oracle_pdb_monitor"
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
To configure the dependency for the modular Oracle ASM database package, you must
set the dependency for Oracle DB package on Oracle ASM instance package. Set the
following parameters in the package configuration file to add the dependency:
Attribute Name
Example
dependency_name
asm_dependency
dependency_condition
<ASM MNP package_name>=up
dependency_location
same_node
Apply the package configuration using the command:
# cmapplyconf -P pkg2.conf
Managing a package
You can manage a package by adding, running, maintaining, halting, and deleting it.
Adding the package to the cluster
After the setup is complete, add the package to the Serviceguard cluster and start it.
$ cmapplyconf -P ORACLE_TEST0
$ cmmodpkg -e -n <node1> -n <node2> ORACLE_TEST0
$ cmmodpkg -e ORACLE_TEST0
For more information on managing packages, see the latest Managing Serviceguard manual
available at http://www.hpe.com/info/linux-serviceguard-docs.
Running a package
To start a package, run the following command:
# cmrunpkg <package_name>
30
Using HPE Serviceguard toolkit for Oracle
Maintaining a package
Maintenance mode is used when user wants to perform maintenance on components of a modular,
failover package while the package is running. It is chiefly used to make modifications to the
networking and generic resource components. Serviceguard will not monitor for any failures
when the oracle toolkit package is in maintenance mode. For more information, see Maintaining
a Package: Maintenance Mode available at Managing HPE Serviceguard for Linux.
The Oracle toolkit package has specific services that monitor critical Oracle processes and
listeners for failures. They also inspects databases for hung situations. When the package services
detect failures of any component, Serviceguard will initiate a corrective action, which can either
be a restart of some of the components or a failover of the package based on the failed component
and the package configuration. When the Oracle toolkit package is in maintenance mode, though
its services continue to run, Serviceguard will not react to any of the failure and no corrective
action like a restart or failover is triggered.
Example 1
•
When the Oracle toolkit package is not in maintenance mode, if the Oracle database hangs
and becomes unresponsive, Serviceguard triggers a failover to alternate mode.
•
When the Oracle toolkit package is in maintenance mode, if the Oracle database hangs and
becomes unresponsive, Serviceguard will stop monitoring database for hung situations.
And, there won’t be any package failover. But as soon as the package is taken out of
maintenance, Oracle toolkit restarts the monitored database processes and may failover to
alternate node in case the database hangs.
Example 2
•
When the Oracle toolkit package is not in maintenance mode and the LISTENER_RESTART
attribute is set to 2, if a listener failure occurs, the Oracle toolkit attempts to restart the
monitored listener twice.
•
When the Oracle toolkit package is in maintenance mode and the LISTENER_RESTART
attribute is set to 2, if a listener failure occurs, Oracle toolkit will not restart the monitored
listener. But as soon as the toolkit package is taken out of maintenance, listener monitor
service restarts the monitored listener.
When a Serviceguard package is in maintenance mode, it ensures the package will not failover
to an alternate node due to any failure in the application that is configured in the package.
To perform maintenance:
1. Enable the maintenance mode for the package using the cmmodpkg command.
cmmodpkg -m on <package-name>
2.
Perform the maintenance actions.
For example, you can change the configuration of the application instance or install an
application patch.
NOTE: You can ignore the errors or warnings in the package log file when the package
is set to maintenance mode.
3.
Disable the maintenance mode for the package using the cmmodpkg command.
cmmodpkg -m off <package-name>
NOTE: Ensure that an application instance is up and running before disabling the
maintenance mode.
Managing a package
31
Online Modification of the Oracle Toolkit package
In addition to the Online modification feature supported by Serviceguard, the online modification
feature of toolkit allows you to modify some of the toolkit attribute values, while the toolkit package
is up and running. Table 5 (page 32) lists the attributes that are supported for online toolkit
package modification.
Table 3 List of Attributes Supported for Online Toolkit Package Modification
32
Attribute
Description
START_MODE
Oracle Toolkit supports online modification of this attribute.
This attribute denotes the database startup. The attribute
value takes effect during the next startup of the database.
LISTENER
Oracle Toolkit supports online modification of this attribute.
This attribute can have either yes or no as its value. By
default, the value is set to yes. If you do not want to
monitor the listener in the toolkit package, LISTENER can
be set to no and comment out LISTENER_NAME and
oracle_listener_service attributes.
LISTENER_NAME
Oracle Toolkit supports online modification of this attribute.
The listener name can be changed, added, or removed
during online modification. The Listeners that are modified
or added must be configured and started manually before
adding to the toolkit package during online modification.
This ensures the listener is monitored when the package
is applied online. Similarly, if the Listener is removed from
the toolkit package, manual intervention is required to
stop the Listener.
LISTENER_PASS
Oracle Toolkit supports online modification of this attribute.
The Listener password can be modified and applied online
only if the values mentioned in LISTENER_PASS array
correspond with LISTENER_NAME array.
LISTENER_RESTART
Oracle Toolkit supports online modification of this attribute.
This parameter defines the number of attempts made to
restart the listeners when a listener failure is detected.
This parameter can be modified and applied online only
if the values mentioned in LISTENER_RESTART array
correspond with LISTENER_NAME array.
PFILE
Oracle Toolkit will support the online modification of this
attribute, provided the user changes the permissions or
ownership for the new location of the parameter file. The
file must be present in all the package nodes at the same
location.
MONITOR_PROCESSES
Oracle Toolkit supports online modification of this attribute.
This attribute defines the Oracle processes to be
monitored by the toolkit package. By default, six critical
Oracle processes are monitored. This array can be
increased or reduced and the processes will be monitored
in the next monitor cycle.
MONITOR_INTERVAL
Oracle Toolkit supports online modification of this attribute.
TIME_OUT
Oracle Toolkit supports online modification of this attribute.
PARENT_ENVIRONMENT
Oracle Toolkit supports online modification of this attribute.
CLEANUP_BEFORE_STARTUP
Oracle Toolkit supports online modification of this attribute.
USER_SHUTDOWN_MODE
Oracle Toolkit supports online modification of this attribute.
KILL_ASM_FOREGROUNDS
Oracle Toolkit supports online modification of this attribute.
Using HPE Serviceguard toolkit for Oracle
Table 3 List of Attributes Supported for Online Toolkit Package Modification (continued)
Attribute
Description
DB_SERVICE
Oracle Toolkit supports online modification of this attribute.
User defined database services can be added or removed
during online modification. Database service must be
configured and running during online modification, so that
toolkit package can use the start or stop control on user
defined database services.
DISABLE_BACKUP_MODE
Oracle Toolkit supports the online modification of this
attribute. This parameter is used to disable the Oracle
database from the backup mode. The valid values for this
parameter are yes and no. By default, the value is set to
no. This means, if the database is in the backup mode
after package failover or restart, the Oracle toolkit package
fails to start. You have to manually get the database out
of the backup mode and then start the package.
When the value of DISABLE_BACKUP_MODE is set to yes
and the database is in the backup mode after failover or
restart, the Oracle toolkit package will alter the database
to remove the backup mode and start the database. The
database is altered to remove the backup mode only if
START_MODE is set to open.
email_id
Oracle Toolkit supports online modification of this attribute.
This parameter is used to provide email addresses for the
toolkit alert notification recipients. Each email_id
parameter can have one of the following values:
• A complete email address
• An alias
• A distribution list
You can list multiple recipients by repeating the email_id
field and address. Toolkit sends an alert email on listener
and package failure. If configured, the alert mail is sent
for Oracle database hang detection. For example:
$SGCONF/scripts/tkit/oracle/tkit_module.sh
oracle_hang_monitor 30 alert. By default,
email_id is commented.
CRITICAL_PDB
The Oracle toolkit supports online modification of this
attribute. This parameter is used to provide the pluggable
database names. This parameter can have one of the
following values:
• PDB names
• value “any”
• value “none”
Monitoring service attributes
Oracle Toolkit supports the online modification of service
attributes. By default, three monitoring services are
started. Services can be added, changed, or removed
depending on monitoring requirements.
To perform online package modification:
1. While package is up and running, retrieve the current package configuration values using
the command cmgetconf.
For example: #cmgetconf -p <<Toolkit_Package_Name>> <<Latest_Toolkit_Conf_File>>
2.
Backup this latest configuration file using a copy utility. This allows you to refer to the previous
configuration values.
For example: # cp -p <<Latest_Toolkit_Conf_File>> <<Backup_Toolkig_Conf_File>>
Managing a package
33
3.
4.
Edit the <<Latest_Toolkit_Conf_File>> file to modify the required attribute values.
Use the cmcheckconf command to verify the package configuration file.
For example: #cmcheckconf -P <<Latest_Toolkit_Conf_File>>
5.
Use the cmapplyconf command to add the new changes in the package configuration to
the Serviceguard cluster environment.
For example: #cmapplyconf -P <<Latest_Toolkit_Conf_File>>
NOTE:
Information related to package reconfiguration is logged in the package log.
If you modify any other attributes, the following error message appears on the console.
ERROR: Online modification of attribute "<Attribute name>" is not allowed
For example: If an user tries to change the value of ORACLE_HOME while the package is running,
the cmapplyconf command fails with an error message.
ERROR: Online modification of attribute "ORACLE_HOME" is not allowed.
Handling failures during online package reconfiguration
This section describes about how to troubleshoot failures during online package reconfiguration.
Table 4 (page 34) describes how to fix the errors in the affected modules that are encountered
during online addition of a package.
Table 4 Modules affected during online addition
Description
Modules affected during
online addition
How to rectify the failures or
apply those that are not
attempted
Example
Adding an
external pre
script to the
package
sg/external_pre_script If an external pre script which is To start the external pre script:
(external.sh)
added to the package
#extern_pre_script.sh start
configuration failed to start, run
the script manually with start
option.
script_name start
Extending
storage to the
package
sg/filesystem
(filesystem.sh)
sg/volume_group
(volume_group.sh)
sg/pr_cntl
(pr_cntl.sh)
If addition of storage has failed, To add PR keys, you must register
ensure the following:
on all the paths and then reserve on
one path.
• Persistent reservation is
To view all the paths:
added to the disk.
#multipath -ll
• Volume group is activated
To view the node PR keys:
with hosttags.
NOTE: In this example, the
• Verify and repair the file
node_pr_key is 72810001.
system.
• The mount point is mounted. #cmviewcl -v -f line |
grep node_pr_key
For more information, see
#sg_persist --out -G
sg_persist(1m),
--param-sark=72810001
vgchange(1m),fsck(1m),
/dev/sde
multipath (1m), and
mount(1m) manpage.
#sg_persist --out -R
--param-rk=72810001
--prout-type=5 /dev/sde
Add hosttags for node
test1.ind.hp.com:
#vgchange --addtag
test1.ind.hp.com vg_dd1
Activate the disk:
#vgchange -a y vg_dd1
Run the following commands:
34
Using HPE Serviceguard toolkit for Oracle
Table 4 Modules affected during online addition (continued)
#e2fsck -y
/dev/vg_dd1/lvol1
#mount -t ext3
/dev/vg_dd1/lvol2 /mnt1
Adding an IP sg/package_ip
to the package (package_ip.sh)
If an IP address which is added
to the package configuration
failed to add or not attempted,
use the cmmodnet command to
add the IP address.
To add the IP to the package
10.149.2.5:
#cmmodnet -a -I 10.149.2.5
10.149.2.0
For more information, see
cmmodnet (1m) manpage.
sg/external_script
Adding an
external script (external.sh)
to the package
If an external script which is
added to the package
configuration failed to start, run
the script manually with start
option.
To halt external pre script:
#extern_pre_script.sh stop
script_name start
Adding a
service to the
package
sg/service
(service.sh)
If a service which is added to the To run the process as service:
package configuration failed to #cmrunserv db1
start or not attempted, use the
/var/opt/db/database1
cmrunserv command to start
the service
For more information, see
cmrunserv (1m) manpage.
Adding Oracle tkit/oracle/oracle
toolkit module
to the package
If an Oracle toolkit module added
to the package configuration fails
to start, then package will not
fail, but only the new modules
will start during next successful
start of the package after
correcting errors, if any.
#Halt the package
cmhaltpkg
<oracle_toolkit_pkg>
# Correct the errors by editing the
package configuration file.
#Apply changes to the package
configuration file
cmapplyconf –P <pkgconfig>
#Run the package
cmrunpkg
<oracle_toolkit_pkg>
Modifying
Oracle toolkit
attributes
values online
tkit/oracle/oracle
If an Oracle toolkit service fails
to start after modifying Oracle
toolkit attribute value while
package is up and running, then
the package will not fail, but the
new values will not be taken in
to effect until package is halted
and successfully started again.
#Halt the package
cmhaltpkg
<oracle_toolkit_pkg>
#Run the package
cmrunpkg
<oracle_toolkit_pkg>
Halting a package
To halt a package, run the following command:
# cmhaltpkg <package_name>
Deleting a package
To delete a package from the cluster, run the following command:
# cmdeleteconf -p <package_name>
Managing a package
35
This command prompts for a confirmation, before removing the package configuration from the
cluster binary file, unless you use the -f option.
36
Using HPE Serviceguard toolkit for Oracle
5 Oracle toolkit features
Alert mail notification
This feature facilitates you to configure email address to get the alert notification from toolkit
package. Alert mails are sent to configured email addresses in case of failure scenarios. It can
be a resource failure or altogether database failure that results in package failure.
Toolkit sends an alert email with failure messages in the following scenarios:
•
Listener failure.
Message: Oracle listener Listener1 failed in Oracle toolkit package Oracle_pkg on node
NODE1.
•
Multiple listeners failed.
Message: Oracle listener Listener1 failed in Oracle toolkit package Oracle_pkg on node
NODE1.
Message: Oracle listener Listener2 failed in Oracle toolkit package Oracle_pkg on node
NODE1.
•
Database hung.
If configured, alert mail will be sent for Oracle database hang detection. For example:
$SGCONF/scripts/tkit/oracle/tkit_module.sh oracle_hang_monitor 30
alert.
Message: Database hang detected in Oracle toolkit package Oracle_pkg on node NODE1.
•
Toolkit package failure.
Message: Oracle toolkit package Oracle_pkg failed on node NODE1.
Cluster verification feature for Oracle toolkit
Cluster verification is a proactive mechanism to identify cluster inconsistencies that adversely
affect toolkit package failover to an adoptive node.
Advantages of cluster verification
The important advantages of cluster verification are:
•
To identify inconsistencies in Serviceguard and the Oracle toolkit resources across package
nodes before you create and configure the Oracle toolkit package.
•
To automate the process of validating certain attribute values specified in the package
configuration file.
•
To achieve complete validation of cluster and application toolkit using single command
execution.
•
To build a steady cluster for high available solution.
NOTE: The verification for the Oracle configuration files and binaries are done only if they are
placed in the root filesystem.
You can use either the cmcheckconf –v or cmcheckconf –P commands to initiate the cluster
verification.
Run the cmcheckconf –v command to perform cluster wide verification of resources. This
command also verifies the packages that are presently deployed in the system. Run the
cmcheckconf -v 2 command to display the ERROR, WARNING, or INFO messages during
the package verification.
Alert mail notification
37
Run the cmcheckconf –P command to perform the package verification irrespective of the
state of the package in the cluster.
The following are the steps to initiate the package verification by using the cmcheckconf –P
command:
1. While the package is running, retrieve the current package configuration values using the
cmgetconf command.
For example: # cmgetconf -p <package_name> <package_name.conf>
2.
Run the cmcheckconf command to verify the package configuration file.
For example: # cmcheckconf –P <package_name.conf>
Use the cluster verification feature for the Oracle toolkit to:
•
Verify the presence of sqlplus, and opatch commands. If the command does not exist,
the console displays the following error messages:
ERROR: The Oracle command "$ORACLE_ORACLE_HOME/bin/sqlplus" is
missing. ERROR: The Oracle command
"$ORACLE_ORACLE_HOME/OPatch/opatch" is missing.
Reason: The Oracle command the sqlplus does not exist in the ${ORACLE_HOME}/bin/
and opatch command does not exist in the $ORACLE_ORACLE_HOME/OPatch/opatch
directories.
Solution: You must ensure that Oracle database software installation has been done properly.
If it is not installed, reinstall the same.
•
Verify for the consistency of package attributes.
◦
If the LISTENER attribute is set to no and if the oracle_listener_service is not
commented. Then, the console displays the following error message:
ERROR: The LISTENER parameter has been set to "no" but the
service_cmd has still the listener service
"oracle_listener_service". Comment out the
"oracle_listener_service" service.
Solution: Comment the oracle_listener_service.
◦
If the LISTENER attribute is set to yes, then, existence of listener.ora is checked.
If the configuration file does not exist, the console displays the following error message:
ERROR: Listener file
"$ORACLE_ORACLE_HOME/network/admin/listener.ora" is missing.
Solution: Create a listener.ora file in $ORACLE_ORACLE_HOME/network/admin
folder along with the required values or copy the configuration file from another node
where the listener is in running condition.
•
Verify if the PFILE/SPFILE files exist on all the nodes.
If both these files do not exist, the console displays the following error message:
ERROR: PFILE "${ORACLE_ORACLE_HOME}/dbs/init${sid_name}.ora" and
SPFILE "${ORACLE_ORACLE_HOME}/dbs/spfile${sid_name}.ora" are missing.
Solution: Create these files by using Oracle command or copy from the package node where
the Oracle database is in running condition.
•
Verify if the ORACLE and ASM users exist on all the nodes.
If both these users do not exist, the console displays the following error messages:
ERROR: Oracle user "$ORACLE_ORACLE_ADMIN" doesn't exist.
38
Oracle toolkit features
ERROR: Oracle ASM user \"$ORACLE_ASM_USER\" doesn't exist.
Solution: Create the Oracle and/or ASM instance users with user ID and Group ID same
as on other package nodes.
•
Verify whether the environment variables ORACLE_HOME and ORACLE_SID are set for the
Oracle user.
If both these variables are not set, the console displays the following info messages:
INFO: ORACLE_HOME environment variable is not set for
"${ORACLE_ORACLE_ADMIN}" user.
INFO: ORACLE_SID environment variable is not set for
"${ORACLE_ORACLE_ADMIN}" user.
Solution: Export the environment variables ORACLE_HOME and ORACLE_SID from the Oracle
user by adding them to the profile file.
•
Verify whether CRITICAL_PDB attribute is specified only for Oracle 12c container database.
If this attribute is set for non container database then the following error is displayed:
"ERROR: Specifying critical PDB is not supported with Oracle version
lesser than 12cR1"
This validation is applicable when package is created and also while performing the online
reconfiguration of the CRITICAL_PDB attribute.
•
Verify whether the PDB names listed is a valid PDB name that exists in the CDB. This
verification is applicable only during online reconfiguration of the package and when the
value specified for the CRITICAL_PDB is the list of names of pluggable databases, which
are considered to be critical. If the specified PDB does not exist, then the following error is
displayed:
"ERROR: Specified Critical PDB PDB_name does not exists in the
Container Database."
Support for sourcing user defined environment variables for Oracle instance
Oracle toolkit provides an option to set the environment variables to specific Oracle instance
during the package startup.
To set the environment variables specific to Oracle instance, do the following:
1. Set the value for the PARENT_ENVIRONMENT attribute to yes in the package configuration
file.
2. Define the environment variables in the {ORACLE_SID}_customer.conf file. For example,
orcl_customer.conf where ORACLE_SID = orcl.
3. Place the file in the $SGROOT/oracletoolkit directory.
WARNING! Hewlett Packard Enterprise recommends you to not override the Oracle toolkit
attributes defined in the package configuration through the {ORACLE_SID}_customer.conf
file.
NOTE: While creating Oracle package through Serviceguard Manager GUI,
{ORACLE_SID}_customer.conf file must be created through CLI.
Cluster verification feature for Oracle toolkit
39
6 Supporting Oracle features
Support for Oracle 12c Multitenant architecture
This section explains support for container databases. It also explains how to package and deploy
Oracle Multitenanat Container Databases (CDB) instances in a Serviceguard cluster using the
Oracle toolkit packages. This feature enables you to start, stop, and monitor the container database
(CDB) and the PDBs configured in it. With A.12.00.50 release, the support for failover of CDB
on PDB failure is introduced. Serviceguard monitors the CDB and PDB and triggers a failover
on a failure. This restarts the entire CDB on the adoptive node.
Starting CDB
During the package startup, CDB instance will be started. If any of the PDB fails to start or fails
to move toNORMAL state, then appropriate log message will be generated in the package log file.
The package start will succeed even if some of the PDB did not start.
The attribute START_MODE is used to determine the startup mode of PDB. When CDB is started,
based on START_MODE attribute value, Pluggable Database (PDB) will be opened. The following
are the three options supported by the Oracle toolkit for startup:
Mount—CDB and all PDB will be started. Also, CDB and all PDB including the seed PDB will
be mounted but not opened.
Open—CDB and all PDB will be started. Also, CDB and all PDB except the seed PDB will be
opened in the READ WRITE Mode.
Nomount—CDB will be started. Neither CDB nor PDB is mounted or opened.
Stopping CDB
During the package halt, all the PDBs are shut before CDB instance is shut down. If any of the
PDB fails to close, then appropriate log message will be generated in the package log file.
The option immediate is supported by the Oracle toolkit for stopping CDB. If the attribute
USER_SHUTDOWN_MODE is set to immediate in package configuration file, then all the PDBs
will be closed with immediate option along with CDB. If the attribute is not set to immediate,
CDB will be halted and all PDBs will be closed with no option. The package log contains the list
of PDBs that are not in NORMAL state during the package halt process. The database hang
detection monitoring service monitors the status of all PDBs.
NOTE:
•
This new feature is of CDB failover on PDB failure. When the status of PDB is UNUSABLE,
the Oracle toolkit sends an email alerts to the email ids that are registered in the package
configuration file. The package will not fail over when any of the PDBs configured in the CDB
fail. Package failover is initiated only when the CDB fails.
•
It is recommended that the filesystem used to store the PDB datafiles must be configured
in the Serviceguard package.
Monitoring PDB
In this release, a new attribute calledCRITICAL_PDB is introduced. You can configure critical
PDBs; failure of which, will initiate the package fail over, which results a restart of the CDB on
the adoptive node. The CRITICAL_PDB attribute can take the following values:
•
PDB names:
Names of the pluggable database, which user considers as critical in nature. If the PDB
names are specified, then on failure of any of the specified PDB, the package will be failed
over to the adoptive node. However, the non critical PDBs are still monitored but the failure
40
Supporting Oracle features
of which will not initiate the package failover. Any PDB failure that is, critical or non- critical
the notifications will be send to the email ids configured in the package.
•
Value as “any”
This value specifies that all of the pluggable database associated with the specified CDB
are monitored. If the health of the any one of the pluggable database fails, then package
fails over after retrying to restart the PDB.
•
Value as “none”
If the value specified is “none”, then the package will not be failed over in case of PDB failure.
The package will failover only in case of CDB failure.
Support for multiple listeners
This section explains support for multiple listeners with the Serviceguard toolkit for Oracle. It
enables you to configure:
•
Single service to monitor all the listeners together
•
A separate service to monitor each listener
NOTE: In a single package, it is not possible to configure the services using both the
approaches. You must either configure all the listeners using a single service, or use a separate
service for each listener.
Ensure that the elements in the LISTENER_RESTART array and the LISTENER_PASS array
correspond to those in the LISTENER_NAME array in the package configuration file. In a situation
where some listeners do not require a restart value and others do not require a password, ordering
the elements in the arrays becomes difficult. In such a case, you must set the
LISTENER_RESTART to 0 and the LISTENER_PASS to empty quotes ("").
The LISTENER_NAME, LISTENER_PASS, and LISTENER_RESTART are configured in the package
configuration file.
For example:
Attribute name
Description
tkit/oracle/oracle/LISTENER
yes
tkit/oracle/oracle/LISTENER_NAME
<listener_name1>
tkit/oracle/oracle/LISTENER_PASS
""
tkit/oracle/oracle/LISTENER_RESTART
0
tkit/oracle/oracle/LISTENER_NAME
<listener_name2>
tkit/oracle/oracle/LISTENER_PASS
""
tkit/oracle/oracle/LISTENER_RESTART
0
Single service to monitor all the listeners together
This is the default behavior. If one of the configured listeners fail, it does not impact the service.
However, if all the configured listeners fail, the service fails, and it leads to the failover of the
package to an alternate node. To enable single service to monitor all listeners, you must not pass
Support for multiple listeners
41
the listener name to the service command. The service_cmd in the package configuration file
appears as follows:
Attribute name
Description
service_name
oracle_listener_monitor
service_cmd
“$SGCONF/scripts/tkit/oracle/tkit_module.sh
oracle_monitor_listener ”
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
A separate service to monitor each listener
This service is recommended if listeners are critical. This approach of packaging listeners provides
high availability to listeners. If any one of the configured listeners fail, it leads to a failure of the
listener service, and the package fails over to an alternate node. In this case, the default
service_cmd in the package configuration file must be modified to include the listener name.
For example:
Attribute name
Description
service_name
oracle_listener_monitor_1
service_cmd
“$SGCONF/scripts/tkit/oracle/tkit_module.sh
oracle_monitor_listener <listener_name1>”
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
You must set the above attributes for each listener. For example:
Attribute name
Description
service_name
oracle_listener_monitor_2
service_cmd
“$SGCONF/scripts/tkit/oracle/tkit_module.sh
oracle_monitor_listener <listener_name2>”
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
Ensure that the LISTENER_NAME passed to the service_cmd, matches one of the listener's
names specified in the package configuration file.
42
Supporting Oracle features
Support for user defined database services
You can start and stop the user defined database services through the database package when
used with the Serviceguard toolkit for Oracle. To use this feature, you must set the Database
Service management policy to manual and start the database in open startup mode.
NOTE:
•
This feature is available only on systems running Oracle Clusterware.
•
If user defined DB services are configured to start with database package startup, then
database must be started in open mode. Set the package configuration file attribute
START_MODE to open.
The attribute, DB_SERVICE is used to start and stop the database service through the database
package. This attribute is commented by default. You can set the attribute DB_SERVICE to all
or specify the database service names that must be started and stopped by the database package.
To start and stop all the database services configured in the database managed by this package,
set this attribute to all.
For example: tkit/oracle/oracle/DB_SERVICE all
To start and stop one or more specific database services through this database package, you
must specify the service names using multiple instances of this attribute, as shown in the following
example:
Example:
•
tkit/oracle/oracle/DB_SERVICE <service1>
•
tkit/oracle/oracle/DB_SERVICE <service2>
•
tkit/oracle/oracle/DB_SERVICE <service3>
Support for database hang detection
You can configure a separate service that will detect the database hang and take appropriate
action. Database hang is detected by connecting to the database and checking its status. If this
process takes an unusually long time or if the toolkit is unable to retrieve the database status,
then it is assumed that the database is hung. You can configure two attributes when you configure
this service:
1. TIMEOUT: This attribute defines how long the script must wait for the database hang check
to return success. If the database health cannot be determined within this time after
connecting to the database, then it is assumed that the database is hung. It can have any
positive integer as value. The default value for TIMEOUT is 30 (seconds). This must not be
confused with the TIME_OUT package attribute or the service_halt_timeout attribute.
2. ACTION: This attribute defines the action that must be taken if a database hang is detected.
Currently, this attribute can take one of the following values:
•
log - Log a message: A message is logged to the package log every time a hang is
detected. If the MONITOR_INTERVAL attribute is set to 30 seconds, then a message is
logged to the package log file every 30 seconds.
•
failure - Failover the package to adoptive node.
The default value for ACTION is failure.
The syntax of the service command is as follows:
service_cmd "$SGCONF/scripts/tkit/oracle/tkit_module.sh
oracle_hang_monitor <TIMEOUT> <ACTION>"
Following is an example in which the TIMEOUT is set to 40 seconds and the ACTION is set to
failure:
Support for user defined database services
43
service_name db_hang_check
service_cmd "$SGCONF/scripts/tkit/oracle/tkit_module.sh oracle_hang_monitor 40 failure"
service_restart none
service_fail_fast_enabled no
service_halt_timeout 300
service_halt_on_maintenance no
Support for Oracle database backup mode
During the oracle database instance package start or package failover on the other cluster node,
if the database is in the online backup mode based on your input through the package attribute
DISABLE_BACKUP_MODE, the toolkit gets the database out of the backup mode. The valid
values for this parameter are yes and no. By default, the value is set to no.
For example: tkit/oracle/oracle/DISABLE_BACKUP_MODE no.
When DISABLE_BACKUP_MODE parameter is set to yes and the database is in the backup
mode, the Oracle database instance package start or failover on other node will alter the database
to remove the backup mode before starting. Also, START_MODE parameter should be set to
open. If Oracle database recovery is needed, then package fails to start with a following message
in the package log:
Unable to get the database out of the backup mode. You must get the database out of the backup
mode before starting the package. The toolkit does not take any action on database recovery.
If DISABLE_BACKUP_MODE is set to no then this feature will not be enabled.
44
Supporting Oracle features
7 Troubleshooting
This section explains some of the problem scenarios that you might encounter while using the
Serviceguard toolkit for Oracle in a Serviceguard cluster.
Problem scenario
Possible cause
Oracle database toolkit packages
startup fails, and package log
contains the following message:
Oracle database failed to start on this Look at the sub-directories for
node.
database and ASM instances under
$ORACLE_BASE/admin/, where log
and trace files for these instances are
deposited.
Oracle Instance startup failed.
Recommended action
Oracle database toolkit packages
startup fails, and package log
contains the following message:
Package attributes such as
Refer the Oracle logs to edit the
ORACLE_HOME, SID_NAME, PFILE package configuration file with
are not initialized with correct values. appropriate values. Apply the
configuration and start the package.
<file name > (parameter file): No such
file of directory.
User defined DB services do not start Oracle database is not started in
with package startup even when they open mode.
are configured to start with database
package.
If user defined DB services are
configured to start with database
package startup, then database must
be started in open mode.
Set the package configuration file
attribute START_MODE to open.
User defined DB services do not start Either the service name mentioned 1. Edit the package configuration file
with package startup and package
in the package configuration file is
with correct service name and
log shows:
wrong or the target Oracle database
reapply the package.
is not registered with Oracle
2. Log in as Oracle administrator to
Could not retrieve the status of
Clusterware.
set the database management
service name.
policy to MANUAL and register the
database with Oracle Clusterware.
For 10g:
#$ORACLE_HOME/bin/srvctl
modify database –d <dbname> -o
<oracle home> y manual.
For 11g:
#$ORACLE_HOME/bin/srvctl
modify database –d <dbname> -o
<oracle home> y manual.
NOTE: You must set the user
defined DB service management
policy to MANUAL in case of Oracle
11.2.0.1.
#$ORACLE_HOME/bin/srvctl
modify service –d <dbname> -s
<service name> -y MANUAL.
Failure in volume group deactivation Failure in killing all the processes
and unmounting the file system,
accessing the Volume group and File
resulting in package halt failure.
system which are currently used by
the Oracle application.
Perform these steps to ensure the
issue does not persist:
1. Edit the package configuration file.
2. Set the value of
fs_umount_retry_count to 5
and apply the configuration and
start the package.
45
8 Support and other resources
Accessing Hewlett Packard Enterprise Support
•
For live assistance, go to the Contact Hewlett Packard Enterprise Worldwide website:
www.hpe.com/assistance
•
To access documentation and support services, go to the Hewlett Packard Enterprise Support
Center website:
www.hpe.com/support/hpesc
Information to collect
•
Technical support registration number (if applicable)
•
Product name, model or version, and serial number
•
Operating system name and version
•
Firmware version
•
Error messages
•
Product-specific reports and logs
•
Add-on products or components
•
Third-party products or components
Accessing updates
•
Some software products provide a mechanism for accessing software updates through the
product interface. Review your product documentation to identify the recommended software
update method.
•
To download product updates, go to either of the following:
◦
Hewlett Packard Enterprise Support Center Get connected with updates page:
www.hpe.com/support/e-updates
◦
Software Depot website:
www.hpe.com/support/softwaredepot
•
To view and update your entitlements, and to link your contracts and warranties with your
profile, go to the Hewlett Packard Enterprise Support Center More Information on Access
to Support Materials page:
www.hpe.com/support/AccessToSupportMaterials
IMPORTANT: Access to some updates might require product entitlement when accessed
through the Hewlett Packard Enterprise Support Center. You must have an HP Passport
set up with relevant entitlements.
Websites
46
Website
Link
Hewlett Packard Enterprise Information Library
www.hpe.com/info/enterprise/docs
Hewlett Packard Enterprise Support Center
www.hpe.com/support/hpesc
Support and other resources
Website
Link
Contact Hewlett Packard Enterprise Worldwide
www.hpe.com/assistance
Subscription Service/Support Alerts
www.hpe.com/support/e-updates
Software Depot
www.hpe.com/support/softwaredepot
Customer Self Repair
www.hpe.com/support/selfrepair
Insight Remote Support
www.hpe.com/info/insightremotesupport/docs
Serviceguard Solutions for HP-UX
www.hpe.com/info/hpux-serviceguard-docs
Single Point of Connectivity Knowledge (SPOCK) Storage www.hpe.com/storage/spock
compatibility matrix
Storage white papers and analyst reports
www.hpe.com/storage/whitepapers
Customer self repair
Hewlett Packard Enterprise customer self repair (CSR) programs allow you to repair your product.
If a CSR part needs to be replaced, it will be shipped directly to you so that you can install it at
your convenience. Some parts do not qualify for CSR. Your Hewlett Packard Enterprise authorized
service provider will determine whether a repair can be accomplished by CSR.
For more information about CSR, contact your local service provider or go to the CSR website:
www.hpe.com/support/selfrepair
Remote support
Remote support is available with supported devices as part of your warranty or contractual support
agreement. It provides intelligent event diagnosis, and automatic, secure submission of hardware
event notifications to Hewlett Packard Enterprise, which will initiate a fast and accurate resolution
based on your product’s service level. Hewlett Packard Enterprise strongly recommends that
you register your device for remote support.
For more information and device support details, go to the following website:
www.hpe.com/info/insightremotesupport/docs
Documentation feedback
Hewlett Packard Enterprise is committed to providing documentation that meets your needs. To
help us improve the documentation, send any errors, suggestions, or comments to Documentation
Feedback (docsfeedback@hpe.com). When submitting your feedback, include the document
title, part number, edition, and publication date located on the front cover of the document. For
online help content, include the product name, product version, help edition, and publication date
located on the legal notices page.
Customer self repair
47
A Sample package configuration file for Oracle database
modular package without ASM
# **********************************************************************
# ****** HIGH AVAILABILITY PACKAGE CONFIGURATION FILE (template) *******
# **********************************************************************
# ******* Note: This file MUST be edited before it can be used. ********
# * For complete details about package parameters and how to set them, *
# * consult the Serviceguard manual.*
# **********************************************************************
package_name
oracle_pkg
package_description
"Oracle Toolkit Package"
node_name
*
script_log_file
$SGCONF/oracle_pkg/pkg.log
# ************ LINUX ORACLE TOOLKIT PARAMETERS ************
tkit/oracle/oracle/INSTANCE_TYPE
database
tkit/oracle/oracle/ORACLE_HOME
/oracle/app/oracle/product/11.2.0/dbhome_1
tkit/oracle/oracle/ORACLE_ADMIN
oracle
tkit/oracle/oracle/SID_NAME
oranew
tkit/oracle/oracle/START_MODE
open
tkit/oracle/oracle/ASM
no
tkit/oracle/oracle/LISTENER
yes
tkit/oracle/oracle/LISTENER_NAME
LISTENER
tkit/oracle/oracle/LISTENER_PASS
lis123
tkit/oracle/oracle/LISTENER_RESTART
2
tkit/oracle/oracle/PFILE
${ORACLE_ORACLE_HOME}/dbs/init${ORACLE_SID_NAME}.ora
tkit/oracle/oracle/MONITOR_PROCESSES
ora_pmon_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES
ora_dbw0_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES
ora_ckpt_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES
ora_smon_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES
ora_lgwr_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES
ora_reco_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_INTERVAL
30
tkit/oracle/oracle/TIME_OUT
30
tkit/oracle/oracle/PARENT_ENVIRONMENT
no
tkit/oracle/oracle/CLEANUP_BEFORE_STARTUP
no
tkit/oracle/oracle/USER_SHUTDOWN_MODE
abort
tkit/oracle/oracle/KILL_ASM_FOREGROUNDS
yes
tkit/oracle/oracle/DB_SERVICE
all
tkit/oracle/oracle/DISABLE_BACKUP_MODE
no
email_id
adc@domain.com
service_name
oracle_service
service_cmd
"$SGCONF/scripts/tkit/oracle/tkit_module.sh oracle_monitor"
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
service_name
oracle_listener_service
service_cmd
"$SGCONF/scripts/tkit/oracle/tkit_module.sh oracle_monitor_listener"
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
service_name
oracle_hang_service
service_cmd
"$SGCONF/scripts/tkit/oracle/tkit_module.sh oracle_hang_monitor 30 failure"
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
ip_subnet_node
nodeA
ip_subnet_node
nodeB
Provide VG name if the database is configured in a VG.
vg
vgoracle
# FILESYSTEMS
fs_name
/dev/vg_ora/lvol1
#fs_server
fs_directory
/mntoracle
fs_type
"ext2"
fs_mount_opt
"-o rw"
#fs_umount_opt
#fs_fsck_opt
48
Sample package configuration file for Oracle database modular package without ASM
B Sample package configuration file for Oracle ASM
instance package and database modular package
ASM multi-node package configuration file
# **********************************************************************
# ****** HIGH AVAILABILITY PACKAGE CONFIGURATION FILE (template) *******
# **********************************************************************
# ******* Note: This file MUST be edited before it can be used. ********
# * For complete details about package parameters and how to set them,*
# * consult the Serviceguard manual.*
# **********************************************************************
package_name
oracle_asm_pkg
package_description
"Oracle Toolkit Package"
node_name
*
script_log_file
$SGCONF/oracle_asm_pkg/log
# ************ LINUX ORACLE TOOLKIT PARAMETERS ************
tkit/oracle/oracle/INSTANCE_TYPE
ASM
tkit/oracle/oracle/ORACLE_ADMIN
oracle
tkit/oracle/oracle/ASM
no
tkit/oracle/oracle/ASM_HOME
/grid/asm_home
tkit/oracle/oracle/ASM_USER
oracle
tkit/oracle/oracle/ASM_SID
+ASM
tkit/oracle/oracle/LISTENER
no
tkit/oracle/oracle/PFILE
${ORACLE_ASM_HOME}/dbs/init${ORACLE_ASM_SID}.ora
tkit/oracle/oracle/MONITOR_PROCESSES
asm_pmon_${ORACLE_ASM_SID}
tkit/oracle/oracle/MONITOR_PROCESSES
asm_dbw0_${ORACLE_ASM_SID}
tkit/oracle/oracle/MONITOR_PROCESSES
asm_ckpt_${ORACLE_ASM_SID}
tkit/oracle/oracle/MONITOR_PROCESSES
asm_smon_${ORACLE_ASM_SID}
tkit/oracle/oracle/MONITOR_PROCESSES
asm_lgwr_${ORACLE_ASM_SID}
tkit/oracle/oracle/MONITOR_PROCESSES
asm_gmon_${ORACLE_ASM_SID}
tkit/oracle/oracle/MONITOR_PROCESSES
asm_rbal_${ORACLE_ASM_SID}
tkit/oracle/oracle/MONITOR_INTERVAL
30
tkit/oracle/oracle/TIME_OUT
30
tkit/oracle/oracle/PARENT_ENVIRONMENT
no
tkit/oracle/oracle/CLEANUP_BEFORE_STARTUP
no
tkit/oracle/oracle/USER_SHUTDOWN_MODE
abort
tkit/oracle/oracle/KILL_ASM_FOREGROUNDS
yes
email_id
abc@hp.com
service_name
oracle_service
service_cmd
"$SGCONF/scripts/tkit/oracle/tkit_module.sh oracle_monitor"
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
Oracle ASM database package
# **********************************************************************
# ****** HIGH AVAILABILITY PACKAGE CONFIGURATION FILE (template) *******
# **********************************************************************
# ******* Note: This file MUST be edited before it can be used. ********
# * For complete details about package parameters and how to set them, *
# * consult the Serviceguard manual.*
# **********************************************************************
package_name
oracle_asmdb_pkg
package_description
"Oracle Toolkit Package"
node_name
*
script_log_file
$SGCONF/oracle_asmdb_pkg/pkg.log
# ************ LINUX ORACLE TOOLKIT PARAMETERS ************
tkit/oracle/oracle/INSTANCE_TYPE
database
tkit/oracle/oracle/ORACLE_HOME
/oracle/app/oracle/product/11.2.0/dbhome_1
tkit/oracle/oracle/ORACLE_ADMIN
oracle
tkit/oracle/oracle/SID_NAME
asmdb
tkit/oracle/oracle/START_MODE
open
tkit/oracle/oracle/ASM
yes
tkit/oracle/oracle/ASM_DISKGROUP
ORA_ASM
tkit/oracle/oracle/ASM_HOME
/grid/asm_home
tkit/oracle/oracle/ASM_USER
oracle
tkit/oracle/oracle/ASM_SID
+ASM
tkit/oracle/oracle/LISTENER
yes
tkit/oracle/oracle/LISTENER_NAME
LISTENER
tkit/oracle/oracle/LISTENER_PASS
lis123
tkit/oracle/oracle/LISTENER_RESTART
2
tkit/oracle/oracle/PFILE
${ORACLE_ORACLE_HOME}/dbs/init${ORACLE_SID_NAME}.ora
tkit/oracle/oracle/MONITOR_PROCESSES
ora_pmon_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES
ora_dbw0_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES
ora_ckpt_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES
ora_smon_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES
ora_lgwr_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES
ora_reco_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_INTERVAL
30
tkit/oracle/oracle/TIME_OUT
30
49
tkit/oracle/oracle/PARENT_ENVIRONMENT
no
tkit/oracle/oracle/CLEANUP_BEFORE_STARTUP
no
tkit/oracle/oracle/USER_SHUTDOWN_MODE
abort
tkit/oracle/oracle/KILL_ASM_FOREGROUNDS
yes
tkit/oracle/oracle/DB_SERVICE
all
tkit/oracle/oracle/DISABLE_BACKUP_MODE
no
email_id
abc@hp.com
service_name
oracle_service
service_cmd
"$SGCONF/scripts/tkit/oracle/tkit_module.sh oracle_monitor"
service_restart
none
service_fail_fast_enabled
no
service_halt_timeout
300
service_halt_on_maintenance
no
ip_subnet_node
ip_subnet_node
50
nodeA
nodeB
Sample package configuration file for Oracle ASM instance package and database modular package
C Sample package configuration file for Oracle multitenant
database with pluggable databases configured for
monitoring# **********************************************************************
# ****** HIGH AVAILABILITY PACKAGE CONFIGURATION FILE (template) *******
# **********************************************************************
# ******* Note: This file MUST be edited before it can be used. ********
# * For complete details about package parameters and how to set them, *
# * consult the Serviceguard manual.*
# **********************************************************************
package_name oracle_pkg
package_description "Oracle Toolkit Package"
node_name *
script_log_file $SGCONF/oracle_pkg/pkg.log
# ************ LINUX ORACLE TOOLKIT PARAMETERS ************
tkit/oracle/oracle/INSTANCE_TYPE database
tkit/oracle/oracle/ORACLE_HOME /oracle/app/oracle/product/11.2.0/dbhome_1
tkit/oracle/oracle/ORACLE_ADMIN oracle
tkit/oracle/oracle/SID_NAME oranew
tkit/oracle/oracle/START_MODE open
tkit/oracle/oracle/ASM no
tkit/oracle/oracle/LISTENER yes
tkit/oracle/oracle/LISTENER_NAME LISTENER
tkit/oracle/oracle/LISTENER_PASS lis123
tkit/oracle/oracle/LISTENER_RESTART 2
tkit/oracle/oracle/PFILE ${ORACLE_ORACLE_HOME}/dbs/init${ORACLE_SID_NAME}.ora
tkit/oracle/oracle/MONITOR_PROCESSES ora_pmon_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES ora_dbw0_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES ora_ckpt_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES ora_smon_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES ora_lgwr_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_PROCESSES ora_reco_${ORACLE_SID_NAME}
tkit/oracle/oracle/MONITOR_INTERVAL 30
tkit/oracle/oracle/TIME_OUT 30
tkit/oracle/oracle/PARENT_ENVIRONMENT no
tkit/oracle/oracle/CLEANUP_BEFORE_STARTUP no
tkit/oracle/oracle/USER_SHUTDOWN_MODE abort
tkit/oracle/oracle/KILL_ASM_FOREGROUNDS yes
tkit/oracle/oracle/DB_SERVICE all
tkit/oracle/oracle/DISABLE_BACKUP_MODE no
tkit/oracle/oracle/CRITICAL_PDB any
email_id abc@hp.com
service_name oracle_service
service_cmd "$SGCONF/scripts/tkit/oracle/tkit_module.sh oracle_monitor"
service_restart none
service_fail_fast_enabled no
service_halt_timeout 300
service_halt_on_maintenance no
service_name oracle_listener_service
service_cmd "$SGCONF/scripts/tkit/oracle/tkit_module.sh oracle_monitor_listener"
service_restart none
service_fail_fast_enabled no
service_halt_timeout 300
service_halt_on_maintenance no
service_name oracle_hang_service
service_cmd "$SGCONF/scripts/tkit/oracle/tkit_module.sh oracle_hang_monitor 30 failure"
service_restart none
service_fail_fast_enabled no
service_halt_timeout 300
service_halt_on_maintenance no
service_name oracle_pdb_moitor_service
service_cmd "$SGCONF/scripts/tkit/oracle/tkit_module.sh oracle_pdb_monitor"
service_restart none
51
Glossary
$SGCONF
/usr/local/cmcluster/conf for RHEL
/opt/cmcluster/conf for SLES
$SGROOT
/usr/local/cmcluster in RedHat
/opt/cmcluster in SLES
ADF
Attribute definition file
ASM
Automatic Storage Management
CDB
Container Database
Configuration
directory mode
of operation
It allows you to maintain your own directory and copy all the toolkits scripts in this directory.
Linux
Computer operating system
MNP
Multi-node package
Non ASM
Non Automatic Storage Management
Package
A grouping of application services (individual Linux Processes) under Serviceguard
PDB
Pluggable Database
52
Glossary
Index
support for multiple listeners, 41
support for Multitenant architecture, 40
support for user defined database services, 43
A
accessing
updates, 46
Alert mail notification, 37
C
Cluster verification of Oracle toolkit, 37
Configuring modular ASM database package, 19
Configuring modular Oracle database package without
ASM, 13
contacting Hewlett Packard Enterprise, 46
customer self repair, 47
D
documentation
providing feedback on, 47
U
updates
accessing, 46
Using Serviceguard toolkit for Oracle, 13
W
websites, 46
customer self repair, 47
With Automatic Storage Management, 12
Without Automatic Storage Management, 11
Multiple instance configuration, 11
Shared configuration, 11
H
Hardware Support, 5
I
Installation
uninstallation, 6
M
Managing a package, 30
adding the package to the cluster, 30
Managing packages
deleting a package, 35
halting packages, 35
maintaining a package, 31
running packages, 30
O
Oracle Toolkit
Advantages, 5
Overview, 5
Prerequisites, 5
Oracle toolkit
troubleshooting, 45
Oracle toolkit after Installation
files and directories, 7
Oracle toolkit features, 37
R
remote support, 47
S
support
Hewlett Packard Enterprise, 46
Support for multiple listeners
separate service to monitor each listener, 42
Single service to monitor all the listeners together, 41
Supported configuration, 11
Supporting Oracle features, 40
support for database hang detection, 43
53