.
Setting up IBM Power Systems 10
Gigabit Ethernet ports and AIX 6.1
EtherChannel for Oracle RAC private
interconnectivity
.
.
.
.
.
.
Ravisankar Shanmugam
IBM Oracle International Competency Center
August 2010
© Copyright IBM Corporation, 2010. All Rights Reserved.
All trademarks or registered trademarks mentioned herein are the property of their respective holders
.
Table of contents
Abstract........................................................................................................................................1
Prerequisites ...............................................................................................................................1
Introduction .................................................................................................................................1
10 Gigabit Ethernet in Power System servers..........................................................................1
AIX EtherChannel on 10 Gigabit Ethernet interfaces ..............................................................2
Setting up Oracle 11g Database RAC nodes with 10 Gigabit Ethernet ports and switches 2
Software version used for the test ................................................................................................................ 3
Set up and configuration of the 10 Gigabit Ethernet switch ...................................................3
Assigning HEA ports to the Oracle RAC nodes in AIX LPARs using the HMC.....................5
Creating the EtherChannel interface in AIX 6.1......................................................................11
Jumbo frames ............................................................................................................................................. 14
Summary....................................................................................................................................20
Appendix A ................................................................................................................................21
About the author .......................................................................................................................22
Trademarks and special notices..............................................................................................23
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
Abstract
The purpose of this white paper is to document the setup and configuration of IBM Power Systems™
10 Gigabit Ethernet ports. The environment used for the tests documented in this paper consist of
the Oracle Database 11g with Real Application Clusters (RAC) software to configure nodes with
private network intercommunication in the AIX® operating system environment. Tests were run on a
POWER6™ processor-based Power® 570 server. The 10 Gigabit Ethernet cards are an option for
the Power 570.
This document additionally covers the setup and configuration of AIX EtherChannel for Oracle RAC
interconnectivity and the Ethernet switches.
This document does not cover the installation of the Virtual IO server (VIOS), Logical Partition
(LPAR) creation, installation of the AIX operating system and various Oracle RAC components.
Refer the Appendix A of this paper for document links covering these specific topics.
Prerequisites
ƒ
ƒ
ƒ
Knowledge of AIX administration, Virtual IO Server and Power Systems POWER6 features.
Knowledge of configuring Force10 S2410 switch.
Knowledge of Oracle RAC concepts.
Introduction
Oracle RAC is a “shared everything” database architecture with two or more nodes clustered together
with shared storage. These nodes are connected together with a high speed interconnect that facilitates
fast communication between the nodes. The nodes use this interconnect at various times, such as
exchanging information during database startup, resolution of data lock situations, exchanging cluster
node information, exchanging transaction information and exchanging data between the Oracle RAC
nodes.
While the database transaction load spreads across multiple Oracle RAC nodes, the Oracle Cache
Fusion architecture can take advantage of a high speed network with high bandwidth and low latency to
move requests from one Oracle RAC node to another. When a high number of Oracle RAC nodes are
created and a high number of database blocks are exchanged between nodes, a high speed, high
bandwidth, low latency private network is beneficial. The 10 Gigabit Ethernet option gives the ideal
solution for high bandwidth and low latency needs in a clustered environment.
10 Gigabit Ethernet in Power System servers
The POWER6 processor-based Power 570 server is a mid range server with various advanced hardware
components and features including an optional dual port 10 Gigabit Ethernet card per building block for a
high speed network and high bandwidth.
For more information about the Power 570 server, refer the Redbook: IBM Power Systems p570
Technical Overview and Introduction.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
1
AIX EtherChannel on 10 Gigabit Ethernet interfaces
IBM POWER6 processor-based servers and above offer Integrated Virtual Ethernet adapters (IVE), which
give integrated high-speed Ethernet adapter ports (Host Ethernet Adapter (HEA)) with hardware assisted
virtualization capabilities.
The POWER6 570 servers used for the Oracle RAC setup in this testing use the 10 Gigabit dual port
Ethernet cards for Oracle cluster node interconnectivity.
The Host Ethernet Adapter can be used for external Ethernet connectivity for LPARs using dedicated
ports without the need of a Virtual IO Server. The HEA adapter directly connects to the internal GX+ bus
of POWER6 processor-based server instead of connecting to the PCIe or PCI-X bus.
IVE adapters also include special hardware features that provide the logical Ethernet adapter, otherwise
called Logical Host Ethernet Adapter (LHEA). These LHEA adapters can be directly assigned to the
LPARs without configuring through the POWER Hypervisor (PHYP). This eliminates the need to move the
packets between LPARs through the Shared Ethernet Adapter (SEA). IVE replaces the need of virtual
Ethernet and SEA in a Virtual IO Server (VIOS) environment and LPARs can share the HEA ports with
improved performance.
Each physical port in HEA can provide 16 logical ports. The setup mentioned in this document uses
logical ports from the HEA and assigns them to the LPARs. Refer to Figure 1 in this document for more
details. From the operating system standpoint in the LPAR, a logical port appears as a standard physical
Ethernet port with its attribute values.
The LHEA ports can be used for setting up an EtherChannel in AIX. EtherChannel is port link aggregation
for grouping one or more Ethernet links with backup ports to create a single logical port for the purpose of
providing high availability and load balancing to the connection between servers, switches and routers.
In this document, the EtherChannel port in the LPAR is used for the Oracle RAC private network
connection.
For more information on the IVE, HEA and LHEA, refer the Redbook: Integrated Virtual Ethernet Adapter
- Technical Overview and Introduction
Setting up Oracle 11g RAC nodes with 10 Gigabit
Ethernet ports and switches
Figure 1 shows the four node Oracle RAC cluster with 10 Gigabit Ethernet interfaces for interconnectivity.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
2
Figure 1 – Oracle RAC node interconnectivity diagram
The Oracle RAC nodes RAC_1 and RAC_2 share 2 x 10 Gigabit Ethernet cards. For high availability and
failover purposes, one card is always active and shared for EtherChannels and other card is in standby.
The Oracle RAC nodes RAC_3 and RAC_4 are setup with 2 x 10 Gigabit Ethernet cards. Only one port is
used from each card and the other port is left unused for testing purposes. There are 2 logical ports
(LHEA) used from each physical port (HEA). The 10 Gigabit Ethernet switches are connected to each
other. All of the 10 Gigabit Ethernet connections from the AIX server to the switches and between the
switches are made using Fibre Channel cables (LC-LC type).
Software version used for the test
ƒ
AIX 6.1 TL4 SP03
ƒ
Oracle 11g Database with RAC version 11.1.0.7
Set up and configuration of the 10 Gigabit Ethernet
switch
For initial basic setup of the switch, refer to the following documents: Installing the S2410 system and
S2410 Quick reference from the Force10 CD or the Force10networks web site.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
3
After finishing the basic setup of the switch, such as network setup to access the switch, activate the
‘telnet’ and the ‘web interface’ to access it remotely.
To activate the telnet and Web access, use the following commands:
Force10 S2410
Password:
Force10 S2410
Force10 S2410
Force10 S2410
# enable
# config
# ip telnet server enable
# ip http server enable
In a brand new Force10 switch, all of the ports are in a disabled mode by default. The ports should be
enabled by using either the command line or Web GUI interface.
The following command enables all of the ports in a switch. Enable the ports on the backup switch.
Force10 S2410 # no shutdown all
Force10 S2410 #show port all
Admin
Physical
Physical
Link
Link
LACP
Interface
Type
Mode
Mode
Status
Status Trap
Mode
---------- ------ ---------------- -------------------------0/1
Enable 10G Full
Down
Enable Enable
0/2
Enable 10G Full
Down
Enable Enable
0/3
Enable 10G Full
Down
Enable Enable
0/4
Enable 10G Full
Down
Enable Enable
0/5
Enable 10G Full
Down
Enable Enable
0/6
Enable 10G Full
Down
Enable Enable
0/7
Enable 10G Full
Down
Enable Enable
<Remaining output is eliminated>
The private network setup for the Oracle11g Database with RAC mentioned in Figure 1 of this document
is created as follows. The following output shows the switch ports status on the switch.
Force10 S2410 #show port all
Interface
-------------0/1
0/2
0/3
0/4
0/5
0/6
0/7
0/8
0/9
0/10
Admin
Physical
Type
Mode
Mode
Status
------ ----------------
Physical
Link
Link
LACP
Status Trap
Mode
---------------------
Enable 10G Full
10G
Enable 10G Full
10G
Enable 10G Full
10G
Enable 10G Full
10G
Enable 10G Full
Enable 10G Full
Enable 10G Full
Enable 10G Full
Enable 10G Full
Enable 10G Full
<Remaining output is eliminated>
Full
Full
Full
Full
Up
Up
Up
Up
--
Enable Enable
Enable Enable
Enable Enable
Enable Enable
Down
Enable Enable
Down
Enable Enable
Down
Enable Enable
Down
Enable Enable
Down
Enable Enable
Down
Enable Enable
Set the “Flow Control” to enabled in the switch. After logging into the switch,
Force10 S2410 #enable
Password:
Force10 S2410 (Config)#storm-control flowcontrol
Force10 S2410 (Config)#exit
Force10 S2410 # write
This operation may take a few minutes.
Management interfaces will not be available during this time.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
4
Are you sure you want to save? (y/n) y
Configuration Saved!
Assigning HEA ports to the Oracle RAC nodes in AIX
LPARs using the HMC
The Host Ethernet Adapter (HEA) ports of the Power 570 server are shown in the Hardware Management
Console (HMC)
The following screen shot shows the list of 10 Gigabit ports in the server. Each port can be configured for
different purposes. For example, a port can be assigned to an LPAR as dedicated (promiscuous) or
shared.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
5
Select a port (port1) in the list and press “configure” button.
If the LPAR is selected in the “Promiscuous LPAR” list, the port is used only for that LPAR. By default, the
ports are assigned to the system LPAR as dedicated (Promiscuous LPAR). In this server, by default the
system LPAR “10-68F72” was Promiscuous LPAR for all of the 10 Gigabit Ethernet ports.
For the Oracle Database with RAC setup, the ports were released from the Promiscuous LPAR and
selected to “None”, so that the ports can be shared with other LPARs.
Select “Flow Control Enabled” for each port.
Repeat the same assignment “None” to all of the HEA ports as needed. Setting up Jumbo Fames in this
configuration can be seen in a later part of this paper.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
6
To assign the logical port for the LPARs, select the LPAR and go to Task ÆConfigurationÆManage
Profiles.
Select the Profile and “Actions”ÆEdit
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
7
The following window shows the list of Host Ethernet Adapters (Physical).
There are 2 x 2 ports Physical adapters are in the Power 570 machine. Select the appropriate port which
is physically connected to the 10 Gigabit Ethernet switch.
For this setup, the port “DQD64A9-P1-C10-T1” and “DQD64A6-P1-C10-T1” are used for the LPAR
“AIX_RAC1_209”.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
8
Select “DQD64A9-P1-C10-T1” port and press “Configure” button
The following window shows the LHEA ports. Select the logical port1 and press OK and this port will be
assigned to the LPAR “AIX_RAC1_209”.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
9
Select the second port (backup port) for the LPAR “AIX_RAC1_209” and click “Configure”.
Select the logical port from the list.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
10
Repeat the above procedures for other LPARs in the RAC configuration.
Shutdown (“shutdown –Fr”) the LPARs and bring them back online.
After the LPARs come online, check the list of Logical Host Ethernet Adapter (LHEA) ports (lhea0 and
hea1) assigned to the LPAR as shown below.
The interfaces “ent1” and ‘ent2” are the corresponding interfaces for “lhea0” and “lhea1”.
# lsdev -Cc adapter
ent0
Available
ent1
Available
ent2
Available
fcs0
Available 00-08
fcs1
Available 00-09
lhea0 Available
lhea1 Available
vsa0
Available
vscsi0 Available
vscsi1 Available
Virtual I/O Ethernet Adapter (l-lan)
Logical Host Ethernet Port (lp-hea)
Logical Host Ethernet Port (lp-hea)
FC Adapter
FC Adapter
Logical Host Ethernet Adapter (l-hea)
Logical Host Ethernet Adapter (l-hea)
LPAR Virtual Serial Adapter
Virtual SCSI Client Adapter
Virtual SCSI Client Adapter
For verifying the mapping of the adapter name “ent1” displayed in AIX for the assigned port, execute the
command “lscfg –vl ent1” in the AIX LPAR.
# lscfg -vl ent1
ent1
U789D.001.DQD64A9-P1-C10-T1 Logical Host Ethernet Port (lp-hea)
IBM Host Ethernet Adapter:
Network Address.............001A64283FB0
In the above output of the interface ent1, “U789D.001.DQD64A9-P1-C10-T1” is the physical adapter
name and location in the Power 570 server.
Creating the EtherChannel interface in AIX 6.1
EhterChannel is a network port aggregation technique that aggregates more than one Ethernet adapter
together to form a single logical Ethernet interface. Then the logical interface is configured with an IP
address for communications. The Ethernet adapters which are part of the EtherChannel should be
connected to the same switch.
EtherChannel can be used for load balancing and high availability of the network connections. This
provides fault-tolerant high speed communications between servers and switches.
The participating Ethernet interfaces for the EtherChannel should not be configured with IP addresses
and should be in “detach” mode before adding to them to the EtherChannel.
Use this command to detach the participating Ethernet interfaces.
chdev –l en3 –a state=detach
## where en3 is the Etherchannel port.
In AIX, EtherChannel provides a backup port option to connect an Ethernet interface to another switch if
the primary switch or the primary EtherChannel ports are down or faulty.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
11
The setup in this document used one primary port in the EtherChannel and a backup port for high
availability. The backup port is in standby mode while the primary port is online.
Refer the Figure 1 for the detailed network connection of the Oracle RAC nodes EtherChannel setup.
The EtherChannel port mentioned in this document is created with one primary Ethernet port (ent1) and a
backup port (ent2). The EtherChannel port (ent3) is created using ent1 and ent2 interfaces. Do not
configure IP or set any attributes for the participating candidate Ethernet ports (ent1 and ent2). If you
have them configured previously, reset the IP address and reset the previously assigned attributes.
The following procedure explains the creation of the EtherChannel interface using the “smitty” tool: #
smitty etherchannel.
Select “Add an EtherChannel / Link Aggregation” in the list.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
12
Select the primary Ethernet ports for the EtherChannel. Do not select the backup port now.
After selecting the primary Ethernet adapters, the following page is displayed. Select the Backup port
from the list now. For “Internet Address to Ping”, enter the pingable private IP address, for example the
other private IP address of other Oracle RAC nodes. Enter “3” for “Number of Retries” and “1” for the
“Retry Timeout”.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
13
Press “Enter” after filling in the screen above. It creates a new interface “ent3” for EtherChannel.
# lsdev -Cc adapter
ent0
Available
ent1
Available
ent2
Available
ent3
Available
fcs0
Available 00-08
fcs1
Available 00-09
lhea0 Available
lhea1 Available
Virtual I/O Ethernet Adapter (l-lan)
Logical Host Ethernet Port (lp-hea)
Logical Host Ethernet Port (lp-hea)
EtherChannel / IEEE 802.3ad Link Aggregation
FC Adapter
FC Adapter
Logical Host Ethernet Adapter (l-hea)
Logical Host Ethernet Adapter (l-hea)
The newly created EtherChannel interface ent3 can be configured with the IP address using: smitty
mktcpipÆen3Æ<Setup the IP address>.
Repeat the above procedures to the remaining Oracle RAC nodes.
For more information on the AIX EtherChannel, refer the AIX 6.1 document: Etherchannel and IEEE
802.3ad Link Aggregation
Make sure the EtherChannel interface is properly setup for the Oracle RAC nodes and that they can
communicate with each other by using “ping” command. Also, a small test would confirm the basic
capacity of the network bandwidth between the Oracle RAC nodes.
The Oracle RAC node1 is configured with the private network IP address 10.0.0.1 on the EtherChannel
interface “en3” and the Oracle RAC node2 is configured with 10.0.0.2 as same.
The test can be done by using “ftp” command as follows: from Oracle RAC node1, as a “root” user do the
“ftp” and monitor the network traffic using the “nmon” tool.
The private IP address 10.0.0.,2 belongs to another Oracle RAC node’s 10 Gigabit Ethernet port
# ftp 10.0.0.2
Connected to 10.0.0.2.
220 dyn9038158218 FTP server (Version 4.2 Wed Oct 1 09:40:05 CDT 2008) ready.
Name (10.0.0.2:root):
331 Password required for root.
Password:
230-Last login: Wed Apr 8 19:35:59 2009 on /dev/pts/1 from sig-9-65-45188.mts.ibm.com
230 User root logged in.
ftp> put "|dd if=/dev/zero bs=1M count=200000" /dev/null
200 PORT command successful.
150 Opening data connection for /dev/null.
The output of network interfaces can be seen by using ‘nmon” tool.
Jumbo frames
Setting up jumbo frames on the private network interface for the Oracle RAC nodes would give an
advantage for sending and receiving large size data frames between the nodes. This could better utilize
the available bandwidth of the 10 Gigabit Ethernet connections between the Oracle RAC nodes.
Enabling jumbo frames on the network interfaces or EtherChannel interface requires certain steps to be
successful in an Oracle RAC environment.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
14
The general rule is to enable the jumbo frames option on both the network interface in the host side as
well as the port in the network switch and the port(s) of the switch interconnection.
Before enabling Jumbo Frames, bring down the CRS and DB resources and “detach” the EtherChannel
interfaces in the Oracle RAC nodes.
As a root user, issue the following commands.
#$CRS_HOME/bin/crsctl stop crs
#ifconfig en3 down detach
In the network setup mentioned in this document for HEA ports the jumbo frames option was set to “Yes”
on the EtherChannel interface “ent3”.
Use the “smitty” tool, #smitty etherchannel then select “Change / Show Characteristics of an
EtherChannel / Link Aggregation” for “ent3” to verify that the jumbo frames setting is set to “yes”.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
15
To enable jumbo frames on the physical ports in the Power 570 server, use HMC menus as follows. In
HMC, select “Servers” in left side of the page, select the server listed on the right side where the Oracle
RAC node is created, click on (>>) pop up menu of the server, navigate to Hardware Information
>Adapters>Host Ethernet. This will bring up a pop-up window.
Select the HEA port used for the Oracle RAC interconnectivity as well as the EtherChannel. Click the
“Configure” button and the following window will be popped up, select “9000 jumbo frame” option in the
list. Select “Flow control enabled” to keep the flow control enabled and select “OK”. Repeat the same
procedure on all of the ports which are involved in the Oracle RAC interconnectivity.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
16
After setting up the jumbo frames option on the host and the HEA ports as mentioned above, enable
jumbo frames on the network switch ports.
In the Force10 S2410 XFP switch, enabling jumbo frames on the ports can be done either with the
Command line or Web GUI.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
17
Using web GUI, login to the switch and enable the Jumbo frame as follows:
Enable jumbo frames by editing the field “maximum frame size” to “9022” and click the “Submit” button.
Follow the same procedure for all of the ports involved in the Oracle RAC interconnect as well as the
switch interconnect ports and “save” the changes made.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
18
After enabling jumbo frames on the EtherChannel port, HEA ports and switch ports as mentioned above
check to be sure the EtherChannel port on the Oracle RAC nodes accept jumbo frames.
Bring up the EtherChannel interface on the Oracle RAC nodes.
#ifconfig en3 up
#chdev –l en3 –a state=up
Execute ‘ping” command with big size blocks from one Oracle RAC node to another and vice versa.
# ping –s8972 10.0.0.2
PING 10.0.0.2 (10.0.0.2):
8980 bytes from 10.0.0.2:
8980 bytes from 10.0.0.2:
8980 bytes from 10.0.0.2:
8980 bytes from 10.0.0.2:
8980 bytes from 10.0.0.2:
8980 bytes from 10.0.0.2:
8972 data bytes
icmp_seq=0 ttl=255
icmp_seq=1 ttl=255
icmp_seq=2 ttl=255
icmp_seq=3 ttl=255
icmp_seq=4 ttl=255
icmp_seq=5 ttl=255
time=0
time=0
time=0
time=0
time=0
time=0
ms
ms
ms
ms
ms
ms
Execute the “entstat” command to make sure the EtherChannel candidate ports accept jumbo frames.
# entstat –d ent3 |grep –i jumbo
RX
RX
RX
RX
RX
TX
RX
RX
RX
RX
RX
TX
Q3 Jumbo Packets: 6 / 1
Q3 Jumbo Packets: 6
Q3 Jumbo Packets: 0
Q3 Jumbo Packets: 0
Q3 Jumbo Packets: 0
and RX Jumbo Frames: On
Jumbo Frame Action: Accept
Q3 Jumbo Packets: 0 / 0
Q3 Jumbo Packets: 0
Q3 Jumbo Packets: 0
Q3 Jumbo Packets: 0
Q3 Jumbo Packets: 0
and RX Jumbo Frames: On
Jumbo Frame Action: Accept
Some issues might be seen while installing Oracle11g CRS version 11.1.0.6 on AIX 6.1.4.3 with root.sh.
On the last node of the cluster, root.sh was timed out with the message: Waiting for the Oracle CRSD and
EVMD to star”. This didn’t bring up the “ons” and “gsd” resources, thus “root.sh” failed. To avoid this
failure, install the CRS without the jumbo frames set to ON in the Oracle RAC interconnect interfaces and
enable them after the install and then apply patches or patch set for CRS.
Another issue while executing root.sh on an Oracle RAC node with AIX 6.1.4.3 with or without jumbo
frames is that the VIP resource won’t come up ONLINE on the Oracle RAC nodes. To fix this issue,
$CRS_HOME/bin/racgvip should be edited as shown below.
The debug trace should be set to capture more details of the VIP failure.
As a root user,
#$CRS_HOME/bin/crsctl
debug log res “ora.dyn9038158218.vip:1”
Part of $CRS_HOME/log/dyn9038158209/racg/ora.dyn9038158218.vip.log looks like:
2010-06-21 12:57:23.664: [
RACG][1] [299130][1][ora.dyn9038158218.vip]: Tue Jun
21 12:57:22 CDT 2010 [ 483342 ] About to execute command: /usr/sbin/ping -S
9.38.158.218 -c 1 -w 3 9.38.158.129
Tue Jun 21 12:57:23 CDT 2010 [ 483342 ] IsIfAlive: RX packets checked if=en0 failed
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
19
2010-06-21 12:57:23.664: [
RACG][1] [299130][1][ora.dyn9038158218.vip]: Tue Jun
21 12:57:23 CDT 2010 [ 483342 ] Interface en0 checked failed (host=dyn9038158218)
Tue Jun 21 12:57:23 CDT 2010 [ 483342 ] IsIfAlive: end for if=en0
Tue Jun 21 12:57:23 CDT 2010 [ 483342 ] checkIf: end for if=en0
2010-06-21 12:57:23.664: [
RACG][1] [299130][1][ora.dyn9038158218.vip]: Invalid
parameters, or failed to bring up VIP (host=dyn9038158218)
To fix this error, edit the $CRS_HOME/bin/racgvip script as below, look for lines starting with _O1 and
_O2 and replace $5 instead of $6.
$CRS_HOME/bin/racgvip:
# get RX packets numbers
_O1=`$NETSTAT -n -I $_IF | $AWK "{ if (/^$_IF/) {print \\$5; exit}}"`
#### the $5 should be replaced with $6 ###
x=$CHECK_TIMES
while [ $x -gt 0 ]
..............
fi
_O2=`$NETSTAT -n -I $_IF | $AWK "{ if (/^$_IF/) {print \\$5; exit}}"`
the $5 should be replaced with $6 ###
if [ "$_O1" != "$_O2" ]
then
# RX packets numbers changed
####
After editing $CRS_HOME/crs/racgvip script,
#$CRS_HOME/bin/crs_stop –all
#$CRS_HOME/bin/crs_start –all
This will bring up the VIP resources on the nodes.
Summary
This document provides the detailed AIX EtherChannel setup procedure when used for a private network
of Oracle RAC nodes. The Oracle RAC nodes’ private network interconnectivity can take the advantage
of 10 Gigabit Ethernet cards as part of the Integrated Virtual Ethernet (IVE) capability available on IBM
POWER6 (or higher) processor-based Power System servers along with the AIX operating system
EtherChannel feature for high availability, load balancing, higher bandwidth and lower latency.
The physical 10 Gigabit Ethernet ports can be shared by the LPARs using the Logical Host Ethernet
Adapter feature, also available in POWER6 (or higher) class Power Systems servers.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
20
Appendix A
The following links refer the documents for installing Virtual I/O Server, AIX and Oracle RAC products.
ƒ
PowerVM Virtualization on IBM System p: Introduction and configuration
http://www.redbooks.ibm.com/redbooks/pdfs/sg247940.pdf
ƒ
AIX 6.1 Installation and Migration
http://publib.boulder.ibm.com/infocenter/systems/index.jsp?topic=/com.ibm.aix.install/doc/ins
gdrf/bos_install_intro.htm
ƒ
Integrated Virtual Ethernet Adapter –Technical Overview and Introduction
http://www.redbooks.ibm.com/redpapers/pdfs/redp4340.pdf
ƒ
Oracle Real Application Clusters Installation Guide - 11g Release 1 (11.1) for Linux and
UNIX
http://download.oracle.com/docs/cd/B28359_01/install.111/b28264/toc.htm
ƒ
Force10 Network S2410 – SFTOS Configuration guide
http://www.force10networks.com/CSPortal20/KnowledgeBase/DOCUMENTATION/CLIConfig
/SFTOS/SFTOS_Config_2.5.3.0_27-Aug-08.pdf
ƒ
Force10 Networks S2410 10 Gigabit switch Command Line reference document
http://www.force10networks.com/CSPortal20/KnowledgeBase/DOCUMENTATION/CLIConfig
/SFTOS/SFTOS_CLI_2.5.3.0_27-Aug-08.pdf
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
21
About the author
Ravisankar Shanmugam is a Senior IT Specialist with IBM Advanced Technical Skills Solutions team
and works at the IBM Oracle International Competency Center based in San Mateo, CA. He provides
Power Systems and System x® platform support for projects at the Competency Center and for
enablement activities at Oracle Corporation in Redwood Shores and Pleasanton, CA.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
22
Trademarks and special notices
© Copyright. IBM Corporation 1994-2010. All rights reserved.
References in this document to IBM products or services do not imply that IBM intends to make them
available in every country.
IBM. the IBM logo, ibm.com, AIX, POWER, System Storage, Power Systems and System x are
trademarks or registered trademarks of International Business Machines Corporation in the United States,
other countries, or both:
Other company, product, or service names may be trademarks or service marks of others.
The information provided in this document is distributed “AS IS” without any warranty, either express or
implied.
The information in this document may include technical inaccuracies or typographical errors.
Information concerning non-IBM products was obtained from a supplier of these products, published
announcement material, or other publicly available sources and does not constitute an endorsement of
such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly
available information, including vendor announcements and vendor worldwide homepages. IBM has not
tested these products and cannot confirm the accuracy of performance, capability, or any other claims
related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the
supplier of those products.
All statements regarding IBM future direction and intent are subject to change or withdrawal without
notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller
for the full text of the specific Statement of Direction.
Some information addresses anticipated future capabilities. Such information is not intended as a
definitive statement of a commitment to specific levels of performance, function or delivery schedules with
respect to any future products. Such commitments are only made in IBM product announcements. The
information is presented here to communicate IBM's current investment and development activities as a
good faith effort to help with our customers' future planning.
Performance is based on measurements and projections using standard IBM benchmarks in a controlled
environment. The actual throughput or performance that any user will experience will vary depending
upon considerations such as the amount of multiprogramming in the user's job stream, the I/O
configuration, the storage configuration, and the workload processed. Therefore, no assurance can be
given that an individual user will achieve throughput or performance improvements equivalent to the
ratios stated here.
Photographs shown are of engineering prototypes. Changes may be incorporated in production models.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in
any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part
of the materials for this IBM product and use of those Web sites is at your own risk.
Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity
http://www.ibm.com/support/techdocs
© Copyright 2010, IBM Corporation
23