Pre-Installation tasks of Installing Oracle 10g Real Application Cluster (RAC) (10.2.0.1) 32-bit on
Redhat AS 3 x86 (RHEL3) / CentOS 3 x86
By Lalit saini
Pre-Installation Task:
Task List:
Minimum Hardware required
Technical Architecture of 2-node RAC
Download Oracle 10g RDBMS softwares from OTN
Software Redhat Packages Required
Memory and Swap Space
Setting up Kernel Paramemeter
Configuring the Public, Private and Virtual Hosts / Network
Creating oracle User account.
Creating required Directories for Oracle 10g R2 RAC software and setting up correct
Permission
Setup shell Limits for the oracle user
Enable SSH oracle user Equivalency on all the cluster nodes.
Configuring System For Firewire Shared Disk
Partitioning Shared Disk
Installing and Configuring OCFS (Oracle Cluster File System)
Creating ASM disks using oracleasm (ASMLib IO) For the Clustered Database
Checking the Configuration of hangcheck-timer module
Required Hardware:
To create 2-node RAC, one would require 2 machine with the following hardware installed on it.
Per Node:
1 GB RAM, at least 8 GB of hard drive, 1 GHz CPU,
1 Firewire Controller, 1 Firewire Cable
2 NIC Ethernet card (one for public and another for private / interconnect network)
Per Cluster:
1 Shared Hard drive
1 Firewire HUB + 1 Firewire cable (for cluster with more than 2 node)
1 Network HUB + 1 network cable (for cluster with more than 2 node)
1 crossover network cable (for cluster with 2 node)
n number of network cable for private network for internode communication (for cluster with n
nodes where n >=3)
n number of network cable for public network (for cluster with n nodes where n >=3)
I used the below hardware to build my 2-node rac.
Server 1
Dell Intel PIII 1.3 GHz, 256 MB RAM, 20 GB HD
Server 2
Dell Intel PIII 1.3 GHz, 256 MB RAM, 20 GB HD
Upgrade Memory to 512MB
256 MB x 2 for Both the Server
Firewire Hard Drive
LaCie Firewire Hard Drive 120 GB
Firewire Controllers
Adaptec AFW-4300 x 2 (for both the server) - Texas Instrument chipset
Firewire HUB
Belkin's Firewire 6-Port Hub
Firewire Cables
1 extra Firewire cable for other node
NICs
D-Link Ethernet card x2
Network Hub
"NETWORK Everywhere"10/100 5-Port Hub
Crossover cable
-------
Total Cost: $913.00
Technical Architecture of 2 node RAC:
Clustered Database Name: RacDB
Node1:
SID: RacDB1
Public Network name (hostname): node1-pub, eth0
Private network Name (for Interconnect): node1-prv, eth1
ORACLE_BASE: /u01/app/oracle
DB file location: +ASM/{DB_NAME}/
CRS file Location: /u02/oradata/ocr mounted on /dev/sda1 (ocfs)
Node2:
SID: RacDB2
Public Network name (hostname): node2-pub, eth0
Private network Name (for Interconnect): node2-prv, eth1
ORACLE_BASE: /u01/app/oracle
DB file location: +ASM/{DB_NAME}/
CRS file Location: /u02/oradata/ocr mounted on /dev/sda1 (ocfs)
Obtaining Oracle 10g Software:
Goto otn.oracle.com and download the appropriate Oracle 10g Softwares into
the /tmp. Make Sure You have enough space under this mount point. You can
check this using df command. I
downloaded 10201_database_linux32.zip (668,734,007 bytes) (cksum 2737423041) file for my 32-bit Linux box. As I am going to create multi-
Instance Database (RAC), I also needed to download 10g
clusterware 10201_clusterware_linux32.zip (228,239,016 bytes) (cksum 2639036338). These files come with a .zip extension which needs to be
unzipped using the unzip utility which is installed as part of CentOS. In case you
do not have one; you can get it from here. After Unzipping this file, you can
optionally write them on the CD. I generally prefer cdrecord command.
I used a CD media of 700MB capacity to copy 10g (10.2.0.1) on it.
[root@localhost root]# unzip /tmp/0201_database_linux32.zip
[root@localhost root]# mkisofs -r /tmp/databases | cdrecord -v
dev=1,1,0 speed=20 –
If you are installing the software from disc, mount the first disc if it is not
already mounted. Some platforms automatically mount the disc when you
insert the disc into the drive.
Packages:
After you install the Linux system and before you start installing Oracle10g software, please make
sure that you have the below packages installed on your Linux box, else you will get error(s)
during the installation process.
make-3.79.1
gcc-3.2.3-34
glibc-2.3.2-95.20
compat-db-4.0.14-5
compat-gcc-7.3-2.96.128
compat-gcc-c++-7.3-2.96.128
compat-libstdc++-7.3-2.96.128
compat-libstdc++-devel-7.3-2.96.128
openmotif21-2.1.30-8
setarch-1.3-1
libaio-0.3.103-3
Please execute the below command as root to make sure that you have this rpms installed. If
not installed, then download them from appropriate Linux site.
rpm -q make gcc glibc compat-db compat-gcc compat-gcc-c++ compatlibstdc++ \
compat-libstdc++-devel openmotif21 setarch libaio libaiodevel
Perform this step on all the nodes.
Memory and swap Space:
Oracle 10g RAC requires to have 1GB of RAM available on each node to sucessfully install 10g
RAC. Well, I have somehow managed to install it with 512 MB RAM. You will get warning during
checking of pre-requisite step of installation step which you can ignore. Please go to Adding an
Extra Swapspace if you want to have an extra Swapspace added.
Kernel Parameters:
Please go to Setting Up kernel Parameter to set the kernel parameters.
Configuring Public and Private network for the Cluster Nodes:
Each node in the cluster must have 2 network adapter (eth0, eth1) one for the public and another for
the private network interface (internode communication, interconnect). You make sure that if you
configure eth1 as the private interface for node1 then, eth1 must be configured as private interface for
the node2.
Follow the below steps to configure these networks:
(1) Change the hostname value by executing the below command:
For Node node1-pub:
[root@localhost root]# hostname node1-pub
For Node node2-pub:
[root@localhost root]# hostname node2-pub
(2) Edit the /etc/hosts file as shown below
[root@localhost root]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1
localhost.localdomain
localhost
# Hostname for the Public Nodes in the RAC (eth0)
216.160.37.154
216.160.37.156
node1-pub.oracledba.org node1-pub
node2-pub.oracledba.org node2-pub
# Hostname for the Private Nodes in the RAC (eth1)
192.168.203.1
192.168.203.2
node1-prv.oracledba.org node1-prv
node2-prv.oracledba.org node2-prv
# Hostname for the Virtual IP in the RAC (eth1)
192.168.203.11 node1-vip.oracledba.org node1-vip
192.168.203.22 node2-vip.oracledba.org node2-vip
[root@node2-pub root]#
(3) Edit OR create the /etc/sysconfig/network-scripts/ifcfg-eth0 as shown below:
If you have static IPs: create the same file on both the nodes as shown below.
[root@localhost root]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
If you DO NOT have static IPs:
Add the entries like below into /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
IPADDR=192.168.10.1 -- For node1-pub Node
IPADDR=192.168.10.2 -- For node2-pub Node
(4) Edit OR create the /etc/sysconfig/network-scripts/ifcfg-eth1 as shown below:
For Node node1-pub:
[root@localhost root]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
PEERDNS=no
IPADDR=192.168.203.1
For Node node2-pub:
[root@localhost root]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
PEERDNS=no
IPADDR=192.168.203.2
(5) Edit the /etc/sysconfig/network file with the below contents:
For Node node1-pub:
[root@localhost root]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=node1-pub
For Node node2-pub:
[root@localhost root]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=node2-pub
(6) Restart the network service OR reboot the nodes:
After I rebooted both the nodes, I verified the network interface configurations by
running the ifconfig command as shown below.
[root@node2-pub root]# ifconfig
Creating oracle OS User Account:
You need OS “oracle” user account created which owns the Oracle software. Oracle
Software installation needs to be proceeds by this account. Oracle software
installation (without Companion CD) requires 6 GB of free space available for the
ORACLE_BASE directory. Please make sure that the mount point where you plan to
install Software has required free space available. You can use “df –k” to check this
out.
[root@node2-pub root]# df -k
Filesystem
1K-blocks
/dev/hda2
18113556
/dev/hda1
101089
none
126080
Used Available Use% Mounted on
3923072 13270364 23% /
14036
81834 15% /boot
0
126080
0% /dev/shm
I had about 13GB of free space available on “/” mount point. So I decided to install Oracle under this
mount point. RAC requires to have oracle user account created on all the nodes with the same
user id and group id. So, create oracle user account with this property by executing the below series of
command on all the RAC nodes.
groupadd -g 900 dba
groupadd -g 901 oinstall
useradd -u 900 -g oinstall -G dba oracle
passwd oracle
Please verify that oracle user has same gid and uid on all the RAC nodes by executing the this command
[oracle@node2-pub oracle]$ id
uid=900(oracle) gid=901(oinstall) groups=901(oinstall),900(dba)
[oracle@node1-pub oracle]$ id
uid=900(oracle) gid=901(oinstall) groups=901(oinstall),900(dba)
Creating Oracle Software Directories:
Perform the below steps on all the nodes in cluster.
[root@node2-pub
[root@node2-pub
OCR files
[root@node2-pub
[root@node2-pub
[root@node2-pub
[root@node2-pub
root]# mkdir -p /u01/app/oracle
root]# mkdir -p /u02/oradata/ocr
root]#
root]#
root]#
root]#
chown
chown
chmod
chmod
-R
-R
-R
-R
-- mountpoint for the
oracle:oinstall /u01
oracle:oinstall /u02
775 /u01/app/oracle
775 /u02
Setting Shell Limits for the Oracle User:
Please go to Setting up shell limits for the oracle user to set the shell limits for the oracle user.
Enable SSH oracle user Equivalency on Both the Cluster Nodes:
To configure SSH user equivalency, you must create RSA and DSA keys on each cluster node and copy
these keys from all the cluster node members into an authorized key file on each node. Follow the
below steps to achieve this task.
su - oracle
mkdir ~/.ssh
chmod 700 ~/.ssh
(A) Generate the RSA and DSA keys on all the RAC Nodes:
/usr/bin/ssh-keygen -t rsa
/usr/bin/ssh-keygen -t dsa
(B) Add keys to the authorized key file and then send the same file to every nodes in cluster:
touch ~/.ssh/authorized_keys
cd ~/.ssh
(1)
ssh
ssh
ssh
ssh
node1-pub
node1-pub
node2-pub
node2-pub
cat
cat
cat
cat
/home/oracle/.ssh/id_rsa.pub
/home/oracle/.ssh/id_dsa.pub
/home/oracle/.ssh/id_rsa.pub
/home/oracle/.ssh/id_dsa.pub
>>
>>
>>
>>
authorized_keys
authorized_keys
authorized_keys
authorized_keys
(2) Copy the authorized keys file to every nodes. For e.g, from node2-pub, I used the below command to
copy the node2-pub's authorized key file to node1-pub node.
[oracle@node2-pub .ssh]$ scp authorized_keys node1-pub:/home/oracle/.ssh/
(C) Repeat Step B on each node one by one.
(D) Change the Permission of authorized_keys file (on each node)
[oracle@node2-pub .ssh]$ chmod 600 ~/.ssh/authorized_keys
During executing step B - (1), you may be prompted as show below. Enter "yes" and continue.
[oracle@node2-pub .ssh]$ ssh node1-pub cat /home/oracle/.ssh/id_rsa.pub >>
authorized_keys
The authenticity of host 'node1-pub (216.160.37.154)' can't be established.
RSA key fingerprint is <**********>.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node1-pub,216.160.37.154' (RSA) to the list of
known hosts.
Warning: No xauth data; using fake authentication data for X11 forwarding.
Now, try executing the date (or any other command) on all the nodes to make sure that oracle is not
asked for the password. You should not receive any error message while you execute these commands
on all the nodes. If you get any error, first fix them before you go further.
ssh
ssh
ssh
ssh
node1-prv
node2-prv
node1-pub
node2-pub
date
date
date
date
Errors / Warnings during the network configurations:
I got the below warning when I tried below command.
[oracle@node2-pub .ssh]$ ssh node1-pub date
Warning: No xauth data; using fake authentication data for X11 forwarding.
Sun Dec 18 02:04:52 CST 2005
To fix the above warning, create the /home/oracle/.ssh/config file (logged in as oracle user) and make
the below entry in it. Then run the same command again and the above warning would not show up.
[oracle@node2-pub oracle]$ cat .ssh/config
Host *
Forwardx11 no
It is observed that when you execute the below command, you will be prompted to enter 'yes' or 'no'.
Simply enter yes and continue. Afterwards, when oracle connect to the remote node, it won’t be asked
for the password. this is explained as below where oracle received below message when it tried to run
the date command on the remote node using ssh very first time. Afterwards,
it does not get any message like this.
[oracle@node2-pub oracle]$ ssh node1-prv date
The authenticity of host 'node1-prv (192.168.203.1)' can't be established.
RSA key fingerprint is <********************************************>
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node1-prv,192.168.203.1' (RSA) to the list of
known hosts.
Sun Dec 18 20:01:09 CST 2005
[oracle@node2-pub oracle]$ ssh node1-prv date
Sun Dec 18 20:01:13 CST 2005
[oracle@node2-pub oracle]$
[oracle@node2-pub oracle]$ ssh node2-prv date
Warning: Permanently added the RSA host key for IP address '192.168.203.2' to
the list of known hosts.
Sun Dec 18 20:14:16 CST 2005
[oracle@node2-pub oracle]$ ssh node2-pub date
Sun Dec 18 20:15:05 CST 2005
If you get then below error message when try to connect to remote node, please make sure that the
firewall is disabled on the remote node.
[root@node2-pub root]# telnet node1-prv
Trying 192.168.203.1...
telnet: Unable to connect to remote host: No route to host
Configuring System for Shared Disk Storage Device (Firewire):
Every node in the cluster must have access to the shared disk. So the shared disk must support the
concurrent access to all nodes in cluster in order to successfully build 10g RAC. I chose Firewire
Disk as a shared storage media because it is a cost effective solution if you just want to have
hands-on practice on 10g RAC without investing more money. After you install the Redhat LINUX
AS 3 system into both the node, please go to the http://oss.oracle.com/projects/firewire/files
and download the appropriate Firewire kernel to support Firewire HD. I downloaded and installed
the below rpms.
[root@localhost root]# uname -r
2.4.21-37.EL
kernel-2.4.21-27.0.2.ELorafw1.i686.rpm
[root@localhost root]# rpm -ivh --force kernel-2.4.2127.0.2.ELorafw1.i686.rpm
This will also update the /etc/grub.conf file with the added entry of this new Firewire kernel. in the
below file, default is set to 1 which means that the system will use the original kernel by default. If
you want to make the newly added Firewire kernel as default, you can simply change the default=1
to default=0. It is required to set this kernel to default in the situation if this node is restarted by
hangcheck-timer or for any other reason, then it should be rebooted with the right kernel.
[root@node2-pub root]# cat /etc/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this
file
# NOTICE: You have a /boot partition. This means that
#
all kernel and initrd paths are relative to /boot/, eg.
#
root (hd0,0)
#
kernel /vmlinuz-version ro root=/dev/hda2
#
initrd /initrd-version.img
#boot=/dev/hda
default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
title CentOS (2.4.21-27.0.2.ELorafw1)
root (hd0,0)
kernel /vmlinuz-2.4.21-27.0.2.ELorafw1 ro root=LABEL=/
initrd /initrd-2.4.21-27.0.2.ELorafw1.img
title CentOS-3 (2.4.21-37.EL)
root (hd0,0)
kernel /vmlinuz-2.4.21-37.EL ro root=LABEL=/
initrd /initrd-2.4.21-37.EL.img
Also update the /etc/modules.conf file and add the below lines at the end of file on BOTH THE
NODES. This will load the Firewire kernel modules and drivers at reboot.
alias ieee1394-controller ohci1394
options sbp2 sbp2_exclusive_login=0
post-install sbp2 insmod sd_mod
post-install sbp2 insmod ohci1394
post-remove sbp2 rmmod sd_mod
Now, Shutdown both the nodes and then connect the Firewire shared disk to them. power on the
Firewire disk and then restart both the nodes using the new Firewire kernel 2.4.21-27.0.2.ELorafw1 one
by one. Confirm that the Firewire disk is visible from both the nodes by running the below command as
root on both the node.
[root@localhost root]# dmesg | grep ieee1394
ieee1394: Host added: Node[00:1023] GUID[0000d1008016f8e8] [Linux OHCI1394]
ieee1394: Device added: Node[01:1023] GUID[00d04b3b1905e049] [LaCie Group
SA ]
ieee1394: sbp2: Query logins to SBP-2 device successful
ieee1394: sbp2: Maximum concurrent logins supported: 4
ieee1394: sbp2: Number of active logins: 0
ieee1394: sbp2: Logged into SBP-2 device
ieee1394: sbp2: Node[01:1023]: Max speed [S400] - Max payload [2048]
ieee1394: Device added: Node[00:1023] GUID[00309500a0042ef9] [Linux OHCI1394]
ieee1394: Node 00:1023 changed to 01:1023
ieee1394: Node 01:1023 changed to 02:1023
ieee1394: sbp2: Reconnected to SBP-2 device
ieee1394: sbp2: Node[02:1023]: Max speed [S400] - Max payload [2048]
[root@localhost root]# dmesg | grep sda
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
SCSI device sda: 240121728 512-byte hdwr sectors (122942 MB)
sda:
Partitioning the Shared disk:
You need atleast two partitions to be created on if you want to go for ASM as an storage options. One or
more partition(s) will be used as ASM disk(s) and one partition is required to store Oracle's CRS (Cluster
Ready Service) files. You cannot create these files under ASM. They need to be either created under the
raw partition (device) or can be on ocfs. This document covers all the options of storing the database
files on shared disk. If you want to use the entire disk as ocfs or raw device (volume), then no need to
create separate partition for the CRS files. I partitioned the disks as shown below by connecting to any
one of the nodes.
As I am going to use ASM for the database files and so I will use ocfs for the CRS files.
[root@node2-pub root]# fdisk /dev/sda
The number of cylinders for this disk is set to 24792.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): n
Command action
e
extended
p
primary partition (1-4)
p
Partition number (1-4):
Value out of range.
Partition number (1-4): 1
First cylinder (1-24792, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +300M
Command (m for help): p
Disk /dev/sda: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot
/dev/sda1
Start
1
End
37
Blocks
297171
Id
83
System
Linux
Command (m for help): n
Command action
e
extended
p
primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (38-24792, default 38):
Using default value 38
Last cylinder or +size or +sizeM or +sizeK (38-24792, default 24792): +70000M
Command (m for help): n
Command action
e
extended
p
primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (8549-24792, default 8549):
Using default value 8549
Last cylinder or +size or +sizeM or +sizeK (8549-24792, default 24792): +70000M
Command (m for help): n
Command action
e
extended
p
primary partition (1-4)
p
Selected partition 4
First cylinder (17060-24792, default 17060):
Using default value 17060
Last cylinder or +size or +sizeM or +sizeK (17060-24792, default 24792):
Using default value 24792
Command (m for help): p
Disk /dev/sda: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot
Start
End
/dev/sda1
1
37
(ocfs)
/dev/sda2
38
8548
/dev/sda3
8549
17059
Clustered database on ocfs
/dev/sda4
17060
24792
Clustered database on raw device
Blocks
297171
Id
83
System
Linux
-- will be used by CRS files
68364607+
68364607+
83
83
Linux
Linux
-- will be used for ASM DSK1
-- reserved for another
62115322+
83
Linux
-- reserved for another
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
[root@node2-pub root]# partprobe -- [ Perform this step on all the nodes in cluster]
[root@node2-pub root]#
Installing and Configuring OCFS (Oracle Cluster File System):
We have 3 storage options to store the clustered database file on the shared disk.
(1) Traditional raw device option (9i).
(2) ocfs (also available in 9i).
(3) ASM (only in 10g and above).
I prefer to use the ASM over the ocfs file system. I will only use ocfs partition to store OCR file as this file
needs to be on the shared disk. You can also use the raw device option to store them.
Download and Install the required rpms:
Please download the below rpms from Oracle's website and install them as shown.
ocfs-2.4.21-EL-1.0.14-1.i686.rpm (For UniProcessor)
ocfs-2.4.21-EL-smp-1.0.14-1.i686.rpm (For SMPs)
ocfs-tools-1.0.10-1.i386.rpm
ocfs-support-1.1.5-1.i386.rpm
[root@node2-pub root]# rpm -Uvh /rpms/ocfs-2.4.21-EL-1.0.14-1.i686.rpm \
>
/rpms/ocfs-tools-1.0.10-1.i386.rpm \
>
/rpms/ocfs-support-1.1.5-1.i386.rpm
Preparing...
###########################################
[100%]
1:ocfs-support
########################################### [
33%]
2:ocfs-2.4.21-EL
########################################### [
67%]
Linking OCFS module into the module path [ OK ]
3:ocfs-tools
###########################################
[100%]
[root@node2-pub root]#
[root@node2-pub root]# cat /etc/ocfs.conf
#
# ocfs config
# Ensure this file exists in /etc
#
node_name = node2-prv
ip_address = 192.168.203.2
ip_port = 7000
comm_voting = 1
guid = 238426EC6845F952C83A00065BAEAE7F
Loading OCFS Module:
[root@node2-pub root]# load_ocfs
/sbin/modprobe ocfs node_name=node2-pub ip_address=192.168.203.2 cs=1843
guid=238426EC6845F952C83A00065BAEAE7F ip_port=7000 comm_voting=1
modprobe: Can't locate module ocfs
load_ocfs: insmod failed
If you get the above error follow the below steps to fix this:
Verify that you have ocfs.o module under /lib/modules/2.4.2127.0.2.ELorafw1/kernel/drivers/addon/ocfs/ directory.
[root@node2-pub root]# ls -l /lib/modules/2.4.2127.0.2.ELorafw1/kernel/drivers/addon/ocfs/ocfs.o lrwxrwxrwx
1
root
root
38 Dec 19 23:14 /lib/modules/2.4.2127.0.2.ELorafw1/kernel/drivers/addon/ocfs/ocfs.o -> /lib/modules/2.4.21-ELABI/ocfs/ocfs.o
IF THIS FILE EXISTS THEN:
open the /sbin/load_ocfs file using vi or another editor and change the below line as shown.
(Line Number 93)
# If you must hardcode an absolute module path for testing, do it HERE.
# MODULE=/path/to/test/module/ocfsX.o
Change to:
# If you must hardcode an absolute module path for testing, do it HERE.
MODULE=/lib/modules/2.4.21-27.0.2.ELorafw1/kernel/drivers/addon/ocfs/ocfs.o
IF THIS FILE DOES NOT EXISTS THEN:
Create a symbolic link as shown below.
mkdir /lib/modules/2.4.21-27.0.2.ELorafw1/kernel/drivers/addon/ocfs/
ln -s /lib/modules/2.4.21-EL-ABI/ocfs/ocfs.o /lib/modules/2.4.2127.0.2.ELorafw1/kernel/drivers/addon/ocfs/ocfs.o
Now try again to load the same module
[root@node2-pub root]# load_ocfs
If you get the error again then modify the /sbin/load_ocfs file as shown in the above step after creating
the symbolic link
[root@node2-pub root]# load_ocfs
/sbin/insmod /lib/modules/2.4.2127.0.2.ELorafw1/kernel/drivers/addon/ocfs/ocfs.o node_name=node2-prv
ip_address=192.168.203.2 cs=1843 guid=238426EC6845F952C83A00065BAEAE7F
ip_port=7000 comm_voting=1
Warning: kernel-module version mismatch
/lib/modules/2.4.21-27.0.2.ELorafw1/kernel/drivers/addon/ocfs/ocfs.o
was compiled for kernel version 2.4.21-27.EL
while this kernel is version 2.4.21-27.0.2.ELorafw1
Warning: loading /lib/modules/2.4.2127.0.2.ELorafw1/kernel/drivers/addon/ocfs/ocfs.o will taint the kernel:
forced load
See http://www.tux.org/lkml/#export-tainted for information about tainted
modules
Module ocfs loaded, with warnings
You may get the above warning but this may be OK. Verify that the ocfs module is loaded by executing
the below command.
[root@node2-pub root]# lsmod | grep ocfs
ocfs
299104
0 (unused)
Creating and Mounting OCFS (Oracle Cluster File System):
Create the file system using mkfs:
Execute the below series of command from any one node to format the /dev/sda1 partition with
the ocfs.
[root@node2-pub root]# mkfs.ocfs -F -b 128 -L /u02/oradata/ocr -m
/u02/oradata/ocr -u 900 -g 901 -p 0755 /dev/sda1
Where
b= blocksize
m= mountpoint
u= UID of oracle user
g=GID of oinstall group
p=permission
Mounting OCFS (Oracle Cluster File System): (Do this on both the node)
[root@node2-pub root]# mount -t ocfs /dev/sda1 /u02/oradata/ocr
Add the below line into the /etc/fstab file to mount the ocfs automatically on every reboot of
system.
/dev/sda1
/u02/oradata/ocr
ocfs
_netdev 0 0
[root@node2-pub root]# service ocfs start
Loading OCFS:
[root@node2-pub root]# chkconfig ocfs on
[
Create the file system using "ocfstool" command line utility:
Please follow the GUI screenshots of creating and mounting ocfs file system. run
the ocfstool from command as shown below:
Perform this step on Both the Nodes.
root@node1-pub root]# ocfstool
Check on the "Tasks" Button --> Select "Generate Config".
---> Select interface = eth1
port = 7000 and
Node Name = node1-prv (For node2, it would be node2-prv)
OK
]
Confirm changes by looking into the /etc/ocfs.conf file. The contents of this file
should be like this:
Now, Click on the "Tasks" button and select 'Format". You will see the screen like below. Select
the appropriate value and click OK button. You need to perform this step only from one node.
Now the /dev/sda1 is formatted with the ocfs. Now this is the time to mount this file system.
Please perform this step on both the nodes. Click on the "Mount" button. You should see the
/dev/sda1 is mounted under /u02/oradata/ocr mountpoint. Also confirm that you see both the
nodes in the "Configured Nodes" section.
Add the below line into the /etc/fstab file on both the nodes to mount the ocfs automatically on
every reboot.
/dev/sda1
/u02/oradata/ocr ocfs _netdev 0 0
Creating Automatic Storage Management (ASM) Disks for the Clustered Database
(both the RAC Nodes):
I will show you how to create the asm disks (stamping the disks as asm) on the FireWire device. I
am going to use one partition /dev/sda2 for ASM disks. I am going to use ASMLib IO for the ASM
disks and for that I need oracleasm kernel drivers as well as binaries and support downloaded
from Oracle's site. Please go to Creating and Configuring ASM instance and Database to get
detailed information on how to create ASM instance and diskgroups and how to use it into the
existing database OR new database. I downloaded the below rpms and installed them as root
user on both the nodes.
[root@node2-pub rpms]# rpm -Uvh
\
>
1.0.4-1.i686.rpm \
>
oracleasm-support-2.0.1-1.i386.rpm
oracleasm-2.4.21-27.0.2.ELorafw1oracleasmlib-2.0.1-1.i386.rpm
Preparing...
#######################################
#### [100%]
1:oracleasmsupport
########################################### [ 33%]
2:oracleasm-2.4.2127.0.2########################################### [ 67%]
3:oracleasmlib
#######################################
#### [100%]
[root@node2-pub rpms]#
Enter the following command to run oracleasm init script with configure option on both the
nodes.
[root@node2-pub root]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM
library
driver. The following questions will determine whether the driver
is
loaded on boot and what permissions it will have. The current
values
will be shown in brackets ('[]'). Hitting <ENTER> without typing
an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: oracle
Default group to own the driver interface []: oinstall
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration:
Creating /dev/oracleasm mount point:
Loading module "oracleasm":
Mounting ASMlib driver file system:
Scanning system for ASM disks:
[root@node2-pub root]#
[
[
[
[
[
OK
OK
OK
OK
OK
]
]
]
]
]
[root@node2-pub root]# /etc/init.d/oracleasm createdisk DSK1
/dev/sda2
Marking disk "/dev/sda2" as an ASM disk:
[
[root@node2-pub root]# /etc/init.d/oracleasm listdisks
DSK1
OK
]
ONLY on one NODE:
On the Remaining Nodes:
you only need to execute the below command to show these disks up there.
[root@node1-pub root]# /etc/init.d/oracleasm scandisks
[root@node1-pub root]# /etc/init.d/oracleasm listdisks
DSK1
Binding the partitions with the raw devices (Both the RAC Nodes):
Add the below lines into the /etc/sysconfig/rawdevices and restarted the rawdevices service
(On both the nodes).
[root@node2-pub root]# cat /etc/sysconfig/rawdevices
# raw device bindings
# format: <rawdev> <major> <minor>
# <rawdev> <blockdev>
# example: /dev/raw/raw1 /dev/sda1
# /dev/raw/raw2 8 5
/dev/raw/raw2 /dev/sda2
[root@shree ~]# service rawdevices restart
Also, you need to change the ownership of these devices to oracle user.
[root@node2-pub root]# chown oracle.dba /dev/raw/raw2
[root@node2-pub root]# chmod 660 /dev/raw/raw2
Please add the below lines to the /etc/rc.local so that these are set back at reboot.
for i in `seq 2 2`
do
chown oracle.dba /dev/raw/raw$i
chmod 660 /dev/raw/raw$i
done
Checking the Configuration of the hangcheck-timer Module:
Before Installing Oracle Real Application Cluster, We need to verify that the
hangcheck-timer module is loaded and configured correctly. The hangchecktimer module monitors the Linux kernel for extended operating system hangs
that could affect the reliability of a RAC node and cause a database corruption.
If a hang occurs than the module restarts the node in seconds. There
arehangcheck_tick and hangcheck_margin parameter that governs the
behavior of the modules:
The hangcheck_tick parameter defines how often, in seconds, the hangcheck-timer check the node
for hang. The default value is 60 seconds.
The hangcheck_margin parameter defines how long the hangcheck-timer waits, in seconds, for a
response from Kernel. The Default value is 180 seconds.
If the Kernel fails to respond within the total of (hangcheck_tick +
hangcheck_margin) seconds, the hangcheck-timer module restarts the system.
Verify that the hangcheck-timer module is running:
(1)
Enter the below command on each node.
[root@node2-pub root]# lsmod | grep hangcheck-timer
hangcheck-timer
2648
0 (unused)
(2) If the module is not listed by the above command, then enter the below command to load the
module on all the nodes.
[root@node2-pub root]# insmod hangcheck-timer
hangcheck_tick=30 hangcheck_margin=180
(3) Also add the below line at the end of /etc/rc.local file to ensure that this module is loaded at every
reboot.
insmod hangcheck-timer hangcheck_tick=30
hangcheck_margin=180
Alternatively, you could also add the same into the /etc/modules.conf file as shown below.
options hangcheck-timer hangcheck_tick=30
hangcheck_margin=180