Oracle RAC 11gR2
RAC Joins the Cloud
RAC Grows Up




RAC 11gR2 (aka 12gR1) adds a number of features that
transforms RAC clusters into database clouds
Many features are geared towards enterprises but there
are new features that any organization can take advantage
of
With today’s high power x86 and 11gR2 RAC it is entirely
possible to replace many silo clusters with fewer 11gR2
“clouds”
Many 11gR1 and older RAC commands are depreciated
11gR2 RAC New Features




Brand New Installer
Clusterware is now Grid Infrastructure
ASM is now part of the Grid Infrastructure installation
Voting and OCR disk are ASM diskgroup

Raw devices are only supported for upgraded clusters


Plug and Play For Clusters



Future Linux release may drop support for raw devices
Cluster is now a DNS service
DHCP for server VIPs
New tools are introduced old tools retired
RAC 11gR2 Install
Non Linear Installer
Grid Plug and Play
SSH Keys
OCR/Voting Disks
OCR/Voting Disks

OCR/Voting disks can be ASM Diskgroup



Raw devices are only supported for upgraded clusters
Other options are NFS and certified clustered filesystems
3 disks for normal redundancy, 5 disks for high
IPMI
IPMI (cont)




Intelligent Platform Management Interface
Defines a cross vendor standard to monitor and manage
hardware
Dell, HP, IBM, Intel, AMD and dozens more
IPMI in relation to RAC allows other nodes to manage
the server regardless of the OS state
ASM Roles
Fix Common Problems


Installer can create a script that can fix some problems
Won’t install missing packages
Finish Install

After GUI finished is will ask to run the command
oraInrootinst.sh and root.sh
Cluster Startup Sequence


Init starts Oracle High Availability Service Deamon
(OHASD)
OHASD starts





cssdagent – starts cssd
orarootagent – manages all root owned ohasd processes
oraagent – manages all oracle owned ohsad resources
cssdmonintor – monitors cssd and node health
OHASD rootagent starts




crsd – main deamon for managing cluster resources
ctssd – time synchronization
Diskmon
ACFS – ASM cluster file system drivers
Cluster Startup Sequence

Oraagent starts – level 2






CRSD – level 3



mdnsd – used for DNS lookups
gipcd – inter-process and inter-node communication
gpnpd – Grid Plug & Play deamon
EVMD – event monitor
ASM – resource for monitoring ASM instance
orarootagent
oraagent
CRSD rootagent – level 4




SCAN VIP(s) - Single Client Access Name
Node VIPs – one per node
ACFS Registry – For mounting ASM Cluster File System
GNS VIP - optional
SCAN

SIMPLE CLIENT ACCESS NAME







A single DNS entry that represents the cluster has a whole
Clients no longer need to know the actual servers that are in a
cluster
THIN connections or connections not using a VIP are better
protected against a node failure
If using GNS SCAN must be in the domain managed by GNS
DNS needs to be able to delegate resolution to GNS
If not using GNS DNS should be configured to round robin up
to 3 IPs for the SCAN name. Recommend use 11.2 client
If using GNS 3 IPs will be requested from DHCP
SCAN Listener





Listener that is dependent on a SCAN VIP
If the SCAN VIP moves to a different node in the cluster the
listener will move with the VIP
Depending on the number of nodes in a cluster a single node
may have more than one SCAN VIP and listener
remote_listener parameter should be set to a SCAN VIP for
11gR2 databases
Simplify RAC Dataguard connections for clients

Two new sqlnet parameters



CONNECT_TIME = timeout in seconds
RETRY_COUNT = number of tries
In a single TNS entry enter both primary and standby servers SCAN
address. If the connection is made to standby it will failover to primary.
SCAN Load Balancing




Client receives IP from DNS or GNS that resolves to
SCAN VIP
SCAN Listener bound to SCAN VIP redirects client to
local listener that has the least load
Client connects to the local listener on the node VIP
Client needs to be able to connect to SCAN VIP and to
the local node VIP and be able to resolve all names
SCAN

SCAN Listeners Overview
Grid - Plug And Play


Greatly simplifies the adding/removing and connecting to
a cluster
Basically run cluster verify script and then addNode.sh


Configures cluster listener and ASM
No longer use static IPs for VIPs
GNS

Grid Naming Service




Handles name resolution for the cluster
Requires DHCP for VIP addresses!
DNS needs to delegate resolution for the cluster to GNS
Use the GNS VIP not server or scan VIP

Bind Example




prodcluster.db.oracle.com NS gns.prodcluster.db.oracle.com
gns.prodcluster.db.oracle.com. 10.90.51.20
When a client connects the cluster will provide the IP address
As RAC servers are added DHCP assigns a VIP address, GNS is
updated with the new server’s VIP address and now can route
connects to the new server.


No client connection strings are never changed
Remove a node is basically reversing the process
Client Connections With GNS
ASM Changes


ASM part of grid infrastructure – one less HOME to
manage
New GUI management tool – asmca








Replaces DBCA for ASM management
Create diskgroups, volumes, ACFS mounts
Enhanced controls to manage ASM attributes
SPFILE is stored in ASM diskgroup
Cluster reads the diskgroup before ASM starts
ASMCMD is greatly enhanced and mostly replaces sqlplus
ASM roles can play a large role in managing ASM
Diskgroups can be cluster resources
ASM Changes (cont)

New O/S aware cluster file system (ACFS)



ASM disks can be used to create filesystem mounted by O/S
Only supported on Redhat/Oracle Linux, other O/S support
coming in the future
If a file can natively support ASM then the file is not supported
on ACFS!






No data, control or redo files
DirectIO is not available
Take advantage of ASM performance over other filesystems
ACFS is able to do snapshots
Still able to access ASM diskgroups with FTP/HTTP via
XMLDB
Copy command can copy out and in of ASM (11.1.0.7)
ASM Changes (cont)

Intelligent placement of data




ASM is able to put hot datafiles on the outer edge of the disk
Can provide a 50% increase in I/O performance
Compatible parameters for both ASM & RDBMS 11.2
Work best with the geometry of the disks are know





JOBD – uses disks sectors for data placment
LUN – logical sectoring number which may not have any relation to
the physical layout of the lun on the disk
Diskgroup should be 50% full for full benefit
alter diskgroup data add template datafile_hot attributes (hot
mirrorhot);
ascmd lstmpl –l –G data
ASM – Access Control Lists








Set permissions at the ASM file level
Set permissions for users and groups
User is the oracle database software owner
Only available for Linux and Unix
Not so much for security as for separation of duties
Files created in ASM are owned by DBUSER
Create a separate OSDBA group for each database using
a separate ORACLE_HOME. Need different groups for
OSASM and OSDBA for ASM
Compatible 11.2 diskgroup both for ASM and RDBMS
ASM – Access Control Lists (cont)

Disk attributes


ACCESS_CONTROL.ENABLED = TRUE
ACCESS_CONTROL.MASK has to be set

Mask values




Mask of 026 sets values of 640



6 removes all
2 removes write
0 removes nothing
read-write for owner, read for group and nothing for everyone else
asmcmd> setattr –d data access_control_enabled true
sql> alter diskgroup data set attribute ‘access_control.enabled’
= true;
ASM Permissions










ALTER DISKGROUP ADD USERGROUP … WITH MEMBER
ALTER DISKGROUP DROP USERGROUP
ALTER DISKGROUP MODIFY USERGROUP ADD MEMBER
ALTER DISKGROUP MODIFY USERGROUP DROP MEMBER
ALTER DISKGROUP ADD USER
ALTER DISKGROUP DROP USER
ALTER DISKGROUP SET PERMISSION
ALTER DISKGROUP SET OWNERSHIP
SELECT * FROM V$ASM_USER
SELECT * FROM V$ASM_USERGROUP
ALTER DISKGROUP ADD USERGROUP myfiles WITH
MEMBER bill;
ASM Roles

OS groups control rights in ASM




OSDBA – can access ASM files and set ACLs
ASMOPER – can start/stop ASM
OSASM – full ASM control
Allow people to administer ASM without having sysdba
rights to the databases
ASM Commandline Tool
ASM Commandline Tool (cont)

Create diskgroups from ascmd

Same syntax as sqlplus



Create diskgroup mydata external redundancy
DISK ‘ORCL:DISK1’ NAME mydata1, ‘ORCL:DISK2’ NAME mydata2;
Create diskgroup from XML via asmcmd

asmcmd> chkdg
ASM Commandline Tool (cont)

lsdg – list diskgroup information


Very similar to select * from v$asm_diskgroup
lsdsk – list disk information

Same information found in select * from v$asm_disk
Cluster Registry & Voting Disks


Raw devices only supported for upgraded clusters
Can move OCR and Voting disks to ASM diskgroup after
upgrade






ocrconfig add <diskgroup name>
ocrcheck
ocrconfig delete <raw device>
crsctl query css votedisk
crsctl add css votedisk <diskgroup name>
crsctl delete css votedisk <raw device>
Cluster Management


crsctl in the grid infrastructure home manages almost
every cluster command
crs_* scripts are depreicated
Depreciated Commands
















crs_stat
crs_register
crs_unregister
crs_start
crs_stop
crs_getperm
crs_profile
crs_relocate
crs_setperm
crsctl check crsd
crsctl check cssd
crsctl check evmd
crsctl debug log
crsctl set css votedisk
crsctl start resources
crsctl stop resources
Server Pools

Create “mini” clusters inside of larger clusters for policy
based databases

Two default pools Free/Generic



Specify min/max nodes and priority



Free pool contains servers not assigned to another pool
Generic pool is for running pre 11.2 databases and non policy
managed databases
Cluster manages the members of the pool based on load, availability,
priority and min/max requirements
Cluster will move servers from free and lower priority pools to meet
the needs of the server pool
Can create ACL on server pools for role based management
Vote/OCR Disk Management


Raw is supported for upgraded clusters but it is possible
to migrate to ASM disks after 11.2 cluster upgrade
Move vote disks to ASM



crsctl query css votedisk
crsctl replace votedisk +asm_disk_group
Move OCR to ASM


ocrconfig -add +new_disk_group
ocrconfig -delete old_storage_location
Troubleshooting Cluster Problems

Golden Rule – Ensure time is synced across cluster





Logs are stored under GRID_HOME/log/hostname



Help with comparing log files
Ensures nodes aren’t evicted due to time skew
If NTP is not used Oracle will use Cluste Time Synchronization
service
Recommend to have cluster down if changing the time
Main alert log
Separate directory for each cluster process
Diagnostic Script

GRID_HOME/bin/diagcollection.pl

Collects cluster and system logs for debugging
Troubleshooting (cont)

Modify time before node is evicted



Default is 30 seconds
crsctl set css miscount 45
On busy systems logs may not flush before node reboots

Crsctl set css diagwait 13 – force


Not set by default
Requires cluster outage to set
Adding A Node



Much easier in 11gR2
Setup ssh keys for new host
Run cluster verify tool

cluvfy stage –pre crsint –n <new host> <-fixup>



Fixes any problems
cluvfy stage –pre nodeadd –n <new host>
$ORACLE_HOME/oui/bin/addNode.sh –silent
“CLUSTER_NEW_NODES=<new host>”
“CLUSTER_NEW_VIRTUAL_HOSTNAMES=<new host vip>”
Instance Caging

Puts a cap on the number of CPUs a instance will use


Set the cpu_count parameter
Enable resource manager with CPU polices
Oracle Joins The Cloud

With 11gR2 it is now possible to create few larger
clusters while being able to maintain performance,
security and reassure customers concerns



Increased utilization rates
Lower administration costs
Happy Management