Red Rock Consulting
Ten RAC lessons You
Can’t Do Without
Presented By John-Paul Tocker
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
Agenda
• Introduction
• Overview of Oracle RAC
• The Cost of RAC
• The Difference between RAC and Dataguard
• The RAC Stack
• Interconnect, bigger is better.
• Questions you should ask yourself.
• Top 10 lessons learnt
• Questions
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
Who Is Red Rock Consulting
• Oracle Partner and Re-seller
• DBA / Middleware Consulting and Out-Task
• Support Oracle, SQL Server, and Sybase
• Also Java and Oracle Application Express custom development
• Also Oracle e-Business Suite, Peoplesoft, JDE
• 50 people in Wellington and Auckland
• Overall, 350 in Australasia
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
Introduction
• The typical New Zealand business is not on scale with a World Wide
enterprise that would that would deploy a large scale geographically
separated RAC environment.
• Most RAC projects overrun budget and timeframe.
• Lessons learnt and what best practises can help you with your
implementation.
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
The Sales Pitch
Could a 1976 Lada Tow your new Yacht to Auckland?
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
To RAC or Not to RAC
• Did you identify and research your requirements before you went
shopping?
• How did you specify your requirements? A solution just to get the job
done or a robust system that will allow for growth and change.
•
What about a proof of concept? Or real working builds?
• Engage or find out who is your local Oracle sales representative, they
have access to the tools and documentation to help you through this
process.
•
RAC is not Bullet Proof!
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
Get Involved In The Project
• Get Involved Early on. Identify the misconceptions “management” often
have of Oracle RAC.
• RAC is an Enterprise Decision and must involve people from all corners
of your technology shop.
• Storage Solutions – ASM or Third Party Clustered file system making the
right choice.
• Network – Identify the requirements of the Interconnect and extra
requirements for RAC such as VIP’s and Public IP’s.
• Infrastructure – HP OTB Intel Based Blade Servers or Large Sun boxes
for example.
• Identify Requirements for the Database – Goes with out saying.
• Application – Has it been written for RAC? Is the application certified for
RAC
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
The Butterfly Effect … Not The Movie
• The Butterfly Effect. Decisions made early on without consideration to the
Database design will have large ramifications later on in the project.
• Your Database Solution should reflect
the following considerations
• Hardware
• Storage
• Availability
• Redundancy
• Network
• Support
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
The RAC Puzzle
• The build process can take more than a week
• Your disk partitions must be presented before the build process.
• ASM, Veritas, GPRS or OCFS2 perhaps
• The interconnect
• Public and Private IP’s configured
• ssh configured on all nodes
• What about the voting disk? Not to mention the OCR, where and how are
we storing those??
• Satisfy CLUVFY
• RAC specific failover and performance testing.
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
Dataguard Or RAC ?
• RAC is scalable, Dataguard isn’t.
• RAC is a complete solution where Dataguard can be dumped anywhere
on any hardware
• Cost factor $$$$
• Dataguard is a redundant copy, But now can be used for reporting with
11g active Dataguard.
• RAC by itself is not a DR solution.
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
Scaling RAC
Instance 3
Node 3
Add node here
Add node here
Instance 1
Interconnect
Node 1
Local
Disk
Add Value
11
Instance 2
Node 2
Shared
Storage
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Local
Disk
Great Employer
What The Documentation Doesn’t Say
• Transparent Application Failover (TAF) is semi transparent. Applications
must handle the error message.
• Rolling Patches aren't all ways rolling. Any changes to the catalogue
mean you will require an outage!
• Your Database still needs outages for Maintenance.
• RAC is not a guaranteed performance increase over a single instance
database.
• RAC is not a DR solution.
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
The Real Cost of RAC
• Hardware
• Interconnect
• Licensing
• Support
• Development costs
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
The Real Cost of RAC
•
Does
•
If you don't go down the Oracle RAC path and more processing power
is required down the track re-platforming maybe be more expensive in
the long run......
•
The majority of RAC incidents and outages are caused by the operator
or DBA.
 With complexity comes risk.
•
Big factor to consider here…..
your
Database
meet
the
required
business
Does the high cost of Oracle RAC offset the costs of
SLAs
failing
to meet SLA’s or down time?
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
What About Standard Edition
• Database for servers with up to four sockets.
• It is also upwardly compatible with Enterprise edition and can easily grow
with you, protecting your initial investment.
• Database files must use ASM
• Must use Oracle Clusterware
• Shared RAW partitions must be used to store the Oracle Clusterware
configuration files, CRS and Voting Disk.
• Third party volume managers and file systems are not supported for this
purpose.
• Could save as much as $200,000! And 30k per year in support.
• Metalink note 271886.1 and 465465.1
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
K.I.S.S.
• Keep it Simple Stupid 
• Why add more pieces to the puzzle?
• Increase support requirements and troubleshooting managing several
vendors to congregate which nodes are being ejected = Headaches!
• Follow the Crowd!!! Choose what is common.
• ASM, CRS on a well supported Platform!!!
• OTB even!
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
Choosing Your RAC Stack
•
•
Choosing your RAC configuration……
•
ASM
•
VERITAS
•
QCFS
•
GPRFS
•
Sun Cluster
•
QFS
Should you go with Oracle for the full stack? I don’t see why not?
Remember the more pieces you add to the puzzle the more grey
areas you will have to manage.
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
Sun Clustering
You may however need to install Sun Cluster in the following scenarios:
1. If there is application running with in the cluster along with Oracle
RAC database that you want to configure for HA and Sun Cluster
provide the cluster resourced (easy to use) to manage and monitor
the application. THIS can be achieved with Oracle Clusterware but
you will have to write your own cluster resource for that.
2. If you want to install cluster file system such as QFS then you will
need to install the Sun Cluster. If this cluster is only running the
Oracle RAC database then you can rely on Oracle technologies
such as ASM, raw devices without installing Sun Cluster.
3. Any certification conflicts.
4. If your application requires a shared filesystem.
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
RAC Certification
•
Certify your Hardware Stack with Oracle and your Vendors.
•
Certify your application.
•
Ensure your hardware has a life time to see you well into the
future.
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
The Interconnect
•
Crossover cable not accepted.
•
Make your network admin RAC aware.
•
Know what traffic is on your Interconnect.
•
Gigabit should be a minimum.
•
Try and get a base line for performance on the Interconnect
•
Fault tolerant network
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
What You Need To Ask Yourself
•
Is your application written for RAC, Does it require a re-write to support the
add RAC functionality such as TAF and load balancing?
•
Is your application RAC certified?
•
Does your project support the monetary costs of RAC and the extra costs of
supporting RAC?
•
Have you identified single points of failure outside the database solution, ie
single points of failure in the network, no UPS single HBA cards and NIC’s.
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
What You Need To Ask Yourself
•
Can you sustain a 20min outage?
•
Do you require a DR function to allow your system to continue
after fatal disaster?
•
Do you have the DBA skills to build and support your RAC?
•
Can your company leverage off an existing RAC solution?
•
Will you have to setup Dataguard to another geographically
separate RAC site.
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
Shared Oracle_Home’s
• It provides another level of redundancy.
• Allows you to mange patching and upgrades, in some cases rolling
patches will allow your RAC to operate while one home is being
patched.
• Reduced downtime.
• In a large RAC where you are managing a possible 15 nodes then it’s
not likely that you will be able to manage each home individually. This
is where you will consider to choose Shared against local. Although
you do have the option of using opatch and ssh and replicating the
patch across all nodes.
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
Solution Best Practices
1.
Use local ORACLE_HOME and CRS_HOMES.
2.
Standardize your RAC OS build and paint it Gold.
3.
Clearly identify requirements and identify either RAC or Dataguard.
4.
RAC is not a DR solution
5.
TAF is not seamless.
6.
Select the right Technology to support your RAC.
7.
Identify if your application is written for RAC or certified for RAC.
8.
Cluster file systems complicate the solution.
9.
One vendor one phone call for support.
10. Don’t compromise on the Interconnect.
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer
Questions
John-Paul Tocker
Red Rock Consulting
email: tockerj@redrock.net.nz
phone: 04 495 3350
MSN: jptocker@gmail.com
virtual: www.redrock.net.nz
Add Value
Deliver Excellence
Copyright (C) Red Rock Consulting
Commercial and in Confidence
Great Employer