Data Guard and RAC: Maximum Availability at
Verizon Wireless
Orlando, April 13, 2011
Session Id: 725
Syamal Bandyopadhyay
Sharath Nasani
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
1
Agenda
• Agenda:
–
–
–
–
–
–
–
–
–
–
–
About The Presenter
Objectives
Maximum Availability Requirements at Verizon Wireless
Database Deployment Architecture
Why Data Guard ?
Challenges with Data Guard Deployments
Why RAC?
Performance Challenges with RAC
Flashback Technology
Conclusion
Q/A
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
2
About The Presenter
Name: Syamal Bandyopadhyay
Expertise in Oracle Database Tuning, Scalability, High Availability,
Deployment Architecture, and Capacity Planning
Working at Verizon Wireless since 2002
Worked at:
AT&T
IBM
Telcordia Technologies
Bell Labs
Contact Information:
syamal.bandyopadhyay@verizonwireless.com
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
3
About The Presenter
Name: Sharath Nasani
Expertise in Oracle Database Tuning, Scalability, High Availability and
Deployment Architecture
Working at Verizon Wireless since 2000
Contact Information:
Sharath.nasani@verizonwireless.com
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
4
Objectives
Sharing Maximum Availability Database
Deployment Architecture at Verizon Wireless
Sharing Lessons learned at Verizon Wireless
Enabling ‘you’ the attendees to be better
positioned to design your Maximum Availability
system
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
5
Maximum Availability Requirements at Verizon Wireless
Maximize Availability
Minimize unplanned outage
System Maintenance without downtime
Cost Efficient (e. g. Utilize Disaster Recovery Site for read only)
Operationally manageable
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
6
•HP-PARISC
•Oracle 10.2.0.4
•Shareplex 6.x
•Veritas Filesystems
Data Center - 1
Database Deployment Architecture
Provisioning/Billing
Data Center - 2
Data Center - 3
•HP-Itanium
•Oracle 11.2.0.2
•Shareplex 7.x
•ASM Storage
•Billing DB size: 50 TBs
•Compress Older partitions.
•Billing Reporting Application
Accesses Active Data Guard
Database
Oracle Data Guard
Prov
DR
Prov
Shareplex replication
Shareplex
replication
Prov
Report
ing
Billing
DR/Re
porting
Oracle
Billing
Active Data
Guard
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
7
Database Deployment Architecture: Customer Care
Data Guard
Shareplex
Data
Replication
CCARE
CCARW
DR
CCARW
Data Guard
CCARE
DR
• 5 Node RAC cluster
• 2 sites
• Oracle 10.2.0.4
• Veritas SF RAC 5.0
• Data Guard for DR
• 2-way data replication using
Shareplex
• Flashback
• DB size: 2.5 TBs
• Solaris 10 / M5000 servers
• Each cluster supports 2
databases (production + DR)
8
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons
or third parties except by an authorized representative of Verizon Wireless.
Database Deployment Architecture – Content Management
….
Global Switch System at Internet level (GSS)
Context Switch System at Web/App
servers level (CSS)
……
……
DB Server:
•Oracle 11.2.0.2
• Oracle Flashback
• SUN M5000 Servers
• Symantec SFRAC 5.0
• Hitachi/EMC Storage
4- Node RAC Cluster
Data Replication Shareplex
CMPEDB
CMPWDB
9
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons
or third parties except by an authorized representative of Verizon Wireless.
Why Data Guard?
•
•
Disaster Recovery for Site Failure
Active/Active Data Guard:
• For Site Failure
• Read Access at DR Site
• Optimal utilization of H/W
10
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
Challenges With Data Guard Deployment
•
•
•
•
•
•
•
•
Define the business purpose and SLA for DR site
Redo Volume
Distance Between Data Centers
Available Network Bandwidth
Network Tuning
Selecting the appropriate mode to support SLA
Select Appropriate Redo Transport mechanism
Oracle 11g is better tuned for redo/archived log transportation
synchronization
11
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
Challenges With Data Guard Deployment (contd….)
•
•
•
•
•
•
•
Which Mode to use?
Reduce Redo volume --- reduced by 80% for massive data load
using DMExpress software from Syncsort
Enforced logging for all DML With the exceptions of tables not
required at the DR site
Tuning
Turned off Automated Failover
Monitoring is extremely critical
Very Reliable Oracle product
12
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
Challenges With Data Guard - Tuning
• SQL*Net Tuning
•
•
TNS Names.ora

(SDU = 32767) ( Default is 2048 Bytes)
•
TNS & Listener.ora


(SEND_BUF_SIZE = 3145728) ( Default is 49152 Bytes)
(RECV_BUF_SIZE = 3145728)
Redo Transport Tuning


•
log_archive_dest_2 service="rssbprddg", LGWR ASYNC max_connections=5
 5 Arch processes are transmitting single file, and 5 RFS processes are receiving at Standby
site.
Log_archive_max_processes=30
Redo Apply Tuning



recovery_parallelism = 8 ( In 11g this parameter is equivalent of Number of CPUs in the system).
parallel_execution_message_size to 8192 ( default is 2152)
db_cache_advice to OFF in standby only
13
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
Challenges With Data Guard - Monitoring
• Monitor MRP process in Standby database

SELECT PROCESS, STATUS, CLIENT_PROCESS,inst_id,THREAD#, SEQUENCE#, BLOCK#, BLOCKS FROM
gV$MANAGED_STANDBY order by sequence#
PROCESS
ARCH
ARCH
ARCH
ARCH
ARCH
ARCH
ARCH
MRP0
RFS
RFS
RFS
RFS
RFS
RFS
STATUS
CLOSING
CLOSING
CLOSING
CLOSING
CLOSING
CLOSING
CLOSING
WAIT_FOR_LOG
IDLE
RECEIVING
RECEIVING
RECEIVING
RECEIVING
RECEIVING
CLIENT_P
ARCH
ARCH
ARCH
ARCH
ARCH
ARCH
ARCH
N/A
LGWR
ARCH
ARCH
ARCH
ARCH
ARCH
INST_ID
THREAD#
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
SEQUENCE#
BLOCK#
BLOCKS
32270
15360
32271
9216
32272
12288
32273
12288
32274
13312
32275
12288
32276
11264
32278
0
32278
944599
32277
812122
32277
212323
32277
232323
32277
43434
32277
2324213
125
709
42
591
940
477
443
0
1
800
323
534
542
663
• Backlog Monitoring From Primary database
select name, sequence#, registrar,standby_dest,archived,applied from v$archived_log
where name='rssbprddg‘ and applied !='YES‘ order by sequence#
NAME
rssbprddg
rssbprddg
rssbprddg
rssbprddg
rssbprddg
rssbprddg
rssbprddg
SEQUENCE#
32547
32548
32549
32550
32551
32552
32553
REGISTRAR
ARCH
ARCH
LGWR
ARCH
ARCH
ARCH
ARCH
STANDBY_DEST
YES
YES
YES
YES
YES
YES
YES
ARCHIVED
YES
YES
YES
YES
YES
YES
YES
APPLIED
NO
NO
NO
NO
NO
NO
NO
14
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
Why RAC?
•
•
•
Maximizing Database availability from server or instance failure in a
single site
Horizontal Scalability
Reduced Cost of Ownership
15
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
Challenges for RAC Deployment
•
•
•
•
•
•
Application Performance ---- faced severe challenges
Database Object Tuning in achieving acceptable response time
Process Isolation
Data Isolation
Massive testing including failure/failover scenarios
Training
16
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
RAC Performance – A Simple Concept
SQL Response Time Equation:
Non RAC:
Response time = CPU time + Wait time (IO wait + Queue time)
RAC:
Response time = CPU time + Wait time (IO wait + Queue time
+ Cluster Wait Time)
17
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
RAC Performance – SQL To Detect Cluster Wait Time
select parsing_schema_name owner, inst_id, sql_id, executions,
elapsed_time/1000/executions elapt,
cpu_time/1000/executions cput, cluster_wait_time/1000/executions cwt,
(cluster_wait_time/1000/executions)/(elapsed_time/1000/executions) CWTtoELAPT ,
buffer_gets/executions lio, disk_reads/executions phyio , rows_processed/executions
rowcnt from gv$sql
where executions > 0 and (elapsed_time/1000)/(decode(executions,0,1,executions)) > 0
and
parsing_schema_name in ( ‘your schema name') and last_active_time > (sysdate - 1/1440)
order by 7 desc
Per Execution
OWNER
IVRPROD
C0MUSLA
ACSSPROD
ACSSWEB
ACSSWEB
ACSSWEB
ACSSWEB
NEW_FOLLOWUP
INST_ID SQL_ID
4
1
4
2
1
2
1
2
EXECUTIONS ELAPT
CPUT
7w6hskya9az3c
2,788 13,304.23 4,400.94
c0h16fgssh5uj
73
112.72 33.94
8phk00fbc0m7b
1,788
59.04 24.98
f29bk27cqtahw
408,199
144.54 14.79
f29bk27cqtahw
382,544
148.50 20.48
8mcgk5cs23jtj
311,375
91.79 12.12
8mcgk5cs23jtj
7,409,683
97.01 17.62
9rdxj8rpgt587
109,109
72.24
7.48
CWT
6,308.60
57.79
32.92
28.98
27.53
25.57
24.82
20.52
CWTTOELAPT LIO
0.47
0.51
0.56
0.20
0.19
0.28
0.26
0.28
PHYIO
ROWCNT
64,788.75 9,939.94 732.57
574.32
6.27 153.85
176.05
5.96
0.01
223.89
26.62 117.16
225.61
26.51 117.60
380.60
16.35
3.60
380.68
16.40
3.60
132.02
12.45
3.15
18
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
RAC Performance – Techniques to Reduce Cluster Wait Time
•
•
•
Identify SQLs and Database Objects having High Cluster Wait
Time
Cluster Wait Time must be reduced
Use the Techniques to reduce Cluster Wait Time
• Define Services for data and process isolation
• Partition / Sub-partition tables
• Partition indexes (preferably Hash)
• Use Freelist Group
• Use Freelist
• Increase initrans
• Remove unnecessary Indexes
• Reduce Full Table Scans, Index Full Scans, Index Fastfull
Scans, Skip Index scans
• Use Multiple block size (2K, 4K, 8K, etc.)
19
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
High Availability Using Flashback
•
•
•
•
•
•
•
Provides point-in-time quick recovery of database objects
Facilitates Testing of planned Disaster Recovery twice a year
Create a flashback point prior to the test
Execute the test
Execute flashback recovery to the pre test flashback point
Continue applying logs
Requires additional disk storage for flashback logs
•
Not a substitute for backup!!!
20
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
Conclusion / Take Home for Attendees
•
•
•
•
•
Various available options used at Verizon Wireless
Maximum Availability achieved using combinations
of technology
Selecting appropriate mode for Data Guard is critical
Reduce Redo Generation by all means
Understanding RAC Performance and Tuning is a
big challenge
21
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
Q/A
22
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.
Data Guard and RAC: Maximum Availability at
Verizon Wireless
Orlando, April 13, 2011
Session Id: 725
Syamal Bandyopadhyay
Sharath Nasani
23
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not
permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.