Infrastructure at your Service.
Data Guard over ODA
Data Guard over ODA
Infrastructure at your Service.
About us
David Hueber
COO
Principal Consultant
Mobile +41 79 963 43 68
david-.hueber@dbi-services.com
www.dbi-services.com
Data Guard over ODA
Page 2
28.05.2014
Agenda
1.  Introduction
2.  Preparing the databases
3.  Network configuration
4.  Standby database creation
5.  Setup Data Guard
Data Guard over ODA
Page 3
28.05.2014
Introduction
ODA & High Availability
>  High availability levels
>  Hardware availability
>  Service availability
>  What about data availability?
Data Guard over ODA
Page 4
28.05.2014
Introduction
High availability levels
3 levels
Hardware redundancy
>  Service availability
>  Data availability
>
Service
Data
Data Guard over ODA
Page 5
28.05.2014
Introduction
Hardware availability
2 x 1 RU x86 Server
2 x 18-cores Intel Xeon E5-2699 v3
>  256 GB RAM (8 x 32GB) – Extensible up to 768GB
>  2 x 600GB Boot Disks
>  Mirrored
>  1 x Internal SAS HBA
>  2 x External SAS HBA
>  4 x 10GbE customer network
>  2 x 40Gb/s Infiniband interconnect
>
1 x 4 RU Storage
4 x 2.5” 200GB SSD (Redo)
>  4 x 2.5” 400GB SSD (Flash Cache)
>  16 x 2.5” 4TB HDD (Data)
>
Data Guard over ODA
Page 6
28.05.2014
Introduction
Hardware availability
Storage redundancy
Controller 1
Controller 2
PCIe2
PCIe3
PCIe2
Expander 1
Expender 2
SSD - 0
SSD - 1
SSD - 2
SSD - 3
SSD - 4
SSD - 5
SSD - 6
SSD - 7
SAS - 8
SAS - 9
SAS - 10
SAS - 11
SAS - 12
SAS - 13
SAS - 14
SAS - 15
SAS - 16
SAS - 17
SAS - 18
SAS - 19
SAS – 20
SAS - 21
SAS - 22
SAS - 23
Storage Shelf backplane
Data Guard over ODA
Page 7
28.05.2014
PCIe3
Introduction
Hardware availability
Storage redundancy
Controller 1
Controller 2
PCIe2
PCIe3
PCIe2
Expander 1
Expender 2
SSD - 0
SSD - 1
SSD - 2
SSD - 3
SSD - 4
SSD - 5
SSD - 6
SSD - 7
SAS - 8
SAS - 9
SAS - 10
SAS - 11
SAS - 12
SAS - 13
SAS - 14
SAS - 15
SAS - 16
SAS - 17
SAS - 18
SAS - 19
SAS – 20
SAS - 21
SAS - 22
SAS - 23
Storage Shelf backplane
Data Guard over ODA
ASM Redundancy
Page 8
28.05.2014
PCIe3
Introduction
Hardware availability
Storage redundancy
>
ASM redundancy
>  Normal redundancy
>  High redundancy
Network redundancy
>
Channel bonding
>  Data network
>  Cluster interconnect
Data Guard over ODA
Page 9
28.05.2014
Introduction
Service availability
Integrated with OAKCLI
RAC One Node
>  RAC
>
>
Requires additional licenses
Please select one of the following for Database
Deployment [1 .. 3]:
1
=> EE : Enterprise Edition
2
=> RACONE
3
=> RAC
3
Selected value is : RAC
Data Guard over ODA
Page 10
28.05.2014
Introduction
Service availability
Not integrated with OAKCLI
>
Failover cluster
>  Requires manual configuration
>  Allows service switch between both ODA nodes
>
Included in Grid Infrastructure licenses
Data Guard over ODA
Page 11
28.05.2014
Introduction
What about data availability?
No out the box solution
Shared storage
Provides cluster abilities
>  Sensible to data logical corruption
>
No site disaster protection
Data Guard
Available in Enterprise Edition
>  Supported for ODA
>  Has to be configured manually
>
Data Guard over ODA
Page 12
28.05.2014
Introduction
What about data availability?
Data Guard setup
Prepare
primary
database
Prepare
standby
node
Data Guard over ODA
Duplicate
database
Declare
resource in
the cluster
Page 13
28.05.2014
Configure
Data Guard
Prepare the databases
Structure setup
>  Input information
>  Primary database
>  Standby database
Data Guard over ODA
Page 14
28.05.2014
Preparing the databases
Input information
Database files storage
Up to ODA version 2.10
>  ASM
>  From ODA version12.1.2.0
>  ACFS - snapshots
>
Files path
All files are Oracle Managed files
>  Destination defined by spfile
>
SQL> show parameter create
NAME
TYPE
VALUE
----------------------------- ------- ----------db_create_file_dest
string
/u02/app/oracle/oradata/datast
ore/.ACFS/snaps/DBITEST
db_create_online_log_dest_1
string
/u01/app/oracle/oradata/datast
ore/DBITEST
Data Guard over ODA
Page 15
28.05.2014
Preparing the databases
Primary database
Configure initialization parameters
db_domain
>  Not set by default
>  dg_broker_config_file1
>  dg_broker_config_file2
>
SQL> show parameter dg_broker
NAME
TYPE
VALUE
------------------------- ------- ---------------------------------dg_broker_config_file1
string
/u02/app/oracle/oradata/datast
ore/.ACFS/snaps/DBITEST/DBITEST/
dr1DBITEST.dat
dg_broker_config_file2
string
/u02/app/oracle/oradata/datast
ore/.ACFS/snaps/DBITEST/DBITEST/
dr2DBITEST.dat
Data Guard over ODA
Page 16
28.05.2014
Preparing the databases
Primary database
Activate force logging
Create services and triggers
>
Service for database access
>  <DBNAME>_PRI
>  <DB_NAME>_STD
>  <DB_NAME>_SNAP
>
Triggers to start appropriated services
NAME
---------------------------------DBITEST_PRI.it.dbi-services.com
DBITEST_SNAP.it.dbi-services.com
DBITEST_STD.it.dbi-services.com
Data Guard over ODA
Page 17
28.05.2014
Preparing the databases
Primary database
Create services and triggers
SQL>
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
DECLARE
DB_NAME
VARCHAR(10);
DB_DOMAIN
VARCHAR(96);
SERVICE_NAME VARCHAR(96);
BEGIN
select VALUE into DB_NAME
from V$PARAMETER where name = 'db_name';
select VALUE into DB_DOMAIN from V$PARAMETER where name = 'db_domain';
--DBMS_SERVICE.DELETE_SERVICE(rtrim(DB_NAME||'_PRI.'|| DB_DOMAIN, '.'));
DBMS_SERVICE.CREATE_SERVICE
(
service_name
=> rtrim(DB_NAME||'_PRI.'|| DB_DOMAIN, '.'),
network_name
=> rtrim(DB_NAME||'_PRI.'|| DB_DOMAIN, '.'),
failover_method => 'BASIC',
failover_type
=> 'SELECT',
failover_retries => 1800,
failover_delay
=> 1
);
END;
/
Data Guard over ODA
Page 18
28.05.2014
Preparing the databases
Primary database
Create services and triggers
SQL>
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
CREATE OR REPLACE TRIGGER startup_trigger
AFTER STARTUP ON DATABASE
DECLARE
DB_NAME
VARCHAR(10);
DATABASE_ROLE VARCHAR(25);
DB_DOMAIN
VARCHAR(96);
BEGIN
select VALUE into DB_NAME
from V$PARAMETER where name = 'db_name';
select VALUE into DB_DOMAIN from V$PARAMETER where name = 'db_domain';
select DATABASE_ROLE into DATABASE_ROLE from V$DATABASE;
IF DATABASE_ROLE = 'PRIMARY'
THEN
DBMS_SERVICE.START_SERVICE(rtrim(DB_NAME||'_PRI.'|| DB_DOMAIN, '.'));
ELSIF DATABASE_ROLE = 'SNAPSHOT STANDBY'
THEN
DBMS_SERVICE.START_SERVICE(rtrim(DB_NAME||'_SNAP.'|| DB_DOMAIN, '.'));
ELSE
DBMS_SERVICE.START_SERVICE(rtrim(DB_NAME||'_STD.'|| DB_DOMAIN, '.'));
END IF;
END;
/
Data Guard over ODA
Page 19
28.05.2014
Preparing the databases
Standby database
Create ORACLE_HOME on standby ODA
>
OAKCLI CREATE DBHOME
[root@dbi-oda2 ~]# oakcli create dbhome
INFO: 2014-11-19 17:35:04: Look at the logfile '/opt/oracle/oak/log/
arwoda1/tools/12.1.2.3.0/createdbhome_94706.log' for more details
...
...
...
Declare standby database in ORATAB
Create the admin structure
/u01/app/oracle/admin/<DBNAME>
>  Using mkdir as oracle user
>
Data Guard over ODA
Page 20
28.05.2014
Preparing the databases
Standby database
Create database files structure in ACFS
>
OAKCLI CREATE DBSTORAGE
>  Available since 12.1.2.2.0
[root@dbi-oda2 ~]# oakcli create dbstorage -db DBITEST
INFO: 2015-06-08 00:00:31: Look at the logfile '/opt/oracle/oak/log/
...
...
INFO: 2015-06-08 00:02:36: Successfully setup the storage structure
for the database 'DBITEST'
INFO: 2015-06-08 00:02:36: Set the following directory structure for
the database DBITEST
INFO: 2015-06-08 00:02:36: DATA: /u02/app/oracle/oradata/
datastore/.ACFS/snaps/DBITEST
INFO: 2015-06-08 00:02:36: REDO: /u01/app/oracle/oradata/datastore/
DBITEST
INFO: 2015-06-08 00:02:36: RECO: /u01/app/oracle/fast_recovery_area/
datastore/DBITEST
SUCCESS: 2015-06-08 00:02:36: Successfully setup the storage for the
database : DBITEST
Data Guard over ODA
Page 21
28.05.2014
Preparing the databases
Standby database
Create database files structure in ACFS
>
OAKCLI CREATE DBSTORAGE
[root@dbi-oda2 ~]# ls -l /u02/app/oracle/oradata/datastore/.ACFS/
snaps/
total 12
drwxrwxr-x 4 oracle oinstall 4096 Jun 8 00:17 DBITEST
drwxrwxr-x 5 oracle oinstall 4096 Apr 28 08:55 DBIPROD
drwxrwxr-x 5 oracle oinstall 4096 Apr 27 12:17 RCAT
[root@dbi-oda2 ~]# ls -l /u01/app/oracle/oradata/datastore/
total 112
drwxrwxrwx 2 oracle oinstall 8192 Jun 8 00:17 DBITEST
drwxrwxrwx 3 oracle oinstall 8192 Apr 28 08:59 DBIPROD
drwxrwxrwx 3 oracle oinstall 8192 Apr 27 12:26 RCAT
drwx------ 2 root
root
65536 Mar 20 10:24 lost+found
Data Guard over ODA
Page 22
28.05.2014
Preparing the databases
Standby database
Create database files structure in ACFS
>
Up to 12.1.2.1.0
>  ACFS structure must be created manually
>  ACFSUTILS for snapshot creation
[root@dbi-oda2 snaps]# acfsutil snap create -w DBITEST /u02/app/
oracle/oradata/datastore/
acfsutil snap create: Snapshot operation is complete.
[root@psrvma322 snaps]# ll
total 20
drwxrwxr-x 5 oracle oinstall 4096 Nov 26 12:21 DBPROD
drwxrwxr-x 4 oracle oinstall 4096 Mar 19 10:25 RCAT
drwxrwxr-x 5 oracle oinstall 4096 Feb 20 17:58 DBITEST
Data Guard over ODA
Page 23
28.05.2014
Preparing the databases
Standby database
Prepare spfile
Use primary one as basis
>  Create a pfile
>  Edit and adjust file
>  Set db_unique_name
>  Not set on primary side by OAKCLI
>  Delete parameter control_files
>  Will be re-created using OMF format automatically
>
Startup nomount the standby instance
Copy the password file to the standby site
Data Guard over ODA
Page 24
28.05.2014
Network configuration
Connect for creation...
>  Listener extension
>  TNSNames entries
Data Guard over ODA
Page 25
28.05.2014
Network configuration
Listener extension
Listener.ora stored in GRID_HOME
>
/u01/app/12.1.0.X/grid/network/admin
Version dependent
Oracle 11g
Requires an DGMGRL static entry
>  Shall be done as grid user
>
(SID_DESC =
(GLOBAL_DBNAME = DBITEST_SITE2_DGMGRL.it.dbi-services.com )
(ORACLE_HOME
= /u01/app/oracle/product/12.1.0.3/dbhome_1 )
(SID_NAME
= DBITEST)
)
Data Guard over ODA
Page 26
28.05.2014
Network configuration
Listener extension
Oracle 12c
Only a temporary static entry is required!
>  Used for database duplication
>  Data Guard does not use DGMGRL entries anymore
>  Replaced by dynamic DGB services
>
(SID_DESC =
(GLOBAL_DBNAME = DBITEST_SITE2.it.dbi-services.com )
(ORACLE_HOME
= /u01/app/oracle/product/12.1.0.3/dbhome_1 )
(SID_NAME
= DBITEST)
)
Data Guard over ODA
Page 27
28.05.2014
Network configuration
TNSNames entries
3 entries required
>
Client access
>  What ever the site
>  Based on Transparent Address Failover using Scan Listener
>  Used for application access
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = dbi-oda1scan.it.dbi-services.com)(PORT =
1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = dbi-oda2scan.it.dbi-services.com)(PORT =
1521))
)
>
Site access
>  Direct access to a each node
>  Used for Data Guard configuration
>  Based on ODA node‘s VIP
Data Guard over ODA
Page 28
28.05.2014
Standby database creation
Duplicate for standby...
>  Duplicate primary database
Data Guard over ODA
Page 29
28.05.2014
Standby database creation
Duplicate primary database
Using RMAN
Connect to primary database
>  Connect to standby database
>
RMAN> connect target sys/manager@//dbi-oda1:1521/DBITEST
RMAN> connect auxiliary sys/manager@//dbi-oda1:1521/DBITEST
>
Run duplicate for standby
RMAN>duplicate target database for standby from active database
dorecover nofilenamecheck;
...
...
archived log file name=/u01/app/oracle/fast_recovery_area/datastore/
DBITEST/DBITEST_SITE2/archivelog/2015_04_28/o1_mf_1_2081_bmynm39y_.arc
thread=1 sequence=2081
media recovery complete, elapsed time: 00:00:03
Finished recover at 28-APR-2015 11:27:07
Finished Duplicate Db at 28-APR-2015 11:27:15
Data Guard over ODA
Page 30
10/05/14
Standby database creation
Duplicate primary database
ODA duplicate issue up to 2.10
>
“out-of-the-box” duplicate fails
Starting backup at 11-AUG-14
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1219 device type=DISK
Finished backup at 11-AUG-14
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 08/11/2014 11:58:34
RMAN-05501: aborting duplication of target database
RMAN-06136: ORACLE error from auxiliary database: ORA-03113: end-of-file
on communication channel
Process ID: 37365
Session ID: 807 Serial number: 7
Data Guard over ODA
Page 31
10/05/14
Standby database creation
Duplicate primary database
ODA duplicate issue up to 2.10
>
Looking to the alert.log shows disk issues
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
ORA-15025: could not open disk "/dev/mapper/SSD_E0_S21_805939325p1"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
ORA-15025: could not open disk "/dev/mapper/SSD_E0_S22_805939344p1"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Data Guard over ODA
Page 32
10/05/14
Standby database creation
Duplicate primary database
ODA duplicate issue up to 2.10
>
Known issue in case of role separation
>  Database creation using DBCA work
>  Manual database creation fails
>  See Note 1084186.1
grid@dbi-oda2:$ ls -l /u01/app/oracle/product/11.2.0.3/dbhome_1/bin/
oracle
-rwsr-s--x 1 oracle oinstall 232754653 Jul 21 09:09 /u01/app/oracle/
product/11.2.0.3/dbhome_1/bin/oracle
grid@dbi-oda2:$ /u01/app/11.2.0.4/grid/bin/setasmgidwrap o=/u01/app/
oracle/product/11.2.0.3/dbhome_1/bin/oracle
grid@dbi-oda2:$ ls -l /u01/app/oracle/product/11.2.0.3/dbhome_1/bin/
oracle
-rwsr-s--x 1 oracle asmadmin 232754653 Jul 21 09:09 /u01/app/oracle/
product/11.2.0.3/dbhome_1/bin/oracle
Data Guard over ODA
Page 33
10/05/14
Setup Data Guard
Start replicating...
>  Create standby redo logs
>  Declare standby database
>  Create Data Guard configuration
>  Run a switchover
Data Guard over ODA
Page 34
28.05.2014
Setup Data Guard
Create standby redo logs
On both Primary and Standby database
>
Always 1 standby group more than Redo Logs
Almost Maximum Available (AMA) case
Real Application Cluster to Single instance
>  Standby redo logs shall be created for each thread of the primary database
>
Primary
instance 1
T1
G1
Primary
instance 2
Data Guard over ODA
T1
G2
T1
SG1
T1
G3
T2
G1
T1
SG2
T2
SG1
T2
G2
T2
G3
T1
SG3
T2
SG2
Standby
instance
T2
SG3
Page 35
10/05/14
Setup Data Guard
Declare standby database
Cluster resource
OAKCLI automatically register the database in the cluster
>  Database created by user must be added manually
>
srvctl add database – main parameters
-db
<db_unique_name>
>  -dbtype
SINGLE
>  -dbname
<dbname>
>  -role
PHYSICAL_STANDBY
>  -startoption
MOUNT
>  Otherwise Active Guard license is required
>  -instance
<instance_name>
>  -node
<node_name>
>
Data Guard over ODA
Page 36
10/05/14
Setup Data Guard
Declare standby database
srvctl add database
oracle@dbi-oda2:$ srvctl add database –db DBITEST_SITE2
-oraclehome $ORACLE_HOME
-dbtype SINGLE
-role PHYSICAL_STANDBY
-spfile /u02/app/oracle/oradata/datastore/.ACFS/snaps/DBITEST/
DBITEST_SITE2/spfileDBITEST.ora
-pwfile /u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/orapwDBITEST dbname DBITEST
-startoption mount
-stopoption immediate
-instance DBITEST
-node dbi-oda2
-acfspath "/u01/app/oracle/oradata/datastore,/u02/app/oracle/oradata/
datastore,/u01/app/oracle/fast_recovery_area/datastore"
Data Guard over ODA
Page 37
10/05/14
Setup Data Guard
Create Data Guard configuration
Standard process
Not ODA specific
>  Use DGMGRL
>
create configuration 'DBITEST' as primary database is 'DBITEST' connect
identifier is 'DBITEST’;
add database 'DBITEST_SITE2' as connect identifier is 'DBITEST_SITE2'
maintained as physical;
edit database 'DBITEST' set property StandbyFileManagement = 'AUTO’;
edit database 'DBITEST_SITE2' set property StandbyFileManagement =
'AUTO’;
edit configuration set property tracelevel=support;
enable configuration;
Data Guard over ODA
Page 38
10/05/14
Setup Data Guard
Create Data Guard configuration
All Data Guard features available
Log Transport
>  ASYNC
>  FAST SYNC (NOAFFIRM)
>  SYNC (AFFIRM)
>  Protection Mode
>  Max Performance
>  Max Availability
>  Max Protection
>
Active Guard
>  Snapshot standby
>
>
Faststart Failover
Data Guard over ODA
Page 39
10/05/14
Setup Data Guard
Run a switchover
Resources state before switchover
grid@dbi-oda1:$ crsctl stat res ora.dbitest.db -p | grep -i role
ROLE=PRIMARY
grid@dbi-oda2:$ crsctl stat res ora.dbitest_site2.db -p | grep -i role
ROLE=PHYSICAL_STANDBY
Perform switchover
DGMGRL> switchover to 'DBITEST_SITE2’;
Performing switchover NOW, please wait...
Operation requires a connection to instance "DBITEST" on database
"DBITEST_SITE2”
Connecting to instance "DBITEST"...
Connected as SYSDBA.
New primary database "DBITEST_SITE2" is opening...
Oracle Clusterware is restarting database "DBITEST”...
Switchover succeeded, new primary is "DBITEST_SITE2"
Data Guard over ODA
Page 40
10/05/14
Setup Data Guard
Run a switchover
Resources state after switchover
Roles have been appropriately switched
>  New primary starts in OPEN
>  New standby starts in MOUNT
>
grid@ddbi-oda1:$ crsctl stat res ora.dbitest.db -p | grep -i role
ROLE=PHYSICAL_STANDBY
grid@ddbi-oda1:$ crsctl stat res ora.dbitest.db -p | grep -i mode
USR_ORA_OPEN_MODE=mount
USR_ORA_STOP_MODE=immediate
grid@ddbi-oda2:$ crsctl stat res ora.dbitest.db -p | grep -i role
ROLE=PRIMARY
grid@ddbi-oda2:$ crsctl stat res ora.dbitest.db -p | grep -i mode
USR_ORA_OPEN_MODE=open
USR_ORA_STOP_MODE=immediate
Data Guard over ODA
Page 41
10/05/14
Setup Data Guard
Run a switchover
Resources state after switchover – 11g
Roles do not change
>  New standby starts in OPEN
>  Active Guard is then activated
>  New primary starts in MOUNT
>  Need to be opened manually
>  After a switchover or a failover resources must be changed manually
>  bug 12596559
>  Note 1670770.1
>
Data Guard over ODA
Page 42
10/05/14
Infrastructure at your Service.
Any questions? Please do ask.
David Hueber
COO
Principal consultant
Mobile +41 79 963 43 68
david.hueber@dbi-services.com
www.dbi-services.com
We look forward to working with you!
Data Guard over ODA
Page 43
28.05.2014