Mika Banda
Applications DBA Team Leader
Amway
Mika.banda@amway.com

Content provided for knowledge sharing purposes only.
Images belong not to me.
Content intentionally intended to educate.
I do not know everything.

• Introduction
– Amway and my background
• Part 1 – Personal High Availability
– Cool to the new guy, okay to old timer
– High Availability starts with you the support person
• Part 2 – Oracle E-Business High Availability at Amway
– Cool to the old timer, wow to the new guy
– Lots of metalink notes
• Part 3 – Bonus [Time Permitting]
– One thing to know about EXA World
• Open Discussion

• Mika Banda
– 18 Years in Oracle
– Old school – command prompt
– Oracle 6 – 11g (Saw ver. 5 & forms 2.5), EBS 10.7 NCA – R12
– Delta*, Government, Enron, GE, and now Amway
• Amway:
– Oracle E-Business R12.1.2 and 12.1.3, Oracle 11.2.0.x
– Shared appltop, PCP, SRDF FS
– Exadata, Exalogic, Blades, AIX, RAC, HACMP, EMC DMX
– Passive DR, active standby

Type Less To Do More
• Unleash your tricks
• Login Time
• Typo Time
• GUI Time
• Format Time
• Security Time
• Translation Time
• Time to connect
Ps –ef pipe cat grape peemon

• Fast Access Tools
• Crackberry Compatible
• Short Coded names
• Team Language
• Tele-Unix
• Manager Friendly
• FAST Diagnostics

• Forms Process Diagnostics
• All or limited to 1+ PIDs
• 4 Characters

• psg = ps –ef| grep $@
• fndlibr is 7 characters:
– Credentials
– SQL*Plus
– Process Details

• 4 characters
• Sub-second turnaround
• Overloaded – 0+ parameters
• A load of information 4 characters away

• Manager Safe
• Needs time – format challenge.
• Change management – team effort

• appnodes
• mycommands
• invalids
• mypatches
• myurl
• dobackup
• backups
• myPO
• allsga
• mydblinks
• mydown
• hotproc
• lastddl
• lastusers
• lcl
• mydirectories
• myPatches
• mytop10segs
• mytopmem
• mytree
• myprocess
• mypmon
• myoraerr
• mysess
• mysidmem
• myspid
• mysgatot
• mycpus
• mymem
• myinstances
• myconcreq
• mywfm
• ASH
• appnodes
• oraports
• COI
• oh
• sgatot
• blocks
• lsnr
• mtns
•
CLONE
• p
• file2patch
• SR
• SHORTS
• getPatches
• psg

• AMP 4.0 Plugin
• OEM 11g/12c
• Nimbus Tool$
• HP PPM Tool$

Application Tier
– Multiple Nodes
– Shared Appltop
– Load-balancing
– Redundant Hardware
– Virtual Servers
– Capacity On Demand
– SRDF
(Symmetrix Remote Data Facility)
Database Tier
– RAC
– Data Guard
– PowerHA/HACMP
– HW Redundancy
– Virtualization
– Replication
– Timefinder BCV

Always in sync, no multiple copy overhead, above all allows
Distributed AD
Node 1:
Start adpatch with Distributed AD options:
$ adpatch workers=27 localworkers=9
Node 2:
Start adctrl - Distributed is yes. Workers prompted.
$ adctrl distributed=y [Enter the worker range: 10-18]
1
Shared Filesystem
2 3
Node 3:
Start adpatch - Distributed is yes. Workers prompted.
$ adctrl distributed=y [Enter the worker range: 19-27]
See Notes 384248.1, 1324602.1, 236469.1, 233428.1

EBS HA: Parallel Concurrent Processing
Multiple Concurrent Processing Nodes (assume use of all three):
Setup - ** Remember FNDSM check before starting **
• Activate Internal Monitors for all nodes (+ worker + shift)
• Define Service Managers for all nodes (+ worker + shift)
• Point $APPLCSF on all nodes to shared file system
• Point $APPLPTMP on all nodes to shared file system
• $APPLPTMP must be part of UTL_FILE_DIR
• Set PO 'Concurrent: PCP Instance Check' to OFF
• Enable Reviver : 's_cp_reviver = enabled’
• Define node1 & node2 for all* managers.
1
Shared Filesystem: tech_st + appl_st
2 3
Interconnect
Notes 1359612.1*, 783044.1, 1389261.1, 551895.1, 466752.1

Multiple Concurrent Processing Nodes (assume use of all three):
Client Connections
Database Load Balancing:
• Point CP_TWOTASK and jdbc_url to loadbalanced entries
• cp_twotask => concurrent managers TWO_TASK
• jdbc_url => DBC file APPS_JDBC_URL (Java Managers – e.g. OPP, WF)
• Remote Listener unset to disable for PCP
1
Load Balance
2 3
Application Load Balancing:
• Web Host entry point = s_webentryhost
• Web domain entry point = s_webentrydomain
• Active Web Port = s_active_webport
• Login Page URL = s_login_page
• External URL info for 3 rd party Access = s_external_url
• Chronos URL = s_chronosURL
• EndUserMonitoringURL = s_endUserMonitoringURL
Load Balance
Interconnect
Shared Filesystem: tech_st + appl_st
See Notes 727171.1, 380489.1, 217368.1
, 477835.1,

Primary Secondary
• Production Load DR
• Read-Only DR
• Reporting offload
• Backup Offload
• Outage Reduction
Options

Pre-reqs :
Duplicate, updated snapshot, updated rapid clone/autocfg, same platform
Unwrapping a 30 hour Patch : a) Copy files b) Bin linking c) JSPs + Jars d) Data + SQL
C
D
= 18hrs
= 4 hrs e) Gen Forms
G = 8hrs f) Gen Reports g) Patch History …
Primary
4
3
1
6
5
Copy
2
Improved 8hr Process :
1 – Clone
2
– Apply Patch to Copy
3
– Sync files
4 - databaseportion
5 – Patch History
6
– Other files
See Note 734025.1, 387859.1,406982.1

4 - $ adpatch options=nocopyportion,nogenerateportion driver=u_merged.drv
6 Do NOT copy the following files:
$APPL_TOP/admin/<SID>
$APPL_TOP/log
$APPL_TOP/patches
$COMMON_TOP/util/apache
$COMMON_TOP/admin/scripts
$COMMON_TOP/html/bin/appsweb.cfg
$COMMON_TOP/html/US/ICXINDEX.htm
$COMMON_TOP/html/_pages
$COMMON_TOP/html/iby_debug.log
$COMMON_TOP/html/iby_error.log
$FND_TOP/log
$FND_TOP/out
Export Patch History:
$ perl $AD_TOP/bin/adphmigr.pl userid=apps/apps startdate='2012/04/10 00:00:00' enddate='2012/04/10
02:00:00' appsystemname=stage appltopname=FastPatch nodatabaseportion=Y
Import Patch History:
Copy 2 files (javaupdates<TIMESPAMP>.txt and adpsv<TIMESPAMP>.txt ) from above to
$APPL_TOP/admin/<Primary DB NAME>.
The next adpatch [now or later]
– will load these files to update history.
See Note 734025.1, 387859.1,406982.1

Understand Your Network Connections
For each Compute Node:
Client TNS entries differ from
Application FB based entries
Client Network (10GB)
Infiniband Network (40GB)
App Compute
Node 1
[A1,A2,A3]
A3
A2
CN One
Administration Network (1GB)
A1
D1
D2
CN One
D3
DB Compute
Node 1
[D1,D2,D3]

Understand Your Network Connections
For each Compute Node:
Client TNS entries differ from
Application FB based entries
Client Network (10GB)
Infiniband Network (40GB)
App Compute
Node 1
[A1,A2,A3]
A3
A2
CN One
Administration Network (1GB)
A1
BAD
TNS
D1
D2
CN One
D3
DB Compute
Node 1
[D1,D2,D3]

Understand Your Network Connections
For each Compute Node:
Client TNS entries differ from
Application FB based entries
Client Network (10GB)
Infiniband Network (40GB)
App Compute
Node 1
[A1,A2,A3]
A3
A2
CN One
Administration Network (1GB)
A1
Inefficient
TNS
D1
D2
CN One
D3
DB Compute
Node 1
[D1,D2,D3]

Understand Your Network Connections
For each Compute Node:
Client TNS entries differ from
Application FB based entries
Client Network (10GB)
Best
TNS
Infiniband Network (40GB)
App Compute
Node 1
[A1,A2,A3]
A3
A2
CN One
Administration Network (1GB)
A1
Best
TNS
D1
D2
CN One
D3
DB Compute
Node 1
[D1,D2,D3]