Twelve Steps to an Oracle Apps Upgrade
11.5.9 to 12.0.6 Upgrade Experiences from Hazelden Foundation
Jeff Vold
Hazelden Foundation
Oracle Apps DBA
jvold@hazelden.org
www.hazelden.org
John Stouffer
Oracle E-Business ACE
Independent Consultant
214 535 6847
john.w.stouffer@gmail.com
www.justadba.com
Agenda
• About Hazelden Foundation
• Environment and Project Overview
• 12 Steps to an Oracle Upgrade
• Other Observations
• Questions
About Hazelden
Since its 1949 founding in a Minnesota lakeside farmhouse,
Hazelden has grown into one of the world's largest and most
respected private not-for-profit alcohol and drug addiction
treatment centers.
We have helped tens of thousands of people reclaim their lives
from the disease of addiction. Hazelden offers the nation’s most
comprehensive approach to addiction by addressing the full range
of patient, family, and professional needs, including treatment and
continuing care for youth and adults, research, graduate school of
addiction studies, public education, and publishing.
We have facilities in Minnesota, Oregon, Illinois, New York, and
Florida. For more information, visit www.hazelden.org
11.5.9 Environment Overview - Functional
•
Modules
– Financials (AR GL AP FA PO)
– Inventory (INV, BOM, Advanced Supply Chain)
– Order Management (including Advanced Pricing and Quoting)
– Components of CRM and Project Costing
– iStore
•
Taxware - 3rd Party Tax Engine
•
Customizations
– Custom Interfaces, code, and objects
– Custom seeded packages
11.5.9 Environment Overview - Technical
•
Three Tiers
– Hardware - Sun Solaris 880s & 480s (2.8 OS)
– DB Tier
• 9.2.0.8 Database - Raid 5 disk
• 8 gig RAM and 4 CPUs
• Admin/Concurrent Managers
– Internal Apps tier
• Load Balanced (11501 Cisco Content Switches)
• Web, Forms, and Reports
• Discoverer 10g (Note:373634.1 – Option B)
– External Apps Tier (In DMZ)
• iStore
• Custom Website using JBoss
– Disk Space Allocated (Production)
• Database Space 198 gig/Used Space 132 gig
• **Environment Space allocated 667 gig
R12.0.6 Environment - Technical
•
•
Sun Solaris M5000 Servers (Solaris 10 OS)
– Reduced physical number of servers from 9 to 4
Still three tiers
– DB Tier
• 10.2.0.4 Database – Raid 10 Disk
• 16 gig Ram and 6 CPUs
• Admin/Concurrent Managers
– Internal Apps tier
• Load Balanced (11503 Cisco Content Switches)
• Web, Forms, and Reports
• Discoverer 10g (Note:373634.1 – Option B)
– External Apps Tier (in DMZ)
• iStore
• Custom Website Using JBoss
– Disk Space Allocated (Production)
• Database Space 283 gig/Used Space 207 gig
• **Environment Space allocated 1170 gig (extra allocated for growth)
Cat Herder or Oracle Upgrade Project?
R12 Environment – Team & Budget
•
Staff
– 1 Project Manager/Apps BA (and now E-Bus Tax Expert)
– 1 Apps SysAdmin/Analyst
– 1 Discoverer Administrator (& SysAdmin backup)
– 1 Java/Web Developer
– 1 Senior Software Engineer
– 1.5 DBAs
– 12 Super Users (No Staff Augmentation)
– Unix, Network, and Desktop Resources
– Total Staff Hours ’08 & ’09 – 18,631
•
Minimal Budget Available for Outside Consultants
– Technical DBA
– iStore Specialist
– Financials & OM Consultant
– Total Consulting Hours ’08 & ’09 – 1,011
12 Steps to an Oracle Upgrade
Step 1 – Getting Ready
•
•
•
•
•
•
•
•
Don’t Underestimate Size, Scope, and Complexity
Read and follow all the breadcrumbs
– Documentation, Metalink, Blogs, and OAUG
Training
Network with Others
Start Team Building
Decide on Architecture
– Hardware Configuration
– Upgrade Method
– Other Details
Testing
– Review and Update Super User Test Plans Prior to Project
– Plan for heavier user testing at the beginning
– Test Both (11.5.9) Migrated Data & Newly Created Data Thoroughly !!
Customizations - Less Issues Than Anticipated
– Spent many hours reviewing/updating/correcting documentation
Step 2 – DBA Project Planning
•
Organization
– Naming Conventions
– Build Documentation
– Unix Directory Structure
– Windows IT Directory Structure
•
R12 Environments
– Patch, Dev, Test, Training, Production
– Managed “Gold” Environment with Backups
•
Expect “bumps in the road”
– Extra clones
– Extra SR’s
– Use Oracle Support Configuration Manager
– Extra Patches (111 patches)
– Other issues
Step 3 – Go-Live Planning
•
Advanced Preparation and Planning Outage Window
– DBA Team
• Run through many iterations of practice build
• Optimize process to reduce manual steps & time
• Detailed build Instructions
– SysAdmin, Web, & Developers
• Same holds true – practice and optimize documents
– Super Users
• Pre-upgrade shutdown plan
• Post upgrade Setups, Validation, Input Orders, Startup
•
Work with Business (and IT) on Go-Live Dates
•
Create end-to-end Go-Live Weekend Plan
Step 4 – Create the Staging Area Directory
•
Installation Guide: Using Rapid
Install R12 B31295-07
•
Mount the Red “Start Here” cd
=> perl /mnt/cdrom/Disk1/rapidwiz/adautostg.pl
•
Check Rapid Installer Version
– cd /Stage12/startCD/Disk1/rapidwiz
– ./RapidWizVersion
– Download latest Rapid install startCD directory.
• (MetaLink Note 549389.1 – 12.0.4 Patch 8582089 – 12.0.4.10)
Step 5 – Apps Installation
•
Run Rapidwiz
– ./rapidwiz -servername hfdvd (Logical Host Name)
– Select Install Oracle Applications Release 12
•
Rapidwiz errors on OCM Details Screen
– Refer to MetaLink Note 746666.1
– Now fixed if using RapidWizVersion 12.0.4.10
•
RW-50004 Step 3 of 5 adrunias.sh
– APPS Tech Stack Registration fails
– Continue with install & then rerun failed command separately
– Now fixed if using RapidWizVersion 12.0.4.10
Step 5 – Continued
•
Apply 10.2.0.4 Database Patch (6810189)
– Apply Additional DB Patches (i.e., Latest CPU/PSU)
• Oracle/Toad Issue – Package names listed multiple times
•
Upgrade OracleAS 10g 10.1.3.4 (.5 is latest)
– MetaLink Note 454811.1 lists latest Oracle AS 10.1.3.x
•
Upgrade OracleAS 10g Forms and Reports
– MetaLink Note 437878.1 lists latest upgrade info
– OC4J Configuration Assistant Fails
– Developer Suite 10g Builders Now Included
Step 6 – Prep & Clone of 11.5.9
•
Upgrade Guide: Release 11i to Release
12.0.4 (E12011-02) / 12.1.1 (E16342-01)
•
Apply Pre-Upgrade Patches to 11i
– 11i.AD.I
– TUMS (Report Lists Steps NOT to Apply)
– SLA Pre and Post Upgrade (MetaLink Note 604893.1)
– Run AD Prep Scripts (Patch 5726010)
– Others
•
Create a Clone of Production to New Server
– Clone 9.2.0.8 Oracle Software
– Copy database and perform recovery as new name
Step 7 – Upgrade to 10g
•
Upgrade Database to 10gR2 – Manually or use DBUA
– 10gR2 DB Prep Guidelines for R12.0.4 (MetaLink Note 403339.1)
– Interoperability Notes 11i with DB 10gR2 (MetaLink Note 362203.1)
– Complete Checklist for Manual Upgrades to 10gR2 (MetaLink Note 316889.1)
•
Database “Connect Role” now only contains “Create Session”
•
utlu102s.sql may differ from dba_registry (MetaLink Note 456845.1)
•
catupgrd.sql ORA-00904 “Other XML” Error
– Recreate Plan Table (MetaLink Note 605317.1)
•
Data Mining and OLAP need to be installed
– Had to rerun olap.sql and dminst.sql to be valid
•
Run Script to Fix Korean Lexers (dr$fnd_lobs_ctx$i max extents)
Step 8 – Patching
•
Review List of All Known Application Patches and Download
– Use admrgpch to Group Patches
•
Prior to Patching
– Review R12 Best Practices
– Increase Tablespaces
– Modify $APPL_TOP/admin/adovars.env - Increase
– Disable Custom Triggers
– Other Fixes to Prevent adworker Errors
•
Enable Maintenance Mode with adadmin and Let it Rip !
•
12.0.4 Upgrade Driver
– E-Bus Tax Pre-Install
– E-Bus Tax Data fixes
– cd $AU_TOP/patch/115/driver (u6394500.drv)
– adpatch options=nocopyportion, nogenerate portion
• Useful Unix utilities - screen, less, & gvim
ADJREOPTS
Step 8 – Continued…
•
Run Pre-Install Merge Patch
•
Run Merge Patch (Includes 12.0.6)
– 12.0.6 long_running_jvm=false (MetaLink Note 761869.1)
•
One-off Patches Not Included in a Merge Patch.
•
Post Patching Steps
– Use of Logical host names created challenges
– Run fnd_conc_clone.setup_clean
– SysAdmin Responsibility => Install => Nodes - Corrected host names and
IP Addresses
– Fix FND_CONCURRENT_QUEUES
– 1000 Concurrent Manager Programs
– ARHMAILS - Mailstop Migration Runs 6.5 Hours
• Support – provided new arhmails.sql (no patch)
Step 9 – Shared Applications Top
•
MetaLink Reference Note 384248.1 (Check Note 785296.1)
•
Two Methods for Creating
– Using Rapidwiz (Section 3)
– Not Using Rapidwiz (Section 4)
•
Share Mount Points from DB Server
– Non-Production One NFS
– Production Five NFS Mount Points
• appl, comn, 10.1.2, 10.1.3, and inst
•
Run adpreclone on appsTier
– perl adpreclone.pl appsTier
– Encountered issue with cookies ( Notes 465583.1 and 268830.1)
•
Run adclonectx.pl to Add a Node
– perl adclonectx.pl addnode contextfile=
Step 9 – Continued
•
•
•
•
•
•
•
•
•
•
•
•
Provide the values required for creation of the
new APPL_TOP Context file.
Target System Hostname (virtual or normal) [pra1] : pra1
Target System Domain Name : cc.hazelden.org
Target System Root Service [enabled] :
Target System Web Entry Point Services [enabled] :
Target System Web Application Services [enabled] :
Target System Batch Processing Services [enabled] : disabled
Target System Other Services [disabled] :
Do you want to preserve the Display [prd:0.0] (y/n) ? : n
Target System Display [pra1:0.0] : localhost:15.0
Database port is 1529
Do you want the the target system to have the same port values as the source
system (y/n) [y] ? : y
Step 10 – AutoConfig, Custom Templates, & ifiles
•
Autoconfig (MetaLink Note 387859.1)
– Meant to make managing configuration easier
– Review changes first using adchkcfg.sh
– Running AutoConfig
– Rolling back an AutoConfig Session using restore.sh
•
Custom Templates
– view httpd.conf
• ## $Header: httpd_conf_1013.tmp 120.25.12000000.9
– Or run adtmplreport.sh target=httpd.conf
– $FND_TOP/admin/template/custom
• Document all changes
•
Include files (IFILE)
– example.. tnsnames.ora
• IFILE=/u19/……/10.1.2/network/admin/DEV_hzdev_ifile.ora
Step 11 – License IEX & ZX
•
Log into OAM with SYSADMIN
– Go into License Manager
– Click Products under License
– Click License Component Application
– Click Continue and Register Modules
•
Through adadmin
- Generate message files
- Generate forms files
- Generate report files
- Generate product JAR files and force generation “Yes”
- Compile APPS schema
- Compile menu information
- Compile flexfields
•
MetaLink Notes 577644.1 and 371016.1
Step 12 – Post Upgrade Steps
•
Reapply Customizations
– Create Custom Directories (Same convention as seeded)
– Register custom tops (Navigation: Applications=>Register)
– Migrate Code from 11i Top to New R12 Custom Top
– Install Custom and Custom Seeded Objects & Code into Database
– Created script to check if custom seeded object were overwritten
•
Examples of post upgrade fixes
– FSG’s Completes with Error ORA-1722 (Note 579655.1)
– R12: Tools -> Copy Function fails with FNDFS Error (Note 419142.1)
– Can’t Insert User Role When Assigning Responsibility (Note 559347.1)
– frmcmp_batch.sh ora-12154 Couldn’t resolve connect (Note 826024.1)
– ORA-01001 When Clicking Cancel-Query Button [ID 436537.1]
•
Check for Invalid Objects, Statistics, Full Backup, Clones, and More
Some Upgrade Statistics
•
Upgrade Time
– DBA “backup to hand off for setups” - 33 hours
– SysAdmin/Super User Setups – 13 Hours
– Super User Non-Transactional Validation – 3 hours
– Super User Transactional Validation (Input orders) – 6 hours
– Super User/IT Transactional Troubleshooting – 19 hours
– Go-Live Decision? => It’s a go!
•
Some Stats as of 4/1/2010
– Internal Problem Requests
• Total 589/Pre Go-Live 445/ Post Go-Live 144
– R12 Oracle Support Service Requests
• Total 235/Pre Go-Live 176/ Post Go-Live 59
• Top Modules
– E-BusTax, Receivables, Order Mgmt, Payments, & Payables
– iStore not as bad as expected
Pre Go-Live SR’s 7/27/08 – 7/27/09
Post Go-Live SR’s 7/28/09 – 4/1/10
Sample of Post Go-Live Issues
•
•
•
•
XML Reports and Logical Host Names
–
Java.Net.ConnectException: Connection Timed Out (MetaLink Note 753395.1)
–
Patch 7449694 and Profile option change
RUP6 – APP-FND-01540: Unsuccessful logins since last session
–
11.5.10 Fix But Disables All Notifications
–
Oracle Provided Fix – Patch 5015899
Credit Card Issues
–
Migrated Credit Card Orders not Masked (Patch 7708802)
–
Credit Card issues with back orders (still issue)
AP Suppliers and Bank Accounts (Now in TCA)
–
Thoroughly check bank accounts migrated correctly
–
External Bank account numbers missing (Oracle API to Update)
Continued ….
•
Tax Issues
– E-Bus Tax Integrated with 3rd Party Taxing Solution Taxware
– EBTax – Many Upgrade Issues (Note 602030.1)
– Multiple Issues with Tax Exemption migrations (Some fixed others not)
– Taxware does not work for Canada
• Need to use E-Bus Tax
•
Workflow Issues
– After upgrading to 12.0.6, WFERROR is ending in Error (Note 815173.1)
– Workflow error alerts when customer is created/updated (Note 1070881.1)
– Workflow error alerts when creating Party Relationship for Org (Note 864353.1)
– Disable Workflow AP email Remittance Advice not used (Note 472389.1)
•
Purge ICX_SESSION_ATTRIBUTES Issue
– R12 ICX: Limit Time = 24 (hours) (Seeded with 999)
•
iStore
– 12.0.6 Lost ability for branded sign-in page (Note 732970.1)
Questions …
Questions
Jeff Vold
jvold@hazelden.org
www.hazelden.org
Hazelden Foundation
And
John Stouffer
Independent Consultant
john.w.stouffer@gmail.com
www.justadba.com