Oracle E-Business Suite Release 12.2: Upgrade Sizing and Best Practices This document contains database schema sizing information, application tier sizing information, upgrade timing, and other best practices, all designed to make your upgrade to Oracle E-Business Suite Release 12.2 as smooth as possible. The sizing data is based on the upgrade of a 456 GB Release 12.1.3 database to Release 12.2. There is a change log at the end of this document. In This Document • • • • • • Application Tier Size Tablespace Size Schema Size R12.2 Best Practices Required Post upgrade Jobs Sample Upgrade Timings Database Size: Before Upgrade DB Size (GB) After Upgrade DB Size (GB) Delta (GB) % Growth 456 481 25 5.5 The 25 GB greater database size after the upgrade reflects the additional system tablespace needed as a prerequisite for enablement of Edition-Based Redefinition (EBR). Environment details Operating system: Server Memory: Number of CPUs: Database Release: Oracle E-Business Suite: Oracle Linux Enterprise Edition Server Release 5.8 34 GB 32 11.2.0.3 12.1.3 Note: The database and application tiers are on the same machine. Database Configuration SGA: Shared Pool Size: PGA: Log Buffer: job_queue_processes: 5 GB 1 GB 3 GB 30 MB 32 During the upgrade of the Admin Tier, batchsize was 1000 and number of workers used was 32. Application Tier Size Oracle E-Business Suite Release 12.2 introduces Online Patching. This requires three application tier file systems: • fs1 (Production file system) – Used by the current users of the system. • • • fs2 (Copy of production file system) – Used by the patching tools. fs_ne (Non-editioned file system) – Stores files containing data that is needed across all file systems (for example, data import and export files, report output files, and log files). Inst (INST_TOP) - Oracle E-Business Suite Release 12 Instance Home, contains all the config files, log files, SSL certificates etc. All three file systems serve a single database. The file system that is currently being used by the running application is never patched: all patches are applied to the file system that is not currently in use. Component Before Upgrade Size After Upgrade Size ORACLE_HOME APPL_TOP INST_TOP fs1 (APPL_TOP+ INST_TOP) fs2 (APPL_TOP+ INST_TOP) fs_ne 3.6 GB 28 GB 20 MB Not applicable Not applicable Not applicable 3.6 GB Not applicable Not applicable 30 GB 29 GB 660 KB Tablespace Size Tablespace size was calculated as “Total TableSpace Size – Free Space”. Tablespace Name SYSTEM CTXSYS OWAPUB SYNCSERVER DCM CWMLITE SYSAUX HTMLDB B2B_IDX MTR DSGATEWAY_TAB DISCO_PTM5_META B2B_LOB OLTS_DEFAULT OPMOR PORTAL PORTAL_DOC B2B_DT OLTS_SVRMGSTORE RE IAS_META WCRSYS_TS OCATS UDDISYS_TS PORTAL_LOG OLTS_ATTRSTORE ODM_DATA Before Upgrade Size (MB) 16489 475 0 25 178 0 1594 311 14 2 5 2 11 2 5 1 0 61 2 7 201 2 2 19 0 0 30 After Upgrade Size Difference (MB) (MB) 16819 330 475 0 25 178 0 1662 311 14 2 5 2 11 2 5 1 0 61 2 7 201 2 2 19 0 0 30 0 0 0 0 0 68 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 % Growth 2% 00% 0.06% 0% 0% 0% 4.26% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% APPS_TS_DISCO_OLAP APPS_TS_QUEUES APPS_TS_TOOLS APPS_TS_INTERFACE APPS_TS_SEED XDB APPS_TS_TX_IDX GEOR_TBS NOETIX APPS_TS_DISCO TEMP2 APPS_TS_MEDIA DISCO_PTM5_CACHE B2B_RT APPS_TS_SUMMARY APPS_TS_ARCHIVE APPS_TS_TX_DATA APPS_CALCLIP PORTAL_IDX ORABPEL OLTS_CT_STORE APPS_OMO TEMP1 BAM OLTS_BATTRSTORE APPS_TS_NOLOGGING NON_APPS_TS_DATA Total 10 7201 14 17456 34184 151 98809 2021 0 766 9192 2536 97 38 11694 938 94631 4676 0 11 7 638 10740 6 0 94 66 315414 10 7201 14 17303 34317 151 99286 2021 0 766 9192 2536 97 38 10580 937 94462 4676 0 11 7 638 10740 6 0 267 66 315159 0 0 0 -153 133 0 477 0 0 0 0 0 0 0 -1114 1 -169 0 0 0 0 0 0 0 0 173 0 -254 0% 0% 0% -0.89% 0.39% 0% 0.48% 0% 0% 0% 0% 0% 0% 0.00% -0.97% 0.10% -0.18% 0% 0% 0% 0% 0% 0.00% 0.00% 0% 185% 0% Schema Size Two new schemas have been created for Release 12.2: 1. APPS_NE: For the Non-Editioned file system 2. GHG: For the Oracle Environmental Accounting and Reporting. Only a few of the schemas are listed below show significant change in the size after the upgrade. Schema Name AP APPLSYS APPS APPS_NE AR BSC CSC CSI EGO INV 12.1.3 Size (MB) 12.2 Size (MB) Difference % Difference 4360.63 38724.44 2395.63 4379.5 39377.81 1765.38 95.25 90567.63 285.69 183.5 6036.94 2690.44 11658.13 18.88 653.37 -630.3 95.25 -141.8 -171.8 -49.25 72.5 34.5 175.57 0.43 1.69 -26.31 Not applicable 90709.44 457.44 232.75 5964.44 2655.94 11482.56 Not applicable -0.16 -37.55 -21.16 1.22 1.3 1.53 JTF MSC OKC PJI SYS XDO XLA 30272.75 5120.44 2969.19 936.25 19893.52 1477.13 3059.06 30340.38 5153.94 3037.69 969.38 20819.65 1626.38 3134 67.63 33.5 68.5 33.13 926.13 149.26 74.94 0.22 0.65 2.31 3.54 4.66 10.1 2.45 Release 12.2 Upgrade Best Practices Recommended initialization (init.ora) parameter settings for the upgrade. • It is recommended to follow AWR pool advisories for optimizing the SGA and PGA size in the Release 12.2 upgrade. • An upgrade involves tens of concurrent sessions. Suitable starting values are: 1. log buffer = 30 to 100 Mb 2. shared pool = 1 to 4 GB 3. pga target = 3 to 20 GB 4. buffer cache = 5-50 GB (multi-GB) • You should ensure that SGA and PGA are allocated appropriately within the available memory of the system. If set higher it may cause excessive paging, or even swapping. • Set “job_queue_processes = No. of CPUs” – adobjcmp.sql (Phases: plb+90 and last+63) This parameter is set at the beginning of the upgrade; it will be most useful in the phases mentioned above. • Setting parallel_max_servers to 2 x no. of CPUs will help with large index creation, statistics gathering, and some large upg+ phase jobs. • In order to maximize multiblock I/O sizes, you should remove "db_file_multiblock_read_count" (if specified). • Ensure you reset the following init.ora parameters after completion of the Release 12.2 upgrade. o recyclebin o parallel_max_servers o job_queue_processes Isolate post-upgrade concurrent programs to a separate manager queue. The downtime portion of the upgrade involves the automatic submission of 95 distinct concurrent programs. For a list of the programs, refer to the "Required Post-Upgrade Jobs" section of this document. In several cases, these programs will run in multiple threads, so the total number of concurrent requests that form part of the post-upgrade step is much higher. These programs will be picked up and executed by the concurrent manager once the system is up, and as such their execution will be mixed with the execution of ongoing concurrent jobs in the system. To improve manageability of this situation, you can define a separate concurrent manager queue to process only these programs. This can be achieved by using inclusion and exclusion rules to prevent other manager queues (such as the standard manager) from picking up these requests, and cause this new manager queue to only process these requests. Doing this allows you to control the number of target processes allocated to these post-upgrade concurrent programs, including dynamically with the use of work shifts. For additional details about configuring new manager queues, target processes, inclusion/exclusion rules, and work shifts, refer to Oracle E-Business Suite Setup Guide from the Onine Documentation Library. Before you shut down all services as part of the “Perform the Upgrade“ tasks mentioned in the Upgrade Guide, you need to deactivate all Request Processing managers. This is needed so that when the services are brought up at the end of the upgrade, the existing Request Processing Managers do not start picking up and executing requests. After the upgrade is complete, you need to perform the following steps to create new program type and managers to run the post-upgrade requests. To create a new program type, and include a program in that program type: 1. Log in to Oracle E-Business Suite as SYSADMIN, and navigate to System Administrator > Concurrent : Program > Type Name: R12PUPT Application: System Administration Description: Program Type for Post Upgrade Request 2. Include programs from the above list into this new program type. 3. Navigate to System Administrator >Concurrent: Program >Define 4. Search for Program "Refresh Materialized Views"Type (choose):R12PUPT 5. Do the same for other programs in the list. To exclude a program type from the Standard Manager: 1. Log in to Oracle E-Business Suite as SYSADMIN and navigate to System Administrator > Concurrent : Manager >Def ine 2. Search for Standard Manager. 3. Click on Specialization Rules. 4. Go at the bottom of the list to add a new record. 5. Include/Exclude (choose): Exclude Type(choose): Request Type Application(choose): System Administration Name(choose): R12PUPT 6. Do the same for all other Request Processing managers. To create a new concurrent manager and include a program type: 1. Log in to Oracle E-Business Suite as SYSADMIN and navigate to System Administrator > Concurrent: Manager > Define 2. Check the Enabled box. 3. Enter the following to create the new concurrent manager: Manager: R12_Post_Upgrade Short Name: R12PU Application(choose): Application Object Library Description: New manager queue for R12 post upgrade requests Type(choose): Concurrent Manager Cache Size: 1 Program Library Name(choose): FNDLIBR Specialization Rules: Include/Exclude(choose): Include Type(choose): Request Type Application(choose): System Administration Name(choose): R12PUPT Work Shifts: Work Shift (choose): Standard Processes: Enter a Value, for example 4 Sleep Second: 30 After completing the above steps, activate all "Request Processing Managers" that you previously deactivated. Note that after all these post-upgrade requests run, you should revert the exclusion/inclusion of managers and programs to the original state, removing the new program type and new manager. This is necessary as some of the programs in the list may need to run in the future, as part of the regular system batch processing requirements rather than part of the upgrade. Determine optimal number of workers for upgrade. While monitoring CPU usage and I/O response times, you should test a range of values between 1 x No. of CPUs to 1.5 x No. of CPUs, . The goal is to maximize the number of workers, while keeping CPU usage below 100% (ignoring occasional spikes) and I/O response times below 10-15 milliseconds (again, ignoring occasional spikes). Required Post-Upgrade Jobs Below is a list of of the concurrent programs that form part of the required post-upgrade steps. These concurrent programs are automatically submitted during the regular downtime upgrade, so no manual submission is required. These jobs took 7 hours for the 456 GB database upgrade, using 30 target processes in the concurrent manager. Application Name Concurrent Program Name User Concurrent Program Name Elapsed Time (min.) Total Requests Advanced Benefits (BEN) Advanced Product Catalog (EGO) Advanced Supply Chain Planning (MSC) Application Implementatio n (AZ) Application Object Library (FND) Application Object Library (FND) Application Object Library (FND) BENCWBMU Compensation Workbench Model Upgrade Process User-Defined Attribute Group Views (AGV) Compilation Program 0.10 1 0.57 1 MSCREFMV Refresh Materialized Views 0.70 2 AZR12UPGRADE iSetup R12 Upgrade Selection Sets Column 0.47 3 FNDFIXRN Fix FND role names 0.10 1 DIAGPATCHINGCP Diagnostics patching CP 12.60 9 FDFVGN Flexfield View Generator 135.10 985 EGOCPAGV Application Object Library (FND) Application Object Library (FND) Application Object Library (FND) Application Object Library (FND) Application Object Library (FND) Application Object Library (FND) Application Object Library (FND) Application Object Library (FND) Application Object Library (FND) Application Object Library (FND) Application Object Library (FND) Application Object Library (FND) Application Object Library (FND) Application Object Library (FND) Application Object Library (FND) Applications DBA (AD) Asia/Pacific Localizations (JA) Collections AFLOBBLD Rebuild Help Search Index 16.72 1 FNDIRLPP iRep Loading Post Processor 20.72 5 FNDRSSTG Request Set Stage 0.87 3 FNDSCMPI Compile Security 0.40 1 FNDLOAD Generic Loader 13.05 5 FNDIRLOAD FNDIRLOAD 24.70 5 FNDRSSUB Report Set 1.00 2 FNDWFAASTATCC Workflow Agent Activity Statistics Concurrent Program 1.98 10 FNDWFWITSTATCC Workflow Work Items Statistics Concurrent Program 2.75 10 FNDWFDSURV Workflow Directory Services User/Role Validation 0.88 2 FNDWFMLRSTATCC Workflow Mailer Statistics Concurrent Program 0.70 10 FNDWFNTFCU Workflow Update Notification Action Comments 1.50 6 FNDWFDSRHP Workflow Role Hierarchy Propagation 0.27 3 FDFCMPN Compile Non-Compiled Flexfields 56.33 7 FNDWFBES_CONTROL_QUEUE_CLE ANUP Workflow Control Queue Cleanup 0.02 2 ADDRPOBS Drop obsolete products schema 51.62 204 JAINDFOBS India - Remove India Localization contexts from DFFs 0.07 3 IEX_COLLECTORS_TO_RESOURCE IEX: AR Collectors To Resource 0.88 1 (IEX) Contracts Core (OKC) Contracts Core (OKC) Human Resources (PER) Human Resources (PER) Human Resources (PER) Human Resources (PER) Incentive Compensation (CN) Incentive Compensation (CN) Incentive Compensation (CN) Incentive Compensation (CN) Incentive Compensation (CN) Incentive Compensation (CN) Incentive Compensation (CN) Incentive Compensation (CN) Incentive Compensation (CN) Inventory (INV) Labor Distribution (PSP) Labor Distribution OKCARTCRTCTX Collectors Create Clause Text Index 0.18 1 OKCCHRCRCTX Create Contract Header Text Index 0.27 1 PAY_GEN_UPG Generic Upgrade Mechanism 2.03 1 PER_FND_SESSIONS_CLEANUP Remove obsolete sessions from fnd_sessions 0.53 12 HRTCAUNMP Unmerge contacts from TCA (TCA purge only) 3.62 1 IRC_CREATE_NOTIFICATION iRecruitment Create Notification Preferences 0.07 8 CN_R1212_CNCMAUPD CN_R1212_CNCMAUPD 1.35 48 CN_R1212_CNCMHUPD CN_R1212_CNCMHUPD 0.30 48 CN_R1212_CNSRPTXN CN_R1212_CNSRPTXN 0.35 48 CN_FORMULA_GEN Generate Formula Packages 7.55 3 CN_COLLECT_GEN Generate Collection Packages 1.90 3 CNUPGPMTRXWRKER Upgrade payment transactions Worker 0.05 10 CNUPGPMTCLWRKER Upgrade payment commission lines Worker 0.05 10 CNUPGPMTRXMGR Upgrade transactions for AutoUpgrade or AutoPatch 0.08 1 CNUPGPMTCOMMTRXMGR Upgrade commissions and transactions for AutoUpgrade or AutoPatch - Manager Items Data Scripts Execution 0.12 1 4.45 6 PSPERUPB PSP: Migrate to OAFramework Effort Reporting 0.03 1 PSPUPREP PSP: Post Upgrade Bad Data Report 0.82 1 INVIDSEP (PSP) Learning Management (OTA) Learning Management (OTA) Lease and Finance Management (OKL) Marketing (AMS) Marketing (AMS) Oracle iProcurement (ICX) Payables (SQLAP) Payables (SQLAP) Payments (IBY) Payroll (PAY) Purchasing (PO) Purchasing (PO) Receivables (AR) Receivables (AR) Receivables (AR) Receivables (AR) Receivables (AR) Receivables (AR) Receivables (AR) Receivables (AR) Receivables (AR) Receivables (AR) Receivables OTARPUPG Upgrade Log Report 1.30 1 OTATZUPG Update Time Zone 0.23 1 OKLSVPTH Service Passthrough Report 0.80 1 AMSUPGHISTORYRECSWKR Upgrade metrics history records process - Worker Upgrade metrics history records process - Manager Catalog Child Data Exceptions Report 1.05 64 1.33 8 3.93 1 0.07 1 APWUPRD Update Expense Status Code in Expense Report Headers Upgrade Report Distribution 0.07 8 IBYUPGCP iPayment FP.G Upgrade Program 0.00 2 PYUSLV POXUPMGR Payroll Worker Process Purchasing Post-Upgrade Process 1.47 0.05 4 1 POXUPWKR Purchasing Post-Upgrade Sub Process Third Party Data Integration Update 0.00 3 0.47 2 1.75 20 HZ_GNR_DYNMK_PKG_GEN Geography Name Referencing Worker GNR Dynamic packages Generation 0.75 6 ARHDCH DQM Sync Import Child Program 0.23 4 ARHSUFXS Data Fix for Party Site Use 0.07 1 ARHDQSYN DQM Synchronization Program 0.32 1 ARHMAILS Mailstop Migration Program 0.03 1 ARHDQCMAL DQM Compile All Rules 17.60 3 ARHDNBCH Create DNB Hierarchy 0.07 1 ARHDQM DQM Staging Program 0.42 3 ARHDQMPP DQM Parallel Sync Index Parent 0.10 1 AMSUPGHISTORYRECSMGR ICXCUCDER APWUPDST HZ_THIRD_PARTY_UPDATE ARHGNRW (AR) Receivables (AR) Regional Localizations (JG) Service Contracts (OKS) Service Contracts (OKS) Service Contracts (OKS) Service Contracts (OKS) Service Contracts (OKS) Service Contracts (OKS) Service Contracts (OKS) Service Contracts (OKS) Service Contracts (OKS) Service Contracts (OKS) Service Contracts (OKS) Service Contracts (OKS) Service Contracts (OKS) Service (CS) ARHDQMPC JGZZVATTRXUPG Program DQM Parallel Sync Index Child Program EMEA VAT: Transaction Upgrade 0.02 4 0.95 6 OKSUHAFMGR Service Contracts Update History Annualized Factor Manager 0.28 1 OKS_TAX_MIGRATION Service Contracts eBTax Migration 0.05 1 OKSAQUMG Service Contracts ASO Queue Migration 0.58 1 OKSUHAFWKR Service Contracts Update History Annualized Factor Worker 0.02 4 OKSUBAFMGR Service Contracts Update Base Annualized Factor Manager 0.20 1 OKS_REPROCESS_SUBREQUEST Service Contracts Subrequest for Reprocessing Rule Data 0.02 1 OKSUBAFWKR Service Contracts Update Base Annualized Factor Worker 0.02 4 OKSLAUNCHKWF Launch Process Workflow for existing Service Contracts 0.10 1 OKSTAXHMGR Service Contracts Update History Tax Columns Manager 0.23 1 OKSTAXHWKR Service Contracts Update History Tax Columns Worker 0.00 4 REPROCESS_SYNC_RULE_DATA Service Contracts Reprocess/Synchronize Rule Data 0.23 1 OKSTAXBMGR Service Contracts Update Base Tax Columns Manager 0.20 1 OKSTAXBWKR Service Contracts Update Base Tax Columns Worker 0.00 4 CSSRAUDCMAN Service: Upgrade for Audit Creation Record - Manager Service: General Upgrade for Audit Records - Worker Service: Upgrade for Audit Creation Record - Worker 0.03 1 0.00 3 0.02 3 Service (CS) CSSRAWGEN Service (CS) CSSRAUDCWKR Service (CS) CS_KB_SYNC_SOLUTIONS_INDEX Knowledge Management Solution Index Synchronization Knowledge Management Statement Index Synchronization Service: General Upgrade for Audit Records - Manager Charges: Upgrade for Line Number Manager Charges: Upgrade for Line Number Worker Service Request: Synchronize Text Index Program Knowledge Management Category Index Synchronization Refresh Materialized View 0.73 2 Service (CS) CS_KB_SYNC_STATEMENTS_INDEX 0.12 1 Service (CS) CSSRAMGEN 0.03 1 Service (CS) CSCHGLINENOMGR 0.03 1 Service (CS) CSCHGLINENOWKR 0.02 3 Service (CS) CS_SR_SYNC_TEXT_INDEX 1.08 2 Service (CS) CS_KB_SYNC_CATEGORIES_INDEX 0.05 1 Trade Management (OZF) Trade Management (OZF) Warehouse Management (WMS) XML Publisher (XDO) Total OZFEARNMV 0.95 3 OZFMIGUTLREC Migrate Accruals for Multi Currency Changes 0.55 9 WMSGRULE Generate All Rules 1.05 2 XDOTMGEN XML Publisher Template ReGenerator 23.53 1 431.05 1692 Sample Upgrade Timings Before applying the main 12.2 upgrade driver (u10124646.drv), AD patch 10117518 must be applied on both file systems (fs1 and fs2). As a result of applying this patch, two new schemas (APPS_NE and GHG) will be created for use in Release 12.2. The overall time taken to apply the patch was about two hours. 1. Time taken for creating the users and granting privileges to them was 20 minutes. 2. Compiling JSPs (using perl) took 1 hour 30 minutes. 3. The following database activity timing information is for the Release 12.2 AD patch (10117518), which took 5.5 minutes. Phase first con seq tab tbm pls plb dat dfr en Time (sec.) 24 55 2 22 18 5 19 11 1 0 last Invoker Total 89 90 336 (5.5 min.) The following timing information is for the Release 12.2 main upgrade driver (u10124646.drv) only Phase-wise breakdown of time for upgrade: Phase first con seq tab tbm os pls vw plb pda daa upa pdt dat upg dfr en last Total Time (in min.) 0.31 2.75 1.15 4.83 16.98 0.03 7.11 3.14 28.04 2.75 21.96 0.08 2.57 34.4 102.2 1.06 8.00 98.04 335.4 min. (5.59 hours) The upgrade is a very I/O intensive process, moving, updating, and indexing large amounts of data. As such, I/O subsystem performance and bandwidth are critical to minimizing the amount of downtime. During many Release 11i to Release 12 customer upgrades, poor I/O sub-system performance was found to be one of the major sources of upgrade performance service requests. Therefore, during the upgrade testing process you should monitor I/O performance and correct any issues found. As a reference, shown below are the IOPS rates and bandwidth measurements that were obtained while testing the upgrade with the 456 GB database. The numbers were calculated based on data from AWR and OS monitoring utilities. IOPS: • • • Maximum IOPS value is 478.9. During upgrade, IOPS range is from 142 to 478.9. Max value is reported during large indexes creation phase. High IOPS value (140 to 478) is seen during UPG phases. I/O Read bandwidth: • Maximum I/O Read bandwidth (in MB): 10.23 • During upgrade I/O Reads ( in MB) per second range is from 1.44 to 10.23 • Max value is reported during dat and upg phase. I/O Write bandwidth: • Maximum I/O Write bandwidth (in MB): 3.09 • • During upgrade I/O Writes per second range is from 0.9 to 3.09 MB. Max value is reported during dat phase. I/O Response Times ( in milliseconds): • Below 20ms for the vast majority of AWR snapshot periods. Higher values were present with spikes above 50ms during the Large indexes create phase and some upgrade phases. Timing information for Release 12.2 Edition-Based Redefinition (EBR) enablement (Patch 13543062) Timing reported here is only for application of the EBR enablement patch. Apart from patch application, many other steps are performed in enabling EBR. The time these take will depend upon the customizations you have. Refer the the following document for the steps needed to enable EBR: • Oracle E-Business Suite Upgrade Guide Release 12.0 and 12.1 to 12.2, Part No. E48839 (Section 4, Oracle EBusiness Suite Online Patching) You need to register all your custom schemas, following the procedures in the above document. Phase-wise breakdown of time needed for the upgrade: Phase Time (min.) first con seq tab tbm pls last Total 57.13 0.46 0.8 0.25 35.13 2.65 3.53 100 min. (1.6 hours) References: • • • Oracle E-Business Suite Release 12.2 Information Center (My Oracle Support Knowledge Document 1581299.1) Database Initialization Parameters for Oracle E-Business Suite Release 12 (My Oracle Support Knowledge Document 396009.1) Oracle E-Business Suite Recommended Performance Patches (My Oracle Support Knowledge Document 244040.1) Change Log Date 04-Nov-2013 Description • Initial publication.