11i Cloning Panel Panel Moderator: Sandra Vucinic Panel Members • • • • • Lynne Paulus - Fair Isaac & Company John Stouffer – Solution Beacon James Lui - Employers Insurance Group Steve Noynaert – RingMaster Software Sandra Vucinic – VLAD Group, Inc. Panel Contributors: • Eric Cole - National Institutes of Health Agenda • • • • • • • • • Available cloning methods Differences between methods Cloning Options available for each method Advanced Rapid Clone Options Future Cloning Options Rapid Clone – Tips, tricks and lessons learned Case Study - 11i Cloning at NIH Custom cloning method Automated third party tools How can I clone an Oracle Applications 11i System? • Available methods: – Cloning using AD Clone utility (adclone.pl) – Cloning using Rapid Clone – Custom cloning methods – Automated tools What are differences between available cloning methods? • Cloning using adclone.pl method – Applicable for all 11i releases up to 11.5.5 that are not AutoConfig enabled – adclone.pl method requires that you run Rapid Install and apply RDBMS and Technology Stack patches to target system • Cloning using Rapid Clone – Applicable for all 11i systems that have migrated to AutoConfig and enabled Rapid Clone Which cloning method is best for your environment? • The best method depends on: – Version of applications: • Prior to version 11.5.7 (adclone.pl) • Version 11.5.7 and higher (Rapid Clone) – Version of Database: • 8i (adclone.pl and Rapid Clone) • 9i (Rapid Clone only) Future Cloning Options • Fully automated RAC cloning with Rapid Clone (available now !!!!!) • Cloning using Oracle Maintenance Wizard version 1.10 and Applications Cloning Assistant for 11.5.10 (planned to be released this summer) What cloning options are available for each method? Singlenode to Singlenode Multinode to Multinode Singlenode to Multinode Cloning 11i Supported Supported Not apps using Available adclone.pl Multi-node to Singlenode Not Available Cloning 11i Supported Supported Supported Supported apps using Rapid Clone Advanced Rapid Clone Options • Cloning a single-node to a multi-node system • Cloning a multi-node system to a multi-node system • Adding a new node to an existing system • Cloning “shared appl_top” • Cloning a multi-node to as single-node system – Follow the procedure for Merging Existing APPL_TOPs doc id: 233428.1 • Use Rapid Clone from OAM Rapid Clone • Advantages of Rapid Clone – All Tech stack is copied unlike adclone • • • • Includes RDBMS, 8.0.6 and iAS Exact copy of Source Environment Able avoid re-applying tech stack patches Create new environment without running Rapid Install • Disadvantage of Rapid Clone = Numerous Steps, not a single program Rapid Clone Tips - RMAN • ‘Cloning Oracle Applications Release 11i with Rapid Clone’ (doc # 230672.1) • Integration with RMAN: – ‘Copy database tier file system’ (Section 2-Step 2b) • Copy rman files, Delay $ORACLE_HOME copy – – – – – Use RMAN, ‘duplicate’ source DB Pre-create ‘tempfile’ for TEMP tablespace Shutdown Target DB after RMAN complete Complete Step 2b, copy Source $ORACLE_HOME ‘Configure target system database server’ (Section 2Step 3a) Rapid Clone Tips - Patching • Rapid Clone and AutoConfig tightly coupled – Patches to either destabilize cloning – Retest Clone method after patches to either • Avoid patches to either unless time for testing – We did not patch either for 1 year after 11.5.8 upg • After patches to either, build new target context file – Do not reuse existing since usually new entries – Use ‘Port Pool’ logic when running adcfgclone.pl – RDBMS context file rebuilt with adbldxml.sh Rapid Clone Tips…. • RDBMS AutoConfig, Rapid Clone may/will clobber files – init_<sid>.ora, tnsnames.ora, sqlnet.ora • Avoid direct changes to AutoConfig generated files • Take advantage of ‘ifiles’, not clobbered • Give ‘ifiles’ meaningful names in RDBMS context file – init_<sid>_ifile.ora, tns_<sid>_ifile.ora Rapid Clone Tips… • Check log files (some not documented) – /tmp/adcfgclone*, $ORACLE_HOME/install/make*log • Unset environment before run adcfgclone.pl – Problems if point to Source by accident – Call <sid>_<box>.env instead of APPSORA.env • Reuse Target context file (except after some patches) – perl ./adcfgclone.pl dbTier <context filename> – perl ./adcfgclone.pl appsTier <context filename> • Develop detailed Rapid Clone cookbook Case Study – 11i Cloning at National Institutes of Health • Five year phased implementation – Multiple, nightly clones (approximately 100/year) – Automated/unattended – cron scripts – Releases 11.5.3 to 11.5.9 – Multiple servers, accounts, mountpoints, ports – Multiple database environments (Oracle/Gelco) Case Study – 11i Cloning at National Institutes of Health Environment Statistics • Oracle E-Business Suite Release 11i - Production – 550 Application Users – 200 Database Users – 225,000 Database Objects – Total Environment Size – 210 GB Disk Allocated, 179 GB Used – 125 GB Database – 30 GB Application Code – 9 GB Patches – Nine Production and Production Support Environments (Production, Quality, Test, Training, Development, Sandbox, Patch, Year-end, Upgrade) • Gelco Travel Manager 8.10 – 33,000 Traveler Users – 2,700 Application Users (Planners, Reviewers, Approvers, Administrators, System Administrators, Super Users) – 10 Database Users – 3,900 Database Objects – Total Environment Size – 30 GB Disk Allocated, 24 GB Used – 14 GB Database Size – 7 GB Application Code – Nine Production and Production Support Environments (Production, Quality, Test, Training, Development, Sandbox, Patch, Year-end, Upgrade) • Total Environment – Production, Development and Project Phases – 44 CPUs – 101 GB RAM – 10 TB Disk Space Case Study – 11i Cloning at National Institutes of Health – Weekly cold backup using gtar/compress to separate disk mountpoint – also used for disaster recovery – Untar cold backup of source environment into $ORACLE_BASE of target environment • gtar –zxvf <sid>_weekly_diskbackup.091304.gz – Supply source and target environment information • Hostname • SID • Starting port – Post clone tasks • Scramble privacy act data • Remove old directories • Implement customizations - WebADI – AutoConfig enabled but not executed during clone process up through 11.5.9 Service Oriented Architecture Presentation Layer STTY StdOUT Service Layer Process Layer SQL Korn Shell Perl OS Java Business Layer SQL*Plus FNDCPASS CONCSUB / startmgr adstrtal.sh / adstpall.sh dbstart / dbshut mailx <java things> Persistence Layer SysAdmin Send e-mail notifying last known users of refresh. Developer Export any indevelopment custom data objects. DBA Process Layer – Part 1 of 2 Shutdown DB remove current datafiles. Update columns reflecting change in DB identity(SID). Restore datafiles from Production to Development Recreate controlfile and open DB. End-date nonauthorized Application users Force FND_USER password changes. DBA Developer SysAdmin Process Layer – Part 2 of 2 Reset SYSTEM and SYS DB passwords. Register APPLSYS-level password changes. Update ICM settings, Profile Options, restart ICM/GSM. Disable / Enable Triggers, Alerts and Concurrrent Programs Restore any custom data objects. Update ORApwd (remote_login_ passwordfile) Send e-mail notice of completed refresh. Architecture Oracle Applications 11i RingMaster Hot or cold Backup Apache PROD RingMaster 9i/10g Repository + Agent SAN Real time Clone NAS STAGING Client: IE TEST Backup based Clone SSH SCP FTP RCP Telnet Mozilla Tar Gzip Compress RMAN Netscape NetApp EMC Hitachi Cloning Prod to Test Pre-clone steps Data transfer (Threaded) PROD TEST Post-clone steps Single click Scheduling request Reports Automatic Audit Trail Downtime, who, what, when? Benefits • • • • Single click - Component complete Reduce time spent on cloning Clone more frequently SOX compliant (audit trail, reports) Q&A Thank You! Lynne Paulus - LynnePaulus@FairIsaac.com John Stouffer – jstouffer@solutionbeacon.com James Lui - jlui@jlui.net Steve Noynaert - snoynaert@ringmastersw.com Sandra Vucinic - sandrav@vladgroup.com