Oracle Database AutoUpgrade Guide September, 2020 TABLE OF CONTENTS INTRODUCTION................................................................................................................................ 3 Oracle 19c Software Installation guide……………………………………………………………4 Oracle 19c Database Upgrade Hands-On Lab…………………………………………………….5 STEP 1 :: Preparing to Upgrade Oracle Database……………………………………………… 11 STEP 2 :: Upgrade and migrate to Container databases …………..…………………………….24 STEP 3 :: Unplug and Plug PDB-Upgrade Pluggable Databases......……………………………28 Oracle SEHUB pg. 2 INTRODUCTION Oracle Database 19c includes new features and enhancements, as well as a long-term support commitment that make it an attractive upgrade target for existing Oracle databases. Moving to Oracle Database 19c may be part of an effort that includes moving to newly purchased server hardware, migrating to different storage architectures, such as Oracle Automatic Storage Management, changing the database character set etc. There are different ways of upgrading to the latest release of Oracle database and Oracle provides multiple methods to upgrade: o Database Upgrade Assistant (DBUA) o Manual Upgrade o Transportable Tablespaces o Data Pump export/import o Oracle GoldenGate o AutoUpgrade In this Lab Guide we are going to discuss today AutoUpgrade Utility Upgrade The act of upgrading an Oracle Database involves modifying the data dictionary to be compatible with a newer version of Oracle Database software. Typical actions that may be part of a database upgrade include: » Adding, dropping, or modifying columns in system tables and views » Creating or modifying system packages or procedures » Creating, modifying, or dropping database types, users, roles, and privileges » Modifying seed data that is used by Oracle Database components All of these actions affect the data dictionary of your database. They do not affect the data stored in your user or application tablespaces. Therefore, the sheer volume of data stored in your Oracle database has little or no bearing on a database upgrade. Oracle SEHUB pg. 3 Oracle 19c Software Installation guide 1. https://www.youtube.com/watch?v=Ra8wHDTV3eY Oracle SEHUB pg. 4 Oracle Database 19c (19.3.0) Upgrade and Migration hands-on Lab https://www.oracle.com/downloads/community/vts-hands-on-labs-downloads.html INSTALLATION Review the hands-on lab instructions them to understand what is included in the larger zip file download. Select the instruction modules under the Hands-on Lab tab Download the VM image zip files. There are a total of 13 files: Setup Information Space and memory requirements – and login Space requirements: 64GB hard disk space Memory requirements: 8GB RAM minimum CPU requirements: Dual-core CPU Inside the lab Oracle SEHUB pg. 5 Username and Passwords All passwords are set to: oracle This applies to the OS user root and oracle as well as to the database accounts for SYS and SYSTEM. Switching environments You can switch between the different environments by typing: . This reads: dot <space/blank> upgr upgr From the picture below you can see the available environments and the shortcuts to switch between any of them. If you need to alter these files or create new ones, find them in /usr/local/bin/ and alter them with root privileges. There is an additional environment variable $OH19 defined for convenience. This points to the 19c Oracle Home, and will be used several times in the lab. Oracle SEHUB pg. 6 HOL overview picture This is what you’ll find inside the hands-on lab virtual image. It contains 3 different Oracle Homes: Oracle 11.2.0.4 – /u01/app/oracle/product/11.2.0.4 Oracle 12.2.0.1 – /u01/app/oracle/product/12.2.0.1 Oracle 19.5.0 – /u01/app/oracle/product/19 The lab contains also 5 different databases: – 11.2.0.4 database (non-CDB) FTEX – 11.2.0.4 database (non-CDB) DB12 – 12.2.0.1 database (non-CDB) CDB1 – 12.2.0.1 database (CDB) CDB2 – 19.5.0 database (CDB) UPGR Oracle SEHUB pg. 7 VERY IMPORTANT: If the login screen does not appear but instead you get a BLACK screen after the reboot process, then please check this link here: https://www.dedoimedo.com/computers/virtualbox-3d-accel-blackscreen.html In the cases we encountered during the first run of the labs, two people struggled with it. The: WaylandEnable=false parameter brought the solution. I set this for the downloadable image already but people who get the lab via a USB drive in the early stages may miss this setting. Keyboard layout and preferences If you need to change the keyboard settings (preconfigured is English-US (default) and German) please do the following: 1. 2. Oracle SEHUB pg. 8 3. On the Linux desktop, click on the PENGUIN icon in the left upper corner, then choose “SYSTEM SETTINGS”, then under HARDWARE: “INPUT DEVICES”. The first item is “KEYBOARD”. Choose the “LAYOUT” tab, then click on “CONFIGURE LAYOUTS” and “ADD” the keyboard you’d want to add. You can then always switch the current keyboard layout by clicking on the “en” (or “de“) characters in the menu bar. As the default is set to “English-US" you may change the default as otherwise every xterm will always open with English keyboard settings at first. You can always alter the keyboard on the fly. There’s no need to open a new xterm to change the setting temporarily for this xterm. Inside the lab Username and Passwords All passwords are set to: oracle Oracle SEHUB pg. 9 This applies to the OS user root and oracle as well as to the database accounts for SYS and SYSTEM. Switching environments You can switch between the different environments by typing: . upgr This reads: dot <space/blank> upgr From the picture below you can see the available environments and the shortcuts to switch between any of them. If you need to alter these files or create new ones, find them in /usr/local/bin/ and alter them with root privileges. There is an additional environment variable $OH19 defined for convenience. This points to the 19c Oracle Home, and will be used several times in the lab. It contains 3 different Oracle Homes: Notation conventions This is VERY IMPORTANT for the entire lab. This color means: Execute on the OS level in an xterm: example: RED means you’ll type this in an xterm This color means: Execute in SQL*Plus: example: GREY means you’ll type this in SQL*Plus And the third color means: Execute in the job console upg>: example: BLUE means you’ll type this in the autouopgrade job console This way you can copy&paste from the instructions if necessary without having trouble with prompts. Oracle SEHUB pg. 10 STEP 1: Preparing to Upgrade Oracle Database 1. Preparation The only task you’ll have to do when using the AutoUpgrade: You need to prepare a config file for the database(s). The environment variable $OH19 is created only for your convenience. It points always to the Oracle 19c Home. . upgr java -jar $OH19/rdbms/admin/autoupgrade.jar create_sample_file config This tells you that the sample file has been created at: Created sample configuration file /home/oracle/scripts/sample_config.cfg You will need to edit it – and then pass it to the AutoUpgrade utility. Created sample configuration file /home/oracle/sample_config.cfg Open the file /home/oracle/sample_config.cfg in your preferred editor (text or graph mode) Text mode: vi /home/oracle/scripts/sample_config.cfg Graphical mode: Oracle SEHUB pg. 11 kwrite /home/oracle/scripts/sample_config.cfg & Adjust the following things: Generated standard config.cfg Make the following adjustments: #Global configurations #Autoupgrade's global directory, ... #temp files created and other ... #Global configurations #Autoupgrade's global directory, ... #temp files created and other ... #send here #send here global.autoupg_log_dir=/home/oracle/upg_logs global.autoupg_log_dir=/default/... # # Database number 1 # upg1.dbname=UPGR upg1.start_time=NOW # upg1.source_home=/u01/app/oracle/product/11.2.0.4 upg1.target_home=/u01/app/oracle/product/19 # Database number 1 upg1.sid=UPGR upg1.log_dir=/home/oracle/logs upg1.upgrade_node=localhost # upg1.target_version=19 upg1.restoration=no upg1.dbname=employee upg1.start_time=NOW upg1.source_home=/u01/... Oracle SEHUB pg. 12 upg1.target_home=/u01/... upg1.sid=emp upg1.log_dir=/scratch/auto upg1.upgrade_node=hol1.localdomain upg1.target_version=19 #upg1.run_utlrp=yes #upg1.timezone_upg=yes Then save the file and name it as UPGR.cfg in /home/oracle/scripts. If you saved it under its original name, sample_config.cfg, rename it as shown below: mv /home/oracle/scripts/sample_config.cfg /home/oracle/scripts/UPGR.cfg Oracle SEHUB pg. 13 2. ANALYZE Phase It is best practice to run AutoUpgrade in analyze mode at first. Once the analyze phase is passed without issues, the database can be upgraded automatically (the below command is a one-line command!). . upgr java -jar $OH19/rdbms/admin/autoupgrade.jar -config /home/oracle/scripts/UPGR.cfg -mode analyze You will see this output: You will see this output: Autoupgrade tool launched with default options +--------------------------------+ | Starting AutoUpgrade execution | +--------------------------------+ 1 databases will be analyzed Type 'help' to list console commands upg> You can monitor the analyze phase in the upg> job console with: Oracle SEHUB pg. 14 lsj status -job 100 Shortly after, the console will reply: upg> Job 100 completed ------------------- Final Summary -------------------- Number of databases [ 1 ] Jobs finished successfully [1] Jobs failed [0] Jobs pending [0] ------------- JOBS FINISHED SUCCESSFULLY ------------- Job 100 FOR UPGR The database can be upgraded automatically. Oracle SEHUB pg. 15 3. Upgrade When you initiate the upgrade with -mode deploy, the tool will repeat the analyze phase, but add the fixups, upgrade and postupgrade steps. java -jar $OH19/rdbms/admin/autoupgrade.jar -config /home/oracle/scripts/UPGR.cfg -mode deploy You will see this output: Autoupgrade tool launched with default options +--------------------------------+ | Starting AutoUpgrade execution | +--------------------------------+ 1 databases will be processed Type 'help' to list console commands upg> At this point you can monitor the upgrade now – enlarge the xterm‘s width a bit to see no line wraps. Type help on the upg> job console to see an overview of available commands. help upg> help Oracle SEHUB pg. 16 exit // To close and exit help // Displays help lsj [(-r|-f|-p|-e) | -n ] // list jobs by status up to n elements. -f Filter by finished jobs. -r Filter by running jobs. -e Filter by jobs with errors. -p Filter by jobs being prepared. -n Display up to n jobs. lsr // Displays the restoration queue lsa // Displays the abort queue tasks // Displays the tasks running clear // Clears the terminal resume -job // Restarts a previous job that was running status [-job [-long]] // Lists all the jobs or a specific job restore -job // Restores the database to its state prior to the upgrade restore all_failed // Restores all failed jobs to their previous states prior to the upgrade logs Oracle SEHUB pg. 17 // Displays all the log locations abort -job // Aborts the specified job h[ist] // Displays the command line history /[] // Executes the command specified from the history. The default is the last command The most important commands are: – this lists the job number and overview information about each active job. Please note that the job number has now changed for the -mode deploy run. lsj lsj upg> lsj +----+-------+---------+---------+--------+--------------+--------+--------+------------+ |Job#|DB_NAME| STAGE|OPERATION| STATUS| START_TIME|END_TIME| UPDATED| MESSAGE| +----+-------+---------+---------+--------+--------------+--------+--------+------------+ | 101| UPGR|PREFIXUPS|EXECUTING| RUNNING|19/10/17 23:15| N/A|23:15:28|Remaining 3/3| +----+-------+---------+---------+--------+--------------+--------+--------+------------+ Total jobs 2 Oracle SEHUB pg. 18 status -job <number> – this gives you more information about a specific job. It displays you also where the log files are located. status -job 101 upg> status -job 101 Progress ----------------------------------- Start time: 19/10/17 23:16 Elapsed (min): 1 End time: N/A Last update: 2019-10-17T23:16:58.468 Stage: PREFIXUPS Operation: EXECUTING Status: RUNNING Pending stages: 6 Stage summary: SETUP Oracle SEHUB pg. 19 <1 min PREUPGRADE <1 min PRECHECKS <1 min GRP <1 min PREFIXUPS <1 min (IN PROGRESS) Job Logs Locations ----------------------------------- Logs Base: /home/oracle/upg_logs/UPGR Job logs: /home/oracle/upg_logs/UPGR/101 Stage logs: /home/oracle/upg_logs/UPGR/101/prefixups TimeZone: /home/oracle/upg_logs/UPGR/temp Additional information ----------------------------------- Details: +--------+----------------+-------+ |DATABASE| +--------+----------------+-------+ Oracle SEHUB pg. 20 FIXUP| STATUS| | +--------+----------------+-------+ UPGR|DICTIONARY_STATS|STARTED| Error Details: None logs – displays the logs folder logs Autoupgrade logs folder [/home/oracle/upg_logs/cfgtoollogs/upgrade/auto/config_files] logs folder [UPGR][/home/oracle/upg_logs/UPGR] Please open a second xterm tab and go to the logs folder. Oracle SEHUB pg. 21 cd /home/oracle/upg_logs/UPGR/101 Explore the subdirectories, especially /home/oracle/upg_logs/UPGR/101 and below. Check the /home/oracle/upg_logs/UPGR/101/prechecks subdirectory. It contains an HTML file with the preupgrade check overview: cd prechecks firefox upgr_preupgrade.html & Check also the preupgrade.log within the same directory: more upgr_preupgrade.log Now change the directoy and see whether the dbupgrade directory has been created. This usually takes up to 4 minutes until the prechecks and fixups have been completed. You will find the 4 upgrade worker’s logs in cd /home/oracle/upg_logs/UPGR/101/dbupgrade. These 4 subdirectories get created before dbupgrade: prechecks prefixups preupgrade drain Oracle SEHUB pg. 22 You can tail -f especially the main worker’s (ending with 0) log to display the upgrade progress. cd ../dbupgrade tail -f catupgrd*0.log Interrupt the tail command with CTRL+C. Depending on the hardware, the upgrade will take about 25-35 minutes. You don’t have to wait but instead we will do some exercises now with the AutoUpgrade tool. The upgrade will take between 20-40 minutes to complete upg> Job 101 completed ------------------- Final Summary -------------------- Number of databases [ 1 ] Jobs finished successfully [1] Jobs failed [0] Jobs pending [0] ------------- JOBS FINISHED SUCCESSFULLY ------------- Job 101 FOR UPGR Congratulations – you upgraded the UPGR database successfully from Oracle 11.2.0.4 to Oracle 19c. Oracle SEHUB pg. 23 STEP 2: Upgrade and migrate to Container databases In this part of the Hands-On Lab you will now plugin UPGR into CDB2. We could have done this with AutpUpgrade already – you can see this in the OPTIONAL AutoUpgrade exercise (Parameter: target_cdb=CDB2). But we rather decided that you should do these steps manually to understand the implications. CDB2 is a Multitenant Container database. And UPGR will be converted into a PDB, and then become a pluggable database. The key is, that – in order to plugin a non-CDB such as the UPGR database – it has to be upgraded first to the same release as the CDB it gets plugged into. Index 1. Preparation UPGR as non-CDB 2. Compatibility check 3. Plugin Operation 1. Preparation UPGR as non-CDB Switch to the UPGR database in 18c environment: . upgr19 sqlplus / as sysdba Shutdown UPGR and start it up read only: shutdown immediate startup open read only; Create the XML manifest file describing UPGR’s layout and information: Oracle SEHUB pg. 24 exec DBMS_PDB.DESCRIBE('/home/oracle/pdb1.xml'); Shutdown UPGR: shutdown immediate exit Switch to CDB2: . cdb2 sqlplus / as sysdba 2. Compatibility check Ideally you do a compatibility check before you plugin finding out about potential issues. This step is not mandatory but recommended. The check will give you YES or NO. Compatibility check: set serveroutput on DECLARE compatible CONSTANT VARCHAR2(3) := CASE DBMS_PDB.CHECK_PLUG_COMPATIBILITY( pdb_descr_file => '/home/oracle/pdb1.xml', pdb_name => 'PDB1') WHEN TRUE THEN 'YES' ELSE 'NO' END; BEGIN DBMS_OUTPUT.PUT_LINE('Is the future PDB compatible? ==> ' || compatible); END; / If the result is “NO” (and it is NO very often), then don’t be in panic. Check for TYPE='ERROR' in PDB_PLUG_IN_VIOLATIONS. In this case, the result should be “YES“. Oracle SEHUB pg. 25 3. Plugin Operation Plugin UPGR with its new name PDB1 – from this point there’s no UPGR database anymore. In a real world environment, you would have a backup or use a backup/copy to plug in. In our lab the database UPGR will stay in place and become PDB1 as part of CDB2. Please use the proposed naming as the FILE_NAME_CONVERT parameter and TNS setup have been done already. Use the NOCOPY option for this lab to avoid additional copy time and disk space consumption. The show pdbs command will display you all existing PDBs in this CDB2. create pluggable database PDB1 using '/home/oracle/pdb1.xml' nocopy tempfile reuse; show pdbs As you couldn’t do a compatibility check beforehand, you’ll open the PDB now and you will recognize that it opens only with errors. alter pluggable database PDB1 open; Find out what the issue is: column message format a50 column status format a9 column type format a9 column con_id format 9 select con_id, type, message, status from PDB_PLUG_IN_VIOLATIONS where status<>'RESOLVED' order by time; As you can see, a lot of the reported issues aren’t really issues. This is a known issue. Only in the case you see ERROR in the first column you need to solve it. The only real ERROR says: Oracle SEHUB pg. 26 PDB plugged in is a non-CDB, requires noncdb_to_pdb.sql be run. Kick off this sanity script to adjust UPGR and make it a “real” pluggable database PDB1 with noncdb_to_pdb.sql. Runtime will vary between 10-20 minutes. Take a break while it is running. The forced recompilation takes quite a bit. alter session set container=PDB1; @?/rdbms/admin/noncdb_to_pdb.sql Now SAVE STATE. This ensures, that PDB1 will be opened automatically whenever you restart CDB2. Before you must restart the PDB as otherwise it opens only in RESTRICTED mode. shutdown startup alter pluggable database PDB1 save state; alter session set container=CDB$ROOT; show pdbs exit Try to connect directly to PDB1 – notice that you can’t just connect without specifying the service name as PDB1 is not visible on the OS level. sqlplus "sys/oracle@pdb1 as sysdba" exit As alternative you could also use the EZconnect (speak: Easy Connect) sqlplus "sys/oracle@//localhost:1521/pdb1 as sysdba" exit Oracle SEHUB pg. 27 STEP 3: Unplug and Plug PDB-Upgrade Pluggable Databases In this part of the Hands-On Lab you will now unplug an Oracle 12.2.0.1 pluggable database (PDB) from CDB1 and plug it into 19c’s CDB2, including all the necessary steps. Index 1. Preparation work in CDB1 2. Preupgrade.jar and Unplug 3. Plugin 4. Upgrade 1. Preparation work in CDB1 The PDB3 we will use in this part of the lab is created already in CDB1 – but you need to startup CDB1 and PDB3. . cdb1 sqlplus / as sysdba startup alter pluggable database pdb3 open; show pdbs exit 2. Preupgrade.jar and Unplug Run the preupgrade.jar but only on container PDB3 java -jar $OH19/rdbms/admin/preupgrade.jar -c 'pdb3' TERMINAL TEXT This will execute preupgrade.jar only in container “PDB3”. Oracle SEHUB pg. 28 Follow the advice of preupgrade.jar‘s output. You can leave all underscore parameters in Be aware: If you’d remove for instance the _fix_control, you’d remove this setting for the entire CDB. This would affect other PDBs as well which may still remain in the 12.2.0.1 environment. That’s why we will leave all underscores as is. Open a second terminal window (or a new tab in your existing one) and logon at first to CDB$ROOT. Then change to PDB3 to complete the steps recommended by preupgrade.jar. . cdb1 sqlplus / as sysdba alter session set container=PDB3; @/u01/app/oracle/cfgtoollogs/CDB1/preupgrade/preupgrade_f ixups.sql alter session set container=CDB$ROOT; alter pluggable database PDB3 close; alter pluggable database PDB3 unplug into '/home/oracle/pdb3.pdb'; drop pluggable database PDB3 including datafiles; Unplugging into a *.pdp does create a zip archive including all necessary files. It will take 30 seconds or more. shutdown immediate exit 3. Plugin In this step you’ll plugin PDB3 into CDB2. . cdb2 sqlplus / as sysdba Oracle SEHUB pg. 29 At first, you’ll do a compatibility check and find out, why the action is classified as “not compatible”: SET SERVEROUTPUT ON DECLARE compatible CONSTANT VARCHAR2(3) := CASE DBMS_PDB.CHECK_PLUG_COMPATIBILITY( pdb_descr_file => '/home/oracle/pdb3.pdb', pdb_name => 'PDB3') WHEN TRUE THEN 'YES' ELSE 'NO' END; BEGIN DBMS_OUTPUT.PUT_LINE(compatible); END; / If the result is “NO“, check PDB_PLUG_IN_VIOLATIONS for the reason: select message from pdb_plug_in_violations where type like '%ERR%' and status <> 'RESOLVED'; You receive two messages: SQL> select message from pdb_plug_in_violations where type like '%ERR%' and status <> 'RESOLVED'; MESSAGE -------------------------------------------------------------------------------- PDB's version does not match CDB's version: PDB's version 12.2.0.1.0. CDB's vers ion 19.0.0.0.0. DBRU bundle patch 200114 (DATABASE JAN 2020 RELEASE UPDATE 12.2.0.1.200114): Not installed in the CDB but installed in the PDB '19.6.0.0.0 Release_Update 1912171550' is installed in the CDB but no release up Oracle SEHUB pg. 30 dates are installed in the PDB The first one is correct and makes sense. The second and third one can be ignored as it doesn’t matter if PDB3 has a different patch level in 12.2.0.1 – you will upgrade it to 19c anyway. You may read a bit more about PDB_PLUG_IN_VIOLATIONS here. Plugin the PDB3, the open it in UPGRADE mode: create pluggable database pdb3 using '/home/oracle/pdb3.pdb' file_name_convert=('/home/oracle', '/u02/oradata/CDB2/pdb3'); alter pluggable database PDB3 open upgrade; exit 4. Upgrade PDB3 As final action, as a PDB has its own data dictionary, you need to upgrade PDB3 now. . cdb2 dbupgrade -c 'PDB3' -l /home/oracle/logs -n 2 Once the upgrade has been completed, you need to recompile and run the postupgrade_fixups.sql as usual: . cdb2 sqlplus / as sysdba alter session set container=PDB3; startup @?/rdbms/admin/utlrp.sql @/u01/app/oracle/cfgtoollogs/CDB1/preupgrade/postupgrade_ fixups.sql alter session set container=CDB$ROOT; show pdbs Oracle SEHUB pg. 31 exit Done. ***************************** Oracle SEHUB pg. 32