Deep Dive in DB2 LUW offline table reorg and index reorg Saeid Mohseni Folksam Insurance Co. Session Code: D11 Wednesday, 16 November 2011, 11:00 – 12:00 | Platform: DB2 for LUW Speaker Bio: Head of database team at Folksam Insurance Co. in Sweden. Saeid has over 30 years of experience in the Information Technology field, and 23 years of experience with the DB2 family of products. His area of expertise include database administration / database architect with focus on performance tuning in OLTP and BI from SQL and DB2 LUW engine point of view, database recovery and system development using DB2. Saeid is former IBM employee and DB2 LUW specialist. He is DB2 LUW instructor - has been teaching IBM DB2 LUW public classes for more than a decade in Sweden and other Nordic countries. Saeid is co-author of IBM redbook (DB2 UDB V8 non-DPF Performance Guide for High Performance OLTP and BI). He is frequently speaker at RUG and member of IDUG - EMEA CPC. 1 Click to edit Master title style 2 Agenda • • • • • REORG background Offline table REORG Index REORG Locking in table REORG & Index REORG Offline REORG with Online table move 2 Click to edit Master title style 3 Agenda • • • • • REORG background Offline table REORG INDEX REORG Locking in table REORG & Index REORG Offline REORG with Online table move 3 Click to edit Master title style 4 Why to we need to REORG • Table level: • • • • Lot of holes in data pages Lot of empty data pages Expanded rows Poor cluster ratio • Index level: • • • • Fragmented index pages Poor physical index page order Too many index levels Lot of pseudo deleted RIDs and/or pseudo deleted leaf pages A table need to be reorged of some reasons: The table has been involved in lot of deletes, update of varchar columns to wider sise, update of the column(s) which have a cluser index on it. These things couses holes in data pages and increased number of datapages in the table which increase number of IO when table is scaned or range delimited scan is used. The index or indexes need to reoeged if exists lot of fragmented 4 Click to edit Master title style 5 When do you need to REORG (1:2) • • • • • High volume of update, insert and delete Low cluster ratio New cluster index has been created Row compression has been enabled Table space has been converted to large The table must be reorged if a new clustering index is created or row compression (Storage Optimizer) has been enabled, of course the dictionary can be created by automatic when number of the new inserted records are greater than 3 MB and new incoming data shall be compressed. Rememer that the old data are not compressed until a table reorg is done. A lage tablespace can hold larger amout of the data in comparison 5 Click to edit Master title style 6 When do you need to REORG (2:2) db2 reorgchk current statistics on table <table_name> • Table level: • F1 = * if more than 5% expanded rows or overflow records • F2 = * if more than 30% holes in the table • F3 = * if more than 20% empty pages • Index level: • • • • • F4 = * if cluster ratio less than 80% F5 = * if more than 50% empty in index entries F6 = * too much empty entries in index levels F7 = * if more than 20% pseudo deleted RIDs F8 = * if more than 20% pseudo deleted empty pages F1. Expanded rows or overflow records occurs when lot of VARCHAR column(s) are updated to longer field and the new record size will not fit in the same page and the record is moved to another page. The original record converted to pointer which refer to the new page and record location. 6 7 Click to edit Master title style Table or Index REORG HIGH • Table REORG needed if F1, F2, F3 = * • Table REORG needed if cluster index exists and F4 = * • Index REORG needed if only F5 or F6 = * • If F7 = * REORG index with CLEANUP ONLY option • If only F8 = * REORG index with CLEANUP ONLY PAGES LOW Table reorganization reorgs INDEXES too! The table REORG reorgs the indexes as well.Don’t reorg the tbale if index reorg is sugessted by reorgchk. The cost table reorg is much higher than index reorg. Avoid the table reorg if F4=* and you don’t have any index defined by cluster option or no index name given during the previus REORG. 7 Click to edit Master title style 8 Agenda • • • • • REORG background Offline table REORG INDEX REORG Locking in table REORG & Index REORG Offline REORG with Online table move 8 Click to edit Master title style 9 Table REORG offline • Up to four phases during table reorg • • • • Scan–Sort Build Replace / Copy Index Recreate REORG TABLE table_name [INDEX index_name] [[ALLOW {READ|NO} ACCESS] [USE tablespace-name] [INDEXSCAN] [LONGLOBDATA [USE long-tablespace-name]] [KEEPDICTIONARY | RESETDICTIONARY] The Offline table REORG passes up to 4 steps (Scan-Sort, Build, Replace/Copy and Index recreate) Scan-sort: REORG process scans and sort the table Build: The shadow copy of the table is created Replace/Copy: Replace: the shadow copy is renamed to the original table – This process is done if shadow copy exists in the same tablespace where the table exists. Copy: The shadow copy is moved from the tempspace to the tablespace where the reorged table exists. Index Recreate: All indexes are created during Offline table reorg. 9 Click to edit Master title style 10 Table REORG offline (1:4) Four phases during table reorg • • • • Scan–Sort Build Replace / Copy Index Recreate 1. 2. 1. A table is scanned and sorted by an INDEX column is specified or 2. CLUSTER index was defined on the table db2 reorg table test.customer index ix1_customer db2 reorg table test.customer The RIDs are logged during reclustring REORG if log retetionen is enabled During step 1 Scan-Sort DB2 sort the table according the Clustring index if exists or data is sorted as the named index in the reorg command. IMPORTANT: The RIDs are logged during reclustring reorg if log archiving is enabled. 10 Click to edit Master title style 11 Table REORG offline (2:4) • Three phases during table reorg • • • • Scan–Sort Build Replace / Copy Index Recreate db2 reorg table Scan-Sort is ignored – compactions is done via table scan IF there is no index with CLUSTER option AND no INDEX option is used. test.customer Scan-Sort is ignored if: - LONGLOBDATA option used - sort record + RID > pagesize in tempspace db2 reorg table test.customer longlobdata Here are some scenarios where step 1 (Scan-Sort) is ignored. 1. Sort-Scan ignored when the reorged table has no index with CLUSTER option or no index name is used during the reorg. 2. Sort-Scan is also ignored if LONGLOBDATA option is used in the reorg syntax. 3. DB2 ignore the Scan-Sort if the sort record size + RIDs are greater than the tempspace page size. 11 Click to edit Master title style 12 Table REORG offline (3:4) Four phases during table reorg • • • • Scan–Sort Build Replace / Copy Index Recreate If INDEXSCAN option used. The index enteries are used to create the shadow copy db2 reorg table test.customer index ix2_customer indexscan The Scan-Sort is ignored if INDEXSCAN option is used. DB2 follows the index to create the shadow copy in that order. Important to know – in this kind of reorg does not use the tempspace att all. 12 Click to edit Master title style 13 Table REORG offline (4:4) • During table reorg • • • • Scan–Sort Build Replace / Copy Index Recreate 1 Sort is not involved during index creation IF unique index DMS or reclustring index SMS. Index is rebuild by scanning the new table 2 All other index recreation are involved in sort Alla indexes belong to the reorged table are recreated during the last reorg phase. The index sort key not sorted if the index is an unique index in DMS tablespace or Clustring index in SMS tablespace. All other indexes recreation are involved in sort. 13 Click to edit Master title style 14 REORG and CLUSTER INDEX Hmm… I forgot to create that index with CLUSTER option! No problem… I can use that index for reclustring when I reorg my table… There is an important difference if the index is defined with CLUSTER option and the table is ordered by that index Or The table is reorged by anouther named index. The defference is during INSERT process which DB2 as ambition to keep the cluster order in the table when the index have CLUSTER option. 14 15 Click to edit Master title style Storage used in offline table REORG (1:2) db2 reorg table idug.tab1 INDEX inx1_c3 tab1 shadow my_ts spill merge tempspace1 my_ts status Before reorg my_ts status during reorg We see in this exampl, USE TEMPSPACE option is not used and the shadow copy is created in the table tablespace and two TDA sort files (spill and merge) created in system temporary tablespace. 15 Click to edit Master title style 16 Storage used in offline table REORG (2:2) db2 reorg table idug.tab1 INDEX inx1_c3 USE tempspace1 tab1 shadow spill merge my_t s tempspace1 In this REORG example – DB2 holds the shadow copy of the reorged table (SQL00002.DTR ) in system temporary tablespace and size of the table’s tablespace is not increased during Sort-Scan and Build steps. During step 3 Replace/Copy the file SQL00002.DTR are copied to the table’s tablespace. 16 17 Click to edit Master title style REORG - partitioned table (1:2) REORG TABLE IDUG.PART_TAB ON DATA PARTION PART1 REORG TABLE IDUG.PART_TAB INDEX INX2 ON DATA PARTITION PART2 ALLOW NO ACCESS | READ ACCESS IF ALL INDEXES ARE PARTITIONED INX1 INX2 <PART1> PART_TAB INX1 INX2 <PART2> PART_TAB INX1 INX2 <PART3> PART_TAB INX1 INX2 <PART4> PART_TAB In DB2 9.7 you can create a partitioned table with partitioned indexes. You can reorg any parition if all indexes are partitioned and ON DATA PARTITION must option is used. 17 18 Click to edit Master title style REORG - partitioned table (2:2) REORG TABLE IDUG.PART_TAB ALLOW NO ACCESS ON DATA PARTITION PART1 ALLOW NO ACCESS IF NON-PARTITIONED EXISTS INX3 Global Index INX1 INX2 <PART1> PART_TAB INX1 INX2 <PART2> PART_TAB INX1 INX2 <PART3> PART_TAB INX1 INX2 <PART4> PART_TAB The ALLOW NO ACCESS is the only access mode supported if the table has non-partitioned index. That means the table is placed in ALLOW NO ACCESS mode. 18 Click to edit Master title style 19 Monitoring REORG GET SNAPSHOT FOR TABLES gives information about current running REORG. Index_id used for clustering Tablespace_id (tempspace1) Being used for shadow copy Current reorg phase Max reorg phase Current number of pages in progress The table snapshot give us a good picture on the REORGed table. You see max number of phases in this reorg is 4. The current phase is step 1 which isw Scan-Sort. The Index ID is used for reclustring is 2. The system temporary tablespace ID=1 is used for holdning the shadow copy and sort 19 Click to edit Master title style 20 Offline REORG & Rollforward recovery • The REORG will be redone during database or table space rollforward. • Important to take a backup after REORGs • Space is taken from the table space or temporary table space if USE option is used • No scan-sort used - the reclustering use the logged RIDs • During rollforward – many REORGs can be started in parallel – risk for out of disk space! It is important to take a backup after REORGs to avoid the redo the REORGs during rollforward operation. The reclusring reorg logs the RIDs. The RIDs are used to redo the reorg during rollforward operation. 20 Click to edit Master title style 21 Offline REORG & Crash recovery • During phase 1,2 (scan-sort or build) revert to existing table • During phase 3 (replace/copy) restart the copy process from shadow copy. Redo copy process is quick if shadow copy and table are in the same table space (DMS) • During phase 4 (index rebuild) marks indexes invalid • Redo index rebuild depends on INDEXREC (access, access_no_redo, restart, restart_no_redo or system) Here you can see what happened with REORGed table during crache recovery: The table shall have the state before reorg if crache occurs during phase 1 or 2 If the carche occurs during phase 3. The Replace/Copy be restarted. All indexes are marked as bad and index recreation is needed if crache is on phase 4. The index recreation process restart depends on database 21 Click to edit Master title style 22 Agenda • • • • • Offline REORG background Offline table REORG INDEX REORG Locking in table REORG & Index REORG Offline REORG with Online table move 22 Click to edit Master title style 23 INDEX REORG • On-line index REORG [allow write access] • Off-line index REORG [allow no access] (default) • Index REORG [allow read access] REORG INDEXES ALL FOR TABLE table-name [ALLOW {READ | NO | WRITE} ACCESS] [{CLEANUP ONLY [ ALL | PAGES] | CONVERT}] 23 Click to edit Master title style 24 Index REORG allow write access • Rebuild phase • Scan sort • Sort keys • Build shadow index • Log catchup phase • Apply the log records (during rebuild phase) to shadow index • Final log catchup followed by shadow switch over • S lock on table followed by final log catchup • Z lock on table followed by switch over to new index • Commit The Index reorg with ALLOW WRITE ACCESS mode is executed in three phases: 1.Rebuild Phase: First the table is scaned to collect the keys – sort the keys and build the shadow index. 2.Log Catchup Phase: All index changes during phase 1 are logged and applys to shadow index 3.Final Log Catchup: call the final log changes and aquire a Z lock on the table followed by switch the shadow index to the original index. 24 25 Click to edit Master title style Index REORG Monitoring db2pd –db <db_name> -reorgs index Index reorg type and status Prev. Index creation time Max phase = 3 Scan, build and catchup ALLOW WRITE ACCESS ALLOW READ ACCESS Index reorg type and status Max phase = 2 Scan and build While index reorg is in progress you can monitor it by using the db2pd –reorg index option. Here we have tow examples: The first one with allow write access id passed through three phases. The second example reorgs index with allow read access option use two phases (without log catchup phase) 25 26 Click to edit Master title style INDEX REORG - partitioned table (1:2) REORG INDEXES ALL FOR TABLE IDUG.PART_TAB ON DATA PARTITION PART1 ACCESS MODE: ALLOW READ ACCESS NO ACCESS WRITE ACCESS INX1 INX2 INX1 <PART1> PART_TAB INX2 <PART2> PART_TAB INX1 INX2 <PART3> PART_TAB INX1 INX2 <PART4> PART_TAB All indexes within a partition can be reorged. An individual index can not be reorged if the index is partitioned index. 26 27 Click to edit Master title style INDEX REORG - partitioned table (2:2) REORG INDEX INX3_NONPART FOR TABLE IDUG.PART_TAB ACCESS MODE : ALLOW READ ACCESS NO ACESS WRITE ACCESS non_partitioned index INX3_NONPART Global Index INX1 INX2 <PART1> PART_TAB INX1 INX2 <PART2> PART_TAB INX1 INX2 <PART3> PART_TAB INX1 INX2 <PART4> PART_TAB In table partitioning - an individual index can be reorg if the index is a global index. 27 Click to edit Master title style 28 Agenda • • • • • Offline REORG background Offline table REORG INDEX REORG Locking and table REORG & Index REORG Offline REORG with Online table move 28 Click to edit Master title style 29 REORG and Locking • REORG TABLE - allow no access • SYSCAT.TABLES IS on table and NS at row level • IX on table space for data • IX on index table space • Z lock on table • REORG TABLE – allow read access • SYSCAT.TABLES IS on table and NS at row level • IX on table space for data • IX on index table space • Z lock on table followed by switch over 29 Click to edit Master title style 30 REORG and Locking • REORG INDEXES - allow read access • SYSCAT.TABLES IS on table and NS at row • IX on table space for data • IX on index table space • U lock on table • REORG INDEXES – allow write access • SYSCAT.TABLES IS on table and NS at row • IX on table space for data • IX on index table space • U lock on table • S lock on table followed by final log catchup • Z lock on table followed by switch over to new index 30 Click to edit Master title style 31 Agenda • • • • • Offline REORG background Offline table REORG INDEX REORG Locking and table REORG & Index REORG Offline REORG with Online table move 31 Click to edit Master title style 32 Online Table Move (ADMIN_MOVE_TABLE Proc) • • • • • • Move the table to the same or new table space Allows UID to the source table Columns can be added or removed Data compression can be added Table type can be changed (MDC, table partitioning, …) Offline REORG while UID against the source table One of the features in DB2 V9.7 is Online Table Move. That means a table can be moved from one tablespace to another while transactions (UID) runing against the table. REORG is one of options in online table move that we are going to look closer to. 32 Click to edit Master title style 33 ADMIN_MOVE_TABLE steps • • • • • INIT: Create target,staging tables and triggers COPY: Copy data from source to target table REPLEY: Target table is updated from staging table VERIFY*: Check if source and target table are identical SWAP: Rename the target table to source table CANCEL: Cancel the ADMIN_MOVE_TABLE process. Drop the target and staging table 33 34 Click to edit Master title style ADMIN_MOVE_TABLE (INIT) C1 C2 C3 C4 C 5 Insert, Update, Delete C1 C2 C3 C1 C2 C3 C4 C5 Target table Source table Applications makes changes in the source table C4 C5 Triggers Staging table During the INIT step, the target table creates and the target table definition can be changed. Creates a staging table that holds an unique key which can be a single column or multi columns. Update, insert and deletes are populated in the staging table by creating triggers. 34 35 Click to edit Master title style ADMIN_MOVE_TABLE (COPY) Copy (Insert or Load) C1 C2 C3 C4 C 5 C1 C2 C 3 C4 C5 Insert, Update, Delete Triggers C1 C 2 C3 C4 C5 Target table Source table Applications makes changes in the source table Staging table Data are populated in tagget table by insert or load. While copy step is in progress the source table is in full access mode and the new changes are applied to the staging table. 35 36 Click to edit Master title style ADMIN_MOVE_TABLE (REPLEY) C1 C2 C3 C4 C 5 C1 C2 C 3 C4 C5 Insert, Update, Delete Triggers Apply changes C1 C 2 C3 C4 C5 Source table Applications makes changes in the source table Target table Staging table If REPLAY_THRESHOLD=100 replay stops if number of records in Staging table less than 100 The first two steps INIT and COPY can be started at anytime. It is recommended to start the REPLY step when the insert, update and delete activities against the base table is very low. The EPLY step, applies all captured changes to the target table. The itertions continues until the configured value for REPLY_THRESHOLD is reached. 36 37 Click to edit Master title style ADMIN_MOVE_TABLE (SWAP) C1 C2 C3 C4 C 5 C1 C2 C 3 C4 C5 2 Insert, Update, Delete Triggers C1 C 2 C3 C4 C5 Source table Apply last changes 3 1 Target table 4 Applications makes changes in the source table Rename the target table to source table Staging table 5 Dropping the staging and old source tables SWAP step: all IUD access against the source table are stopped and the SWAP step call the last applies from the staging table to the target table. The table renames to target table to the source table, the indexes are renamed as well. The old source table and staging table are dropped. 37 Click to edit Master title style 38 ADMIN_MOVE_TABLE & offline TABLE REORG 38 Click to edit Master title style 39 ADMIN_MOVE_TABLE & offline Table REORG • Update SYSTOOLS.ADMIN_MOVE_TABLE by using ADMIN_MOVE_TABLE_UTIL procedure • Set COMMIT_AFTER_N_ROWS > 0 if insert used in COPY step • Set REORG_USE_TEMPSPACE to an existing tempspace • Useful options for ADMIN_MOVE_TABLE procedure • COPY_USE_LOAD: Non_recoverable LOAD in COPY step • FORCE: if COPY_USE_LOAD is used in SWAP step • REORG: target table is REORGed before SWAP in SWAP step The SYSTOOLS.ADMIN_MOVE_TABL E holds all configuration parameters which can be used by ONLINE TABLE MOVE. These configuration values can changed by calling ADMIN_MOVE_TABLE_UTIL procedure. 39 40 Click to edit Master title style ADMIN_MOVE_TABLE & offline Table REORG (init) call sysproc.admin_move_table ('IDUG', ’TAB1', '', '', '', '', '', '', '', '', 'init') Next phase Target table Staging table The information above illustrates the output from INIT step in SYSPROC.ADMIN_MOVE_TABLE function. 40 41 Click to edit Master title style ADMIN_MOVE_TABLE & offline Table REORG (copy) all sysproc.admin_move_table ('IDUG', ’TAB1', '', '', '', '', '', '', '', 'COPY_USE_LOAD,FORCE', 'copy Load Nonrecoverable Next phase The information above illustrates the output from COPY step in SYSPROC.ADMIN_MOVE_TABLE function. You can use INSRT or LOAD to copy the table to target table. The supported Load option is NONRECOVERABLE and you should remember to take a tablespace or database backup. 41 42 Click to edit Master title style ADMIN_MOVE_TABLE & offline Table REORG (replay) call sysproc.admin_move_table ('IDUG', ’TAB1', '', '', '', '', '', '', '', 'REORG', 'replay') REORG option must be added here. The REORG process is during swap phase Unique index is used to manage the staging table during replay Run REPLAY phase when low IUD activities on the source table The information above illustrates the output from REPLYstep in SYSPROC.ADMIN_MOVE_TABLE function. The staging table applies all changes since COPY step to the target table. Remember to mention the REORG option during the REPLY phase and the REORG starts first in SWAP step. 42 Click to edit Master title style 43 ADMIN_MOVE_TABLE & offline Table REORG (swap) call sysproc.admin_move_table ('IDUG', ’TAB1', '', '', '', '', '', '', '', 'REORG', 'swap')" Time for replay and REORG Stop replay if number of rows in Staging table less than 1000 Time for final replay and swap The information above illustrates the output from SWAP step in SYSPROC.ADMIN_MOVE_TABLE function. 43 Click to edit Master title style 44 ADMIN_MOVE_TABLE & offline Table REORG (swap) db2 get snapshot for tables on IDUG_DB Target table REORG Reorg in three phases: Scan-sort is ignored Build step using clustering index to create the shadow copy Clustering index_id =3 The information above illustrates the output from table snapshot while REORG is in progress in SWAP step in SYSPROC.ADMIN_MOVE_TABLE function. 44 Click to edit Master title style 45 USE ADMIN_MOVE_TABLE without REORG option if compaction needed 45 46 Click to edit Master title style ADMIN_MOVE_TABLE without REORG option db2 reorgchk current statistics on table idug.part_tab1 before Lot of gaps and empty data pages after ADMIN_MOVE_TABLE without REORG option Do not use the REPOG option if you need only free the gaps and empty pages. The online table move compact the data and recreates the indexes. 46 Saeid Mohseni Folksam Insurance Co saeid.mohseni@folksam.se Session D11 47