Deep Dive in DB2 LUW offline table reorg
and index reorg
Saeid Mohseni
Folksam Insurance Co.
Session Code: D11
Wednesday, 16 November 2011, 11:00 – 12:00 | Platform: DB2 for LUW
Speaker Bio: Head of database team at Folksam Insurance Co. in
Sweden. Saeid has over 30 years of experience in the Information
Technology field, and 23 years of experience with the DB2 family of
products. His area of expertise include database administration /
database architect with focus on performance tuning in OLTP and BI
from SQL and DB2 LUW engine point of view, database recovery and
system development using DB2. Saeid is former IBM employee and
DB2 LUW specialist. He is DB2 LUW instructor - has been teaching
IBM DB2 LUW public classes for more than a decade in Sweden and
other Nordic countries. Saeid is co-author of IBM redbook (DB2 UDB
V8 non-DPF Performance Guide for High Performance OLTP and BI).
He is frequently speaker at RUG and member of IDUG - EMEA CPC.
1
Click to edit Master title style
2
Agenda
•
•
•
•
•
REORG background
Offline table REORG
Index REORG
Locking in table REORG & Index REORG
Offline REORG with Online table move
2
Click to edit Master title style
3
Agenda
•
•
•
•
•
REORG background
Offline table REORG
INDEX REORG
Locking in table REORG & Index REORG
Offline REORG with Online table move
3
Click to edit Master title style
4
Why to we need to REORG
• Table level:
•
•
•
•
Lot of holes in data pages
Lot of empty data pages
Expanded rows
Poor cluster ratio
• Index level:
•
•
•
•
Fragmented index pages
Poor physical index page order
Too many index levels
Lot of pseudo deleted RIDs and/or pseudo deleted leaf pages
A table need to be reorged of some
reasons: The table has been
involved in lot of deletes, update of
varchar columns to wider sise,
update of the column(s) which have
a cluser index on it. These things
couses holes in data pages and
increased number of datapages in
the table which increase number of
IO when table is scaned or range
delimited scan is used.
The index or indexes need to
reoeged if exists lot of fragmented
4
Click to edit Master title style
5
When do you need to REORG (1:2)
•
•
•
•
•
High volume of update, insert and delete
Low cluster ratio
New cluster index has been created
Row compression has been enabled
Table space has been converted to large
The table must be reorged if a new
clustering index is created or row
compression (Storage Optimizer)
has been enabled, of course the
dictionary can be created by
automatic when number of the new
inserted records are greater than 3
MB and new incoming data shall be
compressed. Rememer that the old
data are not compressed until a
table reorg is done.
A lage tablespace can hold larger
amout of the data in comparison
5
Click to edit Master title style
6
When do you need to REORG (2:2)
db2 reorgchk current statistics on table <table_name>
• Table level:
• F1 = * if more than 5% expanded rows or overflow records
• F2 = * if more than 30% holes in the table
• F3 = * if more than 20% empty pages
• Index level:
•
•
•
•
•
F4 = * if cluster ratio less than 80%
F5 = * if more than 50% empty in index entries
F6 = * too much empty entries in index levels
F7 = * if more than 20% pseudo deleted RIDs
F8 = * if more than 20% pseudo deleted empty pages
F1. Expanded rows or overflow
records occurs when lot of
VARCHAR column(s) are updated
to longer field and the new record
size will not fit in the same page
and the record is moved to another
page. The original record converted
to pointer which refer to the new
page and record location.
6
7
Click to edit Master title style
Table or Index REORG
HIGH
• Table REORG needed if F1, F2, F3 = *
• Table REORG needed if cluster index exists and F4 = *
• Index REORG needed if only F5 or F6 = *
• If F7 = * REORG index with CLEANUP ONLY option
• If only F8 = * REORG index with CLEANUP ONLY
PAGES
LOW
Table reorganization reorgs INDEXES too!
The table REORG reorgs the
indexes as well.Don’t reorg the
tbale if index reorg is sugessted by
reorgchk. The cost table reorg is
much higher than index reorg.
Avoid the table reorg if F4=* and
you don’t have any index defined
by cluster option or no index name
given during the previus REORG.
7
Click to edit Master title style
8
Agenda
•
•
•
•
•
REORG background
Offline table REORG
INDEX REORG
Locking in table REORG & Index REORG
Offline REORG with Online table move
8
Click to edit Master title style
9
Table REORG offline
• Up to four phases during table reorg
•
•
•
•
Scan–Sort
Build
Replace / Copy
Index Recreate
REORG TABLE table_name [INDEX index_name] [[ALLOW {READ|NO} ACCESS]
[USE tablespace-name] [INDEXSCAN] [LONGLOBDATA [USE long-tablespace-name]]
[KEEPDICTIONARY | RESETDICTIONARY]
The Offline table REORG passes up to 4 steps (Scan-Sort, Build,
Replace/Copy and Index recreate)
Scan-sort: REORG process scans and sort the table
Build: The shadow copy of the table is created
Replace/Copy: Replace: the shadow copy is renamed to the original
table – This process is done if shadow copy exists in the same
tablespace where the table exists. Copy: The shadow copy is moved
from the tempspace to the tablespace where the reorged table exists.
Index Recreate: All indexes are created during Offline table reorg.
9
Click to edit Master title style
10
Table REORG offline (1:4)
Four phases during table reorg
•
•
•
•
Scan–Sort
Build
Replace / Copy
Index Recreate
1.
2.
1. A table is scanned and sorted by an
INDEX column is specified
or
2. CLUSTER index was defined
on the table
db2 reorg table test.customer index ix1_customer
db2 reorg table test.customer
The RIDs are logged during reclustring REORG if log retetionen is enabled
During step 1 Scan-Sort DB2 sort the table according the Clustring
index if exists or data is sorted as the named index in the reorg
command.
IMPORTANT: The RIDs are logged during reclustring reorg if log
archiving is enabled.
10
Click to edit Master title style
11
Table REORG offline (2:4)
• Three phases during table reorg
•
•
•
•
Scan–Sort
Build
Replace / Copy
Index Recreate
db2 reorg table
Scan-Sort is ignored –
compactions is done via table
scan IF there is no index with
CLUSTER option
AND
no INDEX option is used.
test.customer Scan-Sort is ignored if:
- LONGLOBDATA option used
- sort record + RID > pagesize
in tempspace
db2 reorg table test.customer longlobdata
Here are some scenarios where step 1 (Scan-Sort) is ignored.
1. Sort-Scan ignored when the reorged table has no index with
CLUSTER option or no index name is used during the reorg.
2. Sort-Scan is also ignored if LONGLOBDATA option is used in the
reorg syntax.
3. DB2 ignore the Scan-Sort if the sort record size + RIDs are greater
than the tempspace page size.
11
Click to edit Master title style
12
Table REORG offline (3:4)
Four phases during table reorg
•
•
•
•
Scan–Sort
Build
Replace / Copy
Index Recreate
If INDEXSCAN option used. The index
enteries are used to create the shadow copy
db2 reorg table test.customer index ix2_customer indexscan
The Scan-Sort is ignored if INDEXSCAN option is used. DB2 follows
the index to create the shadow copy in that order. Important to know –
in this kind of reorg does not use the tempspace att all.
12
Click to edit Master title style
13
Table REORG offline (4:4)
• During table reorg
•
•
•
•
Scan–Sort
Build
Replace / Copy
Index Recreate
1
Sort is not involved during index creation
IF unique index DMS or reclustring index SMS.
Index is rebuild by scanning the new table
2
All other index recreation are involved in sort
Alla indexes belong to the reorged table are recreated during the last
reorg phase. The index sort key not sorted if the index is an unique
index in DMS tablespace or Clustring index in SMS tablespace. All
other indexes recreation are involved in sort.
13
Click to edit Master title style
14
REORG and CLUSTER INDEX
Hmm… I forgot to create that index with
CLUSTER option!
No problem… I can use that index
for reclustring when I reorg my table…
There is an important difference if
the index is defined with CLUSTER
option and the table is ordered by
that index
Or
The table is reorged by anouther
named index.
The defference is during INSERT
process which DB2 as ambition to
keep the cluster order in the table
when the index have CLUSTER
option.
14
15
Click to edit Master title style
Storage used in offline table REORG (1:2)
db2 reorg table idug.tab1 INDEX inx1_c3
tab1
shadow
my_ts
spill
merge
tempspace1
my_ts status Before reorg
my_ts status during reorg
We see in this exampl, USE TEMPSPACE option is not used and the
shadow copy is created in the table tablespace and two TDA sort files
(spill and merge) created in system temporary tablespace.
15
Click to edit Master title style
16
Storage used in offline table REORG (2:2)
db2 reorg table idug.tab1 INDEX inx1_c3 USE tempspace1
tab1
shadow
spill
merge
my_t
s
tempspace1
In this REORG example – DB2 holds the shadow copy of the reorged
table (SQL00002.DTR ) in system temporary tablespace and size of the
table’s tablespace is not increased during Sort-Scan and Build steps.
During step 3 Replace/Copy the file SQL00002.DTR are copied to the
table’s tablespace.
16
17
Click to edit Master title style
REORG - partitioned table (1:2)
REORG TABLE IDUG.PART_TAB ON DATA PARTION PART1
REORG TABLE IDUG.PART_TAB INDEX INX2 ON
DATA PARTITION PART2
ALLOW NO ACCESS | READ
ACCESS
IF
ALL INDEXES ARE PARTITIONED
INX1
INX2
<PART1>
PART_TAB
INX1
INX2
<PART2>
PART_TAB
INX1
INX2
<PART3>
PART_TAB
INX1
INX2
<PART4>
PART_TAB
In DB2 9.7 you can create a
partitioned table with partitioned
indexes.
You can reorg any parition if all
indexes are partitioned and ON
DATA PARTITION must option is
used.
17
18
Click to edit Master title style
REORG - partitioned table (2:2)
REORG TABLE IDUG.PART_TAB ALLOW NO ACCESS
ON DATA PARTITION PART1
ALLOW NO ACCESS
IF NON-PARTITIONED EXISTS
INX3
Global Index
INX1
INX2
<PART1>
PART_TAB
INX1
INX2
<PART2>
PART_TAB
INX1
INX2
<PART3>
PART_TAB
INX1
INX2
<PART4>
PART_TAB
The ALLOW NO ACCESS is the
only access mode supported if the
table has non-partitioned index.
That means the table is placed in
ALLOW NO ACCESS mode.
18
Click to edit Master title style
19
Monitoring REORG
GET SNAPSHOT FOR TABLES
gives information about current
running REORG.
Index_id used for clustering
Tablespace_id (tempspace1)
Being used for shadow copy
Current reorg phase
Max reorg phase
Current number of pages in progress
The table snapshot give us a good
picture on the REORGed table.
You see max number of phases in
this reorg is 4.
The current phase is step 1 which
isw Scan-Sort.
The Index ID is used for reclustring
is 2.
The system temporary tablespace
ID=1 is used for holdning the
shadow copy and sort
19
Click to edit Master title style
20
Offline REORG & Rollforward recovery
• The REORG will be redone during database or table
space rollforward.
• Important to take a backup after REORGs
• Space is taken from the table space or temporary table
space if USE option is used
• No scan-sort used - the reclustering use the logged RIDs
• During rollforward – many REORGs can be started in
parallel – risk for out of disk space!
It is important to take a backup after
REORGs to avoid the redo the
REORGs during rollforward
operation.
The reclusring reorg logs the RIDs.
The RIDs are used to redo the
reorg during rollforward operation.
20
Click to edit Master title style
21
Offline REORG & Crash recovery
• During phase 1,2 (scan-sort or build) revert to existing
table
• During phase 3 (replace/copy) restart the copy process
from shadow copy. Redo copy process is quick if shadow
copy and table are in the same table space (DMS)
• During phase 4 (index rebuild) marks indexes invalid
• Redo index rebuild depends on INDEXREC (access,
access_no_redo, restart, restart_no_redo or system)
Here you can see what happened
with REORGed table during crache
recovery:
The table shall have the state
before reorg if crache occurs during
phase 1 or 2
If the carche occurs during phase
3. The Replace/Copy be restarted.
All indexes are marked as bad and
index recreation is needed if crache
is on phase 4.
The index recreation process
restart depends on database
21
Click to edit Master title style
22
Agenda
•
•
•
•
•
Offline REORG background
Offline table REORG
INDEX REORG
Locking in table REORG & Index REORG
Offline REORG with Online table move
22
Click to edit Master title style
23
INDEX REORG
• On-line index REORG [allow write access]
• Off-line index REORG [allow no access] (default)
• Index REORG [allow read access]
REORG INDEXES ALL FOR TABLE table-name
[ALLOW {READ | NO | WRITE} ACCESS]
[{CLEANUP ONLY [ ALL | PAGES] | CONVERT}]
23
Click to edit Master title style
24
Index REORG allow write access
• Rebuild phase
• Scan sort
• Sort keys
• Build shadow index
• Log catchup phase
• Apply the log records (during rebuild phase) to shadow index
• Final log catchup followed by shadow switch over
• S lock on table followed by final log catchup
• Z lock on table followed by switch over to new index
• Commit
The Index reorg with ALLOW
WRITE ACCESS mode is executed
in three phases:
1.Rebuild Phase: First the table is
scaned to collect the keys – sort the
keys and build the shadow index.
2.Log Catchup Phase: All index
changes during phase 1 are logged
and applys to shadow index
3.Final Log Catchup: call the final
log changes and aquire a Z lock on
the table followed by switch the
shadow index to the original index.
24
25
Click to edit Master title style
Index REORG Monitoring
db2pd –db <db_name> -reorgs index
Index reorg type and status
Prev. Index creation time
Max phase = 3
Scan, build and catchup
ALLOW WRITE ACCESS
ALLOW READ ACCESS
Index reorg type and status
Max phase = 2
Scan and build
While index reorg is in progress
you can monitor it by using the
db2pd –reorg index option.
Here we have tow examples: The
first one with allow write access id
passed through three phases.
The second example reorgs index
with allow read access option use
two phases (without log catchup
phase)
25
26
Click to edit Master title style
INDEX REORG - partitioned table (1:2)
REORG INDEXES ALL FOR TABLE IDUG.PART_TAB
ON DATA PARTITION PART1
ACCESS MODE:
ALLOW READ ACCESS
NO ACCESS
WRITE ACCESS
INX1
INX2
INX1
<PART1>
PART_TAB
INX2
<PART2>
PART_TAB
INX1
INX2
<PART3>
PART_TAB
INX1
INX2
<PART4>
PART_TAB
All indexes within a partition can be
reorged. An individual index can
not be reorged if the index is
partitioned index.
26
27
Click to edit Master title style
INDEX REORG - partitioned table (2:2)
REORG INDEX INX3_NONPART FOR TABLE IDUG.PART_TAB
ACCESS MODE :
ALLOW READ ACCESS
NO ACESS
WRITE ACCESS
non_partitioned index
INX3_NONPART
Global Index
INX1
INX2
<PART1>
PART_TAB
INX1
INX2
<PART2>
PART_TAB
INX1
INX2
<PART3>
PART_TAB
INX1
INX2
<PART4>
PART_TAB
In table partitioning - an individual
index can be reorg if the index is a
global index.
27
Click to edit Master title style
28
Agenda
•
•
•
•
•
Offline REORG background
Offline table REORG
INDEX REORG
Locking and table REORG & Index REORG
Offline REORG with Online table move
28
Click to edit Master title style
29
REORG and Locking
• REORG TABLE - allow no access
• SYSCAT.TABLES IS on table and NS at row level
• IX on table space for data
• IX on index table space
• Z lock on table
• REORG TABLE – allow read access
• SYSCAT.TABLES IS on table and NS at row level
• IX on table space for data
• IX on index table space
• Z lock on table followed by switch over
29
Click to edit Master title style
30
REORG and Locking
• REORG INDEXES - allow read access
• SYSCAT.TABLES IS on table and NS at row
• IX on table space for data
• IX on index table space
• U lock on table
• REORG INDEXES – allow write access
• SYSCAT.TABLES IS on table and NS at row
• IX on table space for data
• IX on index table space
• U lock on table
• S lock on table followed by final log catchup
• Z lock on table followed by switch over to new index
30
Click to edit Master title style
31
Agenda
•
•
•
•
•
Offline REORG background
Offline table REORG
INDEX REORG
Locking and table REORG & Index REORG
Offline REORG with Online table move
31
Click to edit Master title style
32
Online Table Move (ADMIN_MOVE_TABLE Proc)
•
•
•
•
•
•
Move the table to the same or new table space
Allows UID to the source table
Columns can be added or removed
Data compression can be added
Table type can be changed (MDC, table partitioning, …)
Offline REORG while UID against the source table
One of the features in DB2 V9.7 is
Online Table Move. That means a
table can be moved from one
tablespace to another while
transactions (UID) runing against
the table. REORG is one of options
in online table move that we are
going to look closer to.
32
Click to edit Master title style
33
ADMIN_MOVE_TABLE steps
•
•
•
•
•
INIT: Create target,staging tables and triggers
COPY: Copy data from source to target table
REPLEY: Target table is updated from staging table
VERIFY*: Check if source and target table are identical
SWAP: Rename the target table to source table
CANCEL: Cancel the ADMIN_MOVE_TABLE process.
Drop the target and staging table
33
34
Click to edit Master title style
ADMIN_MOVE_TABLE (INIT)
C1 C2 C3 C4 C
5
Insert, Update, Delete
C1 C2 C3
C1 C2 C3 C4
C5
Target table
Source table
Applications makes changes
in the source table
C4 C5
Triggers
Staging table
During the INIT step, the target
table creates and the target table
definition can be changed. Creates
a staging table that holds an unique
key which can be a single column
or multi columns. Update, insert
and deletes are populated in the
staging table by creating triggers.
34
35
Click to edit Master title style
ADMIN_MOVE_TABLE (COPY)
Copy (Insert or Load)
C1 C2 C3 C4 C
5
C1 C2 C
3
C4 C5
Insert, Update, Delete
Triggers
C1 C
2
C3 C4
C5
Target table
Source table
Applications makes changes
in the source table
Staging table
Data are populated in tagget table
by insert or load. While copy step is
in progress the source table is in
full access mode and the new
changes are applied to the staging
table.
35
36
Click to edit Master title style
ADMIN_MOVE_TABLE (REPLEY)
C1 C2 C3 C4 C
5
C1 C2 C
3
C4 C5
Insert, Update, Delete
Triggers
Apply changes
C1 C
2
C3 C4
C5
Source table
Applications makes changes
in the source table
Target table
Staging table
If REPLAY_THRESHOLD=100
replay stops if number of records in Staging table less than 100
The first two steps INIT and COPY
can be started at anytime. It is
recommended to start the REPLY
step when the insert, update and
delete activities against the base
table is very low. The EPLY step,
applies all captured changes to the
target table. The itertions continues
until the configured value for
REPLY_THRESHOLD is reached.
36
37
Click to edit Master title style
ADMIN_MOVE_TABLE (SWAP)
C1 C2 C3 C4 C
5
C1 C2 C
3
C4 C5
2
Insert, Update, Delete
Triggers
C1 C
2
C3 C4
C5
Source table
Apply last
changes
3
1
Target table
4
Applications makes changes
in the source table
Rename the
target table to
source table
Staging table
5
Dropping the staging
and old source tables
SWAP step: all IUD access against
the source table are stopped and
the SWAP step call the last applies
from the staging table to the target
table.
The table renames to target table to
the source table, the indexes are
renamed as well.
The old source table and staging
table are dropped.
37
Click to edit Master title style
38
ADMIN_MOVE_TABLE
&
offline TABLE REORG
38
Click to edit Master title style
39
ADMIN_MOVE_TABLE & offline Table REORG
• Update SYSTOOLS.ADMIN_MOVE_TABLE by using
ADMIN_MOVE_TABLE_UTIL procedure
• Set COMMIT_AFTER_N_ROWS > 0 if insert used in COPY step
• Set REORG_USE_TEMPSPACE to an existing tempspace
• Useful options for ADMIN_MOVE_TABLE procedure
• COPY_USE_LOAD: Non_recoverable LOAD in COPY step
• FORCE: if COPY_USE_LOAD is used in SWAP step
• REORG: target table is REORGed before SWAP in SWAP step
The
SYSTOOLS.ADMIN_MOVE_TABL
E holds all configuration
parameters which can be used by
ONLINE TABLE MOVE. These
configuration values can changed
by calling
ADMIN_MOVE_TABLE_UTIL
procedure.
39
40
Click to edit Master title style
ADMIN_MOVE_TABLE & offline Table REORG (init)
call sysproc.admin_move_table ('IDUG', ’TAB1', '', '', '', '', '', '', '', '',
'init')
Next phase
Target table
Staging table
The information above illustrates
the output from INIT step in
SYSPROC.ADMIN_MOVE_TABLE
function.
40
41
Click to edit Master title style
ADMIN_MOVE_TABLE & offline Table REORG (copy)
all sysproc.admin_move_table ('IDUG', ’TAB1', '', '', '', '', '', '', '', 'COPY_USE_LOAD,FORCE', 'copy
Load
Nonrecoverable
Next phase
The information above illustrates
the output from COPY step in
SYSPROC.ADMIN_MOVE_TABLE
function. You can use INSRT or
LOAD to copy the table to target
table. The supported Load option is
NONRECOVERABLE and you
should remember to take a
tablespace or database backup.
41
42
Click to edit Master title style
ADMIN_MOVE_TABLE & offline Table REORG (replay)
call sysproc.admin_move_table ('IDUG', ’TAB1', '', '', '', '', '', '', '', 'REORG', 'replay')
REORG option must be added
here.
The REORG process is
during swap phase
Unique index is used to manage
the staging table during replay
Run REPLAY phase when low IUD activities on the source table
The information above illustrates
the output from REPLYstep in
SYSPROC.ADMIN_MOVE_TABLE
function. The staging table applies
all changes since COPY step to the
target table. Remember to mention
the REORG option during the
REPLY phase and the REORG
starts first in SWAP step.
42
Click to edit Master title style
43
ADMIN_MOVE_TABLE & offline Table REORG (swap)
call sysproc.admin_move_table ('IDUG', ’TAB1', '', '', '', '', '', '', '', 'REORG', 'swap')"
Time for replay and REORG
Stop replay if number of rows in
Staging table less than 1000
Time for final replay and swap
The information above illustrates
the output from SWAP step in
SYSPROC.ADMIN_MOVE_TABLE
function.
43
Click to edit Master title style
44
ADMIN_MOVE_TABLE & offline Table REORG (swap)
db2 get snapshot for tables on IDUG_DB
Target table REORG
Reorg in three phases:
Scan-sort is ignored
Build step using clustering index
to create the shadow copy
Clustering index_id =3
The information above illustrates
the output from table snapshot
while REORG is in progress in
SWAP step in
SYSPROC.ADMIN_MOVE_TABLE
function.
44
Click to edit Master title style
45
USE
ADMIN_MOVE_TABLE
without
REORG option
if
compaction needed
45
46
Click to edit Master title style
ADMIN_MOVE_TABLE without REORG option
db2 reorgchk current statistics on table idug.part_tab1
before
Lot of gaps and empty
data pages
after
ADMIN_MOVE_TABLE without
REORG option
Do not use the REPOG option if
you need only free the gaps and
empty pages. The online table
move compact the data and
recreates the indexes.
46
Saeid Mohseni
Folksam Insurance Co
saeid.mohseni@folksam.se
Session
D11
47