DBAnalys To The Rescue!
George Potemkin
Russia
Dmitri Levin
alphabroder
St. Petersburg
1703
Philadelphia
1682
ftp://ftp.progress-tech.ru/pub/Users/george/DBAnalys1_Boston.pptx
ftp://ftp.progress-tech.ru/pub/Users/george/DBAnalys2_Copenhagen.pptx
DBAnalys To The Rescue!
Contents
History
Theory
Practice
Optimization
Synergy
DBAnalys To The Rescue
2
PROGRESS Database Analysis
Database: %s
Options: ixanalys chanalys
Date: %s
No tabanalys option
FREE CHAIN ANALYSIS
===================
%l block(s) found in the free chain.
V5
1988
RM CHAIN ANALYSIS
=================
%l block(s) found in the RM chain.
BLOCK ANALYSIS
==============
%l RM block(s) found in the database.
%l.%l% of the RM block space is used.
%l index block(s) found in the database.
%l.%l% of the index block space is used.
Manning Road in Billerica, Massachusetts
Index
%s %d
Blocks
%l
INDEX BLOCK SUMMARY
%
% of all
Bytes
Utiliz.
Indexes
%l
%l.%l
%l.%l
DBAnalys To The Rescue
3
PROGRESS Database Analysis
Database:
Options:
tabanalys
tabanalys option
Date: Mon Nov 6 18:26:53 1995
V7.3A
BLOCK ANALYSIS (3906)
==============
Nov, 9 1994
1 master block(s) found in the database.
RECORD BLOCK SUMMARY (6129)
---Record Size--- ---Fragments--- Scatter
Table
Records
Bytes
Min
Max Mean
Count Index Index
----------------------------------------------------------<329 lines of boring stuff deleted>
Totals:
15021963
642199k
6 1675
43 15275513 1.0 5.3
802392 RM block(s) found in the database.
86.10% of the RM block space is used.
270404 index block(s) found in the database.
400571 free block(s) found in the database.
5 index table block(s) found in the database.
1 sequence block(s) found in the database.
1473374 total blocks found in the database.
14 Oak Park in Bedford, Massachusetts
DBAnalys To The Rescue
4
Segments of dbanalys output
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
3867:
3868:
8361:
14178:
6172:
3870:
14179:
14163:
14157:
14161:
3873:
3880:
14155:
16573:
16574:
3892:
3897:
14172:
7269:
7283:
14176:
10097:
3906:
1)
2)
2.1)
2.2)
2.3)
2.4)
3)
4)
PROGRESS Database Analysis
Database: %s
Blocksize: %i
RecSpaceSearchDepth: %d
Options: %s %s %s
Date: %s%r
CHAIN ANALYSIS FOR AREA "%s" : %l
FREE CLUSTER CHAIN ANALYSIS
LIST OF FREE CLUSTER CHAIN BLOCKS
%J cluster(s) found in the free cluster chain.
FREE CHAIN ANALYSIS
LIST OF FREE CHAIN BLOCKS
%J block(s) found in the free chain of %s object %i
%rNumber of
Object
Object
Partition/Tenant/Group
Blocks
Type
RM CHAIN ANALYSIS
LIST OF RM CHAIN BLOCKS
%J block(s) found in the RM chain of %s object %i
INDEX DELETE CHAIN ANALYSIS
LIST OF INDEX DELETE CHAIN BLOCKS
%J block(s) found in the Index Delete chain of %s object %i
AREA "%s" : %d BLOCK ANALYSIS
BLOCK ANALYSIS (moved to the end at V11)
DBAnalys To The Rescue
5
Segments of dbanalys output
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
msg
6129:
14212:
16667:
16668:
16656:
3921:
3925:
14225:
16657:
16658:
3923:
2314:
16666:
16659:
16579:
3937:
5)
6)
7)
8)
msgs
0000-6130
msgs
7270-9999
msgs 9894-10130
msgs 14155-14225
msgs 16641-17035
RECORD BLOCK SUMMARY
RECORD BLOCK SUMMARY FOR AREA "%s" : %l
RECORD BLOCK SUMMARY FOR SHARED TABLES
RECORD BLOCK SUMMARY FOR %s %s: %l
Summary for AREA "%s": %l
%J RM block(s) found in the database.
INDEX BLOCK SUMMARY
INDEX BLOCK SUMMARY FOR AREA "%s" : %l
INDEX BLOCK SUMMARY FOR SHARED OBJECTS:
INDEX BLOCK SUMMARY FOR %s: %s
%J index block(s) found in the database.
DATABASE SUMMARY
SUMMARY FOR AREA "%s": %l
SUMMARY FOR SHARED OBJECTS:
DATABASE BLOCK ANALYSIS:
database analysis complete %s%r
before 8.2A
8.3A
9.1D
10.2B
11.1
DBAnalys To The Rescue
6
History of changes
V9.0A
proutil db-name -C dbanalys [ [area] area-name ]
Nov 21, 1998
V9.0A
proutil db-name -C dbanalys –Bp 64
Nov 21, 1998
V9.1A
Jan 21, 2000
Index Delete Chains / LOCKCHN (bk_frchn 2)
V10.0A
Free Cluster Chains / CLIST_FREECHN (bk_frchn 3)
Dec 13, 2003
DBAnalys To The Rescue
7
History of changes
V10.1A02
Algorithm for calculation of Scatter Factor for SAT II
Oct 27, 2006 tables has changed
Scatter for Type II is reported as a number of blocks occupied
divided by the minimum number of blocks that table could occupy.
Scatter for Type I is reported as log10 of sum of distances between records
in a table, where distance is calculated as the difference between the
RECID values for the two records, divided by minimum distance possible.
In V11 that formula lost the meaning again when the database is
multi-tenant or has partitioning.
DBAnalys To The Rescue
8
History of changes
V10.1B02
Aug 3, 2007
V10.1B02
Aug 3, 2007
-recspacesearchdepth
The number of RM blocks to move from the front to the back
of the RM chain before allocating a new cluster for data.
Default is 5 * (cluster size / 10). Max value is 100.
Old algorithm: No more than 3 blocks at a time will be
examined and moved to the tail end.
Full report on chains when running chanalys online.
The following lists will be reported:
LIST OF FREE CLUSTER CHAIN BLOCKS
LIST OF FREE CHAIN BLOCKS
LIST OF RM CHAIN BLOCKS
LIST OF INDEX DELETE CHAIN BLOCKS
DBAnalys To The Rescue
9
History of changes
V10.1B03
Nov 2, 2007
proutil <db> -C chanalys [ -scan ]
-scan: fewer locks, less consistent
It instructs chanalys to use minimal locking. This may cause the chains
report to be incomplete , but it will avoid turning on table locks.
Offline chanalys does not use a locking mechanism, the -scan
parameter will be ignored.
[Warning] Unable to acquire table lock on object 2. Chain analysis continuing with -scan option. (14167)
LIST OF FREE CHAIN BLOCKS
next
dbkey free
51 52
CTRL-C pressed - exit free chain analysis
DBAnalys To The Rescue
10
History of changes
Database Analysis (dbanalys) output has changed.
10.2B06
March, 2012 In the past customer's complained that the DBANALYS report widths
were not wide enough resulting in a CR after the table name when
the table name exceeded the Tablename column width.
Now the tabanalys width is 110 chars.
V11
Dance of the broken lines (the ixanalys width is 86 chars)
INDEX BLOCK SUMMARY FOR SHARED OBJECTS:
-------------------------------------------Table
Index Fields Levels
PUB.theTable
IntField
IntField901234567890123456789012
10
1
IntField-1:3
2
IntField-10:2
2
DBAnalys To The Rescue
Blocks
Size
% Util
Factor
215
215
1.6M
1.6M
98.2
98.2
1.0
1.0
11
History of changes
V11.4
Jul 25, 2014
proutil db-name -C dbanalys [ [area] area-name ]
[ tenant tenant-name| group group-name| shared ]
[-csoutput [ -fieldsep sep-value][ -csfilePrefix pref-name ]
[ -verbose ]]
-csoutput
-fieldsep sep-value (default is space )
-csfilePrefix pref-name (default db-name)
-verbose
proutil sports –C dbanalys -csoutput -verbose
DBAnalys To The Rescue
12
Output files
-csfilePrefix
V11.6
sports.block.txt
sports.ch.txt
sports.ix.txt
sports.tbl.txt
sports.lob.txt
-verbose
"Time Stamp" "DBUID" "DB name" "Area" "Area #" "Partition Type" "Tenancy"
"Tenancy #" "Partition Policy" "Partition" "Partition #" "Owner" "Table" "Table #"
"Records" "Size" "Min" "Max" "Mean“
2015-04-06T14:59:00.000-4:00 "njVjvK5a8qfvEyE+7KM4WA" "sports"
"Customer/Order Area" 8 "-" "-" 0 "-" "-" 0 "PUB" "Customer"
-fieldsep
2 83 12377 118 222 149 83 1.0 1.0
2015-04-06T14:59:00.000-4:00 "njVjvK5a8qfvEyE+7KM4WA" "sports"
"Customer/Order Area" 8 "-" "-" 0 "-" "-" 0 "PUB" "Order"
4 207 11730 53 61 56 207 1.0 1.0
DBAnalys To The Rescue
13
Loading sports.tbl.txt in Excel
DBAnalys To The Rescue
14
Loading sports.ix.txt in Excel
DBAnalys To The Rescue
15
History of changes
11.6
Oct 26, 2015
LOB support in PROUTIL DBANALYS
The PROUTIL database analysis utilities are enhanced to
include analysis of LOB data. When LOB data is found in an
area, a “LOB Summary” follows the “Record Block Summary”
in the output, and LOB data is included in the combined
summary. LOB data is also written to a new *lob.txt file when
the -cscoutput parameter is used on the analysis command
line.
DBAnalys To The Rescue
16
DBAnalys To The Rescue!
Contents
History
Theory
Practice
Optimization
Synergy
DBAnalys To The Rescue
17
Space: The Final Frontier… 20 Years After
Disk Space Allocation in the Progress RDBMS
G. Bjorklund, posted on PEG: Aug 15, 1995
Space: The Final Frontier, Engine Crew Monograph No. 2
Gus Bjorklund, Aug 10, 1997
http://www.fast4gl.com/downloads/monographs/space/space.html
Space: the final frontier, OpenEdge RDBMS Internals
Americas PUG Challenge 2012 , Westford, MA
Gus Bjorklund, May 9, 2012
https://community.progress.com/community_groups/openedge_rdbms/w/openedgerdbms/791.space-thefinal-frontier
OpenEdge 10 RDBMS Advanced Storage Architecture
Richard Banville, May, 2004
https://community.progress.com/community_groups/user_conferences/w/exchange/2138.exchange-2004breakout-sessions
DBAnalys To The Rescue
18
Space: The Final Frontier… 20 Years After
Block Types (dbrpr)
1. Master Block
2. Index Block
3. Data Block
4. Free Block
5. Index Dbkey List Block
6. Sequence Block
7. Empty Block
8. Parameter Block (Not Used)
9. Area Block
10. Object Directory Block (Not Used)
11. Extent List Block (Not Used)
12. Object Block
13. Control Block
14. Area Object List Block
15. Area Cluster Allocation Block
16. Object Cluster List Block
17. Object Block Allocation Block
18. RM bit Map List Block
19. Extent Header Block (bk_type 254)
20. Reserved Block
There are 19 known block types.
5 block types do not (should not) exist in database.
4 block types do exist in database but they are not used.
16 block types are mentioned by dbanalys before V11 and 14 block types by V11 dbanalys.
DBAnalys To The Rescue
19
Block types in dbanalys
AREA "misc128_dat": 16 BLOCK ANALYSIS
------------------------------------------------4629499 block(s) found in the area.
Current high water mark: 4629439
56 free block(s) found in the area
4627892 record block(s) found in the area
0 index block(s) found in the area
60 empty block(s) found in the area
372 object block(s) found in the area
372 cluster list block(s) found in the area
2 object list block(s) found in the area
372 cluster map block(s) found in the area
0 row allocation block(s) found in the storage area.
DATABASE BLOCK ANALYSIS:
----------------------66432530 RM block(s) found in the database.
17906763 index block(s) found in the database.
1 master block(s) found in the database.
38 area block(s) found in the database.
1 control block(s) found in the database.
2 object block(s) found in the database.
2905 cluster list block(s) found in the database.
2905 cluster allocation block(s) found in the database.
2905 object block(s) found in the database.
42 object list block(s) found in the database.
2905 object allocation block(s) found in the database.
91506 free block(s) found in the database.
0 index table block(s) found in the database.
1 sequence block(s) found in the database.
692370 empty block(s) found in the database.
DBAnalys To The Rescue
20
Types of Storage Areas
Blocks could be in the areas of the following types:
1. Storage Area Type I
2. Storage Area Type II
3. Storage Area Type III
I + II Hybrid (V10.0B)
Used for temp-tables ( DBI file )
DBAnalys To The Rescue
21
Data Clusters
Storage Area Type II
Three possible cluster sizes: 8, 64, or 512 adjacent blocks
Clusters are allocated from the area and given to a particular object.
They are maintained by a doubly linked list.
First and last blocks in the cluster have the extended headers (80 bytes instead of 64).
First block stores Owner Info (transactionId, serialNumber)
Last block stores Cluster Link (firstFreeCluster, lastFreeCluster)
Database extend: MAX(64, cluster size) – that means that database will be extended for 1 cluster if the
cluster size is 64 or 512 and 8 clusters if cluster size is 8.
DBAnalys To The Rescue
22
Chain Types
1. Cluster chains
2. Block chains
;-)
3. F-chains
Cluster Chains
Block Chains
1. Area Control Clusters
1. Free Chains (FREECHN, bk_frchn 0)
2. Data Object Clusters
2. RM Chains (RMCHN, bk_frchn 1)
3. Free Cluster Chains (CLIST_FREECHN)
3. Index Delete Chains (LOCKCHN, bk_frchn 2)
F-Chains
DBAnalys To The Rescue
Z unofficial name ;-)
23
Free Cluster Chains
proutil sports –C dbrpr
15. Display Cluster Chain (Type II Area)
Cluster Link of object 1/4
Total Clusters: 2 First cluster: 768 Last cluster: 1536
768
1536
1536
0
Clusters of the“Order” table (object 1/4)
2 cluster(s) found
and its “Cust-Order” index (object 2/21).
Cluster Link of object 2/21
Total Clusters: 1 First cluster: 1024
1024
0
1 cluster(s) found
Last cluster: 1024
Fast Object Deletion
Area Free Cluster Link
First cluster: 1024 Last cluster: 1536
1024
768
Free cluster chain after the “Order” table
768
1536
and its “Cust-Order” index were deleted.
1536
0
3 cluster(s) found
DBAnalys To The Rescue
24
Free Cluster Chains
CHAIN ANALYSIS FOR AREA "Customer/Order Area" : 8
---------------------------------------------------------FREE CLUSTER CHAIN ANALYSIS
--------------------------LIST OF FREE CLUSTER CHAIN BLOCKS
next
dbkey
free
1024
768
768
1536
1536
0
3 cluster(s) found in the free cluster chain.
DBAnalys To The Rescue
25
Free Cluster Chains
DBAnalys To The Rescue
26
Block Chains
proutil db –C chanalys [–scan]
LIST OF FREE CHAIN BLOCKS
next
dbkey
free
7026440
7026439
2046268 block(s) found in the free chain of Index object 22
LIST OF INDEX DELETE CHAIN BLOCKS
dbkey
Block
722
2499
3 block(s) found in the Index Delete chain of Index object 33
LIST OF RM CHAIN BLOCKS
next
dbkey
Block
804512704
2372
49
0
8534342 block(s) found in the RM chain of Table object 4
DBAnalys To The Rescue
27
Index Delete chain
Index
Object
Block
Free
Block
Free
Block
Free
Block
Free
Block
Ixdel
Block
Ixdel
Block
Ixdel
Block
Ixdel
Block
(Unique Indexes only)
20731 block(s) found in the Index Delete chain of Index object 622
proutil db-name -C idxcompact item-tax.item-group
0 block(s) found in the Index Delete chain of Index object 622
DBAnalys To The Rescue
28
DBAnalys To The Rescue!
Contents
History
Theory
Practice
Optimization
Synergy
DBAnalys To The Rescue
29
Dbanalys Processing Phases
[2015/10/19@12:49:02.602-0400] CHAIN ANALYSIS FOR AREA "Order Area" : 9
[2015/10/19@13:18:16.944-0400] 1402490 block(s) found in the RM chain of Table object 362
Scan 4 types of the chains:
FREE CLUSTER CHAIN ANALYSIS
FREE CHAIN ANALYSIS
RM CHAIN ANALYSIS
INDEX DELETE CHAIN ANALYSIS
)
Dbanalys offline
Chanalys on/offline
[2015/10/19@13:18:16.960-0400] AREA "Order Area" : 9 BLOCK ANALYSIS
[2015/10/19@13:26:11.385-0400] 4221951 block(s) found in the area.
Sequential scan of all blocks in the area
Scan the Fifth type of the chains: “F-chains”
Timing of dbanalys run
proutil dbname -C dbanalys | \
while IFS="" read Line
do
echo `date '+%H:%M:%S'` "$Line"
done > dbname.dbanalys.log
10:19:52 INDEX DELETE CHAIN ANALYSIS
10:19:52 ----------------10:19:52 AREA "Order-line Area" : 7 BLOCK ANALYSIS
10:19:52 ------------------------------------------------10:19:52
11:55:37 13212671 block(s) found in the area.
11:55:37
11:55:37 Current high water mark: 13212671
11:55:37
DBAnalys To The Rescue
31
Timing of dbanalys run
Use ModPromsgs.p to create the promsgs.dba file:
ftp://ftp.progress-tech.ru/pub/Users/george/Programs/ModPromsgs.p
PROMSGS=$DLC/promsgs.dba
export PROMSGS
DB=dbname
Offset=`ls -l $DB.lg | awk '{print $5}’`
(proutil $DB -C dbanalys && tail +${Offset}c $DB.lg) >$DB.dbanalys.log
[2015/10/06@17:59:12.425-0400] P-10944600 T-1
[2015/10/06@18:01:20.312-0400] P-10944600 T-1
I
I
: (-----) AREA "Order Area" : 9 BLOCK ANALYSIS
: (-----) 2997247 block(s) found in the area.
127.887 sec = 00:02:07.887
DBAnalys To The Rescue
32
Free Cluster Chain Scan Time
[2015/09/29@13:13:48.269-0400] P-37552166 T-1 I Usr 19: (-----) CHAIN ANALYSIS FOR AREA "Misc1 Area" : 21
[2015/09/29@13:13:48.269-0400] P-37552166 T-1 I Usr 19: (-----) FREE CLUSTER CHAIN ANALYSIS
[2015/09/29@13:17:24.353-0400] P-37552166 T-1 I Usr 19: (-----) 56325 cluster(s) found in the free cluster chain.
[2015/09/29@14:55:51.512-0400] P-37552166 T-1 I Usr 19: (-----) CHAIN ANALYSIS FOR AREA "ic-ledger index Area" : 26
[2015/09/29@14:55:51.512-0400] P-37552166 T-1 I Usr 19: (-----) FREE CLUSTER CHAIN ANALYSIS
[2015/09/29@14:56:26.651-0400] P-37552166 T-1 I Usr 19: (-----) 5726 cluster(s) found in the free cluster chain.
[2015/09/29@17:06:05.698-0400] P-37552166 T-1 I Usr 19: (-----) CHAIN ANALYSIS FOR AREA "item-ledger Index Area" : 38
[2015/09/29@17:06:05.698-0400] P-37552166 T-1 I Usr 19: (-----) FREE CLUSTER CHAIN ANALYSIS
[2015/09/29@17:19:24.490-0400] P-37552166 T-1 I Usr 19: (-----) 257338 cluster(s) found in the free cluster chain.
DBAnalys To The Rescue
33
Free Chain Scan Time
DBAnalys To The Rescue
34
Free Chain Scan Time
DBAnalys To The Rescue
35
RM Chains Scan Time
[2015/10/19@10:25:01.825-0400]
[2015/10/19@11:43:01.389-0400]
[2015/10/19@11:56:02.210-0400]
[2015/10/19@12:39:30.437-0400]
[2015/10/19@12:49:03.141-0400]
[2015/10/19@13:18:16.944-0400]
[2015/10/19@13:26:12.032-0400]
[2015/10/19@13:35:16.730-0400]
[2015/10/19@13:38:06.351-0400]
[2015/10/19@13:44:57.849-0400]
[2015/10/19@13:45:48.519-0400]
[2015/10/19@13:53:49.885-0400]
P-14155958
P-14155958
P-27394110
P-27394110
P-11010220
P-11010220
P-7995430
P-7995430
P-37028030
P-37028030
P-31916152
P-31916152
T-1
T-1
T-1
T-1
T-1
T-1
T-1
T-1
T-1
T-1
T-1
T-1
I
I
I
I
I
I
I
I
I
I
I
I
Usr
Usr
Usr
Usr
Usr
Usr
Usr
Usr
Usr
Usr
Usr
Usr
19:
19:
19:
19:
19:
19:
19:
19:
19:
19:
19:
19:
(-----)
(-----)
(-----)
(-----)
(-----)
(-----)
(-----)
(-----)
(-----)
(-----)
(-----)
(-----)
RM CHAIN ANALYSIS
3755255 block(s) found in the RM chain of Table object 370
RM CHAIN ANALYSIS
2136418 block(s) found in the RM chain of Table object 63
RM CHAIN ANALYSIS
1402490 block(s) found in the RM chain of Table object 362
RM CHAIN ANALYSIS
185946 block(s) found in the RM chain of Table object 520
RM CHAIN ANALYSIS
177890 block(s) found in the RM chain of Table object 267
RM CHAIN ANALYSIS
150138 block(s) found in the RM chain of Table object 287
DBAnalys To The Rescue
36
AREA BLOCK ANALYSIS Scan Time
“F-chains” = the fragmented records
proutil db-name –C ixanalys
proutil db-name –C tabanalys
DBAnalys To The Rescue
37
Table Fragmentation Affects the Reading
Table Fragmentation Affects the Reading
Table Fragmentation Affects the Reading
DBAnalys To The Rescue
40
Connect All Fragments
DB-08: A Day in the Life of a Type II Record
Richard Banville
Exchange 2005
Rowid 9794
Block’s DBKEY
Type
Next DBKEY in Chain
Num
Dirs.
Free
Dirs.
Rec 2 Offset
Chain
Backup Ctr
Block Update Counter
Free Space
Rec 0 Offset
Rec 1 Offset
Rowid 2828
Rec n Offset
Free Space
Block’s DBKEY
Type
Chain
Backup Ctr
Used Data Space
Next DBKEY in Chain
Record 1
Num
Dirs.
Record 1
Rec 2 Offset
Record 0
Free
Dirs.
Block Update Counter
Free Space
Rec 0 Offset
Rec 1 Offset
Rowid 10240
Rec n Offset
Free Space
Block’s DBKEY
Type
Chain
Backup Ctr
Used Data Space
Next DBKEY in Chain
Record 1
Num
Dirs.
Free
Dirs.
Record 1
Rec 2 Offset
Record 0
Block Update Counter
Free Space
Rec 0 Offset
Rec 1 Offset
Rec n Offset
Free Space
Used Data Space
Record 1
Record 1
Record 0
Language Record Buffer (used by 4GL clients but not by dbanalys)
DBAnalys To The Rescue
41
DBAnalys To The Rescue!
Contents
History
Theory
Practice
Optimization
Synergy
DBAnalys To The Rescue
42
Record Packing Factor
RPB
One value per area is set in .st file. It is powers of 2 from 1 to 256.
Toss Limit The minimum amount of free space needed in a block for it to remain
on the RM Chain.
Default: 300 bytes (150 bytes for 1K block size)
Range: from 1 (from Create Limit) to (blocksize – 128)
Create Limit The minimum amount of free space required in a block to allow
creation of a new record starting in the block.
Default: 150 bytes (75 bytes for 1K block size)
Range: from 32 to (blocksize – 128)
-recspacesearchdepth
The number of RM blocks to move from the front to the back of
the RM chain before allocating a new cluster for data.
Default: 5 (clustersize / 10)
Range: from 0 to 100
DBAnalys To The Rescue
43
Create/Toss limits
proutil db -C dispTos[sCreateLimits] area*
proutil db -C setAreaCr[eateLimit] area* limit
proutil db -C setBlobCr[eateLimit] lob* limit (deprecated in V11)
proutil db -C setLobCre[ateLimit] lob* limit
proutil db -C setTableCr[eateLimit] table limit
proutil db -C setAreaTo[ssLimit] area* limit
proutil db -C setBlobTo[ssLimit] lob* limit (deprecated in V11)
proutil db -C setLobTos[sLimit] lob* limit
proutil db -C setTableTo[ssLimit] table limit
[ tenant tenant-name| group group-name| shared ]
* - For an area in V11 you can specify either the name or the number. And in V10 only the area number could
be specified. And the same for LOBs. For a table only the name could be specified in any version.
Proutil updates:
_StorageObject._Create-Limit
_StorageObject._Toss-Limit
DBAnalys To The Rescue
44
Toss Limit
proutil db -C chanalys “Order Area” -scan
RM CHAIN ANALYSIS
----------------LIST OF RM CHAIN BLOCKS
free
# free
dbkey
space
slots
hold
58719520
1337
5
58719536
8124
16
58719552
8124
16
0
0
0
58718960
58719456
0
0
...
682
669
5
5
1313186 block(s) found in the RM chain of Table object 362
DBAnalys To The Rescue
45
Number of blocks on the RM chain with free space above “X”
Toss Limit
DBAnalys To The Rescue
46
Number of blocks on the RM chain with free space above “X”
Toss Limit
What value to choose?
DBAnalys To The Rescue
47
Toss Limit
Rule of Thumb
Set toss limit = average record length + 10-20%
DBAnalys To The Rescue
48
Distribution of Free Space in RM Chain Inside the Area
DBAnalys To The Rescue
49
Distribution of Free Space in RM Chain Inside the Area
DBAnalys To The Rescue
50
Distribution of RM Chain Blocks Inside the Area
DBAnalys To The Rescue
51
Chains in dbrpr
DATABASE REPAIR MENU
-------------------1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
A.
P.
DATABASE SCAN MENU
------------------
Database Scan Menu
Test One or More Indexes
Remove Bad Record Fragment
Dump Block
Load Block
Copy Bytes Between Files
Load RM Dump File
Reformat Block to a Free Block
Change Current Working Area
Display the Free Chain
Display the RM Chain
Display the Index Delete Chain
Display Block Contents
Display Record Contents
Display Cluster Chain (Type II Area)
Scan/Fix block checksum (Type II Area)
Try to read additional Index/Table info
Print Info Menu
ON
ON
ON
ON
1.
3.
4.
5.
6.
7.
8.
9.
10.
11.
A.
Report Bad Blocks
Fix Bad Blocks
Report Bad Records
Delete Bad Records
Dump Records to RM File
Rebuild Free Chain
Rebuild RM Chain
Rebuild Index Delete Chain
Change Current Working Area
Fix Cluster Chains in Type II Area
Apply scan to all areas
G. Go
Q. Return to Main Menu
(Current Working Area: Customer/Order Area)
Q. Quit
DBAnalys To The Rescue
52
Long RM chain
proutil db-name -C dbrpr
dbanalys before dbrpr
1313186 block(s) found in the RM chain of Table object 362
Scan Backward (Yes/No)?
Scan of Order Area extent 1
Scan of Order Area extent 1
Scan of Order Area extent 1
Scan of Order Area extent 1
Scan of Order Area extent 1
Scan of Order Area extent 1
Scan of Order Area extent 1
Scan of Order Area extent 1
Scan of Order Area extent 1
Scan of Order Area extent 1
10% complete
20% complete
30% complete
40% complete
50% complete
60% complete
70% complete
80% complete
90% complete
100% complete
dbanalys after dbrpr
120813 block(s) found in the RM chain of Table object 362
DBAnalys To The Rescue
53
Create Limit: For now we see through a glass, darkly
promon db-name
Activity: Space Allocation
USA
KZ
RU1
RU2
LOAD
Per Sec
Per Sec
Per Sec
Per Sec
Per Sec
Database extends
0.29
4.94
7.02
4.99
120.13
Take free block
0.30
1.66
1.99
1.79
33.11
Return free block
0.01
0.04
0.18
0.06
0.00
Alloc rm space
53.69
196.27
130.62
387.88
2083.33
Alloc from rm
53.69
196.26
130.62
387.88
2070.93
Alloc from free
0.00
0.00
0.00
0.00
12.40
Bytes allocated
8538.09 17411.29 22496.94 38779.66 372915.41
rm blocks examined
55.69
198.46
131.42
389.71
2070.93
Remove from rm
2.02
4.23
12.44
12.88
99.21
Add to rm, front
0.0016
0.0129
0.0105
0.0065
12.40
Add to rm, back
0.47
1.11
7.14
9.66
0.00
Move rm front to back
1.18
1.55
0.05
1.42
0.00
Remove locked rm entry
14.54
18.50
20.60
180.48
0.00
Add to rm, front (blocks) 0.82
6.59
5.37
3.34
99.21
DBAnalys To The Rescue
54
Create Limit
a.k.a. Expansion Space Constant
proutil db -C setAreaCreateLimit area limit
proutil db -C setTableCreateLimit table limit
DBAnalys To The Rescue
55
Create Limit. The “Light Breathing” Space
RECORD BLOCK SUMMARY FOR AREA "item-summary Area" : 35
-------------------------------------------------------Record Size (B)- ---Fragments--- Scatter
Table
Records Size Min
Max Mean
Count Factor Factor
PUB.item-summary 34301938 15.2G 419 1066
474
35430542
1.0
1.0
1128604 records (3%) are fragmented.
proutil db -C chanalys "item-summary Area" -scan
CHAIN ANALYSIS FOR AREA "item-summary Area" : 35
Date: Mon Oct 19 13:45:48 2015
Date: Tue Oct 20 12:35:22 2015
150138 block(s) found in the RM chain
2288127 block(s) found in the area.
150460 block(s) found in the RM chain
2288127 block(s) found in the area.
15378
15258
234
0
556
322
blocks
blocks
blocks
blocks
blocks
blocks
were changed (10% of all blocks)
were updated
were removed from the chain
were added to the chain’s head
were added or moved to the bottom
= chain's increase
DBAnalys To The Rescue
56
Create Limit. The “Light Breathing” Space
proutil db -C chanalys "item-summary Area" –scan
RPB 16
LIST OF RM CHAIN BLOCKS
free
free
dbkey
space1 space2
36597840 8036
8035
-1
36597872 8037
8037
0
36597968 7705
7675 -30
36597984 8124
8124
0
36598016 8000
8000
0
36598064 8054
8054
0
36598160 7934
7919 -15
36598192 8124
8124
0
36598208 7990
7960
0
36598336 7935
7874 -61
36598432 8124
8124
0
36598448 6711
6711
0
# free
slots
15
15
12
16
14
15
14
16
15
14
16
13
LIST OF RM CHAIN BLOCKS
free
free
dbkey
space1 space2
35558848 7678
7669
-9
35558864 6879
6882
+3
36235728 7674
7674
0
36235744 7975
7976
+1
35558880 7436
7453 +17
36235760 7223
7223
0
35558896 6539
6540
+1
36235776 7213
7206
-7
35558912 6597
6664 +67
36235792 6506
6498
-8
35558928 7535
7558 +23
36235808 7247
7244
-3
DBAnalys To The Rescue
# free
slots
14
12
15
15
13
12
10
13
10
11
13
14
57
Create Limit. The “Heavy Breathing” Space
proutil db -C tabanalys "Order Area”
RECORD BLOCK SUMMARY FOR AREA "Order Area" : 9
-------------------------------------------------------Record Size (B)- ---Fragments--- Scatter
Table
Records Size Min
Max Mean
Count Factor Factor
PUB.order
40840212 29.1G 450 13519
764
49388337
1.2
1.0
8548125 records (21%) are
1396614 block(s) found in
4174847 block(s) found in
33% of data blocks are on
fragmented.
the RM chain of Table object 362
the area.
RM chain.
Chain reduced by 786 blocks (from 1,403,276 to 1,402,490).
HWM increased by 4096 blocks or by 8 clusters (from 4,221,951 to 4,226,047).
1329 blocks (0.1%) on RM chains were updated.
9 blocks were moved to the chain’s back.
22 records were deleted from blocks on RM chain.
DBAnalys To The Rescue
58
Create Limit. The “Heavy Breathing” Space
proutil db -C chanalys "Order Area" –scan
RPB 16
DBAnalys To The Rescue
59
Create Limit. Table’s Updates vs. Creates
Updates/Creates
0:1?
1:1?
many to any ?
DBAnalys To The Rescue
60
Create Limit. Updates Immediately After Creates
DO TRANSACTION:
CREATE Customer.
ASSIGN Customer.Name = “George”.
TRIGGER PROCEDURE
END. FOR Create OF Customer.
/* Automatically Increment Customer Number using Next-Cust-Num Sequence */
_TableStat-create: 1
ASSIGN Customer.Cust-Num = NEXT-VALUE(Next-Cust-Num).
_TableStat-update: 0 ???
ASSIGN Customer.Name = “George”.
END.
_TableStat-create: 1
_TableStat-update: 1
DBAnalys To The Rescue
61
Min Record Length
DBAnalys To The Rescue
62
Mean Record Length
DBAnalys To The Rescue
63
Create/Toss Limits
"Philosophy"
1.
2.
3.
No fragmentation, no long chains –> don’t change the create and toss limits.
Your efforts will be worthless.
First decide on the value of the create limit. Treat it as a "breathing" space for
all records stored in the same block rather then an expansion of an individual
record. Identify a type of record updates: frequent, only-after-create, almostnever.
The toss limit must be larger than the create limit. Use it to find a balance
between the efficient space usage inside a database and the keeping RM
chains relatively short. If it is impossible to reach the balance than just leave it
and relax.
Defragmentation
Features That Increase Planned Availability
 Online Schema Updates
 Auto-Defragging of the database
 Self Healing Attributes
• Truncate log file online
• Adding extents online
 Enabling AI Online
 SQL Revoke Privileges
 OpenEdge Replication Failback
63
DB-12: Atlantis, The Holy Grail, and 5-9’s: In Search of the Perfect World
DBAnalys To The Rescue
© 2007 Progress Software Corporation
65
AreaDefrag.p
Version 2.0 (“Precision weapon”)
1. Works online (4GL)
2. Defrags only records that are fragmented
3. Defrags one area at a time
4. Allows to specify a range of blocks
5. Could be multi-threaded
6. Could be interrupted
DBAnalys To The Rescue
66
AreaDefrag.p
Syntax:
RUN ScanArea( type, area-number, position, max-blocks)
Type:
TRUE – with defragmentation
FALSE – report only
Position:
Either from block-number or to block-number
Unknown value “?” Means HWM and
then max-blocks should be negative
DBAnalys To The Rescue
67
AreaDefrag.p
Examples:
RUN ScanArea( FALSE, 7, 1, 100000)
scan area 7 without defragmentation from the first block
forward 100 thousand blocks
RUN ScanArea( TRUE, 7, ?, -100000)
scan area 7 with defragmentation from HWM backwards
100 thousand blocks
DBAnalys To The Rescue
68
Testing results
for each <table> no-lock use-index <small-index>:
end.
Before defrag.p : 2 hr 19 min or 139 min
After defrag.p : 22 min or more then 6 times faster
proutil -C dump index 0
Before defrag.p : 102 min
After defrag.p : 10 min or more then 10 times faster
DBAnalys To The Rescue
69
DBAnalys To The Rescue!
Contents
History
Theory
Practice
Optimization
Synergy
DBAnalys To The Rescue
70
_TableStat
Dbanalys
_IndexStat
DBAnalys To The Rescue
71
synergy
DBAnalys To The Rescue
72
synergy
DBAnalys To The Rescue
73
synergy
Dbanalys
DBAnalys To The Rescue
74
synergy
DBAnalys To The Rescue
75
synergy
synergy
DBAnalys To The Rescue
77
synergy
DBAnalys To The Rescue
78
synergy
DBAnalys To The Rescue
79
George potemkin@progress-tech.ru
Dmitri d_levin@hotmail.com
DBAnalys To The Rescue
80
Thank you for your
time!