DBAnalys To The Rescue! George Potemkin Russia Dmitri Levin alphabroder St. Petersburg 1703 Philadelphia 1682 ftp://ftp.progress-tech.ru/pub/Users/george/DBAnalys1_Boston.pptx ftp://ftp.progress-tech.ru/pub/Users/george/DBAnalys2_Copenhagen.pptx DBAnalys To The Rescue! Contents History Theory Practice Optimization Synergy DBAnalys To The Rescue 2 PROGRESS Database Analysis Database: %s Options: ixanalys chanalys Date: %s No tabanalys option FREE CHAIN ANALYSIS =================== %l block(s) found in the free chain. V5 1988 RM CHAIN ANALYSIS ================= %l block(s) found in the RM chain. BLOCK ANALYSIS ============== %l RM block(s) found in the database. %l.%l% of the RM block space is used. %l index block(s) found in the database. %l.%l% of the index block space is used. Manning Road in Billerica, Massachusetts Index %s %d Blocks %l INDEX BLOCK SUMMARY % % of all Bytes Utiliz. Indexes %l %l.%l %l.%l DBAnalys To The Rescue 3 PROGRESS Database Analysis Database: Options: tabanalys tabanalys option Date: Mon Nov 6 18:26:53 1995 V7.3A BLOCK ANALYSIS (3906) ============== Nov, 9 1994 1 master block(s) found in the database. RECORD BLOCK SUMMARY (6129) ---Record Size--- ---Fragments--- Scatter Table Records Bytes Min Max Mean Count Index Index ----------------------------------------------------------<329 lines of boring stuff deleted> Totals: 15021963 642199k 6 1675 43 15275513 1.0 5.3 802392 RM block(s) found in the database. 86.10% of the RM block space is used. 270404 index block(s) found in the database. 400571 free block(s) found in the database. 5 index table block(s) found in the database. 1 sequence block(s) found in the database. 1473374 total blocks found in the database. 14 Oak Park in Bedford, Massachusetts DBAnalys To The Rescue 4 Segments of dbanalys output msg msg msg msg msg msg msg msg msg msg msg msg msg msg msg msg msg msg msg msg msg msg msg 3867: 3868: 8361: 14178: 6172: 3870: 14179: 14163: 14157: 14161: 3873: 3880: 14155: 16573: 16574: 3892: 3897: 14172: 7269: 7283: 14176: 10097: 3906: 1) 2) 2.1) 2.2) 2.3) 2.4) 3) 4) PROGRESS Database Analysis Database: %s Blocksize: %i RecSpaceSearchDepth: %d Options: %s %s %s Date: %s%r CHAIN ANALYSIS FOR AREA "%s" : %l FREE CLUSTER CHAIN ANALYSIS LIST OF FREE CLUSTER CHAIN BLOCKS %J cluster(s) found in the free cluster chain. FREE CHAIN ANALYSIS LIST OF FREE CHAIN BLOCKS %J block(s) found in the free chain of %s object %i %rNumber of Object Object Partition/Tenant/Group Blocks Type RM CHAIN ANALYSIS LIST OF RM CHAIN BLOCKS %J block(s) found in the RM chain of %s object %i INDEX DELETE CHAIN ANALYSIS LIST OF INDEX DELETE CHAIN BLOCKS %J block(s) found in the Index Delete chain of %s object %i AREA "%s" : %d BLOCK ANALYSIS BLOCK ANALYSIS (moved to the end at V11) DBAnalys To The Rescue 5 Segments of dbanalys output msg msg msg msg msg msg msg msg msg msg msg msg msg msg msg msg 6129: 14212: 16667: 16668: 16656: 3921: 3925: 14225: 16657: 16658: 3923: 2314: 16666: 16659: 16579: 3937: 5) 6) 7) 8) msgs 0000-6130 msgs 7270-9999 msgs 9894-10130 msgs 14155-14225 msgs 16641-17035 RECORD BLOCK SUMMARY RECORD BLOCK SUMMARY FOR AREA "%s" : %l RECORD BLOCK SUMMARY FOR SHARED TABLES RECORD BLOCK SUMMARY FOR %s %s: %l Summary for AREA "%s": %l %J RM block(s) found in the database. INDEX BLOCK SUMMARY INDEX BLOCK SUMMARY FOR AREA "%s" : %l INDEX BLOCK SUMMARY FOR SHARED OBJECTS: INDEX BLOCK SUMMARY FOR %s: %s %J index block(s) found in the database. DATABASE SUMMARY SUMMARY FOR AREA "%s": %l SUMMARY FOR SHARED OBJECTS: DATABASE BLOCK ANALYSIS: database analysis complete %s%r before 8.2A 8.3A 9.1D 10.2B 11.1 DBAnalys To The Rescue 6 History of changes V9.0A proutil db-name -C dbanalys [ [area] area-name ] Nov 21, 1998 V9.0A proutil db-name -C dbanalys –Bp 64 Nov 21, 1998 V9.1A Jan 21, 2000 Index Delete Chains / LOCKCHN (bk_frchn 2) V10.0A Free Cluster Chains / CLIST_FREECHN (bk_frchn 3) Dec 13, 2003 DBAnalys To The Rescue 7 History of changes V10.1A02 Algorithm for calculation of Scatter Factor for SAT II Oct 27, 2006 tables has changed Scatter for Type II is reported as a number of blocks occupied divided by the minimum number of blocks that table could occupy. Scatter for Type I is reported as log10 of sum of distances between records in a table, where distance is calculated as the difference between the RECID values for the two records, divided by minimum distance possible. In V11 that formula lost the meaning again when the database is multi-tenant or has partitioning. DBAnalys To The Rescue 8 History of changes V10.1B02 Aug 3, 2007 V10.1B02 Aug 3, 2007 -recspacesearchdepth The number of RM blocks to move from the front to the back of the RM chain before allocating a new cluster for data. Default is 5 * (cluster size / 10). Max value is 100. Old algorithm: No more than 3 blocks at a time will be examined and moved to the tail end. Full report on chains when running chanalys online. The following lists will be reported: LIST OF FREE CLUSTER CHAIN BLOCKS LIST OF FREE CHAIN BLOCKS LIST OF RM CHAIN BLOCKS LIST OF INDEX DELETE CHAIN BLOCKS DBAnalys To The Rescue 9 History of changes V10.1B03 Nov 2, 2007 proutil <db> -C chanalys [ -scan ] -scan: fewer locks, less consistent It instructs chanalys to use minimal locking. This may cause the chains report to be incomplete , but it will avoid turning on table locks. Offline chanalys does not use a locking mechanism, the -scan parameter will be ignored. [Warning] Unable to acquire table lock on object 2. Chain analysis continuing with -scan option. (14167) LIST OF FREE CHAIN BLOCKS next dbkey free 51 52 CTRL-C pressed - exit free chain analysis DBAnalys To The Rescue 10 History of changes Database Analysis (dbanalys) output has changed. 10.2B06 March, 2012 In the past customer's complained that the DBANALYS report widths were not wide enough resulting in a CR after the table name when the table name exceeded the Tablename column width. Now the tabanalys width is 110 chars. V11 Dance of the broken lines (the ixanalys width is 86 chars) INDEX BLOCK SUMMARY FOR SHARED OBJECTS: -------------------------------------------Table Index Fields Levels PUB.theTable IntField IntField901234567890123456789012 10 1 IntField-1:3 2 IntField-10:2 2 DBAnalys To The Rescue Blocks Size % Util Factor 215 215 1.6M 1.6M 98.2 98.2 1.0 1.0 11 History of changes V11.4 Jul 25, 2014 proutil db-name -C dbanalys [ [area] area-name ] [ tenant tenant-name| group group-name| shared ] [-csoutput [ -fieldsep sep-value][ -csfilePrefix pref-name ] [ -verbose ]] -csoutput -fieldsep sep-value (default is space ) -csfilePrefix pref-name (default db-name) -verbose proutil sports –C dbanalys -csoutput -verbose DBAnalys To The Rescue 12 Output files -csfilePrefix V11.6 sports.block.txt sports.ch.txt sports.ix.txt sports.tbl.txt sports.lob.txt -verbose "Time Stamp" "DBUID" "DB name" "Area" "Area #" "Partition Type" "Tenancy" "Tenancy #" "Partition Policy" "Partition" "Partition #" "Owner" "Table" "Table #" "Records" "Size" "Min" "Max" "Mean“ 2015-04-06T14:59:00.000-4:00 "njVjvK5a8qfvEyE+7KM4WA" "sports" "Customer/Order Area" 8 "-" "-" 0 "-" "-" 0 "PUB" "Customer" -fieldsep 2 83 12377 118 222 149 83 1.0 1.0 2015-04-06T14:59:00.000-4:00 "njVjvK5a8qfvEyE+7KM4WA" "sports" "Customer/Order Area" 8 "-" "-" 0 "-" "-" 0 "PUB" "Order" 4 207 11730 53 61 56 207 1.0 1.0 DBAnalys To The Rescue 13 Loading sports.tbl.txt in Excel DBAnalys To The Rescue 14 Loading sports.ix.txt in Excel DBAnalys To The Rescue 15 History of changes 11.6 Oct 26, 2015 LOB support in PROUTIL DBANALYS The PROUTIL database analysis utilities are enhanced to include analysis of LOB data. When LOB data is found in an area, a “LOB Summary” follows the “Record Block Summary” in the output, and LOB data is included in the combined summary. LOB data is also written to a new *lob.txt file when the -cscoutput parameter is used on the analysis command line. DBAnalys To The Rescue 16 DBAnalys To The Rescue! Contents History Theory Practice Optimization Synergy DBAnalys To The Rescue 17 Space: The Final Frontier… 20 Years After Disk Space Allocation in the Progress RDBMS G. Bjorklund, posted on PEG: Aug 15, 1995 Space: The Final Frontier, Engine Crew Monograph No. 2 Gus Bjorklund, Aug 10, 1997 http://www.fast4gl.com/downloads/monographs/space/space.html Space: the final frontier, OpenEdge RDBMS Internals Americas PUG Challenge 2012 , Westford, MA Gus Bjorklund, May 9, 2012 https://community.progress.com/community_groups/openedge_rdbms/w/openedgerdbms/791.space-thefinal-frontier OpenEdge 10 RDBMS Advanced Storage Architecture Richard Banville, May, 2004 https://community.progress.com/community_groups/user_conferences/w/exchange/2138.exchange-2004breakout-sessions DBAnalys To The Rescue 18 Space: The Final Frontier… 20 Years After Block Types (dbrpr) 1. Master Block 2. Index Block 3. Data Block 4. Free Block 5. Index Dbkey List Block 6. Sequence Block 7. Empty Block 8. Parameter Block (Not Used) 9. Area Block 10. Object Directory Block (Not Used) 11. Extent List Block (Not Used) 12. Object Block 13. Control Block 14. Area Object List Block 15. Area Cluster Allocation Block 16. Object Cluster List Block 17. Object Block Allocation Block 18. RM bit Map List Block 19. Extent Header Block (bk_type 254) 20. Reserved Block There are 19 known block types. 5 block types do not (should not) exist in database. 4 block types do exist in database but they are not used. 16 block types are mentioned by dbanalys before V11 and 14 block types by V11 dbanalys. DBAnalys To The Rescue 19 Block types in dbanalys AREA "misc128_dat": 16 BLOCK ANALYSIS ------------------------------------------------4629499 block(s) found in the area. Current high water mark: 4629439 56 free block(s) found in the area 4627892 record block(s) found in the area 0 index block(s) found in the area 60 empty block(s) found in the area 372 object block(s) found in the area 372 cluster list block(s) found in the area 2 object list block(s) found in the area 372 cluster map block(s) found in the area 0 row allocation block(s) found in the storage area. DATABASE BLOCK ANALYSIS: ----------------------66432530 RM block(s) found in the database. 17906763 index block(s) found in the database. 1 master block(s) found in the database. 38 area block(s) found in the database. 1 control block(s) found in the database. 2 object block(s) found in the database. 2905 cluster list block(s) found in the database. 2905 cluster allocation block(s) found in the database. 2905 object block(s) found in the database. 42 object list block(s) found in the database. 2905 object allocation block(s) found in the database. 91506 free block(s) found in the database. 0 index table block(s) found in the database. 1 sequence block(s) found in the database. 692370 empty block(s) found in the database. DBAnalys To The Rescue 20 Types of Storage Areas Blocks could be in the areas of the following types: 1. Storage Area Type I 2. Storage Area Type II 3. Storage Area Type III I + II Hybrid (V10.0B) Used for temp-tables ( DBI file ) DBAnalys To The Rescue 21 Data Clusters Storage Area Type II Three possible cluster sizes: 8, 64, or 512 adjacent blocks Clusters are allocated from the area and given to a particular object. They are maintained by a doubly linked list. First and last blocks in the cluster have the extended headers (80 bytes instead of 64). First block stores Owner Info (transactionId, serialNumber) Last block stores Cluster Link (firstFreeCluster, lastFreeCluster) Database extend: MAX(64, cluster size) – that means that database will be extended for 1 cluster if the cluster size is 64 or 512 and 8 clusters if cluster size is 8. DBAnalys To The Rescue 22 Chain Types 1. Cluster chains 2. Block chains ;-) 3. F-chains Cluster Chains Block Chains 1. Area Control Clusters 1. Free Chains (FREECHN, bk_frchn 0) 2. Data Object Clusters 2. RM Chains (RMCHN, bk_frchn 1) 3. Free Cluster Chains (CLIST_FREECHN) 3. Index Delete Chains (LOCKCHN, bk_frchn 2) F-Chains DBAnalys To The Rescue Z unofficial name ;-) 23 Free Cluster Chains proutil sports –C dbrpr 15. Display Cluster Chain (Type II Area) Cluster Link of object 1/4 Total Clusters: 2 First cluster: 768 Last cluster: 1536 768 1536 1536 0 Clusters of the“Order” table (object 1/4) 2 cluster(s) found and its “Cust-Order” index (object 2/21). Cluster Link of object 2/21 Total Clusters: 1 First cluster: 1024 1024 0 1 cluster(s) found Last cluster: 1024 Fast Object Deletion Area Free Cluster Link First cluster: 1024 Last cluster: 1536 1024 768 Free cluster chain after the “Order” table 768 1536 and its “Cust-Order” index were deleted. 1536 0 3 cluster(s) found DBAnalys To The Rescue 24 Free Cluster Chains CHAIN ANALYSIS FOR AREA "Customer/Order Area" : 8 ---------------------------------------------------------FREE CLUSTER CHAIN ANALYSIS --------------------------LIST OF FREE CLUSTER CHAIN BLOCKS next dbkey free 1024 768 768 1536 1536 0 3 cluster(s) found in the free cluster chain. DBAnalys To The Rescue 25 Free Cluster Chains DBAnalys To The Rescue 26 Block Chains proutil db –C chanalys [–scan] LIST OF FREE CHAIN BLOCKS next dbkey free 7026440 7026439 2046268 block(s) found in the free chain of Index object 22 LIST OF INDEX DELETE CHAIN BLOCKS dbkey Block 722 2499 3 block(s) found in the Index Delete chain of Index object 33 LIST OF RM CHAIN BLOCKS next dbkey Block 804512704 2372 49 0 8534342 block(s) found in the RM chain of Table object 4 DBAnalys To The Rescue 27 Index Delete chain Index Object Block Free Block Free Block Free Block Free Block Ixdel Block Ixdel Block Ixdel Block Ixdel Block (Unique Indexes only) 20731 block(s) found in the Index Delete chain of Index object 622 proutil db-name -C idxcompact item-tax.item-group 0 block(s) found in the Index Delete chain of Index object 622 DBAnalys To The Rescue 28 DBAnalys To The Rescue! Contents History Theory Practice Optimization Synergy DBAnalys To The Rescue 29 Dbanalys Processing Phases [2015/10/19@12:49:02.602-0400] CHAIN ANALYSIS FOR AREA "Order Area" : 9 [2015/10/19@13:18:16.944-0400] 1402490 block(s) found in the RM chain of Table object 362 Scan 4 types of the chains: FREE CLUSTER CHAIN ANALYSIS FREE CHAIN ANALYSIS RM CHAIN ANALYSIS INDEX DELETE CHAIN ANALYSIS ) Dbanalys offline Chanalys on/offline [2015/10/19@13:18:16.960-0400] AREA "Order Area" : 9 BLOCK ANALYSIS [2015/10/19@13:26:11.385-0400] 4221951 block(s) found in the area. Sequential scan of all blocks in the area Scan the Fifth type of the chains: “F-chains” Timing of dbanalys run proutil dbname -C dbanalys | \ while IFS="" read Line do echo `date '+%H:%M:%S'` "$Line" done > dbname.dbanalys.log 10:19:52 INDEX DELETE CHAIN ANALYSIS 10:19:52 ----------------10:19:52 AREA "Order-line Area" : 7 BLOCK ANALYSIS 10:19:52 ------------------------------------------------10:19:52 11:55:37 13212671 block(s) found in the area. 11:55:37 11:55:37 Current high water mark: 13212671 11:55:37 DBAnalys To The Rescue 31 Timing of dbanalys run Use ModPromsgs.p to create the promsgs.dba file: ftp://ftp.progress-tech.ru/pub/Users/george/Programs/ModPromsgs.p PROMSGS=$DLC/promsgs.dba export PROMSGS DB=dbname Offset=`ls -l $DB.lg | awk '{print $5}’` (proutil $DB -C dbanalys && tail +${Offset}c $DB.lg) >$DB.dbanalys.log [2015/10/06@17:59:12.425-0400] P-10944600 T-1 [2015/10/06@18:01:20.312-0400] P-10944600 T-1 I I : (-----) AREA "Order Area" : 9 BLOCK ANALYSIS : (-----) 2997247 block(s) found in the area. 127.887 sec = 00:02:07.887 DBAnalys To The Rescue 32 Free Cluster Chain Scan Time [2015/09/29@13:13:48.269-0400] P-37552166 T-1 I Usr 19: (-----) CHAIN ANALYSIS FOR AREA "Misc1 Area" : 21 [2015/09/29@13:13:48.269-0400] P-37552166 T-1 I Usr 19: (-----) FREE CLUSTER CHAIN ANALYSIS [2015/09/29@13:17:24.353-0400] P-37552166 T-1 I Usr 19: (-----) 56325 cluster(s) found in the free cluster chain. [2015/09/29@14:55:51.512-0400] P-37552166 T-1 I Usr 19: (-----) CHAIN ANALYSIS FOR AREA "ic-ledger index Area" : 26 [2015/09/29@14:55:51.512-0400] P-37552166 T-1 I Usr 19: (-----) FREE CLUSTER CHAIN ANALYSIS [2015/09/29@14:56:26.651-0400] P-37552166 T-1 I Usr 19: (-----) 5726 cluster(s) found in the free cluster chain. [2015/09/29@17:06:05.698-0400] P-37552166 T-1 I Usr 19: (-----) CHAIN ANALYSIS FOR AREA "item-ledger Index Area" : 38 [2015/09/29@17:06:05.698-0400] P-37552166 T-1 I Usr 19: (-----) FREE CLUSTER CHAIN ANALYSIS [2015/09/29@17:19:24.490-0400] P-37552166 T-1 I Usr 19: (-----) 257338 cluster(s) found in the free cluster chain. DBAnalys To The Rescue 33 Free Chain Scan Time DBAnalys To The Rescue 34 Free Chain Scan Time DBAnalys To The Rescue 35 RM Chains Scan Time [2015/10/19@10:25:01.825-0400] [2015/10/19@11:43:01.389-0400] [2015/10/19@11:56:02.210-0400] [2015/10/19@12:39:30.437-0400] [2015/10/19@12:49:03.141-0400] [2015/10/19@13:18:16.944-0400] [2015/10/19@13:26:12.032-0400] [2015/10/19@13:35:16.730-0400] [2015/10/19@13:38:06.351-0400] [2015/10/19@13:44:57.849-0400] [2015/10/19@13:45:48.519-0400] [2015/10/19@13:53:49.885-0400] P-14155958 P-14155958 P-27394110 P-27394110 P-11010220 P-11010220 P-7995430 P-7995430 P-37028030 P-37028030 P-31916152 P-31916152 T-1 T-1 T-1 T-1 T-1 T-1 T-1 T-1 T-1 T-1 T-1 T-1 I I I I I I I I I I I I Usr Usr Usr Usr Usr Usr Usr Usr Usr Usr Usr Usr 19: 19: 19: 19: 19: 19: 19: 19: 19: 19: 19: 19: (-----) (-----) (-----) (-----) (-----) (-----) (-----) (-----) (-----) (-----) (-----) (-----) RM CHAIN ANALYSIS 3755255 block(s) found in the RM chain of Table object 370 RM CHAIN ANALYSIS 2136418 block(s) found in the RM chain of Table object 63 RM CHAIN ANALYSIS 1402490 block(s) found in the RM chain of Table object 362 RM CHAIN ANALYSIS 185946 block(s) found in the RM chain of Table object 520 RM CHAIN ANALYSIS 177890 block(s) found in the RM chain of Table object 267 RM CHAIN ANALYSIS 150138 block(s) found in the RM chain of Table object 287 DBAnalys To The Rescue 36 AREA BLOCK ANALYSIS Scan Time “F-chains” = the fragmented records proutil db-name –C ixanalys proutil db-name –C tabanalys DBAnalys To The Rescue 37 Table Fragmentation Affects the Reading Table Fragmentation Affects the Reading Table Fragmentation Affects the Reading DBAnalys To The Rescue 40 Connect All Fragments DB-08: A Day in the Life of a Type II Record Richard Banville Exchange 2005 Rowid 9794 Block’s DBKEY Type Next DBKEY in Chain Num Dirs. Free Dirs. Rec 2 Offset Chain Backup Ctr Block Update Counter Free Space Rec 0 Offset Rec 1 Offset Rowid 2828 Rec n Offset Free Space Block’s DBKEY Type Chain Backup Ctr Used Data Space Next DBKEY in Chain Record 1 Num Dirs. Record 1 Rec 2 Offset Record 0 Free Dirs. Block Update Counter Free Space Rec 0 Offset Rec 1 Offset Rowid 10240 Rec n Offset Free Space Block’s DBKEY Type Chain Backup Ctr Used Data Space Next DBKEY in Chain Record 1 Num Dirs. Free Dirs. Record 1 Rec 2 Offset Record 0 Block Update Counter Free Space Rec 0 Offset Rec 1 Offset Rec n Offset Free Space Used Data Space Record 1 Record 1 Record 0 Language Record Buffer (used by 4GL clients but not by dbanalys) DBAnalys To The Rescue 41 DBAnalys To The Rescue! Contents History Theory Practice Optimization Synergy DBAnalys To The Rescue 42 Record Packing Factor RPB One value per area is set in .st file. It is powers of 2 from 1 to 256. Toss Limit The minimum amount of free space needed in a block for it to remain on the RM Chain. Default: 300 bytes (150 bytes for 1K block size) Range: from 1 (from Create Limit) to (blocksize – 128) Create Limit The minimum amount of free space required in a block to allow creation of a new record starting in the block. Default: 150 bytes (75 bytes for 1K block size) Range: from 32 to (blocksize – 128) -recspacesearchdepth The number of RM blocks to move from the front to the back of the RM chain before allocating a new cluster for data. Default: 5 (clustersize / 10) Range: from 0 to 100 DBAnalys To The Rescue 43 Create/Toss limits proutil db -C dispTos[sCreateLimits] area* proutil db -C setAreaCr[eateLimit] area* limit proutil db -C setBlobCr[eateLimit] lob* limit (deprecated in V11) proutil db -C setLobCre[ateLimit] lob* limit proutil db -C setTableCr[eateLimit] table limit proutil db -C setAreaTo[ssLimit] area* limit proutil db -C setBlobTo[ssLimit] lob* limit (deprecated in V11) proutil db -C setLobTos[sLimit] lob* limit proutil db -C setTableTo[ssLimit] table limit [ tenant tenant-name| group group-name| shared ] * - For an area in V11 you can specify either the name or the number. And in V10 only the area number could be specified. And the same for LOBs. For a table only the name could be specified in any version. Proutil updates: _StorageObject._Create-Limit _StorageObject._Toss-Limit DBAnalys To The Rescue 44 Toss Limit proutil db -C chanalys “Order Area” -scan RM CHAIN ANALYSIS ----------------LIST OF RM CHAIN BLOCKS free # free dbkey space slots hold 58719520 1337 5 58719536 8124 16 58719552 8124 16 0 0 0 58718960 58719456 0 0 ... 682 669 5 5 1313186 block(s) found in the RM chain of Table object 362 DBAnalys To The Rescue 45 Number of blocks on the RM chain with free space above “X” Toss Limit DBAnalys To The Rescue 46 Number of blocks on the RM chain with free space above “X” Toss Limit What value to choose? DBAnalys To The Rescue 47 Toss Limit Rule of Thumb Set toss limit = average record length + 10-20% DBAnalys To The Rescue 48 Distribution of Free Space in RM Chain Inside the Area DBAnalys To The Rescue 49 Distribution of Free Space in RM Chain Inside the Area DBAnalys To The Rescue 50 Distribution of RM Chain Blocks Inside the Area DBAnalys To The Rescue 51 Chains in dbrpr DATABASE REPAIR MENU -------------------1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. A. P. DATABASE SCAN MENU ------------------ Database Scan Menu Test One or More Indexes Remove Bad Record Fragment Dump Block Load Block Copy Bytes Between Files Load RM Dump File Reformat Block to a Free Block Change Current Working Area Display the Free Chain Display the RM Chain Display the Index Delete Chain Display Block Contents Display Record Contents Display Cluster Chain (Type II Area) Scan/Fix block checksum (Type II Area) Try to read additional Index/Table info Print Info Menu ON ON ON ON 1. 3. 4. 5. 6. 7. 8. 9. 10. 11. A. Report Bad Blocks Fix Bad Blocks Report Bad Records Delete Bad Records Dump Records to RM File Rebuild Free Chain Rebuild RM Chain Rebuild Index Delete Chain Change Current Working Area Fix Cluster Chains in Type II Area Apply scan to all areas G. Go Q. Return to Main Menu (Current Working Area: Customer/Order Area) Q. Quit DBAnalys To The Rescue 52 Long RM chain proutil db-name -C dbrpr dbanalys before dbrpr 1313186 block(s) found in the RM chain of Table object 362 Scan Backward (Yes/No)? Scan of Order Area extent 1 Scan of Order Area extent 1 Scan of Order Area extent 1 Scan of Order Area extent 1 Scan of Order Area extent 1 Scan of Order Area extent 1 Scan of Order Area extent 1 Scan of Order Area extent 1 Scan of Order Area extent 1 Scan of Order Area extent 1 10% complete 20% complete 30% complete 40% complete 50% complete 60% complete 70% complete 80% complete 90% complete 100% complete dbanalys after dbrpr 120813 block(s) found in the RM chain of Table object 362 DBAnalys To The Rescue 53 Create Limit: For now we see through a glass, darkly promon db-name Activity: Space Allocation USA KZ RU1 RU2 LOAD Per Sec Per Sec Per Sec Per Sec Per Sec Database extends 0.29 4.94 7.02 4.99 120.13 Take free block 0.30 1.66 1.99 1.79 33.11 Return free block 0.01 0.04 0.18 0.06 0.00 Alloc rm space 53.69 196.27 130.62 387.88 2083.33 Alloc from rm 53.69 196.26 130.62 387.88 2070.93 Alloc from free 0.00 0.00 0.00 0.00 12.40 Bytes allocated 8538.09 17411.29 22496.94 38779.66 372915.41 rm blocks examined 55.69 198.46 131.42 389.71 2070.93 Remove from rm 2.02 4.23 12.44 12.88 99.21 Add to rm, front 0.0016 0.0129 0.0105 0.0065 12.40 Add to rm, back 0.47 1.11 7.14 9.66 0.00 Move rm front to back 1.18 1.55 0.05 1.42 0.00 Remove locked rm entry 14.54 18.50 20.60 180.48 0.00 Add to rm, front (blocks) 0.82 6.59 5.37 3.34 99.21 DBAnalys To The Rescue 54 Create Limit a.k.a. Expansion Space Constant proutil db -C setAreaCreateLimit area limit proutil db -C setTableCreateLimit table limit DBAnalys To The Rescue 55 Create Limit. The “Light Breathing” Space RECORD BLOCK SUMMARY FOR AREA "item-summary Area" : 35 -------------------------------------------------------Record Size (B)- ---Fragments--- Scatter Table Records Size Min Max Mean Count Factor Factor PUB.item-summary 34301938 15.2G 419 1066 474 35430542 1.0 1.0 1128604 records (3%) are fragmented. proutil db -C chanalys "item-summary Area" -scan CHAIN ANALYSIS FOR AREA "item-summary Area" : 35 Date: Mon Oct 19 13:45:48 2015 Date: Tue Oct 20 12:35:22 2015 150138 block(s) found in the RM chain 2288127 block(s) found in the area. 150460 block(s) found in the RM chain 2288127 block(s) found in the area. 15378 15258 234 0 556 322 blocks blocks blocks blocks blocks blocks were changed (10% of all blocks) were updated were removed from the chain were added to the chain’s head were added or moved to the bottom = chain's increase DBAnalys To The Rescue 56 Create Limit. The “Light Breathing” Space proutil db -C chanalys "item-summary Area" –scan RPB 16 LIST OF RM CHAIN BLOCKS free free dbkey space1 space2 36597840 8036 8035 -1 36597872 8037 8037 0 36597968 7705 7675 -30 36597984 8124 8124 0 36598016 8000 8000 0 36598064 8054 8054 0 36598160 7934 7919 -15 36598192 8124 8124 0 36598208 7990 7960 0 36598336 7935 7874 -61 36598432 8124 8124 0 36598448 6711 6711 0 # free slots 15 15 12 16 14 15 14 16 15 14 16 13 LIST OF RM CHAIN BLOCKS free free dbkey space1 space2 35558848 7678 7669 -9 35558864 6879 6882 +3 36235728 7674 7674 0 36235744 7975 7976 +1 35558880 7436 7453 +17 36235760 7223 7223 0 35558896 6539 6540 +1 36235776 7213 7206 -7 35558912 6597 6664 +67 36235792 6506 6498 -8 35558928 7535 7558 +23 36235808 7247 7244 -3 DBAnalys To The Rescue # free slots 14 12 15 15 13 12 10 13 10 11 13 14 57 Create Limit. The “Heavy Breathing” Space proutil db -C tabanalys "Order Area” RECORD BLOCK SUMMARY FOR AREA "Order Area" : 9 -------------------------------------------------------Record Size (B)- ---Fragments--- Scatter Table Records Size Min Max Mean Count Factor Factor PUB.order 40840212 29.1G 450 13519 764 49388337 1.2 1.0 8548125 records (21%) are 1396614 block(s) found in 4174847 block(s) found in 33% of data blocks are on fragmented. the RM chain of Table object 362 the area. RM chain. Chain reduced by 786 blocks (from 1,403,276 to 1,402,490). HWM increased by 4096 blocks or by 8 clusters (from 4,221,951 to 4,226,047). 1329 blocks (0.1%) on RM chains were updated. 9 blocks were moved to the chain’s back. 22 records were deleted from blocks on RM chain. DBAnalys To The Rescue 58 Create Limit. The “Heavy Breathing” Space proutil db -C chanalys "Order Area" –scan RPB 16 DBAnalys To The Rescue 59 Create Limit. Table’s Updates vs. Creates Updates/Creates 0:1? 1:1? many to any ? DBAnalys To The Rescue 60 Create Limit. Updates Immediately After Creates DO TRANSACTION: CREATE Customer. ASSIGN Customer.Name = “George”. TRIGGER PROCEDURE END. FOR Create OF Customer. /* Automatically Increment Customer Number using Next-Cust-Num Sequence */ _TableStat-create: 1 ASSIGN Customer.Cust-Num = NEXT-VALUE(Next-Cust-Num). _TableStat-update: 0 ??? ASSIGN Customer.Name = “George”. END. _TableStat-create: 1 _TableStat-update: 1 DBAnalys To The Rescue 61 Min Record Length DBAnalys To The Rescue 62 Mean Record Length DBAnalys To The Rescue 63 Create/Toss Limits "Philosophy" 1. 2. 3. No fragmentation, no long chains –> don’t change the create and toss limits. Your efforts will be worthless. First decide on the value of the create limit. Treat it as a "breathing" space for all records stored in the same block rather then an expansion of an individual record. Identify a type of record updates: frequent, only-after-create, almostnever. The toss limit must be larger than the create limit. Use it to find a balance between the efficient space usage inside a database and the keeping RM chains relatively short. If it is impossible to reach the balance than just leave it and relax. Defragmentation Features That Increase Planned Availability Online Schema Updates Auto-Defragging of the database Self Healing Attributes • Truncate log file online • Adding extents online Enabling AI Online SQL Revoke Privileges OpenEdge Replication Failback 63 DB-12: Atlantis, The Holy Grail, and 5-9’s: In Search of the Perfect World DBAnalys To The Rescue © 2007 Progress Software Corporation 65 AreaDefrag.p Version 2.0 (“Precision weapon”) 1. Works online (4GL) 2. Defrags only records that are fragmented 3. Defrags one area at a time 4. Allows to specify a range of blocks 5. Could be multi-threaded 6. Could be interrupted DBAnalys To The Rescue 66 AreaDefrag.p Syntax: RUN ScanArea( type, area-number, position, max-blocks) Type: TRUE – with defragmentation FALSE – report only Position: Either from block-number or to block-number Unknown value “?” Means HWM and then max-blocks should be negative DBAnalys To The Rescue 67 AreaDefrag.p Examples: RUN ScanArea( FALSE, 7, 1, 100000) scan area 7 without defragmentation from the first block forward 100 thousand blocks RUN ScanArea( TRUE, 7, ?, -100000) scan area 7 with defragmentation from HWM backwards 100 thousand blocks DBAnalys To The Rescue 68 Testing results for each <table> no-lock use-index <small-index>: end. Before defrag.p : 2 hr 19 min or 139 min After defrag.p : 22 min or more then 6 times faster proutil -C dump index 0 Before defrag.p : 102 min After defrag.p : 10 min or more then 10 times faster DBAnalys To The Rescue 69 DBAnalys To The Rescue! Contents History Theory Practice Optimization Synergy DBAnalys To The Rescue 70 _TableStat Dbanalys _IndexStat DBAnalys To The Rescue 71 synergy DBAnalys To The Rescue 72 synergy DBAnalys To The Rescue 73 synergy Dbanalys DBAnalys To The Rescue 74 synergy DBAnalys To The Rescue 75 synergy synergy DBAnalys To The Rescue 77 synergy DBAnalys To The Rescue 78 synergy DBAnalys To The Rescue 79 George potemkin@progress-tech.ru Dmitri d_levin@hotmail.com DBAnalys To The Rescue 80 Thank you for your time!