What are you waiting for? A reason for waiting around. Richard Banville Fellow, Progress Software June 7, 2011 Playing Fair and Being Consistent Why do I need to wait? • Shared resource synchronization Implementation restrictions • Wait on only one resource • Concurrent waits – Timer interrupt (timeout processing) – Screen/process interrupt – Network “table1 in use by user1 on p/97. Wait or press CTRL-C to stop.” Focus is on resource waits • How & why we wait 2 © 2011 Progress Software Corporation. All rights reserved. What can a user wait on Some you can avoid; Some are out of your control. • Undo/rollback Database Resource • Internal resource – Shared data structure Schema changes (some) – Buffer lock Quiet point – TXE lock Maintenance activity • Checkpoint • BI/AI (buffer, I/O…) • Transaction end – Unique key insertion • Record lock • Database extends • Backup • Table/index move/activate Replication (pica) Screen input (select) Network message (poll) • Page writers System calls • Index operation I/O (low level) – Search, split 3 • Login (semaphore) © 2011 Progress Software Corporation. All rights reserved. Etc. Wait Mechanisms Latches • Used for “short term” waits • Waiting user NOT visible (activity & waits are visible) • Test, test-and-set (atomically), “nap”, try again Semaphores • Used for “long term” waits • Waiting user visible via promon (VSTs) • Often can be controlled at application level – Example: record lock duration • Sometimes not – Example: I/O duration • Try, queue, wait for explicit wakeup 4 © 2011 Progress Software Corporation. All rights reserved. Semaphores 5 © 2011 Progress Software Corporation. All rights reserved. Semaphores Wait on semaphore summary • “Long term” wait synchronization mechanism • Always server side • Unix semaphore info: – ipcs -sa – -semsets: min 1, max 32,000, default 3 – Kernel control o SEMMNS - max semaphores available system wide o SEMMNI - max semaphore sets o SEMMSL - max semaphores per set • Semaphore usage – Ready semaphore – db startup – Login semaphore – connect/disconnect – Resource wait semaphore – one per connection 6 © 2011 Progress Software Corporation. All rights reserved. Semaphores -semsets (3) #1 #2 26 Connection sems = -n 20 + -Mn 4 + 1 (broker) + 1 (proshut) Ready Login 5 spares User1 User3 User5 … • What could I be waiting on? – Ready/Login semaphore – Schema lock wait – Record lock wait – Buffer lock wait (db, ai, bi) • Ordered wait/wake-up (fifo queue) – Add to queue – Decrement semaphore #3 – OS suspend state User2 User4 User6 … – When resource available, semaphore “ping’ed” – OS activates process again • High semaphore waits – Application contention – Buffer contention 7 © 2011 Progress Software Corporation. All rights reserved. Promon R&D: Other Display 12/07/10 Activity: Other Wait on semaphore Total 1545 Per Min 1862 Per Sec 31.00 Per Tx 0.21 Wait on semaphore summary • High waits can indicate a potential for improvement – Increase resources – Alter application o Improve queries o Avoid full table scans o Alter lock behavior – Physical changes (Use TII storage areas) • Direct connections only • Zero and monitor • Contrast with a more happy time 8 © 2011 Progress Software Corporation. All rights reserved. Performance Indicators (summary) 12/07/10 Activity: Performance Indicators Total waits Lock waits Resource waits Latch timeouts Total 1558 337 1221 1625 Per Min 1870 404 1465 1950 Per Sec 31.16 6.74 24.42 32.50 Per Tx 0.23 0.05 0.18 0.24 • Total waits – Lock waits + Resource waits • Lock waits (application) – Record lock – Schema lock – Transaction end (unique key insertion) • Latch timeouts – #times –spin exhausted 9 © 2011 Progress Software Corporation. All rights reserved. • Resource waits (internal) – Row get lock – Buffered lock in shared memory (db, ai, bi) – Buffer I/O (db, ai, bi) – Micro-transaction end lock – Latches (queue latches) Promon: Other Display vs Performance Indicators 12/07/10 Activity: Other Wait on semaphore 12/07/10 Total 1545 Per Min 1862 Per Sec 31.00 Per Tx 0.21 Activity: Performance Indicators Total waits Lock waits Resource waits Total 1558 337 1221 Per Min 1870 404 1465 Per Sec 31.16 6.74 24.42 Per Tx 0.23 0.05 0.18 Semaphore waits != Total waits – Race condition – stats not (always) latched – Client server record waits – no semaphore wait – Semaphore latch waits – no lock/resource wait – “DB Intent” waits counted twice – excl buffer wait 10 © 2011 Progress Software Corporation. All rights reserved. Promon: Other Display vs Performance Indicators 12/07/10 Activity: Other Total Wait on semaphore 1545 14 * Non-blocking waits 1 * Semaphore latch waits 12/07/10 Per Sec 31.00 0.16 0.00 Per Tx 0.21 0.02 0.00 Activity: Performance Indicators Total waits Lock waits Resource waits 11 Per Min 1862 8 0 Total 1558 337 1221 © 2011 Progress Software Corporation. All rights reserved. Per Min Per Sec 1870 31.16 404 6.74 1465 24.42 Per Tx 0.23 0.05 0.18 Wait Identification 12 © 2011 Progress Software Corporation. All rights reserved. Locks vs Locks Waits 12/07/10 Usr 0 5 6 User Name richb richb richb Other: Performance Indicators: Lock Requests By User --- Record --- ---- Trans --- --- Schema --Locks Waits Locks Waits Locks Waits 0 0 0 0 0 0 12 0 1 0 0 0 222 10 10 0 0 0 Locks from “Performance Indicators” option • Broken out by user • Can indicate a performance issue • Record locks/waits • Transaction locks/waits • Schema locks/waits 13 © 2011 Progress Software Corporation. All rights reserved. Recovery Waits 12/07/10 Activity: BI Log Total Busy buffer waits 10 Empty buffer waits 1 Log force waits 0 Per Min 2 0 0 Per Sec 0.02 0.00 0.00 Per Tx 0.20 0.00 0.00 • Busy buffer waits – # of waits for exclusive BI buffer access – Wait for other user to finish BI buffer update/write – Only seen with page writers (due to MTX latch) • Empty buffer waits – # of waits because all buffers were full – Increase –bibufs and run with biw • Log force waits – # of waits for “commit notes” to be written to disk – Obsolete 14 © 2011 Progress Software Corporation. All rights reserved. BI Buffer Pool – Recording a change Forward Processing -bibufs 10 Free List free - a Current Output Buffer Modified Queue 32 31 free - b free - c 30 New Notes (Actions) 29 free - d free - e 15 © 2011 Progress Software Corporation. All rights reserved. BI PROMON: Total BI Writes Records (notes) written Partial Writes - Busy buffer waits - increase bibufs - Empty buffer waits - increase bibufs - biw, apw Label Confusion 12/07/10 Usr Name 7 richb 8 richb 12/07/10 Usr Name 7 richb 8 richb Status: Blocked Clients Type SELF/ABL SELF/ABL Wait Trans id TRAN 2053 REC 32768 2104 2224 R&D Blocked Clients Status: Blocked Clients Type SELF/ABL SELF/ABL Wait Wait Info Trans id TRAN 2053 2104 REC 32768 2224 Current trid Not blocking trid User Control: -- Wait Info -Usr Name Type Wait Table Dbkey Trans 0 dba BROK -0 0 0 1 dba SERV -0 0 0 6 bob SELF/ABL REC 2 385 2108 24 joe REMC/ABL TRAN 0 2053 2261 16 © 2011 Progress Software Corporation. All rights reserved. Display Users Blocked Clients: Record Lock 12/07/10 Usr Name 7 richb Status: Blocked Clients Type Wait Wait Info Trans id SELF/ABL REC 1282 2104 REC – Waiting on a record lock • “Record Lock” (Activity lock description) • Wait info: Record’s rowid • No table information • Record lock wait in Lock Table option has more detail – Including table information • Easy to identify if “hung” 18 © 2011 Progress Software Corporation. All rights reserved. Record Lock Waits 12/07/10 Status: Lock Table Usr Name Trans ID 6 richb 121759 7 richb 121761 Type REC REC Rowid Rec-id Table 8199 2 8199 2 Flags: Lock request Flags X S QH Tran State Begin Dead (Table only) • S – Shared • X – Exclusive • SX – Shared/Exclusive(?) • IS – Intent shared • IX – intent exclusive • SIX – SHR w/intent EXCL Flags: Lock state • U – Upgrade, L - Limbo, Q - Queued, P - Purge, H – No Hold Tran State • Dead, Begin, Active, Prep, Phase 1, Phase 2 Record lock queuing mechanism X 19 SQ SQ © 2011 Progress Software Corporation. All rights reserved. XQ S S XQ SQ Locking Code Examples find first customer. pause message "about to assign". assign name = "xxx“. S X find last customer. assign cust-num = 39999. release customer. pause. S find first customer exclusive-lock. delete customer. X UQ X X IX Deadlock Alert (on conflict) (no conflict) Limbo record lock L Record lock (‘til purged) L P Limbo Table lock Purge lock until released internally pause message “deleted!”. NOTE: Record locks released Limbo table lock: Serialization Isolation Cooperation (since record lock(s) released) 20 © 2011 Progress Software Corporation. All rights reserved. Lock Waits Statistics 12/07/10 Activity: Lock Table Total Per Min Per Sec Per Tx Waits: Share 59 *Intent Share 0 Exclusive 218 *Intent Exclusive 0 *Share Intent Excl 0 Upgrade 0 Record Get Lock 0 2 0 9 0 0 0 0 0.04 0.00 0.15 0.00 0.00 0.00 0.00 0.01 0.00 0.04 0.00 0.00 0.00 0.00 0 11 0.00 0.19 0.00 0.05 Table Lock Record Lock 0 277 Nice overall wait summary • Helps identify application performance issues *Table locks only 21 © 2011 Progress Software Corporation. All rights reserved. Blocked Clients: Record Get 12/07/10 Usr Name 7 richb Status: Blocked Clients Type Wait Wait Info Trans id SELF/ABL RGET 1282 2104 RGET – Record “get lock” • "Record Get Lock" • Wait info: Record’s rowid • No-lock wait User 6 1. Modify/delete record ( buffer state change) • • 22 User 7 2. 3. No-lock read a record Frag1 Read record fragment #2 Must wait for user6 shm buffer update to get consistent record. New optimistic locking – only needed for fragmented records. © 2011 Progress Software Corporation. All rights reserved. Blocked Clients: Buffer Locks 12/07/10 Usr Name 7 richb 8 richb Status: Blocked Clients Type Wait Wait Info Trans id SELF/ABL BKSH 1280 2104 SELF/ABL BKEX 192 2234 BKSH & BKEX – Database buffer locks • Wait info: Block identifier (dbkey) of block • Area information would be useful • “DB Buf S Lock” - OR – “DB Buf X Lock” • 12 (BKS2), 13 (BKX2) “DB Buf S/X Lock LRU2” – Alternate buffer pool lock Catch all (sequence(192), object block, etc) Record updates: • Record lock first, then buffer lock “Rarely” lock multiple buffers – deadlock avoidance 23 © 2011 Progress Software Corporation. All rights reserved. Record Locator (indicator, block #, row #) Rowid Consists of 3 parts (32 bit example) Block number: 1 to whatever Row #: 0 to 255 (based on RPB) 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Looking for ROWID 42562, Table 1, Area 7 w/64 RPB setting Block # (in area): 665 Row #: 2 Block identifier (dbkey): 42560 Rowid: 42562 (Row # as zeros) (Row # included) 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 1 1 0 0 1 00 00 00 00 10 1 00 Block number: 1 to 2^25 24 © 2011 Progress Software Corporation. All rights reserved. Row #: 0 to 63 Online Backup Wait Online “point in time” backup - can result in BKEX waits 25 © 2011 Progress Software Corporation. All rights reserved. Online Backup Wait Online “point in time” backup - can result in BKEX waits Area 6 Area 7 Area 8 … BI Modify request Modify request Online Backup User 1 Wait… BFP BKEX Backup queue MTX TXE BFP Back it up and mark it • • 26 Wait on (TXE) semaphore during bi backup Wait on BKEX during data backup © 2011 Progress Software Corporation. All rights reserved. .bak Blocked Clients: Read/Write I/O 12/07/10 Usr Name 7 richb Status: Blocked Clients Type Wait Wait Info Trans id SELF/ABL DBRD 1280 2104 DBRD & DBWR – Database read/write • “DB Buf Read” – OR – “DB Buf Write” • Wait info: Block identifier DBW2 (14) • Same as DBWR but for alternate buffer pool BIRD, BIWR, AIRD, AIWR – Bi/Ai read/write • “BI Buf Read” –OR – “BI Buf Write” • Wait info: bi/ai block # (not particularly useful to you) • Can use to determine if waiting on different block (not hung) 27 © 2011 Progress Software Corporation. All rights reserved. Buffer Pool Access BKSH & BKEX vs DBRD & DBWR Flush/overwrite for paging Updating different records in same block User3 Record read/update User1 User2 BK Buffer lock Block in memory DB Buffer lock DB APW DBRD – reading in – all must wait! Flush for performance DBWR – Writing out, cannot be changed BKSH – In memory, read attempt BKEX – In memory, change attempt • 28 Pretty much same mechanism for all buffer pools (-B, -B2, -bibuf, -aibuf) © 2011 Progress Software Corporation. All rights reserved. Blocked Clients: Special Buffer Lock 12/07/10 Usr Name 7 richb Status: Blocked Clients Type Wait Wait Info Trans id SELF/ABL DBSI 1280 2104 DBSI – Intent to update buffer lock • “DB Buf I Lock” • Wait info: Block identifier (dbkey) of conflict • Used by index manager when deleting or inserting entries • “Safe” locking protocol 29 © 2011 Progress Software Corporation. All rights reserved. Insert an Index Key Entry - Shared root Block’s DBKEY Type Next DBKEY in Chain Top Bot Num Entries Chain – DBSI buffer lock – “crabbing” mechanism – Release previous node – Increases concurrency Backup Ctr Block Update Counter Index Reserved No. Bytes Dummy Entry . . . Used . . . Compressed Index Entries . . . ooo . . . Compressed Index Entries . . . Free Space - Shared -DBSI internal (on split) Block’s DBKEY Type Chain Backup Ctr Block’s DBKEY Type Chain Backup Ctr Next DBKEY in Chain Block Update Counter Block’s DBKEY Type Chain Backup Ctr Next DBKEY in Chain Block Update Counter Top Bot Index Reserved Next DBKEY in Chain Block Update Counter Top Bot Index Reserved No. Num Bytes Dummy Entry . . . No. Top Bot Index Reserved Num Bytes Dummy EntryUsed ... Entries No. . . . Compressed Index Entries . . . EntriesEntry . . . Used Num Bytes Dummy . . . Compressed Index Entries . . . Entries Used . . . Compressed Index Entries . . . ooo ooo ooo . . . Compressed Index Entries . . . . . . Compressed Index Entries . . . . . . Compressed Index Entries . . . Free Space Free Space Free Space - DBSI records 30 Type Top Bot Num Entries Chain Backup Ctr Block Update Counter Index Reserved No. Bytes Dummy Entry . . . Used . . . Compressed Index Entries . . . ooo . . . Compressed Index Entries . . . Free Space Block’s DBKEY Type Chain Backup Ctr Block’s DBKEY Type Chain Backup Ctr Next DBKEY in Chain Block Update Counter Type Chain Backup Ctr Block’s DBKEY Type Chain Next DBKEY in Chain Block Update Counter Top Bot Index Reserved Block’s DBKEY Type Chain Backup Ctr Next DBKEY in Chain Block Update Counter Next DBKEY in Chain Block Update Counter Top Bot Index Reserved No. Block’s DBKEY Type Chain Backup Ctr Num Bytes Dummy Entry . . . Next DBKEY in Chain Block Update Counter Top Bot Index Reserved No. Block’s DBKEY Type Chain Backup Ctr Top Bot Index Reserved Entries Used Num Bytes Dummy Entry ... No. Next DBKEY in Chain Block Update Counter . . . Compressed Index Entries . . . Top Bot Index Reserved No. Num Bytes Dummy Entry ... Entries Used Next DBKEY in Chain Block Update Counter Num Bytes Dummy Entry . . . . . . Compressed Index Entries . . . Entries Used Top Bot Index Reserved No. Entries Used Num Bytes Dummy Entry ... . . . Compressed Index Entries . . . Top Bot Index Reserved No. . . . Compressed Index Entries . . . Entries Used Num Bytes Dummy Entry . . . ooo No. . . . Compressed Index Entries . . . Num Bytes Entries Dummy Used Entry . . . ooo . . . Compressed Index Entries . . . Entries Used ooo ooo . . . Compressed Index Entries . . . . . . Compressed Index Entries . . . ooo . . . Compressed Index Entries . . . ooo . . . Compressed Index Entries . . . Free Space . . . Compressed Index Entries . . . ooo Free Space . . . Compressed Index Entries . . . Free Space Free Space . . . Compressed Index Entries . . . Free Space . . . Compressed Index Entries . . . Free Space Free Space Block’s DBKEY Backup Ctr leaf Block’s DBKEY Next DBKEY in Chain © 2011 Progress Software Corporation. All rights reserved. Blocked Clients: Buffer Availability 12/07/10 Usr Name Status: Blocked Clients Type Wait Wait Info Trans id NOBF (11) – Wait for available buffer in -B • Wait statistic accumulated • Actual wait never seen in “Blocked Client” • Wait info: None • Shows up as “DB Buf Avail” in resource waits • Wait mechanism – Sleep for a second, then retry – 5 attempts then hard failure – Error message: “Not enough database buffers (-B)” 31 © 2011 Progress Software Corporation. All rights reserved. Blocked Clients: Transaction End 12/07/10 Usr Name 7 richb Status: Blocked Clients Type Wait Wait Info Trans id SELF/ABL TRAN 2053 2104 Wait info is actually a trid TRAN – Wait for transaction to commit • “Trans Commit" • Wait info: transaction Id waiting on • Can occur when adding a unique index key entry User 6 User 7 1. Begin transaction 3. Begin transaction 2. Delete unique key abc@my.net (or insert) 4. Insert unique key abc@my.net Must wait for user6 to commit or rollback. 32 © 2011 Progress Software Corporation. All rights reserved. Blocked Clients: Micro-transaction 12/07/10 Usr Name 7 richb Status: Blocked Clients Type Wait Wait Info Trans id SELF/ABL TXE 1 2104 TXE - Micro-transaction wait • A micro-transaction? • Wait info: – 1: Share – buffer read/write – 2: Update – multi-buffer change – 3: Commit – commit note & txn table sync – 4: Exclusive – Quiet point and backup • No commit TXE if multi-buffer update already outstanding 33 © 2011 Progress Software Corporation. All rights reserved. Blocked Clients: Schema Lock 12/07/10 Usr Name 7 richb Status: Blocked Clients Type Wait Wait Info Trans id SELF/ABL SCH 10 2104 SCH – Schema lock • “Schema Lock" • Values – 9 - Share lock request – 10 - Exclusive lock request User 6 User 7 1. Login – cache schema 3. Change Schema (w/o online) 2. Share schema lock 4. Exclusive schema lock “Database in use by richb on pts/274. Wait or press CTRL-C to stop.” 34 © 2011 Progress Software Corporation. All rights reserved. Blocked Clients: Miscellaneous 12/07/10 Status: Blocked Clients Usr Name Type Wait Wait Info Trans id 7 richb SELF/ABL SRPL 0 2104 SRPL – OpenEdge Site Replication • "Repl TEND Ack“ • Waiting for TEND ACK from a Sync Repl Agent • Schema lock request • Wait info: none. DBSQ – Service Queue • "DBSQ Send Lock“ • Database service queue message wait. User waiting to queue IPC message to database service queue • Wait info: none. STCA – Statement Cache 35 • Statement caching wait • Wait info: none. © 2011 Progress Software Corporation. All rights reserved. Other Major Waits 36 © 2011 Progress Software Corporation. All rights reserved. Other Major Waits: Checkpoint 12/07/10 Other: Checkpoints Ckpt ------ Database Writes -----No. Time Len Freq Dirty CPT Q Scan APW Q Flushes 2 09:34:57 317 0 170 0 0 0 0 1 09:14:19 1238 1238 272 0 0 0 72 12/07/10 Usr Name 7 richb Status: Blocked Clients Type Wait Wait Info Trans id SELF/ABL TXE 3 2104 Checkpoint – sync in memory changes • TXE – lock – Prevents transaction commit • MTX – latch – Prevents db changes • BIB – latch – Prevents BIW writes • Wait not definitive; Latch waits not shown in “Blocked Clients” 37 © 2011 Progress Software Corporation. All rights reserved. Other Major Waits: Quietpoint 12/07/10 Usr Name 7 richb Status: Blocked Clients Type Wait Wait Info Trans id SELF/ABL TXE 1 2104 Quietpoint • Many wait examples – shared TXE share wait shown above • .lg file – “Quiet point has been enabled by the broker” • TXE – exclusive lock – Prevents data access • MTX – latch – Prevents db changes • BIB/AIB – latch – Prevents BIW/AIW writes 38 © 2011 Progress Software Corporation. All rights reserved. Summary Many different waits conditions Identifying waits • Help id performance bottlenecks Understanding wait conditions • Help remove performance bottlenecks 39 © 2011 Progress Software Corporation. All rights reserved. ? Questions 40 © 2011 Progress Software Corporation. All rights reserved. 41 © 2011 Progress Software Corporation. All rights reserved. Sept. 19 – 22, 2011 Boston Westin Waterfront Hotel and Boston Convention & Exhibition Center Table Lock Types 43 Lock Type Allow Block Intent Share (IS) Row S Table X Intent Exclusive (IX) Row X Table X Shared (S) Row S Row X Shared with Intent Exclusive (SIX) Row S, X other Row X Exclusive (X) Row S, X any other access © 2011 Progress Software Corporation. All rights reserved. Table Lock Compatibility Lock Request Type Other Active Locks 44 IS IX S SIX X None OK OK OK OK OK IS OK OK OK -Q- -Q- IX OK OK -Q- -Q- -Q- S OK -Q- OK -Q- -Q- SIX OK -Q- -Q- -Q- -Q- X -Q- -Q- -Q- -Q- -Q- © 2011 Progress Software Corporation. All rights reserved. Progress Implementation Isolation Level Read Uncommitted Read Committed Repeatable Read Serializable 45 Select Table Record Lock Lock None Intent Share Intent Share Share © 2011 Progress Software Corporation. All rights reserved. None Insert/Update Table Record Lock Lock Not Allowed Not Allowed Share Intent Exclusive Exclusive Share Intent Exclusive Exclusive None Share Intent Exclusive Exclusive