PeopleSoft Instrumentation
for Oracle RDBMS
David Kurtz
Go-Faster Consultancy Ltd.
david.kurtz@go-faster.co.uk
www.go-faster.co.uk
Who Am I?
• Oracle Database Specialist
– Independent consultant
• Performance tuning
– PeopleSoft ERP
– Oracle RDBMS
• Book
– www.psftdba.com
•
•
OakTable
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
2
Instrumentation
Data! Data! Data! I can't make bricks without
clay.
The Adventures of Sherlock Holmes, Arthur Conan-Doyle
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
3
Instrumentation
• PeopleTools
– SQL Trace
– Batch Timings
– Performance Monitor
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
• Oracle
– Dynamic Performance
Views (V$)
– SQL Trace
– Automatic Workload
Repository (AWR)
– Active Session History
(ASH)
©2013 www.go-faster.co.uk
4
ASH: A Brief Overview
• Samples active sessions every second
• Circular buffer in memory
– v$active_Session_history
– It should hold about 1 hour of data
• 1 in 10 samples stored in database
– DBA_HIST_ACTIVE_SESS_HISTORY
– Flushed out during AWR snapshot
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
5
What does ASH retain?
• Most of the columns are on v$session
– Session
• Session ID and serial, query coordinator
– Wait
• event id, name and parameters
– SQL
• SQL_ID, plan hash, opcode, plan line
– Object
• object, file, block and row numbers
– Application
• module, action …
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
7
ASH –v- SQL*Trace
• ASH
–
–
–
–
–
–
–
–
–
Licensed (Diagnostics)
Always there
Real Time
No marginal overhead
Who is blocking me?
Statistical data
SQL ID
Plan if captured by AWR
Estimate of duration
• Per wait event
• Per Plan Operation 11g
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
• SQL*Trace
–
–
–
–
–
–
–
–
–
Free
Enable, File, Profile
Reactive
Run-time overhead
Being Blocked
Every SQL & event
SQL HASH
Actual execution plan
Exact duration
• Operations in Plan
©2013 www.go-faster.co.uk
8
ASH –v- SQL*Trace
• ASH can be used to resolve many of your
performance issues.
• Sometimes, you will still need SQL*Trace
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
9
Application Instrumentation
• It is essential to be able to match
– database sessions
– application processes
• DBMS_APPLICATION_INFO
– set_module, set_action
– Calls in application
• Introduced in PeopleTools 8.50-8.52.
– Appears in SQL*Trace and ASH
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
10
CLIENT_INFO
– PeopleTools 7.53
– Every application server service calls
•
•
•
DBMS_APPLICATION_INFO.SET_CLIENT_INFO
V$SESSION.CLIENT_INFO
Operator ID, …
– Inserts into Audit tables can be done by database
trigger.
•
Trigger uses READ_CLIENT_INFO to obtain PeopleSoft
OPRID
– Not copied to ASH
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
11
CLIENT_ID
• PeopleTools 8.52
• Also set to PeopleSoft OPRID.
• Client_ID is copied to ASH
– So you can identify the PS operator
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
12
PIA Instrumentation
• PeopleTools 8.50
• Module = Component
• Action = Page
– Or XYZZY for search dialogue
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
13
IB Instrumentation
• PeopleTools 8.50
• Module = Service
• Action = Queue
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
14
Application Engine
• PeopleTools 8.50
• Module = ‘PSAE’
• Action = Program
– This is not good
– See http://blog.psftdba.com/2009/03/usingoracle-enterprise-manager-grid.html
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
15
Application Engine
• PeopleTools 8.52
• Module = ‘PSAE.’||AE program
name||session id
– session id is stored in
PSPRCSQUE.SESSIONID_NUM
• Action = Program.Section.Step.Actions
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
16
Cobol / SQR
• No instrumentation
• DIY with trigger
– on Process Scheduler request table
– http://www.gofaster.co.uk/scripts.htm#psftapi.sql
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
17
OEM
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
18
ASH Report in OEM
• You can run ASH reports via EM
– This example is for a specific module
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
19
Enable Trace by Module/Action
• Like setting a watchpoint in a debugger.
• See http://blog.psftdba.com/2012/09/oracle-sqltracing-by-module-action.html
• Can set SQL trace for a specific
component/page.
• But you can’t predict the value of module for
AE so may need to go back to the trigger based
solution.
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
20
Ability to Profile SQL duration
by Process/Process Instance
PROGRAM
-------------------PSQRYSRV
oracle
rman
PSAESRV
rman
FSPCCURR
PSAPPSRV
...
MODULE
ASH_SECS
-------------------------------- ---------PSQRYSRV
633520
oracle
328630
backup archivelog
236050
GL_JEDIT
203120
backup incr datafile
132840
FSPCCURR
119350
VCHR_EXPRESS
111520
---------2982340
sum
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
21
Profile of Whole System
WITH x AS (
SELECT /*+leading(x h) use_nl(h)*/
CASE WHEN h.module IS NULL
THEN REGEXP_SUBSTR(h.program, '[^@]+',1,1)
WHEN h.module LIKE 'PSAE.%' THEN REGEXP_SUBSTR(h.module, '[^.]+',1,2)
WHEN h.module LIKE '%.%'
THEN REGEXP_SUBSTR(h.module, '[^.]+',1,1)
WHEN h.module LIKE '%@%'
THEN REGEXP_SUBSTR(h.module, '[^@]+',1,1)
ELSE h.module END AS module
, REGEXP_SUBSTR(h.program, '[^.@]+',1,1) program
, h.event
FROM dba_hist_snapshot x
,
dba_hist_active_sess_history h
WHERE h.SNAP_id = X.SNAP_id
AND
h.dbid = x.dbid
AND
h.instance_number = x.instance_number
AND
x.end_interval_time
>= TO_DATE('20130730', 'YYYYMMDD')
AND
x.begin_interval_time <= TO_DATE('20130807', 'YYYYMMDD')
AND
h.sample_time BETWEEN TO_DATE('20130730','YYYYMMDD')
AND TO_DATE('20130807','YYYYMMDD')
)
SELECT program, module, sum(10) ash_Secs
FROM x
GROUP BY program, module
ORDER BY ash_secs DESC
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
Snapshots for Period of
interest
ASH Data for period of
interest
Logic to trim module name at
@ or .
©2013 www.go-faster.co.uk
22
Batch Timings by Exec Time
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
23
ASH Profile a Single Process
Process
Instance
--------51159995
51159995
51159995
51159995
51159995
51159995
51159995
51159995
51159995
51159995
51159995
51159995
51159995
51159995
SQL_ID
------------fpj0dmac7rw20
2qnfa4afzvj91
1u37a5hzzc2z6
grhj2f0dcbd70
a6nrbg7sbx401
49yzw3xa50r1t
0wcx4syn7tjk2
44sjax92d6kqa
cm5xjajkfjvkg
djur79tkd6uum
cb1cankq47ky9
1awa6cr95ug3t
03p8ga7kazdg4
5yrur2nv3nn2p
SQL Plan
Hash Value
---------207497280
416825695
768682620
2964999827
721249526
2665694782
1678621177
3752318859
3814124734
2504040148
362952569
3137593480
2504040148
2663443920
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
Exec
ACTION
secs
ASH_SECS
-------------------------------- ----- ---------AR_POSTING.HSS_PYMT.ITEMS.S
2797
410
AR_POSTING.HS_PYMNT.ITEMS2.S
2797
320
AR_POSTING.HS_PYMNT.ITEMS.S
2797
270
AR_POSTING.HSS_DSO.BACK_OUT.S
2797
190
AR_POSTING.HSS_PYMT.UPD_STAT.S
2797
170
AR_POSTING.HS_USER.INSERTS.S
2797
160
AR_POSTING.HS_USER.UPDATES.S
2797
150
AR_POSTING.HS_DSO.ADJ_BAL.S
2797
130
AR_POSTING.HSS_USR1.INSERTS.S
2797
130
AR_POSTING.HS_USER.UPD_ACTV.S
2797
130
AR_POSTING.HSS_USR1.UPDATES.S
2797
130
AR_POSTING.HS_PYMNT.UPD_STAT.S
2797
110
AR_POSTING.HSS_USER.UPD_ACTV.S
2797
100
AR_POSTING.HSS_DSO.NO_ACTIV.S
2797
80
©2013 www.go-faster.co.uk
24
How profile a process (i)
Hint to guarantee sensible
WITH x AS (
join order, and partition
SELECT /*+LEADING(r q x h)*/ h.sql_id, h.sql_plan_hash_value, h.action
elimination
,
SUM(10) ash_secs
,
10*count(t.sql_id) awr_secs
FROM
sysadm.psprcsrqst r
Process Request Table
,
sysadm.psprcsque q
One Row per Process
,
dbA_hist_snapshot x
,
dba_hist_active_Sess_history h
LEFT OUTER JOIN dba_hist_sqltext t ON t.sql_id = h.sql_id
Process Request Queue Table
WHERE x.END_INTERVAL_TIME >= r.begindttm
One Row per Process
AND
x.begin_interval_time <= r.enddttm
AND
h.sample_time BETWEEN r.begindttm AND r.enddttm
AND
h.SNAP_id = X.SNAP_id
ASH data for period for which
AND
h.dbid = x.dbid
process was running
AND
h.instance_number = x.instance_number
AND
q.prcsinstance = r.prcsinstance
AND
h.module = 'PSAE.'||r.prcsname||'.'||q.sessionidnum
Filter ASH data by module
AND
r.prcsname = 'AR_UPDATE2'
AND
r.prcsinstance = 51159995
Specific Process Request
GROUP BY h.sql_id, h.sql_plan_hash_value, h.action) …
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
25
How profile a process (ii)
Ranking by SQL Plan Hash
Value
…
), y AS (
SELECT ROW_NUMBER() OVER (PARTITION BY x.sql_plan_hash_value
order byPlan
DB Time for Execution
x.awr_secs desc) as ranking
, x.sql_id, x.sql_plan_hash_value, x.action
AWR secs 0 if SQL not
, SUM(x.ash_secs) OVER (PARTITION BY x.sql_plan_hash_value)
captured in ASH repository
tot_ash_secs
, SUM(x.awr_secs) OVER (PARTITION BY x.sql_plan_hash_value)
tot_awr_secs
, COUNT(distinct sql_id) OVER (PARTITION BY x.sql_plan_hash_value)
sql_ids
FROM x
)
Select * from y
where y.ranking = 1
AND tot_ash_secs > 30
ORDER BY tot_ash_secs DESC
/
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
26
Profile a Single Process
Process
Instance
--------51159995
51159995
51159995
51159995
51159995
51159995
51159995
51159995
51159995
51159995
51159995
51159995
51159995
51159995
SQL_ID
------------fpj0dmac7rw20
2qnfa4afzvj91
1u37a5hzzc2z6
grhj2f0dcbd70
a6nrbg7sbx401
49yzw3xa50r1t
0wcx4syn7tjk2
44sjax92d6kqa
cm5xjajkfjvkg
djur79tkd6uum
cb1cankq47ky9
1awa6cr95ug3t
03p8ga7kazdg4
5yrur2nv3nn2p
SQL Plan
Hash Value
---------207497280
416825695
768682620
2964999827
721249526
2665694782
1678621177
3752318859
3814124734
2504040148
362952569
3137593480
2504040148
2663443920
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
Exec
ACTION
secs
ASH_SECS
-------------------------------- ----- ---------AR_POSTING.HSS_PYMT.ITEMS.S
2797
410
AR_POSTING.HS_PYMNT.ITEMS2.S
2797
320
AR_POSTING.HS_PYMNT.ITEMS.S
2797
270
AR_POSTING.HSS_DSO.BACK_OUT.S
2797
190
AR_POSTING.HSS_PYMT.UPD_STAT.S
2797
170
AR_POSTING.HS_USER.INSERTS.S
2797
160
AR_POSTING.HS_USER.UPDATES.S
2797
150
AR_POSTING.HS_DSO.ADJ_BAL.S
2797
130
AR_POSTING.HSS_USR1.INSERTS.S
2797
130
AR_POSTING.HS_USER.UPD_ACTV.S
2797
130
AR_POSTING.HSS_USR1.UPDATES.S
2797
130
AR_POSTING.HS_PYMNT.UPD_STAT.S
2797
110
AR_POSTING.HSS_USER.UPD_ACTV.S
2797
100
AR_POSTING.HSS_DSO.NO_ACTIV.S
2797
80
©2013 www.go-faster.co.uk
27
Now investigate the SQL
• Extract SQL from AWR repository with the
dbms_xplan package
– Obtaining and Interpreting Execution Plans
using DBMS_XPLAN
• http://www.gofaster.co.uk/ukougpres.htm#Intro_DBMS_XPLAN
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
28
SELECT * FROM table( dbms_xplan.display_awr(
'duu7n4uzsthxz', 768682620,NULL,'ADVANCED'));
SQL_ID duu7n4uzsthxz
-------------------INSERT INTO PS_HS_CLSITM_TMP4 (…)
SELECT 51164249 , I.BUSINESS_UNIT ,…
FROM PS_ITEM I
,PS_ITEM_ACTIVITY A
,PS_POST_REQ_TMP4 R
WHERE R.PROCESS_INSTANCE = 51164249
AND R.PROCESS_P_HIST ='Y'
-AND I.BUSINESS_UNIT = R.BUSINESS_UNIT
AND I.ITEM_STATUS = 'C'
AND I.HIST_STATUS = 'N'
-AND A.BUSINESS_UNIT = I.BUSINESS_UNIT
AND A.CUST_ID = I.CUST_ID
AND A.ITEM = I.ITEM
AND A.ITEM_LINE = I.ITEM_LINE
AND A.ENTRY_TYPE = I.ENTRY_TYPE
AND A.ENTRY_REASON = I.ENTRY_REASON
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
AND A.ITEM_SEQ_NUM = (
SELECT MIN(ITEM_SEQ_NUM)
FROM PS_ITEM_ACTIVITY X
WHERE X.BUSINESS_UNIT = I.BUSINESS_UNIT
AND X.CUST_ID = I.CUST_ID
AND X.ITEM = I.ITEM
AND X.ITEM_LINE = I.ITEM_LINE
AND X.ENTRY_TYPE = I.ENTRY_TYPE
AND X.ENTRY_REASON = I.ENTRY_REASON)
AND A.ENTRY_AMT_BASE > 0
©2013 www.go-faster.co.uk
29
Execution Plan
Plan hash value: 768682620
----------------------------------------------------------------------------------------------------------| Id | Operation
| Name
| Rows | Bytes |TempSpc| Cost (%CPU)| Time
|
----------------------------------------------------------------------------------------------------------|
0 | INSERT STATEMENT
|
|
|
|
|
222K(100)|
|
|
1 | FILTER
|
|
|
|
|
|
|
|
2 |
HASH JOIN
|
| 23719 | 3613K|
|
111K (1)| 00:22:19 |
|
3 |
TABLE ACCESS BY INDEX ROWID | PS_POST_REQ_TMP4 |
8 |
112 |
|
2
(0)| 00:00:01 |
|
4 |
INDEX RANGE SCAN
| PS_POST_REQ_TMP4 |
8 |
|
|
1
(0)| 00:00:01 |
|
5 |
HASH JOIN
|
| 23719 | 3289K|
11M|
111K (1)| 00:22:19 |
|
6 |
TABLE ACCESS BY INDEX ROWID| PS_ITEM
|
118K| 9944K|
| 30295
(1)| 00:06:04 |
|
7 |
INDEX RANGE SCAN
| PSKITEM
|
280K|
|
|
812
(1)| 00:00:10 |
|
8 |
TABLE ACCESS FULL
| PS_ITEM_ACTIVITY | 3480K|
185M|
| 67704
(1)| 00:13:33 |
|
9 |
SORT AGGREGATE
|
|
1 |
42 |
|
|
|
| 10 |
TABLE ACCESS BY INDEX ROWID | PS_ITEM_ACTIVITY |
1 |
42 |
|
5
(0)| 00:00:01 |
| 11 |
INDEX RANGE SCAN
| PS_ITEM_ACTIVITY |
1 |
|
|
4
(0)| 00:00:01 |
----------------------------------------------------------------------------------------------------------Query Block Name / Object Alias (identified by operation id):
------------------------------------------------------------1 - SEL$1
3 - SEL$1 / R@SEL$1
4 - SEL$1 / R@SEL$1
6 - SEL$1 / I@SEL$1
7 - SEL$1
/ I@SEL$1
UKOUG
Apps
2013 - PeopleSoft
8 - SEL$1 / A@SEL$1
Instrumentation
for Oracle
9 - SEL$2
©2013 www.go-faster.co.uk
30
Limitations
• SQL may not be captured by AWR
– SQL inside a loop
– With bind variables
– Without ReUseStatement
• (binds become literals)
• Increase AWR snapshot frequency
– 15 minutes instead of an hour
– Top N SQL threshold
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
31
Conclusion
• Instrumentation permits you to profile DB
performance with ASH
• Proactively address performance issues
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
32
Nullius in verba
QUESTIONS?
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
33
You know my methods,
apply them.
A Study in Scarlet, Arthur Conan-Doyle
UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle
©2013 www.go-faster.co.uk
34