Profiling Suspicious
Code
Tom Bascom
White Star Software
A Few Words about the Speaker
Tom Bascom; Progress 4gl coder &
roaming DBA since 1987
 President, DBAppraise, LLC





Remote database management service for OpenEdge.
Simplifying the job of managing and monitoring the
world’s best business applications.
tom@dbappraise.com
VP, White Star Software, LLC


Expert consulting services related to all aspects of
Progress and OpenEdge.
tom@wss.com
2
Users want the right
answer, with the best
response time at the
lowest cost.
The performance enhancement
possible with a given improvement
is limited by the fraction of the
execution time that the improved
feature is used.
-- Amdahl’s Law
Performance is not just about
the database.
The most bang for the
performance tuning buck is
often in the application code.
But figuring out where to look is
often hard.
Target the largest response
time component of the most
important Business process
first.
Some Tools
Finding “Likely Suspects”


User Complaints
Compile with XREF




compile “program.p”
xref “tmp/program.xrf”
debug-list “tmp/program.dbg”.
“I/O By User” Data
CRUD Data
Testing – Add Performance Criteria to Test
Plans for New Releases
XREF
c:\examples\t5.p
c:\examples\t5.p
c:\examples\t5.p
c:\examples\t5.p
c:\examples\t5.p
c:\examples\t5.p
c:\examples\t5.p
c:\examples\t5.p
c:\examples\t5.p
c:\examples\t5.p
c:\examples\t5.p
c:\examples\t5.p
c:\examples\t5.p
c:\examples\t5.p
c:\examples\t5.p
1
1
1
7
13
13
13
13
15
15
17
17
22
22
22
COMPILE c:/profiler/examples/t5.p
CPINTERNAL ISO8859-1
CPSTREAM ISO8859-1
STRING "i" 1 NONE UNTRANSLATABLE
STRING "Customer" 8 NONE UNTRANSLATABLE
ACCESS sports2000.Customer Phone
STRING "603 547 9574" 12 NONE TRANSLATABLE
SEARCH sports2000.Customer CustNum WHOLE-INDEX
STRING "->,>>>,>>9" 10 NONE TRANSLATABLE FORMAT
STRING "->,>>>,>>9" 10 NONE TRANSLATABLE FORMAT
STRING "t5" 2 NONE TRANSLATABLE
STRING "x(2)" 4 NONE TRANSLATABLE FORMAT
STRING "i" 1 LEFT TRANSLATABLE
STRING "----------" 10 NONE UNTRANSLATABLE
STRING "CustNum" 7 NONE UNTRANSLATABLE
PROMON – IO By Process
04/08/04
11:00:00
Usr Name
0
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
lakewood
lakewood
lakewood
lakewood
smcnulty
eratclif
sstout
lakewood
lakewood
aracey
lakewood
I/O Operations by Process
-------- Database ----Access
Read
Write
103183
1
1
1
1
1
5443
641020
9441
1434853
366293
7326
21351699
42841
138850
788646
263693
3650
0
0
0
0
0
7
635
30
22840
475
108
7709
77
1262
1171
422
249
0
0
0
22174
13935
0
0
0
0
0
0
1
0
0
0
0
---- BI ----Read
Write
827
0
0
0
0
0
0
0
0
0
0
0
3
0
0
0
1
2999
0
5214
5235
0
0
0
0
0
0
0
0
4
0
0
0
0
---- AI ----Read
Write
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3013
0
0
5244
0
0
0
0
0
0
0
0
1
0
0
0
0
ProTop – IO By User
09:37:10
Sample
Hit Ratio:
Miss% :
Hit% :
Log Reads:
OS Reads:
Chkpts:
Flushed:
Area Full:
UIO
Usr
----31
30
34
9
6
32
16
ProTop -- Progress Database Monitor (release xv)
09/19/04
sports2000 [/data/s2k/sports2000]
Rate
16:1
15:1
Commits:
65
20
Local:
51
6.448%
6.708%
Latch Waits:
37
13
Remote:
0
93.552% 93.292% Tot/Mod Bufs:
1002
370
Batch:
50
20067
13999
Evict Bufs:
10625
330
Server:
0
1294
939
Lock Table:
8192
11
Other:
1
0
0 Lock Tbl HWM:
138
TRX:
1
0
0
Old/Curr BI:
6140
6140 Blocked:
1
1 100.00%
After Image: DISABLED
Total:
52
Name
--------------julia
jami
tucker
tucker
julia
peter
julia
Flags
----SB
SB
SB
SB*
SB
SB
SB
PID
DB Access
OS Reads OS Writes
Hit%
------ ---------- ---------- ---------- ------2776
4109
244
5 94.07%
2772
2171
131
7 93.99%
2788
2003
126
3 93.72%
2656
1315
106
28 91.94%
2644
984
60
0 93.90%
2780
900
62
2 93.13%
2684
452
4
0 99.12%
ProTop – CRUD Data
09:38:33
Sample
Hit Ratio:
Miss% :
Hit% :
Log Reads:
OS Reads:
Chkpts:
Flushed:
Area Full:
ProTop -- Progress Database Monitor (release xv)
09/19/04
sports2000 [/data/s2k/sports2000]
Rate
14:1
14:1
Commits:
62
65
Local:
51
7.239%
7.140%
Latch Waits:
45
46
Remote:
0
92.761% 92.860% Tot/Mod Bufs:
1002
370
Batch:
50
22960
26486
Evict Bufs:
26602
6225
Server:
0
1662
1891
Lock Table:
8192
11
Other:
1
1
0 Lock Tbl HWM:
138
TRX:
1
0
0
Old/Curr BI:
6141
6141 Blocked:
5
1 100.00%
After Image: DISABLED
Total:
52
Table Statistics
Tbl# Table Name
Create
Read
Update
Delete
---- -------------------- --------- --------- --------- --------4 OrderLine
0
5937
152
0
24 POLine
0
2641
56
0
23 PurchaseOrder
0
1699
36
0
18 Order
0
1608
37
0
21 Bin
0
286
14
0
2 Customer
0
206
16
0
12 Vacation
0
111
5
0
Why “Logical I/O” ???

Consistent and Repeatable Measurement




The same query against any given dataset will
always return the same result.
Not subject to external factors such as CPU
speed, disk throughput, user activity or the
buffer cache hit ratio.
Shows Hidden Problems even with small
datasets.
Shows Impact on Other Users.

“Chokepoint” on rate of Logical IO ops.
Why NOT etime() ???


Non-Repeatable
Subject to a host of external factors



CPU speed, disk throughput, other user activity,
buffer cache efficiency, phase of the moon
Granularity is too gross (millisecond)
Does measure non-db activity…
LRTEST.p
define variable i as integer no-undo.
define variable lr as integer no-undo.
find _myconnection no-lock.
find _userio no-lock where
_userio-usr = _myconn-userid no-error.
lr = _userio._userio-dbaccess.
etime( yes ).
find <table> no-lock where <whatever> no-error.
find _userio no-lock where
_userio-usr = _myconn-userid no-error.
display i ( _userio-dbaccess - lr ) etime().
Example
Records
Log I/O
Ratio
etime()
Production
383
335,597
876:1
6,685ms
2,788ms
Development
41
28,517
695:1
647ms
394ms
QA
365
327,284
896:1
4,880ms
2,965ms
1,498
3,001
2:1
64ms
52ms
0
3
N/A
19ms
0ms
1,501
3,010
2:1
64ms
52ms
for each loanfile no-lock where
loan-amount > 500000:
for each loanfile no-lock where
price-locked >= 9/01/2011 and
price-locked <= 9/30/2011:
Production
Development
QA
The performance enhancement
possible with a given improvement
is limited by the fraction of the
execution time that the improved
feature is used.
-- Amdahl’s Law
Target the largest response
time component of the most
important Business process
first.
Wouldn’t it be nice if…
Description: XYZZY
Top Total Time Lines
Program
Line
Avg Time
------------------------------ ----- ----------xtabsms2.p
19281
93.256652
getdocprep2.p
939
63.346967
proc_create_sitm xtabsms2.p
11926
12.611345
xtcountry_x2_x3.p
359
0.000013
proc_read-database sysval.p
536
0.000117
xtcountry_x2_x3.p
360
0.000009
getdocprep2.p
741
0.067411
proc_upd_nref xtmfintb2.p
3582
0.003438
proc_upd_nref xtmfintb2.p
3298
0.003137
proc_process_tasks xttskscn.r
3091
0.012560
findClient sysval.p
328
0.000094
Date: 10/04/11
Time
Calls
---------- ------186.513304
2
126.693934
2
50.445380
4
0.459658 34,967
0.336606
2,879
0.324163 34,894
0.269642
4
0.233802
68
0.213345
68
0.200956
16
0.165167
1,763
The Profiler
Profiler





First introduced with version 8.2 (-zprofile)
“Unsupported” (meaning the analysis tool)
Improved with version 9.0 (session:profiler
handle, no more -zecret)
Microsecond timings
Does not include “think time”
Using the Profiler

-profile



profiler: handle



Non-intrusive
Non-selective
Selective
But requires code insertion or “wrappering”
Analysis tools


$DLC/src/samples/profiler
http://www.greenfieldtech.com/downloads
PROFILER Attributes






DESCRIPTION – optional text describing this
session
LISTINGS – whether or not to create debug
listings
DIRECTORY – where to create debug listings
(default to –T)
FILE-NAME – name of output file (default
profile.out)
ENABLED – yes/no; initializes listings and so forth
PROFILING – turn profiling on or off
Other PROFILER Attributes



TRACE-FILTER – CSV list of “matches” criteria
for procedure tracing
TRACING – line level tracing
COVERAGE - % coverage support
PROFILER Methods


Write-Data() – flush accumulated data to
output file.
User-Data(char) – write user defined data,
such as VST statistics, to the output file.
Minimal Embedded Usage
assign
profiler:enabled = yes
profiler:profiling = yes
.
do i = 1 to 1000000:
/* do something */
end.
assign
profiler:enabled = no
profiler:profiling = no
.
profiler:write-data().
Targeted Profiling

Embedded in Code Being Investigated
define variable i as integer no-undo.
run profiler/on.p ( “batch001” ).
do i =
find
end.
do i =
find
end.
1 to 10:
customer no-lock where customer.cust-num = 1 no-error.
1 to 10:
customer no-lock where customer.phone = "(702) 272-9264" no-error.
run profiler/off.p ( “batch001” ).
Unsupported Profiling Utility
Profiling a Session

Create file called profiler.cfg with 3 lines:
-OUTFILE /tmp/profiler.out
-LISTINGS /tmp
-DESCRIBE someDescription

Add –profile to session startup:
mpro dbName –p start.p –profile profiler.cfg


Run normally.
Terminate cleanly & analyze the output.
Sample Profiling Output
Description: XYZZY
Top Total Time Lines
Program
Line
Avg Time
------------------------------ ----- ----------xtabsms2.p
19281
93.256652
getdocprep2.p
939
63.346967
proc_create_sitm xtabsms2.p
11926
12.611345
xtcountry_x2_x3.p
359
0.000013
proc_read-database sysval.p
536
0.000117
xtcountry_x2_x3.p
360
0.000009
getdocprep2.p
741
0.067411
proc_upd_nref xtmfintb2.p
3582
0.003438
proc_upd_nref xtmfintb2.p
3298
0.003137
proc_process_tasks xttskscn.r
3091
0.012560
findClient sysval.p
328
0.000094
Date: 10/04/11
Time
Calls
---------- ------186.513304
2
126.693934
2
50.445380
4
0.459658 34,967
0.336606
2,879
0.324163 34,894
0.269642
4
0.233802
68
0.213345
68
0.200956
16
0.165167
1,763
Profiler Example
A Calculation Bottleneck?
1
1
1
1
p = 4 * ( 1 - --- + --- - --- + --- ... )
3
5
7
9
Profiler Example
A Calculation Bottleneck?
0009
0010
0011
function piterm returns decimal ( input n as integer ).
return ( 1.0 / (( n * 2 ) + 1 )).
end.
0012
0013
do while abs( newpi - oldpi ) > precision:
0014
oldpi = newpi.
0015
if i modulo 2 = 0 then
0016
0017
0018
pi = pi + piterm( i ).
else
pi = pi - piterm( i ).
0019
newpi = ( 4.0 * pi ).
0020
display i newpi oldpi.
0021
i = i + 1.
0022
end.
Sample Profiling Output
Description: pi
Top Total Time Lines
Program
Line
Avg Time
------------------------------ ----- ----------./pi.p
20
0.000159
piterm ./pi.p
10
0.000010
./pi.p
13
0.000006
./pi.p
18
0.000010
./pi.p
16
0.000009
./pi.p
15
0.000004
./pi.p
19
0.000004
./pi.p
21
0.000004
./pi.p
14
0.000002
/home/tom/p26226_Untitled1.ped
1
0.009446
piterm ./pi.p
11
0.000001
./pi.p
22
0.000001
Date: 10/07/11
Time
Calls
---------- ------3.183243 20,001
0.197060 20,001
0.114640 20,002
0.097591 10,000
0.094585 10,001
0.082780 20,001
0.080964 20,001
0.076913 20,001
0.036911 20,001
0.018891
2
0.017755 20,001
0.013877 20,001
Profiler Example
A Calculation Bottleneck?
0009
0010
0011
function piterm returns decimal ( input n as integer ).
return ( 1.0 / (( n * 2 ) + 1 )).
end.
0012
0013
do while abs( newpi - oldpi ) > precision:
0014
oldpi = newpi.
0015
if i modulo 2 = 0 then
0016
0017
0018
pi = pi + piterm( i ).
else
pi = pi - piterm( i ).
0019
newpi = ( 4.0 * pi ).
0020
if i modulo 100 = 0 then display i newpi oldpi.
0021
i = i + 1.
0022
end.
Sample Profiling Output
Description: pi
Top Total Time Lines
Program
Line
Avg Time
------------------------------ ----- ----------piterm ./pi.p
10
0.000010
./pi.p
13
0.000005
./pi.p
18
0.000009
./pi.p
16
0.000009
./pi.p
20
0.000004
./pi.p
19
0.000004
./pi.p
15
0.000004
./pi.p
21
0.000003
./pi.p
14
0.000002
piterm ./pi.p
11
0.000001
./pi.p
22
0.000001
Date: 10/07/11
Time
Calls
---------- ------0.193776 20,001
0.106857 20,002
0.087636 10,000
0.087096 10,001
0.083103 20,001
0.079651 20,001
0.078491 20,001
0.058005 20,001
0.036486 20,001
0.016168 20,001
0.011794 20,001
Caveat!
define variable i as integer no-undo.
assign
profiler:enabled = yes
profiler:profiling = yes
.
do i = 1 to 1000000:
end.
i = 0.
do while i < 1000000:
i = i + 1.
end.
i = 0. do while i < 1000000: i = i + 1. end.
assign
profiler:enabled = no
profiler:profiling = no.
profiler:write-data().
Caveat!
1
.
2
.
0
.
0
2
2
2
2
2
2
2
.
.
.
.
02/28/2007 "Generic" 07:55:03 "tom"
"profile.p" "" 63126
0 2 1
0 1 0.000000 30.935828
11 1 0.000002 0.000002
8 1000001 4.607678 4.607678
9 1000000 1.719586 1.719586
14 1000000 1.501487 1.501487
12 1000001 3.013981 3.013981
13 1000000 3.032433 3.032433
16 1 0.000003 0.000003
Reminders & Hints

Line numbers are DEBUG LISTING line
numbers




You need to have .DBG files
They need to be created with the same source
and PROPATH as your .r files
Session profiling must terminate cleanly – no
“kill” & no crash.
Temp files can become very, very large.
Summary





Things to be suspicious of.
Tools to narrow your search.
A better way to gauge query effectiveness.
An introduction to Profiling.
Strategies for attacking code performance
problems.
Target the largest response
time component of the most
important Business process
first.
Questions
There is no silver bullet.
-- Fred Brooks
You just have to be persistent.
-- Tom Bascom
Thank you for
your time!
tom@wss.com
http://wss.com