*
OS
Tools
for
Monitoring Progress Applications
*Including
Windows even though
Gus says that I am lying ;)
A Few Words about the Speaker
• Tom Bascom; Progress user & roaming DBA
since 1987
• President, DBAppraise, LLC
– Remote database management service for OpenEdge.
– Simplifying the job of managing and monitoring the
world’s best business applications.
– tom@dbappraise.com
• VP, White Star Software, LLC
– Expert consulting services related to all aspects of
Progress and OpenEdge.
– tom@wss.com
2
Scope
• OS oriented – not specific to Progress or
OpenEdge.
• But useful with Progress and OpenEdge
• Emphasis on Free tools.
• Pre-installed or easily available.
Common Use Cases
• CPU Utilization
• Runaway Processes
• Memory Utilization
•
•
•
•
Disk Space
Disk Activity
Disk Capacity
Progress Temp Files
CPU Utilization
•
•
•
•
Am I really using all of my CPUs? (or “cores”?)
Am I getting useful work done?
Do I have room for growth?
Do I have a “runaway process”?
sar -u
sar -u 5 5
HP-UX pluto B.11.23 U ia64
15:00:00
15:00:05
15:00:10
15:00:15
15:00:20
15:00:25
Average
%usr
5
6
5
30
24
14
%sys
10
10
10
8
2
8
11/11/10
%wio
20
15
25
5
5
14
%idle
65
69
60
57
69
64
sar -u
sar -u 5 5
HP-UX pluto B.11.23 U ia64
15:00:00
15:00:05
15:00:10
15:00:15
15:00:20
15:00:25
Average
%usr
5
6
5
30
24
14
%sys
10
10
10
8
2
8
11/11/10
%wio
20
15
25
5
5
14
%idle
65
69
60
57
69
64
sar -f /usr/adm/sa/sa11 -s 15:00 -e 15:45 -i 600 -u
HP-UX pluto B.11.23 U ia64
15:00:00
15:10:00
15:20:00
15:30:00
15:40:00
Average
%usr
80
82
83
82
82
%sys
5
6
4
5
5
11/11/10
%wio
10
8
13
12
11
%idle
5
4
0
1
3
top
Processes: 72 total, 2 running, 70 sleeping, 325 threads
15:33:28
Load Avg: 0.22, 0.22, 0.17 CPU usage: 1.26% user, 2.10% sys, 96.62% idle
SharedLibs: 8260K resident, 9820K data, 0B linkedit.
MemRegions: 15320 total, 833M resident, 29M private, 676M shared.
PhysMem: 919M wired, 1558M active, 588M inactive, 3064M used, 5128M free.
VM: 154G vsize, 1042M framework vsize, 198231(0) pageins, 0(0) pageouts.
Networks: packets: 2049591/276M in, 2286325/256M out.
Disks: 89638/2507M read, 102460/3753M written.
PID
2465
2462
2461
2460
2447
2444
2443
2407
COMMAND
top
bash
login
mdworker
_progres
bash
login
mdworker
%CPU
5.2
0.0
0.0
0.0
34.0
0.0
0.0
0.0
TIME
00:00.53
00:00.00
00:00.00
00:00.07
00:07.41
00:00.00
00:00.01
00:01.41
#TH
1/1
1
1
3
1
1
1
3
#WQ
0
0
0
1
0
0
0
1
#POR
24
17
22
49
22
17
22
50
#MREG
33
24
53
64
32
24
53
81
RPRVT
1080K
360K
468K
1532K
1580K
364K
468K
9208K
RSHRD
340K
856K
312K
14M
340K
856K
312K
15M
RSIZE
1656K
1024K
1572K
3336K
2156K
1028K
1572K
39M
nmon
nmon 3=Top-by-CPU-use Host=usqadnode1
Refresh=8 secs
18:06.26
CPU-Utilisation-Small-View EntitledCPU= 3.00 UsedCPU= 0.108
Logical CPUs
0----------25-----------50----------75----------100
CPU User% Sys% Wait% Idle%|
|
|
|
|
0 48.3 49.7
0.0
2.0|UUUUUUUUUUUUUUUUUUUUUUUsssssssssssssssssssssssss>|
1 26.6 38.5
0.0 35.0|UUUUUUUUUUUUUsssssssssssssssssss
>|
2
0.0 36.0
0.0 64.0|sssssssssssssssss
>|
3
0.0 26.1
0.0 73.9|sssssssssssss
>
|
4
0.0 24.5
0.0 75.5|ssssssssssss
>
|
5
0.0 26.8
0.0 73.2|sssssssssssss>
|
6
0.0 25.6
0.0 74.4|ssssssssssss >
|
7
0.0 35.1
0.0 64.9|sssssssssssssssss >
|
8
0.0 23.8
0.0 76.2|sssssssssss >
|
9
0.0 24.4
0.0 75.6|ssssssssssss >
|
10
0.0 22.9
0.0 77.1|sssssssssss >
|
11
0.0 24.9
0.0 75.1|ssssssssssss >
|
+-----------|------------|-----------|------------+
taskmgr
Windows Task Manager
View -> CPU History ->
One Graph Per CPU
Memory Utilization
• How much memory am I really using?
• Operating systems “double count” shared
memory…
nmon
Memory
Physical PageSpace |
pages/sec In
Out
% Used
97.2%
3.6% | to Paging Space
6.1
4.9 |
% Free
2.8%
96.4% | to File System 3596.3 441.6 |
MB Used
31834.2MB 1024.4MB | Page Scans
256879.7
|
MB Free
933.8MB 27135.6MB | Page Cycles
21.4
|
Total(MB) 32768.0MB 28160.0MB | Page Steals
3785.5
|
| Page Faults
1785.7
|
------------------------------------------------------------ |
Min/Maxperm
952MB( 3%) 28545MB( 87%) <--% of RAM
|
Min/Maxfree
960
1088
Total Virtual
59.5GB
|
Min/Maxpgahead
2
16
Accessed Virtual
14.4GB 24.2%|
FileSystemCache
(numperm) 54.0%
Process
23.0%
System
20.1%
Free
2.8%
-----Total
100.0%
numclient 54.0%
maxclient 87.1%
User
74.0%
Pinned
19.1%
top
top - 10:51:19 up 408 days, 2:18, 1 user, load average: 7.42, 7.17, 7.10
Tasks: 696 total,
1 running, 695 sleeping,
0 stopped,
0 zombie
Cpu(s): 6.9%us, 0.9%sy, 0.0%ni, 85.0%id, 7.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem:
4049052k total, 4022004k used,
27048k free,
24352k buffers
Swap: 2104472k total,
156k used, 2104316k free, 2055292k cached
free
/home/tom> free
total
Mem:
4049052
-/+ buffers/cache:
Swap:
2104472
used
4022544
1937760
156
free
26508
2111292
2104316
shared
0
buffers
25140
cached
2059644
sar -r
• Totally useless
Windows taskmgr
taskmgr – Windows 7
Sysinternals - RamMap
Sysinternals - VMMap
Disk Space
• How much free space do I have?
df, du
/home/tom> df -vk
Filesystem
/dev/sdb2
udev
/dev/sdc1
/dev/sdd1
/dev/sda3
/dev/sda2
/home/protop> du –k
76
./bin
80
./etc
96
./hlp
364
./mon
28
./os/HP-UX
24
./os/AIX
28
./os/SunOS
28
./os/Linux
24
./os/OSF1
136
./os
96
./tmp
6736
.
1K-blocks
478647540
2024524
480696480
480696480
258033592
2100388
Used Available Use% Mounted on
175136968 279196600 39% /
152
2024372
1% /dev
380900480 75377204 84% /data
233559476 222718208 52% /home
180175200 64750960 74% /srv
4
2100384
1% /windows/C
df, du
/home/tom> df -vk
Filesystem
/dev/sdb2
udev
/dev/sdc1
/dev/sdd1
/dev/sda3
/dev/sda2
/home/protop> du –k
76
./bin
80
./etc
96
./hlp
364
./mon
28
./os/HP-UX
24
./os/AIX
28
./os/SunOS
28
./os/Linux
24
./os/OSF1
136
./os
96
./tmp
6736
.
1K-blocks
478647540
2024524
480696480
480696480
258033592
2100388
Used Available Use% Mounted on
175136968 279196600 39% /
152
2024372
1% /dev
380900480 75377204 84% /data
233559476 222718208 52% /home
180175200 64750960 74% /srv
4
2100384
1% /windows/C
Also available for Windows from Sysinternals!
Disk Activity
• Which disks are most active?
• Are my disks too active?
taskmgr - XP
nmon
Disk
Name
|hdisk4
hdisk6
hdisk7
hdisk9
hdisk10
hdisk13
hdisk17
hdisk18
hdisk21
hdisk22
Busy
Read Write 0----------25-----------50------------75--------100
KB/s
KB/s |
|
|
|
8%
29
0 |RRRR
>
|
4%
0
263 |WW >
|
1%
0
156 |W >
|
6%
47
67 |RRWW
|
13%
71
205 |RRWWWWW>
|
27%
257
47 |RRRRRRRRRRRRWWW
>
|
30%
338
78 |RRRRRRRRRRRRRWWW >
|
5%
26
12 |RRW
>
|
12%
84
104 |RRRWWWW
>
|
2%
16
0 |R
|
+-------------------------------------------------+
sar -d
sar -d 60 1
HP-UX pluto B.11.23 U ia64
11:22:40
11:23:40
device
c3t0d0
c24t0d5
c24t0d2
c24t0d3
c24t0d4
c24t1d1
%busy
3.35
0.02
0.32
0.03
0.03
0.02
11/14/10
avque
0.62
0.50
0.50
0.50
0.50
0.50
r+w/s
2
0
8
0
0
0
blks/s
31
0
128
1
0
0
avwait
0.00
0.00
0.00
0.00
0.00
0.00
avserv
14.52
1.68
0.40
6.62
7.66
6.20
perfmon
perfmon
Disk Capacity
• What can you expect of your disks?
iometer
iometer
randomrd.c
/home/tom> ./randomrd /data/dbappraise/dbappraise_20.d1 8192 5 500
iterations
1
2
3
4
5
reads
500
1000
1500
2000
2500
time(sec)
3.9822
3.9961
3.9709
4.0627
3.9150
reads/sec
125.56
125.12
125.92
123.07
127.71
Progress Temporary Files
• Who do the files in -T belong to?
• What temp files does a process have open?
Progress Temporary Files
# ls -l /protemp
total 64
-rw-r--r-- 1 tom
-rw-r--r-- 1 tom
-rw-r--r-- 1 root
-rw-r--r-- 1 root
-rw-r--r-- 1 tom
-rw-r--r-- 1 tom
-rw-r--r-- 1 tom
-rw-r--r-- 1 tom
users
32768
users
32768
users
0
users
0
users
0
users
0
users
0
users 19283638
2010-11-13
2010-11-13
2010-11-13
2010-11-13
2010-11-13
2010-11-13
2010-11-13
2010-11-13
16:21
17:02
16:21
17:02
17:02
16:21
16:21
17:02
DBI1640184576paLIUh
DBI2036800256vFfHS7
lbi1Bncn0
lbiXuiCqK
rcdD5N6Ik
rcdL2ugyw
srtfrdJsL
srtzgpRzx
Progress Temporary Files
# ls -l /protemp
total 64
-rw-r--r-- 1 tom users
32768 2010-11-13
-rw-r--r-- 1 tom users
32768 2010-11-13
-rw-r--r-- 1 root users
0 2010-11-13
-rw-r--r-- 1 root users
0 2010-11-13
-rw-r--r-- 1 tom users
0 2010-11-13
-rw-r--r-- 1 tom users
0 2010-11-13
-rw-r--r-- 1 tom users
0 2010-11-13
-rw-r--r-- 1 tom users 19283638 2010-11-13
#
#
# fuser -u /protemp/*
/protemp/DBI1640184576paLIUh: 15692(tom)
/protemp/DBI2036800256vFfHS7: 17980(tom)
/protemp/lbi1Bncn0: 15692(tom)
/protemp/lbiXuiCqK: 17980(tom)
/protemp/rcdD5N6Ik: 18972(tom)
/protemp/rcdL2ugyw: 17980(tom)
/protemp/srtfrdJsL: 15692(tom)
/protemp/srtzgpRzx: 17980(tom)
16:21
17:02
16:21
17:02
17:02
16:21
16:21
17:02
DBI1640184576paLIUh
DBI2036800256vFfHS7
lbi1Bncn0
lbiXuiCqK
rcdD5N6Ik
rcdL2ugyw
srtfrdJsL
srtzgpRzx
fuser, lsof
# lsof -p 17980
COMMAND
PID USER
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
_progres 17980 tom
FD
cwd
rtd
txt
mem
mem
mem
mem
mem
mem
mem
mem
mem
mem
0u
1u
2u
3r
4r
5u
6u
7u
TYPE DEVICE
SIZE
NODE NAME
DIR
8,49
8192 33652801 /home/tom
DIR
8,18
4096
2 /
REG
8,18 10444522 5867036 /usr/dlc/bin/_progres
REG
0,0
0 [heap] (stat: No such file or dir)
REG
8,18
133423 20381698 /lib64/ld-2.4.so
REG
8,18
217016 52118852 /var/run/nscd/passwd
REG
8,18
114562 20381728 /lib64/libpthread-2.4.so
REG
8,18
19808 20381713 /lib64/libdl-2.4.so
REG
8,18
49642 20381706 /lib64/libcrypt-2.4.so
REG
8,18
404881 20381716 /lib64/libm-2.4.so
REG
8,18
970192 5575786 /usr/lib64/libstdc++.so.6.0.8
REG
8,18
58128 20381823 /lib64/libgcc_s.so.1
REG
8,18 1505121 20381702 /lib64/libc-2.4.so
CHR 136,2
4 /dev/pts/2
CHR 136,2
4 /dev/pts/2
CHR 136,2
4 /dev/pts/2
REG
8,18 1483758 5866996 /usr/dlc/promsgs
REG
8,18
943825 5866994 /usr/dlc/convmap.cp
REG
8,49
0 33654422 /home/tom/lbiJaKarj (deleted)
REG
8,49 19283638 33654423 /home/tom/srtllSsLf (deleted)
REG
8,49
0 33654424 /home/tom/rcdNAdL5b (deleted)
sysinternals
Resources
• nmon
http://www.ibm.com/developerworks/wikis/display/wikiptype/nmon
• sysinternals
http://technet.microsoft.com/en-us/sysinternals/default.aspx
• iometer
http://www.iometer.org/
• randomrd.c
http://dbappraise.com/
Questions?
Thank-you!
top
Processes: 71 total, 2 running, 1 stuck, 68 sleeping, 336 threads
Load Avg: 0.40, 0.23, 0.17 CPU usage: 1.13% user, 1.36% sys, 97.49% idle
SharedLibs: 8260K resident, 9820K data, 0B linkedit.
MemRegions: 15384 total, 855M resident, 30M private, 679M shared.
PhysMem: 918M wired, 1587M active, 587M inactive, 3092M used, 5100M free.
VM: 152G vsize, 1042M framework vsize, 197921(0) pageins, 0(0) pageouts.
Networks: packets: 2048759/276M in, 2285978/256M out.
Disks: 89504/2506M read, 101810/3748M written.
PID
2448
2447
2444
2443
2407
2392
2277
2273
773
297
293289278
226
181179177-
COMMAND
PubSubAgent
top
bash
login
mdworker
cupsd
AppleSpell
Safari
ssh-agent
Preview
Microsoft AU
Microsoft Ex
Firewall
Terminal
db_daemon
gconfd-2
VMware Fusio
%CPU
0.0
4.0
0.0
0.0
0.7
0.0
0.0
0.0
0.0
0.0
0.0
0.1
0.0
1.0
0.0
0.0
0.0
TIME
00:00.17
00:00.76
00:00.00
00:00.01
00:01.39
00:00.01
00:00.09
00:15.84
00:00.12
00:13.92
00:00.23
00:31.66
00:00.00
00:17.06
00:00.10
00:00.14
00:00.25
#TH
4
1/1
1
1
3
3
2
9
2
2
2
7
1
5
1
1
3
RPRVT
7588K
1572K
364K
468K
9208K
972K
3300K
52M
972K
18M
948K
37M
360K
20M
856K
400K
1168K
RSHRD
14M
244K
244K
244K
14M
248K
14M
65M
280K
54M
1400K
95M
244K
39M
2316K
2468K
18M
RSIZE
9940K
2148K
1028K
1572K
39M
2292K
4960K
74M
2312K
46M
2108K
58M
1304K
35M
2848K
2236K
3992K
VPRVT
60M
17M
17M
18M
92M
57M
47M
230M
42M
55M
49M
88M
10M
65M
18M
18M
50M
VSIZE
2436M
2378M
2378M
2379M
2469M
2418M
2427M
4919M
2411M
2781M
891M
1128M
2379M
2752M
589M
589M
911M
15:30:49
PPID
136
2444
2443
226
1
1
136
136
136
136
136
136
136
136
1
1
136
State
sleeping
running
sleeping
sleeping
running
sleeping
sleeping
sleeping
sleeping
sleeping
sleeping
sleeping
sleeping
sleeping
sleeping
sleeping
sleeping
FAULTS
2727
11679+
451
617
528165+
780
1559
156817
2144
28161
2011
29079
485
57884+
847
716
2902
COW
117
55
102
125
160
87
94
1581
95
788
143
1554
85
525
113
128
219
Network Activity
netstat
perfmon, control panel
sysinternals
Kernel Resources
sar
perfmon, sysinternals
Common Use Cases
•
•
•
•
•
CPU Utilization
Memory Utilization
Disk Space
Disk Activity
Disk Capacity
•
•
•
•
•
Progress Temp Files
Runaway Processes
System Calls
Network Activity
Kernel Resources