MTA Performance
Comparison:
sendmail vs. postfix on *BSD
Brad Knowles
Senior Consultant for
Snow, BV
brad.knowles@snow.nl
http://www.shub-internet.org/brad/papers/mtacomparison/
Entire contents copyright © 2002 by Brad Knowles, all
rights reserved
Overview
• Goal
• Meta Information
– Hardware Used
– Software Tested
– Tools Used
– Methodology
• Test Results
• Conclusions
2002-11-16
Copyright © 2002 by Brad Knowles
2
Goal
• Show you what it looks like to do MTA
performance tuning
– On as many *BSD platforms as I could
• Focus on the process, not the numbers
or expected/desired outcome
– You get to see (most) everything, warts
and all
2002-11-16
Copyright © 2002 by Brad Knowles
3
Goal
• Basically, chapter 3
from the book
sendmail
Performance Tuning by
Nick Christenson
– See
http://www.jetcafe.org/~n
pc/book/sendmail/
2002-11-16
Copyright © 2002 by Brad Knowles
4
Meta Information
•
•
•
•
Hardware Used
Software Tested
Benchmark Tools
Methodology
2002-11-16
Copyright © 2002 by Brad Knowles
5
Hardware Used
• NetBSD
– Twinhead “Twinstation 5G” (Sun SPARCstation 5
clone)
– Not UltraSPARC, but the ancient SPARC 5
•
•
•
•
OS:
CPU:
RAM:
NIC:
NetBSD 1.6-RELEASE
microsSPARC-II @ 110MHz
32MB real, 384MB virtual
On-board “Lance” 10Base-T Ethernet
& SBus QuadFastEthernet
• Unfortunately, disk drives died before testing could be
performed (bug in NetBSD regarding SCSI tagged
command queueing for old drives?)
2002-11-16
Copyright © 2002 by Brad Knowles
6
Hardware Used
• FreeBSD
– Compaq Armada 4131T
•
•
•
•
OS:
CPU:
RAM:
NIC:
• HD:
FreeBSD 4.6.2-RELEASE
Pentium 133
48MB real, 384MB virtual
Asanté FriendlyNET AL1011
“Prism2” 802.11b WiFi PCMCIA
10GB IBM Travelstar 20GN
– 4200 RPM
– 12ms avg. seek
2002-11-16
Copyright © 2002 by Brad Knowles
7
Hardware Used:
FreeBSD
Image copyright © 2001 Sunset Computer Services, Inc. All Rights Reserved.
2002-11-16
Copyright © 2002 by Brad Knowles
8
Hardware Used
• MacOS X (Part 1)
– PowerBook G3 “Pismo”
•
•
•
•
•
OS:
CPU:
RAM:
NIC:
HD:
MacOS X 10.2.1
PowerPC G3 @ 400MHz
1GB real, 2GB virtual
Apple AirPort 802.11b WiFi
48GB IBM Travelstar 40GH
– 5400 RPM
– 12ms avg. seek
2002-11-16
Copyright © 2002 by Brad Knowles
9
Hardware Used:
MacOS X (Part 1)
Image copyright © 2001 Sunset Computer Services, Inc. All Rights Reserved.
2002-11-16
Copyright © 2002 by Brad Knowles
10
Hardware Used
• MacOS X (Part 2)
– PowerBook G4 (DVI)
•
•
•
•
•
OS:
CPU:
RAM:
NIC:
HD:
MacOS X 10.2.2
PowerPC G4 @ 800MHz
1GB real, 2GB virtual
Apple AirPort 802.11b WiFi
40GB Toshiba MK4018GAS
– 4200 RPM
– 12ms avg. seek
2002-11-16
Copyright © 2002 by Brad Knowles
11
Hardware Used:
MacOS X (Part 2)
Image copyright © 2001 Sunset Computer Services, Inc. All Rights Reserved.
2002-11-16
Copyright © 2002 by Brad Knowles
12
Software Tested
• Sendmail 8.12.6
– ftp://ftp.sendmail/org/pub/sendmail/sendmail.8.12.6.tar.gz
• Postfix 1.1.11
– http://www.postfix.org/ftp-sites.html
• Exim 4.10
– Tried every way I could think of, but could
not get it configured to relay back to an IP
address or port on the same machine
2002-11-16
Copyright © 2002 by Brad Knowles
13
Tools Used
• Disk Benchmarking
– PostMark from Network Appliance
• http://www.netapp.com/tech_library/3022.html
– Bonnie++ from Russell Coker
• http://www.coker.com.au/bonnie++/
• SMTP Benchmarking
– smtp-source/smtp-sink from Wietse Venema
• Included with postfix distribution
– Postal from Russell Coker (with smtp-sink)
• http://www.coker.com.au/postal/
2002-11-16
Copyright © 2002 by Brad Knowles
14
Methodology
• Get disk subsystem baseline by
performing benchmarking
– For each platform & filesystem
configuration
• PostMark
– See http://www.shub-internet.org/brad/FreeBSD/postmark.html
• Bonnie++
– Configured to test similar parameters
» File sizes, numbers, & subdirectories according
to the four PostMark tests
2002-11-16
Copyright © 2002 by Brad Knowles
15
Methodology
• For FreeBSD, we only test turning on or off “soft
updates”
– We do not test (turned on by default)
• vfs.vmiodirenable
• dirprefs
• dirhash
– For data on all the possible combinations of the above, see
“Recent Filesystem Optimisations in FreeBSD” at
http://www.usenix.org/events/usenix02/tech/freenix/dowse.html
– For comparison data of Journaling vs. Soft Updates, see
“Journaling Versus Soft Updates: Asynchronous Meta-data
Protection in File Systems” at
http://www.usenix.org/publications/library/proceedings/usenix2000/
general/seltzer.html
2002-11-16
Copyright © 2002 by Brad Knowles
16
Methodology
• For MacOS X
– We test only
• HFS+
– Apple’s Extended Hierarchical File System
• MFS
– Memory-based filesystem created using
RamBunctious 2.0
2002-11-16
Copyright © 2002 by Brad Knowles
17
Methodology
• For MacOS X
– We do NOT test
• UFS
– IMO, Apple broke it while porting HFS+ to Mach
– Many applications do not recognize it, in addition to
MacOS 9 itself
– “Classic” requires that boot device be HFS+ and
MacOS 9 must be installed before MacOS X
• HFS+ with Journaling
– Too new, introduced with MacOS X Server 10.2.2
– Initial reports indicate ~20% performance loss
2002-11-16
Copyright © 2002 by Brad Knowles
18
Methodology
• For MacOS X
– Does not have a native “poll.h” or “libpoll”
(required by postal)
• Install manually from source
http://www.clapper.org/software/poll/
• Install using “fink”
– MacOS X package/port manager based on Debian
“dselect”
– Unable to test Bonnie++
• Need further assistance from the author to find out why it
is crashing
2002-11-16
Copyright © 2002 by Brad Knowles
19
Methodology
• Perform SMTP benchmarking
– For each hardware & software configuration, test
• Test direct source->sink connection
– Demonstrate maximum possible performance
• Test source->relay->sink
– Show relay performance
– Note
• Testing only relaying, not local delivery
• Test generated on, relayed through, and terminated on
“loopback” network
– Eliminate all question of network or NIC performance
2002-11-16
Copyright © 2002 by Brad Knowles
20
Test Results
• Chart data only
– PostMark table data available from
http://www.shub-internet.org/brad/FreeBSD/postmark.html
– Bonnie++ table data not yet available
– smtp-source/smtp-sink data not yet
available
• `hostname` must be legal (I.e., does not end
with “.”)
– Postal data not yet available
2002-11-16
Copyright © 2002 by Brad Knowles
21
Test Results
• Dumb “Driver” Scripts
– My “runpostmark” script available at
http://www.shub-internet.org/brad/runpostmark
– My “runbon” script available at
http://www.shub-internet.org/brad/runbon
– My “runsmtp” script available at
http://www.shub-internet.org/brad/runsmtp
2002-11-16
Copyright © 2002 by Brad Knowles
22
Test Results:
PostMark Sample Run
% cd /var/tmp
% postmark
PostMark v1.5 : 3/27/01
pm>set number 1000
pm>set transactions 50000
pm>show
Current configuration is:
The base number of files is 1000
Transactions: 50000
Files range between 500 bytes and 9.77 kilobytes in size
Working directory: /var/tmp
Block sizes are: read=512 bytes, write=512 bytes
Biases are: read/append=5, create/delete=5
Using Unix buffered file I/O
Random number generator seed is 42
Report format is verbose.
pm>run
2002-11-16
Copyright © 2002 by Brad Knowles
23
Test Results:
PostMark Sample Run
Creating files...Done
Performing transactions........Done
Deleting files...Done
Time:
497 seconds total
492 seconds of transactions (101 per second)
Files:
26014 created (52 per second)
Creation alone: 1000 files (500 per second)
Mixed with transactions: 25014 files (50 per second)
24868 read (50 per second)
24880 appended (50 per second)
26014 deleted (52 per second)
Deletion alone: 1028 files (342 per second)
Mixed with transactions: 24986 files (50 per second)
Data:
161.15 megabytes read (332.02 kilobytes per second)
168.38 megabytes written (346.92 kilobytes per second)
2002-11-16
Copyright © 2002 by Brad Knowles
24
PostMark Test Results
Postmark Test #1: 1000 Files & 50k Transactions
90
8,000.00
2,500
80
7,000.00
70
2,000
6,000.00
KB/Sec
5,000.00
1,500
50
4,000.00
40
1,000
3,000.00
30
Tran s/Sec
60
East
West
North
Dat a Read
Dat a Writ ten
Trans/Sec
2,000.00
20
500
1,000.00
10
0.00
0
FS
U
PF
TM
2002-11-16
S
S
t
+
+
FT
0
5
FS
FT
og
FS
FS
FS
PT
FS
FS
of
30
30
ux
PF
R
R
FS
FS
O
L
D
U
U
M
M
in
SO
M
+
M
+U
+S
S
F3
F6
+
S/
S/
+
+
/
/
/H
/H
+
+
3/
4/
5
L
q
q
+
3
3
5
T
/
3
4
3
D
D
S
S
o
a
a
3
+
G
G
P
U
P
3
r
U
P
p
p
F
F
G
G
O
O
P
5
P
N
N
2x
PP
2x
U
PB
PB
2x
om
om
PB
PB
C
C
1st Qtr
2nd Qtr
Platform
3rd Qtr
0
4th Qtr
Copyright © 2002 by Brad Knowles
25
PostMark Test Results
Postmark Test #2: 20k Files & 50k Transactions
90
3,000.00
700
80
600
70
2,500.00
60
500
50
400
1,500.00
40
300
1,000.00
30
Tran s/Sec
KB/Sec
2,000.00
East
West
North
Dat a Read
Dat a Writ ten
Trans/Sec
200
20
500.00
100
10
0.00
0
FS
U
PF
TM
S
2002-11-16
R
S/
D
O
0
S
t
+
+
FT
5
FS
FT
FS
FS
FS
PT
FS
og
FS
of
30
30
ux
PF
R
FS
FS
O
D
U
U
L
M
M
in
SO
M
M
+U
+S
S
F3
F6
+
+
S/
+
+
/
/
/H
/H
+
+
3/
4/
L
q
q
+
3
3
5
5
T
/
3
4
3
D
S
S
o
a
a
3
+
G
G
P
P
3
r
U
U
P
p
p
F
F
G
G
O
P
5
P
N
N
2x
PP
2x
U
PB
PB
2x
om
om
PB
PB
C
C
1st Qtr
2nd Qtr
Platform
3rd Qtr
0
4th Qtr
Copyright © 2002 by Brad Knowles
26
PostMark Test Results
Postmark Test #3: 20k Files & 100k Transactions
90
2 ,50 0. 00
80
7 00
70
6 00
60
5 00
1 ,50 0. 00
50
4 00
40
3 00
30
2 00
1 ,00 0. 00
Trans/Sec
KB/Sec
2 ,00 0. 00
East
Data Read
Data Written
T rans/Sec
West
North
5 00 .00
20
10
0 .00
0
1 00
S
S
S
+
+
ft
g
0
5
FT
FS
FT
FS
FS
FS
PT
FS
30
30
ux
PF
FS
FS
R
R
UF
So
PF
Lo
U
O
D
U
M
M
M
in
M
SO
/
/
+
+
+
F3
F6
S
H
H
S/
S/
+
+
+
/
/
M
+
/
/
L
+
3
4
q
q
5
T
+
5
/
3
4
D
D
T
P3
G
G
P3
U
P3
ro
pa
pa
FS
FS
U
5+
G
G
O
O
P3
P3
N
N
2x
m
m
2x
U
PP
PB
PB
2x
PB
PB
Co
Co
2002-11-16
1st Qtr
2nd Qtr
Platform
3rd Qtr
0
4th Qtr
Copyright © 2002 by Brad Knowles
27
PostMark Test Results
Postmark Test #4: 20k Files, 100k Transactions, & 100 subdirectories
90
6,000.00
1,400
80
1,200
70
5,000.00
60
1,000
50
800
3,000.00
40
600
2,000.00
30
Tran s/Sec
KB/Sec
4,000.00
East
West
North
Dat a Read
Dat a Writ ten
Trans/Sec
400
20
1,000.00
200
10
0.00
0
/L
P3
ux
in
2002-11-16
2x
FS
+U
P3
P
2x
3+
FT
SO
1st Qtr
2x
FS
+M
3
P
U
5+
U
FS
L
5+
U
og
2nd Qtr
T
5+
U
PF
M
S
PB
FS
H
3/
G
Platform
+
PB
FS
M
3/
G
PB
FS
H
4/
G
3rd Qtr
+
PB
FS
M
4/
G
om
q
pa
FS
+U
om
q
pa
t
of
+S
0
4th Qtr
C
Copyright © 2002 by Brad Knowles
C
28
PostMark Test Results
Postmark Test #1: 1000 Files & 50k Transactions
90
3,500.00
1,200
80
3,317.76
3,174.40
3,000.00
70
1,000
980
60
2,500.00
KB/Sec
50
2,000.00
40
1,566.72
1,500.00
485
30
600
1,638.40
492
Tran s/Sec
800
East
West
North
Dat a Read
Dat a Writ ten
Trans/Sec
400
1,000.00
20
10
489.66 511.62
149
500.00
332.02 346.92
268.32 280.35
82
0
200
235.40 245.96
72
0.00
0
1st Qtr
P BG3/H FS+
2002-11-16
P BG3/MFS
2nd Qtr
P BG4/H FS+
3rd Qtr
P BG4/MFS
Platform
Compaq+UFS
4th Qtr
Copyright © 2002 by Brad Knowles
Compaq+Soft
29
PostMark Test Results
Postmark Test #2: 20k Files & 50k Transactions
90
3,000.00
700
80
2,816.00
666
600
70
2,500.00
60
500
KB/Sec
50
400
40
1,500.00
1,495.04
349
1,392.64
300
30
Tran s/Sec
2,000.00
East
West
North
Dat a Read
Dat a Writ ten
Trans/Sec
1,000.00
20
200
737.57
500.00
10
0
100
82.00
154.75
29
99.14
187.09
36
38.95
0.00
1st Qtr
P BG3/H FS+
2002-11-16
P BG3/MFS
2nd Qtr
P BG4/H FS+
3rd Qtr
P BG4/MFS
Platform
73.51
15
Compaq+UFS
66.57
125.63
23
0
4th Qtr
Copyright © 2002 by Brad Knowles
Compaq+Soft
30
PostMark Test Results
Postmark Test #3: 20k Files & 100k Transactions
90
2,500.00
700
80
2,334.72
613
70
600
2,000.00
60
500
50
KB/Sec
1,500.00
400
40
1,177.60
310
300
1,000.00
30
Tran s/Sec
1,628.16
East
West
North
Dat a Read
Dat a Writ ten
Trans/Sec
819.89
200
20
500.00
10
0
100
90.20
129.65
31
94.42
135.72
33
42.68
0.00
1st Qtr
P BG3/H FS+
2002-11-16
P BG3/MFS
2nd Qtr
P BG4/H FS+
3rd Qtr
P BG4/MFS
Platform
69.72
61.35
15
Compaq+UFS
100.21
22
0
4th Qtr
Copyright © 2002 by Brad Knowles
Compaq+Soft
31
PostMark Test Results
Postmark Test #4: 20k Files, 100k Transactions, & 100 subdirectories
90
2,500.00
700
80
2,129.92
70
600
574
2,000.00
60
500
50
KB/Sec
400
40
300
1,015.93
277
1,000.00
30
708.74
Tran s/Sec
1,484.80
1,500.00
East
West
North
Dat a Read
Dat a Writ ten
Trans/Sec
200
20
500.00
10
0
100
65.43
93.79
24
77.21
110.68
28
38.81
0.00
1st Qtr
P BG3/H FS+
2002-11-16
P BG3/MFS
2nd Qtr
P BG4/H FS+
3rd Qtr
P BG4/MFS
Platform
55.63
14
Compaq+UFS
40.19
57.61
13
0
4th Qtr
Copyright © 2002 by Brad Knowles
Compaq+Soft
32
Test Results:
Bonnie++ Sample Run
% bonnie++ -d /var/tmp/bonnie -n 1:10k:0 -m compaq-soft -r 48m
Writing with putc()...done
Writing intelligently...done
Rewriting…done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.02d
------Sequential Output------ --Sequential Input- -Random-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --SeeksMachine
Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
compaq-soft
300M 2407 67 4584 39 2052 21 1948 53 3644 18 57.8
3
------Sequential Create------ --------Random Create-------Create-- --Read--- -Delete-- -Create-- --Read--- -Deletefiles:max
/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
2002-11-16
Copyright
© 2002 by Brad
Knowles
33
1:10:0
402 31 +++++ +++
832 49
564 46 +++++ +++ 1746
99
Bonnie++ Results
Bonnie++ Block I/O (Sequential Create)
4 50
6 0%
4 02
4 00
5 0%
3 40
3 31
3 00
4 0%
2 50
3 1%
2 00
3 2%
3 1%
3 0%
1 81
1 00
Files Created/Sec
% CPU Busy
2 3%
1 50
1 50
% CPU Busy
Files Created/Sec
5 0%
4 7%
3 50
2 0%
76
1 0%
50
0
0%
comp aq-ufs/1 :10 :0
comp aqufs/2 0:1 0:0
comp aqufs/2 0:1 0:0:1 00
comp aqso ft/1:10 :0
comp aqso ft/20 :1 0:0
comp aqso ft/20 :1 0:0 :1 00
Platform-Filesystem/Test
2002-11-16
Copyright © 2002 by Brad Knowles
34
Bonnie++ Results
Bonnie++ Block I/O (Sequential Read)
8 00
5 0%
7 04
7 00
6 704 3%
4 3%
6 95
4 4%
4 5%
4 0%
6 00
5 00
3 0%
4 00
2 5%
2 0%
3 00
% CPU Busy
Files Read/Sec
3 5%
Files Read/Sec
% CPU Busy
1 5%
2 00
1 65
1 0%
1 0%
1 00
5%
0
0%
comp aq-ufs/2 0:1 0:0
comp aq-ufs/2 0:1 0:0:1 00
comp aq-so ft/20 :1 0:0
comp aq-so ft/20 :1 0:0 :1 00
Platform-Filesystem/Test
2002-11-16
Copyright © 2002 by Brad Knowles
35
Bonnie++ Results
Bonnie++ Block I/O (Sequential Delete)
2 ,50 0
7 0%
2 ,06 26 0%
2 ,00 0
1 ,85 7
5 3%
5 0%
4 9%
4 8%
1 ,50 0
4 0%
3 0%
1 ,00 0
2 5%
% CPU Busy
5 2%
Files Deleted/Sec
6 0%
Files Deleted/Sec
% CPU Busy
8 32
2 0%
5 00
2 15
1 0%
1 88
1 03
0
0%
comp aq-ufs/1 :10 :0
comp aqufs/2 0:1 0:0
comp aqufs/2 0:1 0:0:1 00
comp aqso ft/1:10 :0
comp aqso ft/20 :1 0:0
comp aqso ft/20 :1 0:0 :1 00
Platform-Filesystem/Test
2002-11-16
Copyright © 2002 by Brad Knowles
36
Bonnie++ Results
Bonnie++ Block I/O (Random Create)
6 00
6 0%
5 64
5 7%
5 00
5 0%
4 6%
4 00
4 0%
3 30
3 3%
3 50
3 5%
3 00
3 0%
2 4%
2 00
2 01
% CPU Busy
Files Created/Sec
4 8%
Files Created/Sec
% CPU Busy
2 0%
1 54
1 00
1 0%
78
0
0%
comp aq-ufs/1 :10 :0
comp aqufs/2 0:1 0:0
comp aqufs/2 0:1 0:0:1 00
comp aqso ft/1:10 :0
comp aqso ft/20 :1 0:0
comp aqso ft/20 :1 0:0 :1 00
Platform-Filesystem/Test
2002-11-16
Copyright © 2002 by Brad Knowles
37
Bonnie++ Results
Bonnie++ Block I/O (Random Read)
90
6%
81
80
79
80
5%
5%
5%
5%
60
4%
50
46
3%
40
30
2%
% CPU Busy
Files Read/Sec
70
Files Read/Sec
% CPU Busy
2%
20
1%
10
0
0%
comp aq-ufs/2 0:1 0:0
comp aq-ufs/2 0:1 0:0:1 00
comp aq-so ft/20 :1 0:0
comp aq-so ft/20 :1 0:0 :1 00
Platform-Filesystem/Test
2002-11-16
Copyright © 2002 by Brad Knowles
38
Bonnie++ Results
Bonnie++ Block I/O (Random Delete)
2 ,00 0
1 ,80 0
1 ,81 1
1 ,74 6
1 00 %
9 9%
1 ,60 0
1 ,40 0
8 0%
1 ,20 0
1 ,00 0
6 0%
5 3%
5 3%
8 00
4 2%
Files Deleted/Sec
% CPU Busy
4 0%
6 00
4 00
2 00
% CPU Busy
Files Deleted/Sec
1 20 %
1 ,89 0
2 0%
1 70
407%
3 97 %
0
0%
comp aq-ufs/1 :10 :0
comp aqufs/2 0:1 0:0
comp aqufs/2 0:1 0:0:1 00
comp aqso ft/1:10 :0
comp aqso ft/20 :1 0:0
comp aqso ft/20 :1 0:0 :1 00
Platform-Filesystem/Test
2002-11-16
Copyright © 2002 by Brad Knowles
39
Test Results:
SmtpStone Sample Run
% sudo smtp-sink 127.0.0.1:25 1024 &
% foreach I (1 2 3 4 5 6 7 8 9 10)
foreach? /usr/bin/time smtp-source -l 10240 -f fred@example.net \
-t george@example.net -m 1000 -s 512 127.0.0.1:25
foreach? end
8.06 real
0.22 user
1.65 sys
9.03 real
0.26 user
1.45 sys
smtp-source: fatal: connect: Operation timed out
32.00 real
0.17 user
1.68 sys
13.08 real
0.19 user
1.43 sys
10.98 real
0.31 user
1.28 sys
9.02 real
0.24 user
1.52 sys
8.04 real
0.25 user
1.57 sys
smtp-source: fatal: lost connection while reading server greeting
16.51 real
0.19 user
1.61 sys
10.99 real
0.19 user
1.53 sys
16.52 real
0.25 user
1.50 sys
2002-11-16
Copyright © 2002 by Brad Knowles
40
SmtpStone Results
PowerBook G4 SmtpStone Performance (1000 messages)
2 50 .00
9 .00
8 .05
8 .00
2 08 .29
7 .58
7 .00
6 .00
1 54 .16
1 50 .78
1 50 .00
5 .00
4 .00
1 00 .00
8 4.2 0
7 9.1 0
Standard Dev iation
Messages p er Second
2 00 .00
Msgs/Sec.
Std.Dev.
3 .00
6 8.1 9
2 .00
5 0.0 0
3 9.9 8
0 .98
9 .98
0 .46
0 .42
0 .15
0 .00
5 12
2 56
1 28
1 .00
1 9.9 8
0 .64
64
32
16
0 .10
8
0 .07
4
2
5 .00
0 .00 0 .00
1
Number of Threads
2002-11-16
Copyright © 2002 by Brad Knowles
41
SmtpStone Results
PowerBook G4 SmtpStone Performance (20000 messages)
2 50 .00
4 5.0 0
2 37 .28
4 0.0 0
3 8.2 3
3 5.7 8
Messages p er Second
3 2.9 2
1 88 .79
3 5.0 0
3 2.4 2
2 9.3 4
3 0.0 0
1 55 .99
1 50 .00
2 5.0 0
1 23 .55
1 00 .00
9 7.3 8
9 3.8 7
1 00 .96
1 01 .69
2 1.1 4
9 3.5 7
9 6.0 8
2 0.0 0
1 9.7 6
7 9.2 0
1 5.0 0
3 9.8 3
5 .10
5 .00
2 .58
8 19 2
4 09 6
2 04 8
1 02 4
5 12
2 56
1 28
64
32
2 .56
1 .24
0 .00
1 63 84
Msgs/Sec.
Std. Dev.
1 0.0 0
9 .34
5 0.0 0
Standard Dev iation
2 00 .00
16
0 .00
8
Number of Threads
2002-11-16
Copyright © 2002 by Brad Knowles
42
SmtpStone Results
Compaq SmtpStone Performance (1000 Messages)
3 0.0 0
4 .50
2 8.3 9
4 .23
4 .00
2 5.0 0
2 0.0 0
3 .00
1 6.6 2
2 .50
1 5.0 0
2 .00
1 0.0 0
9 .00
Standard Dev iation
Messages p er Second
3 .50
Msgs/Sec
Std. Dev.
1 .50
1 .00
5 .00
0 .50
0 .17
0 .02
0 .00
4
2
0 .00
1
Number of Threads
2002-11-16
Copyright © 2002 by Brad Knowles
43
Test Results:
Postal Sample Run
% sudo smtp-sink 127.0.0.1:25 1024 &
% postal -m 10k -p 64 -c 1 127.0.0.1 postal.user time,messages,data(K),errors,connections,SSL connections
23:05,7651,41262,0,7715,0
23:06,7795,41656,0,7795,0
23:07,7847,42305,0,7847,0
23:08,7616,41528,0,7616,0
23:09,7670,41659,0,7670,0
23:10,7413,39922,0,7413,0
23:11,7765,41886,0,7765,0
23:12,7752,41960,0,7752,0
23:13,7423,39835,0,7423,0
23:14,7475,39697,0,7475,0
23:15,7458,39981,0,7458,0
23:16,7462,40332,0,7462,0
23:17,7281,39294,0,7280,0
23:18,7408,40022,0,7409,0
23:19,7595,40882,0,7586,0
23:20,7737,41991,0,7746,0
23:21,7646,41122,0,7646,0
^C
2002-11-16
Copyright © 2002 by Brad Knowles
44
Test Results:
Postal Sample Run
% sudo smtp-sink 127.0.0.1:25 1024 &
% postal -m 10k -p 128 -c 1 127.0.0.1 postal.user time,messages,data(K),errors,connections,SSL connections
23:27,8283,43988,0,8411,0
23:28,8965,48165,0,8965,0
23:29,8952,48100,0,8952,0
23:30,8785,46803,0,8785,0
23:31,7948,42811,0,7933,0
Server timed out on read.
Server timed out on read.
[ .. .. Deletia .. .. ]
Server timed out on read.
Server timed out on read.
23:32,8016,43367,28,8059,0
23:33,8960,47748,0,8960,0
23:34,9054,48507,0,9022,0
23:35,8737,47175,0,8769,0
23:36,9057,48586,0,9057,0
23:37,8800,46722,0,8800,0
23:38,8933,48092,0,8933,0
23:39,8969,48155,0,8955,0
23:40,8984,48576,0,8998,0
^C
2002-11-16
Copyright © 2002 by Brad Knowles
45
Test Results:
Postal Sample Run
% sudo smtp-sink 127.0.0.1:25 1024 &
% postal -m 10k -p 256 -c 1 127.0.0.1 postal.user Usage: postal [-m maximum-message-size] [-p processes] [-l local-address]
[-c messages-per-connection] [-r messages-per-minute] [-a]
[-b [no]netscape] [-[z|Z] debug-file]
[-s ssl-percentage]
smtp-server user-list-filename conversion-filename
Postal Version: 0.61
% postal -m 10k -p 32 -c 1 127.0.0.1 postal.user time,messages,data(K),errors,connections,SSL connections
Can't connect to 127.0.0.1 port 25.
Can't connect to 127.0.0.1 port 25.
Can't connect to 127.0.0.1 port 25.
Can't connect to 127.0.0.1 port 25.
Can't connect to 127.0.0.1 port 25.
2002-11-16
Copyright © 2002 by Brad Knowles
46
Postal Results
PowerBook G4 Postal Performance
1 60 .00
4 00 .00
3 78 .02
1 45 .77
1 40 .00
3 50 .00
1 20 .00
3 00 .00
1 00 .00
2 50 .00
7 9.6 9
8 0.0 0
2 00 .00
1 64 .94
6 0.0 0
1 50 .00
3 9.9 2
4 0.0 0
1 0.0 6
0 .00
64
32
5 0.0 0
9 .98
2 7.5 1
1 28
Msgs/Sec
Std.Dev.
1 00 .00
1 9.9 7
2 0.0 0
Standard Dev iation
Messages p er Second
1 26 .46
16
3 .67
8
2 .74
0 .00
4
Number of Threads
2002-11-16
Copyright © 2002 by Brad Knowles
47
Notes for Testing
• When testing real MTAs, you need to watch more
than just the output from the testing program
– Not unusual for MTAs to accept mail faster than they can
process it, especially under heavy load
• Also watch
– Mail queue size
• mailq | grep '[0-9][0-9]:[0-9][0-9]' | wc -l
– CPU load
• iostat 5
– Memory utilization
• vmstat 5
2002-11-16
Copyright © 2002 by Brad Knowles
48
Notes for Postfix
• Make sure that you modify
/etc/postfix/main.cf to be suitable
– Need to boost
• local_destination_concurrency_limit
• default_destination_concurrency_limit
• Check /etc/postfix/master.cf to make
sure there are no artificial limits on processes
like smtp, smtpd, bounce, cleanup,
rewrite etc…
2002-11-16
Copyright © 2002 by Brad Knowles
49
Notes for MacOS X
• Unfortunately, Apple broke iostat
– Still mostly okay for what we want to watch
(CPU user %, system %, and idle %)
• Apple doesn’t even provide vmstat
– vm_stat is not the same program,
although it is similar
2002-11-16
Copyright © 2002 by Brad Knowles
50
Postfix SmtpStone Results
PowerBook G4 Postfix SmtpStone Performance (1000 messages)
7 0.0 0
0 .90
0 .85
0 .84
0 .80
5 7.2 2
0 .70
Messages p er Second
5 2.2 6
5 0.0 0
0 .64
0 .60
3 9.5 0
4 0.0 0
0 .50
0 .40
3 0.0 0
1 9.9 8
2 0.0 0
0 .29
Standard Dev iation
6 0.0 0
Msgs/Sec
Std. Dev.
0 .30
0 .20
1 0.0 0
9 .92
0 .13
4 .99
0 .08
0 .00
0 .10
0 .00
32
16
8
4
2
1
Number of Threads
2002-11-16
Copyright © 2002 by Brad Knowles
51
Postfix SmtpStone Results
Compaq+UFS SmtpStone Performance (1000 messages)
1 2.0 0
1 .40
1 0.9 5
1 0.3 6
1 .20
1 .14
1 .00
8 .00
6 .41
0 .80
6 .00
0 .60
0 .48
4 .00
Standard Dev iation
Messages p er Second
1 0.0 0
Msgs/Sec
Std. Dev.
0 .40
0 .33
2 .00
0 .20
0 .00
0 .00
4
2
1
Number of Threads
2002-11-16
Copyright © 2002 by Brad Knowles
52
Postfix SmtpStone Results
Compaq+Soft SmtpStone Performance (1000 messages)
1 2.0 0
1 .80
1 0.6 5
1 .60
1 .55
1 0.0 0
9 .40
8 .00
1 .20
0 .95
6 .00
6 .14
0 .94
1 .00
0 .80
4 .00
Standard Dev iation
Messages p er Second
1 .40
Msgs/Sec
Std. Dev.
0 .60
0 .40
2 .00
0 .20
0 .00
0 .00
4
2
1
Number of Threads
2002-11-16
Copyright © 2002 by Brad Knowles
53
Postfix SmtpStone Results
Compaq Postfix SmtpStone Results (20000 messages/4 Threads)
1 2.0 0
3 50 .00
1 1.0 8
3 03 .20
Messages p er Second
2 50 .00
8 .00
7 .53
2 00 .00
6 .00
1 50 .00
Standard Dev iation
3 00 .00
1 0.0 0
Msgs/Sec
Std. Dev.
4 .00
1 00 .00
2 .00
5 0.0 0
1 7.7 2
0 .00
0 .00
comp aq-ufs
comp aq+soft
Platform-Filesystem
2002-11-16
Copyright © 2002 by Brad Knowles
54
Sendmail SmtpStone Results
PowerBook G4 Sendmail SmtpStone Results (1000 messages)
2 0.0 0
2 0.0 0
1 8.5 9
1 8.1 9
Messages p er Second
1 6.0 0
1 8.0 0
1 4.8 9
1 6.0 0
1 4.3 6
1 4.0 0
1 4.0 0
1 2.0 0
1 2.0 0
9 .69
1 0.0 0
1 0.0 0
8 .00
6 .00
8 .00
5 .37
4 .97
Standard Dev iation
1 8.0 0
Msgs/Sec
Std. Dev.
6 .00
4 .75
4 .00
4 .00
2 .00
1 .92
2 .00
1 .47
0 .68
0 .37
0 .00
32
0 .00
16
8
4
2
1
Number of Threads
2002-11-16
Copyright © 2002 by Brad Knowles
55
Sendmail SmtpStone Results
PowerBook G4 Sendmail+Async SmtpStone Results (1000
messages)
2 0.0 0
1 .40
1 8.0 1
Messages p er Second
1 .20
1 .16
1 6.0 0
1 .00
1 4.0 0
1 2.0 0
0 .80
9 .92
1 0.0 0
0 .60
8 .00
6 .00
5 .36
0 .42
4 .97
0 .38
Standard Dev iation
1 8.0 0
Msgs/Sec
Std. Dev.
0 .40
4 .00
2 .48
0 .20
2 .00
0 .12
0 .20
1 .25
0 .10
0 .00
0 .00
32
16
8
4
2
1
Number of Threads
2002-11-16
Copyright © 2002 by Brad Knowles
56
Sendmail SmtpStone Results
Compaq Sendmail-UFS SmtpStone Results (1000 messages)
6 .00
3 5.0 0
3 1.0 4
5 .02
3 0.0 0
5 .00
4 .00
2 5.0 0
3 .92
3 .65
2 0.0 0
3 .00
2 .49
1 4.0 8
1 5.0 0
Standard Dev iation
Messages p er Second
4 .56
Msgs/Sec.
Std. Dev.
2 .00
1 0.0 0
9 .67
1 .00
4 .17
5 .00
4 .04
0 .00
0 .00
16
8
4
2
1
Number of Threads
2002-11-16
Copyright © 2002 by Brad Knowles
57
Sendmail SmtpStone Results
Compaq Sendmail+Soft SmtpStone Results (1000 messages)
6 .00
9 0.0 0
6 .61
8 2.2 0
5 .91
8 0.0 0
Messages p er Second
7 2.1 1
7 0.0 0
5 .00
6 2.6 2
6 0.0 0
4 .10
4 .00
5 0.0 0
3 .00
4 0.0 0
2 .81
2 .35
Standard Dev iation
7 .00
Msgs/Sec.
Std. Dev.
3 0.0 0
2 .00
2 0.3 7
1 .00
1 0.9 3
2 0.0 0
1 0.0 0
0 .00
0 .00
16
8
4
2
1
Number of Threads
2002-11-16
Copyright © 2002 by Brad Knowles
58
Performance Tuning: Syslog
• Problem
– Local syslog can cause excessive load on busy mail
system
• Solutions
– Configure syslog to write to logs asynchronously
• Perhaps via a “–” (dash) before the filename
– Run syslog on a central log server, across the network
• Also allows you to perform more in-depth log processing,
without interfering with mail services
• Generally logs are much easier to manage centrally
• Can also combine with other central syslog processing
applications
– E.g., Addamark Log Management System
2002-11-16
Copyright © 2002 by Brad Knowles
59
Performance Tuning: Syslog
• Problem
– Classic syslog uses UDP, can lose 75%
or more of packets sent across the network
• Solution
– Replace classic syslog with nsyslog,
ssyslog, or syslog-ng and use TCP
instead of UDP
• Depending on your particular requirements for
hardware, OS, environment, etc….
2002-11-16
Copyright © 2002 by Brad Knowles
60
Postfix SmtpStone Results
PowerBook G4 (no syslog) Postfix SmtpStone (1000 messages)
7 0.0 0
3 .00
2 .80
6 0.8 9
6 0.0 0
2 .50
5 2.5 9
5 0.0 0
2 .06
4 0.0 0
1 .67
2 .00
3 6.3 2
1 .50
1 .41
3 0.0 0
2 6.7 8
1 .23
Msgs/Sec
Std. Dev.
1 .00
1 9.3 0
2 0.0 0
Standard Dev iation
Messages p er Second
5 2.0 4
0 .80
9 .96
1 0.0 0
0 .35
0 .47
0 .50
4 .98
0 .00
0 .00
1 28
64
32
16
8
4
2
1
Number of Threads
2002-11-16
Copyright © 2002 by Brad Knowles
61
Sendmail SmtpStone Results
PowerBook G4 Sendmail+Async+NoSyslog SmtpStone Results
(1000 messages)
2 0.0 0
8 .00
1 8.0 0
7 .10
1 7.2 0
7 .00
1 6.0 0
5 .50
5 .00
1 2.0 0
9 .92
1 0.0 0
4 .00
8 .00
6 .00
3 .00
5 .24
4 .97
2 .50
2 .00
1 .24
0 .36
0 .10
0 .00
32
16
8
0 .12
4
Msgs/Sec
Std. Dev.
2 .00
1 .76
4 .00
Standard Dev iation
Messages p er Second
6 .00
1 4.0 0
2
1 .00
0 .00
1
Number of Threads
2002-11-16
Copyright © 2002 by Brad Knowles
62
Sendmail SmtpStone Results
Compaq Sendmail-UFS-NoSyslog SmtpStone Results
(1000 messages)
6 .00
3 0.0 0
5 .64
5 .00
2 5.0 0
2 4.9 2
4 .00
2 0.0 0
3 .63
3 .00
1 5.0 0
2 .30
1 1.4 3
2 .00
1 0.0 0
Standard Dev iation
Messages p er Second
4 .46
Msgs/Sec.
Std. Dev.
8 .44
1 .00
5 .00
2 .31
0 .00
0 .00
8
4
2
1
Number of Threads
2002-11-16
Copyright © 2002 by Brad Knowles
63
Conclusions
• This process can be difficult and timeconsuming to get right, but it pays off
• Graphs & charts can help a great deal
– But they don’t tell the whole story
• Software
– Postfix is a lot faster than even I had thought
– Wasn’t able to get sendmail tuned for the
performance I’m confident it can deliver
– Benchmarking tools (esp. Bonnie++ and postal)
need more work to be useful
2002-11-16
Copyright © 2002 by Brad Knowles
64
Conclusions
• Use the Scientific Method
– Try to set aside personal bias
– Trust only what you can prove
– Process
• Generate Hypothesis
• Generate test for Hypothesis
• Test Hypothesis
– Prove or Disprove
• If disproven, modify or generate new
Hypothesis
2002-11-16
Copyright © 2002 by Brad Knowles
65
Related Work
• Nick Christenson
– sendmail Performance Tuning book
http://www.jetcafe.org/~npc/book/sendmail/
– “Performance Tuning Sendmail Systems”
slides
http://www.jetcafe.org/~npc/doc/performance_tuning.pdf
• Brad Knowles
– “Sendmail Performance Tuning for Large
Systems” slides
http://www.shub-internet.org/brad/papers/sendmail-tuning/
2002-11-16
Copyright © 2002 by Brad Knowles
66
Related Work
• Matthias Andree
http://www-dt.e-technik.uni-dortmund.de/~ma/postfix/vsqmail.html
http://www-dt.e-technik.uni-dortmund.de/~ma/postfix/bench2.html
• Nakamura Motonori
http://www.kyoto.wide.ad.jp/mta/eval1/eindex.html
• Ciprian Ascante
http://www.benchmarks.dmz.ro/index.php?topic=LinMSB
2002-11-16
Copyright © 2002 by Brad Knowles
67