EMB
E-Mail Basics
Goals of this lab:
 Learn how e-mail works “under the hood”
 Learn about spam filtering and virus scanning
 Gain experience setting up a full-fledged e-mail service
Prerequisites: LXB, NET, DNS
REVISION: 2.1 [2015-08-28]
©2005-2015 DAVID BYERS
Table of Contents
PREPARATION ......................................................................................................................................................... 1
Exercise 1: Review and preparation ............................................................................................................ 1
MAIN LAB ................................................................................................................................................................ 3
Part 1: Basic mail server...................................................................................................................................... 3
Exercise 2: Mail server installation and basic configuration ....................................................................... 4
Exercise 3: About the mail server ................................................................................................................ 4
DNS and mail service ....................................................................................................................................... 4
Exercise 4: Configure DNS for e-mail service .............................................................................................. 4
Part 2: End user support ..................................................................................................................................... 5
Exercise 5: Implement IMAP server ............................................................................................................ 5
Expert options ................................................................................................................................................. 5
Part 3: Intermediate mail service ....................................................................................................................... 5
User-friendly addresses .................................................................................................................................. 6
Exercise 6: Configuring user-friendly addresses ......................................................................................... 6
Spam protection.............................................................................................................................................. 6
Exercise 7: Questions about spam filtering ................................................... Error! Bookmark not defined.
Exercise 8: Implementing spam filtering ..................................................................................................... 7
Virus filtering (optional) .................................................................................................................................. 8
Exercise 9: Questions about server-based virus scanning .......................................................................... 8
Exercise 10: Implementation of virus scanning ........................................................................................... 8
Expert options ................................................................................................................................................. 8
IDA/ADIT
EMB: E-MAIL BASICS
IDA/ADIT
EMB: E-MAIL BASICS
PREPARATION
Do these exercises before proceeding to the main lab. You should be able to complete them using only
the on-line Postfix documentation and a computer networks textbook.
Exercise 1: Review and preparation
Read the documentation for Postfix (or the mail server of your choice) and answer the following
questions:
1-1
What processes does the mail server consist of? What does each of the processes do?
1-2
How can your mail server communicate with external programs that process mail (e.g. spam
filters or virus scanners)?
1-3
What does a typical SMTP dialog look like? Show how to send a simple e-mail using the SMTP
protocol.
Report:
Answers to the questions above.
Exercise 2: Spam protection
2-1
What type(s) of spam filtering does Spamassassin provide?
2-2
How does greylisting work? Under what circumstances could greylisting cause problems (e.g.
non-acceptance of legitimate e-mail, or excessive delays)?
2-3
If users insist on challenge-response spam filtering, how can the third party impact be
minimized (ideally nearly eliminated)?
2-4
What is SPF? What is DKIM? How do they prevent spam (and do they actually work)?
Report:
Answers to all questions above.
IDA/ADIT
EMB: E-MAIL BASICS
1
MAIN LAB
In this lab you will install and configure a complete mail system, complete from both a network and user
point of view. Note that mail should work equally well on the clients as well as the server. A good test of
the system is to send email from a client to an external address and check that all headers in the email
are correct. Since you do not have access to a graphic display on the UML instances it is tricky to use
graphical e-mail clients like Mozilla Thunderbird to read email. An alternative that many seem to think
works well is to use the text-base e-mail client mut t . If you want to run software that relies on X from
your UML instances you could also install necessary packages and login on the client from a lab
workstation using s s h. Just make sure that X11 forwarding is enabled.
Time taken 2006: 20-30h, average 25h
Past problems: This lab has seen the widest variety of problems of any lab in the past, which is reflected
in the large variation in time taken. Since the mail system is not only complex in itself, but also interfaces
with nearly every other part of the system, there are lots of problem sources, and lots of ways to make
mistakes. Therefore, it is vital to work professionally on this lab. That means understanding the software
and its interactions with e.g. DNS, the network, the file system and the directory service; understanding
the entire configuration and exactly what the role of every configuration statement is; and
understanding how incoming and outgoing mail is processed, where files are stored and what is logged.
It also means taking small, well-thought-out steps and confirming through testing that each step does
exactly what it is supposed to do. You have to slow down if you want to reach the finish line as fast as
possible.
Part 1: Basic mail server
There are lots of different mail servers available. In this course we recommend that beginners use the
Postfix mail server. It is a high-quality production mail server that nevertheless is fairly easy to configure.
More experienced students may want to try their hand at the classic s endmai l software. It is a bit
more difficult to handle than Postfix, but infinitely more flexible.
All Debian Gnu/Linux hosts must have a mail server. For hosts that will only forward all mail to another
server, Postfix and s endmai l are overkill. Many Debian users like to use ex mh for such hosts, but an
even simpler system, such as s s mt p will do the job equally well. You are strongly advised not to install
qmai l unless you have previous experience with it.
Before starting to install the mail server you must know what your goals are. Read the documentation
first and prepare your installation in detail. Postfix documentation is available at
http://www.postfix.org/ and in the Postfix installation itself (man pages and example configuration
files).
Note that access to port 25 (SMTP) is filtered at the ingress to the university network. This means that
you will not be able to send mail to your network from hosts outside the university. You can, however,
send mail from your student account.
Testing tip: When testing, try to keep the interactions between parts of the system to a minimum. For
example, when testing if your server can receive mail, it is a good idea to start with a test case in which
IDA/ADIT
EMB: E-MAIL BASICS
3
you run SMTP manually using e.g. telnet to connect to the server. That way, you won’t encounter
problems with e.g. the mail user agent or DNS (which isn’t completely configured for mail yet). Once you
know raw SMTP is working, you should have a test case that involves a mail user agent. Similarly, when
testing whether a client can send mail through your server, you should try manual SMTP first, perhaps
followed by use of the s endmai l command on the client, and only when that works try using a MUA.
Exercise 3: Mail server installation and basic configuration
3-1
m
Install the mail server of your choice. If you are unsure what to install, use Postfix. Your initial
installation should meet the following requirements (minimum):
(a)
Accept mail from any SMTP server in the world.
(b)
Be able to send mail to any SMTP server in the world.
(c)
Not accept mail for any other destinations than your domain.
(d)
Meet the requirements of RFC 2821, section 4.5.1 concerning the postmaster
address.
3-2
Configure your clients to always use your mail server for outgoing mail. Client configuration is
quite simple: they forward everything, untouched, to the central mail server. Clients must not
accept connections from outside computers.
Report:
Automated test cases that verify that your system meets the requirements.
When you have come this far you should understand how the email server works. The following
questions test your knowledge of key parts of the mail server. You should not proceed beyond this point
before you can answer these questions, or you are likely to run into problems later.
Exercise 4: About the mail server
4-1
Report:
Answer the following questions about your mail server:
(a)
What configuration files does the software use and what are they for?
(b)
Which command displays the current queue of outgoing mail?
(c)
Where is incoming email stored before being delivered?
(d)
Where is outgoing email queued before delivery?
(e)
Where does the mail server log messages about delivery?
(f)
To where (directory and/or file) and in what format does the mail server deliver
mail to end users?
Answers to the questions above.
DNS and mail service
For e-mail service to really work, you will need to configure DNS correctly. When an e-mail message is to
be delivered, a DNS-aware mail server (which means pretty much every mail server currently running)
will search for an MX record for the recipient’s domain. MX records indicate hosts that can accept mail
for a particular domain (known as Mail eXchangers). Each domain may have several MXs, each with a
priority. In your case there should be a single MX – your mail server.
Testing tip: When testing the following exercise, start by testing DNS alone, before you try delivering
mail in a way that depends on DNS. That way, if you run in to problems with mail delivery, at least you
know that DNS is functioning properly.
Exercise 5: Configure DNS for e-mail service
5-1
Add the appropriate resource record(s) to your DNS server.
Report:
No report required.
IDA/ADIT
EMB: E-MAIL BASICS
4
Part 2: End user support
Sending mail isn’t enough. Users tend to want to read mail as well (strange as that may sound). Today,
the preferred way to read mail in a client-server setting is the IMAP protocol. You will implement a basic
IMAP server on your system.
The recommended IMAP server is Courier. However, Courier requires that your mail server delivers mail
to users in so-called maildir format (the default delivery format is usually mbox format). If you are
unwilling to reconfigure your mail server to deliver mail in maildir format, consider using uw-imapd as
the IMAP server instead. You are advised not to use Cyrus. The Cyrus IMAP server is very capable, but its
strengths are in environments with a high volume of mail and a very large number of users. It is not well
suited to small installations.
When installing the IMAP server, one of the things you need to check is that the IMAP server’s idea of
where and in what format user mailboxes are matches the mail server’s. For example, if the IMAP server
expects mail to be delivered in maildir format to a directory in the user’s home directory, but the SMTP
server delivers to mbox files in /var/mail, you won’t be able to read mail using IMAP.
Users must be able to read email via IMAP from both your server and clients.
Testing tip: As with SMTP, you should at least consider having a basic test case in which you run IMAP
manually to ensure that you can distinguish failures in the IMAP software from other failures (e.g.
networking, DNS or client configuration). You should also test IMAP with a real client, such as mutt or
thunderbird (and these can be run on any computer on the Internet).
Troubleshooting tip: In the past, a few groups have had a lot of problems with this part (most haven’t).
If you do run in to problems, you need to understand how IMAP works. That way, you can run a manual
IMAP session (so you can see if the problems you have are in the client or on the server) and you can
snoop the IMAP session (using wi r es har k or by setting up a relay using s oc at – ask your lab
assistant for help) and understand what’s going on.
Exercise 6: Implement IMAP server
6-1
Install and configure and IMAP server that users on your clients and on the lab workstations
can connect to, to read e-mail.
6-2
Where and in what format does your IMAP server expect user mailboxes to be?
Report:
Automated test cases that verify that your system meets the requirements.
If you elect to use Courier, then install the fam package first. Otherwise courier may not work correctly.
Expert options
If you want to try your hand at a more realistic implementation, configure SSL support in the IMAP
server. Real sites will have SSL enabled to avoid sending passwords in the clear. If you want to test IMAP
over SSL, you should be able to use s oc at to make the connection (rather than t el net ). If your IMAP
server doesn’t support SSL natively, then use s t unnel as a frontend.
Part 3: Intermediate mail service
Your mail service up to this point is technically adequate, but would not meet end-user demands. You
still need some basic configuration and some add-on services.
m
The clamav and spamassassin (spamd) processes (spam and virus filtering) use a lot of memory. Your
server is configured to have 128Mb of memory by default, but the default configurations of these tools
will almost certainly exhaust that. If you start running in to memory problems, stop the spamassassin
and clamav processes, then configure the start scripts of spamassassin to start just one child process. If
you still run in to problems, you may consider not running spamassassin and clamav as daemon
processes, but start them on each incoming mail.
IDA/ADIT
EMB: E-MAIL BASICS
5
User-friendly addresses
At this point, all outgoing mail probably has your mail server’s fully qualified domain name as the sender
domain. This is generally undesirable as you don’t want to expose the mail server’s name to end users.
Furthermore, many organizations like outgoing mail to be stripped of internal user IDs. A common (but
seriously flawed) system is to have outgoing mail look like it is sent from firstname.lastname@domain
instead of userid@domain, which is what you would get with no special configuration.
Transforming addresses like this is an example of address rewriting, a major function of all mail servers.
Other examples of address rewriting include appending a domain name to sender addresses that don’t
have one and converting between different address formats. One thing to keep in mind is that rewriting
rules are applied in a specific order, and may be very specific in what they rewrite. When working with
address rewriting, make sure you know which order the various rules are applied in, and exactly to
what, and how, each rule is applied. You will need to review both the Postfix manual (if you are using
Postfix) and the manual pages for the various files involved.
Exercise 7: Configuring user-friendly addresses
7-1
Configure your e-mail server so that all outgoing mail has your domain name, without any
host name, to the right of the @ sign, regardless of which machine the e-mail was sent from.
No configuration is to be done on the clients; just on the server.
7-2
What are e-mail aliases and how are they configured?
7-3
Configure your e-mail server to accept mail to firstname.lastname@your-domain. Mail must
be delivered to the appropriate user. There are at least three different ways of doing this
some of which involve rewriting the address, and some of which that don’t. Solutions that
don’t involve address rewriting are typically easier to work with.
7-4
What are sender canonical maps, and how are they configured?
7-5
Configure your e-mail server to rewrite outgoing mail addresses to firstname.lastname format
automatically. For this to work, you may need to configure your clients to send all mail
without the hostname, with only the hostname, or with something like localhost.localdomain
after the @ sign; the documentation on the address rewriting method you use explains why.
7-6
Formulate a policy for firstname.lastname e-mail addresses that is suitable for a large
organization and that specifies how to handle naming conflicts in a consistent, fair and
management-acceptable way.
Report:
Automated test cases that verify that your system meets the requirements.
Spam protection
Every e-mail server today needs strong spam protection. Spam protection is a fairly hard problem. There
are a number of concerns that must be dealt with:
False positives
False positives are non-spam messages that the system (incorrectly) detects as
spam. False positives generally result in customers complaining about lost e-mail.
False negatives
False negatives are spam messages that are not detected as such. False negatives
generally result in customers complaining about spam.
Processing time Spam filtering can be resource intensive. A large mail system may need several
dedicated high-end computers to do nothing but spam filtering. That generally results in
complaints from management about costs.
rd
3 party impact Spam filtering should never impact a third party (i.e. anyone other than the evil
spammer and good-as-gold victim). I’ll rant about that a little later.
There are a number of different types of spam protection. Some types include content filters, DNSbased blacklists, challenge-response protocols and protocol hacks.
IDA/ADIT
EMB: E-MAIL BASICS
6
Content filters
These tools look at the content of each message and use keywords, combinations of keywords, and
other features to determine if a message is spam. Content filters can be static or dynamic. Static content
filters are like expert systems, with a set of fixed rules that are applied to each message. Dynamic filters
learn from example; each individual user must train the filter by indicating messages that are spam and
messages that aren’t. The filter will automatically learn to distinguish one from the other.
Static content filters are fairly effective if the set of rules is updated frequently, and are suitable in
installations with multiple users. Dynamic content filters are extremely effective, but each user must
train the filter individually, or the number of false positives and false negatives will be too high. Static
content filters are resource intensive.
DNS-based blacklists
DNS-based blacklists use special DNS servers to determine if the sending host is a legitimate sender.
Blacklists can also be used to detect if a message has passed through a known open relay or otherwise
insecure system (this is a good indication of spam). DNS-based blacklists are fast, but controversial as
the policies of the publishers are not always clear and can be somewhat draconian.
Challenge-response
Challenge-response protocols are extremely effective at preventing spam, but they accomplish this by
impacting third parties. Challenge-response systems will reply to messages from unknown senders with
a challenge, which the sender must respond to (usually with another e-mail or by visiting a web site).
Since most spammers use forged sender addresses, and often use valid addresses, challenges to spam
messages tend to go to uninvolved third parties. In effect, users of challenge-response spam filters are
making other people manually filter their spam for them.
Protocol hacks
Some newer methods of filtering use protocol hacks to determine if a message is spam or not. For
example, SPF (Sender Protected Framework) uses DNS records to indicate valid originating mail servers
for a domain (thus making it harder to forge senders in that domain), and Greylisting uses the SMTP
protocol to implement what nearly amounts to a challenge-response protocol without the third party
impact. Other possibilities also exist, but few are in widespread deployment. Nolisting exploits how DNS
lookups for e-mail servers are supposed to be done.
Exercise 8: Implementing spam filtering
8-1
Configure your mail server so Spamassassin can examine each message before it is delivered.
Spam should be marked as such, but not bounced or deleted. Use spamc/spamd.
8-2
Implement greylisting in your mail server. Ensure that local clients are not subjected to
greylisting.
8-3
Configure your DNS server to include SPF records for your domain. The only acceptable as a
source of e-mail from your domain is your server. The SPF record should reflect this.
Report:
Automated test cases that verify that your system meets the requirements.
Testing tip: When testing spam filtering you need a message that will be categorized as spam. Most
spam filters will react to the Generic Test for Unsolicited Bulk Email test string. See
http://en.wikipedia.org/wiki/GTUBE for more information.
Testing tip: Your tests for greylisting should include manual SMTP sessions. In fact, this is the easiest
way to test that greylisting works, once you understand how it is supposed to work. This is also a good
time to do some regression testing to ensure that your own clients can still send mail without it being
greylisted. If you find that your clients are unable to send mail (or are subject to greylisting) at this point,
you probably need to review the configuration changes you made, to ensure that you are aware of all
the implications of the changes.
Testing tip: Testing SPF requires either a testing service or a mail server that uses SPF records. If you
have access to a GMail account, you can view all the headers of mail sent mail to it from your system
IDA/ADIT
EMB: E-MAIL BASICS
7
and see how it reacts to your SPF records. You can also use the on-line SPF tester at
http://www.dnsstuff.com/pages/spf.htm.
Virus filtering (optional)
This section is optional. If you want to try your hand at setting up a more complete service, then
proceed, but if you’ve had enough of e-mail configuration, stop.
Your mail server should have good protection against viruses. Since many viruses are delivered by email
today it has become common practice to implement server-side virus protection, even on Unix type mail
systems. Today, there are a great number of both commercial and free alternatives to choose from.
Exercise 9: Questions about server-based virus scanning
9-1
What is the most important benefit of server-based virus scanning over client-based virus
scanning (where each e-mail client scans for viruses in incoming messages)?
Report:
Answers to the questions above.
Regrettably, many implementations of server-based virus filters have a high impact on third parties. For
various reasons, server-based virus scanning is often done after the SMTP dialog is completed (scanning
is resource intensive, and some antivirus products do not integrate well with mail servers). Many
antivirus products are configured by default to send a reply to the originator of the virus-infected e-mail.
Unfortunately, this reply is hardly ever seen by the real sender (whose host is infected) since practically
every e-mail virus forges the sender address.
Exercise 10: Implementation of virus scanning
10-1
Implement server-based virus scanning on your mail server. Use ClamAV for virus scanning. If
a virus is detected, you should not reply to the sender. It is acceptable to either notify the
recipient that a virus has been found, or generate an SMTP error message (this assumes that
scanning takes place during the SMTP dialog).
Report:
Automated test cases that verify that your system meets the requirements.
Combining virus filtering and spam filtering with amavisd-new
Amavisd-new is free software which can be used to connect an email server with other programs, and it
is quite simple to set up. One problem with amavisd-new is that it prevents spamassassin from rewriting
e-mail headers, so if you choose to use amavisd-new, you need to think hard about how to make spam
filtering really work (and how to demonstrate it).
If you are planning to use amavisd-new together with Postfix, LMTP is the preferred solution.
Expert options
If you want to try your hand at some more advanced SMTP configuration, you could try implementing
support for SSL (encrypted connections) either using a dedicated port for SMTP over SSL or by
supporting STARTTLS. You can also implement SMTP authentication, so that users can use your mail
server to send mail even if they are not on your network, as long as they can authenticate. If you want
to test SMTP over SSL (not sure if this can be made to work with STARTTLS) you should be able to use
s oc at to make the connection (rather than t el net ).
IDA/ADIT
EMB: E-MAIL BASICS
8
FEEDBACK FORM
EMB
Complete this feedback form individually at the end of the lab and hand it to the lab assistant when you
finish. Your feedback is essential for improving the labs. Each student should hand in a feedback form.
Do not cooperate on completing the form.
You do not need to put your name on the feedback form. Your feedback will be evaluated the same way
regardless of whether your name is on it or not. Your name is valuable to us in case you have made and
comments in the last section that need clarifications or otherwise warrant a follow-up.
For each section, please rate the following (range 1 to 5 in all cases).
 Difficulty: Rate the degree of difficulty (1=too easy, 5=too difficult)
 Learning: Rate your learning experience (1=learned nothing, 5=learned a lot).
 Interest: Rate your interest level after completing the part (1=no interest, 5=high interest).
 Time: How long did the part take to complete (in minutes)?
Difficulty
Learning
Interest
Time
(minutes)
Preparation
Part 1: Basic mail server
Part 2: End user support
Part 3: Intermediate mail service
Overall
Please answer the following questions:
 What did you like about this lab?
 What did you dislike about this lab?
 Make a suggestion to improve this lab.
IDA/ADIT
EMB: E-MAIL BASICS
FEEDBACK FORM
EMB
Complete this feedback form individually at the end of the lab and hand it to the lab assistant when you
finish. Your feedback is essential for improving the labs. Each student should hand in a feedback form.
Do not cooperate on completing the form.
You do not need to put your name on the feedback form. Your feedback will be evaluated the same way
regardless of whether your name is on it or not. Your name is valuable to us in case you have made and
comments in the last section that need clarifications or otherwise warrant a follow-up.
For each section, please rate the following (range 1 to 5 in all cases).
 Difficulty: Rate the degree of difficulty (1=too easy, 5=too difficult)
 Learning: Rate your learning experience (1=learned nothing, 5=learned a lot).
 Interest: Rate your interest level after completing the part (1=no interest, 5=high interest).
 Time: How long did the part take to complete (in minutes)?
Difficulty
Learning
Interest
Time
(minutes)
Preparation
Part 1: Basic mail server
Part 2: End user support
Part 3: Intermediate mail service
Overall
Please answer the following questions:
 What did you like about this lab?
 What did you dislike about this lab?
 Make a suggestion to improve this lab.
IDA/ADIT
EMB: E-MAIL BASICS