Email System
– The most important application of computer networks
Dr. David MacQuigg
Research Associate
Autonomic Computing Laboratory
University of Arizona
ECE 478
3 December 2009
1
• The “intractable” problems with email
–
–
–
–
Spam & lost messages, $20B/year
Fraud and other serious crimes
Enabler for most malware
Threats to critical infrastructure
• Reasons for these problems
– Ignorance
– Identity fraud (can’t separate the good guys)
– Investment in the status quo ($2B per year)
• Possible technical solutions
– More of the same (IP blacklists, statistical filters)
– Reputation-based systems
March 14, 2016
2
• The problems with email
–
–
–
–
Spam problem, $20B/year, “intractable”
Fraud and other serious crimes
Enabler for most malware
Threats to critical infrastructure
• Reasons for these problems
– Ignorance (users and admins)
– Identity fraud (can’t separate the good guys)
– Investment in the status quo ($2B per year)
• Possible technical solutions
– More of the same (IP blacklists, statistical filters)
– Reputation-based systems
March 14, 2016
3
• The problems with email
–
–
–
–
Spam problem, $20B/year, “intractable”
Fraud and other serious crimes
Enabler for most malware
Threats to critical infrastructure
• Reasons for these problems
– Ignorance
– Identity fraud (can’t separate the good guys)
– Investment in the status quo ($2B per year)
• Possible technical solutions
– More of the same (IP blacklists, statistical filters)
– Reputation-based systems
March 14, 2016
4
Email
System
detailed model
http://en.wikipedia.org/
wiki/Internet
March 14, 2016
5
The Internet Today
???
X
Trusted Domains
Our Domain
March 14, 2016
6
Textbook Model of the Email System
Figure 9.1 Sequence of mail relays store and forward email messages
{Peterson & Davie, Computer Networks, 4th ed.}
March 14, 2016
7
Real Mail Handling System
P. Faltstrom, mail-flows-0.4, Jan 6, 2004, http://www.ripe.net/ripe/meetings/ripe-47/mailflows.pdf
March 14, 2016
8
Relay-Level Model
Function modules and the
protocols used between them
D. Crocker, "Internet Mail Architecture", 2009,
http://tools.ietf.org/html/rfc5598.
March 14, 2016
9
Administrative-Level Model
Administrative Management
Domains (ADMD)
+--------+
+---------+
+-------+
+-----------+
| ADMD1 |<===>| ADMD2 |<===>| ADMD3 |<===>|
ADMD4
|
| ----- |
| ----- |
| ----- |
|
----|
|
|
|
|
|
|
|
|
| Author |
|
|
|
|
| Recipient |
|
.
|
|
|
|
|
|
^
|
|
V
|
|
|
|
|
|
.
|
| Edge..+....>|.Transit.+....>|-Edge..+....>|..Consumer |
|
|
|
|
|
|
|
|
+--------+
+---------+
+-------+
+-----------+
Legend: === lines indicate primary (possibly indirect)
transfers or roles
... lines indicate supporting transfers or roles
D. Crocker, "Internet Mail Architecture", 2009,
http://tools.ietf.org/html/rfc5598.
March 14, 2016
10
The Email System ( a better textbook model )
|--- Sender's Network ---|
|-- Recipient's Network -|
/
Author ==> MSA/Transmitter --> / --> Receiver/MDA ==> Recipient
/
Border
http://en.citizendium.org/wiki/Email_system
March 14, 2016
11
Shorthand Notation for Email System Models
Simple Setup with four Actors
|--- Sender's Network ---|
|-- Recipient's Network -|
/
Author ==> MSA/Transmitter --> / --> Receiver/MDA ==> Recipient
/
Border
Actors, Roles and Notation
Actors include Users and Agents.
Agents may play more than one role, but no role has more than one Actor.
Typical roles include Transmitting, Receiving, Forwarding, and Delivery.
A Border occurs when there is no prior relationship between Agents.
--> Direction of mail flow (no statement as to relationship)
~~> Indirect relationship (e.g. both directly related to Recipient)
==> Direct relationship between Actors (e.g. a contract)
A/B Roles A and B both played by the same Actor
March 14, 2016
12
Other Common Setups
Simple Forwarding is quite common
|-------- Recipient's Network ---------|
/
--> / --> Receiver/Forwarder ~~> MDA ==> Recipient
/
Border
Chain Forwarding should be discouraged
|------------ Recipient's Network ------------|
/
--> / --> Receiver ~~> Forwarder(s) ~~> MDA ==> Recipient
/
Border
Open Forwarding must be banned
/
/
|-- Recipient's Network -|
--> / --> Forwarder --> / --> Receiver/MDA ==> Recipient
/
/
Border
Border
March 14, 2016
13
Roles and Responsibilities
Author
- Originate messages
- Provide a password or other means of authentication
MSA - Mail Submission Agent
- Authenticate the Author
- Manage Author accounts
Transmitter
- Spam Prevention
- rate limits, content analysis, alerts
- respond to spam reports
- maintain reputation
- Authentication
- RFC compliance
- IP authorization (SPF, SID, CSV, ...)
- signatures & key management (DKIM ...)
Receiver
- Block DoS
- Authenticate Sender
- HELO, Return Address, Headers, Signature
- reject forgeries
- Assess reputation
- whitelists
- Filter spam
- Add authentication headers
- Manage Recipient accounts/options
- whitelisting, blacklisting, filtering, blocking, forwarding
- Process spam reports
March 14, 2016
14
Roles and Responsibilities (continued)
Forwarder
- Authenticate upstream Agent
- Set up forwarding to downstream Agent
- check RFC compliance
- set up authentication records
- submit forwarding request, wait for approval
- Manage Recipient accounts
- maintain database of forwarding addresses
- suspend account when a message is rejected
- communicate w Recipient re "
"
- Maintain reputation as a trusted Forwarder
- certifications
MDA - Mail Delivery Agent
- Authenticate upstream Agent
- Sort and store messages
- Provide access for Recipients
- POP3, IMAP, Webmail
- Manage Recipient accounts/options
- Relay spam reports to Receiver (or don't accept them)
Recipient
- Set up accounts with each Agent
- Select options in each account
- Report spam to Receiver
March 14, 2016
15
Secure Communications
Secure communications may require any or all of:
1)authentication of the source (individual or organization identity)
2)verification of content (digital signature)
3)confidentiality of content (encryption)
4)originality (no duplicates)
5)timely delivery (no unexpected delays)
6)hidden communication (keeping an enemy unaware)
Solving the problems of bulk email abuse (spamming, phishing and other bulk
mail scams) requires that we address items 1 and 4.
To be useful in email authentication, an identity must have three characteristics.
It must be unique, verifiable, and suitable for accumulation of reputation.
http://en.citizendium.org/wiki/Email_authentication
March 14, 2016
16
Identities in an Email Session
Author ==> MSA/Transmitter --> / --> Receiver/MDA ==> Recipient
1
3
6
4
$ telnet open-mail.org 25
220 open-mail.org ESMTP Sendmail 8.13.1/8.13.1; Wed, 30 Aug 2006 07:36:42 -0400
HELO mailout1.phrednet.com
250 open-mail.org Hello ip068.subnet71.gci-net.com [216.183.71.68], pleased to meet you
MAIL FROM:<macquigg@box67.com>
2 Network Owner
250 2.1.0 <macquigg@box67.com>... Sender ok
RCPT TO:<jman@box67.com>
250 2.1.5 <jman@box67.com>... Recipient ok
DATA
354 Enter mail, end with "." on a line by itself
From: Dave\r\nTo: Test Recipient\r\nSubject: SPAM SPAM SPAM\r\n\r\nThis is message 1 from our test
script.\r\n.\r\n
250 2.0.0 k7TKIBYb024731 Message accepted for delivery
QUIT
221 2.0.0 open-mail.org closing connection
RFC-5321
1 Helo Name
Envelope Addresses:
3 Return Address
6 Recipient Addresses
March 14, 2016
RFC-5322
Header Addresses:
From Address
4
Reply-To Address
5
17
Email Authentication – The Challenge
SMTP makes forgery easy
Forger -------> /
/
Author ==> MSA/Transmitter --> / --> Receiver/MDA ==> Recipient
/
/
/
/
Border
/
/
/
-- Secure Channel --
TCP makes IP addresses (relatively) secure
The source address is real, but it may be only a zombie!
DNS offers a (relatively) secure channel
Domain owners can publish their transmitter addresses
Or they can publish a public key
Nothing else can be trusted
March 14, 2016
18
Email Authentication Summary
|--- Sender's Network ---|
|--------- Recipient's Network --------|
/
Author ==> MSA/Transmitter --> / --> Receiver/Forwarder ~~> MDA ==> Recipient
/
/
/
/
Border
/
/
/
------ DNS -------
IP-based Authentication (SPF, SenderID, CSV):
Sender provides a list of authorized transmitter addresses via DNS.
Can be very efficient (no data transfer)
but may have a “forwarding problem” if the MDA thinks it is the Receiver.
Signature-based Authentication (DKIM):
Sender provides a Public Key via a DNS.
Messages are signed with the related Private Key.
Message content can be very secure,
but an un-trusted Forwarder can replay it to millions.
March 14, 2016
19
Analysis of SPF using our models
Simple Forwarding
|-------- Recipient's Network ---------|
/
MSA/Transmitter --> / --> Receiver/Forwarder ~~> MDA ==> Recipient
/
Border
SPF correlates the Return Address to the Transmitter’s IP address.
Forwarders are expected to re-write the Return Address.
Very few forwarders are doing that. “Proselytizing” has failed.
A misconfigured MDA sees the forwarded message as forgery.
The message is quarantined, and possibly lost.
Senders are avoiding the loss by publishing “neutral” SPF records.
Forwarders will not change until senders demand it by publishing
“enforceable” SPF records.
Senders don’t care.
SPF is stuck.
http://en.citizendium.org/wiki/Sender_Policy_Framework
March 14, 2016
20
Reputation – the other half of trust
Millions of legitimate senders are simply unknown
Aggregation of data is essential
Ground Up: Gossip
Top Down: Proprietary Systems
Both: Registry of Internet Transmitters
Some legitimate senders are not qualified to operate a transmitter
Make outsourcing the Transmitter role easy.
Accountability is essential – no excuses.
March 14, 2016
21
Suggested Receiver Setup
March 14, 2016
22
So why isn’t it happening?
Hurdles that proposed solutions have failed to avoid or overcome, in order of
decreasing severity:
1) Required simultaneous upgrades in software or setup (Flag Day)
2) Required widespread adoption by Agents before any benefit is realized by Users
3) Required widespread adoption of one company's method or service (Microsoft patent)
4) Changes that cause a temporary degradation in service
5) Changes in current practices
a) A well-established and standards-compliant practice.
b) A widespread but non-standardized practice. ("Misuse" of Return Address)
c) A widespread but non-compliant practice. (bad HELO name)
d) An already unacceptable practice. (open relays)
6) Costs to senders
a) Must pay a fee, install new software, or incur some administrative cost.
b) Worry about lost messages
c) Need to keep track of their transmitter addresses
The real reason:
Reversed incentives – more spam for everyone else = more money for us
March 14, 2016
23
Bibliography
A short list of the most useful books and articles on email and its underlying technology.
• “Email System”, http://en.citizendium.org/wiki/Email_system - cluster of articles
on how the email system works, email security, authentication methods, etc.
• "Internet Mail Architecture", D. Crocker, http://tools.ietf.org/html/rfc5598 - best
relay-level model, with references to all the relevant RFC standards.
• Computer Networks, Peterson & Davie, 4th ed. – good on all relevant
technologies except email.
• TCP/IP Illustrated, vol. I, The Protocols, W. Richard Stevens, 1994. Very
thorough, yet readable. Good illustrations.
• Pro DNS and BIND, Ron Aitchison, Apress 2005. – Very readable book on the
Domain Name System.
• "CircleID", http://www.circleid.com – a "Collaborative Intelligence Hub for the
Internet's Core Infrastructure & Policies" – current articles by top industry experts.
Project Links
• https://www.open-mail.org – current status of our Identity and Reputation System
• http://purl.net/macquigg/email – articles and notes from early development.
March 14, 2016
24
Economics of Email Abuse
$200B annual benefit of email
$20B cost of abuse
100M users x ($.25/day deleting spam + $100/yr lost emails)
$2B benefit to anti-spam industry
100 companies x $20M/yr
$0.2B benefit to spammers
10K spammers x $20K/yr
$0.02B cost of an effective authentication/reputation system
10M users x $2/yr
100K companies x $200/yr (90% internal, 10% external services)
March 14, 2016
25