Keith W. Ross
Brooklyn Polytechnic
David A. Turner
CSU San Bernardino
Jussi Kangasharju
TU Darmstadt
Secure and Resilient Peer-to-Peer Email:
Design and Implementation
Darmstadt University of Technology
Department of Computer Science
Telecooperation
1
Contribution
• Prototype implementation
– Causes of unavailability in DHT storage
– Degree of replication
• Reliability analysis of P2P storage
– Eliminates need to rely on single server
– Boost resilience of email against attacks
– Provides confidential communications
• Architecture for peer-to-peer email
Darmstadt University of Technology
Department of Computer Science
Telecooperation
2
• Peer-to-peer architecture alleviates problems
• Neighborhood communities?
– Storage stress (big attachments)
– Additional processing (spam & viruses)
• Server-centric has other problems
– Distributed clusters Æ expensive, still vulnerable
• Single-server architecture vulnerable
– Reliable servers expensive
• Email is mission critical for many institutions/people
What is Wrong with Current Email?
Darmstadt University of Technology
Department of Computer Science
Telecooperation
3
Assumptions
• Analyze requirements of P2P email architecture
– Only message delivery; permanent storage locally
– Assume user has dedicated computer (revisit later)
• Store-and-forward email architecture
– Distributed Hash Table (DHT) is available (e.g., Chord)
– DHT gives k closest nodes to given key (currently up)
– No P2P storage layer (e.g., CFS, PAST)
• What we assume to have:
– Peers up/down, peers independent
• Community of peers (nodes)
Darmstadt University of Technology
Department of Computer Science
Telecooperation
4
• Similar to POP-email
– Headers stored in inbox
– Message bodies separately
– Message-ID header as key
• Messages
– Address certificate
– Inbox
• Users:
– UA and SN can be same or
different
alice-inbox
bob-cert
Entities
• Peers in DHT-space
• System nodes store data
• User agents access data
Darmstadt University of Technology
Department of Computer Science
Telecooperation
5
alice-cert
bob-inbox
– Remove object from nodes
– Check authority
– Deletion not guaranteed
Æ garbage collection
• Delete
– Retrieves objects from any
or all of k system nodes
• Fetch
– Stores objects on k system
nodes closest to object’s
identifier
– List of authorized persons
• Store
• Note: No need to enforce
consistency between copies
– Read from all k nodes
– Return all headers from inbox
– Clears inbox atomically
• Read-inbox
– Append headers to inbox
– Inboxes not consistent
Æ Form superset when reading
• Append-inbox
Service Primitives
Darmstadt University of Technology
Department of Computer Science
Telecooperation
6
• Same view to user!
– Headers encrypted
• Store message
• Append headers to Bob’s
inbox
– Alice picks session key
– Encrypt message with
session key
– Encrypt session key with
Bob’s public key
• Alice fetches Bob’s
certificate
• Alice writes message
store(msg, bob)
fetch(bob-certificate)
append-inbox(hdrs)
Alice Sends a Message to Bob
Darmstadt University of Technology
Department of Computer Science
Telecooperation
7
– Delete from all k
– Garbage collection
• Bob fetches message
• Message deleted
– Read from all k nodes,
form superset
– Inboxes cleared
• Bob fetches his inbox
read-inbox(bob)
fetch(msg)
delete(msg)
Bob Reads His Messages
Darmstadt University of Technology
Department of Computer Science
Telecooperation
8
• Simple storage system: Object created once, read
many times, no modifications to object
• Assume I peers, peers homogeneously up/down (p),
uniformly distributed in hash space
– What can go wrong?
• Later peer B wants to read O
– Create k copies on different peers
– k peers determined by DHT for each object (k closest)
• Storage system using a distributed hash table (DHT)
• Peer A wants to store object O
Reliability of Data in DHT-Storage
Darmstadt University of Technology
Department of Computer Science
Telecooperation
9
3 Causes of Loss
i=k
∑
i
−
(
1
)
p
I

 p(1 − p ) 



I

 i 
N
N 1
pl 3 ≈ ∑   i
i =k  i  I
3. At least k peers join and become new closest peers
pl 2 ≈
(1− p ) I
2. Real k closest peers were down when A wrote and
are up when B reads
pl1 = (1 − p ) k
1. All k peers are down when B reads
Darmstadt University of Technology
Department of Computer Science
Telecooperation
10
Results
– Increase k Æ waste storage (maybe not a problem?)
• Needs additional coordination
• Possible if down-events controlled
• Crash Æ others need to detect crash (before they crash)
– Maintain storage invariant Æ O always at k closest
• How to improve?
– For cases 2 and 3 applies:
Search more than k nodes
• First case dominates clearly
Darmstadt University of Technology
Department of Computer Science
Telecooperation
11
Every user’s action needs to access several objects
For each access: ps = 1 – pl1 = 1 – (1 – p)k
Reading and sending need 2 objects
Success for user: pt = (1 – (1 – p)k)2
log(1 − pt )
k=
log(1 − p )
• Solving for k:
•
•
•
•
What the User Sees?
Darmstadt University of Technology
Department of Computer Science
Telecooperation
12
0
0.1
20
40
60
80
100
120
140
160
180
200
0.2
0.3
0.75
0.8
0.85
0.4
0.5
0.6
0.7
Individual peer up probability, p
0
5
10
15
Zoom
0.8
0.9
0.95
0.9
1
pt = 99%
p = 99.9%
t
p = 99.99%
t
p = 99.999%
t
1
• Increase in pt Æ
small increase in k
• When peers mostly
up, k small
13
– This is what user sees
– Failures temporary
• Define target pt
How Large Should k Be?
Darmstadt University of Technology
Department of Computer Science
Telecooperation
Number of copies needed, r
• MX-support needed for others to
send mail to org. X
– Gateway splits messages
• Outgoing gateway can be
sending peer
• Incoming gateway’s address
must be in DNS (MX-query)
– Outgoing and incoming
• Organization X replaces old
email with ours
• How to talk to others?
• Designate gateways
Message
Alice’s mailserver
Headers
Gateway
XX
Interoperability and Migration
Darmstadt University of Technology
Department of Computer Science
Telecooperation
14
Java Prototype
– Some new headers
• Object transfer over
HTTP/1.1
– maxOps
– maxConns
• Parameters for tuning
communications
– System node
– User Agent
• Two components:
Darmstadt University of Technology
Department of Computer Science
Telecooperation
15
No permanent storage
• Turn off garbage collection?
• Enforce replication in UA?
• Still, need 10-20 times
storage of central server
• Storage on peers is free!
Requirements:
• DHT substrate
• No need for storage system
• “No need” for consistency
Æ Mobility = Trusted access
• Problem: Need to access
private key often...
How about mobility?
• Information about folders
and read messages?
• Store email metadata
(folders, etc.) on peers
• Offer permanent storage
Æ Full access on the move
Discussion and Future
Darmstadt University of Technology
Department of Computer Science
Telecooperation
16
Related Work
– POST uses convergent encryption (some weaknesses)
– POST stores inbox on user’s computer
Æ Mobility requires keeping own computer on
Æ Still applies: mobility = trusted access to own computer
– Notification if user is online vs. periodic polling
• POST by A. Mislove et al.
• Independent work in parallel
• POST relies on Pastry DHT, PAST storage layer, and
Scribe multicast system (notifications)
• Main differences:
Darmstadt University of Technology
Department of Computer Science
Telecooperation
17
Conclusion
• Java prototype implementation
– How many copies needed for given target quality
• Reliability analysis of P2P storage
– DHT, ability to store objects, no consistency
• Analyze requirements
– Can be extended
• Design and implementation of P2P email
• Store-and-forward architecture
Darmstadt University of Technology
Department of Computer Science
Telecooperation
18